Dojo 1.2 and Django 1.0 on Google App Engine 1.1.3

Important Update: Use Google App Engine Patch for Django integration instead. It integrates Django seamlessly and includes lots of other goodies. For Dojo, also consider using xdomain and loading Dojo from Google’s AJAX Libraries API. For Google App Engine Patch, see my more recent post. This area changes quickly, so check the dates on any blog posts you’re looking at!

With the release of Google App Engine 1.1.3 (GAE), it’s now much more realistic to serve up Dojo on GAE. LittleShoot uses a lot of Dojo (mostly because it rocks), so getting Dojo up and running on our new GAE LittleShoot port has been a high priority.  

The problem is that vanilla Dojo quickly runs up against GAE’s 1000 file limit per application. Django has the same problem. Our initial solution was to run a custom build that would delete a lot of unnecessary files, with thanks to Peter Higgins from SitePen for lighting our path. This got our build down to about 700 files, solving the problem for the basic LittleShoot site.

We’re getting ready to release the LittleShoot platform, however, that allows any site to detect if LittleShoot’s installed and to call the LittleShoot “P2P 2.0″ API if it’s available. That means we need external sites to be able to load our JavaScript, i.e. we need to support Dojo’s cross-domain builds. The cross-domain build adds about another 500 files, pushing us again over the 1000 file limit.

Thankfully, GAE 1.1.3 introduces “zipserve,” allowing you to serve static files directly from a zip file. Guido van Rossum deployed a similar technique for serving Django for his fantastic Rietveld code review tool and GAE demo app, but that only works for loading python code. So we copied Guido for loading Django 1.0, and now zipserve allows us to do more or less the same thing for Dojo.  

To serve Dojo using zipserve, you basically modify your app.yaml to include something like the following:

- url: /dojo/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /dijit/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /dojox/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

 

This will look for dojo.zip, dijit.zip, and dojox.zip files in your *top-level* directory, the same directory containing app.yaml. You need to separate the zip files to avoid running up against the GAE limit on file sizes (1MB I believe). The zip file name basically acts like a directory in App Engine’s URL resolution. So, when a request goes to http://beta.littleshoot.org/dojo/dojo.js, for example, it loads dojo.js from dojo.zip (not from the dojo directory within dojo.zip). Here’s our script for generating the zips:

#!/usr/bin/env bash

function die()
{
  echo $*
  exit 1
}

cd static/js
dirs=”dojo dijit dojox littleshoot”

for x in $dirs
do
  cd $x || die “Could not cd to $x”
  echo “Building zip for $x”
  zf=../../../$x.zip
  rm $zf
  zip -rv $zf * || die “Could not create zip”
  cd ..
  # We actually delete all the contents because we don’t want to 
  # include them in GAE.
  rm -rf $x
done

exit 0

Even with separating the zip files, dijit and dojox will still come close to the 1MB limit on file sizes.  We use rsync excludes to get rid of the files we don’t need.  Here’s our bash rsync excludes function, which you can customize to your liking (careful not to exclude anything you need!!):

function excludeRsync
{
  rsync –exclude .svn/ \
        –exclude test/ \
        –exclude tests/ \
        –exclude demo/ \
        –exclude demos/ \
        –exclude soria/ \
        –exclude nihilo/ \
        –exclude grid/ \
        –exclude charting/ \
        –exclude util/ \
        –exclude analytics/ \
        –exclude collections/ \
        –exclude README* \
        –exclude *.psd \
        –exclude *.uncompressed.js \
        –exclude *.commented.css \
        –exclude dijit/templates \
        –exclude dijit/form/templates \
        –exclude dijit/layout/templates \
        –exclude *silverlight* \
        –exclude gfx3d/ \
        –exclude dojo/_base/ \
        –exclude dojo/_base.js \
        –exclude dojo/build.txt \
        –exclude functional/ \
        –exclude off/ \
        –exclude presentation/ \
        –exclude sketch/ \
        –exclude storage/ \
        –exclude wire/ \
        –exclude data/ \
        –exclude dtl/ \

        -avz $tmpDir/js $releaseDir || die “Could not sync”

}

We call the rsync script after running our dojo build.  The dojo build puts everything in the “$tmpDir/js” directory listed on the last line of the rsync.

OK, so there are still some hoops to jump through, but it works!  On the Django 1.0 side, I’d highly recommend copying Guido’s Rietveld settings.  The Google App Engine Django Helper is also useful, but Django 1.0 support hasn’t been released as of this writing.  We’re using Guido’s settings for now — basically the Makefile, make_release.sh, settings.py, and main.py.  Oh, I also tweaked his rietveld.py script, turning it into littleshoot.py.  You can find all of these files in the Rietveld svn here.  Most of the tweaks to these files are pretty obvious when you glance at the scripts — nothing super complicated going on.  Not that it’s a breeze, but each files is individually fairly straightforward.  The scripts will also create a zipped version of django from the django directory, so using svn:externals (described below) is handy.

We load both Django and Dojo with svn:externals. You might be on information overload at this point, but here’s another little nugget if you’re still with me. You can run the following in the respective directories where you want Django and Dojo to reside.

svn –editor-cmd=vim pe svn:externals .

For Dojo, enter the following:

dojo http://svn.dojotoolkit.org/src/tags/release-1.2.0b2/

For Django, try:

django http://code.djangoproject.com/svn/django/trunk/django

That will put Dojo 1.2 beta 2 in the “dojo” directory and the Django trunk in the “django” directory when you run “svn up”.  

If you successfully navigate through all of that, the end result is the latest Dojo and the latest Django running on the spanking new Google App Engine 1.1.3. Web app setups just don’t get any sweeter. If you’re a total geek like me, that’s living the good life! Sad, I know…

About these ads

14 Responses to Dojo 1.2 and Django 1.0 on Google App Engine 1.1.3

  1. Dylan says:

    Thanks for the write-up Adam, and I’m glad to see that LittleShoot is using Dojo! Let us know if you want a mention on the Dojo Spotlight when you launch.

  2. adamfisk says:

    Any time Dylan! I’ll be posting a followup based on James Burke’s comments on the forums — a significantly simpler path using the Google xdomain link. The Spotlight at some point would be fantastic. We’re getting very close to public release, so I’ll be sure to let you know. The GAE site is still tucked away at beta.littleshoot.org, but that’ll be live shortly.

  3. [...] Dojo 1.2 and Django 1.0 on Google App Engine 1.1.3 With the release of Google App Engine 1.1.3 (GAE), it’s now much more realistic to serve up Dojo on [...] [...]

  4. Simon B. says:

    How about loading dojo from google’s Ajax Libraries servers? That way you don’t need to store those files on each and every AppEngine-account, as well as cutting down on the AppEngine bandwidth use. I’d be very interested to read about such a solution!

  5. adamfisk says:

    Yes, Simon, that would make a lot of sense. I have to do lots of cross-domain stuff for embedding code on third-party pages, but you’re 100% right that just using xdomain with the Google or AOL libraries makes a lot of sense.

    I was talking with James Burke from Dojo about exactly this, and I’ve been meaning to get a followup post out the door. Putting the finishing touches on the next LittleShoot release has tied my hands though.

    It’s straightforward to do — just include the Google link and I believe a module path for your own modules, although I need to experiment to make sure. Dojo 1.2 should be up on Google’s servers as of yesterday though.

  6. pollux says:

    Hi, how to mini dojo?
    but when I use the build tool comes with dojo, there are still more 2 thousand files.

    would you like to help me to cut dojo down?

    BTW:I’m using dojo1.3,and use the command below :

    build profile=standard mini=”true” copyTests=”false”

  7. pollux says:

    sorry,my mistake
    the command i used to build dojo:
    build profile=standard mini=”true” copyTests=”false” action=clean,release

  8. adamfisk says:

    You have to create a separate layer for your code using a custom profile. That’s likely you’re issue — you need to do something like:

    build.sh stripConsole=normal copyTests=false mini=true profileFile=myapp.profile.js action=clean,release

    I *think* that’s all still valid with 1.3, but worth double checking. Then just have a look at the docs for creating profiles.

    Hope that helps.

    -Adam

  9. pollux says:

    hi Adam:
    a separate layer? How to do it?
    would you like to give me a sample profile?
    Thanks!

  10. pollux says:

    hi Adam:
    I still have some other questions that how to cut dojo down to less than 1000 files? just separate layer to achieve it? you said that you got build down to 500 files. How to do that? would you like to share your profile? or you delete some file by hands?

  11. pollux says:

    hi Adam:
    It seems that the limitation of the file size has been changed.
    http://code.google.com/p/googleappengine/issues/detail?id=78

    good news!
    So I just have to zip the dojo,dojox and dijit directories and upload them.

  12. adamfisk says:

    Right — without the file size limit you can just zip ‘em and send ‘em up. Sorry – I forgot they made that change.

  13. How can I minimize my dojo just like google does?

    Or is there a way to get the contents of:
    http://ajax.googleapis.com/ajax/libs/dojo/1.2/dojo/

    Anyone that could answer that would be my hero

    -nicholas

  14. Joni Wilmoth says:

    Thanks for your post. I am new at python and this will be a big help.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: