Dojo 1.2 and Django 1.0 on Google App Engine 1.1.3

Important Update: Use Google App Engine Patch for Django integration instead. It integrates Django seamlessly and includes lots of other goodies. For Dojo, also consider using xdomain and loading Dojo from Google’s AJAX Libraries API. For Google App Engine Patch, see my more recent post. This area changes quickly, so check the dates on any blog posts you’re looking at!

With the release of Google App Engine 1.1.3 (GAE), it’s now much more realistic to serve up Dojo on GAE. LittleShoot uses a lot of Dojo (mostly because it rocks), so getting Dojo up and running on our new GAE LittleShoot port has been a high priority.  

The problem is that vanilla Dojo quickly runs up against GAE’s 1000 file limit per application. Django has the same problem. Our initial solution was to run a custom build that would delete a lot of unnecessary files, with thanks to Peter Higgins from SitePen for lighting our path. This got our build down to about 700 files, solving the problem for the basic LittleShoot site.

We’re getting ready to release the LittleShoot platform, however, that allows any site to detect if LittleShoot’s installed and to call the LittleShoot “P2P 2.0” API if it’s available. That means we need external sites to be able to load our JavaScript, i.e. we need to support Dojo’s cross-domain builds. The cross-domain build adds about another 500 files, pushing us again over the 1000 file limit.

Thankfully, GAE 1.1.3 introduces “zipserve,” allowing you to serve static files directly from a zip file. Guido van Rossum deployed a similar technique for serving Django for his fantastic Rietveld code review tool and GAE demo app, but that only works for loading python code. So we copied Guido for loading Django 1.0, and now zipserve allows us to do more or less the same thing for Dojo.  

To serve Dojo using zipserve, you basically modify your app.yaml to include something like the following:

– url: /dojo/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

– url: /dijit/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

– url: /dojox/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

 

This will look for dojo.zip, dijit.zip, and dojox.zip files in your *top-level* directory, the same directory containing app.yaml. You need to separate the zip files to avoid running up against the GAE limit on file sizes (1MB I believe). The zip file name basically acts like a directory in App Engine’s URL resolution. So, when a request goes to http://beta.littleshoot.org/dojo/dojo.js, for example, it loads dojo.js from dojo.zip (not from the dojo directory within dojo.zip). Here’s our script for generating the zips:

#!/usr/bin/env bash

function die()
{
  echo $*
  exit 1
}

cd static/js
dirs=”dojo dijit dojox littleshoot”

for x in $dirs
do
  cd $x || die “Could not cd to $x”
  echo “Building zip for $x”
  zf=../../../$x.zip
  rm $zf
  zip -rv $zf * || die “Could not create zip”
  cd ..
  # We actually delete all the contents because we don’t want to 
  # include them in GAE.
  rm -rf $x
done

exit 0

Even with separating the zip files, dijit and dojox will still come close to the 1MB limit on file sizes.  We use rsync excludes to get rid of the files we don’t need.  Here’s our bash rsync excludes function, which you can customize to your liking (careful not to exclude anything you need!!):

function excludeRsync
{
  rsync –exclude .svn/ \
        –exclude test/ \
        –exclude tests/ \
        –exclude demo/ \
        –exclude demos/ \
        –exclude soria/ \
        –exclude nihilo/ \
        –exclude grid/ \
        –exclude charting/ \
        –exclude util/ \
        –exclude analytics/ \
        –exclude collections/ \
        –exclude README* \
        –exclude *.psd \
        –exclude *.uncompressed.js \
        –exclude *.commented.css \
        –exclude dijit/templates \
        –exclude dijit/form/templates \
        –exclude dijit/layout/templates \
        –exclude *silverlight* \
        –exclude gfx3d/ \
        –exclude dojo/_base/ \
        –exclude dojo/_base.js \
        –exclude dojo/build.txt \
        –exclude functional/ \
        –exclude off/ \
        –exclude presentation/ \
        –exclude sketch/ \
        –exclude storage/ \
        –exclude wire/ \
        –exclude data/ \
        –exclude dtl/ \

        -avz $tmpDir/js $releaseDir || die “Could not sync”

}

We call the rsync script after running our dojo build.  The dojo build puts everything in the “$tmpDir/js” directory listed on the last line of the rsync.

OK, so there are still some hoops to jump through, but it works!  On the Django 1.0 side, I’d highly recommend copying Guido’s Rietveld settings.  The Google App Engine Django Helper is also useful, but Django 1.0 support hasn’t been released as of this writing.  We’re using Guido’s settings for now — basically the Makefile, make_release.sh, settings.py, and main.py.  Oh, I also tweaked his rietveld.py script, turning it into littleshoot.py.  You can find all of these files in the Rietveld svn here.  Most of the tweaks to these files are pretty obvious when you glance at the scripts — nothing super complicated going on.  Not that it’s a breeze, but each files is individually fairly straightforward.  The scripts will also create a zipped version of django from the django directory, so using svn:externals (described below) is handy.

We load both Django and Dojo with svn:externals. You might be on information overload at this point, but here’s another little nugget if you’re still with me. You can run the following in the respective directories where you want Django and Dojo to reside.

svn –editor-cmd=vim pe svn:externals .

For Dojo, enter the following:

dojo http://svn.dojotoolkit.org/src/tags/release-1.2.0b2/

For Django, try:

django http://code.djangoproject.com/svn/django/trunk/django

That will put Dojo 1.2 beta 2 in the “dojo” directory and the Django trunk in the “django” directory when you run “svn up”.  

If you successfully navigate through all of that, the end result is the latest Dojo and the latest Django running on the spanking new Google App Engine 1.1.3. Web app setups just don’t get any sweeter. If you’re a total geek like me, that’s living the good life! Sad, I know…