Lantern Bash Configuration Script on OSX

September 4, 2011

I’ve been working for awhile on a piece of software called Lantern. It’s a new censorship circumvention and monitoring-prevention tool built atop the LittleShoot P2P networking layer and also atop LittleProxy, another little project I’ve completely neglected to mention! You’ll be finding out a lot more about Lantern soon, but sign up for the mailing list at if you want to be notified when you can start playing with it. It’ll be fun, and ultimately should help people all over the world who don’t enjoy the relatively unfettered internet most of us take for granted.

I was actually inspired to write this post, because I just added a little bash code snippet for auto-configuring proxies on OSX using networksetup, a pac file, and some nifty for looping to read lines (trickier than you might think!). I thought others might find it useful. Enjoy!

#!/usr/bin/env bash

mkdir ~/Library/Logs/Lantern

function log() {
echo "`date`: $@" >> ~/Library/Logs/Lantern/installer.log

log "Configuring network services"
while read s;
log "Configuring network: $s"
networksetup -setautoproxyurl "$s" file://localhost$HOME/.lantern/proxy.pac || log "Could not set auto proxy URL for $s"
networksetup -setautoproxystate "$s" "on" || log "Could not turn auto proxy on for $s"
log "Configured network: $s"
done < <(networksetup -listallnetworkservices | tail +2)
log "Done configuring network services!!"


GData Maven Deploy Script

July 10, 2010

I just wrote a quick script to deploy the latest GData versions to the LittleShoot maven repository, and I thought others might find it handy. GData doesn’t deploy to a maven repo, so you’ve got to do a little extra work. There are various solutions out there, but this is a little more straightforward. I’m jumping on board with Google’s latest 2.2.x alphas of their GData Java clients, as it looks like that’s where the future action will be. So this only applies to the 2.2.x alphas. The script simply takes the version argument, checks out that version from SVN, and deploys the jar to your repo.

You’ll want to swap out your repository for the LittleShoot repository listed at the end — both the URL and repository ID. If you want to just grab this script from the LittleShoot repository, it’s here.

#!/usr/bin/env bash

# This is a script for automatically deploying the GData jars to the LittleShoot
# maven repository.


function die() {
rm -rf $gdataVersion
echo $*
exit 1

me=`basename $0`

if [ $# -ne 1 ]
"Should be a single GData version argument, as in './$me 2.2.1-alpha'. For versions, see:"
exit 1

echo "Checking out GData version $gdataVersion"
svn co$gdataVersion || die "Could not checkout SVN"

pushd $gdataVersion/gdata || die "Could not move to gdata base dir"
ant dist || die "Could not build with ant"
cd build-bin/dist/ || die "Could not cd to dist directory"

mvn deploy:deploy-file -DartifactId=gdata -Dversion=$gdataVersion -Dfile=gdata-$gdataVersion.jar -Dpackaging=jar -DgeneratePom=true -Durl=scpexe:// -DrepositoryId=littleshoot || die "Could not deploy file"

echo "Cleaning up"
rm -rf $gdataVersion

MIME API from LittleShoot on Google App Engine

July 23, 2009

The other day I needed to add an HTML form to the LittleShoot site to upload files directly to Amazon S3. S3’s “browser-based uploads” allow you to send files directly into S3 using hidden form fields to sign the request as opposed to sending files through your server. Nice and efficient.

Everything was going along fine until I realized one glaring problem: how the heck could I set the Content-Type on the file? S3 requires you to do this in the initial request, in this case using another hidden form field, but how could I determine the MIME type at all? Cut past 30 minutes of digging around, and it turns out you can, of course do the following:

  1. Attach to the form submit event. In JQuery, this is just:
    $("#myForm").submit(function() {...});
  2. Lookup the MIME type in JavaScript (somehow – pay attention to this one)
  3. Dynamically add a hidden “Content-Type” form field just before the upload.

The only problem? To do this, you need to include a full JavaScript library to lookup all MIME types in the submit function above, another 40K+ of JavaScript code. That seems a little excessive for a call you’ll only make every once in awhile.

Now comes the beauty of Google App Engine (GAE). With App Engine, you can fire up a site in a matter of minutes that will scale to millions. With a little help from my favorite App Engine Patch, you can run it all on Django. The speed and ease of deployment are breathtaking: 1 hour to deploy a REST API for MIME lookups, all for free.

The result is  It’s a trivial, ridiculously simple API that basically just exposes the Python “mimetypes” module. It’s the simplest, silliest, and most trivial REST API I’ve ever seen or written, and that’s the whole point. Google App Engine annihilates the barrier to creating this type of API on the only two scales that really matter in this case: time and cost.

I predict we’ll see a lot of similarly trivial services continuing to pop up because they’re so easy. They seem almost stupid at first. Simple building blocks like these are the secret sauce in building any truly powerful platform, however, and with tools like GAE, the web is getting more powerful by leaps and bounds.

If you ever run into the obscure case where you need to lookup a MIME type via an API, check out mimeapi. I built it because it makes more sense than cramming another JavaScript library down my user’s pipes. Maybe it’ll help you too.

Google App Engine Full Text Search From App Engine Patch Team

July 1, 2009

Waldemar Kornewald and the rest of the Google App Engine Patch team have just released the first full text search implementation for Google App Engine (GAE).  LittleShoot has been using App Engine Patch since the early days of App Engine, and we can’t recommend it highly enough.  Its seamless Django integration has saved us countless hours, and features like its tool for combining and compressing all of your JavaScript and CSS using the YUI compressor are just stellar.

While we don’t have an immediate need for full text search over at LittleShoot, we can tell you the team behind GAE full text search is rock solid and battle tested.  Given the various limitations of the GAE datastore API, it’s also quite a technical feat.

If you need full text search, and you’re running on App Engine using python, GAE search will save you a great deal of pain.  Go get it.

Django and More on Google App Engine with App Engine Patch

November 16, 2008

I’ve recently had the chance to play with Waldemar Kornewald’s “Google App Engine Patch” and have come away very impressed. All LittleShoot Google App Engine (GAE) projects now run on it for a couple of simple reasons:

  1. Seamless Django integration (including 1.0.1)
  2. Thorough documentation
  3. Healthy open source development community with an excellent steward in Kornewald and frequent new releases

The Django integration got me first. Almost everything works, such as, Django authentication, Django testing (the original reason I switched), etc. There are also lots of other goodies in there, like support for boto’s SQS module. If you’re unfamiliar with it, boto SQS allows you to call Amazon’s Simple Queue Service (SQS) from Python. That’s a huge step in getting around GAE’s limitation on longer lived, CPU-intensive tasks. Just queue it up in SQS, and your GAE app will keep humming along fine — cloud integration at its finest.

To get started with App Engine Patch, download the zip file from the home page and work off the sample project. The download includes App Engine Patch itself as well as the sample project.  

With Django App Engine Helper not supporting Django 1.0 and seemingly inactive, App Engine Patch is a godsend and gets a huge thumbs up.

Great work Waldemar, and don’t forget to donate.

OSX Uninstallers the Easy Way

October 22, 2008

Back in the old days at LimeWire one of the many tasks I took on was building the installers for all platforms, and I’ve carried on my installer hacking with LittleShoot. While installers are anything but glamorous, I’m oddly obsessed with them I think because they’re your users’ first introduction to your program. They need to be simple, and they need to work.

One of Mac’s quirks has always been a lack of uninstallers, leading to widespread global frustration, decreased productivity, and some say a leading cause of the recent financial crisis (a small minority).

Salvation is at hand.  LittleShoot’s uninstaller is ridiculously simple. It’s a little snippet of AppleScript that just runs a bash file loaded into it’s application bundle using “do shell script” to run an external script file. Within this framework, your entire uninstaller is basically a bash script. Here are the steps to get this working:

  1. Copy the following AppleScript into a file called ‘uninstaller.scpt’ and open it in Script Editor. You can also save the binary script from our SVN here and open it in Script Editor directly (easier).
  2. on onConfirmUninstall()
    set applicationName to “LittleShoot”
    display dialog “Are you sure you want to uninstall ” & applicationName & “?”
    set uninstallScript to quoted form of POSIX path of (path to resource “uninstall.bash”)
    do shell script “bash ” & uninstallScript with administrator privileges
    display dialog “Successfully Uninstalled ” & applicationName buttons {“OK”} default button “OK”

    on error err
    if err contains “User canceled” then
    display dialog “Canceled ” & applicationName & ” Uninstall” buttons {“OK”} default button “OK”
    display dialog “We’re sorry, but there was an error uninstalling ” & applicationName & ” described as: ” & err buttons {“OK”} default button “OK”
    end if
    end try
    end onConfirmUninstall


  3. Change the line ‘set applicationName to “LittleShoot”‘ to ‘set applicationName to “[your application name]”‘
  4. Choose File->Save As… in AppleScript Editor and save this script as an application bundle that’s run only with no startup screen (the options at the bottom when you choose Save As…). 
  5. Navigate via the Terminal to where you saved the bundle and cd into the “Contents/Resources” directory. In our case that’s “,” so it’s Contents/Resources within the app bundle.
  6. Create an uninstall.bash file in Contents/Resources. As you can see in the AppleScript above, that’s the file the AppleScript looks for and executes.
  7. Put whatever you need in uninstall.bash to uninstall your application. The script will run with administrator privileges, so you can really do whatever you want here. Here’s the LittleShoot uninstall script to get you going, although this is a little quirky because we use things like launchd that most applications don’t use. You can also grab this directly from our SVN here.
  8. #!/usr/bin/env bash 

    function die()
      echo $*
      exit 1

    function cleanAndDie()
      die “LittleShoot is already uninstalled”

    function clean()
      local plist=~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist
      test -f $plist && launchctl unload $plist
      rm -rf ~/Applications/
      rm -f ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist
      rm -rf /Library/Receipts/littleshoot.pkg
      rm -rf ~/Library/Receipts/littleshoot.pkg
      rm -rf ~/.littleshoot
      rm -rf ~/Applications/

    function remove()
      rm -rf $1 || die “Could not remove file: $1”

    # If it looks like we’ve already uninstalled, just make sure to remove everything again and die.
    test -e ~/Applications/ || cleanAndDie

    launchctl stop org.lastbamboo.littleshoot || die “Could not stop LittleShoot”
    launchctl unload ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist || die “Could not unload”
    rm -rf ~/Applications/ || die “Could not remove LittleShoot”
    rm -f ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist || die “Could not remove plist”

    # We go through all this because the package file is placed differently on Tiger, Leopard, etc.
    test -e $globalReceipt && remove $globalReceipt
    test -e $localReceipt && remove $localReceipt
    rm -rf ~/.littleshoot || die “Could not remove LittleShoot config folder”
    rm -rf ~/Applications/ || die “Could not remove LittleShoot uninstaller”


So that’s about all she wrote. You basically just have to set up the uninstaller AppleScript application bundle, and then you can just edit your bash script from then on. The application bundle will use whatever script is stored in its Contents/Resources/uninstall.bash file. I’ve played around with a lot of different options for doing this, and the straight script file approach kills the other options in terms of maintainability and flexibility. Note that if the uninstaller encounters an error, the AppleScript will display a dialog to the user with anything your script has echoed, so make those errors informative. You also of course have to include the uninstaller application bundle in your installer.

In the end, your users have a simple uninstaller they can just double click on, and you have a really easy way to write and maintain your uninstaller code.

Back to allowing you to post LittleShoot files to Twitter…

Dojo 1.2 and Django 1.0 on Google App Engine 1.1.3

September 17, 2008

Important Update: Use Google App Engine Patch for Django integration instead. It integrates Django seamlessly and includes lots of other goodies. For Dojo, also consider using xdomain and loading Dojo from Google’s AJAX Libraries API. For Google App Engine Patch, see my more recent post. This area changes quickly, so check the dates on any blog posts you’re looking at!

With the release of Google App Engine 1.1.3 (GAE), it’s now much more realistic to serve up Dojo on GAE. LittleShoot uses a lot of Dojo (mostly because it rocks), so getting Dojo up and running on our new GAE LittleShoot port has been a high priority.  

The problem is that vanilla Dojo quickly runs up against GAE’s 1000 file limit per application. Django has the same problem. Our initial solution was to run a custom build that would delete a lot of unnecessary files, with thanks to Peter Higgins from SitePen for lighting our path. This got our build down to about 700 files, solving the problem for the basic LittleShoot site.

We’re getting ready to release the LittleShoot platform, however, that allows any site to detect if LittleShoot’s installed and to call the LittleShoot “P2P 2.0” API if it’s available. That means we need external sites to be able to load our JavaScript, i.e. we need to support Dojo’s cross-domain builds. The cross-domain build adds about another 500 files, pushing us again over the 1000 file limit.

Thankfully, GAE 1.1.3 introduces “zipserve,” allowing you to serve static files directly from a zip file. Guido van Rossum deployed a similar technique for serving Django for his fantastic Rietveld code review tool and GAE demo app, but that only works for loading python code. So we copied Guido for loading Django 1.0, and now zipserve allows us to do more or less the same thing for Dojo.  

To serve Dojo using zipserve, you basically modify your app.yaml to include something like the following:

– url: /dojo/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

– url: /dijit/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

– url: /dojox/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve


This will look for,, and files in your *top-level* directory, the same directory containing app.yaml. You need to separate the zip files to avoid running up against the GAE limit on file sizes (1MB I believe). The zip file name basically acts like a directory in App Engine’s URL resolution. So, when a request goes to, for example, it loads dojo.js from (not from the dojo directory within Here’s our script for generating the zips:

#!/usr/bin/env bash

function die()
  echo $*
  exit 1

cd static/js
dirs=”dojo dijit dojox littleshoot”

for x in $dirs
  cd $x || die “Could not cd to $x”
  echo “Building zip for $x”
  rm $zf
  zip -rv $zf * || die “Could not create zip”
  cd ..
  # We actually delete all the contents because we don’t want to 
  # include them in GAE.
  rm -rf $x

exit 0

Even with separating the zip files, dijit and dojox will still come close to the 1MB limit on file sizes.  We use rsync excludes to get rid of the files we don’t need.  Here’s our bash rsync excludes function, which you can customize to your liking (careful not to exclude anything you need!!):

function excludeRsync
  rsync –exclude .svn/ \
        –exclude test/ \
        –exclude tests/ \
        –exclude demo/ \
        –exclude demos/ \
        –exclude soria/ \
        –exclude nihilo/ \
        –exclude grid/ \
        –exclude charting/ \
        –exclude util/ \
        –exclude analytics/ \
        –exclude collections/ \
        –exclude README* \
        –exclude *.psd \
        –exclude *.uncompressed.js \
        –exclude *.commented.css \
        –exclude dijit/templates \
        –exclude dijit/form/templates \
        –exclude dijit/layout/templates \
        –exclude *silverlight* \
        –exclude gfx3d/ \
        –exclude dojo/_base/ \
        –exclude dojo/_base.js \
        –exclude dojo/build.txt \
        –exclude functional/ \
        –exclude off/ \
        –exclude presentation/ \
        –exclude sketch/ \
        –exclude storage/ \
        –exclude wire/ \
        –exclude data/ \
        –exclude dtl/ \

        -avz $tmpDir/js $releaseDir || die “Could not sync”


We call the rsync script after running our dojo build.  The dojo build puts everything in the “$tmpDir/js” directory listed on the last line of the rsync.

OK, so there are still some hoops to jump through, but it works!  On the Django 1.0 side, I’d highly recommend copying Guido’s Rietveld settings.  The Google App Engine Django Helper is also useful, but Django 1.0 support hasn’t been released as of this writing.  We’re using Guido’s settings for now — basically the Makefile,,, and  Oh, I also tweaked his script, turning it into  You can find all of these files in the Rietveld svn here.  Most of the tweaks to these files are pretty obvious when you glance at the scripts — nothing super complicated going on.  Not that it’s a breeze, but each files is individually fairly straightforward.  The scripts will also create a zipped version of django from the django directory, so using svn:externals (described below) is handy.

We load both Django and Dojo with svn:externals. You might be on information overload at this point, but here’s another little nugget if you’re still with me. You can run the following in the respective directories where you want Django and Dojo to reside.

svn –editor-cmd=vim pe svn:externals .

For Dojo, enter the following:


For Django, try:


That will put Dojo 1.2 beta 2 in the “dojo” directory and the Django trunk in the “django” directory when you run “svn up”.  

If you successfully navigate through all of that, the end result is the latest Dojo and the latest Django running on the spanking new Google App Engine 1.1.3. Web app setups just don’t get any sweeter. If you’re a total geek like me, that’s living the good life! Sad, I know…