I was actually inspired to write this post, because I just added a little bash code snippet for auto-configuring proxies on OSX using networksetup, a pac file, and some nifty for looping to read lines (trickier than you might think!). I thought others might find it useful. Enjoy!


#!/usr/bin/env bash

log "Configuring network services"
while read s;
do
log "Configuring network: $s"
networksetup -setautoproxyurl "$s" file://localhost$HOME/.lantern/proxy.pac || log "Could not set auto proxy URL for $s"
networksetup -setautoproxystate "$s" "on" || log "Could not turn auto proxy on for $s"
log "Configured network: $s"
done < <(networksetup -listallnetworkservices | tail +2)
log "Done configuring network services!!"


You’ll want to swap out your repository for the LittleShoot repository listed at the end — both the URL and repository ID. If you want to just grab this script from the LittleShoot repository, it’s here.


#!/usr/bin/env bash

echo "Cleaning up"
popd
rm -rf $gdataVersion


One of my favorite moments had to be in the summer of 2000 when Vinnie Falco, creator of BearShare, and my fellow LimeWire developer and partner in crime in the old days, Chris Rohrs, would talk about Gnutella architecture decisions. Vinnie’s an Italian American who was living in Miami Beach at the time. Chris is an MIT grad who grew up one town away from me in Deerfield, MA, one of the oldest towns in the country (close to my beloved Greenfield), and about as culturally far away from Miami Beach as one can get in the US. Chris would give Vinnie some piece of advice about this or that, and Vinnie would respond with the perfect Miami Beach Italian American accent, “Hey, you’re talking to the Vin.” And that’s how the Gnutella network was built.

Chris has been tearing it up at Google for years now, and Vinnie probably owns some tiny island in the South Pacific where he buried a chest full of BearShare gold back in 2002.

Check out Janko’s more thorough article, the best and one of the few histories of Gnutella I’ve seen written down.

I’ll be writing more here soon — been hammering out so much code for so many different things (all feeding back in to LittleShoot), it’s been a struggle to find time to blog about them. Technology’s progressing so rapidly on so many fronts, it’s more fun to keep coding! Stay tuned for news about LittleProxy, a sweet little HTPP proxy I’ve been working on using Netty. LittleProxy will be a core component of a new system for circumventing censors around the world. Oh yeah, and lots of other stuff too.

Everything was going along fine until I realized one glaring problem: how the heck could I set the Content-Type on the file? S3 requires you to do this in the initial request, in this case using another hidden form field, but how could I determine the MIME type at all? Cut past 30 minutes of digging around, and it turns out you can, of course do the following:

1. Attach to the form submit event. In JQuery, this is just:

   $("#myForm").submit(function() {...});

2. Lookup the MIME type in JavaScript (somehow – pay attention to this one)
3. Dynamically add a hidden “Content-Type” form field just before the upload.

The only problem? To do this, you need to include a full JavaScript library to lookup all MIME types in the submit function above, another 40K+ of JavaScript code. That seems a little excessive for a call you’ll only make every once in awhile.

Now comes the beauty of Google App Engine (GAE). With App Engine, you can fire up a site in a matter of minutes that will scale to millions. With a little help from my favorite App Engine Patch, you can run it all on Django. The speed and ease of deployment are breathtaking: 1 hour to deploy a REST API for MIME lookups, all for free.

The result is http://www.mimeapi.org/.  It’s a trivial, ridiculously simple API that basically just exposes the Python “mimetypes” module. It’s the simplest, silliest, and most trivial REST API I’ve ever seen or written, and that’s the whole point. Google App Engine annihilates the barrier to creating this type of API on the only two scales that really matter in this case: time and cost.

I predict we’ll see a lot of similarly trivial services continuing to pop up because they’re so easy. They seem almost stupid at first. Simple building blocks like these are the secret sauce in building any truly powerful platform, however, and with tools like GAE, the web is getting more powerful by leaps and bounds.

If you ever run into the obscure case where you need to lookup a MIME type via an API, check out mimeapi. I built it because it makes more sense than cramming another JavaScript library down my user’s pipes. Maybe it’ll help you too.

While we don’t have an immediate need for full text search over at LittleShoot, we can tell you the team behind GAE full text search is rock solid and battle tested.  Given the various limitations of the GAE datastore API, it’s also quite a technical feat.

If you need full text search, and you’re running on App Engine using python, GAE search will save you a great deal of pain.  Go get it.

1. Seamless Django integration (including 1.0.1)
2. Thorough documentation
3. Healthy open source development community with an excellent steward in Kornewald and frequent new releases

The Django integration got me first. Almost everything works, such as manage.py, Django authentication, Django testing (the original reason I switched), etc. There are also lots of other goodies in there, like support for boto’s SQS module. If you’re unfamiliar with it, boto SQS allows you to call Amazon’s Simple Queue Service (SQS) from Python. That’s a huge step in getting around GAE’s limitation on longer lived, CPU-intensive tasks. Just queue it up in SQS, and your GAE app will keep humming along fine — cloud integration at its finest.

To get started with App Engine Patch, download the zip file from the home page and work off the sample project. The download includes App Engine Patch itself as well as the sample project.  

With Django App Engine Helper not supporting Django 1.0 and seemingly inactive, App Engine Patch is a godsend and gets a huge thumbs up.

Great work Waldemar, and don’t forget to donate.

text pp then your street address and zip to 69866 (eg: pp 101 market st 94105)

That’ll give you the polling place for your address or the number for the Election Protection Coalition if your address isn’t in their system.  Their original post is here.

We’ll be studying this election for centuries.  Great job fellas.  I’m rooting for your servers holding up!

One of Mac’s quirks has always been a lack of uninstallers, leading to widespread global frustration, decreased productivity, and some say a leading cause of the recent financial crisis (a small minority).

Salvation is at hand.  LittleShoot’s uninstaller is ridiculously simple. It’s a little snippet of AppleScript that just runs a bash file loaded into it’s application bundle using “do shell script” to run an external script file. Within this framework, your entire uninstaller is basically a bash script. Here are the steps to get this working:

1. Copy the following AppleScript into a file called ‘uninstaller.scpt’ and open it in Script Editor. You can also save the binary script from our SVN here and open it in Script Editor directly (easier).

on onConfirmUninstall()
set applicationName to “LittleShoot”
try
display dialog “Are you sure you want to uninstall ” & applicationName & “?”
set uninstallScript to quoted form of POSIX path of (path to resource “uninstall.bash”)
do shell script “bash ” & uninstallScript with administrator privileges
display dialog “Successfully Uninstalled ” & applicationName buttons {“OK”} default button “OK”

on error err
if err contains “User canceled” then
display dialog “Canceled ” & applicationName & ” Uninstall” buttons {“OK”} default button “OK”
else
display dialog “We’re sorry, but there was an error uninstalling ” & applicationName & ” described as: ” & err buttons {“OK”} default button “OK”
end if
end try
end onConfirmUninstall

onConfirmUninstall()

2. Change the line ‘set applicationName to “LittleShoot”‘ to ‘set applicationName to “[your application name]“‘
3. Choose File->Save As… in AppleScript Editor and save this script as an application bundle that’s run only with no startup screen (the options at the bottom when you choose Save As…). 
4. Navigate via the Terminal to where you saved the bundle and cd into the “Contents/Resources” directory. In our case that’s “LittleShootUninstaller.app/Contents/Resources,” so it’s Contents/Resources within the app bundle.
5. Create an uninstall.bash file in Contents/Resources. As you can see in the AppleScript above, that’s the file the AppleScript looks for and executes.
6. Put whatever you need in uninstall.bash to uninstall your application. The script will run with administrator privileges, so you can really do whatever you want here. Here’s the LittleShoot uninstall script to get you going, although this is a little quirky because we use things like launchd that most applications don’t use. You can also grab this directly from our SVN here.

#!/usr/bin/env bash 

function die()
{
  echo $*
  exit 1
}

function cleanAndDie()
{
  clean
  die “LittleShoot is already uninstalled”
}

function clean()
{
  local plist=~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist
  test -f $plist && launchctl unload $plist
  rm -rf ~/Applications/LittleShoot.app
  rm -f ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist
  rm -rf /Library/Receipts/littleshoot.pkg
  rm -rf ~/Library/Receipts/littleshoot.pkg
  rm -rf ~/.littleshoot
  rm -rf ~/Applications/LittleShootUninstaller.app
}

function remove()
{
  rm -rf $1 || die “Could not remove file: $1″
}

# If it looks like we’ve already uninstalled, just make sure to remove everything again and die.
test -e ~/Applications/LittleShoot.app || cleanAndDie

launchctl stop org.lastbamboo.littleshoot || die “Could not stop LittleShoot”
launchctl unload ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist || die “Could not unload”
rm -rf ~/Applications/LittleShoot.app || die “Could not remove LittleShoot”
rm -f ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist || die “Could not remove plist”

# We go through all this because the package file is placed differently on Tiger, Leopard, etc.
globalReceipt=/Library/Receipts/littleshoot.pkg
userReceipt=~/Library/Receipts/littleshoot.pkg
test -e $globalReceipt && remove $globalReceipt
test -e $localReceipt && remove $localReceipt
rm -rf ~/.littleshoot || die “Could not remove LittleShoot config folder”
rm -rf ~/Applications/LittleShootUninstaller.app || die “Could not remove LittleShoot uninstaller”

So that’s about all she wrote. You basically just have to set up the uninstaller AppleScript application bundle, and then you can just edit your bash script from then on. The application bundle will use whatever script is stored in its Contents/Resources/uninstall.bash file. I’ve played around with a lot of different options for doing this, and the straight script file approach kills the other options in terms of maintainability and flexibility. Note that if the uninstaller encounters an error, the AppleScript will display a dialog to the user with anything your script has echoed, so make those errors informative. You also of course have to include the uninstaller application bundle in your installer.

In the end, your users have a simple uninstaller they can just double click on, and you have a really easy way to write and maintain your uninstaller code.

Back to allowing you to post LittleShoot files to Twitter…

You can run it with:

ec2-run-instances –url https://ec2.amazonaws.com ami-2bdd3942 -k <your-keypair-name>

You can get more info on running OpenSolaris on EC2 here.

With the release of Google App Engine 1.1.3 (GAE), it’s now much more realistic to serve up Dojo on GAE. LittleShoot uses a lot of Dojo (mostly because it rocks), so getting Dojo up and running on our new GAE LittleShoot port has been a high priority.  

The problem is that vanilla Dojo quickly runs up against GAE’s 1000 file limit per application. Django has the same problem. Our initial solution was to run a custom build that would delete a lot of unnecessary files, with thanks to Peter Higgins from SitePen for lighting our path. This got our build down to about 700 files, solving the problem for the basic LittleShoot site.

We’re getting ready to release the LittleShoot platform, however, that allows any site to detect if LittleShoot’s installed and to call the LittleShoot “P2P 2.0″ API if it’s available. That means we need external sites to be able to load our JavaScript, i.e. we need to support Dojo’s cross-domain builds. The cross-domain build adds about another 500 files, pushing us again over the 1000 file limit.

Thankfully, GAE 1.1.3 introduces “zipserve,” allowing you to serve static files directly from a zip file. Guido van Rossum deployed a similar technique for serving Django for his fantastic Rietveld code review tool and GAE demo app, but that only works for loading python code. So we copied Guido for loading Django 1.0, and now zipserve allows us to do more or less the same thing for Dojo.  

To serve Dojo using zipserve, you basically modify your app.yaml to include something like the following:

- url: /dojo/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /dijit/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /dojox/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

 

This will look for dojo.zip, dijit.zip, and dojox.zip files in your *top-level* directory, the same directory containing app.yaml. You need to separate the zip files to avoid running up against the GAE limit on file sizes (1MB I believe). The zip file name basically acts like a directory in App Engine’s URL resolution. So, when a request goes to http://beta.littleshoot.org/dojo/dojo.js, for example, it loads dojo.js from dojo.zip (not from the dojo directory within dojo.zip). Here’s our script for generating the zips:

#!/usr/bin/env bash

function die()
{
  echo $*
  exit 1
}

cd static/js
dirs=”dojo dijit dojox littleshoot”

for x in $dirs
do
  cd $x || die “Could not cd to $x”
  echo “Building zip for $x”
  zf=../../../$x.zip
  rm $zf
  zip -rv $zf * || die “Could not create zip”
  cd ..
  # We actually delete all the contents because we don’t want to 
  # include them in GAE.
  rm -rf $x
done

exit 0

Even with separating the zip files, dijit and dojox will still come close to the 1MB limit on file sizes.  We use rsync excludes to get rid of the files we don’t need.  Here’s our bash rsync excludes function, which you can customize to your liking (careful not to exclude anything you need!!):

function excludeRsync
{
  rsync –exclude .svn/ \
        –exclude test/ \
        –exclude tests/ \
        –exclude demo/ \
        –exclude demos/ \
        –exclude soria/ \
        –exclude nihilo/ \
        –exclude grid/ \
        –exclude charting/ \
        –exclude util/ \
        –exclude analytics/ \
        –exclude collections/ \
        –exclude README* \
        –exclude *.psd \
        –exclude *.uncompressed.js \
        –exclude *.commented.css \
        –exclude dijit/templates \
        –exclude dijit/form/templates \
        –exclude dijit/layout/templates \
        –exclude *silverlight* \
        –exclude gfx3d/ \
        –exclude dojo/_base/ \
        –exclude dojo/_base.js \
        –exclude dojo/build.txt \
        –exclude functional/ \
        –exclude off/ \
        –exclude presentation/ \
        –exclude sketch/ \
        –exclude storage/ \
        –exclude wire/ \
        –exclude data/ \
        –exclude dtl/ \

        -avz $tmpDir/js $releaseDir || die “Could not sync”

}

We call the rsync script after running our dojo build.  The dojo build puts everything in the “$tmpDir/js” directory listed on the last line of the rsync.

OK, so there are still some hoops to jump through, but it works!  On the Django 1.0 side, I’d highly recommend copying Guido’s Rietveld settings.  The Google App Engine Django Helper is also useful, but Django 1.0 support hasn’t been released as of this writing.  We’re using Guido’s settings for now — basically the Makefile, make_release.sh, settings.py, and main.py.  Oh, I also tweaked his rietveld.py script, turning it into littleshoot.py.  You can find all of these files in the Rietveld svn here.  Most of the tweaks to these files are pretty obvious when you glance at the scripts — nothing super complicated going on.  Not that it’s a breeze, but each files is individually fairly straightforward.  The scripts will also create a zipped version of django from the django directory, so using svn:externals (described below) is handy.

We load both Django and Dojo with svn:externals. You might be on information overload at this point, but here’s another little nugget if you’re still with me. You can run the following in the respective directories where you want Django and Dojo to reside.

svn –editor-cmd=vim pe svn:externals .

For Dojo, enter the following:

dojo http://svn.dojotoolkit.org/src/tags/release-1.2.0b2/

For Django, try:

django http://code.djangoproject.com/svn/django/trunk/django

That will put Dojo 1.2 beta 2 in the “dojo” directory and the Django trunk in the “django” directory when you run “svn up”.  

If you successfully navigate through all of that, the end result is the latest Dojo and the latest Django running on the spanking new Google App Engine 1.1.3. Web app setups just don’t get any sweeter. If you’re a total geek like me, that’s living the good life! Sad, I know…