The other day I needed to add an HTML form to the LittleShoot site to upload files directly to Amazon S3. S3’s “browser-based uploads” allow you to send files directly into S3 using hidden form fields to sign the request as opposed to sending files through your server. Nice and efficient.

Everything was going along fine until I realized one glaring problem: how the heck could I set the Content-Type on the file? S3 requires you to do this in the initial request, in this case using another hidden form field, but how could I determine the MIME type at all? Cut past 30 minutes of digging around, and it turns out you can, of course do the following:

1. Attach to the form submit event. In JQuery, this is just:

   $("#myForm").submit(function() {...});

2. Lookup the MIME type in JavaScript (somehow – pay attention to this one)
3. Dynamically add a hidden “Content-Type” form field just before the upload.

The only problem? To do this, you need to include a full JavaScript library to lookup all MIME types in the submit function above, another 40K+ of JavaScript code. That seems a little excessive for a call you’ll only make every once in awhile.

Now comes the beauty of Google App Engine (GAE). With App Engine, you can fire up a site in a matter of minutes that will scale to millions. With a little help from my favorite App Engine Patch, you can run it all on Django. The speed and ease of deployment are breathtaking: 1 hour to deploy a REST API for MIME lookups, all for free.

The result is http://www.mimeapi.org/.  It’s a trivial, ridiculously simple API that basically just exposes the Python “mimetypes” module. It’s the simplest, silliest, and most trivial REST API I’ve ever seen or written, and that’s the whole point. Google App Engine annihilates the barrier to creating this type of API on the only two scales that really matter in this case: time and cost.

I predict we’ll see a lot of similarly trivial services continuing to pop up because they’re so easy. They seem almost stupid at first. Simple building blocks like these are the secret sauce in building any truly powerful platform, however, and with tools like GAE, the web is getting more powerful by leaps and bounds.

If you ever run into the obscure case where you need to lookup a MIME type via an API, check out mimeapi. I built it because it makes more sense than cramming another JavaScript library down my user’s pipes. Maybe it’ll help you too.

Waldemar Kornewald and the rest of the Google App Engine Patch team have just released the first full text search implementation for Google App Engine (GAE).  LittleShoot has been using App Engine Patch since the early days of App Engine, and we can’t recommend it highly enough.  Its seamless Django integration has saved us countless hours, and features like its tool for combining and compressing all of your JavaScript and CSS using the YUI compressor are just stellar.

While we don’t have an immediate need for full text search over at LittleShoot, we can tell you the team behind GAE full text search is rock solid and battle tested.  Given the various limitations of the GAE datastore API, it’s also quite a technical feat.

If you need full text search, and you’re running on App Engine using python, GAE search will save you a great deal of pain.  Go get it.

I’ve recently had the chance to play with Waldemar Kornewald’s “Google App Engine Patch” and have come away very impressed. All LittleShoot Google App Engine (GAE) projects now run on it for a couple of simple reasons:

1. Seamless Django integration (including 1.0.1)
2. Thorough documentation
3. Healthy open source development community with an excellent steward in Kornewald and frequent new releases

The Django integration got me first. Almost everything works, such as manage.py, Django authentication, Django testing (the original reason I switched), etc. There are also lots of other goodies in there, like support for boto’s SQS module. If you’re unfamiliar with it, boto SQS allows you to call Amazon’s Simple Queue Service (SQS) from Python. That’s a huge step in getting around GAE’s limitation on longer lived, CPU-intensive tasks. Just queue it up in SQS, and your GAE app will keep humming along fine — cloud integration at its finest.

To get started with App Engine Patch, download the zip file from the home page and work off the sample project. The download includes App Engine Patch itself as well as the sample project.  

With Django App Engine Helper not supporting Django 1.0 and seemingly inactive, App Engine Patch is a godsend and gets a huge thumbs up.

Great work Waldemar, and don’t forget to donate.

My buddies Chris Muscarella and Benjamin Stein over at Mobile Commons just released a simple service to find your voting place via text message.  Here’s all you have to do:

text pp then your street address and zip to 69866 (eg: pp 101 market st 94105)

That’ll give you the polling place for your address or the number for the Election Protection Coalition if your address isn’t in their system.  Their original post is here.

We’ll be studying this election for centuries.  Great job fellas.  I’m rooting for your servers holding up!

Back in the old days at LimeWire one of the many tasks I took on was building the installers for all platforms, and I’ve carried on my installer hacking with LittleShoot. While installers are anything but glamorous, I’m oddly obsessed with them I think because they’re your users’ first introduction to your program. They need to be simple, and they need to work.

One of Mac’s quirks has always been a lack of uninstallers, leading to widespread global frustration, decreased productivity, and some say a leading cause of the recent financial crisis (a small minority).

Salvation is at hand.  LittleShoot’s uninstaller is ridiculously simple. It’s a little snippet of AppleScript that just runs a bash file loaded into it’s application bundle using “do shell script” to run an external script file. Within this framework, your entire uninstaller is basically a bash script. Here are the steps to get this working:

1. Copy the following AppleScript into a file called ‘uninstaller.scpt’ and open it in Script Editor. You can also save the binary script from our SVN here and open it in Script Editor directly (easier).

on onConfirmUninstall()
set applicationName to “LittleShoot”
try
display dialog “Are you sure you want to uninstall ” & applicationName & “?”
set uninstallScript to quoted form of POSIX path of (path to resource “uninstall.bash”)
do shell script “bash ” & uninstallScript with administrator privileges
display dialog “Successfully Uninstalled ” & applicationName buttons {“OK”} default button “OK”

on error err
if err contains “User canceled” then
display dialog “Canceled ” & applicationName & ” Uninstall” buttons {“OK”} default button “OK”
else
display dialog “We’re sorry, but there was an error uninstalling ” & applicationName & ” described as: ” & err buttons {“OK”} default button “OK”
end if
end try
end onConfirmUninstall

onConfirmUninstall()

2. Change the line ‘set applicationName to “LittleShoot”‘ to ‘set applicationName to “[your application name]“‘
3. Choose File->Save As… in AppleScript Editor and save this script as an application bundle that’s run only with no startup screen (the options at the bottom when you choose Save As…). 
4. Navigate via the Terminal to where you saved the bundle and cd into the “Contents/Resources” directory. In our case that’s “LittleShootUninstaller.app/Contents/Resources,” so it’s Contents/Resources within the app bundle.
5. Create an uninstall.bash file in Contents/Resources. As you can see in the AppleScript above, that’s the file the AppleScript looks for and executes.
6. Put whatever you need in uninstall.bash to uninstall your application. The script will run with administrator privileges, so you can really do whatever you want here. Here’s the LittleShoot uninstall script to get you going, although this is a little quirky because we use things like launchd that most applications don’t use. You can also grab this directly from our SVN here.

#!/usr/bin/env bash 

function die()
{
  echo $*
  exit 1
}

function cleanAndDie()
{
  clean
  die “LittleShoot is already uninstalled”
}

function clean()
{
  local plist=~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist
  test -f $plist && launchctl unload $plist
  rm -rf ~/Applications/LittleShoot.app
  rm -f ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist
  rm -rf /Library/Receipts/littleshoot.pkg
  rm -rf ~/Library/Receipts/littleshoot.pkg
  rm -rf ~/.littleshoot
  rm -rf ~/Applications/LittleShootUninstaller.app
}

function remove()
{
  rm -rf $1 || die “Could not remove file: $1″
}

# If it looks like we’ve already uninstalled, just make sure to remove everything again and die.
test -e ~/Applications/LittleShoot.app || cleanAndDie

launchctl stop org.lastbamboo.littleshoot || die “Could not stop LittleShoot”
launchctl unload ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist || die “Could not unload”
rm -rf ~/Applications/LittleShoot.app || die “Could not remove LittleShoot”
rm -f ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist || die “Could not remove plist”

# We go through all this because the package file is placed differently on Tiger, Leopard, etc.
globalReceipt=/Library/Receipts/littleshoot.pkg
userReceipt=~/Library/Receipts/littleshoot.pkg
test -e $globalReceipt && remove $globalReceipt
test -e $localReceipt && remove $localReceipt
rm -rf ~/.littleshoot || die “Could not remove LittleShoot config folder”
rm -rf ~/Applications/LittleShootUninstaller.app || die “Could not remove LittleShoot uninstaller”

So that’s about all she wrote. You basically just have to set up the uninstaller AppleScript application bundle, and then you can just edit your bash script from then on. The application bundle will use whatever script is stored in its Contents/Resources/uninstall.bash file. I’ve played around with a lot of different options for doing this, and the straight script file approach kills the other options in terms of maintainability and flexibility. Note that if the uninstaller encounters an error, the AppleScript will display a dialog to the user with anything your script has echoed, so make those errors informative. You also of course have to include the uninstaller application bundle in your installer.

In the end, your users have a simple uninstaller they can just double click on, and you have a really easy way to write and maintain your uninstaller code.

Back to allowing you to post LittleShoot files to Twitter…

The OpenSolaris crew just announced you can run Hadoop AMIs on EC2 running on top of OpenSolaris. That’s just cool. I’m still not ready to abandon my Django code running on App Engine (I’ve got a post coming up on the stellar new update to Google App Engine Patch, by the way), but I’d love to play with it. Anyone else given it a go?

You can run it with:

ec2-run-instances –url https://ec2.amazonaws.com ami-2bdd3942 -k <your-keypair-name>

You can get more info on running OpenSolaris on EC2 here.

Important Update: Use Google App Engine Patch for Django integration instead. It integrates Django seamlessly and includes lots of other goodies. For Dojo, also consider using xdomain and loading Dojo from Google’s AJAX Libraries API. For Google App Engine Patch, see my more recent post. This area changes quickly, so check the dates on any blog posts you’re looking at!

With the release of Google App Engine 1.1.3 (GAE), it’s now much more realistic to serve up Dojo on GAE. LittleShoot uses a lot of Dojo (mostly because it rocks), so getting Dojo up and running on our new GAE LittleShoot port has been a high priority.  

The problem is that vanilla Dojo quickly runs up against GAE’s 1000 file limit per application. Django has the same problem. Our initial solution was to run a custom build that would delete a lot of unnecessary files, with thanks to Peter Higgins from SitePen for lighting our path. This got our build down to about 700 files, solving the problem for the basic LittleShoot site.

We’re getting ready to release the LittleShoot platform, however, that allows any site to detect if LittleShoot’s installed and to call the LittleShoot “P2P 2.0″ API if it’s available. That means we need external sites to be able to load our JavaScript, i.e. we need to support Dojo’s cross-domain builds. The cross-domain build adds about another 500 files, pushing us again over the 1000 file limit.

Thankfully, GAE 1.1.3 introduces “zipserve,” allowing you to serve static files directly from a zip file. Guido van Rossum deployed a similar technique for serving Django for his fantastic Rietveld code review tool and GAE demo app, but that only works for loading python code. So we copied Guido for loading Django 1.0, and now zipserve allows us to do more or less the same thing for Dojo.  

To serve Dojo using zipserve, you basically modify your app.yaml to include something like the following:

- url: /dojo/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /dijit/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /dojox/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

 

This will look for dojo.zip, dijit.zip, and dojox.zip files in your *top-level* directory, the same directory containing app.yaml. You need to separate the zip files to avoid running up against the GAE limit on file sizes (1MB I believe). The zip file name basically acts like a directory in App Engine’s URL resolution. So, when a request goes to http://beta.littleshoot.org/dojo/dojo.js, for example, it loads dojo.js from dojo.zip (not from the dojo directory within dojo.zip). Here’s our script for generating the zips:

#!/usr/bin/env bash

function die()
{
  echo $*
  exit 1
}

cd static/js
dirs=”dojo dijit dojox littleshoot”

for x in $dirs
do
  cd $x || die “Could not cd to $x”
  echo “Building zip for $x”
  zf=../../../$x.zip
  rm $zf
  zip -rv $zf * || die “Could not create zip”
  cd ..
  # We actually delete all the contents because we don’t want to 
  # include them in GAE.
  rm -rf $x
done

exit 0

Even with separating the zip files, dijit and dojox will still come close to the 1MB limit on file sizes.  We use rsync excludes to get rid of the files we don’t need.  Here’s our bash rsync excludes function, which you can customize to your liking (careful not to exclude anything you need!!):

function excludeRsync
{
  rsync –exclude .svn/ \
        –exclude test/ \
        –exclude tests/ \
        –exclude demo/ \
        –exclude demos/ \
        –exclude soria/ \
        –exclude nihilo/ \
        –exclude grid/ \
        –exclude charting/ \
        –exclude util/ \
        –exclude analytics/ \
        –exclude collections/ \
        –exclude README* \
        –exclude *.psd \
        –exclude *.uncompressed.js \
        –exclude *.commented.css \
        –exclude dijit/templates \
        –exclude dijit/form/templates \
        –exclude dijit/layout/templates \
        –exclude *silverlight* \
        –exclude gfx3d/ \
        –exclude dojo/_base/ \
        –exclude dojo/_base.js \
        –exclude dojo/build.txt \
        –exclude functional/ \
        –exclude off/ \
        –exclude presentation/ \
        –exclude sketch/ \
        –exclude storage/ \
        –exclude wire/ \
        –exclude data/ \
        –exclude dtl/ \

        -avz $tmpDir/js $releaseDir || die “Could not sync”

}

We call the rsync script after running our dojo build.  The dojo build puts everything in the “$tmpDir/js” directory listed on the last line of the rsync.

OK, so there are still some hoops to jump through, but it works!  On the Django 1.0 side, I’d highly recommend copying Guido’s Rietveld settings.  The Google App Engine Django Helper is also useful, but Django 1.0 support hasn’t been released as of this writing.  We’re using Guido’s settings for now — basically the Makefile, make_release.sh, settings.py, and main.py.  Oh, I also tweaked his rietveld.py script, turning it into littleshoot.py.  You can find all of these files in the Rietveld svn here.  Most of the tweaks to these files are pretty obvious when you glance at the scripts — nothing super complicated going on.  Not that it’s a breeze, but each files is individually fairly straightforward.  The scripts will also create a zipped version of django from the django directory, so using svn:externals (described below) is handy.

We load both Django and Dojo with svn:externals. You might be on information overload at this point, but here’s another little nugget if you’re still with me. You can run the following in the respective directories where you want Django and Dojo to reside.

svn –editor-cmd=vim pe svn:externals .

For Dojo, enter the following:

dojo http://svn.dojotoolkit.org/src/tags/release-1.2.0b2/

For Django, try:

django http://code.djangoproject.com/svn/django/trunk/django

That will put Dojo 1.2 beta 2 in the “dojo” directory and the Django trunk in the “django” directory when you run “svn up”.  

If you successfully navigate through all of that, the end result is the latest Dojo and the latest Django running on the spanking new Google App Engine 1.1.3. Web app setups just don’t get any sweeter. If you’re a total geek like me, that’s living the good life! Sad, I know…

Can Amazon Build the One-Click Cloud?

It’s a great time to program for the cloud, no matter what Ted Dziuba’s entertaining but barely coherent rants have to say (will someone get that guy some experience?). Amazon and Google are going toe-to-toe, with Amazon’s addition of sorting in Simple DB bringing it up to par with Google App Engine’s Datastore API. Sorting was the biggest missing piece in Simple DB and the most compelling reason to choose the Datastore API instead. No longer.  

But Google App Engine (GAE) and the Datastore API still win. Here’s why:

1. The Datastore API is projected to be 10x cheaper. $0.15-$0.18 per GB-month sounds a lot better than Simple DB’s $1.50 per GB-month.
2. GQL. GAE’s SQL subset is just brain dead simple. As adept as programmers are at learning new frameworks, it’s nice to have something brain dead every once in awhile. Simple DB takes a few more cycles to learn (brain cycles that is — more coffee and such. Modafinil perhaps? Anyone tried it? I’m curious).
3. GAE has better Object Relational Mapping (ORM). GAE basically uses Django’s sweet ORM system. You’ve got to jump through a lot more hoops to get something as nice with Simple DB. 
4. GAE automatically scales the web application, not just the database. With Amazon, you have to add load balancing and bring machines up and down yourself, even if you’re using Simple DB. While there are third-party tools to help, they’re not built-in. Again, GAE is brain dead here.  

Sure, App Engine only supports Python. The ultimate question, though, is what functionality can you get in the end? For web apps, App Engine gives you more, particularly for scaling (which is kind of the whole point). Don’t know Python? Learn it. It will save you time in the end. Instead of endlessly fiddling with your load balancer and custom scripts for bringing instances up and down, you’ll spend your time adding the next killer feature your users will love.

In the end, the Amazon/Google “main event” is a huge win for you, me, and our users. The sorting announcement from Amazon comes on the heals of a flurry of other new features from both companies, including Amazon’s impressive persistent storage addition for EC2 called the Elastic Block Store, querying by attributes on Simple DB, GAE’s support for 10 applications per user instead of 3, GAE’s batch writes, etc. Neither one is pulling any punches, and the tools at our disposal as developers are progressing at a breathtaking pace as a result.

Amazon’s is clearly the more complete offering (you can do anything on it, in any language), but it needs to learn from Google’s focus on the dominant deployment scenarios.  Amazon could easily win if it does the following:

1. Makes Simple DB pricing competitive with Google’s projected prices.
2. Adds a query language for Simple DB along the lines of GQL.
3. Adds automatic scaling for web applications, not just the database.
4. Offers complete deployment solutions for the dominant web applications frameworks, from Tomcat/Spring/Hibernate to Django and Zend, with ORM models already adapted to Simple DB, instances automatically replicated with traffic, etc. Basically the same thing as App Engine for more web app frameworks than App Engine supports and adapted to the Amazon platform. Sure, there are third-party solutions for some of this stuff, but those will never be trusted as much as something offered directly from Amazon.

I’m a big fan of Amazon and Werner Vogels (one of the most innovative people in the industry, and also apparently a pretty nice guy), but Amazon desperately needs to learn from what Google has done. It’s ultimately a question of “usability” for developers. The originators of “one-click shopping” are losing in the game they practically invented. 

Amazon needs to turn on the one-click cloud.

I submitted “P2P 2.0 and the Future of Digital Media” as a panel for SXSW 2009.  Please, please, please vote for my panel.  It’s going to rock.

The panel will include a group of leaders in the P2P world who are together ushering in a new generation of P2P applications that will turn media distribution on its head, enabling any web site to seamlessly integrate P2P.  These applications enable sites to include high resolution videos, for example, streamed directly through the browser but using P2P CDNs behind the scenes.

Users on these new sites won’t even know they’re using P2P.  They’ll just know they’re seeing some amazing content. 

Web site owners will be able to give their users higher resolution files than ever before while lowering their bandwidth bills.

Vote early, vote often.

Things have been on hold here for the public release of LittleShoot as we have awaited word on our TechCrunch 50 application.  We didn’t make it! Darn.  It came down to the wire — we were in the last batch of companies to receive notification we weren’t in the conference on Friday at 1:24 AM EST. The 50 companies that made it in should know now.  Here’s the e-mail:

Dear TechCrunch50 Candidate:

We are sorry to inform you that your company was not selected as a finalist for the TechCrunch50 conference. As you know, we are only able to select a very, very small percentage of the more than 1,000 outstanding applications we receive.

Your company was among a select set of candidates that we considered, and it was a difficult decision driven purely by the limited number of presentation slots. Since we regarded your business so highly, we want to make sure you still get the opportunity to participate in the conference in our DemoPit.
(http://techcrunch50demopit.eventbrite.com).

As a DemoPit company, you will have the opportunity to be nominated for the People’s Choice award and win the 50th spot on the TechCrunch50 main stage. As the 50th company to present, the People’s Choice award winner will be able to compete for the $50,000 TechCrunch50 award. Act fast, as spaces are very limited and first come, first served.

Additionally, all DemoPit companies will benefit from the exposure generated by media attending the event. We do anticipate having approximately 300 members of the international press in attendance.

If you have questions regarding the TechCrunch50 Demo Pit opportunity, please email Dan Kimerling at dan@techcrunch.com.

Sincerely,

–Jason, Heather & Michael
and the TechCrunch50 Team

 

I’ve got a lot of respect for the TechCrunch folks and the way they give unfunded companies a shot, and I thoroughly enjoyed meeting Jason Calacanis down at the Mahalo Tech Meetup last night in one of my first nights in LA. I appreciate the tremendous work Jason, Michael, Heather, and the other folks at TechCrunch have put it to the process.  

That said, it’s on.  I feel like the guy on draft day who didn’t go on the first round.  It’s the “meritocracy” thing that gets me – TechCrunch 50 is touted as a pure meritocracy.  I’d put LittleShoot’s technology up there with anyone, and it just kills me to think 50 startups beat us out.  We can tell ourselves they had better business models, better marketing plans, yada yada yada, but I’m taking it to mean they had better technology.  If there’s anything that motivates me, that’s it. I have great respect for the other applicants, and we all supported each other on the TechCrunch blog as we agonized through the waiting process.  I wish everyone the best of luck, but the LittleShoot public beta is on its way.

Here’s a link to the LittleShoot demo video we submitted for TechCrunch for people unfamiliar with the Little Fella’:

LittleShoot Demo