Lantern Bash Configuration Script on OSX

September 4, 2011

I’ve been working for awhile on a piece of software called Lantern. It’s a new censorship circumvention and monitoring-prevention tool built atop the LittleShoot P2P networking layer and also atop LittleProxy, another little project I’ve completely neglected to mention! You’ll be finding out a lot more about Lantern soon, but sign up for the mailing list at http://www.getlantern.org if you want to be notified when you can start playing with it. It’ll be fun, and ultimately should help people all over the world who don’t enjoy the relatively unfettered internet most of us take for granted.

I was actually inspired to write this post, because I just added a little bash code snippet for auto-configuring proxies on OSX using networksetup, a pac file, and some nifty for looping to read lines (trickier than you might think!). I thought others might find it useful. Enjoy!


#!/usr/bin/env bash

mkdir ~/Library/Logs/Lantern

function log() {
echo "`date`: $@" >> ~/Library/Logs/Lantern/installer.log
}

log "Configuring network services"
while read s;
do
log "Configuring network: $s"
networksetup -setautoproxyurl "$s" file://localhost$HOME/.lantern/proxy.pac || log "Could not set auto proxy URL for $s"
networksetup -setautoproxystate "$s" "on" || log "Could not turn auto proxy on for $s"
log "Configured network: $s"
done < <(networksetup -listallnetworkservices | tail +2)
log "Done configuring network services!!"


GData Maven Deploy Script

July 10, 2010

I just wrote a quick script to deploy the latest GData versions to the LittleShoot maven repository, and I thought others might find it handy. GData doesn’t deploy to a maven repo, so you’ve got to do a little extra work. There are various solutions out there, but this is a little more straightforward. I’m jumping on board with Google’s latest 2.2.x alphas of their GData Java clients, as it looks like that’s where the future action will be. So this only applies to the 2.2.x alphas. The script simply takes the version argument, checks out that version from SVN, and deploys the jar to your repo.

You’ll want to swap out your repository for the LittleShoot repository listed at the end — both the URL and repository ID. If you want to just grab this script from the LittleShoot repository, it’s here.


#!/usr/bin/env bash

# This is a script for automatically deploying the GData jars to the LittleShoot
# maven repository.

gdataVersion=$1

function die() {
rm -rf $gdataVersion
echo $*
exit 1
}

me=`basename $0`

if [ $# -ne 1 ]
then
"Should be a single GData version argument, as in './$me 2.2.1-alpha'. For versions, see: http://gdata-java-client.googlecode.com/svn/tags/"
exit 1
fi

echo "Checking out GData version $gdataVersion"
svn co http://gdata-java-client.googlecode.com/svn/tags/$gdataVersion || die "Could not checkout SVN"

pushd $gdataVersion/gdata || die "Could not move to gdata base dir"
ant dist || die "Could not build with ant"
cd build-bin/dist/ || die "Could not cd to dist directory"

mvn deploy:deploy-file -DgroupId=com.google.gdata -DartifactId=gdata -Dversion=$gdataVersion -Dfile=gdata-$gdataVersion.jar -Dpackaging=jar -DgeneratePom=true -Durl=scpexe://dev.littleshoot.org/var/maven -DrepositoryId=littleshoot || die "Could not deploy file"

echo "Cleaning up"
popd
rm -rf $gdataVersion


MIME API from LittleShoot on Google App Engine

July 23, 2009

The other day I needed to add an HTML form to the LittleShoot site to upload files directly to Amazon S3. S3′s “browser-based uploads” allow you to send files directly into S3 using hidden form fields to sign the request as opposed to sending files through your server. Nice and efficient.

Everything was going along fine until I realized one glaring problem: how the heck could I set the Content-Type on the file? S3 requires you to do this in the initial request, in this case using another hidden form field, but how could I determine the MIME type at all? Cut past 30 minutes of digging around, and it turns out you can, of course do the following:

  1. Attach to the form submit event. In JQuery, this is just:
    $("#myForm").submit(function() {...});
  2. Lookup the MIME type in JavaScript (somehow – pay attention to this one)
  3. Dynamically add a hidden “Content-Type” form field just before the upload.

The only problem? To do this, you need to include a full JavaScript library to lookup all MIME types in the submit function above, another 40K+ of JavaScript code. That seems a little excessive for a call you’ll only make every once in awhile.

Now comes the beauty of Google App Engine (GAE). With App Engine, you can fire up a site in a matter of minutes that will scale to millions. With a little help from my favorite App Engine Patch, you can run it all on Django. The speed and ease of deployment are breathtaking: 1 hour to deploy a REST API for MIME lookups, all for free.

The result is http://www.mimeapi.org/.  It’s a trivial, ridiculously simple API that basically just exposes the Python “mimetypes” module. It’s the simplest, silliest, and most trivial REST API I’ve ever seen or written, and that’s the whole point. Google App Engine annihilates the barrier to creating this type of API on the only two scales that really matter in this case: time and cost.

I predict we’ll see a lot of similarly trivial services continuing to pop up because they’re so easy. They seem almost stupid at first. Simple building blocks like these are the secret sauce in building any truly powerful platform, however, and with tools like GAE, the web is getting more powerful by leaps and bounds.

If you ever run into the obscure case where you need to lookup a MIME type via an API, check out mimeapi. I built it because it makes more sense than cramming another JavaScript library down my user’s pipes. Maybe it’ll help you too.


Google App Engine Full Text Search From App Engine Patch Team

July 1, 2009

Waldemar Kornewald and the rest of the Google App Engine Patch team have just released the first full text search implementation for Google App Engine (GAE).  LittleShoot has been using App Engine Patch since the early days of App Engine, and we can’t recommend it highly enough.  Its seamless Django integration has saved us countless hours, and features like its tool for combining and compressing all of your JavaScript and CSS using the YUI compressor are just stellar.

While we don’t have an immediate need for full text search over at LittleShoot, we can tell you the team behind GAE full text search is rock solid and battle tested.  Given the various limitations of the GAE datastore API, it’s also quite a technical feat.

If you need full text search, and you’re running on App Engine using python, GAE search will save you a great deal of pain.  Go get it.


Django and More on Google App Engine with App Engine Patch

November 16, 2008

I’ve recently had the chance to play with Waldemar Kornewald’s “Google App Engine Patch” and have come away very impressed. All LittleShoot Google App Engine (GAE) projects now run on it for a couple of simple reasons:

  1. Seamless Django integration (including 1.0.1)
  2. Thorough documentation
  3. Healthy open source development community with an excellent steward in Kornewald and frequent new releases

The Django integration got me first. Almost everything works, such as manage.py, Django authentication, Django testing (the original reason I switched), etc. There are also lots of other goodies in there, like support for boto’s SQS module. If you’re unfamiliar with it, boto SQS allows you to call Amazon’s Simple Queue Service (SQS) from Python. That’s a huge step in getting around GAE’s limitation on longer lived, CPU-intensive tasks. Just queue it up in SQS, and your GAE app will keep humming along fine — cloud integration at its finest.

To get started with App Engine Patch, download the zip file from the home page and work off the sample project. The download includes App Engine Patch itself as well as the sample project.  

With Django App Engine Helper not supporting Django 1.0 and seemingly inactive, App Engine Patch is a godsend and gets a huge thumbs up.

Great work Waldemar, and don’t forget to donate.


OSX Uninstallers the Easy Way

October 22, 2008

Back in the old days at LimeWire one of the many tasks I took on was building the installers for all platforms, and I’ve carried on my installer hacking with LittleShoot. While installers are anything but glamorous, I’m oddly obsessed with them I think because they’re your users’ first introduction to your program. They need to be simple, and they need to work.

One of Mac’s quirks has always been a lack of uninstallers, leading to widespread global frustration, decreased productivity, and some say a leading cause of the recent financial crisis (a small minority).

Salvation is at hand.  LittleShoot’s uninstaller is ridiculously simple. It’s a little snippet of AppleScript that just runs a bash file loaded into it’s application bundle using “do shell script” to run an external script file. Within this framework, your entire uninstaller is basically a bash script. Here are the steps to get this working:

  1. Copy the following AppleScript into a file called ‘uninstaller.scpt’ and open it in Script Editor. You can also save the binary script from our SVN here and open it in Script Editor directly (easier).
  2. on onConfirmUninstall()
    set applicationName to “LittleShoot”
    try
    display dialog “Are you sure you want to uninstall ” & applicationName & “?”
    set uninstallScript to quoted form of POSIX path of (path to resource “uninstall.bash”)
    do shell script “bash ” & uninstallScript with administrator privileges
    display dialog “Successfully Uninstalled ” & applicationName buttons {“OK”} default button “OK”

    on error err
    if err contains “User canceled” then
    display dialog “Canceled ” & applicationName & ” Uninstall” buttons {“OK”} default button “OK”
    else
    display dialog “We’re sorry, but there was an error uninstalling ” & applicationName & ” described as: ” & err buttons {“OK”} default button “OK”
    end if
    end try
    end onConfirmUninstall

    onConfirmUninstall()

  3. Change the line ‘set applicationName to “LittleShoot”‘ to ‘set applicationName to “[your application name]“‘
  4. Choose File->Save As… in AppleScript Editor and save this script as an application bundle that’s run only with no startup screen (the options at the bottom when you choose Save As…). 
  5. Navigate via the Terminal to where you saved the bundle and cd into the “Contents/Resources” directory. In our case that’s “LittleShootUninstaller.app/Contents/Resources,” so it’s Contents/Resources within the app bundle.
  6. Create an uninstall.bash file in Contents/Resources. As you can see in the AppleScript above, that’s the file the AppleScript looks for and executes.
  7. Put whatever you need in uninstall.bash to uninstall your application. The script will run with administrator privileges, so you can really do whatever you want here. Here’s the LittleShoot uninstall script to get you going, although this is a little quirky because we use things like launchd that most applications don’t use. You can also grab this directly from our SVN here.
  8. #!/usr/bin/env bash 

    function die()
    {
      echo $*
      exit 1
    }

    function cleanAndDie()
    {
      clean
      die “LittleShoot is already uninstalled”
    }

    function clean()
    {
      local plist=~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist
      test -f $plist && launchctl unload $plist
      rm -rf ~/Applications/LittleShoot.app
      rm -f ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist
      rm -rf /Library/Receipts/littleshoot.pkg
      rm -rf ~/Library/Receipts/littleshoot.pkg
      rm -rf ~/.littleshoot
      rm -rf ~/Applications/LittleShootUninstaller.app
    }

    function remove()
    {
      rm -rf $1 || die “Could not remove file: $1″
    }

    # If it looks like we’ve already uninstalled, just make sure to remove everything again and die.
    test -e ~/Applications/LittleShoot.app || cleanAndDie

    launchctl stop org.lastbamboo.littleshoot || die “Could not stop LittleShoot”
    launchctl unload ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist || die “Could not unload”
    rm -rf ~/Applications/LittleShoot.app || die “Could not remove LittleShoot”
    rm -f ~/Library/LaunchAgents/org.lastbamboo.littleshoot.plist || die “Could not remove plist”

    # We go through all this because the package file is placed differently on Tiger, Leopard, etc.
    globalReceipt=/Library/Receipts/littleshoot.pkg
    userReceipt=~/Library/Receipts/littleshoot.pkg
    test -e $globalReceipt && remove $globalReceipt
    test -e $localReceipt && remove $localReceipt
    rm -rf ~/.littleshoot || die “Could not remove LittleShoot config folder”
    rm -rf ~/Applications/LittleShootUninstaller.app || die “Could not remove LittleShoot uninstaller”

 

So that’s about all she wrote. You basically just have to set up the uninstaller AppleScript application bundle, and then you can just edit your bash script from then on. The application bundle will use whatever script is stored in its Contents/Resources/uninstall.bash file. I’ve played around with a lot of different options for doing this, and the straight script file approach kills the other options in terms of maintainability and flexibility. Note that if the uninstaller encounters an error, the AppleScript will display a dialog to the user with anything your script has echoed, so make those errors informative. You also of course have to include the uninstaller application bundle in your installer.

In the end, your users have a simple uninstaller they can just double click on, and you have a really easy way to write and maintain your uninstaller code.

Back to allowing you to post LittleShoot files to Twitter…


Dojo 1.2 and Django 1.0 on Google App Engine 1.1.3

September 17, 2008

Important Update: Use Google App Engine Patch for Django integration instead. It integrates Django seamlessly and includes lots of other goodies. For Dojo, also consider using xdomain and loading Dojo from Google’s AJAX Libraries API. For Google App Engine Patch, see my more recent post. This area changes quickly, so check the dates on any blog posts you’re looking at!

With the release of Google App Engine 1.1.3 (GAE), it’s now much more realistic to serve up Dojo on GAE. LittleShoot uses a lot of Dojo (mostly because it rocks), so getting Dojo up and running on our new GAE LittleShoot port has been a high priority.  

The problem is that vanilla Dojo quickly runs up against GAE’s 1000 file limit per application. Django has the same problem. Our initial solution was to run a custom build that would delete a lot of unnecessary files, with thanks to Peter Higgins from SitePen for lighting our path. This got our build down to about 700 files, solving the problem for the basic LittleShoot site.

We’re getting ready to release the LittleShoot platform, however, that allows any site to detect if LittleShoot’s installed and to call the LittleShoot “P2P 2.0″ API if it’s available. That means we need external sites to be able to load our JavaScript, i.e. we need to support Dojo’s cross-domain builds. The cross-domain build adds about another 500 files, pushing us again over the 1000 file limit.

Thankfully, GAE 1.1.3 introduces “zipserve,” allowing you to serve static files directly from a zip file. Guido van Rossum deployed a similar technique for serving Django for his fantastic Rietveld code review tool and GAE demo app, but that only works for loading python code. So we copied Guido for loading Django 1.0, and now zipserve allows us to do more or less the same thing for Dojo.  

To serve Dojo using zipserve, you basically modify your app.yaml to include something like the following:

- url: /dojo/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /dijit/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /dojox/.*
  script: $PYTHON_LIB/google/appengine/ext/zipserve

 

This will look for dojo.zip, dijit.zip, and dojox.zip files in your *top-level* directory, the same directory containing app.yaml. You need to separate the zip files to avoid running up against the GAE limit on file sizes (1MB I believe). The zip file name basically acts like a directory in App Engine’s URL resolution. So, when a request goes to http://beta.littleshoot.org/dojo/dojo.js, for example, it loads dojo.js from dojo.zip (not from the dojo directory within dojo.zip). Here’s our script for generating the zips:

#!/usr/bin/env bash

function die()
{
  echo $*
  exit 1
}

cd static/js
dirs=”dojo dijit dojox littleshoot”

for x in $dirs
do
  cd $x || die “Could not cd to $x”
  echo “Building zip for $x”
  zf=../../../$x.zip
  rm $zf
  zip -rv $zf * || die “Could not create zip”
  cd ..
  # We actually delete all the contents because we don’t want to 
  # include them in GAE.
  rm -rf $x
done

exit 0

Even with separating the zip files, dijit and dojox will still come close to the 1MB limit on file sizes.  We use rsync excludes to get rid of the files we don’t need.  Here’s our bash rsync excludes function, which you can customize to your liking (careful not to exclude anything you need!!):

function excludeRsync
{
  rsync –exclude .svn/ \
        –exclude test/ \
        –exclude tests/ \
        –exclude demo/ \
        –exclude demos/ \
        –exclude soria/ \
        –exclude nihilo/ \
        –exclude grid/ \
        –exclude charting/ \
        –exclude util/ \
        –exclude analytics/ \
        –exclude collections/ \
        –exclude README* \
        –exclude *.psd \
        –exclude *.uncompressed.js \
        –exclude *.commented.css \
        –exclude dijit/templates \
        –exclude dijit/form/templates \
        –exclude dijit/layout/templates \
        –exclude *silverlight* \
        –exclude gfx3d/ \
        –exclude dojo/_base/ \
        –exclude dojo/_base.js \
        –exclude dojo/build.txt \
        –exclude functional/ \
        –exclude off/ \
        –exclude presentation/ \
        –exclude sketch/ \
        –exclude storage/ \
        –exclude wire/ \
        –exclude data/ \
        –exclude dtl/ \

        -avz $tmpDir/js $releaseDir || die “Could not sync”

}

We call the rsync script after running our dojo build.  The dojo build puts everything in the “$tmpDir/js” directory listed on the last line of the rsync.

OK, so there are still some hoops to jump through, but it works!  On the Django 1.0 side, I’d highly recommend copying Guido’s Rietveld settings.  The Google App Engine Django Helper is also useful, but Django 1.0 support hasn’t been released as of this writing.  We’re using Guido’s settings for now — basically the Makefile, make_release.sh, settings.py, and main.py.  Oh, I also tweaked his rietveld.py script, turning it into littleshoot.py.  You can find all of these files in the Rietveld svn here.  Most of the tweaks to these files are pretty obvious when you glance at the scripts — nothing super complicated going on.  Not that it’s a breeze, but each files is individually fairly straightforward.  The scripts will also create a zipped version of django from the django directory, so using svn:externals (described below) is handy.

We load both Django and Dojo with svn:externals. You might be on information overload at this point, but here’s another little nugget if you’re still with me. You can run the following in the respective directories where you want Django and Dojo to reside.

svn –editor-cmd=vim pe svn:externals .

For Dojo, enter the following:

dojo http://svn.dojotoolkit.org/src/tags/release-1.2.0b2/

For Django, try:

django http://code.djangoproject.com/svn/django/trunk/django

That will put Dojo 1.2 beta 2 in the “dojo” directory and the Django trunk in the “django” directory when you run “svn up”.  

If you successfully navigate through all of that, the end result is the latest Dojo and the latest Django running on the spanking new Google App Engine 1.1.3. Web app setups just don’t get any sweeter. If you’re a total geek like me, that’s living the good life! Sad, I know…


Vote for ‘P2P 2.0′ Panel at SXSW 2009!

August 22, 2008

I submitted “P2P 2.0 and the Future of Digital Media” as a panel for SXSW 2009.  Please, please, please vote for my panel.  It’s going to rock.

The panel will include a group of leaders in the P2P world who are together ushering in a new generation of P2P applications that will turn media distribution on its head, enabling any web site to seamlessly integrate P2P.  These applications enable sites to include high resolution videos, for example, streamed directly through the browser but using P2P CDNs behind the scenes.

Users on these new sites won’t even know they’re using P2P.  They’ll just know they’re seeing some amazing content. 

Web site owners will be able to give their users higher resolution files than ever before while lowering their bandwidth bills.

Vote early, vote often.


TechCrunch 50, LittleShoot, and the Aftermath

August 15, 2008

Things have been on hold here for the public release of LittleShoot as we have awaited word on our TechCrunch 50 application.  We didn’t make it! Darn.  It came down to the wire — we were in the last batch of companies to receive notification we weren’t in the conference on Friday at 1:24 AM EST. The 50 companies that made it in should know now.  Here’s the e-mail:

 

Dear TechCrunch50 Candidate:

We are sorry to inform you that your company was not selected as a finalist for the TechCrunch50 conference. As you know, we are only able to select a very, very small percentage of the more than 1,000 outstanding applications we receive.

Your company was among a select set of candidates that we considered, and it was a difficult decision driven purely by the limited number of presentation slots. Since we regarded your business so highly, we want to make sure you still get the opportunity to participate in the conference in our DemoPit.
(http://techcrunch50demopit.eventbrite.com).

As a DemoPit company, you will have the opportunity to be nominated for the People’s Choice award and win the 50th spot on the TechCrunch50 main stage. As the 50th company to present, the People’s Choice award winner will be able to compete for the $50,000 TechCrunch50 award. Act fast, as spaces are very limited and first come, first served.

Additionally, all DemoPit companies will benefit from the exposure generated by media attending the event. We do anticipate having approximately 300 members of the international press in attendance.

If you have questions regarding the TechCrunch50 Demo Pit opportunity, please email Dan Kimerling at dan@techcrunch.com.

Sincerely,

–Jason, Heather & Michael
and the TechCrunch50 Team

 

I’ve got a lot of respect for the TechCrunch folks and the way they give unfunded companies a shot, and I thoroughly enjoyed meeting Jason Calacanis down at the Mahalo Tech Meetup last night in one of my first nights in LA. I appreciate the tremendous work Jason, Michael, Heather, and the other folks at TechCrunch have put it to the process.  

That said, it’s on.  I feel like the guy on draft day who didn’t go on the first round.  It’s the “meritocracy” thing that gets me – TechCrunch 50 is touted as a pure meritocracy.  I’d put LittleShoot’s technology up there with anyone, and it just kills me to think 50 startups beat us out.  We can tell ourselves they had better business models, better marketing plans, yada yada yada, but I’m taking it to mean they had better technology.  If there’s anything that motivates me, that’s it. I have great respect for the other applicants, and we all supported each other on the TechCrunch blog as we agonized through the waiting process.  I wish everyone the best of luck, but the LittleShoot public beta is on its way.

Here’s a link to the LittleShoot demo video we submitted for TechCrunch for people unfamiliar with the Little Fella’:

LittleShoot Demo


Where Google App Engine Spanks Amazon’s Web Services: S3, EC2, Simple DB, SQS

May 28, 2008

First off, I loooove Amazon Web Services (AWS), and we make heavy use of S3, EC2, Simple DB, and Elastic IPs for LittleShoot. We run everything on Amazon, but that’s about to change.  I’ve been in the App Engine beta for about a month, and despite all of the astonishing ways AWS makes building web sites easy, Google App Engine makes it far easier. Here’s why:

1) Google App Engine Has Better Scalability  

Google migrates your application across its infrastructure automatically as needed. With EC2, you have to manually detect machine load and bring instances up or down accordingly. You need to set up load balancing and clustering. While Amazon gives you far more control, it’s also far more work.  With Google App Engine, it’s all done for you.

2) Google App Engine Has a Better Database

Google App Engine’s Big Table blows Amazon’s Simple DB out of the water, and that’s coming from a big fan of Simple DB and Amazon CTO Werner Vogels. Simple DB thankfully yanks developers out of the relational database mindset and automatically replicates data across machines for scalability. You do have to learn a completely new query syntax, however, and, as this blog has noted, sorting is not officially supported. Simple DB is also still in beta.  

With App Engine, you’re using the same database, Big Table, Google engineers use to power some of the busiest sites on the Internet. Billions of queries have been hammering out kinks in Big Table for years. You know it will scale.  What’s more, App Engine’s “GQL” gives developers a familiar SQL-like syntax, lowering the learning curve compared to Simple DB.  Big Table also supports sorting.  Perhaps most significantly, Simple DB costs far more. While Google’s final pricing announcement later this year may change, today’s announcement didn’t mention any difference in price for data stored in the database versus anywhere else. On Simple DB, that data costs $1.50 per GB-month. On App Engine, it appears to cost $0.15 – $0.18 per GB-month. Wow.

3) Google App Engine is Cheaper 

Beyond the database, App Engine gives you the first 5 million or so page views per month for free.  That’s a lot of page views. It doesn’t put you up with the Internet’s top dogs, of course, but at 5 million page views you should be making cash. App Engine is free precisely when you’re building your company and keeping costs low is the most important. If you go beyond that 5 million, Google’s I/O event today will reveal newly announced prices that are remarkably similar to Amazon’s current offerings. They both price everything per GB or CPU-hour, and the numbers are barely distinguishable. That first 5 million page views and the apparent huge disparity in database storage pricing are by far the biggest differentiators, both dramatically tipping the scales in favor of Google.

Conclusion

For building scalable web applications quickly, App Engine beats AWS by a surprisingly wide margin.  Note, however, this refers specifically to web applications. For anything custom, you need Amazon. Because App Engine only supports Python, you also need Amazon for running any non-Python code. While this is a significant difference, many good developers are facile with multiple languages and can move rapidly between them.  Amazon’s flexibility makes it win out for many applications, but not for the most common application there is: web sites.  App Engine is more of a “domain-specific cloud” for web applications, but it’s shockingly good at what it does.  

Oh yeah, and it’s cheap.


Follow

Get every new post delivered to your Inbox.