Michael Arrington’s post on creating a decentralized Twitter is theoretically interesting but practically naive. Martin Fowler’s First Law of Distributed Computing continually rings true: don’t distribute your objects. Why? Because it’s hard. In every case I’ve ever seen, it’s orders of magnitude harder to distribute a task than it is to centralize it.
Search is the quintessential example. When I wrote the search algorithms and protocols for the second-generation Gnutella network at LimeWire, the degree of coordination between millions of network nodes was staggering. It worked amazingly well all things considered, but it still could never compete with the speed or collaborative filtering of centralized search. Most importantly, it could never compete with simplicity of centralization. What took us 6 months to distribute would have taken a couple of days to centralize. Distributed networks also make updating much harder, as you’re now forced to update every instance of your software running on any machine in the world. It’s a pain in the a$$. If you have a choice, centralized search wins every time. That’s one of the reasons LittleShoot follows Fowler’s law whenever possible. Something about working better and taking less time makes it appealing.
Distributed computing has shown itself to be particularly useful for moving around large files. In Twitter’s case, you’re working from the opposite extreme: processing a high volume of tiny messages. This screams centralization.
Centralization is not the reason Twitter can’t scale. They can’t scale because, well, they just haven’t written an architecture that scales. Granted the mobile space is still largely DIY, so they have to roll much of their own code. That’s really a pretty lame excuse, though, especially given their resources and the time they’ve had to figure it out. My buddies over at Mobile Commons face similar issues processing huge volumes of mobile messages, and they don’t have these issues. I’m convinced Twitter would be flying if Ben Stein and Chris Muscarella at Mobile Commons were in charge, and I honestly think it’s because they’re just better programmers.
I’m a big Evan Williams fan and was thrilled to meet him for the first time down at SXSW, but that doesn’t mean I have faith in his or his team’s ability to learn how to scale a complex architecture overnight. A distributed architecture would make their task orders of magnitude harder. Don’t do it fellas.