Backbone.js and Capsule and Thoonk, oh my! A scalable realtime architecture

This last year, we've learned a lot about building scalable realtime web apps, most of which has come from shipping &bang.

&bang is the app we use to keep our team in sync. It helps us stay on the same page, bug each other less and just get stuff done as a team.

The process of actually trying to get something out the door on a bootstrapped budget helped us focus on the most important problems that needed to be solved to build a dynamic, interactive, real-time app in a scaleable way.

A bit of history

I've written a couple of posts on backbone.js since discovering it. The first one introduces Backbone.js as a lightweight client-side framework for building clean, stateful client apps. In the second post I introduced Capsule.js. Which is a tool that I built on top of Backbone that adds nested models and collections and also allows you to keep a mirror of your client-side state on a node.js server to seemlessly synchronize state between different clients.

That approach was great for quickly prototyping an app. But as I pointed out in that post, that's a lot of in memory state being stored on the server and simply doesn't scale very well.

At the end of that post I hinted at what we were aiming to do to ultimately solve that problem. So this post is meant to be a bit of an update on those thoughts.

Our new approach

Redis is totally freakin' amazing. Period. I can't say enough good things about it. Salvatore Sanfilippo is a god among men, in my book.

Redis can scale.

Redis can do PubSub.

PubSub just means events. Just like you can listen for click events in Javascript in a browser you can listen for events in Redis.

Redis, however is a generic tool. It's purposely fairly low-level so as to be broadly applicable.

What makes Redis so interesting, from my perspective, is that you can treat it as a shared memory between processes, languages and platforms. What that means, in a practical sense, is that as long as each app that uses it interacts with it according to a pre-defined set of rules, you can write a whole ecosystem of functionality for an app in whatever language makes the most sense for that particular task.

Enter Thoonk

My co-worker, Nathan Fritz, is the closest thing you can get to being a veteran of realtime technologies.

He's a member of the XSF council for the XMPP standard and probably wrote his first chat bot before you knew what chat was. His Sleek XMPP Python library is iconic in the XMPP community. He has a self-declared un-natural love for XEP-60 which describes the XMPP PubSub standard.

He took everything he learned from his work on that standard and built Thoonk. (In fact, he actually kept the PubSub spec open as he built the Javascript and Python implementations of Thoonk.)

What is Thoonk??

Thoonk is an abstraction on Redis that provides higher-level datatypes for a more approachable interface. Essentially, staring at Redis as a newbie is a bit intimidating. Not that it's hard to interface with, it's just kind of tricky to figure out how to logically structure and retrieve your data. Thoonk simplifies that into a few data-types that describe common use cases. Primarly "feeds", "sorted feeds", "queues" and "jobs".

You can think of a feed as an ad-hoc database table. They're "cheap" to create and you simply declare them to make them or use them. For example, in &bang, we have all our users in a feed called "users" for looking up user info. But also, each user has a variety of individual feeds. For example, they have a "task" feed and a "shipped" feed. This is where it veers from what people are used to in a relational database model, because each user's tasks are not a part of a global "tasks" feed. Instead, each user has a distinct feed of tasks because that's the entity we want to be able to subscribe to.

So rather than simply breaking down a model into types of data, we end up breaking things into groups of items (a.k.a. "feeds") that we want to be able to track changes to. So, as an example, we may have something like this:

// our main user feedvar userFeed = thoonk.feed('users');

// an individual task feed for a user
var userTaskFeed = thoonk.sortedFeed('team.andyet.members.{{memberID}}.tasks');


Marrying Thoonk and Capsule

Capsule was actually written with Thoonk in mind. In fact that's why they were named the way they did: You know these lovely pneumatic tube systems they use to send cash to bank tellers and at Costco? (PPSHHHHHHH—THOONK! And here's your capsule.)

Anyway, the integration didn't end up being quite as tight as we had originally thought but it still works quite well. Loose coupling is better anyway right?

The core problem I was trying to solve with Capsule was unifying the models that are used to represent the state of the app in the browser and the models you use to describe your data on the server—ideally, not just unifying the data structure, but also letting me share behavior of those objects.

Let me explain.

As I mentioned, we recently shipped &bang. It lets a group of people share their task lists and what they're actively working on with each other.

It spares you from a lot of "what are you working on?" conversations and increases accountability by making your work quite public to the team.

It's a realtime, keyboard-driven, web app that is designed to feel like a desktop app. &bang is a node.js application built entirely with the methods described here.

So, in &bang, a team model has attributes as well as a couple of nested backbone collections such as members and chat messages. Each member has attributes and other nested collections, tasks, shipped items, etc.

Initial state push

When a user first logs in we have to send the entire model state for the team(s) they're on so we can build out the interface (see my previous post for more on that). So, the first thing we do when a user logs in is subscribe them to the relevant Thoonk feeds and perform the the initial state transfer to the client.

To do this, we init an empty team model on the client (a backbone/capsule model shared between client/server) . Then we recurse through our Thoonk feed structures on the server to export the data from the relevant feeds into a data structure that Capsule can use to import that data. The team model is inflated with the data from the server and we draw the interface.

From there, the application is kept in sync using events from Thoonk that get sent over websockets and applied to the client interface. Events like "publish", "change", "retract" and "position".

Once we got the app to the point where this was all working, it was kind of a magical moment, because at this point, any edits that happen in Thoonk will simply get pushed out through the event propagation all the way to the client. Essentially, the inteface that a user sees is largely a slave to the server. Except, of course, the portions of state that we let the user manipulate locally.

At this point, user interactions with the app that change data are all handled through RPC calls. Let's jump back to the server and you'll see what I mean.

I thought you were still using Capsule on the server?

We do, but differently, here's how that is handled.

In short... it's a job system.

Sounds intimidating right? As someone who started in business school, then gradually got into front-end dev, then back-end dev, then a pile of JS, job systems sounded scary. In my mind they're for "hardcore" programmers like Fritzy or Nate or Lance from our team. Job systems don't have to be that scary.

At a very high level you can think of a "job" as a function call. The key difference being, you don't necessarily expect an immediate result. To continue with examples from &bang: a job may be to "ship a task". So, what do we need to know to complete that action? We need the following:

• member Id of the user shipping the task
• the task id being completed (we call this "shipping", because it's cooler, and it's a reminder a reminder that finishing is what's important)

We can derive everything else we need from those key pieces of information.

So, rather than call a function somewhere:

shipTask(memberId, taskId)

We can just describe a job as a simple JSON object:

{    userId: ,    taskId: ,    memberId: }

The we can add that to our "shipTask" job queue like so:

thoonk.job('shipTask').put(JSON.stringify(jobObject));

The cool part about the event propagation I talked about above is we really don't care so much when that job gets done. Obviously fast is key, but what I mean is, we don't have to sit around and wait for a synchronous result because the event propagation we've set up will handle all the application state changes.

So, now we can write a worker that listens for jobs from that job queue. In that worker we'll perform all the necessary related logic. Specifically stuff like:

• Validating that the job is properly formatted (contains required fields of the right type)
• Validating that the user is the owner of that task and is therefore allowed to "ship" it.
• Modifying Thoonk feeds accordingly.

Encapsulating and reusing model logic

You'll notice that part of that list requires some logic. Specifically, checking to see if the user requesting the action is allowed to perform it. We could certainly write that logic right here, in this worker. But, in the client we're also going to want to know if a user is allowed to ship a given task, right? Why write that logic twice?

Instead we write that logic as a method of a Capsule model that describes a task. Then, we can use the same method to determine whether to show the UI that lets the user perform the action in the browser as we use on the back end to actually perform the validation. We do that by re-inflating a Capsule model for that task in our worker code and calling the canEdit() method on it and passing it the user id requesting the action. The only difference being, on the server-side we don't trust the user to tell us who they are. On the server we roll the user id we have for that session into the job when it's created rather then trust the client.

Security

One other, hugely important thing that we get by using Capsule models on the server is some security features. There are some model attributes that are read only as far a the client is concerned. What if we get a job that tries to edit a user's ID? In a backbone model if I call:

backboneModelInstance.set({id: 'newId'});

That will change the ID of the object. Clearly that's not good in a server environment when you're trusting that to be a unique ID. There are also lots of other fields you may want on the client but you don't want to let users edit.

Again, we can encapsulate that logic in our Capsule models. Capsule models have a safeSet method that assumes all inputs are evil. Unless an attribute is whitelisted as clientEditable it won't set it. So when we go to set attributes within the worker on the server we use safeSet when dealing with untrusted input.

The other important piece of securing a system that lets users indirectly add jobs to your job system is ensuring that the job you receive validate your schema. I'm using a node implementation of JSON Schema for this. I've heard some complaints about that proposed standard, but it works really well for the fairly simple usecase I need it for.

A typical worker may look something like this:

workers.editTeam = function () {  var schema = {    type: "object",    properties: {      user: {        type: 'string',        required: true      },      id: {        type: 'string',        required: true      },      data: {        type: 'object',        required: true      }    }  };  editTeamJob.get(0, function (err, json, jobId, timeout) {    var feed = thoonk.feed('teams'),       result,      team,      newAttributes,      inflated;

async.waterfall([
function (cb) {
// validate our job
validateSchema(json, schema, cb);
},
function (clean, cb) {
// store some variables from our cleaned job
result = clean;
team = result.id;
newAttributes = result.data;
verifyOwnerTeam(team, cb);
},
function (teamData, cb) {
// inflate our capsule model
inflated = new Team(teamData);
// if from the server user normal 'set'
inflated.safeSet(newAttributes);
},
function (cb) {
// do the edit, all we're doing is storing JSON strings w/ ids
feed.edit(JSON.stringify(inflated.toJSON()), result.id, cb);
}
], function (err) {
var code;
if (!err) {
code = 200;
logger.info('edited team', {team: team, attrs: newAttributes});
} else if (err === 'notAllowed') {
code = 403;
logger.warn('not allowed to edit');
} else {
code = 500;
logger.error('error editing team', {err: err, job: json});
}
// finish the job
editTeamJob.finish(jobId, null, JSON.stringify({code: code}));
// keep the loop crankin'
process.nextTick(workers.editTeam);
});
});
};


Sounds like a lot of work

Granted, writing a worker for each type of action a user can perform in the app with all the related job and validation is not an insignificant amount of work. However, it worked rather well for us to use the state syncing stuff in Capsule while we were still in the prototyping stage, then converting the server-side code to a Thoonk-based solution when we were ready to roll out to production.

So why does any of this matter?

It works.

What this ultimately means is that we now push the system until Redis is our bottleneck. We can spin up as many workers as we want to crank through jobs and we can write those workers in any language we want. We can put our node app behind HA proxy or Bouncy and spin up a bunch of 'em. Do we have all of this solved and done? No. But the core ideas and scaling paths seem fairly clear and doable.

[update: Just to add a bit more detail here, from our tests we feel confident that we can scale to tens of thousands of users on a single server and we believe we can scale vertically after doing some intelligent sharding with multiple servers.]

Is this the "Rails of Realtime?"

Nope.

Personally, I'm not convinced there ever will be one. Even Owen Barnes (who originally set out to build just that with SocketStream) said at KRTConf: "There will not be a black box type framework for realtime." His new approach is to build a set of interconnected modules for structuring out a realtime app based on the unique needs of its specific goals.

The kinds of web apps being built these days don't fit into a neat little box. We're talking to multiple web services, multiple databases, and pushing state to the client.

Mikeal Rogers gave a great talk at KRTConf about that exact problem. It's going to be really, really hard to create a framework that solves all those problems in the same way that Rails or Django can solve 90% of the common problems with routes and MVC.

Can you support a BAJILLION users?

No, but a single Redis db can handle a fairly ridiculous amount of users. At the point that actually becomes our bottleneck, (1) we can split out different feeds for different databases, and (2) we'd have a user base that would make the app wildly profitable at that point—certainly more than enough to spend some more time on engineering. What's more, Salvatore and the Redis team are putting a lot of work into clustering and scaling solutions for Redis that very well may outpace our need for sharding, etc.

Have you thought about X, Y, Z?

Maybe not! The point of this post is simply to share what we've learned so far.

You'll notice this isn't a "use our new framework" post. We would still need to do a lot of work to cleanly extract and document a complete realtime app solution from what we've done in &bang—particularly if we were trying to provide a tool that can be used to quickly spin up an app. If your goal is to find a tool like that, definitely check out what Owen and team are doing with SocketStream and what Nate and Brian are doing with Derby.

We love the web, and love the kinds of apps that can be built with modern web technologies. It's our hope that by sharing what we've done, we can push things forward. If you find this post helpful, we'd love your feedback.

Technology is just a tool, ultimately, it's all about building cool stuff. Check out &bang and follow me @HenrikJoreteg, Adam @AdamBrault and the whole @andyet team on the twitterwebz.

If you're building a single page app, keep in mind that &yet offers consulting, training and development services. Hit us up (henrik@andyet.net) and tell us what we can do to help.

filed under &bang, node.js, realtime, scaling, and thoonk 4 comments

posted yesterday by Henrik Joreteg

View the discussion thread.

blog comments powered by Disqus

Redis and client ui goes together like bread and butter.

This is fantastic!

Looks like this is a good article for getting this setup.

The Unix philosophy lies at the core of not only the original UNIX as developed by Ken Thompson at Bell Labs from 1969 but also of its numerous direct descendants and clones, including Solaris, the BSDs¹ and Linux. It has made them, collectively referred to as Unix-like operating systems, into what are widely considered to be the best² operating systems to have been developed to date, despite the fact that they are by far the oldest operating systems in widespread use, and it has been a major factor in the rapid growth and increasing success of Linux.

There is no single, standardized statement of the philosophy. But if it had to be described with only a single word, that word would be modularity, which refers to a system that is composed of components (i.e., modules) that can be fitted together or arranged in a variety of ways.

Modularity is common in nature, and its application to man-made products (both goods and services) has been a key factor in the development and advance of industrial societies. Yet, it was relatively little used for computer software prior to the development of UNIX, and even today its great benefits fail to be fully exploited by other operating systems, most notably the Microsoft Windows systems.

A slightly longer, and more conventional, statement of the Unix philosophy, but one which says essentially the same thing, would be: Design programs to do only a single thing, but to do it well, and to work together well with other programs³.

The philosophy is commonly described with a series of brief rules or tenets⁴. Although they all generally follow logically from the concept of modularity (e.g., small size, efficiency, simple interfaces and ease of understanding), it can be useful to state and discuss them individually.

Among these tenets is the the so-called rule of composition, which states that programs (of which a modular operating system is composed) should be designed to be easily connected to other programs. Another is the rule of silence, which states that programs should, by default, say nothing (i.e., have no output) other than that which is interesting, unusual or surprising.

A serious attempt is made to apply the principle of modularity to everything on a Unix-like system, not only programs but also parts of programs, such as algorithms, and even the kernel (i.e., the core of the operating system). Thus, a Unix-like operating system generally (or at least ideally) consists of a small kernel together with a large number of small, specialized programs that can interact with each other through a variety of well-defined interfaces.

The most familiar of these interfaces, and one of the most important innovations of UNIX, is the pipe. Represented by the vertical bar character in commands typed in by the user, pipes allow the combining of programs so that the output of one becomes the input of another. Such pipelines of commands make it possible to easily perform highly specialized operations that would be difficult or virtually impossible using a non-modular system.

Another major tenet of the philosophy is to use plain text (i.e., human readable alphanumeric characters) rather than binary files (which are not fully human readable) to the extent possible for the inputs and outputs of programs and for configuration files. This is because plain text is a universal interface; that is, it can allow programs to easily interact with each other in the form of text outputs and inputs, in contrast to the difficulty that they would have if each used mutually incompatible binary formats and because such files can be easily interfaced with humans. The latter means that it is easy for humans to study, correct, improve and extend such files as well as to port (i.e., modify) them to new platforms (i.e., other combinations of operating systems and hardware).

In addition to making Unix-like operating systems more efficient and easier to use, the Unix philosophy also facilitates their development and improvement. This is because the use of small, specialized (i.e., modularized) programs makes it far easier for developers to improve them than would be the case with large, multifunctional programs. One reason is that smaller programs can be sufficiently small and simple so that they can be comprehended by a single human mind, whereas large and complex programs generally cannot. It is also because such specialization makes it practical for development (including improvements) to be widely scattered rather than being concentrated at one or a few central locations.

Historical Context

The Unix philosophy grew out of the original design goal of UNIX, which was to create an operating system that was as simple and efficient as possible. This goal was a reaction to what Thompson correctly viewed as the unnecessary complexity of the operating systems that were in use at that time. Such complexity was related to the fact that there was no standard operating system that could be used on a wide variety of computers; rather, each computer manufacturer developed a separate operating system for its own hardware.

Another important factor in Thompson's desire for maximum simplicity and efficiency was the fact that he initially wrote his operating system for a computer, the PDP-7, that had an extremely small main memory of only 4000 18-bit words. (However, this was considered quite adequate when the computer was built in 1965, especially given its low cost, of only U.S.$72,000 for a basic model.)

Although the size and complexity of Unix-like operating systems has increased greatly over subsequent decades in response to the continuous drop in memory costs and increases in CPU (central processing unit) performance, the legacy of the original simplicity lives on, and Unix-like operating systems are still far more modular than most other systems, most notably the Microsoft Windows systems.

The decline in UNIX market share that began in the 1990s relative to the Microsoft Windows systems was sometimes attributed to the perception that UNIX was an old, clunky system that had outlived its usefulness. However, it turned out that this decline was, in fact, not due to any inherent defect in either the philosophy or the operating system itself, nor was it due to any inherent superiority of the Microsoft systems. Rather, it was the result of the fact that the philosophy was designed for a different era, one in which software was treated like mathematics or scientific knowledge. That is, software was something to be shared and improved for everyone's benefit rather than something to be hoarded for the sake of increasing corporate profits.

Thus, rather than discarding the philosophy, something additional was needed so that operating systems based on it could again prosper and grow in this new environment. This addition came in the form of the concept of free software licensing, which keeps software free not only in a monetary sense but also with regard to use (including copying, modifying, extending and redistributing). Particularly important in this context was the development of the concept of copyleft, which precludes the emergence of dominant versions that require the use of expensive hardware and that also promotes compatibility among the various versions without inhibiting innovation. The result has been a returning ownership of Unix-like operating systems and application programs for use with them to the public as a whole and a consequent reinvigoration of technological development.

Contrast With Proprietary Systems

The Unix philosophy is clearly very different from the philosophy behind the Microsoft Windows operating systems. In fact, those systems are characterized by some of the opposite qualities. Most fundamentally, they are monolithic (i.e., lack modularity). Specific characteristics include extremely large program sizes, great complexity of the source code, an apparent lack of craftsmanship on many aspects (such as poor security), a lack of transparency (the source code is a closely guarded secret) and the absence of portability (i.e., can only run on one basic type of processor).

Source code (also referred to as source or code) is the version of software (usually an application program or an operating system) as it is originally written (i.e., typed into a computer) by a human in plain text. It can be written in any of numerous programming languages, some of the most popular of which are C, C++, Java, Perl, PHP, Python and Tcl/Tk.

This sharp contrast is only natural, as the Microsoft Windows operating systems, like much other proprietary (i.e., commercial) software, originated in a different era and are governed by a very different set of priorities. That is, the emphasis is on gaining or maintaining market share and maximizing profits, and little attention is paid to the priorities of the original UNIX developers. The very different development model (e.g., centralized versus dispersed development) also likely plays a role.

Of course, there is nothing necessarily wrong with these market-oriented goals. In fact, they are, in general, beneficial, as they are what has been responsible for much of the tremendous economic growth that has occurred around the world during the past millennium. Yet, as economists would be quick to point out, these goals work for the maximum benefits to society only where free competition (i.e., many sellers and buyers competing on the basis of price and quality) exists, and not where markets have (for whatever reasons) become monopolistic (i.e., characterized by only a single supplier or seller of a product with no good substitutes).

In all fairness, it should be mentioned that the various Unix-like operating systems have not always completely conformed to their idealized philosophy either. In a way, these goals are a bit like democracy -- there is a tendency to stray, but the inherent virtue of the concept keeps pulling it back.

Beyond Proprietary UNIX

The great success of UNIX and its philosophy has had effects far beyond the original operating system as it was developed at Bell Labs and subsequently at the University of California at Berkeley (UCB). Perhaps most significantly, they form the basis for Linux, which is now the most rapidly growing operating system and which many computer experts expect could become the dominant system for many applications in the future.

Linux has been able to bring the advantages of the Unix philosophy back into prominence and restore Unix-like operating systems taken as a whole to their former leadership role largely through the use of free software licensing. Also fundamental to its success, and the growing success of other free software as well, has been the availability of the Internet for allowing collaboration of programmers around the world. This has done a great deal to level the playing field and allow free software to compete effectively with proprietary software (and possibly even endanger the proprietary software development model) despite the massive financial resources of the latter.

In fact, the use of free software licensing together with the ability for programmers (as well as testers and general users) to participate in the development process irrespective of geographical location or organizational membership has further extended the basic concept of the Unix philosophy (i.e., modularity) and has resulted in a large increase in benefits. This is what could be termed the Linux philosophy.

In addition to operating systems, the Unix-philosophy has also played a major role in the swift development and astonishing success of the Internet. It has accomplished this not only by providing an outstanding operating system on which to base much of that network, but also by demonstrating the value of modularization for both the hardware and software aspects of it, which are major factors in its outstanding performance, including its high degree of robustness and scalability.

________
¹In addition to FreeBSD, NetBSD and OpenBSD, this also includes Darwin, which is at the core of the increasingly successful Mac OS X.

²Unix-like operating systems are generally regarded as the most secure, the most robust (i.e., the most resistant to malfunctioning under unusual or stressful circumstances), the most portable (i.e., i.e., the easiest to adapt for use on other types of hardware) and the most configurable.

³Doug McIlroy, the inventor of Unix pipes, summarized the philosophy in Peter H. Salus's 1994 book A Quarter Century of Unix as: "Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface." This is usually severely abridged to "Do one thing, do it well." Of the three parts, only the third is specific to Unix-like operating systems.

⁴The philosophy was well expressed by Mike Gancarz in his 1995 book The UNIX Philosophy as a set of major and minor tenets. The former include: small is beautiful make each program do one thing well, build a prototype as soon as possible, choose portability over efficiency, store numerical data in flat files, use software leverage to your advantage, use shell scripts to increase leverage and portability, avoid captive user interfaces and make every program a filter. The latter include: allow the user to tailor the environment, make operating system kernels small and lightweight, use lower case and keep it short, save trees, silence is golden, think parallel, the sum of the parts if greater than the whole, look for the ninety percent solution, worse is better and think hierarchically.

Created August 6, 2006.
Copyright © 2006 The Linux Information Project. All Rights Reserved.

It is crucial to keep the design of the application ahead of the development. Focus should be placed on wireframing usability, UX, flows, and personas. These larger chunks require wireframes to be done first to be successful with our outside-in process.

Wise words

About

Scala school was started as a series of lectures at Twitter to prepare experienced engineers to be productive Scala programmers. Being a relatively new language, but also one that draws on many familiar concepts, we found this an effective way of getting new engineers up to speed quickly. This is the written material that accompanied those lectures. We have found that these are useful in their own right.

Approach

We think it makes the most sense to approach teaching Scala not as if it's an improved Java but as a new language. Experience in Java is not expected. Focus will be around the interpreter and the object-functional style as well as the style of programming we do here. An emphasis will be placed on maintainability, clarity of expression, and leveraging the type system.

Most of the lessons require no software other than a Scala REPL. The reader is encouraged to follow along, and go further! Use these lessons as a starting point to explore the language.

Vagrant is going to be an essential tool in cloud based systems.

Ship or Die!

There is a tide in the affairs of men.
Which, taken at the flood, leads on to fortune;
Omitted, all the voyage of their life
Is bound in shallows and in miseries.
On such a full sea are we now afloat,
And we must take the current when it serves,
Or lose our ventures.

The quote is from Shakespeare’s Julius Caesar. It sources the ship building enterprise of days gone by when builders couldn’t time the high tides accurately. The smarter builders would launch their vessels even when unfinished, into the waters when the tide hit and hope to complete it at sea. And meanwhile those who chose to ignore the tide in favor of perfectly built ships often dealt with a perfectly build ship rotting away perfectly.

Aside for the navy, the ships were built for a simple purpose - sail to far away lands, acquire goods, raw material, spices for the cheap and return home to sell them at a margin that would cover the cost of the trip and then some more. That’s it.

The essence of the enterprise is to get to these other lands and NOT sit around in the harbor fawning over your creation. Unfortunately that’s what a lot happens in software — gold plated code with unit tests, agile up the wazoo, but too late to the market. Sitting around in the source repo, rotting away perfectly.

I have learnt lessons over a decade plus of writing software and only recently achieving success in shipping it such that the enterprise was successful. We returned home to wine and song and our own sanity. On the way I earned friends and fellow shippers, earned their respect and learned humility to ditch my ideas of what will it take to ship, park that ego or whatever else is in the way and commit to the cause.

Last year (2010), we ended a tremendous journey with exit of (Lil) Green Patch to Playdom/Disney which was one hell of software sprint during which the team reacted to not only changing cloud scale landscape but the very notions of business amidst social media’s nascent awareness of itself. We won. The fat lady sang. 

I now spend my waking hours thinking about what am I shipping today and my sleep about dreaming stuff I want to ship as soon as I wake up, if I don’t ship, I might as well be dead:

And we must take the current when it serves

Acknowledgements: Thanks a lot guys for helping out with the draft! Avichal Garg, Brian Lynch, David King

Don't let good ideas rot in the source repo.

Epic cheat sheet!