Backbone.js and Capsule and Thoonk, oh my! A scalable realtime architecture

This last year, we've learned a lot about building scalable realtime web apps, most of which has come from shipping &bang.

&bang is the app we use to keep our team in sync. It helps us stay on the same page, bug each other less and just get stuff done as a team.

The process of actually trying to get something out the door on a bootstrapped budget helped us focus on the most important problems that needed to be solved to build a dynamic, interactive, real-time app in a scaleable way.

A bit of history

I've written a couple of posts on backbone.js since discovering it. The first one introduces Backbone.js as a lightweight client-side framework for building clean, stateful client apps. In the second post I introduced Capsule.js. Which is a tool that I built on top of Backbone that adds nested models and collections and also allows you to keep a mirror of your client-side state on a node.js server to seemlessly synchronize state between different clients.

That approach was great for quickly prototyping an app. But as I pointed out in that post, that's a lot of in memory state being stored on the server and simply doesn't scale very well.

At the end of that post I hinted at what we were aiming to do to ultimately solve that problem. So this post is meant to be a bit of an update on those thoughts.

Our new approach

Redis is totally freakin' amazing. Period. I can't say enough good things about it. Salvatore Sanfilippo is a god among men, in my book.

Redis can scale.

Redis can do PubSub.

PubSub just means events. Just like you can listen for click events in Javascript in a browser you can listen for events in Redis.

Redis, however is a generic tool. It's purposely fairly low-level so as to be broadly applicable.

What makes Redis so interesting, from my perspective, is that you can treat it as a shared memory between processes, languages and platforms. What that means, in a practical sense, is that as long as each app that uses it interacts with it according to a pre-defined set of rules, you can write a whole ecosystem of functionality for an app in whatever language makes the most sense for that particular task.

Enter Thoonk

My co-worker, Nathan Fritz, is the closest thing you can get to being a veteran of realtime technologies.

He's a member of the XSF council for the XMPP standard and probably wrote his first chat bot before you knew what chat was. His Sleek XMPP Python library is iconic in the XMPP community. He has a self-declared un-natural love for XEP-60 which describes the XMPP PubSub standard.

He took everything he learned from his work on that standard and built Thoonk. (In fact, he actually kept the PubSub spec open as he built the Javascript and Python implementations of Thoonk.)

What is Thoonk??

Thoonk is an abstraction on Redis that provides higher-level datatypes for a more approachable interface. Essentially, staring at Redis as a newbie is a bit intimidating. Not that it's hard to interface with, it's just kind of tricky to figure out how to logically structure and retrieve your data. Thoonk simplifies that into a few data-types that describe common use cases. Primarly "feeds", "sorted feeds", "queues" and "jobs".

You can think of a feed as an ad-hoc database table. They're "cheap" to create and you simply declare them to make them or use them. For example, in &bang, we have all our users in a feed called "users" for looking up user info. But also, each user has a variety of individual feeds. For example, they have a "task" feed and a "shipped" feed. This is where it veers from what people are used to in a relational database model, because each user's tasks are not a part of a global "tasks" feed. Instead, each user has a distinct feed of tasks because that's the entity we want to be able to subscribe to.

So rather than simply breaking down a model into types of data, we end up breaking things into groups of items (a.k.a. "feeds") that we want to be able to track changes to. So, as an example, we may have something like this:

// our main user feedvar userFeed = thoonk.feed('users');

// an individual task feed for a user
var userTaskFeed = thoonk.sortedFeed('team.andyet.members.{{memberID}}.tasks');


Marrying Thoonk and Capsule

Capsule was actually written with Thoonk in mind. In fact that's why they were named the way they did: You know these lovely pneumatic tube systems they use to send cash to bank tellers and at Costco? (PPSHHHHHHH—THOONK! And here's your capsule.)

Anyway, the integration didn't end up being quite as tight as we had originally thought but it still works quite well. Loose coupling is better anyway right?

The core problem I was trying to solve with Capsule was unifying the models that are used to represent the state of the app in the browser and the models you use to describe your data on the server—ideally, not just unifying the data structure, but also letting me share behavior of those objects.

Let me explain.

As I mentioned, we recently shipped &bang. It lets a group of people share their task lists and what they're actively working on with each other.

It spares you from a lot of "what are you working on?" conversations and increases accountability by making your work quite public to the team.

It's a realtime, keyboard-driven, web app that is designed to feel like a desktop app. &bang is a node.js application built entirely with the methods described here.

So, in &bang, a team model has attributes as well as a couple of nested backbone collections such as members and chat messages. Each member has attributes and other nested collections, tasks, shipped items, etc.

Initial state push

When a user first logs in we have to send the entire model state for the team(s) they're on so we can build out the interface (see my previous post for more on that). So, the first thing we do when a user logs in is subscribe them to the relevant Thoonk feeds and perform the the initial state transfer to the client.

To do this, we init an empty team model on the client (a backbone/capsule model shared between client/server) . Then we recurse through our Thoonk feed structures on the server to export the data from the relevant feeds into a data structure that Capsule can use to import that data. The team model is inflated with the data from the server and we draw the interface.

From there, the application is kept in sync using events from Thoonk that get sent over websockets and applied to the client interface. Events like "publish", "change", "retract" and "position".

Once we got the app to the point where this was all working, it was kind of a magical moment, because at this point, any edits that happen in Thoonk will simply get pushed out through the event propagation all the way to the client. Essentially, the inteface that a user sees is largely a slave to the server. Except, of course, the portions of state that we let the user manipulate locally.

At this point, user interactions with the app that change data are all handled through RPC calls. Let's jump back to the server and you'll see what I mean.

I thought you were still using Capsule on the server?

We do, but differently, here's how that is handled.

In short... it's a job system.

Sounds intimidating right? As someone who started in business school, then gradually got into front-end dev, then back-end dev, then a pile of JS, job systems sounded scary. In my mind they're for "hardcore" programmers like Fritzy or Nate or Lance from our team. Job systems don't have to be that scary.

At a very high level you can think of a "job" as a function call. The key difference being, you don't necessarily expect an immediate result. To continue with examples from &bang: a job may be to "ship a task". So, what do we need to know to complete that action? We need the following:

• member Id of the user shipping the task
• the task id being completed (we call this "shipping", because it's cooler, and it's a reminder a reminder that finishing is what's important)

We can derive everything else we need from those key pieces of information.

So, rather than call a function somewhere:

shipTask(memberId, taskId)

We can just describe a job as a simple JSON object:

{    userId: ,    taskId: ,    memberId: }

The we can add that to our "shipTask" job queue like so:

thoonk.job('shipTask').put(JSON.stringify(jobObject));

The cool part about the event propagation I talked about above is we really don't care so much when that job gets done. Obviously fast is key, but what I mean is, we don't have to sit around and wait for a synchronous result because the event propagation we've set up will handle all the application state changes.

So, now we can write a worker that listens for jobs from that job queue. In that worker we'll perform all the necessary related logic. Specifically stuff like:

• Validating that the job is properly formatted (contains required fields of the right type)
• Validating that the user is the owner of that task and is therefore allowed to "ship" it.
• Modifying Thoonk feeds accordingly.

Encapsulating and reusing model logic

You'll notice that part of that list requires some logic. Specifically, checking to see if the user requesting the action is allowed to perform it. We could certainly write that logic right here, in this worker. But, in the client we're also going to want to know if a user is allowed to ship a given task, right? Why write that logic twice?

Instead we write that logic as a method of a Capsule model that describes a task. Then, we can use the same method to determine whether to show the UI that lets the user perform the action in the browser as we use on the back end to actually perform the validation. We do that by re-inflating a Capsule model for that task in our worker code and calling the canEdit() method on it and passing it the user id requesting the action. The only difference being, on the server-side we don't trust the user to tell us who they are. On the server we roll the user id we have for that session into the job when it's created rather then trust the client.

Security

One other, hugely important thing that we get by using Capsule models on the server is some security features. There are some model attributes that are read only as far a the client is concerned. What if we get a job that tries to edit a user's ID? In a backbone model if I call:

backboneModelInstance.set({id: 'newId'});

That will change the ID of the object. Clearly that's not good in a server environment when you're trusting that to be a unique ID. There are also lots of other fields you may want on the client but you don't want to let users edit.

Again, we can encapsulate that logic in our Capsule models. Capsule models have a safeSet method that assumes all inputs are evil. Unless an attribute is whitelisted as clientEditable it won't set it. So when we go to set attributes within the worker on the server we use safeSet when dealing with untrusted input.

The other important piece of securing a system that lets users indirectly add jobs to your job system is ensuring that the job you receive validate your schema. I'm using a node implementation of JSON Schema for this. I've heard some complaints about that proposed standard, but it works really well for the fairly simple usecase I need it for.

A typical worker may look something like this:

workers.editTeam = function () {  var schema = {    type: "object",    properties: {      user: {        type: 'string',        required: true      },      id: {        type: 'string',        required: true      },      data: {        type: 'object',        required: true      }    }  };  editTeamJob.get(0, function (err, json, jobId, timeout) {    var feed = thoonk.feed('teams'),       result,      team,      newAttributes,      inflated;

async.waterfall([
function (cb) {
// validate our job
validateSchema(json, schema, cb);
},
function (clean, cb) {
// store some variables from our cleaned job
result = clean;
team = result.id;
newAttributes = result.data;
verifyOwnerTeam(team, cb);
},
function (teamData, cb) {
// inflate our capsule model
inflated = new Team(teamData);
// if from the server user normal 'set'
inflated.safeSet(newAttributes);
},
function (cb) {
// do the edit, all we're doing is storing JSON strings w/ ids
feed.edit(JSON.stringify(inflated.toJSON()), result.id, cb);
}
], function (err) {
var code;
if (!err) {
code = 200;
logger.info('edited team', {team: team, attrs: newAttributes});
} else if (err === 'notAllowed') {
code = 403;
logger.warn('not allowed to edit');
} else {
code = 500;
logger.error('error editing team', {err: err, job: json});
}
// finish the job
editTeamJob.finish(jobId, null, JSON.stringify({code: code}));
// keep the loop crankin'
process.nextTick(workers.editTeam);
});
});
};


Sounds like a lot of work

Granted, writing a worker for each type of action a user can perform in the app with all the related job and validation is not an insignificant amount of work. However, it worked rather well for us to use the state syncing stuff in Capsule while we were still in the prototyping stage, then converting the server-side code to a Thoonk-based solution when we were ready to roll out to production.

So why does any of this matter?

It works.

What this ultimately means is that we now push the system until Redis is our bottleneck. We can spin up as many workers as we want to crank through jobs and we can write those workers in any language we want. We can put our node app behind HA proxy or Bouncy and spin up a bunch of 'em. Do we have all of this solved and done? No. But the core ideas and scaling paths seem fairly clear and doable.

[update: Just to add a bit more detail here, from our tests we feel confident that we can scale to tens of thousands of users on a single server and we believe we can scale vertically after doing some intelligent sharding with multiple servers.]

Is this the "Rails of Realtime?"

Nope.

Personally, I'm not convinced there ever will be one. Even Owen Barnes (who originally set out to build just that with SocketStream) said at KRTConf: "There will not be a black box type framework for realtime." His new approach is to build a set of interconnected modules for structuring out a realtime app based on the unique needs of its specific goals.

The kinds of web apps being built these days don't fit into a neat little box. We're talking to multiple web services, multiple databases, and pushing state to the client.

Mikeal Rogers gave a great talk at KRTConf about that exact problem. It's going to be really, really hard to create a framework that solves all those problems in the same way that Rails or Django can solve 90% of the common problems with routes and MVC.

Can you support a BAJILLION users?

No, but a single Redis db can handle a fairly ridiculous amount of users. At the point that actually becomes our bottleneck, (1) we can split out different feeds for different databases, and (2) we'd have a user base that would make the app wildly profitable at that point—certainly more than enough to spend some more time on engineering. What's more, Salvatore and the Redis team are putting a lot of work into clustering and scaling solutions for Redis that very well may outpace our need for sharding, etc.

Have you thought about X, Y, Z?

Maybe not! The point of this post is simply to share what we've learned so far.

You'll notice this isn't a "use our new framework" post. We would still need to do a lot of work to cleanly extract and document a complete realtime app solution from what we've done in &bang—particularly if we were trying to provide a tool that can be used to quickly spin up an app. If your goal is to find a tool like that, definitely check out what Owen and team are doing with SocketStream and what Nate and Brian are doing with Derby.

We love the web, and love the kinds of apps that can be built with modern web technologies. It's our hope that by sharing what we've done, we can push things forward. If you find this post helpful, we'd love your feedback.

Technology is just a tool, ultimately, it's all about building cool stuff. Check out &bang and follow me @HenrikJoreteg, Adam @AdamBrault and the whole @andyet team on the twitterwebz.

If you're building a single page app, keep in mind that &yet offers consulting, training and development services. Hit us up (henrik@andyet.net) and tell us what we can do to help.

filed under &bang, node.js, realtime, scaling, and thoonk 4 comments

posted yesterday by Henrik Joreteg

View the discussion thread.

blog comments powered by Disqus

Redis and client ui goes together like bread and butter.

This is fantastic!

Looks like this is a good article for getting this setup.

The Unix philosophy lies at the core of not only the original UNIX as developed by Ken Thompson at Bell Labs from 1969 but also of its numerous direct descendants and clones, including Solaris, the BSDs¹ and Linux. It has made them, collectively referred to as Unix-like operating systems, into what are widely considered to be the best² operating systems to have been developed to date, despite the fact that they are by far the oldest operating systems in widespread use, and it has been a major factor in the rapid growth and increasing success of Linux.

There is no single, standardized statement of the philosophy. But if it had to be described with only a single word, that word would be modularity, which refers to a system that is composed of components (i.e., modules) that can be fitted together or arranged in a variety of ways.

Modularity is common in nature, and its application to man-made products (both goods and services) has been a key factor in the development and advance of industrial societies. Yet, it was relatively little used for computer software prior to the development of UNIX, and even today its great benefits fail to be fully exploited by other operating systems, most notably the Microsoft Windows systems.

A slightly longer, and more conventional, statement of the Unix philosophy, but one which says essentially the same thing, would be: Design programs to do only a single thing, but to do it well, and to work together well with other programs³.

The philosophy is commonly described with a series of brief rules or tenets⁴. Although they all generally follow logically from the concept of modularity (e.g., small size, efficiency, simple interfaces and ease of understanding), it can be useful to state and discuss them individually.

Among these tenets is the the so-called rule of composition, which states that programs (of which a modular operating system is composed) should be designed to be easily connected to other programs. Another is the rule of silence, which states that programs should, by default, say nothing (i.e., have no output) other than that which is interesting, unusual or surprising.

A serious attempt is made to apply the principle of modularity to everything on a Unix-like system, not only programs but also parts of programs, such as algorithms, and even the kernel (i.e., the core of the operating system). Thus, a Unix-like operating system generally (or at least ideally) consists of a small kernel together with a large number of small, specialized programs that can interact with each other through a variety of well-defined interfaces.

The most familiar of these interfaces, and one of the most important innovations of UNIX, is the pipe. Represented by the vertical bar character in commands typed in by the user, pipes allow the combining of programs so that the output of one becomes the input of another. Such pipelines of commands make it possible to easily perform highly specialized operations that would be difficult or virtually impossible using a non-modular system.

Another major tenet of the philosophy is to use plain text (i.e., human readable alphanumeric characters) rather than binary files (which are not fully human readable) to the extent possible for the inputs and outputs of programs and for configuration files. This is because plain text is a universal interface; that is, it can allow programs to easily interact with each other in the form of text outputs and inputs, in contrast to the difficulty that they would have if each used mutually incompatible binary formats and because such files can be easily interfaced with humans. The latter means that it is easy for humans to study, correct, improve and extend such files as well as to port (i.e., modify) them to new platforms (i.e., other combinations of operating systems and hardware).

In addition to making Unix-like operating systems more efficient and easier to use, the Unix philosophy also facilitates their development and improvement. This is because the use of small, specialized (i.e., modularized) programs makes it far easier for developers to improve them than would be the case with large, multifunctional programs. One reason is that smaller programs can be sufficiently small and simple so that they can be comprehended by a single human mind, whereas large and complex programs generally cannot. It is also because such specialization makes it practical for development (including improvements) to be widely scattered rather than being concentrated at one or a few central locations.

Historical Context

The Unix philosophy grew out of the original design goal of UNIX, which was to create an operating system that was as simple and efficient as possible. This goal was a reaction to what Thompson correctly viewed as the unnecessary complexity of the operating systems that were in use at that time. Such complexity was related to the fact that there was no standard operating system that could be used on a wide variety of computers; rather, each computer manufacturer developed a separate operating system for its own hardware.

Another important factor in Thompson's desire for maximum simplicity and efficiency was the fact that he initially wrote his operating system for a computer, the PDP-7, that had an extremely small main memory of only 4000 18-bit words. (However, this was considered quite adequate when the computer was built in 1965, especially given its low cost, of only U.S.$72,000 for a basic model.)

Although the size and complexity of Unix-like operating systems has increased greatly over subsequent decades in response to the continuous drop in memory costs and increases in CPU (central processing unit) performance, the legacy of the original simplicity lives on, and Unix-like operating systems are still far more modular than most other systems, most notably the Microsoft Windows systems.

The decline in UNIX market share that began in the 1990s relative to the Microsoft Windows systems was sometimes attributed to the perception that UNIX was an old, clunky system that had outlived its usefulness. However, it turned out that this decline was, in fact, not due to any inherent defect in either the philosophy or the operating system itself, nor was it due to any inherent superiority of the Microsoft systems. Rather, it was the result of the fact that the philosophy was designed for a different era, one in which software was treated like mathematics or scientific knowledge. That is, software was something to be shared and improved for everyone's benefit rather than something to be hoarded for the sake of increasing corporate profits.

Thus, rather than discarding the philosophy, something additional was needed so that operating systems based on it could again prosper and grow in this new environment. This addition came in the form of the concept of free software licensing, which keeps software free not only in a monetary sense but also with regard to use (including copying, modifying, extending and redistributing). Particularly important in this context was the development of the concept of copyleft, which precludes the emergence of dominant versions that require the use of expensive hardware and that also promotes compatibility among the various versions without inhibiting innovation. The result has been a returning ownership of Unix-like operating systems and application programs for use with them to the public as a whole and a consequent reinvigoration of technological development.

Contrast With Proprietary Systems

The Unix philosophy is clearly very different from the philosophy behind the Microsoft Windows operating systems. In fact, those systems are characterized by some of the opposite qualities. Most fundamentally, they are monolithic (i.e., lack modularity). Specific characteristics include extremely large program sizes, great complexity of the source code, an apparent lack of craftsmanship on many aspects (such as poor security), a lack of transparency (the source code is a closely guarded secret) and the absence of portability (i.e., can only run on one basic type of processor).

Source code (also referred to as source or code) is the version of software (usually an application program or an operating system) as it is originally written (i.e., typed into a computer) by a human in plain text. It can be written in any of numerous programming languages, some of the most popular of which are C, C++, Java, Perl, PHP, Python and Tcl/Tk.

This sharp contrast is only natural, as the Microsoft Windows operating systems, like much other proprietary (i.e., commercial) software, originated in a different era and are governed by a very different set of priorities. That is, the emphasis is on gaining or maintaining market share and maximizing profits, and little attention is paid to the priorities of the original UNIX developers. The very different development model (e.g., centralized versus dispersed development) also likely plays a role.

Of course, there is nothing necessarily wrong with these market-oriented goals. In fact, they are, in general, beneficial, as they are what has been responsible for much of the tremendous economic growth that has occurred around the world during the past millennium. Yet, as economists would be quick to point out, these goals work for the maximum benefits to society only where free competition (i.e., many sellers and buyers competing on the basis of price and quality) exists, and not where markets have (for whatever reasons) become monopolistic (i.e., characterized by only a single supplier or seller of a product with no good substitutes).

In all fairness, it should be mentioned that the various Unix-like operating systems have not always completely conformed to their idealized philosophy either. In a way, these goals are a bit like democracy -- there is a tendency to stray, but the inherent virtue of the concept keeps pulling it back.

Beyond Proprietary UNIX

The great success of UNIX and its philosophy has had effects far beyond the original operating system as it was developed at Bell Labs and subsequently at the University of California at Berkeley (UCB). Perhaps most significantly, they form the basis for Linux, which is now the most rapidly growing operating system and which many computer experts expect could become the dominant system for many applications in the future.

Linux has been able to bring the advantages of the Unix philosophy back into prominence and restore Unix-like operating systems taken as a whole to their former leadership role largely through the use of free software licensing. Also fundamental to its success, and the growing success of other free software as well, has been the availability of the Internet for allowing collaboration of programmers around the world. This has done a great deal to level the playing field and allow free software to compete effectively with proprietary software (and possibly even endanger the proprietary software development model) despite the massive financial resources of the latter.

In fact, the use of free software licensing together with the ability for programmers (as well as testers and general users) to participate in the development process irrespective of geographical location or organizational membership has further extended the basic concept of the Unix philosophy (i.e., modularity) and has resulted in a large increase in benefits. This is what could be termed the Linux philosophy.

In addition to operating systems, the Unix-philosophy has also played a major role in the swift development and astonishing success of the Internet. It has accomplished this not only by providing an outstanding operating system on which to base much of that network, but also by demonstrating the value of modularization for both the hardware and software aspects of it, which are major factors in its outstanding performance, including its high degree of robustness and scalability.

________
¹In addition to FreeBSD, NetBSD and OpenBSD, this also includes Darwin, which is at the core of the increasingly successful Mac OS X.

²Unix-like operating systems are generally regarded as the most secure, the most robust (i.e., the most resistant to malfunctioning under unusual or stressful circumstances), the most portable (i.e., i.e., the easiest to adapt for use on other types of hardware) and the most configurable.

³Doug McIlroy, the inventor of Unix pipes, summarized the philosophy in Peter H. Salus's 1994 book A Quarter Century of Unix as: "Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface." This is usually severely abridged to "Do one thing, do it well." Of the three parts, only the third is specific to Unix-like operating systems.

⁴The philosophy was well expressed by Mike Gancarz in his 1995 book The UNIX Philosophy as a set of major and minor tenets. The former include: small is beautiful make each program do one thing well, build a prototype as soon as possible, choose portability over efficiency, store numerical data in flat files, use software leverage to your advantage, use shell scripts to increase leverage and portability, avoid captive user interfaces and make every program a filter. The latter include: allow the user to tailor the environment, make operating system kernels small and lightweight, use lower case and keep it short, save trees, silence is golden, think parallel, the sum of the parts if greater than the whole, look for the ninety percent solution, worse is better and think hierarchically.

Created August 6, 2006.
Copyright © 2006 The Linux Information Project. All Rights Reserved.

It is crucial to keep the design of the application ahead of the development. Focus should be placed on wireframing usability, UX, flows, and personas. These larger chunks require wireframes to be done first to be successful with our outside-in process.

Wise words

About

Scala school was started as a series of lectures at Twitter to prepare experienced engineers to be productive Scala programmers. Being a relatively new language, but also one that draws on many familiar concepts, we found this an effective way of getting new engineers up to speed quickly. This is the written material that accompanied those lectures. We have found that these are useful in their own right.

Approach

We think it makes the most sense to approach teaching Scala not as if it's an improved Java but as a new language. Experience in Java is not expected. Focus will be around the interpreter and the object-functional style as well as the style of programming we do here. An emphasis will be placed on maintainability, clarity of expression, and leveraging the type system.

Most of the lessons require no software other than a Scala REPL. The reader is encouraged to follow along, and go further! Use these lessons as a starting point to explore the language.

Vagrant is going to be an essential tool in cloud based systems.

Ship or Die!

There is a tide in the affairs of men.
Which, taken at the flood, leads on to fortune;
Omitted, all the voyage of their life
Is bound in shallows and in miseries.
On such a full sea are we now afloat,
And we must take the current when it serves,
Or lose our ventures.

The quote is from Shakespeare’s Julius Caesar. It sources the ship building enterprise of days gone by when builders couldn’t time the high tides accurately. The smarter builders would launch their vessels even when unfinished, into the waters when the tide hit and hope to complete it at sea. And meanwhile those who chose to ignore the tide in favor of perfectly built ships often dealt with a perfectly build ship rotting away perfectly.

Aside for the navy, the ships were built for a simple purpose - sail to far away lands, acquire goods, raw material, spices for the cheap and return home to sell them at a margin that would cover the cost of the trip and then some more. That’s it.

The essence of the enterprise is to get to these other lands and NOT sit around in the harbor fawning over your creation. Unfortunately that’s what a lot happens in software — gold plated code with unit tests, agile up the wazoo, but too late to the market. Sitting around in the source repo, rotting away perfectly.

I have learnt lessons over a decade plus of writing software and only recently achieving success in shipping it such that the enterprise was successful. We returned home to wine and song and our own sanity. On the way I earned friends and fellow shippers, earned their respect and learned humility to ditch my ideas of what will it take to ship, park that ego or whatever else is in the way and commit to the cause.

Last year (2010), we ended a tremendous journey with exit of (Lil) Green Patch to Playdom/Disney which was one hell of software sprint during which the team reacted to not only changing cloud scale landscape but the very notions of business amidst social media’s nascent awareness of itself. We won. The fat lady sang. 

I now spend my waking hours thinking about what am I shipping today and my sleep about dreaming stuff I want to ship as soon as I wake up, if I don’t ship, I might as well be dead:

And we must take the current when it serves

Acknowledgements: Thanks a lot guys for helping out with the draft! Avichal Garg, Brian Lynch, David King

Don't let good ideas rot in the source repo.

Epic cheat sheet!

A great explanation of the confusing terms that Chef uses.

The Facial Hair Theory

This could be a worthy replacement for VirtualBox (which I'm keen to get away from given Oracle's Open Source track record).

Dark music video from @kbee0001

MemoryImage

31 August 2011

database · application architecture

tags:

When people start an enterprise application, one of the earliest questions is "how do we talk to the database". These days they may ask a slightly different question "what kind of database should we use - relational or one of these NOSQL databases?". But there's another question to consider: "should we use a database at all?"

One of the defining characteristics of enterprise applications is the need to store long term data, which naturally leads people to reach for a database. After all persisting data is one of the main things databases do. Using a memory image is a different route to persistence that doesn't involve a database.

The key element to a memory image is using event sourcing, which essentially means that every change to the application's state is captured in an event which is logged into a persistent store. Furthermore it means that you can rebuild the full application state by replaying these events. The events are then the primary persistence mechanism.

A familiar example of a system that uses event sourcing is a version control system. Every change is captured as a commit, and you can rebuild the current state of the code base by replaying the commits into an empty directory. In practice, of course, it's too slow to replay all the events, so the system persists periodic snapshots of the application state. Then rebuilding involves loading the latest snapshot and replaying any events since that snapshot.

Event sourcing has many consequences, including the ability to rebuild past states. But the important property for memory images is that it means that there is no longer any need to worry about keeping the application state in an up-to-date persistent store. Instead you can just keep the application state in main memory. Should the process crash, you can rebuild it from the events (and snapshots).

Using a memory image allows you to get high performance, since everything is being done in-memory with no IO or remote calls to database systems. Perhaps more importantly it means you can get rid of database mapping code, or worrying about synchronizing between in-memory state and database state.

Against that you do have to ensure you ensure you can reliably store the events and process them. You also need to write the code to save and load snapshots and figure out how to restore the system quickly enough to keep your quality of service up. Databases also provide transactional concurrency as well as persistence, so you have to figure out what you are going to do about concurrency.

Another, rather obvious, limitation is that you have to have more memory than data you need to keep in it. As memory sizes steadily increase, that's becoming much less of a limitation than it used to be.[1]

A number of different kinds of systems can make use of a memory image, I'll mention three examples I've come across.

The most recent is LMAX. LMAX is a high performance trading system, which processes 6 million TPS on a single JVM thread. Here the performance advantage of a memory image is obviously a big factor, but they found the simplification in programming model to be equally important. They don't have to worry about concurrency as its all about a single thread. To keep availability high, they run multiple copies of the memory image so if one goes down they can switch over to another instance while keeping their very high transaction rate.

A few years ago I wrote about a couple of systems using an EventPoster architecture. This style provides read access to the in-memory model to lots of UIs for analytic purposes. Multiple UIs mean multiple threads, but there's only one writer (the event processor) which greatly simplifies concurrency issues.

The oldest example is also the inspiration for the name - the Smalltalk development environment. Most development tools rely on text files in a file system which a compiled or interpreted as needed. Smalltalk held all its source code and compiled method inside the image [2]. Every command you executed got stored inside a change log. Most of the time you saved your image (snapshot) but if necessary you could replay the change log from a stable base if you did something foolish.

Like many of these kinds of ideas, it's an approach that's been used and reinvented many times [3], but never got mainstream traction. Having a database hold persistant data continues to be the more common approach.

One problem I've heard of with memory images is around migration. Whenever you are building a software system it's important to understand how it will handle changes. With a memory image, the essential task is to ensure you can continue to rebuild the memory image from the the event log.

One trap here is to use an serialization structure for the event log that doesn't handle evolution gracefully should you want to change the structure of events. If you create specific event classes and serialize them, this may make it difficult to process old events should you change the structure of the event class later. Often it's best to serialize with generic data structures such as maps and lists.

Also it's important to keep a good decoupling between the events and the model structure itself. It may be tempting to come up with some automatic mapping system that retrospects on the event data and the model, but this couples the events and model together which makes it difficult to migrate the model and still process old events.

At some point, it may be worthwhile to migrate the event log itself from an old format to a new one. Migrating the event log is often more hassle, but may be an option if you've evolved a long way from the original event structures.

For a long time, a big argument against using a memory image was size, but now most commodity servers have more memory than we were used to having in disk. As a result most working sets can now be held safely in memory. We noticed this a few years ago, but memory images are still relatively rare. I think that now the NOSQL movement is causing people to re-think their options for persistence, we may see an upsurge in this pattern.

1: You may also be able to reduce memory usage if you only need a subset of the data that's in the events. You can send the same events to different memory image systems for different subsets of data if that makes sense for your needs.

2: I'm using the past tense here for Smalltalk because it's a long time since I used it and it may have changed its behavior over the years. It is still around, although sadly only a niche environment.

3: Some people also may remember the Prevayler project, which is an open-source implementation of this approach. It generated a lot of noise in the Java community a few years ago, but has been rather quiet since. That community uses system prevalence as a generic term for this approach.

Events as a persistance mechanism.

docs

http://docs.nodejitsu.com

community powered rocket-fuel for node.js

We believe in sharing knowledge. So we have assembled this growing collection of node.js how-to articles. These articles range from basic to advanced. They provide relevant code samples and insights into the design and philosophy of node itself.

docs.nodejitsu.com is an open source project and is curated by the Nodejitsu team and friends. If you have articles or ideas that you would like to contribute, we'd very much like to accept your pull request!

Usage

Read Articles, get good at Node.js

Browse /articles/ folder or http://docs.nodejitsu.com

To generate the docs

node bin/generate

To start the docs server

node bin/server

Contribution Guide

coming soon

To add an article:

• make a directory in topics for your article: mkdir articles/how-to-make-an-article (use only letters and dashes)
• next write your article: vim articles/how-to-make-an-article/article.md
• create a metadata with this data: vim articles/how-to-make-an-article/metadata.json

metadata.json

javascript { "title":"What is npm?", "date": "Fri Aug 26 2011 03:08:50 GMT-0700 (PST)", "tags": ["npm", "modules"], "author": "nico", "difficulty": 1 }

Directory Structure

topics/
    article-name/ //url version
        metadata.json
        article.md //file with the real article
        assets/
            ...
    ...

Metadata Format

javascript { "title":"What is npm?", "date": "Fri Aug 26 2011 03:08:50 GMT-0700 (PST)", "tags": ["npm", "modules"], "author": "nico", "difficulty": 1 }

Community node.js articles

How It Works

PhantomJS is a command-line tool that packs and embeds WebKit. Literally it acts like any other WebKit-based web browser, except that nothing gets displayed to the screen (thus, the term headless). In addition to that, PhantomJS can be controlled or scripted using its JavaScript API.

Invoking PhantomJS is easy:

phantomjs script.js [arguments]

where script.js is the script to be invoked and arguments are the optional parameters for the script.

The simplest PhantomJS script looks like this:

console.log('Hello, world!');phantom.exit();

The directory examples in the source code contains various sample scripts. For more information, see also the Quick Start guide. Furthermore, there are more examples in the Service Integration wiki page on how to consume the data from web services using XML, JSONP, and YQL.

Display twitter status

The following script loads twitter URL for a particular user and extract his recent tweets using CSS selector.

var page = new WebPage();page.onConsoleMessage = function(msg) {    console.log(msg);};page.open(encodeURI("http://mobile.twitter.com/Sencha"), function (status) {    if (status !== "success") {        console.log("Unable to access network");    } else {        page.evaluate(function() {            var list = document.querySelectorAll('span.status');            for (var i = 0; i < list.length; ++i) {                console.log((i + 1) + ": " + list[i].innerHTML.replace(/<.*?>/g, ''));            }        });    }    phantom.exit();});

Find pizza in New York

Here is another example of CSS selector: finding pizza in New York using Google Places. This script would list 10 pizzeria, along with the address and the telephone number.

var page = new WebPage();page.open('http://www.google.com/m/local?site=local&q=pizza+in+new+york', function (status) {    if (status !== 'success') {        console.log('Unable to access network');    } else {        var results = page.evaluate(function() {            var list = document.querySelectorAll('div.bf'), pizza = [], i;            for (i = 0; i < list.length; i++) {                pizza.push(list[i].innerText);            }            return pizza;        });        console.log(results.join('\n'));    }    phantom.exit();});

Get approximate location

This ipgeocode.js script shows the approximated location based on the network IP address, using freegeoip JSONP API.

var cb = function (data) {    var loc = data.city;    if (data.region_name.length > 0)        loc = loc + ', ' + data.region_name;    console.log('IP address: ' + data.ip);    console.log('Estimated location: ' + loc);    phantom.exit();};var el = document.createElement('script');el.src = 'http://freegeoip.net/json/?callback=cb';document.body.appendChild(el);

Example output:

> phantomjs ipgeocode.js
IP address: 75.25.137.196
Estimated location: Palo Alto, California

For testing legacy applications

10 Papers Every Programmer Should Read (At Least Twice) 525

Posted by Michael Feathers on Friday, February 27, 2009

I spent most of yesterday afternoon working on a paper I’m co-writing. It was one of those days when the writing came easy. I was moving from topic to topic, but then I realized that I was reaching too far backward – I was explaining things which I shouldn’t have had to explain to the audience I was trying to reach.

When I first started writing, one of the pieces of advice that I heard was that you should always imagine that you are writing to a particular person. It gets your juices going – you’re automatically in an explanatory state of mind and you know what you can expect from your audience. I was doing that, but I noticed that I was drifting. I was losing my sense of audience. I started to explain one thing, and then I realized that I would have to explain something else to help it make sense. I couldn’t imagine that person any more. How could I know what they know and what they don’t?

The problem I was experiencing is only getting worse. People come into programming from many different directions. Some started in other fields, and others started programming as teens. Some started with BASIC, others started with Ruby or C. The industry is filled with knowledge, but it isn’t common knowledge. It isn’t knowledge that we all share. We have to dig for it because of a peculiar fact about our industry: we reinvent our languages and notations every ten years. It’s hard to find deeply technical books and articles which stand the test of time in software: they are all Latin within 20 years.

So, I was thinking about this and trying to not to get too glum. I realized that instead of complaining, I could help by pointing to some papers which are easily available online and which (to me at least) point to some of the most interesting ideas about software. To me, these are classic papers which contain deep “things you oughta know” about code – the material you work with.

We’ve taken an interesting turn in the industry over the past ten years. We’ve come to value experiential learning much more, and we’ve regained a strong pragmatic focus, but I think it would be a shame if we lost sight of some of the deeper things which people have learned over the past 50 years. Rediscovering them would be painful, and (to me) not knowing them would be a shame.

Here’s the original list. It’s a rather personal list of foundational papers and papers with deep ideas. I wrote it “off the cuff” and threw it into a tumblr blog the other day and I got responses from people who suggested others. I’ll add those in a later blog.

Most are easy to read but some are rough going – they drop off into math after the first few pages. Take the math to tolerance and then move on. The ideas are the important thing.

1. On the criteria to be used in decomposing systems into modules – David Parnas
2. A Note On Distributed Computing – Jim Waldo, Geoff Wyant, Ann Wollrath, Sam Kendall
3. The Next 700 Programming Languages – P. J. Landin
4. Can Programming Be Liberated from the von Neumann Style? – John Backus
5. Reflections on Trusting Trust – Ken Thompson
6. Lisp: Good News, Bad News, How to Win Big – Richard Gabriel
7. An experimental evaluation of the assumption of independence in multiversion programming – John Knight and Nancy Leveson
8. Arguments and Results – James Noble
9. A Laboratory For Teaching Object-Oriented Thinking – Kent Beck, Ward Cunningham
10. Programming as an Experience: the inspiration for Self – David Ungar, Randall B. Smith

(edit: Added a brief synopsis of each of them and why I think they are special):

On the criteria to be used in decomposing systems into modules – Parnas

This is a very old paper, but it is more than a classic. In in it, Parnas introduces a forerunner to the Single Responsibility Principle. He introduces the idea that we should use modularity to hide design decisions – things which could change. People still don’t consider this as often as they should.

Another thing I really like in the paper is his comment on the KWIC system which he used as an example. He mentioned that it would take a good programmer a week or two to code. Today, it would take practically no time at all. Thumbs up for improved skills and better tools. We have made progress.

A Note On Distributed Computing – Waldo, Wyant, Wollrath, Kendall

Abstraction is great but it can only go so far. In this paper, the authors lay to rest what was once a pervasive myth – that we could design a distributed system and make distribution transparent. Ever wonder why you had to implement specific interfaces to do remoting in Java? This is why.

In the aftermath it might seem hard to believe that people thought this was possible. I think we can we partially thank this paper for that.

The Next 700 Programming Languages – Landin

Most of us have spent a lot of time working in traditional programming languages, but functional programming languages are slowly seeing an uptick and many OO languages are gaining functional features. This paper (which reads like a tutorial) makes an argument for an expression-oriented style of programming. It also lays the foundation for lazy evaluation.

One of the other neat things about this paper, from a historical point of view, is that there is a discussion section at the end in which there a number of questions and comments about whether making indentation significant in a language is a good idea. I was thrown to see people asking whether or not this would be a problem for functions which span over several pages(!).

Can Programming Be Liberated from the von Neumann Style? – Backus

John Backus is known for a number of achievements in computer science. He received the ACM Turing Award for his work on Fortran. This paper, which he presented at the award ceremony was rather shocking at the time because it said, in essence, “we got it wrong.” Backus took the opportunity to make a plea for pure functional programming. His arguments were convincing and they helped to set a research agenda which is just now starting to make some waves in the mainstream.

Reflections on Trusting Trust – Thompson

I once heard that when this paper was presented, people in attendance rushed back to de-compile their C compilers and look for, er, problems. This paper unveiled a hard problem at the heart of computer security. If you’ve spent any time at all thinking about security, you need to read it.

Lisp: Good News, Bad News, How to Win Big – Gabriel

This paper is a bit atypical in this list. It’s aimed at the Lisp community and it comes off as a bit of a lament. But, hidden deep within it is the Gabriel’s description of the ‘Worse is Better’ philosophy – an idea with profound implications for the acceptance and spread of technology.

An experimental evaluation of the assumption of independence in multiversion programming – John Knight and Nancy Leveson

Behind this dry title lies something very interesting. I first heard about this paper from Ralph Johnson in a newsgroup discussion about program correctness. It turns out that one of the avenues that engineers in other disciplines take to make their products stronger – redundancy – doesn’t really work in software. Multi-version programming was the idea that you could decrease faults in critical systems by handing the spec to several teams, having them develop the software independently, and then having the systems run in parallel. A monitoring process verifies their results and if there is any discrepancy it picks the most common result. Sounds like it should work, right? Well..

Arguments and Results – Noble

I think that all of the other papers in this list are rather well known in some circles. This one isn’t, or if it is, I just haven’t found that circle yet. What I like about this paper is that it takes something which we deal with every day – the set of arguments and results of functions – and it works them through a series of variations which just don’t occur to many people. The fact is, every function that you work with has a number of possible directions if could evolve in. Not all of them are appropriate, but if you know the possible directions, you’re richer for it.

A Laboratory For Teaching Object-Oriented Thinking – Beck, Cunningham

There are an incredible number of papers about there about object orientation. The thing which makes this one great is its directness. OO went through a number of stages. It was once fresh and novel, then it was ornate, and then it became matter-of-fact. This paper hits upon key ideas which many people don’t talk about much any more: anthropomorphism and dropping the top/down perspective. It also shows you how you can design with index cards. It may not sound cool but it is incredibly effective.

Programming as an Experience: the inspiration for Self – Ungar, Smith

How many people know about the Self Project? Not enough in my opinion. Self was an attempt to take two ideas in computing and push them as far as humanly possible. The first was minimalism: the Self programming language was thoroughly in the Lisp and Smalltalk vein – everything was defined in terms of the smallest number of primitives possible. The other idea was direct manipulation – the idea that the object metaphor could be pushed all the way in the user interface – the programmer and user sit with a mouse in a sea of directly clickable objects and use them for everything. If they could’ve gotten away with dropping the keyboard, I think they would’ve.

The amount of technology which the Self project threw off is terrifying also. Self broke ground in dynamic language optimization and VM design. Chances are, your VM uses technology it pioneered. It’s also one of the more perverse ironies that the most widely distributed programming language today (JavaScript) is a prototype-based programming language which borrowed ideas from the hyper-research-y Self.

What would you add to the list?

• Posted in Michaels Musings
• Meta 525 comments, permalink, rss, atom

I should read these at least once!

This looks very useful!

Interesting