Using Comet to update web application data in real time?

Recently I've been looking into a powerful but complex web application technology we're considering using in future versions of the Hub.

The problem: how do we update a browser view when a background process updates the database?

A few examples of places in the Hub where this comes up...

• User launches server, waits for the server to get an IP, for the status icon to change color, etc.
• User associates a server with an elastic IP
• User attaches an EBS or detaches an EBS
• The instance cache expires after N minutes, a background process reloads the information, which is different from the previously displayed information.

You can use "plain old" AJAX polling every N seconds (and most sites with "real-time" data do), but there are a couple of limitations:

1. Will never be real-time
2. It can be tricky to scale

Wouldn't it be neat if instead of polling, any changes to database state could be PUSHED on the fly to the browser without any action on the part of the user?

Well it turns out you can do exactly that via a technique called "reverse AJAX" or "Comet" that uses long waiting requests from the client to the server to make sure there is a connection that's always on and ready to accept "pushed" updates from the server.

But since you don't want to dedicate a synchronous process to every waiting poll, it's best to leverage specially designed server software to help you with that.

Currently Comet is mostly being used for chat-like applications (e.g., GTalk, Meebo, Facebook chat) and seems to add considerable complexity (even more than regular AJAX) so I'm not sure it would be a good idea to use this for something as simple as updating a server view but that depends on implementation details so I wouldn't rule it out off the bat.

Comet is just an umbrella term for this type of technique. The actual protocol used in the implementation is typically STOMP, a messaging protocol supported by RabbitMQ (and many others).

Note that there are many excellent open source components that can be leveraged to implement Comet applications:

• Orbited: an asynchronous COMET server in Python
• js.io: client-side STOMP client

Further reading

• AJAX vs Comet [leahculver.com]
• Comet (programming) [wikipedia.org]
• Concrete Comet examples [cometdaily.com]
• Django Comet tutorial with orbited, stomp and jsio [anirudhsanjeev.org]