1

Getting your feet wet with node.js and socket.io - Part 1

February 21,2011
I havent had a ton of time to work on my node.js project lately so I figured Id just put it out there in its current form as a learning tool for others. In this blog post Ill walk you through everything I did to get it working and, by the end, you should be able to write your own basic webapps using socket.io. First, lets start with an overview of the functionality wed like to have. We want a semi-real time news feed that can display messages passed in from one or more client applications. In order to tell which application passed in which message, wed like to have some color coding as well. Since were using socket.io we also want to avoid AJAX callbacks in favor of server pushes. Finally - it would be cool to see a count of the number of other people also viewing the news feed. For this project Im going to use the Jade templating language, and the Express node.js framework, which provides Sinatra style routing for node applications. Links to each can be found in the project readme. Lets start off by creating our app and setting it up to use Jade, socket.io, and Express. var app = require('express').createServer(); var socket = require('socket.io').listen(app); require('jade'); app.set('view engine', 'jade'); app.set('view options', {layout: false}); We need to import the Express library and call the createServer() method on it to get a server object. Well store it in app. Then we import the socket.io library and attatch it to listen on our newly created server. Finally we import the Jade library, set our app to use it as the view engine and configure some basic settings. Now that we have the basic setup out of the way - lets set up our routes to serve both our main news feed page as well as any javascript or css we may need. If youre familiar with Sinatra in Ruby and Javascripts callback syntax - these will make perfect sense to you. app.get('/*.(js|css)', function(req, res){ res.sendfile("./public"+req.url); }); app.get('/', function(req, res){ res.render('index'); }); Youll see that we match any GET requests for js and css files and pass them along to the public dir, which is where we will serve our static files from. Then well take any root level requests and serve up our index view. As we have it written here, node will look for a file called index.jade in the views directory, run it through the Jade processor, and then render it out to the browser. At this point, we should probably take a quick look at the index view. Theres nothing complicated here, just a couple of lists and a couple of divs. I wont go over the syntax of Jade, but it should be pretty self explanatory. We set up some script tags to include JQuery and socket.io from their respective CDNs and a local javascript file for our application specific code. Then we add a couple of lists for our news feed and a footer to display our connected client count. !!! 5 html(lang="en") head title nodeFeed link(rel="stylesheet", type="text/css", href="/main.css") script(type="text/javascript", src="https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js") script(type="text/javascript", src="http://cdn.socket.io/stable/socket.io.js") script(type="text/javascript", src="/feed.js") body #heading h1 News Feed #content #categories

2
ul#category_list #feed ul#feed_list #footer p span#client_count 0 | connected clients If we were to run the node server now, it would work, but it wouldnt do anything - it would just render out our basically empty view. Not particularly useful. Now that we have the basic structure in place however, the fun can begin. Lets get socket.io hooked up and talking to the clients. Well start by implementing a real-time count of connected clients. On the server side it will look something like this: var activeClients = 0; socket.on('connection', function(client){ activeClients +=1; socket.broadcast({clients:activeClients}) client.on('disconnect', function(){clientDisconnect(client)}); }); function clientDisconnect(client){ activeClients -=1; client.broadcast({clients:activeClients}) } We will bind an anonymous function to the socket connection event that will execute every time a new client connects. When this happens we will increment the current count of connected clients, and then send out a broadcast containing the count. Thesocket.broadcast method sends a message to all connected clients. In this case were passing some JSON that contains our count. We then use the same syntax to bind a function to execute when the client disconnects. This function simply decrements the count and broadcasts the count back to the remaining clients. Were all set on the server side but we need to set up some javascript on the client side to listen for the messages and update the view accordingly. function msgReceived(msg){ $clientCounter.html(msg.clients); } $(document).ready(function () { $clientCounter = $("#client_count") var socket = new io.Socket(null, {port: 3000}); socket.connect(); socket.on('message', function(msg){msgReceived(msg)}); }); This code will need to go in the feed.js file we linked from the main view. Were using JQuerys document ready function to set up our client side socket and cache the selector for the client count on our page. Node.js runs on port 3000 by default so thats what well enter here. Well then bind a function to execute on the sockets message event. This seems like overkill now as the function is only one line, we could just as easily declare it inline anonymously, but well be building on it in the next step. For now well just update the client count. At this point we have a working app - you can start up node.js, navigate to localhost:3000 and see: 1 connected client. If you open the page in another tab, youll see the counter update to 2 on both pages. Likewise, it will decrement as you close tabs. So far weve seen how to set up a very simple node.js app, import the Jade, Express, and socket.io libraries, and pass messages from the server to the clients using comet style message pushes. Now that we have the basics down, well work on our news feed in the next post. Hope you found this helpful - if you want to skip ahead and see the finished product you can take a look at the github repo, otherwise well cover the rest in the next post.

Websockets everywhere with Socket.IO

If you've stayed on top of the advances in the realtime web for the past few years, you've probably heard of different techniques aimed to reduce the latency (ie: speed) of the message exchange between a client and a server. If you're developing a multiplayer game, a chat application, or showing frequent updates of data like tweets or stock price changes, you probably want to reverse the traditional model of communication. So instead of requesting (polling) data on a specific interval of time, you want the server to send (push) data to the client. Nowadays, terms like long polling, comet and WebSocket come to mind when it comes to developing a realtime web application or widget. But it's not always obvious how they work, their advantages, disadvantages and limitations, or even what percentage of the web browser market share supports them. Socket.IO is a lightweight API that runs on the browser and looks like this: socketio-sample.js var socket = new io.Socket(); socket.on('connect', function(){ // connected! }); socket.on('message', function(msg){ // message coming }); socket.send('Hello world!'); If you're familiar with WebSocket, the protocol that aims to simplify bi-directional communication over HTTP once and for all, you'll notice that it looks very similar. The difference is that Socket.IO, under the hood, will enable realtime communication for IE6-9, Firefox 3-4, Safari 3-5, Chrome 3-6, iOS (iPhone and iPad), and other commonplace user agents. Revisiting the history of the realtime web In this day and age, odds are that if you're a web developer you've used AJAX once or twice. Very much like what socket.io does for realtime, libraries like jQuery have provided abstractions that aim to remove the incompatibilities of what browsers offer for asynchronous HTTP requests (IE uses a proprietary ActiveX object, and mostly everyone else uses the standard XMLHttpRequest). Now, if you wanted to make a realtime widget that retrieves data from the server, your first idea might look somewhat like this: setInterval(function(){ $.ajax({ url: '/my/page', success: function(data){ // do something with the data } }); }, 5000); So, every 5 seconds we poll the server for new updates. In my book, that's almost as efficient as thetransmission of IP Datagrams on pigeons. You might also want to try to reduce the interval, and say, put it at 100 milliseconds. setInterval(function(){ // now this should be fast! $.ajax({ url: '/my/page', success: function(data){} }); }, 100); However, we're ignoring two major downsides now: The HTTP latency. Chances are that a complete roundtrip of the packets on a high speed internet connection will be around 200ms. But this is not always the case! If it were 500 or higher, then things might slow down. And they might slow down unnecessarily, because: The server might not have any new data for us. In this case, we'd be producing a lot of network traffic, and request/response cycles overhead, for no purpose.

4
Introducing long polling Long polling addresses the weakness of traditional polling methods by asking the server for new information on a certain interval, but keeping the connection open if the server has nothing new for us. This technique dramatically decreases latency and network traffic, which means it efficiently disguises itself as a server-push technique. function load(){ $.ajax({ url: '/my/page', success: function(){ // do something with the data }, complete: load, timeout: 20000 }); } How about keeping the connection open? If you come from a more traditional programming environment (eg: Desktop software), you're probably wondering why we don't keep the connection open. This is possible with at least two fairly well known techniques: XMLHttpRequest and the multipart/x-mixed-replace MIME type (which is enabled by setting multipart = true in the XMLHTTPRequest instance)

Although it was introduced by Netscape in 1995 (yes, when some of us were still unable to read properly), the only commonplace user agent to support it is Firefox. An <iframe> populated with a response with the headers Transfer-encoding: chunked and Connection: keep-alive.

The technique consists of writing <script> tags that call a function on the parent page as information becomes available to push to the client. The disadvantage of this method is that it'll trigger a never-ending spinner or progress bar in most user agents, severely hurting the user experience. In Internet Explorer, this can be worked around by inserting the <iframe>in a hidden document (via the obscure ActiveX object htmlfile). This technique was exposed to me for the first time thanks to the Gmail Chat team. This gem was analyzed/discovered back in the day by Alex Russell By now, it's obvious that some lower-latency techniques are available to certain user agents, under certain conditions. The fundamental problem is that now the server has to treat HTTP requests differently, altering The headers sent with the response (Content-Type, Connection, etc). The duration (a timeout is required for long-polling, but not all the others) The "framing" of the messages. For multipart, each message has to be accompanied by a delimiter (boundary). Random quirks (IE requires a certain number of dummy bytes at the beginning of the document streamed through the iframe).

All these techniques try to minimize the latency of the incoming data from the server, but normal XMLHTTPRequest have to be used to send data from the client. Which brings us to the most optimal solution available today. One transport to rule them all Meet WebSocket, an effort "to provide a mechanism for browser-based applications that need two-way communication with servers that does not rely on opening multiple HTTP connections", as the author Ian Hicksonputs it. WebSocket takes advantage of the Upgrade header of the HTTP/1.1 specification, which means it's essentially a new protocol for communication: The Upgrade general-header allows the client to specify what additional communication protocols it supports and would like to use if the server finds it appropriate to switch protocols.

5
Examples: HTTP/2.0, SHTTP/1.3, IRC/6.9, RTA/x11 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html WebSocket won't close the connection after sending a message or receiving one. It's essentially "TCP for the web", but with a security model built in the protocol, a fairly rare framing system and UTF-8 encoding (no binary). If we choose to implement it, some of the problems stated above still hold true: The server has to give special treatment to the WebSocket requests, performing a handshake unique to the WebSocket protocol, and implement its new security system. WebSocket is only supported by the most cutting-edge browser engines on the Desktop, a minority of the web population.

Node.JS koolaid to the rescue Node.JS presents developers with a truly unique and exciting possibility: rolling your own scalable non-blocking HTTP server in one of the most popular dynamic scripting languages of all time, with a simplistic API. Writing a (dummy) long-polling server is as easy as: longpolling.js http.createServer(function (request, response) { setTimeout(function(){ response.writeHead(200, {'Content-Type': 'text/plain'}); response.end('Hello World\n'); }, 20000); }).listen(8124); Even with this simple API, consolidating your app logic in a way that works across all transports can be difficult. Socket.IO-node is here to help: Here's a chat server in 10 lines of code that also announces (broadcasts) who connects to the server: broadcasts.js var buffer = []; io.on('connection', function(client){ client.send({ buffer: buffer }); client.broadcast({ announcement: client.sessionId + ' connected' }); client.on('message', function(message){ var msg = { message: [client.sessionId, message] }; buffer.push(msg); if (buffer.length > 15) buffer.shift(); client.broadcast(msg); }); client.on('disconnect', function(){ client.broadcast({ announcement: client.sessionId + ' disconnected' }); }); }); And the best part is, under the hood, it'll handle your WebSocket, WebSocket over Flash, long polling, multipart and iframe connections. Sweet, isn't it?

socket.io and Express. Tying it all together.

Express is a great web development framework for node.js. It provides easy access to stuff like routing, requests and sessions. socket.io is an abstraction layer for Websockets, with Flash and XHR fallbacks, that runs in both node.js and the

6
client.

The Basics
You can have socket.io run with Express easily. By simply invoking socket.ios listen method and passing it the Express app as a parameter:
var io = require('socket.io'), express = require('express'), app = express.createServer(); app.configure(function () { app.use(express.cookieParser()); app.use(express.session({secret: 'secret', key: 'express.sid'})); app.use(function (req, res) { res.end('<h2>Hello, your session id is ' + req.sessionID + '</h2>'); }); }); app.listen(); var sio = io.listen(app); sio.sockets.on('connection', function (socket) { console.log('A socket connected!'); });

However, socket.io will not be aware of Express and the other way around. So if a socket connects, we do not know to which Express session it belongs, but in most scenarios this is an essential information. Since socket.io 0.7 however we can easily obtain this information through the handshake/authorization mechanism. Through this we can tell socket.io to invoke a userdefined function whenever a new Websocket connection is incoming. The important point is, that it is called, before the connection is completed. This provides multiple benefits. For one, we can accept or revoke the connection based on various conditions and for the other part it allows us to inspect the header information of the HTTP request that is trying to establish the Websocket connection including the cookie. The cookie will contain the sessionID of our Express session.
var parseCookie = require('connect').utils.parseCookie; sio.set('authorization', function (data, accept) { // check if there's a cookie header if (data.headers.cookie) { // if there is, parse the cookie data.cookie = parseCookie(data.headers.cookie); // note that you will need to use the same key to grad the // session id, as you specified in the Express setup. data.sessionID = data.cookie['express.sid']; } else { // if there isn't, turn down the connection with a message // and leave the function. return accept('No cookie transmitted.', false); } // accept the incoming connection accept(null, true); });

All the attributes, that are assigned to the data object are now accessible through the handshake attribute of the socket.io connection object.
sio.sockets.on('connection', function (socket) { console.log('A socket with sessionID ' + socket.handshake.sessionID + ' connected!'); });

Getting, serious.
But what I find much more interesting is that we can not only extract the sessionID from the cookie, but we can also load the actual session and use, modify or even destroy it. For that we need to get hold of the session store that Express uses to save our sessions. By default this is a MemoryStore and will be created by Express. Instead we can create our own and pass it to Express. That way we have a reference to that store for ourself.
var io = require('socket.io'), express = require('express'), MemoryStore = express.session.MemoryStore, app = express.createServer(), sessionStore = new MemoryStore(); app.configure(function () { app.use(express.cookieParser()); app.use(express.session({store: sessionStore , secret: 'secret' , key: 'express.sid'})); app.use(function (req, res) { res.end('<h2>Hello, your session id is ' + req.sessionID + '</h2>'); }); });

Now, in our handshake function, we can not only get the sessionID from the cookie, but actually load the session data from the session store.
sio.set('authorization', function (data, accept) { if (data.headers.cookie) { data.cookie = parseCookie(data.headers.cookie); data.sessionID = data.cookie['express.sid']; // (literally) get the session data from the session store sessionStore.get(data.sessionID, function (err, session) { if (err || !session) { // if we cannot grab a session, turn down the connection accept('Error', false); } else { // save the session data and accept the connection data.session = session; accept(null, true); } }); } else { return accept('No cookie transmitted.', false); } });

All the good stuff

Now we have access to all of the sessions data through socket.handshake.session. But to be able to change the session data we will need to create an actual Express Session object. The constructor for the Express session object requires the request associated with the session aswell as the session data. We just acquired the session data from the session store, so we have that. But we do not have access to the HTTP request that is associated to the session. socket.io does not expose it to us. If you look closer at the Session prototype though you see, that there are only 3 properties, that are required on the request object passed to the constructor: sessionID, session and sessionStore. This is good news, since our handshake data object already has the properties sessionID and session. Both (after we created the Session object) with the values that the Session prototype expects to be there. Well and the last property, sessionStore, we can easily add to the data object.

8
var Session = require('connect').middleware.session.Session; sio.set('authorization', function (data, accept) { if (data.headers.cookie) { data.cookie = parseCookie(data.headers.cookie); data.sessionID = data.cookie['express.sid']; // save the session store to the data object // (as required by the Session constructor) data.sessionStore = sessionStore; sessionStore.get(data.sessionID, function (err, session) { if (err || !session) { accept('Error', false); } else { // create a session object, passing data as request and our // just acquired session data data.session = new Session(data, session); accept(null, true); } }); } else { return accept('No cookie transmitted.', false); } });

Now we can use the session object to modify and even destroy the session. Eg. you could use Sessions touch() method to keep the session from timing out for as long as the Websocket connection remains
sio.sockets.on('connection', function (socket) { var hs = socket.handshake; console.log('A socket with sessionID ' + hs.sessionID + ' connected!'); // setup an inteval that will keep our session fresh var intervalID = setInterval(function () { // reload the session (just in case something changed, // we don't want to override anything, but the age) // reloading will also ensure we keep an up2date copy // of the session with our connection. hs.session.reload( function () { // "touch" it (resetting maxAge and lastAccess) // and save it back again. hs.session.touch().save(); }); }, 60 * 1000); socket.on('disconnect', function () { console.log('A socket with sessionID ' + hs.sessionID + ' disconnected!'); // clear the socket interval to stop refreshing the session clearInterval(intervalID); }); });

Sum it all up
From this code base you can keep going. You have everything set up to make your Express/socket.io hybrid application run and interact with each of its components as you wish. If you have any comments or suggestions, please feel free to use the comment box below.