The last 3 months or so I have been busy working on a new backend built in node with mongo as the database. It has been lots of fun, with some frustration, and lots of heureka moments.

This is my attempt at describing those experiences, some good and some bad.

The two many reasons we opted to work with node was the following:

• Development speed, due to the simplicity of node and using the same language in both client and server.
• Performance, most of the information we found indicated that we could easier get more performance from using node than the alternative of ASP.NET.

Development Speed

It seems like the development speed has indeed been increased, but in reality we cannot really measure before we build the next module on top of node. Our first module has also been the proving grounds for the technology stack, as the maturity of the technology means there is not so much information to find on how to write the best node code. Thus, the progress has most likely been slower than what we can expect in the future (yet I still believe I have seen an improvement compared to old projects).

Same Language

So node being javascript, it is in fact the same language is we use in our browser client (built on top of backbone). But, due to the fact that node does not work in the same way as the browser with regards to the code it is not completely trivial to share code between server and client.

A given file containing a function in node would have to use

1

module.exports = myFunction;

To expose it to being available for use elsewhere. In addition this file would have to be required for the code to actually be used elsewhere. One could argue this is much better than the browser where the same function would probably be attached to some global object (pretending to be a namespace).

To give an example of how one could implement code usable for both browser and node, I have taken the liberty of showing a snippet from the brilliant underscore library doing exactly this:

1
2
3
4
5
6
7
8

if (typeof exports !== 'undefined') {
  if (typeof module !== 'undefined' && module.exports) {
      exports = module.exports = _;
   }
   exports._ = _;
} else
  root['_'] = _;
}

Cross OS Platform

Having a cross OS platform gives us more freedom in which development machines to use and servers to run in production, which also gives us more freedom regarding hosting partners, whether in the cloud or not. Actually, all the developers are working on windows, as we have found the Webstorm IDE to be the best tool by far, and it does not work well on linux (when that linux is a virtual box at least.

Running development on windows and production on linux has not been without problems though. The node core is equally good on both OS, but some of the modules we have used have not been properly tested on windows. Additionally, the windows file system is case insensitive, while it is case sensitive on linux, which has led to one error deployment so far.

Pyramid of Doom

Much have been written about the problems the callback nature of most node api’s lead to. Our solution has been to mainly use function hoisting:

Nested:

1
2
3
4

getSomeData(err, function(err, callback){
  doSomeStuff(err, function() {
  });
});

Hoisted:

1
2
3
4
5
6
7

getSomeData(err, dataRetrieved);

function dataRetrieved(err, callback) {
  doSomeStuff(err, someStuffDone);
}

function someStuffDone() {}

This is working ok for scenarios where we have a fixed depth of nesting of functions (the example with 2 above is not a bad pyramid of doom by the way – try throwing in a couple of more levels of nesting and you will see the problem). Unfortunately, this method does not work with a dynamic level of nesting (think an array of functions generated and to be called in sequence), and this is where a library such as Async can help.

Solving the pyramid of doom this way has not been without challenges, as there have been arguments about using Async or similar library even with a fixed amount of nesting. My standing on this is that code written with Async in that scenario (fixed level of nesting) is much harder to read, understand, and reason about.

Express and middleware

Like most developers doing services in node exposed on the web, we have chosen the express web framework. There are a lot of features built into express, and to be honest we only use a fraction of them. Mainly the middleware stack (from the connect sub module) has been used for good, but certainly also for bad.

Cross cutting concerns like logging, authorization, and gzipping are all examples where benefit can be reaped from the stack and it is easy to apply a certain behaviour to all requests and responses. Unfortunately, the ease of implementation has lead to data access and business logic being implemented as middleware. This went well for a while, but it is harder to reuse and test the middleware as they are dependent on the (request, reponse, next) – triple of arguments, as opposed to business relevant objects.

Thus, we are moving towards not using middleware for data access and business logic, having one function, registered as middleware, for each end-point which then delegates to business and data oriented modules.

Require (Cyclic)

As we move along and introduce more files and more modules, I have noticed a tendency to let all dependencies be resolved with require. This leads to some scenarios where we have fairly low level functionality (such as configuration api) require high level functionality (the user module). This easily leads to cyclic require loops, causing bugs that can be hard to resolve. Like any software it is good practise also in node to let high level functionality depend on low level functionality – not the other way.

The solution to reusing bits is, as with many frameworks, to divide and conquer. If a high level module contains a function that would be really usable in a low level module, extract it in a seperate module and break the cycle that way.

Return Callback

Just an observation on the nature of some of the node code I have been writing recently:

1
2
3
4
5
6

function doSomeStuff(err, callback) {
    if (err) {
        callback(err, 'bad stuff happened');
    }
    callback(null, 'wuhuuuu bad stuff did not happen');
}

Did you find the bug? This is the most simple example I could come up with, but when it gets a bit more complex I find myself forgetting the return statement in front of the first callback, leading to some funky debugging session:

1
2
3
4
5
6

function doSomeStuff(err, callback) {
  if (err) {
      return callback(err, 'bad stuff happened');
  }
  callback(null, 'wuhuuuu bad stuff did not happen');
}

Maybe it is just a quirk of how many brain process the code, but consider yourself warned! ;-D

Simplicity

This is the one point where I believe node really shows its strength, it is dead simple to implement a server. Recently, we had a potential candidate who had written the server part of his test in less than 100 lines of coffee-script, and I really think this says it all.

Performance

I have written a lot about my experiences with developing on node, and very little regarding the awesome earth-shattering speed of node.

Some claim that the way node works (event-loop and such) is the coolest thing since the invention of the wheel, while others claim that it is borderline stupid. I am not going to join that discussion, but just tell that what I have seen so far and the experiments done show that node is indeed fast, and also faster than what we did before. That being said our team does not have good large scale measurements or experiments of our own to prove this.

Is node the best choice available for us? Maybe, it could be, but at least until now it has proven to be a very good choice.