Since the beginning of the year I have been working on building a new REST based API to be consumed by both internal and external clients in Node.js.
The main reason for choosing node and mongo as the main elements in our stack when we started the project was to alleviate a number of pains we experienced with the “old” stack of ASP.NET and SQL. Most of this was related to the amount of ceremony that goes in to doing even simple stuff on that stack, which was being made worse by the way the technologies were used. Most if not all of these pains were indeed alleviated, but we gained a couple of new ones, which can be mitigated to some extend.
Debugging and Stack Trace
Having spent many hours the last couple of weeks debugging errors in an application with a codebase that is gaining in size I have come to the conclusion that simply put, the debugging and stack trace in node is inferior to what you would get in a .NET based environment.
I have found some of the error messages to be short non descriptive and very generic, making it often fairly hard to understand what is going on. Likely, it is a consequence of the lack of maturity of the technology combined with the lesser experience I have with it, but reading a stack trace in .NET would often mean I know exactly what the problem was, which is not the case with node.
Probably, the callback nature of most of the API’s also is somewhat to blame for this.
Most of these challenges can be minimized with a good and solid test suite, as it can help pinpoint more precisely where an error has occurred.
A nice little addition to this is that the dynamic nature of the language means simple spelling mistakes in parts of the program that is not executed by tests (bad!) will not get caught. This is an easy problem to fix though – apply JSHint, JSLint, or similar (do it from day one, you owe it to yourself).
I tried to find a good “bad” error message to put here, but could not easily provoke one intentionally. Will keep a look out for it and make a new post next time I find a good one.
Integration vs Unit Tests – Design and Architecture
The way our application is designed means it is mostly implemented using express middleware stack with a little bit of business logic sprinkled here and there. We quickly discovered that this was probably not the best architecture, but still have some leftovers and we have had some challenges to write code that can be unit tested in a meaningful way (without a ton of mocks and fake objects being build).
This is mainly a design challenge and not a consequence of node, nor javascript, but it has lead to a side effect. Because the “units” are not easily testable, a large percentage of the codebase is mainly tested with integration style tests using “restler” (I would actually recommend using “request” instead). These tests are as good or even better than unit tests to find errors, but (un)fortunately the error data returned from our API does not give me enough information to pinpoint what goes wrong.
Thus, the conclusion I have come to is that the middleware should be extremely thin and always defer the bulk of the implementation to another module that can tested. This goes both for middleware implementing cross-cutting concerns and the actual end-point implementation.
The platform does not have a large body of known design patterns for implementing maintainable applications. This easily leads to developers trying out a lot of different approaches with different merits and flaws, which means the codebase could suffer from inconsistencies making it hard to maintain.
Hosting
Figure out very early how you are going to host this. If you are going for a cloud based hosting vendor and you are based in the States, there is a number of options.
The same options are available in Europe, but from the States, leading to a latency challenge. Trying to get a solid agreement with a hosting partner in Europe has been hard and we have been forced to settle for a worse agreement that we would like.
The good part is that due to the traction of the platform almost everyday a new option pops up, and we can always fallback to running off amazon or azure and managing the setup ourselves.
Picking the right Stack
Node.js has been gaining a lot of traction and combined with some of the prime modules it is in general a very nice stack for building the kind of API we are building. I would pick it again if I had to make the choice again today, but this time with the knowledge of what is brings.
It has taken longer, and been harder than expected (was almost expected), but we are getting close to being done and can reflect upon what worked and what did not work.
- Get JSHint or JSLint up and running (as early as possible).
- Get CI up and running (as early as possible) – not after 4 months when you are trying to finish up the project.
- The design and architecture is really important – do not let the speed with which you can whip up new features keep you from worrying about this. Design for tests that can tell you what is wrong with the application when changes are being made.
- Decide as early as possible on standards (to enforce the design and architecture decisions), and make sure the application follows these before moving on. I would recommend starting out with the revealing module pattern, hoisted functions for callback nesting, avoiding control flow libraries such as async, avoiding inheritance, considering every used external module very carefully, and most importantly keeping it dead simple.
- Start out with testing hosting setups from day one:
- Deploy your app to different platform as service vendors, such as:
- Deploy your app to amazon or similar – if nothing else to get an idea of how the server actually work and what is required.
- Use these experiments to figure out how you want to run your application in production. Figure out which are “ real” requirements and which are nice to have.
- Consider before you start whether to use CoffeeScript or javascript to write the application. It might seem like a straight forward choice to you, but at least do yourself the favor of spending just a little bit of time researching.
Happy coding.