March 1, 2016

I’ve written a lot of node applications over the last few years. They’ve varied in their core frameworks, schemas, data stores, and purposes but one thing remains consistent: the use of multiple worker processes. The technique of forking child processes to do the heavy lifting is vital to scaling a node application because otherwise node runs in a single process. Starting up node across multiple processes is easy enough using the native cluster module but how to handle shutdowns is a little trickier.

Consider the (common) case where you have a web server that needs to cleanly close all open connections before shutting down. If you kill the process before closing connections you could have clients with dropped connections and could enter unexpected execution paths on API consumers. Thankfully the pattern for handling clean shutdowns in your application is simple, but there’s nothing unique about it, you’ll end up using nearly the same solution every time you implement it.

There are process managers out there that solve this problem quite well: forever and pm2 among others. These kind of solutions are more involved and required me to rebuild some of my applications in ways that I wasn’t comfortable with, so I decided to take another approach.

The solution I came up with is called adios. Put simply, adios will broker communication between the master and child processes so that when a SIGINT is received by the master (effectively CTRL+C) its child processes will be given an opportunity to cleanly shut down before the application fully stops.

When you start a project with adios you initialize the master adios worker in your master process. This will start a server which listens on a local domain socket for communication with child processes. The decision to use a domain socket was made to avoid collisions during inter-process-communication (IPC). The act of forking a child process in node does provide its own domain socket to communicate over – process.send() – but any message sent over this socket will be received by all listeners. By using its own domain socket adios will not send its messages over the default domain socket and any application that relies on IPC will not have to be aware of and possibly filter out adios events.

When the server has been set up, a promise is resolved and you can fork the children processes:

When adios is initialized in a child process a socket is opened to the master’s server to listen for shutdown events. You then define the actions that should be taken in that process before shutting down, for example, if you wanted to close all connections to an HTTP server before exiting the process you’d do something like this:

Adios achieves custom shutdown behavior by injecting a SIGINT handler on the master process. This means that rather than the normal shutdown behavior our callbacks are run first. When the master process receives a SIGINT it will broadcast on its socket to all the child workers that the application is shutting down and their clean shutdown code needs to be run. After a child resolves the promise in its clean shutdown the process will exit normally and the master will be notified. After all child processes have exited the master will exit normally.

This obviously provides a lot of flexibility in managing multi-process applications but sometimes a child process might have a shutdown task that gets hung up. In these cases, the master process will force the child to halt if the child hasn’t shut down after a given amount of time (ten seconds by default). This will prevent the application shutdown from completely stalling but is something you need to be aware of in the event that you have shutdown tasks that legitimately take a long time. If that’s the case for you the force shutdown timeout is configurable when you initialize the master.

The use of adios has greatly improved process management in our node deployments – it is now in use on all of NBC’s new node.js projects – and has given us a reliable way to cleanly shut down an application.
I hope this tool is useful and improves your node application process management as much as it has improved mine! The examples used in this post are available in a gist here. Let me know what you’re using adios for and what you think of it in the comments. If you see somewhere I can improve feel free to open an issue or pull request.

Posted in

Elliott Foster is a developer and standard nerd.

Comments