From outside the black box

As a software architect specializing in distributed systems, I build black boxes, and lots of them. The users of the systems I help to build don’t need to know how they work, they just care about reliability, speed, and ease of use. Fortunately, one of the biggest advantages of distributed systems is that they are by nature more fault tolerant, easier to scale, and easy to abstract.

The Microservice Swarm

It’s becoming more common for systems I design and build to be made up of lots of smaller systems that all work together to act as a whole—microservices if you’re into buzzwords. This is great for compartmentalizing code, separating concerns, and being able to scale the parts of your system that need it most. And in this world of black boxes that talk to other black boxes, testing is indispensable and, fortunately, easier to handle.

Gone are the days of navigating through a monolithic codebase, reproducing conditions through various stages of your application state. Instead, each microservice acts as its own API, has its own contract on what it expects as input and what it will generate as output. This means that for every system a given service communicates with, you can generate test data and assert output data without having a hard dependency on the external service. Testing for expected and—often more importantly—unexpected input and output becomes very easy to handle in these models. And after a “contract” is established for one service, a dependent service can be built almost entirely by relying on tests to confirm its behavior.

Distribution for Humans and Machines

Putting these ideas into practical use allowed us to develop a new content management and distribution system for NBC. This system is capable of consuming content created by humans and machines, merging that content together in a sane way, and presenting the results to users. The techniques we developed allowed us to build individual components of the system before all the pieces were in place and ready to consume. By establishing contracts between our services, we didn’t need to know all the internals or conditions that would lead to a state at any given place in our code. We could test everything independently and confirm that the service under test would perform as expected. Essentially, we were able to build a black box from the outside.

… And it worked. Additional debugging was necessary after all the pieces were put together, but the changes were very minimal and almost always the result of an external system (that we didn’t control) not honoring the “contract” originally presented to us for the expected I/O.

The Tools

For us, Node.js was the enabler for this kind of development. Compartmentalization and separation of concerns are baked into its DNA, so scaling that up to the systems we build was a natural next step.

Some of the tools we used to help us reach success were:

• Sinon.js for mocking and testing dependencies – including forcing fail states.
• nock for mocking HTTP requests and responses
• nodeunit for our test suites – I realize this is ancient by JavaScript community standards, but it gets the job done and works well for testing our server side code. A move to ava has been discussed for future development.
• JSON Schema for establishing and documenting our API contracts.
• JSON API for defining how our client facing APIs behave.

It’s no secret that microservices are in right now, but when combined with rigorous testing (all of our components have 100% line and branch coverage), developers are empowered to build exceptionally powerful systems in a much more agile way. Let me know in the comments if you’re using microservices and how it’s working out for you!