Software Engineering

Building a Serverless Mesh Processing Microservice in Node.js


In this post we’ll create a mesh processing serverless microservice using Node.js and AWS. While in traditional architectures applications run in long-running servers, in serverless architectures applications run in event-triggered stateless containers. For more information about serverless architectures in general, you can check out http://martinfowler.com/articles/serverless.html.

For this example, we’ll be using AWS Lambda, which is one of the most popular Function-as-a-Service alternatives out there. I think the best way to describe AWS Lambda is that it does for distributed computing what DynamoDB did for high-performance databases and what EMR did for Hadoop. Lambda reduces the level of expertise required to create large-scale distributed applications in the cloud. By using Lambda we don’t need to worry about servers, provisioning or scaling. All of these are automatically taken care of for us.

Model Service

For this example we would like to create a simple service that will allow us to manage 3D models. The service will have the following functionality:

  • List Models: return a list of all available models
  • Add Model: add a new model
  • Simplify Models: simplify (decimate) any of the available models
Screenshot of the demo

Screenshot of the demo

View demo Download source

We’ll use S3 for storing the actual model files and DynamoDB for persisting the list of available models. Now let’s see what alternatives we have for our server-side logic.

First attempt: traditional architecture

The most basic setup would be to have a web server that exposes a REST API running on an EC2 instance. The server would communicate directly with DynamoDB. The Simplify Operation will receive the name of the model to process, download the corresponding mesh file from S3, perform the simplification and upload the new mesh to S3. For this reason, the server will need to communicate directly with S3. The client browser will also fetch the model files from S3. It will look something like this:

attempt1a

Simple Traditional Setup

If we’re talking about a production-level real application, having our server running in one EC2 instance won’t cut it. We’ll want some sort of redundancy in case our server fails. We’ll also probably want some sort of auto-scaling depending on the traffic load and more servers, which will make our setup look something like this:

Adding a load balancer

Adding a load balancer

Having multiple servers running behind a load balancer does indeed address our concerns above. However, this setup has a few disadvantages. The first is that we have EC2 instances running at all times, even when there is no traffic. The second is that we need to program and maintain the web server. Remember that the less code we’re responsible for, the better. Another disadvantage is that we’ll needs DevOps to setup our servers, load balancer, auto-scaling, probably a VPC, etc… Wouldn’t it be nice if we didn’t need to pay anything when our servers were idle, didn’t need to worry about web servers at all and kept DevOps involvement to a minimum? So, let’s try to get rid of our servers.

Second attempt: using AWS Lambda

One of the simplest ways to get rid of our web servers is by having the browser client communicate directly with DynamoDB. This would let the client fetch the list of models and add new models without having to go through any intermediaries. The mesh simplification operation cannot run locally in the browser, so we’ll create a lambda function for it. Then the browser client can just call the lambda function directly. This would look something like this:

Using AWS Lambda

Using AWS Lambda

So we managed to get rid of our web servers. The question is: have things improved? On the one hand, we don’t need worry about redundancy, auto-scaling and traffic loads anymore. S3, Lambda and DynamoDB do all that for us. On the other hand, now our browser client knows too much about the internals of our service. Any client that wants to use our service needs to know about our DynamoDB table, its schema, about the fact that the mesh processing is performed by a Lambda function and so on.

This means that we won’t be able to make future changes to the database schema without affecting the clients that use it if the service communicates directly with DynamoDB. All this coupling is a big disadvantage, especially compared to our previous approach in which the client only communicated with a REST API. What we can do instead is combine these two approaches and hide our lambda functions and DynamoDB database behind a REST API. This would gives us the benefit of a serverless backend while keeping our clients loosely-coupled.

Microservice using AWS Lambda and API Gateway

The first thing will do is create two extra lambda functions: one for listing the models and one for adding models. Here is what our Node.js code looks like:

...
const docClient = new AWS.DynamoDB.DocumentClient();
const params = {
    TableName: 'hecodes-models'
};

exports.insert = function (event, context, callback) {
    params.Item = {
        "name": event.name,
        "url": event.url
    };

    docClient.put(params, function(err, data) {
        if (err) {
            callback(JSON.stringify(err));
        } else {
            callback(null, JSON.stringify(data));
        }
    });
};

exports.simplify = function (event, context, callback) {
    simplifyModel(event.name, event.perc, callback);
};

exports.list = function (event, context, callback) {
    docClient.scan(params, (err, data) => {
        if (err) {
            callback(JSON.stringify(err));
        } else {
            callback(null, JSON.stringify(data.Items));
        }
    });
};

Then we’ll go to AWS API Gateway and create a new API with three resources:

  • /model POST
  • /model/list GET
  • /model/simplify GET

One of the great things about API Gateway is that it’s very easy to map API endpoints to Lambda functions. This way we can configure each of our API resources to execute the corresponding lambda function.

Mapping POST request to lambda insert function

Mapping POST request to lambda insert function

Now we can just have our browser client call our brand new serverless API!

Final design using AWS Lambda and API Gateway

Final design using AWS Lambda and API Gateway

We’ve created a self-contained and serverless microservice for managing 3D models. Clients can interact with the service using a standard REST API. The service manages its own persistence layer and can be deployed independently. If you haven’t tried using Functions-as-a-Service yet I encourage you to do so. Both Azure and Google Cloud offer their own alternatives to AWS Lambda, so you have no excuses!

Edit: Before you start using AWS Lambda, make sure you go through their resource limits page. Like most managed services, it has some constraints that you’ll need to take into account if you decide to use the service in your project.

Programming Patterns
Efficient WebVR Development Using the Adapter Pattern in Three.js
System Architecture
Add git-like functionality to your application using Event Sourcing
Programming Patterns
Building Real-Time Collaboration Applications in Three.js