Using iptables to balance Node.js processes

One of the challenges in building a web application on any platform is making sure it can handle enough visitors. That’s a fairly well known and understood challenge for apps hosted on Apache, which is where most of our experience has been. It’s a new ballgame when we’re talking about Node.js based apps.

The NodePing web application is all Node.js on the server, with lots of jQuery driven Ajax on the client. We actually have two different request patterns within the web app. The application piece is used by customers managing their monitoring checks and accounts. That part is a single page app (SPA). The “static” content pages are simpler. From a request and response point of view they are a more traditional web page, in part so that they’ll be easily crawlable. The components that actually run checks and process results for the NodePing services are a whole different thing, which we’ll write a post about later. This post is just about the web application.

Early on we started looking for information about how fast we should be able to go with Node.js. Good information was hard to find. Most of the benchmark type articles on the net are very simple test cases. They show Node.js doing very well on requests per second, but these typically are just responding with an almost empty response. Of course, they were comparing it to other servers handling similarly simple requests, so those results are fine for what they are trying to do but not really applicable. What happens when you start throwing in real world queries and processing that we’ll see in our web application?

The real answer is we don’t know, and even if the published benchmarks included more data with more processing to handle the requests we still wouldn’t know because they aren’t running our code on our setup. We need to get Slashdotted to find out for sure, and/or get large numbers of customers so we have thousands of real requests to the single page web app. Both of those would be interesting days. We have run a bunch of tests with ab and siege. I’m not going to report numbers, because they won’t be much more useful than the benchmarks I found. The fact is you have to build something and see how it works in your particular case. Feel free to help us get lots of customers, and we’ll report more on how we were able to handle the load in Node.js. It has to be real customers. We already know how we do under simulated load.

What we found in our early testing is that we were running out of performance in our app well before we wanted to. On most servers with at least moderate amounts of memory this was a matter of not enough processing power. We’d hit the host with an ab or siege test, and the processor would peg very shortly.

After looking at various options (mostly making sure we weren’t wasting processing in the code), we concluded we just needed to throw more processing at the app. Node.js typically runs in a single process. We needed to be able to utilize more cores. With Node.js, the most obvious way to do that is to start multiple processes and then balance the requests between the processes.

In our case we’re dealing with a web application, and the logic it needs to run isn’t very intensive. It is mostly serving up data. Each individual request doesn’t need access to more processing. So we don’t need interprocess communications, we just need to be able to run more of them. Also, most page content is in Redis or databases, which are shared, so we don’t even care if requests within a session hit the same process.

The first thing we looked at was various proxy front ends. There are several that might work. One of my favorites is node-http-proxy from nodejitsu.

In the end, we decided the simplest and fastest approach was using iptables to split the requests between multiple processes. We were already forwarding traffic to the server’s port so that we could run the service on a high port and easily run it as a non-root user. I needed to get this going quickly, so I just copied my main index.js file (which basically just starts the server and loads modules) to several files, with the port for each process in each file. It would have been fairly trivial to do this dynamically, or accept it as a command line parameter, but this was quick and it would work well for scripted starts. I ended up with files called something like index8001.js, index8002.js, and so on, with one file for each process I wanted to run.

All that’s left is the iptables bit. We are going to redirect the traffic using iptables statistics module. This could be done randomly, which should end up passing requests fairly evenly. Or it could be done using the nth mode, which forwards the nth request it sees that matches the rule. I opted for the nth mode approach.

Figuring out how to do this was a little tricky, because the statistics module has evolved and there is a lot of old information out there, plus some that is just wrong.

The rules we needed looked something like this:
-A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8001 -m statistic --mode nth --every 3
-A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8002 -m statistic --mode nth --every 2
-A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8003
-A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8101 -m statistic --mode nth --every 3
-A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8102 -m statistic --mode nth --every 2
-A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8103

Most of this is just normal forwarding. In this example we are forwarding traffic to port 80 to ports 8001, 8002, and 8003. Traffic to port 443 we forward to ports 8101, 8102, and 8103. We want a third of the traffic to each destination port to go to each process. Our actual number of ports will vary based on how many processes we want to use on a specific host, which depends in part on how many cores we want to use.

These are terminating targets. That is, once we’ve forwarded a given request we’re done and ready for the next one. We don’t ever forward to port 8001 and also port 8002. So the first rule we want every third request. The other two requests pass to the next line. Here, we want to forward every other request, since 1/3 are already being handled by the previous line. The third line for this destination port doesn’t need the statistics module at all, since it only ever sees every third request and should always forward all the traffic for port 80 that reaches this rule.

Some examples on the Internet list similar set ups with ‘–every 3’ on each of the three lines for port 80. That dumps a few of the requests out on the floor of the data center. The first line picks off one third of the requests, leaving 2/3 of the remaining requests to pass to the next line. If it is set to every 3, it picks off a third of those 2/3. The last line would then pick off a third of what’s left from that. That leaves something like 30% of our requests unhandled. That is bad.

This is not a high availability solution, it is just to spread load between processes. We’re using Forever to run the individual processes. That works fairly well, and we don’t really need to be concerned about fail over between processes on the single server. Load balancing between servers needs to preserve sessions, and is a different scenario from what I’ve described here. Watching the traffic come into this setup spreads the requests across all of the processes, effectively using all of our cores. On two processes we approximately double the number of requests we can handle per second. Four processes can handle roughly four times the number of requests. That is good.