How We Keep Your Blog from Crashing: Load Balancing and Caching

Clayton Stobbs, Compendium's Director of Account Management, recently blogged about how we were able to handle a large influx of inbound traffic to our client's blogs:

As a technical decision maker evaluating Compendium for your company's business blogging platform, you may be interested in what technologies we employ to handle influxes of traffic.  This post will try to pull back the curtain to give you some behind-the-scenes insight into what we do.

Load Balancing

The first thing to keep in mind is that generating a web page with dynamic content is expensive from a time standpoint.  As many as a hundred database queries may be executed in the process of retrieving the content that will be displayed on the page.

If the system is built on a web framework using the model/view/controller design pattern (e.g. Rails, Struts, Zend, or ASP.NET MVC), there is a lot of overhead just in bootstrapping the machinery that will validate and respond to a web page request.  Without resorting to very high end server hardware, the best that a single server could handle is on the order of single digit requests per second.

Since the volume that can be processed by an individual server is pretty low, we scale out horizontally, setting up a cluster of identical application servers that are responsible for processing the requests.  The servers themselves have elevated CPU power, but do not have large amounts of memory.

Incoming requests pass through a load balancing proxy server, which then routes each request in rotating order as shown in the figure below.

The request processing servers just crunch on data.  They do not store any application data themselves.  Instead, each server issues queries against a central database server. 

This kind of separation allows us to handle increasing loads by simply spinning up additional request processing servers and putting them into rotation with the proxy server.  Any page can be generated by any of the processing servers because they all rely on the same database.

The database server itself has both lots of memory and lots of processing power.  Moreover, it uses high performance disk configurations to that ensure fast access and fault tolerance.

The same is true for session data, which stores things like whether a user is authenticated and what roles that user may have.  A separate server, having very large memory capacity, is responsible for storing this data.  Again, since only one session data server is present, the set of HTTP requests needed to display a given page can be distributed across multiple servers since each server has access to the same authentication data.

We also provide a layer of redundancy, with identical sets of request processing and storage servers operating in separate data centers.  Each center has its own copy of the database and changes are kept in sync via master/master replication.  Incoming traffic is further load balanced between these data centers using cookies that enforce stickiness to the data center for a given visitor.

Caching

Sharing request workloads across multiple servers will get you far, but it doesn't address the problems that can arise as the amount of work needed to fetch a set of data increases.  This can take the form of slow database queries that push page render times above the important 1 second level.

This is where caching comes into play.  The use of a caching system allows the application to specify that a chunk of data be stashed away for future usage, usually with a future expiration date so that room can be made for other chunks of data later on.

Caching is practiced at many levels throughout our application. 

First, database queries that usually take larger amounts of time and whose results don't very much over a specific window, are cached for anywhere from a few minutes to several hours.

Second, blocks of data which are obtained from calls to third-party web services are cached for longer periods of time to reduce lag in processing requests and to steer clear of usage rate limits. 

Finally, we cache fully rendered blog pages for a minute to ensure that if a page gets requested frequently over that time period, we only have to hit the back end application servers once during that time.

Technologies Used

What sorts of technologies do we rely to accomplish all of this?  Most of it is comes from the open source community and is in use at other well-known start-ups:

• Data center load balancing -- Amazon Web Services Elastic Load Balancing
• Request server proxy load balancing -- NGINX (pronounced "engine X")
• Request processing -- Agavi PHP Framework atop Apache HTTP Server, running on Amazon Elastic Compute Cloud (EC2)
• Relational database -- MySQL
• Session storage and query caching -- memcached
• Page caching -- Varnish

Any Questions?

This is your chance to ask all those geek questions you were scared to ask :)  I'm happy to explain any part of our system or technology in more detail, just leave a comment.  Thanks!