Performance On A Shoe String

Back in the summer, I happened to notice the official Wimbledon All England Tennis Club site had jumped to the top of Alexa’s Movers & Shakers list — a list that tracks sites that have had the biggest upturn or downturn in traffic. The lawn tennis championships were underway, and so traffic had leapt from almost nothing to crazy-busy in a no time at all.

Many sites have similar peaks in traffic, especially when they’re based around scheduled events. No one cares about the site for most of the year, and then all of a sudden – wham! – things start getting warm in the data centre. Whilst the thought of chestnuts roasting on an open server has a certain appeal, it’s less attractive if you care about your site being available to visitors. Take a look at this Alexa traffic graph showing traffic patterns for superbowl.com at the beginning of each year, and wimbledon.org in the month of July.

Traffic graph from Alexa.com

Whilst not on the same scale or with such dramatic peaks, we have a similar pattern of traffic here at 24ways.org. Over the last three years we’ve seen a dramatic pick up in traffic over the month of December (as would be expected) and then a much lower, although steady load throughout the year. What we do have, however, is the luxury of knowing when the peaks will be. For a normal site, be that a blog, small scale web app, or even a small corporate site, you often just cannot predict when you might get slashdotted, end up on the front page of Digg or linked to from a similarly high-profile site. You just don’t know when the peaks will be.

If you’re a big commercial enterprise like the Super Bowl, scaling up for that traffic is simply a cost of doing business. But for most of us, we can’t afford to have massive capacity sat there unused for 90% of the year. What you have to do instead is work out how to deal with as much traffic as possible with the modest resources you have.

In this article I’m going to talk about some of the things we’ve learned about keeping 24 ways running throughout December, whilst not spending a fortune on hosting we don’t need for 11 months of each year. We’ve not always got it right, but we’ve learned a lot along the way.

The Problem

To know how to deal with high traffic, you need to have a basic idea of what happens when a request comes into a web server. 24 ways is hosted on a single small virtual dedicated server with a great little hosting company in the UK. We run Apache with PHP and MySQL all on that one server. When a request comes in a new Apache process is started to deal with the request (or assigned if there’s one available not doing anything). Each process takes a bunch of memory, so there’s a finite number of processes that you can run, and therefore a finite number of pages you can serve at once before your server runs out of memory.

With our budget based on whatever is left over after beer, we need to get best performance we can out of the resources available. As the goal is to serve as many pages as quickly as possible, there are several approaches we can take:

1. Reducing the amount of memory needed by each Apache process
2. Reducing the amount of time each process is needed
3. Reducing the number of requests made to the server

Yahoo! have published some information on what they call Exceptional Performance, which is well worth reading, and compliments many of my examples here. The Yahoo! guidelines very much look at things from a user perspective, which is always important.

Server tweaking

If you’re in the position of being able to change your server configuration (our set-up gives us root access to what is effectively a virtual machine) there are some basic steps you can take to maximise the available memory and reduce the memory footprint. Without getting too boring and technical (whole books have been written on this) there are a couple of things to watch out for.

Firstly, check what processes you have running that you might not need. Every megabyte of memory that you free up might equate to several thousand extra requests being served each day, so take a look at top and see what’s using up your resources. Quite often a machine configured as a web server will have some kind of mail server running by default. If your site doesn’t use mail (ours doesn’t) make sure it’s shut down and not using resources.

Secondly, have a look at your Apache configuration and particularly what modules are loaded. The method for doing this varies between versions of Apache, but again, every module loaded increases the amount of memory that each Apache process requires and therefore limits the number of simultaneous requests you can deal with.

The final thing to check is that Apache isn’t configured to start more servers than you have memory for. This is usually done by setting the MaxClients directive. When that limit is reached, your site is going to stop responding to further requests. However, if all else goes well that threshold won’t be reached, and if it does it will at least stop the weight of the traffic taking the entire server down to a point where you can’t even log in to sort it out.

Those are the main tidbits I’ve found useful for this site, although it’s worth repeating that entire books have been written on this subject alone.

Caching

Although the site is generated with PHP and MySQL, the majority of pages served don’t come from the database. The process of compiling a page on-the-fly involves quite a few trips to the database for content, templates, configuration settings and so on, and so can be slow and require a lot of CPU. Unless a new article or comment is published, the site doesn’t actually change between requests and so it makes sense to generate each page once, save it to a file and then just serve all following requests from that file.

We use QuickCache (or rather a plugin based on it) for this. The plugin integrates with our publishing system (Textpattern) to make sure the cache is cleared when something on the site changes. A similar plugin called WP-Cache is available for WordPress, but of course this could be done any number of ways, and with any back-end technology.

The important principal here is to reduce the time it takes to serve a page by compiling the page once and serving that cached result to subsequent visitors. Keep away from your database if you can.

Outsource your feeds

We get around 36,000 requests for our feed each day. That really only works out at about 7,000 subscribers averaging five-and-a-bit requests a day, but it’s still 36,000 requests we could easily do without. Each request uses resources and particularly during December, all those requests can add up.

The simple solution here was to switch our feed over to using FeedBurner. We publish the address of the FeedBurner version of our feed here, so those 36,000 requests a day hit FeedBurner’s servers rather than ours. In addition, we get pretty graphs showing how the subscriber-base is building.

Off-load big files

Larger files like images or downloads pose a problem not in bandwidth, but in the time it takes them to transfer. A typical page request is very quick, a few seconds at the most, resulting in the connection being freed up promptly. Anything that keeps a connection open for a long time is going to start killing performance very quickly.

This year, we started serving most of the images for articles from a subdomain – media.24ways.org. Rather than pointing to our own server, this subdomain points to an Amazon S3 account where the files are held. It’s easy to pigeon-hole S3 as merely an online backup solution, and whilst not a fully fledged CDN, S3 works very nicely for serving larger media files. The roughly 20GB of files served this month have cost around $5 in Amazon S3 charges. That’s so affordable it may not be worth even taking the files back off S3 once December has passed.

I found this article on Scalable Media Hosting with Amazon S3 to be really useful in getting started. I upload the files via a Firefox plugin (mentioned in the article) and then edit the ACL to allow public access to the files. The way S3 enables you to point DNS directly at it means that you’re not tied to always using the service, and that it can be transparent to your users.

If your site uses photographs, consider uploading them to a service like Flickr and serving them directly from there. Many photo sharing sites are happy for you to link to images in this way, but do check the acceptable use policies in case you need to provide a credit or link back.

Off-load small files

You’ll have noticed the pattern by now – get rid of as much traffic as possible. When an article has a lot of comments and each of those comments has an avatar along with it, a great many requests are needed to fetch each of those images. In 2006 we started using Gravatar for avatars, but their servers were slow and were holding up page loads. To get around this we started caching the images on our server, but along with that came the burden of furnishing all the image requests.

Earlier this year Gravatar changed hands and is now run by the same team behind WordPress.com. Those guys clearly know what they’re doing when it comes to high performance, so this year we went back to serving avatars directly from them.

If your site uses avatars, it really makes sense to use a service like Gravatar where your users probably already have an account, and where the image requests are going to be dealt with for you.

Know what you’re paying for

The server account we use for 24 ways was opened in November 2005. When we first hit the front page of Digg in December of that year, we upgraded the server with a bit more memory, but other than that we were still running on that 2005 spec for two years. Of course, the world of technology has moved on in those years, prices have dropped and specs have improved. For the same amount we were paying for that 2005 spec server, we could have an account with twice the CPU, memory and disk space.

So in November of this year I took out a new account and transferred the site from the old server to the new. In that single step we were prepared for dealing with twice the amount of traffic, and because of a special offer at the time I didn’t even have to pay the setup cost on the new server. So it really pays to know what you’re paying for and keep an eye out of ways you can make improvements without needing to spend more money.

Further steps

There’s nearly always more that can be done. For example, there are some media files (particularly for older articles) that are not on S3. We also serve our CSS directly and it’s not minified or compressed. But by tackling the big problems first we’ve managed to reduce load on the server and at the same time make sure that the load being placed on the server can be dealt with in the most frugal way.

Over the last 24 days we’ve served up articles to more than 350,000 visitors without breaking a sweat. On a busy day, that’s been nearly 20,000 visitors in just 24 hours. While in the grand scheme of things that’s not a huge amount of traffic, it can be a lot if you’re not prepared for it. However, with a little planning for the peaks you can help ensure that when the traffic arrives you’re ready to capitalise on it.

Of course, people only visit 24 ways for the wealth of knowledge and experience that’s tied up in the articles here. Therefore I’d like to take the opportunity to thank all our authors this year who have given their time as a gift to the community, and to wish you all a very happy Christmas.