Using Google App Engine as Your Own Content Delivery Network

Do you remember, years ago, when hosting was expensive, domain names were the province of the rich, and you hosted your web pages on Geocities? It seems odd to me now that there was a time when each and every geek didn’t have his own top-level domain and super hosting setup. But as the parts became more and more affordable a man could become an outcast if he didn’t have his own slightly surreal-sounding TLD.

And so it will be in the future when people realise with surprise there was a time before affordable content delivery networks.

A content delivery network, or CDN, is a system of servers spread around the world, serving files from the nearest physical location. Instead of waiting for a file to find its way from a server farm in Silicon Valley 8,000 kilometres away, I can receive it from London, Dublin, or Paris, cutting down the time I wait. The big names — Google, Yahoo, Amazon, et al — use CDNs for their sites, but they’ve always been far too expensive for us mere mortals. Until now.

There’s a service out there ready for you to use as your very own CDN. You have the company’s blessing, you won’t need to write a line of code, and — best of all — it’s free. The name? Google App Engine.

In this article you’ll find out how to set up a CDN on Google App Engine. You’ll get the development software running on your own computer, tell App Engine what files to serve, upload them to a web site, and give everyone round the world access to them.

Creating your first Google App Engine project

Before we do anything else, you’ll need to download the Google App Engine software development kit (SDK). You’ll need Python 2.5 too — you won’t be writing any Python code but the App Engine SDK will need it to run on your computer. If you don’t have Python, App Engine will install it for you (if you use Mac OS X 10.5 or a Linux-based OS you’ll have Python; if you use Windows you won’t).

Done that? Excellent, because that’s the hardest step. The rest is plain sailing.

You’ll need to choose a unique ‘application id’ — nothing more than a name — for your project. Make sure it consists only of lowercase letters and numbers. For this article I’ll use 24ways2008, but you can choose anything you like.

On your computer, create a folder named after your application id. This folder can be anywhere you want: your desktop, your documents folder, or wherever you usually keep your web files. Within your new folder, create a folder called assets, and within that folder create three folders called images, css, and javascript. These three folders are the ones you’ll fill with files and serve from your content delivery network. You can have other folders too, if you like.

That will leave you with a folder structure like this:

24ways2008/
		assets/
			css/
			images/
			javascript/

Now you need to put a few files in these folders, so we can later see our CDN in action. You can put anything you want in these folders, but for this example we’ll include an HTML file, a style sheet, an image, and a Javascript library.

In the top-level folder (the one I’ve called 24ways2008), create a file called index.html. Fill this with any content you want. In the assets/css folder, create a file named core.css and throw in a couple of CSS rules for good measure. In the assets/images directory save any image that takes your fancy — I’ve used the silver badge from the App Engine download page. Finally, to fill the JavaScript folder, add in this jQuery library file. If you’ve got the time and the inclination, you can build a page that uses all these elements.

So now we should have a set of files and folders that look something like this:

24ways2008/
		assets/
				index.html
				css/
						core.css
				images/
						appengine-silver-120x30.gif
				javascript/
						jquery-1.2.6.min.js

Which leaves us with one last file to create. This is the important one: it tells App Engine what to do with your files. It’s named app.yaml, it sits at the top-level (inside the folder I’ve named 24ways2008), and it needs to include these lines:

application: 24ways2008
version: 1
runtime: python
api_version: 1

handlers:
- url: /
	static_files: assets/index.html
	upload: assets/index.html

- url: /
	static_dir: assets

You need to make sure you change 24ways2008 on the first line to whatever you chose as your application id, but otherwise the content of your app.yaml file should be identical. And with that, you’ve created your first App Engine project. If you want it, you can download a zip file containing my project.

Testing your project

As it stands, your project is ready to be uploaded to App Engine. But we couldn’t call ourselves professionals if we didn’t test it, could we? So, let’s put that downloaded SDK to good use and run the project from your own computer.

One of the files you’ll find App Engine installed is named dev_appserver.py, a Python script used to simulate App Engine on your computer. You’ll find lots of information on how to do this in the documentation on the development web server, but it boils down to running the script like so (the space and the dot at the end are important):

dev_appserver.py .

You’ll need to run this from the command-line: Mac users can run the Terminal application, Linux users can run their favourite shell, and Windows users will need to run it via the Command Prompt (open the Start menu, choose ‘Run…’, type ‘cmd‘, and click ‘OK’). Before you run the script you’ll need to make sure you’re in the project folder — in my case, as I saved it to my desktop I can go there by typing

cd ~/Desktop/24ways2008

in my Mac’s Terminal app; if you’re using Windows you can type

cd "C:\Documents and Settings\username\Desktop\24ways2008"

If that’s successful, you’ll see a few lines of output, the last looking something like this:

INFO     2008-11-22 14:35:00,830 dev_appserver_main.py] Running application 24ways2008 on port 8080: http://localhost:8080

Now you can power up your favourite browser, point it to http://localhost:8080/, and you’ll see the page you saved as index.html. You’ll also find your CSS file at http://localhost:8080/css/core.css. In fact, anything you put inside the assets folder in the project will be accessible from this domain. You’re running our own App Engine web server!

Note that no-one else will be able to see your files: localhost is a special domain that you can only see from your computer — and once you stop the development server (by pressing Control–C) you’ll not be able to see the files in your browser until you start it again.

You might notice a new file has turned up in your project: index.yaml. App Engine creates this file when you run the development server, and it’s for internal App Engine use only. If you delete it there are no ill effects, but it will reappear when you next run the development server. If you’re using version control (e.g. Subversion) there’s no need to keep a copy in your repository.

So you’ve tested your project and you’ve seen it working on your own machine; now all you need to do is upload your project and the world will be able to see your files too.

Uploading your project

If you don’t have a Google account, create one and then sign in to App Engine. Tell Google about your new project by clicking on the ‘Create an Application’ button. Enter your application id, give the application a name, and agree to the terms and conditions. That’s it. All we need do now is upload the files.

Open your Mac OS X Terminal, Windows Command Prompt, or Linux shell window again, move to the project folder, and type (again, the space and the dot at the end are important):

appcfg.py update .

Enter your email address and password when prompted, and let App Engine do it’s thing. It’ll take no more than a few seconds, but in that time App Engine will have done the equivalent of logging in to an FTP server and copying files across. It’s fairly understated, but you now have your own project up and running. You can see mine at http://24ways2008.appspot.com/, and everyone can see yours at http://your-application-id.appspot.com/. Your files are being served up over Google’s content delivery network, at no cost to you!

Benefits of using Google App Engine

The benefits of App Engine as a CDN are obvious: your own server doesn’t suck up the bandwidth, while your visitors will appreciate a faster site. But there are also less obvious benefits.

First, once you’ve set up your site, updating it is an absolute breeze. Each time you update a file (or a batch of files) you need only run appcfg.py to see the changes appear on your site. To paraphrase Joel Spolsky, a good web site must be able to be updated in a single step. Many designers and developers can’t make that claim, but with App Engine, you can.

App Engine also allows multiple people to work on one application. If you want a friend to be able to upload files to your site you can let him do so without giving him usernames and passwords — all he needs is his own Google account. App Engine also gives you a log of all actions taken by collaborators, so you can see who’s made updates, and when.

Another bonus is the simple version control App Engine offers. Do you remember the file named app.yaml you created a while back? The second line looked like this:

version: 1

If you change the version number to 2 (or 3, or 4, etc), App Engine will keep a copy of the last version you uploaded. If anything goes wrong with your latest version, you can tell App Engine to revert back to that last saved version. It’s no proper version control system, but it could get you out of a sticky situation.

One last thing to note: if you’re not happy using your-application-id.appspot.com as your domain, App Engine will quite happily use any domain you own.

The weak points of Google App Engine

In the right circumstances, App Engine can be a real boon. I run my own site using the method I’ve discussed above, and I’m very happy with it. But App Engine does have its disadvantages, most notably those discussed by Aral Balkan in his post ‘Why Google App Engine is broken and what Google must do to fix it‘.

Aral found the biggest problems while using App Engine as a web application platform; I wouldn’t recommend using it as such either (at least for now) but for our purposes as a CDN for static files, it’s much more worthy. Still, App Engine has two shortcomings you should be aware of.

The first is that you can’t host a file larger than one megabyte. If you want to use App Engine to host that 4.3MB download for your latest-and-greatest desktop software, you’re out of luck. The only solution is to stick to smaller files.

The second problem is the quota system. Google’s own documentation says you’re allowed 650,000 requests a day and 10,000 megabytes of bandwidth in and out (20,000 megabytes in total), which should be plenty for most sites. But people have seen sites shut down temporarily for breaching quotas — in some cases after inexplicable jumps in Google’s server CPU usage. Aral, who’s seen it happen to his own sites, seemed genuinely frustrated by this, and if you measure your hits in the hundreds of thousands and don’t want to worry about uptime, App Engine isn’t for you.

That said, for most of us, App Engine offers a fantastic resource: the ability to host files on Google’s own content delivery network, at no charge.

Conclusion

If you’ve come this far, you’ve seen how to create a Google App Engine project and host your own files on Google’s CDN. You’ve seen the great advantages App Engine offers — an excellent content delivery network, the ability to update your site with a single command, multiple authors, simple version control, and the use of your own domain — and you’ve come across some of its weaknesses — most importantly the limit on file sizes and the quota system. All that’s left to do is upload those applications — but not before you’ve finished your Christmas shopping.