HTML5 contains more than just the new entities for a more meaningful document, it also contains an arsenal of JavaScript APIs. So many in fact, that some APIs have outgrown the HTML5 spec’s backyard and have been sent away to grow up all on their own and been given the prestigious honour of being specs in their own right.
So when I refer to (bendy finger quote) “HTML5”, I mean the HTML5 specification and a handful of other specifications that help us authors build web applications.
Examples of those specs I would include in the umbrella term would be: geolocation, web storage, web databases, web sockets and web workers, to name a few.
For all you guys and gals, on this special 2009 series of 24 ways, I’m just going to focus on data storage and offline applications: boldly taking your browser where no browser has gone before!
Web Storage
The Web Storage API is basically cookies on steroids, a unhealthy dosage of steroids. Cookies are always a pain to work with. First of all you have the problem of setting, changing and deleting them. Typically solved by Googling and blindly relying on PPK’s solution. If that wasn’t enough, there’s the 4Kb limit that some of you have hit when you really don’t want to.
The Web Storage API gets around all of the hoops you have to jump through with cookies. Storage supports around 5Mb of data per domain (the spec’s recommendation, but it’s open to the browsers to implement anything they like) and splits in to two types of storage objects:
sessionStorage– available to all pages on that domain while the window remains openlocalStorage– available on the domain until manually removed
Support
Ignoring beta browsers for our support list, below is a list of the major browsers and their support for the Web Storage API:
- Latest: Internet Explorer, Firefox, Safari (desktop & mobile/iPhone)
- Partial: Google Chrome (only supports
localStorage) - Not supported: Opera (as of 10.10)
Usage
Both sessionStorage and localStorage support the same interface for accessing their contents, so for these examples I’ll use localStorage.
The storage interface includes the following methods:
setItem(key, value)getItem(key)key(index)removeItem(key)clear()
In the simple example below, we’ll use setItem and getItem to store and retrieve data:
localStorage.setItem('name', 'Remy');
alert( localStorage.getItem('name') );
Using alert boxes can be a pretty lame way of debugging. Conveniently Safari (and Chrome) include database tab in their debugging tools (cmd+alt+i), so you can get a visual handle on the state of your data:
Viewing localStorage
As far as I know only Safari has this view on stored data natively in the browser. There may be a Firefox plugin (but I’ve not found it yet!) and IE… well that’s just IE.
Even though we’ve used setItem and getItem, there’s also a few other ways you can set and access the data.
In the example below, we’re accessing the stored value directly using an expando and equally, you can also set values this way:
localStorage.name = "Remy";
alert( localStorage.name ); // shows "Remy"
The Web Storage API also has a key method, which is zero based, and returns the key in which data has been stored. This should also be in the same order that you set the keys, for example:
alert( localStorage.getItem(localStorage.key(0)) );
// shows "Remy"
I mention the key() method because it’s not an unlikely name for a stored value. This can cause serious problems though.
When selecting the names for your keys, you need to be sure you don’t take one of the method names that are already on the storage object, like key, clear, etc. As there are no warnings when you try to overwrite the methods, it means when you come to access the key() method, the call breaks as key is a string value and not a function.
You can try this yourself by creating a new stored value using localStorage.key = "foo" and you’ll see that the Safari debugger breaks because it relies on the key() method to enumerate each of the stored values.
Usage Notes
Currently all browsers only support storing strings. This also means if you store a numeric, it will get converted to a string:
localStorage.setItem('count', 31);
alert(typeof localStorage.getItem('count'));
// shows "string"
This also means you can’t store more complicated objects natively with the storage objects. To get around this, you can use Douglas Crockford’s JSON parser (though Firefox 3.5 has JSON parsing support baked in to the browser – yay!) json2.js to convert the object to a stringified JSON object:
var person = {
name: 'Remy',
height: 'short',
location: 'Brighton, UK'
};
localStorage.setItem('person', JSON.stringify(person));
alert( JSON.parse(localStorage.getItem('person')).name );
// shows "Remy"
Alternatives
There are a few solutions out there that provide storage solutions that detect the Web Storage API, and if it’s not available, fall back to different technologies (for instance, using a flash object to store data). One comprehensive version of this is Dojo’s storage library. I’m personally more of a fan of libraries that plug missing functionality under the same namespace, just as Crockford’s JSON parser does (above).
For those interested it what that might look like, I’ve mocked together a simple implementation of sessionStorage. Note that it’s incomplete (because it’s missing the key method), and it could be refactored to not using the JSON stringify (but you would need to ensure that the values were properly and safely encoded):
// requires json2.js for all browsers other than Firefox 3.5
if (!window.sessionStorage && JSON) {
window.sessionStorage = (function () {
// window.top.name ensures top level, and supports around 2Mb
var data = window.top.name ? JSON.parse(window.top.name) : {};
return {
setItem: function (key, value) {
data[key] = value+""; // force to string
window.top.name = JSON.stringify(data);
},
removeItem: function (key) {
delete data[key];
window.top.name = JSON.stringify(data);
},
getItem: function (key) {
return data[key] || null;
},
clear: function () {
data = {};
window.top.name = '';
}
};
})();
}
Now that we’ve cracked the cookie jar with our oversized Web Storage API, let’s have a look at how we take our applications offline entirely.
Offline Applications
Offline applications is (still) part of the HTML5 specification. It allows developers to build a web app and have it still function without an internet connection. The app is access via the same URL as it would be if the user were online, but the contents (or what the developer specifies) is served up to the browser from a local cache. From there it’s just an everyday stroll through open web technologies, i.e. you still have access to the Web Storage API and anything else you can do without a web connection.
For this section, I’ll refer you to a prototype demo I wrote recently of a contrived Rubik’s cube (contrived because it doesn’t work and it only works in Safari because I’m using 3D transforms).
Offline Rubik’s cube
Support
Support for offline applications is still fairly limited, but the possibilities of offline applications is pretty exciting, particularly as we’re seeing mobile support and support in applications such as Fluid (and I would expect other render engine wrapping apps).
Support currently, is as follows:
- Latest: Safari (desktop & mobile/iPhone)
- Sort of: Firefox‡
- Not supported: Internet Explorer, Opera, Google Chrome
‡ Firefox 3.5 was released to include offline support, but in fact has bugs where it doesn’t work properly (certainly on the Mac), Minefield (Firefox beta) has resolved the bug.
Usage
The status of the application’s cache can be tested from the window.applicationCache object. However, we’ll first look at how to enable your app for offline access.
You need to create a manifest file, which will tell the browser what to cache, and then we point our web page to that cache:
<!DOCTYPE html>
<html manifest="remy.manifest">
<!-- continues ... -->
For the manifest to be properly read by the browser, your server needs to serve the .manifest files as text/manifest by adding the following to your mime.types:
text/cache-manifest manifest
Next we need to populate our manifest file so the browser can read it:
CACHE MANIFEST
/demo/rubiks/index.html
/demo/rubiks/style.css
/demo/rubiks/jquery.min.js
/demo/rubiks/rubiks.js
# version 15
The first line of the manifest must read CACHE MANIFEST. Then subsequent lines tell the browser what to cache.
The HTML5 spec recommends that you include the calling web page (in my case index.html), but it’s not required. If I didn’t include index.html, the browser would cache it as part of the offline resources.
These resources are implicitly under the CACHE namespace (which you can specify any number of times if you want to).
In addition, there are two further namespaces: NETWORK and FALLBACK.
NETWORK is a whitelist namespace that tells the browser not to cache this resource and always try to request it through the network.
FALLBACK tells the browser that whilst in offline mode, if the resource isn’t available, it should return the fallback resource.
Finally, in my example I’ve included a comment with a version number. This is because once you include a manifest, the only way you can tell the browser to reload the resources is if the manifest contents changes. So I’ve included a version number in the manifest which I can change forcing the browser to reload all of the assets.
How it works
If you’re building an app that makes use of the offline cache, I would strongly recommend that you add the manifest last. The browser implementations are very new, so can sometimes get a bit tricky to debug since once the resources are cached, they really stick in the browser.
These are the steps that happen during a request for an app with a manifest:
- Browser: sends request for your app.html
- Server: serves all associated resources with app.html – as normal
- Browser: notices that app.html has a manifest, it re-request the assets in the manifest
- Server: serves the requested manifest assets (again)
- Browser: window.applicationCache has a status of
UPDATEREADY - Browser: reloads
- Browser: only request manifest file (which doesn’t show on the net requests panel)
- Server: responds with 304 Not Modified on the manifest file
- Browser: serves all the cached resources locally
What might also add confusion to this process, is that the way the browsers work (currently) is if there is a cache already in place, it will use this first over updated resources. So if your manifest has changed, the browser will have already loaded the offline cache, so the user will only see the updated on the next reload.
This may seem a bit convoluted, but you can also trigger some of this manually through the applicationCache methods which can ease some of this pain.
If you bind to the online event you can manually try to update the offline cache. If the cache has then updated, swap the updated resources in to the cache and the next time the app loads it will be up to date. You could also prompt your user to reload the app (which is just a refresh) if there’s an update available.
For example (though this is just pseudo code):
addEvent(applicationCache, 'updateready', function () {
applicationCache.swapCache();
tellUserToRefresh();
});
addEvent(window, 'online', function () {
applicationCache.update();
});
Breaking out of the Browser
So that’s two different technologies that you can use to break out of the traditional browser/web page model and get your apps working in a more application-ny way.
There’s loads more in the HTML5 and non-HTML5 APIs to play with, so take your Christmas break to check them out!
Comments
Comments are ordered by helpfulness, as indicated by you. Help us pick out the gems and discourage asshattery by voting on notable comments.
Got something to add? You can leave a comment below.
02/12/2009
Interesting timing considering the announcement today that Google moving away from Gears.
Thanks for shedding light on this new and somewhat still esoteric subject :)
I also recommend Brad Neuberg’s introduction to HTML5 video, which touches briefly on the subject of storage: http://developer.yahoo.com/yui/theater/video.php?v=neuberg-html5
Vote Helpful or Unhelpful
02/12/2009
Great read, just curious about secure this is? Not that you would want to cache sensitive data. Is there a way of encrypting what is stored?
Will add this to the ever expanding to-look-at list!
Vote Helpful or Unhelpful
02/12/2009
Firefox throws an exception when the localStorage data is larger than 1024bytes so it’s a far cry from the 5mb limit in the spec.
Native JSON is in most stable release browsers now: IE8, Safari 4 (need 4.0.3) and Chrome 3 so it has pretty wide support now.
Vote Helpful or Unhelpful
02/12/2009
I was somehow not aware of the fact that IE8 supports session/local storage. Wow, it now seems like the technology is ready to use.
Thanks for the great article!
Vote Helpful or Unhelpful
02/12/2009
Excellent article, although you didn’t mention the third offline technology that only Safari has currently: offline SQL database.
Vote Helpful or Unhelpful
02/12/2009
This is cool . Excellent way to do things .. hats off!
Vote Helpful or Unhelpful
02/12/2009
Very nice writeup, Remy.
Nicholas Zakas mentioned in his writeup on sessionStorage that:
So testing window.sessionStorage can do that..
In Modernizr development, I’ve seen fearphage present this as a solution:
(‘sessionStorage’ in window)Vote Helpful or Unhelpful
02/12/2009
The new capabilities in HTML5 are spell-bounding and will really help push the transition from websites being internet-only to fully fledged desktop apps. Very exciting. Great read!
Vote Helpful or Unhelpful
02/12/2009
Firebug allows you to browse localStorage (see the “DOM” tab).
Vote Helpful or Unhelpful
02/12/2009
Remy, great article! I will definitely be making more use of this part of the HTML5 spec.
I’ve been following Gears since its inception, but never used it because due to its lack of full support across platforms and browsers.
When I found out a certain iPhone OS update brought offline storage to Mobile Safari (through HTML5) and that Google was making use of it in their mobile web apps (such as Gmail), it seemed like the nail in the coffin.
When this really starts to grow (with wider browser support and consumer awareness), you can forget AIR.
Vote Helpful or Unhelpful
02/12/2009
I really enjoyed reading this article – it was put together nicely!
I found it very useful with the example code snippets as I’ve never played about with these APIs before or put any other form of attention to them apart from knowing about their existence…
Vote Helpful or Unhelpful
02/12/2009
@Remy: Any comment on the difference in memory management and performance between storing a ton of data in memory (as an object or array, for example) and using local/sessionStorage?
I’m thinking for example of a Google Maps implementation where you need to manage a lot of locations/markers/data; in that instance, using the equivalent of a local database to store and retrieve the data sounds like a great solution, but only if it’s going to perform better than simply storing it all in a big array.
Vote Helpful or Unhelpful
02/12/2009
I was wondering what spec in HTML 5 would replace Gears, very interesting read. Thanks.
Vote Helpful or Unhelpful
02/12/2009
@RYAN SEDDON – cheers for pointing that out – I hadn’t spotted that the other browsers had JSON support baked in.
@DEVON GOVETT – yep, Safari has Web Databases as well as Web Storage, but from my playing around, I can’t see any advantage database has over storage. Sure you can do complicated queries, but 99% of the time I have a key and just want to get the value out. Similar to the CouchDB model. Also Web Database is only available in Safari (possibly Chrome, I’ve not checked), whereas Web Storage is available in 3 of the big 4.
@JOHANNES GORSET – cheers for pointing out that the DOM tab introspects the storage object. It seems a bit lame compared to Safari’s DB view, but I guess the data’s still the same!
@MATTHEW PENNELL – I’ve not tested memory management or performance yet, so when I find time, I’ll have a play around.
Vote Helpful or Unhelpful
02/12/2009
@REMY SHARP – Before Safari 4 no version of Safari supported Web Storage, but Safari 3.1+ have supported Web Databases, so sometimes you might need to combine solutions to get the widest possible browser support. Also, if you have written a backend in a language like PHP to store data in Mysql, and you want to port it to JavaScript to run client side and offline, it is certainly easier to do with web databases since you can use the same queries. Just a thought.
Vote Helpful or Unhelpful
03/12/2009
HTML5 is great but without IE’s support we don’t have any advantage because unfortunately it’s the most used browser.
And again we going to use JS to rescue. :)
Vote Helpful or Unhelpful
05/12/2009
This was an excellent read, HTML5 is long overdue.
Vote Helpful or Unhelpful
06/12/2009
Great article, thank you!
You may find PersistJS useful. It is a wrapper library for all the offline storage APIs (global/localStorage, Safari’s SQLite, Gears, Flash, and IE’s old userData), so it supports almost every browser. It’s like dojo storage, but standalone and with more backends.
http://pablotron.org/software/persist-js/
Vote Helpful or Unhelpful
14/12/2009
Remy is correct when he says that debugging can be a real PITA when you are using a manifest file. A technique I’ve found useful is to monitor your web server logs while running your app to see exactly what resources your browser is requesting.
For example:
1) ssh into your server
2) cd to your log directory (prolly /var/log/httpd/ or /var/log/apache2/)
3) Run: sudo tail -f access_log
4) Reload your app in the browser; your request(s) should pop into the terminal.
When you are done watching the log, press Control-C to quit the tail process.
You can also inspect the ApplicationCache sqlite database manually using sqlite3 at the command prompt (with Safari on Mac, at least – not sure about others).
More here if y’all are interested:
http://building-iphone-apps.labs.oreilly.com/ch06.html
Cheers!
Vote Helpful or Unhelpful
Impress us