Fast and Simple Usability Testing

Everyone knows by now that they should test the usability of their applications, but still hardly anybody actually does it. In this article I’ll share some tips I’ve picked up for doing usability tests quickly and effectively.

Relatively recent tools like Django and Ruby on Rails allow us to develop projects faster and to make significant changes later in the project timeline. Usability testing methods should now be adapted to fit this modern approach to development.

When to test

In an ideal world usability tests would be carried out frequently from an early stage of the project. Time and budget constraints lead this to be impractical; usability is often the first thing to get dropped from the project plan.

If you can only test at one stage in the project, whatever the size, the most valuable time is before your first public beta — leaving long enough to fix issues and not so late that you can’t rethink your scope.

There are three main categories of usability test:

• Testing design mockups
• Testing a new working application
• Testing established applications

Each category requires a slightly different approach. For small modern web projects you are most likely to be testing a new working application. You will of course have already done functional tests so you won’t be worried about the user breaking things. The main differences between the categories apply in how you word The Script.

Testing an established application is the most fun in my opinion. Humans are remarkably adaptable and rapidly develop coping strategies to work around usability issues in software they are forced to use. Uncovering these strategies may lead you to understand previously unspoken needs of your users. Often small changes to the application will have a dramatic affect on their everyday lives.

Who to test

When you have built a project to scratch your own itch, your intended audience will be people just like you. Test subjects in this case should be easy to find – friends, co-workers etc.

This is not always the case; your users may not be like you at all. When they are not, it’s all the more important to run usability tests. Testing on friends, family and co-workers is better than not doing usability tests at all, but it can’t be compared to testing on actual samples of your intended audience. People who would use the system will provide more genuine feedback and deeper insight.

Never let your test subjects put themselves in the shoes of your ‘actual’ users. For example, you should discourage comments like “Well, I would do this BUT if I was a bus driver I’d do that”. Users are not qualified to put themselves in the position of others. Inaccurate data is often worse than no data.

Aim for five or six test subjects: any more and you probably won’t learn anything new; any less and you’re likely to be overwhelmed by issues stemming from people’s individual personalities.

The Script

The Script is a single side of A4 (or letter) paper, consisting of questions for your testers and reminders for yourself. Have a balance of task-based questions and expectation analysis. This helps maintain consistency across tests. Expectation analysis is more important for testing designs and new applications: “Where would you find X?”, “What would you expect to happen if you clicked on Y?”. In an established system users will probably know where these things are though it can still be illuminating to ask these questions though phrased slightly differently.

Task-based questions involve providing a task for the user to complete. If you are testing an established system it is a good idea to ask users to bring in tasks that they would normally perform. This is because the user will be more invested in the outcome of the task and will behave in a more realistic fashion. When designing tasks for new systems and designs ensure you only provide loose task details for the same reason. Don’t tell testers to enter “Chantelle”; have them use their own name instead. Avoid introducing bias with the way questions are phrased.

It’s a good idea to ask for users’ first impressions at the beginning of the test, especially when testing design mockups. “What are the main elements on the page?” or “What strikes you first?”.

You script should run for a maximum of 45 minutes. 30-35 minutes is better; after this you are likely to lose their attention. Tests on established systems can take longer as there is more to learn from them. When scheduling the test you will need to leave yourself 5 minutes between each one to collate your notes and prepare for the next. Be sure to run through the script beforehand.

Your script should be flexible. It is possible that during the test a trend will come to light that opens up whole new avenues of possible questioning. For example, during one initial test of an established system I noticed that the test subject had been printing off items from the application and placing them in a folder in date order (the system ordered alphabetically). I changed the script to ask future participants in that run, if they ever used external tools to help them with tasks within the system. This revealed a number of interesting issues that otherwise would not have been found.

Running the tests

Treat your test subjects like hedgehogs. Depending on your target audience they probably feel a little nervous and perhaps even scared of you. So make them a little nest out of straw, stroke their prickles and give them some cat food. Alternatively, reassure them that you are testing the system and that they can’t give a wrong answer. Reward them with a doughnut or jam tart at the end. Try to ensure the test environment is relaxed and quiet, but also as close as possible to the situation where they would actually use the system.

Have your subjects talk out loud is very important as you can’t read their minds, but it is a very unnatural process. To loosen up your subjects and get them talking in the way you want them to, try the Stapler Trick. Give them a stapler or similar item and ask them to open it, take the staples out, replace them, shut the stapler and staple some paper – talking all the time about what they see, what they expect to happen, what actually happens and how that matches up. Make them laugh at you.

Say how long the test will take up front, and tell your subject why you are doing it. After the test has been completed, conclude by thanking them for their time and assuring them that they were very useful. Then give them the sugary treat.

What to look for

Primarily, you should look out for incidents where the user stops concentrating on her tasks and starts thinking about the tool and how she is going to use it. For example, when you are hammering in a nail you don’t think about how to use a hammer; good software should be the same. Words like ‘it’ and ‘the system’ and are good indications that the test subject has stopped thinking about the task in hand. Note questioning words, especially where testers question their own judgement, “why can’t I find …”, “I expected to see …” etc. as this indicates that the work flow for the task may have broken down.

Also keep an eye on occasions where the user completely fails to do a task. They may need some prompting to unstick them, but you should be careful not to bias the test. These should be the highest priority issues for you to fix. If users recover from getting stuck, make a note of how they recovered. Prolonged periods of silence from the test subject may also require prompting as they should be talking all the time. Ask them what they are thinking or looking for but avoid words like ‘try’ (e.g. ‘what are you trying to do?’) as this implies that they are currently failing.

Be wary of users’ opinions on aesthetics and be prepared to bring them back to the script if they get side-tracked.

Writing it up

Even if you are the only developer it’s important to summarise the key issues that emerged during testing: your notes won’t make much sense to you a week or so after the test.

If you are writing for other people, include a summary no longer than two pages; this can consist of a list or table of the issues including recommendations and their priorities. Remember to anonymise the users in the report. In team situations, you may be surprised at how many people are interested in the results of the usability test even if it doesn’t relate directly to something that they can fix.

To conclude…

Some usability testing is better than none at all, even for small projects or those with strict deadlines. Make the most of the time and resources available. Choose your users carefully, make them comfortable, summarise your report and don’t forget to leave a doughnut for yourself!