During the software development process, usability evaluations are typically conducted early on to assess whether first-time users understand the purpose of the application, the basic navigation systems, and are able to accomplish simple tasks using those navigation systems. First-time users are great for evaluating these broad concepts. However, when evaluating complex applications — particularly those that are used for productivity, there comes a point when it becomes necessary to move beyond these basic assessments and conduct a more detailed evaluation with more experienced users. Generally, experienced users are not available until the beta test or after the application has been released, so this type of testing is often limited to post-development iterations.
This raises the question — when we’re testing with experienced users, is it really any different than testing with first-time users? The short answer is yes. Experienced users have different needs than first-time users, and the usability evaluation should reflect that. For example, advanced or “power” users will often want to accomplish tasks very quickly. They often like to customize or automate certain features of the application. Therefore, it becomes vitally important to evaluate whether the application works efficiently enough to meet their needs. There is no better way to evaluate whether their needs are being met than by conducting a usability test.
Recruiting
It’s usually very easy to recruit experienced users. If the application is in beta or has already been released, you probably have access to a database with participants’ names and emails, so you can simply use the list to contact people who are already using the product. If possible, try to contact these people by phone, ask them about their level of experience with the product, their background, and try to identify users who articulate their thoughts clearly and descriptively.
You will probably want to recruit a small group of people to represent each type of user. If you have previously defined a set of personas, you will want to make sure that each of your personas are being represented. It is also important to recruit groups of users who have different levels of experience with the product. This allows you to compare them. For example, you may want to recruit a group of first-time users, a group of intermediate users, and a group of advanced or “power” users and test them all together. This will allow you to compare the different groups and gain understanding about what learning is actually taking place over time. From there, you will be able to make informed design decisions and clarify the areas that cause trouble for all groups. You will also gain some insight about how to structure the tips or help documentation for the less-experienced groups.
Preparation
As you’re preparing tasks for experienced users to complete, consider their needs and their specific level of experience. With that in mind, create tasks that will expose answers to the following questions. After using the product for (X) amount of time:
- Which core features are they comfortable / uncomfortable using?
- Which advanced features are they comfortable / uncomfortable using?
- Are they aware of keyboard shortcuts? Do they use them? Which ones?
- Have they adopted any unexpected workarounds?
- Which features do they specifically avoid using?
- What are their reasons for this?
- Which features are they still unaware of?
- Have they developed any habits in using this application?
- What sequences of actions are they taking? Do their actions reveal any cognitive artifacts from previous experience with other applications?
- Do they seem to favor any particular features? Is there any single item that they refer back to very frequently?
- Have they customized and automated the application in a way that meets their needs?
- Have they accomplished tasks quickly enough to meet their needs?
- As they’ve been using this application on their own, was there ever a time when they didn’t know what to do? What did they do to resolve this?
- Did they consult the help file? Keep trying? Ask another person?
And, most importantly…
- Does this application help them to accomplish their goals?
- Are there any needs that have not been met by this application?
Environmental Artifacts
Contextual inquiry (visiting users in their home, office, or wherever they use the application) is usually used as a research tool at the beginning of a development cycle. However, it is also one of the most intriguing advantages of testing with experienced users. You can go out into the wild and get a feel for how users are actually using your application in the context of their everyday lives, rather than the context of a plain white room with a stranger and a video camera. This often serves to curb your assumptions about how people are using the application, when they’re using it, and what they’re using it for. As you conduct a post-development contextual inquiry, look for the following:
- Are they using the application for its intended purpose?
- Are they using the application for any unintended purposes?
- Which browser or operating system are they using? Which version?
- Which applications do they have running simultaneously? Are they using multiple applications together because their needs are not being met?
- Look at their settings — have they changed anything from default?
- Do they use multiple monitors?
- Are they typically distracted or multitasking while using this application?
- What is their workstation like?
- Are they using special hardware to help them use this application?
- Are there any unexpected items nearby — for example, notepads to help them draw diagrams, take notes, or just remember things?
- What have users created or accomplished on their own, using this application? (Ask users to show off some work samples.)
Concerns
Even though there are many benefits to testing with experienced users, they also present a unique set of concerns. Experienced users have mastered the application, so they don’t always have to think about what they are doing. As such, they may not be able to verbalize their reasons for clicking a certain button or doing something a certain way because it has simply become an automated behavior. Furthermore, it becomes very easy for expert users to get overly opinionated and criticize various aspects of your design. Don’t let expert users derail the usability study with their opinions. Try to keep the session focused on their behaviors, as this is what you are most interested in. Their opinions are certainly important, but they should be considered in the context of many other factors and should not be given more weight than necessary. Most importantly, don’t let one user’s opinion bias the whole evaluation.
Sources:
Testing Expert Users, Jakob Nielsen
Observing the User Experience: A Practitioner’s Guide to User Research, Mike Kuniavsky
About Face 3: The Essentials of Interaction Design, Alan Cooper & Robert Reimann