When most people think of usability testing, they picture a moderator sitting in a room watching as a user completes several tasks. This approach is known as in-person moderated usability testing. However, it is not the only type of usability testing.
Increasingly, usability testing is happening remotely and often without the involvement of a moderator.
This unmoderated, remote testing is possible thanks to a new generation of online tools such as UserZoom, UserTesting, Maze, Lookback, or my new personal favorite, Userfeel.
Typically the testing tool will provide you with a custom link to send to participants. When the participant visits this link, they are assigned a task to complete on a given website and recorded for later review as they complete it. As they attempt to complete the task, they are encouraged to speak their thinking aloud so the person reviewing the video can understand their approach.
In this post, I will introduce you to unmoderated usability testing and explain how to run it successfully. But first, let’s explore why you might want to consider unmoderated usability testing over moderated.
Why Should I Use Unmoderated Usability Testing?
You cannot talk about unmoderated usability testing without comparing it to moderated testing, and in some ways, you cannot beat moderated usability testing. When you are watching and interacting with a user of your digital service in real-time, you can get depths of insights into their thinking and behavior that is impossible in any other way.
Moderated usability testing allows you to keep the user on track and prevent them from getting completely stuck. Most importantly, moderated usability testing allows you to ask users questions. But moderated usability testing does have its limitations.
People Don’t Feel As Self-Conscious
For a start, it can be intimidating for participants. Many feel they are the ones being tested, and having somebody watching you while you complete a task can change your behavior.
By contrast, unmoderated testing provides some distance between the participant and the viewer. Although intellectually, the participant knows you will watch their session, they are less self-conscious and more likely to be honest. It is easier for them to criticize an experience when not looking somebody in the eye.
Unmoderated Usability Testing Scales Better
The other problem with moderated usability testing is that it doesn’t scale well. Although we know from Jacob Nielsen’s research that more than six participants lead to diminishing returns regarding new issues uncovered, there are still times when you want to test with more participants.
Avoids Scheduling Issues
Finally, moderated usability testing involves scheduling time for the participant and moderator to meet. That can prove inconvenient for both parties and frustrating for the moderator when some participants fail to turn up.
By contrast, an unmoderated usability session allows the participant to complete the task and the reviewer to look over the recordings at their leisure.
In summary, unmoderated usability testing allows you to:
- test more participants,
- spend less time arranging and running sessions,
- and gives participants a more relaxed, genuine experience.
So knowing this, when should you use unmoderated usability testing.
When Should I Use Unmoderated Usability Testing?
As a general rule of thumb, it is better to carry out usability testing during the early phases of development because it is easier to make changes. For example, usability testing can be superb for competitive analysis. Running unmoderated usability testing on competitor websites can prove highly insightful.
However, in my experience, unmoderated usability testing is better used later in the development process when the website is either getting nearer to launch or has already gone live.
If a user encounters a problem due to the website or prototype being incomplete (such as a broken link), there will not be a moderator available to help them recover.
This fact makes unmoderated usability testing a supplement to moderated testing earlier in the development cycle rather than a replacement.
That said, unmoderated usability testing is an ideal tool for regularly testing a live website. This is because you can intercept people using the website with an exit-intent popup, ask if they are willing to complete a short session there, and then send them to the link provided by your testing tool.
Of course, on a live site, you can also use a session recorder like Microsoft Clarity or Hotjar. Still, these don’t allow specific functionality testing, and you also don’t get the user explaining their thinking.
How, then, do you run unmoderated usability testing?
How Do I Run Unmoderated Usability Testing?
Thanks to the excellent tools now available, running unmoderated usability testing is technically straightforward. However, as with all usability testing, the real challenges are non-technical.
To ensure you get the best results from running your usability testing, I recommend a four-step process starting with clearly defining the question you need to answer.
1. Decide on the Question You Need Answering
Unmoderated usability testing sessions tend to be shorter than their moderated counterpart because they often involve a spontaneous decision to participate by those visiting your site, following you on social media, or reading your emails.
That means, realistically, you can probably only ask users to complete one task, as their commitment will be low, and that means your session needs to be very focused. But, of course, you can always run more sessions later.
The first stage of running any unmoderated usability testing is, therefore, to decide on what question you need answering. Of course, that question will depend on your circumstance but could range from “did people see my call to action?” to “why are users dropping out of my checkout process?“
If you don’t have a clearly defined question, you will almost certainly get less value from any usability session and end up wasting everybody’s time, including your own.
Once you have decided on the question, the next step is to design a task to help you find the answer.
2. Design a Task to Help You Answer Your Question
There are two essential things to consider when explaining and designing the task.
First, you need to be careful not to give participants clues on how to complete the task. For example, let’s say you had concerns that people did not see the subscription call to action for your newsletter signup. Your task could be:
Imagine you are interested in hearing more from this website. Now sign up for their newsletter.
On the surface, this sounds like a reasonable task. However, it has two flaws. First, it tells the user there is a newsletter to sign up to. Second, it encourages them to look for words like “newsletter” or “sign up.” These are clues that can help the user complete the task if those words appeared on the site. They wouldn’t have those kinds of clues in the real world.
Instead, a better task might be:
Imagine you are interested in hearing more from this website. Can you see a way of getting updates, and if so, please show us how you would go about receiving them.
Also remember, you will not be in the session to prompt them when they go silent. That means it is essential to stress they need to articulate what they are thinking as they complete the task; otherwise, you will get little more value than you would from a session recorder.
With that in mind, your task might become:
Imagine you are interested in hearing more from this website. Can you see a way of getting updates, and if so, please show us how you would go about receiving them. Remember, we don’t know what you are thinking, so please tell us what is going through your head as you complete this task.
With your task created, you can now set up your test and recruit participants.
3. Set Up the Test and Recruit Participants
Setting up an unmoderated test is straightforward. Although it varies slightly between tools, generally, you enter your task and define the start and end URLs for it. In other words, where should the test start on your website (the start URL) and how will the tool know when it is complete (the end URL)
When it comes to recruiting participants, there are a few more things to consider. That said, it is not as hard as everybody thinks.
In many situations getting the right people for a usability test session isn’t as important as you might think. As long as they aren’t involved in the company or project, you will probably learn valuable insights. That said, there are a couple of exceptions.
- If your audience has specific cognitive or physical characteristics (e.g., are young, old, or disabled), you might want to consider that during recruitment.
- If you particularly want insights from people with a certain level of knowledge about your site (e.g., new or returning users), you will also need to qualify for that.
You can find the right people either by the method you use for recruiting or by asking one or two qualifying questions before you start.
For example, if you want to recruit existing customers, you can use your mailing list to source participants. However, if you wanted first-time users, an intercept popup on the website might be a better way to go, combined with a qualifying question asking them if they have used the site before.
I often find participants through intercept popups, mailing lists, or social media. That said, most usability testing tools will also recruit participants on your behalf for a small fee. That can often be the best option if you want people to meet a specific demographic profile.
Once you have got participants to complete the task, the final step is to review the results.
4. Reviewing the Results
I’ll be honest, reviewing videos from unmoderated usability testing can be tedious, especially if you have many of them. Fortunately, a lot of tools will allow you to speed up playback or skip inactivity.
Also, some tools like Userfeel will provide you with quantitive data, which can prove helpful too. Things like the system usability scale or time to complete the task. That can help when trying to benchmark against previous testing.
There is no magic formula to spotting issues when watching videos back, but I would encourage you to pay attention to what users fail to say or do.
It is easier to spot people’s mistakes than to spot things that they didn’t do. However, these can often be the reason they fail.
For example, were users who scrolled straight past instructional text less likely to complete the task successfully?
Equally, did users not say something? For example, failing to comment on the value proposition might mean they missed it or found it uninspiring. Also, long periods of silence and inactivity could indicate confusion or even frustration.
You need to learn to read between the lines as much as observe what is happening when watching usability testing. But don’t presume you have interpreted these things correctly. Instead, consider running another test to confirm.
Practice Makes Perfect
Like everything, interpreting the results of unmoderated usability testing takes practice. The same is true for coming up with questions to ask and designing tasks to support them.
The best way of practicing is to start running this testing regularly. Get yourself a good tool and aim to test at least once a month. Not only will you get better at running the tests, but you will also learn a lot and be able to see improvements over time.
