This post is a part of a podcast course on truly lean UX. Listen to the whole episode above, or find out more about the course.
Also in this lesson, Marcus interviews Matt Curry previously from Lovehoney
Finally, Paul chat’s with Mike from Balsamiq about using usability testing post-launch.
There are two characteristics of the web that makes it a unique and exciting medium to design for. Those are its ability to gather unprecedented amounts of data on how users interact with it and the ease with which things can be changed.
Compare that to a printed brochure. Once that brochure goes out into the world, you cannot change it, and you have no idea of how people are interacting with it.
Yet, despite the enormous potential provided by digital, many organisations completely ignore it. They launch a website and other than a few textual changes and updates; they make no substantive changes until the next redesign a few years down the line.
By contrast, companies that get the web are continually optimising their websites post-launch to ensure they reach their potential.
The importance of this ongoing optimisation cannot be overstated.
Why Post Launch Optimisation is Crucial
In truth, you can do all the pre-launch testing you want, but you do not really know how users will respond to a website until it has been launched. Pre-launch testing can certainly avoid costly mistakes, but it cannot give a completely realistic picture of the real experience.
However, once a site is live, and real users interact daily, you can see issues emerge and discover opportunities for improvement.
Improve Critical Metrics
If you have allowed time and resources for post-launch optimisation, you can significantly improve engagement, conversion and word of mouth recommendation.
Small changes post-launch can make an enormous difference to the effectiveness of a website. For example, Jared Spool removed the need for users to log in to an ecommerce website and saw a 45% increase in sales, which equated to $300 million in increased revenue in the first year alone.
But post-launch testing can provide significant cost savings too because it avoids the need for expensive periodic redesigns.
Avoid Expensive Redesigns
Redesigning a website is extremely wasteful for two reasons.
When a site is redesigned, it typically throws out the good with the bad, starting almost entirely from scratch. It is a blunt tool that ignores that some elements of your existing website will be working.
Second, because redesigns are expensive, they only happen every few years. That means that for most of its life, your website is not operating at peak efficiency.
Instead, we should be evolving our websites over time through a rigorous process of post-launch testing. But what exactly does that look like?
The Process for Post-Launch Optimisation
Once a site goes live, and users have had time to adjust to it, an organisation typically adopts a cyclic process of optimisation. This process consists of three steps that the organisation repeats on an almost continual basis. These steps are:
- Find a problem area to address.
- Diagnose the exact problem.
- Test potential solutions.
Let’s explore these in more depth.
1. Find a Problem Area
Optimising a website begins by identifying areas of the site that are not performing as effectively as possible. To do that, you need to be clear what exactly constitutes a problem.
What Problems to Look For
When seeking out areas of improvement on a website, I am typically looking for shortcomings in three key areas:
- Engagement – Are users failing to engage with the content or abandoning the website prematurely?
- Usability – Are users struggling to find the content they require or complete critical tasks on the website?
- Conversion – Are users abandoning the website before completing a call to action? Are those who did act failing to return to the website at a later date?
See Also: How To Define Key Performance Indicators
Looking at these kinds of metrics will give you an indication that there is room for improvement. However, you then need to identify where on the site things are going wrong. Analytics can typically help with this.
Using Analytics to Find Problems
Finding problem areas is not an exact science and does require some intuition. However, analytics are a good starting point for identifying general areas where there may be issues to address.
I am the first to admit I am no analytics expert. However, even with my limited knowledge, I find it relatively easy to find potential issues on a website.
In particular I pay attention to:
- Bounce Pages – These are pages that users arrive on and then immediately leave without viewing any other content. Pages that don’t draw the user deeper into the site and towards a call to action are normally an area for improvement.
- Site Routes – What route are users taking through the website. Are they taking the most direct route to a call to action, or are they being distracted? Are there some routes through the site that perform worst than others? If people follow a particular path, are they likely to abandon the site?
- Drop off Points – What pages cause people to exit the site?
- High Dwell Time Pages – What pages slow the user’s journey? What pages do people tend to get stuck on?
Notice that I am interested in the analytics of specific pages. In other words, knowing the overall dwell time or bounce rate of the site won’t help me narrow in on problem areas. I need to know what specific pages have a high bounce rate or dwell time.
What you should end up with is a list of potential pages that could have problems. Of course, just because a page has a high dwell time or many people abandon the site from it, doesn’t necessarily mean it is failing. Neither does knowing what pages might have a problem allow us to diagnose the exact issue.
To work out exactly what is going on, we need to look at each problem page in more depth. Unsurprisingly I start by focusing on the most poorly performing page based on the criteria above.
2. Diagnosing the Problem
Once I have a problem page to analyse; my next step is to ascertain the problem’s nature. To do that I start with an app like Fullstory or Hotjar. That is because both of these apps have two useful tools for diagnosing problems – heat maps and session recorders.
Using Heat Maps
Heat maps are a great starting point to narrow down potential issues on a page. The two main types of heat maps you are interested in are scroll maps and click maps.
A scroll map gives you a strong indication of where people’s attention is on a page. If people are scrolling straight past a critical piece of content or call to action, then you may have found your problem.
Click maps are useful too because they help see if users understand the page. Are they trying to click things that are not clickable? Are they clicking secondary content that is leading them further from your call to action? Or worst still are they rage clicking randomly out of frustration!?
Heat maps give a useful overview of user behaviour, but watching back some user sessions will provide more specific insights.
Watching Session Recordings
Once you can see possible problems, it is worth watching back some recording of user sessions to see if users are actually behaving as expected.
For example, imagine that users seem to be clicking on a non-clickable page element. Using a tool like Fullstory, you can filter all your recorded sessions and watch back only sessions where users have tried to click that element. Now you can see what they did next or even in some cases what led them to click the element in the first place.
After clicking, did they abandon the site or adapt and continue without an issue? If they abandoned, then there is clearly something that needs fixing.
Of course, session recordings do have their limits. Sometimes you will observe user behaviour and have no idea why they took that action. In those cases where you are left confused, it is time to turn to usability testing.
Running Usability Testing
If time allows, once I have identified a specific issue on a specific page, I like to run some quick usability testing. I set up a simple test that will require the user to encounter the problem area I have identified and see what happens. I observe their behaviour and ask them to explain what they did and why they did it.
See Also: How to Run Regular Usability Testing Yourself for Free
Typically these are carried out over zoom and only take a few minutes to complete. I also often rely on friends and family for this testing, unless the website is highly specialised.
By this point, I will have a pretty solid idea of the problem associated with this page on the website and have a few ideas on how to fix it. But how do I know which approach would be best and be confident I am not going to make things worse?
That brings us on to our final step – test our potential solutions.
3. Testing Potential Solutions
When it comes to testing possible solutions, you have two basic approaches – quantitive or qualitative. With that in mind, let’s look at these two options and when we should use each.
Using Quantitive A/B Testing
One of the most popular options when it comes to optimising a website post-launch is A/B testing. This Quantitive approach to testing works well because it tests with real users, interacting completely naturally on your live website. If a solution performs well in A/B testing, you can be confident it will work when rolled out across the entire site for all audiences.
In case you are unaware, at the most basic level, A/B testing involves showing a percentage of site visitors different versions of a page. This allows you to try one or more potential solutions on the live site to see if any of them perform better than the current version.
By only showing the variations to a small percentage of users, you avoid rolling out a solution that could actually make things worse.
To ascertain whether a solution will perform better, you need to gather enough data to be statistically accurate. This perceived need for a large number of results can put people off using A/B testing, thinking it is a tool for highly trafficked sites like Amazon.
Although A/B testing is more suited to high traffic websites because they will quickly generate the required level of results, it can be used on any site with some adaptation.
See Also: A Beginners Guide to A/B and Multivariate Testing
One answer is for low traffic websites to wait until they get enough results. Yes, this does require some patience, but it will ensure statistically significant results.
Another approach is to make a judgement call on a solution’s effectiveness without waiting until you get a statistically significant dataset. Just because an A/B testing tool says you don’t have enough results, doesn’t mean you have to listen to it!
The final option is to adapt the test to increase the number of results you receive. That can be done by closing the gap between the thing being tested and the point of conversion.
For example, changing the text on a newsletter sign up form is intimately connected with the success criteria of pressing the subscribe button. However, testing the impact of a blog post title on the newsletter sign up is not as strictly related so that the conversion rate will be relatively lower. That means you will have to wait longer for statistically significant results.
Instead of making your success criteria, something that doesn’t often happen (like newsletter sign up) you could look at a smaller, more common action. For example, if you wanted to test those blog post titles, you may be better testing how many users click to view the post, rather than whether they go on to signup.
Without a doubt, A/B testing is powerful and an excellent way of optimising your website, especially when it comes to testing different combinations of text and imagery.
In fact, there are many tools out there which allow you to create variations of content on your site without any coding skills whatsoever. Google even offers a free tool called Optimise that requires no setup if you already have Analytics installed.
However, things become much more complicated if the changes in your variations are more substantial. In those cases, qualitative testing may be more appropriate.
Carry out Qualitative Testing
Qualitative testing essentially revolves around testing a prototype with a small number of users. Instead of testing solutions on a live site, you create a prototype and carry out usability testing.
Qualitative testing does have some advantages over A/B testing, not least because you do not need to rely on statistically significant results. That means there is little waiting around to find out which solution performs the best.
Qualitative testing has another advantage, too; it tells you why a certain variation performed better. That is because you can actually ask users about what they did and why.
However, probably the biggest advantage of qualitative testing is that you are testing with a prototype. You do not need to create a fully working version of your variations, allowing you to test more complex changes with nothing more than a design comp or wireframe.
Both approaches have advantages and disadvantages, and so it makes sense to combine them as appropriate. Together they should lead to a solution that should fix whatever problem you have identified.
Once you have rolled out your solution, you can then turn your attention to the next issue.
Rinse and Repeat
The secret to success with optimising your website is to make it an ongoing process. That means when you fix one problem, you return to the start of our three-step process and begin again by identifying the next biggest issue in the experience.
By continually iterating on a site over time, you will gradually improve its effectiveness and avoid the need for a costly redesign a few years down the line.
Of course, the reality is that even if you are keen to optimise a site over the long term, it is not always easy to get stakeholders to agree. I, therefore, want to end by sharing with you three tips for ensuring post-launch optimisation happens.
Ensuring Post Launch Optimisation Happens
The first piece of advice I would give is to ensure you build post site optimisation into the initial project plan when redesigning your website. A big part of the problem is that most project plans end when the website launches and that is never a good idea even if you do not intend to do ongoing post-launch optimisation.
When a website is launched, there will inevitably be bugs to fix and content that needs tweaking. That is why I tend to favour putting site launch about two thirds through the overall project timeline. That allows ample time for fixing problems and an opportunity to establish a habit of post-launch optimisation.
Second, I would encourage you to talk about post-launch functionality from the very start of a project. Stakeholders will normally have ideas about how the site could be improved, so to avoid scope creep, start a phase two wish list. That will stop scope creep and establish the idea that a website can evolve and change post-launch.
Finally, take every opportunity to talk about post-launch during development. Ask questions about how stakeholders are going to resource and manage the post-launch stage.
One reason I like the UK Government service manual so much is that when they list the phases of a website “live” is a defined stage right alongside “Discovery”, “Alpha” and “Beta”. They even have a phase for “Retirement” which is also often overlooked.
When it comes to post-launch optimisation, something is better than nothing. Even a few weeks of post-launch optimisation is better than none at all. Stakeholders don’t need to commit to improving the site forever, but if you can even get them to do it for a short while, they will quickly see the benefits of doing so long term.
