Testing design: How do you test a design comp?

Paul Boag

We all know that testing the sites we produce is necessary and there are well-established techniques for testing usability. But what about testing design aesthetics? What about validating a design comp?

I have written before about my approach to usability testing, and even how I use an iterative approach to UI design. But, none of these techniques focus much on testing design aesthetics or a design comp. Is testing design something even worth considering or does it undermine the role of the designer?

Is Testing Design Worthwhile?

When I discuss testing design with other designers, I often get an adverse reaction. The perception is that the client is paying the designer for their expertise, so it should be the designer that develops the look and feel without interference. They also express a fear that the design will be compromised somehow if the “masses” get their say.

In someways these reactions are valid. The designer should be making the design decisions, and too many cooks definitely can spoil the broth when it comes to design.

However, in the real world, stakeholders do interfere in the design process, and testing design can be an efficient way of minimising the influence of a single poorly informed person.

I also believe that testing design offers four benefits.

Testing Design Looks Beyond Personal Preference

We all know when we don’t like a design. Many of us even have strong opinions on what quality design looks like. But, none of that matters when it comes to the design of your website. What matters is whether users respond positively to it.

Personal preference shouldn’t be a factor when designing a site, even if that is the preference of the designer. I have created websites I hate, but which the target audience love. That is all that matters.

Testing Design Avoids Confrontations Over Design

I have watched too many projects grind to a halt because the client and designer cannot agree on the design. The designer feels the client should bow to his experience and training, while the client believes that, as he is paying for the site, he should get what he wants.

This kind of stalemate can be incredibly damaging, but testing design is an excellent way of diffusing these differences.

Testing Design Informs the Design Process

Don’t get me wrong. I believe that the designer should be in control of the design. However, testing design is about informing the design process, not replacing the designer.

In my opinion, the more informed the designer is, the better. By meeting real users and having the opportunity to ask them questions about the look and feel of a site, the designer can only be better informed when developing his concepts.

Testing Design Minimises Design by Committee

Often a design is shown to many different people before it is signed off. Either it passes through an official committee or your point of contact hands it around the office. That happens because nobody wants the responsibility of approving a design.

Testing design can offer an alternative approach. By providing hard data on how users will respond to a design it avoids concerns over whether it is the right approach. That deals with both client apprehension over sign-off and different stakeholders debating the right look.

How then do you test aesthetics? How do you test the best colours, typography, styling, etc.? How do you ensure that your design is communicating the right feel and personality to your users?

The first step is to know what you want to learn about the aesthetics.

Asking Users What They Think Isn't Enough

It is not enough to just show users a design and ask them if they like it or not. This approach creates two problems.

First, whether they like a design is not the only criteria by which to judge it. For example, a user might not personally like the design of a law firm (finding it too formal for their personal preferences), but it may still leave them with the right feeling about the company (that it is professional and trustworthy).

Second, when asked what they think, users fall back on comments such as "I don't like the green." Comments like this aren't particularly helpful. You need to know what it is about the green that is a problem.

Focus on Brand Value and Personality

The most important thing to establish with a design is whether it is a good reflection of the company it represents. To do this, you need a strong idea of what the characteristics of the brand are.

There are a variety of ways to go about this process, but my favourite is to imagine the brand as a person. Maybe you can think of a famous person who articulates your brand. Alternatively, you might want to consider the characteristics of a fictional individual you create.

The archetypes book.
Archetypes in Branding is a useful tool for defining the character of your brand.

Whoever this person is they will have specific personality traits that you can express with keywords. Perhaps the person is laid back; maybe they are formal. Whatever the case, this list of words become the core of your brand. The question then becomes, does any design communicate these words?

The only way to know this is to test with real users, because both the designer and stakeholders are too close to the project to be objective. But how do you check a design against a set of words?

Techniques for Testing Design

What follows are the different testing techniques that I use. That is not a comprehensive list, and some I use much more than others. However, it should give you some options.

Each test has its strengths and weaknesses, so a combination is often the way to go.

A/B Design Testing

The most basic test is a simple A/B test. That is ideal when you are not sure of the best approach. Sometimes you can go different directions with a design, and you are unsure which is best suited. In this case, a simple A/B test may be the answer.

Usually, the A/B test involves presenting two designs to a user and asking which they prefer. That is entirely adequate, but you might wish to go a step further.

Helio being used for testing design.
Helio is a tool that makes A/B testing a design straightforward.

Instead of asking them which they prefer, show them your list of brand words and ask which they feel best represents the words. That moves the conversation on from simple likes and dislikes to something directly related to your brand.

The Semantic Differential Survey

The semantic differential survey is a fancy name for a simple idea.

In this test, you present the user with either single or multiple designs and rate them against opposing words.

For example is your design formal or casual, minimal or busy. These choices can be either a black and white choice, or a sliding scale.

Example of using a semantic differential survey for testing design.
A semantic differential survey is an excellent way of seeing how your site is performing against your chosen keywords.

One tip if you go down the sliding scale route. Rate using an odd number of options (e.g., One to five instead of one to ten). If you pick an even number, then users will often choose the middle option (e.g., five). An odd number forces users to lean one way or the other.

There is also a variation of a standard semantics survey that I find useful. In this test I take the target keywords that represent the brand, add their opposite and then sprinkle in some other possible words somebody might use to describe the design. I then post a survey showing the design and asking people to pick which words resinate the most. This is useful for exploring concerns that the design might be engendering the wrong kind of reaction.

Testing design using keyword matching.
Keyword matching can be a useful way of identifying whether you have established the right aesthetic for a brand.

Testing Design Using the Competition

Another approach is to see how well your new design performs compared to the competition.

In this test, you display your design alongside the competition in a single place. You then ask the user to pick which site best represents each of the keywords on your list (e.g., which is the most formal, reliable, professional).

Rating a site design against the competition will give you an idea of whether you have successfully differentiated yourself.

That is an excellent way of seeing whether you are outperforming the competition. It also will flag up a common problem where designs within a sector all end up looking the same. If users struggle to select one site over another, then it is obvious you have not done enough to differentiate yourself.

Snap Tests

The final test focuses on first impressions. We know that users make judgments about sites in a blink of an eye and so we need to accommodate this.

Research shows that if people have an initial negative impression of a site they are more likely to look critically at it regarding content and usability. It's therefore crucial that the site makes an excellent initial impression.

The snap test is as it sounds. You show the user your design for only 1 or 2 seconds and then get their reaction.

Because of the short timeframe you may find that users struggle to judge the design against your list of keywords. However, you can at least discover whether they had a positive or negative reaction to what they have seen.

Aesthetics Matter as Much as Usability

Many web designers like to refer to themselves as "user-centric designers." This term has emphasised usability testing and rightly so. However, usability testing is far from the only form of testing with users.

I would argue that it is just as important to get the aesthetics and brand values of your website right, as it is the usability. A site can be incredibly usable and yet utterly forgettable. It is essential that we test both areas.