Testing, Testing, Testing.

Paul Boag

This week on the Boagworld Show, we give some tips on how to run successful usability testing and how best to introduce multivariate testing.

This week’s show is sponsored by Balsamiq and Fullstory.


Paul Boag: This week on the Boagworld show we give you some tips on how to run successful usability testing and how best to introduce multi-variant testing. This week's show is sponsored by Balsamiq and FullStory.

Hello and welcome to the Boagworld Show the podcast about all aspects of conversion rate of optimization user experience and digital transformation. My name is Paul Boag. I don't have time to introduce Marcus Lillington because really this is just interrupting me wanting to play Red Dead Redemption. Alright. Is it any good. It's freaking amazing. I've taken the whole week off.

Marcus Lillington: No, it's not. It's rubbish.

Paul Boag: It is incredible. It is the evolution of gaming. It is just such a pleasure to play I tell you. That's it. That's all I've been doing this week. Well, since I've got back from Estonia. Which by the way, Estonia is an amazing place. Go to Estonia Marcus. Do it.

Marcus Lillington: I think I'd like Estonia very much. I've got an image of it being a bit of a party place but with lots of history.

Paul Boag: Yeah. I mean so I went to Tallinn, right? Best food, seriously, better than the food I had the week before in New York. Unbelievably good food. That was the first … not that it's all about the food but it is mainly about the food.

Marcus Lillington: Yes. I'd second that.

Paul Boag: I saw some history and shit but the food was amazing.

Marcus Lillington: We went to a museum and there was this priceless works of art but the steak we had after was amazing.

Paul Boag: Exactly. You understand the priorities Marcus.

Marcus Lillington: That reminds me. Can you remember when you and I ended up having an afternoon free in Rome and we'd never been to the Vatican so we ended up getting a taxi and it was like, "We've got like two hours or three hours." That kind of thing. Anyway, various funny things happened. We were jumped on immediately as we got out the taxi and I thought we were gonna be murdered as we were walking behind-

Paul Boag: That felt very dodgy.

Marcus Lillington: Just come with me 'cause we've got to go to the office where I'll hand you on to the other person.

Paul Boag: Down this dark alley.

Marcus Lillington: Anyway-

Paul Boag: We didn't die.

Marcus Lillington: No, we didn't die but what we were talking about reminds me the fact is when you were walking around … I think it was about a tenth of the museum and it was like, "And here is this 2,000 year old marble statue in perfect condition. If it had a monetary value, it would be the same as the sun and there's another marble statue in perfect condition da-da-da-da." Yeah, you just got a bit blaze.

Paul Boag: You did. You got to the point of like, "Yeah, yeah. Get it over with." You became punch drunk with decadence and history and wonder but no, I really liked Tallinn a lot. All I knew about Estonia before I went was that that's where Pipedrive are based who is a web app that I use and it's got quite a big internet community. Gotta say loved it, really loved it. Really good city, very interesting but not as interesting as Red Dead which is much better.

Marcus Lillington: Why? What's so good about it? Come on, tell me. I'm not convinced. How much does it cost for starters?

Paul Boag: It's about 50 quid but there's well over 100 hours worth of game play in it. The main story line is 60 hours and I'm not even part way through. It's a really engaging story, you get to know the people in your camp, in your outlaw gang very well. You discover a little stuff about them even the side quests are really fun and entertaining and enjoyable but it's the richness of the world that blows you away. It's the attention to detail that is … it's just staggering. You can go into all kinds of ridiculous details-

Marcus Lillington: I think you mentioned the cow's testicles didn't you? Or the horses' testicles.

Paul Boag: The horses' testicles, yeah but you skin all the animals accurately, you have to clean your guns and it's got this … it's quite a slow paced game. You have these moments of intense violence and action but then in between that it's a lot of riding around, enjoying the world and it's just stunning, it's just absolutely stunning. I can't stop playing it. This podcast is a massive destruction.

Marcus Lillington: Okay. Well, I recon, should we aim for the shortest podcast we've ever done?

Paul Boag: It's said that I've got a bit of a problem that is driving me nuts and I need you to make a decision for me Marcus.

Marcus Lillington: Okay. inaudible 00:05:06 it will be the wrong one but hey.

Paul Boag: Almost certainly but you see this is my problem, right? It's the old I need a lightweight laptop to carry me around with, to go to conferences and do that kind of stuff and I was hoping the new MacBook Air was gonna be the magical answer for me but actually it would be a good all round computer for most people but I've got my iMac that I sit using at my desk and I need just something lighter weight. I'm either looking at the MacBook which is obviously even lighter and even smaller, the little tiny one, or the new iPad Pro.

Now, I can't make up my mind because on one hand the iPad pro is half the weight, it's 200 quid cheaper, its smaller form factor, it's all those things but it's an iPad Pro. It's not as good for actually doing more complex graphic type stuff. Most of the time I'm just editing a keynote file, doing some emails or writing but I just don't know, I don't know which to get. They really messed up their product line up Apple have. It's all over the place now. Used to know what you were supposed to buy, I liked it when Steve Jobs told me what I had to buy. Now, they're giving you choice and I'm just confused.

Marcus Lillington: I agree. I feel like I'm being … They deliberately won't make the one I need. You know what I mean? I've got the MacBook Pro now, the 13 inch, it's too heavy, the battery runs out on it too quickly. My old Air was the best machine that has ever been made.

Paul Boag: Why don't you get the new Air? Because that would be a good one for you, wouldn't it?

Marcus Lillington: I don't need one but I will. When the time comes around I will definitely go back to an Air because this is just pointless. It's probably better … When we're doing things like this like I'm on video and I'm recording and that kind of thing then yes, alright, this was the better computer but for the majority of the time, it's way better but I will help you with your decision Paul. Firstly, you bought and iPad Pro and you sold it so not good. They're too big as a thing to carry-

Paul Boag: No, I wouldn't get the massive one, I wouldn't get the big one, I'd get the smaller of the two.

Marcus Lillington: Oh yeah, that's the same as a normal iPad, isn't it?

Paul Boag: Yeah, normal iPad size. Now you think inaudible 00:07:34.

Marcus Lillington: If you sat down in an airport, because that's when you want it, isn't it? If you're sobbing on a seat at an airport and you've gotta get that sort of funny seat on keyboard working on you lap, it's never gonna work.

Paul Boag: They're all better. They're supposed to be better the new one but I do tend to agree. I think I need a … and it's really nice, the little iMac even though everybody is saying why would anybody buy that now and not the MacBook Air which I can kind of … That's true.

Marcus Lillington: Is there a little MacBook Air? Is there like an 11 inch one like there used to be?

Paul Boag: No. You've got the 13 inch MacBook Air and then an 11 inch MacBook and it's like I honestly don't … in some ways it's silly buying the 12 inch MacBook because it's actually more expensive than the MacBook Air bottom of the line but it's got … it's such a mess. I can't believe how messy their lineup is. Basic talking about conversion rate optimization, the one thing you don't wanna do is lead people to choice paralysis and that's where I'm at at the moment. I honestly don't know what to buy and I think most people feel like that whatever level they are within Apple's lineup.

Marcus Lillington: This is another major point for them as far as their sales process or how they feel the angle they should be taking is they've gone almost twice the price. We're not all as rich as you Paul but-

Paul Boag: I don't look at spec but I understand that normal people do.

Marcus Lillington: Yes, exactly. You can spec a 15 inch MacBook Pro up to about six grand now which is just bananas. That is insane. Put Lee, because you know what Lee's like who's been on the show a number of times, was so incensed that he went out and bought a Windows machine and then charged it to the company but he's now gone back to Windows because it's ridiculous crosstalk 00:09:38.

Paul Boag: I am getting pretty close to that myself because I look at something like the Microsoft Surface which gives you all the benefits of an iPad Pro with a proper operating system on it.

Marcus Lillington: True but it's still too heavy. We've got on here and it's great. To use as a computer, a lovely thing but to sit on your lap thing, it's a big chunk.

Paul Boag: Oh, is it? Fair enough. Anyway, Marcus, what's your thought for the day?

Marcus Lillington: Right, my thought for the day isn't my thought for the day, it's somebody else's thought for the day.

Paul Boag: You're now telepathic are you? You're channeling other people's thoughts.

Marcus Lillington: Basically, I asked Dan a couple of weeks ago, I said I've done all this stuff about the Headscape website and I've only got two more shows to go left on the series and I've said nothing about any of the backend development that we did. Can you just give me a few bullet points? He wrote an entire blog post on it.

Paul Boag: Excellent. That's good on Dan.

Marcus Lillington: There you go. Link on the show notes. That is the latest blog post on the Headscape site which I will just quickly go through which might sound a bit weird me being him at times but I'll try and translate it into me talking as if it was me talking about that.

Paul Boag: We could've got him on the show, couldn't we? That would've probably been more crosstalk 00:10:52.

Marcus Lillington: He's actually out there but I thought of it like 10 minutes before it was time to go and I think Dan would've probably hit me with something if I said do you wanna be on the show at that point, pretty close to it.

Paul Boag: Yeah. He's not a great fan of that, is he? No, this is interesting because this is about Gutenberg, is it?

Marcus Lillington: It is, yes.

Paul Boag: Now this is something I've been stoically avoiding. I'm interested in this.

Marcus Lillington: Yeah. I have certain opinions about how the usability of it an all that kind of thing but mostly I just said, you did the majority of the backend work on the new Headscape site so tell me all about it. The big thing was experimenting with Gutenberg editor to see what it was like. I'm gonna read this verbatim. This is the important disclaimer at the top of the blog post. I was reluctant to provide any thoughts in writing because for reasons that should become quickly evident, any opinion I express now, will almost certainly be out of date by the time I finish writing it.

Paul Boag: Know that feeling.

Marcus Lillington: Yeah, exactly. It was particularly the case for Gutenberg at the time anyway because it's about to become part of WordPress which I think maybe I should say what it is but everyone listening to this is gonna know what it is, aren't they? I don't know.

Paul Boag: I would've thought so. Gutenberg is a new WYSIWYG kind of build your own page editor within WordPress very much like Squarespace and that kind of thing but more basic.

Marcus Lillington: inaudible 00:12:16 but when we did the Headscape site, it wasn't part of WordPress. I don't think it still is. It's very crosstalk 00:12:25.

Paul Boag: No, it's going to be at any minute now.

Marcus Lillington: Yes, anyway, we've recently redesigned our site. The goal of doing that was three-fold. One was to update the design, making some performance improvements, it's three-fold from a technical point of view. The second one was to decouple the site from the Boagworld site so Paul could play with the site without breaking the Headscape site and thirdly to experiment with Gutenberg to see how we got on with it.

Just to quickly look the first two of those points, one, the simpler thing we did was to … that had a really big impact on performance, I say we in the sense of Dan and Ian, was to basically lazy load a lot of the content including the huge image heavy work listing page but we also set up … we're using a caching tool app thing called Varnish which gives you great performance gain. That was quite easy things that we did there. Apparently, according to Dan, the decoupling from Boagworld was super simple.

Paul Boag: Oh, good.

Marcus Lillington: He was impressed at how well the build in WordPress import/export tool worked and basically took all the content off and started again fresh with the new site.

Paul Boag: But I tell you what, even the … because we had a multi-site installation of WordPress which is why they were joined together. Even after you'd done that and we turned off the multi-site on the original installation 'cause you set up a new one for Headscape, even that was pretty painless. No major issues, we were all a bit terrified it was going to break but it was fine so.

Marcus Lillington: I'm doing an explanation of Dan's post here but one of the angles that we … We liked WordPress a lot. It works. The documentation I'm told is really good but obviously with this new thing that they'd said, this is how it's gonna be, that was changing so often, it was quite a tricky process getting it up and running so I'll just kind of go through each of Dan's points. Getting used to Gutenberg and working out what we could and couldn't do and should and shouldn't do was the major part of the technical aspect of the project.

He says that he's using it as the reason … He's using Gutenberg as the scapegoat for anything that might not mean that the site isn't as perfect as it possibly could be.

Paul Boag: Oh, I see. Right. Yeah. inaudible 00:15:05.

Marcus Lillington: It's actually fair because, lovely quote. Building with Gutenberg in any way at the moment feels a lot like trying to build a house of cards in a light wind while someone imposingly stands behind you revving up a leaf blower every so often just for the sake of it. He says he doesn't like to work like that.

Paul Boag: No shit.

Marcus Lillington: To give you my experience of this is we'd set up something because we set up styling on a block because it's all in blocks. You divide up your content into tech blocks, image blocks, all that kind of thing and you can kind of move them around, do stuff with them which I initially found really quite fiddly but it took me about two hours and then it was really easy. One thing to mention on that is Gutenberg, I think is a very easy editor to use but that's just my front-end user editor view point of it but what kept happening is I'd start working on it the next day and all the captions had moved from left aligned to center for example. That would just be one thing. I said, what's going on there. Dan would go, "No idea."

Then he'd come back and say, "Oh, they've just changed Gutenberg." Full stop. There were so many updates happening all the time, we were working on this thing that would change the settings and the things that we thought were nailed down. That was hell of a particularly hard thing. I'm jumping all over the place. I just need to catch up to where I've got to. Basically Dan's saying he's glad we used Gutenberg to get a feel for it from a content editor point of view and from a developer and themer perspective and he's pleased with what we've done together that the world sees but he said it still needs some polishing from a backend point of view basically because we were dealing with this thing that just changed constantly.

He said it was also good because we've … on the bigger project we've done over the last few years, they've all been Drupal. We've done some smaller WordPress stuff but doing this project gave us the insights into being more confident and working with WordPress effectively and we are now kicking off a new WordPress site for a client that we six months ago would've definitely done in Drupal. It was certainly a worthwhile thing to do.

The lesson that we've learnt I guess that I'm gonna finish on here is that there's still a major debate on whether we should be managing entire page layouts in one WYSIWIG because that's what we're talking about here and because that's how it works, it's great on account of a content page, a blog page, brilliant for that sort of thing but if you want more of like maybe a landing page where you've got content left and right and different widths and all that kind of stuff, it's as Dan puts it, don't expect miracles just yet because it's really hard to do that kind of thing. That's our thoughts on Gutenberg.

Paul Boag: Yeah. I mean I've struggled with it a little bit. I had a bit of a look at it and it depends on the view port you're looking at it from because if you're looking at it from a developer point of view, it makes me very nervous because if they make an update to WordPress it could easily break what I'm doing. Also, what's the quality of the code that it's producing. I mean I haven't looked at it a huge amount but from a content creator point of view, it's like yay! This has got a lot of potential especially as you can create things like custom blocks where you can add in your own stuff into it.

Marcus Lillington: I soon found out they were sort of saying, "I'd like this piece to be like … " We got a couple of customy pages on the site which are more like … yeah, they're one offs. And I said, "Oh, can I do this, can I do that?" It was like, that's a bit trickier. Let's not bother with Gutenberg at all for those pages and then it's like, well, if you're not gonna do that, what's the point in it? Maybe use it for some things and not others but it's just early days. I know that it's got some majorly, major bad press and I suppose what we're saying here is it's not as bad as all that but it was quite a challenge.

Paul Boag: I think it's like anything when they worked very much in the open with it and they put it out there very early and I think it's got a lot better more recently. I don't think I'm gonna be … when they roll out Gutenberg to begin with, I think I will go back to the original editor because I tend to write stuff in html anyway but maybe when I next redesign, I would work with custom blocks from scratch. I don't know. I'm interested. It's sort of an interest to me but I'm not rushing into it. Okay that's cool. Thank you very much for that Marcus.

Marcus Lillington: I'm typing stuff and I'm gonna say it. Andrew in the chatroom says, what would you say are the major benefits of WordPress are over Drupal or is that another podcast in and of itself? Yes, I can't answer that now quickly but it's given me some thoughts for thoughts of the day for the next series. Thank you Andrew.

Paul Boag: Okay. Let's talk about Balsamiq very quickly. This month I am sitting down and going to be recording four videos, four initial impression videos of various people's sites where I'm gonna review four sites and I'm gonna give my feedback on them of how they could be improved, how they could be made better, what user experience enhancements you could do, all that kind of good stuff and I could do that because of the lovely guys at Balsamiq that are making that possible and they're currently collecting submissions for sites that I could review. If you have a site that you'd like me to check out, you can go to balsamiq.com/learn/boagreview and if you submit your website there, I will have a look at it and maybe yours will be the one that I choose to review and give some feedback from.

If however you haven't got a website in place or you're looking at redesigning or something like that and you could just do with some help about knowing how to approach the redesign, maybe some help doing some wireframing and that kind of thing. Balsamiq actually offer you the opportunity to get personal help on doing that wireframing and get someone from Balsamiq to work with you doing that absolutely free of charge which is pretty awesome. You can apply by emailing them at userresearch@balsamiq.com. I'm really quite impressed at these guys, how much they're doing just giving away and encouraging, trying to improve the quality of the sites that are out there.

I met some guys from Balsamiq recently. I can't remember where that was. Was that in New York? Anyway, they're so lovely. They're lovely people. Anyway, that's Balsamiq. In this penultimate episode in the season, we're looking at the subject of testing and how important testing is.

Marcus Lillington: On that subject Paul, sorry to interrupt but-

Paul Boag: No, it's fine.

Marcus Lillington: You said to me inaudible 00:22:45 so we were going up to Christmas. I've got my notes for which I haven't written yet. I need to come up with a Christmas thought for the day on Thursday, on the eighth of November.

Paul Boag: No, that is incorrect.

Marcus Lillington: It was all lies.

Paul Boag: No, we record the last one on the eighth, although it doesn't go out on the eighth, it goes out the week after but we're gonna do a Christmas … You remember we're gonna do a virtual party just in here, right?

Marcus Lillington: Oh, yeah.

Paul Boag: In here being crowd cast. Only for live … we won't put that out as a podcast because it will be irrelevant crap but I will announce the details of that on Thursday's show … the next show if I don't forget. We're gonna come up with something. Is it alright if I carry on now?

Marcus Lillington: I'll probably think of something. Obviously, I can't think anything right now. You'll have to start talking about something and then I'll want to interrupt you.

Paul Boag: If you feel the need to interrupt me go ahead but I think that might be just the end of the show because you know, Red Dead. I'm just saying.

Marcus Lillington: You wanna go and ride your horse, don't you?

Paul Boag: I want to ride on my horse. More importantly, I want to stand behind my horse waiting for the exciting moment when it takes a crap. Which it does at random intervals and it's like yay! Sorry. Anyway, grown up professional podcast. Testing. We wanna talk about testing. Now, we touched a little bit about testing in previous episodes but I wanna look specifically at usability testing and how to make that a success and then later on we're gonna look at multi variant testing but let's look at usability testing first.

First and foremost, most important thing that I can say is you should be doing usability testing but I do completely understand that it's really quite hard to do. It's easy to say it, it's difficult to make it happen.

Marcus Lillington: What do you mean?

Paul Boag: Yeah, exactly. It could be dead, dead simple. It doesn't need to be complicated, it doesn't need to be onerous but it can sound so intimidating. The most important thing actually is not just to test once but to test multiple times throughout a project but just keep it really, really lightweight because the thing is, the more you test, the more issues you're gonna find but also the more you test means the earlier you're testing and the earlier you're testing, the better it is. If you test right at the end of a freaking project, it's too late to change anything.

You're not gonna wanna change stuff at that late date and it's gonna be hard to change. You wanna test really early on. You can test actually anything. You could even test a sketch. For example, you could do a first click test. You could take a design comb 00:25:39 right? And you could literally ask people if you were trying to do dot, dot, dot, where would click first, right? That's it. That's all you need to ask them.

Marcus Lillington: You can ask your colleague in the desk next to you, alright, they are a web professional, you can ask your wife.

Paul Boag: Your wife is a good one. Your friends, children, random people on the street. Preferably somebody outside of the company I say because they know the job and all that kind of thing but yeah, pretty much anybody. Just okay, yes it's better if it's demographically perfect people and all the rest of it but testing with somebody is better than nothing and something like a first click test literally takes seconds. If someone gets their first click right, the chances are they're gonna end up being able to complete the task. Statistically it's something like 85% of people complete the first … correctly end up being able to do their task.

There are also services. Usability hub is a really great service, right? Usability hub, they'll even recruit people for a first click test for you, okay? Yeah, of course you have to pay a bit if you do that but you could just use your social media people as well, the people you've got that follow you on social media. Just go ahead, create a little test and shove it on social media. First click test, get results straight away, perfect. Right. Start early. Very, very important. Test often. Very important. Don't worry about getting the perfect participant. Anybody will do.

The only exception is if you're talking about an audience like children or those with disabilities or the very elderly because they have specific cognitive or physical disabilities that can change the way they interact but most people respond in a fairly similar way to usability hurdles of how to use a website. You don't-

Marcus Lillington: Yeah, it's more sort of if you're asking questions about does a design reflect the character of an institution then you would need to have representative testers but yeah, if it's like can you see the thing, then yeah human will normally do.

Paul Boag: Exactly, yeah. Can you see the thing. That's a brilliant way of putting it. You don't need to test with a lot of people as well, right? Jacob Nelson famously talks about anything over about five or six people and actually you get to mention returns but even if you just test with three people, right? That allows you to get something like 75% of issues, okay? You think about that, three people 40 minutes each with 20 minutes in between. It's an hour in total. That's three people. You can do three people in the morning, right? It only needs to take a morning then you can sit down over lunch and decide what you wanna do. Very straight forward.

You don't need any fancy facilities either or clever software. All you need is a laptop and something like go to webinar or Skype or something like that, Zoom where you can record a session so you can record the screen and you can record what people are doing on camera. You can have someone sitting in another room watching that session so that people can watch it live. Simple as that. There are other tools out there. There's a great tool that I really love using that makes it a little bit easier called lookback.io. I say it's lookback.io, I always wonder whether I get … No, I got it right. Yeah, lookback.io which is a really useful tool for doing both in person and remote testing as well because often it's hard to get people in a room.

By using something like Skype or Zoom, you can talk to people anywhere. Again, no fancy facilities required.

Marcus Lillington: I'm being an actual punter here because I'm doing a lot of testing next week. What are the advantages of lookback.io over I don't know, something like go to meeting?

Paul Boag: It will organize all of your videos that it records and allow you to easily edit them down, right?

Marcus Lillington: Yeah.

Paul Boag: It will also allow you to do both live moderated testing and self testing so you can send a URL out to a person and they just go ahead and do it by themselves. It's just got a really nice interface, it's super simple to use. It's designed specifically for this job. It's like the difference in Photoshop and Sketch. Photoshop is very powerful but it's more generic while Sketch is designed for web designers, you know?

Marcus Lillington: Yup. Cool. I'll have a proper look at that.

Paul Boag: Have a look because you can get a free trial of it. They're not a sponsor although they should be.

Marcus Lillington: He said bitterly.

Paul Boag: He said bitterly. I don't think I've contacted them actually. Few tips to get started with usability testing. First of all, do some preparation beforehand. It doesn't need to be anything great but it's worth having a bit of a script together especially at that introductory bit where you're talking about you've got to remind people to sign a disclosure form or you wanna reassure them that you're not testing them. There are some basic bits and pieces that you wanna kind of get in place and then also you want a clear idea of what tasks you want them to do.

I normally aim for about three tasks over a 40 minute session. About 10 minutes a task and then introduction and then conclusion but you can … it depends on how long the task is really.

Marcus Lillington: Yeah, that's not many. crosstalk 00:31:49.

Paul Boag: No. Do you tend to do more?

Marcus Lillington: Yeah, but I'll have to say I was thinking it depends on what the task is, doesn't it?

Paul Boag: It does totally depend on the task. I think it's just something really simple.

Marcus Lillington: My biggest problem, I'm probably jumping ahead here but my biggest problem with usability testing is I'm a terrible leader of the test. I have to sit on my hands and look the other way.

Paul Boag: I know but it's so easy. This is why I don't do it very much anymore if I'm honest because I'm a very bad facilitator.

Marcus Lillington: I'm good at making people feel comfortable all that stuff but it's sort of I just can't stop myself from helping them.

Paul Boag: Yeah, and of course they'll ask a question as well and you'll answer it and the realize … they're not gonna have someone sit in front of them when they're doing it at home. Also another way you could influence people very easily is by the way you word questions. If you say things like, I don't know. Find out more about the company, right? They'll look for the word about and you've led them so you've gotta kind of word things very neutrally and that kind of stuff which-

Marcus Lillington: It's virtually impossible not to lead somebody, just the level of how much you do it is what you need to think about.

Paul Boag: Yeah. Also, it's being … and this is where I'm terrible, it's being comfortable with silences and I want to feel silences all the time but sometimes you've gotta let them do it, won't you? And if you do feel a silence you've gotta be saying things like, so what are you thinking? What's going on in your head?

Marcus Lillington: That's the most important question, yeah, show me or tell me, that kind of stuff, what are you thinking? They're the important questions.

Paul Boag: The other one that I do quite a lot is don't actually click on anything but tell me what you'd click on and then I say, "What do you expect to happen when you click on that?" That's a good question to ask.

Marcus Lillington: Especially for labels.

Paul Boag: Exactly, yeah. What do you expect to see? Yeah. Try not to influence people. The whole thing about making participants comfortable is a really good point because people do feel very uncomfortable and then of course if the feel uncomfortable, they behave differently. Make them relax, make it clear you're not testing them, right? You're testing the website, lie through the teeth and pretend you weren't involved in the creation of the website in any way because then they feel happy to slag it off in front of you because obviously if you're paying them money, you're giving them a gift and that is something you should be doing, then they're gonna be nice, right?

You need to make it clear that actually if anything you want a bias the other way, you wanna say to people, "Look, we wanna hear your negative feedback, we wanna hear your problems."

Marcus Lillington: Make up bad shit.

Paul Boag: Yeah. Marcus. Make sure you keep the sessions short, right? Because actually it's quite a tiring thing to do to ask people … I think 40 minutes really is about as far as you can stretch people and also that way, you could test with more people as well which is better than doing long sessions with just one or two people.

Marcus Lillington: Yeah, I agree.

Paul Boag: Always record the sessions. You don't always watch those sessions back necessarily depending on whether you've got someone in the room taking notes with you or someone in another room taking notes but it's very good to be able to edit down afterwards, alright?

Marcus Lillington: Yeah, don't record either video or audio and rely on those as your notes. Horrors, horror of horror. You've basically just wasted a week of your life having to go back through it or. Either you, which is not good but I have done it or preferably a colleague or somebody who works for the client you're working with takes written typed notes into a spreadsheet that you've prepared that is … this is the task, these are the notes to go with that person, all that kind of thing. It's all beautifully laid out and you can kind of go back later and go through … Let's say you're testing 10 people or six people or whatever.

You can just go down through what are all the things I can just pull out because we're really good at doing that with words out of this list of all the stuff that you've got on that particular task that people were doing but you'll always say no to yourself or the tester will say when somebody said something brilliant and that's when you go back in to the video and you pull that out and you show it to people.

Paul Boag: You need to put timestamps. You have those moments you go, "Oh that's interesting. Just make a note of the timestamp." Again, that's why a tool like lookback is really good because I think it helps you do that. I can't remember exactly how but I've got a feeling it did. Keep sessions short, always record them, run them regularly, right? I'm a great believer in … run them every month because you're only talking about three people, just test every month whatever it is you're currently doing especially if you're working in house.

Steve Krug talks about that in his second book Rocket Surgery Made Easy and he actually lays out a plan on how to do all of this kind of stuff including script and everything else you need. If you haven't read rocket surgery made easy, then I recommend that you get it. Really, really short book, it won't take you long to read it so it's definitely worth checking out. The other thing that I did want to say related to that is I have actually written an introductory post that summarizes everything that we've been talking about so far. You might wanna checkout that as well. You can see that by going to boagworld.com/usability/usability-testing and you might find that useful as well. Okay.

Talking of testing, brings us on nicely to Fullstory because there are some challenges with usability testing. One of the biggest challenges is getting access to your audience, right? People to test with. Now, we have said that you can go a long way towards doing that by just using any old random person that isn't involved with the project, but let's be honest, it would be so much better if you could use real people that actually were using your website but when you ask on social media if people would be willing to do it, you tend to attract people that either love or hate you as a company.

Some companies are very precious about you bothering their customers and especially sales people can be very defensive about you talking to them and even when you do get users to usability testing sometimes you can find they don't behave particularly naturally and they behave differently. One thing that could be done to deal with that is session recorders which is what Fullstory does. Session recorder allows you to … you're obviously seeing real customers navigating around a website in real time, they're also making real decisions, right? They really are deciding whether to sign up for your newsletter, they really are deciding whether or not to add something to the shopping basket and so on.

There is a realness to session recorders that you don't get from usability testing and of course as well, you've got access to people very easily because it is the people using your website. Now, that said, session recorders don't replace usability testing but they go very nicely and compliment each other very well. I recommend giving Fullstory a go. You can sign up today and you get a free month of their pro account for free. No need for a credit card and you can continue using them for free up to 1,000 sessions per month. To get all of that, go to fullstory.com/boag.

That is usability testing. The other type of testing I wanted to touch on today was multivariant testing, A/B testing. Has Headscape done a lot of this kind of stuff with clients? You don't really-

Marcus Lillington: Some. Not a lot. Some.

Paul Boag: Yeah, because you don't do a huge amount with e-commerce clients?

Marcus Lillington: No. I found this works … I mentioned labeling earlier. If a client can't decide what's … a good example here, let's say a client uses a term that everyone internally within that organization understands and they want to use that on their website because that's what the right word is but actually there's a word that they consider to be a bit daft but actually is what the rest of the world thinks that thing is. That's what something like A/B testing comes into its own because you can inaudible 00:41:17 Paul let's test this but that's a really good test.

You can just say … all you're doing, let's say it's a label on a navigation item. 50% of the audience goes to one, 50% goes to the other and let's see how many click on it and that is a genuine test. What I found with A/B testing is you go, yes we can do an A/B test on this and then you go what are we actually testing? We want to change the color of this page and it's like but how are we gonna know?

Paul Boag: How do you know? Yeah, which is the better one? Yeah. It's a very interesting one A/B testing. That is one common problem is it's not been very clear about what it is that you're testing. The other is not having enough traffic to get statistically accurate results. Let's back up a second. There's A/B testing and there's multivariant testing. They're all very similar to one another but if we're gonna be picky we'll define the difference.

A/B testing is where you're testing one element of a page. For example, you mentioned label or navigation. That would be A/B testing, right? Multivariant testing is where you're changing multiple elements on a page, right?

Marcus Lillington: Funny, that's not what I thought it meant.

Paul Boag: No, I know. This is where it causes a lot of confusion. A lot of people, and I did for a long time, a lot of people think A/B testing is you've got one variation and multivariant is you have multiple variations but actually it's not according to Google's definitely of it as far as I understand it from what I've read.

Marcus Lillington: I think Google's talking silly stuff.

Paul Boag: I know. I've seen different people say different things, alright? Which is why I wanted to clarify for our purposes. Who knows what the right answer is? I'll have a load of hate either way. You'll get hate mail, I'll get hate mail, both of us are wrong.

Marcus Lillington: I can live with it.

Paul Boag: Yeah, so could I. Especially if we don't actually tell people how they email us although that boat has probably sailed. Right, because it's so confusing, because there's all this kind of stuff, right? It's so confusing, because there's all this kind of stuff, it's very tempting not to do it and not to try but actually getting started … who gives a shit about the terminology, right? It's actually very simple to get started. I recommend getting started just using something called Google Optimize, right?

Google Optimize is free product, integrates well with Google Analytics and all the rest of it but who care about that. It just makes it really easy to get started with usability testing so … not usability, multivariant testing. I wanted to simply test various wording for my newsletter signup on my website. You add a bit of JavaScript to my website and then you log into Google Optimize, you create a test and then it uses what's currently there as a base value, loads up the page, you've got like a WYSIWIG editor, you can go in, you can change the wording, change what's on the button et cetera, just in a WYSIWIG editor, hit save, job done. Simple as that. Really is very, very straight forward.

You can create multiple versions of that if you want to or you can edit multiple elements of the newsletter signup form. I might wanna change the button and the text and the title or just the title or whatever, right? Now, there is limitations to this and the limitations come around in order to be sure that one version is performing over another you need a certain number of conversions, right? A certain number of people that have taken whatever you've defined your action as, right? In my case it will be say submitting that newsletter field. Again, you need to be clearly defining what your testing is, what you're trying to improve.

The problem is that you don't always know whether you've got enough traffic, you're not getting statistically accurately enough results because you've got low levels of traffic so you're not getting enough conversions so only once you've got a certain number of conversions, you know whether it's effective or not. How do you deal with sites that have got lower levels of traffic? One response is to say we just have to be patient, right? Who's got time for that? I don't wanna wait three, four months to know whether something has worked or not. I'm too impatient for that.

One option therefore is to reduce your statistical certainty. You can say as long as I've got over this number of people and there's this big enough difference between the two then that's good enough, right? That's-

Marcus Lillington: Okay, depends on what you're testing though, doesn't it?

Paul Boag: It does. Well, yes and no. Of course it does but actually don't forget if you've got a low level of traffic, the stakes are lower so you can afford to take a few more liberties, that's my point. You wouldn't want to do that if you were Amazon but you wouldn't need to do it if you're Amazon because you're gonna get so much traffic and so much conversion that you could have a high level of statical certainty. The trouble is, is that all the people that write about A/B testing are all the kind of we're big and professional.

They set the bar very high just like usability testing. When you read a lot about usability testing, they make it sound high, hard because they're professionals in that and they're setting the bar high and so you should set the bar high if you can but we can't all so reducing the statistical certainty is one way of getting round it but if you reacted like Marcus, feeling slightly uncomfortable with that idea, there are other alternatives available to you. One, is you should only test the single variation.

If you think about it, if you've got 100% of your audience and you're testing four different versions, that means only 25% of your very limited audience is gonna see each version, right? If you're testing between two versions, 50%, double the number of people are gonna hit that. The fewer versions you test, the quicker you're gonna get to that statistically significant threshold if that makes sense.

Marcus Lillington: Totally.

Paul Boag: You know when you say something and you're not sure whether it's clear? Now, that's the complete opposite of what you'd wanna do if you have a high performing website because in a high performing website if you have five or six variations, then you're gonna much more likely to find the best solution possible and it actually you're more likely to increase your conversion rate higher but we don't all have that luxury. A single variation is good when you've got no traffic, another option is to test close to the point of conversion. What do I mean by that?

Let's say for example you were trying to increase the number of people signing up to your newsletter on your blog, just like I was saying a minute ago. If I start testing or I wonder what blog post titles have the biggest impact on conversion rate, there's quite a big distance in terms of user journey between seeing a blog post title and then going and actually pressing the submit button however, if instead I test the text that's immediately around the newsletter signup button, when people are already thinking about signing up for the news letter, there's a much shorter gap, right?

Marcus Lillington: Yeah.

Paul Boag: That's a good way of increasing basically the conversion rate for the particular element you're trying to test, okay? But what if you wanna test blog post titles? That doesn't help you then, does it? The flip side of that is you test micro-conversions. Because you can't test a long journey like from there to newsletter sign up, instead, you might wanna say, "Okay, well I wanna know which are the best performing blog post titles. The more people that view a blog post, the more likely they are eventually to signup to a newsletter. What I wanna find out is which blog post titles encourage the most click throughs to read the blog post.

Okay, now we can test that. Instead of testing the newsletter signup button, we test the number of click throughs instead because that's closer. Does that make sense?

Marcus Lillington: Totally.

Paul Boag: Good. I managed to get through that because it's quite a confusing thing to explain. The most important thing to say about multivariant testing is if you've got low levels of traffic, absolutely give it a go but just don't rely just on A/B testing. Do a bit of usability testing as well and to be honest, I think that applies on every site. I see a lot of really high end sites with lots of traffic that will rely so heavily on usability testing that they're not really picking up on some of the qualitative data that you get from usability testing.

You've gotta mix your different approaches. Now, I've written a new article on this one as well, boagworld.com/design/split-testing. That's a little bit about that. Hopefully that encourages you to have a little go on those two areas and it will make an enormous difference to your conversion rate because to be honest we're all making it up, we're all guessing what might convert well and what might not and the only way to be sure is to actually test. Marcus, do you have a joke to wrap up us up with?

Marcus Lillington: I do. One second. This is from James Shesbe 00:51:39 Thomas. Why don't communists drink loose leaf tea? Because-

Paul Boag: Why don't communists drink loose leaf tea?

Marcus Lillington: Because property is theft.

Paul Boag: I don't get that one.

Marcus Lillington: Property is theft.

Paul Boag: Property. Ooh.

Marcus Lillington: That's quite good. You're just embarrassed you could get it.

Paul Boag: It's more that I was already thinking about where I wanna go next in Red Dead. crosstalk 00:52:13.

Marcus Lillington: You could do that.

Paul Boag: Alright. On that basis, I'm off back to my holiday and we should talk again next week but until then, thank you very much for listening and good bye.

Thanks to robuart from Shutterstock for allowing me to use this image.