The Boagworld Show is back for a season dedicated to making you a better user interface designer. In this first episode we ask a basic question, how do you measure success?
Links Mentioned in the Show
- Task Performance Indicator: A Management Metric for Customer Experience
- The System Usability Scale
- What is Success? How To Define Key Performance Indicators
- Choosing the Right Metrics for User Experience
- Jon Darke, Every Interaction
Paul: The Boagworld Show is back for a season dedicated to making you a better user interface designer. In this first episode, we ask a basic question: How do you measure success? This season of the podcast is sponsored by Balsamiq and Fullstory.
Paul: Yay! That is a good thing. While it's a good thing, I don't know, I quite enjoyed the break. How about you, Marcus? Did you enjoy your break?
Marcus: We've launched a new site today, we're launching another one next week, so I did have a week off, which is great, and I love that, but I didn't have the month-and-a-half off that you had, Paul.
Paul: I took forever off. I don't think I'm ever going to be back. I just faffing around a lot. Anyway, we're being joined this week by Jon, Jon Darke. Jon. How did you-
Paul: Did you have a nice break over? It seems silly to talk about the Christmas period now. By the time this comes out, we're nearing the end of blooming … January.
Marcus: What's the first month of the year called, Paul?
Paul: I don't know. Did you have any kind of break? Because you run your own business, so it's not always easy to do that, is it?
Jon: It isn't, but we've always had the idea of shutting down at Christmas, so we did it again. We had a nice week-long break, but it already seems like a very long time ago.
Marcus: I know the feeling. I'm going on holiday, though, Paul, in two …
Paul: Oh, shut up. We don't care.
Marcus: … three weeks' time.
Paul: We don't care.
Marcus: Yeah, but I might not come back.
Paul: Oh, right.
Marcus: Or I may come back in a box. I'm going skiing. The last time I went skiing, I was 17 years old.
Marcus: We had-
Paul: Jon, would you like to be my new co-host?
Jon: I hear there might be an opening coming up.
Paul: Yeah, I think you might be right. My word.
Marcus: I've been doing actual keep-fit and stuff over the last month or so.
Jon: Oh, no, you're taking this seriously.
Marcus: Well, I had to. Otherwise, I probably would only manage one-day skiing. I could do it when I was 17 years old, and I've got no idea now, but obviously I was using Victorian equipment back in those days, so I suppose it's a bit different now.
Jon: [Crosstalk 00:02:35] some gravity do most of the work for you.
Marcus: That's the plan, and I just have to slow myself down on the really steep bits, or maybe just avoid the really steep bits. That's the way to do it, isn't it?
Paul: I cannot imagine anything worse than going skiing. I would kill myself, without a doubt. Simple as that.
Marcus: It's very beautiful. Really, really beautiful.
Paul: Oh, yeah. No, don't get me wrong. Going places in the snow, fine. Skiing, not so much. There you go. Jon, have you ever been skiing?
Jon: We did it once on a company trip, yes, but I spent half the trip learning on the kiddie slopes, but it's good fun. Quite terrifying, though, is the first time, when doing it first as an adult.
Paul: Was that your business or was that when you worked for somebody else? Because if so, your business was doing really well if you can get company skiing trips out of it.
Jon: Yeah, yeah, it was with Every Interaction. We took everyone away because we had a major development partner we worked with in Prague and we decided to meet up in Austria and do a bit of a thing together and get the teams together, because we worked so closely but we were always so separated, so it was a nice opportunity to get everyone together.
Paul: That sounds wow.
Paul: Yeah. Why doesn't Headscape do that?
Marcus: We are planning, Paul.
Marcus: Nothing quite so grand as that, but we are planning a week away this year, all of us going to go to a farm or something like that and just get away from the grind of the daily tasks and that kind of thing and work on some stuff. That would probably be in the autumn.
Paul: It's not skiing, is it?
Marcus: We might look into it in a bit more detail and work out, actually, we could all go skiing and it would probably be some sort of prize. It depends where we end up going. We'll see.
Paul: These days, Jon, you-
Marcus: A retreat, that's what it's called.
Paul: I've moved on from you, Marcus. I'm bored with you.
Marcus: I'll shut up for a bit. I think, too-
Paul: I want to know a bit about Jon, what he's up to today, because Every Interaction is your agency. What's your role in it?
Jon: I'm a founder and director. It's a relatively small company, so I have to wear many, many hats, but the vast majority of what I do is design direction, I guess, slightly leaning on the UX side of things.
Paul: Sure. Just to give people a little bit of context as we get into the topic today, which works really well that you come from a UI/UX background, because we're talking about UI. That is the topic-
Paul: Yeah, handy. Almost like I planned it. That is the topic for this season. We're really going through … I want to say the "basics" of UI design, but just whatever pops into my head. Today we're going to be looking at, how do you know if you create a good interface, what does success look like.
Paul: We're going to look at topics for understanding your users, we're going to look at establishing some design principles to work with, we're going to look at your workflow as a designer, we're going to get into responsive design a little bit, we're going to look at the future of grid layouts, design systems, testing, persuasive design. We're going to look at the role of developers in interface design and maybe look a little bit about user interface design in the agile world as well. It's going to be a good season, I think.
Paul: Then all of the people that are guests on the show, including dear Jon here, are all people that have come from the Boagworld Slack channel. Basically, if you want to go on the show, you've got to hang out in the Slack channel. That's where the cool kids are these days. They're working web designers. They're not superstars. Or I think Andy Clark's on this season, but he doesn't count as a superstar anymore. It's people that actually do real work, unlike me, which is good. That's why you're on, basically, because I don't do anything anymore.
Paul: That is the plan for this season. Does that meet with your approval, Marcus?
Marcus: Totally, yeah. I love the idea of back-to-basics-type stuff, and user interface design is very much where I spend a lot of my time looking at, or I'm trying to organize it so that our designers can do a good job, so yeah, very much my kind of thing.
Paul: Good stuff! Before we get into that, we're going to look, as I said, today at success criteria and working at how you know whether you've created a good user interface, whether it's actually doing its job and all that kind of stuff. Before we get into that, I want to talk about sponsors, because this season we have two sponsors for the entire season, and I don't think they could've been more appropriate and better sponsors than we've got.
Paul: As I said at the very top of the show, we've got Balsamiq and Fullstory are our two sponsors for this season. Let's talk a little bit about Balsamiq, because Balsamiq have been around forever. They're probably one of the first prototyping tools out there, wireframing tools?
Paul: I had a really interesting conversation. Jon, have you ever used Balsamiq? You must have used Balsamiq at some point. Have you?
Jon: Yeah, absolutely. I've been using it for, oh, maybe nearly a decade.
Paul: You still use it …
Jon: I'm glad they're a sponsor.
Paul: … quite a lot?
Jon: I do. Still today it's my preferred wireframing tool. Everybody these days likes to use Sketch and that, but I prefer to separate the wireframing and design a little bit with the tools. I find if I start wireframing in Sketch, I'd basically end up designing it in black and white.
Jon: What I like about Balsamiq is it feels more like a wireframe. The output is a little bit more rough and ready from a visual perspective because it's not designed to be a precision tool. It's specifically designed to allow you to get the idea down really quick. Even today I can still wireframe in Balsamiq at least twice as fast as I can in anything else.
Paul: I can shut up now. He's pretty much done sponsor shop for me.
Jon: Good sponsor.
Paul: Yeah. It's really funny, because if I'm not amiss, I used to use Balsamiq when it first came along, and then I was like, there are all these other tools starting to come along and Sketch came along and then there were prototyping tools that integrated with Sketch, things like Invision and stuff like that, I stopped using it a little bit.
Paul: When I got talking to them as a sponsor, I was like, "I don't know, should I be recommending them to my audience? Are they really still good?" As I had a chat with them, it was like a light bulb moment. The thing with Balsamiq, other than, I mean, everything you said, Jon, is absolutely spot-on and I totally agree with you, but the thing that they've done is they purposely created a prototyping tool that's really simple, really easy to use, and it's fast, just like you described, and it's not aimed at people like me. It's not aimed at professional UI designers. It's aimed at people like Marcus. No offense, Marcus.
Marcus: Jon still uses it, and he's a designer.
Paul: I think that proves that designers can use it. I'm not saying they can't use it, but their market, the reason that they haven't got into this whole business of making it more and more fancy and integrating with Sketch and all the rest of it is really it's designed for product managers, business owners, consultants, programmers, all of the people that don't use Sketch. I'm used to using fancy prototyping tools. The way they talk about it is it's democratizing the design process, allowing young designers to participate in it, which will no doubt piss off a lot of designers.
Marcus: I talk to Lee about this all the time, because I started off with Axure about five years ago, and it's too faffy. Every time I open it up, I think, oh, I wish I still had Balsamiq. There's my plug for Balsamiq. Then I dive into Axure. It's not that bad. It's the same kind of thing. You can drag little tools onto the interface and move them around, similarly to Balsamiq.
Marcus: What I like about Balsamiq, and I don't know, they might have changed this now, but we still have certain clients that we work with, new clients, who don't get what wireframe is, even after we've explained it in person to their faces. They will get comments coming back about, "We really like the grays," and all this kind of thing.
Marcus: What was right about Balsamiq, and I'm sure it still is the case, is it doesn't look good. It looks like …
Paul: Some sketch!
Marcus: Yeah, it looks like a sketch, and that's what it should look like. This is what I argue with Lee about, because he's merged the whole process of wireframe into design I think too much. There needs to be, as Jon was saying, a kind of, right, these are the wireframes and this is where we're starting to look and feel ideas and look at mocking up pages that I think they should be separate.
Jon: That's the purpose of wireframes, right?
Marcus: Exactly, but it's quite easy if you start often a tool where you can design stuff, as you've already said, Jon, to continue the process and it morphs into this proper design, for want of a better term, which I don't think it should. I think it should-
Paul: Or even worse, that uncanny valley where it looks designed but isn't particularly well-designed because it's not finished. It's not enough like a wireframe for people to go, "Oh, that's a wireframe," but it's not designed enough, either.
Paul: Anyway, try it for yourself. You can get 30 days free trial. Also, if you use the code BALSAMIQBOAG, B-A-L-S-A-M-I-Q-B-O-A-G, one word, capitals, I think, I don't know whether it's to be capitals, but it looks like it has, that would give you three months free to try out their Balsamiq Cloud and really get into it.
Paul: It is very straightforward. You just create an account and you can enter that code alongside the billing information. That should be great. Just go to Balsamiq.com. We'll put a link in the show notes. Try out for yourself. It's a great tool. It's also a great tool for getting clients to articulate their thinking as well, because when they describe stuff, they don't always do a great job. Anyway, we'll get loads more into Balsamiq over the rest of the season.
Paul: Who's that? Who's going to admit that they've got it wrong? I'm going to go with Marcus.
Marcus: Mm-hmm (affirmative). My phone was ringing in the other room, and of course, everything's all joined up these days.
Paul: Oh, yes. I know.
Marcus: I turned my phone off. My phone is off in the other room, and then it's still ringing here. It was an 0-800 number. How annoying is that?
Paul: Yes. Very annoying.
Paul: Okay! Right. Success criteria. It seemed like the logical place to start, if we're going to do a season on user interface design, to start with a basic question of, how do you know whether you've designed a good interface? How do you measure success? Specifically, all the problems that come associated with design of somebody, you produce a design and then people go, "Nah, I don't like that," "That's not what I wanted," "That doesn't make me happy," "Oh, it didn't do what I expected it to do," et cetera, so all that defining success upfront.
Paul: I'm quite interested with you, guy. Marcus, does Headscape still define success criteria at the beginning or has that slacked off since I left?
Marcus: Sometimes. It depends. Those [inaudible 00:15:22] we're going to go on to talk about, measuring success in some cases, it's a hell of a lot easier than it is in others. If you're doing what we used to call a brochure site, a site that is not necessarily …
Marcus: Yeah. Then, you could argue that it's a success if you like it. I'm being obviously deliberately flippant there, but it's much, much harder. Let's say, if the success criteria is to promote the new brand of the organization, there are ways you can measure that, but what I found is when we spent a lot of time in the past writing down KPIs and then creating [inaudible 00:16:06], they often get put in a drawer and forgotten about. From that point of view, there's not as much focus on it maybe as there was maybe five years ago. It depends on the site.
Paul: Oh, you need a slap. You really do. Don't put them … Oh.
Marcus: [Crosstalk 00:16:29]. I'm being honest, Paul!
Paul: You need to refer-
Marcus: No, I don't put them in the drawer; the client does.
Paul: Get them out of the drawer! You've got to follow through-
Marcus: I was trying to be honest in a really, really opposite way of the way we've maybe approached it five, six, seven years ago. Yeah, certainly. I think people, we tend not to do a lot of work from scratch these days. Most of the projects we're doing are rework. There's a lot of measurements, particularly analytical measurements in place, and so therefore there are a lot of KPIs already in place, which is adopting them. We're just trying to improve on something that's already there.
Paul: What about you …
Paul: … Jon? Yeah, go on. You were going to say something. Go on.
Jon: I was just going to add to Marcus's point, really, that I think you've got a point where sometimes these things get put in the drawer, and often it's the client that's doing that, because sometimes the client just isn't necessarily interested. They just want their new website because they think they're all going out of date.
Jon: I think the key there is to, if you can't sell the success criteria into the cloud, you should do it yourself. You should still be taking an initiative and, where possible, measure the things before and after that you think are going to make a difference, so that you have some quantifiable metrics of your own that you can learn from, at the very minimum, and then ideally, you could use that to hopefully convince the client to do more work in the future because if you're looking at these metrics, you can see how you can improve it even further, make it even better.
Paul: I'm very nervous about any client that doesn't want to talk about success metrics, so their success is, "Oh, I want a new pretty site," because that for me is totally unquantifiable. If that is the success for the project, then the result of that is that you just don't know whether you ever succeed or not. You're at the whim of the client. Are you not in that situation?
Marcus: I'm going to have to jump in with a story here. We turned an opportunity to pitch for a piece of work for a very famous chef probably, I don't know, six months ago at the most. Chris, my colleague, went up to visit one of the very famous chef's underlings, a new marketing manager who've worked with us in the past, and was basically told that the audience for the site was the chef, nobody else.
Marcus: She gave me an example of what kind of way he likes to work, by walking Chris around the very posh restaurant in the countryside and showing the way he'd designed each of the rooms, and would regularly change his mind on the décor of the rooms, and each room would have its own theme, and that's the kind of way that he would like to manage his new website, and it was made very clear that this was likely to be a very long contract and one that would be very lucrative, but we would be on famous chef's beck and call, mood swings, mood changes, whenever he wanted.
Paul: At least they were honest.
Marcus: Clearly, and we said, "No, thank you." After thinking about it, a lot, actually, in one of those deep, "Do we want to do something like this?" but we thought we're not big enough to take that on. If we were maybe twice the size we are, then maybe you could allow for that kind of beck and call kind of relationship, because it probably would be quite exciting. We thought it could end up just wrecking other projects, so we thought, nah, that one's not for us. It's just an example of, those kind of projects do exist where the client just says, "I want this beautiful thing."
Paul: I'm fine with that. I'm fine with that if you know that upfront and if you can price the project on a time and materials basis, because the trouble is when you're talking about somebody's whims or you're designing for a wooly set of criteria is that you're never going to know, you can't put any process in place to get you to that end goal because you don't know what the end goal is, so therefore you don't know how long it's going to take and what's going to be involved in it, and that's where you end up losing money.
Paul: That's one of the reasons why I like success criteria is because it allows you to define the work and the direction of the work, but I also agree with Jon that it also allows you to upsell at the end, encourage …
Marcus: [Crosstalk 00:21:13].
Paul: … more investment, isn't it?
Marcus: Mm-hmm (affirmative).
Paul: If you can show that you've succeeded, then they all are willing to spend more money.
Jon: I think even if you haven't succeeded, I mean, even if you do set criteria that you all agree to, maybe you fall short of those, what you can do, it gives you an opportunity to sell in or just do on your own behalf the analysis of what you're looking at and come up with a fix or a second round of work and learn from mistakes, because you can never always with 100% certainly predict you're going to hit these numbers. They're targets, and there are so many unknowns that you can't account for, so you've got to take everything with a pinch of salt, and even if you don't hit them, I think it's just another opportunity to be able to do more work.
Marcus: As an aside, I think with general usage-type stats, I think I'm right in saying that if you've managed to keep things on a par, on a level, then you've done well with the redesign, and that there's usually a dip in-
Paul: Oh, especially at the beginning, yeah, and it's always worth warning the client of that upfront, because otherwise they go into panic and start changing things the minute it's gone live, which is never a good thing.
Paul: It was Daniel Burka who I knew a little bit back in the day when Digg.com was big. Do you remember Digg.com?
Marcus: That's a long time ago.
Paul: I know. He's always said that, basically, he knows the stats and the feedback for about the first two weeks after a big change because everybody throws a wobbly. You see that every time Facebook or Twitter redesign. Suddenly everybody's up in arms. "Oh, they've changed it!" "It's terrible!" "It's awful!" That's what people are like when you change stuff. You always see a downturn to begin with.
Paul: What I'm interested in, Marcus, was what you were saying a minute ago about how, if it's brochure website, it's quite hard to set criteria because, is criteria like increasing brand awareness, but is it like brand awareness you can measure?
Marcus: You can. I think it takes more effort, though. I think you need to-
Paul: Oh, you just can't be bothered. Is that the struggle?
Marcus: Or one's clients can't be bothered. Be careful what I say. It's just more effort to get feedback. If you want actual valuable feedback from your users, that takes a lot more effort than trying to increase analytics in some way. It's just more effort. That's all, really. I'm not saying there's anything wrong with doing it. I'm thinking out loud here.
Marcus: There's also a case of, I'm going to be cynical here, "We want our new site to increase brand awareness" or something along those lines I'm guessing is not as important as, to most people, as "We want more leads." See?
Marcus: That's why it gets the "put in the bottom drawer" syndrome, because I think us and clients, although we must have KPIs, we must be measuring success, that's what you're meant to do, there's probably 30% of them are really important, the other 70% are because they have to do them, a little bit of that.
Paul: That brings me to the "why" approach, that when someone turns around and says, "We want to increase brand awareness," you say, "Why?" and then they'll say, "Because we need a high profile in the market." "Why?" "Because we want to get more new customers." Now you're talking. Now that turns into something that's a bit more measurable, because you can start measuring leads that come through the website or number of people that look at the contact page in order to get the telephone number or whatever else.
Paul: I think there's a couple of things on that. One is that I think sometimes it just takes a bit of imagination to come up with KPIs that you can measure. No KPI is going to be perfect. You might be measuring just the number of people that look at the contact page, or if you're honestly wanting to, it's about brand awareness, the number of times the website is shared, or how long people are spending on the website in terms of attention, how long they're watching a video for if you've got a video on the site, or what their first impressions are, or these kinds of things. I think a lot of the times, it's just a bit of a "can't be arsed" scenario.
Marcus: Yeah. Maybe. I've got it in my head, because we're working with a client at the moment who we've been working with for years and years and years, and we spent a lot of effort, I think you were part of this, Paul, in developing a long set of KPIs for them.
Marcus: I was using the 30-70 example based on them. A 30% off those KPIs, they've kept close eye on, so have we. The other 70% were just periphery, wooly stuff that was unnecessary. That was the point I'm trying to get across there.
Paul: Yeah, no, I agree with that. I totally agree with that. Basically, I think what happens is when you start the client thinking about these things, they start going, "Oh, well, we really ought to consider this," and "We really ought to consider that," and they have more and more stuff in them. That's why I think it's probably worth doing some kind of prioritization exercise to say, "Which of these really matter the most?" and then focus in on the top two or three, but keep the others in mind.
Marcus: Yeah, and also what you were saying there about really drill into, what are we really asking for here? I think that's-
Marcus: Sorry, Jon, carry on.
Jon: I think it could be difficult to actually track a lot of these, because like you say, once you start planning the moment, there's client overheads to keep an eye on these things and to understand. It takes a lot of resource to be able to put all of the useful things out of this mass of data you're collecting and then come up with something, some output from that.
Jon: Again, that's something you can sell in as a sort of maintenance thing. What a lot of people do is saying, "We can keep track of these things for you," and essentially what you're doing is you're feeding a pipeline of future project work for that client. You're telling them, "This is how well it's doing. We spotted away that we could probably improve this. Look at these numbers. If we tried this, it might improve it another 10%."
Jon: It gives you an opportunity to upsell again. It's quite important for I think the people who did the work to have ownership over these things and not just chuck it over the fence at the client and rely on them to analyze all this stuff on their behalf, because they've got their day jobs to get on with and this is not necessarily part of their job description.
Paul: I totally agree with that. I think there's also an element whereby you've got to be a bit careful with KPIs as well. Yes, you want to have a relatively small handful of KPIs. You don't want to be measuring a shitload of stuff that the reasons we've already said, but equally, you don't want to just be measuring one thing, or even multiple things, but all in the same area, because otherwise what can happen is you can skew the experience.
Paul: Let's say, for example, the most important KPI we have is increasing the number of leads. That's what we're going to focus on. That's what leads to those annoying newsletter sign-up boxes. Because, yeah, they do encourage the number of leads, but there is a cost associated to that. If you're only tracking the number of leads, then you don't care about those costs. You're not thinking about the long-term brand impact of that or how it might affect sales over time or whatever else. You've got to have a bit of mix.
Paul: It always makes me think of … I nearly said their name. I suppose it don't really matter because it was blooming ages ago. I must have used this example before. The people that produce-
Marcus: You have. PM software.
Paul: That's it, but other people might not have heard it, so I'm going to tell the story again. Basically you had a sales department and a marketing department that sold project management software. The marketing department were responsible for the website.
Paul: The metric that the marketing department were measured on was the number of leads generated and the sales department were measured on the number of leads converted, but because the marketing department owned the website, they only cared about the number of leads created.
Paul: They set this almost like a log-in firewall on the site which basically said, "If you want to see a demo of our product, you have to provide us with your email address and name." That generated a huge number of leads, which met their KPI, but they were bad-quality leads. The sales department spent hours and hours chasing up a load of bad-quality leads that didn't go anywhere. You can't just look at one thing. You've got to combine multiple things.
Marcus: There's different types of criteria. There's more strategic objectives, which are the business objectives, like wanting to grow traffic or engagement rates or number of people getting to the end of a funnel or something, but then you've got the "relevance to the users" criteria, which is a bit more difficult to get data on perhaps, that might be more achieved through user testing, both before and after.
Marcus: You could get it from analytics and try and draw some conclusions, but it just takes a bit more effort to get the results of these criteria out than the ones that you can just look at a number in Google Analytics and go, "This is bigger than it was before."
Paul: Yeah, and that's the trouble, isn't it, is that we end up tending to favor the easiest analytics or the easiest success criteria or the ones that are easiest to measure. That's why you end up with, "Oh, yeah, we're going to measure the amount of traffic that goes to the site and the dwell time," and those kinds of things because you open up Google Analytics and there it is in front of you.
Paul: Then there are the slightly more complicated ones where you have to create a funnel or something in Google Analytics in order to get that kind of data, and that takes a bit more work. Then, as you say, there are the user-orientated ones, things like task success rate or time to complete task or task performance indicator, all of these different systems that you can use, but they involve actually talking to users and running usability testing and that's a bit like hard work, isn't it, which sucks, but again, that's what we need to be selling in to clients because that's what actually provides the value to build their business.
Paul: If you do that usability testing in order to get your key performance indicator, you don't just get a number at the end of it. You also get shitloads of insights on how to improve that number and how to improve that website, so it's worth doing. You don't get that from analytics. Analytics will show you whether a number has gone up or down, but it won't show you how to influence that number, will it?
Marcus: No. I think analytics, you have to be very careful about ensuring that you're looking at a very wide view, because obviously numbers might go down one month or they might go up twice as high the following month, et cetera, and there's also this, something I've mentioned before on the show is this, should we be aiming for continual growth all the time? All the time, is that realistic? Or that general ticking along nicely, is that a more sensible and realistic view?
Paul: It was really interesting you raise that because that made me think of, do you remember when we worked with Wiltshire Farm Foods?
Marcus: Mm-hmm (affirmative).
Paul: When we first started working with them, it was an e-commerce site, so there were very obvious key performance indicators on that, the site was an absolute mess, so it was easy to increase those key performance indicators.
Paul: There was loads of low-hanging fruit, obvious stuff you could just go in and do. We created an enormous jump in the key performance indicators. We had this ongoing program of work, month in, month out where we were seeing numbers going up and up and it was great.
Paul: Then management started expecting that, but of course, once you've solved all the easy, obvious stuff, then it becomes harder and harder to create the same level of growth, and that's where it can get a bit unrealistic if you're not careful.
Jon: It's about, is it worth being in in order to solve the problem, really, isn't it?
Paul: Mm-hmm (affirmative).
Jon: If it's going to improve things 1%, but that could equal you turning over an extra million quid a year, especially if you're Amazon or someone, but for other people, that could just mean, "Oh, it's not worth doing this project, because this project might cost us 10 grand and it's only going to improve our bottom line by 1."
Paul: Exactly. You're entirely spot-on with that. Of course, the other problem is, with those kinds of situations where you have these expectations about the grower and growees, then it turns into a blame game when you don't make those targets.
Paul: While I'm a great fan of having key performance indicators, I don't think they should be targets. It should be about moving the needle all the time. It doesn't matter whether you're moving it necessarily by 20% or 30% or 40% or whatever, because then you get in situations where somebody doesn't meet the target and marketing blames the web team and the web team blames marketing and it all gets nasty.
Paul: Are we saying we shouldn't bother?
Marcus: It's a funny one, isn't it? No, I think it's important. I lean more towards the value of user testing, and I note on your notes you've got things like top task analysis. I think they're fantastic, and they should lead into, as you've already said, if you're doing user testing, it will guide you on what the right KPIs are. Maybe that's what I was trying to say in my usual waffly way.
Paul: I think it's about striking a balance. The reason I like KPIs is because, too often, the website comes down to whether someone in senior management like it or whether it meets a business need, and the user never gets to look in, while if you've got some kind of formal KPIs that also include some elements of user needs and usability, then it makes sure that the user isn't pushed out of the agenda.
Paul: It just provides a framework within which to work, but they're not perfect. They're not something I think you can obsess too much about because, a) often they can't be measured that accurately, b) you can end up skewing your whole development process because you're just obsessing over improving these metrics. It's like everything, isn't it? It depends, it's a balance, blah blah blah.
Marcus: I think one of the most important things you said, Paul, was that it needs to be holistic and you mustn't focus on just one thing, because you might improve at one area of the site that improves a particular area so much that everything else is forgotten about and you end up losing sales over something, but if you just looked at it wider, had a wider viewpoint, then it would be a lot healthier way of looking at things.
Paul: There's three areas that we're going to look at further reading in just a second, but one of the articles that I'm going to reference in a minute, in fact two of them, mention three criteria that you can use for ensuring a mix of holistic view of your KPIs.
Paul: One is that you should have KPIs around usability, which is fairly obvious. That's things like your top task analysis, task success rate, task performance indicators, et cetera. The other one is engagement, which is things like attention, first impressions, number of shares, that kind of stuff. Then the third is the most obvious area, which is conversion, which is obviously conversion. It's a mix-and-match.
Marcus: Giving a donation, for example, something like that.
Paul: Exactly, yeah. Okay.
Paul: Let's talk about our second sponsor, and then I wanted to share a couple of further reading things before we wrap up. Our second story, as I said, was Fullstory. Fullstory, again, are supporting the entire season and, again, perfect for when you're talking about user interface and, indeed, success criteria and KPIs and stuff, because essentially they offer the next generation of analytics tools, basically, by far the best analytics tool in the market, in my humble opinion.
Paul: What that means is that, let's say you kick off a project, and you're halfway through and you suddenly decide, or you've been running Fullstory on your website for a little while, you've been making improvements to the website, and you suddenly decide, "The KPI we should be tracking now is newsletter signups."
Paul: Now if you were using something like Google Analytics, you would have to go in and add an event handler to the newsletter sign-up form, and that means you can only track from that point, once you've added the event handler onwards, while with Fullstory, you get all the historical data as well for as long as Fullstory has been running on the website, so allows you to be much more flexible in what you're tracking or when you're tracking.
Paul: More than that, it also records sessions, so you can actually watch someone completing these tasks. That gives you insights into why they're doing something or why they're not doing something, which is just awesome. We'll get a lot more into that over the coming weeks.
Paul: No sampling, Fullstory captures every single user session, so it really is incredibly powerful. You can sign up today and get a free month of the pro service for free. No credit card or anything like that. After that month, you can still carry on using it for absolutely free of charge and it will record 1,000 sessions per month for free, which is great for if you want to give it a go in your agency site or somewhere like that where you can just try it out and you're not going to get particularly high levels of traffic. Find out more about it and sign up and go to Fullstory.com/boagworld.
Paul: I was very conscious that we've skipped over a lot in this talk about, well, I think in all of these sessions that we're going to be doing over this season, so I want you to just end each time with a bit of a further reading bit and some next steps, if that makes sense.
Paul: The first article I want to recommend to you goes back to what Marcus, he was talking about task perform … Oh, no, you weren't. You were talking about top task analysis.
Marcus: Top task, yeah.
Paul: Related to top task analysis is another methodology created by the same guy, Gerry McGovern, called task performance indicator. It's a metric that you can start measuring that will show management how well your customer experience is going, whether it's improving, et cetera.
Paul: There's a great A List Apart article on exactly that subject that you can check out. Hang on a second. It's on A List Apart. If you go to AListApart.com and search "task performance indicator," you will find that out and you can check that one out.
Paul: The second article I wanted to mention is one of my own, and is a new article that I've just rewritten, actually. Originally it was published in 2004 and I've just updated it because we're doing this season. It's "What is Success: How to Define Your Key Performance Indicators," which is much more of a step-by-step guide about why key performance indicators is so important and how to go about defining what they are. That one hopefully will help. There will be a link in the show notes to that, but you can also find it by going to BoagWorld.com/digital-strategy/success-criteria.
Paul: Then the very last one I wanted to mention is a really good article on the UXmatters website, which is called "Choosing the Right Metrics for User Experience," which has got some really great metrics on tracking user experience that you can roll into your key performance indicators. Again, if you go to UXmatters and search on "Choosing the Right Metrics," you'll get back a really good article on that, or just go to the show notes.
Paul: Any other articles that anybody's got spring to mind? Jon, you haven't got any? Marcus?
Marcus: Talking about this idea of looking at things holistically, I wrote an article, oh, three yonks ago now just about some bad experiences that underlined the need for that. I probably mentioned it before on the show, my experience with ao.com, the white goods-
Paul: Oh, yeah.
Marcus: Where you've got one department just messing up the whole experience in one tiny little way, but it meant that I didn't ever want to go back there ever again. You've just got to make sure that these things, whatever it is you're doing, it might be testing, it might be trying to define success, what we're talking about today, just make sure you're going at it from all sorts of different angles. That's really important.
Jon: I didn't bring any articles to the table, but one parting word of wisdom I'd say is, make sure you measure before, obviously. I think that's something that a lot of people forget that you do need a baseline for comparison. Make sure you have some stats, and ideally not just a single reading from a random moment in time, that you've got a decent amount of things that you can exclude any weird goings-on and get a bit of a standardized baseline.
Jon: Everyone loves a good stat, so gather them regardless of whether you need them or not. In particular, running an agency, it's good for us to use them for our case studies, so we get to say, "This is what improved after doing this piece of work. These are the things that changed and this is by how much."
Jon: That works really well either if you're an agency to do your case studies or even if you're working in-house, maybe you could use that in your portfolios the next time you come to show people what you've done at your last job. You can say how what you did made an impact, and that's really important to communicate to people.
Paul: It's also good if you can tie that stat to a monetary value, which isn't always easy, I admit, but let's say, for example, you were trying to increase the number of leads that came by the website and you were able to increase it by X number.
Paul: If the company can tell you on average how many leads they convert, and if they can tell you the value of those leads to them as a company, then you can work out how much each lead is worth and put monetary value on the work you've done, which is always good to do if you can get away with it and always impresses people when you can talk about that.
Paul: In terms of next steps, I would recommend that when you start working with a client or even if you work in-house, get a hold of their company strategy, because that will often lay out some broad organizational goals, and that will enable you maybe to identify some areas that are particularly important to the business.
Paul: Then you can basically ask yourself the question, "How could the website help to achieve these organizational goals?" and then you can take those ideas and look at turning them into key performance indicators based on the strategy, but try and get to sign off an agreement on those performance indicators before you start. It makes life a lot easier.
Paul: Jon, can I ask you a question? Where can people find out a little bit more about you and maybe follow what you do, et cetera?
Jon: You can find me on Twitter. I'll be @darkejon, D-A-R-K-E-J-O-N. If you want to follow what we're doing at Every Interaction, that's over at EveryInteraction.com.
Paul: Cool! Okay. Marcus, do you have a joke for us?
Marcus: I do, Paul from the Slack Channel posted a load of Tim Vines ones, so I dig into those because I love him. "I had a dream last night that I was cutting carrots with the Grim Reaper. I was dicing with death."
Paul: I like that. I like that. That's a good bit. Oh, good. We've got a few more like that coming up, have we?
Marcus: Oh, yes, yes. Now I'll just have to remember which ones I've said and what kind of thing, tough things.
Paul: You need to-
Jon: I like this crowdsourcing approach to the jokes. It works well.
Marcus: It does, although I get alarmed every now and then. It's like, there's been nothing in the channel, and then suddenly there's a massive splurge, so I'm right happy again.
Paul: I love it. Talking of Slack, please join us in the Slack channel. It'd be great to discuss some of the stuff that'll come up in this season at the podcast, as we do every now and again in the channel. I'd love to get your thoughts on it, because then what, if you post your thoughts on it, then I can bring those back up into the podcast, if that makes sense.
Paul: If you've got any thoughts about success criteria and key performance indicators, then go along to BoagWorld.com/slacking to join up there. You can join the conversation about various bits and bobs. Then next week, we're going to be talking about how to understand native users, so join us for that. We can talk about that in the Slack channel as well. For now, thank you for listening. Thank you, Jon, for joining us, and goodbye.