This week on the Boagworld Show we are joined by Brooke Baldwin from Facebook to talk about what it is like to do user research for a worldwide audience.
- More on The Digital Project Manager School.
- More on Pactly.
- Nudge: Improving Decisions About Health, Wealth and Happiness by Richard H. Thaler and Cass R. Sunstein.
- Thinking, Fast and Slow by Daniel Kahneman.
Paul Boag: This week on the Boagworld Show we're joined by Brooke Baldwin from Facebook to talk about what it's like to do user research for a worldwide audience. This week's show is sponsored by Pactly and the Digital Project Manager School.
Hello, and welcome to the Boagworld Show, the podcast about all aspects of digital design, development and strategy. My name is Paul Boag, joining me for this week's show as always it's Marcus Lillington, but we're also being joined by Brooke Baldwin from Facebook. Hello, Brooke.
Brooke Baldwin: Hello.
Paul Boag: I thought for a minute you were just gonna wave, having completely forgotten that it was an audio podcast as well.
Brooke Baldwin: Do we wanna mime the interview? I mean, we could do it.
Paul Boag: Yeah.
Brooke Baldwin: It would be weird for a podcast though.
Paul Boag: Well, I was hoping maybe you would do it in creative dance. Is that an option? No? No.
Brooke Baldwin: It's on the table as negotiable.
Marcus Lillington.: You start, Paul.
Brooke Baldwin: Yeah, you start, that's right. You get the tattoo first, I'll go second.
Paul Boag: Okay, fair enough, all right, that sounds fair. I have to say, out of all the guests on this season, it is Brooke that I'm most excited to have on the show.
Marcus Lillington.: What a suck up he is.
Paul Boag: I know.
Brooke Baldwin: Oh, come on.
Paul Boag: No, no, no.
Marcus Lillington.: He says that every week.
Paul Boag: No I don't you liar. There is a specific reason why I'm excited to have Brooke on the show is that Brooke is the inspiration behind this entire season, right. That is because … What was the conference, Brooke, where we met?
Brooke Baldwin: Was it in Brighton, wait?
Paul Boag: No.
Brooke Baldwin: It was at a place with a B. Where was it?
Paul Boag: Bristol?
Brooke Baldwin: Bristol. Okay, so, yes, yes.
Marcus Lillington.: That was good.
Paul Boag: Both of us obviously do far too many conferences. Wherever it was, wherever we met, and Brooke gave this talk, right. You know how everybody normally stands up and they give a talk which is essentially, "This is the best practice," and I do it, I'm the same. I stand up and say this is how you should do whatever it is, user research or user experience design or interface design, whatever. They kind of expound best practice. Brooke stood up and basically, you explained what your job was, didn't you?
Brooke Baldwin: Yeah. I did.
Paul Boag: Which was brilliant, right.
Brooke Baldwin: No, I'm just a simple creature. That's all. I couldn't think of anything else to talk about.
Paul Boag: I don't know, whatever your reason. It was such a great talk because it gave you real insights into what goes on behind the scenes with user experience 'cause obviously I knew it was a job and I knew people did it and I knew even theoretically what it covered, but I couldn't help but think well why does that take all day every day? How do you keep busy?
Marcus Lillington.: You just ask people a few questions don't you? User research, that's all it is.
Paul Boag: Yeah.
Brooke Baldwin: No big deal. Come on, crosstalk 00:03:20 can do it.
Marcus Lillington.: The odd one you know.
Paul Boag: Yeah, exactly. But actually hearing that broken down made me think this would just be a perfect season for the podcast because there are lots of jobs like that. I get asked all the time what the hell I do including by Marcus who has been working with me for 13 years. So, you know, so yeah. We thought it'd be really good thing for a season and it's worked out really well so far. So it's crosstalk 00:03:49 good to finally get you on-
Marcus Lillington.: Paul, I'll stop you there. 20 years. We met 20 years ago and started working together.
Paul Boag: Oh you're kidding me.
Marcus Lillington.: Mm-hmm (affirmative). I always have to pull him up on the numbers. He gets them wrong. Not a numbers guy.
Paul Boag: Yeah, but normally I crosstalk 00:04:04–
Brooke Baldwin: Look at the shock on his face.
Paul Boag: I am shocked. That is terrifying. 20 years, is it really? Crikey.
Marcus Lillington.: 1999, yep.
Paul Boag: Wow. There you go. Good times or something.
Marcus Lillington.: Mostly.
Paul Boag: Well we're speaking together again soon Brooke. Did you know that? Yeah-
Brooke Baldwin: Yeah, yeah I did. I just saw that actually. It's pretty cool.
Paul Boag: Yeah, Better UX which is a conference being run by UserZoom in London. So we're … yeah, that's not too far away. When this goes out, I think it's literally a few days away.
Brooke Baldwin: Oh, yeah that's right. It'll be like the next week. Yeah.
Paul Boag: Yeah, but that's really good because if you're in the London area, here's the great thing about the UserZoom conference. It's free.
Marcus Lillington.: Ooh.
Paul Boag: So if they've still got places, go and have a look and see whether they've got places. Now, it's at info.userzoom.com/betterux-London-2019.html. A really nice snappy website address. You could just go look at the show notes and I'll put it in there. So that's good. So, Brooke-
Marcus Lillington.: Yes.
Paul Boag: Before we get into the real in depth questions, I want to ask you a really basic question that I kind of ask everybody which is how do you describe what your role is to friends and family? In other words, people that don't get digital. You know, because oh yeah I talk to people about using things or do you just say I work for Facebook and leave it at that?
Brooke Baldwin: I think I have a good answer for this. Let's find out, but first I do want to say thank you for having me. I'm delighted to be here and for everyone who's listening in, hey how you doing? I think my brother's listening so that's exciting. Hi Chris, and if anyone else I know's here that's awesome, too. Try to not make a fool out of myself but who knows. I'm-
Paul Boag: You're the only person on the panel right now that wouldn't be making a fool of themselves. I wouldn't worry too much.
Brooke Baldwin: Oh come on, let me join in. I'm delighted to be here. I love my job. So I'm happy to talk about it. So when people ask me what I do and my brother's a great example of like we have such different jobs, we have such different jobs. He's just coming off of a career as a press officer for professional cycling, going to bike races all the time. He doesn't work in an office. I work in an office or I get on a plane to go places, but a few years ago I saw on Reddit somebody posted like in five words can you describe your job, and said that what would mine be? And I thought oh, okay. I was like oh five words that's too much work for me, but essentially I figure like it's job my job to be nosy. That's one piece, but I spend my life asking the question why or even better like hey why'd you do that?
That really is the distillation of my job as a qualitative researcher is why'd you do that? Okay, why'd you do that, and sometimes I'm asking people but sometimes I'm just watching them and discerning that from the context of the environment but it's just my job to be nosy and see how people are using technology or want to use it or it's causing them chaos or problems. Essentially, people are like oh you work at Facebook. Okay, that's cool and I'm like yeah, and they're like what about this, do you see Mark? And I was like he doesn't call me. I am not that important.
Paul Boag: I have trouble to believe that Brooke. Well if he doesn't call you now he will do soon I'm sure. You'll rise through the ranks and conquer the company and you know I don't know whatever-
Brooke Baldwin: I like my hidey hole. I like my hidey hole-
Paul Boag: Do you?
Brooke Baldwin: Oh I really do. I love the work that I do. There are a lot of researchers at Facebook but I think I have among the researchers, one of the best jobs too. So-
Paul Boag: Okay, now explain to the listener why you have one of the best jobs in user research in Facebook because this is the bit where things get interesting and Marcus, I don't think you even know this. So this will be-
Marcus Lillington.: Not a clue.
Paul Boag: It'll be interesting.
Marcus Lillington.: inaudible 00:08:28
Brooke Baldwin: All right, oh god I hope I don't disappoint. So I'm in the ads organization and I work on emerging markets. So international markets but we're focused on emerging markets. So these are the countries around the world where we see the most potential for growth and that are often very different from, you can hear I'm American, very often different cultures from American or British culture and it's my job to go help do more of what we call foundational research or kind of just go figure out … In meetings, I have given this demonstration of what my job is. If I have to pantomime for it. So this is gonna totally go flat for the podcast, but for the seven people who are watching us right now, it's like kinda this is it, and I'll describe it to people. I kinda close my eyes and put my hands out in front of me like I'm in the dark and I'm trying to find things in a darkened room. That's the kind of research that enjoy the most and it's fun. I do, I just get to go be nosy places and talk to businesses, small businesses, single proprietors. I talk to end users and just try to understand what they're trying to get done. And I-
Paul Boag: It sounds a bit … Sorry. Sorry to cut across you. I don't want people to underestimate this, right. You say something like emerging markets and that all sounds very sterile. Give us … no, no. It sounded very good. Give us some idea of the kind of countries we're talking about here.,
Brooke Baldwin: Yeah. This starts to sound so braggy doesn't it? So I'm like all right-
Paul Boag: No, no. Just go for it.
Brooke Baldwin: All right I'm gonna go for it. Everyone'll forgive me. You'll forget who I am in 10 minutes. Okay, so in the last maybe 10 months, Indonesia, Singapore, Mexico, Brazil, just came back from three weeks in the Philippines doing research. So lovely places. Interesting countries. On paper, it's super sexy. When you get down to the nitty gritty and we can talk about that, it becomes way less sexy, but-
Paul Boag: That's good. Go with the sexy at the moment.
Brooke Baldwin: Okay, the sexy. Yeah so I get to go to these places where-
Marcus Lillington.: Globetrotter, yeah.
Brooke Baldwin: And the idea is like we really actually want to put me in and bring some of my teammates with me to put us in places that are not like where we work or where we live now or even where we grew up so that we can understand oh, things are different here and here's how things are different and here's how we need to make sense of that for ourselves.
Marcus Lillington.: So you're, trying to put this into a nutshell for my brain. You're going to try and understand the cultural differences in how basically you can do your advertising model in these different markets because people react very differently to the way … people react differently to advertising in the UK as they do in America, I mean I guess inaudible 00:11:25 states a lot.
Brooke Baldwin: Yeah, and sometimes it's even sort of backing up even a bit from that and being a little bit broader and kind of understanding how is this culture different? How do people communicate? How do two individuals have relationships with businesses? Big businesses, very small businesses and different kinds of businesses and what kind of things do they want? What kind of relationships do they want to have with them? How do they want to communicate with them? So it's really getting into sort of like how can you understand humans and their behaviors and interactions on an individual unique sort of small frame and then also on a much larger societal kind of level. By no means do I know all the things about knowing all the things, but I get to kind of dig in to try to understand that. Even just understanding what are the questions that they have can be useful for me and the team that I work with.
Paul Boag: Absolutely. This is just a random thing that's popped into my head. What did you study in order to end up doing this job?
Brooke Baldwin: Oh god. I was so useless for a while. So I started undergrad as a physics major-
Paul Boag: Oh right.
Marcus Lillington.: Cool subject.
Brooke Baldwin: Super cool subject, but what I see now about the decision I made when I was 17 years old and entering university is that I wanted to understand how things work, and I still do that. I just do it in a very different way. So physics was that for me, but I got to tell you I got part way through my third year and I was like this is awful. I hate it and I'm not that good at it. So I was like oh what can I do? Oh literature, that seems easy. So I finished with the degree in English literature. So that's useful for not much. Conversation at cocktail parties, and then I have a masters of science from NYU and studied business and technology. It was kind of a mish mash of those things. This was back in the day actually when you guys first met, '99, the dotcom boom. So you know just sort of these programs were coming about, but to be fair, most of my education came from working at startups while I was in graduate school. Then after that, working at lots of companies, keep taking classes, keep learning stuff. Learn a lot on the job. So, yeah.
Marcus Lillington.: It seems the focus if you like of where you are at the moment is more sort of sociological-
Brooke Baldwin: Yeah.
Paul Boag: Yeah that's why I asked the question.
Brooke Baldwin: Yeah.
Paul Boag: It sounds like you should have a masters in sociology or something.
Brooke Baldwin: I know. I should. What I'm most interested in sort of that spectrum of study is cognitive psychology and how does the brain make decisions. So when we start to think about behavioral economics and choice architecture and how is the brain making quick decisions and how is the brain making longer term more burdensome kind of decisions. Yeah, that stuff I get super interested and cranked up about. If I were ever to go back to school, I think I'd probably get a PHD in cog-psych, but-
Paul Boag: Yeah, I would be the same.
Brooke Baldwin: It's so interesting.
Paul Boag: Yeah, yeah.
Brooke Baldwin: But at the same time it's like, uh, that would be-
Marcus Lillington.: If it was me it would be English literature. There you go.
Brooke Baldwin: All right.
Marcus Lillington.: Back to almost square one.
Brooke Baldwin: All right.
Paul Boag: Right so-
Marcus Lillington.: The cat's appeared again. crosstalk 00:14:56
Paul Boag: Cats, cats.
Brooke Baldwin: Sorry.
Paul Boag: Yeah, we didn't mention the cat did we? We actually have two guests today. We have Brooke and what was the name of the cat?
Brooke Baldwin: This is Pudge. He is my supervisor and if I try to stop him from doing this he'll just come anyway. So like-
Paul Boag: It's fine.
Brooke Baldwin: Yeah.
Brooke Baldwin: Totally equivalent. Yeah, totally inaudible 00:15:20.
Marcus Lillington.: inaudible 00:15:20
Brooke Baldwin: I'm sure your grandson and your daughter or son are super happy to hear that.
Marcus Lillington.: Absolutely.
Paul Boag: So, let's just take a quick break before we dive in a little bit here. We're getting some good questions from the people listening or watching live. So we'll do some of those as well, but I do just want to talk about our first sponsor. Sorry, talking about advertising it's time to talk about our sponsor but it's a really good one. I'm very picky about which sponsors I'll have on the show and this is a good one. So the Digital Project Manager school basically offers a course in digital project management, who would have guessed? Mastering digital project management and we all have to do a degree of project management whether or not we call ourselves project managers. We all have to deal with shifting priorities. We all have to deal with vague stakeholder feedback, all of that kind of stuff and finding a way to survive all of that can be quite harrowing, and if you run an agency, that's even more important and even more critical. Managing your web projects effectively will benefit your team. It'll benefit your profitability. It'll benefit your deliverables and your client relationship. But most of us are making it up as we go along if we're honest. We have no formal training in it.
So it's about time to change that and that's where the Digital Project Manager school can help you with their course on mastering digital project management which is a seven week, online course. So it's achievable. It's not going and doing a masters or a degree or anything like that. This is just practical stuff that you can actually do and it covers everything from broader topics like project methodologies and leadership qualities to techniques for dealing with that specific stuff that you do on a daily basis like estimating projects, handling budgets, writing briefs, dealing with unexpected challenges from stakeholders. You know, the normal crap that gets thrown at us. The next course is starting on the first of April. So hurry and make sure you get that. You can find out more about it by going to thedigitalprojectmanager.com/boagworld.
Now, so we've got all kinds of questions that I've prepared about your job but we've had a couple of interesting questions that have been dropped in as well that I want to mention. Stewart William is asking, talking of the subject cognitive science, is there any books that you would recommend? The one that sprang to my mind was Thinking Fast and Slow. Have you read that one?
Brooke Baldwin: Of course yes. That's the Danny Kahneman-
Paul Boag: Amazing book. Yeah, yeah.
Brooke Baldwin: Yeah, so Danny Kahneman's great. The thing with that book is that's not where I would start 'cause Danny Kahneman is a really dense writer-
Paul Boag: Yes, he is.
Brooke Baldwin: And for those who are listening, he won the Nobel prize for behavioral economics. He and Amos Tversky were the fathers of behavioral economics. Tversky died but don't worry. He'd gotten like a genius grant, MacArthur grant earlier and they're all settled out, but Kahneman's writing is really dense and it's a bit of a slog. I remember being on the New York City subway reading it and this guy, this really big burly guy stopped me and he was like, and I was kind of like oh he's starting to talk to me when I was reading it one day on the subway, he's like, "It took me a year to get through that," and I was like I understand, yeah. I would start with something that's a little bit more palatable. Probably Nudge is a great book. Like one of the Tversky and Kahneman's disciples wrote that, Richard Thaler. He won the Nobel prize for economics a couple years ago.
But if you don't want to even read a book, honestly go watch the movie The Big Short, and that's getting into behavioral economics. That whole film is about behavioral economics and by the way, Richard Thaler has a cameo in that movie.
Paul Boag: Really?
Brooke Baldwin: Yeah, I give a talk at work about this and the scene where Selena Gomez is actually at the craps table or roulette table. I can't remember what it is and Selena Gomez is describing one of the concepts of BE. Richard Thaler is sitting right next to her, and I can tell you I'm probably one of the few people in the audience when I saw the film was like, oh my god, and I wasn't talking about Selena Gomez. I was like, it's Richard Thaler. So if you want to know the level of nerdiness. I would say any of Thaler's books are really, really good but go watch that movie or even Moneyball, the baseball film about how the Oakland A's put together sabermetrics, and they're using choice architecture and decision-making under uncertainty to figure out how to have a low budget and have this great baseball team. Those are easy reads and it's really interesting subject matter but the movies are even easier to start with I would say. Very consumable.
Paul Boag: Brilliant. Brilliant advice. Okay. Another question that we've got in while I'm just dealing with people's random questions because they're really good, Paul is interested to know what unexpected cultural and social differences you've experienced that have jarred or clashed with your own cultural norms or values? What's kind of surprised you?
Brooke Baldwin: Oh that's such a good question. So I will give another long winded answer. Here we go-
Paul Boag: No, no, that's good.
Brooke Baldwin: I grew up mostly in the United States, but as a kid I lived in a little island called Guam on the Pacific. As a teenager I was an exchange student and lived in South America. So growing up I had some … maybe about a total of six years or so of the teenage life living outside the United States. So that kind of thing has helped me as an American you know we can … You know, you all have your opinions about us. You're not wrong. So that's kind of helped me understand. Sometimes the things that can be most problematic for me are perhaps more gendered kind of issues. I suffer from this delusion that men and women should have the same kind of opportunities and be treated relatively equally. I mean if you're a jerk you're a jerk, but I'm not worried about if you're a man or a woman in that instance. So sometimes those things can be difficult but what I'm really trying to do when I'm thinking about my research is learning about that stuff before I go into a country because it's best if I'm aware.
so if I'm in a country where, you know what, it's just i'm gonna draw attention a lot of times because the way I look is very different. So if I can just understand it's better if I just cover my head and I cover my hair. Wear a long shirt. It's not a huge imposition for me, especially if I can understand like I am an interloper and I want people to talk to me and be comfortable with me. So it's really my job to fit in. So biggest things that have upset me, I don't know. I tend to, in my professional life, really not get cranked up about stuff. That's why I want to push in and learn more actually. In my personal life, certainly as I've traveled or done stuff, there've been things where I'm just like that's not right. Nothing really sticks out to be perfectly honest because part of my job is I've got to go fit in because I gotta get people to open up to me. So it's about trying to understand at least to a degree that I can be respectful around the maybe the gender boundaries or the racial boundaries or the class boundaries that happen. Then I have to be aware or try to, god, how do I maneuver around these, you know.
Marcus Lillington.: And to ask that question maybe slightly more lightly, are there any kind of quirky stuff that you thought, I wasn't expecting that.
Paul Boag: There's definitely something because crosstalk 00:23:05 This face lit up at that point.
Brooke Baldwin: Yeah, yeah, yeah. So when I was on a trip to Indonesia last year doing some research, the partner vendor that I was working with, one of the guys on the team did this thing and I thought it was great. He was like, okay we're gonna give you things to eat and then we'll tell you what they are afterwards. I was like, I was the only one who was like okay. I'm just dumb enough to do it. So sometimes the differences in what cultures eat can be very radically different. Obviously, I have a very American palate, but I discovered I have a deep love for chicken intestine. That is delightful. Fried chicken intestine on a little skewer. Yeah.
Marcus Lillington.: Yum.
Paul Boag: Good.
Brooke Baldwin: It's really good. Calf's brain, not so much. It doesn't have much taste. It just picks up the flavor of whatever sauce it's in. Those can be kind of the funny things. It's best if you don't tell me what it is because then I'll just, let me just eat it. Let me get it down and then we can talk about it later, but I've also discovered that like avocados in America and in the UK for the most part, avocados are a savory thing. You put them in salads. You put them on sandwiches. Not so in the rest of the world.
Avocados can be eaten in some countries with condensed milk and they're a dessert and the greatest thing I've ever discovered is avocado popsicles. Like every time I'm in Asia now I'm like ooh do they have avocado popsicles? Can I get some of those? So those kinds of funny things. When I go to these other places what inevitably ends up happening is I try to learn as much as I can about the people there and the things I need to learn about, but you learn a lot about yourself too, right. You really are about like oh man I'm really uptight about this thing 'cause like, oh, hm, all right, okay. So that happens a lot.
Paul Boag: You talked about preparing before you go so that you're aware of the culture you're going into. I'm interested in what other types of preparation go into a trip like that 'cause I don't imagine you just suddenly rock up to some of these countries.
Brooke Baldwin: No, it's a bit effortful the kind of deep, what I call thick data, right. That's the kind of research I do is thick data. Six to eight weeks often to plan an in country trip. So that is soup to nuts from the very beginning of like hey we need to do a research trip and going through finding the right vendor, getting them prepped, figuring out who do we need to talk to in doing the recruiting for that. Sometimes that can be deeply complicated-
Paul Boag: Sorry, can I interrupt you. You said vendor there. What do you mean by a vendor in that context.
Brooke Baldwin: Yeah, great. Not a vending machine, yes.
Paul Boag: No, I guessed that.
Brooke Baldwin: So I need partners in every country that I go to. So I speak like one and a half languages. Mostly speak English and kind of speak Spanish. So I for sure need people who speak the local language and know how to even literally, physically get around the city or the country that we're in. So I'm partnering with research teams in other countries to help do things as simple as translate the questions that we need asked and do the actual moderation or if I am actually having the conversation with the research participants, I need to find someone to help me get a really good translator, and it can't be any old translator because from my point of view, how we frame questions and the word choice is very, very important because we introduce so much bias. So I need someone who isn't just like oh yeah, sure I can do simultaneous translation. I need someone who understands the difference between translating could versus translating should versus translating would because those all have distinct different connotations in English. I need them to understand English well enough to understand the ramifications in their own local language so that we don't run into trouble when they're doing those translations because that can radically alter the quality of the research and the validity of the research that I'm doing.
So yeah, I get in the weeds on that kind of stuff. So making sure I've got the right team on the ground. We've got all of the logistics planned. Simple things like depending upon who's going, does everyone have the right visa? Has everyone gotten the right … Well, you don't call them shots here but vaccinations. What do you call them-
Paul Boag: Vaccinations.
Brooke Baldwin: Jabs. You call 'em jabs.
Marcus Lillington.: Jabs.
Brooke Baldwin: Jab, you know like all of those kinds of things. Then I have to make sure I'm working deeply with my team before we go with data science, with product management, with engineering and the design team to figure out what questions do we have to ask these people and how do we prioritize them and how do we make sure that we're covering all the topics that need to be covered, but at the same time some of my work is very exploratory. So leaving enough space so that if we find something in conversation with someone or a group of people that we can sort of follow those really golden nuggets. So there's a ton of planning that goes into that and also just figuring out who's doing what on each day because it's not a vacation. So everyone who comes on a research trip with me has to do things each day. They are taking notes. They are writing up a synopsis of the day. They are participating in a debrief. We are editing video. We do all kinds of stuff. So it's quite a bit of prep that goes into it before we get on the ground actually in the country to start doing the actual research.
Paul Boag: So, okay, it's now beginning to sound slightly less sexy because there's a lot of prep that needs to go into it. You've now reached the country. What happens when you're there? So you get in country, what kind of research methodologies you're using? How's your day structured? That kind of thing.
Brooke Baldwin: Yeah, they are long hard days and they are not for the faint of heart. Often they're 12, 14, 16 hour days. For the kind of research I want to do, I'll do a variety of methodological approaches. Sometimes they are in home or in business visits, one on one with a person and we're having a conversation for an hour or two. That's a semi-structured interview so I have specific questions I want to ask. Kind of like what you're doing and then also stuff that we think of extemporaneously depending upon the responses. I may have a video crew with me as well shooting it. People are taking notes. We may be wanting to talk to multiple people at a time. So we may be at rented facility and talking to two or three people at a time, having them show us their mobile phones or whatever devices they're using to connect to the internet to show us how they're doing things. So diads and triads we call them, in depth interviews.
Here I'll throw out the other fancy research terms, contextual inquiries, sort of baby ethnography. The sociologists who are listening if there's any of them, don't worry. I'm not claiming that I'm doing true ethnographic research. I don't have a year to spend with anyone, but we are borrowing from their methodological approaches to sit down, shut up and observe and take notes of what's happening. The sights and sounds and contexts around it all to sort of inform the research. Sometimes even before we go in country, I will do research on moderated remote things like card sorts or time on task things, time out tests. If I need some research prior to the in country visit to inform what we're gonna talk about. If I need to know a little bit ahead of time, I may be doing that kind of stuff. So it really runs the gamut I would say, but it's all within the sort of qualitative research methodological approaches.
Paul Boag: So how are you-
Marcus Lillington.: Do-
Paul Boag: Sorry Marcus, carry on.
Marcus Lillington.: I was just gonna say, especially if you're doing hour, one two hour long interviews with people in their homes, how prepared are they? Do they know what's coming? That's a long time.
Brooke Baldwin: Sometimes they don't. We do let them know how long we're going and that's like part of the initial contact when they're being recruited, you know. That hey we want to come to your home. We may even tell them the topic about what we want to talk to them about, but a lot of times we don't want to tell them too much because I don't want them to rehearse things. We'll tell them how many people are arriving if we are going to be recording things. We tell them all of this in advance. They often forget, and then we show up and they are startled or surprised. So I go through it again. One of the first things I do with a participant is just try to settle them down and ask them like hey have you ever done this before. Let's talk about what's gonna happen and I'll describe how long we're intended to be together. Also, it's really critical that we let them know they can stop at any time. If they don't like a question they don't have to answer it.
They're welcome to ask me questions and I let them know that I actually may or may not answer them. I try to give them as much sort of information about the session and I also tell them like look, we are here to … you're the expert in your opinion. So that's what we're here to get. Sometimes I'll tell people like you're kind of an honorary member of the design team today. So it's your job to-
Marcus Lillington.: That's great.
Brooke Baldwin: Find things that are terrible and point them out to us because if you find them then that helps us fix them. So I'm trying to give them permission also to let it rip and complain about things. Yeah. Mm-hmm (affirmative).
Marcus Lillington.: Cool.
Paul Boag: So you go through this process and it sounds like you're doing some analysis of your results as you go while still in country, but one presumes you do quite a lot when you get back. I mean what do you do? I think someone's asked this question … Yeah, Tim has asked it. Once you've gathered insights into your audience, how do you turn them into actionable items that can actually be applied?
Brooke Baldwin: No, that's a great question and I actually am a little bit different than a lot of other qual researchers. I try to do a lot of it in country. So it can be exhausting. So if we do two or three in home visits a day, after single visit or if we're in the lab, after every single session we do a quick debrief. So everybody gets like one or two post it notes and I want everyone to just write down, and everyone's opinion's equal at this point. What struck you most about that last session? What was most interesting? What was the weirdest? It doesn't really matter. They don't have to defend it too much. So we do a very quick debrief and everybody shares what theirs are and we start to sort of group together like concepts or ideas and maybe we even put a label on it. So we're just trying to sort of capture higher level notes. Everyone has taken notes, but sort of higher level things. So after every single session, we're doing a quick debrief. 15 minutes, 20 minutes at the most. I don't want to spend a lot of time sitting on it.
Then at the end of the day we do an all day debrief. So we're looking at what patterns did we see from session to session and then are there any anti-patterns or are there any weird cool things that stick out that we're like only one person did it but it was crazy. So let's note that. Then day after day we're building that. Then by the end of the … And we're writing up a note usually every day and publishing that internally to the rest of the team and just sort of saying here's the highlights, here's some quotes, here's some short videos or pictures. So we're collecting. We're collecting. I even have a tendency to have a big spreadsheet where we're taking notes person by person, question by question in one big place. So it starts to be this big massive spreadsheet so we can look across for this question how did all these people answer. So we can start to see patterns that way. Then at the end of the trip, we're gonna do probably like a half a day or a full day of like, all right, what did we learn, and going through all of those things.
So it's like short debrief after each session, longer debrief after each day. Longest debrief at the end of the trip. So I'm trying to do as much heavy lifting in country as possible because I don't want to come back and it takes seven weeks for me to do my analysis. I work in industry. So this is not academia. I don't have the luxury of a ton of time. So I need to move very quickly. A lot of other researchers will come back and three or four weeks they're doing their analysis. That suits them. I found over the years that I want to push harder while we're in country and be able to come back as quickly as possible and start sharing that things because there's also some momentum of the research trip has happened. People are stoked about it. If I've brought cross functional team mates with me, like part of the reason I'm also doing the debriefs and the analysis in country because I want them to really get it to the bone.
They learn this stuff, but I also want to coach them through how to do the analysis and how to frame things. Then they become my agents and my advocates. Mine, as if I owned the research, I actually don't, but they become then spokes people for the research and they are very trusted by the rest of the team as well. So I'm also trying to sort of build that capacity and that knowledge while we're in country.
Paul Boag: Yeah. That's very interesting. You talk about taking cross functional teams with you. How do you decide … I'm gonna ask two questions at the same time which is really, really bad etiquette but there you go. I'm gonna do it anyway.
Brooke Baldwin: Okay.
Paul Boag: How do you decide crosstalk 00:37:03 who to take? Also, how do you overcome the problem of ending up with half a dozen people in the room with this poor old participant that's kind of faced with this bank of people?
Brooke Baldwin: Yeah. Yeah.
Marcus Lillington.: Does everyone wear white coats? That's the other one.
Brooke Baldwin: Yeah.
Paul Boag: Clipboards, yeah.
Brooke Baldwin: So there's a few things. Deciding who actually gets to go, we're certainly looking at who makes the most sense from a product or feature perspective. Who has been working on things that are related to the topics. Who can bring sort of the most insight to bear, that kind of thing. As I have done more and more of these trips, I also learn each time and sort of figure out where's the sweet spot and that continues to change. I'm always sort of trying to tweak it and get better and get smarter about it, but I want to sort of try to bring representation from very different skillsets on the team because they've got different points of view and they also can then when they learn this stuff, then they'll have different ideas about how to maybe solve some of these problems that we're seeing and I really want as much diversity in representation on the team when I'm going in country. So I am trying to get as much spread and coverage on that as possible.
To the question about how many people are in there staring at them, there's a few different things that researchers like to do. Number one, I do try to limit the number of people who are actually in the room. So sometimes we will have a two way mirror, yeah, whatever it is and we'll all be hidden behind it. If we're in someone's home, I am kind of a jerk about the rules about how many people can be there because it is, we're coming into someone's house. They are being kind enough to spend time with us and open their world and let me ask a bunch of nosy questions. Often, I'm coming in with a camera too, and a translator. So it's kind of already stacked. So I'm really limiting it to maybe one other person, trying to keep it as tight as possible. To be fair, for some of the really intense like two hour sessions and we're doing three a day and if we're in a country like Jakarta where the traffic is crazy. You're getting picked up at six or seven in the morning. We're not done back in the hotel until about eight and we still have to do that final debrief and then do a writeup that night, oh and eat dinner and maybe take a shower if you're lucky. So it's not for the faint of heart as well.
So there can start to be some self selection within that, but you see it a lot every time when you go to someone's business or their house and they're just like oh there's a lot of people. So it's my job very quickly to try to establish a rapport with them and to try to make them as comfortable as possible so that they can ease up about sort of all that energy that's in the room.
Paul Boag: Yeah. Not, that's so interesting. You've certainly taken the glamor off of it with the long days and these sessions and stuff like that. What about do you ever feel scared going to some of these countries? The reason it's popped into my head is I've just been invited to go and speak at something in Iran and I'm like, ooh. Do I want to do that? How do I feel about that? You must have security issues and things like that sometimes do you?
Brooke Baldwin: Well certainly there are going to be countries that any company I'm working for may decide that are off limits. So each company may have its own rules around that. I lived in New York City for 20 years, so that certainly helps. I've lived in Asia. I've lived in South America. So I've had exposure to some places. When I was a teenager living in Ecuador in '89, that was when all of the judges were being assassinated in Colombia in South America and at 15 years old it was kind of like all the exchange students who had been in Colombia were sent down to Ecuador 'cause they were like get 'em out of here. We don't want to have any trouble. So at a young age I kind of was like oh these things happen. Not to me directly. So I fell a little bit of an advantage because I've had some not scary, but educational experiences in terms of that realizing what's going on in the world. Stuff can happen. Stuff can happen walking down the street in London, too, and certainly things have, but you do the best you can.
Certainly every company that I'm working for, they always know the logistics of where I am, what hotels I'm staying at and what my team is. We have internal practices and we're always trying to hone those and get better and smarter about … In some countries, it's not smart if you're a woman to sort of walk around alone at night. So it's making sure that the team that I'm bringing with me we're highlighting and we're talking about those things in advance. Don't flash your expensive electronics around. Don't wear company branded t-shirts. We're gonna stick out anyway so it's also about doing the best you can to just stick out as little as possible. You know I don't have a probably a really good answer for that but-
Paul Boag: No, no, I'm just curious.
Brooke Baldwin: I'm not going into a war zone. But the kind of research I do, it doesn't really put me in those kinds of situations and this is also a benefit of working with a local vendor is they're gonna have a fixer. So they're also bringing to bear understanding like okay, if there's certain populations we want to do research with they're also there to sort of bring wisdom around what are the places to go and what times of day to go or not go to them. I have to really respect those kinds of things that they tell me because otherwise I'm asking for trouble. Sorry, I've got to push my cat off the table because he's eating all this catnip. You've got a plant here. So important things.
Paul Boag: That's fair enough. Yeah, now get your priorities right.
Brooke Baldwin: Yeah.
Paul Boag: So when you get back after these trips, you're doing however many you are in a year, what are you doing when you're not on a trip or not preparing for a trip?
Brooke Baldwin: Lots of other things. First of all it seems like I go to meetings all the time-
Paul Boag: Yeah. Don't we all.
Brooke Baldwin: That would be the bane of corporate America's existence. So the research trips are often more sort of strategic, longer term kind of research, but my team has other research needs and sometimes it's like hey we've got these three different design options and we don't know which is the best one. Brooke, help us figure that out. Okay. So I do more tactical kinds of research. Shorter term, not as in depth, starting to skew towards a little bit more quantitative but I'm not really a quant, doing that kind of stuff. I'm pretty senior researcher so also I'm helping to mentor and guide other junior researchers. I get involved with interviewing. I get involved with trying to collaborate with researchers across the organization around larger topics. I speak at conferences. I teach classes internally about how to do certain research things, methodologies or using certain tools. Yeah.
Paul Boag: So you're not very busy then.
Brooke Baldwin: Well I don't goof off. I don't get to goof off as much as I'd like to, but when I have my moments to goof off, I take advantage of it.
Paul Boag: Do you get any time to goof off in country? Do you actually get to enjoy these countries you visit?
Brooke Baldwin: I guess two things. I'll push back on that question about do I actually get to enjoy these countries. Doing the work is actually really enjoyable and I love being with these folks and it's so interesting to see man everybody's got hustle. Everyone's trying to do things, something better for themselves and it's an honor and a pleasure and a joy. I had a participant when we were in the Philippines who we met with all of them twice. We had to talk to them twice about something and the first time he wouldn't let us come to his home and he made sure that we met at a research facility and he was kind of like who are you. Then he was like, okay you guys are for real or whatever. When we came to visit him the second time in his home, the night before he had actually cooked chicken adobo which is a really popular Filipino food. He cooked rice and chicken adobo for us. He'd gotten up early in the morning, so did his wife, and she made her special cookies for us. So these user experience where you're just this is dope. I have the greatest job.
Now I forgot what the question was.
Paul Boag: Well I was asking you whether you had a chance to enjoy the-
Brooke Baldwin: Oh to do fun things.
Paul Boag: Country, and I think that's a bit of a bias on my part because what I enjoy about a country is the scenery and the architecture and stuff like that while you seem to enjoy the people which is perfect for your job.
Brooke Baldwin: I like the other stuff, too. I mean I'm not gonna lie. I usually go a few days before the rest of my team goes in country because I want to spend a little bit of time with the vendor. Talk to the moderators or translators. Make sure that sort of everything's lined up. So I may not have 12 and 16 hour days then. It's like an eight hour day instead. So I can go out to dinner somewhere or maybe go sit by the pool for an hour at lunch. I try to take those moments when I can, but I don't always get to. What I'm trying to do now in how I'm structuring research trips is maybe making a day break in the middle if these are two or three week trips. It's nice to have at least one day off. For the trip we just did in the Philippines, I went and just rode all the different types of public transportation in Manila and walked around neighborhoods and wandered into an old church just 'cause I wanted to get a vibe of the country as well, and the city. So those are the kinds of things I do. I don't tack on like a seven day vacation after this stuff.
Paul Boag: No, you couldn't after every single one of those could you?
Brooke Baldwin: No, but my team mates can sometimes. That's a real privilege if they can do that, but often I'm getting that work done and I'm trying to get the analysis done and then I'm trying to share it out and get it to cause impact and have people react and do things about it. So the research is not done just when I'm done with the country or just when I'm done with the analysis. It's a lot about communicating it and trying to get people to care and pay attention and do something about it. Yeah.
Paul Boag: You told that little story a minute ago about the guy who was a bit standoffish to begin with. Its so interesting. Tim's asked a question which kind of ties in with that which is how do you deal with when you're doing qualitative research like that, obviously because they're meeting with you and they're interacting with you, how do you prevent you biasing the relationship because even in those two sessions, his attitude changed how he behaved towards you changed. So I guess you can't completely remove bias.
Brooke Baldwin: I don't want anyone to suffer from this idea that my research is totally unbiased. I think that part of it is acknowledging that that's an impossibility and as much as I think journalism is a noble career choice, this idea of like we're unbiased is complete bullshit. Can I say that?
Paul Boag: Yes.
Brooke Baldwin: And what it is is about recognizing what are the types of biases that we are more prone to have, and I become a better researcher the more I know myself and the more I understand my own biases and can I just figure out that that's happening. Sometimes to be perfectly honest I'm not worried about bias in a situation in the effect of like, okay this guy's welcoming me into his home and his attitude about who we are has changed. It's still my job to ask the right question in the right way and if I sniff that this person isn't telling me the truth, it's my job to push in on that or leave it and then come back around and ask that question in a very different way a couple of more times so that it's part of my analysis I'm looking at … I kind of ask the same question three different ways. Do they have consistency in their response? If they didn't, then I have to take that into account for my analysis, right.
So I have to maneuver around whatever their reactions are and then be aware of my own reactions. I'll tell another quick story. One time I was doing research at this company, this was years ago. Way before Facebook, and we were interviewing nurses and it was for software that they would use and second time I had talked to this nurse and whatever it was she was pissed off about something and she like slammed her hands down and pushed back really hard and stood up and kind of started yelling at me. I had a guy who I was training, like a designer who wanted to learn about research, was in the room taking notes and I could see out of the corner of my eye that his eyes bugged out of his head. He was just freaked out, and my reaction was oh my god this is great. Tell me more. So I was just like … When I'm actually doing research, I kind of shift into this other gear that I'm not quite sure where it comes from. In my personal life I would probably behave very differently, but at work it's really my job if someone gets sort of fired up about something or if I don't believe them to make sense of that and then try to get good thoughtful answers, and yeah. I don't know. That probably really didn't answer your question, did it?
Paul Boag: No, no, no it did.
Marcus Lillington.: I think that's great. I've been doing a lot of kind of testing and things lately and I try to be unbiased, but what I don't do, which is I think is why I am not a user researcher is I don't dig and dig and dig and dig. That's what you just said you do and I think that's what makes a good researcher. That's what it's about isn't it?
Brooke Baldwin: Yeah.
Marcus Lillington.: I just take it on face value. Yep, okay.
Brooke Baldwin: Oh god, yeah no-
Marcus Lillington.: inaudible 00:51:15
Brooke Baldwin: Any researcher who does that is probably not gonna have a long career because there's this massive difference sometimes between what people say and what people mean, but it's very different by culture. It can also be very different by gender and it can be just different from person to person, right. So it's really my job to try to sniff that out and I'm sure I make lots of mistakes, but I remember one time I was doing some research. This was years ago. I was in Boston and I was running a lab and we had the client in the observation room and I remember afterwards we were done for the day and this woman pulled me aside and she said, she's like, "I thought you missed something in session and I was just like oh Brooke should have asked a follow-up question, but then you came back later and you asked it in a different way and I was like I couldn't believe what you got him to tell you," and I was like okay. That's my job. I didn't think that was weird or interesting or anything other than that's kind of what I do, but that gap between what people say and what people mean, it's my job to navigate that.
Sometimes they're aware as a participant that there's a gap there and then a lot of time they're not aware. They're trying to please us. They've often being compensated to spend time with me and especially if I come into their home or business, they want to be very welcoming and these are all totally normal. So it's my job to make sure I'm asking questions in the right way and I'm really listening to their answers and I'm asking good follow-up questions. A good researcher has to be a really good listener and if I can navigate that then I can at least mitigate some of the difference between what they say and what they mean. Not always. I'm gonna make mistakes, but yeah.
Marcus Lillington.: Yeah.
Paul Boag: What's been so fascinating about this season as we've kind of picked into different people's jobs and it's really come across today is how much complexity there is in some of these different roles and how there's different levels of it. So Marcus said he's been doing some user research, but there's a world of difference between what Marcus has been doing, no offense Marcus, and-
Marcus Lillington.: I've just been interviewing stakeholders. You know the usual kind of thing, preparing for a smallish type of project and I don't work with Facebook.
Paul Boag: No, and I find this really fascinating how you can kind of dig deeper and deeper into a role and it becomes more and more nuanced the further down you go and it's the same with design. It's the same with development. It's the same with sales, the same with anything, and that's been really interesting.
Marcus Lillington.: Just to kind of repeat on that, that's the fascinating thing about today 'cause we're always complaining if you're kind of doing agency work that we never have enough time for testing. We never have time for research and sometimes you'll get more than you expect and that's great, but when you get into the kind of depth that you're in it's like wow. Okay, right now I'm learning. I'm picking up things that I need to take on board and hopefully that's what everybody's doing from this series. It's fantastic.
Brooke Baldwin: Yeah. I think I would just say, it is such a privilege the kind of work that I get to do and at the scale that I get to do it, very few people on the planet do. It is not lost on me. I'm grateful and I love my job. I really, really love my job. It's such a privilege. So doing any kind of research, you know I think like is great. I've been doing this 20 years and I did not start out this way. So I would just say any kind of research is good, even that you have the instinct to go talk to stakeholders is amazing. That's the right instinct. Nurture that and grow it as much as you can. For anyone who might be listening who's thinking oh well my company won't pay for us to go to these great countries, very few do. I'm in a super privileged position. It's definitely not lost on me, yeah.
Paul Boag: I'll tell you what's really interesting that you said there. You get this reoccurring thing, right. Whenever you speak to people at the very top of their field and you did it just then, they always say hey don't worry, you're starting out. Anything's good, get going, embrace it, enjoy it, et cetera but there is this middle stream of people that you often find in an industry where they feel the need to prove themselves. So they make their jobs sound really complicated and really important and oh no you're not doing right and you're not using the best practice and all of the rest of that. I think that could be really off putting to people to start in this field, to try it. To give it a go. So it's lovely to hear you saying let's give it a go. Get involved. Something's better than nothing, you know.
Brooke Baldwin: Yeah, it is. Look, if I was in a room just of other researchers like me, I can be the biggest jerk about it too though. Please don't give me extra credit because I kind of don't deserve it. If we're navel gazing and I'm with a bunch of other researchers, I have those sort of stupid debates and fights and I have very strong opinions about these things. I do think that even just having the instinct of hey I should go talk to people and listen to what they have to say. I should put this design in front of them and have them try to use it, and you know what, let me bring the engineer too and we're just gonna shut up, and we're just gonna watch people. So much richness can come out of that. That's literally kind of how I started. I didn't start out knowing how to take two and three week research trips where I'm taking all of these cross functional teammates with me and doing these multiple methodological approaches. I didn't start out that way. I had to learn, and I had to make a lot of bad mistakes too where it was like oh crap. I caused this problem because I didn't think about this thing enough in advance to make it work. Okay, let me learn from it.
So part of it is just also you gotta have time in the seat. You gotta just sort of do it over and over and over again. I am very good at having one on one conversations with participants because I have done hundreds and hundreds and hundreds of that if not thousands of hours of doing it. So it's about practice. There's no secret. Some of us are a little bit better at it than others naturally, but it is about practice.
Paul Boag: Yeah. Yeah. Excellent. Thank you so much Brooke. There was so much-
Marcus Lillington.: Yes, thank you.
Paul Boag: In that. It was a really good one.
Before we wrap up the show, I just want to talk about our second sponsor which is Pactly. We spend too much of our time and money reading contracts just to figure out whether we should sign them or not and I had one only this morning that I had to go through that just you think oh I've lost the will to live. There's got to be a better way isn't there? A faster, more painless way of understanding contracts we sign and that's where Pactly comes in. It supports NDA service agreements, sales contracts, vendor agreements, consulting agreements and lots more contract types being added all the time. It's a really powerful tool. You run your contract through it. You upload it and basically it will flag anything that it thinks oh that looks a bit dodgy or you need to be concerned about that. So I encourage you to bookmark pactly.ai so that you can take it for a spin and give it a go. You can sign up to review a contract in a matter of seconds. It doesn't take very long, and if you are doing contracts regularly and you want to make use of this, if you use the code boagworld you'll get 30% off the lifetime of their paid plan which is really good. But there's a free plan that you can use. You can give them a try. So you can find out more by going to pac.sg/boagworld.
now Brooke, as somebody that hasn't had the privilege of listening to this show over many years, you will not know that we have to endure a Marcus joke.
Brooke Baldwin: I do know about this. I do-
Paul Boag: Oh do you?
Brooke Baldwin: Actually know about this, yes.
Paul Boag: Somebody warned inaudible 00:59:42 did they?
Brooke Baldwin: Oh god.
Marcus Lillington.: This one's not too bad 'cause it's one that Michelle from the Slack channel shared via, it was a joke from Eric Myer. So it's got to be good, yeah.
Paul Boag: Oh yes, this is a good one. I like this one.
Marcus Lillington.: I really like this. So question, how many designers does it take to change a light bulb? The answer, first, let's talk about why you don't want it to be dark. What problem are we really trying to solve here?
Brooke Baldwin: Yeah.
Marcus Lillington.: So I like that one.
Paul Boag: crosstalk 01:00:12 That is so true.
Brooke Baldwin: It resonates.
Paul Boag: Yeah, very appropriate.
Marcus Lillington.: Yes, it does. Oh yeah.
Paul Boag: Very appropriate for this show definitely. Okay, Brooke thank you. Brilliant-
Brooke Baldwin: Thank you.
Paul Boag: Really enjoyed it. I'm looking forward to catching up with you in London very soon and yeah-
Brooke Baldwin: Likewise.
Paul Boag: Hopefully this show should be out by then which is brilliant. Next week, we've got Andrew Miller joining the show. Now Andrew Miller runs digital team in a UK university. So we're going from kind of the wonders of Facebook and all the resources and all the focus on digital to a completely non-digital organization with limited resources, limited time, limited ability. So we're gonna see what life is like at the other end of the spectrum. I should be careful because they have a good crosstalk 01:01:05 team-
Marcus Lillington.: I was gonna say.
Paul Boag: I make it sound like he's a guy sitting in a shed somewhere.
Marcus Lillington.: Yeah.
Paul Boag: Which is horribly unfair.
Marcus Lillington.: They know nothing.
Paul Boag: Yeah, they know nothing. They're ignorant. They're ignorant, and Andrew's in a university, probably even in the chat room because he often turns up in the inaudible 01:01:21. So I've probably just dug a massive big hole, but I'm really looking forward to having him on the show next week, but until then, thank you very much for listening and goodbye.
Stock Photos from sitthiphong/Shutterstock