The podcast where we talk with the people who lead user research in their organization.
Rank #1: 7. Judd Antin of Airbnb.
We kick off the second season with Judd Antin, the Director of Experience Research at Airbnb. Judd and I speak about their model for embedding talented generalists with product teams, skill-sharing among researchers, and just what exactly makes research “sexy.” I don’t know of another way to do things better than to give and get feedback. It should flow like a river. And I think that can be hard, to be open, to focus on strengths rather than weaknesses. To be humble is a thing we’re always seeking to be better at, but that’s how I approach the task of building a team. – Judd Antin Show Links Judd Antin Follow Judd on Twitter Airbnb Trello Alex Schleifer Kathy Lee Janna Bray Steffen Kuhr Natalie Tulsiani Brian Chesky Joe Gebbia Katie Dill Adrian Cleave Improv and user research French Culinary Institute Your Data Are Wrong: The Hype and Reality of Big Data The Incredible Hulk Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript (French version) Steve Portigal: Welcome Judd. Judd Antin: Thanks, Steve. Steve: It’s great to speak with you. Maybe let’s just start as we always do with just kind of the broad strokes. Do you want to introduce yourself? Tell us a bit about you and what you do, what your role is. Judd: Sure. Sure, happy to. So I am the Director of Experience Research here. So experience research at Airbnb is a team that was formerly called Insights, but what we really are is a UX research team. We are embedded in our design organization so we have designers, content strategists, product managers, engineers, data scientists as our partners and we’re a team of 17 folks at the moment. We have a pretty diverse group of people on the team which is something we do on purpose. And everybody is sort of embedded in their teams, working directly on the day to day of products for guests and hosts all the time. So, yeah. Steve: That’s good. And of course those intros, as we know from research, like we could just follow up on all the things that you said. That would probably fill our whole interview. Let me go back to some things. Can you explain Airbnb? Judd: Sure I can. Let’s see if I can do this in a nutshell. So Airbnb is a marketplace that let’s hosts who have space come together with guests who need it. So…we provide an opportunity to travel in a much more local way. So I think you know, it’s – you know Airbnb has been one of the poster children for the sharing economy but I think for us the way that we think about it is that Airbnb is an opportunity to connect hosts and guests together, to have much more genuine local experiences. Local in the sense that you’re traveling to a place that is pro-, potentially just in a neighborhood. You may be staying in a spare room or a place where your host is either there or they’re on vacation and you’re sort of experiencing their city from their point of view and you know getting recommendations from your host about where to go in the neighborhood. So I think that is the thing that makes Airbnb really beautiful and unique is that it’s kind of this view of travel which is to say yeah you show up in San Francisco, the Golden Gate Bridge and Alcatraz and Coit Tower, those are important things that we know from research that everyone wants to see, but if you never left Union Square and Fisherman’s Wharf it would be hard to say that you got a real sense of what San Francisco is like. So Airbnb really lets that happen. Steve: And you started off at that great explanation by contrasting sort of a common or public perception around it – you said “poster child for sharing economy.” Is there something that – and obviously everyone in the company – everyone in any company understands the contrast between how they’re perceived and – I’m just thinking about you in the role that you have and what your team does. Do you deal with that – I don’t know if it’s a gap or delta – the world thinks of us as this. You know we go out and people to sort of understand. I don’t know… Judd: Oh yeah Steve: You’re nodding so I’ll let you answer. Judd: All the time. We spend all of our time talking to guests and hosts on Airbnb. We make a concerted effort through travel and remote research to get outside of the Bay Area bubble for example because, you know, Airbnb is very much present here. But what that allows us to do is see what the everyday experience of traveling or hosting on Airbnb is like for people and I think that means that we get a huge dose of real talk. You know we have a mission – you know sort of our tagline as a company is belong anywhere which is something that we all pretty deeply believe. We want people to be able to have a travel experience which lets them have a really genuine welcoming experience of a place that they can start to feel at home there and belong there. But we learn a lot about how that vision sort of translates to reality in the everyday life of a host, or the everyday life of a guest who is traveling. For host that might mean we expose the diversity of hosts. So for a lot of hosts we talk to the – the money is really an important reason that they do it, but they also feel a strong connection to hospitality. They like to experience people who are visiting them from all over the world and they talk about that as a motivation for doing it. For travelers, similarly, we find out that finding a cheap place to stay is often – I mean let’s – however much – that’s not going to be a part of our vision at Airbnb, but it is a reality of the way that people think about Airbnb when they travel. It is a part of their motivation for getting on the platform. And so I think one of the great things about the research team is that we can be that dose of real talk. We can say like okay this is important. This vision is important. The mission is important. But let’s talk about the real lives of guests and hosts and what matters to them. Steve: How do you characterize – I don’t know, I can only ask it through suggesting an appropriate metaphor that – and I know the synthesis or the ping ponging, if you’ve got – you know here’s public perception, here’s what real people involved – here’s what the real talk says. Here’s what our internal aspirations or you know Kool-Aid or whatever it is for any organization – I mean how – what’s the sort of – tell me about how those things come together. You have to deal with what’s believed and what the aspirations are, with what the world says and with what the real talk from real people is. Judd: I think we have a pretty happy balance between the Kool-aid, or mission driven aspect of what the company does, and the real thing, the real talk of people. What we do most consciously is probably to remind people in this building that we are nothing like the vast majority of people on Airbnb. And that’s where I think the real talk matters because if you make a bunch of assumptions on the basis of what seems obvious or intuitive to us here in this fantastic building in San Francisco in SOMA, like we’re going to get it wrong. We’re going to get it wrong for California, for the United States and certainly for the world, the vast – the majority of our business doesn’t happen in the United States. So I think that is one of the most important roles of real talk is just to say… It’s not that difficult. I don’t think we feel a real tension between the mission and the values and the everyday lived experience of guests and hosts. We just have to remind people that most people’s daily lives are nothing like ours and so we just have to keep that in mind when we make design and product decisions. Steve: I love that that’s not a tension. Those things can work well together. Not every organization is the same and maybe elsewhere you’ve observed that as more of a tension. Judd: Yeah Steve: But, why? How does that work well here? What’s going on? Judd: Here’s my theory. My theory is that it’s because we’re a travel company and here’s a list of things that we don’t have to worry about – advertisers, monetizing engagement. What that means is that to the degree we can get people to find the perfect place for them, where they want to travel, get them out of our digital products and out into the world to travel, that’s success. You know what I mean. And so I think we don’t have to make a lot of assumptions about what motivates sort of digital behavior. We have to get people to travel and I think it attracts a certain type of a person who – and you know that combined with our mission for belong anywhere, we think it’s easy to have that kind of empathy for the world because traveling is such a – as an experience is kind of full of empathy. It’s like understand the world from someone else’s point of view. And as a research team we kind of focus on that too. We’re like hey product team, let us help you understand the world from another point of view, the point of view of guests and hosts that are nothing like you. Steve: That’s wonderful. So the company that’s about getting out of – giving people the chance to experience something differently than what they’ve experienced, in other words travel, is one way that a culture gets created that there’s a hunger to understand – a willingness and a hunger to understand. Judd: Yeah. Steve: The world is different than what we’ve assumed. Judd: Yeah. And I think that’s a big selling point for researchers who want to work at Airbnb because you know they think – travel is such an evocative experience. You know if you look back at the – I think the average person who thinks back over their life to the 10 experiences that were most educational, most instructive, most inspirational for you, a good chunk of them are probably travel experiences. So people’s eyes are open. Like it’s emotional. It’s intellectual. And so as a topic for research, you know facilitating that kind of travel experience is really sexy for most researchers we talk to. It feels really good. It feels really real. You know I think sometimes working in the domain of digital products you can think what does this all add up to? You know what experience I’m really making. Like we don’t have that problem because travel, we’ve all had experiences of travel that are truly sort of transformation. Steve: So what are the experiences that researchers have? Can you give some context or situations where researchers are involved in – are researchers traveling to study travel for example? Judd: Yes. So we have done a fair amount, not as much as I want us to do, but we have started to do travel, primarily as a way of understanding – getting outside that Bay Area bubble. We spend a bunch of time in the homes of hosts. That’s one thing I think we do a lot of. So one thing we haven’t done yet is kind of like I think a travel along project. You know if you research at Uber you could like ride along with an Uber rider or driver. We haven’t done a travel along. I don’t know, that seems a little invasive and weird. But we spend a bunch of time in hosts homes and one of the reasons we do that is because that’s one of those things where it’s very difficult to get a full picture of the day to day of a host, the things that challenge them just in the flow of trying to provide hospitality and use our products without really getting into it with them. Find out what’s their routine of sort of messaging with potential guests – you know scheduling, cleaning, doing key exchange, all of that day to day stuff. It’s really hard to find out the mechanics of that without being there. And it can be very difficult to get the context and depth and empathy we need about motivations for hosting, let’s say, without a really in depth face to face conversation and where better to have that conversation than in the home. Steve: Right. It’s just funny sitting here that you refer to it as their homes. Of course it’s their homes, but I’ve only ever been a guest, never been a host. So the unit that you’re referring to, this environment, it’s a place that I stay. I forgot that it’s somebody else’s home. So to hear you say that, even the language, which I think is probably a key – that’s probably key for you guys to be talking about these as homes. Judd: Yeah. No, I think it’s true. And you know I’ve been thinking a lot about the language we use. You know I think in research – you know I don’t know many people in user experience who use the word subjects. We mostly talk about participants and we use – you know sometimes we talk about users. You may – you know we can talk about whether that’s a good thing or a bad thing, but I kind of always want to turn to guests and hosts – the things we do for guests and hosts because it’s a reminder of – it’s similar to the way you said oh right it’s someone’s home. It’s like right, this is a guest, this is a traveler and this is a traveler who is showing up in somebody else’s space, in their home where they live and sharing that space whether or not the host is physically present at that time – they’re sharing that space. And on the flip-side like this is a host. Not only is hospitality and hosting something that we care a lot about as a company, something we try to promote with our products and look into with our research, but it’s a home. You know it’s really personal. Again, like that is a sexy thing for researchers because it’s like one of the most important things in your life is where you rest your head. And the idea of letting somebody else into that home can seem – you know that’s where the issues of trust come in that are so interesting to look at because you have so many dimensions of that. You have the sense of trust that’s instilled in Airbnb as a brand, Airbnb as a platform. The way we facilitate relationships between guests and hosts. And then you have this face to face personal think where I am letting you into my house if I’m a host or I’m going into your house if I’m a guest. And you know I have a lot of people when I talk about Airbnb that ask about trust and safety and things like that. And the only thing I’ll say about that is it gives me a huge amount of faith in humanity how rarely there’s a serious issue. You know. The number – like millions and millions of nights are stayed in Airbnb’s and the number of serious issues that happen when people trust each other and guests go into homes is just minuscule. From our point of view it’s too many – one is too many. Steve: Right. Judd: But it’s really sort of restoring a faith in humanity and I appreciate that we are constantly on the lookout for doing research that can facilitate that. Like creating wonderful rich, trusting interactions between guests and hosts. Steve: And another reframe for me that’s helpful to think about this is these two groups of people that are being connected as opposed to a bed and a toilet for the night. Judd: And to the end of real talk right. There are some guests who do see it as a bed and a toilet. You know, it’s a cheaper bed and a toilet then they can get at a hotel. But not all guests, probably not most guests. You know the aspect of hospitality, the aspect of getting into a neighborhood, living like a local, it’s really powerful and I think it’s a real motivator for Airbnb. Steve: I want to dive into some other language that you used and maybe get you to unpack that a bit. You said a couple of times that certain kinds of problems or areas are sexy for researchers. So that suggests some insight into what appeals to researchers. Like what we like doing. So what does sexy mean for researchers? Judd: Yeah. So in my experience, when I say sexy – when I say sexy I mean two things. One, a topic for research which is really meaty, which you can really wrap your arms around and spend years at a time diving into. It has a lot of dimensions. It’s not easy. It’s probably a challenge. I think most researchers find the challenge sexy. For example it is something that requires us to deeply understand a problem and figure out how to translate it into design and product and communication. So that’s one aspect of sexy. And then the other thing that’s sexy I think for a researcher is the fact that they can be set up for a direct line to impact product with their work. I think when I talk – I talk to a lot of candidates and when I think about how research is structured in their organization what I hear is well they had to advocate for budget, for a study, and it took them several months to do that. And then they completed the study and then they had to present it up the chain three or four or five levels to get that VP to advocate another VP to get that PM to put it into the roadmap. And it hurts my heart. You know it’s like the worst way to do research. And so I think researchers find it sexy when they don’t have to do that which is – which is one of the beautiful things of Airbnb which is that we don’t have a perfect situation but we have researchers set up as partners, with designers, with PMs, content people, data scientists, engineers, at every stage of the product cycle. The way that they have impact is through being there every day for direct conversations about what we should build? How should we build it and why? Are we doing it right? Iterative development, launch and learn, repeat, you know. And that is sexy to a researcher. Steve: Can we talk more about that staging or how people are set up? So there’s a number of individuals with different roles that are partnered together. Judd: Sure. So, yeah – so we have an embedded model for research where – which I would contrast to the more service-based model where there’s like a central organization that sort of fields requests and sends out a researcher into the field. Our model is an embedded model where a researcher gets situated into a product. Let’s say that product is search. That researcher sits physically next to, hopefully between the PM and the designer, the engineer, the data scientist. And I make a big deal of that physical presence because I think that’s where a lot of the action is. There are a lot of – I’ve had conversations with researchers before where they say I need to get into that meeting. I go, what meeting? And they go that meeting where all the decisions happened. And usually I look into it and find out that meeting doesn’t exist. That meeting happened like in between her desk and his desk at 1:45 and it happened because – like I’m elbowing a shoulder right now. Steve: Um-hmm. Judd: Like that’s why it’s so important to physically have that person there, physically present for everything that’s going on in the product cycle. And that person finds partners in all the other functions. So certainly with design – we have very close relationships with design, product managers. We have a strong and wonderful data science team here. Engineers of course. Content strategists. Everybody is kind of at the table, one very collaborative team trying to make great product. Steve: So I have a dumb question and we can say there are no dumb questions. Judd: There are dumb questions. Steve: Thank you. Thank you for affirming my dumbness. You know product cycles, development process focuses on different tasks are hot and heavy at different points. So I’ll just ask this in the dumbest way possible, what’s the researcher doing at different stages of that? What does it look like for them? Judd: Sure. Yeah. So I guess it would start with an exploratory or formative stage where we’re trying to figure out what we should build. What are the problems? What do they look like? Why are they problems? So very often that’s a place for in-depth qualitative research. We love to have a model where we do in-depth qualitative research and then we follow it up with really rigorous larger scale survey work to check prevalence. So that kind of one-two punch is really powerful for us so we can say here’s a deep, contextual rich understanding of these problems and we know a lot about how prevalent they are and we can segment them a little bit in ways that matter to us. So cool, we did that. Okay now maybe we have a product road-map. Or we’ve decided to build a thing in a certain way and the designers were hopefully participants in that research from the very beginning. Very often they are in the observation room and they’re coming on home visits. They’re participating in feedback on interview guides and surveys, all that stuff. But then I think they really kick into gear at that point where the implications for design kicks in from the study and we are starting to do sketches and ideation and visioning work and I’m a huge advocate of low fidelity prototype testing as a way of sort of pointing the ship. So at that point the researcher might be doing sort of a rolling study where at first mocks maybe Framer prototype, sort of golden path, Wizard of Oz-y things that are a little bit more rich and interactive. Doing repeated user studies with those. And then at a certain point in our cycle we’re sort of iterating and we end up passing off to engineering. Again, like the engineers are often there through the whole process, but then the sort of hot and heavy engineering kicks in. At that point there might be some more design refinement that can go on, or if there’s a concurrent project that the researcher is working on. But we get towards a stage where we have a functional prototype we can test we tweak that before launch. We launch. Now we can talk to real users in the wild and we get all that great ecological validity and then we’re back to square one in a way which is okay. Maybe we worked on a thing, we improved it, but a whole new set of problems reveal themselves. Like no design solution is perfect and so the researcher’s job is never done. Steve: That’s awesome. I feel like applauding that narrative. So everybody’s busy doing this stuff. Again, I guess that’s the embedded model kind of brought to fruition. In a service model you probably wouldn’t either have the resources or wouldn’t choose to do it that way. Judd: Yeah, I mean – so here’s – like the way that I think about that trade-off is that – the greatest thing about a service model that I can see is that you can really provide the person with the perfect expertise for a product – for a research project that’s being requested. We have to have a model that’s a little bit more like a generalist. So people on my team are generally expert at something. So maybe they’re just more passionate and experienced with diary studies, or surveys. They do little stats. You know we’re a very multi-method team. But they kind of have to be a generalist because whatever – if you’re embedded, whatever research need is kind of thrown at you, you need to handle it. You need to handle it. You need to collaborate with another researcher who’s an expert. If it’s sort of at the edge of your skill set you need to work with the data scientist. But it demands this generalist model. And then the other thing that I think is important is that in the service model – like I – we don’t spend a lot – like the amount of time we spend on reporting varies a lot, but it is generally not a huge proportion of their time. I think of a huge amount of time spent on beautiful Keynote or PowerPoint decks as the price you have to pay for a service organization rather than an embedded organization because when a researcher is embedded presumably that PM, that designer, the engineer, they were along for the whole thing. They’ve seen it at every stage. They were in the back room and you debriefed after every session or at the end of every day. They participated in the affinity diagramming and so we may still do a video reel, cut together some clips. We may put together a Keynote, but that’s not the primary deliverable. They’ve been getting it the whole time. It was a constant back and forth. I just think that’s so much more efficient and it lets a researcher develop like real product expertise in that area rather than parachuting in and out all the time. Steve: Are there ways that – as you said they have to work as generalists. But you have diverse staffing, people with different backgrounds and different experiences. Is there any sense of kind of community practice that you get very specifically in that centralized model? Like the researchers sit together and they share stuff, but what’s the gestalt you can create in this model? Judd: I think we have to work just a little bit harder for it because the peripheral awareness isn’t necessarily there, the way that it would be if you had one team and sort of everybody knew who was allocated to which project at which time. And honestly like we don’t have a perfect solution to that yet. I’ve never been a part of a research team where we didn’t have a problem with that type of information sharing and it’s really important and we’re trying to do better at it. Actually we’re talking about it right now because number one we want that community of practice. We want feedback to flow. We want people to find beautiful, like synergy of research questions. We don’t want to have people doing the same project in parallel or reinventing the wheel. And all those things require that kind of peripheral awareness of what everybody is working on, or the ability to get that information quickly. On the other hand you have this sort of classic, collective action problem where if it’s too onerous for any one person to provide that information, or consume it for that matter, then the whole model falls apart because it’s sort of only as strong as its weakest link. So we have to figure out how to create a lightweight model. And to be honest the way we’re doing that right now is with Trello. I don’t know if you ever use Trello? It’s a pretty low tech tool in the sense that it’s like the closest analogy – Alex Schleifer, our VP of Design and my boss, he calls it like a digital whiteboard. It’s got columns and cards and columns. So a product team has a column and a card is an active project. And it might have a link to a research brief or an interview guide, but it’s just that simple. And that’s it. And people can check that to find out what’s going on. They can update it. They can move a card into the done pile. I don’t think it’s a perfect solution, but you call that a real problem that we work hard at. Steve: So everyone can see what’s being researched? Or see what has been? You can look in the done pile and see what’s been done? Judd: Yeah. So we’re still working this out, but the model is that things that are recently completed stay in the done pile. Eventually they kind of get archived and they’ll live in kind of a research repository elsewhere. But things that are active or recently completed, it’s sort of a one stop shop to view them all. We’re going to try it out and see how it works. Like I don’t think there’s a perfect solution for this, but it’s something that I hear from the team that we need to work hard at. Partly because we find those collaborations and cross-pollinations and partly because it helps us feel like a team and like we have that community of practice and that’s something that we work on actively promoting. Steve: That sounds like another sort of – that’s the tension of the distributed – I’m sorry, the embedded vs. the centralized model. Not that the centralized model has it figured out because you know creating deliverables – I don’t know how many times I’ve encountered the like – I’ve had – I’ve been the vendor outside an organization and had people in that organization come into a new role and not know of the project and have to reach out outside the organizational walls to get that collateral or those deliverables from a vendor from a year ago, me. So it’s not like if you – it’s not like the centralized model always works for that. The deliverables just have a shelf life and they disappear. Judd: For sure. Steve: So is this knowledge management we’re kind of talking about here? Judd: Yeah. You have somebody with institutional memory and they’re always there. Although of course moving from team to team to sort of cross-pollinate is a good thing, but we probably do that on the order of like 18 months to 2 years. I think everything is a trade-off. I choose to optimize for perfectly positioning a researcher for impact and to be a voice, a constant voice in the product process. The trade-off of that decision is that we have to work harder at the community of practice bit. Thankfully we have a really passionate and collaborative group of researchers at Airbnb who are devoted to that, actively seeking that and we’re small enough that we can have – you know small enough physically in that we can all sort of see each other in the space of 30 seconds and that we can meet in a reasonable sized room and share ideas on a regular basis. Steve: Are there – do you have regular meet-ups or certain types of activities that are meant to make that happen? Judd: Sure. You know we are in the midst of a concerted effort for information sharing. So we meet all the time. We meet every Monday morning for a standup which is sort of go around the table and everybody say the two things they’re working on that week. We meet every Tuesday for a kind of what I could call more like a team meeting. Like we get into it. We talk about recruiting and hiring. We often have special topics. We talk about growth. We have – growth in the sense of team growth and individual growth. We talk about sort of big picture product and design context stuff that’s going on. We also have something that Kathy Lee (one of the researchers on the team) put together which is Shop Talk. So it’s like a Friday event in which people can have a beer and give a presentation they would have given to their product team to the other researchers. And we’re also instituting skillshares. I sort of demand that everybody on the team – so my model for this is sort of, in my memory as somebody who got a PhD – so when we get a PhD you become the world’s foremost expert at some tiny little thing that nobody cares about and then you write a dissertation which if you’re lucky your advisor will read and then you’re kind of done with it. But, so clearly that’s why I’m not in academia, but one of the great things about that model I think is that you carve out this niche for yourself which is your calling card and that’s a thing that I ask every researcher on my team to do. Yes there’s this baseline that we must be rigorous, beautiful generalists, but what is the thing that you’re uniquely good at as a researcher? What is your calling card? What is the thing that you want to at first sort of really consume all the materials you can find about, go to seminars, take workshops and read books on and then before you know it you’re writing the books and you’re teaching the seminars and you’re giving brown bags to the rest of the team. I just think if you have a team where everybody has that unique team, an aspect of methodology or practice, communication or whatever it is, that they are passionate about developing into a unique skill and they’re all committed to sharing it with each other, that’s my model of a team that grows individually and grows as a team. Like that sounds awesome to me and we’re work- you know I don’t think that we are perfect in that regard, but we have that as an aspiration. Steve: What are some types of things on your team – topics for different researchers, that’s there thing? Judd: Sure. There are methodological topics – just as simple as survey best practices. So we have somebody on the team, Janna Bray, who has made it sort of a thing she’s really vocal and generous about to raise everybody’s game at surveys. We have another researcher, Steffenn Kuhr who is really passionate about rigorous evaluative research in which you are trying to bring as much objectivity to the sort of usability and user testing process as you can and he has some really interesting ideas about how to do that that he’s honed over his career. There are also folks who want to bring aspects that are a little further from sort of core research methodology as their special sauce. So for example, Natalie Tulsiani is a researcher on the team who recently gave a workshop on moderating the observer room. So the topic is as a researcher moderating the group of people who are observing a study. That just kills me. I’m like that is so great. You know what I mean. I worked with a researcher at Facebook who her thing was improv comedy. You know she thought improv could make you a better qualitative researcher and I believe she’s right. So that was her thing. She was passionate about that. I think it could be all over the map, but I just love the idea that everybody on the team has their thing and they’re just chasing it and bringing everyone along with them. Steve: Those are great. Those are good examples. So the idea of the individual passion is a driver here. Judd: Yeah, I mean look, the research shows just in general that when somebody stops growing in their job and they feel like they’re in a rut they’re going to leave. So just from a purely practical point of view, it’s like in my interest as somebody trying to build a team to keep everybody growing around things they’re passionate about. But I also think that’s the shortest path to everybody building everybody else’s game to be stronger. Everybody raising everybody else’s game. We will – that is the path to becoming a world class research team and I think it’s both demanding of an individual to always be raising their game and demanding of that person that they be generous and collaborative with the researchers around them to do the same. Steve: Right. So as a member of that team I am getting energy and reward from being able to follow that passion, but also I’m receiving from others. Judd: Yeah. Steve: From them, so I’m always growing and developing as part of that team. Judd: And if you do that I think developing that community of practice is easier. You know it’s easy to see where it comes from and how it grows. Steve: So you’ve talked about growing the team. Can you go back in time and describe a bit of the history, the evolution? Where you’ve come from? Where you’re looking to go? Judd: Sure. So my history at Airbnb is admittedly a little bit short. I’ve been here since the middle of May, so what amounts to about 6 months. I was the 10th researcher. As of this week we have 17. So we’re growing quickly, but we’re not growing for growth sake we’re growing because – I think because the embedded model we have means that awareness of research is high, the team is producing great work, and it’s just I’m getting more demands than I can reasonably service right now. So I need more researchers and that’s a beautiful problem to have. Steve: So there are teams that don’t have an embedded researcher that… Judd: That’s right. Steve: That team needs somebody. Judd: That’s right and we’re in this moment of accelerated hiring which is necessary in my mind because there are enough teams that have no point of contact, that getting – I mean going from zero to one is sort of technically an infinite improvement, right. And in the research that we do it is really dramatic at that moment. And I think this is – so we’re trying to take advantage of that inflection point. A lot of research teams in industry go on a ratio, right. So a ratio of designers to engineers, researchers to designers, for example. So they might say like 3 to 1 is the right ratio. One researcher to three designers, or one researcher to three designers and PMs together. And what we’re trying to do is just get ahead of that and then back off. And the reason that we’re trying to get ahead of it right now is because we’ve – I think we’re at that moment where having a research voice in all of these teams that have sprung up and don’t have any research representation is going to be disproportionately valuable. So we’re trying to concentrate growth right at this moment and then we’re going to slow down. Steve: So there’s a context here of sort of shift in product development – let me see if I can ask this properly. You’re describing the teams are springing up so that says that what the product is is evolving and growing – do you call the thing a product – what you as a company provide and the different pieces you’re creating to do that… Judd: Yeah. Steve: …is evolving and growing so there’s a thing that’s being made that maybe wasn’t being made at some point and that’s a new team? Judd: Right. Or you know – and I don’t – I’m not an expert in the growth of companies or product teams of course, but in this one, you know I think as the product organization matures there are two things that happen. One, new products spring up. The other, existing products, the pie grows, right, so that the amount of work is – becomes specialized enough to carve out a new team. It’s just too much. So where there was just one team before that for example looked after all the things we do around host dashboards and host tools, for example, well now we’ve a reached a scale and a complexity of the product such that there are several important work streams. And then they break them out and PMs and designers and researchers and data scientists specialize in those areas. And so I think – that’s a thing that’s kind of been happening, like spawning teams at Airbnb, and in that moment having a researcher at the table is super crucial. We don’t want to lose that momentum and there’s enough important work there – a lot of those people have come from other teams where they had a researcher and they’re like hey I’m working on this new thing, where’s my researcher? I love that. I love getting those requests. Why isn’t there a researcher in the room for this. I means I have to say no more than I want to, but I’ll take that problem any day. Steve: Are you trying to hire with specific team roles in mind? Judd: Mostly no. Mostly what I want to hire for is talented generalists and that’s because of what I think of as the virtue of that embedded model and because I don’t want to create – well I don’t want to create an environment in which there’s a bunch of people who are really only either happy or well suited for one type of work because stuff changes. The product moves. The demands of research shift such that okay I need to move some researchers around to tackle this new product area and I don’t in general want to have researchers who I think I’m going to make dissatisfied or their skills aren’t going to be a good fit if they need to slot into a new areas in 6 months or a year. Steve: So you’re using the demand to figure out how big your team needs to be, but it’s not sort of slot filling, it’s capacity building. Judd: Yeah. Overall capacity building. I think if we chase anything at all as a team we chase diversity. Like I want a team full of people with different methods, different perspectives, different backgrounds. We just hired this talented researcher named Andrew Sweeney who is going to be our first researcher in Portland. Portland is where a lot of our Customer Experience team is and so Customer Experience, those are the people who answer the phones. If you have a problem before or during your trip that’s what they’re for and they have a whole suite of internal tools that they use to handle these problems, to track information about cases that are open. Those tools are not as good as they could be and we have never done systematic research on those tools. So in comes Andy Sweeney who has – Andrew has a background working on complex, information rich tools, right. So his approach to the problem is different than the one that I would take. He goes to system usability scale for example which is not the first place that I go, but that’s brilliant for this situation because we’re talking about understanding information flows in a complex information environment. Efficiency is important because we want to enable Customer Experience agents to be going as fast as they can and not blocked by the fact that they have the information tool, like a CRM, and three spreadsheets and a notepad and their workflow is just broken. So chasing that diversity as a team is something that I think goes to the earlier point about raising everybody’s game where everybody is helping each other out. I want to learn about system usability from Andrew. Steve: And so Andrew’s – he’s tackling a new problem space. That’s not hosts or guests? That’s the new organization and being able to deliver a good experience. Judd: That’s right and I think he – so in a way we’re taking on a new sort of constituency which – as a research team – which is our internal CX agents who we call crewbies. So we have hosts and guests and crewbies. Steve: Say the word again? Judd: Yeah, crewbie. People who work in CX, manning the phones and doing Chat and email. They call themselves crewbies. But I mean they’re the most talented, empathetic people on the planet. Like we – many people who work here do these shadowing sessions where we’ll sit with crewbies and sort of just observe their day to day and we do that to gain empathy for the problems that guests and hosts have during and before trips and after trips. But these are some of the most patient, amazing people on the planet and we’re, you know, in the process of working towards better tools for them. Steve: Okay. So that sounds like a marked evolution in what – in how the work that your team is doing is impacting the company. Judd: Yes, it’s a more internal focus than we’ve had before. I think it’s fundamentally the same sorts of things which is like do rigorous research that is deeply understanding the problems and needs that people have and understanding potential solutions, but it’s in the context of making our business more efficient and providing better service. Steve: So will Andy have access to the designers and the engineers that are creating those tools for the crewbies? Judd: Oh yeah, there’s an entire products team that’s spun up in Portland and in San Francisco just to work on this. Steve: So once you have a product team then that’s – in your embedded model – that’s when the request for the researcher comes? Judd: That’s exactly it. And I think I look at a lot of things that way. So I could have hired Andrew 6 months ago when there was no product team, but the reason I didn’t is because he would have had nowhere to land. I think that’s setting a researcher up for failure. You know I want to know – you know a researcher can do the best work in the world, but if there’s no landing zone for that research then they’re not going to be able to have a lot of impact most of the time. Steve: So at the risk of re-asking a question I brought up earlier, we were talking about slot filling or not, is Andrew someone where you saw his particular – what made him a diverse candidate? Can an individual be diverse? I don’t know. What was unique about him that contributed to diversity? It sounds like the thing that he was brought in to do is perfect. For the company, perfect for him. Six months ago were you saying that the right project for him wasn’t there? We talked before about capacity building versus slot filling, but I’m wondering is there sort of a nuance to that that goes with the diversity aspect? Judd: Yeah, I mean – I guess I would be lying if I said that there was never an appropriate time to look for a specific skill set and that it was always just general capacity building, looking for great researchers. This is a case where absolutely I thought I’m sure we can find an incredibly talented researcher who has experience working with information rich internal tools and making them work better and understanding complex workflows and efficiency and all this stuff. I’m sure we can find that person and low and behold we did. At the same time I think, you know Andrew has done a bunch of things in his career, not to emb- I’m sure he’s going to be embarrassed about this and – and he just started on Monday by the way, but – he’s done a lot of things in his career and he would be perfectly capable of filling almost any other slot. He just happens to be a great fit for this role. Steve: Let’s go back in time. As you said you’ve been here “x” amount of time, but you must have some sense of the history. You were #10 you said. Judd: Yeah. Steve: Do you know the story of #1? Like how did research become a thing that was hired for and done? Judd: Well research is in the lore of Airbnb. One of the stories that gets repeated internally is about one of Airbnb’s early investors who at a very early point in the company turned to Joe and Brian (Brian Chesky and Joe Gebbia, two of our founders) and he famously said go to your users. And so they spent several weeks in New York City with some of the first hosts on the platform, deeply understanding what they were going through and what their experiences was like. And Joe and Brian are designers by training. They were at RISD and I think that having designer founders – having you know two of our founders be designers has sort of infused this kind of design and research sensibility into the whole thing . And that’s been beautiful for us as a team because I don’t – let’s say to the degree anyone cares about buy-in from the top, which I think realistically every team needs to care about, relatively speaking we do not have to worry about that. We have people like – we have a whole bunch of leaders saying where’s the research. Let’s look at the research. Go to your users. So it’s really embedded into the lore of the company. I wish I could be more specific about the history of this team in particular, but honestly I can’t. Steve: So with that caveat I’m going to ask another specific question which you can then throw back and say you can’t answer. Especially for this podcast, I think an implicit topic – I think it’s implicit – is this idea of research leadership. And the trajectory often in organizations is a researcher is brought in. Sometimes they’re a junior person and they kind of work for a UXer design person, depending on the industry. And then some organizations there’s a point at which it’s acknowledged that this is a specific thing and it needs to be led, not by someone who’s a designer, but by someone who that’s what they do. That’s my sort of general sense of the pattern and I wonder, do you know when did that happen at Airbnb? When was research a leadership – when was there leadership in research? Judd: Yes I do. Steve: See! You said you couldn’t answer it. Judd: I can answer a few questions. So in the beginning of the team it was just one or two researchers and obviously the design team was much smaller and it was just one big team. Everybody reported to Joe Gebbia. And over time what happened is the design team grew. I think the research team, which was at that time called Insights, was not growing at the same rate, and partly because I think of a – the fact that it’s hard for somebody who’s not a researcher to know how really best to leverage and grow that organization. But the partnership was there. They were doing great work and then at a certain point Alex Schleifer showed up. Alex is our VP of Design. He’s been at Airbnb for about a year and a half and he took over leadership of the organization and almost immediately recognized the need for research leadership. And it took him awhile – it took me awhile to meet him and I think to his credit he gives me a lot of freedom and says like Judd I want you to build a world class organization and I want to support you in that, what do you need? And he gives me a huge amount of context and feedback, but he’s set me up with his team such that my peer now is Katie Dill who runs sort of the organization of designers. Adrian Cleave who runs our design operations organization – but we are Alex’s first team and he treats us like peers and so there is no sense that even though I’m in the design organization I don’t have the sense of – that some – of research being – the sole value of research being to feed design for example. I feel genuinely like we are equal partners in a product process and I give Joe a lot of credit and Alex a lot of credit for recognizing that that was a useful and valuable thing to do with research from a pretty early time. Steve: So the position that you came in to take was a new position? Judd: It was a new position which was to lead the research organization. Yeah. And I was and have been given like a fair amount of leeway to say how should we do this. And that feels great. I don’t have the answers. You know when he said that I was like great I don’t know how to do that, but I’d love to work with you and figure it out. Steve: That seems really key and I don’t know if that’s just part of the overall culture or something that is unique to you, but – and I’ve had this conversation with people a number of times about knowing the solution versus knowing how to get to the solution and the comfort with the ambiguity. You’ve mentioned a number of times in this conversation that we don’t have this figured out. We don’t have the best solution but we’re trying this. Judd: Yeah. I think anybody – you know it’s interesting because we were recently having some conversations about the structure of product teams in general and how hard that is and going why hasn’t this been figured out by now? Shouldn’t there be one canonical way to set up a product team? No. There is no one canonical way because every business is different, every product is different. Values are different and that implies different things for structure. Well I think it’s the same thing for research, for design, for product overall. I feel everything that I do is to make good principal decisions in response to the realities of the situation at Airbnb. To try to build a research team which is uniquely responsive to Airbnb and at the same time embody some qualities that we really care about – world class rigorous research, perfectly positioned researchers every time. Those are kind of our two mantras. Center of excellence, perfectly positioned. And those are sort of – there’s no playbook for that. I think we apply those principals. We apply them systematically and we communicate a lot. And I don’t – it sounds stupid when I say it out loud, but I don’t know of another way. I think I’ve been really, really hard with my team and they are embracing of the idea that feedback is the most important thing we do. Feedback for each other, feedback for me, feedback for me with them – because I say to them I don’t know of another way to do things better than to give and get feedback. It should flow like a river. And I think that can be hard, to be open, to focus on strengths rather than weaknesses. To be humble is a thing we’re always seeking to be better at, but that’s how I approach the task of building a team. I welcome any and all feedback. I don’t pretend to have all the answers. I just think I have a principled way to get there. Steve: I’m going to ask sort of a clichéd research question but I think it follows from what you’re saying. In a purely speculative way, like if we were to be talking in five years let’s say, what do you think – what could you imagine the state of research at Airbnb being? Judd: Five years is a long time. Well, hopefully we will have scaled to the point where we have what we would consider full research coverage. To me that means being sort of lean and agile, but having a researcher represented kind of at every level, from the ground level up to the leadership level and we’re definitely not there yet. It takes time to build that capacity. You want to build it from within too so we’re growing that capacity at the same time. I also think in 5 years we’re going to be a company that has already hopefully long since already embraced the idea of like a really global/local product so that we have product teams that are staffed all over the world and I have researchers embedded in those teams and they are connected back to San Francisco and they are providing this very difficult thing which is how do you create a product which is sort of you know 80 or 90% core, but that 5% or 10% on either edge is the real local bit where the rubber meets the road. So how do we create a uniquely Singapore product or a Germany or Brazil product where we represent belonging in a way that resonates culturally where we have tweaked our onboarding process to highlight different value propositions. Where we do special things that facilitate trust between guests and hosts that are unique to an Indian market for example. And I think that’s going to take an organization which is global. Like I don’t think that sending researchers from here out to those places is going to cover that. So that I hope sooner than 5 years will be true. And then the last thing I’d say is that I hope that by that time we will have created not just a product organization, but an entire company which is totally driven by research and insights in which – you know I don’t think that we need to do all the work, but whether we’re talking about marketing or legal or policy or local operations or customer experience there are research questions in all of those places and I want us to be able to have an organization where somebody on my team is the glue to every one of those teams. Steve: So taking embedded beyond product? Judd: For sure. Because I mean – for example name a – it’s a little bit stupid to me that most organizations have a product research and a market research organization. Name a product question which doesn’t have an influence on marketing and advertising and vice versa. I’m sure somebody could snark and think of an example, but that would be the exception that proves the rule, you know. I’m not saying – there’s no need to have one big kumbaya team, but there is a huge amount of value to saying like look fundamentally we’re just researchers applying our expertise in different disciplines and in different ways, let’s be more together than not. And in my experience most organizations are not like that. So I don’t know. That is a big aspirational goal for this company. Steve: I love it. Just so well put. So that’s kind of the looking ahead. It’s a lovely vision. I want to go back. You mentioned a few things about yourself – PhD and some other places that you’ve worked. Maybe you could just talk about – I’m just curious about sort of parts of your background, whether it’s professional, personal. Judd: Yeah. Steve: Go back and talk about some of the things – what are some of the things that are in your background, experiences or education or whatever, that are really present for you now in the way you’re looking at the world and the kinds of things you’re trying to make happen? Judd: Yeah, I mean, I guess if I – that’s a really – you’re forcing me to be introspective. Well I began – as an undergraduate my major was cultural anthropology and so I began my career as an anthropologist. You know very focused on meaning and understanding and writing culture and the idea of culture and what it meant. And I feel like that’s really important to me now, especially at a company that’s focused on travel. It gives me a lot of empathy. You know I feel like I studied a lot of kind of epistemology that gave me like a fundamentally subjectivist approach to research which I think makes it valuable – makes me a better multi-method researcher and leader in the sense that I think objectivity is a myth. That I think everything has a cultural and social lens and all we’re doing is like seeking confidence and that confidence is built through multi-method research, through looking at the same problems from multiple directions and perspectives. So cool, if you do that there’s no territoriality. We just need all the methods. They’re all flawed and they’re all powerful. So I think that was influential for me. My PhD was in social psychology and information systems so I took a real jag into experimental social psychology and data science. That I think informs me because I, you know, have had now very deep experience with both the most ethnographic qualitative work there is, you know in which I spent six months at an after school arts and media program in the Bayview learning about informal learning outside the classroom. And I’ve done a huge amount of experimental social psychology and data science and I can appreciate – you know I can hold my own with basic statistics and write R and Python and PHP and talk to an engineer in code and stuff like that. So I think having had that experience is deeply influential for me. And the last thing I would say is that I think – so after I graduated from undergraduate I went to culinary school. So I spent six months getting a degree at the French Culinary Institute in Manhattan and then I worked as a chef for awhile. I actually – I was not destined to work as a chef. That was an extremely difficult life and I found that out and was grateful to have kind of something to go back to which was anthropology, and graduate school. But having been a chef, having been somebody who worked with my hands and really embraced that kind of creativity and appreciated flavor and the craft of perfect taillage, like knife skills. I think that helps me have like a little bit of a window into a design world. I don’t think that I have a lot of a window compared to the amazing people I work with on the design team, but I think it gives me an appreciation for craft, for art, for the details, for the menial but beautiful handwork that makes great food for example. I think I can apply that to research and to design. Steve: That’s awesome. You know we talk about researchers as kind of translators and even this – the diagram you drew of somebody elbowing somebody they’re sitting next to. It’s kind of like a translation thing and it seems like the experiences that you’ve had have given you a lot of vocabularies to translate between and maybe even the translation was sort of an ingredient in those. Judd: I agree with that. I think – you know recently I gave a talk – I’m not sure I should say this, but I’ll say it anyway. Recently I gave a talk at a learning lunch here at Airbnb, mostly to a data science crowd, and the talk – this is also a talk that I gave at a design conference in Philadelphia. And it was basically a social psychology talk. It was about the ways that we are all biased. It was about confirmation bias and minimal group bias. It was about post hoc bias. And like the point I was making is research is human and research requires human – humans. Every type of research does and humans are flawed. We cannot avoid our biases. The best we can hope to do is counteract them. By hanging out with each other. By trying multiple things. And that talk got me into a little bit of trouble only because I think it seemed like what I was doing was being unduly negative about different methods, and I was. It was kind of one of those talks where you crap on everything and that was my point. It was like okay everybody spends time exulting the virtues of their particular method and that’s great. They’re all strong, but they are all weak and we are weak. And so I think part of the translation for me is to be able to speak to everybody and go okay I get the strengths of your method and I get its weaknesses and this one too and this one too. So let’s just get past that and work together. Steve: And that’s being human. Judd: How can you avoid being human? Steve: Yeah. Is there anything we didn’t talk about in this conversation we should cover? Judd: I guess one of the things that is on my mind a lot is, because I’m growing a research team, is the idea of responsible growth for a research team. And what does that mean? I’ve been thinking about it because we are growing and I want to make sure that we do it in the right way. I’ve experienced both types of growth, responsible and irresponsible. And the way I think growth gets irresponsible is not even really in a headcount way or a budget way. It’s probably more like a communication and culture way. So the things that I want to promote in the content of responsibly growing this team at Airbnb is radical transparency and communication. The worst outcome in the world is if new researchers look at old researchers look at managers look at me and go I have no idea what these people are thinking. Or I don’t know how to plug into this decision making process and I don’t feel like I have a say. How did this get to be this way? I don’t think it’s right but I don’t have a voice. Okay, terrible, terrible. Need to avoid that at all costs. So the way we do that is by being fundamentally transparent and collaborative. Like everybody from me on down, we talk about what we’re thinking and why we make the decisions we make. We open them to feedback and maybe we form a committee in which everybody has a voice. Okay, committees, that sounds kind of bureaucratic, but what I mean is like involve everybody from the intern to the senior manager in doing something like how should we build a skillsharing system? Cool, we can all be involved in that. And then the other thing I think is responsible growth is making sure that everybody has a path forward in their career because other than feeling like you’re stuck in a rut, the other thing that I think is a recipe for a researcher being dissatisfied is feeling like they have nowhere to grow. And so making sure that everybody feels like they have a path is really important to me as the team grows. That path is building out your unique niche, working on your core skills and expanding them, taking on more stakeholders, taking on more senior stakeholders. And you do not have to become a people manager to be a lead. That is another thing I feel like is really important because people management is a set of skills that is unique and crazy difficult and learned. And you choose to focus on them. Not everybody wants to do that. It is absolutely not the case in this organization that the only path forward toward seniority as a researcher is through people management. It can be through research leadership, through product leadership, not just people leadership. And so as we grow I think the responsible bit is making sure that all of those paths for growth are open to everybody and everybody knows about them. I spend a lot of time thinking about that because it’s scary to think that we were 10 when I started, we’re 17 now. I don’t know, we’ll be 25 or 30 in the next year or year and a half. That’s a lot of growth and everybody thinks the next person in the door is going to be the one that changes the culture, but I don’t think that. But I think we should be deliberate. It should happen on purpose not by accident. That kind of growth, the planning around it. So that’s what I seek by doing it this way. Steve: That’s fabulous. Do you have any questions for me? Judd: Why are you doing this podcast? Steve: Why am I doing this podcast? You know it’s self-serving I think. I’ve been around long enough that the best work was being done by vendors. Let’s just say that. And I still am one. I hate the word vendor… Judd: The “v” word. Steve: …we all know what that means. You know people who work outside organizations. Teams like – companies like this didn’t exist. Teams like this didn’t exist. Leadership roles like yours didn’t exist. It’s a big change that’s happened in the last few years where now – I mean the kinds of vision for what research can be and how it can impact. You know it has to be done inside. It’s not to say that – I’m not saying that my work doesn’t have value, it’s just different. If you work with organizations the context has shifted. There are people inside organizations that have roles and titles and responsibilities that didn’t exist before. So my professional life has changed. So it’s self-serving because this is a chance for me to learn about this shift. It’s fun to be able to do that. It gives me an excuse to have conversations with you and learn things I wouldn’t otherwise learn. And you know I think much of the best work – whatever the percentage is – amazing things are happening inside organizations. So that’s the place to kind of look and learn. I’m not – I like working outside organizations so when you’re a consultant or a vendor you journey from place to place, like in The Incredible Hulk. You know you have these adventures and then you have to leave at the end which is an obscure reference for people that didn’t watch the T.V. show in the 80s or 70s or whenever that was. Judd: I didn’t get it. Steve: Okay. Judd: I liked it though. It’s a good reminder of how young this field really is. Steve: Especially in the form that it’s in. The conversations we’re having about change in design, insourcing and acquisitions and so on. I feel like research follows design and kind of we’re trailing by a couple of years. But we’ve seen research firms get bought too, not just design firms, you know in our recent history. We’ve seen that happen. So I’m curious. Judd: As much there is no one canonical way to build a research organization there’s also no information out there about how to do that. And so to have a resource where you can learn from people who are trying to build those organizations is really valuable to the UX community so thanks for doing it. Steve: Alright. Well it was great to speak with you. Thanks so much for being a guest and for being my host here today. I’m throwing those words out in a really confusing way now. Judd: Nicely done. Thank you, Steve. It was a pleasure.
Rank #2: 15. Leanne Waldal of New Relic.
Welcome back to Dollars to Donuts. This episode features Leanne Waldal, Senior Director of Product Research at New Relic. We talk about establishing research in an organization for the first time, building up a diverse set of research collaborators, and the pleasure of taking on certain types of challenges. I’ve seen hopeful examples in startups recently where even though there are only 3-5 people they bring in someone for research and that’s highly unusual. Usually, if you’re starting up a company you have seed money, or you have your first round, the first people you’re hiring are engineers. You’re not making one of your first 10 hires a researcher. I know some example now around the San Francisco Bay Area where they’ve actually brought someone in whose role is to do research really early in the company. And that either points to a certain amount of humility around, oh, just because I am the user it doesn’t mean I know the users. Or, because they’re going into a space they don’t know. – Leanne Waldal Show Links The Secrets of Lyndon Johnson’s Archives, by Robert A. Caro The Turnaround Leanne on LinkedIn Leanne on Twitter New Relic Autodesk Dropbox Site Reliability Engineering EPIC LUMA Institute RACI model Berkeley School of Information Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes. Transcript Steve Portigal: Well, hi, and if you’re a new listener, welcome to Dollars to Donuts, the podcast where I talk to people who lead user research in their organization. otherwise, welcome back to Dollars to Donuts, the podcast where – okay, you got it. It’s been a little while since the last episode, and I’m really happy to be back making new episodes for you. In a little bit I’ll talk about how you can support me, and the podcast, but first I wanted to mention an interesting article I’m reading. It’s from The New Yorker magazine, from the January 28, 2019 issue. It’s an article by Robert Caro who is an author of among other things, a multi-volume biography of Lyndon Johnson. This article is adapted from his memoir called Working: Researching Interviewing Writing, and his recollections about those activities are what caught my eye. He describes going to do research at the Johnson Library and Museum and going through boxes and boxes of papers. It’s the hard and tedious work of investigative journalism and what struck me was that amount of inferring, and cross-referencing and delving he was doing – he was coming up with facts and perspective that were not there for the taking but were from analyzing and synthesizing the information that he did have as well as what he didn’t have. Later in the article he talks about the importance of “place” in his research, that to understand Johnson he had to leave New York and really spend time in the Texas Hill Country. He writes “As soon as we moved there, as soon as the people of the Hill Country realized we were there to stay, their attitude toward us softened; they started to talk to me in a different way. I began to hear the details they had not included in the anecdotes they had previous told me.” And finally, he talks about silence in interviewing, citing two characters from fiction – Inspector Maigret and George Smiley, and their tactics to quote keep themselves from talking and let silence do its work unquote, where Maigret fiddles with his pipe and Smiley uses his tie to polish his glasses. Carrow has his own brilliant technique: “When I’m waiting for the person I’m interviewing to break a silence by giving me a piece of information I want, I write SU (for Shut Up!) in my notebook. If anyone were ever to look through my notebooks, he would find a lot of SUs.” Speaking of the connections between journalistic interviewing and user research, I want to recommend the podcast from 2017 “The Turnaround” – it’s interviews with different kinds of journalists about doing interviews, about getting to the story. I learned as much from the contrasts between our different objectives as I did from their best practices that apply directly. Check it out! Now, obviously, I’m back with more episodes of Dollars to Donuts, I’m taking more of an open-ended approach to new episodes, and they’ll appear as they’re ready, without any specific frequency. It takes a lot to do this podcast and rather than taking advertising, or doing crowdfunding, I want to ask you to support the podcast – and me. You can hire me! I plan and lead user research projects, I coach teams who are working to learn from their customers, and I run training workshops to teach people how to be better at research and analysis. You can buy my books for yourself and your friends and your colleagues – I’ve got two books – the classic Interviewing Users and more recently, a book of stories from other researchers about the kinds of things that happen when they go out into the field – it’s called Doorbells, Danger, and Dead Batteries. You can also give this podcast a star rating and a review on iTunes, and you can review either book on Amazon. With your support, I can keep doing this podcast for you. Let’s get to my interview with Leanne Waldal. She is the Senior Director of Product Research at New Relic. She has led research at Autodesk and Dropbox as well as running her own research agency for almost 17 years! Thanks for being on the podcast. Leanne Waldal: Absolutely Steve: Let’s start with a basic introduction. Do you want to tell us who you are? What kind of work you do? What your company is? Leanne: I’m Leanne Waldal. I am Senior Director of Product Research at New Relic. New Relic does application performance monitoring. It usually makes people’s eyes glaze over. Basically, we have a tool that developers and engineers, primarily that group, uses to monitor the traffic on websites and mobile apps. So, if you think about streaming media, online retail, financial services – tons and tons of data and traffic – their site reliability team, or engineering team, might use us to monitor that to make sure nothing goes down. I’m based in San Francisco and I’m a part of the design team which is a part of the product team. The product team is mostly in Portland, Oregon. We have a smaller portion of it in San Francisco and we also have a team in Phoenix and in Barcelona. Steve: Can we talk a little more about New Relic and who the customers are? You said to make sure that it’s not down, but I’m assuming it’s not just that binary state, like is Amazon up or is Amazon down? Leanne: It’s to monitor it. So, be watching to see if something happens that’s slower. Usually it doesn’t matter if something is faster. It means you’ve optimized performance and things are running really well. So, if things are going slower or you notice that people are having longer wait times for page loads or checkout loads or media streaming, or whatever sort of thing they’re trying to do in your app. You would have spent a considerable amount of time setting our products up, basically, to then do all that monitoring and raise alerts when something goes wrong. Steve: Okay. So, it’s not up and down. It’s slow and it could be any part. Leanne: Exactly. Lag time. Checkout’s not happening in the amount of time that you expect it to. Steve: Did you use the phrase reliability engineers? Leanne: Yes. So, the new role that basically Google created about 10ish years ago is the site reliability engineer. They wrote a book about site reliability engineering. So, our target user is usually called a dev ops engineer or a site reliability engineer. If you are a more modern company with the way that you’re keeping track of your website and your apps, then you probably actually have a development operations team, or a site reliability team. If you’re not quite there yet, then these people might be a part of an engineering team, or IT operations or something. Steve: That’s good context. Then you said, you’re part of the design team which sits in the product team? Leanne: Yeah. So, at New Relic, engineering and product and design all report up to the chief product officer. Different companies do it different ways. Engineering might be alongside product or design. Design might be alongside. But at our company, if you’re in the product org, you belong to the design team, the product team or the engineering team, or a few different teams that are in the org. Steve: How long have you been in the role that you’re in? Leanne: I’ve been at New Relic for six months. I did a similar job at Autodesk, before New Relic, for about a year and a half. And before that I had a similar role at Dropbox for a couple of years. Steve: How do you compare and contrast those three organizations and what you saw and what the trajectory was in those roles? Leanne: We could start with Dropbox. Dropbox was private, pre-IPO. It was very much a unicorn. When I started there it had 400-600 people. I don’t remember exactly how many. There was no research. They had a small design team. They had, as a lot of tech companies at that time, being a unicorn, had tons of engineers. And also, because Dropbox is a consumer app as well as a business app, everybody used it, so they felt like they knew who the user was. Steve: Everybody that worked there? Leanne: Everybody that worked there, yeah. So, I started setting up a research team there and grew it as the design team was growing. That’s different from Autodesk where I went next. A very old company, very fascinating products with fascinating customers and use cases. And people there who were researchers there, who had been researchers there for a long time. And what I was doing there was combining research and analytics together. At Dropbox, analytics was separate. And I had a small team that was more globally disbursed. At Dropbox, my team was all in San Francisco, to start. Here at New Relic, when I joined, there was one researcher here in San Francisco and one in Portland and a similar sort of tone around a lot of the people who work here at the company. Our site reliability engineers (or have been), so they know a lot about the market and a lot about the users. So, we’re basically here to help the company understand all of the new users it’s acquired since it started because it’s a different company now than it was 10 years ago. Ten years ago, New Relic was a company that was mostly reaching out to developers and engineers to use its product and now we’re mostly focused on enterprise and we have lots of enterprise customers now. Steve: There’s that classic – you referenced this, right. People think they know they’re customer because they are users. Leanne: Yeah, we all do. Steve: But you’re kind of saying yes, to a certain extent. I don’t hear you shutting that down? Leanne: It’s like yes and the marketplace out there has changed and maybe there’s something new that we’re not looking at right now. And also I’ve noticed that for companies where they maybe used to do contextual inquiry and go out and visit their customers and users and understand them deeply, in the last 10-15 years companies have tended to move to surveys and remote interviews and stepped away from that sort of like deep understanding you get from being in a room with someone, or on a construction site, or on the road with them when they’re using your mobile app, or whatever it is. So, you can see everything around them while they’re using your product or your app. And when you move away from that you lose sight of that holistic story of the customer experience. And then research can come in and do that for you. Steve: Why do you think there is that movement towards the remote and movement away from the contextual work? Leanne: Contextual work takes time and effort and can be exhausting. Exhausting to do it and exhausting to come back and know what to do with all that data, how to tell a story out of it and how to decide how it can have impact or value? Or sort of the like, now what do we do? I think also as humans we get really familiar with where we are, and we neglect to notice that the world around us has changed. Steve: So, that’s back to your point then. We started this company based on work that we’d done and skills that we had and users that we knew, that we were, and the world moves on and shifts and new customers… Leanne: And it’s not just New Relic or Autodesk or Dropbox. That happens at any company. You can be in banking or in legal. Ask anybody who has been there a long time, compared with a person who just started. They’re having much different experiences of it. Steve: You described, especially with Dropbox and New Relic, this point at which the company had little or no research, and that is not unique. We know lots of companies like that. Leanne: Oh yes. Steve: Do you have a hypothesis about what’s going on? There’s a point at which someone like you starts talking to these companies and starts saying, “hey, here’s what I would do if you had me come in and work with you.” What’s going on beforehand? What’s the point of need or pain that’s identified inside these companies when they realize… Leanne: Sure. So, what I’ve noticed at the companies I’ve worked at, as well as companies I was interviewing at for this sort of position, and companies I know the story around when they hired someone in sort of like a director or higher role in research, usually something changes in revenue and that change in revenue sets forward sort of like how do we figure out what’s going on. And if they have someone who is currently working there in a role who has had experience in the past with doing market research, or doing churn research, then those people will start to raise the flag of like oh, we have to do research. And then it’s how do we do it. So, that’s one reason that someone brings someone in to do it. Another reason is that there’s sort of like a groundswell of engineers, product managers and designers who are telling the people who make hiring decisions, “we need someone to be in charge of design.” Or, “we need someone to be in charge of research. We’re doing a lot of this ourselves and we don’t feel like we’re doing it well enough.” Or, “we, for example, do lots and lots of interviews. We don’t have time to synthesize the data. So, could you please hire a researcher to work with me.” So, sometimes that happens. Another way it happens, which is how it happened here at New Relic, is that the company decides to hire like a VP or SVP of design and then that person who knows, oh this is what a design team looks like – you have someone who is focusing on content design and language and someone who is focusing on interaction design and visual design. And I also need someone who is focused on research. Sometimes it’s from the top down. Sometimes it’s from the ground up. Sometimes it’s there used to be this role at the company before and someone left or was laid off and then they just didn’t backfill it and then after a while realized oh that’s really a role that’s really important. Steve: We need that. Leanne: Yeah, and it’s not – I used to think that maybe it was sort of like a trend, like everybody is starting to be customer centric, or user centric, or whatever that means. So, then they were like, “oh, well Joe down the block is doing that, so we need to do this too.” And what do we do, we hire someone to run research. But, as I have been interviewing for jobs over the year and talking to different companies and doing these jobs, I’ve realized, no, there’s actually something that happens inside the company that causes that change. It’s not so much like copying a competitor or another company that you care about. You might do that with engineering or something, but research is sort of like the first thing to go if you start running out of money. And, I’ve seen hopeful examples in startups recently where even though there are only 3-5 people they bring in someone for research and that’s highly unusual. Usually, if you’re starting up a company you have seed money, or you have your first round, the first people you’re hiring are engineers. You’re not making one of your first 10 hires a researcher. So, I know some example now around the San Francisco Bay Area where they’ve actually brought someone in whose role is to do research really early in the company. And that either points to a certain amount of humility around, oh, just because I am the user it doesn’t mean I know the users. Or, because they’re going into a space they don’t know. They just had a really good idea about it and they really need to understand the space. Steve: That’s an encouraging sign. Leanne: Yeah. Steve: Is there a distinction between – you gave a number of different scenarios where research comes in – do you see a distinction in sort of the context, or the action that’s being taken when bringing in a person to do research vs. bringing in a person to lead/manage/run/build research? Leanne: Yeah. So, if you’re brought in as a person to do research you’re usually reacting to the things you’re being asked to do. You usually don’t get to pull your head up and do strategic work. You’re doing lots of like compare this design with this, do this survey to answer this question, help me with like interviewing these six users in remote interviews. If you’re being brought in as someone to say like where should research fit into the company, it’s more challenging, but more interesting and exciting because then you’re being told like tell us what we should know? Like where should we go with this? Are you going to focus more on the market research side, or the sales research side? Are you going to focus more on the product design side? What are all the things that we need? What kind of programs do we need? How do we get access to users? How do we interact with users and engage users? There’s just all these little pieces of how a research process and practice runs inside of a company that is great fun to put together because you have to do it based on who you’re working with and the culture of that organization. It’s not just a cookie cutter that you put into a company. Steve: Is there a trajectory from one to the other? Like someone that’s hired because the need is to do research. Is there a trajectory for that person to start answering the kinds of questions that you’re talking about? Leanne: Yeah, yeah. I think if that’s sort of like the way you work, or your personality, or the way you’re motivated, you can get there. For example, I have someone on my team right now who I mentioned to the other day like – she’s a researcher and I said, “you could make some day, if you wanted to, a really amazing research operations manager because you’re really good at all these pieces around managing the research and communicating it.” Because half of research is PR and sales. You have to find the people to engage with it, to sell it to at the end. You know you want your research to have impact on product or a marketing campaign or whatever you’re doing. So, you have to gather the people around you. And you have to keep track of, particularly in a B2B company, who are all the people who have to be involved if I want to do visit customers. So, in a consumer company you just go out and find people. You talk to them and you do whatever you want with them. At a B2B company you have to be engaging with customer success, with sales roles – whatever they’re called – account execs, account managers – whatever that company calls them. You have to often also be engaging with product managers, designers, design managers. Engineers, didn’t used to be, just as a stereotype, interested in research and now at some companies they are very interested. And so managing all those types of people that you either have to get approval from, or you need to make sure that they’re bought into what you’re doing, or you need them to come along with you and do it, takes a lot of work. So, there’s this whole new role of research operations that usually helps keep track of that. And I’ve noticed some researchers have a real knack for sort of like managing all those details around their project and then shining a light on it and sort of going out and selling it, sort of like a marketing campaign. They just know who to schmooze with and talk with. And some researchers are really good at doing the research and writing the report and sharing it. I pointed out to her, “this is something you are doing for our team without being asked. Like, that’s a real superpower.” Steve: Does that start to change the – I think you said this, at least indirectly – does someone who is shining around operations, like your colleague here, does that start to change how research is perceived or experienced by others in the organization? Leanne: Yeah. Especially if you’re an organization that’s not used to research. Or they’re used to doing it themselves. So, they’re not used to partnering with someone else and they’re not used to having someone say like, “oh that’s not really a survey. That’s more of an interview.” Or, “oh, that’s not really an interview. We need to actually go watch people to do this.” Or, “this can’t be answered in an interview survey, or watching people. We need them to be in a diary study.” They aren’t used to having someone bring in all these different methods. And they aren’t used to having someone sort of like find some insights and then tell them. So, that’s why it’s so important to have a collaboration. I want you with me while I’m doing this research so that when I tell you the pens need to be blue, not red, you were there to hear everyone say they like blue better than red, instead of throwing the report over the wall. Steve: I hadn’t really thought before of operations as a – I’m not sure you’re saying this – but, almost a Trojan Horse. That there are these sort of tactical objective problems to be solved under operations. How are we going to find these people? What is it going to take logistically? What is it going to take legally and so on? That starts to change some of the conversations maybe that can happen internally. Like you said, “this is your problem. We recommend you go about it this way.” Leanne: Yeah. Steve: I hadn’t thought of that under an operations lens myself. Leanne: It’s really more in a B2B company than a consumer company because you have to manage so many relationships and oftentimes a researcher wants to focus on sort of like the details of the research. Like ask any researcher, they don’t want to do all the details around recruiting and scheduling people – or, most don’t. Steve: Right. Leanne: And so, you want someone who can do that. And then sometimes researchers also aren’t the people who know how to best sell their work or edit it down into like the three bullet points for an executive presentation. And it’s helpful to have somebody who knows how to do that. You know just sort of like editing and presenting and PR and marketing. And also knowing who to market it to? Like, oh, I know that there’s this sales VP in this office in Toronto who is going to be interested in this. That mentality of like keeping all these dots rolling together. And if you have someone who knows how to do that and also understands how to do research and what goes into research, that’s what I see makes a really good manager from an operations perspective. There’s the people management side too. Being able to mentor and coach and take care of people. But there’s more and more in B2B research a need for someone specifically in this operations role to just sort of like help everybody and raise the brand of the team and the company, or in the product team or the design team. Steve: It’s so cool that we’re at a point in this field and this practice where we acknowledge, as we always did, that there’s a number of very different kinds of skills and expertises that are required. And you just outlined a whole bunch of them. But that we’re able to sort of say, and not be laughed out of the room, like oh, that might be different people. You talked about collaboration and collaboration with people with different strengths or superpowers. Leanne: Yeah. And another expertise and skill that’s being introduced more into companies is the academic anthropologist who is now working inside companies on a research team. Like EPIC used to be primarily all academics and now there’s a bunch of people from business involved in EPIC, and also people who used to only be academic anthropologists and ethnographers working on research teams, inside tech companies or consulting agencies, or whatever. So, I recently hired a few anthropologists on my team. One who is a former professor, one who had applied anthropological research practices for market research. Another who had done research with an NGO, but also worked inside of a company recently. And I think that when you have that academic background and you bring it into a company, it’s like a special sauce. You also bring in a special sauce if you’ve been working in tech for 15 years and you know the ins and outs of how a product team works, but I like mixing those two things together because people learn off of each other. Steve: You like to mix the special sauce. Leanne: Yeah. Steve: So, it’s an extra special sauce. Leanne: Yeah. So, for example, we decided as a team that we wanted to look at all of the internal blog posts and other things that had been written about customers and users, from the point of view of people inside the company, because my team is new to this company. We were like okay, let’s take a look at everything that everybody else wrote. And I was trying to figure out how we would do that. Like, who knows how many internal blog posts there are and how many things that have been written about customer visits and things that different people know about users that they report out internally. And then I realized, I have some former academics on my team, academics who are now working as researchers in industry. They have skills in literature review. So, brilliant. I took them and had them put together like how are we going to review this and code it – special sauce, like they absolutely know how to do that. It was really easy for them to put it together. And then took other people who had more industry experience, or application focused experience, and said, “okay, now you can do what was set up. You can do the review of following the system.” And I have people who know how to set up systems set up the system. Steve: So, that’s an example of the special sauce. They know how to deal with this. Leanne: Yeah. And if you have a team who has mixed experiences like that, then they just sort of like, they lift each other up and they teach each other things. And that’s something that you can’t learn if you’re the only researcher, or you’re just one of a few researchers who all have exactly the same background. So, you all come out of the – whatever the programs are now, at Carnegie Mellon, Stanford and Berkeley and University of Washington and on and on. And you go and get a job and you do remote interviewing and some focus groups and some other things But everybody around you is doing exactly the same thing. Then you don’t get a chance to sort of like up level yourself. And you also, like if you’re the person managing that team, you don’t get a chance to learn new things from people on your team that you hired, that like you don’t have the same background as them. So, it just improves humanity basically. Steve: So how do you think about like what the – if the team is kind of eclectic now and you have all these different mixes of skills and backgrounds and aptitudes, how do you think about what the collective should be? How do you think about the mix? Leanne: Of a whole team? Steve: Yeah Leanne: So, I do miss the team I used to have which had both research and analytics. I think being able to have people on your team who do data science, or who are more like business analysts who know how to count things up and make charts, or someone who knows how to mix data together and come up with reports out of it, you get a much faster feedback between the analytics people and the research people. Analytics is like oh I did this, I want to know why? Research is like I can go out and find out why. Or research says, “I need some access to people who have certain characteristics, can you find me a list of them through analytics? Thank you so much, I’ll go out and talk to them and then I’ll come back to you.” In most companies those roles are separate. So, it is here. Analytics is a separate team. We work super well together, but sometimes when you have them both on the team it’s really nice because you’re both at the same team meeting and you’re both seeing what they’re doing. From a research perspective, having people who are all quant or all qual, or halfway in between or know both, is useful because anybody who’s in the job market right now, when they look for their next job the requirements are going up and up and up. We don’t want you to just be somebody who does interviewing. We need you to know how to design a survey. We need you do know how to do SQL queries. We need you to know how to do – like there’s more and more requirements showing up in job descriptions now. And so, if you can be a part of a team where somebody else knows that thing you don’t know, it just – we’re all going to like leave and get another job someday. Nobody stays somewhere forever. So, it will just help you when you’re going out to look for your next job. Steve: So, skill development, kind of through proximity to your colleagues. Leanne: Yeah. And it’s also diversity. Like any time, you have a diverse team, whatever you’re defining diversity as, just makes for a better team environment because all these different perspectives around the table from all the different places they came from, just makes for like different things to feed into feedback about a project you’re working on, or something that’s going on in the company. You know, we didn’t all come from the same school or the same program, so we don’t have a difference of thought in the way we approach a problem. Steve: So, for you, thinking about the teams that you’ve put together – because you’re talking about different kinds of – I don’t know if diversity is the right word, the way you’re using it, but qual and quant are different orientations, or… Leanne: And also, academic and industry. Or in tech, because I’ve worked in tech for 20+ years, I look for people to come work at the tech company I’m working at who haven’t necessarily worked in tech before. So, look for adjacencies. Like you did research, but you did it for a car company. Or, you did research, but you did it for an agency. So, that they can bring different things into the team. That’s what I mean by diversity. You know research, but you used to be a professor. You know research, but you were working on an astrophysics PhD and decided that’s not what you wanted to do anymore, and you realize things that you learned as a part of your graduate program – so you totally know how to do surveys and how to talk to people and how to do research. Steve: I had this Twitter conversation with someone the other day and they asked me where is a writing course that I can learn to be better at writing the kinds of things I have to create as a researcher. They can’t find things like that. They can find persuasive writing which is for sales. I kind of scratched my head and said, “I don’t know that that exists.” I went into this little sort of pontification, as Twitter invites you to do, about learning from adjacencies like take a creative writing class. Take a journalism writing class. And in doing that, you would start to see, oh here’s how the lessons of doing this adjacent thing well could apply. That was the best advice that I had. I feel sort of affirmed, from what you were saying, that adjacencies for certain kinds of things bring a certain kind of value. Leanne: So, the Luma Institute, who put this framework together around design thinking methods so that it’s easy to teach and easy to learn, calls that alternative worlds. And I think that’s a useful way to think about it, that for your person who is trying to figure out how to write a research report for whatever audience they’re trying to serve, well that’s storytelling. It’s sales, it’s marketing. It’s slide development. It’s all these other things. And when you start thinking of it as how would I think of this from the point of view of a salesperson? Or how would I approach this from the point of view of a marketing person who has to put customer stories on the web and sell that to somebody? Or how would I do this as the point of view of the person I am presenting this research to? Like what do I want to hear? It gets you out of the sort of like oh I have to have 10 slides and one has to have my methodology and the end one has to have like recommendations. You can start thinking about it more as like I want to tell this story so that they land on that naturally. Or, they ask me questions about it when I’m done that will answer all those things, so I don’t have to have it all in my report. Steve: I feel like this is a transgressive notion of expertise though, especially in tech. Leanne: Yeah. It would be nice for all researchers if they could just come in and do their research and research was valued and anything that researchers said they were learning from customers/users was taken and put into the roadmap. I don’t know anywhere that happens. And because there isn’t anywhere that happens and the way the culture in these teams work, you have to have this sort of like marketing and collaboration and partnership mentality around your work. If you’re a consultant, you get a project, you do it, you collaborate, you sell it a little bit – you’re done. You don’t have to still hang out and make sure it’s being used and follow-up. Steve: It’s just all sunshine and roses being a consultant, right! Leanne: Exactly. That’s why I don’t do it anymore. Steve: I had a coach advising me, in my consulting business, and what they said was – they said many things, but one of them was anytime you start talking with a prospective client, show them – show me how you’ve helped someone like me. That adjacency kind of framing – the advice was that wasn’t as persuasive. So, that could be what have you done in my industry, in my vertical? Leanne: Yeah. Steve: You know, and I think researchers, or anyone that just likes to sort of – that gets deep into different kinds of problems, you know we see connections between things that are – you know we know why this problem is like this other problem, even though it’s different. Leanne: And you want the other people to see that and sometimes it’s hard to show them a lens that will help them see that the way you saw it. Yeah. Steve: I think the advice was that’s not where you start the relationship. You start the relationship a little more close in. “Oh, this thing that you’re doing. Yes, I have done that.” Even though every problem is new in ways that – it’s very hard to do that. Leanne: Yes. Yes. Steve: Sort of starting the – showing relevancy right at the outset of the conversation. Leanne: When I ran a consulting agency, that’s how I started it. Somebody asked me if I knew how to do something and I’d never done it before and I said I knew how to do it. And all of a sudden, I had a consulting. Steve: So, your career is just built on lies. Leanne: Yes. Steve: Is what we’re saying here, okay. Leanne: Yes. Steve: And we’re leaving this in. Yeah, well I think that’s true too. There’s the confidence to explore. I mean can you do this? Yes. Have you done it? Well, no, but I can do it. Leanne: Yeah. And sometimes the question never comes up, have you done this? It’s just the can you? Steve: So, that’s – so, we’re talking sort of requests in the consultant side of things. Inside organizations, in any of the ones you’ve worked in, like how are projects formed? Who decides? Leanne: So many different ways. So, I would say that if you’re beginning research in a company, you’re taking requests and you’re probably using your boss or some other key stakeholders you recognize to decide which to work on. And you’re probably prioritizing those in some sort of backlog, if that’s in a spreadsheet or it’s a Jira or whatever. So, you’re showing people sort of like here’s what’s in my backlog. Here’s what’s coming up next. If you’re a team, the way I prefer to run a research team, which is the way I’m gradually getting it set up here – how it was set up at my last couple of jobs – is to have researchers who are paired or embedded, or whatever you want to call them, but they’re primarily working with a product area. A product team, a feature team, a product area, however your organization is set up. And they are servicing that team and partnering and collaborating with that team. And they own the research for that. But because they’re a part of the research team they also have a holistic view across the organization, because they’re also meeting with other researchers who are doing the same thing and seeing where their work, or where their product area or features sort of overlap, or interlock with each other. And then having some people who are more senior, principal researchers, just working on special strategic projects. So, that could be, depending on a company, that you have someone who is more focused on market research or competitive research. It might be someone who is focused more on research to understand where we should go next? So, sort of like future casting and sort of like looking at what’s going on, finding edge cases, seeing spots for innovations and other sorts of insights. And then also the operations side. People to manage panels of users and recruit people. And then there’s always, depending on how many people there are, some sort of management tier for people management and coaching and that sort of thing. Steve: And when you talk about embedded you – it’s certainly a frame that you hear described with some regularity, but your words are a little different. Some people talk about embedded, that that researcher is on that team, sort of fully on that team. You’re almost describing like a workstream or a customer or something. Leanne: Yeah, because the way people work on teams now is not the way that HR software recognizes. So, HR software, for example, recognizes I have a boss. He’s a VP of design. Recognizes I have people who report to me, all the people on my team. Um, maybe recognizes that I have people that might be good for 360 reviews. So, my direct colleagues who also report to my boss. But, that’s such a limited view of what I call my team because – I’ll just talk for me. Like I work with lots of different people around the company who, if I was going to be reviewed, I would want them to weigh in. People on my team – actually, we had a research team meeting this morning. I had them all bring a stakeholder map because I wanted to see – now that I have people that are sort of like embedded or paired within teams, like who are they working with and what does it look like, compared with people on my team whose work is sort of nebulous around all the different people that they’re managing and all the different work that they’re doing. And it turned out that way. So, if you’re a researcher on a team – a lot of people refer to the three-legged stool, but if you have a researcher it’s a four-legged stool. So, you have someone from research, from engineering, from product and from design, all working together with like a common focus on how are we going to get this out the door and make sure it meets customer needs, solving a problem, all that sort of stuff. If you are somebody in operations or a principal researcher working on a strategic project, you just have a mess of people around you. And so, all these people sort of like vacillate in between teams. Designers do the same thing. Designers are a part of the scrum team and they’re a part of the design team. Engineers are a part of a team for some sort of product or feature. They’re also a part of an engineering team. And if gets even more complicated when your manager is in one city and you’re in another city. Because if you’re enough time zones away you probably have somebody who is sort of like your dotted line manager in that city to help you with like all the things around your job. For example, I have a researcher in Barcelona. We can’t look over each other’s shoulders. There’s only two hours a day when we’re both at work. So, there’s a director of design in that office who is her sort of – who is her manager there. So, if she has questions about how does this work? Or how do I do this? Or who is in charge of this? Or help me review this? He can do that. But no HR software recognizes that. It doesn’t recognize she basically has two bosses. It doesn’t recognize that I have one boss, but actually these three other people are kind of my bosses too. It doesn’t recognize that that designer has a boss for whoever is running that scrum team, but that designer also reports to a design manager or director. So, most people work that way now. There’s very few people inside companies anymore who will say they belong to one team and that’s all they belong to. Everybody belongs to like multiple teams and we have this archaic structure around this person decides like your performance review and hopefully gets input from all these people. But, just lines people up in like lines and doesn’t realize that it’s a little more messy than that. Steve: So, does that gap between the reality and the structure of the software, what’s – are there consequences to that for how we work? Leanne: I don’t think there’s consequences. I think it’s just the way it is. I think there’s more 360 reviews now, just in my experience, then there were 20 years ago, because you work with so many other people now. I can remember jobs from early to mid-90’s. Nobody was doing 360 reviews of any of the jobs I had then. Like I worked with a set of people in a geographical space. I didn’t work with anybody else. I think it’s just more like it is what it is and things have come in to help manage and understand that, but we, the people who sort of like do to the sort of organizational management, haven’t figured out how to accurately represent that in a structure of the software that manages the company. Some companies try to go – what was it, like Zappos, that decided nobody had a manager. That seems like part of a reaction to that – Holacracy, that’s what it was called. Steve: So, we got into this because we were talking about what does it mean to be – there sometimes seems a tension between researchers are embedded, or researchers are centralized. And I think what you’re saying is they are neither, or both. And that that’s how work happens. Leanne: Yeah, and you have to look at the culture of your org. How does your org work? Who wants to participate and who wants – it’s that old RACI model. Like who wants to be involved? Who wants to be informed? Who wants to partner with you? Steve: So, wait. Leanne: Do you know the RACI model? Steve: I know it, but I need to clarify it. Leanne: It’s where you define – I might not get this right. It’s like where you define who is responsible/accountable, who is interested in consulting on it and who just wants to be informed. So, if you’re thinking about a project, you always have people who sort of fall into those categories. So, I just think it depends – some cultures are very top down and as you go up the ladder they only want to be informed and not as involved. Some cultures are more participatory and as you go up the career ladder, everybody wants to participate and know and be involved. So, it just depends where you land and get a sense of who just needs an inform, who actually wants to come along and sit with you while you do a remote interview, and travel 3 blocks away to visit the customer. You know how far up the ladder does that go? Steve: So, coming into these organizations that have done very little with research to date – like understanding the culture, it seems like that’s got to be an initial step to get the lay of the land? Leanne: Yeah. Who’s making decisions and what data are they using to make those decisions? Do they want any more data? Usually the answer is yes. Anybody wants to know more to make better decisions and, in a corporation, make more money. So, figuring out who those people are, where the gaps are, what they need to know. And then another thing you can do when you start at a company is be the researcher yourself of that org because what you’ll start to notice, just like any research project, is that you’ll start hearing the same thing from multiple people. So, you’ll start hearing like, “the one thing we’ve never figured out is whether people really like blue or red better.” You know there’s always something that’s like the one thing where people aren’t aware that everybody has the same question, but because you got them to open up and talk and had casual conversation and used all your interview tips and tricks to interview multiple, different people who don’t work with each other, you’ll start to see a theme around something that everybody wants to know, but nobody knows, but nobody is like trying to figure it out because they don’t know everybody else wants to know. I’ve seen that at every company. We’re working on a project like that here right now where there’s basically two things that I heard across the entire company when I first came here. So, I was like okay, that’s where I’m going to start some key research projects and then I’m going to bring in some people who are interested in it. And then as I hire people in, I’m just going to start attaching researchers to different product areas and work with the people who run design to sort of think about which designers need the most help right now. Or, what meets the business directives, or the key priorities, or the goals, or whatever – the product org of the company. Steve: Are there stages of evolution? I don’t know if it’s milestones or some framework that at these points you’re kind of coming in where there’s very little and you start building in a way that is specific to what you’re learning about the culture, are there stages that you could identify that you’re passing through or moving towards? Leanne: Oh, yes. So, once you get sort of like programs and tools set up you don’t really have to do as much anymore. So, programs and tools like is there decent survey software? Do we have access to users? How do we access them? Is there a panel? When we do research, do we have standard agreements that we sign with the humans that we’re doing research with? Repositories – where are we storing everything? How are we storing it? How are we accessing it? Do we need transcripts? Do we have a tool for that? Do we need to do card sorting? Do we have a tool for that? Just sort of like sorting out all the tools and processes for the practice. What was the question? I got lost on a rabbit hole of process and tools. Steve: You were answering the question. So, what are the stages or milestones? Leanne: Oh yeah, okay. So, looking at what’s needed that is more like capital improvements. So, like we need a new roof. Okay, we’ll get a roof, it will last for 20 years. Done. So, things like how you pay incentives and research agreements, that’s sort of like the roof. Then you need like everything that pays the electric bill and the gas bill all the time. So, those are projects. So, what projects need attention right now. And you learn that from interviewing sort of like all the key stakeholders you’re going to work with, which in my example earlier was once you interview a whole bunch of them you start to find out that there are some key things that everybody wants to know. And then figuring out, either with your boss, if she or he knows, or with key stakeholders, like if I was to start bringing in researchers one at a time, where would we start? Who would benefit the most? What would be most impactful? Or said another way, sort of like what are the things launching soon, or later? What are the business goals, or what’s in the strategy? Or what are the business directives? Or all those sort of things that make a company run. And that helps you because then you’re making sure you’re focusing on something that is actually going to eventually affect the company’s bottom line. So, when the research comes out and it helps that, then you’ve been like okay, this is how we do it at this company. We’re not academic. We don’t go sit back and sort of think about things and just, you know, go research things out of curiosity. We look at where the company is going to make sure we’re helping it and helping it do better with the research that we’re doing. Steve: And then what’s a five year – that’s a presumptive question. They’re all presumptive questions. What’s sort of the horizon for which you would have vision? The time horizon? Leanne: For the times that I’ve come into a company and there’s been no research, or one or two, like here, individual researchers, what I’ve told people is it’s going to take 12-18 months, with budget and with head count, to get this into a modern research practice. And here’s how we’ll do it incrementally and here’s how we’ll check in and benchmark and measure it along the way. When I was at Autodesk and I inherited a team and then I sort of hired in and added on to it, it was much different. These were really old products that had a really amazing install base. And some of them were market leaders. It wasn’t like companies that were two years old, five years old or ten years old, or fifteen years old. Like, once you’re at 30 years you’re settled in a certain way. So, I did different things. I tried to better understand the products and better understand the company to see where we would could add value. And then also, because that was a much larger company, see where research was happening across the company to see if we could sort of like combine efforts. For example, I went to some company that’s like 25 years old, that has say 1,000 researchers. Because there are some companies out there now that have that many. And I was plopped in as a director of research with 20 to 50 direct reports. It’s a much, much different role than coming into the company and setting something up. They’re you’re coming in and you’re saying how is this done? Is there any way it could be improved? Is there any way I could sort of like do – no, there isn’t? Okay. So, how do we prioritize things? How do we have impact? How do we collaborate and share out and be good corporate citizens and be a part of the team? I’ve realized, with this third job that I’m on now, after doing consulting for many years, that I do really like growing things. There’s a challenge to it that like gets me into work every day. Like, oh, we have to figure this out. Oh, there’s lots of problems to solve. Oh, how are we going to prioritize this? Or how are we going to solve for this? You know, how am I going to find someone to fill this role? Steve: You just listed a bunch of things as positives that someone – if you’re in a different emotional state or a different sort of mindset – could frame those all as negatives. Leanne: Yeah. Yeah. You have to understand the challenge you like to work on. Is the challenge I want to be the person who works with the team to help them figure out what they need to solve? Or do I want to solve an organizational challenge while also working on business priorities and solving problems? Steve: So, this is something that kind of sparks you? Leanne: Yeah. And I didn’t realize that until – it was like 3 months ago or so. We hosted a breakfast here and somebody was asking me what I was doing, and they were like, “Well it sounds like you’re really good at that because you’ve done that three times now, and you started your own company.” And I was like, oh, that’s the first time I realized like this is my thing that I like to do. Steve: Yeah. That’s great. Leanne: Sometimes you have to be almost 50 to have finally found yourself. Steve: Right. It’s the journey, not a destination, right. Leanne: Yes. Steve: I’m sure you’re going to keep finding yourself. Leanne: Yes. Steve: Anything else that I should have asked you about that you want to share with us? The collective us. I know it’s just you and I in a room. Leanne: Well, there is one other thing I’ve noticed in the past few years. I have used a community platform, that’s sold by a company and meant for marketing and brand teams, and have used it for product and design research. There’s something about – so, support always has its help forums. Marketing and brand usually do focus groups, but now sometimes they run online communities where you can get badges and win things for participating in their research, or telling stories, or whatever. There’s a space in between that people who are on research teams at companies sometimes use and sometimes don’t. So, a typical team might have a panel, particularly if you’re B2B and it’s hard to get access to people. Or even if you’re a consumer-focused company and you just want to make sure you have like your 10,000 people who you can access at any time to easily invite to research. But I think there’s something about the way that people on social media interact with each other in a group in one-to-many fashion and many-to-many fashion that is happening in some ways in research teams, and sometimes not. So, if you’re a researcher and you can, at any time, poke into a community that belongs to you – this is my panel, but your panel, they can’t all see each other. In a community they can see each other. And so, it acts a bit like a support forum, a bit like this marketing and branding, sort of like what you think of our new colors sort of thing. Or this is our new video for our marketing campaign. But instead it’s for product and design. I’ve noticed there’s a certain fear of it – oh my God, if we get our users and customers to talk to each other, what will they say about us? But I think if you get over that fear and you open it up and put together a private space where they’re all talking together and you and your product design, engineering, marketing sales, whatever partners are also in there, you create this sort of like brand loyalty and product loyalty that you don’t necessarily get just from marketing brand and sales and just from support. So, I’ve done that in my last two companies, created this sort of community where people could see each other, and we could interact with them in little like pop-up sort of projects. Haven’t done it here yet. But when I talk to the vendor that I’ve used for doing this, the last time they demoed it for me, I saw that the examples they were giving me were for a product and design team. So, I was like huh, you’re now selling to product and design teams. So, there’s something going on with people picking that up and doing that. Steve: How do you set people’s expectations, the people that are participants in this community, for what – as you said, it has elements of other kinds of communities that are maybe more familiar… Leanne: Well, you have to make sure, particularly in business settings, that – you know don’t share anything that you wouldn’t want your competitors to see, because you don’t know if your competitor is in this community, or not. We’re focusing on sort of like your work and your use cases of these sorts of technology. Don’t disclose anything that you shouldn’t be disclosing. But, if you’re in a space where people are using a technology where they get a lot out of finding out how other people are using it too, then you create relationships with each other and then you create your own relationship with them. Another downside is you can get bullies, so you need to have guidelines. We can kick you out if you behave in one of these ways. And with panels, because it’s the company to the user or customer, and that’s it, it’s harder to find sort of like an intrinsic incentive for someone to participate in research. In a community you usually don’t have to have any incentives that are hoodies or cash or gift cards or whatever because the intrinsic incentive is that they’re getting value from other people being in there. And then every once in a while being able to talk to a product manager or talk to a designer or talk to somebody else. So, I think that – I don’t know if that’s something that will keep going, but I see it as a way, particularly in B2B research, to provide value for your customers that actually helps out your sales people and gives your product people and design people a way to interact with users and then also give the user support that they’re probably also getting over in the support forum, but a different kind of support because you feel like you’re part of a group. Steve: It’s compelling to me to think about just blurring the different kinds of interactions that companies have with people. Leanne: Yeah Steve: Like you said, there’s marketing, there’s research, there’s support and we structure the companies around those kinds of functions and the tools and so on. Leanne: Yeah. And it can make it hard to see the end-to-end customer experience. Yeah. Steve: What – maybe we can rewind. Like how did you end up in research as a profession? Leanne: Back, when I graduated from college, in the early 90s, I went to work for a startup cellphone company as a statistician. My background is in statistical computing and economics from my university education. So, I was analyzing what were then considered very, very large sets of data – terabytes of data. I was running a neural net program on a Sun SPARCstation. I don’t even think it had a gigabyte of RAM. Yeah. It had two processors. It was amazing. It would take like 4 to 6 days to run something, to spit out a model. And if I got it wrong I had to do it all over again. What I was doing was I was working in a marketing organization and I was there to help them like figure out things like who should we sell call waiting to? Like should we just sort of try to sell it to everyone? Or should we target people? So, I was figuring out targeted lists based on behavior that we saw in cellphone use data, to help the marketing team understand who to sell things to. When they did that, what I realized was I didn’t know why the people were buying it? Like, just because I could predict like because you have these like usage analytics, it makes you more likely to buy, but why is that? And a couple of things happened then. The company I was working for got bought by AT&T, became what is now AT&T Wireless Services. When you get bought by a much larger company, a bunch of chaos ensues. Your job often changes. Your boss often changes. I was very young. I was in my early 20s. And at that exact time Mosaic came out and the web became graphical. So, there were all these things happening that occurred at this point in my life where I was like, huh, this is my first job out of college, but now I’m interested in knowing why these things happen? I just taught myself HTML. I figured out how to put together a webpage. Oh my goodness, there’s this thing called Match.com and I met this girl and she lives in San Francisco and I want to date her. Oh, I’ll just go take this job at this company in San Francisco, this web company that has like 20 people and they’re making websites and they’re from all different sorts of places. And I became a QA manager there. Once again, sort of being asked, “can you do this?” Not, have you done this. And I said, “yes.” And then from QA realizing that we didn’t really know what the users were doing. And I, at this point, had never heard of user research. But I was friends with people who were doing that at different companies. So, I asked them about it. And then I left that company and went to a startup. Was also a QA manager. Got laid off when the company didn’t get its next round of funding and went and did some consulting. Doing like web development for the summer. Set up a company that I had intended to be a company that did QA for web apps and somebody came to me and said, “can you do some researcher interviews with users while you’re doing browser testing?” And I said, “yes, we can.” And I was like here’s my chance. I’ve always wanted to know why people are doing things. And so, I happened into it that way. And in the mid to late 90s anybody could do that. You could teach yourself PHP. You could declare yourself to be a web marketer. You could say I’m an information architect. Like you just said what you were, and you could just do it. So, I quickly went and found some friends and said, “I have this consulting project. I don’t know what to do, help me out.” And they were like well you do AB…like the things I just mentioned. You have to do A and B and C and D. I was like okay. So, I did A, B, C and D. I did the project. I got paid. I thought it was amazing. I was like I want to do this again. I was like a kid who gets ice cream for the first time. Like, I will have more of that. And that’s how it got started. In consulting, what you need, from my experience, is you need one or two projects as reference projects and then that helps you get the next project and the next project and the next project. And then it was 17 years later. Steve: Wow. You’re saying that there was a time when you could sort of declare yourself to be this thing. Leanne: Yeah. When the web was new. When this new technology was coming out and consumers were grabbing onto it and companies were seeing that money could be made there and – people think now that there are more jobs than people, but then there really was more jobs than the labor market could supply because the labor market didn’t have the skills and the education system wasn’t educating people with those skills. For example, in looking for research interns for this coming summer, I interviewed people who were getting degrees in user research. Like a Bachelors of Science in user research. And I was like wow, like companies are now preparing people and there’s a whole education system out there in these D Schools and I Schools and everything now that didn’t exist in the 90s. Steve: Right. I sometimes feel like oh I’m – ‘cuz I came up through the same system, or lack of systems, that you did. Leanne: Yeah. If I wanted to do this job now – say, if I had gotten my Masters at the Berkeley I School, or whatever, and I went and got a job as like a senior researcher at a tech company, and I worked myself up, way up, it would be much, much different than my experience of teaching myself and figuring it out. Steve: What’s lost or gained in that evolution, if anything? Leanne: Well, I can clearly see that my sort of like – YouTube didn’t exist then, but it would have been like the university of YouTube, the way that I went about teaching myself how to do research and starting up a whole consulting agency around it – it would have been much different if I had been schooled in a particular way in how to do research. I think some things would have been lost because it wouldn’t have been sort of like me and a couple of other people just sort of figuring out how to do something. You’d come up with new stuff. But I think some things might have been gained. I might have more confidence – like I have a Masters in this, so I know how to do it. Or, I did my thesis on this and I am – yeah. There’s a certain amount of imposter syndrome that goes around, even if you have that education. But sometimes when you don’t, and you make it up yourself, you can doubt whether or not you really know what you’re doing. Fake it ‘til you make it. Steve: I remember just being in a consultancy kind of around that period of time and that every prospective client that came in we would have to write a proposal for what we were going to do. And it just felt like writing a Master’s thesis every time. Or some just impossible thing to give birth to. Like explaining what it was, deciding the steps, trying to articulate those steps. We just didn’t have any point of reference and so we were sort of figuring it out every time until I think at some point we started to settle in and like oh, here’s kind of what – what it looks like. And now like writing a proposal is not hard. There’s many other hard things. Leanne: But now you have templates and you can copy it. Steve: And you know – I mean I have a narrative that runs through my head that oh yeah it kind of goes like this. We have more tribal knowledge as well as… Leanne: Yes. More people to reach out to, to get examples from too, than used to exist. Steve: Right. In this era, this consultancy, not only were we figuring it out, we were doing so in pretty much isolation. Because you knew a few people, but most of us at that era – there wasn’t a community of researchers to kind of connect to. Leanne: Yahoo Groups did not exist yet. Google Groups was not there. Google wasn’t even here. Steve: Yeah. This is great. It’s a very interesting conversation. I learned a lot. Leanne: It was great chatting with you Steve. Steve: I appreciate you taking the time. Leanne: Absolutely. Steve: Anything else? Are we done? Any last thing? Leanne: It’s fantastic. Yes. Perfect ending. Steve: Mic drop. Alright, thank you. That’s it for this episode of Dollars to Donuts Go to portigal.com/podcast for the transcript as well as links for this episode. At the website you can also check out all the other episodes, and subscribe to the podcast on iTunes, Spotify, and all the usual places. My books are available at Amazon and rosenfeldmedia.com. Our theme music was written and performed by Bruce Todd.
See Podcast
