Latest / Fit Happens: Why CEOs and Leaders Succeed or Fail / How Market Research Tricks Leaders into Making The Wrong Decisions

Transcript

How Market Research Tricks Leaders into Making The Wrong Decisions · Fit Happens: Why CEOs and Leaders Succeed or Fail

• Jason Baumgarten 00:00 I'm Jason Baumgarten and you're listening to Fit Happens, the podcast where top leaders, investors and board directors share the stories, surprises and hard earned lessons behind finding the right fit. Let's get into it. Jason Baumgarten 00:17 Most people get very good at one narrow thing. Eventually they get promoted out of it. Jon Cohen ran polling at ABC News, the Washington Post and Pew and then became chief research officer at SurveyMonkey. He's now founder and CEO of an amazing AI company, TrueDot. Today we'll get into why he refused to be promoted out of the most critical job he's ever had and tell you the one thing that actually matters is true experience. Jason Baumgarten 00:47 I'm Jason Baumgarten and you're listening to Fit Happens. Jon, welcome to the show. Jon Cohen 00:47 Pleasure to be with you. Thanks for having me. Jason Baumgarten 00:57 Absolutely. I'd love to go back to the beginning. You studied history in college. And I couldn't help but note that history is the study of things that already happened and now you're in the business of betting on the things that might happen. What made you cross that line? That's extremely interesting. Jon Cohen 01:16 I hadn't thought about that in a very long time or in particularly that way. So I mean, I think, look, context is everything. And I think kind of in the work that we do today and the work that pollsters across the world do, which is trying to understand the current moment and also to look around the corner. Obviously we can't cast ourselves into the future, but we're trying to understand moving things. And you can't really understand them or certainly the meaning of them unless you understand the context, unless you understand the history. Jon Cohen 01:49 So it's not a connection I've made before, but thank you for giving me the opportunity to go down that thought path. I think that does really matter. You want to understand, of course, kind of the signal from the noise and Barbara Wohlstetter's famous book about Pearl Harbor and kind of the things that got lost there into tragic consequence and kind of thinking about how do you do that? And the only way to effectively do that is to kind of have an understanding of what actually might make a difference. What are the anomalies that we should be paying attention to and what is all the many deviations that we shouldn't. Jon Cohen 02:28 And that comes from judgment. That's taste, the new word that's so popular in AI circles, but that matters and that comes from historical context. And it's great to see all the books on the wall behind you because that's received wisdom, and how do we take that and apply it in our daily lives? So, you know, your answer really makes me think about this moment we're living in, where it's all about transient views and not knowing things deeply. And I think that connection from past to future is important. Jason Baumgarten 02:57 You then went from a policy institute to a newsroom, both consumers of the same core data, but very different environments. And what did that tell you early about kind of what you were good at and what environment you wanted to work in? You think about those early formative years. Jon Cohen 03:24 So it's a great question. I actually have to take you back to actually my transition out of being a history major. And I did have a sojourn in the consulting world before going to grad school in political science. And that was actually as a systems consultant for what was way back then called Anderson Consulting and is now Accenture. And they took me, the history major from Johns Hopkins, and pulled up the little coding school bus and sent me off and taught me how to program. And so I learned how to program. And I worked on the conversion of state tax systems from the old world to a slightly more modern mainframe world. Jon Cohen 04:00 And so I learned how to program. And so when I went to graduate school in political science and then had the jobs you mentioned, I was really good at the systems front. So not only did I have the academic historical interest in context, but I was able to go into the weeds and build better systems. And when you're dealing with history or whether you're dealing with state tax systems or kind of a mess of public opinion data, which I have done subsequently over the decades to build, how do you look at this information systematically and build a workflow where you're learning and have the opportunity to build that system of understanding so you can do something else? So whether it was in policy or whether it was in the newsroom, kind of, that is the core skill that made me better at my job and frankly, better at my job at a time where everywhere I've been was a time of increasing scarcity, particularly in the newsroom. Jon Cohen 05:01 How do you do the proverbial more with less, which is almost always impossible, but with the right system, some things are possible. And so when I arrived in the Washington Post newsroom and all of our historical trend was sitting in binders on the wall, or ABC before that, same situation, how do you build a system where you can extract from that kind of a working system to do work on deadline? And so I got really good at working on deadline because I built a system that could effectively ingest all of that information from my predecessors into being something that was useful in the moment. Jason Baumgarten 05:50 You were very ahead of your time. I hear that's the new trend, is ingesting workflow and inference data. Jon Cohen 05:50 Very much. What's old is new again. Jason Baumgarten 05:50 You mentioned going to the Washington Post, and I have to ask because when I was refreshing myself about your background, I was thinking at least one, likely two presidential elections during your time at the Washington Post. And that's gotta be an insanely stressful moment to poll a presidential election. Most of us have only watched it on TV. Maybe we've been in a campaign headquarters, but certainly never making that call. What does that feel like? And maybe take us into the room for a moment. Jon Cohen 06:19 Well, I mean, it's certainly nerve wracking. You kind of, all of it. And it's not just about covering the presidential contest, but in the Washington Post newsroom, I was also responsible for understanding the gubernatorial contests in Maryland and Virginia and then kind of the Senate races as well. So it was a lot. I was hired into the Washington Post in 2006, yesteryear for many of your listeners probably. And I was hired kind of seven weeks before the midterms, and those were the midterms where the Democrats kind of took back control of the House and Nancy Pelosi became the first female speaker. And also there were races in Maryland, Virginia. So I had to get up to speed really fast on a lot. And I worked 80 straight days. Jon Cohen 07:09 I had to build that system that I talked about and kind of do the work. And it was incredibly nerve wracking. And everything you put out in that context, it's called the silly season, kind of what happens now after Labor Day, all the way through the election. And everything you put out gets an immediate reaction. We would post something on the washingtonpost.com at 5:00pm and at 5:01 I would get a call from Rahm Emanuel and Rahm would kind of extract everything from me about what I had published, what I decided not to publish, or he would try to anyway, but it mattered. Jon Cohen 07:46 And that was an incredibly heady but nerve wracking time because it wasn't just like you put out a number and no one pays any attention. Everyone paid attention to it and therefore it had to be right. And kind of if there's a through line kind of over the decades for me, in addition to kind of that nervous moment of putting a number in publication, it's like doing everything you possibly can to make sure that number is correct. And understanding everything, if you're talking about trend, how it got from that binder on the wall into kind of the newspaper kind of all those years later, understanding the origins, the different systems that the numbers pass through, how we do the adjustments. So much about polling, not just pulling the data from the wall, but kind of on the new data that we collect. How are you weighting it? Jon Cohen 08:40 How does it show up in a crosstab? How are we retrieving it? All of that, that system has to be perfect. The number of times I popped up at 3 o'clock in the morning wondering if the number was right in the newspaper, many, many times. But like, all you can fall back on is the system. Jon Cohen 08:57 And that's what I pulled through and kind of, I know we'll talk later about my current work, but how do you do that really well, particularly in this world where you can't count on some systems the way we used to. They're amazing, but they're unreliable. And so building that trustworthy system is everything. Jason Baumgarten 09:14 What do you think's gotten easier about polling and about the extension into market research? And what do you think's gotten harder? Because you've seen it for so many years. Jon Cohen 09:32 That's a great question. I mean, the workflow certainly gotten easier because of these tools that I just disparaged by being somewhat unreliable. You can build trustworthy tools on top of these amazing and but unreliable AI models as we have and others have, I think as well. So like the system is better. You don't have to spend as much time crunching the numbers. You can start doing analysis even before you've finished fielding the survey. And so you can start earlier, you can go deeper and you can do so with more reliability. Jon Cohen 09:59 And it's just amazing. So that first job out of college where I was converting state tax data, that trained me for this current moment. I mean, you mentioned the through line. I think it's also just a bunch of serendipity as everywhere. But I know how hard it is to convert certain kinds of data to others. Jon Cohen 10:17 And that has made me remark about the current moment and just be so amazed by it. And it's incredible. And so I can, as an individual, my small team, as a small group of people, can operate as efficiently and as effectively as a major market research company. And that's incredible. And that's on the workflow front. Also, a lot of things have gotten harder. Jon Cohen 10:41 I mean, there's just much more noise out there. So to have a breakthrough is harder. It's harder to get people to answer surveys. There's a decades long decrease, well documented in the response rates that we can get as survey researchers. It has become harder to get people. Some of that is our own fault, which we can get back to as researchers, but some of it's not. Some of it's just technological and sociological. And so it's a lot harder and so kind of it's harder to have the confidence that we're getting representative samples in many cases. And we have to work harder to overcome the bias that can creep in in lots of different areas. Jason Baumgarten 11:23 You talked a little bit about some of the self harm in the industry. What do you think the biggest thing, if you could go back and change it, if you would, that you wish you could undo? Jon Cohen 11:24 I don't know what I would undo. I know how to fix things. I know one of the problems was Qualtrics, which has been a dominant software player in this space, did a terrific job relabeling research as experience management. It was very effective from a business standpoint. Jon Cohen 11:52 I think that for an industry that is now under this rubric of experience management, I think far too little attention is paid to the actual experience of respondents, and this goes to the self harm that I mentioned earlier, and also their experience of researchers. So we have addressed the experience of researchers, just how do you make our own workflow easier? We're also trying to do it on the respondent side. I think researchers have gotten too over committed to very long surveys with frankly repellent interfaces. Jon Cohen 12:27 The number of times that we pull up a form and have to go through it or answer formulaic questions. There's a reason that we ask the same question the same way over time. Trend is a wonderful thing to have to look at movement over time. Also, I think we get kind of too stuck in certain wording and the opportunity to kind of change it and mix it up is really important. You and I've had the opportunity to work with one another. How do you ask a better question? How do you have a different format of asking where you're actually respecting the experience and the expertise of your respondent, and so you don't give them a boring question in the boring old way that leads to kind of, they're either not spending time thinking about it or just getting annoyed at you and rushing through the survey. So I think there are a number of things that one can do, and that goes to thinking through the questions and then also having just a more fun and engaging way to kind of give your respondents an opportunity to answer them. Because people spend lots of time on their phones, and give them a good experience, they'll give you their time. Jon Cohen 13:42 And researchers haven't really done that well. Jason Baumgarten 13:42 John, that brings us very naturally to what you've built. And as you mentioned, I've had the benefit of working with TrueDot and with your team, and it's such a pleasure because not only do you bring incredible expertise to the conversation, but an amazing piece of software and a very innovative approach to what is a fairly old practice. I'd love you just to tell listeners a little bit about TrueDot, what you're all about, what you've built to combat a lot of these things that you saw in the industry for so many years. Jon Cohen 14:19 Thank you for the opportunity. Loved working with you in the last few years. I want to start with actually your expertise, because I think it's really important for someone like you, me, and for the company that we're building to understand what our role is. Our role is not everything. I've wanted for decades to put the tools of professional research kind of in the hands of experts. We talked about the political context earlier. Jon Cohen 14:43 How do you have a campaign strategist, have the right tools to test hypotheses that they might have, get actionable data that they can then use in campaigns? How can you, as the leader, know kind of that you are and lead other leaders at Spencer Stuart, kind of get the supporting data around them to do their work. And so it's important to not just say like, I don't believe everything needs to be self service and we're going to come and give you an answer. That answer is going to come from some kind of iterative work. So you have expertise on substance, we have expertise on survey design, execution and presentation, and it's together which make the good product. Jon Cohen 15:28 So I just wanted to level set there, because I don't think we come in and solve all problems magically, but we have a workflow, so we have that. But this company really was born of my grievances that I've assembled over time in the field. A lot of them I've already talked about in terms of the workflow. I just think the tooling in the service space has been inadequate for the moment and has led to a lot of thinness in the work. If you look at most polling write ups today, whether they're in the news, around politics, or from market research firms. Jon Cohen 16:02 I think there's a thinness to it, which is just kind of tremendously at odds with how interesting the times we live in are and how important it is that we understand ourselves, the voters, kind of the consumers, the employees. It's an incredibly important moment. And most of the write ups are pretty formulaic. I think that a lot of that comes from the fact that the tooling that we have means that we spend collectively 80% of our time as an industry just collecting numbers and organizing them and putting them out and leaving far too little time for thinking, for thinking at the front end about what we're trying to do, and certainly at the back end about going deeper. If you just look at the same cross tabs, that same spreadsheet that you get after every survey, you're going to write some version of the same story. Jon Cohen 16:55 Now, the trend might have moved and that's a good story, but you're not digging deeper, you're not asking a new question. And so what we've built at TrueDot is, how do you get rid of that 80% or move that 80% of drudgery to 10% of coordination and spend more time, have a tool that allows you to explore what we're really after, to benefit from that context that we've built in an easily retrievable way, and then to kind of start to do this statistical weighting at an early enough time in the workflow and the process that you can start to explore. You don't have to just look at every poll by party. Yes, we are really divided by party in this country, and that has increased over time. Jon Cohen 17:36 And also, if you only look at the differences between Republicans and Democrats and independents, and that's what you write your story on, that same column every time, maybe there's more to it, maybe there's more to the story. And kind of having the right tooling allows you to do that exploration. And so that's really what we've built, which is kind of, for me, incredible not only as kind of running the business, but incredible as a researcher. This is the tool I always wanted. This is what I wanted in that newsroom. Jon Cohen 18:08 And you took me back to that feeling. And I was there at the Washington Post for both of the Obama victories and the stress of producing those numbers. And we did it daily in the last few weeks. To think that I could have had a tool like this. And you know, I'm proud of the work I did. Jon Cohen 18:28 We did great work. It was highly accurate. I also think we could have gone deeper. And with a tool like TrueDot, we can go deeper today. And that's really exciting for me. Jason Baumgarten 18:39 Well, I think what really was amazing, being able to work with you and the team and the tool, is it also democratizes something that was a real specialty. You went to your market research team or you went to your polling team, and they did it and then brought it back. As a leader, as an executive, often we don't get told the truth. I mean, it's one of the big challenges. The more you're running things, the less, you know, often ironically. And I think putting these kinds of tools back in somebody's hands, say, you don't need to run this through eight layers. Jason Baumgarten 19:11 You can actually test your own hypothesis. Sort of like the vibe coding moment for market research, perhaps, or polling. So I think that it's something to think about. I always remember I had this market research professor at Stanford, and she always said, don't ask the question unless you've thought through what you're going to do with the answer. Jason Baumgarten 19:31 And it sounded really easy, right? You'd sort of like, but of course, if I know the answer is this, I would do that. And she would force us to say, okay, if the answer is anywhere on this spectrum, what would you do differently? And you would realize how quickly so many of the questions didn't need to be asked because it wouldn't change your insight or it wouldn't change your outcome. It wouldn't change your behavior, and it really forced you to do a better job. And I think your comment about having surveys that allow you to kind of put slop out there quickly are not good, because they reinforce this notion that I'm doing research, but actually I'm not doing anything that will change my behavior or my decision. And I think that ability with TrueDot to get much deeper allows you to have much more nuanced view of like, oh, I would actually change what I say because I have that depth that I didn't have with a quick, sort of sloppy version. Jon Cohen 20:25 Totally agree. I think it's a really smart comment. And as a Berkeley person, it's hard to credit Stanford anywhere, but I think that's incredibly smart. I mean, I often work with clients on kind of like thinking through the headlines that they want, and I really push them to think, not just the headlines they might want for kind of their product, and not say like 90% want this product. It's easy to push people in that way and write advice questions. But the best kind of questions are ones where it's a useful, interesting answer regardless of how the data come out. So exactly what that professor was going for, right. Jon Cohen 21:00 That is going to be the difference. Because if you just want one particular answer, you're going to write an overly biased question. You just are. And then what's the point of that? Now, I do have to say, in market research, that is most of the work. Yes, a tool like TrueDot gives an executive the opportunity to seek the truth in many circumstances. Jon Cohen 21:20 That's not what they're after. They're after filling the next line on the trend, showing NPS up into the right and only up into the right. So kind of like most of the market research industry isn't geared toward truth seeking. It's for number generation, and with a purpose. I'm not saying there's no purpose, but it is to run the system, not to learn something new. Jon Cohen 21:45 I had a conversation with the chief customer officer of a very large multinational corporation who spends eight figures on one of these experience management software systems. I asked him, what's the value you get for learning about your customers from this investment? And the answer was zero. Now, obviously the answer is not zero for value in the system. It's a business that has franchises, and those franchisees have to meet a certain threshold to get paid. Jon Cohen 22:18 And so there's a value in the collection of that number for the business. But there was no net new learning about the customers. And so the truth seeking part of the industry is small. Jason Baumgarten 22:37 What an amazingly missed opportunity. I mean, you think about the speed at which business is getting disrupted. And I was reading an article somebody wrote recently about false moats or false walls that businesses have around them. The notion that it's defensible because we have amazing customer service, or it's defensible because we delight our customers. And you're like, do you actually delight your customers or you're just perpetuating a system? So I think that's kind of a tragic story to me, that there's not that sort of improvement bias in the seeking of the data. Jon Cohen 23:06 Like there's some, right? I mean, there are truth seekers out there. My own kind of view is that it's a real market, certainly big enough for a startup like mine. And then there's the next circle around that. Are the people that we can teach to care, that you can give them reasons to care, that you can share why it's better to take one approach versus another. Jon Cohen 23:26 And you mentioned missed opportunities. I totally agree. I don't think there's any reason why you can't have both in a customer survey thing. So go for both. But the larger share of the market, the biggest part of the 150 billion dollar insights market, frankly, we're going to have to shame into caring. Jason Baumgarten 23:44 Well, hopefully customers in the market does that, and it's out of need that people move. I want to dig into a story in your polling journey that I thought was fascinating. Maybe tell everybody a little bit about the Michigan Senate primary that you ran and sort of what the lesson was there. Jon Cohen 24:20 Referring to the Michigan Senate primary from just a few weeks ago, we did not actually do any original polling or proprietary polling in that race. What I was doing, I advised an organization called DVHQ, which does election night race calling and has set up an operation to do that. Jon Cohen 24:20 And so I was in their offices in Washington D.C. on primary night, and it was deemed to be an uninteresting evening. One of the candidates had an 11 to 18 point lead in the polls. And so we were just kind of like working on the system and kind of watching a night, and it was meant to be uneventful by the pre election polling. I have a system that goes out and I send agents out every day to get all of the polling on every Senate race. So I had this repository, and so what I did is I looked at my repository of pre election surveys and said, what should we watch for over the course of this evening that might indicate that all the polls are wrong, that Al Sayed won't win this very easy win over Stevens. Jon Cohen 25:10 And so kind of I workshopped 8 things to watch for over the course of the evening, and kind of all of a sudden they started going yellow. Meaning like, oh, this isn't turning out the way that the pre election polls had indicated. And in fact, he ended up winning by, I believe, under a percentage point or maybe a percentage point. And so it was another one of these, and we've had a lot of these over the past decade, the polls were wrong moments. Jon Cohen 25:39 I looked at it as a chance to say, what can we learn about pre election polling and also election analysis going forward, I should say, because the polls were all so heavily favoring Al Sayed. The prediction market started the evening at 99%, he's going to win, right? Because they always overestimate kind of the polling average. Jon Cohen 26:03 So you had this, again, part of the reason people thought it would be uneventful is because everyone, on call she and Polymarket, nearly everyone, said the same thing that the pre election polls did. And then what it became is I had already effectively written the postmortem on the pre election polling because we had these hypotheses going in about what could be wrong in the polling that would show up on election night. Jason Baumgarten 26:57 You wrote something that I'd love you to unpack because I think a lot of executives have this exact scenario play out. It may not be a poll, it might be some other form of insight or data that their teams are providing. But I think this idea of even the framing of what are the things, if this were to go wrong, that would show us it going wrong before we get hit in the face with it, and how do we react to it, and how do we do that before it's an issue. Jason Baumgarten 26:58 But you wrote, plausibility is not validity. And I thought it was an interesting framing. And as you think with your CEO hat on, if another CEO is sitting with you saying, what can I learn from all that? I'm not in the polling business. But what are the broader leadership lessons that you would extract from that? Jon Cohen 27:17 When we get data leading up to a product launch or leading up to anything that we have to make decisions about as an executive, I mean, it's so interesting. I mean, I think you always have to interrogate your priors, right? So often you look at a set of data and it's the confirmation bias, which is everywhere. You see in something what they want to believe, and how you pause and understand that that bias is everywhere in all of us and really understand the numbers we're looking at, where they came from, why they were collected, and what they might mean. And the Michigan example is one where it's like, write that autopsy. Jon Cohen 27:58 It's the same thing as the headline that you wanted to write. The Stanford professor suggested, what would we have to believe to think of things in different scenarios? And so it's a pressure test in political polling. You look at different likely voter models. There's not one set of numbers. Jon Cohen 28:17 Yes, I had to publish one set of numbers when I was at the Washington Post and ABC News. You come up with, here's the estimate. But really behind that were 20 different scenarios. And we picked the most plausible one on a variety of factors. And the same thing's true kind of in a product launch. Jon Cohen 28:36 The same thing's true in understanding the divide between boards and CEOs, it's like there are different scenarios. The data is one input, but they don't only say one thing. And so kind of like having that bigger, wider view. And also, I do think there's a value in about election polling. You get a result, and then you can go back and look at the methodology. Jon Cohen 29:01 So in the Michigan example, it turned out that the survey that was done with live interviewers turned out to be more accurate when it came to estimating the vote choices of black voters in Michigan than the ones done through other methodologies. So that emerged from that exercise. So kind of like, how are we always testing our methods over time? How do we bring the concept of an election day to a corporate moment? And so we can start to assess right from wrong? Jon Cohen 29:32 So the plausibility versus is not validity item is like, hey, how do we get better? You and I, off camera, were talking about the role of expertise. Expertise comes in and seeing that something's valid, understanding that one thing is better than another, and that comes from experience. That comes from things other than just like, is that number right or wrong? We don't know. Jason Baumgarten 29:56 One of the things you're touching on, though, which I think is so important, is there's this narrative right now, certainly in tech communities, about humans have judgment and somehow we all have taste as humans. And I think that's really dangerous. I know enough about myself to know many areas of life where I have no taste and no judgment. And I'm still learning new areas that are pointed out by others regularly where I have no taste and no judgment. But one of the things that's interesting about polling in particular is that you get the results and you get the outcome, and then you get to go back and figure out exactly, as you were saying, what was right, what was wrong about the methodology. Jason Baumgarten 30:30 And that gives you this learning loop that a lot of people don't have. Like, as CEO, you don't often always know whether you are right or wrong, or you don't know for so long that you've forgotten what the decision was that led you to being right or wrong. So I know you gave a talk called the researcher in the loop. How do you think about that, the benefit of having the data in such a pristine way, like polling, where you can go back and interrogate, versus when you think about your CEO hat on, all the difficulty of getting that loop right? Jon Cohen 31:02 Sure. Well, first of all, it's not pristine, let's dismiss that notion. Jason Baumgarten 31:02 Don't burst my bubble, John. I'm living in it, I appreciate it. Jon Cohen 31:10 It can be. It's increasingly so. So perhaps with the right tooling like TrueDot. So thank you for the opportunity to do a product plug. But no, I think, look for the opportunities to have something like an election day. Jon Cohen 31:21 I do think that these models should be tested. I do think you should aim at something that you can have a notion for. I mentioned NPS earlier. Most systems are not run to kind of learn stuff, but 60% of the bonuses in the Fortune 500 are based on it. And so it's no surprise that you have a sampling strategy and an analysis strategy that shows those increasing at opportune moments. Jon Cohen 31:50 And so kind of like, how do those relate to corporate performance over time? I think the evidence was pretty paltry to start, but kind of put it through the steps, like work on it. Having a stress test for systems of both customer feedback, employee engagement stuff. Too often it's just these numbers are performative or meant and designed to be performative, or again, just to keep the system running. Jon Cohen 32:24 But how do you have a system of learning? And that is really an opportunity. And that's not just about more frequent surveys, although I think that is possible. It is about why are we doing this? How do we have these moments where we can see, are we right or wrong? Jon Cohen 32:40 Is it helping? Are the businesses improving because we have the system? I think that speed isn't everything, but if you're not fast enough, it's a completely irrelevant system from a learning standpoint. And I think we now have the opportunity to have fast enough systems that can meet the speed of business and the decision making that needs to happen. And that means that there's a higher bar on the connection of these data that we're collecting to the business outcomes. Jason Baumgarten 33:12 I mean, certainly in my world, I always laughed at, it's like Lake Wobegon when we're recruiting, in that everyone says, I'm a great recruiter, I've got great. And I said, what's your stats? Do you have a list of everybody you've recruited and how they did? And they're like, no, but I know I'm good. Jason Baumgarten 33:27 I'm like, all right, we'll just make the list of the last ten. How'd they do? And they're like, well, maybe I'm not so good. And I say, by the way, the people you didn't hire, what happened to them? And they're like, oh, I'm really not so good. Jason Baumgarten 33:38 So I do think having that feedback loop and having the data can be really powerful. On the other side, you said that when you're thinking about a good AI research tool, that something could be good enough to say, I don't have confidence, the data is not good enough. How have you thought about that in terms of TrueDot? And have you thought about that even when, in your own decision making, when you don't have enough confidence in the data to make a call? Jon Cohen 34:09 Any good system has to be able to say no. And this is one of the biggest things with AI, a well documented tendency to be sycophantic, to tell you how great you are, how smart you are. That's exactly the wrong thing when it comes to building a trustworthy system. One of the things that we've built is, and going back to my Anderson Consulting days, how do you take data in different formats, very messy formats, how do you take the chicken scrawl on someone's handwritten tax form and put it in a system? Jon Cohen 34:42 And the pathways to doing that, normalizing data, is exceptionally important and very difficult. And you have to be able to say, well, these two things don't match, these don't go together. And so what we've built is a system that can take data from anywhere, not just those that are collected on TrueDot, but those collected by any system, by any methodology. How do you bring them together and put them together when it's appropriate and say you can't when it's not? Jon Cohen 35:12 So even the Michigan example, there were those different methodologies. How do you have a system that can understand that it matters whether a survey was done one way versus another. It matters whether I'm looking at two subgroups that are compatible and two that are not. It matters when you say, well, I don't have enough executives in Brazil to break apart this particular finding or that one. And so a lot of that you can build in. Jon Cohen 35:40 You can build in a minimum column size, sample size, kind of threshold, but you have to build in these things to an AI system so it can help you very quickly discriminate on things that we should say and things that we shouldn't, and things that are backed up methodologically and things that aren't. And so a lot of that we can build and have built into a system. And some of that is ongoing human judgment, which means we're still a little bit ahead of the machines. Jason Baumgarten 36:21 One of the things I'm curious about, you've had this incredible ability of a very consistent through line in your career that many people haven't had. You've done something at the core in so many different places and evolved it, but you've had an amazing kind of an apprenticeship approach to becoming a true master at something. I'd be curious as you think about the work AI is eating and the willingness of people to let AI do things instead of giving that learning curve to humans. Where does the next Jon Cohen come from in our world today? Jon Cohen 36:51 So, first of all, it's true that I can tell a consistent story from college to now. As you well know, journeys are not always so straightforward, and certainly this was not one that benefited from a tremendous amount of planning, at least not kind of long term planning. So there is a consistency and I do think there's been a core drive, but that wasn't always by design and a lot of good fortune, kind of serendipitous meetups with people who became mentors and a lot of good fortune along the way. But I do think really hard about how this goes forward. I'll give you one example. Jon Cohen 37:33 A lot of my colleagues in the profession get worried when they see our software because one of the things I like to show off is here, you no longer have to check these numbers. I'll show you why you should trust this system over time. But more importantly, you put in a draft and it'll check it for you, and it'll check it more accurately and obviously faster than anything you could do manually. Well, the reason they get nervous is that checking numbers is the first way that you train a new researcher. At many, all of the organizations I've worked for over time, go check this. Jon Cohen 38:12 And then in checking it, they have to go, well, maybe I'll go check the crosstabs, or maybe I have to go back and run something in SPSS on the raw data. So checking numbers is core to how you train people about systems and the work and also the underlying integrity that it takes to understand the process, and it has to be right 100% of the time. So checking numbers, that's the way you train people. Well, I come in with a system that you don't have to do that. That's disturbing. Jon Cohen 38:45 Now I then have to say, well, what we were doing in teaching people how to check numbers, which I should say is how people are trained today. So this isn't just how I was trained or people coming a little bit after, but today it's somewhat controversial. Today I say, well, yes, we need to focus on training, but the tools that we have used to do that should change. It's like people are talking about writing in AI, maybe a separate subject. But the idea is, at the end of the day, how do you teach someone to think. How do you teach someone to think about polling in a way that allows them to develop their own judgment, their own taste over time? Jason Baumgarten 39:28 It's a good point because I think the training on the tools is often a backdoor into training something broader that you're trying to get. And I can recognize what you're saying in all the different apprenticeship environments I've been in, where it's actually an indirect route to the thing you're really trying to get at. And what you're really trying to get at is takes more work, so you don't do it. So you sort of give them the easy task. But it's a good push for people, and especially for all the leaders that have teams out there, to think about. Jason Baumgarten 39:57 How do I have to apprentice differently and not rely on the maybe the way I've done it, that isn't as good as it could be, but put real thought into it. And the implications, I think, are enormous, not just for how we train the next generation, but how we price our products, right? This whole notion of, like, it used to be pretty straightforward about how you priced, and there was this system that you needed to fund, and all of that is being thrown out slowly, as things, things are slow to change, of course. Jason Baumgarten 40:28 But this whole notion of, what's the ultimate value of what we're providing and therefore, what's the right way to charge for that? I think all of that is being challenged right now. And it starts from, like, how many people do we need to run things the way they're currently run, or to your point we're getting to is, how do you come with a system where it's like, oh, you don't just have to think about a system that fills a chart, but also gives you an opportunity to learn well, that's now more valuable and maybe requires more people or fewer. Jason Baumgarten 41:10 It's so interesting. I was chatting with another CEO who said, we're trying to get the more of an employee mentality pricing into software because of AI? And I said, well, that's great. When do I get to give it a performance report? And he sort of nervously laughed, and I said, no, the problem is with my employee, I can decide where on the spectrum of a bonus that employee is, including whether to have a job or not. The AI system might produce a lot of output, but a lot of it I might judge poorly. And you want to fix a set price, which doesn't work in that world. Jason Baumgarten 41:35 And he hadn't been thinking about it that way. But I said, until you can actually reliably recreate a top decile employee, you can't charge like one. Which I think was upsetting to him, because that was the plan. But we're in this new exploration, I think we're all sort of learning as we go what works and what doesn't in AI, both in our own lives and how we use it, but also the translation to business models. John, you've had this incredible career and so many true institutions around how people think about consumer insights, polling data more generally, as you think about, and I take your point, the through line is sometimes messier, especially looking forward than backward. Jason Baumgarten 42:17 But if you take a neater historical view with the benefit of hindsight, what's the thing about you that makes that all work? Jon Cohen 42:17 One of the things I've always loved about the polling profession is that people do come from a lot of different backgrounds. One of my bosses at ABC News was an English major. Other colleagues have been PhDs in political science. Others, increasingly, have outsized data skills. Jon Cohen 42:43 And so it's a diversity of talent and perspective on the pursuit of understanding. And that has been kind of deeply, I've benefited a lot from that. Those are the kinds of people I like working with. Not a single minded or kind of narrowly trained type. Jon Cohen 43:04 And so I've appreciated the different perspectives, and in some ways that reflects on me and kind of my diverse interests, and that I can bring diverse interests to something to understand. So I don't have one narrow perspective that's not only a data perspective, not only a historical perspective, not only one about language, but it's about all of them. And kind of the way I tie it together again is with the systems approach that I alluded to or talked about earlier. It was just like, how do I bring all of that together in a way that makes sense to build the system? And I had to do that because I was hired at the Washington Post to be director of polling when they were like, hey, we had two people doing this, but maybe we could have one. And of course, that was crazy. Jon Cohen 43:53 And I couldn't take a day off for a very long time, and ended up hiring someone else. But I was able to, because of scarcity, build that system, because I did have that. I was again the history major who they taught to program. And that has benefited me in ways that I never could have anticipated throughout my career. Jason Baumgarten 44:19 The joy for all of us that learn difficult languages like Pascal and C is that we can now vibe code. And it feels like magic. Mine was COBOL. In an earlier episode you talked about your first job coming because you knew how to type. I was the... Jon Cohen 44:30 I knew how to program in SPSS. That was, turns out COBOL is still in demand, apparently. Jason Baumgarten 44:38 Apparently, I'm gonna have AI teach me how to fall back, update my skills. Jon Cohen 44:38 And I must, my fallback. COBOL, you may not need to update much. Jason Baumgarten 44:47 John, let's do our speed round. We do this every time. I'd love to start with just a book that you'd recommend that has changed how you think in the last year or so. Jon Cohen 45:19 So I don't read as often or as much as I should, but I did read London Falling, or listen to London Falling, as that counts as reading on Audible, which is a terrific book by Patrick Radden Keefe, who I admire so much as an author. And it was just a tremendous tale told. And it was nice to kind of engage with the book because I'm mostly reading Substack and listening to podcasts and kind of have different inputs in my life. I tried to do fast and slow. So I think that there's a lot of fast things, and then sometimes the benefit of a book is just it took them a long time to do it, so you had to, you couldn't fire it off. Jason Baumgarten 46:05 Okay, the worst career advice you ever got that was well intended, or bad career advice that you think gets given out that is just bad. Jon Cohen 46:06 I think the worst advice that was true when I graduated from college, and I see today, is that you actually have to have deep passion for something and choose a career. And I think that, particularly when one is just graduated from college, I think there's far too much pressure to be something. And I think that I benefited from advice that was like, you know what, just go out and do something that you find edifying that puts you in a position to make better choices. When you have more constrained situation, whether you are in debt, whether you have a family, other things will happen that kind of make it harder for you just to choose for yourself. And I think there's this magical time from graduating college, and maybe we all have it now with AI, I don't know, but there's this time where you can just choose things, or if you're lucky enough to be in the position where you can just choose things that you find make you better. Jon Cohen 47:01 This is why I went to work for Anderson Consulting. If I thought I was choosing a career as a computer programmer, I never would have taken that job, for a variety of reasons, never would have done it. And yet we've spent the better part of an hour talking about how much that benefited me in my career. But it was because I didn't succumb to the pressure of choosing a career when I was 22 years old. Jason Baumgarten 47:35 It's such good advice and such good framing to give yourself the permission that if you don't know your life's purpose at 22 or your calling at 22, it's okay. It's okay to just jump into something that gives you learning and gives you exposure and helps you find it. I think there is a lot of pressure on young kids, especially to have it all wrapped up, and even how people apply to college now. It's like you're supposed to write the retrospective of your life when you're 16. Speaking of that, what advice would you give your younger self? Jon Cohen 47:56 I mean, since I benefited from that, and I will, there's a longer version of that story, which I'll tell you sometime, about that career advice that was imparted to me, actually, in Krakow, Poland, on a study trip. But my younger self did get that advice, and I did take it to heart and benefited from it. I think it is just to kind of be kind. Jon Cohen 48:17 I mean, how generic is that? But I think, how one shows up in the world is everything. I think it's increasingly difficult in situations like this. You and I are talking over video conferencing. Jon Cohen 48:31 To focus on how you show up and be engaged and focused in a world where we're interacting increasingly virtually is exceptionally difficult. How we do that when we have AI at our fingertips. How do you learn to think critically and do that? So I mean, none of this is particularly unique, but I think it's increasingly difficult. The other thing is, put the phone away. Jon Cohen 48:56 Kids or adults, right? I think we all have attention challenges. I've done lots of reading about ADHD and the notion that, and I think some of the original ADHD researchers are trying to relabel the whole thing. Vast variable attention stimulus trait. Like, we're all symptomatic, and how do we get a little bit of focus in our lives on ourselves and others is paramount. Jason Baumgarten 49:23 John, I have to ask this one, survey question you would just ban, could never be asked again. Is there something that comes to mind? Jon Cohen 49:41 Agree, disagree scales kind of have a proven acquiescence bias. People like to be agreeable, particularly in this country, although there are some cultures that are even more so. And I would just get rid of the entire answer category. I would also go after NPS. I know you asked for one, but I'm going after two. The survey methodologist in me, NPS is a bad scale. People ascribe way too much power to it. Jon Cohen 50:00 There's no one perfect question in the world, and I think it's often a stand in for deeper, potentially more valuable work. And I say that, and both of those pieces of advice can be totally wiped away because people want trend, and that's fine, that's okay. There's a reason to ask a redi-cook question that was asked in the 50s, see how it is now. But anyway, I could go on. Jason Baumgarten 50:26 I'm sorry, but that's okay, you got the magic. That's good. All right, we're going to end with a job you would be genuinely terrible at. Jon Cohen 50:51 Oh gosh, I am genuinely terrible when it comes to doing the same thing the same way all the time. I'm just too interested in systems, I'm too obsessed with AI tools that kind of allow me to rethink how I do things. Jon Cohen 50:51 So if I had to do the same thing the same way every day, I mean, I don't do it because I can't. My mind doesn't work that way. It repels me and I'm bad at it. Jason Baumgarten 51:11 Well, John, thank you for your wisdom, your insight, your humor. It's always fun to spend time with you. I think for everybody, hopefully they gathered something of interest, value and curiosity. Probably the word I would use to wrap the conversation is, your level of curiosity about the world is amazing, and you've given us all a tool to be more curious. So here with John Cohen, the founder and CEO of TrueDot, if you ever have insight, market research or customer service or polling or all the things we tend to ask questions about, please check it out. Jason Baumgarten 51:40 It's been fun to partner with them over the years and get lots of insights from John and his team, and listen to another episode of Fit Happens. Thank you so much, John. Jon Cohen 52:13 Thank you. Pleasure to be with you.