Latest / Razib Khan's Unsupervised Learning / Jesse Singal: after the replication crisis and into the youth gender medicine debate
Transcript
- 0:00SPEAKER_01: This podcast is brought to you by the Albany Public Library Main Branch and the
- 0:04SPEAKER_01: generosity of listeners like you.
- 0:09SPEAKER_01: What is a podcast?
- 0:16SPEAKER_01: God, Daddy, these people talk as much as you do.
- 0:26SPEAKER_00: Razeeb Khan's Unsupervised Learning
- 0:28SPEAKER_00: Even if you and your partner are healthy,
- 0:30SPEAKER_00: there's still a chance your child can develop a serious genetic disease.
- 0:33SPEAKER_00: This is because every embryo has new changes not present in either parent.
- 0:37SPEAKER_00: Most of the time, these are benign, but sometimes they can be catastrophic.
- 0:40SPEAKER_00: Orkin's whole genome embryo reports directly screen the embryo and analyze these de
- 0:44SPEAKER_00: novo genetic mutations.
- 0:46SPEAKER_00: Discuss embryo screening and IVF with a genetics expert.
- 0:50SPEAKER_00: Hey, everybody.
- 0:51SPEAKER_00: I am here with the Unsupervised Learning Podcast with Jesse Singel,
- 0:56SPEAKER_00: who is – he has the podcast Blocked and Reported with Katie Herzog,
- 1:00SPEAKER_00: and that's on Substack.
- 1:03SPEAKER_00: And he also has a Substack single-minded.
- 1:06SPEAKER_00: He is – was an erstwhile contributor, I guess, to NYMAG.
- 1:11SPEAKER_00: He covered the social science beat.
- 1:13SPEAKER_00: And I'm going to mostly be talking about – I think mostly I want to talk about – Jesse,
- 1:17SPEAKER_00: I want to talk about your book.
- 1:19SPEAKER_00: about psychology that you came out with a couple of years ago.
- 1:22SPEAKER_00: I think that was it, like two or three years ago?
- 1:25SPEAKER_00: Yeah, 2001.
- 1:26SPEAKER_00: Yeah, so it was three years ago.
- 1:28SPEAKER_00: Quick Fix by Fadso Psychology.
- 1:31SPEAKER_00: Can't cure our social ills.
- 1:32SPEAKER_00: So yeah, I think I read it when it came out.
- 1:34SPEAKER_00: It was a lot of topics.
- 1:37SPEAKER_00: What I will say is it was about a period.
- 1:40SPEAKER_00: I'll tell the listeners,
- 1:40SPEAKER_00: I feel like there was a period between 2005 and 2015 to some extent overlapping
- 1:46SPEAKER_00: actually with the first term Obama era with nudges and other things like that and
- 1:51SPEAKER_00: also in the wake of the financial crisis.
- 1:53SPEAKER_00: where psychology became a really big deal, cognitive biases and all these other things.
- 2:00SPEAKER_00: Priming really got roided up, I think.
- 2:03SPEAKER_00: And, you know, it was just in the news all the time.
- 2:06SPEAKER_00: Dan Ariely had a NPR, you know, thing and all this stuff.
- 2:12SPEAKER_00: And then reproducibility crisis and everything happened and kind of came crashing down.
- 2:16SPEAKER_00: I'm going to say you have another book on gender transitions, I think.
- 2:21SPEAKER_00: We'll talk about the second half.
- 2:22SPEAKER_01: Yeah, youth gender medicine to be specific.
- 2:24SPEAKER_00: Youth gender medicine.
- 2:26SPEAKER_00: Yeah, youth gender medicine.
- 2:28SPEAKER_00: Okay,
- 2:28SPEAKER_00: yeah,
- 2:28SPEAKER_00: which I don't know anything about this stuff because I'm not a youth and I don't
- 2:31SPEAKER_00: think I need gender medicine.
- 2:33SPEAKER_00: But maybe you can educate me on that.
- 2:36SPEAKER_00: But you tweeted out years ago – I'll tell you a story.
- 2:40SPEAKER_00: You tweeted out years ago that you were going to write –
- 2:42SPEAKER_00: An article for New York Mag about some big thing where the social science was all
- 2:48SPEAKER_00: wrong or not robust,
- 2:51SPEAKER_00: and you ask people to guess.
- 2:52SPEAKER_00: I guessed correctly.
- 2:53SPEAKER_00: I think I guessed this was about implicit bias.
- 2:55SPEAKER_00: which is like,
- 2:56SPEAKER_00: it's kind of a real effect,
- 2:59SPEAKER_00: but it's not nearly as robust or replicable or powerful as people think.
- 3:04SPEAKER_00: The implicit bias,
- 3:05SPEAKER_00: just so the people know,
- 3:07SPEAKER_00: the sexiest thing about this is you go take a test and you look at a black face or
- 3:12SPEAKER_00: a white face or there's words related to men or women or whatever,
- 3:15SPEAKER_00: and it figures out what your implicit bias is.
- 3:17SPEAKER_00: It gives you a result back and
- 3:19SPEAKER_00: You could be like,
- 3:19SPEAKER_00: oh,
- 3:19SPEAKER_00: I'm racist,
- 3:20SPEAKER_00: but I'm not sexist or I'm both or I'm sexist,
- 3:23SPEAKER_00: not racist,
- 3:23SPEAKER_00: these sorts of things.
- 3:24SPEAKER_00: And it was really cool party factoid, you know, like, oh, there's these implicit biases.
- 3:30SPEAKER_00: Everyone understands it.
- 3:31SPEAKER_00: So I said I said that that was what you would write about because and I believe
- 3:35SPEAKER_00: what year did you write that article?
- 3:37SPEAKER_00: Was it like 2016, 2015?
- 3:37SPEAKER_00: Yeah, I want to say 2017, but somewhere right in there.
- 3:41SPEAKER_00: Okay, so I have a friend, Chris.
- 3:44SPEAKER_00: Actually, his real name, well, Hunt Stilwell.
- 3:47SPEAKER_00: He's not in academia anymore, so I can say his name now.
- 3:49SPEAKER_00: He's a cognitive psychologist at the University of Texas.
- 3:52SPEAKER_00: And I had a discussion with him before 2010.
- 3:55SPEAKER_00: And he basically told me, look, everybody within the field
- 3:59SPEAKER_00: uh aside from i mean other things as well like priming but he's basically like
- 4:03SPEAKER_00: implicit bias priming all of this stuff it's not going to pan out uh you know i see
- 4:09SPEAKER_00: you like writing about this like it's a real thing yes what year did what year did
- 4:14SPEAKER_01: you say he told you that uh probably 2008 wow okay prescient prescient
- 4:21SPEAKER_00: Well,
- 4:22SPEAKER_00: I don't actually – so I talked to another person I knew who was in a social
- 4:27SPEAKER_00: psychology lab at Berkeley at the time,
- 4:29SPEAKER_00: actually,
- 4:29SPEAKER_00: at a party.
- 4:31SPEAKER_00: And I knew they had dropped out of grad school.
- 4:33SPEAKER_00: They were working at startups, and I asked them why.
- 4:35SPEAKER_00: And he basically – he didn't basically.
- 4:38SPEAKER_00: He just literally told me, I think the research that I'm doing is bullshit.
- 4:43SPEAKER_00: Because what they would do is basically his advisor,
- 4:50SPEAKER_00: his postdoc or whatever that was advising him would just be like,
- 4:52SPEAKER_00: just look for correlations of the data and just fit it to some theory.
- 4:56SPEAKER_01: Yeah.
- 4:56SPEAKER_00: Basically, that's what – and he's just like, I don't want to be doing this with the rest of my life.
- 5:01SPEAKER_00: This is 2008.
- 5:02SPEAKER_00: This is what was being published.
- 5:03SPEAKER_00: So I started having an awareness and a skepticism.
- 5:07SPEAKER_00: So I wasn't entirely shocked with the – but it wasn't just me.
- 5:10SPEAKER_00: If I know about it, that means a lot of people within psychology do.
- 5:13SPEAKER_00: And so I've been talking a lot.
- 5:15SPEAKER_00: Talk about your book, what motivated it.
- 5:17SPEAKER_00: I know you were on the social science beat.
- 5:20SPEAKER_00: And I think it came out at a time when it was kind of the denouement.
- 5:25SPEAKER_00: It was kind of like the ending of the replication crisis.
- 5:28SPEAKER_00: People were just kind of like –
- 5:29SPEAKER_00: It was like, you know, kind of stages of grief for a lot of people, especially in parts of academia.
- 5:34SPEAKER_00: But I think people are like,
- 5:35SPEAKER_00: okay,
- 5:35SPEAKER_00: yeah,
- 5:36SPEAKER_00: we just need to accept that the death has occurred and we need to move forward.
- 5:40SPEAKER_01: Yeah.
- 5:40SPEAKER_01: I mean, I don't know.
- 5:41SPEAKER_01: I've just always been interested in how science goes wrong and debunking popular
- 5:45SPEAKER_01: ideas that might be incorrect.
- 5:47SPEAKER_01: And the more I learned about social psychology in general, the more shocked I was.
- 5:52SPEAKER_01: And I think my shock was maybe motivated a little bit by guilt because in some of
- 5:57SPEAKER_01: my earlier science writing at New York Mag from 2014 or 2015,
- 5:59SPEAKER_01: I had definitely contributed to them.
- 6:03SPEAKER_01: These three tricks will make you more productive or whatever.
- 6:05SPEAKER_01: Look at a photo of Einstein.
- 6:08SPEAKER_01: That's not a real one,
- 6:09SPEAKER_01: but it's actually sort of indistinguishable from some of the most overheated
- 6:12SPEAKER_01: priming claims.
- 6:13SPEAKER_01: So, yeah, I'm curious in why shoddy scientific ideas spread.
- 6:18SPEAKER_01: I'm curious in which cultural niches they fill or what sort of wish fulfillment they're performing.
- 6:25SPEAKER_01: Because I posit this worldview called prime world, where basically...
- 6:31SPEAKER_01: Things would get better.
- 6:32SPEAKER_01: There'd be less poverty,
- 6:33SPEAKER_01: less racism,
- 6:34SPEAKER_01: more educational equality if we could just tweak things the right way.
- 6:37SPEAKER_01: And I think that gives short shrift to the complexity of the world and how
- 6:41SPEAKER_01: complicated it is to fix anything.
- 6:44SPEAKER_00: yeah and like a lot of the stuff wasn't like it wasn't even like ideal ideological
- 6:48SPEAKER_00: it would be like oh like you put some eyes above a tip jar or something like that
- 6:53SPEAKER_00: and people will tip more they won't there's no risk of stealing these little things
- 6:57SPEAKER_00: that you could do to you know you know there was like there was that uh this isn't
- 7:02SPEAKER_00: like psychology as such but it's like the nutritional psychology stuff the guy at
- 7:06SPEAKER_00: cornell with the suit yeah
- 7:09SPEAKER_00: Yeah, yeah.
- 7:10SPEAKER_00: And so the issue is, like, he wasn't actually a... I'm going to say he wasn't a bad person in his intent.
- 7:16SPEAKER_00: You could tell that he was actually sincere about trying to make people eat better.
- 7:21SPEAKER_00: But...
- 7:23SPEAKER_00: his methods were all crap, you know?
- 7:25SPEAKER_00: And so ultimately it's bad.
- 7:29SPEAKER_01: I forget how it turned out,
- 7:30SPEAKER_01: but Stephanie Lee,
- 7:31SPEAKER_01: who's a really good science reporter now at the Chronicle,
- 7:33SPEAKER_01: then at Buzzfeed,
- 7:35SPEAKER_01: I think in his case,
- 7:36SPEAKER_01: if memory serves,
- 7:37SPEAKER_01: and I did not cover this,
- 7:39SPEAKER_01: there was some evidence of actual malfeasance.
- 7:41SPEAKER_01: I think in many cases,
- 7:43SPEAKER_01: social scientists convinced themselves they had found something because they just
- 7:46SPEAKER_01: didn't have the quant training to realize that
- 7:50SPEAKER_01: how easy it is to sort of simulate a signal when you're looking through a lot of noise.
- 7:55SPEAKER_00: Yeah,
- 7:55SPEAKER_00: talk about the difference between that,
- 7:56SPEAKER_00: because there were fraudulent people like Diedrich Staple,
- 7:59SPEAKER_00: and that's a problem.
- 8:01SPEAKER_00: But really,
- 8:01SPEAKER_00: I think what you're mostly talking about is the structural biases within social
- 8:07SPEAKER_00: psychology research that kind of went unchecked for a decade.
- 8:11SPEAKER_01: Yeah.
- 8:11SPEAKER_01: So there's both all these biases in how science is produced,
- 8:14SPEAKER_01: including just the bias toward positive findings rather than null findings,
- 8:19SPEAKER_01: to the point where some scientists have actually gone out of their way to say,
- 8:23SPEAKER_01: we should publish null findings,
- 8:24SPEAKER_01: which,
- 8:25SPEAKER_01: when you think about science,
- 8:26SPEAKER_01: is a complete...
- 8:28SPEAKER_01: Because otherwise,
- 8:29SPEAKER_01: the literature is completely dominated by positive findings,
- 8:32SPEAKER_01: not because there's more of them because they get published.
- 8:36SPEAKER_01: And there's also the extent to which if you don't have guardrails like
- 8:40SPEAKER_01: pre-registration or published study protocols...
- 8:44SPEAKER_01: You can find whatever you want,
- 8:45SPEAKER_01: especially as data becomes more and more available and we get more and more seduced
- 8:49SPEAKER_01: by data.
- 8:50SPEAKER_01: You can tell any story you want.
- 8:53SPEAKER_01: I mean, there's a cliche that I include in the book, torture numbers and they will confess to anything.
- 8:57SPEAKER_01: I don't think when I was starting out as a science journalist,
- 9:00SPEAKER_01: I understood how easy it is to just do enough,
- 9:04SPEAKER_01: quote unquote,
- 9:04SPEAKER_01: exploratory analyses till you find something and then you retrofit your hypothesis
- 9:09SPEAKER_01: to it,
- 9:09SPEAKER_01: which...
- 9:11SPEAKER_01: Again, by that method, you can prove anything, and science is pretty useless.
- 9:16SPEAKER_01: One of my goals with the book was to – there's obviously a lot of other people
- 9:20SPEAKER_01: talking about this – but to help push back against that and to nudge science toward
- 9:23SPEAKER_01: better practices.
- 9:25SPEAKER_00: Well, I think you have some – you put yourself through some quantitative training, right?
- 9:29SPEAKER_00: You know how the statistics work somewhat.
- 9:33SPEAKER_01: Yeah.
- 9:33SPEAKER_01: I got a master's degree that was quant-heavy as far as these things go, a public policy degree.
- 9:40SPEAKER_01: I know how to read a paper.
- 9:41SPEAKER_01: I know how to read a regression table.
- 9:42SPEAKER_01: I definitely need to ask a statistician for anything complex.
- 9:48SPEAKER_01: Now,
- 9:49SPEAKER_01: I do think it's been a marked advantage for me that most science journalists,
- 9:55SPEAKER_01: when it comes to stats,
- 9:56SPEAKER_01: are at a one.
- 9:58SPEAKER_01: If 10 is like a PhD in a quant field, I may be at a three or a four.
- 10:02SPEAKER_01: That alone makes it way easier to do my job and helps me avoid certain errors I
- 10:05SPEAKER_01: would have otherwise made.
- 10:07SPEAKER_00: Yeah.
- 10:07SPEAKER_00: I mean,
- 10:08SPEAKER_00: so there was a period like in the early teens when Nate Silver was making a big
- 10:12SPEAKER_00: splash about data journalism and all that stuff.
- 10:16SPEAKER_00: And,
- 10:16SPEAKER_00: you know,
- 10:17SPEAKER_00: you didn't like do data journalism as such,
- 10:19SPEAKER_00: but you're definitely quantitatively literate journalism,
- 10:24SPEAKER_00: you know.
- 10:26SPEAKER_00: Okay, so this is a Jesse Singel response on Twitter.
- 10:29SPEAKER_00: Well,
- 10:29SPEAKER_00: actually,
- 10:30SPEAKER_00: if you look at the supplements and you look at the distribution of p-values,
- 10:34SPEAKER_00: I'm joking,
- 10:35SPEAKER_00: but you know what I'm saying.
- 10:36SPEAKER_00: I appreciate that, but I wonder what the demand for that really is.
- 10:43SPEAKER_00: Because you're showing why you actually are saying what you're saying instead of
- 10:49SPEAKER_00: just saying it,
- 10:50SPEAKER_00: asserting it,
- 10:51SPEAKER_00: right?
- 10:52SPEAKER_01: I think...
- 10:53SPEAKER_01: Yeah,
- 10:53SPEAKER_01: so,
- 10:54SPEAKER_01: I mean,
- 10:54SPEAKER_01: my two main jobs are writing about social science and talking about internet
- 10:58SPEAKER_01: bullshit with Katie on the podcast.
- 11:01SPEAKER_01: I...
- 11:02SPEAKER_01: make more money talking about bullshit, but not by that much.
- 11:06SPEAKER_01: I think there's demand,
- 11:08SPEAKER_01: especially because of the gender stuff,
- 11:09SPEAKER_01: I'm sure we'll get into,
- 11:11SPEAKER_01: for someone who can explain to readers why a study is good or bad,
- 11:16SPEAKER_01: much more often bad in my case.
- 11:19SPEAKER_01: And Substack lets me do that at interminable length, like article 5,000 words.
- 11:25SPEAKER_01: And I'm under no illusions.
- 11:27SPEAKER_01: I'm sure a lot of my readers don't read at all.
- 11:28SPEAKER_01: But
- 11:29SPEAKER_01: Oftentimes,
- 11:32SPEAKER_01: I used to often be the only person who would look at these studies at all skeptically,
- 11:38SPEAKER_01: both youth,
- 11:38SPEAKER_01: gender,
- 11:39SPEAKER_01: and other stuff.
- 11:40SPEAKER_01: Or if there were other people doing it, they didn't have my platform.
- 11:42SPEAKER_01: So I think there is a market for the right kind of nerdery,
- 11:46SPEAKER_01: as long as you explain stuff in terms like normal people understand.
- 11:50SPEAKER_01: And that's easy for me to do because I don't understand super complex stuff.
- 11:53SPEAKER_01: I need to explain down to me in the first place.
- 11:57SPEAKER_00: Yeah,
- 11:58SPEAKER_00: so going back to some of the topics in your book,
- 12:02SPEAKER_00: some of these things I know a fair amount about or I read about it.
- 12:08SPEAKER_00: One aspect that I was only tangentially familiar with, and honestly, I felt a little...
- 12:15SPEAKER_00: How do I say it?
- 12:16SPEAKER_00: Just kind of like over reading about scientists that overhyped their research and made a lot of money.
- 12:25SPEAKER_00: And now we have to destroy them.
- 12:26SPEAKER_00: And yes, I understand.
- 12:28SPEAKER_00: So the whole Amy Cuddy power posing thing.
- 12:30SPEAKER_00: Tell us about that,
- 12:31SPEAKER_00: because honestly,
- 12:32SPEAKER_00: like I never dug deeply because I kind of felt sorry for her because she seemed
- 12:36SPEAKER_00: like a nice person.
- 12:37SPEAKER_00: Honestly, that's one of the problems with this stuff.
- 12:40SPEAKER_01: Yeah.
- 12:41SPEAKER_01: I mean,
- 12:41SPEAKER_01: Amy Cuddy came up with the idea of power posing,
- 12:44SPEAKER_01: which is that if you do an expansive pose,
- 12:46SPEAKER_01: like hold your arms out,
- 12:49SPEAKER_01: that changes something in your brain and could make you more confident and more
- 12:54SPEAKER_01: effective during negotiation.
- 12:55SPEAKER_01: And it was particularly marketed to women because this sort of overlapped with the
- 12:59SPEAKER_01: Sheryl Sandberg lean in thing.
- 13:02SPEAKER_01: The idea being that we can close some of these gaps between men and women by making
- 13:05SPEAKER_01: women basically act a little bit more like men,
- 13:08SPEAKER_01: more assertively,
- 13:09SPEAKER_01: more...
- 13:10SPEAKER_01: making fair demands rather than retreating and not speaking up for themselves.
- 13:14SPEAKER_01: So Amy Cuddy,
- 13:15SPEAKER_01: who I'm sure she is a perfectly nice person,
- 13:20SPEAKER_01: I think made some pretty overhyped claims both in her TED Talks and in her research.
- 13:25SPEAKER_01: And things really came unraveled for her when one of her colleagues and co-authors,
- 13:30SPEAKER_01: Dana Carney,
- 13:32SPEAKER_01: posted something on her faculty webpage.
- 13:34SPEAKER_01: I think she's at one of the UCs.
- 13:36SPEAKER_01: basically saying, I don't think these effects exist.
- 13:39SPEAKER_01: I think we were p-hacking and didn't really know it.
- 13:43SPEAKER_01: My understanding is that Amy Cuddy has not really backed down.
- 13:47SPEAKER_01: And one of the interesting personality differences is between what different
- 13:51SPEAKER_01: scientists do when their ideas come under attack and the way they react.
- 13:55SPEAKER_01: And I thought Amy Cuddy was more in the camp of doubling and tripling down,
- 14:00SPEAKER_01: though I should say I haven't checked in on power posting in maybe a couple of years.
- 14:05SPEAKER_00: Yeah,
- 14:05SPEAKER_00: I mean,
- 14:06SPEAKER_00: a lot of times when scientists are confronted,
- 14:08SPEAKER_00: as you know,
- 14:09SPEAKER_00: by the falsification of something they've spent years working on,
- 14:15SPEAKER_00: they have excuses for why the evidence that you're presenting doesn't count,
- 14:22SPEAKER_00: right?
- 14:22SPEAKER_00: Whereas you know that in a different circumstance, they would be like, oh, yeah.
- 14:27SPEAKER_00: But it's like all of a sudden motivated reasoning kicks in and they got a PhD.
- 14:32SPEAKER_00: They're a smart person.
- 14:33SPEAKER_00: They're going to figure out an angle to slip through,
- 14:36SPEAKER_00: and I think that that's what happens to people like Amy Cuddy.
- 14:39SPEAKER_00: Can you talk about – have you followed – I mean this is after your book,
- 14:43SPEAKER_00: but have you followed the Francesca Gino as well as Dan Ariely stuff?
- 14:48SPEAKER_00: Yeah.
- 14:49SPEAKER_01: Yeah, to be honest, not as closely just because I've like I've shifted a bit since the book came out.
- 14:54SPEAKER_01: But yeah,
- 14:55SPEAKER_01: that that clearly is much more I need to speak carefully because there's been
- 14:59SPEAKER_01: certain legal threats level that seems to be more in the vicinity of potential
- 15:04SPEAKER_01: actual fraud as far as I can tell.
- 15:05SPEAKER_01: And I have read some of them.