Latest / Razib Khan's Unsupervised Learning / Werner Zagrebbi: Adventures in Forbidden Data
Transcript
- 0:00SPEAKER_00: This podcast is brought to you by the Albany Public Library Main Branch and
- 0:03SPEAKER_00: the generosity of listeners like you.
- 0:09SPEAKER_00: What is a podcast?
- 0:16SPEAKER_00: God, Daddy, these people talk as much as
- 0:24you do.
- 0:26SPEAKER_00: Razib Khan's Unsupervised Learning.
- 0:29SPEAKER_01: Hey, everybody.
- 0:30SPEAKER_01: I'm here today with my friend Werner Zagrebi from Twitter.
- 0:36SPEAKER_01: He is also a student at CMU in statistics, correct?
- 0:42SPEAKER_02: Statistics and machine learning is actually the name of it.
- 0:44SPEAKER_01: Okay.
- 0:45SPEAKER_01: Machine learning is just part of statistics now.
- 0:48SPEAKER_02: For real, for real.
- 0:49SPEAKER_01: I mean, it's like it's all merged together.
- 0:51SPEAKER_01: So you've been, like, tweeting out and writing a bunch of things.
- 0:54SPEAKER_01: And I just thought we should, like,
- 0:56SPEAKER_01: talk about all the different data that you've been discovering because it's kind
- 1:00SPEAKER_01: of cool and it's interesting.
- 1:04SPEAKER_01: So what's going on with admissions?
- 1:07SPEAKER_01: So you have the Columbia data, Columbia University.
- 1:10SPEAKER_02: Yeah, thanks for, you know, our great friend, Crem.
- 1:14SPEAKER_01: Yeah, yeah, Crem U. Yeah, yeah,
- 1:16SPEAKER_01: Crem So, yeah,
- 1:19SPEAKER_01: tell us, this is a Manifest, Manifest Manifold Markets Manifest Conference 2026.
- 1:24SPEAKER_01: I gave a presentation there, and you were also there, and you talked about admissions.
- 1:30SPEAKER_01: Tell me what we know about admissions.
- 1:32SPEAKER_01: What did you find out from the Columbia data that's interesting?
- 1:34SPEAKER_02: Yeah, so, and thanks so much for having me on, Razib.
- 1:39SPEAKER_02: Really, really, it's great to be here.
- 1:41SPEAKER_02: Love the show.
- 1:43SPEAKER_02: Um, you know,
- 1:44SPEAKER_02: I think,
- 1:44SPEAKER_02: I think admissions,
- 1:45SPEAKER_02: like it's,
- 1:46SPEAKER_02: it's a kind of social science adjacent topic that a lot of people have interacted
- 1:50SPEAKER_02: with, have strong opinions about.
- 1:52SPEAKER_02: Um, I think the Columbia data is kind of,
- 1:56SPEAKER_02: you know, novel in a couple of ways that like, first of all, it's not anonymized.
- 2:00SPEAKER_01: Like, you know, we, we, you could do the cool things with surname.
- 2:04SPEAKER_02: You could do cool things with, uh, linking to Wikipedia.
- 2:08SPEAKER_02: Um,
- 2:09SPEAKER_02: And I think using this has
- 2:15SPEAKER_02: gotten me a much better understanding of admissions than
- 2:20SPEAKER_02: the normal Twitter discourse.
- 2:21SPEAKER_02: Because this is something that has a lot of discourse and there are a lot of myths.
- 2:26SPEAKER_01: Well, I mean, with admissions, people just say whatever.
- 2:28SPEAKER_01: They kind of have some big stylized facts, whatever.
- 2:31SPEAKER_01: You have granular data.
- 2:32SPEAKER_01: You have really granular data.
- 2:35SPEAKER_01: So talk about what you found out that surprised people.
- 2:39SPEAKER_01: Yeah.
- 2:39SPEAKER_01: So I think,
- 2:40SPEAKER_02: I think,
- 2:40SPEAKER_02: so the broad strokes, people,
- 2:43SPEAKER_02: people are certainly right that,
- 2:46SPEAKER_02: you know,
- 2:46SPEAKER_02: elite colleges have become much,
- 2:47SPEAKER_02: much more selective that,
- 2:49SPEAKER_02: and that this is like, this is the kind of the, like the central trend.
- 2:55SPEAKER_02: What's, what is,
- 2:57SPEAKER_02: like that being said,
- 2:58SPEAKER_02: like that, there,
- 2:59SPEAKER_02: there are a couple of questions that I think,
- 3:01SPEAKER_02: you know,
- 3:02SPEAKER_02: follow up questions to that is like, In what respect has it become more selective?
- 3:07SPEAKER_02: So obviously, like the average SAT is much higher.
- 3:10SPEAKER_02: The SAT has also gotten much easier.
- 3:12SPEAKER_02: You know, everyone agrees.
- 3:14SPEAKER_02: And you could kind of see that in this data that the SAT has gotten much worse
- 3:21SPEAKER_02: at predicting how you'll do in any class.
- 3:24SPEAKER_02: Now, part of this is also that GPA has also been,
- 3:27SPEAKER_02: you You need to tell people what Goodhart's Law is.
- 3:39SPEAKER_02: Yeah, you know,
- 3:40SPEAKER_02: this is the rationalist famous Goodhart's law that when a measure becomes a target,
- 3:47SPEAKER_02: it ceases to be a good measure because people just focus on,
- 3:52SPEAKER_02: you know, devaluing the metric instead of, and it's not as good as measuring.
- 3:57SPEAKER_01: Well, I mean, I think another way you can think about it is like a metric is measuring
- 4:01SPEAKER_01: an underlying variable.
- 4:04SPEAKER_01: Yes.
- 4:04SPEAKER_01: Right?
- 4:05SPEAKER_01: But all you see is the metric.
- 4:09SPEAKER_01: And the correlation, let's say the correlation is 0.8 between the variable and the metric.
- 4:15SPEAKER_01: Well, people are not getting rewarded for the variable.
- 4:21SPEAKER_02: Yes.
- 4:22SPEAKER_01: They're getting rewarded for the indicator.
- 4:24SPEAKER_01: And so they get better and better at gaming the indicator.
- 4:27SPEAKER_01: You know, here's an example, I think concrete example for people.
- 4:30SPEAKER_01: You know, there used to be something called stack ranking.
- 4:34SPEAKER_01: I mean, some people still do it ad hoc.
- 4:36SPEAKER_01: You know, places like Microsoft stack ranking.
- 4:38SPEAKER_01: And Oracle famously did it.
- 4:39SPEAKER_01: You would fire the bottom 5% to 10% of engineers based on their metrics.
- 4:49SPEAKER_01: Sometimes the metrics are really stupid, like how many lines of code you produce.
- 4:54SPEAKER_01: Okay?
- 4:55SPEAKER_01: I don't think this was like the Microsoft issue.
- 4:57SPEAKER_01: I think that was more like, you know, evaluations by your report.
- 5:02SPEAKER_01: But, like, let's say, and I think as a stylized factor, as like a thought experiment, it works.
- 5:06SPEAKER_01: Let's say that the number of lines of code you produce is the metric.
- 5:11SPEAKER_01: At the beginning,
- 5:12SPEAKER_01: when you're measuring that, that is having a rough correlation to productivity.
- 5:16SPEAKER_01: But once you tell people that they're being judged by the number of lines
- 5:22SPEAKER_01: of code they produce, Everyone's going to start producing massive amounts of code.
- 5:27SPEAKER_01: And the measure is just going to be totally useless, you know?
- 5:31SPEAKER_01: And so things get good-hearted,
- 5:34SPEAKER_01: their utility disappears because people know exactly what you're measuring
- 5:37SPEAKER_01: and they know how to game to the measure, right?
- 5:41SPEAKER_01: Yeah.
- 5:43SPEAKER_02: And a very universally applicable concept.
- 5:45SPEAKER_02: I think it's like, you know, a kind of social science version of a Lucas critique even.
- 5:49SPEAKER_02: You know, it's beautiful.
- 5:50SPEAKER_02: You could apply it to A lot of areas of life.
- 5:54SPEAKER_02: And, you know, in college admissions, this is, you know, kind of the central.
- 5:59SPEAKER_01: When you say Lucas critique,
- 6:00SPEAKER_01: a lot of people know that that's an economist or actually very few people don't talk
- 6:03SPEAKER_01: about what the Lucas critique is.
- 6:07SPEAKER_02: You know,
- 6:09SPEAKER_02: Lucas critique in macroeconomics that,
- 6:12SPEAKER_02: you know,
- 6:12SPEAKER_02: when you look at past economic data and try to judge policies based
- 6:18SPEAKER_02: on past economic data,
- 6:20SPEAKER_02: you know,
- 6:20SPEAKER_02: this is actually fraught because people change,
- 6:24SPEAKER_02: you know, based on what the policy is.
- 6:26SPEAKER_02: People change how they act to match the policy.
- 6:30SPEAKER_02: So this makes old data and old math models wrong for, you know, the kind of planning you want.
- 6:34SPEAKER_02: Yeah.
- 6:35SPEAKER_01: I mean, you know,
- 6:36SPEAKER_01: the problem is social science,
- 6:37SPEAKER_01: unlike physical science,
- 6:38SPEAKER_01: biological science is kind of in the middle, I would say, for various reasons.
- 6:41SPEAKER_01: The problem with social science,
- 6:42SPEAKER_01: unlike physical science, is,
- 6:44SPEAKER_01: you know,
- 6:46SPEAKER_01: The laws of the past are not always the laws of the future because social organisms,
- 6:53SPEAKER_01: individuals, human beings in particular, dynamically respond.
- 6:57SPEAKER_01: And this, again, is Goodhart's law, is one of the examples of this.
- 7:01SPEAKER_01: And so the truisms of the past may not be the truisms of the future.
- 7:05SPEAKER_01: So before 1900, the truism is higher socioeconomic status.
- 7:09SPEAKER_01: People have more offspring.
- 7:11SPEAKER_01: That's just the truism.
- 7:13SPEAKER_01: After the demographic transition,
- 7:15SPEAKER_01: that shifted and the correlation was roughly higher SES is lower fertility,
- 7:19SPEAKER_01: except at the very high levels.
- 7:22SPEAKER_01: It's a very high level.
- 7:22SPEAKER_01: So it's kind of U-shaped,
- 7:24SPEAKER_01: but since only a small portion of the population,
- 7:27SPEAKER_01: by definition,
- 7:28SPEAKER_01: is in the top couple of percent,
- 7:31SPEAKER_01: that bump at the top couple of percent
- 7:33SPEAKER_01: of very high net wealth individuals didn't really matter too much
- 7:36SPEAKER_01: the overall correlation,
- 7:38SPEAKER_01: which was really driven by middle
- 7:39SPEAKER_01: and upper middle class people really shrinking their family sizes vis-a-vis compared
- 7:44SPEAKER_01: to working class
- 7:46SPEAKER_02: and the poor.
- 7:46SPEAKER_01: Anyway, just saying that these correlations shift all over the time, all the time.
- 7:52SPEAKER_01: And I've seen that even in my own life.
- 7:55SPEAKER_02: Yeah.
- 7:56SPEAKER_02: Um, and, and so, and,
- 7:58SPEAKER_02: and everyone kind of realizes like,
- 7:59SPEAKER_02: you know, at the,
- 8:01SPEAKER_02: you know,
- 8:01SPEAKER_02: a 1500 SAT,
- 8:02SPEAKER_02: you know, 15 years ago was damn, that's, you know, that's, that's incredible.
- 8:06SPEAKER_02: That will,
- 8:06SPEAKER_02: you know,
- 8:07SPEAKER_02: does, that certainly doesn't guarantee you getting into a good school anymore.
- 8:11SPEAKER_02: Even at the,
- 8:11SPEAKER_02: you know, at the,
- 8:12SPEAKER_02: at the highest,
- 8:13SPEAKER_02: you know, SAT,
- 8:14SPEAKER_02: you know, level, if you,
- 8:15SPEAKER_02: if you have a perfect SAT,
- 8:16SPEAKER_02: only one in five of those kids who apply to Columbia get in, um, And,
- 8:22SPEAKER_02: you know,
- 8:22SPEAKER_02: and unfortunately,
- 8:23SPEAKER_02: you know,
- 8:24SPEAKER_02: this medium is not very favorable to graph posting in a certain way,
- 8:28SPEAKER_02: the podcast medium.
- 8:29SPEAKER_02: But, you know, we could describe some of these graphs to you, you know.
- 8:33SPEAKER_02: In 1996, a whiter Asian kid who got a 1550 had a 72% chance of getting into Columbia.
- 8:38SPEAKER_02: Now it's 15%.
- 8:39SPEAKER_02: Yeah.
- 8:45SPEAKER_02: And the number of Columbia applicants.
- 8:48SPEAKER_01: Okay, two issues.
- 8:50SPEAKER_01: So recentering and whatever, like 1994, spring recentering changed things.
- 8:55SPEAKER_01: You know, compressed at the top.
- 8:57SPEAKER_01: Okay, so basically,
- 8:58SPEAKER_01: you know,
- 8:58SPEAKER_01: back in my day,
- 9:00SPEAKER_01: because I am of a certain age,
- 9:02SPEAKER_01: when I took the SAT, I was right at the tail end right before recentering.
- 9:06SPEAKER_01: And the average score was in the low 900s.
- 9:08SPEAKER_01: They recentered it.
- 9:09SPEAKER_01: The average score went to 1,000.
- 9:11SPEAKER_01: That really, really inflated math in particular, I think, for some reason.
- 9:15SPEAKER_01: And so math is compressed at the top, so you can't differentiate people.
- 9:19SPEAKER_01: So that happened.
- 9:20SPEAKER_01: The other thing that I do have to ask when it comes to acceptance rates,
- 9:24SPEAKER_01: common applications and just like the fact that like way more people are applying,
- 9:29SPEAKER_01: that is reducing the
- 9:30SPEAKER_02: acceptance rates, right?
- 9:32SPEAKER_02: Absolutely.
- 9:33SPEAKER_02: Now, unfortunately,
- 9:34SPEAKER_02: like within the data that I have,
- 9:36SPEAKER_02: which is –
- 9:38SPEAKER_01: Just real quick,
- 9:38SPEAKER_01: I just want to bring that up because when you say
- 9:40SPEAKER_01: the acceptance rates are dropping – That doesn't mean fewer people are going
- 9:44SPEAKER_01: to college, like more people are going to college.
- 9:46SPEAKER_01: It just means there's more and more applications and there's all these kids
- 9:50SPEAKER_01: that apply to every single Ivy League school and it may be too late.
- 9:55SPEAKER_01: And so obviously you're going to get like just decreased acceptance rates.
- 10:00SPEAKER_01: But it doesn't mean that the kids not accepted at Columbia are not going to get
- 10:03SPEAKER_01: into Penn or Cornell or Brown or even.
- 10:06SPEAKER_01: Yes.
- 10:07SPEAKER_01: I know a woman.
- 10:09SPEAKER_01: This is a long time ago, so I can say it now.
- 10:11SPEAKER_01: She applied to all the IVs at Tulane and Stanford,
- 10:17SPEAKER_01: and she didn't get in anywhere, including Tulane and Duke.
- 10:21SPEAKER_01: She didn't get into anywhere except Stanford.
- 10:23SPEAKER_01: Which just shows the randomness because arguably Stanford is better than Tulane or Duke.
- 10:31SPEAKER_02: Arguably, yeah.
- 10:32SPEAKER_01: Almost certainly it is, right?
- 10:34SPEAKER_01: So I'm just saying that there's a stochastic aspect.
- 10:39SPEAKER_02: I've heard rumors that someone we might know has access to a lot more data from
- 10:46SPEAKER_02: a lot more colleges.
- 10:47SPEAKER_02: I would love to see Stanford's data because you hear a lot of these stories
- 10:53SPEAKER_02: about Stanford being,
- 10:54SPEAKER_02: I think, rather more stochastic than most in who it chooses to admit.
- 10:59SPEAKER_02: So I'd be very interested in seeing if that's actually the case because I I,
- 11:04SPEAKER_02: you know,
- 11:04SPEAKER_02: I think,
- 11:05SPEAKER_02: I think,
- 11:05SPEAKER_02: I think that's probably,
- 11:06SPEAKER_02: that's probably,
- 11:07SPEAKER_01: you know,
- 11:07SPEAKER_01: so I've been interrupting you partly because you,
- 11:09SPEAKER_01: you drop in like these terms that I know,
- 11:12SPEAKER_01: but I don't think the listeners know,
- 11:13SPEAKER_01: but like talk about your manifest talk and what you, what you discovered.
- 11:16SPEAKER_01: Okay.
- 11:16SPEAKER_01: So go back to it.
- 11:18SPEAKER_02: Yeah.
- 11:18SPEAKER_02: And, and, you know, I'll actually, um, you know, they, they, I'll actually pull it up right now.
- 11:23SPEAKER_02: Um, so, and, and so everyone kind of, You know, college has gotten more difficult to get into.
- 11:30SPEAKER_02: And there's also this, you know, ambient cultural knowledge.
- 11:33SPEAKER_02: It's increasingly salient how much more difficult,
- 11:35SPEAKER_02: you know,
- 11:36SPEAKER_02: time intensive and grindy upper middle class American childhood has become in
- 11:40SPEAKER_02: the 21st century.
- 11:41SPEAKER_02: You know, there's a lot of discourse about this.
- 11:43SPEAKER_02: David Brooks wrote a good article in The Atlantic about it a year ago.
- 11:46SPEAKER_02: You know, and,
- 11:48SPEAKER_02: you know,
- 11:49SPEAKER_02: I think emblematic of this,
- 11:50SPEAKER_02: you know,
- 11:51SPEAKER_02: there was a time in the 90s,
- 11:52SPEAKER_02: you know, kids would go to the mall and They would hang out with their friends.
- 11:56SPEAKER_02: Now, okay,
- 11:57SPEAKER_02: you know,
- 11:57SPEAKER_02: they're on the screens,
- 11:58SPEAKER_02: but also, you know, their parents, you have to drive them to trombone practice.
- 12:02SPEAKER_02: You have to, you know, drive them to Chinese lessons.
- 12:05SPEAKER_02: There are all these,
- 12:06SPEAKER_02: you know,
- 12:07SPEAKER_02: extra obligations that fall, you know, and the parents bear a lot of these costs.
- 12:13SPEAKER_02: And everyone knows this is basically because of college.
- 12:16SPEAKER_02: Now,
- 12:20SPEAKER_02: I think one reading of these facts is this could be basically good.
- 12:24SPEAKER_02: Maybe it's better.
- 12:25SPEAKER_02: Maybe in order to beat China, we have to have all the kids working harder.
- 12:30SPEAKER_02: They have to be studying for their SATs instead of going to the mall.
- 12:33SPEAKER_02: I think Vivek Ramaswamy, I'm a big fan.
- 12:36SPEAKER_02: I'm a big fan, but when he missteps, we got to correct him.
- 12:41SPEAKER_02: He had this viral thread that the culture that valorizes sleepovers is not going to win.
- 12:48SPEAKER_02: It's China where they valorize the engineers and study for their SATs.
- 12:54SPEAKER_02: This, I think, is not the right reading of the data.
- 13:00SPEAKER_02: What's going on?
- 13:00SPEAKER_02: If you have this Goodhart's Law view of the SAT, Why have scores gone up so much?
- 13:07SPEAKER_02: Well, one, you know, they made everything easier.
- 13:11SPEAKER_02: What's more,
- 13:11SPEAKER_02: you know,
- 13:12SPEAKER_02: the Democrat Asians are just doing dynamite on the SAT
- 13:15SPEAKER_02: and there are significantly more of them than there used to be in this country.
- 13:19SPEAKER_02: You know,
- 13:19SPEAKER_02: the population has been a tremendous amount in the last couple of decades and that
- 13:23SPEAKER_02: in a lot of ways was a great thing.
- 13:28SPEAKER_02: Also,
- 13:29SPEAKER_02: kids are spending a lot more time prepping on the SAT and there's social science
- 13:32SPEAKER_02: about this, you know, and especially among Asian Americans.
- 13:36SPEAKER_02: All right, all right, okay.
- 13:38SPEAKER_01: So, but it used to be that the prepping of the SAT only got you, like, 40 or 50 points.
- 13:42SPEAKER_02: Yes.
- 13:43SPEAKER_02: And, you know,
- 13:43SPEAKER_02: and people do, like,
- 13:44SPEAKER_02: you know, toss out studies that, like, it's not that effective to prep for the SAT.
- 13:50SPEAKER_02: You know, I think, first of all, the SAT has gotten much more preppable.
- 13:56SPEAKER_02: You know, it's a lot less G-loaded than it used to be.
- 13:58SPEAKER_02: And you can see this, like, you know, it doesn't predict as well how you're going to
- 14:02SPEAKER_01: do it.
- 14:03SPEAKER_01: Tell the listener what G is.
- 14:05SPEAKER_02: It's not as correlated with the underlying latent true intelligence
- 14:09SPEAKER_02: that we're trying to measure,
- 14:10SPEAKER_02: which the IQ test is the best thing at measuring G.
- 14:14SPEAKER_02: That's why we use the IQ test in social science.
- 14:17SPEAKER_02: But the SAT used to be pretty good at measuring G.
- 14:20SPEAKER_02: In fact, the original SAT was near the upper limit.
- 14:24SPEAKER_02: It correlated with one SAT test you took correlated with the next SAT test you take
- 14:32SPEAKER_02: to the same extent that it would correlate with an IQ test.
- 14:35SPEAKER_02: This is not really the case anymore.
- 14:37SPEAKER_02: For multiple reasons, this isn't the case anymore.
- 14:48SPEAKER_02: Now, and I think,
- 14:52SPEAKER_02: you know,
- 14:52SPEAKER_02: so if you have this Goodhart's Law view of the SAT,
- 14:55SPEAKER_02: it's not actually good if people are prepping for the SAT.
- 14:59SPEAKER_01: It's actually deadweight loss.
- 15:01SPEAKER_01: You know, they're engaging
- 15:04SPEAKER_02: in this zero sum,
- 15:06SPEAKER_01: negative sum kind of.