Latest / Azadeh and Guy on AI / AI Slop and the Collapse of Peer Review

Transcript

AI Slop and the Collapse of Peer Review · Azadeh and Guy on AI

• 0:01
  SPEAKER_00: Hello, everyone.
• 0:02
  SPEAKER_00: We are recording episode three of season two.
• 0:06
  We are still excited that we are recording season two of Azadeh
• 0:12
  and Guy on AI.
• 0:14
  And today we're talking about the topic that we've touched upon
• 0:18
  a number of times
• 0:20
  SPEAKER_03: and has a great title.
• 0:21
  SPEAKER_03: So go with the title.
• 0:23
  SPEAKER_00: Jointification of research.
• 0:25
  SPEAKER_03: Yes.
• 0:26
  SPEAKER_03: And just to be clear,
• 0:28
  SPEAKER_03: the jointification of research is a paper,
• 0:30
  SPEAKER_03: we'll talk about the paper in a second,
• 0:33
  SPEAKER_03: that describes the phenomena and AI is not the reason for the phenomena.
• 0:38
  SPEAKER_03: So we're not saying they're the same.
• 0:40
  SPEAKER_03: What we are saying is AI is one potential.
• 0:44
  SPEAKER_03: AI slop is one potential of accelerating this phenomena.
• 0:49
  SPEAKER_03: So how about you talk about the phenomena?
• 0:51
  So,
• 0:53
  first in the paper,
• 0:54
  they offer a kind of definition of junkification.
• 0:58
  SPEAKER_00: And it happens when platforms scale rapidly,
• 1:03
  SPEAKER_00: they prioritize profit,
• 1:05
  and they slowly degrade user experience by flooding the system
• 1:09
  with low-quality
• 1:11
  SPEAKER_00: content.
• 1:12
  SPEAKER_03: So you can immediately see where it's coming from.
• 1:16
  But they particularly junkification came from thinking about
• 1:21
  Amazon and how Amazon
• 1:23
  SPEAKER_03: onboarded a lot of small providers that were of varying qualities.
• 1:28
  SPEAKER_03: And it became really, really hard to find good products.
• 1:31
  You could find cheap products,
• 1:33
  but you were never sure what you were getting.
• 1:35
  So that's the idea that when the marginal cost of adding things
• 1:40
  to the platform
• 1:41
  SPEAKER_03: goes too close to zero, it encourages junkification.
• 1:47
  SPEAKER_00: Junkification, yes, very true.
• 1:50
  SPEAKER_00: And the same thing is happening, unfortunately, in academic world.
• 1:55
  SPEAKER_00: So nowadays we are seeing a lot of papers,
• 2:01
  SPEAKER_00: I would say,
• 2:01
  SPEAKER_00: with good enough quality that are not that much innovative.
• 2:07
  And we're seeing it actually on both ends of the spectrum,
• 2:11
  I would argue.
• 2:12
  SPEAKER_03: We're seeing it in papers that are like, they're good enough.
• 2:15
  SPEAKER_03: Sometimes they're just not good.
• 2:18
  SPEAKER_03: There's that too.
• 2:20
  And or do not add a lot of new or even talking with the
• 2:25
  literature in a way that
• 2:26
  SPEAKER_03: helps them add to the existing literature.
• 2:29
  SPEAKER_03: Mm-hmm.
• 2:30
  SPEAKER_03: And on the other hand,
• 2:31
  SPEAKER_03: what we're seeing,
• 2:32
  SPEAKER_03: and we have talked about this a little bit,
• 2:34
  is a growth in the number of journals and the numbers of papers
• 2:38
  they're publishing.
• 2:38
  SPEAKER_03: Because,
• 2:39
  SPEAKER_03: again,
• 2:40
  SPEAKER_03: with digital,
• 2:41
  SPEAKER_03: the marginal cost,
• 2:42
  SPEAKER_03: if you're publishing 500 journals,
• 2:44
  SPEAKER_03: adding the 501st journal adds very little cost to the publisher.
• 2:50
  SPEAKER_03: And the publisher is in the business of making money.
• 2:52
  SPEAKER_03: This is a weird business where...
• 2:55
  The academics provide the labor,
• 2:57
  the two kinds of labor that are most important.
• 2:59
  SPEAKER_03: They provide the labor for writing the damn thing,
• 3:02
  SPEAKER_03: collecting the data,
• 3:03
  SPEAKER_03: writing it,
• 3:03
  SPEAKER_03: doing all of this,
• 3:05
  SPEAKER_03: going back and forth with their reviewers,
• 3:07
  SPEAKER_03: but also as their reviewers.
• 3:09
  SPEAKER_03: And we do all of that for free.
• 3:11
  SPEAKER_00: Exactly.
• 3:11
  SPEAKER_00: They do voluntarily.
• 3:13
  SPEAKER_00: So, yeah, it's free.
• 3:16
  SPEAKER_03: And as a result, what you're seeing is...
• 3:20
  SPEAKER_03: As long as they have enough people to keep writing,
• 3:23
  they can keep publishing and making money either on the front
• 3:26
  end by asking the
• 3:28
  SPEAKER_03: authors to pay and then it's open access.
• 3:31
  SPEAKER_03: Mm-hmm.
• 3:34
  SPEAKER_03: Which I love.
• 3:35
  SPEAKER_03: I love people having access to my papers.
• 3:37
  SPEAKER_03: I want everything.
• 3:38
  SPEAKER_00: You need to pay a lot.
• 3:39
  SPEAKER_03: Yeah.
• 3:39
  SPEAKER_03: I mean, it depends.
• 3:40
  SPEAKER_03: Sometimes you pay a few hundred.
• 3:42
  SPEAKER_03: Sometimes it's thousands.
• 3:43
  SPEAKER_03: Thousands, yeah.
• 3:44
  SPEAKER_03: And I've seen publications that actually asked for $10,000 or more.
• 3:51
  SPEAKER_03: And that is, those are incredible sums.
• 3:53
  SPEAKER_03: You cannot do that if you're publishing five, six articles a year.
• 3:56
  SPEAKER_03: I mean, the numbers just go up exponentially.
• 3:57
  SPEAKER_00: But even if it's $500 or $600 or $700.
• 3:58
  SPEAKER_03: That's still a lot, yeah.
• 3:59
  SPEAKER_03: Yeah.
• 4:03
  SPEAKER_03: That's a lot of money.
• 4:04
  SPEAKER_03: You can load it.
• 4:05
  And that's the assumption in some of these journals that you
• 4:08
  load it on your grant
• 4:09
  SPEAKER_03: or something else.
• 4:09
  SPEAKER_03: But if you don't have one, you're stuck.
• 4:12
  SPEAKER_03: So instead of equity,
• 4:13
  SPEAKER_03: because there's equity for the consumer because it's open access,
• 4:16
  SPEAKER_03: but for the producer,
• 4:18
  SPEAKER_03: it's actually less equity.
• 4:19
  yeah and so so that's part of the challenge but they might be
• 4:24
  making money on the
• 4:24
  front end and often they're making money on their back end and
• 4:27
  that is they're
• 4:28
  selling it to libraries so if they're not charging you
• 4:31
  somebody's got to pay
• 4:31
  somewhere for this thing and so if they're not charging you
• 4:35
  they are charging at
• 4:36
  the back end they're charging the university or the libraries
• 4:40
  or whoever is
• 4:41
  collecting these and want to have this on their virtual shelves
• 4:45
  because it's more
• 4:46
  SPEAKER_03: and more virtual
• 4:49
  So the authors,
• 4:51
  and today we are talking about a paper that comes from two
• 4:56
  Australian scholars.
• 4:58
  SPEAKER_00: And it's been published in 2025.
• 4:59
  SPEAKER_00: So it's a very recent paper.
• 5:03
  SPEAKER_00: And they talk about three overlapping forces.
• 5:08
  SPEAKER_00: They talk about publish or perish.
• 5:11
  SPEAKER_00: which is very kind of prevalent in academia.
• 5:15
  SPEAKER_00: They talk about platform capitalism meets academia,
• 5:19
  SPEAKER_00: and they also talk about pay-to-publish explosion.
• 5:24
  SPEAKER_03: Sorry about that.
• 5:26
  SPEAKER_03: My cold is not exactly gone, but it is what it is.
• 5:30
  So they're noticing this kind of what sometimes people refer to
• 5:35
  late-stage
• 5:35
  SPEAKER_03: capitalism entering all the arenas.
• 5:39
  SPEAKER_02: Mm-hmm.
• 5:39
  SPEAKER_03: And it starts with the universities and the way universities are ranked,
• 5:43
  SPEAKER_03: which is linked to publishing statistics,
• 5:47
  SPEAKER_03: citation statistics,
• 5:49
  SPEAKER_03: and all of these other things that co-occur.
• 5:52
  So there's pressure on the institutions who transfer that
• 5:55
  pressure because they are precious.
• 5:57
  SPEAKER_03: They transfer the pressure to faculty in some places.
• 6:00
  SPEAKER_02: Mm hmm.
• 6:01
  not in our institution,
• 6:03
  but in some places,
• 6:04
  you actually get bonuses if you get published.
• 6:06
  SPEAKER_03: You get extra money.
• 6:08
  That obviously puts extra pressure on you because you want that
• 6:10
  money because...
• 6:13
  SPEAKER_03: We tend to count our chickens.
• 6:14
  SPEAKER_03: Right.
• 6:15
  SPEAKER_03: It's like if I can make an extra thousand bucks, I assume I can make it.
• 6:19
  And that puts pressure on me to go and publish at the right
• 6:22
  place at the right time.
• 6:23
  SPEAKER_03: So so there are indefinitely just to publish or perish.
• 6:27
  SPEAKER_03: You want to keep your job in academia.
• 6:28
  SPEAKER_03: You've got to keep producing.
• 6:29
  SPEAKER_03: And that's when it becomes enticing.
• 6:33
  SPEAKER_03: And I do want to separate because we're talking about junkification.
• 6:37
  I want to say not all open access or all pay for play journals
• 6:43
  are bad and they're not all junk.
• 6:47
  And that we have to be careful because some of it is that the
• 6:50
  number of researchers
• 6:53
  SPEAKER_03: has grown around the world.
• 6:54
  SPEAKER_03: There are more academics.
• 6:56
  SPEAKER_03: There are more people exploring.
• 6:58
  SPEAKER_03: There are more people producing research results.
• 7:00
  So some of it is just that the pool of people producing
• 7:05
  knowledge has grown,
• 7:07
  SPEAKER_03: but not quite as much as the number of papers have grown.
• 7:10
  SPEAKER_03: And I've talked about this.
• 7:12
  Since I've been in the field in the last 25 years,
• 7:15
  the number of papers have grown tenfold.
• 7:18
  SPEAKER_03: That is a lot of damn papers.
• 7:20
  SPEAKER_00: Yeah, yeah.
• 7:24
  SPEAKER_00: And they also said that...
• 7:26
  SPEAKER_00: All of those things that we talked about,
• 7:28
  SPEAKER_00: they create a burden and the burden is not shared equally.
• 7:31
  So the system disproportionately harms early career
• 7:35
  researchers,
• 7:37
  which we know about.
• 7:38
  Many assistant professors are pushed to publish a lot,
• 7:41
  many graduate students.
• 7:43
  SPEAKER_00: Especially if you are an international graduate student,
• 7:46
  SPEAKER_00: you feel the pressure to publish a lot to be kind of known in academia.
• 7:50
  SPEAKER_00: Scholars in the Global South,
• 7:53
  SPEAKER_00: non-native English speakers,
• 7:55
  SPEAKER_00: and also researchers outside elite institutions.
• 7:59
  SPEAKER_03: And this is where AI enters the game, right?
• 8:02
  SPEAKER_03: Because AI can accelerate your productivity, and it does.
• 8:06
  SPEAKER_03: Some of it, as we've talked many times, in a good way.
• 8:09
  SPEAKER_03: We talked to Inusa last time.
• 8:11
  SPEAKER_03: This is good.
• 8:11
  This is not all negative,
• 8:13
  but we have to be careful because it does allow lots of
• 8:18
  production.
• 8:20
  And now,
• 8:21
  for the first time,
• 8:23
  the authors can get to minimal marginal cost.
• 8:26
  SPEAKER_03: That is, if you're not committed to truth,
• 8:29
  SPEAKER_03: If you're not committed to actually generating data or ideas,
• 8:33
  SPEAKER_03: you can have AI produce a lot of texts very,
• 8:37
  SPEAKER_03: very quickly.
• 8:38
  SPEAKER_01: Yeah.
• 8:38
  SPEAKER_03: And we are not especially good in academia at catching people.
• 8:43
  SPEAKER_03: The system eventually works on two things, on trust and on peer review.
• 8:47
  SPEAKER_02: Mm-hmm.
• 8:48
  SPEAKER_02: Mm-hmm.
• 8:49
  The problem is that some people don't operate with trust and
• 8:53
  that creates a break.
• 8:54
  And the second thing is that the peer review process is at the
• 8:58
  verge of breaking
• 9:00
  SPEAKER_03: because the number of papers has grown so much.
• 9:05
  And some of the journals are giving so very little time to read
• 9:09
  and give reviews
• 9:11
  SPEAKER_03: that the peer review may be at the point of collapse,
• 9:15
  SPEAKER_03: which will increase.
• 9:17
  SPEAKER_03: So if people are reading less or using less expert reviewers,
• 9:21
  SPEAKER_03: or I'm getting sometimes calls recently,
• 9:25
  SPEAKER_03: academia,
• 9:26
  SPEAKER_03: which is a platform,
• 9:28
  SPEAKER_03: has been asking me to review papers about mental health.
• 9:32
  SPEAKER_03: And I'm going, I know nothing about mental health.
• 9:35
  SPEAKER_03: I mean,
• 9:36
  SPEAKER_03: if you saw me,
• 9:37
  SPEAKER_03: you would say,
• 9:37
  SPEAKER_03: this is the last person to review papers about mental health,
• 9:41
  SPEAKER_03: unless you need somebody who's suffering.
• 9:44
  SPEAKER_03: And this is not,
• 9:46
  SPEAKER_03: this is not,
• 9:46
  there is nothing in my record that would say I would be
• 9:50
  appropriate for this.
• 9:52
  SPEAKER_03: And I'm reviewing a lot of papers.
• 9:54
  SPEAKER_03: Last year I reviewed 41 papers.
• 9:56
  SPEAKER_00: That's a lot of papers.
• 9:57
  SPEAKER_03: That is a lot of papers.
• 9:58
  SPEAKER_03: Many of them, you do a few cycles on them.
• 10:00
  SPEAKER_03: So I'm reviewing maybe 30 papers with some repeat offenders.
• 10:05
  SPEAKER_03: But it's a lot of reviews.
• 10:07
  SPEAKER_03: And I tend to do now about one a week.
• 10:10
  But I am,
• 10:11
  I have been doing this for a long time and I can do that fairly
• 10:13
  quickly.
• 10:14
  SPEAKER_03: I spend a few hours on an article and
• 10:17
  what i love about that is i'm learning and this is mostly all i
• 10:20
  read is ai so i'm
• 10:21
  getting really current on what people are doing in ai right now
• 10:25
  it's worth my time
• 10:27
  but it is very i mean the quantities are overwhelming and so
• 10:32
  can you can you make
• 10:33
  that work is a real question so this is the concern and we are
• 10:37
  afraid that ai will
• 10:38
  SPEAKER_03: just accelerate accelerate yes yes because the truth is
• 10:43
  SPEAKER_03: that you can ask AI to generate pretend data, and it will.
• 10:50
  SPEAKER_03: And it is very hard for reviewers to know from that point.
• 10:54
  SPEAKER_03: This is where,
• 10:55
  SPEAKER_03: and we've talked about this,
• 10:57
  open science frameworks are really critical and that journals
• 11:04
  need to start
• 11:05
  SPEAKER_03: thinking about
• 11:07
  SPEAKER_03: Do we have to ask everybody to be registered?
• 11:10
  SPEAKER_03: And so we have a certain level of confidence that something has happened.
• 11:14
  SPEAKER_03: And it's not somebody saying to chat GPT or in my case to Claude,
• 11:18
  SPEAKER_01: hey,
• 11:19
  what would data for a study about reading comprehension look
• 11:25
  like generate tables
• 11:27
  SPEAKER_03: and effect sizes and something like that?
• 11:30
  SPEAKER_03: And I, you know, I've tried that a few times just to see what happens.
• 11:35
  SPEAKER_03: It's kind of scary.
• 11:36
  SPEAKER_00: Yeah, yeah.
• 11:37
  SPEAKER_00: And I was thinking to myself,
• 11:39
  SPEAKER_00: to be honest today,
• 11:40
  SPEAKER_00: when I went through the paper,
• 11:41
  I was thinking maybe eventually we need to come up with a
• 11:45
  different outlet for
• 11:47
  SPEAKER_00: academia other than publishing.
• 11:50
  SPEAKER_03: Or, yeah, somebody needs to take responsibility.
• 11:57
  SPEAKER_03: But part of it has to do with reducing any singular value
• 12:05
  Focus on one metric because there is that law of social science
• 12:10
  that if you have
• 12:12
  SPEAKER_03: one metric,
• 12:13
  SPEAKER_03: no matter what it is,
• 12:14
  SPEAKER_03: no matter how well it is measured,
• 12:16
  SPEAKER_03: the minute you have everything vested in one metric,
• 12:20
  SPEAKER_03: that metric will be corrupted.
• 12:22
  SPEAKER_02: Okay.
• 12:24
  Because there are real consequences for people's careers,
• 12:27
  for people's earnings.
• 12:29
  SPEAKER_03: The minute it's like that, it doesn't matter.
• 12:32
  SPEAKER_03: If you shift this,
• 12:33
  only things that were approved by universities or only no
• 12:37
  matter what you will do,
• 12:38
  SPEAKER_03: it eventually corrupts that metric.
• 12:40
  What you need are multiple metrics that reduce the chance that
• 12:44
  it'll get corrupted.
• 12:45
  SPEAKER_02: Okay.
• 12:46
  and again it's got something to do with consequences so um i
• 12:50
  think we're very close
• 12:51
  to the point where we will have to start making adjustment and
• 12:54
  the cope framework
• 12:56
  is another place we will have the link in the notes but the
• 12:59
  cope network is another
• 13:01
  place where we're trying so we've got open science we've got
• 13:03
  cope we've got
• 13:04
  different lists but it's also on institutions to make sure that
• 13:08
  they're not
• 13:09
  SPEAKER_03: measuring faculty through one metric alone if
• 13:12
  SPEAKER_03: Because that will create corruption.
• 13:14
  SPEAKER_03: If it's about,
• 13:15
  SPEAKER_03: we've talked about this outside this podcast,
• 13:20
  but we talked about the fact that there are people who are
• 13:23
  generating false
• 13:24
  SPEAKER_03: citations by asking their friends to cite them.
• 13:27
  And they have a network and they all cite each other and those
• 13:30
  kind of things.
• 13:31
  SPEAKER_03: Now, if it's relevant, please cite.
• 13:32
  SPEAKER_03: But you can create very quickly information.
• 13:35
  SPEAKER_03: an acceleration in number of citation that is unhealthy because,
• 13:39
  SPEAKER_03: again,
• 13:39
  SPEAKER_03: if that is the main metric,
• 13:41
  SPEAKER_03: then it's easy to create ways around it.
• 13:44
  SPEAKER_03: And it's very hard work to find those out.
• 13:46
  And I'd rather not spend the time on being detective,
• 13:50
  although that's fun sometimes.
• 13:51
  SPEAKER_01: Yeah.
• 13:52
  SPEAKER_03: And instead,
• 13:55
  making sure that we're judging people on multiple metrics so
• 13:58
  it's harder to game
• 13:59
  SPEAKER_03: them and it's more worth doing the actual work than gaming the work.
• 14:03
  SPEAKER_00: Yep.
• 14:03
  SPEAKER_00: Okay.
• 14:04
  SPEAKER_00: Yep.
• 14:05
  SPEAKER_03: So junkification is a fun thing to say.
• 14:10
  SPEAKER_03: And sometimes it's hard for me.
• 14:12
  SPEAKER_03: So I'm just sorry about that.
• 14:14
  But so this is something to keep an eye on and keep thinking
• 14:19
  about as AI slop
• 14:21
  SPEAKER_03: becomes one of these things that are out there.
• 14:24
  SPEAKER_00: Yeah, very true.
• 14:29
  SPEAKER_03: And we'll see you next time on Azadeh.
• 14:32
  SPEAKER_00: And Guy.
• 14:33
  SPEAKER_03: On AI.
• 14:34
  SPEAKER_03: Bye.