Latest / What's Your Problem? / Did AI Write This?

Transcript

Did AI Write This? · What's Your Problem?

• 0:15
  Speaker 1: Pushkin. I'm Jacob Goldstein, and this is What's Your Problem?
• 0:33
  Speaker 1: My guest today is Max Spiro. He's the co-founder and
• 0:37
  Speaker 1: CEO of a company called Pangram. And Max's problem is this.
• 0:41
  Speaker 1: How do you build a tool that that reliably identifies
• 0:44
  Speaker 1: whether a piece of writing or an image was created
• 0:47
  Speaker 1: by AI. If you've heard about any of the recent
• 0:51
  Speaker 1: controversies over whether something was written by AI, whether it
• 0:56
  Speaker 1: was a prize-winning novel or an op-ed in a major newspaper,
• 1:02
  Speaker 1: there's a good chance that you've actually heard about Pangram,
• 1:05
  Speaker 1: whether you know it or not, because Pangram is the
• 1:08
  Speaker 1: tool people almost always cite in those cases. It is
• 1:12
  Speaker 1: generally viewed as the most reliable, widely available AI identification tool.
• 1:18
  Speaker 1: Max and I talked about a lot of different things
• 1:21
  Speaker 1: in our conversation. We talked about changing norms around when
• 1:25
  Speaker 1: it's OK to use AI and when it's not OK.
• 1:28
  Speaker 1: We talked about the risk of false positives, of PanGram
• 1:32
  Speaker 1: wrongly accusing people of using AI to write things. And
• 1:36
  Speaker 1: we talked about the new PanGram tool to detect whether
• 1:39
  Speaker 1: an image was AI generated. But to start, we talked
• 1:42
  Speaker 1: about two domains where PanGram is already in wide use,
• 1:46
  Speaker 1: education and publishing. Tell me about PanGram and sort of
• 1:51
  Speaker 1: where you are now. Who are your customers? Who's using it?
• 1:54
  Speaker 2: Our customers, a bunch of customers in education, higher ed,
• 1:58
  Speaker 2: and also like high schools, K-12, they want to know
• 2:02
  Speaker 2: whether their students are submitting AI-generated essays. They want a
• 2:05
  Speaker 2: way to teach writing or, like, have any writing-based assignments
• 2:13
  Speaker 2: without having it sort of just devolve into students using AI.
• 2:16
  Speaker 1: It seems like an extremely strong case. Like, you know,
• 2:20
  Speaker 1: writing is a way of thinking hard about things. I
• 2:23
  Speaker 1: buy that it is useful still for students to write.
• 2:27
  Speaker 1: My sense is that students are using AI a lot and,
• 2:32
  Speaker 1: you know, going to great lengths to sort of humanize
• 2:34
  Speaker 1: what the AI says, to sort of write through AI
• 2:37
  Speaker 1: or do what they have to do. Like, what is
• 2:38
  Speaker 1: your sense of how it is going?
• 2:40
  Speaker 2: Yeah, something like pangram here would be more of like,
• 2:43
  Speaker 2: we're adding friction to the process of cheating. And, you know,
• 2:47
  Speaker 2: it's going to make it less enticing if you have
• 2:49
  Speaker 2: to go start with AI bones and rewrite it.
• 2:52
  Speaker 1: Yes, you can't one-shot your assignment.
• 2:54
  Speaker 2: Exactly.
• 2:55
  Speaker 1: Yeah, yeah.
• 2:55
  Speaker 2: And that's helpful in some way. And, you know, you
• 2:59
  Speaker 2: can encourage students to, you know, start from scratch and
• 3:02
  Speaker 2: not use AI at all.
• 3:03
  Speaker 1: But I mean, certainly there are teachers who are requiring
• 3:06
  Speaker 1: students to do their writing in the room. Right? Like,
• 3:10
  Speaker 1: they're just going hard at, you have to sit here
• 3:13
  Speaker 1: in front of me. So what did somebody call it?
• 3:14
  Speaker 1: AI jail or something? You have to sit here in
• 3:16
  Speaker 1: front of me and write an essay.
• 3:18
  Speaker 2: It's kind of a bummer. Like, I feel like we
• 3:20
  Speaker 2: lose a lot here. We lose the in-class instruction. We
• 3:24
  Speaker 2: lose the ability to assign longer written pieces. It's true.
• 3:29
  Speaker 2: When I was in college, I wrote a lot of
• 3:31
  Speaker 2: like eight and 10, 12 page essays.
• 3:34
  Speaker 1: Yeah.
• 3:34
  Speaker 2: Like those, you can't do that in class. That takes
• 3:37
  Speaker 2: a lot of hours.
• 3:38
  Speaker 1: I mean, but that's just the nature of the world now, right? Like,
• 3:40
  Speaker 1: it sort of is what it is.
• 3:42
  Speaker 2: Yeah, and it's a little bit adversarial, as many of
• 3:46
  Speaker 2: our use cases are, where, like, the way that Pangram
• 3:49
  Speaker 2: is used is somebody is trying to get away with
• 3:52
  Speaker 2: AI writing in a way. And so there is this
• 3:55
  Speaker 2: adversarial component.
• 3:56
  Speaker 1: Yes. I mean, it's got to be mostly adversarial, right?
• 3:58
  Speaker 1: The weird non-adversarial one would be, I actually wrote this thing,
• 4:02
  Speaker 1: and I don't want people to think it's AI, so
• 4:04
  Speaker 1: I'm going to put it in Pangram, right? Like, that's
• 4:06
  Speaker 1: an edge case. Like, I've started using fewer em dashes, actually.
• 4:10
  Speaker 2: Oh, man. You got to use the em dashes.
• 4:13
  Speaker 1: Journalists love em dashes. And actually, maybe because I love them,
• 4:17
  Speaker 1: I didn't even notice it in AI pros. But once
• 4:21
  Speaker 1: everybody started talking about the em dash as a giveaway,
• 4:23
  Speaker 1: I just started doing, like, shorter independent clauses, basically.
• 4:27
  Speaker 2: Em dashes are, like, they're no longer an AI tell.
• 4:30
  Speaker 2: This was, like, a GPT-4 era thing. maybe like 4.0,
• 4:34
  Speaker 2: really overuse the em dash. We're over it now. Em
• 4:37
  Speaker 2: dashes are used a normal amount.
• 4:38
  Speaker 1: I can bring the em dash back? I can bring
• 4:40
  Speaker 1: it back?
• 4:40
  Speaker 2: Yes, you should bring it back.
• 4:42
  Speaker 1: Okay, great. So education is obviously a big use case.
• 4:47
  Speaker 2: I'll tell you about publishing as well.
• 4:49
  Speaker 1: Oh yeah, publishing. Well, so that one has been in
• 4:51
  Speaker 1: the news, right? There have been a couple of people,
• 4:53
  Speaker 1: one person lost a seven-figure book deal, right? Because, I
• 4:56
  Speaker 1: don't want to say because of you, but because they
• 4:59
  Speaker 1: were accused of using AI to write the book and
• 5:03
  Speaker 1: you were definitely involved in that.
• 5:04
  Speaker 2: Yeah, there have been a series of scandals this year
• 5:07
  Speaker 2: on AI use, either in books that are already published
• 5:10
  Speaker 2: or like big book deals. But kind of what it
• 5:14
  Speaker 2: comes down to is that a lot of people in
• 5:16
  Speaker 2: the industry are on the left side of the bell
• 5:19
  Speaker 2: curve where they're not heavy AI users. They're like, they're
• 5:23
  Speaker 2: book editors and publishers. They read like real books all day.
• 5:31
  Speaker 2: So they're not like, you know, talking to chat. So
• 5:34
  Speaker 2: I think like a lot of them, they like never
• 5:36
  Speaker 2: really built up this intuition in a way that like
• 5:39
  Speaker 2: all of a sudden, like AI getting good enough to
• 5:42
  Speaker 2: like produce publishable prose was like very surprising and kind
• 5:46
  Speaker 2: of caught them by surprise.
• 5:48
  Speaker 1: So recently, famous investor Stanley Druckenmiller wrote an op-ed in
• 5:53
  Speaker 1: the Wall Street Journal about the bond market. And everybody's like, wait,
• 5:57
  Speaker 1: AI wrote that. And then Druckenmiller was like, you're damn
• 6:00
  Speaker 1: right AI wrote that. I'm 80 years old and I
• 6:02
  Speaker 1: know how to use AI. And then maybe most interestingly,
• 6:06
  Speaker 1: the editor of the journal op-ed page was like, it's
• 6:09
  Speaker 1: fine that AI wrote it. Like Druckenmiller is a famous investor.
• 6:12
  Speaker 1: He really believes this. He stands behind it. Any famous
• 6:16
  Speaker 1: person writing an op-ed in the paper usually has somebody
• 6:19
  Speaker 1: else write it for him anyway. There's no problem here.
• 6:21
  Speaker 1: Sort of surprisingly, I might think I would have a
• 6:23
  Speaker 1: problem with it, but upon reflection, I don't. Do you
• 6:26
  Speaker 1: think it's bad?
• 6:27
  Speaker 2: I think the argument against is that it still does
• 6:30
  Speaker 2: a disservice to the readers. Like, if you read it, it's, like,
• 6:34
  Speaker 2: full of AI prose.
• 6:36
  Speaker 1: So your critique is as a piece of writing. It's
• 6:39
  Speaker 1: bad writing, and that is the criticism.
• 6:41
  Speaker 2: Yeah, and, like, the Wall Street Journal has great editors.
• 6:45
  Speaker 2: I'm sure, like, he could have just, like, written up
• 6:47
  Speaker 2: a draft himself that was, like, not particularly flowery or
• 6:51
  Speaker 2: whatever and then, you know, sent it to his editors and, like,
• 6:54
  Speaker 2: they help him work it into something that's, like, very readable.
• 6:58
  Speaker 1: You could also argue, like, it certainly would be less
• 7:02
  Speaker 1: objectionable if it just said at the bottom, this was
• 7:05
  Speaker 1: written with the assistance of AI, right? Like, that would
• 7:08
  Speaker 1: seem to, like, if you have any criticism, it's like, well,
• 7:11
  Speaker 1: you should tell me because it's weird. Yeah. Although you
• 7:14
  Speaker 1: could make the same argument for ghostwriters and, like, pretty
• 7:16
  Speaker 1: much anything you see written by a CEO, in most cases,
• 7:21
  Speaker 1: of a big company is going to be ghostwritten, right? Like,
• 7:23
  Speaker 1: there are people whose job is writing op-eds for CEOs.
• 7:26
  Speaker 2: I'm going to make this other argument now, which I
• 7:28
  Speaker 2: think is maybe more relevant. So I think there's like
• 7:31
  Speaker 2: two types of AI use and AI writing. There's basically
• 7:36
  Speaker 2: making something more concise or distilling. So for example, if
• 7:39
  Speaker 2: Druckenmiller's like, hey, I wrote like, you know, five pages,
• 7:42
  Speaker 2: here's all my research, here's all my thoughts. I need
• 7:45
  Speaker 2: AI to help me like make this concise and get
• 7:47
  Speaker 2: it into like op-ed size. And so I think typically
• 7:49
  Speaker 2: AI writing in this case ends up being quite helpful
• 7:53
  Speaker 2: because there's a lot to work with and it's distilling
• 7:56
  Speaker 2: it down to its core components. The other side of
• 7:59
  Speaker 2: AI writing that I see is expanding or padding, where
• 8:03
  Speaker 2: somebody's like, here's a few bullet points. Here's a few
• 8:07
  Speaker 2: things that I believe. Turn this into an email or
• 8:10
  Speaker 2: an op-ed. I think that's where it's really doing a
• 8:14
  Speaker 2: disservice to the readers because you're just putting more words
• 8:17
  Speaker 2: that are basically padding and don't mean that much.
• 8:19
  Speaker 1: Just publish the bullet points, bro.
• 8:21
  Speaker 2: Exactly, exactly.
• 8:22
  Speaker 1: Yeah, publish it in Axios. They love bullet points over there.
• 8:26
  Speaker 1: There's a broader thing that's interesting to me about the
• 8:30
  Speaker 1: Druckenmiller Wall Street Journal thing, and that is, like, it
• 8:33
  Speaker 1: does make me wonder to what extent we're right now
• 8:38
  Speaker 1: in a transitional moment where, you know, I could imagine
• 8:42
  Speaker 1: in a year, a few years.
• 8:45
  Speaker 1: In many settings, saying, did a person use AI to
• 8:48
  Speaker 1: write this, would be like today saying, did they use
• 8:50
  Speaker 1: the internet to do the research? Like, probably, and so
• 8:54
  Speaker 1: what are the answers? Does that seem plausible to you?
• 8:59
  Speaker 2: I think we're absolutely in this space or this time
• 9:03
  Speaker 2: period where the norms are completely up in the air
• 9:06
  Speaker 2: and people are so polarized on one side or the other,
• 9:09
  Speaker 2: or some people are like, I'm going to let AI
• 9:12
  Speaker 2: automate my entire online presence. And other people are swearing
• 9:15
  Speaker 2: they'll never use AI ever. And so we have this
• 9:20
  Speaker 2: really wide range of norms. Nobody's really settled on what
• 9:25
  Speaker 2: is acceptable versus unacceptable. I think AI and LLMs, they're
• 9:29
  Speaker 2: very powerful technology. They could do a lot of work
• 9:33
  Speaker 2: for me. They can go do research for me. They
• 9:36
  Speaker 2: can compile this into a spreadsheet for me to read
• 9:39
  Speaker 2: and peruse.
• 9:40
  Speaker 1: They can determine whether people are using AI to write.
• 9:44
  Speaker 2: Yeah. But yeah, I think the answer is we really
• 9:48
  Speaker 2: haven't settled on any norms societally. And the norms that
• 9:53
  Speaker 2: we will settle to are probably that some degree of
• 9:56
  Speaker 2: AI use is okay, but using AI to replace your
• 9:59
  Speaker 2: voice completely, especially in settings where it's going to be
• 10:04
  Speaker 2: read by thousands or millions, is probably less acceptable. Yeah.
• 10:08
  Speaker 1: Well, and disclosure. I do feel like disclosure can solve
• 10:11
  Speaker 1: a lot. It's also interesting to me. You see this
• 10:14
  Speaker 1: certainly on LinkedIn and on Twitter. There are a lot
• 10:18
  Speaker 1: of things that are so obviously AI. And initially, I
• 10:22
  Speaker 1: was like, don't people know? But now I think maybe
• 10:24
  Speaker 1: people know and don't care. Like, I don't know. It
• 10:27
  Speaker 1: does feel... I want to be clear, I don't like
• 10:30
  Speaker 1: AI prose. I'm not speaking of a moral objection, but
• 10:32
  Speaker 1: just as a matter of taste. And then there is
• 10:36
  Speaker 1: a little bit, like, it feels a little rude. It
• 10:38
  Speaker 1: feels a little spammy. Like, I'm reading this thing, and
• 10:41
  Speaker 1: this person didn't even take the time to lightly edit
• 10:44
  Speaker 1: it to get rid of the obvious AI-isms.
• 10:47
  Speaker 2: Yeah, or if you aren't going to take the time
• 10:50
  Speaker 2: to write it yourself, then I'm so much less inclined
• 10:53
  Speaker 2: to read it.
• 10:54
  Speaker 1: So the question I have is, do the people who
• 10:56
  Speaker 1: are posting that know that it is obviously AI and
• 10:59
  Speaker 1: they just don't care? I mean, I guess it varies,
• 11:01
  Speaker 1: but that's a weird one to me still.
• 11:03
  Speaker 2: I think there's a little bit of a bell curve.
• 11:05
  Speaker 2: There's a lot of people who are either new to
• 11:07
  Speaker 2: using AI or they haven't really used that much AI before,
• 11:11
  Speaker 2: and they can't really look at text and immediately grok
• 11:13
  Speaker 2: that it's AI-generated. They can't immediately tell. And so, yeah,
• 11:17
  Speaker 2: obviously you have the people who are just like, don't
• 11:19
  Speaker 2: use AI much. And then you also have the people
• 11:21
  Speaker 2: who like just discovered this technology and they're like, wow,
• 11:25
  Speaker 2: it like writes so well, this is amazing. And I
• 11:27
  Speaker 2: don't think they fully realized that like regular AI users
• 11:31
  Speaker 2: can read this and immediately smell that it's AI generated. Yeah. Then,
• 11:35
  Speaker 2: then you get to the middle of the bell curve
• 11:36
  Speaker 2: where it's like, yeah, like, like people who use AI
• 11:40
  Speaker 2: on a somewhat regular basis, they, they can kind of
• 11:42
  Speaker 2: just tell. And I think you get to the other
• 11:44
  Speaker 2: end where there's people who are like so deep in
• 11:47
  Speaker 2: AI psychosis that, that they also can't tell because like 90%
• 11:51
  Speaker 2: of what they're reading is like AI already. They talk
• 11:54
  Speaker 2: to chatbots more than other people and they just like
• 11:57
  Speaker 2: have lost this ability to differentiate. Uh-huh.
• 12:00
  Speaker 1: So maybe we'll all be at that far end of
• 12:02
  Speaker 1: the curve in a year.
• 12:04
  Speaker 2: I really hope not.
• 12:05
  Speaker 1: Yeah, I suppose I also hope not. So the companies
• 12:10
  Speaker 1: that are making frontier models, Anthropic, OpenAI, arguably Alphabet, Google,
• 12:17
  Speaker 1: are giant companies with wild amounts of money, full of geniuses.
• 12:23
  Speaker 1: You'd think those companies could just do what you're doing.
• 12:27
  Speaker 1: They could just build their own AI detectors. Why don't they?
• 12:31
  Speaker 2: I agree. I don't disagree here. But I also think
• 12:35
  Speaker 2: that this is just not in their top 10 priority list.
• 12:40
  Speaker 2: They're building AGI. They're automating white-collar work.
• 12:44
  Speaker 1: Probably the right answer. I mean, there's an answer, which
• 12:47
  Speaker 1: is like, is it contrary to their incentives to do
• 12:52
  Speaker 1: what you're doing, right? Is it in their interest for
• 12:56
  Speaker 1: people to be able to distinguish AI pros from human pros,
• 12:59
  Speaker 1: or is it contrary to their interest?
• 13:00
  Speaker 2: I think it is contrary to their interest. There was
• 13:02
  Speaker 2: an interesting article that I was exploring a couple of
• 13:07
  Speaker 2: years ago, actually, when OpenAI had watermarking technology, they'd built it,
• 13:12
  Speaker 2: but then they didn't want to release it. There was
• 13:14
  Speaker 2: a lot of internal politics on this, and this was
• 13:16
  Speaker 2: largely because they were afraid it would hurt their core
• 13:20
  Speaker 2: user base, which was at least a plurality of students
• 13:23
  Speaker 2: at this time.
• 13:24
  Speaker 1: They would get caught cheating if there was a watermark.
• 13:26
  Speaker 1: It's basically what you're saying.
• 13:27
  Speaker 2: They would get caught cheating, or they would churn to
• 13:30
  Speaker 2: a different AI service that didn't have a watermark.
• 13:33
  Speaker 1: So you brought up watermarks, so let's talk about watermarks.
• 13:37
  Speaker 1: The EU, as I understand, just passed a law requiring...
• 13:41
  Speaker 1: watermarks on text from language models. Is that true?
• 13:45
  Speaker 2: Yes, that's true. And OpenAI and Anthropic are both working
• 13:49
  Speaker 2: on this. Google already watermarks text outputs.
• 13:52
  Speaker 1: Is that bad for your business?
• 13:54
  Speaker 2: I think it's really good that people are taking this
• 13:56
  Speaker 2: problem seriously. It is competitive with us, but I think
• 14:00
  Speaker 2: Pangram has a lot of advantages that watermarks don't have,
• 14:03
  Speaker 2: which is that a watermark is just going to tell
• 14:05
  Speaker 2: you either Claude was here or not, and Pangram will
• 14:07
  Speaker 2: be able to tell you with higher granularity of like,
• 14:10
  Speaker 2: What was the degree of AI assistance? Was it just
• 14:12
  Speaker 2: helping or did it write the whole thing?
• 14:14
  Speaker 1: I mean, presumably, there's some universe where Claude outputs the
• 14:20
  Speaker 1: copy that's watermarked, and then you literally just retype it
• 14:24
  Speaker 1: into a Google Doc. I mean, naively, I would think
• 14:28
  Speaker 1: the watermark would not follow. Or is the watermark somehow
• 14:31
  Speaker 1: encoded in the prose so that if you retype it,
• 14:34
  Speaker 1: it's still there?
• 14:34
  Speaker 2: Yes, the watermark is statistically encoded in the word choices themselves.
• 14:40
  Speaker 1: Oh, that is rad. So even retyping it, the watermark
• 14:43
  Speaker 1: is still there.
• 14:44
  Speaker 2: Yes. With that said, you could run some automated paraphraser
• 14:50
  Speaker 2: to just paraphrase all the words, and that would erase
• 14:53
  Speaker 2: the watermark.
• 14:54
  Speaker 1: Yes. Well, so there are humanizers now, right? There are
• 14:59
  Speaker 1: things you can upload your language model pros into that
• 15:03
  Speaker 1: claim to evade detection. Although I will say I tried one,
• 15:07
  Speaker 1: I won't name it, but I tried one in preparing
• 15:09
  Speaker 1: for this interview, and you'll be glad to know it
• 15:11
  Speaker 1: didn't work. I had the language model output pros, I
• 15:14
  Speaker 1: ran it through a humanizer, put it into PanGram, and
• 15:16
  Speaker 1: PanGram said 100% AI.
• 15:18
  Speaker 2: Amazing. Yeah, I mean, this is part of the adversarial
• 15:21
  Speaker 2: nature of the business is We are also training on
• 15:24
  Speaker 2: humanizer outputs because we need to make sure that pangram
• 15:28
  Speaker 2: can detect anything that's been run through a humanizer. Otherwise,
• 15:33
  Speaker 2: it would kind of be trivial to evade pangram.
• 15:36
  Speaker 1: Why can't they beat you? I would think you could
• 15:39
  Speaker 1: be at a degree of randomness at the level of
• 15:44
  Speaker 1: word choice and syntax that would defeat you. Why can't they?
• 15:50
  Speaker 2: I think these are kind of like two, there's two
• 15:52
  Speaker 2: competing poles for text when you're humanizing. You want it
• 15:57
  Speaker 2: to be coherent and, you know, retain its logical flow.
• 16:02
  Speaker 2: And then on the other side, you want to just
• 16:05
  Speaker 2: degrade it or perturb it a little bit enough that
• 16:07
  Speaker 2: it is outside of the range of AI detectors. There
• 16:11
  Speaker 2: are humanizers that can beat pangram and you could do
• 16:15
  Speaker 2: this by just like perturbing the text way more just
• 16:19
  Speaker 2: a lot more.
• 16:19
  Speaker 1: Well, sure. I mean, there's a difference between degrading and perturbing, right? Like,
• 16:25
  Speaker 1: you can make it really bad, but that's obviously not useful.
• 16:28
  Speaker 2: Well, I think degrading is just a more extreme version
• 16:31
  Speaker 2: of perturbing.
• 16:32
  Speaker 1: Well, you could improve. You could perturb it a lot
• 16:35
  Speaker 1: and improve it. Oh, yeah, yeah.
• 16:37
  Speaker 2: But these humanizers, they don't have the resources to actually
• 16:39
  Speaker 2: do that, right?
• 16:40
  Speaker 1: Yeah.
• 16:41
  Speaker 2: Most of them are, like, one- to ten-person companies.
• 16:45
  Speaker 1: There's a more fundamental question here that is interesting to me,
• 16:48
  Speaker 1: and I don't really understand the answer, which is, like,
• 16:52
  Speaker 1: why does AI slop all sound the same? You know
• 16:55
  Speaker 1: what I mean? Like, that's the thing that's so annoying.
• 16:57
  Speaker 1: Why can't you just turn up the volume, not on, like,
• 17:00
  Speaker 1: make it bad, but just, like, don't use that same rhythm.
• 17:04
  Speaker 1: Don't do, like, the one sentence. Don't use these annoying
• 17:06
  Speaker 1: words that are AI words. Like, why is that at
• 17:10
  Speaker 1: a technical level?
• 17:12
  Speaker 2: It's really hard to say. I think... So there's a
• 17:15
  Speaker 2: concept in machine learning called mode collapse. And that means like,
• 17:18
  Speaker 2: you know, once an AI model learns what is best
• 17:23
  Speaker 2: or most common, which is the mode, then it will
• 17:27
  Speaker 2: kind of like collapse the edges of its distribution and
• 17:30
  Speaker 2: just output the mode much more often.
• 17:33
  Speaker 1: Yes. The mode being the modal word, the most common word.
• 17:36
  Speaker 2: Yeah, the most common word or phrase or sentence construction.
• 17:39
  Speaker 2: I think that's part of it. And I think also...
• 17:42
  Speaker 2: a lot of these labs, they do post-training. They'll create
• 17:45
  Speaker 2: a reward model and they'll give the language model a reward.
• 17:49
  Speaker 2: And so I can only suspect that there's somebody in
• 17:52
  Speaker 2: here who's saying there's some reward for, quote, like good writing.
• 17:57
  Speaker 2: And that might include, you know, writing with em dashes
• 18:02
  Speaker 2: is probably correlated with good writing in the pre-AI world.
• 18:05
  Speaker 2: Same thing with the like Like, it's not just X
• 18:08
  Speaker 2: but Y sort of metaphors.
• 18:10
  Speaker 1: Yes. That's another one that I have taken out of
• 18:12
  Speaker 1: my own writing, actually. Yeah, yeah.
• 18:13
  Speaker 2: Like, it's a very reasonable construction, but then you feed
• 18:16
  Speaker 2: it into the reward model, and then the AI learns
• 18:20
  Speaker 2: in its prose to turn that up to 11.
• 18:21
  Speaker 1: It likes it too much, yeah.
• 18:23
  Speaker 2: Yeah. So it just overuses what would have normally been
• 18:26
  Speaker 2: considered good writing.
• 18:31
  Speaker 1: We'll be back in just a minute. Okay, let's talk
• 18:43
  Speaker 1: about false positives, right? This is a thing people care
• 18:45
  Speaker 1: about and have written about. So obviously in any test,
• 18:48
  Speaker 1: you can have a false positive or a false negative.
• 18:50
  Speaker 1: A false negative, you're saying this is human generated and
• 18:52
  Speaker 1: actually it's AI. That one people don't get so upset about.
• 18:55
  Speaker 1: The one people are really afraid of is wrongly accused, right?
• 18:59
  Speaker 1: Pangram says it was AI. Actually, it was written by
• 19:02
  Speaker 1: a person. I've seen you say one in 10,000 is
• 19:07
  Speaker 1: the false positive rate. And I've also seen outside studies that,
• 19:12
  Speaker 1: find a very low false positive rate, a similar order
• 19:14
  Speaker 1: of magnitude, one from the University of Chicago I found
• 19:17
  Speaker 1: particularly compelling. But like, I don't know how many writing
• 19:20
  Speaker 1: assignments students have in high school and college, but it's
• 19:23
  Speaker 1: probably order of magnitude of hundreds, right? And so that
• 19:27
  Speaker 1: means that some share of students are going to be
• 19:29
  Speaker 1: wrongly accused, right? If it's 500 writing assignments, then it
• 19:35
  Speaker 1: would be 5% of students will be wrongly accused by
• 19:38
  Speaker 1: Pan Graham of using AI to write a thing that
• 19:41
  Speaker 1: they actually wrote?
• 19:42
  Speaker 2: Yeah, I do think like 1 in 10,000 is beyond
• 19:45
  Speaker 2: the threshold where like you can use this as a
• 19:49
  Speaker 2: tool and generally typically be confident that when it says
• 19:52
  Speaker 2: it's AI, it's correct.
• 19:54
  Speaker 1: It's true. That's definitely true. Like, I mean, I will
• 19:56
  Speaker 1: say just I was actually accused by a teacher in
• 19:59
  Speaker 1: high school of plagiarizing. That's the old school. And I
• 20:02
  Speaker 1: had not plagiarized, for the record. And so that is
• 20:05
  Speaker 1: the old school human version of this. And I'm sure
• 20:08
  Speaker 1: his error rate was higher than 1 in 10,000. But
• 20:12
  Speaker 1: I am sensitive to it. Like, that feeling of being
• 20:14
  Speaker 1: wrongly accused is a terrible feeling. Mm-hmm.
• 20:17
  Speaker 2: Yeah, so I think one thing that pangram helps with
• 20:19
  Speaker 2: actually is it's much better than like the average human intuition.
• 20:24
  Speaker 2: And it's also better than any other AI detector. So
• 20:27
  Speaker 2: because of that, the existence of pangram is, I truly
• 20:32
  Speaker 2: believe this, reducing the number of false accusations for AI
• 20:35
  Speaker 2: use in a scenario like education.
• 20:38
  Speaker 1: Yeah, that's fair. So like notionally, like, of course, teachers
• 20:42
  Speaker 1: are on the lookout for AI one way or the
• 20:43
  Speaker 1: other if they want students to write it.
• 20:45
  Speaker 2: Exactly. And then the other side, I think, which is
• 20:47
  Speaker 2: really important, is that it's not the only source of information.
• 20:53
  Speaker 2: Once Pangram says that something's AI, we can always go deeper.
• 20:56
  Speaker 2: We can either ask the student for drafts. If you
• 21:01
  Speaker 2: use Google Docs, Pangram actually has this Chrome extension where
• 21:04
  Speaker 2: you could see the full revision history, you could see
• 21:07
  Speaker 2: that the document being typed out.
• 21:09
  Speaker 1: And that history is available even without PanGram, right?
• 21:11
  Speaker 2: Exactly.
• 21:12
  Speaker 1: And if you're using Chrome, you can see the version
• 21:14
  Speaker 1: history and go back.
• 21:15
  Speaker 2: Exactly. Yes. Yeah. So that version history is there. But
• 21:19
  Speaker 2: I think PanGram is very important because you don't want
• 21:22
  Speaker 2: to go look at the version history for every single doc.
• 21:24
  Speaker 2: You only want to look into it once you suspect
• 21:27
  Speaker 2: that something was wrong or there are some AIUs there.
• 21:30
  Speaker 1: That's funny. I thought, well, you could automate looking at
• 21:33
  Speaker 1: the version history. And then I was like, oh, but
• 21:34
  Speaker 1: you could also automate a little script that would make
• 21:37
  Speaker 1: it look like you wrote it. It's the adversarial thing.
• 21:41
  Speaker 1: It is amazing. I was reading an article in New
• 21:42
  Speaker 1: York Magazine about students using AI to cheat. And one
• 21:44
  Speaker 1: of the things they pointed out is like how much
• 21:46
  Speaker 1: time students spend cheating, like trying to use AI and
• 21:51
  Speaker 1: not get caught. And like that amount of time you
• 21:53
  Speaker 1: could have written the paper, right? Which is a classic thing,
• 21:56
  Speaker 1: but maybe it's more fun to cheat. Well, yeah.
• 21:57
  Speaker 2: I mean, I think that's part of the like, friction,
• 22:00
  Speaker 2: like adding friction to this cheating process where fewer students
• 22:04
  Speaker 2: are going to do it if it is time consuming
• 22:06
  Speaker 2: to cheat. If it's just one prompt, I think that's
• 22:10
  Speaker 2: where like a lot of students and faculty, like everyone's
• 22:14
  Speaker 2: kind of been frustrated with this problem. It also sucks
• 22:16
  Speaker 2: when your peers are doing one sentence prompts and getting
• 22:19
  Speaker 2: away with it.
• 22:21
  Speaker 1: Right. That's a bad equilibrium. That's a really bad equilibrium. So, okay,
• 22:24
  Speaker 1: so out of every 10,000 times you say AI had
• 22:28
  Speaker 1: anything to do with it, you only get it wrong once.
• 22:30
  Speaker 2: Correct. It's a little bit more of a spectrum in
• 22:32
  Speaker 2: the middle when somebody believes that they had 50% input
• 22:38
  Speaker 2: into the document and AI had 50% input and then
• 22:42
  Speaker 2: Paragram said it's 100% AI. Like, this is sort of
• 22:46
  Speaker 2: the gray area where it's much harder to measure if
• 22:49
  Speaker 2: we're right or not.
• 22:50
  Speaker 1: Uh-huh. And so that's the area where you get it
• 22:54
  Speaker 1: wrong more is kind of at that margin. So it's like, no, no,
• 22:57
  Speaker 1: I wrote it and I got some edits from AI,
• 23:00
  Speaker 1: but PanGram returns 100% AI. Like that happens more often
• 23:04
  Speaker 1: than 1 in 10,000.
• 23:05
  Speaker 2: Yeah, that happens at least for like light AI assistants
• 23:09
  Speaker 2: or medium AI assistants. It's like a few percent of
• 23:13
  Speaker 2: the time where we say it's still like 100% AI.
• 23:17
  Speaker 1: So a few percent is orders of magnitude higher than
• 23:21
  Speaker 1: one in 10,000. It happens kind of a lot.
• 23:24
  Speaker 2: We try to be very transparent about all of our benchmarks.
• 23:27
  Speaker 2: I think this is also the area of greatest debate
• 23:31
  Speaker 2: where it's not actually so black and white. What the
• 23:35
  Speaker 2: degree of AI assistance was, people on average tend to under-disclose.
• 23:40
  Speaker 2: They're like, oh, I use AI to help me. I
• 23:42
  Speaker 2: talked to someone who's like, I used AI for copy edits.
• 23:45
  Speaker 2: And then you go through what that means. It's like
• 23:47
  Speaker 2: they actually, like, every word was produced by AI. And
• 23:52
  Speaker 2: it had gone through so many revisions with AI. And
• 23:56
  Speaker 2: so I think this is the biggest gray area where
• 23:59
  Speaker 2: Pangram is not going to be, like, a perfect oracle
• 24:02
  Speaker 2: and tell you this was the exact prompt and these
• 24:05
  Speaker 2: were the exact human edits. But I think it can, like,
• 24:08
  Speaker 2: get approximately there.
• 24:10
  Speaker 1: Are there... groups of people or kinds of speakers where
• 24:14
  Speaker 1: pangram is worse at detecting whether their writing is their
• 24:18
  Speaker 1: own or generated by AI?
• 24:20
  Speaker 2: So we've run this on three different groups. I think
• 24:25
  Speaker 2: there's been different studies in the past that have found
• 24:29
  Speaker 2: that AI detectors are biased against English as second language speakers.
• 24:36
  Speaker 2: There's also been questions of if AI detectors are biased
• 24:41
  Speaker 2: against autistic or neurodivergent speakers. And so what we've done
• 24:47
  Speaker 2: is we ran studies where we looked at 25,000 texts
• 24:52
  Speaker 2: by English as second language speakers and found that there's
• 24:56
  Speaker 2: no elevated false positive rate. We have one false positive
• 25:00
  Speaker 2: in the 25,000. And then so similarly, somebody else who
• 25:05
  Speaker 2: is not us asked for credits to run a similar
• 25:09
  Speaker 2: study for autistic speakers and they did a really great job.
• 25:14
  Speaker 2: They had like a 20 page report on their findings
• 25:17
  Speaker 2: and they also found no bias against autistic speakers, which
• 25:21
  Speaker 2: is really great. And then the third one that we've
• 25:25
  Speaker 2: looked at was just like race. Cause I think this
• 25:28
  Speaker 2: has also been a question and we again found no
• 25:32
  Speaker 2: statistically significant difference, which it was really cool to see. What?
• 25:37
  Speaker 1: Yes.
• 25:38
  Speaker 2: Yes.
• 25:38
  Speaker 1: One hopes that would not be a problem. You could
• 25:40
  Speaker 1: imagine how for someone who was not a native speaker,
• 25:43
  Speaker 1: for example, you might have a higher rate of false positives.
• 25:46
  Speaker 1: Was it a problem in the past? Like were earlier
• 25:48
  Speaker 1: versions of PanGram less reliable in that way?
• 25:52
  Speaker 2: I would say like PanGram 2, like definitely had a
• 25:54
  Speaker 2: bit of this issue. And then starting like PanGram 3,
• 25:58
  Speaker 2: we were taking it very seriously. But I think every,
• 26:02
  Speaker 2: we've had a lot of candidate models that were biased
• 26:04
  Speaker 2: against English second language speakers and And so because of that,
• 26:08
  Speaker 2: we kind of threw away the model and never released it.
• 26:11
  Speaker 1: And when was Pangram 2?
• 26:12
  Speaker 2: How long ago was that? Like a year and a
• 26:16
  Speaker 2: half ago. Even Pangram 2, like our false positive rate
• 26:20
  Speaker 2: was like much lower than the other AI detectors. But
• 26:25
  Speaker 2: I think it was still like compared to our baseline
• 26:28
  Speaker 2: false positive rate, it was elevated on ESL. And I
• 26:32
  Speaker 2: think like part of the difficulty here is going back
• 26:35
  Speaker 2: to the humanizers. a lot of them are trying to
• 26:38
  Speaker 2: inject spelling mistakes, grammar mistakes as a way to evade
• 26:42
  Speaker 2: AI detection. And so training on these naively will make
• 26:48
  Speaker 2: your model correlate these spelling and grammar mistakes with AI, actually,
• 26:53
  Speaker 2: which is not correct.
• 26:54
  Speaker 1: Well, only if those are occurring at a higher rate
• 26:57
  Speaker 1: in the humanizers than they are in the human texts
• 27:00
  Speaker 1: you're putting in, right? Yeah.
• 27:02
  Speaker 2: I think the humanizers, some of them are perturbing and
• 27:05
  Speaker 2: degrading the text more than your average human. So it's
• 27:09
  Speaker 2: taking it like all the way down to the other side. Yeah.
• 27:12
  Speaker 1: There have been a number of cases where, you know,
• 27:15
  Speaker 1: Pangram found a piece of writing to be AI generated.
• 27:17
  Speaker 1: The person said, no, it wasn't. I wrote it. Has
• 27:20
  Speaker 1: there been a case where the person was validated? A
• 27:23
  Speaker 1: high profile case?
• 27:24
  Speaker 2: Yeah. So there was like one case, Taylor Lorenz had
• 27:28
  Speaker 2: this like article in Vanity Fair. So Pangram has a
• 27:31
  Speaker 2: 50 word minimum. Somebody selected a 54 word segment and
• 27:37
  Speaker 2: Pangram said this was AI generated. But what they did
• 27:40
  Speaker 2: was they just posted the like a hundred percent AI
• 27:44
  Speaker 2: for the 54 word segment. And so then I looked
• 27:46
  Speaker 2: into it. If I checked like the full Vandy fair article,
• 27:50
  Speaker 2: Pangram actually said a hundred percent human. And then, you know,
• 27:53
  Speaker 2: Taylor's like, Hey, like here's my edit history. Like we
• 27:56
  Speaker 2: went through a lot of provisions with, with an editor,
• 27:58
  Speaker 2: like this is all my writing. And so, yeah, like it's,
• 28:01
  Speaker 2: it was absolutely a false positive and, Yeah, partially because
• 28:05
  Speaker 2: it was a small piece of text and partially, I think,
• 28:07
  Speaker 2: because this person was kind of using pangram in an
• 28:10
  Speaker 2: inappropriate way of, like, I don't know, trying to go
• 28:12
  Speaker 2: for the gotcha rather than, like, going for the full story.
• 28:15
  Speaker 1: Well, that 1 in 10,000 false positive rate, is that
• 28:19
  Speaker 1: contingent on some minimum number of words? And if so,
• 28:21
  Speaker 1: what number of words?
• 28:23
  Speaker 2: This is aggregate over positive. all of our texts, but
• 28:26
  Speaker 2: it's a little bit higher for 50 words.
• 28:28
  Speaker 1: If you put in 50 words, is the false positive
• 28:31
  Speaker 1: rate notably higher?
• 28:32
  Speaker 2: Yes, it's closer to 1 in 5,000. Uh-huh.
• 28:36
  Speaker 1: I mean, maybe the person looked for the most AI-sounding 50.
• 28:39
  Speaker 2: Words, right? I mean, that's what they said, is they
• 28:41
  Speaker 2: said this sounded like AI to them. So it's a
• 28:43
  Speaker 2: false positive to them and also to PanGram.
• 28:46
  Speaker 1: Okay, let's talk about images. Images are new. I think
• 28:50
  Speaker 1: when we booked you, maybe, I don't even know if
• 28:52
  Speaker 1: the images thing was out, but tell me about your
• 28:54
  Speaker 1: image business. Yeah.
• 28:55
  Speaker 2: So it's a new model. We released it in July,
• 28:59
  Speaker 2: end of July, but it is state-of-the-art for AI image detection.
• 29:04
  Speaker 2: So what that means is you can put in an
• 29:06
  Speaker 2: image and we'll tell you, yes, this is AI generated
• 29:10
  Speaker 2: and it works on images from nano banana, like Gemini,
• 29:15
  Speaker 2: GPT image, mid journey, all of the major AI image models.
• 29:20
  Speaker 2: And so I think this is pretty cool because it
• 29:23
  Speaker 2: A couple of years ago, it was like really easy
• 29:24
  Speaker 2: to tell whether an image was AI generated. You just
• 29:27
  Speaker 2: like count the fingers and you look at the text.
• 29:28
  Speaker 1: Count the fingers, right. Or see if there's something that
• 29:30
  Speaker 1: defies the laws of physics, right?
• 29:32
  Speaker 2: Yeah. And today that happens a little bit, but like
• 29:35
  Speaker 2: you can get some really hyper-realistic AI images. It's scary.
• 29:39
  Speaker 2: And I think it's really hard to tell.
• 29:42
  Speaker 1: Certainly hard for people to tell. I mean, I certainly
• 29:44
  Speaker 1: buy that you are at the forefront of language detection.
• 29:49
  Speaker 1: What is the competitive landscape like for image detection?
• 29:52
  Speaker 2: There's a number of AI image detectors. I think many
• 29:57
  Speaker 2: of them are more focused on deepfakes and less focused
• 30:02
  Speaker 2: on AI images in general.
• 30:04
  Speaker 1: Yes, I talked to the head of Lodi. And they're
• 30:07
  Speaker 1: focused in particular on videos. Like his claim, at least,
• 30:10
  Speaker 1: was that they are good at understanding video, ingesting video
• 30:13
  Speaker 1: in a way that other people are not.
• 30:15
  Speaker 2: Oh, that's pretty cool.
• 30:15
  Speaker 1: Because video is hard.
• 30:17
  Speaker 2: Yeah. Yeah, we've been working a bit with video. So
• 30:19
  Speaker 2: if you put in a single frame from a video
• 30:21
  Speaker 2: into the AI image detector, it will do pretty well.
• 30:24
  Speaker 1: What kinds of businesses are natural customers for your image product?
• 30:28
  Speaker 2: Yeah, I think naturally it would be someone like Pinterest
• 30:33
  Speaker 2: or like any like stock photo company. image website, things
• 30:38
  Speaker 2: like that, where you want your users to have some
• 30:41
  Speaker 2: degree of trust that what they're seeing is real.
• 30:44
  Speaker 1: Dating apps or websites was the first one I thought of.
• 30:47
  Speaker 2: Yeah, I think dating apps, absolutely. You don't want AI-generated
• 30:51
  Speaker 2: people or AI catfishing.
• 30:54
  Speaker 1: Real estate apps was another one I thought. I mean,
• 30:56
  Speaker 1: I know they will explicitly sometimes be like, and here's
• 30:59
  Speaker 1: what it could look like. But like, they don't always
• 31:01
  Speaker 1: say that. I feel like a lot of the time,
• 31:04
  Speaker 1: like my neighborhood has all these old houses that people
• 31:06
  Speaker 1: have lived in for 50 years, but they sell for
• 31:08
  Speaker 1: a lot of money now, right? Which is the classic
• 31:09
  Speaker 1: thing where the realtor wants to be like, look, you
• 31:11
  Speaker 1: can pay a lot of money for this house because
• 31:13
  Speaker 1: it could look like this. But they don't show you
• 31:15
  Speaker 1: what it looks like now for a reason.
• 31:17
  Speaker 2: Oh, that's horrendous.
• 31:18
  Speaker 1: Yeah, yeah.
• 31:19
  Speaker 2: I mean, I think that sort of stuff, like it
• 31:21
  Speaker 2: should be illegal to... to put an AI photo instead
• 31:25
  Speaker 2: of the actual photo in your listing?
• 31:27
  Speaker 1: They may well say it. I don't want to be
• 31:28
  Speaker 1: accusing people of malice, but, like... No, I've.
• 31:32
  Speaker 2: Definitely seen it for apartment listings in New York. Yeah. Like,
• 31:34
  Speaker 2: where I show up, and it's like, huh, it looks
• 31:37
  Speaker 2: a little bit different than the photos. And I go
• 31:39
  Speaker 2: back to the photos, and, like, some of the details
• 31:41
  Speaker 2: are different. And I'm like, okay, I think... I'm, like,
• 31:44
  Speaker 2: pretty sure this was AI.
• 31:45
  Speaker 1: Wait, it's a shady apartment listing in New York? No,
• 31:48
  Speaker 1: I don't believe you. I.
• 31:51
  Speaker 2: Whenever it's too good to be true, half the time
• 31:54
  Speaker 2: it's AI.
• 31:55
  Speaker 1: One time I called for an apartment in New York
• 31:58
  Speaker 1: to go look at it, and the guy's first question was,
• 32:00
  Speaker 1: how tall are you? I was like, I'm out. I'm
• 32:03
  Speaker 1: not going to go look. We'll be back in a
• 32:10
  Speaker 1: minute with The Lightning Round. Okay, let's finish with the
• 32:23
  Speaker 1: lightning round. Cool. How has building PanGram changed the way
• 32:28
  Speaker 1: you think about language?
• 32:31
  Speaker 2: I don't know if I have an answer for this.
• 32:34
  Speaker 1: Maybe it hasn't. Maybe it hasn't.
• 32:35
  Speaker 2: No, I mean, I think some of the biggest learnings
• 32:39
  Speaker 2: for me were that, oh, like, these language models are
• 32:43
  Speaker 2: not at all as diverse as I thought. Yeah. When
• 32:47
  Speaker 2: I started, like, ChatGPT had come out, and we were
• 32:50
  Speaker 2: all like, this is amazing. incredible technology. And then spending
• 32:54
  Speaker 2: a little more time with it made me realize, oh, actually,
• 32:56
  Speaker 2: it's kind of just like saying the same thing every time.
• 32:59
  Speaker 2: It's quite mode collapse. And humans are much more diverse.
• 33:02
  Speaker 2: And I think that's valuable.
• 33:05
  Speaker 1: Well, what's the last thing you read that you loved?
• 33:08
  Speaker 2: I read this book called Sublimation by Isabel J. Kim.
• 33:11
  Speaker 2: It was about sort of this concept where if somebody immigrates,
• 33:15
  Speaker 2: they split into two people. And one person stays, one
• 33:20
  Speaker 2: person goes. It was very interesting. It's a good book.
• 33:22
  Speaker 1: It's a novel?
• 33:23
  Speaker 2: Novel, yeah.
• 33:24
  Speaker 1: You ever read anything by AI that you loved?
• 33:27
  Speaker 2: There's this kind of funny green text from an old
• 33:31
  Speaker 2: AI model about being a bottomless pit supervisor.
• 33:37
  Speaker 1: Wait, I don't know what green text is. What's green text?
• 33:40
  Speaker 2: Green text, it's this format of storytelling from 4chan where
• 33:46
  Speaker 2: it's basically describing a story in first person. Every line
• 33:52
  Speaker 2: starts with this carrot or this like.
• 33:54
  Speaker 1: I see. Kind of like looking at a command line
• 33:56
  Speaker 1: or something.
• 33:57
  Speaker 2: Yeah, like the command line thing. And then I think
• 33:59
  Speaker 2: because of that, like the formatting, it was green.
• 34:02
  Speaker 1: Okay, so go on. So you saw something that was
• 34:04
  Speaker 1: like this and it was about?
• 34:06
  Speaker 2: Yeah, it was kind of just like a joke story
• 34:08
  Speaker 2: about being a bottomless pit supervisor. And I was like, okay,
• 34:12
  Speaker 2: this is like, this is kind of funny. This is
• 34:14
  Speaker 2: the first time I've like laughed at something that was
• 34:16
  Speaker 2: AI generated.
• 34:18
  Speaker 1: How did you come across it?
• 34:19
  Speaker 2: It was posted on Twitter. I can read it for
• 34:22
  Speaker 2: you if you like.
• 34:23
  Speaker 1: Yeah, that'd be great.
• 34:24
  Speaker 2: Okay. So it starts with the human prompt is write
• 34:28
  Speaker 2: me a 4chan green text, be me bottomless pit supervisor.
• 34:33
  Speaker 2: And then AI continues this with in charge of making
• 34:37
  Speaker 2: sure the bottomless pit is in fact bottomless. Occasionally have
• 34:40
  Speaker 2: to go down there and check if the bottomless pit
• 34:42
  Speaker 2: is still bottomless. One day I go down there and
• 34:45
  Speaker 2: the bottomless pit is no longer bottomless and The bottom
• 34:48
  Speaker 2: of the bottomless pit is now just a regular pit. Distress.jpg.
• 34:53
  Speaker 2: Ask my boss what to do. He says, just make
• 34:56
  Speaker 2: it bottomless again. I say, how? He says, I don't know.
• 35:00
  Speaker 2: You're the supervisor. Rage.jpg. Quit my job. Become a regular
• 35:05
  Speaker 2: pit supervisor. First day on the job, go down to
• 35:08
  Speaker 2: the new hole. It's bottomless.
• 35:11
  Speaker 1: That's great. It also doesn't sound like AI. Yeah. To me,
• 35:17
  Speaker 1: to my ear, anyway.
• 35:19
  Speaker 2: Yeah, I think that's because this was, like, old AI.
• 35:23
  Speaker 2: I want to say it's, like, GPT-2 or 3.
• 35:26
  Speaker 1: I also wonder, like, it sounds like it's not a
• 35:28
  Speaker 1: form that I'm familiar with, but it sounds like it
• 35:30
  Speaker 1: is this very particular form of writing that exists in
• 35:33
  Speaker 1: the world. And so, just like if you say, write
• 35:35
  Speaker 1: a sonnet or whatever, I don't know, maybe Pangram can
• 35:38
  Speaker 1: tell an AI sonnet from a human-generated sonnet, or maybe
• 35:40
  Speaker 1: it's not enough words. But, like... When there is a form,
• 35:45
  Speaker 1: maybe we all sort of collapse. Humans collapse to the
• 35:48
  Speaker 1: form as well. And so the collapse of AI is
• 35:50
  Speaker 1: maybe less obvious. I don't know.
• 35:52
  Speaker 2: I think so. Yeah. The collapse of the form and
• 35:54
  Speaker 2: adherence to this structure is, I think, makes it interesting.
• 35:59
  Speaker 2: And it's very interesting that AI is able to adhere
• 36:01
  Speaker 2: to the structure so well.
• 36:03
  Speaker 1: And it gets the joke, right? Like bottomless pit supervisor
• 36:07
  Speaker 1: is an absurd premise, right? And it treats it just right.
• 36:11
  Speaker 1: Like it's a deaf comic touch, right?
• 36:14
  Speaker 2: Yeah, it was good.
• 36:15
  Speaker 1: Tell me about your Twitter game, your X game. You're
• 36:18
  Speaker 1: on a lot and you're kind of a poster. Like
• 36:20
  Speaker 1: your persona on Twitter is a little bit more, I
• 36:23
  Speaker 1: don't know if combative is too strong of a word,
• 36:26
  Speaker 1: than your persona in this interview.
• 36:28
  Speaker 2: I like to be on Twitter a lot. I post
• 36:31
  Speaker 2: all the time. I kind of just like post what
• 36:33
  Speaker 2: I think. I don't think too hard about it. But
• 36:36
  Speaker 2: I think Twitter is a really great community because... First off,
• 36:40
  Speaker 2: I like text. I'm not on TikTok. I'm not filming
• 36:43
  Speaker 2: videos of myself. I like just typing things out. So
• 36:46
  Speaker 2: I think it's a great place for me. And it's
• 36:49
  Speaker 2: also kind of like where all the AI happenings happen.
• 36:53
  Speaker 2: So a lot of the people I follow are people
• 36:55
  Speaker 2: who work in AI in various capacities, either AI researchers
• 36:59
  Speaker 2: or independent people doing cool things with AI. So I
• 37:04
  Speaker 2: post a lot.
• 37:05
  Speaker 1: Are you not more combative on Twitter? Am I misreading that?
• 37:09
  Speaker 2: Sometimes I think, so I often will get like, somebody
• 37:13
  Speaker 2: will say, you know, pangram was wrong on like so-and-so
• 37:15
  Speaker 2: and then I'll have to jump in and be like, Hey,
• 37:18
  Speaker 2: you know, can you send me the text? Like we
• 37:20
  Speaker 2: want to go like triage it and figure that out.
• 37:22
  Speaker 1: Yeah.
• 37:23
  Speaker 2: Uh, in terms of combativeness, I, I'm not like picking
• 37:28
  Speaker 2: fights all the time, but I, I mean, I think
• 37:30
  Speaker 2: maybe it's just a function of the platform and the
• 37:33
  Speaker 2: format where like, you know, here we're having such a
• 37:36
  Speaker 2: like nice, pleasant conversation. But, like, there's no pleasant conversations
• 37:40
  Speaker 2: on Twitter.
• 37:41
  Speaker 1: Yeah, it's interesting. I will occasionally get angry emails, as
• 37:45
  Speaker 1: one does. And almost every time, if I write back
• 37:48
  Speaker 1: in a polite way, the person apologizes. I don't even
• 37:51
  Speaker 1: ask them for an apology. I just say, yes, thank
• 37:53
  Speaker 1: you for pointing that out. Here is my thinking. And
• 37:57
  Speaker 1: more often than not, if I do that, the person
• 37:59
  Speaker 1: will write back apologetically, basically, or certainly gratefully. What's your
• 38:06
  Speaker 1: least favorite AI thing?
• 38:10
  Speaker 2: My least favorite is when it's like, and honestly. I'm
• 38:14
  Speaker 2: just like, that's horrible.
• 38:18
  Speaker 1: Load-bearing and doing the work are the two.
• 38:21
  Speaker 2: That, yeah, those make me cringe.
• 38:32
  Speaker 1: Max Biro is the co-founder and CEO of Pangram. Today's
• 38:36
  Speaker 1: show was produced by Gabriel Hunter Chang. It was engineered
• 38:40
  Speaker 1: by Sarah Bruguere. And our showrunner and editor is Jake Harper.
• 38:44
  Speaker 1: I'm Jacob Goldstein, and we'll be back next week with
• 38:47
  Speaker 1: another episode of What's Your Problem?