Latest / Azadeh and Guy on AI / From Grammarly to GitHub

Transcript

From Grammarly to GitHub · Azadeh and Guy on AI

• 0:01
  SPEAKER_00: Hi and welcome to Azadeh and Gai on AI.
• 0:04
  This is episode 11 and today we're talking about Grammarly and
• 0:07
  the new things that
• 0:08
  SPEAKER_00: we can see on Grammarly because I'm really excited about that.
• 0:12
  SPEAKER_02: But before that, let's introduce ourselves.
• 0:15
  SPEAKER_02: Hello everyone, my name is Azadeh.
• 0:18
  I'm a PhD student here at UNL and my major is Innovative
• 0:22
  Learning Technologies.
• 0:24
  SPEAKER_00: And I'm Guy Training, and I obviously got this wrong.
• 0:26
  And the second thing that we're going to talk about today is
• 0:30
  the novice problem,
• 0:32
  SPEAKER_00: and we're exploring it through a few papers on programming.
• 0:36
  SPEAKER_00: So let's go to Grammarly.
• 0:37
  SPEAKER_02: Yeah.
• 0:38
  SPEAKER_00: I've been using Grammarly a long time.
• 0:40
  SPEAKER_02: Yeah, I've been using Grammarly for a year.
• 0:44
  I think that many people whose first language is not English
• 0:48
  started using
• 0:49
  Grammarly because it was a very good way to get the writing
• 0:55
  assistance that you
• 0:56
  SPEAKER_00: need.
• 0:58
  SPEAKER_00: while you're writing in your second language.
• 1:00
  SPEAKER_00: I know a lot of people use it when their first language is English.
• 1:03
  SPEAKER_00: So I'm not saying this has to be your second language.
• 1:07
  SPEAKER_00: But I actually found it really useful to pay for it very,
• 1:10
  SPEAKER_00: very early on because it was helping me.
• 1:12
  SPEAKER_00: And really what it does is reduce a lot of anxiety.
• 1:17
  SPEAKER_02: Yeah, it's interesting.
• 1:18
  SPEAKER_02: With all of those AI tools that currently exist, it is still popular.
• 1:23
  SPEAKER_00: Yeah.
• 1:23
  SPEAKER_00: They've been able to,
• 1:25
  SPEAKER_00: I think because they had a large user base,
• 1:29
  SPEAKER_00: they were able to say,
• 1:30
  SPEAKER_01: hey,
• 1:30
  SPEAKER_00: we've got an AI that we've been using on you for a long time.
• 1:34
  We just didn't tell you it was AI,
• 1:36
  at least not in the way that we've always done.
• 1:38
  SPEAKER_00: And that really worked interestingly.
• 1:41
  SPEAKER_00: And what's new about Grammarly is I opened it about a week ago.
• 1:44
  SPEAKER_00: Mm-hmm.
• 1:46
  and it's it opens up automatically and suddenly it looked
• 1:49
  different and i was
• 1:50
  wondering if it is different and not just looking different and
• 1:54
  it definitely is
• 1:55
  you showed me some of those features if i correctly remember
• 1:58
  they added features
• 1:59
  one of the most interesting features was uh they added an
• 2:04
  option that helps you to
• 2:06
  SPEAKER_02: look at your text from the perspective of readers that was my favorite
• 2:11
  SPEAKER_00: Yeah.
• 2:11
  So really what they've done more conceptually is they've
• 2:15
  stopped thinking about
• 2:16
  this as a platform that has certain services and they divided
• 2:21
  them up into agents
• 2:23
  SPEAKER_00: that can do different things.
• 2:24
  SPEAKER_00: So you can activate or deactivate agents.
• 2:27
  SPEAKER_00: So they've always had the proofreader.
• 2:32
  SPEAKER_00: Right?
• 2:33
  SPEAKER_00: And so you can have that.
• 2:35
  SPEAKER_00: And you can have an expert review and the AI detector.
• 2:40
  SPEAKER_00: They've had that for a while and the plagiarism check.
• 2:44
  SPEAKER_00: All of those were there.
• 2:46
  But now they have a host of other agents that can connect you
• 2:51
  in multiple ways.
• 2:52
  SPEAKER_00: And as you said,
• 2:53
  SPEAKER_00: one of the interesting ones is that ability to ask,
• 2:58
  SPEAKER_00: what is my audience going to think?
• 3:00
  SPEAKER_00: So in that one,
• 3:02
  SPEAKER_00: It's called reader reactions.
• 3:04
  SPEAKER_00: And if you click on it, you can choose.
• 3:09
  SPEAKER_00: to see what their reactions might be by different audiences.
• 3:13
  SPEAKER_00: So you get an outside view of what your writing does and doesn't do,
• 3:18
  SPEAKER_00: which is a different kind of critique that is usually really hard to get.
• 3:23
  SPEAKER_00: You can do that on a normal LLM, but not at one press of a button.
• 3:27
  SPEAKER_02: You actually have to write a prompt and think about it.
• 3:31
  SPEAKER_02: And is it possible to customize those agents?
• 3:34
  SPEAKER_02: Yes.
• 3:35
  It is possible to customize them to a certain degree,
• 3:38
  and it depends on the agent.
• 3:40
  So,
• 3:41
  for example,
• 3:41
  if you're using the AI detector,
• 3:43
  there's no customization that I know.
• 3:45
  SPEAKER_00: No, obviously.
• 3:46
  SPEAKER_00: But if you want something else, yes, you can customize it.
• 3:51
  SPEAKER_00: You can give it specific directions about how to go about it.
• 3:57
  SPEAKER_00: doing these kind of things.
• 3:59
  And what I love is they've created a whole plethora of agents
• 4:03
  so you can start
• 4:04
  SPEAKER_00: thinking about what makes sense to you and what doesn't.
• 4:08
  SPEAKER_00: For example,
• 4:10
  there's a Wikipedia agent that can tell you what Wikipedia says
• 4:13
  about what you're
• 4:14
  SPEAKER_00: saying.
• 4:14
  SPEAKER_02: That's interesting.
• 4:16
  SPEAKER_00: There's a trigonometry agent.
• 4:19
  SPEAKER_00: I'm pretty sure I'm never going to use that one.
• 4:21
  SPEAKER_00: Okay.
• 4:23
  SPEAKER_00: But it's the last one.
• 4:24
  SPEAKER_00: There's a latex agent.
• 4:26
  SPEAKER_00: Now, that sounds fantastic for anybody who's battled with latex.
• 4:31
  SPEAKER_00: That's an add-on for publication,
• 4:33
  SPEAKER_00: and it's a specific format,
• 4:35
  SPEAKER_00: and it usually gives me a lot of trouble.
• 4:38
  SPEAKER_00: So all of these things are fantastic as agents you can add to your...
• 4:43
  SPEAKER_00: to your workspace through Grammarly.
• 4:46
  SPEAKER_00: So I'm excited about this.
• 4:48
  SPEAKER_00: There's another project that I was excited about early on,
• 4:53
  and that is they promised to build the platform that registers
• 4:57
  everything you do so
• 4:58
  SPEAKER_00: you know where it came from.
• 5:00
  So quick question,
• 5:01
  are they charging their clients for those added features or...
• 5:05
  SPEAKER_00: I have no clue.
• 5:07
  SPEAKER_00: They're not charging extra, but I'm on a paid account.
• 5:09
  SPEAKER_02: Okay.
• 5:10
  SPEAKER_00: So I'm using my paid account, and that seems to work.
• 5:14
  SPEAKER_00: I am pretty sure that just like what happened with Claude last night,
• 5:18
  SPEAKER_00: I mean,
• 5:18
  SPEAKER_00: I'm still burning about that.
• 5:19
  SPEAKER_01: Okay.
• 5:21
  and it had told me i was out of tokens and i need to wait till
• 5:24
  2 a.m i was not
• 5:26
  gonna wait till 2 a.m so i did go to chat gpt claude is usually
• 5:30
  when i'm writing or
• 5:31
  trying to figure things out claude is my go-to but obviously
• 5:35
  last night it was not
• 5:37
  talking to me and so i'm pretty sure that at one point or
• 5:41
  another they'll ask for
• 5:43
  SPEAKER_00: more money and
• 5:45
  SPEAKER_00: That's the great thing about checking it out right now.
• 5:48
  SPEAKER_00: You want to check that this is worth paying extra for,
• 5:52
  whereas my basic Grammarly subscription is worth it for me
• 5:55
  because I'm using it
• 5:57
  SPEAKER_00: constantly for punctuation,
• 6:01
  SPEAKER_00: for checking,
• 6:03
  SPEAKER_00: for plagiarism,
• 6:04
  SPEAKER_00: all of those things.
• 6:05
  SPEAKER_00: It's really nice add-on that at the click of a button,
• 6:08
  SPEAKER_00: I get some answers,
• 6:10
  SPEAKER_00: and I can decide what I want and don't want.
• 6:13
  SPEAKER_00: I do like that they've added a humanizer, for example.
• 6:17
  SPEAKER_00: So they have four voices that they've created.
• 6:21
  So you've got the everyday voice,
• 6:23
  the precisionist,
• 6:24
  the executive,
• 6:25
  and the scholar.
• 6:26
  SPEAKER_00: But you can create a voice.
• 6:28
  So you can use a prompt to create a voice,
• 6:31
  and then it rewrites it in that voice.
• 6:34
  So can you create that new voice using a prompt or should you
• 6:40
  train grammarly based
• 6:41
  SPEAKER_02: on the text that you already have?
• 6:43
  SPEAKER_00: So you create a prompt and...
• 6:48
  SPEAKER_00: they ask you for a few examples of what that would look like.
• 6:52
  SPEAKER_00: Oh, okay.
• 6:52
  SPEAKER_00: So you can just create a description,
• 6:56
  SPEAKER_00: but what they're saying is,
• 6:58
  SPEAKER_00: they're saying paste 200 words or more,
• 7:01
  so there's robustness enough there that has a style that then
• 7:07
  they can borrow from.
• 7:08
  SPEAKER_01: Okay.
• 7:09
  SPEAKER_01: Yeah, that's interesting.
• 7:11
  Which is very much what I try to do when I'm writing
• 7:15
  recommendation letters.
• 7:17
  SPEAKER_00: Yes, many of them are at least starting with AI.
• 7:21
  SPEAKER_00: Because AI is better than me at writing these letters.
• 7:24
  SPEAKER_00: This is something,
• 7:24
  SPEAKER_00: by the way,
• 7:25
  I noticed Ethan Oleg wrote about recommendation letters and
• 7:29
  about the fact that
• 7:29
  SPEAKER_00: he's trying to write them by hand.
• 7:32
  SPEAKER_00: Or not by hand, but himself without AI.
• 7:36
  SPEAKER_00: And he...
• 7:37
  SPEAKER_00: admitted that he thinks it might even hurt his students.
• 7:40
  I don't think so because his name,
• 7:42
  his signature at the end of the page will indicate something.
• 7:46
  But as somebody who's always struggled to write very good
• 7:50
  recommendation letters
• 7:51
  SPEAKER_00: and would agonize for days,
• 7:52
  SPEAKER_00: the ability to do that quickly,
• 7:55
  SPEAKER_00: and I'll be honest,
• 7:57
  it's often better than me at drawing at the main details and
• 8:00
  connecting the
• 8:01
  SPEAKER_00: paragraphs and all of that.
• 8:03
  SPEAKER_00: that I think it's a great tool.
• 8:06
  I always check it,
• 8:07
  and I always add to it,
• 8:08
  and I'm very careful to even direct it.
• 8:10
  SPEAKER_00: So this is something very similar.
• 8:12
  SPEAKER_00: And I have a very strong voice.
• 8:14
  I created a very strong voice inside Cloud for recommendations
• 8:18
  because I've written
• 8:19
  SPEAKER_00: over 400 of them.
• 8:20
  So I could give all of them to Cloud and say,
• 8:22
  create something similar to this.
• 8:24
  SPEAKER_00: So there is my voice in there.
• 8:27
  That is very different,
• 8:28
  though,
• 8:28
  from people who are starting to use it before...
• 8:32
  SPEAKER_00: Okay, yeah.
• 8:46
  SPEAKER_02: Sounds interesting.
• 8:48
  SPEAKER_00: All right.
• 8:48
  SPEAKER_00: Are we off to the novice program?
• 8:51
  SPEAKER_02: I think so, yeah.
• 8:52
  So the topic of the second part of this episode is on effects
• 8:57
  of AI on novice programmers.
• 8:59
  SPEAKER_02: The first paper that we went through is the widening gap,
• 9:02
  SPEAKER_02: the benefits and harms of generative AI for novice programmers.
• 9:07
  SPEAKER_02: The authors come from the U.S.,
• 9:09
  SPEAKER_02: Finland,
• 9:09
  SPEAKER_02: and also Ireland,
• 9:10
  SPEAKER_02: and the paper has been published in 2024.
• 9:12
  SPEAKER_02: Okay.
• 9:14
  SPEAKER_02: So this paper tackles a huge question,
• 9:17
  SPEAKER_02: and they are asking,
• 9:18
  are tools like GitHub Copilot and ChatGPT helping or hurting
• 9:23
  students who are just
• 9:25
  SPEAKER_02: learning to code?
• 9:27
  So before we talked about the details of this paper,
• 9:31
  what do you think,
• 9:31
  Kai,
• 9:32
  personally?
• 9:34
  SPEAKER_00: I am really intrigued.
• 9:35
  SPEAKER_00: This is why we chose this.
• 9:36
  SPEAKER_00: I'm intrigued by the novice problem.
• 9:38
  It's very clear to me that people that have some knowledge can
• 9:43
  really benefit from
• 9:44
  SPEAKER_00: the use of AI.
• 9:44
  SPEAKER_00: It accelerates what you can do.
• 9:46
  SPEAKER_00: It fine-tunes it sometimes.
• 9:49
  And as I said just before,
• 9:51
  if you have a lot that you've done,
• 9:52
  you can create a voice.
• 9:54
  SPEAKER_00: You can use it in multiple ways.
• 9:55
  SPEAKER_00: But if you're just entering into a topic, it's going to be really hard.
• 10:00
  SPEAKER_00: And AI gives you...
• 10:03
  SPEAKER_00: potentially a false sense of confidence of I can do this or dependence,
• 10:08
  SPEAKER_00: I can't do this without AI.
• 10:11
  SPEAKER_00: And so when you're building basic skills,
• 10:14
  SPEAKER_00: it is a real challenge because AI can do it so much better.
• 10:18
  It will always be when you're entering any new field,
• 10:21
  it can do it better than you.
• 10:22
  Yeah,
• 10:23
  so their results are also completely aligned with what you just
• 10:27
  said.
• 10:29
  SPEAKER_02: Talking about their methodology,
• 10:30
  SPEAKER_02: and I won't go long,
• 10:31
  SPEAKER_02: they kind of brought 21 novice programming students into the lab,
• 10:38
  and they use participant observation interviews and also eye
• 10:42
  tracking to see how
• 10:42
  SPEAKER_02: they are doing.
• 10:44
  SPEAKER_02: So jumping to their findings,
• 10:46
  SPEAKER_02: for students who were already on the right track and not struggling,
• 10:50
  SPEAKER_02: the AI tools were like an accelerant.
• 10:52
  They used them to quick generate code they already understood
• 10:57
  and intended to write.
• 10:59
  So they were also better at spotting and ignoring bad
• 11:03
  suggestions from AI.
• 11:05
  SPEAKER_02: But it was not the same thing for students who were struggling,
• 11:09
  SPEAKER_02: because the AI tools often made things worse.
• 11:13
  And the researchers found that Gen AI can compound the classic
• 11:16
  difficulties novices have.
• 11:18
  SPEAKER_02: For example,
• 11:19
  SPEAKER_02: a student might have a flawed mental model of the problem,
• 11:23
  SPEAKER_02: and the AI just helps them build on that flawed idea faster.
• 11:29
  SPEAKER_02: Yeah,
• 11:30
  and they also said that using AI from that perspective creates
• 11:35
  a kind of illusion
• 11:36
  SPEAKER_02: of competence for novice programmers.
• 11:39
  SPEAKER_02: And I think it's not limited to novice programmers.
• 11:42
  SPEAKER_02: It could be true about any novices in any other field.
• 11:46
  Yeah,
• 11:47
  definitely for me,
• 11:48
  the reason I use programmers is for really two reasons.
• 11:53
  SPEAKER_00: One is we have quite a bit of literature.
• 11:56
  SPEAKER_00: On programmers and what they need.
• 11:59
  SPEAKER_00: And programming was also and still is one of the best ways to use AI.
• 12:05
  SPEAKER_00: It's the most accelerating use of AI.
• 12:08
  SPEAKER_00: It's where people are making the most money.
• 12:10
  SPEAKER_00: So we know in that area it's helpful to use AI.
• 12:14
  You know,
• 12:14
  Google says between a third and half of all of their code is
• 12:18
  written by AI.
• 12:19
  SPEAKER_00: Obviously, it's working.
• 12:20
  Google does not do something that doesn't work,
• 12:22
  at least not for an extended period of time.
• 12:26
  SPEAKER_00: So obviously, it's working.
• 12:27
  SPEAKER_00: But the question is how it's impacting.
• 12:29
  SPEAKER_00: So it's a way to examine the novice problem.
• 12:32
  SPEAKER_00: But it's definitely not the only way.
• 12:34
  SPEAKER_00: And if we go to the second paper, it's got a few advantages.
• 12:38
  SPEAKER_00: One of the advantages is the sample is a little bit better,
• 12:41
  SPEAKER_00: but the second,
• 12:42
  SPEAKER_00: and they're measuring a lot more.
• 12:44
  SPEAKER_02: 69 young learners with an age range of 10 to 17.
• 12:52
  SPEAKER_02: So the topic of the paper,
• 12:54
  actually the title of the paper is Studying the Effect of AI
• 12:57
  Code Generators on
• 12:58
  SPEAKER_02: Supporting Novice Learners in Introductory Programming.
• 13:02
  SPEAKER_02: And the others come from Canada and the US.
• 13:06
  SPEAKER_02: Let me see.
• 13:07
  So the question that they are trying to answer was if we give
• 13:11
  AI code generators
• 13:13
  like OpenAI's codecs to young novices who have never written
• 13:18
  text-based code
• 13:19
  SPEAKER_02: before,
• 13:19
  SPEAKER_02: what happens?
• 13:20
  SPEAKER_02: Does it help them learn?
• 13:22
  SPEAKER_02: Does it make them independent?
• 13:24
  SPEAKER_02: Or does it hurt their ability to retain information?
• 13:27
  SPEAKER_02: Yeah.
• 13:28
  SPEAKER_02: Their methodology was interesting.
• 13:30
  SPEAKER_02: They had a kind of multi-phase study.
• 13:34
  They had an introduction phase,
• 13:36
  training phase,
• 13:37
  and also evaluation phase.
• 13:39
  SPEAKER_02: So they figured out that during the training phase,
• 13:44
  a group of students who had access to the AI performed
• 13:48
  significantly better than
• 13:49
  SPEAKER_02: the group without it.
• 13:51
  SPEAKER_02: And even during the training,
• 13:54
  SPEAKER_02: after an AI-assisted authoring task,
• 13:56
  SPEAKER_02: students had to do a modifying task without the AI.
• 14:00
  And on those manual tasks,
• 14:02
  the AI group performed just as well as the group that never had
• 14:06
  AI.
• 14:08
  But the most important finding is that the benefit wasn't the
• 14:13
  same for everyone.
• 14:14
  The researchers had given all the students a pretest to measure
• 14:19
  their prior
• 14:20
  SPEAKER_02: programming knowledge.
• 14:21
  For students who already had higher programming competency,
• 14:24
  the AI was a massive benefit.
• 14:27
  SPEAKER_02: They performed significantly better on the retention test a week later,
• 14:32
  SPEAKER_02: but it was not the same for students with lower competency levels.
• 14:37
  SPEAKER_02: The AI helped them get through the training tasks,
• 14:40
  SPEAKER_02: but did not lead to better long-term retention.
• 14:44
  And we are talking eventually about long-term retention because
• 14:48
  if there is no
• 14:49
  SPEAKER_00: long-term retention,
• 14:50
  SPEAKER_00: you can still use AI.
• 14:51
  SPEAKER_00: In a world where AI is everywhere,
• 14:53
  SPEAKER_00: you can still use AI to do everything,
• 14:55
  but you're starting from scratch with each problem because
• 14:59
  there's no learning that
• 15:00
  SPEAKER_00: generalizes.
• 15:01
  SPEAKER_00: Now, generalizing learning...
• 15:03
  SPEAKER_00: is incredibly hard.
• 15:04
  That's the holy grail of any kind of learning is if I teach you
• 15:09
  in a literature
• 15:09
  SPEAKER_00: class,
• 15:10
  will you be able to read another novel and be able to analyze
• 15:13
  it and understand it
• 15:14
  SPEAKER_00: in the same way?
• 15:15
  SPEAKER_00: That's a hard problem anyway.
• 15:16
  SPEAKER_00: But what we're saying again is what we call the Matthew effect,
• 15:21
  SPEAKER_00: and that is the richer getting richer.
• 15:22
  SPEAKER_00: Yep.
• 15:23
  SPEAKER_00: And the people at the bottom are not getting as much.
• 15:26
  SPEAKER_00: They're not being hurt.
• 15:27
  SPEAKER_00: But by the fact that others are getting richer,
• 15:30
  SPEAKER_00: it's going to be harder and harder for them to catch up.
• 15:34
  So that's another manifestation with younger learners of what
• 15:38
  the differences are.
• 15:39
  SPEAKER_02: Yep.
• 15:41
  SPEAKER_00: And do we want to talk about that?
• 15:44
  The third paper is more specific because they try to understand
• 15:50
  performance,
• 15:51
  SPEAKER_02: workload,
• 15:51
  SPEAKER_02: emotion,
• 15:52
  SPEAKER_02: and self-efficacy for novice programmers using AI code generation.
• 15:57
  And they wanted to know what happens when you give a novice
• 16:00
  programmer GitHub
• 16:02
  SPEAKER_02: Copilot,
• 16:02
  SPEAKER_02: but specifically when they are under time pressure.
• 16:05
  SPEAKER_02: Yeah.
• 16:06
  SPEAKER_02: So let me go to their findings.
• 16:10
  SPEAKER_02: So they had four highlights.
• 16:13
  SPEAKER_02: Using GitHub Copilot significantly increased programming efficiency.
• 16:18
  SPEAKER_02: Students simply got more done in the allotted time.
• 16:23
  SPEAKER_02: which is not kind of surprising.
• 16:25
  SPEAKER_02: But it also significantly reduced their mental workload and effort.
• 16:31
  This is a huge deal for preventing burnout in students who are
• 16:34
  already struggling with coding.
• 16:37
  And at first,
• 16:39
  using the AI did not significantly impact their emotion or
• 16:42
  their self-efficacy.
• 16:45
  The researchers suggest this is because the students attributed
• 16:48
  their improved
• 16:49
  SPEAKER_02: performance performance
• 16:50
  SPEAKER_02: to the AI, not to themselves.
• 16:52
  SPEAKER_02: So this part is also interesting for me.
• 16:55
  People are doing things and are accomplishing things,
• 16:57
  but they know that it's not them,
• 17:01
  it's AI.
• 17:02
  SPEAKER_00: Yeah, and...
• 17:04
  Having a reasonable growth or no growth in your self-efficacy
• 17:10
  is really important
• 17:11
  SPEAKER_00: because if you're starting to have self-efficacy that is irrational,
• 17:14
  SPEAKER_00: it wasn't you that translated this paper into Chinese.
• 17:18
  SPEAKER_00: It was the machine.
• 17:19
  SPEAKER_00: You actually did nothing.
• 17:21
  Then you don't have to have increased self-efficacy as far as
• 17:26
  Chinese is concerned.
• 17:27
  SPEAKER_00: And that's where our brains are very good to lying to us.
• 17:31
  SPEAKER_00: And it feels like we have competencies we actually don't have.
• 17:36
  SPEAKER_00: That's why in the first study,
• 17:38
  SPEAKER_00: when they checked an assignment that wasn't using AI,
• 17:43
  SPEAKER_00: it's a really important check to see how much transfer.
• 17:46
  SPEAKER_02: Yeah.
• 17:47
  SPEAKER_02: Right?
• 17:48
  happens but again this much like the first study has 17
• 17:53
  participants so and i wish
• 17:56
  we would have larger studies that would be able to stabilize
• 17:59
  for us the results and
• 18:01
  not rely on these small highly selective samples that somebody
• 18:07
  in their head in
• 18:07
  their classroom it always happens when we start research but
• 18:10
  i'm kind of eager to
• 18:12
  SPEAKER_00: get to research
• 18:13
  where we're doing this with samples varied enough and large
• 18:16
  enough that we can
• 18:16
  start generalizing the results and not just noting that this is
• 18:20
  really interesting.
• 18:21
  SPEAKER_02: Interesting, yes.
• 18:24
  SPEAKER_02: So that's it.
• 18:25
  SPEAKER_00: That's it for this episode.
• 18:26
  SPEAKER_00: So we talked about Grammarly.
• 18:29
  SPEAKER_02: And also the effect of AI on novice programmers.
• 18:32
  And the question that we're all left with is,
• 18:35
  why is Guy talking to Claude before 2 a.m.?
• 18:38
  SPEAKER_00: What are you doing?
• 18:39
  SPEAKER_00: Go to sleep.
• 18:40
  SPEAKER_00: Which is what I did.
• 18:43
  SPEAKER_00: All right.
• 18:43
  SPEAKER_00: We'll see you next time on Azadeh and Guy.
• 18:45
  SPEAKER_00: On AI.