Latest / Azadeh and Guy on AI / Critical Thinking vs. Cognitive Debt

Transcript

Critical Thinking vs. Cognitive Debt · Azadeh and Guy on AI

• 0:01
  SPEAKER_03: Hi, and welcome to Azadeh and Guy on AI.
• 0:05
  SPEAKER_03: And today we have two topics.
• 0:07
  We want to talk about the research paper that kind of raised
• 0:11
  our curiosity.
• 0:12
  SPEAKER_03: And it is a brand new paper from 2026.
• 0:17
  And one of the things that I find challenging right now is just
• 0:21
  staying up with the
• 0:22
  SPEAKER_03: literature and with the development of AI.
• 0:24
  SPEAKER_02: Exactly, yes.
• 0:25
  SPEAKER_02: Mm-hmm.
• 0:25
  And so one of the things that I think we can help is by sharing
• 0:29
  some of the
• 0:30
  research that is out there that we think is very strong and
• 0:34
  talk through it.
• 0:36
  SPEAKER_03: And then you know something that you didn't know before.
• 0:39
  SPEAKER_03: It's impossible to chase all the literature, by the way.
• 0:42
  SPEAKER_03: It is exploding just like AIs.
• 0:45
  SPEAKER_03: And so that's very exciting.
• 0:46
  SPEAKER_03: All right.
• 0:47
  SPEAKER_03: And this paper is called...
• 0:49
  SPEAKER_02: The effects of critical thinking intervention on reliance behaviors,
• 0:54
  problem solving quality and creativity during human-gen AI
• 0:58
  collaborative learning.
• 1:00
  SPEAKER_03: And it is done by a group from Singapore led by... Chen Yu Hu.
• 1:09
  SPEAKER_03: And we hope you pronounce it correctly, but if not, it's my fault.
• 1:13
  SPEAKER_03: She was really trying to find out how it's pronounced.
• 1:16
  SPEAKER_03: And colleagues.
• 1:18
  And a lot of this research is really produced by research
• 1:22
  groups,
• 1:22
  and we applaud that.
• 1:23
  SPEAKER_03: We work in research groups as well,
• 1:25
  SPEAKER_03: and it's in a highly productive way,
• 1:27
  SPEAKER_03: especially when you need to move quickly with the way AI is developing.
• 1:33
  SPEAKER_03: So we salute them.
• 1:37
  I find this really interesting because it's trying to answer
• 1:41
  the question that we
• 1:42
  all have on our mind is that idea from the famous paper that
• 1:46
  has yet to be
• 1:46
  SPEAKER_03: published about cognitive debt.
• 1:49
  SPEAKER_03: And that is,
• 1:50
  how do we prevent students from just relying blindly on AI and
• 1:54
  not doing any
• 1:55
  SPEAKER_03: critical thinking?
• 1:56
  Yeah,
• 1:56
  the other good thing about their research is that they did an
• 1:59
  intervention.
• 2:03
  I'm not saying that survey designs are bad,
• 2:06
  but doing an intervention is very powerful.
• 2:09
  SPEAKER_03: It moves us quicker.
• 2:11
  It moves us forward much quicker,
• 2:13
  especially when it's well-designed and well-powered.
• 2:16
  SPEAKER_02: Exactly.
• 2:17
  SPEAKER_03: And it seems like they've done both.
• 2:18
  SPEAKER_03: That is,
• 2:19
  SPEAKER_03: there's good documentation of the intervention itself,
• 2:21
  SPEAKER_03: which is incredibly important because without the good documentation,
• 2:26
  SPEAKER_03: you cannot follow and say, hey, this is a good idea.
• 2:30
  SPEAKER_03: I want to try it again.
• 2:31
  And it is well powered because in each condition they had over
• 2:35
  100 participants,
• 2:36
  which means that you have enough power to discover even
• 2:40
  relatively medium,
• 2:42
  SPEAKER_03: even maybe small effects on the large effect size.
• 2:46
  SPEAKER_03: And if it's any smaller than that,
• 2:48
  SPEAKER_03: then maybe you don't want to discover them because,
• 2:50
  SPEAKER_03: yeah,
• 2:50
  SPEAKER_03: you can power it up to a thousand people per
• 2:53
  SPEAKER_03: And then anything becomes significant.
• 2:56
  SPEAKER_03: But you want to be able to detect meaningful differences,
• 2:59
  SPEAKER_03: especially when the intervention time is not very long,
• 3:02
  SPEAKER_03: which is this case.
• 3:04
  SPEAKER_02: They had over 200 undergraduate students as their participants.
• 3:09
  And they were first and second year students enrolled in the
• 3:13
  university in 2023 or 2024.
• 3:17
  SPEAKER_02: So basically they started when Gen AI had emerged at that time.
• 3:22
  SPEAKER_03: And this is important that it's coming from Singapore and a technological
• 3:27
  SPEAKER_03: university because Singapore has made AI one of its goals.
• 3:32
  And there's a lot of emphasis both in public in general,
• 3:37
  but definitely in research.
• 3:40
  SPEAKER_01: Exactly.
• 3:40
  SPEAKER_03: And so what did they do in the intervention itself?
• 3:45
  SPEAKER_02: So like many interventions,
• 3:48
  SPEAKER_02: they divided their participants into an experimental and a control group.
• 3:53
  SPEAKER_02: And I think the number of participants in each group was almost similar.
• 3:58
  SPEAKER_02: So for control group,
• 4:03
  they were allowed to use AI freely without having any
• 4:07
  constraints or without having
• 4:09
  SPEAKER_02: any instructions about how to use AI.
• 4:12
  SPEAKER_02: But the other group, intervention group,
• 4:14
  SPEAKER_02: They were provided with three specific instructional scaffolds,
• 4:19
  SPEAKER_02: and they were designed to promote critical thinking.
• 4:21
  SPEAKER_02: So one of them was authentic instruction.
• 4:24
  Students completed a pre-task exercise to identify potential
• 4:28
  biases in AI-generated content.
• 4:31
  SPEAKER_02: The other scaffold was mentoring.
• 4:33
  So groups were supported by a customized GPT-based peer tutor
• 4:38
  that provided
• 4:39
  SPEAKER_02: adaptive feedback,
• 4:40
  SPEAKER_02: monitored their progress,
• 4:41
  SPEAKER_02: and promoted critical thinking through Socratic questioning.
• 4:45
  SPEAKER_02: And last one was dialogue or role play.
• 4:48
  So the students were assigned a specific stakeholder roles like
• 4:53
  administrator,
• 4:54
  SPEAKER_02: instructor,
• 4:54
  SPEAKER_02: or student to evaluate the ethical and pedagogical issues from multiple
• 5:00
  SPEAKER_02: perspectives.
• 5:01
  SPEAKER_03: And for data collection, they had two layers of data collection.
• 5:06
  SPEAKER_03: One is on the individual level, they had survey.
• 5:09
  SPEAKER_03: So they're still using survey to measure some effects,
• 5:12
  which is important because then you can get fairly efficient
• 5:15
  information from 226
• 5:15
  SPEAKER_03: students,
• 5:15
  SPEAKER_03: which is hard to do any other way.
• 5:23
  SPEAKER_03: They used those to measure trust in AI,
• 5:26
  SPEAKER_03: perceived critical thinking,
• 5:30
  SPEAKER_03: using actually a subscale that is very common,
• 5:32
  SPEAKER_03: the MSLQ,
• 5:35
  which is a great instrument that has been validated multiple
• 5:38
  times and being used
• 5:39
  SPEAKER_03: here a little bit differently.
• 5:40
  SPEAKER_03: And the AI reliance behavior,
• 5:42
  SPEAKER_03: so whether the mean reflection,
• 5:45
  SPEAKER_03: the reflective,
• 5:45
  SPEAKER_03: cautious,
• 5:46
  SPEAKER_03: collaborative,
• 5:48
  SPEAKER_03: or thoughtless.
• 5:49
  And so the idea is how much are they interacting and how much
• 5:53
  they are just relegating.
• 5:54
  So back to that idea of cognitive debt,
• 5:57
  I'm just letting the AI do everything.
• 5:59
  SPEAKER_02: Exactly.
• 6:02
  SPEAKER_02: So how about the results?
• 6:05
  SPEAKER_03: So before the results,
• 6:07
  they used a number of different procedures for the different
• 6:11
  pieces of data.
• 6:12
  SPEAKER_03: So they used regression analysis for the surveys.
• 6:15
  SPEAKER_03: They used content analysis to look at those joint discussions.
• 6:23
  SPEAKER_03: So the group level had problem solving problems.
• 6:27
  SPEAKER_03: And they did it in groups.
• 6:31
  SPEAKER_03: And so there aren't 200 and something.
• 6:33
  There are 38 problem solving reports and 28 chat histories that
• 6:39
  help show exactly
• 6:41
  SPEAKER_03: how they used this specific GPT.
• 6:45
  SPEAKER_02: They also used creativity assessment.
• 6:48
  So the creativity of the group reports was measured using a
• 6:52
  hybrid approach.
• 6:53
  Computational text mining tools assessed semantic originality
• 6:58
  and idea density,
• 7:00
  SPEAKER_02: while human writers evaluated the novelty of the ideas and the writing.
• 7:04
  And I love this multifaceted approach because we are seeing
• 7:09
  these people who are
• 7:10
  SPEAKER_03: interested in AI are also using AI in research,
• 7:14
  SPEAKER_03: but not exclusively.
• 7:15
  They're leaving the human in the loop as part and actually in
• 7:21
  this case orthogonal.
• 7:22
  So they're not supervising as much as they are just providing
• 7:27
  more information.
• 7:28
  SPEAKER_02: And also AI adoption rates.
• 7:31
  So the researchers calculated the cosine similarity between the
• 7:37
  final group reports
• 7:38
  SPEAKER_02: and the AI chat transcripts.
• 7:40
  SPEAKER_03: So what you want is as much difference as possible.
• 7:43
  SPEAKER_03: So you're using it, but you're not overly relying to it.
• 7:46
  SPEAKER_03: So a perfect difference.
• 7:48
  a perfect correlation between the chat result and the paper
• 7:52
  that was finally the
• 7:53
  SPEAKER_03: final report would mean there's no humanly original text.
• 7:58
  SPEAKER_03: And I love the use of AI in this case as well,
• 8:01
  SPEAKER_03: because it allows you not to focus on the words themselves,
• 8:04
  SPEAKER_03: which would be a primitive way to do that,
• 8:06
  but actually about ID units and how much correlation there is
• 8:10
  between the ideas.
• 8:11
  SPEAKER_03: So this is really if you want the details,
• 8:14
  I think it's worth reading because there are methods here that
• 8:19
  if you're interested
• 8:20
  in doing that kind of research or interested in the nitty
• 8:23
  gritty of research,
• 8:24
  SPEAKER_03: which I personally love,
• 8:26
  SPEAKER_03: this is something to go back to and see how exactly they've done it.
• 8:31
  SPEAKER_03: So let's see what they found.
• 8:32
  SPEAKER_03: What did they find about creativity?
• 8:35
  So the first finding goes back to the survey that they
• 8:38
  administered and they
• 8:41
  believe that there was not a lot of difference between the
• 8:44
  experimental and control
• 8:45
  SPEAKER_02: group.
• 8:46
  SPEAKER_02: And they had explanation for that, right?
• 8:48
  SPEAKER_03: Yeah,
• 8:49
  SPEAKER_03: they at least speculated that the intervention,
• 8:52
  SPEAKER_03: because really they spent about two hours with students.
• 8:55
  SPEAKER_03: Two hours is not a very long time.
• 8:57
  SPEAKER_03: And so what they are saying,
• 8:58
  SPEAKER_03: and I think we're sensing it from our research as well,
• 9:01
  SPEAKER_03: is that even one or two training sessions are not enough.
• 9:05
  SPEAKER_03: We need to be repeatedly there.
• 9:08
  SPEAKER_03: We, for example, are writing a grant right now.
• 9:10
  And one of the things that I want to make sure we're doing is
• 9:13
  we need to revisit on
• 9:15
  SPEAKER_03: a monthly basis what
• 9:16
  for about two years before we think we will have real impact on
• 9:21
  the way people use AI.
• 9:23
  SPEAKER_03: I don't know exactly how much.
• 9:25
  And that's one of the things that I'm excited if we get funded
• 9:28
  to go and measure is
• 9:31
  SPEAKER_03: when is enough enough,
• 9:32
  SPEAKER_03: right?
• 9:32
  SPEAKER_03: When will we see a significant uptick that is consistently used?
• 9:37
  SPEAKER_03: So that's something that I'm excited about.
• 9:39
  SPEAKER_03: But two hours is definitely on the low end.
• 9:42
  SPEAKER_02: Exactly.
• 9:43
  SPEAKER_02: The other finding is about the shift in students' behaviors.
• 9:47
  SPEAKER_02: So survey is a kind of self-report measure.
• 9:52
  And they said that they didn't feel different,
• 9:54
  but their behavior absolutely changed.
• 9:57
  SPEAKER_02: And the intervention group engaged in less thoughtless use of AI.
• 10:01
  SPEAKER_02: They used AI more thoughtfully.
• 10:03
  SPEAKER_03: Yes.
• 10:03
  SPEAKER_03: So for example, and they showed a lot more semantic originality.
• 10:08
  SPEAKER_03: So that's again,
• 10:09
  going back to that idea of how different are they from their
• 10:13
  original text and ID
• 10:16
  SPEAKER_03: density was higher for the experimental group.
• 10:22
  SPEAKER_03: And that's, that's important.
• 10:24
  SPEAKER_03: And there were multiple measures of that.
• 10:26
  SPEAKER_03: Um,
• 10:29
  SPEAKER_03: Some of them more than others.
• 10:31
  SPEAKER_03: So that's that's really something that you need to look at the detail.
• 10:35
  SPEAKER_03: We're not going to talk about all of that here,
• 10:37
  SPEAKER_03: but definitely pick it up and look at it if you're interested.
• 10:40
  SPEAKER_03: And then the novelty scores,
• 10:44
  the human writers evaluated the novelty scores and the
• 10:47
  experimental group scored
• 10:50
  SPEAKER_03: significantly higher.
• 10:52
  SPEAKER_03: And remember, this is with a two-hour training.
• 10:54
  SPEAKER_03: This is the amazing thing.
• 10:56
  SPEAKER_03: We can see an impact almost immediately.
• 10:59
  SPEAKER_03: Now, these are products that are produced immediately.
• 11:01
  SPEAKER_03: And even in my class,
• 11:02
  SPEAKER_03: when we're using AI,
• 11:03
  SPEAKER_03: we see when we prompt them to use better prompts,
• 11:07
  SPEAKER_03: they do that.
• 11:08
  SPEAKER_03: And the result is better.
• 11:09
  SPEAKER_03: But the question is, is it sustained?
• 11:12
  SPEAKER_01: Is the real question.
• 11:13
  SPEAKER_03: And what we're seeing is...
• 11:14
  SPEAKER_03: Our students revert quickly to old bad practices,
• 11:18
  SPEAKER_03: short prompts without enough detail,
• 11:21
  SPEAKER_03: and then they're getting crap in return.
• 11:23
  SPEAKER_03: Crap, by the way, is an official research term.
• 11:28
  So human critical thinking is the essential ingredient for true
• 11:33
  innovation based on
• 11:35
  SPEAKER_02: what they found.
• 11:37
  SPEAKER_03: And really,
• 11:37
  the direct adaption of AI content was significantly lower in
• 11:41
  the experimental
• 11:42
  SPEAKER_03: group.
• 11:43
  So they were doing the hard work of working with the AI and not
• 11:47
  just saying,
• 11:48
  SPEAKER_03: well,
• 11:48
  SPEAKER_03: the AI has solved it.
• 11:49
  SPEAKER_02: Exactly.
• 11:50
  SPEAKER_02: Accepting what it says, yes.
• 11:52
  SPEAKER_03: So that's what we learned from this research study.
• 11:55
  SPEAKER_03: Please pick it up.
• 11:57
  SPEAKER_03: Computers and Education is a great journal.
• 12:00
  SPEAKER_03: Lots of things about AI.
• 12:01
  SPEAKER_03: It's one of the places I like going to.
• 12:03
  SPEAKER_03: I review for them.
• 12:04
  SPEAKER_03: So please, if you want to stay up to date, this is one of the places.
• 12:12
  SPEAKER_03: All right.
• 12:12
  SPEAKER_03: Do we want to delve into the second delve, by the way, as an AI word?
• 12:16
  SPEAKER_03: That's why I used it.
• 12:17
  SPEAKER_03: And do you want to delve into the second topic?
• 12:19
  SPEAKER_03: Because we have a bone of contention between us.
• 12:23
  SPEAKER_03: I have thoughts and Azadeh thinks I'm prematurely thinking them.
• 12:30
  SPEAKER_03: So I'll start because I had the thoughts.
• 12:33
  So I've been reading everything about AI and coding and I'm
• 12:41
  interested in coding
• 12:42
  SPEAKER_03: just because,
• 12:43
  SPEAKER_03: well,
• 12:43
  SPEAKER_03: I'm interested in computer science.
• 12:45
  SPEAKER_02: You have been vibe coding for a while, right?
• 12:46
  SPEAKER_02: Yeah,
• 12:46
  SPEAKER_03: and I've been vibe coding for a while with Claude,
• 12:49
  SPEAKER_03: co-work,
• 12:50
  SPEAKER_03: love it,
• 12:52
  SPEAKER_03: wrote about it on Substack and other places.
• 12:55
  SPEAKER_03: It's delightful.
• 12:56
  SPEAKER_01: But...
• 12:58
  SPEAKER_03: I'm also thinking,
• 12:59
  SPEAKER_03: and this is not an original thought,
• 13:01
  SPEAKER_03: this is a thought that others have had,
• 13:03
  is whatever is happening in coding right now may be what is
• 13:08
  going to happen to
• 13:10
  SPEAKER_03: either,
• 13:10
  SPEAKER_03: to the next wave of information-based professions.
• 13:17
  So whatever's happening in coding now will come eventually to
• 13:21
  other jobs in the market.
• 13:23
  SPEAKER_03: Now, this is a big proposition.
• 13:25
  SPEAKER_03: This is the leap of faith you've got to take.
• 13:28
  SPEAKER_03: So if you're saying,
• 13:29
  SPEAKER_03: no,
• 13:29
  SPEAKER_03: this is going to stop at the coding door and not go anywhere else,
• 13:34
  SPEAKER_03: then I will be wrong.
• 13:36
  SPEAKER_02: I completely agree.
• 13:37
  SPEAKER_02: Yeah.
• 13:37
  SPEAKER_02: That I'm going to be wrong.
• 13:40
  No.
• 13:40
  You can,
• 13:41
  because we started having that argument and I said,
• 13:44
  let's save it for the podcast.
• 13:46
  SPEAKER_03: So I think that one of the places that we might see it is in research,
• 13:53
  SPEAKER_03: because right now we have researchers,
• 13:55
  we have different ranking of universities and we have research
• 13:58
  heavy universities
• 13:59
  SPEAKER_03: and research light universities.
• 14:01
  SPEAKER_03: And I,
• 14:02
  SPEAKER_03: If this same cycle,
• 14:04
  SPEAKER_03: because the cycle,
• 14:05
  SPEAKER_03: and I didn't explain what the cycle is,
• 14:06
  SPEAKER_03: but we've had AI produce code for a while now.
• 14:11
  SPEAKER_03: For the last year,
• 14:12
  SPEAKER_03: year and a half,
• 14:13
  a growing percentage of the coding that is done in the world is
• 14:18
  done by AI.
• 14:19
  SPEAKER_03: There are multiple platforms that do this well,
• 14:22
  SPEAKER_03: whether it's lovable or code code,
• 14:25
  SPEAKER_03: a cloud code or whatever.
• 14:29
  SPEAKER_03: They're everywhere.
• 14:32
  SPEAKER_03: And they have been doing that.
• 14:34
  SPEAKER_03: And humans have been verifying them.
• 14:36
  And that has become a bottleneck because AI can produce a lot
• 14:40
  more code than people can check.
• 14:42
  And lately,
• 14:43
  Claude and others have been producing agents that can check the
• 14:47
  code.
• 14:48
  SPEAKER_02: Okay.
• 14:49
  SPEAKER_03: If that becomes good, then the human is out of the loop.
• 14:53
  The human just generates idea and goes to have a cup of coffee
• 14:58
  and has a working
• 15:00
  prototype that keeps getting fixed because we have review
• 15:03
  agents that can look at
• 15:04
  SPEAKER_03: the code,
• 15:05
  SPEAKER_03: say what's working,
• 15:05
  SPEAKER_03: what's not working,
• 15:07
  SPEAKER_03: and fixes them.
• 15:07
  SPEAKER_03: Now, not even all coders believe it's possible.
• 15:11
  SPEAKER_01: Mm-hmm.
• 15:12
  SPEAKER_03: But a growing number does.
• 15:14
  Now,
• 15:14
  we always have to separate because we're working with
• 15:17
  engineers,
• 15:18
  right?
• 15:18
  SPEAKER_03: And we've started talking about the main problem,
• 15:22
  SPEAKER_03: and that is that it threatens our professional ego,
• 15:28
  SPEAKER_03: right?
• 15:28
  SPEAKER_02: Yeah.
• 15:29
  SPEAKER_03: It goes to the heart of that.
• 15:30
  So when people say,
• 15:31
  no,
• 15:32
  it can't,
• 15:33
  I want to see empirical evidence,
• 15:34
  not your gut feeling.
• 15:36
  SPEAKER_01: Yeah.
• 15:36
  SPEAKER_03: Because my gut feeling is nobody can replace me.
• 15:40
  SPEAKER_03: I am the best whatever profession is.
• 15:43
  SPEAKER_03: And a machine can never beat me.
• 15:45
  SPEAKER_02: Yeah.
• 15:46
  SPEAKER_03: But maybe it can.
• 15:47
  SPEAKER_02: And if we take the ego out of it... That's where I'm not... Yeah.
• 15:52
  SPEAKER_02: Maybe.
• 15:53
  SPEAKER_02: Because we... Yeah.
• 15:54
  SPEAKER_03: Go ahead.
• 15:55
  SPEAKER_02: We have tried qualitative coding with AI before.
• 15:59
  SPEAKER_02: And to be honest, we were not 100% satisfied with the output.
• 16:03
  SPEAKER_03: Yeah.
• 16:04
  SPEAKER_02: So...
• 16:05
  SPEAKER_03: And so that is definitely true.
• 16:07
  So my suggestion is that research universities will do less and
• 16:11
  less research or
• 16:12
  there will be probably less research universities and less
• 16:16
  researchers because they
• 16:17
  SPEAKER_03: would be able to use AI,
• 16:20
  SPEAKER_03: an army of agents,
• 16:21
  SPEAKER_03: a swarm of agents.
• 16:23
  SPEAKER_02: So you mean there is no human in the loop?
• 16:25
  SPEAKER_02: AI is doing all... Almost all of the work.
• 16:28
  SPEAKER_03: Humans summarize it and serves it.
• 16:31
  SPEAKER_03: Kind of the next...
• 16:31
  SPEAKER_03: And I don't know how long that is,
• 16:33
  SPEAKER_03: but human has to generate some of the ideas.
• 16:36
  SPEAKER_03: We're not... We don't... I think that that is the portion.
• 16:40
  SPEAKER_03: But I think that we may get to the point where...
• 16:43
  SPEAKER_03: To produce research programs,
• 16:45
  we would need maybe one-tenth of the people producing research
• 16:48
  right now because
• 16:49
  right now the constraint is the speed of writing and the speed
• 16:55
  of data collection
• 16:58
  SPEAKER_03: and the speed of ideating.
• 17:00
  SPEAKER_03: Well,
• 17:01
  SPEAKER_03: you still need to ideate,
• 17:02
  but you have a machine that can help you do that faster and
• 17:06
  build a research
• 17:06
  SPEAKER_03: program out of it.
• 17:08
  And you can run experiments much faster depending on how you
• 17:11
  collect the data.
• 17:12
  SPEAKER_03: And so in writing,
• 17:14
  SPEAKER_03: I mean,
• 17:15
  SPEAKER_03: I know some people are attached to writing as thinking,
• 17:17
  SPEAKER_03: but there are sometimes,
• 17:18
  many times in science where I have concrete results and I just
• 17:23
  need to describe
• 17:24
  SPEAKER_03: them so somebody can build the next rung.
• 17:26
  SPEAKER_02: Yeah.
• 17:27
  SPEAKER_03: And that's where maybe there's a need for a human to do a little bit of
• 17:32
  SPEAKER_03: interpretation,
• 17:33
  SPEAKER_03: kind of cherry on top,
• 17:34
  SPEAKER_03: conclusions and expansions and what comes next.
• 17:37
  SPEAKER_03: But the majority of the work can be done, right?
• 17:40
  I mean,
• 17:41
  I think about how technical a method section is and a result
• 17:44
  section is.
• 17:45
  And even some of the discussion can be,
• 17:48
  here's what other researchers said.
• 17:50
  SPEAKER_03: This is what we added to the information.
• 17:52
  And I can see how that can become a cycle where I need a lot
• 17:58
  less researchers out there.
• 18:00
  SPEAKER_03: Now, universities also do teaching and service and other things.
• 18:03
  SPEAKER_03: So it's not like we will have nothing to do.
• 18:05
  SPEAKER_02: Yeah.
• 18:07
  SPEAKER_02: I'm still thinking about that.
• 18:10
  Especially,
• 18:10
  you suggested that we have a separate agent that reviews the
• 18:16
  code.
• 18:17
  SPEAKER_02: Yes.
• 18:17
  SPEAKER_02: But it can review the code from multiple perspectives.
• 18:21
  SPEAKER_02: I mean, I'm talking about programming.
• 18:23
  SPEAKER_02: So for programming,
• 18:24
  SPEAKER_02: we are talking about memory management,
• 18:26
  SPEAKER_02: I don't know,
• 18:27
  SPEAKER_02: time management,
• 18:28
  SPEAKER_02: efficiency of the code.
• 18:30
  SPEAKER_02: So basically, the agent can come up with several different plans for
• 18:35
  SPEAKER_02: for improving the quality of the code.
• 18:37
  But who is going to decide which one is the best plan for that
• 18:41
  specific project?
• 18:43
  SPEAKER_03: Well, nobody is deciding.
• 18:45
  SPEAKER_03: The agents are deciding.
• 18:47
  SPEAKER_03: They're iterating.
• 18:48
  SPEAKER_03: They're trying that out in short increments.
• 18:51
  One of the things that has emerged fairly recently is very
• 18:55
  short incremental,
• 18:56
  SPEAKER_03: like evolution,
• 18:58
  SPEAKER_03: is you create the next version.
• 19:00
  SPEAKER_03: Is it performing better?
• 19:01
  SPEAKER_03: If it's not performing better, you go back to the beginning.
• 19:03
  SPEAKER_03: If it's performing better, you said it's good.
• 19:06
  SPEAKER_03: What is the next problem to solve?
• 19:07
  SPEAKER_03: So what it's taking is it's taking shorter pieces of the problem,
• 19:12
  SPEAKER_03: identifying them,
• 19:13
  SPEAKER_03: solving them.
• 19:15
  SPEAKER_03: Again,
• 19:16
  SPEAKER_03: if the output is improved,
• 19:20
  SPEAKER_03: whether you decide efficiency,
• 19:22
  SPEAKER_03: number of,
• 19:23
  SPEAKER_03: you know,
• 19:24
  SPEAKER_03: you can decide what the criteria are.
• 19:26
  And the one problem with that is that some things are really
• 19:31
  easy to quantify.
• 19:32
  So it's easy for an agent to decide if it's gotten better or
• 19:35
  not based on the correction.
• 19:37
  SPEAKER_03: In research, that's not always possible.
• 19:39
  SPEAKER_01: Mm-hmm.
• 19:40
  So the more squishy the result is,
• 19:46
  the harder it is for AI to have a scale.
• 19:54
  SPEAKER_03: But I think it's possible.
• 19:55
  SPEAKER_03: And that's exactly where I have a soft belly here.
• 19:59
  SPEAKER_02: So maybe it's possible, but not in every single area of research.
• 20:03
  SPEAKER_02: For example...
• 20:05
  SPEAKER_02: If you're talking about it like this,
• 20:07
  SPEAKER_02: maybe in programming it could be easily applied,
• 20:09
  but maybe in other parts of the research when critical thinking
• 20:13
  and innovation
• 20:14
  SPEAKER_02: matters a lot.
• 20:16
  SPEAKER_02: I'm not sure about that.
• 20:18
  SPEAKER_03: So I don't know either.
• 20:20
  SPEAKER_03: And I think that...
• 20:23
  SPEAKER_03: We will have to develop some markers about how education,
• 20:27
  SPEAKER_03: how in our case,
• 20:28
  SPEAKER_03: educational research is done.
• 20:30
  SPEAKER_01: Yeah.
• 20:30
  SPEAKER_03: But we have concrete outcomes.
• 20:32
  SPEAKER_03: We can say, can kids do this better?
• 20:35
  SPEAKER_03: Right.
• 20:36
  SPEAKER_03: And does this design work or not work fairly efficiently?
• 20:40
  SPEAKER_03: And especially with our digital platforms,
• 20:42
  SPEAKER_03: building curriculum is definitely faster right around an idea.
• 20:46
  SPEAKER_03: And so I am I'm just wondering out loud.
• 20:50
  SPEAKER_03: I don't know.
• 20:50
  But I think that we might face a situation where our jobs at
• 20:56
  some universities,
• 20:57
  SPEAKER_03: not all universities,
• 20:58
  some universities will be those bastions of research and
• 21:02
  everybody in them will do
• 21:03
  SPEAKER_03: maybe even more research because they will be 10 times more productive.
• 21:07
  SPEAKER_01: Mm hmm.
• 21:08
  But in other places,
• 21:09
  the role may revert to let's teach everybody about AI.
• 21:14
  SPEAKER_03: Let's just teach our courses really, really well.
• 21:16
  SPEAKER_02: There's nothing wrong with that.
• 21:17
  SPEAKER_02: Yeah, yeah, yeah.
• 21:18
  SPEAKER_03: Right.
• 21:19
  SPEAKER_03: And we have not found the multiplier for education.
• 21:24
  SPEAKER_03: It may be that education is one of those places where AI can help people.
• 21:29
  SPEAKER_01: Mm hmm.
• 21:30
  SPEAKER_03: But at the edges.
• 21:31
  SPEAKER_01: Yeah.
• 21:32
  SPEAKER_03: But to this point,
• 21:34
  technology has helped the technology has not revolutionized
• 21:37
  education as far as the
• 21:39
  SPEAKER_03: need for people to do their job.
• 21:42
  SPEAKER_01: Yeah.
• 21:43
  SPEAKER_03: And so it may be that we come to that,
• 21:44
  SPEAKER_03: but that's actually a much less solvable problem,
• 21:47
  SPEAKER_03: I would argue,
• 21:48
  SPEAKER_03: than being able to have a research production.
• 21:52
  SPEAKER_03: Anyway, that's where my brain is.
• 21:55
  SPEAKER_03: Azadeh tells me, I'm full of it.
• 21:57
  SPEAKER_03: It's just premature.
• 21:59
  SPEAKER_03: But I'm trying to imagine the future as I think about,
• 22:02
  SPEAKER_03: for example,
• 22:03
  SPEAKER_03: as we're graduating people into the world,
• 22:06
  SPEAKER_03: what do I need to make sure that our doctoral students know?
• 22:10
  And I think knowing AI will be the only chance to be a
• 22:14
  researcher in this new world
• 22:16
  SPEAKER_03: and also to make sure that many of them know that
• 22:19
  SPEAKER_03: that they are much more likely to teach.
• 22:21
  SPEAKER_03: And that is true even now, but that very few of them will do research.
• 22:27
  And it might be that that pyramid gets much,
• 22:29
  much narrower because we can do so much with AI.
• 22:34
  SPEAKER_03: So I think that cycle may be coming sooner than later.
• 22:39
  I know that there are a number of labs that work exclusively on
• 22:43
  the idea of
• 22:44
  SPEAKER_03: creating AI that improves science.
• 22:45
  SPEAKER_02: Okay.
• 22:47
  SPEAKER_02: Yeah.
• 22:49
  SPEAKER_03: All right.
• 22:52
  SPEAKER_03: I thought I would get more fire here.
• 22:53
  A little bit more pushback will be because I need to think
• 23:03
  through these things.
• 23:04
  SPEAKER_03: And I appreciate the pushback.
• 23:09
  SPEAKER_02: Yeah.
• 23:09
  SPEAKER_03: Until next time.
• 23:11
  SPEAKER_02: See you.
• 23:12
  SPEAKER_03: On Azadeh.
• 23:13
  SPEAKER_02: And Guy.
• 23:13
  SPEAKER_03: On AI.