Latest / Azadeh and Guy on AI / New Studies on AI, Emotions, and Student Learning Gains

Transcript

New Studies on AI, Emotions, and Student Learning Gains · Azadeh and Guy on AI

• 0:02
  SPEAKER_00: Hi and welcome to episode two of season three of Azadeh and Guy on AI.
• 0:09
  SPEAKER_00: That's fantastic.
• 0:11
  SPEAKER_00: All right.
• 0:12
  SPEAKER_00: And today we're really interested.
• 0:14
  SPEAKER_00: We got a few papers that were just published and they are all,
• 0:20
  all three of them are in some way involved with emotions while
• 0:26
  using AI or the way
• 0:28
  SPEAKER_00: you interact with AI in an emotional way.
• 0:32
  SPEAKER_00: And for most of you,
• 0:35
  SPEAKER_00: you probably know,
• 0:35
  I'm really interested in emotions and how emotions are
• 0:38
  connected to achievements or
• 0:41
  SPEAKER_00: achievement emotions.
• 0:43
  SPEAKER_01: And we have published episodes on emotions.
• 0:46
  So this is really a fascinating area that I think is important
• 0:50
  to keep looking at.
• 0:52
  SPEAKER_00: There's this claim.
• 0:54
  And this conversation around whether AI actually has emotions,
• 0:58
  and I think the answer is not.
• 1:00
  SPEAKER_00: But we as humans have emotions,
• 1:03
  SPEAKER_00: and our brains are trained to see emotions in other things,
• 1:06
  SPEAKER_00: whether they're real or not.
• 1:08
  SPEAKER_01: Exactly.
• 1:08
  SPEAKER_01: We interpret AI's output as emotional.
• 1:11
  SPEAKER_00: Yes.
• 1:12
  SPEAKER_00: And it's built to evoke that because it's imitating us.
• 1:17
  SPEAKER_00: But at least right now it's imitating us.
• 1:19
  When does it stop imitating and being truly autonomous in that
• 1:23
  way is unclear.
• 1:25
  SPEAKER_00: It could be that the answer is never.
• 1:27
  SPEAKER_00: If you ask the Pope last week, he said definitely never.
• 1:31
  SPEAKER_00: It's just mimicking.
• 1:32
  But there will be other people who claim like that engineer out
• 1:36
  of Google who said,
• 1:37
  SPEAKER_00: I think it's alive a couple of years ago already.
• 1:41
  SPEAKER_00: And what is clear is even if it would be,
• 1:45
  SPEAKER_00: if we would get to that moment,
• 1:47
  it'll be hard for us as people to recognize the difference
• 1:50
  because we're already
• 1:51
  SPEAKER_00: giving it a sense of agency.
• 1:53
  SPEAKER_01: Yeah, yeah.
• 1:54
  And so it's not we're not interpreting it because we're
• 1:57
  under-interpreting its
• 1:59
  SPEAKER_00: agency,
• 2:00
  SPEAKER_00: but in some ways because we over-interpret this.
• 2:03
  SPEAKER_00: And I can't tell you how many times,
• 2:05
  SPEAKER_00: for example,
• 2:05
  SPEAKER_00: people just this afternoon,
• 2:08
  SPEAKER_00: I had an interaction with a colleague who...
• 2:11
  Jokingly,
• 2:11
  but not jokingly,
• 2:13
  it talked about the fact that she thinks technology hates her.
• 2:17
  SPEAKER_00: And it's like,
• 2:18
  SPEAKER_00: again,
• 2:18
  SPEAKER_00: we're using this emotional language about things that we definitely know,
• 2:22
  SPEAKER_00: you know,
• 2:23
  SPEAKER_00: an iPad has zero emotions towards you.
• 2:26
  SPEAKER_00: You might have lots of emotions towards it.
• 2:29
  SPEAKER_00: But it's a way we look at the world as people.
• 2:31
  SPEAKER_00: So this is important to explore.
• 2:33
  SPEAKER_00: And that's why I'm happy these three papers that are very, very recent.
• 2:38
  One note that I always want to go back to is I personally
• 2:42
  refuse to deal with
• 2:44
  SPEAKER_00: pre-publication articles because
• 2:48
  SPEAKER_00: They have not been peer reviewed yet.
• 2:51
  SPEAKER_00: And I think peer review is important.
• 2:53
  SPEAKER_00: So talking prematurely,
• 2:55
  because if you think about the famous Stanford study that
• 2:58
  talked about the fact
• 2:59
  SPEAKER_00: that AI or MIT,
• 3:02
  SPEAKER_00: I don't remember,
• 3:03
  SPEAKER_00: so don't hold me to it.
• 3:05
  SPEAKER_00: But there was this huge discussion about how it's reducing activity, but
• 3:15
  SPEAKER_00: But it's important.
• 3:15
  SPEAKER_00: It's really important to remember that that was a preprint,
• 3:20
  which means it has not been pre-reviewed and it has not been
• 3:22
  published yet.
• 3:23
  SPEAKER_00: So it makes me think about it.
• 3:25
  SPEAKER_00: There was quite a bit of critique of the methodology and other things.
• 3:28
  SPEAKER_00: So we're dealing with papers that have been printed.
• 3:31
  SPEAKER_00: And in this case, at least all three of them are also publicly
• 3:36
  You can start reading tomorrow So let's go with the first one
• 3:47
  Because I find this
• 3:49
  SPEAKER_00: really,
• 3:49
  SPEAKER_01: really interesting So the first paper,
• 3:52
  SPEAKER_01: as Guy said,
• 3:53
  has been published just this year The title of the paper is
• 3:56
  Decoding Student
• 3:57
  SPEAKER_01: Chatbot Dialogues How Interaction Structure is Associated with
• 4:02
  SPEAKER_01: Learning Gains in AI-Assisted Programming.
• 4:05
  SPEAKER_01: It's been published in AI in Education,
• 4:09
  SPEAKER_01: and the authors are Kor and Kapoor,
• 4:12
  SPEAKER_01: if I'm pronouncing their names correctly.
• 4:14
  SPEAKER_00: And if not, we apologize.
• 4:16
  SPEAKER_01: Yeah.
• 4:17
  SPEAKER_01: So if you want to talk about the methodology of that paper,
• 4:21
  SPEAKER_01: they employed an exploratory quasi-experimental pre-post design,
• 4:27
  SPEAKER_01: and the students completed an unassisted pre-test
• 4:32
  SPEAKER_01: Then they engaged in chatbot-supported programming tasks,
• 4:35
  SPEAKER_01: and then they completed an unassisted post-test.
• 4:40
  SPEAKER_01: Then the transcripts,
• 4:42
  SPEAKER_01: I mean the chatbot interactions were analyzed using two methods,
• 4:48
  SPEAKER_01: epistemic network analysis and also descriptive sentiment analysis.
• 4:53
  And what's important here on top of that to know is we're
• 4:57
  dealing with secondary students.
• 4:59
  SPEAKER_00: Yeah.
• 4:59
  SPEAKER_00: And this is Singapore.
• 5:00
  And the reason it's interesting is Singapore has made AI a goal
• 5:07
  for all of its schools.
• 5:09
  And there's a lot of research and experimentation coming out of
• 5:12
  Singapore.
• 5:13
  SPEAKER_00: So this is immensely fascinating.
• 5:17
  SPEAKER_00: As they looked into it, what is interesting is theoretically...
• 5:23
  SPEAKER_00: And they talk from a very much a learning perspective.
• 5:27
  So they talk about cognitive load,
• 5:29
  constructivist theory,
• 5:30
  and zone of proximal development.
• 5:33
  SPEAKER_00: All great concepts.
• 5:34
  What is really interesting is because they come from computer
• 5:38
  science on the
• 5:38
  SPEAKER_00: sentiment analysis,
• 5:40
  SPEAKER_00: they use these boxes that were built a long time ago.
• 5:44
  Probably the first publicly available AIs were the sentiment
• 5:49
  analysis AIs.
• 5:53
  And the thing that is always a little bit harder to me is when
• 5:59
  they do that work,
• 6:00
  SPEAKER_00: they don't connect it very strongly to theoretical
• 6:06
  SPEAKER_00: Understanding of emotions.
• 6:08
  SPEAKER_00: So while they're looking at whether people are positive or negative,
• 6:11
  SPEAKER_00: they're not really going any deeper.
• 6:13
  And there's a lot of research about emotions and maybe it's
• 6:17
  time to go beyond is it
• 6:19
  SPEAKER_00: negative or positive to a more diverse,
• 6:22
  SPEAKER_00: multi-factor approach to emotions.
• 6:25
  SPEAKER_00: That being said,
• 6:26
  I think the study is still good because it includes that as
• 6:29
  part of a larger set of
• 6:34
  SPEAKER_00: goals within the study.
• 6:36
  SPEAKER_01: Mm-hmm.
• 6:38
  SPEAKER_01: So do we want to talk about their findings?
• 6:42
  SPEAKER_00: Well, I do want to talk briefly about the methodology.
• 6:45
  SPEAKER_00: So they really,
• 6:48
  SPEAKER_00: part of their methodology was to look at students in three buckets,
• 6:51
  SPEAKER_00: students that made no gains,
• 6:53
  SPEAKER_00: students that made gains,
• 6:54
  SPEAKER_00: and students that seem to have lost ground.
• 6:57
  SPEAKER_01: Mm-hmm.
• 6:59
  These are somewhat problematic because it is very unlikely that
• 7:04
  students have
• 7:05
  SPEAKER_00: become less knowledgeable over this period of time.
• 7:07
  SPEAKER_00: Exactly.
• 7:07
  SPEAKER_01: I was thinking about the same thing.
• 7:09
  SPEAKER_01: So just by being part of that study, you will learn something.
• 7:13
  SPEAKER_01: So learning loss was kind of...
• 7:16
  SPEAKER_00: Yes.
• 7:16
  SPEAKER_00: It's like, so has the learning activity made you stupider?
• 7:21
  That's a problematic proposition and that's something that
• 7:24
  requires exploration.
• 7:25
  SPEAKER_00: But even if you put into the bucket people
• 7:28
  SPEAKER_00: People who didn't make any progress or people who have made even loss.
• 7:34
  SPEAKER_00: That would be a bigger bucket.
• 7:36
  SPEAKER_00: And actually it would make the two groups almost equivalent.
• 7:40
  SPEAKER_00: So that's important to know.
• 7:43
  SPEAKER_00: But what have they found about emotions?
• 7:45
  SPEAKER_00: I think that's where we want to focus.
• 7:47
  SPEAKER_01: So you just talked about those three buckets.
• 7:50
  SPEAKER_01: So students with learning gains and the number was 26.
• 7:54
  SPEAKER_01: They showed a strongly connected multi-term
• 7:58
  SPEAKER_01: ENA patterns by ENA,
• 8:00
  SPEAKER_01: as I said before,
• 8:01
  SPEAKER_01: its epistemic network analysis,
• 8:04
  SPEAKER_01: and including solution attempt,
• 8:06
  SPEAKER_01: feedback,
• 8:07
  SPEAKER_01: clarification,
• 8:08
  SPEAKER_01: and knowledge contribution.
• 8:11
  So there was a kind of iterative feedback integrated cycle that
• 8:16
  was the defining
• 8:17
  SPEAKER_01: structure feature of
• 8:18
  SPEAKER_01: Productive Engagement.
• 8:21
  SPEAKER_01: And then we have students with no gain.
• 8:23
  SPEAKER_01: The number of those students was 15.
• 8:26
  They showed tighter coupling between solution attempts and
• 8:30
  knowledge contributions
• 8:32
  SPEAKER_01: with feedback playing a secondary supportive rather than a central role.
• 8:37
  SPEAKER_01: So interaction was task-focused but less recursively linked.
• 8:42
  SPEAKER_01: And lastly, we have students with
• 8:46
  Learning Loss,
• 8:48
  so they showed feedback-rich exchanges but limited
• 8:52
  self-regulatory refinement.
• 8:56
  Feedback appeared to confirm rather than trigger deeper
• 8:59
  metacognitive adjustment.
• 9:01
  And really what we're talking about here is the fact that
• 9:06
  students that have done
• 9:08
  SPEAKER_00: well seem to have had these rich,
• 9:10
  SPEAKER_00: long conversations with clarifications,
• 9:13
  SPEAKER_00: with digging deeper and using the feedback to get more information.
• 9:19
  SPEAKER_00: So it's the engagement with the bot.
• 9:22
  SPEAKER_00: I mean, if I simplify this, what they're saying is
• 9:25
  People with learning gains were much more engaged with the bot
• 9:29
  and with the idea
• 9:30
  SPEAKER_00: that the bot is designed to go deeper.
• 9:35
  SPEAKER_00: But on the emotional stuff, we found not very much.
• 9:39
  SPEAKER_00: So all three groups were the same and there wasn't much emotion.
• 9:42
  SPEAKER_01: Okay.
• 9:44
  SPEAKER_00: And I think that's connected to the chatbot design.
• 9:47
  SPEAKER_00: That chatbot is not designed to evoke emotions.
• 9:50
  SPEAKER_00: It's designed to give you feedback and answer.
• 9:54
  SPEAKER_00: So the responses were moderately positive, but very, very low numbers.
• 10:02
  SPEAKER_00: And really, it's technical discussion.
• 10:05
  SPEAKER_00: So much like,
• 10:07
  SPEAKER_00: for example,
• 10:07
  SPEAKER_00: I think about my own use of AI,
• 10:10
  SPEAKER_00: it is almost completely devoid of emotions.
• 10:13
  SPEAKER_00: It is, can you help me summarize where I am on my project?
• 10:17
  SPEAKER_00: There's no emotion.
• 10:18
  SPEAKER_01: Yeah.
• 10:19
  SPEAKER_00: I mean,
• 10:19
  SPEAKER_00: I can be frustrated about one of the projects,
• 10:21
  SPEAKER_00: but there's no emotion in my interaction itself.
• 10:26
  SPEAKER_00: I ask questions.
• 10:27
  SPEAKER_00: I say, hey, can you give me more information about that?
• 10:29
  SPEAKER_00: When I do voice interface, there's emotion.
• 10:32
  SPEAKER_00: It's really interesting.
• 10:35
  SPEAKER_00: And because I get upset at it because it doesn't understand me.
• 10:39
  SPEAKER_00: It's my pronunciation, I think, more than anything else.
• 10:42
  SPEAKER_01: But that's very interesting, Guy.
• 10:43
  You said that you think that there are no emotions involved in
• 10:48
  your interaction
• 10:49
  SPEAKER_01: with AI,
• 10:50
  SPEAKER_01: almost no emotions.
• 10:52
  And when I talk with my friends,
• 10:54
  I mean,
• 10:55
  many of them are graduate students.
• 10:56
  SPEAKER_01: They say that they get mad at AI.
• 11:00
  SPEAKER_01: It is very sassy and cheeky.
• 11:03
  SPEAKER_01: They don't like AI responses.
• 11:05
  SPEAKER_01: So I can hear a lot of emotions.
• 11:08
  SPEAKER_01: Even when their interactions are highly technical.
• 11:13
  SPEAKER_00: That's interesting.
• 11:14
  SPEAKER_01: Yeah.
• 11:15
  SPEAKER_00: We should look into that.
• 11:18
  We've got lots of opportunities to interact,
• 11:20
  especially with students with a similar background.
• 11:23
  SPEAKER_00: I mean, they're undergraduates, but in engineering.
• 11:26
  SPEAKER_00: So that might be something to explore.
• 11:28
  SPEAKER_01: And I can remember one of the conversations.
• 11:30
  SPEAKER_01: Sorry, I'm just... No, no, no.
• 11:32
  So one of them said that it thanked Claude after solving a
• 11:37
  problem and it said,
• 11:38
  SPEAKER_01: instead of giving me compliments,
• 11:40
  SPEAKER_01: it's better for you to go and try to solve the problem yourself.
• 11:47
  SPEAKER_01: So...
• 11:48
  SPEAKER_00: Yes, which is good advice.
• 11:51
  SPEAKER_00: Partially, I think it is the large... the large...
• 11:58
  SPEAKER_00: Firmes that are doing this work,
• 11:59
  SPEAKER_00: the large companies,
• 12:01
  SPEAKER_00: and OpenAI and Claude,
• 12:03
  SPEAKER_00: especially,
• 12:03
  SPEAKER_00: I'm seeing this a little bit in Gemini as well,
• 12:07
  they have put more and more guidelines around that because of
• 12:11
  the critique,
• 12:12
  SPEAKER_00: I think,
• 12:13
  SPEAKER_00: out there that AI is trying to help students too much.
• 12:18
  SPEAKER_01: And there's a learning mode.
• 12:20
  And so I think that part of it is in the post-training,
• 12:25
  they're putting these guardrails
• 12:27
  SPEAKER_00: To try and sense if somebody is learning,
• 12:29
  trying to learn something and then making more self-regulatory
• 12:34
  suggestions.
• 12:37
  SPEAKER_00: Okay, let's go to the second one.
• 12:40
  SPEAKER_00: And the second one was published by PARC.
• 12:42
  It's how learners interpret emotion-aware feedback in
• 12:45
  AI-supported learning,
• 12:47
  SPEAKER_00: evidence from a classroom study.
• 12:49
  SPEAKER_00: And this is a study that was conducted in Korea.
• 12:55
  SPEAKER_00: And again,
• 12:55
  SPEAKER_00: a country that seems to be steeped in experimenting with AI,
• 13:01
  SPEAKER_00: using AI,
• 13:02
  SPEAKER_00: learning about AI.
• 13:04
  SPEAKER_00: And so very happy to see that happening.
• 13:10
  SPEAKER_00: Tell us a little bit about it.
• 13:11
  SPEAKER_01: So talking about the methodology of that study,
• 13:15
  SPEAKER_01: this is an exploratory classroom study and they use the between subjects
• 13:19
  SPEAKER_01: experimental design.
• 13:22
  SPEAKER_01: And let me see.
• 13:24
  So the condition-specific affective feedback was delivered by a
• 13:30
  human operator
• 13:31
  SPEAKER_01: using predefined templates at fixed interaction points.
• 13:37
  SPEAKER_01: This was about the design of the study.
• 13:39
  SPEAKER_01: The study compared three learner-facing feedback conditions.
• 13:44
  SPEAKER_01: So condition A was inference with explanation.
• 13:48
  Affective hypothesis plus brief rationale grounded in
• 13:52
  observable task behavior and
• 13:54
  SPEAKER_01: reflective prompt.
• 13:56
  SPEAKER_01: Condition B was inference without explanation,
• 13:59
  SPEAKER_01: the same affective hypothesis with no rationale or prompt.
• 14:04
  And condition C was no inference,
• 14:06
  deliberate non-reference,
• 14:08
  inference,
• 14:09
  sorry.
• 14:10
  SPEAKER_01: So it was a neutral acknowledgement with no affective claims.
• 14:15
  SPEAKER_00: And before we get into it,
• 14:18
  the one thing that I'm a little bit sad about this study is
• 14:22
  that they didn't
• 14:23
  SPEAKER_00: include an example or two of what each condition looks like.
• 14:27
  SPEAKER_00: So they explained it.
• 14:28
  SPEAKER_00: But in my brain, I'm still trying to understand what does it mean?
• 14:33
  SPEAKER_00: So give me an example.
• 14:34
  SPEAKER_00: So if you're writing a paper right now and you're using these wonderful
• 14:39
  SPEAKER_00: simulations,
• 14:40
  SPEAKER_00: in this case,
• 14:41
  SPEAKER_00: it's simulated responses.
• 14:45
  SPEAKER_00: Give us an example of what might.
• 14:46
  SPEAKER_01: Invest in your appendix section.
• 14:50
  SPEAKER_01: Or even in the text.
• 14:51
  SPEAKER_00: I love things in the text.
• 14:53
  SPEAKER_00: I'm not a huge fan of appendices, sometimes because of language.
• 14:57
  SPEAKER_00: We have to put things in appendix.
• 14:59
  SPEAKER_00: But when I'm reading,
• 15:00
  SPEAKER_00: I really don't want to go to the appendix,
• 15:02
  SPEAKER_00: although in this one I did go to go and see if it's there.
• 15:06
  SPEAKER_00: It just helps your readers understand.
• 15:08
  And if they want to emulate that kind of structure,
• 15:11
  they have somewhere to go and get it.
• 15:13
  SPEAKER_00: So,
• 15:14
  SPEAKER_00: Dr.
• 15:15
  SPEAKER_00: Park,
• 15:15
  SPEAKER_00: if you're hearing us,
• 15:17
  SPEAKER_00: if you're listening to us,
• 15:18
  SPEAKER_00: we would love to have an example of what that looks like.
• 15:21
  SPEAKER_01: Exactly.
• 15:22
  SPEAKER_01: So the sample consisted of 78 undergraduate students.
• 15:26
  SPEAKER_01: They participated in a 90-minute course session and completed a five-item
• 15:31
  SPEAKER_01: conceptual quiz,
• 15:33
  SPEAKER_01: a short written reflection,
• 15:35
  SPEAKER_01: and an applied data analysis task.
• 15:38
  SPEAKER_01: And they were randomly assigned to one of those three conditions.
• 15:43
  SPEAKER_01: The theoretical framework, they used sensible AI and
• 15:47
  SPEAKER_01: As their primary interpretive lens,
• 15:50
  it's a framework concerned with designing AI systems that are
• 15:54
  intelligible,
• 15:55
  SPEAKER_01: contextually grounded,
• 15:57
  SPEAKER_01: and trustworthy for the humans who interact with them.
• 16:02
  SPEAKER_01: And do we want to talk about key findings of the study or...?
• 16:07
  SPEAKER_00: Yeah, let's talk about key findings.
• 16:09
  So they had three conditions and the conditions that had the
• 16:12
  best responses across
• 16:13
  SPEAKER_00: the board was inference with explanation.
• 16:16
  SPEAKER_00: So the explanation was helpful.
• 16:19
  SPEAKER_00: And the second one was actually no inference, just as it is.
• 16:26
  SPEAKER_00: And so no emotion.
• 16:27
  SPEAKER_00: And the condition B, the inference without explanation,
• 16:32
  SPEAKER_00: Was the worst one.
• 16:33
  SPEAKER_00: So it's like, when did you get, how did you get there?
• 16:35
  SPEAKER_00: All of that was really, really important.
• 16:38
  SPEAKER_00: So the emotion-aware AI
• 16:42
  SPEAKER_00: Dermotors is not just this is their conclusion.
• 16:45
  It's not just about how it's inferred,
• 16:49
  but how it's presented and contextualized.
• 16:52
  SPEAKER_00: And so we need to be careful.
• 16:54
  Really what it seems to be saying is when we use the inference
• 17:00
  about emotions,
• 17:01
  SPEAKER_00: we need to be very,
• 17:01
  SPEAKER_00: very careful.
• 17:05
  I have an example yesterday I was at a professional learning
• 17:07
  and I used an AI bot
• 17:09
  SPEAKER_00: and one of the terms it used was rookie.
• 17:13
  It's like way way too fast rookie let's start with the basics
• 17:18
  and there were two
• 17:19
  people in the audience me and somebody else and somebody else
• 17:22
  was offended and a
• 17:25
  little bit it's like who are you calling rookie we're teachers
• 17:28
  we're not rookies
• 17:30
  SPEAKER_00: and my response was a little bit different my response was to say hey
• 17:34
  SPEAKER_00: Why are you using the term?
• 17:35
  SPEAKER_00: So I had that conversation with AI and it explained it.
• 17:39
  It said,
• 17:40
  oh,
• 17:40
  I'm using that because here's the context I'm working within.
• 17:44
  SPEAKER_00: And within this context of this simulation,
• 17:46
  SPEAKER_00: it makes sense to call your rookie because you're supposed to be,
• 17:50
  SPEAKER_00: the envelope story is that you're a beginning reporter.
• 17:53
  SPEAKER_01: And I'm like, okay, now I understand.
• 17:55
  SPEAKER_01: Okay, now it makes sense.
• 17:57
  SPEAKER_00: I think there's something to it, right?
• 17:59
  SPEAKER_00: Oh, you explained it to me.
• 18:01
  SPEAKER_00: Now contextually I understand it and I can understand what's happening.
• 18:05
  SPEAKER_00: But again, it's an explorer.
• 18:08
  We have an emotional response,
• 18:10
  but that emotional response really depends on context.
• 18:13
  SPEAKER_00: And without context, we're like, what is going on?
• 18:15
  SPEAKER_01: Yeah.
• 18:16
  All right,
• 18:16
  let's go to the next one because I found that one interesting
• 18:20
  as well.
• 18:21
  SPEAKER_01: So the last one has been published this year,
• 18:23
  SPEAKER_01: Generative AI in Higher Education,
• 18:25
  SPEAKER_01: a Meta-Analysis of Intellectual and Social-Emotional Outcomes.
• 18:30
  SPEAKER_01: It's been published in Frontiers in Psychology.
• 18:33
  SPEAKER_01: The authors are Liu, Zhu, and Xi.
• 18:36
  SPEAKER_00: Again, sorry if we made mispronunciation.
• 18:39
  SPEAKER_00: We welcome your comments and we will try to do this.
• 18:43
  SPEAKER_00: Anyway, this again is a great publication and it's a meta-analysis.
• 18:48
  SPEAKER_00: So it's a study of studies and they studied 33 experiments.
• 18:52
  SPEAKER_00: They summarized 33 experimental and quasi-experimental studies.
• 18:57
  SPEAKER_00: Just showing you how much work is happening in a short amount of time.
• 19:01
  Because this is studies published only between 2021 and 2024,
• 19:07
  which is basically the span of AI.
• 19:09
  SPEAKER_00: It's a little longer than the span of AI.
• 19:11
  SPEAKER_00: Sometimes we're doing so much work in the space that
• 19:14
  SPEAKER_00: Sometimes we forget.
• 19:15
  SPEAKER_00: It's been less than four years and the world has changed.
• 19:20
  SPEAKER_00: It is over 3000 participants.
• 19:22
  SPEAKER_00: So that's great as well.
• 19:24
  SPEAKER_00: And they're looking at multiple outcomes.
• 19:26
  So they're looking at outcomes in learning or what they call
• 19:30
  intellectual outcomes
• 19:33
  SPEAKER_00: and social emotional outcomes.
• 19:34
  SPEAKER_01: So they are looking at knowledge,
• 19:36
  SPEAKER_01: skills and academic performance for the intellectual part and motivation.
• 19:41
  Self-efficacy engagement and satisfaction for the social
• 19:44
  emotional outcomes.
• 19:47
  And what they really looked at is that you always summarize it
• 19:52
  in effect size.
• 19:53
  SPEAKER_00: In this case,
• 19:55
  SPEAKER_00: they used Hedges G.
• 19:57
  SPEAKER_00: My favorite,
• 19:58
  SPEAKER_00: of course,
• 19:58
  SPEAKER_00: is Cohen's D.
• 19:59
  SPEAKER_00: But,
• 20:00
  SPEAKER_00: you know,
• 20:01
  SPEAKER_00: I didn't get to pick them.
• 20:02
  SPEAKER_00: They did.
• 20:03
  SPEAKER_00: The difference is very small.
• 20:05
  SPEAKER_00: There are some differences.
• 20:06
  SPEAKER_00: If anybody wants to talk about this, find somebody else.
• 20:10
  SPEAKER_00: I want to talk about AI.
• 20:12
  SPEAKER_00: Anyway, the effect size for intellectual outcomes was large.
• 20:17
  However,
• 20:19
  it was very clear that there was some selection bias in the
• 20:23
  publications.
• 20:25
  SPEAKER_00: And so with a positive skew,
• 20:29
  SPEAKER_00: which means that at least from a theoretical perspective,
• 20:32
  SPEAKER_00: there's a big chance there's a lot of,
• 20:35
  SPEAKER_00: not a lot,
• 20:35
  but some studies that were not published because the effect
• 20:38
  size was too small and
• 20:40
  SPEAKER_00: their authors didn't
• 20:43
  SPEAKER_00: Chose for whatever reason not to publish,
• 20:46
  SPEAKER_00: whether it was underpowered,
• 20:47
  SPEAKER_00: which happens often,
• 20:49
  SPEAKER_00: or they were disappointed and they didn't want to publish it.
• 20:52
  SPEAKER_00: It's hard to publish null results or low results.
• 20:55
  SPEAKER_00: And so that's part of the problem.
• 20:58
  And so there's definitely a bias in the intellectual outcomes,
• 21:02
  which makes
• 21:04
  SPEAKER_00: It makes the very large hedges G,
• 21:07
  SPEAKER_00: it's over one,
• 21:09
  SPEAKER_00: which is a very significant positive effect,
• 21:14
  SPEAKER_00: biased.
• 21:15
  SPEAKER_00: We know it's biased.
• 21:17
  SPEAKER_00: And on the other hand, the social-emotional outcomes
• 21:23
  SPEAKER_00: The hedges G is 0.3, which is much more reasonable.
• 21:28
  SPEAKER_00: And it's much less likely to have bias.
• 21:31
  SPEAKER_00: And when they did the test for bias, they did not find any bias.
• 21:34
  SPEAKER_00: And the reason there's a lot less bias is because for most likely,
• 21:40
  SPEAKER_00: most of these studies did not focus on the emotional,
• 21:46
  SPEAKER_00: social-emotional outcomes.
• 21:48
  SPEAKER_00: They focused on intellectual outcomes.
• 21:50
  SPEAKER_00: And they also measured others.
• 21:52
  SPEAKER_01: Mm-hmm.
• 21:53
  SPEAKER_00: So it's natural if there's a selection bias,
• 21:56
  SPEAKER_00: if they had large intellectual outcomes,
• 21:58
  SPEAKER_00: but low social emotional outcomes,
• 22:01
  SPEAKER_00: they definitely published it.
• 22:03
  And so we're seeing a much more normalized results and we're
• 22:08
  suspecting a lot less
• 22:09
  SPEAKER_00: bias,
• 22:09
  SPEAKER_00: but also we can see that the impact on social emotional aspects is much,
• 22:17
  SPEAKER_00: much lower.
• 22:18
  SPEAKER_00: And so there is a small
• 22:21
  SPEAKER_00: And even statistically significant, but it's still a small effect.
• 22:29
  SPEAKER_00: So what did they find?
• 22:30
  SPEAKER_00: And this is mostly in higher education.
• 22:32
  SPEAKER_00: And what they said is AI is very useful for intellectual things,
• 22:37
  SPEAKER_00: but not as useful for the social and emotional.
• 22:43
  SPEAKER_00: And this is where we keep seeing the advantage of people.
• 22:49
  SPEAKER_00: And that is
• 22:50
  SPEAKER_00: The live teacher who can respond in real time,
• 22:53
  SPEAKER_00: who can empathize,
• 22:54
  who can help navigate social emotional difficulty is doing a
• 22:58
  lot more of the
• 23:00
  SPEAKER_00: burden,
• 23:01
  SPEAKER_00: whereas that availability,
• 23:03
  SPEAKER_00: the advantages of
• 23:05
  The always patient,
• 23:07
  always available AI makes it much more likely to have impact on
• 23:13
  outcomes.
• 23:13
  SPEAKER_00: The one thing to remember is this is in higher education.
• 23:17
  SPEAKER_00: This is not in K-12.
• 23:20
  And we've seen this with a number of things where agency is a
• 23:24
  huge part of the learning.
• 23:26
  SPEAKER_00: The older the students are,
• 23:28
  the more likely are these technologies to be working because
• 23:31
  you have to engage on
• 23:32
  SPEAKER_00: your own.
• 23:32
  SPEAKER_00: You have to have
• 23:33
  SPEAKER_00: Based on the first study, right?
• 23:36
  SPEAKER_00: You have to have more complex conversations to get the most out of it.
• 23:40
  So this is more likely to happen with more mature,
• 23:45
  more motivated learners.
• 23:48
  SPEAKER_00: And higher education students are that for the most part.
• 23:53
  SPEAKER_01: Yep.
• 23:54
  SPEAKER_00: All right.
• 23:54
  SPEAKER_00: So what do you think we learned today about AI and emotions?
• 23:58
  SPEAKER_01: We can use,
• 23:59
  SPEAKER_01: as you said,
• 24:00
  SPEAKER_01: we can use human beings as a kind of,
• 24:03
  SPEAKER_01: I mean,
• 24:03
  AI can complement what human beings offer in the realm of
• 24:08
  social emotional things.
• 24:10
  SPEAKER_01: Yeah.
• 24:11
  SPEAKER_00: And,
• 24:11
  SPEAKER_00: and yeah,
• 24:12
  SPEAKER_00: people still have the advantage here,
• 24:14
  SPEAKER_01: I think,
• 24:16
  SPEAKER_00: because even with all that biased effect size was very small.
• 24:19
  And we saw very little with the chatbot in the first study and
• 24:24
  not much in the
• 24:25
  SPEAKER_00: second study either.
• 24:27
  SPEAKER_00: So it is very interesting.
• 24:29
  However,
• 24:29
  we do need to remember that each one of these were done in a
• 24:33
  very specific location.
• 24:35
  SPEAKER_00: With a very specific context.
• 24:37
  SPEAKER_01: Context, yeah.
• 24:38
  SPEAKER_00: Because the first one is in Singapore, the second one is in Korea.
• 24:44
  And so this is something that we need to consider because when
• 24:48
  we're talking about
• 24:48
  SPEAKER_00: emotions,
• 24:50
  SPEAKER_00: there are two structures that have an effect here.
• 24:55
  SPEAKER_00: And I think one is culture.
• 24:57
  SPEAKER_01: Mm-hmm.
• 24:58
  SPEAKER_00: Just the way you interact with machines,
• 25:01
  SPEAKER_00: the way you consider machines and all of that is mediated by culture.
• 25:04
  But on top of that,
• 25:06
  the way you respond to surveys is also mediated by culture.
• 25:11
  SPEAKER_00: And there's a significant literature.
• 25:13
  SPEAKER_00: So even if people are feeling those emotions,
• 25:16
  SPEAKER_00: it might not come out in the way they write,
• 25:19
  SPEAKER_00: if that's the interpretive scale,
• 25:21
  SPEAKER_00: like the sentiment analysis,
• 25:23
  SPEAKER_00: or in the way they respond to Likert scale.
• 25:27
  SPEAKER_01: Yeah.
• 25:27
  SPEAKER_00: And so there's that to worry about as we think about what's going on.
• 25:33
  The best evidence here is the meta-analysis because it's
• 25:36
  multiple places and all of that.
• 25:38
  SPEAKER_00: And that gives us a little bit more confidence.
• 25:41
  SPEAKER_00: And even there, there's a very low yet positive analysis.
• 25:47
  SPEAKER_00: All right.
• 25:48
  SPEAKER_00: An exciting week talking about emotions.
• 25:51
  And in the next few weeks,
• 25:52
  we'll have a few interviews and some more exciting things.
• 25:56
  SPEAKER_00: We'll see you next time See you