Latest / Azadeh and Guy on AI / Stealth Assessment, Creativity & the Future of Learning

Transcript

Stealth Assessment, Creativity & the Future of Learning · Azadeh and Guy on AI

0:00
SPEAKER_02: All right.
0:01
SPEAKER_02: Hi, and welcome to Azadeh.
0:03
SPEAKER_00: And Guy.
0:04
SPEAKER_02: On AI.
0:04
SPEAKER_02: And we have been really excited about the interviews we've had this season,
0:11
SPEAKER_02: and this is another one.
0:12
SPEAKER_02: So why don't you do the introduction?
0:14
SPEAKER_00: Yeah, sure.
0:15
SPEAKER_00: So today's guest is Dr. Rahimi, and I'm going to do a brief introduction.
0:21
SPEAKER_00: Then I will ask him to introduce himself even more, and then we get to the questions.
0:27
SPEAKER_00: So we are delighted to welcome Dr. Seyyed Ahmad Rahimi to the podcast.
0:31
SPEAKER_00: Dr. Rahimi is an Associate Professor of Learning Sciences and AI in Education
0:37
SPEAKER_00: at Georgia Tech's College of Lifetime Learning.
0:44
SPEAKER_02: It is a mouthful, to be honest.
0:46
SPEAKER_00: I know.
0:46
SPEAKER_00: By the way, I love the name of your college.
0:49
SPEAKER_00: His interdisciplinary work brings together learning sciences,
0:54
SPEAKER_00: artificial intelligence,
0:55
SPEAKER_00: assessment,
0:56
SPEAKER_00: and creativity with a particular interest in how we can design technologies
1:01
SPEAKER_00: that support learning while preserving uniquely human capabilities like creativity
1:07
SPEAKER_00: and critical thinking.
1:08
SPEAKER_00: Dr. Rahimi also investigates the safe and sound integration of generative AI
1:13
SPEAKER_00: into education with particular attention to how cognitive work should be allocated
1:18
SPEAKER_00: between learners and AI so that these systems support rather
1:22
SPEAKER_00: than undermine human thinking, agency and learning.
1:26
SPEAKER_00: His broader research includes productive failure,
1:29
SPEAKER_00: game-based learning,
1:30
SPEAKER_00: formative assessment,
1:31
SPEAKER_00: human AI collaboration,
1:33
SPEAKER_00: and the design of AI tools that support both learners and educators.
1:37
SPEAKER_00: We are very excited to have him with us today.
1:40
SPEAKER_00: Dr. Rahimi, welcome to the podcast.
1:43
SPEAKER_01: Wow, thank you so much for that very nice introduction.
1:46
SPEAKER_01: You covered a lot of grounds there.
1:48
SPEAKER_02: Yes, but she forgot to mention the most important thing, and that is you're a fellow podcaster.
1:55
SPEAKER_01: Oh, that's very important.
1:57
SPEAKER_01: It should have been mentioned at the beginning.
1:58
SPEAKER_02: Expectations just went up because you're familiar with the medium.
2:03
SPEAKER_01: I am, and I think it's a great,
2:05
SPEAKER_01: powerful medium to reach people and to Be creative,
2:11
SPEAKER_01: feel expressive,
2:13
SPEAKER_01: and I would say make a very small dent in the world in the direction that you want.
2:20
SPEAKER_01: Hopefully people will hear it, if not now, maybe later.
2:24
SPEAKER_01: A very quick thing that I would add to that introduction is that I'm the director of
2:29
SPEAKER_01: the CLASS Lab,
2:30
SPEAKER_01: and it stands for Creative Learning through AI and Stealth Assessment in STEM.
2:39
SPEAKER_01: Way more mouthful than Lifetime Learning College.
2:43
SPEAKER_01: For that reason, we turned that into class.
2:46
SPEAKER_01: And class means, you know, first of all, classy.
2:50
SPEAKER_01: It's classy, what we do.
2:52
SPEAKER_01: But it also refers to the context of what I usually conduct research in,
2:58
SPEAKER_01: which is a classroom and K-12 education.
3:01
SPEAKER_01: But I go beyond that.
3:04
SPEAKER_01: I'm not like, I don't identify as a K-12 teacher.
3:11
SPEAKER_01: But I find that period very important for learners' development.
3:17
SPEAKER_01: One other thing I would add,
3:18
SPEAKER_01: if you look at the logo of our lab,
3:22
SPEAKER_01: in the center of class,
3:23
SPEAKER_01: we have C,
3:25
SPEAKER_01: L, A, and then on top of that, a circle, which is AI, on top of each other.
3:31
SPEAKER_01: If you put A and I on top of each other, it becomes like a human.
3:35
SPEAKER_01: The A is the body and the circle is the face of a human.
3:41
SPEAKER_01: And that's how I approach AI in my research.
3:44
SPEAKER_01: It's human-centered approach.
3:46
SPEAKER_01: Human is in the center and the technology goes around it rather than the other way around.
3:53
SPEAKER_00: Awesome.
3:54
SPEAKER_00: Thank you so much.
3:55
SPEAKER_00: Anything else you want to add?
4:00
SPEAKER_01: Are you asking me or Guy?
4:01
SPEAKER_00: Yeah, yeah, yeah.
4:03
SPEAKER_01: No, I think The only thing I would add is that I very recently moved to Georgia Tech.
4:09
SPEAKER_01: Before that, I was an assistant professor at the University of Florida.
4:13
SPEAKER_01: I got my PhD at Florida State University.
4:16
SPEAKER_01: I worked with Valerie Shute,
4:19
SPEAKER_01: who came up with the idea of stealth assessment,
4:22
SPEAKER_01: one of the pioneering people who worked on that idea and coined the term.
4:27
SPEAKER_01: And I used to be a Floridian in the past.
4:31
SPEAKER_01: I don't know, 12 years?
4:32
SPEAKER_01: From 2014, I lost count.
4:35
SPEAKER_01: And then now we are moving to Georgia.
4:38
SPEAKER_01: So it's very recent.
4:40
SPEAKER_02: Both places are wet.
4:43
SPEAKER_01: I agree.
4:44
SPEAKER_01: But Florida is wetter, like wetter.
4:48
SPEAKER_01: I don't know.
4:50
SPEAKER_02: Yeah, I will take that.
4:51
SPEAKER_01: That is correct, English-wise.
4:54
SPEAKER_01: And when you mentioned associate professor, It still is new to my years.
4:59
SPEAKER_01: So I usually have struggle with introducing myself.
5:03
SPEAKER_01: I always start by saying, I'm an assistant professor, then I correct myself.
5:07
SPEAKER_01: No, I just became an associate professor.
5:11
SPEAKER_02: So congratulations.
5:12
SPEAKER_02: Thank you.
5:12
SPEAKER_02: Nothing else.
5:15
SPEAKER_02: I have to admit that when I read about stealth assessment,
5:18
SPEAKER_02: and we'll get some questions so we may get into it.
5:23
SPEAKER_02: It's not my favorite term.
5:24
SPEAKER_02: Can I say that?
5:26
SPEAKER_01: It is an interesting term.
5:27
SPEAKER_01: First of all, it will grab your attention to understand what is going on.
5:31
SPEAKER_01: And that was the intention.
5:32
SPEAKER_01: It was not the intention for the learners.
5:34
SPEAKER_01: We don't go to the classrooms or we don't label these products
5:38
SPEAKER_01: as like there would be a stealth assessment going on.
5:42
SPEAKER_01: But it's for researchers.
5:44
SPEAKER_01: To mention, we couldn't say embedded assessment.
5:50
SPEAKER_01: We couldn't say all those other terms because they will not Mm-hmm.
6:26
SPEAKER_02: All right.
6:27
SPEAKER_00: Perfect.
6:28
SPEAKER_00: Sounds awesome.
6:29
SPEAKER_00: So let's get to our questions.
6:31
SPEAKER_00: Our first question is,
6:33
SPEAKER_00: when you think about AI and education right now,
6:36
SPEAKER_00: what excites you most and what concerns you most about where we are heading?
6:43
SPEAKER_01: This is a great question.
6:44
SPEAKER_01: I think this can make people think more deeply about the moment that we are in.
6:49
SPEAKER_01: I think what excites me really the most is finally getting High-quality,
6:56
SPEAKER_01: personalized one-to-one learning possible at scale.
7:01
SPEAKER_01: If you know for sure,
7:03
SPEAKER_01: you know about Bloom's Two Sigma problem in this study that he showed that
7:09
SPEAKER_01: the students
7:09
SPEAKER_01: who had one-on-one tutoring performed almost two standard deviations better than
7:16
SPEAKER_01: the other students.
7:18
SPEAKER_01: And to this day,
7:19
SPEAKER_01: I think we haven't actually We've done a good job in that regard to make
7:24
SPEAKER_01: that really possible at scale.
7:26
SPEAKER_01: But now with this natural language processing at the hand of everybody
7:32
SPEAKER_01: with large language models, we probably can make that happen.
7:37
SPEAKER_01: AI in education field has been around way before ChatGPT came out.
7:42
SPEAKER_01: As you know,
7:42
SPEAKER_01: the field of AI in education and AI tutoring,
7:45
SPEAKER_01: intelligent tutoring systems have been around.
7:49
SPEAKER_01: At some point,
7:50
SPEAKER_01: even like in one of the papers that my advisor wrote many years ago,
7:55
SPEAKER_01: the field was kind of like giving up on making it work because it was very difficult
8:01
SPEAKER_01: to understand the natural language at that time, maybe 1990s or late 1980s.
8:07
SPEAKER_01: But now,
8:09
SPEAKER_01: at that time,
8:09
SPEAKER_01: we didn't know this powerful The power of computation and all the possibilities
8:16
SPEAKER_01: that can come around with the data that we have at our hands.
8:20
SPEAKER_01: But we are now seeing a new field to be playing in, which is sometimes unknown.
8:29
SPEAKER_01: So that's excitement for me,
8:31
SPEAKER_01: like new opportunities,
8:33
SPEAKER_01: one-on-one tutoring, solving that problem of two sigma that Bloom came up with.
8:39
SPEAKER_01: What I'm very concerned about is unproductive success.
8:45
SPEAKER_01: Kapoor, in his work on productive failure,
8:48
SPEAKER_01: divides success and failure into four quadrants, like four sections.
8:53
SPEAKER_01: Unproductive success, productive success, unproductive failure, and productive failure.
9:00
SPEAKER_01: So AI, as we know it,
9:02
SPEAKER_01: like the off-the-shelf vanilla AI,
9:04
SPEAKER_01: if you will, like JGPT or Claude, they can do whatever you want.
9:08
SPEAKER_01: Without limiting whatever you're asking them to do for you.
9:13
SPEAKER_01: And if they do what the learner actually is supposed to do to learn something,
9:20
SPEAKER_01: then that's the successful performance,
9:22
SPEAKER_01: which is not productive in terms of learning.
9:26
SPEAKER_01: That is way more at the hands of people these days.
9:31
SPEAKER_01: They open up their phone and ask a question, or maybe sometimes students ask to do my homework.
9:38
SPEAKER_01: We have seen this banning of AI at schools,
9:41
SPEAKER_01: in the schools, and students are very creative at going around these bans.
9:47
SPEAKER_01: And they,
9:48
SPEAKER_01: for one example,
9:49
SPEAKER_01: they asked the AI,
9:50
SPEAKER_01: which is in Snapchat, not ChatTPT, not Claude, to do their homework.
9:56
SPEAKER_01: That AI was not meant for This type of use, but they asked that to do their homework.
10:02
SPEAKER_01: So I hope my research and other people researching this area can help us create safe
10:08
SPEAKER_01: and sound AI for learning, which I can talk about it later.
10:15
SPEAKER_00: All right.
10:16
SPEAKER_02: And so as you're thinking about that,
10:21
SPEAKER_02: I'm wondering Because you're talking about designing really systems
10:27
SPEAKER_02: that would enhance that experience and make sure that we've got productive struggle
10:32
SPEAKER_02: and not unproductive struggle.
10:35
SPEAKER_02: And productive success and not unproductive success.
10:38
SPEAKER_02: So we're productive whatever we do.
10:41
SPEAKER_02: But how are you thinking about how that interacts with that access
10:46
SPEAKER_02: that everybody's got all the time?
10:50
SPEAKER_02: So what's the bridge that will help us cope with the fact that kids have access?
10:56
SPEAKER_02: Kids, and when I say kids,
10:58
SPEAKER_02: I mean all students,
10:59
SPEAKER_02: regardless of age, actually have access and may choose to use it or misuse it.
11:08
SPEAKER_01: Well, I think we are kind of behind the industry in this game.
11:14
SPEAKER_01: And they dictate what we should do or not do Based on this broad availability
11:23
SPEAKER_01: of this technology, I saw a very interesting, thought-provoking graph.
11:30
SPEAKER_01: Do you know how many days or how many years it took people to have access to radio?
11:37
SPEAKER_01: A while.
11:38
SPEAKER_01: A while.
11:39
SPEAKER_01: I can say that as well.
11:40
SPEAKER_01: Thirty-seven something years.
11:43
SPEAKER_01: And then it drops to almost 17 days.
11:48
SPEAKER_01: To equalize to that number with radio for Chan GPT.
11:54
SPEAKER_01: 17 days or maybe less than that.
11:56
SPEAKER_01: So it's astronomically different technology.
12:02
SPEAKER_01: And the temptation of using it is there always.
12:08
SPEAKER_01: We are interacting with the technology without guardrails
12:14
SPEAKER_01: before even understanding what Guardrails we need to create to protect ourselves
12:22
SPEAKER_01: from the damage it can cause.
12:25
SPEAKER_01: From many years ago,
12:26
SPEAKER_01: when it comes to technology,
12:28
SPEAKER_01: I have a book actually called The Mind and the Machine, which was in 1980-something.
12:34
SPEAKER_01: Very famous MIT writers wrote,
12:38
SPEAKER_01: I don't remember their names,
12:39
SPEAKER_01: but they were talking about this issue of technologies and
12:44
SPEAKER_01: the damage they can cause.
12:45
SPEAKER_01: And what we need to do.
12:47
SPEAKER_01: In that time,
12:48
SPEAKER_01: they thought that because we know how humans go from novice to expert,
12:53
SPEAKER_01: how they change levels to get to an expert level,
12:57
SPEAKER_01: and how machines or AI at that time is designed,
13:02
SPEAKER_01: they thought machines never can do things that humans can do at an expert level.
13:08
SPEAKER_01: But they mistakenly predicted the future because what the machines,
13:13
SPEAKER_01: AI can do these days, Actually looks expert level product.
13:19
SPEAKER_01: We don't care.
13:19
SPEAKER_01: I don't care if the way it predicts or the way it works is like predicting
13:24
SPEAKER_01: the next word or something like that.
13:26
SPEAKER_01: But what I look at is somehow expert level, especially these latest models are really powerful.
13:33
SPEAKER_01: So this notion that machine can cause these damages has been around for many years.
13:44
SPEAKER_01: And we have been thinking about what is the effect with technology and what is
13:51
SPEAKER_01: the effect of technology.
13:54
SPEAKER_01: The with technology relates to performance.
13:56
SPEAKER_01: And everybody can use these tools to enhance their performance or to say
14:01
SPEAKER_01: that I did this, I wrote this, but AI does that for you.
14:04
SPEAKER_01: That's the effect with technology.
14:06
SPEAKER_01: And sometimes it's very good quality, high quality, if that is the purpose.
14:12
SPEAKER_01: The learning is really concerning the effect of technology.
14:16
SPEAKER_01: So what happens, what is the residual of that interaction that comes with you?
14:21
SPEAKER_01: That's the effect that we as educators need to be concerned with.
14:26
SPEAKER_01: And I think I wrote even in that paper called SAIL,
14:30
SPEAKER_01: the Framework Safe and Sound AI for Learning at the end,
14:33
SPEAKER_01: that if we design these learning environments,
14:36
SPEAKER_01: AI-powered learning environments,
14:37
SPEAKER_01: with these guardrails,
14:38
SPEAKER_01: which I call them external guardrails,
14:42
SPEAKER_01: And the learners from early ages interact with these environments.
14:48
SPEAKER_01: We can tackle the problem,
14:50
SPEAKER_01: the two sigma problem,
14:52
SPEAKER_01: and we can train the humans that a responsible use of AI is this way.
14:58
SPEAKER_01: So then they internalize those guardrails when they get to a certain age.
15:05
SPEAKER_01: Then that person can use these off-the-shelf AI tools responsibly.
15:11
SPEAKER_01: They can realize what is the purpose of my interaction with this tool.
15:16
SPEAKER_01: Is it for learning something?
15:17
SPEAKER_01: If it's learning for something, I'm not going to let AI do it for me.
15:21
SPEAKER_01: I'm going to use AI to my advantage.
15:23
SPEAKER_01: If it's performance, I can delegate what I need to do that way.
15:27
SPEAKER_01: So we need,
15:28
SPEAKER_01: as I said,
15:28
SPEAKER_01: we are behind this game and now it's in the hands of students and everybody else.
15:33
SPEAKER_01: But we have to really go faster as a field To come up with these frameworks,
15:38
SPEAKER_01: design these tools,
15:39
SPEAKER_01: and then push people to,
15:42
SPEAKER_01: instead of banning these technologies in schools,
15:45
SPEAKER_01: providing the well-designed technologies in the schools
15:48
SPEAKER_01: to help students develop those internal guardrails for the long term.
15:53
SPEAKER_01: I hope I answered your question.
15:54
SPEAKER_01: I don't know.
15:55
SPEAKER_02: I think you did.
15:56
SPEAKER_02: And if not, it was better than my question.
16:01
SPEAKER_02: This is perfectly fine.
16:02
SPEAKER_02: What I'm thinking about, so we just had a small gathering around AI this weekend.
16:10
SPEAKER_02: And that's one of the things that came is people who were interested in AI.
16:14
SPEAKER_02: And there's this world of I think we're wise to realize that AI is a major force
16:23
SPEAKER_02: in our society.
16:24
SPEAKER_02: I think it's unavoidable.
16:26
SPEAKER_02: But we have a lot of colleagues,
16:27
SPEAKER_02: whether in K-12 or in higher education,
16:31
SPEAKER_02: that are I want to say that for whatever reason,
16:35
SPEAKER_02: and I think there are multiple reasons here,
16:37
SPEAKER_02: there's resistance,
16:38
SPEAKER_02: there's this sense of, I mean,
16:41
SPEAKER_02: I have colleagues that feel lost, right,
16:44
SPEAKER_02: in this wave of innovation that, again,
16:47
SPEAKER_02: as you said,
16:47
SPEAKER_02: In 17 days we've had millions,
16:50
SPEAKER_02: hundreds of millions of people using ChatGPT and that's unprecedented.
16:55
SPEAKER_02: And so they're ignoring it.
16:57
SPEAKER_02: I go to meetings and conferences and things and I'm thinking,
17:03
SPEAKER_02: if AI is not at least one of the top two topics you're talking about,
17:08
SPEAKER_02: I think we have a problem.
17:10
SPEAKER_02: If all the administrators,
17:11
SPEAKER_02: my favorite example is I went to administrator days in July here in Nebraska,
17:17
SPEAKER_02: and all the administrators from around the state get together to talk about
17:20
SPEAKER_02: the next school year, planning, they consult, they meet, they talk.
17:25
SPEAKER_02: They think together.
17:26
SPEAKER_02: They do all of these great things.
17:28
SPEAKER_02: There was very little about AI.
17:30
SPEAKER_02: And all I kept thinking is, how is this not the thing that everybody is talking about?
17:35
SPEAKER_02: I mean, that's all I talk about.
17:37
SPEAKER_02: And I understand that I'm in a bubble.
17:39
SPEAKER_02: So there's a risk of being in the bubble and not recognizing
17:43
SPEAKER_02: that there's life outside the bubble.
17:45
SPEAKER_02: But we are also seeing it affecting everything in education at the same time.
17:49
SPEAKER_02: So I'm 100% with you.
17:51
SPEAKER_02: We have to figure this out.
17:52
SPEAKER_02: And we cannot wait.
17:53
SPEAKER_02: I mean, in some ways, we need society to figure this out.
17:57
SPEAKER_02: Because our society is so fixated on outcomes.
18:01
SPEAKER_02: And it's exactly the outcomes that AI is very good at doing.
18:05
SPEAKER_02: And that is, these things look polished and they look done.
18:09
SPEAKER_02: And it's an industrial production of memos.
18:12
SPEAKER_02: I mean, I always joke that,
18:14
SPEAKER_02: you know,
18:14
SPEAKER_02: all of our white collar work is about moving pieces of papers,
18:18
SPEAKER_02: even if they're digital, from one desk to the next.
18:21
SPEAKER_02: I mean, when I publish, what do I do?
18:22
SPEAKER_02: I want to finish the paper so I can send it to Somebody else can read it.
18:25
SPEAKER_02: I'm done.
18:26
SPEAKER_02: I want some vacation from it, right?
18:28
SPEAKER_02: That's all I'm doing.
18:29
SPEAKER_02: And then their goal is to read it as fast as possible,
18:32
SPEAKER_02: respond, and send it back so it's not on their desk.
18:35
SPEAKER_02: We're all moving things around desks.
18:37
SPEAKER_02: And the problem is that came into education as if that is the goal of life,
18:43
SPEAKER_02: or at least the work life, is just to move papers.
18:45
SPEAKER_02: The goal is not to move papers.
18:47
SPEAKER_02: The goal is actually to be able to do things.
18:50
SPEAKER_02: And with AI, it's really easy to confuse the two and say, I'm done.
18:57
SPEAKER_02: But I want to take you back to the stealth assessment because that's the thing
19:03
SPEAKER_02: that you've talked about early on.
19:07
SPEAKER_02: And you're really talking about something that I've been thinking about for a long time.
19:11
SPEAKER_02: So I love that fact that people are actually doing the work and not just talking.
19:15
SPEAKER_02: I'm very good at talking, doing the work.
19:17
SPEAKER_02: I don't know.
19:18
SPEAKER_02: There are some things I know how to do, but this is not one of them.
19:21
SPEAKER_02: But I've been intrigued by that idea that we can use digital traces especially
19:26
SPEAKER_02: to see what kids are learning without putting them in that occasion where we say,
19:31
SPEAKER_02: now you're taking a test.
19:33
SPEAKER_02: And this will count very,
19:34
SPEAKER_02: very highly and put the anxiety and the pressure,
19:37
SPEAKER_02: but also differential performance and all of that.
19:40
SPEAKER_02: So tell us a little bit about that and tell us where do you think it's going,
19:46
SPEAKER_02: especially in the age of AI?
19:50
SPEAKER_01: Do we have two hours?
19:51
SPEAKER_01: No.
19:52
SPEAKER_02: Okay.
19:52
SPEAKER_02: Well, we do.
19:53
SPEAKER_02: I'm not sure the listeners want to listen for two hours, but I do as much as possible.
19:59
SPEAKER_01: But yeah, just the AI know.
20:01
SPEAKER_01: Sure.
20:02
SPEAKER_01: So let me give you a very short background.
20:05
SPEAKER_01: First of all, hats off to my advisor, Valerie Shute, for coming up with this creative idea.
20:12
SPEAKER_01: And then also on the other giants,
20:15
SPEAKER_01: we are standing on the shoulders of the giants,
20:17
SPEAKER_01: like Robert Mislevy,
20:18
SPEAKER_01: who passed away like two years ago,
20:21
SPEAKER_01: the person who came up with the Evidence Center Design Framework for Assessment,
20:27
SPEAKER_01: Russell Amund,
20:28
SPEAKER_01: another advisor of mine when I was getting my master's degree
20:31
SPEAKER_01: from Florida State University, and others who worked in this area.
20:38
SPEAKER_01: When we wrote those papers when I was a PhD student about stealth assessment,
20:43
SPEAKER_01: the first thing that I remember,
20:45
SPEAKER_01: like the first paper that I helped with Val to write about,
20:49
SPEAKER_01: And I think she was kind enough to just involve me in the writing to learn.
20:53
SPEAKER_01: It was not co-authoring with her.
20:55
SPEAKER_01: I always see those opportunities as opportunities to grow.
20:58
SPEAKER_01: And I'm trying to provide the same thing for my students these days.
21:03
SPEAKER_01: One sentence stood with me was that if we think about this pattern,
21:09
SPEAKER_01: teach, stop, test, stop, then teach, stop.
21:15
SPEAKER_01: Those stop moments are taking a lot of If
21:26
SPEAKER_01: we replace those with learning, what happens?
21:31
SPEAKER_01: If a student comes to a learning environment,
21:33
SPEAKER_01: whether it wants to be a school,
21:35
SPEAKER_01: classroom,
21:35
SPEAKER_01: a digital environment,
21:37
SPEAKER_01: and they always think that I am learning,
21:41
SPEAKER_01: and the assessment is something that happens continuously, unobtrusively, and
21:50
SPEAKER_01: So we have two types of assessments in general,
21:53
SPEAKER_01: assessment of learning and assessment for learning.
21:56
SPEAKER_01: Stealth assessment is known to be an assessment for learning.
22:01
SPEAKER_01: The reason I said that it is invisible but transparent is that the learner knows
22:07
SPEAKER_01: that he or she is being assessed on this thing,
22:10
SPEAKER_01: and that's because the system wants to provide appropriate personalized support
22:15
SPEAKER_01: after that in an ongoing manner.
22:19
SPEAKER_01: Knowing that the purpose of this technology or method of assessment is to blur
22:26
SPEAKER_01: the boundaries between learning and assessment and to kind of harness those times
22:33
SPEAKER_01: that would be spent in preparation for assessment and then grading the assessment
22:38
SPEAKER_01: and then sometimes providing feedback which never gets to the learners in time.
22:44
SPEAKER_01: We are kind of trying to bring that into the system so the educational
22:49
SPEAKER_01: and learning system can be very responsive to each individual and provide
22:54
SPEAKER_01: the right support at the right time and to
23:00
SPEAKER_01: prevent the learning gaps
23:02
SPEAKER_01: that can prolong for a long time and then cause issues in the future.
23:08
SPEAKER_01: So that's the main goal.
23:12
SPEAKER_01: ECD,
23:12
SPEAKER_01: which is the underlying framework that we use to design these type of assessments,
23:17
SPEAKER_01: is a top-down approach.
23:19
SPEAKER_01: Meaning,
23:20
SPEAKER_01: you first define what is it that you're trying to assess with
23:24
SPEAKER_01: the competency model development.
23:26
SPEAKER_01: If I'm saying creativity, what do I mean by that?
23:29
SPEAKER_01: How do I operationalize it?
23:31
SPEAKER_01: And then I think about the evidence.
23:35
SPEAKER_01: Okay, if I say creativity,
23:37
SPEAKER_01: for example,
23:37
SPEAKER_01: is about divergent thinking,
23:39
SPEAKER_01: and divergent thinking is about being fluent in my ideas that I come up with,
23:44
SPEAKER_01: what are the things that a person who is high on divergent thinking
23:48
SPEAKER_01: or fluency can do?
23:50
SPEAKER_01: I need to specify that.
23:51
SPEAKER_01: That's how I define the competency.
23:53
SPEAKER_01: And then those indicators that I'm trying to mention are the things that I can
24:00
SPEAKER_01: then include in the learning environment to capture.
24:05
SPEAKER_01: As we say, after the indicators or evidence, we go to the tasks.
24:09
SPEAKER_01: We know what we want to assess in the competency model.
24:12
SPEAKER_01: We know what evidence can inform that in the evidence model.
24:15
SPEAKER_01: And we know what type of tasks can provide those evidence for us in the task model.
24:21
SPEAKER_01: That's how we design ECD.
24:23
SPEAKER_01: It's top-down.
24:25
SPEAKER_01: It's not like bottom-up, like other methods like machine learning or deep learning.
24:29
SPEAKER_01: We create this learning environment,
24:31
SPEAKER_01: we design it well with the intentions that we have in mind, and we trace everything.
24:37
SPEAKER_01: Then we kind of like send all of these variables into a machine learning model
24:41
SPEAKER_01: and see how that model can predict students' learning or other outcome variables.
24:46
SPEAKER_01: That's bottom up.
24:47
SPEAKER_01: You look at the patterns.
24:49
SPEAKER_01: Sometimes those models,
24:50
SPEAKER_01: if they are not following the latest learning analytics methods
24:55
SPEAKER_01: that use some new methods called feature engineering and stuff like that,
25:01
SPEAKER_01: You can't really explain why the model can predict learning.
25:06
SPEAKER_01: You can't really get to the explanation unless you have
25:09
SPEAKER_01: the explainable AI layer included in those models that can
25:17
SPEAKER_01: disentangle
25:19
SPEAKER_01: the messy data or model that you built.
25:23
SPEAKER_01: But the top-down approach is very explainable.
25:27
SPEAKER_01: You have diagnosis assessment because you have all of those sub-facets of
25:31
SPEAKER_01: the competency model that are linked to evidence, that are linked to tasks.
25:35
SPEAKER_01: It's very easy to explain where the learner is lacking what knowledge and what skills.
25:41
SPEAKER_01: So that's how I differentiate between machine learning approaches,
25:45
SPEAKER_01: bottom-up approaches, and top-down approaches.
25:49
SPEAKER_01: To finalize my response to you in terms of how modern AI can change this,
25:57
SPEAKER_01: I would say it can make that process of evidence identification
26:02
SPEAKER_01: and evidence accumulation most probably the evidence identification.
26:06
SPEAKER_01: The evidence identification is one process that you look at all the data
26:09
SPEAKER_01: that you have, you identify the good evidence that comes from that interaction.
26:15
SPEAKER_01: That part can be richer with the interaction with AI.
26:18
SPEAKER_01: Even if AI can ask a very simple question and get a very simple reflection from
26:24
SPEAKER_01: the student,
26:25
SPEAKER_01: that can show misconceptions that students might have,
26:28
SPEAKER_01: that can show learning that the student might have.
26:30
SPEAKER_01: In my own project,
26:33
SPEAKER_01: the NSF Career Project,
26:34
SPEAKER_01: which just got funded,
26:35
SPEAKER_01: thankfully,
26:36
SPEAKER_01: after three times,
26:37
SPEAKER_01: two times trying,
26:38
SPEAKER_01: this was my third time, which was the last time when I finally got it, I want to...
26:43
SPEAKER_01: So the students go through this learning environment to design their
26:50
SPEAKER_01: They need to think about,
26:56
SPEAKER_01: oh, I'm going to create a level for physics, Newton's first law, for example.
27:04
SPEAKER_01: But I want to also be creative and make a very interesting creative level.
27:08
SPEAKER_01: At the end,
27:09
SPEAKER_01: when they're done,
27:10
SPEAKER_01: the coach,
27:11
SPEAKER_01: the AI coach, which I'm going to design, is going to start a conversation with them.
27:17
SPEAKER_01: Explain to me, for example, I don't know exactly how we will design that part.
27:21
SPEAKER_01: Explain to me how this level is related to Newton's first law.
27:25
SPEAKER_01: What is your thinking behind it?
27:27
SPEAKER_01: So that explanation can bear a lot of evidence for the students' understanding
27:31
SPEAKER_01: of Newton's first law.
27:33
SPEAKER_01: And also explain to me why do you think this is creative?
27:35
SPEAKER_01: That also can bear evidence for creativity.
27:38
SPEAKER_01: Apart from other pieces of evidence that we can gather from that level
27:42
SPEAKER_01: for assessment of creativity,
27:45
SPEAKER_01: we can enrich the evidence that we can gather from the interaction with the AI.
27:50
SPEAKER_01: There are so many other things that we can do,
27:52
SPEAKER_01: like have AI for assessment of the products, the process, and so on.
27:57
SPEAKER_01: But I just think
27:58
SPEAKER_01: that large language models specifically can Enrich both evidence identification
28:04
SPEAKER_01: and also evidence accumulation process in the self assessment technique.
28:10
SPEAKER_02: Yeah, and I was thinking about the role of AI,
28:13
SPEAKER_02: especially as you're defining things like creativity,
28:16
SPEAKER_02: because there's some subsets of skills that are very easy on a top-down
28:21
SPEAKER_02: to know what comes first, what comes second, and all of that.
28:24
SPEAKER_02: There are those places like creativity where that is almost orthogonal
28:30
SPEAKER_02: to those almost developmental steps where you can – Track the development
28:37
SPEAKER_02: and suddenly you put this thing here which can be very different and when you talk
28:44
SPEAKER_02: about divergent thinking, that's much harder to assess out of the old boxes, right?
28:51
SPEAKER_02: Of the old learning management systems and all of the ways
28:56
SPEAKER_02: that were considerably more mechanical even if you had a chatbot
28:59
SPEAKER_02: and we've been working in chatbots long before Before
29:04
SPEAKER_02: the large language models were there,
29:07
SPEAKER_02: but now there's that ability to assess something that is a little more ephemeral
29:13
SPEAKER_02: through that.
29:14
SPEAKER_02: And that's the piece that I really love as an added layer that makes this a lot more doable.
29:20
SPEAKER_02: And as you said, really scalable.
29:24
SPEAKER_01: Exactly.
29:24
SPEAKER_01: What you asked me earlier about what excites me,
29:27
SPEAKER_01: I forgot about this,
29:28
SPEAKER_01: I had this in my responses,
29:30
SPEAKER_01: that I'm very excited about measuring hard to measure or hard
29:35
SPEAKER_01: to assess constructs like creativity,
29:37
SPEAKER_01: persistence,
29:38
SPEAKER_01: problem solving,
29:39
SPEAKER_01: collaboration, all of those now could be done at scale, as you mentioned.
29:46
SPEAKER_01: Creativity is like,
29:48
SPEAKER_01: in some older papers and research on creativity,
29:53
SPEAKER_01: Very famous creativity researchers,
29:55
SPEAKER_01: I don't want to name names,
29:56
SPEAKER_01: but they mentioned we need to drop the objective measure of creativity
30:05
SPEAKER_01: because it's not possible.
30:06
SPEAKER_01: We need to have the subjectivity always included.
30:09
SPEAKER_01: And there were some ways we could go around that,
30:13
SPEAKER_01: but when it comes to pure creativity or really that creativity
30:16
SPEAKER_01: that scholars are talking about,
30:19
SPEAKER_01: we need to do some sort of like Methods like,
30:23
SPEAKER_01: for example,
30:23
SPEAKER_01: if 10 experts untrained tell you that this is a creative work,
30:31
SPEAKER_01: you need to accept that as a creative work.
30:34
SPEAKER_01: That's the consensual assessment technique by Theresa Immobile.
30:39
SPEAKER_01: But can we now use AI to assess creative work more objectively And mimic
30:49
SPEAKER_01: that subjective process.
30:51
SPEAKER_01: I think we are getting closer to that these days.
30:53
SPEAKER_02: Well, I would be careful.
30:57
SPEAKER_02: This is crazy.
30:58
SPEAKER_02: We can have three-hour conversation about this.
31:00
SPEAKER_02: I've been thinking about creativity for a long time because part of my work is
31:05
SPEAKER_02: in arts education.
31:06
SPEAKER_02: And so I've been thinking about those aspects for a long time.
31:12
SPEAKER_02: When we talk about the AI,
31:13
SPEAKER_02: the subjective and objective is a little bit problematic in definition, right?
31:19
SPEAKER_02: Is it objective just because the machine produces it, even produces it reliably?
31:24
SPEAKER_02: We can talk about reliability.
31:26
SPEAKER_02: And from that perspective, yes.
31:29
SPEAKER_02: I don't know about subjective and objective.
31:32
SPEAKER_02: So we'll get there maybe one day.
31:34
SPEAKER_02: And maybe in your project.
31:36
SPEAKER_02: And I will be happy to be wrong about that, by the way.
31:39
SPEAKER_01: Very happy.
31:40
SPEAKER_01: In a very recent study that we published in the Creativity Research Journal,
31:45
SPEAKER_01: and in the same context that I was talking about,
31:48
SPEAKER_01: the game environment,
31:49
SPEAKER_01: we call it physics playground, where students design their own game levels.
31:54
SPEAKER_01: This was actually my dissertation project.
31:57
SPEAKER_01: I had 421 game levels that students designed,
32:01
SPEAKER_01: but I had two raters, two human raters to rate those levels.
32:06
SPEAKER_01: At that time, there was no LLM.
32:09
SPEAKER_01: But then I saw this opportunity.
32:11
SPEAKER_01: Now,
32:13
SPEAKER_01: how can I make the LLM judge these levels for their creativity that kind of matches
32:20
SPEAKER_01: the average of those two raters' judgment?
32:23
SPEAKER_01: And we got to a very high correlation of about 0.74,
32:29
SPEAKER_01: if I'm not mistaken, which is very promising.
32:31
SPEAKER_01: But one thing that we did test was the systematic way of getting there.
32:36
SPEAKER_01: You can't simply ask AI to judge this for creativity.
32:39
SPEAKER_01: A, you needed a rubric.
32:41
SPEAKER_01: With rubric, we got 0.7 something correlation.
32:44
SPEAKER_01: Without rubric, we got 0.5 or 0.6 correlation.
32:46
SPEAKER_01: So...
32:49
SPEAKER_01: There is some systematic way to get there, not just ask AI to do it.
32:53
SPEAKER_01: Oh, for sure.
32:54
SPEAKER_02: For sure.
32:56
SPEAKER_01: Anyway.
32:58
SPEAKER_01: Yes, let's talk about it later.
32:59
SPEAKER_02: Yeah, because I love that idea.
33:01
SPEAKER_02: And I want to swim with the data a little bit.
33:04
SPEAKER_02: But I think that there are multiple facets of this.
33:08
SPEAKER_02: And what I love about this is that you examined this within a very specific domain.
33:12
SPEAKER_02: I think that domain transfer is a little bit harder.
33:16
SPEAKER_02: But that being said, you know, always excited to be wrong.
33:22
SPEAKER_02: As they will say, I'm really excited to be wrong.
33:24
SPEAKER_02: Every time I get reviewers responding to my papers, I am so positive.
33:29
SPEAKER_02: You are so correct and I was wrong.
33:31
SPEAKER_02: No, I never said that.
33:33
SPEAKER_02: No, but we're always getting better.
33:35
SPEAKER_02: That being said, I'm gracious and thank you to all of my reviewers.
33:40
SPEAKER_01: Very good.
33:41
SPEAKER_01: You reminded me of this book called Think Again by Adam Grant.
33:47
SPEAKER_01: I'm not sure if you've read that.
33:48
SPEAKER_01: There's a chapter called The Joy of Being Wrong.
33:51
SPEAKER_01: So as scientists, we need to embrace
33:54
SPEAKER_02: the wrongness of our ways.
33:56
SPEAKER_02: Yes.
33:58
SPEAKER_00: Okay.
33:58
SPEAKER_00: So let's wrap up with the final question.
34:01
SPEAKER_00: Yes.
34:02
SPEAKER_00: If you look five or ten years ahead,
34:04
SPEAKER_00: how do you think AI will change What it means to be a successful learner
34:09
SPEAKER_00: and what human capabilities will become even more important?
34:14
SPEAKER_01: Well, I think this is a great question to end with because it kind
34:17
SPEAKER_01: of sums up what we talked about so far.
34:21
SPEAKER_01: I want to just quote Chris Didi here, one of my best mentors.
34:27
SPEAKER_01: He always says that lifetime learners are people who can thrive on chaos.
34:35
SPEAKER_01: Not just survive, but thrive.
34:38
SPEAKER_01: And I always ask myself, like, what a lifetime learner, who is a lifetime learner?
34:44
SPEAKER_01: What characteristics do they have?
34:47
SPEAKER_01: In the college,
34:48
SPEAKER_01: I think we have had this conversation because the name of the college
34:52
SPEAKER_01: at Georgia Tech is called Lifetime Learning.
34:55
SPEAKER_01: What do we mean by that?
34:57
SPEAKER_01: How different that is from lifelong learning?
35:00
SPEAKER_01: To me, the word lifetime Is there,
35:04
SPEAKER_01: I think,
35:05
SPEAKER_01: to emphasize the difference from lifelong learning,
35:08
SPEAKER_01: which is like you can continuously learn about new things as you go to a new job and
35:13
SPEAKER_01: so on.
35:14
SPEAKER_01: But lifetime learning is actually bringing all
35:16
SPEAKER_01: of these durable transversal skills like creativity,
35:20
SPEAKER_01: love for learning,
35:21
SPEAKER_01: persistence,
35:22
SPEAKER_01: and all of those characteristics that can help you thrive on chaos with it.
35:28
SPEAKER_01: And I think in the college or in the field of lifetime learning,
35:32
SPEAKER_01: if there is a field Named after that after a couple of years.
35:37
SPEAKER_01: We need to work on two branches of things.
35:40
SPEAKER_01: One is how to help develop these lifetime learners.
35:45
SPEAKER_01: Research go into that curriculum and plans that go into helping more people
35:51
SPEAKER_01: to become lifetime learners and also infrastructure for lifetime learners.
35:57
SPEAKER_01: For example,
35:58
SPEAKER_01: if Suzanne in Idaho,
36:00
SPEAKER_01: in her 60s,
36:02
SPEAKER_01: who has a farm,
36:04
SPEAKER_01: wants to get a credential in something to do something different in her life,
36:10
SPEAKER_01: receives this coaching suggestion from an AI companion or coach
36:18
SPEAKER_01: that maybe this credential,
36:20
SPEAKER_01: which is only two weeks,
36:21
SPEAKER_01: is good for you to go and take it from Georgia Tech or something else.
36:25
SPEAKER_01: That is the type of infrastructure that we don't have in place.
36:28
SPEAKER_01: For example,
36:28
SPEAKER_01: if I go to System 1 and learn something,
36:31
SPEAKER_01: System 2 understands what I did in System 1, and so on and so forth.
36:36
SPEAKER_01: So I would say a successful lifetime learner is a person that I hope we can see in
36:42
SPEAKER_01: the next 5-10 years more broadly out there.
36:45
SPEAKER_01: Someone who has strong internal guardrails,
36:49
SPEAKER_01: who can Protect their learning when it comes to AI,
36:52
SPEAKER_01: who can be adaptive and agile when it comes to disruption of workforce,
36:58
SPEAKER_01: you know, as we know these days.
37:01
SPEAKER_01: And who is creative and has all of these great skills that comes with Lifetime Learner.
37:09
SPEAKER_00: Awesome.
37:09
SPEAKER_00: Yeah.
37:10
SPEAKER_00: Thank you so much.
37:11
SPEAKER_02: Thank you very much for giving us your time and your attention.
37:17
SPEAKER_02: And we hope we'll have more conversations because I'm excited about all of this,
37:23
SPEAKER_02: from stealth to creativity to we may need to block some time off and continue
37:31
SPEAKER_02: that conversation.
37:33
SPEAKER_01: Thank you so much for having me and for the invitation.
37:35
SPEAKER_01: Hopefully we can talk about this more in the future.
37:38
SPEAKER_01: All right.
37:39
SPEAKER_00: Thank you.