Latest / Azadeh and Guy on AI / AI Delegation, Overreliance & Literacy in Engineering Ed

Transcript

AI Delegation, Overreliance & Literacy in Engineering Ed · Azadeh and Guy on AI

• 0:00
  SPEAKER_01: Hello, everyone.
• 0:02
  SPEAKER_01: Welcome to the third episode of
• 0:03
  SPEAKER_00: Azadeh
• 0:05
  SPEAKER_01: on AI.
• 0:06
  SPEAKER_00: And today we've got Azadeh has traveled to Norman, Oklahoma.
• 0:12
  SPEAKER_01: Yeah.
• 0:13
  SPEAKER_00: Where she... Super hot, super hot.
• 0:16
  SPEAKER_00: And I want to say, I've been in this country for almost 30 years at this point.
• 0:23
  SPEAKER_00: And I've never been to Norman, Oklahoma.
• 0:25
  SPEAKER_01: Oh, I didn't know about that.
• 0:26
  SPEAKER_00: Yeah.
• 0:27
  SPEAKER_00: And how long have you been here?
• 0:30
  SPEAKER_01: Since February 2023.
• 0:32
  SPEAKER_00: So, three and a half years?
• 0:35
  SPEAKER_00: And she's already been to Norman Oklahoma.
• 0:37
  SPEAKER_00: I'm just saying, this is where it's at.
• 0:40
  SPEAKER_01: Anyway, why did you go there?
• 0:43
  SPEAKER_01: Actually,
• 0:43
  SPEAKER_01: a
• 0:47
  SPEAKER_01: couple of folks from the engineering education program and also I joined together
• 0:54
  SPEAKER_01: to go to ASE Midwest Conference.
• 0:57
  SPEAKER_00: And you went really with the fabulous Megan Pitka, who's part of our React-E group.
• 1:05
  SPEAKER_00: And they went to represent the findings from one of our studies.
• 1:09
  SPEAKER_00: So if you're dying to know what we're thinking about and what we learned
• 1:12
  SPEAKER_00: from students in engineering, read the paper.
• 1:15
  SPEAKER_00: It's in the proceedings.
• 1:16
  SPEAKER_01: Yeah, yeah, absolutely.
• 1:18
  SPEAKER_00: Anyway, so you've gone.
• 1:20
  SPEAKER_00: So set the scene for us about the people that were there.
• 1:24
  SPEAKER_00: What was the conference about this year?
• 1:26
  SPEAKER_00: Who was in the room?
• 1:28
  SPEAKER_00: And what was the mood when AI came up?
• 1:31
  SPEAKER_01: That's a very good question.
• 1:33
  SPEAKER_01: So the population is very different from the people I usually work with.
• 1:38
  SPEAKER_01: All of them are engineers and most of the presentations were very focused
• 1:44
  SPEAKER_01: on obviously engineering education and the things that are happening in engineering,
• 1:48
  SPEAKER_01: curriculum design, other kind of things.
• 1:51
  SPEAKER_01: But as we all expect,
• 1:53
  SPEAKER_01: part of the conversation was about AI and the use of AI for engineering,
• 1:58
  SPEAKER_01: both for faculty and also for students.
• 2:01
  SPEAKER_01: So that was, I mean,
• 2:03
  SPEAKER_01: the other part of the presentations were also interesting,
• 2:06
  SPEAKER_01: but that part was the section of the conference that I have the most experience
• 2:12
  SPEAKER_01: with.
• 2:13
  SPEAKER_01: And it's also very connected to all of the things that we are doing in React,
• 2:17
  SPEAKER_01: our project about the use of AI for engineering education.
• 2:22
  SPEAKER_00: And what I love about this is one of the things that I always talk about
• 2:27
  SPEAKER_00: with junior faculty and with graduate students is you've got to find the place
• 2:33
  SPEAKER_00: where people are interested in what you're interested in.
• 2:35
  SPEAKER_00: And you've got to find Basically, you've got to find your people.
• 2:40
  SPEAKER_00: And this was an opportunity to find at least some of your people, right?
• 2:44
  SPEAKER_00: Absolutely.
• 2:45
  SPEAKER_00: And that's really useful.
• 2:47
  SPEAKER_00: So that's great.
• 2:48
  SPEAKER_00: So what was the loudest theme across the sessions that you attended?
• 2:54
  SPEAKER_00: Obviously, it was AI because you already told us that.
• 2:57
  SPEAKER_00: But what was the tone and tenor of that conversation?
• 3:04
  SPEAKER_00: And was it the thing that you expected to hear before you went?
• 3:10
  SPEAKER_01: One of the things that I haven't thought about before attending that conference is
• 3:15
  SPEAKER_01: that most of these people are engineers.
• 3:17
  SPEAKER_01: So they mostly lean toward the technical side of AI.
• 3:22
  SPEAKER_01: So I attended a workshop which was about designing an agentic AI to do stuff
• 3:28
  SPEAKER_01: for you, to design a website for your student.
• 3:31
  SPEAKER_01: And that was very interesting.
• 3:33
  SPEAKER_01: I have a background in computer software engineering, so that part really resonated with me.
• 3:38
  SPEAKER_01: Things have changed a lot since I graduated, obviously.
• 3:42
  SPEAKER_01: But again,
• 3:43
  SPEAKER_01: that part, I mean,
• 3:44
  SPEAKER_01: working with GitHub,
• 3:45
  SPEAKER_01: creating a code space or trying to implement a project, it was beyond prompting.
• 3:52
  SPEAKER_01: I mean, you reach a point that you understand, okay,
• 3:55
  SPEAKER_01: prompting is good,
• 3:56
  SPEAKER_01: but I need to have some technical knowledge to know how to operationalize all
• 4:00
  SPEAKER_01: of those things.
• 4:02
  SPEAKER_01: So that was a new part of this conference.
• 4:04
  SPEAKER_01: And now I have some ideas to work more in that area and
• 4:09
  SPEAKER_01: to strengthen my coding abilities,
• 4:13
  SPEAKER_01: coding more with GitHub to see how can I implement a kind of agent with AI.
• 4:19
  SPEAKER_01: So from that perspective was really interesting.
• 4:23
  SPEAKER_01: In the same session that Dr.
• 4:25
  SPEAKER_01: Pitko and I presented our research,
• 4:27
  SPEAKER_01: we had four other presentations that were about AI and I explained more about them.
• 4:33
  SPEAKER_00: Yeah, in a sec.
• 4:34
  SPEAKER_00: So that's really interesting and that's really important that we are all kind
• 4:41
  SPEAKER_00: of working within a certain bubble where people are doing similar things and
• 4:45
  SPEAKER_00: the advantage of going,
• 4:47
  SPEAKER_00: I mean, again, And going back to why would you go to a conference like that?
• 4:50
  SPEAKER_00: Definitely a regional conference.
• 4:52
  SPEAKER_00: Regional conferences are great, but we often try to go to national or international.
• 4:57
  SPEAKER_00: But it's the chance to interact and get some of that hands-on knowledge
• 5:03
  SPEAKER_00: But what I really want to push on,
• 5:24
  SPEAKER_00: For the people listening is that there are layers here.
• 5:27
  SPEAKER_00: And as you advance, this is a question I get often.
• 5:31
  SPEAKER_00: People say, oh,
• 5:32
  SPEAKER_00: I've created this thing because I told AI to make a website or to make me
• 5:38
  SPEAKER_00: an application.
• 5:39
  SPEAKER_00: I'm not sure how to take that and make it something I can share with others safely
• 5:46
  SPEAKER_00: and And I can park it somewhere where it lives and it's independent
• 5:51
  SPEAKER_00: and it doesn't just live inside cloud or inside everything.
• 5:54
  SPEAKER_00: And this is where you need some technical knowledge.
• 5:57
  SPEAKER_00: You need to understand what a back end of a software like that should look like
• 6:02
  SPEAKER_00: and what to even ask for.
• 6:05
  SPEAKER_00: So even if the AI can code it or code most of it,
• 6:10
  SPEAKER_00: you still need to understand what you're looking for,
• 6:12
  SPEAKER_00: what is the structure,
• 6:14
  SPEAKER_00: how do you want to protect it,
• 6:15
  SPEAKER_00: and obviously the issues of privacy and everything else around it.
• 6:18
  SPEAKER_00: So I think that's really incisive.
• 6:21
  SPEAKER_00: And if you're working with AI and you're getting to the point
• 6:23
  SPEAKER_00: that you're making things,
• 6:25
  SPEAKER_00: this is where a little bit of technical knowledge can take you a very long way.
• 6:30
  SPEAKER_01: Absolutely.
• 6:30
  SPEAKER_01: That's very right.
• 6:32
  SPEAKER_01: And the other thing is that you reach a point where you understand you need
• 6:37
  SPEAKER_01: to use one of the paid versions of the AI because with a free version,
• 6:41
  SPEAKER_01: you are very limited in terms of the credits and everything.
• 6:45
  SPEAKER_01: So if you want to operationalize something and use it in real life,
• 6:51
  SPEAKER_01: you need to have a paid version of AI.
• 6:53
  SPEAKER_00: So tell me about the session that we presented it.
• 6:56
  SPEAKER_00: We, I was of course there in spirit, not in body.
• 6:59
  SPEAKER_00: Yeah.
• 7:01
  SPEAKER_00: Tell me a little bit.
• 7:02
  SPEAKER_00: So let's start with a little bit of description of our paper and then
• 7:06
  SPEAKER_00: the other papers in the session.
• 7:09
  SPEAKER_01: So our paper was a qualitative study.
• 7:13
  SPEAKER_01: We interviewed four undergraduate engineering students.
• 7:19
  SPEAKER_01: And then we used two framework,
• 7:22
  SPEAKER_01: two theoretical framework,
• 7:23
  SPEAKER_01: human-AI interaction and also entrustment framework to analyze what they got
• 7:29
  SPEAKER_01: from students.
• 7:31
  SPEAKER_01: And it was mostly about the delegation and which responsibilities we want AI to take.
• 7:39
  SPEAKER_01: So we explored how students used AI in their personal life,
• 7:44
  SPEAKER_01: in their professional life,
• 7:46
  SPEAKER_01: and how they decided to selectively use AI for doing different things.
• 7:52
  SPEAKER_00: Yeah, and I think about that.
• 7:54
  SPEAKER_00: I mean, Ethan Nolik talks about the jagged edge of AI.
• 7:58
  SPEAKER_00: And I think that beyond that jagged edge,
• 8:01
  SPEAKER_00: there's also this permeable,
• 8:04
  SPEAKER_00: very flexible border between What the AI is doing and what we're doing
• 8:11
  SPEAKER_00: and how much we're letting it and then pulling back and figuring out what are
• 8:16
  SPEAKER_00: the things that we, again,
• 8:17
  SPEAKER_00: going back to entrustment,
• 8:18
  SPEAKER_00: what are the things that you trusted to do and what are the things
• 8:21
  SPEAKER_00: that we're keeping for ourselves.
• 8:23
  SPEAKER_00: And this is not a permanent hard border.
• 8:27
  SPEAKER_00: We're constantly testing it and we're constantly trying things
• 8:31
  SPEAKER_00: and feeling comfortable or uncomfortable around it.
• 8:35
  SPEAKER_00: And that's an ongoing conversation between us and AI and each person is at a different point.
• 8:42
  SPEAKER_00: All right, tell us about the other papers in that session.
• 8:45
  SPEAKER_01: So I'm really trying to talk briefly about those sessions,
• 8:49
  SPEAKER_01: lots of useful knowledge, but one of them was between opportunity and over-reliance.
• 8:55
  SPEAKER_01: So the main thing that they talked about is that faculty weren't simply anti-AI.
• 9:01
  SPEAKER_01: Mm-hmm.
• 9:24
  SPEAKER_01: The other session was about,
• 9:28
  SPEAKER_01: or the other presentation was about longitudinal patterns of generative AI use
• 9:33
  SPEAKER_01: in advanced software engineering.
• 9:36
  SPEAKER_01: And it showed faculty have extremely low trust in students' responsible AI use.
• 9:42
  SPEAKER_01: And so they actually follow the students longitudinally and found
• 9:49
  SPEAKER_01: that they were using AI more for understanding than generation.
• 9:54
  SPEAKER_01: So their verification increased over time.
• 9:59
  SPEAKER_01: And I thought putting those things together, I mean, was interesting.
• 10:05
  SPEAKER_01: The other one was teaching agentic AI as a reliability engineering discipline.
• 10:11
  SPEAKER_01: So for this one, my favorite example from the conference was actually an AI failure.
• 10:19
  SPEAKER_01: The agent confidently told them that it had successfully built something,
• 10:23
  SPEAKER_01: but nothing had been deployed.
• 10:26
  SPEAKER_01: And the instructor point was,
• 10:27
  SPEAKER_01: don't verify what the AI says it did, just verify the actual artifact.
• 10:34
  SPEAKER_01: And I thought that was a powerful way of thinking about AI literacy.
• 10:42
  SPEAKER_01: Last session that I'm going to talk
• 10:43
  SPEAKER_01: about was AI-Augmented Entrepreneurial Mindset-Based In-Class Activities.
• 10:51
  SPEAKER_00: Yeah, that's a- Yes, that's a mouthful.
• 10:54
  SPEAKER_01: Yes.
• 10:56
  SPEAKER_01: So students in a digital communication course use AI within two structured in-class activities.
• 11:03
  SPEAKER_01: And one thing I appreciated was that the student reflections complicated the idea
• 11:09
  SPEAKER_01: that simply adding AI makes an activity more engaging.
• 11:13
  SPEAKER_01: Students recognized the efficiency,
• 11:16
  SPEAKER_01: but some actually said,
• 11:18
  SPEAKER_01: The speed and the time pressure made them consume information less deeply
• 11:23
  SPEAKER_01: or feel anxious.
• 11:25
  SPEAKER_01: So AI-supported inquiry isn't automatically curiosity-driven inquiry.
• 11:32
  SPEAKER_00: So what I'm hearing you say is we're all having very similar questions.
• 11:37
  SPEAKER_00: We want our students to learn about AI and to use AI,
• 11:40
  SPEAKER_00: but we're really worried that it replaces their thinking.
• 11:44
  SPEAKER_01: Exactly.
• 11:46
  SPEAKER_00: If you look at all of this,
• 11:51
  SPEAKER_00: what is it that you think you heard that you haven't thought about before
• 11:57
  SPEAKER_00: or haven't thought as deeply about before?
• 12:01
  SPEAKER_01: That's a very good question.
• 12:04
  SPEAKER_01: All of the episodes, I'm the host and I'm asking the questions.
• 12:07
  SPEAKER_01: Now you know how hard it is
• 12:09
  SPEAKER_00: to think on your feet.
• 12:10
  SPEAKER_01: Yeah.
• 12:12
  SPEAKER_01: So as I said before, I mean, the technical side was really interesting to me.
• 12:17
  SPEAKER_01: I haven't spent that much time To examine a lot of different technical aspects of AI.
• 12:26
  SPEAKER_01: So that gave me a lot of hints about going and trying and see what happens.
• 12:32
  SPEAKER_01: So that part was specifically agentic AI.
• 12:39
  SPEAKER_00: So you came back with something you want to try and test.
• 12:44
  SPEAKER_00: So what are your next steps?
• 12:47
  SPEAKER_01: I'm trying to understand how I can implement GitHub,
• 12:53
  SPEAKER_01: asking a lot of questions,
• 12:55
  SPEAKER_01: obviously from AI,
• 12:57
  SPEAKER_01: and working with ChatGPT and Gemini to see what tools I can use to start designing
• 13:04
  SPEAKER_01: an agent.
• 13:06
  SPEAKER_00: All right.
• 13:06
  SPEAKER_00: Yeah.
• 13:07
  SPEAKER_00: So that's where we're at.
• 13:08
  SPEAKER_00: We're making agents now.
• 13:09
  SPEAKER_00: We're making agents.
• 13:10
  SPEAKER_00: Secret agents.
• 13:11
  SPEAKER_01: Yes.
• 13:12
  SPEAKER_00: Not telling anybody about it.
• 13:14
  SPEAKER_00: Yeah.
• 13:14
  SPEAKER_00: She just told everybody about that.
• 13:17
  SPEAKER_00: All right.
• 13:18
  SPEAKER_00: So thank you very much for sharing.
• 13:20
  SPEAKER_00: Yeah, yeah.
• 13:21
  SPEAKER_00: And we will see everybody on our next episode.
• 13:24
  SPEAKER_01: Abazade.
• 13:26
  SPEAKER_00: And Guy.
• 13:26
  SPEAKER_01: I and I.