Latest / Azadeh and Guy on AI / Beyond Surface-Level: Teaching AI to Analyze Qualitative Data Deeply

Transcript

Beyond Surface-Level: Teaching AI to Analyze Qualitative Data Deeply · Azadeh and Guy on AI

• 0:00
  SPEAKER_01: Hello, everyone.
• 0:01
  SPEAKER_01: Welcome to a new episode of Azadeh and Guy on AI.
• 0:05
  SPEAKER_00: And today we're back to a theme that we talked about before,
• 0:09
  SPEAKER_00: and that is qualitative analysis with AI.
• 0:12
  SPEAKER_01: Do we want to introduce ourselves?
• 0:14
  SPEAKER_01: Sure, you can introduce yourself.
• 0:16
  SPEAKER_00: If you want to talk about yourself, just say so.
• 0:19
  I just thought that we haven't introduced ourselves for a
• 0:22
  while,
• 0:22
  so maybe for our new audience.
• 0:26
  SPEAKER_01: I'm Azadeh,
• 0:28
  and I'm a first-year PhD student here at the University of
• 0:31
  Nebraska-Lincoln,
• 0:33
  SPEAKER_01: and my specialization is Innovative Learning Technologies.
• 0:36
  SPEAKER_00: And I'm Guy Trinin,
• 0:37
  SPEAKER_00: professor of education,
• 0:40
  SPEAKER_00: teaching in the Innovative Learning Technologies program here at UNL.
• 0:45
  SPEAKER_01: Okay.
• 0:46
  SPEAKER_00: And we're coming to you from Podcast Tower on top of the CHS building.
• 0:54
  SPEAKER_01: Yeah.
• 0:54
  SPEAKER_01: Yeah.
• 0:55
  SPEAKER_00: Third floor.
• 0:56
  SPEAKER_00: Third floor.
• 0:57
  SPEAKER_00: Yeah.
• 0:57
  SPEAKER_00: All the way at the end.
• 0:58
  SPEAKER_00: Anyway.
• 1:00
  SPEAKER_00: We want to talk about qualitative analysis,
• 1:03
  mostly because Azadeh gave me a face when I started talking
• 1:06
  about the fact that AI
• 1:08
  SPEAKER_00: is getting better and I think we should get back to looking at it.
• 1:12
  SPEAKER_00: And we found a lovely paper that was published in 2025.
• 1:15
  SPEAKER_00: So it's fairly recent.
• 1:19
  even though it's 2026 and there's probably something newer
• 1:24
  coming just around the corner.
• 1:25
  And I hope it is because I think that the models we're dealing
• 1:28
  with now are
• 1:29
  SPEAKER_00: considerably more capable.
• 1:31
  SPEAKER_01: And we previously also talked about using AI for qualitative coding.
• 1:36
  SPEAKER_01: I think we talked about it in one of our episodes in season one.
• 1:42
  SPEAKER_00: And we are talking about the paper by Naim and colleagues.
• 1:47
  And they did a very specific study of a step-by-step process of
• 1:53
  using chat GPT in
• 1:55
  SPEAKER_00: thematic analysis.
• 1:57
  So the first thing that I love is that it's very clearly a
• 2:00
  specific methodology.
• 2:01
  SPEAKER_00: It's the Brown and Clark methodology of thematic analysis.
• 2:05
  So Brown and Clark were very gracious in offering six steps to
• 2:09
  begin with.
• 2:09
  SPEAKER_01: Sure.
• 2:10
  So anything that outlines a clear series of steps is always
• 2:15
  easier for AI to follow.
• 2:19
  SPEAKER_00: And if you haven't listened to our last episode,
• 2:22
  SPEAKER_00: which you should,
• 2:24
  one of the conclusions that we keep coming back to is AI is
• 2:27
  better when you are
• 2:28
  SPEAKER_00: asking it to do step by step,
• 2:31
  SPEAKER_00: task by task,
• 2:32
  SPEAKER_00: and you're not putting all of the tasks together,
• 2:34
  SPEAKER_00: at least not yet.
• 2:36
  SPEAKER_00: Maybe one day we will, but right now it muddies the water.
• 2:40
  SPEAKER_01: Yeah,
• 2:40
  and I think it's better for AI and also for the human because
• 2:44
  you can go back,
• 2:45
  SPEAKER_01: you can revise,
• 2:45
  SPEAKER_01: you can see where AI is going.
• 2:48
  And then based on that,
• 2:49
  you can change your prompt or improve or refine your prompt.
• 2:53
  And that reminds me of the work we've done on the same topic
• 2:56
  that we published last
• 2:57
  SPEAKER_00: year and presented at site.
• 3:01
  SPEAKER_00: If you're coming to site, which is next week, please come talk to us.
• 3:05
  SPEAKER_00: We're both going to be there, but only Wednesday.
• 3:08
  SPEAKER_00: I'm sorry.
• 3:08
  SPEAKER_00: Not going to be there on Tuesday.
• 3:10
  SPEAKER_00: Got to teach.
• 3:12
  But what we found was that we needed to prompt a few times
• 3:17
  until we got what we wanted.
• 3:19
  SPEAKER_00: because it was,
• 3:21
  SPEAKER_00: again,
• 3:21
  SPEAKER_00: muddying the water and not doing exactly what we wanted and even invading
• 3:25
  SPEAKER_00: quotations,
• 3:26
  SPEAKER_00: which now is a lot easier to prevent.
• 3:29
  SPEAKER_00: But that's always a danger.
• 3:31
  SPEAKER_00: So going step by step,
• 3:32
  SPEAKER_00: as Naim did here,
• 3:34
  is helping control for any kind of errors like that and stop it
• 3:37
  before you're
• 3:38
  SPEAKER_00: moving to the next step.
• 3:40
  SPEAKER_01: Exactly, yes.
• 3:41
  SPEAKER_01: And when we are talking about those six steps,
• 3:44
  SPEAKER_01: we mean selection of quotations,
• 3:46
  SPEAKER_01: keywords,
• 3:47
  SPEAKER_01: coding,
• 3:48
  SPEAKER_01: themes,
• 3:49
  SPEAKER_01: conceptualization,
• 3:50
  SPEAKER_01: and finally,
• 3:51
  SPEAKER_01: development of conceptual model.
• 3:54
  SPEAKER_00: And in their model,
• 3:58
  SPEAKER_00: after each phase,
• 3:59
  SPEAKER_00: you are doing some work to make sure that it's accurate,
• 4:03
  SPEAKER_00: right?
• 4:05
  SPEAKER_01: Yeah.
• 4:05
  SPEAKER_00: And on top of that,
• 4:06
  SPEAKER_00: what they've done is through the prompting,
• 4:09
  SPEAKER_00: they do something that they sometimes call post training,
• 4:12
  SPEAKER_00: but really it's prompting.
• 4:14
  SPEAKER_01: Yeah.
• 4:14
  SPEAKER_01: Yeah.
• 4:15
  SPEAKER_00: And making sure that the AI has in its memory
• 4:20
  SPEAKER_00: the steps and the theoretical assumptions about this kind of work.
• 4:26
  So it's not devoid of theory and it's not devoid of
• 4:28
  understanding the conceptual
• 4:31
  framework around which you organize this specific qualitative
• 4:36
  inquiry with the idea
• 4:37
  SPEAKER_00: that AI knows about everything,
• 4:40
  SPEAKER_00: right?
• 4:40
  SPEAKER_00: It's read almost everything that's been published, at least on the web.
• 4:44
  So it has grounded theory and thematic analysis and 17 other
• 4:49
  variations on qualitative.
• 4:51
  SPEAKER_00: And if we are not very direct with it,
• 4:53
  SPEAKER_00: it can start drifting as you're prompting into other modes.
• 4:57
  SPEAKER_00: And so this helps put some frame around the work that's being done.
• 5:02
  SPEAKER_01: Which is very important when you are working with AI.
• 5:05
  SPEAKER_01: I mean,
• 5:06
  SPEAKER_01: putting boundaries and making AI understand the context of your work,
• 5:11
  especially when you are talking about qualitative analysis is
• 5:15
  super important,
• 5:16
  SPEAKER_01: I think.
• 5:17
  And I think that the other thing that is important here is how
• 5:21
  structured is your data.
• 5:25
  SPEAKER_00: Depending on what you collected and how it is collected,
• 5:29
  one of the things that I like to do when I start with
• 5:33
  qualitative inquiry is
• 5:35
  And,
• 5:35
  you know,
• 5:36
  I'm raising my hand,
• 5:37
  probably the wrong hand,
• 5:38
  but I'm raising my hand.
• 5:39
  SPEAKER_00: I'm a quantitative researcher originally.
• 5:42
  SPEAKER_00: I have done some qualitative work.
• 5:44
  We have done together some qualitative work,
• 5:46
  but I inherently look at the world in units.
• 5:49
  And so one of the first things that I always try to do is try
• 5:53
  to think about what
• 5:54
  SPEAKER_00: is that unit of analysis?
• 5:56
  SPEAKER_00: What are the boundaries of the quotations?
• 5:58
  SPEAKER_00: And this is one of the things they do here.
• 6:00
  SPEAKER_01: Yeah, we had a lot of discussions about that guy while coding...
• 6:04
  SPEAKER_01: different things.
• 6:05
  SPEAKER_01: Yeah.
• 6:05
  SPEAKER_01: What is the unit of analysis?
• 6:07
  SPEAKER_01: Should we code a whole paragraph or just a sentence, sometimes a phrase?
• 6:13
  SPEAKER_00: Yeah.
• 6:14
  And the more you can dictate that to AI,
• 6:17
  the clearer the analysis would be.
• 6:20
  So that's one of the first phases that they did was selecting
• 6:23
  the quotations.
• 6:25
  SPEAKER_01: Exactly.
• 6:26
  SPEAKER_01: Yes.
• 6:26
  SPEAKER_00: And then they really specified the qualities at each level.
• 6:34
  So they went into this description of what are the qualities of
• 6:39
  each one of those steps.
• 6:41
  SPEAKER_00: And they have the...
• 6:45
  SPEAKER_00: seven,
• 6:46
  SPEAKER_00: eight steps,
• 6:47
  SPEAKER_00: I don't know,
• 6:48
  10 steps of feeding and familiarizing ChatGPT with a topic and
• 6:53
  then selecting the
• 6:54
  SPEAKER_00: quotations.
• 6:55
  SPEAKER_00: And so this is a slow process of adding to it.
• 6:59
  And then in each step,
• 7:01
  they have,
• 7:02
  you know,
• 7:03
  in keywords,
• 7:04
  they have the six R framework,
• 7:08
  right?
• 7:08
  SPEAKER_00: Yeah.
• 7:08
  SPEAKER_00: And that's one set of six R's.
• 7:11
  SPEAKER_00: And then in step three in coding,
• 7:13
  SPEAKER_00: They again have six R's.
• 7:15
  SPEAKER_00: They really love their R's.
• 7:16
  SPEAKER_01: Yes.
• 7:17
  SPEAKER_01: Yes.
• 7:17
  SPEAKER_01: Again.
• 7:18
  SPEAKER_00: And then in themes, they have the four R's.
• 7:22
  SPEAKER_00: So they've got lots of R's.
• 7:23
  SPEAKER_01: Yeah.
• 7:24
  SPEAKER_01: So I love their figure.
• 7:26
  I mean,
• 7:27
  I'm talking about figure two,
• 7:28
  how they explained everything super clearly.
• 7:31
  So talking about coding and those six R's,
• 7:34
  they talk about robust,
• 7:36
  reflective,
• 7:38
  robust.
• 7:39
  SPEAKER_00: Resplendent.
• 7:40
  SPEAKER_01: Resplendent, radical, relevant, and also righteous.
• 7:46
  SPEAKER_00: So, yes.
• 7:47
  SPEAKER_00: So it is really the words they are using, right?
• 7:52
  The terms they are using,
• 7:53
  besides the fact that they want to play with their R.
• 7:55
  Mm-hmm.
• 7:57
  SPEAKER_00: everywhere is that they are going to extreme.
• 8:00
  SPEAKER_00: And that reminds me a little bit of survey methodology.
• 8:03
  SPEAKER_00: When you want a very good reaction,
• 8:06
  you sometimes ask extreme questions because that allows you to
• 8:13
  define things more
• 8:15
  SPEAKER_00: clearly.
• 8:15
  SPEAKER_00: Is it really extreme?
• 8:17
  SPEAKER_00: Right.
• 8:18
  SPEAKER_00: Radical.
• 8:18
  SPEAKER_00: Radical is a big word.
• 8:20
  SPEAKER_00: It has it has a lot of meaning.
• 8:22
  And that kind of analysis allows you to surface not just
• 8:27
  surface things that are
• 8:28
  SPEAKER_00: repeated,
• 8:28
  SPEAKER_00: but actually things that deviate or are surprising.
• 8:33
  SPEAKER_00: And that's part of what we're looking for.
• 8:34
  SPEAKER_00: We're not just looking for confirmation.
• 8:36
  SPEAKER_00: We're looking also for what are the things that are surprising in this.
• 8:39
  SPEAKER_00: kind of frameworks.
• 8:40
  So that's part of the way I think about what do we want AI to
• 8:45
  do and how do we help
• 8:46
  SPEAKER_00: AI see the patterns in a way that's not reductive?
• 8:50
  Because our experience doing this work is AI was very surface
• 8:55
  level and very reductive.
• 8:57
  And we've played with a number of other applications that have
• 9:01
  been built for
• 9:02
  SPEAKER_00: software that does qualitative analysis.
• 9:04
  SPEAKER_00: And it seemed to have functioned at that level as well.
• 9:07
  SPEAKER_01: Yeah.
• 9:08
  SPEAKER_01: Yeah.
• 9:08
  SPEAKER_01: And I think in most of the research that we've been reading so far,
• 9:12
  SPEAKER_01: the researchers found the same thing.
• 9:14
  AI focusing on surface level data and having a kind of issue
• 9:19
  for analyzing more contextual data.
• 9:23
  SPEAKER_00: Yeah.
• 9:24
  And one of the things to say about the name paper is they have
• 9:30
  a very specific
• 9:32
  SPEAKER_00: context that they're exploring,
• 9:34
  SPEAKER_00: and that's customer feedback.
• 9:37
  And customer feedback tend to be fairly short most of the time
• 9:41
  and fairly structured.
• 9:43
  SPEAKER_00: So that's a little bit easier.
• 9:44
  SPEAKER_00: I mean, there are some natural things you're going to attach yourself to.
• 9:50
  SPEAKER_00: Yeah.
• 9:51
  SPEAKER_00: And that's something to remember.
• 9:52
  So the affordance of AI is sometimes may have to do with the
• 9:58
  clarity of the theory
• 9:59
  SPEAKER_00: you're using for the thematic analysis.
• 10:03
  SPEAKER_00: So if you have a strong theory that has been proven again and again,
• 10:07
  SPEAKER_00: I think that's where AI can be highly effective.
• 10:11
  SPEAKER_00: When you have to build a theory, I'm still wondering about that.
• 10:17
  SPEAKER_01: Yeah, exactly.
• 10:20
  And the other things that always I have on my mind is that how
• 10:26
  much do we need to
• 10:27
  SPEAKER_01: immerse AI with enough,
• 10:30
  SPEAKER_01: I don't know if enough is the correct word,
• 10:32
  SPEAKER_01: but with enough contextual information to make sure that it's coding the
• 10:37
  SPEAKER_01: transcriptions or anything else the way that we want,
• 10:40
  SPEAKER_01: I mean,
• 10:41
  SPEAKER_01: in terms of depth.
• 10:42
  SPEAKER_00: Yeah.
• 10:43
  SPEAKER_00: Yeah.
• 10:44
  SPEAKER_00: But one of the things I think I appreciate more than anything,
• 10:47
  SPEAKER_00: and this is why we chose this one.
• 10:49
  SPEAKER_00: There are other articles out there,
• 10:51
  SPEAKER_00: but they really introduced that idea of an AI toolkit for analysis.
• 10:56
  SPEAKER_00: Yeah.
• 10:57
  SPEAKER_00: And they said, here are the steps.
• 10:58
  SPEAKER_00: You can go through the steps.
• 10:59
  Now,
• 11:00
  one of the things that we know has happened over the last year
• 11:03
  or so is the specificity.
• 11:06
  SPEAKER_01: Yeah.
• 11:07
  SPEAKER_01: Yeah.
• 11:08
  SPEAKER_00: of prompting.
• 11:09
  SPEAKER_01: Happens to me all the time.
• 11:13
  SPEAKER_00: The specificity of prompting used to be a huge thing.
• 11:20
  SPEAKER_00: It's still an important thing,
• 11:22
  SPEAKER_00: but its importance in,
• 11:24
  SPEAKER_00: for example,
• 11:24
  SPEAKER_00: saying,
• 11:25
  SPEAKER_00: imagine you're a qualitative researcher.
• 11:27
  That is not needed anymore,
• 11:28
  but the steps themselves are really,
• 11:31
  really needed.
• 11:32
  SPEAKER_00: So that's where you want to be careful and not just say,
• 11:35
  SPEAKER_00: Do a thematic analysis and tell me what happened.
• 11:37
  SPEAKER_01: No, no, no, no, no, no.
• 11:39
  SPEAKER_01: Don't do that.
• 11:39
  SPEAKER_01: Don't do that.
• 11:41
  All right.
• 11:42
  So now I'm shifting and I want to ask you,
• 11:45
  we talked about qualitative analysis.
• 11:46
  SPEAKER_00: What have you been using AI for?
• 11:50
  SPEAKER_01: Let me see.
• 11:51
  SPEAKER_01: I always have to pause when I want to answer that question.
• 11:54
  SPEAKER_01: I don't know if it is a very innovative use of AI,
• 12:02
  SPEAKER_01: but sometimes I use AI for...
• 12:05
  SPEAKER_01: I'm submitting a lot of IRB applications,
• 12:09
  SPEAKER_01: which is a tedious task,
• 12:10
  SPEAKER_01: to be honest.
• 12:11
  SPEAKER_01: And I use a lot of AI for formulating the responses.
• 12:16
  SPEAKER_00: Yeah.
• 12:16
  SPEAKER_00: And that's something we've both been using.
• 12:18
  SPEAKER_00: And that is when it's filling out forms,
• 12:21
  SPEAKER_00: we know what we want to say,
• 12:23
  but we can say it in two short sentences and then it builds a
• 12:26
  paragraph around
• 12:27
  SPEAKER_00: that.
• 12:27
  SPEAKER_00: You check it, you fix it.
• 12:28
  SPEAKER_00: So it doesn't say anything you don't want.
• 12:30
  But the speed at which we've been able to do these kind of
• 12:34
  tasks has improved tremendously.
• 12:37
  SPEAKER_00: We did the same thing.
• 12:37
  SPEAKER_00: We did a data plan.
• 12:42
  as to how do we mean to preserve the data,
• 12:45
  store the data,
• 12:46
  all of these things.
• 12:47
  SPEAKER_00: It's very technical.
• 12:49
  SPEAKER_00: It's very formulaic.
• 12:51
  And so we know what we want to say and we just have AI help us
• 12:55
  say it in full
• 12:55
  SPEAKER_00: sentences instead of bullet points.
• 12:58
  I'm not sure that the people that receive this don't use AI to
• 13:02
  turn it back into
• 13:03
  SPEAKER_00: bullet points,
• 13:04
  SPEAKER_00: but they seem to want a more robust description,
• 13:07
  SPEAKER_00: so here they are.
• 13:09
  SPEAKER_01: How about you guys?
• 13:10
  SPEAKER_00: Well,
• 13:11
  SPEAKER_00: I've been using it in a few ways that,
• 13:13
  SPEAKER_00: so the technical aspect,
• 13:15
  SPEAKER_00: I've been using it to answer emails that have been challenging to me.
• 13:20
  So I've received a few emails that were demanding or a little
• 13:25
  bit unpleasant.
• 13:26
  SPEAKER_01: Yeah, yeah.
• 13:26
  SPEAKER_00: And in the moment,
• 13:28
  it's very hard for me to be gracious in the way I respond and
• 13:31
  think about all of
• 13:33
  SPEAKER_00: the aspects.
• 13:34
  SPEAKER_00: So sometimes what I do is I say to AI,
• 13:36
  SPEAKER_00: this is what I want to say,
• 13:37
  but can you help me say it in a way that will be nicer and will
• 13:42
  help me respond in
• 13:43
  SPEAKER_00: a thoughtful way?
• 13:44
  SPEAKER_00: to this specific matter.
• 13:47
  And that's kind of on my mind as I'm thinking about creating
• 13:53
  that distance so I can
• 13:55
  SPEAKER_00: say things,
• 13:56
  SPEAKER_00: it adds grace to my life.
• 13:59
  SPEAKER_01: Yeah, sure.
• 14:00
  SPEAKER_00: So those are the ways we've used AI.
• 14:03
  SPEAKER_01: Lately, yeah.
• 14:04
  SPEAKER_00: Yes, lately.
• 14:06
  SPEAKER_00: And that's partially because it's semester break and we're enjoying.
• 14:11
  SPEAKER_00: The sun came out today.
• 14:13
  SPEAKER_01: Nice weather, yes.
• 14:14
  SPEAKER_00: It's finally not cold.
• 14:16
  SPEAKER_00: May there not be snow until next November.
• 14:24
  SPEAKER_01: Yes.
• 14:25
  SPEAKER_01: Thank you so much for listening to us.
• 14:27
  SPEAKER_00: Azadeh.
• 14:28
  SPEAKER_01: And Guy.
• 14:29
  SPEAKER_00: On AI.
• 14:30
  SPEAKER_01: Bye.