Latest / Azadeh and Guy on AI / From Data Overload to Clarity

Transcript

From Data Overload to Clarity · Azadeh and Guy on AI

• 0:00
  SPEAKER_01: I do love podcasting.
• 0:03
  SPEAKER_01: This is all I have to say.
• 0:04
  SPEAKER_01: I do love podcasting.
• 0:05
  SPEAKER_01: So welcome to the Azadeh and Guy on AI,
• 0:08
  the show that explores how artificial intelligence is
• 0:11
  transforming research in
• 0:13
  SPEAKER_01: education and beyond.
• 0:14
  SPEAKER_01: I'm Guy.
• 0:15
  SPEAKER_00: And I'm Azadeh.
• 0:17
  SPEAKER_00: I'm a PhD student at UNL and Guy is my PhD advisor.
• 0:21
  And today we're really jumping into the way we think about AI
• 0:26
  in the research
• 0:27
  SPEAKER_01: workflow with a specific emphasis on qualitative analysis,
• 0:30
  SPEAKER_01: not because we think it's the most important thing,
• 0:32
  SPEAKER_01: but it's the thing we've been doing lately.
• 0:34
  SPEAKER_01: So there's a lot there, but before we get there,
• 0:38
  We really want to cover some of the new things we've been
• 0:41
  seeing around AI.
• 0:42
  SPEAKER_01: And there's lots of these.
• 0:44
  And there are publications every day and new models and
• 0:47
  everything coming out today
• 0:48
  SPEAKER_01: and yesterday and the day before.
• 0:50
  SPEAKER_01: I mean, it's literally a world that changes by the day.
• 0:53
  SPEAKER_01: But today we want to talk about...
• 0:58
  SPEAKER_01: Notebook.lm and the new feature in Notebook.lm.
• 1:00
  SPEAKER_00: That's very true, yes.
• 1:01
  We want to talk about Notebook.lm and its new feature,
• 1:06
  which is Video Overview,
• 1:08
  right?
• 1:09
  SPEAKER_00: Yes.
• 1:10
  SPEAKER_01: So Notebook.lm is one of my favorites.
• 1:13
  SPEAKER_01: I have a lot of favorites.
• 1:15
  SPEAKER_00: That is true.
• 1:15
  SPEAKER_00: I was going to say, could you pick one of them?
• 1:19
  SPEAKER_01: No, I can't.
• 1:20
  SPEAKER_01: That's why I have four kids.
• 1:21
  SPEAKER_01: I have four kids because I can't pick just one.
• 1:23
  SPEAKER_01: I want to be able to say I'm excited about all of them.
• 1:27
  SPEAKER_01: And I am.
• 1:28
  SPEAKER_01: And so Notebook LM has been a product that came out last year.
• 1:33
  SPEAKER_01: We loved it from the moment it got out.
• 1:36
  SPEAKER_01: And we're not special.
• 1:37
  SPEAKER_01: I mean, Azadeh is special.
• 1:39
  SPEAKER_00: I'm not that special.
• 1:41
  SPEAKER_01: And as a result, many, many people loved it.
• 1:45
  And I think even the people who may not work LM when they were
• 1:48
  interviewed,
• 1:49
  SPEAKER_01: not by us yet,
• 1:50
  SPEAKER_01: but maybe if we get famous one day,
• 1:52
  SPEAKER_01: they'll come on our podcast,
• 1:55
  SPEAKER_01: really said that they were even surprised by how popular it became.
• 1:58
  SPEAKER_01: But it is a great little application of AI.
• 2:02
  SPEAKER_01: And the reason I loved it, it was immediately usable.
• 2:06
  SPEAKER_01: It was great for education,
• 2:09
  and it was a really good illustration of how AI can be useful
• 2:13
  in your life right
• 2:14
  SPEAKER_01: now.
• 2:15
  SPEAKER_00: Yeah, and I think the whole platform is very intuitive.
• 2:18
  SPEAKER_00: I mean, I've seen many people using Notebook LM.
• 2:21
  SPEAKER_00: The first time they opened the platform,
• 2:24
  SPEAKER_00: they just start using it,
• 2:26
  and they do not need to do a long training or something like
• 2:30
  that to understand all
• 2:31
  SPEAKER_00: of the features.
• 2:32
  SPEAKER_00: So it's, yeah, you're right.
• 2:34
  SPEAKER_00: It's one of the most favorite tools, AI tools.
• 2:37
  SPEAKER_01: And you can create lots of notebooks.
• 2:39
  SPEAKER_01: And in each one of the notebooks, you can bring some information in.
• 2:43
  SPEAKER_01: It can be audio.
• 2:44
  SPEAKER_01: It can be most people are bringing PDFs, especially in the beginning.
• 2:48
  SPEAKER_01: We were bringing text and being able to do a few things with the text.
• 2:52
  And the new thing is on top of having the ability to have a
• 2:57
  quiz or to have a chat
• 2:59
  SPEAKER_01: with the text,
• 3:00
  SPEAKER_01: which is kind of wonderful.
• 3:01
  You can ask it questions and it answers based on the text that
• 3:06
  you've provided.
• 3:07
  SPEAKER_01: And then it also provides notes for studying.
• 3:12
  SPEAKER_01: Again, great.
• 3:14
  SPEAKER_01: And we had this podcast feature that a lot of people loved.
• 3:18
  SPEAKER_01: And we were a little apprehensive about it.
• 3:20
  SPEAKER_01: It was both really interesting.
• 3:24
  And one time we listened as we were traveling across Nebraska,
• 3:28
  we were listening to a podcast.
• 3:30
  SPEAKER_01: And it was like, after a while, it gets repetitive.
• 3:33
  SPEAKER_01: Yeah.
• 3:33
  SPEAKER_00: Yeah, yeah.
• 3:34
  SPEAKER_00: And I remember that it mispronounced some of the key terms, actually.
• 3:40
  SPEAKER_01: So it wasn't without problems, which is a great way to explore AI.
• 3:44
  SPEAKER_01: We are never saying that AI is perfect.
• 3:46
  SPEAKER_01: It's just it has some affordances.
• 3:48
  And if listening is your thing,
• 3:50
  they really broke it down and they did all of these things.
• 3:52
  SPEAKER_01: So now they added this feature and you played with it a little bit.
• 3:55
  SPEAKER_01: So what does it look like?
• 3:58
  SPEAKER_00: I used it for checking some of the PDFs.
• 4:02
  SPEAKER_00: I mostly,
• 4:03
  SPEAKER_00: to be honest,
• 4:03
  SPEAKER_00: I prefer to read the papers myself,
• 4:08
  but sometimes I try the AI tools just because I want to see how
• 4:12
  their features work
• 4:13
  SPEAKER_00: and what are their capabilities.
• 4:16
  SPEAKER_00: So I think it's been really helpful.
• 4:18
  SPEAKER_00: I think I used this one time for summarizing a couple of papers,
• 4:23
  SPEAKER_00: and I was kind of happy with the quality of the results.
• 4:26
  SPEAKER_00: So it could be very helpful, yeah.
• 4:29
  And they really initiated that idea that you can take anything
• 4:34
  and everything and
• 4:34
  SPEAKER_01: turn it into a podcast.
• 4:36
  SPEAKER_01: Sure.
• 4:38
  SPEAKER_01: is what they call, what do they call it?
• 4:40
  SPEAKER_01: Video?
• 4:41
  SPEAKER_00: Video overviews.
• 4:42
  SPEAKER_01: Video overviews.
• 4:43
  And so just so you're clear,
• 4:45
  I mean,
• 4:46
  we had expectations when we just heard about this.
• 4:48
  SPEAKER_01: It's like,
• 4:48
  SPEAKER_01: it's going to be this wonderful movie,
• 4:51
  SPEAKER_00: and it's going to have all of these characters.
• 4:53
  SPEAKER_00: Yeah, I was really excited, yes.
• 4:55
  SPEAKER_01: And that's not exactly what it is.
• 4:57
  SPEAKER_01: It doesn't mean we don't think it's awesome, because it is.
• 5:00
  SPEAKER_01: But it is basically PowerPoint.
• 5:04
  SPEAKER_00: Yeah.
• 5:04
  SPEAKER_01: So it creates a set of slides to go with the audio.
• 5:09
  SPEAKER_01: So you've got audio and a set of slides.
• 5:11
  SPEAKER_01: This is why I love it.
• 5:12
  SPEAKER_01: And it's exactly what you said.
• 5:14
  SPEAKER_01: And that is when you're reading hard texts,
• 5:18
  SPEAKER_01: listening is an okay strategy,
• 5:22
  SPEAKER_01: but it's actually less good than reading.
• 5:25
  SPEAKER_01: Sure.
• 5:26
  Because in reading,
• 5:28
  there's,
• 5:28
  first of all,
• 5:28
  another way to get the information.
• 5:30
  So you're getting multimodal information if you are getting
• 5:34
  these overviews.
• 5:35
  SPEAKER_01: But more to the point, listening is linear.
• 5:38
  SPEAKER_01: That is, there's a beginning, middle, and end.
• 5:41
  And most of us are not stopping mid listen and moving it back
• 5:46
  and re-listening to
• 5:47
  SPEAKER_01: the last paragraph.
• 5:48
  SPEAKER_01: We don't actually know even where it is.
• 5:50
  SPEAKER_01: So if we miss something, we might do that.
• 5:52
  SPEAKER_01: But most of the time, people listen through.
• 5:54
  SPEAKER_01: And we know that when we're reading hard texts,
• 5:57
  SPEAKER_01: We're not reading through.
• 5:58
  SPEAKER_01: We were reading a sentence.
• 6:00
  SPEAKER_01: We're looking up a word.
• 6:02
  SPEAKER_01: We're clicking on something.
• 6:04
  SPEAKER_01: We're really grappling with the text.
• 6:06
  And again,
• 6:07
  an easy text,
• 6:08
  yes,
• 6:08
  listening to a podcast about murder,
• 6:10
  let's be honest.
• 6:12
  SPEAKER_01: All the podcasts that aren't mine are about murder apparently.
• 6:17
  SPEAKER_01: And when you're listening to a story, you're fine.
• 6:19
  SPEAKER_01: You can follow the story and you'll be fine.
• 6:22
  SPEAKER_01: But when it is a difficult text,
• 6:24
  it turns out that in our brains we're doing all of these
• 6:27
  strategies that allow us
• 6:28
  SPEAKER_01: to keep the meaning.
• 6:30
  SPEAKER_01: And sometimes we jump two pages backwards,
• 6:33
  SPEAKER_01: right,
• 6:34
  SPEAKER_01: to go,
• 6:34
  SPEAKER_01: wait a minute,
• 6:35
  SPEAKER_01: there was something there.
• 6:36
  SPEAKER_01: I need to go back to it so I can understand it.
• 6:38
  SPEAKER_01: Yeah.
• 6:39
  SPEAKER_01: In audio, you can do that as well.
• 6:41
  And I think that the overview bridges some of that by having
• 6:46
  visual text and some
• 6:49
  SPEAKER_01: kind of graphic representation.
• 6:50
  SPEAKER_00: Yeah, yeah.
• 6:51
  SPEAKER_00: It could be helpful.
• 6:52
  I mean,
• 6:54
  not very interesting because it's basically a PowerPoint with
• 6:57
  narration.
• 6:58
  SPEAKER_00: Yeah.
• 6:58
  SPEAKER_00: But, yeah, I can see some uses of that new feature in probably very...
• 7:07
  SPEAKER_00: I would say basic, yeah, applications of that kind of video, yeah.
• 7:13
  SPEAKER_01: Yeah, and let's put Notebook LM where it needs to be.
• 7:16
  SPEAKER_01: Notebook LM is not there to solve the problems of the world.
• 7:20
  It's really a study aid and a way to interrogate some text that
• 7:25
  we're grappling with.
• 7:26
  SPEAKER_01: Mm-hmm.
• 7:27
  SPEAKER_01: It's not the end all and be all of everything,
• 7:29
  SPEAKER_01: but it is a way to get through some complicated text in an easier way,
• 7:35
  SPEAKER_01: in a way that,
• 7:36
  SPEAKER_01: again,
• 7:36
  SPEAKER_01: going back to the point you made before,
• 7:38
  what's great about it is the threshold of skill to get going is
• 7:43
  close to zero.
• 7:44
  SPEAKER_00: Yeah, yeah.
• 7:45
  SPEAKER_01: Which is true of many of the AI features out there.
• 7:48
  SPEAKER_01: One of the things that I loved about them,
• 7:49
  SPEAKER_01: and I think that's the reason that it caught on so fast,
• 7:53
  SPEAKER_01: is the interface was so easy.
• 7:55
  SPEAKER_01: You went to chat GPT and you typed something and you got an answer.
• 7:59
  SPEAKER_01: You didn't need to know anything.
• 8:01
  SPEAKER_01: We're starting to need to know something like…
• 8:04
  SPEAKER_00: Probably that's one of the reasons they are really popular.
• 8:07
  SPEAKER_00: You basically need no training or other thing to start working with them.
• 8:12
  SPEAKER_01: Exactly.
• 8:13
  SPEAKER_01: You get a result immediately.
• 8:14
  And that ability to get a result immediately is critically
• 8:17
  important for people to
• 8:19
  SPEAKER_01: sustain their interest and to say,
• 8:21
  SPEAKER_01: I don't need to know coding.
• 8:23
  SPEAKER_01: It's basically like working in Google.
• 8:25
  SPEAKER_01: We all know how to work Google.
• 8:27
  SPEAKER_01: This is similar.
• 8:28
  SPEAKER_01: It's not the same.
• 8:28
  SPEAKER_01: But if you know Google, you have enough to go.
• 8:32
  SPEAKER_01: And be able to do something.
• 8:33
  SPEAKER_01: So that's a celebration for today.
• 8:35
  SPEAKER_01: It's not a new feature.
• 8:36
  And by the time you hear this,
• 8:38
  it'll be probably an old feature,
• 8:40
  like two weeks old.
• 8:41
  SPEAKER_01: And that's in the AI calendar.
• 8:45
  Two weeks is like,
• 8:46
  I don't know,
• 8:48
  a hundred years,
• 8:48
  a century in development.
• 8:50
  SPEAKER_01: But we're still excited about it.
• 8:51
  SPEAKER_00: Yeah.
• 8:51
  SPEAKER_00: And the other thing is that you can rate the videos.
• 8:54
  SPEAKER_00: You can say good video or bad video.
• 8:56
  SPEAKER_00: So you are providing them with feedback.
• 8:58
  SPEAKER_00: Yes.
• 8:59
  SPEAKER_01: So maybe they should pay us for using it.
• 9:01
  SPEAKER_01: But then again, we're getting it for free.
• 9:03
  SPEAKER_01: So I don't know.
• 9:04
  SPEAKER_01: It's always, but we are retraining and training and retraining the...
• 9:10
  The AI world has been wondering about how do we get better
• 9:15
  without all of this vast
• 9:18
  SPEAKER_01: amount of data that we've already processed.
• 9:20
  So they've processed everything they could get their hands on
• 9:24
  legally and illegally.
• 9:25
  SPEAKER_00: Mm-hmm.
• 9:28
  At this point,
• 9:29
  they're thinking,
• 9:30
  what are the solutions to keep improving?
• 9:31
  And one of the solutions is obviously looking at the queries we
• 9:37
  make and what do we
• 9:38
  SPEAKER_01: make of them.
• 9:39
  Another way for them to do additional training is those
• 9:43
  thinking modes where the AI
• 9:45
  SPEAKER_01: is actually prompting itself to explore a little bit further.
• 9:49
  SPEAKER_01: And that provided another layer of capacity.
• 9:52
  SPEAKER_01: And,
• 9:52
  SPEAKER_01: of course,
• 9:52
  SPEAKER_01: the last layer is the layer of human training,
• 9:57
  SPEAKER_01: saying this is a good answer,
• 9:58
  SPEAKER_01: this is not a good answer.
• 10:00
  SPEAKER_01: So they're using the hundreds of millions to billions of users,
• 10:04
  SPEAKER_01: definitely billions of queries,
• 10:06
  to start training the AI and give feedback about what works and
• 10:10
  what doesn't work.
• 10:11
  SPEAKER_01: And that makes it stronger.
• 10:12
  SPEAKER_01: So every time you click on it,
• 10:14
  SPEAKER_01: they're getting more data,
• 10:15
  SPEAKER_01: but you're potentially getting a better result at the end.
• 10:18
  SPEAKER_01: You've got to do your own math, I guess.
• 10:20
  SPEAKER_01: Is it worth it?
• 10:21
  SPEAKER_01: Am I working for Google or am I working for myself?
• 10:23
  SPEAKER_01: I've been working for Google myself for a long time.
• 10:26
  SPEAKER_01: Yeah.
• 10:27
  SPEAKER_00: Yeah.
• 10:29
  SPEAKER_01: But Notebook LM is probably my favorite Google feature so far.
• 10:33
  SPEAKER_00: Yeah, I know.
• 10:35
  SPEAKER_00: It's very user-friendly, yeah.
• 10:37
  SPEAKER_01: Yeah, that is definitely there.
• 10:39
  SPEAKER_01: Okay,
• 10:40
  so now we've talked about Notebook LM and we're really excited
• 10:44
  and we're thinking
• 10:45
  SPEAKER_01: about what happens next.
• 10:48
  SPEAKER_01: And I had a chance this week.
• 10:53
  to really dive in with a group of researchers who wanted a
• 10:58
  primer on using AI in
• 11:01
  SPEAKER_01: the research workflow.
• 11:03
  SPEAKER_01: And that was yesterday.
• 11:04
  SPEAKER_01: Okay.
• 11:05
  SPEAKER_01: Yesterday's making a face because she was not invited.
• 11:09
  SPEAKER_01: She was busy.
• 11:10
  SPEAKER_00: Yeah, I was like, what is happening?
• 11:13
  SPEAKER_01: Why didn't I know about this?
• 11:14
  SPEAKER_01: Well...
• 11:15
  SPEAKER_01: That's how it is.
• 11:16
  So they asked me to come and talk about this,
• 11:18
  and there were about,
• 11:20
  I think,
• 11:21
  30 or 40 people.
• 11:23
  SPEAKER_01: This was an online presentation,
• 11:25
  SPEAKER_01: and I was thinking about what would be useful for researchers to know.
• 11:31
  SPEAKER_01: And so there were two features,
• 11:32
  SPEAKER_01: and we're going to talk about one,
• 11:34
  SPEAKER_01: which is the thing we've been grappling with,
• 11:36
  SPEAKER_01: and that is really thinking about what does it mean to get AI help in the
• 11:41
  SPEAKER_01: qualitative coding process.
• 11:43
  SPEAKER_01: Sure.
• 11:47
  I think that as we think about research,
• 11:50
  we're always thinking about how can we do this better?
• 11:53
  SPEAKER_01: How can we do this faster?
• 11:56
  SPEAKER_01: How can we do this more reliably?
• 11:58
  SPEAKER_01: Yeah.
• 11:59
  So those are the things that are on my mind as I'm thinking
• 12:03
  about why even use AI
• 12:05
  SPEAKER_01: in the process,
• 12:06
  SPEAKER_01: right?
• 12:07
  SPEAKER_01: It has to be worth it.
• 12:09
  And the first thing that I talk about is really about this
• 12:14
  concept of where are you
• 12:17
  SPEAKER_01: on the knowledge scale or more specifically thinking about the inverted
• 12:24
  relationship between your level of expertise and how much you
• 12:27
  can get out of AI.
• 12:28
  SPEAKER_01: Yeah.
• 12:29
  And the way I think about it is novices and experts really
• 12:33
  don't need AI or can't
• 12:34
  SPEAKER_01: use AI very effectively.
• 12:35
  SPEAKER_01: So novices, their real challenge is that they are novices.
• 12:41
  So even if they get – even if they prompt AI,
• 12:45
  they have no way to evaluate.
• 12:49
  Verify the accuracy of the data or the knowledge that they are
• 12:53
  getting from AI.
• 12:54
  SPEAKER_00: So –
• 12:55
  SPEAKER_00: they may be even more confused.
• 12:58
  SPEAKER_01: Yeah,
• 12:59
  SPEAKER_01: and they either take it and say,
• 13:01
  SPEAKER_01: looks good to me,
• 13:02
  SPEAKER_01: which often AI does because it's very sure of itself.
• 13:05
  SPEAKER_00: Yeah, I saw the experience with some of my classmates.
• 13:09
  SPEAKER_00: They were trying to learn coding using it.
• 13:12
  SPEAKER_00: and it was a source of confusion.
• 13:14
  I have a bachelor's degree in computer software engineering,
• 13:18
  so using AI was not a huge problem.
• 13:21
  SPEAKER_00: I could remember some of my knowledge,
• 13:22
  SPEAKER_00: past knowledge,
• 13:23
  SPEAKER_00: and then combine it with AI to came up with solutions.
• 13:27
  SPEAKER_00: For other students, it was a kind of a struggle.
• 13:30
  They were just copy and pasting the code that AI gives them,
• 13:34
  so it was a kind of problem.
• 13:36
  They were even more confused and grappling with how we can
• 13:40
  figure out the solution.
• 13:42
  SPEAKER_00: Getting more errors, going back to AI, so it's a kind of vicious cycle.
• 13:47
  SPEAKER_01: And vibe coding is a thing,
• 13:49
  but a little bit of knowledge goes a long way to help it along
• 13:53
  and make sure that
• 13:54
  SPEAKER_01: you don't get stuck and that you know what to do when you do.
• 13:56
  At least for troubleshooting,
• 13:58
  which is one of the most important parts of the coding.
• 14:02
  SPEAKER_01: Yeah,
• 14:02
  SPEAKER_01: because we always,
• 14:03
  SPEAKER_01: and especially when AI can produce code,
• 14:06
  then it's all about problem solving when you've got errors in
• 14:10
  the code that you're
• 14:11
  SPEAKER_01: trying to figure out.
• 14:13
  SPEAKER_01: So that's one side of the equation.
• 14:16
  SPEAKER_01: The other side is the experts.
• 14:17
  SPEAKER_01: Sure.
• 14:18
  SPEAKER_01: And if you are an expert,
• 14:19
  SPEAKER_01: if you've been doing something your whole life,
• 14:22
  SPEAKER_01: you're probably faster,
• 14:24
  SPEAKER_01: more accurate and more capable than AI.
• 14:27
  SPEAKER_01: And in the case of academics, you also have a voice.
• 14:30
  SPEAKER_01: So when you're writing, you have a voice that is your voice.
• 14:34
  SPEAKER_01: And AI is never going to sound like you, even in writing.
• 14:38
  SPEAKER_01: And so...
• 14:39
  SPEAKER_01: You work with it,
• 14:40
  and I've had a few opportunities where I try to use AI for the
• 14:44
  purpose of writing a
• 14:45
  SPEAKER_01: report or something like that,
• 14:46
  SPEAKER_01: and I often find myself as,
• 14:48
  SPEAKER_01: this is overcomplicating things.
• 14:51
  SPEAKER_01: This is what I need to write, and I write it, and it's faster.
• 14:54
  SPEAKER_01: So on both sides of the equation, novice and experts,
• 14:59
  SPEAKER_01: we really, AI is not as useful as we'd like it to be.
• 15:03
  SPEAKER_01: And it's not as helpful.
• 15:05
  It can save you if at the last minute you need something and
• 15:08
  something is better
• 15:08
  SPEAKER_01: than nothing,
• 15:09
  SPEAKER_01: then fine.
• 15:10
  SPEAKER_01: But if you have the time, if you're a novice, you should probably learn.
• 15:13
  And if you're an expert,
• 15:14
  you can probably produce what you need to produce.
• 15:16
  And I think we have evidence for that based on one of our
• 15:19
  recent research.
• 15:20
  SPEAKER_00: I had an interview with a
• 15:23
  with a faculty member in computing school at UNL and they said
• 15:27
  that I never let my
• 15:28
  students use AI tools at the beginning of the course when they
• 15:32
  have no knowledge of
• 15:33
  SPEAKER_00: the concepts or they don't know how to do things.
• 15:35
  And when I give them some quizzes and then I make sure that
• 15:40
  they have reached to a
• 15:41
  SPEAKER_00: specific level of knowledge,
• 15:43
  then they are allowed to use AI and to compare results with
• 15:47
  their own solution to
• 15:48
  SPEAKER_00: see how it works.
• 15:49
  SPEAKER_01: So this is a solution.
• 15:51
  This is an intuitive understanding of we need to cross a
• 15:54
  certain threshold before we could.
• 15:56
  SPEAKER_01: And then in the middle of the distribution,
• 15:58
  SPEAKER_01: when you have some knowledge so you can evaluate,
• 16:00
  SPEAKER_01: but you don't remember everything.
• 16:02
  SPEAKER_01: I mean,
• 16:02
  SPEAKER_01: I think your example as a software engineer is like,
• 16:05
  SPEAKER_01: do you remember all the languages and all the intricacies?
• 16:09
  SPEAKER_00: Yeah.
• 16:10
  SPEAKER_01: Probably not.
• 16:11
  SPEAKER_00: Yes.
• 16:12
  SPEAKER_01: And so, but...
• 16:14
  SPEAKER_01: you know enough.
• 16:15
  SPEAKER_01: So this is where it's super helpful,
• 16:17
  and this is where in the research cycle and when you're
• 16:22
  thinking about research,
• 16:23
  SPEAKER_01: this is where it can be super helpful.
• 16:26
  And so this is really for us as we're thinking about what do we
• 16:32
  do with AI in research.
• 16:33
  SPEAKER_01: That's kind of the pointers is we need to think about this.
• 16:37
  SPEAKER_01: But there are other challenges, right?
• 16:39
  SPEAKER_00: Mm-hmm, yeah.
• 16:40
  SPEAKER_01: And hallucinations is a challenge.
• 16:43
  SPEAKER_01: So it invents stuff.
• 16:45
  SPEAKER_00: It invents, yeah.
• 16:48
  SPEAKER_00: Lots of that doesn't make sense.
• 16:49
  SPEAKER_00: And I think I will go back to your point.
• 16:52
  SPEAKER_00: When we do not have enough fundamental knowledge,
• 16:54
  it's really hard to do a kind of critical analysis of the AI's
• 16:59
  output to see which
• 17:01
  SPEAKER_00: part worked for us if the whole thing doesn't make sense,
• 17:04
  SPEAKER_00: yeah.
• 17:05
  SPEAKER_01: And in the paper we published a few months ago,
• 17:09
  SPEAKER_01: we actually looked at,
• 17:10
  SPEAKER_01: yeah,
• 17:11
  SPEAKER_01: you should go look that up.
• 17:12
  SPEAKER_01: It's out of the site conference.
• 17:14
  SPEAKER_01: And it's about qualitative coding.
• 17:16
  And one of the things we found with ChatGPT is when we asked
• 17:18
  for direct quotes from
• 17:20
  SPEAKER_01: the text,
• 17:22
  SPEAKER_01: it gave us a wonderful quote.
• 17:23
  SPEAKER_01: Yes.
• 17:24
  But then we looked further and said,
• 17:27
  wait a minute,
• 17:28
  I don't remember that.
• 17:29
  SPEAKER_01: And we went back to the text and it wasn't there.
• 17:32
  So it was paraphrasing to the point that it's like,
• 17:35
  this was actually a good indicator.
• 17:37
  SPEAKER_01: That is, it was a good descriptor.
• 17:40
  SPEAKER_01: It just wasn't a direct quote.
• 17:41
  SPEAKER_01: So presenting it as such is actually a hallucination, right?
• 17:46
  SPEAKER_01: It's not off topic, but it never actually happened.
• 17:49
  SPEAKER_01: And we can't trust it at that level.
• 17:51
  SPEAKER_01: So that's something everybody worries about.
• 17:53
  SPEAKER_00: Yeah,
• 17:53
  and I think we experienced the same phenomenon with other AI
• 17:56
  tools that we tried
• 17:57
  SPEAKER_00: with that research.
• 17:59
  SPEAKER_00: I mean, they change.
• 18:01
  SPEAKER_00: If they didn't invent the whole quotation,
• 18:04
  SPEAKER_00: they changed it and they modified it to be aligned with their findings.
• 18:08
  SPEAKER_01: Yeah, and obviously that's not what you want.
• 18:10
  SPEAKER_01: You want to have the evidence concrete and not questioned.
• 18:15
  SPEAKER_01: The second concern,
• 18:17
  SPEAKER_01: and that is definitely concern,
• 18:18
  was a concern with a group I was talking to because many of
• 18:22
  them are health
• 18:23
  SPEAKER_01: professionals,
• 18:23
  SPEAKER_01: and that is the issue of privacy.
• 18:25
  SPEAKER_01: Oh.
• 18:26
  SPEAKER_01: And so when you think about the Institutional Review Board,
• 18:29
  SPEAKER_01: what data you can share and how,
• 18:32
  SPEAKER_01: not knowing exactly how much it is protected.
• 18:35
  SPEAKER_01: Right now,
• 18:36
  SPEAKER_01: there's an order for OpenAI to keep all queries,
• 18:39
  SPEAKER_01: even the ones we ask them to not keep out.
• 18:43
  SPEAKER_01: And there's a court order, so obviously the information is out there.
• 18:47
  SPEAKER_01: Privacy is a major concern.
• 18:48
  SPEAKER_01: You can't just take research data and put it in.
• 18:50
  You've got to go through a few steps to anonymize it and to
• 18:54
  make sure that it
• 18:55
  doesn't include anything identifiable and anything that would
• 18:59
  risk any of the
• 19:00
  SPEAKER_01: promises you made to the participant.
• 19:03
  SPEAKER_01: If you're wondering about this and you're a researcher, consult your IRB.
• 19:06
  SPEAKER_01: There are processes in place to accommodate.
• 19:09
  SPEAKER_00: And that's one of the biggest concerns of researchers who are using AI.
• 19:12
  SPEAKER_00: And some of them,
• 19:13
  SPEAKER_00: I've talked with a few people,
• 19:16
  and I think it isn't worth going through all of those steps
• 19:19
  just to analyze your
• 19:20
  SPEAKER_00: data with AI.
• 19:22
  SPEAKER_00: Some of them think like this.
• 19:24
  SPEAKER_01: Yeah, absolutely.
• 19:25
  SPEAKER_01: And that's the expert conundrum, right?
• 19:27
  SPEAKER_01: So the question on the table is,
• 19:30
  SPEAKER_01: if this has worked and we have been doing research,
• 19:33
  SPEAKER_01: qualitative research,
• 19:35
  SPEAKER_01: before AI...
• 19:37
  SPEAKER_01: Why use AI?
• 19:43
  SPEAKER_01: She's making a face at me.
• 19:44
  SPEAKER_01: Why use AI?
• 19:46
  SPEAKER_01: What's the benefit?
• 19:47
  SPEAKER_01: What could be the benefit of using AI in qualitative research?
• 19:50
  SPEAKER_00: So my first response,
• 19:53
  the first thing that comes to my mind is that when you are
• 19:56
  dealing with a large
• 19:57
  SPEAKER_00: data set,
• 19:59
  pages and pages of transcripts,
• 20:02
  and it's really hard to go through all of that.
• 20:05
  SPEAKER_00: AI could be a good help, or as we talked about it, a good intern.
• 20:11
  SPEAKER_00: You can probably code the first few transcripts yourself,
• 20:15
  SPEAKER_00: reach intercoder agreement and everything,
• 20:17
  SPEAKER_00: and then train AI to do the rest of the work with you.
• 20:21
  SPEAKER_00: And then you can go back and verify the output that you get from AI.
• 20:26
  SPEAKER_01: Yeah, and...
• 20:28
  SPEAKER_01: Traditionally, qualitative research has dealt with...
• 20:32
  SPEAKER_01: Fairly small piles of data.
• 20:34
  SPEAKER_00: Sure, yeah.
• 20:35
  SPEAKER_01: And either short excerpts or, and potentially lots of them.
• 20:41
  SPEAKER_01: I once had a database that had about 1,400 separate data pieces,
• 20:47
  SPEAKER_01: but they were all two to four sentences long.
• 20:50
  SPEAKER_01: Yeah, yeah.
• 20:51
  SPEAKER_01: So it was a lot of reading, but not impossible.
• 20:54
  SPEAKER_01: Or you have in-depth interview, but with very few people.
• 20:58
  SPEAKER_01: You know, we usually say eight, eight, ten, twelve.
• 21:01
  SPEAKER_01: Mm-hmm.
• 21:02
  SPEAKER_01: Beyond that, it's hard to do reliable and consistent coding.
• 21:07
  SPEAKER_01: Now, you can reach saturation.
• 21:09
  There are other instruments that we've invented to make sure
• 21:14
  that we are not concerned.
• 21:16
  SPEAKER_01: But sometimes you do want a much richer database.
• 21:19
  SPEAKER_01: Sure.
• 21:21
  SPEAKER_01: becomes really almost impossible.
• 21:24
  It is incredibly expensive and almost impossible to analyze
• 21:28
  these large data sets.
• 21:30
  SPEAKER_01: And here is an instrument that,
• 21:32
  SPEAKER_01: if you do proper training,
• 21:33
  might be able to provide you a better understanding of a large
• 21:38
  set of documents.
• 21:39
  It gives you the bigger picture of the data that you are trying
• 21:42
  to understand and analyze,
• 21:44
  yeah.
• 21:44
  SPEAKER_00: Mm-hmm.
• 21:45
  SPEAKER_01: The second piece is that AI can do it as many times as you want.
• 21:50
  SPEAKER_00: Yeah, that's the best thing.
• 21:53
  So one of the affordances we don't think about,
• 21:56
  and we just reanalyze data.
• 21:57
  SPEAKER_01: So I did an analysis,
• 21:59
  SPEAKER_01: and as they gave me a look and said,
• 22:01
  SPEAKER_01: I'm not so sure about this thing,
• 22:03
  SPEAKER_01: and we reanalyzed it.
• 22:05
  SPEAKER_01: It was the right thing to do.
• 22:06
  SPEAKER_00: I still regret that question.
• 22:10
  SPEAKER_01: But...
• 22:13
  SPEAKER_01: We will not do that more than twice,
• 22:14
  SPEAKER_01: but you can ask AI to do that 20 times,
• 22:17
  SPEAKER_01: 30 times until it reaches convergence.
• 22:19
  SPEAKER_01: And we're like, okay, we know these are the things that are there.
• 22:22
  SPEAKER_01: So you can have multiple ways to do that.
• 22:25
  SPEAKER_01: So the repeated analysis is definitely a strength.
• 22:27
  SPEAKER_01: But then as I was thinking about this,
• 22:30
  I was thinking about the thing that is most difficult for us is
• 22:34
  guard against bias.
• 22:35
  SPEAKER_01: Now,
• 22:35
  I'm going to say something controversial because I'm going to
• 22:38
  say AI can help us
• 22:39
  SPEAKER_01: guard against bias.
• 22:41
  SPEAKER_00: Okay.
• 22:42
  SPEAKER_01: Even though AI is super biased itself.
• 22:45
  SPEAKER_01: Sure, yeah.
• 22:46
  SPEAKER_01: So we know AI is biased, but we are too.
• 22:50
  That's why we write a statement about positionality and all of
• 22:53
  these things to say,
• 22:54
  SPEAKER_01: here's where I'm coming from.
• 22:56
  But we can actually ask AI to guard against our biases,
• 23:01
  to be the other voice,
• 23:04
  right?
• 23:04
  SPEAKER_01: And say, here's my analysis from this perspective.
• 23:07
  SPEAKER_01: Can you look at it from another perspective?
• 23:09
  SPEAKER_00: Yeah, and this is my experience.
• 23:11
  SPEAKER_00: This is my background.
• 23:12
  SPEAKER_00: So, yeah.
• 23:13
  SPEAKER_01: So what would somebody who came from a different background?
• 23:17
  SPEAKER_00: Show me the hints that are representative of my bias in my data analysis.
• 23:22
  SPEAKER_00: Yeah, I love that.
• 23:23
  SPEAKER_01: And so it's a way to counter bias,
• 23:27
  which I haven't thought about until basically yesterday when we
• 23:30
  started thinking
• 23:30
  SPEAKER_01: about this.
• 23:31
  SPEAKER_01: And I'm like,
• 23:31
  SPEAKER_01: this is actually an interesting area because you can say,
• 23:34
  SPEAKER_01: you know,
• 23:35
  SPEAKER_01: I have training.
• 23:36
  SPEAKER_01: I come from a very specific way of thinking.
• 23:38
  SPEAKER_01: I come from information processing theory.
• 23:41
  SPEAKER_01: And I think in those terms, in very cognitive terms in many ways.
• 23:46
  SPEAKER_01: But...
• 23:47
  SPEAKER_01: What would somebody who's coding it with critical theory in mind, right?
• 23:53
  SPEAKER_01: Or with a postcolonial theory in mind or with feminist theory in mind,
• 23:59
  SPEAKER_01: what would the coding look like then?
• 24:01
  Is an interesting counter argument and to see the convergence
• 24:04
  and the divergence.
• 24:06
  SPEAKER_01: And then I am still, as long as I can...
• 24:10
  SPEAKER_01: do this,
• 24:10
  SPEAKER_01: and the AI isn't the PI,
• 24:13
  SPEAKER_01: I'm the PI,
• 24:14
  SPEAKER_01: I can make decisions about what do I do with this information,
• 24:17
  SPEAKER_01: but it can definitely show me where I'm missing things.
• 24:20
  SPEAKER_00: I love that idea, yeah.
• 24:21
  And we've seen this because we've been reading quite a bit of
• 24:24
  the research around
• 24:27
  SPEAKER_01: qualitative coding with AI,
• 24:29
  and consistently what we're seeing is that AI and people are
• 24:33
  not exactly on the
• 24:34
  SPEAKER_01: same page,
• 24:35
  SPEAKER_01: they're mostly on the same page,
• 24:36
  SPEAKER_01: and sometimes AI comes up with things that humans say,
• 24:40
  I haven't thought about this,
• 24:41
  but now that you told me about it,
• 24:43
  it's kind of obvious.
• 24:45
  SPEAKER_01: So it's definitely pointing to those cognitive biases that are existent.
• 24:49
  SPEAKER_01: Now, again, AI has a bias too.
• 24:52
  SPEAKER_01: So you can't accept what AI said is better than human,
• 24:55
  SPEAKER_01: but I think about it as complementary to human.
• 24:58
  And it's providing a different perspective and it can,
• 25:03
  through prompt engineering,
• 25:06
  SPEAKER_01: help us do this.
• 25:07
  SPEAKER_01: And so...
• 25:11
  when you think about when,
• 25:13
  when qualitative researchers ask me,
• 25:18
  why would I be using it?
• 25:19
  SPEAKER_01: These are the things I'm talking about.
• 25:21
  SPEAKER_00: Yeah.
• 25:21
  SPEAKER_00: So let me ask you a question.
• 25:23
  SPEAKER_00: If you are going to use AI to reveal your biases in your data analysis,
• 25:28
  SPEAKER_00: then how would you talk about that in your paper or in your manuscript?
• 25:32
  SPEAKER_01: I think we're an uncharted grounds.
• 25:34
  SPEAKER_01: I mean, that's where I find myself at least half of my life.
• 25:38
  SPEAKER_01: Um,
• 25:39
  SPEAKER_01: I think you have to write out what you've done and how you've done it.
• 25:43
  SPEAKER_01: I think at this point, I keep going back to it, we have established ways.
• 25:48
  SPEAKER_01: This is how you triangulate.
• 25:50
  SPEAKER_01: This is how you do this.
• 25:52
  SPEAKER_01: We have,
• 25:52
  SPEAKER_01: both in qualitative and quantitative research,
• 25:55
  SPEAKER_01: we have established ways for reporting certain things,
• 25:58
  SPEAKER_01: and we have code words,
• 26:00
  SPEAKER_01: right?
• 26:00
  SPEAKER_01: Triangulating is this.
• 26:02
  SPEAKER_01: And you don't need to do that much explaining.
• 26:04
  SPEAKER_01: You just need to say...
• 26:06
  SPEAKER_01: What were the results?
• 26:07
  SPEAKER_01: But this is a new area,
• 26:10
  so I think you have to write it out and even potentially
• 26:13
  include some of the
• 26:14
  SPEAKER_01: outputs as an appendix or added data for people,
• 26:19
  SPEAKER_01: especially the reviewers,
• 26:20
  SPEAKER_01: but the readers in general,
• 26:23
  SPEAKER_01: to start understanding what's going on.
• 26:25
  SPEAKER_01: And I think eventually we'll settle on accepted ways to do this.
• 26:28
  But right now there's nothing accepted,
• 26:30
  so the only thing you can do is write it out.
• 26:33
  SPEAKER_01: This is what I did.
• 26:34
  SPEAKER_01: This is what I did then.
• 26:35
  SPEAKER_01: This is what I did then.
• 26:37
  SPEAKER_01: This is the result.
• 26:38
  SPEAKER_01: This is what I think it means.
• 26:39
  SPEAKER_00: Be super clear about your procedures.
• 26:42
  SPEAKER_01: And that's what I worry about.
• 26:45
  As a reviewer,
• 26:46
  and I talk about this a lot,
• 26:47
  as a reviewer,
• 26:47
  I'm reading a lot.
• 26:49
  It's very clear that people are using more and more AI,
• 26:52
  both in writing and in analysis.
• 26:55
  SPEAKER_01: I'm not opposed to it.
• 26:56
  I'm just really wishing people would be more forthright and
• 27:01
  explain what they're
• 27:02
  SPEAKER_01: doing and how they're doing it and how they,
• 27:04
  SPEAKER_01: for example,
• 27:05
  SPEAKER_01: try to verify that this is good.
• 27:07
  SPEAKER_01: And this is where...
• 27:09
  SPEAKER_01: In our work right now,
• 27:10
  SPEAKER_01: we are doing a lot of work with AI,
• 27:12
  SPEAKER_01: and everything we're doing,
• 27:13
  SPEAKER_01: we're doing multiple times.
• 27:15
  SPEAKER_00: Sure.
• 27:16
  SPEAKER_01: Because any one time is anecdotal.
• 27:18
  SPEAKER_01: It's like working with people, right?
• 27:20
  SPEAKER_01: Because AI is not consistent.
• 27:23
  You've got to do it enough times to know that you're getting
• 27:25
  the consistent result
• 27:27
  SPEAKER_01: that you can trust versus I just tried it once and it came out.
• 27:32
  SPEAKER_00: Sure, yeah.
• 27:34
  And that's the thing that I'm worried about is people are not
• 27:36
  relying on this
• 27:37
  affordance that is that ability to iterate and have AI do it 30
• 27:42
  times or 10 times
• 27:44
  SPEAKER_01: or 11 times.
• 27:45
  There's that paper by Tai and colleagues that talked about when
• 27:51
  did their automatic
• 27:52
  SPEAKER_01: coding through AI reach convergence.
• 27:55
  SPEAKER_01: And we need more of that work,
• 27:57
  SPEAKER_01: but we also need everybody to understand that that's how you do this.
• 28:01
  SPEAKER_00: Yeah,
• 28:02
  and probably one of the factors that prevents people from
• 28:06
  iterating through AI is
• 28:07
  SPEAKER_00: that when you get all of those results,
• 28:10
  SPEAKER_00: you need to compare them to see how consistent they are.
• 28:13
  SPEAKER_00: So when you think about the time that you need to spend for that,
• 28:17
  SPEAKER_00: probably that's one of the things that you have.
• 28:21
  So let's wrap up quickly before we move on and say,
• 28:25
  can you use AI in qualitative analysis?
• 28:27
  SPEAKER_01: We think yes.
• 28:29
  SPEAKER_01: Know the literature.
• 28:30
  SPEAKER_01: Know what others have found about the consistency and about strategies.
• 28:35
  SPEAKER_01: So build on what others have done.
• 28:37
  And you should craft very detailed prompts for AI,
• 28:41
  give it enough context,
• 28:42
  explain the terms.
• 28:44
  SPEAKER_00: Even if you feel that it knows everything, it doesn't.
• 28:47
  SPEAKER_00: So,
• 28:47
  SPEAKER_00: for example,
• 28:48
  SPEAKER_00: talking about grounded theory,
• 28:49
  SPEAKER_00: you may want to define axial coding,
• 28:52
  SPEAKER_00: open coding,
• 28:53
  SPEAKER_00: selective coding,
• 28:54
  SPEAKER_00: and then do it step by step.
• 28:56
  SPEAKER_00: Ask AI not to do the whole work at once.
• 28:59
  SPEAKER_00: Just do it step by step.
• 29:01
  SPEAKER_00: Do the open coding.
• 29:02
  SPEAKER_00: Give me the result.
• 29:03
  SPEAKER_00: Check the result.
• 29:04
  SPEAKER_00: And then ask it to go over other steps.
• 29:08
  SPEAKER_01: And the last thing is, remember, you are the human overlord.
• 29:12
  SPEAKER_00: Yeah.
• 29:13
  SPEAKER_01: For now.
• 29:16
  SPEAKER_01: So act like it.
• 29:18
  SPEAKER_01: We are, as the researchers, ultimately responsible.
• 29:21
  SPEAKER_01: We cannot surrender responsibility to the machine.
• 29:23
  SPEAKER_00: You have the agency, not AI.
• 29:26
  SPEAKER_01: Yes.
• 29:27
  SPEAKER_01: Again, at this point.
• 29:28
  SPEAKER_01: This might change by next week in ChatGPT 5.0, but...
• 29:33
  SPEAKER_01: Until then, feel assured.
• 29:35
  SPEAKER_01: All right.
• 29:36
  SPEAKER_01: Last segment for this show is how are you using AI right now?
• 29:43
  Right now I'm using AI for one of the qualitative papers that
• 29:47
  I'm working on.
• 29:49
  SPEAKER_00: So we previously coded, Guy and I previously coded a few transcripts and
• 29:56
  We had this idea that after anonymizing and after doing all of
• 30:00
  those things,
• 30:01
  SPEAKER_00: important things that Guy just mentioned,
• 30:03
  SPEAKER_00: we want to see how ChatGPT does the qualitative coding.
• 30:08
  SPEAKER_00: And we took three different approaches.
• 30:11
  SPEAKER_00: The first approach,
• 30:12
  I just asked AI to analyze the thing without giving it any kind
• 30:15
  of details about
• 30:18
  SPEAKER_00: qualitative work or different approaches that you take.
• 30:22
  And it surprised me with choosing the right theory and also
• 30:27
  talking about all of
• 30:28
  SPEAKER_00: the steps that we need to do when we take that approach to code the data.
• 30:33
  SPEAKER_00: But the result was not as I thought, and it was kind of disappointing.
• 30:39
  SPEAKER_00: You remember, Guy, I showed you the results for that.
• 30:42
  So it didn't do the line-by-line coding as we usually do in
• 30:46
  qualitative analysis.
• 30:48
  SPEAKER_00: And it just gave me some samples of the data with codes.
• 30:52
  SPEAKER_00: But in terms of extracting the themes,
• 30:56
  SPEAKER_00: we felt that maybe it did a fairly,
• 30:59
  SPEAKER_00: well,
• 31:00
  SPEAKER_00: kind of a good job,
• 31:02
  SPEAKER_00: I think.
• 31:03
  SPEAKER_00: But yeah, what was your idea about the first approach and the AI output?
• 31:09
  SPEAKER_01: I think that one of the things we fail when we over-engineer the prompts,
• 31:13
  SPEAKER_01: we fail to uncover the advances that
• 31:21
  that AI is making,
• 31:22
  generative AI is making in understanding those whole worlds.
• 31:28
  So I think it's good to give specific prompts and walk the
• 31:32
  process very carefully.
• 31:34
  SPEAKER_01: But two things stick in my mind.
• 31:35
  One is they are getting better and they know things like they
• 31:39
  basically spelled out
• 31:40
  SPEAKER_01: how to do grounded theory in that feedback.
• 31:44
  SPEAKER_01: So obviously something is happening that isn't required.
• 31:46
  SPEAKER_01: And maybe if you don't want grounded theory, you should tell it.
• 31:49
  SPEAKER_01: Yeah.
• 31:50
  SPEAKER_01: But it's defaulting into that.
• 31:52
  The second thing that emerged in my mind is the paper that
• 31:57
  Ethan Malik was really
• 31:59
  SPEAKER_01: pointing to about maybe the outcome is more important than the steps.
• 32:05
  SPEAKER_01: And so I'm still somewhat baffled.
• 32:09
  So a lot of the work we've done and others have done is,
• 32:13
  is AI like human,
• 32:15
  right?
• 32:16
  SPEAKER_01: Are we going to get the same results?
• 32:18
  It might be that we're getting a little bit different results,
• 32:21
  but it's just as good.
• 32:22
  SPEAKER_01: So complementary or just different.
• 32:26
  SPEAKER_01: So that's an interesting way to think about it.
• 32:28
  And that's why when I try things,
• 32:31
  I try them in different levels of depth to see if it matters.
• 32:35
  SPEAKER_01: And am I still getting what I want to get?
• 32:37
  SPEAKER_01: Mm-hmm.
• 32:38
  SPEAKER_01: And is accuracy,
• 32:40
  SPEAKER_01: I'm going to say something really controversial,
• 32:41
  SPEAKER_01: but it's going to be,
• 32:42
  SPEAKER_01: is accuracy really that important if the basic coding is the same?
• 32:47
  SPEAKER_01: So that's a question in my mind.
• 32:49
  We'll talk about this another time because we're running out of
• 32:51
  time and we want to
• 32:53
  SPEAKER_01: make sure that people are not just switching off immediately.
• 32:56
  SPEAKER_01: So today we talked about using AI.
• 33:00
  SPEAKER_01: specifically generative AI, but we use AI to meet them.
• 33:04
  SPEAKER_01: We talked about Notebook LM.
• 33:06
  SPEAKER_01: We talked a lot about what are we doing with qualitative research,
• 33:09
  SPEAKER_01: what are our thoughts,
• 33:11
  SPEAKER_01: and next time we'll just pick it up there and talk about some new things.
• 33:15
  SPEAKER_00: And we love your feedback.
• 33:18
  SPEAKER_01: Yes,
• 33:18
  we would love as many emails as you can send us because we
• 33:23
  don't get enough email
• 33:24
  SPEAKER_01: is really where it's at.
• 33:26
  SPEAKER_01: All right, we'll see you next time.
• 33:28
  SPEAKER_00: See you.