Latest / Azadeh and Guy on AI / Can AI Be Your Next Interviewer?
Transcript
- 0:00SPEAKER_00: And we are on episode eight.
- 0:05SPEAKER_01: Hello, everyone.
- 0:06SPEAKER_01: We are recording the eighth episode of Azadeh and Guy on AI.
- 0:14SPEAKER_00: And we are doing this.
- 0:16SPEAKER_00: This time it's an experiment through Zoom.
- 0:18SPEAKER_00: So if you have quality problems, please complain to Zoom.
- 0:24SPEAKER_01: Yes.
- 0:24SPEAKER_00: But if it's great, let us know.
- 0:27SPEAKER_01: Sure.
- 0:28SPEAKER_01: Yeah.
- 0:29SPEAKER_01: So today we are talking about using AI to conduct interviews.
- 0:35SPEAKER_00: So this is something I've been fascinated with.
- 0:40Ever since I found out that one of the companies whose founder
- 0:44I listened to on one
- 0:46of the podcasts talked about the fact that they conduct a lot
- 0:49of their interviews
- 0:50SPEAKER_00: with the help of
- 0:53SPEAKER_00: And I was immediately really,
- 0:55really interested in the potential of collecting a lot more
- 0:59data and actually
- 1:01SPEAKER_00: creating.
- 1:02SPEAKER_00: So that's one thing.
- 1:03SPEAKER_00: The second thing that I thought about is that interviewing with AI can be
- 1:08SPEAKER_00: potentially uncomfortable,
- 1:09SPEAKER_00: right?
- 1:11SPEAKER_01: Yes, yeah.
- 1:13SPEAKER_00: AI and it's artificial and all of that.
- 1:16But if you're upfront about it and you give people an option
- 1:19and if they say,
- 1:20SPEAKER_00: no,
- 1:20SPEAKER_00: I want to talk to a person,
- 1:21SPEAKER_00: you give them that option.
- 1:22I think that there's an advantage where AI doesn't judge you,
- 1:26doesn't make faces.
- 1:27SPEAKER_00: It's not a potentially as biased.
- 1:30SPEAKER_00: So there's some advantages to the AI as an interview agent.
- 1:36SPEAKER_00: So we decided to do an episode on this.
- 1:39SPEAKER_01: Yeah, yeah, yeah.
- 1:41SPEAKER_01: And I don't think AI is still that much advanced in terms of,
- 1:47SPEAKER_01: I mean,
- 1:48SPEAKER_01: I'm talking about doing interviews with AI,
- 1:50SPEAKER_01: obviously.
- 1:51So I don't think there's still any application that reads human
- 1:59facial expressions
- 2:00SPEAKER_01: while doing the interviews.
- 2:02SPEAKER_01: Yeah.
- 2:03Yeah,
- 2:03I don't think we're quite there,
- 2:04but I think we will be there fairly soon.
- 2:08SPEAKER_00: But on the other hand,
- 2:10SPEAKER_00: not over-reading facial expression might be an advantage and not just a
- 2:15SPEAKER_00: disadvantage.
- 2:17Because sometimes we over-read or we over-interpret facial
- 2:26expression that may not
- 2:27SPEAKER_00: at all be connected to what we're talking about.
- 2:29SPEAKER_00: So there's an advantage there.
- 2:32SPEAKER_00: to try to do it a different way.
- 2:35SPEAKER_00: And all I'm saying is when we talked about doing analysis,
- 2:39SPEAKER_00: for example,
- 2:39SPEAKER_00: qualitative analysis with AI,
- 2:41SPEAKER_00: I'm not sure we're there yet either,
- 2:43but I think it's really important to play with it because it's
- 2:46going to be a
- 2:47SPEAKER_00: powerful research tool when it gets there.
- 2:50SPEAKER_00: And I do think you'll get there.
- 2:52SPEAKER_01: Yep, yep, yep, yep.
- 2:54SPEAKER_00: So the app or website that I'm talking about is actually 11 Labs.
- 3:00SPEAKER_00: And 11 Labs is an app that provided us with voice agents very early on.
- 3:06SPEAKER_00: And you can play with them.
- 3:07SPEAKER_00: You can learn English with them.
- 3:09SPEAKER_00: And that's lovely.
- 3:11But more recently,
- 3:13they actually created the ability for the user to create an
- 3:17agent.
- 3:18SPEAKER_00: What do I love about it?
- 3:20SPEAKER_00: First of all,
- 3:20SPEAKER_00: you tailor the agent so you can actually provide them the script,
- 3:26SPEAKER_00: the questions,
- 3:27SPEAKER_00: what prompt to use,
- 3:28SPEAKER_00: what kind of voice to use.
- 3:31SPEAKER_00: And you do also input a personality.
- 3:34SPEAKER_00: So just like you can do with any AI,
- 3:37SPEAKER_00: you can say,
- 3:38SPEAKER_00: so their initial one,
- 3:40SPEAKER_00: for example,
- 3:40SPEAKER_00: says,
- 3:41SPEAKER_00: you are Alexis.
- 3:42SPEAKER_00: That's the direction to the AI, not to us.
- 3:44A friendly,
- 3:45proactive,
- 3:45and highly intelligent female in a world-class engineering
- 3:48background.
- 3:49SPEAKER_00: Your approach is warm,
- 3:51SPEAKER_00: witty,
- 3:51SPEAKER_00: and relaxed,
- 3:52SPEAKER_00: effortless,
- 3:53SPEAKER_00: balancing professionalism with the chill,
- 3:56SPEAKER_00: approachable vibe.
- 3:58SPEAKER_00: So they give you that,
- 4:00SPEAKER_00: not just so you can use that specific voice,
- 4:04SPEAKER_00: but because it shows you how to prompt the AI in a robust way.
- 4:09SPEAKER_00: So there's actually much more of that.
- 4:14SPEAKER_00: in the prompt.
- 4:15SPEAKER_00: The prompt is quite long.
- 4:16It's a two,
- 4:17three page long prompt with guardrails and with,
- 4:21so there's a lot of detail.
- 4:23So you have,
- 4:24you can develop it very carefully,
- 4:25but you can think about this.
- 4:26SPEAKER_00: When we do interviews,
- 4:28we try to do a practice run and to provide a lot of guidelines
- 4:34to the people who
- 4:35are doing this because they need to know what they're doing and
- 4:37they need to know
- 4:38SPEAKER_00: how to do it.
- 4:39SPEAKER_00: This is the same thing, but you write it out in a very clear way.
- 4:43SPEAKER_00: And then you can choose a language,
- 4:47SPEAKER_00: which I love,
- 4:48because one of the things we do is we contact families and kids
- 4:53with different
- 4:53SPEAKER_00: first languages.
- 4:54SPEAKER_00: And here's our opportunity to do interviews in a variety of languages.
- 5:00SPEAKER_00: There are about 50 languages or so.
- 5:02SPEAKER_00: I'm not sure about the count, but there are a lot of languages.
- 5:05SPEAKER_00: So it gives you the ability to download the need
- 5:12SPEAKER_00: To have people speaking a lot of languages or to decide that you're not
- 5:17SPEAKER_00: interviewing anybody that speaks a language,
- 5:20SPEAKER_00: you don't have the capacity on the project.
- 5:22SPEAKER_00: Suddenly you can do that.
- 5:24SPEAKER_00: And I think that's magical.
- 5:26SPEAKER_01: Yes, it is.
- 5:29SPEAKER_00: And then you have a choice of voice.
- 5:34SPEAKER_00: You can have a choice of testing it.
- 5:37SPEAKER_00: So you can test it.
- 5:38SPEAKER_00: You can provide security on top of it.
- 5:40And you can create a widget so it can be launched within your
- 5:43website and you don't
- 5:44SPEAKER_00: have to bring everybody into 11 labs.
- 5:50SPEAKER_00: So this, to me, has tremendous potential now.
- 5:54SPEAKER_01: Go ahead.
- 5:59SPEAKER_01: Yeah, I have a question.
- 6:00So does it,
- 6:01the agent that you just described,
- 6:03does it act like a chat bot?
- 6:09SPEAKER_00: It is a voice agent.
- 6:11SPEAKER_00: So it acts like a chatbot, but it's voice.
- 6:15SPEAKER_00: So it listens to you, it responds to you, and it asks the question.
- 6:20There's also the option,
- 6:20of course,
- 6:21to record everything,
- 6:22which is what we want in an interview.
- 6:23SPEAKER_00: We want to record.
- 6:24SPEAKER_00: And there's the capacity to tell it exactly what the protocol is.
- 6:31SPEAKER_01: Okay, okay.
- 6:32So there's a variety of ways where you can interact with it and
- 6:39add all the
- 6:40SPEAKER_00: components that you want in an interview,
- 6:42SPEAKER_00: including,
- 6:43SPEAKER_00: again,
- 6:44SPEAKER_00: the tone,
- 6:45SPEAKER_00: including the exact language.
- 6:48SPEAKER_00: The questions,
- 6:49SPEAKER_00: including how to respond to follow-ups and all of this,
- 6:53SPEAKER_00: requires quite a bit of tinkering,
- 6:56SPEAKER_00: but you can tailor it,
- 6:58SPEAKER_00: you can test it,
- 6:59SPEAKER_00: and my favorite thing,
- 7:00SPEAKER_00: again,
- 7:01SPEAKER_00: is you can also put guardrails on it.
- 7:03SPEAKER_00: I think you should also let your participants know and...
- 7:08give them the option to opt out of working with an agent and
- 7:12work with a human
- 7:13because you want to be sensitive and you don't want people to
- 7:16not participate
- 7:17SPEAKER_00: because it's an AI.
- 7:20SPEAKER_01: Exactly.
- 7:21SPEAKER_01: But a huge part of that process is very similar,
- 7:25SPEAKER_01: as you mentioned,
- 7:26SPEAKER_01: to what we are doing in other interviewers.
- 7:30SPEAKER_01: I mean, we give them guidelines.
- 7:32SPEAKER_01: So the same thing is true about AI.
- 7:34But we need to be just more specific to make sure that it's
- 7:37asking the way that we intend to.
- 7:41SPEAKER_00: Yes.
- 7:42SPEAKER_00: And one of the things, for example, is a measure they call temperature.
- 7:46And the measure of temperature is how much leniency do you give
- 7:52the AI model,
- 7:53the LLM.
- 7:55SPEAKER_00: So you can say deterministic,
- 7:56SPEAKER_00: ask only the questions we have specified,
- 7:59SPEAKER_00: and you can get it to be creative or more creative.
- 8:02SPEAKER_00: And the way,
- 8:03SPEAKER_00: I think,
- 8:03to play something like that is to test it multiple times and
- 8:09see what happens when
- 8:10SPEAKER_00: you change it.
- 8:12SPEAKER_00: And then you can say, do I want it to go that way or not?
- 8:16I like,
- 8:17at least at the beginning,
- 8:18to think about it as more deterministic.
- 8:20I want to make sure it asks all the questions and in exactly
- 8:23the right way.
- 8:24SPEAKER_00: It means it won't have really smart follow-ups most of the time,
- 8:28SPEAKER_00: but it means I will get the content I really want right there.
- 8:33SPEAKER_00: So that's the balance of things that I'm thinking about.
- 8:36SPEAKER_01: Yes, yes.
- 8:37SPEAKER_01: I think we need to be on the safe side.
- 8:39SPEAKER_01: First,
- 8:39when you are trying the bot to make sure that it's doing
- 8:42exactly what you have to
- 8:44SPEAKER_01: do,
- 8:44SPEAKER_01: what you need to do.
- 8:45SPEAKER_01: Yeah.
- 8:47SPEAKER_00: Yeah.
- 8:48SPEAKER_00: And so that's part of the tools that are emerging.
- 8:53SPEAKER_00: And I think if you want to do a large scale interview study and research,
- 8:59you want to try and get all of this much quicker,
- 9:04this is the opportunity.
- 9:06If you're doing a study like we just did with four
- 9:09participants,
- 9:10I don't think we need that.
- 9:11SPEAKER_01: No, I don't think so, yeah.
- 9:13SPEAKER_00: I would still say stay on the safe side, be with humans.
- 9:17SPEAKER_00: But if you suddenly say, I want to do interview with 120 people,
- 9:21SPEAKER_00: That is hard, even from a scheduling perspective.
- 9:24So the ability to do that quickly with an agent allows you to
- 9:31spend your time otherwise.
- 9:32SPEAKER_00: Now, it's still going to cost you because 11 Labs is not free.
- 9:37SPEAKER_00: And with each conversation and all of that,
- 9:40you will eventually have to pay for either the API or for
- 9:44tokens or whatever the
- 9:46SPEAKER_00: mechanism says.
- 9:48SPEAKER_00: So you want to find out what it is,
- 9:50SPEAKER_00: but it does allow you,
- 9:51SPEAKER_00: if you have the funding,
- 9:52SPEAKER_00: it allows you to operate a lot faster.
- 9:54SPEAKER_00: And I think that's worth at least considering.
- 9:58SPEAKER_01: Yeah, yes, that's very true.
- 10:00SPEAKER_01: Yeah.
- 10:01SPEAKER_00: All right.
- 10:02SPEAKER_00: And we have two papers that connect to that.
- 10:04SPEAKER_00: So do you want to talk about the first one?
- 10:06SPEAKER_01: Sure.
- 10:07SPEAKER_01: Yeah.
- 10:08SPEAKER_01: So one of the papers is a kind of literature review.
- 10:11The title of the paper is AI Supported Individual Interviews
- 10:15Opportunities and
- 10:16SPEAKER_01: Threats for Evaluation.
- 10:18SPEAKER_01: It's been published in 2024,
- 10:22SPEAKER_01: and the last name of the author is Kupiek,
- 10:27SPEAKER_01: if I'm pronouncing it correct.
- 10:31SPEAKER_01: So they take a kind of evaluative view on...
- 10:36SPEAKER_01: not let's try AI interviewing now,
- 10:38but what does the body of literature suggest about integrating
- 10:42AI into every step
- 10:44SPEAKER_01: of qualitative interviewing in evaluation settings?
- 10:48SPEAKER_01: So what they do is that they break down the
- 10:52SPEAKER_01: all of the stages of a qualitative interview process into six categories,
- 10:58SPEAKER_01: which are planning,
- 10:59SPEAKER_01: developing the interview protocol,
- 11:01SPEAKER_01: sampling and recruitment,
- 11:03SPEAKER_01: data collection,
- 11:04SPEAKER_01: transcribing,
- 11:04SPEAKER_01: and finally coding and analysis.
- 11:07SPEAKER_01: And they try to see how much AI has been used in any of those stages.
- 11:14SPEAKER_00: Yes.
- 11:15SPEAKER_00: And how much research there is.
- 11:17SPEAKER_00: So as a literature review.
- 11:19SPEAKER_01: Yeah, exactly.
- 11:20SPEAKER_01: Yes, yes, yes.
- 11:21SPEAKER_01: How much research has been done in any of those areas.
- 11:26SPEAKER_01: So for the first one,
- 11:27SPEAKER_01: planning and instrument design,
- 11:30SPEAKER_01: we all know that AI is very good at generating ideas.
- 11:35SPEAKER_01: We can use it for drafting interview questions.
- 11:38SPEAKER_01: We can...
- 11:39SPEAKER_01: use AI to test and refine protocols.
- 11:42SPEAKER_01: And I know that they had a table.
- 11:44SPEAKER_01: I'm just going to pull up that table.
- 11:47SPEAKER_01: Yeah.
- 11:47SPEAKER_01: And the number of records for planning is five.
- 11:51SPEAKER_01: So people are using it in planning their interview.
- 11:56SPEAKER_01: The next stage is sampling and recruitment.
- 11:59SPEAKER_01: And this is a kind of weak spot.
- 12:02I'm also curious about knowing how AI is going to be useful to
- 12:06recruit interviewees.
- 12:09But the other idea that they have is that we can use AI to do a
- 12:13simulation.
- 12:16SPEAKER_01: And this part I also find useful.
- 12:19SPEAKER_01: I thought about the same thing for my own research,
- 12:23kind of doing simulating respondents for a kind of pilot phase
- 12:28of the study.
- 12:29SPEAKER_01: could be helpful.
- 12:32SPEAKER_01: And the number of records that they have is two for that one.
- 12:35SPEAKER_01: Sampling and recruitment.
- 12:37SPEAKER_01: We talked about that.
- 12:38SPEAKER_01: Data collection.
- 12:39SPEAKER_01: I mean, AI conducting the interviews.
- 12:45SPEAKER_01: For that one,
- 12:46SPEAKER_01: they have seven instances of AI using for doing interviews,
- 12:51SPEAKER_01: and there are emerging studies.
- 12:56SPEAKER_01: For example, researchers have used chatbots to collect data.
- 13:02SPEAKER_01: But there are still limitations around emotional sensitivity,
- 13:06SPEAKER_01: dynamic probes,
- 13:07SPEAKER_01: and also maintaining rapport with interviewees.
- 13:13SPEAKER_00: And we talked about the difficulties when we talked about 11 labs,
- 13:17SPEAKER_00: how fast and how well we can do that is important.
- 13:22SPEAKER_01: Exactly.
- 13:23Transcription,
- 13:24and I think you agree that we can use AI fairly well for doing
- 13:31the transcription.
- 13:34SPEAKER_01: The number of studies that have used AI for that purpose is 14.
- 13:40SPEAKER_00: And I think that that's just the studies that reported on it,
- 13:43SPEAKER_00: because everybody that's doing interviews right now is either using the
- 13:47SPEAKER_00: transcribing in Zoom or using one of the other platforms.
- 13:50And the human,
- 13:51instead of being the main transcriber,
- 13:54the human becomes just the checker.
- 13:59SPEAKER_00: Where, yeah, where there are two challenges.
- 14:04One is when there is a heavy accent or an unfamiliar to the
- 14:07machine accent that
- 14:09becomes a little more difficult and that requires more human
- 14:13intervention.
- 14:14It's not that it doesn't work,
- 14:15but it will require more human intervention.
- 14:18And the second one is in languages that are more rare,
- 14:23the accuracy goes down.
- 14:25SPEAKER_01: Yeah.
- 14:26SPEAKER_01: And the last category is coding and analysis.
- 14:28SPEAKER_01: So the number of instances that they mentioned in their study is 25.
- 14:34SPEAKER_01: And they say AI tools have been applied to generate initial codes,
- 14:38SPEAKER_01: group themes,
- 14:39SPEAKER_01: assist in inductive or deductive coding.
- 14:41SPEAKER_01: And I think
- 14:43SPEAKER_01: People are using it more and more,
- 14:44especially that AI has been embedded in many of those software
- 14:49that we use for
- 14:50SPEAKER_01: coding.
- 14:51SPEAKER_01: For example, MaxQDA and Vivo or other software.
- 14:54SPEAKER_01: Atlas CI, all of them have AI enabled in their software.
- 15:01SPEAKER_00: And so we've talked about those and we have a specific interest.
- 15:06But what I love about this review is really it's looking at the
- 15:10lifespan of the
- 15:11interview from design all the way to coding and basically
- 15:16showing how AI can be
- 15:18SPEAKER_00: used in almost all of them,
- 15:19SPEAKER_00: which is really fascinating to me.
- 15:22SPEAKER_01: Exactly, exactly, yes.
- 15:24And the thing that they are suggesting is that still we need
- 15:27human in the process.
- 15:29SPEAKER_01: You cannot say that AI can be used as a complement, not a replacement.
- 15:35So it can boost efficiency,
- 15:37it can expand the scale,
- 15:40but not supplant human expertise.
- 15:43SPEAKER_01: And there are still ethical issues.
- 15:45We have still transparency issues,
- 15:48bias and contextual sensitivity issues.
- 15:51SPEAKER_01: And all of them, they still remain as key challenges.
- 15:54SPEAKER_00: Yeah.
- 15:57SPEAKER_00: All right.
- 15:58SPEAKER_00: The second paper is a working paper from CES IFO.
- 16:02SPEAKER_00: It's CES IFO Conducting Qualitative Interviews with AI.
- 16:06SPEAKER_00: And it's by Chopra and Halland.
- 16:09And I really liked this one because they've used it in the
- 16:14research and measured a
- 16:15SPEAKER_00: variety of measures that are connected to how acceptable
- 16:22SPEAKER_00: this interview process is.
- 16:26SPEAKER_00: And what I'm seeing in some of the results,
- 16:30SPEAKER_00: and we're not going to go all over it because it's an 87-page document.
- 16:34SPEAKER_00: If you're interested, go look it up.
- 16:37SPEAKER_00: But...
- 16:40SPEAKER_00: When they talked about,
- 16:41SPEAKER_00: for example,
- 16:42SPEAKER_00: the estimation of the people participating,
- 16:44SPEAKER_00: about the naturalness of interviews,
- 16:47SPEAKER_00: if you take the very natural to extremely natural,
- 16:52SPEAKER_00: you're already at 41% of the participants.
- 16:55SPEAKER_00: And if you're taking somewhat natural, then you were at 75%.
- 16:59SPEAKER_00: What it says to me is...
- 17:02SPEAKER_00: It's natural enough for people not to feel off-put by it,
- 17:06SPEAKER_00: but there is 25% of people who are feeling not so okay about it.
- 17:11SPEAKER_00: So 75% were okay with it, 25% were like, this felt really awkward.
- 17:17SPEAKER_00: And the same thing about the overall experience,
- 17:23SPEAKER_00: the very positive and extremely positive were 60% of participants.
- 17:30That's a lot when you consider the fact that agents are new and
- 17:37some people feel
- 17:38SPEAKER_00: really strongly about AI.
- 17:39SPEAKER_00: And the same percentage almost,
- 17:44SPEAKER_00: so 55% or 53%,
- 17:47SPEAKER_00: sorry,
- 17:48SPEAKER_00: said they actually would have preferred AI to a human.
- 17:54SPEAKER_00: So more than half prefer the AI to a human.
- 18:00The other interesting thing about their research is that they
- 18:03developed the chatbot
- 18:06SPEAKER_01: and they published their paper in 2023.
- 18:10SPEAKER_01: So it was kind of... That was an old AI.
- 18:15SPEAKER_01: Yes, yes.
- 18:17SPEAKER_01: But they developed a multi-agent system and that's interesting.
- 18:21They have a security agent,
- 18:23they have a summary agent,
- 18:24topic agent,
- 18:25probing agent,
- 18:28and...
- 18:29SPEAKER_01: Yeah,
- 18:29so the process was interesting how they developed that kind of
- 18:33system to use AI for
- 18:35SPEAKER_01: doing the interviews.
- 18:38SPEAKER_00: So what you're really saying is
- 18:40SPEAKER_00: There's potential here.
- 18:42We're definitely going to start playing with it and seeing how
- 18:45people respond to it.
- 18:47SPEAKER_00: But we are encouraging you to just take a look.
- 18:50SPEAKER_00: If you're doing that kind of study or if it's just intriguing to you,
- 18:53SPEAKER_00: check out those two papers.
- 18:55Check out 11 Labs and see how you can use it and drop us a word
- 19:01and tell us how it went.
- 19:03SPEAKER_01: Sure.
- 19:03SPEAKER_01: Yeah.
- 19:04SPEAKER_01: It will be an interesting process.
- 19:09SPEAKER_00: All right.
- 19:10SPEAKER_00: So I think that's it for this time.
- 19:12SPEAKER_00: Eight episodes.
- 19:13SPEAKER_00: I feel like we're hitting our groove now.
- 19:17SPEAKER_01: Yes, definitely.
- 19:18SPEAKER_00: All right.
- 19:22SPEAKER_00: All right.
- 19:23SPEAKER_00: So we'll see you next time.
- 19:26SPEAKER_00: On Eaza Day.
- 19:28SPEAKER_01: And Guy.
- 19:28SPEAKER_00: On AAI.
- 19:30SPEAKER_01: All right.
- 19:34SPEAKER_00: We'll see you next time.