Latest / Inside DuckDuckGo / Duck Tales: Why we built a ‘second opinion’ option in Duck.ai and how you can use it to increase confidence in responses. (Ep.48)
Transcript
- 0:00SPEAKER_00: Hello, and welcome to DuckTales,
- 0:02SPEAKER_00: where we go behind the scenes at DuckDuckGo and talk about the people, stories,
- 0:05SPEAKER_00: and technology that are building privacy tools for everyone.
- 0:08SPEAKER_00: This week, I'm talking to DuckDuckGo Chief Product Officer Bia about a feature
- 0:12SPEAKER_00: in DuckAI called Second Opinion that lets you compare answers
- 0:16SPEAKER_00: from different AI models side by side.
- 0:18SPEAKER_01: Bia, thank you so much for coming on DuckTales to tell me about Second Opinion.
- 0:23SPEAKER_00: My pleasure. Thanks for having me.
- 0:25SPEAKER_00: Will you explain what this feature is?
- 0:28SPEAKER_00: Sure.
- 0:29SPEAKER_01: And in fact, let me show you, which I think will be even better. So we're at DuckAI.
- 0:34SPEAKER_01: I'm going to ask a question that's at the forefront of everybody's mind,
- 0:38SPEAKER_01: which is best trick for removing
- 0:44SPEAKER_01: a butternut squash soup.
- 0:49SPEAKER_01: Stain from white t-shirt.
- 0:53SPEAKER_01: And I'm getting a response here.
- 0:54SPEAKER_01: The model that provided was GPT-6 Luna, and it says to blot off excess,
- 0:59SPEAKER_01: flush the stain from the back of cold water, liquid laundry detergent.
- 1:03SPEAKER_00: So we've got about two short paragraphs of instructions from GPT-6 Luna.
- 1:08SPEAKER_00: And then at the bottom of the answer, we see a couple buttons, like a share button,
- 1:13SPEAKER_00: a copy button, and a button that says second opinion. Yes.
- 1:17SPEAKER_01: So I'm like, all right, this answer seems OK,
- 1:21SPEAKER_01: but I don't want to get this wrong and forever have a yellow stain
- 1:24SPEAKER_01: on my white T-shirt.
- 1:25SPEAKER_01: So I'm going to click the second opinion.
- 1:28SPEAKER_01: And see what another model's advice is.
- 1:30SPEAKER_01: So in this case, DuckAI is using GEMMA4, which is an open weight model,
- 1:36SPEAKER_01: asking the same question and we get a slightly different answer here.
- 1:40SPEAKER_01: So we got to treat the grease first and the color second.
- 1:42SPEAKER_01: So looks like there's a lot of overlap with the first answer,
- 1:46SPEAKER_01: but some additional detail and some additional steps that give me
- 1:50SPEAKER_01: a little bit more confidence here.
- 1:53SPEAKER_01: And so, yeah, the idea, like roughly with the second opinion is
- 1:56SPEAKER_01: that sometimes you ask AI a question, you get an answer and you're like, maybe.
- 2:01SPEAKER_01: OK, I don't know.
- 2:02SPEAKER_01: Like I would ask another friend if I was sitting at a table looking for advice.
- 2:07SPEAKER_01: And so you get to roll the dice again and ask another model and get another question.
- 2:12SPEAKER_01: And then you can kind of figure out the diff.
- 2:15SPEAKER_00: Interesting. OK, so you just used the image of like rolling a dice.
- 2:20SPEAKER_00: Why that?
- 2:22SPEAKER_01: Yeah, because answers from AI are non-deterministic,
- 2:26SPEAKER_01: meaning they are generated
- 2:30SPEAKER_01: on the spot using a prediction, basically,
- 2:34SPEAKER_01: of what you want to hear based on some huge data training set.
- 2:38SPEAKER_01: And they can be wrong and they can be like totally wrong.
- 2:41SPEAKER_01: They can hallucinate and be crazy.
- 2:45SPEAKER_01: That, I think, is one of the reasons that people are rightly cautious about using AI
- 2:51SPEAKER_01: to answer their questions because they are not guaranteed right answers.
- 2:57SPEAKER_01: And so running the same question on two different models gives you kind of a sense
- 3:01SPEAKER_01: of the universe of possibilities and a little bit more data to make
- 3:05SPEAKER_01: an informed decision.
- 3:08SPEAKER_00: Yeah, and it seems like DuckDuckGo is uniquely able to build a feature like this
- 3:13SPEAKER_00: because an AI company would never encourage you and has no reason to show you
- 3:18SPEAKER_00: an answer from a different company's models.
- 3:21SPEAKER_00: Whereas, you know, we have all these models in the same place with a privacy layer
- 3:25SPEAKER_00: that is now letting us do this kind of fun and useful thing
- 3:30SPEAKER_00: of seeing two answers side by side.
- 3:32SPEAKER_00: How did this feature come to be?
- 3:33SPEAKER_00: Like what kind of problem is it solving for people as they search?
- 3:38SPEAKER_01: There's a few answers to that.
- 3:40SPEAKER_01: Anecdotally, when I'm using AI, which I use DuckAI every day many times over,
- 3:45SPEAKER_01: I often find myself kind of wanting to like second guess or fact check
- 3:51SPEAKER_01: or get another opinion on the answers that I'm getting.
- 3:54SPEAKER_01: Sometimes you're like asking for some like, you know, What's Tom Hanks' birthday?
- 3:59SPEAKER_01: There's one answer to that that's correct.
- 4:01SPEAKER_01: All the other answers are wrong.
- 4:04SPEAKER_01: Most of the things that I'm using AI for don't have one right answer.
- 4:08SPEAKER_01: And so getting kind of a spread of perspectives is part of my reasoning
- 4:14SPEAKER_01: and decision making process.
- 4:17SPEAKER_01: That's like the anecdotal answer.
- 4:19SPEAKER_01: There's like a more kind of meta societal answer,
- 4:22SPEAKER_01: which is people don't trust AI answers for good reasons because They often spit out
- 4:28SPEAKER_01: an answer that is just one perspective and may be wrong in varying degrees,
- 4:34SPEAKER_01: but they present it with great confidence and don't really necessarily signal
- 4:39SPEAKER_01: the extent to which it could be wrong or how it might be wrong
- 4:42SPEAKER_01: or what other answers might be.
- 4:44SPEAKER_01: And so I think second opinion gives you literally a one tap or one click way
- 4:48SPEAKER_01: to mitigate that downside of using AI.
- 4:52SPEAKER_00: Yeah, so it seems like it solves a problem for the user or it's useful
- 4:57SPEAKER_00: to that one person on their search of getting more information
- 5:01SPEAKER_00: or a second perspective.
- 5:03SPEAKER_00: But it seems to also be useful as a critical thinking tool or useful to shape
- 5:09SPEAKER_00: the way that people use AI search.
- 5:12SPEAKER_00: This made me think of some research that Microsoft did.
- 5:16SPEAKER_00: Amy Quinton, Ph.D.: About how relying
- 5:19SPEAKER_00: on AI help affects people's cognitive engagement with the tasks that they're doing.
- 5:25SPEAKER_00: And a lot of people pointed at this study saying like, look, using AI makes you stupider.
- 5:30SPEAKER_00: When I went and read the study,
- 5:32SPEAKER_00: what it seemed to be more related to was actually your confidence
- 5:35SPEAKER_00: in the AI's answers, the more kind of The confident you become that it's correct,
- 5:41SPEAKER_00: then the less critical thinking you're doing,
- 5:45SPEAKER_00: Whereas when you have a healthy amount of skepticism,
- 5:48SPEAKER_00: it doesn't affect your cognitive engagement the same way.
- 5:52SPEAKER_00: So how can a feature like this kind of help shape the way that people use AI
- 5:57SPEAKER_00: to make it kind of less risky, more useful?
- 6:00SPEAKER_01: Yeah, I mean, I think like an interesting analog is like if you were doing research
- 6:05SPEAKER_01: on something the old fashioned way without AI,
- 6:07SPEAKER_01: you would probably read several sources.
- 6:09SPEAKER_01: Like if you're planning a trip,
- 6:10SPEAKER_01: you might read like this guidebook and this person's blog post
- 6:14SPEAKER_01: and this official website.
- 6:17SPEAKER_01: You know, AI is handy and that you can pick a button and get like one tidy answer.
- 6:24SPEAKER_01: But if that tidy answer is misleading or incomplete, then that's problematic.
- 6:28SPEAKER_01: And so my hope with second opinion is that it gives you an equally low Is it fair
- 6:33SPEAKER_01: to say that the round out those data points a bit and things?
- 6:38SPEAKER_01: And as a layperson, is there any hope for me a little bit more critical thinking No!
- 6:43SPEAKER_00: Hope for you Tatum. No!
- 6:50SPEAKER_00: Yeah, no what they are?
- 6:53SPEAKER_01: There's no hope for you a good
- 6:55SPEAKER_00: question. I mean,
- 7:02SPEAKER_01: I do think the short answer is no.
- 7:04SPEAKER_01: I think they're like the things are changing too fast and the nuances are too great.
- 7:10SPEAKER_01: I think that there are some dimensions at which a consumer could make informed choices.
- 7:17SPEAKER_01: Like one that I think probably is reasonable is you might have an intuition
- 7:21SPEAKER_01: of whether your question is like a very...
- 7:25SPEAKER_01: Quick answer is sufficient or whether you needed to do some like deeper thinking
- 7:29SPEAKER_01: and research and challenging itself.
- 7:30SPEAKER_01: And so on a given model, you can turn the reasoning up or down.
- 7:34SPEAKER_01: But like in terms of the distinction, you know,
- 7:36SPEAKER_01: when you're trying to figure out how to like remove butternut squash soup
- 7:40SPEAKER_01: from a T-shirt, the distinction between GPT-6 Luna and GPT-5.6 Luna and...
- 7:49SPEAKER_01: Claude's latest model, I do not think that a normal human could or should need
- 7:53SPEAKER_01: to hold those nuances in their head and be able to make a reasonable choice.
- 7:59SPEAKER_00: Yeah.
- 8:00SPEAKER_00: And it seems like that's built into this feature where instead of having to go back
- 8:03SPEAKER_00: to the search page and choose a second model from the dropdown menu,
- 8:07SPEAKER_00: because when you use DuckDuck.ai,
- 8:08SPEAKER_00: you can see all the models from different companies that we offer,
- 8:12SPEAKER_00: you can hit that button almost like a randomizer,
- 8:14SPEAKER_00: see a second opinion without having to feel like you have any insight
- 8:18SPEAKER_00: into why this one might be more or less useful.
- 8:22SPEAKER_01: Yeah, that's right.
- 8:23SPEAKER_01: And in fact, before second opinion existed,
- 8:26SPEAKER_01: and still we do have an option for you to go and Rerun your prompt
- 8:31SPEAKER_01: on a different model of your choosing.
- 8:33SPEAKER_01: But the cognitive friction, I think, is pretty significant because like I said,
- 8:37SPEAKER_01: most people are not going to know the right answer to which model.
- 8:41SPEAKER_01: And so you get to that dropdown and like you just give up.
- 8:45SPEAKER_01: And so, yeah, second opinion makes that choice for you.
- 8:48SPEAKER_00: So there's going to be a lot of instances, I think,
- 8:50SPEAKER_00: especially where people aren't searching for info.
- 8:52SPEAKER_00: Like if I'm using a chatbot to say,
- 8:54SPEAKER_00: please make me weekly schedule to make sure I get all my chores done
- 8:58SPEAKER_00: before Thursday, I'll know which one I like better. Easy, right?
- 9:02SPEAKER_00: But if I am looking for information,
- 9:03SPEAKER_00: any advice for people to know which answer might be more accurate and helpful
- 9:08SPEAKER_00: for them?
- 9:10SPEAKER_01: Yeah, that's a tough one.
- 9:12SPEAKER_01: I'd say the simplest tip would be if you asked a question,
- 9:17SPEAKER_01: Especially one that requires like fairly specified knowledge or recent information.
- 9:23SPEAKER_01: Check to see whether the answer is citing sources like, you know, web content.
- 9:28SPEAKER_01: If you see links like in DuckAI,
- 9:30SPEAKER_01: we'll present these clearly both in a row at the bottom and in line and you can see
- 9:36SPEAKER_01: Whether it actually went and read content to give you that answer.
- 9:40SPEAKER_01: And so first of all, just whether it did in fact read to give you
- 9:43SPEAKER_01: an answer is one clue of what happened and how credible it is.
- 9:47SPEAKER_01: And then secondly, you can actually see those sources,
- 9:51SPEAKER_01: click on them or hover over them to see where they're coming
- 9:54SPEAKER_01: from and whether they're credible sources or sources that seem appropriate to you.
- 9:58SPEAKER_01: And that will tell you a lot about how the answer got divined and whether it's the right one.
- 10:06SPEAKER_00: Gotcha. Because the chatbot can either perform an internet search in real time
- 10:12SPEAKER_00: or it can look backward to the training data that its developers fed into it
- 10:16SPEAKER_00: when it was being built.
- 10:17SPEAKER_00: And if it does the second one,
- 10:19SPEAKER_00: we can kind of infer that the information might not be up to date
- 10:23SPEAKER_00: or we just don't have the same visibility into where it's getting drawn from.
- 10:28SPEAKER_01: Yeah, because even like the looking backwards part,
- 10:30SPEAKER_01: it's not actually like recalling facts that it learned.
- 10:33SPEAKER_01: All it is doing is like predicting what a logical next token or set
- 10:39SPEAKER_01: of letters would be to try to complete the sentence.
- 10:42SPEAKER_01: And so like Even if it saw good training data that would be relevant
- 10:47SPEAKER_01: to your question, there's no guarantee that it's going to actually properly spit out
- 10:52SPEAKER_01: the right answer because of the sort of non-determinism factor.
- 10:56SPEAKER_01: Whereas if you see it giving you sources,
- 11:00SPEAKER_01: it's actually like reading those sources as guardrails on its answer.
- 11:05SPEAKER_00: And how long did it take us to build this?
- 11:08SPEAKER_00: And are people into it?
- 11:10SPEAKER_01: Yeah.
- 11:12SPEAKER_01: I think people are into it. I'm into it.
- 11:14SPEAKER_01: I remember when we initially released it, we got some feedback from a user.
- 11:19SPEAKER_01: We have feedback buttons all over our UI and we read the feedback that you send us,
- 11:24SPEAKER_01: so please do.
- 11:25SPEAKER_01: And it said something like, wow,
- 11:28SPEAKER_01: second opinion was the feature that I didn't realize I needed,
- 11:32SPEAKER_01: which I love because that's kind of how I feel about it and what I was hoping for.
- 11:37SPEAKER_01: It's not something that we were getting demand for,
- 11:39SPEAKER_01: but I think the sort of There was an underlying demand that is maybe difficult
- 11:46SPEAKER_01: to articulate in the form of a feature request for people to have some way of like,
- 11:53SPEAKER_01: Understanding the veracity of answers or the universe of possibilities or not,
- 12:01SPEAKER_01: you know, being kind of at the mercy of a single model's like stream of thought.
- 12:06SPEAKER_01: Once a model goes in a given direction,
- 12:09SPEAKER_01: it wants to keep going in that direction because it's just predicting what
- 12:12SPEAKER_01: a rational or what a likely probable next token would be.
- 12:16SPEAKER_00: It's also really hard, I think,
- 12:18SPEAKER_00: when you're doing a standard internet search to get a snapshot
- 12:21SPEAKER_00: of whether a certain question or topic is contested at all,
- 12:24SPEAKER_00: whether there's some nuance or complexity to the correct answer.
- 12:28SPEAKER_00: Whereas with second opinion, for example, with the butternut squash, Substance.
- 12:32SPEAKER_00: If both answers told me to do the exact same thing.
- 12:34SPEAKER_00: I'm like, OK, I can feel quite confident that this is an established answer to what
- 12:39SPEAKER_00: to do with this topic.
- 12:40SPEAKER_00: Whereas if I get two quite different answers now,
- 12:43SPEAKER_00: not only do I have more ideas to play with,
- 12:45SPEAKER_00: I also understand more about my own question that this you know,
- 12:50SPEAKER_00: this is a little bit complex and I am going to get different answers
- 12:53SPEAKER_00: from different sources.
- 12:54SPEAKER_01: Yeah, if nothing else, it gives you some good follow up questions, right?
- 12:57SPEAKER_01: Like you can ask the second model, like,
- 12:59SPEAKER_01: should I really scoop it off with a spoon or should I skip right
- 13:02SPEAKER_01: to and like push back in maybe ways that would have been hard for you
- 13:07SPEAKER_01: to to devise yourself without.
- 13:09SPEAKER_01: Another friend sitting around the table, right?
- 13:12SPEAKER_00: Yeah.
- 13:13SPEAKER_00: And you mentioned before this conversation that you can either continue talking,
- 13:18SPEAKER_00: you know, continue talking to that model, or you can say, no, no,
- 13:20SPEAKER_00: let's go back to that first one I got and pick up the conversation there.
- 13:24SPEAKER_01: That's right.
- 13:24SPEAKER_01: You can use the second opinion as a tool to get a better answer from that first one
- 13:29SPEAKER_01: or vice versa and take it in whatever direction you want
- 13:33SPEAKER_01: because of my role building product at DuckDuckGo.
- 13:36SPEAKER_01: Often when I'm using AI chat,
- 13:39SPEAKER_01: I will run the same prompt on two or three products to do sort of a compare
- 13:43SPEAKER_01: and contrast. Like how does DuckAI answer this?
- 13:45SPEAKER_01: How does Gemini answer this?
- 13:47SPEAKER_01: How does ChatGPT answer this?
- 13:49SPEAKER_01: And I am constantly like, you know,
- 13:53SPEAKER_01: weaponizing the best parts of the The competitive answer against my chosen,
- 13:58SPEAKER_01: you know, bring in this little tidbit from this answer and try to kind of like force
- 14:04SPEAKER_01: the conversation that I continue into a better place.
- 14:07SPEAKER_01: And so not everybody wants to have like three windows open doing that, I imagine,
- 14:12SPEAKER_01: but you can kind of do it in one place with one click in DuckAI now.
- 14:16SPEAKER_00: So it seems like a big takeaway here is
- 14:18SPEAKER_00: that not only can you use these Popular chatbots completely privately,
- 14:25SPEAKER_00: you can use them in the same conversation.
- 14:28SPEAKER_00: You can access them in one place and even within one train of thought.
- 14:33SPEAKER_00: Yep, exactly.
- 14:35SPEAKER_00: One thing I'll say is that we are still working on this
- 14:39SPEAKER_01: feature and improving it
- 14:41SPEAKER_01: and learning from feedback.
- 14:43SPEAKER_01: So, you know, I think there's some aspects of the design that could be simplified or clarified.
- 14:50SPEAKER_01: If you're using it and enjoying it, stay tuned for some potential updates.
- 14:55SPEAKER_01: And if you do have feedback,
- 14:56SPEAKER_01: please send it our way so we can work that right into our design
- 15:01SPEAKER_01: and decision-making process as we continue to iterate on a second opinion.
- 15:07SPEAKER_00: Awesome. Bia, thank you so much for coming and telling us about this.
- 15:11SPEAKER_00: Thank you. It's my pleasure.