Latest / Inside DuckDuckGo / Duck Tales: Building useful, private, optional AI directly into search, with Search Assist (Ep.14)
Transcript
- 0:00SPEAKER_00: Hello, everybody.
- 0:00SPEAKER_00: Welcome to DuckTales.
- 0:02SPEAKER_00: I'm Gabriel, founder of DuckDuckGo.
- 0:05SPEAKER_00: With me is Ewa.
- 0:05SPEAKER_00: You want to introduce yourself, Ewa?
- 0:06SPEAKER_01: Ewa Zobula Hi, everyone.
- 0:09SPEAKER_01: I'm Ewa Zobula.
- 0:10SPEAKER_01: I'm a product person at DuckDuckGo based in Poland.
- 0:17SPEAKER_00: Cool.
- 0:17SPEAKER_00: We are going to talk about Search Assist today.
- 0:20SPEAKER_00: We've done one episode before when we introduced the more button.
- 0:26This is the stuff on the top of search results where our anonymous AI is
- 0:32answering
- 0:34SPEAKER_00: queries for you.
- 0:34SPEAKER_00: And you can click more,
- 0:36butwe'regoingtogotakeastepbackandjustkindoftalkaboutthefeaturein
- 0:40SPEAKER_00: general.
- 0:43SPEAKER_00: As a precursor,
- 0:44SPEAKER_00: I've said this a bunch of times at AI episodes,
- 0:47but our kind of our guiding principles for AI features are that they're
- 0:54useful,
- 0:54SPEAKER_00: private and optional.
- 0:56SPEAKER_00: So in this case,
- 0:58SPEAKER_00: and I know people really appreciate that,
- 1:01SPEAKER_00: we think search assist is extremely useful and we'll get into that,
- 1:05SPEAKER_00: but it is also optional.
- 1:06SPEAKER_00: So you can turn it off if you like.
- 1:09SPEAKER_00: It's really easy to do so.
- 1:10SPEAKER_00: There's toggles like actually within search assist itself,
- 1:13SPEAKER_00: but also in the browser and search settings.
- 1:17SPEAKER_00: And of course, it's private, like the rest of our search results.
- 1:20SPEAKER_00: It's completely anonymous.
- 1:22SPEAKER_00: So with that, yeah, let's just jump from the highest level.
- 1:25SPEAKER_00: Yeah.
- 1:26SPEAKER_00: What is search assist and how does it work?
- 1:30SPEAKER_01: So as you mentioned,
- 1:32search assist is our AI-generated instant answer to search queries that we
- 1:39showup
- 1:39SPEAKER_01: on top of search results page.
- 1:42Once we are confident that this is going to be the optimal answer to use a
- 1:48query,
- 1:48meaning for queries where you either ask a question or really are looking for
- 1:52a
- 1:53SPEAKER_01: quick summary,
- 1:55SPEAKER_01: And we are now showing it on like roughly quarters of our searches.
- 2:02SPEAKER_01: And we are using LLNs to create the answer,
- 2:06but what is important is that we actually ground it in the right sources and
- 2:09like
- 2:12SPEAKER_01: verified and checked sources.
- 2:14SPEAKER_01: So it's not like generated literally just by an LLN,
- 2:18SPEAKER_01: but we find the relevant sources to the search query.
- 2:23SPEAKER_01: We analyze them and we synthesize the concise one,
- 2:27two sentences answer that we show on top of SERP when it's really relevant.
- 2:33Or we also showed like lower down the page where you might have different
- 2:39intent.
- 2:41SPEAKER_01: But still, it could be useful if it's something that you scroll down to.
- 2:48SPEAKER_00: Yeah, absolutely.
- 2:51SPEAKER_00: And I want to get into the you mentioned it being concise.
- 2:54SPEAKER_00: And that's one of the main differences I feel we have with Google.
- 2:57SPEAKER_00: And I want to get into a bunch of those.
- 2:59SPEAKER_00: But before we do that, let's continue with the basics.
- 3:04SPEAKER_00: I mean, so I actually started working on this feature.
- 3:08SPEAKER_00: Maybe people think it was in
- 3:10SPEAKER_00: reaction to Google or something, but it wasn't.
- 3:13SPEAKER_00: We actually started working on this as soon as ChatGPT came out,
- 3:17SPEAKER_00: um,
- 3:18SPEAKER_00: to really initially focus on Wikipedia and helping,
- 3:22SPEAKER_00: you know,
- 3:23SPEAKER_00: giving people better Wikipedia answers,
- 3:25SPEAKER_00: which,
- 3:26SPEAKER_00: um,
- 3:26SPEAKER_00: you mentioned searches just now appears in about 25% of searches,
- 3:30SPEAKER_00: but Wikipedia appears there at about 10%.
- 3:31SPEAKER_00: So, um,
- 3:34And we had gotten lots of feedback over the years that,
- 3:38hey,
- 3:39it'd be great.
- 3:40SPEAKER_00: You're showing me the beginning of Wikipedia,
- 3:42SPEAKER_00: but it'd be great if you could just show me the answer from Wikipedia.
- 3:46SPEAKER_00: And we had tried that in different ways and we accomplished that.
- 3:50somewhat but until lms came along where we could really pull back the
- 3:55paragraphsof
- 3:56wikipedia and ask the lm to pull out the answer within that paragraph which is
- 4:00much
- 4:01better than just showing the paragraph and making you find it we weren't
- 4:04really
- 4:04abletounlockuhsearchassistsoyeahwestartedworkingonitrightwhich
- 4:10attributescameoutumandkindofrushedtowikipediaandtheniknowyougot
- 4:15SPEAKER_00: involved later when we um
- 4:18SPEAKER_00: you know, started adding lots of other sources.
- 4:20SPEAKER_00: And as you mentioned, we're trying to use the best sources we can.
- 4:26SPEAKER_00: But I'd say more broadly,
- 4:27SPEAKER_00: given that it's kind of a broader thing that Wikipedia now,
- 4:30SPEAKER_00: like,
- 4:30SPEAKER_00: what do you see the problem that it's solving in search results?
- 4:36SPEAKER_00: Just kind of for at large,
- 4:37SPEAKER_00: like you mentioned,
- 4:38SPEAKER_00: sometimes we put it on top,
- 4:39SPEAKER_00: sometimes we put it on bottom.
- 4:41SPEAKER_00: Obviously, that's a choice.
- 4:43We're putting it on top because we think it's solving a search problem,
- 4:45right?
- 4:47So I think the key problem that we're solving with Assist is that it shortens
- 4:54the
- 4:54SPEAKER_01: path from when you know what you want to ask and you formulate a query
- 5:02SPEAKER_01: to actually finding what you're looking for.
- 5:05SPEAKER_01: And to your point,
- 5:07SPEAKER_01: we've already been doing Wikipedia or other modules in the past,
- 5:11SPEAKER_01: and we're still doing them for many of the searches.
- 5:15SPEAKER_01: But Assist allows us to cover more of these informational queries,
- 5:21SPEAKER_01: including the long tail ones,
- 5:23meaning people use different language to ask search engine about what
- 5:27they're
- 5:27SPEAKER_01: looking for.
- 5:29SPEAKER_01: And with ASSIST,
- 5:31we've been able to understand more of these natural language queries or
- 5:36queries
- 5:36SPEAKER_01: that really ever are asked only once to a search engine,
- 5:40SPEAKER_01: which is a huge portion of search queries.
- 5:45But we still can understand them and can present an answer that is good enough
- 5:51to
- 5:52SPEAKER_01: answer what you're really looking for.
- 5:54SPEAKER_01: But also,
- 5:55SPEAKER_01: with the more button that you've already mentioned,
- 5:57alsoallowsyoutodigdeeperandgetmoreinformationondemandwhilestill
- 6:04SPEAKER_01: keeping you in the search engine context,
- 6:05in the context that a lot of people are familiar with because we've been using
- 6:09it
- 6:09SPEAKER_01: for years,
- 6:10SPEAKER_01: years,
- 6:10SPEAKER_01: some of us ever since they were born.
- 6:14And so it's kind of like bridges the gap between the value that LLMs bring and
- 6:20how
- 6:20they can enrich the experience of finding information without having to
- 6:26movetoa
- 6:28SPEAKER_01: totally different user interface,
- 6:30SPEAKER_01: to move to more like conversational chat experiences.
- 6:35SPEAKER_01: It's still search results that are familiar.
- 6:37SPEAKER_01: It's the search results that these answers are grounded for.
- 6:42but we're making use of this technology to present it in a more suitable way
- 6:46for
- 6:47SPEAKER_01: larger volume of different types people ask search engine.
- 6:52SPEAKER_00: Yeah.
- 6:53SPEAKER_00: And so maybe I summarize that by it really is saving people time.
- 6:59SPEAKER_00: And I think as a primary benefit,
- 7:03I think as a secondary benefit in aligning with our vision of RAID's standard
- 7:07of
- 7:07SPEAKER_00: trust online is that we're trying to,
- 7:09SPEAKER_00: um,
- 7:12SPEAKER_00: you know,
- 7:13understandwhatisthekindofthebestinformationinthesearchresultsand
- 7:18SPEAKER_00: service that for you in a concise way.
- 7:21SPEAKER_00: So that not only saves you time,
- 7:22but it on average should be giving you better information higher up on the
- 7:27page,
- 7:28SPEAKER_00: which is kind of really what you want in a search engine.
- 7:31SPEAKER_00: And just to restate for people who really don't,
- 7:34SPEAKER_00: One AI, you can turn it off.
- 7:36SPEAKER_00: However, this is not, to your point earlier, AI making up the answer.
- 7:40This is us grounding the answer on actual search pages that are crawling in
- 7:45real
- 7:45SPEAKER_00: time to look for that answer for you.
- 7:48SPEAKER_00: And then the sources are annotated there,
- 7:52which you can click through and both check and read more information because
- 7:57we're
- 7:58SPEAKER_00: only giving you a concise summary.
- 7:59SPEAKER_00: So if you want more, you click through.
- 8:02SPEAKER_00: So with that in mind, yeah,
- 8:04SPEAKER_00: How, yeah, you mentioned the more button in the UX.
- 8:06SPEAKER_00: Before we kind of dive into kind of differences,
- 8:12what has the general user reaction been like over time with Search Assist?
- 8:19SPEAKER_01: So we've been getting a lot of really positive reactions from our users.
- 8:25SPEAKER_01: Assist has been like one of the highest rated parts of search experience
- 8:31SPEAKER_01: historically at DuckDuckGo.
- 8:34And I think what people usually appreciate is both that it saves time to get
- 8:42this
- 8:42SPEAKER_01: concise answer.
- 8:43Thefactthatit'sreallyconciseandit'snotliketakingoveryoursearch
- 8:48SPEAKER_01: experience is just there when you need it,
- 8:50but still doesn't make it hard to get to organics if that's what you're
- 8:55looking
- 8:56SPEAKER_01: for.
- 8:57is another thing that people have appreciated and the ability to drill down
- 9:01to
- 9:01sources as you've mentioned is also something we've heard been getting like
- 9:08really
- 9:08SPEAKER_01: good reception and we are using
- 9:13SPEAKER_01: the feedback we're getting from people a lot in improving assist.
- 9:17SPEAKER_01: On one hand, that's because we really don't track our users.
- 9:21We have very little information about how people interact with our search
- 9:26results.
- 9:27SPEAKER_01: So we really rely on when people make effort to click thumbs up,
- 9:33SPEAKER_01: click thumbs down,
- 9:35SPEAKER_01: leave some additional comment.
- 9:37SPEAKER_01: We use this information both in automated way to improve our answers.
- 9:44And also we really do read through them and take lessons and figure out how we
- 9:50can
- 9:50SPEAKER_01: continue improving a system,
- 9:52SPEAKER_01: which is,
- 9:52SPEAKER_01: for instance,
- 9:53SPEAKER_01: how we've gotten the more button.
- 9:56SPEAKER_00: Yeah, I mean, so...
- 9:58That seems like a big difference from Google right there for what it's worth.
- 10:01SPEAKER_00: I mean, I guess I'm not inside Google, so I don't know.
- 10:03SPEAKER_00: But from reading comments on Hacker News and other places,
- 10:06SPEAKER_00: it does not necessarily seem like they read every piece of feedback.
- 10:10SPEAKER_00: But we actually do.
- 10:12SPEAKER_00: And so that really is a distinction.
- 10:16SPEAKER_00: We mentioned some others, too.
- 10:19SPEAKER_00: So I mentioned that it was optional.
- 10:21I think we should clarify now that it's not just you can turn it on or off,
- 10:24SPEAKER_00: which you can,
- 10:26SPEAKER_00: but you can also change the frequency of when Search Assist appears.
- 10:31SPEAKER_00: We have often and sometimes, as well as on demand.
- 10:35SPEAKER_00: So you can basically make it so it doesn't show up automatically,
- 10:39SPEAKER_00: but if you want to click on it,
- 10:40SPEAKER_00: you can still click the Search Assist button on the screen.
- 10:44underneath the search box and it'll show just for when you want it on demand.
- 10:48SPEAKER_00: And if you really,
- 10:48SPEAKER_00: really don't want it,
- 10:49SPEAKER_00: including not even seeing that button on the page,
- 10:52SPEAKER_00: you can get rid of that.
- 10:54SPEAKER_00: So that's another one.
- 10:55SPEAKER_00: Another one that you mentioned.
- 10:58SPEAKER_00: So the conciseness, I think...
- 11:02SPEAKER_00: you know, it does a couple things.
- 11:04SPEAKER_00: One,
- 11:05SPEAKER_00: it means that the less information there is,
- 11:08the less kind of surface area there is for making stuff up or getting stuff
- 11:12wrong.
- 11:14But also what you had just mentioned,
- 11:16it just takes up less space on the page.
- 11:18SPEAKER_00: So I think some complaints that people have about Google's
- 11:22SPEAKER_00: AO reviews is they really are just taking up,
- 11:24SPEAKER_00: you can't see the organic links,
- 11:26SPEAKER_00: they're taking up too much of the page.
- 11:28SPEAKER_00: We're really keeping it tight.
- 11:30SPEAKER_00: And the way we're doing that is we're kind of forcing it to be small,
- 11:34SPEAKER_00: but also to your point,
- 11:36SPEAKER_00: we have the more button there as an option to
- 11:40SPEAKER_00: But we're not starting out with that long explanation.
- 11:44SPEAKER_00: We're starting out with this concise explanation.
- 11:48Isthereanythingelseyouwanttohighlightintermsofkindofdifferences
- 11:53SPEAKER_00: between us and Google?
- 11:55SPEAKER_00: If not, I can probably come up with something.
- 11:59Maybe like building on top of what you just talked about is I think what I can
- 12:03tell
- 12:05SPEAKER_01: about our philosophy of building assist,
- 12:08SPEAKER_01: which I think also applies to the whole of search,
- 12:12SPEAKER_01: is that we like to give control to our users.
- 12:16SPEAKER_01: We trust that they know they can...
- 12:20own their experience,
- 12:22which is why we're giving you search results,
- 12:24which some of it is assist.
- 12:27SPEAKER_01: You get the organics, you get other modules.
- 12:30None of these experiences is like overly dominating the search results
- 12:35page.
- 12:36And you can kind of like choose your own adventure,
- 12:38which I think is,
- 12:42again,
- 12:42I'm not in Google.
- 12:43SPEAKER_01: I don't know exactly how decisions are made in there.
- 12:47SPEAKER_01: But from my observation,
- 12:49it's like a different approach to how we give control versus kind of like
- 12:55deciding
- 12:56for the user what they should be seeing when they search for a specific topic
- 13:01or
- 13:02SPEAKER_01: query.
- 13:05SPEAKER_00: Yeah, that's a great point.
- 13:06I mean,
- 13:06I would even build on that,
- 13:08which is we've also been given control to publishers.
- 13:13There are some publishers who don't,
- 13:16you know,
- 13:16want to be part of AI results.
- 13:19Wearetryingourbesttomaketheclick-throughrateactuallyprettydecent
- 13:22SPEAKER_00: because we have a short summary.
- 13:24And so really,
- 13:25if you want more,
- 13:25you click through and the links are prominent.
- 13:29SPEAKER_00: Like on Google,
- 13:30SPEAKER_00: sometimes they're hidden actually behind the click and on DuckDuckGo,
- 13:32SPEAKER_00: they're not.
- 13:34morefundamentallyinthatifyoudowanttooptoutwehaveauhkindofwayfor
- 13:39publisherstojustoptoutoftheaisummariesumviarobust.txtumandstill
- 13:46appear totally fine in search results yes um whereas google has been trying
- 13:50to
- 13:50SPEAKER_00: bundle those things together and basically force you to opt out at google
- 13:55altogether um so that's another decently big difference aligned with
- 13:59controlum
- 14:01SPEAKER_00: okay so yeah what what's um
- 14:05SPEAKER_00: What's next?
- 14:05SPEAKER_00: What are the negative feedback complaints we're getting?
- 14:08SPEAKER_00: And what are we working on?
- 14:10SPEAKER_00: Where are we going?
- 14:12SPEAKER_01: So one of the things that we continue working on,
- 14:16SPEAKER_01: and we will pretty much continue,
- 14:17SPEAKER_01: I think,
- 14:18SPEAKER_01: forever,
- 14:19SPEAKER_01: is that we want to keep the high bar for answers quality.
- 14:25SPEAKER_01: There are still cases where we get the answer wrong.
- 14:29SPEAKER_01: It's not incorrect, or maybe the sources weren't right.
- 14:34SPEAKER_01: or we misinterpret the query.
- 14:38SPEAKER_01: The percent of these incorrect answers is really low.
- 14:42SPEAKER_01: But as I said, we want to really take correctness seriously.
- 14:48SPEAKER_01: So we're working to add even more loops,
- 14:53SPEAKER_01: both with user feedback,
- 14:55SPEAKER_01: with observing the queries,
- 14:57SPEAKER_01: especially the fresh ones.
- 15:00SPEAKER_01: like trending topics.
- 15:02This is where information changes really often and we want ASSIST to always
- 15:07give
- 15:08SPEAKER_01: you the fresh and correct answer.
- 15:10So definitely answer quality is something that we will continue working on.
- 15:15SPEAKER_01: But we also are moving to other languages besides English.
- 15:22SPEAKER_01: ASSIST has been available in English for a while already.
- 15:27SPEAKER_01: That hasn't been true for other languages.
- 15:29SPEAKER_01: So it's already available in Italy.
- 15:33SPEAKER_01: For some queries, we're only rolling it down.
- 15:35You'llstartseeingitratherfurtherdownthepagebeforewebuildenough
- 15:40SPEAKER_01: confidence to show it on top of your results.
- 15:43SPEAKER_01: But you can already start seeing it there.
- 15:45SPEAKER_01: And we plan to roll out to a few next languages,
- 15:50SPEAKER_01: which are going to be Spanish,
- 15:52SPEAKER_01: Dutch,
- 15:53SPEAKER_01: French,
- 15:54SPEAKER_01: German.
- 15:56SPEAKER_01: And we won't stop there.
- 15:57SPEAKER_01: But this is like the current focus, which I'm personally excited about.
- 16:03SPEAKER_01: Because...
- 16:06SPEAKER_00: I'm sure a lot of team members are.
- 16:08SPEAKER_00: I mean, we have, I mean, this is more intro.
- 16:11SPEAKER_00: We have people,
- 16:12SPEAKER_00: I think across like 30 something countries now,
- 16:15SPEAKER_00: and a lot of them are not English as their first language.
- 16:18SPEAKER_00: And I think they would prefer to have non-English search assist answers.
- 16:22SPEAKER_00: So that's exciting.
- 16:23SPEAKER_01: Absolutely.
- 16:24SPEAKER_01: But also our users.
- 16:25I think we,
- 16:26weknowthatpeopleallovertheworldareusingtech.cosearchand
- 16:31Ipersonallycan'twaittohavethemuseAssistandalsogetbenefitsofthese
- 16:39SPEAKER_01: short and concise answers.
- 16:43We're also looking into improving and expanding or continue improving the
- 16:49user
- 16:50SPEAKER_01: experience of Assist.
- 16:52SPEAKER_01: We've been already mashing the more button a few times.
- 16:56We know that there are sometimes like different expectations people have
- 17:01when
- 17:01SPEAKER_01: clicking more button.
- 17:02SPEAKER_01: Sometimes it's really,
- 17:03SPEAKER_01: you want to have the longer version of the answer,
- 17:05but sometimes it's more like you want to know more information or like
- 17:11additional
- 17:12SPEAKER_01: thing that wasn't mentioned,
- 17:13SPEAKER_01: a concise answer.
- 17:15SPEAKER_01: Or maybe you want to drill down, do more like
- 17:18learn about related things so we're now looking into exploring these
- 17:22optionsand
- 17:23and making sure that expound answers are like really helpful depending on
- 17:29what
- 17:29you'relookingforcoolIknowwe'realsoalwaystryingtospeedthingsupso
- 17:38SPEAKER_00: reducing latency getting answer faster it's always on the list
- 17:43SPEAKER_00: Yeah,
- 17:43SPEAKER_00: I guess the only thing else that comes to mind is,
- 17:46SPEAKER_00: you know,
- 17:46SPEAKER_00: we've been,
- 17:48SPEAKER_00: it might be worth pointing out is,
- 17:50SPEAKER_00: which we mentioned in the past,
- 17:51SPEAKER_00: that this actual technology stack is completely independent from Bing and
- 17:57SPEAKER_00: Microsoft.
- 17:59SPEAKER_00: You know, we've been kind of obviously working with the LM partner.
- 18:02SPEAKER_00: We're not building a foundational model ourselves, but the stack is,
- 18:07youknowiscomingviaourowncrawlinguhtheumthewebpageswherewe're
- 18:13generating the answers and that index is improving every day and so we're
- 18:19working
- 18:19on that as well um and ultimately we hope to say more about that um i think
- 18:27that's
- 18:27itimeantheumanythingelseyouwanttoaddordowehiteverythingumforthe
- 18:34SPEAKER_00: basics
- 18:35SPEAKER_01: I think we hit everything.
- 18:38SPEAKER_01: The one thing which is interesting to me,
- 18:44SPEAKER_01: I don't know if it's going to be interesting to others,
- 18:46SPEAKER_01: but I think it's also worth mentioning how we determine,
- 18:52SPEAKER_01: for instance,
- 18:53answer correctness and how we've been actually using a lot of manual reviews
- 18:59of
- 18:59SPEAKER_01: queries and answers.
- 19:01SPEAKER_01: within the team, which I think is not obvious.
- 19:06SPEAKER_01: Some people might assume that if you're using AI-generated also,
- 19:09SPEAKER_01: you also delegate to the AI to determine if it's correct or not,
- 19:14SPEAKER_01: or if it includes enough details or not too much details.
- 19:19Sothewaywe'vebeenbuildingAssististhatwe'vedonealotofthatmanual
- 19:25SPEAKER_01: judgment ourselves and we rely on these manually within the datasets.
- 19:32I'mprettysurequiteafewpeopleintheteammightcomplainalotabouthow
- 19:41SPEAKER_01: And maybe it's not the most exciting work while you do it.
- 19:46SPEAKER_01: But I think it's extremely useful in really ensuring that these answers,
- 19:51SPEAKER_01: when we say our correctness rate is high,
- 19:55SPEAKER_01: this doesn't mean that someone or AI tells us we're doing a good job.
- 20:00SPEAKER_01: It's really, we've reviewed it.
- 20:01SPEAKER_01: a huge,
- 20:02SPEAKER_01: huge,
- 20:03hugedatasetofqueriesandwelikeretestthemovertimetoensurethatthis
- 20:10SPEAKER_01: quality stays high.
- 20:13SPEAKER_00: That's good.
- 20:13SPEAKER_00: That's a good way to end.
- 20:15SPEAKER_00: I guess I have been forgetting to say,
- 20:17SPEAKER_00: if you're listening to this and you do have follow-up questions,
- 20:20SPEAKER_00: email us.
- 20:22You can reply to the email that you get if you subscribe to us on Substack.
- 20:27Otherwise,
- 20:28we probably need to develop an email address that other people can send to us.
- 20:32SPEAKER_00: So I'll have to get back on that.
- 20:33SPEAKER_00: But if you subscribe to Substack,
- 20:34youcanjustreplytothatemailandit'llcometousandwe'llgetitforfuture
- 20:39SPEAKER_00: episodes.
- 20:39SPEAKER_00: So thank you everyone for coming on and thanks everyone for tuning in.
- 20:43SPEAKER_01: Thanks, Fabril.
- 20:44SPEAKER_01: Thanks, everyone.