Latest / Inside DuckDuckGo / Duck Tales: Why DuckDuckGo is building its own web search index (Ep.22)
Transcript
- 0:00SPEAKER_01: Hello, welcome back to DuckTales.
- 0:02SPEAKER_01: I haven't been here in a while.
- 0:04SPEAKER_01: And I am Gabriel, the founder of Duck Tales
- 0:07And I have with me someone who I don't think has been on DuckTales at all yet,
- 0:10SPEAKER_01: but you should know,
- 0:13SPEAKER_01: Caine
- 0:15SPEAKER_01: who I know very well,
- 0:17SPEAKER_01: who's the first employee of DuckTales.co and now our CTO.
- 0:22SPEAKER_01: Hi, Caine.
- 0:24SPEAKER_01: Hi, Gabe.
- 0:26SPEAKER_01: Uh, we've worked together for a very long time.
- 0:28SPEAKER_01: Um,
- 0:29SPEAKER_01: and we're here today to talk about something we've both been working on,
- 0:32SPEAKER_01: uh,
- 0:33SPEAKER_01: came more than me,
- 0:34SPEAKER_01: but,
- 0:35SPEAKER_01: uh,
- 0:35SPEAKER_01: I'm working on some,
- 0:36SPEAKER_01: which is our web search index.
- 0:40SPEAKER_01: So, um,
- 0:42SPEAKER_01: As some background, first some background.
- 0:45SPEAKER_01: Yeah, for sure.
- 0:45SPEAKER_01: DuckDuckGo started as a search engine, as many people know.
- 0:49SPEAKER_01: And it was actually started by me.
- 0:51SPEAKER_01: I was by myself for a few years.
- 0:53And the first thing I did was start crawling the web and building a web index.
- 0:58But I soon realized that that is very expensive,
- 1:04especially as one person.
- 1:08SPEAKER_01: And there were other places to get a web index at the time.
- 1:12SPEAKER_01: what was more interesting was maybe adding value on top of the web index.
- 1:15SPEAKER_01: So building other indexes,
- 1:17SPEAKER_01: um,
- 1:18SPEAKER_01: this was a time,
- 1:19SPEAKER_01: this is the mid two thousands,
- 1:20SPEAKER_01: you know,
- 1:20SPEAKER_01: there weren't,
- 1:21SPEAKER_01: there obviously wasn't AI,
- 1:22SPEAKER_01: but there wasn't even really many instant answers.
- 1:25I mean,
- 1:25that's what we were working on together at the very,
- 1:28very beginning.
- 1:29SPEAKER_00: Like we were working on, um, you know, you had the knowledge graph.
- 1:34SPEAKER_00: It wasn't called a knowledge graph at a time,
- 1:35SPEAKER_00: but you were doing all the structured content,
- 1:38SPEAKER_00: um,
- 1:39SPEAKER_00: from Wikipedia and otherwise,
- 1:41SPEAKER_00: um,
- 1:42SPEAKER_00: We worked on some other smaller indices.
- 1:44SPEAKER_00: So, yeah.
- 1:46SPEAKER_00: And then actually,
- 1:47SPEAKER_00: fun fact,
- 1:49SPEAKER_00: in hiring,
- 1:50our backend project is still based on some of the original spam and content
- 1:55farm
- 1:56SPEAKER_00: crawling.
- 1:57Likeoneoftheprojectsisbasedonsomeofthespamandcontentfarmcrawlers
- 2:00SPEAKER_00: that you originally wrote.
- 2:03SPEAKER_00: Yeah.
- 2:03SPEAKER_00: So that lives on 15 years.
- 2:05lotsofindexingandlotsofcrawlingyeahexactlyuhjustnotuhyouknowwe
- 2:11started but then we stopped doing a full web index but just as examples right
- 2:15weum
- 2:17started like the code you're talking about indexing wikipedia which became
- 2:21our
- 2:21knowledge graph you know which is powers a lot of answers which also we used
- 2:26when
- 2:26SPEAKER_01: we started working on ai answers um we've been doing local indexing for
- 2:32SPEAKER_01: I don't know, over a decade.
- 2:34SPEAKER_01: Local businesses.
- 2:35SPEAKER_00: 15 years.
- 2:36SPEAKER_01: Things like that.
- 2:39SPEAKER_01: You know,
- 2:39SPEAKER_01: then all sorts of kind of nature indexes that involve some crawling,
- 2:43SPEAKER_01: like lyrics and things like that.
- 2:46So indexing technology is not new to us,
- 2:49despite what some people say about it.
- 2:51Sometimes we do lots of search indexing,
- 2:54butwehadn'tbeendoingafullwebindexuntil
- 3:02SPEAKER_01: relatively recently, the last few years ish.
- 3:06SPEAKER_01: But now we are.
- 3:07And so the question is the questions and why you're here and we'll talk about
- 3:12it
- 3:12SPEAKER_01: for a few minutes is kind of why what's going on,
- 3:16SPEAKER_01: how all the main all the main questions,
- 3:18SPEAKER_01: which we're obviously not going to answer today.
- 3:20SPEAKER_01: But we can start with, I think, like, you know,
- 3:24SPEAKER_01: Kind of the why,
- 3:25SPEAKER_01: but like kind of why are we well positioned,
- 3:29SPEAKER_01: you know,
- 3:30SPEAKER_01: to work on this,
- 3:31SPEAKER_01: you know,
- 3:32SPEAKER_01: so to speak.
- 3:33SPEAKER_01: And you're kind of at the center of it.
- 3:34SPEAKER_01: So I think you're a good person to ask.
- 3:36SPEAKER_00: Yeah, I mean, I think the why now is a mixture of like our needs.
- 3:44SPEAKER_00: Like we want to support our own AI use cases.
- 3:48We have two primary agent-driven products out,
- 3:53Search Assist,
- 3:53which is on the SERP.
- 3:55SPEAKER_00: search engine results page, duck.go.com.
- 3:58SPEAKER_00: And then we have Duck AI, which is our chat bot.
- 4:02SPEAKER_00: And both of those products are hungry for this kind of data.
- 4:07SPEAKER_00: Yeah, in particular, right?
- 4:09SPEAKER_00: Yeah, it just makes sense for that.
- 4:10SPEAKER_01: Maybe talk percentages,
- 4:12SPEAKER_01: but like there's some percentage of search results now,
- 4:15SPEAKER_01: what is it,
- 4:15SPEAKER_01: like 25%,
- 4:16SPEAKER_01: I think,
- 4:16SPEAKER_01: that have search assist answers.
- 4:18SPEAKER_01: And then,
- 4:19SPEAKER_01: you know,
- 4:19SPEAKER_01: the percentage better made for DuckAI,
- 4:20SPEAKER_01: but some significant percent call the web.
- 4:23SPEAKER_00: I always do.
- 4:24SPEAKER_00: 15% maybe?
- 4:25SPEAKER_00: I do.
- 4:27SPEAKER_00: I have my numbers based on sometimes absolutes make more sense.
- 4:33SPEAKER_00: Yeah, doesn't matter.
- 4:34SPEAKER_01: Good percentage of queries and DuckAI prompts require web search.
- 4:39SPEAKER_01: And so we need a web mix for it, essentially, right?
- 4:42SPEAKER_00: Yeah, I mean, I think on the chatbot side, it's really good to ground.
- 4:47SPEAKER_00: If you're deciding whether or not to ground and you're on the line,
- 4:50SPEAKER_00: you should probably use RAG,
- 4:52SPEAKER_00: retrieval augmented grounding and go out to a third-party data set.
- 4:56SPEAKER_00: For us,
- 4:56SPEAKER_00: raising the standard of trust online,
- 4:58SPEAKER_00: we want to do that because the more that you ground,
- 5:01SPEAKER_00: it's known empirically the answers are better.
- 5:03So we err on the side of grounding where I think maybe not everybody does.
- 5:09So it's really good that we need to build our own index in order to be able to
- 5:13accommodate that.
- 5:15SPEAKER_00: So that's kind of some of the why now.
- 5:18SPEAKER_00: And again, it's on Search Assist and it's on DuckAI.
- 5:23SPEAKER_00: One of my favorite parts about this whole thing is like,
- 5:27SPEAKER_00: we're very used to working for customers like our end user.
- 5:32SPEAKER_00: For the search index, DuckDuckGo itself is the customer.
- 5:36SPEAKER_00: It's a very nuanced,
- 5:38SPEAKER_00: unique thing for us to be able to serve ourselves,
- 5:41SPEAKER_00: which creates this really tight feedback loop internally.
- 5:45Soit'sbeencooltouseourownandweareliveforsomeamountofthetraffic
- 5:53SPEAKER_00: today that's just growing day over day.
- 5:58SPEAKER_00: for these use cases.
- 5:58SPEAKER_01: That's a good point because like,
- 6:00SPEAKER_01: I think in terms of like our position,
- 6:02SPEAKER_01: well,
- 6:02SPEAKER_01: position to do it is,
- 6:03SPEAKER_01: you know,
- 6:04SPEAKER_01: being live,
- 6:06SPEAKER_01: you know,
- 6:07SPEAKER_01: maybe we can talk about that a little bit,
- 6:08but like that creates a feedback loop that we have that a lot of people don't
- 6:12have
- 6:12SPEAKER_01: because we have,
- 6:13SPEAKER_01: you know,
- 6:15SPEAKER_01: many,
- 6:15SPEAKER_01: many millions of people using our search engine and now Duck AI.
- 6:18And so we're getting constant feedback about the relevancy of the search
- 6:23results
- 6:23SPEAKER_01: that we're serving,
- 6:24SPEAKER_01: not to mention the fact that we have
- 6:26SPEAKER_01: almost 20 years of evaluating relevancy ourself.
- 6:29SPEAKER_00: Exactly.
- 6:31SPEAKER_00: Yeah, exactly.
- 6:31SPEAKER_00: And so like humans are unsurprisingly and appropriately more,
- 6:39SPEAKER_00: more critical of results than agents are.
- 6:42SPEAKER_00: So it kind of creates this higher fidelity feedback loop because,
- 6:48SPEAKER_00: you know,
- 6:48SPEAKER_00: through our,
- 6:48SPEAKER_00: through anonymization and whatever else,
- 6:50SPEAKER_00: like we can privately understand what is,
- 6:54SPEAKER_00: most relevant on the internet for customers and users.
- 6:57And that really helps us to,
- 6:59and positions us to be pretty competitive in the space quickly.
- 7:03SPEAKER_00: So like, I think that's kind of interesting and it's exciting.
- 7:07And like the true DuckDuckGo way,
- 7:09as you and I know well,
- 7:10like we like to ship stuff.
- 7:12SPEAKER_00: And so it's been really cool to,
- 7:14SPEAKER_00: yeah,
- 7:16SPEAKER_00: like it's just been really cool to be using it already and in production,
- 7:20SPEAKER_00: our own index.
- 7:21SPEAKER_00: And it creates that flywheel and we could,
- 7:24you know,
- 7:24use buzzwords like reinforcement learning and this,
- 7:26that,
- 7:27and the other thing.
- 7:27SPEAKER_00: But at the end of the day,
- 7:28it's just really the relationship of consuming your own internal API
- 7:32product.
- 7:33That's the flywheel and allows you to like establish relative priority
- 7:37really
- 7:38SPEAKER_00: quickly and be like,
- 7:39SPEAKER_00: oh,
- 7:39SPEAKER_00: I ran this experiment.
- 7:40Like we really think this query set is going to be well suited to our own index.
- 7:46And then we tried and we're like,
- 7:47oh,
- 7:47we're not working that well on that.
- 7:48SPEAKER_00: Let's move to this other one.
- 7:49And then it just,
- 7:51it just changes the game for how quickly you can iterate.
- 7:55SPEAKER_00: which has been really exciting for,
- 7:57SPEAKER_00: and I know the team's really excited about it too,
- 7:59SPEAKER_00: because engineers like to ship things.
- 8:02SPEAKER_00: Perfect.
- 8:02SPEAKER_01: I think that's a good intro.
- 8:05SPEAKER_01: But let's do,
- 8:07SPEAKER_01: to your point about buzzwords,
- 8:09SPEAKER_01: let's do a few more buzzwords in terms of like,
- 8:11SPEAKER_01: just give us the broad tech flow,
- 8:15SPEAKER_01: like,
- 8:16SPEAKER_01: you know,
- 8:16SPEAKER_01: without getting too deep into anything,
- 8:18SPEAKER_01: but just to give people a sense of kind of how it works.
- 8:21SPEAKER_01: And then maybe we could wrap up after that.
- 8:23Yeah,
- 8:24so kind of the way that I think about this is like a little pipe or a train or
- 8:29whatever.
- 8:30You have your frontier that kind of is the web that you're looking to crawl
- 8:38because
- 8:39SPEAKER_00: you have to pick what your frontier is.
- 8:42SPEAKER_00: Then you crawl that.
- 8:44SPEAKER_00: And all of these components are extremely complicated by themselves.
- 8:51SPEAKER_00: To crawl, it means like,
- 8:53SPEAKER_00: you need to crawl politely.
- 8:55SPEAKER_00: Some sites want you to crawl, some sites don't want you to crawl.
- 8:59And so to be a good,
- 9:01trustworthy netizen,
- 9:03you have to respect those things.
- 9:06SPEAKER_00: And that's an important part of crawling.
- 9:08It's also important to have the bandwidth and the throughput to crawl at the
- 9:12scale
- 9:12SPEAKER_00: that you need to crawl.
- 9:15SPEAKER_00: And so fortunately for us, we've had a lot of experience with that.
- 9:18SPEAKER_00: So we have that.
- 9:21SPEAKER_00: On the rendering side,
- 9:24SPEAKER_00: like you have to,
- 9:25SPEAKER_00: when you fetch content,
- 9:26SPEAKER_00: you have to render,
- 9:28SPEAKER_00: including JavaScript and everything else like that.
- 9:30SPEAKER_00: The only way to get the content is to literally run the whole webpage.
- 9:34SPEAKER_00: Otherwise, like you get no content.
- 9:38SPEAKER_00: So that's quite an expensive process.
- 9:40So we kind of do a naive approach and then a more complicated rendered
- 9:44approach.
- 9:46SPEAKER_00: Then you have content extraction, which is like the next step.
- 9:49SPEAKER_00: where you think about your title,
- 9:51SPEAKER_00: your description,
- 9:52SPEAKER_00: your headings,
- 9:54SPEAKER_00: metadata,
- 9:56SPEAKER_00: main body stuff,
- 9:58SPEAKER_00: where you extract the content,
- 9:59SPEAKER_00: what the page actually means.
- 10:02SPEAKER_00: And then we're very fortunate in today's day and age to have semantics.
- 10:08SPEAKER_00: And so semantic search is a big part of the pipeline.
- 10:12SPEAKER_00: And what that means is
- 10:14SPEAKER_00: you know,
- 10:14SPEAKER_00: what people are calling embeddings and,
- 10:16SPEAKER_00: and you calculate embeddings on,
- 10:19SPEAKER_00: on extracted content.
- 10:21SPEAKER_00: Um, and then we use a database, which I quite like called Vespa.
- 10:26SPEAKER_00: Um, and it's all ingested into Vespa.
- 10:29SPEAKER_00: Like in my opinion,
- 10:31SPEAKER_00: uh,
- 10:32SPEAKER_00: kind of your indexing,
- 10:33SPEAKER_00: your ingestion,
- 10:34SPEAKER_00: your features,
- 10:35SPEAKER_00: how you calculate those things and how they describe,
- 10:40SPEAKER_00: um,
- 10:42SPEAKER_00: you know,
- 10:43that's a big description of your product because it limits what you can do in
- 10:48the
- 10:49SPEAKER_00: ranking phases,
- 10:50SPEAKER_00: which we could get into.
- 10:50I don't know if you wanna get into the ranking funnel as well or just the data
- 10:54pipeline.
- 10:54SPEAKER_01: I think that's a pretty good overview.
- 10:55SPEAKER_01: I mean, I think,
- 10:58SPEAKER_01: I think we'll do more of these in the future.
- 11:00SPEAKER_01: But like, if people have questions, they can write me or write us.
- 11:04We need to get an email going where people,
- 11:07we can have proper feedback for these.
- 11:08SPEAKER_00: A mini series, as it were.
- 11:10SPEAKER_00: Yeah,
- 11:10SPEAKER_00: because I'd like to bring in,
- 11:12SPEAKER_00: you know,
- 11:13SPEAKER_00: I can give you the overview,
- 11:14SPEAKER_00: but there's a lot of experts on the team.
- 11:16SPEAKER_00: Yeah,
- 11:16soIthinkweplaningeneraltocomebackandwecantalkdeeperaboutspecific
- 11:22SPEAKER_01: things in the future.
- 11:23SPEAKER_01: But I think that was the right overview.
- 11:25SPEAKER_01: Okay.
- 11:25SPEAKER_01: I think we hit...
- 11:27SPEAKER_01: hit it.
- 11:27SPEAKER_01: Anything, anything that we missed that you want to cover or we're good?
- 11:33No,
- 11:34I mean,
- 11:34I think,
- 11:36I think that the space is quite interesting and it's been really fun.
- 11:40SPEAKER_00: Like when we started out,
- 11:41SPEAKER_00: you know,
- 11:41SPEAKER_00: I,
- 11:42Istartedoutwithyou15and15yearsagoandlikeweweredoingsomething
- 11:46SPEAKER_00: different back then.
- 11:47SPEAKER_00: The concept of a customer being an AI agent was not really a thing.
- 11:52SPEAKER_00: And I think AI
- 11:56agents as search customers is quite interesting because they generate the
- 12:03queries.
- 12:03SPEAKER_00: AI agents are generating the queries.
- 12:06SPEAKER_00: The queries are formulated quite well.
- 12:10SPEAKER_00: They don't feature misspellings.
- 12:12AI agents can consume large amounts of information quickly,
- 12:16like lots and lots of information.
- 12:19SPEAKER_00: They can speak in embeddings.
- 12:21SPEAKER_00: So I just think it's a really exciting time.
- 12:25And I think we made the right call to kind of forego it until more recently
- 12:31because
- 12:31SPEAKER_00: now we're quite well equipped to support ourselves.
- 12:36And we have the products out that really express the needs and the
- 12:41requirementsof
- 12:42SPEAKER_00: an API like this.
- 12:43SPEAKER_00: So I know I'm very excited for that and the team's very excited for that.
- 12:47SPEAKER_00: So yeah,
- 12:48we should do more of these and get deeper into what a polite crawler really
- 12:52means
- 12:52SPEAKER_00: as opposed to just glossing over it.
- 12:55SPEAKER_01: Uh, cause it's cool.
- 12:56SPEAKER_01: Well, thank you again for joining.
- 12:59SPEAKER_01: Of course.
- 13:00SPEAKER_01: Uh, we'll do more again later.
- 13:01SPEAKER_01: If anyone's got questions, email them in to us.
- 13:03SPEAKER_01: Otherwise see you till next time.
- 13:06SPEAKER_01: Bye everybody.
- 13:06SPEAKER_01: Later.
- 13:07SPEAKER_01: Thanks.