Latest / Inside DuckDuckGo / Duck Tales: How we use AI at DuckDuckGo while staying true to our principles (Ep.45)
Transcript
- 0:00SPEAKER_00: Hi, welcome to DuckTales,
- 0:01SPEAKER_00: where we go behind the scenes at DuckDuckGo and discuss the stories,
- 0:04SPEAKER_00: technology, and people that help build privacy tools for everyone.
- 0:09SPEAKER_00: In each episode,
- 0:09SPEAKER_00: you'll hear from employees about our vision,
- 0:11SPEAKER_00: product updates, engineering, or approach to AI.
- 0:15SPEAKER_00: My name is Mark, and I'm on the engineering team.
- 0:19SPEAKER_00: I have with me Zach.
- 0:20SPEAKER_00: Zach, you want to introduce yourself?
- 0:22SPEAKER_01: Hi, yeah, I'm Zach Dean Merrim on our data science team here at DuckDuckGo.
- 0:28SPEAKER_01: I am still a duckling,
- 0:30SPEAKER_01: so I've been here for about 10 months,
- 0:33SPEAKER_01: and I work on building our internal AI systems.
- 0:37SPEAKER_00: Great.
- 0:39SPEAKER_00: Yeah, so today we're going to talk even more about AI, which is Zach's specialty.
- 0:47SPEAKER_00: Today we're going to talk specifically about how we use,
- 0:51SPEAKER_00: sorry, how we use AI internally at DuckDuckGo.
- 0:56SPEAKER_00: So, yeah,
- 0:58SPEAKER_00: I guess to start out,
- 1:00SPEAKER_00: As a company,
- 1:02SPEAKER_00: we've adopted AI pretty heavily internally as a tool to help us move faster.
- 1:08SPEAKER_00: Can you talk a little bit, Zach,
- 1:10SPEAKER_00: about the approach that we've taken here,
- 1:13SPEAKER_00: given that I think you were in on the ground floor with this, right?
- 1:16SPEAKER_01: Yeah.
- 1:20SPEAKER_01: DuckDuckGo as a company, Takes,
- 1:23SPEAKER_01: I think,
- 1:23SPEAKER_01: a somewhat unique approach to AI as we take unique approach to everything we do.
- 1:29SPEAKER_01: We very much have a culture of things like first principles thinking and our own spin on things.
- 1:38SPEAKER_01: And a very important part of what we do are our privacy and security.
- 1:45SPEAKER_01: So before we rolled out many of these tools,
- 1:47SPEAKER_01: we spent quite a bit of time internally Getting comfortable with their failure modes
- 1:53SPEAKER_01: and the blast radius potential for those failures.
- 2:00SPEAKER_01: One of the first places we started using AI a lot was with coding tools.
- 2:05SPEAKER_01: So right off the bat with Cursor and then CloudCode.
- 2:09SPEAKER_01: And one, the models are very, very good at writing code.
- 2:14SPEAKER_01: But then two,
- 2:16SPEAKER_01: if you are effectively using tools like Git,
- 2:19SPEAKER_01: And GitHub,
- 2:22SPEAKER_01: an agent that's limited to writing code can only do so much damage
- 2:25SPEAKER_01: if your code base is backed up in GitHub and you don't let,
- 2:28SPEAKER_01: for example,
- 2:29SPEAKER_01: people force push domain or whatever like that,
- 2:31SPEAKER_01: like if your GitHub is properly configured.
- 2:33SPEAKER_01: And so we had a lot of questions like that early on.
- 2:36SPEAKER_01: And then,
- 2:37SPEAKER_01: you know,
- 2:38SPEAKER_01: once Once we were confident,
- 2:40SPEAKER_01: we really were able to kind of open the floodgates
- 2:42SPEAKER_01: and let people really start using both cursor,
- 2:45SPEAKER_01: cloud code,
- 2:46SPEAKER_01: and now to a lesser degree codex to write code and automate as much of engineering
- 2:51SPEAKER_01: as they can.
- 2:53SPEAKER_00: Yeah, that's great.
- 2:55SPEAKER_00: You mentioned privacy, right?
- 2:58SPEAKER_00: Obviously, it's a core tenant of our products.
- 3:00SPEAKER_00: It's also a core tenant of how we operate here.
- 3:04SPEAKER_00: And yeah, to your point, we do tend to do things a little bit differently.
- 3:09SPEAKER_00: So I'm curious how the privacy kind of shaped our internal workflows with respect
- 3:15SPEAKER_00: to AI,
- 3:16SPEAKER_00: given that even though we don't collect user data,
- 3:22SPEAKER_00: we do still have things that we obviously don't want to share more broadly.
- 3:26SPEAKER_01: Yeah, absolutely.
- 3:29SPEAKER_01: So our privacy engineers very much think of like an information hierarchy of stuff that is...
- 3:39SPEAKER_01: I don't know what the word is I want to use for the scale, but there's a high to low scale.
- 3:45SPEAKER_01: And so there's certain information that we don't even save to desk in our production servers.
- 3:50SPEAKER_01: There's certain information that is stored in memory and is never retained anywhere
- 3:56SPEAKER_01: in our systems.
- 3:57SPEAKER_01: Right.
- 3:59SPEAKER_01: That is fundamentally the safest form of data.
- 4:02SPEAKER_01: If you don't store a piece of information,
- 4:05SPEAKER_01: it's very hard to accidentally store it in the wrong place and in a way
- 4:09SPEAKER_01: that you don't want to do.
- 4:12SPEAKER_01: And that kind of principle thinking applies to AI too.
- 4:15SPEAKER_01: Like if an AI system doesn't have access to certain data,
- 4:19SPEAKER_01: it can't do the wrong thing with that data.
- 4:22SPEAKER_01: And one of those ways to prevent access is just by fundamentally not storing
- 4:26SPEAKER_01: the data in the first place.
- 4:28SPEAKER_00: As you go through that hierarchy,
- 4:29SPEAKER_00: you know,
- 4:30SPEAKER_00: there's information that
- 4:31SPEAKER_01: we're more and more comfortable with working on internally.
- 4:34SPEAKER_01: Like, for example,
- 4:37SPEAKER_01: you know,
- 4:37SPEAKER_01: time series traffic of,
- 4:39SPEAKER_01: you know,
- 4:39SPEAKER_01: how many users are on our various websites,
- 4:44SPEAKER_01: like that sort of aggregated data that can't be tied to an individual person.
- 4:50SPEAKER_01: That's a lot safer to start saying, hey, I'm going to, you know, I'm on the data science team.
- 4:53SPEAKER_01: I'm going to do some data science work and look at.
- 4:56SPEAKER_01: Whatever it is that the set of questions I have.
- 5:00SPEAKER_01: And then I think the kind of the bottom piece that is incredibly important is we pay
- 5:05SPEAKER_01: a lot of attention to our data processing agreements that we have with all
- 5:10SPEAKER_01: of our vendors, but especially our AI vendors.
- 5:14SPEAKER_01: And the contractual agreements you have with people about what they are
- 5:18SPEAKER_01: and are allowed to do with your data are important.
- 5:21SPEAKER_01: Also very important.
- 5:24SPEAKER_01: So, yeah,
- 5:25SPEAKER_01: there's a spectrum of things and you go talk to any privacy engineer
- 5:28SPEAKER_01: and you can very quickly learn just how deep the problem goes.
- 5:33SPEAKER_00: Yeah, it's interesting you mentioned principled thinking, right?
- 5:37SPEAKER_00: I think that's something that is a core part of the culture here.
- 5:42SPEAKER_00: I think it's fair to say that we have,
- 5:45SPEAKER_00: you know,
- 5:46SPEAKER_00: spent a lot of time talking about how to adapt AI to our principles as opposed
- 5:51SPEAKER_00: to adapting our principles to AI.
- 5:52SPEAKER_00: Yeah.
- 5:53SPEAKER_00: Yeah.
- 5:53SPEAKER_00: Which I feel like you see a lot in the industry right now.
- 5:56SPEAKER_01: Yeah.
- 5:57SPEAKER_01: Yeah.
- 5:58SPEAKER_01: And that's, I mean, that's a great point.
- 6:01SPEAKER_01: Like,
- 6:05SPEAKER_01: Two of our principles that matter a lot to me personally are questioning assumptions
- 6:10SPEAKER_01: and validating direction.
- 6:11SPEAKER_00: Absolutely.
- 6:12SPEAKER_01: And, you know,
- 6:13SPEAKER_01: I'm like the AI,
- 6:15SPEAKER_01: you know,
- 6:16SPEAKER_01: waving, you know, the flag and the pom-poms and saying, let's go AI internally.
- 6:21SPEAKER_01: Right.
- 6:22SPEAKER_01: Well, like,
- 6:23SPEAKER_01: I find I have to, like,
- 6:25SPEAKER_01: you know,
- 6:25SPEAKER_01: remind people just because we're using AI,
- 6:29SPEAKER_01: I just mean we throw our principles out the window.
- 6:32SPEAKER_01: Absolutely.
- 6:35SPEAKER_01: Again, as a leader here, I am more interested in...
- 6:41SPEAKER_01: I want to validate direction too.
- 6:42SPEAKER_01: And so I am very interested to hear from people places where AI isn't working.
- 6:47SPEAKER_01: I want to see the PR that's a 7,000 line of code wall of text where a team's going,
- 6:53SPEAKER_01: whoa, we can't review this, let alone ship it.
- 6:57SPEAKER_01: And those kinds of failure modes, AI is rife with them.
- 7:01SPEAKER_01: And we don't just look at an AI's output and say, well, an AI made it, so let's ship it.
- 7:05SPEAKER_01: Yeah.
- 7:06SPEAKER_01: You should use AI as a lever
- 7:13SPEAKER_01: with your principles,
- 7:14SPEAKER_01: not as a reason to throw your principles out the window.
- 7:17SPEAKER_01: But again, just questioning assumptions and validating direction.
- 7:22SPEAKER_01: AIs are notoriously bad at questioning assumptions.
- 7:24SPEAKER_01: This will happen all the time, even with the latest models, where you'll be like, hang on a sec.
- 7:29SPEAKER_01: You said X, and then you said Y, which implies not X.
- 7:34SPEAKER_01: But you never connected the pieces together in the chain of logic.
- 7:38SPEAKER_01: And so I think your principles matter more when you're using AI.
- 7:43SPEAKER_00: Oh, that's interesting.
- 7:44SPEAKER_00: Yeah.
- 7:45SPEAKER_00: Yeah.
- 7:45SPEAKER_00: I think that's really well put.
- 7:47SPEAKER_00: Actually leads us into our next question to a degree.
- 7:52SPEAKER_00: There's...
- 7:54SPEAKER_00: There's a lot of hype right now, obviously, with AI.
- 7:57SPEAKER_00: I mean, to put it mildly, right?
- 8:00SPEAKER_00: The hype cycle has been super interesting.
- 8:04SPEAKER_00: There's a lot of concern about AI in the workforce today, right?
- 8:08SPEAKER_00: And we're certainly not immune to that, although we don't really see it that way as a company.
- 8:14SPEAKER_00: So maybe can you talk a little bit about how we're getting so many people
- 8:19SPEAKER_00: to use it effectively and where we're trying to get to?
- 8:23SPEAKER_01: Yeah, so AI is a very powerful tool.
- 8:33SPEAKER_01: And it's also in some ways a slippery tool.
- 8:36SPEAKER_01: Like it's changing so fast.
- 8:39SPEAKER_01: There's not like a long history of like,
- 8:44SPEAKER_01: I don't know, I can't think of an equivalent technology, but there's not like,
- 8:53SPEAKER_01: Yeah.
- 8:53SPEAKER_01: Yeah.
- 8:54SPEAKER_01: Yeah.
- 9:12SPEAKER_01: And it's a great joke,
- 9:13SPEAKER_01: but it's like, you know, one, it interprets everything you said too literally.
- 9:16SPEAKER_01: And then two, it forgets tomorrow everything that you said to it today.
- 9:20SPEAKER_01: And like those two problems, like, you know, it's...
- 9:26SPEAKER_01: I've been playing around with Fable a lot recently.
- 9:29SPEAKER_01: It's an amazing model for one-shotting really difficult programming problems.
- 9:33SPEAKER_01: But you still get that sometimes over-literality,
- 9:36SPEAKER_01: like what you say wasn't quite interpreted the way you meant it,
- 9:41SPEAKER_01: and you have to be careful of that.
- 9:42SPEAKER_01: And then the other side of like,
- 9:44SPEAKER_01: you come in tomorrow and you have to curate context that you take over
- 9:50SPEAKER_01: between sessions.
- 9:50SPEAKER_01: And I think honestly that latter part has been the part for me that Has been
- 9:55SPEAKER_01: the most difficult to adapt to is just what do you remember and how do you remember
- 10:01SPEAKER_01: and how do you persist that memory?
- 10:03SPEAKER_01: And that's very much something you have to build for yourself.
- 10:07SPEAKER_01: And from the individual level of the team level,
- 10:11SPEAKER_01: how are we storing information in when an AI goes wrong,
- 10:17SPEAKER_01: where do we put that learning so that we don't forget about it again in the future?
- 10:22SPEAKER_00: Right, right.
- 10:37SPEAKER_00: Right.
- 10:37SPEAKER_00: Right.
- 10:38SPEAKER_00: Right.
- 10:38SPEAKER_00: Right.
- 10:58SPEAKER_00: Like that didn't obviate the need for strong engineers, right?
- 11:02SPEAKER_00: It just changed the nature of what they're doing on a daily basis.
- 11:05SPEAKER_00: And I think at most that's what we're looking at here potentially,
- 11:09SPEAKER_00: but obviously it's a hot topic.
- 11:11SPEAKER_01: Yeah, no.
- 11:12SPEAKER_01: So I was literally just listening in a podcast.
- 11:16SPEAKER_01: I'm on a podcast.
- 11:17SPEAKER_01: I was listening to a podcast.
- 11:17SPEAKER_01: I love it.
- 11:19SPEAKER_01: This morning with Fiona Fung, who's the manager of the Cloud Code team.
- 11:24SPEAKER_01: And she had just like this will-in-one-off anecdote.
- 11:25SPEAKER_01: She was like, oh,
- 11:27SPEAKER_01: my old boss,
- 11:27SPEAKER_01: like my first engineering manager I ever had, you know, called me up the other day.
- 11:31SPEAKER_01: And she said, this guy started his career programming in punch cards.
- 11:35SPEAKER_00: Yeah, right.
- 11:36SPEAKER_01: Programming in punch cards.
- 11:37SPEAKER_01: And the guy called her up and was like, hey, I'm using Cloud Code.
- 11:40SPEAKER_01: And you run the Cloud Code products.
- 11:41SPEAKER_01: You just want to call you and tell you how cool Cloud Code is.
- 11:44SPEAKER_01: And I thought that was like a really great piece of perspective.
- 11:46SPEAKER_01: Like, just like imagining the transition, right?
- 11:51SPEAKER_01: Between programming and punch cards to programming and Python.
- 11:54SPEAKER_01: And I didn't experience that because I wasn't around.
- 12:00SPEAKER_01: But you forget how long the history is of computing and how much it already has changed.
- 12:06SPEAKER_00: Absolutely, absolutely.
- 12:08SPEAKER_00: To think about people, I mean, I think most of us have at least tried some assembly, right?
- 12:13SPEAKER_00: Like, I don't know if they still teach that in university today,
- 12:16SPEAKER_00: but they certainly did, even if it was just to expose you to it.
- 12:19SPEAKER_00: But yeah,
- 12:20SPEAKER_00: that's not something that,
- 12:21SPEAKER_00: you know, the average person or the average engineer is going to be doing.
- 12:24SPEAKER_00: But anyway, so yeah, I guess we can start wrapping up.
- 12:30SPEAKER_00: I think we've talked a lot about engineering examples.
- 12:35SPEAKER_00: I'm wondering if
- 12:38SPEAKER_00: You know, we have some non-engineering examples of how we're using this at the company.
- 12:42SPEAKER_01: Yeah.
- 12:43SPEAKER_01: One of my favorites is spreadsheets.
- 12:46SPEAKER_01: I've spent a lot of time early in my career building models in Excel,
- 12:50SPEAKER_01: and I definitely developed like I've been frustrated a lot recently with trying
- 12:55SPEAKER_01: to build a model in Excel and
- 13:14SPEAKER_01: I can't find the commands I want or necessarily even remember them.
- 13:19SPEAKER_01: And what's really interesting to me is a lot of Excel models,
- 13:23SPEAKER_01: you can give a dataset to Claude,
- 13:24SPEAKER_01: you can describe The problem you want to solve,
- 13:27SPEAKER_01: you can describe the shape of the model that you want.
- 13:30SPEAKER_01: You know,
- 13:30SPEAKER_01: I want one tab with my input parameters,
- 13:33SPEAKER_01: one tab with my data and one tab with the calculations and output or something like
- 13:37SPEAKER_01: that.
- 13:38SPEAKER_01: And it'll do a remarkably good job of building things like that.
- 13:41SPEAKER_01: And there is, you know, there's a lot of work that still happens in spreadsheets.
- 13:45SPEAKER_01: And it's really nice to be able to go to Cloud Code and give it a spreadsheet
- 13:48SPEAKER_01: and say, hey, Can you change the formatting?
- 13:51SPEAKER_01: I've got all these cells where the important information is bolded,
- 13:55SPEAKER_01: and I want to remove the bolding and replace the bolding with a red background.
- 14:00SPEAKER_01: And that's really tedious to do with pointing
- 14:02SPEAKER_00: and clicking.
- 14:03SPEAKER_01: In Cloud Code,
- 14:04SPEAKER_01: it's just reading the XML,
- 14:05SPEAKER_01: and so it's very good at a spreadsheet as an XML file,
- 14:10SPEAKER_01: and it can just say, cool, yeah,
- 14:11SPEAKER_01: I'm just going to change around the XML structure,
- 14:14SPEAKER_01: and now instead of bolding, you've got red highlighting.
- 14:17SPEAKER_00: Got it.
- 14:17SPEAKER_00: Yeah, yeah.
- 14:19SPEAKER_00: No, it's interesting.
- 14:20SPEAKER_00: I feel like as much as I've embraced it in my daily workflow,
- 14:25SPEAKER_00: I still have not yet gotten the nerve up to trust it with something like
- 14:29SPEAKER_00: a spreadsheet.
- 14:30SPEAKER_00: But I know that a lot of people do, so I probably need to get over that.
- 14:34SPEAKER_01: I mean, if your spreadsheets are in version control,
- 14:37SPEAKER_01: as mine all are,
- 14:38SPEAKER_01: because I'm a huge nerd,
- 14:41SPEAKER_01: there's limited room for it to really, really screw something up.
- 14:45SPEAKER_00: Sure.
- 14:46SPEAKER_00: All right, great.
- 14:48SPEAKER_00: Yeah, please go ahead.
- 14:49SPEAKER_01: One more is just like summer camp forms.
- 14:53SPEAKER_01: Like I've got these forms.
- 14:54SPEAKER_01: Like I actually know it was a school form.
- 14:55SPEAKER_01: I got a form from my kid's school and it was like,
- 14:58SPEAKER_01: it was like a scan, but it was like a little bit sideways.
- 15:02SPEAKER_01: So all the fields are a little bit off.
- 15:04SPEAKER_01: And so, you know,
- 15:05SPEAKER_01: like filling out a PDF form is annoying enough when there's like little boxes
- 15:09SPEAKER_01: that you can type in,
- 15:11SPEAKER_01: but it's like, okay, I don't want to print this thing out, handwrite everything.
- 15:14SPEAKER_01: My handwriting's terrible.
- 15:15SPEAKER_01: Scan it, send it back.
- 15:17SPEAKER_01: And I was like, I said to ClaudeCowork, here's the form.
- 15:22SPEAKER_01: I dictated to it what I wanted in each field.
- 15:25SPEAKER_01: And I said, can you just iterate?
- 15:27SPEAKER_01: Actually, it was ClaudeCowork, not Cowork.
- 15:28SPEAKER_01: Can you iterate on,
- 15:31SPEAKER_01: you know,
- 15:31SPEAKER_01: take my text,
- 15:32SPEAKER_01: turn it into a text box,
- 15:33SPEAKER_01: put it into PDF,
- 15:35SPEAKER_01: and then, you know, line them all up correctly with the fields, which was not easy.
- 15:39SPEAKER_01: And it built itself a little loop where it was like,
- 15:42SPEAKER_01: you know, it got like, My information to bullet points is pretty easy.
- 15:45SPEAKER_01: Map them to the fields is pretty easy.
- 15:47SPEAKER_01: And then it just did a narration of, okay,
- 15:50SPEAKER_01: write them out to the spreadsheet and then just like adjust the X and Y coordinates
- 15:54SPEAKER_01: of the box until it lined up well enough that I could email it back to the school.
- 15:58SPEAKER_00: Wow.
- 15:59SPEAKER_00: That's awesome.
- 15:59SPEAKER_00: Yeah.
- 16:00SPEAKER_00: I have not, I wouldn't even have thought to do that with AI.
- 16:03SPEAKER_00: So I feel like I need to be a little more ambitious.
- 16:05SPEAKER_00: Yeah.
- 16:06SPEAKER_01: It was the same where like I spent five minutes talking to Claude, gave it the file.
- 16:11SPEAKER_01: Sent it off in the background and it came back and it was done and I sent,
- 16:15SPEAKER_01: you know, I just, I checked the day.
- 16:16SPEAKER_01: I was like, yep, that is correct.
- 16:17SPEAKER_01: And, you know,
- 16:18SPEAKER_01: emailed it to the school and instead of,
- 16:20SPEAKER_01: you know, I mean,
- 16:21SPEAKER_01: who knows,
- 16:21SPEAKER_01: it would have been probably 20 minutes of printing it out,
- 16:24SPEAKER_01: filling it out,
- 16:25SPEAKER_01: scanning it, sending it back, put a bunch of steps and I didn't want to do them.
- 16:28SPEAKER_00: Yeah, sure, sure.
- 16:30SPEAKER_00: All right, great.
- 16:31SPEAKER_00: Yeah, I feel like we could talk about this all day.
- 16:33SPEAKER_00: Me too.
- 16:33SPEAKER_00: Yeah.
- 16:35SPEAKER_00: But we should probably wrap up.
- 16:36SPEAKER_00: So, yeah.
- 16:38SPEAKER_00: Thanks a lot for your time.
- 16:39SPEAKER_00: And thanks for everyone else for listening in.
- 16:43SPEAKER_01: Yeah.
- 16:44SPEAKER_01: Great talking to you as always, Mark.