Latest / The Pragmatic Engineer / Uber’s AI infra, from the inside

Transcript

Uber’s AI infra, from the inside · The Pragmatic Engineer

• 0:00
  SPEAKER_01: What kind of internal tools do software engineers at Uber use to build software today?
• 0:03
  SPEAKER_01: We get an exclusive,
• 0:04
  SPEAKER_01: insider,
• 0:05
  SPEAKER_01: and visuals-heavy look from one of Uber's longest-tenured engineers,
• 0:08
  SPEAKER_01: Udaykira Mendicetti.
• 0:10
  SPEAKER_01: Today, Uday leads much of the work on agentic AI at Uber.
• 0:13
  SPEAKER_01: In this conversation,
• 0:14
  SPEAKER_01: we get into internal developer systems Uber has built and used pre-AI like UOwn,
• 0:19
  SPEAKER_01: M3DB, Michelangelo, Cadence, Morpheus, and others.
• 0:23
  SPEAKER_01: AI Foundational Systems is a company built like its MCP Gateway, AI Guard, and AI FX.
• 0:28
  SPEAKER_01: Recent systems used by nearly all engineers like Minion,
• 0:30
  SPEAKER_01: CodeInbox, Slate, Shepard, Wisdom, and others.
• 0:33
  SPEAKER_01: If you want to understand what AI native development looks inside of
• 0:36
  SPEAKER_01: a modern tech company with thousands of engineers like Uber,
• 0:39
  SPEAKER_01: and why it's much easier to retrofit AI on top of BattleTest's developer Infra,
• 0:43
  SPEAKER_01: this episode is for you.
• 0:45
  SPEAKER_01: This episode is presented by TurboPuffer, Vector, and Full-Text Search built on object storage.
• 0:49
  SPEAKER_01: It's fast, cheap, and extremely scalable.
• 0:52
  SPEAKER_01: Today's episode is about the close to two dozen internal systems that Uber has built
• 0:55
  SPEAKER_01: to help engineers ship with AI agents.
• 0:58
  SPEAKER_01: One of the takeaways will be that production-ready code is far more complicated
• 1:02
  SPEAKER_01: than just using an AI coding harness.
• 1:04
  SPEAKER_01: The harness is the easy part.
• 1:06
  SPEAKER_01: What's hard is all the things around them.
• 1:08
  SPEAKER_01: This is where I need to mention our season sponsor, WorkOS.
• 1:12
  SPEAKER_01: One piece that is always a hard problem with agents is permissions.
• 1:15
  SPEAKER_01: Do you want an agent to have standing access to everything it might touch?
• 1:19
  SPEAKER_01: Or do you just approve every single prompt until you just stop reading the prompts?
• 1:23
  SPEAKER_01: Worker was built airlock for this.
• 1:26
  SPEAKER_01: Airlock is an authorization layer that sits between your agents and
• 1:29
  SPEAKER_01: the tools they call,
• 1:30
  SPEAKER_01: and it judges every single request against the agent's intent and your rules.
• 1:35
  SPEAKER_01: You write the rules in plain English,
• 1:36
  SPEAKER_01: agents never see your credentials, and every verdict is logged.
• 1:40
  SPEAKER_01: It works with coding agents like Cloud Code and Codex.
• 1:42
  SPEAKER_01: Workhorse Airlock is how agents do real work without over-permissioning.
• 1:46
  SPEAKER_01: Check it out at workhorse.com slash airlock.
• 1:49
  SPEAKER_01: Uday, welcome to the podcast.
• 1:51
  SPEAKER_00: Thank you so much for having me.
• 1:53
  SPEAKER_00: I've been following your work.
• 1:55
  SPEAKER_00: It's amazing to see the kind of impact you've had and all of
• 1:59
  SPEAKER_00: the great content you publish for all of the engineers.
• 2:02
  SPEAKER_00: So I'm super honored to be here and talk to you today about what you're doing.
• 2:07
  SPEAKER_01: Well, we worked together well before The Pragmatic Engineer was born back at Uber.
• 2:12
  SPEAKER_01: Not together together, but I knew of your work as well.
• 2:14
  SPEAKER_01: When did you join Uber exactly?
• 2:16
  SPEAKER_00: Yeah, so I joined Uber in May 2015.
• 2:20
  SPEAKER_00: In fact, like the two engineers who referred me, they are still here.
• 2:24
  SPEAKER_00: So I joined in our real-time API team.
• 2:28
  SPEAKER_00: It was during the period where we were trying to break out the Break our monoliths
• 2:32
  SPEAKER_00: into microservices and I was working on our API Gateway for all of our mobile apps
• 2:37
  SPEAKER_00: and like I worked on changing our mobile protocol.
• 2:40
  SPEAKER_00: Since then I worked on many different business platforms from our Fulfillment,
• 2:44
  SPEAKER_00: Uber for Business,
• 2:45
  SPEAKER_00: on our earner onboarding,
• 2:46
  SPEAKER_00: marketplace pricing and the last one year has been all agentic AI both
• 2:50
  SPEAKER_00: for our users, customer support, end users and so on.
• 2:54
  SPEAKER_00: So yeah it's been like a roller coaster last one year on working on this.
• 2:57
  SPEAKER_01: Yeah.
• 2:58
  SPEAKER_01: And for those people who didn't work there with us,
• 3:01
  SPEAKER_01: can we recall a little bit what real-time API or RT APIs,
• 3:04
  SPEAKER_01: we used to call it inside,
• 3:06
  SPEAKER_01: what it was and why it was a big deal and why it had so much load?
• 3:09
  SPEAKER_00: So when I joined, We had just two services at Uber.
• 3:14
  SPEAKER_00: It was called Dispatch written in Node.js and API written in Python.
• 3:19
  SPEAKER_00: Dispatch used to handle all of the rider and driver traffic at that point in time.
• 3:23
  SPEAKER_00: And the protocol was pretty simple at that point.
• 3:26
  SPEAKER_00: It was like a four second ping protocol we used to call that would give you
• 3:30
  SPEAKER_00: the entire state of the world.
• 3:31
  SPEAKER_00: And when the trip is completed, we would hand off to API.
• 3:34
  SPEAKER_00: It would handle all of the post trip processing, payments, receipts and everything.
• 3:38
  SPEAKER_00: RT API was born at that point in time to abstract all of the internal services
• 3:43
  SPEAKER_00: and mobile and also changing the protocol from simple four-second protocol
• 3:48
  SPEAKER_00: to back like where we have both REST APIs and we also have server to client.
• 3:53
  SPEAKER_00: We used to call it push protocol.
• 3:55
  SPEAKER_00: So we would,
• 3:56
  SPEAKER_00: because so much of our app is very dynamic,
• 3:58
  SPEAKER_00: everything keeps changing every few seconds.
• 4:01
  SPEAKER_00: So we needed a way to efficiently update the state of the app across both riders
• 4:06
  SPEAKER_00: and drivers as and when we get newer Updates as in when things change
• 4:09
  SPEAKER_00: in physical world.
• 4:11
  SPEAKER_00: So it used to be at that point, like one of the highest QPS services.
• 4:15
  SPEAKER_00: And over the years, we kind of because all of our stack was just no JSON Python.
• 4:19
  SPEAKER_00: So that was the natural choice at that point.
• 4:21
  SPEAKER_00: But in the subsequent years,
• 4:22
  SPEAKER_00: we have converted that into Go and it's still one of our highest QPS services
• 4:26
  SPEAKER_00: at Uber.
• 4:27
  SPEAKER_01: And I think for people listening,
• 4:28
  SPEAKER_01: they're going to be surprised when they hear like, oh,
• 4:30
  SPEAKER_01: there was only two services in 2015 or around that time at Uber,
• 4:34
  SPEAKER_01: because of course, Uber was known for the microservices boom.
• 4:37
  SPEAKER_01: And Tuan was on this podcast.
• 4:39
  SPEAKER_01: He talked a little bit about this.
• 4:40
  SPEAKER_01: But would you mind just recalling how those microservices came to be?
• 4:45
  SPEAKER_01: And from your perspective, you know, like what it meant to suddenly have so many services?
• 4:50
  SPEAKER_00: Yeah, I mean, I think there's like so many blogs and like so many people talk about that.
• 4:54
  SPEAKER_00: So I think like when I joined, right,
• 4:56
  SPEAKER_00: like we were,
• 4:57
  SPEAKER_00: as I said,
• 4:58
  SPEAKER_00: like we were just two services and there and you also think going back
• 5:03
  SPEAKER_00: in those days, the kind of hyper growth.
• 5:05
  SPEAKER_00: We were constantly expanding to new countries,
• 5:09
  SPEAKER_00: we were constantly expanding our product portfolio across mobility
• 5:12
  SPEAKER_00: and later we started Uber Eats and that also was expanding.
• 5:15
  SPEAKER_00: We needed a mechanism for teams to really build all kinds
• 5:18
  SPEAKER_00: of functionality very quickly to match up to the scale at which
• 5:22
  SPEAKER_00: the business was growing.
• 5:23
  SPEAKER_00: So we have invested a lot in all kinds of service infrastructure making it easy
• 5:27
  SPEAKER_00: for teams to build services, metrics, tracing.
• 5:30
  SPEAKER_00: Monitoring, observability, deployment, the whole bunch of stuff around that.
• 5:34
  SPEAKER_00: And there's not much off-the-shelf software for these things at that point at
• 5:38
  SPEAKER_00: the scale and rate of change that we had.
• 5:41
  SPEAKER_00: And there's a lot of stuff that we had to build in those days internally.
• 5:45
  SPEAKER_00: That created a boom cycle of a lot of,
• 5:48
  SPEAKER_00: we went from like two services to order of thousands of services.
• 5:53
  SPEAKER_00: Then around 2018, 2019 was when we standardized on our languages.
• 5:58
  SPEAKER_00: We picked Go and Java as our primary languages.
• 6:01
  SPEAKER_00: We modernized a lot of our tech stack.
• 6:04
  SPEAKER_00: We spent a lot of time in thinking about the layers of our architecture.
• 6:08
  SPEAKER_00: How do we platformize?
• 6:09
  SPEAKER_00: And over the years,
• 6:10
  SPEAKER_00: in fact,
• 6:11
  SPEAKER_00: when I worked on one of our fulfillment platform,
• 6:13
  SPEAKER_00: We actually reduced some of the services to a few services.
• 6:17
  SPEAKER_00: Even now, we are kind of in the similar ballpark of number of services.
• 6:20
  SPEAKER_00: Even though our business is 10 times larger, we are doing many more products.
• 6:25
  SPEAKER_00: And this whole bunch of platformization and years of investment that has gone
• 6:31
  SPEAKER_00: into it,
• 6:31
  SPEAKER_00: like we are still running one single global tech stack across all
• 6:36
  SPEAKER_00: the countries we operate in.
• 6:38
  SPEAKER_00: There's many,
• 6:38
  SPEAKER_00: many platforms that Serve all of the mobility verticals and all of
• 6:42
  SPEAKER_00: the delivery verticals using one shared building block.
• 6:46
  SPEAKER_00: And I think that kind of allows us to build more mobility verticals,
• 6:50
  SPEAKER_00: more delivery verticals, because we have platformized a lot over the years.
• 6:54
  SPEAKER_01: Would you mind telling us when you joined,
• 6:56
  SPEAKER_01: how big was Uber's engineering team and how did it go over time?
• 6:59
  SPEAKER_01: Because I don't think people from the outside appreciate just how big that growth was.
• 7:04
  SPEAKER_00: So when I joined I was around the first 600 engineers and in fact the whole
• 7:12
  SPEAKER_00: of marketplace team at that point used to be like maybe 50,
• 7:16
  SPEAKER_00: 60 engineers and over the next few years As we've scaled our business,
• 7:21
  SPEAKER_00: as we scaled our traffic,
• 7:24
  SPEAKER_00: we grew to thousands of engineers and around 2019-2020 we started also expanding
• 7:30
  SPEAKER_00: across different sites.
• 7:31
  SPEAKER_00: Now we have 12 global tech sites across the world in almost Every continent we operate in.
• 7:37
  SPEAKER_00: In fact, that helps us build products for those hyper-local markets.
• 7:43
  SPEAKER_00: And some of those products are then rolled out globally across all the markets
• 7:46
  SPEAKER_00: where we could roll out that kind of functionality.
• 7:49
  SPEAKER_00: So I think that gives us a lot of ability to tap into talent globally
• 7:54
  SPEAKER_00: and also build nuanced local products.
• 7:57
  SPEAKER_00: And that also helps us build one global tech stack.
• 8:00
  SPEAKER_01: And speaking of building products, tech stacks, You still work on the platform team, right?
• 8:07
  SPEAKER_00: Yeah, so at least until my first 10 years, I worked on different business platform teams.
• 8:15
  SPEAKER_00: And in the last one year,
• 8:16
  SPEAKER_00: I'm more spending my time on a developer platform,
• 8:20
  SPEAKER_00: on an agentic AI for engineers, agentic AI for the rest of the company.
• 8:25
  SPEAKER_00: And in the process, I've also worked on some user-facing AI initiatives as well.
• 8:28
  SPEAKER_01: Can you take us back to 2015 on how...
• 8:31
  SPEAKER_01: Uber's internal tooling changed because again,
• 8:34
  SPEAKER_01: I was there and I was really surprised to see just how much custom tooling we had.
• 8:39
  SPEAKER_01: We talked about RT API, but there were things like Morpheus Edge Gateway and other things.
• 8:43
  SPEAKER_01: Would you mind talking through like back in 2015, 2016,
• 8:46
  SPEAKER_01: if an engineer joined Uber,
• 8:48
  SPEAKER_01: what kind of tools did they use and how did those evolve?
• 8:51
  SPEAKER_01: I'm talking obviously the main ones that most engineers back
• 8:56
  SPEAKER_01: under mobile engineers touched at Uber.
• 8:58
  SPEAKER_00: Yeah, absolutely.
• 9:00
  SPEAKER_00: Like in those days,
• 9:01
  SPEAKER_00: given the way in which we were scaling,
• 9:04
  SPEAKER_00: there's not many off-the-shelf products that we could tap into.
• 9:07
  SPEAKER_00: So I think we talked about RTPI and Edge Gateway.
• 9:11
  SPEAKER_00: Some of the other stuff that we had to build was a system called YouOwn
• 9:15
  SPEAKER_00: that will enable anyone in the company to identify who owns a specific asset.
• 9:20
  SPEAKER_01: Like it was like services, right?
• 9:22
  SPEAKER_01: And you could look at during on-call,
• 9:25
  SPEAKER_01: if there's an outage,
• 9:25
  SPEAKER_01: you could figure out which team owns it,
• 9:27
  SPEAKER_01: who's on-call for that kind of stuff, right?
• 9:29
  SPEAKER_00: Yeah, absolutely.
• 9:30
  SPEAKER_00: And as the engineering size grew,
• 9:31
  SPEAKER_00: we also had to build like end to end,
• 9:35
  SPEAKER_00: like our engineering design docs,
• 9:36
  SPEAKER_00: product design docs, getting all of those into one single place.
• 9:40
  SPEAKER_00: We had to work on getting all of our incident manager
• 9:43
  SPEAKER_00: into We are like really obsessed about how we handle our incidents and the learnings
• 9:49
  SPEAKER_00: from that.
• 9:49
  SPEAKER_00: So we build a lot of really good tooling around that.
• 9:52
  SPEAKER_00: I think there was some of the more complex ones was our metrics,
• 9:55
  SPEAKER_00: like even back in 2015, 2016,
• 9:57
  SPEAKER_00: we used to probably We have around like 500 million metrics emitted per second with
• 10:02
  SPEAKER_00: the cardinality and there's no off the shelf metric system that we could use and
• 10:07
  SPEAKER_00: that gave rise to M3DB which was open sourced later
• 10:12
  SPEAKER_00: and there's even companies formed around that.
• 10:15
  SPEAKER_00: And that enabled us to build a handle kind of metrics volume that we had.
• 10:22
  SPEAKER_00: We also had to build our own end-to-end machine learning platform because we used
• 10:27
  SPEAKER_00: to have hundreds of ML models.
• 10:29
  SPEAKER_00: And if you take the whole lifecycle from creating the features,
• 10:33
  SPEAKER_00: building the models, monitoring, serving.
• 10:35
  SPEAKER_01: This is Michelangelo, right?
• 10:37
  SPEAKER_00: Michelangelo and like we recently just few couple of months ago,
• 10:40
  SPEAKER_00: we open sourced our deep learning and part of the Michelangelo infrastructure.
• 10:45
  SPEAKER_00: So there's a lot of like really good solid foundational building blocks there.
• 10:49
  SPEAKER_00: The other interesting one is also Cadence,
• 10:51
  SPEAKER_00: which is If you think about some of the workflows, right,
• 10:55
  SPEAKER_00: like they span across multiple days,
• 10:57
  SPEAKER_00: whether it's a driver onboarding,
• 10:58
  SPEAKER_00: whether it could be some CRM campaign,
• 11:00
  SPEAKER_00: you could see a lot of engineers doing their own resiliency stuff,
• 11:04
  SPEAKER_00: their own state management,
• 11:06
  SPEAKER_00: and that gave rise to Cadence,
• 11:08
  SPEAKER_00: which is a durable workflow engine, and later even that got open sourced.
• 11:12
  SPEAKER_01: I think not many people will know Cadence, but a lot of people will know Temporal, right?
• 11:17
  SPEAKER_01: Which was, as I understand, either inspired by Cadence or a lot of the ideas came from there.
• 11:21
  SPEAKER_01: I'm not even sure it was the open source version of Cadence,
• 11:23
  SPEAKER_01: but it had a lot of,
• 11:25
  SPEAKER_01: like it's kind of the external version of whatever Cadence is for Uber or used
• 11:28
  SPEAKER_01: to be.
• 11:29
  SPEAKER_00: Yeah, and I think those folks like this,
• 11:30
  SPEAKER_00: there's a real need for something like this,
• 11:33
  SPEAKER_00: and they were able to go and build, scale it even further beyond Uber.
• 11:38
  SPEAKER_00: But I think that the pattern that I'm trying to say,
• 11:42
  SPEAKER_00: there's a whole bunch of things that We had to like invent in those days for
• 11:47
  SPEAKER_00: the scale that we had and over the years then we are like now as
• 11:51
  SPEAKER_00: the industry also mature like now we have few few products that we build
• 11:54
  SPEAKER_00: so somewhere we have we think there's some unique need we buy and
• 11:58
  SPEAKER_00: if we build anything if we think that's interesting for the rest of the industry
• 12:01
  SPEAKER_00: then we try to open source wherever possible.
• 12:03
  SPEAKER_01: Yeah I think two more tools that I think is worth mentioning
• 12:06
  SPEAKER_01: because I just remember them as one is Morpheus It's basically launched directly
• 12:11
  SPEAKER_01: for Uber, but it was way more integrated into everywhere, right?
• 12:14
  SPEAKER_01: Like you could go in there and run experiments, you could see the results immediately.
• 12:18
  SPEAKER_01: It was almost like feature flagging and experimentation,
• 12:21
  SPEAKER_01: but integrated into mobile feature flags.
• 12:23
  SPEAKER_01: And it was like,
• 12:24
  SPEAKER_01: I remember inside Uber,
• 12:25
  SPEAKER_01: like all of these integrations,
• 12:26
  SPEAKER_01: it was just integrated into a bunch of tools, all of them, right?
• 12:31
  SPEAKER_00: Yeah, and since we are also expanding to so many countries,
• 12:35
  SPEAKER_00: you need a really good configuration platform,
• 12:38
  SPEAKER_00: whether it's for configuring our backend services or also having feature flag system
• 12:43
  SPEAKER_00: and also experimentation.
• 12:44
  SPEAKER_00: Uber is a very data-driven company and everything goes through a feature flag,
• 12:49
  SPEAKER_00: everything goes through an XP.
• 12:52
  SPEAKER_00: And since we are a two-sided marketplace and three-sided marketplace,
• 12:58
  SPEAKER_00: some of the experiments are also not
• 13:02
  SPEAKER_00: We also have switchbacks and we have to build
• 13:05
  SPEAKER_00: a really deep experimentation that's baked into all of our tools
• 13:09
  SPEAKER_00: and it can automatically generate analysis and what are the p-values,
• 13:13
  SPEAKER_00: what metrics to use and that also led to working a lot on our metrics foundations,
• 13:19
  SPEAKER_00: data foundations so that whatever analysis that we generate
• 13:23
  SPEAKER_00: from this metrics are trustworthy and we can make business decisions based on that.
• 13:27
  SPEAKER_01: And I remember I was always wondering, why do we have our own feature flag system?
• 13:31
  SPEAKER_01: But then I was thinking like,
• 13:33
  SPEAKER_01: hmm, like show me a feature flagging solution for a vendor that supports cities.
• 13:37
  SPEAKER_01: In fact,
• 13:37
  SPEAKER_01: cities are first class citizens because inside Uber,
• 13:40
  SPEAKER_01: like I think cities were so important and like a lot of our tools had cities as,
• 13:45
  SPEAKER_01: for example, for cardinality, like every city had its own metrics.
• 13:49
  SPEAKER_01: And then you start to realize like,
• 13:50
  SPEAKER_01: oh, very few things treat, like Uber had a bunch of interesting constraints, right?
• 13:55
  SPEAKER_01: Like that no one else had at this scale.
• 13:57
  SPEAKER_00: Yeah, absolutely.
• 13:58
  SPEAKER_00: Yeah.
• 13:58
  SPEAKER_00: Yeah.
• 13:58
  SPEAKER_00: I think the city one is an interesting one.
• 14:00
  SPEAKER_00: It's like so deeply baked into like our infrastructure.
• 14:04
  SPEAKER_00: Like, for example, some of our like critical platform services, we also do sharding by city.
• 14:10
  SPEAKER_00: Oh, sharding by city.
• 14:11
  SPEAKER_00: Yeah.
• 14:12
  SPEAKER_00: Because then we can also observe across different tiers of cities.
• 14:18
  SPEAKER_00: Because we need to make sure,
• 14:20
  SPEAKER_00: even now,
• 14:22
  SPEAKER_00: across all of the regions where we serve our traffic,
• 14:25
  SPEAKER_00: we shard a city to a specific region.
• 14:27
  SPEAKER_00: We can also at any point in time failover that city to across regions and even
• 14:32
  SPEAKER_00: and we also have like a logical deployment groups where a group of cities are routed
• 14:37
  SPEAKER_00: to a group of containers and this is for few services we don't do it for all
• 14:42
  SPEAKER_00: the services we do it for some of the services that are where we need that level of
• 14:47
  SPEAKER_00: that level of sharding and that level of reliability.
• 14:51
  SPEAKER_01: One other system I wanted to ask you about that I remember using
• 14:55
  SPEAKER_01: and it was being developed in 2017, I believe, which is DevPod.
• 14:58
  SPEAKER_01: And it felt really ahead of the industry at the time.
• 15:01
  SPEAKER_01: How did that come along and what did it turn into?
• 15:03
  SPEAKER_00: Yeah.
• 15:04
  SPEAKER_00: So I think the backstory for DevPod is because we already talked about the growth
• 15:09
  SPEAKER_00: of microservices.
• 15:10
  SPEAKER_00: Initially, we had micro reposts where every service was its own repository.
• 15:15
  SPEAKER_00: And we didn't have a standard build system at that point.
• 15:17
  SPEAKER_00: Each service had their own,
• 15:19
  SPEAKER_00: like, and we had at least two, three build systems, two, three testing systems.
• 15:23
  SPEAKER_00: And it was very hard for even for a central team when we wanted
• 15:26
  SPEAKER_00: to do library upgrades and go across all the services and do upgrades.
• 15:30
  SPEAKER_00: So around 2017-18,
• 15:32
  SPEAKER_00: we then changed the strategy to getting to Monorepos,
• 15:37
  SPEAKER_00: even though they are microservices,
• 15:38
  SPEAKER_00: but all of the services were deployed from the Monorepos.
• 15:42
  SPEAKER_00: We had one Monorepo per language,
• 15:44
  SPEAKER_00: so we had a Monorepo for Go,
• 15:46
  SPEAKER_00: one for Java, one for mobile iOS, one for Android and so on.
• 15:50
  SPEAKER_00: Once the scale of that repository was pretty big enough then like it was pretty hard
• 15:55
  SPEAKER_00: to run the build and like getting all of those systems set up
• 16:00
  SPEAKER_00: on your local laptop also then you have to like go through all kinds
• 16:04
  SPEAKER_00: of setup issues.
• 16:05
  SPEAKER_00: So we took on this thing of like DevPod which is like cloud developer boxes
• 16:11
  SPEAKER_00: and you can have any number of them.
• 16:14
  SPEAKER_00: We had one devpod per monorepo.
• 16:16
  SPEAKER_00: So if I'm working primarily in Java monorepo,
• 16:18
  SPEAKER_00: I create the Java version of the flavor of the devpod and it has all
• 16:22
  SPEAKER_00: the build system cached.
• 16:24
  SPEAKER_00: I can have faster builds.
• 16:25
  SPEAKER_00: It's working on cloud machines as a beefier than even the laptops.
• 16:29
  SPEAKER_00: So then you could have like we could have really good SLAs on your build time,
• 16:33
  SPEAKER_00: on your test time.
• 16:35
  SPEAKER_00: And we don't have to worry about the setup issues because we can we can show
• 16:39
  SPEAKER_00: the right tools in place.
• 16:41
  SPEAKER_00: And now the interesting part is as we got to the AI era,
• 16:44
  SPEAKER_00: like even the agents need the similar workspaces and cloud boxes.
• 16:49
  SPEAKER_00: And we are able to kind of latch on to that.
• 16:52
  SPEAKER_00: And we also created like now a mega dev pod with all of the repositories.
• 16:56
  SPEAKER_00: Repositories, because then I can have my agents create PRs across the repositories.
• 17:01
  SPEAKER_00: It has all the tools it has to work in any repository, to look at context across any repository.
• 17:08
  SPEAKER_00: This also helped us to create the AgentsMD,
• 17:10
  SPEAKER_00: CloudMD at the top folder because we have standardized builds, standardized testing.
• 17:15
  SPEAKER_00: Once we set up the top level files, most teams will get like really good value out of the bat.
• 17:20
  SPEAKER_00: They don't have to like really go into per module setup.
• 17:24
  SPEAKER_00: And like on the DevPod, all of the AI tools are installed by default.
• 17:27
  SPEAKER_00: So you can get a new DevPod.
• 17:30
  SPEAKER_00: You don't have to go through even installing like Cloud Code, Codex, OpenGo.
• 17:33
  SPEAKER_00: They're all there in the DevPod.
• 17:35
  SPEAKER_00: So this made it really easy even for non-engineers in the company to create a DevPod
• 17:40
  SPEAKER_00: and start playing with Cloud Code, start playing with Codex out of the bat.
• 17:44
  SPEAKER_01: Yeah.
• 17:44
  SPEAKER_01: So I guess like having cloud development environments just like really early helped a lot.
• 17:49
  SPEAKER_00: Yeah, like I think all of those years of platformization bets also helped a lot
• 17:55
  SPEAKER_00: as we expanded on our agentic journey because you could use all the things
• 17:59
  SPEAKER_00: that we built making the life easier for engineers and it makes it easy
• 18:04
  SPEAKER_00: for our agents as well.
• 18:05
  SPEAKER_01: I wanted to talk a little bit about before we got into what's on today,
• 18:09
  SPEAKER_01: because I think a lot of things are connected, right?
• 18:12
  SPEAKER_01: So like there was so many things being built for platforms, for tools, often custom for Uber.
• 18:18
  SPEAKER_01: But having talked enough about that, what does it look like today?
• 18:24
  SPEAKER_01: What does development look like today?
• 18:25
  SPEAKER_01: What kind of tools do you use?
• 18:26
  SPEAKER_01: And maybe you could...
• 18:28
  SPEAKER_01: Give us a little bit of a primer of the foundations that you put
• 18:32
  SPEAKER_01: in place specifically for AI and what you built and why.
• 18:36
  SPEAKER_00: Especially in the AI world,
• 18:38
  SPEAKER_00: one thing we realized very early on was because
• 18:41
  SPEAKER_00: the landscape is changing pretty quickly,
• 18:44
  SPEAKER_00: harnesses are changing pretty quickly, models are changing pretty quickly.
• 18:47
  SPEAKER_00: We wanted to invest in the right set of building blocks that are hardness
• 18:51
  SPEAKER_00: and model agnostic so that no matter any of these things change,
• 18:55
  SPEAKER_00: we can easily port across different models, different harnesses.
• 18:59
  SPEAKER_00: So few things that we invested like,
• 19:01
  SPEAKER_00: so we started working on a model gateway from The initial introduction of ChatGPT,
• 19:06
  SPEAKER_00: like late 2013, early 2014.
• 19:09
  SPEAKER_00: So we had a model gateway back then,
• 19:12
  SPEAKER_00: which would expose an OpenAI compatible API or Anthropic compatible API.
• 19:17
  SPEAKER_01: Sorry, so like this is 2023, 2024, right?
• 19:19
  SPEAKER_00: Yeah, so 2023, 2020,
• 19:21
  SPEAKER_00: early 2024, when ChatGPT was first launched and GPT models were first launched.
• 19:25
  SPEAKER_00: And we quickly noticed,
• 19:27
  SPEAKER_00: you know,
• 19:27
  SPEAKER_00: we wanted to make it easy for any teams in the company who want to like leverage
• 19:33
  SPEAKER_00: and build agents with these models.
• 19:34
  SPEAKER_00: And so we had a model gateway that would abstract all of the complexity behind
• 19:38
  SPEAKER_00: where we had all of the security and policy guards in place.
• 19:42
  SPEAKER_00: We had all of the visibility in place.
• 19:44
  SPEAKER_00: So you could switch across different models or different vendors for these models.
• 19:49
  SPEAKER_00: And you could just have the ID for the organization or the project and
• 19:54
  SPEAKER_00: then you can use any of the models exposed.
• 19:57
  SPEAKER_01: So this is what you just expose it into like a model selector and it piped into
• 20:01
  SPEAKER_01: the model gateway and it routed to the right,
• 20:04
  SPEAKER_01: you know,
• 20:04
  SPEAKER_01: may that be ChatGPT at the time or Opus at the time, that kind of those models.
• 20:09
  SPEAKER_00: Yeah, and it was built on Michelangelo itself.
• 20:12
  SPEAKER_00: So we create a Michelangelo Gen AI product.
• 20:15
  SPEAKER_01: Oh, Michelangelo.
• 20:15
  SPEAKER_01: But that's an ML platform, right?
• 20:19
  SPEAKER_00: Yeah.
• 20:19
  SPEAKER_00: And then now there's a Gen AI set of foundations on Michelangelo platform.
• 20:23
  SPEAKER_00: So then you go to create a Gen AI project or a Gen DKI project on Michelangelo.
• 20:28
  SPEAKER_00: Then you can set up the project.
• 20:30
  SPEAKER_00: What is Model Gateway?
• 20:59
  SPEAKER_00: I can show like a quick primer into like you know what we had built
• 21:02
  SPEAKER_00: and this was late 2023 when the first set of LLMs were launched
• 21:09
  SPEAKER_00: and we quickly realized that we had a whole bunch of internal teams wanting
• 21:12
  SPEAKER_00: to use it for different use cases and we wanted to make it really easy for them
• 21:17
  SPEAKER_00: and this was built on top of the Michelangelo platform and one more component inside
• 21:20
  SPEAKER_00: that they can set up
• 21:22
  SPEAKER_01: And Michelangelo,
• 21:23
  SPEAKER_01: just to recap, that's the ML platform that was built from like 2015, 2016.
• 21:29
  SPEAKER_01: So it was built a long time before LLMs.
• 21:32
  SPEAKER_00: Exactly.
• 21:32
  SPEAKER_00: And now we have like a Gen AI or Agentic AI set of components also inside Michelangelo.
• 21:38
  SPEAKER_00: And we're also building a lot more right now that those are not in the open source version yet.
• 21:45
  SPEAKER_00: But we are actively building a lot of stuff for agentic AI world.
• 21:49
  SPEAKER_00: But my model gateway was the first such one and
• 21:52
  SPEAKER_00: the initial version we didn't have all of these middle version components
• 21:55
  SPEAKER_00: and we built some of this over the last one, two years.
• 21:58
  SPEAKER_00: But what the experience for any team in the company is,
• 22:01
  SPEAKER_00: you know,
• 22:02
  SPEAKER_00: they create a client,
• 22:03
  SPEAKER_00: they give their project ID, which they initialized in Michelangelo.
• 22:07
  SPEAKER_00: And then the caller token, and that's all.
• 22:10
  SPEAKER_00: And we exposed one OpenAI Compatible API or Anthropic Compatible API.
• 22:15
  SPEAKER_00: And all of the models across all the vendors,
• 22:18
  SPEAKER_00: open-weight or closed-weight, they're all behind this gateway.
• 22:23
  SPEAKER_00: And the first thing we added there was what we call AI Guard.
• 22:27
  SPEAKER_01: What is AI Guard?
• 22:29
  SPEAKER_00: Essentially, we wanted to make sure we are not sending any sensitive content back to the user.
• 22:36
  SPEAKER_00: So this was one entry point where we could add this middleware
• 22:41
  SPEAKER_00: that we can enforce consistently across all the projects.
• 22:45
  SPEAKER_00: And security team had ability to like,
• 22:47
  SPEAKER_00: you know,
• 22:47
  SPEAKER_00: then they could iterate on all of the AI guard specific models,
• 22:50
  SPEAKER_00: whether it's on content,
• 22:51
  SPEAKER_00: whether it's on safety policy,
• 22:52
  SPEAKER_00: whether it's on like a whole bunch of comms policy,
• 22:55
  SPEAKER_00: where like five different set of fine-tuned models that they built.
• 23:00
  SPEAKER_00: Which runs on every request and every response with strict latency guardrails.
• 23:05
  SPEAKER_00: This helped a lot to make sure that all of our projects have same standard baselines.
• 23:10
  SPEAKER_00: And then because every project is tracked here, We could have per user, per caller attribution.
• 23:17
  SPEAKER_00: Then as we progressed later this year,
• 23:20
  SPEAKER_00: we were able to add spent years for projects, spent years for users.
• 23:25
  SPEAKER_00: And since all the requests go through one place,
• 23:27
  SPEAKER_00: we could also add audit logging,
• 23:28
  SPEAKER_00: session traces, and a whole bunch of infrastructure around this.
• 23:32
  SPEAKER_00: Every model request at Uber goes through this.
• 23:35
  SPEAKER_00: So there's no model request that doesn't go through this.
• 23:38
  SPEAKER_00: So that gives us one single entry point across all the models.
• 23:42
  SPEAKER_01: And I assume like you can just try billing as well, like costs and all of those things.
• 23:47
  SPEAKER_00: Exactly.
• 23:47
  SPEAKER_00: Like you can track billing, cash sheet rate and all kinds of things.
• 23:52
  SPEAKER_00: You know as we progress into like there's a lot of discussion
• 23:54
  SPEAKER_00: of like you know model routing and all of that I think this could be like
• 23:57
  SPEAKER_00: a good entry point that we can leverage because all of our models go
• 24:00
  SPEAKER_00: through this I think on the same journey of like being able to make it easy
• 24:04
  SPEAKER_00: for engineers to switch across models the next thing was how do we make it easy
• 24:08
  SPEAKER_00: for engineers to switch across harnesses like if I'm using cloud code then
• 24:14
  SPEAKER_00: if I switch to codex then I need to like carry forward all of my settings all
• 24:18
  SPEAKER_00: the skills all the tools all the plugins So we internally wrap all of the harnesses
• 24:25
  SPEAKER_00: with a wrapper called AIFX, AI Foundations.
• 24:29
  SPEAKER_00: And this is pre-installed on Devpods or you can install on your local machine.
• 24:34
  SPEAKER_00: And we have a concept of personas.
• 24:37
  SPEAKER_00: So then that personas comes with a whole bunch of default plugins and skills so
• 24:42
  SPEAKER_00: that you don't even have to install.
• 24:44
  SPEAKER_00: Like any engineer, they get a whole bunch of default skills that they can use out of the bat.
• 24:50
  SPEAKER_00: We wrap everything, all of the settings are managed.
• 24:53
  SPEAKER_00: So even if you are using any of the hardnesses on your local or on DevPod or any
• 24:58
  SPEAKER_00: of your DevPod, the settings are synced.
• 25:01
  SPEAKER_00: So then centrally we can manage one set of settings that we can propagate across all
• 25:07
  SPEAKER_00: the users and any user-specific settings,
• 25:09
  SPEAKER_00: they are synced across all of their environments.
• 25:12
  SPEAKER_00: And again,
• 25:12
  SPEAKER_00: this makes it really easy because now we have one entry point
• 25:15
  SPEAKER_00: for switching the harnesses, switching the tools.
• 25:17
  SPEAKER_00: And then we also made it like there's a status line where we can give tips,
• 25:22
  SPEAKER_00: where we can have tracking of your spend across harnesses,
• 25:25
  SPEAKER_00: within that harness, in that session.
• 25:28
  SPEAKER_00: So all of those additional things that we could add without any explicit setup from
• 25:33
  SPEAKER_00: the engineers.
• 25:34
  SPEAKER_01: Nice.
• 25:34
  SPEAKER_01: And then speaking of tracking spend, how do you approach that right now?
• 25:38
  SPEAKER_01: Like as an engineer, I do,
• 25:39
  SPEAKER_01: or if I just join as a,
• 25:41
  SPEAKER_01: you know,
• 25:41
  SPEAKER_01: like a senior engineer somewhere, like, do I need to be aware of spend, of costs?
• 25:46
  SPEAKER_01: What's the thinking there?
• 25:47
  SPEAKER_00: See, we want to make sure engineers have access to all,
• 25:51
  SPEAKER_00: like any of the tools that are available.
• 25:55
  SPEAKER_00: And we have this notion of spend tiers.
• 25:57
  SPEAKER_00: So by default, you get certain, certain tier.
• 26:01
  SPEAKER_00: And then, then We have many levels of tiers.
• 26:05
  SPEAKER_00: Once you are close to that tier,
• 26:07
  SPEAKER_00: then you can raise a request to go to the next tier and you just need
• 26:14
  SPEAKER_00: a simple manager approval so then you can discuss with the manager.
• 26:17
  SPEAKER_00: The ROI for that project, like, how are you spending?
• 26:21
  SPEAKER_00: And it is just a conversation around responsible use of this, right?
• 26:25
  SPEAKER_00: But we don't want to restrict engineers, like,
• 26:27
  SPEAKER_00: even if they're working on, like,
• 26:31
  SPEAKER_00: if this is going to the right project use,
• 26:33
  SPEAKER_00: They should be able to spend whatever they need because ultimately we want
• 26:37
  SPEAKER_00: to maximize the gain from this.
• 26:40
  SPEAKER_00: But you also want to make sure there's a responsible use.
• 26:43
  SPEAKER_00: So we try to do the minimal amount of paperwork needed to cross the tiers
• 26:49
  SPEAKER_00: and they will see across any of our tools how much they're spending within
• 26:56
  SPEAKER_00: that hardness and across all the hardnesses.
• 26:58
  SPEAKER_00: And we also have different tiers for different managed regions.
• 27:02
  SPEAKER_00: So like then for all of the interactive tiers,
• 27:04
  SPEAKER_00: interactive harnesses,
• 27:05
  SPEAKER_00: we have some spent here for some of our background coding agents.
• 27:08
  SPEAKER_00: We have a separate independent tier.
• 27:11
  SPEAKER_00: So we make sure they can easily manage across all of these.
• 27:14
  SPEAKER_01: Yeah, but I guess it just makes sense.
• 27:16
  SPEAKER_01: Even if I do my side projects, I will be aware of my own spend on my whatever limit I have.
• 27:22
  SPEAKER_01: And I guess,
• 27:23
  SPEAKER_01: I guess,
• 27:23
  SPEAKER_01: do I sense it correctly that the thinking is just don't use tokens on stupid stuff
• 27:29
  SPEAKER_01: that is that makes no sense or accidentally,
• 27:32
  SPEAKER_01: you know, you might be just burning and you don't know.
• 27:34
  SPEAKER_01: So like, just get that feedback.
• 27:36
  SPEAKER_01: Like these things are not free.
• 27:37
  SPEAKER_00: Exactly like I think just having that feedback and we also have like
• 27:41
  SPEAKER_00: a plugin called cost check that will run across all of your sessions
• 27:46
  SPEAKER_00: and give you personalized tips based on your usage to say hey like this is the kind
• 27:50
  SPEAKER_00: of thing this much you're spending on this project These are the tips
• 27:55
  SPEAKER_00: that you can use to further improve your cost profile.
• 27:59
  SPEAKER_00: And I think all of this creates a good conversation between managers and engineers,
• 28:05
  SPEAKER_00: but also engineers will also see good feedback
• 28:07
  SPEAKER_00: into how they are efficiently spending across.
• 28:11
  SPEAKER_00: And that helped us a lot with all of the real-time coaching with context help, model sizing,
• 28:20
  SPEAKER_00: So that engineers are also aware of working in this new age.
• 28:23
  SPEAKER_00: We also don't want inefficient spend.
• 28:26
  SPEAKER_01: And what other foundations that you built are related to AI?
• 28:30
  SPEAKER_01: Oh, MCP gave way.
• 28:31
  SPEAKER_00: Oh, yeah.
• 28:32
  SPEAKER_00: I think the other backstory was around, I think, early 2025.
• 28:39
  SPEAKER_00: We started as Cursor kind of picked up.
• 28:42
  SPEAKER_00: We enabled Cursor and MCP was a Spec that was created around that time.
• 28:49
  SPEAKER_00: And we quickly realized that we can really increase the power of AI by exposing all
• 28:57
  SPEAKER_00: of our internal tools in an easy way across things.
• 29:01
  SPEAKER_00: We could potentially leverage this beyond just coding.
• 29:05
  SPEAKER_00: We could leverage it for all kinds of software engineering tasks.
• 29:07
  SPEAKER_00: So around April of 2025, we started working on MCP Gateway, similar to Model Gateway.
• 29:15
  SPEAKER_00: This is one gateway for all of the tools.
• 29:19
  SPEAKER_00: So what we did was because we already had good ideal registry across all
• 29:24
  SPEAKER_00: of our services,
• 29:25
  SPEAKER_00: whether they're using Thrift or Proto,
• 29:28
  SPEAKER_00: since we have the ideal specs,
• 29:30
  SPEAKER_00: we auto-generate the MCP specs for any of these services.
• 29:34
  SPEAKER_01: And just to add,
• 29:35
  SPEAKER_01: I guess,
• 29:35
  SPEAKER_01: folks who don't know,
• 29:36
  SPEAKER_01: like Uber services either emit Thrift or Prada buffers as strong contracts, right?
• 29:42
  SPEAKER_00: Yeah.
• 29:42
  SPEAKER_00: And like we initially we are just ripped and later we added support for protobuf as well.
• 29:48
  SPEAKER_00: But all of the new services in the last few years, they're all protobuf.
• 29:52
  SPEAKER_00: The ones that were created like a few years And as they go through rewrites,
• 30:00
  SPEAKER_00: usually teams switch to proto.
• 30:02
  SPEAKER_00: But whether they're in thrift or proto,
• 30:06
  SPEAKER_00: since we have all of the ideals in one single registry,
• 30:10
  SPEAKER_00: we could use that registry to generate A default mcp spec for them
• 30:15
  SPEAKER_00: so they could anyone in the company they can go
• 30:17
  SPEAKER_00: to this tool mcp gateway they can see all of the mcp servers that we auto-generated
• 30:25
  SPEAKER_00: and if you go to any of that you could see all of the Tools that were generated from
• 30:30
  SPEAKER_00: that existing API.
• 30:31
  SPEAKER_00: So these might be the APIs from those services that are projected as MCP tools.
• 30:37
  SPEAKER_00: And we auto-generate the request pack, response pack, the description.
• 30:44
  SPEAKER_00: Later, we also,
• 30:45
  SPEAKER_00: based on the API descriptions,
• 30:46
  SPEAKER_00: you generate LLM optimized descriptions for the APIs and fields so that we can,
• 30:52
  SPEAKER_00: like,
• 30:52
  SPEAKER_00: we don't, like,
• 30:52
  SPEAKER_00: there are,
• 30:53
  SPEAKER_00: we saw some tools with,
• 30:54
  SPEAKER_00: like, 3,000, 4,000 tokens, so then we can optimize it for modeling.
• 30:58
  SPEAKER_00: We have a framework for creating a custom MCP and proxy their custom MCP also
• 31:04
  SPEAKER_00: through MCP Gateway.
• 31:05
  SPEAKER_00: The other interesting thing we did was
• 31:20
  SPEAKER_00: Even all of our vendor tools,
• 31:23
  SPEAKER_00: whether it's Google, Slack, all of them are also exposed through MCP Gateway.
• 31:27
  SPEAKER_00: This reduced a lot of burden on configuration management since we always spoke
• 31:33
  SPEAKER_00: about AI effects, right?
• 31:34
  SPEAKER_00: So I could install any of the tools in the same manner,
• 31:38
  SPEAKER_00: whether it's internal MCP tools or external MCP tools.
• 31:42
  SPEAKER_00: In the last few months, we also projected all of these MCPs as CLIs.
• 31:47
  SPEAKER_00: So you could execute any MCP as a CLI.
• 31:51
  SPEAKER_00: So then if some agentic hardness wants to use it as CLI, they can use it as CLI.
• 31:55
  SPEAKER_00: If they want to use traditional MCP, they can use MCP.
• 31:58
  SPEAKER_01: And do I understand that this all comes because the APIs to start
• 32:02
  SPEAKER_01: with were already either generated or written in,
• 32:05
  SPEAKER_01: like a lot of them were generated that you already had a platform for the APIs.
• 32:09
  SPEAKER_01: So adding MCP was somewhat complicated.
• 32:11
  SPEAKER_01: I mean, trivial, of course, it's work, but you know what I mean.
• 32:14
  SPEAKER_01: And now generating CLIs is also somewhat trivial because you had a foundation before AI, right?
• 32:20
  SPEAKER_00: Yeah.
• 32:21
  SPEAKER_01: And so when you have it, that's what I'm starting to see with Uber.
• 32:24
  SPEAKER_01: You built a bunch of stuff before 2022 and you could just kind
• 32:28
  SPEAKER_01: of add AI infra pretty easily on it.
• 32:31
  SPEAKER_00: Yeah, and the same team that works on our API Gateway also worked on MCP Gateway.
• 32:36
  SPEAKER_00: So they're already in the business of building API Gateway.
• 32:39
  SPEAKER_00: So they know how to do any kind of protocol transformations, any kind of API Gateways.
• 32:46
  SPEAKER_00: So it was like we were able to pretty quickly build this.
• 32:50
  SPEAKER_00: So I think this was probably like,
• 32:53
  SPEAKER_00: especially once we got our Google Workspace and all of our Slack tools also
• 32:58
  SPEAKER_00: into this, it kind of like created all kinds of Like amazing agency workflows.
• 33:03
  SPEAKER_00: People could automate anything right at that point,
• 33:06
  SPEAKER_00: like from Slack to Google, Jira, Linear,
• 33:08
  SPEAKER_00: like any any tool, vendor tool or any internal service is at a CLI call away.
• 33:15
  SPEAKER_00: And then the other set of foundations as we like started thinking
• 33:19
  SPEAKER_00: about this space was See I think all of these go in this camp
• 33:22
  SPEAKER_00: of like vendor-neutral hardness-neutral building blocks that we think like you know
• 33:28
  SPEAKER_00: as like you would have to do this irrespective of what happens
• 33:31
  SPEAKER_00: because you would need agents need some way to access your tools.
• 33:34
  SPEAKER_00: Then the other thing was around skills like, you know, we jumped onto this earlier this year.
• 33:40
  SPEAKER_00: Since we already have this notion of monorepos,
• 33:43
  SPEAKER_00: we created a skill monorepo in some sense that has all of the core skills
• 33:48
  SPEAKER_00: that central platform teams build and all of
• 33:50
  SPEAKER_00: the team specific skills individual teams can build.
• 33:54
  SPEAKER_01: And are these skills,
• 33:55
  SPEAKER_01: these are just like the skills folders,
• 33:57
  SPEAKER_01: like the plugins that you can use for cloud code, etc.
• 34:00
  SPEAKER_00: Exactly.
• 34:01
  SPEAKER_00: So there's plugins and skills that you can use across all the harnesses.
• 34:04
  SPEAKER_00: Then we had like one single quality gates that can run across all of them.
• 34:10
  SPEAKER_00: And we made it portable across all of the harnesses.
• 34:14
  SPEAKER_00: So you install with the same AIFX tooling that I mentioned.
• 34:18
  SPEAKER_00: And like if you do plugin add that skill,
• 34:22
  SPEAKER_00: it will update the configurations across all of the harnesses,
• 34:26
  SPEAKER_00: whether you are at that point using that harness or not.
• 34:30
  SPEAKER_00: So that even if you switch across harnesses, you have these plugins.
• 34:36
  SPEAKER_00: And
• 34:40
  SPEAKER_00: since all of the skills are using the tools from the MCP Gateway,
• 34:44
  SPEAKER_00: we also have one standard mechanism for all the skills to access any of the tools.
• 34:49
  SPEAKER_00: Now we are working on some tools where engineers can go select their skill,
• 34:54
  SPEAKER_00: It will tell them all kinds of tips to improve their skills.
• 34:58
  SPEAKER_00: The next thing that in-flight that you're working is how do we then collect
• 35:03
  SPEAKER_00: the traces from the harnesses and generate improvement patches for the skills so
• 35:08
  SPEAKER_00: that we can continue
• 35:11
  SPEAKER_00: Because there's a lot of discussion around the skill quality,
• 35:15
  SPEAKER_00: skill evals and how do we improve that continuously.
• 35:19
  SPEAKER_00: Like when you first build a skill,
• 35:20
  SPEAKER_00: they might not be perfect and you need some mechanism to continuously improve that.
• 35:24
  SPEAKER_00: I think both the MCP and skills, this gave us like a lot of like benefits.
• 35:32
  SPEAKER_00: And like if I look at like in the last few weeks,
• 35:36
  SPEAKER_00: there's around probably 30,000 skill executions per day.
• 35:39
  SPEAKER_01: This is kind of interesting for me to see because if you told someone like, oh,
• 35:42
  SPEAKER_01: we're building a skills repo,
• 35:44
  SPEAKER_01: like what most people would think inside a startup, right?
• 35:46
  SPEAKER_01: I'm not talking about Uber, but like, oh, you just do a get...
• 35:50
  SPEAKER_01: Just a repo and you have a bunch of scales there,
• 35:53
  SPEAKER_01: just a bunch of scale folders and that's kind of it.
• 35:55
  SPEAKER_01: But it's pretty clear that it's more complex, right?
• 35:59
  SPEAKER_01: Is it more complex because of how many people, engineers you need to support?
• 36:04
  SPEAKER_01: Is it because you're also running evals because you keep cleansing these scales
• 36:09
  SPEAKER_01: or like tracking if they're used or not used, those kind of things?
• 36:12
  SPEAKER_00: Yeah, absolutely.
• 36:13
  SPEAKER_00: I think it's like now with any of these latest models,
• 36:18
  SPEAKER_00: it's very easy to create like a V0 version of the scale.
• 36:22
  SPEAKER_00: It's very hard to make it really good across all kinds of use cases.
• 36:26
  SPEAKER_00: And that's why you want to To really make sure we have one place,
• 36:31
  SPEAKER_00: we can update all of these skills.
• 36:34
  SPEAKER_00: In fact,
• 36:34
  SPEAKER_00: when we exposed the CLI projection for all of our MCPs,
• 36:39
  SPEAKER_00: just a couple of engineers using AI,
• 36:43
  SPEAKER_00: they updated all of the skills in this repository to use the CLI pattern
• 36:47
  SPEAKER_00: because it's more token efficient.
• 36:49
  SPEAKER_00: Because if every skill says I want MCP ABC,
• 36:52
  SPEAKER_00: MCP DEF,
• 36:55
  SPEAKER_00: Then if I install these two plugins,
• 36:58
  SPEAKER_00: I'm going to have like six MCPs across these two always in my context and it is
• 37:02
  SPEAKER_00: so much context load right and like then we replace all the skills with
• 37:06
  SPEAKER_00: the CLI pattern so that that had a lot of impact on our like token efficiency
• 37:11
  SPEAKER_00: as well and the other thing is because We auto install this skill marketplace
• 37:19
  SPEAKER_00: for everyone.
• 37:21
  SPEAKER_00: It reduces one more step of configuration and so then like
• 37:24
  SPEAKER_00: the discoverability is easy,
• 37:26
  SPEAKER_00: the setup and configuration and management is easy and we could build whole bunch
• 37:30
  SPEAKER_00: of like eval pipelines around that, discoverability pipelines around that.
• 37:35
  SPEAKER_00: We still have teams that have like their own repositories but majority of
• 37:39
  SPEAKER_00: the skills go through this.
• 37:40
  SPEAKER_00: My favorite is like context graph.
• 37:44
  SPEAKER_01: What is contact draft?
• 37:45
  SPEAKER_00: The backstory is back in December,
• 37:48
  SPEAKER_00: one of the engineers built an AI assistant that you can go to,
• 37:54
  SPEAKER_00: we call it Cortana,
• 37:55
  SPEAKER_00: you can go to web or you can go to Slack and ask any question from any code base.
• 37:59
  SPEAKER_00: So it would search across all of our repositories and answer questions.
• 38:04
  SPEAKER_00: And then we saw people asking all kinds
• 38:07
  SPEAKER_00: of questions like it could be research it could be like what is how
• 38:10
  SPEAKER_00: to run this query like what is the data source for this
• 38:13
  SPEAKER_00: who owns this like can you do RCA for this whole bunch of software engineering tasks
• 38:18
  SPEAKER_00: and when we're looking at some of these traces like it was clearly obvious
• 38:22
  SPEAKER_00: that you know we could optimize the number of turns it takes for the models
• 38:26
  SPEAKER_00: to get grounding because we have so many systems Historically,
• 38:31
  SPEAKER_00: all of these are siloed systems.
• 38:32
  SPEAKER_00: Engineer has to go into the task management platform,
• 38:36
  SPEAKER_00: engineer has to go to our on-call dashboard,
• 38:39
  SPEAKER_00: has to go to our own dashboard that we talked about.
• 38:41
  SPEAKER_00: Then we're like, okay, how do we plug all of that into work?
• 38:45
  SPEAKER_00: We have 100 plus node types and 130 plus edge types and this is live
• 39:00
  SPEAKER_00: If you click on any person,
• 39:04
  SPEAKER_00: you know what assets they manage,
• 39:06
  SPEAKER_00: what design docs they wrote,
• 39:08
  SPEAKER_00: what features they're working on,
• 39:10
  SPEAKER_00: what on-call events they worked on,
• 39:12
  SPEAKER_00: what are the configuration changes, what incidents they manage.
• 39:17
  SPEAKER_00: And if you take any node, it has a relationship to all other nodes.
• 39:20
  SPEAKER_00: And what this allows us is now the same assistant I talked about.
• 39:25
  SPEAKER_00: It has the context graph as the default grounding.
• 39:28
  SPEAKER_00: So if you ask any question,
• 39:30
  SPEAKER_00: It will first use this to like navigate,
• 39:32
  SPEAKER_00: okay, like what is this person asking about?
• 39:34
  SPEAKER_00: Once you get grounded into, oh,
• 39:36
  SPEAKER_00: this is about this domain,
• 39:37
  SPEAKER_00: this set of services,
• 39:39
  SPEAKER_00: then it can go deep and read other documents, read other things.
• 39:43
  SPEAKER_00: So it reduces your agent races and the amount of turns it needs to give you a good answer.
• 39:49
  SPEAKER_00: And we already see like in a whole bunch of categories of questions,
• 39:53
  SPEAKER_00: The quality goes up,
• 39:55
  SPEAKER_00: the cost goes down,
• 39:56
  SPEAKER_00: the latency goes down,
• 39:58
  SPEAKER_00: and we are still early in harnessing the full potential of this,
• 40:02
  SPEAKER_00: and we are super excited about all of the things that we can build on top
• 40:07
  SPEAKER_00: of this going forward.
• 40:08
  SPEAKER_01: And then the nodes, are they people?
• 40:10
  SPEAKER_01: Are they tools?
• 40:11
  SPEAKER_00: This is the schema visualization.
• 40:13
  SPEAKER_00: So every person in the company will have like, there's a node for me, a node for every employee.
• 40:19
  SPEAKER_00: And then it will have relationships to all of the instances,
• 40:23
  SPEAKER_00: like every PR that I generate, there will be a node for that.
• 40:27
  SPEAKER_00: And there's a link saying, okay, Uday created this PR.
• 40:31
  SPEAKER_00: Uday manages this asset.
• 40:35
  SPEAKER_00: I work on this product feature.
• 40:36
  SPEAKER_00: I contributed to this design doc.
• 40:39
  SPEAKER_01: So then the nodes are both people, but they're also assets.
• 40:43
  SPEAKER_00: Yeah, exactly.
• 40:44
  SPEAKER_00: Like one simple example is,
• 40:45
  SPEAKER_00: let's say if you ask any question like how many mobility trips around the world are
• 40:53
  SPEAKER_00: with cash payment type.
• 40:55
  SPEAKER_00: Like you first need to understand what tables exist, what columns exist.
• 40:59
  SPEAKER_00: Are there other people who wrote these queries?
• 41:02
  SPEAKER_00: Are they from science team?
• 41:03
  SPEAKER_00: Are they from mobility science?
• 41:05
  SPEAKER_00: Like now all of this data is interconnected.
• 41:08
  SPEAKER_00: You can go and check are there other queries that were done in
• 41:12
  SPEAKER_00: the last 30 days around this and what kind of tables exist.
• 41:16
  SPEAKER_00: Go and explore those columns because again all of this is interlinked.
• 41:20
  SPEAKER_00: Then you can quickly get to Like the blessed queries that we have
• 41:25
  SPEAKER_00: and answer these kind of even complex questions pretty quickly.
• 41:29
  SPEAKER_00: Otherwise, you would have to go through many, many dogs, many, many like tool calls.
• 41:34
  SPEAKER_00: This makes it really efficient.
• 41:35
  SPEAKER_00: Like it's much harder for us to manually write all of these cipher queries,
• 41:39
  SPEAKER_00: but agents are really good at navigating the graph,
• 41:42
  SPEAKER_00: like how they're really good at navigating file system with bash.
• 41:45
  SPEAKER_01: It seems very futuristic to me, but it's pretty cool.
• 41:47
  SPEAKER_01: And I assume it's a ton of data to store just
• 41:49
  SPEAKER_01: because Uber has like 10,000 plus people working there easily.
• 41:52
  SPEAKER_01: So...
• 41:53
  SPEAKER_00: Yeah, the good part is,
• 41:54
  SPEAKER_00: you know, similar to how our API Edge Gateway team worked on MCP Gateway effort.
• 42:00
  SPEAKER_00: Our graph platform team that manages graph or our production graph with billions
• 42:04
  SPEAKER_00: of nodes, they also helped build this graph.
• 42:08
  SPEAKER_00: So this is, in compared to our production graph, this is much smaller.
• 42:11
  SPEAKER_00: So this is, so it's not like a scale challenge for them anymore.
• 42:17
  SPEAKER_00: I already talked about the AI assistant Cortana which kind of plugs some of all of
• 42:26
  SPEAKER_00: the context craft, tools, skills and it's omnipresent in Slack, Web, CLI.
• 42:32
  SPEAKER_00: So anyone in the company like even someone in Ops
• 42:35
  SPEAKER_00: in some country they can ask any question about how is this built,
• 42:38
  SPEAKER_00: where is this located.
• 42:40
  SPEAKER_00: Who works on this?
• 42:41
  SPEAKER_00: What was the history behind this?
• 42:43
  SPEAKER_00: And like it is able to take all of this data,
• 42:45
  SPEAKER_00: all of our Google documents and then and then give them like a really good answer.
• 42:49
  SPEAKER_00: And it breaks down kind of the information silos across orgs, right?
• 42:53
  SPEAKER_00: Like they're able to like self serve, ask anything, execute any skill anyone has built.
• 42:58
  SPEAKER_01: And I guess it must have access to the data sources,
• 43:02
  SPEAKER_01: at least for the right people,
• 43:03
  SPEAKER_01: so it can go into things like logs and monitoring, those kind of things.
• 43:07
  SPEAKER_00: Yeah, it has logs, monitoring, you can ask, okay, what is the health of this service?
• 43:11
  SPEAKER_00: It can go look at all of its dependencies,
• 43:14
  SPEAKER_00: what kind of storage systems it uses,
• 43:16
  SPEAKER_00: run all the queries and give you a 360 degree health report for that asset.
• 43:20
  SPEAKER_00: And we carry forward all of the existing user permissions,
• 43:23
  SPEAKER_00: so then if I'm only able to access the stuff that I had access to.
• 43:28
  SPEAKER_00: Otherwise, that data is not accessible.
• 43:31
  SPEAKER_01: Yeah, because inside Uber,
• 43:33
  SPEAKER_01: I remember it used to be called break glass when you added a permission for
• 43:36
  SPEAKER_01: a system you didn't have access to.
• 43:38
  SPEAKER_01: You go through an approval flow, which goes through your manager or director or whatnot.
• 43:41
  SPEAKER_01: But I know Uber was really big.
• 43:43
  SPEAKER_01: At my time, they were really big on this already.
• 43:45
  SPEAKER_00: Would I just mention how Uber wrapped up AI foundations work
• 43:48
  SPEAKER_00: but already more foundations work is to come.
• 44:12
  SPEAKER_01: Our presenting sponsor, TurboPuffer, knows this all too well.
• 44:14
  SPEAKER_01: They are rebuilding their storage architecture from first principles.
• 44:18
  SPEAKER_01: They are doing this to make more types of storage faster, cheaper, and more reliable at scale.
• 44:23
  SPEAKER_01: The new system called TurboPuffer v3 is not yet in prod, but it already passes CI.
• 44:28
  SPEAKER_01: Still, it's not nearly as performant as the current production version, v2,
• 44:33
  SPEAKER_01: and so the TPuff team is now perf-maxing query plans until they are as fast
• 44:37
  SPEAKER_01: or faster than v2.
• 44:39
  SPEAKER_01: They're doing this in public,
• 44:40
  SPEAKER_01: documenting how they increase performance all the way up to
• 44:43
  SPEAKER_01: the v3 production launch.
• 44:45
  SPEAKER_01: For example,
• 44:46
  SPEAKER_01: we can see how the team increased full-text search performance 9x by batching reads.
• 44:51
  SPEAKER_01: A lot of this is basic foundational work right now,
• 44:53
  SPEAKER_01: but they will be sharing some very cool systems work over the next few weeks.
• 44:57
  SPEAKER_01: But hang on, why is the team doing a full rewrite?
• 45:00
  SPEAKER_01: It's because search is changing, and so is TurboBuffer.
• 45:03
  SPEAKER_01: T5 started as a Vector database, but it is no longer VectorDB.
• 45:07
  SPEAKER_01: It has evolved into a high-performance database for search.
• 45:10
  SPEAKER_01: The V3 discards the Vector Primary Index,
• 45:12
  SPEAKER_01: and all indexes will share the same columnar storage system.
• 45:16
  SPEAKER_01: Every query will run on vectorized query engines,
• 45:18
  SPEAKER_01: which is something you don't usually see in search databases,
• 45:21
  SPEAKER_01: Which will result in both very fast search and just generally a very fast database.
• 45:26
  SPEAKER_01: The TurboPuffer team is keeping a work log of their journey and you can follow along
• 45:29
  SPEAKER_01: at turbopuffer.com slash v3.
• 45:32
  SPEAKER_01: It's a blast to see how it's being built.
• 45:34
  SPEAKER_01: I also want to mention our season sponsor Lanier,
• 45:36
  SPEAKER_01: the place to coordinate work between humans as well as agents.
• 45:40
  SPEAKER_01: One of the biggest changes for us software engineers is the sheer amount of code
• 45:43
  SPEAKER_01: that we need to review.
• 45:45
  SPEAKER_01: Uber has built custom tools to deal with this but so has Linear.
• 45:48
  SPEAKER_01: It's called Linear Diffs and it lets engineers review PRs right inside of Linear.
• 45:52
  SPEAKER_01: One thing I really like about Linear Diffs is how a diff sits next to
• 45:55
  SPEAKER_01: an issue inside a project and so it's super simple to see all of the context
• 45:59
  SPEAKER_01: in one place and know why the change has happened without needing to jump
• 46:03
  SPEAKER_01: through different tools.
• 46:04
  SPEAKER_01: For large PRs,
• 46:05
  SPEAKER_01: guided reviews walk you through the core change with an explanation
• 46:08
  SPEAKER_01: of what has changed.
• 46:09
  SPEAKER_01: Guided reviews also group the glue code and the supporting edits separately.
• 46:14
  SPEAKER_01: These small but important changes make a large PR readable in one sitting.
• 46:19
  SPEAKER_01: And linear also supports stacked PRs.
• 46:21
  SPEAKER_01: If you've used stacked PRs, you'll know that they're a great way to break up complex work.
• 46:25
  SPEAKER_01: And if you've not yet used them, you'll probably love them when you try them out.
• 46:29
  SPEAKER_01: How we do code reviews keeps changing as AI agents take on more of the work.
• 46:33
  SPEAKER_01: These neat changes coming from Linear help us engineers understand code reviews more
• 46:38
  SPEAKER_01: and expect Linear to evolve diffs to keep up with how we work more with AI agents.
• 46:42
  SPEAKER_01: To learn more and to get six months off of Linear, head to linear.app.pragmatic.
• 46:48
  SPEAKER_01: With this,
• 46:48
  SPEAKER_01: let's get back to Uber and how building a feature looks like today in terms
• 46:52
  SPEAKER_01: of tools used by devs at the company.
• 46:54
  SPEAKER_01: ERD and ERD, I guess for those who don't know,
• 46:56
  SPEAKER_01: product requirements documents and ERD was engineering requirements documents,
• 47:01
  SPEAKER_01: right?
• 47:01
  SPEAKER_01: Like it's like the product spec and engineering plan for those not familiar inside Uber.
• 47:07
  SPEAKER_00: Yeah,
• 47:07
  SPEAKER_00: so I think we went through some of the AI foundations and like now if we think about
• 47:12
  SPEAKER_00: the lifecycle of a feature,
• 47:14
  SPEAKER_00: then how can we Truly uplevel each of those life cycle with the right set
• 47:19
  SPEAKER_00: of AI foundations and also rethinking those tools for the AI era.
• 47:23
  SPEAKER_00: And some we are more farther ahead,
• 47:26
  SPEAKER_00: some we are early in the discussion,
• 47:28
  SPEAKER_00: like I think even the notion of a combined There's a lot of cool ideas,
• 47:32
  SPEAKER_00: but we also have all kinds of agentic skills for working with any of your
• 47:46
  SPEAKER_00: Design documents.
• 47:47
  SPEAKER_00: And as I was saying before,
• 47:49
  SPEAKER_00: whether it's a product design doc,
• 47:50
  SPEAKER_00: engineering design doc,
• 47:51
  SPEAKER_00: launch reviews,
• 47:52
  SPEAKER_00: XP reviews, production readiness reviews, all of those things are in one place.
• 47:58
  SPEAKER_00: I'm filtering some really old projects so that we can look through them.
• 48:03
  SPEAKER_00: But you will know exactly who is working on them, track the status of that.
• 48:10
  SPEAKER_00: And all of this data also goes into the graph as well so
• 48:12
  SPEAKER_00: that we know like full history of what are the source of truth design docs,
• 48:17
  SPEAKER_00: approval status, who works on them.
• 48:19
  SPEAKER_00: And each of these also have like the Jira link or the linear link.
• 48:23
  SPEAKER_00: So then we can trace from this project, how was the execution translated to.
• 48:28
  SPEAKER_00: This gives like a source of truth system for both humans and agents to know
• 48:32
  SPEAKER_00: the specs for every project.
• 48:33
  SPEAKER_01: Yeah, I guess planning is still really important,
• 48:37
  SPEAKER_01: especially for like something that is not just a trivial,
• 48:39
  SPEAKER_01: you know, like one prompt or one shot project.
• 48:42
  SPEAKER_00: Yeah, exactly.
• 48:43
  SPEAKER_00: And because we need to like,
• 48:44
  SPEAKER_00: there's so many with the kind of Product heterogeneity
• 48:49
  SPEAKER_00: and country heterogeneity like we need to like really think about all kinds
• 48:53
  SPEAKER_00: of corner cases more than like they know the the happy path to think about all kinds
• 48:57
  SPEAKER_00: of corner cases and like how they should exist and since we also operate
• 49:01
  SPEAKER_00: in physical world the number Yeah, absolutely.
• 49:03
  SPEAKER_00: I think that's why it really becomes important to think about the right plan upfront.
• 49:06
  SPEAKER_00: And this will also
• 49:32
  SPEAKER_00: Make it easy for both the engineers or agents to take that plan and decompose
• 49:37
  SPEAKER_00: and execute on that.
• 49:38
  SPEAKER_00: Once we have the plan,
• 49:39
  SPEAKER_00: then the next thing is how do you convert that into, decompose that and execute.
• 49:44
  SPEAKER_00: Now we're going to what we call Minion internally.
• 49:50
  SPEAKER_00: So this is our background coding agent.
• 49:52
  SPEAKER_00: So around 15% of all of the production PRs go through Minion right now.
• 50:00
  SPEAKER_00: Essentially,
• 50:00
  SPEAKER_00: you know,
• 50:02
  SPEAKER_00: this whole bunch of PRs that some PRs engineers want to do interactively,
• 50:07
  SPEAKER_00: but we also want to make it really easy.
• 50:09
  SPEAKER_00: Once you have for the task that I have really good spec or I want
• 50:13
  SPEAKER_00: to do minimal steering, I can hand off that to a Minion.
• 50:18
  SPEAKER_00: And we had again we had we did multiple iterations is
• 50:20
  SPEAKER_00: the latest iteration we call it Mahini and v2 but like you can use it
• 50:24
  SPEAKER_00: from web you can use it from slack you can do a plan mode and then let it execute
• 50:29
  SPEAKER_00: behind the scenes or you give one you can give the prompt once and let it execute
• 50:33
  SPEAKER_00: behind the scenes if it's like a simple simple prompt and it has access
• 50:37
  SPEAKER_00: to this is using the mega dev pod so then it has access to All of the repositories,
• 50:43
  SPEAKER_00: all of the code base that we have,
• 50:45
  SPEAKER_00: it can work across repositories and generate one PR or multiple PRs
• 50:50
  SPEAKER_00: across repositories based on the plan and spec that you have.
• 50:54
  SPEAKER_00: When we initially started, mostly maintenance kind of tasks used to be offloaded to Minion.
• 51:01
  SPEAKER_00: As we progress on this journey,
• 51:03
  SPEAKER_00: as we are adding more and more tools into the Minion environment
• 51:07
  SPEAKER_00: where Minion can Self-validate its work in web or mobile or backend APIs.
• 51:14
  SPEAKER_00: Our goal is we can increase the kind of tasks people can do with Minion over time,
• 51:19
  SPEAKER_00: increase the percentage of tasks over time.
• 51:22
  SPEAKER_00: I think we can also see how it's
• 51:26
  SPEAKER_01: integrated in Slack.
• 51:27
  SPEAKER_01: You can just mention, do you mention at Minion or what do you say?
• 51:30
  SPEAKER_00: You can do at minion and then it will show you the status.
• 51:35
  SPEAKER_00: And you can also like you don't need to go to the web.
• 51:38
  SPEAKER_00: You can steer it on Slack itself.
• 51:42
  SPEAKER_00: Maybe you have like a Slack thread with the With the EM,
• 51:46
  SPEAKER_00: with the PM, you can iterate on the design.
• 51:49
  SPEAKER_00: If it's a simple thing, you are just iterating and get alignment there.
• 51:52
  SPEAKER_00: And then say, hey, at Minion, follow this thread.
• 51:54
  SPEAKER_00: And can you create a PR?
• 51:57
  SPEAKER_00: And then it will take that Slack thread context and then create a PR for you.
• 52:01
  SPEAKER_00: So it's been really useful.
• 52:03
  SPEAKER_00: Even in our support channels,
• 52:04
  SPEAKER_00: I would see like someone reports a bug,
• 52:06
  SPEAKER_00: the on-call ask Minyan, hey, can you see what is the issue?
• 52:11
  SPEAKER_00: And then Minyan will come back with, oh, this is the root cause.
• 52:13
  SPEAKER_00: Oh, this looks simple enough.
• 52:15
  SPEAKER_00: Let's just fix it right there.
• 52:16
  SPEAKER_00: And then we would like fix it.
• 52:17
  SPEAKER_00: Like there's no then loop of like adding it to backlog, doing in the next.
• 52:21
  SPEAKER_00: It's improved since hasn't it?
• 52:35
  SPEAKER_00: Yeah, so this is showing all of my devpods.
• 52:39
  SPEAKER_00: Since I'm working on context graph, you see that.
• 52:42
  SPEAKER_00: But yeah, if you want to create a new devpod, you'll see which flavor you want.
• 52:49
  SPEAKER_00: For minion,
• 52:50
  SPEAKER_00: there is a devpod flavor that has all of the repositories,
• 52:55
  SPEAKER_00: and optimized for minion usage.
• 52:57
  SPEAKER_00: But for interactive,
• 52:59
  SPEAKER_00: it will create a devpod, and within a matter of seconds, You will have that running.
• 53:06
  SPEAKER_01: That's impressive.
• 53:07
  SPEAKER_01: I'm sure you do some clever stuff like having things warm or pre-built
• 53:12
  SPEAKER_01: so you can copy those kind of things.
• 53:14
  SPEAKER_00: Yeah, it's like,
• 53:15
  SPEAKER_00: it's running on, all of,
• 53:17
  SPEAKER_00: this is,
• 53:18
  SPEAKER_00: because this is developer stuff, like,
• 53:19
  SPEAKER_00: we have, like,
• 53:20
  SPEAKER_00: a separate multiple clouds that we are using for this,
• 53:22
  SPEAKER_00: and it's, like,
• 53:23
  SPEAKER_00: Kubernetes cluster, and, like,
• 53:25
  SPEAKER_00: everything is warmed up,
• 53:27
  SPEAKER_00: so then when you ask it, like,
• 53:28
  SPEAKER_00: we are able to just clone from the pre-built image,
• 53:31
  SPEAKER_00: and it has build caches,
• 53:33
  SPEAKER_00: everything,
• 53:34
  SPEAKER_00: repositories,
• 53:34
  SPEAKER_00: everything,
• 53:35
  SPEAKER_00: already cloned, like,
• 53:36
  SPEAKER_00: it's ready for you to build, so,
• 53:40
  SPEAKER_00: we use Bazel across all of our repositories for build systems, so,
• 53:44
  SPEAKER_00: I was able to go into a repository like run Bazel for that and
• 53:49
  SPEAKER_00: then just like we went we went through this in the order
• 53:52
  SPEAKER_00: of this like you know seconds where any engineer like if they if I want to make
• 53:56
  SPEAKER_00: a code change here like within order of seconds I create DevPod It's one single way
• 54:01
  SPEAKER_00: to build any module and then anyone knows this command.
• 54:05
  SPEAKER_00: And now agents also knows the same way, right?
• 54:08
  SPEAKER_00: In the ClaudeMD,
• 54:08
  SPEAKER_00: AgentsMD,
• 54:09
  SPEAKER_00: they know how to build,
• 54:11
  SPEAKER_00: test any module in any repository and they're able to use the
• 54:13
  SPEAKER_01: exact same commands.
• 54:15
  SPEAKER_00: Yeah, so I think that covers kind of like,
• 54:16
  SPEAKER_00: you know, what the kind of stuff that you have to do in code generation.
• 54:19
  SPEAKER_00: So you can do interactive hardness, you can do like through Minion.
• 54:23
  SPEAKER_00: The next thing is, okay, like you have the PR.
• 54:26
  SPEAKER_00: What do you do after that?
• 54:27
  SPEAKER_00: Then, um, Maybe go through self-healing CI first.
• 54:31
  SPEAKER_00: One thing that we realized is because our PR volume has gone up year over year.
• 54:36
  SPEAKER_00: Oh yeah, it must have explored.
• 54:38
  SPEAKER_00: Yeah, our P50 cycle time,
• 54:39
  SPEAKER_00: P90 cycle time,
• 54:41
  SPEAKER_00: both for inner loop where you have the PR and you want to get it to a green state
• 54:47
  SPEAKER_00: and create the remote PR.
• 54:50
  SPEAKER_00: Or once you have your PR created, Getting the human review, getting all the CI things to pass.
• 54:56
  SPEAKER_00: I think reviews become a big bottleneck, same as like any other company.
• 55:01
  SPEAKER_00: And we see year over year the delta and we want to like really,
• 55:04
  SPEAKER_00: we're like really focused on like breaking down every aspect of this.
• 55:08
  SPEAKER_00: And like how can we go really deep and optimizing it for the AI era.
• 55:12
  SPEAKER_00: So first thing that we did was we took a whole bunch of like simple CI jobs
• 55:17
  SPEAKER_00: that used to run once you create the PR and we moved it to inner loop
• 55:21
  SPEAKER_00: and we kept all of this behind one single command like AI FX validate.
• 55:26
  SPEAKER_00: So then when agent creates a PR,
• 55:28
  SPEAKER_00: it can run the linter,
• 55:29
  SPEAKER_00: the coverage rules,
• 55:30
  SPEAKER_00: the build rules,
• 55:31
  SPEAKER_00: the template rules,
• 55:32
  SPEAKER_00: like all of these things so that the probability of this PR is in
• 55:36
  SPEAKER_00: a good condition like goes up.
• 55:38
  SPEAKER_00: Then we hooked on ability to do visual validation where I can spin up emulator and test it.
• 55:45
  SPEAKER_00: I can spin up like a web and test it.
• 55:47
  SPEAKER_00: I can spin up like a staging environment where I can deploy this change,
• 55:51
  SPEAKER_00: run the APIs and test it.
• 55:53
  SPEAKER_00: And now Minion or even my interactive harnesses,
• 55:56
  SPEAKER_00: these agents know how to have the skills for this.
• 56:00
  SPEAKER_00: And they can go through this verification loop.
• 56:02
  SPEAKER_00: And let's say you create the PR and still something fails.
• 56:07
  SPEAKER_00: Then we have like a self-healing CI.
• 56:10
  SPEAKER_00: You can apply to all the PRs or per PR.
• 56:13
  SPEAKER_00: So this is like a uber specification.
• 56:15
  SPEAKER_00: Oh, nice.
• 56:35
  SPEAKER_00: And once you do all of this,
• 56:37
  SPEAKER_00: then we have this thing called Code Inbox,
• 56:40
  SPEAKER_00: this is showing a mock version of that,
• 56:43
  SPEAKER_00: where any engineer can go,
• 56:46
  SPEAKER_00: they can see all the PRs that are waiting for them,
• 56:48
  SPEAKER_00: all the PRs that they created, You can just quickly take a look.
• 56:52
  SPEAKER_00: You have a video as well.
• 57:12
  SPEAKER_00: Yeah, I can see the bill checks and all of that.
• 57:14
  SPEAKER_00: And we're also adding risk score, blast radius of this PR and so on.
• 57:19
  SPEAKER_00: So all of this will give engineers really good signals to figure out,
• 57:23
  SPEAKER_00: okay, is it good to land and which parts of the PR I should focus on.
• 57:27
  SPEAKER_00: And, yeah,
• 57:28
  SPEAKER_00: I think we are still, like,
• 57:29
  SPEAKER_00: we are aggressively, like,
• 57:31
  SPEAKER_00: working on this whole bunch of ideas that we have,
• 57:33
  SPEAKER_00: but we want to make sure, like,
• 57:34
  SPEAKER_00: you know,
• 57:35
  SPEAKER_00: as we are ready for the influx of PRs, we,
• 57:38
  SPEAKER_00: in fact, had to, like,
• 57:39
  SPEAKER_00: add internal code read for our CI system because year over year,
• 57:43
  SPEAKER_00: the number of CI jobs also doubled or more than doubled.
• 57:47
  SPEAKER_00: So we had to, like, also, like,
• 57:48
  SPEAKER_00: beef up a whole bunch of foundations in this space so that we can be ready for
• 57:54
  SPEAKER_00: the influx of new PRs.
• 57:57
  SPEAKER_01: Yeah,
• 57:57
  SPEAKER_01: and I guess you're also seeing that there's no way engineers can review all PRs with
• 58:03
  SPEAKER_01: the same rigor as we did in the past.
• 58:05
  SPEAKER_01: So I guess you're building tools to signal which ones to pay more attention to,
• 58:10
  SPEAKER_01: which ones might be low risk in so many ways,
• 58:13
  SPEAKER_01: maybe in impact, maybe in blast radius.
• 58:15
  SPEAKER_01: What is your take on what will happen with code review over time?
• 58:19
  SPEAKER_01: Do you think there will always be human code review or for some things it might go away?
• 58:23
  SPEAKER_00: Yeah, absolutely.
• 58:24
  SPEAKER_00: See I think one is for us like we want to err always on the safer side because
• 58:31
  SPEAKER_00: the cost of an incident for uber is like really high so even with all
• 58:37
  SPEAKER_00: of this tooling we like right now we still require an engineer to review Every PR
• 58:43
  SPEAKER_00: that goes in,
• 58:45
  SPEAKER_00: once the more confidence we get in our risk scoring,
• 58:50
  SPEAKER_00: and we can also backtest it with years of data, right?
• 58:53
  SPEAKER_00: We have all of this data,
• 58:55
  SPEAKER_00: and the more confidence we have in our AI code reviews,
• 58:58
  SPEAKER_00: then maybe for some class of low risk PRs,
• 59:02
  SPEAKER_00: we could enable, we call it green lane, like auto landing.
• 59:08
  SPEAKER_00: But we want to be really,
• 59:09
  SPEAKER_00: really sure about all of the dimensions that we want before we go there.
• 59:13
  SPEAKER_00: But right now,
• 59:13
  SPEAKER_00: the main priority is how do we make it really easy for engineers to get all
• 59:17
  SPEAKER_00: the signals they need to review faster.
• 59:20
  SPEAKER_00: And if we do a good job there,
• 59:22
  SPEAKER_00: then I'm sure we can increase the percentage of Autoline PRs over time.
• 59:27
  SPEAKER_00: But that's kind of P1, P0 is first getting all the signals in the right place.
• 59:31
  SPEAKER_01: Nice.
• 59:32
  SPEAKER_01: Yeah, but CodeInbox is nice.
• 59:33
  SPEAKER_01: I like it.
• 59:34
  SPEAKER_00: This is a thing that we should touch upon is like maybe how I can do sanity testing.
• 59:39
  SPEAKER_01: And sanity testing,
• 59:40
  SPEAKER_01: just for those who don't know the internal lingo, that's the manual testing, right?
• 59:44
  SPEAKER_00: Yeah, manual testing.
• 59:45
  SPEAKER_00: Yeah, I want to do, let's say I'm working on some new feature.
• 59:49
  SPEAKER_00: It touches two or three services.
• 59:52
  SPEAKER_00: How do I know if this feature is working end-to-end?
• 59:55
  SPEAKER_00: So then typical workflow for engineers is like pre-agent equalities.
• 1:00:00
  SPEAKER_00: We call it Slate.
• 1:00:02
  SPEAKER_00: Essentially,
• 1:00:05
  SPEAKER_00: you can create your own instance that deploys the branch that you are working on
• 1:00:12
  SPEAKER_00: and you can route your test account UID directly.
• 1:00:15
  SPEAKER_00: To the instance that you created.
• 1:00:17
  SPEAKER_01: So when you log in you are executing that branch.
• 1:00:20
  SPEAKER_00: Exactly, that branch and rest of the upstream and downstream goes to the other staging setups.
• 1:00:26
  SPEAKER_00: So that way you can test end-to-end without worrying about other parts of
• 1:00:32
  SPEAKER_00: the infrastructure and this will carry forward across the service graph.
• 1:00:36
  SPEAKER_00: So you can say from your mobile app or this is the test account I want
• 1:00:41
  SPEAKER_00: to test it will go through other services and when it touches the service under test
• 1:00:47
  SPEAKER_00: then it will go to your slate instance if you're testing two or three
• 1:00:50
  SPEAKER_00: then it will go to the service under test across all two or three
• 1:00:53
  SPEAKER_00: so you can do even multi-service tests with this now that we had this we we had
• 1:00:59
  SPEAKER_00: to just teach agents how to deploy slate And we also give it all of the past APIs
• 1:01:07
  SPEAKER_00: that you can use.
• 1:01:09
  SPEAKER_00: Oh, nice.
• 1:01:10
  SPEAKER_01: Basically,
• 1:01:11
  SPEAKER_01: this was a thing where I as an engineer wanted to sanity test,
• 1:01:15
  SPEAKER_01: meaning I want to manual test up my changes.
• 1:01:17
  SPEAKER_01: I had a way to do this in a pretty cool way.
• 1:01:20
  SPEAKER_01: And now we can just give it to agents
• 1:01:21
  SPEAKER_01: and they can actually test their own changes end to end in a safe way.
• 1:01:26
  SPEAKER_00: Yeah so here we are showing a web version of that and you can see in fact like
• 1:01:31
  SPEAKER_00: the this is like the CLI pattern we went
• 1:01:34
  SPEAKER_00: through it's using slate mcp create new slate environment and it's saying this is
• 1:01:39
  SPEAKER_00: the environment name and then once it creates
• 1:01:42
  SPEAKER_00: the environment it's saying deploy service to this environment and giving
• 1:01:45
  SPEAKER_00: the service name this is a test service and now
• 1:01:50
  SPEAKER_00: It has its own environment.
• 1:01:53
  SPEAKER_00: This is like the Slate UI.
• 1:01:56
  SPEAKER_00: And if you see here, you can pin test account UIDs to that particular instance.
• 1:02:03
  SPEAKER_00: And you will see all the services that are in that particular environment.
• 1:02:06
  SPEAKER_00: You can see the Gitsha and...
• 1:02:09
  SPEAKER_00: Now, the agent internally can use all of these skills and tools.
• 1:02:16
  SPEAKER_00: So if you see what it did was, it did A-B testing of that web thing.
• 1:02:22
  SPEAKER_00: It ran agent browser in that slate environment with what is on main and what is on my branch.
• 1:02:29
  SPEAKER_00: And it created a report for me to review.
• 1:02:33
  SPEAKER_00: The full set of changes that happened and so then like after this we can go to
• 1:02:40
  SPEAKER_00: the like a report that it generates like see generated a report that shows before
• 1:02:46
  SPEAKER_00: and after and showing me like all that and there's no regressions
• 1:02:50
  SPEAKER_00: and this is very easy for like you know engineers to validate and we can attach some
• 1:02:54
  SPEAKER_00: of the screenshots and videos to to the PR and the same thing we have
• 1:03:00
  SPEAKER_00: for iOS Android web And if you want to do back-end APIs,
• 1:03:04
  SPEAKER_00: then you can also test the back-end APIs.
• 1:03:07
  SPEAKER_01: I just wanted to add one of the things that, again,
• 1:03:11
  SPEAKER_01: you might not appreciate if you're not outside of Uber,
• 1:03:12
  SPEAKER_01: but Slate is so important because Uber literally today still has thousands
• 1:03:16
  SPEAKER_01: of microservices,
• 1:03:17
  SPEAKER_01: or maybe not micro,
• 1:03:19
  SPEAKER_01: but to run a change on just one service when you want to do end-to-end,
• 1:03:22
  SPEAKER_01: that's just difficult, right?
• 1:03:24
  SPEAKER_01: This is why Slate took years to build to start with.
• 1:03:27
  SPEAKER_00: Yeah, and you also have to think about the...
• 1:03:31
  SPEAKER_00: Routing of the pinned accounts.
• 1:03:35
  SPEAKER_01: This is the test account that I want you to route to my branch that is just deployed
• 1:03:41
  SPEAKER_01: in a staging environment.
• 1:03:42
  SPEAKER_01: That's not trivial at all.
• 1:03:45
  SPEAKER_00: In fact, how this works is one of the things that we We built back in maybe 2017-18.
• 1:03:51
  SPEAKER_00: It's called Jaeger, which is a distributed tracing infrastructure.
• 1:03:57
  SPEAKER_00: It's open source now.
• 1:03:59
  SPEAKER_00: And all of our services have Jaeger spans.
• 1:04:02
  SPEAKER_00: Essentially, wherever the call originates, there's a parent span.
• 1:04:06
  SPEAKER_00: And at every hop, there's a child span.
• 1:04:09
  SPEAKER_00: And there is a,
• 1:04:10
  SPEAKER_00: you can add any baggage on that trace,
• 1:04:14
  SPEAKER_00: which can be propagated across the call stack.
• 1:04:17
  SPEAKER_00: So when you pin from the slate web,
• 1:04:21
  SPEAKER_00: we are essentially adding,
• 1:04:22
  SPEAKER_00: saying that for these are the Services under test and this is the account
• 1:04:26
  SPEAKER_00: that I want to route.
• 1:04:27
  SPEAKER_00: So as it is going through our service mesh,
• 1:04:30
  SPEAKER_00: service mesh knows this baggage and then it can say, oh,
• 1:04:33
  SPEAKER_00: I want to route this request not to this environment, but to the slate environment.
• 1:04:38
  SPEAKER_00: So because we had tracing built out and because it's like propagated across
• 1:04:42
  SPEAKER_00: the stack,
• 1:04:43
  SPEAKER_00: now you could actually test and we can reliably route these kind of test requests to
• 1:04:47
  SPEAKER_00: the right environments so that it makes it really easy to test.
• 1:04:50
  SPEAKER_01: This thing took years to make sure that all existing services were retrofitted
• 1:04:54
  SPEAKER_01: with this and all new services were built with this in mind.
• 1:04:57
  SPEAKER_01: I'm starting to sense a trend where Uber built stuff over years,
• 1:05:02
  SPEAKER_01: pre-AI, just infrastructure that is easy to test for humans or easy to automate.
• 1:05:09
  SPEAKER_01: And now with AI,
• 1:05:10
  SPEAKER_01: it seems like, oh,
• 1:05:11
  SPEAKER_01: you just built this the last year,
• 1:05:13
  SPEAKER_01: but actually you built on top a foundation that was built for years, right?
• 1:05:17
  SPEAKER_00: Yeah.
• 1:05:17
  SPEAKER_00: I'm sure I think if we did not have system-like slate,
• 1:05:21
  SPEAKER_00: it would have been really hard for agents to self-validate.
• 1:05:24
  SPEAKER_00: Then you cannot automate this thing.
• 1:05:26
  SPEAKER_00: And as you said, this kind of stuff really helps.
• 1:05:32
  SPEAKER_00: And I'll also talk about one more pretty interesting thing, which is so uber-specific.
• 1:05:37
  SPEAKER_00: What is Capri?
• 1:05:39
  SPEAKER_00: Yeah.
• 1:05:39
  SPEAKER_00: And this is like very, very hard to get right.
• 1:05:42
  SPEAKER_00: And like,
• 1:05:43
  SPEAKER_00: we had like even many versions of build like this in the past,
• 1:05:47
  SPEAKER_00: but I think this is probably our best version and we can like really scale it to all
• 1:05:52
  SPEAKER_00: the teams.
• 1:05:52
  SPEAKER_00: But essentially,
• 1:05:54
  SPEAKER_00: the idea is like context,
• 1:05:57
  SPEAKER_00: because we have, even if you look at our core Rider app, right?
• 1:05:59
  SPEAKER_00: Like we have hundreds of screens.
• 1:06:01
  SPEAKER_01: And the Rider app is what most people think is Uber app, right?
• 1:06:04
  SPEAKER_01: Yeah, Uber app, exactly.
• 1:06:06
  SPEAKER_00: Yeah, like we have different business,
• 1:06:08
  SPEAKER_00: different product lines, different cities, different like whole bunch of dimensions.
• 1:06:12
  SPEAKER_00: Like if I want to test any new thing,
• 1:06:14
  SPEAKER_00: how do I know every combination of these has no regression?
• 1:06:19
  SPEAKER_00: And if you were to like create all permutations across this,
• 1:06:22
  SPEAKER_00: you would have to maintain thousands and thousands of tests across the stack.
• 1:06:27
  SPEAKER_00: And that alone is like, again, really hard problem in itself.
• 1:06:31
  SPEAKER_00: So for some
• 1:06:32
  SPEAKER_00: of our most complex platforms like we call it Capture Replay like essentially you
• 1:06:37
  SPEAKER_00: can capture real production traffic with just one middleware
• 1:06:41
  SPEAKER_00: because now we have standardized language frameworks across all
• 1:06:44
  SPEAKER_00: the services is very easy to hook this into and It will capture not just the inbound
• 1:06:52
  SPEAKER_00: and outbound at that service layer,
• 1:06:53
  SPEAKER_00: but also any of the downstream APIs that service calls,
• 1:06:58
  SPEAKER_00: those requests and those responses are also captured.
• 1:07:00
  SPEAKER_00: So in some sense,
• 1:07:01
  SPEAKER_00: you have like full request response and all
• 1:07:04
  SPEAKER_00: of your outbound request responses captured.
• 1:07:07
  SPEAKER_00: Now, with all of these captured requests,
• 1:07:09
  SPEAKER_00: they go through some pipeline and then you can build simple rules that says, oh,
• 1:07:14
  SPEAKER_00: if you see this kind of field in the request, this is the value for that dimension.
• 1:07:20
  SPEAKER_00: So then you can capture all kinds of dimensions that that platform team cares about.
• 1:07:26
  SPEAKER_00: And once you have all the requests and all the dimensions,
• 1:07:28
  SPEAKER_00: you can in some sense generate the full set of live dimensions and live combinations
• 1:07:34
  SPEAKER_00: that that platform is currently supporting,
• 1:07:37
  SPEAKER_00: which we call it MatchGuard that I can run pre-deploy in our CI CD that shows
• 1:07:44
  SPEAKER_00: for all of the PRs that are going in this deploy,
• 1:07:47
  SPEAKER_00: let me validate against The full combinatorial set
• 1:07:51
  SPEAKER_00: of like combinations this platform supports and show the data before and after.
• 1:07:57
  SPEAKER_00: So then you get really high confidence if am I breaking maybe cash trips only
• 1:08:03
  SPEAKER_00: in Brazil with this configuration,
• 1:08:05
  SPEAKER_00: which would have been really hard for engineers to manually test or agents also
• 1:08:08
  SPEAKER_00: to manually test.
• 1:08:10
  SPEAKER_01: So step one is you capture what you want to capture and then you have the system
• 1:08:13
  SPEAKER_01: that will rerun it, check that there's regressions.
• 1:08:17
  SPEAKER_00: Yeah.
• 1:08:17
  SPEAKER_00: So like then I'm going to one of the domains that has this enabled
• 1:08:22
  SPEAKER_00: so you can see all the sessions that it captures.
• 1:08:25
  SPEAKER_00: I can go to any of the sessions.
• 1:08:27
  SPEAKER_00: Then you can see all of the metadata and tags.
• 1:08:30
  SPEAKER_00: So essentially it's saying, oh, this request is tagged as marketplace type eats.
• 1:08:34
  SPEAKER_00: It's tagged as line of business eats.
• 1:08:37
  SPEAKER_00: It is in this kind of terminal state.
• 1:08:39
  SPEAKER_00: It knows captured all the requests.
• 1:08:42
  SPEAKER_00: You can also see all of the outbounds that it used.
• 1:08:45
  SPEAKER_00: If we click on that, it will also show you The request and response for the outbounds.
• 1:08:50
  SPEAKER_00: And if you see at the top, there is slate workflow UID or search environment name.
• 1:08:55
  SPEAKER_00: So then let's say if someone,
• 1:08:58
  SPEAKER_00: if I'm working on a bug that's only affecting for some reason,
• 1:09:01
  SPEAKER_00: San Francisco, Uber Eats trips, I fix that bug.
• 1:09:06
  SPEAKER_00: Then I can create a slate environment for that bug for my code changes.
• 1:09:11
  SPEAKER_00: I can go here and then say replay Just this request on my slate environment
• 1:09:18
  SPEAKER_00: and show me the delta here.
• 1:09:20
  SPEAKER_01: Show me the difference and you're expecting a fix or some change.
• 1:09:23
  SPEAKER_00: Yeah, I'm expecting the fix because I know the before request,
• 1:09:25
  SPEAKER_00: after request, what I'm expecting.
• 1:09:28
  SPEAKER_00: And then I can do it for a specific dimension or I can use it.
• 1:09:34
  SPEAKER_00: And then we also generate what we mentioned like match guard,
• 1:09:37
  SPEAKER_00: which is the full set of dimensions possible for that.
• 1:09:40
  SPEAKER_00: So then you can create like a When I was there,
• 1:09:43
  SPEAKER_00: we didn't have this,
• 1:09:44
  SPEAKER_00: but we have something called Deputy and Cerberus,
• 1:09:46
  SPEAKER_00: but those were, I guess, very early versions.
• 1:10:00
  SPEAKER_00: Yeah, so in fact,
• 1:10:01
  SPEAKER_00: what Deputy and Cerberus allowed was ability to route
• 1:10:07
  SPEAKER_00: any request from production
• 1:10:10
  SPEAKER_00: to your testing instance and Cerberus allowed from your local testing instance back
• 1:10:16
  SPEAKER_00: to the staging servers.
• 1:10:19
  SPEAKER_00: But now that we have Slate,
• 1:10:22
  SPEAKER_00: we don't necessarily have to use Deputy and Cerberus because you can deploy
• 1:10:26
  SPEAKER_00: a Slate environment and we don't need to use any of them.
• 1:10:29
  SPEAKER_01: I just want to stress because I think it's hard to see from the outside,
• 1:10:32
  SPEAKER_01: but that's really hard to build this because every single, again,
• 1:10:35
  SPEAKER_01: you have thousands of services,
• 1:10:37
  SPEAKER_01: all of them have to support these things from routing,
• 1:10:40
  SPEAKER_01: for example, to capturing production, to replaying, right?
• 1:10:44
  SPEAKER_01: Like this was a lot of work, I assume.
• 1:10:46
  SPEAKER_00: Yeah, and now I think the same,
• 1:10:48
  SPEAKER_00: like similar to how we exposed like slate to agents,
• 1:10:51
  SPEAKER_00: we can expose all of the Capri functionality to agents so agents can check,
• 1:10:55
  SPEAKER_00: oh, I'm trying to fix this.
• 1:10:57
  SPEAKER_00: Let me get all of the sessions of this type and then see what request looks like.
• 1:11:01
  SPEAKER_00: Can I compose a request of that type on my slate instance and see did it fix or did it not fix?
• 1:11:07
  SPEAKER_00: Let me go in a loop until I fix this.
• 1:11:09
  SPEAKER_01: Yeah, yeah.
• 1:11:10
  SPEAKER_01: And now the agent can,
• 1:11:12
  SPEAKER_01: even though it cannot see,
• 1:11:13
  SPEAKER_01: but it can see the delta and it can tell that nothing has changed, for example.
• 1:11:16
  SPEAKER_01: So it can get feedback loops.
• 1:11:18
  SPEAKER_00: Exactly, exactly.
• 1:11:19
  SPEAKER_00: I think that helps a lot.
• 1:11:22
  SPEAKER_00: Now, the agent can also capture replay on Slate with any of these.
• 1:11:27
  SPEAKER_00: We already talked about the Edge Gateway.
• 1:11:29
  SPEAKER_01: Edge Gateway, that's just kind of Uber's internal API, right?
• 1:11:32
  SPEAKER_01: It used to be RT API, real-time API, but now it's called Edge.
• 1:11:37
  SPEAKER_01: It's a Go version in Go, if I remember, right?
• 1:11:40
  SPEAKER_00: Yeah, it's in Go and now it supports a whole bunch of endpoint types.
• 1:11:44
  SPEAKER_00: So there's a sectional API where there's gRPC, webhooks, GraphQL.
• 1:11:48
  SPEAKER_00: So again,
• 1:11:49
  SPEAKER_00: all APIs go through one single entry point where we can build a whole bunch
• 1:11:54
  SPEAKER_00: of tooling around that on monitoring, on alerting,
• 1:11:59
  SPEAKER_00: And like even there is like a like where there's all the reliability features.
• 1:12:05
  SPEAKER_00: The Edge also has its own capture replay similar to the Capri.
• 1:12:09
  SPEAKER_00: It's using Capri behind the scenes.
• 1:12:11
  SPEAKER_00: And like if you want to look at We also automatically tagged
• 1:12:16
  SPEAKER_00: which service service service service service.
• 1:12:29
  SPEAKER_00: Which is a required dependency, which is an optional dependency.
• 1:12:32
  SPEAKER_00: So that metadata is also there.
• 1:12:34
  SPEAKER_00: So as we are even looking at alerts and diagnosing alerts,
• 1:12:39
  SPEAKER_00: all of this rich context really helps both engineers to debug faster and also agents
• 1:12:43
  SPEAKER_00: to debug faster.
• 1:12:44
  SPEAKER_00: Yeah, we already talked about experiments.
• 1:12:47
  SPEAKER_00: The last thing we will cover is like what we call Shepard and Fox.
• 1:12:53
  SPEAKER_00: After we moved to Monorepo,
• 1:12:54
  SPEAKER_00: like then we moved to Monorepo because upgrading libraries was very hard, right?
• 1:12:58
  SPEAKER_01: On all of the different repos, yeah.
• 1:13:01
  SPEAKER_00: Exactly.
• 1:13:01
  SPEAKER_00: Then you move to monorepo,
• 1:13:02
  SPEAKER_00: but then now someone has to go and upgrade this library usage across all of
• 1:13:08
  SPEAKER_00: the modules in that repository.
• 1:13:11
  SPEAKER_00: So how do we automate that?
• 1:13:12
  SPEAKER_00: So we first took a stab at using open rewrite policies.
• 1:13:17
  SPEAKER_00: If it's a simple signature change, you can write that in open rewrite policy.
• 1:13:23
  SPEAKER_00: And then Shepard knows what are the unique modules owned by different teams.
• 1:13:28
  SPEAKER_00: And then it automatically creates PR for each of those modules.
• 1:13:32
  SPEAKER_00: When AI came into picture,
• 1:13:34
  SPEAKER_00: then we could add like a AI-based shepherd where, okay,
• 1:13:38
  SPEAKER_00: if I want to do fixed context propagation across all the services so
• 1:13:42
  SPEAKER_00: that tracing works better,
• 1:13:43
  SPEAKER_00: I can create set of prompts and skills that then agent can run across every module
• 1:13:48
  SPEAKER_00: and fix all the issues in every service faster.
• 1:13:51
  SPEAKER_00: So this is kind of a large scale Changes whether you want deterministic changes or
• 1:13:56
  SPEAKER_00: whether you want like agentic changes across the modules
• 1:14:00
  SPEAKER_00: that gave us like we were able to do a lot of migrations over the years through this
• 1:14:05
  SPEAKER_00: and we call each one as a campaign and if you see J unit 4 to So JUnit 5,
• 1:14:09
  SPEAKER_00: there will be a YAML file with the policy and you'll see a whole bunch
• 1:14:13
  SPEAKER_00: of PRs created for each module and a campaign owner can track it,
• 1:14:19
  SPEAKER_00: the status of that and go through that.
• 1:14:21
  SPEAKER_00: So it makes it really easy to manage these migrations.
• 1:14:24
  SPEAKER_00: A few months ago, then we took the same platform and added like one more feature.
• 1:14:29
  SPEAKER_00: We also want to make it easy.
• 1:14:31
  SPEAKER_00: We want to continuously improve the code base, not one-time migrations.
• 1:14:35
  SPEAKER_00: Let's say I want to...
• 1:14:36
  SPEAKER_01: So like code maintenance?
• 1:14:37
  SPEAKER_00: Yeah, code maintenance.
• 1:14:38
  SPEAKER_00: So we call it Fox.
• 1:14:39
  SPEAKER_00: Essentially,
• 1:14:40
  SPEAKER_00: we want to create self-improving code bases since we already have a whole bunch
• 1:14:45
  SPEAKER_00: of agent skills.
• 1:14:46
  SPEAKER_00: Essentially,
• 1:14:47
  SPEAKER_00: any service at Uber,
• 1:14:48
  SPEAKER_00: they can go here and enroll their service into different skills.
• 1:14:51
  SPEAKER_01: They can say,
• 1:14:53
  SPEAKER_00: I want dead cloud cleanup or performance analysis or noisy logs.
• 1:14:56
  SPEAKER_00: There's a whole bunch of these skills that you can onboard to.
• 1:15:00
  SPEAKER_00: Oh, smart.
• 1:15:02
  SPEAKER_00: And what Fox does is every Monday,
• 1:15:05
  SPEAKER_00: essentially every Sunday,
• 1:15:08
  SPEAKER_00: when we don't have that much CI load,
• 1:15:10
  SPEAKER_00: it will generate a new PR for this service using that code cleanup.
• 1:15:16
  SPEAKER_00: Can I clean up any code?
• 1:15:17
  SPEAKER_00: Can I improve the performance of this?
• 1:15:19
  SPEAKER_00: Can I fix any noisy logs?
• 1:15:21
  SPEAKER_00: So it's kind of automated then code base improvement.
• 1:15:26
  SPEAKER_00: I like
• 1:15:30
  SPEAKER_01: it because I heard people do this thing where they would set an AI loop,
• 1:15:35
  SPEAKER_01: let's say in cloud code or codec saying,
• 1:15:38
  SPEAKER_01: look for ways to improve my code base and they would execute it every night or once
• 1:15:41
  SPEAKER_01: a week.
• 1:15:42
  SPEAKER_01: But it sounds like this is just a lot more optimized.
• 1:15:46
  SPEAKER_01: And I love that you can enroll your code base because a code base is owned by
• 1:15:52
  SPEAKER_01: an engineer or team at Uber, right?
• 1:15:54
  SPEAKER_00: Yeah, exactly.
• 1:15:55
  SPEAKER_00: And you can see, because you're rapidly scaling, we are still in the opt-in phase.
• 1:16:02
  SPEAKER_00: Once we improve some of the skills, we'll move some of them to opt-out phase.
• 1:16:07
  SPEAKER_00: But any time you obtain a skill for a service,
• 1:16:10
  SPEAKER_00: then every week,
• 1:16:11
  SPEAKER_00: in some sense, you are getting a PR created, like a managed agentic loop.
• 1:16:18
  SPEAKER_00: So then you don't have to like, you are continuously improving this.
• 1:16:22
  SPEAKER_00: In some sense,
• 1:16:23
  SPEAKER_00: the more toil we reduce,
• 1:16:25
  SPEAKER_00: the more amount of time engineers can spend on the true feature work.
• 1:16:28
  SPEAKER_00: So like constantly thinking of,
• 1:16:30
  SPEAKER_00: okay, how do I eliminate more toil from engineers and automate all of these things.
• 1:16:35
  SPEAKER_00: And so we constantly look at all kinds of PRs with maintenance category PRs.
• 1:16:42
  SPEAKER_00: Do we need to create a new skill?
• 1:16:43
  SPEAKER_00: Do we need to improve that skill so that That class of PRs can be onboarded
• 1:16:48
  SPEAKER_00: to this automated manage loop.
• 1:16:50
  SPEAKER_01: This is nice.
• 1:16:51
  SPEAKER_01: I like it.
• 1:16:51
  SPEAKER_01: And then finally, I heard about wisdom.
• 1:16:53
  SPEAKER_01: What is wisdom?
• 1:16:55
  SPEAKER_00: Oh, yeah.
• 1:16:56
  SPEAKER_00: So we use wisdom because all of our employees dog fooder products.
• 1:17:03
  SPEAKER_00: So if you're using a rider app or earner app,
• 1:17:07
  SPEAKER_00: and if you see any bug, then you can, from the app, report that bug.
• 1:17:12
  SPEAKER_00: We now have all kinds of AI stuff where like it will automatically route it to
• 1:17:16
  SPEAKER_00: the right team based on the stuff and then when you report a bug because it's from
• 1:17:22
  SPEAKER_00: the employee like beta apps we can capture whole bunch of logs and telemetry around
• 1:17:27
  SPEAKER_00: that that makes it really easy for the team owning that feature to fix it so
• 1:17:33
  SPEAKER_00: In this case, it's like an internal feature.
• 1:17:35
  SPEAKER_00: It's essentially like if you want to test any feature in a different country,
• 1:17:39
  SPEAKER_00: you can mock the location on the device.
• 1:17:41
  SPEAKER_00: So if someone reported that issue, the first thing
• 1:17:47
  SPEAKER_00: Now how do you reproduce that before it is it used to be a huge problem to reproduce
• 1:17:51
  SPEAKER_00: so now we know that the set of screens the user has gone through because we have
• 1:17:57
  SPEAKER_00: that in the logs and then we can give the engineers like this is
• 1:18:01
  SPEAKER_00: the reproduction steps and this is the expected behavior and this is
• 1:18:04
  SPEAKER_00: the observed behavior and so that we can boil down to the issue quickly And all of
• 1:18:11
  SPEAKER_00: the telemetry that we collected from the app,
• 1:18:13
  SPEAKER_00: you can see all the logs in one place,
• 1:18:15
  SPEAKER_00: whether it's analytics, network requests, all kinds of feature flags.
• 1:18:21
  SPEAKER_00: So that also gives a lot of context.
• 1:18:23
  SPEAKER_00: And now that we have all of this context,
• 1:18:25
  SPEAKER_00: we build a managed agent called DebugAssist that looks all of this,
• 1:18:29
  SPEAKER_00: looks at the code base,
• 1:18:30
  SPEAKER_00: and it will try to pinpoint what is the root cause and what is the impact,
• 1:18:35
  SPEAKER_00: evidence timeline around that.
• 1:18:38
  SPEAKER_00: Now, once I think, okay,
• 1:18:40
  SPEAKER_00: this is the right root cause, I can hand off this to Minion that we talked about.
• 1:18:48
  SPEAKER_00: This root cause looks good.
• 1:18:49
  SPEAKER_00: Like, why don't you create a PR for this?
• 1:18:51
  SPEAKER_00: Minion will take all of that context that we have.
• 1:18:54
  SPEAKER_00: It creates a PR.
• 1:18:55
  SPEAKER_00: It will attach that PR to the Jira task.
• 1:18:58
  SPEAKER_00: The Jira task also has the root cause analysis.
• 1:19:00
  SPEAKER_00: So anyone who's working,
• 1:19:01
  SPEAKER_00: they know The root cause analysis,
• 1:19:03
  SPEAKER_00: if someone updated the root cause on the comments or like a new PR, again,
• 1:19:08
  SPEAKER_00: all of that is data that we can then figure out,
• 1:19:10
  SPEAKER_00: OK, how do we improve the debug assist?
• 1:19:12
  SPEAKER_00: How do we improve minion?
• 1:19:13
  SPEAKER_00: We didn't get the right RCA or we didn't get the right PR from the RCA.
• 1:19:17
  SPEAKER_00: Like both will be able to continuously improve.
• 1:19:20
  SPEAKER_01: And so do I understand correctly that you can capture and reproduce bugs so much easier, right?
• 1:19:25
  SPEAKER_00: Yeah, and we also added ability to,
• 1:19:27
  SPEAKER_00: because for the employee reported bugs,
• 1:19:30
  SPEAKER_00: we have all the logs,
• 1:19:32
  SPEAKER_00: I can take my test device and I can ask it to simulate what
• 1:19:36
  SPEAKER_00: that employee has experienced,
• 1:19:38
  SPEAKER_00: so then Once I give the bug UID,
• 1:19:43
  SPEAKER_00: we can essentially serve the same network responses that we served to the person
• 1:19:49
  SPEAKER_00: who filed so that I can experience the same issue in my test device app too.
• 1:19:55
  SPEAKER_00: So that way,
• 1:19:56
  SPEAKER_00: then once you fix it,
• 1:19:57
  SPEAKER_00: then I can see this actually got fixed and then it makes it really easy
• 1:20:01
  SPEAKER_00: to reproduce.
• 1:20:03
  SPEAKER_01: I love the recording of the thing because that's so important.
• 1:20:07
  SPEAKER_01: It's hard enough to do even on the web.
• 1:20:09
  SPEAKER_01: You need to build custom tools for yourself if you want to do a website.
• 1:20:12
  SPEAKER_01: But it's awesome that you did it.
• 1:20:14
  SPEAKER_00: Yeah, I think this helps a lot like then in reproducibility and like getting to
• 1:20:19
  SPEAKER_00: the right root cause.
• 1:20:20
  SPEAKER_01: So you built all of these systems and especially last year,
• 1:20:24
  SPEAKER_01: it seems so many brand new systems that are just doing things like,
• 1:20:27
  SPEAKER_01: you know, they're maintaining the cold quality agents are validating themselves.
• 1:20:33
  SPEAKER_01: How do you think the job of software engineers inside of Uber is changing?
• 1:20:38
  SPEAKER_01: And what are you learning from all of these new tools?
• 1:20:41
  SPEAKER_01: It seems like you're learning yourself.
• 1:20:43
  SPEAKER_00: You know like in some sense like if you're
• 1:20:45
  SPEAKER_00: the agentic AI world like you're constantly trying to find the next bottleneck
• 1:20:50
  SPEAKER_00: and trying to like remove that and figuring out how you can like both using AI
• 1:20:56
  SPEAKER_00: and both using like both systems engineering like reduce that bottleneck
• 1:21:01
  SPEAKER_00: because ultimately our goal is to maximize the number of features that we can ship
• 1:21:05
  SPEAKER_00: across all of our products and that will require us to look at
• 1:21:09
  SPEAKER_00: the entire life cycle How do we rewire that for the new world and what kind of tools
• 1:21:17
  SPEAKER_00: that we need to build and I think this is going to be like an ongoing journey right
• 1:21:20
  SPEAKER_00: and there's a lot of discussion of you know this concept called capability overhang
• 1:21:24
  SPEAKER_00: where I feel like even right now like there's so many things
• 1:21:28
  SPEAKER_00: that we could you leverage the latest models for but we don't have
• 1:21:33
  SPEAKER_00: the Sufficient orchestrating capabilities or the harnesses or the tools around that
• 1:21:39
  SPEAKER_00: to leverage those models in the right places in the right way.
• 1:21:42
  SPEAKER_00: I feel like the more we work on this,
• 1:21:44
  SPEAKER_00: the more we can unlock,
• 1:21:47
  SPEAKER_00: the more features we can ship, the more productivity that we can unlock.
• 1:21:51
  SPEAKER_00: And we are trying to measure a whole bunch of this and constantly think
• 1:21:55
  SPEAKER_00: about what is the next set of foundations that we need to build to make it easy
• 1:21:59
  SPEAKER_00: for engineers to work and to make it easy for agents to work.
• 1:22:02
  SPEAKER_01: You mentioned productivity.
• 1:22:03
  SPEAKER_01: How do you think about engineering productivity?
• 1:22:06
  SPEAKER_01: Because when I was at Uber, this was again up to 2020, us managers did look at things.
• 1:22:11
  SPEAKER_01: It wasn't the main thing,
• 1:22:11
  SPEAKER_01: but we did look at things like diffs per engineer just to show
• 1:22:14
  SPEAKER_01: if someone didn't contribute for like two weeks,
• 1:22:16
  SPEAKER_01: it was like, what are you doing here?
• 1:22:18
  SPEAKER_01: Or if someone did a lot of code reviews, that's actually what we wanted to notice.
• 1:22:21
  SPEAKER_01: But it sounds like a lot of those things,
• 1:22:23
  SPEAKER_01: the code output, the code artifacts are less relevant potentially.
• 1:22:28
  SPEAKER_01: What is a productive engineer in your mind right now inside of Uber or inside of your team?
• 1:22:33
  SPEAKER_00: Yeah, so I think before the AI world,
• 1:22:35
  SPEAKER_00: we were measuring the standard stuff like code throughput and stuff like that.
• 1:22:39
  SPEAKER_00: Those are the easy ones.
• 1:22:40
  SPEAKER_00: And then we also have every few months, we do the developer experience surveys.
• 1:22:44
  SPEAKER_00: We look at NPS, sentiment across all kinds of dimensions.
• 1:22:49
  SPEAKER_00: We look at our code quality dimensions,
• 1:22:51
  SPEAKER_00: what's our incidence per def,
• 1:22:53
  SPEAKER_00: what's our code coverage, and all of these metrics, right?
• 1:22:57
  SPEAKER_00: I think now,
• 1:22:58
  SPEAKER_00: once you go to AI,
• 1:22:59
  SPEAKER_00: I feel like we can leverage the new set of capabilities to get more nuanced
• 1:23:05
  SPEAKER_00: into this discussion.
• 1:23:07
  SPEAKER_00: Because now,
• 1:23:07
  SPEAKER_00: we are internally talking about not necessarily just diff velocity,
• 1:23:11
  SPEAKER_00: but feature velocity.
• 1:23:12
  SPEAKER_00: Like, how much of all of this additional throughput is going towards unlocking new features?
• 1:23:18
  SPEAKER_00: And how can we even change our organizational processes so that we can reduce
• 1:23:24
  SPEAKER_00: the P50 feature time?
• 1:23:25
  SPEAKER_00: Because Like,
• 1:23:26
  SPEAKER_00: you can create 100 divs,
• 1:23:28
  SPEAKER_00: but if they are not all feature divs,
• 1:23:29
  SPEAKER_00: then like,
• 1:23:30
  SPEAKER_00: maybe it's not necessarily increasing the value that we can provide to the users,
• 1:23:35
  SPEAKER_00: right?
• 1:23:35
  SPEAKER_00: So, like one,
• 1:23:36
  SPEAKER_00: for example,
• 1:23:37
  SPEAKER_00: one of the things we did is tagging all of our PRs to different categories,
• 1:23:41
  SPEAKER_00: like on complexity level,
• 1:23:43
  SPEAKER_00: and is it a feature PR,
• 1:23:44
  SPEAKER_00: bug fix,
• 1:23:46
  SPEAKER_00: maintenance,
• 1:23:47
  SPEAKER_00: testing,
• 1:23:47
  SPEAKER_00: documentation on different categories,
• 1:23:49
  SPEAKER_00: and we could see year over year, how is that mix changing?
• 1:23:52
  SPEAKER_00: Luckily, like, at least like right now, I can see Our percentage of feature PRs is going up.
• 1:23:58
  SPEAKER_00: Our percentage of maintenance PRs is going down.
• 1:24:01
  SPEAKER_00: So that gives us good indication that we're in the right direction.
• 1:24:05
  SPEAKER_00: We see the code volume go up and the percentage of PRs for features going up.
• 1:24:10
  SPEAKER_00: Then we are investing a lot.
• 1:24:13
  SPEAKER_00: Because before,
• 1:24:14
  SPEAKER_00: even though we had the U-Plan tool,
• 1:24:16
  SPEAKER_00: sometimes you might not have the right tracing from this product design dog
• 1:24:20
  SPEAKER_00: to this engine design dog to this particular task.
• 1:24:23
  SPEAKER_00: So it becomes a bit harder to measure what is the cycle time end to end reliably
• 1:24:27
  SPEAKER_00: across all the features.
• 1:24:28
  SPEAKER_00: So now we're using AI to automatically cluster these things
• 1:24:32
  SPEAKER_00: and tag any missing things so that we can reliably get what is
• 1:24:38
  SPEAKER_00: the P50 feature velocity for different complexity of features for every half over
• 1:24:44
  SPEAKER_00: the last two years, three years.
• 1:24:46
  SPEAKER_00: And then we can go into the bottleneck In the PRD phase,
• 1:24:50
  SPEAKER_00: is the bottleneck in the launch phase, is the bottleneck in the rollout phase?
• 1:24:54
  SPEAKER_00: What is going on there and what kind of things we need to invest to accelerate
• 1:24:59
  SPEAKER_00: the number of features that we can ship per half?
• 1:25:01
  SPEAKER_01: And how is your mental model changing of software engineering compared to pre-AI
• 1:25:05
  SPEAKER_01: and pre-Ds tools?
• 1:25:07
  SPEAKER_01: Is it still the same job?
• 1:25:08
  SPEAKER_01: Have things changed?
• 1:25:10
  SPEAKER_00: I mean, at least in my total tenure as in engineering,
• 1:25:14
  SPEAKER_00: there's never been this much change to how I work.
• 1:25:18
  SPEAKER_00: Like, for example, like,
• 1:25:19
  SPEAKER_00: you know, we say, like,
• 1:25:21
  SPEAKER_00: all the engineers at Uber are hands-on, like,
• 1:25:24
  SPEAKER_00: even distinguished engineers at Uber,
• 1:25:25
  SPEAKER_00: like, are hands-on, they, like, they create PRs.
• 1:25:28
  SPEAKER_00: And I used to think I was hands-on engineer,
• 1:25:30
  SPEAKER_00: but my hands-on definition year over year has completely changed.
• 1:25:33
  SPEAKER_00: I'm like, even personally, I'm doing like way more PRs than what I was doing last year.
• 1:25:38
  SPEAKER_00: Like it's more easy to do all of these things, all of these things, right?
• 1:25:42
  SPEAKER_00: And I'm able to like explore more number of ideas than what I could possibly do in the past.
• 1:25:48
  SPEAKER_00: But the job of the engineering is,
• 1:25:49
  SPEAKER_00: you know,
• 1:25:49
  SPEAKER_00: in some sense,
• 1:25:50
  SPEAKER_00: how do you build the right context for the agents to produce reliable output like
• 1:25:57
  SPEAKER_00: in some sense like you need to get really good at context engineering
• 1:26:00
  SPEAKER_00: before I was operating with some context to do some tasks so I'm always thinking
• 1:26:04
  SPEAKER_00: about okay if I want to offload some of this work to an agent what kind
• 1:26:08
  SPEAKER_00: of context it would need and how do I set up the environment with that right context
• 1:26:11
  SPEAKER_00: so that it can produce that work then you have to become really good
• 1:26:16
  SPEAKER_00: at like you know how do you give the right specs right constraints for
• 1:26:20
  SPEAKER_00: the agents apart from building the right context
• 1:26:23
  SPEAKER_00: then multitasking has become really like a thing before if you're like it would take
• 1:26:29
  SPEAKER_00: so much time to create a PR so like at least laser focused on that
• 1:26:33
  SPEAKER_00: but now you're constantly switching between so many things like you so you need
• 1:26:37
  SPEAKER_00: to become really good at multitasking across all
• 1:26:40
  SPEAKER_00: the sessions figuring how do you scope the how do you scope
• 1:26:42
  SPEAKER_00: the work how do you track the work How do you verify the work really quickly?
• 1:26:47
  SPEAKER_00: Because if you're not able to verify the work and pick any code smells from
• 1:26:52
  SPEAKER_00: the agent, then that's hard.
• 1:26:54
  SPEAKER_00: Then you're shipping some recreations into production, right?
• 1:26:58
  SPEAKER_00: So the context engineering,
• 1:27:01
  SPEAKER_00: ability to multitask,
• 1:27:03
  SPEAKER_00: ability to review and verify the outputs,
• 1:27:06
  SPEAKER_00: these things have become much more important now because you're able to parallelize
• 1:27:11
  SPEAKER_00: a lot more than before.
• 1:27:13
  SPEAKER_01: One thing I'm picking up is maybe a new primitive is we used
• 1:27:17
  SPEAKER_01: to build software thinking, how do I build this software and how do I verify it?
• 1:27:22
  SPEAKER_01: Whereas now the question that you said is often,
• 1:27:25
  SPEAKER_01: how do I set up an agent to be able to build this and to be able to verify it?
• 1:27:29
  SPEAKER_01: So it's kind of like almost a meta, like there's more of that, right?
• 1:27:34
  SPEAKER_00: Yeah, I'll give an interesting example.
• 1:27:35
  SPEAKER_00: I was working with some of the engineers that work on our customer support team.
• 1:27:42
  SPEAKER_00: We are going through the journey of having AI agents handle some
• 1:27:46
  SPEAKER_00: of our customer support policies.
• 1:27:47
  SPEAKER_00: That process would go from, oh, you have some runbook that human agents would follow.
• 1:27:53
  SPEAKER_00: So then from that runbook,
• 1:27:54
  SPEAKER_00: you are going through a whole bunch of steps to create a policy
• 1:27:58
  SPEAKER_00: that AI agent can follow.
• 1:28:00
  SPEAKER_00: Then you have to think about, okay,
• 1:28:01
  SPEAKER_00: there is some dynamic elements here,
• 1:28:02
  SPEAKER_00: then which APIs I need to call to get that data,
• 1:28:06
  SPEAKER_00: which attributes I need to use,
• 1:28:08
  SPEAKER_00: then I need to configure my system,
• 1:28:10
  SPEAKER_00: change bunch of configurations, then I need to do manual evals before launching.
• 1:28:15
  SPEAKER_00: Then I had to do continuous evals post-launch.
• 1:28:18
  SPEAKER_00: And then in the last few months, we were able to take down all of the steps.
• 1:28:22
  SPEAKER_00: How do I now take all of the context and all of the principles that I want to use
• 1:28:27
  SPEAKER_00: for the policy conversion and teach agents to be able to do that so that I have
• 1:28:31
  SPEAKER_00: a really good V0 policy.
• 1:28:33
  SPEAKER_00: That an agent can create.
• 1:28:35
  SPEAKER_00: Then you automate all of the configuration setup,
• 1:28:38
  SPEAKER_00: manual testing,
• 1:28:39
  SPEAKER_00: finding the right APIs,
• 1:28:40
  SPEAKER_00: finding the right properties so that we can hook the right dynamic elements
• 1:28:44
  SPEAKER_00: into this.
• 1:28:45
  SPEAKER_00: Again, like that used to take weeks,
• 1:28:47
  SPEAKER_00: now it's like order of Then,
• 1:28:50
  SPEAKER_00: instead of doing manual evals,
• 1:28:53
  SPEAKER_00: then how can you use all of the past context to create synthetic evals on top of
• 1:28:57
  SPEAKER_00: that so that you can accelerate, you can deploy this and then verify this quickly.
• 1:29:03
  SPEAKER_00: Essentially,
• 1:29:04
  SPEAKER_00: what we were discussing,
• 1:29:05
  SPEAKER_00: which is Now,
• 1:29:06
  SPEAKER_00: instead of doing that work,
• 1:29:09
  SPEAKER_00: how can you build the right tools and harnesses on top of this so
• 1:29:13
  SPEAKER_00: that you can automate the workflow?
• 1:29:15
  SPEAKER_00: Obviously,
• 1:29:15
  SPEAKER_00: you need the right guardrails and human verification steps in between,
• 1:29:18
  SPEAKER_00: but you're designing a system that can do these things instead of relying on
• 1:29:23
  SPEAKER_00: so much manual work across all of these steps.
• 1:29:26
  SPEAKER_01: Yeah, it sounds like you're learning a lot because these are all new concepts
• 1:29:29
  SPEAKER_01: which you're now using, you're learning about it, you're doing the limitations.
• 1:29:34
  SPEAKER_01: I see what you mean by more learning, more PRs.
• 1:29:37
  SPEAKER_01: And as closing, what is a tool that you really like using?
• 1:29:42
  SPEAKER_01: This can be a digital tool, this can be a non-digital tool, something you find helpful.
• 1:29:46
  SPEAKER_00: Yeah, I know the last few days I've been hooked to like some of
• 1:29:50
  SPEAKER_00: the personal agents like Instang, Muse, Grokbots of the world.
• 1:29:53
  SPEAKER_00: I'm playing around with all of them.
• 1:29:55
  SPEAKER_00: It's helping automate a whole bunch of manual work that I do in my personal life.
• 1:30:00
  SPEAKER_00: And I can in some sense follow the same process,
• 1:30:03
  SPEAKER_00: give it all of the context it needs to do that task for me,
• 1:30:06
  SPEAKER_00: give me all of the inputs I need.
• 1:30:08
  SPEAKER_00: Then I can have human in the loop verification step and do that task.
• 1:30:12
  SPEAKER_00: But it's helping a lot to automate all of the manual personal work.
• 1:30:17
  SPEAKER_01: Awesome.
• 1:30:17
  SPEAKER_01: Well, Uday,
• 1:30:17
  SPEAKER_01: thank you so much for walking through all of the internals that are being used
• 1:30:22
  SPEAKER_01: at Uber.
• 1:30:22
  SPEAKER_01: I found it super interesting, especially that I haven't seen this for quite a while.
• 1:30:27
  SPEAKER_00: Thank you so much for giving us the opportunity.
• 1:30:29
  SPEAKER_00: I feel like we are still pretty early in this.
• 1:30:32
  SPEAKER_00: There's so much to work on.
• 1:30:34
  SPEAKER_00: There's so much to build.
• 1:30:37
  SPEAKER_00: And we are super excited to be on this journey,
• 1:30:40
  SPEAKER_00: to build the right stuff and to see how the software engineering changes and how at
• 1:30:45
  SPEAKER_00: the end of the day,
• 1:30:46
  SPEAKER_00: unlock the value for the company,
• 1:30:47
  SPEAKER_00: value for our business by working on the right side of things.
• 1:30:50
  SPEAKER_01: I hope you enjoyed this look into Uber's tooling as much as I did.
• 1:30:53
  SPEAKER_01: It's now been six years since I left Uber,
• 1:30:55
  SPEAKER_01: so a lot of the tooling that Ulay showed me was also new to me.
• 1:30:58
  SPEAKER_01: Uber was known for having about 5,000 services with about 2,000 engineers 10 years ago.
• 1:31:04
  SPEAKER_01: But since then,
• 1:31:04
  SPEAKER_01: the company started to consolidate services into well-sized services
• 1:31:08
  SPEAKER_01: and today it has fewer microservices than they did a decade ago.
• 1:31:11
  SPEAKER_01: Today it's in the ballpark of 4,500 to 5,000 even though these services are not
• 1:31:16
  SPEAKER_01: as micro and they're much more well-sized instead.
• 1:31:19
  SPEAKER_01: One piece of software that I really like is Capri.
• 1:31:22
  SPEAKER_01: The event capture and replay software.
• 1:31:24
  SPEAKER_01: Back in my time at Uber,
• 1:31:25
  SPEAKER_01: between 2016 and 2020,
• 1:31:27
  SPEAKER_01: we had two tools called Deputy and Cerebrus that could do some level
• 1:31:30
  SPEAKER_01: of production data replay,
• 1:31:32
  SPEAKER_01: but it wasn't all that easy to use and it was just really hard to capture
• 1:31:35
  SPEAKER_01: a real base to start with.
• 1:31:37
  SPEAKER_01: With Capri,
• 1:31:38
  SPEAKER_01: it's far more trivial to capture sessions,
• 1:31:40
  SPEAKER_01: then replay them,
• 1:31:41
  SPEAKER_01: and this works really well to reproduce bugs and also to test that a fix worked for
• 1:31:45
  SPEAKER_01: a certain bug fix.
• 1:31:46
  SPEAKER_01: Building a system at Capri is especially complex when you have closer to 4,500
• 1:31:49
  SPEAKER_01: or 5,000 services running like Uber.
• 1:31:52
  SPEAKER_01: Another thing that was interesting is how we're seeing where CodeReview is headed.
• 1:31:56
  SPEAKER_01: It's slowly dying everywhere.
• 1:31:58
  SPEAKER_01: But looking at the tooling Uber built around CodeReview with CodeInbox,
• 1:32:02
  SPEAKER_01: every PR still gets merged after a human engineer decides that it's ready to do so.
• 1:32:07
  SPEAKER_01: But Uber is building a lot of automated checks.
• 1:32:09
  SPEAKER_01: Many of them are with AI to give more signal about the riskiness of the change and
• 1:32:14
  SPEAKER_01: the quality that AI agents make of it.
• 1:32:16
  SPEAKER_01: And so in the end,
• 1:32:17
  SPEAKER_01: it's a lot easier for engineers to decide which PRs they should spend time looking
• 1:32:21
  SPEAKER_01: at it in depth Versus ones that are safe to approve with the symbol LGTM,
• 1:32:26
  SPEAKER_01: AKA you might decide you're not going to read the code.
• 1:32:29
  SPEAKER_01: Finally,
• 1:32:30
  SPEAKER_01: I had the sense that Uber investing 10 plus years
• 1:32:32
  SPEAKER_01: in developer productivity is really, really paying off with their AI integrations.
• 1:32:36
  SPEAKER_01: For example, they built cloud development environments called Devpods back in 2017.
• 1:32:41
  SPEAKER_01: Uber got to reuse almost all of this infra in building out their cloud agents called Minions.
• 1:32:46
  SPEAKER_01: The same is true for existing systems to validate software,
• 1:32:49
  SPEAKER_01: to safely rule them out, and now agents can use these as tools or as feedback loops.
• 1:32:54
  SPEAKER_01: So it feels like companies that are getting some of the most benefit from AI are
• 1:32:58
  SPEAKER_01: the ones that have had developer platform teams pre-AI as well,
• 1:33:02
  SPEAKER_01: at least in the case of Uber.
• 1:33:03
  SPEAKER_01: Do check out the show notes for more of The Pragmatic Engineer deep dives
• 1:33:06
  SPEAKER_01: on Uber's engineering culture.
• 1:33:08
  SPEAKER_01: Uber is probably one of the companies I have covered in the most detail so far.
• 1:33:11
  SPEAKER_01: If you've enjoyed this podcast,
• 1:33:12
  SPEAKER_01: please do give it a rating and tell your colleagues and friends about it.
• 1:33:15
  SPEAKER_01: Thanks, and I'll see you in the next one.