Latest / The MonkCast / Dealing with the Next Bottleneck: Build Management at AI Scale with Helen Altshuler

Transcript

Dealing with the Next Bottleneck: Build Management at AI Scale with Helen Altshuler · The MonkCast

• speaker-0: Hey, it's James Governor. We're here for another Monk Cast. Today I am joined by Helen Olchula, CEO and co-founder of Engeflow, and we're going to be talking about the channels of build at scale, build management at AI scale. So I think from my perspective, and I'll jump in a little bit, well, first I should welcome you, Helen. speaker-1: Thank you. Pleased to be here. speaker-0: Right. So what we're seeing in this industry at the moment is pretty clearly, ⁓ with the onslaught of software ⁓ being driven by agents, being driven by ⁓ frontier models and assistive coding, ⁓ we're just seeing more and more software being built. And there are bottlenecks being created in terms of our processes for getting that software into production. This is a very real problem. ⁓ I'd say the industry is sort of creaking at the seams. ⁓ we've certainly seen ⁓ problems, for example, Outages ⁓ at vendors like GitHub. ⁓ and so all in all, this is a real industry problem. ⁓ we're creating so much software at such at such pace. ⁓ that that yeah, the the the the bottleneck is, as I say, getting that into production. Now, I think what's interesting is that historically, perhaps, we haven't all had these sorts of scale challenges. We haven't had ⁓ engineering organizations that are large enough. ⁓ we we we haven't had sophisticated enough. ⁓ operations that we can ⁓ be thinking about even rolling out the applications that we're building at this kind of speed. ⁓ but it seems like now with AI, ⁓ the world has changed. And I think that's kind of the the the topic of the conversation today. So Helen, ⁓ I think one of the the the the the key questions for me is ⁓ you came out of an environment of h high scale. And I'm gonna ask you a bit about a bit about your history, how you got here. But do we all face the kinds of challenges that an organization that a web scale organization faces? Is that is that really where we are with with AI? So what let's let's hear a bit about your journey, why you are well you're the right person to ask about build management at scale. speaker-1: Sounds great. ⁓ so yes, we our industry is going through a seismic shift. ⁓ it's quite clear. The code growth significantly accelerated ⁓ with AI tooling and as you mentioned, causing outages, really pushing the limits of every developer infrastructure on its way. And so ⁓ it's interesting to see this because it is familiar environment, ⁓ and ⁓ it is something that I've ⁓ witnessed ⁓ the successful results of while at Google. And that's for that reason ⁓ we're seeing the same kind of challenges now faced by the industry, challenges that Google itself faced ⁓ in 2006, 2008. ⁓ and so I would like to kind of bring us back into what ⁓ what happened with code growth at Google and w what did Google do and now the rest of the industry, why are we all looking ⁓ at Google's build system called Basil. And so ⁓ in in fact ⁓ what we've seen now is that Basil is having a moment ten years after its open sourcing. And so what happened? What happened to the rest of the speaker-0: There's a couple of things here. One is almost all great overnight overnight success stories took 10 years. That is a that is a common pattern. So we've we've got, you know, we've got the the overnight success, ⁓ baseball having a moment. But yeah, also we love lessons from history at Red Monk. ⁓ we do think that, you know, tr to truly understand what's going on in technology, ⁓ it always makes sense to understand the foundations of how we got here. So yeah, I'm I'm I'm super interested to hear about. the the problems Google was facing and and Basil. And so yeah, over to you again. speaker-1: Sure. So so Go so Google ⁓ was facing the building at scale challenges ⁓ in in those years in two thousand six, two thousand eight. ⁓ now Google d is doing something unique which the rest of the industry caught on to as well, which is monorepo style development, which means all of the code base ⁓ is in one place and everything is built from from source and everything all of the tests and builds across different projects are in one repository. And so that allows Google to do a ton of efficiencies and automation, but it also means that Google's builds are huge and Google's ⁓ tests are taking a long time. And so over that period of time, Google noticed that ⁓ the ⁓ Python scripts that it that they used to ⁓ maybe to compile the language and wrap around the two million lines of code make file ⁓ cause significant problems and ⁓ That's when i in in two thousand eight ⁓ Google decided to staff this project called Revolution Fix It, which one of our current employees at Enschlow, ⁓ Mike Blant, was the lead for ⁓ while he was leading the testing group that at at Google. And that Revolution Fix It was aimed to bring together the ⁓ powerful tools that were in place and powerful tools that were being developed into one cohesive ⁓ system. that ⁓ is ⁓ changing how builds are working and changing how developer workflows ⁓ are. And in that time frame, Google staffed the team and ⁓ did a lot of prep ⁓ leading to it. Some of these projects were in the nascent form form. ⁓ and essentially it led the effort to revolutionize its developer infrastructure ⁓ and with a goal to get to five-minute builds ⁓ for any project and ⁓ that ⁓ that meant that builds could no longer be done sequentially. There need to be that there had to be a system that does it in parallel. And so it led to the creation of Google's internal build system called Blaze, which which was the starting point for Basil that we'll talk about later today. ⁓ and also building a parallel parallel engine called Forge, which is the parallel execution of builds at scale, as well as the file system that ⁓ the That really was meant to address the source scalability. And so what's really interesting to me here is that around the same time frame, Google noticed that it needs to do massive change in the developer ⁓ built infrastructure. So, like that's where it kind of like thought of about build management at scale without actually calling it build management at scale. At the same time, there were also efforts to do source at that scale. And so it's also Quite interesting for me to see that today we've seen ⁓ lots of companies starting to re envision ⁓ GitHub, the new GitHub, the source source management at scale, because ⁓ they pushed to the limit. And back in 2008, this is also when Google migrated from one code system ⁓ code management system into its own created one. So scale is the driver, the ⁓ Yeah, ten years ago, fifteen years ago, it really was about Google building better it internally as it was growing as a company. And ⁓ today we see in parallels that ⁓ teams started ⁓ started adapting AI and now everyone is reaching Google level scale faster than ever before. speaker-0: Yeah, some of the the the growth curves are ⁓ you know, honestly, frankly absurd. you know, i i in this industry, I guess w we keep seeing again and again, you know, the the fastest growing ever thing. but but that doesn't seem to stop. You know, the the whether we we get the the fastest growing company ever to, you know, a billion ARR or just the the sort of the scaling characteristics, ⁓ you know, sort of ⁓ ChatGPT is an amazing example. ⁓ you know, fastest growing consumer product ever. we're seeing sort of new markets. It the this sandbox market at the moment, I think, is is a funny one where there are like multiple vendors all doing almost geometric sort of growth. ⁓ because of the, yeah, just frankly the growth is is as they say, ⁓ off the charts. speaker-1: Yeah, and and this ⁓ j just a few weeks ago, or maybe even just last week, ⁓ Anthropic published ⁓ what's happening internally with respect to code ⁓ contributed by person by quarter. And we see in Q1 of 2026, it jumped from previous quarters, which were around one to two X ⁓ to almost ⁓ six times. And Q two of twenty twenty six is ⁓ eight times ⁓ contributions ⁓ by. by person or I imagine they mean agent as well in this particular case. And so that level of ⁓ growth really ⁓ requires ⁓ rethinking of the developer infrastructure. And the symptoms we're hearing are first felt on the code side. And so you can hear companies complain about pull requests. So the volume of pull request reviews has has increased significantly. ⁓ we see we we have some of our customers that are reporting, you know three to ten times more. ⁓ but pull request is now the bottleneck, which is throttling the build problem. And so ⁓ the first issue that we've seen, and and GitHub famously ⁓ created a lot of interesting charts about the growth of ⁓ code ⁓ and git ⁓ and pressures that it put on GitHub. So source is the front of it. Source is under pressure. The next one is review. So code reviews are the challenges and are becoming a little bit of this like throttling mechanism for the real problem that is starting to surface, which is build and test infrastructure. And as soon as we address the issues with codes, w ⁓ issue issues with reviews, we will start seeing the challenges ⁓ that basically we've been working for more than a decade on ⁓ with builds. And so This is the moment. This is the moment. So Basil has been adopted in its nascency ⁓ fifteen years ago, 10 years ago by companies like SpaceX and Uber and Robinhood. And it was mostly because even in its beta state, it was the belt the best built platform at for scale. ⁓ but now we're seeing Basil is having its moment really as the only ⁓ solution for this level of scale. And I'm happy to talk more about just the basal adoption and what we've seen here as well. speaker-0: Well, I'd I'd love to hear about that. I mean, ⁓ you know, I I think that we are always ⁓ yeah, as I say, that the the the history of the project, ⁓ w we've seen this in other areas. ⁓ you know, when we were thinking about container orchestration, you know, an organization that had been doing that at scale and built something that was gonna benefit the industry in this case, Kubernetes, ⁓ was Google. ⁓ Certainly Google meets school scale challenges ⁓ well before a lot of the rest of us. ⁓ but yeah, no, I'd love to, you know, drill in a little bit then into what you're currently seeing. As you say, there were some early adopters there, but I think also the balance between ⁓ you know, you've got an open source project. ⁓ so let let's talk a bit about the actual scale challenges and then a little bit about why, you know, why englow ⁓ is the right organization. ⁓ to be standing this up. speaker-1: Of course. So so as far as the challenges, obviously scale is a depressing one. But there's also other ⁓ benefits of using Basil. It's hermeticity and security, ⁓ and its ability to know that ⁓ all of your inputs were accounted for and you using the ⁓ exactly tested versions in creating a ship shippable artifact. And so Basil ⁓ is now being adopted primarily for scale, but also for the its extensibility and ability to ⁓ help organizations standardize on the common developer infrastructure. And so in in 2000 in in 2017, when I actually created the Post Basilcon while while still working at Google ⁓ together with my colleagues, and we we saw ⁓ 200 people packed it into the packed into the room. in s ⁓ in in Sunnyvale and ⁓ those were the early adopters of Basil. So two hundred engineers, ⁓ maybe one or two per company, but behind each of them there are thousands of engineers using built pi ⁓ pipelines that these folks were creating. And this was the first ⁓ kind of community ⁓ mind share kind of coming together and deciding that ⁓ this is now the time to ⁓ create create more ⁓ tooling to create more ⁓ community governance and start exchanging ideas around making basal scale. ⁓ and as as the community grew, so I've been involved ever since in open source community, create creating the ⁓ the ⁓ the conference, giving giving talks. speaker-0: By the way, I mean, I guess a lot of our a lot of our our listeners will know about this, but that that's a very magic moment, that first conference ⁓ for a project or or ⁓ product where yeah, I mean, having been involved in a few of those, and certainly there's something, you know, it's just around that Dunbar scale, it's the first time the village gets together ⁓ and you probably have been working together sort of offline and coming together. Yeah, no, it's a it's a beautiful moment. So that must have that must have felt that must have felt great. speaker-1: Yeah, i it it was actually interesting to see that at the time Google did zero marketing for Basil. So Google really built it for itself and ⁓ open sourced it only because other companies ⁓ started creating similar ⁓ solutions like ⁓ Meta that's created Buck and ⁓ and Twitter at the time created created its own pants system. And so this was a wake up call for Google that the standards are gonna start ⁓ being be ⁓ really really having a a challenge if Google does not open source the build system. So while we've we've created that conference to bring together the mind share, we actually had lots of collaboration with engineers at Meta and at Twitter at the time and converging on the language that is used for Basal definition, which is Starlark. ⁓ and the later build systems also started started adopting that language. It it is also something that BAC2 is using ⁓ from Meta as well. And really this was a place to bring together folks working on the scale ⁓ challenges at various companies and discuss how to make Basil extensible and adaptable. And so that was the moment where rules ecosystem became more more prevalent. And so Basil in itself is a powerful tool, but what makes it more valuable is that it has extensive rules ecosystem where developers can have the same build platform for JavaScript or Java or C, where traditional tools you need to have different build platforms building these components. So one of Basil's unique elements was that, and ⁓ a lot of the talks in the early Basil cons were all about rules, creating rules. creating migrations, using rules to ⁓ to have more effective migrations, using Gazelle, which is the open source tool ⁓ that's created by one of our Enschlow engineers while still at Google at the time. ⁓ and so community saw the value and community saw the saw the need for ⁓ strong investment. At the time that investment came a lot from Google and over the years the balance was shifting with obviously with ⁓ companies like Enschlow being created. ⁓ with Linux Foundation that we've just established to create governance around Basil community ⁓ tooling and ecosystem, we are now seeing the move ⁓ and maturity of this ecosystem. And that to me is ⁓ also a good signal why more companies now than ever before are adapting Basil right right from the start, because the community matured But also more importantly, things that were rough edges of Basil, AI is smoothing those over. So for example, one of Basel's rough edges is its learning curve and documentation. And my team actually is also investing in this and making Basil documentation better, collaborating with engineers from Astranis and ⁓ several other companies making it better. But really, I think we're now making documentation better for agents ⁓ to read. More and more developers are interacting with ⁓ with ⁓ their agents to learn how to use basil to do basil migration itself, which is also one of the rough edges. ⁓ and then perhaps the most phenomenal one is basil never had like really strong ID integration. And so now ⁓ we are living in a world where ⁓ IDs are at least the like the visual ones or less over A dominant ⁓ code creation engine. speaker-0: A hundred percent. So the the the sort of the the clawed code ⁓ I mean, clawed code is is pretty good at handling those kinds of integrations. speaker-1: Yeah, absolutely. What what we've seen from our customers is that they before they would put error messages from Basil into Stack Overflow or ⁓ other ⁓ or just ask on the on Basil Slack. And overwhelmingly now ⁓ Claud is pretty good at answering those questions. ⁓ yet ⁓ and ⁓ I hear good feedback from developers using Codex and other platforms as well. speaker-0: Yeah, well that's ⁓ it's a great story. ⁓ you know, a thing with with a a few, you know, ⁓ a few, as you say, rough edges ⁓ can get smoothed out in this in this era of AI tooling. ⁓ and to your your your argument about, you know, basil's time is now, yeah, definitely ⁓ I mean it's just interesting because so much of the developer experience that we've said is so important ⁓ is now agent experience. ⁓ so Let me ask you a quick question about that. W I mean, what are some of the things that you're doing to make sure the docs are agent ready? I mean, there's the obvious, you know, publishing and markdown and so on, but what are some of the specifics ⁓ that are about making I mean, well, it's a very different definition of accessibility, but agent accessibility. What are you doing in that regard? speaker-1: Yeah, let me let me in in that point also hi reveal another cool feature of Basil and use it as an example. So Basil has tremendous amount of data that it outputs, which is really hard for humans to process. It's got build build event stream, ⁓ it's got it's got ⁓ basil query, it's got ⁓ profile information, basal profile itself, the tracing information. And all of that data, ⁓ today we're processing through ⁓ well or at least for the past six years, we've been processing by building relational stores, by creating observability as part of our platform. ⁓ but but more and more I'm now starting ⁓ starting to see ⁓ agents are using that information ⁓ much better but burning a lot of tokens ⁓ on the way. And so what we ⁓ implemented at Enschlow more recently with this customer insight that we've gotten is ⁓ MCP and skills that are using deterministic information from profiles to help agents come to the conclusions faster and ⁓ complement the in and default to this ⁓ discovery when deterministic approach is not working. speaker-0: Yeah, well we're certainly we're certainly not living speaker-1: Ready that's the big the that's that's the big ⁓ part, like using what you can about your data and deterministic way to answer it. speaker-0: Yeah, it's super important. I mean, you know, look, it's ⁓ July of twenty twenty six. You're not gonna be talking to any organization that isn't concerned about how many tokens they're burning. ⁓ so so yeah, that's that's super important. So we've we've we've got some of the the the this sort of the journey from Google, you know, to more general purpose. ⁓ maybe let's cut over a bit to To yeah, to the these questions of of customers and and scale. And yeah, what are what you know, let's it it's always you know, v engineers love love love stories, numbers about high scale and so on. So yeah, have you got any examples ⁓ about, you know, current state of the art and what you're seeing in your customer base? speaker-1: Indeed, indeed. I we actually did that analysis recently for our internal team summit just to just to understand what is the scale that we are processing today across all of our customers and how that changed ⁓ in just one year. And remarkably, what we saw was that ⁓ in May of twenty twenty five, across all of our customer clusters, ⁓ all of the companies combined that we're managing were ⁓ executing ⁓ 200 f 50 million build actions per week. And so this is your build and test workflows. So look, it's a good metric for your developer velocity, but also for the complexity and scale you you're working with. And just within a year, by May by May of this year, we're seeing close to two billion ⁓ actions executed by the same kind of set of customers ⁓ per week. And that to absorb that level of scale. Like Ench flow had to do a lot of work of over the past year. And the reason we did this internal study is because it didn't feel like we were absorbing that scale ⁓ on a day-to-day basis. From time to time, we have a customer request to look at some tuning, cost tuning, or some performance tuning. And so we've been investing in this along the way. But when we look at the results, we're seeing that our customer plaid more recently actually commented on that that they've absorbed three times the number of pull requests without any change in their ⁓ c cloud or li or anchful license ⁓ kind of infrastructure. And so that's that's an a a way that we can help companies scale ⁓ sublinearly scale their costs sub ⁓ and and and and and have ⁓ have a better efficiency in the overall cycle. And this is where like I'm gonna bring it back to like again why Basil, why Basil is having its moment. And we do have customers that are not using Basil, but Basil is uniquely positioned here, ⁓ is because speaker-0: Well, I just want to hear a little bit about that. When you say g customers that aren't using basil, because like, you know, th there is a world out there that is not all basil. ⁓ there are customers that have alternatives. Migration can be hard. What what you know, what what what do you mean so your customers aren't using basil? I thought you were the basil company. speaker-1: That that's that's right. It's i it's interesting. I let the cat out of the back. So we are a basel company, but we actually our own thinking as the team that built Basil ⁓ at Google and like we are at this point the largest concentration of basil experts anywhere in the world here at Ench flow, we actually weren't bullish that basil will win the market in twenty twenty one when we started ⁓ implementing some of these ⁓ some of these systems. And we looked at the approach of Well, developers will w always want to have choices. So, can we bring some of Basil's best features into the ⁓ older generation build systems or into other build systems that may be compatible with Basil level of thinking? And so actually, Basil has a ton of advantages, but the magic is not only Basil, it's remote execution API. And any build system that implements that remote execution API. means that they are able to parallelize their workload on large number of smaller machines as opposed to smaller number of large machines, which is where a lot of the performance and cost savings come from. And so we basically implemented a remote execution service that works with any ⁓ any built ⁓ system that implements remote execution API. So MethodsBUC2 is implementing that. And ⁓ Android platform is implementing that, AOS P. And so we increased the number of ⁓ customers that are using Basil ⁓ within sorry, that are using remote execution within within that that stream. We also acquired a company TP build ⁓ a year ago, and they have been ⁓ working on making CMake build system. More compatible with the modern concepts. And so we connected that through remote execution API. And so we're able also to get them the benefits of parallel execution. But with all of this, while remote execution API and parallelism is really, really essential for speed and performance, Basil has other advantages. And this is why we we've seen companies benefiting from having more. ⁓ tooling that is built on basil that may not exist for other build systems. And that tooling is what's enabled what's enabling companies to scale. It's the security that I mentioned, being able to create a software, a bill of material as part of the build process, which is becoming much more, much more and more revel relevant. And it's analyzing vast amount of data from Basil to improve the underlying performance of their their build processes. And so we call this build graph analysis. Basil uniquely provides visibility into this that allows companies to parallelize further and perhaps not continue to scale through adding more hardware at the problem. speaker-0: Is this is this something that organizations can like run themselves? I mean it's an open source project. I mean you're making it it sounds very cloud like, but I mean obviously there must be organizations that that that that are ⁓ that have reasons that they'd want to be running their own infrastructure. speaker-1: Indeed. And and what I've what I've been also ⁓ saying recently ⁓ to my team is that I'm seeing signals that AI future is multi-cloud, and by multi-cloud it means run everywhere where you can get good amount of resources. So Basil itself is is an open source project that you can install on any infrastructure, be it on-prem, your your your personal machine or ⁓ cloud. ⁓ Enchflow provides the platform that distributes basal builds and parallelizes them with observability ⁓ in a most efficient, cost efficient, and performant way. ⁓ in the cloud, we support multiple cloud environments such as ⁓ AWS, ⁓ GCP, ⁓ and also ⁓ building out the ⁓ Kubernetes-based solution that allows ⁓ you to run it anywhere, on-prem or ⁓ or on any other flavor of Kubernetes like Azure Kubernetes MCN also is on the rise right now. And then we're hearing nail clouds like Nebus and Core Week. Yeah. Oracle Cloud has ⁓ reached migrator as well recently. So it's it's actually quite quite remarkable that ⁓ from Basel's perspective, it does it doesn't really care where it runs. But ⁓ being able to paralyze it effectively, you need to know a little bit more about The auto-scaling of a particular cloud, the ⁓ storage system of a particular cloud on-prem infrastructure. And that's where we focused the past two years at NSHUR specifically, and we built ⁓ more efficient autoscalers than GCP and AWS native ones. So Snowflake is one of our customers that actually ran it themselves, ⁓ that ran Basel and Remote Execution open source project ⁓ on their ⁓ own ⁓ AWS infrastructure. And by simply switching to Ench flow, they cut their their cloud costs in half because we are that much more efficient in ⁓ utilizing the cloud infrastructure ⁓ for the same performance benefit. speaker-0: So Snowflake made the transition from running it itself to running it on end flow. speaker-1: Yes, yes. And the case the case study is coming up. speaker-0: Of your salespeople definitely made their quota that that quarter. speaker-1: This th th this was one of those ⁓ deals where where ⁓ I got a call from Snowflake CFO just trying to understand how they're making money on the software that we are selling them. ⁓ so this was this was a really interesting opportunity for us. And since then we've seen more and more companies. And so you asked me to share some numbers. ⁓ last ⁓ last November at Basilcon, I actually gave a talk together with a staff engineer at Databricks, ⁓ also our customer, and ⁓ they they revealed the numbers. So Databricks is at 3,000 developers. Their code base grew to a hundred million line lines of code in their monorepo. ⁓ I expect that it it's close to double now at least. They have seen twice as much code growth in a year. And ⁓ they were ex they were like executing a billion actions a day. And so that is a really huge scale. And now when we're talking to customers at this scale when they're anticipating more and more load, ⁓ mostly agent-based ⁓ load, ⁓ we are asking them questions. have you looked at your dependency ⁓ at your dependencies? Did you look at the build graph? I actually published a blog post ⁓ a few months ago about my week in a field, ⁓ just like being ⁓ being you know on calls with customers and meeting customers at various industry events and d and dinners. And hearing these signals and the signals are overwhelming that CI is under pressure, the company is trying to figure out how to parallelize their their their builds, or they're already using the latest tooling like remote execution, and now they're investing in their build graph optimizations, which is what we can surface with our analytics for companies to scale smarter. And so the goal for me creating that blog post was ⁓ how can we k help companies scale smartly because even if you can throw that much hardware at a problem, we've also seen our customers that are lodging that are running two hundred thousand core clusters l getting ⁓ cloud provider stuck out issues. So there simply isn't enough ⁓ compute available at the time that they need speaker-0: By the way, we'll we'll ⁓ we'll put a link to this blog post in the in the description. ⁓ but but ⁓ yeah, no, that the you know, we are seeing these panelists. And also, I mean, you know, quite a nice position to be in to say, Hey, ⁓ we're running builds for the two, you know, arguably the the key players in data infrastructure of the current era. speaker-1: Yeah, I would I would say that ⁓ there's there are clearly leaders in seeing the volume ⁓ from AI and investing in build developer infrastructure early. And ⁓ by o b obviously like Databricks and ⁓ Snowflake are those leaders. Monga D B another customer also has to kinda invested speaker-0: The bongo as well. You're really racking up all the all the data of the speaker-1: We essentially right, we we o we o we own the large scale data data space, ⁓ but more importantly, ⁓ at this point there is no company that can handle the scale that we're running at, which is like two hundred thousand core infrastructure, ⁓ you know, like two billion ⁓ actions per week. And so we see we started by having these companies as early adapters because they are seeing signals before everybody else. ⁓ the early majority is coming. And like one of the examples I wanted to actually also highlight, ⁓ Tinder published a really, really great blog about how their investment in Basil and ⁓ in parallelization and hermetic tools positioned them so well that now with Cursor and Basil they're able to do their software upgrades ⁓ much, much faster than ever before. And so we've seen companies learn from each other, they read the blogs, they learn how ⁓ like what their peers are using, which is by the way one of the reasons why we run Basilcon is bring all these peers together to talk about ⁓ latest tooling that they that they can take advantage of. And and this is an example of fundamental kind of AI infrastructure investment. So Tinder made that investment early ⁓ fairly early. ⁓ so did Databricks, ⁓ so did Zooks, like an ⁓ a another company that we work very closely with. And Zooks has migrated from their DIY solution to Angeflo. And again, like with cost savings, but really extensibility and supportability of the new ⁓ level of scale that they need ⁓ to be working with. speaker-0: You a question. So you're, I mean, look, if I if I if I cut you, Helen, you'd bleed basil. I mean, pretty clearly, ⁓ you know, this is the the the you know, you're right? Absolutely, you are absolutely ⁓ but you're you're sort of you know the way you described it, and if we go back to that 2021 moment where you're like, what if basil doesn't win? And sort of I'm quite interested in, I mean, that's ⁓ a really interesting position, but also. Th this this question about you don't wanna just be the basel company. You wanna be the build management at AI scale company, right? speaker-1: We are. We absolutely are at this point. And so we're running that level of scale for major stock exchange with CMake, ⁓ for major gaming company with Bactu, ⁓ for most of the browsers out there ⁓ with Chromium ⁓ build system. Like the we we are definitely the company ⁓ that owns build ⁓ management at scale. ⁓ and interestingly enough, Basil ⁓ is is having its moment. For the first time ⁓ like since it was open sourced in in like much with much higher number of people not just adopting Basil, but also a sign of like job rec so you can see every single company at this point that has large code bases is looking for people with basel expertise. So we see in the future is convergent towards using Basil for its core features. We also seen these other build systems that I mentioned being ⁓ essential for the audiences that they serve. So like Chromium is very much in in Chromium build space. ⁓ and gaming is ⁓ is certainly adopting buck to ⁓ and and definitely meta is having its moment with buck to because of scale as well. speaker-0: Yeah, well I mean, you know, the the bet on parallelisation is generally a good bet. Indeed. ⁓ let me ask you that that question about GitHub then. you know, ⁓ I think that when I look around there are people sort of looking at actions, trying to work out what ⁓ what their strategy should be. I mean what w how do you embrace how do you embrace the GitHub customer? speaker-1: Well, GitHub customers, ⁓ first of all, like they come with ⁓ a number of ⁓ other tools that they use. And so as I mentioned, C code code is is step one, ⁓ reviews step two, then there's build and CI. And so ⁓ we see we see in this as a challenge. Most of our customers are have are pushing GitHub to its limits. And ⁓ at this point, a lot really the solution is ⁓ Parallelization of the build workflows, parallelization of the test workflows. So we have we natively ⁓ integrate with GitHub Actions, for example, making them more scalable and more parallelizable through the use of our CI runner infrastructure. ⁓ this is a fairly ⁓ busy space as well. There's there's there's lots of companies now trying to plug the gaps that GitHub has created until GitHub itself fixes them. But more importantly, I would say it's really the build. metadata that we are the experts in. So to be the ⁓ leader in build management at scale for us, it's a combination of our team's expertise and the data that we accumulated in our platform to make smart decisions around how to restructure dependencies, how to squeeze more from compute, how to parallelize data. And so to us for anyone any company that's starting with their ⁓ kind of GitHub infrastructure, looking at builds ⁓ and looking at their CI, ⁓ the next logical step for us is to ⁓ to work through the ⁓ GitHub Actions, ⁓ kind of self-hosted runners that we're able to provide or manage. ⁓ and ultimately we've we've actually ⁓ even seen seen the benefit of having this kind of forward deployed engineer ⁓ impact where we started helping our customers to move more workload from GitHub actions to core remote execution. And by doing that, they're s saving a ton a ton on compute, but they're also getting much faster performance improvements. And so we'll also be publishing the blog that that we've that we've done with companies like Apex and Iterable. And they're a combination of CI runner system plus remote execution and really understanding what Can be parallelized in the code base and using agents to parallelize those things more effectively gets the company's ultimate kind of performance and speed and scale and makes them really ready to absorb the scale that's coming at them. And the example that I brought up in the beginning with Plaid absorbing three times more without any changes to their cost is really because. There there's so much parallelism in their system and we work together to implement more parallelism as we go. And I would I would also say another kind of data point to to give you, back to the anthropic example that I've given, it's it's not surprised that AI companies are predominantly convergent and b and basal. And so ⁓ open AI is is investing heavily. There's a lot of open source ⁓ efforts with codex ⁓ driven by ⁓ some of the prominent community leaders like ⁓ Zbarski and and several others that are ⁓ openly in the community are showing the impact of running this ⁓ on Basil. ⁓ we we s we're seeing that traditional built systems are not scaling and we knew that that there's like about a million lines of code in your Mana Repo when you really need to start looking at different ⁓ solution for build management at scale. And so every company is getting closer to that metric ⁓ than before. But also we've seen that AI companies, even even AI companies, I would say they started not with Basil and they're converging on Basil and they're migrating now that they have doubled or tripled their code base within short amount of time. ⁓ and so what I wanted to kind of like kind of l le leave the folks ⁓ with ⁓ who are watching this is ⁓ how do you know that you need to take a different ⁓ look at the build ⁓ infrastructure, at the build management of scale? ⁓ it's first of all, you're using variety of build systems ⁓ for different languages. Like that's that's that that's a that's a sign that you have complexity in your system. the other one is that ⁓ for ⁓ builds ⁓ f for languages that do not require build system, interpreted languages like Python ⁓ or JavaScript, you ⁓ you're not using build system, but you're generating so many tests that you're not sure how to ⁓ how those tests will be ⁓ operated together and how to parallelize them. And what we've seen at that happens at that point is that companies are starting to ⁓ turn their CI system into these manual shards. so that they can create parallelism by doing these ⁓ shards of ⁓ tests instead of letting the build system like basil take care of all of that ⁓ for you. speaker-0: There you go. Because I I mean I have to say, sort of summing up, ⁓ the thing that has really surprised me recently, and you you spoke about, you know, ⁓ you know development, sort of verification, deployment, where build sits in that. The the thing that I have been ⁓ really surprised by is the Rapidity with which organizations, even in regulated industries, are saying things like, we're not gonna do manual code review anymore. At least not for everything we're building. ⁓ and that is definitely even a year ago, I would have I would not have expected, you know, I wouldn't have expected to go in and be talking to, yeah, a bank in the UK. in in the last few weeks, you know, Manufacturing Company, Bank of the UK, Gaming Company, where they're like, actually and one of these organizations in particular, and I don't think I should name them, they've started saying, Well, we don't call them it's not about the branches, it's about the leaves. You know, branches we will do some human verification, but for a lot of what we do, they're just leaves. And I I think that that if you think about the ⁓ you know, cattle versus pets that we talked about with infrastructure, we're seeing applications and they're just much more ⁓ disposable now. And that of course accelerates ⁓ how much software is being built. Because if you've got a good process where you can do a rollback if there is a problem, then you've got a much more much higher tolerance for risk. And yeah, y you know, t as I say, w when you had this conversation with the bank, it was it was really surprising. speaker-1: I I heard the same conversation two weeks ago when I talked with with some of our ⁓ right sharing and ⁓ autonomous driving customers ⁓ that ⁓ they're looking at risk based pull request reviews. ⁓ so figuring out like what goes automatically and what's not. And as I mentioned in in the beginning of this conversation, pull requests are the current lever ⁓ like like that that once you pull it, the throttling on your build infrastructure Will ⁓ not be in place and companies that have not invested in build management at scale will suddenly start facing exactly the same issues that GitHub is facing when code bases blew up. And so this currently is a really cool moment to observe in our industry is that ⁓ pull requests are still throttling how many builds are ⁓ are happening, and yet most of our customers have seen from two to ⁓ four times an improv ⁓ increase in their build in the number of builds ⁓ that are coming happening on a weekly basis. And so but once that lever is pulled, because you're now ⁓ doing a lot more automatic reviews, the build infrastructure will show its cracks. And the test ⁓ for ⁓ you know are you gonna have cracks or not is predominantly are you using basil and are you using speaker-0: Yeah. speaker-1: remote execution and caching system that Basil ⁓ is natively ⁓ like provide providing and with all of the acceleration that we can offer on top of it. speaker-0: Okay, well, I think you've been even more bullish about basil than I had expected. ⁓ but I did you know, I was I I was pleased we're gonna have this conversation. What what's important, ⁓ as I say, as as we sum up, is it is this yeah, like and it's Again, I every day in the AI era sort of reassess your priors day. And it again, like one year ago, I was like, well, actually the agent pull request, that makes sense, right? You have a human, ⁓ get the agent to do some work, makes a pull request, ⁓ look at the code, ship the code. That's just nothing nothing like how we're actually using agents. That's just completely ⁓ that's broken down. And I I think that that that fundamental primitive ⁓ in terms of workflows that that has propelled the industry for you know the last well twelve, fifteen years or so, ⁓ yeah, fundamentally breaking down. So so yeah, new we new new laws of physics basically. speaker-1: Yeah, essentially what's changed really isn't Basil, it's everyone else. And the pressures that once defined a few large codebases, ⁓ like the one at Meta Google and Google Twitter, are now showing up everywhere. ⁓ in languages, like there's more languages in a code base, ⁓ and all of this is ⁓ putting CI ⁓ pipelines under pressure. And ⁓ I'm I'm actually like looking at this on a on a weekly basis with our customers and ⁓ the more the more investment that they're able to make in ⁓ understanding their own code bases and restructuring and using an agent workflows to restructure their own dependencies, the better they will be ready for continued increase that is happening. It's not even increase in the code growth, it's increase in velocity, is that there's more changes. And more changes means that more changes need to be reviewed and built. And I do hope that with AI ⁓ code bases, are ⁓ growing, but ⁓ hopefully there's AI is also good at deleting code as much as it is creating code as well. speaker-0: Well, we're gonna have to see a lot of that, because as I say, there's so much of it is dungible. ⁓ yeah, we're gonna need to throw away an awful lot of code. ⁓ otherwise things are gonna get very, very expensive. And on that happy note, ⁓ yeah, that that's been amazing, Han. ⁓ I I ⁓ I really appreciate your time. ⁓ thanks for joining us on the show. Like I say, I really enjoy ⁓ a good good sort of ⁓ lesson from history. ⁓ you know, we recently had ⁓ Sterenion from Google's Cloud Run team learned a bit about the foundations of of that product. You know, I always think it's interesting, yeah, of course things are interesting that happened today and over the last eighteen months. speaker-1: And course the fun fact is that Sterin was my colleague working on the Basil project in twenty seventeen. speaker-0: There you go. So this is where goes a couple. Come to the Red Monk ⁓ podcast if you want to hear these interesting ⁓ interesting stories about Google technology. Yeah, Sterren and Helen ⁓ working on that project. So yeah, ⁓ what I would say, everyone, I hope you've enjoyed the show. ⁓ the classic things. Please do share it if you think anyone's ⁓ hitting some ⁓ build limits and they might need some help. ⁓ but yeah, share it, like it, subscribe, do all of those good things. ⁓ but yeah, really, Helen, ⁓ thanks so much for joining us. speaker-1: Sounds great. And join us on a Basil World Tour where we do bring together people that care about it. And ⁓ we we're publishing our events at meetup.build for for that as well. speaker-0: Okay, so there you go, everyone. That's your call to action. ⁓ thanks very much, Helen, and thanks all for joining us. speaker-1: Thank you.