Latest / The Adults In The AI Room / Earley AI Podcast – Ep. 86: Open Source, Observability, and AI-Driven Engineering with Tom Wilkie

Transcript

Earley AI Podcast – Ep. 86: Open Source, Observability, and AI-Driven Engineering with Tom Wilkie · The Adults In The AI Room

• 0:01
  SPEAKER_00: This is the Early AI Podcast hosted by Seth Early.
• 0:04
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science, your digital transformation journey with design and deployment of innovative technology solutions, as well as Vector.
• 0:26
  SPEAKER_00: Now enjoy the show.
• 0:28
  SPEAKER_01: Good morning, good afternoon, good evening, depending upon your time zone.
• 0:32
  SPEAKER_01: Welcome to the Early AI Podcast.
• 0:34
  SPEAKER_01: I'm Steph Early.
• 0:36
  SPEAKER_01: Today I'm joined by Tom Wilkie, who is VP of product at Grafana Labs.
• 0:40
  SPEAKER_01: For those who don't know Grafana, think of it this way: every modern company is a software company, and every software company needs to understand what its software is doing.
• 0:49
  SPEAKER_01: That's observability.
• 0:50
  SPEAKER_01: And Grafana Labs has become one of the most important players in that space with 25 million users, and their tools are used worldwide with over 400 million in revenue.
• 0:59
  SPEAKER_01: But what makes this conversation really interesting for audience is not just the technology, it's the philosophy.
• 1:05
  SPEAKER_01: Grafana Labs has built its entire strategy around openness, open source, open standards, open data.
• 1:11
  SPEAKER_01: And the philosophy has some fascinating and unexpected consequences in the AI era.
• 1:15
  SPEAKER_01: Tom, welcome to the show.
• 1:17
  SPEAKER_01: Hey, thank you for having me.
• 1:18
  SPEAKER_01: So our audience includes technical practitioners and business leaders.
• 1:23
  SPEAKER_01: For somebody who's not deep in the infrastructure world, set up a problem for us.
• 1:27
  SPEAKER_01: What does observability actually mean and why should business leaders care about it?
• 1:31
  SPEAKER_03: Yeah, of course.
• 1:32
  SPEAKER_03: Yeah.
• 1:32
  SPEAKER_03: So I mean, if you're uh if you're a modern business leader, you're probably familiar with the term like software is eating the world, right?
• 1:38
  SPEAKER_03: And this is the idea that every business is now effectively become a tech business.
• 1:42
  SPEAKER_03: So if your like key kind of competitiveness hinges on your ability to build and leverage like technology, then you're gonna probably want to hire a bunch of software engineers.
• 1:52
  SPEAKER_03: And those software engineers are gonna build some software that's gonna be running your business, and they're gonna need to like understand how that software behaves, and they're gonna need to be able to iterate on and improve that software rapidly, and that's really gonna probably dictate your competitiveness.
• 2:07
  SPEAKER_03: You want to give them the best tools, right?
• 2:09
  SPEAKER_03: You want to give them the tools to help them safely iterate and improve their software, understand what happens when it goes wrong, respond quickly, and that's observability.
• 2:16
  SPEAKER_03: That's that kind of like set of technology and tools that helps software engineers respond to incidents and improve performance and ship features quickly.
• 2:25
  SPEAKER_01: Observability has been around for a while, but of course, uh in the AI era, we have a lot more opacity of certain functions and internal mechanisms, and of course, the language models themselves are so highly complex.
• 2:39
  SPEAKER_01: These are, you know, uh these are the statistical outputs, not deterministic outputs in many cases, right?
• 2:45
  SPEAKER_01: But so so tell me how this new world of AI, machine learning, and uh large language models and generative AI, how does that impact observability?
• 2:55
  SPEAKER_01: What are the biggest issues and challenges?
• 2:57
  SPEAKER_03: Yeah, no, so I look at it really kind of from three angles here, right?
• 3:00
  SPEAKER_03: So one is the whole problem of observability becomes, you know, potentially a little bit easier when you apply agentic kind of techniques to it.
• 3:08
  SPEAKER_03: So using agents to automatically root cause analyze incidents in your production infrastructure, right, is now or has been for a while now possible and really quite effective, right?
• 3:20
  SPEAKER_03: You know, if you've got and really have consolidated observability into something that has a nice interface that agents can leverage, then suddenly, you know, it is very easy to tell what happened.
• 3:29
  SPEAKER_03: It's very easy to automate a lot of the processes around release management and so on.
• 3:34
  SPEAKER_03: So, firstly, yeah, leveraging agents to make observability more easy.
• 3:37
  SPEAKER_03: The second big angle is like obviously, and and this is probably goes without saying, like AI, LLMs, these have completely revolutionized how we build software, right?
• 3:47
  SPEAKER_03: And so, you know, with with technology like Clawed Code being so capable of autonomously building software to a specification, it is quite a natural extension to give these desktop agents access to your observability system.
• 4:01
  SPEAKER_03: And so suddenly, like the role of a set of observability tools, you know, really helping an engineer understand a problem is shifting towards helping agents understand those problems and making these agents accessible, making these tools accessible to Claude, to, you know, all of the various tools, and actually it has this nice effect of like combining and tightening the dev test loop, like and and really like allowing engineers to iterate more quickly.
• 4:23
  SPEAKER_03: And third, third thing, third example is like, you know, one of the one of the patterns you see in the software industry is every time there's kind of an architectural revolution, every time the way we build software changes, be it going from like uh the mainframe to you know the cloud to you know, from monoliths to microservices, from event-driven architecture, every time there's a new architectural style, there's a new wave of tooling to understand its behavior, you know, and a new wave of best practice.
• 4:50
  SPEAKER_03: And we're seeing exactly the same thing with this agentic architectures that are that are becoming common, you know, requiring a new set of observability tools to really understand their behavior, which is kind of I think what you were alluding to at the beginning.
• 5:02
  SPEAKER_01: Absolutely.
• 5:02
  SPEAKER_01: Yeah, absolutely.
• 5:03
  SPEAKER_01: And we had just uh done a project for a client where we were helping with their software development lifecycle and automated testing was a huge piece of this, just to affecting the culture change to get developed engineers to think differently about this and to get leadership to realize that you don't need uh a massive team of offshore developers anymore, right?
• 5:23
  SPEAKER_01: You can do these things with you know fewer people, with uh people that have more capabilities and experience, and and let all that grunt uh programming be done by by the by the tools, by the uh by the clawed uh uh claws of the world.
• 5:36
  SPEAKER_01: And uh it is uh astounding.
• 5:38
  SPEAKER_01: It's it's it's revolutionary, it's it's amazing.
• 5:41
  SPEAKER_01: But as you say, you know, this new wave does require kind of a new way of thinking this.
• 5:45
  SPEAKER_01: And and you just you just said that there's a proliferation of new tools.
• 5:48
  SPEAKER_01: The market is massive.
• 5:49
  SPEAKER_01: I think you mentioned that there are 30 billion dollar plus uh players in the space.
• 5:55
  SPEAKER_01: Uh it's interesting that unlike something like CRMs, where vendors, uh a small number of vendors tend to dominate, observability hasn't consolidated.
• 6:03
  SPEAKER_01: Why why do you think that is?
• 6:04
  SPEAKER_03: Oh, yeah, I love I love this about the observability market, right?
• 6:07
  SPEAKER_03: Like fundamentally, I think the reason this market can support so many competitors is because the switching costs are relatively low.
• 6:15
  SPEAKER_03: So, you know, the the fact is for an observability system, you get a lot of value from the most recent data that you've put into it, right?
• 6:22
  SPEAKER_03: You really only need kind of the last few hours of data, you know, before you can start to, you know, notice trends and patterns and and start to understand the behavior of systems, right?
• 6:32
  SPEAKER_03: And that fundamentally very low switching cost is is, I think, one of the many reasons why this market really rewards the most innovative players, right?
• 6:41
  SPEAKER_03: And then once you start layering on, like there's there's a nice trend in the market for open standardization, like a trend I hope being uh being really encouraged by Grafana Labs, right?
• 6:51
  SPEAKER_03: That's really increasingly lowering these switching costs, right?
• 6:54
  SPEAKER_03: So I think you know, we do a lot of migrations off of our competitors onto Grafana Cloud, and we've found, you know, there's an AI angle to this as well, where we're leveraging kind of Gen AI to translate dashboards and alerts and all of these kind of uh approaches.
• 7:08
  SPEAKER_03: But the easier we make it to migrate off of our competitors onto Grafana Cloud, the easier, like and more customers do that.
• 7:14
  SPEAKER_03: And yeah, and like I love how competitive the market is because of that.
• 7:18
  SPEAKER_01: So, you know, you you do have this, you call the approach the big tent philosophy.
• 7:22
  SPEAKER_01: So walk us through what that means in practice and why it matters for enterprise buyers who who are evaluating observability platforms.
• 7:30
  SPEAKER_03: This is this this was a really interesting kind of aspect of the market, right?
• 7:34
  SPEAKER_03: So you've got so many players in the market, I said like$30 billion plus players, and yet all of them work on a very similar model of send us your data and we will keep that data for you and we will give you tools that help you understand that telemetry, right?
• 7:46
  SPEAKER_03: That data.
• 7:47
  SPEAKER_03: Grafana Labs fundamentally from day zero started with a very different philosophy.
• 7:51
  SPEAKER_03: We were like, keep your data wherever it is, we we don't care, right?
• 7:53
  SPEAKER_03: What we'll do is we'll build tools that allow us to connect to the data where it lives in our competitors, in other observability stacks, and then allow you to visualize it all in one place in Grafana.
• 8:03
  SPEAKER_03: And to this day, 12 years later, like Grafana is still pretty much the only thing that does that, right?
• 8:08
  SPEAKER_03: Um, you know, and that's that was weird, that was strange, but that was what that was our way, our initial wedge that got us into peak places and became really the only dashboarding technology that worked when you had, you know, tens of different vendors in your organization.
• 8:21
  SPEAKER_03: I will say, you know, since the end of uh of of the zero interest rate period, like we've moved to a world where most organizations are now consolidating onto a much smaller number.
• 8:31
  SPEAKER_03: And that Big Ten angle is maybe less of a differentiator, less of a wedge than it used to be, you know.
• 8:37
  SPEAKER_03: But like a lot of people are now choosing to consolidate on Grafana and because we're more cost effective, because they get better value for their money, because we've got better technology around AI, because we've got, you know, more competitive solutions.
• 8:48
  SPEAKER_03: One of the one of the angles that I really like about Big Ten is because we, you know, because to build that kind of 200 plus different data sources for Grafana, you know, it naturally fits into kind of an open source ecosystem.
• 9:00
  SPEAKER_03: We we built, you know, the majority, like 90 plus percent of our code in the open.
• 9:04
  SPEAKER_03: Right.
• 9:05
  SPEAKER_03: And that gave us gave us a lot of advantages that that really, like a lot of people didn't see us almost as competitors here, like it gets us in the door with the lowest possible kind of uh barriers to entry because it's just go to GitHub and download it.
• 9:19
  SPEAKER_03: But what it also got when in the age of AI is it got models that started to know how to use our software.
• 9:25
  SPEAKER_03: And this was like an aha moment for us, right?
• 9:27
  SPEAKER_03: We realized that by having, you know, 25, I think it's more than that now, like million users of our software worldwide, they're out there blogging, publishing examples, tweeting, publishing videos, you know, just generating so much content on the open web about how to use Grafana, how to use Prometheus, how to use Loki, that these models, these foundation models from you know Anthropic and OpenAI and Google and so on, they're trained on that data.
• 9:51
  SPEAKER_03: They're trained on the open web.
• 9:52
  SPEAKER_03: They know fundamentally how to use our software better, actually, than they know how to use proprietary competition.
• 9:57
  SPEAKER_03: Like that for us, like open source is such cheap.
• 10:00
  SPEAKER_01: That's so fascinating.
• 10:01
  SPEAKER_01: I mean, it really is so interesting how uh being that openness has created that competitive advantage when you think that openness would not create that competitive advantage, right?
• 10:11
  SPEAKER_01: It is very counterintuitive.
• 10:13
  SPEAKER_01: You also mentioned three things that differentiate you, and again, this isn't meant to be a sales pitch on Grafana, but the principles are very interesting.
• 10:21
  SPEAKER_01: So just my my audience keep that in mind.
• 10:23
  SPEAKER_01: You know, I'm not pitching you guys, but I think you're you're very ill illustrative of a lot of fascinating uh principles and dynamics.
• 10:29
  SPEAKER_01: You mentioned three things that differentiate you the adaptive telemetry, big ten interoperability, which we talked about, and AI.
• 10:36
  SPEAKER_01: Let's unpack or start with adaptive telemetry because the economics angle is kind of faceting.
• 10:41
  SPEAKER_01: Facetely.
• 10:43
  SPEAKER_03: And this this is a recent one, as I mentioned, like with the interest rate change, like suddenly, you know, it used to be if we rewind four or five years ago, as a software engineer, I could justify purchasing a tool pretty easily, right?
• 10:54
  SPEAKER_03: I could go to my uh, you know, C uh CFO, I could say, like, I need some money, and this is gonna make us money in this way, right?
• 11:00
  SPEAKER_03: And and it was, you know, generally, generally pretty easy to get that investment.
• 11:04
  SPEAKER_03: Like, you know, money was looking for ways to generate more money.
• 11:07
  SPEAKER_03: That changed, right?
• 11:08
  SPEAKER_03: Suddenly that changed with interest rates going up, and suddenly everyone tightened their belt.
• 11:12
  SPEAKER_03: And and everyone was looking to, you know, consolidate because that gave them more purchasing power, and they were looking to not grow their spend like linearly with revenue, right?
• 11:22
  SPEAKER_03: And you saw this with people pulling back on hiring, you've seen this with like many impacts, but one of the one of the big ones is in the observability market, observability has been seen as like, why am I paying so much to understand the behavior of my software?
• 11:34
  SPEAKER_03: So the response from the market's generally been discounts, generally been bundling, generally been like relatively kind of simple approaches.
• 11:41
  SPEAKER_03: And we wanted to do something a bit differently at Grafan Labs, right?
• 11:44
  SPEAKER_03: We didn't want to just do this by cutting into our margins, right?
• 11:47
  SPEAKER_03: The so we built technology to solve this.
• 11:49
  SPEAKER_03: We're fundamentally a tech company.
• 11:50
  SPEAKER_03: We built technology that identifies the data you actually use on a day-to-day, week-to-week basis, right?
• 11:56
  SPEAKER_03: And for the data that you don't use, that, you know, and it's commonly held that like you are sending a ton of data to an observability vendor that that that's going unused, you know, for that data that does go unused.
• 12:07
  SPEAKER_03: You have to pay for it.
• 12:08
  SPEAKER_03: Yeah, it's consumption-based, right?
• 12:09
  SPEAKER_03: For that data you're not using, we can, you know, we can give you ways of saving money on that.
• 12:13
  SPEAKER_03: We can aggregate away kind of high cardinality dimensionality of it.
• 12:17
  SPEAKER_03: We can, you know, we can sample it, we can, and we can do this all automatically.
• 12:21
  SPEAKER_03: So it's almost, you know, almost as if by magic you turn it on and you halve your bills with Grafana Cloud.
• 12:26
  SPEAKER_03: Right.
• 12:26
  SPEAKER_03: And that I think that's just, you know, illustrative of our approach.
• 12:30
  SPEAKER_03: We don't want to just do the standard thing that everyone's doing.
• 12:33
  SPEAKER_03: We want to try and find a more innovative approach.
• 12:35
  SPEAKER_03: You know, another example, this is not just about this one piece of technology.
• 12:38
  SPEAKER_03: You know, we're trying to align the value people get from their observability observability tools with the costs that they incur.
• 12:44
  SPEAKER_03: So another thing we've done uh in the past year has been launch a product called Bring Your Own Cloud, BYOC.
• 12:50
  SPEAKER_03: And this is where we will come for our very, very largest customers, we will come and run a full region of Grafana Cloud in their Amazon, Google, or Azure account, right?
• 13:00
  SPEAKER_03: And they pick up the cost for the hardware and they pay us a relatively flat licensing fee.
• 13:05
  SPEAKER_03: Right.
• 13:05
  SPEAKER_03: And so this for the for the massive like AI hyperscalers, this allows them to really subvert the usual consumption-based like SaaS economics, where effectively, you know, you're paying, you know, you're paying, you know, our Amazon bill plus our margin, right?
• 13:19
  SPEAKER_03: For for BYOC, you don't.
• 13:21
  SPEAKER_03: You can actually get this like an order of magnitude more cost-effective at scale.
• 13:25
  SPEAKER_03: And it's just part of that wider kind of brand promise of like innovative solutions to this, like this, this challenge with SaaS economics.
• 13:33
  SPEAKER_03: Right, right.
• 13:34
  SPEAKER_01: And and it is counterintuitive because you're building technology that reduces your revenue per customer, right?
• 13:40
  SPEAKER_01: Yeah.
• 13:40
  SPEAKER_01: And the alignment with that customer value is the kind of thing that's going to build that longer term trust and that uh retention.
• 13:47
  SPEAKER_01: How do you think about the trade-off internally?
• 13:49
  SPEAKER_01: So again, it might have been a little bit of a hard sell, right?
• 13:52
  SPEAKER_03: Yes, yeah.
• 13:52
  SPEAKER_03: I can remember when we um I can remember when we proposed this idea and like there were legitimately like seven-figure deals that the sales team were working on in Cycle that got halved, like when we when we launched this.
• 14:04
  SPEAKER_03: And I, you know, they like that because they get compensated based on the size that exactly, exactly.
• 14:08
  SPEAKER_03: Like understandably, the sales team were some of the members of the sales team were quite upset about this.
• 14:13
  SPEAKER_02: Yeah.
• 14:13
  SPEAKER_03: But I think in hindsight, or not, I think, like I know I've got a very good relationship with our with our go-to-market organization.
• 14:19
  SPEAKER_03: And in hindsight, they now realize that the majority of the customers that that choose Grafana Cloud choose it because of this reason, you know, because of this, right?
• 14:26
  SPEAKER_03: And fundamentally, yeah, there might be some short-term pain, but that long-term They're not losing deals because of pricing from that perspective.
• 14:34
  SPEAKER_01: Not at all.
• 14:35
  SPEAKER_01: So it's really fascinating.
• 14:37
  SPEAKER_01: Again, I I love your philosophy of you know, big tent, of openness, of transparency, of really focusing on the value proposition for the customer.
• 14:45
  SPEAKER_01: That's really all great stuff.
• 14:47
  SPEAKER_01: Um, I I also like to understand, you know, when you're out there in the marketplace, what beliefs are you encountering that that you have to disabuse people of, right?
• 14:56
  SPEAKER_01: What are the misconceptions?
• 14:58
  SPEAKER_01: What are the things that people don't understand about this space?
• 15:01
  SPEAKER_01: About, you know, and and you can define it broadly, it can be more about uh AI more broadly, it can be about observability.
• 15:08
  SPEAKER_01: What are the things that people are not understanding?
• 15:10
  SPEAKER_03: So I think the interesting one for me is, you know, we we sell observability tools and therefore like we're in inherently selling to like SREs, DevOps practitioners, like it's an inherently conservative space.
• 15:22
  SPEAKER_03: And so these are people who are held accountable for the reliability of systems, right?
• 15:27
  SPEAKER_03: And they take that very seriously.
• 15:29
  SPEAKER_03: And it does tend to mean in some ways they're a bit apprehensive of new technology, right?
• 15:34
  SPEAKER_03: And I think there was that famous kind of IBM quote, like a computer can't be held accountable, right?
• 15:38
  SPEAKER_03: So they're incredibly kind of apprehensive of AI, right?
• 15:42
  SPEAKER_03: And we are definitely like we we definitely saw when we first started launching kind of agentic capabilities in Grafana Cloud that the market was was a bit scared, right?
• 15:52
  SPEAKER_03: And um we had to do it in a very careful way, we had to build that level of trust.
• 15:56
  SPEAKER_03: You know, a lot of our kind of messaging around this was that we've built AI that's actually useful for our users.
• 16:01
  SPEAKER_03: So there was a lot of education for the first year or so about like what it's capable, what what's what the art of the possible is and how to do things safely, right?
• 16:10
  SPEAKER_03: So, you know, the nice thing about observability is most of it is read-only, like the the ability to actually go and like break production from an observability system is relatively low.
• 16:19
  SPEAKER_03: And so there's there's loads of opportunities to show people the value add, to show people you can make it easier to understand the behavior of a system, you can make it easier to generate visualizations that you can share with your team, you can make it easier to root cause analyze production incidents and easier and quicker.
• 16:34
  SPEAKER_03: And just demonstrating that and building that trust and you know, really kind of persuading people that um that what what a gentic AI is capable of was like the initial set of kind of, I guess, misunderstandings that the market had.
• 16:46
  SPEAKER_03: I will say I think we're at, you know, we're at the inflection point now in observability that um, you know, that that maybe development and and traditional software engineering was at a year ago, where now people believe that that this stuff is useful, right?
• 17:02
  SPEAKER_03: They they don't need convincing of what it's capable of.
• 17:05
  SPEAKER_03: And now the market's shifting to be like, how do we how do we make this more autonomous?
• 17:10
  SPEAKER_03: How do we really start to like take the recommendations coming out of the automated root cause analysis and apply them autonomously?
• 17:17
  SPEAKER_03: And that's raising a lot of really interesting questions about human in the loop, about you know, what does it mean for these systems to be given production access?
• 17:25
  SPEAKER_03: And you know, you've seen probably a lot of press over the last couple of weeks about like the number of instances Amazon's had and and how there's a I'm not sure how true this is, at least I don't work for Amazon, but like how there's a you know a higher incident of of AI generated code causing instance and reliability problems at Amazon, right?
• 17:43
  SPEAKER_03: So that's kind of where the market's at right now.
• 17:45
  SPEAKER_03: You've got people who are inherently conservative and held accountable for reliability, you know, starting to believe that these tools could radically transform the operation space.
• 17:55
  SPEAKER_03: And like I think we have a we have a very strong culture of reliability at Grafana Labs, as you can imagine.
• 17:59
  SPEAKER_03: We build tools that help people, you know, improve their reliability, and therefore we kind of practice that culture ourselves.
• 18:05
  SPEAKER_03: But we're also, you know, we're also very um we we rely heavily on a lot of automation.
• 18:10
  SPEAKER_03: It's why we're able to run so many regions of Grafana Cloud around the world and why we you know do that with a relatively small team.
• 18:16
  SPEAKER_03: And I think we've found ways to apply automated changes safely.
• 18:22
  SPEAKER_03: You know, we we have a lot of automated non-agenti checks for like blast radius to make sure that that any change doesn't like take out the entire system, for cost increases to make sure that any change doesn't suddenly like balloon a system up by a thousand X and blow our margin, right?
• 18:37
  SPEAKER_03: You know, many, many automated checks, and we we make sure a lot of our automated checks are non-agentic, so they can't be prompt injected.
• 18:44
  SPEAKER_03: And this is the way we're kind of starting to come to terms with like safely applying, you know, in a very controlled and confined spaces, like recommendations that are coming out of our root cause analysis technology.
• 18:55
  SPEAKER_01: You know, one of the things that we talked about uh was build build versus buy.
• 18:59
  SPEAKER_01: And you know, you you were saying that engineers want to build the thing that you're selling them, right?
• 19:04
  SPEAKER_01: And many times they underestimate the you know the level of effort or the complexity or whatever.
• 19:08
  SPEAKER_01: But uh talk a little bit about that.
• 19:10
  SPEAKER_01: We see it across software uh overall.
• 19:13
  SPEAKER_01: Um, but again, it's it's very strong with organizations have have deep engineering talent.
• 19:19
  SPEAKER_01: And what's what's the what's the misconception around build versus buy?
• 19:24
  SPEAKER_01: Because again, people are saying, well, I can build a lot of this stuff because it's you know, I have all these great tools and and accelerators and so on.
• 19:30
  SPEAKER_03: Yeah, this is the this is the SaaS pocalypse, right?
• 19:33
  SPEAKER_03: Like, you know, suddenly it's become significantly easier to build software, I mean, which is definitely true, and therefore that's going to lead to you know all SaaS companies not existing in the in the not so distant future.
• 19:44
  SPEAKER_03: I'm not sure it's ever been difficult to build software, though.
• 19:47
  SPEAKER_03: Like there's always been some young, naive engineer who's like, I can re-implement Salesforce in the weekend.
• 19:52
  SPEAKER_03: And honestly, there's some truth to that as well.
• 19:54
  SPEAKER_03: The value of, you know, let's pick on Salesforce here, is not is not that it's particularly good software.
• 20:00
  SPEAKER_03: I mean, no offense.
• 20:00
  SPEAKER_03: Like the value of Salesforce is the ecosystem it has, right?
• 20:03
  SPEAKER_03: The value of Salesforce is the integrations, the the the availability of people that know how to use it.
• 20:09
  SPEAKER_03: You know, all our sales team know how to use Salesforce.
• 20:12
  SPEAKER_03: We can't hire salespeople that know how to use other CRMs.
• 20:15
  SPEAKER_03: You know, or like the example of like it's also the accountability.
• 20:19
  SPEAKER_03: Salesforce have a team that are on the hook for making sure Salesforce works and is available.
• 20:24
  SPEAKER_03: And honestly, like a lot of companies don't want to have that accountability, don't want to be running software and and and you know, care and feeding software engineers to keep it alive so so that they can go and implement all their own technology.
• 20:36
  SPEAKER_03: And that hasn't changed fundamentally, right?
• 20:38
  SPEAKER_03: So I don't know.
• 20:39
  SPEAKER_03: I think you know, there are some interesting aspects to the AI software engineering that are going to drastically change the competitive dynamic here.
• 20:47
  SPEAKER_03: Like it's I'm I'm not sure the ecosystem around popular SaaS software is is as much of a moat as it used to be because integrating has suddenly become trivially easy, right?
• 20:57
  SPEAKER_03: Like the, you know, and there will of course be software companies that that are very good at running software, like I would include ourselves in that, for which like it's not actually a big lift to marginally add another service to our platform, you know, even for internal usage.
• 21:11
  SPEAKER_03: But yeah, fundamentally, like there are groups of companies that have the capability to build and run software, and like they will always just want to build and run software.
• 21:19
  SPEAKER_03: It's what they incentivize their team to do.
• 21:21
  SPEAKER_03: I've actually been trying to incentivize our team for the past few years to buy more software because like you know, it's uh you know, you start to like run out of engineers eventually if you just build everything.
• 21:31
  SPEAKER_03: But fundamentally, like we still build the vast majority of our software and we don't actually uh at least in engineering at Grafana Labs, in RD, we don't buy like a lot of software.
• 21:40
  SPEAKER_03: Like there's very few packages that we've we've bought in.
• 21:42
  SPEAKER_03: And it's kind of I recognize the hypocrisy of that as well, as a SaaS vendor, like I'm trying to encourage uh, you know, my customers to buy my SaaS, and I don't buy a lot of SaaS software myself.
• 21:52
  SPEAKER_03: You know, it's um it's it's an interesting it's an interesting dilemma.
• 21:55
  SPEAKER_03: I the last thing I guess I'll say on this is we we built a whole business.
• 22:00
  SPEAKER_03: Very early on, about kind of around this piece of software called Graphite.
• 22:03
  SPEAKER_03: So Graphite was an early time series database, like an early piece of monitoring software.
• 22:07
  SPEAKER_03: And the business we built about it was like Graphite was very trendy for a while.
• 22:11
  SPEAKER_03: You would get a bunch of engineers, download it, you know, build a build a service around it internally and monitor the behavior of their software using Graphite.
• 22:18
  SPEAKER_03: Very successful.
• 22:19
  SPEAKER_03: Those engineers would lose interest and move on to other projects.
• 22:22
  SPEAKER_03: And then the organization would be left holding the bag for an unmaintained piece of software.
• 22:27
  SPEAKER_03: And Grafana Labs would come in and offer them help with that.
• 22:30
  SPEAKER_03: We would offer to take over the maintenance of that and put it onto Grafana.
• 22:33
  SPEAKER_03: And that's how Grafana Cloud was born.
• 22:34
  SPEAKER_03: And it was very successful.
• 22:36
  SPEAKER_03: And that I think like that's really the risk of building versus buying.
• 22:40
  SPEAKER_03: Is like you let you know an engine a set of engineer, a very set of very capable engineers will go out and build a very good solution.
• 22:46
  SPEAKER_03: Be very excited about it.
• 22:47
  SPEAKER_01: We get a creation, solve the problems, and then once that's done.
• 22:51
  SPEAKER_01: And then what?
• 22:52
  SPEAKER_01: How do you make that sustainable long-term?
• 22:54
  SPEAKER_03: And I I interestingly want to go build something else.
• 22:58
  SPEAKER_03: And they will, right?
• 22:58
  SPEAKER_03: They they will vote with their feet, and if you don't let them, they'll go somewhere where they can.
• 23:02
  SPEAKER_03: The interestingly, I I think there's a ray of hope at the ray of light at the end of this tunnel because you can maybe in the future, if we get really good at that thing I was describing earlier of like automated, like agentic operations where the software runs itself and updates itself and fixes and patches itself and scales itself and tunes and tweaks itself, you know, in a safe way that also encourages like velocity and iteration.
• 23:26
  SPEAKER_03: Like maybe there's a world where this where AI can can allow us to operate software at scale with a much smaller set of uh set of people.
• 23:33
  SPEAKER_03: Sure.
• 23:33
  SPEAKER_03: You know, and a set of people that aren't necessarily the people that built that software and are experts in that software.
• 23:38
  SPEAKER_03: You know, maybe that expertise can be encoded in, you know, plain text that the the agents can understand.
• 23:44
  SPEAKER_03: You know, this is almost like going back to a dev versus ops world, but where the ops is done by by or majority of the ops is done by an agent.
• 23:52
  SPEAKER_03: We're not there yet, like, and there's there's definitely some work to get there.
• 23:55
  SPEAKER_01: Yeah, you're getting there incrementally.
• 23:56
  SPEAKER_01: You said something really interesting about how customers are asking how you build project products so quickly with a small team, and the answer um traces back to the engineering culture.
• 24:07
  SPEAKER_01: Tell us about the what you call the bazaar model.
• 24:10
  SPEAKER_03: Yes, yeah.
• 24:10
  SPEAKER_03: So this is like this is not my idea at all, by the way.
• 24:13
  SPEAKER_03: Like, this is a very famous book called The Cathedral and the Bazaar, and about like how open source software won over commercial software in the infrastructure space in particular.
• 24:22
  SPEAKER_03: And this is because it promotes this idea that an empowered engineer can scratch their own itch, can solve a problem that they acutely feel, and can often do that better than like a team of engineers and product managers trying to solve a problem that they don't actually experience themselves, right?
• 24:36
  SPEAKER_03: And so this is how most of like the you know the Linux ecosystem was built, right?
• 24:40
  SPEAKER_03: Is uh and this is the bizarre model, and it's all about kind of not centrally planned, like, you know, chaos almost like rains and and then the best stuff floats to the top.
• 24:50
  SPEAKER_03: And that's kind of how we try and run Grafana Labs internally, right?
• 24:52
  SPEAKER_03: We have a a group of incredibly empowered, autonomous engineers that are that are kind of told to go and do what they think is the right thing to do.
• 25:00
  SPEAKER_03: And it works very well for us because we build tools for software engineers, right?
• 25:05
  SPEAKER_03: And we build tools for software engineers with with a team of software engineers.
• 25:09
  SPEAKER_03: So we are our own biggest user, and that team can kind of be left to, to some extent, to their own devices to kind of figure out what the biggest pain point at any one moment with our software is and fix it and move on and iterate and continue to.
• 25:22
  SPEAKER_03: And and it's not just, I won't say it's completely insular, that paints a picture of the team just kind of sitting in a room and and ideating over and over again.
• 25:29
  SPEAKER_03: Like we also, with the majority of our software being open source, we get a really tight feedback loop that doesn't involve money, which is kind of an interesting angle, with our community, right?
• 25:38
  SPEAKER_03: And so our community use our software and give us feedback and and occasionally contribute changes to that software as well.
• 25:44
  SPEAKER_03: And you know, it's very easy for an engineer at Grafana Labs to go and talk to a user of our software because they just go and do it in the guise of open source, right?
• 25:52
  SPEAKER_03: And then, of course, like we've got customers and partners on top of that.
• 25:54
  SPEAKER_03: So we've got many, many layers here of like tight, tight feedback loops that allow the engineers to very quickly discover new problems to solve, get excited about them, meet people that have those problems, maybe have those problems themselves, and then solve them very quickly.
• 26:09
  SPEAKER_03: And we optimize in Grafana Labs for safely delivering code at very high speed.
• 26:14
  SPEAKER_03: You know, a very automated deployment process, very, very easy to put changes out and to test them at small scale without like without worrying our large user base and like and progressively deliver them to more and more people and de-risk the whole process.
• 26:29
  SPEAKER_03: And it's it's majority fully automated, gets us to that point where you know, uh, um an engineer with like high agency in Grafana Labs is able to really materially change the direction of the business in relatively like short, short order.
• 26:43
  SPEAKER_03: And that's like that describes like a startup, right?
• 26:45
  SPEAKER_03: Like a small company, and yet we're we're like, you know, we're 17, 1800 people now, right?
• 26:50
  SPEAKER_03: So like that's what I love.
• 26:52
  SPEAKER_03: We're also lots of small teams, right?
• 26:54
  SPEAKER_03: Like we, you know, I take the kind of Amazon two pizza team principle to heart, and we generally organize ourselves around a very large number of very small teams.
• 27:03
  SPEAKER_03: So the the trick there is very much like dividing responsibilities and workups so that teams can fully own a problem, you know, and that's that's not always possible, but like we try our best to do that, and that's how like, and once they can fully own a problem and a small group of people can keep effectively all the context for that in their head, they can be incredibly productive.
• 27:21
  SPEAKER_03: Right.
• 27:21
  SPEAKER_03: I introduced you know, I I don't want to sound like uh uh Savonia, like this is to most people in this space, this is relatively obvious.
• 27:28
  SPEAKER_03: It works very well in observability because it's engineers building software for other engineers.
• 27:32
  SPEAKER_03: I have no idea how you would do this in other markets where that's not the case, right?
• 27:37
  SPEAKER_03: You know, you hear horror stories about like um people building software for the legal profession, where you need like, you know, it takes years to become an expert in in the legal matters, and it takes years to become an expert in software engineering, and you it's very difficult to expect like a software engineer to be a legal expert and vice versa, right?
• 27:55
  SPEAKER_01: Right, right, right.
• 27:56
  SPEAKER_01: But what you're what you're doing is you're you're reducing the layers and the friction between the engineers solving the problem and the and the customers experiencing the problem because they're kind of one in the same.
• 28:06
  SPEAKER_03: Exactly.
• 28:07
  SPEAKER_01: And you're allowing for that massive throughput and that high velocity of problem solving, and then you're breaking it up into components that are manageable with a small team where people can get their heads around things rather than these, you know, enormous uh monolithic uh projects that no one can ever understand.
• 28:22
  SPEAKER_01: Exactly.
• 28:23
  SPEAKER_03: Yeah.
• 28:23
  SPEAKER_03: And then it's about like, you know, we famously like don't particularly publish a roadmap at Grafana Labs, right?
• 28:28
  SPEAKER_03: You know, because we want the ability to chop and change and be very flexible, but also because kind of it's very hard to predict what the outcome of this system will be.
• 28:35
  SPEAKER_03: It just just you look at the history, it tends to, you know, we we've tended to respond very quickly to changes in the market and and customer demands.
• 28:43
  SPEAKER_03: So that's what we tend to lean on.
• 28:45
  SPEAKER_03: But it's also about picking and choosing culture points that you want to emphasize in your organization that reinforce this.
• 28:52
  SPEAKER_03: So I I reference like, you know, we we run a lot of software, obviously, at scale at Grafana Labs, right?
• 28:57
  SPEAKER_03: We run like 50 regions of Grafana Cloud.
• 28:59
  SPEAKER_03: And every engineer at Grafana Labs at some point is on call for their software that they write.
• 29:03
  SPEAKER_03: So they are they're responsible for the performance and reliability and and cost of that software in production, right?
• 29:10
  SPEAKER_03: And you know, that blends two very distinct skills, like development and operations.
• 29:14
  SPEAKER_03: And you know, DevOps is a relatively common practice nowadays, but like one of the other things it really forces at Grafana Labs, it forces you as an engineer to use your own software, right?
• 29:24
  SPEAKER_03: It forces you to use Grafana.
• 29:25
  SPEAKER_03: Eat some cooking, yeah.
• 29:27
  SPEAKER_03: Exactly.
• 29:27
  SPEAKER_03: Yeah, eat your own dog food, we call it.
• 29:28
  SPEAKER_03: Yeah.
• 29:29
  SPEAKER_03: And it's in cooking, but okay.
• 29:31
  SPEAKER_03: And that, but it's that kind of forcing function that means you tend to then discover the problem, right?
• 29:36
  SPEAKER_03: You know, it's very nice to say, well, just go and use your software, but unless you've got a reason to, like everyone's under pressure, everyone's got priorities, everyone's got other stuff to do.
• 29:44
  SPEAKER_03: You know, unless you've actually got a day-to-day practice that forces you to log into Fana and try and figure out, you know, unless you're getting paged, you know, in the middle of the afternoon to be like, you know, your software's gone wrong.
• 29:54
  SPEAKER_03: You know, this is why we saw internally such good adoption of the AI tools we built.
• 29:58
  SPEAKER_03: You know, we built most of these tools to solve our own problems.
• 30:01
  SPEAKER_03: We were convinced we we only started making them open and and and and and available to customers once we were convinced they helped us.
• 30:07
  SPEAKER_03: And I think it was that as well.
• 30:09
  SPEAKER_03: It was that culture of dog fooding, that culture of DevOps, um, you know, and that that that culture of autonomous kind of empowerment that meant we actually built a decent, you know, a decent agentica system because it was the engineering feedback internally of like, oh, this stuff actually works.
• 30:24
  SPEAKER_03: This stuff's actually helpful.
• 30:25
  SPEAKER_03: This this stuff actually kind of, you know, accelerated my my on-call journey, helped me like get to the root cause quicker that convinced us that this was worth something and worth talking about.
• 30:34
  SPEAKER_01: Yep.
• 30:35
  SPEAKER_01: So so I think that's that makes a lot of sense.
• 30:37
  SPEAKER_01: And it's a very interesting balance.
• 30:39
  SPEAKER_01: And it's something that we talk to our customers about around AI adoption.
• 30:43
  SPEAKER_01: You know, organizations want to let a thousand flowers bloom, right?
• 30:46
  SPEAKER_01: And you want to have that creativity, you want to tap into that autonomy, that that ability, but they also feel they need to impose some top-down structure, right?
• 30:54
  SPEAKER_01: Because you can't, you know, just let everything be canada.
• 30:58
  SPEAKER_01: You're making the case that right moves emergent standards, not prescriptive.
• 31:02
  SPEAKER_01: So, how does that work in practice?
• 31:04
  SPEAKER_01: And, you know, what about things like semantic standards?
• 31:07
  SPEAKER_01: What are those important to your engineering and to your organization?
• 31:12
  SPEAKER_03: It's it's a bit of a you maybe hit a bit of a nerve with me now.
• 31:14
  SPEAKER_03: Yeah, like this is an interesting one, right?
• 31:16
  SPEAKER_03: So I think, you know, to come back to the very premise of the question, like a thousand flowers bloom is uh, you know, obvious to most people is a very way to to really encourage productivity and innovation, right?
• 31:27
  SPEAKER_03: So I don't think that's that's contentious.
• 31:29
  SPEAKER_03: It's the the the level of um top-down kind of consolidation standardization required that that's the contentious thing.
• 31:36
  SPEAKER_03: I think I think leaders tend to think more is needed than is actually needed.
• 31:40
  SPEAKER_03: Right.
• 31:40
  SPEAKER_03: So so firstly, like as long as you have a nice open culture of like sharing ideas and letting the best ideas float to the top naturally, and as long as you have a bit of patience, you know, the bad ideas will generally get forgotten pretty quickly.
• 31:54
  SPEAKER_03: Like we, you know, one of the things we don't do at Grafana Labs is have like a big monolithic issue tracker that doesn't allow the company to forget.
• 32:01
  SPEAKER_03: You know, we use we use GitHub issues, you know, we we close issues that go stale very quickly, and we have, you know, the institutional ability to forget bad ideas, right?
• 32:11
  SPEAKER_03: And that's like really important, I think, often overlooked.
• 32:14
  SPEAKER_03: So, first, like let the best ideas float to the top, encourage the the spread of those ideas and and and prototypes and implementations in the organization.
• 32:22
  SPEAKER_03: And you'll find naturally, most of the time, you didn't have to like be top-down at all.
• 32:27
  SPEAKER_03: Like the best idea won quite quickly, and and everyone standardized on the de facto standard, the emergent standard.
• 32:35
  SPEAKER_03: Right.
• 32:35
  SPEAKER_03: And then, you know, sometimes that doesn't happen.
• 32:37
  SPEAKER_03: Sometimes two two two um technologies or or solutions like float to the top and stay and and both reach that kind of escape velocity or critical mass or whatever you want to call it.
• 32:48
  SPEAKER_03: Then you've got to ask yourself, is that really a bad thing?
• 32:50
  SPEAKER_03: Like maybe you've got the the engineering bandwidth to sustain two, and maybe actually killing one and consolidating does more damage than good, right?
• 32:59
  SPEAKER_03: And you know, maybe, maybe not, I don't know.
• 33:02
  SPEAKER_03: It depends on the problem and the engine and the organization.
• 33:04
  SPEAKER_03: And then I'll say, like, you know, even in Grafana Labs for a while, there was still one team that deployed their services on EC2 VMs and everyone else was on Kubernetes, and eventually it took a little bit of top-down direction to be like, look, guys, you know, just just just just stop it and get on board.
• 33:18
  SPEAKER_03: And and it's little things like, you know, just a gentle nudge was all that was needed there.
• 33:23
  SPEAKER_03: But I think the main message is like patience, like often engineering leaders will say, like, oh, but look at the opportunity cost, we're wasting resources.
• 33:31
  SPEAKER_03: And I'm like, Yeah, but the cost of consolidation is often underestimated and is often dangerous to the culture because as soon as you start telling engineers that have poured their heart and soul into this project to drop it or to like go and go and do that other thing.
• 33:45
  SPEAKER_01: It's devastating to people.
• 33:46
  SPEAKER_01: It's like yeah, exactly.
• 33:55
  SPEAKER_03: Engineering is is like there's a lot of creativity to this.
• 33:58
  SPEAKER_03: There's a lot, you know, it isn't just like it isn't like designing a bridge, although I'm sure I've never designed a bridge, I'm sure there's creativity in that as well.
• 34:04
  SPEAKER_01: I'm sorry, you never did what?
• 34:05
  SPEAKER_03: Designed a bridge, like you know, traditional or like civil engineering is a lot of people.
• 34:09
  SPEAKER_01: All the civil engineers out there are gonna take offense to that.
• 34:11
  SPEAKER_01: Yes, yeah, or be uninvited for mechanism.
• 34:14
  SPEAKER_01: Anyway, carry on.
• 34:15
  SPEAKER_01: I understand your point.
• 34:16
  SPEAKER_03: There's a lot of, you know, you do need a lot of like, you know, product ideation and and a lot of free thinking in software engineering to find, you know, the I like I like the idea of the art of the possible, right?
• 34:28
  SPEAKER_03: It's about finding problems that are worth solving and that you're able to solve, you know, and that you know, we can solve almost any problem, but a lot of them just aren't like worth the effort, right?
• 34:36
  SPEAKER_03: And so you want to find that intersection.
• 34:37
  SPEAKER_03: And honestly, it's often just an engineer that's you know, a software engineer that's able to do that.
• 34:42
  SPEAKER_03: This is just again, not not an original thought, like the whole software engineers are the new kingmakers, like is the is the kind of general philosophy of explaining that, right?
• 34:51
  SPEAKER_03: Right, right, right.
• 34:52
  SPEAKER_01: And you know, we i I think the analogy to complex adaptive systems is useful here.
• 34:58
  SPEAKER_01: You're providing attractors and guardrails that you think of complexity, uh, and it's not a prescriptive blueprint, but the standards emerge from the work, the feedback loops keep things from going off the rails.
• 35:08
  SPEAKER_01: It's much more resilient than uh a top-down uh mandate.
• 35:12
  SPEAKER_01: Um so I think that that makes a lot of sense.
• 35:15
  SPEAKER_01: We kind of skirted the semantic, right?
• 35:18
  SPEAKER_01: But that's okay.
• 35:19
  SPEAKER_01: I think I mean I think semantic standards are are super important when it comes to unstructured um content and maybe uh your documentation uh and and those types of things abide by some uh kind of a framework, some semantic framework.
• 35:34
  SPEAKER_01: Maybe uh it's less important in the coding itself.
• 35:37
  SPEAKER_01: I I don't know.
• 35:38
  SPEAKER_01: I don't know enough about your space.
• 35:39
  SPEAKER_03: Yeah, I I I think I used to be in the camp of arguing that this stuff was more important because mostly like I used to make the argument to get the best out of like the Grafana system, for instance, or you know, a lot of these agentic tools.
• 35:51
  SPEAKER_03: They would they would make assumptions and almost like a leap of faith, and you know, and it's better when those assumptions turn out to be correct.
• 35:58
  SPEAKER_03: So if you do something one way, then other things should be done the same way, you know.
• 36:02
  SPEAKER_03: And so that's how I used to argue like, oh, it's better to just put all your data in Grafana Cloud.
• 36:06
  SPEAKER_03: It's better to have everything in consistent APIs and consistent query languages, you know, and consistent mod models, you know, it's better if the job label means the same thing everywhere.
• 36:15
  SPEAKER_02: Yeah.
• 36:15
  SPEAKER_01: Well, and MCPs require metadata structures and metadata standards, right?
• 36:20
  SPEAKER_03: So so what is the role of of Right, but that's actually a really good example because like if that used to be my thinking is like it's better if things are done in a regular way because it, you know, it allows higher levels of reasoning, right?
• 36:33
  SPEAKER_03: But I, you know, I'm becoming increasingly convinced it's not important, right?
• 36:36
  SPEAKER_03: And the MCP example is great because like I'm not using MCP anymore.
• 36:40
  SPEAKER_03: My my most of my tools are just using command line tools and you know, just you know, actually just calling dash H on the command to discover what the command's capable of doing, right?
• 36:47
  SPEAKER_03: And and I actually think like to a certain extent, MCP might even be going out of fashion, like for that reason.
• 36:53
  SPEAKER_03: So it's kind of interesting that the models, the newer models, are actually much more capable of dealing with inconsistency and dealing with multiple different ways of doing things and and competing standards.
• 37:04
  SPEAKER_03: Yeah, and we we did a lot of work and we have this knowledge graph in Grafana Cloud, which like tames a lot of the chaos in the open source, in the open source instrumentation world, right?
• 37:13
  SPEAKER_03: And kind of you know, makes it much more coherent and consistent and standardized and gives what we have definitely found is like the you know, if you've got the knowledge graph enabled and you're leveraging it properly, the assistant is able to more effectively root cause analyze instance, right?
• 37:29
  SPEAKER_03: But I've actually been impressed with how easy or or how how good the assistant is at root cause analyzing instance, even if you don't have the knowledge graph, even if it's having to go back to this inherently kind of inconsistent open source telemetry that that is just the Wild West.
• 37:44
  SPEAKER_03: Like because not these models have been trained on the open source code bases and that you know, and they actually do tend to know for some for some definition of know like what the different labels mean, what the different telemetry signals mean, even though they mean different things depending on the context.
• 37:59
  SPEAKER_03: Right.
• 38:00
  SPEAKER_03: So yeah, I I think these models, the newer ones, and I I know very little about like how they work or how they're built, but the emergent property seems to be they're able to deal with inconsistency a lot better recently.
• 38:11
  SPEAKER_01: Right.
• 38:12
  SPEAKER_01: I think it's a different that's it's a different use case than the ones that we see around say retrieval augmented generation, where you really are dealing with content that does need to be structured.
• 38:22
  SPEAKER_01: So I can see how you can have that kind of emergent uh capability and the ability for the tools to derive those structures and then to map those structures.
• 38:31
  SPEAKER_01: So so I I don't I I I I'll concede the argument uh from that perspective.
• 38:35
  SPEAKER_01: I mean, I've worked.
• 38:37
  SPEAKER_01: We're saying it works, it works, right?
• 38:39
  SPEAKER_03: Yeah, I mean it's starting to increasingly work, I think.
• 38:42
  SPEAKER_03: Like that's the uh that's the impressive thing.
• 38:44
  SPEAKER_03: Yeah, yeah.
• 38:45
  SPEAKER_01: Uh let's see.
• 38:46
  SPEAKER_01: You mentioned uh, you know, well, your your remote first company with 1,600 employees, and a lot of organizations struggle with that.
• 38:54
  SPEAKER_01: How do you maintain cross-functional relationships, uh, especially with go-to-market, you know, without a physical office?
• 39:02
  SPEAKER_01: And you know, obviously, people, you know, I've been in situations where even in physical offices, people will be in their their little respective locations and all be meeting virtually, right?
• 39:12
  SPEAKER_01: So I think it's less and less necessary, but it's still an interesting uh perspective because some things happen in a much more happenstance uh and and uh um you know lucky coincidence way with when you have physical proximity.
• 39:28
  SPEAKER_01: But what's the analogy to that in in your remote world?
• 39:31
  SPEAKER_03: Yeah, my um my impression, I mean, I haven't worked in an in-office environment for over a decade now.
• 39:38
  SPEAKER_03: But my impression is people massively overestimate the happenstance, like and actually the the common thing is that people stay in their little, you know, in their cubicle and do their work and go home and don't like the the Hawaii, the water cooler track doesn't exist to the extent people think it exists in in in-office companies, uh in in-person companies.
• 39:57
  SPEAKER_03: And so to be very clear, like you have to engineer that in a remote company, of course.
• 40:01
  SPEAKER_03: But I would even argue in an in-office culture, you shouldn't rely on it happening spontaneously.
• 40:05
  SPEAKER_03: You should engineer it as well, right?
• 40:07
  SPEAKER_03: Right.
• 40:07
  SPEAKER_03: And especially like, you know, the minute you reach any form of scale, like you're not gonna have 1,600 people in one office.
• 40:13
  SPEAKER_03: I'm sure there's some companies that do, but those companies probably have multiple offices with more than 1,600 people in, right?
• 40:19
  SPEAKER_03: You know, we we have customers all over the world.
• 40:21
  SPEAKER_03: We're always gonna have salespeople all over the world.
• 40:23
  SPEAKER_03: Why not have engineers all over the world as well and have those customers get to know their local engineering team?
• 40:29
  SPEAKER_03: Right.
• 40:29
  SPEAKER_03: And then, you know, we try and run engineering internally a lot like an open source project, right?
• 40:33
  SPEAKER_03: And in with so much of our software being open source, like you don't necessarily know, you know, the contributor where it's coming from.
• 40:40
  SPEAKER_03: You don't necessarily, you're not necessarily overlapping in time zone, right?
• 40:44
  SPEAKER_03: So you're encouraged to form strong kind of asynchronous communication patterns.
• 40:48
  SPEAKER_03: You're encouraged to write things down, share content asynchronously, like through through other presentations.
• 40:54
  SPEAKER_03: And that's just a much more inclusive way of working, right?
• 40:56
  SPEAKER_03: And it's a much more durable way of working as well.
• 40:59
  SPEAKER_03: Right.
• 40:59
  SPEAKER_03: So and then finally, like, and I my my pet peeve, like I can see in-office cultures working, to be very clear.
• 41:05
  SPEAKER_03: Like, I'm not anti-in-office, like, and I can see them working very well, right?
• 41:08
  SPEAKER_03: There's many successful companies, but the thing I really can't see working is hybrid, right?
• 41:13
  SPEAKER_03: Is where you have like some people in office and some people not in office because you've got all of the disadvantages of being remote, like you have to force people to communicate in writing or sit on, you know, sit in their desks and zoom in to a meeting, even if some people are in the same building as them.
• 41:27
  SPEAKER_03: And you've got all the disadvantages of being in office, like you're only like, you know, you have the uh the potential for like cliques to form and for people to feel excluded or feel like second-class citizens if they weren't in the room when the decision was taken.
• 41:39
  SPEAKER_03: Like, so I I for us, like the ability to attract talent no matter where it is, the ability to build teams that span the globe and therefore have like follow the sun on cool rotations, right?
• 41:50
  SPEAKER_03: The ability to be close to our customers and and be embedded in our custom, you know, in the different cultures around the world and you know, have have people sensitive to that.
• 41:58
  SPEAKER_03: And fundamentally, like you mentioned sales and and and engineering there.
• 42:02
  SPEAKER_03: It's it I find it really interesting for Grafana labs, it's the go-to-market teams that are are more keen on having an in-person presence, right, and wish we had an office occasionally.
• 42:11
  SPEAKER_03: And the engineering teams love not being in the office.
• 42:13
  SPEAKER_03: But I would always I would always see it as like logically the other way around, right?
• 42:16
  SPEAKER_03: The it's the engineering teams that should be the ones benefiting from the collaboration improvements, you know, and the communication enhancements from being in person.
• 42:25
  SPEAKER_03: And it's that those go-to-market organizations that I really wish just were always in the field, in with our customers spending time, not in the office, right?
• 42:32
  SPEAKER_03: So I find I find that kind of um I find that interesting, obviously biased because I'm an engineer, but so let's let's take a talk a little bit about the case for junior engineers, right?
• 42:42
  SPEAKER_01: I know that there's a narrative that AI will eliminate the need for junior software engineers, but you're taking a contrarian view of that.
• 42:50
  SPEAKER_01: Why don't you make that case?
• 42:51
  SPEAKER_03: Yeah, I I I like to pick contrarian views just you know, just for the sake of it half the time.
• 42:55
  SPEAKER_03: But they're more interesting, right?
• 42:57
  SPEAKER_03: Yeah, exactly.
• 42:58
  SPEAKER_03: As a remote first company, right, we've always struggled to onboard junior people.
• 43:02
  SPEAKER_03: I remember when I first started a professional software engineering, I was constantly bothering the person sat next to me who was my boss about how do I do this.
• 43:08
  SPEAKER_03: You mean I basically didn't know how to do code reviews, I didn't know how to use version control, I didn't know how to use an IDE.
• 43:14
  SPEAKER_03: Like, you know, I had a good academic background, but I was pretty naive when it came to software engineering.
• 43:19
  SPEAKER_03: And I was constantly bothering him.
• 43:21
  SPEAKER_03: And it's that tight kind of feedback and and and low cost of interruption that that bootstrapped me into my career, right?
• 43:28
  SPEAKER_03: And you just can't do that in a remote organization.
• 43:30
  SPEAKER_03: You've never found an effective way.
• 43:32
  SPEAKER_03: So as a consequence, Grafan Labs just basically has very, very few junior software engineers.
• 43:38
  SPEAKER_03: And the ones we do have are ones that have proved themselves like able to work in a remote company by effectively coming in through our open source communities, right?
• 43:47
  SPEAKER_03: They they picked up the project at university, contributed, became a known entity, and then as soon as they graduate, we tend to offer them a job, right?
• 43:54
  SPEAKER_03: Right.
• 43:54
  SPEAKER_03: They self selected, they self selected until it's such an exception to the rule, right?
• 43:58
  SPEAKER_03: Most people, you can't, it's not reasonable.
• 44:00
  SPEAKER_03: To expect most software engineers would behave like that.
• 44:02
  SPEAKER_03: Right.
• 44:03
  SPEAKER_03: But I will say, you know, yes, AI, the idea that AI might get rid of like the whole need to be a software engineer, or it presents like, how do I junior software engineer?
• 44:12
  SPEAKER_03: Or it also presents this like, how do I learn what good code looks like so that I can review the output of an AI if I never actually wrote code to begin with, because no one writes code anymore.
• 44:20
  SPEAKER_03: The counter though is like suddenly, as a remote first engineer, junior engineer, I've got something I can interrupt all the time, which is an LLM, right?
• 44:28
  SPEAKER_03: It's like it's like the death of Stack Overflow.
• 44:31
  SPEAKER_03: On Stack Overflow was where you used to go to ask stupid questions, and of course the first answer would be you're an idiot, and the second answer would be you're asking the wrong question.
• 44:37
  SPEAKER_03: Maybe the third or fourth answer is.
• 44:39
  SPEAKER_03: You're in the wrong forum.
• 44:40
  SPEAKER_01: Maybe you're asking the wrong question and it's the wrong place.
• 44:44
  SPEAKER_03: So then like you can now go and like very with very low kind of like ego ask an LLM a stupid question because no one's just suddenly it might be possible to onboard with a kind of agentic buddy, like onboard a junior engineer into an organization again.
• 45:00
  SPEAKER_03: And I will also say like it's the junior engineers, you know, coming out of university now who are actually, in my experience, more capable with the AI tools because they've just been using them from an from an earlier age.
• 45:12
  SPEAKER_03: You know, they've not they don't really know anything else, right?
• 45:15
  SPEAKER_03: Like they are, you know, they're more comfortable.
• 45:18
  SPEAKER_03: It's it's it's sometimes, and I'm I'm grossly stereotyping and overestimating here, but like it's sometimes the older, you know, senior tenured engineers who are like, Oh, I don't need any of this AI tooling.
• 45:27
  SPEAKER_03: Like I'll write it all myself the old-fashioned way, and I'm gonna just, you know, you know, dial all the ones and zeros in, and that's how we write software now.
• 45:33
  SPEAKER_03: You know, it's it's I write in machine code, so yeah, exactly.
• 45:36
  SPEAKER_03: Yeah.
• 45:37
  SPEAKER_03: And so I I think it's the um, you know, honestly, I I'm I'm I'm relatively kind of optimistic.
• 45:42
  SPEAKER_03: Like I think, you know, this this breed of tooling and and technology and will help us, we'll help Grafana Labs in particular, on board more junior engineers.
• 45:51
  SPEAKER_01: Yep, that makes a lot of sense.
• 45:52
  SPEAKER_01: So this has been a great, wide-ranging conversation.
• 45:55
  SPEAKER_01: I really appreciate your time, but if you could leave the listeners one or two things to think about, whether you're evaluating their observability strategy, rethinking their engineering culture, figuring out how AI fits in operations, what do you want them to walk away with?
• 46:07
  SPEAKER_03: The one thing that I've learned about the AI ecosystem and market at the moment is how quickly it's changing.
• 46:14
  SPEAKER_03: Right.
• 46:14
  SPEAKER_03: So all of the assumptions I had, even I think Seth, when you and I first have been radically changed in the space of a few months, right?
• 46:21
  SPEAKER_03: Like, so the one, you know, I know it's a cliche, but the one constant here is change.
• 46:25
  SPEAKER_03: I just did not realize how true that was and how much you have to constantly be learning and trying and experimenting and staying, you know, staying in touch with with the uh with the latest developments.
• 46:36
  SPEAKER_03: And that's always been true, but it's even more true now.
• 46:39
  SPEAKER_03: So like, you know, everything I you know, I suspect every conversation I have about AI, I kind of end with saying this is not going to be a very evergreen conversation.
• 46:46
  SPEAKER_03: Because I suspect half the stuff that I've said will be provably wrong in six months' time.
• 46:51
  SPEAKER_03: And I'm kind of excited by that.
• 46:52
  SPEAKER_03: I think it's I, you know, you've got to you've got to be excited by that, or else, like, you know, it's incredibly scary.
• 46:57
  SPEAKER_03: I think it's a little bit of both.
• 46:58
  SPEAKER_01: So I think uh for our listeners that the the through line here is that openness, whether it's open source, open standards, open culture, is not just a philosophy.
• 47:06
  SPEAKER_01: It really is that competitive strategy uh lowers switching costs and you know builds trust.
• 47:11
  SPEAKER_01: It really is is uh and in the area of AI turns out to be the best way to make sure your models know how to use your technology, right?
• 47:18
  SPEAKER_01: The whole uh proprietary keep it behind a a firewall or paywall or or whatever means there's less visibility.
• 47:25
  SPEAKER_01: And and your approach means that you're being referenced in in GEO and and and search uh generative search results, whereas your competitors are not.
• 47:36
  SPEAKER_01: So I think it's really fascinating to hear that in practice.
• 47:39
  SPEAKER_01: And and Tom, thank you for your time today.
• 47:43
  SPEAKER_01: Where can people find you?
• 47:44
  SPEAKER_01: We'll have some information in the show notes, but where do you where do you want to point them?
• 47:47
  SPEAKER_03: Yeah, just check out grafano.com or any of our projects on GitHub, and uh you'll see me at like every one of our events and and many kind of open source concert uh conferences as well.
• 47:56
  SPEAKER_03: So thank you very much for having me, Seth.
• 47:57
  SPEAKER_03: I really enjoyed that.
• 47:58
  SPEAKER_01: Yes, that's Tom Wilkie.
• 47:59
  SPEAKER_01: He's VP of product at Grafano Labs.
• 48:01
  SPEAKER_01: And thank you for joining us, and thank you for everybody listening, and we'll see you next time at the next Early AI podcast.
• 48:09
  SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
• 48:13
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 48:19
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science and Vector.
• 48:24
  SPEAKER_00: To learn more, visit early.com.
• 48:27
  SPEAKER_00: That's E A R L D Y dot com.
• 48:30
  SPEAKER_00: Thank you for tuning in.