Latest / The Pragmatic Engineer / Measuring the impact of AI on software engineering – with Laura Tacho

Transcript

Measuring the impact of AI on software engineering – with Laura Tacho · The Pragmatic Engineer

0:00
SPEAKER_01: What happens with the time saving as a developer?
0:02
SPEAKER_01: What would that mean for me as a developer?
0:04
SPEAKER_00: When Dora researched this question,
0:06
what they found was that many developers were actually feeling
0:09
less satisfied
0:10
SPEAKER_00: because AI is accelerating the parts that they enjoy.
0:14
SPEAKER_00: And so what was left over was more stuff that they didn't enjoy,
0:18
SPEAKER_00: the toil,
0:19
SPEAKER_00: the meetings,
0:20
SPEAKER_00: the administrative work.
0:21
SPEAKER_00: It gave me pause when I read that.
0:22
SPEAKER_01: What is the actual impact of AI tools on software engineering?
0:26
SPEAKER_01: Lara Taco is a CTO at DX,
0:28
SPEAKER_01: a company with a mission to measure developer productivity with data,
0:31
SPEAKER_01: and has been doing so even before AI tools went mainstream from 2022.
0:34
SPEAKER_01: Today,
0:35
we discuss why most of the hype in the media about AI gets
0:38
things wrong thanks to
0:39
SPEAKER_01: oversimplification,
0:41
SPEAKER_01: and why the burden is on us engineers to set the record straight.
0:44
SPEAKER_01: The actual data of the impact of rolling out AI tools for development at
0:48
SPEAKER_01: companiesbooking.com and WorkHuman.
0:50
How developers report that their most timing saving use case is
0:53
not actually AI
0:54
SPEAKER_01: code generation,
0:56
SPEAKER_01: but debugging tricky stack traces and doing it faster.
0:59
SPEAKER_01: The paradox of AI tools,
1:01
how using AI tools to help with coding can make developers less
1:04
satisfied with our
1:04
SPEAKER_01: jobs because we actually like to code.
1:07
SPEAKER_01: And many more interesting topics.
1:09
If you're a lead or engineer interested in data about what
1:12
works today with AI and
1:13
SPEAKER_01: how to stay on the ground with all the hype in the media,
1:16
SPEAKER_01: this episode is for you.
1:17
If you enjoy the podcast,
1:18
please subscribe to it on any podcast platform and on YouTube.
1:22
SPEAKER_01: All right, so Laura, welcome to the podcast.
1:25
SPEAKER_00: Hey, Gerger.
1:26
SPEAKER_00: Good to see you again.
1:27
SPEAKER_01: So to kick off,
1:28
SPEAKER_01: one thing I hear a lot,
1:30
SPEAKER_01: and I see it as well,
1:31
SPEAKER_01: is how AI is and has been and remains overhyped.
1:36
SPEAKER_01: So when I look at media headlines,
1:38
SPEAKER_01: some of the headlines are just frankly,
1:41
SPEAKER_01: they sound ridiculous.
1:43
SPEAKER_01: There are some headlines around how... Actually, let me read you a few.
1:48
Some are a bit scary as developers,
1:50
and some just feel like over the moon.
1:52
SPEAKER_01: So...
1:53
SPEAKER_01: Here's one from Forbes.
1:55
SPEAKER_01: Are coders jobs at risk?
1:56
SPEAKER_01: AI's impact on the future of programming.
1:58
SPEAKER_01: CIO Magazine.
2:00
SPEAKER_01: AI coding assistants wave goodbye to junior developers.
2:03
SPEAKER_01: Gizmodo.
2:04
SPEAKER_01: OpenAI just released a coding tool to,
2:06
SPEAKER_01: quote,
2:07
SPEAKER_01: help programmers,
2:08
SPEAKER_01: in brackets,
2:09
SPEAKER_01: replace their jobs,
2:09
SPEAKER_01: probably.
2:10
SPEAKER_01: Software development may never be the same.
2:12
SPEAKER_01: Now, these are all, like, you know, they're not, like,
2:15
SPEAKER_01: tech,
2:15
SPEAKER_01: like not publications where software engineer would say like,
2:18
SPEAKER_01: okay,
2:18
SPEAKER_01: they're like amazing with tech,
2:19
SPEAKER_01: but you know,
2:20
SPEAKER_01: Gizmodo and Forbes,
2:21
like decision makers read them and they often get forwarded as
2:26
developers or you
2:27
SPEAKER_01: see them.
2:28
SPEAKER_01: What is your take on what we're hearing with mainstream media?
2:31
SPEAKER_01: And as you're talking with developers and engineering leaders,
2:34
SPEAKER_01: how are,
2:35
SPEAKER_01: what are they telling you about these?
2:37
SPEAKER_00: Yeah.
2:37
SPEAKER_00: I mean, these headlines are headlines for a reason, right?
2:40
SPEAKER_00: They get clicks, they get engagement, they're sensational.
2:43
SPEAKER_01: They're at supported media usually.
2:46
SPEAKER_00: Absolutely.
2:47
And I think anytime,
2:49
I think in general,
2:51
media literacy is really important.
2:53
SPEAKER_00: Data literacy is really important.
2:54
SPEAKER_00: And this is no different.
2:56
So whenever I see a headline like that,
2:58
I always trace it back to the money.
3:00
SPEAKER_00: Like who's getting paid?
3:02
SPEAKER_00: As you said, it's at supported media.
3:03
SPEAKER_00: Who's being covered?
3:04
SPEAKER_00: Are they a vendor?
3:05
SPEAKER_00: Are they selling an AI tool?
3:07
SPEAKER_00: Like I ask all of these questions.
3:08
SPEAKER_00: You should be asking all of these questions as well.
3:10
SPEAKER_01: Are they paying for coverage?
3:12
SPEAKER_01: Sometimes they are.
3:13
SPEAKER_01: And even when they're not paying,
3:14
they might be paying PR agencies who pitch these ideas to
3:17
magazines and the
3:18
SPEAKER_01: magazines who need to do output.
3:21
SPEAKER_01: So they, for example, they're being paid by ads.
3:23
SPEAKER_01: They will often produce these shallow articles,
3:25
SPEAKER_01: which are kind of pre-written for them,
3:26
SPEAKER_01: by the way.
3:27
SPEAKER_01: This is a fun fact.
3:28
SPEAKER_00: I think there's a challenge in AI,
3:30
which is that simplifying something to the point where it can
3:34
be understood by
3:35
someone who doesn't have a background in development can
3:39
oversimplify to the point
3:40
SPEAKER_00: of being incorrect.
3:41
And so one recent example that I read was actually in the Wall
3:44
Street Journal,
3:45
SPEAKER_00: and they talked about how companies have AI employees.
3:50
SPEAKER_00: And these AI employees had line managers.
3:53
SPEAKER_00: I'm using air quotes here.
3:55
SPEAKER_00: Line managers and company credentials.
3:58
SPEAKER_00: Well,
3:59
SPEAKER_00: sure,
3:59
you could think about Copilot or Claude or any agentic workflow
4:05
as having a line
4:06
manager being the person who's dispatching the work or
4:10
verifying the work.
4:11
SPEAKER_01: Or insatiated the agent, yeah.
4:14
SPEAKER_00: Yeah, or like company credentials.
4:17
SPEAKER_00: Does that mean it has access to commit on GitHub?
4:20
Like,
4:20
does my Dependabot have company credentials because it can open
4:23
a pull request?
4:25
SPEAKER_00: And so, yeah, I could see in a journalistic world that is accurate.
4:31
SPEAKER_00: But is that really a reflection?
4:33
Is there an engineering manager out there hiring AI agents as
4:38
employees and giving
4:39
SPEAKER_00: them company email addresses?
4:40
SPEAKER_00: Like, that's just not really what's happening.
4:42
SPEAKER_00: And I think the...
4:43
SPEAKER_00: oversimplification can be really sensational.
4:47
SPEAKER_00: And everyone wants a piece of the AI hype right now.
4:50
SPEAKER_00: So those are the things that I think about when I read this.
4:54
I think it's really unfortunate because it puts a big burden on
4:57
engineering leaders
4:58
SPEAKER_00: to educate their business counterparts who maybe don't have,
5:03
they just don't have the background knowledge and experience to
5:05
understand what is
5:07
SPEAKER_00: authentic and what is not authentic.
5:09
SPEAKER_00: What are the real limitations of these tools?
5:10
SPEAKER_00: What is possible now?
5:12
SPEAKER_00: And,
5:13
SPEAKER_00: you know,
5:13
SPEAKER_00: it is our job,
5:14
SPEAKER_00: like it or not,
5:15
to be that person who can translate those and explain it
5:19
because ultimately it
5:19
hurts us and it hurts developers when we don't do that because
5:23
nobody wins in this
5:24
SPEAKER_00: hype cycle.
5:25
SPEAKER_00: Developers are like, this is super gimmicky.
5:28
SPEAKER_00: That's another reason for lower adoption.
5:29
SPEAKER_00: Like this is super gimmicky.
5:31
SPEAKER_00: There's no way that this is going to work as good as it says.
5:33
SPEAKER_00: They try it once.
5:35
SPEAKER_00: It gives them spaghetti code and they're like, this is just a load of BS.
5:38
SPEAKER_00: And then on the other side, on the executive side,
5:41
There's CEOs saying like,
5:42
hey,
5:42
I heard that Microsoft is writing 30% of their code with AI.
5:46
SPEAKER_00: Like, why aren't we doing that?
5:48
SPEAKER_00: You know,
5:48
those headlines suggest that 30% of Microsoft's code that's
5:51
running in production
5:52
SPEAKER_00: was authored by AI.
5:54
SPEAKER_00: That is not at all realistic.
5:57
We don't have data to support that from any of the companies
5:59
that we work with.
6:00
SPEAKER_00: And,
6:01
SPEAKER_00: you know,
6:01
SPEAKER_00: hundreds of companies,
6:02
SPEAKER_00: we've never seen data consistent with that kind of sensational claim.
6:06
SPEAKER_01: If you want to build a great product, you have to ship quickly.
6:09
SPEAKER_01: But how do you know what works?
6:11
SPEAKER_01: More importantly, how do you avoid shipping things that don't work?
6:15
SPEAKER_01: The answer, Statsig.
6:18
SPEAKER_01: Statsig is a unified platform for flags,
6:20
SPEAKER_01: analytics,
6:21
SPEAKER_01: experiments,
6:21
SPEAKER_01: and more,
6:22
combining five plus products into a single platform with a
6:25
unified set of data.
6:27
SPEAKER_01: Here's how it works.
6:28
SPEAKER_01: First, StatSick helps you ship a feature via FeatureFly or Config.
6:33
SPEAKER_01: Then it measures how it's working,
6:35
SPEAKER_01: from alerts and errors,
6:36
SPEAKER_01: to replays of people using that feature,
6:38
SPEAKER_01: to measurement of top-line impact.
6:41
SPEAKER_01: Then you get your analytics,
6:42
SPEAKER_01: user account metrics,
6:43
SPEAKER_01: and dashboards to track your progress over time,
6:45
SPEAKER_01: all linked to the stuff you ship.
6:47
SPEAKER_01: Even better,
6:48
SPEAKER_01: Statsik is incredibly affordable with a super generous free tier,
6:52
SPEAKER_01: a starter program with $50,000 of free credits,
6:54
SPEAKER_01: and custom plans to help you consolidate your existing spend on flags,
6:58
SPEAKER_01: analytics,
6:58
SPEAKER_01: or A-B testing tools.
7:00
SPEAKER_01: To get started, go to statsik.com slash pragmatic.
7:03
SPEAKER_01: That is S-T-A-T-S-I-G dot com slash pragmatic.
7:07
SPEAKER_01: Happy building.
7:08
SPEAKER_01: This episode was brought to you by Graphite,
7:10
SPEAKER_01: the developer productivity platform that helps developers create,
7:13
SPEAKER_01: review,
7:14
SPEAKER_01: and merge smaller code changes,
7:15
SPEAKER_01: stay unblocked,
7:16
SPEAKER_01: and ship faster.
7:18
SPEAKER_01: Code review is a huge time sink for engineering teams.
7:21
Most developers spend about a day per week or more reviewing
7:24
code or blocked
7:25
SPEAKER_01: waiting for a review.
7:27
SPEAKER_01: It doesn't have to be this way.
7:29
SPEAKER_01: Graphite brings stack pull requests,
7:31
the workflow at the heart of the best-in-class internal code
7:33
review tools at
7:34
SPEAKER_01: companies like Meta and Google,
7:35
SPEAKER_01: to every software company on GitHub.
7:38
SPEAKER_01: Graphite also leverages high-signal,
7:40
code-based-aware AI to give developers immediate actionable
7:43
feedback on their pull
7:44
SPEAKER_01: requests,
7:45
SPEAKER_01: allowing teams to cut down on review cycles.
7:48
SPEAKER_01: Tens of thousands of developers at top companies like Asana,
7:51
SPEAKER_01: Ramp,
7:51
SPEAKER_01: Tekton,
7:52
SPEAKER_01: and Vercel rely on Graphite every day.
7:55
Start stacking with Graphite today for free and reduce your
7:58
time to merge from days to hours.
8:00
SPEAKER_01: Get started at gt.dev slash pragmatic.
8:03
SPEAKER_01: That is g for graphite, t for technology.dev slash pragmatic.
8:07
SPEAKER_01: At Google, they had, I think, 25% a few months ago, a similar claim.
8:11
SPEAKER_01: And then I talked with someone at Google,
8:13
SPEAKER_01: and they actually brought up saying,
8:15
SPEAKER_01: like,
8:15
SPEAKER_01: you know what?
8:16
They're like,
8:16
yeah,
8:17
we have all these,
8:17
like,
8:17
AI integrations,
8:18
AI code review.
8:20
SPEAKER_01: We have the autocomplete.
8:21
SPEAKER_01: They have pretty much,
8:23
SPEAKER_01: as usual,
8:24
SPEAKER_01: you know this because you also talk with Google,
8:25
SPEAKER_01: but they have the whole...
8:28
internally they have the the stack that's available for anyone
8:31
using cursor etc
8:32
just the internal version which is kind of trained on google's
8:36
uh data and this
8:37
engineer was saying like i'm pretty sure they're counting
8:40
accepted completions as
8:42
ai generated but that just seems weird because like yeah i
8:46
accept it when it makes
8:47
SPEAKER_01: sense but i'm i'm like reading it and i'm i'm
8:50
reading it out,
8:51
then they're like,
8:52
we don't know where this number comes from.
8:54
SPEAKER_01: Like, who's measuring it?
8:55
SPEAKER_01: They didn't even tell them.
8:56
SPEAKER_01: And is acceptance really AI generated?
8:58
SPEAKER_01: Well, technically, yes.
8:59
SPEAKER_01: But even before AI,
9:01
then we could have said it was machine generated a lot of your
9:03
code because
9:04
SPEAKER_01: autocomplete has always been very good at predicting,
9:07
SPEAKER_01: you know,
9:07
SPEAKER_01: as your start to type out the first two letters of a class,
9:10
SPEAKER_01: was our code machine generated?
9:11
SPEAKER_01: Technically, yes.
9:12
SPEAKER_01: But so yes, I agree.
9:13
SPEAKER_01: It's confusing.
9:14
SPEAKER_00: Yeah.
9:15
SPEAKER_00: And I think your point about acceptance rate is exactly it.
9:17
That's a lot of these studies that are producing these numbers
9:20
for headlines are
9:21
SPEAKER_00: looking at accepted suggestions.
9:24
SPEAKER_00: And that's just not a great measure of the first of all,
9:28
SPEAKER_00: the business impact,
9:29
SPEAKER_00: but also even if that code made it to production.
9:32
SPEAKER_00: There is absolutely no,
9:33
there's not a line between I accepted the suggestion and now
9:36
it's running in
9:37
SPEAKER_00: production.
9:37
SPEAKER_00: So that's that's quite misleading.
9:39
I think,
9:40
you know,
9:40
we could say are 30 percent of pull requests being assisted by
9:46
AI?
9:46
SPEAKER_00: Like, I think that is probably true at a majority of companies.
9:49
SPEAKER_00: And so that's just a really different approach.
9:53
SPEAKER_00: magnitude of influence than 30% of code being written by AI.
9:57
We can also say that 100% of PRs at most companies or like most
10:02
larger companies
10:03
SPEAKER_01: have been kind of robotically checked,
10:06
SPEAKER_01: right?
10:06
SPEAKER_01: Like the linter has been run on them.
10:08
SPEAKER_01: Static analysis has run on them.
10:09
SPEAKER_01: This has been going on for years and it catches all the obvious things.
10:14
SPEAKER_01: So
10:15
SPEAKER_00: I mean,
10:15
SPEAKER_00: can you imagine that headline,
10:16
SPEAKER_00: like,
10:17
SPEAKER_00: ACME Corp only ships code to production that's been,
10:20
SPEAKER_00: you know,
10:21
SPEAKER_00: read by robots.
10:22
SPEAKER_00: Is this the end of software engineering?
10:24
SPEAKER_00: Like,
10:24
SPEAKER_00: you know,
10:25
we could certainly come up with a sensational headline to just
10:28
talk about CI,
10:29
SPEAKER_00: CD,
10:29
SPEAKER_00: as you said.
10:30
SPEAKER_01: You talk with a lot of engineering leaders, right?
10:32
SPEAKER_01: Like on a company's engineering leaders, tech leads, those kind of folks.
10:36
These days,
10:37
what are some of the most common questions you get related,
10:40
obviously,
10:41
to AI?
10:42
And what kind of sentiment are you gathering when it comes to
10:46
AI from these same
10:47
SPEAKER_01: engineering leaders?
10:48
SPEAKER_00: Yeah, I think the most common question I get is, what should I be doing?
10:56
I think as engineering leaders,
10:58
we have operated in a space where we can pattern match.
11:02
SPEAKER_00: in a lot of things, right?
11:04
SPEAKER_00: Like if I'm trying to modernize my CICD pipeline,
11:07
I can go talk to another customer of a different tool and see
11:10
what they're doing
11:11
SPEAKER_00: and how they've modernized and look at their before and after.
11:13
And in AI,
11:14
we just don't have that because it's frontier work for
11:17
everyone.
11:18
SPEAKER_00: And I think that's very exciting,
11:19
but also very distressing when you're holding the bag of money
11:22
and you have to
11:23
SPEAKER_00: figure out where you should spend that money.
11:26
SPEAKER_00: That's really tricky.
11:27
SPEAKER_00: So what should I do
11:29
SPEAKER_00: The other thing is,
11:30
SPEAKER_00: you know,
11:30
how do I measure it and how do I prove that I've made the right
11:33
decisions right
11:34
SPEAKER_00: now?
11:34
Because that is something that every engineering leader is
11:37
being held to account by
11:39
SPEAKER_00: their exec team,
11:40
SPEAKER_00: by their board.
11:41
SPEAKER_00: How are you investing in AI and can you show me the results?
11:46
SPEAKER_01: Tapping exactly on this, how can we actually measure the impact of AI?
11:52
SPEAKER_01: Actually, in the pragmatic engineering, we did some deep dives.
11:54
SPEAKER_01: Actually,
11:54
we did one of them with you on how we figured out all the
11:58
things that don't work,
11:59
SPEAKER_01: like lines of code or the single metrics,
12:01
SPEAKER_01: and we started to make progress with things like a space framework,
12:03
SPEAKER_01: later the DevX framework.
12:05
What are measurements here that might work even for developer
12:08
productivity or just
12:09
SPEAKER_01: measuring the efficiency of AI?
12:11
SPEAKER_01: What have you seen work out?
12:13
SPEAKER_00: Yeah,
12:13
SPEAKER_00: what's so tricky is that,
12:15
SPEAKER_00: as you said,
12:15
developer productivity is just that it's a really hard problem
12:19
on its own.
12:20
And when we add AI on top of it,
12:22
we certainly don't reduce complexity here.
12:24
SPEAKER_00: So,
12:25
SPEAKER_00: you know,
12:25
companies that have invested a lot in understanding developer
12:28
experience and
12:29
developer productivity are in a better spot right now to
12:32
understand the impact
12:33
SPEAKER_00: because they have that baseline of understanding of how their teams and
12:38
SPEAKER_00: organization operated before.
12:40
SPEAKER_00: And then we can just do experimentation style.
12:43
SPEAKER_00: We can look at what the impact of AI is.
12:45
I think for any leader who is out there sort of feeling like
12:49
lost in the forest,
12:50
not quite understanding what are even the measurements to look
12:54
for when it comes to
12:56
SPEAKER_00: telling the story about the impact of AI,
12:59
SPEAKER_00: Abhinoda,
12:59
SPEAKER_00: who's the co-founder of DX,
13:01
SPEAKER_00: and I have just put together a new AI measurement framework,
13:05
SPEAKER_00: which I'll just share on the screen.
13:06
SPEAKER_00: We can kind of talk through it.
13:07
SPEAKER_00: This is the DX AI measurement framework.
13:10
And what we recommend based on our field experience working
13:13
with hundreds of
13:14
SPEAKER_00: companies who have been working with AI from the infancy,
13:18
SPEAKER_00: the very beginning when AI was a glimmer in everyone's eyes,
13:21
SPEAKER_00: to now sort of full-scale rollouts where they're seeing some
13:24
SPEAKER_00: pretty impressive results from using AI,
13:28
SPEAKER_00: want to look at it across to utilization,
13:31
SPEAKER_00: impact and costs.
13:32
And these are really the three areas that will give you
13:36
together a really complete
13:37
SPEAKER_00: picture about how AI is working or not working,
13:41
SPEAKER_00: what you should do next,
13:42
SPEAKER_00: and then how you can tell that story about impact in your organization.
13:46
SPEAKER_01: Yeah, because I guess in the end, everyone's looking for impact, right?
13:51
SPEAKER_01: It should result in something tangible.
13:55
SPEAKER_01: Am I feeling this right in terms of either you're,
13:58
SPEAKER_01: as a software engineering organization,
13:59
SPEAKER_01: either you're building more stuff,
14:02
SPEAKER_01: building better stuff,
14:03
SPEAKER_01: or generating more revenue?
14:05
SPEAKER_01: Somehow,
14:06
SPEAKER_01: if it doesn't help with any of those or some related things,
14:09
SPEAKER_01: then what am I even doing,
14:10
SPEAKER_01: right?
14:11
SPEAKER_01: Yeah.
14:11
SPEAKER_00: Yeah,
14:12
SPEAKER_00: and I think,
14:13
SPEAKER_00: you know,
14:13
SPEAKER_00: as an industry,
14:14
SPEAKER_00: we've looked for output metrics to kind of quantify that end result.
14:19
SPEAKER_00: And at the beginning,
14:20
and actually what you'll see in a lot of the headlines is about
14:24
the quantity of
14:25
SPEAKER_00: code that can be produced with AI.
14:27
SPEAKER_00: But this is really disconnected from everything we know about developer
14:31
SPEAKER_00: productivity,
14:32
SPEAKER_00: developer experience,
14:33
SPEAKER_00: like quantity of code doesn't actually mean business impact.
14:37
SPEAKER_00: And so when we think about measuring the impact of AI,
14:39
SPEAKER_00: we need to sort of track it across the lifecycle,
14:42
SPEAKER_00: but also really stay focused on the end result,
14:44
SPEAKER_00: which is,
14:44
SPEAKER_00: as you said,
14:45
SPEAKER_00: it's more revenue,
14:46
SPEAKER_00: it's reduced cognitive load for developers,
14:48
SPEAKER_00: it's a better developer experience,
14:50
SPEAKER_00: it's more time,
14:51
SPEAKER_00: more time to innovate.
14:53
SPEAKER_00: These things are really important and focusing on something like
14:56
SPEAKER_00: acceptance rate,
14:57
SPEAKER_00: for example,
14:57
SPEAKER_00: just by itself isn't really going to tell you the whole story there.
15:01
I wonder if we're going to have a bit of a speed run of what
15:03
we've learned about
15:04
SPEAKER_01: developer productivity over,
15:05
SPEAKER_01: let's say 20 years compressed in a few years,
15:08
because I still remember that when the first kind of developer
15:11
productivity
15:12
SPEAKER_01: products came out measuring,
15:15
SPEAKER_01: they start to measure things like lines of code per developer.
15:17
SPEAKER_01: And then they looked at number of commits per developer on average.
15:21
SPEAKER_01: And, you know, the first product said like, this is, this is good.
15:23
SPEAKER_01: Look at this.
15:24
SPEAKER_01: And then as an industry, we started to say that's, that's BS.
15:28
SPEAKER_01: Like,
15:28
SPEAKER_01: I'm sorry,
15:29
the developer who pushes the most code to production or writes
15:32
the most lines of
15:33
code might not be your best developer because they might just
15:37
be doing boilerplate
15:38
SPEAKER_01: stuff,
15:39
SPEAKER_01: updating frameworks.
15:41
SPEAKER_01: Actually, they might just be...
15:43
fixing their own bugs because they ship so many of them so i i
15:47
feel but we've had
15:48
that conversation let's say maybe like 10 years ago and
15:50
everyone agreed like you
15:51
know like lines of code is not not the best metric in fact some
15:54
of the best
15:55
developers sometimes don't even write lines they like delete
15:58
them but now we're now
15:59
back here like oh yeah ai generates a lot of lines of code
16:02
therefore it should be
16:02
SPEAKER_01: productive
16:03
Yeah,
16:04
and I think one of my more controversial opinions is that
16:07
source code is a liability.
16:08
SPEAKER_00: I think it sounds controversial,
16:10
SPEAKER_00: and then when people think about it,
16:12
SPEAKER_00: they realize that,
16:12
SPEAKER_00: yeah,
16:13
SPEAKER_00: it actually is.
16:13
And now we're in a world where it is trivially easy to produce
16:19
a tremendous amount
16:19
SPEAKER_00: of source code.
16:21
And so what does that actually mean for productivity and
16:24
business impact when what
16:27
SPEAKER_00: could have been written in one line is now written in five lines?
16:29
SPEAKER_00: Do we really want to measure...
16:32
SPEAKER_00: AI impact in terms of lines of code generated.
16:34
SPEAKER_00: I certainly don't.
16:35
SPEAKER_00: We don't recommend it.
16:36
SPEAKER_00: We did not include acceptance rate in our framework for good reason.
16:41
SPEAKER_00: I think it does give insight into whether the tools are fit for purpose,
16:45
SPEAKER_00: but when we're looking at broadly
16:47
measuring business impact and the impact on developer
16:50
experience and the impact on
16:52
SPEAKER_00: the business,
16:53
SPEAKER_00: acceptance rate is just such a tiny part of the story.
16:56
SPEAKER_01: And by acceptance rate,
16:57
you mean like what percent or how many lines did the developers
17:01
say except the tap
17:02
SPEAKER_01: suggestion or whatever the AI is spitting out?
17:05
SPEAKER_00: Yeah,
17:05
so we can use that to figure out if it's just spitting you
17:08
spaghetti code and,
17:10
SPEAKER_00: you know,
17:11
SPEAKER_00: suggestions that are not accurate,
17:14
SPEAKER_00: then acceptance rate is going to be low.
17:15
SPEAKER_00: And we can use it as a signal to understand that,
17:18
SPEAKER_00: okay,
17:18
SPEAKER_00: these tools are not sufficiently robust for the use cases.
17:21
But if we're just looking at,
17:23
oh,
17:23
well,
17:23
you know,
17:24
developers are accepting...
17:26
SPEAKER_00: 95% of the suggestions,
17:28
SPEAKER_00: that doesn't really tell us anything in terms of,
17:32
SPEAKER_00: is it increasing their velocity?
17:33
SPEAKER_00: Is it saving them time?
17:34
SPEAKER_00: Is it going to help us innovate faster?
17:37
SPEAKER_00: Those are the things that we actually want to look at.
17:39
SPEAKER_00: And that's what we've included in this AI measurement framework,
17:41
not necessarily just those granular measurements of acceptance
17:45
rate or lines of
17:46
SPEAKER_00: code.
17:47
SPEAKER_01: I'm glad to hear a little bit of like kind of more grounded approach.
17:52
SPEAKER_01: And I guess it helps that you're not an AI vendor and you're not,
17:57
SPEAKER_01: you know,
17:58
SPEAKER_01: I guess your goal is to try to figure out like what actually works.
18:00
Now,
18:00
you did an interesting research slash case study with
18:03
Booking.com and how they use AI.
18:06
SPEAKER_01: What did you find there?
18:08
SPEAKER_00: Yeah,
18:08
SPEAKER_00: so for booking,
18:09
what they really realized was adoption is the key to getting a
18:13
better result.
18:14
SPEAKER_00: What they found was that the developers who were adopting the tool,
18:19
so going from non-user to periodic but consistent users and
18:22
moving the population
18:24
SPEAKER_00: up from
18:25
from not using into daily weekly usage that was where they were
18:29
seeing the most
18:30
benefit so they did some very concerted organization-wide
18:33
efforts around enablement
18:35
for adoption things like office hours things like you know
18:39
workshops and trainings
18:40
and they got their adoption up to 65 percent of developers
18:43
using these tools on a
18:46
weekly daily basis which is actually well above the median
18:50
which is 50 percent
18:51
industry-wide and the top quartile is 60 percent so they're
18:55
doing quite well across
18:57
you know according to the industry benchmark so so just to
19:00
pause here just so i
19:00
understand so booking was like okay we'd like as many devs as
19:04
possible to use these
19:06
SPEAKER_01: tools github copilot or whatever or any other copilot that they had the
19:11
SPEAKER_01: the chat, et cetera, let's say on a weekly basis.
19:14
SPEAKER_01: And we're going to do training on this.
19:16
SPEAKER_01: Leadership will say, hey, please use it or try it out.
19:20
SPEAKER_01: And they did office hours.
19:22
SPEAKER_01: They had teams on this.
19:24
SPEAKER_01: And even after doing this,
19:25
SPEAKER_01: we're saying that about 65% of devs use it weekly or maybe daily,
19:29
SPEAKER_01: but mostly weekly.
19:31
SPEAKER_01: So that means 35% are still like, no, I'm good.
19:34
SPEAKER_01: I'm just going to do what I did before, right?
19:37
SPEAKER_00: Yeah, for a variety of reasons, right?
19:40
SPEAKER_00: But that 65%,
19:41
SPEAKER_00: what's interesting is like that's still above the 20,
19:44
SPEAKER_00: you know,
19:44
SPEAKER_00: above the P75 industry wide.
19:47
SPEAKER_01: But I'm just pausing here because I think there's two types of people,
19:50
SPEAKER_01: right,
19:50
SPEAKER_01: who are listening as well.
19:51
SPEAKER_01: It depends on what environment you are.
19:53
But if you're in a startup environment or you're just like an
19:56
early adopter,
19:57
SPEAKER_01: you'll be like,
19:57
SPEAKER_01: well,
19:58
SPEAKER_01: why would you not use it?
19:59
Like everyone using it,
20:00
like I'm not going to use it all the time,
20:02
not for everything.
20:03
SPEAKER_01: But like, yeah, it's there.
20:04
SPEAKER_01: Like I know when to use it, when to not, and they'll use it daily.
20:07
SPEAKER_01: And then there might be some people saying, why would anyone use this?
20:11
SPEAKER_01: But it's interesting that in a company that is trying to say like,
20:13
SPEAKER_01: okay,
20:14
SPEAKER_01: let's get everyone to use it.
20:15
SPEAKER_01: To me,
20:17
it still feels interesting how it's only 65% when they do all
20:21
these things that a
20:22
SPEAKER_01: bunch of places don't do.
20:23
SPEAKER_01: From the training,
20:23
SPEAKER_01: from the enablement,
20:25
SPEAKER_01: from the investing,
20:26
SPEAKER_01: from partnering with companies like you.
20:29
SPEAKER_01: So what have you learned about that 35%?
20:34
SPEAKER_01: What is either holding them back or are they right to be skeptical?
20:37
SPEAKER_00: Yeah,
20:38
I think the biggest learning for me is that it's not
20:40
necessarily that these
20:42
individuals are skeptical Luddites who don't want to use any
20:45
new technology.
20:46
Some of it is just that the organization doesn't make a license
20:49
available to them.
20:50
SPEAKER_00: They would like to use it, but the licenses aren't available.
20:54
SPEAKER_00: And I'm not suggesting that's the case for booking.
20:57
SPEAKER_00: I want to make it clear, but I have seen that pattern repeated regularly.
21:00
SPEAKER_00: in many organizations.
21:02
SPEAKER_00: So we have to,
21:03
SPEAKER_00: when we think about utilization,
21:04
SPEAKER_00: in our framework,
21:05
we recommend looking at the number of daily active users and
21:08
weekly active users.
21:09
SPEAKER_00: If you use DX to measure that,
21:11
SPEAKER_00: you can actually look at that as a percentage of your total population.
21:15
SPEAKER_00: And then you can actually look at where the licenses line up to be
21:19
across your population because it might be that the people who
21:22
would like to use it
21:23
SPEAKER_00: can't yet because the licenses are not available to them.
21:26
SPEAKER_00: You know,
21:26
SPEAKER_00: some companies right now as they're experimenting will say,
21:29
SPEAKER_00: okay,
21:29
SPEAKER_00: we're going to get 500 licenses for Copilot,
21:32
SPEAKER_00: 500 licenses for Cody and whatever.
21:34
SPEAKER_00: So there's a limited pool to pull from.
21:37
SPEAKER_00: And so there's no scenario where 100% of developers could be using it.
21:43
They simply haven't invested the money in making licenses
21:47
available to 100% of their developers.
21:49
And so I would say that's a fairly big reason where we see not
21:52
100% development or
21:54
SPEAKER_00: 100% adoption.
21:57
SPEAKER_00: I think the other thing,
21:58
SPEAKER_00: and you covered this in your lead dev talk,
22:00
SPEAKER_00: is that for certain services,
22:03
SPEAKER_00: components,
22:04
SPEAKER_00: product areas,
22:06
it's just not that effective because of the very novel or
22:11
greenfield nature of the
22:12
SPEAKER_00: kind of code.
22:13
So like we can think about this on a spectrum and one is like
22:16
writing Terraform
22:17
files or like one is writing something in YAML in like a really
22:21
defined way,
22:22
SPEAKER_00: like for Terraform.
22:23
SPEAKER_00: And the other one is like,
22:25
SPEAKER_00: doing something that no one has ever done before.
22:28
AI is amazingly excellent at stuff that has a lot of structure
22:32
and a lot of pattern.
22:33
SPEAKER_00: But when,
22:34
SPEAKER_00: you know,
22:34
you use the example of that healthcare startup who wanted to
22:37
remain nameless
22:37
because they didn't even want to go on record saying that they
22:40
don't use AI because
22:41
SPEAKER_00: it just doesn't work for them.
22:43
SPEAKER_00: And so,
22:44
SPEAKER_00: you know,
22:44
SPEAKER_00: you can imagine at a company as big as Booking or as big as Meta,
22:48
SPEAKER_00: Dropbox,
22:49
there are going to be pockets of developers who just aren't
22:52
well served yet by the
22:54
SPEAKER_00: tools.
22:55
SPEAKER_01: Well,
22:56
SPEAKER_01: and I can also see cases where these tools just really fall flat,
23:00
SPEAKER_01: where you're trying to do something very,
23:01
SPEAKER_01: very specific and in a concise or very performance effective way,
23:08
SPEAKER_01: which is usually about understanding the whole structure,
23:10
SPEAKER_01: making small tweaks.
23:12
SPEAKER_01: For example,
23:12
SPEAKER_01: if I think of,
23:13
SPEAKER_01: I was talking with a stable Linux branch maintainer,
23:16
SPEAKER_01: Greg KH,
23:18
SPEAKER_01: I was saying,
23:18
SPEAKER_01: like,
23:18
SPEAKER_01: how do you use AI?
23:19
And he was like,
23:20
well,
23:20
I mean,
23:20
like,
23:21
we don't really,
23:21
they use it for tools,
23:22
but
23:23
SPEAKER_01: If I look at every Linux commit to the kernel, it's a few lines.
23:27
And those lines have been thought for so long,
23:30
and they need to be as concise as possible.
23:32
SPEAKER_01: Your performance matters, all those things matters.
23:34
SPEAKER_01: And for those use cases,
23:35
SPEAKER_01: especially with big companies,
23:36
SPEAKER_01: I can imagine that if you're in a platform team,
23:39
SPEAKER_01: you're optimizing the P95 performance of your Android app.
23:44
SPEAKER_01: You might use it for brainstorming or here and there,
23:47
SPEAKER_01: but in the end,
23:48
SPEAKER_01: the changes you make are so small in terms of lines of code,
23:52
SPEAKER_01: but they're so large in impact.
23:54
SPEAKER_01: And so much of it is about testing,
23:56
SPEAKER_01: about wins that have not been made before or seeing connections.
24:02
SPEAKER_01: So I wonder if some of that is also that.
24:04
SPEAKER_00: Yeah,
24:05
I want to show this to you because I think that exactly that
24:08
point that you made
24:09
SPEAKER_00: that maybe the biggest gain is not in code generation,
24:13
SPEAKER_00: but in like you could still use it for brainstorming,
24:15
SPEAKER_00: can still use it for error analysis.
24:18
This is part of that enablement and training that companies can
24:21
offer in order to
24:22
SPEAKER_00: increase adoption,
24:23
SPEAKER_00: increase impact.
24:24
SPEAKER_00: We did a study of 180 plus companies,
24:28
and we looked at the developers who were saving a serious
24:31
amount of time with AI.
24:32
SPEAKER_00: And we tried to understand, like, what are you actually doing?
24:35
And interestingly enough,
24:37
code generation,
24:38
like mid-loop code generation,
24:40
is the third thing.
24:42
SPEAKER_00: highest use case for saving time,
24:44
SPEAKER_00: but actually stack trace analysis and refactoring existing code,
24:48
SPEAKER_00: we're saving more time than the mid loop code generation.
24:52
And so this is also really important for companies and platform
24:55
engineering teams to understand
24:58
SPEAKER_00: Because I think the idea,
24:59
SPEAKER_00: you know,
25:00
SPEAKER_00: is like,
25:01
SPEAKER_00: well,
25:01
we give our developers a license for Copilot and then just
25:05
expect them to kind of
25:06
SPEAKER_00: figure it out.
25:07
And a lot of us go to mid-loop code generation because that's
25:10
kind of what is
25:12
SPEAKER_00: mostly talked about.
25:13
SPEAKER_01: Yeah, most demos and most obvious things, right?
25:16
SPEAKER_01: Yeah, absolutely.
25:16
SPEAKER_01: That's what I thought about until we talked.
25:18
SPEAKER_00: It's the most obvious thing.
25:20
But things like,
25:21
you know,
25:22
putting in a hundred line stack trace analysis and being like,
25:25
what?
25:26
SPEAKER_00: Why is this happening?
25:27
SPEAKER_00: Give me like give me a diff that would fix this problem.
25:31
SPEAKER_01: Or give me four possible ideas.
25:33
SPEAKER_00: Totally.
25:34
SPEAKER_00: Yeah.
25:34
SPEAKER_01: And then two of them might be things that I didn't think about.
25:36
SPEAKER_01: And now I can go off and research.
25:39
SPEAKER_00: Yeah.
25:39
And so there's no ceiling on the different kinds of use cases
25:45
that AI can help,
25:46
especially when it has really good context and understands your
25:49
code base really
25:50
SPEAKER_00: thoroughly.
25:51
SPEAKER_00: Code documentation,
25:53
SPEAKER_00: brainstorming and planning,
25:55
SPEAKER_00: unit tests are an area that are really well served by AI,
26:00
SPEAKER_00: anything that's very well defined.
26:02
But I was really surprised to hear about stack trace analysis
26:05
being the top time
26:06
saver and not necessarily being mid-loop code generation
26:09
because as you said,
26:10
SPEAKER_00: it's the most obvious thing.
26:11
SPEAKER_01: And when you say mid-loop, this is like, what does mid-loop stands for?
26:16
SPEAKER_00: Yeah,
26:16
kind of I can write out the scaffold of whatever function I
26:19
want to write and I can
26:20
SPEAKER_00: give it the input and the output and then just say,
26:23
SPEAKER_00: give me,
26:24
SPEAKER_00: finish my function,
26:25
SPEAKER_00: make it complete this thing for me.
26:28
SPEAKER_01: But this is really interesting as well,
26:29
SPEAKER_01: because I feel it goes a little bit against the mainstream narrative,
26:33
SPEAKER_01: even with developers,
26:34
SPEAKER_01: as you said,
26:34
SPEAKER_01: that what is AI good for?
26:36
SPEAKER_01: Code generation.
26:37
SPEAKER_01: It generates code faster, because I think that's what we see, right?
26:39
SPEAKER_01: It does spit it out.
26:40
SPEAKER_01: It is superhuman in terms of speed.
26:43
SPEAKER_01: You cannot write this fast.
26:46
SPEAKER_01: But if we see that these are the main use cases,
26:48
SPEAKER_01: stack trace,
26:49
SPEAKER_01: refactoring,
26:50
SPEAKER_01: okay,
26:50
SPEAKER_01: code journey is still down there,
26:51
SPEAKER_01: and then we have the other stuff.
26:53
Maybe it suggests that,
26:56
yeah,
26:56
as you say,
26:57
maybe we're thinking a bit wrong.
26:58
SPEAKER_01: And I wonder if this might also impact how,
27:01
SPEAKER_01: you know,
27:01
SPEAKER_01: there's this narrative that now anyone can be a developer,
27:04
SPEAKER_01: you don't need to be a developer to write software.
27:06
But if these tools really help with,
27:07
let's say,
27:08
stack trace analysis and refactoring,
27:11
SPEAKER_01: unless you're a developer, you're not going to use that.
27:14
SPEAKER_01: So maybe these tools are actually a lot better for experts,
27:17
SPEAKER_01: you know,
27:18
SPEAKER_01: professional software engineers who know what they're doing.
27:20
SPEAKER_00: You know,
27:21
SPEAKER_00: to me,
27:21
the sticking point about things like stack trace analysis or
27:25
refactoring code is
27:26
SPEAKER_00: that it's about time savings and not interaction with the tool.
27:31
SPEAKER_00: And what I mean by that is like,
27:33
SPEAKER_00: number one, typing speed has never been the bottleneck in development.
27:37
But now we have all this code generated faster than we can
27:41
type,
27:42
that's great.
27:43
SPEAKER_00: But it still takes me time to review that code,
27:48
SPEAKER_00: to cognitively make sure that I understand it and that it's accurate,
27:54
SPEAKER_00: time to review it.
27:55
And so the time savings,
27:57
it's not that we're saving time because we don't have to type,
28:01
a lot of that time we're just reallocating to reviewing or
28:05
other parts of code
28:07
SPEAKER_00: authoring that's not typing for stack trace analysis we're actually just
28:10
eliminating the toil completely of parsing through this you
28:16
know huge output trying
28:18
to figure out what's going wrong and then going spelunking in
28:21
the code and so that
28:22
is truly like a net positive time saving to say you know give
28:26
me four examples or
28:27
SPEAKER_00: like what's the what's the most likely cause of this
28:30
I can just totally leapfrog that whole 45 minutes that I would
28:34
have spent banging
28:35
SPEAKER_00: my head against my keyboard trying to figure out.
28:37
SPEAKER_00: Whereas if I'm using it for code generation,
28:39
SPEAKER_00: yeah,
28:40
SPEAKER_00: it's faster,
28:41
but I also have to invest time reviewing that code to make sure
28:44
that it's accurate.
28:44
SPEAKER_01: Reviewing it, iterating on it, you might refactor it.
28:48
SPEAKER_01: Oftentimes,
28:48
SPEAKER_01: you know,
28:49
SPEAKER_01: I think as developers,
28:50
SPEAKER_01: we know that it generates something,
28:51
SPEAKER_01: but if you have some coding styles or if you have a way of coding,
28:54
SPEAKER_01: you will tweak it,
28:55
SPEAKER_01: rewrite it,
28:56
SPEAKER_01: change it,
28:57
SPEAKER_01: it gets it wrong,
28:58
SPEAKER_01: etc.
28:58
SPEAKER_01: So, okay, so this makes sense.
29:00
SPEAKER_01: Going back to the first one, we're like, okay, it truly saves time.
29:04
SPEAKER_01: What happens when
29:05
SPEAKER_01: It saved time.
29:06
SPEAKER_01: Like,
29:07
SPEAKER_01: let's say,
29:07
SPEAKER_01: okay,
29:08
SPEAKER_01: until now,
29:09
I've usually had to spend a bunch of time analyzing the SAC
29:11
trace and I was stuck
29:12
SPEAKER_01: on it for,
29:13
SPEAKER_01: you know,
29:13
SPEAKER_01: 15 minutes at first.
29:15
But by the way,
29:16
with experience,
29:17
it would have gone down to like 10 and then five.
29:19
SPEAKER_01: And now I'm a senior engineer and boom, I look at it.
29:21
SPEAKER_01: I know what that is.
29:23
SPEAKER_01: But what happens with the time saving?
29:26
As a developer,
29:27
what would that mean for me as a developer or me as an
29:30
organization?
29:31
SPEAKER_01: Do I just clock out earlier?
29:35
Will I now have a bit more space to help out others to start to
29:39
think about bigger
29:41
SPEAKER_01: things instead of the day-to-day and some strategic stuff?
29:45
SPEAKER_01: Where have you seen it?
29:45
SPEAKER_01: Because this is not new, right?
29:47
SPEAKER_01: Developers saving time, we've seen this with other tools as well.
29:52
SPEAKER_01: The big question is the organization thinks like,
29:54
SPEAKER_01: oh,
29:54
SPEAKER_01: if my developers spend 10% time,
29:56
SPEAKER_01: each of them on average,
29:57
SPEAKER_01: I can either fire 10% of them.
30:00
SPEAKER_01: This is the,
30:00
SPEAKER_01: you know,
30:01
SPEAKER_01: the big evil corporation,
30:02
SPEAKER_01: or I can just not hire and my productivity goes up 10%.
30:05
This is what the business thinks,
30:06
but it's not really what happens,
30:07
does it?
30:08
SPEAKER_00: It's definitely not what happens.
30:10
SPEAKER_00: I think one thing to keep in mind is that on the very best day,
30:14
SPEAKER_00: developers are not spending even 80% of their time coding.
30:17
SPEAKER_00: I think the industry average is like 25%.
30:20
There was a study at AWS that an average AWS engineer only
30:24
spends 20% of their time coding.
30:26
SPEAKER_00: And so when we apply AI to the coding tasks,
30:31
SPEAKER_00: We're only working with 20% of that time to begin with.
30:34
SPEAKER_00: And when we save 10% of that time,
30:35
SPEAKER_00: that actually doesn't amount to,
30:38
SPEAKER_00: oh,
30:38
SPEAKER_00: we can ship 10 new product lines overnight.
30:43
SPEAKER_00: That's just not realistic.
30:45
SPEAKER_00: I think this is, though, where things get a little bit weird.
30:49
SPEAKER_00: So what happens with that time?
30:51
Because we would like to think like,
30:53
oh,
30:53
this is going to be really great for developers.
30:56
SPEAKER_00: They're going to be saving, you know, they're saving time.
30:59
SPEAKER_00: They can reinvest that time in tech.repayment or, you know, other stuff.
31:05
SPEAKER_00: When Dora researched this question, what they found was that...
31:10
Many developers were actually feeling less satisfied because AI
31:14
is accelerating the
31:15
SPEAKER_00: parts that they enjoy.
31:18
SPEAKER_00: And so what was left over was more stuff that they didn't enjoy.
31:22
SPEAKER_00: The toil, the meetings, you know, the administrative work.
31:26
SPEAKER_00: And so that was an interesting result.
31:30
And that's from their guide on AI engineering that came out a
31:34
couple of months ago.
31:35
It gave me pause when I read that because I thought I've always
31:38
had the very strong
31:38
conviction that AI time savings is not going to come from the
31:43
coding task.
31:44
SPEAKER_00: It makes sense that that's the obvious place where we all started.
31:47
But organizationally,
31:49
how fast we can create code has never been the bottleneck.
31:52
SPEAKER_00: It's been everything around it.
31:54
SPEAKER_00: And now when we take away or make that faster,
31:57
SPEAKER_00: the code authoring process for people who like to author code,
32:01
SPEAKER_00: that has some impact.
32:01
SPEAKER_01: Well, I think you...
32:04
SPEAKER_01: You know,
32:04
SPEAKER_01: when I was thinking,
32:05
SPEAKER_01: when I was a developer and then was a manager with developers,
32:08
SPEAKER_01: like,
32:08
SPEAKER_01: what is a good day as a developer?
32:10
SPEAKER_01: And I think,
32:12
SPEAKER_01: you know,
32:12
SPEAKER_01: an average good day,
32:13
SPEAKER_01: like,
32:14
SPEAKER_01: again,
32:15
SPEAKER_01: there can be different days,
32:15
SPEAKER_01: but it was like,
32:16
SPEAKER_01: I come into work,
32:18
SPEAKER_01: you know,
32:18
SPEAKER_01: we say hi to people,
32:19
SPEAKER_01: you know,
32:20
SPEAKER_01: maybe we talk about something.
32:22
SPEAKER_01: Usually it's, we don't have meetings.
32:24
SPEAKER_01: I have a clear goal in mind.
32:26
SPEAKER_01: it's something challenging that I, I want to complete something.
32:29
SPEAKER_01: Maybe it might've been from yesterday or I'm now just starting as fresh.
32:33
SPEAKER_01: I get into the zone.
32:35
SPEAKER_01: I, you know, get it together.
32:36
SPEAKER_01: I get it working.
32:38
SPEAKER_01: Uh, I clean it up.
32:40
SPEAKER_01: It's working.
32:42
SPEAKER_01: It's amazing.
32:43
SPEAKER_01: I committed or I tested, I check it.
32:45
SPEAKER_01: I put up a pull request and I'm done.
32:48
SPEAKER_01: And you know,
32:48
SPEAKER_01: if this happens at 2 PM and it was something challenging,
32:51
SPEAKER_01: it should have taken me eight hours,
32:52
SPEAKER_01: but I got it done in four and I'm really proud of it.
32:55
SPEAKER_01: Uh, it,
32:56
SPEAKER_01: It works.
32:57
SPEAKER_01: It does.
32:57
SPEAKER_01: Maybe I could clean up.
32:58
SPEAKER_01: And then I help out some people.
32:59
SPEAKER_01: But when I had that, like, it's been a good day.
33:02
SPEAKER_01: And the bad day is the opposite.
33:03
SPEAKER_01: It's like, I get into work.
33:05
SPEAKER_01: I have this thing that I need to do.
33:07
SPEAKER_01: And I get interrupted.
33:08
SPEAKER_01: I go to a meeting.
33:09
SPEAKER_01: And I try to go back.
33:10
SPEAKER_01: And now another meeting.
33:12
SPEAKER_01: Or I finally get back into it.
33:14
SPEAKER_01: But now I'm just stuck.
33:15
SPEAKER_01: It's more complex.
33:16
SPEAKER_01: And I go home.
33:17
SPEAKER_01: And when I'm falling asleep, I'm still thinking about this goddamn thing.
33:20
SPEAKER_01: I actually want to open my laptop.
33:22
SPEAKER_01: And I don't sleep well.
33:23
you know we used to do things before ai uh just like things
33:26
like no meeting days
33:27
and like bunching meetings to give people the chance to be in
33:31
the flow because
33:31
there is something about that when when you are in the flow for
33:35
a day like for for
33:36
some time it's it's it's good like right like good yeah like
33:40
you're you're you're a
33:41
SPEAKER_01: developer
33:42
SPEAKER_01: Yeah.
33:44
SPEAKER_01: You still build software or used to build software, right?
33:46
SPEAKER_00: Yeah, both.
33:46
SPEAKER_00: Yeah.
33:47
I think this is going back to that question about measurements
33:51
and how do we
33:53
SPEAKER_00: measure the impact of AI?
33:54
SPEAKER_00: This is one area that I'm personally very curious about,
33:57
SPEAKER_00: which is does AI allow us to manage interruptions better?
34:01
SPEAKER_00: Does it help developers stay in flow state?
34:03
SPEAKER_00: Does it reduce cognitive load?
34:05
And these are numbers that aren't going to show up necessarily
34:08
in time savings.
34:10
SPEAKER_00: Per developer,
34:11
SPEAKER_00: but will show up in other areas of developer experience,
34:14
SPEAKER_00: because one hypothesis is that with the use of AI tooling,
34:19
SPEAKER_00: the tax that you pay when you have to switch tasks.
34:21
SPEAKER_00: So going from,
34:22
SPEAKER_00: you know,
34:24
maybe you have from 10 to noon focus time and then you have a
34:28
meeting from 12 to
34:29
SPEAKER_00: 1230 and then you can focus for the rest,
34:31
SPEAKER_00: you know,
34:31
SPEAKER_00: that tax beforehand and the tax after.
34:34
Does having an AI coding assistant reduce the amount of time
34:38
that it helps you or
34:39
SPEAKER_00: that it takes you to get back into flow?
34:40
SPEAKER_00: Because you have basically a body double or a pair programmer,
34:46
SPEAKER_00: so to speak,
34:46
SPEAKER_00: who is holding context and keeping context for you.
34:49
SPEAKER_00: And it's easier for you to pick up where you left off.
34:52
This is really difficult to measure systemically with only
34:56
workflow data alone.
34:57
which is maybe another thing that I didn't emphasize when I
35:01
talked about the
35:01
SPEAKER_00: measurement framework,
35:02
but combining self-reported metrics with system and workflow
35:06
metrics is absolutely
35:07
SPEAKER_00: essential when measuring the impact of AI tools,
35:11
SPEAKER_00: because it does have an impact on the authoring experience as well.
35:15
SPEAKER_00: And some of that we cannot observe from our systems.
35:17
SPEAKER_00: We actually have to talk to developers in order to figure it out.
35:20
SPEAKER_00: And this is one area.
35:21
SPEAKER_00: So talking about, you know, change competence,
35:25
SPEAKER_00: developer experience measurements, CSAT for the AI tools.
35:30
Those are all things that are really important parts because we
35:32
might actually miss
35:33
important signal about how AI is impacting the code authoring
35:38
experience or other
35:40
SPEAKER_00: parts of the software development lifecycle.
35:43
But we might miss them if we're only looking at the workflow
35:46
tools themselves.
35:47
So we need to have a more robust,
35:48
comprehensive way of measuring across the organization.
35:52
SPEAKER_01: So let's talk about what is working and what you've seen working.
35:55
And one of the things that we previously talked about is how
35:58
some of the teams who
36:00
are actually making pretty good use of AI are starting to make
36:03
some architectural
36:03
changes to their code base and make some architectural
36:08
decisions to make it easier
36:10
SPEAKER_01: to reach for,
36:11
SPEAKER_01: let's say,
36:11
SPEAKER_01: agentic modalities.
36:13
SPEAKER_01: What have you seen work and how is this coming along?
36:17
SPEAKER_00: Yeah, there's two kind of two broad things that I'll talk about here.
36:20
SPEAKER_00: And one is the architecture itself.
36:22
And then the other is sort of the discoverability of the
36:25
architecture and of the system.
36:26
SPEAKER_00: And there's a lot more going on there.
36:28
SPEAKER_00: On the architecture itself,
36:29
what I have seen to sort of anecdotally from my own
36:32
conversations is leaders are
36:35
SPEAKER_00: recommitting to like clean interfaces between services.
36:38
SPEAKER_00: I would say that's probably the top thing that comes up into.
36:40
SPEAKER_01: That's kind of a nice thing to come out of this.
36:44
SPEAKER_00: Great.
36:44
SPEAKER_00: I love to see it.
36:45
I think,
36:46
you know,
36:47
we can think about sort of AWS as everything is an API as a
36:51
model here.
36:52
SPEAKER_00: Like when your own systems operate like that,
36:56
SPEAKER_00: the interfaces are so clear and well-defined,
36:58
it becomes easier for agentic models to use your code base
37:01
because the boundaries
37:02
SPEAKER_00: are more clearly defined.
37:03
SPEAKER_01: By the way, also for humans, right?
37:05
SPEAKER_00: Also for humans.
37:06
SPEAKER_00: Yeah.
37:07
SPEAKER_00: It's like it works really great.
37:10
And so that actually that point also for humans is the
37:13
interesting point about
37:14
SPEAKER_00: documentation,
37:15
SPEAKER_00: because this is the shift that I'm seeing more often.
37:17
SPEAKER_00: I was actually in Amsterdam while you were in Mongolia.
37:20
SPEAKER_00: Otherwise, we could have had another stake.
37:23
But while I was in Amsterdam,
37:24
I did a fireside chat with about 45 other people.
37:28
SPEAKER_00: engineering leaders and,
37:29
SPEAKER_00: you know,
37:29
SPEAKER_00: very quickly into Q&A,
37:31
SPEAKER_00: the question was,
37:31
SPEAKER_00: should we be writing documentation for AI or for humans?
37:35
SPEAKER_00: And my question, my answer to that question is, yeah, both.
37:40
But here's one thing that I've seen kind of pick up,
37:43
I would say,
37:43
in the last six weeks.
37:45
SPEAKER_00: So human documentation often has like visual dependencies.
37:50
SPEAKER_00: They'll be
37:51
SPEAKER_00: a screenshot of something.
37:53
SPEAKER_00: It needs this sort of like narrative flow.
37:56
SPEAKER_00: Whereas for AI, it's really good to have the coding examples.
38:00
SPEAKER_00: There can't be visual dependencies.
38:03
It doesn't work great that way because developers aren't going
38:08
to a documentation
38:09
SPEAKER_00: page necessarily or watching a YouTube tutorial.
38:12
SPEAKER_00: to see how to use your thing.
38:14
SPEAKER_00: They're going to Claude to ChatGPT.
38:17
They're interacting in their IDE with an AI assistant and
38:21
trying to implement it.
38:22
And so documentation needs to be there for AI so that that
38:28
developer gets the
38:29
SPEAKER_00: information they need in the best way.
38:31
SPEAKER_00: And I think companies that are
38:33
SPEAKER_00: going AI first,
38:34
SPEAKER_00: like Vercel Clerk,
38:35
SPEAKER_00: for example,
38:36
SPEAKER_00: they have really,
38:37
SPEAKER_00: really solid examples of AI first documentation because it's a flywheel.
38:43
SPEAKER_00: Like, you know, they have great documentation.
38:46
SPEAKER_00: Then when a developer is trying to implement this thing,
38:49
they actually get a good suggestion and can do it successfully
38:52
from within their ID
38:53
SPEAKER_00: with whatever coding assistant.
38:55
then that coding assistant has more data about what actually
38:57
works and it just
38:58
SPEAKER_00: keeps reinforcing it.
39:00
SPEAKER_00: And so for internal development teams,
39:02
SPEAKER_00: like platform teams,
39:03
SPEAKER_00: this is a great model to think about.
39:05
How can you make documentation that is getting people the
39:09
information they need at
39:10
SPEAKER_00: the moment they need it,
39:11
SPEAKER_00: which is now in the editor,
39:13
SPEAKER_00: not in your documentation necessarily.
39:17
And then for external developers,
39:19
if you're making a dev tool,
39:20
that's like...
39:22
out there in the ecosystem this is the way that people are
39:24
discovering your tool
39:25
and implementing it um there you know the just the way that
39:28
people that developers
39:29
are are coming across tools and using them is really different
39:32
so that's been the
39:33
biggest way that i've seen companies think about or already
39:36
start trying to change
39:37
the way that they architect their their services but also
39:41
document their services
39:42
SPEAKER_00: to make them work better in this sort of ai assisted coding world
39:48
SPEAKER_01: That's interesting.
39:48
And I also like your thinking of how this also creates a bunch
39:51
of opportunities for
39:53
especially company startups who are building APIs or things for
39:56
developers to use,
39:57
if you make it easy for them to use and also make it a bit more
40:00
friendly for AI
40:02
SPEAKER_01: crawlers or any of these to ingest,
40:05
SPEAKER_01: you might be getting more users later on or unblocking your users.
40:08
SPEAKER_01: Because in the end,
40:09
SPEAKER_01: I'm going to guess in the future,
40:12
SPEAKER_01: two years from now,
40:13
SPEAKER_01: developers will be like,
40:14
SPEAKER_01: all right,
40:15
SPEAKER_01: I want to create a project using this technology.
40:17
SPEAKER_01: As a developer, you will specify this technology.
40:20
SPEAKER_01: If not, it'll default to whatever.
40:22
SPEAKER_01: But if you have too much trouble with the technology,
40:24
SPEAKER_01: eventually,
40:25
SPEAKER_01: you know,
40:25
SPEAKER_01: there's going to be this learning thing.
40:26
SPEAKER_01: You will choose a technology that as today,
40:29
SPEAKER_01: the one that you're familiar with,
40:31
SPEAKER_01: you know,
40:31
SPEAKER_01: it works.
40:32
SPEAKER_01: You know, if you get stuck, it's easier to do.
40:34
SPEAKER_01: So these things will remain important.
40:36
SPEAKER_01: Technology, SDK, vendor, you know, reliability, maintainability.
40:42
These things are going to remain important for professional
40:44
software engineers
40:45
SPEAKER_01: that,
40:45
SPEAKER_01: you know,
40:46
SPEAKER_01: we are and we will be.
40:47
I like what you said about this is also good for human beings
40:51
and it's good for AI.
40:54
There's so many areas about developer experience and the world
40:58
that I operate in
40:58
where what's good for the developer and good for the business
41:01
is a circle,
41:01
SPEAKER_00: that Venn diagram.
41:03
And this is what's good for the AI agent and good for the human
41:06
being is also a
41:06
SPEAKER_00: circle,
41:07
SPEAKER_00: like clear boundaries between services.
41:10
SPEAKER_00: I think that's such an interesting space to operate in.
41:13
It's like we needed AI as sort of like the business kick in the
41:17
pants of like,
41:17
SPEAKER_00: hey,
41:18
we're not going to get as much out of our investment in AI
41:21
because now our wallets
41:22
SPEAKER_00: are open.
41:23
Before it was like,
41:23
well,
41:24
read the manual,
41:26
kind of work around bad tooling.
41:28
SPEAKER_00: But now that there's significant financial investment around it,
41:33
SPEAKER_00: not that development teams aren't a significant financial investment,
41:37
SPEAKER_00: but I think the tolerance is different,
41:39
SPEAKER_00: you know?
41:39
SPEAKER_00: Yeah.
41:40
SPEAKER_01: Yeah, I like your thinking.
41:41
SPEAKER_01: This is a good Venn diagram to draw.
41:46
SPEAKER_01: Because again, when you onboard to a new company, it's always a problem.
41:50
SPEAKER_01: Onboarding has been just difficult.
41:52
SPEAKER_01: The onboarding documentation is out of date.
41:55
SPEAKER_01: The presentation is out of date.
41:56
SPEAKER_01: No one tells me how to do this.
41:57
SPEAKER_01: So then people take a month to get productive.
42:00
SPEAKER_01: And now you can say, okay, let's update it.
42:03
SPEAKER_01: So now our AI tool can also be helpful.
42:05
SPEAKER_01: But also it goes both ways, right?
42:07
SPEAKER_01: Like when people onboard, they can also read their thing.
42:09
They can now turn to the chat agent,
42:11
which will actually give them accurate information.
42:14
SPEAKER_01: I love it.
42:14
SPEAKER_01: Like I feel there's a lot of like these wins.
42:17
SPEAKER_01: Like it's always nice to discover these.
42:19
SPEAKER_01: So speaking of wins,
42:21
SPEAKER_01: As an engineering leader at an organization that is adopting AI,
42:27
SPEAKER_01: you know,
42:27
SPEAKER_01: there's either a mandate or you want to do it or both.
42:30
SPEAKER_01: How can you measure what is working?
42:32
SPEAKER_01: And before you told me about this company called WorkHuman,
42:36
SPEAKER_01: who did something similar,
42:37
SPEAKER_01: where they actually figured out what to measure,
42:40
SPEAKER_01: how to measure in a practical way.
42:42
SPEAKER_01: How did this work?
42:43
SPEAKER_01: What was the story there?
42:45
WorkHuman,
42:45
I think,
42:46
like many other companies,
42:49
started working with Copilot.
42:51
SPEAKER_00: It's very accessible to most developers.
42:54
SPEAKER_00: And what they found was they knew that it was working.
42:57
SPEAKER_00: They'd heard from the developers that they enjoyed using this tool.
43:01
SPEAKER_00: But what was really hard was figuring out
43:04
SPEAKER_00: how to quantify how much it was working and where it was working.
43:08
And so what WorkHuman did was they used those metrics in the AI
43:14
measurement
43:15
SPEAKER_00: framework across utilization,
43:16
impact and cost to try to figure out how we can actually kind
43:21
of draw a map of
43:22
SPEAKER_00: where things are going well and where things are not going well.
43:25
SPEAKER_00: So what they found was that looking at developer experience more broadly,
43:31
SPEAKER_00: they were able to figure out that AI
43:34
has a good impact on our company because it improves developer
43:36
experience.
43:37
SPEAKER_00: And I think this is sort of,
43:39
SPEAKER_00: very broadly speaking,
43:40
the advice that I give to every single company when they're
43:42
trying to figure out,
43:43
SPEAKER_00: like,
43:43
SPEAKER_00: how do I reason about AI's impact on my company?
43:48
SPEAKER_00: I tell them, AI is a tool to improve developer experience.
43:51
SPEAKER_00: When you improve developer experience, you have better outcomes.
43:54
SPEAKER_00: It follows like that.
43:55
SPEAKER_00: It's the same pattern at every single company.
43:57
SPEAKER_00: AI isn't this like magic bullet that's going to solve everything.
44:01
SPEAKER_00: We're talking about improving developer experience.
44:03
SPEAKER_00: And so WorkHuman found that it's an 11% boost in developer experience.
44:08
SPEAKER_01: And how did they find, was it via kind of a survey?
44:12
SPEAKER_01: Did they measure data?
44:13
SPEAKER_01: Is it a mix?
44:15
yeah they're measuring a mix so they're using they use dx so
44:18
they use the developer
44:19
experience index which is our kind of composite metric of 14
44:24
research-backed
44:26
developer experience drivers so everything from incident
44:29
management to local dev
44:30
iteration speed lots of these different factors that play a big
44:34
important role in
44:35
the day-to-day work of developers so they're seeing you know an
44:40
11 percent gain and
44:41
SPEAKER_00: that's all correlated as well to time savings so
44:44
SPEAKER_01: And just so I understand,
44:45
SPEAKER_01: I think for anyone like saying,
44:47
SPEAKER_01: so they were measuring these things before,
44:50
SPEAKER_01: right?
44:51
SPEAKER_01: And then as they rolled it out, they kept measuring it.
44:53
SPEAKER_01: And now they're seeing a gain, right?
44:56
SPEAKER_01: Because you can only get an improvement on something that you measure.
44:58
SPEAKER_01: So if I'm working at a company which never measured anything,
45:01
SPEAKER_01: I'm going to have like,
45:03
SPEAKER_01: I can measure now and have a baseline.
45:04
SPEAKER_01: But unless I did it before AI,
45:06
SPEAKER_01: it's going to be a bit harder for me to tell,
45:08
SPEAKER_01: you know,
45:09
SPEAKER_01: how much has it improved because I don't have data before.
45:12
SPEAKER_00: Yeah, exactly.
45:13
SPEAKER_00: And so my other general advice is start measuring now.
45:18
SPEAKER_00: We can pull in historical data to cover some gaps.
45:22
SPEAKER_00: We can look at GitHub and Jira and those tools historically.
45:26
When it comes to surveying on the developer experience and from
45:29
those self-reported
45:30
SPEAKER_00: things,
45:30
SPEAKER_00: just start small.
45:32
SPEAKER_00: Just get started.
45:33
SPEAKER_00: Don't wait to hire that other person.
45:34
SPEAKER_00: Don't wait to do anything.
45:36
SPEAKER_00: You're just delaying success.
45:38
SPEAKER_00: So
45:39
SPEAKER_00: Get started, get a baseline.
45:40
That's what helped WorkHuman be able to figure out what the
45:44
biggest gains were.
45:45
SPEAKER_01: And so what were the gains that they found?
45:47
SPEAKER_01: Is it 11% across the board?
45:49
SPEAKER_01: What does that mean?
45:50
SPEAKER_00: Yeah.
45:51
So they were they kind of measured organization wide and found
45:54
that developer
45:55
SPEAKER_00: experience went up 11 percent across the whole organization,
45:58
SPEAKER_00: which was great.
45:59
SPEAKER_00: The developers that they found that use AI.
46:04
So we're just segmenting here like daily,
46:06
weekly users with users who are not.
46:09
had a 15 higher velocity so they were able to ship more code to
46:14
production get more
46:16
stuff done than non-users these numbers are from several months
46:21
ago and what
46:21
they've noticed is like it just keeps compounding and getting
46:25
higher and higher so
46:26
this is a great example of you know we know and if you and i
46:31
are developers like we
46:32
SPEAKER_00: know that these tools are
46:34
delightful to use most of the time we know that they make a big
46:38
difference when it
46:39
comes to you know enjoying work and and bringing the joy back
46:43
to doing development
46:44
work for a lot of us um but going to your vp or cto or board
46:50
and saying well the
46:50
developers like to use them is not a you know like that's a
46:55
that's a that's a hard
46:56
SPEAKER_00: sell to keep the wallet open for getting more funding for these tools
47:00
SPEAKER_01: Yeah,
47:01
SPEAKER_01: yeah,
47:01
SPEAKER_01: because these things cost money,
47:02
SPEAKER_01: especially now with the agentic tokens,
47:05
SPEAKER_01: you know,
47:05
SPEAKER_01: they can burn through a lot of money.
47:06
SPEAKER_00: Yeah, absolutely.
47:08
SPEAKER_00: That's actually another thing that,
47:10
SPEAKER_00: you know,
47:10
SPEAKER_00: is worth discussing here because in a lot of these,
47:14
SPEAKER_00: you know,
47:14
companies that have been using AI for a longer time and are a
47:18
little bit more
47:18
SPEAKER_00: mature,
47:19
SPEAKER_00: they were sort of operating in that binary,
47:21
SPEAKER_00: you have a license and you don't have a license.
47:23
SPEAKER_00: Right now,
47:24
one of the biggest challenges when it comes to measurement in
47:26
front of us is
47:27
looking at consumption-based pricing and figuring out who are
47:32
the developers that
47:32
SPEAKER_00: have the most to gain?
47:34
Who are the developers that have the least to gain from being
47:38
able to have access
47:40
SPEAKER_00: to more tokens?
47:41
SPEAKER_00: What are the use cases that have the biggest impact?
47:45
SPEAKER_00: For example, stack trace analysis.
47:48
Because what I'm hearing from engineering leaders right now on
47:50
the ground is like,
47:52
SPEAKER_00: I just don't know how to allocate the buckets of money.
47:55
Do I give a bigger bucket of money to senior engineers than
47:59
junior engineers?
48:00
SPEAKER_00: Or is it the other way around?
48:02
Do I give a bigger bucket to junior engineers because I can get
48:05
more productivity
48:07
value from them with AI assistance and the senior engineers
48:11
don't need as much?
48:12
this is just so interesting because i feel a little bit of deja
48:15
vu now if you
48:16
remember back to the last time in in the tech industry where
48:20
we've had companies
48:22
spend thousands of dollars per year on developer tools was
48:26
around 2000 to 2010
48:29
where a lot of startups and a lot of tech companies spent about
48:33
three thousand to
48:34
SPEAKER_01: up to five or up to eight thousand dollars per year per developer
48:39
on visual studio licenses which was not just for visual studio
48:43
but it was for
48:43
documentation this was pre-internet documentation was terrible
48:47
on the internet and
48:48
that was amazing and access to early release software where
48:51
they could use like cql
48:53
SPEAKER_01: server all these things
48:54
SPEAKER_01: And it was huge amounts of money.
48:56
And even startups paid because they could get an all-in-one
49:00
development kit for
49:01
SPEAKER_01: like thick clients,
49:02
SPEAKER_01: so like Windows applications,
49:03
SPEAKER_01: the web database servers,
49:05
SPEAKER_01: et cetera.
49:06
SPEAKER_01: And this lasted for a few years.
49:08
SPEAKER_01: It kind of died out.
49:09
SPEAKER_01: But there was a case where almost every company who said,
49:12
SPEAKER_01: and they did it because,
49:13
SPEAKER_01: well,
49:13
SPEAKER_01: they could just use open source,
49:15
SPEAKER_01: but it was slower.
49:16
SPEAKER_01: And they were like,
49:17
SPEAKER_01: oh,
49:17
SPEAKER_01: if we're paying 100K for a developer per year,
49:20
SPEAKER_01: even back then it was that,
49:21
SPEAKER_01: we'll pay 8K more
49:22
or 10K more per year to get them more productive and to get
49:26
ahead of the competition.
49:27
SPEAKER_01: So we've had this before,
49:29
but now it feels we're getting back to this where there will be
49:32
companies that say,
49:33
SPEAKER_01: you know what,
49:34
SPEAKER_01: let's just bite the bullet,
49:35
SPEAKER_01: do it even if we don't have the data.
49:36
SPEAKER_01: Shopify is doing this.
49:37
SPEAKER_01: They have no budget limits.
49:39
SPEAKER_01: And there are the companies that are like, I'm not sure.
49:43
SPEAKER_01: It feels expensive.
49:45
We're used to like $200 per year per developer allocation,
49:48
that kind of stuff.
49:49
SPEAKER_00: Yeah.
49:50
SPEAKER_00: And I think history always repeats itself.
49:53
SPEAKER_00: Right.
49:53
SPEAKER_00: And when you're of a certain vintage,
49:55
SPEAKER_00: as you and I are,
49:56
SPEAKER_00: we see these patterns over and over again.
49:58
SPEAKER_00: And so,
49:59
SPEAKER_00: you know,
49:59
SPEAKER_00: that's been providing maybe some comfort to me is like,
50:02
SPEAKER_00: you know,
50:02
SPEAKER_00: I've lived through quite a lot of hype cycles I was talking with.
50:05
SPEAKER_00: Jesse Ademetz from Twilio yesterday who leads their developer platform.
50:09
SPEAKER_00: And like he and I were both like in the thick of the Kubernetes hype.
50:12
SPEAKER_00: We were kind of comparing notes of like,
50:13
SPEAKER_00: how does this feel compared to like the Kubernetes and container hype?
50:17
SPEAKER_00: You know, there's a lot that's the same.
50:18
SPEAKER_00: There's a lot that's different.
50:20
SPEAKER_00: And the hype eventually...
50:23
SPEAKER_00: it all concludes one way or another.
50:26
SPEAKER_00: So we're not still living in the container hype cycle.
50:30
SPEAKER_00: Eventually we're gonna see some stabilization.
50:33
Right now we're in this like Cambrian explosion of tool sprawl
50:38
and there's just so
50:39
SPEAKER_00: much unknown,
50:40
SPEAKER_00: pricing is unknown.
50:42
And eventually we're gonna kind of consolidate and come to
50:46
something that's a
50:47
SPEAKER_00: little bit more stable,
50:48
SPEAKER_00: but it's gonna probably take us a year or two to get there.
50:51
SPEAKER_00: But Gerger,
50:51
SPEAKER_00: I would not be surprised in 18 months if we're spending,
50:57
SPEAKER_00: yeah,
50:59
SPEAKER_00: like what did you say?
50:59
SPEAKER_00: $3,000?
50:59
SPEAKER_00: It was $3,000 to $8,000 per year.
51:00
SPEAKER_01: So that was per year.
51:01
SPEAKER_01: So on a monthly basis,
51:02
that would have been something like $400 or $300 to like $800
51:04
per month per
51:05
SPEAKER_01: developer on those tools back in 2000.
51:16
SPEAKER_00: Yeah.
51:16
SPEAKER_00: And so,
51:16
SPEAKER_00: I mean,
51:17
SPEAKER_00: we can adjust that for inflation,
51:18
SPEAKER_00: but my sort of prediction has always been,
51:22
SPEAKER_00: you know,
51:23
SPEAKER_00: or is right now,
51:24
SPEAKER_00: if we think about 18 months in the future,
51:26
I don't think it's unrealistic to spend $1,200 to $2,000 US
51:31
dollars on an agent who
51:33
SPEAKER_00: can complete tasks autonomously,
51:36
SPEAKER_00: even if they have to be verified by a human in the loop.
51:38
And I think that there are going to be companies who are
51:42
willing to open their
51:43
wallets because maybe this does allow them to avoid increasing
51:47
headcount at the
51:49
SPEAKER_00: previous rate,
51:50
or it just allows their senior developers to be spending more
51:53
time on more complex
51:55
SPEAKER_00: work.
51:56
um which is another thing that we've seen in our data at dx so
52:00
we have a core four
52:03
measurement framework which is our sort of evergreen solid
52:06
foundation of developer
52:08
productivity i'll actually i can show that right here for those
52:11
who are curious to
52:12
see it but one of the things that we look at is speed and when
52:16
we think about
52:17
SPEAKER_00: measuring ai we see ai as like an enabler
52:21
And we're going to see the impact on all of the core
52:24
measurements of productivity
52:25
SPEAKER_00: and performance.
52:26
SPEAKER_00: It's not that we need to rethink everything because AI exists.
52:29
SPEAKER_00: We still need to just go back to our fundamentals,
52:31
SPEAKER_00: understand what performance means,
52:32
SPEAKER_00: and then see the AI impact on it.
52:34
And specifically in this speed category,
52:37
what we've seen is that the dips per engineer increase.
52:41
So we're able to get more throughput,
52:43
but also the complexity of them increases as well.
52:47
SPEAKER_00: So AI users are able to work on
52:50
more complex work and get more of that work through the systems
52:55
to production,
52:57
SPEAKER_00: which is interesting.
52:58
SPEAKER_00: Part of me thinks, is that complexity good or bad?
53:02
SPEAKER_00: We see when we triangulate this data.
53:04
SPEAKER_01: It also goes back a little bit to source codes being a liability, right?
53:06
We're also seeing diffs increase and eventually opportunities
53:11
for bugs will increase.
53:12
In fact,
53:13
probably bugs will increase unless you have more thorough
53:16
testing.
53:17
SPEAKER_00: You know what?
53:17
SPEAKER_00: You're very right about that.
53:20
SPEAKER_00: And actually, that's a trend that we're seeing already.
53:22
SPEAKER_00: Let me show you this from the... I'm speaking the future here.
53:26
This is from the Dora's Impact of AI study that they released a
53:28
couple months ago.
53:35
What they've seen already is that delivery throughput is
53:38
actually slowing a little
53:41
SPEAKER_00: bit because my hypothesis is batch size is increasing.
53:45
SPEAKER_00: And this is the thing about AI.
53:47
Like it doesn't change the fundamental physics of things we
53:49
already understand to
53:50
SPEAKER_00: be true about software development.
53:52
SPEAKER_00: Bigger batch sizes are riskier.
53:54
SPEAKER_00: So we want to keep batch size small.
53:57
SPEAKER_00: But here's the wild thing.
53:58
SPEAKER_01: And batch size usually it's, you know, the diff size.
54:01
SPEAKER_00: Yeah, exactly.
54:02
SPEAKER_00: How much...
54:03
SPEAKER_00: How much work is being shipped?
54:05
SPEAKER_00: Is it a small amount, small chunk, big chunk?
54:08
SPEAKER_00: And then here's this kind of forecasting,
54:10
SPEAKER_00: like if AI adoption increases by 25%,
54:13
they actually predict that it's going to be a minus 7.2%
54:18
reduction in delivery
54:19
SPEAKER_00: stability.
54:21
I mean,
54:21
we can hypothesize about the number of different reasons that
54:24
might be.
54:25
I mean,
54:25
part of it goes back to that fundamental thing that bigger
54:28
changes are riskier.
54:29
SPEAKER_00: AI makes it trivially easy to write very, very big changes all at once.
54:34
And this is something that when we put together the AI
54:38
measurement framework,
54:40
I think one of the biggest risks about measuring AI is that
54:42
when we do get tunnel
54:43
SPEAKER_00: vision on stuff like lines of code or acceptance rate,
54:48
SPEAKER_00: we miss the picture about quality,
54:51
SPEAKER_00: stability,
54:51
SPEAKER_00: reliability,
54:52
SPEAKER_00: maintainability.
54:54
And we can't get short term gains at the sacrifice of long term
54:59
stability.
55:00
SPEAKER_00: Like we know that that's not a viable solution.
55:03
And in order to be able to protect yourself,
55:05
you have to have good measurements in place.
55:07
that you're seeing all parts of the picture and not just hyper
55:10
focusing on the speed gains.
55:13
SPEAKER_01: Yeah.
55:13
SPEAKER_01: And as a software company,
55:14
SPEAKER_01: just taking this a little bit further,
55:16
SPEAKER_01: again,
55:17
SPEAKER_01: let's put our optimistic glasses on where this thing works.
55:20
And like,
55:21
you know,
55:21
most of the code is good and we can even do a bit better
55:23
testing with AI.
55:24
So like the quality will not like degrade that much or it might
55:27
still stay the same.
55:30
SPEAKER_01: What is the best case outcome?
55:31
Because people are starting to ask this question I'm seeing in
55:34
social media.
55:35
SPEAKER_01: AI has been around for two and a half years, or coming up three years.
55:39
SPEAKER_01: Starting from chat, GPT has been around for earlier.
55:41
SPEAKER_01: But again, let's just take that for the sake of it.
55:44
SPEAKER_01: But as an end user of a customer,
55:46
SPEAKER_01: of a company that has invested heavily in AI,
55:49
SPEAKER_01: maybe Google,
55:50
SPEAKER_01: Microsoft,
55:51
SPEAKER_01: a startup.
55:52
SPEAKER_01: What should you be seeing?
55:54
SPEAKER_01: Obviously,
55:54
SPEAKER_01: you know,
55:54
SPEAKER_01: from the company's perspective,
55:56
SPEAKER_01: maybe they're doing the same with fewer people,
55:59
SPEAKER_01: but should you be seeing higher quality,
56:02
SPEAKER_01: more frequent iteration,
56:04
SPEAKER_01: better bang for your buck,
56:05
SPEAKER_01: if you will,
56:05
SPEAKER_01: no price changes and more functionality?
56:08
SPEAKER_01: What might we see?
56:09
SPEAKER_01: Or is it just like the cloud,
56:10
SPEAKER_01: which is,
56:11
SPEAKER_01: it's been a cost exercise,
56:13
SPEAKER_01: reliability,
56:14
SPEAKER_01: you know,
56:14
SPEAKER_01: might be higher in some cases or dependability will be go,
56:18
SPEAKER_01: but end users don't see anything.
56:20
SPEAKER_01: You know, you don't know if you're,
56:21
your service or your company that you're using has their own
56:25
infrastructure or the
56:26
SPEAKER_01: cloud,
56:27
SPEAKER_01: you don't care.
56:27
SPEAKER_01: The company very much cares and they can do all sorts of tricks there.
56:32
SPEAKER_00: I think it's all of the above, most likely.
56:35
SPEAKER_00: So as an end user, what I expect is faster time to market.
56:40
SPEAKER_00: And that's really on the other side,
56:42
SPEAKER_00: on the building side,
56:44
what we're trying to emphasize and what a lot of our
56:46
conversations have focused
56:47
SPEAKER_00: around with other executives and engineering leaders.
56:50
SPEAKER_00: We're really trying to reduce the amount of time to market.
56:53
SPEAKER_00: Yeah.
56:54
SPEAKER_00: So I think this has a lot of implications.
56:56
SPEAKER_00: You know, software is developed usually right now very sequentially.
57:00
We have a roadmap and maybe we have a Gantt chart of what we're
57:03
doing now.
57:04
SPEAKER_01: We have a PRD.
57:06
We have a meeting with all these business stakeholders because
57:09
we know it will be
57:09
SPEAKER_01: expensive to change later,
57:11
SPEAKER_01: right?
57:11
SPEAKER_00: This is another one of my maybe unconventional right now,
57:15
SPEAKER_00: maybe a little off the rails opinions,
57:17
SPEAKER_00: but I think roadmaps are on their way out in the age of AI.
57:22
I think companies that are going to win with AI are not ones
57:26
that think about
57:26
SPEAKER_00: things in roadmap sequential form,
57:28
SPEAKER_00: but think about it more as experiment portfolios.
57:31
And so rapid experimentation and trying to figure out what does
57:35
delight your
57:35
SPEAKER_00: customers is going to help companies win.
57:39
SPEAKER_00: I think the companies that will win are the ones that focus,
57:41
SPEAKER_00: that already have muscles to do experimentation,
57:44
SPEAKER_00: A-B tests,
57:45
SPEAKER_00: trying to figure out,
57:46
SPEAKER_00: you know,
57:46
SPEAKER_00: how to delight their customers.
57:48
SPEAKER_00: I think as an end user, what I don't want is thrashing as an end user.
57:54
SPEAKER_00: And I can see that happening.
57:55
It's just like,
57:55
oh,
57:55
we're just gonna,
57:56
because now there's fewer reasons to say no to things.
57:59
SPEAKER_00: Yeah.
58:00
And there's probably good reason that some of those things
58:03
weren't built yet and
58:04
SPEAKER_00: they're sitting on your backlog.
58:06
SPEAKER_00: And now that it's not trivial,
58:09
SPEAKER_00: but much easier to build those things,
58:11
SPEAKER_00: that doesn't mean that as an end user,
58:12
SPEAKER_00: I'm gonna find it useful.
58:14
SPEAKER_00: And I don't wanna see thrashing and just like feature bloat.
58:17
But what I do wanna see is faster time to market for the things
58:22
that have been
58:22
SPEAKER_00: validated and experimented already that really do delight end users.
58:27
SPEAKER_00: The same is true, I don't care
58:29
SPEAKER_00: Whether the application I'm using is running on Kubernetes or not,
58:35
SPEAKER_00: or if it's running in Azure or it's running in AWS really doesn't,
58:40
SPEAKER_00: I don't care.
58:41
SPEAKER_00: I just want the end user experience to be great.
58:44
SPEAKER_00: And I don't care if AI was used to development or not.
58:48
SPEAKER_00: I just want the great experience as an end user.
58:52
As a company who might be rolling out either as an engineering
58:55
leader or you're a
58:56
SPEAKER_01: tech lead there,
58:57
SPEAKER_01: have you seen a good rollout be for these AI tools?
59:01
Is there a case study or a company that you might have observed
59:05
who actually did a
59:06
SPEAKER_01: pretty good job in figuring out what to measure,
59:08
SPEAKER_01: how to roll out,
59:09
SPEAKER_01: how to deal with things like reliability,
59:12
SPEAKER_01: those kind of things?
59:14
SPEAKER_00: Yeah.
59:14
SPEAKER_00: You know what's wild,
59:15
SPEAKER_00: Gerge,
59:15
SPEAKER_00: is that I'm seeing that highly regulated industries,
59:19
SPEAKER_00: financial,
59:20
SPEAKER_00: insurance,
59:21
SPEAKER_00: pharma,
59:22
SPEAKER_00: are having the best results from introducing AI tools.
59:26
Here's my reason,
59:29
is because they have to be so deliberate and structured in
59:32
rolling out.
59:33
SPEAKER_00: And what we have found is that structured rollouts get the best results.
59:39
And so it's that whole like slow is smooth and smooth is fast
59:43
kind of thing.
59:44
SPEAKER_00: So,
59:46
SPEAKER_00: you know,
59:47
SPEAKER_00: I've had many conversations with very large banks who are far ahead,
59:52
SPEAKER_00: I would say,
59:52
of their kind of tech counterparts at even smaller companies
59:56
because they've had to
59:57
SPEAKER_00: be so intentional.
59:59
SPEAKER_00: with acceptable use policies,
1:00:01
SPEAKER_00: with licensing,
1:00:03
with budget and finance and kind of making sure everything is
1:00:07
very intentional.
1:00:08
SPEAKER_01: Making sure how you're not going to have sensitive PII data leak.
1:00:11
SPEAKER_00: Absolutely.
1:00:12
SPEAKER_00: And the more intentional and structured a rollout it is,
1:00:15
SPEAKER_00: the higher chance it has to be successful.
1:00:18
SPEAKER_00: So that's one thing.
1:00:19
SPEAKER_00: Structure is everything.
1:00:20
So one of the companies that has actually a really good story
1:00:23
about structured
1:00:24
SPEAKER_00: rollout and adoption is Indeed.
1:00:26
SPEAKER_00: So Indeed.com is a global job site talent matching service.
1:00:32
They do a lot with helping people find careers that are
1:00:36
meaningful to them.
1:00:37
SPEAKER_00: Indeed is operating at enterprise scale.
1:00:41
And one of the things that is very much built into the fabric
1:00:44
of what Indeed does
1:00:46
SPEAKER_00: is this experimental mindset.
1:00:48
And so one of the ways that showed up with their structured
1:00:51
rollout was actually
1:00:53
trialing a bunch of different tools and doing like a cohort
1:00:56
analysis of which tools
1:00:59
SPEAKER_00: were working better for which use cases and which individuals.
1:01:03
SPEAKER_00: Because,
1:01:03
SPEAKER_00: you know,
1:01:04
SPEAKER_00: as I said before,
1:01:05
we're in kind of like the Cambrian explosion when it comes to
1:01:08
all the different
1:01:09
SPEAKER_00: tools.
1:01:09
And it can be very overwhelming for engineering leaders or for
1:01:13
executive teams to
1:01:14
SPEAKER_00: like have a bag of cash.
1:01:16
SPEAKER_00: And then they want the competence that they're actually
1:01:20
spending that bag of cash on the right stuff their wallets are
1:01:23
open and so what
1:01:24
they did was kind of segment groups and do trials of different
1:01:29
tools and then look
1:01:30
at the results comparatively and then from there they figured
1:01:33
out okay well let's
1:01:34
do you know these tool these two i've seen other companies take
1:01:37
the same approach
1:01:38
SPEAKER_00: and they ended up with just one vendor but this is like a very structured
1:01:42
methodical approach to sort of run a controlled experiment and
1:01:46
figure out what are
1:01:47
SPEAKER_00: the tools that are giving us the most gains
1:01:50
I think the other thing that they did experimentation wise is
1:01:55
just trying to figure
1:01:56
SPEAKER_00: out like,
1:01:58
SPEAKER_00: can we,
1:02:00
SPEAKER_00: we have a hypothesis.
1:02:01
SPEAKER_00: So for example, high latency code reviews are bad for everyone.
1:02:05
They've got teams all over the world and say you're a developer
1:02:08
in Seattle working
1:02:09
SPEAKER_00: with a developer in,
1:02:11
SPEAKER_00: I don't know,
1:02:12
SPEAKER_00: Vienna,
1:02:12
SPEAKER_00: Austria is where I am.
1:02:13
SPEAKER_00: So,
1:02:14
SPEAKER_00: you know,
1:02:15
SPEAKER_00: obviously,
1:02:15
SPEAKER_00: if I'm in Seattle and I do something at the end of or middle of my day,
1:02:21
SPEAKER_00: my coworker in Vienna is already probably sleeping or offline.
1:02:25
SPEAKER_00: And this just leads to high turnaround time.
1:02:27
And so they had an idea like,
1:02:29
what if we could use an AI tool for code review?
1:02:33
which maybe it's not going to be perfect or as thorough as that
1:02:36
human code review,
1:02:37
SPEAKER_00: but at least I could unblock someone early,
1:02:40
SPEAKER_00: you know,
1:02:41
SPEAKER_00: at the time that they need it and give them preliminary feedback.
1:02:43
So closing the feedback loop,
1:02:45
which we know is an important part of developer experience.
1:02:47
SPEAKER_00: And so does that hypothesis hold up?
1:02:50
I think that's the one thing that I really appreciate about
1:02:53
their approach is that
1:02:54
SPEAKER_00: they treat these like experiments with hypotheses that they're trying to,
1:02:58
SPEAKER_00: you know,
1:02:58
SPEAKER_00: validate or not,
1:03:00
SPEAKER_00: and not just like,
1:03:01
SPEAKER_00: oh,
1:03:01
SPEAKER_00: well,
1:03:02
SPEAKER_00: let's just use AI for code review.
1:03:03
SPEAKER_00: That certainly sounds like it's going to help.
1:03:06
SPEAKER_00: They're taking a really methodical approach.
1:03:08
And so,
1:03:09
you know,
1:03:09
code review was one of those things to shorten feedback loops.
1:03:12
They're also looking at different use cases to figure out which
1:03:15
ones are the most impactful.
1:03:16
SPEAKER_00: So unit tests, for example, are...
1:03:19
a really good use case and figuring out how they can then
1:03:21
spread that across.
1:03:22
SPEAKER_00: Also having these sort of organization-wide change initiatives.
1:03:29
SPEAKER_00: We can think about adoption.
1:03:31
This is a use case that's come up actually a bit in the
1:03:35
academic research world.
1:03:36
There was a paper about how Google is using AI for migration
1:03:39
and some different
1:03:41
SPEAKER_00: tricks,
1:03:41
SPEAKER_00: which I see that pattern.
1:03:44
SPEAKER_00: Actually, I think I had probably four conversations about that.
1:03:46
SPEAKER_00: This week, it's only Tuesday morning.
1:03:48
SPEAKER_00: That's a really common pattern right now is using AI for migration.
1:03:51
AWS is also doing something like that with their QDeveloper Pro
1:03:55
migrating from one
1:03:56
SPEAKER_01: Java older version to a newer one.
1:03:58
SPEAKER_01: Apparently their tool is trained on that.
1:04:00
SPEAKER_01: So I've heard that from internally on like one of the,
1:04:03
SPEAKER_01: and apparently what I've heard is hit or miss,
1:04:05
SPEAKER_01: but sometimes it can work really great.
1:04:07
SPEAKER_01: So like, yeah, just plus one to migrations.
1:04:09
SPEAKER_00: Yeah, absolutely.
1:04:10
SPEAKER_00: And so they're taking this approach of like,
1:04:12
let's look for those use cases and run some experiments to
1:04:14
validate that they're
1:04:15
SPEAKER_00: actually working and then try to roll them out.
1:04:18
SPEAKER_01: You know what you mentioned at the beginning,
1:04:20
like how developers can get demotivated if AI is doing the
1:04:24
stuff that you like
1:04:25
SPEAKER_01: doing.
1:04:25
SPEAKER_01: But migrations is something that I don't know.
1:04:27
SPEAKER_01: I'm going to speak for myself.
1:04:29
SPEAKER_01: I always hated doing it and it always took longer and it was a drag.
1:04:33
And I think everyone hated doing it because you just want to
1:04:35
get it over with.
1:04:37
SPEAKER_01: You want to get it right, but... Undifferentiated heavy lifting.
1:04:42
SPEAKER_01: Yes, undefriation and heavy lifting.
1:04:44
SPEAKER_01: The business doesn't care about it.
1:04:46
SPEAKER_01: As a developer, you're not going to be a much better developer.
1:04:48
SPEAKER_01: Obviously,
1:04:49
SPEAKER_01: you're going to be a better professional,
1:04:50
SPEAKER_01: but it's not the thing you're excited about.
1:04:53
Well,
1:04:53
can AI reduce the cost of change or reduce the complexity of
1:04:57
change?
1:04:57
SPEAKER_00: And so Indeed and many other companies that I speak with are thinking,
1:05:01
what if we could proverbially knock at the door of our
1:05:05
development team with a PR
1:05:07
SPEAKER_00: that includes the migration?
1:05:08
SPEAKER_00: It's already been tested.
1:05:09
SPEAKER_00: It has an ephemeral environment.
1:05:11
SPEAKER_00: It's already been run.
1:05:12
SPEAKER_00: And all they have to do is press approve.
1:05:14
like well and you review it but but but suddenly like reading
1:05:18
through it the
1:05:18
changes that have been done it's a lot easier with the
1:05:20
migration then and figuring
1:05:22
out does this match the you know like our coding standard with
1:05:26
the modern language
1:05:27
features that i expect as opposed to looking it up because with
1:05:30
the migration
1:05:31
especially from an old framework version you have to look up
1:05:33
how it used to work
1:05:35
which hard to find documentation is outdated and then obviously
1:05:39
you want to get the
1:05:40
SPEAKER_01: modern right but yeah i
1:05:42
SPEAKER_01: I wonder if this will be a great use case,
1:05:44
SPEAKER_01: by the way,
1:05:44
SPEAKER_01: because who wants to do it to start with?
1:05:47
SPEAKER_00: Here's a top tip, one of my favorite in my arsenal of tips.
1:05:52
So I think a lot of folks approach a migration and they give
1:05:56
one file and say,
1:05:57
SPEAKER_00: OK,
1:05:57
SPEAKER_00: migrate this to whatever,
1:05:58
SPEAKER_00: upgrade this to the next version.
1:06:00
SPEAKER_00: And that's going to give kind of mixed results.
1:06:02
SPEAKER_00: And so a different technique to use is actually do one of them by hand,
1:06:06
SPEAKER_00: like do one of the migrations by hand.
1:06:09
SPEAKER_00: Give the diff or give both files then to your,
1:06:13
SPEAKER_00: you know,
1:06:14
SPEAKER_00: model of choice and say,
1:06:16
give me a prompt that will reproduce this result for subsequent
1:06:21
files that match
1:06:22
SPEAKER_00: the structure and format of file A.
1:06:25
And you're going to get a prompt that more closely matches the
1:06:28
actual work that
1:06:29
SPEAKER_00: needs to be done instead of starting with like,
1:06:32
SPEAKER_00: oh,
1:06:32
SPEAKER_00: you know,
1:06:32
SPEAKER_00: upgrade this file from whatever version X to whatever version X.2.
1:06:37
SPEAKER_00: I guess we can call that prompt engineering.
1:06:39
SPEAKER_00: We have a lot of tips like that in the guide to AI-assisted engineering,
1:06:45
SPEAKER_00: which you can find on DX's website.
1:06:47
But we went through and interviewed 180 different companies and
1:06:51
power users of AI
1:06:53
to try to figure out what were some of the things like prompt
1:06:55
engineering,
1:06:56
Great system prompts,
1:06:58
recursive,
1:06:58
adversarial prompting,
1:06:59
all of those different things.
1:07:01
My tip about migrations isn't in there because that came at a
1:07:05
different time.
1:07:05
But there's lots of different techniques that developers can
1:07:09
use to get more out of
1:07:11
their tools because it's not just as easy as throwing a license
1:07:15
at your developers
1:07:16
SPEAKER_00: and hoping that they figure it out.
1:07:17
SPEAKER_00: They really do need support and enablement and training.
1:07:21
SPEAKER_00: And so that guide can play a good part of that.
1:07:23
SPEAKER_01: Amazing.
1:07:24
SPEAKER_01: And as closing for engineering leads,
1:07:27
tech leads who want to take groundage and they do want to use
1:07:31
AI tools when it
1:07:32
SPEAKER_01: makes sense,
1:07:33
SPEAKER_01: how can they get better at kind of,
1:07:35
SPEAKER_01: you know,
1:07:35
SPEAKER_01: avoiding the hype and sticking to what works?
1:07:38
SPEAKER_01: And what is your advice to them?
1:07:40
SPEAKER_00: Data beats hype every time.
1:07:42
SPEAKER_00: So I think it's,
1:07:44
SPEAKER_00: of course,
1:07:45
SPEAKER_00: important to keep an eye out in the industry and look for opportunities,
1:07:48
look for new and novel approaches and think how you might be
1:07:50
able to fold that into
1:07:51
SPEAKER_00: what you're doing.
1:07:52
SPEAKER_00: But ultimately,
1:07:53
SPEAKER_00: AI to work on an organizational scale needs to be thought about as an
1:07:58
SPEAKER_00: organizational problem.
1:07:59
SPEAKER_00: And you need to have
1:08:01
really solid organizational hygiene when it comes to measuring
1:08:05
your performance,
1:08:07
SPEAKER_00: treating AI as an experiment,
1:08:08
SPEAKER_00: trying to figure out then what's the impact of AI.
1:08:10
So get your baseline measurement as quickly as you can and then
1:08:16
start running experiments.
1:08:18
Don't just expect AI to be a silver bullet or that every
1:08:21
engineer is going to
1:08:22
SPEAKER_00: inherently know exactly how to use it.
1:08:24
SPEAKER_00: Because just like any other tool, we still need training and enablement.
1:08:28
SPEAKER_00: So when you have the data and you can storytell around the data,
1:08:31
that's going to also protect you from the hype cycle and maybe
1:08:35
some unnecessary
1:08:37
SPEAKER_00: pressure from inflated expectations from the media.
1:08:39
SPEAKER_00: So that's my advice.
1:08:41
SPEAKER_00: Data beats hype.
1:08:43
SPEAKER_01: Love it.
1:08:43
SPEAKER_01: So to wrap up, I'll just do some rapid questions if that's okay with you.
1:08:47
SPEAKER_01: I ask and then you just shoot what comes.
1:08:50
SPEAKER_01: What's a tool, either digital or physical, that you love using and why?
1:08:55
I have to shout out to Granola because there has never been a
1:08:59
tool that I've used
1:09:00
that's like dramatically increased my quality of life as much
1:09:04
as Granola has.
1:09:05
SPEAKER_00: I'm a bad note taker and a forgetful person.
1:09:10
SPEAKER_01: And this is the AI meeting note taker, right?
1:09:13
SPEAKER_00: Yeah.
1:09:14
SPEAKER_00: So what Granola does,
1:09:16
SPEAKER_00: of course,
1:09:16
make sure that you get permission from the person that you're
1:09:19
in a meeting with.
1:09:19
SPEAKER_00: But, you know, it's an AI note taker.
1:09:21
But what I love about it is that I can take my own sort of like
1:09:24
disjointed notes
1:09:25
and then Granola comes back and fills them in with all of the
1:09:29
context that I
1:09:29
SPEAKER_00: missed.
1:09:30
SPEAKER_00: It's really magical.
1:09:32
And this was like one of the times that I was like,
1:09:36
wow,
1:09:36
AI really is amazing.
1:09:38
SPEAKER_00: But it's been such a transformational tool.
1:09:40
SPEAKER_01: What is a book that you would recommend and why?
1:09:44
SPEAKER_00: There's two books that I want to recommend, if that's okay.
1:09:46
First one is like a more business book and it's called Write
1:09:48
Useful Books by Rob Fitzpatrick.
1:09:50
This will get your writing clear and snappy and just cut out a
1:09:55
lot of the fluff.
1:09:56
SPEAKER_00: So it's incredibly useful.
1:09:58
SPEAKER_00: It's meant to be skimmed and you can really digest it in like an hour.
1:10:02
SPEAKER_00: So use it.
1:10:03
SPEAKER_00: I think the other book that's not a professional book,
1:10:06
SPEAKER_00: but I think useful nonetheless is called Unsavory Truth by Marian Nestle.
1:10:11
SPEAKER_00: She's like a food...
1:10:12
scientist author historian and it really breaks down like food
1:10:18
marketing and the
1:10:19
food lobby more more in the united states and about how like
1:10:23
everything is kind of
1:10:24
constructed and i think right now we're talking about ai hype
1:10:27
we're talking about
1:10:28
like media literacy data literacy there has been no book that
1:10:31
has really just
1:10:32
SPEAKER_00: strengthened my understanding of like
1:10:34
how deep this all goes with politics and funding and lobbyists
1:10:38
than that particular book.
1:10:39
SPEAKER_00: It is about food and not about tech,
1:10:41
but a lot of the concepts are really transferable and it's just
1:10:45
very fascinating.
1:10:46
SPEAKER_01: Well, amazing.
1:10:48
SPEAKER_01: Lara,
1:10:48
this was nice to go a bit more deeper into data and seeing what
1:10:55
actually works and
1:10:56
SPEAKER_01: doesn't work.
1:10:57
SPEAKER_01: So it was really good to have you here.
1:10:59
SPEAKER_00: Yeah, thanks, Gary.
1:11:00
SPEAKER_01: I hope you enjoyed this data-driven conversation with Laura.
1:11:03
I really like how Laura said that the best way to deal with
1:11:05
hype is with data.
1:11:07
SPEAKER_01: To check out more data and detailed case studies that Laura referenced,
1:11:10
SPEAKER_01: see these collected in the show notes below.
1:11:12
For more in-depth reading about how we can use AI tools as
1:11:14
software engineers in a
1:11:15
SPEAKER_01: grounded way,
1:11:16
SPEAKER_01: check out the Pragmatic Engineer Deep Dives,
1:11:18
SPEAKER_01: also linked in the show notes.
1:11:19
SPEAKER_01: If you enjoyed this podcast,
1:11:20
SPEAKER_01: please do subscribe on your favorite podcast platform and on YouTube.
1:11:23
This helps more people discover the podcast,
1:11:25
and a special thank you if you leave a rating.
1:11:27
SPEAKER_01: Thanks, and see you in the next one.