Latest / The Pragmatic Engineer / Beyond Vibe Coding with Addy Osmani

Transcript

Beyond Vibe Coding with Addy Osmani · The Pragmatic Engineer

0:00
SPEAKER_01: What does vibe coding mean to you?
0:01
SPEAKER_01: Vibe coding is not the same as AI-assisted engineering,
0:04
and I feel like that distinction is kind of critical because we
0:06
don't want to
0:07
SPEAKER_01: devalue the discipline of engineering.
0:09
SPEAKER_00: How do you personally use these tools?
0:11
I have more recently been focusing on the idea of spec-driven
0:16
development,
0:16
SPEAKER_01: having a very clear plan of what it is that I want to build.
0:20
If you are open to tasks,
0:21
it can be a great way of de-risking your use of LLMs in coding.
0:25
One thing that I'm kind of noticing on myself already is losing
0:28
a little bit of criticality.
0:29
It's going to continue to be very important for us to be able
0:31
to think through how
0:33
SPEAKER_01: things work,
0:33
be able to problem solve without necessarily relying on the AI
0:36
testing and
0:37
SPEAKER_01: retesting your critical thinking skills are going to be important.
0:39
You're working inside a larger team,
0:41
the Chrome team,
0:42
other teams as well.
0:43
SPEAKER_00: What are things that you're observing?
0:44
SPEAKER_00: At a company like Google, with AI, what we've realized is...
0:48
How do professional software engineers and the likes of Google
0:50
go beyond Vibe
0:51
SPEAKER_00: Coding to speed up their day-to-day work with AI?
0:54
SPEAKER_00: Adi Osmani has worked on the Chrome team for 13 years,
0:57
SPEAKER_00: and if you ever opened up the developer tab on Chrome,
0:59
SPEAKER_00: you've definitely used the stuff he built.
1:02
SPEAKER_00: He is also a prolific author.
1:03
His latest book is titled Beyond Vibe Coding and is aimed at
1:07
professional software engineers.
1:08
SPEAKER_00: Today,
1:09
SPEAKER_00: we go into Vibe coding versus AI-assisted engineering,
1:12
and why Vibe coding isn't useful for much more than just
1:15
prototyping something
1:16
SPEAKER_00: quick and dirty.
1:17
SPEAKER_00: The importance of understanding what the model does,
1:20
why Addy always reached to the thinking log of the model he
1:23
uses to make sure he
1:24
SPEAKER_00: fully understands what it did and why before approving changes.
1:27
SPEAKER_00: New development workflows with AI,
1:29
SPEAKER_00: how things like speculative and development,
1:32
SPEAKER_00: asynchronous coding background agents,
1:33
and parallel coding with several agents are new and unexplored
1:36
areas of software
1:37
SPEAKER_00: engineering.
1:38
SPEAKER_00: and many more.
1:39
If you're a software engineer who wants to work better with AI
1:41
coding tools on the
1:42
SPEAKER_00: day-to-day and wants to build reliable software with AI tools,
1:45
SPEAKER_00: then this episode is for you.
1:46
SPEAKER_00: This podcast episode is presented by Statsik,
1:49
SPEAKER_00: the unified platform for flags,
1:50
SPEAKER_00: analytics,
1:51
SPEAKER_00: experiments,
1:51
SPEAKER_00: and more.
1:52
Check out the show notes to learn more about them and our other
1:55
season sponsor.
1:56
SPEAKER_00: So Adi, welcome to the podcast.
1:58
SPEAKER_00: Thank you.
1:58
SPEAKER_00: It's a pleasure to be here.
2:00
You're an author of a lot of different books,
2:02
Leading Effective Engineering Teams.
2:03
SPEAKER_00: This came out about a year ago.
2:05
SPEAKER_00: And now your latest book is called Beyond Vibe Coding.
2:09
SPEAKER_00: And on your substack,
2:11
you also write a lot about your learnings about Vibe Coding
2:15
slash AI-assisted
2:16
SPEAKER_00: development.
2:17
SPEAKER_00: In this book and also on your blog,
2:18
SPEAKER_00: you talk about Vibe Coding versus AI-assisted software engineering.
2:22
SPEAKER_00: In your mind,
2:23
SPEAKER_00: what does Vibe coding mean to you specifically,
2:26
SPEAKER_00: and how is it different or similar to AI-assisted software engineering?
2:30
SPEAKER_01: Yeah,
2:30
so I tend to tell folks that I personally think that Vibe
2:35
coding is not the same as
2:36
SPEAKER_01: AI-assisted engineering,
2:37
and I feel like that distinction is kind of critical because we
2:41
don't want to
2:42
devalue the discipline of engineering and give folks that are
2:46
new to the industry
2:47
an incomplete picture of what it takes to build robust
2:49
production-ready software.
2:51
SPEAKER_01: But I kind of have, I guess, two definitions for them.
2:54
I think the Vibe coding is really about fully giving in to the
2:58
creative flow within AI.
2:59
SPEAKER_01: So very much focused on high-level prompting.
3:02
SPEAKER_01: And in many ways, forgetting that the code exists.
3:06
So you would,
3:06
you know,
3:06
this goes back to Andhraaj Karparathy's like original
3:10
definition,
3:10
but...
3:11
It's about accepting AI suggestions without necessarily having
3:14
a deep review and
3:16
SPEAKER_01: focusing on rapid iterative experimentation.
3:19
SPEAKER_01: Personally, I find it really great for prototypes, MVPs, and learning.
3:26
What I've seen in production teams is that this has been very
3:32
useful for them when
3:33
SPEAKER_01: it comes to trying out ideas rapidly.
3:36
And building out an intuition for what the shape of an idea
3:40
might look like,
3:41
SPEAKER_01: what a component might look like,
3:43
SPEAKER_01: what an MVP might look like.
3:45
And that tends to prioritize speed and exploration over things
3:49
like correctness and
3:50
SPEAKER_01: maintainability.
3:51
Things that we perhaps care a little bit about when it comes to
3:55
building things for
3:56
SPEAKER_01: large production audiences.
3:57
And I would say that there's a little bit of a spectrum between
4:00
vibe coding and
4:03
doing what falls a little bit closer into traditional software
4:06
engineering.
4:06
SPEAKER_01: You know,
4:07
SPEAKER_01: doing more planning,
4:08
SPEAKER_01: more specification-driven development,
4:12
SPEAKER_01: including sufficient context.
4:15
And really what is AI-assisted engineering across the full
4:20
software development lifecycle.
4:22
SPEAKER_01: So to me,
4:23
SPEAKER_01: AI-assisted engineering is where AI is this powerful collaborator,
4:27
SPEAKER_01: but it's not a replacement for engineering principles.
4:30
SPEAKER_01: And so in that model,
4:31
SPEAKER_01: it is a force multiplier,
4:33
SPEAKER_01: but it can help you across that whole cycle,
4:37
SPEAKER_01: whether it is with boilerplate,
4:39
SPEAKER_01: debugging,
4:40
SPEAKER_01: deployment.
4:41
But the big difference is that the human engineer remains
4:45
firmly in control.
4:46
SPEAKER_01: You are responsible for thinking about the architecture,
4:50
SPEAKER_01: for reviewing the code,
4:51
SPEAKER_01: and for understanding every line,
4:54
SPEAKER_01: if not most of what AI is generating for you.
4:58
SPEAKER_01: And you're on the hook,
4:59
SPEAKER_01: really,
4:59
SPEAKER_01: for making sure that the final product is secure,
5:03
SPEAKER_01: scalable,
5:03
SPEAKER_01: and maintainable.
5:04
SPEAKER_01: Maybe AI is helping you with velocity, which is great.
5:08
But that doesn't mean that you can shrug off your
5:14
responsibility for quality at the
5:16
SPEAKER_01: end of the day.
5:17
And I tend to think that a lot of people have said that they
5:22
found AI encoding to
5:23
SPEAKER_01: be a force multiplier.
5:26
But I found that the greater expertise that you have in
5:30
software engineering,
5:31
SPEAKER_01: the better the results you can get when you're using LLMs.
5:35
SPEAKER_01: And I think sometimes if you're new to the industry or you're a junior,
5:40
maybe there are some what we would consider to be traditional
5:43
best practices that
5:45
SPEAKER_01: maybe you haven't had to experience them yet or think about them.
5:48
SPEAKER_01: Right.
5:48
SPEAKER_01: You know,
5:49
SPEAKER_01: if you care about production quality programming,
5:52
you should probably only be committing code to your repo that
5:55
you can fully explain
5:56
SPEAKER_01: to somebody else.
5:57
Because just expecting that the AI is going to help you
6:01
untangle whatever mess
6:02
SPEAKER_01: happens later on is probably not going to work out long term.
6:06
SPEAKER_00: And how do you personally use these tools?
6:10
SPEAKER_00: May that be Vibe coding and especially AI-assisted engineering?
6:14
I have more recently been focusing on the idea of spec-driven
6:19
development,
6:20
SPEAKER_01: having a very clear plan of what it is that I want to build.
6:24
I think that,
6:25
you know,
6:26
there are definitely places where I still vibe code.
6:29
SPEAKER_01: If it's for a personal tool,
6:31
SPEAKER_01: something throw away or,
6:33
SPEAKER_01: you know,
6:33
SPEAKER_01: in the old days,
6:34
SPEAKER_01: if an engineer or a PM had an idea,
6:38
SPEAKER_01: maybe we would put together,
6:39
SPEAKER_01: you know,
6:40
SPEAKER_01: a quick mock.
6:41
Maybe we would put together a wireframe or a sketch or
6:44
something like that.
6:45
perhaps work with UX to come up with something a little bit
6:48
more polished.
6:49
SPEAKER_01: These days,
6:50
the fact that you can vibe code a prototype and actually show
6:54
somebody in a pull
6:55
SPEAKER_01: request or in a chat,
6:56
SPEAKER_01: like,
6:57
SPEAKER_01: hey,
6:57
here's an even more clear version of the vision that I had in
7:01
mind for this.
7:02
SPEAKER_01: I think there's something very powerful to that.
7:05
SPEAKER_01: And I'm loving vibe coding for that.
7:07
Just being able to give me a slightly higher quality way to
7:11
deliver sharing an idea.
7:13
SPEAKER_00: Vibe coding is having its moment.
7:15
SPEAKER_00: Describe an app in a prompt and boom, you've got something running.
7:19
SPEAKER_00: It feels like magic.
7:20
SPEAKER_00: But here's what Vibe coding doesn't account for.
7:23
SPEAKER_00: Institutional knowledge.
7:24
Every prompt stems from what you tell it,
7:26
not what your team already knows.
7:28
SPEAKER_00: Real product development is the opposite.
7:31
SPEAKER_00: It's accumulated context.
7:32
SPEAKER_00: For example, every book has a kind of history.
7:35
SPEAKER_00: Every feature connects to customer requests.
7:37
SPEAKER_00: Every PR fits into your team's roadmap.
7:40
A short prompt will not share all this additional context for
7:42
the agent to use.
7:43
SPEAKER_00: Linear agents work inside this additional team context.
7:46
SPEAKER_00: They live inside your development system where actual work happens.
7:49
SPEAKER_00: They see the issue blocking your sprint,
7:51
SPEAKER_00: the related PRs,
7:52
SPEAKER_00: the project goals,
7:53
SPEAKER_00: the discussions that your team already had about this exact problem.
7:57
SPEAKER_00: And because linear is your team's shared workspace,
7:59
SPEAKER_00: agents don't only see your context,
8:02
SPEAKER_00: they see your entire org's context.
8:04
SPEAKER_00: The bug report from support,
8:06
SPEAKER_00: the design spec from your PM,
8:07
SPEAKER_00: the architecture notes from the tech lead.
8:09
SPEAKER_00: So when you ask an agent to draft PR or build a feature,
8:13
SPEAKER_00: it's not improvising just from a prompt.
8:15
SPEAKER_00: It's using the same context that your team already uses.
8:18
This is what AI-powered development looks like when you're
8:20
building something real.
8:21
SPEAKER_00: Not just building fast, but building intentional.
8:25
SPEAKER_00: See how agents work in Linear at linear.app slash agents.
8:29
SPEAKER_00: And if you want to try them yourself,
8:31
SPEAKER_00: get Linear Business for free by visiting linear.app slash pragmatic.
8:36
That doesn't mean that we take the vibe coded prototype or that
8:39
code and just stick
8:41
SPEAKER_01: it in production.
8:42
SPEAKER_01: Once you have clarity around the vision that you have for a component,
8:46
SPEAKER_01: for a feature,
8:46
SPEAKER_01: for a view,
8:47
SPEAKER_01: you should probably be writing out like,
8:49
SPEAKER_01: OK,
8:49
SPEAKER_01: well,
8:49
SPEAKER_01: what are the actual expectations around this?
8:51
SPEAKER_01: What do we actually consider to be the requirements?
8:54
And that will give you a much higher quality outcome,
8:57
typically from the LLM.
9:00
SPEAKER_01: Because otherwise,
9:00
SPEAKER_01: if you're giving into the vibes,
9:02
SPEAKER_01: you're also sort of giving into,
9:05
SPEAKER_01: okay,
9:05
SPEAKER_01: well,
9:05
SPEAKER_01: you figure out the architecture.
9:07
SPEAKER_01: You figure out, like, what this should do.
9:09
SPEAKER_01: And while that's fine for ideation,
9:11
SPEAKER_01: it's probably not sufficient for production sort of product engineering.
9:16
SPEAKER_01: So for me, spec-driven development has been a big thing.
9:19
SPEAKER_01: I think that tests are great.
9:23
SPEAKER_01: And if you are open to tests,
9:25
SPEAKER_01: it can,
9:26
SPEAKER_01: you know,
9:26
SPEAKER_01: be a great way of de-risking your use of LLMs and coding.
9:29
SPEAKER_01: Because sometimes,
9:30
SPEAKER_01: you know,
9:30
SPEAKER_01: even if you're using state-of-the-art models,
9:32
you can end up in a situation where maybe the code looks more
9:36
convoluted than you
9:37
SPEAKER_01: would expect.
9:38
Or maybe your first couple of,
9:41
you know,
9:41
prompts have been generating really good code.
9:43
SPEAKER_01: And then for whatever reason, things go off the rails.
9:46
But if you can prove that things are working with tests and
9:51
that if something did
9:52
SPEAKER_01: go off the rails,
9:53
SPEAKER_01: it's clearer to you what that was,
9:55
SPEAKER_01: I think that that can help you keep your project green the whole time.
9:59
SPEAKER_01: And that's been something that's helped me quite a lot as well.
10:04
SPEAKER_01: So specter-driven development, testing.
10:07
SPEAKER_01: I try to also make sure that I'm leveraging.
10:12
SPEAKER_01: I guess this is a shout-out.
10:15
SPEAKER_01: My team just released Chrome DevTools MCP.
10:17
SPEAKER_01: So I use... Oh, amazing.
10:19
SPEAKER_01: Yeah, so we just released Chrome DevTools MCP.
10:22
SPEAKER_01: I care a lot about quality.
10:24
SPEAKER_01: And I think that in the last couple of years,
10:27
we've seen a number of cases where if you notice that something
10:30
is broken...
10:32
SPEAKER_01: I will see a lot of engineers that will just say,
10:34
SPEAKER_01: okay,
10:34
SPEAKER_01: well,
10:34
SPEAKER_01: hey,
10:35
SPEAKER_01: LM,
10:35
it looks like this button is off or it looks like this thing is
10:38
not quite exactly
10:39
SPEAKER_01: what it should be.
10:40
SPEAKER_01: Go fix it.
10:41
And that's,
10:41
again,
10:42
going a little bit closer to just rolling with the vibes.
10:46
But if you can keep things like a browser in the loop or
10:50
something that can
10:50
SPEAKER_01: actually see the page,
10:52
SPEAKER_01: That is what kind of Chrome DevTools MCP and related solutions do.
10:56
SPEAKER_01: They give your LLM eyes in many ways.
10:58
SPEAKER_01: So we can see what the browser sees.
11:00
SPEAKER_01: It can see what's rendering or what isn't.
11:03
SPEAKER_01: It can detect if there are warnings,
11:06
SPEAKER_01: errors in the console,
11:07
SPEAKER_01: maybe even get a deeper sense of what is broken.
11:10
SPEAKER_01: And that can just improve the feedback loop that you have.
11:14
And so I've been just excited about MCP for being able to help
11:17
us sort of evolve
11:19
our workflows a lot more thanks to this idea of being able to
11:22
call in to other
11:23
SPEAKER_01: tools.
11:24
SPEAKER_01: So that has been great.
11:27
SPEAKER_01: Otherwise, for me, generally speaking, I've found that, you know...
11:32
You really do need to put in the effort to get proficient with
11:36
these tools.
11:37
SPEAKER_01: And I still find myself doing that.
11:39
SPEAKER_01: If there are new models,
11:40
SPEAKER_01: new tools,
11:40
SPEAKER_01: new platforms coming out,
11:41
SPEAKER_01: I'm generally finding that I experiment a lot every week.
11:47
I try to encourage my teams to share with each other,
11:50
with me,
11:51
like,
11:51
how are things going?
11:52
Are there any insights that are worth us bubbling up or things
11:55
that we should try out as a team?
11:58
SPEAKER_01: And if your team see that you are very open to learning together,
12:02
I think that that can just create this nice culture of
12:04
psychological safety that
12:06
can set your team up for success as we're all going through
12:09
this big period of
12:09
SPEAKER_01: change.
12:10
SPEAKER_00: Yeah, and I feel what you said about it takes time to learn this thing.
12:15
SPEAKER_00: I have so many smaller and larger aha moments just by using it.
12:19
SPEAKER_00: And there's plenty of people who are skeptical,
12:23
SPEAKER_00: especially including engineers,
12:25
SPEAKER_00: who are skeptical about LLMs,
12:27
SPEAKER_00: whether it may be the theory,
12:29
SPEAKER_00: may that be energy footprint,
12:30
SPEAKER_00: other things.
12:32
But I found that a lot of people who are skeptical have either
12:35
not tried it or they
12:37
SPEAKER_00: haven't given it some time because it does take some time,
12:39
SPEAKER_00: some playing around.
12:40
SPEAKER_00: And, you know, like Rangers, it doesn't work everywhere.
12:42
SPEAKER_00: It makes mistakes.
12:45
SPEAKER_00: It screws up.
12:46
SPEAKER_00: But...
12:47
SPEAKER_00: It does.
12:47
SPEAKER_00: I think if you've not found...
12:50
SPEAKER_00: I can only tell for myself,
12:51
SPEAKER_00: but I found a bunch of ways where it helps even my smaller projects.
12:55
SPEAKER_00: And as you said, it's about what works for you and what works for others.
13:00
SPEAKER_00: Speaking of which, you're working...
13:03
SPEAKER_00: in a larger,
13:04
SPEAKER_00: inside a larger team,
13:05
SPEAKER_00: larger production team,
13:06
SPEAKER_00: you know,
13:06
SPEAKER_00: you've got the Chrome team,
13:08
SPEAKER_00: other teams as well.
13:09
What are things that you're observing on how others are using
13:13
it in interesting
13:14
SPEAKER_00: ways,
13:14
SPEAKER_00: maybe unexpected ways,
13:16
or even ways that maybe didn't work out for those specific
13:20
people or engineers?
13:22
SPEAKER_01: I would say that at a company like Google,
13:25
SPEAKER_01: we have this very long-term,
13:29
SPEAKER_01: very battle-tested way of thinking about software engineering at scale.
13:34
SPEAKER_01: And with AI, what we've realized is a lot of that doesn't go away.
13:40
SPEAKER_01: You still want to care about quality.
13:42
SPEAKER_01: You still want to care about doing due diligence.
13:44
The things that I think we've found interesting kind of mimic
13:49
what people have seen
13:50
SPEAKER_01: in startups and other kinds of companies.
13:52
So the importance of mastering understanding,
13:56
like what is prompt engineering,
13:57
right?
13:58
Like making sure that you are constructing the right set of
14:01
incantations to get the
14:03
SPEAKER_01: best outcome from an LLM and then context engineering,
14:06
SPEAKER_01: right?
14:07
SPEAKER_01: more recently?
14:08
How can you make sure that you are optimizing the context
14:11
window to increase the
14:13
SPEAKER_01: chances that those outcomes are higher quality?
14:16
SPEAKER_01: We spend a lot of time thinking about that,
14:18
SPEAKER_01: making sure that the right sets of descriptions,
14:21
SPEAKER_01: details,
14:22
SPEAKER_01: files,
14:23
SPEAKER_01: examples,
14:24
SPEAKER_01: any additional content that is
14:27
specific to a project that the LLM may not necessarily have in
14:31
its training data.
14:33
That has been very interesting and important for us as we've
14:36
been working through.
14:37
I have been somebody that has been trying to explore using AI
14:43
in every facet of my
14:44
SPEAKER_01: life in many ways over the last couple of years.
14:49
SPEAKER_01: That's been very eye-opening in terms of places where there is meaningful
14:53
SPEAKER_01: productivity gain to be had,
14:55
as well as places where either model quality or system prompts
15:00
and tool quality are
15:02
SPEAKER_01: not quite there just yet.
15:04
SPEAKER_01: And so I've been also sort of nudging my teams in that direction as well.
15:09
If you are thinking about this idea of us eventually all being
15:14
AI native engineers,
15:16
SPEAKER_01: then one prompt for you is before you try to solve a problem yourself,
15:21
SPEAKER_01: if you gave this to a model,
15:22
SPEAKER_01: you gave this to AI,
15:25
SPEAKER_01: what would it do?
15:26
Would it actually help you accomplish your goal much faster or
15:30
would it likely slow you down?
15:31
SPEAKER_01: And if it would slow you down, like why is that?
15:34
But even that prompt is something that I think helps us to
15:37
learn a lot about what
15:39
SPEAKER_01: is possible and what isn't.
15:40
SPEAKER_01: Many of us have been on this journey where,
15:43
SPEAKER_01: you know,
15:43
SPEAKER_01: if you're a classic software engineer,
15:46
SPEAKER_01: if you're a web developer,
15:47
SPEAKER_01: whatever,
15:49
SPEAKER_01: you probably don't have this very deep understanding of AI.
15:54
SPEAKER_01: For myself and for some of my leads,
15:56
SPEAKER_01: I set us this task of like,
15:57
SPEAKER_01: okay,
15:58
SPEAKER_01: well,
15:58
let's start to become slightly deeper experts in these areas so
16:01
that we can guide
16:02
SPEAKER_01: our team towards,
16:04
SPEAKER_01: well,
16:04
SPEAKER_01: where would it be useful for you to build up this expertise to?
16:07
SPEAKER_01: So again,
16:08
SPEAKER_01: in the last year,
16:09
I've been spending a lot more time on thinking about things
16:11
like evals and
16:13
benchmarks and how much should we be caring about rag versus
16:18
fine tuning and so on.
16:20
SPEAKER_01: And this ends up contributing as well,
16:22
because at the same time that we are talking about AI assisted
16:26
engineering,
16:27
SPEAKER_01: many of the products that we work on also have this consideration for,
16:32
SPEAKER_01: well,
16:32
SPEAKER_01: is AI going to help you deliver a better customer experience in some way?
16:37
And so I'm trying to look at like,
16:39
how can the work that we're doing for our coding workflow
16:43
SPEAKER_01: also benefit the product work that we're going to have to do.
16:46
SPEAKER_01: So that's been a good learning for us.
16:49
SPEAKER_01: I think,
16:49
SPEAKER_01: honestly,
16:50
just the importance of always maintaining human oversight has
16:53
been one of the
16:55
SPEAKER_01: bigger learnings.
16:57
SPEAKER_01: We have, of course, like I know a lot of people have run into this.
16:59
SPEAKER_01: We've,
16:59
SPEAKER_01: of course,
16:59
SPEAKER_01: seen,
17:00
SPEAKER_01: you know,
17:01
cases where people very often like external contributors will
17:05
sometimes be very
17:07
SPEAKER_01: passionate about like,
17:08
SPEAKER_01: hey,
17:08
SPEAKER_01: I want to contribute to your project.
17:09
But then they'll use an LLM where you're going with the use an
17:12
LLM and submit
17:13
SPEAKER_00: something that I think Mitchell,
17:17
SPEAKER_00: uh,
17:17
Hashimoto was blowing up on social media because he just had
17:21
enough that people
17:21
SPEAKER_00: were submitting things that,
17:24
SPEAKER_00: as you said,
17:25
SPEAKER_00: out of good intention,
17:26
SPEAKER_00: but putting way more load on maintainers.
17:28
I love reading studies about how different segments of the
17:33
developer population are
17:34
SPEAKER_01: finding AI and what that means for their teams.
17:38
SPEAKER_01: And one of the studies that I recently read
17:41
highlighted that if you are increasing the velocity of code and
17:47
improvements that
17:48
SPEAKER_01: can land in the team and you are doing human oversight,
17:52
SPEAKER_01: the human review is going to become the bottleneck.
17:55
SPEAKER_01: And that is what teams are starting to realize.
17:57
SPEAKER_01: Like, oh, wait, we're starting to see a lot more PRs come through.
18:01
SPEAKER_01: But who's going to review those?
18:03
SPEAKER_01: Right.
18:03
SPEAKER_01: So I'm glad that teams are,
18:06
SPEAKER_01: you know,
18:07
at least some teams seem to be caring about that quality
18:10
dimension sufficiently to
18:11
SPEAKER_01: actually hand review themselves.
18:13
SPEAKER_01: But that does also mean that our our workflows may have to evolve.
18:20
SPEAKER_01: I have seen,
18:21
SPEAKER_01: you know,
18:21
SPEAKER_01: a lot of talk about like,
18:22
SPEAKER_01: yeah,
18:22
SPEAKER_01: why aren't we using,
18:23
SPEAKER_01: you know,
18:24
SPEAKER_01: LLMs to also do the code review?
18:26
And that's a little bit of a slippery slope because if the AI
18:30
is writing the code
18:31
SPEAKER_01: and you haven't studied it carefully,
18:33
SPEAKER_01: but the AI is also reviewing the code,
18:36
SPEAKER_01: are you actually sure about what's landing?
18:38
So I think that the best practices around code review are still
18:41
something that are evolving.
18:43
SPEAKER_01: And I'm excited to see where that goes.
18:45
SPEAKER_01: For me personally, there are a lot of tools that I use.
18:48
SPEAKER_01: Some of my favorite ones include using Klein in VS Code.
18:53
SPEAKER_00: You're a big fan of people follow your writing on Klein.
18:56
SPEAKER_01: Yeah, I love Klein.
18:58
I think that people can do a lot with Cursor and Copilot and VS
19:02
Code these days as well.
19:03
But one of the things that I enjoy doing is most of these tools
19:08
will show you the
19:10
thinking that is happening behind the scenes as a solution is
19:13
being built out.
19:14
SPEAKER_01: And even if that happens quickly,
19:16
SPEAKER_01: I will try to go back,
19:17
SPEAKER_01: scroll through,
19:18
SPEAKER_01: expand,
19:19
SPEAKER_01: and read through,
19:20
SPEAKER_01: okay,
19:20
what was your thinking process through actually being able to
19:22
build this out?
19:24
SPEAKER_01: What decisions did you make?
19:25
SPEAKER_01: What did you generate?
19:26
SPEAKER_01: And I will review that code before it ends up in a pull request.
19:30
There is going to be some likelihood later on that I may have
19:33
to maintain that
19:34
SPEAKER_01: code,
19:35
that I may have to make some tweaks to that code that the LLM
19:38
is not going to be
19:39
SPEAKER_01: able to help me with.
19:40
SPEAKER_01: Just last night, I was working through a problem.
19:43
SPEAKER_01: And the code looked, you know, on paper, code looked right.
19:48
SPEAKER_01: It wasn't working the way it was supposed to.
19:50
SPEAKER_01: I asked the LLM a few times to like, hey, can you go use your tools?
19:53
SPEAKER_01: Go figure out what the problem is.
19:55
SPEAKER_01: It continued to make changes, didn't actually fix the underlying problem.
19:59
And so tonight I'll have to roll up my sleeves and go and
20:02
manually debug it.
20:03
SPEAKER_01: If I didn't understand how that code worked or I hadn't been reading it,
20:07
I would be feeling like I'd just been dropped into a jungle and
20:11
having to navigate
20:12
SPEAKER_01: it myself.
20:13
SPEAKER_00: Well, plus, I mean, I'm reflecting on this, right?
20:16
SPEAKER_00: Like, I feel this is the difference between...
20:18
a software engineer who's worth their money just in terms of
20:22
being employed.
20:24
SPEAKER_00: And one, that's not.
20:26
SPEAKER_00: If all you can do is just prompt and prompt, I mean, anyone can do that.
20:29
SPEAKER_00: I'm exaggerating, but a new grad can do that or someone still in college.
20:35
SPEAKER_00: But not everyone will have taken the effort to understand,
20:38
SPEAKER_00: to know how it works and be able to roll up their sleeves.
20:41
SPEAKER_00: And when these models fail, which they fail,
20:45
SPEAKER_00: They can fix it.
20:46
SPEAKER_00: And also they can also explain it to people.
20:48
SPEAKER_00: They can explain it in meeting coherently without rambling.
20:51
SPEAKER_00: So I feel like this is what a professional is, right?
20:54
SPEAKER_00: Like in any industry, it's like they know how the things work.
20:57
SPEAKER_00: Same with the car mechanics, same with anything else.
21:00
SPEAKER_00: So maybe it's just a reminder that if you're not doing this,
21:03
SPEAKER_00: if you're kind of letting go and like,
21:04
SPEAKER_00: oh,
21:04
SPEAKER_00: it solves my problems,
21:06
SPEAKER_00: you're kind of at the risk of,
21:08
SPEAKER_00: well,
21:09
SPEAKER_00: if you don't know how it works,
21:10
SPEAKER_00: I mean,
21:11
SPEAKER_00: why do we need you?
21:12
SPEAKER_00: Anyone can prompt another LLM.
21:14
SPEAKER_01: Exactly, exactly.
21:15
SPEAKER_01: And,
21:16
SPEAKER_01: you know,
21:17
SPEAKER_01: as we start to talk more about,
21:20
SPEAKER_01: you know,
21:21
SPEAKER_01: in the last year,
21:21
SPEAKER_01: you know,
21:23
SPEAKER_01: people working in their terminal,
21:24
their CLIs has become more of a thing once again with Cloud
21:28
Code and Gemini CLI and
21:29
SPEAKER_01: OpenCode and so on.
21:30
SPEAKER_01: And then people started talking about, OK, well, how do we orchestrate
21:34
SPEAKER_01: multiple parallel agents being able to complete work for us.
21:38
As we start thinking about this idea of each engineer almost
21:42
having a virtual team
21:44
SPEAKER_01: of their own,
21:45
being able to go off and work through your backlog and get all
21:48
of these different
21:49
SPEAKER_01: tasks implemented concurrently,
21:52
SPEAKER_01: I think that you start to quickly realize,
21:54
SPEAKER_01: well,
21:54
SPEAKER_01: that all sounds great in the abstract and in theory,
21:59
but then you're going to compound all of these other problems
22:02
where a lack of
22:03
SPEAKER_01: manual review is going to probably lead you to some level of tech debt.
22:08
SPEAKER_01: Maybe not immediately, but at some point, it very likely will.
22:12
And my experience of this kind of led me at one point to write
22:15
about what I called the 70%.
22:18
SPEAKER_01: kind of problem.
22:19
SPEAKER_00: Let's talk about that because you wrote about that extensively.
22:22
SPEAKER_00: We actually published a shared article,
22:26
SPEAKER_00: a guest article based on your article,
22:27
SPEAKER_00: which we'll also link in the notes below.
22:30
What is the 70% problem and how did you come across it and how
22:34
do you think it
22:35
SPEAKER_00: might have changed since you published it,
22:36
SPEAKER_00: which was about six months ago?
22:38
Yeah,
22:38
obviously,
22:39
you know,
22:40
model quality,
22:41
tooling quality continues to get better.
22:45
So the 70% problem is really about this idea that LLMs can
22:49
produce very roughly
22:50
SPEAKER_01: like 70% of a working application very quickly,
22:54
SPEAKER_01: but they tend to struggle with that last 30%.
22:57
SPEAKER_01: And that last 30%,
22:59
SPEAKER_01: you can consider it like the last mile,
23:01
SPEAKER_01: but it includes lots of different kinds of patterns that,
23:05
SPEAKER_01: you know,
23:06
your audience probably run across or maybe they will run across
23:09
things like the two
23:10
SPEAKER_01: steps back pattern.
23:11
SPEAKER_01: So, you know, you have used a couple of prompts to build up something.
23:16
You give an LLM one more prompt and it happens to go in a
23:20
completely different direction.
23:22
Maybe it has fully rewritten your UI or the functionality
23:26
behind your component.
23:27
SPEAKER_01: Things like that.
23:28
There are very often hidden maintainability costs,
23:32
places where you're not being specific.
23:34
You are delegating the responsibility back to the LLM and you
23:38
may end up getting
23:38
SPEAKER_01: diminishing returns.
23:40
As we've seen time and time again on Hacker News,
23:43
security vulnerabilities.
23:46
SPEAKER_01: This is exactly where we see people accidentally leaking API keys,
23:52
SPEAKER_01: where there are XSS issues,
23:53
SPEAKER_01: where there are all kinds of problems because
23:56
People did not think as holistically about the problem that
24:00
they were solving and
24:01
SPEAKER_01: just sort of gave into the vibes.
24:03
SPEAKER_01: So a vibe coded proof of concept,
24:06
SPEAKER_01: you know,
24:07
SPEAKER_01: is fine for MVP for that prototyping phase,
24:10
SPEAKER_01: but it likely needs to be rewritten with production quality in mind.
24:15
If you are going to be landing this in a code base where you're
24:17
working with other
24:18
SPEAKER_01: people,
24:18
SPEAKER_01: working with a team.
24:20
SPEAKER_01: And, you know, dealing with a real user base of people.
24:24
SPEAKER_01: I think that the security and quality aspects,
24:27
SPEAKER_01: you know,
24:28
SPEAKER_01: really speak to the need for keeping the human in the loop there.
24:31
SPEAKER_00: Yeah,
24:31
and we keep hearing stories about product managers and less
24:36
technical founders who
24:38
SPEAKER_00: get really into vibe coding,
24:39
SPEAKER_00: excited,
24:40
SPEAKER_00: you know,
24:40
SPEAKER_00: they spin up a prototype or a better version,
24:42
and then they want to build something that they can put out
24:45
into production.
24:46
SPEAKER_00: And there's a bunch of stories,
24:47
SPEAKER_00: I'll link one in the show notes,
24:49
SPEAKER_00: on how they just get stuck or it takes them a very long time.
24:53
SPEAKER_00: What they thought would take a day or two ends up being 10, 20, 30 days.
24:55
SPEAKER_00: And in the 70% problem,
24:59
SPEAKER_00: You went into something relevant for this,
25:00
which is you said that more experienced engineers can finish
25:06
the last 30% a lot
25:07
SPEAKER_00: easier,
25:08
and then less experienced engineers get actually a lot more
25:12
stuck or get this false
25:13
SPEAKER_00: confidence.
25:14
SPEAKER_01: Yes, yes, absolutely.
25:16
SPEAKER_01: I think that with that last 30%,
25:20
what you will often see junior engineers or interns or new
25:24
grads doing is they will
25:27
not really know what to do next beyond just constantly
25:32
reprompting the LLM to fix
25:34
SPEAKER_01: the issues.
25:34
SPEAKER_01: And if it's not able to do it,
25:35
they're not necessarily going to have all of those skills just
25:38
yet for debugging
25:39
SPEAKER_01: the problem or understanding where the issues are.
25:42
And so what this speaks to is the importance of having a really
25:47
good critical
25:48
SPEAKER_01: thinking,
25:49
SPEAKER_01: problem solving mindset.
25:50
SPEAKER_01: You know,
25:51
we always talked about the importance of this for people who
25:53
are going into
25:53
SPEAKER_01: computer science.
25:54
SPEAKER_01: I think that that remains now,
25:57
SPEAKER_01: but that diligence required to actually read the code,
26:00
SPEAKER_01: understand the system,
26:02
SPEAKER_01: understand how all of those pieces connect together.
26:04
SPEAKER_01: I don't think that that necessarily goes away.
26:07
SPEAKER_01: As you said earlier,
26:08
SPEAKER_01: With the tools that we have today and the models we have today,
26:11
almost anybody can give a high level prompt to a tool and have
26:16
something that seems
26:18
SPEAKER_01: like it works come out the back of it.
26:20
SPEAKER_01: But I wouldn't necessarily trust that in production.
26:24
Too many stories at this point of things just going a little
26:28
bit off the rails.
26:29
SPEAKER_00: It reminds me when this was before AI tools,
26:31
I had an engineer who joined Uber who came from a smaller
26:34
startup and a new joiner.
26:37
SPEAKER_00: So after a week, I was this person's manager.
26:39
SPEAKER_00: I asked, like, how are things going?
26:41
SPEAKER_00: And he was really distressed.
26:42
SPEAKER_00: I'm like, oh, it's really difficult.
26:43
SPEAKER_00: It's like, what's difficult?
26:44
SPEAKER_00: He's like, I'm trying to read all the code and it's just too much.
26:48
SPEAKER_00: And I was like,
26:49
SPEAKER_00: why are you trying to read all the code of Uber,
26:51
SPEAKER_00: which was like,
26:52
SPEAKER_00: you know,
26:52
SPEAKER_00: our backend system.
26:53
SPEAKER_00: It was like, you know, like just so it was a mobile app.
26:55
SPEAKER_00: It was like more than a million lines of code.
26:57
SPEAKER_00: He's like,
26:57
SPEAKER_00: well,
26:58
SPEAKER_00: I do that because whenever I join a new company,
26:59
SPEAKER_00: I read all the code to understand everything.
27:02
SPEAKER_00: And I was like, I was like, I get what you're coming from.
27:04
SPEAKER_00: And I think that's great.
27:06
SPEAKER_00: I was just like, let me explain to you how the code base works.
27:09
SPEAKER_00: You shouldn't read all the code,
27:10
SPEAKER_00: but you should understand the structure,
27:12
SPEAKER_00: where you can find things,
27:13
SPEAKER_00: because this is on the code base where we had,
27:16
SPEAKER_00: I think,
27:16
SPEAKER_00: 200 or 300 engineers working on it.
27:18
SPEAKER_00: And I thought...
27:19
This person had a really great intention,
27:23
which is,
27:23
I'm coming to a new place.
27:24
SPEAKER_00: I want to understand it.
27:26
SPEAKER_00: I'm going to spend the first few weeks understanding.
27:28
SPEAKER_00: And those people excel.
27:29
SPEAKER_00: And I'm kind of thinking we keep getting back to this thing.
27:33
SPEAKER_00: But what I'm hearing from you is if you can actually,
27:36
SPEAKER_00: to some extent,
27:37
SPEAKER_00: outsource this to an LLM,
27:38
SPEAKER_00: which is a bit of a hit or miss,
27:39
SPEAKER_00: or it might or might not succeed.
27:41
SPEAKER_00: But if you do, you're now hopelessly dependent on it.
27:44
And at some point when the context window fills up or when the
27:47
model is not having
27:48
SPEAKER_00: a good day because these things are non-deterministic,
27:50
SPEAKER_00: you're kind of stuck.
27:52
SPEAKER_00: You know,
27:53
your best thing is to try a new model or empty the context
27:56
window or I don't know,
27:57
SPEAKER_00: do something else.
27:58
SPEAKER_00: But it doesn't really feel like you're in control.
28:01
SPEAKER_01: Yes, yes, you're absolutely right.
28:03
And I think that whether your gateway into this new world was
28:08
Vibe Coding or you
28:09
SPEAKER_01: are a senior engineer and you've been evolving your workflow for AI,
28:14
there are a few things I think that everybody should keep in
28:16
mind that can help you
28:17
SPEAKER_01: get the best outcomes.
28:19
SPEAKER_00: Adi just mentioned getting the best outcomes.
28:22
SPEAKER_00: Here's what I'm seeing with AI and Vibe coding.
28:24
SPEAKER_00: These tools can give you incredible velocity.
28:26
SPEAKER_00: You can ship features a lot faster than before.
28:28
SPEAKER_00: But velocity without precision means you're just shipping more things,
28:32
SPEAKER_00: not necessarily the right things.
28:34
SPEAKER_00: How do you know if what you build actually works?
28:37
SPEAKER_00: Did that new checkout flow improve conversion or hurt it?
28:39
SPEAKER_00: Is a feature you shipped helping retention or causing a drop-off?
28:42
Without precision in how you roll out a measure,
28:45
you're making decisions blind.
28:46
SPEAKER_00: That's where Statsik comes in.
28:48
Statsik is our presenting partner for the season and they built
28:51
a precision toolkit
28:52
SPEAKER_00: that matches your AI accelerated velocity.
28:54
SPEAKER_00: Here's what precision looks like.
28:56
You ship a feature to 10% of users in a controlled environment
28:59
and see if it's
29:00
SPEAKER_00: moving your metrics up or down.
29:03
If conversion drops,
29:04
you catch it early and roll back before it affects everyone.
29:07
You're making precise data-driven decisions at the same pace
29:11
you're shipping code.
29:12
SPEAKER_00: Take Graphite as an example.
29:13
They build granular control and rapid iteration into their
29:16
development workflow using Statsig.
29:18
SPEAKER_00: When they roll out a new feature,
29:20
SPEAKER_00: built-in analytics show exactly how it affects their key metrics.
29:23
SPEAKER_00: They can see if engagement is up,
29:25
SPEAKER_00: if the feature is causing errors,
29:27
SPEAKER_00: or if users are dropping off.
29:28
SPEAKER_00: They do this with feature gates and metrics working together,
29:31
and they're running over 300 of these controls rollouts at any
29:34
given time.
29:34
During production incidents,
29:36
this approach cut their resolution times by over 50%,
29:39
because they could quickly identify which feature flag was
29:41
causing issues and fix it instantly.
29:43
SPEAKER_00: Most teams stitch together separate systems,
29:46
SPEAKER_00: wait on queries,
29:47
SPEAKER_00: and try to correlate user segments that don't match.
29:49
By the time they know if something worked,
29:51
they've already moved on to the next feature.
29:52
SPEAKER_00: With Statsig, you have everything in one place.
29:55
SPEAKER_00: Feature flags, experimentation, and analytics with the same user data.
29:58
SPEAKER_00: Statsig has a generous free tier to get started,
30:00
SPEAKER_00: and pro pricing for Teams starts at $150 per month.
30:03
To learn more and get a 30-day enterprise trial,
30:06
go to statsig.com slash pragmatic.
30:08
With this,
30:08
let's get back to the conversation about AI and development
30:11
workflows.
30:12
SPEAKER_01: Understand that models tend to have finite context windows.
30:15
SPEAKER_01: They've been getting larger over time.
30:17
You probably need to,
30:18
at some level,
30:19
adopt sort of a project manager mindset.
30:23
SPEAKER_01: Break tasks into small, verifiable chunks.
30:26
I've seen a lot of people who will throw the kitchen sink at
30:30
the LLM and say,
30:30
SPEAKER_01: hey,
30:31
SPEAKER_01: yes,
30:31
SPEAKER_01: build all these requirements at once.
30:33
SPEAKER_01: And that doesn't necessarily work the best.
30:36
SPEAKER_01: So,
30:36
SPEAKER_01: you know,
30:37
SPEAKER_01: small verifiable chunks,
30:39
SPEAKER_01: clear expectations and be prepared to iterate with the AI.
30:43
Don't just feel like one shotting is going to give you the best
30:47
outcome because it
30:48
SPEAKER_01: probably isn't.
30:49
And that decomposition is really pretty similar to planning a
30:52
sprint or writing
30:54
SPEAKER_01: pseudocode,
30:55
SPEAKER_01: the type of thing that we would do in the older days.
30:57
SPEAKER_01: And it reduces the risk of context loss and compounding errors.
31:01
I think that a lot of best practices in software engineering
31:04
remain timeless,
31:05
SPEAKER_01: even with AI coding.
31:07
SPEAKER_01: So caring about modular, testable code, enforcing those code reviews...
31:13
SPEAKER_01: You know,
31:13
SPEAKER_01: AI is going to introduce a few new habits,
31:16
like thinking about the input and output constraints and
31:19
seeding enough context.
31:20
SPEAKER_01: But that doesn't mean that you can't get good output from it.
31:23
It does mean that you're going to have to apply a level of
31:26
diligence with that
31:27
human in the loop to make sure that you are setting yourself up
31:31
for success.
31:32
SPEAKER_01: The more that you sort of give up your responsibilities to the LLM,
31:37
SPEAKER_01: the higher the risk is that something's going to go off the rails.
31:40
SPEAKER_00: Yeah,
31:40
SPEAKER_00: and one thing that I heard from an engineer,
31:44
SPEAKER_00: actually,
31:44
SPEAKER_00: it was Armin Ronatra,
31:45
SPEAKER_00: who's a longtime century engineer,
31:49
SPEAKER_00: the creator of Flask and a bunch of open source frameworks.
31:52
And he was telling me that he's observed something interesting
31:55
with engineers who
31:56
SPEAKER_00: are firmly in control and very confident in their work.
32:00
SPEAKER_00: He said that he sees they're just seeing a lot more success with AI.
32:04
And the people who feel that they don't have that much control
32:07
over their work
32:08
SPEAKER_00: over their tasks, they're just assigned tickets.
32:10
SPEAKER_00: And they also have a bit of a, like, you know, the world's control.
32:13
I mean,
32:14
mine said they're freaking out a lot more about AI and what it
32:17
will do.
32:17
SPEAKER_00: And this was so interesting of,
32:20
SPEAKER_00: It just came to me as we're talking about it,
32:22
SPEAKER_00: because what I'm sensing from a conversation is,
32:25
SPEAKER_00: again,
32:25
SPEAKER_00: a lot of your advice boils down to be in charge,
32:28
SPEAKER_00: understand everything,
32:29
SPEAKER_00: be confident,
32:30
SPEAKER_00: you know,
32:30
SPEAKER_00: that if this thing is taken away,
32:32
SPEAKER_00: you can make progress,
32:33
SPEAKER_00: like no problem,
32:34
SPEAKER_00: it'll just be a bit slower.
32:35
SPEAKER_00: And as long as you have this mindset, it feels like everything is easier.
32:39
SPEAKER_00: And keeping this, as you said, you still read through the thinking.
32:43
SPEAKER_00: You prefer the models where it explains to you.
32:45
SPEAKER_00: And then you can go back if you wish.
32:47
SPEAKER_00: And you often do it.
32:49
SPEAKER_00: You read through to make sure to learn.
32:51
SPEAKER_00: Plus, it's kind of fun.
32:52
SPEAKER_00: You keep learning professionally.
32:53
SPEAKER_00: You keep getting better every day.
32:55
SPEAKER_01: Absolutely.
32:55
SPEAKER_01: Absolutely right.
32:57
SPEAKER_01: I'm reminded of, you know, AI is just another tool in your tool belt.
33:03
SPEAKER_01: You know,
33:04
we've gone through so many different moments over time of
33:08
engineering and developer
33:09
SPEAKER_01: experience getting a little bit better with every generation.
33:14
You know,
33:14
when I was coming up,
33:16
I remembered getting excited when templates became a thing.
33:20
SPEAKER_01: You could generate code?
33:22
SPEAKER_01: What templates do you mean?
33:23
SPEAKER_01: Oh, no, just downloading templates for ideas, for UI, for starting off.
33:29
SPEAKER_00: Oh, on the web, right?
33:30
SPEAKER_01: Yeah, on the web.
33:31
SPEAKER_01: And this was just the simplest of things.
33:34
SPEAKER_01: It was like a zip file, right?
33:35
SPEAKER_01: Somebody else had created it.
33:36
SPEAKER_01: Okay, so I've improved my starting point.
33:39
And then we got stronger command line interfaces,
33:42
scaffolding,
33:43
that type of thing.
33:44
SPEAKER_01: And then you didn't have to worry about the exact starting point.
33:48
SPEAKER_01: And I feel like this is just taking us a few steps forward,
33:51
like it's making it easier for us to bootstrap a solution that
33:56
kind of works,
33:57
but still requires you to really understand what you are
34:01
putting into your code
34:01
SPEAKER_01: base shipping out to users and not,
34:04
SPEAKER_01: you know,
34:05
SPEAKER_01: getting rid of your responsibilities at the end of the day.
34:08
SPEAKER_01: I find that,
34:10
SPEAKER_01: at least for me,
34:11
SPEAKER_01: in the last year or two,
34:13
SPEAKER_01: going back to first principles,
34:15
SPEAKER_01: as always,
34:16
SPEAKER_01: has helped a lot.
34:17
I've been working a little bit more closely with the Google
34:21
DeepMind team on Gemini
34:24
SPEAKER_01: and its coding capabilities.
34:26
And that has been a really good reminder for me of just
34:29
understanding how a lot of
34:30
SPEAKER_01: these things work behind the scenes.
34:32
SPEAKER_01: If you remember that, you know, what is training data?
34:36
SPEAKER_01: Well,
34:36
it's very likely going to be looking at permissively licensed
34:40
code that's on GitHub
34:42
SPEAKER_01: or out on the open web.
34:45
SPEAKER_01: The patterns in that code are probably going to be reflective of,
34:48
SPEAKER_01: in many cases,
34:49
SPEAKER_01: like lowest common denominator.
34:51
um things that maybe just work are they going to be the most
34:55
secure the most high
34:56
performance the most accessible possibly not and so if you
35:01
remember that the
35:01
training data itself um still requires a lot of work to like
35:05
actually get things to
35:06
a place where you know it maybe is of production quality you
35:10
sort of start to set
35:11
your expectations a little bit lower um when working with llms
35:15
um you realize like
35:16
SPEAKER_01: Yeah,
35:17
it will probably do a better job than me just copying and
35:20
pasting snippets of code
35:21
SPEAKER_01: from somewhere else,
35:23
SPEAKER_01: obviously.
35:24
But I still very likely need to do a level of manual work or a
35:29
level of diligence
35:30
SPEAKER_01: on top of this to get the best outputs.
35:32
SPEAKER_00: Yeah,
35:32
SPEAKER_00: it reminds me a little bit of 10 years ago or so,
35:36
SPEAKER_00: five,
35:36
SPEAKER_00: 10 years ago,
35:37
SPEAKER_00: we use a lot of Stack Overflow because when you search for something,
35:40
SPEAKER_00: it was there.
35:41
SPEAKER_00: And so,
35:41
SPEAKER_00: for example,
35:42
SPEAKER_00: for email validation,
35:43
SPEAKER_00: like you would search,
35:44
SPEAKER_00: like how do I validate an email address?
35:46
SPEAKER_00: There was a Stack Overflow question and it had like 20 answers.
35:48
SPEAKER_00: One of them was top rated.
35:49
SPEAKER_00: It was a regular expression.
35:51
SPEAKER_00: And what I did,
35:53
SPEAKER_00: and I think what a lot of people did,
35:54
is I just copied and pasted because I cannot be bothered to
35:57
define exactly the
35:58
SPEAKER_00: regular expression of emails,
35:59
SPEAKER_00: which is a lot more complicated than just things.
36:02
So I just kind of...
36:02
And if it was for nothing important,
36:05
it kind of worked.
36:06
SPEAKER_00: But there were then complaints and flags raised that...
36:11
SPEAKER_00: A few of them,
36:12
SPEAKER_00: not this specific example,
36:13
SPEAKER_00: but they were insecure or they didn't account for edge cases.
36:16
SPEAKER_00: And a lot of devs,
36:18
SPEAKER_00: again,
36:18
when you were in the mindless mode or it didn't matter or you
36:20
know better,
36:21
SPEAKER_00: you did that.
36:22
SPEAKER_00: And I guess we're going to have the same thing at larger scale.
36:25
SPEAKER_00: Like, why did the bug happen?
36:27
SPEAKER_00: And you're going to look back at the history.
36:29
SPEAKER_00: Yeah,
36:30
someone just looks good to me for that big pull request that
36:34
was in the end
36:34
SPEAKER_00: generated by an AI.
36:35
SPEAKER_00: Yeah.
36:36
SPEAKER_01: I think that this is very much a smaller point,
36:39
SPEAKER_01: a side discussion point,
36:41
SPEAKER_01: but I've seen a number of cases where people will now say,
36:45
SPEAKER_01: hey,
36:46
do I still need to use third-party libraries if I can just
36:50
prompt a very slightly
36:52
SPEAKER_01: smaller version of a solution myself?
36:55
Which reminded me exactly of this Stack Overflow copy-pasting
36:59
the top response thing.
37:00
SPEAKER_01: And if you're a senior engineer, what you realize is,
37:03
SPEAKER_01: Well, hold on.
37:04
That also means that you are now taking on the responsibility
37:07
of making sure that
37:08
SPEAKER_01: this is future proof for security issues,
37:11
SPEAKER_01: for all sorts of platform issues that could happen.
37:14
SPEAKER_01: If you're relying on a library,
37:15
it's perhaps easier for there to be a central point of leverage
37:18
where those fixes
37:18
SPEAKER_01: can be made and then deployed out to people.
37:21
SPEAKER_01: If you own all of these different patterns in your code base yourself,
37:25
SPEAKER_01: that also means that you're taking on that responsibility.
37:28
SPEAKER_01: Can be totally okay,
37:29
but it's something that I sometimes see people just not being
37:31
mindful of requiring
37:32
SPEAKER_01: a little bit of extra work.
37:33
And I feel this is where we just need to remind ourselves that
37:36
it's just engineering,
37:37
right?
37:37
SPEAKER_00: It's trade-off.
37:38
Do you take on the responsibility and the risk of maintaining
37:42
this thing and the
37:43
SPEAKER_00: fact that it might have missing edge cases or whatever,
37:45
SPEAKER_00: or might not work correctly or securely?
37:48
SPEAKER_00: Or do you take on a dependency, which has its own problems, right?
37:51
SPEAKER_00: There's now dependency, security, et cetera.
37:54
SPEAKER_00: But speaking of the need for software engineering...
37:58
SPEAKER_00: What are some new workflows,
38:00
some things that we have not been able to do before as software
38:04
engineers that we
38:05
SPEAKER_00: can now experiment with,
38:06
SPEAKER_00: that you might be trying out with these AI tools that is just like,
38:11
SPEAKER_00: is brand new?
38:11
SPEAKER_01: Yeah,
38:12
SPEAKER_01: I think for me,
38:13
the thing that I'm most excited about seeing evolve is this
38:18
idea of asynchronous
38:20
SPEAKER_01: background coding agents.
38:22
A number of teams playing around with these ideas at the
38:24
moment,
38:25
Jules,
38:26
Codex,
38:28
SPEAKER_01: We also see GitHub experimenting with some of these ideas as well.
38:33
I think that the idea of you being able to delegate parts of
38:39
your backlog and have
38:41
SPEAKER_01: a system be able to
38:43
SPEAKER_01: implement that asynchronously is a very interesting idea.
38:48
If we can do that without merge conflicts in a way that is
38:54
easily human verifiable,
38:56
SPEAKER_01: so again,
38:56
SPEAKER_01: going back to that human in the loop,
38:58
SPEAKER_01: I think that's very interesting.
38:59
SPEAKER_01: I have been finding that that is very interesting
39:03
very much already in a place where if you are having one agent
39:08
work on writing or
39:10
updating your tests or you have a number of agents working
39:13
together on trying to
39:15
SPEAKER_01: migrate your code base from one version of a library to another or
39:20
SPEAKER_01: you know,
39:20
SPEAKER_01: a version of a dependency to another,
39:23
SPEAKER_01: that they're pretty okay at doing that kind of work right now.
39:26
SPEAKER_01: Smaller changes,
39:27
SPEAKER_01: like there are lots of things that,
39:28
SPEAKER_01: you know,
39:29
SPEAKER_01: we all have to do,
39:30
SPEAKER_01: like adding dark mode,
39:31
SPEAKER_01: for example,
39:32
SPEAKER_01: like smaller kinds of changes,
39:33
where maybe being able to delegate those kinds of changes to
39:37
asynchronous agents
39:38
SPEAKER_01: are going to,
39:39
SPEAKER_01: you know,
39:39
SPEAKER_01: get very good.
39:41
SPEAKER_01: I'm excited about that.
39:42
I think that the jury is still out on exactly what is the
39:46
surface for being able
39:47
SPEAKER_01: to,
39:48
SPEAKER_01: if you are the conductor of this orchestra,
39:51
what is the right surface for you being able to manage all of
39:54
these things?
39:54
And what is a realistic number of tasks for you to be managing
39:59
at the same time?
40:00
SPEAKER_01: Because,
40:01
SPEAKER_01: you know,
40:01
SPEAKER_01: even I like a number of people have shown off like,
40:04
SPEAKER_01: hey,
40:04
I've got like 20 different terminals open and I have quad code
40:09
running in half of
40:09
SPEAKER_01: them and then Gemini running.
40:10
SPEAKER_01: And it's it looks it's funny.
40:13
SPEAKER_01: But the reality is that you only have a finite amount of attention.
40:18
And if you are going to be actually putting in diligence into
40:22
your code reviews and
40:23
SPEAKER_01: into each of these workflows,
40:25
you are probably only going to be able to do a couple of things
40:28
at the same time.
40:29
SPEAKER_01: It all comes back to multitasking best practices.
40:34
SPEAKER_01: But I'm interested in that.
40:35
SPEAKER_01: I'm interested in seeing where that goes.
40:37
SPEAKER_01: Another thing that's very interesting is vibe designing.
40:40
I'm seeing that part of product engineering,
40:45
product development evolving a little bit.
40:48
It was very exciting to see Figma's MCP moving a little bit
40:53
closer in this direction.
40:55
Things that allow designers and developers to work more closely
40:59
together or to at
41:00
SPEAKER_01: least enable designers to be able to take their vision and turn it into a
41:04
SPEAKER_01: functional prototype,
41:05
something perhaps a little bit closer to code that can then be
41:09
productionized and
41:10
SPEAKER_01: is not just a one-off demo.
41:12
SPEAKER_00: Yeah,
41:13
I heard an engineering lead or design leader at Shopify said
41:17
that her team,
41:19
SPEAKER_00: every designer on her team,
41:20
SPEAKER_00: so not all of Shopify,
41:22
SPEAKER_00: has cursor.
41:23
And what they do is they create a Figma design and then they
41:26
ask cursor to implement it.
41:27
SPEAKER_00: And then they show it to engineers.
41:29
SPEAKER_00: And this is not saying ship it.
41:31
SPEAKER_00: It's just more,
41:32
we now have something interactive that we can work with as
41:34
opposed to just what
41:35
SPEAKER_00: they used to do,
41:36
SPEAKER_00: which is sharing a Figma design.
41:37
SPEAKER_00: And I was like, huh.
41:39
SPEAKER_00: This is something I've not heard before,
41:41
SPEAKER_00: like,
41:42
SPEAKER_00: truly,
41:42
SPEAKER_00: like,
41:43
SPEAKER_00: like all designers or at least on a team using the developer tool,
41:46
SPEAKER_00: like I couldn't imagine designers using Visual Studio code for,
41:50
SPEAKER_00: I mean,
41:50
SPEAKER_00: I can imagine them being able to open it,
41:52
SPEAKER_00: but I couldn't imagine them.
41:53
SPEAKER_00: It's just not built for them.
41:54
SPEAKER_00: So this was really, really interesting to hear.
41:57
SPEAKER_00: And again, this wasn't some vendor advertisement or anything.
42:00
SPEAKER_01: Yeah, shout out to the Shopify team.
42:03
SPEAKER_01: Honestly,
42:03
I feel like a number of them have been very open to sharing
42:07
what's been working out
42:08
SPEAKER_01: well for their teams.
42:09
SPEAKER_01: And I've enjoyed following their story.
42:12
I think that more of these patterns spreading is going to
42:15
depend a little bit on
42:17
SPEAKER_01: training,
42:18
SPEAKER_01: on governance,
42:18
SPEAKER_01: on clear boundaries between what is prototype code and production.
42:22
But already being able to turn something static into a
42:26
semi-functional prototype is very cool.
42:29
SPEAKER_01: Are we going to see, like, everybody using cursor?
42:31
SPEAKER_01: I don't know.
42:32
SPEAKER_01: I think that I,
42:35
SPEAKER_01: at least for me,
42:36
I see folks being able to accomplish the same outcomes from the
42:39
tools that they
42:40
spend the most time in or the bridges between tools getting
42:43
better over time.
42:44
SPEAKER_01: But I still think it's very, very exciting.
42:46
SPEAKER_01: In the same way,
42:47
SPEAKER_01: you know,
42:48
we're also seeing lots of good conversations about PM and EM
42:52
roles changing.
42:54
SPEAKER_01: PMs maybe, you know, spending...
42:57
SPEAKER_01: more time on problem framing and metrics and policies for agents,
43:00
EM spending more time on evals and safety reviews and really
43:04
enabling their teams
43:05
SPEAKER_01: to work with AI confidently.
43:07
SPEAKER_01: It won't change accountability for outcomes,
43:11
but I am seeing a lot of good discussions about the need for
43:14
taste in product
43:15
SPEAKER_01: engineering,
43:16
SPEAKER_01: where that's going to be what differentiates people.
43:19
SPEAKER_01: Because
43:19
If anybody can look at what you've built and can use prompts to
43:23
accomplish a
43:24
SPEAKER_01: similar set of functionality,
43:26
the taste piece is going to continue to be very interesting for
43:30
folks to focus on
43:31
SPEAKER_01: in the future.
43:32
SPEAKER_01: You and I have talked about juniors and seniors.
43:36
I think that there's going to be interesting times for new
43:39
grads versus seniors.
43:42
SPEAKER_01: AI definitely raises the floor, but it also raises the ceiling too.
43:47
SPEAKER_01: And juniors are going to be able to get moving faster,
43:52
SPEAKER_01: but seniors who can write specs,
43:55
SPEAKER_01: decompose work,
43:56
SPEAKER_01: understand system architecture,
43:58
SPEAKER_01: review effectively,
43:59
SPEAKER_01: I think that they're going to become even more valuable.
44:02
SPEAKER_01: And that last 30% that we talked about, I think it's leverage.
44:06
SPEAKER_01: Not just busy work, but actually gets leverage.
44:09
SPEAKER_01: And,
44:10
SPEAKER_01: you know,
44:11
a lot of surveys have been showing that trust is in a cautious
44:17
but optimistic place
44:19
SPEAKER_01: at the moment.
44:20
But that cautious piece speaks to the need for that human
44:24
oversight remaining pretty central.
44:26
SPEAKER_00: Yeah,
44:26
SPEAKER_00: and if I think about...
44:28
SPEAKER_00: I was just thinking that when it comes to this idea of parallel agent,
44:32
SPEAKER_00: I don't think it's ever been done in the sense that as a developer,
44:35
SPEAKER_00: you were never able to do this.
44:36
SPEAKER_00: I mean, we couldn't even kick off and talk with natural language to...
44:41
SPEAKER_00: machine that would spit out code that actually compiles.
44:43
I think this is new as well,
44:44
but the fact that you can do it with multiple.
44:46
SPEAKER_00: It's not happened before,
44:48
SPEAKER_00: and programming is a very,
44:50
SPEAKER_00: you're in the flow,
44:51
SPEAKER_00: you work on one problem.
44:52
SPEAKER_00: When you stop working on it,
44:53
SPEAKER_00: you switch over,
44:55
SPEAKER_00: you get your context almost like the stack clears,
44:58
SPEAKER_00: you load the new thing.
44:58
SPEAKER_00: It kind of feels like it, right?
45:00
SPEAKER_00: But on the other,
45:01
when I think back to who are the best senior engineers I know
45:05
and what their days
45:06
SPEAKER_00: look like,
45:06
SPEAKER_00: well,
45:06
SPEAKER_00: they're on a team with a few mid-level,
45:09
SPEAKER_00: maybe some interns or new grads,
45:11
SPEAKER_00: and they're working on their stuff.
45:13
And then they're the ones who get the slack pink saying,
45:16
hey,
45:16
could you please unblock me?
45:18
SPEAKER_00: So they context switch, review the code, you know, like criticize it.
45:23
SPEAKER_00: or they block out time and they review, review, review.
45:25
SPEAKER_00: And like for every kind of senior slash,
45:27
SPEAKER_00: often these are tech leads,
45:29
SPEAKER_00: they have like a few people who are,
45:30
SPEAKER_00: you know,
45:31
SPEAKER_00: they're not agents,
45:31
SPEAKER_00: they're people,
45:32
but they just review their work and they kind of orchestrate
45:35
them a little bit on
45:36
SPEAKER_00: the standup,
45:36
SPEAKER_00: you know,
45:37
SPEAKER_00: they're in the planning meeting,
45:38
SPEAKER_00: they nudge them,
45:39
SPEAKER_00: they mentor them.
45:41
SPEAKER_00: So in some sense,
45:42
SPEAKER_00: I feel like seniors already do this,
45:45
SPEAKER_00: and if I had a magic wand saying,
45:47
SPEAKER_00: like,
45:47
who would I expect in a few years' time to be able to manage
45:51
multiple agents?
45:52
SPEAKER_00: Well, the senior, for sure.
45:53
SPEAKER_00: Like,
45:54
SPEAKER_00: new grads,
45:54
SPEAKER_00: probably,
45:55
SPEAKER_00: like,
45:56
SPEAKER_00: I mean,
45:56
SPEAKER_00: I feel there will be a stretch for them,
45:57
SPEAKER_00: but they're not expected to have that,
45:59
SPEAKER_00: and they don't have the expertise.
46:00
So I wonder if some of these skills are somewhat transferable
46:03
in a sense that,
46:04
SPEAKER_00: and you know,
46:05
SPEAKER_00: that senior,
46:06
SPEAKER_00: why could that senior do that?
46:07
SPEAKER_00: Well, because they understood the code base.
46:09
SPEAKER_00: They knew what good looked like.
46:10
SPEAKER_00: They were always thorough on their reviews.
46:12
SPEAKER_00: They never let things slip.
46:13
SPEAKER_00: They call out even the smallest thing.
46:15
SPEAKER_01: I completely agree with you.
46:17
And I think that there's a lot to be said about how developer
46:22
education in teams
46:24
SPEAKER_01: may also evolve for this moment.
46:27
Historically,
46:28
I remember when I was coming up,
46:29
you know,
46:30
mentorship was always a big topic.
46:32
SPEAKER_01: And we talked about,
46:33
SPEAKER_01: you know,
46:34
SPEAKER_01: especially for folks who are joining a new team,
46:37
SPEAKER_01: the importance of like pair programming.
46:40
SPEAKER_01: I think we're going to see, you know, perhaps even situations of trio.
46:44
SPEAKER_01: programming where it's a junior senior and the AI.
46:48
And maybe the senior is going to be there asking you to explain
46:53
the code that the
46:54
AI is generating in some way or walking you through how that
46:58
code perhaps connects
46:59
SPEAKER_01: to other parts of the system.
47:00
SPEAKER_01: And really,
47:00
SPEAKER_01: again,
47:01
using it as an additional tool in their arsenal to be able to
47:04
help build up
47:06
SPEAKER_01: confidence and awareness of how the full app actually works.
47:10
We're also seeing some interesting discussion about potentially
47:14
new roles or
47:15
SPEAKER_01: refinements of roles,
47:16
SPEAKER_01: things like forward deployed engineers.
47:19
SPEAKER_01: I'm seeing interest in developers who are a little bit more embedded with
47:24
SPEAKER_01: customers,
47:25
SPEAKER_01: who can build features rapidly with AI while feeding back requirements.
47:30
SPEAKER_01: And that type of thing might blur the lines between the developer,
47:33
SPEAKER_01: PM,
47:34
SPEAKER_01: and designer roles a little bit.
47:36
SPEAKER_01: I'm interested to see where that may end up going.
47:40
SPEAKER_01: And I'm also interested in how,
47:42
SPEAKER_01: you know,
47:42
SPEAKER_01: just generally speaking,
47:43
SPEAKER_01: AI engineering is going to evolve how,
47:46
SPEAKER_01: you know,
47:47
SPEAKER_01: we approach education,
47:48
SPEAKER_01: whether you're in high school or you're in college.
47:50
Like,
47:51
are we going to be teaching people prompt and context
47:54
engineering best practices?
47:56
SPEAKER_01: What does that all actually look like?
47:57
SPEAKER_01: How do we
47:58
continue to enable people to think with systems design and
48:02
engineering in mind.
48:03
SPEAKER_01: But I'm very excited where the education side of this goes.
48:06
So one area that we've talked about was code reviews and how
48:09
it's really important.
48:10
SPEAKER_00: It's a bottleneck and we should review the code.
48:13
One thing that I'm kind of noticing on myself already is when I
48:17
use these AI tools,
48:19
SPEAKER_00: may that be cloud code or agents or even just the autocomplete,
48:24
I have a tendency to do tap-tap or accept or just like accept
48:28
all the things,
48:29
SPEAKER_00: especially when I'm working on something that,
48:31
SPEAKER_00: I mean,
48:32
SPEAKER_00: it's not the most critical thing in the world.
48:34
SPEAKER_00: And it's kind of, it's easy.
48:35
And especially after I started to trust that it gets it right
48:38
most of the time,
48:38
SPEAKER_00: in the end,
48:39
SPEAKER_00: I know I'm going to review.
48:41
SPEAKER_00: But I find myself losing a little bit of...
48:44
SPEAKER_00: criticality.
48:45
SPEAKER_00: I'm not as critical,
48:46
SPEAKER_00: you know,
48:47
the second day and the third day and when I was the first time
48:51
when I didn't trust
48:51
SPEAKER_00: this thing.
48:52
SPEAKER_00: And I worry a little bit that with code reviews, the same could happen.
48:54
SPEAKER_00: You know, LGTM looks good to me.
48:56
SPEAKER_00: I mean, I understand that Google has it built in as a
49:01
as a feature in your code review tool,
49:03
which is a really fun tool,
49:04
as I understand.
49:05
SPEAKER_00: But there is this risk.
49:07
There's always been this risk,
49:08
of course,
49:08
that you just kind of like,
49:10
it's a lot of work.
49:11
SPEAKER_00: It kind of looks good, and you're not looking critically.
49:13
SPEAKER_00: How do you think, especially on proper teams,
49:19
SPEAKER_00: we could battle or we could do this kind of fighting against like,
49:23
SPEAKER_00: yes,
49:23
SPEAKER_00: giving it a proper review,
49:25
SPEAKER_00: especially if we haven't written the code that much.
49:27
SPEAKER_00: Because I feel with writing the code, you have like two reviews.
49:30
SPEAKER_00: One is you're writing your own code,
49:31
SPEAKER_00: you're typing it out,
49:33
SPEAKER_00: which we're not doing as much anymore.
49:34
And then someone else gives a review knowing that it was you
49:37
who typed it out.
49:38
SPEAKER_01: I mean, it's always more fun to write code than to read and review it.
49:43
SPEAKER_00: Yeah, a little bit, right?
49:45
Yeah,
49:45
and I think more and more of our job is going to become about
49:50
reading and reviewing code.
49:51
SPEAKER_01: There are a few ideas that I've tried to suggest to folks.
49:55
SPEAKER_01: Things like for certain features or certain days of the week,
50:01
SPEAKER_01: maybe you intentionally try to not write
50:05
lean on using ai or an llm and just try to see like okay well
50:10
can i can i still
50:11
solve some of these problems myself um to preserve your
50:15
critical thinking skills
50:16
and force you to think about okay well let's say that all of
50:20
the top llm providers
50:21
were down for the day uh what would you do being cheeky you
50:25
know i would probably
50:26
say like yeah i'm just i'm just gonna go you know use olama and
50:29
a local model and
50:30
SPEAKER_01: i'll be totally fine i will find some fallback
50:33
But the reality is that I do think that it's going to continue
50:36
to be very important
50:36
SPEAKER_01: for us to be able to think through how things work,
50:40
SPEAKER_01: be able to problem solve without necessarily relying on the AI.
50:45
SPEAKER_01: The models are going to continue getting better.
50:47
Their ability to take in sufficient context from your code base
50:51
is going to
50:51
SPEAKER_01: continue getting better.
50:52
SPEAKER_01: But it is going to take quite some time,
50:55
SPEAKER_01: in my opinion,
50:56
SPEAKER_01: before you're going to be able to fully trust that in every situation,
51:01
SPEAKER_01: whatever requirements you throw at it,
51:02
SPEAKER_01: it's going to be able to get it right.
51:04
SPEAKER_01: And if you get stuck,
51:05
SPEAKER_01: if you're trying things out and you've tried it five,
51:07
SPEAKER_01: ten times and it still hasn't solved the problem,
51:10
SPEAKER_01: you're going to have to solve the problem yourself.
51:12
So I think that being able to force yourself into situations
51:15
where you're testing
51:16
SPEAKER_01: and retesting your critical thinking skills are going to be important.
51:19
I also think that there's a little bit of value in doing some
51:22
game theory around this.
51:24
Teams or individual people are probably going to start leaning
51:28
on AI to do more of
51:30
the code reviews to keep up with the pace and velocity of
51:34
change coming in.
51:36
SPEAKER_01: What are those workflows going to look like?
51:38
SPEAKER_01: If you have an agent that's saying like,
51:40
SPEAKER_01: Yeah, I've reviewed this PR and it looks good to merge in.
51:43
Are you actually going to trust it or are you going to go and
51:46
do a human level
51:47
SPEAKER_01: review,
51:48
SPEAKER_01: even if it is maybe more shallow than you historically would have done?
51:52
Or maybe you're OK with spending 10,
51:55
20 minutes or something on the review.
51:57
SPEAKER_01: What does that work will look like?
51:59
SPEAKER_01: In my case,
51:59
SPEAKER_01: even if an agent,
52:01
SPEAKER_01: an agent telling me that something looks good to merge is a signal.
52:06
SPEAKER_01: It's similar to a signal of like,
52:07
SPEAKER_01: OK,
52:07
SPEAKER_01: well,
52:08
SPEAKER_01: maybe a junior person on my team has also,
52:10
SPEAKER_01: you know,
52:11
SPEAKER_01: given this an LGTM.
52:12
SPEAKER_01: I'm still going to probably go back if it's critical enough.
52:15
SPEAKER_00: Or the CICD passes all the tests, right?
52:17
SPEAKER_00: It's green.
52:18
SPEAKER_00: All tests pass.
52:19
SPEAKER_01: Exactly, exactly.
52:22
SPEAKER_01: It's a signal.
52:23
SPEAKER_01: Signals are useful,
52:24
but you still want to apply your lens on quality to it and take
52:30
a look through and
52:30
SPEAKER_01: just make sure that you're confident.
52:32
Things like having those tests,
52:34
having whatever quality gates that you and your team
52:38
discussed like having those in place all of these are good
52:41
signals for building up
52:42
confidence so the more signals that you have can build up to
52:44
build up confidence
52:45
that things aren't going to go off the rails when you merge in
52:48
i think that's good
52:49
but um i try to you know be very intentional with making sure
52:53
that i'm not leaning
52:54
on ai for absolutely everything i may use it for many aspects
52:58
of my life on a
52:58
SPEAKER_01: day-to-day including ai coding
53:00
SPEAKER_01: But I still try to make sure that I'm,
53:02
SPEAKER_01: you know,
53:03
whether it's 20 or 30 percent or whatever amount of my tasks
53:07
don't necessarily
53:07
SPEAKER_01: require AI.
53:08
I still try to make sure that I'm using my brain to solve those
53:12
problems myself.
53:13
SPEAKER_01: And I think that intentionality,
53:14
SPEAKER_01: like just be proactive with maintaining your critical thinking skills,
53:17
SPEAKER_01: I think that's going to help people.
53:18
Yeah,
53:18
and one of these things of like,
53:21
you're still hands on two things came to my mind.
53:23
SPEAKER_00: One was a cloud code actually recently shipped.
53:25
SPEAKER_00: You can change modes.
53:27
SPEAKER_00: You can have exponential learning mode where it explains things.
53:30
SPEAKER_00: And you have a thing called learning mode,
53:32
SPEAKER_00: which is it pauses and says,
53:33
SPEAKER_00: you do this part,
53:35
SPEAKER_00: which I thought was really clever.
53:35
SPEAKER_00: I actually turned it on for something that I was building.
53:38
It didn't work as I expected because it gave me a really weird
53:41
task to do.
53:42
SPEAKER_00: But I see the potential and I actually...
53:45
SPEAKER_00: I actually intended to lean a bit more on that.
53:47
And I think it's,
53:48
I hope other tools will also do this as part of the developer.
53:51
SPEAKER_00: Like I feel, I do want to know that I can do it.
53:55
SPEAKER_00: I can do it, but how am I going to know if I'm not doing it?
53:58
SPEAKER_00: Of course, you feel around a little bit with it.
54:00
SPEAKER_01: I think that is such a fantastic idea.
54:02
And if somebody is a junior or you are coming into a code base
54:07
that you're not
54:07
SPEAKER_01: familiar with,
54:08
fight that feeling of the first thing I'm going to do is try to
54:13
provide value and
54:14
SPEAKER_01: I'm going to just prompt a new feature into existence.
54:18
SPEAKER_01: Maybe use the LLM to just explain how the code base works.
54:22
And just spend time like soaking yourself up in the richness of
54:25
like,
54:25
what is the code base?
54:26
SPEAKER_01: How does it work?
54:27
SPEAKER_01: How does everything connect together?
54:28
SPEAKER_01: Before you start prompting things into existence.
54:31
SPEAKER_01: I think that using it as a learning tool is a very powerful thing.
54:34
SPEAKER_01: We need more of it.
54:35
And this is also something,
54:37
what you said,
54:39
new joiners using a learning tool.
54:40
I'm hearing from a few companies that they're seeing new
54:44
joiners onboard faster.
54:46
SPEAKER_00: The ones where they can use these AI tools to explain stuff to them.
54:52
SPEAKER_00: outlier the club base is just a kind of a trusted person.
54:55
SPEAKER_00: 24-7,
54:56
SPEAKER_00: you can ask questions on like a senior engineer who,
54:59
SPEAKER_00: I mean,
54:59
SPEAKER_00: you can ask questions anytime,
55:00
SPEAKER_00: but obviously you don't want to do it like,
55:02
SPEAKER_00: you know,
55:03
SPEAKER_00: eight hours of the eight-hour working day.
55:04
SPEAKER_00: Like, you're going to keep your limits.
55:07
And it's interesting how that is changing the dynamics at a
55:10
bunch of workplaces
55:11
SPEAKER_00: that do have these tools.
55:12
SPEAKER_01: Absolutely.
55:14
I think that my hope is that it's going to become a much more
55:18
standard thing,
55:20
SPEAKER_01: treating AI as a learning tool.
55:21
SPEAKER_01: And it's not just for understanding a new code base.
55:27
It can be very useful for understanding programming concepts or
55:30
frameworks or
55:31
SPEAKER_01: architectural patterns.
55:33
There are times when I want to be able to bring a feature over
55:36
from one code base
55:38
that is very different or written in a different programming
55:40
language over to
55:41
SPEAKER_01: another.
55:42
And I have found AI very much indispensable for those
55:46
situations of helping me
55:47
understand one code base and the path to being able to bring
55:51
over something to a
55:52
SPEAKER_01: new one.
55:53
SPEAKER_01: And so encouraging your team to experiment with AI tools,
55:56
SPEAKER_01: sharing their best practices,
55:58
making it a regular thing that people know it's okay to use it
56:01
as a learning tool,
56:02
SPEAKER_01: I think is just going to be great for you as leads and managers.
56:07
SPEAKER_00: And you're also a manager of a team.
56:10
SPEAKER_00: And as part of this,
56:11
SPEAKER_00: obviously,
56:12
SPEAKER_00: you do performance reviews,
56:13
SPEAKER_00: promotions,
56:14
SPEAKER_00: help people basically professionally get better.
56:17
How do you think the kind of your definition or what you're
56:21
seeing in our team,
56:21
a definition of a really standout solid software engineer has
56:24
changed in the last
56:25
SPEAKER_00: three to four years since these tools have come out?
56:28
SPEAKER_00: What is new and what is the same?
56:30
SPEAKER_01: I think that what is new is the importance...
56:35
One of the things that I have found has always held true is the
56:38
importance of being
56:39
SPEAKER_01: a lifelong learner.
56:41
SPEAKER_01: That has always remained true.
56:42
It doesn't matter if frameworks come and go,
56:45
tools change,
56:47
the industry evolves.
56:48
SPEAKER_01: Being a lifelong learner and being very open to trying out new things,
56:54
SPEAKER_01: failing,
56:54
SPEAKER_01: building up those skills,
56:56
SPEAKER_01: I think that that remains very,
56:57
SPEAKER_01: very important.
56:58
The people on my team that I think were the most successful
57:01
early on at leveraging
57:03
AI for coding and for product engineering were the ones that
57:07
went in with this
57:08
SPEAKER_01: mindset of,
57:10
SPEAKER_01: I'm happy to learn new things to try out and to have this growth mindset.
57:15
SPEAKER_01: And if it doesn't work, that's fine.
57:17
SPEAKER_01: At least I've tried it out.
57:18
At least I understand the constraints and maybe I'll try out
57:21
different models for
57:22
SPEAKER_01: these different use cases in the future.
57:24
I think that that has been very much a consistent thing that's
57:27
been important.
57:29
SPEAKER_01: I feel like if you are a lead,
57:32
now is the time for you to be helping your team through this
57:35
moment in terms of
57:36
SPEAKER_01: showing that you're OK with learning as well.
57:40
SPEAKER_01: One of the things that I do every week actually is I spend a lot of time.
57:43
SPEAKER_01: I love reading.
57:43
SPEAKER_01: I spend a lot of time reading.
57:45
SPEAKER_01: I will, of course, read your newsletter a lot.
57:49
SPEAKER_01: I will read a number of different papers,
57:51
SPEAKER_01: white papers,
57:53
SPEAKER_01: blogs,
57:54
SPEAKER_01: watch videos,
57:54
SPEAKER_01: watch courses,
57:55
SPEAKER_01: you know,
57:56
SPEAKER_01: read announcements about what is new in AI,
57:58
SPEAKER_01: what is new in AI engineering during the week.
58:01
SPEAKER_01: And I will surface those things to my team in a newsletter.
58:05
I will also surface...
58:06
Oh,
58:06
you have like an internal newsletter for your team?
58:08
SPEAKER_01: Yeah, I got an internal newsletter.
58:09
SPEAKER_01: I write it every Monday.
58:11
SPEAKER_01: So I'll be I'll be writing it after after this.
58:13
SPEAKER_01: And I will include like, what am I hacking on?
58:16
SPEAKER_01: What am I writing?
58:16
SPEAKER_01: What am I thinking?
58:18
What are the things that I think are important for our team to
58:21
actually pay attention to?
58:23
And in this time where,
58:24
you know,
58:24
you see so many people struggling to stay up to date.
58:28
If you follow along on Twitter or other social networks,
58:31
it can feel very overwhelming.
58:34
SPEAKER_00: Yeah.
58:35
Because every few hours,
58:36
it feels like something has changed in a fundamental way.
58:39
And being able to sift through that and help people to just pay
58:43
attention to like
58:43
SPEAKER_01: what is actually important.
58:44
SPEAKER_01: I think that's a great thing for leadership to be doing right now,
58:47
SPEAKER_01: especially if you can guide folks towards,
58:49
SPEAKER_01: hey,
58:50
SPEAKER_01: maybe spend a little bit more time poking at this rather than that.
58:53
SPEAKER_00: I just want to say,
58:54
SPEAKER_00: I think it's not done because you're also staying closer to,
58:57
SPEAKER_00: you know,
58:58
SPEAKER_00: the industry,
58:59
SPEAKER_00: like to being technical.
59:00
SPEAKER_00: Maybe it's a stretch to say hands on,
59:02
SPEAKER_00: but you're pretty close to that just by keeping up with all of this.
59:06
SPEAKER_01: Yeah, yeah.
59:08
SPEAKER_01: And I've,
59:09
SPEAKER_01: you know,
59:10
SPEAKER_01: I'll defer to people on my team to say things,
59:14
SPEAKER_01: but I think that I've generally had good feedback about this.
59:18
SPEAKER_01: I've even had like other execs saying,
59:20
SPEAKER_01: hey,
59:21
SPEAKER_01: actually,
59:21
SPEAKER_01: I'm finding it really hard to stay on top of this.
59:23
SPEAKER_01: And your newsletter is helping me navigate this moment as well.
59:28
SPEAKER_01: And so I think that as a lead,
59:29
SPEAKER_01: being able to stay on top of what is happening at whatever level you are
59:34
SPEAKER_01: comfortable with or that your bandwidth allows,
59:36
I think that that can be a really powerful thing for your team
59:39
and will help you
59:40
SPEAKER_01: just keep your your own skills relatively sharp.
59:42
SPEAKER_01: You know, we've been talking a lot about AI assisted engineering.
59:45
SPEAKER_01: If your team do have to build out like AI features,
59:49
SPEAKER_01: a lot of this stuff is also going to continue to be relevant for them,
59:52
SPEAKER_01: too,
59:52
because you can help connect the dots between what is happening
59:55
in the industry
59:56
where AI is concerned that is actually important to their work
59:59
versus what is not.
1:00:01
And that's especially I think it's exceptionally important in
1:00:03
this moment where
1:00:04
SPEAKER_01: maybe there are things that they have not had to historically do.
1:00:07
SPEAKER_01: Think about like,
1:00:09
SPEAKER_01: oh,
1:00:09
SPEAKER_01: hey,
1:00:09
SPEAKER_01: well,
1:00:09
SPEAKER_01: Model X is really good at image generation,
1:00:13
SPEAKER_01: but Model Y is really good at generating sound.
1:00:16
SPEAKER_01: I'm just making things up.
1:00:17
But I think that it is really useful for you to be able to
1:00:21
maybe highlight some of
1:00:22
SPEAKER_01: these opportunities in a more concrete way.
1:00:24
SPEAKER_01: And then your team can go off and spend time digging into it themselves.
1:00:27
SPEAKER_01: But they're not starting from a blank slate.
1:00:29
SPEAKER_00: Plus, I guess you're just showing that, look, look, I'm learning.
1:00:34
SPEAKER_00: I'm spending time reading, understanding, digesting.
1:00:37
I'm doing some stuff on the side,
1:00:39
which kind of gives the permission that it's okay to do it.
1:00:41
SPEAKER_00: And if anything, I mean, you know, everyone for themselves.
1:00:44
SPEAKER_00: But I would assume that right now,
1:00:46
SPEAKER_00: because we are having a big change in this technology,
1:00:50
SPEAKER_00: it's a new tool,
1:00:51
SPEAKER_00: a lot of capabilities.
1:00:53
SPEAKER_00: it should be okay at most companies and most teams,
1:00:56
unless you're like has done crunch time shipping something like
1:00:59
on Friday for
1:01:01
SPEAKER_00: engineers to spend some time experimenting,
1:01:03
SPEAKER_00: figuring out,
1:01:04
SPEAKER_00: hey,
1:01:04
SPEAKER_00: can I use this thing?
1:01:06
SPEAKER_00: Will it work for me?
1:01:07
SPEAKER_00: When I talk with, again, the folks at Shopify, they do a lot of this.
1:01:10
SPEAKER_00: And in the end,
1:01:11
SPEAKER_00: actually,
1:01:12
SPEAKER_00: what some people think might happen is like,
1:01:14
SPEAKER_00: well,
1:01:14
SPEAKER_00: people do less work.
1:01:15
SPEAKER_00: Actually, what happens is
1:01:16
They do the same work as they did before,
1:01:18
but they're way more motivated.
1:01:19
SPEAKER_00: They tried a bunch of new stuff.
1:01:21
SPEAKER_00: Some of it doesn't work,
1:01:22
SPEAKER_00: and it's kind of a...
1:01:23
SPEAKER_00: You would think it's a waste,
1:01:24
SPEAKER_00: but it's actually not.
1:01:25
SPEAKER_00: It was learning.
1:01:26
SPEAKER_00: And now they're a lot more confident about what works, what doesn't.
1:01:30
SPEAKER_00: They'll confidently say, like, okay, we're not going to use...
1:01:33
AI to generate this part of the code base for prototyping is
1:01:36
great and so on.
1:01:37
SPEAKER_00: So I feel like everyone's figuring things out.
1:01:40
SPEAKER_00: So it's almost a waste to not get permission.
1:01:44
And how better way to get permission than to show that you're
1:01:47
doing this as well?
1:01:48
SPEAKER_01: Yes, yes, absolutely.
1:01:49
SPEAKER_01: AI is a tool.
1:01:51
Tools that can compress work,
1:01:53
I think,
1:01:54
can also lead to good moments of clarity.
1:01:56
SPEAKER_01: And it is faster than ever to try out certain classes of ideas.
1:02:00
And I think that that efficiency can also sometimes highlight
1:02:02
the importance of our
1:02:03
SPEAKER_01: human qualities when it comes to judging.
1:02:05
SPEAKER_01: What should we be moving forward with?
1:02:07
SPEAKER_01: What should we be ignoring?
1:02:08
SPEAKER_01: What should we be minimizing?
1:02:09
I think that trying to embrace AI in this moment and figure out
1:02:14
what works well for
1:02:15
SPEAKER_01: you and your teams is a great use of time.
1:02:18
SPEAKER_01: I did want to say,
1:02:19
SPEAKER_01: if there are folks in your audience who are in enterprise,
1:02:24
SPEAKER_01: especially,
1:02:25
SPEAKER_01: I know that having gone through this journey,
1:02:27
SPEAKER_01: there may be teams who feel like,
1:02:28
SPEAKER_01: oh man,
1:02:29
SPEAKER_01: it feels like startups are able to move so much faster than we are,
1:02:34
SPEAKER_01: or they're able to use all of these great new tools and these models,
1:02:37
but we are still waiting on the enterprise-friendly version of
1:02:40
these things.
1:02:41
And I've talked to teams where they can sometimes feel like
1:02:44
we're waiting,
1:02:45
SPEAKER_01: we're waiting,
1:02:45
SPEAKER_01: we're waiting to be able to try out more of these tools.
1:02:47
SPEAKER_01: What I ended up doing in my team was we were similarly,
1:02:50
SPEAKER_01: you know,
1:02:51
SPEAKER_01: a couple of years ago now,
1:02:52
we were similarly waiting for the company embraced official way
1:02:57
of doing things.
1:02:58
SPEAKER_01: But that didn't stop us from being able to learn.
1:03:01
So I encourage people,
1:03:02
well,
1:03:03
a lot of us are hacking on side projects right at the weekends.
1:03:06
SPEAKER_01: We can try out whatever,
1:03:07
SPEAKER_01: you know,
1:03:08
SPEAKER_01: third party tools,
1:03:09
SPEAKER_01: third party models,
1:03:10
SPEAKER_01: our own models that we can try out and learn.
1:03:12
SPEAKER_01: And that doesn't have to be a big blocker for you.
1:03:15
SPEAKER_01: So you can still,
1:03:16
SPEAKER_01: you know,
1:03:16
help your team along this journey without necessarily having to
1:03:21
do a lot of that
1:03:22
SPEAKER_01: waiting.
1:03:22
So just wanted to let folks know there are many paths to being
1:03:25
able to embrace this moment.
1:03:26
Yeah,
1:03:27
and I think the consensus is pretty much that these tools will
1:03:30
be widespread.
1:03:30
They keep changing and they're already contributing to a lot of
1:03:34
Teams code bases
1:03:35
SPEAKER_00: and people are using it.
1:03:36
SPEAKER_00: So might as well just, you know, use it because it takes time in the end.
1:03:40
So the earlier you start,
1:03:42
you'll have plenty of learning to do regardless.
1:03:45
SPEAKER_00: Yes, absolutely right.
1:03:46
SPEAKER_00: So as closing,
1:03:47
SPEAKER_00: I just have a few rapid questions,
1:03:49
SPEAKER_00: so I'll just fire them and then you tell me what comes to mind.
1:03:52
SPEAKER_00: What's your favorite programming language and why?
1:03:55
SPEAKER_01: My favorite programming language is JavaScript.
1:03:58
SPEAKER_00: I thought you might say this, but I wasn't sure.
1:04:02
SPEAKER_00: Of course.
1:04:02
SPEAKER_00: You wrote books about this.
1:04:04
My favorite programming language is JavaScript,
1:04:06
and it's not for the personal reasons.
1:04:10
SPEAKER_01: There are probably better programming languages.
1:04:12
I like JavaScript because it enables anybody on the planet to
1:04:15
be able to build and
1:04:17
SPEAKER_01: ship something to the web in a way that doesn't require gatekeepers.
1:04:20
SPEAKER_01: It's very open.
1:04:22
SPEAKER_01: And I think that there's something very liberating about that idea.
1:04:25
SPEAKER_01: So that is one of the reasons why I like JavaScript probably the most.
1:04:28
SPEAKER_00: I like it.
1:04:29
I'm always surprised when a software engineer says JavaScript
1:04:34
because we know that
1:04:34
SPEAKER_00: compared to other languages,
1:04:36
it has a lot of limitations and there's been books written
1:04:38
about it as well.
1:04:40
SPEAKER_00: But I cannot disagree with it.
1:04:43
SPEAKER_00: I love it.
1:04:44
SPEAKER_00: It is true.
1:04:45
SPEAKER_00: What's a tool that you really like using and what does it do?
1:04:48
SPEAKER_01: I would say that one of my favorite tools at the moment, and I'm biased.
1:04:54
SPEAKER_01: I have a book out about this as well.
1:04:56
SPEAKER_01: It's probably Bolt, Bolt.new.
1:04:59
Bolt.new is one of those Vibe coding scaffolding tools that
1:05:03
people can use.
1:05:04
They very recently added support,
1:05:06
actually,
1:05:07
for being able to use custom agents.
1:05:10
SPEAKER_01: So you can use Cloud Code, for example, to build your Vibe coded app.
1:05:15
SPEAKER_01: And the output is generally very high quality, great design.
1:05:18
SPEAKER_01: So I've been really enjoying that.
1:05:20
And the team have been,
1:05:22
I would say,
1:05:22
on the edge of offering some really great integration.
1:05:25
So I've been liking this idea that many Vibe coding platforms
1:05:28
are now starting to
1:05:29
SPEAKER_01: think about the integration layer.
1:05:32
SPEAKER_01: So how can we automate the idea of you needing to use,
1:05:35
SPEAKER_01: you know,
1:05:36
SPEAKER_01: like a super base or an authentication provider or these other things?
1:05:40
You still need to pay,
1:05:41
of course,
1:05:41
attention to all of the code that's being generated.
1:05:43
But being able to remove more of that setup friction,
1:05:45
I think,
1:05:46
is amazing.
1:05:47
And I guess the fun fact on Bolt.new is they started off as a
1:05:50
company called
1:05:50
SPEAKER_00: StackBlitz,
1:05:51
SPEAKER_00: who built a really cool online editor,
1:05:54
SPEAKER_00: really advanced.
1:05:56
SPEAKER_00: And then they moved over from that to Bolt.new.
1:05:58
And there are software engineers behind it,
1:06:00
again,
1:06:00
who really know their craft.
1:06:02
SPEAKER_00: Yes, yes, absolutely.
1:06:04
SPEAKER_00: And finally, what's a book that you would recommend?
1:06:08
SPEAKER_01: So the book I'm going to recommend is one that I think covers,
1:06:13
SPEAKER_01: I would say,
1:06:14
career paths and best practices for software engineers in a way
1:06:17
that I found very
1:06:18
SPEAKER_01: compelling.
1:06:19
SPEAKER_01: It is by you.
1:06:19
SPEAKER_01: It's the Software Engineer's Guidebook.
1:06:21
And of course,
1:06:22
I'm playing to the crowd,
1:06:23
but it is genuinely a very,
1:06:26
very sharp book.
1:06:28
SPEAKER_01: From all the reviews that I've read about it,
1:06:30
SPEAKER_01: other people have also found it to be an excellent read.
1:06:32
SPEAKER_00: It's on my desk, but we didn't talk about this, by the way.
1:06:35
SPEAKER_00: So this recommendation came just as a surprise to me as others.
1:06:38
SPEAKER_00: Thank you.
1:06:38
SPEAKER_01: Genuinely an excellent book.
1:06:42
SPEAKER_01: I guess if I wasn't going to,
1:06:45
SPEAKER_01: of course,
1:06:45
SPEAKER_01: recommend that one,
1:06:47
I think that if you are trying to learn more about the
1:06:51
foundational aspects of AI
1:06:52
SPEAKER_01: engineering in this moment,
1:06:54
there's a great book called AI Engineering by Chip Huynh that
1:06:58
is also worth taking
1:06:59
SPEAKER_01: a look at.
1:07:00
very very well reviewed a very very thorough book and i also
1:07:03
recommend folks check
1:07:04
that out it's such a good book i i feel if if you know the
1:07:07
concepts there you're
1:07:08
pretty well off and if you don't know them you're probably
1:07:11
gonna be you know still
1:07:12
finding your way so big plus one for that addy this has been
1:07:16
great thanks so much
1:07:17
SPEAKER_00: for for doing this and this really awesome conversation
1:07:21
SPEAKER_01: Well, thank you.
1:07:21
SPEAKER_01: I really appreciate it.
1:07:23
SPEAKER_01: I hope the folks,
1:07:24
SPEAKER_01: you know,
1:07:24
SPEAKER_01: if you are interested in this space,
1:07:26
SPEAKER_01: Beyond Vibe Coding is out now from O'Reilly.
1:07:29
SPEAKER_01: Hopefully it'll be useful to some people.
1:07:31
SPEAKER_01: But it's been a pleasure having this conversation with you.
1:07:33
SPEAKER_00: Yeah, and I've been reading the book.
1:07:35
I really like how it goes into the practicality,
1:07:38
so I can also very much recommend it.
1:07:40
SPEAKER_00: Thank you.
1:07:41
I hope you enjoyed going beyond Vibe coding,
1:07:44
pun intended,
1:07:45
with Addy Osmani.
1:07:46
One of the things I took away from this conversation is how
1:07:49
it's really important
1:07:50
SPEAKER_00: that,
1:07:50
SPEAKER_00: as an engineer,
1:07:51
SPEAKER_00: we keep understanding exactly what the LLM does and why.
1:07:55
And if we don't understand it,
1:07:57
we just stop,
1:07:57
understand it,
1:07:58
and then move on.
1:07:59
SPEAKER_00: As long as you understand the LLM, you are in charge.
1:08:02
SPEAKER_00: But once you stop understanding,
1:08:04
SPEAKER_00: it's in charge and you can kind of become helpless and out of your depth.
1:08:08
SPEAKER_00: For more tips on how to get up to speed with AI engineering,
1:08:11
SPEAKER_00: check out Deep Dives in the Pragmatic Engineer that I wrote,
1:08:14
SPEAKER_00: which are linked to the show notes below.
1:08:15
SPEAKER_00: If you've enjoyed this podcast,
1:08:17
SPEAKER_00: please do subscribe on your favorite podcast platform and on YouTube.
1:08:20
SPEAKER_00: A special thank you if you also leave a rating for the show.
1:08:23
SPEAKER_00: Thanks and see you in the next one.