Latest / The Pragmatic Engineer / What is a Principal Engineer at Amazon? With Steve Huynh

Transcript

What is a Principal Engineer at Amazon? With Steve Huynh · The Pragmatic Engineer

0:00
SPEAKER_01: If you're going to optimize for performance,
0:02
SPEAKER_01: saying,
0:02
why can't we be at one millisecond or why can't we be at 10
0:05
milliseconds and start
0:06
SPEAKER_01: from there instead of sort of saying,
0:07
SPEAKER_01: hey,
0:08
SPEAKER_01: let's try to decrease latencies by 50% or 25%.
0:11
Let's just start from what is the conceptually fastest thing
0:14
that we could do.
0:15
SPEAKER_01: And that's actually how Amazon was created.
0:17
Amazon's principal engineering level is unique in many ways
0:19
across big tech.
0:21
Steve Hewden was a software engineer at Amazon for 17 years and
0:24
worked as the last
0:25
SPEAKER_00: four years as a principal engineer.
0:27
SPEAKER_00: Today,
0:27
SPEAKER_00: we talk about the ins and outs of this role,
0:29
SPEAKER_00: including why being promoted from senior to principal is so hard,
0:33
even though Amazon usually has hundreds of principal
0:35
engineering openings and
0:37
SPEAKER_00: thousands of seniors trying to get into these positions,
0:40
SPEAKER_00: the Amazon principal engineering community,
0:42
SPEAKER_00: the in-person events,
0:43
SPEAKER_00: the Slack group,
0:43
SPEAKER_00: and the principles of Amazon internal presentation series.
0:47
SPEAKER_00: Engineering concepts at Amazon are on reliability,
0:49
SPEAKER_00: such as brownouts and COE,
0:51
SPEAKER_00: correction of errors,
0:52
SPEAKER_00: and many more topics.
0:54
If you're interested in understanding one of the hardest
0:56
engineering levels to get
0:57
SPEAKER_00: into across big tech,
0:58
SPEAKER_00: together with stories of how Steve thrived in this position,
1:01
SPEAKER_00: this episode is for you.
1:03
Subscribing on YouTube and on your favorite podcast player
1:05
greatly helps more
1:06
SPEAKER_00: people discover this show.
1:07
SPEAKER_00: If you enjoy it, thanks for doing so.
1:10
SPEAKER_00: So Steve, welcome to the podcast.
1:12
SPEAKER_00: Thanks for having me.
1:13
SPEAKER_00: How long were you at Amazon?
1:15
SPEAKER_00: 17 years?
1:16
SPEAKER_01: Yeah, I was there for 17 and a half years.
1:19
SPEAKER_01: And yeah, I just quit last year.
1:22
SPEAKER_01: So I've been basically a year doing other things now.
1:26
SPEAKER_01: And what were the things that you worked on while you were there?
1:29
SPEAKER_01: People always talk about my long tenure there,
1:32
SPEAKER_01: but I feel like I've had five or six jobs over that time period.
1:38
SPEAKER_01: I started off on a project called Search Inside the Book.
1:42
SPEAKER_01: I worked on the first Kindle launch.
1:45
SPEAKER_01: Wow.
1:46
SPEAKER_01: I worked on the precursor to Prime Video.
1:49
SPEAKER_01: I sort of worked there at the beginning part of my career,
1:51
and then I sort of ended my career there for the last five
1:55
years of my time there.
1:56
SPEAKER_01: I worked in payments.
1:59
SPEAKER_01: I worked in Amazon Local,
2:01
which was sort of our Groupon project when that type of
2:04
business was looking like
2:06
SPEAKER_01: it was going to take over.
2:08
SPEAKER_01: I worked on Amazon Restaurants.
2:09
SPEAKER_01: I worked on Amazon Tickets, which was our Ticketmaster clone.
2:13
And then my last five years was working on live sports
2:16
streaming on Prime Video.
2:18
SPEAKER_00: If you want to build a great product, you have to ship quickly.
2:22
SPEAKER_00: But how do you know what works?
2:24
SPEAKER_00: More importantly, how do you avoid shipping things that don't work?
2:28
SPEAKER_00: The answer, Statsig.
2:30
SPEAKER_00: Statsig is a unified platform for flags,
2:32
SPEAKER_00: analytics,
2:33
SPEAKER_00: experiments,
2:34
SPEAKER_00: and more,
2:35
combining five plus products into a single platform with a
2:37
unified set of data.
2:39
SPEAKER_00: Here's how it works.
2:41
SPEAKER_00: First, StatSick helps you ship a feature via FeatureFly or Config.
2:45
SPEAKER_00: Then it measures how it's working,
2:47
SPEAKER_00: from alerts and errors,
2:49
SPEAKER_00: to replays of people using that feature,
2:51
SPEAKER_00: to measurement of top-line impact.
2:53
SPEAKER_00: Then you get your analytics,
2:54
SPEAKER_00: user account metrics,
2:55
SPEAKER_00: and dashboards to track your progress over time,
2:58
SPEAKER_00: all linked to the stuff you ship.
3:00
SPEAKER_00: Even better,
3:00
SPEAKER_00: Statsik is incredibly affordable with a super generous free tier,
3:04
SPEAKER_00: a starter program with $50,000 of free credits,
3:07
SPEAKER_00: and custom plans to help you consolidate your existing spend on flags,
3:10
SPEAKER_00: analytics,
3:10
SPEAKER_00: or A-B testing tools.
3:12
SPEAKER_00: To get started, go to statsik.com slash pragmatic.
3:16
SPEAKER_00: That is S-T-A-T-S-I-G dot com slash pragmatic.
3:19
SPEAKER_00: Happy building.
3:21
SPEAKER_00: This episode was brought to you by Graphite,
3:23
SPEAKER_00: the developer productivity platform that helps developers create,
3:26
SPEAKER_00: review,
3:26
SPEAKER_00: and merge smaller code changes,
3:28
SPEAKER_00: stay unblocked,
3:29
SPEAKER_00: and ship faster.
3:31
SPEAKER_00: Code review is a huge time sink for engineering teams.
3:34
Most developers spend about a day per week or more reviewing
3:37
code or blocked
3:37
SPEAKER_00: waiting for a review.
3:39
SPEAKER_00: It doesn't have to be this way.
3:41
SPEAKER_00: Graphite brings stack pull requests,
3:43
the workflow at the heart of the best-in-class internal code
3:46
review tools at
3:46
SPEAKER_00: companies like Meta and Google,
3:48
SPEAKER_00: to every software company on GitHub.
3:51
SPEAKER_00: Graphite also leverages high-signal,
3:53
code-based-aware AI to give developers immediate actionable
3:55
feedback on their pull
3:56
SPEAKER_00: requests,
3:57
SPEAKER_00: allowing teams to cut down on review cycles.
4:00
SPEAKER_00: Tens of thousands of developers at top companies like Asana,
4:03
SPEAKER_00: Ramp,
4:04
SPEAKER_00: Tekton,
4:04
SPEAKER_00: and Vercel rely on Graphite every day.
4:07
Start stacking with Graphite today for free and reduce your
4:10
time to merge from days to hours.
4:12
SPEAKER_00: Get started at gt.dev slash pragmatic.
4:15
SPEAKER_00: That is g for graphite, t for technology.dev slash pragmatic.
4:20
SPEAKER_00: So that's a lot of different teams.
4:23
SPEAKER_00: How did you work on so many teams?
4:24
SPEAKER_00: Is it just like there's a lot of internal transfers?
4:27
SPEAKER_00: Did you get bored?
4:28
SPEAKER_00: Was it just you followed your manager?
4:30
SPEAKER_00: How does it work inside Amazon?
4:31
SPEAKER_00: Because when people think about companies,
4:33
SPEAKER_00: people who have not worked on Amazon,
4:34
SPEAKER_00: they would kind of assume you go,
4:36
SPEAKER_00: you work there,
4:36
SPEAKER_00: you're on a team for like,
4:37
SPEAKER_00: you know,
4:38
SPEAKER_00: four or five,
4:38
SPEAKER_00: six years.
4:39
SPEAKER_00: Clearly not the case.
4:40
SPEAKER_01: You know,
4:40
it depends a little bit on like corporate policy and then where
4:43
you are with your
4:44
SPEAKER_01: career.
4:45
SPEAKER_01: I started as a support engineer.
4:47
SPEAKER_01: So sort of like operationally.
4:50
SPEAKER_01: focused person.
4:51
SPEAKER_01: And then I was basically like, I want to be a software developer.
4:55
SPEAKER_01: And so I think getting into the company was pretty difficult.
5:00
SPEAKER_01: But once I was there, sort of set that target and changed roles.
5:04
And when I changed the role,
5:07
it was a natural time to move to another team.
5:11
SPEAKER_01: There's also some internal policy.
5:14
SPEAKER_01: So basically at Amazon,
5:15
it used to be that you had to stay on a team for at least a
5:20
year before you
5:21
SPEAKER_01: transferred.
5:21
SPEAKER_01: And if you wanted to transfer,
5:24
like a senior manager or director or whoever up top could block
5:28
your transfer.
5:29
And what that ended up meaning was that certain teams that were
5:33
just terrible to
5:34
SPEAKER_01: work on,
5:35
those teams actually had more than 100% attrition over the
5:39
course of a year because
5:40
SPEAKER_01: you measured attrition with a year-long time unit.
5:43
SPEAKER_01: Amazon did something actually smart at the corporate level.
5:47
SPEAKER_01: They basically said, okay, well, you have freedom of movement now.
5:52
This sort of happened,
5:53
I don't know,
5:54
probably like 13 years ago,
5:55
10,
5:56
13 years ago.
5:57
SPEAKER_01: And so they said, you have freedom of movement now.
6:00
SPEAKER_01: A VP or a director can't block you.
6:03
They can say,
6:03
okay,
6:03
well,
6:04
we need another month to get a transition plan going.
6:07
SPEAKER_01: But essentially,
6:08
you have freedom of movement as long as you're not on a
6:10
performance improvement
6:11
SPEAKER_01: plan,
6:12
which meant that certain teams were sources of high-quality
6:15
engineering talent and
6:16
SPEAKER_01: certain teams were sinks of high-quality engineering talent.
6:19
SPEAKER_02: Mm-hmm.
6:20
SPEAKER_01: And it sort of created an internal marketplace for different roles.
6:25
SPEAKER_01: Now,
6:25
SPEAKER_01: what that ended up meaning was that certain teams,
6:29
SPEAKER_01: they basically didn't want you to know what the policy was.
6:32
SPEAKER_01: They wanted you to sort of think that you were kind of stuck.
6:36
But,
6:36
you know,
6:37
despite that sort of like local gamesmanship that was going.
6:41
SPEAKER_00: Yeah, like basically some managers didn't want their best people to leap.
6:43
SPEAKER_01: Exactly.
6:44
SPEAKER_01: Let's just say it how it is.
6:45
SPEAKER_01: But ultimately,
6:46
I think it's a great strategy because if there was a team that
6:52
was difficult to
6:52
SPEAKER_01: staff,
6:54
SPEAKER_01: the problem was on the management.
6:56
It wasn't something that had to be bared or borne from the
7:01
employee themselves.
7:03
SPEAKER_01: And so,
7:04
SPEAKER_01: getting back to my own career journey,
7:07
SPEAKER_01: at a very large company like Amazon,
7:08
SPEAKER_01: there are so many awesome things that are going on.
7:12
SPEAKER_01: And I decided to just kind of go where my curiosity took me.
7:18
Now,
7:18
there were some times where there were reorgs or a line of
7:22
business got spun down.
7:25
But ultimately,
7:26
I think freedom of movement was one of the smartest things that
7:29
Amazon did.
7:30
SPEAKER_00: Yeah.
7:31
I think this is something that people don't really appreciate
7:33
about some large companies.
7:34
So,
7:34
you know,
7:35
not all companies are like Amazon and every company changes,
7:37
right?
7:37
SPEAKER_00: Like today,
7:38
SPEAKER_00: I'm assuming it will be hard to move as many teams within Amazon,
7:42
SPEAKER_00: depending on where you are.
7:43
SPEAKER_00: You know,
7:43
SPEAKER_00: if you're in a satellite office where there's two teams,
7:47
SPEAKER_00: you can probably move on to the other team at max.
7:49
SPEAKER_00: But I think this is one of the underrated things of large companies.
7:52
Like once you are in,
7:53
it's almost always easier to get that job at another team from
7:57
the inside.
7:57
SPEAKER_00: Yes.
7:58
SPEAKER_00: Especially because you can talk to them.
7:59
SPEAKER_00: You know,
8:00
SPEAKER_00: I talked with the Reddit mobile team and I asked like,
8:03
SPEAKER_00: oh,
8:03
SPEAKER_00: how can you become a platform engineer on the mobile team?
8:06
SPEAKER_00: And they said like, well, you know, most of our hires have been internal.
8:09
SPEAKER_00: They just helped us out on hackathons.
8:11
SPEAKER_00: They come around.
8:12
SPEAKER_00: They commit stuff.
8:13
SPEAKER_00: We know them.
8:14
SPEAKER_00: It's a low risk hire.
8:15
SPEAKER_00: Yeah.
8:15
I think it's just nice to remember that when you think of like
8:17
a big company like
8:18
SPEAKER_00: Amazon or Meta or Microsoft,
8:20
SPEAKER_00: it's just so many small teams.
8:21
SPEAKER_00: And once you're in,
8:22
you actually have almost priority access to those teams if you
8:26
play your cards
8:27
SPEAKER_00: right.
8:27
SPEAKER_01: Absolutely.
8:28
SPEAKER_01: And,
8:28
SPEAKER_01: you know,
8:29
SPEAKER_01: you might interview for that team,
8:30
SPEAKER_01: but it's such lower stakes than an external interview.
8:34
SPEAKER_01: And,
8:35
SPEAKER_01: you know,
8:35
SPEAKER_01: just all things being equal,
8:37
SPEAKER_01: would you rather take somebody that's,
8:39
SPEAKER_01: you know,
8:40
SPEAKER_01: internal and knows the culture,
8:42
SPEAKER_01: they know how software is developed within a particular context,
8:45
SPEAKER_01: or somebody that's just as good,
8:48
SPEAKER_01: but doesn't,
8:48
SPEAKER_01: you know,
8:49
SPEAKER_01: hasn't been onboarded?
8:50
And I think ultimately you're going to pick the person that's
8:53
internal,
8:53
all things being equal.
8:54
SPEAKER_00: Yeah, it's just kind of like business rationality for the most part.
8:57
So one thing about Amazon and about large companies like Amazon
9:01
is people talk
9:02
SPEAKER_00: about externally about the scale.
9:03
SPEAKER_00: And it's hard to imagine,
9:05
but can you give us a sense of the scale that you've seen or
9:08
like some tough
9:08
engineering challenges that you worked on that would have been
9:10
just really hard to
9:12
SPEAKER_00: work at a smaller startup?
9:13
SPEAKER_00: Yeah.
9:13
SPEAKER_01: Yeah,
9:14
SPEAKER_01: I think that's the thing that you just,
9:17
SPEAKER_01: you will not see at most other places is the scale of things.
9:22
SPEAKER_01: I'll give you a couple of examples.
9:23
SPEAKER_01: So, you know, Prime is the exclusive club that everybody is a member of.
9:29
And,
9:30
you know,
9:30
in the U.S.,
9:31
the shipping benefit is probably,
9:34
you know,
9:34
the most popular.
9:35
SPEAKER_01: But...
9:36
SPEAKER_01: Globally,
9:38
SPEAKER_01: Prime Video is,
9:40
SPEAKER_01: you know,
9:40
SPEAKER_01: it's the thing that people use the most with their subscription.
9:45
SPEAKER_01: And so if you think about,
9:47
SPEAKER_01: you know,
9:47
SPEAKER_01: our service-oriented architecture and,
9:49
SPEAKER_01: you know,
9:50
SPEAKER_01: just loading up the app,
9:52
SPEAKER_01: the Gateway page is the place where all of our requests come in,
9:57
SPEAKER_01: right?
9:57
SPEAKER_01: And so it's just like Netflix.
9:59
SPEAKER_01: It's this infinite scroll of carousels.
10:02
SPEAKER_01: So the Gateway page, is it the Amazon Prime landing page?
10:04
SPEAKER_01: Yeah, it's the landing page there.
10:06
SPEAKER_01: And so you're like, okay, cool.
10:09
SPEAKER_01: If let's say 90,
10:10
SPEAKER_01: 95,
10:20
99% of all of your requests are coming from that page and that
10:13
page needs to be
10:14
SPEAKER_01: personalized,
10:16
SPEAKER_01: you know,
10:16
and you have a service oriented architecture with a bunch of
10:19
microservices.
10:21
SPEAKER_01: Um,
10:22
SPEAKER_01: One request to that page turns into,
10:25
SPEAKER_01: let's just say,
10:26
SPEAKER_01: hundreds of downstream requests to different services.
10:29
SPEAKER_01: It might even be more than that.
10:31
SPEAKER_01: It's actually kind of hard to count.
10:33
SPEAKER_01: Yeah.
10:34
SPEAKER_01: And is this page, right?
10:35
SPEAKER_01: Like all the stuff flowing, all personalized stuff.
10:37
SPEAKER_01: So that's the retail one.
10:39
SPEAKER_01: But I was talking about the Prime Video one.
10:41
SPEAKER_01: The Prime Video.
10:41
SPEAKER_01: But essentially, it's the same thing.
10:42
SPEAKER_01: Yeah.
10:43
SPEAKER_01: And so, you know, same thing for the retail website as well.
10:46
SPEAKER_01: And so if you have one request sort of spidering out into,
10:50
SPEAKER_01: you know,
10:51
SPEAKER_01: two orders of magnitude more requests internally,
10:54
SPEAKER_01: you start to see like really,
10:56
SPEAKER_01: really large scale for these microservices.
10:59
So a microservice will have like a reverse proxy or a load
11:02
balancer in front of it.
11:03
And you are sort of unironically talking about things like tens
11:06
of thousands of
11:08
SPEAKER_01: requests per second or hundreds of thousands of requests per second.
11:12
SPEAKER_01: coming into your service.
11:13
SPEAKER_00: So like the services that are like behind,
11:15
SPEAKER_00: you know,
11:16
SPEAKER_00: like there's the Prime,
11:17
SPEAKER_00: there's all the things loading,
11:18
SPEAKER_00: they're spidering out,
11:19
SPEAKER_00: like making,
11:20
SPEAKER_00: you know,
11:20
SPEAKER_00: to render that one recommendation,
11:22
SPEAKER_00: for example,
11:23
SPEAKER_00: for,
11:23
SPEAKER_00: I don't know,
11:23
SPEAKER_00: the video that you would like,
11:25
SPEAKER_00: it will make a lot of requests of different services.
11:27
SPEAKER_00: And then so when you're operating a smaller service inside of Amazon,
11:31
SPEAKER_00: suddenly you're going to be hit with what you just said,
11:34
SPEAKER_00: 10K,
11:34
SPEAKER_00: 100K requests per second,
11:35
SPEAKER_00: that kind of scale.
11:36
SPEAKER_00: Exactly.
11:37
SPEAKER_01: And you will essentially be DDoSing yourself.
11:42
SPEAKER_01: You're just like, okay, cool.
11:44
SPEAKER_01: Let's change a caching configuration on some item details.
11:49
SPEAKER_01: And it turns out you've just browned out like a critical service, right?
11:55
SPEAKER_01: What does browned out mean?
11:57
SPEAKER_01: Oh, sorry.
11:58
SPEAKER_01: I'm using some jargon.
11:59
SPEAKER_01: So if you want to talk about availability,
12:02
if you suppose you are DDoSing a service or sending a lot of
12:07
requests over to them,
12:08
SPEAKER_01: you can,
12:09
SPEAKER_01: you know,
12:09
SPEAKER_01: you can,
12:10
SPEAKER_01: you can just take them down.
12:11
SPEAKER_01: That would be like a blackout.
12:12
SPEAKER_01: Yeah.
12:13
And so like you send a request,
12:15
oh,
12:15
you can't establish a connection immediately comes back.
12:18
SPEAKER_01: But there's a type of outage where they brown out.
12:22
SPEAKER_01: So basically, they're reachable.
12:23
SPEAKER_01: They might accept the connection.
12:26
But,
12:26
you know,
12:27
they'll essentially time out or they might return partial
12:31
results or bad results.
12:32
Or the only thing that they do return is,
12:34
you know,
12:35
500 for some percentage or proportion.
12:37
SPEAKER_01: After you waited a bunch of time for that.
12:38
SPEAKER_01: Yeah.
12:39
SPEAKER_01: And so...
12:40
Now we start talking about availability and resilience in the
12:45
face of all of this
12:46
SPEAKER_01: DDoSing that you're doing to yourself.
12:49
And so the thing on top of scale that is going to really
12:53
complicate things is your
12:54
SPEAKER_01: dependency chain.
12:56
SPEAKER_01: Right.
12:56
And so,
12:57
you know,
12:58
your service is a dependency of some of the process that's
13:01
going on.
13:02
SPEAKER_01: It depends on, you know, maybe AWS.
13:05
SPEAKER_01: It may depend on another service.
13:06
SPEAKER_01: You know,
13:07
SPEAKER_01: how do you make sure that if,
13:09
SPEAKER_01: you know,
13:10
suppose there's a failure for a primary dependency and that
13:13
dependency comes back
13:14
SPEAKER_01: up?
13:15
How do you make sure you don't just like inundate it with a
13:17
bunch of requests as
13:18
SPEAKER_01: it's trying to recover?
13:19
SPEAKER_01: Right.
13:20
SPEAKER_01: And so you have all of these sort of like odd dynamics that occur.
13:23
I used a brownout as something that is a perennial problem that
13:28
we have,
13:29
right?
13:29
Where there's maybe a dependency on a base service like S3 or
13:33
DynamoDB or whatever it is.
13:36
There might be some increased latency that may cause a chain
13:40
reaction of a
13:41
SPEAKER_01: dependency going down.
13:43
SPEAKER_01: And then one of these sort of middle tier services would brown out.
13:46
SPEAKER_01: So you're an owner of the services for your team.
13:52
SPEAKER_01: And so then it's like, okay, what do we do in those situations?
13:56
SPEAKER_01: How do we know that they're browning out?
13:58
SPEAKER_01: What do we do in the face of a dependency outage?
14:02
SPEAKER_01: And then critically,
14:03
SPEAKER_01: if there is an outage and then the service comes back up,
14:06
SPEAKER_01: how do we make sure that we give it enough space so that it can breathe?
14:09
SPEAKER_02: Mm-hmm.
14:10
SPEAKER_01: So that,
14:11
SPEAKER_01: you know,
14:11
SPEAKER_01: you know,
14:12
SPEAKER_01: as they're trying to recover from some sort of outage,
14:15
SPEAKER_01: we don't just take them down immediately again.
14:17
And I guess for like most of us who are not working right now
14:22
on these services,
14:23
SPEAKER_00: like these sound pretty cool in theory.
14:25
SPEAKER_00: But you're saying this was actually like, like, this is not theory.
14:27
SPEAKER_00: This actually was like, oh, this service is going down.
14:30
SPEAKER_00: We are literally having 100K requests per second.
14:33
SPEAKER_00: And we're like...
14:34
Pushing that on to like other three services with the same
14:37
because we need to
14:38
SPEAKER_00: invoke three other services.
14:39
SPEAKER_00: One of them has browned out.
14:40
SPEAKER_00: What do we do now?
14:41
SPEAKER_00: How do we fix it?
14:42
SPEAKER_01: Yeah.
14:43
And I think for certain other large tech companies,
14:48
you know,
14:49
you can do best effort.
14:52
SPEAKER_01: Which is basically like,
14:53
SPEAKER_01: hey,
14:54
SPEAKER_01: we're temporarily down,
14:56
SPEAKER_01: but you have some sort of degraded service that makes sense.
15:01
But if you're on,
15:02
say,
15:02
a website that does purchases,
15:05
now we're talking about transactions.
15:07
SPEAKER_01: Or if you're in the Prime Video,
15:09
SPEAKER_01: like live video streaming use case,
15:12
SPEAKER_01: now we're talking about a football game that you're unable to see.
15:17
SPEAKER_01: And then when we recover, the game might be over.
15:20
SPEAKER_01: And so it's much higher stakes.
15:22
SPEAKER_01: And so I think the scale with transactional semantics,
15:27
SPEAKER_01: right,
15:27
like that's actually the challenge that you're not going to see
15:31
unless you sort of
15:31
SPEAKER_01: like work for a payment processor or something like that.
15:34
Yeah,
15:35
I guess that real world pressure challenge,
15:38
like you are losing money.
15:39
SPEAKER_00: I'm starting to understand why.
15:40
SPEAKER_00: I have noticed that startups love to hire from certain companies.
15:45
They usually start stuff to hire from other startups because
15:47
it's similar environment.
15:48
SPEAKER_00: From large tech companies, it's a bit of a maybe.
15:50
SPEAKER_00: I'm generalizing, obviously.
15:51
SPEAKER_00: This will not be true 100% of the time.
15:53
SPEAKER_00: But for example,
15:54
SPEAKER_00: hiring from Google,
15:55
a lot of startups are not as happy because the people coming
15:57
from Google are used
15:58
SPEAKER_00: to having this amazing team around them,
16:00
SPEAKER_00: internal tools.
16:01
SPEAKER_00: But most startups love hiring from Amazon.
16:03
SPEAKER_00: And I'm starting to get a sense of why this actually is.
16:06
SPEAKER_01: Yeah, I think that's part of the culture.
16:08
SPEAKER_01: You get hired as a software developer and they hand you a pager.
16:13
And before phone apps and things like that,
16:17
it was like this pager from the 90s.
16:21
It's really great because you have to operate the software that
16:25
you write.
16:26
SPEAKER_01: If you actually,
16:27
SPEAKER_01: you cannot write the software,
16:30
hand it over to the testing team and then throw it over to the
16:32
SRE team after
16:33
SPEAKER_01: you're done.
16:34
SPEAKER_01: Like you own that piece of software.
16:36
SPEAKER_02: Yeah.
16:36
SPEAKER_00: Yeah.
16:37
SPEAKER_00: At every team, right?
16:38
One interesting thing that we talked about yesterday over
16:41
dinner with Casey
16:42
Moratory is you said something interesting on how Amazon
16:45
measured how on their
16:46
SPEAKER_00: retail website,
16:47
SPEAKER_00: I think it was retail,
16:48
SPEAKER_00: maybe Amazon Prime,
16:49
SPEAKER_00: the
16:49
SPEAKER_00: lower the latency of something loading,
16:52
SPEAKER_00: like a page loading,
16:53
SPEAKER_00: like a purchase stage or a purchase button loading,
16:55
SPEAKER_00: the more revenue they got.
16:56
And they started to measure and there was a linear correction
16:59
as the faster it was,
17:00
SPEAKER_00: the more people converted.
17:01
SPEAKER_00: And it seemed it had no end.
17:04
SPEAKER_00: And the question Casey asked is like, okay, if this is the case,
17:07
SPEAKER_00: What would stop Amazon,
17:09
SPEAKER_00: because you have the best technologies in the world,
17:11
SPEAKER_00: you have AWS,
17:12
SPEAKER_00: you can build whatever you want,
17:14
SPEAKER_00: to get the latency of the website down to,
17:16
SPEAKER_00: let's say,
17:16
SPEAKER_00: like 10 milliseconds or even one millisecond?
17:19
SPEAKER_00: Because if this goes up, you would maximize revenue.
17:22
SPEAKER_00: So can you tell me about how this measurement actually happened?
17:27
SPEAKER_00: And why is Amazon's website still maybe not the fastest in the world,
17:34
SPEAKER_00: even though it would generate so many more billions,
17:36
SPEAKER_00: right?
17:36
SPEAKER_01: Yeah.
17:37
SPEAKER_01: Well,
17:37
SPEAKER_01: there are a couple of questions embedded in there,
17:39
SPEAKER_01: but we'll start with the latency to gross revenue measurement.
17:45
SPEAKER_01: So essentially somebody way back when,
17:48
SPEAKER_01: because we invest in logs and telemetry,
17:51
started tracking how much gross revenue we would make based off
17:55
of the latency for
17:56
SPEAKER_01: detail pages,
17:57
SPEAKER_01: based off of latency of gateway,
17:58
SPEAKER_01: based off of latency of the checkout pages.
18:01
And they noticed this dynamic where it's like,
18:03
if you're faster,
18:04
you just make more money.
18:06
SPEAKER_01: It's a pretty clear correlation.
18:08
SPEAKER_01: I think you would even go as far as to say as causation.
18:12
SPEAKER_01: And so there was this really big focus on latencies.
18:17
SPEAKER_01: I love the idea that,
18:19
SPEAKER_01: you know,
18:19
SPEAKER_01: if you're going to optimize for performance,
18:21
SPEAKER_01: saying like,
18:22
SPEAKER_01: why can't we be at one millisecond?
18:24
SPEAKER_01: Or why can't we be at 10 milliseconds and start from there?
18:27
Instead of sort of saying like,
18:28
hey,
18:29
let's try to decrease latencies by 50% or 25%.
18:31
Like,
18:32
let's just start from what is the conceptually fastest thing
18:36
that we could do.
18:38
SPEAKER_01: And I think in a vacuum...
18:40
The conceptually fastest thing that we could do is sort of like
18:44
a monolith,
18:45
SPEAKER_01: which is how Amazon started,
18:47
SPEAKER_01: where you have a web server with all of your catalog information,
18:52
SPEAKER_01: so all of the items that are there,
18:54
SPEAKER_01: and then transaction processing on the host.
18:55
SPEAKER_01: That would be the fastest process.
18:58
SPEAKER_00: way to run the retail website.
19:00
SPEAKER_00: And basically like a web request would be, it opens the HTTPS handshake.
19:05
SPEAKER_00: It hits the server.
19:06
SPEAKER_00: The server in an ideal world has everything cached or calculated.
19:10
SPEAKER_00: It sends it back.
19:11
SPEAKER_00: So the total like latency would be the time for this request,
19:15
SPEAKER_00: the time to transfer that data based on your internet speed.
19:18
SPEAKER_00: And that's it.
19:18
SPEAKER_00: That is the absolute, you cannot be faster than that.
19:20
SPEAKER_00: I don't think so.
19:21
SPEAKER_00: Maybe there's some exotic sort of thing that's there.
19:23
SPEAKER_00: Maybe you can do some exotic process.
19:24
protocol that I know predicts the future I'm like with UDP
19:27
sends it but but yeah
19:28
but this is your baseline I guess the the optimal would be like
19:31
zero click instead
19:32
of like a one click checkout right so we just send you stuff
19:35
before like you know
19:36
you want it that that would be the I guess the theoretical
19:38
maximum but you know if
19:40
you if there's some sort of like web request right so some HTTP
19:43
request and then
19:44
SPEAKER_01: some sort of like buy button that would be the fastest
19:47
SPEAKER_01: Right.
19:48
SPEAKER_01: And that's actually how Amazon was created.
19:50
We bought this,
19:51
you know,
19:51
sort of the opposite of horizontal scaling is vertical scaling.
19:54
SPEAKER_01: We bought these big sun boxes and,
19:57
SPEAKER_01: you know,
19:57
SPEAKER_01: we hacked up our own web server and C++ and,
20:02
SPEAKER_01: you know,
20:02
SPEAKER_01: to scale up,
20:03
SPEAKER_01: we bought bigger hardware.
20:06
SPEAKER_01: And then when that didn't work,
20:07
SPEAKER_01: you know,
20:07
SPEAKER_01: we bought like six of these big boxes and that ran Amazon.
20:10
SPEAKER_01: And we ran that wave up until the early 2000s.
20:14
SPEAKER_01: And then what we realized,
20:16
SPEAKER_01: we ran into a wall,
20:17
SPEAKER_01: which was that,
20:20
SPEAKER_01: you know,
20:20
SPEAKER_01: when you built the C++ binary,
20:23
SPEAKER_01: the binary could only be four gigabytes.
20:26
SPEAKER_01: And that was a hard limit based off of the 32-bit software,
20:30
SPEAKER_01: the architecture that we're running on before.
20:32
SPEAKER_01: We could not get above four gigabytes.
20:35
SPEAKER_01: And so these product managers would come and just be like,
20:37
SPEAKER_01: well,
20:37
SPEAKER_01: just make a change for me,
20:39
SPEAKER_01: right,
20:39
SPEAKER_01: to the devs.
20:40
And then they would just be like,
20:41
I don't think you understand that this is a hard constraint.
20:43
And so the size of the code or the binary code,
20:46
the compiled one,
20:47
it was there.
20:48
And you had so much business logic by then that it just filled
20:51
up four gigabytes.
20:52
SPEAKER_01: Yeah.
20:52
SPEAKER_01: Yeah.
20:52
SPEAKER_00: Yeah.
20:53
SPEAKER_01: And, you know, we had a distributed C++ build.
20:56
So,
20:57
you know,
20:57
you could,
20:58
you know,
20:58
it would take many,
20:59
many hours for it to compile.
21:00
SPEAKER_01: And so we would distribute it across desktops.
21:02
SPEAKER_01: And it was this whole big thing.
21:04
SPEAKER_01: But we ran into that wall.
21:06
SPEAKER_01: And so what we decided to do,
21:08
SPEAKER_01: and I think this was super smart,
21:09
SPEAKER_01: was like to lean into service-oriented architectures,
21:12
SPEAKER_01: right,
21:13
SPEAKER_01: and microservices.
21:14
SPEAKER_01: Yep.
21:15
SPEAKER_01: And when you break it down,
21:17
SPEAKER_01: a web service call is essentially,
21:19
SPEAKER_01: it's a remote procedure call,
21:21
SPEAKER_01: right?
21:21
SPEAKER_01: So you have this execution pointer and then you're like,
21:23
SPEAKER_01: okay,
21:23
SPEAKER_01: well,
21:24
SPEAKER_01: I need to do some computation or I need to gather some data.
21:27
I'm going to turn,
21:27
in turn,
21:28
make a HTTP request downstream to another service.
21:31
SPEAKER_01: And then you can sort of chain those things together.
21:33
SPEAKER_01: And so getting back to the original thing about performance,
21:37
In a world where you have to because you have thousands and
21:40
thousands of developers
21:42
SPEAKER_01: building,
21:42
SPEAKER_01: you know,
21:43
the stuff and the fact that you cannot have a monolith as big
21:47
as Amazon retail,
21:49
SPEAKER_01: you know,
21:49
SPEAKER_01: past something that's sort of like circa 2002 Amazon size.
21:53
SPEAKER_01: You have to lean into remote procedure call.
21:55
SPEAKER_01: You have to say that there is a web service.
21:58
The best performance that you can actually get is always going
22:00
to be bounded by the
22:01
SPEAKER_01: number of web requests that you end up making,
22:04
SPEAKER_01: whether it's the first order calls to say,
22:07
SPEAKER_01: go get the item details,
22:09
SPEAKER_01: but then also any blocking call that happens downstream.
22:13
And by blocking call,
22:15
we mean like you need to wait for this to finish to get your
22:17
data.
22:18
SPEAKER_00: Like,
22:18
SPEAKER_00: you know,
22:18
SPEAKER_00: it's a service that like returns,
22:20
SPEAKER_00: I don't know,
22:20
SPEAKER_00: your top five most likely to buy things.
22:23
It might need to make those,
22:24
let's say,
22:24
five requests or just one request.
22:26
SPEAKER_00: It needs to wait for that before it can return.
22:28
SPEAKER_01: Exactly.
22:28
SPEAKER_01: Exactly.
22:29
SPEAKER_01: And you can do this telemetry stuff.
22:30
SPEAKER_01: You can do this observability stuff to figure out,
22:33
SPEAKER_01: you know,
22:33
SPEAKER_01: within that service call chain,
22:35
SPEAKER_01: what the blocking call is.
22:36
SPEAKER_01: Yeah.
22:37
SPEAKER_01: And you can get some amount of visualization on it.
22:40
SPEAKER_01: And so then you can get down to the point where it's like,
22:43
SPEAKER_01: okay,
22:43
SPEAKER_01: if we're going to start from first principles,
22:45
SPEAKER_01: what's the least amount of latency that you can get for,
22:49
SPEAKER_01: say,
22:49
SPEAKER_01: like a web request or a checkout page call?
22:51
SPEAKER_01: You're going to run into, like, the absolute minimum, right?
22:56
SPEAKER_01: And it's going to be based off of,
22:58
SPEAKER_01: like,
22:58
SPEAKER_01: what are the required operations,
23:01
SPEAKER_01: you know,
23:01
SPEAKER_01: evaluation or transactions or whatever for that particular request.
23:05
SPEAKER_01: Yeah.
23:06
SPEAKER_00: And then basically,
23:06
SPEAKER_00: so as I understand,
23:08
SPEAKER_00: as it became more microservices and services,
23:10
SPEAKER_00: this was great for maintainability.
23:12
SPEAKER_00: And also you first just solved the issue of the monolith size.
23:17
And as we know with history,
23:19
of course,
23:20
now teams could be more autonomous.
23:21
SPEAKER_00: They're not as dependent.
23:22
SPEAKER_00: They could do the APIs, but it was a trade-off for latency.
23:26
And now you had to go back and figure out the blocking calls,
23:30
how to speed those up.
23:32
SPEAKER_00: How to do,
23:32
SPEAKER_00: I guess,
23:33
SPEAKER_00: you know,
23:33
SPEAKER_00: trade off things like caching,
23:35
SPEAKER_00: like,
23:35
SPEAKER_00: you know,
23:35
SPEAKER_00: you can have things fast,
23:36
but it might not be as correct on the first one or like just
23:39
tricky UI where you
23:40
SPEAKER_00: don't show the data just yet,
23:42
SPEAKER_00: but it's coming and the users sense a sense of like progress,
23:46
SPEAKER_00: those kind of things.
23:47
SPEAKER_01: And it also,
23:48
SPEAKER_01: I think,
23:48
SPEAKER_01: forces teams to really end products to really say,
23:51
SPEAKER_01: OK,
23:51
SPEAKER_01: like what is the strictly necessary processing that happens on this page?
23:56
Some of the work that I was doing before I left Prime Video was
23:59
basically like you
24:00
SPEAKER_01: have these really,
24:01
SPEAKER_01: really big,
24:01
SPEAKER_01: heavy gateway page or landing page requests.
24:06
And if you're in a situation with high load,
24:10
can you preemptively reduce the amount of...
24:15
SPEAKER_01: say,
24:15
SPEAKER_01: personalization that's going on to sort of speed up that page or,
24:20
SPEAKER_01: you know,
24:21
SPEAKER_01: to increase the amount of,
24:22
SPEAKER_01: like,
24:22
SPEAKER_01: throughput that you're able to have.
24:23
SPEAKER_01: So to serve more customers,
24:25
SPEAKER_01: can you do that in a smart way,
24:28
SPEAKER_01: right,
24:28
SPEAKER_01: that sort of anticipates load that's coming onto that page,
24:33
SPEAKER_01: say,
24:33
SPEAKER_01: if there's a football game coming up or something like that?
24:35
SPEAKER_00: Yeah.
24:37
SPEAKER_00: Sounds like these are just,
24:38
SPEAKER_00: like,
24:38
SPEAKER_00: A,
24:40
SPEAKER_00: they seem just hard to solve,
24:42
SPEAKER_00: but now you have to solve them.
24:43
SPEAKER_00: So it sounds like this...
24:45
this kept you busy and not everyone else busy at amazon to this
24:48
date right like is
24:50
is this do you think is this is this ongoing engineering
24:52
challenge for amazon
24:53
because you know what i would imagine the tricky thing being
24:57
here is like okay you
24:58
can optimize whatever you have you can find the critical paths
25:01
but amazon keeps
25:01
growing right like there's new teams new services new
25:04
everything coming on so this
25:06
SPEAKER_00: thing will change all the time it's an ongoing puzzle to solve
25:09
SPEAKER_01: Yeah, absolutely.
25:10
Yeah,
25:10
I think,
25:11
you know,
25:11
they definitely have a ton of work in front of them.
25:15
SPEAKER_01: Also,
25:15
SPEAKER_01: you know,
25:16
it's part of their ethos to really like launch new lines of
25:19
businesses really
25:20
SPEAKER_01: quickly.
25:21
SPEAKER_01: And so,
25:21
SPEAKER_01: you know,
25:22
the ability for a team to go from zero to launch product within
25:27
the confines and
25:28
SPEAKER_01: the context of a large corporate entity,
25:30
SPEAKER_01: I think that's...
25:31
SPEAKER_01: you know, part of the DNA that's there.
25:33
SPEAKER_01: So as long as they're planting seeds,
25:35
SPEAKER_01: as the sort of like internal terminology is,
25:37
SPEAKER_01: I think that,
25:38
SPEAKER_01: you know,
25:38
SPEAKER_01: software developers will be in demand for quite amount of time.
25:43
SPEAKER_00: And I guess it's a good reminder that,
25:45
SPEAKER_00: you know,
25:45
there's every now and then we have the monoliths versus
25:46
microservices debate that
25:48
it sounds it kind of just makes sense for a startup to start
25:50
with a monolith.
25:51
Like you can always do what Amazon did and you have the
25:53
benefits of latency.
25:54
SPEAKER_00: Everything is in one place.
25:56
Like I'm sure there might be reason to start with microservices
25:59
to start with.
25:59
But if you're a small team,
26:01
I mean,
26:02
even today,
26:02
I don't think that argument changes,
26:04
right?
26:04
Like Amazon got really big wins by starting with a monolith
26:08
back in the day.
26:09
SPEAKER_01: Yeah, absolutely.
26:10
SPEAKER_01: I think it just makes a ton of sense to start with a monolith,
26:15
SPEAKER_01: wait till it breaks,
26:17
and then the part where it breaks is when you have like 50
26:20
developers working on
26:21
SPEAKER_01: the same piece of code.
26:23
SPEAKER_01: Once that sort of breaking point occurs,
26:25
Then you start to like try to figure out like how you can sort
26:28
of break things up.
26:29
SPEAKER_01: But starting with a microservice architecture,
26:31
SPEAKER_01: especially when you're small,
26:32
SPEAKER_01: like what a waste of time and energy.
26:35
SPEAKER_00: Totally.
26:35
SPEAKER_00: So you were a principal engineer on Amazon.
26:38
And apparently I learned that,
26:40
you know,
26:41
most companies,
26:42
they have different levels.
26:43
SPEAKER_00: And again,
26:43
SPEAKER_00: this principal engineer,
26:44
SPEAKER_00: some companies have like staff level,
26:46
SPEAKER_00: but it's usually like entry level,
26:48
SPEAKER_00: mid level,
26:48
SPEAKER_00: senior.
26:49
SPEAKER_00: And then you have staff or in the case of Amazon, it's principal.
26:53
I've learned that Amazon's principal level is both really hard
26:56
to get into compared
26:57
SPEAKER_00: to a lot of other companies,
26:59
SPEAKER_00: and it's pretty special in some ways,
27:00
SPEAKER_00: so we'll talk about that.
27:01
SPEAKER_00: But can you tell me, like, how is the career kind of...
27:06
SPEAKER_00: development?
27:06
Because most people imagine like,
27:07
oh,
27:08
it should be pretty straightforward.
27:09
SPEAKER_00: I spend like,
27:09
SPEAKER_00: I don't know,
27:10
SPEAKER_00: two years as a junior,
27:11
SPEAKER_00: two years as a mid roughly,
27:12
SPEAKER_00: and two years as a senior,
27:13
SPEAKER_00: then I get to principal.
27:14
SPEAKER_00: How does it actually work at Amazon?
27:16
SPEAKER_01: I think it's linear up until you hit principal, right?
27:19
So,
27:20
you know,
27:20
you join,
27:21
you're a junior developer,
27:22
you get promoted to mid.
27:23
SPEAKER_01: At mid, you know, you're starting to influence the team, but...
27:27
SPEAKER_01: But then you get to senior.
27:29
SPEAKER_01: And so now your expected impact is at the team level.
27:33
SPEAKER_01: And then there's this jump that you get to principal.
27:37
SPEAKER_01: And principal is L6?
27:39
SPEAKER_01: Principal is L7.
27:40
SPEAKER_01: L7, yes.
27:41
SPEAKER_01: Yeah.
27:41
And so I think you really have to start with like,
27:44
why is that jump so big?
27:46
Because I think at pretty much any other company,
27:48
it's just a linear progression.
27:50
SPEAKER_01: Like there's nothing necessarily special about staff.
27:53
SPEAKER_01: You know, you can just sort of
27:54
SPEAKER_01: Go to that level of senior staff and then principal.
27:57
But for some reason,
27:58
Amazon decided that they weren't going to have a staff level.
28:03
And so,
28:05
and I think they sort of like couched it around like having
28:07
high standards.
28:08
Basically,
28:09
to get from senior to principal,
28:11
you have to do like two and a half level jump.
28:14
SPEAKER_00: From L6, L7.
28:15
SPEAKER_00: Yes.
28:16
SPEAKER_00: Technically,
28:16
SPEAKER_00: it sounds like one level,
28:17
SPEAKER_00: but at some other companies,
28:19
SPEAKER_00: this might be like,
28:20
SPEAKER_00: you know,
28:21
SPEAKER_00: L8,
28:21
SPEAKER_00: L9 or L8 and a half.
28:23
SPEAKER_01: Yeah.
28:23
SPEAKER_01: Yeah.
28:24
SPEAKER_01: And so the hand wavy argument is like,
28:26
SPEAKER_01: hey,
28:26
SPEAKER_01: we have high standards and it means something to get to that level.
28:30
SPEAKER_01: It's like, fine.
28:31
SPEAKER_01: But I noticed that some of the best engineers that I'd ever worked with
28:36
were having such problems getting to principal engineer that
28:39
they ended up moving
28:40
to Facebook or to Meta or to all these other places where the
28:43
progression was just
28:45
SPEAKER_01: sane.
28:45
SPEAKER_01: So they got a staff or a senior staff.
28:47
Now they're senior staff and,
28:49
you know,
28:49
principal and distinguished engineer at other companies.
28:52
SPEAKER_01: And so because we had high standards, we actually had this brain drain.
28:57
SPEAKER_01: And it wasn't a brain drain at lower levels.
28:59
SPEAKER_01: It was the brain drain at sort of like the higher levels.
29:02
SPEAKER_02: Mm-hmm.
29:03
And it's just an example of something where it's just like,
29:06
why did you do that to yourself?
29:08
SPEAKER_01: And so that's the context for being a principal at Amazon.
29:12
SPEAKER_01: It's safe to say it's wicked hard to get internally, right?
29:16
SPEAKER_01: So I'm a colleague with Ethan Evans.
29:19
SPEAKER_01: And so we talk about what's the hardest promotion at Amazon.
29:24
SPEAKER_01: And I had made the argument that it was senior engineer to principal.
29:29
SPEAKER_01: And he's like, yeah, that's hard.
29:31
SPEAKER_01: Actually,
29:31
SPEAKER_01: the hardest one,
29:32
SPEAKER_01: Steve,
29:33
is VP to senior VP because there's only eight spots or 10 spots
29:38
for that and maybe
29:39
SPEAKER_01: 300 VPs that are all trying to get this.
29:42
SPEAKER_01: That's more of a supply and demand thing.
29:44
I will say that at Amazon,
29:46
there is gigantic demand for principal engineers.
29:50
SPEAKER_01: And so there are roles that have been open for years,
29:53
I think something on the order of like 13 months or 17 months
29:57
or something like
29:57
SPEAKER_01: that to get an external hire to join as a principal engineer.
30:02
SPEAKER_01: But that metric is only calculated when the role is filled.
30:05
SPEAKER_01: Yeah.
30:05
And so probably,
30:07
you know,
30:07
there are hundreds of principal engineer openings at Amazon.
30:11
SPEAKER_01: And there are thousands of senior engineers.
30:14
SPEAKER_01: Who desperately want to get there.
30:15
SPEAKER_01: And they're putting in the work.
30:17
SPEAKER_01: You know, and so there's this sort of like, there's this tension, right?
30:22
SPEAKER_01: And I don't think you see that at the lower levels.
30:25
SPEAKER_01: I don't think that that's happening at senior or mid or junior.
30:27
SPEAKER_01: And so like that incongruity, I think, is super interesting.
30:31
SPEAKER_01: Yeah.
30:32
SPEAKER_00: But once you do get to principal engineering,
30:34
one thing that I've never heard any other company have is there
30:37
is apparently a
30:38
SPEAKER_00: principal engineering community,
30:39
SPEAKER_00: which is,
30:40
SPEAKER_00: I've heard,
30:41
SPEAKER_00: again,
30:41
SPEAKER_00: from other people that it's tightly knit.
30:43
SPEAKER_00: It's actually special.
30:44
SPEAKER_00: It's actually just a really nice organization.
30:46
SPEAKER_00: Can you talk about that?
30:47
SPEAKER_00: So like,
30:47
SPEAKER_00: you know,
30:48
SPEAKER_00: once you got in there somehow,
30:50
SPEAKER_00: I don't know,
30:51
SPEAKER_00: was it Bloodsfoot and Sears at promotion?
30:53
SPEAKER_01: There is a community.
30:54
SPEAKER_01: I think it's actually really great.
30:55
SPEAKER_01: Yeah.
30:56
SPEAKER_01: My own history,
30:58
SPEAKER_01: you know,
30:58
I went from support engineer to senior engineer in like four
31:02
years at Amazon.
31:04
SPEAKER_01: But then from senior to principal, it took me eight years.
31:07
SPEAKER_01: And I got promoted in Q1 of 2020.
31:11
Turns out to be a consequential like year for in the industry
31:14
for the world.
31:15
SPEAKER_01: That was forceful remote work.
31:17
SPEAKER_01: Yeah.
31:18
And so,
31:18
you know,
31:19
I got promoted and everybody's like,
31:20
you know,
31:20
congratulations.
31:21
SPEAKER_01: Yeah.
31:22
They used to have like a principal engineer offsite where they
31:24
just flew everybody
31:25
SPEAKER_01: into Seattle or nearby and then to sort of like,
31:29
SPEAKER_01: you know,
31:30
SPEAKER_01: mingle and to talk to other folks.
31:32
SPEAKER_01: That stopped during the pandemic.
31:34
SPEAKER_01: And then,
31:35
SPEAKER_01: you know,
31:36
SPEAKER_01: by the time the pandemic restrictions started leaving,
31:39
SPEAKER_01: the population of principal engineers had essentially doubled.
31:42
SPEAKER_01: That's still to say,
31:43
SPEAKER_01: like,
31:43
SPEAKER_01: there are still hundreds and hundreds of openings for principal engineer.
31:47
SPEAKER_01: But then the,
31:48
SPEAKER_01: you know,
31:48
the sort of like offsite community shifted over to the senior
31:52
principals that I
31:52
SPEAKER_01: didn't have access to.
31:54
SPEAKER_01: But,
31:55
SPEAKER_01: you know,
31:55
SPEAKER_01: at the moment,
31:56
the manifestation of the principal engineering community is
32:00
essentially through the
32:00
SPEAKER_01: Slack channel.
32:02
SPEAKER_01: which is absolutely awesome.
32:04
SPEAKER_01: And then we had principal offsites for like our local organization.
32:09
SPEAKER_01: So like Amazon Music, Prime Video, Twitch, that sort of thing.
32:12
SPEAKER_01: Those meetups were amazing.
32:14
So the reason they were is because of this high standard that
32:19
Amazon had created.
32:20
And so what it meant is that everybody that was able to achieve
32:24
that overly high
32:25
SPEAKER_01: standard,
32:26
SPEAKER_01: there's something exceptional about them.
32:28
SPEAKER_01: There's there's,
32:29
SPEAKER_01: you know,
32:30
they're super deep in a particular technology or they were
32:33
associated with,
32:35
SPEAKER_01: you know,
32:36
SPEAKER_01: the growth of a really large line of business,
32:39
SPEAKER_01: either within Amazon or externally.
32:41
SPEAKER_01: They were essentially leaders within the industry.
32:45
SPEAKER_01: And you could just literally you could just scoop out.
32:50
SPEAKER_01: five people, and then put them into a room.
32:52
SPEAKER_01: And the conversation is just amazing, right?
32:55
SPEAKER_01: And I would sort of be like, I don't even belong here.
32:58
SPEAKER_01: Like, look at this guy.
32:59
SPEAKER_01: He wrote a book on a particular topic.
33:03
SPEAKER_01: And this guy, he is a luminary in a particular field.
33:10
And then this person just like is an amazing code machine and
33:13
can just write an
33:14
SPEAKER_01: entire application over a weekend.
33:17
SPEAKER_01: And then you're like,
33:18
SPEAKER_01: What am I doing here?
33:19
SPEAKER_00: I do wonder if that community might be coming back now.
33:23
SPEAKER_00: I know you've left, but now Amazon is now in-person.
33:26
Because it sounds like a lot of the benefit was the in-person
33:29
part as well.
33:30
SPEAKER_00: Because this is what I never heard.
33:31
Even before the pandemic,
33:33
I didn't hear other companies say,
33:35
for example,
33:35
Uber.
33:36
I've heard that the senior staff engineers do get together
33:38
every now and then.
33:39
SPEAKER_00: But it was very like roots.
33:41
SPEAKER_00: So it was bottoms up.
33:43
SPEAKER_00: But my understanding is that Amazon actually invested...
33:46
SPEAKER_00: not just some principal engineers saying,
33:48
SPEAKER_00: hey,
33:48
SPEAKER_00: let's get together,
33:49
SPEAKER_00: but also just kind of making sure that that group really had something.
33:55
SPEAKER_00: I think it's smart.
33:56
SPEAKER_00: I think more companies should do it, but I'm just not seeing it.
33:59
SPEAKER_01: The investment was...
34:02
SPEAKER_01: Also in terms of headcount.
34:04
SPEAKER_01: So there are program managers and product managers,
34:08
SPEAKER_01: essentially,
34:08
SPEAKER_01: that are bringing the folks together.
34:12
SPEAKER_01: There's a wonderful series.
34:15
It's called the Principles of Amazon series where principal
34:18
engineers will just do
34:20
SPEAKER_01: a presentation and it's recorded.
34:22
SPEAKER_01: That's been happening for years.
34:24
you know 20 years and you know we record everything that's
34:27
there but it takes work
34:28
to actually that's an internal series that and is that open to
34:33
like everyone at
34:34
amazon or it's for the principles oh it's it's open uh for
34:36
everybody at amazon to
34:38
consume to consume and then um you know there might be some
34:40
senior engineers and
34:41
stuff like that that that would make a presentation that's part
34:44
of their promotion
34:44
SPEAKER_01: packet it was be able to make an amazon-wide presentation
34:48
on a particular thing my point was though that that stuff
34:51
doesn't just happen on
34:52
its own yeah like you have to like you need a program manager
34:56
or multiple folks to
34:58
sort of like herd the cats and to like schedule the off sites
35:02
and to make sure that
35:04
the you know the slack channel doesn't go off the rails right
35:07
and it's still useful
35:07
and it's just not going to happen like grassroots with just
35:12
like throwing a bunch
35:12
SPEAKER_01: of people into a room
35:14
SPEAKER_00: This episode is brought to you by Augment Code.
35:17
SPEAKER_00: You're a professional software engineer, vibes will not cut it.
35:20
SPEAKER_00: Augment Code is the AI assistant built for real engineering teams.
35:24
SPEAKER_00: It ingests your entire repo,
35:25
SPEAKER_00: millions of lines,
35:26
SPEAKER_00: tens of thousands of files,
35:28
SPEAKER_00: so every suggestion lands in context and keeps you in flow.
35:32
SPEAKER_00: With Augment's new remote agent,
35:33
SPEAKER_00: cue a parallel task like buck fixes,
35:35
SPEAKER_00: features,
35:36
SPEAKER_00: and refactors,
35:37
SPEAKER_00: close your laptop,
35:38
SPEAKER_00: and return to ready-for-review poll requests.
35:40
SPEAKER_00: Where other tools stall, Augment Code sprints.
35:44
AugmentCode never trains or sells your code,
35:46
so your team's intellectual property stays yours.
35:49
SPEAKER_00: And you don't have to switch tooling.
35:50
SPEAKER_00: Keep using VS Code, JetBrains, Android Studio, or even Vim.
35:54
SPEAKER_00: Don't hire an AI for vibes.
35:56
SPEAKER_00: Get the agent that knows you and your code base best.
35:59
SPEAKER_00: Start your 14-day free trial at augmentcode.com slash pragmatic.
36:04
I think these are the things,
36:05
I mean,
36:07
we're now exposing a few of these things here and there.
36:09
SPEAKER_00: But some of these companies like Amazon is a great example.
36:12
SPEAKER_00: There's more to the eye than what meets the surface.
36:15
SPEAKER_00: So like once you're inside Amazon,
36:16
SPEAKER_00: for example,
36:17
SPEAKER_00: you now as an engineer,
36:18
SPEAKER_00: even if not a principal engineer,
36:19
SPEAKER_00: you now have access to the whole 20 years of principal presentations.
36:23
SPEAKER_00: Like when I joined Uber, I was amazed at how we had the RFCs available.
36:28
SPEAKER_00: Like I could read all historic ones.
36:30
SPEAKER_00: So I think there is,
36:31
SPEAKER_00: and every company has its own,
36:33
SPEAKER_00: of course,
36:33
SPEAKER_00: once you're in there,
36:34
SPEAKER_00: you have access to this knowledge base,
36:35
SPEAKER_00: which it will just never be published.
36:37
SPEAKER_00: It cannot because it has business sensitive things, et cetera.
36:40
SPEAKER_00: So I think as an engineer,
36:42
SPEAKER_00: you can just really just be a sponge when you join,
36:46
especially one of the companies that is known to be a bit more
36:48
open internally.
36:50
Amazon,
36:50
I think a really interesting one,
36:52
because externally,
36:52
it's very closed,
36:53
is my sense.
36:54
SPEAKER_00: They're very careful about what they share.
36:56
SPEAKER_00: For example,
36:56
SPEAKER_00: the postmortems for AWS is very few are published externally,
37:00
SPEAKER_00: but internally they're all there.
37:02
SPEAKER_00: As I understand there,
37:03
SPEAKER_00: as an engineer you can access,
37:04
SPEAKER_00: you can learn from them,
37:05
SPEAKER_00: like really cool real world learnings.
37:08
SPEAKER_01: Absolutely.
37:08
SPEAKER_01: You know,
37:09
SPEAKER_01: it is an open place internally and we are so selective about what we,
37:14
SPEAKER_01: I say we as though I still work there.
37:17
What they publish externally and,
37:19
you know,
37:20
the postmortems,
37:21
we call them COEs.
37:22
SPEAKER_01: It's a correction of error.
37:25
SPEAKER_01: It's,
37:26
SPEAKER_01: you know,
37:26
SPEAKER_01: it's this idea that,
37:27
SPEAKER_01: you know,
37:27
you have like holes in Swiss cheese and you have like a failure
37:32
requires...
37:33
SPEAKER_01: that there's a whole across layers.
37:36
SPEAKER_01: That's the best reading.
37:37
Like I would just subscribe to the email list where they were
37:40
published internally.
37:41
So you have this like stream of like of disasters that are
37:44
going on within the company.
37:46
SPEAKER_01: And you just,
37:46
SPEAKER_01: you know,
37:47
you grab some popcorn and you pop open one of these COEs and
37:51
you learn so much from
37:52
SPEAKER_01: that.
37:53
SPEAKER_01: And I think that that's part of the secret sauce.
37:55
SPEAKER_01: The idea,
37:56
SPEAKER_01: and I don't know if it's like this for 100% of them,
37:59
SPEAKER_01: is that it's a blameless culture sort of thing.
38:02
SPEAKER_01: And so to really screw up requires that multiple people drop the ball.
38:08
SPEAKER_01: And you learn so much from that sort of stuff.
38:12
SPEAKER_01: The brownouts,
38:15
these lessons that you would learn from trying to recover from
38:18
really large
38:18
SPEAKER_01: dependencies –
38:20
SPEAKER_01: Those things are immortalized inside some of these COEs.
38:22
SPEAKER_01: So there's some very famous outages that happened within Amazon.
38:27
SPEAKER_01: And, you know, there were an egg on our face.
38:29
SPEAKER_01: And we really, really learned those lessons through those postmortems.
38:33
SPEAKER_01: They're absolutely wonderful.
38:34
SPEAKER_00: As a principal engineer,
38:36
SPEAKER_00: so far we kind of glamorize the role saying,
38:38
SPEAKER_00: you know,
38:38
SPEAKER_00: it is hard to get into,
38:39
SPEAKER_00: but once you're there,
38:40
SPEAKER_00: you have the community,
38:41
SPEAKER_00: you do this really impactful work.
38:43
But one of the principal engineers at Amazon who's still there
38:46
called Bavik Kotari,
38:48
SPEAKER_00: he collected some things that are
38:51
SPEAKER_00: Maybe not as glamorous or more challenging about principal engineering.
38:54
SPEAKER_00: He had five of these things or five or six.
38:57
SPEAKER_00: I just want to go through with you and your take on this.
39:00
SPEAKER_00: So first he wrote,
39:01
SPEAKER_00: there is this paradox of belonging that you're part of all teams,
39:05
SPEAKER_00: yet you're part of none.
39:07
SPEAKER_00: What does that mean?
39:08
SPEAKER_01: Yeah, no, so Bavik was actually a peer of mine.
39:13
SPEAKER_01: We worked in Prime Video together.
39:14
SPEAKER_01: Oh, awesome.
39:16
SPEAKER_01: So he's an awesome dude.
39:17
Yeah,
39:18
there are all of these paradoxes,
39:19
and this paradox of belonging is...
39:24
SPEAKER_01: This is a really interesting one.
39:26
SPEAKER_01: You know, you work for the organization, right?
39:28
SPEAKER_01: You're working cross teams, right?
39:30
SPEAKER_01: So as a senior engineer,
39:32
SPEAKER_01: you're working on,
39:32
SPEAKER_01: you're embedded on a team and,
39:35
SPEAKER_01: you know,
39:35
SPEAKER_01: you own the team's architecture,
39:37
SPEAKER_01: the operations,
39:38
SPEAKER_01: you know,
39:39
SPEAKER_01: the software development lifecycle and the design.
39:42
But when you get to that next level where you're working across
39:45
teams,
39:45
right?
39:47
SPEAKER_01: You kind of operate in this weird layer where,
39:50
SPEAKER_01: you know,
39:50
SPEAKER_01: you're not on pager duty for a particular team.
39:55
SPEAKER_01: You have visibility across all of these teams that are there.
39:58
You're helping to guide and make decisions,
40:01
but you're literally not on the ground floor.
40:05
SPEAKER_01: And so when you work with a particular team,
40:08
you might call the senior engineers or the mid-level engineers
40:11
in and be like,
40:11
SPEAKER_01: hey,
40:12
SPEAKER_01: let's whiteboard some stuff.
40:13
SPEAKER_01: Let's try to figure out what's going on.
40:15
SPEAKER_01: You're not on the team.
40:16
SPEAKER_01: You're kind of this advisor that's sort of coming in
40:20
But then maybe a director or a VP would call you in and say,
40:24
hey,
40:24
what do I own?
40:25
SPEAKER_01: What's going on?
40:26
SPEAKER_01: Explain to me this outage or tell me why we can't build this thing.
40:30
And then you're trying to whiteboard the architecture and the
40:33
system and you're
40:34
SPEAKER_01: trying to say,
40:34
SPEAKER_01: hey...
40:36
SPEAKER_01: You know, this is what's going on on the ground floor.
40:40
SPEAKER_01: But you weren't, you know, you weren't part of that team.
40:42
SPEAKER_01: So you're just sort of operating in this this sort of strata where,
40:46
SPEAKER_01: you know,
40:46
SPEAKER_01: you don't really belong on a team.
40:48
SPEAKER_01: You know, I'm a I'm an immigrant.
40:50
SPEAKER_01: I think you are as well.
40:52
SPEAKER_01: And, you know, my parents came from from Asia.
40:56
SPEAKER_01: I'm not Asian, right?
40:57
SPEAKER_01: So when I go back to Asia, I'm definitely from the US.
41:00
And then growing up in this country is just like,
41:02
you know,
41:03
I'm not quite an American,
41:06
right?
41:07
SPEAKER_01: And so you sort of operate in this sort of,
41:10
SPEAKER_01: you know,
41:10
area in the gaps where your identity is really defined by not
41:15
being squarely in one
41:17
SPEAKER_01: of these predefined categories.
41:18
SPEAKER_01: And so it's very similar to that.
41:20
SPEAKER_01: As a principal engineer, you're not on the ground floor.
41:22
SPEAKER_01: You're not checking in.
41:23
You will check in code,
41:24
but you're not necessarily part of that team embedded on that
41:28
team.
41:28
SPEAKER_00: And even if you are for a short time, it's usually a short time.
41:31
SPEAKER_00: And like tomorrow,
41:32
SPEAKER_00: the director will call you up and say,
41:33
SPEAKER_00: like,
41:33
SPEAKER_00: hey,
41:34
SPEAKER_00: Steve,
41:34
SPEAKER_00: we need you on this other team.
41:35
SPEAKER_00: They're in trouble.
41:36
SPEAKER_00: Move over.
41:37
SPEAKER_01: Yeah, and you parachute in, and then they're like, oh, who's this guy?
41:41
SPEAKER_01: And then your director is like, what's going on?
41:45
SPEAKER_01: What happened during this outage?
41:48
SPEAKER_01: Why is the press writing about us?
41:51
SPEAKER_01: And then you're like, well, here's what's happening on the ground floor.
41:53
SPEAKER_01: But you're not really embedded on that team.
41:56
SPEAKER_00: Which leads us to the next paradox that Bavik said.
41:59
SPEAKER_00: He lists a few of the paradoxes, which is the freedom of responsibility.
42:02
And he writes that you enjoy significant autonomy in being able
42:05
to choose what you work on.
42:07
However,
42:07
there's an implicit expectation and accountability for
42:10
resounding impact.
42:11
SPEAKER_00: Yeah.
42:12
SPEAKER_01: So, you know, I reported to a VP right before I left the company.
42:18
SPEAKER_01: So they were your manager, basically.
42:19
SPEAKER_01: Yeah, my manager was a VP.
42:21
SPEAKER_00: Oh, wow.
42:22
SPEAKER_00: I don't hear many companies having engineers report into VPs.
42:29
SPEAKER_02: Yeah.
42:29
SPEAKER_00: That doesn't seem very standard.
42:31
SPEAKER_01: You know,
42:31
SPEAKER_01: and so the org that he owned,
42:33
SPEAKER_01: you know,
42:33
I considered myself the tech advisor for that organization was
42:37
about 450 people.
42:39
SPEAKER_01: 450 software developers.
42:41
SPEAKER_01: And what did our one-on-ones consist of, right?
42:45
SPEAKER_01: Like when I would have our one-on-one,
42:47
SPEAKER_01: it wasn't like,
42:48
SPEAKER_01: hey,
42:48
SPEAKER_01: here's,
42:49
SPEAKER_01: you know,
42:50
SPEAKER_01: he didn't assign me work.
42:51
SPEAKER_01: He wasn't like, hey, I need you to build this thing.
42:54
SPEAKER_01: I need you to design this thing.
42:56
The context that he said was basically like,
42:58
here's a direction that you need to go.
43:02
SPEAKER_01: And the way that you can achieve that type of impact was up to me.
43:08
So he might say something like,
43:09
hey,
43:10
availability is so important for live sports.
43:15
SPEAKER_01: We just signed billion-dollar contracts with these sports leagues.
43:19
SPEAKER_01: And so we need to increase our availability posture.
43:23
SPEAKER_01: And then I would be like, okay.
43:25
SPEAKER_01: Yeah.
43:26
SPEAKER_01: And then I would go away and it would come back and I would be like,
43:30
SPEAKER_01: you know,
43:30
SPEAKER_01: here's what I'm working on.
43:32
SPEAKER_01: Right.
43:33
SPEAKER_01: Like that type of.
43:35
Dynamic does not exist at the senior engineer below level where
43:39
you're basically
43:40
SPEAKER_00: telling your boss what's happening.
43:42
I was about to say that when you said my manager one-on-ones,
43:45
he didn't tell me what to do.
43:46
SPEAKER_00: I'm like, most engineers would be like, sign me up.
43:48
SPEAKER_00: Like, I don't want, you know, we all hate micromanagement.
43:51
SPEAKER_00: But now when you're telling me like,
43:52
SPEAKER_00: he would say like,
43:53
SPEAKER_00: oh,
43:53
SPEAKER_00: so we just signed a billion dollar contract.
43:55
SPEAKER_00: Availability is important.
43:56
SPEAKER_00: And then stops talking.
43:57
SPEAKER_00: I'm like, that sounds uncomfortable.
44:00
SPEAKER_00: Yeah.
44:01
SPEAKER_00: And basically,
44:02
SPEAKER_00: you're kind of expected a little bit to understand what he's expecting,
44:06
SPEAKER_00: even though he doesn't know.
44:06
SPEAKER_00: And I'm assuming there's two ways of going, right?
44:09
SPEAKER_00: You go back on the next one-on-one and you say something and he's like,
44:13
SPEAKER_00: Steve,
44:14
SPEAKER_00: you're a principal engineer.
44:14
SPEAKER_00: This is not what I expect of you.
44:16
SPEAKER_00: And you don't want that.
44:17
SPEAKER_00: Whereas this,
44:18
SPEAKER_00: you know,
44:19
SPEAKER_00: if you bring back the right things,
44:20
it sounds like you really need to up level and like
44:22
understanding how these people
44:24
SPEAKER_00: think.
44:25
SPEAKER_01: Absolutely.
44:25
SPEAKER_01: And so he's, you know, he's accountable to his boss as well.
44:29
SPEAKER_01: And, you know, don't get me wrong.
44:30
SPEAKER_01: I didn't, you know, I had a I owned aspects of availability.
44:34
You know,
44:34
there's a multi thousand person organization at Prime Video
44:38
doing this stuff.
44:39
SPEAKER_01: But we own the live sports aspect of this.
44:42
SPEAKER_01: And,
44:42
SPEAKER_01: you know,
44:42
SPEAKER_01: there are playback teams,
44:44
SPEAKER_01: there are,
44:44
SPEAKER_01: you know,
44:44
SPEAKER_01: recommendation teams,
44:46
SPEAKER_01: there are,
44:46
SPEAKER_01: you know,
44:46
there's so many different teams that are there that had to
44:49
really step up and make
44:51
SPEAKER_01: sure that availability was good.
44:53
But he would say something like,
44:55
hey,
44:55
you know,
44:56
what is our availability posture for...
44:59
SPEAKER_01: certain aspects, and I would have to go and figure it out.
45:03
SPEAKER_01: What are we measuring?
45:04
SPEAKER_01: What are we not measuring?
45:05
There's a deadline for the start of a season where we're
45:09
expecting millions and
45:10
SPEAKER_01: millions of concurrent to come in.
45:13
SPEAKER_01: What can we do between now and then?
45:16
SPEAKER_01: And then if we do write some software,
45:18
Like what is the highest leverage piece of software that we
45:22
could create that would
45:23
SPEAKER_01: increase our availability posture?
45:24
And so the way that I sort of describe it to people is you are
45:28
assigned not a
45:30
SPEAKER_01: problem,
45:31
SPEAKER_01: not even a problem space.
45:32
SPEAKER_01: You're assigned a direction.
45:33
SPEAKER_01: You can solve the problem with code.
45:34
SPEAKER_01: You can solve the problem with system design and architecture.
45:38
SPEAKER_01: But you could also solve the problem, say, by, you know, I don't know.
45:41
SPEAKER_01: Hey, maybe there's some off-the-shelf software we should purchase.
45:44
Maybe there's a dev team that we should start to spin up right
45:48
now whose job it is
45:50
SPEAKER_01: to do this particular thing.
45:52
Maybe we've identified a piece of software and it's already
45:55
been scoped that this
45:57
SPEAKER_01: team needs to go and build.
45:59
SPEAKER_01: But...
46:00
SPEAKER_01: it's not a priority for them.
46:02
Now we need to go and figure out like,
46:03
you know,
46:03
how we can get them to do it.
46:05
SPEAKER_01: Can we shuffle around resources?
46:07
SPEAKER_01: That sort of thing.
46:07
SPEAKER_01: And so the way I describe it is like,
46:09
there's so many more things on the menu that you can use to
46:12
solve the problem.
46:14
SPEAKER_01: And I don't think people recognize that.
46:16
SPEAKER_01: They think that it's just,
46:17
SPEAKER_01: oh,
46:17
SPEAKER_01: when you're a principal,
46:18
SPEAKER_01: like you just like code a lot and it's just really complicated.
46:22
SPEAKER_01: Or do more meetings.
46:23
SPEAKER_01: You know, that's what often happens.
46:24
SPEAKER_01: I mean, at the end of the day, like, don't get me wrong.
46:26
SPEAKER_00: There's a ton of meetings that go on.
46:28
SPEAKER_00: Yeah, yeah.
46:29
SPEAKER_00: But this is...
46:30
I think it's good to shine a light because I also feel like
46:32
once it sounds like a
46:34
SPEAKER_00: big change,
46:35
SPEAKER_00: but I also kind of feel if you get good at this,
46:37
SPEAKER_00: you might not really want to go back to having a manager who's like,
46:41
SPEAKER_00: all right,
46:42
SPEAKER_00: here's a project.
46:43
SPEAKER_00: We need to solve, scope it out, which you can do, right?
46:46
SPEAKER_00: Yeah.
46:46
SPEAKER_00: That's cool.
46:47
SPEAKER_00: And now the next challenge that Bavic said was,
46:50
SPEAKER_00: this all sounds great,
46:51
SPEAKER_00: but there's apparently a bandwidth challenge.
46:52
So it's easy to become this social resource where people just
46:57
pull you into
46:57
SPEAKER_00: everything and you're breathing...
46:59
SPEAKER_01: Yeah,
47:00
SPEAKER_01: no,
47:00
SPEAKER_01: you know,
47:00
SPEAKER_01: I think I wish I had taken a screenshot,
47:03
SPEAKER_01: but,
47:03
SPEAKER_01: you know,
47:03
SPEAKER_01: I have my Outlook calendar.
47:05
SPEAKER_01: So it's my schedule.
47:06
SPEAKER_01: My day looked like most people's week.
47:10
SPEAKER_01: So it looked like somebody had just like blew up a Tetris factory.
47:13
Like there was like I would have triple or quadruple booked on
47:17
a Monday all through the day.
47:19
SPEAKER_01: So you would have the manager calendar as an IC.
47:21
SPEAKER_01: Yeah.
47:22
And it's absolutely crazy because,
47:25
you know,
47:25
for that large org that I was supporting,
47:28
SPEAKER_01: Everybody just added me as optional or,
47:31
SPEAKER_01: or they might try to say like,
47:32
SPEAKER_01: no,
47:32
SPEAKER_01: you're actually required for all of these meetings.
47:34
SPEAKER_01: But when you have,
47:35
SPEAKER_01: you have a triple booked calendar and you're required for this stuff,
47:38
SPEAKER_01: you just learn that you're going to have to disappoint a lot of people.
47:41
SPEAKER_00: Yeah.
47:42
SPEAKER_01: And so it's, it's this sort of like, uh,
47:45
this thing where it's almost easier to say no now that you're
47:49
obscenely overbooked
47:51
SPEAKER_01: versus when you're a senior engineer,
47:52
SPEAKER_01: you're like,
47:53
SPEAKER_01: I don't have time to write code,
47:55
SPEAKER_01: but there's just barely enough time in between the cracks.
47:59
SPEAKER_01: And so I think that it's almost like when your schedule breaks,
48:04
that's when you are finally freed because you know that you can
48:07
sort of say no to
48:07
SPEAKER_01: stuff.
48:08
SPEAKER_01: But ultimately,
48:09
if I just went to all of the meetings that everybody said that
48:11
I would have to go
48:12
SPEAKER_01: to,
48:12
I would be a professional meeting attender and I would
48:15
literally have no time to do
48:16
SPEAKER_00: the work.
48:17
And then Bavik follows up on this next challenge,
48:20
which is being truly present.
48:21
And he writes,
48:22
I think it's almost like,
48:23
you know,
48:24
he was sitting next to you.
48:25
You find yourself physically present in one meeting while your
48:27
mind is already
48:28
SPEAKER_00: racing against next three.
48:29
you know it's it's a it's a really big challenge you know i i
48:33
pride myself on being
48:34
a good communicator and being present and when there are there
48:38
are 20 things that
48:39
are going on in the air or 100 things that are going on it's
48:42
just really really
48:43
SPEAKER_01: difficult to to say single threaded um and
48:48
SPEAKER_01: What I ended up having to do is to sort of say like,
48:51
SPEAKER_01: OK,
48:51
SPEAKER_01: I could do all of these things and they would be really impactful.
48:55
But I just had to aggressively prioritize and say,
48:57
you know,
48:58
for the availability.
48:59
SPEAKER_01: I'm just looking at availability.
49:00
SPEAKER_01: There's all these other fires that are going on, which is disappointing.
49:05
SPEAKER_01: Because there's so many things that, you know, you could be focusing on.
49:09
SPEAKER_01: It's super difficult.
49:11
And so,
49:11
you know,
49:12
I work with a lot of people to try to get them to the next
49:14
level.
49:14
SPEAKER_01: And they say, Steve, oh, I'm completely overwhelmed.
49:16
SPEAKER_01: There are, like, 20 things that are going on.
49:20
SPEAKER_01: And I tell them, like, do you think it gets easier?
49:22
SPEAKER_01: Yeah.
49:23
When you get higher level,
49:24
there's just going to be more and more things on your plate.
49:27
SPEAKER_01: Why wait until you burn out or you break?
49:30
SPEAKER_01: You can just start implementing these things now.
49:31
SPEAKER_01: So every high level tech I see,
49:33
SPEAKER_01: I know,
49:33
SPEAKER_01: and managers included,
49:35
they have a wonderful system in order to like isolate signal
49:40
and then cut out the
49:40
SPEAKER_01: noise.
49:42
SPEAKER_01: And if you don't have that, you literally won't survive.
49:44
But it just at the principal level and above,
49:46
it's just amplified that much more.
49:48
I'm getting a sense that a lot of the work as you do as a
49:52
principal engineer,
49:52
SPEAKER_00: I mean,
49:53
there's huge amounts of software engineering and you need to be
49:57
just really good at
49:59
SPEAKER_00: building resilient systems,
50:01
SPEAKER_00: learning about new technologies.
50:03
SPEAKER_00: For example, today, I'm assuming Kurov is a principal engineer.
50:06
SPEAKER_00: At Amazon,
50:06
SPEAKER_00: they're expected to just know everything about LLMs,
50:09
SPEAKER_00: trade-offs,
50:10
SPEAKER_00: characteristics,
50:11
SPEAKER_00: et cetera,
50:11
SPEAKER_00: because they're...
50:12
SPEAKER_00: Anyway,
50:13
SPEAKER_00: but you also need to just become...
50:15
SPEAKER_00: do the skills that managers have,
50:17
SPEAKER_00: which is managing your time,
50:19
SPEAKER_00: changing contacts,
50:21
SPEAKER_00: figuring out how to get that focus time.
50:23
SPEAKER_00: Contrary to popular belief, managers actually need focus time.
50:27
SPEAKER_00: So I will always try to carve out some time.
50:30
SPEAKER_00: But you're now doing it while your title is not manager.
50:33
SPEAKER_00: But actually,
50:34
SPEAKER_00: it feels like you combine a manager,
50:36
SPEAKER_00: a lot of manager responsibilities,
50:38
SPEAKER_00: and a lot of,
50:38
SPEAKER_00: you know,
50:39
SPEAKER_00: like,
50:39
SPEAKER_00: experienced engineer.
50:40
SPEAKER_00: And boom, you get the principal engineer role.
50:42
Oh,
50:42
the only upside is,
50:43
like,
50:43
you don't need to do performance reviews for people.
50:45
SPEAKER_00: Congratulations.
50:45
SPEAKER_00: You saved a little bit of that.
50:46
Well,
50:47
actually,
50:48
during performance review season,
50:50
they pull the principal engineers in.
50:51
Because if you're,
50:53
so,
50:53
you know,
50:54
if you're stack ranking people,
50:55
okay,
50:56
cool.
50:56
SPEAKER_01: Well, we'll need to take a look at their performance and tech.
50:59
SPEAKER_01: So I reported to a VP,
51:00
SPEAKER_01: you know,
51:01
SPEAKER_01: one of my peers was a director,
51:02
SPEAKER_01: and he was basically like,
51:03
SPEAKER_01: hey,
51:04
SPEAKER_01: Steve,
51:04
SPEAKER_01: I would like you to show up to my performance review for my entire org of
51:08
SPEAKER_01: 100-something people.
51:09
SPEAKER_01: And I'm like, I can't do that for you and for everybody else.
51:13
SPEAKER_00: Okay,
51:13
SPEAKER_00: so now it makes sense why,
51:15
SPEAKER_00: as a principal engineer,
51:16
SPEAKER_00: your compensation package will be similar to,
51:18
SPEAKER_00: like,
51:19
SPEAKER_00: is it a senior engineering manager or something like that?
51:21
SPEAKER_00: Around that.
51:22
SPEAKER_00: Around that, but basically, like, the job is...
51:25
has a lot of overlaps okay the benefit is you're not the one
51:28
delivering the
51:29
performance reviews of the direct report but you're doing
51:32
almost everything else or
51:33
in terms of the effort i'm talking about yeah okay so having
51:38
been a principal
51:39
engineer for four years what are the good things that you
51:42
really really liked about
51:43
amazon specifically amazon's principal engineer role and what
51:46
are some of the you
51:48
SPEAKER_00: know not so good or it could have been better things
51:51
SPEAKER_01: I mean,
51:52
the great parts are you get visibility that you just couldn't
51:56
possibly have at the
51:57
SPEAKER_01: team level,
51:58
SPEAKER_01: you know,
51:59
SPEAKER_01: within a large organization like Prime Video or wherever you're at.
52:02
There are many thousands of people that are working within that
52:06
organization doing
52:07
SPEAKER_01: so many things.
52:08
SPEAKER_01: Right.
52:09
SPEAKER_01: And typically the performance of these people is really high.
52:12
SPEAKER_01: There's so many different directions that are going on.
52:14
SPEAKER_01: And so to survive,
52:16
SPEAKER_01: you kind of have to look inward and you say,
52:17
SPEAKER_01: okay,
52:18
SPEAKER_01: well,
52:18
SPEAKER_01: here's my service boundary.
52:19
SPEAKER_01: Here's all the software I own.
52:21
SPEAKER_01: I'm going to own everything within the sphere of ownership.
52:24
Because you've built this wall up,
52:26
you tend not to be able to see like that broader picture.
52:28
SPEAKER_01: Yeah.
52:29
SPEAKER_01: And so as a principal engineer,
52:31
I think it's really awesome to be able to sort of like spelunk
52:34
and be able to go to
52:35
SPEAKER_01: different teams and sort of see that broader picture.
52:38
SPEAKER_01: Yeah.
52:39
I just don't see a way that you would be able to get that type
52:43
of visibility that's
52:43
SPEAKER_01: super interesting at a lower level.
52:47
SPEAKER_01: You know,
52:47
SPEAKER_01: I think the other thing is like,
52:48
SPEAKER_01: you know,
52:49
SPEAKER_01: whether it's warranted or not,
52:51
SPEAKER_01: you do get some amount of status when you go to a meeting.
52:53
SPEAKER_01: People just listen to you.
52:55
SPEAKER_01: They listen to your harebrained ideas.
52:57
And it's kind of nice because you don't necessarily have to
53:00
like prove yourself
53:01
SPEAKER_01: over and over again.
53:03
SPEAKER_00: This is a bit less like professional,
53:05
SPEAKER_00: like not fights,
53:06
SPEAKER_00: but just establishing that you know what you're talking about.
53:09
SPEAKER_01: Yeah, yeah.
53:11
SPEAKER_01: Now,
53:11
SPEAKER_01: the bad things are,
53:13
SPEAKER_01: you know,
53:14
there's a lot of folks that are really good in tech and being
53:17
really effective as a
53:18
SPEAKER_01: principal engineer.
53:19
SPEAKER_01: But then they also, you know, myself included, they're like, OK, cool.
53:23
SPEAKER_01: Well, that sort of makes me an expert in pretty much everything.
53:26
SPEAKER_01: And so you would get these principal engineers together.
53:28
SPEAKER_01: We had a weekly meeting.
53:30
SPEAKER_01: And so it'd be like,
53:31
SPEAKER_01: okay,
53:32
if you wanted to talk about establishing a constitution for a
53:35
small island nation,
53:36
SPEAKER_01: all of a sudden they would just be like,
53:38
SPEAKER_01: well,
53:38
SPEAKER_01: sure,
53:38
SPEAKER_01: the main considerations is nobody has a background in government policy.
53:42
SPEAKER_01: Yeah.
53:43
But all of a sudden,
53:44
just because you're sort of trained to do so,
53:46
you start to pitch in.
53:48
SPEAKER_01: You're like,
53:48
SPEAKER_01: well,
53:48
SPEAKER_01: actually,
53:49
maybe we should have two branches of government or three
53:51
branches of government.
53:52
SPEAKER_01: And it just sounds like we would know what we're doing, but we don't.
53:57
SPEAKER_01: And so there's this trap.
53:59
SPEAKER_01: And again,
53:59
I've fallen into it many times where you actually think you're
54:02
an expert in one
54:04
SPEAKER_01: thing.
54:05
SPEAKER_01: But you're actually not.
54:06
SPEAKER_01: Right.
54:06
SPEAKER_01: And so, you know, take LLMs.
54:08
SPEAKER_01: There's a ton of folks that understand AI.
54:11
I left before it was sort of like allowed to use internally,
54:14
but I think you can use it now.
54:17
SPEAKER_01: I'm not an expert in LLMs at all.
54:20
But I do think that the expectation would be that you
54:25
understand how they work.
54:27
SPEAKER_01: But then the expectation is also like, hey, what should our policy be?
54:31
SPEAKER_01: How should we be thinking about this stuff?
54:33
SPEAKER_01: And I think that's fine for mature technologies, potentially.
54:38
SPEAKER_01: You can ramp yourself up for it.
54:39
SPEAKER_01: But that particular landscape is changing so quickly.
54:43
SPEAKER_01: I think there's this sort of trap where you speak as an authority,
54:48
SPEAKER_01: even though you haven't had the requisite time to ramp up on something.
54:51
SPEAKER_00: And you've been there for 17 years at Amazon.
54:54
SPEAKER_00: What are your favorite parts of the culture?
54:57
SPEAKER_00: There's a lot of things that there's a values that we all know,
55:01
SPEAKER_00: like the frugality,
55:02
SPEAKER_00: customer obsession.
55:04
What were the things that you found to be the most interesting
55:08
or the ones that had
55:09
SPEAKER_00: a lasting impact?
55:10
SPEAKER_00: And how did they change?
55:11
SPEAKER_00: How did Amazon change over 17 years?
55:13
SPEAKER_00: They must have changed.
55:14
SPEAKER_01: No, I think the things I missed the most and the secret sauce.
55:19
SPEAKER_01: Yeah,
55:19
SPEAKER_01: the leadership principles are good,
55:21
SPEAKER_01: but I think the actual secret sauce there is principled thinking.
55:25
SPEAKER_02: Mm-hmm.
55:26
SPEAKER_01: Right.
55:26
SPEAKER_01: And so, yeah.
55:27
So,
55:28
you know,
55:28
there's,
55:28
you know,
55:29
invent and simplify and bias for action and all of this stuff.
55:33
SPEAKER_01: But like,
55:33
SPEAKER_01: ultimately,
55:34
the thing that is amazing about those leadership principles
55:39
aren't the specific
55:40
SPEAKER_01: stances that they took.
55:41
SPEAKER_01: So they decided that customer obsession is a big deal.
55:43
SPEAKER_01: They decided that bias for action is a big deal.
55:45
SPEAKER_01: Yeah.
55:46
SPEAKER_01: all of these things.
55:47
SPEAKER_01: But really,
55:48
SPEAKER_01: if you look at a meta level,
55:49
SPEAKER_01: you'd be like,
55:50
SPEAKER_01: oh,
55:50
SPEAKER_01: these guys have principles that they won't budge on.
55:53
SPEAKER_01: I sort of think about it in terms of math and axioms.
55:56
SPEAKER_01: Like, you just take certain things to be true,
55:59
SPEAKER_01: you know,
56:00
SPEAKER_01: two lines that are parallel,
56:01
SPEAKER_01: if you extend them out to infinity,
56:03
SPEAKER_01: won't touch them and won't touch with each other.
56:05
SPEAKER_01: Yeah, you assume that's true.
56:07
SPEAKER_01: Yeah, you don't prove that.
56:09
SPEAKER_01: It's an axiom.
56:09
And then based off of that,
56:11
you're able to build a system of mathematics,
56:13
right?
56:14
And so it's the same thing with the corporate leadership
56:17
principles at Amazon.
56:18
SPEAKER_01: They basically said, okay, we are going to fix these things to be true.
56:23
SPEAKER_01: There are 16 or 12 or I don't know.
56:24
SPEAKER_01: They just sort of bolted some on.
56:26
SPEAKER_01: They were 14 and now they're 16.
56:27
SPEAKER_00: Yeah.
56:30
SPEAKER_01: But there are like four or five that are just really core to Amazon.
56:35
SPEAKER_01: And we just fix those things to be true.
56:37
SPEAKER_01: Which ones were the ones that you felt were the most present?
56:40
SPEAKER_01: Customer obsession.
56:42
SPEAKER_01: We are absolutely customer obsessed.
56:43
SPEAKER_01: We'll just burn money to delight a customer.
56:46
SPEAKER_01: You can be in a meeting with a VP as an intern,
56:49
SPEAKER_01: and you say,
56:49
SPEAKER_01: hey,
56:50
SPEAKER_01: that's a bad customer experience.
56:52
SPEAKER_01: It would be like a needle coming off a record.
56:54
SPEAKER_01: It would just be like, what?
56:55
SPEAKER_01: What are you talking about immediately?
56:57
SPEAKER_01: Bias for action.
56:58
SPEAKER_01: So just get some stuff done.
57:01
SPEAKER_01: Stop asking for permission.
57:02
SPEAKER_01: Just go and do it.
57:03
SPEAKER_01: Ownership.
57:04
SPEAKER_01: It's just like you own your software.
57:06
SPEAKER_01: You do the operations.
57:09
SPEAKER_01: You own the bug count, all of this stuff.
57:12
SPEAKER_01: So those are the ones that are fixed.
57:15
SPEAKER_01: And then you start layering things on top of it.
57:18
SPEAKER_01: And I think it's really great.
57:19
SPEAKER_01: But,
57:19
SPEAKER_01: you know,
57:19
SPEAKER_01: you could take Amazon and you could have like the,
57:22
SPEAKER_01: you know,
57:22
SPEAKER_01: evil goatee version of Amazon,
57:24
SPEAKER_01: which is just sort of the opposite of those things.
57:26
SPEAKER_01: And that would still be a really valid and awesome company.
57:29
SPEAKER_01: So you could say, okay, well, it's the opposite of customer obsession.
57:32
SPEAKER_01: It's not customer obsession or not being customer obsessed.
57:36
SPEAKER_01: I think it's, you know, like being about your staff.
57:39
SPEAKER_01: Yeah.
57:39
SPEAKER_01: Which is Google.
57:40
SPEAKER_02: Yeah.
57:42
It could be like,
57:42
hey,
57:43
we really care about our people above everything else.
57:45
SPEAKER_01: Or it could be, you know, let's not mince around it.
57:48
SPEAKER_01: We care about top line or bottom line revenue.
57:50
SPEAKER_01: Yeah, that's totally valid.
57:52
SPEAKER_01: Right.
57:53
SPEAKER_01: And then you could just fix that.
57:54
You wouldn't you can't prove that,
57:56
you know,
57:56
being,
57:57
you know,
57:57
staff focused is a bad thing.
57:59
SPEAKER_01: You just build that.
58:00
And then,
58:00
you know,
58:00
a certain set of things will happen like great things are going
58:03
to happen.
58:04
SPEAKER_01: And then like not so great things are going to happen.
58:06
SPEAKER_01: Those not great things that happen,
58:08
SPEAKER_01: you can try to mitigate them,
58:09
but you can't fix them because you have started with this
58:12
principled approach to
58:14
SPEAKER_00: everything.
58:14
SPEAKER_00: Yeah.
58:15
SPEAKER_00: Yeah.
58:15
SPEAKER_00: It all goes like everything has.
58:18
SPEAKER_00: Yeah.
58:18
SPEAKER_00: I see what you mean,
58:19
SPEAKER_00: but I think what you're saying is like,
58:21
SPEAKER_00: it might be less about what the specific principles are.
58:24
SPEAKER_00: I mean,
58:25
SPEAKER_00: Amazon has theirs and we know about them,
58:26
SPEAKER_00: but it's just sticking to them and not keeping wiggling.
58:29
SPEAKER_00: Because if you keep wiggling, it's like,
58:31
SPEAKER_00: What was the point, right?
58:32
SPEAKER_00: Then you're going to have a really kind of mediocre,
58:35
SPEAKER_00: truly not standout company,
58:37
SPEAKER_01: whatever you do.
58:37
SPEAKER_01: What does it actually mean to be principled and to not bend?
58:41
SPEAKER_01: It could be really easy to do so.
58:43
SPEAKER_01: So that's an amazing secret sauce of Amazon's.
58:46
People look at the leadership principle and I'm like,
58:47
no,
58:47
it's principled thinking.
58:49
SPEAKER_00: A lot of this,
58:50
SPEAKER_00: honestly,
58:51
SPEAKER_00: from what I understand,
58:51
SPEAKER_00: talking to you earlier and some other people,
58:53
SPEAKER_00: a lot of it probably comes from Jeff Bezos.
58:56
SPEAKER_00: Being from the top down,
58:57
SPEAKER_00: being very principled on not giving,
58:59
SPEAKER_00: not saying we will do this,
59:01
SPEAKER_00: whatever it takes.
59:03
Sounds like it was customer obsession initially and then some
59:05
other things.
59:06
SPEAKER_01: Yeah.
59:06
SPEAKER_01: Yeah, absolutely.
59:07
SPEAKER_01: And he's he was he was an absolute genius when it came through.
59:11
SPEAKER_01: So I'm a you know, I'm a Jeff Bezos fanboy for sure.
59:15
SPEAKER_01: Like it just it just worked.
59:17
SPEAKER_01: Another thing that's Amazon secret sauce is just the writing culture.
59:23
And so I spent on the order of like one to four hours every day
59:27
reading while I was
59:28
SPEAKER_01: a principal engineer.
59:30
SPEAKER_01: And we had a standard format.
59:32
SPEAKER_01: It was a six-page memo.
59:35
SPEAKER_01: And, you know, that would be our business strategy.
59:38
SPEAKER_01: That would be a system design.
59:40
SPEAKER_01: That would be, you know, what we call the PR FAQ.
59:44
So a press release and frequently asked questions for like a
59:46
new line of business
59:48
SPEAKER_01: or a new initiative.
59:49
SPEAKER_01: And everybody was sort of constrained to the six page format.
59:54
And everybody just produces documents in that format for
59:57
whatever they need to do.
59:58
SPEAKER_01: And so when I would try to get up to speed on a particular thing,
1:00:01
SPEAKER_01: I would just be like,
1:00:02
SPEAKER_01: give me your six pages.
1:00:04
SPEAKER_01: Give me all your documents.
1:00:05
And I just got really,
1:00:06
really good at just reading these documents to get up to speed.
1:00:11
SPEAKER_01: Which is a self-fulfilling and virtuous cycle,
1:00:13
SPEAKER_01: which is just like,
1:00:14
SPEAKER_01: OK,
1:00:14
SPEAKER_01: well,
1:00:14
SPEAKER_01: now I need to express myself.
1:00:16
And so I will write a six pager and that will set the context
1:00:19
for whatever we're working on.
1:00:21
SPEAKER_01: We'd go to a meeting.
1:00:22
SPEAKER_01: You would read the six pager.
1:00:23
And it was just super great to to just actually just have
1:00:28
people do study hall at
1:00:29
the beginning part of a meeting where you just everybody just
1:00:32
gets fast forwarded.
1:00:34
SPEAKER_01: And then you have a really great discussion at the end.
1:00:36
That is what an amazing culture that I think that almost every
1:00:41
other company should
1:00:42
SPEAKER_01: replicate if they could.
1:00:44
But I think that the difficulty would be like you actually have
1:00:47
to be disciplined
1:00:48
and actually have a reading and principled and have a reading
1:00:52
culture and then
1:00:52
SPEAKER_01: actually value writing.
1:00:55
SPEAKER_00: Yeah.
1:00:55
SPEAKER_00: I almost wonder if unless it comes from the top,
1:00:58
SPEAKER_00: some of these things might just be really,
1:00:59
SPEAKER_00: really hard to do.
1:01:00
SPEAKER_00: Yeah.
1:01:01
One thing that I figured is we're in your studio right now and
1:01:05
you have a lot of these blocks.
1:01:07
SPEAKER_00: And I asked them what they are.
1:01:08
SPEAKER_00: Are they for promotions or projects or whatever?
1:01:10
SPEAKER_00: They're for patents.
1:01:12
SPEAKER_00: Yeah.
1:01:13
SPEAKER_00: And this is for patent number 10,000, 10,824,964.
1:01:15
SPEAKER_00: Can you tell me about why you have these?
1:01:17
SPEAKER_00: How they come about?
1:01:18
SPEAKER_00: Yeah.
1:01:18
SPEAKER_00: What you needed to do for them?
1:01:25
SPEAKER_01: So the highest order bit is like,
1:01:28
SPEAKER_01: you know,
1:01:29
SPEAKER_01: for better or for worse,
1:01:29
SPEAKER_01: there are software patents that exist.
1:01:32
SPEAKER_01: Amazon,
1:01:33
they'll say that basically the reason they have them is
1:01:36
defensively because,
1:01:37
SPEAKER_01: you know,
1:01:38
SPEAKER_01: other people will assert that,
1:01:39
SPEAKER_01: hey,
1:01:40
SPEAKER_01: you're in violation of our patents or our IP.
1:01:42
SPEAKER_01: Yeah.
1:01:43
SPEAKER_01: And then, you know, we'll use them reactively.
1:01:45
SPEAKER_01: Okay, fine.
1:01:46
SPEAKER_01: But, you know, you're also in violation of these other things.
1:01:50
SPEAKER_01: And so,
1:01:50
SPEAKER_01: you know,
1:01:51
SPEAKER_01: there is a culture of trying to make sure that,
1:01:54
SPEAKER_01: you know,
1:01:54
SPEAKER_01: we protect ourselves in that way.
1:01:56
SPEAKER_01: But,
1:01:56
SPEAKER_01: you know,
1:01:56
SPEAKER_01: there's the other part of software patents,
1:01:58
SPEAKER_01: which is basically like,
1:01:58
SPEAKER_01: hey,
1:01:59
SPEAKER_01: can you really patent like math or whatever?
1:02:01
SPEAKER_01: Yeah.
1:02:02
SPEAKER_01: And so what I learned over time is that,
1:02:04
SPEAKER_01: you know,
1:02:04
SPEAKER_01: I'm just a really bad IP lawyer,
1:02:06
SPEAKER_01: even though,
1:02:07
SPEAKER_01: you know,
1:02:07
SPEAKER_01: as a principal engineer,
1:02:08
SPEAKER_01: I might cosplay as somebody that really understands software patents.
1:02:11
SPEAKER_01: Right.
1:02:13
SPEAKER_01: At the end of the day,
1:02:14
SPEAKER_01: you know,
1:02:15
what we would do is we would take our important six pagers and
1:02:17
we would hand them
1:02:18
SPEAKER_01: over to the legal team.
1:02:19
SPEAKER_01: And then they would just be like, oh, this stuff is really interesting.
1:02:22
SPEAKER_01: Like, let's explore that.
1:02:24
And so it turned into this awesome thing where like we just had
1:02:27
ready inputs to go
1:02:28
SPEAKER_01: into like the,
1:02:30
SPEAKER_01: you know,
1:02:30
SPEAKER_01: into that particular system.
1:02:32
SPEAKER_01: A writing culture turns out has a bunch of benefits.
1:02:35
SPEAKER_01: Exactly.
1:02:36
And I think that there's this sort of like it's the concept is
1:02:40
called like the
1:02:40
SPEAKER_01: curse of knowledge,
1:02:41
SPEAKER_01: which is essentially like if you understand something,
1:02:44
SPEAKER_01: you discount how long,
1:02:46
SPEAKER_01: like how easy that concept is.
1:02:48
SPEAKER_01: And so it's just like, you don't get it.
1:02:50
SPEAKER_01: You don't get it.
1:02:51
SPEAKER_01: You don't get it.
1:02:51
SPEAKER_01: And then you get it.
1:02:52
SPEAKER_01: And then you're like, oh, that's trivial.
1:02:54
SPEAKER_01: Right.
1:02:54
SPEAKER_01: Even though,
1:02:55
SPEAKER_01: you know,
1:02:55
SPEAKER_01: there could have been,
1:02:56
SPEAKER_01: you know,
1:02:56
SPEAKER_01: it could actually be novel or it could actually be interesting.
1:02:59
And so what ends up happening is that you would just throw
1:03:01
these documents over to
1:03:02
SPEAKER_01: the lawyers and then they would basically be like,
1:03:05
SPEAKER_01: oh,
1:03:05
SPEAKER_01: this stuff is great.
1:03:06
SPEAKER_01: And
1:03:07
SPEAKER_01: You would just be like,
1:03:08
SPEAKER_01: well,
1:03:08
that's just regular software development or that's just the
1:03:11
context and domain that
1:03:12
SPEAKER_01: we were living in.
1:03:12
SPEAKER_01: You know, it turns out that there's some interesting stuff.
1:03:15
SPEAKER_01: This particular patent I'm proud of.
1:03:17
So there's a system design interview question that seems to be
1:03:21
popular right now,
1:03:22
SPEAKER_01: which is like design ticket master.
1:03:24
SPEAKER_02: Mm-hmm.
1:03:25
And so I worked on Amazon tickets and we ended up shuttering
1:03:29
that business,
1:03:29
but we ended up building one of the world's fastest ticket
1:03:33
selling systems in the
1:03:35
SPEAKER_01: world.
1:03:35
SPEAKER_01: We could do many, many orders per second.
1:03:38
So the use case is basically at T0,
1:03:40
that's for a really big ticket on sale.
1:03:43
SPEAKER_01: That's when the maximum amount of demand and requests are coming in.
1:03:47
And you want to sell out all of your ticket supply as quickly
1:03:51
as possible.
1:03:52
SPEAKER_01: The problem is, I think, one where you have seated concerts.
1:03:59
SPEAKER_01: And so when you purchase a ticket,
1:04:02
SPEAKER_01: you know,
1:04:02
SPEAKER_01: most of the time with the system design stuff,
1:04:04
SPEAKER_01: it'll be like general admission or it won't be a high ticket on,
1:04:07
SPEAKER_01: you know,
1:04:08
SPEAKER_01: like one with a bunch of demand.
1:04:10
SPEAKER_01: You have to find contiguous seats.
1:04:12
SPEAKER_01: Yeah, so the ones are next to each other.
1:04:14
SPEAKER_01: Yes, exactly.
1:04:16
SPEAKER_01: And so, you know, it's actually really hard.
1:04:20
SPEAKER_01: Like, suppose it was a SQL database as your backing store.
1:04:23
Like,
1:04:23
how do you come up with a SQL query that's just like,
1:04:26
hey,
1:04:26
give me the best...
1:04:28
four tickets,
1:04:29
you know,
1:04:29
within this particular price range that are sitting next to
1:04:33
each other.
1:04:33
SPEAKER_00: Yeah.
1:04:34
SPEAKER_00: Now you're thinking,
1:04:35
SPEAKER_00: so this is a real world thing where you need to,
1:04:37
SPEAKER_00: you want to be as efficient as possible in terms of resource usage.
1:04:41
SPEAKER_00: May that be,
1:04:41
SPEAKER_00: maybe you want to minimize your CPU or memory,
1:04:43
SPEAKER_00: depending on what you have,
1:04:44
SPEAKER_00: I assume.
1:04:45
And you need to do as quick,
1:04:47
as rapidly as possible to give this to people.
1:04:51
Okay,
1:04:51
so now we're talking about a problem that seems pretty novel in
1:04:55
some ways,
1:04:56
right?
1:04:56
SPEAKER_01: Yeah, and so I did this patent with a senior principal.
1:05:00
SPEAKER_01: I was a senior engineer at the time.
1:05:02
SPEAKER_01: But the idea is like...
1:05:05
SPEAKER_01: you know,
1:05:05
SPEAKER_01: what is the theoretical maximum speed by which we could,
1:05:10
SPEAKER_01: you know,
1:05:10
SPEAKER_01: show this inventory to people?
1:05:12
SPEAKER_01: And it turns out that,
1:05:15
SPEAKER_01: you know,
1:05:15
SPEAKER_01: even if you have a high ticket on sale,
1:05:17
SPEAKER_01: you only have like thousands of tickets at the end of the day.
1:05:20
So instead of making a request to like a backend that would
1:05:24
conduct some sort of
1:05:24
SPEAKER_01: search across the space,
1:05:27
What if you actually inverted it and then you basically had
1:05:31
each of the individual
1:05:32
SPEAKER_01: hosts have like some view on the entire arena or venue that was there.
1:05:38
And you loaded up all of that availability and inventory into
1:05:43
like L2 cache on a CPU.
1:05:44
SPEAKER_01: Yeah.
1:05:45
SPEAKER_01: Because it's actually not that many.
1:05:46
SPEAKER_01: So if you had this compact.
1:05:47
SPEAKER_01: Yeah.
1:05:48
SPEAKER_01: We'll kind of cash it a pretty big.
1:05:49
SPEAKER_01: Yeah.
1:05:50
SPEAKER_01: Then what you can do is you can do bit manipulation to,
1:05:53
SPEAKER_01: like,
1:05:54
SPEAKER_01: really,
1:05:54
SPEAKER_01: really quickly get contiguous seeds that are there.
1:05:58
SPEAKER_01: And then what you do is you can,
1:06:00
SPEAKER_01: like,
1:06:00
SPEAKER_01: send in that particular request and try to,
1:06:03
SPEAKER_01: like,
1:06:03
SPEAKER_01: reserve those particular seeds.
1:06:04
SPEAKER_01: Now there's a logging problem.
1:06:06
SPEAKER_01: Which is much more tractable than, like, hey, there's, you know...
1:06:12
SPEAKER_01: 2 million people that have just hit your onset.
1:06:15
SPEAKER_01: And each of them, I'm wanting to search for each of them.
1:06:18
SPEAKER_01: Yes.
1:06:18
So the inversion of that ordering process by which you like
1:06:21
actually send out the
1:06:22
inventory to the individual nodes and then like load it up into
1:06:27
CPU cache and then
1:06:28
just do bit manipulation and then try to lock that resource
1:06:32
from the individual
1:06:33
SPEAKER_01: nodes.
1:06:33
SPEAKER_01: That was the basis of this particular patent.
1:06:36
awesome that's that's clever and like that sounds like some you
1:06:39
know people are
1:06:40
always asking like oh you know on my job i don't use the
1:06:43
algorithm stuff or any of
1:06:45
the formal methods sounds like there are some uses of it
1:06:48
especially when you're
1:06:49
trying to figure out what is it like when you just taking away
1:06:52
from the pattern
1:06:53
like just having a problem like like like this and saying like
1:06:56
what is the
1:06:56
theoretical limit that we can do what is the fastest possible
1:07:00
like to answer that
1:07:00
SPEAKER_00: you probably want to have access to these tools like you know like
1:07:04
SPEAKER_00: So it's not always a time and effort to actually get into these things.
1:07:08
And so what are you up to now that you've left Amazon a year
1:07:13
ago after like 17,
1:07:16
SPEAKER_00: 18 very long years?
1:07:17
SPEAKER_01: You know, I'm just, you know, I'm just making content.
1:07:19
SPEAKER_01: I'm just sort of living the dream there.
1:07:21
SPEAKER_01: You know, making YouTube videos.
1:07:22
SPEAKER_01: It started up a newsletter.
1:07:23
SPEAKER_01: I've had Discord community and...
1:07:27
SPEAKER_00: Yeah, and we're going to link all of those below.
1:07:30
SPEAKER_00: I actually got to first know you before we started talking.
1:07:33
SPEAKER_00: This was probably a few years ago from your YouTube videos,
1:07:36
SPEAKER_00: which you shared a lot about Amazon things,
1:07:41
SPEAKER_00: software engineering things,
1:07:42
SPEAKER_00: and just your general thinking.
1:07:43
But yeah,
1:07:44
your newsletter is a new one,
1:07:45
so we'll link it in the show notes below.
1:07:48
It's always a good way to keep in touch,
1:07:49
and also on your YouTube channel.
1:07:51
SPEAKER_00: Awesome.
1:07:52
SPEAKER_00: So as closing, I have some rapid questions.
1:07:55
SPEAKER_00: So I'll just ask and you just shoot what comes to mind.
1:07:58
SPEAKER_00: What is career advice that greatly helped you in your path?
1:08:02
SPEAKER_01: Yeah, I mean, this is, you know, I talk a lot about this.
1:08:05
SPEAKER_01: It's kind of like, oh, what's your favorite food or your favorite movie?
1:08:08
SPEAKER_01: It's just like there's so much there and it's hard to pick one.
1:08:11
SPEAKER_01: What I would say is instead of saying like,
1:08:13
SPEAKER_01: hey,
1:08:14
SPEAKER_01: what's the technology that I should learn that's really going to,
1:08:17
SPEAKER_01: you know,
1:08:18
SPEAKER_01: make my career better?
1:08:20
you know,
1:08:20
solid,
1:08:21
instead sort of flip it around and say like,
1:08:24
how can I quickly learn skills?
1:08:27
SPEAKER_01: that makes you sort of like recession-proof, right?
1:08:31
SPEAKER_01: That sort of makes you valuable.
1:08:32
SPEAKER_01: It's essentially meta-learning.
1:08:34
SPEAKER_01: It's like, how can I learn something faster and faster?
1:08:37
SPEAKER_01: If that's your focus,
1:08:39
SPEAKER_01: then you'll always be,
1:08:40
SPEAKER_01: you'll never have a problem finding a job and you'll never have a problem
1:08:44
SPEAKER_01: progressing in your career.
1:08:46
SPEAKER_01: Now,
1:08:46
SPEAKER_01: some of the skills may be difficult to find resources on online,
1:08:50
SPEAKER_01: but I think if you just sort of think about like,
1:08:53
SPEAKER_01: what's a valuable skill that if I knew right now,
1:08:56
would you know make my you know job search easier would like
1:09:00
make me you know
1:09:01
perform better on the job and then just sort of thinking about
1:09:05
acquiring that skill
1:09:06
as quickly as possible and do it now like don't wait yeah well
1:09:09
people tend to
1:09:10
postpone themselves they'll be like oh well I'll start when you
1:09:14
know everything is
1:09:15
SPEAKER_01: lined up but like
1:09:17
SPEAKER_01: To begin, you just need to begin.
1:09:18
Like when you start something that only then will you know what
1:09:21
you need to do
1:09:22
SPEAKER_01: instead of saying like,
1:09:23
SPEAKER_01: oh,
1:09:24
SPEAKER_01: I need to get everything that I need to do first before I start.
1:09:27
SPEAKER_01: You've used a lot of programming languages.
1:09:29
SPEAKER_00: Which one's your favorite and why?
1:09:31
SPEAKER_00: And which one you do dislike most?
1:09:33
Yeah,
1:09:35
you know,
1:09:35
I have like,
1:09:37
you know,
1:09:37
obviously there's no perfect programming language.
1:09:40
SPEAKER_01: What I would say is like, I really enjoyed Perl.
1:09:45
SPEAKER_01: And nobody would ever give that answer.
1:09:47
But I just like this concept of like,
1:09:49
there's just so many different ways to do it.
1:09:51
SPEAKER_01: It's a write-only language.
1:09:53
SPEAKER_01: Like you can't read anybody else's Perl.
1:09:55
SPEAKER_01: And it's actually one of the languages that like uses up the most power.
1:09:58
SPEAKER_01: It's like the least efficient.
1:10:00
SPEAKER_01: It's interpreted.
1:10:00
SPEAKER_01: Yeah.
1:10:02
SPEAKER_01: It's just, like, terrible.
1:10:04
SPEAKER_01: Most of Booking.com still runs it, or some of it.
1:10:06
SPEAKER_01: Amazon's back end was,
1:10:08
SPEAKER_01: you know,
1:10:08
SPEAKER_01: for a long time,
1:10:09
SPEAKER_01: it still might be,
1:10:10
SPEAKER_01: you know,
1:10:11
sort of like Pearl Mason and sort of like web technology bolted
1:10:14
onto Pearl.
1:10:15
SPEAKER_01: But I just kind of like it.
1:10:16
SPEAKER_01: I just feel like I can express myself,
1:10:17
SPEAKER_01: and there's just,
1:10:18
SPEAKER_01: like,
1:10:18
SPEAKER_01: there's just...
1:10:19
SPEAKER_01: However you'd like to express yourself,
1:10:21
SPEAKER_01: you can.
1:10:23
It also looked like an ASCII factory blew up sometimes,
1:10:25
and so it's just,
1:10:26
like,
1:10:26
it's...
1:10:28
SPEAKER_01: Now that it's on a podcast, I wouldn't really advertise that fact.
1:10:31
SPEAKER_01: The best programming languages right now,
1:10:33
SPEAKER_01: I think Rust is pretty interesting,
1:10:35
SPEAKER_01: so I might pick that up.
1:10:38
SPEAKER_01: At the end of the day, I really love the boring languages.
1:10:43
SPEAKER_01: So Java,
1:10:46
SPEAKER_01: for all of its stuff,
1:10:47
SPEAKER_01: like its verbosity,
1:10:48
SPEAKER_01: I think it's just a great language,
1:10:50
SPEAKER_01: like a JVM-based language.
1:10:53
um that has essentially like great like library support and a
1:10:58
bunch of stuff
1:10:58
written for it but it's just like super boring maybe it's just
1:11:01
because i'm from
1:11:02
amazon and we do this like enterprise stuff like it's a fine
1:11:06
language and then i
1:11:07
see you have a large bookshelf here you also read a lot
1:11:11
especially at amazon all
1:11:12
the most internal documents what is a book that you would
1:11:14
recommend something
1:11:15
around software engineering that that you enjoyed and it cannot
1:11:18
be that book
1:11:19
SPEAKER_01: It can't be your book.
1:11:21
SPEAKER_01: What I would say is,
1:11:22
SPEAKER_01: you know,
1:11:23
SPEAKER_01: I just given the advice about,
1:11:26
SPEAKER_01: you know,
1:11:26
SPEAKER_01: meta learning and career growth.
1:11:29
SPEAKER_01: I think that most software developers should read a book by Cal Newport.
1:11:32
SPEAKER_01: It's called So Good They Can't Ignore You.
1:11:35
SPEAKER_01: And so the concept there is around career capital.
1:11:38
SPEAKER_01: So like, what are the skills that are in the most demand?
1:11:41
SPEAKER_01: And if you can just like, learn those skills, then you become in demand.
1:11:44
SPEAKER_01: And then,
1:11:45
SPEAKER_01: you know,
1:11:45
SPEAKER_01: from there,
1:11:46
SPEAKER_01: you can choose what type of lifestyle that you'd like,
1:11:48
SPEAKER_01: you know,
1:11:49
SPEAKER_01: you can also like sort of lean into,
1:11:51
SPEAKER_01: you know,
1:11:51
SPEAKER_01: some of the science of meta learning.
1:11:53
SPEAKER_01: So deliberate practice, space repetition, that sort of thing.
1:11:56
In terms of,
1:11:57
like,
1:11:57
tech books,
1:11:58
I think the new AI engineering book by Chip Wynn is amazing.
1:12:05
SPEAKER_01: I think DDIA, so the design of data intensive.
1:12:10
SPEAKER_01: It's so good.
1:12:10
SPEAKER_01: A new version is coming at the end of the year, actually.
1:12:13
SPEAKER_01: I'm excited about that.
1:12:14
SPEAKER_01: I think that'll be pretty good.
1:12:15
SPEAKER_01: But, you know, at the end of the day, like...
1:12:17
SPEAKER_01: You don't want one book on your bookshelf.
1:12:19
SPEAKER_01: You want 50 books on your bookshelf.
1:12:22
SPEAKER_01: And so,
1:12:23
SPEAKER_01: you know,
1:12:23
SPEAKER_01: I think within a particular subgenre of tech books,
1:12:27
SPEAKER_01: you know,
1:12:28
SPEAKER_01: I'd have recommendations there.
1:12:29
SPEAKER_01: Steve, this was great.
1:12:31
SPEAKER_01: Awesome.
1:12:31
SPEAKER_00: Really enjoyed it.
1:12:32
SPEAKER_00: Yeah, great.
1:12:33
SPEAKER_00: Thanks so much for having me.
1:12:34
SPEAKER_00: Thanks a lot for Steve for sharing all these details.
1:12:37
Although Amazon's principal engineering level feels
1:12:39
surprisingly difficult to get
1:12:40
SPEAKER_00: promoted to,
1:12:41
I have yet to hear of such a strong principal engineering
1:12:43
community than what
1:12:44
SPEAKER_00: Amazon builds and keeps investing in.
1:12:47
This community itself could be a reason enough to consider the
1:12:49
company after the
1:12:50
SPEAKER_00: principal plus level,
1:12:52
SPEAKER_00: should you have the opportunity to do so.
1:12:53
SPEAKER_00: For a deep dive into Amazon's engineering culture,
1:12:56
SPEAKER_00: including the details on compensation,
1:12:58
SPEAKER_00: career ladders,
1:12:59
SPEAKER_00: performance reviews and engineering processes,
1:13:01
check out the Pragmatic Engineer Deep Dive linked in the show
1:13:04
notes below.
1:13:05
SPEAKER_00: If you've enjoyed this podcast,
1:13:06
SPEAKER_00: please do subscribe on your favorite podcast platform and on YouTube.
1:13:10
This helps more people discover the podcast and a special thank
1:13:13
you if you leave a rating.
1:13:14
SPEAKER_00: Thanks and see you in the next one.