Latest / The Pragmatic Engineer / Stacked diffs and tooling at Meta with Tomas Reimers

Transcript

Stacked diffs and tooling at Meta with Tomas Reimers · The Pragmatic Engineer

0:00
SPEAKER_01: What does actual development look like when you join?
0:02
SPEAKER_01: Fabricator is our internal code review tooling.
0:04
SPEAKER_01: Fabricator is well integrated with Sandcastle,
0:06
SPEAKER_01: which is our internal CI tooling,
0:08
SPEAKER_01: which is integrated with OnDemand,
0:09
SPEAKER_01: which are our internal dev boxes,
0:11
SPEAKER_01: which is integrated with Landcastle,
0:13
SPEAKER_01: which is how code makes its way out to users.
0:16
SPEAKER_01: And sort of the thing goes from there.
0:17
SPEAKER_01: And then it doesn't just stop at code review,
0:19
SPEAKER_01: it's integration across the rest of the developer platform.
0:22
SPEAKER_01: For example, Meta built its own task system.
0:24
And so tasks and pull requests could be very deeply integrated
0:28
in a way that we
0:29
SPEAKER_01: just don't see out here.
0:30
One tool that I kind of experienced because of Fabricator was
0:34
called Herald.
0:35
Herald,
0:35
if I recall correctly,
0:36
was the rules engine,
0:37
which later got replaced by ButterflyBot.
0:39
SPEAKER_01: And the idea of both of them was the same.
0:41
It was being able to match events that might happen in the
0:44
normal course of code review.
0:45
SPEAKER_01: I used to work on internal developer tools there.
0:47
SPEAKER_01: When we were deprecating APIs,
0:49
you might go ahead and do something like create a Herald rule
0:51
that would say
0:52
SPEAKER_01: something like,
0:53
SPEAKER_01: if a user calls this function or uses this API,
0:56
SPEAKER_01: post a comment letting them know,
0:57
SPEAKER_01: hey,
0:57
SPEAKER_01: this is deprecated.
0:59
SPEAKER_01: Also, the new syntax is this.
1:01
SPEAKER_01: And then as you got later and later into the rollout,
1:03
SPEAKER_01: you could start to do things instead like,
1:05
SPEAKER_01: hey,
1:05
notify me if anyone's adding call sites of this API so that I
1:08
can go ahead and
1:09
SPEAKER_01: leave a comment saying,
1:11
SPEAKER_01: hey,
1:11
SPEAKER_01: like,
1:11
SPEAKER_01: no,
1:11
SPEAKER_01: we're really trying to deprecate this.
1:13
SPEAKER_00: Meta is famous for its customer internal tooling.
1:15
Devs at the company use the likes of Fabricator,
1:17
Sandcastle,
1:18
or ButterflyBot.
1:19
But what are these tools and why did Meta not use a more
1:22
standard stack to build software?
1:24
We get answers from Thomas Rainer who was previously at Meta
1:26
working on developer
1:27
SPEAKER_00: tooling before co-founding Graphite,
1:29
SPEAKER_00: a developer productivity platform.
1:31
Today we talk about what stack diffs are and how they became
1:35
the de facto way devs
1:36
SPEAKER_00: at Meta work with pull requests.
1:38
Why Meta moved to a monorepo and why there seems to be an
1:40
industry-wide trend in
1:41
SPEAKER_00: companies moving from polyrepos to monorepos.
1:44
SPEAKER_00: Why AI coding tools will make code reviews a lot more important and more.
1:48
If you're interested in how effective engineering teams work
1:50
these days and
1:51
SPEAKER_00: approaches trending in this group,
1:53
SPEAKER_00: this episode is for you.
1:54
If you enjoy the show,
1:55
please subscribe to the podcast and consider leaving a rating.
1:58
SPEAKER_00: This is great to help the show.
1:59
SPEAKER_00: And with this, let's jump in.
2:01
SPEAKER_00: Thomas, it's great to have you on the podcast.
2:03
SPEAKER_00: Great to be here.
2:05
SPEAKER_00: So before you started Graphite,
2:08
SPEAKER_00: you worked at Meta,
2:10
SPEAKER_00: and Meta is known for a lot of their in-house tooling.
2:13
SPEAKER_00: Can we talk about when you joined,
2:16
SPEAKER_00: what was interesting tools that you didn't see at other companies before?
2:20
SPEAKER_01: So funny.
2:20
SPEAKER_01: It's a great question, too.
2:21
SPEAKER_01: And yes, certainly.
2:24
SPEAKER_01: When I joined, I was a new grad.
2:26
SPEAKER_01: So I had just graduated college.
2:27
SPEAKER_01: I joined Meta.
2:28
SPEAKER_01: I used a lot of the tooling.
2:30
SPEAKER_01: My frame of reference was personal projects.
2:33
SPEAKER_01: I used to do a lot of hackathons back in college and internships.
2:36
SPEAKER_01: And so I think when I joined Meta, a lot of it came in sort of...
2:42
SPEAKER_01: eyes wide,
2:43
SPEAKER_01: sort of like,
2:43
SPEAKER_01: I was about to say eyes wide open,
2:45
SPEAKER_01: but I mean the opposite of that,
2:46
of I came in sort of just like blissfully unaware that this
2:49
wasn't industry standard.
2:51
SPEAKER_01: When I left was when I had the bigger shock,
2:52
SPEAKER_01: because I think I had used tools like GitHub before,
2:55
SPEAKER_01: I'd used analytics tools before,
2:57
I had left Meta after having really gotten sort of immersed in
3:02
all of this
3:03
SPEAKER_01: wonderful tooling.
3:04
SPEAKER_01: And then when I had to start to
3:07
SPEAKER_01: do all of these same activities myself,
3:09
I found myself reaching for tools that didn't exist and there
3:11
weren't clear analogs for.
3:13
SPEAKER_00: Yeah, it's usually the other way around, right?
3:14
Like usually people who work in industry at a startup or even
3:19
at a larger company go to meta.
3:20
SPEAKER_00: And unless you've been in like Google where,
3:23
SPEAKER_00: you know,
3:23
SPEAKER_00: it's all custom,
3:24
SPEAKER_00: it's just a shock because I think most people imagine like,
3:27
SPEAKER_00: okay,
3:27
SPEAKER_00: you know,
3:28
SPEAKER_00: they must use GitHub or at least something similar.
3:30
SPEAKER_00: But what does actual development look like?
3:33
SPEAKER_00: You know, when you joined, right?
3:35
SPEAKER_00: Like you went into bootcamp, I assume.
3:37
SPEAKER_00: What does it look like to like,
3:39
SPEAKER_00: you know,
3:39
SPEAKER_00: like they didn't even push a pull request,
3:41
SPEAKER_00: but that's not what they call it,
3:42
SPEAKER_00: do they?
3:43
SPEAKER_01: No, no, they call it a diff.
3:46
SPEAKER_01: Good, good, good trivia for it, by the way.
3:49
SPEAKER_01: I think that,
3:51
SPEAKER_01: though,
3:51
SPEAKER_01: if I had to use one word to describe Meta's internal tooling,
3:54
SPEAKER_01: it would be integrated.
3:55
SPEAKER_01: Everything, everything, everything integrates with each other.
3:58
And I tell this to people,
4:00
and they never really understand the extent of it.
4:02
SPEAKER_01: Meta builds its own internal calendar.
4:05
So meeting room scheduling can be integrated in with the rest
4:08
of the tooling.
4:09
SPEAKER_01: And that really sort of permeates the whole thing.
4:12
SPEAKER_01: Fabricator is our internal code review tooling.
4:14
SPEAKER_01: Fabricator is well integrated with Sandcastle,
4:17
SPEAKER_01: which is our internal CI tooling,
4:18
SPEAKER_01: which is integrated with OnDemand,
4:20
SPEAKER_01: which are our internal dev boxes,
4:21
SPEAKER_01: which is integrated with Landcastle,
4:24
SPEAKER_01: which is how code makes its way out to users.
4:26
SPEAKER_01: And sort of the thing goes from there.
4:28
SPEAKER_01: And then it doesn't just stop at code review.
4:30
SPEAKER_01: It's integration across the rest of sort of like the developer platform.
4:33
SPEAKER_01: For example, meta built its own task system.
4:36
And so tasks and pull requests could be very deeply integrated
4:39
in a way that we
4:40
SPEAKER_01: just don't see out here.
4:42
SPEAKER_01: What was Soundcastle?
4:44
SPEAKER_01: So Sandcastle was the internal CI system.
4:47
SPEAKER_01: So out here, a lot of people use GitHub Actions.
4:50
SPEAKER_01: They use BuildKite.
4:52
SPEAKER_01: And Sandcastle was just our flavor of that.
4:54
It allowed us to build versions of our apps and websites before
4:58
we released and
5:01
SPEAKER_01: went ahead and tested them for us.
5:03
SPEAKER_00: Yeah.
5:04
SPEAKER_00: You mentioned this integration.
5:06
When you say it,
5:07
it sounds like it's just a word,
5:08
but in practice,
5:09
what did it mean?
5:11
SPEAKER_00: You did mention that,
5:11
SPEAKER_00: for example,
5:12
SPEAKER_00: tasks,
5:12
SPEAKER_00: which is,
5:13
SPEAKER_00: I guess we can give it the JIRA or linear or whatever that is,
5:17
SPEAKER_00: that was integrated with the other stuff.
5:19
SPEAKER_00: And this was like 10 years ago, right, or so?
5:24
SPEAKER_01: Actually,
5:24
SPEAKER_01: yeah,
5:26
SPEAKER_01: I think integration here,
5:28
I think one of the best examples I have of integration was
5:30
actually a translation system.
5:32
SPEAKER_01: So at scale, internationalization becomes a real problem, right?
5:35
You need to be able to include internationalization as product
5:38
developers start to
5:39
SPEAKER_01: update the website,
5:40
SPEAKER_01: create new pages,
5:41
SPEAKER_01: et cetera.
5:42
SPEAKER_01: There's always this question or this tension that happens of,
5:45
SPEAKER_01: well,
5:45
how do we know that the string that we're pushing has a
5:48
translator equivalent?
5:50
At Facebook,
5:50
that could actually just be built into the code review tooling,
5:53
right?
5:53
SPEAKER_01: And so the code review tooling can tell you something like,
5:55
SPEAKER_01: hey,
5:56
SPEAKER_01: this diff is using a string for which the translations don't exist.
6:00
SPEAKER_01: And because of that, you can't deploy it yet.
6:03
Or what was more interesting to me was when you looked at a
6:05
diff that had been merged,
6:06
SPEAKER_01: that had been landed,
6:08
SPEAKER_01: That was what we called it.
6:10
You would see on that same page like,
6:11
hey,
6:12
this diff has been rolled out to employees.
6:14
SPEAKER_01: This diff has been rolled out to 1% of users.
6:16
SPEAKER_01: This diff has been rolled out to 10% of users.
6:19
SPEAKER_01: Also,
6:19
SPEAKER_01: if this diff included any feature flags or sort of like AB tests,
6:23
SPEAKER_01: here are the results of those AB tests.
6:25
And so you had this wonderful ecosystem where everything was
6:28
tightly tied together.
6:29
And so you as a developer could just go to one place and be like,
6:33
I know what I'm looking for.
6:34
SPEAKER_01: I'm looking for this piece of like code that I landed.
6:37
SPEAKER_01: Let's go see like how it's changed statistics or how it's impacted users.
6:43
SPEAKER_01: And you could see that all in one place.
6:45
And then there was a bunch of other stuff that started to make
6:47
its way into the
6:47
SPEAKER_01: public tool chain now of if you wanted to revert a diff,
6:50
SPEAKER_01: you could just go ahead and do that directly from Fabricator.
6:54
SPEAKER_00: This episode is brought to you by Swarmia,
6:56
SPEAKER_00: the engineering intelligence platform for modern software organizations.
7:00
Swarmia gives everyone in your organization the visibility and
7:02
tools they need to
7:03
SPEAKER_00: get better at getting better.
7:05
Engineering leaders use Swarmia to balance the investment
7:07
between different types
7:08
SPEAKER_00: of work,
7:09
SPEAKER_00: stay on top of cross-team initiatives,
7:11
SPEAKER_00: and automate the creation of cost capitalization reports.
7:15
Engineering managers and team leads get access to a powerful
7:17
combination of
7:18
research-backed engineering metrics and developer experience
7:21
surveys to identify
7:22
SPEAKER_00: and eliminate process bottlenecks.
7:24
Software engineers speed up their daily workflows with Swarmy
7:27
as two-way Slack notifications,
7:29
SPEAKER_00: working agreements,
7:30
SPEAKER_00: and team-focused insights.
7:32
You can learn more about how some of the world's best software organizations,
7:35
SPEAKER_00: including Miro,
7:36
SPEAKER_00: Docker,
7:36
SPEAKER_00: and Webflow,
7:37
SPEAKER_00: use Swarmia to build better software,
7:39
SPEAKER_00: faster,
7:40
SPEAKER_00: at swarmia.com slash pragmatic.
7:42
SPEAKER_00: That is S-W-A-R-M-I-A dot com slash pragmatic.
7:47
SPEAKER_00: Fabricator was a really central part of this.
7:49
SPEAKER_00: I know because at Uber, we use Fabricator.
7:52
It was an open source tooling,
7:54
and I think Uber,
7:55
and you mentioned Dropbox,
7:56
also took it on.
7:58
SPEAKER_00: Maybe a few other companies, but not as many.
8:00
SPEAKER_00: So for a while, we kind of...
8:02
SPEAKER_00: used it.
8:02
SPEAKER_00: I saw some of this integration, but
8:06
SPEAKER_00: It sounds a little wild because I don't really know any other company,
8:09
SPEAKER_00: maybe outside of Google,
8:10
SPEAKER_00: where you can have this thing that,
8:13
SPEAKER_00: as you say,
8:13
SPEAKER_00: you deploy a code and you can see the experiments,
8:18
SPEAKER_00: the localization,
8:19
SPEAKER_00: the rollout,
8:21
SPEAKER_00: all from one UI.
8:21
SPEAKER_00: Usually it's different systems.
8:22
SPEAKER_00: Even at Uber, it was different systems.
8:24
SPEAKER_00: We had an experimentation system.
8:26
SPEAKER_00: It was called Morpheus at the time or Flipper, the feature flags.
8:29
SPEAKER_00: I had this list of
8:33
SPEAKER_00: like systems that I,
8:34
SPEAKER_00: you know,
8:34
SPEAKER_00: like a cheat sheet of where I would go to check the rollout,
8:37
SPEAKER_00: the internationalization,
8:39
SPEAKER_00: et cetera,
8:39
SPEAKER_00: et cetera.
8:40
SPEAKER_00: So all of this was really kind of like one click away.
8:44
SPEAKER_01: I think that's the way it exists for a lot of people today.
8:46
SPEAKER_01: You know,
8:47
SPEAKER_01: I think when we look of right now,
8:49
SPEAKER_01: best in class dev tooling is starting to become more integrated.
8:52
SPEAKER_01: I think around like linear,
8:53
SPEAKER_01: it's now starting to post the linear issue in the GitHub pull request.
8:58
And you can see,
8:58
and it updates the status as the pull request moves through review.
9:02
SPEAKER_01: You're also seeing the same with Rissell, right?
9:03
SPEAKER_01: When you create a PR,
9:05
SPEAKER_01: Rissell can go ahead and be like,
9:06
SPEAKER_01: hey,
9:07
SPEAKER_01: went ahead and like made a preview environment for you.
9:09
SPEAKER_01: You can go ahead and click that there.
9:11
I think what was so magical about Facebook is that all of that
9:14
was native and it
9:15
SPEAKER_01: didn't appear to be integrations,
9:17
SPEAKER_01: but just part of the system.
9:19
SPEAKER_00: And who built this tool, right?
9:23
Like the developer tooling,
9:25
you know,
9:25
Fabricator,
9:27
the Sandcastle,
9:29
all of these things.
9:29
SPEAKER_00: Was it just like devs?
9:30
SPEAKER_00: Like, OK, I'm going to build this.
9:32
SPEAKER_00: Obviously, it was internal teams, right?
9:33
But was it like dedicated teams or people just kind of went and
9:37
like went and tweaked stuff?
9:38
SPEAKER_01: I think it really depends on the tool.
9:41
SPEAKER_01: Some internal dev tools were just built by developers themselves.
9:44
SPEAKER_01: They saw a problem.
9:45
SPEAKER_01: They're like, nothing's addressing this.
9:47
SPEAKER_01: I need to fix it.
9:50
It's a wonderful culture to be a part of where you can solve
9:54
your own problems.
9:55
SPEAKER_01: But some of it came more from the like,
9:57
SPEAKER_01: oh,
9:58
SPEAKER_01: we as a company have noticed that there's an issue here.
10:01
SPEAKER_01: We need to address the issue.
10:03
SPEAKER_01: Let's spin up a team to do it.
10:04
And then the place where all of these tools ultimately found
10:08
their home was under
10:09
SPEAKER_01: the dev infrastructure team.
10:10
So developer infrastructure at other companies called Developer
10:13
Platform and
10:13
Developer Velocity really maintained the bulk of Facebook's
10:18
developer tooling.
10:19
SPEAKER_00: Yeah, and I guess one tool that...
10:23
SPEAKER_00: I kind of experienced because a fabricator was called Herald.
10:27
SPEAKER_00: I'm not sure if you know.
10:29
SPEAKER_00: Yeah, of course.
10:31
SPEAKER_00: You could set up special rules on the code base, anything.
10:34
Like if basically on a commit,
10:36
anything happens,
10:37
like weird rules could run.
10:40
SPEAKER_00: Were there like,
10:41
SPEAKER_00: do you remember some,
10:43
like there were some really creative examples of like using
10:45
Herald rules for
10:46
developers to start to create their own rules or code ownership
10:50
or that kind of stuff?
10:52
SPEAKER_01: Totally.
10:53
SPEAKER_01: So Harold,
10:55
SPEAKER_01: if I recall correctly,
10:56
SPEAKER_01: was the rules engine,
10:57
SPEAKER_01: which later got replaced by ButterflyBot.
10:59
SPEAKER_01: And the idea of both of them was the same.
11:01
It was being able to match events that might happen in the
11:04
normal course of code review.
11:06
Something like someone puts up a PR that includes some line of
11:09
code or calls some
11:10
SPEAKER_01: function or CI fails regarding something.
11:14
SPEAKER_01: And then what it would do is it could take a variety of actions.
11:16
So some of the examples that I remember was when I used to work
11:20
on internal
11:21
SPEAKER_01: developer tools there.
11:22
SPEAKER_01: When we were deprecating APIs,
11:24
you might go ahead and do something like create a Herald rule
11:26
that would say
11:27
SPEAKER_01: something like,
11:28
SPEAKER_01: if a user calls this function or uses this API,
11:31
SPEAKER_01: like post a comment letting them know,
11:33
SPEAKER_01: hey,
11:33
SPEAKER_01: this is deprecated.
11:34
SPEAKER_01: Also, the new syntax is this.
11:37
SPEAKER_01: And then as you got later and later into the rollout,
11:39
SPEAKER_01: you could start to do things instead like,
11:41
SPEAKER_01: hey,
11:41
notify me if anyone's adding call sites of this API so that I
11:45
can go ahead and go
11:46
SPEAKER_01: leave a comment saying,
11:47
SPEAKER_01: hey,
11:48
SPEAKER_01: like,
11:48
SPEAKER_01: no,
11:48
SPEAKER_01: we're really trying to deprecate this,
11:49
SPEAKER_01: like,
11:49
SPEAKER_01: please stop.
11:51
SPEAKER_01: And then you could take it from there.
11:53
SPEAKER_01: Other examples were simpler.
11:54
SPEAKER_01: I remember there were some cases where there were,
11:57
SPEAKER_01: let's call it privileged or important parts of the code base.
11:59
SPEAKER_01: And what you'd say is like,
12:00
if someone goes ahead and creates code that touches that part
12:03
of the code base,
12:03
just make sure this team is added as a reviewer or at very
12:06
least as a subscriber.
12:07
SPEAKER_01: Subscribers were an idea and fabricator of being able to CC people.
12:11
It wasn't that you expected a review,
12:13
more just that you wanted to keep their eyes on it.
12:16
SPEAKER_01: And there were a handful of rules like that.
12:18
SPEAKER_00: And do I remember correctly that code ownership was super,
12:21
SPEAKER_00: super important for Fabricator and for Facebook?
12:24
SPEAKER_00: And it was just really trivial to say,
12:26
SPEAKER_00: I own parts of the code and then set up different rules around that.
12:30
SPEAKER_01: No, so that's a great question.
12:31
So code ownership actually became,
12:34
code ownership went through a few stages at Facebook.
12:36
SPEAKER_01: There was a while where code owners were required at Facebook.
12:39
And so you could have like different parts of the code base
12:42
could have different owners.
12:44
SPEAKER_01: Then the company took it away.
12:46
And they said,
12:46
actually,
12:47
we don't believe that we should have code owners.
12:48
SPEAKER_01: We're a collaborative environment.
12:50
SPEAKER_01: We should go ahead and get rid of that.
12:52
SPEAKER_01: It was then reintroduced.
12:54
And I think it was then again taken away with the idea of like,
12:58
we're not sure.
12:59
The company alternated between,
13:00
oh,
13:01
they're really privileged parts of our monorepo.
13:04
And we can talk more around sort of how Facebook structured
13:06
their code that need
13:07
SPEAKER_01: certain people to review it.
13:09
SPEAKER_01: And while actually we should trust our developers,
13:14
SPEAKER_01: developers need to be able to understand organizational practice.
13:17
SPEAKER_01: One thing that was always existing within the platform,
13:20
SPEAKER_01: though,
13:21
SPEAKER_01: was this idea of if you're in some part of the code base,
13:24
SPEAKER_01: tag the correct reviewers.
13:25
And so I think one of the things that actually I most missed
13:28
when I got out here
13:29
SPEAKER_01: was the ability to structure repository in a way where you could say,
13:32
SPEAKER_01: oh,
13:32
SPEAKER_01: if you touch this part of it,
13:33
SPEAKER_01: we should tag these people in.
13:35
SPEAKER_01: I think in GitHub,
13:36
SPEAKER_01: code owners is a very coarse approximation of that,
13:38
SPEAKER_01: where you have to define one file.
13:40
SPEAKER_01: It's at the root of the repository.
13:42
SPEAKER_01: It's difficult to use.
13:43
SPEAKER_01: A person who I've heard has done this even better company is Google.
13:46
SPEAKER_01: So in Google, code reviewers were assigned hierarchically.
13:50
SPEAKER_01: And so in every folder,
13:51
SPEAKER_01: you could basically say like,
13:52
SPEAKER_01: hey,
13:53
SPEAKER_01: if you touch code in this folder,
13:54
SPEAKER_01: like you need to go talk to these people,
13:56
SPEAKER_01: either as a heads up or as a requirement.
13:59
SPEAKER_01: And it allowed people to get, there were two really nice aspects of that.
14:04
SPEAKER_01: One was that it was really easy to define ownership.
14:06
And then the second was that it was easy as you were writing
14:09
pull requests there to
14:10
SPEAKER_01: be able to go ahead and say like,
14:12
SPEAKER_01: who does own this code?
14:14
SPEAKER_01: Who can I go ask for?
14:15
SPEAKER_01: And so if you're in a
14:16
SPEAKER_01: say that you're using an API that you weren't certain of,
14:18
SPEAKER_01: you need to figure out like,
14:19
SPEAKER_01: who can I go talk to about this API?
14:21
SPEAKER_01: What you could do is you could just navigate the file tree,
14:24
SPEAKER_01: find the team or people in charge of it,
14:27
SPEAKER_01: and then just go DM them.
14:29
SPEAKER_01: And so that became a really powerful system there.
14:31
SPEAKER_00: Now,
14:31
SPEAKER_00: you've gone back and forth between Facebook,
14:33
SPEAKER_00: there was code owners,
14:34
SPEAKER_00: no code owners,
14:35
SPEAKER_00: and you're now working at a startup.
14:37
SPEAKER_00: You actually work with a lot of startups and larger companies.
14:40
SPEAKER_00: What is your unfiltered view on code ownership?
14:43
SPEAKER_00: Is it a good thing?
14:44
SPEAKER_00: Is it a bad thing?
14:45
SPEAKER_00: Is it necessary only when you're bigger or when you're smaller?
14:50
SPEAKER_00: I'm sure you've got some opinions now.
14:52
SPEAKER_01: I work at a startup now, Graphite.
14:53
We have the privilege of working with some of the largest
14:55
DevTools teams in the world,
14:56
SPEAKER_01: which is really cool.
14:58
SPEAKER_01: At this point, I think I've heard all sides of the story.
15:00
SPEAKER_01: We are in a financially regulated market,
15:04
SPEAKER_01: and because of that,
15:05
SPEAKER_01: we have a legal requirement to have code owners.
15:07
SPEAKER_01: And so I've heard that side of the world.
15:09
I've also heard the side of the world of,
15:11
oh,
15:11
we couldn't possibly do code owners here.
15:13
SPEAKER_01: It would just slow us down too much.
15:14
SPEAKER_01: And so I think it really depends on
15:17
SPEAKER_01: the nature of the business.
15:18
SPEAKER_01: I once saw this matrix which I really liked,
15:20
which is if you trust people a lot and are willing to tolerate
15:26
small mistakes,
15:27
SPEAKER_01: you should lean on culture,
15:28
SPEAKER_01: not process.
15:30
SPEAKER_01: As you start to get into a place of,
15:33
SPEAKER_01: oh,
15:34
SPEAKER_01: actually,
15:35
SPEAKER_01: maybe I trust the people less or I can't tolerate mistakes at all,
15:38
SPEAKER_01: you move to automation and enforcement.
15:41
SPEAKER_01: And so it really depends on where your business falls on that.
15:44
SPEAKER_01: I also will say that I think that as Facebook's grown,
15:47
SPEAKER_01: the way they treat different parts of their code base has changed.
15:50
SPEAKER_01: Whereas some parts of the code base absolutely cannot go down.
15:53
SPEAKER_01: Things that deal with privacy cannot go down.
15:55
SPEAKER_01: That's not a place where you can move fast.
15:58
SPEAKER_01: Some other things,
15:59
SPEAKER_01: perhaps,
16:00
SPEAKER_01: maybe the way that you display comments if it's not a big deal,
16:04
SPEAKER_01: that actually is a little bit easier or a new product even.
16:08
There you have a little bit more leeway to move quickly and so
16:10
maybe code owners
16:11
SPEAKER_01: isn't the right match.
16:12
SPEAKER_00: Yeah,
16:12
and one thing that I've kind of learned over time is maybe
16:16
inside a company,
16:17
SPEAKER_00: as much as you'd like to think that there's one engineering culture,
16:19
SPEAKER_00: maybe that's not the answer.
16:20
SPEAKER_00: Maybe a new startup inside a company,
16:24
SPEAKER_00: like an established company like Meta,
16:26
SPEAKER_00: a new startup,
16:26
maybe it should operate differently because maybe they'll be
16:28
shut down in six
16:29
months if they don't find product market fit and they should
16:31
just throw away the things.
16:34
SPEAKER_00: At the same time, I think companies don't really like to admit it, but...
16:39
SPEAKER_00: And maybe even inside a smart company, right?
16:42
SPEAKER_00: Like if you start just a small prototype project,
16:46
SPEAKER_00: see if it goes anywhere,
16:47
you're probably not going to have the same rigid thing as when
16:50
you're deploying to
16:52
SPEAKER_00: customers who actually count on it.
16:53
SPEAKER_00: It needs to stay up.
16:55
SPEAKER_01: I think that's a really wise point.
16:56
SPEAKER_01: I think that ultimately...
16:59
SPEAKER_01: Engineering anywhere exists within a system of constraints,
17:02
SPEAKER_01: and I think those constraints within a company can be very different.
17:04
And so recognizing what are the constraints on the product I'm
17:07
building right now
17:09
and architecting your systems around it is an engineering
17:11
problem like any other.
17:13
I think the place where this actually comes to me most clearly
17:18
is when I deal with
17:19
SPEAKER_01: developer tools teams who release mobile apps versus web apps.
17:23
SPEAKER_01: So web apps tend to release on an hourly basis,
17:27
SPEAKER_01: maybe multiple times a day,
17:28
SPEAKER_01: but probably not slower than that.
17:30
SPEAKER_01: Whereas mobile apps release on a weekly cadence.
17:32
SPEAKER_01: Because of that,
17:33
SPEAKER_01: just the shape of problems they deal with are so different,
17:36
SPEAKER_01: even within a company.
17:37
So mobile app teams generally don't have to hold pagers because
17:41
even if something
17:42
SPEAKER_01: goes down,
17:42
SPEAKER_01: what are you going to do?
17:43
SPEAKER_01: Submit a fix to Apple?
17:44
SPEAKER_01: Great, that's two days.
17:45
SPEAKER_01: Yeah.
17:46
Versus a web app team,
17:47
like paging is an important part of the culture,
17:50
right?
17:50
SPEAKER_01: As you're shipping things really quickly,
17:52
SPEAKER_01: things will break,
17:53
SPEAKER_01: things will go down,
17:54
SPEAKER_01: and you're on the hook to fix them.
17:56
SPEAKER_01: And so watching the difference between those cultures,
18:00
SPEAKER_01: I think,
18:01
SPEAKER_01: is probably the two extremes you see within a company.
18:04
SPEAKER_01: And then,
18:04
SPEAKER_01: again,
18:05
SPEAKER_01: as you start to work in different parts,
18:06
if you work on payment processing versus abuse and fraud
18:10
detection versus the
18:11
SPEAKER_01: consumer front end versus authentication,
18:13
SPEAKER_01: you actually have different constraints,
18:15
SPEAKER_01: which are subtly different.
18:16
SPEAKER_01: And I think that,
18:17
SPEAKER_01: as you're saying,
18:18
engineering organizations can develop different cultures around
18:20
how they need to
18:21
SPEAKER_01: treat them.
18:22
I like to think of constraints and how,
18:25
you know,
18:25
at least the different practices.
18:27
SPEAKER_00: Now,
18:27
SPEAKER_00: one thing that really surprised me at Uber,
18:29
SPEAKER_00: which came really from Meta,
18:31
SPEAKER_00: from Fabricator,
18:32
SPEAKER_00: is we use this thing called stack diffs.
18:34
SPEAKER_00: And we did a deep dive into Pragmatic Engineer.
18:37
SPEAKER_00: It'll be linked in the notes below.
18:39
But it's something that just really clicked for me and also
18:43
everyone around me,
18:44
SPEAKER_00: everyone I talk with at Meta.
18:46
SPEAKER_00: You've now actually built a business on top of this.
18:49
SPEAKER_00: But can we talk about what is stack diffs and how did you come across it?
18:54
SPEAKER_00: And why is it so useful once you get a hang of it?
18:59
SPEAKER_00: Absolutely.
19:02
SPEAKER_01: So you're right.
19:02
SPEAKER_01: We've spent a lot of time talking to people about stacking,
19:05
SPEAKER_01: about stack diffs,
19:06
SPEAKER_01: and so stack VRs,
19:07
SPEAKER_01: as they're called on GitHub.
19:10
SPEAKER_01: And I think that we've had a lot of practice about how to talk about it.
19:14
I think the way I like to introduce stacking is by talking
19:17
around the problem that it solves.
19:19
SPEAKER_01: And the problem that it solves is in any given engineering organization,
19:22
you have a bunch of engineers who at any given moment might
19:24
tell you they're blocked.
19:25
SPEAKER_01: They're blocked on code review.
19:27
SPEAKER_01: And I think when you really start to ask, well, why are they blocked?
19:31
SPEAKER_01: Why is CodeReview blocking them?
19:32
SPEAKER_01: What it comes down to is in most engineering organizations,
19:35
SPEAKER_01: the way development works is you have a main line.
19:38
SPEAKER_01: So you have a main branch or a master branch.
19:41
SPEAKER_01: You go ahead, you fork off of that.
19:43
SPEAKER_01: You go ahead and you create your feature branch.
19:45
And then you kind of get stuck because before you can continue
19:49
to develop on top of
19:49
SPEAKER_01: that feature branch,
19:51
SPEAKER_01: it needs to merge back into main.
19:54
SPEAKER_01: Exactly.
19:55
You need to merge it back because you only create feature
19:58
branches off of main.
19:59
SPEAKER_01: The downside of that,
20:00
SPEAKER_01: though,
20:01
SPEAKER_01: is that your personal development process now becomes blocked on review.
20:05
SPEAKER_01: Because you've created some PR,
20:07
SPEAKER_01: you've created the perfect way to do,
20:10
SPEAKER_01: I don't know,
20:11
SPEAKER_01: authentication,
20:11
SPEAKER_01: login.
20:12
SPEAKER_01: And you now want to build forgot password.
20:14
And so your options here are you can either add forgot password
20:17
back into that branch,
20:19
but suddenly that branch is going to become really big and your
20:21
reviewer is not
20:22
SPEAKER_01: going to be able to review it.
20:23
Or if they review it,
20:23
they're going to give it a rubber stamp,
20:25
looks good to me.
20:26
SPEAKER_00: Yeah, yeah.
20:29
SPEAKER_00: There's no point in a code review when you do that.
20:32
SPEAKER_01: Exactly.
20:33
And so Google has a funny academic paper where they refer to it
20:37
as reviewer frustration.
20:38
SPEAKER_01: As pull requests get longer,
20:41
reviewers buy out of the process and end up just approving
20:43
things rather than
20:44
SPEAKER_01: actually reading them.
20:46
SPEAKER_01: And so the solution...
20:49
SPEAKER_01: The solution which Meta stumbled upon is what they called stacking.
20:52
SPEAKER_01: Google, I've heard, has stumbled on something similar.
20:55
SPEAKER_01: There might be some cross-pollination with Meta there.
20:57
SPEAKER_01: But the idea is you create your feature branch,
20:59
SPEAKER_01: you put it up for review,
21:00
SPEAKER_01: and rather than waiting for it to be reviewed for you to start building,
21:03
SPEAKER_01: you just branch off of that and keep going.
21:05
SPEAKER_01: And you create another feature branch.
21:07
SPEAKER_01: And then you put that up for review.
21:09
SPEAKER_01: And then while you wait for that to be reviewed,
21:10
SPEAKER_01: you create another feature branch and you create that to be reviewed.
21:13
Now,
21:13
people who have been using Git long enough,
21:16
most of them have tried something like this.
21:18
SPEAKER_01: And I think inevitably you end up with a few questions.
21:20
One is,
21:21
well,
21:21
what if like someone rightly calls out that that earlier PR
21:25
needs to be updated?
21:27
SPEAKER_01: The general answer is that's actually fine.
21:29
SPEAKER_01: Generally,
21:30
SPEAKER_01: when you need to update a PR,
21:31
SPEAKER_01: you're not changing the abstraction boundary between PRs.
21:34
An example of how you might break it out is if you're building
21:36
a new feature,
21:37
SPEAKER_01: I have server and then I have front end.
21:39
SPEAKER_01: And so even if someone's like, hey, can we refactor the server?
21:43
SPEAKER_01: Usually the endpoint abstraction is actually staying in place.
21:46
SPEAKER_01: And so the next PR can continue to be stacked on top of it.
21:50
The other question is,
21:52
well,
21:53
how do I know what I'm supposed to break up?
21:56
SPEAKER_01: And I think for that,
21:57
SPEAKER_01: generally,
21:58
SPEAKER_01: and this is actually really interesting to me,
22:00
SPEAKER_01: generally developers build their systems in order,
22:03
SPEAKER_01: right?
22:04
SPEAKER_01: You might start building the server and then building the front end.
22:06
I think stacking is just a way to communicate that same order
22:09
to your reviewers.
22:11
SPEAKER_01: A lot of people do this automatically using commits, right?
22:14
SPEAKER_01: They just say like,
22:15
SPEAKER_01: oh,
22:15
SPEAKER_01: I'm going to build commits in the order I would build them,
22:17
SPEAKER_01: and you should review them by commits.
22:19
SPEAKER_01: Stacking goes a step further and says,
22:21
SPEAKER_01: well,
22:21
SPEAKER_01: if the first three commits in a branch are approved,
22:24
SPEAKER_01: why not merge those while you wait for the latter ones to be approved?
22:27
SPEAKER_01: And that has a few other benefits.
22:29
SPEAKER_01: You can run CRM separately,
22:30
SPEAKER_01: you can reduce merge conflicts,
22:32
SPEAKER_01: and you tend to be able to reduce your time to merge.
22:36
Because if there are sticky things,
22:38
you get stuck there but can still merge a lot of other code.
22:41
SPEAKER_01: The other advantages then happen sort of afterwards of...
22:44
SPEAKER_01: In the case that there's an error or something like that,
22:46
SPEAKER_01: you can go ahead,
22:47
SPEAKER_01: quickly identify which PR was offending.
22:49
And if you tell me a 10-line PR is offending as opposed to a
22:52
2,000-line PR,
22:53
SPEAKER_01: it's much easier for me to identify the issue.
22:56
SPEAKER_00: So basically, you get to work in just smaller increments.
23:00
SPEAKER_00: It's almost like forking off of your...
23:03
SPEAKER_00: Your current branch,
23:04
SPEAKER_00: you know,
23:04
SPEAKER_00: like smaller and smaller,
23:06
SPEAKER_00: have a bunch of small forks,
23:07
SPEAKER_00: except it's just a tooling,
23:09
SPEAKER_00: right?
23:09
SPEAKER_00: That like, so why do people not do this?
23:13
SPEAKER_00: You know,
23:13
SPEAKER_00: like,
23:13
SPEAKER_00: just forget about like the tooling that might have,
23:15
SPEAKER_00: but you could just literally,
23:17
SPEAKER_00: you know,
23:17
SPEAKER_00: create a branch and then create yet another branch.
23:19
But I guess it's just really complex to look at what's happening,
23:22
complex to merge it back.
23:23
SPEAKER_00: Yeah.
23:25
SPEAKER_01: I think Git makes it scary, and I think GitHub makes it scary.
23:28
So in Git,
23:29
the most basic way to do this would be to create branches off
23:32
of branches.
23:32
But then if you need to update something,
23:34
you need to go ahead and rebase it.
23:35
Now,
23:36
that's where I just use a really scary word for a lot of people,
23:38
rebase.
23:38
SPEAKER_00: Rebase.
23:39
SPEAKER_00: Oh, we hate rebasing.
23:40
I mean,
23:41
okay,
23:41
I can't talk for anyone,
23:42
but I'm going to admit it always freaks me.
23:45
SPEAKER_00: Having the rebase conflict and figuring out what I need to do, yeah.
23:52
SPEAKER_01: It scares me, and I've been doing this for years.
23:54
SPEAKER_01: I actually have a Stack Overflow post bookmark,
23:56
SPEAKER_01: which is the three types of rebase,
23:58
SPEAKER_01: because depending on how many arguments you give it,
24:00
SPEAKER_01: it behaves differently.
24:01
SPEAKER_01: And so super frightening.
24:03
SPEAKER_01: I think that's the place.
24:05
I'm very fortunate that I learned this at Meta because I think
24:07
what Meta has and
24:08
what I've learned Google has and what I've learned places like
24:12
Stripe and Uber have
24:12
SPEAKER_01: built are internal tools to handle this for you,
24:15
SPEAKER_01: where you can basically say,
24:16
SPEAKER_01: like,
24:16
SPEAKER_01: I'm creating a stack now.
24:18
SPEAKER_01: I now need to update a branch in my stack.
24:20
SPEAKER_01: Do that for me.
24:21
SPEAKER_01: And under the hood,
24:22
it takes care of this intricate set of rebases that while you
24:25
can do by hand are
24:26
SPEAKER_01: easy to mess up.
24:27
SPEAKER_01: Now, there's a lot of tooling that does that client side portion.
24:32
SPEAKER_01: The other half of this is the pull request side.
24:36
If wherever you have your pull request does not have support
24:41
for stacking,
24:42
SPEAKER_01: I think stacks can look very foreign.
24:44
Because what you end up looking with is a lot of PRs that
24:48
appear to be recursively
24:50
SPEAKER_01: merging into different branches that they were all based off of.
24:53
SPEAKER_01: And so it can be hard to understand what's going on.
24:55
SPEAKER_01: And without any clear explanation or affordance in the UI,
24:58
SPEAKER_01: your reviewer might become confused too.
25:00
SPEAKER_00: So I'm kind of like wondering if you have a theory,
25:04
SPEAKER_00: why did,
25:04
SPEAKER_00: you know,
25:05
SPEAKER_00: like a few companies like Meta and Google invent this?
25:10
SPEAKER_00: And why is it just not part of Git or GitHub?
25:15
SPEAKER_00: I'm sure you've speculated about this.
25:16
SPEAKER_00: You must have.
25:19
SPEAKER_01: Certainly.
25:19
SPEAKER_01: So why I think some companies built this.
25:23
I think first and foremost,
25:24
the current tools weren't solving their own needs.
25:26
SPEAKER_00: Current tools being Git and GitHub-like.
25:29
SPEAKER_01: Git or whatever they had at the time, right?
25:33
SPEAKER_00: Mercurial.
25:33
SPEAKER_00: Yeah.
25:36
SPEAKER_01: I think they were shipping a high volume.
25:37
SPEAKER_01: I think they were really early on monorepos.
25:39
And I think that they were getting to this point where,
25:42
okay,
25:43
I'm blocked on review.
25:44
SPEAKER_01: Progressively larger parts of this org are blocked on review.
25:47
SPEAKER_01: And as you sort of like start to grow up as a company,
25:50
SPEAKER_01: you realize like,
25:52
SPEAKER_01: oh,
25:52
SPEAKER_01: like if developers are spending 20% of their time waiting,
25:55
that's actually a lot of inefficiency that's being created
25:58
across the org,
25:58
SPEAKER_01: right?
25:59
SPEAKER_01: Most people that run an org of over 100 engineers would be like,
26:02
SPEAKER_01: I would love to have 20 more engineers.
26:05
SPEAKER_01: Second thing is that they saw the potential impact.
26:10
SPEAKER_01: So I think for a lot of people,
26:11
SPEAKER_01: they sort of like saw that lift of like,
26:14
SPEAKER_01: okay,
26:15
SPEAKER_01: Facebook,
26:16
SPEAKER_01: for example,
26:16
really prioritized being able to improve their ship velocity
26:19
for whatever reason.
26:20
SPEAKER_01: It mattered to them a lot.
26:21
SPEAKER_01: Google, I think, did as well of, well, we're a web company.
26:24
SPEAKER_01: If we can continuously deploy, we can continue to improve things.
26:29
SPEAKER_01: And thirdly, I think they had near infinite resources.
26:31
SPEAKER_01: And this is the thing that I think most of us don't have,
26:33
SPEAKER_01: which is they saw the potential impact,
26:35
SPEAKER_01: they saw a problem,
26:36
SPEAKER_01: they had the resources,
26:37
SPEAKER_01: and so they could build a solution.
26:39
I think it's much harder when you're a startup or even just a
26:43
smaller scale company
26:44
than one of the three most valuable companies in the world to
26:48
be able to say,
26:49
SPEAKER_01: I'm going to take a few engineers and I'm going to build this.
26:51
SPEAKER_00: Yeah.
26:52
SPEAKER_00: I mean,
26:52
SPEAKER_00: these are companies who are printing money despite having,
26:55
SPEAKER_00: you know,
26:55
SPEAKER_00: thousands of engineers,
26:56
SPEAKER_00: even tens of thousands.
26:57
SPEAKER_00: So, so they, I mean, obviously they're going to like, like be rational.
27:01
SPEAKER_00: They're not just going to create a team out of thin air,
27:03
SPEAKER_00: but this is very interesting because what you mentioned is,
27:05
SPEAKER_00: I remember I talked with Nicole,
27:07
SPEAKER_00: with the doctor,
27:08
SPEAKER_00: with Nicole Forsgren about developer productivity.
27:11
SPEAKER_00: And, you know, one of the things that,
27:13
SPEAKER_00: came up when we discussed this,
27:15
SPEAKER_00: well,
27:15
SPEAKER_00: you do want to measure certain things just to get a sense.
27:18
SPEAKER_00: And one of them is typically code review.
27:20
SPEAKER_00: Or how long does it take from writing the code to getting it in there?
27:23
SPEAKER_00: So I have a feeling that companies like Meta,
27:27
SPEAKER_00: Google,
27:27
SPEAKER_00: and some of these bigger ones who did have the resources,
27:29
SPEAKER_00: they start to measure it earlier.
27:30
SPEAKER_00: Therefore, they decided, let's build better tooling.
27:34
SPEAKER_00: And one interesting part that I've only learned recently,
27:37
SPEAKER_00: I was talking with one of the three Linux fellows,
27:41
SPEAKER_00: Greg
27:46
And he told me something interesting,
27:48
that Git was created by the Linux community.
27:50
SPEAKER_00: Well,
27:50
SPEAKER_00: specifically not Star Wars,
27:52
SPEAKER_00: but it was to solve their issue of Linux development,
27:55
SPEAKER_00: where they had about three to four changes per hour or so.
28:01
SPEAKER_00: And it was an email-based list.
28:03
SPEAKER_00: So they created Git to solve their own problem.
28:06
SPEAKER_00: And now I think they have something like 40,
28:09
SPEAKER_00: 50 changes per day or per hour,
28:11
SPEAKER_00: something like that.
28:12
SPEAKER_00: I think it's per day, actually.
28:14
SPEAKER_00: That's their rate.
28:15
SPEAKER_00: And they built it.
28:17
SPEAKER_00: It's all for them.
28:17
SPEAKER_00: They open-sourced it, but it was for them.
28:20
SPEAKER_00: There's this Linux.
28:22
SPEAKER_00: It's also a monolith.
28:23
SPEAKER_00: The changes are...
28:25
SPEAKER_00: pretty small for the most part.
28:27
SPEAKER_00: There's not as much, let's say, user-facing code, etc.
28:30
SPEAKER_00: But they sell for themselves.
28:32
So I wonder if they just didn't really have the problem that
28:35
StackDiff does because
28:36
SPEAKER_00: they don't have...
28:38
SPEAKER_00: If you look at the average Linux change,
28:40
SPEAKER_00: it's a lot smaller than,
28:41
SPEAKER_00: let's say,
28:42
SPEAKER_00: a new feature being added to some of the codebases like Meta or Uber.
28:47
I think you hit the nail on the head with it's really important
28:50
to use tools that
28:51
SPEAKER_01: are built for your use case.
28:52
SPEAKER_01: I think that open source development generally,
28:55
SPEAKER_01: and Linux specifically,
28:57
SPEAKER_01: stacking makes no sense for.
28:58
SPEAKER_01: So people always ask me sort of like, who is stacking not for?
29:01
SPEAKER_01: My answer is open source.
29:02
SPEAKER_01: And the reasoning is that in open source,
29:05
SPEAKER_01: you're dealing with a lot of untrusted authors who come to you.
29:08
SPEAKER_01: They're like, hey, I have this feature.
29:10
SPEAKER_01: It's partially built.
29:11
SPEAKER_01: I don't have tests for it yet.
29:13
SPEAKER_01: I haven't fully tested it.
29:14
SPEAKER_01: Can I merge it?
29:16
And it's funny in that in a company,
29:18
if you're like,
29:18
I have this feature,
29:19
it's partially built.
29:20
Maybe you even have some tests for it,
29:21
but the feature's not totally there.
29:23
SPEAKER_01: You're like, I know you're going to be here tomorrow, right?
29:26
SPEAKER_01: And so, cool.
29:27
SPEAKER_01: Thank you for doing your job.
29:28
I appreciate that you're contributing back to the code base and
29:30
that you're going
29:31
SPEAKER_01: to prevent me from having to resolve rebase conflicts later,
29:33
SPEAKER_01: merge conflicts.
29:35
SPEAKER_01: I think that in Linux, you kind of see the opposite, right?
29:38
SPEAKER_01: And if you've ever spent time on those email lists,
29:40
SPEAKER_01: you'll see people throw back and forth sort of like patches.
29:43
And what will happen is that they'll say like,
29:45
oh,
29:45
like this isn't complete.
29:47
SPEAKER_01: You're not going to work on this edge case, right?
29:49
SPEAKER_01: And rather than saying like, you know what?
29:51
SPEAKER_01: Like we can put this behind a feature flag and merge it.
29:53
SPEAKER_01: They're instead like,
29:54
SPEAKER_01: go take this back,
29:56
SPEAKER_01: fix it,
29:57
SPEAKER_01: fully finish it,
29:58
SPEAKER_01: wrap it in tests and documentation and everything else.
30:00
SPEAKER_01: And only then can you merge it.
30:03
and that makes a ton of sense in open source right having
30:05
worked on open source
30:06
repos myself the biggest fear is that the contributors is going
30:09
to disappear it
30:10
happens more often than we like to admit they get busy
30:12
something happens and so in
30:13
that case you don't want to land partial stacks you want to
30:16
make sure that the
30:17
thing is fully complete before you even consider like reviewing
30:20
it truthfully yeah
30:22
and this is actually like when i had a discussion this trust
30:26
came up so much with
30:27
SPEAKER_00: with open source so i
30:29
SPEAKER_00: It's interesting how,
30:31
SPEAKER_00: like,
30:31
SPEAKER_00: you know,
30:32
SPEAKER_00: as you said,
30:32
SPEAKER_00: the context,
30:33
SPEAKER_00: like inside a company where you have people,
30:34
you can actually trust them that they'll follow up or if they
30:36
don't follow up,
30:37
SPEAKER_00: it's not a big deal.
30:38
SPEAKER_00: You know, like it might be deleted.
30:41
SPEAKER_00: Interesting how this context makes such a big difference.
30:44
SPEAKER_00: Now,
30:44
SPEAKER_00: another thing that is unique,
30:46
SPEAKER_00: I guess,
30:47
SPEAKER_00: maybe not just to large companies,
30:50
SPEAKER_00: but it kind of started there,
30:51
SPEAKER_00: monorepos.
30:53
SPEAKER_00: So what was,
30:55
SPEAKER_00: you know,
30:55
Meta and Google were probably one of the two companies to start
30:58
doing monorepos.
30:59
SPEAKER_00: You've seen it at Meta.
31:00
What was the history with the monorepo story there and the pain
31:05
points you saw?
31:06
SPEAKER_00: Yeah.
31:07
SPEAKER_00: Yeah, totally.
31:09
SPEAKER_01: I think you have to remember,
31:10
SPEAKER_01: so first things first,
31:11
SPEAKER_01: you have to remember that source control at Meta has a funny history.
31:13
SPEAKER_01: When that company started,
31:14
SPEAKER_01: and this is well before my time,
31:17
rumor has it the way they did source control is emailing back
31:19
and forth zip files.
31:21
SPEAKER_01: And so at some point, I mean, that wasn't so uncommon, right?
31:27
SPEAKER_01: Git itself is a fairly new innovation, came around in 2007, 2008.
31:29
SPEAKER_01: And so
31:33
SPEAKER_01: I think that when Facebook started to do source control,
31:38
SPEAKER_01: they had this idea that,
31:40
SPEAKER_01: well,
31:41
SPEAKER_01: we want to be able to version everything together.
31:44
SPEAKER_01: And I think there are a lot of advantages to that.
31:46
SPEAKER_01: I think one is it makes collaboration a lot easier.
31:48
SPEAKER_01: It makes discovery a lot easier.
31:50
SPEAKER_01: I think it makes it easy to put dependencies between different services.
31:54
SPEAKER_01: So I think all of us have had that case where it's like,
31:56
SPEAKER_01: oh,
31:56
SPEAKER_01: well,
31:57
SPEAKER_01: I need to change this,
31:58
but that also means I need to change the call sites or that
32:00
also means I need to
32:01
SPEAKER_01: change this other thing.
32:02
SPEAKER_01: Um, monorepos allow you to do that, right?
32:05
Because what it's saying is there's one like unified state of
32:08
the world that we can
32:09
SPEAKER_01: look at,
32:10
SPEAKER_01: can make sense and everything's good.
32:11
SPEAKER_01: We can test against even,
32:13
SPEAKER_01: uh,
32:14
SPEAKER_01: poly repos,
32:15
SPEAKER_01: the opposite of that when an organization has many,
32:17
SPEAKER_01: many repositories,
32:19
SPEAKER_01: I think have different advantages.
32:21
SPEAKER_01: Um, you can version them separately.
32:23
SPEAKER_01: You can, uh, have different practices within them.
32:25
SPEAKER_01: You can, uh,
32:27
SPEAKER_01: have entirely different owners within them.
32:29
I actually think that GitHub on many companies pushes a culture
32:32
of polyrepos
32:33
SPEAKER_01: because it's what's common in open source.
32:36
SPEAKER_01: So again,
32:37
SPEAKER_01: going back to this narrative of sort of like,
32:39
SPEAKER_01: oh,
32:39
SPEAKER_01: open source is different than closed source.
32:41
SPEAKER_01: I think in open source you actually want things like,
32:43
I don't want to have to version at the same time that all of my
32:46
dependencies are
32:46
SPEAKER_01: versioning on.
32:47
SPEAKER_01: That doesn't make sense.
32:48
SPEAKER_01: I'm not really collaborating with my dependencies.
32:50
Yes,
32:50
I depend on other open source projects,
32:52
but perhaps I'm not contributing to them.
32:54
SPEAKER_01: I think within a company, those constraints change.
32:57
SPEAKER_01: Within a company,
32:58
SPEAKER_01: you're looking at constraints of like,
32:59
SPEAKER_01: I want to collaborate with other people.
33:01
SPEAKER_01: I want to be able to version things.
33:03
And so you get pushed towards sort of like one unified source
33:06
of truth for like,
33:07
SPEAKER_01: what is the state of our code base?
33:11
SPEAKER_01: By the time I was there, Facebook wasn't even one monorepo.
33:14
SPEAKER_01: It was multiple monorepos.
33:15
SPEAKER_01: I remember hearing someone refer to it as a polyleth,
33:18
SPEAKER_01: which made me laugh a lot,
33:20
SPEAKER_01: where there was the main web monorepo.
33:22
SPEAKER_01: There was Instagram, which was separate.
33:24
SPEAKER_01: There were the mobile apps, which were separate than that.
33:26
SPEAKER_01: There was internal tooling.
33:28
SPEAKER_01: Yeah, which was completely separate.
33:30
SPEAKER_01: And they were on a mission to try and unify these into all of them,
33:34
SPEAKER_01: into one massive monorepo,
33:36
because the negatives of not having one monorepo were starting
33:39
to affect the company.
33:40
Some examples of that might be that if you wanted to do an
33:42
end-to-end test on
33:43
SPEAKER_01: mobile apps,
33:44
I remember there was some tests that would pull down a copy of
33:47
the other repository,
33:49
SPEAKER_01: spin up a server from that repository,
33:51
SPEAKER_01: and then spin up the mobile app such that they could test against that.
33:55
SPEAKER_01: And that led to all sorts of conflicts.
33:57
SPEAKER_01: Right.
33:58
SPEAKER_01: Like it makes your CI flakier.
34:00
It makes it hard to version against when we had dependencies
34:03
between repositories.
34:04
SPEAKER_01: That was a nightmare.
34:05
SPEAKER_01: And so by bringing everything under one roof,
34:07
SPEAKER_01: it allowed you to really collaborate across the company more actively.
34:11
And it allowed you to collaborate across the company more
34:14
actively and allowed you
34:15
SPEAKER_01: to share dependencies.
34:18
SPEAKER_00: So Facebook was being slowed down despite having a few mega repos,
34:24
SPEAKER_00: if you will,
34:24
SPEAKER_00: or these large ones.
34:25
SPEAKER_00: So they saw this advantage.
34:26
In the end,
34:27
did they move to one unified monitor between iOS,
34:31
Android,
34:32
and even web?
34:34
SPEAKER_01: When I was leaving, they were on the journey to.
34:36
SPEAKER_01: Wow.
34:37
SPEAKER_01: I remember hearing recently that they're still on the journey to.
34:40
SPEAKER_00: That's really, it's such a different code, especially for native mobile.
34:43
SPEAKER_00: Yeah.
34:46
SPEAKER_01: But the advantage of it,
34:47
SPEAKER_01: though,
34:48
SPEAKER_01: and again,
34:48
SPEAKER_01: going back to it,
34:49
is I think that it allows you to enforce a common set of eng
34:51
practices and requirements,
34:54
SPEAKER_01: right?
34:54
Like the idea of like,
34:56
oh,
34:56
every repository should have at least one reviewer,
34:59
right?
34:59
SPEAKER_01: In a company, yeah, that probably is a company-wide requirement.
35:03
SPEAKER_01: But in open source,
35:05
I can't be guaranteed that every open source project wants the
35:08
same set of requirements,
35:09
SPEAKER_01: right?
35:09
SPEAKER_00: And it's interesting because it's not just meta.
35:11
Other companies,
35:12
when they grow a certain size,
35:14
they start to have the same journey.
35:15
SPEAKER_00: I've seen it at Uber.
35:17
SPEAKER_00: As far as I understand, Shopify is doing similar things.
35:20
SPEAKER_00: I assume that as companies go larger,
35:23
SPEAKER_00: it typically starts with,
35:25
SPEAKER_00: oh,
35:26
SPEAKER_00: what about a vulnerability of a library being updated?
35:28
SPEAKER_00: Would it not be nice if it was in one place?
35:31
SPEAKER_00: Now...
35:33
You've seen more of these obviously through Graphite when
35:37
you're integrating with
35:38
SPEAKER_00: some of these companies.
35:40
Do you see some trends of when monorepos are starting to become important?
35:45
SPEAKER_00: And if they are starting to become more important,
35:47
SPEAKER_00: why do you think we're not seeing GitHub support too much of this still?
35:53
SPEAKER_01: It's a great question.
35:54
So first and foremost,
35:56
I think that we're watching an industry-wide move towards monorepos.
35:59
Most companies we're talking to are already in a monorepo or
36:02
are migrating to a monorepo.
36:04
SPEAKER_01: At least for the people that we talk with in Silicon Valley,
36:06
SPEAKER_01: a lot of people have decided,
36:08
SPEAKER_01: you know,
36:08
SPEAKER_01: this is actually the future.
36:09
SPEAKER_01: I see the advantage.
36:10
SPEAKER_01: The tooling's there.
36:12
SPEAKER_01: Someone should be moving here.
36:14
And so we're actually seeing a broader sort of like trend
36:17
towards monorepos.
36:18
SPEAKER_01: Why do I not think GitHub supports it?
36:20
SPEAKER_01: I think it doesn't make sense for open source.
36:22
I think we're just starting to see the advent of monorepos in
36:26
open source.
36:27
SPEAKER_01: I think React is closer to a monorepo than not.
36:29
I think that Vercel follows the same pattern,
36:32
probably a lot of the same energy there.
36:35
I think we're seeing a lot of repos say,
36:37
well,
36:38
maybe multiple packages should be in one repo.
36:42
But there are a lot of advantages to being able to put them all together.
36:46
SPEAKER_01: But I still think it's not the norm.
36:48
When you look at the JavaScript ecosystem or the Rust ecosystem
36:51
or the Python ecosystem,
36:52
we still live in a world of many separate packages from many
36:54
different authors with
36:55
SPEAKER_01: many different cultures.
36:57
SPEAKER_01: And the way that you want to manage those is through polyrepos.
36:59
SPEAKER_00: And then when you say you see a lot of companies having,
37:02
SPEAKER_00: can you just get a sense of what types of companies in terms of the size,
37:05
SPEAKER_00: the funding stage?
37:07
SPEAKER_00: I appreciate you might not be able to say specific names here,
37:11
SPEAKER_00: do not advertise it,
37:11
but it was pretty surprising for me to say that a lot of the
37:15
companies you work with.
37:16
SPEAKER_01: A lot of them are moving.
37:18
I'd say that Graphite tends to work with Silicon Valley based
37:21
companies that are
37:22
SPEAKER_01: somewhere between,
37:22
SPEAKER_01: call it like 100 and high thousands of people,
37:27
SPEAKER_01: maybe 10,000 people.
37:29
SPEAKER_01: And so for us,
37:30
I think it's a lot of the forward thinkers who are really
37:33
investing in developer velocity,
37:35
SPEAKER_01: as you were talking about earlier,
37:36
SPEAKER_01: that are starting to invest in the like,
37:37
SPEAKER_01: how do I speed up my team,
37:39
are very quickly realizing that the coordination overhead of
37:42
poly repos makes it
37:44
SPEAKER_01: harder to speed up your team.
37:46
It both slows your team down in code review and weird,
37:49
insidious and second order effect ways.
37:52
SPEAKER_01: And it makes it hard to enforce a common culture.
37:55
SPEAKER_01: So when we talk to DevTools teams and they're like,
37:57
SPEAKER_01: oh,
37:57
SPEAKER_01: we want to migrate CI,
38:00
SPEAKER_01: when you're in a monorepo,
38:01
SPEAKER_01: that is a somewhat straightforward operation.
38:04
SPEAKER_01: When you're in a polyrepo, that is a multi-year journey that you're on.
38:06
SPEAKER_00: Oh, yeah.
38:09
SPEAKER_00: I think I can emphasize with this.
38:11
SPEAKER_00: Yeah, and I think what you said, it needs to be some investment.
38:14
SPEAKER_00: It doesn't just happen like this.
38:15
SPEAKER_00: I mean,
38:16
SPEAKER_00: there's better tools now,
38:17
but you need to have dedicated people looking at vendors who
38:21
are doing it,
38:21
SPEAKER_00: etc.
38:21
So it probably makes sense that the teams who are investing in
38:26
better tooling are
38:27
SPEAKER_00: probably the ones who are looking at these alternatives.
38:31
SPEAKER_01: And there's probably some selection effect in who we talk to.
38:35
SPEAKER_01: When teams want to invest in tooling, they start talking to us.
38:38
And at that point,
38:38
they've already started to consider the monorepo journey.
38:41
Well,
38:41
I mean,
38:41
you don't need to tell me because I've seen the benefits of monorepo.
38:44
I mean,
38:44
I see the costs as well,
38:45
but it was so good at Uber,
38:47
so many benefits.
38:48
And like I would,
38:49
you know,
38:50
if I had a magic wand,
38:51
I would say,
38:51
let's do monorepo.
38:52
SPEAKER_00: Obviously, there's a cost.
38:53
SPEAKER_00: So I see this.
38:55
I'm just surprised that it's kind of an encouraging way that
38:58
there's so much of this happening.
39:00
SPEAKER_00: Yeah.
39:01
SPEAKER_01: I think another place where things got confused,
39:04
SPEAKER_01: let's call it,
39:05
SPEAKER_01: is that people associate microservices with many repos.
39:08
SPEAKER_00: I think,
39:09
SPEAKER_01: again,
39:09
that's because that's a way that I think GitHub very much
39:13
encourages you to think
39:13
SPEAKER_01: of like,
39:14
SPEAKER_01: oh,
39:14
SPEAKER_01: you have abstractions between code,
39:15
SPEAKER_01: you should have different repos.
39:17
And I think that I have a friend also from Facebook who likes
39:21
to joke that the
39:22
SPEAKER_01: discovery of monorepos is really just a discovery of folder structure.
39:26
SPEAKER_01: Yeah.
39:26
it's like well yeah so you have multiple you have multiple uh
39:29
different pieces of
39:29
code but you can just put them in different folders that's okay
39:32
yeah i think you
39:34
know the probably the one of the best examples uber is known
39:36
for advertising how
39:37
they have 5 000 plus microservices and they do have mono repos
39:40
i think per language
39:41
like four or five but yes you you can it's not dependent it's
39:45
totally independent
39:46
SPEAKER_00: from one another
39:47
SPEAKER_00: Exactly right.
39:48
SPEAKER_00: Now,
39:49
SPEAKER_00: one other trend that's happening now everywhere,
39:52
SPEAKER_00: especially with the forward thinking companies,
39:54
SPEAKER_00: is AI tools for code generation or for coding.
39:57
SPEAKER_00: How do you see this changing how code is written,
40:02
SPEAKER_00: both through the companies that you work at or maybe even at Graphite?
40:06
SPEAKER_01: It's incredible.
40:07
It's been a whole conversation at Graphite and it's definitely
40:09
been a whole
40:10
SPEAKER_01: conversation with a lot of the companies we work with.
40:12
I think the most immediate impact of how will AI impact
40:16
software engineering is
40:17
SPEAKER_01: there will be more software written.
40:19
SPEAKER_01: There is going to be more code written as the tools make it work.
40:22
SPEAKER_01: easier and easier for developers to ship code.
40:25
SPEAKER_01: I think what's not as obvious is the second-order effect, right?
40:28
SPEAKER_01: If you have more code written, well, what needs to happen?
40:30
SPEAKER_01: Well,
40:31
SPEAKER_01: more code needs to be reviewed,
40:32
SPEAKER_01: and if more code is reviewed,
40:33
SPEAKER_01: more code needs to be tested,
40:34
SPEAKER_01: and more code needs to be merged,
40:35
SPEAKER_01: and more code needs to be deployed.
40:37
That's suddenly the place where we're seeing people run into
40:40
issues where they say,
40:42
SPEAKER_01: okay,
40:43
I think that we're ready to adopt one of these AI tools that
40:46
helps us generate code,
40:47
SPEAKER_01: that helps our developers code faster.
40:50
SPEAKER_01: Let's do that.
40:50
And then what immediately happens is we see the volume of pull
40:53
requests go up.
40:53
SPEAKER_01: We see the size of those pull requests go up.
40:55
SPEAKER_01: We see the amount of bugs or CI bills also go up.
40:59
And I think getting ready for that transformation is a thing
41:01
that the industry is
41:02
SPEAKER_01: just starting to grapple with.
41:03
SPEAKER_00: This episode was brought to you by Sentry.
41:06
Buggy lines of code and long API calls are impossible to debug
41:09
and random app
41:10
SPEAKER_00: crashes are things no software engineer is a fan of.
41:13
This is why over 4 million developers use Sentry to fix errors
41:17
and crashes and
41:18
SPEAKER_00: solve hidden or tricky performance issues.
41:21
SPEAKER_00: Sentry calls debugging time in half,
41:23
SPEAKER_00: no more soul-crushing lock sifting,
41:25
SPEAKER_00: or vague user reports like,
41:27
SPEAKER_00: it broke,
41:27
SPEAKER_00: fix it.
41:29
SPEAKER_00: Get the context you need to know,
41:30
SPEAKER_00: what happened,
41:31
SPEAKER_00: when it happened,
41:32
SPEAKER_00: and the impact,
41:33
SPEAKER_00: down to the device,
41:34
SPEAKER_00: browser,
41:34
SPEAKER_00: and even a replay of what the user did before the error.
41:37
Sentry will alert the right dev on your team with the exact
41:40
broken line of code so
41:41
SPEAKER_00: they can push a fix fast,
41:43
or let Autofix handle repetitive fixes so your team can focus
41:46
on the real problems.
41:49
Sentry helped Monday.com reduce their errors by 60% and sped up
41:52
time to resolution
41:53
SPEAKER_00: for next door by 45 minutes per dev per issue.
41:57
Get your whole team on Sentry in seconds by heading to
41:59
sentry.io slash pragmatic.
42:02
SPEAKER_00: That is S-E-N-T-R-Y dot I-O slash pragmatic.
42:07
Or use the code pragmatic on sign up for three months on the
42:10
team plan and 50,000
42:11
SPEAKER_00: errors per month for free.
42:13
Now,
42:14
you know,
42:14
this is speculative,
42:15
but it's good to speculate sometimes.
42:17
SPEAKER_00: How do you think successful companies will cope with it?
42:20
SPEAKER_00: And this is, we can just speculate, right?
42:22
SPEAKER_00: Like we know there's going to be more code being churned out.
42:25
SPEAKER_00: Devs will be able to generate stuff faster, build features faster.
42:29
SPEAKER_00: And yes, it will have all these.
42:31
SPEAKER_00: A lot of them will just tap, tap, tap, accept.
42:33
SPEAKER_00: They're going to not notice certain things being there.
42:36
SPEAKER_00: We'll even have like attacks of like,
42:38
SPEAKER_00: you know,
42:38
SPEAKER_00: some companies trying to have malicious,
42:40
SPEAKER_00: a bit like,
42:40
SPEAKER_00: you know,
42:41
SPEAKER_00: stack overflow,
42:41
SPEAKER_00: copy paste,
42:42
SPEAKER_00: and you have the memory leak.
42:43
SPEAKER_00: We'll have more of those.
42:45
SPEAKER_00: What, you know, let's just do the thought experiment.
42:48
SPEAKER_00: Like how, how will good teams deal with this?
42:54
SPEAKER_01: I think AI necessarily is both the problem and the solution.
42:58
So I agree with,
42:59
I think that the fastest moving teams are going to adopt AI.
43:02
I think we've moved past the world where you can say like,
43:06
ah,
43:06
this is a fad,
43:06
this will pass.
43:07
AI is going to mean a lasting change on the software
43:10
development industry,
43:11
period.
43:12
SPEAKER_01: I think that you're right.
43:13
SPEAKER_01: It's going to create some problems, right?
43:14
SPEAKER_01: You have developers just clicking tab nonstop,
43:17
SPEAKER_01: probably importing code that they neither read nor reviewed.
43:19
SPEAKER_00: Well, we already have it, right?
43:19
SPEAKER_00: It's called vibe coding.
43:20
SPEAKER_01: Yeah.
43:23
SPEAKER_01: I love the Twitter memes about it.
43:25
SPEAKER_01: But yeah, vibe loading is a thing, right?
43:27
You have developers at your company who are probably just
43:29
talking to their computer,
43:30
SPEAKER_01: telling it like,
43:30
SPEAKER_01: yeah,
43:30
SPEAKER_01: create this website.
43:31
SPEAKER_01: No, I'm not going to read that code.
43:33
SPEAKER_01: And they're pointing up for code review.
43:34
SPEAKER_01: And I think that to me, that just emphasizes the point of code review.
43:38
Because if the author didn't read the code,
43:39
then someone else certainly should.
43:41
SPEAKER_01: And the question is, how are we going to make sure that happens?
43:43
SPEAKER_01: And so I have two answers for you.
43:46
SPEAKER_01: One is,
43:46
I think a lot of the practices that already existed at these
43:49
major companies,
43:50
SPEAKER_01: at the Google and Facebook,
43:51
SPEAKER_01: and to a lesser extent,
43:52
SPEAKER_01: but still an extent,
43:54
SPEAKER_01: Uber,
43:54
SPEAKER_01: Twitter,
43:55
SPEAKER_01: Snapchat,
43:56
which are used to seeing a high volume of changes written by a
44:01
diverse employee pool,
44:03
SPEAKER_01: a lot of those practices are going to percolate down.
44:05
SPEAKER_01: Because those used to be the problems of like,
44:07
SPEAKER_01: well,
44:07
SPEAKER_01: you only run into this issue if you have 100 developers.
44:10
SPEAKER_01: We see it on Twitter.
44:11
People are saying teams of 10 can start acting like teams of
44:13
100 developers.
44:14
SPEAKER_01: And so that also comes with the problems of hundreds developers.
44:18
The second solution is I think that AI is going to transform
44:20
the way that we view code review.
44:22
I think the largest change which has happened,
44:26
something which we are trying to...
44:30
keep up with internally is that I think we're seeing the same
44:34
advent that happened
44:35
when something like Grammarly or the grammar checker in Word
44:38
happened where AI can
44:39
SPEAKER_01: actually check a lot of the minutiae or mechanics of code review.
44:42
It can tell you that like,
44:43
yes,
44:43
this code works or yes,
44:44
this code does what it intended to do.
44:47
SPEAKER_01: The question though of code review then isn't
44:49
SPEAKER_01: Is this code literally bug free?
44:51
SPEAKER_01: It's is this aligned with what we wanted it to do?
44:54
SPEAKER_01: Right.
44:55
SPEAKER_01: Is this I originally we spec'd out the system.
44:58
SPEAKER_01: It was supposed to work this way.
45:00
SPEAKER_01: Or maybe we have built the system and it does work this way.
45:03
SPEAKER_01: What are the other effects of that?
45:04
SPEAKER_01: How does this interplay with everything else we do at this company?
45:07
SPEAKER_01: What's going to happen there?
45:08
SPEAKER_01: And I think that we're going to see AI tools built.
45:10
SPEAKER_01: We're building one ourselves,
45:11
SPEAKER_01: our AI reviewer,
45:12
that let developers automatically check this code for sort of
45:16
like the simple stuff,
45:18
SPEAKER_01: allowing the reviewers to focus on the more complicated stuff.
45:22
Do you think testing will become more important in this world
45:27
where we do have a
45:28
SPEAKER_00: lot more AI tools,
45:29
SPEAKER_00: a lot more AI code,
45:30
SPEAKER_00: a lot more pushing to production?
45:32
SPEAKER_00: Yeah.
45:34
SPEAKER_01: I think testing is going to change.
45:35
SPEAKER_01: And so is it going to become more important?
45:42
SPEAKER_01: Probably.
45:42
SPEAKER_01: In some form, definitely.
45:44
SPEAKER_01: Because as I said,
45:45
SPEAKER_01: I think that if developers are not putting thought into every keystroke,
45:49
SPEAKER_01: but instead we're just seeing this code written quickly,
45:52
SPEAKER_01: checking that code for correctness,
45:54
both the literal does the function work as intended and the
45:58
end-to-end correctness
45:58
SPEAKER_01: of are the buttons in the right place and is it usable,
46:02
SPEAKER_01: are critical.
46:04
SPEAKER_01: Now,
46:04
I think what's really interesting or the trend that we're
46:06
seeing is that it appears
46:07
SPEAKER_01: that AI can actually write tests for you in some limited cases.
46:11
SPEAKER_01: And when we look at some of the more advanced AI agents,
46:13
SPEAKER_01: they actually do this as part of their coding,
46:15
SPEAKER_01: right?
46:15
If you ask an agent to generate a change and also test it,
46:19
it'll do a pretty good job.
46:22
And I think that a lot of the previous types of tests that
46:24
before were restricted
46:26
SPEAKER_01: to QA people or to humans of like,
46:29
look at this website and try to like click around it with the
46:32
mouse where before we
46:33
SPEAKER_01: had to use,
46:34
SPEAKER_01: we had to use an intent testing tool.
46:36
SPEAKER_01: It was hard.
46:36
SPEAKER_01: We'd kind of be like, oh, go click at like XY location this.
46:41
SPEAKER_00: At Uber, we even hired testers who we call the sanity testing.
46:47
SPEAKER_00: You describe what to do, go there, take a credit card.
46:50
SPEAKER_00: And that was because I was on the payments team.
46:53
We actually had some of the fake credit cards that were take
46:55
one of them,
46:56
enter it there.
46:57
And we did it because we couldn't really automate it for
47:00
different reasons.
47:01
SPEAKER_00: The payments is a bit tricky, but yeah.
47:04
SPEAKER_00: It was,
47:05
SPEAKER_00: it sounded pretty terrible job to do,
47:08
SPEAKER_00: but someone had to do it and someone was being paid.
47:10
SPEAKER_00: So.
47:11
SPEAKER_01: Exactly.
47:12
SPEAKER_01: And it's, uh,
47:13
SPEAKER_01: I think that, again, the machines can start to do a lot of this work.
47:18
SPEAKER_01: I think the question that we have to grapple with then is,
47:21
SPEAKER_01: so what is the purpose of this review step,
47:23
SPEAKER_01: right?
47:24
SPEAKER_01: And I think that for a lot of people,
47:27
SPEAKER_01: this is a question we used to talk about like three years ago.
47:30
SPEAKER_01: And I think with time,
47:31
SPEAKER_01: with advantage of time,
47:32
SPEAKER_01: we've had a lot more clarity of like,
47:35
SPEAKER_01: the purpose of code review isn't just the mechanical.
47:37
SPEAKER_01: It's not just,
47:38
SPEAKER_01: hey,
47:38
can you proofread my essay for me because I want to make sure
47:40
there are no grammar
47:41
SPEAKER_01: mistakes in it.
47:42
SPEAKER_01: I think there's also a type of shared learning that happens of,
47:46
SPEAKER_01: hey,
47:47
I want other people to code review this because I want to
47:49
distribute my knowledge
47:50
SPEAKER_01: of how the system works.
47:51
SPEAKER_01: And I think that there is an alignment checking that happens of,
47:54
SPEAKER_01: hey,
47:55
SPEAKER_01: this is how we build software here.
47:57
And there's tribal knowledge that maybe more senior staff level
48:00
engineers at your
48:00
SPEAKER_01: company have that they disseminate through the process of code review.
48:04
And I think that as we take away these mechanical parts or
48:08
reduce their need,
48:10
SPEAKER_01: you're going to see those latter two become a lot more emphasized.
48:13
SPEAKER_00: I am a little worried,
48:14
SPEAKER_00: though,
48:14
SPEAKER_00: about there is a big...
48:16
SPEAKER_00: We do see that AI tools are very capable of generating code.
48:20
SPEAKER_00: And I think,
48:21
SPEAKER_00: you know,
48:21
like experienced developers who've been in the industry for 10
48:24
plus years and have seen,
48:26
SPEAKER_00: for example,
48:26
SPEAKER_00: they work with the teams of junior engineers and you see what happens.
48:30
SPEAKER_00: You know, it's a bit of a mess and you need to really clean it up.
48:32
SPEAKER_00: Like they...
48:34
SPEAKER_00: I think they see that this has limitations, right?
48:37
SPEAKER_00: It's a very powerful tool when you know what good output is,
48:39
SPEAKER_00: when you can check it,
48:40
SPEAKER_00: when you can step in and say,
48:41
SPEAKER_00: this is BS,
48:41
SPEAKER_00: stop.
48:42
SPEAKER_00: Or you just stop prompting after a while because it's not going anywhere.
48:45
SPEAKER_00: And you start to take over and you just write whatever you need to do.
48:48
SPEAKER_00: But I do feel that people overindex on how good it generates code.
48:53
SPEAKER_00: I have a feeling that a lot of companies,
48:56
SPEAKER_00: especially company leaders,
48:58
SPEAKER_00: even CTOs,
48:59
SPEAKER_00: they'll kind of ignore this part of learning because it's invisible.
49:03
SPEAKER_00: You don't really see the difference between two teams.
49:06
SPEAKER_00: One, there's no code review or just AI code review.
49:09
SPEAKER_00: Everyone's doing their own thing and they're both shipping for a while.
49:12
SPEAKER_00: And at some point, something breaks.
49:14
SPEAKER_00: And this team where people are actually doing code reviews,
49:16
SPEAKER_00: they're thoughtful,
49:17
SPEAKER_00: they understand the code.
49:18
they can start,
49:20
nothing slows down,
49:20
they build the new features,
49:22
they innovate.
49:23
SPEAKER_00: Whereas this one kind of gets stuck, there's just bugs, no one knows why.
49:26
SPEAKER_00: But because it's not visible,
49:29
SPEAKER_00: I have this feeling that there might be,
49:32
SPEAKER_00: I don't wanna be all negative about it,
49:35
SPEAKER_00: but
49:36
It's just so many steps removed from the work that I wonder if
49:40
few companies will
49:41
SPEAKER_00: appreciate it.
49:42
SPEAKER_00: A bit like how few companies used to appreciate developer tools.
49:45
SPEAKER_00: It's only fashionable these days, right?
49:47
SPEAKER_00: If you think about it 10 or 15 years ago,
49:49
SPEAKER_00: everyone was looking like,
49:51
SPEAKER_00: why is Google having a dedicated team to build internal dev tools?
49:56
SPEAKER_00: That's weird.
49:56
SPEAKER_00: Yeah.
49:58
SPEAKER_01: I think it became popular with... I'll give a lot of credit to Heroku.
50:02
SPEAKER_01: I think I remember just growing up in that era.
50:04
SPEAKER_01: There was...
50:06
there was a lot of love for like,
50:08
oh,
50:08
this is what good can look like,
50:10
you know?
50:10
And I think it really set a standard of like,
50:12
oh,
50:12
we don't have to accept worse.
50:13
We can actually,
50:14
we can have better tooling sort of like in the popular opinion.
50:17
SPEAKER_01: And then a lot of Heroku developers went on to go to companies,
50:20
SPEAKER_01: brought a lot of that energy and feeling with them.
50:22
SPEAKER_01: And then we started to see it there.
50:24
SPEAKER_01: But yeah,
50:25
SPEAKER_01: I remember when Google,
50:26
SPEAKER_01: there was a lot of sort of like question marks around like,
50:29
SPEAKER_01: what is Google doing?
50:30
SPEAKER_01: Like, why would you invest so much in this?
50:32
SPEAKER_01: And meta too.
50:33
SPEAKER_01: I think that,
50:35
SPEAKER_01: Where this will be different with AI is the fact that...
50:41
SPEAKER_01: At the end of the day, code has really profound business decisions.
50:46
If you decide that you want to give a refund or you want to
50:49
charge this or the user
50:50
SPEAKER_01: experience wants to look like this or this is the way the product works,
50:53
SPEAKER_01: that's at the end of the day a business decision.
50:55
And I think that what leaders will agree with is that they
50:57
don't want the machines
50:58
SPEAKER_01: making business decisions.
50:59
And so at that point,
51:00
at some level,
51:01
that means that a human needs to be reviewing it.
51:03
SPEAKER_01: And now the question is sort of like,
51:05
SPEAKER_01: what is that level and what is that granularity that we review on?
51:08
SPEAKER_01: But I think that we can...
51:11
SPEAKER_01: We can find a common understanding and like,
51:14
SPEAKER_01: okay,
51:15
SPEAKER_01: humans will need to at least check for intent the same way humans prompt.
51:18
SPEAKER_01: And that's not to say that prompting is bad or it doesn't work.
51:21
It's more that just,
51:22
I think fundamentally,
51:23
human natural language communication is flawed.
51:26
You and I have a conversation and you can understand something
51:28
different than what I mean.
51:29
And so at a company,
51:30
the way we get around this is we then check on the other side.
51:33
SPEAKER_01: We align at the start and we check at the end.
51:36
SPEAKER_01: And I don't think that's going to go away.
51:38
SPEAKER_00: Yeah, I like this thinking.
51:40
SPEAKER_00: And also, I wonder if we might have more companies discover.
51:44
SPEAKER_00: Again,
51:44
SPEAKER_00: I'm just a little bit inspired by Linux,
51:46
SPEAKER_00: as I recently talked with Greg from there,
51:49
SPEAKER_00: that the way Linux works,
51:51
SPEAKER_00: you know,
51:51
SPEAKER_00: when I talked about how the whole project works,
51:53
SPEAKER_00: I was really surprised in how much it's about people.
51:55
And the core concept there is when you take a patch,
51:58
they call a PR a patch there.
52:01
SPEAKER_00: you now own it and you are personally responsible for it.
52:05
SPEAKER_00: And this actually,
52:06
SPEAKER_00: you know,
52:06
SPEAKER_00: if something goes,
52:07
SPEAKER_00: and this is how there's a pyramid of like subsystem owners,
52:10
SPEAKER_00: system owners,
52:11
SPEAKER_00: and then in the end there's Linux Torval.
52:12
So every single release for Linux,
52:14
it goes through,
52:15
it's still to this day,
52:16
it goes through him.
52:17
SPEAKER_00: He owns it.
52:18
SPEAKER_00: And, you know, he's very happy to take that responsibility.
52:20
SPEAKER_00: But,
52:21
And Greg was telling me that because of this,
52:23
if you think about it,
52:24
it's very open.
52:25
SPEAKER_00: The whole world sees your changes.
52:27
SPEAKER_00: So because of this, people take pride in this.
52:31
SPEAKER_00: They will both review, but there's also trust levels, right?
52:33
SPEAKER_00: So if you're a system owner and there's a subsystem owner,
52:36
SPEAKER_00: you've worked now for 10 years,
52:38
SPEAKER_00: you know this person is solid.
52:39
SPEAKER_00: You're not going to spend all the energy.
52:42
SPEAKER_00: And I wonder if the correct response to AI is like, yeah, just go wild.
52:47
SPEAKER_00: Use these tools.
52:48
SPEAKER_00: But responsibility...
52:50
SPEAKER_00: Maybe it needs to go back to developers.
52:54
SPEAKER_00: So we'll see.
52:56
SPEAKER_01: I suspect that is what we'll see.
52:58
And as I said,
52:59
that's not to say that AI is not going to solve your problems.
53:01
I think AI will actually free the humans to focus more on those
53:05
bigger picture things.
53:06
SPEAKER_01: I think AI is going to be able to take away the like,
53:08
SPEAKER_01: oh,
53:09
SPEAKER_01: does this function do what it intends to do?
53:11
SPEAKER_01: Because it can understand that.
53:12
SPEAKER_01: It can tell us that.
53:13
SPEAKER_01: But I think it's going to be a tool in them saying,
53:16
SPEAKER_01: and so does this holistic change do what you want it to do?
53:19
SPEAKER_00: Speaking of change,
53:21
SPEAKER_00: one thing that I was really surprised that when I worked at Uber,
53:25
I talked with other companies and stacking didn't really take
53:26
off anywhere else
53:27
SPEAKER_00: because Fabricator was just not there.
53:30
Why do you think that didn't take off and how did you get the
53:32
idea to start
53:32
graphite around this idea of building this tooling to make it
53:38
accessible for a lot
53:38
SPEAKER_00: more people?
53:39
SPEAKER_01: Yeah, of course.
53:40
SPEAKER_01: Well,
53:40
so I'll start by saying I actually don't think that stacking
53:43
caught on inside Meta
53:44
SPEAKER_01: at the start.
53:46
SPEAKER_01: So originally when it was built, it was built into Mercurial.
53:50
SPEAKER_01: It was advertised at the company.
53:54
There were a few senior engineers who saw the vision,
53:57
felt blocked,
53:58
and picked it up.
53:59
And I think that,
54:01
not in an exaggerating way,
54:02
they saw themselves get 3x faster.
54:04
And what happened was those senior engineers started to become evangelists.
54:08
SPEAKER_01: I had one on my team who was like the stacking evangelist.
54:11
And they used to go throughout the organization and literally
54:13
give PowerPoint
54:14
SPEAKER_01: presentations of like,
54:15
SPEAKER_01: this is how stacking speeds you up.
54:17
SPEAKER_01: Like, you need to believe me.
54:19
SPEAKER_01: I've seen the other side.
54:20
And at the time,
54:21
I think you had a lot of developers who were very skeptical.
54:23
SPEAKER_01: Who were like, I've been doing Git for so long.
54:25
SPEAKER_01: Like, I don't need you to tell me how to change my workflow.
54:28
SPEAKER_00: Why would I do that?
54:29
SPEAKER_00: I hear all the signs from current people who hear about stacking.
54:34
And then what ended up happening was that as these evangelists
54:39
made inroads and
54:40
SPEAKER_01: started to convince people,
54:41
SPEAKER_01: like,
54:41
SPEAKER_01: no,
54:41
SPEAKER_01: no,
54:41
SPEAKER_01: no,
54:42
SPEAKER_01: it's actually speeding you up,
54:43
SPEAKER_01: people would adopt it,
54:44
SPEAKER_01: see the value,
54:45
SPEAKER_01: and become that evangelist themselves.
54:48
SPEAKER_01: We like to joke that once you stack, you don't go back.
54:51
um i i think why it works so well is it did two things really
54:56
it was like the
54:56
perfect uh convergence of two forces one is that it gave
55:00
individuals what they want
55:01
so i think no developer likes being blocked right if you ask a
55:03
developer like what
55:05
is the feeling towards code review um or towards having to wait
55:08
on code review oh
55:09
yeah it's almost always negative no one's like i love that
55:12
feeling negative
55:13
SPEAKER_01: People are like, it sucks.
55:15
SPEAKER_01: It's the worst part.
55:16
And I think that even if you have a more senior developer
55:19
that's been doing,
55:20
SPEAKER_01: that's had code review for 20 years,
55:22
SPEAKER_01: if you ask them like,
55:23
why do you enjoy coding on personal projects as opposed to
55:25
company projects?
55:27
SPEAKER_01: The overhead of collaboration becomes quite clear there of like,
55:32
SPEAKER_01: yeah,
55:33
SPEAKER_01: because when I work with others,
55:35
SPEAKER_01: things just slow down.
55:36
SPEAKER_01: We need to review things.
55:37
SPEAKER_01: We need to align on them.
55:38
SPEAKER_01: They get a lot slower.
55:39
SPEAKER_01: Yeah.
55:40
And I think what stacking did is it took away some of that overhead,
55:44
right?
55:44
Where rather than having to wait on other people to review your
55:47
code that you
55:47
SPEAKER_01: already thought was pretty good and you had already aligned on before,
55:50
you could just keep going and keep building without needing to
55:54
sort of wait for them.
55:55
SPEAKER_01: The second side is on, I guess it was great for three parties.
55:59
SPEAKER_01: The second side was the reviewer.
56:00
SPEAKER_01: The reviewer started to get smaller PRs.
56:02
SPEAKER_01: Yeah.
56:03
SPEAKER_01: And whether you know it or not,
56:05
SPEAKER_01: when I send you a 10-line PR,
56:07
SPEAKER_01: you are so much happier than if I send you a 2,000-line PR.
56:10
SPEAKER_00: Yeah,
56:10
SPEAKER_00: but so many companies eventually,
56:11
SPEAKER_00: when they write down code review guidelines,
56:13
SPEAKER_00: they're always going to have small peers.
56:15
SPEAKER_00: At Uber,
56:16
SPEAKER_00: at some point,
56:16
SPEAKER_00: we had recommendations within the team for 50 or 70 lines,
56:20
SPEAKER_00: whatever that is.
56:21
SPEAKER_00: But you realize,
56:22
SPEAKER_00: I mean,
56:23
SPEAKER_00: there's jokes going around,
56:24
and I think it was viral on social media of the 5,000-line
56:26
review looks good to me,
56:28
SPEAKER_00: and then the 10-line,
56:30
SPEAKER_00: which gets like 10 comments.
56:32
SPEAKER_01: Exactly right.
56:33
SPEAKER_01: And I think people, as we were saying before, people buy out of it.
56:35
And the third is I think it really gave the company something
56:38
they needed,
56:39
which is that they started to see their engineering
56:41
organization speed up.
56:42
SPEAKER_01: They started to see time to merge go down.
56:44
SPEAKER_01: That's usually a metric that people measure,
56:46
SPEAKER_01: and we can talk about engineering metrics separately.
56:49
And in the case that something went wrong,
56:51
it made it much easier to correct things,
56:52
right?
56:53
In the case that QA or even users found something that wasn't
56:57
working the way it
56:57
SPEAKER_01: should be,
56:58
SPEAKER_01: it was much easier to then go to history and be like,
57:01
SPEAKER_01: which one of these changes did it?
57:02
Then when the changes were like monolithic feature,
57:04
monolithic feature,
57:05
monolithic feature.
57:06
SPEAKER_00: Yeah.
57:06
Now,
57:07
you know,
57:07
you build this tooling and you are still a pretty small startup.
57:11
SPEAKER_00: How do you convince companies to,
57:13
SPEAKER_00: you know,
57:14
SPEAKER_00: like use a pretty small startup,
57:16
SPEAKER_00: pretty unknown methodology?
57:18
You know,
57:18
rumors are Meta uses it a lot because I assume it's not a
57:21
really easy sell,
57:22
is it?
57:24
SPEAKER_01: No, it's not an easy sell at all.
57:26
SPEAKER_01: Well, so back to the company's origins.
57:28
SPEAKER_01: So I don't know if you know this.
57:29
SPEAKER_01: We didn't actually start as a code review company.
57:32
SPEAKER_01: So we actually started by building tools for mobile releases.
57:35
SPEAKER_01: That's the background of the founders.
57:36
SPEAKER_01: That's the area we wanted to build in.
57:40
I was fortunate enough to have two of my teammates from Meta
57:43
join the team.
57:44
SPEAKER_01: And I think one of the things that the three of us felt was like,
57:47
SPEAKER_01: oh,
57:47
SPEAKER_01: like the tooling out here,
57:48
SPEAKER_01: I think they were both new grads when they joined Meta,
57:51
SPEAKER_01: had been there for a handful of years,
57:53
SPEAKER_01: call it like four.
57:54
SPEAKER_01: And then when they left,
57:55
SPEAKER_01: they had the same experience I did,
57:57
SPEAKER_01: which was like,
57:58
SPEAKER_01: oh,
57:58
SPEAKER_01: like the tooling out here is different.
58:00
SPEAKER_01: And at the start, we kind of were like, well...
58:03
SPEAKER_01: We must be the weird ones.
58:05
And after about a few months,
58:07
we were like,
58:08
no,
58:09
there's actually something missing here.
58:10
SPEAKER_01: And I think the context that I now have that I wish I had then was,
58:17
SPEAKER_01: one,
58:17
SPEAKER_01: most companies,
58:18
SPEAKER_01: if you have more than 50 people,
58:20
SPEAKER_01: you have at least one person who builds tooling on top of GitHub.
58:23
SPEAKER_01: And I think that if you've worked at one of these companies,
58:24
SPEAKER_01: you kind of just take that as a truth of like,
58:26
SPEAKER_01: yeah,
58:26
SPEAKER_01: I have the DevTools team,
58:27
SPEAKER_01: DevPlatform.
58:29
SPEAKER_00: But these days, Jenkins is GitHub Actions, so same difference.
58:33
SPEAKER_01: Exactly.
58:34
SPEAKER_01: You have someone building tooling there to make your developers faster.
58:37
SPEAKER_01: The second thing is that these companies,
58:38
SPEAKER_01: so like Meta and Google,
58:40
SPEAKER_01: so GitHub released their pull request page back in 2013.
58:44
SPEAKER_01: They haven't really made a ton of major edits to it since.
58:48
those companies had their changes then and have continued to
58:53
iterate on their
58:54
SPEAKER_01: tooling for like the past decade.
58:55
SPEAKER_01: And so when we say their tooling is like a decade more advanced,
58:58
SPEAKER_01: that's a quite literal statement,
59:00
SPEAKER_01: actually.
59:00
SPEAKER_01: That's not a figurative one.
59:02
SPEAKER_01: And so we realized that.
59:04
SPEAKER_01: We were like, hold on.
59:06
SPEAKER_01: Like, there's missing tooling out here.
59:08
SPEAKER_01: We tried a lot of the alternatives.
59:10
SPEAKER_01: As I said, stacking was sort of that first pain point.
59:12
And so we tried a lot of the sort of like open source CLI first alternatives.
59:16
SPEAKER_01: Realized it
59:18
SPEAKER_01: It was like an approximation of what we wanted, but it wasn't quite it.
59:23
SPEAKER_01: And then we built it.
59:24
SPEAKER_01: And then what happened actually was a lot of our old coworkers,
59:27
SPEAKER_01: many of whom had gone on to startups,
59:29
SPEAKER_01: ended up asking us like,
59:30
SPEAKER_01: hey,
59:30
SPEAKER_01: code review out here is different.
59:34
SPEAKER_01: How do you all deal with it?
59:35
SPEAKER_01: And we're like, we built that.
59:37
SPEAKER_01: Let us know if you ever want us to tell you how to build it.
59:38
SPEAKER_01: And they're like,
59:40
SPEAKER_01: I don't, but like, can I just use your thing?
59:43
SPEAKER_01: And after we heard that for a few months,
59:45
SPEAKER_01: we ended up pivoting back in,
59:47
SPEAKER_01: I guess this would have been November of 2021.
59:49
SPEAKER_01: Okay.
59:50
So quick pivot in the beginning of the,
59:53
just,
59:54
so basically,
59:54
did you just feel a pull?
59:55
SPEAKER_00: Like, okay, like this is a good idea.
59:57
SPEAKER_00: Seems like there's a man for it.
1:00:00
SPEAKER_01: I don't think it was... The other company idea was working quite well.
1:00:06
I think what we felt was we had all of our friends asking us
1:00:10
for this thing that we
1:00:11
SPEAKER_01: had built and it wasn't the thing that we were selling.
1:00:14
SPEAKER_01: And you can only listen to that for so long before you decide to say,
1:00:18
SPEAKER_01: maybe we got this one wrong.
1:00:20
SPEAKER_01: Maybe this is the thing.
1:00:22
SPEAKER_00: It's just fascinating.
1:00:23
SPEAKER_00: I guess it's hard to...
1:00:25
SPEAKER_00: Appreciate if you're not working at a startup,
1:00:27
SPEAKER_00: because,
1:00:28
SPEAKER_00: you know,
1:00:28
SPEAKER_00: my my imagination of joining a startup,
1:00:30
SPEAKER_00: I have friends and my brother was working at a startup,
1:00:32
but I didn't join a super early stage startup ever is you would
1:00:35
imagine you're joining.
1:00:36
SPEAKER_00: This is what we're doing.
1:00:37
SPEAKER_00: This is what we're going to do.
1:00:38
SPEAKER_00: But,
1:00:39
SPEAKER_00: you know,
1:00:39
sounds like you need to be flexible enough because you might do
1:00:42
something different
1:00:43
SPEAKER_00: that actually works and there's a bigger demand for it.
1:00:46
Yeah,
1:00:46
we were fortunate enough that at the time that we pivoted,
1:00:48
we were just six people.
1:00:49
SPEAKER_01: And so it was six people and three of us had felt the pain from Facebook.
1:00:54
SPEAKER_01: So it was us telling people like, no, no, no, we've heard it.
1:00:57
SPEAKER_01: Like our coworkers have heard it.
1:00:58
We'd gotten other people at the company to buy into this
1:01:00
workflow at that point.
1:01:02
And yeah,
1:01:03
it was certainly stressful,
1:01:05
but it was also a thing we just had enough conviction in.
1:01:09
SPEAKER_01: We were like, this is the way the world will build software.
1:01:11
SPEAKER_01: It's more a question of who's going to bring it into the world, not if.
1:01:14
SPEAKER_00: Yeah.
1:01:15
SPEAKER_00: You previously mentioned engineering metrics.
1:01:17
What are engineering metrics you see these fast-moving
1:01:20
companies measure and
1:01:21
SPEAKER_00: actually care about and want to improve?
1:01:25
So I'll start with the statement that all engineering metrics
1:01:28
are a proxy.
1:01:29
SPEAKER_01: So I think that measuring engineering velocity is an art, not a science.
1:01:34
And I think that the value of engineering metrics is to create
1:01:38
some holistic
1:01:39
SPEAKER_01: metrics so that you can see major outliers and be like,
1:01:42
SPEAKER_01: what's going on here?
1:01:43
SPEAKER_01: Yeah.
1:01:44
The kinds of metrics which I see most people measure are number
1:01:50
of pull requests as
1:01:51
SPEAKER_01: just a proxy for sort of like how's velocity doing,
1:01:54
SPEAKER_01: time to merge of those pull requests.
1:01:57
SPEAKER_01: And then my favorite metric I've actually ever seen came out of Uber.
1:02:00
SPEAKER_01: And so maybe you might have more context on this than I,
1:02:03
but it was basically time that a pull request is just spent
1:02:06
waiting in review with
1:02:08
SPEAKER_01: no clear action.
1:02:09
And so the idea was if an author puts up a pull request and is
1:02:12
just waiting on review,
1:02:13
SPEAKER_01: you take that,
1:02:14
SPEAKER_01: then the reviewer goes ahead,
1:02:16
SPEAKER_01: requests changes,
1:02:17
SPEAKER_01: and you then ignore that time.
1:02:23
SPEAKER_01: You then put it back up for review,
1:02:25
SPEAKER_01: you start summing that time again,
1:02:26
and you end up with this composite of this is the amount of
1:02:29
time this pull request
1:02:30
SPEAKER_01: just sat waiting on review without clear next steps.
1:02:33
SPEAKER_00: So this metric was introduced after I left Uber.
1:02:36
I think I might have covered it in one of the deep dives on how
1:02:39
Uber did it.
1:02:40
But this was a really big problem at Uber because we were
1:02:43
distributed sites and my
1:02:44
team specifically had this problem where we would have to wait
1:02:48
almost 24 hours for
1:02:50
SPEAKER_00: something to happen in the pull request because we had the...
1:02:52
We had strong code ownership,
1:02:54
and I think Uber still might have it to this day.
1:02:57
SPEAKER_00: When I was there, we still had it.
1:02:59
So the mobile platform team would own the platform code in San
1:03:03
Francisco and Amsterdam.
1:03:05
Last thing at night,
1:03:07
you would make a change,
1:03:08
and you would have to head off.
1:03:10
SPEAKER_00: And it would go to them,
1:03:11
they would reject it because of whatever reason you come back
1:03:13
in the morning,
1:03:14
SPEAKER_00: you know,
1:03:14
SPEAKER_00: it's now nighttime in San Francisco,
1:03:18
SPEAKER_00: you fix it quickly.
1:03:19
SPEAKER_00: And then you have to wait another like eight hours and then stay late.
1:03:23
SPEAKER_00: So like,
1:03:24
SPEAKER_00: I'm glad that they started to measure it because,
1:03:26
and this is one of the things where it's not really a big deal
1:03:29
if you're a startup,
1:03:30
SPEAKER_00: if you're in the same time zone,
1:03:31
SPEAKER_00: if you're in the same location,
1:03:33
SPEAKER_00: but
1:03:33
SPEAKER_00: And then, you know, we came up with creative workarounds.
1:03:35
SPEAKER_00: Like we started to get like,
1:03:37
SPEAKER_00: we onboarded someone to be an owner of,
1:03:39
SPEAKER_00: essentially,
1:03:39
SPEAKER_00: you can do a lot of things,
1:03:40
SPEAKER_00: but it was really killing us.
1:03:42
Like,
1:03:42
and I knew it because I was the manager of the team and this
1:03:44
was the number one complaint.
1:03:45
SPEAKER_00: So we actually, you know, we took it into our hands and we solved it.
1:03:48
SPEAKER_00: But so many teams were doing this independently.
1:03:51
SPEAKER_00: And then it kind of goes back to like,
1:03:53
You know,
1:03:53
like as a dev,
1:03:54
I didn't really like to hear too much about strategy and whatnot.
1:03:56
But there is a thing of like,
1:03:57
you know,
1:03:57
who owns what,
1:03:59
how do you place your teams?
1:04:00
It's kind of easier when you have teams working on similar
1:04:02
stuff close to each
1:04:03
SPEAKER_00: other in terms of time zone.
1:04:05
SPEAKER_00: So, yeah.
1:04:06
SPEAKER_00: And,
1:04:06
SPEAKER_00: you know,
1:04:07
SPEAKER_00: one other thing that Uber measured,
1:04:08
SPEAKER_00: which I'm not sure how successful it was,
1:04:10
SPEAKER_00: but it was an interesting thing,
1:04:11
is they tried to measure focus time on how much uninterrupted
1:04:15
focus blocks devs had.
1:04:16
SPEAKER_00: Yeah.
1:04:18
SPEAKER_00: I think it's really noble.
1:04:19
SPEAKER_00: It's just really hard to measure.
1:04:20
But it's true that when I was always happier developing when I
1:04:24
actually had time,
1:04:25
SPEAKER_00: it wasn't fragmented with the random meetings here or there,
1:04:29
SPEAKER_00: etc.
1:04:31
SPEAKER_01: Yeah, totally.
1:04:31
SPEAKER_01: I mean, maker's schedule, manager's schedule is super real.
1:04:34
SPEAKER_01: I think that as we were going back to before,
1:04:37
SPEAKER_01: though,
1:04:37
SPEAKER_01: these problems,
1:04:38
SPEAKER_01: which used to be more isolated to big companies,
1:04:40
I think as we start to see a higher volume of pull requests and
1:04:43
more of the time is
1:04:43
SPEAKER_01: spent in code review,
1:04:45
SPEAKER_01: not authoring pull requests,
1:04:48
SPEAKER_01: they're going to become everyone's problems.
1:04:50
SPEAKER_01: And I think the question is how ready are companies for that change?
1:04:54
SPEAKER_00: Yeah.
1:04:56
SPEAKER_00: I like this thinking because this will happen.
1:05:00
So it's interesting how small companies might need to take some
1:05:04
of the lessons from
1:05:05
SPEAKER_00: the larger companies on how they solve it and prepare for it.
1:05:08
SPEAKER_00: Because it's going to come.
1:05:09
SPEAKER_00: I'm kind of tempted to agree with you on this one.
1:05:12
SPEAKER_00: It'll be fascinating to see.
1:05:13
It'll be also fascinating to see if small companies speed up a
1:05:16
lot more than,
1:05:16
SPEAKER_00: let's say,
1:05:17
big companies where there's larger teams and there's now more dependencies.
1:05:21
SPEAKER_00: It'll be an interesting one for us to...
1:05:23
I think small companies are certainly better positioned to
1:05:25
embrace the change.
1:05:26
SPEAKER_01: I think big companies will be slow.
1:05:28
I think small companies,
1:05:29
we've already seen them adopt a lot of AI tools faster.
1:05:32
I think some of the most advanced big companies are just trying
1:05:36
to adopt them at
1:05:37
SPEAKER_01: the same pace as these small companies,
1:05:39
SPEAKER_01: but they're definitely not trying to outperform them.
1:05:41
SPEAKER_01: And I think we've seen this actually in quite a few areas.
1:05:45
SPEAKER_01: Every time that you have a new technology that makes something...
1:05:49
SPEAKER_01: more accessible easier.
1:05:50
SPEAKER_01: The same way I think AI is making coding more accessible easier.
1:05:53
I think what ends up happening is that practices that were
1:05:56
previously only meant
1:05:57
for the tippity-top of the market start to distill their way
1:06:01
down to everyone.
1:06:03
SPEAKER_00: So you're a co-founder of a DevTools company,
1:06:07
SPEAKER_00: and I think this is a little bit of a dream of a lot of engineers.
1:06:11
I actually see this pattern pretty much as you join a company,
1:06:14
you work on a product.
1:06:15
SPEAKER_00: If it's a big enough company, there's a platform team.
1:06:17
A lot of engineers gravitate to there because you can solve
1:06:19
pretty cool problems.
1:06:21
SPEAKER_00: In your case, you were on a DevTools team.
1:06:24
SPEAKER_00: And then you...
1:06:26
you find the courage or find the right people to actually start
1:06:29
a startup.
1:06:29
SPEAKER_00: And what better thing to do as a dev and what,
1:06:33
SPEAKER_00: you know,
1:06:33
SPEAKER_00: you already know how to do this is built for other developers.
1:06:37
What is something that you've kind of learned the hard way of
1:06:41
starting a dev tools company?
1:06:42
SPEAKER_00: Because I assume it's a bit trickier
1:06:46
SPEAKER_00: than what you imagined?
1:06:47
SPEAKER_00: What was different than when you started out?
1:06:48
SPEAKER_00: Because it sounds like the dream, right?
1:06:50
SPEAKER_00: You kind of went out,
1:06:51
SPEAKER_00: you actually managed to raise some funding,
1:06:52
SPEAKER_00: and now you're going to build these cool tools for developers.
1:06:54
SPEAKER_00: And you're a developer.
1:06:55
SPEAKER_00: It's great.
1:06:55
SPEAKER_00: You know, your customer is wonderful.
1:07:00
SPEAKER_01: I mean, a ton of things.
1:07:02
SPEAKER_01: I think that building a company can be very rewarding in a lot of ways.
1:07:05
SPEAKER_01: I think there are a lot of challenges.
1:07:06
SPEAKER_01: I think some of the things that you learn to deal with,
1:07:08
some of the things I think we've learned to deal with here are
1:07:11
sort of the
1:07:13
SPEAKER_01: diversity of engineering workflows across,
1:07:15
SPEAKER_01: even across Silicon Valley and non-Silicon Valley style companies,
1:07:19
SPEAKER_01: right?
1:07:19
SPEAKER_01: Of like, what does code review mean to them?
1:07:21
SPEAKER_01: What are they trying to get out of it?
1:07:22
SPEAKER_01: What's the purpose of it?
1:07:23
SPEAKER_01: And finding the middle road to blend all of those things
1:07:30
SPEAKER_01: to blend all of those,
1:07:32
SPEAKER_01: I don't know,
1:07:33
SPEAKER_01: needs,
1:07:33
SPEAKER_01: desires,
1:07:34
SPEAKER_01: et cetera,
1:07:35
SPEAKER_01: is difficult,
1:07:35
SPEAKER_01: is challenging,
1:07:36
SPEAKER_01: is something we build.
1:07:37
I think one of the things that has most surprised me coming out
1:07:40
of Facebook is what
1:07:42
SPEAKER_01: we talked around earlier,
1:07:43
SPEAKER_01: is the tooling.
1:07:44
SPEAKER_01: I think working within a company,
1:07:47
SPEAKER_01: you probably have that team that is building really awesome tooling,
1:07:50
SPEAKER_01: and they really benefit from that.
1:07:52
SPEAKER_01: When you come out here into the wild,
1:07:54
SPEAKER_01: as I call it,
1:07:55
SPEAKER_01: a lot less of that exists,
1:07:56
SPEAKER_01: and you find yourself being like,
1:07:58
SPEAKER_01: okay,
1:07:58
SPEAKER_01: how can I set that up?
1:08:00
SPEAKER_01: How can I get back to that?
1:08:01
SPEAKER_01: A company which I'll give a shout out to is Statsig.
1:08:07
So Facebook had really wonderful experimentation tooling,
1:08:13
feature flagging tooling.
1:08:14
And when I left immediately,
1:08:15
not immediately,
1:08:16
but six months in,
1:08:17
I was like,
1:08:17
okay,
1:08:17
cool.
1:08:18
SPEAKER_01: We probably have our first feature flag.
1:08:20
SPEAKER_01: How do we go ahead and turn this on?
1:08:21
SPEAKER_01: And I think you just mentioned you're at Uber.
1:08:23
SPEAKER_01: You had wonderful tooling there too.
1:08:26
There's all of this stuff that's set up for you that you just
1:08:28
take for granted.
1:08:29
SPEAKER_01: And once you come out and have to build it,
1:08:31
SPEAKER_01: you have to start to be like,
1:08:33
SPEAKER_01: OK,
1:08:34
SPEAKER_01: how do we do this from scratch again?
1:08:36
SPEAKER_00: You know, or who can do this for me?
1:08:39
SPEAKER_00: Awesome.
1:08:39
SPEAKER_00: So with that, let's wrap up with some rapid questions.
1:08:41
SPEAKER_00: I'll ask a question and you just shoot whatever comes to your mind.
1:08:44
SPEAKER_00: Cool.
1:08:44
SPEAKER_00: So first,
1:08:45
what is a tool that you saw inside Facebook that is now outside
1:08:50
Facebook as well
1:08:50
SPEAKER_00: and that can be used and you think is pretty cool?
1:08:53
SPEAKER_01: I think Statsig is still the answer I'm going to give here.
1:08:56
SPEAKER_01: It's very cool having the feature flag system that it's used to.
1:08:59
And it's actually an amalgamation of a handful of different
1:09:01
Facebook tools.
1:09:02
Our feature flagging system,
1:09:03
Gatekeeper,
1:09:03
and then our experimentation feature,
1:09:05
Deltoid.
1:09:06
SPEAKER_01: And so it's cool to see both of those.
1:09:07
SPEAKER_01: And QE2, I guess.
1:09:09
SPEAKER_01: It's kind of cool to see that reborn outside of Facebook.
1:09:14
SPEAKER_00: That's awesome.
1:09:14
SPEAKER_00: And this is the beauty of like,
1:09:16
SPEAKER_00: I think,
1:09:16
SPEAKER_00: you know,
1:09:17
there was this like several years where DevTools got funded
1:09:21
because I feel a lot of
1:09:22
the things that were hidden inside these companies are now
1:09:24
outside as well.
1:09:26
SPEAKER_01: Totally.
1:09:28
SPEAKER_01: I think also one of the things that's,
1:09:29
SPEAKER_01: having worked at DevTools within Facebook and out of Facebook,
1:09:31
SPEAKER_01: I think one of the things that's so funny to me is that in Facebook,
1:09:34
SPEAKER_01: building DevTools is a lot easier.
1:09:35
SPEAKER_01: You don't need to worry about things like marketing.
1:09:37
SPEAKER_01: You don't need to worry about things like hiring.
1:09:39
And I think it takes a lot of courage and it's a very difficult
1:09:42
path to build a
1:09:43
SPEAKER_01: tool outside of one of these large companies and say,
1:09:46
SPEAKER_01: no,
1:09:46
SPEAKER_01: I'm going to figure out how to make this work for the market.
1:09:48
SPEAKER_01: Because again, the engineering cultures are different.
1:09:51
SPEAKER_01: The constraints are different.
1:09:52
SPEAKER_01: The
1:09:53
SPEAKER_01: The other tools you're integrating with are different.
1:09:55
SPEAKER_01: And so you have to start to rebuild all that.
1:09:58
SPEAKER_00: And so what is your favorite programming language?
1:10:01
SPEAKER_00: Always an interesting question.
1:10:03
SPEAKER_01: TypeScript, by far.
1:10:04
SPEAKER_01: I think TypeScript is one of the most beautiful languages.
1:10:07
And you're talking to a person that learned JavaScript when he
1:10:10
was like 10.
1:10:11
SPEAKER_01: And then the type system on top of TypeScript,
1:10:14
SPEAKER_01: I think,
1:10:14
SPEAKER_01: is really clean and does a great job with the constraints they have.
1:10:19
I remember when TypeScript was created and I remember there was
1:10:22
a bit of skepticism
1:10:23
SPEAKER_00: around A,
1:10:24
SPEAKER_00: it was Microsoft.
1:10:25
SPEAKER_00: B, do we even need this?
1:10:26
SPEAKER_00: Because,
1:10:27
SPEAKER_00: you know,
1:10:27
the dynamic typing system is kind of like this wonderful thing
1:10:30
that allows you to
1:10:31
SPEAKER_00: do so many crazy things.
1:10:33
SPEAKER_00: And yeah,
1:10:34
SPEAKER_00: like I think everyone has turned,
1:10:35
SPEAKER_00: well,
1:10:35
SPEAKER_00: not everyone,
1:10:36
but like most devs have slowly or surely seen the benefits of
1:10:41
what we get with
1:10:43
SPEAKER_00: typing and with a nice language.
1:10:45
My controversial opinion is I'm glad Microsoft doesn't want to
1:10:48
do it because I
1:10:49
think that Microsoft has some really wonderful compiler
1:10:51
knowledge locked inside of it.
1:10:54
SPEAKER_01: And I similarly had that skepticism.
1:10:55
I remember at the time thinking,
1:10:56
we have flow,
1:10:58
what's the purpose of this?
1:11:00
And I think that the formal grammars and proofs that they've
1:11:04
done around it are really,
1:11:06
SPEAKER_01: really excellent.
1:11:07
You can go really deep on group theory or category theory just
1:11:11
by looking at TypeScript.
1:11:12
SPEAKER_01: But
1:11:13
SPEAKER_01: Yeah, I frequently look at other untyped languages.
1:11:17
SPEAKER_01: I'm looking at Ruby and I'm looking at Python,
1:11:19
and I don't think they've gotten close to what TypeScript has
1:11:21
done to JavaScript yet.
1:11:23
SPEAKER_00: And then finally,
1:11:24
SPEAKER_00: what's a book that you read and would recommend,
1:11:27
SPEAKER_00: fiction or nonfiction,
1:11:29
SPEAKER_00: one or two books?
1:11:30
SPEAKER_01: So I am an avid mystery reader.
1:11:32
SPEAKER_01: So the mystery book I always recommend people is The Last Days of Night.
1:11:35
It's a book around a Tesla,
1:11:38
Edison and Westinghouse and the race to invent the light bulb.
1:11:41
SPEAKER_01: It's written as historical fiction.
1:11:42
SPEAKER_01: I think it's phenomenal.
1:11:43
SPEAKER_01: As one who currently works in technology in New York,
1:11:46
SPEAKER_01: there isn't a better book I can recommend to anyone in that position.
1:11:50
SPEAKER_01: Nonfiction, I read a lot of books these days.
1:11:53
SPEAKER_01: I've been actually reading a lot more product books.
1:11:56
The Timeless Way of Building is a really interesting book about architecture.
1:12:00
SPEAKER_01: The beginning is a little abstract to get through,
1:12:04
SPEAKER_01: but once you get to the sort of latter half,
1:12:07
it talks around the importance of having the people who
1:12:10
ultimately use tools or
1:12:12
SPEAKER_01: buildings be the ones who build those tools or buildings.
1:12:16
SPEAKER_01: Because that's the only way that you create a self-sustaining cycle.
1:12:20
SPEAKER_00: I guess it kind of resonates with developers, developer tools.
1:12:24
SPEAKER_00: Yeah.
1:12:24
Which seems kind of obvious because who else would program the
1:12:27
tools that we use
1:12:28
SPEAKER_00: than developers?
1:12:29
SPEAKER_00: It's one of these weird things, but yeah.
1:12:32
SPEAKER_01: There's an MC Escher painting to be made about that.
1:12:34
SPEAKER_00: Well, this is really interesting.
1:12:36
I'm glad we were able to go a bit deeper into stacking than we
1:12:40
previously touched
1:12:41
SPEAKER_00: on that article as well.
1:12:43
SPEAKER_00: And thank you for your time.
1:12:44
SPEAKER_00: Cool.
1:12:45
SPEAKER_00: Of course.
1:12:46
SPEAKER_00: Thank you.
1:12:46
SPEAKER_00: This was always very fun.
1:12:47
Thanks a lot to Thomas for all the details on Meta's tooling,
1:12:50
stag diffs,
1:12:50
and on tooling trends.
1:12:52
SPEAKER_00: You can find Thomas on social media as listed in the show notes below.
1:12:56
SPEAKER_00: For more deep dives on tooling at Meta and more details on stag diffs,
1:12:59
check out the pragmatic engineer deep dives linked in the show
1:13:02
notes below.
1:13:04
If you enjoy the podcast,
1:13:05
please do subscribe on your favorite podcast platform and on YouTube.
1:13:08
SPEAKER_00: This helps more people discover the podcast.
1:13:10
SPEAKER_00: A special thank you if you leave a rating.
1:13:12
SPEAKER_00: Thanks and see you in the next one.