Latest / The Pragmatic Engineer / How Linux is built with Greg Kroah-Hartman

Transcript

How Linux is built with Greg Kroah-Hartman · The Pragmatic Engineer

0:00
There's a nine-week release,
0:01
because every nine weeks there's a new release going out,
0:04
right?
0:04
SPEAKER_00: So there's, Linus does a release at this point in time.
0:07
SPEAKER_00: And then the merge window is considered open.
0:09
SPEAKER_00: And then for two weeks,
0:10
all the maintainers send Linus all the stuff they've had
0:13
pending from the last
0:14
SPEAKER_00: release.
0:15
SPEAKER_00: We have two weeks to add all new features.
0:17
SPEAKER_00: And then he does release candidate one.
0:18
SPEAKER_00: From there on, it's bug fixes only for the next seven weeks.
0:22
SPEAKER_00: So it's bug fixes only, bug fixes only, bug fixes.
0:24
SPEAKER_00: It's regression fixes, we'll revert things.
0:26
SPEAKER_00: No new features.
0:27
SPEAKER_02: Do I understand correctly that in the case of Linux,
0:29
SPEAKER_02: is this a thing where every nine weeks there will be a release?
0:32
SPEAKER_00: It's time-based.
0:33
So we have that two-week window of merging all the new features
0:35
to Linux that have
0:36
SPEAKER_00: been in our tree and accepted already and proven to work.
0:39
SPEAKER_00: And the window is short, nine weeks.
0:40
SPEAKER_00: We used to have three-year-long development cycles.
0:42
SPEAKER_00: And the problem there is even if you have six-month development cycles,
0:45
SPEAKER_00: there's that fear of you have a feature.
0:47
SPEAKER_00: I want to take your feature, but it's not quite ready.
0:49
SPEAKER_00: Do I want to wait?
0:50
SPEAKER_00: And things like that.
0:51
SPEAKER_00: But if you know that you can get your feature in nine weeks from now,
0:54
SPEAKER_00: and it's just not ready,
0:55
SPEAKER_00: it's not ready.
0:55
The pressure's off me as a maintainer to take your new feature
0:58
until it's ready.
1:00
Linux is the world's most widely used operating system thanks
1:02
to powering most
1:03
SPEAKER_02: Android devices,
1:04
SPEAKER_02: servers,
1:05
SPEAKER_02: smart TVs,
1:06
SPEAKER_02: and embedded systems.
1:07
SPEAKER_02: But how is it actually built?
1:09
SPEAKER_02: Today,
1:10
SPEAKER_02: we sat down with Greg Crow Hartman,
1:12
SPEAKER_02: a Linux kernel maintainer for 13 years,
1:14
SPEAKER_02: who is one of the three Linux Foundation Fellows.
1:17
SPEAKER_02: In today's conversation,
1:18
we cover details on how widespread Linux is and why mobile
1:22
versions of Linux have
1:23
SPEAKER_02: three times the lines of code as the server versions.
1:26
What exactly it takes to get a change accepted to the Linux
1:28
kernel and merged by
1:30
SPEAKER_02: Linux Torwards himself.
1:31
SPEAKER_02: How Linux manages to have 4,000 contributors per year,
1:34
SPEAKER_02: yet have no product managers or project managers.
1:37
SPEAKER_02: and many more details.
1:39
SPEAKER_02: If you're a software engineer,
1:40
SPEAKER_02: you will use Linux directly or indirectly,
1:42
and this episode will help you understand why it's so
1:44
widespread and how it's a lot
1:46
SPEAKER_02: easier to contribute than most people would assume.
1:49
If you enjoy the show,
1:50
please subscribe to the podcast on any podcast platform and on
1:52
YouTube.
1:53
SPEAKER_02: Thank you.
1:55
SPEAKER_02: So Greg,
1:56
it's really just nice to have you here because you're one of
1:59
the most well-known
2:01
SPEAKER_02: Linux contributors,
2:02
SPEAKER_02: one of the longest standing ones as well.
2:05
SPEAKER_02: So just welcome to the podcast.
2:07
SPEAKER_02: Thanks for having me.
2:07
SPEAKER_02: I think as software engineers,
2:09
we know Linux is important in the sense of it's running on most
2:12
web servers that we
2:14
SPEAKER_02: use and run.
2:17
It's a desktop OS that some people use,
2:19
and it's,
2:20
of course,
2:20
a fork of it is powering Android.
2:23
SPEAKER_02: But what is there to know about Linux?
2:26
SPEAKER_02: How big is this thing?
2:27
SPEAKER_02: How complex is this thing?
2:28
SPEAKER_00: Well, it's an operating system, so it's a kernel.
2:31
SPEAKER_00: We took over the world without anybody noticing.
2:34
I joke,
2:35
Android devices are 4 billion Android Linux users out there,
2:39
and they don't realize it.
2:41
Everything else is a rounding error,
2:43
which doesn't make the server people happy with me.
2:44
SPEAKER_00: But it's true.
2:45
SPEAKER_00: It's in everything.
2:47
SPEAKER_00: It's in all the embedded devices.
2:48
SPEAKER_00: It's in the air conditioning units, electric charging ports, satellites.
2:53
SPEAKER_00: Runs the International Space Station.
2:55
SPEAKER_00: Really?
2:56
SPEAKER_00: Yeah.
2:57
SPEAKER_00: Air traffic control for Europe and probably the US.
2:59
SPEAKER_00: All the financial markets.
3:02
SPEAKER_00: I don't think... It's in the cameras that we're using.
3:06
SPEAKER_00: So it's, yeah, I don't know of any place that hasn't taken over.
3:12
SPEAKER_00: The top five selling laptops for the past 15, 10, 15 years, Chromebooks.
3:17
SPEAKER_00: Those are all Linux-based.
3:18
SPEAKER_00: Not Apple, but Chromebooks are.
3:21
SPEAKER_00: Oh, iPhones.
3:22
SPEAKER_00: So every 5G modem out there is running a copy of Linux.
3:25
SPEAKER_00: Really?
3:26
SPEAKER_00: Yeah.
3:26
SPEAKER_00: Wow.
3:27
SPEAKER_00: So now with Apple doing their new chip,
3:28
SPEAKER_00: I don't know if it's a new one,
3:29
SPEAKER_00: but Qualcomm,
3:31
SPEAKER_00: all the 5G modems,
3:32
SPEAKER_00: probably the 4G,
3:33
SPEAKER_00: I'm not sure,
3:33
SPEAKER_00: but I know all the 5G modems have Linux inside it.
3:36
SPEAKER_02: This episode is brought to you by WorkOS.
3:38
SPEAKER_02: If you're building a SaaS app,
3:40
at some point your customers will start asking for enterprise
3:42
features like SAML
3:43
SPEAKER_02: authentication,
3:44
SPEAKER_02: SKIM provisioning,
3:45
SPEAKER_02: and fine-grained authorization.
3:47
SPEAKER_02: That's where WorkOS comes in,
3:48
SPEAKER_02: making it fast and painless to add enterprise features to your app.
3:52
SPEAKER_02: Their APIs are easy to understand,
3:53
SPEAKER_02: and you can ship quickly and get back to building other features.
3:57
WorkOS also provides a free user management solution called
4:00
AuthKit for up to 1
4:01
SPEAKER_02: million monthly active users.
4:03
It's a drop-in replacement for Auth0 and comes standard with
4:05
useful features like
4:06
SPEAKER_02: domain verification,
4:07
SPEAKER_02: role-based access control,
4:09
SPEAKER_02: bot protection,
4:10
SPEAKER_02: and MFA.
4:11
SPEAKER_02: It's powered by Radix components, which means zero compromises in design.
4:14
You get limited customizations as well as modular templates
4:17
designed for quick integrations.
4:19
SPEAKER_02: Today,
4:20
SPEAKER_02: hundreds of fast-growing startups are powered by WorkOS,
4:22
SPEAKER_02: including ones you probably know,
4:24
SPEAKER_02: like Cursor,
4:24
SPEAKER_02: Vercel,
4:25
SPEAKER_02: and Perplexity.
4:26
SPEAKER_02: Check it out at workos.com to learn more.
4:29
SPEAKER_02: That is workos.com.
4:32
SPEAKER_02: Well, first of all, I'm...
4:34
I'm just reflecting on why I never kind of,
4:38
you know,
4:38
like thought about it like this.
4:40
Because in my mind,
4:41
it was always like,
4:42
you know,
4:43
Debian,
4:43
Red Hat,
4:44
it's on a server side.
4:45
SPEAKER_02: And maybe that's because that's where I actually see where it is.
4:49
SPEAKER_02: Of course,
4:49
SPEAKER_02: you know,
4:50
SPEAKER_02: there's the...
4:52
SPEAKER_02: Right now,
4:52
SPEAKER_02: I'm a Mac user and there's the Unix influence,
4:55
SPEAKER_02: which is an influence.
4:55
SPEAKER_02: You know, it gets pretty close.
4:57
SPEAKER_02: I think it's a good time to reflect on how many things it actually runs.
5:02
SPEAKER_02: In terms of the...
5:04
SPEAKER_02: The kernel itself, how large is it?
5:07
SPEAKER_02: I know for different devices,
5:09
SPEAKER_02: it'll be spread differently for server-side Linux,
5:12
SPEAKER_02: for an Android device.
5:15
SPEAKER_02: It'll use different parts of the kernel.
5:16
SPEAKER_02: How big is this in terms of contributors, lines of code?
5:20
SPEAKER_02: I know lines of code is not a great measure, but it is a measure.
5:22
SPEAKER_00: Lines of code is interesting.
5:23
SPEAKER_00: So we have just under 40 million lines of code right now.
5:26
SPEAKER_00: That's a lot.
5:27
SPEAKER_00: That's all the kernel.
5:28
SPEAKER_00: That's the kernel.
5:29
SPEAKER_00: The core part is like 5% of that that everybody runs.
5:32
SPEAKER_00: And then everybody, the rest of it is hardware support.
5:35
Different drivers,
5:35
different devices,
5:36
different architectures,
5:37
different chips.
5:39
SPEAKER_00: So your laptop runs about two, two and a half million lines of code.
5:42
SPEAKER_00: Your server runs about one and a half million.
5:45
SPEAKER_00: Servers are really easy.
5:46
SPEAKER_00: Those are very simple.
5:47
SPEAKER_00: Wow.
5:48
SPEAKER_00: Your phone runs about four million.
5:50
Your phone SOCs are the most complex pieces of CPU and
5:54
interaction out there.
5:55
SPEAKER_00: They're just crazy complex chips.
5:59
SPEAKER_02: Can we just pause for a second?
6:00
SPEAKER_02: So again, lines of code, we know, is not a perfect measure of complexity.
6:04
But in this sense,
6:05
comparing it between the two of them with the same code base is
6:08
somewhat so.
6:08
SPEAKER_02: You said roughly, give or take, a server is 1 and 1 half million.
6:12
SPEAKER_02: A phone is 4 million.
6:13
SPEAKER_02: Three times the lines of code for a phone.
6:15
SPEAKER_02: Why the difference?
6:17
Even though I would think that the server,
6:19
you know,
6:19
does all this mission critical stuff.
6:21
SPEAKER_00: A server is really simple.
6:22
SPEAKER_00: CPU and a network card and storage and storage.
6:26
SPEAKER_00: That's it.
6:27
SPEAKER_00: So,
6:28
SPEAKER_00: SOC on a phone has,
6:30
SPEAKER_00: you have power control,
6:31
SPEAKER_00: you have clocks,
6:32
you have five different buses on there talking to different
6:35
types of devices.
6:35
SPEAKER_00: You have battery control, you have
6:37
SPEAKER_00: Talking to your modem, you have another version of Linux in the modem.
6:41
SPEAKER_00: You got USB out the back.
6:42
SPEAKER_00: You got USB bypass to talk to the audio side.
6:45
SPEAKER_00: You have audio drivers.
6:47
You have a zillion different clocks and files and all sorts of
6:50
stuff in there.
6:51
SPEAKER_00: And it's an eight core machine.
6:53
SPEAKER_00: There's eight processors in that thing.
6:54
SPEAKER_00: Those are not trivial things.
6:56
SPEAKER_00: And sometimes those processors are different sizes.
6:58
SPEAKER_00: So you have big and little sizes,
6:59
SPEAKER_00: which add the complexity just for some control,
7:03
SPEAKER_00: for some power management.
7:04
SPEAKER_00: But they all run the same core of the Linux.
7:06
SPEAKER_00: But it's the drivers and the devices and things like that.
7:09
SPEAKER_00: So your Pixel phone.
7:11
SPEAKER_00: I look at Pixel phone.
7:12
SPEAKER_00: Google ships a core kernel that all Android devices pick.
7:16
SPEAKER_00: Not hardware specific, just says ARM64.
7:18
SPEAKER_00: Pixel has 300 other drivers they add to get the Pixel phone working.
7:23
SPEAKER_00: I mean, some of these are tiny.
7:24
SPEAKER_00: This is for this tiny chip, this is this tiny thing.
7:26
But your phone is really one of the most complex beasts out
7:29
there for software.
7:30
SPEAKER_02: Is it safe to say that,
7:32
SPEAKER_02: you know,
7:32
SPEAKER_02: the complexity and,
7:33
SPEAKER_02: you know,
7:34
SPEAKER_02: the lines of code will,
7:35
SPEAKER_02: to some extent,
7:36
SPEAKER_02: scale with that has to do with the hardware,
7:38
SPEAKER_02: the capabilities,
7:40
SPEAKER_02: and,
7:40
SPEAKER_02: you know,
7:41
SPEAKER_02: not about,
7:42
SPEAKER_02: you know,
7:42
SPEAKER_02: like how mission critical?
7:43
SPEAKER_02: Because,
7:43
SPEAKER_02: of course,
7:43
SPEAKER_02: the phone OS needs to be stable,
7:46
SPEAKER_02: the server needs to be stable,
7:47
SPEAKER_02: my TV needs to be stable.
7:48
SPEAKER_02: So, you know, that's just kind of a given, right?
7:49
SPEAKER_02: Yeah.
7:49
SPEAKER_00: Yeah, and all TVs for the past 15 years are all running Linux.
7:53
SPEAKER_02: Oh, so my Samsung TV is running Linux?
7:54
SPEAKER_00: Oh, yeah.
7:56
SPEAKER_00: My Samsung washer and dryer are running Linux.
7:59
SPEAKER_00: So your Samsung watch is running Linux.
8:01
SPEAKER_00: So Samsung has their own, yeah, distro.
8:04
SPEAKER_00: It all works really nice.
8:07
SPEAKER_00: It's all down to the complexity of the hardware.
8:09
SPEAKER_00: So the kernel controls the hardware.
8:11
SPEAKER_00: The job of Linux is to make...
8:14
SPEAKER_00: all the hardware look agnostic to programs.
8:17
So you can write the same user space program and run it on
8:21
different hardware,
8:21
SPEAKER_00: and it just works.
8:23
A kernel's job is to manage memory and devices in a common way
8:28
and provide that to user space.
8:30
SPEAKER_00: We joke from the kernel that user space is just a test load,
8:34
SPEAKER_00: but it's a tool there for you to actually solve your problems.
8:37
SPEAKER_00: So when you're running servers, you're wanting to
8:39
SPEAKER_00: put message through and network and storage and stuff.
8:42
SPEAKER_00: That's your load.
8:43
SPEAKER_00: And that's what they're there.
8:44
SPEAKER_00: For a phone, you want to control a display.
8:46
SPEAKER_00: You want to talk out the modem.
8:47
SPEAKER_00: You want to talk on the thing.
8:48
SPEAKER_00: You want to listen to audio.
8:50
SPEAKER_02: Lots of different things there.
8:51
And I just want to touch back on the kernel because I'm not a
8:57
Linux developer.
8:58
SPEAKER_02: I know I've heard of the kernel.
9:00
SPEAKER_02: In my assumption, it is the critical parts, as you said, the...
9:05
the thing that runs immediately,
9:07
and then the user space will run on top of it.
9:10
SPEAKER_02: But what is the differential?
9:13
SPEAKER_02: What makes a kernel?
9:14
SPEAKER_02: And you said it's about 5% of all of these things.
9:16
SPEAKER_02: How do you split this?
9:18
SPEAKER_02: Or is there a definition of, again, you're a kernel developer.
9:23
SPEAKER_02: I'm trying to get a sense,
9:25
SPEAKER_02: how can someone who,
9:27
SPEAKER_02: let's say,
9:28
SPEAKER_02: I'd like to contribute to Linux and understand how it is.
9:31
SPEAKER_02: Eventually, I'm going to figure out what this kernel is.
9:33
SPEAKER_02: But what is it?
9:34
SPEAKER_02: What makes kernel and non-kernel?
9:37
SPEAKER_00: So kernel versus user space.
9:38
SPEAKER_00: So there's an idea.
9:40
Chips have a protected mode and a not protected mode in a very
9:44
simplified way.
9:44
SPEAKER_00: There's different levels of protection.
9:46
SPEAKER_00: So the protected mode is where the operating system runs, the kernel.
9:50
SPEAKER_00: And that is where we share all the resources.
9:52
SPEAKER_00: It's one flat address space.
9:53
SPEAKER_00: Got it.
9:54
SPEAKER_00: And we are not isolating processes.
9:56
SPEAKER_00: Got it.
9:56
SPEAKER_00: So a user space process then runs on top of that, and we isolate them.
9:59
And they all individually think they have the whole machine,
10:02
but they don't.
10:02
SPEAKER_00: Yeah.
10:02
SPEAKER_00: So it's multitasking.
10:04
SPEAKER_00: You can run multiple programs at the same time.
10:06
SPEAKER_00: And the kernel is there to give you memory,
10:08
SPEAKER_00: to give access to storage,
10:10
SPEAKER_00: to give access in a common way,
10:11
SPEAKER_00: to give access to the network in a common way.
10:13
to give or provide the pipes to go around the network stack in
10:17
the user space.
10:18
SPEAKER_00: Some people don't like using Linux as network stack.
10:20
SPEAKER_00: They have their own.
10:21
To provide a way for all your different mice to show up to user
10:24
space in a common
10:25
SPEAKER_00: way where you know all the different mice,
10:27
SPEAKER_00: USB to storage devices,
10:28
SPEAKER_00: your graphics controller.
10:30
SPEAKER_00: We provide a way to make it so that user space can talk to
10:34
the kernel in an agnostic way and it'll their stuff will just
10:38
work because it all
10:39
the graphics work the same interface we talk to keyboards all
10:41
the same way things
10:42
like that so it's a commonality of providing a shim layer above
10:46
the hardware and
10:47
then for example drivers do they always live in the user space
10:51
no all the drivers
10:52
live in the kernel so the kernel and drivers are all linux is
10:55
not a micro kernel
10:57
architecture it's a monolithic so the code is all in the same
11:00
address space so a
11:01
SPEAKER_00: bug in any one of them has a chance to take any part of the kernel down
11:04
So Linux ships all the drivers for all the architecture in one
11:09
big tarball.
11:10
SPEAKER_00: That's 40 million lines of code.
11:12
SPEAKER_00: Other operating systems try and go out there and split things off.
11:15
So the core Windows is their kernel,
11:17
and then you can put drivers additional on the top.
11:21
SPEAKER_00: We tie everything together in one big giant blob.
11:25
SPEAKER_00: Theirs is still monolithic.
11:26
SPEAKER_00: Any driver in theirs can crash the kernel.
11:29
SPEAKER_00: within reason.
11:30
In that way,
11:31
we can refactor the way the interfaces between drivers and the
11:34
kernel are.
11:36
SPEAKER_00: Linux drivers are,
11:37
SPEAKER_00: on average,
11:37
one-third smaller than other operating system drivers because
11:40
we can see the
11:41
SPEAKER_00: commonalities.
11:41
SPEAKER_00: If you send three different drivers for three kind of same hardware,
11:44
SPEAKER_00: well,
11:45
SPEAKER_00: let's combine them all.
11:46
Make it smaller and refactor things and make it easier and,
11:49
oh,
11:49
let's change this API.
11:51
SPEAKER_00: And this has to do with the open source approach, right?
11:53
SPEAKER_00: Yeah,
11:54
so we see all this common code and we can refactor it and we
11:57
can make it better and
11:58
SPEAKER_00: cleaner.
11:58
SPEAKER_00: And we're not tied to any fixed interface.
12:01
SPEAKER_00: Our fixed interface is between the user space and the kernel.
12:04
SPEAKER_00: We will not break that.
12:05
SPEAKER_00: That's our guarantee.
12:07
SPEAKER_00: We've guaranteed it for a long, long time.
12:09
And so we always want you to be able to upgrade your kernel and
12:11
not feel worried
12:13
SPEAKER_00: that your old programs are going to crash.
12:15
SPEAKER_00: So you should always be able to upgrade.
12:16
SPEAKER_00: That's our guarantee to you.
12:17
SPEAKER_00: If it does break, then it's our fault, we'll regress.
12:20
SPEAKER_00: There are some exceptions,
12:21
SPEAKER_00: there's some gray areas,
12:22
there's some really low level parts between the user space and
12:25
kernel that we kind
12:25
SPEAKER_00: of work around and we argue about these all the time.
12:28
SPEAKER_00: But we never try and break user space on purpose.
12:30
SPEAKER_00: A lot of times we do it accidentally, we'll fix it up.
12:32
That's our only really rule of kernel development,
12:35
don't break user space on purpose.
12:37
SPEAKER_02: And so when we're talking about the,
12:39
SPEAKER_02: you know,
12:39
SPEAKER_02: the 1.5 million lines roughly for server,
12:42
SPEAKER_02: we're talking about the kernel.
12:42
SPEAKER_02: Yes, kernel plus drivers.
12:44
SPEAKER_02: Kernel plus driver.
12:45
SPEAKER_02: Well, because it is part of the kernel.
12:47
SPEAKER_02: And then,
12:47
SPEAKER_02: you know,
12:48
SPEAKER_02: you have this 40 million line of Tarbo,
12:50
SPEAKER_02: and then every platform will kind of take their parts of it.
12:53
SPEAKER_02: They'll take.
12:55
SPEAKER_02: what is relevant for their use case capabilities, drivers, other parts.
12:59
And then this is why,
13:00
I guess,
13:01
Raspberry Pi,
13:01
you're going to say it's going to run.
13:02
SPEAKER_02: It's laying on Linux, right?
13:04
SPEAKER_00: Of course.
13:05
SPEAKER_00: Raspberry Pi is huge.
13:07
SPEAKER_00: Those things are everywhere.
13:08
SPEAKER_00: That's what's in all the electric charging stations.
13:10
SPEAKER_00: Those are Raspberry Pis.
13:12
SPEAKER_00: Really?
13:13
SPEAKER_00: Yeah, where you plug your car into.
13:13
SPEAKER_00: Yeah, those are Raspberry Pis.
13:14
SPEAKER_00: Yeah, those are all Raspberry Pis.
13:15
SPEAKER_00: Wow.
13:16
SPEAKER_00: Because it's a really cheap industrial thing.
13:19
SPEAKER_00: Lots of signages now.
13:20
SPEAKER_00: Those are all running Raspberry Pis.
13:22
Yeah.
13:23
SPEAKER_02: I guess it was safe to assume they're not running Windows, to be fair.
13:26
SPEAKER_00: So the Dutch signage for the train, those are all running Linux.
13:30
SPEAKER_00: Sometimes you'll see a crash Linux machine up there.
13:32
Can we look at a specific example of how development actually
13:36
flows through with a
13:38
SPEAKER_02: specific patch?
13:39
Before I show a specific example,
13:40
so I'll say,
13:40
so we had 4,000 developers last year.
13:43
SPEAKER_00: So they make a change.
13:43
SPEAKER_00: So those 4,000 developers will send an email to a maintainer,
13:46
SPEAKER_00: and a maintainer maintains a subset of the kernel.
13:49
SPEAKER_00: Every part of the kernel is owned by somebody.
13:52
SPEAKER_02: And then you are one of these maintainers as well.
13:54
SPEAKER_00: Yeah, I'll maintain some drivers and things like that.
13:56
But then those maintainers send things off up the tree to a
13:59
subsystem maintainer.
14:00
So like USB serial,
14:02
then we'll get sent up to USB,
14:03
and then USB will go to Linus.
14:05
SPEAKER_00: Mm-hmm.
14:06
SPEAKER_00: So it's kind of a pyramid scheme that way.
14:08
SPEAKER_00: We have that.
14:08
SPEAKER_00: And so we have like 800 maintainers and we have the middle section,
14:12
SPEAKER_00: we maybe have about 200 different trees there.
14:15
And then in our testing environment,
14:17
all those trees are tested every day.
14:18
SPEAKER_00: They're all merged together and things that happen and whatnot.
14:22
SPEAKER_00: So we have this kind of hierarchy of developers and maintainers that way.
14:27
SPEAKER_00: And part of the hierarchy is the human aspect.
14:29
So if I take code from you as a maintainer,
14:33
I'm now responsible for it because my name's on it.
14:35
So if it's a simple one-off or it's a simple driver that nobody
14:38
cares about except you,
14:40
great.
14:40
SPEAKER_00: I know you're the only one that's going to be affected by it.
14:42
SPEAKER_00: It's fine.
14:43
SPEAKER_00: But if it's the core part of the kernel and I take changes from you,
14:46
SPEAKER_00: now I'm responsible for it if you disappear.
14:49
So I have to trust that either you're going to be here or that
14:52
I understand it good
14:52
SPEAKER_00: enough that I can maintain it.
14:53
SPEAKER_01: Mm-hmm.
14:54
SPEAKER_00: So part of Linux development trust or issue or model is trust,
14:59
SPEAKER_00: and it's trust in human interaction.
15:00
SPEAKER_00: Like, I will take stuff from people.
15:03
SPEAKER_00: Whatever they send me,
15:04
SPEAKER_00: because I trust,
15:05
SPEAKER_00: not that they got it right,
15:05
SPEAKER_00: but they'll be there to fix it when they get it wrong.
15:07
SPEAKER_00: Because we all get it wrong.
15:09
SPEAKER_00: And that's the part.
15:10
SPEAKER_00: So that's the trust model we have.
15:12
And then we've been burned in the past by some major features
15:16
where it landed in
15:17
SPEAKER_00: the networking core subsystem a long time ago.
15:19
And then once they landed and were merged and taken,
15:23
the email address behind it disappeared.
15:24
SPEAKER_00: And the network developers took six months to unwind the mess.
15:29
SPEAKER_00: So it's hard to change the core part of the kernel.
15:31
SPEAKER_00: For a good reason, because it affects everybody.
15:33
And also a good reason in that we want to make sure that you
15:36
are going to be there
15:36
SPEAKER_00: to fix it if it breaks.
15:38
SPEAKER_00: Yeah.
15:38
SPEAKER_00: But for drivers and things like that, we'll take anything.
15:40
SPEAKER_00: Drive by.
15:42
SPEAKER_00: It's really simple.
15:42
SPEAKER_00: Yeah.
15:43
SPEAKER_00: It's a very simple thing that way.
15:44
SPEAKER_00: But that's the hierarchy.
15:46
SPEAKER_00: So a change flows up the tree that way.
15:48
SPEAKER_00: Yeah.
15:48
SPEAKER_00: So I can show that.
15:49
SPEAKER_02: All right.
15:50
SPEAKER_02: So what are we going to see?
15:52
SPEAKER_00: So here is a change.
15:54
SPEAKER_00: So this was written by somebody named Chester.
15:58
um he made a change to the usb serial driver uh it's an option
16:04
the chip is called
16:05
option um these are a usb the serial devices um they're in
16:08
modems they're in lots
16:09
of different things there's a ton of different ones and there's
16:12
no standard for
16:13
these types of devices so you have to add a custom device id
16:15
for every single one
16:16
that you want to use it's just the way they work um so here's a
16:20
patch and here's
16:21
the description of it um this is just email the description
16:24
here there's the
16:25
SPEAKER_00: subject line
16:26
SPEAKER_02: Yeah, you have a serial option, adding whatever.
16:29
SPEAKER_00: Adding that device.
16:30
SPEAKER_00: And then here's,
16:31
SPEAKER_00: so good part about,
16:33
hardest part about writing a kernel change is the description
16:37
of what's going on.
16:38
SPEAKER_00: Really?
16:38
SPEAKER_00: Yeah, I mean, the code is easy.
16:39
SPEAKER_00: It's the description.
16:40
SPEAKER_00: Explaining it is hard, right?
16:41
SPEAKER_00: You don't explain usually what it's doing.
16:44
SPEAKER_00: You have to explain why you're doing it.
16:46
SPEAKER_00: For something as simple as this, it's like, it's really easy.
16:48
SPEAKER_00: So this person says, this driver is part of a Cat6 modem.
16:53
SPEAKER_00: The product ID is shared across multiple modems.
16:55
SPEAKER_00: It gives a little dump of what it looks like in the device.
16:58
SPEAKER_00: And then there's some more information.
17:00
SPEAKER_00: There's a signed off by line.
17:02
And signed off by is what we created a long time ago that shows
17:05
that I have the
17:07
proper authorship of this and ownership and I give it to this
17:10
project under the
17:11
SPEAKER_00: license by which the project is run.
17:14
SPEAKER_00: So it's saying I'm licensing this thing under the GPL.
17:18
SPEAKER_00: And then way down below is probably the one line patch.
17:22
SPEAKER_00: This is all just
17:23
SPEAKER_02: So this is all the description.
17:24
SPEAKER_02: This person is giving context on,
17:26
SPEAKER_02: like,
17:26
SPEAKER_02: here's what needs to do with the modem,
17:28
SPEAKER_02: the different specifications or whatever.
17:30
SPEAKER_00: And here's the change.
17:31
SPEAKER_00: So somebody changed this, removed that line.
17:33
SPEAKER_00: The red is removed.
17:34
SPEAKER_00: The green is add.
17:36
So somebody added and had to reformat the lines based on some
17:39
new ones they added.
17:41
SPEAKER_00: And that's it.
17:42
SPEAKER_00: So they add a few new device IDs.
17:44
SPEAKER_02: And then there's a device ID, and then we see a few hex numbers.
17:48
SPEAKER_02: Those are like some IDs here.
17:50
SPEAKER_00: So for USB, USB devices have a product and a vendor and a product ID.
17:56
SPEAKER_00: That's how they're vendor, and then they have products.
17:59
SPEAKER_00: And then there's some sub-device and sub-product IDs.
18:01
SPEAKER_00: Got it, got it.
18:02
SPEAKER_00: So that's what this is for.
18:04
So they're saying,
18:04
we're just adding support for some,
18:06
the driver already works for these chips.
18:09
But we just have a new ID because a new vendor came along and
18:11
they wanted to put
18:12
SPEAKER_00: their own vendor ID on it.
18:13
SPEAKER_00: Very common.
18:15
So it sounds like this change is as simple as I get in terms of
18:17
the code change.
18:18
SPEAKER_02: But still, the description was very extensive, right?
18:23
SPEAKER_00: So very extensive.
18:25
Part of the description was also just here is a dump of the
18:28
description of the hardware.
18:29
SPEAKER_00: Just so that we can verify, yeah, that is going to match with this.
18:33
SPEAKER_00: Got it.
18:33
SPEAKER_00: So we have tools that create those things.
18:37
SPEAKER_00: But yeah, it's a lot of work for a four-line change.
18:40
SPEAKER_02: Well, but this is like if we talk GitHub language, although I'm not sure.
18:46
SPEAKER_00: This will be a PR, right?
18:47
SPEAKER_00: This should be in the patch itself, not the PR.
18:52
SPEAKER_00: Ah.
18:52
SPEAKER_00: So a PR would be, so say you have 10 changes you want to make.
18:55
SPEAKER_00: So a PR would be the patch 0 out of 10.
18:58
SPEAKER_00: Got it, got it.
18:59
SPEAKER_00: Yeah, this is the commit.
19:00
SPEAKER_00: This is the commit itself,
19:00
which is a big problem of why I don't like the GitHub model is
19:03
because people don't
19:04
SPEAKER_00: put the changes in the GitHub,
19:06
SPEAKER_00: in the git commit.
19:07
SPEAKER_00: No.
19:07
SPEAKER_00: Because the git repo.
19:08
SPEAKER_00: We don't.
19:09
SPEAKER_00: So there's a problem.
19:09
SPEAKER_00: When you're looking at the repo later and you look at the change,
19:13
SPEAKER_00: you can't see the pull request information.
19:15
SPEAKER_00: Yeah.
19:16
SPEAKER_00: And it's gone.
19:17
SPEAKER_00: And that's a big problem, I feel, with the GitHub model.
19:19
SPEAKER_02: Well,
19:19
SPEAKER_02: I feel this goes back to,
19:21
SPEAKER_02: you know,
19:21
SPEAKER_02: like you built the tool or,
19:23
SPEAKER_02: you know,
19:24
Linux Group built a tool for your use case and you're using it
19:26
the way you intend
19:27
SPEAKER_02: to use it.
19:28
Whereas GitHub built the pull request workflows built on top of
19:31
this and it is not
19:32
SPEAKER_02: part of Git for whatever reason.
19:34
You know,
19:34
maybe GitHub could have made it part of Git whatnot,
19:37
but it's not,
19:38
right?
19:38
SPEAKER_00: Well, no.
19:39
SPEAKER_00: So we have pull requests.
19:42
SPEAKER_00: We created pull requests.
19:44
SPEAKER_00: In Linux, we email those git create pull requests.
19:47
SPEAKER_00: Oh, okay.
19:48
SPEAKER_00: That was a good command.
19:49
SPEAKER_00: It makes an email.
19:50
SPEAKER_00: Is that part of the git?
19:51
Yeah,
19:51
it makes an email that says pull from this repo and here's
19:54
everything that's in this repo.
19:56
When we do a merge of that,
19:59
that merge commit has all that information in it.
20:02
SPEAKER_02: Okay.
20:03
SPEAKER_00: And then so you'll,
20:04
SPEAKER_00: if you look at the Linux kernel and you see when you merge,
20:06
SPEAKER_00: when Linus merges in the USB tree,
20:08
SPEAKER_00: he sees my little message at the top saying,
20:11
SPEAKER_00: here's everything that's going to be in this pull request.
20:13
SPEAKER_00: You got it.
20:13
SPEAKER_00: And because Git is the source and that's where all the data is, right?
20:18
SPEAKER_00: And so you can see that we don't have pull requests, it's not external.
20:21
GitHub could change that model and put that in the merge
20:23
request,
20:23
but it doesn't.
20:24
SPEAKER_02: I was about to say that,
20:26
SPEAKER_02: like,
20:26
SPEAKER_02: because you did it,
20:27
SPEAKER_02: they could,
20:28
SPEAKER_02: but it's a matter of,
20:29
SPEAKER_02: I guess,
20:29
SPEAKER_02: preferences and,
20:30
SPEAKER_02: yeah.
20:31
SPEAKER_02: That's fine.
20:31
But the good thing about this is you can track any single line
20:34
of code back to who
20:36
SPEAKER_00: made it and what they did and what was the change log?
20:40
SPEAKER_00: What was the reasoning behind this?
20:43
SPEAKER_02: Which is great.
20:44
SPEAKER_02: So this person sends it to the module?
20:48
SPEAKER_00: So the owner to this is Johan.
20:52
SPEAKER_00: There's a script we have that says take any patch and
20:55
SPEAKER_00: Give me the people who are responsible for this and the mailing list.
20:58
So Johan and me picked this and sent it to the USB list and
21:03
copies a bunch of other
21:03
people that I guess they worked with and that have changed this
21:06
driver in the past.
21:07
SPEAKER_02: And then the copy is also done with the tool.
21:10
SPEAKER_02: As you said, it kind of looks to who touched this code or who might.
21:13
SPEAKER_02: Yeah, exactly.
21:14
SPEAKER_00: Oh, nice.
21:15
SPEAKER_02: This is all automated.
21:16
SPEAKER_00: Yeah, all automated.
21:16
SPEAKER_00: We do that.
21:17
SPEAKER_00: Awesome.
21:18
SPEAKER_00: So that was great.
21:19
SPEAKER_00: He sent it.
21:20
SPEAKER_00: And just the mailing list has two copies of it.
21:22
SPEAKER_00: That's just because I wanted two different mailing lists.
21:24
SPEAKER_00: But then they said, oops, I messed up.
21:29
SPEAKER_00: There's an email from the person instantly after he sent it off.
21:33
SPEAKER_00: It said, I messed up.
21:34
SPEAKER_00: There's an interface.
21:34
SPEAKER_00: Maybe it would be a good idea to change this comment.
21:38
SPEAKER_00: So they go and change the comment.
21:40
SPEAKER_00: And then they resend it.
21:41
SPEAKER_00: And you just send a new version.
21:43
SPEAKER_02: And then in this case, they send a new patch?
21:48
SPEAKER_02: Or do they just add one more?
21:51
SPEAKER_02: No, you want to have a clean commit, right?
21:53
SPEAKER_00: Yeah, we don't do it.
21:54
SPEAKER_00: So here they sent a version 2 patch, if you can see that.
21:58
SPEAKER_00: It says version 2, right?
22:00
SPEAKER_00: Oh, yeah.
22:01
SPEAKER_00: Version 2.
22:02
SPEAKER_00: Got it.
22:03
SPEAKER_00: And then here's the same information.
22:05
And then there should be some comments about what changed
22:07
between the two versions.
22:08
SPEAKER_00: Hopefully.
22:10
SPEAKER_00: Yes, changes in version two from the previous one.
22:13
SPEAKER_00: And there's a link back to the first one.
22:15
SPEAKER_00: Nice.
22:16
SPEAKER_00: Very nice.
22:17
SPEAKER_00: So that's what we do.
22:17
SPEAKER_00: So we want to see the changes because, I mean, I get 1,000 emails a day.
22:21
SPEAKER_00: And when I review patches and stuff,
22:23
SPEAKER_00: I'll review them,
22:24
SPEAKER_00: and then they're gone because I'm reviewing the next one.
22:26
SPEAKER_00: But when you send a next version,
22:29
I want to see what changed from the previous one because I
22:31
don't want to go back
22:31
SPEAKER_00: and dig through all the old stuff.
22:33
SPEAKER_00: So they added some information to it.
22:35
SPEAKER_00: Wonderful.
22:36
SPEAKER_00: And then what happened?
22:40
SPEAKER_00: Johan,
22:41
SPEAKER_00: who is the maintainer of the subsystem,
22:44
SPEAKER_00: wrote,
22:44
SPEAKER_00: said,
22:45
SPEAKER_00: hey,
22:45
SPEAKER_00: thanks for the patch and help for documenting it.
22:47
SPEAKER_00: Oh, I did something else.
22:49
SPEAKER_00: I got that out of order.
22:51
First,
22:51
they said,
22:53
oh,
22:54
Chester wrote,
22:55
hey,
22:56
please apply this after two weeks,
22:58
after a week.
22:59
SPEAKER_00: He said, because after a week or after two weeks, it's nice.
23:02
SPEAKER_00: Hey, what happened to this?
23:03
SPEAKER_00: What's going on?
23:04
SPEAKER_00: Johan said, you submitted this patch during the merge window.
23:08
SPEAKER_00: I'll talk about how we do our development model,
23:10
but there's a two-week merge window when we do releases that
23:13
Linus takes all the
23:13
changes from all the maintainers that have been in their
23:16
development trees.
23:17
SPEAKER_00: We can't add new changes at that point in time.
23:19
SPEAKER_00: So there's a two-week kind of blackout for new development.
23:22
But this is where all the stuff is flowing into Linus for the
23:24
next release.
23:25
SPEAKER_00: So during that time,
23:27
SPEAKER_00: if you send me a patch,
23:28
SPEAKER_00: I can't really do anything with it,
23:29
SPEAKER_00: but it'll stick in my mailbox until then.
23:31
SPEAKER_00: So this happened to hit that little window of time.
23:34
SPEAKER_02: Just to understand, there's a nine-week release.
23:37
SPEAKER_02: Because every nine weeks, there's a new release going out, right?
23:41
SPEAKER_02: And then there's a window where patches are gathered.
23:44
SPEAKER_02: So yeah, Harold, let's talk about that.
23:45
SPEAKER_00: So Linux does a release.
23:47
SPEAKER_00: Yeah.
23:47
SPEAKER_00: At this point in time.
23:50
SPEAKER_00: And then the merge window is considered open.
23:53
SPEAKER_00: And then for two weeks,
23:54
all the maintainers send Linus all the stuff they've had
23:57
pending from the last
23:59
SPEAKER_00: release.
23:59
SPEAKER_00: Yeah.
24:00
SPEAKER_00: We have two weeks to add all new features.
24:02
SPEAKER_00: Yeah.
24:03
SPEAKER_00: And then he does release candidate one.
24:05
SPEAKER_00: And then from there on, it's bug fixes only for the next seven weeks.
24:09
SPEAKER_00: So it's bug fixes only, bug fixes only.
24:11
SPEAKER_00: It's regression fixes.
24:12
SPEAKER_00: We'll revert things.
24:13
SPEAKER_00: And so it's no new features.
24:15
SPEAKER_00: But during that seven weeks, people are sending me new features.
24:18
SPEAKER_00: So I have a separate tree.
24:20
SPEAKER_00: Which is my next tree.
24:22
SPEAKER_02: You're not batching it for when the window will open.
24:24
SPEAKER_00: Yeah, so we call it next, Linux next.
24:26
So we have a next tree where all these are merged together on a
24:29
daily basis to see,
24:31
SPEAKER_00: to make sure they work.
24:32
SPEAKER_00: Because be prepared for Linux's next one.
24:34
SPEAKER_00: And then when he does a release after everything's good,
24:37
SPEAKER_00: we all throw things at him again in another two weeks.
24:40
SPEAKER_00: Now, Linus doesn't pull automatically from all those merged trees.
24:44
SPEAKER_00: We have to explicitly ask him.
24:46
SPEAKER_00: Yeah.
24:46
SPEAKER_00: Because sometimes our trees aren't good.
24:48
SPEAKER_00: Yeah.
24:48
SPEAKER_00: So sometimes, like, I've maintained the TTY in Serial.
24:51
SPEAKER_00: One time, famously, it was a mess.
24:53
SPEAKER_00: It just wasn't working.
24:54
SPEAKER_00: There was new features added.
24:55
SPEAKER_00: So I'm like, I'm skipping this release cycle.
24:57
I'm going to pull out some of these bug fixes and send them to
24:59
you off the side and then go.
25:00
But if it was,
25:01
like,
25:01
automatically being merged in,
25:03
we'd have to deal with that mess.
25:04
it's just interesting because most companies just you know
25:08
reflecting on you know
25:09
the companies that use get large tech companies they often have
25:12
let's say let's
25:12
talk about native mobile development where there is a concept
25:15
of releasing every
25:18
week or two weeks because of the app store review process or
25:21
same with like desktop
25:22
SPEAKER_02: apps that you can't really just continue to release there's usually a
25:26
an aim for something but it's not as strict so every now and
25:30
then it would also
25:30
happen that you know it's just not stable enough we'll push it
25:34
back but there is
25:35
not this rigid like clockwork like I think you know most
25:38
companies that I've seen
25:40
they just treat it a bit more flexible because again you know
25:42
they come up with the
25:44
SPEAKER_00: thing that they're in charge with it your feature you want to have added
25:47
SPEAKER_02: Yeah,
25:47
SPEAKER_02: and then as we know,
25:48
SPEAKER_02: when you have a milestone,
25:50
SPEAKER_02: features might be cut,
25:52
SPEAKER_02: deadline might be moved.
25:53
SPEAKER_02: But do I understand correctly that in the case of Linux,
25:58
SPEAKER_02: is this a thing where every nine weeks there will be a release?
26:01
SPEAKER_00: It's time-based.
26:02
So we have that two-week window of merging all the new features
26:05
to Linux that have
26:06
SPEAKER_00: been in our tree and accepted already and proven to work.
26:09
SPEAKER_00: And the window is short, eight, nine weeks.
26:11
SPEAKER_00: And that's good,
26:12
SPEAKER_00: because we used to have two-year-long development cycles,
26:15
SPEAKER_00: three-year-long development cycles.
26:16
SPEAKER_00: And the problem there is,
26:18
SPEAKER_00: even if you have six-month development cycles,
26:20
SPEAKER_00: there's that fear of,
26:21
SPEAKER_00: you have a feature.
26:22
SPEAKER_00: I want to take your feature, but it's not quite ready.
26:24
SPEAKER_00: Do I want to wait?
26:26
SPEAKER_00: But if you know that you can get your feature in nine weeks from now,
26:30
SPEAKER_00: and it's just not ready,
26:32
SPEAKER_00: it's not ready.
26:32
And it's much more like,
26:34
OK,
26:35
the pressure is off me as a maintainer to take your new
26:37
feature.
26:38
SPEAKER_00: until it's ready.
26:39
SPEAKER_02: You can say,
26:40
SPEAKER_02: like,
26:40
SPEAKER_02: look,
26:40
SPEAKER_02: if it'll make it into the next one,
26:42
or let's make sure it's going to work properly if it's a more
26:45
complex one.
26:45
SPEAKER_00: Yeah, we have lots of features.
26:46
I mean,
26:46
famously,
26:47
there's a USB feature that's on patch version number 35.
26:50
SPEAKER_00: This 25 patch series, it's on the 35th version.
26:54
SPEAKER_00: And it's just not ready.
26:55
And I just got emailed today saying,
26:56
well,
26:57
maybe we need to change this to this other way.
26:58
SPEAKER_00: I mean, so I feel so bad for that developer.
27:01
SPEAKER_00: But he's been working hard.
27:02
SPEAKER_00: And it's a complicated feature.
27:04
SPEAKER_00: And it's taken him a year and a half to get there.
27:07
SPEAKER_00: I have other patches that are in version 3.
27:09
SPEAKER_00: But that's version 3.
27:10
SPEAKER_00: And it's been two years.
27:11
SPEAKER_00: Because the developer just took a lot of time in between.
27:14
SPEAKER_02: Okay.
27:14
SPEAKER_02: So,
27:15
SPEAKER_02: in this case,
27:15
SPEAKER_02: this is a good example that,
27:17
SPEAKER_02: you know,
27:18
SPEAKER_02: the person,
27:19
SPEAKER_02: the contributor said like,
27:21
SPEAKER_02: hey,
27:21
SPEAKER_02: a reminder,
27:22
SPEAKER_02: I'd like this patch applied.
27:23
SPEAKER_00: Yeah, it was fine.
27:23
SPEAKER_02: And then Johan replied, reminding of the timeline of how it works, right?
27:28
SPEAKER_02: Yeah, exactly.
27:29
SPEAKER_02: And Chester wrote back.
27:30
SPEAKER_02: And then really friendly, it'll be in the next one, don't worry.
27:33
SPEAKER_00: Yeah.
27:33
SPEAKER_02: Which is nice, very positive.
27:34
SPEAKER_00: Yeah.
27:35
SPEAKER_00: We're not mean people.
27:37
SPEAKER_00: Yeah.
27:38
SPEAKER_00: And then reminding,
27:38
don't ever feel bad about reminding me that I haven't reviewed
27:41
your patch in two
27:42
SPEAKER_00: weeks.
27:43
SPEAKER_00: Now, if I haven't reviewed it in two days, yeah, I'll be a little testy.
27:46
SPEAKER_00: But two weeks is a good idea.
27:48
SPEAKER_00: And then, Chester, thanks a lot for keeping an eye on it.
27:50
SPEAKER_00: Keep up the good work.
27:51
SPEAKER_00: And that's it.
27:52
SPEAKER_00: So then Johan has it.
27:53
Johan applied it to his tree because he then wrote saying,
27:56
hey,
27:57
and Johan was very nice here.
27:58
SPEAKER_00: He said, you kind of didn't do the comments in the proper format.
28:01
SPEAKER_00: I fixed it up for you.
28:03
SPEAKER_01: Oh, nice.
28:03
SPEAKER_00: So for drive-by changes like that, we want to make it really easy.
28:06
SPEAKER_00: We're not mean people.
28:07
I mean,
28:08
clearly,
28:09
this feels like it's a person who is unlikely to become a
28:13
regular contributor.
28:14
SPEAKER_02: They're getting their work done, right?
28:15
SPEAKER_02: They're adding.
28:17
SPEAKER_00: They have a device that they have to ship.
28:18
SPEAKER_02: Yeah, pretty much.
28:20
We want to be friendly and open and easy to everybody because
28:23
everybody submits
28:24
SPEAKER_00: their first patch at one time,
28:25
SPEAKER_00: right?
28:26
Famously,
28:26
when I did my very first patch,
28:28
I wrote an email saying,
28:28
how do I make a patch?
28:29
SPEAKER_00: Because we didn't have good documentation.
28:30
SPEAKER_00: Somebody wrote back and said, hey, here's how you do it.
28:33
SPEAKER_00: He became my boss eight years later.
28:35
SPEAKER_00: It was like I ended up working for him.
28:37
SPEAKER_00: It's just funny.
28:38
SPEAKER_00: It's just like a small world and whatnot.
28:39
SPEAKER_00: But yeah, and we want to make it easy.
28:42
So Johan takes this,
28:43
and he's got the patch,
28:44
and it's in his tree now,
28:45
which is great.
28:46
SPEAKER_00: But that's in his local little tree.
28:48
SPEAKER_00: Then he has to get it off somewhere else.
28:50
SPEAKER_00: Johan then makes a pull request to me.
28:53
SPEAKER_00: So this is an output of the git make pull request.
28:57
SPEAKER_00: I don't know what the actual command is.
28:59
SPEAKER_00: And this is what a pull request from git will look like.
29:01
SPEAKER_00: And this is because Johan is a subsystem maintainer?
29:03
SPEAKER_00: Yes, Johan maintains the USB to serial drivers.
29:06
SPEAKER_00: There's a bunch of drivers for these types of things.
29:08
SPEAKER_00: And then he sends it off to me, the USB maintainer.
29:11
SPEAKER_00: Got it.
29:12
SPEAKER_00: And he says, take this...
29:14
SPEAKER_00: Patch or pull from this tree of this tag.
29:17
SPEAKER_00: And it's a sign tag.
29:18
SPEAKER_00: So it's signed with his GBG key.
29:19
SPEAKER_00: So I can verify that.
29:20
SPEAKER_00: It's really him when I pull from it.
29:22
SPEAKER_00: And it says, take these patches.
29:23
SPEAKER_00: And here's the information.
29:24
SPEAKER_00: It's going to be some USB device IDs.
29:26
SPEAKER_00: And they have all been in Linux Next with no reported issues.
29:29
SPEAKER_00: So they've been tested.
29:30
SPEAKER_00: In our integrated testing, we test all this stuff every day.
29:33
SPEAKER_02: And what does testing mean?
29:35
SPEAKER_02: Is it automated testing?
29:37
SPEAKER_02: Is it pushing it out on devices and device labs?
29:40
SPEAKER_02: Is it a mix?
29:41
SPEAKER_02: Yes, it's all of that.
29:42
SPEAKER_00: So we have one.
29:43
SPEAKER_00: So Linux Next gets merged every day as a developer in Australia.
29:46
He merges all the trees together and builds them and boots them
29:50
in virtual machines.
29:52
SPEAKER_00: That's a nontrivial thing for a kernel to do.
29:55
SPEAKER_00: Just to boot.
29:55
SPEAKER_00: If it can boot, it's usually, hey, things are going well.
29:58
SPEAKER_00: It isn't testing on real devices.
30:00
SPEAKER_00: Now,
30:00
SPEAKER_00: there are other labs out there with Kernel CI,
30:02
SPEAKER_00: which is our CI infrastructure,
30:04
SPEAKER_00: that can run on all individual labs.
30:05
SPEAKER_00: And we do push things out there.
30:07
SPEAKER_00: And there are people testing Linux Next on their real hardware,
30:09
SPEAKER_00: sending us reports back in an automatic fashion.
30:12
SPEAKER_00: Those are less rare.
30:14
SPEAKER_00: Linus's tree gets tested more on that.
30:16
SPEAKER_00: The stable trees,
30:17
SPEAKER_00: I can talk about stable trees in a little bit,
30:18
SPEAKER_00: get tested more on the real hardware more.
30:21
SPEAKER_00: Linux Next gets build and boot tested pretty well.
30:24
um i don't run linux next i run the my development trees online
30:28
so i don't run all
30:29
the mix of them all sometimes they interact because we don't
30:32
have any fiefdoms um
30:34
if i have a usb change that needs to actually go through the
30:37
networking stuff i can
30:38
change the networking code and whatnot like that and they can
30:41
say hey maybe you
30:42
shouldn't do that and we try and get approval you review my
30:44
patches but it's now we
30:46
SPEAKER_00: can touch any any but he can touch any part of the kernel in a way
30:49
SPEAKER_00: But he sent me a pull request,
30:50
SPEAKER_00: and a pull request is that I don't actually review the changes in it.
30:54
SPEAKER_00: I'm not reviewing each individual patch through email.
30:56
SPEAKER_00: I'm trusting that he sent me four patches here and that they're good.
31:01
And I have known Johan,
31:03
and I know that he will be there if something goes wrong.
31:05
SPEAKER_02: Yeah,
31:06
and will you read the description and then every now and then
31:11
you might decide,
31:11
SPEAKER_02: for example,
31:12
SPEAKER_02: to deep dive into a change?
31:14
SPEAKER_00: Yes, totally.
31:14
SPEAKER_00: I mean,
31:15
SPEAKER_00: for USB device IDs,
31:16
SPEAKER_00: it's like,
31:16
SPEAKER_00: okay,
31:16
SPEAKER_00: yeah,
31:17
SPEAKER_00: well,
31:17
SPEAKER_00: they're all attached to the same driver.
31:18
SPEAKER_00: Yeah, these are common.
31:20
SPEAKER_00: They're nothing simple.
31:21
SPEAKER_00: Sometimes they're a little more complex.
31:23
I don't pull from a lot of different trees,
31:25
but I pull from some that I trust.
31:27
Some subsystems that I don't necessarily trust as well,
31:30
I will make you send them an email.
31:32
SPEAKER_00: And I'll actually review them,
31:33
SPEAKER_00: and then I'll review them,
31:34
SPEAKER_00: and then when I review them,
31:35
SPEAKER_00: I add my signed off by to it.
31:37
SPEAKER_02: And I guess part of trust will be here.
31:39
I'm just going to assume that since you and Johan know each
31:42
other well and you've
31:43
SPEAKER_02: worked for a while,
31:44
SPEAKER_02: Johan will probably also every now and then give a comment saying,
31:47
SPEAKER_02: hey,
31:47
SPEAKER_02: Greg,
31:48
SPEAKER_02: there's this change.
31:49
SPEAKER_02: Can you take an extra look on this thing, etc. ?
31:52
SPEAKER_00: Yeah, so sometimes Johan makes changes to the code himself.
31:56
SPEAKER_00: Or I make changes to the code myself.
31:57
SPEAKER_00: I put it out for review.
31:59
SPEAKER_00: And I have other people review my changes.
32:01
SPEAKER_02: This is just fascinating for me to tell you explain how trust
32:08
SPEAKER_02: between people, maintainers, is so important for efficient development?
32:13
SPEAKER_02: Yeah.
32:14
SPEAKER_00: It's all,
32:15
SPEAKER_00: it's,
32:16
SPEAKER_00: yeah,
32:16
SPEAKER_00: it's,
32:17
SPEAKER_00: and then also the trust is,
32:18
somebody once told me that Linux development was the scariest
32:21
thing they ever did
32:22
SPEAKER_00: because not because it was like difficult or whatnot,
32:24
SPEAKER_00: but it's because my name is on this change and it's public.
32:27
SPEAKER_00: That makes you as an engineer do really, really good work.
32:30
SPEAKER_01: Mm-hmm.
32:31
SPEAKER_00: I mean,
32:31
so much so that this person who said this patch went back and
32:34
looked at it
32:35
SPEAKER_00: instantly and said,
32:35
SPEAKER_00: oh,
32:35
SPEAKER_00: wait,
32:36
SPEAKER_00: the comment could be made a little bit better.
32:38
SPEAKER_00: And they're like, oh, yeah.
32:39
So,
32:39
I mean,
32:39
that's not a normal development process in a company that I
32:42
commit to Go.
32:44
SPEAKER_02: It makes me wonder about a few things that I kind of took for granted.
32:49
SPEAKER_02: For example, could this mean that closed-door software were...
32:54
SPEAKER_02: the outside world does not know how it's done.
32:56
SPEAKER_02: Maybe there's just a bit less incentive to do such great work.
33:00
SPEAKER_02: And actually, it's just a reflection.
33:01
SPEAKER_02: I do remember when I worked at a company and when we,
33:05
SPEAKER_02: actually my team,
33:06
SPEAKER_02: we open sourced a component that we built.
33:08
And I just remember how I put in way more work into that to
33:13
make it look good,
33:14
SPEAKER_02: to have the document,
33:15
SPEAKER_02: not just look good,
33:16
SPEAKER_02: but to make it clean.
33:17
SPEAKER_02: We cleaned up actual tech depth before we published it.
33:21
SPEAKER_02: And we didn't do that with our stuff.
33:24
SPEAKER_00: So open source development,
33:27
SPEAKER_00: by virtue of just human pressure,
33:30
SPEAKER_00: makes a better engineering product.
33:32
SPEAKER_00: It's a better engineering.
33:33
And we've kind of shown that through the years that this
33:36
development model creates
33:38
SPEAKER_00: a better software.
33:39
I'm kind of revisiting some of my,
33:41
like,
33:42
not assumptions,
33:42
but I never thought of it like this.
33:44
SPEAKER_02: But it's just awesome to see this.
33:47
SPEAKER_02: So then what happens next after Johan sends it to you?
33:50
SPEAKER_00: So Johan sends it to me.
33:51
SPEAKER_00: And then I take it, and I put it in my tree.
33:53
SPEAKER_00: I think I sent.
33:54
SPEAKER_00: I sent him saying I took it.
33:56
SPEAKER_00: And then if you take it, you're responsible, right?
33:59
SPEAKER_00: I pulled it and pushed it out, yes.
34:01
SPEAKER_00: And there's my email that says that.
34:03
SPEAKER_00: So now I'm responsible.
34:04
SPEAKER_00: It's in my tree.
34:05
SPEAKER_00: So now, since this is a device ID, these can go to Linus at any time.
34:09
SPEAKER_00: We can add bug fixes or new device IDs.
34:12
SPEAKER_00: These are tricky.
34:13
SPEAKER_00: So then a few days later, I send this change off to Linus.
34:16
SPEAKER_00: So I send Linus.
34:17
SPEAKER_00: I said, hey, Linus.
34:19
Take all these following changes,
34:20
these changes,
34:21
and here's a whole bunch of USB fixes.
34:23
SPEAKER_00: So here's some small driver fixes, some new device IDs.
34:26
SPEAKER_00: And so I summarize it all.
34:28
SPEAKER_00: I say these are all the things in here.
34:30
Yeah,
34:30
and these are going to be like a few dozen of patches,
34:34
something like that.
34:35
SPEAKER_00: Yeah, but here's the list of the patches down below.
34:38
SPEAKER_00: And here's the diff stat of them.
34:39
Here's the diff of them to make sure that this diff matches
34:42
what he pulls from.
34:44
SPEAKER_00: This is signed with Mikey.
34:45
SPEAKER_00: I do say almost all of these have been Linux Next.
34:48
SPEAKER_00: I guess some of them slipped in.
34:50
SPEAKER_00: But we also have another testing.
34:51
SPEAKER_00: When you send patches to the mailing list, we call it a zero-day bot.
34:55
SPEAKER_00: We'll go through and start applying them and build testing them.
34:58
SPEAKER_00: And that's run.
34:58
And then our own trees that we create also does verification
35:02
that they did build and boot.
35:03
SPEAKER_00: And it will run some benchmarks.
35:05
SPEAKER_00: For drivers, it doesn't really run benchmarks.
35:08
SPEAKER_00: And so then Linus takes this and he puts it in his tree.
35:12
SPEAKER_00: So then it got picked up.
35:13
SPEAKER_00: So it got picked up another day later.
35:15
SPEAKER_00: And then let's talk about how we do our model.
35:18
SPEAKER_00: So Linus does a release every nine weeks.
35:21
SPEAKER_00: Bug fixes come in during those nine weeks for the last release.
35:23
SPEAKER_00: You're running the last release, right?
35:25
SPEAKER_00: You want those bug fixes.
35:27
SPEAKER_00: You had a device that's running those bug fixes.
35:29
SPEAKER_00: A long time ago, we realized that people don't want to wait eight weeks.
35:32
So let's create a model of we have a development tree and we
35:35
have a stable tree.
35:36
So when Linus does a release,
35:38
I fork off Linus's branch and I say,
35:40
this is a stable branch.
35:41
SPEAKER_00: So 6.4, I do 6.4.1.2.3.4.5.
35:44
SPEAKER_00: And our release numbers are just numbers.
35:48
SPEAKER_00: They mean nothing.
35:49
SPEAKER_00: They're not semantic versioning.
35:50
SPEAKER_00: We were around way before that happened.
35:52
SPEAKER_00: They're just meaning this number is later than that number.
35:55
SPEAKER_00: Yep.
35:55
SPEAKER_00: That's all.
35:56
SPEAKER_00: When we switch from 4.x to 5.x, it's just because the x got too big.
36:00
SPEAKER_00: Yeah.
36:01
And in your brain,
36:02
when you see a number between like 14 and 18,
36:04
it looks smaller than 4 to 8.
36:07
SPEAKER_00: Yeah.
36:08
SPEAKER_00: So we just bump it up every couple of years.
36:11
SPEAKER_00: So we take stable.
36:13
SPEAKER_00: We have stable releases.
36:14
SPEAKER_00: I do a release every week.
36:15
SPEAKER_00: And what I do is the patches have to be in Linus's tree first.
36:19
SPEAKER_00: We can't diverge.
36:20
So if it's in Linus's tree first and it's a bug fix and it
36:22
meets its criteria,
36:23
SPEAKER_00: I put it in the stable tree and I do a release.
36:25
SPEAKER_00: And so we do new releases every week.
36:27
for that so during those nine weeks i'll take new device ids
36:31
i'll take bug fixes
36:32
and whatnot and then you can tag the fixes that are going into
36:36
the tree with a
36:36
special way that i'll automatically take them i know to look at
36:39
them the other
36:39
stable tree maintainer with me sasha he runs through them and
36:42
runs a whole bunch of
36:43
fuzzing he's been doing ai before it was called ever ai um it's
36:47
just pattern
36:47
matching i mean and we have a whole body of here's a whole
36:50
bunch of bug fixes
36:51
SPEAKER_00: here's a whole bunch of changes did anybody do these kind of match
36:55
SPEAKER_00: Oh, yeah.
36:55
SPEAKER_00: Because some people don't realize that, oh, this was a bug fix.
36:58
SPEAKER_00: It should go into the stable tree.
37:00
SPEAKER_00: They've written academic papers on it for years.
37:02
SPEAKER_00: It's fun stuff.
37:04
SPEAKER_00: So just pattern matching, right?
37:05
So they will pick up a whole bunch of stuff that,
37:07
hey,
37:07
maybe you forgot about that.
37:08
And it'll give you a chance to respond before it goes into the
37:11
stable tree.
37:12
SPEAKER_00: And we do those releases.
37:13
When Linus does a new release,
37:14
then I throw that stable tree away,
37:15
and I make a new stable tree.
37:18
SPEAKER_00: That's great for things that I can update more often.
37:21
SPEAKER_00: People want to make a device.
37:22
SPEAKER_00: You want to make something that's going to last a long time.
37:24
SPEAKER_00: So what we've come up with, the idea is long-term stable trees.
37:27
SPEAKER_00: And there I pick one kernel a year and I maintain it for,
37:31
SPEAKER_00: to start with two years,
37:32
SPEAKER_00: sometimes six years.
37:34
SPEAKER_00: So your Android phone is running off a kernel that's five years old,
37:37
SPEAKER_00: but it's still getting bug fixes back to it.
37:38
SPEAKER_00: So I maintain like four long-term stable trees at the same time.
37:43
SPEAKER_00: And we backport all these fixes to all the different branches,
37:45
SPEAKER_00: and then we pick one a year and we maintain these.
37:47
SPEAKER_00: So there's six of them going at a time.
37:49
And in this case,
37:50
like,
37:51
is it you,
37:52
like,
37:53
there's one maintainer for each of these long-term?
37:56
SPEAKER_00: No, it's just me.
37:57
SPEAKER_00: Oh, it's just you.
37:58
SPEAKER_00: Wow, okay.
37:59
SPEAKER_00: Yeah, it's the two of us.
38:01
The longer,
38:01
the interesting thing is the older the code is,
38:04
the harder it is to maintain.
38:05
And the company's like,
38:06
oh,
38:06
I'll put a junior developer to maintain old code.
38:08
That's harder because it's more diverged from what the latest
38:11
developers are using.
38:12
SPEAKER_02: Can you tell me a little bit more about this?
38:14
SPEAKER_02: Because the older the code, the harder it is to maintain.
38:18
SPEAKER_02: I think it feels true, but why is this the case?
38:24
SPEAKER_00: Is it just lost context?
38:26
SPEAKER_02: So development moves on, goes forward, right?
38:29
So say a change I make today to the code base that fixes the
38:32
bug that affects the
38:33
SPEAKER_00: code that I look back,
38:34
SPEAKER_00: it's affected the code for the past 10 years.
38:37
SPEAKER_00: If I try to start backporting this change to code that's 10 years old,
38:41
SPEAKER_00: code has evolved in that time.
38:43
SPEAKER_00: And making that change to older code is harder.
38:45
And the more I have to change it,
38:47
the more it diverges from the original fix.
38:49
So the more context and skill you have to have to make the
38:53
change to the older code
38:54
SPEAKER_00: base than even the developer who made the first change.
38:57
SPEAKER_00: It's not intuitive.
38:59
SPEAKER_00: Companies make this mistake all the time,
39:00
SPEAKER_00: thinking,
39:01
SPEAKER_00: oh,
39:01
SPEAKER_00: I'll just maintain this old code base for a long time.
39:03
SPEAKER_00: We have major security bugs like Spectra meltdown with chips.
39:08
Some of those Spectra fixes have not been backported to some of
39:11
the long-term
39:11
kernels that are still being supported because it was just too
39:14
hairy of a fix.
39:15
SPEAKER_00: Anybody who cared, move to a new kernel.
39:16
SPEAKER_00: Yeah, yeah, yeah.
39:17
SPEAKER_00: So I look at a lot of these older kernels as it's,
39:19
SPEAKER_00: again,
39:20
SPEAKER_00: if you're using it,
39:21
SPEAKER_00: you will provide the resources to maintain it.
39:23
SPEAKER_00: Google,
39:23
SPEAKER_00: I'll call out,
39:24
SPEAKER_00: and Lenaro,
39:25
SPEAKER_00: Google's another group,
39:27
do a lot of work in testing these old kernels because Google
39:29
cares a lot about
39:29
SPEAKER_00: these kernels.
39:30
So they provide testing infrastructure and merges and
39:34
reproducibility and running
39:35
on real devices to make sure that these kernels still work on
39:38
them and they work
39:38
SPEAKER_00: well.
39:39
SPEAKER_00: And that way I know that if I make a change back there, it'll still work.
39:42
SPEAKER_00: If I didn't have that resources for them doing that work,
39:45
SPEAKER_00: I wouldn't be able to maintain these old kernels.
39:46
SPEAKER_02: Yeah.
39:48
And then going back to the bug fix,
39:49
so like every week there's a new stable branch release.
39:53
SPEAKER_02: And then when does the big release come, the nine-week release come?
39:57
SPEAKER_02: That's after this has been kind of baking,
39:59
SPEAKER_02: right,
40:00
SPEAKER_02: for the stable branch has been...
40:01
SPEAKER_00: So stable's independent of Linux's tree.
40:03
SPEAKER_00: Oh, stable's independent.
40:03
SPEAKER_00: So the only tie is it has to be in Linux's tree first.
40:07
SPEAKER_00: We do not want divergent...
40:08
We don't want you to make a fix to a stable tree only in
40:11
non-Linus's tree.
40:12
SPEAKER_00: Got it.
40:12
So sometimes I will have bugs in the stable tree due to other
40:15
changes I've taken.
40:16
SPEAKER_00: I mean, fixes need fixes.
40:18
And I'm like,
40:19
I can't take the fix for this until you get the fix in Linux's
40:21
tree.
40:22
And it's kind of a forcing function on a developer to get a fix
40:25
to Linux before
40:27
SPEAKER_00: I'll take it for the stable tree.
40:28
SPEAKER_00: Sometimes I'll revert the change in the stable tree.
40:30
SPEAKER_02: And do I understand the way to get a fix into Linux is,
40:34
SPEAKER_02: well,
40:34
SPEAKER_02: of course,
40:34
SPEAKER_02: you need to get a fix into Linux's tree,
40:37
which means you need to go through one of the maintainers who
40:41
maintains one of the
40:43
SPEAKER_02: subsystems.
40:46
SPEAKER_00: Yeah.
40:47
SPEAKER_02: And you just need to go up the tree, up the pyramid.
40:49
SPEAKER_00: Right.
40:49
SPEAKER_00: So famously, Bluetooth always breaks every other release.
40:53
SPEAKER_00: Bluetooth is crazy complex.
40:55
SPEAKER_00: The hardware is horrible.
40:56
SPEAKER_00: And if you need to get a fix in there,
40:57
SPEAKER_00: it has to go to the Bluetooth tree,
40:58
SPEAKER_00: and then that gets sucked into the networking tree,
41:00
SPEAKER_00: and then that network tree goes to Linux.
41:02
SPEAKER_00: So it's like a two-stage process.
41:04
SPEAKER_00: Sometimes, and then we have somebody tracking regressions.
41:07
SPEAKER_00: Regressions are really important.
41:08
SPEAKER_00: We don't want anything to regress.
41:09
Sometimes Linus will say,
41:10
I'll just take these bug fixes that are regressions.
41:12
SPEAKER_00: I'll just take them now.
41:12
SPEAKER_00: Boom.
41:13
SPEAKER_00: I'll just take them.
41:14
SPEAKER_00: So it depends on what they are.
41:16
SPEAKER_00: If they affect hardware that's really common,
41:18
SPEAKER_00: we prioritize that over hardware that isn't as common.
41:22
SPEAKER_00: Just by virtue of, hey, this broke my laptop.
41:24
SPEAKER_00: Right.
41:25
SPEAKER_00: I want to keep working.
41:25
SPEAKER_00: So, yeah.
41:26
SPEAKER_00: It's a little thing that way.
41:27
SPEAKER_00: So, we have two branches going at once.
41:29
SPEAKER_00: Development and then stable releases happening.
41:31
SPEAKER_00: So, then this went into Linus's tree.
41:34
SPEAKER_00: I picked this out as part of the stable trees.
41:36
SPEAKER_00: And then they ended up in the stable tree somewhere as well.
41:41
SPEAKER_00: And then I can give you dates for all this stuff.
41:45
SPEAKER_00: This whole process took about a week and a half.
41:49
SPEAKER_00: And that was it.
41:49
SPEAKER_00: Okay.
41:50
SPEAKER_00: Okay.
41:51
SPEAKER_00: And then here is, it ended up in the 6.13.4 kernel as well.
41:56
SPEAKER_00: And other ones as well.
41:59
SPEAKER_02: Trust isn't just earned, it's demanded.
42:02
Whether you're a startup founder navigating your first audit or
42:05
seizing security
42:05
SPEAKER_02: professional skill in your governance risk and compliance program,
42:09
proving your commitment to security has never been more
42:11
critical or more complex.
42:13
SPEAKER_02: That's where Vanta comes in.
42:15
Vanta can help you start or scale your security program by
42:18
connecting with auditors
42:19
and experts to conduct your audit and set up your security
42:22
program quickly.
42:23
SPEAKER_02: Plus,
42:24
SPEAKER_02: with automation and AI throughout the platform,
42:26
SPEAKER_02: Vanta gives your time back so you can focus on building your company.
42:29
Businesses use Vanta to establish trust by automating
42:32
compliance needs across over
42:33
SPEAKER_02: 35 frameworks like SOC 2 and ISO 27001.
42:37
SPEAKER_02: Thank you.
43:01
So we saw what it takes to get a fix into Linux,
43:05
and it actually wasn't that complicated.
43:07
SPEAKER_00: No, it really isn't.
43:08
SPEAKER_00: I mean, it's just you email, it change off.
43:10
SPEAKER_02: You email, you use the Git workflow.
43:12
SPEAKER_02: If you're familiar with Git, it's pretty simple.
43:14
SPEAKER_02: Obviously,
43:15
SPEAKER_02: you need to be able to build Linux,
43:18
SPEAKER_02: test it yourself,
43:20
SPEAKER_02: validate locally that it works,
43:22
SPEAKER_02: the basic things,
43:23
SPEAKER_02: and then
43:23
SPEAKER_02: It's straightforward.
43:24
The fun thing is,
43:25
so I can take a change like that without really testing it,
43:28
because it built.
43:28
SPEAKER_00: It obviously works for your hardware.
43:30
SPEAKER_00: I didn't test it, but it works, and I assume that it goes.
43:33
SPEAKER_00: And yeah, it's a very fast workflow as far as getting a project.
43:38
So it was like a two-week window from sending the first change
43:42
as the merge window
43:43
SPEAKER_00: to getting it out into stable kernels to the world.
43:45
SPEAKER_00: That was pretty fast overall for a worldwide project that is everywhere.
43:51
So I think I understand what it's like to be someone who
43:55
contributes to Linux every
43:57
SPEAKER_02: now and then.
43:57
SPEAKER_02: But over time, some people start to contribute more.
43:59
SPEAKER_02: They become more regular contributors.
44:01
SPEAKER_02: And eventually,
44:02
you're one of the few people or one of the few or many people
44:06
who works on Linux
44:07
SPEAKER_02: full-time.
44:08
SPEAKER_02: Are there many people working on it full-time?
44:09
SPEAKER_00: So Linux has almost always been paid to be worked on.
44:14
SPEAKER_00: So I started keeping the numbers back in 2006 or something.
44:18
SPEAKER_00: And at that point in time,
44:19
80% of the people that contributed were being paid to do it
44:22
full time for their
44:23
SPEAKER_00: employer.
44:24
And their employers want people who know how to do Linux
44:27
because they want to solve
44:28
SPEAKER_00: their problems.
44:30
It's much cheaper to pay a few engineers to add a few new
44:32
features than it is to
44:33
SPEAKER_00: write your own operating system.
44:34
SPEAKER_00: That's the beauty of Linux.
44:35
SPEAKER_00: That's why IBM put a bunch of money into it.
44:37
SPEAKER_00: That's why everybody uses it.
44:38
SPEAKER_00: It's a tool for people to get their work done.
44:40
right you want to you want to run your battery you want to run
44:43
your car charger you
44:44
had a little driver for the one device you had you had an
44:47
engineer do that and it's
44:48
good to go and it'll be maintained for forever because we
44:50
maintain it in the
44:51
SPEAKER_00: community it's all good
44:52
SPEAKER_00: So it's cheaper.
44:53
SPEAKER_00: So we've been doing it.
44:54
And the joke used to be you get three changes into the kernel
44:56
and you get a job.
44:57
SPEAKER_00: It's not really a joke.
44:59
SPEAKER_00: As long as they aren't spelling fixes.
45:01
SPEAKER_00: But some people do spelling fixes, which is great.
45:04
SPEAKER_00: We have people that do janitorial work to the kernel.
45:06
They sweep the tree for common problems and they just clean
45:09
stuff up and keep code
45:10
SPEAKER_00: alive and make sure it's fresh,
45:12
SPEAKER_00: proper coding style.
45:13
SPEAKER_00: We have coding style issues.
45:14
SPEAKER_00: We have people just fixing spelling mistakes and comments,
45:17
SPEAKER_00: which is great because you've got to start somewhere.
45:18
SPEAKER_00: In fact,
45:19
spelling mistakes and comments is a great place to start
45:22
because it makes you get
45:23
SPEAKER_00: the workflow down.
45:24
SPEAKER_00: You figure out how to make a patch,
45:25
SPEAKER_00: you figure out how to send an email,
45:26
SPEAKER_00: fix your email client and not send HTML and things like that.
45:30
SPEAKER_00: And you can't use a web email client to send an email.
45:33
SPEAKER_00: It just doesn't work.
45:35
SPEAKER_00: Good email.
45:35
SPEAKER_00: There's lots of really good email tools out there.
45:37
SPEAKER_00: Use them.
45:38
SPEAKER_00: But you're now a full-time kernel maintainer.
45:44
SPEAKER_02: What does your kind of day-to-day or week-to-week look like?
45:47
Because I'm going to assume it's going to be a little bit
45:50
different than most
45:50
SPEAKER_02: developers who,
45:51
SPEAKER_02: you know,
45:52
SPEAKER_02: like write code,
45:52
SPEAKER_02: review code,
45:53
SPEAKER_02: do those kind of things.
45:54
So,
45:55
yeah,
45:55
I mean,
45:55
I've been working for the Linux Foundation for,
45:57
what,
45:58
13 years now.
45:58
SPEAKER_00: Before that,
45:58
SPEAKER_00: I used to work at Novell and Sousa,
46:00
SPEAKER_00: before that IBM,
46:02
SPEAKER_00: and then a little startup,
46:02
SPEAKER_00: all doing Linux stuff all the time.
46:04
SPEAKER_00: And then before that, I did embedded work.
46:05
SPEAKER_00: When I worked for a company,
46:07
SPEAKER_00: you end up working on features that your company wants,
46:10
SPEAKER_00: or reviewing code from the other developers of your company,
46:12
SPEAKER_00: and then sending off changes.
46:13
SPEAKER_00: Or if you're a maintainer,
46:15
SPEAKER_00: a maintainer is,
46:16
SPEAKER_00: and the networking maintainer said this the best,
46:18
SPEAKER_00: we're like editors.
46:19
SPEAKER_00: We used to be a writer.
46:21
SPEAKER_00: All we do is critique other people's stuff now.
46:23
SPEAKER_00: But because
46:23
SPEAKER_00: As we were a writer, we have a little side project.
46:25
SPEAKER_00: So we do have little things that we do dabble in stuff.
46:27
SPEAKER_00: So like I looked,
46:27
SPEAKER_00: I did 80 changes,
46:28
only 80 changes last year because I have a few little things I
46:31
want to do that was
46:32
SPEAKER_00: low.
46:33
But working for the Linux Foundation as a full-time maintainer,
46:36
that's rare.
46:37
SPEAKER_00: I think there's only maybe five people,
46:39
SPEAKER_00: maybe a handful of people that just work on whatever they want to do.
46:43
So Linux Foundation rule is they can't tell me what to do and I
46:46
can't tell them what to do.
46:47
SPEAKER_00: Works out great.
46:48
SPEAKER_00: Me and Linus and Shua Khan,
46:50
SPEAKER_00: we're all fellows there,
46:51
SPEAKER_00: and we work on improving Linux for however we feel like it.
46:55
SPEAKER_00: Me and Linus do a lot of review, a lot of other stuff.
46:58
SPEAKER_00: Linus still contributes.
46:59
He famously rewrote the core locking primitives in Linux a
47:03
couple years ago.
47:04
SPEAKER_00: I had a Microsoft developer say,
47:05
SPEAKER_00: there's no way any of us would be even allowed to do that on Windows.
47:08
SPEAKER_00: You don't test, change core bits and pieces.
47:12
SPEAKER_00: For one of the security features and one of these stable releases,
47:15
Lena said to rewrite the call path from how a user space calls
47:19
into the kernel,
47:20
SPEAKER_00: the core syscall path.
47:22
SPEAKER_00: Nobody really noticed that it got rewritten, but it did.
47:24
SPEAKER_00: And he did it in a stable way, and then it worked like that.
47:27
SPEAKER_00: So we're also part of the kernel security team.
47:30
SPEAKER_00: We get security bug fixes all the time.
47:32
And if they're easy,
47:33
we'll just fix them ourselves and send out the fixes.
47:35
SPEAKER_00: So we do security fixes a lot, as far as that goes.
47:39
SPEAKER_00: So my day-to-day is I read other people's stuff.
47:41
SPEAKER_00: Like I said, I get 1,000 emails a day to do something with.
47:44
SPEAKER_00: You're not extricating.
47:45
SPEAKER_00: No.
47:47
SPEAKER_02: Yeah.
47:47
SPEAKER_02: You get 1,000 emails a day-ish.
47:48
SPEAKER_02: Yes.
47:49
SPEAKER_02: Wow.
47:49
SPEAKER_00: So I don't have to just file off.
47:52
SPEAKER_00: And it's like,
47:52
SPEAKER_00: oh,
47:53
I subscribe to a number of kernel subsystem mailing lists to
47:56
see what's going on.
47:57
SPEAKER_00: Yeah, yeah.
47:58
SPEAKER_00: And I don't have to do something with all of those.
48:00
SPEAKER_02: Yeah, but some of them you need to do something with.
48:02
SPEAKER_00: Yeah, some of these I do need to do something with.
48:04
And some like,
48:04
so I'll,
48:05
so say for USB is one of the subsystems I maintain.
48:07
I show them all off to a mailbox and then once a week I'll go
48:09
through them all and
48:10
SPEAKER_00: say,
48:10
SPEAKER_00: okay,
48:10
SPEAKER_00: let's review all these.
48:12
SPEAKER_00: And so I'll look in my inbox.
48:13
I'll have 200 USB emails to patches to go through and other
48:17
people review them and
48:18
SPEAKER_00: other stuff like that.
48:19
SPEAKER_00: And okay, this maintainer said this was good, not good and whatnot.
48:22
And I apply them to my trees,
48:23
see if they build,
48:24
if they failed,
48:25
I'll report those and whatnot.
48:27
SPEAKER_02: You know,
48:27
SPEAKER_02: what you're doing reminds me a little bit of when I used to work at Uber,
48:31
SPEAKER_02: we had this concept of RFCs,
48:33
SPEAKER_02: which I think got inspired by the RFC process.
48:36
So people would just send off a document of here's what I'm
48:40
planning to do.
48:41
And there would be mailing lists for like back and mobile,
48:44
different parts.
48:45
SPEAKER_02: And I noticed after a while that the more tenured engineers and the more
48:49
experienced engineers were spending increasingly more of their
48:51
time reading through
48:53
SPEAKER_02: these things,
48:53
SPEAKER_02: critiquing,
48:54
SPEAKER_02: giving feedback,
48:55
SPEAKER_02: giving pointers,
48:56
SPEAKER_02: connecting the dots.
48:57
Like it just hit me when one of the first mobile engineers at
49:01
Uber was telling me
49:02
that he has one day blocked out per day just to go through all
49:06
of these things,
49:07
SPEAKER_02: which again,
49:07
SPEAKER_02: it wasn't kind of part of his role,
49:09
SPEAKER_02: but he felt responsible.
49:10
SPEAKER_02: He had all the context.
49:11
He actually helped so many people avoid certain things just by
49:14
pointing it out.
49:15
SPEAKER_02: It's the same thing, or something similar happened here.
49:18
SPEAKER_00: It's the same.
49:18
SPEAKER_00: But we also,
49:19
SPEAKER_00: the different part of this,
49:21
SPEAKER_00: and I'll call this out,
49:22
SPEAKER_00: we don't have grand proposals sent to the kernel list.
49:25
SPEAKER_00: We don't say, hey, wouldn't it be great if you did this?
49:28
SPEAKER_00: I don't want to see that.
49:29
SPEAKER_00: I want to see code that works.
49:32
SPEAKER_00: I love it.
49:32
SPEAKER_00: As proof,
49:33
SPEAKER_00: then,
49:33
SPEAKER_00: code that works matters is because you've taken the time,
49:37
SPEAKER_00: you've proved that this can be done.
49:39
Now,
49:39
not necessarily that it's done right or done the best way,
49:42
but it could be done.
49:44
SPEAKER_00: And that's now you have the skin in the game,
49:45
SPEAKER_00: and now I'm willing to work with you,
49:47
SPEAKER_00: and let's go on that.
49:47
SPEAKER_00: People do send off RFCs of patches.
49:50
SPEAKER_00: If it's an area I care about, I'll look at it.
49:53
SPEAKER_00: Sometimes you can get away with this.
49:54
SPEAKER_00: This is a fun trick with maintainers.
49:57
If you send me a patch set that solves your problem in such a
50:00
way that it's
50:00
SPEAKER_00: horrible,
50:01
SPEAKER_00: but I hate it so much,
50:02
SPEAKER_00: that I'll rewrite it myself.
50:04
SPEAKER_00: Because they'll be like,
50:04
I can't say no because it solved the problem and I want to
50:07
solve your problem.
50:08
SPEAKER_00: But if I don't say no, then I have to take that.
50:10
SPEAKER_00: So you can do that like once a year to a maintainer.
50:13
I sense that you're eliminating busy work because I've seen at
50:16
different companies
50:17
SPEAKER_02: when you have the proposal process,
50:19
SPEAKER_02: again,
50:19
SPEAKER_02: a lot of companies for good for,
50:20
SPEAKER_02: you know,
50:21
SPEAKER_02: it sounds logical.
50:22
SPEAKER_02: Instead of starting the work,
50:23
SPEAKER_02: instead of investing time,
50:25
SPEAKER_02: maybe we would all save time by do a little planning up front.
50:28
SPEAKER_02: Right.
50:28
But then every now and then what happens is you get into this
50:31
never ending planning.
50:32
Nothing happens until either the project is abandoned or
50:35
someone just sits down,
50:36
SPEAKER_02: writes some code,
50:38
SPEAKER_02: and kind of just cuts all the discussions are done because now it works.
50:42
SPEAKER_00: Yeah, well, you have to prove that it can work.
50:44
SPEAKER_00: And so inside companies,
50:46
SPEAKER_00: I'll say,
50:46
SPEAKER_00: we did,
50:47
SPEAKER_00: like when I worked for companies,
50:48
SPEAKER_00: IBM,
50:48
SPEAKER_00: it's like we had planning.
50:49
Okay,
50:49
we need to implement this feature to match this parity with
50:51
this old version of Unix.
50:53
SPEAKER_00: How are we going to do that?
50:53
SPEAKER_00: Let's figure out how to do this.
50:55
SPEAKER_00: Is this going to work?
50:55
SPEAKER_00: Yada, yada, yada.
50:56
SPEAKER_00: And we have planning and things like that.
50:57
One of the fun things is when you're dealing with open source,
51:00
and this happened at IBM,
51:03
SPEAKER_00: engineer over here was tasked with fix this problem.
51:06
SPEAKER_00: Great.
51:07
He came up with a solution,
51:08
submitted all the changes upstream,
51:09
lots and lots of discussion.
51:10
SPEAKER_00: Turns out his solution was not very good.
51:13
Somebody else saw that it was a problem,
51:15
rewrote it,
51:16
submitted it,
51:17
and got it accepted.
51:18
SPEAKER_00: But it wasn't the original engineer's work.
51:21
And so the end of the year came,
51:23
it was like,
51:23
how is this person going to be reviewed?
51:24
SPEAKER_00: And we're like, he caused the feature to get done.
51:28
SPEAKER_00: It wasn't that his code made it in,
51:30
SPEAKER_00: but he influenced the community and made,
51:33
SPEAKER_00: the goal was you wanted to see Linux support this,
51:34
SPEAKER_00: right?
51:35
SPEAKER_00: Linux supports this now.
51:36
And it was a change in mentality of how management had to treat
51:39
engineers.
51:41
SPEAKER_00: And also the same thing with who owns the code.
51:43
We had people come in and end up becoming maintainers of
51:47
certain subsystems.
51:48
SPEAKER_00: And that's great.
51:49
SPEAKER_00: And they were maintaining this part of the kernel.
51:51
SPEAKER_00: And then they were reassigned to do something else within the company.
51:53
SPEAKER_00: So that's great.
51:54
SPEAKER_00: But you're going to still have to give them time to do that other thing.
51:56
SPEAKER_00: It's like, no, no, we'll reassign it to somebody else.
51:58
SPEAKER_00: It's like, no, no, no, no.
51:59
SPEAKER_00: The community gave that to him.
52:01
SPEAKER_00: It follows him.
52:01
SPEAKER_00: If he goes to a different company, it follows him.
52:03
SPEAKER_00: If he goes to a different part of the company, it follows him.
52:06
SPEAKER_00: And that's actually why Linux is so good, I think.
52:09
SPEAKER_00: When you work at a big company,
52:10
SPEAKER_00: you're forced to work on new things every couple of years,
52:12
SPEAKER_00: right?
52:13
SPEAKER_00: And that's part of moving up in a company.
52:14
SPEAKER_00: You get different tasks when not.
52:15
Famously,
52:16
Windows has had like eight,
52:18
no,
52:18
five different teams work on their USB stack.
52:20
SPEAKER_00: Linux has had one team working on their USB stack for 20 years.
52:24
SPEAKER_00: And then we know this stuff and we have this development in depth there.
52:27
SPEAKER_02: We just keep coming back to like,
52:29
SPEAKER_02: I was kind of expecting a little bit of a discussion about,
52:33
SPEAKER_02: I came in here just using Linux or indirectly using Linux a lot,
52:37
SPEAKER_02: but not knowing of the depths.
52:38
And I kind of thought that we would talk a lot about the tech,
52:42
the processes.
52:43
SPEAKER_02: And every time we come back to the people,
52:46
SPEAKER_02: the trust,
52:47
SPEAKER_02: I wanted to ask why you think Linux has won so big that it's everywhere.
52:54
SPEAKER_02: But I'm starting to get the answer.
52:57
SPEAKER_02: It's true.
52:58
SPEAKER_02: Because I was thinking, why Linux?
53:01
SPEAKER_02: Why not a commercial?
53:02
SPEAKER_02: If I naively asked myself before this conversation, we have two teams.
53:08
SPEAKER_02: One is commercially funded.
53:09
SPEAKER_02: They're selling their software.
53:10
SPEAKER_02: They're paying the developers really well.
53:13
SPEAKER_02: And then the other one is giving it away for free.
53:15
SPEAKER_02: They figured out a model where people are still paid.
53:17
SPEAKER_02: But it's open source.
53:19
SPEAKER_02: Anyone can use it.
53:20
SPEAKER_02: Anyone can contribute.
53:21
SPEAKER_02: Which one will win the long term?
53:23
I naively would have said maybe the commercial one because
53:26
they're incentivized.
53:27
They're going to create all these professional things where
53:30
here it's more intrinsic value.
53:32
SPEAKER_02: It is intrinsic, but...
53:34
SPEAKER_00: So Linux has been contributed to by companies in their own interest.
53:39
SPEAKER_00: So it turns out everybody contributes in a selfish way.
53:41
SPEAKER_00: We want to solve my selfish problem.
53:43
SPEAKER_00: But it turns out everybody has the same problems.
53:45
SPEAKER_00: So your problem being solved is the same problem as their problem.
53:49
SPEAKER_00: Famously, we had this when it came down to embedded.
53:51
SPEAKER_00: So embedded happened.
53:53
They came up saying,
53:53
we need to change Linux to make it work better on batteries.
53:56
SPEAKER_00: You know, power is really important.
53:57
SPEAKER_00: Power is very, very important.
53:59
SPEAKER_00: A lot more efficient.
54:00
So we wanted to do this when Linux was first getting into
54:02
embedded and like,
54:03
SPEAKER_00: we need to make this very efficient.
54:04
SPEAKER_00: And we're like, great.
54:05
SPEAKER_00: That's a wonderful little solution.
54:06
SPEAKER_00: Make it work for everybody.
54:08
SPEAKER_00: Like, no, no, no.
54:09
SPEAKER_00: We just care about embedded over here.
54:10
SPEAKER_00: That's the only person that's going to care about power.
54:11
SPEAKER_00: It's like, no, you really just make a generic.
54:13
SPEAKER_00: It'll all be good.
54:14
SPEAKER_00: Turns out.
54:15
data centers save billions of dollars in money because of power
54:18
management.
54:19
SPEAKER_02: And it turns out everybody, so the mainframes.
54:21
SPEAKER_02: If it's more efficient on a mobile phone,
54:23
SPEAKER_02: suddenly it's a good candidate for it to be a mobile OS.
54:26
SPEAKER_00: Yes, it works for everything.
54:28
SPEAKER_00: Same thing for multiprocessors.
54:30
SPEAKER_00: Multiprocessor came out, there's two.
54:32
SPEAKER_00: We have two processors in the big data center.
54:33
SPEAKER_00: Who's going to care about that?
54:34
SPEAKER_00: In your pocket now, you have 16 processors.
54:36
SPEAKER_00: It just works for everybody now, right?
54:38
SPEAKER_00: Data trends shrink, go different places.
54:41
SPEAKER_00: But because we solved it in a generic way,
54:45
We forced you to solve it in a generic way,
54:47
but you contributed in a selfish manner.
54:49
SPEAKER_00: And that's a good way that IBM knew they could put money into it,
54:53
SPEAKER_00: hire developers,
54:54
SPEAKER_00: and get the money back.
54:56
SPEAKER_00: So it was cheaper for them in the long run to do that.
54:59
SPEAKER_00: And they make money selling support and selling hardware.
55:01
SPEAKER_00: Red Hat makes money selling support.
55:03
SPEAKER_00: And it's like that.
55:04
SPEAKER_00: Intel makes money selling chips.
55:06
And that's who contributes to Linux is the people who they want
55:09
to sell a different product.
55:11
SPEAKER_02: Now, one other thing that's interesting about efficiency,
55:14
SPEAKER_02: We have 4,000 developers contributing.
55:17
SPEAKER_02: Some of them only contribute one chain.
55:18
SPEAKER_02: Some of them contribute a bunch.
55:21
SPEAKER_02: 300 to 500 companies per year.
55:22
SPEAKER_02: We're talking about per year.
55:23
SPEAKER_02: If you told me this is inside a kind of commercial company,
55:28
SPEAKER_02: a tech company,
55:29
SPEAKER_02: I would assume that in order to make this work,
55:31
SPEAKER_02: oh,
55:32
SPEAKER_02: for 4,000 developers,
55:33
SPEAKER_02: we probably need to hire 400 PMs.
55:36
SPEAKER_02: We'll have about, for every 50 developers, we'll have about 80 TPMs.
55:41
SPEAKER_02: This is how it would run.
55:42
SPEAKER_00: It's very true.
55:43
SPEAKER_00: No, I know.
55:44
SPEAKER_00: I've been there.
55:44
SPEAKER_02: You've been there, but I only come from here.
55:47
SPEAKER_02: Now,
55:47
SPEAKER_02: one thing you told me,
55:48
because I was asking you how many project or very technical
55:52
project managers you
55:53
SPEAKER_02: have,
55:53
SPEAKER_02: and you said zero.
55:55
SPEAKER_02: How?
55:56
SPEAKER_00: Well,
55:56
SPEAKER_00: so in a way,
55:57
the project managers already happened on the back end before
56:00
the patches got to us.
56:01
SPEAKER_00: So at a company, say, IBM, I want to solve this problem.
56:04
SPEAKER_00: They've said, how do we solve this problem?
56:06
SPEAKER_00: Let's put this task.
56:07
SPEAKER_00: Let's figure this out.
56:08
SPEAKER_00: And then the patches come out to us.
56:10
SPEAKER_00: So we don't see that.
56:11
SPEAKER_00: So we just see the feature when it lands on us.
56:13
SPEAKER_00: That's fair.
56:13
so they're there forking for the individual companies to get
56:17
their thing sometimes
56:18
sometimes they're not sometimes they're just developers are
56:20
spitting things out and
56:21
like this person who needed to get a new device id it saves
56:24
company time and money
56:24
if they contribute their changes upstream than to keep it as a
56:27
fork because they
56:27
have to keep maintaining that fork so wise companies have
56:30
realized let our
56:31
developers work upstream do what they need to do there with
56:35
limited project
56:35
management and it just works out better and again we're only
56:39
taking things when
56:40
SPEAKER_00: they're ready
56:42
SPEAKER_00: We're not having to track.
56:43
SPEAKER_00: We do have tools.
56:44
SPEAKER_00: We set everything through email.
56:45
We have tools like the networking subsystem has a web page you
56:48
can go to see what
56:48
SPEAKER_00: the status of your patch is,
56:50
SPEAKER_00: if it's past all the CI,
56:52
SPEAKER_00: if it's been reviewed by the maintainer,
56:53
SPEAKER_00: and things like that.
56:53
So we have a bunch of automatic tools based on top of email
56:57
that'll help you out.
56:58
And those project managers can go look at those if they want to
57:00
wonder what the
57:01
SPEAKER_00: status of their employees' patches were at,
57:03
SPEAKER_00: things like that.
57:03
SPEAKER_00: But yeah, it's a different model.
57:05
SPEAKER_00: But it's not like they're hidden behind the solution for that company.
57:10
SPEAKER_02: It's fair, and I think it's also good to remind that that's the case.
57:13
But I feel Linux still figured out a way to just focus on just
57:18
rootless efficiency
57:19
SPEAKER_02: with automation,
57:20
SPEAKER_02: with focusing on the work when it's done.
57:23
SPEAKER_02: So as you said, all these things do happen, but they happened before.
57:26
SPEAKER_02: And then this part of the process will just be more efficient by design.
57:31
SPEAKER_00: Yeah,
57:31
SPEAKER_00: and we also,
57:31
SPEAKER_00: but once a year we get together,
57:33
SPEAKER_00: the core maintainers,
57:34
and we talk about not technical things because we can't have
57:36
enough technical
57:37
SPEAKER_00: people in the room for a topic.
57:38
SPEAKER_00: We talk about process.
57:40
SPEAKER_00: Is our process working?
57:41
SPEAKER_00: Is it not working?
57:41
SPEAKER_00: And we refine it and say, oh, maybe we need to do this a little better.
57:44
SPEAKER_00: Oh, wouldn't it be nicer to do this?
57:45
SPEAKER_00: Hey, we need more testing over here.
57:47
SPEAKER_00: Hey, can we do this type of stuff?
57:48
SPEAKER_00: So we do.
57:48
SPEAKER_00: We talk about our process all the time.
57:50
SPEAKER_00: Famously,
57:50
the lead up to that meeting is another public mailing list that
57:54
we all talk about
57:55
SPEAKER_00: processes.
57:56
SPEAKER_00: And that process,
57:57
SPEAKER_00: that once a year bike shedding of our process in public,
58:00
SPEAKER_00: it helps shake out a lot of things and work out.
58:04
SPEAKER_00: And there are problems.
58:05
SPEAKER_00: I'm not saying this development model is perfect.
58:07
SPEAKER_00: It works really well.
58:08
SPEAKER_00: One thing that's odd about Linux is that we keep going as fast as we are.
58:12
SPEAKER_00: We're running at nine to 10 changes an hour.
58:14
SPEAKER_00: In the stable kernels, we're running 30 changes a day.
58:17
SPEAKER_00: 30 to 40 changes a day.
58:18
SPEAKER_00: 10 CBEs a day.
58:21
SPEAKER_00: A bug at our level is a CBE, almost.
58:23
SPEAKER_02: Yeah.
58:23
SPEAKER_02: So CBEs are the critical...
58:25
SPEAKER_00: It's a security bug.
58:26
SPEAKER_00: It's a vulnerability.
58:28
SPEAKER_00: That could be as stupid as a memory leak somewhere,
58:31
SPEAKER_00: or I rebooted the machine,
58:33
SPEAKER_00: or I took over and got permission.
58:35
SPEAKER_00: I don't know.
58:36
When I create a CVE,
58:37
I don't know how you use Linux,
58:39
so I can't tell the severity of it.
58:41
SPEAKER_00: But I can just say, here's a bug.
58:42
SPEAKER_00: You should look at this.
58:44
SPEAKER_00: So we're responsible for that.
58:45
SPEAKER_00: So we're running at a huge rate of change.
58:47
SPEAKER_00: Most large software projects have a huge ramp up,
58:49
and then they plateau with developers and rate of change and
58:52
whatnot because
58:52
SPEAKER_00: they've solved the problem.
58:53
SPEAKER_00: Linux has never solved the problem.
58:55
And I had a manager at IBM,
58:57
every year he'd come to me and said,
58:58
hey,
58:58
is Linux done yet?
58:59
SPEAKER_00: I was like, no.
59:00
SPEAKER_00: It took me 10 years to finally come up with the answer of,
59:03
SPEAKER_00: it'll be done when you stop making new hardware.
59:05
And when they stop making new hardware or having different work
59:08
classes,
59:08
then we'll stop.
59:09
But we're one of the few projects that keep having to add new
59:12
features because of new hardware.
59:14
We're not doing it just because,
59:15
I mean,
59:15
this has been working for all of us for 20 years.
59:18
We're doing it to support new hardware,
59:19
to support new use models,
59:21
to support things.
59:22
SPEAKER_00: We don't add things for fun, generally.
59:23
SPEAKER_00: We add it to solve a problem that somebody had.
59:26
SPEAKER_02: Most of Linux is written using C or C++, right?
59:29
SPEAKER_02: No C++.
59:31
Just C.
59:32
And I guess for some hardware drivers,
59:34
is there assembly ever involved or no?
59:36
SPEAKER_00: No,
59:37
assembly will drop down into the early boot of a processor and
59:40
then some core
59:41
functionality like locking and that drivers or other people
59:44
will call will go down
59:47
like string functions and whatnot will go down to good assembly
59:50
language that's
59:50
SPEAKER_00: tuned for the different processors.
59:52
SPEAKER_00: Also, when you boot,
59:53
SPEAKER_00: Linux looks at the processor you're running on,
59:56
patches itself to figure out the best functions that those
1:00:00
assembly would work,
1:00:01
SPEAKER_00: and then it continues on moving,
1:00:03
SPEAKER_00: which is crazy.
1:00:03
SPEAKER_00: It patches itself at boot time.
1:00:04
SPEAKER_00: So hold on, but some of that is assembly?
1:00:07
SPEAKER_00: Some of that's assembly in the very beginning,
1:00:09
SPEAKER_00: and some of those low-level functions,
1:00:10
SPEAKER_00: but drivers don't ever touch.
1:00:12
SPEAKER_00: OK,
1:00:12
SPEAKER_02: so basically from a Linux contributor,
1:00:15
SPEAKER_02: it's all C.
1:00:15
SPEAKER_02: Now,
1:00:17
one thing that actually the way we started talking is there is
1:00:20
a proposal to
1:00:22
SPEAKER_02: introduce Rust because it's just more memory safe.
1:00:24
SPEAKER_02: It's also language growing in popularity.
1:00:26
SPEAKER_02: And some people would like to do more Rust development.
1:00:29
SPEAKER_02: What is your take on this?
1:00:32
SPEAKER_02: Do you think Linux at some point might support Rust?
1:00:36
SPEAKER_02: And what is your thinking of doing things outside of C?
1:00:41
SPEAKER_00: So we have 25,000 lines of Rust in the kernel already.
1:00:44
SPEAKER_00: Oh.
1:00:45
SPEAKER_02: We do.
1:00:46
SPEAKER_02: OK, awesome.
1:00:47
SPEAKER_00: Yeah.
1:00:47
SPEAKER_00: So most of that is just bindings.
1:00:49
SPEAKER_00: There's no real functionality.
1:00:51
SPEAKER_00: In the latest release, if the kernel crashes, it'll put up a QR code.
1:00:55
SPEAKER_00: You can take a picture of it to get the crash jump.
1:00:57
SPEAKER_00: That code was written in Rust.
1:00:58
SPEAKER_00: Oh, nice.
1:00:59
SPEAKER_00: That's in Rust.
1:01:00
SPEAKER_00: So the Rust for Linux developers has been working for a long time.
1:01:04
A couple years ago,
1:01:04
they came to us and said,
1:01:05
we think we're ready to do this.
1:01:07
SPEAKER_00: Do you want it?
1:01:08
SPEAKER_00: And we said, yeah, let's try this experiment.
1:01:10
SPEAKER_00: You're willing to do the work?
1:01:12
SPEAKER_00: Who am I going to tell no to?
1:01:15
SPEAKER_00: Classic Linux, right?
1:01:16
SPEAKER_00: Yeah.
1:01:18
SPEAKER_00: Now,
1:01:18
the problem with Linux and Rust is it would be easier to write
1:01:22
a core piece of
1:01:23
SPEAKER_00: Linux and Rust than it would be to write a driver.
1:01:25
SPEAKER_00: A driver is consumed from everywhere in the kernel.
1:01:28
SPEAKER_00: So you want to talk locking.
1:01:30
SPEAKER_00: You want to talk input and output.
1:01:31
You want to talk to the driver model,
1:01:33
talk to the USB port,
1:01:34
all this stuff.
1:01:35
Drivers can be really tiny because they take resources from the
1:01:38
rest of the kernel.
1:01:39
SPEAKER_00: In Rust, you need to have a binding between the C code and the Rust code.
1:01:43
SPEAKER_00: There's an intermediate layer.
1:01:46
The kernel in C has these very opinionated model ideas of how
1:01:49
it handles objects
1:01:50
SPEAKER_00: and how it does memory and how it has its memory model.
1:01:53
SPEAKER_00: Rust has its very opinionated model of how it does this type.
1:01:56
SPEAKER_00: Same idea.
1:01:57
SPEAKER_00: This meshing is tough.
1:02:00
SPEAKER_00: This meshing is also the most crazy complex REST code you've ever seen.
1:02:04
SPEAKER_00: So from a new REST developer like me,
1:02:06
SPEAKER_00: I can barely read the bindings,
1:02:08
SPEAKER_00: but I trust other people are doing it.
1:02:09
SPEAKER_00: So yes,
1:02:10
the trick is we now need to write a binding for every different
1:02:14
part of the kernel
1:02:15
SPEAKER_00: in order to write a REST code.
1:02:17
SPEAKER_00: a Rust driver.
1:02:17
SPEAKER_00: If you want to do the QR generator, that's simple.
1:02:19
SPEAKER_00: That was one function.
1:02:22
So over the year,
1:02:23
the past couple years,
1:02:24
people have been writing bindings to try and do things.
1:02:27
SPEAKER_00: We've had a bunch of example drivers,
1:02:29
SPEAKER_00: like a new disk driver,
1:02:30
SPEAKER_00: let's write a driver in C versus Rust.
1:02:32
It turns out there are still some performance issues with Rust
1:02:34
code versus C code,
1:02:36
SPEAKER_00: because we can do some tricks and see that they can't do yet in Rust.
1:02:38
SPEAKER_00: And the tooling and the Rust developers are doing it.
1:02:41
The core Rust developers,
1:02:42
the language,
1:02:43
some of them are Linux kernel developers.
1:02:44
SPEAKER_00: They've always wanted Rust to be working for Linux.
1:02:47
SPEAKER_00: The Rust model is good.
1:02:49
SPEAKER_00: Memory safety at our level does not mean that you can't crash the kernel.
1:02:53
SPEAKER_00: You can still overwrite things.
1:02:55
Memory safety in Rust just means the memory that you pass
1:02:58
around you think you have
1:02:59
SPEAKER_00: an ownership of or it isn't an ownership of.
1:03:02
SPEAKER_00: And when things go out of scope, they'll get cleaned up properly.
1:03:06
SPEAKER_00: So I've seen every single kernel bug for the past 18 years.
1:03:09
SPEAKER_00: Half of them will be fixed with Rust.
1:03:11
SPEAKER_00: It's just going to be fixed with Rust.
1:03:13
SPEAKER_00: It's the stupid one-off bugs.
1:03:16
SPEAKER_00: It's the, oops, I overwrote an array, and I didn't realize it by one.
1:03:19
SPEAKER_00: Oops, I forgot to clean up this error path.
1:03:22
SPEAKER_00: I forgot to unlock this lock.
1:03:24
SPEAKER_00: It's stupid little things like that.
1:03:26
SPEAKER_00: There's logic bugs, of course.
1:03:28
SPEAKER_00: You can write logic bugs in Rust.
1:03:30
SPEAKER_02: You always have those.
1:03:31
SPEAKER_00: Right.
1:03:31
SPEAKER_00: So but famously,
1:03:32
SPEAKER_00: the code,
1:03:32
SPEAKER_00: the QR code for that in Rust that made the QR,
1:03:36
SPEAKER_00: C passed into the Rust code a pointer to a buffer and the buffer size.
1:03:40
SPEAKER_00: The Rust code forgot to look at how big the buffer was.
1:03:42
SPEAKER_00: And it scribbled right over memory.
1:03:43
SPEAKER_00: So you can write memory unsafe code in Rust.
1:03:46
SPEAKER_00: Yeah.
1:03:47
SPEAKER_00: And you can crash things in Rust.
1:03:48
So memory safety here means it's the safety of object life
1:03:53
cycles and things like that.
1:03:54
SPEAKER_00: It doesn't mean it's going to remove all bugs.
1:03:56
SPEAKER_00: It's not a golden bullet or anything like, silver bullet.
1:03:59
SPEAKER_00: But I think,
1:04:01
SPEAKER_00: yes,
1:04:01
I think Rust needs to come in because it should be easier to
1:04:05
write drivers in this
1:04:06
SPEAKER_00: stuff.
1:04:07
We have a lot of issues with lifetime rules of when you yank
1:04:11
out a device,
1:04:12
devices are dynamic.
1:04:13
And dealing with these reference counting of things like that
1:04:16
is very tricky to get right.
1:04:17
There's parts in the kernel we still do not have it right,
1:04:19
and we know we don't have it right.
1:04:21
SPEAKER_00: Rust is forcing us to actually document our C code better.
1:04:26
SPEAKER_00: And it's cleaning up.
1:04:26
So if Rust disappeared tomorrow,
1:04:28
I've had to clean up code in the driver core.
1:04:30
SPEAKER_00: It's like,
1:04:30
SPEAKER_00: oh,
1:04:30
SPEAKER_00: yeah,
1:04:31
I guess we can do things better and safer in the C code in
1:04:34
order to make Rust
1:04:34
SPEAKER_00: easier.
1:04:35
SPEAKER_00: And we have.
1:04:36
And so it's making us rethink how we do a lot of our existing
1:04:39
code in the kernel.
1:04:41
SPEAKER_00: To be fair, a lot of core kernel people are pretty resistant to that.
1:04:45
SPEAKER_00: They don't like change, don't like different languages.
1:04:48
SPEAKER_00: One core kernel developer said,
1:04:49
I don't like working with a project that has multiple languages
1:04:52
in it just because
1:04:53
SPEAKER_00: it's tricky.
1:04:54
SPEAKER_00: And they are free to do that.
1:04:56
SPEAKER_00: They're not stepping on anybody's toes.
1:04:57
A lot of it's miscommunication and a lot of it comes down to
1:05:00
people again.
1:05:00
SPEAKER_00: Famously,
1:05:02
SPEAKER_00: in this binding,
1:05:04
SPEAKER_00: I wrote the driver core many,
1:05:05
SPEAKER_00: many years ago of how drivers work in the kernel.
1:05:08
SPEAKER_00: There had to be a binding for that in Rust.
1:05:12
SPEAKER_00: This code I saw, I said, this is horrible.
1:05:14
SPEAKER_00: This isn't going to work at all.
1:05:15
SPEAKER_00: It's miserable.
1:05:16
SPEAKER_00: I went and actually met with the developers.
1:05:18
SPEAKER_00: And there's a Rust Linux conference.
1:05:19
SPEAKER_00: We sat down.
1:05:20
SPEAKER_00: I think they gave a whole presentation just for me.
1:05:23
SPEAKER_00: Turns out I was wrong and they were wrong.
1:05:25
SPEAKER_00: We both were wrong.
1:05:26
And they were doing crazy things like they had a thousand lines
1:05:29
of Rust code that I
1:05:30
SPEAKER_00: do in two lines of C code.
1:05:32
SPEAKER_00: I'm like, well, why?
1:05:33
SPEAKER_00: They're like, well, we didn't want to change the C code.
1:05:34
SPEAKER_00: I'm like,
1:05:35
we can change the C code because I just did that because it was
1:05:37
easy in C.
1:05:38
SPEAKER_00: But if I change that,
1:05:39
SPEAKER_00: you get rid of a thousand lines of rust.
1:05:40
SPEAKER_00: Let's do that.
1:05:41
SPEAKER_00: And again,
1:05:41
SPEAKER_00: it comes down to,
1:05:42
SPEAKER_00: okay,
1:05:42
SPEAKER_00: understanding what your problems are,
1:05:44
SPEAKER_00: understanding what my problems are,
1:05:45
SPEAKER_00: and let's work together.
1:05:46
And now we have bindings in the kernel that you can actually
1:05:48
write some drivers with.
1:05:50
And the Red Hat developers are starting to write the new NVIDIA
1:05:52
GPU drivers in Rust.
1:05:54
SPEAKER_00: And they're starting to put proposals out there.
1:05:57
SPEAKER_00: The Apple GPU drivers for the Apple MacBooks are written in Rust.
1:06:00
Those patches are not merged,
1:06:02
but they're written in Rust and Prove on a fork.
1:06:05
SPEAKER_00: That works great.
1:06:06
There's a whole bunch of crazy object lifecycle issues with
1:06:09
graphics drivers,
1:06:10
SPEAKER_00: and Rust makes it a lot easier for them to do.
1:06:12
SPEAKER_00: I think you'll see a lot more of the simple,
1:06:15
SPEAKER_00: stupid drivers for hardware devices being written in Rust.
1:06:18
SPEAKER_00: Because all they want to do is read and write to some random memory bits.
1:06:21
SPEAKER_00: And it's really easy to do that in Rust,
1:06:22
SPEAKER_00: and you can do it in actually less code than you can do it in C code.
1:06:25
SPEAKER_00: And I think that's good.
1:06:26
We now have the infrastructure in there,
1:06:28
so I think we've hit the tipping point.
1:06:29
SPEAKER_00: We'll start seeing new stuff in there.
1:06:31
SPEAKER_00: And we need to do that.
1:06:31
SPEAKER_00: I mean,
1:06:32
there's mandates from governments that you can't use
1:06:35
memory-unsafe languages like C
1:06:36
SPEAKER_00: in products.
1:06:37
SPEAKER_00: And if I want to see Linux succeed, which I do,
1:06:40
SPEAKER_00: We're going to have to change.
1:06:41
And I can say going forward,
1:06:43
if you want to write in Rust,
1:06:44
you can write in Rust.
1:06:45
SPEAKER_00: Now, that being said, we still have 40 million lines of C code.
1:06:47
SPEAKER_00: So we have some very,
1:06:48
very good developers out there working on mitigating the
1:06:50
problems we have in C.
1:06:52
SPEAKER_00: We now have bounds checking for our stuff.
1:06:53
SPEAKER_00: We now have other,
1:06:54
we call them C belts and airbags that protect your C code from
1:06:59
doing stupid things.
1:07:00
And we're working with the compiler authors to add new
1:07:02
extensions to C and make
1:07:04
things safer for the C code because we want to protect the code
1:07:07
that we have today
1:07:08
SPEAKER_00: because you're not going to rewrite code in Rust.
1:07:09
SPEAKER_00: Don't worry about that.
1:07:10
Google famously published something recently saying over the
1:07:13
past couple of years,
1:07:13
we've written our new code in Rust and we got overwhelmingly
1:07:17
more secure because we
1:07:18
SPEAKER_00: didn't touch the old code and bugs degrade over time.
1:07:22
SPEAKER_00: There's still going to be bugs in the older stuff,
1:07:23
SPEAKER_00: but most bugs happen in your new code,
1:07:25
SPEAKER_00: not in your old code.
1:07:26
SPEAKER_02: That's awesome.
1:07:27
SPEAKER_02: I'm sensing you're excited about Rustin.
1:07:30
SPEAKER_02: It's also just nice to see the evolution.
1:07:32
SPEAKER_00: Yeah, it's evolution and see what happens.
1:07:34
SPEAKER_00: And if it fails tomorrow, we can rip it out.
1:07:36
SPEAKER_00: But we have developers willing to do this work for us.
1:07:38
SPEAKER_00: It's not intruding on other people's stuff.
1:07:40
SPEAKER_02: Well, then, I think
1:07:41
SPEAKER_02: It does go back to what you said earlier.
1:07:45
SPEAKER_02: I understand that a big part of Linux is show the work.
1:07:49
SPEAKER_02: If it works,
1:07:49
SPEAKER_02: and same thing,
1:07:50
SPEAKER_02: it sounds like that's how Rust started,
1:07:52
SPEAKER_02: and it's also how it's progressing.
1:07:53
SPEAKER_02: People are showing that it works.
1:07:55
SPEAKER_02: They're proving that it solves their problem.
1:07:59
SPEAKER_02: It maybe even works better for them, and then step by step.
1:08:01
SPEAKER_00: Yeah, like people are like, well, why not Zig or Hair?
1:08:03
SPEAKER_00: Those are other good languages.
1:08:04
SPEAKER_00: I'm like, that's great, but nobody's proposed that.
1:08:06
SPEAKER_00: Yeah.
1:08:07
SPEAKER_00: So yeah, they want to do that.
1:08:08
And I think those developers who work on those languages don't
1:08:11
care about Linux,
1:08:11
which is fine.
1:08:12
SPEAKER_00: They don't have to.
1:08:13
SPEAKER_02: So looking ahead,
1:08:14
SPEAKER_02: outside of Rust,
1:08:17
what are other things that you're kind of excited about that's
1:08:21
coming in Linux,
1:08:23
SPEAKER_02: either projects,
1:08:24
SPEAKER_02: changes?
1:08:25
SPEAKER_02: I don't know if... We haven't said LLMs except for once here.
1:08:29
I don't know if that,
1:08:31
for example,
1:08:31
will LLMs have any impact on how development is done?
1:08:36
SPEAKER_00: No, there's not.
1:08:38
I mean,
1:08:38
they're all trained on Linux kernel code,
1:08:40
so you write out another driver.
1:08:41
SPEAKER_00: LLMs are great for writing boilerplate code and things like that.
1:08:45
SPEAKER_00: In Linux drivers,
1:08:46
you don't have much boilerplate code because we've stemmed that
1:08:49
down into the core
1:08:50
SPEAKER_00: and made that work better.
1:08:52
LLMs are used to find bugs and find the bugs fixes to match
1:08:57
that we should be taking.
1:08:59
SPEAKER_00: But again, we've published papers on that for eight years.
1:09:03
SPEAKER_00: There's been lots of research on that.
1:09:05
SPEAKER_00: So we've been using that for a while.
1:09:07
SPEAKER_00: I mean, LLM is just applied statistics, right?
1:09:09
SPEAKER_00: So it's just pattern version.
1:09:10
SPEAKER_00: So code for us at this level, it doesn't matter that much.
1:09:14
SPEAKER_00: So no.
1:09:15
And then as far as I don't know what's coming tomorrow because
1:09:18
I just see what
1:09:18
SPEAKER_00: people send to me.
1:09:19
SPEAKER_00: So we don't have a plan.
1:09:20
I mean,
1:09:20
we always joke,
1:09:21
you know,
1:09:22
Linux is evolution,
1:09:23
not intelligent design.
1:09:25
SPEAKER_00: It's just whatever shows up, right?
1:09:27
SPEAKER_00: Because you're solving your problem,
1:09:28
and we'll figure out how to fit it in there with everybody
1:09:30
else's stuff and make
1:09:31
SPEAKER_00: sure it doesn't work out.
1:09:33
SPEAKER_00: People are working on new features.
1:09:34
SPEAKER_00: I mean, Linux, people are like, oh, it's an old model.
1:09:37
SPEAKER_00: It's the old Unix model.
1:09:38
It's like,
1:09:38
yeah,
1:09:38
we can run code from 20 and 30 and 40 years ago,
1:09:41
but we can also run new stuff.
1:09:42
SPEAKER_00: We have new features.
1:09:43
SPEAKER_00: We have new IO paths that are even better.
1:09:46
SPEAKER_00: new types of functionality.
1:09:47
SPEAKER_00: We have new security models.
1:09:49
SPEAKER_00: We have new capabilities.
1:09:50
SPEAKER_00: We have new types of stuff for the new stuff.
1:09:51
SPEAKER_00: But we didn't break the old stuff.
1:09:53
SPEAKER_00: So we can do both stuff.
1:09:54
SPEAKER_00: You can rewrite your code.
1:09:55
SPEAKER_00: I know the databases are rewriting it to use IOU ring,
1:09:58
SPEAKER_00: which is a new way to do I.O.,
1:10:00
which gets the user space to kernel boundary out of the way and
1:10:04
does faster paths.
1:10:05
SPEAKER_00: So they're speeding up the databases by porting to new Linux features.
1:10:09
SPEAKER_00: But their old databases still run just fine.
1:10:11
SPEAKER_00: And so it's like people look at it like, oh, nothing's changed.
1:10:15
SPEAKER_00: Because the old stuff still works.
1:10:17
SPEAKER_00: That was the goal.
1:10:18
SPEAKER_00: The old stuff still works.
1:10:20
SPEAKER_00: So I don't know.
1:10:20
SPEAKER_00: Just see what new, I mean, new hardware features.
1:10:22
SPEAKER_00: I see the new hardware coming all the time.
1:10:25
SPEAKER_00: We get told by the CPU vendors, like, look at this new chip.
1:10:27
SPEAKER_00: It's like, great.
1:10:29
SPEAKER_00: So that's always fun.
1:10:31
SPEAKER_02: And then in terms of contributing to Linux,
1:10:33
SPEAKER_02: we just went through this example,
1:10:34
SPEAKER_02: and it seems pretty easy to contribute,
1:10:37
SPEAKER_02: honestly.
1:10:38
SPEAKER_02: I wanted to ask advice to contribute, but my sense is just do it.
1:10:43
SPEAKER_02: It's not that difficult.
1:10:44
SPEAKER_02: But from a professional point of view,
1:10:47
SPEAKER_02: what do you think a developer who is building other stuff at a company,
1:10:53
what would they get professionally out of contributing even one
1:10:57
change or a few
1:10:58
SPEAKER_02: changes to Linux?
1:11:00
SPEAKER_02: How could their outlook change or what could they learn?
1:11:04
SPEAKER_00: Well, the best thing is it's your resume.
1:11:07
SPEAKER_00: So I talk to college students.
1:11:08
SPEAKER_00: I talk to college students at VU, other universities all the time.
1:11:11
SPEAKER_00: Say, hey, contribute to the kernel while you have time.
1:11:13
SPEAKER_00: And then when you go to get hired,
1:11:15
SPEAKER_00: somebody can look at you and say,
1:11:16
SPEAKER_00: oh,
1:11:16
SPEAKER_00: yeah,
1:11:16
SPEAKER_00: look,
1:11:16
SPEAKER_00: you do play well with others.
1:11:17
SPEAKER_00: And you do contribute at our stream.
1:11:19
SPEAKER_00: Because when you come in as a company, you're not...
1:11:21
SPEAKER_00: writing code from scratch,
1:11:22
SPEAKER_00: you're working with other people,
1:11:23
SPEAKER_00: you're working with existing code bases.
1:11:25
SPEAKER_00: If you contribute to Linux,
1:11:27
SPEAKER_00: or any open source project,
1:11:28
SPEAKER_00: you show that you can work with others,
1:11:29
SPEAKER_00: you can work with existing code base.
1:11:31
SPEAKER_00: So it shows a great skill set.
1:11:32
SPEAKER_00: When I hired people when I was at IBM,
1:11:34
SPEAKER_00: if you contributed versus not,
1:11:35
SPEAKER_00: it's like,
1:11:36
SPEAKER_00: oh,
1:11:36
SPEAKER_00: that's an easy sell,
1:11:37
SPEAKER_00: I'd rather take that.
1:11:38
So from a personal point of view,
1:11:40
contributing can get a job easier,
1:11:42
get a next job.
1:11:43
For another point of view is from an engineer,
1:11:45
you get to learn new things.
1:11:46
SPEAKER_00: I wrote my first driver and sent it out.
1:11:48
SPEAKER_00: So here it is.
1:11:49
SPEAKER_00: It's all perfect, whatnot.
1:11:50
SPEAKER_00: Everybody's like, no, this is wrong, this is wrong, this is wrong.
1:11:51
SPEAKER_00: And have you ever heard of multiprocessors?
1:11:53
SPEAKER_00: I'm like, what?
1:11:54
SPEAKER_00: What is all this?
1:11:55
SPEAKER_00: And that's great.
1:11:55
SPEAKER_00: From an engineering point of view, I want to know better.
1:11:57
SPEAKER_00: I mean,
1:11:57
SPEAKER_00: the Linux kernel developers,
1:11:59
you can never have all the best developers in the world at the
1:12:01
same company.
1:12:02
SPEAKER_00: But when an open source,
1:12:04
the best operating system people can all work on the operating
1:12:06
system together.
1:12:07
So the depth and talent of the people that are working on Linux
1:12:10
is just amazing.
1:12:11
SPEAKER_00: Take advantage of that.
1:12:13
I'll say the Rust developers that are working on Rust for Linux
1:12:16
are core Rust developers.
1:12:18
SPEAKER_00: These people are really, really, really good.
1:12:20
SPEAKER_00: They maintain core parts of the Rust infrastructure.
1:12:22
SPEAKER_00: Take advantage of them.
1:12:23
SPEAKER_00: I mean, I'm learning so much from them.
1:12:25
SPEAKER_00: So from an engineering point of view,
1:12:27
There's these people that are really out there and willing to
1:12:30
help you and grow as
1:12:32
an engineer and learn different processes and learn different
1:12:35
skills much better.
1:12:36
SPEAKER_00: I mean,
1:12:36
I learned so much more working in the community than I ever did
1:12:39
working at
1:12:40
SPEAKER_00: companies because you have better review process,
1:12:42
you have more exposure to crazy corner cases that you hadn't
1:12:45
thought of that,
1:12:46
SPEAKER_00: oh,
1:12:46
SPEAKER_00: yeah,
1:12:47
SPEAKER_00: in the real world,
1:12:47
SPEAKER_00: yes,
1:12:47
SPEAKER_00: that would have been one in a million,
1:12:49
SPEAKER_00: but we do have to take that because we have a million bucks,
1:12:51
SPEAKER_00: four billion machines out there.
1:12:53
SPEAKER_02: Plus, I guess, the more curious you are, everything is open in Linux.
1:12:57
SPEAKER_02: So I remember when I joined Uber, I was just amazed by the RFC process.
1:13:01
SPEAKER_02: And internally, I could read all the RFCs.
1:13:03
And I spent a week or two just kind of reading and trying to
1:13:06
take it all in.
1:13:07
SPEAKER_02: Linux is here.
1:13:09
SPEAKER_02: Obviously, it's overwhelming if you just start at once.
1:13:12
SPEAKER_02: But you can target something.
1:13:14
SPEAKER_02: And so you can just,
1:13:16
SPEAKER_02: even if you contribute little or even before you contribute,
1:13:18
SPEAKER_02: you could just learn.
1:13:19
SPEAKER_02: You can see how the changes are made.
1:13:21
SPEAKER_02: You can try to understand these things.
1:13:22
SPEAKER_00: Yeah, I will say it's not the best learning operating system.
1:13:25
SPEAKER_00: There's really good learning operating systems out there.
1:13:27
SPEAKER_00: We're not this.
1:13:28
That being said,
1:13:28
I mean,
1:13:29
people still write academic papers about it and all this stuff.
1:13:31
We want to rewrite the schedule or do all this fun stuff with
1:13:34
it because it is a
1:13:34
SPEAKER_00: real world tool.
1:13:37
SPEAKER_00: I mean,
1:13:37
SPEAKER_00: I learned from Minix and Linus learned from Minix,
1:13:39
SPEAKER_00: which was a learning operating system.
1:13:41
SPEAKER_00: And then we took those ideas and that and we made Linux with it.
1:13:45
SPEAKER_00: I mean, Linus did it way before me.
1:13:47
SPEAKER_00: But learning operating systems are great,
1:13:49
SPEAKER_00: but working on a real-world one system is a little bit different.
1:13:52
That being said,
1:13:53
there's parts of the kernel that are very easy to get into for
1:13:56
newbies.
1:13:57
We have a whole section of code with really bad,
1:13:59
crummy drivers that are the wrong coding style.
1:14:02
SPEAKER_00: They have the wrong formatting.
1:14:04
SPEAKER_00: They have just a lot of dead code.
1:14:07
SPEAKER_00: That's there for beginners to take up and take your first patch,
1:14:11
SPEAKER_00: fix the spelling mistakes,
1:14:12
SPEAKER_00: fix the coding style,
1:14:13
SPEAKER_00: learn how to do this stuff.
1:14:14
SPEAKER_00: And there's a whole website,
1:14:15
SPEAKER_00: kernelnewbies.org is a wiki that has a whole bunch of stuff on it,
1:14:18
SPEAKER_00: how to write your first kernel patch,
1:14:20
SPEAKER_00: how to get involved.
1:14:21
SPEAKER_00: I've given old YouTube talks of research how to write a kernel patch.
1:14:24
SPEAKER_00: I need to do a newer one.
1:14:25
SPEAKER_00: It's fun.
1:14:26
SPEAKER_00: I've gone to universities and said,
1:14:27
SPEAKER_00: here,
1:14:28
I gave everybody a file that you're going to write a kernel
1:14:29
patch for this file.
1:14:30
SPEAKER_00: It's like, what?
1:14:31
SPEAKER_00: Okay.
1:14:31
SPEAKER_00: And they do it,
1:14:32
SPEAKER_00: and by the end of the class,
1:14:33
SPEAKER_00: end of two hours,
1:14:34
SPEAKER_00: they send a patch off,
1:14:35
SPEAKER_00: and they got it accepted.
1:14:36
SPEAKER_00: It's very simple.
1:14:38
SPEAKER_00: It's not a difficult thing to do.
1:14:41
SPEAKER_00: And we want new people to get involved because we want...
1:14:44
We don't know who's out there or what they can contribute if we
1:14:46
just want to do
1:14:47
SPEAKER_00: something for fun or do something for real.
1:14:50
SPEAKER_02: It's great.
1:14:50
SPEAKER_02: Awesome.
1:14:51
SPEAKER_02: Well,
1:14:52
this has been really interesting and I'd just like to close off
1:14:54
with some rapid
1:14:55
SPEAKER_02: questions where I just ask and then you do what comes to mind.
1:14:59
SPEAKER_02: What's the most memorable patch that you've contributed to in Linux?
1:15:03
SPEAKER_00: This is going to be about people again.
1:15:06
Early 2000s,
1:15:08
we were starting to get,
1:15:09
Microsoft was saying Linux is a cancer.
1:15:11
SPEAKER_00: We were all worried about Linux.
1:15:12
SPEAKER_00: Oh, I remember.
1:15:12
SPEAKER_00: Yes, you remember that stuff.
1:15:14
SPEAKER_00: We started getting some really,
1:15:15
really good patches for some hardware that we really didn't
1:15:18
know that well.
1:15:19
SPEAKER_00: It was showing some really good stuff.
1:15:21
SPEAKER_00: And it was like, this is really good.
1:15:24
SPEAKER_00: And we're like, where did you get this information?
1:15:26
SPEAKER_00: How did you know this stuff?
1:15:27
SPEAKER_00: Is this like somebody trying to sneak this in?
1:15:29
SPEAKER_00: And the person wrote back and said, here's how I found this.
1:15:31
SPEAKER_00: Here's how I tested it, how they did this.
1:15:33
SPEAKER_00: I'm like, OK, all right, we took this.
1:15:34
SPEAKER_00: And over time, we took all these patches over time.
1:15:36
SPEAKER_00: And then we have this conference once a year for all the maintainers.
1:15:40
SPEAKER_00: And you get invited to it.
1:15:41
SPEAKER_00: And we're like, oh, let's give them an invite because it was really good.
1:15:44
SPEAKER_00: And it was in Canada every year for a number of years for some reason.
1:15:48
SPEAKER_00: And they came and they showed up and he showed up and he's like,
1:15:51
SPEAKER_00: oh,
1:15:52
SPEAKER_00: sorry,
1:15:52
SPEAKER_00: I had to bring my mom because he was in high school.
1:15:55
SPEAKER_00: He was 17 years old.
1:15:56
SPEAKER_00: None of us knew.
1:15:58
SPEAKER_00: And he contributed.
1:15:59
SPEAKER_00: And it was like, OK, great.
1:16:00
And it turns out he later went on to MIT,
1:16:02
and now he's a professor at Stanford.
1:16:04
SPEAKER_02: Wow.
1:16:05
SPEAKER_02: Yeah, all you see is an email address in Linux, right?
1:16:07
SPEAKER_00: Yeah, just say, yeah.
1:16:09
SPEAKER_00: It's like, OK, yeah.
1:16:10
SPEAKER_00: It's like that.
1:16:11
SPEAKER_00: I mean, it's things like that.
1:16:12
SPEAKER_00: Wow.
1:16:12
SPEAKER_00: That's good.
1:16:13
Another one I'm really happy about is there's lots of drivers
1:16:16
that have been
1:16:17
SPEAKER_00: sitting outside the kernel tree for many,
1:16:18
SPEAKER_00: many years,
1:16:19
SPEAKER_00: just because people never got them upstream or whatnot.
1:16:22
One of them is the subsystem to handle Braille keyboards,
1:16:26
so Braille displays.
1:16:28
SPEAKER_00: Just feel that those are outside the tree.
1:16:29
I and a couple other developers worked with those people and
1:16:32
got them in the tree
1:16:33
SPEAKER_00: and got them working,
1:16:34
SPEAKER_00: and now they're shipping with all devices.
1:16:36
So we made sure that these people who were always having to
1:16:39
patch this out-of-tree
1:16:41
SPEAKER_00: stuff because these devices only needed for a very shiny subset,
1:16:45
SPEAKER_00: but now it's available for everybody.
1:16:46
SPEAKER_00: I'm very happy to see that happen.
1:16:47
SPEAKER_02: Wow.
1:16:48
SPEAKER_02: And I guess this goes back to Linux of what you said,
1:16:50
SPEAKER_02: why companies will contribute a few developers per year.
1:16:53
Because now when you take Linux,
1:16:55
you get an OS,
1:16:56
for example,
1:16:56
that also has Braille support.
1:16:59
SPEAKER_02: That itself,
1:16:59
SPEAKER_02: like adding it to an existing product or if you built an OS,
1:17:03
SPEAKER_02: that itself would be a massive undertaking.
1:17:05
SPEAKER_00: Yeah.
1:17:06
SPEAKER_00: So now it supports all the devices out there.
1:17:08
SPEAKER_00: Awesome.
1:17:08
SPEAKER_00: What's your favorite programming language?
1:17:10
SPEAKER_00: Still C. Still C. I've been doing C for, what, 30 years every day?
1:17:14
SPEAKER_00: So yeah, C, yeah.
1:17:16
SPEAKER_00: I've been doing a lot of Rust lately.
1:17:18
SPEAKER_00: Rust, I feel like I'm able to write really sloppy code, and it works.
1:17:25
I don't feel like I have to be as precise as C,
1:17:27
which is,
1:17:27
I don't know if that's good or bad.
1:17:30
SPEAKER_00: And what's a book or two that you would recommend reading?
1:17:34
SPEAKER_00: The old Code Complete book.
1:17:36
SPEAKER_00: was a really good one.
1:17:36
SPEAKER_00: That was a really good one.
1:17:37
SPEAKER_00: It taught me that coding style matters.
1:17:40
SPEAKER_00: It doesn't matter what the coding style is.
1:17:42
It's just a generic set coding style matters because our brains
1:17:47
work on patterns.
1:17:48
SPEAKER_00: And as programmers, we're reading patterns.
1:17:50
And when the pattern's the same,
1:17:51
the metadata goes away and we can see the logic easier.
1:17:54
SPEAKER_00: Code Complete has aged a little bit weirdly.
1:17:56
SPEAKER_00: If you look at the first book, it has a lot more C examples and whatnot.
1:17:59
SPEAKER_00: But it talks about the basics behind programming and all that stuff.
1:18:02
SPEAKER_00: And that was a really, really good book.
1:18:05
On the flip side,
1:18:05
another really fun one that's really tiny,
1:18:07
programming pearls.
1:18:09
And it's like bit fiddling and cute little algorithms and neat
1:18:13
stuff like that,
1:18:14
SPEAKER_00: which surprisingly we still do today.
1:18:16
SPEAKER_00: We're talking about adding parity functions in a common way.
1:18:19
And everybody's like,
1:18:20
no,
1:18:20
if you do it this way,
1:18:21
you do it this way,
1:18:21
it'll be faster.
1:18:23
So we're still messing with these things that people have
1:18:25
messed with for 40,
1:18:27
50,
1:18:27
60 years.
1:18:27
SPEAKER_00: And these things still matter.
1:18:28
And they matter to people because cycles matter and power
1:18:31
matters and things like that.
1:18:32
SPEAKER_00: So between those two, those are my favorite ones.
1:18:35
SPEAKER_02: Well, this is awesome.
1:18:36
This has been such an interesting and,
1:18:38
for me,
1:18:38
just really educational and eye-opening chat.
1:18:41
SPEAKER_02: So I'm glad we did it.
1:18:42
SPEAKER_02: Well, thanks for having me.
1:18:43
SPEAKER_02: I found this episode to be a really interesting one about Linux.
1:18:46
I'm still amazed that an open source project managed to become
1:18:49
the most widespread
1:18:50
SPEAKER_02: operating system in the world,
1:18:51
SPEAKER_02: despite not being a commercial business.
1:18:53
SPEAKER_02: It's such an interesting and inspiring project.
1:18:56
SPEAKER_02: You can find Greg on social media as linked in the show notes below.
1:18:59
And if you'd like to try your hands on contributing to Linux,
1:19:02
visit kernelnewbies.org.
1:19:04
SPEAKER_02: For more deep dives related to backend engineering,
1:19:06
SPEAKER_02: check out the Pragmatic Engineer articles linked in the show notes below.
1:19:10
SPEAKER_02: If you enjoyed this podcast,
1:19:11
SPEAKER_02: please do subscribe on your favorite podcast platform and on YouTube.
1:19:14
This helps more people discover the podcast and a special thank
1:19:17
you if you leave a rating.
1:19:19
SPEAKER_02: Thanks and see you in the next one.