Latest / Oxide and Friends / Paths into Systems Programming

Transcript

Paths into Systems Programming · Oxide and Friends

• 0:00
  Speaker 1: Well, okay. First of all, I do feel that we just need to just tie off a little bit on on last week. So amazingly, in a rant about how I was mispronouncing words and everything I knew was wrong, I mispronounced hagiography, which, Dan, you corrected me on and then I ordered you to get out. So I'm sorry about that because he was you you you your correction was in fact correct. So So I it was just a bridge too far for me on the last Monday.
• 0:30
  So, anyway,
• 0:30
  Speaker 2: I I just wanted to
• 0:31
  Speaker 1: Dan, I wanna formally apologize, and thank you for correcting my my grievous mispronunciation of hagiography.
• 0:37
  Speaker 3: Because by by popular demand, this can be, like, a standing weekly, apologies for words pronounced wrong in the last week.
• 0:44
  Speaker 1: Yeah. Exactly. That's right. So, yeah. Well, I just love the fact that you were like, who cares?
• 0:57
  And then immediately corrected me, like, moments later. My so it's like, well, who cares? Clearly, there's a fine line between the policing mind and the criminal mind in this particular dimension because you obviously care. So I care too, so we all care. But that was a lot of fun.
• 1:11
  Last week, that was super interesting. So, the pronunciations aside, the the reliving deck was really interesting. So, so, Adam, I I did this kinda I I've been, excited to see, how many folks are really interested in this topic, which
• 1:27
  Speaker 3: is Yeah. Yeah. Very fired up to see the response. I don't know if about you, but, about Sunday mornings, I started thinking about, you know, if we haven't already, telegraphed what we're talking about, I start getting a little itchy And, this one in particular, because often we go dumpster diving into your recent tweets to see what what went well.
• 1:50
  Speaker 1: Uh-oh.
• 1:51
  Speaker 3: And no. No. I don't I mean, I I nothing had had lipped on the radar. So I was
• 1:55
  Speaker 4: kind of wondering what we
• 1:56
  Speaker 3: were gonna talk about, but then, you know, you had this this this great, kind of out of the blue comment that that I think you received.
• 2:07
  Speaker 1: Yeah. I mean, normally out of the blue. I mean, a a a, DM, and I'll I'll definitely anonymize it here. But, you know, my are your DMs open, actually,
• 2:17
  Speaker 3: Adam? Yes. Any follow-up question? Like, what? What?
• 2:21
  Speaker 1: Yes. No. I'm curious, like, how many people have I I I have open DMs, and it to me, it's actually really important because I yes. There is some amount of spam, but there's also, like, a lot of people who, have interesting thoughts that Yeah. Damn
• 2:35
  Speaker 3: it. Absolutely. And, I don't I mean, the amount of spam I get is very minimal and the amount of, like
• 2:42
  Speaker 1: I am getting a lot of spam. I get a lot of I get a lot of, like, crypto cards and spam. Oh. You do you like any of this?
• 2:49
  Speaker 3: No. I well, maybe. I don't check my, you know, they they have the 2 folders. The the, like, we think it's spam and we and we think it's not spam. And I don't check the folders very often.
• 2:57
  Speaker 1: Why must I stare into sun at the one that you think is spam?
• 3:00
  Speaker 3: The one I get a lot is, like, the I get added to some group of, like, 25 random people.
• 3:06
  Speaker 1: That's all cryptocurrency spam.
• 3:08
  Speaker 3: Oh, okay.
• 3:09
  Speaker 4: Good to know.
• 3:09
  Speaker 1: Anyway, ignore all that. But so I, so someone who didn't know came into my DMs and asked a really interesting question, which is what has prompted this discussion. And so the, the reason I'm writing to you is to ask you for a piece of advice on how to do a transition from where I am in the tech field to a full time systems programmer. Bit of context, I come from a user space background. I've been doing that for 15 plus years.
• 3:33
  I tend to gravitate towards hardware related subjects, and because of that, I've gone through a bunch of different things, solutions architecture, operations in the last 5 to 10 years, and I've really enjoyed that. I've taken various courses on Linux kernel development, and I've really fallen in love with that. And I've been really trying to invest my free time in understanding this stuff and and bridging the gap between my software knowledge and hardware knowledge such as I've been, you know, working on a hobbyist OS, and I've been blogging about it. I've been trying to contribute to the Linux kernel. My problem is that I am having a hard time breaking in.
• 4:06
  I'm I'm not getting interviews at the HR level, And I, you know, try to ask for advice for people and, you know, sometimes it gets some insights, but a lot of them try to talk me out of this and tell me that this is, like, this is not worth doing. So, I don't know how to proceed, and I'm I'm hoping that, you can't you, me in the sense, can can help me proceed. And I'm I was thinking this is a great I wanna hear from everybody on this because this is I I've got my own ideas, but I I think a lot of people gonna have their ideas on how to break into systems programming. And I think for starters, Adam, I feel we should define systems programming. Program.
• 4:47
  Speaker 3: Yes. I we definitely should. Why don't you go first? Because as I was writing it down, gosh, it is it is kind of it's easy talk about characteristics, and it's hard to give a or the hard for me to give a succinct definition.
• 5:02
  Speaker 1: I feel like I have the same problem actually with software. Just just defining software. You've have you done this? You've tried to, like I I'm gonna come up with a concrete definition of software right now. Have you ever done this?
• 5:13
  This is No. It does not go well. Basically, it's really hard to come up with a definition of software such that Euclid wasn't writing it.
• 5:23
  Speaker 3: I was just thinking that. Just thinking, like, that, yeah, that that the Greeks weren't writing definitions of algorithms and that, product managers through the specs aren't aren't programming?
• 5:35
  Speaker 1: I mean, any definition of software that relies on definition of hardware is just gonna get I mean, it's just too easy to make. What if we all annihilate all the hardware, all the software no longer exists? Like, that doesn't make any sense. So they I you you because you and I both know Mike Olsen at Cloudera.
• 5:51
  Speaker 4: Yeah. Yeah.
• 5:51
  Speaker 1: Yeah. So I had actually I went to Cloudera in, like, super early days. This is when we were at FishWorks. This would have been, like, 2,000 and, like, 7, 2008. Mhmm.
• 6:01
  Speaker 5: So
• 6:01
  Speaker 1: we were like, Jeff Hammerbacher was there. We knew we knew a couple of folks. Yeah. And was there a couple of couple of folks over there. And I think I I went by the office.
• 6:09
  There were, like, 5 people there that worked at Cloudera.
• 6:13
  Speaker 3: And this is in the old, Fry's Electronics?
• 6:17
  Speaker 1: This was in Burlingame. Oh. Oh, okay. And I feel that they it was ruining Mike here. I feel that they went through, like, 6 offices in, like, 4 months.
• 6:26
  So this was a very brief stuff. It was a super weird office. I can definitely see why. It was it was like, it it kinda defies description. It was like the set of WKRP.
• 6:40
  That that is not that is yeah. You know what? That that that's not a clarifying comment. But so it it a super weird office and the I had just come from this interesting interview with Arthur Whitney that had me Have have you, like, what are you doing? Why are you here?
• 7:14
  What is happening right now?
• 7:15
  Speaker 3: Right. Is it 2 AM?
• 7:17
  Speaker 1: Is it 2 AM? And you're very high. Are you extraordinarily high in my office at, like, 2 PM on a Tuesday? Like, get out of here. Like, look at these guys anyway.
• 7:24
  So it's like, no. I'm not I'm I'm high on life, Mike. I'm just high on software right now. High as a kite. Okay.
• 7:31
  So systems so I I with the with the kind of the caveat that I apparently, I can't even define software, let alone systems. I do think that, when we talk about systems, we are talking about software software interaction. We're talking about software that runs other software is my most concise definition. Software that is responsible for the the creation of other software, runs other software, executes other software, debugs other software, that it it it it the the interaction is not with an end user, but with that that software software.
• 8:04
  Speaker 2: How bad is that?
• 8:05
  Speaker 3: Yeah. I see. It's it's better than what I had because the the the not as good definition that I was monkeying with was anything that's not really sitting at the top of the stack and that's really insufficient, which is why I like yours better. Because my definition includes lots of libraries and and frameworks, which which, I don't know, felt a little squishier. But but yours where it has more to do with the execution of other software is is interesting.
• 8:37
  Speaker 1: So I think it'd be inclusive of a lot of those is yeah.
• 8:41
  Speaker 4: Is LS systems programming?
• 8:44
  Speaker 2: Yeah. For sure.
• 8:45
  Speaker 1: Yes. Yeah. Yes. Yeah. I think so.
• 8:48
  I I I
• 8:49
  Speaker 4: What about x what about x term?
• 8:51
  Speaker 1: Are you high right now, Josh? Are you on drugs? Are you are you talking about this phase high?
• 8:56
  Speaker 3: Feel like
• 8:56
  Speaker 4: this is well, I'm I'm in Australia.
• 9:00
  Speaker 3: There you go. Yeah. There we if
• 9:02
  Speaker 4: I don't know.
• 9:02
  Speaker 2: I don't know
• 9:02
  Speaker 4: if that's
• 9:04
  Speaker 3: I I I will say yes to Xterm, but I would say that, like, it's very easy because we don't feel good about these these definitions to go from, you know, you know, into into suddenly is Minecraft systems programming and and feeling, like, compelled to say yes to that.
• 9:22
  Speaker 4: Is a is a web browser systems programming.
• 9:25
  Speaker 3: See, that's what I'm saying. So oh, man. Yeah. Because, like, you got, like, your I mean, clearly, you have your
• 9:30
  Speaker 4: I'm I'm clearly. I'm just in there. I'm here to ruin ruin the space.
• 9:34
  Speaker 1: Yeah. No. Exactly.
• 9:35
  Speaker 3: Right. Right. So so I know we're, like, going into CSS's systems programming. I understand that's what you're you're you're getting us towards.
• 9:43
  Speaker 1: It totally is though.
• 9:44
  Speaker 2: Oh, that's
• 9:45
  Speaker 6: I mean
• 9:47
  Speaker 1: Oh, okay. So so okay. So, like, the
• 9:48
  Speaker 4: system So with the Programming is is an approach as much as anything else.
• 9:52
  Speaker 1: Yeah. Yeah. Yeah. I agree. So with the with the let's define it by some of its attributes, because you were saying earlier that, like, I I know some of the attributes about it.
• 9:58
  Because I I definitely feel like I I have got a handle on some of these amorphous attributes. What what are some of the attributes you think about?
• 10:05
  Speaker 3: Well, I mean, I'm embarrassed that that what I kept on coming up with was related to oxide values, and in particular, rigor and curiosity. Mhmm. And, I I think maybe less in terms of systems programming, but more in terms of systems programming, like the mindset of the author. And I think that curiosity about the systems below and above you and about, kind of understanding the nuances of those, and then rigor in terms of thinking through, not just the happy path.
• 10:40
  Speaker 1: Yeah.
• 10:41
  Speaker 3: But I think one of the characteristics of systems programming is thinking about all the ways that things can go wrong and handling those in in ways that are comprehensible.
• 10:50
  Speaker 1: Yeah. That's a very good point. Where do, like, software gets actually a lot harder when our expectations are really high for it. When we actually expect the software to work all the time and not just kinda seem to work in a lab, we we there's a whole bunch of actually really hard problems that we need to go solve. So, you know, I like that.
• 11:06
  I also get that, like, system software and then you're you're switching to be a more positive kind of, amorphous feeling than I've got. System software is hard to love. It's hard to appreciate. It because I feel like, you know, we talk about the systems demos. The we we like to give systems demos inside of oxide.
• 11:22
  And it to to the untrained or even trained eye perhaps, they they might look very unimpressive to see a machine boot, to see a packet go over the network, to see a to see these things that that are seemingly simple, but are fiendishly complicated. And I feel like that's kinda getting into your curiosity point too, Adam, where, like, in order to be successful in systems programming, you constantly need to be kind of pulling at these abstractions because the abstractions are actually trying to seal you off. The the abstractions are trying to get you to not think about them, and yet so much of system software is understanding these the the abstractions, understanding the interactions across the abstractions, evolving the abstractions, it's like you you can't be content to you really do have to understand that you are at a at a new way you described it. Like, you gotta understand the software below you and above you. Yeah.
• 12:16
  Is this a big mess at this point? Are we are we I can't imagine that.
• 12:20
  Speaker 3: Well, I mean, I've I got I get I kept, like, coming to, like, the Justice, Potter Stewart line. You know it when you see it. Yeah. And and I'm not sure that that's that everyone is gonna agree on any particular body of software being, system software or not. But but I think that, you know, the the mindset and the approach of, of robustness and of sort of complexity and perhaps constraint.
• 12:45
  And this is something we've talked about on a bunch of previous spaces where I think perhaps systems program is also defined by, you know, the constraints around you, that it that, that there are kind of spaces you need to fit into or compatibility you need to consider above.
• 13:03
  Speaker 1: Yeah. Interesting. Is hardware facing software always system software?
• 13:11
  Speaker 3: Hardware facing software. I I that is to say software that directly
• 13:15
  Speaker 1: Is is directly on the metal, whether it is in a privileged mode or not. Whether it is I feel that's a good question.
• 13:22
  Speaker 3: Oh, yeah. I mean, it's so tough too because, like, for first of all, I mean, I was thinking about the privileged mode because I, I, you know, 20 years ago, 25 years ago when, when we were starting, there was sort of this clarity of is the privbit set or not. And now, like, what does that even mean? Like you're, you know, you're sitting in some virtualization layer and it's like, what prove that are
• 13:42
  Speaker 1: you the proof that is a lie. The proof that is, yeah,
• 13:45
  Speaker 7: It's right.
• 13:46
  Speaker 1: You're you're talking about software.
• 13:48
  Speaker 3: And it's like, well, okay. If I'm talking to hardware, it's like, well, you know, does that mean that anything that is compiled into an ELF binary is systems programming? Because, like, that's sort
• 14:00
  Speaker 4: of talking to hardware,
• 14:01
  Speaker 3: I guess. Right? Like, machine code instructions. Yeah.
• 14:05
  Speaker 1: And I I feel that, like, also going back I'm just trying to go back to your I like your rigor and curiosity point. System software is hard. Right? Can we agree on that? It is hard.
• 14:15
  Speaker 3: I mean, yes. I think so. Right. I mean, I I mean, it's hard for me.
• 14:19
  Speaker 4: As as a subset of old software, it's hard. I mean
• 14:23
  Speaker 1: The the right. I mean, I feel that, like, it's hard in a particular way. Like, numerical programming, I think, is extraordinarily hard in a different way. System software, I mean, is that
• 14:34
  Speaker 3: No. I think that's absolutely right. And I think that in the same way that, like, user interface programming
• 14:40
  Speaker 1: is hard.
• 14:40
  Speaker 2: It's really hard.
• 14:41
  Speaker 3: I think in the same way that, I I mean, you probably have this experience with user interface programming and with systems programming where to a person who is incurious or, you know, not adept. Like, there are, you know, I've seen, engineers produce user interfaces where it's like, you know, the things aren't right, and they kinda don't see it. And I think in the same way, I've seen people produce system software, which fails in a bunch of ways. And it's until you point it out, it's hard for them to see it. So I think that it's it's a skill certainly that
• 15:15
  Speaker 4: can be
• 15:15
  Speaker 3: taught, like all these others, but hard in ways that is differentiated and where, you know, folks have different levels of interest in getting into those details.
• 15:25
  Speaker 1: Yeah. I
• 15:26
  Speaker 6: think that stuff can really be taught. I you know, like, it's not about
• 15:32
  Speaker 3: Hey, hey, Dan. You're very faint.
• 15:34
  Speaker 6: I'm is this better?
• 15:36
  Speaker 3: A little bit. Yeah.
• 15:37
  Speaker 6: Okay. What I was saying is I'm not sure that user interface stuff can really be taught. Or if it can be taught, it it it's taught in the same way that art is taught. I've often thought that if somebody said to me, Hey Dan, go write user interface stuff. I would probably fail miserably.
• 15:54
  Not because I wouldn't try, but rather because I just don't think that I have either the aesthetic or artistic ability to create a user interface that's truly good. You know what I mean?
• 16:07
  Speaker 1: And I do think yeah. Dan, I do know what you mean. And I kinda think that, like but in system software, we do create a lot of interfaces, but that we are often creating them for effectively ourselves or our doppelgangers where Yeah. So it's easier for us to kinda intuit about the abstraction to create, I think, the interface to create.
• 16:25
  Speaker 4: I think both both of those areas have a strong requirement for taste to produce good artifacts.
• 16:34
  Speaker 7: But I think there's a something that
• 16:37
  Speaker 6: Brian hit on, which is that we create those interfaces, but we do it for people like us. Right? Like, other systems people. And and let's be honest, like, we there are a lot of interfaces that were designed by people with exceptional taste and say the 19 seventies or eighties that really haven't changed very much. You know, look at, like, the read system call for example.
• 16:58
  It takes 3 arguments. It takes a file descriptor, a pointer, and a length of a buffer. Like, that was brilliant at the time, but you you kinda like, how do you improve on that? You know?
• 17:08
  Speaker 1: Yeah. So okay. So in terms of I mean, I I don't know if we've if we've I don't know.
• 17:15
  Speaker 3: Well, certainly no answers there. Right?
• 17:17
  Speaker 1: Account with that. But but
• 17:18
  Speaker 3: I think, like, to you know, in terms of some of the characteristics that the person who, who DMed you was talking about, they're probably talking about things like embedded programming, like kernel programming or building compilers or virtual machines, things of that sort. Or building compilers or virtual machines, things of that sort, I'd imagine.
• 17:36
  Speaker 1: Yes. And I think that the I mean, to me, for whatever reason, I guess it's like, well, let's talk about actually how we got interested in this. And it because I feel that because system software is it's hidden from view. Right? I think that this is one I I do feel that, like, one of the attributes of it is it is hard to appreciate.
• 17:57
  You need to get kind of into it to really understand how complicated these things are. And I feel like there's this moment in one's education, be it formal or informal, where you realize, like, holy god. I can't believe anything works at all. I can't believe that, like, when I sit here at a prompt and, you know, get a directory listing. The amount of complexity that's happening, like, every time pitch perfect to deliver that.
• 18:22
  I just feel like that there's, like, there's gotta be that kind of a moment where one says, like, that is either gives you a a it's a pounding headache and you're, like, I never wanna think about this again or I think add them to your kind of curious curiosity and and rigor point, you think yourself, like, wow. That's super interesting. I actually really I I wanna dig in. I want it, like, I bet there's a lot of other stuff I don't know, and kind of taking you down the rabbit hole. I feel like is is that a fair common theme about how people got into this?
• 18:52
  Speaker 8: Can I say that I remember the systems we love talk on YouTube that said, what happens when you exit with 0? Do I remember the title correctly? And when I saw that talk, I was like, oh, okay. So a lot of things can go wrong. How is this actually still working?
• 19:10
  1. And 2, apparently, well, the the the the person who was showing it was for on Joyant. I remember that correct. Right? And they were showing it on the SmartOS, code base.
• 19:23
  And, like, how much legacy is actually in there for all of these years, like Dan said, like, things don't ever change. That that's also fascinating. But what's what's also more fascinating is, not well, actually, I do have a question about the fascination on that part is, since you guys been in this thing for 25 years, has it actually changed, like, a lot or or only in the hardware, you know, the the privileged stuff or only the visualization or only abstractions? Is is there an actual major change, that has happened or will happen?
• 20:00
  Speaker 3: I mean, Brian, I don't know if we're gonna have to say it at the same time. Yeah. But I assume we're both thinking Rust. Yeah.
• 20:07
  Speaker 8: Yeah. Damn.
• 20:08
  Speaker 9: Yeah.
• 20:08
  Speaker 1: Yeah. I I I mean, yes. I I think Rust is I I do think Rust is a very big deal. And I think that the the, Rust is also a reflection of a lot of big changes, that the I I think that it systems programming has changed a lot. I think that one of the major ways that it's changed, this is not a deep thought, but Russ has reflect this as well.
• 20:32
  That whole open source thing is a really good idea as it turns out. And the and this kinda gets to in turn, like, the the kind of a point that you're making about these, you know, the these these kind of old software systems that are still functional. I mean, to me, another light that went on in system software was I was accustomed to when I was just kind of writing my own programs. I kind of everything was broken all the time. And as you get deeper and deeper into system software, you guys, like, oh, wow.
• 20:58
  There's a lot of software down here that actually has been working correctly or working as is, I guess, in some cases, for a really, really, really long time. And that kind of Adam, to your point about rigor kind of changed my own expectations for the kind of software that I could write, where I thought, like, wait a minute. I we can actually write perfect software. Why not? And I feel that there's a lot of software in c that is gonna stand, a long test of time, but c, as we all know, has got some really serious
• 21:31
  Speaker 2: shortcomings that make
• 21:32
  Speaker 1: it hard get lost.
• 21:41
  Speaker 3: So, just a quick diversion on that. Brian, you remember when I rewrote Nohup, so, Solaris' story?
• 21:48
  Speaker 10: Yes. Okay.
• 21:49
  Speaker 3: So I rewrote Nohup
• 21:51
  Speaker 1: written Can you explain Nohup a little bit?
• 21:53
  Speaker 3: So Nohup is a command that you use to, to hang up to background a process that you've run interactively. So the canonical use case is like I run a long running build. I forgot to put it under no hub or I forgot to run it in a screen session. And then I need to go home for the night and I don't want to pack up my laptop and kill the session. So I run no hop, minus p to background, and that was the the utility that I added.
• 22:23
  Speaker 1: And being a sig hub, it's the hang up signal that dates for I mean, I think it is signal 1. Right? It dates from the very dawn of the rest of the curve.
• 22:30
  Speaker 3: And this utility I rewrote was written by Joseph Asana in, I mean, just the first, like, moments after the big bang of UNIX.
• 22:40
  Speaker 1: Before the flood, for sure.
• 22:41
  Speaker 3: Right. So now we're talking, like, seventies. And I just wanna point out, Brian, that it has been 20 years since I did that.
• 22:50
  Speaker 1: Yeah.
• 22:50
  Speaker 3: Just to, like, show how I'm kinda old we are and how durable the software is.
• 22:55
  Speaker 1: Yeah. Tom, did you know actually, Tom in here. I I don't know if if Tom knew Joseph Asana, but Joseph Asana, died tragically. He died young, in in
• 23:06
  Speaker 3: a car crash as I recall.
• 23:09
  Speaker 6: Car crash or a heart attack? Heart attack. Sonja died of a heart attack.
• 23:13
  Speaker 1: No. There you go. That that we were waiting for alright.
• 23:15
  Speaker 2: Agiography. In 77. Heart attack.
• 23:18
  Speaker 1: In 77. Yeah. Did you know him, Tom?
• 23:20
  Speaker 4: Yeah. He was there the summer I worked and he he died that fall, I think.
• 23:25
  Speaker 1: Oh, wow. And but so he died with, like, a lot of software that people were not able to rewrite. So I Adam, I recall joking with you that you're gonna be you're gonna be haunted by Asana's ghost.
• 23:40
  Speaker 3: Yeah. And that has borne out, and it's been terrible ever for the last 20 years.
• 23:44
  Speaker 1: Oh, did really does he visit you 3 times? Right? Did he get get, like is it, like, past, present, future? I mean, what's it?
• 23:49
  Speaker 3: Like, oh, hey, Joe. Yeah. Yeah. Yeah. Right.
• 23:52
  Speaker 1: Oh, it's a it's kind of like a cast for the friendly ghost.
• 23:55
  Speaker 3: Got it. Time to go to bed. Right? So so Brian, you, you know, in terms of your own entry into systems programming, you you were writing as a as a high school student. You were writing a bunch of games as I recall.
• 24:07
  Speaker 2: I was doing a bunch of this bullshit.
• 24:09
  Speaker 1: I mean, I I but the the I I feel like a lot of people, I thought I understood computer science when I didn't.
• 24:17
  Speaker 3: Well, but okay. But you're you're writing some code And you went to college thinking that you were, you know, maybe gonna be an economics major.
• 24:24
  Speaker 1: I was gonna be an economics major. Yes.
• 24:26
  Speaker 3: Okay. So, so then what what opened your eyes?
• 24:30
  Speaker 1: Well, I think that the understanding that there was an actual discipline behind computer science was eye opening number 1. It's like, oh my god. This is actually, like, real. There there's, like, actually, you you can, like, reason about algorithms, and I thought that was just that was amazing. And we can actually it's not just programming.
• 24:45
  There's actually computer science. So that was kind of like also, I flunked my first economics exam, which also happened.
• 24:51
  Speaker 3: Hopefully it helps. Yeah.
• 24:53
  Speaker 1: I I I had a c me right on top of the my I had convinced my roommate to take this very, quantitative intensive micro course and I had never done partial differential equations. So I was like learning partial differential equations on the fly as an idiot 18 year old or whatever. And I just remember getting the exam back, and this guy very in the kind of very old school fashion plotted in chalk, all of the scores on the exam, and all of them are, like, clustering in kind of the nineties eighties and then maybe just one of the seventies, maybe, like, one of the sixties. And then there were 2 little dots next to one another at, like, 33. And that was me and my roommate who had had gotten a different 33% of the exam correct.
• 25:41
  Like, we definitely did not. I mean, we got the exact same score, but but but between us, if we combined our knowledge, we would get a d minus on that exam, basically. So that helps. That you know, don't don't underestimate the power of the universe to kinda point you in one direction.
• 25:57
  Speaker 2: Be like Nice.
• 25:59
  Speaker 1: But so that that was kind of realization number 1. I think realization number 2, though, was when and I and I feel like I got very lucky. This happened all very young. I I got a very clear sense of what I wanted to go do at an a very young age. I think this is very unusual.
• 26:16
  But taking an operating systems course for me, and this is why I do think that, like, formal pedagogy is actually important even if you do it online or in an informal capacity, Ben Eater or what have you. I when you kinda sit down and are forced to do a course and a lab intensive course, I this is just this moment of, like, oh my god. I know nothing. I know absolutely nothing. I thought I had some idea about the way computers work, and I have no idea how computers work and no idea how system software works.
• 26:48
  And little did I know that that feeling would typify me for the next 30 years. I feel like at Oxide, I'm always like, well, it looks
• 26:55
  Speaker 4: like I picked the wrong day
• 26:56
  Speaker 1: to know how computers work because I'm learning about something, you know, totally some other aspect that I do not know at all, and I wonder how I've gotten this far not understanding this dimension of how computers work or how system software works. So that for me was the big, like, like going on of the this is why I like your curiosity point because it did actually really arouse my curiosity of, wow, I knew so little, and now I want to learn so much more, and I wanna affect these abstractions. And for me, the ultimate system software demo is the linker assignment we did in that in that course where you wrote a linker. Because, of course and that to me is, like, when you write a linker, it's, like, well, linker's just a program. It just, like, reads things off of the disk and, like, does some things into memory and then, like, jumps to an address.
• 27:44
  But that was just, like I I I mean, I felt like Von Neumann was sitting in my lap, like, whispering into my ear. I felt like it was just, like, it was a hot and heavy romance with Johnny.
• 27:56
  Speaker 4: That was perfect.
• 27:58
  Speaker 1: Lovely. You know what? That's a
• 27:59
  Speaker 3: great image.
• 28:00
  Speaker 1: Yeah. Maybe that was a little too much. I got, like
• 28:02
  Speaker 3: I'm I'm here for it.
• 28:03
  Speaker 7: I'm trying
• 28:03
  Speaker 1: to find out I'm biting on my earlobe. Maybe a little bit too much. But the
• 28:06
  Speaker 4: So what's that movie with like that movie with the pottery?
• 28:10
  Speaker 1: Yeah. Like Ghost. That's exactly what it was. It was like Ghost, but it was but Patrick Swayze was being played by John von Doyman. Yeah.
• 28:16
  That's what it is, and I was Demi Moore, and and the pottery was a Laker. There we go. Well, thanks, Josh. I think we I think we've heard that other
• 28:26
  Speaker 4: I'm I'm here
• 28:26
  Speaker 7: to help.
• 28:27
  Speaker 1: Yeah. There you go. So that to me and that and that was, like, this big revelation. And then there were a couple other revelations that came out of that of, like, I decided I thought I was gonna go to PhD, and then I actually wanted to go to industrial system software. But the one thing that I heard that I think a lot of people have heard is nobody does this.
• 28:43
  Like, you should that there is no they're there. Like, there's no there's no career for you in system software. And I feel that this comes out of the fact that system software is so hidden that the people that work on it are often hidden. And so people don't appreciate. And that's, like, no.
• 28:59
  Like, this is really important stuff, and a lot of people do work on it.
• 29:02
  Speaker 6: So Well, that that's relative. I mean, it took me something like 20 years to to, you know, gently maneuver the entry vehicle to the point where I was actually writing system software, like, of the kind that I wanted to.
• 29:16
  Speaker 1: Oh, so Dan Dan talk about your story a little bit because you've got definitely an online story.
• 29:22
  Speaker 6: Yeah. I mean, I I so I got really interested in operating systems when I was in high school. And in in large part because I wanted to understand how the system worked and I like, really what I wanted to understand was this notion of multiprocessing. And, you know, on a uni processor system, like, what does that mean? How does the operating system do that?
• 29:42
  And, you know, eventually, somebody told me, oh, there's a clock. And, like, you know, that clock ticks every so often. And on a clock tick, you get an interrupt and you do something. And I was like, oh, wow. You know, that was a huge revelation.
• 29:52
  I was like, well, how do you do a context switch? And then, eventually, I sat down
• 29:56
  Speaker 3: and I I wrote a
• 29:58
  Speaker 6: a coroutine jumper team in assembly language for 68,000. And I was like, oh, that's how you do it. You know? And then I was like, well, how do you switch from one process to another? It's like, well, you know, really it turns out in the kernel there's a effectively a thread which represents every process and you trap into the kernel and do something and then you do a context switch from one thread to another and now you're running in the kernel, and now you switch an address space.
• 30:20
  It's like, well, how does that work? Because I was in a different address space. So, well, the kernel is mapped into every address space, every every process. And it was like, oh, okay. You know, so I kinda got these little inklings of how this stuff worked, and that to me was very compelling.
• 30:32
  And I was like, well, I wanna do this. But at the time where I was really interested in working on that, you know, or by the time I I I was in a position where I was, you know, okay, this is really what I wanna do, It was so commodified, especially with Linux and open source, that in many cases, people were just like, no, we're not gonna work on that because that's not interesting. And we don't generate a lot of revenue for that. And so I think if you look at the industry writ large, you you have, you know, many hundreds of thousands, if not millions of programmers who are working on all sorts of things, but they're not doing that sort of low you know, like, there are people who are doing that low level software development, but their numbers are much, much smaller. It's a it's a tiny fraction of a percent of the complete of the total industry.
• 31:16
  Speaker 1: And then so and so but then you were doing all that in high school? Like, all the the 68 k context switching code and so on? Would and this is, like, before
• 31:24
  Speaker 9: or after?
• 31:25
  Speaker 1: I I I wrote
• 31:26
  Speaker 6: I wrote the eventually, I wrote the context switching code kind of after high school. But, yeah, I mean, I was I was in high school when I was trying to figure out how all these things sort of fit together and and and reading operating systems textbooks and and kinda hanging around the local university. And the sysadmins there were kinda my buddies, and so they gave me accounts on a bunch of the Unix machines and so forth and and the vaccine and and the PDP 10 and all of that sort of stuff. And so I got to kinda play around with all of that gear. But I, you know, as a high school student, I didn't really get it at the level that I get it now, for sure.
• 31:59
  And then when I went off to industry and I was like, I wanna work on this, there was just no space to do that, you know?
• 32:05
  Speaker 1: Yeah. Interesting.
• 32:06
  Speaker 2: And so
• 32:06
  Speaker 6: I ended up, like, when I when I got when I worked at Google, for example, most of my career there was working in in at at the user space layer. I I never worked on web stuff. Again, I don't I just don't think I'm talented in the in the right kinds of ways to be successful with that sort of thing. But I spent a lot of time working on code that kind of would, you know, take an RPC and extract some arguments and do something with them and call another RPC or something along those lines. That's kinda most of what programmers at Google do.
• 32:37
  Speaker 1: But that qualifies as system software, doesn't it? Does it not? Is that not system software?
• 32:41
  Speaker 6: I don't think so.
• 32:41
  Speaker 1: I don't think so. Alright. So your definition is more more restrictive maybe.
• 32:45
  Speaker 6: Well, I mean, my definition of system software is system software is software that we write for the use of other software systems. Right? Which is it kinda kinda sounds circular and roundabout, but Yeah.
• 32:56
  Speaker 1: That's sort of my new question. Okay. Yeah. Yeah. Makes sense.
• 32:58
  Speaker 6: Yeah. And and so and that's I mean, like, the stuff I was working on was what we I think back in the day, what are called application level software. Yeah. You know, as distinct from the system software, which is, like, for the running of the computer and and and the consumption of other programs. Like, you write a text editor or something like that.
• 33:14
  To me, that's system software.
• 33:16
  Speaker 1: Yeah.
• 33:16
  Speaker 6: You know, because you're writing that for a programmer or, you know, a user in that sort of
• 33:32
  Speaker 1: system software in the same way. Yeah. Fair enough. I so I want and, Trent, I wanna get to you in a second here. Before we do though, Patrick, if you're here, I wonder if you might, give your story, because I think it's actually I wanna I wanna get to kinda answering this question in the DMs about, like, what what can people how do do you kind of break into system software?
• 33:53
  Patrick, I don't know if you'd you wanna talk about your story a bit because I think it's it's pretty interesting.
• 33:58
  Speaker 10: Sure. Although I'm I'm not certain that it's it answers that question so much. I mean, but I I was working doing, system administration stuff at a serial company here in the Midwest, and started doing I I unlike your experience of having, like, a really crisp operating systems course that kind of crystallized all that interest for you. The the OS course at Iowa State was fine, but it it it was not, you know, I I hear about, you know, like the folks from Brown talking about the OS course there. And it sounds very magical and and the course that I took was not that.
• 34:37
  So
• 34:38
  Speaker 1: Right. Less magical. Well, yeah.
• 34:41
  Speaker 10: I mean, it Which
• 34:42
  Speaker 1: I think is honestly yeah. I think it's honestly I think the more typical one for
• 34:46
  Speaker 3: me. Yeah. Absolutely. I mean, I think that is the norm. And and I've sat in a in a surprising number of OS classes in in, like like, schools that are renowned for computer science and seen some very odd descriptions to sophomores and freshmen.
• 35:01
  Speaker 10: Yeah. I mean, it it it was kind of much more abstract and, like, the big project was just, like, right a shell and it it it just wasn't, you know, you you talk about, like, scheduling in the abstract and paging in the abstract, but it it wasn't, like, you know, go implement part of an operating system
• 35:19
  Speaker 1: Yeah.
• 35:20
  Speaker 10: Which I I think make makes a lot of that more crisp. So but I mean, that that there were there were, like, extracurriculars and and things through I I worked as a web developer and, like, had a group of friends that did, like, a bunch of goofy stuff with computers in a lab, that sort of maintained that interest. I I thought that Adam's point about, about rigor and curiosity, I found to be very on point for getting into system stuff. So when when I was doing kind of sysadmin ops work, that it was often like we we would have systems that were misbehaving and it was like, how how do you gain visibility into this? And, and these were a lot of it was like HP UX machines.
• 36:08
  So it wasn't wasn't like open source where you can go read the source code to the thing. It was like, here are the tools that they give you, but, like, how can you how can you understand what's going on in the system? But from the standpoint of of not a system software developer, but, like, an operations person, like, what what tools do they they give you for that sort of thing?
• 36:32
  Speaker 1: So and Patrick, that's really interesting because when you're debugging a problem like that, especially in the proprietary era, but even but now as well. I mean, you when you're debugging something, you're debugging a system that's not working properly, you really are right at that confluence of rigor and curiosity, where and you you are are and I I mean okay. Like, yeah. I'm talking my book, but I really do feel that one of the things that typify system software is a real emphasis on debugging and that and the methodology of debugging. The idea that, like, hey, this stuff is knowable.
• 37:04
  You can actually like, something that looked like it was totally magical that you had no understanding of, you were able to grind away for some period and, like, oh, we figured out what it was. Project, you must have had that happen a couple of times. I I assume that that was is that where kind of the light was beginning to go on of, like, wow. This is really interesting down here?
• 37:23
  Speaker 10: Some I mean, like, I I I was interested in system stuff and and trying to understand it certainly wasn't it it it wasn't something that, like, I was being paid to do. Right? It it was just like the system would be misbehaving, especially for for the things like the the HP UX machines. You're kind of at the mercy of HP for again, they you you're not reading the source code. You're you're you're not gonna get to the to ground on a problem like you would if you were the developer of the software or had access to the source.
• 37:54
  But it was, you know, like, when we would we would have like, a performance problem or a correctness problem, it was like try and get as many answers as you could out of, you know, if if you found an experienced engineer that you were working with. Like, they would give you answers and and that would offer kind of a a better window into what it was you were seeing or, you know, what why it was happening. How how you might avoid that in the future, that sort of thing.
• 38:21
  Speaker 1: And then so, you should talk about how we cross paths initially, which was how many years is that is that 10 years ago now? I feel like it might be. It's getting there.
• 38:31
  Speaker 10: Getting there. So I I have been working on, there was a need in the company, a non wholly owned subsidiary, at General Mills needed access to part of the active directory for, like, an address book thing. And I I had found, Mark's work of LDAP JS, which was like the only means by which you could, kind of conjure up a LDAP server of like arbitrary data, not not like open LDAP or anything. I I wanted to do filtering on the fly of the LDAP from our active directory so that the this non wholly owned subsidiary could like have an address book, but not get everything in AD. So I was writing patches to LDAP JS so that I could do these goofy, goofy things to filter an address book.
• 39:27
  And apparently, y'all at Joint, appreciated that work.
• 39:32
  Speaker 1: Definitely. So, I mean, there are there's a bunch there. I I don't know if you know this or I
• 39:36
  Speaker 6: don't know if you knew that.
• 39:36
  Speaker 3: Do that. No. Not
• 39:37
  Speaker 2: at all.
• 39:38
  Speaker 3: This all
• 39:38
  Speaker 1: just to me. Yeah. Yeah. So the I mean, first of all, like, you are do not adjust your Twitter spaces. You are hearing correctly.
• 39:44
  This is an LDAP server written in JavaScript. If you're just wondering if that was, like, you were hallucinating that or not, it I mean, it was it was it was the fashion of the day. And so this is an LDAP server written in Node. And, Patrick, I think there's a bunch that's actually really interesting about that. So first of all, like, you had this problem that is a pretty filthy problem, and I think you've you came up with, like, a pretty I mean, it's an inter I mean, I think it would have been a less curious person.
• 40:13
  Would have been like, I don't know. I'm gonna get some proprietary software. I don't know. I got I actually don't know how you can solve that problem because the because you said the existing servers weren't gonna cut it. How would you have solved it without without JS?
• 40:24
  I mean, I guess you
• 40:25
  Speaker 10: I like, I think you would have had to, like, replicate out of AD into some custom like, you you would have to would have had to use client software to, like, periodically replicate. But, like, then it's it's how often do you replicate? Is it out of date? Like, I I didn't have the the privileges with the Windows folks to, like, get some stream of data, so you would have needed to pull it. And being, like, a big corporate environment, active walking over this thing once every 5 minutes or whatever.
• 41:03
  So Right. Like, acting as a server that proxied requests was like clearly the way to go. And I'm sure JavaScript wouldn't have been my my first choice, but it was for something that could do that was like the only choice in that moment.
• 41:18
  Speaker 3: Patrick, did your colleagues at the time, did they did they agree with this approach? Did was it obvious to everyone? Or,
• 41:28
  Speaker 10: I I just went and did this.
• 41:30
  Speaker 1: Yeah. Right. Just solved the problem. Right. Exactly.
• 41:32
  Speaker 3: Got it. Got it.
• 41:33
  Speaker 2: Got it.
• 41:33
  Speaker 10: That that And and people were people were thrilled that the problem was solved. It was like, I I don't wanna hear how it was solved. Like Right.
• 41:41
  Speaker 3: I I I You know what I think? Like, this is this is my experience of being your colleague now too.
• 41:46
  Speaker 1: Right. Exactly.
• 41:46
  Speaker 3: There was a problem and now it's solved or maybe I never even knew that there was a problem.
• 41:59
  Speaker 1: Wait. What? But so the part of the reason I think that your that story is so interesting, Patrick, is because you were using the software as open source, and you started contributing patches to it. And meanwhile, on the other end of this, we're like, who the hell is this person? Like, all of a sudden, we're getting, like, not just, you know it's kinda, like, not, like, easy stuff or, like, you know, kind of the the kind of the the drive by fixes.
• 42:28
  But getting some, like, some deep stuff that was clearly showing some real use of this, And I just remember Mark Cabbage coming to me being like, I've got no idea who this person is, but we absolutely gotta hire them because they are like, I'm going through the code and the code is, like, really clean. And, Patrick, you and I think and, I mean, funny because Josh here as well. Because Josh, I think you were about the same time, also came in through open source. And you, and I don't know if you knew this, but Josh had done the original, SVN work for, for KVM. Right, Josh?
• 43:06
  I think I'm remembering that correctly.
• 43:08
  Speaker 4: On on a Lumos.
• 43:09
  Speaker 1: On a Lumos. Right? Yeah.
• 43:11
  Speaker 4: Because the only machine I have was an AMD machine, and I was extremely sad that some other ones didn't work on
• 43:17
  Speaker 1: And but it's same kind of thing. I mean, it's a little bit different in that you were it was more coming out of a community as opposed to solving a problem at work, I think, Josh. Right? I think that was more just like, this is the machine I've got.
• 43:30
  Speaker 4: Yeah. I mean, it was at the time, it was the it was the the large scale distraction that I needed to prevent myself from doing my final year project at uni. There you go. Right?
• 43:42
  Speaker 1: But you got this
• 43:44
  Speaker 4: I did I did get a ton in
• 43:46
  Speaker 1: there. It being uni or it being the AMD support
• 43:52
  Speaker 2: degrees. Yeah.
• 43:52
  Speaker 4: Yeah. No. I mean, I'm here now. So that's you know, but the, but, like, I had been in the open source community since that was started, which is, like, 2,005, I think.
• 44:07
  Speaker 1: It it but I think
• 44:08
  Speaker 4: Then we started the Lumos, like, when Oracle did the thing that they did.
• 44:13
  Speaker 1: They did the thing that they the Oracle thing that they do. But I think it's
• 44:16
  Speaker 4: pleasantness. Yeah.
• 44:17
  Speaker 1: Yeah. Pleasantness. And, Josh, do you is our first conversation as vivid for you as it is for me?
• 44:24
  Speaker 4: Yeah. I remember you said we talked on the we were on Skype, and you were in an airport, and you were surprised at the fidelity of the the fidelity of the phone call.
• 44:36
  Speaker 1: He said, it's like you're
• 44:37
  Speaker 4: inside my brain.
• 44:40
  Speaker 1: Okay. That's what you remember from that story is like That's what I remember from that.
• 44:43
  Speaker 6: Should
• 44:43
  Speaker 2: I talk about John
• 44:44
  Speaker 1: Von Neumann? Like, was there any any
• 44:46
  Speaker 2: I didn't
• 44:46
  Speaker 4: get any color I'm talking. Well, I mean, I couldn't I couldn't I couldn't see him, obviously.
• 44:51
  Speaker 1: That's right. So what I remember from that conversation is me saying, I just need to ascertain that you're the same person that did the this work that I'm looking at. Because the work Right. Is and I feel like, Patrick, it's in your case as well. I mean, I I I definitely
• 45:05
  Speaker 3: You're asking him to cut himself in half and prove that he's not made of cake?
• 45:08
  Speaker 1: I I that's exactly what I asked him to do. And, unfortunately, he did cut himself in half and he grievously injured himself. But he was not made of cake, so we had not a win. Perfect.
• 45:16
  Speaker 4: But then several months later after, I've been reassembled.
• 45:19
  Speaker 1: That's right. But I think that in both of these cases because I I do think that this is, like, this is actionable, I feel, for people looking to I mean, this is where open source has changed, like, I feel everything about the way we do software development. Internet who you have not even in the same country, in your case, Josh, not same part of the country, in in in your case, Patrick. Someone I've got, like, no overlap with. Patrick, I I yours in particular.
• 45:49
  Is, like, we've gotten we had, like, 0 overlap, but we shared a a zeitgeist and ethos and interest. And, and, obviously, like, we we hired you at at Joynt. I'm absolutely terrific at Joynt, and then you, you know, kept going deeper and deeper and deeper into into kernel work. And I remember and, Patrick, I'm not sure if I'm making this up or not, but I do feel there was a time when you were like, I just I didn't envision myself. Like, I'm actually doing kernel development.
• 46:20
  I I I for whatever reason, I had this kind of mystique about it. I didn't envision myself doing it, but, hey, here I am. As it turns out, it's just a program.
• 46:27
  Speaker 10: Yeah. I mean, I I came in doing mostly upstack stuff, at least for the first little while before, before you suggested working on LX, which is where I I got into the kernel stuff. And, yeah, I I don't know if it's your line or Josh's line about kernel programming where
• 46:44
  Speaker 4: It's just a big c program.
• 46:46
  Speaker 10: Basic program. Yeah. And it's true.
• 46:50
  Speaker 1: It it is a big complicated asynchronously program. That's yeah. Exactly right. And then, of course, like, as, Adam, to your earlier point, like, the kernel's not even in charge of anything anymore because it's sitting on virtual hardware. I mean, obviously, it is.
• 47:00
  But it's sitting in its own little sculpted reality, and there's turns out there are tons of layers previously hitherto unseen layers experience? Would because it feels to me like something we can take away from that is it is actually really worth not just understanding what what a body of software is, but really getting deeply in to a community that you find interesting for whatever reason, and get to the point where you're actually, like, helping them debug the software. I think everyone loves someone who's helping them debug software. Is that a fair inference?
• 47:46
  Speaker 7: Yeah. I I
• 47:49
  Speaker 10: may I it it's so hard to to make generalizations about a a path to success there, I think.
• 47:56
  Speaker 1: Right.
• 47:57
  Speaker 6: I I agree.
• 47:58
  Speaker 4: And and and and, like, I know that I was extremely fortunate to have all of the spare capacity in my life and and all of the, like, advantages that allowed me to spend a bunch of time doing frivolous, like, non university related stuff, for instance, like, on on the side because it was interesting to me and and because it helped me with something that I was trying to do. But, like, if I had to work 2 jobs or something, like, already, I would definitely wouldn't have had time to do all that stuff. So it's really, like, not a one size fits all approach, I think.
• 48:26
  Speaker 6: I I think it's also really weird to generalize because a lot of this is very dependent on community. And I think, you know, much of this discussion, just given people's backgrounds, is sort of gravitating towards the Alumos community and and things that came out of joint and so forth. But elsewhere in the systems world, things are not quite so collegial very often. It you know, like like, seriously. And and I I I think that it can be really difficult to break into that community unless you're kind of already somebody in the community.
• 49:01
  So I
• 49:01
  Speaker 4: I think that that's not just Collegial, but we're a very small pond. And like the LDAP JS pond was even smaller than the Olumoz pond for instance.
• 49:08
  Speaker 1: The LDAP JS pond had had 2 fish on it.
• 49:11
  Speaker 4: Like, you you as a, you know, a person of reasonable competence and and drive and with a problem to solve can show up and become an outsized influence in that pond generally, but merely by being easy to work with and doing lots of good things, which is probably not true in Linux, for instance, just because there are so many voices in that room, it would be hard to be noticed and prominent without also having done other brand building work on your own, I think.
• 49:37
  Speaker 1: Yeah. And which I think is actually an important point. I love small communities. I think people get too hung up on the size of communities. Small communities are often amazing.
• 49:45
  And the and Dan totally agree with you. There's there's plenty of bad behavior in groups and so on. But small communities can be something to gravitate towards. And, again, one should not be, you know, one should be pulling to the stock of interest.
• 49:57
  Speaker 2: But
• 49:57
  Speaker 3: And and important to to just hit on the head again, motivated by something you're his job needed, although they didn't they didn't know how he was doing it, or in Josh's case, in his own self interest or a personal interest. But,
• 50:17
  Speaker 1: you know,
• 50:17
  Speaker 3: I think showing up to a community and saying, what can I do is is a tough way to get in? And
• 50:23
  Speaker 2: I think
• 50:23
  Speaker 3: that in particular, that's gonna be tough, you know, in Linux. But if you're, if you've got some interesting problem, I think small or large community, if you're doing something that is interesting to you, you're gonna produce some artifacts that are interesting to other members of those communities.
• 50:41
  Speaker 1: Totally. Antranik, I know you had your hand up a long time ago, so let's get to you and then and then send me. I Antranik, sorry about that if you're still there.
• 50:48
  Speaker 8: I'm here, but I I I'm here. Well, the last time in this space, I fall asleep, by the way, but
• 50:52
  Speaker 1: now Well, listen. That's why I wanted I you know, I was trying to find the browse button, but I know it is I mean, people should know that it is like, what time is it? Is it 3 in the morning there? 4 in the morning? I mean, it is
• 51:02
  Speaker 8: 4 51.
• 51:03
  Speaker 1: 4 51. Exactly. So those those good reports.
• 51:06
  Speaker 8: I think I forgot. Oh, yeah. I just remembered. So, so we have a a team in the company that basically their only job is to write the trace code. Well, it's a one one one man team, to be honest.
• 51:17
  And, he was always, you know, saying in his, like, Slack bio, like, a a system programming wanna be. And I just wanna, like, tell him that that is system programming. You are doing system programming. So my question now, is that system programming, and are we in a state that if someone is not writing code for the user, we should always ask ourselves, is this system programming?
• 51:41
  Speaker 1: I mean, obviously, for me, I think that is absolutely system programming even though he the he's running in a non Twonie complete language. And, but I think it is definitely system programming because I think it is you know, again, Adam, circling around on your rubric of of rigor and curiosity, like, those are being exhibited in spades in someone who is instrumenting system via d tracer or how you're however you're doing it. Writing that that that kind of meta code or meta thought to go debug the system, I think is absolutely systems programming in my opinion. And and
• 52:13
  Speaker 8: and one so my my last question is I I had the honor to be a I think it's called a TA, a teacher assistant in a university for an OS class. And, what I realized is that, students don't have interest in, like, low level stuff. Is is there is like, why is that? Is it because it's less demo y? And on the point that Dan said about is not being profitable.
• 52:38
  I just realized that we are maybe right now in a state where it's like the exactly the opposite. For example, let's take Netflix with free BSD. The more they invested in free BSD as assist in system programming, the more they got out of it, you know, like bandwidth wise and performance, stuff like that. Are we like reversing the action of system programming to revenue in in this age? With Rust coming in, making it more safe and easier and faster to write this software?
• 53:04
  Speaker 6: I I don't I don't know. I mean, look, you know, I I think one of the things with the with, like, a Netflix is that there's kinda, you know, half a dozen companies or something in the world that operate at that scale. And for those companies, yeah, it makes a lot of sense to hire a lot of system programmers. I mean, Microsoft might be the canonical example here. I mean, it's like they actually market and sell an operating system that people continue to pay good money for.
• 53:31
  So, you know, surely they need to hire a lot of systems people. And, you know, but, like, most places are just not like that. And, you
• 53:40
  Speaker 7: know, it for for a lot
• 53:42
  Speaker 6: of these places, like like Netflix, it makes sense because they have this you know, they can leverage economies of scale, but a much smaller organization cannot. And for them, it makes a lot more sense to treat, say, an operating system like a commodity.
• 53:55
  Speaker 1: Well, I mean, I do think though, Dan, that that that system software software that software that is responsible for any other software is very important in many different spots, and there's there's more demand, I think, still than there is supply of those folks. And so I I think, and, frankly, your question about, like, why are why do students seem incurious about it? My answer to that is students need to be just just putting myself, when I was, you know, 18 years old, I needed to be broken a little bit. I needed to appreciate that all of these abstractions that had been created for me to to give me this kind of delusion of how simple the software is because the the the abstractions are simple just like Dan, your, you know, your your read system calls. It gives a it's a straightforward abstraction.
• 54:45
  It's a simple abstraction and, you know, on and on and on up the stack, we give people these insulating simple abstractions, and then we say, like, well, why are you incurious about all the software below? And I think people are like, wait. What software below? What what what's going on? It's, you know, we need to actually, like you've spent all of your life as a as as a programmer, someone experimenting with software being insulated by these subtractions, and now we actually need to break them down.
• 55:12
  And we actually need to to show you that this is actually these are very complicated systems, and we can't insulate you from them. So I think that the the and that requires a great teacher, I think. I do think that I'm a I guess I'm an old schooler in that regard that I think, great teachers can make a really profound difference for the right person.
• 55:32
  Speaker 8: Yeah. We we ended up having only one person who got interested, and he ended a, a package manager for a language called Oberon, which I I thought
• 55:41
  Speaker 1: I said There you go. Yeah.
• 55:43
  Speaker 8: Which yeah. But but the only one person out of, like, 200. But but it it was good enough, you know, to have something that a very small community of 15 people was interested for 20 years. But, it sounds like a good book, you know, breaking the stack. What That's right.
• 55:59
  Longer there.
• 56:00
  Speaker 1: That's right. That's right. That's exactly right. And I do feel like that there are more now that I think that the positive side is there are so many more resources available now than there ever have been before. And I love Ben Eater's series.
• 56:13
  I love I mean, you got the and, you know, Tom, you talked about the stack book last time on on the the kind of building up the the small system and being able to fit the whole thing in your head. There's more and more of that available than than ever before. So I think that there's an a lot of opportunity for people to learn, but then the question is, like, how do you to kind of mature that. Simeon, you've had your hand up for a long time. Whoop.
• 56:38
  Simeon, you still there?
• 56:41
  Speaker 2: Sorry, Simeon.
• 56:42
  Speaker 1: Yep. We can hear you.
• 56:45
  Speaker 11: Yeah. I think it's a very interesting topic. This, like, how
• 56:47
  Speaker 1: do you how do you
• 56:48
  Speaker 11: get in? One of the thoughts I had so so, Brian, I love the idea of helping people debug stuff. I I recently heard somebody describe reverse engineering as the debugging nobody asked you to do. That seems like a way to break down the doors. Something that that I'd be interested in getting, the folks' input on is how do you do this sort of how do you get into communities beyond just looking for an open source project in a GitHub?
• 57:18
  You know, go find GitHub issues and and solve them for people if you if you wanna get into hardware and that sort of stuff. Like, I know that, Edwin, who's not on this call, today, but he he said he met you folks, first time at, at, like, I think, a open hard hardware meet up, where
• 57:35
  Speaker 1: Yeah. With our
• 57:37
  Speaker 11: you're taking a motherboard apart and, like, trying to, like, reflash a, you know, reflash a, you know, a bias onto a motherboard or something like that. You know, what are what are the venues of the tools, the ways ways that you can get into that kind of thing?
• 57:53
  Speaker 1: Yeah. That's actually a great and, Rick, I was wondering if you might wanna answer that because, actually, Rick was there Rick, Josh, and I were all there when Edwin was there at the OSFC in 2019. Tom was there too. But, Rick, I wonder if you might wanna take I because I think the the reverse engineering point is actually a really good point. Another great way and I think it's reverse engineering, I think, has got more, I mean, from my perspective, more currency and relevance than ever because security is so important.
• 58:18
  But, maybe I'm, I don't know. Rick, is that
• 58:24
  Speaker 12: I I think that's fair to say. I mean, I I came to systems programming via a very roundabout way, and literally by accident. So, you know, part of my career path is I I actually started much more at the hardware level. Growing up, I I had some people in my life who were very much into electronics and electrical work. And so, like, I understood that part really well, and I understood how to use a computer really well.
• 58:52
  But I really didn't understand anything of the the intermediate parts. But I was always curious about how they worked. And just, you know, my my career path, I I grew up in a very rural area in Ohio. I went to a state school in Ohio. I did not have a a particularly good chance of entering into high-tech, at the big companies, yet here I am.
• 59:17
  Partly, that was literally due to a clerical error. So sometimes when people are worried about, you know, what's the path in, just part of it is luck, and part of it is keeping your eyes open for different opportunities, and looking, you know, past kinda what the surface value of an opportunity is. But specifically getting into, like, the curiosity part and and reverse engineering, like, my I was dropped into a situation where I was literally told, hi. You now work at Apple. We are designing a new part a a new machine based on a new processor that nobody's talked about, and it has bugs.
• 59:56
  And we need you to go characterize the performance of it. Have fun. And so I was just given a ton of material and told to go figure this stuff out, and I didn't know what I was doing. But I also knew that because it was a system that was not fully defined, I had a lot of room to experiment, and I shouldn't be afraid to break things. So a lot of Yeah.
• 1:00:18
  Speaker 6: Thanks.
• 1:00:19
  Speaker 12: Reverse engineering, a lot of curiosity boils down to feeling like you have the permission to go try things and experiment and realizing that you're not you're unlikely to permanently damage something. Like, there are things that can do that. But for the most part, you can just do really deep things to your system and see what happens. And then when you don't understand what happened, then you go find the community that can answer that question. And that's when you start to find these communities where you're like, you know, there's the, OS dev channel on Libera chat that talk about all the, like, system mode bring up stuff.
• 1:01:00
  But they don't necessarily get down into the deeper details of the electrical side of of how the hardware works. And, you know, that's a different community. And on reverse engineering, it's kinda coming at it from this perspective of, I understand how the lowest levels of the system work, but I don't necessarily know how the upper layers work. Or I know how the upper layers work, but I don't necessarily know how the lower layers work. How can I do experiments to kind of figure out what's going on?
• 1:01:26
  How can I use the information I have to put together a picture of how how it probably works and then go test that theory? And, you know, that's that's my career in a nutshell. It's basically just poking at things, seeing what happens. When I can't get a clear answer, when I'm stumped, going and finding somebody who at least can point me in the right direction or be a really good rubber duck to kinda let me figure out what direction to go explore.
• 1:01:53
  Speaker 1: Yeah. And, actually, Rick, you're getting you another I mean, on brand for us, but another oxide bio as well in terms of resilience and kind of continuing to grind and, like, okay. I've try I I I don't understand this. What's going on? And instead of hitting, you know, a dead end where other people might might kinda walk away, be like, no.
• 1:02:11
  No. I need to go like find the resource, find the whether it's online or find and ask the questions and, and try different things. I really like your your point about that too because I feel that, you know, often when you're stuck on a problem, it can help you just to get out of a local minimum by just trying some different things. And that can be hard to do, I feel. I feel like we can get ossified in our thinking.
• 1:02:36
  It could be hard to to to remind ourselves to, like, to play around a bit. And do you because, Rick, you also made another really interesting point about doing things that were not assured of success. Of, like, I'm just going to explore this. I don't know if this is a path that's fruitful or not. I feel that's also a a very kind of important theme, for doing systems work.
• 1:03:02
  Speaker 6: That's the agony and the ecstasy of systems programming. And I think that Rick hit on something really important that we don't emphasize enough, which is that these systems have become so incredibly complex that you cannot reason about them from first principles anymore. You have to take an experimental scientific approach to understand the behavior of non trivial systems. Yeah. Period in the story.
• 1:03:24
  Speaker 1: Yep. Yeah. That's very true. And they they that's a very good point that these are, like these systems are very even the simple ones are are finnishly complicated, and you've gotta have an idea. Because, Ricky, you described having an idea of, like, I think the system works this way.
• 1:03:39
  What can I go kinda construct to answer that question definitively? I think it's a is a very and so, Rick, I'm I'm I have to say I'm I'm dying of curiosity about the clerical error. Is that because is Apple your first big break, or is it was there one before Apple?
• 1:03:55
  Speaker 12: Yeah. So, my university I went to University of Cincinnati, and they're the origin of of co ops. Like, they they literally created that entire concept. So it's a mandatory part of the curriculum is is to take alternating 6 month periods and go work in industry and then come back to school and go to industry and come back to school. And the school helps, find jobs for you.
• 1:04:23
  And, usually, they found things in the Cincinnati area, but sometimes people got more interesting things. And it just happened that a friend of mine the placement office accidentally sent his resume to Apple, and he got selected. And they were completely aghast and, like, trying to tell him he can't possibly go do this, and then they were Apple's like, no. Seriously. We really want you to come out.
• 1:04:51
  And from there, it led into a word-of-mouth, like, these are the people who are really good from this school, and it set up a pipeline of of people for at least a few years. But, yeah, I I literally am sort of a one step removed from a literal clerical A
• 1:05:08
  Speaker 1: literal mistake.
• 1:05:09
  Speaker 2: From Ohio to. And,
• 1:05:12
  Speaker 1: Rick, do you think that in all increasingly in a remote world, regardless of whatever trolling VCs say, Founders Fund can go jump in a lake. In increase increasingly in a remote world, certainly in an open source world, do do some of those geographical boundaries break down a little bit? Does the the does the Rick Alpherter graduating from the University of Cincinnati today? Is that person less dependent upon a clerical error to get into this stuff?
• 1:05:45
  Speaker 12: I would like to hope so.
• 1:05:48
  Speaker 1: That's it. Well, that's, I think it's pretty good.
• 1:05:51
  Speaker 3: Yeah. I mean, it could, it could, it could really go either, either go away where, you know, in terms of acting, you would go to LA to be discovered. And I think there was certainly a time when in computer, in, in software engineering, you would go to San Francisco and the Bay Area to be discovered, which created a hurdle in one sense, but another, sort of a leap of faith that folks could take. But now with an all remote world, I think it's more egalitarian certainly, but I think may actually be harder to distinguish yourself as well.
• 1:06:21
  Speaker 1: Interesting. Yeah. So you heard that there's there's a bigger pool for sure. I mean, we benefit I mean but, boy, there are so many folks that we have the privilege of working with at Oxide that I just feel like we would not be working with in a nonremote world. So it's been really, really transformative and important.
• 1:06:38
  Nathaniel, I think you had your your hand up next.
• 1:06:41
  Speaker 7: No. Actually, I think it was Matt.
• 1:06:43
  Speaker 1: Oh, it's Matt. Sorry, Matt. Go ahead.
• 1:06:47
  Speaker 13: Hey, guys. So, getting back to the topic of incurious students and and and, you know, why why people aren't more curious about the low level stuff. I wonder if part of the problem is that on our modern computers, especially I would say the sandboxed mobile and tablet type devices, the insulation that you were talking about earlier is too perfect. Like
• 1:07:15
  Speaker 1: It's awfully good. Yeah. I do feel this way about the I love the Chromebook, and I think the Chromebook is an incredibly important and, undersung product, but the Chromebook is so secure that kids can't I mean, it it's very hard for kids to, you know, the the old war games of, you know, hacking it and and changing their grade to an a or whatever. Like, you're definitely not doing that via Chromebook.
• 1:07:38
  Speaker 13: And Well, even even as far back, I would say, as Windows XP, you were running atop a robust foundation that that you couldn't well, I so maybe this is just nostalgia talking but, I I remember on my family's first computer, the Apple 2 GS, I started getting at least little glimpses of how things worked on a low level as, you know, when I was a little kid. Like Brian, I'm sure you remember on the 2gs, if you pressed, I think it was control open Apple escape, it would, like, trigger an interrupt that would go into this control panel UI. And
• 1:08:19
  Speaker 1: I would say that 100% of my time on an Apple 2 GS was at my neighbor's house getting my ass kicked on Epic's games.
• 1:08:25
  Speaker 13: Ah, right. Okay. So you didn't get near as deep into it probably. But yeah. So and and, of course, the 2 GS wasn't a multitasking machine, but still it it it had this this, you know, this command that could trigger and interrupt and, you know, interrupt whatever you were doing and take you somewhere else.
• 1:08:47
  And at some point, I noticed that if I was running some 2gs game that was, like, playing music in the background, then I could switch into this control panel UI and but the music would keep playing. How does that work?
• 1:09:00
  Speaker 1: Interesting.
• 1:09:01
  Speaker 13: And and and it's some and and it didn't take me long to figure out that if I dropped into the the monitor, which was that this Apple 2, basically a REPL for assembler, for like, where you could like enter lines of assembly language and run them, as I recall. And it didn't take me long to figure out that, like, if I disabled interrupts, the sound would start looping on what on whatever half second it had been playing or something like that.
• 1:09:34
  Speaker 1: So so to give you some okay. So I I I hear what you're saying, certainly. I so I I do think that there and I and totally, like, I think the phones in particular are very hard to I mean, they're they're far too sophisticated. You're not gonna hack in it's too hard to hack in your phone. I I do feel that the the two things I would say are, one, I see at least with my own kids, Minecraft getting I do think you were talking about is Minecraft's programming?
• 1:10:00
  I think I have to say it it probably is getting there, based on some of the things that kids today are are building with Minecraft where you are getting into kind of the the the blocks of way the way things work. The other thing I would say is the thing I do love about this day and age is these eval boards where you can have these super cheap computers that have everything on it. Now they don't have it's not a 2 GS, so you're not playing, you know, you're not and maybe that's what makes it hard for for kids to get captivated captivated by it. But there's a sense in which it's actually things are are very accessible in way that they that they haven't been.
• 1:10:36
  Speaker 3: Yeah. Yeah. Matt, it's funny you took this this route because as you were talking about Chromebooks and these locked down phones, I was thinking about, you know, the the point of curiosity that a lot of kids go through is cheating at video games. And I, like, I certainly did. Like like, you know, kind of figuring out how to, you know, hack shit where games
• 1:10:54
  Speaker 13: would have to pay for it. I would have actual I would have had to, yeah, be able to play the games in order to cheat at them. But yeah.
• 1:11:02
  Speaker 1: Right. Fair fair fair
• 1:11:03
  Speaker 3: fair enough. But, you know, obviously, on on the you know, one of the first kind of, pieces of hex editing my my now 16 year old, but then, like, 9 year old was doing was to cheat at a video game on the phone where they could extract kind of data files from the phone to the computer and then stuff them back in after editing, you know, aspects of the save file. So I think that even as things feel locked down and things feel kind of curated in this walled garden, there are lots of little edges for curious fingers to get get underneath.
• 1:11:41
  Speaker 1: That's an awesome anecdote. And I also feel that because, I mean, you've got especially that you your older has got that real disposition, a did not exist when I was a kid. And that's, I think, another way that that kids are really understanding how things work in a way that's exciting to them. Certainly, my daughter is just loving that stuff. Hey.
• 1:12:07
  Quick note. We got a lot of folks got their hands up. I wanna get to them, and then we've got actually a lot of folks that are waiting to speak because Twitter Spaces has a very low cap on speakers. If if you have requested to speak and can't speak yet, it's because, we can only have whatever it is, 12 speakers. So
• 1:12:24
  Speaker 3: Yeah. So if you've said your piece to, you know, making making space for someone else would be appreciated. Yeah.
• 1:12:29
  Speaker 1: It'd be great. On that note, Nathaniel, and then I think, Colleen, and then I know, Patrick, you had your hand up, and then Dan.
• 1:12:36
  Speaker 7: Yeah. So so it's funny that you sort of brought up the computing for kids thing, because I've been thinking about this a bit lately. Because one of the things that I did in in undergrad when I was bored was I ported ported Inferno to l 4 just because. So if you don't know what Inferno is, it's a sort of reimagined plan 9 of sorts that runs inside its own sort of virtual machine. And, it has, like, the language called limbo in it.
• 1:13:06
  There's a compiler for it. Limbo is kind of like Golang except came with generics at first, which was kind of interesting. But, no, I've been thinking about this because if you wanna sorta of get kids interested in systems programming, plan 9 on the Raspberry Pi 4 is actually a small enough system that like, the c compiler is, like, 55,000 lines. It's tiny, and, like, it's a it's a small system. There aren't that many Sys calls to it.
• 1:13:34
  Everything is through 9 p, And, like, it's it might be worth looking into if you wanna sort of try to get somebody who's, you know, undergrad high school level to get interested in, like, c programming and, like, how does an OS work? I think plan 9 is also hard real time, so you kinda get that aspect as well as well. But, that's yeah. It's I I'll I'll let I'll let I'll let Dan grumble about it. There was also a a mention, a note from Dan too about OS as a commodity.
• 1:14:04
  So I used to work for a, boutique hosting company, who owes who hosts, like, InfoQ and, like, used to host Atlassian's corporate infrastructure. And we were contracted out to build out the, hosting platform for Jira Studio back in the day. And I wrote the cluster scheduler for that back in, like, 2010, back before containerization was actually kind of a a household name. It was one of those things that we just did because we needed we needed the tenancy. And, you know, running a whole bunch of virtual machines across, you know, however many tenants we had.
• 1:14:43
  I think we had, like, 15,000 tenants or something. It was low 10 of the 5th. Was just not tenable. Like, you just couldn't pack VMs tight enough for that. So we wrote this piece of software in c plus plus o three, like ISO standard c plus plus that just farmed out processes to machines and did the thing.
• 1:15:05
  And they'll come to find out that Google had Borg at the time.
• 1:15:09
  Speaker 1: And Right. Yeah. Right.
• 1:15:11
  Speaker 7: And, so so what's really funny about about the the thing that we had internally was that it was called babysitter. And come to find out that Google actually had their own thing called babysitter that was sort of like the, as you would probably phrase it, Brian, the the, the. Right? The sort of, like, the primordial version of Borg. And it was basically just a just a, you know, a process supervisor.
• 1:15:35
  So if you're, you know, familiar with daemon tools or or run it, it was sort of in in that same vein, except we just made it distributed. So we had a, you know, a job queue sort of sitting out way out in front on a on a on a machine somewhere. It was just an append only, database, append only log. But yeah. So, actually, the way that I got into systems programming, sort of like to cap this all off, was to, to, I think, Adam's point, I was I was looking at, like, cheating at games, and this is actually sort of my entrance into the demo scene.
• 1:16:08
  I used to I used to write, cracks for, like, Commodore 64 games. You know, you'd you'd get the you'd get the disc from your friend who worked at the, you know, the video game store, and you would spend, like, the weekend, cracking the bootloader because there would be copy production or something on it. So you go into the, you know, c 64 assembly monitor, and you, you know, figure out how to sort of not slide your way out of it. And, and that was that. And then if you had enough space left over, you could just write, like, a, you know, a short scroller or or whatever in there to, you know, draw pretty colors on the display while the while the, you know, the game loaded up.
• 1:16:43
  What's funny is when I when I was in high school, we had a an electronics program. So, like, you did solid state in digital and the whole 9 yards, and there was a sort of microcomputing component to it. And we had this, like, z 80 trainer for, like you know, you would hand assemble z 80 code, and you would enter it into, like, this this panel with switches, and then, like, you would press the step switch. And, one of my yeah. Yeah.
• 1:17:08
  Yeah. Yeah. Yeah. And this was so That
• 1:17:09
  Speaker 1: feels very dreamlike. That actually happened, or is that I mean, I just feel like the No.
• 1:17:13
  Speaker 7: It was no. It's it's a it's a real thing. So so what I what I wound up what I wound up doing was actually for my senior project in high school was I, I put together a version of that, also for the z eighty, but with a few a few different sort of, takes on it. So there was, there's some documentation on the Game Boy and, like, the sound hardware, and the Game Boy was also a zed machine. So I was like, you know, maybe I can do something with, like, you know, some oscillators and, like, build my own sound hardware for my own version of this and, like, you know, just sort of make a, you know, a chip tune sent out of it.
• 1:17:45
  Speaker 1: Yeah. Game Boy Hacking is, like, a huge thing. We got a we got a colleague who's super into Game Boy hacking, and it's Yeah. Really interesting. I mean, it's Yeah.
• 1:17:55
  Speaker 7: It There's there's a lot to it. It's it is a it is a surprisingly deep little piece of hardware. And, like, this is this is a thing from 1988. This is a little, you know, like, double a powered device with a z80 in it and, like, I some some sound hardware to sort of, you know, make beeps and bloops, and something to refresh to to refresh the display. But, like, it's actually surprisingly deep.
• 1:18:16
  So that was my senior project in high school is I I put together sort of this, like, chiptune synthesizer thing.
• 1:18:23
  Speaker 5: And then,
• 1:18:24
  Speaker 1: And is it so they know in terms of, like, the get kinda getting back to the the kind of the the question at hand. So if someone's looking to kinda break into this stuff professionally Who's who's been who's been doing the maybe higher level software, more application focused software for a peer like, what's the I so I I do wanna kinda get us to, like, what's the what's the advice for folks who are who who wanna get lower? And kinda while you're thinking about that, I wanna get, Colleen, you your hand's been up for a long time. I wanna get I wanna
• 1:18:58
  Speaker 4: get you in here as
• 1:19:00
  Speaker 1: well. Yeah.
• 1:19:00
  Speaker 14: Thank you, Ryan. Can you can you hear me first of all?
• 1:19:03
  Speaker 1: Yep. We can hear you.
• 1:19:05
  Speaker 14: Yeah. So I wanna take this, narrative of, you know, system programming and why, you know, new folks are not doing it. I think I'm gonna share my experience. I think that's very relatable to this. I think the way, if you look at the old docs, right, they are just like books.
• 1:19:25
  So whenever I was learning, like, Java Spring, like, couple of years back, I would read the docs. I would be like, say, okay. I wanna go deep into this. And then the documentations are horrible, to be honest. They are not interactive in a sense that they are not on the point.
• 1:19:41
  So there there is no hook. The way
• 1:19:44
  Speaker 9: I see that the the
• 1:19:46
  Speaker 14: the I I just had a Twitter space, by the way, with Dan on this topic. He writes docs for the Solid JS. So he's a very so he had a very strong views there. So the thing is as long as even, you know, if you want to build interest, this is in general true, you need to provide the active value. If you if you if your docs are just like books in, like, 700 page, I don't necessarily see anybody would be interested.
• 1:20:17
  I don't think this is a problem with system programming, but this is true in general. If you look at, like, Springboks, I see this because this is a very relatable experience I share. If you look at Springboks and you just see you wanna learn, you would see that they're teaching you XML, which is almost obsolete, and you would you you you just want to get started in, like, 20 minutes. One more thing I see, if you look at the modern way that the things are taught, is educative.io.
• 1:20:49
  Speaker 1: It So therefore, actually, what can we before we get to that, can we just talk about docs for a second? Because I think you've hit on a really important point here, Killeen. The it it because I do feel that a hallmark of a great system of great system software is great documentation. And I I feel that, you know, we our our colleague, Cliff Beffel, has done if you I've gone into the Hubris documentation, really spent a lot of time it takes a lot of time to sculpt that documentation, and I I do feel that's more the exception than the rule. So, Klim, are there particular systems that I don't know.
• 1:21:26
  Have you read the the Hubert Stocks, by the way?
• 1:21:27
  Speaker 3: Yeah. Unbelievable. I mean, like like everything that Cliff has done, it's just an incredible job.
• 1:21:32
  Speaker 1: Just extraordinary. And and and very it makes it very accessible too because it means that someone can kinda roll up to it. They Clint, to kinda your point, I wanna understand, like, why what like, what is the good system software documentation has that crisp why. And I yeah. I think it is a bit lacking.
• 1:21:52
  Klima, are there are there systems that you think today have particularly compelling documentation or is it something that your in your steam has just kind of been going downhill?
• 1:22:02
  Speaker 14: Yeah. I think, let me put it this way from the learning perspective. Right? I'm just trying to be very generic, in a sense, and that can apply to system software. If you look at, you know, educative.
• 1:22:15
  Io, in a sense, because I was just, enrolled in there. One of the courses and the way they make it so easy is that their websites, So it's not just dogs in terms of dogs, it's a whole learning experience. Because if you look at it how we learn or how today we learn, the textbook way is kind of getting overpassed by the web. Right? Anything you put if you're if you I what I'm trying to focus is more on, you know, overall learning experience.
• 1:22:56
  If we have, let's say, systems, some kind of docs that is, you know, that has a does visual elements that are like, one example I would go is that if you go to educated dot io, one thing I like about them is that if they're explaining you how URL shortener works. Right? This is a recent example. Instead of telling you everything, you have a visual way which you can, you know, play back and forth and then read the, their description. So you it is very relatable experience.
• 1:23:29
  I don't necessarily think this is a The way newcomers come is that they go to Mernestech. They go for, you know, I can't remember. Every framework's like, this is how things are evolving. I think this is a perspective, because I come from that era, like recent era, so I don't come from, like, 10 years back. So I'm this is a new perspective.
• 1:24:01
  I think if we start thinking from, a learning perspective, experience perspective rather than saying that, hey, this doc is perfect because this matches some standard. Of course, it does, but you have to see that every reader is not gonna understand your crisp, like concise language. It must be it might be perfect. But if you look at somebody's trying to get in, they are gonna have a hard time. They might need to read it like 10 time or 7 time, which which is not necessarily desirable.
• 1:24:42
  So if you have a beginner friendly, like, informal language that explains docs, I think that's a very good, learning experience, and that's gonna grow your interest if if in simple words rather than saying that this is the perfect docs because it it is compliant with this standard.
• 1:25:01
  Speaker 1: Yeah. Interesting. Interesting. It's yeah. No.
• 1:25:03
  That I've not gone on edgegrape.0, but it definitely looks, looks interesting. And it certainly feels like there's a lot of new ways to to learn systems out there. In terms of who is, the n o o seven seventy, I think. Are are you are you next?
• 1:25:19
  Speaker 9: I think Ian was first.
• 1:25:21
  Speaker 1: Okay. There you go. Go for it. You are so you're so courteous.
• 1:25:26
  Speaker 5: So, you're currently hiring for a couple of positions that could be considered systems programming positions. As a hiring manager, you're if you're hiring someone who is changing roles from a non a a a a role maybe higher up the sec to this role, what sorts of things are you looking for in terms of someone who is curious about going to that role?
• 1:25:52
  Speaker 1: Like, how would they
• 1:25:53
  Speaker 5: pick up how would they pick up rust, which is likely the language that you're that they're gonna be working in. And potentially, like, you've mentioned a Arduino pet washing coffee maker. If you got examples of projects that people could pick up on the side to be able to, like, brush up their skills in that area.
• 1:26:10
  Speaker 1: Yeah. And I feel I mean, I don't know how number of folks that we have that wanna work for the company, which is great. And the, I and I I I feel that it it's actually giving me confidence that there are lots of systems folks out there because we see so many we have people do these materials where they're talking about things that are meaningful for them or or that they're proud of. It's a portfolio of work, and there's a lot of really, really, really great stuff out there. I think that we've got a lot of folks that are, that have changed directions or are, and I think I do think that what I mean, my my colleague speaks to this as well, but I I do feel that we are looking for, pretty intense curiosity.
• 1:27:01
  And then it you're looking for a kind of doggedness, I do feel. And I think that we we kind of we you know, I I feel Rick hit on this earlier with the resilience. I I feel that systems are will resist being understood, and you need a certain, monomaniacal kind of focus to actually, will these systems into being understood sometimes. So I think we're kinda looking for that as well. But, again, I don't know that we're representative
• 1:27:30
  Speaker 3: You already you know, from the perspective of a hiring manager to a degree already answered this with, you know, Patrick and, and Josh's stories of folks who showed up and contributed in really meaningful ways. And
• 1:27:44
  Speaker 1: And that has been true to Oxnard as well. Where we
• 1:27:46
  Speaker 3: And then Yeah. I think, like, you know, if if if folks were, for example, using something like Dropshot, our our web framework, or,
• 1:27:54
  Speaker 6: like,
• 1:27:54
  Speaker 3: Hubris, our our embedded, operating system. Like, the like, and we're using it and contributing positively, you know, that would, speak volumes and, you know, be an overweight factor compared with, you know, what what they had done. Necessarily. But but part of that is the authentic interest. You know, peep we've we've also had folks show up and say, here are a bunch of clipping nets that we drop in.
• 1:28:21
  And that's not unhelpful, but it doesn't show a depth of curiosity and use. And, you know, it's a different kind of contribution.
• 1:28:29
  Speaker 1: Yeah. No. I think you're totally right, Adam. And I think it's like we're not looking for, oh, you know, you're you're interested in our software projects as kind of like as as a as a sycophant, we are it's we're really looking for that that deep natural interest because it represents more of an alignment on how software should be built. And because we know that, like, the way we're doing things are not a fit for everybody, and that's okay.
• 1:28:54
  And so we, but, yeah, when we do find out those kind of deep contributions just like Patrick's l dot JS back in the day, or or Josh back in the day, you do realize, like, okay. Yeah. This is this is a kindred spirit. And it is it is definitely important. Ian, does that answer your question?
• 1:29:11
  Speaker 5: Yeah. It does. I was just, kind of ruminating internally about how you could, extrapolate the, oxide specificness of that to a a broader, like, okay. Now someone is hire is applying to a certain job for another organization. What might that hiring manager be looking for for a job change And it sounds like there's a common commonalities there might be, you know, contributions to open source projects that that are that are relevant.
• 1:29:41
  And and that level of curiosity to be able to apply, that knowledge in a non trivial fashion or like a in a more direct fashion? Is that would that be an app?
• 1:29:54
  Speaker 1: I think that that would be app and I would also say and if anyone is if if anyone is hiring technologists, please, please, please have a writing intensive process that allows people to demonstrate their their who they are and not just a resume and a cover letter. I feel that I mean, Adam, I don't know if you feel the same way, but I I I know that as I've said before, like, the hiring process that we have is a consequence of having done it grievously wrong in in previous lives. And I can't imagine another way to do it because we've had so many people, I feel, where you look at the resume and you're like, yeah. I don't know. Yeah.
• 1:30:33
  Interesting. And then you look at the materials and you're like, woah. Okay. Wow. Really interesting.
• 1:30:37
  And conversely, you've had people who are like, wow. This resume is amazing. Then you look at the materials, they're like, these materials are a lot less amazing. I don't know if you've had that.
• 1:30:45
  Speaker 3: Yeah. No no disagreement, but I would also say that, you know, having hired at some other companies, we also have a fairly privileged position of that many folks wanting to show up and throwing up what to some might feel like an obstacle of our fairly intensive writing intensive application process, you know, and not necessarily, creating too much of a bottleneck. Whereas for a lot of organizations, it
• 1:31:09
  Speaker 10: is gonna create that bottleneck.
• 1:31:11
  Speaker 1: Totally agreed. I do feel that the the the the two questions that everybody should ask every candidate in writing before I feel, what are you proud of? Talk about something that you're proud of, and why do you wanna work here? Yes. I feel like the the
• 1:31:25
  Speaker 3: Yeah. Totally agree. Give it giving people the space to to opine on on what excites them is really motivating, and then understanding their connection to your company and your mission and your team, absolutely. Couldn't agree more.
• 1:31:41
  Speaker 1: So yeah. You bet. N o o seven seventy. I don't know how to pronounce that. You know?
• 1:31:46
  I I
• 1:31:47
  Speaker 9: that's perfectly fine. I wanted to, like, give the question a small twist, namely, why didn't I end up going to system programming? Although, I I was well in the way. And, basically, I've been into computer science for a long time, did open calls. We have staff structure of interpretation and computer programming, the list version when I was 14, 16 because I found it interesting.
• 1:32:15
  And, then around the age of, like, 17, 18, I heard of operating system family I had never heard before. And the Tron operating system family from Japan, and it's, like, really impressive stuff. They have, like, plans for embedded operating system, desktop operating server operating system, custom CPUs, which are better for the real time properties, which all those had and that kind of stuff. And anyhow So I
• 1:32:42
  Speaker 1: I can guarantee Adam does not know that there is a Tron reference in the in the kernel, does not come up pretty frequently. It is a, this was like a would would compete against, like, p sauce. This is for, like, engine controls. This is like super hard real time stuff. Anyway, go ahead.
• 1:33:10
  Speaker 9: And, I basically was fascinated and dig through all their, like, 60, 80, I'm not I'm not quite sure, English publications because they're Japanese. They don't have that many English publications. But because there were, like, a lot of people working on their complete architecture, with, like, all the different branches I told about, there were, like, quite a little to dig through. And, like, for a project for my school, I boiled all this down into, like, 60, 80, roughly, but something that magnitude of, like, latex pages, which were made to be understandable by a layperson understanding all the terms. And I, like, had to take all the terms from all its different disciplines.
• 1:33:53
  One later in history didn't turn out because, like, US imperialism, but that's beside the point. And, the Japanese people, including the professor which got it started, were, like, that impressed by the questions I asked doing my research to them. And they didn't have didn't have a full scope what I was doing, but they were so impressed that they said, you know what? We invite a person every year to our conference, to annual conference. We we only do embedded stuff, and it's no longer called Itron as it was before for industrial.
• 1:34:25
  It's now, I'm not quite sure what the name is. I forgot it. Anyway, I was there, and life was nice. Shaking hands, was fun. 3 years later, I port their embedded operating system to an open source reimplementation of the SuperEdge CPU architecture, which originally comes from Japan, was made by the people which previously worked on the Tron CPU, and there was, like, a preexisting Itron port I could, like, draw from.
• 1:34:59
  And the open source implementation had basically no documentation. So I went into the VHDL code, and I tried to find out what the device wedges is or try to learn all that from scratch, document what I learned, document what kernel changes.
• 1:35:13
  Speaker 1: Yeah. It's
• 1:35:14
  Speaker 9: okay. What's the compiler changes we have made? That kind of stuff. And, then, I I, like, sent them that and that. Yeah.
• 1:35:22
  That's interesting. We are welcome to have you again, but, like, you will have to pay your own flight and your own stay this time. Okay. That was fine. I managed to do that.
• 1:35:32
  And, then I was there, and, I was, like, in the UNIOT trick or something like that for the conference. And nobody there talked about embedded or, like, one person talked about a payment system for a mobile payment system for buses. Another person talked about smart home other thing. And and at the in the same year, there was, like, a new version of this, kernel release, a new specification because it's all specification based, and then there is 1 default implementation. Anyhow, I didn't know there was a new version.
• 1:36:08
  So I ported so I saw that I almost spotted the old version. And then I learned that, but what was even worse, there was no person there to, talk with me about the embedded stuff I did, about the work I did. Just like, yeah, we have a new version. Yeah. And we have the spec, And the spec also has PDFs available, but, if you want it in in print, it's like $60 or something like that.
• 1:36:34
  So
• 1:36:36
  Speaker 1: what there's an interesting point here though, I think, in terms of, you know, talking about how this one, you know, get engaged. You know, you got an open source project that's being worked on. It's got a company behind it. And one of the a good way that I to me that that highlights of getting engaged is going in where there is Spartan documentation and adding the docs. And that is something that that can be that can be really, represent a lot of hard work that generally is gonna be pretty welcome and is gonna, I think, turn some heads.
• 1:37:07
  Speaker 9: I sort of did that with my, paper and that ended up in the trick. It's published somewhere probably under Springer. I don't know. Anyway, that that was, like, the one thing. And, like, nothing came of it.
• 1:37:19
  I talked with them, they there was nobody to talk about the better thing, but that was not all. I also, like, studied all the VHDL code for these open source documentation, and tried to dig into that, and there was no documentation how the fuck. Can I say, that word here? How does the interrupt controller work? Well, I dug into the VHDL and found an odd, off by 1 error in the, interrupt controller in the, like, nanoseconds time register, where, in the first run through, it will go from 0 to 9999.
• 1:37:54
  And in all further ones, it will go from 1 to, like, 10 to the, 9. So there was, like, a one off inconsistency between this thing. And I, like, approached the people which did the open source implementation of the process and, like, release it as a 12 and said, yeah. I found this bug here. I replicated it on, the hardware you gave me, Sato, you sent me.
• 1:38:16
  And I replicated that bug. It exists. I'm not imagining this. It's a minor bug, but, can I get this upstream merge, something like that? And like 2 weeks later, I said, we had a look at the code.
• 1:38:29
  That part of the code base has to flow out anyway. No, we do not merge your code. No. We will not include your thing in the new tarball release. And that was also it.
• 1:38:39
  And that was it. And, then I just,
• 1:38:43
  Speaker 1: Dan's point that even a small community can be can be not welcoming. That's definitely annoying.
• 1:38:50
  Speaker 9: And then, like, with that experience back in Germany where I'm from, basically, I talked with, someone a few years older than me, which I came to visit our school, what he's doing in computer science. And then, just it sounded like the first one and a half, 2 years will be full of stuff I already taught myself. So, I then kinda knew, what I do as open source, minimal, viable product where it actually can have an impact. My idea was, by popularizing and giving this open source reimplementation of old CPU architecture, which is now out of patent, which has a stoic connection that specific operating system, by, like, using this overlappingness, that connectedness, I can like slightly push the operating system market in that particular Japanese area, maybe a bit in that direction. No feedback at all.
• 1:39:46
  Nothing came
• 1:39:46
  Speaker 2: from it.
• 1:39:47
  Speaker 9: I go into different discipline. I find out how differential equations work.
• 1:39:52
  Speaker 1: Yeah. There you go. Right. Exactly. Well, the yeah.
• 1:39:54
  And so, I mean, it it's definitely not it's not foolproof, certainly. But that's definitely an interesting tale. So, Timon, I think you're next, and I think we're definitely going way over here. So I don't we'd love to to maybe that's gonna be your last comment. Timon, do you wanna
• 1:40:10
  Speaker 2: Yeah. Sure. Can you hear me?
• 1:40:12
  Speaker 1: Yep. We can hear you. Awesome.
• 1:40:14
  Speaker 2: Yeah. My plan was, like, long time ago. You were, I think, talking a little bit about, like, yeah, smaller companies, you know, not really needing systems programmers. And, yeah. I just wanna give you a point of, like, just a lot of the examples here, I think they're, like, around, like, operating systems or, like, anything kind of in that realm of, like, very close to operate kind of systems.
• 1:40:37
  But, yeah, I I come more from, yeah, I guess you couldn't call it media systems. Like, it it's it's a, like, very different abstraction level. So, like, we we build on top of Chorus operating systems. But,
• 1:40:54
  Speaker 1: I thought did you say on top of Chorus?
• 1:40:56
  Speaker 2: Sorry. On top of operating systems. So, like, we we we build on top of the people that, like, operate systems. That's the we we build systems on top of systems for systems.
• 1:41:05
  Speaker 1: Okay. And so so do you view it as system software? I think to to kind of to to end where we started, do you view that as system software?
• 1:41:13
  Speaker 2: Sorry. I didn't catch that.
• 1:41:14
  Speaker 1: Do do you view that as system software? Because I think this is one of the things we're
• 1:41:17
  Speaker 4: talking about.
• 1:41:18
  Speaker 1: Right. Yeah. So I guess Yeah.
• 1:41:19
  Speaker 2: So, like, my my my definition of system software or, like, systems is, like, it's software that is made up of little thing, but on individually, they are useless. And I think the differentiating factor, because that could also be an application, but the differentiating factor is then that this these always little blocks enable a whole different thing, another application, or even another system, to function. And that that I do or did, actually. I switched industries. But, past 9 years, I was making media systems.
• 1:41:55
  So, for example, today, museums, they are very interactive, and, they are can be extremely complicated. So you have, like, 30, 40 different exhibits. That quite often talk to 3 or 4 different other exhibits, because you can interact with them, and then something else happens over there. And you have, like, embedded systems that talk to, you know, a Linux system that does something, and you have a lot of small little things. And, you know, I I actually come from, like, visual effects and, like, from a from a logistic standpoint.
• 1:42:30
  And I kinda grew more into programming and into the space. And at some point, I realized, I'm actually building systems to enable these applications. So, like, because they get so complex, because there's so much complexity in the interactions involved there, you have to write software to actually be able to write the software for the client because
• 1:42:53
  Speaker 1: Yeah. Interesting.
• 1:42:54
  Speaker 2: If you would do that from scratch every time, it's way too complicated. So you need to have abstract away a lot of these complexities into something like a system that, you know, abstracts away some of these things for these, yeah, little implementations that are specific to a client. And that was for a very small company. I wouldn't say it's also you know, there's not a lot of money in that either. And I think if if you reframe a little bit what a system is and are open to different things, if you're not you know, maybe you're interested in embedded systems.
• 1:43:28
  I do a lot of embedded systems as well. And, like, that is can be part of that. But, you know, it's it's a it's a very different type of systems. But the same thinking involved, like, thinking about how things have to interact with each other, how all these little blocks have to, you know, mesh into each other, how how they talk to each other, and how all of these things together make up something that, you know, make things work. I think that's a very similar methodology to an operating system.
• 1:43:54
  Speaker 1: Totally. Well and I think I mean, this this I think this brings us around to a good finishing point because this is actually kinda where we started in terms of the you know, Adam, your your curiosity and rigor that I think seems to be a a theme we've seen throughout. And I think people being curious and then being rigorous in their own thinking, but whether they're debugging a system or documenting a system or reverse engineering a system, being tenacious and resilient, and following that curiosity to the depths of the system feels to be like a a pretty common theme that we're hearing in terms of how people got into this.
• 1:44:30
  Speaker 3: Yeah. Yeah. You know, one last anecdote I want to share was, you know, a a guy I went to high school with, we were both into computers. Like, he hosted a BBS, like, on his second phone line, so I dial up and, and every once in a while, like, his mom would pick up. And I went into computer science, and he went into German studies and humanities.
• 1:44:51
  20 years later, I bumped into him at Berkeley and he, you know, now as like a a grown up with kids decides to go back to school to earn a master's in computer science. Oh, it's interesting. So spend a bunch of time, you know, like, learning a totally new discipline, new field. And and so there was, and forgive me because I've I've been, reading, bowling alone. So I've got the nomenclature of of human capital, which is what he was building.
• 1:45:18
  Speaker 1: I love
• 1:45:18
  Speaker 2: the yeah.
• 1:45:19
  Speaker 1: I love bowling.
• 1:45:19
  Speaker 3: But but then, you know, a lot of what we've been talking about with these communities of whether it's, you know, on GitHub, an open source project, or or live in these conferences, or whatever is human capital and who you know. And so, you know, my buddy got a, got a new job out of out of this, master's program, which is awesome. Not something he was fired up about. But then, another friend of mine, my buddy, Lucas Brian, you're a buddy, Lucas, as well.
• 1:45:46
  Speaker 1: He's got a job for him. Like, I just wanna make sure it's the same person.
• 1:45:49
  Speaker 3: You know, working in the start up, knew the guy, said, hey. I think you can do this job. But, you know, I've met you. I've I've talked to you about stuff. So, and I've evaluated your talents and and goes to the start up.
• 1:45:59
  Start up gets acquired by Apple. You know, it's a terrific success story for him. I'm very excited for him. Really pleased for him. But but also that it's it's that confluence of, you know, learning and demonstrating what you know, but then also making those connections, whether through open source communities, in person, you know, through your own networks of of school and friends and community and whatever.
• 1:46:22
  Yeah. That get you to those. And there's there's probably no silver bullet, but it is it is building both knowledge and connection.
• 1:46:30
  Speaker 1: Yeah. And don't you you can't just rely on the kind of the the cover letter or apply especially if you're coming from a non traditional background or one that's kind of a, you know, a a later discovery that this is as a calling, you've really probably have to emphasize that that human capital even more. Yeah. Yeah. No.
• 1:46:47
  That that is, that's good. Well, that's that brings us right back to to you to we're all just humanists in the end. Right?
• 1:46:54
  Speaker 6: So Exactly.
• 1:46:55
  Speaker 1: Great. Alright. Well, I know we went a little long, but this is a this is an exciting one. A a lot of great stories from from, a lot of folks. So, I I hope this is useful for people who are, hope that there's some tidbits in there.
• 1:47:09
  I know that we, we I think we've spent most of the time just to finding some software was. I'm not even sure we got there. But Nope.
• 1:47:14
  Speaker 3: But close enough. Right?
• 1:47:16
  Speaker 1: Close enough. Close enough. But thanks, everyone, and we'll talk to you next time.