Latest / Oxide and Friends / Solutions Software Engineering with Matthew Sanabria

Transcript

Solutions Software Engineering with Matthew Sanabria · Oxide and Friends

• 0:00
  Bryan Cantrill: Is that better?
• 0:01
  Adam Leventhal: Say more. Say more.
• 0:02
  Bryan Cantrill: Okay. Sorry. Is that better?
• 0:04
  Adam Leventhal: Yeah.
• 0:04
  Chat, what do you think?
• 0:06
  Bryan Cantrill: Chat, what do you think?
• 0:07
  Adam Leventhal: It's a lot better. Not quite as hot.
• 0:10
  Bryan Cantrill: Okay. I was too close to it maybe.
• 0:13
  Adam Leventhal: If you guys wanna hear a nice sounding voice, listen to Matthew. Matthew, welcome.
• 0:16
  Bryan Cantrill: Matthew, you should go ahead and say really anything. Mean, you're just gonna I mean, it's very soothing.
• 0:22
  Matthew Sanabria: Oh, thank you. Is this an ASMR episode now? I don't know what we're doing.
• 0:26
  Bryan Cantrill: It is now. It it it it is now. I we we you kicked it off. Yeah. No.
• 0:31
  We we and and the thing is people are gonna hear your audio setup. You saw someone on Blue Sky is threatening to infiltrate the company to give us good audio.
• 0:39
  Matthew Sanabria: I might have to provide safe passage for that infiltration. We'll see.
• 0:43
  Bryan Cantrill: That's right. It's like, hey. I I got bad news. Oxide was breached. But strangely, they have left audio equipment and instructions on how to use it.
• 0:53
  So I'm not I don't know.
• 0:56
  Adam Leventhal: Instructions, throw those right out.
• 0:58
  Bryan Cantrill: Exactly. So, I understand we have some, some e waste news to revisit. How do we as we kinda go? So I mean Headlines, we have e waste and then maybe a little maybe a ten second CNCF update. We have peace in our time for the CNCF with respect to CNCF NAT's shootout.
• 1:20
  Adam Leventhal: Yeah. So
• 1:20
  Bryan Cantrill: He's sort potentially in the Neville Chamberlain sense. We don't know.
• 1:26
  Adam Leventhal: Yeah. I mean, allegedly, the only reporting I saw on the CNCF was an interview on Runtime News, which I guess is a thing that I get in my inbox and noticed that said that Derek Carlson said now they can have the trademark. We'll figure out our own thing. But I haven't seen it reported anywhere else. There's apparently an AMA this coming Wednesday at 8AM Pacific.
• 1:49
  So maybe we'll find out more. But seems like That could be
• 1:53
  Bryan Cantrill: a hot ticket. That could be a hot ticket, that AMA. Oh, yeah. But yes, Detente. Peace in our time.
• 1:59
  So that that that was a thank you very much for that. That was a good conversation. It was obviously great to have Adam and Rachel on, and getting Eliza's perspective as well. And then, in the e waste department, people have been going through the back catalog and like, hey, are all of the dead gimlets spoken for? The answer is not yet.
• 2:19
  So I think I've got like four of them currently spoken for. So put that in the chat if you want to claim some of that e waste. I think that they, I don't know if they're going to last to the end of this episode, and then I don't know how I'm gonna get them to you yet, but I've got, like, posted through that. Name
• 2:35
  Adam Leventhal: of it.
• 2:35
  Bryan Cantrill: Yeah. Yeah. I'm yeah.
• 2:36
  Adam Leventhal: Okay. Brian, proposal. What about for one of them? One of them only, we open it up to a silent auction benefiting a charity of your choice.
• 2:46
  Bryan Cantrill: That's a great that's that's a great idea.
• 2:49
  Adam Leventhal: It could be over a dozen dollars for that charity.
• 2:52
  Bryan Cantrill: But even a dozen dollars would necessitate a single bidder. I just wanna I mean Oh. Behind it.
• 2:59
  Adam Leventhal: I'll I'll juice stuff if you need to. Don't worry.
• 3:02
  Bryan Cantrill: Oh oh oh, a rigged auction. Yes. Absolutely. I'm sorry. I I I you know, I couldn't see you winking over the audio here.
• 3:08
  Yes. For absolutely a rigged auction. Yes.
• 3:10
  Adam Leventhal: When I step on your foot and say auction.
• 3:14
  Bryan Cantrill: It's like, oh, yeah. Oxite, you do those guys did the the crooked e waste charity auction? Yeah. Yeah. Yeah.
• 3:21
  Those guys. Yeah. I know. You know. Though I I think it's a great idea.
• 3:27
  I I And put it
• 3:30
  Adam Leventhal: on a posted note right next to the others.
• 3:34
  Bryan Cantrill: I threw out the one that said Crooked E waste charity auction. That one surely was I mean, that that obviously is not real. So alright. That's well, enough of our news headlines. Matthew, welcome.
• 3:50
  This is it is it's great to have you here.
• 3:53
  Matthew Sanabria: Yeah. Thanks for having me. Yeah. I should be much louder now for those who are saying I was quiet. Just let me know.
• 3:57
  Bryan Cantrill: You are loud and yet just somehow just as soothing, maybe even more soothing at at greater volume. You sound great. It is great to have you here. You've been you've been in the do you call it the Oxide community, you've in Oxide and Friends before you worked for Oxide, you were definitely someone who was following the company and vice versa. So maybe we could turn it the dial back a little bit And how did you, I mean, stumbled across oxide at some point.
• 4:33
  Was this when you were at HashiCorp or when did you stumble across oxide?
• 4:37
  Matthew Sanabria: Yeah, I think when I was in HashiCorp is when I actually stumbled across oxide, I think one of my coworkers actually told me about it. They're like, hey, this company over here is doing really cool stuff. You should take a look and keep an eye on them. And I was like, what company? And they're like, oh, it's oxide.computer.
• 4:53
  And I was like, wait, .computer? Like, people use other TLDs except for ..com? Like, cool.
• 5:00
  Bryan Cantrill: $9.99 a year, my friend. Yeah. I
• 5:05
  Matthew Sanabria: don't know where the decimal is in that, but I hope it's in, you know, $9.99.
• 5:09
  Bryan Cantrill: Percent, 99¢. Yes. Let's just say that it was much more reasonable price than oxide.com, which they wanted, I believe, €400,000 for. They did not take them off of that. Alright.
• 5:20
  So you and then but you came out to the office as part of OSFC. Right? You were the the with the open source firmware conference. Because what I remember is you taking a selfie with Adam, the the Grimace, the the Right.
• 5:38
  Adam Leventhal: Right. From, our our cover art, may, from the episode talking about Ashley Corporan, the relicensing.
• 5:48
  Bryan Cantrill: The relicensing. Wow.
• 5:50
  Matthew Sanabria: When when was that? Was that 2023 or 2022? Don't remember.
• 5:53
  Bryan Cantrill: It was 2023, I think. I think it was 2023.
• 5:55
  Matthew Sanabria: Yeah. So at that point, I've had known of Oxide for, like, a few years by that point. And I was in between jobs, I think. I was going from HashiCorp to Cockroach, I think. Yeah.
• 6:07
  Yeah. I was going from HashiCorp to Cockroach. Because actually at the Open Source Firmware Conference, I remember the Cockroach compute Cockroach recruiter email bouncing. Like, I I sent them an email, and it bounced. And that's how I found out that Cockroach had layoffs.
• 6:20
  And then I was like, oh. I was like, oh, I'm about to start there in, a week or two. I should probably rethink my life. But I did, ended up joining and staying there for a year. But yeah, that's when I first met the Okay.
• 6:33
  Bryan Cantrill: This is gonna be unfair, but I actually unfair not to you, but to cockroach. I was actually describing for my mom that you were coming on the podcast in your own history. And I was describing that you worked for Cockroach. And my mom was like, that's not the name of the company. I'm like, it is.
• 6:50
  She's well, that's a that's a terrible name for a company. I'm like, I okay. We're doing this again. But so did you with with your own, people who were not in technology, when you told them you were going to cockroach, did that necessitate some explanation? How did you
• 7:04
  Matthew Sanabria: I just I just stopped saying the word cockroach, honestly. Like, even my wife's like, I can't tell people where you work because it's not fun. You know? So I just kept saying a database company.
• 7:14
  Bryan Cantrill: That's really all I said. Yeah. Okay. Yeah. See, I work for CRDB.
• 7:18
  You haven't heard of it.
• 7:19
  Matthew Sanabria: Right. Exactly. But
• 7:21
  Bryan Cantrill: that's how you but I'm not wrong that this was an issue that had to be navigated where your wife is just like, I'm actually like a little sick of having to have a five minute conversation every time I say where you work.
• 7:30
  Matthew Sanabria: Yeah. Yeah. She's just like, oh, my husband works in tech. That's all she ended up saying at
• 7:33
  Bryan Cantrill: that point. That's okay.
• 7:36
  Matthew Sanabria: Even with people that were in the tech community, they were like, what's cockroach? I'm like, you know what? It's fine.
• 7:43
  Bryan Cantrill: It's a database. That's all you need to know.
• 7:45
  Matthew Sanabria: It's a database.
• 7:46
  Bryan Cantrill: Yeah. It's good point. Like a yeah. That's right. He's not a boy.
• 7:49
  I think I think I'm a little worried about Matthew. Am I the only one? He I think he has made up a company that he works for. It's very awkward.
• 7:57
  Adam Leventhal: It's like, what? Honey, why do you keep telling everyone I'm unemployed? It's like, it's just easier. I don't know. Like I'm I'm less.
• 8:04
  Bryan Cantrill: That's right. Just less of a conversation. So you were at, you're a cockroach and you are sorry, excuse me, you were at a database company. And as someone in the chat is saying reminds me of B, it definitely reminded me of Bee. This is Matthew Bee was John Blake.
• 8:21
  I this could be no. Because when I was trying to decide between working for Bee or working for Sun and telling, you know, my mom again, I'm like, I am this company Bee. She's like, wouldn't B, B what? I'm like, no, B. She's like, B Bumblebee?
• 8:38
  I'm like, no, B like, 2B. She's like, 2B. I'm like, yes. And she's like, that's a terrible name for a company. And I'm like, that is a terrible name for a company, actually.
• 8:49
  It is it did kind of affect my I'm like, you know, I don't wanna be having this conversation for three years.
• 8:54
  Matthew Sanabria: So basically ask ask your mom for, like, feedback on naming is what I'm hearing.
• 8:59
  Adam Leventhal: That's right. Do you do you remember when Apple when Apple end up buying NeXT? And they but they had been in conversations with Bee and Steve Jobs came in and said that we went with plan a and not plan b. Anyway, that was one of problems. Yeah.
• 9:13
  Bryan Cantrill: And and Matthew, you're exactly right. And let me tell you, one of the very first people in terms of, like, run a name by like, just like anyone's mom or whatever, like your wife, spouse, whomever, like, people that aren't in tech. Right? You wanna, like, run a name by them. And I did run OXIDE by my mom and my mom was very strong proponent of the name OXIDE.
• 9:32
  She's like, I really like the name. Nice. So there we go.
• 9:36
  Matthew Sanabria: Yeah, exactly.
• 9:37
  Bryan Cantrill: There's a lot of merit to that. Okay. So you are at CRDB, and meanwhile, Oxide is like we're like developing, I mean getting our product in market, and I know you were kind of following the company, and then, you saw that we were kinda starting up a new function with respect to solution software engineering.
• 9:58
  Matthew Sanabria: Yeah. Yeah. So right before I joined CRDB, like I mentioned, I got notified of the of the layoffs that happened.
• 10:05
  Bryan Cantrill: Like that.
• 10:05
  Matthew Sanabria: And I was Yeah. Yeah. The database company. So I got notified of the layoffs while was at Open Source firmware conference, and I can't remember if that was before or after all things opened that same year. I can't remember which one was which.
• 10:20
  Right.
• 10:21
  Bryan Cantrill: But yeah. Yeah. Yeah. Yeah. Yeah.
• 10:22
  Matthew Sanabria: But either way, I saw Oxide, like, twice in the span of a month. Right? One once at Open Source Former Conference and then got to visit your office and hang out with people and take the nice selfie in front of the beautiful beautiful image. And then again at All Things Open where I met, I believe it was Travis and Steve, I think, there, Klabnik. And I got to talk with them and just kinda piss out the rack and get, like, you know, hands on keyboard of the actual console and the API.
• 10:47
  And I made an API key for myself and took it to my machine and kept playing with it. So I'm sorry.
• 10:53
  Bryan Cantrill: Yeah. Can I tell you my take from the other side of that? So you are at CRDB, you're at unnamed database company. And Travis comes back to that and he's like, man, Matthew was, like, running the booth. It was like, people would come by and, like, they would ask, like you know, he he would ask a question and he would jump in and and I was so it was really it was great.
• 11:14
  It was fun for you to actually get on the actual thing and and start playing around with it.
• 11:18
  Matthew Sanabria: I I did get a t shirt out of it. So it was a good. It was a fair trade, I think.
• 11:21
  Bryan Cantrill: Very good. Yeah. Okay. So then so fast forward then. Did you and the intersect then with Oxide?
• 11:33
  Matthew Sanabria: I think at that point, I was talking to Travis a bit and I was like, hey, like, I like what you're all doing, you know, what are you what's your hiring plans, blah blah blah. And just kind of asking, you know, what your roadmap and for hiring and for the product looks like and where you're at with things. I think you were I think you were just starting to sell the Right? When when? Late twenty twenty three?
• 11:51
  Yeah.
• 11:52
  Bryan Cantrill: Yeah. I mean, yeah. The the the that's right.
• 11:54
  Matthew Sanabria: Yeah. So you were just starting to sell the rack and then Travis was basically like, you know, we're gonna be hiring soon. Like, keep keep in mind. Then I was like, oh, well, I just signed an offer to go to Cockroach, and, like, I would hate to rescind. I'll I should see out the the offer at least and, like, start there, and we'll talk in, like, February or something, I said.
• 12:13
  We'll we'll talk later, I said I said to to Travis. And later later definitely came. Later definitely came. I was like, you know, the my time at Cockroach was was fun. I met a lot of good people, but I ultimately wasn't the fit for me.
• 12:27
  The SaaS the SaaS database market is probably not my favorite market. Was always interested in more on premises computing and that style of of work. So I reached out to Travis and I was like, hey, like, what's up? You know, like,
• 12:40
  Bryan Cantrill: what's what's up you're hiring? Matthew, I I don't I hope it's not revealing too much. I'll also say that they had an RTO mandate that was, like, really not uplifting. Is that a fair statement?
• 12:50
  Matthew Sanabria: I really Yeah. You know what? Why did you have to bring up past trauma? I was having a great day today. Know?
• 12:56
  Bryan Cantrill: I was
• 12:56
  Matthew Sanabria: having such a good day. No. You're not wrong. So we were we were, like, mandated to come in three times a week if you were in if you were within 40 miles of the office or something like that. And I was within 40 miles of the of the New York office.
• 13:09
  Bryan Cantrill: Okay. I'm like, yeah. You like, hey, honey. Well, I need to go house shopping for like, we I need to move 42 miles away from the office.
• 13:16
  Matthew Sanabria: Like, I actually wanna get I
• 13:17
  Bryan Cantrill: wanna wait for did you really oh my god.
• 13:20
  Matthew Sanabria: I I actually Oh my god. I'm so sorry. Considering using my family members', like, addresses to, like, get outside of that.
• 13:26
  Bryan Cantrill: Be further away from the office. Yep. Oh my god. Oh my god.
• 13:30
  Matthew Sanabria: And it was like an hour and fifteen minutes one way. So my commute and that was with with no traffic or no nothing.
• 13:36
  Bryan Cantrill: Brutal.
• 13:36
  Matthew Sanabria: Yeah. Very, very bad.
• 13:38
  Bryan Cantrill: Brutal. Brutal. Yeah. Anyway, so those contemplating RTO mandates, I mean, I'm I I hopefully the bloom is off that particular rose, but they really are very bad news. And Adam, this is when we ring the bell to a link to our previous podcast on RTO or GTFO on our personal perspective on RTO mandates.
• 13:56
  Anyway, so yes, Matthew, sorry.
• 13:58
  Matthew Sanabria: Yes, I think we're right around probably what this time last year, actually. So I think we're in May, June ish '20 '20 '4. And I reached out to Travis and I was like, hey, like, you know, what's going on with with hiring? Like, I'm I'm thinking I don't think I'm gonna stay at Cockroach long term. Like, what do you what do you what's your roadmap?
• 14:19
  And then that's when I got access to the preview silo because I was gonna use it internally at Cockroach to, like, run Cockroach on on Oxide to see, like, performance and whatnot. Yeah.
• 14:28
  Bryan Cantrill: Yeah. Interesting.
• 14:29
  Matthew Sanabria: Yeah. Cool.
• 14:29
  Adam Leventhal: And to be clear, a silo is our mechanism for multitenancy, and our previous silo means we hand those out to prospects who are considering Oxide. So it's a way for people to evaluate the console and the API and the tooling and all that stuff without before they bought the rack.
• 14:47
  Bryan Cantrill: Exactly.
• 14:47
  Matthew Sanabria: Yep. So I I got access to one of those, and it was I was gonna start using it. And then it was just funny how timing works. Like, around this time, I was notified kind of internally at Cockroach that they were gonna change their license. And I was like, Oh, we're doing this again.
• 15:03
  Like, HashiCorp did this. Cockroach is gonna be like, Okay, we're doing this again. And then between that, between the commute, and between then my wife wanted to, like, quit her job at the chocolate factory and start her own chocolate business, like brick and mortar. And between all of that happening, I was like, I can't be going one hour and fifteen minutes into the city three days a week or whatever, like one way. Right?
• 15:24
  Two and a half hours of of every day. Like, this sucks. So that's kind of all prompted. And I looked around and I was like, who's even hiring remotely anymore in, you know, in this in this year of our lower twenty twenty four at the time? And I was like, who do I even wanna work for though?
• 15:40
  Like, what company do I even, like, care about? You know what I mean? And that's where I was like, wait a minute. I know one company. It's a long shot.
• 15:47
  And then that's when I reach out to to Travis. My wife is with Mo Wonka. Yeah. Kind of.
• 15:53
  Bryan Cantrill: Yeah. That is awesome. And the oh, okay. And again, I don't wanna personalize this, but you know, you're at HashiCorp, HashiCorp changes its license. You go to an unnamed database company, it changes its license.
• 16:07
  Like at what point Matthew, do you take a long look in the mirror and ask yourself some tough questions of like, am I the monkey's paw of open source? Do I mean, and maybe now is a good time to have a message to our community, Adam, about our
• 16:24
  Adam Leventhal: Important update. Important update to our community.
• 16:28
  Bryan Cantrill: I mean, Matthew, don't know. There's something about something about this guy. He just like, you get him on board and you just gotta relicense it.
• 16:35
  Matthew Sanabria: I don't know if you checked the PRs repos lately, but we're well underway. Okay? We're
• 16:40
  Adam Leventhal: a j no. Just kidding.
• 16:42
  Bryan Cantrill: Oh, god. Okay. Okay. That was fair. I deserved that, but it definitely gave me a little bit of a fight or flight reaction.
• 16:48
  But I can
• 16:49
  Matthew Sanabria: see you switching right now. You're just like, wait. For real?
• 16:52
  Bryan Cantrill: No. I know. Know. Absolutely totally. So the and then at that point, because I think we had the solution software ensuring JD up.
• 17:00
  Right? I mean, it's the and I'm under is my understanding by the way that this is a meet cute story. Adam, am I the last person on the planet to hear that? Do you know that term meet cute?
• 17:09
  Adam Leventhal: I do know it. Yeah. Yeah. Absolutely. We yes.
• 17:13
  Comes up You've for a long time. Comes up surprisingly often in my house. I better not unpack that.
• 17:21
  Bryan Cantrill: I let's just say that I had a you learning of GitHub moment when someone asked me about a meet cute story. I like making them repeat it like six times. They're
• 17:31
  Adam Leventhal: just like, oh, this is alright. Is this like sliders? Like tiny burgers? Like what's the what is the cute meat? Because it sounds delicious.
• 17:39
  Bryan Cantrill: Are you saying meat with two e's or one e? This what a meat I mean, what what are we talking about? Like, Meat cute. I'm like, you have invented that term. It's like, I assure you I have not.
• 17:51
  Anyway, this is our so this is the oxide Matthew meet cute story. Because I think like this is right about the time we put the job description up Matthew, and which honestly was obviously a great fit for your disposition and aptitude. I think people know this about us, we take the job description really seriously. Like we really want to put a lot of time into the job description to describe the job. This is this, like, very novel idea that we have.
• 18:24
  You don't want ten years of experience,
• 18:26
  Matthew Sanabria: for something that was released yesterday? What do you mean?
• 18:28
  Bryan Cantrill: Exactly. Well, I think a lot of people don't in terms of, like, we want to give you a feel for like what it's going to be like in the job. And we spend time on that and I think it's important that when people read it, like, you should be like, yeah, I can like I can see myself in this role. And you could see yourself in this role as we described it.
• 18:51
  Matthew Sanabria: Yeah. I mean, my primary point of contact thus far was Travis. Right? And so like I was getting I was getting the perspective from Travis of what he would like like this role to be, you know, for his sales sort of needs. And then I was getting the, you know, secondhand feedback from Travis from you.
• 19:06
  Right? Like, you were talking to Travis and Travis was relaying it back to me a little bit of what you would like this role to be. And like, I was like, you know what? I want this role to be that too. Like, I align with this and I think that this will help me use more of my skills.
• 19:19
  So like, part of the problem with most of the roles in general, like at any company, is what we said before about job descriptions not accurately describing what the role is. And you don't really get a chance to use maybe all of the skills you have as a person, Right? Like, we we all have many more skills than just coding or And I felt like in previous roles, I was only using a subset of my skills. Whereas in this solution software engineering role, I could be using not only my tech skills but my personal skills and some writing skills and, like, I could use all of those skills, you know? And that's why I felt like it was just a good fit for me personally.
• 19:57
  Bryan Cantrill: That's interesting. Okay. So so tell me more about that. That's really interesting because you mean yeah. I mean, this role, solution software engineering really, I mean, it really is sitting the confluence of software engineering and having a solution.
• 20:11
  We're actually making the rack, don't know, do something that's like, I guess useful for someone who's trying to solve a problem. But it does require a lot of soft skills and a lot of, like, hard tech skills. So yeah. That's unusual from your perspective.
• 20:27
  Matthew Sanabria: Yeah. Yeah. It it's someone asked a question earlier saying, like, what sort of solvent do we use for our solutions for this role? And I, like, reply saying the tiers from previous roles. It's not totally untrue.
• 20:39
  Right? Like, in previous roles, you kind of you can probably be in a place where you're kind of pinned, right, where you only do this function and no more. Oh, you wanna talk to the customer and find out how they're really using your stuff? You can't do that. You have to go through some PM or some other process.
• 20:54
  And then by the time the feedback gets back to you about how they're actually using your product, it's been translated or they didn't ask the right questions, and you're just you're talking past one another. So, like, a five minute chat between two engineers can, like, unblock hours or weeks of work, but instead we put it behind two, three hours worth of meetings and email trains. It's like, why why do we do these things? And that's the sort of stuff I was experiencing in previous roles where I was like, I can do better than this if you just let let me out. You know what I mean?
• 21:22
  Like, let me like, yeah. Like, take take take the collar off of me. Like, let me off the leash. It's okay. I'll be fine.
• 21:29
  But, yeah, it's it's just weird. It really is.
• 21:31
  Bryan Cantrill: I mean, and I know people know we ask this a lot. When you have you been unhappiest in your career and why? And, I mean, they all basically rhyme and it's all basically like people were trying to get the right thing done and they were being actively prevented from getting the right thing done. I mean, you're just like, it is really sad how often you have like people actually wanna get the right thing done. You just need to get out of their way actually.
• 21:53
  You don't even need to, you don't need to OKR them or MBO them or do this other crap. Actually if you actually just get out of their way, they would actually have the right thing. It's just like, oh, God.
• 22:02
  Matthew Sanabria: And not only would be the right thing, it would be done faster. It would be better than what you would do normally if you just let people do the job that they're supposed to be doing. You know? Like, if you actually cared about customer satisfaction and making sure you're building the right things for right customers, why are the people that are building the things and the customers so far removed from one another? Why are they not talking to
• 22:22
  Bryan Cantrill: each other? So why do we think that is? Because I actually I mean, why is it that this is such so common where customers are where these things are stratified?
• 22:36
  Matthew Sanabria: I don't know.
• 22:37
  Bryan Cantrill: Don't know.
• 22:37
  Matthew Sanabria: I don't know actually.
• 22:39
  Adam Leventhal: Yeah. I mean, what I've seen is that I think that you've got yeah. I've complicated about feelings about product management. I think there can be extremely, extremely good valuable product management, and there can be product management that is actually causing active harm. And I think one of the emergences of product management, I think one of the ideas is engineers should do the engineering and experts should talk to the customers to be able to infer the the real needs and translate them for engineers.
• 23:10
  And actually some of the I've worked with a few great product product managers and they they do that great. Like, they aren't a filter. They aren't a obstacle. They aren't impeding communication. They're facilitating it.
• 23:23
  They're getting at the root of the problem and so forth. But that is a minority, a very small minority. But I do think that this whole there's a concept in executive management where you say, we want this delegation of responsibilities, and we don't want engineers popping in and out and, you know, engineers who we don't who we do and don't trust variously to have those kinds of sensitive potentially sensitive conversations directly with customers.
• 23:49
  Bryan Cantrill: So do you think the fear here is that if we allow people kind of to talk directly like that, then they're exactly what was it just like when we're talking about the return to office mandate about the, I feel like that is coming from the place of fear where executive management doesn't know its own role. Do you think it's like, oh god. If I allow if I allow these two groups to just directly interact, like, what's my job? It's like, I don't know. To facilitate that communication and get out of the way?
• 24:16
  Is that like, are you not comfortable? Like Yeah.
• 24:19
  Adam Leventhal: I think it's great. I think I think there is that some of that fear. I think there's, like, kind of lack of control too. Right? Like, if I'm if I'm running the conversation, I know good or bad, I am in control of how it goes.
• 24:30
  Whereas if I hand that over, I think that I mean, of control is scary in lots of different contexts. So the but I think you're right that fear is probably at the core of that.
• 24:40
  Bryan Cantrill: Well, and I think it's usually comes to lack of control because it's more of the, I go back and back again and again to the, like my favorite passage from The Solve a New Machine where Tom West is bonding the team with trust even though trust is risk, which I think is like this great to me was kind of this galaxy brain, like, insight from Tom West that trust is risk because I sometimes I struggle to understand, like, why are you not trusting these two people to talk to one another? It's like, oh, because it's oh, it's risk. Right. Oh, okay. That's weird.
• 25:12
  Adam Leventhal: Yeah. Brian, this all reminds me of a conversation that you had with a customer back in our Sundays. We were we were taking the DTrace roadshow out, and I think we were both visiting a customer. The the salesperson had brought us in. Given this demonstration about how DTrace can make their application more efficient, you know, we found some performance wins.
• 25:33
  And the customer, I think, had an insight like, oh, wait a minute. So are you saying that with the same sun gear I've already bought, with D Trace, I can make my stuff go faster and not need to buy more sun gear? You've got the you've got the sales guy in the back of the room, like, melting, no. No. No.
• 25:52
  And you're kinda like, yeah,
• 25:54
  Bryan Cantrill: I think so.
• 25:54
  Adam Leventhal: And like the sales guy in my recollection, this might be wrong, burst into flames in that moment.
• 26:00
  Bryan Cantrill: No, the only time I actually, in the defense of Sun sales, they would grip the chair, I'm sure. But there was always, I mean, was one of actually Sun's great virtues that we very much have tried to recreate at Oxide that Sun always believed that if you win the customer's trust, good things will come from that. That if you that yeah, maybe you I allow you to go much more efficiently. So yeah, maybe on this purchase maybe you do buy less of it, but actually in the limit, you trust us. And I think that the reason we did not just flat out go out of business in the .com bust is because customers did trust us.
• 26:40
  They actually wanted us to like I mean, it was kind of sad where customers were like, come on little guy, like don't die. I actually need you to be like, I I kind of need
• 26:51
  Matthew Sanabria: you
• 26:51
  Bryan Cantrill: here. And the we had an in particular part of my kind of the inspiration for did you do you ever deal with the the MDE folks, Adam at Sun, market development engineering?
• 27:03
  Adam Leventhal: No. I don't think so.
• 27:05
  Bryan Cantrill: They were great and I realized kind of in hindsight that I think like I've not seen that role anywhere else. Like I'm not sure that role existed before or after Sun but I loved the and Matthew, to kind of your point about like using all of your skills, it was both like the market and engineering. And I mean, one because I did you ever deal with those folks? They they loved DTracing, needless to say.
• 27:34
  Adam Leventhal: Oh, yeah. So the are these folks kind of in marketing doing engineering activities to, demonstrate the capabilities of of, you know, all of what Sun was producing?
• 27:44
  Bryan Cantrill: They yes. They are like, I guess, putatively in a marketing org, but they were they were technical going to ISVs doing engineering work to actually integrate this stuff and to make it working and to make it work and like saying for for. And they it was like it was a great like confluence, Matthew, get your point about all these skills. It's a great confluence of, you know, a bit of evangelism, a lot of empathy and listening to folks, understanding what customers needed, understand what ISVs needed and then like the actual like technical work to go like go make it work. Right.
• 28:22
  And I went looking for like, I'm like where did that term come from? Where did it go? And I'm like, I can't find
• 28:28
  Matthew Sanabria: it outside. So I'm
• 28:29
  Bryan Cantrill: not sure like that one may have got down to the ship.
• 28:35
  Matthew Sanabria: Yeah. I mean, PM thing is interesting. Like fear, I think, is a big motivator of why people are afraid to let engineers and customers talk to one another. But at the same token, like, I've been with bad PMs. I've been with great PMs.
• 28:49
  And I agree with what Steve was saying before. Like, you kind of shield that you kind of like, of act like an API and connect teams together essentially. And I agree. But what I wanted to get at there with the whole PM kind of talk is customers want to know that the human that they're talking towhether it's a PM, an engineerit doesn't really matter who it isthat they empathize with them, right? That they've actually used the product that they're using and can see it from their customer's perspective.
• 29:16
  Because there's something really nice when the customer's like, Oh, you get it because you've done it too. Right? Not just like you're gonna you're gonna, you know, forward my message along like the the postal service. Like, you know, you actually did it. You know what you're talking about.
• 29:30
  And there there's a there's a trust there, and I think that's powerful.
• 29:35
  Bryan Cantrill: Totally. When I also feel that, like, it it allows because you are now, like, seeing how this thing is being used. I I think it's it's really hard to build something that you yourself are not using. Yep. And which we're all engaged in at some level, right?
• 29:51
  And I think that like you've got to compensate for that by making sure that you are getting folks as close to kind of that. I mean, I love the fact that we use the oxide rack internally. It's great, right? And that's not like the beginning and end, but it is really important to get that empathy for how this thing is being used. And you get a lot of folks, you're just like, they're building something that is totally being thrown over a wall and it shows.
• 30:17
  You're just like, am I the only one using this thing? I mean, am I Microsoft does Microsoft use Microsoft Teams? Serious question.
• 30:26
  Adam Leventhal: The real question is does the Microsoft Teams team use it? Because I'm sure they force everyone else to.
• 30:31
  Bryan Cantrill: God, can you imagine? God, can you imagine if my Teams team did not use like, would burn that place to the ground. I'd there'd be just you'd think it'd be a revolt.
• 30:39
  Matthew Sanabria: It has to come from the top two, though. Right? Like, it's it's a difference. It's it's I've been at companies where they have a product, and then they don't encourage employees
• 30:48
  Bryan Cantrill: to use it. They say, Oh yeah, we sell XYZ. You know, you can use it if
• 30:52
  Matthew Sanabria: you want, but whatever. Be mindful of your costs or be mindful of this. And it's like, why? Don't you want your employees to be empowered to know that the product acts this way and that way? Whereas here at Oxide, it's like, nah.
• 31:04
  Go go throw your personal project on the rack right now. Go ahead. Have fun. Let us know what breaks. Oh, wait.
• 31:09
  That weird thing broke? Woah. That's weird. Let's fix it. And it's like, it comes from the top here, so it's encouraged.
• 31:14
  And then now everyone feels empowered to use a rack. It also helps that we have an on premises product that's not chewing up some cloud bill somewhere, right, and and increasing spend. But still, it does the culture is here, which is nice.
• 31:26
  Bryan Cantrill: Yeah. That that's awesome. So the okay. So then coming on board, Matthew, I mean, we've got a there's a bunch of work to do. So and a lot of of kind of of opportunities.
• 31:39
  So what are some of the the kind of the specific things that you've tacked into since coming on board?
• 31:46
  Matthew Sanabria: Yeah. So to paint a picture, I started in mid early to mid November of twenty twenty four. And that's like right with the holidays and everything. Right? So I spent really the first maybe three ish months kind of just taking a lot in, right?
• 32:00
  Just understanding more of the product, reading some RFDs, making connections internally with employees and understanding where everyone works and what they do and using the RAC, right? Deploying Kubernetes to it, deploying my own applications to it, just getting familiar with it, reporting issues, doing all that stuff. And that really helped, right, just taking some time to really and everyone kept warning me like, hey, there's a lot of work. Take your time to learn learn the product, learn everything, get your bearings. There will be there will be much work.
• 32:28
  Don't worry. Don't you don't have to worry about jumping, hitting the ground running. Then I kind of found, like, my my my opening, so
• 32:35
  Bryan Cantrill: to speak. Right? Because you don't you don't really know where
• 32:37
  Matthew Sanabria: you're gonna start working your first day or your first couple weeks or whatever. You don't know what what you're gonna write or what you're gonna do. But I started seeing, like, email threads come in from customers or or prospects having questions. And I'm like, oh, wait. I know the answer to that.
• 32:51
  I just, like, did that yesterday. Like, let me let me let me reply to that and let me update the docs and oh, wait. That that salesperson needs someone, a technical resource for a call? Yeah. Let me help there.
• 33:01
  Like, oh, there's a bad part of this code. Let me submit this pull request. And I just started kind of slowly kinda like water. Right? You just fill the cracks a little bit.
• 33:09
  Where you see a crack and you can fill it, you just fill it. And that helped me learn the product. That helped me understand, like, what how customers are using it and so on and so forth and make connections with people at Oxide.
• 33:20
  Bryan Cantrill: That is awesome. I mean, it's very gratifying to hear too, where you feel empowered to be like, hey, if there's something like, I can go do that. Like, don't know, I'm kind of waiting to like, for someone to tell me or to task me on that or what have you. I can just like, I can see that this needs to be done and I know how to do it. So I'm I and it's always also a great feeling.
• 33:36
  You're like, oh, wait a minute. Like, it's totally like the Jurassic Park thing of, you know, the old Unix, I know this. I'd be like, oh, hey, know this answer. I feel good about this one. I can actually answer this one, which yes, my bad.
• 33:46
  Matthew Sanabria: Yeah. Was weird at first because again, I've come from previous roles where this sort of behavior and responsibility is not encouraged. Right? So I'm not gonna lie to you. The first maybe three, four months, I was like, can I even do this?
• 34:03
  Is Brian gonna come yell at me? Like, you know what I mean? Like what's gonna happen here? Like am I even empowered to do this thing? Can I just push the button or no?
• 34:11
  Is someone gonna be in my DM saying, Matthew, you did this wrong? Because in the past, I've had those experiences. So learning, kind of like resetting my mind that that that corporate blocking structure is no longer there and I can just do the thing and be trusted to do the thing was a hard thing for me to get over, honestly. And I I probably just recently got offered that.
• 34:30
  Adam Leventhal: Is this is this the smallest company that you've worked on?
• 34:35
  Matthew Sanabria: In a long time. Yeah. But I've worked at Wicker, and it was, like, 30 people when I worked there early in
• 34:39
  Bryan Cantrill: my career.
• 34:40
  Matthew Sanabria: Got it. Got it.
• 34:41
  Adam Leventhal: Yeah. Because, you know, we, structure is not something that we have in abundance, obviously.
• 34:49
  Matthew Sanabria: Feel like I'm a position. What do you mean? Flat is a structure. I don't know what you're talking about.
• 34:54
  Bryan Cantrill: Yeah. Exactly. Yes. Thank you, Matthew. Somebody gets it around here.
• 34:58
  I'm gonna mark that on your OKRs right now.
• 35:01
  Matthew Sanabria: Perfect. Thank you.
• 35:02
  Bryan Cantrill: Matthew exceeds performance, so I'll say. Yeah, I would say that we are, I would say that we, because I also do think Adam, I mean, yes, that I mean, you say like structure is not a strong suit. Think we, you wanna connect people to the problems that need to be solved and then help people answer the question, am working on the right thing? Do I have the right priorities? So we actually really do need that structure.
• 35:32
  It's just not siloed, traditional or hierarchical. Like, I mean, God, like Megan and Angelopins are so important for helping us figure out what needs to be done and helping us connect what needs to be done. And it does mean that it's like it's kind of harder because I do feel that for everybody it's like kind of I want to make sure that I'm working on the most important thing. And you can absolutely get paralyzed on that. Right?
• 36:00
  Like I tell people like, don't overthink it. Like if there are eight things that are important, you can just pick up one and do it. Like you don't need to worry about like the stack ranking the eight important things because you can go out of your mind and you can get paralyzed.
• 36:12
  Adam Leventhal: But
• 36:15
  Bryan Cantrill: I do think that the so you need people who can like help facilitate that, but it is more of a it's facilitation as opposed to I mean, and I think also we're pretty good at like whether, you know, when it's pretty obvious when something is the highest priority at the company.
• 36:32
  Matthew Sanabria: Yeah. But
• 36:33
  Adam Leventhal: but but you want a healthy I mean, I think healthy culture generally, but in startups even more acutely, they wanna tell people what they should be doing, not what they can't do. And I think that's even more interesting to start up where like, you know, my I remember when I when I first joined a startup coming out of Sun, you know, Sun, everyone was standing kind of shoulder to shoulder for any given task. There were two people grabbing it. And And when I joined First Startup, like, it was, you know, for a given task, there was nobody whose responsibility it was. I think initial the the that was most emphatically made to me that point when I we decided to, like, open an office in San Francisco.
• 37:12
  And then I was like, alright. Now what, boss? Like, who does that? And they're like, you do that. Like, you find the office in San Francisco.
• 37:18
  I'm like, oh, okay. I get it. I now I understand how startups work. Like, whoever kind of mentions it, like, it's probably their problem.
• 37:27
  Matthew Sanabria: It's that's right. Pretty factual. Yep. And that's kind of what I was seeing, right, when I was when I was here. And I was starting to see, like, oh, okay.
• 37:35
  This customer prospect has this problem. There's these open items that weren't yet addressed because they weren't as part like, as priority until you, you know, until I came on. And so I started getting more of those those list of things to do, like Kubernetes integrations, customer, like, help with with sales calls and such. Oh, we we need a plugin for this or a plugin for that. And I started to kind of see what was the priority.
• 37:58
  Right? Like, the Go SDK needed to work. The the Terfan provider needed to work. We need a Kubernetes plugin for Rancher. Da da da.
• 38:04
  And then pull requests were getting dumped into my lap. Like, hey, nobody has the bandwidth to review this Rancher thing. Can you go review it? And then now I'm doing that. And so I got this big list of things that kind of were important and I had to prioritize them.
• 38:17
  And that's when I Angela and and and Megan are great, obviously. And as you as you said before, that's when I was like, I have too much crap written down. I need to, like, organize this stuff. And that's where I put my sort of PM hat on and I reach out to Megan. I was like, hey.
• 38:33
  How do we do things here like this? You know, I I need to just see what my work is so that I can know what eight things I have to pull one from. Right? And that's when I started, like, putting things into into GitHub projects and organizing things and getting I I now have, like, the next this quarter up to the next two quarters, like, mapped out of what I want this team to do. Right?
• 38:53
  And it took a minute to get there, though. Not gonna lie.
• 38:57
  Bryan Cantrill: Totally. Well, and then yeah. But then you have what you end up having is, like, a very organic like, you I mean, the things you're working on are based conversations with real people trying to solve. There's no question about those things being important because we can connect them all to like actual use cases, which is great.
• 39:15
  Matthew Sanabria: Yeah. Yeah. I mean, when you have customers that have a rack in their data center or wherever they have the rack and they have they're blocked by some pull request or some feature that you're responsible for, that's that's obvious that that's we have to work on. Right? Like, whatever long term project you're working on or whatever, you kinda have to to bump it up a little bit or sorry, to, like, you know, preempt it a bit and work on the thing the customer needs.
• 39:38
  And that's fine because it keeps our customers happy and, you know, keeps them coming back to us. And and we get really good feedback from that too. Right? It's It's not just, Oh, drop this, drop whatever you're doing because customers said jump. That's not what it means.
• 39:52
  It's really just we want to make sure that the customers aren't blocked and successful and then they turn around and say, Listen, nobody is treating us this way. Nobody is even, like, putting our GitHub issues out in the public so where we can see them. Nobody's giving us updates on them, much less an engineer actually that's gonna work on it doing the updates. So, like, just that treatment of the issues and the pull requests and reviewing them and talking to the customer, like, in real plain language, it just makes the customer turn around and be like, yeah, we like you. You're the only one solving our problems.
• 40:24
  Bryan Cantrill: Well, just it turns out it's, like, kinda nice to hear. It's like it's very, like, self reinforcing where you're just like, that's that's great to hear. This is really appreciated. And it's very uplifting. And and, of course, from their perspective, it's it's uplifting to have and I think we've all lived the other side of that.
• 40:40
  I think it's the other thing that's important, Matthew. So, like, we've all like, know, the vendor that's not treating us very well or we're kind of chasing this kind of opacity or we're not, the problem's not being taken seriously or I mean, you say, it's like the solution is the solvent is coming from our tears from having in the side, know, we call it all your scar tissue based development. But we've kind of seen the opposite approach, it just does not work. Or it's very frustrating anyway to be on the other side of it. So what they would ask about Matthew because the materials for this position are slightly different than we ask, than we've asked.
• 41:21
  And we do like a little, I mean we always have the oxide, we call the oxide questionnaire asking you when you've been happiest and why and unhappiest and why and so on. But we did tweak the materials a little bit for this in terms of the integration samples. I wonder if you wanted to talk a little bit to that and kind of what your perspective is on that. Because now you've been on both sides of this, you've been both in as, you know, applying to OXIDE and then now reviewing materials on the other side. What's your perspective on that?
• 41:50
  Matthew Sanabria: Yeah, yeah, that's a really good question. And for the listener, I've reviewed about, I think, 50 materials so far since being at Oxide, which it does a lot of reading, right? Materials are generally on average probably around 10 pages each. So it's a lot of reading and then writing on your review. The integration section for these materials are primarily around tell us times that you've integrated systems, right?
• 42:13
  It can be and it's kind of open ended on purpose so that you can say, you know, have you integrated small low level systems? Have you integrated large systems together? And but we're basically looking for can you make two things that were not designed to talk to one another talk to one another and in a way that it solves a problem so that the sum of the parts are greater? You know? And that's the key there.
• 42:37
  Because a lot of people are like, Yeah, I wrote software to make HTTP requests and I got the response and I did that. Like, great. What did that do? Right? Like, what sort of problem did that solve for people?
• 42:46
  Did that, like, reduce, you know, some sort of, like, latency? Did that sort of, like, enable the customer to move quickly? What did you do there? What was the problem? And people forget to kind of cover that part in the materials where it's like they don't focus enough on what problem they're solving for their audience.
• 43:02
  They just say, Yeah, I took system A and system B and they're talking. It's like, great. Now they're talking. We're all talking. You and I and Adam are talking.
• 43:08
  What does that mean? Right? Are we providing value or not? And that's where where it kind of falls down
• 43:13
  Bryan Cantrill: for a ourselves because, you know, exactly. Sorry. Self review. I don't know. I need Audio needs improvement for sure.
• 43:20
  Matthew Sanabria: I keep getting totally
• 43:21
  Bryan Cantrill: Keep getting that on my performance review. That's really interesting, Matthew, actually, that you because you're focusing on a this is gonna change the way I actually read materials for this because that really I mean, I'm embarrassed to say this like but focusing on that aspect of it, of not just the technical aspects of the integrated and we ask you for the nontechnical aspects of it, but really looking for people to explain what this actually affected, why it wasn't just integration for its own sake. Like we were combining these two systems for a reason. And what was that reason from the actual use of the system perspective? Am I saying that back?
• 44:00
  Is that a recent paraphrase?
• 44:01
  Matthew Sanabria: Yeah, exactly. And the reason why I kind of stress the nontechnical aspect is, first of all, pretty much every candidate we get, we know they have the technical acumen, right? They know what they're doing. They've written code before. They understand how systems work.
• 44:14
  Great. That's great. So does everybody else that's out there. But what we want to focus on too in this is the non technical side, right? How was it received?
• 44:23
  Did it solve the problem? And even more so, what would have been the result if you didn't implement that, right, if you didn't integrate these systems together? Because there are times where saying no and not integrating systems was the correct move. Perhaps when because you didn't know enough about the customer problem yet to know that they didn't really need this integration. They were doing something completely different that they didn't need this to be integrated on.
• 44:48
  Like, that kinda comes out in materials when you ask about that nontechnical stuff.
• 44:52
  Bryan Cantrill: That's really interesting in terms of like that judgment of when when not to integrate two systems because also Even just just the Yeah, right, right, right. The charge of integrating systems comes from like you do actually have these like these two things are kind of like a pre existing systems in their own right. So if you can connect them in the right way, you can create something that's greater than the sum of its parts, which must be I mean, that's like really exciting to get working, but it can also be brutal. What are some of the technical things you look for in because I think it takes a certain personality type disposition. Like what are some of the aspects of character that, like, lend itself because I mean, don't you have to have a high pain threshold?
• 45:40
  Matthew Sanabria: Do you have to have a high pain was gonna perseverance.
• 45:43
  Bryan Cantrill: So you
• 45:44
  Matthew Sanabria: kind of hit it right on the head anyway. Perseverance. Right? We've all, like, probably banged our head against some bug or something for a while and like had to persevere through the gnarly things of the documentation is saying it works one way, your testing is saying it works another way, the person that wrote it is saying it works a third way. And you're like, what the heck is going on here?
• 46:05
  You know? And you have to persevere through that and kind of document how it really works, like, as you go with it. And it's very it's kind of demoralizing sometimes, especially when you have to work with vendors or tooling that they barely have a read me or they barely have any documentation. Now you're just like, you're dash dash helping your way through everything to see, like, how does this really work? Because I need to integrate against it.
• 46:28
  And I I am making the I am the open API spec at this point. You know? Oh my god. It's frustrating.
• 46:33
  Bryan Cantrill: Yeah. This reminds me of when I and he will correct if I get the year wrong, but I think it was it's certainly no later than 2016, but I think it might have been 2015 when I dispatched our colleague, Patrick, that had joined Adam to go into like into Kubernetes and learn more about Kubernetes and like get just get Kubernetes configured. And Patrick is like, I am certain that I am the first person to do that. I I can I feel like I prove that actually this system is not actually configurable because things were in were so I mean, God, nascent is putting it way too generously? But this is also like in the era where everyone's kind of pretending they're using Kubernetes.
• 47:14
  But Patrick is like, I'm convinced none of these people is actually using Kubernetes because like this thing doesn't work. This
• 47:21
  Adam Leventhal: is like solving ten years of impostor syndrome for me because in approximately that same timeline, I was evaluating Kubernetes, which again, everyone was deploying. And my conclusion was, I think I might be able to get it stood up. But if it breaks, there's no fucking way I'll ever be able to debug this thing.
• 47:41
  Matthew Sanabria: I hear that. The other side of the perseverance thing though is going to be like breath, right? Not breath as in breathing, like breath as in like, you know, wideness of skill set. Because you are going to have to integrate disparate systems and you're going to have to talk to customers about their disparate problems that wide. And if you dive so far deep down one rabbit hole and you can't come up for air and like look the other way and see other problems, you're not gonna be good at in this sort of role.
• 48:10
  Right? You have to have that sort of t shape, v shape, whatever you wanna call it nowadays, development where you have breath in a in a in a bunch of different topics because you're the even even in, like, a twenty minute conversation with the customer, you can cover everything from the high level API to the storage layer to the networking layer to some more lower level stuff, then we can jump back up to Kubernetes or this and that, you can just go all over the place. And if you can't speak to those those topics at more than just a surface level and understand what these things are and how they connect, they're not gonna trust you. Right? They're gonna be like, get Matthew out of my face right now.
• 48:43
  Don't wanna talk to this guy. You know, he doesn't know what he's doing. Get me the real guy. And it's like, you don't want that to happen obviously so you have to maintain that breadth. So I look for those things really.
• 48:51
  Perseverance of integrating things and, like, breadth of knowledge. Like, don't just tell me you integrated two HTTP servers. Great. What else? You know, what what there has to be more than that.
• 49:01
  We've all made a GET request. We've all, like, serialized and deserialized things. What else do you have there?
• 49:07
  Bryan Cantrill: Yeah. And I think that like it's well, we just didn't necessarily fit for every insured. There are plenty of folks that are late. And I remember I got it the first time, you know, we we are in kernel development. You're like, you're at depth really.
• 49:16
  Right? And you're kind of the you you you necessarily have to be. And then you you know, you do not you know, you get with a I remember getting with a customer for the first time. And, you know, sometimes when you get get to depth, you can kind of have this idea of like, oh, like, I'm dealing with someone who's, you know, less technical or what have you. And then they ask you, like, they they're running in terms of breadth and where it's like, oh my God, they're asking you about so many things that I don't know anything about.
• 49:42
  And like, wow, this technologist knows so has got such breadth and like, okay, I've got depth over here, but the depth is going to be of limited utility because there are so many things I do not understand. And I mean, think it's actually really important for people to kind of have exposure to what that breadth is like someone who's actually trying to deploy a solution to solve a problem has to have that breadth. And sometimes I think we overly enshrine depth and don't value breadth enough. And the breadth is, as you say, Matthew, is essential for this role.
• 50:18
  Matthew Sanabria: Yeah. And I'm one person, right? Like, obviously, that's why we're hiring for this role. I'm one person. I can only do so much.
• 50:24
  I can only hold so much breath, only hold so much depth. And I have I have friends for that. Right? I have friends for depth. Right?
• 50:30
  Like, there's there's so many resources at Oxide that that, like, so many people here that just have such an an incredible amount of depth in in different areas. Right? Everyone has very, very deep knowledge in different areas. I can tap them on the shoulders if I'm not sure about something and be like, hey, can I just talk to you about something real quick? The customer mentioned this to me and I I just wanna kinda understand how this really works because I only was able to go to this this level with them.
• 50:53
  And they it's funny too because the customer loves that because you're telling them the truth. Like, I actually this is where my knowledge, yeah, this is where my knowledge stops. But I do have access to, you know, other people that that where where their knowledge can continue on. I can talk to them for you and get a better answer. Or let's we can get we we can talk all together and we can just get it all done now.
• 51:15
  And they love that
• 51:16
  Bryan Cantrill: because representative it is of of, like, how, like, how much institutional mistrust there is out there when you're like, hey. You know what customers love? The truth. It's amazing. I just
• 51:25
  Adam Leventhal: write write this down.
• 51:26
  Bryan Cantrill: Write this this down. Hey. Write this down because I think I think this idea has legs.
• 51:30
  Matthew Sanabria: Yeah. The And it's I was gonna say it's funny. Like, the customers love it because you're telling the truth, but then the engineers that you're talking to on the other side that have the depth, they love it too because you're telling them the truth as well. You're no longer mistranslating what the customer is saying. You're saying, hey.
• 51:46
  Customer's stuck right here. They're getting this error. Like, I don't I I think we're doing this. And you can give them more information than than just customer x has problem. Go find out.
• 51:55
  You know what I mean? You can bring the technical details to them in the context and the engineer's like, oh, didn't know the customer was using it in this way. Yeah. No. We'll we'll change that.
• 52:03
  You know? And they love that. Like, I get many DMs from our colleagues saying, thank you for bringing this to our attention. You know?
• 52:10
  Bryan Cantrill: Totally. Or it's being used at all. I mean, I think it's like it's
• 52:13
  Adam Leventhal: so like,
• 52:13
  Bryan Cantrill: sometimes it can feel like such a vacuum. Like, I don't know. Did did anyone care about this thing that I did? Because I I mean so, I mean, you get feedback when things are broken, but you don't necessarily get feedback when things are working. Right?
• 52:24
  And so can be really useful to know that, like, oh, yeah. No. This is being used, and here is the way we would like to to extend it. By the way, because we thought we couldn't extend the system, we did this like this this horrific thing. No.
• 52:35
  No. You don't need to do the horrific thing. Like, we can extend the system. We can know, we were talking to a potential partner this morning, Matthew, and they're like, hey, the the API, like, how set in stone is the API? Like, is we could like, these things are immutable.
• 52:49
  Like, let's let's we can improve these things, which is always very empowering to, know that we all collectively can actually improve the system.
• 52:56
  Matthew Sanabria: Yeah. Yeah. I mean, there's, you know, there's obviously two sides to that. One, yes, integrations and the system itself can be mutated and changed and and made better over time. Obviously, some of those changes will have to be breaking changes.
• 53:07
  So then the other side is understanding the customer well enough to know, like, wait a minute. This customer is using this feature very deeply. If we do change this, we do need to, like, be careful and notify them and make them very, very aware that we're changing this
• 53:20
  Bryan Cantrill: behavior. Interesting.
• 53:21
  Matthew Sanabria: Yeah. Yeah. Yeah. That's interesting. You can't get that if you don't talk to your customers.
• 53:25
  Right? Like, you just don't know.
• 53:28
  Bryan Cantrill: Yeah. That is interesting. And then so how do you I mean, because you're, like, constantly moving, I mean, fluidly between, interacting with customers and understanding their issues and then like, okay, now I'm like actually like it like do me software engineering, right? I'm actually
• 53:45
  Matthew Sanabria: Writing code, yeah.
• 53:46
  Bryan Cantrill: Writing code, like how do you kind of, if you want me asking like how do know, how do you kind of structure your day to do that? Or how do you how do you do it?
• 53:58
  Matthew Sanabria: Yeah. Yeah. I'll preface this and say it to all the listeners. I don't drink coffee at all. Only drank coffee, like, once or twice in my entire life.
• 54:06
  So I don't drink any coffee. So there's no caffeine meme or whatever that I'm gonna say here. The real answer I
• 54:12
  Bryan Cantrill: do do actually I I smoke crack every morning.
• 54:18
  Matthew Sanabria: Right. I don't drink coffee, but I've replaced it with this beautiful other no. No. No. No.
• 54:23
  The the real answer here is just there's a lot that happen like, I have the benefit of being on the East Coast at Oxide. And and I'm not saying this in a bad way or a good way, I'm just saying I have the benefit of being on the East Coast. Most of our meetings at Oxide start at, like, noon or later for me. So I have this nice block of time before Yeah. And no customer is gonna be pinging me that time or even if they are, I don't have, like, some resources that I might need.
• 54:50
  Like, my my colleagues are most certainly not awake yet or whatever. Or maybe they are, they're just not working yet. So I have that block of time that I just use for for the software stuff. Because that's the software side demands the focus. Right?
• 55:03
  It demands the focus time so you can actually get some code done and and test it and whatnot. I can't do that in between my meetings. Right? Like, if you have a 30 block in between meetings, you're not gonna get code done. Let's be real.
• 55:13
  Bryan Cantrill: You can do it. Yeah.
• 55:14
  Matthew Sanabria: Yeah. So I shift that all to the morning, for me. And that's how I get, like, the the code stuff done.
• 55:21
  Bryan Cantrill: Yeah. And I found that actually for me personally kinda over my career that time that is like the most productive time of day has also shifted around too. And I don't if you'd found the same like, but that definitely, know, gone through eras even post kids where like the night owl thing has been great. And then other eras where it's like actually like the morning actually kind of beating and getting everyone and getting up early and been a But you're right, Matt, you've got to find that kind of time of day wherever it is where you can get, you can just you can get, like, get into it and get to depth on whatever it is that you're doing.
• 55:57
  Matthew Sanabria: Yeah. I mean and and it's it's it's interesting. Right? Because we have the benefit of working remotely and my colleagues, you know, the meetings for me start at, like, 12:30 and on. And so there's times where maybe I'm staying up a little later.
• 56:09
  Like like today, right, we're recording this podcast and it's it's yeah. It's it's like hey. Know? Asshole.
• 56:14
  Bryan Cantrill: I'm at work, like, PM, by the way. I hope anyone's we can talk about that. Like, when are we when is sorry. When is the other back half of this podcast come up from, like, the yeah. Sorry about that.
• 56:22
  Matthew Sanabria: Thank you so much. But it's good. It's good. Like, I I take advantage of that. Right?
• 56:26
  I know that nobody's breathing down my neck to make sure that I'm in a chair from nine to five. As long as I'm making my meetings, as long as I'm doing my work, maybe from 1PM to 2PM, I I do take a break and I go walk my dogs or I go to the gym or something. Fine. Because I know I have that five, 06:00 meeting tonight, and I'll just I'll just shift time around. You know what I mean?
• 56:45
  So I take advantage of that fully. And there's times where I want to talk to colleagues in vastly different time zones. Right? Like, you know, we have a colleague over in Australia and New Zealand, and I want to talk to them. And I can't do that on my normal time.
• 56:58
  I have to do that during sort of off time and and it works. But I choose to do that. Right? And I choose to shift the time around for that because I need to, you know, make progress on something that I need their expertise on or blah blah blah. So I that flexibility really just it really helps.
• 57:14
  Bryan Cantrill: Yeah. It's interesting. And just in terms of, like, being very deliberate about how you and then manage that block of focus time. And Adam, what do you when do you find that focus time today? What what and has that shifted over time for Yeah,
• 57:27
  Adam Leventhal: for sure. I think I love our no meeting Wednesdays. That's that's huge for me.
• 57:32
  Bryan Cantrill: Oh, God. I know.
• 57:34
  Adam Leventhal: It used to be evenings, but the kid has ground me down. So evenings are just too exhausting. So, yeah, it's certainly I think often it's like mornings and those Wednesday, no meeting days are are when when I actually find the time to to like focus.
• 57:50
  Bryan Cantrill: Yeah. I well, as I mean, we know we talked about this before, but like the no meeting Wednesdays, like that was a survival tactic for me. I was gonna it was really getting ugly without the with I I just was not finding any window for that focus time. And I for me personally, like if I don't have that thing that I can focus on and do actual like the artifact creation, whether that's that's writing or writing code, I become unhappy. Yeah, I get super Yeah.
• 58:21
  Yeah. I
• 58:21
  Matthew Sanabria: tend to do my artifact reading at like night, like later on. That's when I have the brainpower to do the artifact reading. And, I read
• 58:30
  Bryan Cantrill: The reading. Oh, interesting. Yeah.
• 58:32
  Matthew Sanabria: I read the Kubernetes docs cover to cover on the website. And I did that all at night because I was like, how does this actually work? I need to know. I've used Kubernetes for a while, but how does this really work?
• 58:42
  Bryan Cantrill: Not the bullshit that the cloud people are telling us that it just magically works, create the service type load balancer and it just does the thing.
• 58:48
  Matthew Sanabria: No, no, no. How does this really work? Because I need to build the thing that makes it work.
• 58:52
  Bryan Cantrill: Matthew, I've known that for a long time. And I I I would like I have never had the confidence that I am speaking for both of us the way I do when I say there is no human way I could be reading Kubernetes documentation after like 8PM. I that would be unless I was like, no. I am am suffering from it. I mean, I just it would put me into such a deep slumber.
• 59:16
  Matthew Sanabria: So about that crack that we're talking about. Okay?
• 59:18
  Bryan Cantrill: No. No. No.
• 59:19
  Adam Leventhal: No. Great. For that for that to be happening, like, a family member is in danger, and the only way out is to, like, get a coop Kubernetes cluster back up. Like, that's the only scenario I can imagine.
• 59:29
  Bryan Cantrill: And my family member dies in that scenario. It's very sad. A %. I'm sorry, sorry, family. I all I could.
• 59:38
  Did all But even like, you know, even my natural urges were not enough. My flight or flight was not actually enough to, I'm so impressed Matthew, you were able to do that. Maybe we should experiment with this caffeine free lifestyle. Presuming you are sampling your wife's wares, you are getting the theobromides my friend, Lest you be
• 1:00:00
  Matthew Sanabria: too true. Yeah. I do do your single chocolate. Style.
• 1:00:04
  Bryan Cantrill: Yeah. Exactly. You'd be able to their mainlining chocolates. The You're not wrong. That is really, really impressive.
• 1:00:15
  But then I think it also like, mean yet another reason why you've got to trust people to structure their own day and find their own rhythm that works for them, whatever it is. And I think it's just like when you, you know, you kind of put that commute in the middle of it, it's just like really, really tough.
• 1:00:36
  Matthew Sanabria: Yeah. People lie to you. They say, oh, you can just do that reading on commute. No, you can't. I don't wanna hear that.
• 1:00:43
  No. You can't. It's it's you you really can't, honestly. You're tired. You're making it you're running late all the time because, you know, schedules and stuff, and you can't do that sort of reading.
• 1:00:51
  Bryan Cantrill: So I I just did it at night
• 1:00:52
  Matthew Sanabria: before bed because, yeah, as people are joking in the chat, like, Kubernetes documentation does make you wanna conk out. So it was good. So I I got through a lot of documentation that way, and I was able to write the Kubernetes RFD of how we're gonna, you know, what plugins we're gonna build for it and and move forward. So it worked out well.
• 1:01:08
  Bryan Cantrill: That that's amazing. When I get I'm I'm just amazed that you did have, like, you you would either conk me out or you get, like, really vivid strange nightmares about, like, Ist.io or whatever. Or, you know, I had, you know, I was able to I was I was only able to ping half the IP addresses, and then I kinda woke up in a cold sweat. But the, and so what have been some of the, you know, I can as you've been, because I mean, one of the things about this role that may be surprising folks, maybe not. I mean, it's like a lot of the work you're doing is in Go, right?
• 1:01:40
  I mean, I think that we are, which is gonna be the lingua franca for a lot of this integration kind of work. What's your kind of thought on like the role of programming language in all of this of Go versus obviously our software, much of it is in Rust, but we
• 1:01:54
  Matthew Sanabria: You know what's funny about all this, Brian? Like, so I went to the office. I saw the Go book for free. Right? We we were joking about that earlier today.
• 1:02:01
  And I came into oxide like, oh, I'm probably gonna be one of the only people writing Go here. And that's true, by the way. I am one of only people writing Go. That is that is a % true. Like me and, like, maybe two other people writing Go.
• 1:02:13
  Yeah. If that and the two other people trying to write Go are trying not to write Go anymore. I came into the oxide like, oh, yeah. I'll write some Go. Da da da.
• 1:02:22
  It'd be great. I'll I'll you know, and maybe I'll write some more Rust or whatever. I'm starting not to let Go. I'm not gonna lie to you. I'm I'm getting different feelings for Go than when I first started here.
• 1:02:36
  Bryan Cantrill: Not even
• 1:02:36
  Matthew Sanabria: gonna lie
• 1:02:36
  Bryan Cantrill: to you.
• 1:02:37
  Matthew Sanabria: Not to speak anything bad about Go or whatever. It's just there are some things that really frustrated me now that I'm, like, seeing it in a different light. You know? So, yeah, thank you for ruining that for me.
• 1:02:50
  Bryan Cantrill: God. I'm so sorry. I'm I am so sorry. Yeah. The Atomar long intervention is working.
• 1:02:55
  I told you this is gonna be a split. I'm like, the the the
• 1:02:59
  Matthew Sanabria: Don't come to Oxide if you wanna be corrupted. Okay?
• 1:03:01
  Bryan Cantrill: This is what we do. But but you also do need that, like, I mean, I think part of the challenge of being able to do these kind of integrations is you need to be kind of a polymath when it comes to right? You need to be able to because you don't always get to pick the language that you're integrating in. And you've got to be able to
• 1:03:20
  Matthew Sanabria: Yeah, you're totally right. I mean, hit up the nail on the head. It's like the lingua franca of the cloud integration. So like basically, all of my near term and semi medium term projects are Go. Right?
• 1:03:32
  The Rancher Node driver, the Terraform provider, the Go SDK, the Kubernetes integrations, all these things are gonna be written in Go because that's the language you use for these integrations. And Yeah. That doesn't mean that you should only know Go, though. You have to know other things because the the software that we're writing for for Oxide is Rust. So you need to know how to read Rust and understand, like, oh, this Go call this this Go SDK is gonna call this API in Rust.
• 1:03:56
  Let me see how that's handled. Oh, it's an option of a vector on the Rust side. I need to be able to, like, set this.
• 1:04:03
  Bryan Cantrill: You know, I need to
• 1:04:04
  Matthew Sanabria: be able to set that to nil and Go or I need to be able to you know what I mean? You have to know how to translate those things. And we've we run into that in the Go SDK with, like, options of primitive types in the Rust side. In Go, you don't have an option of a primitive type. You have to make it a pointer automatically.
• 1:04:19
  So now you have pointers to integers. And it's you know, these little things add up. And if you can't if you can't read both sides of the language and understand these things, good luck. You know? Like, you're you're gonna have a very bad time.
• 1:04:33
  Bryan Cantrill: Yeah. And this is tough, right? Because I feel that we and we've got our it was the front end of the computer episode, right Adam, where we were talking about all the terrific work that you and Dave that then allowed Crespo and built the And is And I love all the progenitor stuff. It is like one of these challenges of like the Where you just don't have like, Go just isn't as expressive as Rust is in this regard. And it's tough, you know?
• 1:05:02
  I like, I don't and I don't know because we wanna come up with a on the one hand, you wanna come up with a with a fit for the language. On the other hand, you don't wanna sacrifice, like the power that another language might have. And certainly like for Rust, we don't want
• 1:05:18
  Adam Leventhal: to Yeah. I don't know. It's tough. Yeah. I mean, once you start thinking in some types, like what the enums and Rust, it is sort of hard to imagine doing without, like, do you remember back in the day and see like you'd have, oh, you'd have a union and you'd have a struct with it and you'd have a, you know, just a bunch of hash defines that told you which one and that felt fine.
• 1:05:38
  And now it just feels disgusting. It feels like embarrassing that we were so loosey goosey in a way that we thought was like pretty robust. And I don't know.
• 1:05:49
  Bryan Cantrill: Feels like hand coding assembly.
• 1:05:51
  Matthew Sanabria: It feels like just like,
• 1:05:52
  Bryan Cantrill: wow, how do you ever like go, wow. You got that working? Okay. That's great. But, like, that is like, you know, there's a compiler that'll do that for you.
• 1:05:59
  You don't need to
• 1:06:01
  Adam Leventhal: Right.
• 1:06:01
  Bryan Cantrill: And I feel that same way, but certainly about and it's it's tough because, yeah, know I agree with I mean, in many ways, like, sometimes algebraic types, like the thing that is for especially for error handling. I and I'm Matthew, I dare say that that might be the thing that you're just like, okay, the error handling and goes getting a little bit long in the tooth. I'm actually saying there is actually another way of doing it.
• 1:06:21
  Matthew Sanabria: Yeah. It's it gets a little frustrating. Mean, I typing it out is whatever. Right? You have, like, auto completion or snippets or whatever for that.
• 1:06:29
  But it's just a matter of how you handle the errors. You know? I mean, I know Go treats errors as values and that's fine. It's just a lot of the freaking libraries and such. They just don't let you do anything with them or they don't even pass too much Yeah.
• 1:06:44
  They don't I've seen libraries where they're asserting on the error text rather than like the type of the error. I'm like, what are we doing here? And then, you know, that you have to use that library because that's how you build the plugin and blah blah blah. Or it's just it gets really frustrating, you know, where where you're you're forced to use this language and then you can't use all the features because the library or plugin that you're writing against doesn't support that. And it's just Yeah.
• 1:07:06
  Totally. What am I doing here?
• 1:07:08
  Bryan Cantrill: You know? It's just so so it brings up another question, because I mean, of the things you have integrated with or are integrating with are open, I think. Right? I don't think we're yet using anything. Maybe you would view the the the cloud APIs as the proprietary bits.
• 1:07:24
  I think I'll lead up with the question of like, I assume that, I mean, you probably already have found bugs in things you're integrating with. And maybe that is just like or maybe it would be extremely uncommon to not find bugs in things you're integrating with. And, like, what's your disposition when you find these things that, like, don't work or, like, you know, plainly no improvement?
• 1:07:45
  Matthew Sanabria: Yeah. You know what pisses me off is that you have so basically in Go sorry. It's hard to be to be blunt. No. That's good.
• 1:07:50
  Yeah. Yeah. Absolutely.
• 1:07:52
  Bryan Cantrill: You know what makes me mad?
• 1:07:53
  Matthew Sanabria: You wanna see when I'm angry? No. No. No. The in go or sorry.
• 1:07:58
  In the plug in in the cloud plug in ecosystem, basically, all the plug ins are like, here's the interface, implement this interface, and that's your plug in. Right? Like, there you go. Have fun. Build that binary, whatever.
• 1:08:10
  But nobody documents the interface behavior. They're just like, here's the five methods you need to implement. Go have fun. Nobody writes anything down to say, like, oh, we're gonna call start in these cases. We're gonna call stop in these cases.
• 1:08:25
  We're gonna call this or they don't document the behavior of these interface methods that the concrete type should should be aware of. Right? Like, stop is guaranteed to be called only if this and that and and whatever. They don't they don't document any of that. Nothing about the errors, nothing about any of this stuff.
• 1:08:39
  And you're just left to kind of wonder, and like, how is my concrete type gonna be called in this interface way? When I implement this interface and it's being used by the plugin, how is it gonna be called? What do I have access to? What can I do? What can't I do?
• 1:08:52
  You just have to figure all that out through trial and error. Right? Building the plugin, running it in the ways that you think is gonna be used by the customer, and then stepping on the nil pointer exceptions and stepping on the errors here or whatever. And one of the integrations that we're working on
• 1:09:08
  Bryan Cantrill: I just think, yeah, it's frustrating.
• 1:09:10
  Matthew Sanabria: Yeah. Anytime you, like, log to standard out a certain error, you just don't see those logs ever again. Ever. And you're just like, how do I debug this thing when it's going wrong? Because it's going wrong and I have no idea why.
• 1:09:20
  So you you you log and you you dump, like, you know, type information or whatever. Nope. Don't have access to it. You can't run it in a debugger because you're meant to pass this binary to some other thing that it's it's running it, like, way out there. So I just instrumented it with tracing, sent traces to, like, Honeycomb or Acxiom or whatever.
• 1:09:37
  And I was like, well, if I can't log, I have Internet access. I'm sending traces up there. And I I was able to debug this thing in, like, two seconds. You know? And I was like, why do you do this to me?
• 1:09:46
  You know, why why do you make me do this? You just let
• 1:09:48
  Bryan Cantrill: me see the logs that I
• 1:09:49
  Matthew Sanabria: was trying to to output in the first place. So these little frustrating things are are what you have to deal with when you're integrating with other people's software.
• 1:09:58
  Bryan Cantrill: Okay. But that's really interesting about the way you debug that though in terms of, I mean, frustrating that you needed to add tracing to do it. But I mean, good for you that you had that arrow in the quiver and Matthew, you've got a great blog entry if I recall correctly when you were looking at all these different solutions and kind of comparing and contrasting them, I mean, is where the breadth really begins to pay off. It's like, oh, wait a minute, actually, I know a bunch of these systems really well. It's actually, I mean, because this is your state and I'm like, God, how would I solve that problem?
• 1:10:25
  And let's just say it would have taken me a lot longer because I would not have been able to leverage as much breadth on that problem. Yeah. Remind me of that
• 1:10:40
  Matthew Sanabria: blog There's like observability vendors to watch in 2024. I think I wrote it like a a year ago now. And I was just kind of talking about all the observability vendors out there that are kind of worth checking out and ones that I've used before and one that I recommend and blah blah blah. So I was able to use all that knowledge and use that when I was creating plug ins, and it it really, really helps. So check that out if you're in the observability field and you want to, like, be better with tracing and understand where where to send your data.
• 1:11:07
  Because even if I were to send traces to standard out and SAR error, which is what you would normally do if you're running locally, right, you don't always have to send it to Honeycomb or Acxiom or whatever. You can keep it local. But even if I were to do that in these cases, I was never getting the standard out or standard error anyway, so it would've been useless for me. So, yeah, you really do have to navigate some of that breath and be like, you just have to be creative. Right?
• 1:11:29
  And how can you
• 1:11:29
  Bryan Cantrill: be creative if you don't even know
• 1:11:31
  Matthew Sanabria: what else is out there? How else you can get that information?
• 1:11:36
  Bryan Cantrill: Yeah, absolutely. And so I dropped a link to the blog entry in the chat there, we'll obviously have the show notes of reviewing that my source on that is LinkedIn. I don't know how embarrassed I should be, but I feel like I'm having less, I'm having a lot less shame about using LinkedIn. Actually, you although maybe maybe maybe I need to get some of that shame back. If that Adam, you'd stage intervention, right, if I were being too LinkedIn LinkedIn centric.
• 1:12:03
  Adam Leventhal: Yeah. We're getting there. You're not there yet. We're getting
• 1:12:05
  Bryan Cantrill: Exactly. Makes notes. Draws Hashmark, yeah, another LinkedIn reference. But the so Matt but I dropped in a link to that, that blog entry, which I remember reading when you because you referred to RP 68 in this, I loved your reference to RP 68 and this is what our partnership has shared values. Actually, do you wanna describe that a little bit?
• 1:12:29
  Because that was not something I anticipated on this blog entry when it originally went up.
• 1:12:34
  Matthew Sanabria: Yeah. I mean, because I had just done a whole, like observability pipeline migration at Cockroach, and we were like at the tail end of that. And I've evaluated many observability vendors and I've talked to their people and I've demoed their product and all of this stuff, and I've dug into the depths of OpenTelemetry and the Collector and all the stuff that's with that. And I was like, Man, some of y'all are just selling a product. You don't actually want to partner with people.
• 1:13:03
  You don't actually care if I'm successful with your product or not. You just want me to pay for for the observability tool you have, send the data and then be trapped there. You know, you don't actually care about me providing us with a good a thing that's solving our problem. Yeah.
• 1:13:17
  Bryan Cantrill: Right. And some of you are using the verb trap, actually. Some of you are surprising. Yeah. Yeah.
• 1:13:24
  Interesting.
• 1:13:25
  Matthew Sanabria: I was just so fed up with it, and that's when I was like, you know what? Let me write this down because there's there were some vendors we went with or some vendors we talked to. And I say vendor not in the way of like I'm just saying it as a neutral thing. There were some companies that evaluated their product and I was like, okay, they treated me more like a partner. Like, they gave a shit about what I was doing.
• 1:13:45
  They wanted to, like, understand my problem and some of them even told me, like, yeah, no, this part of our product is not a fit for your use case. And when they say that, you know they're not trying to just sell you something. Yeah. Others were just like, no, we have a we have a circle shapes problem for that circle there. And we also oh, that could be a triangle for that triangle shape problem.
• 1:14:08
  No problem. No problem. We can do all of the things. We're all the shapes all the time. And I'm like, you just want my money.
• 1:14:13
  That's all you want. And you don't actually care about whether or not we're successful with your product. You just want us to pay and then, you know, be miserable and then give us all the promises that, yeah, we'll we'll we'll look at your support tickets. Sure. Sure.
• 1:14:26
  Sure. Sure. And I was like, you're a vendor, you don't you don't have a real product that in a partnership way.
• 1:14:33
  Bryan Cantrill: Yeah, that is and I loved your I mean, again, was very unanticipated that you had discovered this rp 68 that actually was written. Don't know, Matthew, you know the kind of the history of that, but the the we were having all of those same frustrations much further down the stack when we were looking at things like voltage regulators and the service processor. Then we were looking at all these different decisions of band vendors and so on. And in particular one of our colleagues Keith Luszewski was getting extremely frustrated that folks did not have our same disposition. And I'm like, would it help if I put together a kind of a rubric for this?
• 1:15:21
  That really I felt was writing down Adam the stuff that we had already felt internally, but that thing has had, I guess I should have anticipated it, but that has been a really important document for us for like up down and all around. That thing has had way more relevance to us than I would have imagined when I wrote it for sure. Matthew, it was great to see it had real resonance for you to it like at a totally different spot. And I mean, now of course I send that document and we've long since made it public. Before it was public, would send it to folks as like early in the conversation.
• 1:16:01
  And it was actually really interesting that the other of you you've done this experiment sending people r p 68. It's almost like a test to see if they read it. Yeah. Yeah. Yeah.
• 1:16:11
  Yeah. It's a it's Because
• 1:16:13
  Matthew Sanabria: I was gonna say, like, what you said to me the other day, like, some people just tell on themselves, and that's a way to get people to tell themselves.
• 1:16:20
  Bryan Cantrill: Absolutely. And they just don't read it and you and you said, well, this is really important to us. And then conversely, people are like, you know what's funny? This is exactly the way we think and it's so great that you think the same way. Like, okay, think we've got a real potential for our partnership here.
• 1:16:35
  And that is extremely important to us because we and I think that this is a bit foreign in the hardware space where there's this idea that you should dual source absolutely everything. And we don't have that idea because what we actually wanna do is really be able to differentiate our product. We wanna find the right partners and we wanna put rings on fingers. And Adam, I'm just kind of realizing this now. This is kind of a dumb thing to be realizing so late in the afternoon here of Oxide.
• 1:17:06
  But because I've always like wondered like why is that so iconoclastic in the kind of the in the hardware space, but it feels so natural to us. Because in software it's really hard to dual source anything.
• 1:17:17
  Adam Leventhal: Right. You know what I It's
• 1:17:19
  Bryan Cantrill: so bespoke. Yeah. Yep. Yeah. It's like, well, sorry, we have to put rings on fingers.
• 1:17:25
  Like we are not able to like we know, like we actually have to make decisions and like we don't have the option. It's like we have to make sure like we're making the right decision and yeah, we've got to research everybody just like you did in this piece Matthew of like, sorry, I can't like rip out my observability solution and replace it with a different one, nine months from now, rather that's gonna be way disruptive to go do. Preserving that optionality is like not really practical. I actually need to like pick one and then go be willing to go deep with it.
• 1:17:58
  Matthew Sanabria: Yeah. Yeah. And and, like, there were there were companies on that list where I wanted to like, we aligned on as partners and I wanted to use them, but I I couldn't. We weren't ready for for what they needed for us. You know?
• 1:18:08
  There was too much transformation that needed to happen in order to be useful with them. And that's okay. Right? You could always revisit that later. But in that case, I ended up I ended up standardizing OpenTelemetry for for this use case because it allowed us that sort of agreement to dual source a bit, you know, to send data for here and there and and get treated the same or as as same as same can be, right?
• 1:18:30
  And it's true. It's harder to do that in the software space sometimes because you have to agree. And like that's what we're doing here at Oxide, like with our approach to infrastructure and such. That's why Kubernetes has been such a key component in what we're building here, like, integration wise and for customer satisfaction wise, because it's like the only agreement people have nowadays for containers is Kubernetes. That's why every prospect, every customer that approaches us are like, hey, what are you doing for Kubernetes?
• 1:18:56
  It's like, oh, we're actually in the process of building our our cloud controller manager and our and our our cluster API provider. Like, we're we're in the process of building all that. Oh, but what about Distro x and Distro y? It's like, yeah. Well, we need to get the base layers on first because that's what we can all agree on.
• 1:19:09
  And then from there, we can get
• 1:19:10
  Bryan Cantrill: a little
• 1:19:10
  Matthew Sanabria: bit more bespoke. You know?
• 1:19:13
  Bryan Cantrill: Yeah. Interesting. And then, yeah. And obviously a lot of both breadth and depth to tackle as we do all that.
• 1:19:23
  Matthew Sanabria: Yeah. Yeah. I mean, it's a ton, ton of breadth and depth that you need for that.
• 1:19:28
  Bryan Cantrill: That's awesome. And you found like lots of issues. I know you've had like obviously lots of bugs that we've fixed in terms of an oxide software stack. And hopefully that's been collaborative and getting all that stuff fixed. But hopefully it's fun to be operating at this kind of this nexus of because I mean, ultimately like that you're the one that's actually, I would say on the coalface, but I've been led to believe that I'm the only one that says on the coalface.
• 1:20:00
  But are the coal face of actually like making this thing into something that's actually useful for somebody does useful work. So I think that the work you're doing is just so critically important. It requires, mean, all of the, what you said at the top that like this is a job that brings all of your skills to bear. And I really feel one of your new colleagues, you can really feel all those skills being brought to bear. It's been really important.
• 1:20:28
  Matthew Sanabria: Yeah, yeah. And a lot of people are like, they ask me outside of the company or whatever, or even when you and I were talking Brian about what the role is going to shape up to be, like what do you see yourself doing? Like during the interviews, right? I got all that question. Like, what do you see yourself doing at Oxide?
• 1:20:41
  What's success? And Adam was laughing because he asked me this specifically, right? Like, Adam asked me these things specifically.
• 1:20:47
  Adam Leventhal: Yeah. Was like, what do you think this job is? I mean, I know. I'm just wondering what you think it is.
• 1:20:51
  Matthew Sanabria: Yeah. Yeah. Or even like, what do think Oxide is? You asked me that too. I remember.
• 1:20:55
  You're like, what do you what is Oxide? Can you explain it to me? And I explained it to you. And, you know, obviously, I guess I said something right because I'm here. You know?
• 1:21:03
  I guess that's
• 1:21:04
  Bryan Cantrill: my conclusion. Sure. Yeah. I mean, I like to think of Oxide as the progenitor company. People don't I don't know how people refer to it that way.
• 1:21:10
  Matthew Sanabria: Sorry. We're like a podcast company. How dare
• 1:21:12
  Bryan Cantrill: you? No.
• 1:21:14
  Matthew Sanabria: But but it's it's it's true. Like, that those sort of questions made me realize that this role is for me because I knew that, yes, I'm going to bring technical skills here. I'm going write software for you all. Right? For sure.
• 1:21:23
  But I also need to make sure that I'm enabling our salespeople to win customers, to have materials that they need to get customers across the board and prospects across the board. I need to make sure our documentations read like a customer read read for an audience of a customer, right, rather than an audience of engineers. And, you know, oh, marketing needs someone that's building the actual thing to, like, say something about the actual thing, and it can't just always be Brian because your your your resources are tapped all the time. Right? And it's like, Oh, I can do that too because I know what the hell this all is.
• 1:21:54
  And it's like, it's just a mixed bag and it stretches me in different ways, right? I get to use my interpersonal skills, get to use my coding skills, my writing skills to, like, set a vision for things. And, you know, that's why this role needs needs the depth and the breadth that we were talking about. And that's why this role needs more people because there's only so much one person can do, obviously. Exactly.
• 1:22:15
  Yeah. If I'm writing the Packer integration this week that I that I demoed last week or whatever, I Great job. Writing the Kubernetes
• 1:22:21
  Bryan Cantrill: thing. Thank
• 1:22:22
  Matthew Sanabria: you. Thank you. Then I'm not writing the Kubernetes thing because I can't write two things at once. Right? I'm only one person.
• 1:22:28
  So that's why this role is hiring. But I did wanna take some time on this on this podcast episode to, like, kinda dispel the whole title
• 1:22:37
  Bryan Cantrill: ish myth y sort of deal
• 1:22:39
  Matthew Sanabria: because a lot of people are like, I've worked with solutions engineers before. I've worked with support engineers before. I've worked with software engineers before. Like, you're not a traditional software engineer. You must suck.
• 1:22:51
  It's like, no, I have a different set of skills, you know. And Brian, I know I might trigger you with the whole comment of, you know, support engineers aren't worth that much. Solution engineers aren't
• 1:23:00
  Bryan Cantrill: worth that God. Yeah. Oh my God. And sorry, not to, I mean, I'll try to walk past this one as quickly as possible. But we, I mean, we think that support ensuring is a critical function, how to block entry describing more or less exactly this.
• 1:23:19
  And literally top hacker news comment is someone just saying loose, bold faced like basically I thought I was constructing a straw man that they then basically said in earnest about denigrating support and cheering. It just like makes my brain blow up about what I mean, it makes me, it really, yes, sorry, it really does. Infuriates me at a very deep level. It is very, I think disrespectful about how hard that job in particular is about support engineering and how hard it is to be have great breadth, great depth and be cool under fire at a time by the way when customer is like potentially pretty upset. And, like, that is, like say, sorry.
• 1:24:03
  I've I've
• 1:24:03
  Matthew Sanabria: said that. Yeah. I mean, like, I've gotten that from people. People that ask me, like, oh,
• 1:24:08
  Bryan Cantrill: I don't wanna I wanna apply, but I don't
• 1:24:09
  Matthew Sanabria: wanna be a solutions engineer. And I'm like, what does that mean? Like, what what are you saying to me right now? You like, as is as if it's a job that's beneath you? Like, as
• 1:24:18
  Bryan Cantrill: if Right.
• 1:24:18
  Matthew Sanabria: You know, as if it's it's, you know, inferior to to true, quote, unquote, software engineering. It's like, you do realize that a company is made up of more than just software engineers. Right? In fact, if every company was just full of software engineers, they wouldn't exist, you know, tomorrow.
• 1:24:33
  Bryan Cantrill: So they
• 1:24:33
  Matthew Sanabria: they just go bankrupt or whatever. You need more skills than just the the the strictly software skills. And I just I just get so, like I I get mad when people say this too because it's like there's more than just writing the code for you can have the best product in the world, you know, with designed with the most rigor and all of that. And if you can't get customers by telling and tell them in plain language what you do and show that empathy to them and, like, be responsive to building integrations for them, you're not gonna make any sales. No matter how good you think your product is, you're not gonna make any sales.
• 1:25:04
  And so that's why I like this this role because I get to to see that. I get to see what the customer needs from the eyes of an engineer, from the eyes of an end user, and work with, you know, engineering, sales, design, work with the various aspects that we have at us at Oxide and make that come to reality in the best that I can. Right? Obviously, I can't see everything across the finish line myself. I have to turn things over to other teams to work on first.
• 1:25:30
  But I at least get to have that initial push and initial direction of things. And that's really, really, really powerful. So when people tell me like, oh, Matt, you know, you're just solutions engineer. That that kinda sucks. It's like, I think you're mistaken there.
• 1:25:43
  You know? Like, I I think you should revisit what you think
• 1:25:46
  Bryan Cantrill: when you say that because I think you should revisit some of your assumptions, my friend. Yeah. Yeah. No. Absolutely.
• 1:25:54
  And I I feel it's also folks saying in the chat, it's like it's a team sport. It requires a bunch of different kinds of roles. And and one thing I realized, especially with that Hacker News comment, why did that grind my gear so much? I feel that like, and I know you and I have talked about this in the past in terms of Richard Scarry versus Doctor. Seuss and I'm all in Richard Scarry.
• 1:26:16
  You know what I mean? I feel like like, Richard Scary has very much informed my worldview of, like you know what I mean?
• 1:26:23
  Adam Leventhal: Just just one of the members of Busytown? Is that
• 1:26:25
  Bryan Cantrill: Yeah. It's one
• 1:26:25
  Matthew Sanabria: of members
• 1:26:26
  Bryan Cantrill: of Busytown. Like, it meets everybody in Busytown. Like, Richard Scary, it's not like it's like, here's Busytown. And, like, you've got, like, the police officer and you've like the butcher and the librarian, you've got the teacher, you've got the nurse, like everybody is together in busy town making busy town work. Yeah.
• 1:26:41
  And this is like, I feel that, yeah, this is like my, this is very much either informed or reflected by worldview, but like in busy town, you need sorry. You need solution software engineering in busy town. Damn it. It's like busy does not function without it.
• 1:26:57
  Adam Leventhal: So my defense mechanism on those kinds of comments, the reason I drive over them is I think you're probably talking about supporting a much simpler product. And I think the products that I've worked on, I think without exception, have been incredibly complicated, you know, and at the intersection of lots of externalities, like lots of things outside of your control. Like for example, at Oxide, we talked about solving an incredibly hairy networking problem, right? It's like, tell me your, you know, support engineer who is not as, as you know, that it's somehow lesser than could have through that. Walk me through that.
• 1:27:38
  Like, and that has been true, I think of every product that I've worked on that like the intersection of the product and all of the externalities, all of the use cases, all of the ambitions of the customers is insanely complicated and requires people with a broad, broad set of of skills. And, yeah, it takes all the members of build busy time to make it go.
• 1:28:02
  Bryan Cantrill: Slowly worm, mister Fumble, sergeant Murphy, the works. Bananas gorilla, all working together.
• 1:28:08
  Matthew Sanabria: Yes. Funny too about that. Like, was when I was at HashiCorp, I transferred from the support engineering org to software engineering. Right? So I I went to actual, like, from Terraform support engineering to Terraform software engineering.
• 1:28:22
  And I remember being on a call with a customer, and the customer was talking to one of our support engineers. And they're like, we want you to escalate this to an engineer. Get an engineer on the on
• 1:28:32
  Bryan Cantrill: the call.
• 1:28:34
  Matthew Sanabria: And they got me.
• 1:28:34
  Bryan Cantrill: Talk to
• 1:28:35
  Matthew Sanabria: you over the buzzer. Yeah.
• 1:28:36
  Bryan Cantrill: They got me.
• 1:28:37
  Matthew Sanabria: So I came onto the call and because I was on call that week or whatever it was. And then they were like, can you explain to us how to do this Terraform? Like, why is Terraform being so slow here with the parallelism, blah blah blah blah. And I explained it to them. And they're like, oh, that's what the support engineer said.
• 1:28:52
  And I was like, yeah, he trained me. Like, what do you what do you think that they do here? You know what I mean? And I just remember seeing the look at that customer's eyes like, oh, shit. Like, this person, this support engineer that I just kind of dissed knows what the hell like, knows their shit.
• 1:29:09
  And after that, I was just like, wow. People really do have a a weird perception about support engineers, solutions engineers, all of these things. And I I know it doesn't come from nothing. I know I know there's there's some truth to it in some in some regard. I get it.
• 1:29:24
  Not everyone does their job the best, but when you can dispel those stereotypes and show people like, yeah, that's not how we do things here. We're better than that. It really, really, really shows.
• 1:29:36
  Bryan Cantrill: Absolutely. Absolutely. Well, Matthew, it has been so great having you aboard. I feel we we gotta let you plug you've got a podcast of your own that Adam, you don't need to go like, hey. We need go join Matthew's podcast at
• 1:29:50
  Matthew Sanabria: some point.
• 1:29:51
  Bryan Cantrill: But you gotta plug that because that's been a lot of fun. I've ever caught episodes of that, which is really great.
• 1:29:58
  Matthew Sanabria: Yeah. Yeah. Yeah. It's fallthrough.fm. It was it's basically like it came out of the ashes of GoTime.
• 1:30:04
  So GoTime was discontinued by Changelog. And we said we wanted to continue some some of that conversation but get away from Go, so we've created FallThru. It's myself yeah. It's it's a keyword of Go, it was kind of, you know, like, kinda little homage there. So myself, Chris, Ian, Dylan, couple of other Angelica as well.
• 1:30:24
  So we we host people there. And, yeah, please please come by and and hang out. We've had a bunch of people on. So
• 1:30:31
  Bryan Cantrill: A lot of fun conversations on that, and it's obviously Matthew, it's been so great to count you now as a colleague. It's been so much fun to work with you and looking forward to expanding this team and get you some help here.
• 1:30:46
  Matthew Sanabria: Same. Yeah. I'm really excited about what we're doing at Oxide. My colleagues are the greatest. I love all the colleagues at Oxide.
• 1:30:51
  Everyone's doing such fantastic, fantastic work. I'm not gonna sit here and, like, take credit for any of this stuff. Like, I I just get the honor of, like, showing this work off via integrations and on customer calls and through sales things. I get the honor of, like, showing this work off, But there's many, many, many great fantastic people behind the scenes, like, just doing such good work. So thank you for that, everyone that's listening.
• 1:31:15
  Bryan Cantrill: It takes everyone in busy town. That's what I gotta say. It sure does.
• 1:31:18
  Matthew Sanabria: Alright.
• 1:31:19
  Bryan Cantrill: Thanks, everyone. Talk to you next time.