Latest / Code Story | Startup Podcast for CTOs, CEOs and Technical Founders / S3 E18: Defining Modern Observability: How Honeycomb Redefined Systems Engineering with Charity Majors

Transcript

S3 E18: Defining Modern Observability: How Honeycomb Redefined Systems Engineering with Charity Majors · Code Story | Startup Podcast for CTOs, CEOs and Technical Founders

• 0:02
  [SPEAKER_01]: I'm a little embarrassed to do it now.
• 0:03
  [SPEAKER_01]: I had to be chemically anti-product manager sort of stance.
• 0:08
  [SPEAKER_01]: I didn't want any of them to touch my mind.
• 0:12
  [SPEAKER_01]: It felt to me like giving it control.
• 0:14
  [SPEAKER_01]: It felt to me like giving up the vision.
• 0:15
  [SPEAKER_01]: I've never wanted to build a team where engineers take orders.
• 0:20
  [SPEAKER_01]: That's what it represented to me.
• 0:22
  [SPEAKER_01]: I'm from Ops every single year.
• 0:24
  [SPEAKER_01]: I've been like, every year I've been like, well, this is definitely a year we're gonna fail.
• 0:32
  [SPEAKER_01]: We might not fail, which might mean that we're doomed.
• 0:38
  [SPEAKER_01]: My name is Charity Measures, I'm the CTO and Prof. Ruffinicom.io.
• 0:48
  [SPEAKER_00]: This is Code Story, a podcast bringing you interviews with tech visionaries, who share in the critical moments of what it takes to change an industry, and build and lead a team that has your back.
• 1:02
  [SPEAKER_00]: I'm your host, Noel Lappart, and today, how charity majors point the term of observability and built a tool to surface the relevant infrastructure pins.
• 1:14
  [SPEAKER_01]: All this and more, on code story.
• 1:19
  [SPEAKER_00]: Charity majors didn't touch a computer until she was in college.
• 1:23
  [SPEAKER_00]: In fact, she was raised in a religious, fundamentalist compound in rural Idaho, homeschooled and cultivating all her own food.
• 1:31
  [SPEAKER_00]: She went to college to study classical performance in piano, and though she loved piano, she decided to switch keyboards so to speak and pursue something in computers.
• 1:39
  [SPEAKER_00]: She's been in San Francisco since she was 19, and never wants to leave.
• 1:44
  [SPEAKER_00]: She's firmly motivated by using code to get stuff done.
• 1:47
  [SPEAKER_00]: IE, she doesn't do tech for fun.
• 1:50
  [SPEAKER_00]: Several years ago, she was the first infrastructure hire at Parse, while supporting the mobile back-end as a service before and after the Facebook acquisition.
• 1:58
  [SPEAKER_00]: She had access to a tool where she could slice and dice her infrastructure to gain visibility into a particular section of services and answer questions.
• 2:07
  [SPEAKER_00]: When she left, she realized that a tool of that nature was paramount to doing her job well.
• 2:12
  [SPEAKER_00]: So, she set out to build it again and figure out how to coin the term observability.
• 2:19
  [SPEAKER_00]: This is the creation story of honeycomb.io.
• 2:25
  [SPEAKER_01]: Five years ago or so, seven years ago or so.
• 2:29
  [SPEAKER_01]: I was doing my thing in the engineer at Paris.
• 2:32
  [SPEAKER_01]: I was the first emperor.
• 2:34
  [SPEAKER_01]: I came on board when, you know, pre-beta, and that first year was the same, right?
• 2:39
  [SPEAKER_01]: Between when I came on board, that acquired by Facebook, we brought it down over 60,000 mobile apps.
• 2:45
  [SPEAKER_01]: The technical problems were astoundingly hard and fun and hard.
• 2:50
  [SPEAKER_01]: Every other day, like a different app is hitting the iTunes top 5 or whatever, we'd be scrambling to figure out why parse was done because parse was going on all the time.
• 3:00
  [SPEAKER_01]: It was a really hard set of problems.
• 3:02
  [SPEAKER_01]: We were doing like microservices before buying the services for the thing.
• 3:05
  [SPEAKER_01]: We were kind of head of the curve in a lot of ways.
• 3:08
  [SPEAKER_01]: But like the core root of the problem was for our API server, we had a fixed pool of HTTP workers, unicorn workers.
• 3:17
  [SPEAKER_01]: And back in it, we have multiple databases, and any time any of these backing databases or other services got slow, within the second, all of those workers would fill up with requests that were in flight to that slower bucket, and the output go down for everyone else.
• 3:35
  [SPEAKER_01]: and I tried everything, I tried every tool that was available, you know, monitoring tools, metrics tools were great because they would tell you that something was wrong, but they wouldn't tell you what was wrong.
• 3:45
  [SPEAKER_01]: You couldn't slice and dice your way down to figure out what exactly was wrong.
• 3:49
  [SPEAKER_01]: And logging tools were great if you knew what you were looking for.
• 3:52
  [SPEAKER_01]: and you had to remember to log it in the first place.
• 3:55
  [SPEAKER_01]: But if it was a new problem, if it was an unknown unknown, you were screwed because you didn't know what to look for.
• 4:01
  [SPEAKER_01]: The first insight we got in this problem was at Facebook.
• 4:05
  [SPEAKER_01]: There was this tool called school app.
• 4:07
  [SPEAKER_01]: That was not ugly.
• 4:08
  [SPEAKER_01]: Like aggressively hostile to users.
• 4:10
  [SPEAKER_00]: Like it was not fun.
• 4:12
  [SPEAKER_01]: but it let us slice and dice in near real time and all of a sudden I could follow it like a trail of backgrounds, right?
• 4:19
  [SPEAKER_01]: Like I could start from something's wrong.
• 4:21
  [SPEAKER_01]: What's wrong?
• 4:22
  [SPEAKER_01]: Oh, there's a bunch of errors.
• 4:23
  [SPEAKER_01]: Like what are they going to?
• 4:25
  [SPEAKER_01]: It seems like they're going to the right end points.
• 4:27
  [SPEAKER_01]: Okay, right on the end points?
• 4:28
  [SPEAKER_01]: Well, is it all of them?
• 4:30
  [SPEAKER_01]: No, no, it's not all of them.
• 4:32
  [SPEAKER_01]: It's just the ones that are backed by MongoDB with Sanctuary.
• 4:35
  [SPEAKER_01]: Well, is it all of them?
• 4:36
  [SPEAKER_01]: No, it's just the ones that are for this replica set or for this cluster.
• 4:40
  [SPEAKER_01]: Well, is it all of some, you know?
• 4:43
  [SPEAKER_01]: And I could just like, I could follow that step after step to the end.
• 4:46
  [SPEAKER_01]: The amount of time it took us to diagnose these previously intractable problems dropped like a block, like from...
• 4:54
  [SPEAKER_01]: Open ended days to like maybe minutes, but but it wasn't even engineering problems.
• 5:00
  [SPEAKER_01]: It's a support problem, right?
• 5:02
  [SPEAKER_01]: You could always find the past problem.
• 5:05
  [SPEAKER_01]: This made a big impact on me, like it changed my life in a very real way.
• 5:08
  [SPEAKER_01]: I started to sleep again, and then when I was leaving Facebook a couple years later, I kind of went, oh crap.
• 5:15
  [SPEAKER_01]: I don't know how to engineer anymore without this tool that we've built.
• 5:18
  [SPEAKER_01]: Like, this is not just how I get the site back up to the sites down.
• 5:22
  [SPEAKER_01]: This is, these are my five senses for production.
• 5:24
  [SPEAKER_01]: This is how I know that what I built is doing what I wanted it to, you know?
• 5:29
  [SPEAKER_01]: And so Christina and Christina, they co-founder, sort of building this.
• 5:32
  [SPEAKER_01]: And like we had to start by writing a story and scratch, which was not trivial.
• 5:38
  [SPEAKER_01]: But it was actually the easier problem compared to trying to figure out how to talk about this.
• 5:44
  [SPEAKER_01]: We knew that we weren't building a modern solution.
• 5:46
  [SPEAKER_01]: But trying to figure out a talk about what we were building, was the hardest problem that I've ever had in my entire career.
• 5:52
  [SPEAKER_01]: It was half of the way through the first year that I first I googled the term observability, which was not a term that was widely used to the tribe.
• 5:59
  [SPEAKER_01]: But like I googled it, and I, the definition, like it had this really rich history in mechanical engineering, control systems theory, that was all about, can you look at the outside and understand what's going on in the inside?
• 6:12
  [SPEAKER_01]: right?
• 6:13
  [SPEAKER_01]: Can you understand any system state that it's ever gotten itself into without being able to predict that you would see that system state before, right?
• 6:20
  [SPEAKER_01]: And like I just had like light bulbs going off my breath like, oh my god, this is what we're building.
• 6:24
  [SPEAKER_01]: This is so great.
• 6:25
  [SPEAKER_01]: Fast forward a year or so in the entire industry has adopted all of our language.
• 6:31
  [SPEAKER_01]: Most of the people who are out there building food called observatory tools or observatory teams, they're just building monitoring tools.
• 6:36
  [SPEAKER_01]: They're just building monitoring teams.
• 6:38
  [SPEAKER_01]: They're not actually making a leap from
• 6:40
  [SPEAKER_01]: be known at no one's to be unknown at no one's been all of the technical around the person with that choice.
• 6:47
  [SPEAKER_00]: That's interesting.
• 6:47
  [SPEAKER_00]: So, it is observability, but it's not monitoring.
• 6:50
  [SPEAKER_00]: How do you describe that, and that's what you're saying, that was a really hard thing to be able to describe that.
• 6:56
  [SPEAKER_00]: How have you landed on describing the difference there?
• 6:58
  [SPEAKER_00]: That's a key.
• 7:00
  [SPEAKER_01]: Yeah, well, it kind of depends on the audience, right?
• 7:02
  [SPEAKER_01]: Like, there are a lot of particular organizations that follow if you accept the definition that observability means being able to just ask questions from the outside to understand any systems at the inside.
• 7:14
  [SPEAKER_01]: There's actually a lot of things that fall from that.
• 7:16
  [SPEAKER_01]: It means it's a total high cardinality.
• 7:18
  [SPEAKER_01]: high-dimensionality.
• 7:19
  [SPEAKER_01]: It needs to, you know, pack a bunch of really rich context.
• 7:22
  [SPEAKER_01]: You need to be gathering up all this information from the right-rival instruction such that you can ask any question without needing to ship custom code to describe that systemsick.
• 7:33
  [SPEAKER_01]: The way to solve this problem becomes done is the data format.
• 7:36
  [SPEAKER_01]: You can't gather metrics like a metric of a single number, where you have a tendency in tags to it.
• 7:42
  [SPEAKER_01]: It doesn't support high card
• 7:45
  [SPEAKER_01]: I believe that the source of truth for this needs to be is arbitrarily wide, structured data blogs, where you can just pack in all of the contacts, everything that's interesting.
• 7:56
  [SPEAKER_01]: Another way to think of this is that just became necessary when we blew up the monolith because when you had a monolith, you could attach a debugger and you could just step through the code.
• 8:04
  [SPEAKER_01]: Now, when you got rid of the monolith, now your code is in all of these different services, so it has to hop the network.
• 8:11
  [SPEAKER_01]: So you're cutting it into bundle
• 8:16
  [SPEAKER_01]: You see it with everything that we know about the request, any parameters that we're shipped in, all of the stuff about the environment, say the container or the east, east to instance, the language internals, anything we know, we just stuff it in there.
• 8:40
  [SPEAKER_01]: You know, when your request is pretty to exit or error, you ship that single arbitrarily wide circuit data block off to a service like Honeycomb.
• 8:47
  [SPEAKER_01]: In the future, when you're trying to figure out what's going on, right, when you're structured at this way, you're able to go, oh, what all of these errors have in common is, you're all in US East 1A, you're all on this type of instance type, they're all, you all have this in the large packet size, like whatever the hell it is.
• 9:05
  [SPEAKER_00]: If you didn't structure that way, then you couldn't do that post-talk
• 9:10
  [SPEAKER_01]: If you're more of a client-side person or user experience person or whatever, I would describe it more in terms of the questions that you're able to ask.
• 9:18
  [SPEAKER_01]: Questions like what went wrong, what did these errors have in common?
• 9:22
  [SPEAKER_01]: It's like right now like people are used to having metrics tools or anything that's something went wrong.
• 9:26
  [SPEAKER_01]: And then the login tools, which I didn't see, they can see what went wrong.
• 9:30
  [SPEAKER_01]: And then maybe the tracing tools where they can see how it went wrong.
• 9:33
  [SPEAKER_01]: And observability means even ask all of those questions together because it's just the same data, right?
• 9:37
  [SPEAKER_01]: It's two sides in the same core and you're just flipping back and forth between visualizing it as a water fob in your tracing tool or like slicing and dicing it in your logging tool, right?
• 9:47
  [SPEAKER_01]: You should be able to do all that at once.
• 9:52
  [SPEAKER_00]: So tell me about the MVP, tell me about that first product you build, how you went about building it, and I would even say when you recreated it or recreated your five senses as you were saying.
• 10:04
  [SPEAKER_00]: Tell me about that MVP and what sort of tools you used to bring it to life.
• 10:08
  [SPEAKER_01]: Well, I would probably say that I'm getting all wrong.
• 10:13
  [SPEAKER_01]: And our investors were telling this from the very beginning they were just like, shouldn't you find product market fit before you write a storage engine?
• 10:19
  [SPEAKER_01]: No, definitely not.
• 10:21
  [SPEAKER_01]: It was just, it could feel like every other tool out there, like it needed to be different.
• 10:25
  [SPEAKER_01]: We were really, really lucky that we managed to survive.
• 10:27
  [SPEAKER_01]: Because we built for like a year and a half when we even went out there.
• 10:29
  [SPEAKER_01]: But the MPP was basically just as arbitrarily wise for our data blogs so we could slice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice dice
• 10:46
  [SPEAKER_01]: You know, bundling up in, you know, wrapping up of the information and ship it up to any come, right?
• 10:50
  [SPEAKER_01]: We did that like two, three years in.
• 10:53
  [SPEAKER_01]: The other thing that was really helpful was we built sort of the APM home.
• 10:57
  [SPEAKER_01]: people get their daily in and they didn't even face with this pre-builder, which is hostile.
• 11:03
  [SPEAKER_01]: A pre-builder is similar, how easy you make them are never easy.
• 11:05
  [SPEAKER_01]: And so instead of facing them with that, we would show them like the same three graphs together where you request right, your error rate and your latency.
• 11:13
  [SPEAKER_01]: And that made it much friendlier.
• 11:15
  [SPEAKER_01]: But in the beginning like the MVP, the very basic thing was basically just a slightly better law school.
• 11:22
  [SPEAKER_00]: So when you're building that, you know, you had to decide like I had this amazing thing when I was at Parson Facebook and now I've got to rebuild it and I know you said you rebuilt it all wrong.
• 11:34
  [SPEAKER_00]: But even in building that first, you know, first version, you had to make some decisions and tradeoffs about what you were going to do and what you weren't going to do.
• 11:42
  [SPEAKER_00]: Talk to me about those and how did you cope with those decisions?
• 11:47
  [SPEAKER_01]: To make clear, we didn't have a great thing in Facebook.
• 11:50
  [SPEAKER_01]: It was a thing that was necessary and also horrible to use.
• 11:55
  [SPEAKER_01]: We wanted to make something that was delightful to you.
• 11:57
  [SPEAKER_01]: You know, at Facebook, they could just cram it down on drugs.
• 12:00
  [SPEAKER_01]: You use this, you use nothing, right?
• 12:02
  [SPEAKER_01]: And this is a mistake in many ex-Google, ex-facebook, etc.
• 12:06
  [SPEAKER_01]: teams make, which is they're just like, oh, we were forced to use this thing.
• 12:10
  [SPEAKER_01]: It was life changing for us.
• 12:11
  [SPEAKER_01]: We're just going to build it and we're going to force it and we're going to
• 12:16
  [SPEAKER_01]: For the very beginning, we were talking a lot about how do we make this, finally, also what we're trying to do is revolutionize the way engineers think about production systems.
• 12:26
  [SPEAKER_01]: We want to invite them in, instead of it being this terrible thing that they have to be responsible for, we want it to be something where they understand that it makes the better region here.
• 12:37
  [SPEAKER_01]: We talk a lot about, you know, bringing everyone up to the level of the best debugger every area of your system, because when you're working on a system, you're working on your service, you know, it intimately, you know, all the end of the house of it.
• 12:49
  [SPEAKER_01]: you know how they interact with it, like an expert and you should wear grooves in the system by interacting with it.
• 12:55
  [SPEAKER_01]: But when you're keep bugging it, you're not just keep bugging your corner of the system.
• 12:58
  [SPEAKER_01]: You have to be able to understand if you plug the full request.
• 13:02
  [SPEAKER_01]: So like what if we could just give you access to your team's brain, right?
• 13:07
  [SPEAKER_01]: Like everyone on your team is responsible for a different corner of the system.
• 13:11
  [SPEAKER_01]: Every one of you interacts with it in this
• 13:14
  [SPEAKER_01]: And, you know, five months from now when you move on to a different part of the system, you're going to have forgotten how interact with this part of the system, if you're going to want to talk to them before.
• 13:22
  [SPEAKER_01]: Like, what could we do to just make it so that, you know, you can go, oh, okay, I don't call for this thing, whatever, I don't know how to use it, I get paged, but like I know that my team's expert is bed or Emily.
• 13:34
  [SPEAKER_01]: So I'm just going to go look at how Emily interact with the system or like the last of me had a problem that felt like this, you know, I think like Alice is not called.
• 13:42
  [SPEAKER_01]: So I'm going to go see how she did, and I want to just like retrace her steps through the system.
• 13:47
  [SPEAKER_01]: History may not repeat, but it rhymes.
• 13:50
  [SPEAKER_01]: And that should like, taping within the shadow distance of the answer.
• 13:54
  [SPEAKER_00]: So from that point, you've got your MVP built.
• 13:59
  [SPEAKER_00]: You've considered how to build it.
• 14:01
  [SPEAKER_00]: You have a good point of view on how you're going to build it.
• 14:04
  [SPEAKER_00]: And you get the tool out there.
• 14:05
  [SPEAKER_00]: How did you progress the product from there?
• 14:07
  [SPEAKER_00]: And kind of what I'm looking for is, how did you figure out what was the next most important thing to build and how you built your roadmap?
• 14:15
  [SPEAKER_01]: You know, so in the beginning, I had this, I'm a little embarrassed to admit it.
• 14:19
  [SPEAKER_01]: Now I had this, we haven't linked anti-product manager sort of stance.
• 14:24
  [SPEAKER_01]: I didn't want any of them to touch my, it's hard for me to even project myself into the way.
• 14:31
  [SPEAKER_01]: I was thinking about it then.
• 14:33
  [SPEAKER_01]: It felt to me like giving up control.
• 14:35
  [SPEAKER_01]: It felt to me like giving up the vision.
• 14:36
  [SPEAKER_01]: It felt to me like sacrificing.
• 14:39
  [SPEAKER_01]: Like I've never wanted to build a team where engineers take orders.
• 14:43
  [SPEAKER_01]: That's what it represented to be.
• 14:44
  [SPEAKER_01]: Like the all the other years I would just be taking taking orders if I hired new product people.
• 14:50
  [SPEAKER_01]: And that was done.
• 14:54
  [SPEAKER_01]: I have an amazing product manager now, who I rely on very much.
• 14:57
  [SPEAKER_01]: I mean, also in the early days, I think that I was telling the team not to listen to our customers because I knew that our customers were just asking for faster horses.
• 15:06
  [SPEAKER_01]: They all had come from the metrics world.
• 15:09
  [SPEAKER_01]: And I knew that we were building something dramatically different.
• 15:13
  [SPEAKER_01]: So I was reluctant to listen to our customers very much.
• 15:17
  [SPEAKER_01]: And now it's like 30% we have built, we wanted to build.
• 15:21
  [SPEAKER_01]: We've gotten further than 70%.
• 15:23
  [SPEAKER_01]: I actually feel like the world has come around to us.
• 15:26
  [SPEAKER_01]: You know, when I started talking about observability on the stuff, literally I had so many people just tell me you're too late.
• 15:32
  [SPEAKER_01]: It's done.
• 15:32
  [SPEAKER_01]: Data-done's going public.
• 15:34
  [SPEAKER_01]: There's nothing left to be done in space.
• 15:35
  [SPEAKER_01]: Just give up and go home.
• 15:37
  [SPEAKER_01]: And I think that over the past few years, a lot of space is open up there.
• 15:42
  [SPEAKER_01]: partly as a result of, you know, it's really stuff that we haven't been talking about, partly as a result of, you've now got like logging APM metrics, times through database, there's like five or six different markets.
• 15:54
  [SPEAKER_01]: They've changed their product roadmap and they're like we'd observe the ability now too.
• 15:58
  [SPEAKER_01]: I think the customers have been educated a lot, they're expecting different things.
• 16:02
  [SPEAKER_01]: This is all great.
• 16:03
  [SPEAKER_01]: Now we can listen to our users.
• 16:05
  [SPEAKER_01]: They're asking for the right things.
• 16:08
  [SPEAKER_01]: We've been operating with like nine people writing codes for the past two years, which is insane.
• 16:14
  [SPEAKER_01]: It's like an order of magnitude or two of us back to less than any of our competitors.
• 16:19
  [SPEAKER_01]: And the reason for that was because when we last raised, you know, our VCs were like, we believe that you can build product.
• 16:25
  [SPEAKER_01]: We aren't sure that we believe we can sell it.
• 16:27
  [SPEAKER_01]: So, you know, we went heads down like pastime, focusing on sales, me focusing on marketing or just trying to figure out when we're trying to get our house in order of the past two years.
• 16:35
  [SPEAKER_01]: And as of this last summer, we've shown that we can, our sales are great.
• 16:40
  [SPEAKER_01]: Right, we're great.
• 16:41
  [SPEAKER_01]: So now like we were double in the size of our engineering team, like there was a long time there were just like, we just have to keep the wheels on the bus while we figure out our business side and now that's opening up a little bit more.
• 16:52
  [SPEAKER_01]: So now, you know, we've gone through the exercise
• 16:58
  [SPEAKER_01]: and really that what our product vision is is to fulfill the promise of observability.
• 17:02
  [SPEAKER_01]: I think that honeycomb is really the only legit observability tool out there at this point, and yet it doesn't fully fulfill the promise of observability.
• 17:10
  [SPEAKER_01]: There are still questions that you can't ask using honeycomb.
• 17:13
  [SPEAKER_01]: And in order to be able to understand any internal system state, you need to be able to ask any question given the data that you have.
• 17:20
  [SPEAKER_01]: So that's really what's driving us right now.
• 17:23
  [SPEAKER_00]: doubling back a little bit what when you build the tool what what sort of tools are using to build an observability tool.
• 17:29
  [SPEAKER_00]: I mean what what are you writing this platform and you know things like that?
• 17:34
  [SPEAKER_01]: On the back end it's going on the front end it is react as part.
• 17:40
  [SPEAKER_00]: Gotcha so going and react and then microservices type architecture I assume.
• 17:47
  [SPEAKER_01]: Yeah, microservices is ish, you know, the API is just one service, but we've just one of other services too.
• 17:54
  [SPEAKER_01]: We also have this one thing that is pure proxy.
• 17:57
  [SPEAKER_01]: For customers who aren't able to fully use a cloud, they can run the service inside their secure network and proxy other events through it.
• 18:04
  [SPEAKER_01]: It either stores the hash value of the events or our presentation, the events themselves into just forwards on the 50 version of us so that people who have really high data security needs or HIPAA or whatever they can still use this.
• 18:20
  [SPEAKER_00]: Okay, fast forward a little bit back to where we were.
• 18:23
  [SPEAKER_00]: So you're about to expand the team.
• 18:25
  [SPEAKER_00]: You've been coding with nine people and now you're going to double it.
• 18:29
  [SPEAKER_00]: How do you go about building your team?
• 18:31
  [SPEAKER_00]: And what I'm specifically looking for is, what do you look for in those people to indicate that they're the winning horses to join you?
• 18:39
  [SPEAKER_01]: We've been in the Bay Area for a long time, but we have a lot of amazing friends, right?
• 18:44
  [SPEAKER_01]: And this is something where we could have just hired all of our ex-school ex-face with friends, right?
• 18:50
  [SPEAKER_01]: From the very beginning, we were building a tool for everyone.
• 18:53
  [SPEAKER_01]: We need to have a very diverse group.
• 18:56
  [SPEAKER_01]: And I don't just think diverse in terms of gender and race, but diverse in terms of where they come from.
• 19:01
  [SPEAKER_01]: You know, we very intentionally went out in the higher to a bunch of tech academy grads.
• 19:05
  [SPEAKER_01]: It was like, this is really the most representative of the upcoming developers.
• 19:10
  [SPEAKER_01]: The first two people we hired, I will admit, were people that we knew well, because it was like the five of us for the first two and a half years or so.
• 19:18
  [SPEAKER_01]: But even there, like we were very conscious of, because we see in our book, we're colleagues.
• 19:22
  [SPEAKER_01]: We hired two young dads with young girls who were always home by four or five p.m. like every night.
• 19:29
  [SPEAKER_01]: So we were like, well, if we can't set a good role model for future employees, at least we can hire some people who do that for us.
• 19:35
  [SPEAKER_01]: something that we've always done in terms of our interviewing is we do have a coding interview but like the code is not the interview.
• 19:42
  [SPEAKER_01]: We send them like a piece of code the night before we ask them to extend or modify it in some way because green field code is just not most of what we care about right.
• 19:50
  [SPEAKER_01]: We ask this been an hour or two like extending it or like adding features like that but not more than that but that's something the interview the interview is the next day when they have a code review.
• 20:00
  [SPEAKER_01]: with, you know, a couple of people from our team.
• 20:02
  [SPEAKER_01]: And the interview is just, you know, we walked through a lot of, why did you make these choices?
• 20:06
  [SPEAKER_01]: What else did you have in mind?
• 20:07
  [SPEAKER_01]: What would you have done if you had more time to talk me through the trade-offs here?
• 20:11
  [SPEAKER_01]: That's the interview.
• 20:12
  [SPEAKER_01]: And we have found selecting the communication skills is a great predictor of success in our economy.
• 20:17
  [SPEAKER_00]: I mean, text skills, you know, they come and go.
• 20:20
  [SPEAKER_01]: If you could learn one, you could learn another.
• 20:22
  [SPEAKER_01]: You know, like,
• 20:23
  [SPEAKER_01]: But what we care about is do they think clearly and can they communicate about what they're doing and do they not have an ego wrapped up in their code?
• 20:31
  [SPEAKER_00]: Five hired, I think, nine to ten now, graduates from Dev Mountain here in Dallas and a great success working with them.
• 20:40
  [SPEAKER_00]: And great folks, I really love how you said that's representative of tech folks today and moving forward.
• 20:46
  [SPEAKER_00]: I totally agree with that.
• 20:48
  [SPEAKER_00]: And it's this combination of practical skills and view of the world actually as an end user, which is really interesting.
• 20:55
  [SPEAKER_01]: I went to the marm, like sort of mid-career transfer folks, too, who, you know, one of them trained as, like, biologists, one of them was Canadian, like, reporter security guard, but, like, they changed careers and, like, there's some of the most driven motivated folks that I've ever gotten to work with, which is fantastic.
• 21:09
  [SPEAKER_00]: Absolutely.
• 21:10
  [SPEAKER_00]: Gritty and hungry.
• 21:11
  [SPEAKER_01]: Yeah, they are.
• 21:12
  [SPEAKER_00]: Well, this is, this is going to be interesting.
• 21:14
  [SPEAKER_00]: Did you build this in the beginning to scale efficiently or were you fighting this as you grew?
• 21:21
  [SPEAKER_01]: Oh, yeah, no, no, no.
• 21:22
  [SPEAKER_01]: We, we, like, this is kind of it.
• 21:25
  [SPEAKER_01]: the benefit that you get from having an opt co-founder is like skills, they didn't do very big.
• 21:30
  [SPEAKER_01]: It's all, it's a distributed data store, it's skills horizontally, it's not even a problem.
• 21:34
  [SPEAKER_00]: Sort of what you knew and what you walked in with, so it's how you're gonna do it from the beginning.
• 21:39
  [SPEAKER_01]: The infrastructure is still large and what I don't almost five years ago, it just works.
• 21:45
  [SPEAKER_01]: I think more people should really have opt co-founders.
• 21:48
  [SPEAKER_01]: They're incredibly rare and that's really too bad because so many of the classic scaling
• 21:53
  [SPEAKER_01]: Horror stories, problems that we hear about are just the result of people not knowing what they were doing and starting wrongs And besides these things are not hard, it's so much harder to dig yourself out of a hole than not falling into the hole in the first place
• 22:06
  [SPEAKER_00]: Do you have this template that you're able to replicate?
• 22:09
  [SPEAKER_00]: Essentially, say, this is how I'm going to build out a system or is it something that you have to decide how you're going to build it moving, you know, like for Honeycomb or say you started something new, you would do it a different way for that company or, or is it, this is the charity template.
• 22:25
  [SPEAKER_01]: You know, I think that there's a set of reasonable defaults in a given time.
• 22:30
  [SPEAKER_01]: The set of these little defaults that I used five years ago, I think that they would shift a little bit.
• 22:35
  [SPEAKER_01]: Like, for example, I didn't use containers.
• 22:38
  [SPEAKER_01]: I used EC2 instances, because like Docker Kubernetes will kind of on the verge of maybe being to winter at that point.
• 22:46
  [SPEAKER_01]: At this point, I think I would go all in on those, but at that point they weren't ready yet, and so I didn't choose them.
• 22:51
  [SPEAKER_01]: But otherwise, I don't know what I would do differently.
• 22:54
  [SPEAKER_00]: Well, as you step out on the balcony and you look across what you've built with Honeycomb, what do you most proud of?
• 23:01
  [SPEAKER_01]: I am most proud of the fact that we have built a team where you do not have to be the largest person in the room to be heard, to be heard, to be noticed, to be listened to, and be noted, I'm really proud of that.
• 23:15
  [SPEAKER_00]: Absolutely, what did you do to cultivate that culture?
• 23:19
  [SPEAKER_01]: I would give most of the credit to the person's daughter
• 23:23
  [SPEAKER_01]: We hired her early on.
• 23:25
  [SPEAKER_01]: And I knew I knew that I needed her because she comes from a friend-in-back ground and she was really sharp and really good at her job.
• 23:34
  [SPEAKER_01]: But I also feel like she's someone who's been kind of consistently looked over across her career because she is very quiet.
• 23:39
  [SPEAKER_01]: You have to prod her to speak up or you did.
• 23:42
  [SPEAKER_01]: You don't so much anymore, but you did.
• 23:43
  [SPEAKER_01]: And you tell us she just didn't shut down a lot of times when the time she came to us.
• 23:47
  [SPEAKER_01]: But I needed her because I am not someone with any friend and experience or design experience or product experience.
• 23:54
  [SPEAKER_01]: And the fact that I leaned on her so much, I was traveling half the time for the first three or four years while I was CEO.
• 24:00
  [SPEAKER_01]: I leaned on her so much, and I think that she really built this culture where quiet people are celebrated.
• 24:07
  [SPEAKER_00]: Well, let's flip the script a little bit.
• 24:08
  [SPEAKER_00]: Tell me about a mistake you made and how you and your team responded to it.
• 24:13
  [SPEAKER_01]: You know, honestly, I think that your original mistake was me being CEO.
• 24:17
  [SPEAKER_01]: We had originally had three founders that would didn't work out, and I kind of was left holding the bag of CEO.
• 24:23
  [SPEAKER_01]: but I didn't want to do it and I had nightmares for like two years.
• 24:29
  [SPEAKER_01]: I had nightmares almost every night about being basically unemployable for the rest of my career because I had to take a big sideways step and learn all this stuff about sales and marketing it.
• 24:38
  [SPEAKER_01]: My heart wasn't doing that.
• 24:39
  [SPEAKER_01]: I didn't love it.
• 24:41
  [SPEAKER_01]: and fundamentally it's just like my personality type is not.
• 24:44
  [SPEAKER_01]: I am not a very structured person.
• 24:46
  [SPEAKER_01]: So there's a great book called before tendencies where she talks about what motivates you, internally and externally, right?
• 24:53
  [SPEAKER_01]: You can have your internal, the goals you set for yourself, do you find it easier hard to meet them and the expectations other people have of you, do you find it easier hard to make them
• 25:02
  [SPEAKER_01]: And it's basically four possible combinations here, right?
• 25:04
  [SPEAKER_01]: I think you find easy to uphold other's expectations of you against pertainence of yourself, or you're the kind of person who, like, you need the gym buddy, right?
• 25:13
  [SPEAKER_01]: You can't set the structure for yourself, but you follow someone else's external structure, or you're the kind of person who provides easy to meet your own expectations for yourself, but can't meet other's expectations, or you're the type who basically is a rebel and just burns it all down, can't meet anyone's expectations.
• 25:29
  [SPEAKER_01]: and that's me.
• 25:32
  [SPEAKER_00]: I am a volunteer, you know, I'm not someone who provides structure.
• 25:36
  [SPEAKER_01]: So three, three and a half years in, my co-founder and I switch places, Christina's now the CEO, and I like sort of instinctively dominate, and she instinctively doesn't, and so having us in this configuration is just a much better people partnership.
• 25:54
  [SPEAKER_00]: Yeah, a CEO, like I have never been so miserable,
• 25:58
  [SPEAKER_01]: When I would wake up in the morning, we'll have a bed.
• 26:01
  [SPEAKER_01]: The first thing that I would think about is the weight of the responsibility.
• 26:04
  [SPEAKER_01]: You know, like all these people have followed me off this cliff.
• 26:08
  [SPEAKER_01]: And I didn't know if we were going to land anywhere.
• 26:11
  [SPEAKER_01]: And it was kind of slowly killing me.
• 26:16
  [SPEAKER_00]: So, what's the future look like for the product and for your team?
• 26:21
  [SPEAKER_01]: I should promise this first.
• 26:23
  [SPEAKER_01]: I'm from Ops every single year.
• 26:25
  [SPEAKER_01]: I've been like, I was sure we were going to fail from the start.
• 26:28
  [SPEAKER_01]: Like every year I've been like, well, this is definitely the year we're going to fail.
• 26:32
  [SPEAKER_01]: And this is the first year that I'm like, we might not fail, which might mean that we're doomed.
• 26:40
  [SPEAKER_00]: Things are looking up.
• 26:41
  [SPEAKER_01]: Like it's kind of crazy.
• 26:43
  [SPEAKER_01]: The entire world is kind of come around to our way of thinking over the past four years.
• 26:48
  [SPEAKER_01]: And it's a little wild.
• 26:49
  [SPEAKER_01]: It's really crazy here, you know, the stuff that I labored over, like, trying to figure out how to explain this stuff.
• 26:55
  [SPEAKER_01]: Now, just seeing it, like, every confidence I go to, it's being echoed, like, every other day, it feels like there's a product release from some tech company who's, like, you know, using my words and talking about simply, I talked about them.
• 27:10
  [SPEAKER_01]: from a business perspective like we've gotten some amazing executives at Honeycomb and you know who know how to do the things that Christina and I don't right sales and marketing and stuff.
• 27:20
  [SPEAKER_01]: We have some amazing customers right now we're balancing people every day who come to Honeycomb who want to use us but we haven't made it easy enough.
• 27:29
  [SPEAKER_01]: We haven't armed them with the right information, we haven't made it easy enough for the data end, we haven't made it clear enough with it.
• 27:34
  [SPEAKER_01]: choices and so I think there's some sharpening that we need to do.
• 27:37
  [SPEAKER_01]: It's like we've built this great super highway and the on ramp is 10 miles from everyone's front door.
• 27:45
  [SPEAKER_01]: So we need to like build some driveways, right?
• 27:48
  [SPEAKER_01]: It's just in communities to make it easier for them to get on.
• 27:51
  [SPEAKER_01]: But the product does what its test it does, which is a kind of amazing thing.
• 27:55
  [SPEAKER_01]: Ultimately, I mean, this is about helping teams build software in a better way.
• 27:59
  [SPEAKER_01]: Like I come from ops where we're notorious for our
• 28:05
  [SPEAKER_01]: I'm over 30 now, and I don't want to get woken up anymore either, right?
• 28:08
  [SPEAKER_01]: Like, the goal is not to invite software engineers to invite us here in this hellhole.
• 28:14
  [SPEAKER_01]: It's to make it better, and it turns out that ownership is really key to that.
• 28:19
  [SPEAKER_01]: Making these tight virtuous feedback loops, so the person who's making the change and who's empowered to change it,
• 28:25
  [SPEAKER_01]: who knows what the context is, is getting the alerts in a timely manner.
• 28:30
  [SPEAKER_01]: And I feel like, you know, it's not just us, there's also like a lot of darkling with feature flags, there's this people doing, you know, casting it during, it's all about swinging the center of gravity away from pre-production environments to production, right?
• 28:45
  [SPEAKER_01]: Making that the first class citizen, I think we think about first and build for first and we test for first.
• 28:52
  [SPEAKER_01]: I finally think that we might not fail.
• 28:56
  [SPEAKER_01]: It's just a question of the scale of our success.
• 29:01
  [SPEAKER_01]: And so that really boils down, I think, honestly, we're making some really big investments in design, design and product.
• 29:08
  [SPEAKER_01]: I think that our success over the next one to three years is really going to be governed by how well we can shift from, you know,
• 29:21
  [SPEAKER_01]: I think the design is how we're going to make sure that our engineering isn't wasted anymore.
• 29:26
  [SPEAKER_01]: Like we've built a lot of things that people don't, they can't find or they don't see or they don't know how to use or the vision isn't going to transmit it to them somehow.
• 29:33
  [SPEAKER_01]: And the design is my hope for how we bridge that gap.
• 29:37
  [SPEAKER_00]: Let's switch to you, charity, who influences the way that you work.
• 29:41
  [SPEAKER_00]: You know, CEO or CTO or an architect, really any person,
• 29:49
  [SPEAKER_01]: Oh, I look up to Adam Jacob a lot.
• 29:51
  [SPEAKER_01]: Adam Jacob, former CEO of Chef.
• 29:53
  [SPEAKER_01]: In fact, the best compliment I ever received in my life was after we finished pitching to, I think it was storm, it was the last round, round before us, as I was leaving their life.
• 30:04
  [SPEAKER_01]: You're just like a female Adam Jacob, and I was just blown away.
• 30:07
  [SPEAKER_01]: So flattered.
• 30:10
  [SPEAKER_01]: I really admire him.
• 30:11
  [SPEAKER_01]: I think that he's made the
• 30:13
  [SPEAKER_01]: the leap from engineer to executive in a way that is impacted so many people's lives for the better, you're just a warm and wonderful human being, and I also feel like the honeycomb is carrying the torch, you know, just the story of how to build software better.
• 30:28
  [SPEAKER_01]: I was impacted a lot by chef, five, six, seven, eight years ago, or whatever, it made a big impact on my career, and I helped the honeycomb to make the same kind of impact on people's careers.
• 30:41
  [SPEAKER_00]: Well, if you could go back to the beginning, what would you do differently, or where would you consider taking a different approach?
• 30:48
  [SPEAKER_01]: I would start talking to users sooner.
• 30:50
  [SPEAKER_01]: You know, because it made all the mistakes that I said I was going to make, even though I thought that I was not making them.
• 30:56
  [SPEAKER_01]: I was like, ah, people like me, we often start companies and leave design for an afterthought.
• 31:01
  [SPEAKER_00]: We've run into a foreign afterthought.
• 31:03
  [SPEAKER_00]: And I thought I wasn't making those mistakes, and I totally made those mistakes.
• 31:07
  [SPEAKER_01]: I wish that we've been working as hard on the user experience as we were on, you know, the backend, the storage engine, the making, the queries run, that's one thing I would do differently.
• 31:18
  [SPEAKER_01]: Also, I think that I would quit sooner or less to, you know, I think that
• 31:22
  [SPEAKER_01]: I'm a very stubborn person, very stubborn, and of course the best way to motivate me is to tell me that I can't do something.
• 31:31
  [SPEAKER_00]: I've gotten a lot of mileage out of that trait, but it's not all right.
• 31:36
  [SPEAKER_01]: You know, I think that I could have saved myself a lot of suffering if I just been willing to give it a little bit sooner and go, yeah, this has got me so much to do this job.
• 31:47
  [SPEAKER_00]: So you're getting on a plane and you're sitting next to a young entrepreneur builder who has created the next big thing.
• 31:54
  [SPEAKER_00]: They're jazzed about it.
• 31:55
  [SPEAKER_00]: They can't wait to show it off to the world.
• 31:57
  [SPEAKER_00]: They can't wait to show it off to you.
• 31:59
  [SPEAKER_00]: What advice do you give that person having gone down this road a bit?
• 32:04
  [SPEAKER_01]: First of all, I really hate founders.
• 32:07
  [SPEAKER_01]: Like, I just hate them.
• 32:09
  [SPEAKER_01]: The cult of the founder of Silicon Valley is grotesque.
• 32:13
  [SPEAKER_01]: I find it in love for some.
• 32:15
  [SPEAKER_01]: Who to present to answer that?
• 32:16
  [SPEAKER_01]: Did they shut down?
• 32:17
  [SPEAKER_01]: It was just sit down, shut up and go away.
• 32:20
  [UNKNOWN]: Ha ha.
• 32:21
  [SPEAKER_01]: Maybe valid, maybe not.
• 32:23
  [SPEAKER_01]: But if it was genuinely great,
• 32:26
  [SPEAKER_01]: I think that first I would ask if you have a co-founder because going through this without a partner I just can't even imagine I just I just can't imagine it and I think they can really do need a partner to get through it That's great advice.
• 32:39
  [SPEAKER_00]: We'll try to thank you for being on the show today Thank you for telling the creation story of Honeycomb Absolutely my pleasure.
• 32:47
  [SPEAKER_00]: Thank you so much for having me And this concludes another chapter of code story
• 32:54
  [SPEAKER_00]: Coat Story is hosted in produced by Noah Labpart.
• 32:57
  [SPEAKER_00]: Be sure to subscribe on Apple Podcasts Spotify or the podcasting app at your choice.
• 33:02
  [SPEAKER_00]: Support the show on patreon.com slash Coat Story for just five to ten bucks a month.
• 33:08
  [SPEAKER_00]: And when you get a chance, leave us a review both things help us out tremendously.
• 33:15
  [SPEAKER_00]: And thanks again for listening.