Latest / Code Story | Startup Podcast for CTOs, CEOs and Technical Founders / S13 Bonus: Dynamic Observability: Eliminating Telemetry Waste at Scale with Peter Morelli, Co-Founder & CEO of Bitdrift

Transcript

S13 Bonus: Dynamic Observability: Eliminating Telemetry Waste at Scale with Peter Morelli, Co-Founder & CEO of Bitdrift · Code Story | Startup Podcast for CTOs, CEOs and Technical Founders

• 0:00
  [SPEAKER_03]: Most of their ability tools are built with the same pattern and that you have an agent or an SDK or a sidecar that collects data and then it pushes it into a big database somewhere.
• 0:08
  [SPEAKER_03]: Well it's time series database or store whatever it is.
• 0:11
  [SPEAKER_03]: That works really well in data centers where you have a lot of bandwidth that you're network is reliable, you have a lot of CPU and storage running all the time.
• 0:19
  [SPEAKER_03]: Now that's actually true for mobile, and so what we found was if you had a good network connection for a little bit and you caught some philometry and tried to push it up to a big database in the cloud, it worked.
• 0:29
  [SPEAKER_03]: But that really isn't the normal case.
• 0:31
  [SPEAKER_03]: A lot of time your network isn't good or you walk from inside the outside or into a tunnel and suddenly your network connection drops.
• 0:36
  [SPEAKER_03]: We were seeing that the day it was being dropped on the ground.
• 0:40
  [SPEAKER_03]: My name is Pete Marley, co-founder and CEO of The Drift.
• 0:49
  [SPEAKER_02]: a podcast bringing you interviews with tech visionaries.
• 0:52
  [SPEAKER_02]: Six, six months moonlighting goes.
• 0:53
  [SPEAKER_00]: It's the last and last of the big hands.
• 0:55
  [SPEAKER_02]: Who share what it takes to change an industry?
• 0:58
  [SPEAKER_00]: I don't exactly know what to do.
• 1:00
  [SPEAKER_02]: It doesn't go as to get right.
• 1:02
  [SPEAKER_04]: who built the teams that have their bad company is its team's help each other, a team's proud of her team.
• 1:09
  [SPEAKER_02]: Keeping scalability top of mind, all that infrastructure was up there.
• 1:13
  [SPEAKER_02]: Yes, we've been fighting it as we grow.
• 1:14
  [SPEAKER_02]: Total waste of time.
• 1:15
  [SPEAKER_02]: The stories you don't read in the headlines.
• 1:18
  [SPEAKER_02]: It's not an easy thing to achieve.
• 1:19
  [SPEAKER_02]: To get yourself a deficit of off, try to begin to ride the ups and downs of the start-up line.
• 1:26
  [SPEAKER_04]: To really want it.
• 1:27
  [SPEAKER_01]: Not just about technology.
• 1:29
  [SPEAKER_01]: All this and more on code story.
• 1:32
  [SPEAKER_01]: on your host Nualab part, and today, how Peter Morelle is making your mobile observability not suck, so you can define what matters most to capture.
• 1:49
  [SPEAKER_01]: Peter Morelli was born and raised in San Francisco, one of the few natives of the area.
• 1:54
  [SPEAKER_01]: He was front and center watching the Texan explode in front of him, and also blow up in front of him.
• 1:59
  [SPEAKER_01]: He's lived all over, like New York, San Diego, Texas, London, but always ends up returning to the Bay Area.
• 2:06
  [SPEAKER_01]: Outside of Tech, he's married with three kids, one in college and the other two in high school.
• 2:10
  [SPEAKER_01]: He's a foodie, and loves cooking, enjoying the communal, connective aspect of having a meal with his family and friends.
• 2:20
  [SPEAKER_01]: Peter spent many years at Lyft, where he and his team became frustrated by traditional observability tools and how they waste money storing unused data.
• 2:29
  [SPEAKER_01]: They started to build a solution based on dynamic real-time control paired with local on-device telemetry storage.
• 2:38
  [SPEAKER_01]: Once they figured this out and realized their breakthrough, they spun off the solution into its own company.
• 2:46
  [SPEAKER_01]: This is the creation story of BitDrift.
• 2:53
  [SPEAKER_03]: BitRef is a company that focuses on mobile and edge and durability, so not the normal durability and data centers, but stuff on your phone, stuff on devices like kiosks that you see out world, and there's a fundamental difference in how that works if you go into that, but that was when it was founded to serve.
• 3:12
  [SPEAKER_03]: Its origin stories will unique.
• 3:13
  [SPEAKER_03]: The co-founders and a lot of the initial group of engineers still at BIFRIFT, all are out of lift the right sharing company.
• 3:22
  [SPEAKER_03]: I was a VP of engineering there for eight years.
• 3:24
  [SPEAKER_03]: My co-founders were there at the same time or longer.
• 3:27
  [SPEAKER_03]: And we all ran into a lot of the problems that trying to make a good consumer service, trying to understand what your customers are experiencing is really difficult on phones.
• 3:37
  [SPEAKER_03]: And so a lot of the things that we faced, that's what we started building the first parts of the drift.
• 3:42
  [SPEAKER_03]: And about three years ago, we spun out after building it for, I think, six or seven years there.
• 3:47
  [SPEAKER_03]: And so it's a pretty mature and scalable product, but that's where the origin story really was to solve an actual problem that we had in spades at, left and we couldn't really find in the market and still can't.
• 3:58
  [SPEAKER_03]: So that's sort of a bit to a solution story.
• 4:02
  [SPEAKER_01]: Let's dive into the MVP then.
• 4:05
  [SPEAKER_01]: So maybe it's the version inside a lift or maybe it's when spun out.
• 4:09
  [SPEAKER_01]: You tell me, tell me about the MVP in that first version you built and how long it took to build and what sort of tools you're using to bring it to life.
• 4:18
  [SPEAKER_03]: The the first version really was driven by the fact that most
• 4:24
  [SPEAKER_03]: Most of your ability tools are built with the same pattern.
• 4:26
  [SPEAKER_03]: And now you have an agent or an SDK or a sidecar that collects data and then it's pushing it into a big database somewhere.
• 4:32
  [SPEAKER_03]: Well there's just time series database or store whatever it is.
• 4:36
  [SPEAKER_03]: And that works really well in data centers where you have a lot of bandwidth that you're network is reliable.
• 4:40
  [SPEAKER_03]: You have a lot of CPU and storage running all the time.
• 4:44
  [SPEAKER_03]: now that's actually true for mobile.
• 4:45
  [SPEAKER_03]: And so what we found was if you had a good network connection for a little bit and you caught some philometry and tried to push it up to a big database in the cloud, it worked.
• 4:54
  [SPEAKER_03]: But that really isn't the normal case.
• 4:56
  [SPEAKER_03]: A lot of time your network isn't good or you walk from inside the outside or into a tunnel and suddenly internet or connection drops.
• 5:01
  [SPEAKER_03]: We were seeing that the day was being dropped on the ground.
• 5:04
  [SPEAKER_03]: That's those of case.
• 5:05
  [SPEAKER_03]: We'll try for a few times and then drop it on the floor.
• 5:08
  [SPEAKER_03]: when you have those really frustrating experiences where you don't have enough bars and if you do have bars that's not working and you can't get to your car or it couldn't see that it was there or it wasn't updating.
• 5:18
  [SPEAKER_03]: That was when we lost the visibility and so when your customers are having a bad experience still go to your bigs competitor and maybe never come back.
• 5:24
  [SPEAKER_03]: So that's the existential to us to have that visibility.
• 5:27
  [SPEAKER_03]: So the first version really was collecting all that data onto the the phone itself and just making sure that we could upload it later when we did have a good network connection.
• 5:37
  [SPEAKER_03]: Sounds very simple.
• 5:38
  [SPEAKER_03]: It's actually really hard to do.
• 5:39
  [SPEAKER_03]: You don't want to blow out the the storage on the device, so you have to make sure you're not collecting too much data.
• 5:46
  [SPEAKER_03]: You don't want to affect the user experience, you have about 16 milliseconds before and you start to, by human, to notice a lagger freeze.
• 5:53
  [SPEAKER_03]: And so how do you do and collect a lot of data with them not affecting the rest of your apps performance?
• 5:57
  [SPEAKER_03]: So a lot of them initial work we did.
• 6:00
  [SPEAKER_03]: was our performance in memory safety, making sure everything was asynchronous and then the effects of the UX, getting network reliability detecting when it was going working well or not.
• 6:09
  [SPEAKER_03]: So we ended up with a solution that really wrapped a sort of native libraries.
• 6:14
  [SPEAKER_03]: We broke it and rushed and it still is to the state.
• 6:17
  [SPEAKER_03]: It was a lot of speed performance in memory safety.
• 6:20
  [SPEAKER_03]: And so we wrapped that in sort of the native layers of swift and common.
• 6:24
  [SPEAKER_03]: So a lot of that work was done there.
• 6:26
  [SPEAKER_03]: And so that really was the use cases that we were trying to do were the live consumer cases.
• 6:31
  [SPEAKER_03]: In fact, we were actually at the time exploring writing our own native networking stack, based on all-boy, one of our micro-founders.
• 6:38
  [SPEAKER_03]: At the server version of that all-boy proxy, not cliented that.
• 6:42
  [SPEAKER_03]: And we were looking at all-boy mobile phones.
• 6:44
  [SPEAKER_03]: So debugging and networking stack and all of the varied conditions that the network sees out in a while, that was the initial use case that we started this on.
• 6:52
  [SPEAKER_03]: Expand that it sees that.
• 6:53
  [SPEAKER_03]: The other part that we faced is where the lift was still scaling quite a lot, and so we had tens of millions of clients on daily basis, like connecting and uploading data.
• 7:04
  [SPEAKER_03]: And so at that scale, most of their solutions are too expensive, or they couldn't scale for that level.
• 7:09
  [SPEAKER_03]: I know we took down a couple of providers when we tried to turn them on past five or six percent.
• 7:15
  [SPEAKER_03]: So I think the other thing, and they handle it by sampling.
• 7:18
  [SPEAKER_03]: And so the other thing that we weren't really wanted to solve was being able to deal with the mobile scale, which is millions and millions of devices.
• 7:26
  [SPEAKER_01]: I'm curious about then from that point.
• 7:28
  [SPEAKER_01]: So that, that MVP is built and you're achieving what you want to.
• 7:32
  [SPEAKER_01]: I'm curious about how you progress it from that point and matured it.
• 7:35
  [SPEAKER_01]: And I think to wrap that in a box a little bit, what I'm looking for is how do you build your robot?
• 7:39
  [SPEAKER_01]: How do you go about deciding that?
• 7:40
  [SPEAKER_01]: Okay, this is the next most important thing to build or to address with bid trip.
• 7:46
  [SPEAKER_03]: I think we built a lot of the core infrastructure and I'll go into the sort of the next two generations that we built.
• 7:51
  [SPEAKER_03]: Right now, it's a very customer focused.
• 7:53
  [SPEAKER_03]: We're very lucky to have very large customers and a lot of customers coming to us.
• 7:57
  [SPEAKER_03]: And so we have a very priority driven approach.
• 7:59
  [SPEAKER_03]: We try not to build point solutions like just one offshore work customer.
• 8:03
  [SPEAKER_03]: And so what are the ways that we can extend our platform in a way that benefits lots of customers?
• 8:07
  [SPEAKER_03]: And so we tend to prioritize them based on the customer questions coming in.
• 8:12
  [SPEAKER_03]: And though we have a pretty healthy sort of roadmap for the long term, and we make big investments where we see, I think a lot of it is very customer driven now.
• 8:20
  [SPEAKER_03]: I think early on we were our own customer, and so we had a lot of pain points.
• 8:24
  [SPEAKER_03]: And the next thing we did was we realized that we now had reliable upload of a lot of telemetry that you can collect.
• 8:31
  [SPEAKER_03]: What are the things never being looked at?
• 8:33
  [SPEAKER_03]: That's actually true of most observability systems, like 90% of the data is never looked at.
• 8:37
  [SPEAKER_03]: It's only the bad conditions.
• 8:40
  [SPEAKER_03]: And so why were we wasting all the stand-with storage and ingest for day we weren't looking at?
• 8:46
  [SPEAKER_03]: And so the next thing we did was add a little bit of processing at the edge.
• 8:50
  [SPEAKER_03]: So instead of saying, hey, I'm going to upload all these logs, maybe I just summarize or metricize them at the edge.
• 8:56
  [SPEAKER_03]: And so instead of magnabytes of logs, I could send very small counters.
• 9:00
  [SPEAKER_03]: So the huge amount of bandwidth.
• 9:01
  [SPEAKER_03]: And really allowed us to create metrics on the fly, based on what was happening on the device.
• 9:07
  [SPEAKER_03]: So that was a really powerful feature we added.
• 9:09
  [SPEAKER_03]: And then the next thing we did was making this all runtime configurable.
• 9:13
  [SPEAKER_03]: Now, most observability systems are unique decisions at compile time.
• 9:17
  [SPEAKER_03]: You add a log line, you add a metric, and that is done when you compile them and you release.
• 9:22
  [SPEAKER_03]: If you're doing five or six hundred releases a day on the server side, which is what we were, that's not a problem.
• 9:27
  [SPEAKER_03]: That's like a few minutes or 10 minutes, like 20 minutes of delay.
• 9:31
  [SPEAKER_03]: Unmobile, an average release cycle was anywhere between 10 and 15 days.
• 9:36
  [SPEAKER_03]: You have your weekly trains, you have your alphas of betas the next week, then you have to go to the app store or you have to get it approved, and then it has to propagate to everyone's customer devices.
• 9:45
  [SPEAKER_03]: So, by the time you hear about a customer problem and at a log line, all of a sudden you had a really long delay.
• 9:51
  [SPEAKER_03]: And the God forbid you didn't get it right, you'd have to get it again.
• 9:54
  [SPEAKER_03]: And then after all that, you'd have to turn it off at the end because it was go expensive.
• 9:58
  [SPEAKER_03]: So,
• 9:59
  [SPEAKER_03]: The other thing we did to work, and this is, I think, one of the really things that unlock the lot of the rest of what we do with BitRift, is now we at runtime can say do I want this log or not, for these conditions with these set of people, I can but runtime choose what I want to do it, so observability has not become runtime for mobile, which makes it really nice, you can say I have this problem, get more information, I don't even need to be more turned off, and that's something that used to take you weeks or months and now you can do it a few minutes.
• 10:25
  [SPEAKER_03]: So you combine like the local storage, the ability to edge process, and the ability to be able to turn it on and control it on runtime.
• 10:32
  [SPEAKER_03]: And that's the result that lasts part.
• 10:34
  [SPEAKER_03]: That really is the core of the platform.
• 10:36
  [SPEAKER_03]: And we've layered all the sort of traditions of webletting use cases on top.
• 10:40
  [SPEAKER_01]: I'm curious about team, how'd you go about building your team?
• 10:43
  [SPEAKER_01]: I know some of the folks came from Lyft.
• 10:46
  [SPEAKER_01]: I'm curious about what you looked for in those people to indicate that they are the winning horses to join you.
• 10:51
  [SPEAKER_03]: For us, going from a larger, more established company to a small startup world, it was just going to be a little bit more chaotic.
• 10:58
  [SPEAKER_03]: You needed to have people who were in for that type of ride, whether they'd done it before or interested in doing it.
• 11:04
  [SPEAKER_03]: You're looking for people who are self-starters who understand that not someone's not going to be sitting there telling you exactly what to do every single second of the day.
• 11:12
  [SPEAKER_03]: We were also going to be a remote company remotely.
• 11:14
  [SPEAKER_03]: There's no central office where still like that.
• 11:17
  [SPEAKER_03]: And so we generally skewed towards senior people who had done things across our wide variety of technical areas they were just specialist.
• 11:26
  [SPEAKER_03]: and just for willing to be able to understand and go after problems on your own, so it's think of self-starvers.
• 11:31
  [SPEAKER_03]: And so we took about, I think, about a third of the team that we were working on when we spun out of Lyft, and we've since hired specifically for that style of engineer.
• 11:40
  [SPEAKER_03]: Again, tended to skewer a senior, it's harder to have it as a junior engineer.
• 11:44
  [SPEAKER_03]: just to have no person sitting next to you and helping you coach.
• 11:48
  [SPEAKER_03]: I think it's one of the tougher problems the industry faces now.
• 11:50
  [SPEAKER_03]: But that's how we selected for them.
• 11:52
  [SPEAKER_03]: Obviously, you look for the highest talent that you can.
• 11:55
  [SPEAKER_03]: And I think I've been very lucky in my two co-founder, who's some of the best engineers that I've ever worked with.
• 11:59
  [SPEAKER_03]: They aren't best engineers that have ever worked with, and they tend to attract that talent.
• 12:02
  [SPEAKER_03]: And I'd say, of the initial team that's been out of left, we probably are still have two-thirds of them.
• 12:11
  [SPEAKER_01]: Okay, I'm curious about scale, right?
• 12:13
  [SPEAKER_01]: And specifically, this type of problem, especially when you solved it early, you're collecting data on the device with the new got a scale and get that data somewhere.
• 12:21
  [SPEAKER_01]: And it's a lot of data and all the things, right?
• 12:23
  [SPEAKER_01]: I'm curious about how you approach that in the beginning, but I'm also interested in, have there been interesting areas where you've had to find that as you've grown.
• 12:32
  [SPEAKER_03]: The fun part of this is that we originally built to solve less scale, which is tens of millions of mobile clients communicating every day.
• 12:40
  [SPEAKER_03]: And you have the drivers and the writers, and they're all trying to talk to each other in real time.
• 12:45
  [SPEAKER_03]: So an nasty problem, in terms of scale, and something that we had to solve as we built this.
• 12:50
  [SPEAKER_03]: So out of the gate, we, I think, had the ability to get to tens of millions of devices.
• 12:55
  [SPEAKER_03]: And as we started talking to various customers, the ones that were the most interested,
• 13:00
  [SPEAKER_03]: in what we did were that size were bigger.
• 13:02
  [SPEAKER_03]: And so we have a lot of very large words were installed now over a billion times.
• 13:06
  [SPEAKER_03]: I think we do a trillion logs a day at the edge.
• 13:09
  [SPEAKER_03]: And we can deal with hundreds of millions of concurrent connections.
• 13:12
  [SPEAKER_03]: Because a lot of our clients have these real-time use cases.
• 13:15
  [SPEAKER_03]: So we have a streaming company in India that they were streaming cricket matches and hitting 120 million concurrent connections a day.
• 13:21
  [SPEAKER_03]: There's actually a great post we did with the Amazon technical teams about how we were able to scale with that.
• 13:26
  [SPEAKER_03]: So I think very quickly we started to get customers at the scale that pushed even where we were.
• 13:31
  [SPEAKER_03]: But a lot of it is really how you think about building these systems in a resilient way.
• 13:36
  [SPEAKER_03]: And I think I'll credit my co-founders who've done this at many companies beyond lifts, like Twitter and Amazon all sorts of places where you've had to deal with sort of this type of scale.
• 13:46
  [SPEAKER_03]: And the bitriff is built with that in mind.
• 13:49
  [SPEAKER_03]: Because I think the whole point of it is that is as efficient as possible with the resources that it has, but it still has to deal with consumer scale and mobile devices.
• 13:58
  [SPEAKER_03]: Just to give you a sense, I think, like, left I think was interesting, 45 terabytes or more a day of data from mobile devices.
• 14:05
  [SPEAKER_03]: And again, like most of that wasn't being looked at.
• 14:07
  [SPEAKER_03]: We have customers who 70% of their data bandwidth was being used for telemetry, not for their actual, like customer production usage.
• 14:15
  [SPEAKER_03]: And we can come in and cut that down to 2% because of how the system works.
• 14:19
  [SPEAKER_03]: It keeps data at the edge on the devices until something on the control plane.
• 14:23
  [SPEAKER_03]: Our server layer that says, hey, what am I interested in?
• 14:26
  [SPEAKER_03]: What do you do in the right now?
• 14:27
  [SPEAKER_03]: It sort of says, hey, go out and find this.
• 14:30
  [SPEAKER_03]: inherent in the system is a very different way of thinking about data, storage, and architecture and where it is.
• 14:36
  [SPEAKER_03]: And so instead of having this one big database that you have to scale as you dump everything possible in the world at it, you need to charge it or split the data.
• 14:44
  [SPEAKER_03]: It's still a vertical scaling exercise there.
• 14:46
  [SPEAKER_03]: We have a much more distributed approach to how we collect and use data.
• 14:50
  [SPEAKER_03]: That's a fundamental architecture gives us huge advantages in terms of how we scale, the level of data that we're able to collect and also pricing, which is what you'll see.
• 14:59
  [SPEAKER_03]: The redesign that we did helps us do that scale, but there's a lot of systems thinking and how we built the system to be resilient, assume a lot more failure.
• 15:08
  [SPEAKER_03]: Like I said, the data center is a lot more.
• 15:11
  [SPEAKER_03]: of a sanitary environment as much cleaner as it goes to mobile phones.
• 15:15
  [SPEAKER_03]: Just think of your mobile phone.
• 15:16
  [SPEAKER_03]: How many apps are you using in an eight given day?
• 15:18
  [SPEAKER_03]: You use Wi-Fi, then you walk outside, then you're driving down the street of 50 miles an hour, switching cell towers, and maybe it starts to rain, or like a concert gets out in your neighborhood and saturates the cell towers.
• 15:29
  [SPEAKER_03]: There's a lot of noise in the system.
• 15:32
  [SPEAKER_03]: And I think building for that is one of the things that you know, I had to do very early on in this continued to invest.
• 15:41
  [SPEAKER_01]: So as you step out on the balcony, you look across all that you've built with BitRift, what do you most proud of?
• 15:48
  [SPEAKER_03]: It's taken the focus away and observability of managing the spend and managing these data flows and putting it squarely back on the customer experience.
• 15:57
  [SPEAKER_03]: Right?
• 15:57
  [SPEAKER_03]: You spend all these people in all this time saying, how do I get the data in there?
• 16:01
  [SPEAKER_03]: I pay for it, okay, who shouldn't log now and who should log there that feels like the incentives were really misaligned.
• 16:07
  [SPEAKER_03]: It was all about like a taxes on your system.
• 16:10
  [SPEAKER_03]: When in fact, you should be saying, are my customers having a good experience and not just the main ones, but every single one, right?
• 16:17
  [SPEAKER_03]: Like you don't have that nice clean parade of curve with like everything is a big sort of fat.
• 16:21
  [SPEAKER_03]: A percentage, you have a very long tail of terrible experiences.
• 16:25
  [SPEAKER_03]: And most teams are getting till 10 or 15, maybe 20% of them, just in terms of prioritization in terms of engineering.
• 16:32
  [SPEAKER_03]: And I think the system allows you to address all of those things.
• 16:36
  [SPEAKER_03]: You're not missing the things that you normally would drop or normally would sample away.
• 16:41
  [SPEAKER_03]: You're able to, like, now access it.
• 16:42
  [SPEAKER_03]: Like, we just did release an AI layer on top of our platform that allows agents to go out and do it.
• 16:47
  [SPEAKER_03]: So it's not just humans doing if agents can go out and do a lot of the investigations.
• 16:51
  [SPEAKER_03]: And some of the fixes are supposed to fix it.
• 16:53
  [SPEAKER_03]: And so all of a sudden,
• 16:55
  [SPEAKER_03]: You've moved what I think the industry standard is of addressing a very small percentage of the issues, certainly in mobile, it's smaller than most, and being able to actually, now we can actually do a fairly awesome task.
• 17:05
  [SPEAKER_03]: And the customer, you're going to have a great experience of it.
• 17:07
  [SPEAKER_03]: So I think we need to shift the industry toward that.
• 17:10
  [SPEAKER_03]: I like working on things that shift the industry forward a bit.
• 17:13
  [SPEAKER_03]: And I think it is sort of a shift in how you're allocating your time and your dollars and its durability.
• 17:19
  [SPEAKER_03]: And I think it's a much healthier one.
• 17:23
  [SPEAKER_01]: Let's flip the script a little bit.
• 17:24
  [SPEAKER_01]: Tell me about a mistake you made and how you and your team responded to it.
• 17:29
  [SPEAKER_03]: I think the early contracts that you deal tend to be, you're trying to get your customers, you're trying to get those things.
• 17:35
  [SPEAKER_03]: I think there are a certain ones where we got a two good of a deal.
• 17:38
  [SPEAKER_03]: I thank you to basically where either break even or losing a little bit on some of the original deals.
• 17:43
  [SPEAKER_03]: I think we got those quickly in a lot of those.
• 17:46
  [SPEAKER_03]: I think we find a belief in our pricing model which is different than everyone else's.
• 17:50
  [SPEAKER_03]: And I think at a line in centers in a good way, and that we were able to work with the customers and get to, I think things that we did not expect led to very high cost over runs on our side.
• 18:00
  [SPEAKER_03]: And you were able to handle them with the customer and get to a pretty healthy state.
• 18:03
  [SPEAKER_03]: And a lot of that was very good engineering to help the system scale well.
• 18:07
  [SPEAKER_03]: And we were fortunate to have great customers who were going to work with us and say, OK, that makes sense to us, who can happily tweak
• 18:12
  [SPEAKER_03]: this sort of how we use it on our side to make it work.
• 18:15
  [SPEAKER_03]: And those are still customers that we are with us today and are growing with us today.
• 18:20
  [SPEAKER_03]: So I think for the most part is we try to best to predict what how people would use our systems, but everyone is very different through real wars and really interesting.
• 18:28
  [SPEAKER_03]: So not everything was optimal.
• 18:31
  [SPEAKER_03]: Those sort of the first few conflicts that we made.
• 18:33
  [SPEAKER_03]: I think those are the ones that like highlight to me or the things that we had to spend a lot of time on.
• 18:38
  [SPEAKER_03]: There are lots of hay features that we thought would be really popular that didn't end up being, that we had to recover from some scale decisions we made early on that were painful that we had to fix pretty quickly as we grew.
• 18:49
  [SPEAKER_03]: That was a pretty standard in any ground company.
• 18:55
  [SPEAKER_01]: Okay, Peter, let's move forward and this will be exciting.
• 18:58
  [SPEAKER_01]: I'm curious about what the future looks like for BitDrift.
• 19:01
  [SPEAKER_01]: The industry is moving very fast.
• 19:03
  [SPEAKER_01]: Where things are heading with AI and the way people are using software, tell me what the future looks like for BitDrift.
• 19:11
  [SPEAKER_01]: I think it's still super exciting.
• 19:12
  [SPEAKER_03]: I'm one of those people who think AI is like a net positive to engineers.
• 19:17
  [SPEAKER_03]: And I think I've seen the best engineers that I work with pick up even better.
• 19:20
  [SPEAKER_03]: I think they're very powerful tools.
• 19:22
  [SPEAKER_03]: I've seen our customers use it in fantastic ways.
• 19:25
  [SPEAKER_03]: I think things that used to take weeks.
• 19:28
  [SPEAKER_03]: or months to integrate our systems and start using us now to get a day or two.
• 19:31
  [SPEAKER_03]: And I think we've really been helped by that, and I think I've seen our customers, both their internal development, as well as interacting with us.
• 19:39
  [SPEAKER_03]: It's greatly accelerated.
• 19:41
  [SPEAKER_03]: I think that where do we see that going and what parts are going to be built, what they're going to be aggregated to the big framework and front-sure model sort of providers?
• 19:50
  [SPEAKER_03]: That'll be interesting.
• 19:51
  [SPEAKER_03]: I think we're in for a huge amount of change and how things going.
• 19:54
  [SPEAKER_03]: I think
• 19:55
  [SPEAKER_03]: For us though, when I look at what we provide, we're giving customers access to the data set that they typically have not access to.
• 20:03
  [SPEAKER_03]: They have sampled it at one percent, they haven't done it, they haven't longed enough or they have they've lost it because they haven't used it to get it due to instability.
• 20:11
  [SPEAKER_03]: So I think we're giving both their customers engineers and their agents a lot of the data that they really haven't been able to use before.
• 20:20
  [SPEAKER_03]: That's great for existing mobile applications and solutions.
• 20:24
  [SPEAKER_03]: What I've seen happen is that the sixth celebration we made to our chief to build, not as chief to maintain and observe, but I'm seeing us continuing to invest and how do we make this more automated?
• 20:34
  [SPEAKER_03]: Like I said earlier, capture more of the issues that people are seeing and facing.
• 20:38
  [SPEAKER_03]: And beyond that, it's an expansion beyond just people and mobile phones and interacting, computing, moving to the edge and all these different ways.
• 20:48
  [SPEAKER_03]: I think that was a trend before.
• 20:49
  [SPEAKER_03]: I think it's all in your next story.
• 20:51
  [SPEAKER_03]: How many devices are now being built to the edge?
• 20:54
  [SPEAKER_03]: To TV is running Android, right?
• 20:56
  [SPEAKER_03]: Both Kiosk that you see around the world, they're running the Liberia into the anterior Linux.
• 21:01
  [SPEAKER_03]: So I see that compute at the edge is just happening more and more.
• 21:05
  [SPEAKER_03]: We have several customers who are effectively divisive, but not more tones.
• 21:09
  [SPEAKER_03]: So I think software is only going to get more prolific.
• 21:12
  [SPEAKER_03]: It's only going to get more expanded beyond just our core business of mobile apps, which is doubling and tripling the add, you know, submissions to the app store.
• 21:19
  [SPEAKER_03]: You're going to see it permeate the rest of our lives even more.
• 21:23
  [SPEAKER_03]: So for me, that's exciting.
• 21:24
  [SPEAKER_03]: I think it's an interesting like air at 11, which has a little bit of chaos and the industry is like you said is moving super fast.
• 21:33
  [SPEAKER_03]: So what worked one week does not work the next week.
• 21:35
  [SPEAKER_03]: So I think you'll have to be able to roll with those punches and comfortable with that.
• 21:39
  [SPEAKER_03]: I find it exciting.
• 21:42
  [SPEAKER_01]: Good deal.
• 21:42
  [SPEAKER_01]: Okay, let's switch to you, Peter.
• 21:44
  [SPEAKER_01]: Who influences the way that you work?
• 21:50
  [SPEAKER_03]: I feel I'm always drowning under a huge amount of work.
• 21:53
  [SPEAKER_03]: So it's interesting.
• 21:54
  [SPEAKER_03]: I think my co-founders do.
• 21:56
  [SPEAKER_03]: They both have different ways of approaching work and thinking about systems in a very deep way.
• 22:01
  [SPEAKER_03]: I go back and a lot of my job has been hiring engineers and verbs of engineers.
• 22:06
  [SPEAKER_03]: And they're always the people that are influenced with me the most to bend those spectacular engineers that you've worked with.
• 22:13
  [SPEAKER_03]: And you just watched them as they approach problems and decompose them, the best ones knew that the business mattered as well.
• 22:20
  [SPEAKER_03]: They would build software in a way and technology in a way that would serve the business.
• 22:25
  [SPEAKER_03]: And for me, that was all the thing where I was like, I learned the most.
• 22:30
  [SPEAKER_03]: The other one was, as I've been doing new crawl, I do a lot more sales and marketing, and those are just areas that as a VP of engineering, you were perfectly aware of them, they were important, you enabled them.
• 22:40
  [SPEAKER_03]: But it wasn't really the core thing that you would spend your time on, and now that's what I did most of my time on.
• 22:47
  [SPEAKER_03]: So I think it's like a whole deal set of people and industry leaders in that area, and it's for me then great to learn that's also been frustrating sales as a frustrating experience, a lot of rejection.
• 22:59
  [SPEAKER_03]: And I think for me, if it's been great to stretch what my set of expertise knowledge was that I would apply here.
• 23:08
  [SPEAKER_01]: Okay, Peter, last question, so you're getting on a plane and you're sitting next to a young entrepreneur who's built the next big thing.
• 23:14
  [SPEAKER_01]: They're jazzed about it.
• 23:15
  [SPEAKER_01]: They can't reach showed off the world and can we show off to you right down the plane?
• 23:19
  [SPEAKER_01]: What advice to give that person having gone down this road a bit?
• 23:23
  [SPEAKER_01]: I'm going to assume that it was a very technical entrepreneur, given that sort of my world, but I think not to underestimate the story that you tell.
• 23:32
  [SPEAKER_03]: Around these things, I think a lot of times, every engineer does this, no matter how experienced they are, they focus on the technology, and they tend not to focus on the customer and the story and how the people will use it.
• 23:44
  [SPEAKER_03]: And so,
• 23:45
  [SPEAKER_03]: One of my former experiences working very early at Salesforce, I was like, in January, 30 or 40 of them.
• 23:52
  [SPEAKER_03]: And I remember how Mark Benioff made every year go through customers for customer support, customer support people were on their team, right?
• 24:01
  [SPEAKER_03]: And it was such a customer focused company as opposed to a technology company that had customers.
• 24:07
  [SPEAKER_03]: That's always stayed with me, no matter where I've gone, is just to say, hey, you need to understand your customer deeply, and to spend the time with them more time than you have.
• 24:16
  [SPEAKER_03]: I don't understand what they are.
• 24:18
  [SPEAKER_03]: So that would be one of the pieces of advice.
• 24:21
  [SPEAKER_03]: The other one would be like, do not underestimate how hard it is getting.
• 24:25
  [SPEAKER_03]: and what you do and how awesome it is in front of people.
• 24:28
  [SPEAKER_03]: Everyone hears and sees the viral explosions of things and you've seen a lot of them these days, but for the vast majority of products it's really hard rising up the noise.
• 24:37
  [SPEAKER_03]: And there's a lot of noise and a lot of speed because it's easy to generate software these days.
• 24:41
  [SPEAKER_03]: So that part of being an entrepreneur I think is the harder and more underestimated one.
• 24:48
  [SPEAKER_03]: everyone focuses on what they know, which is technology.
• 24:50
  [SPEAKER_03]: Even I think the other parts are actually equally as hard and equally as interesting problems, and so they need to spend more time on that.
• 24:57
  [SPEAKER_01]: I think that's amazing advice.
• 24:58
  [SPEAKER_01]: It's fantastic.
• 24:59
  [SPEAKER_01]: Peter, thank you for being on the show today.
• 25:02
  [SPEAKER_01]: Thank you for telling the creation story of BitTrip.
• 25:05
  [SPEAKER_01]: Thanks very much.
• 25:08
  [SPEAKER_01]: And this concludes another chapter of Coat Story.
• 25:18
  [SPEAKER_01]: code story is hosted and produced by Noah Labhart.
• 25:22
  [SPEAKER_01]: Be sure to subscribe on Apple podcast, Spotify, or the podcasting app at your choice.
• 25:27
  [SPEAKER_01]: And when you get a chance, leave us a review.
• 25:30
  [SPEAKER_01]: Both things help us out tremendously.
• 25:34
  [SPEAKER_01]: And thanks again for listening.