Latest / The Pragmatic Engineer / Observability: the present and future, with Charity Majors

Transcript

Observability: the present and future, with Charity Majors · The Pragmatic Engineer

0:00
SPEAKER_0: The three pillars model.
0:01
SPEAKER_0: What is that?
0:02
The famous phrase goes,
0:03
I think it was coined by Peter Borgen back in 2017,
0:06
the observability has three pillars,
0:07
metrics,
0:08
logs,
0:08
and traces.
0:09
And a lot of vendors glommed onto this because cynically speaking,
0:12
they have a metrics product to sell,
0:14
a logging product to sell,
0:15
and a tracing product to sell.
0:16
SPEAKER_1: But it's actually kind of worse than that, right?
0:18
Every request that enters your system historically people have stored it in,
0:22
sure,
0:22
like metrics,
0:24
storage and dashboards,
0:25
and maybe structured logs and unstructured logs,
0:27
in a tracing tool,
0:28
in a profiling tool,
0:30
in an analytics tool again and again and again and again and again.
0:33
SPEAKER_1: And what connects them?
0:34
SPEAKER_1: Not much.
0:35
You,
0:35
the engineer,
0:36
sitting in the middle going,
0:37
"Well,
0:37
that shape looks like that shape,
0:39
sothey'reprobablythesamething."
0:40
SPEAKER_1: Or maybe copy pasting IDs.
0:42
SPEAKER_1: And the cost multiplier is obscene.
0:46
Charity Majors is an observability expert and the author of the book,
0:49
Observability Engineering,
0:51
she worked at a software engineer at Parse then at Facebook and then
0:54
co-founded Honeycomb,
0:55
an observability startup.
0:57
She believes that observability tools should be a lot easier and faster than
1:00
they are today,
1:01
and that every engineer should have the kind of magical observability
1:04
experience like Devzet Meta have with their internal tools.
1:08
In today's episode,
1:08
we go into what is observability and what do things like the three pillar
1:12
model and cardinality mean?
1:14
SPEAKER_0: What is observability 2.0 and why is everyone so excited about it?
1:18
Things engineering teams frequently get wrong about observability and so
1:22
many more things worth knowing about modern observability practices.
1:25
If you enjoy the show and would like to support it,
1:27
please subscribe to the podcast on any podcast platform and on YouTube.
1:31
SPEAKER_0: Charity, welcome to the podcast.
1:34
SPEAKER_1: Thank you so much for having me.
1:35
SPEAKER_1: You and I go way back and I feel like we don't get to talk enough.
1:38
SPEAKER_1: So this is a real treat for me.
1:40
SPEAKER_0: We go way back.
1:41
SPEAKER_0: In fact, the way we started talking was so funny.
1:46
We both were on our blogs,
1:47
wewerewritingadvicecolumnswhowerepeople-
1:52
Oh,
1:52
that guy said the same advice question to both of us,
1:53
and we both answered and we were like,
1:55
"Hey,
1:56
hi there,
1:57
friend."
1:58
Yeah,
1:59
someone asked something about I think measuring developer productivity
2:02
actually funny enough.
2:04
And then I think you wrote something and I wrote something and we didn't know
2:06
about each other.
2:07
SPEAKER_0: And then we discovered it later.
2:09
And I read your article and I'm like,
2:11
"Oh,
2:11
Icouldhavewrittenthesame."
2:12
SPEAKER_0: And I realized we're little bit common souls.
2:15
SPEAKER_1: Yeah, exactly.
2:16
SPEAKER_1: That was funny.
2:19
This episode was brought to you by Sonar,
2:21
the creators of SonarQube Server,
2:22
Cloud,
2:23
ID,
2:23
and Community Build.
2:25
Sonar helps prevent bugs,
2:26
code quality and security issues from reaching production,
2:29
amplifies developer's productivity in concert with AI assistance and
2:33
improves the developer experience with streamlined workflows.
2:37
Sonar analyzes all code regardless of who writes it,
2:40
your internal team or GenAI resulting in more secure,
2:43
reliable,
2:44
and maintainable software.
2:46
Combining Sonar's AI Code Assurance capability and SonarQube with the
2:49
power of AI coding assistance like GitHub Copilot,
2:52
Amazon Q Developer,
2:54
and Google Gemini Code Assist boosts developer productivity and ensures
2:58
that code meets rigorous quality and security standards.
3:01
Join over 7 million developers from organizations like IBM,
3:04
NASA,
3:05
Barclays,
3:05
and Microsoft who use Sonar.
3:08
SPEAKER_0: Trust your developers, verify your AI-generated code.
3:11
SPEAKER_0: Visit sonarsource.com/pragmatic to try SonarQube for free today.
3:16
SPEAKER_0: That is sonarsource.com/pragmatic.
3:21
SPEAKER_0: Trust isn't just earned.
3:23
SPEAKER_0: It's demanded.
3:24
Whether you're a startup founder,
3:25
navigating your first audit or a seasoned security professional scaling
3:29
your governance risk and compliance program,
3:31
proving your commitment to security has never been more critical or more complex.
3:36
SPEAKER_0: That's where Vanta comes in.
3:37
Vanta can help you start or scale your security program by connecting with
3:41
auditors and experts to conduct your audit and set up your security program quickly.
3:46
Plus,
3:46
with automation and AI throughout the platform,
3:48
Vanta gives your time back so you can focus on building your company.
3:52
Businesses use Vanta to establish trust by automating compliance needs
3:55
across over 35 frameworks like SOC2 and ISO27001.
4:00
With Vanta,
4:00
they centralize security workflows,
4:02
complete questionnaires up to five times faster,
4:05
and proactively manage vendor risk.
4:07
Join over 9,000 global companies to manage risk and prove security in real time.
4:12
For a limited time,
4:13
my listeners get $1,000 off Vanta at vanta.com/pragmatic.
4:17
SPEAKER_0: That is V-A-N-T-A.com/pragmatic for $1,000 off.
4:23
So to kick things off,
4:24
can you tell us how you got first exposed to observability?
4:29
Beforehand,
4:30
when we talked,
4:31
I remember that there was something about this startup that you worked at
4:34
Parse and then it got acquired and you worked at Facebook and then something
4:38
happened there.
4:39
SPEAKER_0: How did this go?
4:40
Well,
4:41
so around the time that Parse was getting acquired,
4:43
this was in the free money days,
4:47
rocket ship growth,
4:48
and we were taking off and we had 60 some thousand mobile apps when we got acquired.
4:54
By the time I left,
4:55
we had over a million mobile apps and the traffic was so unpredictable,
5:00
A different app would hit the top 10 in iTunes every day,
5:02
just be like...
5:03
And-
5:05
SPEAKER_0: This was back in 2013, 2014, right?
5:07
SPEAKER_1: Yeah, exactly.
5:08
SPEAKER_1: Like 2012, 2013.
5:10
And we had built this on the Ruby on Rails stack,
5:13
which was a smart enough idea at the time,
5:15
but fixed pool of workers.
5:18
And so any app takes off and suddenly boom,
5:21
Parse goes down because all of the in-flight workers get caught up on threads.
5:25
SPEAKER_1: Any of the backends get slow, boom, Parse goes down.
5:28
SPEAKER_1: And I'm like the infrastructure lead, right?
5:30
And as a reliability engineer,
5:31
this is just professionally humiliating for me because it's constantly
5:34
going down.
5:36
And I tried everything,
5:38
every tool out there,
5:41
and the first glimmer of light that we got was we get to Facebook.
5:44
And I was really cynical about it,
5:46
because all of the tools have been built in-house for in-house workloads,
5:50
you have the luxury of forcing people to use your stuff.
5:53
SPEAKER_1: And I'm like, "This is not built for us."
5:55
But we started feeding some data sets into a tool there called Scuba,
5:59
and suddenly we're able to slice and dice in real time on high cardinality dimensions.
6:04
We were able to instead of just going,
6:06
"Oh..." Because Parse goes down,
6:08
we're looking through the logs,
6:10
and it might be full of a particular request,
6:13
but that doesn't necessarily mean that that's the app that caused it to go down.
6:16
SPEAKER_1: There might just be lots of stuff backing up.
6:17
SPEAKER_1: It might a few very slow requests that's taking.
6:21
SPEAKER_1: You just don't know.
6:23
And having the ability to break down slice by things like app ID,
6:27
user ID,
6:28
raw query,
6:29
normalized query,
6:30
the amount of time it took for us to pinpoint the cause and do something about
6:33
it dropped like a rock from hours.
6:38
Sometimes it would just recover and we'd never actually know what happened
6:41
to seconds.
6:42
SPEAKER_1: It wasn't even an engineering problem anymore.
6:44
SPEAKER_1: It was like a support problem.
6:47
SPEAKER_1: And this was life-changing, right?
6:51
So I'm coming at this from an operational lens,
6:55
and Christine,
6:56
my co-founder,
6:57
had written the Parse analytics product and she had built it on top of
7:01
Cassandra and she was constantly being professionally humiliated by the
7:05
fact that this product she had written for our users to understand their applications,
7:11
you had to pre-define in advance how to capture the data and what questions
7:14
you'd be able to ask.
7:16
SPEAKER_1: And they'd be like, "But I need to ask this new question."
7:18
SPEAKER_1: And she'd be like, "Ah, how embarrassing."
7:20
She'd go and look it up by hand in Scuba and reply to them and she'd just like,
7:25
"Ah."
7:26
So we had both had this experience from very different sides of the stack and
7:29
we were just like...
7:31
So when we left Facebook,
7:34
it's looking back on how ill prepared we were to start.
7:37
SPEAKER_1: I had never heard the words product market fit deep in infrastructure.
7:42
SPEAKER_1: We didn't know that categories existed or Gartner existed.
7:46
So we were not setting out to build a category,
7:48
we were setting out to try and explain this life-changing experience that we
7:54
had had.
7:54
We knew it wasn't monitoring,
7:56
we knew it wasn't logging,
7:58
which is why it took so much longer and was so much harder than it needed to be.
8:03
And then just to get a sense,
8:05
so what was Scuba at Facebook or how would you describe it?
8:08
Because it sounds like it's a mix of just listening to events,
8:13
but also monitoring and querying.
8:16
SPEAKER_1: Scuba is this weird beast.
8:18
SPEAKER_1: It's kind of a columnar store, but it was in memory.
8:21
SPEAKER_1: They wrote a white paper about it.
8:23
One of my favorite things was that replication,
8:27
it's a C++ binary and it shelled out to rsync to do replication.
8:30
SPEAKER_1: It was quick and dirty.
8:34
SPEAKER_1: And the evolution of Scuba was actually an interesting one.
8:37
They developed it about a decade earlier when they were trying to get a handle
8:40
on their MySQL issues.
8:42
SPEAKER_1: There was a time when like PHP and MySQL just crashing all the time.
8:47
Facebook was being professionally humiliated,
8:49
which is why I think it's so interesting,
8:51
the genesis of scuba and thus Honeycomb is not connected to the entire 20,
8:57
30-year history of telemetry and monitoring and three pillars.
9:03
SPEAKER_1: It came out of complete left field.
9:06
SPEAKER_1: It's much closer to the history of business analytics, I think.
9:13
SPEAKER_0: So that's where it started.
9:14
And now jumping a little bit more back to basics,
9:19
you've been in the observability field for a long time now.
9:22
SPEAKER_0: We can save now it's called that.
9:23
But what does observability mean to you from a software engineer's point of view?
9:28
SPEAKER_0: How would you define it?
9:31
SPEAKER_1: Yeah, there are so many definitions floating around.
9:33
SPEAKER_1: I'm probably responsible for more than my fair share of them.
9:40
SPEAKER_1: But really it's about understanding your software.
9:42
It's about understanding the intersection of your code,
9:46
your systems and your users,
9:48
right?
9:49
SPEAKER_1: That's all it is.
9:50
And I feel like for a long time the observability field was obsessed with
9:55
errors and bugs and outages and nines and crashes and all these things.
10:01
And I feel like one of the directions that we're starting to pull the industry
10:04
in is that it's not just about problems.
10:08
Understanding the need to understand impact of what you're putting out into
10:12
the world is so much bigger and more interesting than that,
10:14
which is part of why I feel like it's not just an operational tool.
10:20
SPEAKER_1: This is the tool that underpins or development feedback loops.
10:25
SPEAKER_0: Yeah.
10:25
And I guess maybe not just development,
10:27
but at some level product as well,
10:29
like how your stuff works.
10:31
SPEAKER_1: Yes, exactly.
10:33
And ultimately,
10:34
Christine and I just did this exercise to refresh our mission and vision and
10:38
the phrase that we landed on was we're here to help engineers explain and
10:44
understand our software in the language of the business.
10:47
SPEAKER_1: I feel like there's this void.
10:50
If you look at all of the C-level roles out there,
10:55
the only one that has no template or definition is CTO all over the fucking map.
11:00
SPEAKER_1: Right?
11:01
And if you look at VPs and executive teams,
11:04
engineering VPs have traditionally been like the junior varsity league.
11:08
SPEAKER_1: They're not really in the core group.
11:12
SPEAKER_1: Why is that?
11:13
It's because for so long engineers have been kind of the artist of the company.
11:18
SPEAKER_1: You just trust us, right?
11:20
SPEAKER_1: We can't really justify it.
11:23
SPEAKER_1: Because what is it about executive teams?
11:25
The point of having an executive team is that you're each other's first team
11:29
and you co-own the most fundamental decisions about where companies invest
11:36
their resources,
11:38
which means you have to have to co-understand them.
11:43
Everyone needs to understand enough about marketing to be like,
11:46
"In the upcoming year,
11:47
our priorities are to move the needle in these ways.
11:50
SPEAKER_1: So we're going to allocate these."
11:52
SPEAKER_1: For engineering, it's just like, "We need 20% more people.
11:55
SPEAKER_1: Why?
11:56
SPEAKER_1: Well, we just do."
11:57
SPEAKER_1: That's about as deep as it goes.
11:59
And so I feel like over the next five years or so,
12:03
really helping engineering get that,
12:06
engineering,
12:06
product,
12:07
even design,
12:08
get that first class seat at the table,
12:11
it all comes down to being able to explain and understand our work in the same
12:15
language as everyone else,
12:16
which is money.
12:17
So you're saying that let's say that the CFO,
12:19
so the finance team,
12:20
the CMO,
12:21
the marketing,
12:21
then the COO,
12:22
the operations team,
12:23
they can all explain,
12:25
"Here's what we do,
12:26
here's how that translates to the business.
12:28
Here's why we are important and here's the things that we are moving for the business.
12:33
SPEAKER_0: And if I get 20% more people, I can do this for the business."
12:36
SPEAKER_0: Right?
12:37
SPEAKER_1: Exactly.
12:37
And then you're saying that engineering at a lot of startups,
12:40
if I understand,
12:41
don't really have that ability and observability or understanding how our
12:46
stuff connects to the business gets us there.
12:48
SPEAKER_1: It's a translation layer, yeah.
12:50
Being able to understand reason about the work that you're doing and tie it
12:54
back to top level goals.
12:56
SPEAKER_1: I don't think this is just a startup problem either.
12:58
I talked to big company execs and nobody wants to say this in public,
13:01
but there's a lot of...
13:03
It's hard.
13:04
SPEAKER_1: It's really freaking hard.
13:06
SPEAKER_0: Yeah.
13:07
So going back to observability as a concept,
13:11
it is an industry and you've written a large book about it.
13:15
Can we go through some kind of common terms where if I'm a software engineer,
13:20
I heard about observability,
13:22
what are some things that I should probably know about?
13:25
And you mentioned just before you mentioned high cardinality,
13:28
normalized query,
13:29
but there's stuff like traces,
13:30
instrumentation,
13:31
sampling.
13:32
SPEAKER_0: What are some of the basics that you think?
13:34
SPEAKER_0: Look, if you're an engineer, figure these things out.
13:37
Look it up,
13:37
read a book,
13:38
talk to people and then you'll be able to start with.
13:41
Yeah,
13:42
I think it's important to know the difference between metrics through small
13:47
M metrics and big M metrics,
13:48
right?
13:49
SPEAKER_1: We use the term metrics a lot just as a syn-
13:51
SPEAKER_0: Sorry, you say small M metrics.
13:53
SPEAKER_1: Small M, like the generic term for telemetry.
13:56
SPEAKER_1: We just like, "Oh, the metrics are blah, blah, blah."
13:58
SPEAKER_1: And then there's the Metric, which is a number, with some tags appended.
14:03
SPEAKER_1: And I think this causes a lot of confusion with folks.
14:06
The metric is a very small,
14:09
fast,
14:10
efficient data type,
14:11
but it's supremely limited because it doesn't sort any contextual
14:14
relationship data.
14:18
I think one of the big shifts in the industry right now is away from the sort of
14:21
three pillars model where every request that enters your system,
14:25
you store it in a bunch of different places to a model where you have unified storage.
14:30
So I think understanding the difference between the metric and structured
14:34
data is pretty key.
14:36
I think sampling is emerging as a really important lever and it scares a lot of
14:44
people in large part because the big logging vendors have put decades now
14:47
into telling people every log is sacred,
14:50
don't drop a log line,
14:53
which always reminds me of this sort of Monty Python Every Sperm is Sacred
14:56
sketch and it's like...
15:00
Anyway.
15:01
SPEAKER_1: Yeah.
15:01
SPEAKER_1: So I think sampling is important.
15:03
SPEAKER_1: I think understanding the data structures is important.
15:05
You mentioned something that I think is also kind of a given in the
15:10
observability industry or people who do it,
15:12
the three pillars model,
15:13
what is that?
15:14
Yeah,
15:15
so the famous phrase goes,
15:16
I think it was coined by Peter Borgen back in the 2017.
15:20
SPEAKER_1: The observability has three pillars, metrics, logs, and traces.
15:23
And a lot of vendors lumped onto this because cynically speaking,
15:27
they have a metrics product to sell,
15:28
a logging product to sell,
15:30
and a tracing product to sell.
15:32
SPEAKER_1: But it's actually kind of worse than that.
15:34
Every request that enters your system,
15:36
historically,
15:36
people have stored it in,
15:38
sure,
15:39
like metrics,
15:40
storage and dashboards and maybe structured logs and unstructured logs,
15:44
in a tracing tool,
15:45
in a profiling tool,
15:47
in an analytics tool again and again and again and again and again.
15:51
SPEAKER_1: And what connects them?
15:55
The engineer sitting in the middle going,
15:56
"Well,
15:57
that shape looks like that shape.
15:58
SPEAKER_1: So they are probably the same thing," or maybe copy-paste your IDs.
16:03
And some of the bigger fancier tools like Datadog have built sort of bridges
16:07
that you can pre-define,
16:08
like this metric here ties into that log over there.
16:11
But again,
16:12
you're in this situation where you're having to define in advance what
16:15
information is going to be important and where you're going to need to
16:17
connect it.
16:18
SPEAKER_1: And the cost multiplier is obscene, right?
16:23
Some people,
16:24
every request that enters our system,
16:25
they're storing it in 15 different tools for 15 different use cases.
16:29
And the more than you get,
16:30
the more expensive it gets,
16:31
the more strong it gets and the harder it gets to correlate everything.
16:34
So the 1.0 to 2.0 shift,
16:38
data is data,
16:39
there are a lot of different ways to climb the mountain,
16:41
but fundamentally it's about moving from many sources of truth to unified storage.
16:48
Okay,
16:48
so just so I get that right,
16:50
observability 1.0 is just three pillars,
16:53
the metrics,
16:54
logs and traces,
16:54
which it started like that.
16:57
And apparently it is really good to sell a bunch of services and you can sell
17:00
expensive services with that as well.
17:02
SPEAKER_0: So what is 2.0?
17:03
Okay,
17:04
because you mentioned unified storage,
17:05
butcanyouexpandonwhythatis-
17:08
SPEAKER_1: Yeah.
17:09
SPEAKER_0: It just sounds a bit too simple.
17:09
SPEAKER_0: I mean-
17:10
SPEAKER_1: I know.
17:10
SPEAKER_0: -it sounds like people could have come up with this earlier, right?
17:14
SPEAKER_1: Well, they've had.
17:14
SPEAKER_1: They've had nice things in the business side for decades.
17:17
They've been extremely expensive,
17:18
but we're in the very like the cobbler children have no shoes in the software side.
17:22
It's just like,
17:23
"Well,
17:23
wecouldmakeitworkwithourducttapeandbailingwire."
17:28
I remember Vertica came out 15,
17:28
20 years ago,
17:32
and the business...
17:33
Anyway,
17:35
a lot of folks were talking about unified observability,
17:37
but a lot of situations when you look under a hood,
17:40
what they're talking about is either a unified bill or best a unified visualization.
17:45
And the beautiful thing about having unified storage is you have no dead ends.
17:50
SPEAKER_1: Right?
17:50
You click on a log,
17:53
you can turn it into a trace,
17:54
you can visualize it over time,
17:56
you can derive your metrics from it.
17:57
SPEAKER_1: You can derive your SLOs from it.
17:59
You can take your SLO data,
18:01
click on it,
18:02
jump into it,
18:03
see exactly which events are violating your SLO and why and what's different
18:08
about them.
18:09
SPEAKER_1: But you're right, there is a lot more to it.
18:11
I try to emphasize that because when I started writing about this,
18:15
I started to notice all these other observability 2.0 articles popping up
18:19
all over the place where people are like,
18:20
"Oh yeah,
18:21
we do that too because blah,
18:22
blah,
18:22
blah,
18:22
blah,
18:22
blah."
18:22
SPEAKER_1: And it's like, all right.
18:24
Some of the other things that I think are associated with 2.0...
18:27
And by the way,
18:27
I'm not trying to police this and be like,
18:29
"OnlyHoneycombdoesthis."
18:30
SPEAKER_1: I don't think that at all.
18:31
In fact,
18:32
one of the most exciting things about this last year to me was that I feel like
18:36
the batch of baby observability startups that we're seeing are no longer
18:41
looking like cheaper Datadogs.
18:43
SPEAKER_1: They're looking more cheaper Honeycombs.
18:46
SPEAKER_1: They're built on ClickHouse, they're built in columnar stores.
18:48
SPEAKER_1: They're using OTel Native.
18:50
SPEAKER_1: They're using wide-structured events organized around units of work.
18:55
SPEAKER_1: And I'm so ecstatic.
18:57
SPEAKER_1: This is a better world for the industry and I'm excited.
19:01
But some other things that I think have in common with this are,
19:05
I think it really does parallel the shift from observability being an ops tool.
19:09
It's organized around errors and downtime and outages and crashes and
19:14
towards being something that underpins the whole development cycle.
19:19
It's what underpins your feedback loops and allows you to...
19:23
One of the most exciting use cases I think is in the CI/CD,
19:26
being able to visualize it as a trace,
19:28
see where your tests are breaking,
19:30
where your time is going,
19:31
because keeping that time between when you're building it and when it's in
19:35
production and you're looking at it,
19:37
keeping that as small as possible is the most fundamental part of building
19:42
great software and great teams,
19:43
I think.
19:44
So when you say it underpins development,
19:46
are you saying that these new tools,
19:50
either that exist today or that will exist,
19:52
you're kind of envisioning it as I am coding my stuff and as I push to just
19:57
development to CI/CD and know there's a crash,
20:01
something went wrong,
20:02
I should be able to look at either if it's the build crash,
20:06
I should be able to look at there.
20:07
If the app crash,
20:08
I can just look at this magical dashboard,
20:10
which in the past I had to look at the logs or if I had a trace,
20:15
look at the trace.
20:16
So you're saying that I can just use this to actually develop faster,
20:20
like me as a dev.
20:22
SPEAKER_1: Basically, yeah.
20:23
I mean,
20:25
it can be hard to generalize because there are so many different ways that
20:27
people do this and everything.
20:29
I think that having observability in the pre-production environment is good,
20:35
but I really think it's so much of it about accelerating time to value,
20:38
accelerating time to insights,
20:40
accelerating getting it into production.
20:43
Some of the most exciting things that I think we're doing right now are about getting...
20:49
I mean,
20:49
I usually have my test and prod or live a life shirt on today,
20:52
I have my database shirt on.
20:55
But getting things into production immediately and safely,
21:00
like using progressive deployment,
21:02
using feature flags,
21:04
and the combination of observability of the 2.0 framework plus things like
21:10
progressive deployments,
21:11
canaries,
21:12
feature flags.
21:13
It's greater than the sum of its parts because when you can slice and dice and
21:17
break down,
21:18
say you get your code out within 15,
21:13
20 minutes,
21:20
it's so fresh in your head what you did,
21:22
why you did it,
21:23
how you did it,
21:24
and you're looking,
21:25
but it's not like you're blasting it out to everyone immediately.
21:28
SPEAKER_1: Of course, of course not.
21:30
You ship it behind a feature flag,
21:32
you ship it to a canary,
21:33
you ship it to 10%,
21:35
and then you flip it on,
21:37
you send some test requests,
21:38
you do it in this very control,
21:40
it's like using a scalpel.
21:42
SPEAKER_1: You have this precision tooling that gives you confidence in moving fast.
21:47
So am I hearing it correctly that what you're saying is when you have these
21:51
modern and helpful engineering practices,
21:53
things like feature flags,
21:54
things like being able to test in production because you have some
22:00
separation of tendencies,
22:02
for example,
22:04
all of those things.
22:05
If you have those things and then you kind of invest in observability trying
22:08
to have it better,
22:10
that is just a much bigger win than let's say just,
22:13
I don't know,
22:13
let's get into this,
22:14
whatever,
22:14
observability 2.0 is,
22:16
let's modernize it,
22:16
but by itself,
22:17
it's not going to be as big of a wind,
22:18
right?
22:19
SPEAKER_1: Correct.
22:20
SPEAKER_1: Correct.
22:21
SPEAKER_1: I use this metaphor a lot.
22:22
SPEAKER_1: I'm super blind.
22:23
So it's like putting on your glasses before you go barreling down the freeway,
22:27
right?
22:27
SPEAKER_1: When you're driving, you don't want to be just veering.
22:31
SPEAKER_1: It shouldn't feel like you're always course correcting or driving.
22:35
SPEAKER_1: It should feel like you're just driving, right?
22:38
And when you're building,
22:39
you should have these feedback loops that are so intuitive and so fast and so
22:42
integrated that it feels like you're just building,
22:44
right?
22:45
SPEAKER_1: That's the dream.
22:48
SPEAKER_0: Yeah.
22:49
Now,
22:50
one other question I have related to...
22:54
We've talked about developers of observability,
22:56
but how do you think o11y observability relates to...
23:03
The short form.
23:04
SPEAKER_0: How does it relate to SRE, DevOps, and other roles?
23:08
Because I feel that there are some companies where they're saying,
23:14
"Oh,
23:14
observability?
23:16
SPEAKER_0: That is owned by our SRE team or our DevOps team."
23:19
SPEAKER_0: What's your view about this?
23:28
I think obviously any vendor relationship,
23:30
any product has to have an owner.
23:32
SPEAKER_1: I think that's not necessarily a terrible idea.
23:35
I think that a lot of the center of gravity is moving to platform teams in a lot
23:39
of places because the arena is really managing that thin line,
23:45
the fuzzy line between our code and infrastructure code.
23:49
SPEAKER_1: It's permeable.
23:50
SPEAKER_1: There's always things that somebody's got to own that line.
23:54
But the other thing I like about the platform engineering model is that your
23:58
customers are internal,
24:00
right?
24:00
SPEAKER_1: It's a product focused development organization.
24:03
SPEAKER_1: Your customers are internal.
24:05
And I like that model because some of the historical flaws of the sort of SRE,
24:12
DevOps,
24:12
whatever models have been that,
24:14
oh,
24:14
they own it.
24:15
SPEAKER_1: They own monitoring.
24:18
And the platform model is so explicitly you own your code and we're here to
24:22
help you with that.
24:23
SPEAKER_1: And that is, I think, such a critical change.
24:26
SPEAKER_0: I could not agree more.
24:28
It's interesting how it's a platform team,
24:30
suddenly it's not they,
24:31
but they kind of either reflect and it's you know that you have skin in the game
24:35
and you cannot expect them to,
24:36
for example,
24:37
go on call for your stuff.
24:38
SPEAKER_0: Platform teams just don't do that, right?
24:40
SPEAKER_0: They will build a tooling for you.
24:46
And to be clear,
24:46
as someone at is deep in my DNA,
24:47
and it's not like SREs are going anywhere.
24:50
The systems are only getting harder and more complex,
24:52
and there's an area for expertise.
24:55
SPEAKER_1: And the consultative model is I think a great one.
24:57
DevOps,
24:58
if I might go on a little side rant here,
25:00
I do feel like is there any term in computing that's been more contested than
25:06
the word DevOps?
25:07
I'm not sure,
25:08
but whereas I feel like the DevOps philosophy of being very...
25:13
You work together,
25:16
empathy,
25:17
collaboration is eternal and not restricted to software.
25:21
I do feel like we're sort of in the waning days of the DevOps movement because
25:28
it's no longer considered a good thing to do to split up a dev team and an ops
25:33
team to then collaborate,
25:35
right?
25:36
Increasingly,
25:36
there are only engineers who write code and own their code in production.
25:40
SPEAKER_1: And I think this is really exciting.
25:42
I think we can understand why dev versus ops evolved,
25:49
but it was always kind of a crazy idea that half your engineers could build the
25:53
software and the other half would understand and operate it.
25:56
SPEAKER_1: That's just not a great way to break down.
25:58
SPEAKER_1: It doesn't lead to excellence in either domain.
26:02
Yeah,
26:02
I feel it's a little bit similar to waterfall where people still talk about waterfall,
26:06
but there is no waterfall.
26:07
Waterfall used to be literally three or 4-year-old projects,
26:11
long projects,
26:12
but they don't exist.
26:13
And so when people talk about waterfall,
26:15
they talk about a two-month project.
26:16
SPEAKER_0: That is not waterfall.
26:18
If it's one or two months,
26:19
you can call it whatever you want,
26:21
which is a good thing,
26:22
right?
26:22
SPEAKER_0: It's a good thing that we don't have waterfall projects anymore.
26:24
Even government projects no longer take,
26:27
I mean,
26:28
most of them don't take thankfully that long.
26:30
Yeah,
26:31
and to be clear,
26:32
I get just as mad as anyone else when people are like,
26:35
"DevOpsisdead."
26:37
SPEAKER_1: No, it's not.
26:38
SPEAKER_1: That's the wrong takeaway.
26:39
SPEAKER_1: It's like any good movement eventually fulfills its purpose, right?
26:44
SPEAKER_1: And I feel like the movement of DevOps is in the fulfillment stage.
26:50
SPEAKER_0: I agree with that.
26:52
So I have a question I've been wanting to ask you on why do you think
26:59
observability is so darn hard?
27:02
Every single developer has their story of why observability is hard if they
27:07
actually did it wherever they sit.
27:09
First of all,
27:10
what was your first time where you just realized this is just really hard even
27:14
though it shouldn't be?
27:16
SPEAKER_1: If I can be honest, I actually always hated observability and monitoring.
27:23
True story,
27:29
I would do anything to weasel out of being the one to own it,
27:32
including three companies in a row.
27:35
SPEAKER_1: I hired my friend, Ben Hartshorn, who loves this stuff.
27:37
SPEAKER_1: I'd be like, "Ben, please come work with me because I hate this shit."
27:39
And he would come in and he would build all the graphs and I would bookmark them.
27:41
SPEAKER_1: I have always hated this.
27:42
SPEAKER_0: Oh my gosh, look where that got you.
27:43
SPEAKER_1: I know, right?
27:50
SPEAKER_1: Anyway, it's hard.
27:53
It's hard because software is hard and observability the first line of defense,
27:59
and it's like not only do you have to build the software,
28:02
but you also have to have this sort of meta thread that's watching what you're
28:06
doing and going,
28:07
"Whatisfuturemegoingtoneedtoknow?"
28:09
SPEAKER_1: Or "What is future me going to need to understand at 2:00 A.M.?"
28:13
SPEAKER_1: And that's just a muscle that takes a long time to build it.
28:17
SPEAKER_1: Right?
28:18
And I also think that historically we've had a lot of tools that were so...
28:24
Some of my gripes about the past generation of tooling is just that you really
28:28
expected people to master at least two disciplines.
28:32
So many of the tools for so long required you to convert the code that you were
28:37
writing into physical resources like what is this doing to my CPU and my RAM?
28:42
It's just like,
28:45
okay,
28:46
this is a little too much to ask of your average software engineer.
28:50
SPEAKER_0: Okay, I'll ask an easier question.
28:53
One of the biggest frustrations I think like non-technical people like the
28:57
CEO or CFO have is they look at the bill of an organization.
29:01
Number one is going to be the cloud costs or infrastructure depending if
29:03
you're self-hosting.
29:04
Number two is almost always observability,
29:07
and it just feels like people feel that it gets out of hand.
29:10
And when you ask engineers,
29:12
obviously engineers are told optimize the bill or make it lower.
29:16
SPEAKER_0: Why do you think the costs just get out of hand in so many cases?
29:21
SPEAKER_0: I think if they don't get out of hand, that is the exception.
29:23
SPEAKER_0: What makes it so darn expensive?
29:26
I mean,
29:26
what makes it so darn expensive is the complexity of our systems and our high bar.
29:34
The easiest way to slash observability bill is to give fewer about your
29:38
customer experience.
29:40
And to be clear,
29:43
there's a wide...
29:44
There are companies who every single request,
29:47
delivery companies,
29:48
banks,
29:49
you don't really have a choice.
29:50
SPEAKER_1: You have to understand every single request.
29:52
SPEAKER_1: So there's a certain built-in, and then there are advertising companies.
29:56
SPEAKER_1: It's more spray and pray and you can get buy and buckets.
29:59
SPEAKER_1: And so I think that's a legit angle to look at.
30:03
The second one that I would call out is the thing I mentioned before,
30:05
which is just the multiplier effect.
30:08
How many times are you storing a record of this data for every request that
30:13
enters your assistant?
30:14
I think that Honeycomb,
30:16
we've been building this for maybe nine years on January 1st.
30:21
SPEAKER_0: Wow.
30:22
SPEAKER_1: I know.
30:23
We were so small for several years and in the last couple of years it just taken off.
30:26
But a big part of the driver from 1.0 to 2.0 is people suddenly,
30:31
money's not free anymore.
30:32
And they're taking a look at the multiplier effect,
30:37
and it's just unsustainable.
30:38
No matter how tightly you're trying to keep control of costs,
30:41
it's unsustainable if you're storing 15 different copies for every single request.
30:45
SPEAKER_1: It's just unsustainable.
30:47
The third one that I'll point out is the one that personally bothers me the most,
30:51
which is cardinality.
30:53
If you look at actual observability engineering teams using your
30:56
traditional three pillars or bottles,
30:58
they usually,
30:59
and this is true,
31:00
spend an outright majority of their time trying to govern cardinality.
31:06
You can go to bed Friday night,
31:08
have a $200,000 a month Datadog bill,
31:11
make no code changes over the weekend and wake up Monday with a $2 million a
31:16
month Datadog bill just because the cardinality changed out from under you.
31:20
Can we pause here because I don't think every software engineer will
31:24
actually understand what cardinality does,
31:26
especially if you've not worked with observability?
31:30
SPEAKER_0: Can we break it out?
31:31
SPEAKER_0: What does it mean?
31:32
SPEAKER_0: It is wicked important.
31:35
I know,
31:35
but the first time you meet at it,
31:36
it's not an obvious concept,
31:37
is it?
31:40
SPEAKER_1: Yeah.
31:41
SPEAKER_1: No, not at all.
31:42
Cardinality refers to,
31:44
I think the mathematicians call it the number of unique items in a set.
31:48
And basically it means if you've got a collection of a hundred million users,
31:53
any unique ID is going to be the highest possible cardinality.
31:56
So like request ID or in America,
31:59
like social security numbers,
32:00
if you have a hundred million users,
32:02
you have a hundred million values.
32:05
And then the lowest possible cardinality would be a field with just one value.
32:09
SPEAKER_1: Like species equals human.
32:11
SPEAKER_1: Right?
32:12
SPEAKER_0: Yeah.
32:13
Now,
32:13
the point of big M metrics tools is they are built to handle low cardinality data,
32:18
full stop.
32:20
And there's this very traditional experience that everyone who uses these
32:26
tools which they start using it,
32:29
they're happy with it,
32:30
they append something like host name,
32:33
and then they get to have more than a hundred hosts and suddenly it all breaks
32:37
or it becomes absolutely good.
32:39
SPEAKER_1: And they're just like, "What the hell?"
32:41
And just to explain behind the scenes of why it gets so expensive,
32:45
whatnot.
32:49
SPEAKER_1: There's no relational data in time series databases.
32:53
Every unique combination of number and value you have to store again and again.
33:02
SPEAKER_1: So you have the term custom metrics.
33:07
I always thought that that meant like,
33:09
"Oh,
33:09
a custom metric that you've defined in your code.
33:11
SPEAKER_1: It's a line of code."
33:12
SPEAKER_1: No, it refers to unique combinations of metrics, values.
33:18
SPEAKER_0: So basically every single unique combination will take up more space.
33:21
And this is,
33:22
as you said,
33:23
a rookie mistake I've seen when people use one of these.
33:26
SPEAKER_1: Everybody.
33:27
Many products is you just literally,
33:29
you're like,
33:29
"Okay,
33:30
I don't know,
33:32
I want to query for what city,
33:34
whatcountrydoesthiseventhappen."
33:37
And then they add IP address,
33:39
and then IP address is unique to everyone,
33:41
and suddenly it just adds so much to your bill.
33:45
SPEAKER_1: Yeah, it can like 100X overnight.
33:48
And so world-class observability engineering teams end up spending an
33:52
outright majority of their time just trying to govern,
33:55
because the irony is that is the most valuable data.
33:59
The more unique the data is,
34:00
the more identifying it is,
34:01
which means the easier it is to debug your systems and understand what's happening.
34:05
And it's not just storing the data,
34:07
it's also being able to use it to slice and dice and break down and group by and explore.
34:13
And so you just can't do this with stuff that's built on the metric data type.
34:17
SPEAKER_1: It's actually impossible.
34:20
SPEAKER_0: And this is what we talk about observability 1.0 or so.
34:22
SPEAKER_0: So how does it change?
34:26
It seems,
34:27
this feels a little bit to me like if I had to '60s,
34:33
you actually had to worry about where your program,
34:37
you need to tell where it goes on a tape,
34:38
it goes back and you structured your code accordingly.
34:42
And it feels to me that if you're using these observability products,
34:46
when you're thinking about,
34:47
I want to lock stuff,
34:48
you need to be thinking about how expensive will this be If I add this field,
34:53
which is like an IP or a website,
34:56
oh,
34:56
I shouldn't add that.
34:57
SPEAKER_0: I should transform it, which I mean in 2024, that sounds a bit silly.
35:02
SPEAKER_1: It does, [inaudible 00:35:03].
35:02
Because you're like you're optimizing machine time,
35:05
even though machine time is cheap,
35:07
but I guess storage is expensive.
35:08
SPEAKER_0: And so what's the solution?
35:11
The solution is we have to move away from tools that are backed by big M metrics.
35:17
We have to move towards tools that use structured data where you can have high cardinality,
35:24
where you can store lots of demand.
35:28
SPEAKER_1: I've often said the bridge from 1.0 to 2.0 is logs, right?
35:32
SPEAKER_0: Mh-hmm.
35:33
SPEAKER_1: It's emitting fewer logs, but wider logs.
35:38
The wider your log,
35:39
the more context you're attaching to each event,
35:43
and the more context you have,
35:45
the better your ability to identify outliers and correlate things.
35:49
We have this thing called bubble up,
35:51
which is just like any graph you have,
35:53
you can be like,
35:53
"What'sthat?"
35:54
Draw a little bubble around it and we'll compute for all the dimensions
35:58
inside the bubble versus the base outside the bubble.
36:00
SPEAKER_1: So you're like, "What's this little spike?"
36:02
SPEAKER_1: And you're like, "Oh, I see.
36:04
All of these are for requests coming from Android devices,
36:07
from this region going to the secondary with the batch size of blah,
36:10
blah,
36:10
blah,
36:11
andit'stakingthis."
36:12
And so much of debugging boils down to,
36:14
here's the thing,
36:15
I care about why,
36:16
what are all the ways it's different.
36:18
And gathering up your data in this way,
36:20
you ask why it's so expensive and it's because the model does not fit the needs
36:25
that we have for this data.
36:27
SPEAKER_0: And do I understand correctly?
36:28
SPEAKER_0: Because the software engineers always trade off.
36:30
I think it's easy enough to understand that observability 1.0,
36:34
it optimizes for...
36:37
When it stores the data,
36:38
you can immediately query it and you will get it almost immediately.
36:42
SPEAKER_0: You don't need too much computation.
36:44
When you're doing the opposite of,
36:46
okay,
36:46
I can just store whatever,
36:47
it's not going to expand my storage costs.
36:49
SPEAKER_0: I'm assuming there's a trade-off with, let's say compute, right?
36:52
You're going to compute it later,
36:53
or you'll have post-processing or something,
36:55
right?
36:56
SPEAKER_0: You don't get anything for free.
36:57
SPEAKER_0: No, you're shaking your head.
37:00
SPEAKER_1: So you're right.
37:00
There are always trade-offs,
37:01
but it might not be exactly the trade-offs that you think,
37:03
so this is made possible by the falling costs of storage and compute and all
37:09
these things.
37:09
SPEAKER_1: Absolutely no.
37:11
Metrics were optimized for a world where all of your resources were so expensive.
37:18
SPEAKER_1: You're just like, "I can't afford to store this NGINX log.
37:21
SPEAKER_1: I'm going to derive some metrics and store that."
37:25
SPEAKER_1: Right?
37:26
SPEAKER_1: Yeah, no, you should be able to slice and dice in real time.
37:31
Another aspect of this is I don't think you can really move to 2.0 world
37:34
without taking your wide structured data and feeding it into a columnar store.
37:38
If you're using a traditional relational database,
37:39
you have to define and advance the indexes,
37:43
the schemas,
37:44
all these things.
37:45
SPEAKER_1: You want it to be flexible.
37:46
SPEAKER_1: You want to be able to just drop in, "Oh, this might be useful someday."
37:49
SPEAKER_1: Drop it in and immediately-
37:50
Well,
37:50
yeah,
37:51
for logging,
37:51
you shouldn't have to...
37:52
It feels a royal pain to do it.
37:55
It feels a little bit like typically what happens when you don't have good
37:59
logging or good practices,
38:01
you have a bug,
38:02
something that crashes with the customer,
38:04
you realize you have zero logs,
38:05
you have no way to look at it.
38:06
SPEAKER_0: So what do you do?
38:07
You ship something to production back in a mobile app,
38:10
you add logs,
38:11
and then you tell the customer,
38:12
"We now have logs.
38:13
SPEAKER_0: Can you please update the app and try it again?"
38:15
And it's like,
38:17
I mean,
38:18
it's doable,
38:18
but it's pretty darn embarrassing,
38:19
right?
38:20
SPEAKER_1: Yeah.
38:20
The goal is to be capturing enough rich telemetry all the time that you can go
38:26
back and you can just be like,
38:27
"Oh,
38:27
whatdidtheydo?"
38:28
SPEAKER_1: Right?
38:29
SPEAKER_0: Yeah.
38:30
So a common worry about any observability these days,
38:35
I mean,
38:36
you could build your own observability,
38:37
but unless you're Facebook or Google,
38:40
you probably shouldn't do it.
38:40
And they do it already,
38:42
but there's this worry about vendor lock-in thinking,
38:46
okay,
38:46
whatever I choose,
38:48
I might be locked in or should I try to choose a vendor that's not locked in?
38:51
Now you work for a vendor,
38:52
so you know you're on one end you're going to be biased,
38:55
on the other end you often speak truth to power.
38:59
SPEAKER_0: How big of a deal do you think vendor lock in is?
39:01
SPEAKER_0: Can you avoid it?
39:02
SPEAKER_0: Should you even want to avoid it?
39:04
SPEAKER_0: Is it even possible?
39:06
SPEAKER_1: This is a rare spot of good news in the landscape.
39:08
SPEAKER_1: Historically, this has been a huge problem.
39:12
SPEAKER_1: OpenTelemetry is changing everything.
39:15
The goal with OpenTelemetry has always been you instrument your code with
39:19
OTel and you can basically take your fire hose and point it to whatever vendor
39:24
you want,
39:25
which is,
39:26
there's a little bit,
39:27
but it is 90,
39:15
95% true.
39:30
SPEAKER_1: This is a game changer.
39:31
Forcing vendors to compete for your business based on being excellent and
39:35
responsive and a good value instead of keeping you locked in their ecosystem is,
39:42
and honestly,
39:43
this is the first year where I've really seen this start to come true,
39:46
and it's really exciting.
39:48
SPEAKER_0: It's interesting because my next note was exactly this.
39:50
SPEAKER_0: I made a note, OpenTelemetry.
39:52
SPEAKER_0: I've heard about it.
39:53
SPEAKER_0: And again, don't forget, I'm a bit of an outsider for observability.
39:57
SPEAKER_0: I know software, but not the details of this.
40:00
SPEAKER_0: What is OpenTelemetry?
40:02
SPEAKER_0: You said how great it is, but what is it and why should we care about it?
40:05
Yeah,
40:06
I mean,
40:06
it's the inheritor to Google had OpenCensus and OpenTracing,
40:11
both kind of flopped.
40:13
SPEAKER_1: Ben Sigelman and the folks at LightStep actually expect this out.
40:18
SPEAKER_1: Get this.
40:18
This is also the year where it overtook Kubernetes as the number one CNCF project.
40:23
SPEAKER_0: OpenTelemetry.
40:27
SPEAKER_1: OpenTelemetry is now the top project in terms of [inaudible 00:40:31].
40:31
SPEAKER_1: Yeah, it's huge.
40:32
SPEAKER_1: It's amazing.
40:33
SPEAKER_1: And it's a few years old now.
40:36
SPEAKER_1: It gets critiqued a bit for being kind of big and bloated.
40:41
SPEAKER_1: It does the job it needs to do.
40:45
SPEAKER_1: I think what most people need to understand is this.
40:49
SPEAKER_1: It does a lot of jobs.
40:50
It does them well,
40:51
and increasingly you don't need to understand everything about it to get the
40:55
value out,
40:56
because for a long time it was clunky and you had to really invest into
41:01
understanding it.
41:02
Increasingly,
41:03
it's getting to the point where it just accelerates a lot of the value is you
41:06
get your data even if you don't think about OTel at all.
41:10
If you get your data into an OTel-enabled pipeline,
41:14
it gets consistent naming,
41:17
consistent structure.
41:19
SPEAKER_1: There's semantic conventions and stuff.
41:20
And what this means is then when it gets to the server side,
41:24
your vendors can do amazing shit for it.
41:26
SPEAKER_1: We can intuit, we can derive, we can compute, we can calculate.
41:30
We know what the data is,
41:32
so we can do a lot of really exciting things with it.
41:34
And I think you're going to see a lot more of that in the next couple of years.
41:37
And so just to get a sense of what exactly it is,
41:39
I'm on the website and it says how OpenTelemetry is a collection of APIs,
41:43
SDKs,
41:43
and tools.
41:45
It's built to make telemetry portable and effective and it writes that you
41:50
can use it,
41:50
an instrument generate,
41:51
collect and export telemetry data.
41:53
So do I understand it correctly that a lot of languages or frameworks have
41:57
APIs that you can use and then you can plug in vendors underneath it?
42:02
SPEAKER_0: Or how do the vendors come in this picture here?
42:05
SPEAKER_1: Yeah, some vendors support the own collectors.
42:08
SPEAKER_1: A lot of are contributing back to the core project.
42:12
But basically the idea is however you do your instrument trying to provide
42:19
just frameworks for consistency,
42:20
right?
42:21
SPEAKER_1: It's a little bit too big to just generalize it.
42:24
SPEAKER_1: If you're like, "Does the OTel do this?"
42:25
SPEAKER_1: The answer is probably yes.
42:28
But basically it's just getting people's telemetry into a consistent
42:30
format with consistent naming and semantic conventions,
42:34
which means that we can do a lot of great stuff with it.
42:36
SPEAKER_0: Okay.
42:36
So is it safe to say that if I'm working at now a midsize company or a project
42:41
that I know will need observability or already needs it,
42:44
it's kind of a safe bet to look at OpenTelemetry,
42:48
see if I can at least parts of my pipeline adhere to it,
42:52
because A,
42:53
this will hopefully make it a bit better.
42:55
And then B,
42:56
if the question of portability comes up,
42:57
it'll be way easier to move vendors if it ever comes up because we know that a
43:02
lot of companies will never move,
43:03
butIthinkknowingthatyoucould-
43:06
SPEAKER_1: How do you [inaudible 00:43:08]?
43:08
Well,
43:08
it also helps,
43:09
I mean,
43:09
I'm talking a little bit against possibly your business,
43:13
but as any business,
43:15
the best negotiation is saying,
43:17
"We could move if we wanted to.
43:19
SPEAKER_0: So let's talk about should we do a longer term commitment?
43:23
Do you have new features that you ship that are unique and your competitors
43:26
don'thaveitandweneedit?"
43:27
SPEAKER_0: And I have the same thing.
43:30
It force the vendors to compete on the territory that they should be
43:33
competing on.
43:35
What are some common things that you've seen engineering teams get wrong
43:39
about putting observability in place?
43:41
SPEAKER_0: So one of the most common ones.
43:43
SPEAKER_1: Oh, boy.
43:46
Feeling like they don't need to have any until it's in production and things
43:50
start breaking.
43:52
SPEAKER_1: You really want to be developing with it.
43:54
SPEAKER_1: You want to get in the habit of understanding yourself, like what?
43:57
SPEAKER_1: You're going to attach a GDB to it.
44:02
No,
44:02
you want to debug your software the way you're going to debug it in production,
44:06
like shift left,
44:07
shift right,
44:07
whatever the fuck you want to call it.
44:08
SPEAKER_1: You want to do that early.
44:14
Other areas,
44:16
I think a lot of folks feel like,
44:21
well,
44:22
the dashboards versus...
44:24
A lot of folks get really attached to their dashboards,
44:25
and I really don't feel like unless your dashboard is dynamic and allows you
44:29
to ask questions,
44:30
I feel like it's a really poor view into your software.
44:34
SPEAKER_1: You want to be like, "And then what?
44:35
SPEAKER_1: And then what?
44:36
SPEAKER_1: And then what?"
44:36
SPEAKER_1: You want to be interacting with your data.
44:39
If all you're doing is looking at static dashboards,
44:42
I think it limits your ability to really develop a rich mental model of your software.
44:49
And it means that there are things that you don't think to ask or graph or
44:53
dashboard so you don't see them,
44:55
right?
44:56
Yeah,
44:56
it's interesting because I built so many dashboards and whenever you walk
44:59
into an office where there's a team,
45:02
usually they have a dashboard and it looks cool.
45:03
SPEAKER_0: And all of the dashboards we've had always looked cool.
45:06
SPEAKER_0: And I'll be honest, they were kind of useless.
45:09
I mean,
45:09
they were good when someone important walked by,
45:12
a director or VP,
45:13
they're like,
45:14
"Oh,
45:14
this is the team.
45:15
SPEAKER_0: Here's their sets."
45:16
And after a while,
45:17
what we started to do,
45:17
because we realized that's what we mostly used it for,
45:20
we made sure that nice numbers,
45:21
and it was always green.
45:23
SPEAKER_0: It sounds silly to say it.
45:25
SPEAKER_1: No, I get it.
45:26
SPEAKER_1: I get it.
45:27
Like a public dashboard,
45:27
you didn't really...
45:28
So then we had private ones where we actually had the real stuff.
45:34
Relatedly,
45:35
I think that a habit the more and more teams are picking up,
45:38
which I think is super important,
45:40
is using SLOs as their entry point instead of using dashboards as their entry point.
45:45
SPEAKER_0: Oh.
45:47
SPEAKER_0: So you use it as an entry point for what?
45:49
SPEAKER_1: For understanding, debugging, interacting.
45:53
SLOs,
45:54
I think I have this sticker that I made that's like SLOs are the API's for
45:57
engineering teams,
45:59
and SLO is your agreement.
46:00
You're like,
46:01
"We will provide a level of service that we all agree internally and
46:05
externallyisgood."
46:07
SPEAKER_1: This means we have a budget.
46:08
We can use what's left over in our budget for running chaos engineering experiments.
46:13
One of the things that we did at Honeycomb a few years ago was we had Kafka nodes
46:17
that kept just vanishing on us,
46:20
and it was really frustrating.
46:21
And so we took some of our SLO budget and we started killing Kafka machines
46:28
every day and working on the automated recovery process so that they would.
46:34
SPEAKER_1: And to this day, every Monday, we kill the oldest Kafka node.
46:38
SPEAKER_1: We just shoot it in the head.
46:39
SPEAKER_1: And so we're always testing the bootstrapping process.
46:43
We stopped getting paged in the middle of the night of Kafka nodes because
46:46
we're constantly testing this thing.
46:48
But if you have SLOs,
46:50
it's also the greatest hedge against micromanagement,
46:52
I think.
46:53
Because if you're meeting your obligations,
46:56
then how you spend your time as a team is below the fold.
47:01
SPEAKER_1: Nobody should care about that if you are meeting your obligations.
47:04
It's also a way for you to negotiate and be like,
47:06
"Hey,
47:06
we're not meeting your obligations,
47:08
so we have to put a hold on this feature work because we have to do this
47:11
reliability work because we have to get ourselves back to a place where we can
47:16
deliveronourobligations."
47:17
So there's just so many ways that I think...
47:19
An antipattern I think is when your SLOs are not derived from the same data
47:25
that you're using to debug.
47:27
When it's something that's out there on a satellite it's not connected,
47:30
it's like,
47:30
well,
47:30
now you have one more problem,
47:32
right?
47:32
SPEAKER_1: It's like you really want it to be like, "Here's my SLOs.
47:36
SPEAKER_1: Ooh, I don't understand that my budget is going down faster."
47:39
SPEAKER_1: Click on it and see why.
47:43
I really like you saying that SLOs can be a way to avoid micromanagement,
47:49
and it should be because this is no one's micromanagement.
47:53
SPEAKER_0: I think it's fair, right?
47:54
I think micromanagement is warranted when you're doing a terrible job,
47:57
then the manager or tech leader,
47:59
whoever,
48:01
should come and look at you,
48:02
but otherwise they should leave you alone.
48:03
SPEAKER_0: And I think it's kind of fair.
48:05
SPEAKER_0: It's a good way to think of it.
48:07
I wish more teams would take this and hopefully they can take this as an inspiration.
48:18
SPEAKER_0: So with Honeycomb, it's an observability startup that you're building.
48:23
What was a major exciting or interesting engineering challenge that you
48:28
have to solve to actually build this product?
48:30
SPEAKER_1: I mean, we had to write our own database.
48:33
SPEAKER_0: Your own database.
48:34
SPEAKER_1: Yeah.
48:34
SPEAKER_0: Really?
48:34
SPEAKER_0: You're kidding me.
48:36
SPEAKER_0: Why?
48:37
Because when we started in 2016,
48:40
to be clear,
48:42
I've spent my entire career telling people,
48:44
"Never write a database.
48:46
SPEAKER_1: Don't do it.
48:46
SPEAKER_1: Just never write a database."
48:48
If you think you want to write a database,
48:50
trust me,
48:50
you don't want to write a database.
48:53
SPEAKER_1: But Christine and I got started and we were just like, "Well, shit."
48:57
SPEAKER_1: ClickHouse wasn't around.
49:00
And ironically,
49:01
we would have I guarantee if ClickHouse was around,
49:04
we would've used it.
49:05
And I'm now really grateful that it wasn't and we didn't because being able to...
49:10
The data model is so custom to us,
49:13
right?
49:13
And being able to iterate on it,
49:15
add traces,
49:17
add all these things has been a real force multiplier for us.
49:21
SPEAKER_1: I will say it's why we lost.
49:23
When I was CEO,
49:25
our earliest investor,
49:27
I met with him a year in,
49:28
I was like,
49:28
"Well,
49:28
we're starting to get some interest,
49:30
butwe'regoingtoneedmoremoney."
49:33
He's like,
49:33
"Well,
49:35
if you're going to succeed,
49:36
you would've succeeded by now,
49:37
andI'mnotgivingyouanymoremoney."
49:39
SPEAKER_1: And I'm like, "Well..." And he's like, "Well, you know what?
49:42
SPEAKER_1: You shouldn't have spent all this time fucking around writing a database.
49:45
Youshouldhavefoundproductmarketfitfirstandthenwrittenadatabase."
49:50
SPEAKER_1: And the thing is, as snotty as he was, he's right.
49:54
SPEAKER_1: He's absolutely right.
49:54
SPEAKER_1: That is the common wisdom.
49:56
SPEAKER_1: That is 99 times out of a hundred, that is the right smart thing to do.
50:01
SPEAKER_1: We were too dumb to know better, so we accidentally did the right thing.
50:06
SPEAKER_0: And so how did your database help you?
50:09
SPEAKER_0: What's it called even?
50:10
SPEAKER_0: It's internal to you, right?
50:11
SPEAKER_0: You never opened it up.
50:12
SPEAKER_1: Yeah.
50:12
SPEAKER_1: Yeah.
50:12
SPEAKER_1: It's called Retriever.
50:13
SPEAKER_1: So all of our services are called after dogs.
50:16
We have Dalmatian and Poodle and Retriever and Bassett and Hound and all
50:22
these things.
50:24
SPEAKER_1: How has it helped us?
50:25
SPEAKER_1: I mean, people-
50:29
SPEAKER_0: What kind of database is it?
50:31
SPEAKER_1: It's a columnar store.
50:32
SPEAKER_1: It's a columnar store.
50:33
SPEAKER_1: So it's been through a few different evolutions.
50:36
And actually,
50:36
so Sam Stokes gave a great talk at Strange Loop and I think 2018 about this,
50:44
about the internals.
50:44
SPEAKER_1: People can go and look it up.
50:46
And then a couple years later,
50:47
at the very last Strange Loop,
50:49
Jessitron gave a talk about how we've evolved,
50:52
because at some point around 2020 or so,
50:55
we actually serverless our database.
50:58
SPEAKER_1: We were like, "Okay."
50:58
SPEAKER_1: So initially we're using the columnar store.
51:01
It was all on local SSDs and EC2,
51:06
but the vast majority of data that gets written to disk never gets queried by anyone,
51:10
right?
51:10
SPEAKER_1: And it was really expensive.
51:12
And we're just like,
51:13
"You can't build a business on this where 99% of data never gets queried,
51:17
andyetwehavetopaytostoreit."
51:18
And so Ian Wilkes,
51:20
one of our principal engineers who's been here since the very beginning,
51:23
he actually moved the query planner to Lambda jobs.
51:28
And shortly after the data gets laid down on the SSDs,
51:33
we actually age out to S3,
51:34
and then we do this massive fan out and merge at query time.
51:41
SPEAKER_0: All right.
51:43
SPEAKER_0: That's pretty crazy that you built it.
51:44
SPEAKER_0: I mean, congrats.
51:49
SPEAKER_1: Yeah.
51:49
No,
51:49
I still tell people never write a database,
51:50
but there's an asterisk once in a while,
51:52
youreallycan't-
51:54
Well,
51:55
what I also like about this is there are startups that succeed because they
52:02
don't follow the beaten path.
52:03
In fact,
52:04
some of the more interesting ones I talk with,
52:07
we covered some in the news that I'll link it in the show notes below,
52:10
but there's,
52:12
Figma,
52:12
for example,
52:13
ignored the wisdom of launch in six months.
52:16
SPEAKER_0: They took three or four years to build their first version.
52:19
SPEAKER_0: Again, they burned a lot of money, but it was the right thing to do.
52:22
There was another company,
52:24
Antithesis,
52:26
which built a advanced debugging tool for four years.
52:31
SPEAKER_1: I just talked to the CEO of Antithesis just a few days ago.
52:35
They also built their own database,
52:36
but they did something wild,
52:38
which is they didn't write any tests,
52:39
and they're having their platform tested.
52:41
SPEAKER_0: It is wild.
52:43
SPEAKER_0: But again, he told me the same thing.
52:45
SPEAKER_0: People say, "Don't write your own thing."
52:46
SPEAKER_0: So I think it's kind of a little bit reassuring that-
52:48
SPEAKER_1: I didn't know the rules write the rules.
52:54
When you know where you want to go,
52:54
I mean,
52:55
there's a chance that you might run out of money,
52:57
whatnot,
52:57
but when you know,
52:58
just do it,
52:58
I mean,
52:58
take all the advice,
52:59
but you don't need to control C,
53:00
control V to succeed.
53:05
SPEAKER_1: Yeah.
53:05
SPEAKER_1: No, 100%.
53:05
SPEAKER_1: I'm a big fan of the innovation token metaphor.
53:08
You've got two or three innovation tokens as startups who spend them wisely.
53:12
SPEAKER_1: And we definitely spent two on our internal storage engine.
53:18
SPEAKER_0: Of the three that you had in total back in the time.
53:19
SPEAKER_1: Of the three, yeah.
53:20
But it pays off every time we go to write a feature,
53:23
we're not fighting our storage engine.
53:25
We can build our storage engine to do what we want to do,
53:27
and it's actually an incredible force multiplier.
53:30
SPEAKER_0: Okay.
53:30
So let's jump into a pretty interesting topic,
53:33
which is observability and AI.
53:36
SPEAKER_0: LLMs are a super hot topic these days, and they're everywhere.
53:41
SPEAKER_0: How do you think about observability and AI systems?
53:47
SPEAKER_1: Yeah, that's such a good question.
53:49
So I feel like there are three places where AI really intersects with observability.
53:54
SPEAKER_1: Number one is when you're building and training a model.
53:56
SPEAKER_1: Number two is when you're developing with LLMs.
53:59
And number three is the everyone problem of we're all now dealing with this
54:06
influx of software of unknown origin.
54:08
It used to be you could pretty much guarantee that someone somewhere
54:13
understood the software at some point,
54:15
and you can no longer take that for granted.
54:20
And that is,
54:20
I think,
54:21
so funny because it really hearkens back to the origin story of Honeycomb at Parse.
54:26
We had developers all over the world just writing snippets of JavaScript and
54:29
uploading and we just had to make it work,
54:31
right?
54:31
MongoDB queries,
54:33
they'd write them and upload them,
54:34
we just had to make it work.
54:35
And so many of the things we just sort of forged in fire to understand this
54:40
unknown software is like,
54:41
"Oh,
54:41
nowthisisaworldwideproblemthateveryoneishaving."
54:43
SPEAKER_1: So it's a little fun.
54:46
SPEAKER_1: I wrote a blog post about observability in AI last week, and...
54:51
SPEAKER_0: Oh, cool.
54:52
Yeah,
54:53
it's not...
54:54
I don't think it's super mind-blowing or anything,
54:56
but a couple of the conclusions that we're coming to is basically,
55:00
first of all,
55:00
if you can compute the answer,
55:01
you should probably compute the answer.
55:03
We see a lot of observability vendors out there who are like,
55:06
"AI,
55:07
thisandthat."
55:08
SPEAKER_1: And it's like, yeah, but it's actually, it's a guess.
55:12
And if you had gathered the context,
55:14
then you could have computed it and it would've been faster,
55:16
cheaper,
55:16
easier,
55:17
and better.
55:17
But instead,
55:18
you're like,
55:18
"We put AI on it and it's a get," that's not better.
55:21
SPEAKER_1: It's not better.
55:21
SPEAKER_1: Just it has AI on it.
55:23
SPEAKER_1: AI can make things worse too.
55:26
There are certain problems where AI is the right tool for their job when it
55:29
comes to calculation computation,
55:30
not one of them.
55:34
Another thing that we're really seeing a lot of,
55:37
and unfortunately we have actually some customers who are really
55:42
sophisticated model builders and stuff,
55:44
but everybody in the AI community is so tight-lipped,
55:48
they don't want to talk about anything.
55:51
SPEAKER_1: And it's like, "Guys, come on."
55:54
But I feel like one of the early lessons for Philip and me has been that you
56:01
can't have good AI observability in isolation.
56:06
SPEAKER_1: You have to have it embedded in good software observability.
56:11
There are all these startups out there that are raising just buckets of cash
56:14
to solve the problem of AI observability,
56:16
and they're all focusing on the sort of self-contained models,
56:21
but the inputs from all these different services and data and humans and stuff.
56:27
SPEAKER_1: It's a trace, right?
56:28
SPEAKER_1: It's a trace shaped problem.
56:30
You have to be able to trace it all the way from all these inputs up here in
56:33
software land through the model to the human feedback.
56:39
SPEAKER_1: It's a classic high card now.
56:41
And by AI observability,
56:42
you mean that the problem is like,
56:43
"Okay,
56:43
I have an LLM or a system that uses LLMs in my software and I want to add
56:49
observabilitytoittounderstandhowitworks."
56:52
And you're saying that a lot of the startups are focused on,
56:54
"All right,
56:54
let's just wrap the model into observability inputs and outputs and we'll
56:58
seeit."
56:58
Whereas this thing,
57:00
it actually it is built around other parts of your software.
57:03
You want to see the user interface,
57:05
you want to connect to customer support tickets,
57:08
that kind of stuff.
57:09
SPEAKER_1: Yeah, absolutely.
57:10
SPEAKER_1: It's a trace shaped problem.
57:14
SPEAKER_1: Yeah, it's a trace shape problem.
57:16
It's a high cardinality shaped problem and it's a high dimensionality shape problem.
57:20
SPEAKER_1: This is a software problem with non-deterministic elements.
57:25
SPEAKER_1: It's not an AI problem is how I think of it.
57:28
Yeah,
57:29
well,
57:29
there's going to be a lot of money in it for sure just because how much money
57:33
there is in AI.
57:34
But I really liked your third point,
57:37
which is because you said that the three buckets are,
57:39
number one is observability for these LLM models or when companies are
57:43
writing AI.
57:43
Number two was observability for developers who are writing code using LLMs.
57:48
And number three,
57:49
which I think is the most important is observability for this code generated
57:52
by AI,
57:53
which we know is hard to tell if it's good quality or not,
57:57
but there's going to be just more of it.
57:58
And this is what you said that back at Parse,
58:00
you were used to just adding observability for basically all sorts of code,
58:05
like JavaScript and whatever that people upload it to power their mobile apps.
58:10
SPEAKER_1: Yep, exactly, exactly.
58:12
SPEAKER_1: Production is where code meets reality.
58:15
It's where it doesn't matter how pretty it looked,
58:17
doesn't matter how great.
58:20
You don't know if your code is good or not until you've watched it run in production.
58:24
So I feel like so many things in the age of AI,
58:27
this is not new,
58:29
it's just a really intensified version of what we're actually dealing with.
58:33
SPEAKER_0: Yeah.
58:35
And what it tells me is any company that wants to use these AI agents,
58:40
maybe these new AI agents or just more AI code,
58:44
probably the prerequisite of doing that and not like burning,
58:48
have good observability so that when stuff breaks and from there on,
58:52
it might be a next step of figuring out,
58:54
can there be a feedback loop?
58:55
Can I actually allow some of this AI to push production and all of those things?
58:59
But without that you're going to be flying blind,
59:01
which is just stupid to do with something as unreliable as an LLM.
59:06
SPEAKER_1: Exactly.
59:07
A different topic,
59:08
but also pretty important one,
59:11
every company these days has a choice of building,
59:14
buying or using open source.
59:16
Now in the case of observability,
59:17
usually this goes down to should we buy or should we use open source?
59:21
SPEAKER_0: Because again, building from scratch doesn't really make sense.
59:24
SPEAKER_0: All of these have upsides.
59:25
Do you predict any of these to gain more momentum based on what you're seeing?
59:30
Do you see more companies might be going to use open source and try to host them,
59:35
or maybe more of them are giving up and going to vendors and maybe using OpenTelemetry,
59:40
those kinds of things?
59:42
I think insofar as OpenTelemetry is open source,
59:46
I think it has a really bright future and I'm so relieved to see it when it comes
59:51
to using open source,
59:53
running your own versus using vendors.
59:56
SPEAKER_1: The main trend that I'm seeing in this space is consolidation.
59:59
I think it's part of people just trying to get a handle on their bills,
1:00:03
which totally makes sense.
1:00:05
If you're paying five different vendors and each of those vendors is going
1:00:09
for your 15%,
1:00:10
I feel like a reasonable benchmark for observability spend is like 15 to 20%
1:00:17
of your cloud spend depending on the type of business or whatever.
1:00:21
SPEAKER_1: I think it's a good rule of thumb.
1:00:22
But if you're paying for five different vendors and each of them are gunning
1:00:25
for that 15 to 20% of your cloud spend,
1:00:28
that's nuts.
1:00:30
SPEAKER_1: Right?
1:00:30
SPEAKER_1: You can't do that.
1:00:32
SPEAKER_1: So there's a big consolidation in the industry.
1:00:35
The only open source vendor that's really involved is Grafana,
1:00:39
and Grafana is doing very well.
1:00:40
SPEAKER_1: They just raised a huge round.
1:00:44
SPEAKER_1: I think different models... Although I guess should count Prometheus too.
1:00:51
I think of Prometheus and Datadog as the last best capital metrics products
1:00:58
that will ever be built.
1:01:01
Nobody's ever going to try and launch another big project because what's the point?
1:01:05
These are mature,
1:01:06
they're great,
1:01:07
honestly,
1:01:07
and they're just very mature technologies and metrics,
1:01:13
for all the shit talking that I do,
1:01:15
metrics have a place in the ecosystem right now.
1:01:19
It's like 80% of what people use is metrics and 20% is structured data and they
1:01:23
need to invert that.
1:01:24
SPEAKER_1: It needs to be 80% structured data and 20% metrics.
1:01:27
But metrics are still great for cheaply plotting trends over long periods of
1:01:35
time or for counters.
1:01:39
Counters are a very essential metric thing or at a certain scale,
1:01:46
which is a lot higher than a lot of people think.
1:01:49
SPEAKER_1: Structured data is too expensive and you should use metrics.
1:01:51
SPEAKER_1: So there are niche use cases for metrics.
1:01:53
SPEAKER_1: And so Prometheus I think will continue to be a contender.
1:01:55
SPEAKER_1: I think Grafana is great.
1:01:57
I think that Datadog,
1:01:59
et cetera,
1:02:00
but in general,
1:02:01
most people are using vendors and most people don't want to have to deal with
1:02:04
this stuff under the hood.
1:02:06
When everything breaks at 2:00 A.M.,
1:02:08
you don't want to also be dealing with a broken observability tool on top of
1:02:12
your other broken software.
1:02:15
SPEAKER_1: So I mean, it makes sense to me.
1:02:17
SPEAKER_1: And I'm not saying that just because I'm a vendor.
1:02:21
SPEAKER_1: I mean, I wouldn't say that, I wouldn't.
1:02:24
SPEAKER_0: Yeah, a question I got actually from a reader on social media.
1:02:28
SPEAKER_0: What about front end and mobile observability?
1:02:31
Because when we talk about it,
1:02:32
it feels so much a bit as about the backend.
1:02:35
SPEAKER_1: Yes.
1:02:36
SPEAKER_1: Yes.
1:02:37
So we actually launched our RUM replacement this year for front end trying to
1:02:43
do the same thing for RUM that we have done for the backend.
1:02:47
SPEAKER_0: What does RUM stand for?
1:02:48
SPEAKER_1: Real user monitoring, and it's basically the front end.
1:02:52
It's organized around browser sessions and user sessions instead of
1:02:56
backend requests.
1:02:57
But I think it's so critical that I feel like so often the borders of tools are
1:03:02
what create silos.
1:03:04
So if you've got one team over here that's using a completely different view
1:03:08
into their software,
1:03:09
then the other teams,
1:03:11
it's like you spend more time arguing about the nature of reality than trying
1:03:14
to solve the problem together when you have a common...
1:03:17
Again,
1:03:18
back to that unified storage,
1:03:19
right?
1:03:20
Many different views,
1:03:21
many different entry points,
1:03:22
but a unified view all the way from your mobile device or your browser to the
1:03:27
database and back,
1:03:28
it's such a powerful thing.
1:03:32
Mobile is a different beast and we've started dipping our toe into it this year.
1:03:39
It's no surprise that mobile is the only standalone solution out there
1:03:45
that's sort of left.
1:03:45
SPEAKER_1: And I think a lot of folks are trying to figure out how.
1:03:47
I mean,
1:03:48
you come from a mobile background,
1:03:49
so you probably know all the reasons for this better than I do.
1:03:52
SPEAKER_0: Yeah.
1:03:52
SPEAKER_0: I mean, right, there was Crashlytics, which was acquired.
1:03:56
It's such a weird thing because what I've heard is Crashlytics around like
1:03:56
2012 or something was acquired by Twitter and for a while kept alive and then
1:04:06
it was destroyed.
1:04:07
SPEAKER_0: And two things happened there.
1:04:08
I think they bought it for about $300 million,
1:04:09
and apparently it was such a low return that the VCs really got spooked and
1:04:14
they never invested in another company that was doing this because they said
1:04:17
if this is the biggest exit done was there and it always feels that mobile has
1:04:22
been left by itself.
1:04:24
SPEAKER_0: There are some tools, but none of them are really first class.
1:04:27
None of them really come from the vendors that are doing the proper backend
1:04:32
observability monitoring.
1:04:33
And then I think everyone thinks it's such a small market,
1:04:37
which I disagree with honestly.
1:04:39
It's not small,
1:04:40
but I do have a theory for why this is,
1:04:43
and it's because the build pipeline is so alien and different because you
1:04:49
can't use CI/CD,
1:04:50
right?
1:04:50
You've got the Apple Store gating or you've got the Android diaspora and the
1:04:55
inability to fold it into the best practices of software development,
1:05:01
I think that's why it's out in the island by itself.
1:05:03
No,
1:05:04
mobile remains surprisingly archaic in this,
1:05:07
and a lot of it has to do with Apple and then Google going along,
1:05:11
not allowing binary code to be shipped directly without their permission.
1:05:16
Obviously companies,
1:05:19
every team is going around,
1:05:20
they're doing feature flags and there's some JavaScript here,
1:05:23
JavaScript there,
1:05:24
but it's all hush,
1:05:24
hush under the hood.
1:05:26
SPEAKER_0: Apple kind of knows, but it's different.
1:05:29
SPEAKER_0: It's a fun, fun world.
1:05:33
SPEAKER_0: Every area has its own challenges.
1:05:35
SPEAKER_1: Every are does.
1:05:36
SPEAKER_1: Yeah.
1:05:37
And the last question I got from,
1:05:40
again,
1:05:40
someone on social media from this person wrote,
1:05:46
"I'm starting a new company today.
1:05:48
What is the right time to start investing in observability and how can I
1:05:52
designforitupfront?"
1:05:53
SPEAKER_0: Startup, fresh idea.
1:05:58
SPEAKER_1: I think it's as integral to building software as tests.
1:06:02
SPEAKER_1: I do.
1:06:03
I think it's the same sort of...
1:06:05
The best time to instrument,
1:06:08
the best time to write tests,
1:06:09
the best time to instrument is while you're writing the code.
1:06:11
Anything that you try to slap on after the fact is not going to get at that
1:06:15
original intent of what you were trying to do when you had it in your head.
1:06:20
And I think that done correctly,
1:06:21
it actually accelerates your development.
1:06:24
SPEAKER_1: It doesn't hold you back.
1:06:25
It accelerates your ability to get stuff out,
1:06:28
to understand,
1:06:29
to keep moving.
1:06:30
And so yeah,
1:06:31
I would say as soon as the code that you're writing is real,
1:06:35
something you intend to put in front of a user,
1:06:37
as soon as you start thinking about writing tests,
1:06:40
you should be thinking about writing observability.
1:06:44
It sounds like to me we're saying when you're prototyping and you're doing
1:06:47
throw away stuff,
1:06:48
and again,
1:06:48
you wouldn't write any tests,
1:06:50
don't worry about it,
1:06:51
but when you're like,
1:06:52
"Okay,
1:06:52
this might go out," I really like the test analogy.
1:06:55
SPEAKER_0: And you know the interesting thing with tests is?
1:06:57
SPEAKER_0: There's two sides of people.
1:06:58
SPEAKER_0: When you say, "Oh, tests will speed you up the long term."
1:07:03
People who haven't written tests or haven't seen it,
1:07:05
they're like,
1:07:05
"That's BS.
1:07:06
SPEAKER_0: It takes time.
1:07:07
SPEAKER_0: It cannot slow me.
1:07:08
SPEAKER_0: I'm just going to skip it."
1:07:10
And then people who've seen it,
1:07:11
they're like,
1:07:11
"No,
1:07:11
no,
1:07:12
no,
1:07:12
youdonotunderstand."
1:07:14
And now you see the two startup founders,
1:07:16
one of them is starting by writing the tests and they actually do get sped up
1:07:20
where the other one says,
1:07:21
"It'ssilly."
1:07:22
SPEAKER_0: There's a comic about pushing the car on the square wheel.
1:07:29
SPEAKER_0: Someone brings a round one saying, "Oh, let's change it."
1:07:31
SPEAKER_0: "No, no, no.
1:07:32
SPEAKER_0: It will take too much time."
1:07:34
SPEAKER_1: So real.
1:07:36
SPEAKER_0: I feel this might be the case with observability.
1:07:39
SPEAKER_0: Once you've seen it, you probably cannot unsee it.
1:07:41
SPEAKER_1: Exactly.
1:07:43
SPEAKER_0: So let's close with some rapid questions.
1:07:45
SPEAKER_0: I'll just shoot a question and you tell me whatever comes to mind.
1:07:50
SPEAKER_0: People told me I need to ask something around management for you.
1:07:53
SPEAKER_0: So I'll ask, do you like being an engineer or a manager more?
1:07:58
SPEAKER_1: I love being an engineer.
1:08:00
SPEAKER_1: I love it.
1:08:01
One of the hardest things about the early years of Honeycomb were really rough.
1:08:05
I didn't expect to have to be CEO and I wasn't a very good one,
1:08:09
but so much of my identity came from being an engineer,
1:08:12
and it was really,
1:08:13
really hard for me to move past that.
1:08:15
I'll say that when I was an engineering manager,
1:08:17
I hated it,
1:08:18
but now I look back,
1:08:19
I'm like,
1:08:19
"Oh,
1:08:19
I kind of missed that," which is why I actually have this open calendar link
1:08:23
where people can set up time to just bring their problems to me and talk about them,
1:08:27
and I get such a kick out of it.
1:08:28
SPEAKER_1: I miss those aspects of doing engineering management.
1:08:31
SPEAKER_1: But I think being an engineer is you get paid to solve puzzles all day.
1:08:37
That dopamine hit of just figuring things out,
1:08:40
making things work,
1:08:41
yet I would just go around feeling high all day.
1:08:43
SPEAKER_1: It was really fun.
1:08:46
SPEAKER_0: Yeah.
1:08:48
SPEAKER_0: What is a controversial thing that you believe is true?
1:08:52
SPEAKER_1: Absolutely nothing.
1:08:53
SPEAKER_1: Gergely, how could you say such a thing to me?
1:08:58
SPEAKER_0: What am I thinking?
1:08:59
SPEAKER_0: All right.
1:09:00
SPEAKER_1: What are you thinking?
1:09:01
SPEAKER_1: A controversial thing that I believe to be true.
1:09:03
I actually wrote an article yesterday I got out something about founder mode
1:09:09
and I...
1:09:13
So I'm not sure if this counts as controversial or not,
1:09:16
but I think it counts as controversial is Silicon Valley like YC.
1:09:20
I do not think that it's a good idea for the CEO to have to approve everything
1:09:24
that goes out.
1:09:24
SPEAKER_1: I think that is egotistical.
1:09:27
SPEAKER_1: I think it is wrong.
1:09:29
I think it hobbles good decision making and judgment in other parts of the organization.
1:09:34
SPEAKER_1: I wouldn't want to work for someone like that.
1:09:37
There's this group of folks who just idolize Steve Jobs,
1:09:40
Gergely,
1:09:41
and I think that Steve Jobs obviously very successful person,
1:09:44
and I think in large part,
1:09:46
despite the fact that he was a raging control freak asshole,
1:09:49
I think he was successful despite that,
1:09:51
not because of that.
1:09:52
It makes me really sad to see so many bright people be like,
1:09:55
"Oh,
1:09:56
well,
1:09:56
Ishouldalsobeanassholeandcontrolfreak."
1:09:59
Think of how many brilliant,
1:10:00
wonderful people probably left because they couldn't stand to be
1:10:03
micromanaged in that way.
1:10:04
Well,
1:10:05
I think it also comes to show there's just so many ways to succeed that there's
1:10:09
no one way.
1:10:11
SPEAKER_1: There's so many ways, there's so many ways.
1:10:11
SPEAKER_0: Then also to be fair, so many personalities, right?
1:10:13
SPEAKER_0: So I'm going to just do a wild stab.
1:10:17
SPEAKER_0: You don't operate like that, right?
1:10:21
SPEAKER_1: No, you don't.
1:10:21
For better or for worse,
1:10:22
I've been a worker and an employee much longer than I've been a founder at a C-level.
1:10:27
SPEAKER_1: And I don't know.
1:10:29
It's so clear to me that the way that you bring out greatness in people is by
1:10:34
supporting them and empowering them and giving them agency and giving them control.
1:10:38
SPEAKER_1: And yes, be in the details.
1:10:41
The advice of hire great people and get out of their way is terrible advice.
1:10:46
You need to be doing the work to create alignment,
1:10:48
to make sure that you have a shared view of what good looks like,
1:10:51
what great looks like,
1:10:52
so that you can course correct early when you start to diverge.
1:10:55
So yes,
1:10:56
be in the details,
1:10:57
but you don't take people's agency away from them.
1:11:00
So if you were not building an observability startup,
1:11:04
what would you be doing?
1:11:05
SPEAKER_1: I would be a staff engineer someplace.
1:11:07
SPEAKER_1: In fact, that's what I plan to do next, is just go be an 5:00 P.M.
1:11:12
I go home and I turn my brain off and it's like,
1:11:14
"It's your problem now,
1:11:15
bud."
1:11:16
SPEAKER_1: It's going to be amazing.
1:11:20
For some reason,
1:11:21
I want to believe you,
1:11:22
but I don't think you're going to be that even later.
1:11:25
SPEAKER_0: The turning of the brain at 5:00 P.M.
1:11:29
SPEAKER_0: I mean specifically.
1:11:30
SPEAKER_1: Oh, that part, yeah.
1:11:31
SPEAKER_1: Yeah.
1:11:31
SPEAKER_1: Maybe not.
1:11:33
SPEAKER_1: I mean it though.
1:11:34
SPEAKER_1: I plan on going and being an engineer for a while.
1:11:36
SPEAKER_1: I think you can only... I hate the term thought leader.
1:11:40
It just makes me a little nauseous and I really think that the farther
1:11:44
detached you get from the work,
1:11:47
just the lower quality.
1:11:48
SPEAKER_1: I don't know.
1:11:49
For me,
1:11:49
at some point,
1:11:50
I need to circle back and do something with my hands in order to not cringe when
1:11:54
I hear myself speaking.
1:11:57
SPEAKER_0: I absolutely hear you.
1:11:58
SPEAKER_0: So you're a big fan of whiskey.
1:12:02
SPEAKER_1: I am.
1:12:02
SPEAKER_0: What is your current favorite?
1:12:03
For the longest time,
1:12:04
I really like peated Scotches and it,
1:12:07
I'm still kind of coming to terms with my identity as being more of a bourbon
1:12:11
person now.
1:12:13
SPEAKER_1: Yeah, no, interesting.
1:12:15
SPEAKER_1: I really like WhistlePig, and I really like the Rye WhistlePig too.
1:12:23
SPEAKER_1: My favorite ever is impossible to find now though.
1:12:26
SPEAKER_1: It's called George T. Stagg, and it is like 190 proof.
1:12:30
SPEAKER_1: It's so good.
1:12:32
You can't find it,
1:12:33
but they started making something called Stagg Jr.,
1:12:35
which is like 80% is good.
1:12:38
SPEAKER_1: So if you haven't tried it, try Stagg Jr.
1:12:40
SPEAKER_0: All right.
1:12:40
SPEAKER_0: And what's a nonfiction book you'd recommend for software engineers?
1:12:44
SPEAKER_1: The book is called Fluke by Brian Klaas.
1:12:49
SPEAKER_1: Fluke: Chance, Chaos, and Why Everything We Do Matters.
1:12:53
I'm not a religious person,
1:12:54
but I do really look for ways of having meaning in my life.
1:13:00
And the takeaway from you is that everything we do really does matter because
1:13:06
you don't know when the thing...
1:13:08
A straight comment that you drop in someone's presence sets off something in
1:13:12
their head that changes their life or causes them to stop happening or causes
1:13:17
them to get sober,
1:13:18
or all these ripple effects happen.
1:13:21
Like the things that we do and say,
1:13:24
yeah,
1:13:24
maybe 95 don't actually set off,
1:13:29
but things that we do really do matter.
1:13:31
SPEAKER_1: It's called Fluke.
1:13:33
SPEAKER_1: I've read it three times in the past year.
1:13:34
SPEAKER_1: Cannot recommend it highly enough.
1:13:38
SPEAKER_1: Yeah.
1:13:41
Well,
1:13:41
Charity,
1:13:42
this was a really interesting and fun conversation as usual.
1:13:46
SPEAKER_1: Thank you so much.
1:13:46
SPEAKER_1: It's so nice to see you.
1:13:47
SPEAKER_1: I wish we lived closer so we could get together more often.
1:13:50
It's always a delight getting to spend a little bit of time just shooting the shit.
1:13:54
SPEAKER_0: Thank you to Charity for sharing all these interesting details.
1:13:57
Charity is a prolific writer,
1:13:58
and you can read more from her on her blog at charity.wtf,
1:14:02
which is linked in the show notes below.
1:14:04
For a deep dive on how to build an observability startup and for details on how
1:14:08
a scale up managed to have a $65 million observability built for a single year.
1:14:14
SPEAKER_0: See The Pragmatic Engineer Deepdives linked in the show notes below.
1:14:17
If you enjoyed this podcast,
1:14:18
please do subscribe on your favorite podcast platform and on YouTube.
1:14:22
SPEAKER_0: Thank you and see you in the next one.