Latest / Agentic Conversations (formally mlops.community) / Build Reliable Systems with Chaos Engineering // Benjamin Wilms // #237

Transcript

Build Reliable Systems with Chaos Engineering // Benjamin Wilms // #237 · Agentic Conversations (formally mlops.community)

0:00
My name is Benjamin Wimps. I'm the cofounder and CEO of
0:04
Steady Bid, a chaos engineering platform, and you can call me
0:08
the paranoid guy because I don't trust any system.
0:11
I need to see how the system reacts under any conditions.
0:14
Coffee. Let's talk about coffee.
0:16
So there are two types of coffee.
0:17
I look like like the most in the early morning.
0:20
There's one with milk and sugar. Don't, don't judge me.
0:24
And of course, there's espresso. Just pure espresso.
0:28
What is happening in Mobs community?
0:30
This is another one of them good old podcasts and I am your host
0:34
Dimitri OST today talking to Benjamin and I wanted to have
0:37
him on here because I feel like what he is doing is so closely
0:44
related and needed in the Milos world.
0:48
It's just not something that we talk about that much yet.
0:52
Yet maybe. And I think that he has some
0:57
really unique ideas around running experiments to try and
1:01
purposefully inflict chaos on your systems to make sure you
1:06
know and understand where things can fail.
1:09
And when you do that, you then get to go and you upgrade your
1:14
systems and you continue to run the experiment until you feel
1:17
like you've got it down. And then you just row that
1:21
experiment and all of those learnings into a test that is
1:25
part of your CI CD pipeline. Benjamin explained it so well.
1:30
It was a great conversation. Hope you enjoy it.
1:32
Let's get into the episode. And one thing that he did say
1:35
that I thought was so cool is I usually ask people before we
1:40
start recording, I say what would be the ideal outcome of
1:44
this episode? And he said for him, one of the
1:48
goals that he wanted to inspire in people is to just have
1:54
somebody that has never heard of chaos engineering go down the
1:58
rabbit hole and look deeper at it and maybe start incorporating
2:02
it into their repertoire of tools that they can use to keep
2:06
their systems more reliable. Let's talk with Benjamin.
2:09
And if you enjoyed this episode, you know it means the world to
2:13
me if you leave a star, or if you give us a review or my
2:19
favourite, share it with one close friend.
2:23
I'll see you all later. All right?
2:31
Benjamin, you are not too far away from me.
2:33
You're almost like my neighbour in Poland.
2:36
I'm in Frankfurt. I think you're just a little bit
2:39
further north. Yeah, it's just a yeah, a 45
2:43
minute ride with the famous German Deutsche Bahn.
2:46
Yes. That is never late or never has
2:49
any problems. We all know.
2:50
That yeah, we all know that, yes.
2:53
And so I, I wanted to get you on here because I wanted to talk
2:56
about chaos engineering and what better of a person to talk about
2:59
chaos engineering than the person who built Chaos Monkey
3:02
for Spring Boot. And for those uninitiated people
3:05
who don't know what Spring Boot is, it's a framework for Java.
3:08
What is Kdot chaos engineering? Can you give us the TLDR of it?
3:13
Yes, for sure. So the real need behind chaos
3:18
engineering is that people are building very complex and
3:21
distributed systems and systems is not only like something from
3:25
the technology point of view, it's really something where a
3:28
lot of interaction is is is getting is getting started or is
3:32
inside of that system. So it's very hard to understand
3:36
how the system works and how you should react under specific
3:42
conditions. So it's really hard to
3:43
understand what's going on inside of that system.
3:47
Now with cash engineering, you can train yourself, you can
3:51
train your organisation, you can train your system, your
3:53
technology to handle specific conditions.
3:57
And most of the time we are talking about some bad
4:00
conditions, bad conditions from production like a latency spike,
4:04
some delay in the in the traffic, maybe an area of your
4:07
network is no longer responding or maybe something went wrong
4:10
during your deployment. So really like those moments
4:13
where you are under high stress and you would like not to get in
4:18
the same moment again and again. And now with chaos engineering,
4:22
you can inject those bad moments, but in a safe
4:26
environment. You can do it in a preproduction
4:28
stage, You can be more proactive and you can learn from it.
4:32
You can train your system and yourself.
4:35
And so I've been kind of on this kick about change management and
4:38
how many processes there are in software development.
4:42
When you make changes to code, you want to make sure that it is
4:46
in the most reliable way possible.
4:49
Correct. And so this kind of falls under
4:52
that category. You get to not have to test in
4:55
prod, right? You can't do things beforehand,
4:59
hopefully. You should, because otherwise
5:01
it's too late. Exactly.
5:03
And it's funny because these are high stress situations that you
5:06
were talking about. I think they tend to happen at
5:09
like 3:00 AM when you have one person on call and they get
5:15
woken up in the middle of the night.
5:16
So is that the idea like less 3:00 AM calls?
5:22
Correct, correct. That's like that's the outcome.
5:24
And but the more valuable outcome is for yourself and of
5:28
course for your customers. They, your customers are still
5:31
able to use your product. They are able to use your
5:35
service you are offering. And to be honest, your customers
5:38
don't care about any technical issues.
5:40
They don't care about AWS or Azure or Google issues.
5:45
They would like to buy something on your marketplace.
5:48
They would like to interact with your app on your mobile phone.
5:51
And today's market conditions are quite tough because if your
5:54
customer is not able to do it, they will continue.
5:57
They will go to your competitor, say we'll find a way to get
6:00
their product. And so you need to be able to
6:04
handle those outages. You need to be able to make your
6:07
customer happy. This will lead to revenue and
6:11
happy customers. And you mentioned complex
6:14
systems and how valuable this is for complex systems.
6:17
Is it also useful for like the smaller startups that are just
6:21
getting rocking and don't have that complex of systems in
6:24
place? Yes, you are correct.
6:27
Why? Let's imagine you're a young
6:30
startup. You as a young startup, you
6:32
would like to get your idea out of the market and you're highly
6:36
focused on, on, on, on specific use cases, something you would
6:40
like to implement. And maybe as a young startup you
6:43
don't should, you're not able because of missing resources to
6:47
invest a lot of energy in reliability engineering to make
6:52
it more perfect. So if something is there that
6:57
you can just ask, OK, how safe I am, how big is the risk I'm
7:01
taking to go into production with this new feature?
7:03
Is it still so that this system with a new feature can handle in
7:08
sole outage of my cloud environment, can handle some
7:11
hiccups in incubators. That's something you as a young
7:15
startup needs. You need to be able to see it
7:18
and yeah to to handle the risk. Now I was talking to a CTO of a
7:24
big technology company and I was talking to them about chaos
7:28
engineering and what they thought about it.
7:29
And one thing that they said was they don't wanna know how
7:34
brittle their system is. Yeah.
7:37
What does he say to that? Because it's like, yeah, it's
7:40
almost like they know that things are hanging on by a
7:42
shoestring, but let's not mess things up because then we have
7:47
to go and we have to stop and backtrack and make it more
7:51
robust. OK, I need to be careful now
7:54
because this story is something I also know maybe from a
7:58
different company, but so what's the story behind?
8:01
And there was a company reaching out to us.
8:04
It was an inbuilt request, it was the inbound lead.
8:07
And they told us our man, our system is so under pressure and
8:11
we as engineers and sores we, we realised we created a monster
8:16
and this monster is our system and we are not able to handle it
8:19
anymore. O we need to do something with
8:22
scarce engineering. We would like to get rid of all
8:24
those incidents. We are spending hours each day
8:27
on fixing something. So we would like to turn this.
8:32
So they did a PC with us, they did the POC in their internal
8:35
system and they identified a lot of risk in the system which they
8:40
are we are not able to handle. So next step OC was done.
8:46
Let's get a report out of the OC.
8:49
Let's resent this to our management and the management
8:51
was very impressed about the findings about this huge list of
8:57
risk in the system. But there was no commitment
9:01
coming from the management. They told the team, no, let's
9:05
focus on, on, on, on on fire drills that focus on you need to
9:09
be more reactive. We should spend more money on on
9:13
people that are taking care about all those files in your
9:17
system. But don't fix it.
9:18
No, no, let's continue the old way.
9:21
So it was really, it is still really hard for me to understand
9:25
if you are aware of such a big risk in your system, which can
9:28
lead to a many bad scenarios for you as a company.
9:33
That's why you are not choosing the right way from my
9:37
perspective. Yeah.
9:39
It does feel like one thing that you were mentioning is that it's
9:44
just as much organisational as it is technical.
9:47
It is part of the culture, it is part of the motivation.
9:51
So you need to ask yourself, what is, is there, let's say, a
9:55
compensation plan for some levels in your company?
9:58
What? What's in such a plan?
10:01
Is there any motivation to to improve something from, from
10:07
specific levels and where is the pain located and how much
10:12
momentum can this group of people, whether pain is located,
10:16
create to improve something? And this reflects into the
10:20
culture. And most of the companies we
10:22
spoke to and we are still engaged with are, are building
10:25
such a failure culture where you are improving because you
10:31
failed, where you can learn from failures and where you're when
10:34
no one is doing something with like finger pointing or that's
10:37
bad, it's your job. It's not my fault, so it's
10:41
really deep in the culture for company.
10:43
Yeah, those postmortem, the blameless postmortem, blameless
10:48
1. One thing that I also think
10:51
about is how you play a lot in the DevOps sphere, and this what
10:57
you're talking about is a lot and it has to do a lot with the
11:00
infrastructure of code or just like the infrastructure of a
11:06
company or a tech, The technological piece of it, yes.
11:10
Do you also see a lot of people wanting to do this with data?
11:16
And basically the way that I envision, can I just tell you
11:19
how I envision? Yeah, please.
11:20
Yeah. Engineering.
11:21
And I think chaos monkey is my favourite way of envisioning it
11:25
because it's like there's a monkey that got loose in your
11:28
system and it's just turning off and on a bunch of different
11:31
pieces of your system and you get to see how resilient that
11:35
is. And so I wonder, do you also see
11:38
people wanting to come and say, can you just turn off and on my
11:41
databases or can you make data flows go the wrong way or stuff
11:45
like that to simulate that trouble?
11:48
I mean, to be honest, a lot of people probably say I don't even
11:51
need a chaos monkey. There's a few people at my
11:54
company that do that on their own.
11:57
I don't need some special software for that.
11:59
But I want to hear what your story on the data side of things
12:02
too. Let me start was we see data
12:05
side because also I would like to talk about the chaos monkey
12:07
side and from the data perspective chaos engineering as
12:13
we are doing it in, in, in, in our region of study, but is not
12:16
limited to only infrastructure or hardware components.
12:19
It is really like a multi level approach for what I'm talking
12:23
about. You are getting started on on a
12:25
on bare metal, on a virtual machine.
12:28
A lot of stuff can go wrong on that machine on many different
12:31
levels like your network, resource consumption, storage.
12:37
But then also you can go up the levels where you would like to
12:41
do something on the platform level.
12:43
And platform could be like Kubernetes or could be a
12:46
distributed database system or even go higher.
12:50
Until you are on the application level, you are going through the
12:54
cloud provider level. So on many different levels
12:57
something can go wrong and also on many different levels you
13:00
need to inject those behaviour, which also means that people are
13:06
doing it on the database side. They are injecting latency on
13:11
only all reading requests to the database or they are trying to
13:16
do how to say database shutdown but also like bring UA new
13:24
instance, how fast is the data but distributed to the new
13:27
instance is is a hot standby whatever.
13:29
So they are running those so systems and tests and even on
13:34
the network layer, so our customers are doing on Kafka
13:37
something on red is something they are handling with the
13:40
messages. They are holding back some
13:42
messages and that newer messages can bypass source messages over
13:47
the network that maybe let's say an update event of an the
13:51
customer is executed before the create event as a simple
13:55
example. So you can do that every level
13:58
is the short answer. Getting to the Chaos Monkey
14:01
site. I was and I'm still a big fan of
14:04
Chaos Monkey. Why the core concept?
14:07
The core idea behind the Chaos Monkey was all the engineers,
14:11
all the product engineering teams.
14:14
They are enforced to do something, to be more reliable.
14:18
Why? Because they know the Chaos
14:20
Monkey is running and I've got no idea what this freaking tool
14:24
is doing and at what point in time.
14:26
I don't want to followed on a Sunday morning at 3:00 AM or
14:32
whatever. So, ha, it's really like this.
14:35
You need to be prepared for those moments.
14:37
And that was a great job done by the Chaos Monkey.
14:40
Now after a couple of years the chaos engineering approach is
14:45
more shifting into um science or running experiments where you
14:53
are always asking yourself is a system still healthy or not and
14:59
if not, the system should not be part of an experiment.
15:02
So you need to find an chaos engineering solution that is
15:06
able to understand and interact with your system.
15:09
To answer the question if my system is still healthy.
15:12
If not, the experiment needs to stop immediately and should do a
15:15
rollback, which the chaos monkey doesn't care and he's always
15:19
like hitting your system. And that's like the big
15:20
difference between the old approach of random injecting bad
15:24
behaviour ends in nowadays approach of chaos engineering
15:27
platforms. Yeah.
15:29
So I can see the value of that, especially when you're talking
15:32
about on that first part, if you're doing stuff with Kafka
15:37
and you're recognising data streams are being piped in in
15:42
the right time or you're holding back data streams.
15:44
And then for a machine learning model, that could be very
15:48
useful, especially if it's real time and it needs those data
15:51
streams and it needs that. So how do you make sure that the
15:55
machine learning model that's real time downstream is still
15:59
effective if all this upstream chaos is happening?
16:02
Correct. And then on on the flip side,
16:06
the chaos monkey versus the science and experimentation, it
16:09
does seem like a very clear distinction and it seems like a
16:13
more mature way of doing it. Like the chaos monkey was the
16:15
infant way of doing it. And now we're kind of gotten
16:19
into, we've graduated and we're in high school.
16:22
That's exactly. And so we can be a bit more
16:27
focused on how we create those those different experiments.
16:31
And we can also know if we have a hunch that I think this side
16:36
of the system is a little bit unreliable or I've seen or I've
16:40
heard about something happening with my friend's system.
16:43
In this way you can run an experiment and make it happen
16:46
and it's more directed. Exactly.
16:48
And that's a, that's a very good point.
16:51
What people are doing is there is an experiment and this has
16:54
been yeah, run the first time in in one area of the system of of
16:59
their system and it was successful.
17:01
Now they are sharing it with other teams inside because
17:05
nowadays they are all running on one central platform.
17:08
Maybe it's the Kubernetes based platform, I don't know.
17:10
But OK, this team was quite successful to now handle all
17:15
incidents with let's say some storage issues.
17:20
Now let's use this collection of experiments they have executed
17:24
with success and hand over those collection to another team that
17:28
they can do it on their applications they are running
17:31
and doing. And that's like really this
17:35
sharing is caring moment where people are able to help each
17:38
other and they are helping each other because there was a bad
17:42
moment in outage and they can learn so much from those
17:46
outages, which is, yeah, driving a strong value.
17:49
Yeah, learn from my mishaps. And so hopefully you don't have
17:53
to go through them yourself. Exactly.
17:57
All right, real fast, I want to tell you about the sponsor of
17:59
today's episode, AWS Tranium and Influencia.
18:02
Are you stuck in the performance cost trade off when training and
18:05
deploying your generative AI applications?
18:08
Well, good news is you're not alone.
18:11
Look no further than AW's Tranium and Influencia, the
18:16
optimal infrastructure for generative AI.
18:20
AWS, Tranium and Influencia provide high performance compute
18:25
infrastructure for large scale training and inference with
18:29
Ellms and diffusion models. And here's the kicker, you can
18:34
save up to 50% on training costs and up to 40% on inference
18:40
costs. That is 50 with A5 and A0.
18:44
Whoo. That's kind of a big number.
18:46
Get started today by using your existing code and frameworks
18:50
such as Pytorch and Tensorflow. That is AWS, Tranium and
18:55
Influencia. Check it out.
18:56
And now let's get back into the show.
18:59
And one thing I also think about is how have you seen people
19:03
doing this ad hoc, if at all? Like if I didn't have a tool
19:08
that should do this, what would I do?
19:11
Turning off a machine shell command, shutdown, whatever,
19:14
that's the easiest way to do. Or if you are able to unplug the
19:18
power, yeah, do it. But.
19:20
Cut the wires. Exactly.
19:22
But then you're more getting back to the chaos monkey
19:25
approach and you, you first of all, before before you do
19:30
anything chaos engineering to start with.
19:33
The core issue is not like turning off something or like
19:37
the challenge not issue. The challenge is not to turn off
19:39
something. The challenge is define your
19:42
expectations. Start from your point of view.
19:44
What is your expectation, what your system needs to do and then
19:49
under which conditions O For example, my expectation is that
19:53
my customer needs to purchase my product in my ecommerce online
19:58
store even if my cloud provider is losing. 60% of all zones.
20:05
OK, how to make, how to make that happen, how to survive?
20:08
That's something you need to build in your system and then
20:11
you need to test if your expectation is fulfilled.
20:14
And that's something you can do, of course, without any chaos
20:17
engineering platform. It's getting more easier to do
20:19
it, yes, but that's like the normal starting point with your
20:24
expectation. So it feels like you've probably
20:27
seen a million different ways that systems fail.
20:31
Do you notice that customers maybe are not as creative in
20:35
these experimentations and you almost have to help give them
20:38
templates on? Well, what about if you have
20:40
this scenario or that scenario? Or is is there anything like
20:44
that where you can suggest different ways that they can
20:48
test it? The hardest answer ohh sorry no
20:54
see hardest question is inside of chaos engineering.
20:59
Where should I start and how should I start?
21:01
So where should I start in my system?
21:03
What is the good area under, in inside of my system where a lot
21:08
of stuff can go wrong? And the second one is OK, now I
21:13
found that spot. Hmm.
21:15
But how? What's next?
21:16
Is it something on the network layer?
21:18
Is it something on the cloud level?
21:20
It's Kubernetes, Kafka, Redis, whatever, all the database by
21:23
starting point. That is the biggest challenge.
21:26
So there is a way where you can use some tools that can
21:32
highlight some, let's say, risky areas in your system.
21:35
And they also, those tools should be also able to tell you
21:40
why is it risky. Maybe those tools are able to do
21:44
a pre-check of best practises. So there are open source tools
21:48
out there for Kubernetes and others other technologies where
21:53
you can run a check of your system.
21:55
So maybe let's talk about Kubernetes.
21:58
The CPU resource consumption is not configured or maybe you are
22:02
running not enough replicas, not enough instances of your of your
22:05
service. Maybe your instances are not
22:08
distributed across all the zones of your cloud environment that
22:11
are questions that are yet you need to take care that you need.
22:16
They need to be answered and hopefully with an easy approach.
22:21
Yeah, and I could imagine that machine learning engineers can
22:24
resonate with this. And then you just swap out CPUs
22:27
for GPUs and it all makes sense. Like ohh Gu access my data
22:33
scientist need more of that or whatever it may be.
22:35
On network On the network, maybe there are some rules on the
22:38
network about how the traffic is being routed, or maybe there are
22:43
some firewalls involved, or maybe some packets packages are
22:48
corrupted or lost ever so a lot of stuff can go wrong.
22:53
Well, and also I think there's the other hand in this which
22:59
could be data access and people in Germany can't get the data
23:06
that is in Canada because that data can't go outside of Canada
23:09
if there's special Pi or it has a certain use case.
23:13
And so trying to almost like do white hat hacking inside of the
23:18
system to see if there is possible for that data to be
23:21
leaked in some way or another could be quite useful.
23:25
If the data distributed across all your areas or around the
23:29
world fast enough? Or is there one area where,
23:34
let's say in Canada again, everything is working perfectly,
23:37
but the rest of the world is not getting actual data?
23:41
Or maybe between the synchronisation there is
23:45
something going wrong. Maybe that's funny funny attack
23:49
you can do. It's it's called time machines
23:52
where you can change the time of the system O.
23:55
This leads to very bad scenarios like all the all your
23:58
certificates are outdated so your system is no longer working
24:02
because there is no secure communication possible.
24:05
Or let's say the data in another region is now created on a on a
24:12
newer time as around the rest of the world, which is quite hard
24:17
about yeah, when the system has been distributed all the Aussie
24:21
data around the world. So it's you can do it on many
24:25
different levels. So it does feel like chaos
24:30
engineering goes hand in hand with observability.
24:34
And by observability, I think that we should probably like
24:38
define what that means to. And in my understanding, you can
24:44
tell me where I'm wrong. My understanding is that
24:47
observability is just there to clock and read what is happening
24:52
and maybe give you some alerts when things start to go wrong.
24:56
And this is much more proactive, but it still can be plugged into
25:01
an observability tool. Is that a fair assessment?
25:04
Yes, you need to have an observability tool in place and
25:09
a why for multiple reasons. First of all, to understand
25:14
what's going on and, uh, how is my system?
25:19
Yeah, reacting under acidic conditions which were injected.
25:23
And the important part is if you are doing case engineering,
25:27
let's imagine you are doing a CPU or memory attack where you
25:32
will use all the memory of one dedicated machine or one region
25:36
of the cloud environment. You need to be able to
25:39
understand which was injected by your chaos tool and which is
25:45
something like a cascading failure.
25:48
So something like a reaction of your system.
25:50
So why is it important? This is the job of your
25:53
observability tool. So that's why both tools needs
25:56
to work very closely. The chaos engineering tool needs
25:59
to tell the observability tool, OK, that was injected by my
26:03
chaos tool. This was the group of virtual
26:07
machines. I injected a lot of CPU load.
26:10
If something is going crazy outside of that group or we
26:14
found something that's a bad scenario.
26:16
Getting errors are in your system.
26:18
So it's really like you need to B direction communication.
26:21
You need to consume data in your chaos engineering tool to see is
26:27
everything safe or should I stop immediately?
26:30
And you need to send out data so that also all the people on call
26:34
are able to understand. OK, that's done by a chaos tool,
26:38
and that's something bad in my. System.
26:41
And it does feel like, as you mentioned before, one of the
26:44
hardest questions is where do I start?
26:46
Now? When you do this and you run an
26:49
experiment, you see what the chaos tool has done versus what
26:53
the cascading effects were. Do you also get recommendations
26:58
on how to fix that or is that just the engineers should know
27:02
or should spend some time doing that root cause analysis?
27:07
Let's see, hardest part sometimes, um, you can check for
27:13
something and you are able to, yeah, do a kind of advice.
27:19
But the chaos tool or the people that have created those chaos
27:25
tools are not the experts for your system.
27:28
You are the only expert for your system.
27:31
It's very hard to understand how you should fix something.
27:37
And that's like where you need to be able to have the
27:41
experience, the knowledge, how can I fix it?
27:44
There are some starting points here, but no tool on that market
27:48
or maybe in the future will be able to tell you exactly OK, in
27:52
a complex system. We found something and that's
27:55
how we can fix it. It's just an advice.
27:57
It's just something where you get a little bit of guidance,
28:00
but don't trust the tool which is telling you where you're just
28:04
pull request and you are safe to go.
28:06
Hmm, that's hard because the complexities is so big.
28:10
That's yeah, hot. Does that mean that you run
28:16
various experiments in the same area to see, OK, we think we
28:20
updated it, let's try again. Exactly experiments.
28:24
They need to be repeatable. You need to be able to just
28:27
press a button and tell the system, please do it again
28:30
because I just changed this little attribute.
28:35
Now we are talking again about a CPU limit, or let's say about
28:39
the distribution on the network layer.
28:41
Next one, are we now save enough?
28:43
No. Ah, we need to tweak it again.
28:44
And that's like this iteration you're in.
28:46
It's a process. Yeah, I was hoping that you
28:51
would say you have some magical AI solution that can give you
28:55
the root cause analysis today. But there is a way where you can
29:02
use AI for your doing those iterations in a more efficient
29:07
way as you are able to do it as as a as a person.
29:11
Yeah, there's. Yeah, that's incredible.
29:14
And I do see the value in this and you mentioned how you have
29:21
the systems teams and you, I imagine you're speaking a lot to
29:25
shares or what. So it's Ares that are trying to
29:29
make sure that their systems are the most reliable as possible.
29:33
Do you find that you ever talk to like data platform teams or
29:38
machine learning platform teams or is it mainly at that Sr
29:41
level? So right now it is at 3 levels
29:44
or tiers. So first of all of course is the
29:47
biggest group right now is the Sr group.
29:49
Why? That's where the pain is.
29:51
They are more hunted by their own systems.
29:53
They are always under, under pressure and if you are under
29:57
pressure, you are not able to fix something.
29:59
So that's the first group. The second group is coming from
30:02
QA performance teams. So they are they would like to
30:07
see if the system is still able to deliver as needed and under
30:13
with high quality even under those conditions.
30:15
So they are really like running their load performance tests in
30:18
combination of Chaos engineering.
30:20
And the third group is right now the platform engineering team.
30:26
So people that are building a a big platform that has been used
30:30
internally for other product engineering teams, developer
30:34
teams to build something. So maybe on the company level
30:39
there was a decision done. We need to run on a multi cloud
30:43
provider setup. And your engineers, they don't
30:47
care about the underlying Kubernetes cluster.
30:50
Is it managed by Azure or Google or AWS?
30:53
I don't care. So you need to have a platform,
30:55
a central platform O those teams.
30:57
They need to make sure that the new created platform is
31:00
bulletproof. That's why they are doing cars
31:02
engineering and they are also integrating CARES engineering
31:05
into their CICD system in the pipeline process.
31:09
In the past, some of the conversations we had were
31:14
getting started with people that are from the machine learning
31:19
area and it is always very closely connected to the data,
31:25
how the data is coming into those system and how the
31:28
interaction is now with the rest of the system.
31:30
That is like their biggest pain point.
31:32
It's not about running the system.
31:36
It's really like, is it even working if some streams are not
31:41
responding anymore, are not producing the data correct?
31:44
That's like the core issue, but it's not a big group right now.
31:48
Yeah, yeah. And I would imagine that people,
31:51
like I said earlier, people get that in their day-to-day and
31:55
they don't need any special tool to have these data streams be
31:58
shut off. So if this is only happening in
32:01
production, it's maybe not a good place.
32:03
If you are able to get this up and not running in a pre
32:07
production system and you can train your tool, that's a good
32:11
approach. I was going to ask a little bit
32:15
more to DoubleClick on this idea of incorporating the Chaos
32:20
engineering into your CICD pipeline.
32:22
What does that look like? There are some steps you have to
32:26
do first before you can go the CCD way.
32:30
First of all, you need to answer the question where should I
32:34
start and how you need to create experiments.
32:36
If you were successful in the execution of those experiments,
32:40
they are a very strong candidate to automate everything.
32:44
So you are really like creating a list of experiments that have
32:48
been executed. Why are your pipeline system
32:52
after the deployment in a preproduction system?
32:55
So, but our customers are doing is there's a set of experiments
32:59
and this experiment set needs to be executed after really every
33:04
deployment in let's say the QA stage, the performance or
33:09
preproduction system. Why?
33:11
Because inside of source experiments, there are alt
33:16
incidents from the past, They have turned that incident into
33:20
kind of regression experiment, regression test to really make
33:24
sure that they are not getting back in the same incident
33:28
situation again. Another type of experiments they
33:32
are automating is there are some, yeah, nonfunctional
33:37
requirements. That's something we need to see
33:39
in the system. Let's make safe that the system
33:42
is able to do that and able to yeah, work as needed and then
33:50
they are doing it in the CCD systems.
33:53
So I like that. It's like, first you're
33:56
experimenting. Once you know that you've gotten
33:58
it down pat, then you can add it into the CCD system just to make
34:02
it one more test so that in case you're changing something later
34:06
on down the line, you never forget the pain that you could
34:12
have caused. That's, that's funny, but also a
34:16
little bit offset. Um, so, so I'm, I'm a developer
34:19
by heart and, and, but we need to be honest, developers don't
34:24
care about security and they don't care about reliability.
34:27
Why? Because they are measured on
34:29
features and how fast they can get features into production.
34:32
So everything else like security or liability proof, I don't care
34:38
as long as no one is telling me to do it.
34:40
Yeah, but now it is still very important.
34:44
And if you are not taking care this will lead to very bad
34:47
scenarios and a lot of pressure on your shoulders again because
34:51
then someone is telling you this hmm very hard words hey you have
34:57
to fix it that's you messed U said the situation we are not
35:02
able to get in anymore. And is it a fair assessment to
35:07
say that this is only for the reliability aspect of what your
35:11
tools are capable of doing when they're put under pressure?
35:18
But I'm trying to find the words to figure out how to explain
35:21
this. It's it doesn't have anything to
35:24
do with security threats since you brought U this idea of how
35:27
developers don't really care about reliability or security
35:30
when it comes to chaos engineering.
35:32
It's not like you're doing any security vulnerability scanning
35:36
or security threats. It's only on that reliability
35:39
side, like if things turn off or things don't act the way that
35:42
they should. Correct.
35:44
It's really a focus on reliability, availability of
35:47
your system and um, it's not about, yeah, finding or really
35:54
likes. The difference is if you're
35:58
starting from the security perspective, security is most of
36:01
the time starting from the outside of the system.
36:04
Someone would is trying to get in your system from the outside.
36:08
Reliability and chaos engineering is something where
36:11
you are taken care from the inside that your system can
36:14
handle a lot of load of outages in areas of your system.
36:20
And it's really like improving your system from the inside and
36:23
not make it make it more secure. So yeah.
36:27
I like that and that's a really good way of visualising it,
36:31
someone from the outside. I also think that I guess with
36:34
security you have a nefarious actor that is trying to cause
36:40
you harm. With Chaos engineering, it's
36:42
like you are trying to see if you are causing yourself harm.
36:47
Yes, exactly. Excellent.
36:49
So what are some ways that you've seen people do this
36:52
wrong? You can get started with chaos
36:54
engineering and you will find a very big list of tools to get
36:57
started. So let's, let's put all those
37:01
tools in one toolbox and inside of the toolbox there's a
37:06
screwdriver, there's a hammer, there's whatever else inside.
37:09
And now what people are doing. Ah, I will take the the biggest
37:14
hammer I can find in this toolbox.
37:15
OK now what's next? I need to use this hammer.
37:20
So they are very focused on the chaos engineering tool and not
37:23
on the value they are able to get if they have been using this
37:28
tool, correct? So the outage, the attack, it's
37:33
it's not like the value that is more like to to create value.
37:39
It's just a tiny tool, but people are focusing so damn
37:43
freaking such big amount of resources on I need to do a
37:49
regional outage on AWS. Oh, we are not able to handle
37:52
and regional outage here are a big surprise.
37:54
I could tell you up front if you're running in one region hat
38:00
hat. So that's the point is don't
38:02
focus too much on the tooling. It's, it's it, it, it.
38:07
You'll need to really, yeah, start with the with your
38:11
expectation. Again, what your system needs to
38:15
do for you is like your, that's where the value is.
38:19
And then you can use some tools to verify if the system is still
38:24
providing the value as needed. Well, it kind of goes back to
38:27
what you were saying earlier is how you decide on the
38:30
experiment, how you decide on where and what.
38:33
And so by choosing the right tool, you are also deciding
38:38
where you're going to be swinging this hammer and on
38:41
what. And if the value that you can
38:45
gain, basically the insights that you can extract from
38:50
swinging that hammer it, those need to be something that is not
38:57
blatantly obvious. Correct.
38:59
And that's again why it is so important that you have a strong
39:03
integration into existing observability tools that you can
39:07
get the data, and also that you are able to rerun the same
39:13
conditions or rerun the same experiment under the same
39:16
conditions every time. So if you're running an
39:19
experiment under changing conditions, that's something not
39:22
repeatable, That's something which is not helping you to
39:25
improve. And it's again, you can imagine
39:28
like like in software testing, like in unit integration
39:31
testing, you are starting a new test every time under the same
39:35
conditions to really make sure that your test case is the same.
39:40
But if the starting conditions are every time different, that's
39:44
you're getting crazy. That's again is a big yeah, need
39:49
for a specific tool that can help you to understand, OK, this
39:54
experiment, are we safe to execute this experiment because
39:57
the same conditions are in the system, yes or no?
40:00
If no, OK, there's a new finding.
40:03
That's something where you have to take care.
40:06
I do like this viewpoint of saying our end goal or our
40:11
maturity level #2 is being able to incorporate this into the
40:16
CICD pipeline and running it as one more test.
40:21
Yes, correct. Yeah, that feels like a very
40:25
lofty goal, but it is quite useful.
40:28
But it's doable. And make sure it's before you go
40:31
into production. Yeah, that's always useful
40:36
information. Yeah, not a lot of us follow.
40:40
Yeah. Yeah.
40:42
Then one thing that I did want to call out that I think people
40:46
are understanding the beauty and the value of this is that I was
40:50
at cube con whatever a month ago and you all were there.
40:54
I saw the Steady Bit booth and I was on one side of the venue and
41:01
there's a million different booths at Cube Con.
41:03
For anybody that has never been, it is an absolute zoo.
41:06
It is. I think they said there were
41:08
like 13,000 people there and from the other side of the venue
41:15
I had two people walking in front of me and they said they
41:18
saw your booth and it had a big thing that said Chaos
41:22
Engineering on it. And these two developers were
41:25
like, ohh, Chaos engineering, let's go there, let's go check
41:28
that out. So I do think that this idea is
41:32
something that people really understand and respect and want
41:36
to incorporate more into their products.
41:40
And that's also something we can see in in our daily doing.
41:46
The need is growing. People are more aware of the
41:49
topic. It is an old topic.
41:50
It's not something really new. But like every time it there's
41:54
like this hype cycle, a lot of people are getting up or a new
41:57
topic. Chaos engineering cells.
41:59
Awesome. And then again, it's getting
42:02
down on on on a specific level. And now we can really see where
42:06
it's been mentioned by many different companies around the
42:09
world. It's been done by a lot of
42:12
industries. So it's and our customer base.
42:15
It's not, it's not just one type of industry.
42:18
It's a a wide mix about retail, ecommerce, insurance companies,
42:23
financial companies, even a heating manufacturer is is is on
42:27
it. So nice.
42:28
It's a wide mix about the industries.
42:32
Why there's one in common? A lot of pain because something
42:37
is not working as expected and that's something they would like
42:40
to get rid of it again. Why they would like to implement
42:43
new features. They would like to make their
42:45
customers happy and. Once they are.
42:48
Maybe is that something that they need to make sure?
42:54
So the last question that I got for you, it may be the the
42:59
kicker on this. What do you say to the person
43:01
who says, you know, what is our downtime is a few hours every
43:12
year or every quarter, and I have to look at the ROI of
43:20
implementing a tool like Chaos Engineering or just going
43:25
offline for a few hours. I think I'm just gonna stick to
43:30
the what I don't know is not going to kill me.
43:34
And I would rather let it go offline for a few hours, let the
43:38
people get pinged at 3:00 AM and not implement that tool.
43:43
So the first author, before I will ask some questions is.
43:47
The first answer is hope is not a strategy.
43:51
That's not something you should do.
43:53
It's really like, maybe it will go wrong, maybe not.
43:56
I don't care. But no, you should care.
43:59
Why? It's your company you're in.
44:01
It's your you're there for, for your customers.
44:06
And the first question I would like to ask those people is do
44:11
you know how much revenue you are making in just one hour?
44:15
Are you connected enough that you are able to understand what
44:19
the impact is if there is a 10 minute outage time or one hour?
44:24
So it's really you can do it on your own.
44:30
Just take a look at the numbers if you are able to get them, do
44:33
the math and you can see how big the impact could be.
44:36
And the next one is there is something very important.
44:40
It is very closely with your customers and it's the
44:43
connection between your customer and your company and it's about
44:46
the trust. So if a system fails one type 2
44:50
* 3 times, you are losing trust. You are losing trust, trust of
44:53
the most important group for your customers.
44:55
And that's something you are not getting back overnight.
44:57
That's something you you're working on heart.
45:01
And yeah, please make yourself aware of of, Yeah.
45:05
So scenarios and conditions. Yeah, you do have to have a lot
45:09
of brand loyalty for someone to suffer through a few outages.
45:13
Yeah. And if you take a look at your
45:15
at your smartphone, if there's one app not working, go to the
45:19
App Store, go to to the next one.
45:22
There's a lot of yeah, pressure on today's markets and we are
45:28
not the only one with the solution.
45:30
Well, Benjamin, this has been great.
45:32
I really appreciate you schooling me on everything chaos
45:36
engineering, especially when it comes to how it is not only a
45:39
technological solution but it is very much this idea of people
45:45
and organisational mentality that you need to bring into the
45:51
reliability sector of your company.
45:54
I also like how you speak about this and you've been dealing a
46:00
lot with scores and platform teams, but it has so many
46:04
repercussions in the machine learning field, especially for
46:07
those that are having to deal with machine learning models
46:09
that need to be reliable. And you, like you said, do you
46:14
know how much money you lose if a machine learning model does
46:19
what we've heard various times on this show and it just
46:23
recommends the same item over and over and over again to every
46:27
single person that comes to your ecommerce shop, Do you know how
46:30
much that loses for the company? Well, it is probably very useful
46:35
to figure that out and then become proactive on how you can
46:39
make your systems more reliable and.
46:41
Robust, well said and nothing more to add to.