Latest / Agentic Conversations (formally mlops.community) / Open Source and Fast Decision Making // Rob Hirschfeld // #164

Transcript

Open Source and Fast Decision Making // Rob Hirschfeld // #164 · Agentic Conversations (formally mlops.community)

0:00
I am Rob Hirschfeld, CEO, cofounder of Racken, and I like
0:06
my coffee black and wheat, much to my wife's chagrin.
0:10
She thinks I am weak nature because of it.
0:13
Like, I like weak black coffee. Hello, Hello everyone.
0:18
Welcome back to the Mill OPS Community Podcast.
0:21
Today I am flying solo talking to my man Rob Hirschfeld.
0:27
He is the CEO and cofounder of Rackin.
0:30
And boy oh boy was this conversation.
0:34
Awesome. Why is that, you say?
0:37
Well, Rob has the DevOps mentality baked into his blood.
0:42
He's been doing this for so many years.
0:45
I don't even think he realises how much of A fundamental piece
0:50
he is and has been for the DevOps movement.
0:54
I mean, the guy invented the cloud.
0:59
Can we just get that out of the way real fast?
1:02
Do you hear him talk about it? I asked him all about that rainy
1:06
day when he created. The cloud and what he thinks of
1:09
it now and how he feels like. It's been done justice.
1:14
By Amazon, I mean, he freely admitted that he did not know
1:19
how to monetize it. He did not realise what he had
1:22
on his hands when it was created.
1:25
And Amazon stepped in and they did know what they had and they
1:28
were able to make a pretty penny off of it.
1:30
But. That is not to say that Rob did
1:34
not and still has not been. Killing it when it comes to
1:39
products after inventing the cloud.
1:42
Right now he is the CEO of Rackin as I mentioned, and they
1:46
are making DevOps so much easier.
1:50
The user experience for the DevOps engineer is incredible
1:55
and he breaks that down in this conversation.
1:59
He also said something I think for me as far as my key takeaway
2:04
if I leave you with anything here, he harped.
2:08
On the idea of exercising your code.
2:12
And that stuck with me. So I'm not going to spoil
2:17
anything or explain what it means because I think he does a
2:20
much better job and I probably won't be able to do it any
2:23
justice. But listen out for when he
2:26
mentions exercise your code and without further ado, let's just
2:31
jump into this conversation with Rob.
2:33
All right, hope you all enjoy, and if you like the ML OPS
2:38
Community podcast, why not subscribe or leave us a review?
2:43
Give us some feedback. But you know what can be the
2:46
highest leveraged action that you can take?
2:49
That would be if you send this to one friend that you think
2:52
would also enjoy it. So bring in the DevOps
2:56
fundamentals into the ML OPS mentality.
3:10
I want to talk first about this stat that I saw, which is you
3:16
have copywritten the term the cloud or you came up with that
3:21
term. What is this?
3:22
About. Dave McCrory and I are patent
3:25
patented the cloud. That is nuts.
3:27
So you were the first one to call it the cloud.
3:31
We were, yeah. Tell me that story.
3:33
What and how? Like when was that and how did
3:38
it happen? Ohh boy.
3:40
That is a great story in in 1999, Dave, Dave McCrory and I.
3:45
And if you don't need Dave Mccrary, Dave McCreary is famous
3:48
for work he did actually when he and I were reunited at Dell
3:52
around Data Gravity. So Data he's, he's the data
3:55
Gravity guru. Wow.
3:57
And so he and I have been inventing together since 1999.
4:01
We formed a company. To do application service
4:06
providers or ASPs. It's early days of the web.
4:10
I had been doing consulting where I was writing applications
4:13
and delivering them into people's offices and things like
4:17
that. It was hugely painful and so I
4:20
was asking how do I make it so that I can Web API an
4:23
application and make it more available.
4:26
Something that people still were doing, hardly doing at that
4:29
point, and Dave had come in doing Citrix work, which was
4:34
remote desktop. The application sharing and
4:37
things like that. And we got together and started
4:40
and built this startup where we would help companies take
4:44
software that they had running on desktops and turning it into
4:48
a cloud offering. Although it wasn't called cloud
4:50
at the time, it was just an Internet offering, right.
4:54
And what happened is so we got funding on that model 1999 was
4:58
crazy up until there's a crash. And so we we had, we had an
5:03
interesting ride through this story through this.
5:06
But in order to do the work. We had to create a complete
5:12
application stack for each server for each customer.
5:16
So we would have a customer come in, they didn't know if they
5:18
could sell anything or not, but we would put 5 servers together
5:22
so that they could do their dev process to see if they could
5:26
actually build a Citrix enabled application.
5:28
We needed a SQL Server, we needed all this stuff and so we
5:33
were. This was the pizza box service
5:34
that was one U servers had just come out and we were getting
5:38
stacks of those servers and we were.
5:40
Buying like $30,000 worth of gear for each customer.
5:43
It was insane. Wow.
5:44
And so we could not grow the business because we had.
5:49
This need for this need for so much hardware, Yeah, Dave took
5:54
VMware the desktop version. And ran a prototype where he
6:00
built that whole 5 server stack on running on a desktop.
6:04
And so this is really cool, we're going to do that and we've
6:06
tried to install on servers that didn't work.
6:09
We call it VMware, and VMware said yeah, we have this beta of
6:14
this thing called ESX. Why don't you try and get it
6:17
working? We were the first company to get
6:20
SX working anywhere outside of VM's, VMware Labs, anywhere.
6:25
And so we we, we virtualized all of those functions onto a single
6:29
server. We were super excited and things
6:31
were things were pretty good, right.
6:32
That was that was the beginning of like OK, this is saving us
6:35
costs. But we had enough servers that
6:38
we were like managing ESX on five or six servers, right?
6:43
There was no shared storage. There was no Sans were
6:46
ridiculously expensive. There wasn't even this was, this
6:49
was the very early days of Net app.
6:51
There wasn't even good shared storage, shared block storage.
6:56
And just to set the scene because it's important, right,
6:58
when you 1999 inventing the cloud, yeah, we went to, yeah,
7:03
we went to VM Ware's headquarters and we said this is
7:05
really cool, We love this software, but it's sort of hard
7:08
for us to manage. Is it OK for us to write some
7:11
management software? And we talked to literally to
7:15
Diane Greene, who was CEO for a long time.
7:17
But she said, yeah, we're just worried about making this server
7:21
better. We we're not so worried about
7:23
managing it yet. And we said no, no problem.
7:26
We, we sense a business opportunity and we came back
7:29
from that meeting and started writing patents about how to
7:34
manage and provide an API which is really what that pattern is
7:37
about for infrastructure managed by another company.
7:44
So it was all about multi machine virtual management in
7:48
that system. And then we wrote I think 7 or 8
7:50
patents down around those concepts, around managing VMS,
7:54
around globally distributing them, all sorts of cool.
7:57
Things that one got, that one got granted.
7:58
It's almost been 20. It's the parents about to
8:02
expire. Dave and I are gonna have a
8:03
party when that patent expires. It's owned by Dell.
8:07
Tried to get them to donate it to the Open Stack Foundation
8:09
back in the day. Nobody's anybody enforces it.
8:12
Right? The patent, the way these
8:14
patents work. A tonne of people have referred
8:15
to it. But yeah, my wife likes to tease
8:18
me. She's like, yeah, he invented
8:19
the cloud. And I don't know if it's a story
8:21
of valour or shame because at the same time Amazon showed up.
8:26
Yeah, 7-8 years later with the right business model for it.
8:30
We turned it into a dev test system and never never never
8:34
figured out how to monetize it, or figured out too late how to
8:37
monetize it. So one thing that is fascinating
8:40
to me is that you've seen this 25 years of cloud.
8:45
What has changed and what's improved?
8:48
What is still around for better, for worse?
8:54
Less has changed the people than people think.
8:58
Want to believe? They they feel like a lots
9:00
changed but they're they're the there's things that are very
9:04
persistent in this one is the confusion of the business model
9:08
and the technology. Right.
9:12
So like the thing that Dave and I were doing at the time was
9:15
pretty novel, which was you would rent infrastructure, rent
9:19
infrastructure as a service that has gotten so commingled with
9:23
cloud, which I usually think of as API driven infrastructure or
9:27
more elastic dynamic infrastructure.
9:29
And I think one of the things that has was less confusing at
9:34
the beginning is, is actually even more confusing now as
9:37
commingling this idea of the business model and the
9:39
technology. And so that that definitely has
9:42
been there the behind the scenes stuff actually hasn't improved
9:46
that much. I I we actually like to talk
9:48
about about how little we've really improved the operations
9:52
and the work of operations in this.
9:54
It's still like we were joking in 99 IBM was doing all this
9:58
work on data centre autonomics right self healing all that and
10:02
it's we haven't really made a lot of progress with that that
10:04
stuff it's keep adding new services and micro services and
10:09
all of those little services are just.
10:11
The complexity of that environment has really exploded
10:15
in ways that people. Complain about that, they don't
10:19
know how to, how to chase. And the same is true.
10:22
We've made these very, very narrow businesses back in the
10:25
day you you really started sort of a bigger footprint of it.
10:30
You assumed you were going to have a bigger footprint of a
10:32
business to make this stuff work.
10:33
And now with some of the SAS offerings, you can start in a
10:36
very narrow targeted way for a business, which is amazing.
10:40
But we end up with some these, these very, very small use cases
10:44
that become standalone businesses that actually is not
10:48
that that has part of this cloud generation that's very
10:51
different. And what we saw.
10:54
Is that? Causing more headaches because
10:57
you mentioned the microservices adding more complexity like what
11:01
are the? Can you give us some examples of
11:03
that and how you feel that potentially is has its pros and
11:07
cons I imagine and what those pros and cons are.
11:11
Yeah, it's it's sad to me cause Rakan is a software provider.
11:14
We we sell software to people. They run that software.
11:17
The level of control in that and the disciplines necessary to
11:21
distribute updates and things like that are essential to our
11:25
business, which is about automation.
11:27
But if you look at the services that you're consuming as a
11:31
business, right, it feels like every week I add a new service
11:36
into the mix and that service is generally not integrated to
11:41
anything else. And so.
11:44
I I now have my day spread through using a tonne of apps.
11:50
I have sensitive information spread across a tonne of apps,
11:53
right. I've been playing with GPT
11:55
ChatGPT a lot. That's like I'm cutting pasting
11:59
that everywhere, which is OK, I don't want it scanning my
12:02
systems, but right, the next generation of this, they're
12:05
going to be clawing through my slack.
12:07
They're going to be running through my git repos, right?
12:09
The number of times you turn around and say, oh, I want a
12:12
security scanner. For Go, I'm going to go to the
12:17
company that writes security scanners for Go and then give
12:19
them access without restrictions to all of my git repos and maybe
12:24
my my my G drive for right? And so now I have a company that
12:29
has access. For a very narrow reason into
12:33
all of my information and that trend line should be terrifying
12:38
people. Hmm.
12:39
And we're very used to it. Frog boiling and water used to
12:43
it, where we're potentially vulnerable to some startup
12:47
smallest security violation or going out a bit.
12:50
But it's really frustrating. Like if I wanted to pull slack
12:54
back in and run it myself, I'd have to switch to matter most,
12:57
but even matter most to really now focused on hosting we've
13:00
just given up on. People running their own stuff.
13:05
Mm-hmm. And.
13:08
Man, that is fascinating. I 100% agree with you on that,
13:13
that a lot of these times you get access to way more than.
13:18
It's like you bite off more than you can chew in a way, because
13:21
you give access for one thing. Like you were talking about this
13:25
Go example, but you give access to the whole You let someone in
13:30
the mansion when really they only needed to be in the
13:33
courtyard. And so there is a I feel like
13:37
you probably have a very strong opinion about.
13:41
The better way of doing this and what what we should be doing
13:46
instead and how to provision and how to make sure that things are
13:49
secure and how to make sure that the data and the information is
13:55
only given when it needs to be given and to it who it needs to
14:00
be given to. Ohh boy I I wish I actually had
14:03
answers to that problem. It's ohh boy we we keep.
14:11
We keep sort of bypassing the question of security and access
14:15
and controls because it's so hard, The UX is so hard for
14:20
people to get right. And it's definitely something
14:23
that the industry hasn't moved towards standards on.
14:28
We we're starting to see a little bit like in the
14:30
Kubernetes space around like some ingress controllers and
14:34
some some spaces like that, but even that's really not helping
14:38
us create standards around this. Even like single sign on is not
14:44
particularly robustly supported by all apps or if it does it's
14:48
using single sign on providers that are mining my information
14:53
for habits and behaviours. It's it's a little bit of a
14:57
dystopia compared to. To what we were hoping.
15:01
Like when we did the startup, it was called Pro Tear.
15:03
I never, I don't think I ever named it back in the day.
15:06
Part of the idea was that you could put a couple of
15:08
applications together but still have shared storage and things
15:11
like that and we've. Every, every.
15:15
App. And this is actually a big
15:17
problem in operations tooling. We keep designing stuff that's
15:21
designed to have its own data store, its own source of truth,
15:24
and you're not designing things that are good at sharing
15:27
information. And that to me, even more than
15:30
security piece, which I think is important, is really, really the
15:35
starting point of the problem is you have to have things that
15:39
expect not to be the sole source of truth in an environment.
15:44
But we keep, we keep doing that to ourselves.
15:46
Isn't it kind of like the we could draw a parallel here with
15:50
the ads and what you see on Facebook or any social media
15:55
platform. And it's because they've had the
15:57
cookies that have been following you around the Internet.
15:59
And so you are now they have this picture of who you are and
16:05
they have all of this data about you and so then they can better
16:08
serve you ads. I look at it like, ohh, well, if
16:11
we want the apps that we're using to better perform in our
16:16
ecosystem. On all of our data, then they
16:18
have to have that access to all of the data.
16:22
A lot of times the people who are benefiting from that data
16:25
are that's their monetization strategy, is that data?
16:29
And we have to we've we've gotten used to not doing this.
16:32
So there's a barrier here. But you know the idea that
16:35
you're going to pay for the service that you're using or pay
16:40
to run the software to increase your control and privacy on
16:43
that. The inconvenience of that is
16:46
high and people, the consequence of not doing it is pretty low.
16:52
Right, so you were making a comment about my my camera
16:55
stream from my background, right?
16:57
That I had to hunt around for a device that had the option to
17:01
locally let me retrieve the stream inside of my own land.
17:06
Most of the cameras that you get, and they're much cheaper,
17:10
which is stunning, actually will only send the video out to their
17:15
service and you can only retrieve the video from the
17:18
their their Internet site. So you can't connect to the
17:22
camera directly at all. That has become a premium
17:25
feature. And so we we have to figure out
17:30
as an industry. How to make it easier?
17:34
Because I don't think this is just the consumers.
17:35
I think this is where the integration comes in, How to
17:38
make it easier to keep our users in control or operators in
17:43
control, keep our data local. But that might mean slowing
17:47
down, it might mean agreeing to protocols, it might mean saying,
17:51
oh, I'm not the sort, I'm not the source of truth, but if you
17:53
were going to be like, ohh, that sounds great, Rob, I want to
17:56
integrate into a control protocol.
18:00
There's Which one of the six are you gonna pick?
18:02
Yeah, and I can. I can give you an example out of
18:04
the OPS space if it's if it's helpful.
18:06
Yeah, yeah. Yeah.
18:07
Please. Do so, Hugh, a couple of years
18:09
ago. HP, did you know the Yeomans
18:12
work? I have to give them some credit
18:13
for establishing this protocol called Redfish.
18:16
No, if you're familiar with it. If you're familiar with bare
18:19
metal, it is. It is a protocol that's used for
18:23
out of band management on servers.
18:26
Before that there was something called IPMI, which was very old,
18:31
incredibly fragmented. And then there were each vendor
18:34
had their own. And so HP said I'm going to
18:37
consolidate, we're going to, we're going to open source this
18:41
protocol, we're going to build a consortium of vendors and we're
18:43
going to standardise on redfish trumpets sounded angels, angels
18:47
came down, blessed the servers, but the reality is.
18:53
Now you have another standard. You have you.
18:56
Well, the problem is not only do you have one standard, every
18:59
vendor to make it work has their own variation on that standard.
19:04
And so you went from N standards to 2N standards.
19:09
And so it it helped a little. Because there's sort of some
19:14
there. There's definitely a
19:15
commonality, but it's not enough to make it where you can plug in
19:19
a Dell server instead of an HP server and have your management
19:22
tooling just work. And so that's a real deficit in
19:27
how all those pieces fit together.
19:29
And then the other thing that people forget, that's very real
19:32
in this is time is actually a problem because the protocols
19:37
improve over time. And what ends up happening is
19:41
you don't. You don't just get to say, am I
19:42
using redfish? This is true of every protocol,
19:45
but it's not just Am I using redfish.
19:47
You have to be like, oh, am I using redfish before they fix
19:50
the bug in how the security protocols interact with the
19:54
server, or after? And of my fleet of servers,
19:58
which ones are patched or not? So when you're dealing with
20:01
this, and This is why it's more work and we sort of just throw
20:05
up our hands and say, I'm going to let the SAS provider do that,
20:08
You have to deal with which protocol am I using, which
20:12
version of that protocol. You have to deal with systems
20:14
that are patched or not patched. There's a lot of work.
20:18
It's valuable work, but it's work.
20:21
Well, that is something that I wanted to get into you with get
20:24
into with you about the impact of using open source.
20:31
And having that capability to go and know everything about what
20:37
you're doing and roll your own in a way versus.
20:42
The using a SAS provider and being able to go fast as you
20:48
were mentioning, and being able to then not necessarily have to
20:54
think about all of these bells and whistles and knobs that you
20:58
have to turn or patches that you have to be updated on.
21:01
And so you inevitably have thought a lot about these
21:05
trade-offs over the years. How do you look at that these
21:09
days? There's there, vendored open.
21:16
Open Source which I actually differentiate and because open
21:19
source itself becomes a layered piece and then A and even as has
21:24
the open open backed and and and non open backed.
21:27
Although the end of the day I think if you're using a SAS it's
21:31
very you're you're you're not really using open source
21:33
technology that might that people might get upset when I
21:36
say that but it's it's the the person the operational expertise
21:41
to run software is its own distinct thing and no SAS.
21:46
Provider I've seen opens and I'm sure there's an example.
21:51
But by and large, they don't tell you what they're doing,
21:53
certainly not like the hyperscalers.
21:55
That is incredibly secret sauce on how that works.
21:59
So the what what I see in that is that.
22:06
The effort to maintain your own stuff.
22:10
Track the versions, track the dependencies, patch the
22:13
operating systems right. All of that that you have to do.
22:18
We have not done a good job as an industry helping people do
22:22
that in a repeatable way. So the the this is actually the
22:25
travesty that started Rack end. If you want to indulge you with
22:28
the story, yeah please do about this.
22:31
So we started before we started Rack and the team was at Dell in
22:35
the early open Stack and Hadoop days.
22:38
And we built the first open stack installer, something
22:41
called Crowbar. And we were it was beautiful.
22:45
You could you could basically plug in a server and then build
22:48
an entire cluster automatically. It was literally just did all
22:52
the work for you. And we would take that to a
22:56
customer site. And we would ship a rack of
22:59
servers, they would plug it in and they would build open stack
23:02
or Hadoop on that stack of servers.
23:05
Took about an hour and it was lovely.
23:09
And but what would happen is every time we went, we would
23:12
learn things from that experience.
23:14
We'd improved the software, we'd fix it.
23:16
We'd come back to the office, we'd fix things, we'd add
23:19
capabilities, the versions would change, we'd update all that
23:23
stuff, and we'd go to the next site, improve the experience.
23:26
And by the time we got through like ten of those, we were
23:29
really good and fast about building that whole experience
23:32
and making it go. I will tell you, never worked
23:34
out-of-the-box. Because each site.
23:38
Has is different. We show up in people like I I
23:40
thought I had this, but I really have that.
23:42
Can you reset? But that is normal.
23:45
Yeah. But that wasn't a travesty.
23:47
We got really good at showing up, getting it done.
23:50
And then we actually had a benchmark where our customers,
23:53
the benchmark of success was when you were flying home and
23:56
the customer had done the reset tune and reset themselves while
24:00
you're on the way home. Nice.
24:02
So that was it was, it was amazing.
24:04
But. When we got to that 11th.
24:08
Install, we said. Alright, you know what?
24:10
We really have learned a lot. Let's improve installs 1 to 10.
24:14
We had no way to fix anything that we've done before.
24:20
Every site, even though we'd use a completely automated process,
24:24
ended up as unique, which is not software.
24:28
So when we look at this problem, if you're a software provider,
24:32
you're writing a product. You want you must have all of
24:36
your customers able to migrate together so you can support
24:40
them. Otherwise you're just making a
24:42
faster consulting practise which is not solving this problem.
24:46
Right now. Each person is still an island.
24:49
What what we set out to fix in this was how do you make it so
24:54
automation can be reused? So that if I give you a new
24:58
version of my open stack install, you can go back to the
25:02
site you already installed and put that in and then rerun that
25:05
patch, right? So from an engineering
25:07
perspective, I can actually build something and have people
25:09
keep reusing it. I can have a community of people
25:12
actually collaborating and helping each other.
25:15
And this, to your question, is, what's missing, right?
25:18
If we go back to my camera, right?
25:20
It would be really cool if I could download, Hey, I want to
25:23
store my videos locally. Here's a standard way to do it
25:26
and run that software and make it work.
25:28
And be confident that I could buy this gear and buy this
25:31
camera and install this software and have it go, and then be able
25:35
to hit a button and upgrade and patch all that stuff.
25:38
There's not a lot of incentives in the market at the moment that
25:41
make it easy to do that. We're living standards that make
25:45
it easier right there. There's a lot of barriers to it.
25:49
The benefits of being able to do that yourself and have that type
25:51
of autonomy is really high, isn't it?
25:54
Then you're not paying somebody yet another subscription fee to
25:59
service whatever camera thing or not paying them a service fee
26:04
and assuming that they've gotten the rights to watch all your
26:07
videos. I neither.
26:10
Neither scenario is particularly attractive to me.
26:14
Well, you do talk. I mean in this story that you
26:18
lay out, you talk about the automation and one through 10
26:22
not being automatable, that word that I'm looking for.
26:27
It's not it's not automatable, it's it's the.
26:31
And ohh, there's there's a a thing I I'd like to talk about
26:34
about the way to combat complexity is to exercise the
26:38
systems that you have. And what what happens is that
26:41
the the challenge here is that you attend people, 11 people all
26:45
using Open Stack successfully and they're all pretty happy,
26:48
but none of them actually can participate as a community.
26:52
In improvements in the future and the the the complexity
26:56
piece, what I mean when I talk about the the antidote for
26:59
complexity is exercise. Is.
27:02
I've I've got these ten people. They're all slightly different,
27:04
but they're using 99% the same automation if I find a bug on
27:09
one site. The chances are they all have
27:12
the buck. So I have reduced the complexity
27:16
of the whole system by exercising, by having 10 people
27:19
exercising the code instead of 1 by 1.
27:22
But what I'm missing, and this is actually what we set out to
27:25
fix at rack end, is I want to be able to say well if if if one is
27:30
gets the benefit of exercising, then I want to be able to repeat
27:33
that across all of the other customers.
27:35
They need to be able to take that fix.
27:38
And then what happens is the community gets bigger and bigger
27:41
everybody's. Experience everybody.
27:43
Finding those edge cases and testing things creates a mutual
27:47
benefit. Yeah, yeah.
27:49
But that doesn't happen by accident.
27:51
It's actually something you have to invest in.
27:54
But why is that not happening? I didn't understand that.
27:57
It doesn't happen because we, especially with all these SAS
28:02
right right now, the cloud disincentives that from
28:04
happening. They they don't want you to do
28:06
it. They they want you to let them
28:09
take the expertise and do that work it it doesn't happen
28:13
because it requires more effort. For the individuals to stay on a
28:19
system or participate in a system that they can share.
28:24
Right, right Now when people build systems they have a
28:26
tendency to build the automation they need and the it's they're
28:30
like this happens all the time. So if you download chefs,
28:33
answerable script or something like that you're like, this is
28:35
really cool, but it has all this extra logic that I don't need
28:38
for my environment and you start ripping it out because it's it's
28:41
opaque complexity. And so you to make it easier for
28:46
you to maintain that playbook. You take out all of the stuff
28:52
that people have added for other environments that aren't yours
28:55
or defence mechanisms that you're not going to hit.
28:57
You don't keep it in because there's really no incentive for
29:00
you to be like, well, I'm going to take out, I'm only in Amazon
29:05
and I'm going to take out the Google and Azure stuff because I
29:08
don't, I can't support it and I'm not planning to take the
29:12
upgrades now. If you are planning to take the
29:14
upgrades, those other cloud profiles are actually valuable
29:18
because they mean that. You're now exercising that
29:21
playbook on on 3 clouds and not just one.
29:26
But we don't look at the systems as an operator.
29:30
We don't typically look at. Those added pieces as beneficial
29:36
to us if we're not actively using them?
29:40
Right. Does that, does that help them
29:42
when we build an infrastructure pipeline?
29:45
The pipeline does a whole bunch of work that most people don't
29:49
care about. Hey, this is Mike Delbalso,
29:53
Cofounder and CEO of Techkon. ML OPS Community is the best way
29:58
to stay in the loop on the latest ML OPS news and best
30:01
practises. It's also a great way to connect
30:03
with experts and get support from an amazingly helpful
30:06
community. Subscribe and They're trying to
30:16
solve their very narrow part of that pipeline.
30:19
They get the benefit of the of the whole pipeline working.
30:22
But most of the time this is the behaviour that we have to
30:26
overcome. People, if you hit code that
30:29
doesn't do what it is, if you're like, well, I don't understand
30:32
why that code is doing it. I'm just going to rip that out.
30:35
You used to have this fight internally in the team too, by
30:37
the way. I have a story about this, might
30:40
illustrate it. Boy, we had built these, these
30:43
beautiful pipelines for bare metal hardware.
30:45
We do a tonne of bare metal hardware.
30:47
And so my cofounder, our CTO had this pipeline that literally you
30:52
plug in servers and it discovers inventories, qualifies, updates,
30:56
patches, installs the OS, installs the application.
30:59
It just this beautiful, completely automated pipelines.
31:04
And I took it because I'm guilty of this too.
31:07
And I said, you know, that that's really cool, Greg.
31:09
But I just need to do the cloud. And in the cloud systems I get
31:13
the server, all that stuff is already done.
31:15
I just want to skip to the end of the pipeline.
31:18
And so I took his pipeline, I copied it, cut out all the bits
31:22
that were for bare metal and was running it.
31:26
And I'm like, look, this is great.
31:27
Look how fast it is. And I mean he just, he
31:29
literally, I could hear his forehead hit the desk and he's
31:33
like, Rob, you missed, you missed it.
31:35
To leave all that stuff in, I'm like, yeah, but some of it
31:37
doesn't work. The bio stuff doesn't work when
31:39
you're on the cloud. And he's like, then fix that.
31:41
It doesn't work. The.
31:43
Because what what happens is, and this is the way it is today,
31:48
when you run that, it's exercise when you run the whole pipeline
31:52
in every environment. That means that every time I run
31:56
the cloud pieces, I'm actually exercising and able to move that
32:02
automation into any environment. So now if I run a whole bunch of
32:06
stuff in the cloud and somebody calls up and says, yeah, this is
32:08
really cool, can you also run it in bare metal?
32:11
I'm like, well, of course you can.
32:12
That pipeline is exercised. In the cloud, in virtual
32:16
machines and on bare metal. And so now I've gotten the
32:19
benefit of the operational Right practise and the exercise across
32:25
the whole spectrum because I fixed the place where it was
32:29
failing to apply BIOS to a VM. Silly, but instead of removing
32:33
it, I fixed that behaviour so I could keep the exercise in the
32:38
system. As opposed to looking at it.
32:42
So instead of looking at it as operational bloat.
32:47
And thinking this is something that we're not going to need
32:50
ever. So I'm going to get rid of it
32:52
because it's not. It doesn't have anything to do
32:55
with me. It's not my monkeys, not my
32:57
circus. I'm gonna just get that out of
33:00
my life so I don't have to deal with the headache of it not
33:03
working and getting errors every time I want to do something.
33:07
You're saying change the frame of mind because.
33:12
Who knows what the future holds and you want to be able to
33:16
change on the drop of a dime? Correct when as you exercise
33:22
that code, as you participate in the community.
33:25
Even though you're not actively thinking you're participating in
33:27
the community, the fact that you're using something that is
33:31
versioned and updatable, right, those things all support this
33:36
reuse and accelerative effect that is much more powerful.
33:41
And this is the thing I think a lot of us, I've watched this
33:43
because some people think the antidote to complexity is, is
33:46
simplicity, is removing items, which is what we see all the
33:51
time. And I hear this, it's like,
33:53
well, we standardise our data centres on Dell because.
33:56
We're just going to be a single vendor and it's simpler for us
33:58
to build our tooling and I'm like that works until Dell has a
34:02
new version that breaks the old stuff because that happens or
34:05
you acquire somebody and they switch or pandemic hits and I
34:09
can't acquire servers anymore from that vendor and I have to
34:11
switch, right. It's just and it's just bad
34:14
business or you're for no technical reason except that
34:17
you're procurement people say I've got a good deal, right?
34:20
Or your your CEO played golf with the Cisco vendor and so all
34:24
of those things are real. What we've we've gotten used to
34:28
is just tearing out stuff that we we don't really we don't
34:31
think is there and it's it's just a it's a it's across the
34:34
industry we have the same problem right There's a joke
34:36
about legacy code and people like what's another name for
34:41
legacy code. It's like proven, tested,
34:44
resilient. It's it's.
34:47
A lot of stuff in in code is there because somebody learned a
34:51
lesson and had to put in protections.
34:55
And we just need to reframe it from that perspective.
34:57
Doesn't mean I'm not in favour of refactoring or streamlining
35:00
when you can. But there's a lot of cases where
35:03
we. Don't look at the system effect
35:06
when we do that. Well, there is something that I
35:09
think you mentioned beforehand about automating infrastructure
35:13
inefficiently and it feels like this what we're talking about
35:16
kind of plays into that and I also would love to look at it
35:19
through the lens of. The machine learning engineer
35:24
and who's building with machine learning use cases and
35:27
potentially building on top of the data foundations that the
35:31
company has and how you see that when it comes to these
35:35
automations and especially like automating code, automating
35:39
infrastructure and doing it inefficiently as opposed to
35:43
things that you've seen work. Ohh my goodness.
35:47
Especially for for ML workloads. I I think that we because
35:54
because they tend to be big, they tend to be ones that have a
35:57
fair bit of complexity in the infrastructure and there's a
36:00
high desire to have very rigorous repeatability across
36:05
that fleet. But what that does is it
36:09
actually undermines some of the counterintuitive pieces and I'll
36:14
explain that for this. So one of the big barriers I hit
36:18
when we started talking about immutability.
36:21
And one of our, our, my chief solutions architect and I used
36:24
to have this argument all the time.
36:25
He's like, I want to reimage a whole server, I want to burn a
36:29
server down and I want to build it back up from immutable
36:32
artefacts. And I'm like, there's no way
36:34
that's slow. It's hard.
36:36
I don't want to risk breaking the whole server because we're
36:39
going to reimage it. And I'm like, well, why don't we
36:42
just take the server and patch it, just just apply the little
36:45
patches. It's so fast, we can just
36:47
surgically tweak it. And he was entirely right.
36:52
What what we've seen as we implemented those practises is
36:59
that when you take this sort of an artefact approach, this
37:03
immutability of saying this is what I want my server to look
37:07
like, and then you reset it from zero back into that pristine
37:11
state, that immutable state. What what it does is it gives
37:15
you a huge amount of certainty in what you have deployed and
37:18
how. And the the thing I was reacting
37:22
to and that this I think is one of the bad habits that that we
37:24
get into is you, you have battle scars that say, yeah, that that
37:29
that system didn't reset correctly or I have a high
37:32
failure rate on the resets. And I see this a lot.
37:36
A lack of reliability is the crisis in automation that people
37:41
have to think about that if you're nervous about running an
37:43
automation script or pushing a button to reset a server and
37:46
burn it down to zero and reset it, then address your
37:49
reliability problem. 1st. Right.
37:53
That process needs to be reliable and what we've seen is
37:56
1 the artefact. Patching is actually a full
37:59
system reset is actually a faster reset than patching in a
38:06
lot of cases because that surgical patch that you think is
38:09
very surgical ends up actually taking a lot of time and
38:12
bringing us machine in from an already patched everything's
38:16
right artefact with the applications installed.
38:18
If you get, if you start with that actually laying that image
38:21
in and booting it is very fast. If you go through like an app
38:25
get, update and app get this app get, and you're pulling things
38:29
in from all sorts of repos, that is the most fragile process in
38:33
the industry. Any time I do it like app get
38:36
update or Yum install, whatever, it goes to the Internet, it goes
38:40
to repos. It's going to tell you you're
38:42
out of date. It's going to pull in all sorts
38:43
of random stuff that you weren't thinking you had to do.
38:46
And those systems are incredible.
38:48
That process is incredibly fragile.
38:50
It's very slow in comparison and so if you.
38:54
Build that image You do all that work once and then clone it to
38:58
your entire fleet and then solve the reliability issue so that
39:01
you can confidently roll machines over like that.
39:04
The difference in performance of your system is remarkable and
39:08
then right we have customers. To rebuild an Esxi cluster, they
39:13
just push a button, they reset the whole cluster, BIOS,
39:15
everything, and they push a button.
39:18
They go get coffee, they come back and they resume their work.
39:21
Right, but it's mind changing because now they're like well.
39:25
It's out of date. Let me just reset and get I know
39:28
what the correct state looks like.
39:29
Let me go do that. They stopped wondering, like, is
39:32
my system all the way patched or partially patched?
39:35
Is it patching this break? Am I in?
39:37
There's all sorts of mess that people create with incomplete
39:43
updates or partial updates or not knowing the state of their
39:46
system. And then there's no standalone
39:49
system, so everything's interconnected to other things.
39:51
And if you're not sure the version it's at or the patch
39:54
date it's at, or if somebody hacked it.
39:56
Right. This is it's it's it's
39:58
beautiful, but it's a completely different way of thinking about
40:01
it. The other thing I would I would
40:06
say in this is. Turn rates are really valuable,
40:09
like keep talking about exercise, right?
40:12
A part of another thing of exercise is just being very
40:15
comfortable with turning things over and having the the days
40:19
when you would have a system and be like it's been up for a year.
40:23
You sell pop champagne corks and be excited nowadays if you have
40:27
systems that haven't been reset in 30 days.
40:30
Right. You should actually be like ohh
40:31
wait, I've all my servers are too old.
40:34
This is cloud or or or on premise.
40:36
I don't care where they are. If you're not able to roll all
40:39
of your infrastructure on a 30 day basis or faster, then you're
40:43
not. You're not keeping up with
40:45
patches. You're not refreshing, you don't
40:46
have the resilience in your operations to actually deal with
40:52
an emergency patch or change or something like that, right?
40:55
If you're every 30 days you're cycling through stuff and
40:58
somebody comes out with the next Heartbleed virus and you're
41:01
like. OK that's that's going to impact
41:03
all my systems And now you've lost like when Java we had the
41:07
Java things and people lost months of work finding,
41:10
discovering, fixing patching. You want to be able to say oh I
41:14
know how to fix this. I got the fix queued up.
41:16
Start a rollout process and you want to be able to say I'm just
41:20
instead of taking 30 days, I'm going to do my rollout over the
41:22
next week. If you don't have those
41:24
processes in place you're. Vulnerable and as you were
41:26
mentioning earlier about the legacy code and that being
41:29
resilient and tested and basically battle.
41:32
And how do you look at these two things and how do they fit
41:35
together? The you mean patching versus the
41:40
immutable. The the thinking about
41:41
immutability? Yeah, I'm I'm thinking more
41:45
along the lines of having this legacy code and then also being
41:49
agile enough to burn everything down every 30 days.
41:53
I I don't see them as as incompatible.
41:56
The the if you have legacy code, the thing that makes it scary to
42:01
people. Is they don't know how to
42:03
recreate the build, they don't know how to patch it, they don't
42:07
know how to upgrade it, they don't know what the environment
42:09
is. And so the the challenge with
42:12
the legacy piece is not the language that it was written in
42:17
or the age of the code. The thing that people gets
42:20
people in trouble when it gets to legacy is they don't have,
42:23
they don't know how to maintain it, they don't know how to patch
42:25
it, they don't know how to upgrade it.
42:27
They basically created a static artefact.
42:30
And so it is worth noting right when I talk about.
42:33
Having these immutable artefacts and version controlled
42:35
everything, it's only useful if you have an automated process to
42:39
build that artefact. Right.
42:42
People forget I Immutability doesn't just mean that I have a
42:45
whole bunch of stuff sitting in a locked vault.
42:48
What it actually means is that I have a way to reproduce those
42:51
artefacts in a predictable way and then apply those into my
42:55
environment in a predictable way.
42:57
The legacy thing that gets people nervous is when they have
43:00
a a Fortran programme that nobody knows how to build debug
43:04
tests, right, that that's installed on a mainframe that
43:07
that four generations ago the person who set that up and made
43:11
the decisions went away. That should make that should
43:14
make people nervous, but you can have modern programmes.
43:17
Right. Well, here's the magic Docker
43:20
container that you run in Kubernetes.
43:22
And how did I build that? I don't know how I built it.
43:24
That person's gone. Don't mess with the container,
43:27
just run it. That should send off the alarm
43:30
bells, just like that's what we're talking about here.
43:33
The legacy is. The lack of repeatability in the
43:37
system. Hmm.
43:39
And how do you look at that when it comes to throwing or
43:44
sprinkling on data? Because it feels like a lot of
43:47
the stuff that you're talking about is very much like.
43:51
In the software world and very DevOps focus.
43:54
But then when you start to incorporate data in there, and
43:58
data isn't as I would say, agile as you're making these things
44:06
out to be, how do you see the difference there?
44:09
Ohh my goodness it can't be as agile.
44:12
Without a doubt. I do think that the metadata
44:17
collection of metadata I think access to that data how it's
44:21
partitioned and shared. The way the way I see it is, you
44:25
should have automated, repeatable processes for doing
44:29
those things. So this, this actually goes back
44:32
to our ability, what we were talking about with improving
44:36
access to things. Some of it is the the tools
44:39
actually have pretty good access controls.
44:42
But we have a tendency as humans to interact right Turn the
44:47
knobs, set it up, and say we're done.
44:51
And spend less time actually figuring out how to automate
44:54
what those processes and controls are.
44:58
And so I would what I would suggest.
45:00
When I'm I, I realised my world is DevOps so I'm I'm putting a
45:03
very DevOps he spin on this. But you know, you should be able
45:07
to get hands off in providing the access to that data,
45:12
replicating the pieces of the data, right?
45:14
We're back to is there an immutable process or a version
45:19
to process? Is there a process that other
45:21
people can see and inspect? Understanding how that the the
45:26
parts of your data are replicated and put together.
45:29
We do need to understand how all that stuff fits together and
45:34
that stuff. And I'm glomming a whole bunch
45:35
of data under that stuff. But it's it you do.
45:40
You do have to have a way to manage all of those pieces,
45:44
protect all those pieces, standardise how that stuff
45:47
works. So Rob, I wanna talk about the.
45:53
Idea of Rackin and what you're doing over there and get into
45:57
what exactly it is and how it helps people and how potentially
46:02
it can help us make sure that we have these automated processes
46:08
and we're doing things effectively and not
46:12
ineffectively. And basically leveraging and
46:15
standing on the shoulders of giants who have been in the
46:18
cloud business for the last 25 years.
46:21
I am happy to try and try and shed some light on this.
46:26
What what we do with with rack and and a lot of it addresses
46:30
the problems we've been laying out right obviously that's
46:32
that's that's where we see things that are going on reckon
46:36
writes software product called digital rebar.
46:39
Digital rebar is it's a workflow engine for automation and
46:43
infrastructure, so it's not like there's some orchestrators out
46:46
there that people are familiar with their job schedulers and
46:48
the mill O space. Digital rebar is specifically in
46:52
infrastructure tools. So the the the semantics, the
46:55
nomenclature, the objects in that system are infrastructure
46:58
focus and what it does is it builds an infrastructure
47:02
pipeline. Which is very similar to like a
47:04
CICD pipeline or a data pipeline in that you can sequence a whole
47:09
bunch of tools and operations together in a repeatable way.
47:12
So you can start a process and then flow data.
47:17
In our case, data is state of the infrastructure through a
47:22
series of transforms. For us that means Terraform and
47:26
Ansible or shell scripts or talking to other APIs, but
47:30
you're talking to all of the components of the infrastructure
47:33
in. A version managed sequence to
47:37
get to a desired state. And then one of the things we
47:40
did that's really been fun is that we also have a way to do
47:44
jobs. I'll put this in ML terms, but
47:47
you know, we we have transforms, which is the infrastructure
47:51
pipeline going from one state to another or resetting a rack of
47:55
servers, things like that. But we also found that that
47:58
automation can be applied in work orders, little jobs.
48:02
And so people want to be able to say, you know what, I just need
48:04
you to run a security scan. I don't want you to change
48:06
anything. And so that type of of small
48:10
operation or work order is also part of the system.
48:14
So transforms and then you can schedule a whole bunch of work.
48:18
And then the other thing we've done about with this is we've
48:22
created a shared state system in that it's very easy to send
48:27
state from digital rebar to other systems because you have
48:31
to participate in the infrastructure or provide access
48:35
to from other systems into digital rebar.
48:37
So we find people wanting to do that like putting dev portal in
48:41
front of digital rebar and then let the infrastructure do its
48:44
thing, CI, CD systems, Terraform in front of it.
48:48
You know what, what we find from an OPS perspective is that
48:52
people want to consolidate and consistent operational
48:55
experience. They want to abstract out is it
48:58
cloud, is it VM, is it bare metal?
48:59
Have that same pipeline run everywhere like we were talking
49:02
about, but then not force people into having an operations
49:08
experience when they just want a server when they want to dev
49:13
experience. So for us a dev or I think a
49:16
data scientist, right would say. I just need.
49:20
You to build me a cluster that can do ML analysis.
49:23
I don't care about the process that you use.
49:25
I don't care about the operating system, I don't care about the
49:27
cloud or the infrastructure. Just just set everything up.
49:29
I just want to click a button and get a cluster back.
49:32
There's a tonne of operational work that has to happen to make
49:34
that go. That's what digital rebar does.
49:38
Simple API, whole bunch of operational work that's
49:40
consistent, portable and reusable.
49:43
But back into a very simple API for a consumer to use and then
49:47
we do a tonne of work as a company.
49:50
Because it's software. To allow that to be a platform
49:54
that somebody can use without giving up control of their
49:58
systems, especially if you're talking about data here, right?
50:02
None of our customers want us to tunnel in to their data centre
50:08
or their data set like this whole thing we're talking about.
50:11
That's no, we don't want it. I don't want access to anybody's
50:15
data. That's the last thing I want.
50:17
The other thing that we that nobody wants is for us to make
50:20
changes to their automation platforms that are in our
50:24
benefit because we want everybody on the same version or
50:28
upgraded or patched. So the fact that we're software
50:32
which is really important for automation especially we do
50:34
distributed multi site control planes with customers so they
50:38
have global global fleets of infrastructure.
50:41
But this works even for teams and multiple teams collaborating
50:45
around the automation. You don't want to have to say,
50:47
well, I patched my automation to include this new feature.
50:52
And force it on everybody else, huh.
50:55
So because of the way we do versioned automation, you can
50:59
actually control when people pick up that automation, they
51:02
can pick it up that's, but they don't have to pick it up.
51:06
That control is really important for people and all that just
51:10
comes out as immutable artefacts that you download into digital
51:12
rebar and and then make it run. But then each customer has their
51:16
own version and they maintain it themselves and we work really
51:19
hard to make that simple. And so is there the capability
51:23
then you mentioned if you want to use?
51:27
Or you can just ask for certain things and you don't care about
51:32
other things and you just say give me this what I want.
51:36
I don't care about the operating system, I don't care about the
51:38
cloud any of that. I imagine there is the
51:40
capability to say I do want it on this cloud, I do want it with
51:44
this operating system and you can get that granular.
51:47
Yes, yeah, but the but the system is uses declarative APIs.
51:52
So the goal is if you start a pipeline, the system can fill in
51:57
or. More more importantly, the
51:58
operators who set it up can actually provide ways to fill
52:02
all those details. So you can say I need to use
52:05
this cloud specifically or you can say I just, I'm just gonna
52:08
call, I need you to fulfil this request and not hear that not
52:12
care how that abstraction is really important, right?
52:15
Because what we've built over these pipelines that run
52:17
anywhere and so and we do this internally all the time, I will
52:21
test new builds and new development in cloud
52:25
infrastructure because it's dynamic and inexpensive.
52:28
Because of the way we build that pipeline, that pipeline then
52:31
will work against bare metal system because the abstractions
52:34
are fine for it. Because right here what we've
52:37
got is this long pipeline of work that is consistently
52:41
applied, and if I add something into the pipeline then I can
52:44
make it go. Now I might only build and this
52:47
is where the community effect gets really interesting.
52:49
I might only build for Ubuntu if I'm only testing for Ubuntu or
52:52
Amazon Linux or something like that.
52:54
But once I have that code. Then we can come back and say,
52:58
yeah, you know what, I'll also make it work for CentOS or
53:00
Windows or right, we we have ways to keep extending
53:04
individual tasks in the pipelines to broaden their scope
53:08
or add different APIs or detect where they are and make
53:10
adjustments to what's going on. That's a really powerful
53:15
feature. When people look at it the first
53:18
time, they're like, oh wait a second, they're doing a tonne of
53:20
stuff. And that can be a little
53:22
overwhelming, so important for people to be like, oh wait a
53:25
second, they've baked in. These protections and standards
53:29
and things like that. And even if I don't understand
53:31
why it's doing this task instead of turning it off, we have
53:35
people who do that and then they come back and they're like a
53:37
couple of months later, I turned into and we're like, all right,
53:41
did you turn off the? They're like I did.
53:42
I copied all that stuff and took out the things I didn't need.
53:45
And we're like, were they breaking?
53:47
Like we were talking about earlier.
53:48
Yeah, no. And then they pull it out.
53:50
Even if they weren't, they were working.
53:51
They just thought they were unnecessary complexity.
53:54
They just hadn't reached that point in their journey yet.
53:58
And so that's, that's part of how all these these things work.
54:01
Most operators now understand they're like, OK, wait a second.
54:05
If you can do all I maybe I only buy Lenovo Gear, but the fact
54:10
that you can do the other types of gear is good protection for
54:14
me. It's but it's a different
54:16
mentality in looking at how these systems could just like
54:19
switching to immutability. People like I don't want to do
54:22
that. I've got to build images.
54:24
Now that's painful and you're like, well, yes, but once you've
54:28
built that image and everything you do is going to be more
54:31
resilient and faster. And operators have to have the
54:34
time to do that. It feels like that is something
54:37
that you learn with experience after, as you mentioned, you
54:40
could rattle off a few different scenarios just off the top of
54:44
your head when shit hits the fan.
54:46
Yeah, and I imagine those first couple of years before you run
54:50
into the first whatever, maybe the pandemic hits and you can't
54:53
get servers anymore or whatever. The five other scenarios that
54:57
you were able to just talk about, obviously from battle
55:00
scars that you have gone through over the years.
55:04
That seems like it's it's very clear as to why you would want
55:08
to keep that as opposed to rip things out and say, well, we're
55:11
never going to need this. It's it.
55:14
This is, to me, the reframing of perceived complexity.
55:19
As right as as exercise or, you know, something useful.
55:25
Yeah. And I do a lot of thinking about
55:27
how because everybody runs around pulling on their hair
55:30
being like Ohh systems are so complex, I'm scared of
55:32
operations now. Which which I understand the
55:36
feeling and I I spent a couple of years ago, I was like
55:39
alright, how do we how do we deal with this?
55:41
And I went down a whole bunch of rabbit holes until I realised
55:44
that this is what it's actually well understood.
55:47
There's a whole bunch of academic research on complexity
55:49
and dealing with complex systems and what failures and complex
55:53
systems look like. And and I can't summarise all
55:56
that research that quickly but one of the things that made me
56:00
reevaluate was complex systems are much more.
56:05
Resilient to failure. Then simple systems.
56:11
And and so. When you look at a system and
56:15
see complexity that you're you're question shouldn't be is
56:19
why is that system so complex? I can't understand it.
56:21
You might not understand everything about it but you need
56:24
to be like how do I exercise this system as it operates, how
56:29
observable and transparent it is.
56:31
Those are things like for rack end we work really hard on that.
56:34
We have boy, I've done whole podcasts on nothing but this
56:37
fact that when things break instead of doing retries we stop
56:41
and tell people to fix the the root cause of the problem.
56:44
Rather than just banging through, because when you're
56:47
fighting complexity, you don't want unexpected behaviours,
56:51
right? So there's there's all sorts of
56:52
design elements that go into making these systems really
56:55
resilient. But we're very used to needing
56:58
comprehensible systems. Maybe, maybe ChatGPT is going to
57:02
teach us something indirectly that not understanding what's
57:05
going on behind the covers is not as bad as we thought it
57:09
would be as we thought it was. We've sort of gotten, we're sort
57:12
of very willing to accept that we don't know how these models
57:15
got trained. But that's part of these systems
57:18
is we can lean into making a complex system usable and secure
57:24
and reliable and resilient, but there's a little bit more
57:28
cognitive lift the first time you look at it.
57:31
Hmm, so you have to have had played around with some ChatGPT
57:35
and its abilities to use racking or potentially like lay with the
57:41
API or do these operational loads under the covers.
57:44
Have you discovered anything and is that has it been anything?
57:50
Good I'm I'm actually doing some talks about what I'm calling
57:53
generative DevOps that I would suggest people to look at where
57:57
we're we're way down the rabbit hole at this point.
58:00
But yeah if you search for generative.
58:01
DevOps. You'll see some of the early
58:04
thinking that I have with this, and the teaser on it is that.
58:10
It's making expertise much more accessible and so I'm using this
58:14
phrase, the 10X operator. Like we've been talking about
58:20
10X developers for a long time, but we might be entering an era
58:25
where there is the possibility of a 10X operator.
58:30
And that. Completely changes assumptions
58:34
on all sorts of things. How you run gear, can you run
58:37
gear? How many people you need, how
58:40
effective? Like like edge, like like the
58:42
10X operator concept with generative DevOps completely
58:47
could remake the landscape we had.
58:50
I love it. I'm going to Google that right
58:52
now. Well, Rob, thank you so much for
58:54
coming on here and talking to me about all this.
58:57
The past, present and future really.
58:59
And bringing this DevOps feel to it.
59:03
I. There's so much that the Milos
59:06
world borrows from DevOps and so it's great to hear your spin on
59:11
things and and also see what you're working on and what
59:14
you're dedicating your. Time and energy into these days
59:19
and it's fascinating stories you've got.
59:22
I really appreciate this and I think that's it, man.
59:25
We'll end it. Here, I appreciate it.
59:26
Thank you for the conversation. It has been a pleasure.
59:33
Hi, this is Michael Eric. I'm the founder of Pametan Data
59:36
Innovation and Confetti AI and this is a quick public service
59:40
announcement. If you're tired of having
59:42
nightmares about the machine learning models you have in
59:44
production, then I encourage you to subscribe to the ML OPS
59:47
Community Podcast. It's where all the cool ML OPS
59:50
kids hang out and learn how to tame their ML models.