Latest / Agentic Conversations (formally mlops.community) / Speed and Scale: How Today's AI Datacenters Are Operating Through Hypergrowth

Transcript

Speed and Scale: How Today's AI Datacenters Are Operating Through Hypergrowth · Agentic Conversations (formally mlops.community)

0:00
The couple are observability of the physical and the logical
0:02
with our context about how this stuff is supposed to be working,
0:07
marry all that stuff up. What you figure out is the delta
0:10
between our like our observations and what's really
0:13
happening and what our intent says or what the design says.
0:17
And then that delta is the problem you have to go fix
0:19
that's. Probably a really good place to
0:27
kick it off. Like, how is this new world
0:30
shaken down and what does it look like now?
0:35
Yeah, well, I mean, and 1st now that now that we're up and
0:40
running, you know, just to contextualize and then for the
0:43
audience a little bit, I'm Chris, I'm Co founder CEO, I'm
0:47
an engineer and I work on infrastructure.
0:50
That's me. So like you know, what we're
0:52
talking about today, I think mostly is all of this
0:56
infrastructure investment that is happening in support of AI
1:02
and machine learning and these as we as we have talked about
1:07
these things that are taking over the world right now.
1:10
And you know the, the infrastructure investment here,
1:13
I mean, it is, it is enormous, right?
1:16
It is some, some meaningful percentage of GDP growth in
1:21
America this year, right? Like is AI infrastructure.
1:24
And So what is going on there, I think is, you know, our big
1:29
theme today and then just to contextualize like, you know,
1:35
myself and Netbox and how we fit into this and what we see,
1:41
Netbox is the system record for infrastructure.
1:43
That's what it is, right? And so if you distill that way
1:46
down, at its core, Netbox is a model that the teams that are
1:51
building this kind of infrastructure are using to keep
1:54
track of everything in their infrastructure.
1:57
Everything from space and power and cooling.
2:00
These sort of like foundational fundamentals that we hear about
2:03
all the time all the way up to like, I mean, certainly racks
2:07
and GPU servers and switching and cabling and then into
2:12
logical stuff like IP addresses and switch configs and server
2:15
configs and all of the automation that happens around
2:19
this. So like really from from as
2:21
early as I've decided I want to design a new big AI data center
2:25
to this thing is running and scaling and serving, you know,
2:30
inference or training workloads or whatever it is at at mega
2:33
scale. We have this incredible lens
2:36
into to that journey. And then the other simple point
2:42
of context I'll share is I I don't think there's any AI data
2:44
center in the world that is built without netbox.
2:46
And so we get this incredible cross cutting view into all the
2:51
variations and all the things that people are trying and what
2:54
it what is happening in the like the word we were using a few
2:57
minutes ago is chaos, right? Like the chaos in this space
3:00
right now. And it is insanely chaotic just
3:04
because of the pace and the amount of demand that there is
3:08
in the level of investment that is happening and how many teams
3:13
and companies are being formed all at the same time to tackle
3:15
this in the geopolitics of it even like there's so many
3:19
factors, right? And so, yeah, that's a little
3:22
bit of the context I think of what we're going to talk about.
3:25
And the the headline here is like, everybody's investing here
3:29
right now. There is, there's no secret
3:33
answer to how to build AI infrastructure.
3:36
It's a lot of hard work and a lot of logistics and a lot of
3:40
like supply chain type problems. And where am I going to get the
3:43
power and, and how about to get to get the power in 45 days when
3:47
I've made a commitment to get, you know, 300 megawatts online?
3:50
And, and where am I going to get the servers and the GP us and
3:55
the cabling and the racks and the humans to do all this stuff
3:59
or the robots increasingly, right?
4:01
Like all of that is what is happening now.
4:04
So fascinating time in this space.
4:07
And so it feels like every single one of these steps is a
4:13
potential bottleneck. Are there some that you see
4:19
being more bottleneck than others?
4:21
I know that back in the day it was like GPU constrained.
4:25
Has that been alleviated and now it's more on the the actual
4:29
wiring or the power? Yeah, it's a it's such a great
4:34
question. And building these kinds of
4:37
infrastructures at the speed and scale that the world demands
4:42
right now is a constraint satisfaction problem.
4:45
In any given time, there is a constraint and it is the primary
4:49
bottleneck. And there have been several of
4:51
them over the last couple of years.
4:54
And there will continue to be several of these as long as
4:56
demand is what it is, which is, you know, almost like infinitely
5:00
more demand for AI infrastructure than there is
5:02
supply, right? And so that's, that's the root
5:05
cause I think of like, you know, the furiosity with which
5:11
investment is happening in the space and the pace at which, you
5:14
know, teams are trying to trying to build in this space.
5:17
So let's just break down some of the different kinds of
5:20
constraints that we will hear about, right?
5:24
We all hear about power as a constraint.
5:27
For an example, where am I going to get 3 gigawatts of power?
5:32
I, I did a call last, not last week, the week before last with
5:38
a public company that, you know, building AI data center
5:44
infrastructure that I hadn't really heard much about in the
5:49
past. They grew up in kind of Bitcoin
5:51
land. And I don't call it the CTO.
5:56
And he said, you know, we have 3 gigawatts of power capacity.
5:59
And my jaw dropped, right? Like, because that, that's
6:02
problem number one that you usually hear like where, where
6:05
are you going to get all of this?
6:07
And the, the, the kinds of anecdotes we're hearing just to
6:11
give everyone a sense of the, the chaos and the scrappiness
6:16
with which this power problem is being approached.
6:18
We have, we have a, a saying we use in our team Nutbox Labs for
6:25
how we need to operate to serve this kind of customer.
6:29
And that saying is turbines in the parking lot.
6:31
And that's a real anecdote. You know, a few months ago,
6:35
we're talking with one of these, you know, real hyperscale AI
6:39
data center builders who's got 10s of billions of dollars in
6:43
commitment from one of the research labs to bring
6:45
infrastructure online incredibly fast.
6:47
And what they told us was, hey, Chris, hey Netbox Labs team.
6:53
This infrastructure is coming online real fast.
6:55
We're buying turbines, we're putting them in the parking lot.
6:58
That's how we're getting our power fast enough because we
7:00
can't the, the grids can't operate this quickly, right?
7:03
And can't scale this quickly, or we can't build nuclear reactors
7:06
this quickly, right? And so it power is a big
7:09
constraint that everybody's investing in, obviously.
7:12
Wait, and so how did this other company have the three gigs of
7:17
spare capacity? Yeah, luck in history.
7:21
And you know, one of the one of the sort of interesting trends
7:24
that we've seen is companies that invested maybe 7-8, nine
7:30
years ago in tapping, you know, cheap power or building
7:37
infrastructure in environments suitable for, you know, large
7:42
scale data center infrastructure with good cooling
7:44
characteristics or access to hydro, both for power and for
7:48
cooling, like those kinds of things for Bitcoin.
7:52
These are companies that are really well positioned actually
7:55
in sort of this this AI infrastructure world.
7:59
And we've all seen that with core weave, who's sort of the
8:01
flagship, you know, builder of infrastructure in this space.
8:06
I think a lot of their historical contemporaries in the
8:09
Bitcoin mining space have recognized like, hey, it's way
8:13
more lucrative to shift over to this AI stuff and use the the
8:18
fact that they've already invested in solving these big
8:21
constraints, the power and the cooling and the space and even
8:24
just like the logistics. I think that's the next big
8:29
constraint that we see that maybe a lot of, a lot of the
8:34
time is not as talked about as it should be, which is think,
8:38
think about everything that has to line up to bring a big, I
8:42
don't know, 300 MW AI data center full of NVIDIA Superpods
8:48
online. There's a lot of moving parts
8:52
and they all have to converge and they start as really as like
8:57
I've purchased a parcel of land and I have some power
8:59
commitment. Great.
9:00
What am I going to do with that? I need to design a physical
9:03
space. I need to interlock that with my
9:07
ability to procure racks and GPU servers and switching and fiber
9:16
optic cabling and turbines and power infrastructure and liquid
9:22
cooling infrastructure and all of this stuff has to converge in
9:26
a design has to drive a purchasing workflow.
9:31
All of those components have to arrive before you can bring
9:35
anything online. And so like that, the problem of
9:38
even just what am I going to order to make sure that it's all
9:41
going to work together at massive scale, right?
9:44
And how am I going to make sure that it all arrives in a, in a
9:47
way that, you know, I can, I can interlock that with my design
9:53
and make sure I've got everything I need to bring this,
9:55
this AI infrastructure up and running.
9:58
And then how am I going to even deal with the fact that I've
10:01
got, you know, thousands of pallets of stuff arriving at a
10:04
loading dock and it needs to get rocked and stacked and plugged
10:08
in and cabled and burned in and tested and configured and
10:11
deployed and observed and, and then ultimately delivered, you
10:16
know, to an end customer for training or, you know, sliced
10:20
and diced cloud style for, for retail like inference type use
10:25
cases, incredible logistics problems.
10:28
And let's just make it a little worse.
10:31
Every one of those components is sold by a different vendor.
10:35
And you know, one of the really interesting challenges that we
10:39
see is that every one of these vendors operates differently.
10:41
They skew differently. They expose their catalogues
10:45
differently. They expose life cycle data
10:46
about their catalogues differently.
10:48
Like if I want to buy AGP server from Dell over here and I need
10:52
to get some fiber optic cabling from Corning over there, and I
10:55
need to switch from Arista over here.
10:57
I've got three different procurement problems to solve, 3
11:00
different kinds of data I need to figure out how to work with,
11:03
and then let's compound it one more time and then I'll stop.
11:08
The other big problem is that there's so much demand in the
11:11
space and the challenges are so hard that the componentry
11:15
vendors are iterating on their own offerings incredibly fast to
11:23
meet the needs of this space. And so the, the ground is
11:26
shifting under the feet of the folks who are seeking to build
11:29
these data centers, the, the, the kinds of switches they can
11:31
buy or the kinds of service. Every few months, NVIDIA has a
11:34
new architecture, right? And, and then that nullifies
11:39
like all the other stuff that attached to it.
11:41
And you have to rethink all of this.
11:42
So it's a really hard problem space at the moment.
11:46
And I imagine when when I want to learn something, I usually
11:51
will go to YouTube and search how to do X or I'll read some
11:56
articles and some blog posts about how to do that and kind of
12:00
immerse myself in it that way. I get the feeling you don't have
12:04
these engineers going around making YouTube videos on how to
12:11
put together the GPUs and the date data centers.
12:15
Yeah, this is not a space like, I don't know, call it, call it
12:20
software development, for example, where there's a vast
12:24
array of humans and some people are inclined to tinker and share
12:27
and that kind of thing. In fact, you know, I would say
12:30
there are a few 100 people in the world roughly who know how
12:33
to build this kind of infrastructure, this sort of
12:35
speed and scale. And they all know each other, of
12:37
course. Like mostly they live in like A5
12:40
block radius in Soma in California or something like
12:43
that. They probably all have drinks
12:44
with each other every Thursday or whatever, but they're too
12:47
busy to, you know, be sitting down and recording YouTube
12:51
videos and so on. These are the kinds of folks
12:54
who, I mean, honestly, on Christmas Day, they're
12:57
operating, you know, infrastructure and they're
12:59
procuring stuff. That's the pace at which this
13:03
this space is moving right now. And again, like the furiosity
13:07
with which these folks are operating.
13:08
And so they're, they're figuring this out on the fly.
13:12
They're sharing with each other within their community.
13:15
I just don't think there's been enough, almost like enough
13:19
duration for the learnings to propagate.
13:24
And you know, also because of the rate of change in this
13:27
industry, the learnings are going stale really fast too.
13:32
So this is not an easy space to dive into.
13:36
So how then when something is updated and you have better
13:41
throughput with one provider, does that not necessarily the
13:48
knowledge get propagated? But just like how do you update
13:52
and keep tabs on it when you're building out your data center
13:58
and you recognize, whoa, there's actually a better way of doing
14:01
it. So all of the inventory that I
14:04
was going to order now let's upgrade to this.
14:06
But that means that these five things downstream are going to
14:10
be affected too. Or how do you know like the
14:13
second and 3rd order repercussions I guess of?
14:17
Such a great question and let's let's break this problem up a
14:20
little bit because I think there are a few different answers.
14:23
1st, the, the teams that have started to figure out the
14:28
answers to this question are the teams that have been at this a
14:31
while. And by a while I mean more than
14:33
a year, right? Like the teams building, you
14:37
know, real generative AI infrastructure at you know, 100
14:40
plus MW scale. You can count them mostly on a
14:44
couple couple hands. And these are teams that are
14:48
practiced a life cycle management of their
14:52
infrastructure. And I think what this roughly
14:55
looks like is because of the rate of change and the rate of
14:59
turnover in these environments, it's not like let's RIP and
15:05
replace all that stuff that we built 18 months ago.
15:09
It's well, we're building 40 more data centers this year.
15:14
And so the new data centers are going to get the new
15:16
architecture and that new architecture is, is changing, is
15:21
turning over every, you know, four to six months roughly with
15:24
a lot of alignment and similarity, right?
15:26
Like there's a lot of convergence on some of the
15:28
basics. So I think that's roughly what
15:32
it looks like. And then, you know, the other,
15:34
the other segment here is, and this is the honest reality,
15:39
probably about 100 neoclouds have popped up in the last
15:42
121824 months, right? And these folks have not tackled
15:46
this problem yet. They're still building
15:49
Greenfield. You know, they're they're first
15:52
or second generation of data center infrastructures.
15:57
And you know, we in in a totally different world in enterprise
16:03
where infrastructure has lived for decades.
16:06
You know, we think and talk all the time about life cycle
16:09
management, end of life, end of support.
16:13
When is it time for us to take out that old switch and replace
16:16
it with the next generation? How do we think about network
16:19
refresh teams in AI infrastructure?
16:22
Haven't, this is an absolute, but as a rule of thumb haven't
16:28
gotten to life cycle management yet.
16:29
And so like this is a problem that is coming and not a problem
16:33
that these teams generally have. A great set of answers for
16:37
today. Well, if you have a new data
16:41
center that has all the newest and best components, but you
16:47
also have the second third generation data centers, is the
16:52
demand that you've been seeing for those second and third
16:57
generation data centers and what they can provide for GPU usage,
17:03
is that just skipping to the newest one or is it still like
17:08
there's just as much for as far down and as as many generations
17:15
back as it can go? So one thing I won't pretend to
17:18
be is an actual operator of AI data centers.
17:21
And so, you know, only they know they're real like business
17:24
models and you know kind of what the the demand looks like across
17:29
their different kinds of footprints.
17:31
But my perception is that the demand here is deep.
17:36
And you know, what you get with a current generation, I don't
17:41
know, you know, NVIDIA Super Pod, you know, with, call it,
17:47
you know, GB 300 architecture or something like that is
17:51
efficiency relative to prior generations in a bunch of
17:57
dimensions, power efficiency, cooling efficiency and so on,
18:01
right. So really that's what we're
18:03
trading off. Like there's a ton of demand
18:05
that is still being served by older architectures because
18:09
they're available and they're online and they work.
18:11
But as we transition toward newer generation architectures,
18:15
you know, we're serving more demand per Watt for example, or
18:18
more demand per well, I think Watt is really the right, the
18:22
right measure. I do know that some friends of
18:26
mine did some calculations on their usage and it was cheaper
18:29
for them to use the newer versions.
18:34
Even though it's more expensive per hour, they could get more
18:37
compute and get things done faster, and so they didn't have
18:41
to use as many hours. Yeah, I think that's that's
18:45
roughly a simple and good way to think about it.
18:48
Now put yourself though in the shoes of, you know, a large
18:52
Neocloud operator or you've made, you know, billions of
18:57
dollars of investment in, you know, last year's generation of
19:03
infrastructure. And I think this is a general, a
19:07
general question in the market today that we don't have great
19:11
answers to how long is the useful life of that
19:15
infrastructure? And we haven't reached it yet.
19:18
This whole, this whole push really roughly has only been
19:21
going on for, you know, 24 months or or thereabouts, right?
19:25
And so I think there's a lot of learnings we're going to have
19:28
downstream over the next 234 more years about what happens to
19:32
the two 3-4 year or four four generation old infrastructure
19:38
and is do the economics work to RIP and replace it?
19:43
Do we just keep it online serving less, you know, less
19:48
critical workloads or workloads that are more amenable to the
19:52
lack of efficiency there. What do we do?
19:55
We don't know yet, I don't think.
19:57
Yeah. And so now can you take me
19:59
through this stack again from the the tangible stuff you can
20:04
touch all the way to the more software type intangibles and
20:11
how you can keep track of all of it and then know what needs to
20:16
be done where or how much of this we have?
20:19
Is that enough of what we've got and just be able to keep the
20:24
trains running on time? Yeah, so, so much about this
20:30
spanning, you put it really well like the tangible all the way up
20:34
to the sort of the the actual outcomes which are token
20:39
generation, you know, roughly. So much of that is about the
20:45
data through thread of what are all of the things that make up
20:50
our ability to produce this outcome and how do they interact
20:53
with each other and how do they interlock.
20:55
And so if we, if we rewind just a little bit, we talked about
21:02
the earliest phases of these AI infrastructure builds being
21:07
we've got, you know, we've, we've procured some acreage
21:10
roughly and we've got a power commitment from somebody.
21:13
From that moment forth, the teams that are doing this best
21:18
are managing from intent, they're designing and they're
21:21
carrying the data of their design all the way through to
21:25
token generation. And what I mean by that is, OK,
21:29
we've got, you know, 200 acres of space and we've got 300
21:32
megawatts of capacity. What's the shape of the building
21:36
we're going to build? How many racks can fit in it?
21:38
What are the thermal calculations say about that?
21:41
What kind of power and cooling density can we achieve?
21:46
What sort of the inference or training infrastructure can
21:54
align with those parameters? What NVIDIA gear are we going to
21:58
buy or what switching footprint are we going to buy?
22:02
All of that stuff ultimately needs to end up in a data model.
22:06
And this is what Netbox is that captures the inventory.
22:14
Like what are all the components, the
22:18
interrelationships between that inventory.
22:20
Like on this interface in this switch, we're plugging in this
22:22
cable with this length and it's going to, you know, rat in this
22:25
cable run to this server, in this rack, in this, you know,
22:29
geometry in the data center. And all of that has to flow all
22:33
the way back to like power and cooling and so on.
22:36
All the other that has to flow from design intent.
22:40
This design is going to meet our, our parameters for thermal
22:46
thermals and cooling characteristics and power
22:49
consumption and token generation.
22:52
What happens if you don't do that is at some point you, you
22:59
have so much logistical complexity that you can't keep
23:02
track of all this stuff. And we see this one team start
23:05
with spreadsheets, right? Like the canonical way to do
23:07
this would be like, OK, let's do some power calculations and all
23:10
right, we need 18,372 cables and they all have these different
23:16
lengths. At some point, you know, that
23:18
spreadsheet gets messed up and you ordered the wrong thing and
23:21
that, you know, fun anecdote from a couple months ago.
23:24
For me now spending time with one of the largest producers of
23:29
fiber optic cabling in the world, their number one business
23:32
problem returns. Everybody orders the wrong cable
23:36
lengths, right? And why does that happen?
23:38
Bad management of design data and incorrect cable calculations
23:44
that are maybe not taking into account things like cable bend
23:48
radius or, you know, obstructions in the the physical
23:54
facility. So data really has to be the
23:57
through thread from, you know, as early as like I've got some
24:01
space and some power all the way through to this is running.
24:06
And then, you know, one other dimension this that I think is
24:09
important to start to pull apart is, you know, we're talking
24:15
about AI infrastructure is sort of this monolithic concept, but
24:18
there are different goals for different kinds of
24:21
infrastructure. And you know, we see a range
24:26
from I'm building a gigantic, you know, 500 MW facility to do
24:31
at scale, training for a research lab, single tenant
24:35
infrastructure, you know, all the way to things like what what
24:39
I call retail Neocloud infrastructure.
24:43
We're going to let people come and swipe a credit card and rent
24:45
some GPU capacity for a few hours and then, you know, return
24:49
it to the pool. So a lot of ephemerality and
24:52
then over to what I usually think of as AI factory
24:57
infrastructure and maybe like pre canned builds, even things
25:01
in, you know, shipping container sized prefab pods that might get
25:07
shipped off to some enterprise, a bank or whatever.
25:10
Who wants to buy some AI and doesn't have the capacity
25:12
themselves? And so we also have to consider
25:15
like, what's the, what's the end use case?
25:18
Who's going to consume this? How are they going to consume
25:20
it? That has to feed into the design
25:22
as well. Oh man, to just trying to grok
25:29
this idea of thousands of different cables and laying them
25:36
out across a gigantic data center.
25:39
And I know that when I chatted with Andy from Vast, he was
25:43
saying that a lot of times folks will break up the data centers
25:47
into four different grids or 8 different grids.
25:50
So that if for some reason you lose some power, hopefully you
25:54
don't lose power in the whole thing and you're able to kind of
25:59
salvage whatever you lose. What you just touched on there
26:02
is yet another kind of like outcome constraint, a resiliency
26:07
or redundancy constraint on power.
26:10
And we see the same thing on connectivity and we see the same
26:12
thing on cooling and so on, right.
26:16
So you know, really what we've just described as a gigantic
26:21
math problem where there are all these constraints on one end and
26:26
all all these inputs on the other end.
26:29
And right now a mix of expert humans and software like Netbox
26:36
or other like a CAD tools and so on are going together and out
26:41
the other end is popping, you know, kind of a design that you
26:45
know to best guess meets those constraints.
26:48
And you said something about like the bend constraint that
26:52
some cables have. I imagine that's just because
26:55
you can't put a 90° angle in it like a hose.
26:59
The water doesn't come out. Yeah, exactly, exactly like
27:01
that. Like light is traveling through
27:03
these fiber cables and so you bended 90° like the light
27:06
doesn't doesn't make that bend. And so there's a bend radius you
27:09
have to take into account. So like all kinds of fun math in
27:14
in these data centers. And does I think I heard
27:18
something about NVIDIA giving a lot of suggestions on how to set
27:25
up the data centers or they have standards?
27:28
Well, this is such a great topic for us to broach a little bit
27:30
because one, one of the, I think one of the drivers of speed in
27:39
these, you know, kind of hyper complex, many dimensional like
27:45
constraint satisfaction logistics problems is going to
27:49
be data standardization. And here what you're referring
27:53
to are, you know, blueprints or reference architectures that
27:56
that NVIDIA provides for. Here is our blueprint for a, you
28:02
know, an H-100 super pod or something like that.
28:05
And, you know, you, you need power in this configuration and
28:09
you need cabling that looks like this and you need this kind of
28:11
switching footprints and you need to plug things in this way
28:14
and these sort of cooling characteristics.
28:16
And, you know, think of it as like an instruction set for how
28:19
to build a working pod in their blueprint with, you know, kind
28:24
of all of the physical and logical elements preconsidered
28:28
for you. So that you, if you are not a,
28:33
you know, an expert at designing these things from scratch can
28:36
say, OK, I want to implement that blueprint.
28:38
Here's everything I need to buy. It's it's almost like the Lego
28:41
brick instructions, right? For how do I get, how do I get a
28:44
working in a pod in in Nvidia's definition and now extend this.
28:50
We see this from fiber optic cabling providers.
28:54
They have they have blueprints or reference architectures.
28:56
We see it from data center fabric, network fabric vendors
29:00
like, you know, high density InfiniBand fabric blueprints and
29:07
so on. These blueprints, the way they
29:10
manifest today mostly is PDFs, right?
29:14
So if you, you know, if you're, if you're going to build a big,
29:17
huge, you know, 300 MW data center, what are you doing?
29:20
First, you're reading a lot of PDFs that are very technical and
29:25
very and then you're needing to interlock them with each other.
29:27
You're needing to take this fabric design and this compute
29:31
design and this cabling design, etcetera, right?
29:35
And figure out how do I marry these things up.
29:38
And we're not here yet in our ecosystem or generally today.
29:42
But what we are working on here is how do we start to create
29:45
some programmatic representations of not just
29:51
these reference designs, but also the componentry that goes
29:53
into them. This is this is one of the
29:55
things that Netbox has really brought to the space is the
30:00
notion of a de facto industry standard way of modeling all of
30:04
this information. The vendors and data center
30:10
operators and tooling companies building, building software used
30:16
by these teams and really everybody can sort of count on
30:19
and align to here is how we're going to share data about cable
30:23
lengths and cable bends and cable geometry and cable types.
30:25
Here's how we're going to share information about switches and
30:29
servers and routers and firewalls and storage devices
30:34
and so on. Physical dimensions, logical
30:36
dimensions, configuration details, interfaces.
30:40
And if we can come up with a programmatic language for that
30:43
information to be shared, then we can start to automate much
30:46
faster. Then these teams can start to
30:50
automate the design process by consuming vendor data in
30:54
canonical forms. So this is one of the areas that
30:57
we're working on. And this is all about all the
30:59
way back to what we talked about earlier, the, the bottle, the
31:02
logistics bottleneck, like how do we solve for that logistics,
31:08
the procurement bottleneck of I've got stuff that I got to get
31:12
from 100 different vendors and it's all going to arrive and it
31:16
all has to work together. And right now that's a very
31:19
human problem and humans are the bottleneck in that scenario.
31:22
And so if we can programmatize and automate that logistics
31:26
process around a standardized data, as an industry, we're
31:30
going to move a lot faster. Well, it does feel like that's a
31:35
perfect use case for trying to automate it.
31:39
And knowing that there's, you have all these constraints and
31:44
you have your inventory, how can we make sure that what we have
31:49
matches our constraints? I also was thinking as you were
31:54
talking about that, like I imagine there's people, as you
31:58
were saying, those 200 people in the world that understand how to
32:03
do this and have been doing it for a while that are trying to
32:08
figure out the best way to do it and they've got some tried and
32:12
true tested tricks or they're pushing the boundaries.
32:15
I remember reading A blog post back in the day from the Meta
32:20
engineering team about how they had two different clusters of
32:26
24,000 GPU's and one was using the traditional InfiniBand, but
32:30
then another one they were doing something different to try and
32:33
get a little bit more speed out of it.
32:36
And ultimately it it was kind of the same, I think.
32:39
But there's going to be that R&D that's happening too.
32:43
And so in a way, what I was reminded of was like, it kind of
32:49
feels like you could do some flight simulator program or
32:55
something akin to that where you get to go in and try and say,
32:59
what if we did this and we tweaked these things and maybe
33:03
you would get a simulation. And then before going and buying
33:07
all that inventory or before spending all those hours, you
33:11
can at least like sanity check it.
33:14
So there's a term that you'll often hear in infrastructure
33:18
management or in all all other kinds of industries and the term
33:22
is digital twin. And that's roughly what you're
33:24
describing, right? Like we want to create a digital
33:26
twin of the eventual mega scale infrastructure that we're going
33:31
to build so we can pressure test it.
33:33
We already talked about a few of the, you know, the really simple
33:36
ways that people are doing that kind of thing today.
33:39
You don't really want to go build a 300 MW data center if
33:43
you haven't through your digital twinning and design process
33:48
proving to yourself that your design, you know, meets power
33:53
redundancy constraints, for example, right, Like, like that
33:56
would be you wouldn't be able to meet your SL as to your end
33:59
customers without considering those characteristics in your
34:03
design. Where we where we start to run
34:06
into constraints is the the depth, the efficacy, the
34:13
fidelity, I guess with which we can digitally twin these
34:17
environments in part because there are not standard ways of
34:22
sharing data about the characteristics of the
34:25
components and how they're going to work in practice and and so
34:28
on, right. So this is this is a key part of
34:31
the design process that we want ultimately to make higher
34:35
fidelity by having more current, more aligned and more granular
34:46
data about everything that goes into these environments.
34:49
So that we can pressure test, you know, in many different ways
34:53
and be confident that when we do go by, you know, a billion
34:58
dollars worth of stuff to deploy this data center, we're going to
35:01
meet those design constraints that we've laid out for
35:04
ourselves. And that fidelity isn't there
35:07
because it's still so new. That fidelity can be there with
35:13
a lot of hard manual labor today, and again, it goes back
35:18
to reading PDFs like design specs for individual servers or
35:23
switches or cables or whatever. That's where that information
35:28
resides today. Where it doesn't reside
35:32
generally without a lot of hard work by these teams is in a
35:37
common data model like Netbox where you can say, you know,
35:41
like I'm going to design this data center.
35:44
It's going to have this layout, it's going to have these
35:46
physical characteristics. It's going to have this cabling
35:48
architecture. It's going to have this network
35:51
configuration. It's going to have these, this
35:53
switching model. It's going to have these server
35:55
models, It's going to have etcetera, the the data about all
35:58
of those components that feeds the call it like the evaluation
36:04
of that design in a programmatic sense.
36:09
It's really hard to get at today.
36:11
It's sitting in PD FS, right? And now we look, we know where
36:14
that's going. Like data in PD FS is not is not
36:18
as mysterious as it used to be, like circular reasons to our
36:22
conversation, right. But like so ironic, yeah, isn't
36:25
it? But, but what we need to arrive
36:28
at is a, is a, is a way for vendors to expose that data
36:31
programmatically so that the people designing these massive
36:36
infrastructures as new component trees being designed every few
36:40
weeks for, for their new needs and new constraints can slurp up
36:46
that, that data about this infrastructure and pressure test
36:48
new designs, drive effective procurement workflows and
36:52
compress. This time NVIDIA has a term that
36:55
they use which is time to 1st train.
36:58
And that's really the North Star metric that we're talking about
37:02
often. You know, we have design
37:04
constraints we want to meet, we want to compress the time from
37:09
I've got space and power to I'm training my my model in my now
37:15
online infrastructure. And one of the biggest
37:17
constraints to that is all of these data logistics and all of
37:22
the pressure testing of the, the design through digital twinning
37:27
and effective aligned data in a data model like that box.
37:30
So that's what we're working on. Well, can you just imagine the
37:35
amount of time it takes to just plug everything in that and
37:40
that's just one piece of it. That's like probably you're in
37:43
the home stretch when you start plugging things in.
37:46
Well, now let's talk about that because I think this is the,
37:49
the, the, the, the act of racking, stacking, cabling,
37:53
plugging things in, right? That act is, is an emerging
37:58
constraint and that constraint is going to get solved in the
38:01
way we all think it is 2 pretty soon, which is robots roughly.
38:06
And AI is going to help us with that as well.
38:09
And So what do you need for a robot to walk into a gigantic
38:13
data center and know what to rack where and how to plug
38:17
things in and how to run cables and like all of that kind of
38:20
stuff. You need accurate data for that
38:23
robot to act on. And so I think that's another
38:25
another thing that we find ourselves thinking about or
38:29
having conversations with all these operators about
38:31
constantly. Like how am I going to give that
38:36
robot accurate, instruct accurate what I call field
38:38
operations instructions right now today, even a, even a human,
38:45
you know, like a data center technician, someone were asking
38:47
to walk into the data center and go down this cold aisle and, you
38:51
know, walk around to the front of this rack and unplug this
38:54
thing in this interface and plug it in over there.
38:57
That person isn't an architect. They don't often have all the
39:01
information and all the knowledge to, to, to deduce
39:05
exactly what to do without really clear instructions.
39:09
Often we'll call these instructions like a cut sheet or
39:12
a, like a, a standard operating procedure or something like
39:16
that. And, and now think about how
39:21
precise, you know, that cut sheet or those, those operating
39:25
procedures need to be. If the, if the thing we're
39:28
instructing is no longer a human technician, but it's a, it's an
39:32
optimist robot or something that's walking through the, the,
39:35
you know, the, the, the data hall and taking actions that's
39:40
happening today, for what it's worth, in certain large
39:44
hyperscale facilities. So that's another constraint
39:48
that we're going to start to compress is like the the racking
39:50
stackling, the stacking like the day 0 field operations and then
39:54
also the ongoing field operations because things break
39:56
all the time or need to be re racked or reconfigured.
40:00
I recorded a few musical albums back in the day and it was at a
40:07
studio that was owned by a guy who absolutely adored complexity
40:17
in the way that there were so many wires everywhere and he
40:21
really was into analog gear. And so things were going through
40:25
here and there and I didn't understand half of it.
40:28
But I spent enough time because I recorded like 3 albums at the
40:33
studio and I, I was there so much that he eventually just
40:37
gave me the keys and said, you know, like record when you want.
40:41
But 9 times out of 10 when I would go to record when I
40:46
wanted, I was troubleshooting for the whole session because I
40:52
couldn't figure out what the hell was going on with the
40:55
complexity that he had set up. And this was in a studio the
40:58
size of the room that I'm in right now.
41:00
I can't imagine a like 5 different football field length
41:05
data center. And and that's such a great
41:09
microcosm example because what you needed was something written
41:15
down somewhere that you could refer to this red cable here,
41:18
like plug it in over here if you want this outcome, right?
41:21
And that's if we distill it down, that's actually all Netbox
41:26
really is, right? Like it is that documentation in
41:30
a programmatic form that gets used to affect an outcome,
41:34
whether it's a physical outcome like plug this cable in here or,
41:39
or, you know, deploy this rack with, you know, this much space
41:46
around it for thermal reasons or whatever.
41:49
Or it's a logical outcome like assign this IP address to this
41:52
server so that it can be exposed to this end customer in this
41:55
way, right? That's, that's roughly all it
41:58
is. And I think the the simplest and
42:00
biggest take away here a huge amount of complexity, huge
42:04
amount of chaos, huge amount of demand, a huge amount of speed.
42:08
And what, what that requires is good data.
42:13
No surprise, right? Like when you can operate with
42:16
good data, whether it's at studio scale or whether it's at
42:19
mega scale, you eliminate those inefficient you would have been
42:22
recording in 3 minutes instead of, you know, using all your
42:25
time to figure out analog cabling.
42:27
So many long nights trying to figure out why the hell I can't
42:32
hear anything. Everything's working, but no
42:35
sound is coming out. What is going on?
42:37
Oh, man. So yeah, those, it's almost like
42:39
the the SO PS of the data centers need to be so clear and
42:47
crisp and precise because you have so many more variables that
42:52
are involved. And now if you're trying to
42:58
eliminate constraints in every single move because you, you
43:03
speed is king right here. King is royalty really.
43:07
And you're saying, well, could we throw robots at this or could
43:10
we throw, can we make this programmatic?
43:12
Can we try and use a different InfiniBand the next generation?
43:18
There's always something that you're trying to iterate on.
43:22
I can see how something like Netbox is going to be immensely
43:28
valuable, but also how people are just probably pulling their
43:33
hair out left and right when it's amazing that any data
43:36
centers are actually online right now, the time that I think
43:38
about it. It's such a, it's such a
43:40
fascinating space. And I like, you know, many
43:44
minutes into our conversation, one thing I'll admit, I'm not a
43:47
data center engineer. I have no idea how any of this
43:49
stuff really works. What we see is these teams that
43:53
are the best in the world at building these infrastructures,
43:55
like how they are doing it in this really interesting sort of
43:59
cross cutting view because they all are using Netbox to solve
44:03
this data problem. And I think you're right,
44:05
there's a there's a reason why there's only a few 100 people in
44:08
the world who are really pros at this.
44:10
It's a really hard problem to solve.
44:12
It's really dynamic. And I think there are some
44:14
really interesting analogies by the way, to like the early days
44:17
of the Internet. You know, 20 years ago, you
44:20
could get in a room with roughly the 200 people who ran the
44:23
Internet. They all knew each other.
44:25
They knew how it worked. They talked to each other.
44:27
It was evolving at a really fast clip.
44:30
That's what's happening in AI infrastructure right now.
44:33
Did this knowledge will propagate.
44:34
It'll start to normalize over time how you do this stuff.
44:38
What we don't know yet is whether, like, the diversity
44:43
that we're seeing in the ecosystem will persist.
44:46
Will there continue to be hundreds and hundreds of, you
44:48
know, operators of AI infrastructure?
44:51
Will we see more consolidation? There are real economies of
44:54
scale in this space, you know, to the, you know, ultra
44:58
hyperscalers. I think that's also going to
45:00
shake out over the next couple years.
45:03
It is. Absolutely intriguing how until
45:09
a few years ago there were three clouds and that was really it.
45:13
And now there's been this Cambrian explosion and so much
45:18
so we actually just released AGPU guide on the whole
45:23
ecosystem because I had, I just kept seeing all of these funding
45:28
announcements and the majority of them that were raising a
45:32
whole boatload of money were very low level infrastructure
45:38
plays. And it was like, well, what are
45:40
the value props that each one of these folks have?
45:43
I don't understand what the difference is between a modal
45:46
and a base 10 or a fireworks or together.
45:50
And then are they even going and building their own data centers?
45:55
Not really. They're kind of just renting
45:57
them off of other people's data centers that are already out
45:59
there. So then you have a little bit
46:01
lower that you could go to like the Lambdas and they are
46:04
building their own data centers and they're trying to do that.
46:06
And so I was intrigued by all that, which led me down the
46:11
rabbit hole of, well, OK, well, what are the value props here?
46:15
Like what do I when would I want to use one versus the other?
46:18
And I, I think you use a great term here, which is Cambrian
46:22
explosion. The market answers, we don't
46:25
know the answers to like the, the actual value props that are
46:30
going to stick in the long term. And that's why this Cambian
46:33
explosion is happening. There's an exploration happening
46:35
in the market right now, right? And so it could be that all
46:39
these different business models make sense and all end up
46:43
persisting. And you know, we retain this
46:45
diversity of kinds of infrastructure providers to meet
46:49
these ultimate value propositions, whether it's, you
46:53
know, really lightweight, ephemeral, ephemeral ability to
46:57
pop up new infrastructure really quickly on top of existing
47:00
collocation footprints, or whether it's in a mega single
47:04
tenant purpose built facilities with, you know, specialized
47:09
cooling and power and whatever characteristics data centers in
47:13
space, who knows, right? Like, so we're seeing an
47:17
exploration happen right now. Some of the some of the value
47:20
props that I do think are going to continue to to like way into
47:28
a diversity of infrastructure types and providers at least for
47:34
for some time. 1 is sovereignty. And this is an interesting one.
47:39
But if you think about how important to the world AI has
47:44
become in such a compressed time, what we're seeing is
47:48
countries, nations, right, say this is strategically important.
47:52
We need sovereign AI infrastructure.
47:54
We can't be dependent on AI infrastructure sitting in the
47:57
United States, for example. And so you're not going to solve
48:01
sovereignty with big monolithic hyperscale infrastructures.
48:06
We're seeing sovereign AI infrastructures pop up all over
48:08
the place. Now there are companies that
48:10
specialize in building sovereign AI infrastructure.
48:14
So that may be a specialization that persists.
48:18
Another kind of specialization that we already touched on a
48:21
little bit is the the spectrum from, you know, single tenant
48:28
mega scale, like, you know, training footprints for research
48:33
labs all the way to, you know, slice and dice retail type
48:39
infrastructures. These are very different and use
48:42
cases and value propositions that may not all be best served
48:45
by the same kind of company or architecture or
48:51
operationalization. And then one more that we see
48:56
often is, and this is akin to the sovereignty 1, certain kinds
49:03
of enterprises really want to own their own AI infrastructure.
49:05
We see this a lot in financial services.
49:08
We see it a lot in healthcare, in particular in pharma where
49:12
high performance compute is really valuable to have for drug
49:15
discovery and things like that. We see it a lot in government
49:18
obviously. And one of the most interesting
49:21
places that we see this a lot is in energy.
49:25
If you're if you're generating, you know gigawatts of of power,
49:32
it's a pretty natural extension your business to like put some
49:35
GP us next to that power. But what all of those kinds of
49:39
organizations are not are hyperscale data center builders
49:43
and operators, right? And so, and when I think of the
49:46
term AI factory, which we hear a lot, it's mostly about that sort
49:50
of value proposition. How do I provide like the
49:55
ability for, you know, financial services or pharma or government
50:00
or utility to get an AI data center without themselves having
50:05
to develop all of this expertise?
50:07
So there's a bunch of different business models that we're
50:09
seeing be explored right now. They aren't obviously all served
50:13
by the same exact kind of like converged, you know, hyperscale
50:19
company. So for a while, I think we're
50:21
going to continue to have this diversity.
50:24
Well, what you said earlier about a shipping container,
50:28
basically a data center in a box.
50:31
Yep is new to me, I had never heard about that.
50:35
Actually something that has existed for some time, even back
50:39
in, you know, back when we all talked about cloud as the next
50:43
big thing, which ship has sailed a little bit, we've all consumed
50:46
cloud and now like some repatriation or whatever.
50:48
But deploying private clouds in shipping containers or like
50:54
think of them as just prefab cloud infrastructures or edge
50:58
infrastructures, pretty well established concept.
51:02
And so maybe no surprise that the same methodology is playing
51:07
out in in AI or GPU infrastructure as well for
51:10
organizations that need to be able to deploy, you know, not
51:14
hyperscale, but owned single tenant footprints.
51:18
Yeah, and I like how you say the classic quote from Amazon,
51:25
right, where stick to what makes your beer taste good.
51:30
And if you're a financial services company, building data
51:34
centers is probably not what your bread and butter is.
51:38
So how can you reap the benefits of all of this without having to
51:44
become a data center expert? Is it that you just hire one of
51:48
these 200 people or is it that there's ways now that companies
51:54
are popping up to service that demand?
51:57
And that's exactly it. They're, they're companies that
51:59
we're seeing that are, I call them AI factory companies
52:02
roughly that whose value prop is exactly that I'm going to
52:09
deliver like an AI data center to meet the needs of this bank
52:14
or whatever it is. Or there's, there, there are
52:20
spaces that have existed forever and what we call systems
52:22
integration, right? So the the big systems
52:27
integrators who forever have solved problems of building IT
52:30
infrastructure or OT infrastructure or whatever for
52:34
enterprises all now have AI factory or high performance
52:40
computing practices who consolidate some of that
52:43
expertise. And repeatedly for their
52:45
customers are able to procure all the equipment, solve all
52:48
these problems that we've just talked about and pump out AI
52:51
data centers, you know, to spec to meet the needs of their
52:54
customers. So that's a business model that
52:56
I think will persist. Now, when you are looking up the
53:04
stack, how high up do you go in terms of that maintenance piece
53:11
or if we need to do updates to the GPUs?
53:15
I know that can be a headache in itself because every second that
53:19
the GPU is offline, you're burning cash.
53:23
Are you also looking at that or is there a level where you you
53:28
say, all right, cool, we've we made it to here and we kind of
53:30
stop. We have spent a lot of time
53:33
today talking about like the build process time, time to 1st
53:37
train to use NVIDIA term, right? I would say at least as
53:42
important is the kind of operational processes around
53:46
this infrastructure is up and running and living.
53:49
And then that blade in that rack died and what do we do, right?
53:54
Or yeah, we need to do some software upgrades on, you know,
53:58
these 18 racks or something like that, right?
54:01
So here again, the data is critical.
54:07
All of these systems in real time are interacting according
54:11
to some configuration. This server has this IP address
54:15
on this interface, it's connected to this switch, it's
54:18
running this software, it's allocated to this tenant or this
54:23
end customer or something like that, right?
54:25
So all of that data context equally important once you're up
54:30
and running for things like I want to drain this
54:34
infrastructure so that I can do a bunch of field operations on
54:37
it or reconfigure it or something like that.
54:41
And you know, I would also say like the pace in an operational
54:47
state is another order of magnitude faster, right?
54:50
Like you were talking about operations or sorry, automation,
54:55
things can happen that are not driven from design when you're
54:59
operating infrastructure like a backhoe can cut a piece of fiber
55:04
and that just happened. Now what now what has to happen
55:07
to flow around that? All of that must be automated.
55:09
There's no way to to deal with that situation in a satisfying
55:13
way without automation. And so automation demands good
55:16
data and. I, I just keep coming back to
55:22
like, there's so many ways that this can fail from human error
55:28
to just, yeah, power outages or you blow a fuse and you now have
55:36
to troubleshoot. You not have to troubleshoot.
55:40
I mean, and you're right, there's so many ways this can
55:42
fail. This is part of why as early as
55:44
the design phase, we need to define our reliability or
55:49
resiliency constraints and design for failure.
55:53
Everything always fails in infrastructure.
55:55
That's that's just a truism, right?
55:57
At some point a backhoe will cut that, you know, piece of fiber
56:01
or you know, that generator will blow up or something will some
56:05
that will happen, right? And so we design for those
56:08
failures with appropriate redundancy and so on.
56:11
But then we have to resolve those failures.
56:14
And this is what you're getting at.
56:16
We need the data and the visibility to understand what is
56:20
that thing, what's it connected to, what is impacted by this,
56:26
and what are our options for doing something about it.
56:30
And you know, a, a, a model like Netbox encodes all of the
56:35
logical and physical information.
56:37
To understand that stuff, you need to couple it with tools
56:40
that are observing behavior of these systems in real time.
56:43
We call these observability tools for understanding the flow
56:47
of network traffic or understanding thermal
56:50
characteristics in the data center, like actual heat sensors
56:53
on all the racks or power load characteristics or anything you
56:59
can imagine. We want to have visibility into
57:01
as well, to couple that data with the context of what we've
57:05
built and how it is supposed to be working so that we can ask,
57:09
is the design intent being met based on our observations, based
57:13
on what we can sense in this operational environment?
57:16
And if not, what are we going to do about that?
57:18
Right? That's that's when we get humans
57:20
to work right? Like these switches are behaving
57:24
differently than we expect for our design humans.
57:28
Can you look at all the data and decide what we should do about
57:31
this? And then increasingly, of
57:32
course, like we hope that AI operations sort of closes that
57:36
loop as well. AI agent like this seems to be
57:39
behaving differently than we expect.
57:41
What you diagnosis the issue and what would you do about it?
57:45
Well, this is fascinating because you're, when you're
57:47
talking about observability, you're referencing like the
57:52
physical, the actual things you can touch.
57:54
And whatever I think about observability, it's like, Oh
57:57
yeah, is there like a data dog type of tool and solution is
58:03
what my mind instantly goes to when I think of observability.
58:07
And and that matters as well, right?
58:09
Like because again, think about the value proposition in these
58:14
infrastructures. Ultimately it's tokens are being
58:16
generated roughly. And so you'd need to observe
58:21
what what I think of as the logical, like the flow of
58:25
network traffic or is the server up or down or whatever that's
58:28
that's a part of the process of generating those tokens.
58:32
But the logical parts of that process are underwritten by the
58:36
physical parts of that process. There are servers and there is
58:39
power and there are fiber optic cables and so on that make up
58:45
the ability to generate those to run the software that's
58:47
generating those tokens. And so it's not enough just to
58:49
observe things at the logical level, especially if you want to
58:53
diagnose like why did that IP address suddenly go offline?
58:57
Well, it could be because somebody cut, cut the cable and
59:02
we need to be able to diagnose down to that level to remediate
59:05
that issue and get that IP address back online, right?
59:08
So the couple are observability of the physical and the logical
59:11
with our context about how this stuff is supposed to be working,
59:15
marry all that stuff up. What you figure out is the delta
59:18
between our like our observations and what's really
59:22
happening and what our intent says or what the design says,
59:26
and then that delta is the problem that we have to go fix.
59:30
Man, this is so cool to learn about.
59:32
There is so much involved in it and it is a very high value
59:43
problem because there's so much on the line and as we know, like
59:50
GPUs are measured in seconds and so you need to keep them online
59:55
at all costs. And this is the operational
59:59
problem they were talking about IA 100% agree.
1:00:01
You need to keep them online at all costs when there's a.
1:00:05
This is why these spaces like observability for example are
1:00:10
such high value spaces because the outcome when something.
1:00:17
Fails is deeply negative, right The the other, the other value
1:00:24
equation going back to conversation we had earlier is
1:00:29
the speed to to to deliver value from these operating
1:00:33
infrastructure to get them operational.
1:00:35
And I think the other thing that we're seeing in this, in this
1:00:38
space is the demand is so high and the expectations are so high
1:00:41
to meet that demand that delays in speed, delays in logistics
1:00:46
like when those constraints are bottlenecks create drag to bring
1:00:53
a infrastructure operational that has a really negative
1:00:56
impact on value for the world too and certainly for the
1:00:59
companies that are building this infrastructure.
1:01:01
So constraints both pre operations and post operations
1:01:04
on speed, resiliency, scale effectiveness, that kind of
1:01:08
thing. What are you seeing as far as
1:01:11
the average of time it takes to stand up a data center?
1:01:17
It's compressing really, really fast now.
1:01:19
I mean I think if you asked me that same question five years
1:01:22
ago, I would have said a typical data center is a 2 year build at
1:01:26
least. And today I would say, look, I
1:01:31
had, I had a week in November and I had calls with seven or
1:01:38
eight of our customers building a infrastructure where every
1:01:43
single one of them said, listen, we're going to be 10 Xing our
1:01:46
infrastructure over the next six months.
1:01:49
You know, we need your help in these ways or we need need to
1:01:51
make sure we're ready in, in way XY or Z.
1:01:54
So I mean, just think about that math, right?
1:01:57
These are organizations that have a large scale data centers
1:02:02
online today and they're saying it's 10 Xing in the next 6
1:02:05
months. They're building a data center a
1:02:07
week roughly, you know, it's, and now there's now let's be
1:02:14
clear, right? They're not starting from
1:02:17
scratch and getting it online in a week, right?
1:02:20
These things are operate, they're happening in parallel.
1:02:24
It it, it is a complex product project management, you know,
1:02:28
and logistics management problem to line all these things up to
1:02:31
pump out that many data centers that quickly.
1:02:34
But that's roughly the pace that we're seeing now.
1:02:38
Where do you feel there are pieces missing right now with
1:02:45
the, whether it's in the build or it's in the operation parts
1:02:49
of the data center? Yeah.
1:02:51
Like where can we improve? Yeah, I think, I think the
1:02:55
integration of all these things that we've talked about is a
1:02:58
really tough challenge today. It's way tougher than it should
1:03:01
be. Today we, we delved into some of
1:03:03
this in talking about the notion of standardization of say
1:03:09
component data, for example, from, from networking and server
1:03:13
and GPU and cabling and whatever vendors, right.
1:03:16
Like the more we can start to standardize on how vendors share
1:03:21
that information, the more we can compress the timed
1:03:25
operations of the time to 1st train.
1:03:27
So that's, that's one example of the integration problem.
1:03:30
Another example we talked about once these infrastructures are
1:03:34
online, they're all these systems that these teams are
1:03:42
using to understand and to observe like how they're
1:03:46
behaving and integrate that with, how are they supposed to
1:03:48
be behaving and integrate that with.
1:03:50
We want to drive change and we want to scale this.
1:03:54
We want to support new kinds of workloads and so on.
1:03:59
And so, you know, right now these teams are having to cobble
1:04:04
too much of that together, glue too much of that together on
1:04:07
their own. And so I think we'll see
1:04:09
continued like tightness in the way tools like Netbox or we have
1:04:16
observability capabilities or drift detection capabilities or
1:04:22
things like that all glued together tightly to form kind of
1:04:26
a cohesive approach for these teams from as early as design
1:04:32
intent all the way through this thing is running and I'm
1:04:35
changing it without having to, you know, themselves integrate
1:04:41
many different components or data sets.
1:04:43
Is it going to be the different providers start to integrate
1:04:48
tighter or is it going to be something where you see Netbox
1:04:51
being the one single pane of glass that you can look through?
1:04:56
My general point of view is there it's going to be both
1:05:01
roughly like one of the things that we think about is that
1:05:05
there will be black platform consolidation in this space.
1:05:08
It does make sense to have a tightly integrated set of
1:05:14
tooling and capabilities to serve the end to end needs,
1:05:18
given how tightly these operators need to need to
1:05:22
operate from as early as design all the way through to, you
1:05:26
know, kind of managing change in operational infrastructure.
1:05:29
And we're building for that. However, the reality is every
1:05:33
infrastructure is different there.
1:05:37
There's no such thing as like a magic blueprint as as hard as
1:05:40
NVIDIA and others are trying that everyone is going to follow
1:05:45
that. The Cambrian explosion that we
1:05:46
talked about just sort of dictates that right?
1:05:48
There are different value propositions, there are
1:05:51
different ideas that people are trying, and so there probably
1:05:54
will not be 1 magic, you know, tooling stack to rule them all.
1:05:58
We'll build, we'll build for that.
1:06:00
But the other guiding principle that I think is really important
1:06:04
in a space like this, and we have this applies outside of AI
1:06:06
data centers too, into enterprise infrastructure or OT
1:06:09
infrastructure. All kinds of infrastructure is 1
1:06:13
of openness and composability. So, you know, one of the things
1:06:17
that we haven't talked about very much is the fact that
1:06:19
Netbox is open source. Almost everything that we build
1:06:22
has open source elements and that's really purposeful and
1:06:26
important. We need it to be easy for these
1:06:28
teams to decide. I'm going to use all the net box
1:06:30
stuff to run this stuff except for this one piece because I
1:06:33
really want to custom build something over here for, I don't
1:06:37
know, observing or driving automation in my infrastructure
1:06:40
or some design element that's unique to my business, but that
1:06:45
needs to be able to integrate with the rest of the stack.
1:06:47
And this is where openness and APIs and composability of the
1:06:52
tool chain that these that these operators use to manage their
1:06:56
infrastructures is really, really important.
1:06:58
And that's I think a fundamental to how we think about the space
1:07:01
and how I expect it to to continue to evolve.