Latest / Agentic Conversations (formally mlops.community) / Building Out GPU Clouds // Mohan Atreya // #317

Transcript

Building Out GPU Clouds // Mohan Atreya // #317 · Agentic Conversations (formally mlops.community)

0:01
We're firing on all cylinders. GPUs are hard.
0:07
You've been dealing with them a lot.
0:09
Why are they so hard? The first is they are hard to
0:13
get hold of the one number one exactly the the the ones you
0:18
want. They may not be available and,
0:22
and usually, I mean, if you, if you look back, you historically
0:26
only had two options. Your company probably would have
0:30
purchased a bunch of GPUs in a data center exactly hardware and
0:34
set things up. And if, if the company is large,
0:37
the enterprise is large, they probably have a good number of
0:39
GPUs that IT has set up and made available to you or the last 15
0:45
years or so when you had the folks like AWS as your Google
0:51
and they help, you know, they provide GPUs for rent.
0:54
Essentially, these were the only two games in town.
0:59
And the problem that I think people run into is for many
1:05
tasks tasks, you need the exact kind of GPU to do the thing
1:08
optimally, and you may not get hold of that, or it might be too
1:12
expensive. Then you're kind of stuck
1:14
between a rock and a hard place. Especially if your company is
1:18
okayed a lot of spend on AWS and AWS either doesn't have the
1:22
capacity you're looking for or. So I see this a lot with some
1:27
when I talk to some customers, one of the complaints they they
1:31
actually two things they'll end up saying is 1 is IT doesn't
1:34
understand what I exactly need because IT operates in standards
1:38
based world. So if it's outside the known
1:41
bubble for them, it, you know, it's a lot more work for you to
1:45
convince them and say, hey, I need that.
1:47
Even on a cloud like AWS. The second problem they run into
1:51
is they really don't know if that thing is going to work for
1:56
them or not. A lot of lot of stuff in AIML is
1:59
experimentation because you learn about it and then you have
2:02
to try it. And if it's too hard to get
2:05
access to something before you can try it because you may
2:08
conclude this is not the thing I need.
2:10
And then some of these clouds also make it so hard for you to
2:13
get access, like do a one year commit.
2:15
So true. If you want to do something with
2:18
just reserve pricing, the pricing is astronomical.
2:21
Exactly. Exactly.
2:22
And that gets in the way of doing business.
2:24
Right now, none of that exists with general purpose compute,
2:29
but we struggle with that with AI infrastructure today.
2:33
Yeah. Well, you did mention there's
2:34
there's kind of these two paths, which is you either buy the
2:37
hardware and set up your own cloud or your own GPU cloud, we
2:40
could say, or you go to the cloud providers.
2:43
In the recent, I would say 2 years, three years, we've had
2:46
the explosion of the models and the Lambda labs and the base
2:50
10s, All of those folks, correct, have come out of
2:53
nowhere or seemingly nowhere. And there's a lot of demand for
2:57
that type of thing too. Yeah, that that we believe will
3:01
be a game changer for the market in and like you said correctly,
3:06
in the last two years things have cropped up and then there's
3:10
large number of providers who who are looking to set this up.
3:15
In fact, that's that's, you know, the kind of entities we
3:18
guys work with now, it's hard for everyone.
3:22
And for them, the problem is, you know, they do the hard work
3:26
of finding the real estate data center power because these GPUs
3:30
are power hungry. Then you have to buy the GPUs,
3:34
rack them and and they do that. Then they start looking at, OK,
3:39
I got a bunch of GPUs, how do I convert this into a GPU cloud?
3:43
And a lot of them have hardware shops, data center shops,
3:47
software shops and running a service like naws kind of an
3:52
experience is not easy. So that gap is what we guys fill
3:58
because they need to go to market fast because they are
4:01
probably paying interest on on the GPUs and if the GPUs are
4:06
idling and they have no, no customers when they're burning
4:09
money at the time. Yeah, if you're a core Weave,
4:11
they've done some very interesting financial
4:14
engineering to make those GPUs worth the while.
4:17
And they want to be saturating those GPUs 100% all the time,
4:21
correct? They are also a little lucky in
4:24
my opinion because if you look at their ten Ki mean the S1 that
4:27
they published before they went public, about 6070% of the
4:32
revenue comes from the one customer at Microsoft.
4:35
So effectively, they are, you know, they are lucky because
4:38
they had one big customer who could, you know, fund the whole
4:42
thing. But if you are a new GPU cloud,
4:45
you may not be that lucky because there's only so many
4:49
companies that do foundation models that need that kind of
4:52
scale. Microsoft, through Open AI, I
4:55
guess, was funding Core Weave's investment in that.
4:58
But what about the rest? So the kind of organizations,
5:02
the GPU clouds we work with, they seem to sell to enterprises
5:06
like our universities. We'll just give an example.
5:11
This was, this was a university that said, hey, if you want to
5:15
launch an AIML lab, we want to train the next generation of
5:19
data scientists. The thing is to do that, well,
5:22
you need GPUs. The university doesn't have
5:24
anything and it's too hard to set up.
5:27
But they want to launch the course now, right?
5:30
So they are working with these new GPU clouds, the Neo clouds,
5:35
and these guys are giving them experiences like, hey, I'll give
5:39
you a notebook as a service, I'll give you Ray as a service,
5:43
I'll give you Cube Flow as a service.
5:44
Of course, underneath the covers, they all have GPUs.
5:48
And I think this is the way by which the next generation of
5:51
data scientists, etc. Because when they graduate, they
5:56
need to have practical experience with real things.
5:59
Yeah, and a collab that you get a free 300 bucks a month or 300
6:03
bucks until you burn that 300 bucks.
6:05
Yes, that's not enough. Correct.
6:07
Exactly. And then this also has to be
6:09
structured, right, Like in a university setting, how do I
6:12
give you a degree? How do I, how do I get you a lab
6:16
where you can practice stuff? So these are the kind of
6:19
examples we see. And then the neo clouds, the GPU
6:22
clouds are enabling. In my opinion, the next
6:25
generation, the next wave of people will hit the market fully
6:29
trained on how to do AIM at at scale.
6:32
So you were talking about how basically you've got from the
6:37
very low end on the infrastructure stack to the
6:41
frameworks and like the pie torches of the world and all in
6:46
between, you've been playing in that dimension.
6:49
What are some things I imagine over the last two years that
6:54
you've seen as far as gotchas? Yeah, let's talk about a few
6:58
things. Of course, it's a complex stack.
7:01
A lot of stuff can go wrong, but let's talk about things that are
7:04
interesting. So when you buy a computer, you
7:11
kind of expect the CPU to last a long time.
7:13
Now, we're not talking about a single GPU here.
7:16
We're talking about multiple GPUs connected together, working
7:19
well together. And when you have that many
7:22
moving parts in a complex system, failures are possible,
7:28
right? In fact, in fact, Meta had
7:32
published an interesting stat in their LAMA white paper, I think
7:37
when they launched 3, I believe it's, I think a year old at this
7:40
time. I think they saw about 30%
7:43
failure rate. You don't expect that on a
7:46
computer, you never even see anything like that, right?
7:50
This kind of talks to the complexity of the environment.
7:53
Now the question is, if you are in the middle of a training run
7:57
and you have a GPU failure, what do you do now?
8:01
Do you abandon the training run that has been running for two
8:03
weeks? That would be bad.
8:07
So how do you replace capacity like a literally a swap
8:12
underneath the covers where you can still deliver an SLA?
8:15
That's amazing. So I think as the GPU cloud,
8:19
cloud, as any provider, whether you're an enterprise Oregon, a
8:22
wannabe GPU cloud, you have to take care of these things
8:26
because your customers expect that because they just want to
8:29
do a training run. Yeah, and trying to tell
8:31
somebody expect 30% problems, 30% of the time shit's going to
8:36
hit the fan. Exactly.
8:37
That's a really hard sell. Yeah.
8:39
And interestingly, you know, so one of our advisors, he worked
8:44
on Lama, right? So he kind of gave us some
8:47
inside stories on how difficult it was as an end user.
8:52
Within Meta is big. They have their enormous
8:56
internal infrastructure, but all the teams don't get access to it
9:00
at the same time. You get 2 weeks and the two
9:03
weeks you do your stuff. You better do everything you can
9:06
in those two weeks exactly. And a lot of times these teams
9:10
just have a hypothesis, right? They have a hypothesis that I
9:13
want to run this, and if it doesn't work, they got to wait
9:17
again in line, right? This is pretty frustrating as a
9:20
user because you don't know you. You you have a general sense of,
9:24
yeah, I think I should try this, this may work.
9:26
And I think that's the problem in general with EIML, because
9:29
you start with a hypothesis. It's not deterministic.
9:32
It's and as depending on what you find from that, you tune
9:37
your hypothesis a little more. This is difficult because the
9:41
data is informing you what is possible.
9:44
And it's not like a formula that you can say, if I do this, this
9:47
magical stuff will happen. So this is the day in the life
9:51
of an end user. And and if you bolt on a 30%
9:54
failure on that and say that too bad your two weeks are up, I
9:58
mean, that's terrible, right? And these failures are coming
10:02
from just GPUs being finicky. That and, and the connectivity
10:06
issues because, you know, I think I think the thing that
10:09
maybe a lot of us forget is it's actually a connected GPU system,
10:16
right? You know, you may have like 100
10:18
GPUs that have to work together interconnected and this
10:22
connectivity is where kind of you guys play, right?
10:25
How, how do I, if it's if it's in the same node, if I have 8
10:28
GPUs in the same node, how can they talk to each other fast?
10:31
If I have two, two servers, 2 nodes, how does internode
10:36
connectivity work? What if something, what if a
10:40
switch dice in between? What happens to that connection?
10:43
How do you have resiliency built in there?
10:47
How do you detect reroute without causing an issue
10:50
upstream to the training job that's going on?
10:54
So these are all and then this thermals are a crazy problem,
10:58
right? Like heating.
11:00
I remember reading one of the blogs from Meta again on how
11:04
they would optimize their GPU clusters and one thing they said
11:08
was what about updating? We have these 24,000 GPUs
11:13
basically at any given moment, we have to be updating the GPUs
11:17
constantly. And so making sure that you're
11:23
updating in the most efficient way possible.
11:26
And, and the people that are using those GPUs, if they had a
11:29
two week long job that they're running and now it's like, oh,
11:33
we need to update this. You have to figure out how
11:36
you're going to kick them off of those GPUs or just swap it out.
11:38
Like you said, under the covers, they don't know that anything
11:41
went differently, correct? It's just that you as that
11:46
engineer that's working on the clusters, you understand, OK,
11:50
we, we swap this out and now we're updating these GPU's.
11:53
Or it could be something as simple as maybe I'm swapping out
11:55
an SSD, popping out memory. It's not that the GPU's, but the
11:59
stuff around GPU's stuff can. Stuff happens.
12:03
Yeah, yeah. Getting back to the gotchas,
12:06
what are things that in your eyes, when you've been playing
12:10
with it, when you've been seeing it, it's like this is very
12:14
common, one that will get people into trouble.
12:17
Yeah. So there's steady state issues
12:19
at steady state. But let's talk about what
12:22
happens before that and I'll talk about what I've heard from
12:27
some Neo clouds when they service certain big customers.
12:33
And when some of them described how they onboard them, I
12:37
literally fell off the chair. It is good to know because this
12:40
is, this is a state-of-the-art in many new clouds, right?
12:46
And, and, and you can actually see this like if you go to some
12:50
new GPU cloud that exists and you say, hey, can I, why can't I
12:53
just simply sign up, login, put in my card and use it?
12:57
Why are you asking me to submit a form and you'll call me in a
13:00
day or two? And is it because you don't have
13:03
enough capacity? Remember the, the data center
13:06
provider here, they have a bunch of GPUs sitting in a, in a one
13:11
or two data centers. They, what we found is they
13:15
don't even have a good sense of inventory.
13:17
They don't even know what's out there really, right?
13:20
So these are the curve walls we guys have to encounter and say,
13:25
hey man, we can help you solve these problems.
13:27
When I say inventory, I'm not talking about just GPUs.
13:31
Which server is it on? What kind of memory does it
13:33
have? How is it connected?
13:34
How is it connected? Or is it some random one?
13:37
Yeah, What's it, Mac address? Yeah.
13:38
And we were surprised and things like firewall policies because
13:42
there's multiple themes and all of that, right?
13:44
These are all, many of them manage these in Google Sheets or
13:49
Excel spreadsheets, right. So that's the state of art at
13:52
many of these companies, right? So no wonder when you submit a
13:56
form and someone negotiates finite GPUs that you're buying
14:01
this take two weeks to do all of these things.
14:03
Actually provision you those GPUs, correct?
14:06
And ten teams have to do their stuff and update the Excel
14:09
spreadsheet. I mean, this is not good, right?
14:13
So what we have done in our platform is we said, hey, why
14:17
not solve this problem? What if there's a system, system
14:22
source of truth where you maintain this inventory where we
14:27
don't want to be simply a replacement for an Excel
14:29
spreadsheet. The inventory is important
14:32
because it can dynamically then carve out capacity from the
14:36
infrastructure and allocate it to a tenant.
14:40
And so that is now done right now.
14:43
The next thing is it's not just the the GPU allocation, you have
14:49
to also now create what is called as a tenant access
14:53
network, what the industry call as a tenant access network.
14:56
So what happens is imagine, imagine, I'll take an analogy.
15:01
You know, when you go to a hotel, there's a room, right?
15:04
Imagine the room can auto expand and shrink based on what you
15:07
need, right? So how do you automatically put
15:10
up the walls? How do you put the right things
15:13
in the room that you need? All that is done manually today
15:17
and what we do is automate that. If I say I need 100 GPUs, I do
15:22
the East West network, East West is the InfiniBand, the
15:25
north-south, because I need a storage network, a high speed
15:27
storage network where I store my models, my data, etcetera.
15:31
And then my East West network cannot be interrupted because if
15:35
it is, then a latency kills, right?
15:39
How do I then let people access that network and how do I manage
15:44
multiple tenants on my one single pool of infrastructure,
15:49
fully knowing tomorrow this university is going to come and
15:52
say, hey, I need 100 more GPUs so there has to be elastic.
15:56
I shrink and contract over time. This is table stakes for a GPU
16:00
cloud, right? But really hard issues like when
16:03
you say that, I think, Oh my gosh, that is brutal to be able
16:09
to be that elastic, but also have all of these things that
16:13
are in your mind with that inventory, if it was being done
16:16
on a Google Sheet, by the way, always a great business idea.
16:19
If you see somebody doing something on a Google Sheet, you
16:21
can probably think of like 10 companies that have come because
16:25
they saw someone doing something on a Google Sheet or just like
16:28
Excel and they thought, you know what, we could probably make a
16:31
product around that. Totally.
16:33
I mean like even even products like air table.
16:35
I was thinking of air table. Air table is a classic one where
16:39
it is just such a better experience but at the end of the
16:41
day it's a excel sheet. Glorified Excel sheet.
16:44
Yeah, exactly. So, so you look at the bottom of
16:48
the stack. This is kind of what we saw.
16:49
And then what happens? Your inventory also is changing.
16:52
So it's a dying. So it's a getting replaced.
16:54
You have new switches. Your data center is not static.
16:57
You're adding capacity, removing capacity, retiring GPU's, adding
17:00
new GPU's. How do you keep up with that?
17:02
Yeah, so stuff around you is changing.
17:04
It's like quicksand. Constantly.
17:06
Stuff on top is changing, so we have a middle layer that is
17:10
helping you make sense of what's happening on top.
17:14
What's happening below? I like this.
17:17
You're mentioning that on the bottom on that layer, it's
17:21
constantly changing because of whatever reason.
17:24
And then on the top, it's changing because of the
17:27
customers wanting more capacity or wanting less capacity,
17:30
correct? And so being able to be that in
17:34
a way the broker of the capacity, knowing what capacity
17:38
there is what's actually online, because the last thing you want
17:42
to do is give a customer something that's not on.
17:45
You say it's online, but they're trying to load up and it's like,
17:48
why is this not loading and something's broken here?
17:51
And. That would be such a bad
17:52
experience, right? Like, like, think about, we all
17:57
expect, like when you go to AWS, press a button and say, I want
18:00
this EC2 instance, it comes up like clockwork.
18:04
You don't have to make a support call.
18:06
That's the experience people expect.
18:09
Can you be that tall to have a shot in the market?
18:14
And that's a pretty hard mountain to climb.
18:17
The things that you get into as you're trying to provision GPUs
18:23
are all these hard pieces, the ways that when working with
18:29
GPU's you've seen them fail. I, I know that we, when you gave
18:33
the presentation at the conference the other day,
18:36
someone asked about the noisy neighbors situation.
18:38
Yes. And I, I wanted to get into that
18:40
with you because I think that's one thing that people who have
18:43
tried this at their, if they're at a company that has the luxury
18:47
of buying hardware or just being the one who's provisioning this,
18:53
they get into those situations. I remember 3-4 years ago when we
18:57
had the run AI team on here, they talked about how difficult
19:01
it was too. And how do you go about solving
19:05
for these noisy neighbor situations?
19:07
Maybe breakdown for myself even. Give me a reminder of what that
19:11
even means. Sure.
19:12
Absolutely. So this all kind of was what
19:17
forced us to get into the inventory management space,
19:19
right. So if you go talk to in NVIDIA
19:24
or AMD etcetera, right, the technologies for this exist,
19:29
right? You have ways by which you can
19:31
isolate like using EVPN technologies, we expand all
19:36
these stuff exist. I mean this is how data centers
19:38
work today, right? Orchestrating those dynamically,
19:43
making sure you can do this programmatically based on some
19:48
higher level ask like higher 100 GPUs.
19:51
Yesterday I was using 50 GPUs. Where am I going to allocate
19:55
them? How am I going to connect them
19:56
to the same VPC, essentially the same technologies like the
20:01
things that are familiar with like a VPC, like a VX LAN, an
20:06
EVPN? How do you do route linking to
20:09
my storage network? Right?
20:11
All these are known devils. This is orchestrating them
20:14
automatically and programmatically.
20:16
And then making sure that I didn't have any, you know,
20:19
misses in the process. But is this just like Terraform
20:22
that we're talking about? And it's not.
20:25
Terraform is a way to achieve that.
20:28
Really. So Terraform has no
20:30
understanding of the inventory, right?
20:31
It has to talk to something. Yeah, right.
20:34
So the interfaces we support, Terraform is one of the
20:36
interfaces we support because people want to automate their
20:39
way as they want. But Terraform itself has no
20:43
understanding of inventory, right?
20:45
It doesn't know the current state, like how many switches do
20:49
I have? So and then some of these
20:52
technologies that exist, some of them will support certain kinds
20:56
of automation, others will support other kinds of
20:58
automation. So you have to effectively
21:01
present a unified interface and say that I'm going to shield you
21:06
from the vagaries of 20 different languages.
21:09
You got to speak it. We become the universal
21:11
translator of sorts, A unified orchestrator and a translator to
21:15
talk to all these various technologies that exist.
21:18
And why do they exist? If I'm an enterprise, I can go
21:21
in and say I'm going to go with the gold standard because my
21:25
universe is small. I'm going to pick vendor ABC.
21:29
Or if I want to reduce that even further, I'll say, you know
21:31
what, I'll pay the extra money. I'll buy an NVIDIA DGX, which is
21:35
a closed system. I don't want to deal with that,
21:38
right? I want to avoid all these
21:40
problems. I get a box, plug it in, but
21:43
it's limited to 8 GPUs and if I have more money then I'll go buy
21:48
a super pod. But I'm paying through my nose
21:50
at that point of time because nothing is modular.
21:54
A GPU cloud cannot do that. They need modularity because
21:57
their demands are different. The margins are the nature.
22:02
They make money. So they'll have an Arista,
22:05
Juniper, Cisco for the network, Ethernet switches.
22:08
They'll have, you know, Mellanox for the InfiniBand.
22:12
They'll have 20 different storage vendors.
22:14
They're trying to, you know, save dollars here and there
22:18
because they can't pass this on to the user.
22:20
They have to remain competitive in the market and they have
22:23
maybe 12 months to actually make money, after which the next
22:28
generation GPU comes out and then this game over.
22:31
Yeah, then they have to make another investment to bring on
22:34
the new GPU that folks are going to be asking for.
22:37
So that variety is what causes them grief.
22:40
The variety gives them the margin.
22:43
But because there is no consistency across all of these
22:46
things, you need something that blew it all together, right?
22:52
And or like we all use an iPhone or a Mac or something like that,
22:56
man is impossible to change anything in those, right?
22:59
It's a black box wall garden. Or you can buy a wall garden
23:03
system, which not affordable for the GPU clouds.
23:06
So anyway, I guess we digress a little more than we thought, but
23:11
these are some of the challenges they face in which may not exist
23:15
for an enterprise per SE, because their scale is, let's
23:19
talk about scale, right? The typical GPU cloud we work
23:22
with has 5000 GPUs, 10,000 GPUs, multi generations of GPUs,
23:29
right? Sometimes spanning four or five
23:31
datacenters, sometimes not just datacenters.
23:33
It's Kolo here, Kolo there, stitched together with some
23:36
weird network, right? And they don't, they have data
23:40
center talent, they have no software talent, right?
23:44
Enterprise, very different, right?
23:45
They have some homogeneity. They have 64 GPUs, 100 GPUs.
23:52
Yeah, unless you're meta and you're 100 GPUs is great scale.
23:56
Yeah, exactly. And a lot more software shops
23:59
than hardware shops you would imagine.
24:01
Exactly. Now, even within that 64, let's
24:04
talk enterprise for a second, right?
24:06
The GPU cloud's different scale. The same problem also exists
24:10
from the enterprise, which is if I have, let's say I have only
24:14
100 GPUs, how do I slice that and give it to a data scientist
24:19
for X hours and reclaim it and give it to someone else and they
24:24
need it? That two weeks, the medic I was
24:25
talking about. Yeah.
24:26
How do I do that efficiently, right.
24:29
In the old days, you'd get on a queue like, you know, if you're
24:34
family with Slurm, you know, a lot of universities still do it.
24:38
This is a 20 year old HPC tech where you said, hey, I'm going
24:41
to submit a job. I'm going to describe what I
24:44
want to my Slurm job. It'll wait in line and Slurm
24:48
will say when that capacity is available, your job will run.
24:52
Now how do you achieve that in the modern containerized world?
24:56
That's the challenge. What's the replacement for
24:58
Slurm? Yeah, a lot of people still just
25:00
use Slurm because of that exact point Exactly.
25:03
And then you have others who say, all right, well, we're
25:05
going to go about it with Kubernetes and try and figure it
25:07
out that way. So and Slurm has its own
25:11
problems, like all the nodes have to look exactly the same,
25:14
all the GPUs have to look exactly the same.
25:17
If I have to update an OS when I have a customer who's doing
25:20
that, they took it offline for two weeks because they had to do
25:23
an OS update across their Slurm cluster.
25:26
That's what I was talking about with the with the meta paper
25:30
when they were talking about these rolling updates that they
25:32
have to do. That's, that can cripple you and
25:34
you don't even realize it. And until you're in that
25:36
situation where you go, oh, we'll just update real fast and
25:38
then your whole cluster is offline for two weeks.
25:41
And then you talk to the data scientist there, they say, what
25:43
am I going to do, Forced vacation?
25:47
Yeah, they get to hang out. Well, that's not bad.
25:49
It's not the worst thing in the world.
25:50
But yeah, I imagine they want to be making an impact.
25:53
They want to contribute. And then, yeah, forced vacation.
25:56
Yeah. And and it's frustrating for
25:58
some of them, right? Because, you know, again, I'm
25:59
friends with some of these people, at these customers, and
26:02
they come and say, look, I have a month to finish, finish my
26:06
analysis, submit a paper. Only then I get to go present at
26:11
this event, some big bioinformatics event or
26:14
something like that, right? They have their endings.
26:18
What do they do now? This thing got offline because
26:20
some security guy came and said he got a patch.
26:22
Yeah, some update. Yeah, that is a great one.
26:27
Yeah. So then we were talking about
26:28
scale and we were saying enterprise is 1 type of scale,
26:32
but then these neo clouds are a whole different level of scale.
26:37
So what are some of these problems that you'll run into
26:41
when you're at this type of scale?
26:42
Yeah, we talked about two things.
26:44
We talked about how do I onboard users?
26:48
How do I handle them with some elasticity because the demand
26:52
and require elasticity in an environment there stuff under me
26:58
is in flux, right? And how do you keep that going?
27:03
Whether you're an enterprise Oregon Neocloud, similar
27:05
problems, this scale is different.
27:07
And then the next problem people run into is you can't give the
27:13
end user a data scientist, a bare metal server and say OK,
27:17
bye bye. You know, what's that person
27:19
going to do, right? So the problem then becomes, how
27:23
do I like, what is the end user experience that an envelope or
27:28
a, you know, a data scientist expects?
27:30
What do they need right now? If I go to a cloud, what do I
27:34
get? Stage maker.
27:35
Yeah. And you click a button, you get
27:38
a Jupiter notebook. You click a button.
27:40
Somehow, magically, GPU show up. You train stuff.
27:43
Run some pipelines, you're good. Yeah, it is very managed in that
27:47
regard if you want it to. Be exactly.
27:49
Now how can I bring that experience in my in my data
27:53
center if I'm an enterprise? Because I may be regulated, I
27:57
may not be able to move all the data to McCloud.
28:00
You know, there are so many constraints that people have.
28:03
And at some point we should talk about data gravity because that
28:06
is 1 massive intractable challenge that a CEO
28:10
organization struggle with. And it's kind of outside our
28:14
scope, but it's fun to talk about.
28:16
Well, it is funny that you mentioned that because we've
28:18
been going around and I've been asking folks, hey, we are
28:20
putting together this GPU providers guide.
28:23
What are some things that you would want to see in it?
28:25
And during that process, I've been interviewing folks in the
28:29
community and saying, hey, you bought a bunch of GPUs.
28:32
You rented. Really, I think bought isn't the
28:35
wrong word for it. What when you rented these GPUs,
28:39
do you wish you would have known?
28:40
What do you wish you would have asked yourself about?
28:43
And one of them that came up was, you know, I didn't realize
28:46
how much I would have to pay an egress fees to get my data to
28:50
these GPUs. Exactly.
28:52
So that that is a definitely a challenge and I think let me
28:56
kind of double click on that a little bit because that's a fun
28:58
topic and I'm a technologist. I love these crazy problems.
29:04
Now that the problem there is you have this concept of cold
29:08
data, hot data. And what I mean by that is, you
29:13
know, let's say you have 1,000,000 photographs that
29:16
you've taken over 20 years. You don't touch them all at the
29:20
same time. In fact, your viewer where
29:22
you're viewing stuff, you're probably looking at a few of
29:24
them, right? And let's say it's indexed and
29:28
all of that and stored in some. So what would the provider do?
29:33
If I'm Apple or Google, I'm not going to put everything in hot
29:37
storage. You don't need that.
29:39
So I'm going to do some tricks there.
29:41
I'm going to move some to cold storage, cheaper storage, and
29:43
only keep certain things in memory.
29:45
That's how most of these systems work.
29:49
Now when you overlay AIML on top and let me take an example of
29:53
let's say I'm, I'm a clinical diagnosis company.
29:57
I have a new, I'm a data scientist looking at, hey, this
29:59
is new, new kind of treatment, but I want to compare against
30:04
patient data that is 20 year old.
30:07
Now, for me, cold data has no meaning because I had to compare
30:11
against all the things that exist and get my answer quickly.
30:16
If I'm going to wait for the data to come back from tape onto
30:20
hot storage, everything is slow. Yeah.
30:23
What do I do now? Now I'm talking petabytes of
30:26
data that I need instant access to.
30:28
And if you overlay that with the example I took, like that Slurm
30:32
system going offline, what if I have to move that system to the
30:35
cloud, like a Bush to the cloud or shift to the cloud?
30:39
How am I going to move my petabytes of data there?
30:42
Yeah, what do you do? That seems to be.
30:45
Open question. Yeah, yeah.
30:47
It's a question that is hard to solve and you have constraints
30:50
like the egress charges, etcetera that that people have
30:53
to deal with. Yeah, I guess at the end of the
30:55
day you can always figure out a way to make it work.
30:58
But does it work within your budget?
31:01
Yeah, exactly. See, when does this happen?
31:05
When you have things like I got to do patching or something
31:07
else. And now this is where Kubernetes
31:09
comes in handy because it's, it's kind of more flexible than
31:14
things like Slurm. It's, you know, and have to deal
31:19
with 20 year old tech, right? So people end up spinning up a
31:23
second parallel system close enough to the data.
31:28
The investment you had to make here is more hardware and then
31:32
you can deal with your upgrades and updates here on the system
31:36
that needs to be touched. And then you roll over slowly,
31:41
right? You don't have to do Big Bang
31:43
updates. So I've seen companies do that
31:46
where they instead of operating at 100% capacity, they shrink it
31:49
down to 80%. At least people are able to do
31:52
their jobs and slowly, gradually, they back away at
31:56
the. So at any given time you can
31:58
expect 20% to be offline. So really, you buy 100 million
32:03
worth of GPUs and you only use 80 million?
32:08
It's a practical approach. There's no other way out because
32:11
you're updating something all the time and then you want to
32:16
minimize the time it's offline, right?
32:20
Bring it down to hours if possible, right?
32:24
And some of this you can do if, if you do some intelligent
32:30
things like store all the updates, etc, locally, you're
32:34
not pulling down terabytes from the Internet to update.
32:38
So local cache systems, right? Like if it's a software update,
32:41
what do you want to download Ubuntu updates from the Internet
32:45
for every system, right? So people do these intelligent
32:48
things and we help with that to make sure that, first of all,
32:52
can I reduce that from 20% to 10% and I keep that really,
32:56
really, really quick so that I can sweep through my
32:59
infrastructure as quickly as I can, like patching, etcetera.
33:05
Yeah. So we also were mentioning
33:07
before I distracted you with the data gravity thing, how you
33:12
don't want to just have bare Meadow.
33:16
I guess for some Neo Clouds, that is one of their value
33:22
props. They say, look, we're as close
33:24
to the middle as you can get and some folks want that.
33:27
But for the most part, you want to give people something at
33:32
least the minimal resources so that they can do their jobs on
33:37
there if they want. Yeah, it's actually a great
33:39
point. So let me again use an analogy
33:42
here because it'll, it'll, I think, help explain the the
33:45
user's needs, right? Let's say I want to, I'm hosting
33:49
a party tonight and I'm going to give them pizzas.
33:53
I have two options, right? I can buy a ready made pizza
33:57
from the freezer at the local store.
34:00
All I have to do is reheat it and give it.
34:02
Or I can buy all the ingredients and make it myself.
34:06
And different users may fall on the 2 ends of the spectrum,
34:10
right? Nothing right and wrong.
34:12
It's this preference. Yeah, preference.
34:14
Similarly here, what is the catalog, what's the menu of
34:17
items that the Enterprise Oregon, the cloud provider has
34:22
to offer? Some will say, hey, just give me
34:25
bare metal, I'll do everything myself on top.
34:27
This is akin to someone saying just give me the tomatoes,
34:31
flour, I'll make my own pizza base, etcetera.
34:34
And they may have a genuine reason to do that.
34:37
Some may say that, hey, I don't need all that, just give me a
34:39
notebook. If you only sell the raw
34:42
material, I mean, you're not an interesting storefront, right?
34:47
If you sell only the ready made stuff, again, you're not an
34:49
interesting storefront from a user perspective.
34:51
So as a as an enterprise Oregon, a GPU cloud, you kind of want to
34:55
have both. Yeah.
34:56
And actually this brings up another point.
34:58
I guess the metaphor may fall down here, but a lot of these
35:03
folks that are these NEO clouds are now offering tokens.
35:08
They're saying don't even mess with the sage makers.
35:11
We've got the Bedrocks yes, of this and just tell us what model
35:15
you want and we'll give it to you in tokens.
35:17
Yes, that's how we're. Trying exactly.
35:19
It's actually fantastic you brought that up because the
35:22
future with Gen. AI probably might be a token
35:25
cloud, right? Where if in the modern Gen.
35:29
AI workloads that run, they consume tokens and they consume
35:37
a lot of tokens and people don't want to run a dedicated
35:41
inference endpoint. Like it costs a lot of money to
35:44
run Llama yourself, right? So then they'll end up saying,
35:47
look, I just want to write my app.
35:50
I want to write my Python code, give me an API endpoint and a
35:53
token and let me consume tokens and I'll pay you by the tokens.
35:59
I consume maybe a million tokens a day.
36:01
Now, why is it interesting for a for a GPU cloud?
36:04
A lot of them have they like it or not, they have spare capacity
36:08
lying around, right? How do they monetize it?
36:11
It's higher profit margins, right?
36:12
Exactly. Instead of it idling if if I can
36:16
be running a serverless inference and be somehow
36:21
monetizing the GPU, but I'm selling something different to
36:24
the user. The user here is expecting
36:26
tokens, so I'm selling them tokens, not GPU's, but I'm not
36:30
idling my GPU. 'S and can you also use the
36:34
second or third generation these older GPU's you now can get a
36:38
bit more juice from? Exactly.
36:40
So that's a beautiful point you made.
36:41
So like we talked about, there's a 12 to 18 month lifespan and
36:45
then people say, hey, I don't want to buy today.
36:48
Everyone wants an H100IN. In a few months probably it'll
36:52
be something else. Exactly right.
36:54
And the interest level on those today's generation systems are
36:59
going to go down. What do you do with that now?
37:01
You don't want to idle it, you don't want to throw it away.
37:04
It's a wasted investment. So now you can extend the life
37:07
of these systems. But exactly what he said, if I
37:10
can run the token cloud on that and make sure that I can deliver
37:14
them tokens, now I'm protecting my revenue stream.
37:18
I'm running a higher margin service.
37:21
And whether you're an enterprise Oregon, a GPU cloud, the same
37:25
problem, except in one case it's a, it's a CFO of the GPU cloud
37:30
worrying about margins in the enterprise is the CFO of the
37:35
enterprise talking to IT and saying, hey, what do you mean
37:39
you just bought, spent $5,000,000 on this H1 hundreds.
37:42
What do you mean it's not useful anymore?
37:44
Same problems. But yeah, token clouds is the is
37:47
where the market I believe will go.
37:49
The lot of the Gen. AI use cases because a lot of us
37:53
who've been in AML for a while maybe think about ML OPS.
37:58
We think you're building your own model from scratch.
38:01
You're dealing with the data, all those data pipelines.
38:04
It made me think that it's a very different persona or you're
38:10
almost attacking different folks who work on ML in different
38:14
ways, correct? I've heard a lot of people say
38:18
outsource everything you can when it comes to this.
38:20
If you can just get tokens, then start there because it makes
38:23
your lifeway easier. If you don't have to deal with
38:26
any of these GPU's and then set up the services on top of those,
38:29
then great. If you do, that's one type of
38:34
Persona and maybe you're setting up the platform on that.
38:39
You're the platform engineer that is 1 Persona that's
38:43
worrying about that reliability. But then if you're just building
38:47
the app, you're hitting that API and you're creating the product
38:51
that uses that service, that's a whole different persona.
38:54
Exactly. And there's various personas
38:57
that could be that consumer, yes.
39:00
And it's not just a developer. I think what we see is an
39:03
interesting pattern there. I think.
39:06
I think this token cloud will be an enabler for what I would call
39:09
as a citizen scientist. We talk about a data scientist.
39:12
We seeing a pattern there. The you know, maybe someone from
39:16
the CFO team who doesn't know how to write code.
39:19
What they're expecting is saying, hey, how do I unlock
39:22
value from my data? I have all these spreadsheets.
39:24
I don't know what to do with it. It can either sleep there in
39:27
storage or maybe I can unlock some value out of it.
39:32
If only I can train my LLM and LLM, whichever is suitable, that
39:39
can somehow understand this data and became a domain expert.
39:43
As an example, literally earlier today I was talking to one
39:47
company that said we have 20 years of investor relations
39:52
content. There's one guy in the company
39:56
who's been there with them for like 6-7 years who knows
39:59
everything if that person is on vacation.
40:02
Or gets it by bus. Exactly.
40:03
Big problems. No one else knows because these
40:06
documents are massive, right? But they're so proprietary to
40:09
the company, so they're basically saying, man, how can I
40:12
build a machine with this knowledge so we can ask
40:16
questions to it? And I think the default that
40:19
most people go to is they say, well, AWS has like the private
40:24
cloud and so we just do it on AWS in a bedrock type of way.
40:30
I'm not going to sit here and just talk a bunch of shit about
40:32
AWS, but some people run into allegedly problems with not
40:37
getting good service and paying a lot for it, correct?
40:41
They offer a good platform, but it's not for everybody, right?
40:47
There are many organizations and geographies where this is not
40:50
going to work. Like many of the partners and
40:55
customers we work with. They work with like a defense
40:58
entity and country A, country B. Then I'm going to go to the
41:01
public cloud. At least not in this
41:04
geopolitical climate, right? They're very sensitive now, like
41:08
if I'm a defense agency or if I'm doing something sensitive in
41:14
that country, I prefer to use a sovereign cloud that I have
41:18
control over, but I have guarantee my data will never
41:21
leak. So some of them feel that way,
41:23
right? Again, right or wrong, AWS is
41:27
not for everybody or any other public cloud.
41:30
And yeah, I mean, like, like for example, at at GTCI heard
41:35
something interesting. And since you're from Germany,
41:38
I'll talk about this. So this person came along and
41:41
said, hey, you know, we have some factories that have have
41:46
been shut down and what the government wants to do is to
41:49
repurpose them into AI data centers because they have power,
41:55
they have real estate. And apparently it's set up in a
41:59
way where you can set up racks and convert them into.
42:02
They've got cash to burn. Exactly.
42:04
I think people are thinking of very innovative ways to get
42:08
started. So I'm pretty excited about the
42:10
situation because if if access to AI infrastructure is
42:16
available to everybody, think about all the stuff is going to
42:20
unlock. I think we are at that
42:22
inflection point. Is it one year out, two years
42:25
out? I don't know.
42:25
But we are at a very interesting point in the market.
42:28
Yeah, And it is fascinating when we go back to that CFO who just
42:32
in the basic iteration of now I have my ChatGPT that I can talk
42:37
to and I don't have to worry about the data leaking to
42:39
whatever. And we've heard that for a
42:41
while, right? That's the whole thing, the
42:43
whole reason why open source folks are banging the drum.
42:46
But then if you take it a few steps further and you can say,
42:50
well, now teams can build services that will be useful for
42:56
our CFO products that are going to help make that persona's life
43:02
easier. And I think a lot of different
43:05
teams and product folks are doing that work right now inside
43:10
of the organization. They're going around and saying,
43:13
just show me your job, show me what you're doing.
43:15
Let's see if we can plug in some kind of AI service here,
43:20
correct? Correct.
43:21
To make that person become more productive, like the example I
43:25
was talking about the the CF OS team gathering all the data and
43:31
they don't have a developer or any for that matter.
43:34
Anyone who understands, you know, what is temperature is in
43:38
a token or, or how do I quantize a model?
43:41
I mean that all is like, you know, Greek and Latin for them,
43:44
right? They are finance people.
43:47
So how do I get this person to become more productive without
43:53
having a massive development team that has to do some
43:56
requirements gathering and? I've seen, it's funny you
43:58
mentioned that because I saw recently a team that has an
44:01
embedded like data scientist, ML persona on the finance side.
44:08
Yeah. So you've got that finance team
44:09
of 40 people and they have one embedded engineer in there who
44:14
is very deep in the AI world and is trying to help them
44:18
streamline their processes and make it more.
44:20
Productive, exactly. That's one approach, the
44:22
approach where I see the market shifting and this is something
44:25
that we guys are very interested in because we're getting dragged
44:27
into it from the spoken cloud thing, which is what I'd call as
44:32
zero code fine tuning, right? Because in the end it's this
44:38
workflows. So give me my data, select a
44:42
model, you can choose ABC, whatever you want, say OK,
44:48
somehow click a button called fine tune.
44:50
They don't need to know what it does.
44:51
Behind the scenes, just optimize.
44:53
Right. And then it fine tunes it.
44:55
And then it says, OK, here's your endpoint right now here's a
44:58
chat bot which is pointing at that endpoint.
45:02
Now test it. Yeah.
45:04
See if it's actually doing its job.
45:07
And they'll just iterate very quickly.
45:10
And these models are getting so intelligent that you can fine
45:14
tune them quickly and all the complexity that we guys deal
45:19
with can be abstracted away. Yeah, I like that zero code fine
45:23
tuning. I've heard it.
45:25
My friends at Prem are doing like autonomous fine tuning is
45:29
what they're calling it, but it's that same general idea.
45:31
And recently we had Tanmei on here and he was talking about
45:34
the benefits of fine tuning because, and I personally have
45:39
seen a lot of people dunking on fine tuning.
45:41
I probably have been guilty of this myself where it's like fine
45:45
tuning is a lot of work and it can potentially be for nothing
45:50
if you're not doing it right. You may get a worse result after
45:54
you've fine-tuned and within the base model.
45:56
And so you have to go into it with a certain understanding of
46:01
what you're getting yourself into.
46:02
I think we are at a point now where we can make this a 0 core
46:07
experience to a certain extent, right?
46:10
Like we can even go and say, hey, for the kind of data set
46:13
you have, maybe these models are the right kind of models to pick
46:18
the kind of budget you have. Maybe you want to pick this
46:21
because you only have access to XGPUS, right?
46:24
Or these kind of tokens like you think about like lawyers, if
46:27
they have to do an analysis of, you know, 50 years of cases, oh,
46:32
man, that's a lot of work. What if something can just
46:36
autonomously help you there and just give you an assist?
46:40
Fine tuning is a great tool for certain scenarios where the
46:43
domain set is kind of reasonably static.
46:46
It doesn't change all the time. I don't know what kind of shelf
46:50
life it has. There are people who say fine
46:53
tuning will take over the world. There are some people who say,
46:55
no, it won't. And I think the jury's out on
46:58
that. We'll see.
47:00
When you're doing fine tuning, you're consuming AI
47:02
infrastructure to go back to that store analogy that I
47:06
mentioned, right? You may be buying a ready made
47:08
pizza, you may be buying the ingredients, or you may just buy
47:13
tokens, right? Or you may say, you know what?
47:16
I don't care what tokens are, I just want a workflow that is
47:20
going to upload my data and fine tune in the process.
47:24
I may be using tokens and GPUs. I don't care as a user.
47:28
So that's the spectrum of things I think the market needs.
47:33
And if I'm a cloud provider or an enterprise, these are the
47:36
kind of services I need to offer my users, the spectrum.
47:40
Otherwise I'm an uninteresting storefront for the user.