Latest / Agentic Conversations (formally mlops.community) / Economics of Building Data Centers, GPU Clouds, Sovereign AI

Transcript

Economics of Building Data Centers, GPU Clouds, Sovereign AI · Agentic Conversations (formally mlops.community)

0:00
I got into building clouds inside those data centers, so I
0:03
probably built more clouds. And I care to admit to you, some
0:05
are great, some are awful. Yeah.
0:12
Deep conversations are coming. Beware.
0:16
So dude, you've been building data centers.
0:19
Huge investments, big time production.
0:24
I want to know everything about it.
0:25
OK, absolutely. So just just for your audience,
0:28
Craig Tavares, I'm President, COO, Buzz High Performance
0:32
Computing. We're a Canadian cloud service
0:35
provider. We we own data centers
0:38
facilities in both Sweden and Canada.
0:40
We operate GPU clouds at scale. We, we certainly cater to again,
0:46
a lot of the sovereign mandates that you see in a lot of those
0:48
nations, especially Canada being Canada owned and operated.
0:52
You know, we have a huge interest and obviously play
0:56
within the, the, the sovereign economy of, of supporting
0:59
sovereign mandates for AI. What that means is again, you,
1:02
you have to look at the full integrated stack.
1:04
So yeah, my background, I've been building telecom
1:07
infrastructure, you know, right out of the gate, right out of
1:10
school, putting fiber in the grounds for communications
1:14
networks, building telecom companies from the ground up
1:18
both nationally and then internationally after that.
1:22
And when you're in that infrastructure game and you're
1:24
you're building all these massive high bandwidth networks,
1:28
you quickly got into data centers, right?
1:30
So again, everybody knew that the network in the data center
1:33
went hand in hand. Like you can't have a data
1:35
center without a network. And networks led to a data
1:38
center somewhere because again, that's where the other
1:40
information sits. So that that got exacerbated
1:44
when, you know, super, super hyper skill when again, all the
1:48
AWS and Azures and Googles came into the market.
1:53
So I do have to roll back the clock for you a little bit.
1:55
Didn't tell you that continuum or just go back on that
1:57
continuum. When again, you know, we went
1:59
from being a data center provider and then I got into
2:02
building clouds inside those data centers.
2:04
So I probably built more clouds than I care to admit to you.
2:06
Some are great, some are awful. And then somewhere right at that
2:09
cusp when again, like the hyperscalers came in and took
2:12
over the world. So the, the relevant public
2:15
clouds became what we called, or we became what we know as
2:19
regional clouds. And the regional clouds, the non
2:23
Awss, the non Googles still had relevance because one, there
2:28
were, there was a big customer base that just wanted to get a
2:31
simple cloud to access. But more importantly, there was
2:34
a big customer base that wanted to know where the data was.
2:37
Right. So again, if I'm going to put my
2:38
data in a cloud again, I, I kind of wanted, you know, within
2:42
again, a, a domestic region. And it, it, it only is to
2:47
satisfy again, certain data residency policies that
2:50
Enterprise Oregon government might have.
2:52
So again, Fast forward, I did some time at some large Fortune
2:57
500 companies went back to venture, actually just venture
3:01
where we, we purchased power plants.
3:03
So we, we bought power plants, we operationalize them.
3:06
These are what we call combined cycle natural gas power plants.
3:09
And we did that in Canada. And then I, I built an HPC
3:13
facility actually directly connected to one of these power
3:15
plants. So what we know as completely
3:17
off grid, a data center powered by its own power source and, and
3:22
then, you know, off the fence, so that the, the power that I
3:25
was metering was going directly to my data center.
3:28
But again, it was my power generation that was supplying
3:30
electricity to that data center. And that that's going back a few
3:33
years. But now that's, that's what you
3:35
hear a lot of the big data center companies talking about.
3:38
They're like, Hey, I don't have access to energy where I want
3:41
it, when I want it at the price I want it at.
3:44
So, you know, how do I scale my data center by creating my own
3:50
power generation and, and allowing for infinite scale
3:54
based on, you know, me controlling all the elements
3:57
myself, right? Not dependent on the, the, the
4:00
local grid, not dependent on, you know, the, the region I'm
4:03
in, whether or not again, it has like, say, renewable energy
4:07
sources. So again, those are the things
4:09
that that keep coming up all the time.
4:11
Like you were saying earlier when we were talking, you know,
4:13
one what what is that? What is that basic building
4:16
block that you have to have first?
4:17
Well, yeah, it is power, right? That's where it starts then then
4:21
now I can look at the data center appetite in that market
4:24
and then start looking at the the data center strategy.
4:28
So how do you? How do you look at that?
4:29
Well, not that they now that with AI like the there's
4:33
infinite. Appetite, you don't have to look
4:35
too. Much it is like, yeah, because
4:37
training, when you're, when you're looking at model training
4:40
again, and it it, it really changed the design philosophy #1
4:45
and it changed localization or, or geographic relationship.
4:49
What I mean by that is, again, look at how we built data
4:53
centers around the world. We built a data center so that
4:56
the data center is closest to the user, right?
4:58
So again, I'm, I'm building in like key markets and they're
5:01
usually large population centers.
5:03
So as, as I chose a location for my new data center building, and
5:07
this is again, regional clouds or hyperscale clouds, again, I
5:11
wanted to be in key markets. And then if you look at the US,
5:14
we knew these as NFL cities. So if, if in in the US, if there
5:17
was a state with an NFL team, you could guarantee there was a
5:20
cloud provider in that very, very first, right?
5:23
And for Canada, it's if it has a hockey team.
5:26
Yes, actually, no, it, it is true.
5:28
Yeah, but we don't, we don't actually have a lot of hockey
5:30
teams, believe it or not, but but yeah, no, there, there are a
5:33
few key markets in Canada and you know, you, you could, you
5:36
could call, you could count them in one hand.
5:38
But Toronto being the big epicenter, Montreal being an
5:43
another massive market, Vancouver another market,
5:46
Alberta, Manitoba is coming up as a, as a very relevant market
5:50
as well too. So yeah, we we do see again the
5:52
growth in Canada. Quebec grew really fast for
5:57
another reason is because we had very cheap hydroelectric energy.
6:02
So one. So it enabled the build out.
6:05
It did, yeah. Because one there, there was a
6:07
lot of it. So when Quebec started investing
6:11
its power infrastructure years ago, it was to enable
6:15
industrialization. So they wanted the forestry guys
6:17
to come in, They wanted the mineral mining guys to come in.
6:20
So they built lots of hydroelectric energy because
6:22
those are industries that take a lot of electricity, use a lot of
6:25
electricity. It didn't really happen.
6:28
So it didn't actually happen. And again, you were talking
6:31
about 20 years ago when these investments and its
6:33
infrastructure development started happening.
6:35
But one thing that did happen is data centers saw the opportunity
6:40
and data centers came in really quickly.
6:42
So we we saw a lot of the consumption and all the, the
6:45
power demand really get eaten up fast.
6:48
And then the other thing that happened too was a lot of
6:53
battery storage companies that came in from battery
6:55
manufacturing perspective came into the market too.
6:58
So again, this this is more in the manufacturing side and to
7:01
build batteries it, it there is a huge energy demand as well
7:06
too. So electricity is a key element
7:09
in developing batteries. And if you're developing
7:12
batteries, you're doing it under a green flag, you're in mandate.
7:15
Therefore you don't want to be creating or, or developing
7:19
batteries with fossil fuels. So again, because Quebec was
7:23
hydroelectric, all the battery manufacturers really enjoyed
7:26
going to Quebec to to use again abundance of cheap energy to to
7:31
do what they could with renewable energy sources.
7:34
So I'd like that you are basically going full stack on
7:38
the building out of the data center saying we're going to
7:40
produce our own energy, We're going to make sure that we have
7:44
the capabilities that we need. How are you producing that
7:48
energy besides the hydroelectric?
7:49
So today, today at Buzz, we don't actually build and produce
7:53
any energy ourselves. What we have done is, you know,
7:57
this, this comes from our parent company, High Digital.
8:01
We found locations around the world that has that that
8:05
situation I'm explaining to where there's an abundance of
8:07
hydroelectric energy. One, we wanted to really stay
8:12
and, and remain a, you know, a sustainable company where again
8:15
we're using a, a renewable energy sources and in this case
8:18
again, hydroelectric or geothermal.
8:20
So Sweden was again a, a big anchor site for us.
8:23
It was a it was a primary location for us and, and where
8:26
we built some our first facilities so that you know,
8:28
again we're using 100% green energy there.
8:31
Again, Quebec again another obviously a home base for us.
8:34
And then you know, a lot of our GPU clusters say that that's we
8:38
offer to the market are are based in Quebec.
8:40
And you know, my parent company High Digital just made a large
8:44
acquisition in Paraguay where we purchased a a very large piece
8:49
of land facility connected to one of the largest dams in the
8:53
Western Hemisphere, so 100% hydroelectric vien.
8:56
It feels like if you build it, they will come.
8:59
Right now, yeah, I think you're planning a road map like you're
9:01
you're creating that energy road map so that again, as as you
9:05
build out that data center infrastructure, again, you have
9:07
the energy to back it up and energy to scale with you.
9:09
So again, for, for what we need to do and cater to the market
9:13
that we cater to, we're building fully integrated data centers,
9:16
which means that you know, right now the the locations and the,
9:23
the axis energy and the hydro energy that I have, it actually
9:27
caters to demand and for the next couple years will cater to
9:30
whatever I need to do from a GPU cloud perspective.
9:32
So data centers, you know, come in different shapes and sizes,
9:38
right. Again, the traditional data
9:40
center that I was talking about earlier, you know, we were
9:44
building really sub 10 kilowatt racks.
9:46
So I could, you know, I could put in any single rack, a bunch
9:49
of servers and I could see them up to 10 kilowatts of energy in
9:51
that rack. And then I think, you know, if
9:54
if you sit any of infrastructure talks here at the conference,
9:57
you know, you'll, you'll hear that that shift again where
10:00
again quickly people had to go from that 10 kilowatt rack or
10:03
sub 10 kilowatt rack to a 40 kilowatt rack when the the H
10:06
Series GP is released, right? So he just went kind of anywhere
10:11
from like 5:00 to 6:00 to even 8X depending what your, your
10:15
power consumption that rack was overnight.
10:17
And and then you quickly went to an 80 kilowatt rack really, you
10:22
know, like a three to six month period.
10:24
And now what we're planning for IS130135 kilowatt racks with the
10:29
GB 2 hundreds. With no view of slowing down.
10:33
At all. Yeah, I I think again, data
10:36
center design and data center construction will naturally
10:40
limit the the absorption in the market.
10:44
You know if you talk to Jensen, he's already preempting.
10:46
Hey guys, start thinking about 400 kilowatt racks.
10:50
Yeah. And you you get into other
10:51
issues with that one. Yes, again, power density is is
10:55
a huge problem. You probably have to get out of
10:58
this rack mindset. So a data center will not be the
11:01
same data center that you remember it like, you know, a
11:03
few years ago. And then the heat dissipation is
11:06
a real issue. That's a real thing.
11:08
And that's why we had to shift from air cooled to liquid or
11:12
water cooled, because you can't get the same heat dissipation or
11:16
the heat, the same thermal transfer through air as you
11:19
would with liquid. So again, you're, you're trying
11:22
to create that, that thermal efficiency and removing heat off
11:26
those chips as fast as humanly possible and at scale.
11:31
So yeah, it, it, it's not, again, it's not a, it's not an
11:33
easy engineering feat to, to again, design data centers, but
11:37
also design data centers. So they're modular and future
11:40
proof, right? Because again, I'm trying to
11:42
plan for what that rack next rack density might be.
11:45
And they're just getting more and more dense and more power
11:47
hungry. Do you see a world where there's
11:49
400 kilowatt racks? Yeah.
11:52
And how does that look? How do you make that transition,
11:54
especially when you're trying to plan forward proof and you're
11:57
like, dude, I just got to 130, give me a little bit of a rest.
12:00
So I we have ongoing joke with our engineers is don't screw
12:05
anything down to the ground so. Put it all on wheels.
12:08
You can just wheel them in and out.
12:10
That's right. So yeah, we're and and if you
12:12
think about this for a second, like the the size pipes we're
12:16
putting to data Centers for the plumbing just to move again
12:19
liquid fast enough at volume through these servers.
12:23
I mean it, it's already massive. So yeah, if you have to now
12:27
again, prepare for the future and say, OK, I need the right
12:30
plumbing infrastructure in place, I need the right
12:31
electrical infrastructure in place.
12:33
These things are just going to get bigger and bigger.
12:34
So yeah, it's not like I don't think anyone has actually solved
12:38
for all those answers except like I said, like don't don't
12:41
get used to 1 standard. That's why I'm saying don't
12:43
screw anything down to the ground.
12:45
Like you could change it next year again.
12:48
I remember that. I made a joke.
12:49
Oh, yeah, you're in Canada and Sweden because it's cold there,
12:52
right? And you're like, well, that's
12:54
kind of true, actually. It helps a lot, yeah.
12:57
So the, the metric that we use to determine your data center
13:03
efficiency is something called PUE.
13:04
So it's like your power utilization efficiency metric.
13:07
And in most data centers, especially cool climates, we try
13:11
to leverage what's known as free air cooling, right.
13:13
So I want to use as much as the ambient air when it's cool,
13:16
bring it into the data center, use it instead of firing up my
13:19
chillers and, and cool that water loop, like if it's liquid
13:23
or if it's just air that I'm putting to the data center,
13:25
right? Take the cold air from outside
13:26
and, and use it. And yeah, in in climates like
13:29
the, you know, we have in, in northern Canada and if we have
13:32
northern Sweden again, yeah, you, you, you want to leverage
13:35
that cold air again for 80% of the year.
13:39
Again, you have cold air, why not use it?
13:41
So then that way I drive down my P UE and you know, on average
13:46
throughout the entire year, I have what we know is like a, a
13:49
1.3 P UE. So for every one kilowatt of
13:54
usable energy or capacity to have my server, you know I'm I'm
13:59
using about 30% more to cool the environments and and manage
14:03
environmentals within the data center.
14:06
I hear a lot of people talking about building data centers in
14:08
the desert in the Middle East. Yeah.
14:10
Does that just not make sense? Well, you it you can do it, you
14:15
just wouldn't use them much for your cooling.
14:17
So there are other cooling techniques that you would
14:19
incorporate, such as an evap cooling.
14:22
Yeah, definitely. You might have to fire up the
14:25
chillers more often, but they're innovating.
14:27
I mean, even in climates like that, they're always looking at
14:29
innovation techniques to, again, keep efficiencies really low.
14:32
And one thing that we see work really well around, you know,
14:36
what we know is district energy plants is 1.
14:41
If, if I do generate heat, I want to reuse it somewhere else.
14:43
So the other thing that we do when we look at when we design
14:45
data centers, especially if they're industrial parks, yeah.
14:48
Can I take that heat waste and then hey give it to my neighbor
14:51
if they need heat? For a solar.
14:52
Yeah, or, or anything like that. Yeah, I figured.
14:56
Yeah, absolutely. So you, you know, you whatever,
15:00
whatever you can use that heat waste for.
15:01
Yeah. Go, go create again a district
15:04
energy system with, with neighboring businesses to, to
15:07
reuse that heat or another idea. And, and you know, we, we've did
15:12
this, we've done this historically.
15:13
Like if you're, if you're, say, near a large body of water, it
15:17
could be a lake, it could be the ocean.
15:20
And you have the ability to to actually leverage deep lake
15:24
cooling or, or deep water cooling.
15:26
What you'll do is take again, cold water in from that that
15:29
water source and then run it through your data center.
15:32
So again, there's another district energy type technique.
15:35
We do it in Toronto quite a bit like the whole all of downtown
15:38
Toronto or most of downtown Toronto's on this district
15:40
energy system where we take all the lake water, pump it through
15:44
pipes and all the commercial buildings.
15:46
And, and that's how we create the cooling for the buildings.
15:49
Yeah, same concept for a data center, just again, more dense,
15:52
more compact way to do it. So there's.
15:55
A lot of different chip companies here at the Rays
15:59
Summit. I talked to you about this and
16:01
you're like, yeah, they're kind of a dime a dozen.
16:03
They have their different perks. But you are standardizing on
16:08
NVIDIA, right? You're not doing any other kind
16:10
of chips you're not really interested in.
16:13
It's true. Yeah.
16:13
We, we have, we have kind of strong partner in NVIDIA and you
16:18
know that we, we also get a lot of support from NVIDIA too.
16:21
So they, you know, they're, they're really big on supporting
16:23
their NVIDIA cloud partners. We're, we're one of them.
16:25
So, you know, we're very, very fond of that partnership.
16:30
You know, certainly again, when you look at the NVIDIA team, you
16:33
know, they're, they're, they're pioneering and, and carving
16:36
through, again, new technology fees, right?
16:38
They, they have this innovation engine that's always going.
16:41
And the thing the thing I love about NVIDIA is they're not
16:44
afraid also to experiment and get people to come and
16:47
collaborate and experiment with them in the market.
16:49
Are they the ones coming and helping you with the cooling and
16:52
the design? They do.
16:53
They have a very, they have a, they have a great team, team
16:56
that again collaborate to say, hey, listen, we want to build a
16:59
standard. And the reason why they want to
17:01
build a standard is they want to create a consistent experience
17:03
for all the customers. So again, if you're going to
17:06
spend a quarter $1,000,000 a server, you as as a consumer or
17:09
me as a, a service provider and trying to monetize that,
17:12
definitely, you know, you want to deliver the best experience
17:16
to your customer possible, which means performance reliability.
17:21
And that reliability is really around uptime because again,
17:24
I'm, I'm spending lots of dollars to build my a, a
17:26
project. We'll listen it better, you
17:28
know, the hardware better work. So yeah, they, they do have a
17:31
strong influence. They, they bring in, as you
17:34
know, reference architectures at the NCPS use and and listen
17:38
again, standardization is not a bad thing because that
17:43
standardization can also be updated when you find new
17:45
innovation techniques to make something better.
17:48
So it's not to say that you standardized to lock into a
17:50
rigid framework. It's to say that, hey, once we
17:53
find a good recipe, let's all, let's all follow the process.
17:56
But when we see process improvements for the next
17:59
iteration, we'll incorporate those.
18:01
Do they also let you know what's coming down the pipe and that's
18:04
where you can recognize that in the next three, 610 months we
18:09
should also be starting to prepare for XYZ?
18:13
Yeah, that, that and and it, it, it from 2 angles. 1 is the
18:17
hardware perspective, right? Yeah.
18:19
How do I have to plan my data center design around that new
18:23
technology coming to future is telling you again, you're going
18:25
more dense. And then the second part is
18:29
software, right. So again, you know, I want to
18:31
say that NVIDIA equally invests as much time and hardware in
18:35
their software stack. So yeah, we're we get we
18:38
definitely get early preview of all these things.
18:40
And then, you know, again, the goal is like, yeah, how do I
18:43
create like the, you know, the the best AI platform and
18:47
democratize that for easy access to everybody that's innovating
18:50
because we want to accelerate the speed of innovation in the
18:52
market right now. That's the one thing that we
18:54
don't want to do is slow down anybody that that again, it's
18:57
building the AI project and they themselves have to get to market
19:00
really quick because they got to see an ROI and they got to
19:02
monetize it themselves. Yeah, this probably a good point
19:05
to talk about the hardware, software bottlenecks that you
19:08
were just talking about on the panel.
19:10
What are some key points that you've been seeing out there?
19:14
On the hardware side, you know, this was more about keeping up
19:19
with long lead times and supply chains.
19:22
So, you know, in some areas and some businesses, you know, you
19:27
got guys that are GPU rich, that means they have a like Open
19:30
Access GPU's, huge budgets, a massive amount of capital.
19:34
So you see a lot of wastage, you see a lot of unutilized GPUs.
19:38
But then on the other spectrum, you do have a big part of the
19:41
market. And again, you see this with
19:42
research, as you see this with startups that don't have big
19:45
budgets. So again, they're they're a
19:47
little bit GPU poor. I want more and more GPUs and
19:49
I'd love to innovate faster, but the GPU is expensive.
19:52
It's not cheap. So what we do on the hardware
19:55
side again is we have a planning sort of a, a consumer one from a
19:59
data center infrastructure perspective.
20:01
That's probably the longest lead item because it takes a long
20:04
time to build a data center. We're we're talking 12 to 18
20:06
months in some cases, right? So I have to have that data
20:09
center pipelining capacity ready to even deploy or install a GPU.
20:13
And then obviously the GPU itself, again, you have to, you
20:16
have to alignment with again, your OEMs, all the server
20:18
vendors that, that again are packaging those NVIDIA GPUs and
20:22
then you, when you install them. And you know, we've got really
20:25
good at this it, it used to take a long time to turn up these
20:28
clusters. They're highly complex.
20:30
It's like wiring a brain and making it work with 1000 network
20:34
connections and you know, again, all the soft just.
20:36
One plug is not in the right place.
20:39
Yeah, no, you've got all these errors and problems.
20:40
Yeah, man. It sucks.
20:42
Sounds like a nightmare. No, it sucks, right?
20:44
So, so again, yeah, but we have, you know, we've got a model down
20:47
where again, we can deploy really quick.
20:49
But when I say deploy quick, we're still talking about weeks
20:51
sometimes, right? So again, install the racks
20:55
connected from a network perspective, then Commission the
20:58
servers and then pressure test them, right.
21:00
So again, again, that that's making sure I get all the right
21:02
firmers in there, all the right drivers are in there and then I
21:06
got to get the OS and patch properly and then I go back to
21:09
test it. So again, yeah, that takes time.
21:11
It definitely takes time. And then I, I, you know when I
21:13
have a clean system that I could you, I put in the market and
21:16
then start to monetize it. And then you throw the software
21:19
on top of that. Yeah.
21:20
And that that's the other part of the bottleneck you're talking
21:22
about. So on the software side now as
21:25
a, as a service provider, what I want to do is offer some level
21:29
of orchestration to my customer too, to say, hey, number one,
21:33
let me make sure again, I, I, you know, I put you on the, the
21:36
right platform for the right reason.
21:39
And whether you give a customer bare metal or whether again,
21:42
I'm, you know, I'm giving him Kubernetes to consume or maybe a
21:45
Slurm it it is around again, me optimizing that platform for the
21:50
customers use and developers again, you you you look at
21:54
historically, developers love bare metal.
21:57
The problem with that is, again, you have a lot of it is, yeah,
22:00
like and you see start, you see wasted start to come out too,
22:03
right? Because, you know, people start
22:06
to treat servers like pets instead of cattle and and then
22:11
that's. The whole dev OPS movement it
22:12
is, yeah. Around that, no, it came back,
22:15
it came back again, right? People are hoarding GP us,
22:18
right. People are actually holding on
22:20
GP us as they can because again again, you you see this this
22:23
cycle and even the hyper scalers where it it's actually hard to
22:28
get a GPU sometimes, right? And then you know, unless you're
22:31
willing to lock into a long term contract and people lock into
22:34
long term contracts. Why?
22:36
Because when they need the GPU, they want to make sure they have
22:38
it. So anyways, it, it's a, it's a
22:41
behavior that we see in market all the time when again, there,
22:43
there's a resource that that's highly desirable and when you
22:47
need it, you need it, right. So the, the software stack on
22:51
this side is one. Yeah, we do see optimizations at
22:54
least from a model perspective. So you look at, you know, you
22:58
look at compression, right? Really, really simple things
23:01
that, that everybody from a model billing perspective always
23:04
tries to improve. And, and then you, you go down
23:08
the stack to orchestration. That's where again, as a search
23:11
provider now I want to schedule workloads and put them in the
23:14
most efficient places and to make sure I'm creating maximum
23:17
usage out of my cluster. Because why if I have wastage in
23:20
my cluster, well, I'm not monetizing and making money.
23:22
I need to to pay back the investment.
23:24
I was talking to Paul yesterday and he was mentioning how for
23:27
him he's OK paying a premium for a more managed service on this
23:32
because it's so hard to get talent that really understand
23:36
and how the hell these things work.
23:38
No, you hit, you hit the nail on the head.
23:41
And you know, when you go to enterprise, you can't go and sit
23:45
down with the company and say, oh, how many GPUs do you need?
23:50
You can't have that conversation.
23:51
Nobody is going to know how the hell that right number.
23:54
Yeah. And no one's no, no one's going
23:55
to know what to do with it either afterwards.
23:57
So again, yeah, that that's where again, I think the token
24:00
economy is starting to become or, you know, more and more real
24:02
thing. And especially as again, as we
24:04
start to shift away from guys that are just doing foundation
24:07
models and, you know, really, again, still going back to tune
24:10
models because again, you get into a lot of organizations
24:13
where again, they're just taking proprietary data and they want a
24:16
closed environment. So they want on on Prem
24:18
environments. So.
24:19
Yeah, you will still get a lot of tuning happening, but then
24:22
again, you quickly shift to that inference side of the equation.
24:26
And then, yeah, now you're abstracting the hardware and
24:28
which is good that that is actually the right thing to do.
24:31
Because like I said, back to the the bare metal conversation that
24:34
we're having and trying to sell GPU hours or trying to sell bare
24:38
metal, you do see a lot of wastage there, right?
24:41
You can't, you can't maximize efficiency very easily because
24:44
you're leaving it to an individual or someone else and
24:46
their behaviors to maximize that hardware themselves and in multi
24:50
tenant environments. Yes, as a service provider, and
24:53
it may sound counterintuitive that I want people to save money
24:56
because if people are spending money, well, that means I make
24:58
more money. It's not true.
25:00
I want people to have efficient usage because again, it extends
25:03
the life of the project. I want to add value as a service
25:06
provider, so my service becomes more sticky as well.
25:08
So anyways that it, it's, it's actually great again for
25:13
innovation. It's great for the industry and
25:15
then me, it's great for my brand because if I could send you a
25:18
recommendation and say, Hey, listen, look at, you know, look
25:21
at my tooling. Here's where you have wastage,
25:24
here's where you can optimize or here's where you could be on a,
25:27
a cheaper GPU for the workflow that you're doing.
25:30
I want to provide information to to my customer.
25:32
Are you also renting out GPUs to some of these neo clouds that
25:36
are just doing inference like the models on the base 10s or do
25:41
they get enough in other places? No, though we do have a pretty
25:45
broad customer base and yeah, a a lot of them are inference as a
25:49
service providers. So yeah, we do that.
25:51
And again those, those guys are great too, again because they're
25:54
downstream customers are more interested in OK, how many
25:56
tokens am I using rather than how many GPGPU hours do I need,
26:00
right. So that you know, again, again,
26:03
you do see again a, a huge shift in the market so that, but you
26:07
still need a metric. That's why the token is
26:09
important. However, again, like I said, the
26:12
GPU hour is just really, really inefficient and it was a little
26:15
bit more hard to place in terms of me trying to quantify again
26:21
how many GPS I need for a specific model or specific
26:25
project that it that I needed to deliver.
26:27
Well, I know some friends that bought some GPUs, and one of
26:31
their main decision criteria was the level of support that they
26:36
were going to get, because they knew it was going to be so hard
26:38
to just go in there and figure out how to make this work with
26:41
their project. And there was going to be a lot
26:44
of learning for both parties, not just the team that was
26:48
buying the GPUs, but also the folks that were renting the
26:52
GPUs. There was a lot of times where
26:54
they were encountering things for the first time.
26:56
And it's like, all right, well, let's troubleshoot this
26:59
together, I guess. So, so we still see that even as
27:02
a service provider and operator, like again, the new, a new
27:05
firmware from an OEM comes out and it changes everything.
27:08
Then we got to rollback firmwares because I'm not
27:10
getting the bandwidth that I was supposed to get.
27:12
Or again, like I said, you know, you, you got to install patches
27:15
or you can install all you got to, you install packages,
27:18
they're in the right order just to get again, the throughput you
27:20
need. Because this, this is around
27:22
moving data too, if you think about it within a cluster, like,
27:25
you know, the network has to be operating at full bandwidth
27:28
efficiency, your memory has to be at full bandwidth efficiency,
27:32
right? And you know, this is where NV
27:34
link came in. Because again, yeah, you want
27:36
the cluster to act really as a single server almost where again
27:40
that GPU parallel processing again, they could all talk to
27:43
each other at again the maximum efficiency.
27:45
So you go back to what we know is primitives.
27:48
Primitives again are your, your CPU, your memory, your disk and
27:52
your network, right. And then you need that, that
27:55
high throughput across all those elements.
27:58
So again, you know, there there is one vendor that we use on the
28:01
storage side of vast data. So again, fast has been
28:05
extremely, extremely powerful for us because again, you know,
28:10
you don't see bottlenecks to storage and it's network
28:12
storage. And then our orchestration like,
28:15
you know, we, we partnered with folks like Raffe who again
28:18
create massive orchestration automation for us, you know,
28:22
even at the Kubernetes level of creating dev pods, for example.
28:25
And you know, it it again, when you talk about customer
28:28
experience, again, this is where again, we want to make our
28:30
customers, you know, easy to consume or, or provide the
28:35
infrastructure or customers easy to consume or they hit that easy
28:37
button so that they're not think like you said, they're not
28:40
thinking about it too much. Yeah, I remember we had Bohan on
28:43
from Raffe and he was talking about how they're this middle
28:48
layer and it's constantly like the ground is shifting
28:51
underneath them because there's not a clear thing on how many
28:55
GPU's are actually being utilized, how much of the GPU's
28:58
are being utilized. If there's rolling updates
29:00
happening, which ones are we taking offline, putting online?
29:03
So underneath them nothing is stable and then above them, the
29:07
demand isn't stable either. It's like I want 50 GPUs right
29:10
now. I want 50,000 GPUs right now.
29:12
And so they're constantly trying to play this game of like cat
29:15
and mouse on all right, well, we can connect these here.
29:18
And it reminds me of my daughter's like little playbook
29:21
where you draw the line of this is a mouse.
29:25
And so then mouse word goes there with the picture and then
29:28
you draw the line here and it's like this is a cat and cat goes
29:31
there, but you're just doing it at a ginormous scale with a a
29:34
lot of money on the line. I, I think the best story I can
29:37
give you is around, you know, historical technology
29:41
evolutions. And it, I had, you know, the
29:44
opportunity to listen to Geoffrey Hinton in Toronto not
29:48
too long ago. And he, he was asked the same
29:50
question. Like he, he, you know, they,
29:51
they kind of asked, hey, what was the efficiency gainer like?
29:55
What was the thing that allowed us to get to where we are right
29:57
now? And I was surprised and not so
30:00
surprised by his answer, but he actually said it was, you know,
30:03
the progression and evolution in GPU's.
30:05
So it was the advancement performance of the hardware, you
30:09
know, because when you look at efficiencies and you look at
30:12
like, you know, evolution, you see it across the entire
30:15
spectrum, hardware and software. So, you know, based on what
30:18
you're saying, even with Raffate, like you, yeah, again,
30:21
you do need the the software optimizations at different
30:23
levels, whether again, it's in the model itself, whether it's
30:27
the orchestration layers or just be behavior.
30:30
Like how do you create more education for people around
30:33
their behaviors around usage? But the the example that I
30:37
wanted to draw to is like, like, you know, you think about
30:40
autonomous driving as a technology.
30:42
It's it it it's a solution, it's an outcome.
30:45
But the technologies behind it were two fold.
30:47
One is yet you have that AI right, AI, you know, making
30:51
decision that this is almost like real time inference by the
30:54
way, too. But here's a big but you
30:57
probably wouldn't have got it and Jeffrey engines right
30:59
without the GPU. Another another example of that
31:03
is OCR. So it's optical development in
31:07
the camera, right? So if you think about a self
31:09
driving car, well, guess what, if, if that camera didn't
31:12
involve to where it is today, you wouldn't have self driving
31:14
cars either. So it is a it's like that
31:16
convergence of all technologies evolving almost at the same pace
31:20
and then reducing or lowering the barrier of entry for for
31:24
that innovation to start. And economics has a lot to do
31:27
with it too, right. So is the GPU at a price now
31:30
that I could offer to the market that the market could finance or
31:36
manage or fund to go do the things that we do today?
31:41
The GPU played a big role in that because of their training.
31:45
The camera played a big role in that.
31:47
You have this intersection and this convergence of all these
31:50
different pieces. And so then when you look at it
31:53
and you extrapolate it out to the companies and the
31:56
businesses, now when you're talking about the economics, it
32:00
is clear that companies need to be able to say this makes sense
32:07
economically for us to spend a lot of money on whatever it is
32:11
that we're doing. And so that conversation is now
32:14
being had more and more. Are we spending money on GP US
32:19
just to do AI or do we start to see that ROI from it?
32:25
And I think more and more people are quantifying what is the lift
32:31
that AI is giving us. We've moved out of that POC
32:33
phase and now we're not willing to burn a lot of money on the GP
32:39
US just because we want to be AI literate.
32:42
Yeah. And so it's very cool to see.
32:46
No, no, no again. Yeah, you struck 1/4 to me on
32:49
that one too. And, and more so because
32:51
obviously Buzz being a, a GPU and AI centric cloud provider
32:58
means that again, yeah, we have purpose builds tooling, we have
33:01
purpose builds software in our cloud for the development of AI.
33:06
But what we've done too is we've, we've offered a product
33:09
in the market that's a fraction of the cost of like a hyper
33:12
scale type offering. So again, if you compare us to
33:16
like the Awss and Googles and Microsoft of the world, yeah,
33:19
we, we, we are much, much cheaper to go get that AI
33:23
project done. We're quicker and more simpler
33:25
to use, and it allows you to fail fast, right?
33:30
So when you're in a platform like ours, you can.
33:32
You could throw pizza dough on the ceiling and see if it works.
33:35
Again, you're not, you're not running too high.
33:37
I have a budget and because of the concept around cloud, well,
33:41
guess what? I could hit the reset button and
33:42
start over again. Yeah, if you were to go straight
33:45
to, I'm going to go buy hardware for my own data center, try to
33:48
figure out how to use the GPU, figure out if I got the right
33:50
optimization techniques in there, figure out if I brought
33:52
the right scale. That that's a dangerous venture.
33:55
Yeah, that that's a really dangerous investment.
33:58
So it's really good to at least start in cloud, get that POV
34:00
done, see if you could build an ROI really quick.
34:03
And like I said, you could fail fast if, if the pizza dough
34:05
doesn't stick, well, hey, the pizza doesn't stick, right?
34:08
So, and then, yeah, once you have that, you may stay in cloud
34:10
or you may, and we've seen this a lot.
34:12
We, we may create hybrid environments where, yeah, now I
34:15
do edge into my own on Prem data center.
34:17
I want to have my own dedicated stack and economics may work
34:21
better in that case because I could drive high utilization on
34:23
that hardware that I own. And then I use cloud for
34:26
bursting and I use cloud for different projects, whether
34:28
again, it's maybe just my inference platform that I want
34:31
running cloud and I'll keep training on Prem or vice versa.
34:34
So yeah, I, I think hybrid is a great future for everybody,
34:38
right? Being able to figure out again
34:40
where the dollar should go based on how much hardware I own
34:42
versus how much hardware I rent. The hybrid narrative for the
34:48
cloud was always something that people like to have, but I don't
34:54
know if it was ever a really strong need except for a few
34:59
companies that definitely needed to do it for one reason or
35:02
another. But at the end of the day, folks
35:06
like to talk about it more than they like to actually do it.
35:08
But here I think there's a it flips it on its head because of
35:11
what you're saying where you need it, whether you're trying
35:17
to get more access to GPU's and you can't get it or you're just
35:21
seeing that unit economics work better when I do certain things
35:25
on my own or with this cloud. And this cloud provides this
35:29
type of service. So it's better for these use
35:30
cases. And you have this almost like
35:33
new paradigm that's emerging where hybrid is very common
35:38
design. Yeah.
35:40
And but I want to say one thing. Hybrid's not easy to pull off.
35:44
Yeah, So what what we've done which.
35:46
Is why I think traditionally people weren't doing it.
35:48
They were talking about it and it's like, yeah, we're going to
35:50
go hybrid or we we like hybrid. And I remember somebody telling
35:53
me that the CEO of Databricks loved it when people would talk
35:57
about hybrid because they were like, yeah, everybody loves that
36:00
idea. And then they end up using US
36:04
and. Execution failed.
36:06
Yeah, it's like it's a great in theory, but not in practice.
36:10
So, so one thing I could tell you is that again, when you look
36:15
at a lot of the AI projects today, we see a lot of data
36:19
movement. And that data movement happens
36:22
because, you know, I have my all my applications in hyper scale
36:27
cloud right now. All the data sitting there, you
36:30
know, could be an S3, but I need a specialized platform to do AI,
36:34
you know, I don't want to pay the hyperscaler price to do it
36:38
in their cloud. So now you see again, a lot of
36:41
data movement happening from hyperscaler to a platform like
36:44
Buzz. And then, hey, I may train
36:47
something, I may tune something, I may, I may run an inference
36:50
endpoint. But what I don't want to do is
36:52
have to move the data back in. So I don't want to create a
36:54
movement of data from one place to another, then another place
36:57
to another. So again, I think that's one
36:59
thing that Buzz has tried to do is we've tried to create a
37:02
holistic platform. It's an enterprise grade
37:05
platform, which means that, you know, I could take something
37:08
from again, it, its infancy from its, its genesis and that that's
37:13
again, maybe again, even building a training model to how
37:17
that turns into an app and hosts the app as well too.
37:19
So again, we, we provide VMS, we provide the storage services, we
37:22
provide the security services and provide the resiliency.
37:26
And really he again, it, it at least allows a user consumer to,
37:32
to not have to move the data back and forth.
37:34
So you could, you could host everything and run that app in
37:37
one place. So at least again, it goes back
37:39
to what we were talking earlier. Like I, I want to create the
37:41
proper customer experiences. I want to create the proper
37:44
environment for my base to be able to to use my infrastructure
37:49
without making it too complicated, right?
37:51
And yeah, moving data is not easy and it's not cheap.
37:54
That's what I've heard, that when folks are going and
37:59
shopping around for GPU's, a lot of times you forget about that
38:03
and then you are starting your project and you're like, Oh
38:08
yeah, eager recipes are kind of expensive.
38:11
We didn't realize that it was going to be this expensive.
38:14
And if you talk about the hybrid model, there are ways that
38:18
you're going to have to get really creative with that data
38:21
architecture so that it doesn't end up not making financial
38:25
sense. Again, you're like, you did all
38:26
this gymnastics to make it hybrid so that you could save a
38:30
buck here and there, and then it doesn't work.
38:32
Yeah. No, it's, it's again a good
38:35
point. I, I think when we look at
38:38
hybrid and what we saw traditionally, especially in
38:40
cloud was there were a lot of enterprises and companies that
38:46
preferred to keep certain applications on Prem.
38:49
And they, if they understood cloud and they understood that
38:54
the best thing to do with cloud is go cloud native, not lift and
38:57
shifts, then they were able to harness the value of cloud a
39:01
little bit better and then it became economically more
39:03
feasible too. But yeah, everyone that just did
39:05
a lift and shift to say, hey, I'm going to throw an
39:06
application cloud for no reason at all, that that wasn't a great
39:09
idea. And that became very, very
39:10
expensive. So again, hybrid was more a
39:13
force methodology previously where again, I think some
39:17
customers that needed to keep apps on Prem did that and that
39:22
could be compliance reasons too. Again, it goes back to what I
39:24
was saying earlier on sovereignty and data sovereignty
39:27
specifically where if if you couldn't show that again your
39:32
data was in one place and it was secured and ring fenced the way
39:35
you need it to be, well, you weren't meeting a certain
39:37
compliance and controls that you had within an organization.
39:43
So again that that that really kept again the Co location
39:47
business alive. It kept again a lot, lot of data
39:50
centers that where customers again just had their own racks,
39:53
they had their own space, you know, traditional enterprise IT
39:57
data centers that existed again, those are still there for that
40:00
reason, right? It's around a, a lot of it is
40:02
around the data sovereignty and the the data residency
40:05
requirements that they need to meet.
40:08
And then, yeah, you go to hyperscale and again, they were
40:10
great. They were great front end apps.
40:11
So they're great web apps that love to be in hyperscale cloud.
40:16
And they were suited for April, yes, but again, especially if
40:18
you're using cloud native techniques.
40:21
So again, come back to AI now. And again, AI loves on Prem
40:27
massive amount of data that you got to train.
40:29
But again, when you go to inference again, it's, it's not
40:32
again, the same data set. However, one thing I'll share
40:34
with you is like you, why would you create multiple data sets
40:38
for different types of phases within that that AI journey,
40:42
right? I, if I have my data in one
40:44
place, I want to keep my data in one place and I want to iterate
40:46
on it. Therefore, again, if I have to
40:48
go back and tune it too, I, I'd rather not create a second copy
40:51
of that data. So, yeah, I think I think you're
40:55
absolutely right. Like the the concept of hybrid
40:59
sounds attractive. It's not easy to execute, it's
41:02
not easy to manage. And typically if you get
41:05
everything in one place, you, you, you want to do that.
41:08
What are you most excited for that's coming down the pipe?
41:14
I, I, I see. So again, if you look at 1
41:18
sovereign around the world and what a, what a sovereign mandate
41:23
is, again, countries and nations, they're saying, Hey,
41:27
listen, I, I, I get that AI super important.
41:33
You know, I hate to use the arms race term because it rely it, it
41:36
kind of refers to, you know, trying to acquire as many GPUs
41:38
as possible. But, but the truth is, I think,
41:41
I think what a lot of countries have recognized is that they do
41:44
have to have AI strategy. And you know, for for certain
41:48
countries that AI strategy means, well, one is keeping a
41:51
talent pool or growing a talent pool within the country, right.
41:54
To help businesses to, to grow your GDP.
41:57
Yeah. And and it it it's super
41:59
important because one, yeah, you don't you don't want brain drain
42:03
too. Where again, yeah, you know, you
42:05
have your talent coming out your your universities and schools
42:08
just jumping to another country because of a higher paycheck as
42:11
well too Well. And they can't get access to any
42:15
cool stuff to work on. Yeah, no, it's, it's, it's
42:18
super, it's super. It, it, it's a big problem.
42:21
And then for governments, governments that other big, you
42:25
know, thing behind sovereign, right, government itself is
42:28
going to be a massive consumer of AI.
42:30
So again, yeah, they they do see it as a strategic position.
42:33
So you have to adopt AI as government.
42:35
There's only government departments and factions to
42:37
inside of that. But again, what's sovereign is
42:39
really it's, you know, it's control over the infrastructure,
42:42
right? Do I have domestic control of
42:43
that infrastructure? Do I have domestic control of
42:46
the operations? And that's all digital
42:48
operations too. And the last one is data like
42:51
that, that sovereign piece is really a data story, like is my
42:55
data itself sovereign as well? And now you can talk about
42:58
sovereign AI and again, the way the government supporting this
43:01
and Canada has been great for this, by the way.
43:03
Like, you know, the the first thing we did was we elected like
43:06
a minister of AI. That brilliance.
43:08
Yeah, we need a guy that that's hyper focused on this, like, you
43:11
know, fueling innovation, fueling the economy around AI
43:14
itself. And then we create a grant
43:16
program. So you know that and and you
43:18
hear, you hear about the headlines all the time too,
43:20
right? You know, you just allocated
43:22
$100 billion to it. So again, data center
43:25
infrastructure and AI from Saudi, you know, they stood up
43:28
humane and you know, they, they got another, you know, X billion
43:31
of dollars there. And then you got Stargate where
43:34
again, they, they, they earmarked $500 billion to go
43:37
build data centers, which again, if you look, you know, under the
43:41
covers, it's no, we're going to deploy, you know, the 1st $30
43:44
billion. And you know, this is this
43:45
progression of yeah, yeah. So, but you know, again, it it
43:50
is, it is a lot of it, a lot of those headlines where you're
43:52
going to see again, investments and but it's a stake in the
43:55
ground too. And, and countries saying, yeah,
43:58
no, I get this is hyper important.
44:00
It's one of the biggest revolutions that we've seen in
44:02
our time. So yeah, I want to make sure
44:04
one, that the funding is there so that, you know, we protect
44:08
our serenity. What number two, that we help
44:10
businesses innovate and, and move forward.
44:12
So again, yeah, data center GPU access isn't the bottleneck or
44:16
the threshold to go innovate and build.
44:19
Is there any other topics that you want to touch on that we
44:22
didn't touch on? As Buzz, we've been building
44:24
enterprise platforms forms, but we've also started to corporate
44:28
a sovereign philosophy and a sovereign firm framework in
44:31
that. So when I say enterprise
44:32
platform that that's going above again your basic cloud and AI
44:36
tooling that you would see to develop an AI project.
44:40
It is again incorporating very rigid compliance, very rigid
44:43
security techniques and technologies and systems.
44:48
It's incorporating high availability, reliability, tight
44:51
Slas for our customers, right? And then bring in the sovereign
44:57
it, you know, it's around again, adhering to certain national or
45:00
local policies and and keeping governance in control over in
45:05
the environment. So sovereign actually has become
45:07
the new standard. And if you say, hey, I'm a
45:09
startup and I don't even care about standard sovereign, you
45:12
might be right. You may not need Sovereign, but
45:14
because Sovereign is setting a high bar, you enjoy the
45:17
reliability of that cloud. You enjoy the reliability and
45:20
performance of those of those you don't have to.
45:22
Sacrifice just to be sovereign, Correct?
45:25
Exactly. Perfect dude, we got it.