Latest / Agentic Conversations (formally mlops.community) / How Universal Resource Management Transforms AI Infrastructure Economics

Transcript

How Universal Resource Management Transforms AI Infrastructure Economics · Agentic Conversations (formally mlops.community)

0:00
People in Africa, for example, they are trying to get second
0:02
hand data centers from banks in order to build AI data centers.
0:09
So you can only imagine like what is a second hand data
0:12
center that was used in the bank by a bank for over 10 years?
0:18
What does it look like? You know, can you even program
0:21
that thing? I'm sure you get this all the
0:29
time being in Oklahoma. People mispronounce your name, I
0:32
guess, left and right. They.
0:34
Yeah. And it's spelled so perfectly.
0:37
It's like Wilder. You are the Wilder one, but you
0:42
pronounce it differently, right? Yes, we say Wilder in Brazil,
0:47
but you can call me Will. Yeah.
0:49
All right, Will or Will the Wild, I'm excited to talk with
0:56
you about this whole generation of hardware and accelerators
1:02
that is being almost like forgotten about because we think
1:07
that we need the newest H100B100 whatever insert, whatever the
1:15
top notch hardware is from NVIDIA that just came out and
1:22
you're taking a bit of a different approach.
1:24
Can you explain this approach to me?
1:27
Sure. So the basic idea here is that
1:31
we already have enough compute power outside of any of the
1:37
GPU's in order to run a lot of this AI workloads.
1:43
So what is this compute power? This is basically CPUs and other
1:49
accelerators that people are not using.
1:52
So we need to understand that since 3:00 or four years ago,
1:57
the situation changed from, you know, prioritizing training to
2:03
prioritizing inference. And this is a huge, you know,
2:08
mentality change. In my first company that was
2:12
founded in 2018, we are just focusing on training.
2:15
How can we make training better? How can we make that more
2:18
efficient? And the answer was usually like
2:21
either making better algorithms for the best hardware or no,
2:25
just buy like more hardware. And this usually meant NVIDIA
2:30
GPUs great. And then comes like to town 21
2:33
to town 22, we see inference start to start becoming this
2:39
main workload. And then GPUs, the way they were
2:42
made before, were not the silver bullet anymore.
2:47
So now you see a lot of companies, I run a lot of
2:51
workloads, AI workloads, inference workloads, using just
2:54
a machine with CPUs. And how is this possible?
2:59
Well, there are many things in there that are maybe too
3:01
technical for this conversation, but that's the kind of
3:06
understanding that I think that a lot of people forgot because
3:11
hardware became so cheap, cheap up to 2020 one that OK, let's
3:18
just buy more harder. But now we don't even, we don't
3:21
even have the resources to make the harder that we need.
3:25
You know, we have to wait for six months.
3:27
So are we going to wait six months to get the best hardware
3:30
in the world or are we going to start doing work right now?
3:33
And that's the mentality that I think we need to readopt because
3:38
we forgot about that. Well, so I know that there's
3:43
been a ton of work that happens around things like Llama, CPP or
3:50
CCPI can't remember which one. I was getting confused.
3:54
And there's ways that you can obviously distill models or you
4:01
can make them smaller, but our first instinct is let's try and
4:06
get the biggest model on the biggest machine possible.
4:10
Can you just talk to me about what the shift is like if we're
4:15
going to be running it on CPU's or if we're going to be running
4:18
it on a bit older hardware? How do you think about?
4:23
That so the, the great thing about the GPU approach is that
4:29
you have to consider that the models are very big, but the
4:32
data is even bigger. So what you're trying to do is
4:36
to get a bunch of GPU's together to work as one and then they're
4:39
going to have this pool of memory and that we're going to
4:44
be able to fit the the model and the data.
4:48
Great. But doing this is very, very,
4:50
very expensive, especially because you know, those GPU's
4:54
are they have a lot of power, computing power embedding them.
5:00
When you go to inference, the main roadblock is not a compute,
5:05
it's memory. So if you're trying, if you're
5:10
thinking about creating or designing a system for inference
5:13
only, then you want to prioritize your resource
5:17
resources to get as much memory as possible while still keeping
5:21
a good compute power. But you don't need to have the
5:24
best, best, best in the world. So with this, if you understand
5:30
this, you can see that systems today that are equipped with
5:34
CPUs only can benefit from memory expansion in order to run
5:40
inference. And there is a technology that
5:43
has been, you know, in the market for a while now that's
5:46
called CXL, which is basically RAM on asteroids.
5:51
And a few key companies around the world sell that, like for
5:55
example Micron here in the US. And these are like chips that
6:00
they're basically only people who deal with data center know
6:03
what CXL is. If you are just like an AI guy
6:06
or if you're working or not in other areas, people never even
6:10
heard of that. But this can be really helpful
6:14
in order to expand the RAM, the memory capabilities of a server.
6:20
So there are companies as already mentioned that are
6:23
selling just an expansion card and then you can get a server
6:27
that has already a good amount of RAM and then expend that even
6:31
more, enabling you to load to load models and load data on the
6:37
go. This way, you know, you don't
6:41
need to spend hundreds of thousands of dollars getting a
6:46
GPU system that will have a lot more compute power that you need
6:51
just because you are after the run that it carries.
6:54
So that's, that's the idea. So there are technologies that
6:57
are getting more and more popularized such as CXL and you
7:01
will see a lot more now because you know, the price of RAM is,
7:06
you know, increasing a lot since the since December 2025.
7:12
So you will see a lot more people trying to leverage CXL.
7:16
What are the trade-offs that you have to be thinking about when
7:19
you are doing something like that, like leveraging CXL?
7:25
I wouldn't say like CXL per SE, I mean there is a a matter there
7:31
is an issue with capability here and compatibility because not
7:37
all servers are CXL compatible. But this is going to change
7:42
quickly. But I think the main trade off
7:45
here is going to be tooling. So as I said, people are so
7:50
focused on GPUs and the tools work so well, they, they, they
7:54
don't even want or can take the time to do the transition, get
7:59
that all the tooling that works out-of-the-box with GPUs to work
8:03
on other chips or CPUs. And by tooling you mean like
8:08
Slurm and Kubernetes or what kind of tooling are you talking
8:12
about? Slurm, Kubernetes and mainly
8:15
like the lower level tools such as CUDA for example, how do you
8:18
program a GPU? How do you program, you know,
8:21
all those advanced chips? So if you compare side by side
8:27
like CUDA with other tool that that I don't know, HIP for
8:31
example from, from AMD, CUDA seems to be way more developer
8:38
centric or developer friendly, at least for the AI crowd.
8:43
If you are a data center guy, you will see, OK, I can, I can
8:46
handle that. But this is the main trade off.
8:50
And that's exactly, you know, one of the points where I've
8:54
been working with with a few leads and clients.
8:59
Yeah, because a lot of times, especially for the folks that
9:05
are doing AI workloads, all of the CUDA stuff is abstracted
9:10
away. And it's even like sometimes
9:14
you're looking at managed Slurm and you're looking at managed
9:19
Kubernetes. You don't have to even be
9:21
thinking about that layer either.
9:24
You're just doing your stuff and it works.
9:27
And so it feels like what where you're saying there is this
9:33
issue, like we've got all this hardware, we've can leverage
9:37
this hardware that we haven't been thinking about, but we need
9:41
to make it more developer friendly so that you don't have
9:45
to be thinking like, oh, I was proficient in CUDA, or maybe I
9:51
wasn't even, I didn't even think about CUDA.
9:54
Now I've got to learn some other random thing, because this
9:59
doesn't even run with CUDA. Right.
10:02
And, you know, I'm all about all four abstraction.
10:08
However, abstract abstractions are great if you great if you
10:12
live in a ideal world. If we were able to get all the
10:16
compute power we needed from NVIDIA and if we were able to
10:21
fabricate that, yeah, sure. But that's not how the world
10:26
works. And on top of that, there are
10:28
different applications. And there is, you know,
10:31
companies have already invested hundreds of millions of dollars,
10:36
even billions of dollars buy infrastructure that wasn't AI
10:40
friendly. And now they just want to, they,
10:44
they are asking people like me and say, Hey, how can I make
10:48
this thing that I already have to work for AI?
10:53
And I mean, there are many ways you can answer this question.
10:55
And that's a real need. You're just, you're not going to
11:00
just throw that hardware away. I mean, that's a waste.
11:04
Yeah. So you're almost like dragging
11:07
forward this expense that companies have had.
11:13
You're bringing it into the new age of AI.
11:16
Right, right. One very common use case and you
11:20
will see more and more in 2026 is the we call it like the
11:27
conversion of data centers that were previously dedicated to
11:31
Bitcoin mining into AI powerhouses.
11:36
There's a lot of money into that, like from private equity
11:41
and venture capitalists who are investors in data centers that
11:45
today are not profitable anymore because Bitcoin mining is not
11:50
profitable. But they do have the compute
11:53
power sitting there and they have the contracts for, you
11:57
know, power and like electricity.
12:00
So with those two things combined, if you can have tools
12:04
and knowledge in order to convert that to be AI friendly,
12:08
then you already have a data center ready to go to run AI at
12:11
least inference. And coming back to the developer
12:17
who is just trying to run an LLM on a machine that isn't
12:22
necessarily a GPU, where and how should they be thinking through
12:29
like the best ways to go about that?
12:34
I still keep running through my head like, is it should we be
12:40
thinking about smaller models? Should we be thinking about a
12:44
shirt, a certain shape of AI workload?
12:48
Or is it that you can kind of retrofit any type of hardware to
12:56
the needs that we have? Right.
13:00
So let's start with this, the model size.
13:04
So there is a lot of interest in smaller models and I think
13:08
that's the way to go for most use cases.
13:11
I mean, many people wrote papers like proving that the small
13:14
models perform even better than bigger ones for a very
13:18
self-contained use case. So I think that's the way to go.
13:21
I see a lot of developers going this direction and yeah, that's
13:24
great. The thing is, when you start to
13:29
scale, even if it's a small model, you still have that data
13:33
sitting there. You want to still process that
13:35
data, right? So, and you want to process that
13:39
fast. This if you continue to use the
13:44
same tools or if you continue to adopt the same hardware that we
13:49
do today for bigger models, you're not going to get that
13:53
because first, you're working with a small use case, so you
13:56
probably don't have the money to spend the latest and greatest
14:00
hardware. Second, once you if even if you
14:04
had that thing, it would be an overkill.
14:07
So what is, how can you solve this equation of like the
14:13
relevance and the size of your problem and being compatible
14:18
with the compute power that you have and the compute power that
14:21
we have? As you know, a society is
14:24
already great. I mean, think about it like when
14:28
was the last time that you consider like, you know,
14:31
changing your personal computer, like getting a new one?
14:34
Like the computer I'm talking to you from right now is like it's
14:39
three years old and I'm a computer guy.
14:42
Like, you know, I could, I could easily be be buying like the
14:48
latest and greatest and I just don't need I can extract maximum
14:52
performance from an older machine.
14:55
So this philosophy I think will be necessary for developers to
15:03
continue to move forward because again, supply chain is a
15:07
problem. Even if you had the money, you
15:09
couldn't buy the latest and greatest today.
15:13
So there's the small model revolution that we're seeing,
15:18
and I also wonder about the idea of coding agents and the shape
15:25
of that workload and if you have any thoughts there.
15:31
I've been working a lot with agents from my user standpoint,
15:36
like I use 10s of agents to do several work in parallel for me.
15:42
And what is great I think that the way agents work is like from
15:47
a developer perspective, I give them very well defined tasks,
15:51
meaning that they don't divert, they don't get out of their
15:55
lane. This reduce computational need.
16:00
So you can do that work for one agent in a very, you know, small
16:05
containerized, so to speak, hardware and software space.
16:10
So if we scale our work in order to use hundreds of thousands of
16:18
agents, then we can have like smaller machines or hundreds of
16:23
thousands of smaller machines providing the infrastructure for
16:26
those little guys. So again, I see that you don't
16:31
need the latest and greatest to run agents.
16:36
So that's the way I think this is very important to emphasize
16:40
because when I talk to people who are not technical, they are
16:43
like, you know, they are CF OS, they are CE OS who are just
16:47
trying to, to go into the AI era.
16:52
They always ask me, hey, but I don't, I need to buy a GPU
16:58
server. No, you can start with this
17:01
regular Dell server that you already have in your office.
17:06
And this is going to be more and more evident as people start to
17:09
get or to embed AI agents within their, you know, software.
17:14
SAS software is using a lot of this and also in the developer
17:21
workload. Yeah, it's almost like how far
17:24
can you get with just the consumer hardware that you have?
17:27
And once you've maxed that out, then you can upgrade to the pro
17:31
version and really see what is needed.
17:35
But I appreciate your vision a lot because it's it's kind of
17:40
saying, hey, let's take a step back and let's leverage what we
17:45
already have. And for a lot of the workloads
17:50
that we instantly just default to let me go grab some GP us off
17:56
of wherever it may be like the newest Neo cloud, we probably
18:03
don't need that in a way that we think we do.
18:06
And so getting creative on how we use the hardware that we
18:14
have, it is fascinating. And it makes me think, like up
18:21
until now, we've been talking about how to leverage hardware
18:27
that potentially you've already bought, but then you can
18:29
retrofit in and you can add it and almost like pimp my ride of
18:33
my old hardware. Right.
18:37
I feel like there's also a world where we're going to start
18:42
seeing these offerings happening, just like we have the
18:46
Neo clouds that are offering GPU time.
18:50
Why? And it probably is already
18:53
happening and I just haven't seen it as much.
18:55
But why not have the hey, this is a really cheap option for you
19:03
and it gets the job done. I think this will happen.
19:07
People are already starting to change that because in a sense,
19:11
cloud providers have done this for over a decade now.
19:15
If you go to AWS and you try to get a machine there, what you're
19:19
getting is what is a virtual CPU?
19:22
It's basically it's a virtual things.
19:24
You're not getting the whole the whole hardware.
19:27
So what they're doing here is if they're basically proving that
19:31
you can run a lot of those tasks using less hardware, and we've
19:36
done that for over a decade. Obviously they charge a lot for
19:40
that and I believe it can be way cheaper.
19:44
So as people wake up to this problem, you know, they they've
19:51
perceived that they're not going to be able to buy the latest in
19:53
graders from NVIDIA. So how can they better use the
19:58
machines that are in the cloud? The way to do this is to change
20:04
the way to deploy stuff. So with the NVIDIA tools, with
20:09
the AMD tools it is. Somewhat easy.
20:12
So everything is abstracted as he said.
20:14
So you are seeing a Python code and everything is done in the
20:18
back, in the back end for the other chips.
20:22
And here I'm including as well the new chips that are arriving
20:26
in the market, right? So we can talk about that later
20:28
on. But there are one of the biggest
20:33
attractive points for developers when they see a new chip is, oh,
20:38
I can do the same task for, you know, it's much cheaper and it's
20:43
going to be faster. However, I need to learn an
20:46
entirely new language. And no professional developer,
20:51
at least very few will do that. They don't have the time.
20:55
They just don't have the time. They they are excited about it,
20:58
but they need to get work done. So in this point, like we need,
21:04
I see that there is a huge gap there in terms of infrastructure
21:09
to help those guys. So how can we help them to do
21:12
this transition as easily as possible, as smoothly as
21:18
possible? And there aren't enough tools
21:20
out there. Yeah, the idea of learning a new
21:26
language is great in theory, but then I guess what is probably
21:32
happening now is that you'll just send Claude code over to
21:36
learn the language for you and hope everything goes well.
21:40
Right, Yeah, the Cloud Code is a very good, you know, window into
21:47
this world because it already has some might have some
21:50
knowledge about this, either not a language per SE, but a
21:55
framework that's so obscure that almost nobody knows.
21:59
And it can give you like a, a nice little platform in order
22:05
for you to build your application at as things get get
22:12
better or faster, then we're going to be able to just, hey,
22:16
we're going to get some agents in charge to doing this and then
22:20
get just get the result. But again, when you're talking
22:25
about enterprise and real like production grade software, we
22:31
are not at the point right now where we can leave agents by
22:35
themselves to do that. So we need still like to give
22:40
the tools to the human developer so they can learn and assess if
22:45
what the agents are doing is correct.
22:49
Yeah, the phrase that comes to my mind is hope is not a
22:53
strategy if you're just throwing a bunch of coding agents at it.
22:58
That is literally the definition of hoping everything is going to
23:03
work out. Right.
23:06
The other thing that I I was thinking about too is talk to me
23:10
more about the other chips because I think we hear a lot
23:17
and we see a lot, especially now recently Crock got semi bought
23:22
by NVIDIA and then you have what is it?
23:24
Cerebrous is another new chip. There's all of these chips that
23:30
are coming out and saying for inference we're faster and we
23:34
are going to be a better bet, cheaper, faster, whatever.
23:40
Better, right? There were multiple bats in the
23:46
last, I'd say five years, seven years.
23:51
And let's go through their perspectives.
23:53
A lot of them were about saving energy.
23:57
Let's make a very efficient chip that can do the same job for a
24:03
fraction of the power consumption.
24:07
To my knowledge, most of those companies have failed because
24:13
unfortunately in this area, nobody is thinking about saving
24:18
energy. You know, that's not a first
24:20
priority. And a lot of those companies had
24:23
like amazing chips and they they weren't just picked up because
24:28
people don't care about that feature specifically.
24:30
And again, the developers needed to learn something completely
24:34
new. The technology still there can
24:37
still be useful, but it didn't become a mainstream.
24:41
The other perspective was about how can you do the math inside a
24:49
chip in a completely different way.
24:51
And I was involved with that in my very first company.
24:55
We were doing some stuff. We are starting to make a chip
24:59
in order to do the mathematical operations in a different way,
25:04
not based on linear algebra, based on something else.
25:08
And this kind of stuff is also very, very, very interesting and
25:12
people get excited about it. But then you hit the wall of
25:16
fabrication. There are so many problems that
25:20
need to be solved at the hardware level that it becomes
25:25
impossible. Like you can't even model that,
25:29
you know, like using a regular FPGA, because the benefits that
25:33
come from those new mathematical strategies, you cannot even
25:38
prove them physically implementing an FPGA.
25:42
You would have to do like a new FPGA or a new chip completely
25:45
from scratch just to prove that it works.
25:48
And as you know, this is very expensive, so we need to have
25:51
hundreds of millions of dollars and this is.
25:54
On the level of the like the silicon or the wafers that
25:59
you're creating? Not only silicon, not only the
26:02
silicon and the actual architecture and you know how
26:05
you build, but also in the micro code which is like think of like
26:09
a firmware for the chip. So, you know, the operations are
26:13
done differently there. So technology still exists, is
26:18
still exciting, but it requires like it didn't find a killer
26:22
application. What I see today, especially in
26:26
the last two years, it has become more and more mainstream.
26:30
Is it still like coming from this idea that by changing the
26:33
math you can do something better?
26:37
There are companies like AI, think a Tenserdyne, they're
26:41
actually from Germany, the headquarters in Munich, and
26:46
they're doing something that's great.
26:47
It's basically they're using logarithm logic, logarithm
26:52
arithmetics in order to do AI computation.
26:56
Because in AI, if you know, I mean I'm sure you know this, you
27:00
don't need this huge precision in order to calculate the
27:04
weights. So you can do that with much
27:09
lower precision and when you adopt logarithm arithmetics, you
27:13
can even like decrease it further.
27:15
So I mean, I recommend you to have a look at those guys.
27:18
I think I think it's great what they're doing.
27:20
I don't think they have a chip yet, but that's something I've
27:24
been in touch with them. I wanted to test their chip.
27:29
And those are basically the approaches.
27:31
And I think again, just to summarize, if you chips that
27:35
have prioritized saving energy have failed because nobody cares
27:39
about it, unfortunately, the ones who are probably going to
27:43
win are the ones who really focus on one thing only, which
27:49
is in our case, AI inference. How can we make the best
27:53
inference chip in the world? And that's what Grok was trying
27:56
to do. Grok is amazing.
27:58
I mean, what they were doing with such like every chip has,
28:02
you know, the amount of memory there, like it's ridiculous,
28:04
very small. So they needed to put thousands
28:08
of chip together just to run. And I don't know, OK, don't put
28:11
me on that. I don't know if it's hundreds or
28:13
thousands, but just to run one LLM.
28:16
But it was fast. Very, very, very, very fast.
28:20
And that's why, like, even NVIDIA had to go out and shop
28:25
for them. You know, they had to admit to
28:28
the world, yeah, our technology cannot do that.
28:33
And do you see fundamental approaches that are different
28:41
coming forward? Like on top of this, do you feel
28:43
like there's still room for the innovation in that regard?
28:48
Yeah, but it's not in silicon. I think photonics, there are a
28:54
few photonic chip start-ups coming out right now and they
29:01
have some impressive benchmarks. And you know, photonic you're
29:07
doing like a competition straight with light.
29:10
So when you do that, you can get rid of a lot of the limitations
29:16
that exist in the silicon world. But what was what's probably
29:21
going to happen is to have like a photonic ship chip working
29:26
together with ACPU like it always is, right?
29:29
There is always an accelerator that communicates with ACPU.
29:34
So I don't have, you know, hands on experience in with photonic
29:40
chips, but from the conferences I've been attending, like I've
29:45
been talking to those people, it is very exciting.
29:47
And another thing that I've been seen as well as like a revival
29:52
in HPC. So AI has become so big that
29:57
started to, you know, incorporate HPC into it and it's
30:02
great. AI can solve a lot of the
30:04
traditional HPC problems much, much faster.
30:07
But there are still problems that cannot be solved by AI, at
30:11
least the AI that we have today. And we still need like this high
30:17
precision, like 64 bits. So I see companies coming out
30:21
with new CPUs and GPUs that are focused on high precision, like
30:26
64 bits. And I, I, I see that there is a
30:30
lot of space for this right now as well.
30:32
Because, you know, NVIDIA has, I want to say abandoned, but they
30:38
focus so much on AI that their chips became way too expensive
30:43
for HPC. So if you are a regular HPC
30:46
shop, you probably wouldn't be doing this with any of it
30:50
anymore. You can do this with other chips
30:52
that are will give you the precision that you need and are
30:56
going to be much cheaper and. Can you talk to me a bit about
31:02
how you've seen the best hardware combinations as far as
31:08
maybe it's GPUs working with CPUs and what that looks like?
31:14
Well in terms of architecture, I think each company or each team
31:21
is different because they have different problems.
31:24
I have a ton of experience with the traditional CPU plus some
31:29
accelerator and this could be an NVD GPU, an AM DGPU or even
31:35
Fpgas. So to make those guys to
31:38
communicate with each other, because they have to you
31:42
traditionally you have to use PCIe and there has been
31:50
throughout the years a bunch of upgrades to the PCIe protocol in
31:55
order to be able to exchange as much data as possible.
31:59
And this is great for this vision of, you know, let's use
32:07
the compute file that you already have.
32:09
Because if you, if you can update PCIe in order to exchange
32:14
as much data as possible, then you, you're going to have a
32:17
fairly fast machine to perform your AI workloads.
32:22
Now, if you're building something like the the Starship
32:27
from Open AI, you need a completely different way of in a
32:32
completely different protocol to communicate data.
32:35
Like how can you make those hundreds of thousands of GPUs to
32:38
perform as well? That's why, you know, Innovita
32:42
has an Envelink and other things that are so specific because
32:48
those are the extreme use cases and it's great.
32:51
I think that's the way to go. But that's one company. 99% of
32:58
other companies don't need that thing for the use case.
33:03
So how can we still like use or reuse the protocols that we we
33:09
are used to, like the PCIe, for example.
33:12
And I see that this continues to be very strong.
33:17
I, I went to supercomputing last year in yeah, I was in, I was in
33:23
November, December. Now, I don't remember.
33:25
And people are, you know, continue to push the limits of
33:29
this thing. And there is so much coming out
33:32
from this era. And with PCIe, is it possible to
33:37
mix and match different GPUs? Yes, yes, you can do that.
33:44
I used to have a machine where I was running different GPUs from
33:49
different vendors and it works great.
33:52
It's, it's always about the work you're willing to put in in
33:57
order to make those tools work. You know, we've done this so
34:00
much and I personally have done this so much that we started to
34:05
create like a framework on how to do this.
34:08
And that's what we are trying to to deliver to our clients today.
34:13
So how can you mix and match have an heterogeneous system
34:18
that works for you? Yeah, talk to me more.
34:21
About what you're working on and what you're building.
34:23
Because I I find that fascinating too.
34:27
We are a dev tools company, so we are developer first.
34:30
We're trying to help the developer to have an easier life
34:34
when deploying AI. So what we build is a tool
34:39
that's based on a. We call it a transpiler.
34:42
We assume that the world is the world speaks Scuda, the world
34:46
speaks NVIDIA, but we still need to speak other languages because
34:51
we have different applications O how can we have like an
34:54
instantaneous translator from the CUDA world to another world?
34:59
And that's what we're trying to do here is like a plug and play
35:02
solution, a transpiler that you know, it's a common line, very
35:06
low level, but for the developer who is doing this every day,
35:09
it's not difficult to use. So you just need to install the
35:13
transpiler and then you go inside the folder you're working
35:16
the folder that contains your CUDA code.
35:18
And then this will convert that CUDA code to the computing power
35:24
that you have in a given computer.
35:26
Let's say that you you develop in your laptop, you develop
35:30
everything to run on an NVIDIA capable server, but then the
35:36
server that's available to you, which will be much cheaper, only
35:41
has a MDG. PS:.
35:43
How do you do that? I mean, you can't, you won't be
35:47
able to just rewrite everything from scratch.
35:49
Those are different animals. So how can you quickly move from
35:54
one world to another? And that's what we are we are
35:57
working on today. So we have our first MVP.
36:01
We are working on a use case, specifically the use case
36:03
already mentioned, which is helping the previously Bitcoin
36:08
centric data centers to to transition into AI.
36:14
But yeah, there is so much more to be done because it's a very
36:17
low level work. Like we we're not even touching
36:21
Python. So we're doing a lot of stuff
36:23
with C++ and using, you know, the LLVM framework in order to
36:31
break down stuff and see how intermediate representation
36:35
works for several different types of workloads.
36:38
Yeah, it's not easy. It's not supposed to be easy.
36:41
Otherwise everybody could do that and it would be great.
36:45
So we're trying to make it easier for people to use because
36:48
we believe in a world where, you know, CUDA will be a lingua
36:53
franca, but there there's going to be huge communities that
36:56
don't speak CUDA, and we want to enable those developers to tap
37:01
into those communities. And by community I mean
37:04
computing for. Hardware, yeah, hardware, of
37:07
course. And you want to stick to the
37:10
software layer as opposed to because I, I feel like there's
37:14
an easy play for someone to say, look, I'm just going to go and
37:19
I'm going to grab a lot of these abandoned hardware accelerators
37:26
and create like a misfit cloud neocloud where you could have
37:31
that as your, your value prop too.
37:34
Oh yeah. That's, that's our very next
37:36
step because it is this all depends on capital.
37:41
So you're like at the very beginning of the company, but if
37:45
things work out, one idea, and I dream about this every day, is
37:51
to have this, you know, I don't know the best way to put this,
37:55
but it's a cloud of, you know, heterogeneous cloud.
37:58
So we can offer you every chip that exists under the sun and
38:05
you as a developer have two options.
38:08
You can give us the code or the binary and let us handle that
38:12
for you. So we're going to optimize for
38:14
what is the best hardware to run your code, or you can have the
38:18
power to choose from all of those options.
38:24
The second option already kind of exists in the traditional
38:28
hyper scalers, but it's so difficult to use.
38:31
So if you're trying to use not CPUs, but like for example, if
38:36
you are like at the at the leading edge or trying, trying
38:41
new ideas on the hardware, you want to use an FPGA.
38:44
AWS offers Fpgas in the cloud. It is so complicated for
38:49
somebody like me who is experienced with Fpgas, the
38:52
whole, the whole tools, the tooling and everything.
38:59
It is so difficult to make it work that I would love to have
39:03
just like a translator to just hey, grab it, grab my code and
39:06
just put in there. Actually, this is something
39:10
hopefully I'll be able to publish this in a couple of
39:11
months. I'm working this is small FPGA
39:15
from this company Rocketry. It's a very small FPGA and I'm
39:20
creating like an example here on how to help AI developers on how
39:25
can they make like a minimal, very minimal accelerator.
39:30
So they plug this into their computer and then they can just
39:33
play with, you know, moving operations, let's say a matrix
39:38
multiplication to this little guy here.
39:41
The focus here is not to be fast, it's just to teach them or
39:45
to give them more visibility on what it takes to actually make
39:50
an accelerator. And once you understand that,
39:53
you can see like the possibilities like, oh wow, I
39:56
can do a lot if I just, if I have the chance to, to spend
40:02
more time on the hardware side. And that's, I think that's the
40:06
main thing that we lost in the last 20 to 30 years as a
40:11
community. Hardware has become so good that
40:18
people abstracted everything and they just assumed that it would
40:21
be there. And they can just code something
40:24
and it will work. And for the most part, it did.
40:29
Now we hit another wall and we need to go back to harder.
40:32
And I'm an electrical engineer by training.
40:35
I'm not a computer scientist. So I didn't start with coding.
40:37
I started with like, actually designing this sort of, you
40:41
know, little boards. So I see the problems that we're
40:46
facing today in the AI community from a different lens.
40:51
I see this as a really hard or problem understanding what you
40:56
can do with the you know as little hard as possible.
41:01
I do love the vision of bring your code to the Neocloud and I
41:11
don't have to think about what kind of hardware I'm going to
41:14
get. I just know I'm going to get a
41:16
screaming deal and I have it in my I write it in whatever
41:22
language I'm trying to write it in.
41:24
And then you do all the hard work of figuring out how to make
41:27
it work on all these random red headed stepchilds of hardware.
41:35
Right, Exactly, exactly. And I think chip designers are
41:41
not going to stop. There's going to be every week I
41:44
see like a new idea either coming out, you know, it's a
41:47
spin off from a lab or somebody who is a veteran in the industry
41:53
like spent, I don't know, 10's, the 15 years at Intel and now
42:00
they're coming out with a new company that promised a new kind
42:03
of chip. So this is not going to stop,
42:07
but they are only going to be successful if the tools that
42:13
connect those chips to the developers are there. 100%
42:20
otherwise you have this fragmented ecosystem and folks
42:25
aren't going to put in the time to learn how to leverage your
42:30
accelerator. If it's like, yeah I may
42:33
encounter that in a random job, but let's be honest, I'm going
42:40
to get way more bang for my buck if I learn CUDA.
42:44
Right. And another spect of that is
42:47
like, if you think about geopolitics, we are like, I
42:51
mean, you're in Germany, I'm in the US relatively, you know,
42:56
they're privileged countries when it comes to access to
42:59
compute power. If you start talking to people
43:02
in South America, in Africa, and I, I do like I'm from Brazil and
43:08
I know people also doing similar work in other parts of the
43:11
world. You will see that people in
43:15
Africa, for example, they are trying to get second hand data
43:18
centers from banks in order to build AI data centers.
43:24
So you can only imagine like what is a second hand data
43:27
center that was used in a bank by a bank for over 10 years?
43:33
What does it look like? You know, can you even program
43:36
that thing? Most, most, most programmers
43:40
today they they can't, but that's the kind of material they
43:44
have and they're not stopping because of that.
43:46
You know, they know they can't get the, the NVIDIA GPUs.
43:51
So we, we, we these people, they have to outsmart their their
43:59
competitors in order to be able to do the same work with less
44:02
harder. So changing gears for a second,
44:06
I know that you are in Oklahoma and around there you got a lot
44:12
of oil men. I'm currently watching this
44:15
series called Land Men, and I feel like I am living
44:21
vicariously through the oil men through that series.
44:27
But that's your day and day. And I know that you have some
44:30
thoughts on like the middle of America and the AI revolution as
44:39
they perceive it versus the coastal cities and how they
44:44
perceive it. Right.
44:46
So we, you and I, as I said, we are used to the Silicon Valley
44:51
mentality like very open, let's build the future.
44:56
There is a lot of here I would say like it.
45:00
It's not they are not excited about the future as the problem
45:04
is they are very risk averse and that's most of the country
45:09
actually, I think people in the US who are actually excited and
45:16
excited and building the future, those are only two cities, it's
45:19
New York and San Francisco. And in Silicon Valley when you
45:23
go everywhere else, you talk to people, they are excited, they
45:29
see the opportunities. They just don't take the first
45:31
step because there are many roadblocks.
45:34
And this, in my opinion, this comes from education and
45:39
exposure. We simply don't have enough
45:43
people with experience living those areas.
45:46
I came here because of family, but you know, it's hard to find
45:52
people to even talk about this on, on on the street.
45:57
Like I obviously I, after three years, I found a little
46:00
community here, but it's hard to find those people have this
46:05
experience like in big tech or, you know, big enterprise.
46:09
And we're in charge of making these decisions in the past.
46:14
So you just don't have the culture.
46:17
How do you change this? Because there is money here,
46:20
there, there are use cases. You know, it's a perfect place.
46:23
You know, we have energy, which is the commodity that you need
46:26
for AI today. You have tons of energy.
46:30
How do you change that? Well, you need first more people
46:36
with the the right background to be here.
46:39
We need to create more opportunities for people to
46:41
leave the coast. And this is hard to do.
46:44
Yeah, you need to create more opportunities here for those
46:47
people to meet. It's even hard to meet those
46:50
people we we don't know we exist.
46:51
Like I have to go out hunting here like on LinkedIn.
46:56
Hey, what are you working on? And once you do this connection,
47:00
then people get excited, like, and start work inside projects
47:03
like what I'm doing today. But I don't have a perfect
47:07
answer for that. I think if we are able as
47:13
technologists, if we are able to show them that we have a killer
47:18
solution for the problems that they have here today, then
47:22
they're going to be excited about it.
47:24
It's not about selling the technology.
47:25
This doesn't help here. I can go to Silicon Valley
47:29
tomorrow and start talking about this and everybody's going to be
47:33
excited. Like even at I get out of the
47:35
airplane and start talking to people and Oh yeah, let's do
47:38
something here. You cannot sell the technology.
47:41
You can only sell, you know, like a real close solution.
47:47
And since AI is very experimental, experimental right
47:51
now, we're probably not in the right phase to do this.
47:55
But this is going to change. And I think 2026 is the year for
47:58
that because WOW data centers are being built and this region
48:03
is very attractive for this area.
48:06
Yeah, you got the energy.