Latest / Agentic Conversations (formally mlops.community) / Efficient GPU infrastructure at LinkedIn // Animesh Singh // MLOps Podcast #299

Transcript

Efficient GPU infrastructure at LinkedIn // Animesh Singh // MLOps Podcast #299 · Agentic Conversations (formally mlops.community)

0:00
Hi, this is Animesh Singh. I'm the Director at LinkedIn
0:03
leading our GPU fleet and infrastructure for training and
0:05
inferencing. In addition, I lead the
0:08
distributor training software stack across all of LinkedIn,
0:12
additionally contributing to a leading a lot of initiatives
0:15
that are on MO efficiency and performance optimizations.
0:20
Prior to that, I was a Distinguished Engineer and
0:23
Senior Director at IBM, leading a lot of core Watson
0:27
initiatives. Woo, I'm bubbling from this
0:31
conversation. So many gems when it comes to
0:39
working with LLMS at scale, GPU infrastructure, what you want to
0:44
be optimizing for, how you can think about optimizations, and I
0:51
had. A.
0:52
Incredible question at the end. Like how does the platform and
0:57
the GPU infrastructure that you're dealing with differ when
1:03
it comes to working on MLMS versus traditional ML?
1:09
Did not disappoint. Let's get into the conversation.
1:19
Well. Let's start with this, man.
1:21
And I'm so happy that we got to do this because we've been
1:25
jiggling around our calendars to make this work for probably
1:29
about six months. I think we were going to do this
1:31
when I was last in San Francisco in June.
1:33
But one thing led to another, and here we are in 2025, having
1:39
the conversation. Finally, persistence paid off.
1:43
I think this is great and I've been following your work
1:47
throughout as well. You're doing excellent work in
1:49
terms of, you know, bringing communities together and
1:52
disseminating that knowledge, right?
1:54
Like what's all happening in the space, right?
1:58
And what in these cases are springing up?
2:00
What are the industries they are targeting?
2:03
There is some excellent work you are driving in that and I'm so
2:05
glad it's happening in 2025. We have now some better
2:11
experience of, you know, what is working, what is not working,
2:17
what may be a little bit of the hype, right, What is realistic,
2:20
what is going to be the trends in 2025.
2:22
So I think the timing is is working out, yeah.
2:26
What is working? One thing which has definitely
2:31
proven that it's here to stay is NLMS, right?
2:36
I feel throughout, you know, 20, 2223, there was a lot of, you
2:47
know, discussions how effective LLMS are going to be right in
2:53
the industry, in the space. There are a bunch of, you know,
2:56
modelling architectures like, you know, recommendation ranking
3:00
LPMS, graph neural networks, GNS, right, But the efficacy of
3:06
LLMS in the use cases being powered by LLMS was quite a bit
3:12
of question mark, right? Like, yes, there was a promise.
3:14
What we saw that magic moment with Chad GBD coming in that
3:19
literally, you know, woke every up everyone uprightly.
3:22
It does seem seamless. It it does seem that it's it's
3:26
not yet another chart what you are talking to.
3:29
And that sprung up the industry, right when I joined LinkedIn at
3:35
that time, you know, the chargeability moment hasn't
3:37
happened. And as soon as I joined, a month
3:39
later, chargeability came in and what I came in here to do, a lot
3:43
of that changed within a period of a month.
3:46
And I think through the course of that period, multiple
3:50
companies, multiple industries have identified, right,
3:53
different use cases which are working well with this, right?
3:58
And people are being productive, be it, you know, either, you
4:04
know, generating code, be able to do certain automation
4:09
leveraging this. The interface does seem very
4:12
common like and a lot of the generative VI use cases which we
4:16
launched even on LinkedIn, for example, you know, profile
4:20
summarization, right? So based on what you have, you
4:24
know, create a headline for me, create a summary for me, use
4:28
cases like, you know, assisting falling the learning courses,
4:33
use cases like, you know, resume targeting recruiter emails for
4:38
candidates. Because a lot of the things
4:40
which we had seen that, you know, if you're getting cold
4:43
call emails from recruiters, they are not hyper personalized,
4:47
right? Like at times they sound like a
4:50
template generative AI in LLMS helped us immensely in that
4:54
right where they take into account the candidates profile,
4:59
the company in which candidate is working, the company from
5:01
which the recruiter is and create these very personalized
5:05
emails which you know, we are seeing success candidates
5:08
actually this morning much more opening those emails.
5:12
So I think you know, a bunch of use cases we have seen working
5:15
really well with LLMS and we are obviously doubling down, right?
5:20
Like if you see the talk of 2025, anyone before that last
5:25
year there was quite a bit of discussion on agents, what they
5:30
can do, what they cannot do. And we did our own experiments,
5:36
right, like invested a lot internally in terms of building
5:42
agent infrastructure, right? First of all, right, what does
5:47
it take to create agents? How's it different than the
5:50
traditional generative AI applications or use cases?
5:54
We were building more than once is right?
5:56
What makes it different? And then finally, we launched,
5:59
you know, closer in the last quarter of last year, LinkedIn
6:04
hiding assistant, right? So which is essentially an agent
6:07
for the recruiters, which based on certain criterias they define
6:11
will actually go work behind the scenes, find candidates for
6:15
them, relevant candidates, summarize their experience and
6:19
profile to the recruiters. And then based on that they can
6:22
tell, OK, you know, reach out to this particular candidates and
6:27
you know, let's start having a discussion.
6:29
And there is much more. We are doubling down on that
6:31
whole LinkedIn hiring assistant and we have seen some great,
6:34
great enthusiasm from our partners and customers, right.
6:38
So it's it's seeing some very good results.
6:41
So there is much more we are doing now on other areas off
6:44
LinkedIn which will be powered by agents.
6:46
So they are here to stay. I think it's how in which use
6:52
cases really benefit from it. That will be a nuanced
6:55
discussion. You cannot throw it at every
6:56
single thing. There are so many things which
7:00
can be just powered by them, which then frees you up to do
7:04
things right, which are probably the more creative aspects of the
7:08
work you are doing right And and we are seeing quite a bit of
7:11
that happening. So what's not working?
7:14
I, I wouldn't say what's not working, but a thing which
7:18
probably needs a lot of improvement moving forward is
7:24
the cost and the ROI of launching these LLM based,
7:31
either agentic use cases or traditional, you know,
7:35
prescriptive rag based generative AIE apps or even, you
7:40
know, leveraging LLMS for use cases beyond generative AIE.
7:45
The cost is a big hindrance. I think 1 if you see like
7:50
training itself used to be the biggest barrier for entry and
7:55
that God saw to some extent because first of all, you know,
8:00
even within LinkedIn we invested quite heavily in building out
8:04
our scale out training infrastructure which can power
8:06
elements. I think when I came in, you
8:09
know, we were working off we 100 fleet and then since then we
8:14
have scaled our LinkedIn fleet by 7X.
8:17
We have a one hundreds, we have H1 hundreds, we have H2 hundreds
8:21
and and the fleet is as modern as it could be and we have
8:25
scaled our training tremendously, right.
8:27
Like it's 150X increase in the the size of the models we are
8:32
training, right. Our, our data processing has
8:35
increased many Ford, we actually, you know, completed
8:40
1,000,000 training runs on the platform and we're training big
8:44
foundation models now investment in the infrastructure renting
8:48
and I think it was well understood for companies at
8:52
scale of LinkedIn etcetera, right, where a lot of content
8:56
based data is being consumed, being produced, you will have
9:00
tons of data, right? And when you have these tons of
9:03
data, you need to make sure that you have the infrastructure to
9:05
train models on these data, right.
9:07
And then the other thing which actually helped a lot of the the
9:12
training landscape is the emotions of open source models.
9:15
I think, you know, Meta led the way, followed by many of those
9:20
in the industry, right? Where for smaller companies or
9:24
for companies, which there is not a need to train a model on
9:28
word data, right? Like you have your own specific
9:31
data, which is probably, you know, not as big as as the word
9:35
data. Getting these open source models
9:38
as starting on top of them, because these models already
9:41
know what exists in the world. They can answer your questions
9:44
around, you know, they've gone through Wikipedia, they've
9:47
called through, you know, public libraries, all the articles and,
9:50
and they're being trained on them.
9:51
Then you can bring in and do more fine tuning instead of, you
9:55
know, training on huge amounts of data.
9:59
So then the infrastructure cost can go down further, right?
10:03
So fine tuning became a big mechanism.
10:05
Plus, you know, the mergers in the industry around a lot of
10:08
techniques around supervised fine tuning. 0 short, few short
10:11
learning which emerge where, you know, prompt optimization
10:15
techniques which emerge which essentially brought the cost
10:19
down heavily on the training side.
10:23
What is now, you know, more and much more in the picture is the
10:27
cost of inferencing, right? And, and I think it's, it's
10:32
humongous, it's big and there are a lot of efforts, right,
10:35
which we are doing, which the industry is also doing right.
10:38
Overall, how to bring down the cost of inferencing these
10:40
models? Now if you take a look at use
10:45
cases specifically as well, there is some thinking which is
10:49
built in like when you are interacting with model, right
10:55
where you are asking certain queries, asking you to analyze,
10:58
you're prepared mentally, it will take some time, it will
11:00
think through it, right? Even with the emergence of you
11:03
know, the latest open AI models, right, where there is a lot of
11:07
reasoning going on, right? So there is a lot of it is
11:09
actually analyzing its own output, then you know refining
11:13
it's own output, then the second output is further analyzed.
11:15
There is a lot of back and forth reasoning.
11:16
So there are multiple inferencing calls happening.
11:19
And I think as a consumer we are prepared if we are going into a
11:22
scenario in this particular context, the model may have some
11:26
thinking time specifically if you are interacting with the
11:28
latest open AI models etcetera, then you know that you know
11:33
you're all asking complex queries which need that
11:36
analysis. So the latency you are are
11:39
willing to tolerate now the the argument and, and even to get to
11:45
that latency, right? Like there is tons of
11:47
infrastructure investment which has been made, right.
11:51
The general realization is that, you know, with even with all
11:56
these investments, we are not able to get our GPU's to perform
12:01
at maximum utilization, right? Like inferencing is becoming
12:05
very costly because you are optimizing a lot for latency
12:08
through port and you know, you have a lot of failover
12:13
mechanisms which you need to build like almost any company
12:16
needs to account for hey, I, I have two or three data centers,
12:20
right? Like if one data center goes
12:22
down. So there is a lot of redundancy
12:24
you need to build for applications which are user
12:28
facing. That means you know the cost of
12:30
GPU is ballooning, right? And and that's essentially is,
12:35
is an infrastructure problem which needs to be solved
12:39
specifically. Now when we do take elements in
12:42
use cases where the consumer appetite for latency might not
12:45
be there at all, right? Like there are efforts happening
12:51
across the industry, right? Like, hey, the traditional
12:54
Rexis, right, recommendation ranking, like so for example,
12:58
you go to social media sites, you, you get recommendations,
13:01
you get feed, you get people you want to connect with all these
13:05
things like, you know, as soon as you go to the site, this
13:09
should be there, right? You should be as you're
13:12
scrolling through the feed, the feed should be just mean, you
13:15
know, updated and and customized for you in real time.
13:18
There is no appetite for latency in those scenarios.
13:21
Now, if, if you, if you need to see if elements can be effective
13:26
in those world, that means, you know, you nearly really need to
13:30
optimize for latency. And if you're really doing that,
13:32
you are potentially throwing more money at the problem.
13:36
And, and so that problem, I think, you know, how do we take
13:40
this large language model architecture, the transformer
13:43
architecture and make it really, really optimized for
13:46
inferencing, right? Is is becoming a big thing
13:48
right? Which needs to be solved full
13:50
scale, right? Do you feel like it's a bit of
13:55
trying to fit a round peg in a square hole?
13:58
Because when you throw LLMS at like a Rexis problem, just
14:03
because we've been doing Rexis for a while and we figured out
14:09
how to make it real time, why do we need to add an LLM on top of
14:13
it? It almost in my eyes is over
14:17
complicating things just to try and use a shiny new tool.
14:20
But maybe you've seen there's better performance, there's
14:23
better personalization there or something that I haven't?
14:27
I think I would more than speaking on my behalf, right?
14:33
Like in general, there are like, you know, research papers,
14:37
emerging companies are trying that now.
14:40
Why would you try something like that, right?
14:42
I mean, and that's a fair question, right?
14:43
Like Rexis is already well established, right?
14:46
And as as a architectural pattern, right?
14:48
Like the traditional recommendation ranking models,
14:51
retrieval models including, you know something like graph neural
14:56
networks and GNS, etcetera, right.
14:57
They do a very, very solid job at at this and you have seen
15:01
companies like. And it's fast.
15:05
It's fast and and like you, you take a look at TikTok, often
15:11
times you know, the recommendation algorithm is, is
15:14
is talked about. So you're fairly right like hey,
15:17
is why is is that why I think there are couple of things in
15:22
heuristic, right. So the way we have solved
15:26
recommendation ranking problems in the industry is obviously you
15:29
have created models, a lot of the companies have smaller
15:34
models, right. These are not traditional
15:36
foundation models which have been trained on word data
15:38
etcetera, right. They have potentially not seen a
15:42
lot of user interactions and patterns.
15:44
So then you know, you add things like real trying training,
15:50
there's a lot of data being ingested in real time.
15:52
Online training is happening. You know, there is there is a
15:55
lot of feature ingestion which is happening in real time.
15:59
What is user interacting with? So there is this new paradigm,
16:03
OK, which is these models, the LLM models, these are foundation
16:07
models, they have potentially seen maybe 95% of the patterns.
16:12
So maybe the what you need to do in real time to update these
16:16
models is is probably a lesser investment, right?
16:20
Models have seen majority of the patterns and and if you feed
16:23
what the user is done right like they, they would be able to
16:27
predict much more comprehensively.
16:29
You don't need to do a lot of online training etcetera,
16:31
writing your time. That's one of the the you know,
16:34
thinking. The other thing is the
16:35
simplification of the architecture for something like
16:40
GN NS, right? What we need to do is you know
16:44
when you are doing the training GNS, all the data is in the
16:46
graph format, graph structure format, right?
16:49
You need to traverse the nodes and the edges in real time,
16:52
right? Because you don't know pre hand,
16:54
right? Like how much data you will be
16:56
processing, which will be the right node, which edge you need
16:58
to traverse, right? So there is no data pre
17:01
processing happening and there is a different architecture and
17:06
it's inherently hard to scale GNS beyond a certain limit
17:09
because there is live data processing happening while the
17:12
training is going on or while inferencing is going on, right.
17:17
So an architecture and then GNN is one example.
17:20
There are different, you know, recommendation ranking
17:22
architectures and then companies would have a proliferation of
17:26
these recommendation ranking models, right?
17:27
Like every team would create for each of their use cases, they
17:32
would start from scratch and and create a model which is
17:35
potentially small model. It does very targeted job.
17:37
It does a really good job at it, right and then build all these
17:40
things. So you have bespokes models with
17:42
different architectures. There are huge number of them if
17:47
you take the LM route, right, like the the trend which is
17:51
emerging in the industry right here.
17:52
If I do create a giant foundation model, right?
17:55
And and if you, you're obviously very present in what's
17:59
happening, distillation as a technique is becoming very
18:03
prominent in the industry. Like I will create smaller
18:05
models for inferencing, but I will distill it from this giant
18:09
foundation model. So what you have done is you
18:12
know, you have sort of centralized the, the, the
18:15
creation of models in, in potentially a central place.
18:21
So you can think of a scenario in future, right?
18:23
Like there is one central team instead of having every
18:27
different use case in every different vertical within your
18:30
or creating your own models for their own use cases, which are
18:32
very targeted. 1 central team, which is, you know, the holder
18:36
of your organization's data, which is curating that whole
18:39
data and creating, you know, maybe one or two or three.
18:43
OK, very simplistic scenario. You're creating 2 giant
18:46
foundation models, one for generative AI use cases, one for
18:48
non generative AI use cases. And then you know, the models
18:52
for all particular use cases are being disturbed from this.
18:56
So you have simplified the, the overall architecture.
19:00
You are essentially worrying about the compliance majorly
19:04
like there is compliance is a big thing as well as you are
19:07
seeing right there was the the DMA act, there is AI act, there
19:12
will be other acts that will be coming.
19:14
If you if you curtail the surface area, how many models
19:18
you have, right, what data it was trained on, right, then
19:22
you're not worrying about, you know, the many hundred other
19:27
models which your organization has created and make sure
19:29
everyone is compliant. What data it was trained on was
19:32
that data compliant? You have sort of centralized and
19:35
simplified that problem. Hiding becomes easy, right?
19:38
Right Now you do tend to hire like when when I'm looking at
19:44
certain use cases, like, you know, GNS people who can run GNS
19:49
scale, right. You need a very targeted skill
19:51
set. People who understand you know
19:52
how to target graph data, and then you need to go into the
19:55
depths of the GPU's architecture where is.
20:00
Because what a lot of data transfer is happening during
20:02
that training. So OK is Nvilink good enough?
20:05
How much HBM memory I have? How much on disk memory I have
20:08
on on the GPU nodes? Then you know what is the
20:11
network bandwidth. There is a lot of in depth GPU
20:15
knowledge you will require right just to go there and and start
20:18
solving. Plus you know the graph
20:19
traversal. What algorithms do I need to
20:21
introduce to? So your sourcing of skills and
20:25
talent also become simplified. So there is that whole value
20:28
proposition which could be achieved provided if LLMS do
20:34
prove themselves. I think many are are, are
20:38
looking at this problem space and, and figuring that out.
20:41
So it's yet not solved. And as I said, like, you know,
20:45
like generative AI as well, this is a problem of scale, like
20:52
generative use cases. You have to explicitly as a user
20:55
go and invoke that, right? You will go like I'm giving a
20:59
LinkedIn example, right? You will go and say, hey,
21:02
summarize my profile. Or a recruiter will say, launch
21:05
this particular agent or are you, you know, LinkedIn learning
21:10
user will go and say, summarize this course for me or explain to
21:13
me this nuance. So these are, you know, discrete
21:16
transactions, users going to feed, logging into feed,
21:21
browsing through the content. This is happening all the time.
21:24
It is happening at scale, right? So, so then that's a bigger,
21:28
much bigger scale problem which needs to be solved with LMS
21:33
overall, right, in a cost effective manner.
21:35
I think it's it's and very, very, you know, latency
21:40
sensitive manner. So that's that's the thing,
21:42
right, which needs to be happening.
21:43
And we'll see whether they do prove efficacy, like, you know,
21:48
based on all the results you see, they're able to pass a lot
21:51
of bar exams, PhD exams, math exams, right?
21:54
So you'll see, hey, they are intelligent, right?
21:56
Like, can they be very intelligent for this specific
21:59
set of targeted problems? I hadn't thought about the
22:02
simplicity in the architecture and also the simplicity in being
22:07
able to attract talent that understands this architecture
22:12
because it is more simplified. And you were talking about a,
22:17
the cost of having GPUs at scale.
22:20
And when you have this many GPUs and you're trying to utilize
22:25
them, you don't want to have any percentage going idle.
22:31
I imagine you think about that a lot.
22:33
You're thinking, wow, we're burning money just letting this
22:36
GPU sit around and we're not utilizing it to its maximum
22:39
capacity. It does.
22:42
Is that what you're trying to do, or is that what Liger
22:46
kernels are trying to help out with?
22:49
Can you explain that? Any, any infra and platform team
22:53
which is an Emily infra at this point you would talk, this is a
22:56
burning thing, right? You have on one hand, you know,
23:04
you cannot run these LMS, for example, or or these modern
23:09
recommendation ranking models without GPUs, right training
23:14
definitely right. And even with influencing now
23:18
with elements, you have to go on GPUs, right?
23:21
Like the gender trend in the industry, like if you're going
23:25
two years ago was here, we will train on GPUs would sell on
23:28
CPUs, right? That's all.
23:30
Most of the companies are architected and CPUs were not
23:34
that expensive. The modern architecture, it
23:37
doesn't lend you to that, right? So we need to get.
23:39
So the whole investment in GPU efficiency becomes very, very
23:43
paramount. And it starts from at every
23:47
single layer like in our case, like, OK, there is the general
23:50
thing, right, which also happens in companies right here, you
23:52
need to allocate a maintenance budget.
23:54
Now you can be much more generous with CPUs, right, in
23:58
terms of maintenance budgets, how much spare capacity do you
24:02
allocate so that if you're 20% of your fleet is being
24:05
maintained on a regular basis. So that's the delta you need.
24:09
If you need high availability, you need to spread it across
24:11
three data centers. OK, let's go ahead and do it
24:14
right. Those decisions were much
24:16
easier. They're becoming much more
24:17
harder when you have GPU's in house because yes, there are
24:21
certain companies which are potentially invested a lot,
24:25
right? And and they're deep coffers,
24:27
but a lot of the companies want to be cost conscious about it,
24:31
right, like you are. So every decision, Hey, even to
24:35
the point like looking at certain use cases, what would be
24:39
the right maintenance budget, right?
24:41
Like if if you have, let's say you know, in in one maintenance
24:45
own thousand GPUs, can I allocate 100 just for
24:49
maintenance, right? Like then you do the cost and
24:51
it's like, hey, there's a lot of wastage which is going on you.
24:53
Start sweating. You start start.
24:55
Thinking about how much money that is and.
24:57
Yeah. And so, so how do you first of
25:01
all, start then looking at the workload so that they become
25:05
more resilient to maintenance? Like so, OK, if I cannot take
25:09
the previous, the previous approach used to be like, once I
25:11
know 20% of my fleet is going to be maintained, I will not
25:15
schedule anything, right? We don't need to worry too much
25:17
about, you know, for the next 24 hours, even if they're empty.
25:21
Now you're worrying about that. I cannot leave 200 GPUs lying
25:25
empty for 24 hours, right? Just because you know, the
25:28
maintenance is going to happen. So then you look at your
25:30
workload, OK, how can I make the workloads more resilient?
25:34
Now if you look at distributor training workloads like majority
25:38
in the industry are gang scheduled, right?
25:40
So you do gang scheduling, you know beforehand this distributed
25:43
training workload will attack X amount of GPU's, right?
25:47
And there is a lot of you know, synchronization happening as the
25:51
distributed training is going on.
25:52
And if if you know few of the nodes fail, you will have to you
25:58
know, put it back in the queue. Like you obviously need to have
26:01
checkpoint restore, put it back in the queue, do again gang
26:04
scheduling so that you know it can resume.
26:09
So then you look at this workload, OK, maybe in this
26:11
world we have to rethink the whole lotion of gang scheduling,
26:14
right? Do we need to build more
26:18
elasticity, right? Like if if a workload has been
26:21
launched, 100 GPUs and 30 GPUs are are going to be taken away,
26:25
can it shrink? Can it expand on its own right
26:28
and continue going Same for inferencing.
26:30
I think the need for scale down, scale up is becoming big.
26:36
Serverless architecture has to become a prominent scheme in the
26:39
thing of things like you cannot optimize for Max traffic like
26:45
and and a lot of us are are doing that.
26:47
A lot of the industry is doing that where you put what is the
26:50
peak traffic you get and that's your capacity.
26:53
But if out of 24 hours, 2 hours is your peak traffic for those
26:57
less of the 20 rest of the 22 hours these GPUs are sitting
27:00
idle because they are being provisioned to handle their peak
27:03
traffic. Again, a lot of vested.
27:05
So, so many of these decisions are now becoming very prominent.
27:09
So I feel the the emergence of elastic architecture, serverless
27:13
architecture to handle workloads which which, you know, should be
27:19
able to scale down, scale up as the capacity is, you know,
27:23
shrinking, expanding. This is, you know, a big
27:27
investment area and something like which, which we are doing
27:31
like, and, and I'm, I'm assuming like, you know, a lot of people
27:34
are, are solving this problem at different means and mechanisms,
27:38
right? So yeah, that's that's
27:40
essentially the the key characteristic there.
27:44
Well, and you haven't even mentioned the reliability of the
27:48
GPUs because they're notoriously unreliable.
27:50
They are like there are, I mean all these NVIDIA GPUs you get
27:54
for example, right? And, and one of the decisions we
27:57
took early on and I think for the simplicity part of it,
28:05
right, like, hey, let's start at least on the training side,
28:08
right? We'll go with NVIDIA.
28:10
And so far we have standardized on NVIDIA throughout, but then
28:15
you know, you have a lot of these checks things which can go
28:20
wrong because it's just taking a distributed training example.
28:24
When you are running at scale, you are touching your, in our
28:28
case, you know, we have Kubernetes as the computing for
28:30
layer, right? So you have Kubernetes, you have
28:33
the network, you have the hardware, you have the storage
28:36
training is, you know, reading and pumping a lot of data
28:39
throughout any of these points going down is a weak link in
28:44
addition to the GPUs itself, right?
28:47
And act with the planned maintenance.
28:49
So, so, and, and sometimes, you know, for foundation models,
28:52
your training can run for one week, two week, three-week at
28:55
stretch, right? When you are actually running
28:57
training for these large foundation models, you're
28:59
guaranteed to have some part of the infrastructure having some
29:02
problem. Sometimes it will be the
29:04
messaging queue, sometimes it will be the storage system,
29:06
sometimes it's the the something layer or sometimes it could be
29:10
just the GP node itself failing health checks ECC errors.
29:15
Nickel is a problem InfiniBand of the GP US right, which is
29:18
connecting all this. So this this poses a huge
29:23
challenge right in in terms of ensuring because these are so
29:26
long running jobs. How do you make sure that you
29:29
are being effective, right? And that combined with the
29:32
problem of gang scheduling because him they need to go all
29:35
together at once or otherwise they will need to wait if a if a
29:38
training job needs 500 GPUs, it will be in the queue until those
29:41
500 GPUs are free, right? You cannot launch it before that
29:45
because you know you have not I mean, that's how the the the
29:51
architectures around gang scheduling are built.
29:53
So this this is a big thing, right, which we have invested a
29:59
lot in terms of building, you know, very fast checkpointing
30:03
restore because you assume failure is happening,
30:05
maintenance is happening. So a lot of investment has gone
30:08
in from outside in terms of ensuring that, you know, there
30:10
is automated checkpointing restore which could be done and
30:15
which is, you know, hierarchical.
30:16
So we first checkpointed memory, then distributed those
30:18
checkpoints to some async block storage, then you have to read
30:22
right and and then you know, there is also discussions around
30:25
whether these checkpoints can reside on the GPUSSDZ.
30:28
So because there is a cost of, you know, reading once you have
30:32
like the training can continue checkpointing in memory.
30:35
But once you have send it to some remote storage, a block
30:37
storage or a file based storage, then when you're reading, if
30:40
something has gone wrong, that takes time, you have to bring
30:43
that through the network and and bigger models will have you
30:47
know, many gigabytes of checkpoints, right, bringing
30:49
them through the network. So then OK, can we leverage the
30:52
GPUSSDS under the covers itself, right?
30:55
GPUSSDS obviously have some limited capacity.
30:58
How much can you store? So bunch of those, you know,
31:02
architectural elements need to be crafted.
31:05
So we are investing quite a lot and quite heavily into that
31:09
Liger you talked about right? Was the other part of the
31:12
efficiency problem, right, Which is essentially, as we went on
31:16
this journey, we saw like beyond the, the reliability and, and
31:22
the scalability, a lot of the use cases, as the model started
31:24
getting more complex, more bigger, and we had an X amount
31:28
of GPU fleet, we started seeing a lot of pulls from our
31:34
customer. Hey, I'm IDE GPUs, I'm not
31:37
getting GPUs. All my training is running for
31:40
like, you know, so long, right? And, and I really want to get it
31:44
done faster. Multiple of these use cases and,
31:47
and literally people were because, you know, GPUs was our,
31:52
our scars entity, at least, you know, a year ago, we, we had
31:55
quite a bit of that, right? We were, as we were scaling the
31:58
fleet, right? And we were ordering NVIDIA has
32:00
its own supply chain and timeline, right?
32:03
The use cases were just springing out.
32:05
So the we looked at the problem, OK and and you follow the
32:08
traditional methodology of data parallelism, model parallelism,
32:12
let's introduce tensor parallelism.
32:13
We invested in technologies like 00 plus plus because our a 100
32:18
fleet had very constrained network bandwidth.
32:21
So how you can scale up training even with that constrained
32:25
bandwidth? 0 plus plus helped in that.
32:28
So after we have exhausted every single option, then OK, what can
32:33
we do? So as we started going deeper,
32:36
one of the things which we used to do infrequently for our
32:40
customers was, you know, rewriting CUDA kernels, right?
32:43
Like a lot of the use cases which are very sensitive to that
32:48
the training should complete within X number of hours
32:51
etcetera. When we looked at the the model
32:54
training code which our models used to produce, there was a lot
32:58
of improvement. So at times we will go and
32:59
rewrite the CUDA kernels few certain operations, right?
33:03
No, you, you bring this to the LLM.
33:06
The thought which we had was a, we can do this on a model by
33:09
model basis, but this is not scalable, right?
33:13
Like there are multiple users coming different models.
33:18
So the thought, OK, how can we make what what exists to
33:22
actually solve this problem, which can be a little bit more
33:25
scalable. So that's where, you know,
33:27
fortunately at that point, Open AI has launched a Triton, which
33:31
was essentially a Pythonic programming interface, right,
33:35
where you can do CUDA kernel programming at a much abstracted
33:39
layer. Most of our users would be, you
33:42
know, familiar with Python. So using that.
33:45
Now the second thing was like, you know, GPU memory is
33:47
hierarchical, right? Like you have the DRAM, then you
33:52
have the the HBM memory, then you have SRAM, the streaming
33:56
multiprocessor, SSMS and GPUs, they interact mostly with SRAM,
34:01
right, which is very little. So every time.
34:04
And so all these multiple kernels you have and multiple
34:09
operators you have in your training code, there's a lot of
34:12
data transfer happening right from CPU to to HBM to SRAM.
34:15
Like that is your biggest bottleneck, right?
34:18
Even though the GPUs can massively parallelize the amount
34:22
of IO data IO which is happening between different hierarchies of
34:25
memory, that's becoming a big bottleneck for the training
34:28
time. So that's when we thought, OK,
34:30
let's combine what trade on is brought to the table.
34:35
Let's take this problem and create certain first of all
34:39
custom kernels for our distributed training workloads.
34:42
And we started seeing huge gains, right?
34:43
Like in one of the cases, like just three, 100% increase in, in
34:46
the overall efficiency majority of the cases like more than 50%
34:50
decrease in the memory which was being used.
34:53
What we did was we started fusing kernels, right?
34:56
Like a certain operations, they can be just combined fused
35:00
together. Operators can be fused together.
35:02
You don't need to have 5 kernels to do this different, five
35:05
different operators, right. And there is some human judgment
35:07
involved in that to, to make that decision, which, and we
35:12
have also seen flash attention in the industry, sort of saw
35:14
that quite a bit, right? And, and it became very popular
35:18
and it was built on the principles of kernel fusion.
35:21
So we took that and combined that and it's, it's sold and
35:26
helped immensely. A lot of our internal developers
35:30
huge decrease in training time because memory efficiency got
35:34
much bigger. And once we realized, hey, this
35:37
looks good and, and let's just open sources.
35:40
There was not a lot of planning. There was not a lot of thought.
35:44
It was also like, hey, the all of open source models are coming
35:48
in and and maybe you know, if community likes it, they will
35:52
potentially create kernels for those open source models.
35:54
And if we end up consuming it, it eventually also benefits us,
35:57
right? Yeah, it just took off, right?
36:00
I think multiple companies were in the same position, right?
36:05
It's it's when you like go and put a problem out.
36:09
They were all, you know, getting this this big load of generative
36:12
use cases. Lot of LMSGPU crunch was only
36:16
present and almost everybody right needed to solve this
36:22
problem for distributed training right at at scale.
36:25
And and we got a solid reception right in the community right
36:29
from Andre hugging face. We one thing which we did spend
36:35
some time on making the use very easy, right.
36:37
So we integrated with the Hugging Face ecosystem right off
36:40
the back right etcetera. So and we just last week it
36:46
completed more than 1,000,000 downloads.
36:50
Right. This is amazing like we never
36:53
thought you know this will go all the way there right So and
36:57
we have got insert lots of good feedback from from the users who
37:00
are benefiting from it. Yeah, It's funny that you
37:03
mentioned the memory aspect because we had a guest on here a
37:08
few months ago talking about how his whole thesis was that we are
37:13
memory constraint, we're not GPU constraint.
37:16
GPU is almost like not the way to look at it.
37:19
It should be really looking at memory and that is the
37:22
bottleneck right now. And, and I think very directly
37:27
so right Lexi, ultimately ML is a data processing problem right
37:33
now. So that does mean that, you
37:36
know, there's a lot of data moving in between these GPU's
37:41
right now. The fastest you can process data
37:44
is if the data is in the memory right now.
37:47
You go through the hierarchical thing OK GPUHBMS, RAM.
37:52
The second thing is OK the the CPU on system memory of the CPU
37:57
on on the GPU nodes. Then you go OK, I I will go
38:01
outside the GPU nodes right. Maybe the GPU SSD is right
38:04
itself. But the more you can keep in
38:07
memory, the faster you can process more.
38:10
So for LM oriented workloads, workloads because if you look
38:16
at, you know, people want to bigger context length, right?
38:20
Even when you are training the model, right, if you increase
38:22
the context length, you potentially can train bigger
38:25
models inferencing time, right? Like the amount of information
38:30
users are sending, right for the model to process, the amount of
38:35
information the models are generating.
38:37
Like see Rexis output used to be very straightforward, right?
38:42
Like it's Rexis recommends you whether this is a good content
38:47
to show or not in case of generative AID output is also
38:51
huge, right? So a lot of that data processing
38:54
is going back and forth. And the more you can, you know,
38:57
leverage the memory effectively, the biggest gain you can get.
39:01
So it's, it's rightly so there's a lot of even during inferencing
39:05
time like KV cache is becoming a big technique, right?
39:08
Like where you can reuse KV cache as the inferencing calls
39:12
are coming right for sequence of tokens acceptor right which we
39:15
have seen before you calculated I.
39:17
Can save a lot of costs too. Yeah, yeah, you've collect,
39:20
you've already calculated attention scores for all these
39:22
different, you know, tokens you are swinging the sequences you
39:24
are seeing, right? So why do this again?
39:27
So let's keep it in memory, But how much can you keep in memory?
39:29
The other part is like, you know, if you start leveraging
39:32
GPU memory for KV caching, then model itself needs GPU memory
39:36
for its own processing. So what is that route right
39:38
boundary and, and most of the H1 hundreds, right, which is now
39:43
dominantly like if you, if you look at what the industry has
39:49
probably most in bulk is right now H1 hundreds, right?
39:52
Or to some extent H2 hundreds. And and 80 gig memory per GPU
39:58
is, is turning out to be low, right, based on the model of
40:02
data you're processing. And I think part of it is also,
40:05
as the use cases emerge, that that starts becoming clearer.
40:10
Because the way maybe the the whatever went into the decision
40:15
making was a disk memories for GPU processing whatever GPU is
40:19
computing. When people start leveraging
40:21
this memory as a cache, like carry cache like in case of GN
40:24
NS. Also, we are keeping some part
40:27
of the graph structure in memory.
40:28
When you start using it as a data storage mechanism in
40:31
addition to being used for computing needs or of of your
40:35
model itself, then it's less than you need more.
40:38
So I'm assuming you know the the that that realization is already
40:45
like if you see the Grace Hopper architecture and the Grace Black
40:49
architecture where they're combining ARM CPUs with the GPU
40:53
nodes. And then they're creating a very
40:55
high bandwidth data transfer link between that the CPU and
40:59
the GPU. So that you can then leverage
41:01
the the more memory on on the other side of the CPU nodes
41:04
because there is a very high bandwidth data transfer were
41:07
realization to this particular use case which is emerging from
41:13
generative AI, right. It's yeah.
41:17
I want to give a shout out to the person who who said that to
41:19
me because I was trying to remember their name.
41:21
It was Bernie Wu was the one who said that we are memory
41:25
constrained and you just went very deep in that and makes a
41:28
lot of sense. Everything that you're saying.
41:30
You did mention one thing that you worked on quite heavily too,
41:34
which I've also heard is in absolute pain around checkpoints
41:40
and speeding up the checkpoints because a, it's not clear when
41:45
things are going to fail. So you don't know if you need to
41:47
checkpoint every second or every 5 seconds or every 5 minutes or
41:51
every day. And so you, if you over optimize
41:55
for checkpointing all the time, then you're potentially
41:57
transferring around a ton of data.
42:00
And you don't need to be because as you mentioned, these
42:03
checkpoints are huge. And so especially when you're
42:06
training LLMS that are very, very big, the checkpoints can be
42:10
in the terabytes. And that's just like if you're
42:12
doing that every 5 seconds, that's a whole lot of data
42:16
that's going around. So how did you specifically
42:20
speed up the checkpoint and make that process better?
42:25
I think we decided like, so initially the very naive
42:31
implementation of checkpointing was, you know, so our, our
42:35
majority of our data is on HDFS, right?
42:39
And we will the, the mechanism was your checkpoint and the
42:44
checkpoint goes to SDFS and when you are reading right, you read
42:46
from there. So the realization specifically,
42:50
yes, with LLMS like a first of all these checkpoints are big
42:54
and the training pauses while you are checkpointing, right.
43:00
So, so there's a pause in the training workload and this gets
43:05
done. There's a transaction which is
43:06
happening, right? So the first thing which we did
43:09
and, and you know, change that architecture as LLMS became more
43:13
prominent is, hey, let's make it, you know, two phase
43:17
transaction, right? In essence, we will checkpoint
43:21
in memory, right? So, and, and from there onwards,
43:25
any copy of that checkpoint is a sync.
43:28
So, so two points, right? Like the checkpointing in memory
43:32
is very fast and the second thing is going on waiting for
43:36
their checkpoint to be transferred to a remote storage,
43:39
right. So it's hierarchical
43:40
checkpointing strategy which we developed and now we are, you
43:43
know, streaming that checkpoint. We also changed our our back end
43:47
for checkpointing storage from HDFS to a block based storage.
43:51
Now we are, you know, investing in, in figuring outgoing and
43:55
optimizing it even further. So GPU's if you request like
44:01
when you are ordering GPUs, NVIDIA GPUs, you can request SSD
44:05
storage within the GPUs, right? And there are technologies which
44:12
I'm assuming you know some of the companies may have in house,
44:16
but even in open source, right, where you can then start looking
44:19
at creating a cache or distributed cache onto the
44:23
GPUSSD. So maybe 1GPU SSD storage is not
44:26
enough, right? It's for your needs.
44:30
Then you can combine the GPU, SSD storage across all your GPUs
44:35
and then create a distributed cache to store this, right.
44:37
So that's the second phase we are on in terms of ensuring so
44:41
that even and and the advantage with that will be in the
44:44
restore. I think checkpointing will still
44:46
be fast because it's in memory. It's then we'll go to the
44:49
GPUSSDS. But when you are reading back,
44:52
if you don't have to take it outside of the GPU network and
44:55
bring it back into the GPU network, you save tons while
44:58
restoring. And when reading, the other
45:00
thing which we did was like to your .1 of the questions was how
45:03
often do you checkpoint, right? I mean, that's a big question.
45:06
And, and we have mostly sort of, you know, left it to the
45:11
modelers like to take that common sense decision.
45:15
How soon should you checkpoint right?
45:18
How much of like for example, for jobs which take like more
45:21
than a week to train, I think they can have a hey, how much
45:25
I'm willing to lose? I'm willing to lose 5 hours
45:28
worth of data or am I willing to lose if if something goes wrong
45:31
and I have to restart, can I restart from previous days
45:34
checkpoint? So it's it's a decision making
45:36
right heuristic in certain use cases, like very deterministic
45:41
use cases when you're doing incremental training like you
45:43
need to train every few hours, then it is very clear like hey,
45:47
it's driven by checkpoints, right.
45:49
So you have to do a checkpoint, you know, every two hours,
45:52
etcetera. So depending on the use cases,
45:55
the other thing which we have done is like disruption
45:58
readiness. So if in cases of planned
46:02
maintenance, right, where we know that the nodes are going to
46:06
go down, right, we trigger a checkpoint, right?
46:09
So it's a trigger checkpoint. And if the modulus have
46:12
implemented checkpointing restore, right, that trigger
46:15
will go in Walker checkpointing restore.
46:18
So this is not a plan. This is not it's happening
46:20
before we take that workload of those nodes, right?
46:25
So essentially this is signal from the underlying infra that
46:30
hey, these nodes are going away, I'm triggering a notification,
46:34
this notification and this is all automated right now.
46:36
We'll go to the the model running model that will be
46:39
checkpointed moved and then, you know, we will put it back.
46:43
And then we are invested in things like priority queuing.
46:45
So if things which are being disrupted, they should be moved
46:50
to the front of the queue like a Disney front of the line pass
46:54
and rescheduled like as fast as possible, right?
46:56
So you need to invest in those mechanisms to make sure that,
46:59
you know, that goes on smoothly. Yeah.
47:03
I like the idea of first of all, just getting the checkpoint
47:07
offloaded as quickly as possible so you can get back to training
47:11
and then do what you want with the checkpoint, but don't have
47:14
everything stopped because you're trying to offload the
47:18
checkpoint. So like make that as quick as
47:21
possible. I think I've heard that before
47:22
as as a design pattern and then also make sure that there's no
47:27
surprises for the model that is training.
47:29
And then it's like, hey, what happened?
47:30
I thought I had all these resources and it all went
47:33
offline. So you make sure that these
47:37
things are, that they feel kind of standard, but it's like
47:42
unless you think through it or you have a bad experience, I
47:45
imagine you probably had an experience where you realized,
47:48
wow, we should probably do something about that because
47:51
there was a whole training job that just kind of went to
47:55
nothing. And we could probably provision
47:58
or at least put a bit more anticipation in place for that.
48:05
One thing that I was thinking about is how we've been
48:09
centering this conversation a lot on the new paradigm of LLMS
48:14
and agents, but LinkedIn still has a ton of traditional ML use
48:20
cases, right? So how do you think about
48:23
bridging the gap or creating a platform that can service both
48:29
the LLM, the agent use cases and then also the traditional ML use
48:36
cases? I think it's, it's happening.
48:42
So certain things are common, right?
48:45
Like obviously all the investments we are doing in the
48:50
GPU infrastructure, GPU monitoring, observability,
48:53
resiliency, and you know, building things like distributed
48:59
checkpointing, restore automated disruption readiness, then you
49:02
know, it, it goes, then you start going into like, so one of
49:07
the things which I did once I came here was, you know, we too,
49:12
we took a major decision to rehaul our machine learning
49:15
training pipelines. The pipelines or the
49:19
orchestration engine was built very much in our solid
49:24
prescriptive Bay, right? Like it, it was multiple
49:28
components which we had in our training pipeline.
49:31
It was based on Tensorflow and, and sort of, you know, it's a
49:35
similar paradigm which DFX follows Tensorflow extended.
49:38
You look at from Google and then, you know, every single
49:41
step is, is sending metadata to the subsystem.
49:44
So very rigid. Now, I think that realization
49:48
was very early on here this and, and part of this was also, you
49:54
know, there is a lot of traditional feature engineering
49:56
which happens with the traditional Nexus models, right?
50:00
As opposed to LLMS which don't have the notion of feature
50:04
engineering, you are feeding, you know, bubbles of text.
50:07
So, so the feature engineering as a discipline is, is sort of
50:12
disappearing in, in, in this new world.
50:16
So, and, and there is a lot of new, you know, rag based
50:19
architectures which are bringing up right, you're doing more in
50:21
your time. So the machine learning
50:24
pipelines engine, we, we rewrote and we redesigned on an open
50:29
source orchestration engine called flight, because the
50:32
notion was that experiments will be very heavy right in, in, in
50:39
this new world. And, and so, you know, there is
50:41
a lot of quite changing of code. I want to do very short fine
50:44
tuning jobs or sometimes very long running jobs, but the user
50:48
experience has to be, you know, very nifty.
50:51
We launch something called interactive dev where you can,
50:55
you know how VS Code connect to a remote running job, right?
51:00
And put debug pointers and trace through what is going wrong
51:05
because otherwise the previous experience used to be like, you
51:07
know, you don't know what has gone wrong.
51:09
So once it has gone wrong, you need to rebuild, recompile, re
51:12
upload. So this this whole new modern
51:16
pipeline training architecture which is very much focused on
51:19
very quick changes, very much experiments.
51:21
You don't need to re upload. Go directly debug there.
51:23
See what values are being passed in real time through your debug
51:27
pointers. Go 1 by 1 and then, you know,
51:30
once we standardized LLMS on it, we are migrating a lot of our
51:35
traditional Rexis models onto this new platform as well.
51:40
And the feedback from users is, is really great.
51:43
Like they're all liking this, a little bit less rigid, very, you
51:48
know, experimentation oriented architecture where you know,
51:52
the, the back and forth things which they need to change,
51:55
right, to change their model and train their model is, is, and
51:59
there is a very inbuilt robust versioning mechanism.
52:02
One thing which was missing in our previous versions of machine
52:05
learning pipelines was you have a config, you have a model, you
52:10
have a data set, all these are different version things, right?
52:13
And you do an experiment, then you have to go and look at every
52:17
different entity what was correlated with this particular
52:20
experiment or training run. So in in this new machine
52:23
learning pipeline, right, there is a very, very robust
52:25
versioning, right? Like in, in terms of all the
52:27
entities, your configuration parameters, your pipeline, your
52:30
model, as well as you know what data version these are all
52:33
bundled together, right? So every experiment is
52:36
associated with the right lineage, like, OK, this was all
52:38
the things which were used to create this particular model,
52:41
right? So versioning is is very robust
52:44
because it's sort of built with that assumption.
52:46
You will experiment a lot, right?
52:49
So that's one change. I think one thing which
52:53
potentially and again it's early, early days of the what
52:58
will change in REXIS with service.
53:02
You don't want the non LM architecture.
53:04
I think the LM architecture itself like 1 hypothesis is,
53:06
hey, the LM architecture itself will take place in REXIS, right.
53:10
So then OK, whatever you have done at upper layers, right for
53:14
or you're doing for LMS like vector DBS embedding.
53:17
So they become the prominent mechanism over all the rag
53:20
becomes a prominent mechanisms, right.
53:22
So your traditional feature engineering pipelines, the way
53:25
you were constructing, they will change, you know, the rear time
53:28
feature pipelines, not the offline, right.
53:29
They will change to to accommodate to this new
53:31
paradigm. Tools like Lang chain, Lang
53:33
graph, which are very much dominant in in the LLM space.
53:37
They may become prominent in that space as well.
53:41
So far we haven't gone that far. I don't think we have looked at
53:45
hey, can the holder the land chain or land graph is being
53:49
used right for elements, whether you know prescriptive
53:53
orchestration graphs when the models are inferencing in real
53:56
time or non prescriptive agentic graphs which are formed like
54:00
where the agent decides what part to travels.
54:02
Can they be used in in the the traditional space where you know
54:07
there is a lot of feature ingestion happening there
54:09
featuring engineering pipelines which you have built, But I
54:13
think the rate is on to see itself.
54:18
I think the thing which will simplify this can LLMS solve
54:21
that part of the problem Then you can take the LLM
54:23
infrastructure as is even at the upper layers and start
54:26
replicating it for access. Until then, I think it's in fact
54:29
majority of our workloads are still Rexis workloads are not
54:33
powered by LLMS, right. And and rightly so, they are
54:36
very targeted, very effective in what they do.
54:40
And so it's the first hypothesis has to be at the model layer
54:44
itself. But at lower layers, we are
54:46
definitely ensuring that there is standardization across like
54:49
so it's now the same machine learning pipelines engine which
54:52
is being used all across LinkedIn, whether it's Forex,
54:55
Rexis or for LLM workloads, right.
54:57
So we have standardized all that layer, the topmost layer when
55:00
you, you know, move towards inferencing and all some of
55:03
those changes, you know, we'll figure out as we go.
55:06
Yeah. Yeah.
55:07
Where, where exactly does it divert?
55:09
Because it is fascinating to think about how the lower levels
55:15
you've recognized there, you can share resources or you can have
55:19
the same type of stuff going on, but at some point it's diverting
55:24
right now. So where where is that point if
55:28
you know where it might be case by case basis?
55:31
I think see first thing is so, so you can look at every single
55:35
layer, right? Hey, do we need so for example,
55:37
when you are going to inference, right?
55:42
Do you, so people have either written their own inferencing
55:46
engines right in different languages or you have
55:48
standardized on something like TF serving from open source
55:51
Tensor Rd. from NVIDIA, which were very heavily optimized for
55:55
models right in the in the Rexis space, when you look at the LLM
56:01
space, there is VLLM, there is TGIE, there is SG Lang, right?
56:05
The inference engine itself, there are custom inference
56:08
engines being written for LLMS. You just even look at the
56:11
hardware architecture we didn't talk about, right.
56:13
There are a lot of custom chips which are coming like you, you
56:17
see many startups. Grog is another example which
56:20
are just developing custom hardware.
56:24
Yeah, yeah, for. Sure.
56:25
So, so it's, it's if for people who are investing and taking a
56:30
bet that, hey, it's much simpler for them because they can just
56:34
standardized on one that the one of the reasons right, where
56:37
where you may feel, Hey, NVIDIA GPUs could be better for LLMS
56:42
might be if you write what it says, you know, people who are
56:44
investing in custom ASICS, custom chips, which are
56:47
potentially right, better for elements in terms of
56:52
inferencing, etcetera. Is is the better on the
56:54
architecture of the transformer architecture is going to replace
56:57
every single thing, then you can have custom chips.
56:59
So right, there is some diversion.
57:00
Some companies are just going with very custom chips, right
57:03
for elements. So the engine is different.
57:06
The chips are different, right? Like if you build something
57:09
general purpose right now, we are we are all, you know,
57:12
standardized. The hardware layer is is there
57:15
is no specific thing apart from the kernels work, right, which
57:19
we have been doing which has been pretty specific.
57:22
Then you also look at the the the feature processing infra
57:27
right. So a lot of real time feature
57:30
processing, feature ingestion in case the kind of tooling which
57:36
has emerged in the LLM word, it's you are talking, you know,
57:40
Lanting a land graph vector DBS are prominent, right.
57:45
It's not the traditional feature processing, right?
57:47
And and embeddings have become the dominant landscape.
57:50
And I think we have been on a journey where you know, we are
57:53
using embeddings all across as well started heavily bought for
57:58
access as well as LMS. But some of the tooling chain there is
58:01
different, right, What you will use for something like Beam for
58:06
example, you know for feature processing in the past now you
58:09
know a lot of that is happening in the LLM word through the Lang
58:13
chain and the Lang grass like you know when you are actually
58:15
orchestrating the data. So those are the divergences
58:17
which are happening and and are there for the right reasons.
58:21
I think you know, over a period of time it will reconcile.
58:24
But yes, that that layer is different when you are handling
58:28
traditional Rexes versus this, the inferencing engines are
58:30
different when you are handling there.
58:32
In some cases people are even taking different chips, hardware
58:35
chips for these layers. So I think it's bottoms up.
58:40
We have to push, we have to make sure, you know, continue
58:43
standardizing on bottom layers, right and go up the stack and
58:45
see, you know what else you know, as as we become more
58:48
mature, what else can be standardized and can be the
58:51
same? If LLM start solving Rexes then
58:56
potentially maybe you know that itself solves a lot of this
58:59
problem.