Latest / Agentic Conversations (formally mlops.community) / PyTorch's Combined Effort in Large Model Optimization // Michael Gschwind // #274

Transcript

PyTorch's Combined Effort in Large Model Optimization // Michael Gschwind // #274 · Agentic Conversations (formally mlops.community)

0:00
My name is Michael Schwimm. I'm an engineer on the Pie Torch
0:03
team. I help create Torch Chat, which
0:07
is the Pie Torch library for LLM Large Language Model inference,
0:14
and my coffee is usually a latte.
0:18
I take it with a double espresso.
0:21
I feel honoured that I got to talk to Michael today.
0:25
This is another episode of the ML OPS Community Podcast.
0:28
I'm your host, Dimitri OS, and there's one of those times where
0:32
I almost have to pinch myself. This man has contributed so much
0:37
and maybe he has been behind the scenes or if you have been in
0:42
the ecosystem for a few years, you probably know who he is and
0:47
that's for a good reason. The dude has done a lot and
0:51
recently with his product or project Torch chat, he talked
0:59
about how he was able to combined all of these different
1:04
projects that he's worked on while at Meta in the Pie Torch
1:07
ecosystem. And really when you talk to him,
1:10
you can tell that he is all about that optimization.
1:14
Another thing that became blaringly, blazingly obvious as
1:19
I spoke with him is he gives a lot of people a lot of credit
1:26
for everything. He recognises that proverb, if
1:31
you want to go fast, go alone. If you want to go far, go
1:35
together. He basically embodies that and I
1:39
can tell you that because one of the first things that he told me
1:41
was, OK, I want to give a shout out to that team, but I don't
1:44
want to talk about anything that team is doing because I want
1:47
them to come on here and talk about it themselves.
1:49
I don't want to act like I can take credit for that.
1:52
So, you know, you've got a gem of a person when that's what
1:56
they're telling you. Let's get into it with Michael.
1:58
And as always, if you enjoy this episode, share it with one
2:03
friend. I'll see you on the other side.
2:11
Yeah, I've been around the block for a couple of times.
2:13
I built what arguably the first accelerator that was in the cell
2:19
PlayStation 3. So if you play PlayStation
2:23
three, you used my chip. If you used the Xbox three, six,
2:28
yeah, worked on that one too. So everyone like.
2:33
Myself who enjoyed Tony Hawk growing up, It was thanks to
2:37
you. She is it.
2:39
That sounds exciting. Yeah.
2:42
And I think there is a path that LED from there.
2:46
It was terribly hard to programme initially the Gold
2:50
West, but all the performance that you can get into one chair
2:56
and to do that we sort of had to make compromises.
3:04
It's not, you know, the comfort of a traditional CPU where you
3:10
know you have branch prediction and all those caches, whatever.
3:14
It was much more like, OK, you want the pure performance.
3:17
We're going to give you as much performance as possible.
3:21
But it means there is nothing else, no comfort on this chip.
3:24
It's all on you to manage that. And so users did find that
3:30
somewhat hard to programme to say the least.
3:33
And I ended up spending a good part of my time building
3:39
software, environments, ecosystems, compilers, and that
3:45
straight up led to AI, which turned out to be the most
3:51
voracious, most consumer performance and in the end
3:58
accelerators, GPUs, etcetera. So that was really cool from
4:03
enabling accelerators for a broad range of applications
4:08
within HRB. Accelerators power the first
4:11
petaflop computer, the first computer delivering petaflop per
4:17
second. And there is a direct connection
4:23
from there to the use of accelerators in AI today.
4:28
They are the largest consumer today of compute performance,
4:34
and they have a particular affinity to the sorts of things
4:43
that work well on accelerators, large matrices, very repetitive
4:50
operations, but many of them similar to what you have for
4:55
graphics processing, where you want to process a lot of pixels
5:00
at the same time. Here you're processing a lot of
5:03
neurons, a lot of layers at the same time with the same general
5:09
operations, matrix multiplies in particular.
5:12
And that was a match made in heaven.
5:16
And so, yeah, the history of accelerators in many ways has
5:23
become, you know, enabler to AII.
5:27
Think Jeff Hinton and his team were the first who saw that
5:33
potential when they trained models for the image net
5:38
competition and beat the traditional image understanding
5:46
entrance in the image net competition back in 2012.
5:51
How time flies. And were you, because you were
5:55
working on the PlayStation 3 and then the Xbox and mainly in the
5:59
gaming world I guess, or the chips side of the gaming world?
6:04
And how did the transition go into AI?
6:08
Was it fairly straightforward where you recognised OK, this is
6:13
same chip, it's just different application?
6:18
So the story is a little bit longer.
6:22
After Cell and the game computers, I spent a
6:30
considerable amount of time on supercomputers.
6:33
So I built three of the world's fastest supercomputers.
6:42
That was the Roadrunner, the first petaflop supercomputer,
6:48
then Blue Chain, and at the end the Summit system, which was
6:55
using GPU's to get about the IB Miss Power architecture to power
7:03
National Labs applications. And it's from there that I was
7:11
looking around, hey, what's the next cool thing you could do
7:15
with this technology? We have the Summit style
7:19
supercomputers where we connect GPU's with CPU's in a very tight
7:30
collaborative fashion. This was pre Grace Hopper CPU
7:38
GPU that NVIDIA later released. We have been working with NVIDIA
7:44
on printing the NVIDIA GPUs to to the IDM systems.
7:50
And so as I was looking around back then over 10 years ago,
7:56
there were these image Med results that Jeff Hinton had and
8:01
I was like we need to try that. And so we built a system to
8:08
train what were CNNS that in the Bay, Alexnet in particular.
8:15
And we trained the first Alexnet model in under an hour.
8:23
And so that was like, hey, this is this is working.
8:27
This is a really good direction. Of course, Facebook beat us
8:33
shortly after that, which you know, as I mentioned,
8:36
competition is good in our industry.
8:38
You want to see that because somebody comes and says like
8:42
what you're doing is cool, but I have these ideas how we can do
8:46
that even better. So I think this collaborative
8:50
competitive framework is, is a great way of bringing people and
9:01
harnessing their ideas, allowing them to build on the ideas of
9:06
others. And I wish I have to say
9:08
research is a team sport. So there, there is a lot of
9:15
wisdom in teams, in collaboration where you might
9:19
not have all the answers, but you have a part of the answer
9:22
and some of your colleagues or competitors for that matter, may
9:28
have the other part of that. And overall, you know, by having
9:33
that corporation competition, whatever you want to call it, we
9:39
get an overall better outcome. And I think that was goodness
9:46
and super exciting to see. So I was really thrilled when I
9:53
joined Facebook to have the opportunity to work with some of
9:59
those people that, you know, had built those exciting training
10:07
systems. And Pythorch was really just
10:13
coming into its own. And I think Pythorch's
10:17
commitment to open source to to a community based model has
10:22
served it super well is the foundation a lot of innovations,
10:31
improvements that people brought to that ecosystem.
10:40
Having that openness means you can pursue your own ideas, you
10:45
can contribute, but you don't have that bank cost of where you
10:51
first have to build the framework before you can prove
10:55
that, you know, I have a better whatever it is, a matrix
11:00
multiply kernel or a better way of doing quantization or
11:03
whatever. You can build on the work of
11:07
other you know and focus on your own interests and your own ideas
11:17
and improvements a while relying on the community of other uses
11:24
of other contributors to take care of the things that are
11:29
important but maybe not your core competence.
11:34
So talk to me a little bit about Torch Chat now, what you've been
11:37
working on recently and the three goals as to why you
11:43
created it. So, OK, Torch Chat is in many
11:49
ways the combination of several development strands or if you
11:57
will, in the Pythorge ecosystem, in the LLM ecosystem, there was
12:05
the optimization of Pythorge for LLM.
12:10
So we've done a lot of investment over the years to
12:14
optimise high Torch for LLMS. There was better transformer,
12:19
the initial acceleration for LLM inference, then the accelerated
12:26
Transformers that brought in flash attention that brought the
12:31
new scaled up product detention operator, many other
12:35
infrastructure improvements. There was the Torch compile work
12:45
that allows you to take those models that you have in pythorge
12:50
and export them such that you can run them stand alone outside
12:55
of Python hosted environment like a normal application.
13:02
And then there was the executorge on device work that
13:09
brought model inference, LLM inference with LLM optimizations
13:17
to on device AI. And so one of the thoughts here
13:23
or one of the goals in building Torch Check was to integrate all
13:31
of these different development strains, these different silos
13:38
of optimization, both to show users how these different
13:46
technologies can be used, but also to integrate them, how you
13:50
can integrate them. And to build that end to end
13:54
solution that provides a seamless environment to deploy
14:02
LLMS for inference from servers on the one side all the way to
14:08
on the by applications at the other end of the spectrum.
14:14
So that was the first underlying impetus that was both enabled by
14:24
these Pythorch developments. Then Korcheck is also a driver
14:36
for ongoing optimization for LLMS. In pipe words, in exactly
14:41
court, just by having that integrated environment, you can
14:46
now benchmark. We know in engineering that if
14:51
you benchmark something, if you measured regularly, it gets
14:55
better not by magic, but by some engineer looking at this, this
15:01
is not good. We can do this way better.
15:04
You know, if if you don't have those numbers out there that
15:09
give somebody the hey, I think I can do this twice as fast yes,
15:17
please come send us your PR. That's exactly what we want,
15:21
right. So giving giving the platform
15:25
where people can measure where people can try out their own
15:28
ideas and measure them and and sort of almost like a dialogue
15:33
with the community. Now, what can I do here to
15:38
contribute to bring my ideas to the table?
15:45
And it's been an important driver for Pythorch
15:52
optimizations. We found kernels.
15:54
We found the opportunities for, hey, this should do better by
15:59
comparing with other inference solutions, by comparing between,
16:05
say, the math and a Linux platform.
16:12
ARM code generation hadn't been until very recently a big focus.
16:19
And so with Torch Chat running on the Mac, people started to
16:27
ask questions. Hey, why is this performance the
16:30
way it is? We thought, you know, based on
16:34
the rating of this machine, you'd get better performance.
16:38
And so we looked at again, what what gets measured gets
16:43
improved. We looked at the code, we
16:46
partnered with the Metazone LLBM team, Metazone compiler teams,
16:53
like what can we do to generate better code for Arm?
16:57
We partnered with ARM, we partnered with Apple to bring
17:04
better kernels. A lot of the performance comes
17:07
from the BLAST kernels, gem, generalised matrix multiply
17:13
kernels. And so getting the best kernels,
17:20
partnering with the community that is most intimately aware of
17:30
what improvements can you do for that particular platform.
17:34
And again, bringing that into the common ecosystem gave us
17:43
gives the community a better outcome by by bringing another
17:50
set of eyes to the table, another set of ideas into the
17:58
ecosystem. And it's not just porch chat.
18:01
It becomes, and I would love for porch chat to be this catalyst
18:07
in some ways that people can look at it like what are they
18:12
doing to get performance, whatever.
18:16
And for people to blatantly plagiarise it for their own
18:20
application. That's what open source is good
18:23
for. Hey I saw this idea.
18:26
I may not be doing LLM inference but this is a cool idea which I
18:30
can use for my own application over here.
18:33
So there is some of that were just aggregating all of these
18:40
that libraries ideas, ways to structure kernel to do things
18:47
better. Just provides a reference for
18:52
for others. I'm a really big fan of this.
18:55
It's to summarise, you were able to leverage many different veins
19:01
of the pie torch ecosystem that was already there to create a
19:04
more complete product. We could say in a vague sense, I
19:09
don't know if product is the right word, but to create
19:13
something new that is leaning on or standing on the shoulders of
19:17
giants. And then you recognised, you
19:21
know what would be really good, As if we can benchmark these,
19:25
but not just benchmark them for ourselves, benchmark the
19:27
performance and let everyone see because we want to know and we
19:33
want the numbers to continuously get better.
19:35
And I think one thing that I really appreciate about what you
19:41
just said is the competitiveness.
19:45
It's almost like, you know that whenever you see a number and
19:49
you think, how could this be faster?
19:53
You, you get that itch. And so there's going to be other
19:56
people that are going to be out there and they're going to get
19:58
that itch. And if you throw it out and make
20:00
it public, the numbers are only going to get faster.
20:05
Exactly. I think that's what good
20:07
engineering is about, looking at a solution, asking, well this is
20:13
great. Well, what's the next thing we
20:16
can do with that? Either how can we use it and
20:20
give them more value by using it in a different space, or how can
20:24
we take it in this space and make it more performance, lower
20:28
power and more reliable. There are so many attributes
20:32
that people care about where you can't imagine improvements.
20:38
And hey, I would like to run it without Python because Python is
20:42
cool, but it's also a very large ecosystem, so it's not going to
20:49
run on my small own device. We've been running Torch Chat on
20:55
a Raspberry Pi, which I thought was one of the coolest things
21:01
ever. That's the other thing in
21:03
engineering. Every time you achieve something
21:06
you have like the next goal. So I think the next goal for
21:11
Twitch chat should be to run a model on an Apple Watch or
21:16
whatever on the mark watch. Like natively.
21:19
Not just talk to the Internet and pull down the answer from a
21:23
server, but run it on the watch. Or device.
21:27
Wow. Yeah, it it's fascinating you
21:30
say that too, because we've had conversations in the ML OPS
21:33
community Slack where folks are saying I need to optimise
21:37
battery. I need to know how I can put
21:40
this model on device and make sure that it just doesn't kill
21:45
the battery right away. So there is, like you say, so
21:49
many different vectors that you want to optimise for.
21:52
And it's really cool when you can give that to the community
21:55
and each different use case is going to have their vector that
22:00
they're trying to optimise for. And so bind you're able to get
22:06
to places that you probably wouldn't be able to get to if it
22:09
was just one team trying to work on their one use case.
22:13
And I, I'm wondering as I talked to you, what are some things
22:18
that surprised you that gave you just an enormous amount of lift
22:23
as you are trying to optimise that either you created or and
22:29
was because of stuff that you and your team has done or just
22:33
the community has done and it surprised you because you
22:37
wouldn't have thought to try and go in that direction.
22:41
A big shout out to the developers of X&N Pack and to
22:47
our own team members that are a part of X&N Pack development.
22:55
One of the engineers on our team developed a really cool 4 bit
23:06
matrix multiply forearm that uses a instruction that only
23:14
exists in arm that does like a vector of four bit by 4 bit
23:23
multiplications and then accumulates.
23:26
It all integrated into a quantization kernel, but scaling
23:32
off, but GroupWise quantization but and and the scaling gave a
23:40
amazing performance that made CPU inference of lens in
23:49
particular for on device scream in ways that were just amazing.
23:59
It's another example where if you integrate different
24:04
technologies or different dimensions, there's the
24:07
instruction set architectural farm, there is the XNM pack
24:13
architecture, there is the specific kernel, and that's
24:18
integrated with the exact torch and comes the torch jet.
24:22
Just if you look at the steps of how many communities and experts
24:28
were necessary, it's hard to imagine that you can build that
24:34
in a clean room without these amazing community that all bring
24:42
their own ideas to the table. 100% in agreement with
24:49
everything that you just said. And Speaking of which, you've
24:55
built products that go on different ships and interface
25:01
with different ships at the a very low level.
25:05
Is there tips or tricks that you could give us?
25:09
Because I, if I was talking to a friend who is working on these
25:15
very problems and he was saying, man, it's just so hard because
25:19
GPU's aren't written in CUDA. It's like if I need something to
25:24
happen with the GPU or if I really want to figure out how to
25:27
squeeze the most out of it. Like I, I got to read that
25:30
manual. And sometimes you just got to
25:33
know somebody at NVIDIA to be able to get that little.
25:37
I don't want to say trade secret, but you, you want to
25:40
like figure out or sometimes even it's like petitioning or
25:44
lobbying so that something can get changed in a GPU.
25:48
And I wonder if with all your experience you found there are
25:53
easy ways to do things. There are hard ways to do
25:55
things. There are tips and tricks that
25:57
can help us as we're we're trying to squeeze the most that
26:00
we can out of the juice. I think the number one matching
26:05
is collaboration. There are a number of open
26:12
source solutions and teams all around the space, and so
26:22
deciding to go something completely alone will make it
26:30
difficult to engage the right people that have the right idea.
26:36
But you're missing this one thing and you don't know where
26:39
to get it. Well, somebody else may have
26:43
that and I'll give you a couple of examples.
26:46
There is the Cutlass team at NVIDIA and they're producing a
26:53
amazing library for doing numerically intensive processing
27:01
on GPU's and that's been an amazing relationship.
27:09
A lot of capabilities we get from Cutlass.
27:15
Conversely, we have partnered and contributed optimizations
27:20
and these guys work at NVIDIA. So if you have your kernel and
27:25
you contributed to, you know, that library A, you make it
27:32
available to many more users that might benefit from it, but
27:39
also might improve it because they have, you know, insights or
27:44
ideas that build on top of your own solution.
27:51
And with Catlett being linked to NVIDIA teams that think about
28:01
how to optimise the architecture, the GPU
28:06
architecture and the library architecture, you're much more
28:11
likely to find that person that have either the ability to say,
28:21
hey, there's this little known approach that you know, if I
28:26
integrated here improves performance significantly, or
28:31
hey, this is really good. But if we optimise this or that
28:37
in our next chair, then we could build something that's even more
28:43
amazing. So now through these
28:46
collaborating communities, you get this continuum where they're
28:52
the chip manufacturers, the vendors, whether that's NVIDIA
28:57
or ARM or Qualcomm there or Apple on the one hand that build
29:05
the infrastructure on which you're running all the way to
29:10
the software end users. And there is the opportunity
29:15
really to do end to end optimizations that way without
29:21
needing to own or understand the entire stack because frankly
29:28
it's it's very unlikely that you're going to find an LLM
29:31
expert who also happens to do chip design.
29:36
The other thing that I wanted to talk to you about is this idea
29:40
of the cloud versus on device. And I know you have some strong
29:47
thoughts about that and how it's almost like this false dichotomy
29:52
or false paradigm, In the past maybe it was more valid, but now
29:56
because of advancements, it is more realistic to be able to do
30:03
what you want to do on device on device.
30:07
There are many reasons why on device is exciting and on the
30:16
brink off a breakout if it hasn't already broken out.
30:22
On the one hand, there is, well, I want to run in a disconnected
30:26
environment. Whether it's I want to use the
30:29
LLM as a translator and I'm in a place where I don't have network
30:34
access but I still need translations or running in a car
30:40
and you know, I don't have to write connectivity or I just
30:44
care about privacy. Get some explanations about
30:47
something or other that involves data that are private and I
30:54
don't want to upload them into cloud.
30:57
There's there's a range of applications.
31:00
So in the path on device, I have been seen as this special thing
31:07
on the side where you have to develop a model that are
31:15
specifically for on device. And that meant that the on
31:22
device models were, you know, a small set of models like for
31:28
keyboard prediction or gesture recognition and sort of very
31:36
niche. Well, you need that on device,
31:40
but it it wasn't benefiting from the broader research, the
31:48
broader investment into AI model because they were always the
31:55
thing on the side. With the release of Executorge
32:02
and in particular also including LLM optimizations for Executorge
32:08
and Executorge is by purchase back end for on device embedded
32:15
edge type applications, it became practical to take models
32:24
that are channel purpose models and run them on device.
32:31
Then Twitch chat took that and created this continuum where
32:36
it's actually the same software infrastructure that you can use
32:40
to run and export and exercise models, optimise model from
32:49
server all the way to on device. So there's a consistent model
32:54
definition that will run on either servers or on device.
33:00
So you can use the same models and try it out on server.
33:07
Then ask the question, so how does this run on device?
33:10
There will be evidently some sort of different goal like you
33:17
will possibly want to deploy a larger model on the server.
33:22
On device, you're more memory limited, you're compute
33:25
constrained, but so Llama 7 billion or Llama 1 billion
33:31
versus Llama 70 or 405 billion, but it's the same definition.
33:38
And if you look at some of the challenges that you're facing
33:44
from server to on device, they're in fact very similar.
33:49
For example, on device, one of the big challenges is memory
33:54
footprint, also compute power. And one of the answers to that
33:59
is quantization, taking the 32 bit or 16 bit floating point
34:05
values and asking the question, well, how many bit can I get
34:10
away with? And I mentioned earlier, we have
34:13
this streamer 4 bit kernel for ARM processors that we use for
34:21
embedded and that comes that. So that that's called
34:27
quantization. That quantization allows to get
34:31
really amazing performance on, on device.
34:35
But the same approach also applies to servers, to GPU's.
34:41
GPU's are also memory limited. They have a fixed device memory
34:47
size on the card and so you're facing some of the same
34:54
constraints, only with slightly bigger numbers.
34:58
Plus on GPU's you also find, like on all other platforms,
35:03
that your bandwidth limited. So quantization helps you not
35:08
just with your footprint, but also just taking advantage of
35:14
the bus, which has a limited amount of pins, you know, a
35:19
limited number of bits that can be simultaneously transmitted.
35:24
And so if you need to transmit more data, it takes a longer
35:29
time because you send more of these packets.
35:37
So if you look at that, you're, you're basically looking for the
35:41
same sort of basic technologies like quantization.
35:46
And Torch Chat uses a library called Torchio architecture
35:55
optimization that one of the pie church teams is developing and
36:01
that takes a look at your model and finds ways to compress the
36:09
computation, to compress the weights make computation more
36:14
efficient. And that applies equally well on
36:18
the one side hand to servers where you care about the inverts
36:22
or the amount of memory weights that you can store on a single
36:29
GPU, because the largest models today are multi GPU inference
36:33
because you can't fit all the weights on a single device
36:38
anymore. So you're talking fundamentally
36:41
about the same constraints that you have with on device.
36:45
Yeah. And so integrating that for LLMS
36:50
and Torch Chat and more broadly for AI with libraries such as
36:55
Torch L really allows different communities or different use
37:00
cases to benefit from that shared investment.
37:05
That's an important opportunity. And I think with the Torch Chat
37:13
library in particular, but more generally, but the Executorage
37:19
and Pi Torch, creating this community that has a much
37:27
broader reach enables models that used to be either just, you
37:33
know, for this one domain or that other domain to cross
37:38
domains. And will it benefit all the
37:40
users? There will be differences, for
37:43
example, in size, if you're going on device, you might pick
37:47
a smaller Llama model than if you're running on a bigger
37:52
server. But it's the same technologies
37:56
and the same primitives and optimizations that you can share
38:03
between users and get the benefit of shared investment.
38:09
So users do not have to reinvent or.
38:14
Recode. The same optimizations, just for
38:18
different domains. Well, this was the first time
38:24
that I've heard someone put it like that.
38:26
Where on device and GPUs or accelerators you're still, you
38:33
don't have to think of them as two separate things.
38:36
It's just one has bigger numbers than the other.
38:38
And so don't get into the trap of thinking that you need to
38:43
have this one with the bigger numbers in order to do what you
38:47
want to do. And almost the idea of
38:50
standardising it across all of the places that you want to run
38:55
your model is a fascinating way of of abstracting the device and
39:05
just recognising what am I trying to do and what are my
39:09
constraints. And then you can talk to the
39:13
device experts and have a conversation.
39:16
What else can we do to optimise for that target?
39:19
There is for example, the GPU mode community that is very
39:27
focused on optimizations you might do for GPU's and some of
39:35
the engineers that are very active on that are engineers
39:38
that work on the Torch AO, the architecture optimization
39:44
libraries. So you get that benefit there.
39:48
But we also have for example, engineers that worry and work on
39:54
on device AI contribute to that library and partnering with ARM
39:59
and embedded manufacturers. So having that continuum is
40:05
good. And then once you have that and
40:09
the same primitives, you can ask, well, what's the best
40:12
implementation for my device? Or are there additional things I
40:17
can do for my device that are very specific here?
40:21
To what this device offers me. Yeah, exactly.
40:27
It's, it's almost like you don't want to, as you were mentioning
40:31
before, have to reinvent the wheel or do things very custom
40:38
for each device that you're on. You want to get 80% of the way
40:42
there or 90% of the way there, and then make those final tweaks
40:46
to try and optimise for whatever device you're on if you need it,
40:50
because some of the time just out-of-the-box will probably be
40:54
good enough all. Right, Especially if the
41:00
stakeholder, the developers of a particular device make sure is
41:07
ARM, is NVIDIA, if increasingly AMD are working, but the the
41:15
Pytharch community, but other communities to optimise or
41:22
contribute their specific functions, their specific
41:28
optimization. So it becomes transparent where
41:34
the ecosystem basically depending on where it runs can
41:40
pick the best implementation to solve your problems.
41:46
There's just one linear operator and pipe porch, but under the
41:51
hood there are different libraries, from kublas for GPU's
41:59
to composable kernels on AMD to ARM library, etcetera to X&M
42:08
pack. Just providing implementations
42:17
that the Pytorch ecosystem and Torch Chat can rely to to to get
42:24
the past performance on a particular device.
42:29
There's one thing that I believe everyone who has ever used
42:33
Pytorch is in strong agreement with, and that is the developer
42:38
experience is strong. And thinking about it in that
42:42
way that you're talking about is one of those reasons where you
42:48
recognise that, OK, the standard way or the the way that a
42:54
developer is going to think is along these lines.
42:59
So let's try and make it as easy as possible for them to go about
43:03
it and not add unintentionally or intentionally friction to the
43:09
workflow. And so having the
43:12
standardizations and having recognising where you can get a
43:16
little extra oomph out-of-the-box if needed is is
43:22
great, but if it's not needed, that's even better.
43:26
Maybe you just transparently get that optimization period.
43:31
So that's one of the great things I've seen the Torch chat
43:38
in response to Torch Jack, there've been improvements to
43:43
high Torch, to Torch Ale to some of these other libraries that
43:49
have looked at what does a model do here and where the
43:56
opportunities for further improvement.
43:58
I mentioned our work with the ARM for on device.
44:03
There is similar work with the GPU vendors to to create better
44:11
kernels, either Brittany, CUDA or with Triton, which is the GPU
44:23
language that's coming out of Open AI that we use for for a
44:30
lot of applications that we generate code for.
44:34
When you use the Torch export to export the model, it will
44:38
generate the Triton code if you're targeting a GPU and then
44:43
run it through that compiler and you can get a lot of
44:49
optimization that has been developed for Triton is part of
44:55
this ecosystem. And there's another of these
44:58
pillars where it's a parallel community.
45:01
But by integrating them into this ecosystem, it now offers
45:07
benefits to all the pipers users that use export and including
45:14
for Torch chat. If you export that for GPU but
45:21
the Torch Jet export, it'll generate the Triton code and and
45:29
built both the model that way. Are there any anti patterns that
45:35
you've seen recently that are worth talking about I.
45:40
Think the biggest anti pattern is premature optimization.
45:48
There is this tendency that people sometimes develop that
45:57
oh, my problem is so different, whatever is out there won't
46:02
apply here. And they start to build a very
46:10
specific, very niche solution. And sometimes that is necessary,
46:15
like if you have, you know, crazy constraints that just
46:21
nobody else can satisfy. But you'd be surprised how often
46:29
you find out that constraints or challenges similar to the ones
46:36
that you're facing in your application exist in so many
46:41
other places in the ecosystem. And if you're operating in an
46:46
environment where you can just turn around and grab that
46:51
solution and bring it to your problem, it's so much more
46:56
efficient. So cutting yourself off by
47:01
premature optimization or oh I can't use general AI framework I
47:12
think becomes a very limiting anti pattern.
47:21
At the end of the day you might need some optimization for your
47:25
target, but it might actually be better to build the overall
47:29
solution and then ask which parts here do I truly have to
47:33
optimise for my target and where else can I just get a free ride.
47:41
I'm often joking that good engineers are lazy and that
47:45
doesn't mean they don't want to work.
47:48
It means like, oh, I felt this two times already, why should I
47:52
implement the same solution for the third time?
47:55
Let me take this solution over there so now I can spend my
47:59
engineering time on creating something new of value rather
48:04
than redoing what already did. Maybe in a slightly different
48:09
context. So cutting yourself off from
48:14
from the the broad sharing of code and ideas I think is is one
48:24
of those and anti patterns that I see people sometimes fall
48:31
into. So there's a funny thing that
48:35
you mentioned there with the over optimising too early.
48:40
And really sometimes I can imagine that you're thinking
48:47
you're taking 2 steps forward, but really you're taking 2 steps
48:51
back and you recognise it a little bit later and you
48:55
recognise wow, if if actually this wasn't optimised, maybe we
49:00
could do what I'm trying to do a little bit better.
49:03
How do you know when it is time to optimise?
49:07
First, get it to run. If it's not running in whatever
49:13
form, shape or colour, it's probably too early to optimise.
49:19
And then I would say you want to figure out what the obstacles
49:29
are to run. I don't know what the
49:34
performance or the battery life or whatever your constraints are
49:41
for, for your target and to ask, well, specifically what are the
49:46
pieces I might optimise to get there so I can focus on those
49:53
pieces that are that make the difference, that are make or
49:58
break. And you can still reuse
50:03
everybody else's work for the rest because that allows you to
50:09
focus what the real value of your creation is.
50:15
I can make this run on whatever this device or but that much
50:22
battery life or whatever. So there is, there is that part.
50:28
And then there is, well, you know, like I need the gazillion
50:33
of other things that I need functionally, but like a file
50:38
system, I'll, I'll not rewrite a file system for LLMS, but yet I,
50:47
you, you need a file system because you got to store stuff,
50:51
right? I think everybody will see it
50:53
for the files in the case of the file system.
50:56
But there are many other things like quantization algorithms.
51:00
OK, so I, I want to compress the data.
51:04
It doesn't actually, you know, matter what device it is.
51:08
It's the same sort of quantization approaches.
51:13
Well, if I can go to Torch ALE, the architecture optimization to
51:17
pick up my quantization algorithm, that's goodness.
51:23
Maybe I have a RIB 5 and I don't have a RIB 5 kernel yet.
51:28
Then I can use the quantization algorithm that compresses the
51:33
way and if my contribution is running the first LLM on the
51:41
RISC 5 smartwatch, the optimizations or the work that I
51:47
have to do specific to that are the RISC 5 kernels.
51:52
XMM pack has some of those already, but I much like for ARM
51:59
our team working, but XM Pack and ARM teams developed a better
52:06
matrix multiply form this matrix multiply for ARM.
52:10
I can see that there is, you know, a similar opportunity say
52:15
for RISC 5 and that's the contribution that enables us to
52:20
run to scream on that device, maybe partnering, but RISC five,
52:25
hey, wouldn't it be cool? There is this instruction that
52:28
ARM has if you had something similar, but hey, I also have an
52:34
idea how to do that. Even better if you're building
52:37
your RISC 5 Smartlatch that speaks LLM that does LLM.
52:45
It may come down to you just need a few kernels and most of
52:48
the rest like the quantization algorithms itself, like the LLM,
52:56
like the model export or the executorch.
53:00
You can build on all of that and you know, focus on getting your
53:08
product, your idea out by focusing on on what is needs to
53:18
be different to enable what you want to do rather than
53:23
rebuilding the world. Are you bullish on types of new
53:32
training architectures or or new model architectures for this
53:37
optimization? Like you said, hopefully we can
53:40
see the smartwatch, the LLM running natively on a
53:42
smartwatch. And I feel like it's going to be
53:46
a little bit of each area that you can as as you've talked
53:50
about, there's such a Broadway that you can go in so many
53:54
different vectors where you can optimise.
53:56
And I wonder if if you're bullish about the model
53:59
architectures or if there's other places where you're
54:01
feeling very bullish on that you want to talk about?
54:06
There are a couple of things that I think looking forward to
54:10
the next innovations at the model architectural level there
54:16
Transformers are everywhere and in some form, shape or cola,
54:19
they're here to stay. But there is a discussion of are
54:23
there improved transformer implementations, the Amamba
54:31
architecture. So that'll be exciting.
54:34
There's, there's definitely more opportunity there, typically
54:42
more focused on infrastructure. So not there, there's probably
54:49
more, many more of these out there in research.
54:54
And having those researches use pipe bridge means that when when
55:01
they find something, we can bring it in, plug it in very
55:05
quickly as as a community, because, you know, it might need
55:15
a couple of new operators or whatever, but it's still the
55:19
same tokenizers, it's still the same, you know, linear kernels,
55:24
etcetera. Other areas.
55:29
I'm definitely excited about numeric, the the whole brand
55:36
that we've been seeing to exploring numeric format.
55:41
I mean, we started out with B 3216 brain float 16 FB 8.
55:49
Now we're down to FB four people looking at 4 bit floating point
55:53
numbers. So I think there's a lot of just
55:59
innovation, the experimentation that's going on and the
56:09
infrastructure and the environment that we have I think
56:13
is very optimised for that. Typically insights are a
56:20
function of how many iterations, like a lot of this is very
56:25
experimental. So if you can run more
56:28
experiments, you get more insights and you get better
56:30
results. So the optimising for turn
56:34
around, for iteration speed, it is something that's been on my
56:40
mind and I know on other people's minds as well.
56:45
Andrew and had pointed out how that is critical if you want to
56:51
drive success in a startup, for example.
56:54
Like it it duration speed is directly proportional to rate of
57:01
innovation. Almost so excited about both
57:06
what we can do with the duration speed, but also what can we do
57:11
to speed up how long it takes to to run 1 iteration.
57:19
One experiment so that we might get new insights that that allow
57:26
us to build better, faster, more reliable, more efficient
57:30
systems.