Latest / Agentic Conversations (formally mlops.community) / Efficient Deployment of Models at the Edge // Krishna Sridhar // #284

Transcript

Efficient Deployment of Models at the Edge // Krishna Sridhar // #284 · Agentic Conversations (formally mlops.community)

0:00
I'm Krishna. I'm I'm VP of Engineering at
0:02
Qualcomm. I work on Qualcomm's new
0:06
Qualcomm AI hub. I take my coffee always as a
0:10
latte with 2% milk. Today we're taking it to the
0:14
edge. Yes, I did just say that.
0:17
All about AI on the edge. Welcome back to the MLR
0:21
Community podcast. I'm your host Demetrios and
0:25
talking with Krishna, I got to learn all about what Qualcomm is
0:33
doing in the chip sector, but mainly to make it easier for AI
0:39
and ML developers to grab their models, bring them into the
0:44
Qualcomm AI hub and then easily deploy them onto Qualcomm chips
0:51
or any chip for that matter, on the edge.
0:54
They give you so many different stats and help you optimise for
0:58
what you need to optimise for. It was really cool to talk to
1:03
him because I feel like AI on the edge is something that I do
1:06
not know enough about and there's a lot of people doing
1:10
some magnificent stuff. I asked him what some of his
1:13
favourite use cases were on the edge and he gave me a whole list
1:17
of them. He rattled off a few, one being
1:20
having to do with cricket, which is not a sport that comes up
1:24
much. Let's get right into it.
1:27
As always, if you enjoy this episode, it would mean the world
1:30
to me if you share it with one friend.
1:32
And a huge shout out to Qualcomm for sponsoring this episode.
1:37
It is because of sponsors like you that we get to do this cool
1:41
stuff. So if you're deploying on the
1:44
edge, definitely go check out the AI Hub that Qualcomm has set
1:48
up. Let's go.
1:57
I'm sensing a pattern in how you do things.
2:01
You start companies and then you sell them to big companies.
2:07
Yeah, yeah, yeah. Well, well, I've done it once
2:11
and I've been part of a company that's that that for which it
2:16
happened the second time. So yes.
2:20
It's very cool. And what were the companies?
2:23
Is it always been in the AI space?
2:26
Yes, It's always been in the AI space.
2:27
I've been in the AI space for, I don't know, almost 1112 years.
2:34
So it's been a, it's been a while.
2:37
Very cool. What made you want to get into
2:39
it? I actually have an interesting
2:43
story. I did, I, I was doing my PhD in
2:48
Wisconsin on numerical optimization.
2:52
And I'm, I'm, I'm actually being deadly serious about this.
2:56
My PhD thesis was how to help oil companies optimise for taxes
3:03
in their drilling in their, you know.
3:08
You found your life's purpose. Yes, you know, and, you know,
3:11
and at that time I was, you know, obviously much younger.
3:13
And I was like, oh, this is so cool, such interesting problems
3:16
to solve. There's a lot of cool
3:18
technology. And then, you know, one day I
3:20
wake up and I'm like, wait a second, I'm actually helping oil
3:24
companies, you know, optimise their drilling with taxes.
3:29
And I mean, the story of how this whole thing came about was,
3:32
is, is, is, is, is kind of fascinating in itself, which is
3:36
a lot of governments in, in, in Africa, they started imposing
3:40
these complicated tax laws so that, you know, oil companies
3:44
would end up paying their taxes to them.
3:47
And, and then, you know, it's sort of a game of cat and mouse
3:50
where you're like, OK, now let me optimise how to, you know,
3:53
how to do this. And it, it became an interesting
3:56
problem in itself. So I was doing that.
4:00
And then, you know, a lot of interesting problems in machine
4:04
learning was what, what it was called back then what around how
4:10
to do numerical optimization. So that's how I got into it,
4:12
into ML or AI and, and then I've been there since I, I figured
4:18
it's a much more useful, you know, useful than ever than
4:23
helping oil companies with taxes.
4:26
I can imagine since then you've you've been around the block per
4:30
SE and you were doing stuff with Apple and building a platform,
4:35
right? Yeah.
4:36
So I had this unique opportunity to, to, to help design Core ML,
4:42
which is Apple's inference engine to deploy models on on
4:46
the edge. And what was fascinating about
4:49
deploying models on the edge is, you know, is, is the fact that
4:58
you can do extremely innovative stuff in a way that's very
5:03
snappy from an experience perspective, as well as
5:08
extremely privately privacy sensitive.
5:11
So the fact that your data doesn't need to leave the phone
5:17
And well, you know, of course, the, you know, the
5:20
quintessential example, if it is kind of Face ID or face
5:23
identification that's now there in every phone.
5:26
I mean, there's no opportunity for you to release your
5:29
biometric data to the cloud, get it verified and then get it sent
5:32
back. Doesn't quite make sense.
5:35
But at the same time, you need, you need your Face ID to work
5:37
within, let's say 300 milliseconds.
5:41
So you have to run some pretty complicated neural networks to
5:45
be able to do face on the authentication locally on your
5:49
phone in a way that's extremely secure.
5:53
And, and that was sort of an example of of on device MLA.
6:00
And, and I'm going to, I'm going to ask you, I'm going to do a
6:02
little pop quiz with you. Let's say you take a modern
6:06
smartphone and you take a picture with it.
6:08
How many AI models do you think run in that you know in that 300
6:13
to 700 millisecond time frame when you press OK?
6:17
Well, I do know that it will get better because I see it visually
6:22
with my eyes. I'll I'll take a photo,
6:24
especially when you're in portrait mode with like the
6:26
iPhone and it looks like it's not that good at first and then
6:33
it refreshes or it updates the pixels and boom, it's like, wow,
6:37
that was magic. Now it looks really good.
6:39
I can imagine what if we say I don't want to like go too crazy
6:44
and make you. I'm going to I'm going to
6:47
lowball it and say. Five models, OK, that's actually
6:52
not a not a terrible guess. I think it's order of 2525.
6:58
Yeah. So that's not a small number for
7:00
you don't have more than 300 to 500 milliseconds.
7:04
And if you, if you include post processing, you don't have more
7:06
than a second. That's all the time you have.
7:10
And it has to do everything, including, you know, framing,
7:13
sometimes face detection, recolouring.
7:15
So that like all these complicated things have to be
7:18
done in 2025 milliseconds. You get a perfect shot.
7:21
And as, as, as you see, the phones are getting better and
7:25
better at it. I don't know if you saw the
7:26
latest Android phones. You have these really cool
7:29
features where, you know, you have a group of friends, you
7:32
know, all of their eyes are not open at the same time.
7:34
So, you know, you know, can you find the shot where everybody's
7:37
eyes are open at the same time? So you got to process a bunch of
7:40
different frames, find the shot, you know, make the perfect
7:43
photo. So all of that, you know, has to
7:45
happen real time. You don't have an opportunity to
7:48
like send all this information to the cloud.
7:50
You want to see the picture right away.
7:51
So you have to do it locally on the device, even with
7:55
assistance. I mean, even though you're, if
7:59
you talk about the Google Assistant or in a Siri, a lot of
8:04
the work has to get done locally.
8:07
Of course, a bunch of the heavy processing around answering the
8:10
questions happens in the server, but the speech processing, the
8:13
text to speech, a lot of that happens locally.
8:16
So you can have a pretty snappy experience.
8:20
And you also have, you know, a tonne of compute that's there
8:23
sitting right there on, on, on your phone like, you know, the
8:26
latest Snapdragon, you know, you have, you're looking at 4045
8:29
tops. So you know, you can compute,
8:32
you can use all of that compute to take pictures, to do speech
8:36
processing, to do video processing.
8:38
You can do a lot of that locally.
8:41
So that's what sort of got me into this field is pretty
8:43
fascinating. And going back to that, well,
8:47
just so I can place it too. What level do you like to play
8:50
at in the stack and how do you look at it?
8:54
Like where? Where is your sweet spot or your
8:57
special sauce? So I've had the experience over
9:00
the years to interact with almost all the layers of the
9:04
stack rate ranging from actually, before I answer that
9:08
question, I'll, I'll tell you what makes this a fascinating
9:10
problem. So I talked about, you know,
9:12
2025 different, you know, models running on a camera.
9:16
I think on a modern smartphone you can have, you know, order of
9:20
1000, you know, models running, you may have an order of, you
9:24
know, 100,000 applications that are built on top of on device
9:28
AI. So these are some pretty large
9:31
numbers. And, and what's important is
9:34
the, the, the approach that, you know, we've always had to this
9:37
problem is build systems that other people can use because
9:41
we're not building all the models that go into all of this
9:43
technology. We don't, we don't, that's not
9:45
our special source. Our special source is building
9:47
systems so that the the people who are innovating with the AI
9:52
models can bring them into our hardware as easily, as quickly
9:55
as possible. So, So what that means is we
9:59
need to be be able to build a system that can translate
10:02
anything a researcher is doing in the cloud to train the models
10:07
to be able to bring them onto to a device, to run them on the
10:09
device. And what makes this a pretty
10:13
challenging problem is, is the fact that the, the pace at which
10:18
AI innovation is happening is absolutely, you know, insane.
10:22
It's so rapid, it's changing all the time.
10:26
The software people are using the cloud to train models are
10:29
changing all the time. They're using different
10:31
frameworks. You know, back in the day with
10:33
like 10s of flowers, flavour of the week.
10:35
Today it's all pidarch, you know, and, and things are
10:38
changing so fast in, in, in that ecosystem.
10:42
At the same time, your hardware innovation is happening at an
10:46
incredibly rapid pace. The kind of hardware that was
10:48
being used in a phone or laptop even five years ago, it looks
10:53
nothing like what it is today. Almost all the phones today and
10:57
all the computers today have dedicated newer processes, newer
11:01
engines to run specialised, you know, specialised AI models.
11:05
So you have your hardware that's changing and you have, you have
11:09
the AI and frameworks that are fast moving.
11:11
So you have two fast moving things, one on the top and the
11:13
bottom. So that's what makes this a
11:15
pretty interesting problem because you got to map a fast
11:18
moving thing at the top to a fast moving thing at the bottom.
11:20
So you got to have pretty stable infrastructure.
11:23
And and that that's what you know is exciting about this
11:26
particular problem is how do you build stable infrastructure that
11:30
can handle a fast moving AI ecosystem and a fast moving
11:33
hardware ecosystem that allows someone to target their latest
11:36
models to not only the latest hardware, but also potentially
11:40
previous generation hardware. Yeah, it's, it's a nice
11:44
metaphor, how you talk about the top and bottom moving so quickly
11:48
and almost like you're trying to set up that foundation and have
11:54
a strong foundation, but at the same time you're trying to be
11:57
able to hit the target and the target's moving.
12:01
So the foundation's moving, the target's moving, and you're
12:04
having fun trying to play between both of those.
12:08
Yeah, and that, that's what makes this this whole space
12:10
super. Fascinating.
12:12
OK, So now getting more along the lines of you're working at
12:16
Qualcomm, you're doing all kinds of cool stuff there with on
12:20
device AI. How are you looking at it in
12:22
that sense of coming back to the metaphor, like being able to
12:29
understand how everything is moving in these lower levels,
12:34
but also servicing all of the innovation and how everything's
12:40
moving at the upper level? Yeah, so a few things have have
12:44
gotten a little bit easier. So, I mean, the way we look at
12:50
it is, you know, there's still a lot of stuff going on in your
12:54
classical computer vision, classical speech spaces, but
12:58
there's also a lot of new stuff going on with generative AI
13:01
with, you know, with new language models and new image
13:08
generation models. So what's what's interesting
13:12
there is, you know, the cloud model, the best cloud model from
13:17
let's say two years ago performs worse than the best device model
13:24
today. And that's, you know, a
13:26
fascinating thing in itself. So what, what that basically
13:31
means is things that we were able to do with, you know, an
13:36
order or two orders of magnitude more compute a year or two years
13:40
ago, we can do now on a local device.
13:43
So almost all the latest flagship Android smartphones
13:48
today, they, they have a language model that's in there
13:53
to do things like summarization, you know, completion, you know,
13:58
text, you know, replies do all of these things can be done
14:02
locally on your phone. So let's say you get, you get a
14:06
long e-mail, you know, you know, your boss likes to write a super
14:10
long emails with like, you know, 25 pages and you can just hit
14:13
summarise and you get a summary of it.
14:16
And that information can stay local on your phone.
14:19
Now those are the kinds of tasks that are now being done with,
14:23
with LLMS on the device, which look a little bit different from
14:28
let's say you're capturing a smartphone and doing, you know,
14:32
tone mapping that looks a bit different from from your text
14:37
summarization use cases. So to, to answer your question,
14:42
what has changed is, is sort of there has been more use cases
14:49
that are coming on, especially with Gen AI.
14:51
But luckily a lot of the, the architectures for these models
14:56
there, they look similar is from a compute perspective, it's
15:00
typically Transformers and they don't look that different.
15:04
So that makes it slightly easier.
15:07
So if everything is fast moving in the top, but kind of the
15:11
techniques that are being used are similar, then that allows us
15:15
to build great systems because we're not necessarily building
15:19
widely different systems for your camera and your, you know,
15:23
language models. So that, that's one thing that
15:26
has, that has changed in the last 2-3 years is there's been
15:28
more and more Transformers that have been deployed.
15:31
They look pretty similar. In fact, the, the, the
15:34
architectures of models that are shipping on your phone to do
15:38
language tasks is not very different from let's say you're
15:45
a DAS or driver assistance systems that are shipping and
15:47
Qualcomm chips in your car. It's kind of fascinating that
15:51
those two kinds of problems are using similar techniques.
15:55
So that has made our life a little bit easier.
15:59
And, but the hardware is still innovating at like at, AT and
16:04
there's always new features of the hardware.
16:05
So that part hasn't quite changed.
16:08
But I would say the, the, the, the difference is become in, in,
16:13
in more nuanced things like precision.
16:16
And to give you, you know, a perfectly good example, when
16:19
you're doing something like a large language model
16:22
summarization, it's not as sensitive to precision, which
16:25
means you can, you can, you can do something with four bit
16:30
weights, let's say, whereas when you're doing, you know, a dash
16:35
in your car and you're doing precise calculation of, you
16:38
know, where is this next car or where is the pedestrian, that
16:41
stuff is pretty sensitive to precision.
16:43
So you can't, you can't afford to really go lower bit.
16:47
You got to keep it higher bit to keep the accuracy high.
16:51
So that is what I see as as, as differences these days is like
16:56
in, in terms of hardware is that you have slightly more nuanced
17:03
requirements or changes in the hardware to accommodate
17:06
different kinds of precisions for roughly similar
17:09
architecture. So that that I think is what's
17:11
been what's different now than it was let's say 2-3 years ago.
17:16
Do you think it's the hardware's job to be doing that kind of
17:20
optimization or is it able to be done in the software layer?
17:26
This is a great question and, and I kind of have a good
17:29
analogy for this site. So almost all hardware today on
17:35
modern smartphones, cars, laptops, you know, industrial
17:41
devices, pretty much everything Qualcomm ships today.
17:47
We, we do this thing which, you know, it's called heterogeneous
17:49
compute, which means you have ACPU, you have a GPU and you
17:53
have a newer processor. And the analogy I give is your,
17:58
it's, it's a spectrum where the CPU is, is the most flexible and
18:04
programmable piece of technology where you can pretty much do
18:07
anything. You can programme it to do
18:09
anything. Now there's no guarantee that
18:11
everything will be fast, but you can do everything.
18:16
On the other side, you have your neural processor which is
18:19
extremely specialised. So it can do some things really,
18:24
really fast. It's designed to do somethings
18:27
really, really fast. And when you make a design at
18:30
the hardware level, you can typically get an order of
18:32
magnitude more efficiency and order of magnitude more power
18:35
efficiency and order of magnitude more more performance.
18:39
And that is, but then it comes at the cost of flexibility.
18:45
You can't programme anything like you can't, even though it's
18:50
technically Turing complete, it's not easy to programme
18:52
everything. So it's only efficient at
18:54
certain things and then you have your GPU which is somewhere in
18:57
the middle, which is which is more programmable, but it's also
19:01
more efficient at certain things.
19:04
But it's kind of a spectrum that way.
19:06
So it's always a dance of like, OK, what are the features that
19:11
are so critical that we need to bring it in hardware to get that
19:17
extraordinary more efficiency versus what can we make do with
19:22
a slightly less efficient software implementation maybe on
19:27
the CPU? And that's the balance.
19:29
You're always playing with this, with this technology.
19:33
Yeah, you have all these knobs that give you these different
19:35
trade-offs, right. And I imagine you've seen some
19:39
surprising use cases or trade-offs happening.
19:44
What are some areas or or different specific situations
19:49
where you didn't think it was going to work or it was
19:52
optimised in a certain way that you were a little bit surprised
19:56
by? Oh, this is a good.
19:59
Question. I would say universally over the
20:03
past past five years, every time I see some new features that are
20:07
described in the hardware of the silicon, there's a certain
20:10
expectation of what people are going to do with it and it's
20:14
usually have. I would say I've had 0% success
20:20
rate in predicting how people end up using it because
20:25
obviously people will end up using the things that you tell
20:28
them to use. But then people always find
20:30
creative ways to, to deploy stuff on, on your hardware.
20:34
And, and that was, was right from the very beginning of
20:38
accelerated computing in this, in this neural network space has
20:42
always been the case because people when, when, when you
20:44
write the spec, you write a spec a certain way.
20:46
But when a programmer is reading the spec, they read it
20:48
differently. And they do some pretty
20:51
fascinating things that that surprised me.
20:55
And, you know, Transformers was a great example.
20:57
I don't think the hardware necessarily was optimised for
21:02
deploying Transformers, but we've figured out ways to do it
21:06
in software to optimise it in ways that I wasn't expecting.
21:10
And you know, it's now I would say an order of magnitude more
21:14
efficient to deploy in software primarily because of how we, you
21:18
know, optimise the software for deploying these kinds of
21:21
networks. So it happens all the time that
21:24
people find more interesting and creative and innovative ways to
21:28
deploy models on on hardware. OK.
21:33
So we've got the spectrum of easy, hard trade-offs, fast,
21:39
slower or flexibility, I guess is is 1X axis, Y axis type of
21:45
thing. And we've also got the different
21:49
ways that the chips are being optimised for certain scenarios,
21:57
whether it's a chip that's inside of a car and that's being
22:01
optimised for that precision, or if it's a chip that's inside of
22:05
a phone, it doesn't necessarily need that precision.
22:08
The thing that I'm fascinated by is as you are looking at on
22:15
device AI and ML, what do you think some challenges are that
22:21
we're hitting up against or some walls that maybe are hopefully
22:28
we're going to get over in the next couple months or, or what
22:32
are just some some things that you feel like people are
22:35
grappling with that they haven't had before?
22:39
Because now as you mentioned, things are just getting so good.
22:44
So we're starting to push the limits in different ways that we
22:48
didn't necessarily get to in the past.
22:51
Yeah, this, this is a great question.
22:53
It actually ties in nicely with your previous question as well.
22:55
So the thing with especially with these SLMS and LLMS, the
23:02
things that at least I didn't expect was how quickly we'd run
23:07
up against memory issues. So now it's, it's, it's not just
23:13
about compute. There's also memory that you
23:15
have to factor in. And now that's actually become
23:18
the biggest bottleneck where the base at which we can bring in
23:24
larger models, I'd say somewhat correlated to what kind of
23:28
memory technology innovations can be made, what kind of, you
23:32
know, algorithmic innovations can be made to consume less
23:36
memory. And, and that's become, you
23:39
know, a bottleneck more than the raw compute per SE.
23:44
So that, that's one thing that that that's, that's been
23:48
different in the last, I'd say 2 years.
23:52
And the second one is, which is another fascinating constraint
23:58
is, is power efficiency or battery life, right for a lot
24:03
of, you know, mobile compute, when I say mobile, I mean PCI
24:08
mean an industrial chip. I mean, you know, even the car
24:13
battery life is pretty important and you don't want to be
24:16
spending too much battery or power to do some of these
24:22
things. So that's another area where you
24:25
may, you know, you may have the raw horsepower to go, let's say
24:27
at, you know, 100 miles an hour, but you prefer to go at 30 miles
24:31
an hour because you know it, it consumes less power.
24:35
So the, the example I give is, hey, I have this LLM that can
24:39
generate 50 tokens a second where I'm like, I can't read 50
24:42
tokens a second. So you don't need to give me the
24:45
information at 50 tokens a second.
24:47
You can give it to me at 10 tokens a second.
24:50
I can still, I can still barely read it, but it, you know,
24:54
consumes a lot less battery life.
24:56
So that's the, you know, the the trade off.
24:59
Sometimes you you, you can choose to make and that and
25:02
that's been pretty interesting to to balance out over the last
25:07
couple years. And the third thing, which I
25:10
think is I still think is the bottleneck is just being able to
25:17
do more high quality things with Gen AI in a limited, limited
25:23
compute constrained environment setting.
25:27
So I mean, in obviously the dream is to have something like
25:30
entirely like chat CPD running locally, but I would assume it,
25:34
it still consumes maybe 2 orders of magnitude more compute than a
25:40
constrained device can handle. So there's still a lot of
25:43
innovation that has to get made to be able to compress that
25:46
technology to, to smaller sizes while improving the compute.
25:52
So that, that still is, is an ongoing exercise.
25:55
And I, and I think that in order for us to see, and I expect that
26:00
over the next couple years that will, you know, more and more of
26:04
that will happen and you'll see more and more use cases that are
26:08
fascinating and can happen locally and quickly and snappily
26:12
on any device without needing, you know, an incredible amount
26:17
of compute in the cloud to do everything.
26:20
Yeah. And I wonder if it is we just
26:22
move towards like you were talking about with the photos
26:25
and how there's 25 models that are working on a photo every
26:28
time you take a photo. Are there just going to be a
26:31
whole tonne of models that we can use that are small language
26:34
models? Or I guess that might not be the
26:37
ideal architecture or the ideal way to do it to load up 25
26:40
different models and you just use Laura's.
26:42
But there's got to be a way that we can take advantage of the
26:47
small language models in that respect.
26:50
I mean, it it, the similar analogy I give is it's with
26:54
especially with a lot of photo processing, it's not like
26:57
they're 25 totally different models.
26:59
They shared a common backbone and they had different heads.
27:03
So there was a lot of shared compute.
27:05
So that allows you to do a lot of things with, you know,
27:09
efficient compute. So with Laura's and things like
27:12
that there, you know, it's similar analogy.
27:14
You got to share compute. You got to share not just
27:17
compute, but also memory as as I said, that's become the biggest
27:21
constraint as well as as space. So like you as a, as a, let's
27:27
say a smartphone, you go buy a smartphone, you don't want 30
27:31
gigabytes of that to be consumed by, you know, models.
27:34
You want, you want that to be, you know, maybe one or two
27:37
gigabytes because you need space to do stuff with your, you know,
27:40
store photos, store your memories, things like that.
27:43
So, you know, space and memory, all of that factors in as well.
27:45
So I'm sure we'll find more efficient ways to to share use
27:52
cases and and and share compute and share memory and shares, you
27:56
know, space the same use cases. But those are those, you know,
28:00
those I expect to happen. Yeah, it's interesting you bring
28:05
up memory because we had Bernie Wu on here from Memberge a few
28:11
months ago and he was very much talking about how we've come to
28:16
this place where memory is the biggest constraint.
28:19
And one thing that he mentioned he's he's trying to enable is
28:26
the shared memory pools. But that works with a big GPU
28:33
cluster, right? That doesn't necessarily or even
28:36
potentially you get a bunch of CPUs together, I don't know.
28:39
But if you don't get a bunch of phones together to share memory,
28:44
so it's almost like a hard out, you can't do that.
28:48
Yeah, I mean you, but you do have the option of having
28:50
multiple, let's say multiple cores or multiple NP US or
28:54
multiple GP us within the phone and you can share memory within
28:59
those. I mean those those kinds of
29:01
architectures you know can be done as well.
29:03
So, yeah, I mean, those technologies I think are still
29:06
really key for us to, you know, look into before and we can do
29:11
more cool, more cool things locally.
29:14
Yeah. Yeah.
29:15
And it it's also a great point that I don't want to buy my
29:18
phone and half of my space is already taken up by all these
29:22
small language models or large language models, semi large
29:25
language models that are on there.
29:27
Since you have to buy the upgraded phone just so that you
29:32
can have a little bit of space for your photos and your videos.
29:38
Yeah, definitely. In fact, I think almost all the
29:41
phones today come with at least 8 to 12 gigabytes of RAM 'cause
29:45
you can't, you can't, you can't do LMS. It's, it's fascinating
29:49
that they have to have more RAM just so they can do more AI on
29:52
it. Not, you know, they don't
29:54
necessarily need more RAM for it's a more regular
29:58
applications. But it's like primarily because
30:00
we had they're upgrading the RAM across all phones in the premium
30:03
segment. Yeah.
30:06
So talk to me a bit about what you're doing at like Qualcomm AI
30:11
Hub, because I think there's some fun stuff there for the
30:14
developers that want to develop on the edge and they want to put
30:19
models onto the edge. Yeah, I can kind of summarise
30:21
that very, very quickly. So, you know, for for those who
30:28
are unaware, Qualcomm is the largest manufacturer of silicon
30:33
in the world and we manufacture silicon for phones, for PCs, for
30:38
cars, and, you know, industrial automation, IoT devices.
30:45
So we manufacture silicon for a lot of, you know, power
30:49
constrained settings and. One of the.
30:55
One of the things that we're trying to solve is to make it
30:58
extremely easy for a developer, whether that be a manufacturer
31:02
of these devices or cars or phones or PCs, or someone who's
31:07
building an application on these devices.
31:09
They want to make their job of being able to bring the latest
31:13
and greatest AI innovations to our devices as easily as
31:16
possible. And you know, we have a little
31:19
saying, which is they have to be able to do this within, within 5
31:23
minutes, within 5 lines of code. So we built a system which
31:28
allows people to, you know, as soon as they finish training,
31:31
they can say, hey, this is my model, this is the device I want
31:35
to run it on and go. And what our system does is
31:39
it'll take the model, it'll translate it to how to run it
31:43
most efficiently on our new processor.
31:45
It'll optimise it. We actually even have physical
31:49
devices in a, in a cloud where we'll measure performance right
31:53
there, we'll run it, we'll measure accuracy and we'll get
31:56
back a result in 5 minutes. And to the developer saying, OK,
32:00
this model will run in, you know, 60 milliseconds on the on,
32:04
on these devices. It here's how, here's how you
32:07
should run it on the device. Here's the model for you to
32:10
download and run. And, and these are the
32:14
performance characteristics of the model.
32:16
If you want to tweak it some more.
32:18
And here's a link so you can look at it and, you know, share
32:22
it with your colleagues so that they also understand what you're
32:25
deploying. And and here's an automation for
32:28
you to do this programmatically in your in your in your system.
32:33
So we have sort of automated the process of being able to deploy
32:37
models on all our devices. And that we believe is the key
32:41
to more and more innovation and more and more iteration and more
32:45
and more deployments of more complex things across all our
32:50
different devices. So that in a nutshell is what
32:52
qualcoming I have this. OK.
32:54
So, so many questions I have for you on this, because I'm
32:58
assuming we have a lot of different generations of chips
33:07
that are getting put into different devices.
33:10
And so I'll think of like a random one because this is just
33:13
where my mind goes, a refrigerator that has chips in
33:18
it. And I want to put some kind of a
33:19
model on that, but the refrigerator might have lots of
33:25
generations of chips within the AI hub.
33:28
Does it also give you hey, run it like this on this chip, run
33:32
it like this on that chip and. We do one step better, which is
33:36
we also have a configuration which allows you to deploy the
33:39
same model on multiple generations.
33:43
And then, and then our, our, we actually have a small little
33:47
system on the device that detects which generation it is,
33:51
knows what features are there in that particular generation and
33:55
efficiently maps the, the, you know, the operations of the
34:02
compute to that chips generation accordingly.
34:05
So it's it. We also allow for developers to
34:08
be able to, to go back your example to, to create one model
34:12
that'll run differently on different generations of the
34:15
refrigerators. Maybe on the latest generation
34:17
it'll run faster. On the older generation, it may
34:21
run slower, but it'll still run so that that sort of flexibility
34:25
is also something we can we can do.
34:27
So we all know about the autonomous driving use cases or
34:31
just like chips inside of cars. We also know about chips inside
34:35
of our phones because we play with them everyday.
34:37
What are some fun use cases that you've been seeing on developers
34:43
deploying AI into chips or places that we wouldn't
34:47
necessarily think? OK.
34:48
I'll give you a couple, couple of use cases that I think were
34:52
pretty fascinating. I'll start with the, with the
34:55
mobile one because you know, it's I, I thought it was a
34:59
pretty fascinating scenario. So I'm a big, you know, cricket
35:05
buff. I play, you know, I play in the
35:07
recreation cricket league. You know, it's like it's, it's a
35:10
pretty intense league. We play, you know, play for 8
35:12
hours on a weekend and, and one of my buddies once he, you know,
35:17
brings his phone and he like, you know, connects it to, you
35:21
know, a tripod and he starts recording the, the, the, the,
35:24
the, the practise session that we have for before our game.
35:28
And he's like, hey, you know, I built this app that can do, you
35:32
know, real time tracking. I don't know if you watched, if
35:36
you watched cricket or tennis, or I think it might be there in
35:40
soccer too. There's a system called Hawkeye
35:43
which can do real time ball tracking, real time, you know,
35:48
prediction. So in tennis it tells you
35:50
whether whether the ball was in or out.
35:53
In cricket it you know, sort of tells you whether the ball's
35:56
hitting the stumps, you know, and in, in soccer, I'm guessing
36:00
it predicts whether it's goal is gone in or not, whether enough
36:03
of the ball is gone in or not. Or offsides maybe, or offsides,
36:07
right? I mean, offsides is more like
36:09
people tracking, but this is more like ball tracking.
36:11
So Hawkeye is the system, which I think they charge something
36:14
like $100,000 or something per international game for a game
36:20
just to do real time tracking. So he's like, hey, you know, I
36:23
built this thing which can do it on your, on your phone.
36:26
You just plug in your phone and it can do real time tracking.
36:30
It can do, you know, a lot of the, a lot of the important
36:33
things that, that you, that you want from a tracking system, it
36:37
can do it just with the camera technology on your phone.
36:40
You don't need sensors, don't need, you know, light hours,
36:43
don't need anything, just just phone tracking.
36:46
And he made it into a commercial app and, and I think they got
36:49
something like 100,000 downloads or something.
36:51
They had, last I spoke to them, they were at, they were at
36:54
$100,000 a day because people started using it to, you know,
37:00
at home for their, you know, either for their little, you
37:03
know, recreation games or street games or whatever, they started
37:07
using that. So I thought that was a pretty
37:08
fascinating use case where they would do real time tracking.
37:12
They would, you know, automatically, you know,
37:14
automatically take your, let's say 30 minute, you know,
37:19
practise session and clip it down into just the important
37:22
bits. So to give the baseball analogy,
37:27
it's like, you know, you just record your entire baseball
37:30
session and they break it down into just, you know, just
37:33
important clips where things are, you know, the important
37:36
things are happening and everything else is clipped out.
37:38
And they do all of this locally on the device, which I thought
37:42
was pretty fascinating that there's enough compute to do all
37:44
of that on the device That I thought was a pretty fascinating
37:49
use case, even though it's, it's on a mobile, it's a pretty,
37:53
pretty fascinating use case. So that's one I'm, I'm trying to
37:58
think of one from on the PC. On the PC, there was a bunch of
38:05
really fascinating use cases I've seen around, you know,
38:10
music and auto tuning and automatic generation of music
38:15
like auto DJ ING and things like that.
38:17
I've seen stuff like that, which is, you know, pretty
38:19
fascinating. You listen to the music and
38:21
you're like, Oh my God, it's actually automatically DJ ING
38:24
some music generating some stuff in the middle.
38:27
All of that again, happening locally on the PCI thought that
38:30
that that's pretty fascinating. I actually thought the the use
38:34
case that Microsoft released. I don't know if you played it on
38:37
with it on the latest, on the latest PCs.
38:41
It's called, it's called recall. That's pretty fascinating
38:47
because it can do a lot of things locally and help you do
38:53
effectively searches across across pictures, across
38:59
screenshots, across actual real time things that are happening
39:03
on your computer, which I think are incredibly fascinating that
39:08
you can ask questions like, hey, I was in a meeting at noon.
39:11
Who are all the people in that meeting?
39:13
And it can kind of answer that question by looking through
39:16
stuff in your computer, you know, looking through, you know,
39:21
stuff around your calendar, invitations, potentially
39:24
screenshots of who's in the meeting, you know, just
39:29
fascinating things that can be done entirely locally.
39:32
I've been super impressed with with with that particular
39:35
feature. I'd definitely recommend people
39:36
give that a spin. Then I'll throw in a few more
39:41
that I've been pretty interesting, really interested
39:43
with actually in Jenny I the thing that I've been the most
39:48
impressed with has been the automatic code generation.
39:52
That stuff's just getting so good it can practically finish
39:57
your functions. Like the programme of
40:00
productivity has just gone up to the point where when people
40:03
start using those, they just can't go back to.
40:06
That's I think another interesting use case.
40:11
In terms of actual devices, I think I've seen a lot of
40:17
innovation in these in in the space of physical security.
40:24
It sounds, sounds pretty boring, but I, I think it's pretty
40:28
interesting where you can have a, a tiny little device that's
40:33
plugged into your to your home and, and network and it can pull
40:37
in all the different cameras that might be there or not.
40:40
Just Home Office 2 pulls in all the feeds from all the different
40:44
cameras, analyses it and it can, you know, do this all in real
40:49
time and then only send the important things back to the
40:53
cloud. So that's, that's also pretty
40:57
fascinating use case for physical devices in insecurity
41:03
where previously it it required a lot of cloud compute.
41:07
Now a lot of stuff is happening locally.
41:09
I don't know if you've recently flown in, in airlines in Europe,
41:16
but they have a tonne of real time security stuff that's going
41:21
on all locally on the device. You know, they compute your, you
41:25
know, facial features and, and they do real time protection.
41:28
They do a lot of things that are happening locally in the space
41:32
of security. And, and you think about it, the
41:34
number of cameras and systems that are there in an airport to
41:38
be able to secure that. Not super straightforward, but,
41:42
but again, as compute gets more efficient, it becomes more
41:45
economical to do things like that.
41:47
So in terms of new devices, I'm I'm generally fascinated by
41:50
things in their physical security space.
41:54
And when you say compute gets more efficient, it's just
41:56
because we are able to only grab the important bits and do
42:03
everything on device until something is flagged as well.
42:08
This is important and it's able to create that processing on
42:11
device that knows when something is important.
42:15
That's correct. That's the part that I think is
42:18
really cool is you have a little, let's say a little, you
42:24
know, SoC or little chip that's plugged in the camera that can
42:28
do a tonne of processing and and you know, you have and that just
42:35
philtres out all the stuff that's not important.
42:38
Yeah, yeah. So going through the AI hub
42:42
again and I've got a model I want to come.
42:46
I, I trained in the Pytorch and I want to come and deployed onto
42:50
the AI Hub on one of these chips for my refrigerator so that it
42:55
can scan and tell me if my spinach has gone bad or
43:00
something like that. Maybe there's a inside my
43:02
refrigerator. I've got a little camera head so
43:05
it knows all right, this spinach get new spinach and it sends me
43:09
a text message or something like that.
43:11
How do I go about deploying it on the AI Hub and making sure
43:14
that it works on my refrigerator?
43:18
Yeah. So you know, you train your
43:20
model in the cloud and once you're done training the model,
43:23
you just use the Python API, have AI hub, you say, OK, these
43:27
are the devices I want to I want to deploy to and we literally
43:31
translate to all those devices. We actually have physical
43:35
devices in the cloud that map to those, to those SOC's.
43:38
So we'll take the translated model, run it on those devices
43:41
that we have in the cloud, tell you exactly what kind of latency
43:44
you're going to get, what kind of accuracy you're going to get,
43:47
and then give you back all those results in, in a few minutes
43:50
that that it's really as simple as that.
43:51
It's certainly nothing more complicated to that.
43:53
And, and what has been fascinating as a result of that
43:59
is because it's such an automated system, it allows us
44:02
to build on top of it to do more interesting things.
44:07
And what we've been able to do is bring in a tonne of
44:11
integrations into the system. So we have an AWS integration,
44:16
we have, you know, a bunch of third party integrations with
44:20
companies like Dataloop. And what is integrations that
44:23
say the AWS integration allows us to do?
44:26
It allows a developer to fully automate the process.
44:29
So they, they use AWS, launch a bunch of training jobs in their
44:34
system, then use AI Hub, launch a bunch of bunch of, you know,
44:38
optimization, inference compilation and, you know,
44:42
inference jobs on the physical devices.
44:44
And you have the whole system automated where they can retrain
44:47
a new model, deploy a new model and everything is automated for
44:50
them. So that is one cool integration
44:54
that's happened by automating the process of deploying on the
44:56
edge. And, and by doing that, the next
44:59
cool thing that has happened is now we've done a whole bunch of
45:03
partnerships with, with model makers.
45:07
So these model makers, companies like, you know, Mistral and,
45:11
and, and, you know, Microsoft for that matter, or even Meta,
45:14
they typically release their models for the cloud, you know,
45:18
and they partner with a lot of cloud vendors saying, Hey, you
45:20
can make our models available for inference in the cloud.
45:23
Now they're also partnering with us to make all their devices
45:26
available locally on the device. So all of their models are
45:31
available locally on their device on Qualcomm AI hub.
45:34
They can just go to AI hub, see the see that Oh, Mr's latest mod
45:37
is available, download the model and start using it.
45:41
And this allows us to create an entire ecosystem around the
45:45
automation of deploying on the edge.
45:49
It's just one more step in the DAG basically.
45:53
Yeah, and we'll just keep building, building along.
45:59
We talked about optimising for battery life earlier.
46:03
I wonder about when I'm playing in the AI hub, if I have those
46:07
specific requirements of I need to optimise for X, how do I go
46:11
about it? That's a great question.
46:14
So you know, we're, we're at the point where we give the first
46:18
sets of important information around what's actually happening
46:23
on the device. So as I'd mentioned before, all
46:27
of Qualcomm's SOC's are entirely heterogeneous, which means you
46:31
have CPU's, you have GPU's and you have newer processing units.
46:36
So now the first thing you as a developer you want to know, and
46:40
here's my model. Is it running on the CPU?
46:43
Is it running on the GPU? Is it running on the neural
46:44
processing unit? Is it, you know, partially on
46:48
the neural processing unit? Is it, you know, what fraction
46:50
of it is on the neural processing unit?
46:52
How can I get all of it on the neural processing unit?
46:54
So that's the first step towards optimising for more efficiency
46:58
because the more of it that lands up on the neural
47:00
processing unit, typically the more power efficient it is.
47:04
And So what developers can do is play around with their models
47:07
or, or work with us sometimes too improve the functionality of
47:12
the processing unit from a software perspective so they can
47:15
get more of their use cases running on the new processing
47:18
unit. So that allows them to really
47:20
optimise for efficiency and get more of their models running,
47:28
you know, as power efficiently as possible.
47:30
And I'm not sure if I fully understood your packaging it all
47:35
up there and then you're also deploying it into the devices.
47:41
How does it actually look? Or do you package it up?
47:45
And then I grab it and then I go and I deploy it on device.
47:48
It's more the latter. We package it all up and then
47:51
you download that package and then it's super easy for you to
47:54
deploy that on your device locally.
47:55
So what we've done is it's we've automated the, all the, all the,
48:00
all the like gnarly stuff around, you know, getting this
48:05
model, translating it in a measuring performance, all of
48:08
that we've automated. So you, all you need to do is,
48:10
you know, you train your model, you're like, OK, I want to
48:13
deploy this and then you get a little package and then you
48:15
deploy that. It's just super straightforward.
48:18
And we think we have all kinds of examples to allow you to get
48:21
that package and deploy it to. Yeah.
48:25
So then it it takes away a lot of those questions of like, oh,
48:30
do I need to bake this model into a Docker container?
48:33
What's going to be the best optimization here so that I can
48:38
optimise for this chip? And yes, but I can see the value
48:43
of that. All of that's been automated
48:44
away and, and to showcase the real breath of it, what we've
48:49
done is we've taken I'd say about the, you know, 150 most
48:53
popular models across all different use cases, speech,
48:56
audio, video, text, vision, small, small language models, AI
49:00
generation, everything. And we've made all of the
49:03
recipes for deploying all of those models available publicly.
49:07
So you, if you go to the website, you can find, you know,
49:09
150 different recipes of how to deploy all of these models in
49:13
the most efficient ways across all of our different chips.
49:16
And all of this has been fully automated.
49:19
Nice. Now here's the real test.
49:24
Is it only Qualcomm chips or have you branched out too?
49:28
Great question. So you know there is in the in
49:32
the mobile and the PC space you typically deploy in a multi chip
49:43
environment. If you're a developer, let's say
49:45
that, you know, I gave you that example of the developer that
49:48
made this ball tracking application, they're shipping
49:51
this model on, on, on Android phones.
49:55
They're shipping the model on different Android phones or
49:58
iPhones or all kinds of different phones, right.
50:00
So at least in the Android ecosystem as, as the, the role
50:05
we play as a leader in the Android ecosystem is to make
50:08
sure that these kinds of applications can deploy their,
50:11
their models across, across various different, SO CS in the
50:15
Android space. So the actual artefact that we
50:20
provide to developers can actually be deployed on other
50:24
Android phones that don't have Qualcomm SoC.
50:28
They, they'll run on the CPU and the GPU, but they run pretty
50:31
efficiently on those non, non Qualcomm SoC phones.
50:36
And the reason that's important is we want to make it super easy
50:40
for a, a, a, an application developer to build 1 model and
50:44
deploy that on, on any Android phone.
50:46
And our viewpoint is it'll run fast on an Android phone, but if
50:50
it has a Qualcomm SoC, it'll run really fast.
50:53
And that's kind of our take on it is like we're not making it
50:56
run slow on other things, we're making it run faster on
51:00
Qualcomm. So we do make sure that from a
51:03
compatibility perspective, especially in Android and PC, we
51:06
are compatible with all the standards, we're compatible with
51:09
the rest of the community and and these models can be deployed
51:13
on other SoC, but they run super fast on the Qualcomm associates.
51:19
That's our sort of philosophy.