Latest / Agentic Conversations (formally mlops.community) / On-Device AI Agents in Production: Privacy, Performance, and Scale // Varun Khare & Neeraj Poddar // #340

Transcript

On-Device AI Agents in Production: Privacy, Performance, and Scale // Varun Khare & Neeraj Poddar // #340 · Agentic Conversations (formally mlops.community)

0:00
Nobody's going to want to talk to their phone, interact with
0:05
their apps if they aren't 100% confident that that interaction
0:12
isn't staying on device. Yes.
0:20
Guys, we wanted to talk about on device AI and machine learning.
0:26
It's been a hot topic and it reminds me of the Internet of
0:30
Things because every time I talk to somebody about on device AI
0:34
and machine learning, they give me the same thing that I hear
0:38
from the IoT folks where it's like, yeah, this year is the
0:44
year it's happening. It's going to happen, I promise
0:48
you. Do you feel like you've been on
0:51
the precipice for the past 10 years now?
0:56
Like in start, so, so definitely like last six years we have been
1:01
thinking, OK, next year is the time when the stock actually
1:07
gets ready and we take it into production.
1:09
But the funny stuff is by the time the stock gets ready, AI
1:13
changes, right. So we used to have traditional
1:16
ML models and you know, the season trees and all of those
1:19
kind of stuff and then charge everybody came and everything is
1:23
Transformers today. Yeah, I think it's a cat and
1:28
mouse game in some sense. You build the stack, newer
1:32
models come, you add that option to those and then finally make
1:35
it possible with at least the genuine models.
1:39
I do think we are close, yeah. I'll just say sometimes the
1:44
selling is way too hard than reality.
1:46
Demetrius and with IoT, that's what happened.
1:49
You know, IoT has worked wonders for industrial and you know it
1:54
has been doing on for the last decade plus.
1:56
But then all the household and the home automation was way
1:59
overtook or so right. Nobody really wants a
2:02
refrigerator in the microwave to be that smart to you.
2:04
So I think that's what happened with on device also, when there
2:08
was no need, there was a lot of selling just because some of the
2:11
stuff was working. I think now with AI, things are
2:15
going to get merged and they're meeting in the right spot.
2:18
There is a need. AI can't be where everyone
2:22
becomes meta and creates A5 GW data center.
2:27
And at the same time, AI cannot be where it is a repeat of your
2:32
search and social era mistakes where privacy is sacrificed.
2:36
So that's where I think on Device AI does have its unique
2:39
space and hopefully it is going to be the beginning of the many
2:44
years of on device AI and it will be more real.
2:47
Yeah, I should say probably it's it might be a little bit
2:53
presumptuous or precarious. I'm not sure the exact vocab
2:57
term I'm looking for here of me to say that we don't have on
3:02
device AI right now because I know that there are tons of
3:06
models that are power powering lots of the experiences that I
3:11
have on my phone. And I've seen as like a meme
3:16
folks starting up their dishwasher machine and it says
3:20
powering up the AII. Think my friend Nick shared that
3:24
and he was saying like what the hell do I need AI in my
3:28
dishwasher for? Exactly.
3:31
Exactly right, It's the same thing right.
3:33
We oversell and every industry when the CI wave they want to
3:36
get some marketing. So might as well introduce AI in
3:39
your dishwasher, in your microwave, in your car key.
3:43
But but you're right, I mean so so I mean Varun has been working
3:47
with on device for now close to how many years on I'll let him
3:50
answer what he has seen the evolution.
3:52
In the ETS, I think, yeah. And to that point, right, so all
3:58
of this IoT ecosystem, the platforms from all the major
4:03
players came out in open source trying to automate stuff, make
4:10
things get going in the factory sides, you know, places like oil
4:16
drilling spaces where it's very hard to get Internet.
4:21
But the entire consumer space, the devices that we have
4:26
ourselves, right? The smartphones, the
4:28
smartwatches, hopefully our glasses that are coming soon,
4:34
that set of devices has still not gotten a platform to even
4:39
find some of the very basic machine learning models, right?
4:43
And I think the primary challenge over there is the
4:46
diversity of devices that you keep on seeing, right?
4:49
So you have iPhones and Galaxies with state-of-the-art chips,
4:53
trillions and trillions of transistors inside it.
4:56
And then you have a nine year old Android smartphone with less
4:59
than two gigabytes of RAM. But the user is equally using
5:03
both of those devices, right? Different people, right?
5:07
So how when you want to unlock on device, you have to build all
5:12
the stack from the ground up while handling this diversity of
5:17
devices and then unlock the experiences that we are talking
5:22
about. So while the future looks good,
5:24
the challenges are in terms of building all the different
5:27
layers of the stack that are needed.
5:31
So early days then, like in 2018-2019 for me, I was
5:37
converting operations between Pytouch and Tensorflow because
5:41
those used to be the major machine learning platforms.
5:44
And we had to run Tensorflow JS for web and by touch or touch
5:52
strip for mobile. So to deploy a model, we had to
5:56
compile it into two different formats from the same source.
5:59
And the only way we could figure out as a 20,000 member community
6:03
was to write a map between each operator from pipers to 10s of
6:07
luck, right? So it's that it was that
6:10
primitive. But fortunately, all of those
6:12
complications have been resolved.
6:14
The major run times have come up for devices that can run equally
6:18
well on different sets of devices, whether it's AI glasses
6:22
or it's a smartphones. That simplifies the stack and
6:28
then you need the next layer of interface which is actually
6:32
interacting with your application.
6:34
So we thought, let's just build it out, make it possible for
6:38
developers and take it from there.
6:41
Maybe now is a good point where we can dive into some of these
6:45
unique challenges. You kind of give us this quick
6:48
overview of what the evolution has been with on device and the
6:53
how the stack changed a bit. But today, where do you feel
6:59
like we're still hitting places that could be optimized or
7:05
they're still inherently difficult just because the
7:08
problem is a tricky 1? So the way so there are two
7:13
parts of the puzzle. One is the text tag Dimitrios.
7:16
Another is from the benefits and the impact side, right.
7:19
So I'll reverse this. I'll talk about the benefits and
7:22
the impact side first. So I believe with AI, you know,
7:26
a lot has changed in our daily way of interfacing with
7:31
technology. We're not Googling as much.
7:33
We are going to ChatGPT and whatever your favorite tool is.
7:37
But when you come to mobile and look at the apps, I can pretty
7:40
much say the apps are still quite dumb, right?
7:43
Like that has been my punchline for the last one year.
7:45
Like why are my apps still dumb? Nothing has really changed.
7:49
My recommendations are pretty much the way it used to be.
7:52
I still can't do NLP searches on half the things, including some
7:56
of the most advanced apps where if I say if I like, if I'm a
8:01
visual thinker and a person who remembers things visually and
8:05
you remember a dance scene and you search about it, nothing in
8:08
the app world can give you oh, this is the movie you're
8:10
thinking, right? So there's a lot of gaps where
8:14
AI has not really penetrated the app landscape the way we want.
8:22
And the impact of that is the apps are pretty much the
8:28
advanced use cases and all are used by people who are
8:31
technology capable. But there is a whole myriad of
8:33
people, whether it's my dad who doesn't interface with
8:36
technology well, and my kid who is 3 1/2 year old, but he wants
8:40
to use the technology, but I'm concerned about his privacy,
8:43
right. So that's the impact side.
8:45
I would say from the tech side, I think our challenges are
8:49
interestingly even more complex compared to the machine learning
8:54
world. So now with AI, the models are
8:57
no longer few kilobytes or few MVS.
9:00
Now we are dealing with gigabytes of models and how do
9:04
we make sure we run them efficiently on devices which are
9:08
inherently resource constrained. And we want to make sure that
9:12
they run like Warren was saying on the whole diversity of
9:16
devices. We can't be the apples of the
9:19
world and say only iOS 18 and the next generation will support
9:23
things, right? We have a whole variety of users
9:26
who already are using their devices, so we need to get the
9:29
experience for them also optimized, right.
9:31
So that's a pretty big challenge.
9:34
And another technical challenge I will add is the developer
9:38
ecosystem where how Kotlin developer or Swift developer who
9:46
is used to creating, you know, Android apps or you know, iOS
9:51
apps, how do they bring AI into their apps?
9:54
Where is that ecosystem? Are they going to write complex
9:59
agentic workflows in Copland? Have you written any?
10:02
It's not easy. And the other part of this is
10:08
typically the machine learning code is written by machine
10:10
learning teams, The AI teams, they write in Python.
10:13
How many of them love writing Cortland?
10:15
Almost 0. So how do we bridge this gap,
10:18
right? So those are two aspects I would
10:21
say both from the impact side and the technology side where if
10:24
you have something more to add. Yeah, I think what I can share
10:30
is what is possible on the device front today, Demetrius.
10:40
And you know, the kinds of technical challenges that are
10:43
there to run some of these models, get the agentic
10:47
workflows out. So one piece is at least
10:51
supporting these 1 billion, two billion parameter models that
10:54
Neeraj was sharing. They should be able to talk to
10:57
you. They should have voice
10:59
understanding inside them, maybe multilingual capabilities, OCR,
11:05
text understanding, vision and visual understanding.
11:09
They might not be great at solving PhD level research
11:13
problem statements, but I don't need to do that on my
11:16
smartphone. So for that cloud is still
11:20
there, all the charge and the trillion parameter models, they
11:23
can do that, but it's about the personalization.
11:26
To your point, a model that is more tuned to me has the memory
11:31
more associated with the kinds of interactions that I've been
11:34
doing and utilizes them to work with all the different agents or
11:41
tools or even the cloud based models to get the final response
11:46
that you want at the end of the day, right.
11:49
And for that you need do need some interesting technical
11:55
capabilities to be solved. It's one of the things for
11:58
instance we open sourced recently was Fast Transformers,
12:04
which helps you run these models 2 times faster but reduce your
12:10
memory consumption by over 30%. Now what that means is you can
12:15
run a big model that is bigger than the available RAM on your
12:18
device. In general, your storage and
12:21
hard disk is much larger than the RAM on the device, right?
12:24
And you don't need all of these billions of parameter parameters
12:29
at the same time, right? So if I'm thinking about food
12:33
and recipes and cookbooks, I don't need a model that is
12:38
explored in history to at least answer the questions in my
12:41
current session, right? So that's what sparsity leverage
12:44
is. It turns off those levels in the
12:47
models that are not supposed to be active for answering the
12:51
current context, which helps you save memory, but at the same
12:55
time increase the inferences speeds because now you're
12:59
computing a lot lesser. So these kinds of optimizations
13:03
are there on the software stack and it's a open-ended research
13:08
problem statement where some of our community members are also
13:12
contributing to actually make these kinds of technology
13:15
possible. And then there is the layers
13:18
above it, basically the technical challenges that needs
13:23
to be solved in this space, you know, and there are research
13:27
coming up possible directions that so we should be able to
13:33
unlock bigger and bigger models on these smaller and smaller
13:37
devices. So it's almost like we have the
13:43
difficulty in the tech of running large models on device
13:48
which is there. But then if I'm also hearing you
13:50
both correctly, there is a difficulty on in unlocking what
13:56
use cases we really want to use these models for?
14:02
Is it in each application we're going to just have LLM
14:07
capabilities so we can talk to it?
14:09
Is it that we want an LLM that is in Overlord and it's watching
14:15
our phone and doing things and then it can have agent calls
14:18
when we talk to it? And so it seems like since it is
14:23
so new, that hasn't shaken out. At the same time, we're still
14:28
trying to figure out like what's the best way to power these on
14:31
device. And so there's these parallels
14:35
that are probably, it's almost like the chicken and the egg
14:37
problem where we say if we can get these models running super
14:43
efficiently on device, the use cases will open themselves up.
14:48
Or if we know one specific use case, then we can optimize it
14:54
for running on the device. And I wanted to mention one
14:59
thing around this memory piece, which I find fascinating.
15:02
And and I wanted to just dive in how you're doing that because I
15:06
had heard that one path forward was just hot swapping out
15:11
Laura's in the moment of doing it it or did you take that
15:17
approach or is it a whole different route?
15:20
Actually it's a mix of some of these approaches that you
15:24
mentioned to to us. So 1 is when we think of multi
15:29
agent systems running on these mobile phones, right?
15:32
One is more like a Marcus marketplace and an orchestrator
15:38
agent, which is working for you across the context of multiple
15:42
applications, right? Which is kind of looking as a
15:47
supervisor of all the different agents.
15:49
And each of the mobile applications have agents inside
15:53
them. And both of these agents are
15:55
interacting with each other, communicating whatever they want
15:59
to do. Let's say if you ask the
16:00
assistant to order, order food for you, it can go to your food
16:04
delivery app. And then the agent inside that
16:06
will figure out the right preferences based on what you
16:09
have bought in the past suggested back to the
16:12
orchestrator agent and then show it to the user.
16:15
And so all of these, there are these multiple agents that would
16:20
be running, but interestingly, the LLMS that would be powering
16:24
all of these agents, they'll be similar same behind it, right?
16:27
Because you have to fit off 3 billion, four billion parameter
16:31
model on a device. You cannot have multiple LLMS in
16:33
those cases. So in those places we do need
16:39
Lora adapters in some of the cases where you we want to
16:42
personalize the responses of the model to add some additional
16:46
information or generalization about the domain that that
16:49
particular agent needs to work upon.
16:52
But another aspect is more of the general model capabilities
16:57
itself, right? So you don't have to build an
17:01
explicitly train load adapters. Every model has this inherent
17:07
clustering of weights in some sense and you can dynamically
17:11
choose. There are these mixture of
17:13
export models that are coming up right where there are 10 or 20
17:17
exports and each export is only maybe let's say 200 million
17:21
parameters in a trillion parameter model case.
17:24
But you choose one export and only that goes into your memory
17:27
and everything else is out. In Flash, you use the same
17:33
approach just to increase the number of exports from 10 and 20
17:37
that we see today to something like about 100 million experts.
17:44
These are all smaller experts. I was going to say like a
17:48
hundred 100 million experts. Oh my God.
17:52
When you're talking about a billion parameter or 2 billion
17:55
parameter model, and then you have to cluster those into
17:58
topics, the topic clustering itself becomes a 200 million
18:03
parameter model in that case. But it does give you the
18:09
advantage of turning off your memory, turning off unnecessary
18:12
compute that you were running. Yeah.
18:15
I would just add here, Dmitrios, that the question that you were
18:19
hinting at is very interesting. Where, how will this shape out?
18:23
Like like we can't imagine every app has its own LLM, but we
18:28
can't imagine every app have its own set of agents.
18:32
We think that the future is going to be like, if you think
18:37
about multiple steps it will take, I think multiple apps will
18:40
come up, which will be AI native.
18:42
First. They will have agents running,
18:44
they'll deliver experiences to the users which are driven
18:47
through LLM, some private on device and some with the cloud.
18:51
And then eventually, you know, you do want the interface of the
18:57
phones and maybe not even the phones.
18:59
Maybe there's a future, a different device to through a
19:02
different kind of an assistant. The assistant is your primary
19:05
interface which decides which app to dispatch a particular
19:10
action. So in that case, that assistant
19:13
is not like the Overlord AI, but it is an AI layer that has the
19:17
Super intelligence, it has the memory across apps and it can
19:21
kind of lease out its capabilities to the agents
19:25
running inside the apps. So this kind of fundamentally
19:30
changes the way the apps have been working in the phones.
19:33
And I believe like the apps, the way they work in the phones are
19:36
the same way the apps have worked on the laptops and our
19:38
desktops, then nothing has changed, right?
19:41
You go to a particular app, you finish your work.
19:42
If you have two apps, you decide, oh, I'll go in one, then
19:45
I go in another and then do your work.
19:47
And then, you know, Steve Jobs very nicely added the Sheen of
19:50
that screen, which was great. So now we can use our fingers to
19:55
interact with it. But still, if your dad wants to
20:00
book a cab and he's used to making a call, now he hates your
20:06
phone because he has to go through Uber and Uber changes
20:08
its screen with the new promotion.
20:09
And then he's lost whatever. He just wants to do the same
20:12
thing he was doing in his 90s, which is like, hey, in the
20:15
night, be like, let me just call.
20:16
And in the technology I need can use Uber, you can use Dyft or
20:20
whatever it is. So I believe that.
20:22
I really hope that happens, you know?
20:25
That that we create this interface which becomes the
20:28
personal interface depending on how capable you are and the AI
20:31
adapts underneath to your needs. Yeah, there are a few points on
20:35
that that I have been pondering quite a bit.
20:39
And the first one is, if all of the apps are going to be OK with
20:45
succeeding this power to the overlord agent type of thing,
20:53
because then they lose out on you going to their application.
20:57
They're just almost like a secondary LLM call or a tool
21:02
call. And I know it kind of is likened
21:07
to the way that agent to agent framework wanted to make it
21:12
work. And so there is movement, there
21:17
is energy going in that direction.
21:19
It just feels like if I'm Amazon, I've now lost the
21:24
platform. I'm just a tool call.
21:27
I'm not the platform that you go on and you search around and you
21:31
end up buying a bunch more stuff because my recommender system is
21:34
so good. And I find that really difficult
21:39
to see Amazon just giving up. Yeah, so Dimitrios with the
21:44
Internet has happened, right? Think about it.
21:46
I used to own a website where I used to put recipes.
21:50
I used to stuff the recipes at the bottom and in the beginning
21:53
I used to have all the story and the nonsense about how.
21:55
It ends. And eventually, when it became
21:59
popular, I used to monetize it. How do you monetize it now with
22:02
ChatGPT, right? How do you monetize it with
22:06
perplexity? So I think the resistance here
22:09
is just that the reward has to be high enough that there might
22:15
be a net new bead of apps coming which will displays a lot of the
22:20
existing apps or the existing apps will evolve in this
22:23
ecosystem. I think the bigger problem I
22:26
believe might be from the OS Windows, honestly, how will they
22:31
react to it? What kind of control are they
22:33
willing to give in this new AI native experience?
22:37
And because it's very hard for the AI for the apps to trust the
22:43
OS with all the personal data also on how will they monetize
22:48
it because many of. Them and even, Yeah, you don't
22:52
really expect it not to be done by the OS, but everything we've
22:58
seen from Apple so far shows no inclination of being able to
23:04
pull something like that off. Yeah.
23:07
And there's a bigger piece of trust and privacy there, which I
23:11
believe a lot has to play with. Whoever wins this will be
23:15
Vaughn. Because it started being an
23:17
open, it has to be open sourced. The primary objective of this
23:22
platform and the assistant cannot be that you're directly
23:25
monetizing it. You're enabling this ecosystem
23:28
and people are trusting it because they know it's open.
23:31
Just like, you know, how Google displays a lot of the things
23:35
with the Android ecosystem and the Chrome ecosystem, the open
23:38
nature of things was a lot to do with it's, you know, popular
23:43
popularity in the beginning. So I think open source the
23:46
benefits and then I fundamentally hope that there's
23:50
a net new beat of apps coming which are going to displace a
23:54
lot of the existing apps. Well, diving into the apps and
23:59
as you mentioned, like AI first apps that come up, which I think
24:04
for all of us is still a little bit intangible to think what
24:08
that looks like. And we, it's hard to imagine
24:11
rethinking Jira AI first. It's like, OK, is that linear?
24:15
I I guess maybe we'll. Yeah, I know.
24:19
Yeah. But besides that, you mentioned
24:24
it before and we got to call it out, every single app is not
24:28
going to have a model because I can have three apps on my device
24:34
before I fill up all of my storage or all of my
24:37
capabilities, unless the hardware capabilities just
24:41
explode or like you guys are saying, the model capabilities
24:46
are able to distill down and you're able to really take
24:50
advantage of these models. So I think now it's probably a
24:54
good moment to just talk about what some of these constraints
24:57
are. If you are building an app that
25:00
is AI native and you are expecting it to go on the phone,
25:05
it's going to be LLM or AI First.
25:10
But you can't just have a reasoning model there, can you?
25:14
You can offload some stuff to the cloud when you need to, but
25:19
is A1B type of model going to give a good enough experience
25:24
for an AI First app? Yeah.
25:29
So I can take a Java dip here as you please add.
25:33
So Dimitrios, I think it depends for instance how the what AI is
25:39
actually controlling within the application, right.
25:43
So there are different layers just like you mentioned, but for
25:46
instance, you just at one level want multimodal, multimodal
25:52
interactions with the application, right?
25:54
You should be able to talk to it, but at the same time
25:57
interact with all the UI and the widgets.
25:59
Now there is a smaller model that is just transcribing what
26:02
you're seeing into an LLM, generating the response,
26:07
understanding what you said right, and respond back to your
26:11
invoice. Now there is a full loop of
26:14
multiple LLMS or AI models running on devices to get a full
26:21
duplex conversation going with you, right?
26:25
And in terms of the experience that gets unlocked.
26:31
So there was this example that came from Tinder actually.
26:36
So the standard left swipe and right swipe matching your
26:40
approach. This swapped it out with a
26:43
gamified version, and in that game the AI was more like the
26:49
orchestrator of the game, asking the users to engage more with
26:54
each other to actually solve the puzzle of the game.
26:58
And that was a way for them to, you know, get more engagement
27:03
going amongst the people. They understand each other
27:06
better and take things from there.
27:08
Now this is just strapping on where the decision making is
27:12
happening. And in this case that AI model
27:15
doesn't need to be a very complicated model, right?
27:18
It could be a simple model just understanding OK, what the guy
27:23
is saying, what person B is saying and how they are
27:26
matching, what kind of puzzle should I throw for them to
27:30
solve. Think a very simple, simple,
27:31
simple task, but you get a bit more different experience.
27:37
Banda Snatch, I think it's Netflix on Black Mirror.
27:40
Banda Snatch, they tried with this, right?
27:42
They had multiple storylines, you interact with it and the
27:45
story changes. Now, I assume all of these kinds
27:48
of interactions are also getting mobilized by the AI agents who
27:54
are working within different context, working over your
27:56
memory and your preference to do this.
27:59
I'll just add, I think we are what we see today based on our
28:03
experience and working with the apps.
28:06
Around 1 billion parameter models are pretty good at
28:09
summarization Right now at 1 billion parameter models, we
28:15
don't see reliable tool calling and agentic workflows.
28:18
You end up using around 1.5 to 1.71 point 9 billion parameter
28:23
models where it is not hallucinating to pick up tools
28:27
and then incorporate the response of the tools in
28:30
orchestrate more tools. And then the sweet spot is
28:33
probably around 2 to 2.5 to 3 billion parameters where
28:37
hopefully we'll get multimodal models like Warren was saying,
28:40
which will be full voice and maybe even some of the OCR
28:46
capabilities which are all big into the model.
28:49
So you don't have to worry about creating a pipeline with an ASR
28:54
11 or TTS yourself. So we do think, I think in the
28:58
next 6 to one year, six months to a year, we will see some of
29:01
these models coming like Quinn had already released very
29:03
powerful models in Quinn 3, which are pretty good in doing
29:09
all sorts of tool calling agentic workflows we see
29:12
already. So the next step will be how can
29:14
we pack multi modality, especially voice in voice out
29:18
duplex probably in a 2.5 billion range.
29:23
I like this vision because as we were mentioning, some of the
29:28
hiccups here are that we don't have use cases clearly defined.
29:33
And one that you mentioned that could be a potential future is
29:41
when we're able to talk to our app just as much as we're
29:45
touching the screen and the app is doing things and then we can
29:50
use the touch screen when we want, but we can also talk to it
29:54
and it understands it. So far, every time that I've
29:58
interacted with Siri, the assistant, it's fallen flat on
30:03
its face. So I'm very skeptical that
30:05
that's actually going to be a reality.
30:08
And I I almost feel like Siri was better before and somehow it
30:13
got worse. It did it did it.
30:17
Did. So I think there.
30:18
Are two parts here. Dimitrios 1 is sometimes when I
30:23
come and I want to interact with my phone, my intent is very
30:26
clear. This is what I want to do.
30:28
So in that case, I would like my phone to act like a real
30:32
assistant so that they're smart, they're capable, they can fill
30:37
in the dots based on what my personal history is a pre memory
30:41
is so none. If I say I have to fly to US
30:45
tomorrow, it should already know what kind of airlines are like,
30:49
what times I pick. I have to book my Ubers, I do my
30:53
hotels, this is my work. I want my breakfast done, all of
30:56
the nonsense. I don't need to deal with this.
30:58
So that's what I meant. And intent is clear.
31:01
And with the user preferences and users understanding, a lot
31:06
of this can be done across apps by an assistant if you have the
31:09
right stitching done. The second part is I don't know
31:14
what I want to do. So that is where we are doing
31:17
exploration and sometimes doom scrolling, right?
31:19
Like Netflix is one of them where we are continuously do
31:22
scrolling nowadays, right? We don't know what we want
31:26
honestly. That UI and UX designing and
31:29
thinking is the next, I think frontier for us.
31:33
How do we enable AI LED exploration where we can
31:39
seamlessly transition from like an assistant to an actual app
31:44
and give them the space and then come back to it?
31:47
Because as you are exploring, the user will be forming that
31:51
intent and once that intent is formed then you can finish the
31:54
action. We have to enable exploration
31:57
through AI and that interface doesn't exist today.
32:01
I do want to have the ability when I'm scrolling through
32:07
Netflix to say like, I'm looking for a Western, but like a modern
32:11
Western. I like this show and that show,
32:15
and then I'm still scrolling and it's populating it with things
32:19
that I have. But I understand the vision too,
32:22
that you're saying nobody's going to want to talk to their
32:28
phone, interact with their apps, if they aren't 100% confident
32:33
that that interaction isn't staying on device?
32:38
Yes, because we already, we're already like convinced that our
32:43
devices are listening to us all the time.
32:46
Now, when we actually start interacting with it like that,
32:50
it's going to be really weird when we're thinking like, OK,
32:54
everything that I'm saying here, how does it know when I'm
32:56
talking to it about my recommendations or when I'm
33:00
talking to my wife about whatever logistical things in
33:03
the house? Yeah.
33:05
I think one of the interesting pieces on the application side
33:12
is also how do you make these apps work, especially in these
33:18
kinds of explorative use cases. Coming back to the point, you
33:22
mentioned to matures earlier, right?
33:24
Why would apps want to do this? So in this kind of an interface,
33:29
it's a merged experience across the application and the agency
33:33
open up your assistant, but it then actually fires up the
33:36
application which evolves your intent, helps you explore, work
33:41
towards the directions, all for which it will require data which
33:46
is proprietary at least for that application.
33:49
So that agent has to run within the context of the application.
33:53
It cannot be that orchestrator agent in that case.
33:56
And that's where both of these intra app and inter app
34:02
communication needs to be orchestrated together.
34:05
Yeah. And and like so I think for me
34:11
inherently like an OS vendor doing it will have its own
34:15
problems because first of all, we have two giant OS vendors and
34:19
trying to find commonality in AI or workflows between them is
34:23
going to be next to impossible. And then which app is going to
34:27
give them access to their internal data is going to be
34:32
another. Problem.
34:35
So this has to be something outside and like I was
34:39
mentioning, you know, before we still have to think about this
34:43
swapping of context that is going to happen both within the
34:46
LLM and then within this apps like from an assistant
34:49
seamlessly going to an app. Even the app should be more
34:54
adaptive, right? So one of the things that we did
34:57
as part of deploying earlier machine learning models,
35:00
Dimitrios is hooking each and every interaction of the user.
35:04
As the user is navigating through the app, the the
35:07
impressions and the clicks are getting captured on the device
35:11
and it gets captured. Oops, sorry, my light is
35:14
flickering and it gets captured on the device right as the
35:18
events are flowing. And in Python, you can do
35:21
feature engineering on it to understand what the user is
35:24
trying to do and run machine learning and AI models to
35:27
provide that level of personalization and
35:29
recommendations. Right.
35:31
All of this also is nearly impossible to do it on the cloud
35:34
because the amount of firehose it takes to get this click
35:38
stream event, stream data into the cloud, then run it and then
35:41
bring the output back at scale of 50 million, 100 million users
35:45
is just impossible. Doesn't work.
35:47
Doesn't work. Doesn't work.
35:49
Yeah, it's scaling, it's speed that I'm not sitting there for a
35:53
little bit and thinking I wonder what would be a good movie.
35:57
And it's populating for ages. I do, I do find the level that
36:04
we play at again, I, I'm keep coming back to it where what
36:10
would be really nice is that I'm not locked into Netflix when I'm
36:15
looking for a movie. And I have my Netflix and my
36:19
Apple subscription and my, all my different movie catalog
36:24
subscriptions that the agent could go pull from when I say I
36:27
want something funny, However, like Netflix is never going to
36:33
want that because then I'm not locked in to Netflix.
36:37
And, and so I, I think that you raised two really good points
36:41
that Netflix isn't going to want Apple or Google to be getting
36:46
all their data and having all of that information.
36:49
And Apple and Google also, we inherently aren't necessarily
36:55
going to want them to be having all that information.
36:57
So it feels like there's this utopian world that we could be
37:01
living in that would be amazing, and we're trying to push towards
37:06
it. But potentially the hardest
37:10
challenge here is going to be user adoption, or everybody
37:15
getting along and playing nicely together so that the users can
37:19
have that utopian experience. Absolutely.
37:22
I mean, at the end of the day, right now we I've been
37:25
sacrificing on the user experience and user privacy
37:28
primarily to make sure that the apps have that hold over us,
37:32
which I understand all of them are running a business, but they
37:36
are also inherently making sure that their computers are not
37:40
enabled, which is the main driver.
37:42
I think if we eliminate that piece where if it can be a layer
37:46
which is open source driven, which is built by the community
37:50
and the information stays on device, nobody's taking that
37:54
information out, right? So the the level of
37:59
personalization and details that Netflix has to give will only be
38:03
given to the assistant and assistance context to make the
38:07
recommendations. And depending on the level of
38:10
corporation and the level of details they provide, assistant
38:14
might be incentivized to invoke Netflix rather than Hulu, right?
38:18
So they can be very interesting rewarding behaviour here also on
38:24
how you build this new app ecosystem.
38:28
In fact, we can start even thinking about how do we reward
38:32
the users for the data, because some of them might want to give
38:37
it for personalization to the apps.
38:41
And now you give the control to the user.
38:43
Some of them might be, I don't care, give everything away,
38:46
fine. But for people who care, they
38:48
can even monetize that piece so we can build a whole new way of
38:53
not just interacting with the technology, but also making this
38:57
whole experience so much personal and accessible that I'm
39:03
not. For me, it's very exciting just
39:04
because I feel like, you know, when, when I grew up, for me,
39:08
mobile was the new thing. Like when we saw mobile, it was
39:12
like, Oh my God, the things have changed.
39:13
And then the social era came even more, but then it has just
39:16
stagnated for the last 6-7 years.
39:18
We just keep, we are cranking out the same thing in your, you
39:22
know, like a lipstick on a pig thing.
39:25
Going on. Yeah.
39:28
I just wanted to add 1 interesting thing on what
39:31
happened did recently and how inefficient it was.
39:35
You know, so just to get this context on what an application
39:39
is doing or the user is doing on that application, Apple built
39:44
models where they were screenshotting your app and then
39:48
doing the OCR over it to understand the context.
39:53
Now, because this happens as an OS, because you are seeing every
39:57
application as a binary, right? So it's running within its own
40:01
container. So the only way for you to take
40:03
some information out of that app is to screenshot the app, right?
40:08
But on the other hand, if it's an open source community working
40:11
with the applications and the application developers as the
40:14
first first focus, they can build integrations directly
40:21
inside the application. Which means that you don't,
40:24
first of all need a computer vision model or multimodal
40:28
capability, right? In that case, the app is
40:31
filtering out the data, doing the right things, and giving you
40:34
that compressed information that even a smaller model can handle.
40:37
Yeah, I mean, think about what we're doing on the browser
40:40
automation on the desktops. We have to literally take
40:44
screenshots, figure out the pixels and then automate them.
40:48
And now, you know, publicity rightly is waking up and saying
40:53
this can't be the way. And that's where they came out
40:56
with Comet. So I think similar ecosystem has
40:58
to emerge in the mobile landscape.
41:02
Demetrios and the primary interactive engine is not going
41:07
to be a browser because nobody's going to the browser on the
41:10
phone. So it can be a new AI assistant
41:13
which can unlock this. So we'll see.
41:15
You're right, there are lots of challenges here, which is
41:17
exciting. It almost feels like you would
41:19
need a new OS. That's where the world is
41:27
heading, you know, and every often new technology comes
41:32
because of which we'll have to rethink things.
41:34
I think cloud native does this for a lot of things.
41:37
I think for AI it's not far fetched that for the tools that
41:43
you have been using that are the most important tools, you'll
41:46
need them in AI native ways where a native browser, a native
41:49
OS an an AI native marketplace. Interestingly, on that front,
41:58
what Neeraj was mentioning, right, So Andre Karpati recently
42:01
was talking about software 2.0, software 3.0 and how this thing,
42:08
I think that will happen for OS as well.
42:11
If not a separate OS, at least AI layer of OS should definitely
42:15
come up right? And that would be the
42:18
three-point O version there. Rather than writing Kotlin or
42:21
C++ code, you are prompt engineering the entire thing
42:25
out. Yeah.
42:27
It doesn't feel that far fetched.
42:29
I think one question I've got for you guys who are deep in
42:32
this space is these different constraints on device.
42:36
And I know that we've talked to Qualcomm folks back in the day
42:41
and they mentioned how when you are playing at these on device
42:47
levels, you need to be picking up not only the size and just
42:53
how much storage you've got and how much memory you've got, all
42:57
of that like the engine capacity, but also you can do
43:05
operations that drain the battery really quickly.
43:09
And so you want to be looking at all the different vectors and
43:15
maybe you've seen not only with the phones, but with other IoT
43:20
devices we could say that have had AI enabled in them.
43:26
I'm thinking about like factories primarily.
43:29
How have you tried to play around with maximizing the
43:36
models on every vector? Right.
43:39
So I can share some early thoughts on this.
43:43
SO at least on the IoT world, I think one biggest difference
43:50
between designing stack for IoT and Android slash iOS, the
43:54
consumer world is the Control Center flips, right?
43:59
In IoT, there's a central entity, the hub, which is
44:03
orchestrating all of these sensors which are continuously
44:05
generating some data, right? In this case, the user who's
44:10
sitting with the device is the one who's actual source of data,
44:14
right? So your API designs, the control
44:17
centers, they flip when you move from IoT to the consumer world.
44:22
And secondly, some of these optimizations that you
44:30
mentioned, those are the hardest pieces in managing the
44:33
diversity, right? Some people will have strong
44:36
Internet connectivity, but as soon as they sit in the car and
44:39
start moving, their Internet connectivity starts breaking
44:42
down, right? It's much easier to do battery
44:47
intensive tasks when the phone is on the charge at night rather
44:52
than doing it while it's being heavily used for that server
44:56
video call, right? So that is what a platform that
45:01
is specifically catered for consumer use cases can handle as
45:05
compared to an IoT platform. And we do all of this management
45:11
for the applications, so they don't have to worry about.
45:14
We directly look into whether this model needs to run on CPU
45:17
or an NPU or a GPU and what's getting bottlenecked and
45:22
throttled right now. One of the interesting things,
45:25
for instance, we saw it's more expensive for the battery to
45:29
make a API call to the cloud than to run a 300 KB or a 500 KB
45:36
decision tree model on device well.
45:39
Isn't that interesting, huh? Just another reason to optimize
45:45
on device. Exactly.
45:47
And you cannot come across these challenges unless or these
45:51
optimizations unless you actually go in production and
45:55
deploy these systems out to understand these terms.