Latest / Agentic Conversations (formally mlops.community) / AI Agents for Consumers // Shaun Wei // #244

Transcript

AI Agents for Consumers // Shaun Wei // #244 · Agentic Conversations (formally mlops.community)

0:00
I'm Sean and Sean Wade. I'm the CEO of CEO of Real Char
0:06
and I take my morning coffee with with old milk with I Do you
0:15
do half, half, half coffee, half milk?
0:17
I do like canned sugars and I like to light my days off like
0:24
and have little bit sweet in my mouth.
0:28
What's going on, everyone? Welcome back to another ML OPS
0:30
Community podcast. I'm your host Dimitrios and
0:35
today we're talking with Sean all about what he is working on
0:39
his product Rivia and the company's Real Char Oh my I'm so
0:47
excited by what he's doing because it is a consumer
0:50
application using Gen AI and traditional mill and he
0:56
basically holes opened the curtain to show us what and how
1:02
they are combining the two. I cannot say I fully understand
1:09
what exactly he's doing, but he broke down for me after we
1:14
stopped recording. Of course, this idea on how in
1:20
robotics it's very common to run everything in parallel and look
1:25
at something. I think he called it Design O
1:28
and everything needs to be done in parallel so that you can
1:32
scale and so that you can do things quickly.
1:35
Now his product is a consumer a that you can basically use AI to
1:41
call different customer service lines and have a I take your
1:47
customer service burdens off your shoulders.
1:51
So we as consumers are getting put up to the task of having to
1:58
deal with a bunch of shitty AI. Why don't we fight back by
2:02
giving all of these shitty implemented AI, probably a
2:06
little bit half baked AI solutions that are everywhere
2:11
with enterprises and companies. Why don't we start to implement
2:16
our own? And so that's the idea.
2:20
You get AI to make calls for you, wait over three hours on a
2:26
customer service phone call and press buttons for you.
2:29
If it's like if you'd like to go to whatever be transferred to
2:33
this, then press 1. So they're doing incredible
2:36
things with that. If you want to check it out, you
2:39
can go to revia dot tech slash signup.
2:43
We left the link in the description.
2:44
I know I've already signed up because I get the feeling it's
2:48
not going to be fully polished. He said it in the call right
2:51
now. He said, you know what, it's not
2:53
like we are 100%. We're basically if you analyse
2:57
this like a self driving car, we can get the car to drive
3:01
straight on the road for 15 metres.
3:04
And so going into it with the expectations that it's not
3:09
polished, but it is going to be something that is immensely
3:14
valuable if it can at least offload 10 percent, 15% of those
3:20
phone calls that you have to make that you don't want to make
3:23
because they're unnecessary. All right, that's it.
3:28
Let's get into it with Sean. As always, if you enjoy this,
3:31
share it with one friend so we can get the word out about the
3:34
good old Milos community. And Spotify has that cool
3:37
feature where you can leave comments.
3:41
I read them all, the good and the bad.
3:42
Actually, I love the bad ones. Talk as much shit as you want.
3:45
To me, that is my daily fuel. I'll see you on the other.
3:50
Side. O Sean, dude, I'm excited to
3:59
talk to you again. I know that we had our first
4:02
conversation on the Deep Grand podcast that I run and we got to
4:05
talk about your founders journey and being a founder and what it
4:09
means to develop AI products these days.
4:13
But I told you, come on the ML OPS Community podcast because I
4:17
had such a great time talking to you and learning from you.
4:21
And your experience in being like in the weeds on the
4:27
engineering is so valuable. And so being a technical founder
4:31
that you are, I thought you would be able to still talk to
4:34
us about coding. And some of the stuff I'm really
4:39
putting AI into production is I think the theme of today's chat
4:44
that we want to talk about. But as a quick overview for
4:47
those who did not listen to the AI Minds podcast that we did, I
4:52
want to let everybody know that you were working on Google
4:56
Assistant and then you transitioned from there.
4:59
You said, you know what, I'm going to go do something a
5:03
little bit more audacious and help build self driving cars.
5:07
You were doing that for a while and I guess all the while you
5:10
had this bug that was telling you there was something there
5:15
when it comes to Google Assistant.
5:16
And so now you started your own company.
5:18
You guys are about to be featured in the next YC batch
5:22
and it's all about assistance for humans.
5:26
And so let's just breakdown a quick overview of what the
5:31
product is before we go into the tech of it.
5:35
Yes. So what are we build is called
5:37
the Riviera REVIA and we are like soon to have a version on
5:46
App Store. So I think by the time you
5:50
listen to the podcast, you should be able to download that.
5:54
But we're still doing like a small beta testing right now,
5:57
and hopefully I can give this assistant to power your daily
6:02
life. Yeah.
6:04
And it's is it only in the US? It's it's the App Store.
6:09
I think we can open it up to all the like regions.
6:12
Ohh, that's dude. So that's ambitious, right?
6:14
Because there's so many different cultural things and
6:17
languages and pieces that you have to deal with, especially
6:21
when it comes to me not knowing. And basically you're making it
6:27
open-ended. You're saying that this tool can
6:32
do anything that you want, right?
6:34
When it comes to like dealing with customer service type
6:37
issues, phone calls with products or product companies.
6:42
Like I think about my mobile company, they call me all the
6:44
time trying to offer me upgrades and I just want to know what the
6:48
upgrade is. I don't want to sit on the phone
6:49
call with them for a long time. Or if I need to get my plumbing
6:53
redone or I break a pipe or something springs a leak in the
6:58
house and I need to have a plumber come by.
7:01
You can go and have the AI assistant call the plumber and
7:05
schedule a time for them to come over and you just get a text
7:08
message that says plumber will be over in again tomorrow or
7:13
Tuesday between 4:00 and 6:00, something like that.
7:17
Ohh yeah, that's the hope, right?
7:19
So like, ohh, based on my past experience, like that's people
7:24
really want like they want using the I to replacing you in those
7:29
like really boring phone calls. So that's the mission.
7:32
Like we're building on this. Like people should be spending
7:35
your time on meaningful tasks with your families, with your
7:40
work. You don't want to like really
7:42
spending two hours like dealing with like the plumbers.
7:45
They always, like you, are like utility companies.
7:50
None of them, yeah. Like the least amount that I
7:53
have to be organising my life, the better.
7:57
Yes, that is for sure, dude. And that's the vision.
8:02
There's a tonne of technical challenges that you have with
8:07
this. So why don't you walk us through
8:09
what some of these have been and how you've taken?
8:12
Like, I imagine you had a tonne of learnings when you were
8:14
working on Google Assistant, and that was back in the day.
8:18
I remember being so inspired when I saw that.
8:20
And I remember a specific point in my life when I saw that, and
8:25
I thought, wow, the world's going to change if this is real.
8:30
Yes, I think the the court system cability you were
8:35
referring to was Google Duplex. So it's gonna be tight announced
8:41
the Google Duplex in Google IO, I think 2018 when this AI was
8:47
supposed to representing you to make phone calls to hair salons
8:52
for you, book like restaurant for you.
8:55
So I was actually on both ends, right.
8:59
So on the building and on the receiving end.
9:02
So I interviewed people from both ends.
9:05
So what I found was firstly, you know, during 2018, we're trying
9:11
to build an app and in our model that can handle natural
9:15
conversation in a phone call setting.
9:18
It's extremely hard. You have to have tonnes of
9:22
labelling data to handle a lot of corner cases when you're
9:28
trying to make it sound natural. So it requires a total training
9:32
and, and it's spending like like a larger team actually just
9:38
making that two use cases and it may not work most of the time,
9:42
right. So on the that's on the engine
9:46
side, like during that time, it's really hard and the we
9:51
don't have a larger model by that time.
9:53
So how we, what we did was trying to do a lot of
9:59
classifications, right? Whenever you're trying to say,
10:01
OK, I want to book a hair salon in this location time O we
10:06
capture those intent trying to understand the user intent and
10:10
map those to the actual a sentence, right?
10:15
So the audio bytes. So during that time we were
10:18
using I think or Ireland models trying to do that.
10:26
And there's another piece in the in the front to do the internal
10:29
classifications. Then there's like the speech to
10:32
text or text to speech. So in the end, but so yeah, R
10:38
was like running extremely fast. So that's like a advantage over
10:44
like the the larger model, right?
10:47
But on that part is like, since there's so many intent you have
10:50
to capture, right? There are so many quantum cases,
10:53
why interview the other end? Like we're receiving the phone
10:56
calls. They're very, very interesting.
10:58
Like for them, hey, it's a little bit odd in the beginning,
11:02
right? There's like starting, there's a
11:06
starting calling them, you know, that's the first experiment they
11:08
starting to experience. Ohh, there's a, I actually
11:10
holding another conversation with them.
11:12
Then soon they realised the a, I wasn't able to handle all the
11:18
cases like humans. So one of the example they gave
11:21
me was they were a hair salon. So they were trying to receive a
11:26
phone call from like the Plex saying OK the client will want
11:31
to adhere cut. So, but when they ask what type
11:35
of haircut do you want, like a female like haircut or like a
11:39
male haircut, that's when the system break.
11:43
So, so that's why like the during that time, the NPU wasn't
11:49
there to understand all the like small, smaller cases.
11:55
That's what eventually break. And I think they were eventually
12:00
receiving like human phone calls.
12:02
So they were, I think Google might be using like human to
12:06
represent the, the users to call them.
12:09
It's just like so hard during during time, we realised that
12:12
people really want to kind of a hand off this phone call
12:17
experience to the assistant because those are really boring
12:21
and like the phone call by nature is a blocking experience,
12:26
right? When you're making phone calls,
12:28
you cannot do anything. So people really hate that
12:30
experience. So that's why I got a lot of
12:32
inspiration from Google to Facts when they made that S like work.
12:36
All right, real quick, let's talk for a minute about our
12:38
sponsors of this episode Making it all Happen.
12:41
Lattice Flow AI Are you grappling with stagnant auto
12:45
performance? Gartner reveals a staggering
12:48
statistic that 85% of models never make it into production.
12:51
Why? Well, reasons can include poor
12:53
data quality, labelling issues, overfitting, underfitting and
12:58
more. But the real challenge lies in
13:00
uncovering blind spots that lurk around until models hit
13:04
production. Even with an impressive
13:06
aggregate performance of 90%, models can plateau.
13:10
Sadly, many companies optimise for prioritising model
13:14
performance for perfect scenarios while leaving safety
13:19
as an afterthought. Introducing Lattice Flow AI, the
13:23
pioneer in delivering robust and reliable AI models at scale.
13:28
They are here to help you mitigate these risks head on
13:31
during the AI development stage, preventing any unwanted
13:34
surprises in the real world. Their platform empowers your
13:37
data scientists and ML engineers to systematically pinpoint and
13:41
rectify data and model errors, enhancing predictive performance
13:46
at scale. With Lattice Flow AI, you can
13:49
accelerate time to production with a reliable and trustworthy
13:52
models at scale. Don't let your model stall.
13:56
Visit Lattice Flow Dot AI and book a call with the folks over
14:00
there right now. Let them know you heard about it
14:04
from the MLS Community podcast. Let's get back into the show.
14:09
All right. So moving on from from that
14:11
Google experience, you, you still set out to create
14:19
something that is more open-ended and that just feels
14:22
like really hard. How have you been able to deal
14:27
with some of these issues that you encountered back in 2018
14:31
when you were doing this now? Is it just OK cool NLP is much
14:35
better with Transformers and open AI or is it a much more in
14:41
depth problem than that? Yeah, I think ohh, the the first
14:45
experience I'm learning from Google was before a lot of
14:52
scenarios cases you have to do like human labelling or you have
14:58
to have some type of fall back mechanism, right?
15:03
If it doesn't work, if you still want to get this work done, you
15:07
have to have some type of fullback.
15:10
And yeah, I also learned how how how human interaction with AI
15:18
for the first time, right? What are the like how they
15:22
actually train themselves to be able to understand like AI?
15:26
Then I think it was the 2nd experience of my building self
15:30
driving cars that really helped me to to understand to deploy an
15:35
AI in an actual world, right? Thinking about.
15:40
So, you know, we can talk more about like what I have learned
15:44
he's running cars and or if there's any topic you want to
15:48
like me to like discuss a little bit more.
15:52
Yeah, I would love to go into that because it, it basically
15:54
went from like now you're learning about, you're almost
15:58
learning about the the way that humans interact and the design
16:06
piece, the product piece. And then you went into the
16:12
technical piece and you were like, alright, how do I deploy
16:14
this and how do I make it super fast, right?
16:16
Because with self driving cars you can't be slow.
16:20
Yes, yeah. So you know, for for the cool
16:24
experience, I learned like the problem, right and with the
16:26
potential solution. So then in the industry when I
16:30
was like in polling, that's when I really learned how to actually
16:34
solve this, do things like for submarine cars.
16:38
But most people probably don't realise this.
16:41
We'll think about self driving car.
16:43
It's pretty much the most advanced AI integrated systems
16:48
humans have ever deployed on the road, right?
16:51
It's trying to negotiate with humans, behave like humans in
16:55
trying to replace a human driver in a way like using all the
17:00
different models. That's that's like what we have
17:06
been building for the last couple of years, whether we move
17:09
or the all the company has been building, it's trying to replace
17:12
or at least mimic how human interacts, traffics or like
17:19
control. Like they like cars on the road.
17:25
A few things. Ohh, units are running cars.
17:27
Like I really have learned our first like how do you simulate
17:35
an environment that's safe in a virtual world, but still can
17:40
representing a actual human world, right?
17:43
Because like it doesn't make sense for a car to drive around
17:49
then hit something that's not safe world.
17:54
You can do that million times right with no human heart in the
17:58
in the simulation. A second is like, how do you
18:03
make everything running super fast in milliseconds?
18:07
A buzz is can still like scale, right?
18:11
Ohh, So that's second things like you'll you'll be able to
18:14
learn like in suffering car. The third thing you learn is
18:17
like people also probably don't realise is sometimes required.
18:21
Is also multi model in a sense, right?
18:23
It's have the lightest data, the camera data.
18:27
Later on, they also added like the audio input because you have
18:32
to hear the sirens because when you hear the sirens, you have to
18:35
know where the sirens coming from and how to how do I pull
18:39
over on the street? That's safe, right?
18:42
So the Supreme Court itself also evolving model so we can talk
18:46
about each while those like and give you a little bit more
18:51
understanding how the technical behind it to make it more
18:55
reliable and be able to conquer the actual tasks in the human
19:02
world. Ohh, that's so cool and that's
19:06
so true. The multimodal Now how does that
19:09
play into what you're doing today?
19:11
Yes. So let's start with multi model,
19:14
right? Good.
19:17
When people thinking about phone calls, you think it might be
19:22
just all, it might be just audio.
19:24
The reality is a little bit more complex than that.
19:28
So when you're making phone calls as a human, what do you
19:32
usually do is you have a mental model of what you're trying to
19:36
do. Like phone calls.
19:38
It can be audios. Also, like during phone calls,
19:42
there's very interesting, there's button presses when
19:48
you're dealing with all the customer support and everything,
19:51
you have to understand those like button presses like and
19:54
they take actions instead of just the audio, audio bytes you
19:58
are hearing. And like when you're dealing
20:02
with the phone call, thinking about the scenario, you usually
20:05
have to also deal with incoming text messages, right?
20:10
So they will say, oh, I'm sending you a message or send
20:12
you a link to this. It's the model or the EI we're
20:19
building. You have to understand, OK, if
20:21
you receive something additional then the audio bytes, how do you
20:25
deal with that? So that's why the multi model
20:29
has to be there or you can complete the circle like
20:33
handling the handling the phone calls.
20:38
Wow. And it's so complex because
20:42
sometimes you get like, yeah, I'm sending you the text message
20:45
with your code on it, and then you have to say the code.
20:49
And so you have to have the system understand all of that.
20:53
How are you going about understanding it?
20:55
And it seems like you are really doing stuff with agents that
21:02
people only wish they could be doing.
21:04
But from what it sounds like, you've figured out a way to put
21:08
them into production. Yes, so like I said, I'm like
21:12
highly inspired by the suffering current, right, like self
21:16
driving car was able to take the text or take the audios and you
21:22
can also command it like take the text, take the different
21:28
radar data in real time and make the essential on those.
21:32
So when we build the assistant Rivia or we use the similar
21:38
methodology. So the system taking any type of
21:43
strings, it can be audio streams, can be text stream, can
21:46
be any type of data, it will be able to understand it on the
21:52
fly. Yeah, we we have a we call it,
21:55
it's also called perception models.
21:57
It understand like any type of data format and try to
22:00
understand those on the fly. And what does that look like
22:03
technically? Technically thinking about don't
22:08
think about the video recording right O when you record a video,
22:13
you have this like audio, audio bytes like streaming in using
22:18
web sockets or like web RTC to the back end.
22:22
Then you have the video stream like that's like the frames also
22:28
streaming into your back end. So in the perception models, you
22:33
have to handle like 2 streams at once, right?
22:36
Then you have your ML models like once more like.
22:39
Once the audio bytes or the video bytes comes in, the models
22:43
will be able to start to handle those.
22:46
You can handle those in a batch setting for example.
22:48
You can process this every two frames or every hundred bytes of
22:55
audio bytes. So that's depends on how
22:58
frequent or how fast your system runs.
23:02
For most self driving cars or most like, if not just other
23:05
robotics, it's like processing the data in a a yay frequency,
23:11
right? So you see like like probably
23:13
figure you see like the processing as probably hundred
23:17
100 Hertz or so even more even faster.
23:22
So we're adopting a similar technology is like the system
23:25
runs on a clock cycle, right? So once 100 milliseconds is
23:31
starting to process all the frames for the last 100
23:35
milliseconds, you use that data to generate signals for the
23:39
downstream systems. Damn that is wild.
23:43
So basically you're doing it at consistent intervals and
23:48
justice, checking out what's going on and it's almost like
23:53
you're processing whatever comes in.
23:55
It could be any modality. Modality, yes.
23:59
And then once you get a trigger for a certain modality, you
24:03
know, OK, if it's speech, we're going to translate that to text
24:08
and then we're going to work with that through Transformers,
24:12
like a transformer model or what is, I'm not sure how you can
24:16
actually like take action on it. I understand the the data coming
24:20
in part and then you saying OK now we have this data but how do
24:25
you actually go and take action on it and do that in a reliable
24:28
way? Yeah.
24:30
So I think eventually it will be end to end models, right.
24:34
So it will be like the any type of data like like the how the
24:38
GPI for all does, right. It will take the Omni data like
24:42
for Omni channels like format into the systems.
24:46
You have a daily cases like models taking those different
24:49
inputs then map to whatever actions you want to take.
24:53
For example, it can be speak, can be voice, can be anything
24:56
right to eventually become any to any model.
24:59
But right now we are still in the lake processing and
25:03
collecting data phase, right? You do this a lot on data like
25:06
to train our model. So you will have wide call is
25:10
more like a what you just described you, you have those
25:15
informations, then that's when you engage in like traditional,
25:18
like another piece, right? So you're starting to have those
25:21
intent or triggers or signals you can use for your
25:26
downstreams. So for those you're starting to
25:28
engage with like the first traditional piece, then there's
25:32
like the newer generation EIS because you do have to
25:35
understand what's what's happening for all the audio
25:40
videos like for the last Sprint, right?
25:42
So then you use those to generate more, even more
25:46
signals, kind of signal or expanding the context.
25:50
You can think of that way expanding the context just how
25:53
you do a vector search like Rack, right?
25:57
So it has to enhance the context so they so they model does not
26:02
hallucinate right on the task because you still, you still
26:06
need to finish the user's task, right?
26:09
They assigned to you. So you do like the enhancement,
26:12
do validations. Then for once you have enough
26:16
like context, you, you send us to the downstream.
26:19
For example, if that's a transformer AI models like, then
26:25
you have eight text to speech or or like direct like the direct
26:31
like actual text depends on your output, right?
26:34
So yeah, that's the second part, right?
26:36
So once you have enough signals, you can use that to ask your
26:41
general AI models to generate the response for you.
26:46
I like this description of saying you're just trying to
26:51
gather as much information and as much context through the
26:55
device itself and through the interaction itself.
26:58
It's not like you're going to a vector database to gather this
27:02
information. You're gathering it on the fly
27:06
with all the different inputs that are coming through on the
27:09
phone. I presume it's only through
27:11
phone, right? Yes, it's only through phone.
27:14
Yeah, Yeah. But I think in balls like let's
27:17
refer back to Saffron part, right?
27:21
When you think about when you try to build a self driving car,
27:24
what is the first thing you're trying to do?
27:26
You're you're trying to ask a summary driving car to start,
27:32
drive in a straight line, then stop, right?
27:35
So that's the first simple task you have to ask yourself do
27:39
similar to our cases. We start with something really
27:43
simple, right? For example, can it's like
27:45
starting making phone calls and say hello to the and then hang
27:49
out the phone call, right? So once you're starting to
27:52
collect those data, you have better understanding how the EI
27:57
and the the the human interacts, just like how you hang on sound
28:00
like when the Supreme Court is going to drive the straight
28:02
line, like what are the signal you trying to collect?
28:05
That is a similar a similar methodology.
28:09
So basically these basic agents are very much like my 14 year
28:14
old self prank calling different people and saying hello and then
28:17
hang it up. Well, to you, well, I think
28:22
about like we don't want to harm the human in the process, right.
28:26
So they like what describe like the the virtual worlds we put
28:32
them into a virtual world meaning like we are not using
28:36
like the actual humans to sometimes to engage those phone
28:41
calls. We have the other end.
28:43
It can be recorded. Humans to interact with the ISO
28:49
feel like a lot of random spending costs.
28:51
We have this like testing a controlled environment.
28:55
So this is fascinating to me because you're training or it's
29:00
not training. It's really like validating your
29:03
capabilities through this virtual world.
29:07
Are both both right? Like people talk a lot,
29:11
especially for generally I like people talk a lot about like how
29:14
to create a synthetic data, right?
29:16
You can think of that as like we also use that to generate
29:20
sensitive data so we can later on use that to train the model.
29:25
The 2nd is like the validation part, right?
29:27
You know, you asked me like, ohh, the multiple agents, you
29:31
know, once they hit the human world is starting to break,
29:35
right? And then, you know, it's very
29:37
soon easy to create a really appealing demos with your agents
29:43
work only 1% of the time they have recordings and send out
29:47
right. But how do you repeat the
29:49
process make it reliable. That's much, much, much harder.
29:53
I would think like you spend actually 90% of the time effort
29:57
to make it reliable. So that's why we have this like
30:00
virtual worlds and we can, you know, if something breaks, we
30:04
will be able to detected by millisecond what happened on
30:09
that particular moment and also inspired a lot of other several
30:13
card, right? I will be able to detect what
30:14
happens on a millisecond to find.
30:17
Oh, is that something like relative to the generate a model
30:20
or is related to the machine learning traditional NYPL you
30:24
model that makes the wrong decision?
30:27
Or is that because like open edges went down for a couple of
30:32
seconds, right. So we have those simulations
30:36
like England systems. So we will be able to like
30:39
inspect and you know, to kind of fine tune the system in a way
30:43
that we'll be able to reliably finish the task you have given
30:48
it. Well, I know that the last time
30:50
we talked it was how you can hand over the controls to a
30:55
human almost. And so it's like you have real
30:58
time observability to know when things go off the rails.
31:02
Is that a homegrown solution that you created or are you
31:08
using some like data dog plus whatever like evaluation tool
31:14
out there? How does that even look under
31:16
the hood? If you search like the real
31:19
charter I, you will find like we have open source projects,
31:22
right? And we the my team is really
31:26
specialised in the audio processing, right, audio and
31:29
streaming processing. So we build a lot of web sockets
31:33
in the very low level web socket to handle all the incoming
31:36
traffics and the forwarding the audio bytes like across
31:39
different systems. So it's more like a homegrown
31:42
systems like we we have built in house and be able to, you will
31:48
do pretty much like observe what's going on and take over
31:51
the control if you think the AI is making wrong decisions.
31:57
It's also like like self driving or like autopilot, right?
32:01
So you can still like supervise the phone call or you can still
32:06
supervise the driving. And if you find something odd
32:10
like you can always grab the driving wheel saying, OK, I'm in
32:14
control. But when you say, oh, I think
32:17
it's much safer for you, would you like to take control?
32:19
Now you can still engage the autopilot.
32:22
I want to build the same experience.
32:24
I think it's for people didn't realise this.
32:28
If you're trying to build that experience, the audio bytes, the
32:31
audio processing is very hard problem.
32:35
Because like for text, you don't feel when you see a text, if
32:40
it's selected for a second or two seconds, you don't feel the
32:44
text slagging, right? So you find always fine.
32:47
It's the transformer is slow, but when the audio bits slow,
32:51
you will immediately realise it. You know, even by probably half
32:56
second you're starting to realise, oh, there's like really
32:59
odd audio is starting to cut off or it's starting to slow.
33:03
You'll realise immediately. So that's why it's required a
33:07
really fast systems. And when you process in those
33:12
audios, it has to be really mindful making sure you don't
33:17
generate lacks into the systems. That's pretty hard.
33:22
So that's another hard problem. We have realised we have like
33:27
encounters because you know how to transformer works right.
33:30
If you, if you have ever monitors how the open AI give
33:34
you a response, you suddenly realise their response is not
33:38
stable. So it can be very interesting
33:42
way for the GT4 Turbo because we have the internal data, we have
33:46
internal like benchmarks for all the different larger models, how
33:50
fast they run and how stable they run.
33:54
And you were surprised GPT for Turbo, the first token they were
34:00
running is about 800 milliseconds to one second the
34:05
first token, right? So that's way slow for way too
34:09
slow for the human conversation, like the real time conversation
34:12
and the we're not really stable. So they can go up to like 3
34:16
seconds to four seconds. And we encounter this problem at
34:21
20% of the time that that's the work to do as the agent, right?
34:25
If you're trying to handle the audio wise, then they announce
34:29
this GPT for all right, the even faster model.
34:32
They said this is the 50% reduction.
34:36
So yeah, they were true for the for the launch date, we were
34:42
seeing seeing like 300 milliseconds, like 500
34:46
milliseconds for the first token.
34:47
But then it's starting to like getting slow again.
34:51
I think with the traffic like put this started receiving
34:53
right. So, so that's why I like I, I
34:56
will be super surprised if they're like for like end to end
35:01
model can reach to the level they demoed during their opening
35:06
day. Yeah, Yeah, it's really hard
35:10
problem to think about at the scale and then that time to 1st
35:13
token. Yes, I know when you think
35:16
about, I'm sure a lot of people whenever they think about time
35:21
to 1st token and the fast providers out there, I always
35:24
think about perplexity and how fast they are.
35:27
But you can't really use perplexity as an API.
35:31
I'm imagining for your use case. Ohh well previously they do have
35:36
an API, but perplexity is they have all their own models right?
35:44
Like the. I think that's mostly used for
35:47
RAG, not for natural conversation.
35:52
So that's why I like, you know, one way when we're finding like
35:55
the models to make the, the, the, our solution run faster,
36:00
faster. So that's why we use like deep
36:02
breath, right? So that's why you use deep
36:04
grammes like speech to text and text to speech models.
36:06
So it's like help us on both ends, try to reduce the latency.
36:12
Yeah. And then the models in between.
36:14
Are you just putting something in house?
36:18
Pillow, we run everything in parallel, right?
36:22
So you're now we have built a mechanism like it doesn't matter
36:29
if opening is running slow. So there's always easy to fall
36:35
back to a faster systems. So like the different tiers of
36:39
systems to handle the request to make it running faster and
36:43
accurate, let's say at the same time.
36:46
And and also like I said, we we have live, but we have real time
36:50
tracings on each of the requests.
36:53
So if it's going I like a young 13 thresholds, we have a fall
37:00
back mechanism, right? Think about like so that's why I
37:02
also use like several parties like if the seven time card, one
37:05
of the AI models starting to break, does the system going to
37:10
stop on the road? This does not right.
37:12
So it has a way or not? Yeah, it has a way to recover
37:16
itself. So we are using the same
37:18
mechanism to recover from if there's any general models
37:22
running slow and we can recover from that.
37:26
So I like this design pattern. It's basically saying you have a
37:29
preferred model that you're going to be trying to use unless
37:34
it's too slow and then you have fall back options.
37:37
And I've also seen a different design pattern where you've got
37:40
like the gateway and depending on the prompt, it can choose
37:46
which model would be best. If it's not that complex of a
37:49
prompt, it can go to a smaller open source model and save a
37:53
little bit of money or whatever. But it's, it's a little
37:56
different when you think about it as a gateway because you you
38:00
hit the gateway and then you kind of make the best choice
38:03
from the gateway. What you're saying is we have a
38:06
priority list and we are always going to try and go for this
38:11
model, whatever that model, that Top Model is.
38:14
But if we can't get that Top Model, then we're going to try
38:17
and go down the priority list. Ohh, it's slightly more complex
38:22
than that, right. So there is a priority certain
38:25
request. So we also have like I said,
38:28
like everything is running in parallel.
38:30
So we have similar to your gateway.
38:33
So we have also a gateway for distributing tasks for different
38:37
like different requests, different, different identified
38:42
intent. So they all play in a role for
38:45
the downstream system to decide OK, should I?
38:49
Should I wait or I like starting press buttons?
38:53
OK. Ohh, cool.
38:55
And I know that you mentioned you are dealing with Gen AI and
39:00
traditional ML, yes. Are there pieces first of all,
39:04
just breakdown how that looks like that you have the Gennai
39:07
side and then it goes down and it it tells the actions to the
39:10
traditional MMO and then it ends there or what does that look
39:13
like and how do those how do those two play together?
39:17
Like I said, everything is running in parallel, right?
39:20
So you can think of like how I use something used to like the
39:26
terminal, like the event bus, right.
39:28
So everything happening in Middlesex and is going to
39:31
publish into an event bus. So any downstream systems, if
39:37
you think you can handle the tasks, you're going to subscribe
39:42
to those events or like in the event bus, then once you're
39:47
processing the host, you're going to submit your response
39:50
back to the event bus. Then the downstream system if
39:54
they are interested in your response, still like same layer
39:56
to subscribe to your event. So that way you have a unified
40:02
like communication channel, right?
40:04
So it will make your system much simpler to handle.
40:12
Wow, OK, this is really cool to think about.
40:16
So it's, it's this event bus. And basically if there is the
40:20
notion that, hey, this might be for me, then the whatever the
40:26
task where the system is going to subscribe to it.
40:29
And then if it is for it, it'll publish back the results.
40:32
And if it's not for it, it'll also publish back results.
40:35
Like Nope, this actually wasn't for me.
40:37
Something like that, yes. Yeah, like the simplest version,
40:41
right? Like most people can't
40:43
understand, but there's like, you know, in, in practise, like
40:46
it's in engineering, it's all in the details.
40:48
There's so many things you have to.
40:50
Make it work. Yeah.
40:53
Like what? Yeah, So for example, how do you
40:56
scale, right? So that's like really our
40:58
problem, right? How do you scale Like now you
41:00
have millions of like requests coming in milliseconds.
41:05
Remember, you have to make sure the audio biases doesn't cut off
41:08
right in high resolution. And the handle like there's like
41:13
the critical path is cannot be slowed down because of all you
41:17
have something, you have more events.
41:19
But then there's things like, oh, if the system breaks, you
41:23
haven't fall back. All those like play into make
41:27
the system run faster, right? Yeah.
41:30
So those are the like a lot of engineering efforts going to
41:34
make the system reliable. That's it.
41:36
And for anybody out there listening, we're laughing about
41:39
the scaling part because I think before we hit record, we were
41:42
saying that the typical question you'll hear at talks or meetups
41:50
is somebody has to raise their hand and say, OK, this is great,
41:53
but how does it scale? And so now we just basically did
41:59
that to ourselves here. This is great, but how does it
42:01
scale? Ohh, that is classic.
42:05
So this, this is fascinating to think of.
42:08
I'm still trying to like get the picture in my head, but I, I
42:11
think I have a more clear idea of it.
42:14
Are there other challenges that you've encountered that you're
42:17
like, damn, that one was particularly nasty and I'm glad
42:22
I'm over it now. Ah, I feel is like, you know, I
42:27
built on the traditional AI. I find everything we're trying
42:33
to build on the general EI is nasty because you're constantly
42:39
dealing with like slow response or inconsistent response or
42:46
like, or like the agent doesn't behave like what we expected or
42:54
like, ohh, it worked yesterday, but why it's not working this
42:57
today? Ohh, because OK, I just upgraded
42:59
their model. Or they just make the model like
43:01
Dumber, right? Those are the things we are
43:05
constantly dealing with the the non deterministic behaviour of
43:10
the general AI is hard, right? So we have never encountered
43:17
things like this before. So that's why I mentioned like,
43:21
you know, you can build a really cool demo probably work only one
43:27
out of 100 times. But if you if you make sure
43:31
you're trying to make sure everything is working like
43:33
probably 90% of the time a you need to like that's a very hard
43:39
engineering problem to solve. OK, yeah.
43:43
So that's why I, I find like anything that's it's involved in
43:48
the general AI model, it's going to be hard.
43:51
And if you really want to productize it, it's going to be
43:54
really, really even harder problems.
43:59
So another thing I. Realised is it?
44:01
Yeah. So the long nights when you said
44:06
that just all of that stuff is what we've been hearing time and
44:12
time again on this podcast and you're not alone there.
44:16
The inconsistent responses. That's why like evaluation is
44:21
such an important piece of this. The slow responses I think
44:26
people have. A lot of people have abandoned a
44:30
lot of use cases because they don't want to have to try and
44:34
figure out how to make it faster.
44:36
And so it's like, we'll just wait until it gets faster.
44:41
Cause for some people, like, that's not how the beer tastes
44:46
good. It's not like if I make this and
44:49
spend all this time and energy making this faster, what am I
44:54
going to have to trade off in order to do that, right?
44:57
And so that's, that's another hard problem.
45:00
So I feel for you. Yes, yeah, I think he'll make
45:05
his stable makes it run faster. It's going to be it's a so
45:11
different experience like in the in the in the traditional a I
45:14
world and yeah, you have. So that's why you have to have
45:19
really good engineering practise like monitoring benchmarks,
45:23
evaluations. Most people when they launch
45:27
their product, they don't talk about those where they only talk
45:30
about like all the fancy products.
45:32
But the the harder problem is to how do you make there is you
45:35
know you have all the like tools you need to make it running
45:39
stable. Yeah, the reliability piece is
45:43
so important, especially if you're going to put something
45:46
out there for the public to use. One thing is putting it in the
45:51
video and cutting out the pieces that you like, but we've seen
45:55
time and time again that people will put it out for the public
45:59
and then it just gets demolished because it's not that reliable.
46:03
And so you were in. I think what happens is you get
46:08
the marketing teams that will create these videos and show
46:13
that like, this is magical. And then it gets out into
46:17
people's hands and they have such high expectations for what
46:20
it's going to do. And it doesn't live up to that.
46:22
It doesn't even live up to half of it.
46:24
And so then you're like, this is a scam.
46:26
What is this crap? Ohh yeah, I I think this this is
46:32
so true. Like, because like, the thing
46:34
was, there's so many economists out there, right?
46:38
Ohh, you'll have to get your the story of heard first, right?
46:48
But also like, you know, on the other side is we all know the
46:52
general model is growing super fast, right?
46:56
Maybe like the claims you trying to sign right now, you know, you
47:02
cannot achieve that, but within a year or within half a year you
47:06
might be able to do that, right? So who knows?
47:09
Yeah. So that's why like people,
47:11
people fray like if they don't talk about this, they're going
47:15
to miss out the opportunity, right?
47:18
Another thing is like, I'm not claiming I had no all the
47:21
answers, right? I was still like trying to
47:23
improve the system at the same time, right.
47:27
I can only claim ohh, my self driving car couldn't drive
47:30
driving St like we're we're like we're 10 metres like that's I
47:34
can guarantee. But how it can go into like
47:36
highways like to do like detours a lot of things.
47:39
I don't have like the everything like radio like right now right.
47:45
So yeah. And do you think that you just
47:49
will need the training data like once you put it out there, just
47:53
like with self driving cars, the more miles that a car drives the
47:57
better it performs or is it something that because you are
48:03
using so many third party models?
48:06
Yes, I think the one I designed the systems of all the in the
48:11
intent was as we collecting more data, more real interactions,
48:16
the system going to get smarter automatically, right.
48:19
So we, so that's why we, we trying to launch early, like
48:25
maybe, maybe it doesn't work like, however, some time, but
48:30
like we are able to collect in those interactions during the
48:34
phone calls, like to see how AI interacts with the actual humans
48:39
in those phone calls. So by then like we're starting
48:43
to have more data. Like I think like, like I
48:46
mentioned, eventually for different use cases, you will
48:49
have a dedicated model for that case, right?
48:52
For us, we want to have that dedicated model to for
48:57
representing you to make phone calls.
49:00
And when you say use cases, you mean like if I call my plumber
49:04
versus if I call my hair salon? It's more like a, you know, what
49:10
we are thinking of is people feel most painful is to dealing
49:14
with customer support, dealing with random phone calls.
49:18
We want to have those data like how people actually engage in
49:22
those conversations and how AI deciding to do this phone calls.
49:27
For example, if you're trying to call your utility companies like
49:33
saying, ohh, you have power outage, right?
49:37
What is their system look like, right?
49:39
For example, it's more like, oh, you're driving to a new city,
49:41
right? How the their roads looks like,
49:43
right? How to get to the to the
49:46
destination. And we're going to know, OK, how
49:50
you get actually be able to navigate through that.
49:52
And we are recording kind of understanding during the
49:58
process. So something like that.
50:02
I see. So right now you're just
50:04
exploring roads and yes, gathering that data.
50:07
And so different companies that you call into, you're getting
50:12
their customer service agents, which are probably also using AI
50:17
these days. So, so that's why I talk about
50:21
like a business has been using this for ages.
50:25
They're trying to take advantage of you, right?
50:28
Like because you don't have like 2 hours or three hours waiting
50:32
for their humans like saying, oh, I want to cancel my
50:35
subscription. And you know, they gonna say,
50:36
ohh, just get on this phone call like 3 hours later they're going
50:40
to cancel the conclusion. They know you won't do that,
50:43
right? So they're taking advantage of
50:45
you. But now we are giving you the
50:47
similar or even better a I to like to give them the option
50:52
back to you saying, ohh, you don't need to wait in line for
50:55
that, right? Your AI going to do everything
50:57
for you and then the business won't be able to take advantage
51:01
of you. Yeah, and it's almost like
51:04
throwing it back in their face. Like if I'm going to have to
51:06
deal with these shitty chat bots, then I might as well have
51:11
my own chat bot that can deal with it.
51:14
And then if I need to be plugged in at some point, I can be
51:18
that's incredible. I, I'm very happy with the
51:23
vision and I appreciate you kind of peeking behind the hood and,
51:28
or under the hood and showing us what you're doing and how you're
51:32
working on these hard challenges.
51:34
Is there any other like thing that comes up that you might
51:37
want to give as words of wisdom to the listeners that that are
51:43
dealing and working with AI and ML day in day out?
51:49
There's like the the, the newer AI going to redefine all the new
51:55
systems, right? But it's not going to be an easy
52:00
path, an easy solution or easy like we asked like how to
52:05
deployment, to deploy those in the systems.
52:08
It's going to take probably it's still taking years for the
52:13
agents really representing you to do a lot of things.
52:17
I think right now is be be open minded to all the potential
52:22
solutions and try them out right.
52:24
Try the all the AI tools, try the AI things like I think in
52:28
the future you will be the level of intelligence you can leverage
52:35
only really going to define how productive you are right?
52:39
Thinking about like oh I can probably utilise 10 GPUs to like
52:45
GPUs of intelligence to make my life easier or with someone can
52:50
utilise hundreds GPUs or AI to make their life better.
52:54
I think in the future that's how we think you are rich right?
52:59
You can leverage it even more intelligence to make your life
53:02
better. That is a wild way of looking at
53:06
it, like it's gonna be, but it's not currency.
53:10
We're not gonna play in U.S. dollars.
53:12
We're gonna play in in NVIDIA chips.
53:13
Yes. Wonderful man.
53:16
Well, I am super thankful that you came on here and you did
53:21
this with me. It was a blast talking to you.
53:23
I am so excited for what you're building.
53:27
And I mean, I wonder though on your launch, not that I know
53:32
much about launching companies or any of that.
53:34
So it's almost like a armchair quarterback right here.
53:37
So take it with a grain of salt, but don't you want to take a
53:41
page out of the self driving car book and just start in one area
53:47
of like the US and start with one specific accent or one
53:53
specific like company that you're trying to do or one
53:57
specific use case? Yes, like like I mentioned, like
54:02
we don't claim we have all the solutions, right.
54:04
So, but we really want to hand the technology to you.
54:09
I want to like we have our own a critical use cases we think
54:15
we're going to handle really well, but most of the cases we
54:17
may not handle really well. But we want to yield the user to
54:22
get a feeling of how the power A is and be able to utilise this
54:27
to better like the bigger corporate.
54:32
So that's why like we're trying to make sure like people who are
54:37
who were suffered from the phone calls and at least get the
54:40
access, you know, to see like the our mobile app 1st.
54:47
What is the price structure look like for the consumer?
54:51
Yes, so for us, we want an easel mind and very interesting like I
54:59
because I would never making phone calls.
55:02
It's really, I don't know like why just like in the US, like
55:08
the first time I'm trying to make a phone call is making me
55:10
super nervous. Want this to be an easy mind.
55:14
Like when you're trying to make phone call, you're thinking of
55:16
like the solution we have built for you.
55:18
And so that's why we charge a kind of flat subscription
55:23
models. So then for any phone calls, you
55:26
can go over that. So you don't need to worry about
55:29
all have to make phone calls today or I have to make phone
55:31
calls tomorrow or like there's something like or something or
55:35
the other side is saying speaking some language I don't
55:37
understand, right. So I want ease of mind.
55:41
So that's why we are reject a subscription fee so that you
55:44
don't need to worry about any phone calls at all.
55:47
And the reason I ask that is because I've heard people talk
55:51
about this idea of I really want my customers to engage with my
55:56
product, but because I'm using third party APIs, the more a
56:01
customer engages with my product that means the more API calls
56:05
I'm making and the more money I'm spending.
56:09
Of few things like we, we know everyone is doing token economy
56:15
right and but I think as an end user you care more about like
56:22
end user you care more about like can I get 10 years solve my
56:27
problem as a company? Yeah, you know if we have we, we
56:32
find cheaper and still working solutions we are making more
56:37
profit, right. Think about that way right like
56:41
we do our own calculation of how much token we consume and for
56:46
each phone calls. I think it's still it makes
56:48
sense for us to open up this in a subscription models.
56:53
Me as a consumer, I love that much more.
56:55
I would rather do it like that than have to think about like,
56:58
OK, should I use my credit for this AI phone call or should I
57:05
just get on the phone? Yeah, So I, I feel you 100%.
57:09
I'm super excited about what you're doing, Sean.
57:11
This is awesome. OK.
57:12
See you. Thanks.