Latest / Agentic Conversations (formally mlops.community) / Reliable Voice Agents

Transcript

Reliable Voice Agents · Agentic Conversations (formally mlops.community)

0:00
We're recording, yeah. Yeah, we're recording, we're on.
0:03
Oh, OK. We'll try again.
0:07
We'll try. So we did the Agent Builder
0:17
Summit. You gave a workshop before that.
0:20
It was awesome. I had a blast.
0:21
Hopefully you did, but I wouldn't expect you to tell me
0:24
if you didn't right now in front of everyone.
0:27
It was, no, it was amazing. We built a voice agent from
0:31
scratch with live kits and then actually went and built out
0:34
evals as we went along, which was really fun because it was
0:37
actually how you build agents in production.
0:39
Like how do you actually take a voice agent, build a test, added
0:44
it, build a test appends like you know, keep new capabilities,
0:48
add in like different things and improve the agent over the
0:52
course of the workshop? Yeah, there was three guys that
0:54
left and I was like, how was it? And they both looked like shell
0:57
shopped because they were like, it was really good.
1:00
I wasn't expecting that, but it was, it was good.
1:04
And they, like, almost couldn't talk because of how good it was.
1:07
And so that was a little bit of like, yeah, you know, it's
1:11
Goodwin. That's amazing.
1:13
Yeah, yeah, it was cool. It was very cool to see.
1:15
And there was a lot of voice AI companies at the agent builders,
1:20
a big event too, which is cool because I tried to get a a few
1:26
of them there to make sure and showcase that like this is
1:31
happening and a lot of really good products are being made
1:37
with voice agents. Totally.
1:39
I think I was really excited how many people were interested in
1:42
voice AI yesterday because in my head, I think over the past
1:46
year, it's certainly become, you know, last summer it was still
1:50
very niche of a couple of companies were playing around
1:52
with voice AI. People were saying, oh, this is
1:55
finally working. So this is really exciting
1:58
space. But it was small startups
2:00
experimenting and now I think we see large companies, almost
2:04
every company is thinking about how does voice AI fit into my
2:07
agentic kind of vision or our agentic plans.
2:11
And that's really exciting to see everyone from big companies
2:15
to small start-ups thinking how can I incorporate voice AI into
2:19
my product. So we saw a lot of that
2:21
yesterday. And I told you, I go nuts when I
2:26
have to deal with the old voice. Like, what is it, IVR?
2:30
Yeah, I go absolutely crazy whenever I get on a call.
2:34
And it's that. And so I hope that transition
2:38
that you're talking about of every company having this now
2:43
happens sooner than later. Well, it's funny because I think
2:46
people when they say I don't want to be talking to a bot on
2:49
the phone, are really saying I don't want to be talking to a
2:52
bad bot on the phone. But really, if you could go, I
2:55
think similar things were said about websites.
2:58
And why would I want to go to a website versus just calling
3:01
someone on the phone and making it happen or like, I don't trust
3:04
a website with my credit card or I don't trust things that are on
3:08
on this website. And then obviously, there's a
3:10
shift where millennials would much rather go to a website than
3:14
talk to anyone. And I think that with voice AI,
3:17
it's actually creating this really natural interface where
3:20
you can really quickly, just like imagine you can just call
3:23
and change your appointment versus going to a website,
3:25
logging in for getting your password, finding a new time,
3:29
etcetera. It's so funny.
3:30
That you mentioned how much easier that is when we created
3:33
these websites and you would think like oh, we've got it all
3:37
on the website why don't you use the website you can change
3:40
everything there you can do it but then the actual act of doing
3:44
it is so painful sometimes for that very fact like clicking
3:48
around you don't know the interface, you forget your
3:50
password, you got to sign in all that so yeah, IA 100% agree
3:55
yeah. And I think that we forgot that
3:57
actually talking is a very efficient way of doing things.
4:00
Humans are have, you know, created speech over thousands of
4:05
years and very efficiently communicate information.
4:08
And so there's a reason why when you're working on something,
4:11
you're like, it's easier to just talk over the phone about this.
4:14
And I think voice AI has this capability to create this very
4:17
natural interface, especially in age of AI when you're talking to
4:21
either like a humanoid robot. But even if you're just trying
4:24
to get a task done, you can say, OK, here's what I'm thinking.
4:27
I need to I need to book a reservation or I need to like
4:30
get this task done. Voice can be really natural.
4:33
And so I think what we're going to see is that almost every
4:36
company is going to have a voice interface of some kind for the
4:39
interactions that make sense in a similar way that happened to
4:42
mobile after web, where you had not everything is on mobile, but
4:47
a lot of things are on mobile in different forms.
4:49
And so I think every company will have to think about what
4:52
interfaces make sense via voice for us, and how can we make that
4:55
really seamless so that instead of having to wait on the phone
4:58
for 20 minutes to talk to a human, you actually have like a
5:02
really seamless, fast interface to get things done.
5:06
I know that the customer support is a huge use case.
5:11
What other ways are companies thinking about employees?
5:14
Yeah. I think what's interesting about
5:16
voice right now is 80% of the use cases kind of fall in the
5:20
same couple buckets where customer support is
5:23
overwhelmingly the use case. This is because I think the
5:27
infrastructure is already there. So it's an interesting case of
5:30
like you already had some level of automation with IVR.
5:35
It's over the phone. So it's actually like a very
5:37
defined channel, like you have one person talking to another
5:40
person. So it's much easier to navigate.
5:43
You have like a pretty predefined side of things in
5:46
place. And most companies for customer
5:48
support will have like standard operating procedures.
5:51
So like if you're a customer support person, these are the
5:54
things you have to say. So I think that's why it's taken
5:58
off so much is because it's and also it's a very high large cost
6:01
Center for a lot of companies. But then we're also seeing
6:05
things like healthcare, so being able to like file claims, book
6:10
appointments, all of these things that have to happen over
6:13
the phone because these legacy industries had very little
6:17
software permutation permeation because of there's just so many
6:23
providers. There's so many like such high
6:26
need for security and all of this.
6:28
But voice AI is cool because you've had this universal API
6:31
where you can have a human talking to an automated system.
6:36
So you don't need to have software integrations.
6:38
I think we see this with like logistics too.
6:41
So truck drivers, there is so many different parts of the
6:45
logistics system and a lot of those don't have, you know, a
6:49
truck driver isn't going to integrate a software system for
6:52
if they're just doing, you know, maybe they have 50 trucks or
6:56
maybe they have 50 drivers. They're a small business and
7:00
they're not necessarily looking to adopt all of these new
7:02
software systems. And voice AI again is really
7:06
cool because it doesn't matter whether or not all of them adopt
7:09
it. One of them can adopt it,
7:10
automate some things, and then still be talking to a human on
7:15
the other side. Wow, the universal API is such a
7:19
good metaphor. It's like you can use tech now,
7:23
but you don't have to have everyone else using tech.
7:27
Yeah. Because everybody can talk,
7:29
mostly everybody. And so yeah, me sometimes better
7:34
than others or better. No, I wanted to make a joke
7:40
there, but didn't, didn't come out.
7:42
But the the idea is like having almost like this behind the
7:49
scenes going on where you can leverage tech and the folks that
7:55
are using the tech don't even need to know.
7:59
Yeah, in a way they'll know because it's AI.
8:02
And so, yeah, I'm talking to a bot, but they don't need to know
8:05
that like, well, you might have a really cool complex system in
8:08
the background and they can just sync into it with this universal
8:12
API. Exactly.
8:13
And so it doesn't, Yeah. And so you can have like
8:17
doctor's offices or claims providers or truck drivers or
8:23
HVAC system providers all of a sudden be able to use this voice
8:27
interface where especially if you're, let's say you're driving
8:30
around in the car all day servicing different HVAC
8:33
systems. You're not necessarily in front
8:36
of a computer. You're not necessarily looking
8:38
to acquire a bunch of complex software integrations and
8:42
setups. UI's they were just talking
8:44
about were so painful. Yeah, exactly.
8:46
But you can get in the car and say like, OK, what's my next
8:48
appointment or have someone answer the phone for you while
8:51
you're installing something and now you haven't lost that next
8:54
client. Yeah.
8:56
And I think there's also all, there's all of these use cases
8:59
that haven't been explored as much because they're, there
9:03
weren't even really like automated systems there before.
9:06
So you know, all these people that are in cars all day, how
9:09
can you help augment those? Like if you are like if you're a
9:14
delivery person or if you are a police officer, or if you are
9:19
like doing a, if you're like a Uber driver or like, how can we
9:24
integrate this into Waymo and self driving cars?
9:27
How could you control that interface with with voice AI?
9:30
Yeah, Yeah. I was thinking about how you
9:33
might want to help. I know I've heard of a company
9:37
that is doing something around for doctors because they have to
9:41
write a lot of crap after they see a patient and a lot of their
9:46
time is spent writing things down about the visit that they
9:50
just had. And so you can kind of
9:53
extrapolate that out with a lot of different jobs.
9:58
You have to have this component of I need to write down what I
10:02
just did and can we have it be a voice agent that interviews you
10:10
about what you just did and then writes it down for you So you
10:13
can just answer questions and you don't have to sit there and
10:17
go into the UI and type it down or whatever.
10:20
Or even better, like, can the voice agents just be with you in
10:24
whatever you're doing and then write it down and say, hey,
10:27
here's what I got, Do you want to change anything?
10:29
Totally yeah, I think that's it definitely a big it's like so
10:33
much of our day-to-day lives happens via voice, right.
10:38
You're like talking to your Co workers, you're talking to
10:40
someone and then you want to recap it.
10:42
You're like talking about the you're trying to communicate
10:46
your plans, you're trying to communicate changes.
10:49
And so I think all of these places that you're talking to
10:53
other people, there's opportunities for voice AI to do
10:56
to free up a lot of that time to then do other more interesting
10:59
things. Yeah.
11:00
So now there's a big idea that we want to hit on, which is how
11:06
to make this reliable. Exactly.
11:08
I think that's the biggest, that's the biggest thing that
11:10
people are nervous about with voice AI is like, that's great
11:14
that I could automate this, but these are my customers or these
11:17
are my business partners or people that are important to
11:21
what I'm doing everyday. I don't want the AI to all of a
11:24
sudden go rogue which it. Has been known to do.
11:28
Exactly. I don't want to be promising
11:30
refunds. I don't want to be giving
11:32
discounts, giving discounts, saying the wrong thing.
11:36
And I think that's where something that's really hard is
11:39
that you have to, for every step in the conversation, it's an
11:43
opportunity for failure, right? So if you have, if you have a
11:48
10% failure at every point, it's pretty high.
11:52
But let's say you have some percent of a failure at every
11:55
point, that failure could potentially compound throughout
11:59
the stack. So the chance that anyone turn
12:03
fails then all of a sudden becomes very high.
12:07
But so you might say, how will voice AI ever be reliable in
12:10
that case? But we've actually built out
12:13
systems like this across all sorts of other parts of
12:16
software. So for example, like the
12:17
Internet or, you know, a web application, every single stack
12:22
part of the stack is super unreliable.
12:24
Servers are unreliable, network connections are very unreliable.
12:29
You know, all of your APIs and all that jazz, but you create
12:34
reliability out of redundancy. So you can have these fall
12:37
backs. You can.
12:38
We've actually gotten to six nines of reliability for large
12:41
scale web applications. And so I think the thing that
12:44
I'm thinking about all day every day is how do you create that
12:46
same amount of reliability for agents while also still making
12:50
them really autonomous? It reminds me a little bit and
12:53
I'm no expert and I definitely wasn't around in tech.
12:57
I just heard stories of like databases being super unreliable
13:01
back in the day. And so that's why you had to
13:04
create different kinds of databases that would maybe you
13:07
Shard databases and Cassandra comes up or maybe you have these
13:12
new ways of thinking about databases so that you have a
13:17
database that's specifically never going to fail and you have
13:20
cockroach DB that comes up. You know, it's meant for that
13:24
real like reliability. And so now when we're at the
13:29
beginning of this in the voice spectrum of how it's going to
13:34
take over you, you like are going to probably start seeing
13:39
some of that. I imagine like you're going to
13:41
start seeing, OK, this this voice agent is specifically,
13:45
it's like the cockroach DB of voice models.
13:48
It's not going to be have any problems.
13:50
Or maybe what you're talking about too is also each piece of
13:55
the system needs to be like a cockroach and that it doesn't
14:00
fail. Or if it does fail, we have ways
14:04
so that the end user doesn't realize it fails.
14:07
Yeah, exactly. And then like that that
14:11
redundancy or fall back mechanisms.
14:14
But yeah, I think that's something that people forget is
14:17
that building reliable web applications feels so obvious
14:20
today. Like the idea that your
14:22
application would go down if you have, you know, 10,000 users go
14:26
to your site, it's actually just by default, like if you deploy
14:30
on AWS server cell and use serverless, etcetera.
14:34
Like it just works. It just works out-of-the-box.
14:36
Like everything is pretty secure out-of-the-box.
14:40
Like there's a lot you have to try pretty hard to make a
14:43
terrible web application today. But I mean, that's, that's not
14:48
to say that it's still easy, but there's a lot of things that
14:51
have just become infinitely easier than they used to be.
14:55
And like even like using RDS from AWS or like what any of
15:01
these like cloud databases, there's like redundancy.
15:04
They have like leader replicas and then read replicas and like
15:10
all these things that you don't even have to think about.
15:11
You don't have to think about your server backs.
15:14
You don't have to think about all these things, but today with
15:17
voice AI, you still very much have to think about a lot of
15:20
these things of like if my TTS provider and my LLM provider and
15:26
my STT provider all have some variance in the reliability of
15:32
how of their latency. You have to be ready for all
15:37
three of those things to not respond for two seconds.
15:40
That's going to be a six second latency.
15:43
And if you don't respond for six seconds, I'm going to be like,
15:46
hello, anyone there? What's happening?
15:49
I'm totally checked out, yeah. Exactly.
15:51
It's super unnatural, yeah. 6 seconds is an eternity in a
15:56
conversation. Yeah, maybe for like normal.
16:02
Yeah, that's. Like if I don't, I don't respond
16:04
for six seconds. Hello, I see you on there.
16:13
So you can, and that's not even, you know, API reliability, like
16:17
99th percentile of like 1 seconds delay is, is notable.
16:22
And so I think when you're building voice AI applications,
16:26
you have to think about your latency budget across
16:29
everything. And not only your latency budget
16:32
on average, but your latency budget in an on the end
16:34
percentile. Because if you have a 20 turn
16:37
conversation or if you have a 30 minute conversation, think about
16:41
how many turns that is to the conversation and how many
16:44
opportunities for failure. And if any one of those fails,
16:47
like if you, if the agent doesn't respond for six seconds
16:50
at any point along that conversation, there's a chance
16:53
the user just hangs up or like thinks that nothing is happening
16:57
or ETC, and now that whole conversation has failed.
17:01
But the fail back or the ideas that you've seen work is it just
17:07
like if something fails, you have a go to line that you say,
17:12
sorry, something's not working, I'm working to make it happen
17:16
better or I don't know what. How do you deal with that?
17:19
If you're noticing, all right, there's these failures that we
17:24
kind of have figured out, but we know since there's such a high
17:30
unpredictability, we're gonna have some at some point.
17:33
And so we need to like give the user some idea that we're still
17:38
there. It might be taking 6 seconds,
17:40
but I have like a quick canned response that I can use.
17:46
Well, so this is where you actually need to have if like
17:50
open AI doesn't respond or like this particular model provider,
17:53
like model instance, you can use same, similar models of the same
17:58
family and it hit those. But you can also have a fall
18:02
back of like if Open AI doesn't respond, then I'm going to use
18:05
Gemini. And so that you can kind of have
18:09
these like quick fall backs or like whichever 1 responds first.
18:12
I use that this fall back logic is actually pretty non trivial.
18:16
So that's why voice AI orchestration libraries like
18:20
Pipe Cat or Life Kit are really useful.
18:23
I think even more so than agentic applications because a
18:27
lot of this real time streaming voice orchestration is pretty
18:31
non trivial. So how do you actually create
18:34
these fall backs? You can actually configure them
18:36
with these libraries to fall back like they handle all the
18:40
fall back orchestration so that you don't have to implement that
18:43
from scratch. Yeah, that's cool.
18:45
And the other thing is like, isn't it a big time
18:50
vulnerability if at any one point you fail?
18:57
And then it's almost like this can ruin everything, Maybe for
19:01
the user. Let's say that the user is
19:03
patient enough to stick around, but do you have the potential of
19:08
like losing the context or is it not like I, I think you were
19:13
telling me that on the reliability aspect, you have
19:17
like all of this, it's like more high stakes, but is it just
19:22
because of the user that is not going to be patient?
19:25
Or is it because they're if you lose something, then you
19:28
potentially can lose the whole conversation and you got to like
19:31
start over from scratch? Yeah, it's more the the users on
19:36
impatience. So I think like that's why we do
19:40
a lot of these benchmarks around like model, not only model
19:44
latency, but also model variation in latency.
19:48
So the thing that's just as important as the, as I said,
19:52
like the 50th percentile is also the 99th percentile, because if
19:56
the 50th percentile is amazing, but 99th percentile is much
20:00
higher. Then that that person's going to
20:03
be really impatient where you don't respond for 30 seconds.
20:07
All of a sudden, it's a crazy amount of time that that
20:09
person's just going to assume that the the whole thing failed.
20:13
Oh yeah, yeah. If you don't respond for 30
20:15
seconds, you're somebody's hanging up.
20:17
Somebody's hanging up 6. Seconds.
20:19
It's awkward. Yeah. 30 seconds.
20:21
Game over. Game over.
20:23
Yeah, you see that? And so I think that's similar.
20:27
So that's just one type of failure, but then there's all
20:29
these other types of failures that I think these are the ones
20:32
that get more airtime for enterprise readers where you are
20:36
issuing refunds or you're saying factually incorrect information.
20:40
Air Canada. Yeah, exactly.
20:42
You're you're selling. Your voice.
20:44
Selling some of the car for $1.00.
20:46
Yeah, the Chevy, it's saying that Tesla's better than Chevy
20:51
dealer. All those, yeah, all those
20:53
problems come up and like you said, it's a very valuable
21:00
conversation that you're having, so you could fuck it up big
21:03
time. Yeah, totally.
21:06
And and it's with customers too, which is a business.
21:09
That's it. It's like a hot It's high
21:11
stakes. Yeah, exactly.
21:13
It's high stakes. And so I think that's where it's
21:17
not only just performance, but now you have this like quality
21:20
aspect that I think is very different than SAS tools
21:23
previously as well, where before it was like the button should
21:26
work every time, but that's really a performance thing.
21:29
Like what percentage of the time is a button working?
21:31
And it's usually pretty deterministic.
21:34
So with voice AI, it's now super non deterministic and what
21:37
quality looks like is very different.
21:38
Like there's obvious things where it goes right and obvious
21:41
things where it goes wrong. And there's this whole in
21:43
between of wasn't quite perfect. And so how do you do like
21:48
instruction following evaluation or like, did it do the right
21:52
thing at the right time? Did it take the right action on
21:54
your behalf? Be a function calling and all of
21:58
this jazz, which opens up a whole can of worms of how do you
22:01
make sure that not only your performance and you're saying
22:04
the right thing, but you're saying the right thing at the
22:06
right time. But you tested this, right?
22:09
Yeah, with all the different providers.
22:11
Yeah. So we've done the benchmarks
22:13
across all of these different areas.
22:15
So TTS like how do you measuring latency, measuring word error
22:20
rate where like the accuracy of the model cost and how do you
22:24
balance these? I think there's a very common
22:27
balance across all of these models is like cost, latency and
22:31
quality that's. The Venn diagram.
22:34
Yeah, that's the Venn diagram. And you can kind of like some
22:36
models are just going to be like better at everything, but really
22:41
there's no one-size-fits-all. Like every model is going to
22:44
have it's trade-offs. And so how do you choose between
22:47
for TTS, for STT, for LLMS, voice activity detection?
22:52
The like really interesting thing about voice AI is that you
22:55
have so many models all working together in order to orchestrate
22:58
this. It's not just one LLM.
23:00
You have like all of these models chained together in order
23:03
to produce a really high quality output.
23:06
And so that's why we've done a lot of the benchmarks is because
23:09
we want to be able to show like as you're evaluating all these
23:13
different models that are constantly, every single week,
23:15
there are new models coming out, you know, like Open AI just
23:18
released their new real time models.
23:19
Should I use that one? And you know, like all of these
23:25
TTS providers in STT are like constantly releasing new things.
23:30
And so I think with voice AI more than ever having but real
23:33
time benchmarks are constantly monitoring the performance is
23:37
super important for being able to understand what's happening.
23:40
I guess that's the important part for the majority of people.
23:43
It's like, how much is it going to cost, how fast is it going to
23:46
be, and how well is it going to actually say what it needs to
23:48
say? But then on the how well does it
23:50
actually say what it needs to say?
23:52
There's a lot of new arts there, I would imagine.
23:54
Definitely, yeah. So that's why like instruction
23:57
following benchmarks are really interesting of like is it
23:59
following the right set of steps for a 5 minute conversation, for
24:04
a 10 minute conversation, for a 30 minute conversation.
24:08
And that's where I think with real time models, it's still not
24:13
obvious that you should use real time over cascading models
24:15
because the controllability isn't there.
24:18
The instruction following is getting better and better, but
24:22
it's still, if it's not following the instructions, once
24:26
it's happening, it's kind of just happening.
24:28
There's all sorts of ways you can inject context, which is
24:30
interesting. And I think we're still early
24:33
days in different voice architectures with real time.
24:36
What do you mean by inject context?
24:37
So for example, like as the conversation is going, you can
24:40
have a background agent saying like, is this conversation going
24:44
well? And or like, is this, are we
24:47
missing any information? And so you can imagine that you
24:50
have like a real time model that's going back and forth and
24:52
then you have another model that's saying, what other
24:54
information do I need? Like the person asks a question
24:57
and then it goes and does a look up and then feedback that
25:00
context into the next into the next turn.
25:04
So I think regardless of if you're doing real time or
25:08
cascading model, cascading is where you have like text to
25:11
speech LM sorry, speech to text to speech versus using a voice
25:18
to voice model. Regardless, you still have this
25:22
kind of as the conversation progresses, you want to inject
25:29
the right context for what that turn is trying to do.
25:33
And so I think that's where actually voice AI has kind of
25:36
been very early to context engineering because there's just
25:38
far too much context to feed into every single turn.
25:41
That's exactly what I was saying.
25:42
And like, how do you keep that context on and the the most
25:46
important things right on a 30 minute call?
25:49
Yeah, exactly. Like you're exceeding the
25:52
context window. But also even beyond that, it
25:54
you start to very quickly have this needle in the haystack
25:56
because there's all sorts of extra conversation or if you're
26:01
doing a rag look up, you can't, you can't.
26:06
Those are like tend to be costly look ups.
26:10
And so doing that in a really efficient way that you can get
26:13
still fit within that one second latency budget when you're
26:17
already using the latency budget for lots of other things, I
26:20
think has always been a very hard thing for voice AI.
26:24
And so there is it's like a lot of interesting things are
26:28
happening around people having, you know, graphs.
26:30
So like as you go, as you move through the graph, it's like
26:34
dynamically injecting context around like once you get to
26:37
appointment scheduling, you know the availability is already been
26:40
pre loaded or like making sure that you inject the right step
26:47
for the next turn. So instead of saying at the
26:49
beginning throughout this whole conversation, I want you to 1st
26:52
ask the person for their name and their phone number and their
26:55
availability and then send them an e-mail follow up.
26:59
Putting that all in one prompt might not perform as well as
27:02
having a graph that's heuristics based.
27:06
And you say you're on, you're at node one on this graph.
27:09
And then after the next turn, you're like, OK, now you're at
27:11
node 2, now you're at node 3. And this is the step that you
27:15
should be doing right now. Yeah.
27:17
And I could see that for if you're trying to also close
27:24
somebody on something close like a ticket or close a deal or
27:28
whatever, And remember, you're stuck at node 4.
27:32
Like please try to move to node 5.
27:34
It seems like you're not getting past node 4 for some reason.
27:37
What's going on? Yeah, you can also add that
27:39
little context there. So I could see that.
27:42
That seems like it could be very valuable.
27:44
Now, what else was there on the reliability side?
27:50
I think, yeah, there's like so many interesting voice AI
27:53
architectures that are coming out of ending in the earlier
27:58
this year or even last year. Most people are just trying to
28:01
make things work end to end. You're just trying to say, does
28:04
the voice AI application respond to you with something relevant
28:08
in a reasonable amount of time? And now I think that is a that
28:12
was kind of stuff like that's the first problem.
28:15
Yeah, exactly. The lowest part.
28:18
And I think now we're seeing, it's been really cool to watch
28:21
the industry evolve over the last year where that's like the
28:25
first thing everyone cares about.
28:26
Even in the workshop yesterday as we were setting things up, we
28:29
talked a lot about latency. And the reason we talked about
28:32
about latency is nothing else matters if you can't even
28:36
respond, right? That's the Step 1.
28:38
And that's a that's still not, it's still not trivial to get
28:43
really low latency for voice AI. So then Step 2 is can you finish
28:48
the conversation? Can you have some direction to
28:50
the conversation and get to the end?
28:53
And then can you do more and more complex actions?
29:00
But I think that's where we're seeing a lot of really
29:02
interesting voice architectures of, you know, having this graph,
29:07
the this graph method or having background agents or having
29:11
multiple agents responsible for different things.
29:14
Having like one agent that's responsible for responding,
29:19
another agent that's responsible for coming up with context or
29:23
like all of these different things I think people are coming
29:26
up with is a really, it's been cool to watch these
29:29
architectures kind of evolved real time in front of us.
29:32
Yeah, Zach, yesterday was saying that they have a constellation
29:37
of models or agents. And a lot of times if when the
29:44
conversation is happening or when the other, when the user is
29:48
talking and they think that they might need something, they'll
29:52
just fire it off. Because in the unlikely case
29:56
that they might need it, they would rather like have it and
29:59
not need it than not have it than need it.
30:01
Yeah, exactly. It reminds me a lot of like
30:04
Instagram in the early days was able to have insane latency, not
30:10
because their latency was better in loading images than anyone
30:13
else, but because they did predictive loading.
30:16
So they were even before you entered your password, they were
30:19
pulling up all of your like pulling up the first images and
30:23
they were pulling up all sorts of like content that you would
30:27
see it immediately as you logged in or like pre loading, you
30:31
know, the one to the left, the one to the right, the previous
30:33
and the next one. I think there's tons of
30:36
opportunity there to do similar things to the voice AI where
30:38
it's, I think this conversation is going in this direction.
30:41
So I'm just going to pre load this type of information or I'm
30:45
going to start thinking about what to do next.
30:49
Oh, yeah, I didn't even think about the idea of, hey, the
30:52
conversation seems to be moving to node 5.
30:55
Let's get all of that stuff ready right now because it may
30:59
take us a minute. I was thinking about like in the
31:01
moment of I'm a slow talker, especially for a computer.
31:06
I talk very slow, right? That didn't come out right.
31:11
That made me sound like a computer, but especially as a
31:15
robot. As a robot, I found it really
31:18
hard. This entire conversation has
31:21
just been generated. Video working.
31:25
Demetrius have not actually been sitting here.
31:28
It would be amazing once it gets this good, but until then, I we
31:33
were talking about this earlier, how it's still a little uncanny
31:35
valley on the video generation, Yeah.
31:38
Yeah, like a little awkward. The head Bob.
31:43
Now you're trying to act like it.
31:44
I mean, yeah, so we are human as far as we know.
31:51
Yeah. And we're here.
31:54
And anyway, I was thinking like, in the moment that I'm saying
31:58
things, then it's grabbing it and it's like, oh, well, that
32:05
the whole thing that Dimitrios just said was 2 seconds long.
32:08
They can get a lot of information in two seconds and
32:11
then they'll have that ready in case they need it and then be
32:15
able to respond. If the whole process took 3
32:19
seconds, it's a second response. At least what I feel is a second
32:24
response, right? So I like that pre loading idea.
32:28
Yeah. I think there's also really
32:29
interesting things around more as you.
32:32
So following the right steps, when the steps are known, like
32:35
the standard operating procedure is customer service, these types
32:38
of things. But what about when you want
32:39
special exceptions or like negotiating or these really
32:43
autonomous behaviors? I think those are really
32:46
interesting. And how people can kind of start
32:49
to handle like this is like how an agent can negotiate with
32:54
these are the bounds in which I'm kind of comfortable.
32:56
This is how I would be reasoning about this social interaction
33:00
and how could you start to get agents to navigate those types
33:03
of situations autonomously is really interesting.
33:07
Yeah, I'm always worried about the negotiation.
33:09
I always feel like I'm not going to get a good deal here or the
33:14
agent is going to just go to the lowest amount.
33:18
Yeah. Well, I think it's, I think
33:20
there's a lot of interesting techniques from sales of or like
33:24
recruiting, right? Like large companies when
33:26
they're hiring have hiring bands and kind of you can create like
33:31
incentives and trade-offs. So I think, I'm not sure it's
33:34
still very early days in that area.
33:36
But like, how do you give agents a set of tools where maybe they
33:40
have a negotiation budget for a customer service or maybe they
33:46
have a like this is the amount of refunds we can give out.
33:49
Like based on all the conversations you've had today,
33:52
how many of these, which ones are the the most, you know,
33:55
deserving of those refunds? That or like memory across
33:59
conversations. That's not, I haven't seen that
34:02
happening today, but I think that's a really interesting area
34:06
where the memory. Across conversations like, Oh
34:09
yeah, Susie was just dealing with this problem too.
34:12
Yeah, yeah. And we solved it this way.
34:14
Yeah. I think that's actually another
34:17
area that is more people are exploring more is kind of
34:21
reinforcing certain pathways. So like I know I solved this
34:24
problem this way previously and that worked.
34:26
How can you in, how can you embed that within your
34:31
architecture so that you can have memory of maybe it's memory
34:36
or like reinforcement learning or like there's lots of ways
34:39
that you could implement this, but like how could you have a
34:45
how can you remember like what things worked and then use that
34:49
to make the agent better and better over time?
34:51
Yeah, that better and better over time and knowing like,
34:54
yeah, I've seen it this way and I can do it this way.
34:59
So that's how I'm going to do it.
35:01
I remember. So again, referencing the
35:05
conversation yesterday with Zach, he was like, still to this
35:09
day, like 80% of the Google searches have never been
35:13
searched before. Yeah.
35:14
And so The thing is that a lot of them are, like, similar but
35:20
not the same. And so he said that's when the
35:24
agents will get a little bit messed up because they think
35:28
they know and then they don't because they're like, Oh yeah,
35:31
this is the same as Susie. And yeah, no, but actually it's
35:34
a little different in this way. And the voice agent will be
35:37
like, no, it's Susie. No, no, I know how to do this.
35:40
This is how to do it. And so that's, that can get
35:43
super tricky, I imagine. Definitely.
35:45
I think memory in those cases is still really, really hard.
35:49
Yeah, 80% of use cases haven't been searched before.
35:51
It also resembles kind of really autonomous systems like self
35:56
driving, where a lot of the scenarios that you've seen,
36:00
especially down to like you've never seen before this exact
36:03
scenario, right? Like that exact person with that
36:06
exact dog with this exact street at this, with this lighting and
36:10
this, you know, weather conditions, all this with that
36:12
car is every single step is new. And how can you create a system
36:18
that's really autonomous but also being incredibly reliable?
36:23
I think there's a lot of parallels with voice AI of
36:25
you're going to have the same situations where you haven't
36:28
heard this voice before, but this background noise with these
36:30
frequencies with this exactly asking for this thing.
36:35
And yet you want to, so you want to be able to autonomously
36:38
navigate that with incredibly high reliability.
36:41
And I think that's just like a very exciting areas like how can
36:45
we create these really truly autonomous systems?
36:48
Yeah, that's such a great point. It's such a like great parallel.
36:54
And I think people kind of again, the like sentiment
36:58
towards voice eye is just so like reminds me so much of how
37:03
people talk about self driving where they're both saying
37:07
either, you know, tomorrow we're going to have self driving cars.
37:10
Like no one will be driving a car like in in one year from
37:14
now, no one's going to be driving a car and no one will
37:15
have a license. So we'll all just be like
37:17
driving autonomous cars. There won't even be like regular
37:21
cars a lot down the road. And then there's this other
37:23
extreme of people saying self driving is never going to work.
37:27
Look, it failed right there. It will never work like we'll
37:30
never have autonomous cars. And the true answer is somewhere
37:34
in the middle. It's going to take longer than
37:35
we expect. It's going to look different
37:37
than we expect. But the getting really
37:41
autonomous systems that are reliable is totally in the realm
37:43
of possibility for voice. And so I think we're both
37:46
underestimating and overestimating voice at the same
37:49
time, where we throw a voice at something and say like it
37:52
should, it sounds really realistic.
37:53
So it should just work for everything today.
37:55
And then on the flip side, people are saying it failed on
37:58
this one case and we can never trust it.
38:01
And I think there's like there's somewhere in between, right?
38:04
It's going to take a few years but and like iterations, but
38:07
we'll totally get there. What is it?
38:18
ISR. Maybe you're.
38:20
Oh my God.