Latest / Agentic Conversations (formally mlops.community) / The Challenge with Voice Agents

Transcript

The Challenge with Voice Agents · Agentic Conversations (formally mlops.community)

0:00
What's up everybody? We are back for this limited
0:03
edition series that we're doing with the process team, talking
0:07
about all of their work they have been doing on AI agents.
0:12
Today it's voice and we go deep into the experiences that
0:17
they've had building out voice AI agents.
0:19
I talked with Paul a little bit at the beginning about the
0:22
landscape, about what he's seeing, what I've been seeing.
0:25
And then we go into the tactical stuff with Flores to talk
0:30
through how he has been using open AI, real time API, and
0:36
specifically what the learnings have been because voice is a
0:41
whole different beast, as we will find out.
0:44
Let's get into this conversation.
0:50
All right, we're back with episode 2.
0:52
We made it this far and we're coming hot out the gate with
0:56
voice AI agents because we talked yesterday and I want to
1:02
give a bit of background. The way that we were going to do
1:06
episode 2 was say, hey, let's talk about all the frameworks
1:08
you guys used while building out some of these agents, what you
1:12
liked, what you didn't like. And then we said, you know,
1:14
frameworks have been around for the last two years.
1:17
There's a lot of content on the Internet about frameworks.
1:21
You have some people that love them, some people that hate
1:23
different ones and what we don't have a lot of content around or
1:29
experienced stories around our people building voice, AI
1:35
agents. Yeah, So what what we saw
1:41
already for a long time is that, you know, the ability to
1:43
generate a synthetic content with voice.
1:47
So you basically get take text and you have voice allows a new
1:51
way of interfacing or interacting with AI systems.
1:54
So it's a it's, it's a important next step for us to be able to
1:59
open up in particular in AB to C world agents to consumers.
2:04
And for a long time, we're experimenting, doing voice
2:07
cloning, you know, working with avatars, you know, Synthesia 11
2:12
Labs and, and many others out there just to kind of see how
2:16
this, you know, how this could look and feel.
2:18
And typically what was kind of the, the limiting factor is that
2:22
this was not possible real time because you could, you could
2:26
generate some voice. You know, there's a lot of sort
2:28
of light voice libraries that you could use standard voices
2:32
that you could generate to give them text and they would
2:35
generate audio text to speech essentially.
2:39
And they were great. They were becoming very good,
2:42
but they were still it was an offline interaction.
2:44
You have to take the text, and I had the voice, wait a little bit
2:47
and then go back. And it wasn't really real time.
2:49
Yeah. And about a year ago or so,
2:53
there was a moment where this started to become possible.
2:55
I remember very, very vividly a demo we had with a team from
3:00
Grok with a cue. And of course, their whole aim
3:04
is to kind of accelerate. Yeah, they're fast.
3:06
They're very fast. And that's what they optimize
3:08
for inference. Yeah.
3:09
And they were using that to enable real time voice
3:12
interactions. And then of course we saw the
3:15
Real Time Voice API from Open AI.
3:18
Well, we should mention with Grok, there's there's almost two
3:21
ways of doing this these days. It's clear that there's this
3:26
large push towards speech to speech models.
3:30
So the whole model can do everything that you need done
3:35
and the model takes input a speech and the model gives
3:38
output a speech. But before that became the trend
3:43
is what you're talking about with Grok, because Grok's so
3:44
fast on the inference side that you could set up these pipelines
3:48
and go input voice transcribe that, send it to a large
3:55
language model that is on Grok and it can do things really
3:59
fast. And I think someone told me
4:02
where it was like the token streaming time is like 300
4:07
tokens per second or something. Yeah.
4:10
So it's it's insanely fast. And then you output that with
4:13
some text to speech model. So there's that pipeline.
4:17
Three steps. So you had three steps.
4:18
You have to basically switch the bridge of the modalities with
4:21
different models and that that changed.
4:24
And then you had the open AI, real time voice API and others
4:28
that we were, you know, doing early testing with and so on.
4:31
And all of a sudden you could have, you know, agents that you
4:34
could talk to that was you could stream voice in and they would
4:37
stream voice out. And we did some tests also with
4:41
I Food, you know, one of the largest food delivery companies
4:44
in the world that were operates out of Brazil, seeing if we
4:47
could use that, for example, for riders as they were out there
4:51
and they were delivering and we were trying to figure out.
4:54
And by the way, just take this, take this for for a second.
4:57
One thing is to have a demo where you and I can do a, you
4:59
know, a demo with one of these voice agents.
5:01
Now take that same model, put it into the context of a food
5:07
delivery partner who is out on the streets in Sao Paulo who
5:12
speaks with, you know, Portuguese, the Brazilian
5:15
accent, of course, maybe even a colloquial accent with a lot of
5:19
traffic noise in the background. Is in a hurry trying to get
5:23
figure out, you know, which restaurant does do they need to
5:25
go to to pick up the order or what direction of the road is
5:27
closed or whatever. And they then expect a voice
5:32
model to be able to operate. So that was a great test because
5:34
it sort of takes something that works in a lab to being able to
5:38
test real life, you know, in real life.
5:40
That's complicated for so many reasons.
5:42
And this was them calling the restaurant to say, hey, where's
5:45
the order? They were calling I food to
5:47
basically ask the question. So normally they would otherwise
5:50
have to stop the vehicle, take out their phone text.
5:55
And so for to not lose time and to do that safely, basically the
5:59
ability to interact real time through voice was one of the use
6:02
cases that we were, we thought was, was a great way to kind of
6:04
really stretch the limits of the technology while also figuring
6:09
out if it would work for a real use case.
6:12
And of course, there are lots of things you want to test.
6:15
For us, it was 1 technical feasibility.
6:18
Can we actually get a model into this app in a way to kind of
6:22
test this, but also desirability, right?
6:24
Does a writer find this good enough?
6:26
Is it useful? Is it help buy food, answer
6:28
questions faster, safer, and so on.
6:31
And and then you enter a whole new world of complexity, right?
6:34
Because, well, what formats is audio being streamed in?
6:37
You're not. You're actually streaming, not
6:39
doing batch processing. You still need to retrieve real
6:41
data that you have normally in your agent tool, calling sort of
6:44
workflows from the I food. And that memory aspect that we
6:47
talked about last episode is so important.
6:50
How do you keep different things in context without losing what
6:55
is important and the thread of the call?
6:58
So with the one thing that I I am not clear about when it comes
7:03
to voice agents is on the memory aspect.
7:07
When you're doing the pipelines, you're just shoving new things
7:12
into the prompt or you're shoving the whole conversation
7:16
into the context window. How does that?
7:18
Work. Yeah, so a variety of things.
7:22
Of course you have a system prompt you need to stream the
7:24
content into the depends on the model we're using.
7:27
In this case, we were putting everything into the context
7:29
window. That of course has its own
7:31
limitation. It's, you know, we tested that
7:34
also on a bunch of other use cases internally where we wanted
7:37
it to be an expert on a certain topic.
7:39
We took like an internal education course and people
7:42
could call, for example, we get an audio interface for Tokan or
7:45
internal assistant. They could call Tokan and ask
7:47
about this course. And you could see that there's a
7:50
lot of these behaviors were, you know, if you just use a simple
7:53
quote UN quote text to text interaction and LM, then you,
7:58
you know how the context and the stuff you put in the context
8:01
will perform generally with the with the at the moment it
8:04
generates for voice. It didn't behave the same way.
8:07
Of course, we're testing different languages.
8:10
So you start with Portuguese and the moment you mention English
8:13
term, it started responding and in English and it doesn't go
8:16
back to Portuguese anymore, even though that's in the system
8:18
prompt. It's much more prone to
8:20
hallucinations. So this thing that we kind of
8:22
solved and LLMS in the voice model world, boom back like it
8:27
was just making up stuff. One thing that I wanted to talk
8:30
about was a bit of the landscape in this whole voice AI space.
8:38
You've got different players that are doing different things
8:41
and you've got folks that are are focusing on different ends
8:45
of this spectrum. Like in 11 labs that is doing
8:49
the speech or text to speech aspect.
8:52
And so if you have the text, you can turn it into speech.
8:54
And then there's like the Deep Graham, which is doing the text
9:00
to speakers. Yeah, I'm getting all confused
9:03
with which ways we're coming in. But voice to text is 1 aspect
9:08
and they also started just going from the text to speech and and
9:14
creating that whole voice API piece because I think everyone
9:18
knows that's where the future is.
9:20
And then you've got other ones like Cartesia is doing a speech
9:25
to speech model. Obviously Open AI is doing a
9:28
speech to speech model and doing the real time.
9:31
Are there other ones that you can think of in this space that
9:36
I've grabbed some attention? Because this is like the
9:38
foundation of what you need. And then you build apps on top
9:42
of it. And when you think about what
9:44
you need for this, you are looking at a way to get the
9:51
voice to the model. So whether that is on a Zoom
9:55
call or via phone, you need to get that information to the
9:59
model. The model then needs to do
10:02
something with it. It's either the speech to speech
10:04
model and it spits back out speech, or you do that pipeline
10:07
that we were talking about. And oh, the other one that I
10:10
wanted to mention, just the little tangent, is Hume.
10:13
Hume is doing like emotion detections and all that kind of
10:16
stuff, and they're creating that tool kit, but what else is
10:20
there? Yeah, I think they're, you
10:22
touched on a lot of them. I think there's one more which
10:26
we, we, we typically also look at is the ability to clone
10:30
voices, right. So if you can, so another great
10:34
example where you used to need a lot of good, rich, high quality
10:39
audio content to be able to clone someone voice.
10:41
Now you can do that with 10s of seconds, you can create a pretty
10:45
high fidelity voice clone. So there are, let's say, the
10:51
elements you mentioned. So the ability to transcribe
10:53
text to speech do that high in high fidelity.
10:56
But also that in itself requires, for example, language
10:59
detection, right? So some like open the eyes
11:02
Whisper does that. Well, you may, depending if
11:05
you're doing transcription for meetings or, or or conversation
11:08
with customers, you need speaker detection.
11:10
So there's actually, whenever we've put these things into
11:13
systems, agents, products, it's typically a bunch of things that
11:18
need to come together. So like I mentioned, speaker
11:22
attention, like language detection, the actual
11:24
transcription we do benchmarking against various languages
11:27
because again, prose is is a global group.
11:29
We've got lots of different languages.
11:31
We hardly work only in English. So if you do this in, in Hindi
11:36
or any other Hindic language or South America with his
11:40
languages, Eastern Europe, we typically try and measure how
11:43
well does this model perform just for that task.
11:46
Transcription, word error rates in that language, you've got a
11:50
translation, the ability to actually move across languages.
11:55
You know, if you have a use case, for example, you want to
11:57
create educational content or you've got a lot of education
12:00
companies in the portfolio, Udemy Skill Soft, Stack
12:04
Overflow, Brainly and many others or even others that do
12:07
corporate learning and so on. They typically want to, you
12:11
know, translate dub and so on. Well, you need to be able to do
12:16
that with high quality can be done offline.
12:18
So you don't need to do that like you and I are in a
12:19
conversation now. Then the other kind of
12:23
environment is you're in like the 1:00 we we mentioned in AB
12:27
to C context like I food or marketplace.
12:30
You want to be able to interact with the consumer or driver or
12:34
restaurant real time and there the ability to use this much
12:39
richer medium than just characters in a chat message
12:43
because you can, like you said, you can detect speakers,
12:46
emotions, intonation, accent, all those different things and
12:51
using that to better generate answers.
12:53
It's also something that you know, offers again, the whole
12:56
you need to actually know how to do that.
12:58
You can't just say, well, I'm getting voice and I'm giving the
13:00
right answer. What you think?
13:01
Well, me, is this person annoyed?
13:04
Are they? There's a lot of different ways
13:05
of saying are they? Yeah, exactly 1 word.
13:07
Yeah. And if in a learning context,
13:09
are they still engaged? Right.
13:11
And so, you know, language learning, how do you help them
13:14
with their accents? So all those different signals
13:16
you can now get out of this much richer medium.
13:18
Yeah, utilizing them is in a whole new problem space as well.
13:22
Well, there is two things that on these buckets that you're
13:24
talking about, it's almost like the first one is not so agentic
13:28
and the second one is very agentic because the first one,
13:31
if we're just dubbing my voice, which actually I have a funny
13:34
story about that. I was on a Brazilian podcast and
13:38
for the first half of the Brazilian podcast, I stumbled
13:41
through with my broken Portuguese and it was all about
13:44
data and I was talking about ML OPS.
13:46
So you can imagine how fun that was for me to be sweating
13:49
talking about ML OPS in Portuguese.
13:52
And then for the second-half, I think they just got sick of
13:55
hearing me stumble in my Portuguese.
13:57
They said OK, now talk in English and they would ask me
14:02
the questions in Portuguese, I would say them in English.
14:04
And later after the fact they went and they dubbed it mm hmm.
14:07
In Brazilian Portuguese. And the agentic part and really
14:11
where you start to see the voice AI medium come in is when you
14:17
can go back and forth and get information or get the agent to
14:21
do something. Yeah.
14:23
And that that's, I think we're, we're this is sort of really at
14:26
the frontier now, right. So being able to use function
14:28
calling with these voice models in real time, we found we're
14:33
struggling with that. It's going to be required to get
14:35
to get rid of these hallucination issues.
14:38
The system prompt and instruction following is
14:43
depending on the use case also not quite there yet.
14:46
So but we're getting there. I mean, it's like all these
14:48
things, right? It's this is a, this is not a
14:50
point in time, it's a vector. So we see the improvements over
14:53
the last 12 months on this front and, and, and once we get those
14:59
things right, in particular the function calling, that will make
15:03
it a lot easier to then also use the agentic workflows, you know,
15:07
be able to access tools and so on to get higher fidelity
15:11
conversations. Let's talk about some of the use
15:14
cases that we've seen, because I feel like there are some really
15:20
cool ways folks are building apps on top of this base that we
15:24
just mapped out. Yeah, a ton.
15:26
So we're focused on the e-commerce space.
15:28
So typically with AB to C and AB to C world, there are lots of
15:33
customer touch points that you can, you know, you know, you can
15:36
think about now changing because of voice.
15:38
Our geographies, you know, we, we, we, we sit here in
15:41
Amsterdam, but the people we work with the, and the consumers
15:44
of our platforms there in Brazil, again, like I mentioned,
15:46
India, Eastern Europe, they're very, you know, they're very
15:50
different in the way they use technology.
15:52
There's much higher usage of voice in general.
15:54
If you think about the percentage of, for example,
15:56
voice messages on WhatsApp in Brazil is significantly higher
15:59
than than here it would be in the Netherlands.
16:02
So voice is also in many ways a much more natural, let's say
16:09
capability in some of those markets, especially in the B2C
16:13
application. You know, if as we work with our
16:16
restaurant partners with you know, real estate partners that
16:19
are listing stuff on our platforms, being able to confirm
16:23
certain things with them and check that the menu is menu is
16:27
fine and using that as sort of more seamless touch points.
16:31
And then the category I still think is, is heavily unexplored
16:36
as we give users the ability to interact with us to with, with
16:39
our platforms through voice. How do we use that as a way to
16:43
help them better detecting the intent that they have?
16:47
Because they speak to us and say, hey, you know, we we're
16:50
looking I'm looking for let's say something simple and light
16:54
tonight, right. Then you say, well this is Paul
16:57
is using this account he's ordering such and such thing.
17:00
He's maybe agitated or whatever. So get him the fast, the fast
17:04
order with discount, right? Yeah.
17:06
So, but it's, it's a much more richer medium.
17:08
And then all these things apply not just to the food ordering
17:13
space, but also as you may have seen, we, we know we recently
17:16
announced that we'll be partnering with this big R,
17:19
which is a travel platform in, in, in South America.
17:23
They will join the group. And so there, there's going to
17:25
be a ton of additional travel use cases.
17:30
So I think this is in fairly unexplored space.
17:33
Lots of ideas and many things to test and try and and and build
17:37
still. Some use cases that I've seen
17:39
that have been quite novel to me have been in the funny enough,
17:45
very regulated space of healthcare, health tech, and
17:50
helping doctors or psychologists or whoever it may be work faster
17:57
and spend less time on admin than they would need to if they
18:01
because we don't know this because we're not doctors,
18:03
right? Or at least as far as I know,
18:05
you're not a doctor. I'm a doctor in the wrong kind
18:08
of of engineering, so not very useful.
18:11
All right, Doctor, I didn't know that.
18:13
That's cool. No.
18:15
So the thing about it is that for the most part doctors can
18:20
spend, let's just take the doctor example.
18:22
Doctors can spend a lot of their time on admin work, whether that
18:27
is for insurance stuff or just talking about what they think
18:32
the patient has or where they think there's maybe pain points,
18:37
what prescription they're giving and why, etcetera, etcetera.
18:40
So that is the less agentic use case because maybe you can
18:45
create a solution that will give someone or you the doctor brings
18:51
in their phone and records the conversation and then it helps
18:54
them fill out that form after. Or you can have it where an
18:58
agent actually just asks the doctor after says, what do you
19:01
think about this? And the doctor talks to the
19:04
agent like they would talk to their nurse or whoever.
19:08
Yeah. Well, there's one company in the
19:11
group, it's called Corki dot AI. And there, as I mentioned
19:15
previously, we invest in lots of companies, many of them AI
19:17
companies. Corki dot AI is one of them in
19:19
the healthcare space. And what these guys have done is
19:23
they, they were in the space already for some time.
19:25
And of course, this is a good example where this super smart
19:28
team, very capable AI native, they're building on the, on all
19:33
of these waves of or new capabilities to come out,
19:37
including voice. And they, their proposition, one
19:40
of the important ones is helping healthcare workers, doctors, but
19:44
also others that are working in emergency departments, picking
19:46
up phones and so on to better assist, you know, whoever's on
19:52
the other side of the call. And they do that by basically
19:56
giving them an interface that during the call and they, their
20:01
models listen in and help the, the, the healthcare worker ask
20:07
the right questions. You start to code and say, this
20:10
is likely an emergency of this and this type, you may want to
20:14
ask such and such questions. So they help them basically
20:16
triage and navigate down the tree of possible options next
20:20
best. Action.
20:21
Yeah, exactly. But in a, in a very assistive
20:23
way and to help the folks as they go through the calls and,
20:29
and a lot of that is live in real time.
20:31
So they've had to build a lot of their own technology.
20:34
Of course, in the healthcare space, you can't afford to make
20:36
mistakes. So you need to make sure these
20:39
models are, are, are accurate. So they have a fair fairly
20:42
sophisticated set up to evaluate the accuracy of the models, but
20:46
also had to do a lot of, let's say their own training, deep
20:51
model development to be able to extract healthcare specific
20:55
terms. And and that in fact is, you
21:00
know, it's helping, you know, thousands of folks around the
21:03
world as they are putting this into in front of healthcare
21:05
workers in that daily basis. Yeah, and that one.
21:08
'S fascinating because it's not necessarily that full pipeline
21:11
that we talked about. It is voice in and then agents
21:14
helping, but you don't have voice out.
21:17
So it's still is agentic workflows, it's still is using
21:21
voice AI, but it is not necessarily everything we talked
21:24
about. And I like that you give that
21:26
because it's a bit more creative thinking.
21:28
Yeah, that's. You actually bring up another
21:30
topic, which also again, isn't specific to voice, but
21:33
especially as we think about more sophisticated agent is the
21:36
ability to proactively reach out to at the right time.
21:40
Right now, all of these systems, whether or voice or not, let's
21:45
say these are genetic systems. They're basically dependent on
21:47
you coming in with a question or task.
21:49
And if you're lucky, then maybe the the agent will ask you one
21:54
or two clarifying questions before it starts to execute the
21:57
task. Because most of the time they
21:58
just greedy execution, right? They just go and they they
22:01
execute as we learned in the. Last episode.
22:04
It can't say no. It can't say.
22:06
No. And there and so you know, when
22:09
you are in a, let's say in a process, you're trying to do a
22:14
task, in this case, fill your taxes, which may take you hours,
22:17
right? You want to have the ability for
22:20
a assistive system to come in at the right moment in time and
22:24
say, hey, shouldn't you think about this or have you forgotten
22:26
about that? And sort of it's almost like
22:28
another prediction, like should it intervene?
22:30
Should it proactively reach out to you?
22:33
Hey, haven't you forgotten to also take this form or whatever?
22:37
And of course, voice is just one way to kind of connect with you,
22:42
but it's not the only one, right?
22:44
One that I. Have not seen in the voice space
22:46
that I know we've made predictions around that there's
22:50
going to be agents in the e-commerce area.
22:55
But I want my personal agent to go and do things for me and
22:59
especially in the voice realm where I'm stuck on the phone
23:02
with Vodafone, for example. And it is so painful because
23:07
they're passing me around to different agents, actual human
23:10
agents. But each human agent doesn't
23:12
know what I'm calling for. And I would like to be equipped
23:17
to send my agent out to do my bidding for me and figure out my
23:21
phone bill. Why did I get charged extra?
23:23
I can tell the agent, go figure out, go call Vodafone, figure
23:27
out why I got charged extra. And so it's almost like this
23:29
personal agent that can sit on the phone and interact with all
23:34
these companies, especially because already the companies
23:37
that you are faced with on a day-to-day basis, they have some
23:41
really bad voice. It's not agents, it's just
23:46
robotic calling and you have to interact with that.
23:50
And so it would be really nice if a voice agent could do that
23:54
for me. I.
23:55
I completely agree. It's kind of an asymmetric
23:58
reality that typically the call centers you're trying to call
24:02
the big companies have their agents first.
24:04
It's like IVR and then become something a little bit more
24:07
sophisticated while we are still stuck with our own human time.
24:11
I know. So we should have our own
24:13
calling agent. Yeah, yeah.
24:15
And by the way, I don't think that's that hard to implement
24:18
to, you know, exactly to to to build their test.
24:22
You know, we'll have a Florissant to talk about some of
24:25
the tests that we've done where we've used this interface to
24:28
kind of do and test things that basically you send out an agent
24:34
that you interact with with through voice to do things for
24:36
you in an e-commerce space. Go order me some food.
24:38
Go buy me this, go find me this. So we're, I think close you'll,
24:43
you'll find 252025 to be a good year for you.
24:46
And I hope so send these agents out on behalf of yourself.
24:49
Yeah, well, Speaking of. Flores, let's bring him in now
24:51
and let's have a conversation on what he has found while creating
24:56
different voice agents. I know he's played extensively
24:59
with the real time APII also know he's had some insights
25:02
because again, this is like, let's pull back the curtains.
25:06
Let's see, you've banged your head against the wall so many
25:09
times. How can we help other developers
25:11
not do that same thing? Welcome back, dude.
25:17
Talk to me about voice and the differences that you've
25:20
experienced between voice and working with voice AI agents
25:24
versus text, OK? Yeah, so it was a journey, you
25:29
know, it's because I can remember the first time I heard
25:32
about the the audio agents coming live.
25:36
I was like, oh, but we can just copy paste the prompt we have
25:40
for a certain agent. It's easy.
25:41
We put it in the real time API and it still works.
25:45
Yeah, that was kind of the dream I had and it wasn't true.
25:48
It didn't happen. No, it didn't happen.
25:51
And yeah, so there were, there were a few things.
25:55
I think the, the, the first one was of course, that if you type
26:01
something and I type something, it's exactly the same.
26:05
But the moment I'm I'm, I'm, I'm, I'm talking that or I'm
26:10
speaking that sentence out loud and you're speaking that
26:12
sentence out loud. They are.
26:14
They are quite different. That's true.
26:16
Well, just in the pauses or in the intonations, the way that I
26:21
pronounce words, there's so many variables there.
26:24
Yeah, yeah. And they add up like it's a,
26:26
it's an error that can propagate.
26:28
Like if the beginning of the sentence something was unclear
26:30
and then there's not another word that is unclear.
26:33
And that whole, that whole paragraph that you're just
26:37
saying to the agents is just becoming super vague to it,
26:41
which is, which is just a dimension that is really hard to
26:44
to tackle right of the start. Well, it's funny.
26:48
That you mentioned there's different dimensions that you
26:51
have to be thinking about because I know you, you said to
26:55
me at some point you have so many different events that fire
27:00
when you're working with the agents voice agent specifically,
27:04
right. It's not just, oh, hey, there's
27:07
some text and now we interpret that text.
27:10
There's so many different pieces, so maybe you can map out
27:13
some of those pieces when we have texts.
27:14
You know, we send the messages and the messages are really
27:17
structured. You know, we have or yeah, we
27:19
have the tools and the messages and it's like the user says
27:22
this. This isn't says this, but if we
27:25
move to real time, you know, this all becomes real time.
27:29
So it's asynchronous. You know, you're connected over
27:31
a web socket and you're not having like, hey, these are the
27:35
messages. Now give me one response.
27:38
You know, you can constantly are you're you're getting responses
27:41
all the time and you can send responses all the time.
27:45
So knowing exactly how did this process and saved in the, in the
27:50
session as they call it, the real time is really important
27:54
because then you can know how you can manipulate this.
27:57
So a, a funny thought experiment is like if I would now continue
28:01
to talk and you would interrupt me halfway.
28:02
Yeah, I I would still remember what I wanted to say to you.
28:07
You know, if you have the old system where it was text to
28:11
speech and speech to text and then to LLM, the LLM would have
28:17
the memory that it would told you that part.
28:20
Real time API changed this because they they have the
28:23
interruption event. I'm not sure it's not the right
28:26
word, but they have this event saying like the speaker
28:29
interrupted you and they would trim the output from the LLM
28:34
perspective as well. So the LLM knew where where it
28:38
got interrupted and knows that you don't know that part.
28:42
So if you would ask something, it would not say like, oh, I
28:44
told you that last time, but actually it wasn't a trim part.
28:47
So that's like that's, that's just a whole new way of working
28:50
with that session. And they have a few nice visuals
28:54
on the, the API reference page. So for the viewers I would
29:00
recommend watching that instead of me trying to visualize it
29:03
with voice you can say. Things a certain way, you can
29:08
say them in certain accents. So you have all of these
29:14
different vectors that you pull on, but you also have the
29:18
latency requirements that you have in any tech.
29:22
And you also have the piece around like, is what is being
29:26
said actually relevant to the question I asked?
29:30
Yeah, yeah. And.
29:31
Like one of these parameters within the real time, which is
29:35
quite difficult is the turn detection.
29:37
So if, if I'm talking now, you know, there are there are some
29:42
pauses which are natural, which are just like, you know, give me
29:46
room to think or because I can't find the next word.
29:51
But others are like, I'm done talking.
29:53
Yeah, if I'm done talking, you know, you want this agent to, to
29:58
talk back. And that's yeah.
30:00
And that's what the the, the, what the turn detection is
30:03
doing. It wants to have the the least
30:05
amount of time of of silence, but still kind of have leaves
30:10
room for you to have these stinking moments.
30:13
Yeah, not interrupt. Yeah, yeah.
30:14
But this is. Highly personal yeah.
30:16
So if I would now make making a voice agent for for me, I would
30:20
tune these parameters so that so that my way of talking would be
30:25
perfectly aligned. Interesting.
30:26
And then I gave you this agent. I was like oh, it's so
30:28
incredible. You know it will never interrupt
30:30
you and you will start talking to it and it just starts to
30:33
interrupt you because you're a stylist there.
30:35
Exactly I. Have long pauses thinking or
30:38
whatever and the agent doesn't realize that and so you have to
30:42
tune it in a different way. But the Speaking of which I had
30:46
a friend who who is building agents and he was saying that a
30:50
lot of times when the user will say something, the agent will
30:56
think that it is trying to talk. So normally you have an agent
31:02
that will be talking. And if it was with a human, I
31:07
would be nodding along and maybe I would say, yeah, OK.
31:12
And my friend was saying that a lot of times the agent will stop
31:15
talking because it thinks, oh, you want to talk?
31:17
I just heard you say something. This is one of.
31:19
Those parameters you can can change, you can say like what is
31:23
what is talking. You know if it exit if you talk
31:26
for longer than X amount of milliseconds, that's considered
31:30
talking. So if you mm hmm, can be
31:32
detected as as as as something you want to say and not, you
31:35
know, so it's it's it's all all these different things.
31:38
And like I said, highly personal.
31:40
Yeah, actually another. Insight from my friend who's
31:43
building these agents, he said that he found the product worked
31:48
better when they made the agent a little less good.
31:56
I guess in a way. So if you make the agent worse
32:01
and not as good as it is possible, then humans are more
32:07
sympathetic and they're more patient with it.
32:09
And they also are able to, you know, they talk slower, they
32:14
enunciate their words more, and it's much better for him and his
32:18
product to get that. And so if you make the agent as
32:23
good as it is capable of right now on these different factors,
32:28
you mistakenly have the customer thinking that it is better than
32:33
it is and you get a worse result and that's an.
32:37
Amazing this amazing view you have this like this mirroring
32:40
aspect that people have naturally, you know that you
32:43
kind of adjust to the way all the person is talking that's
32:46
actually a really cool one I've I've not heard that before, so
32:49
I'm going to write that down for later so make it yeah tell.
32:52
Me if you make it worse. There you go.
32:54
Amazing. Now I know you guys did some
32:57
really wild stuff with evals to see what was working, what
33:02
wasn't working. Can you talk to me about the
33:03
custom evals that you did for the voice agents?
33:06
Yeah, so. It was pretty cool.
33:08
Like, you cannot just launch this, you know?
33:10
You cannot just say like, hey, put it out in the wild.
33:13
We have a we. Have a assistance?
33:15
Works by the phone. Good luck.
33:18
So we needed a way to know the boundaries.
33:21
You know, we talked about it earlier like, you know, knowing
33:24
where it feels also kind of breeds confidence in deploying.
33:29
And since it was like real time, maybe I was a month old, you
33:34
know, so there, there was no testing framework.
33:37
So we came with this, this crazy idea of any like, OK, why don't
33:41
we use real time API to test the real time API.
33:45
Since, you know, we had dozen of experiments where we were able
33:48
to make the real time API move in all these weird places where
33:52
like, you know, talk with a German accent, you know, or talk
33:55
really slow or super fast. You know, it was lots of
33:58
emotion. Yeah, exactly.
33:59
Like you were able to prompt it to, to, to talk differently.
34:03
So we build an entire system where we would randomize
34:07
different characters of different properties saying
34:11
like, you know, the talk fast, slow, talk with an accent or
34:15
not, you know, British or American, but also be sad, you
34:21
know, or be happy. And yeah, we had like a wild
34:24
range, you know, we went all out because if we thought like, we
34:27
can automate this, then we can just test whatever.
34:30
You know, that's the amazing thing about automation.
34:32
You will do much more the moment you automate it.
34:34
But yeah, so we tried this and we just ran like hundreds and
34:39
hundreds of experiments saying the same stuff and then seeing
34:42
how the phone agent would react and it needed to like order a
34:47
pizza, but then in all these different styles.
34:50
And the fun thing was that the result of that experiment was
34:53
that we saw that Italian and Spanish were like mix and match.
34:57
So the moment there were Italian or Spanish, it would just swap
35:00
over to one another. Like if it if it didn't really
35:02
understand that there was a different language, which was
35:05
really funny. It's the hybrid.
35:06
Language. Yeah, so I have the same.
35:08
Mistake sometimes, you know, like amigo, that's Italian,
35:10
right? But so it's good to see that
35:15
these smart models also struggle with that.
35:17
And you know, it's, it's it's the same where the German accent
35:23
would sometimes convince the IT to talk German back.
35:27
So you had like the Italian accent, it was out of fine.
35:30
But then, you know, when you had the German one, it would just
35:33
start talking back Germans. The model would respond in.
35:36
German sometimes, yeah, but we. Saw that, that, that, that that
35:40
accents could indeed trigger it to kind of go over that language
35:43
barrier. And the final like what are we
35:46
discussed with the the turn detection, you know, that is
35:49
something we saw with the slow and fast speaking that the
35:53
moment we had slow speaking, it would randomly trigger that turn
35:58
detection. So people who were talking
36:00
really slow were really difficult to have consistent
36:04
because if the pulse was just a few milliseconds too long, then
36:08
you know, it will blow the experiment.
36:10
And those were all the things we were seeing by generating all
36:13
this synthetic data sets. So that was quite, quite a
36:15
innovative approach. And since this whole thing was a
36:18
month old, you know, we're like, we're probably the first doing
36:21
this. It was pretty exciting, but and
36:24
do you think with that turn? Detection.
36:26
It will become more personalized later on and be able to adapt
36:32
real time. Is that like, How do you see
36:34
that problem being solved? Yeah.
36:37
It's, it's, it's a difficult one.
36:39
Like I think first of all, people just need to learn, you
36:43
know, with to, to how to work with the normals.
36:47
Yeah, that's the quickest approach.
36:49
But I do see the opportunity for companies because also in the
36:55
real time API, you can say I do my own turn detection.
36:58
You don't have to do it on the open AI side.
37:01
So Open AI is just a turn detection model.
37:04
So it would be amazing if the open source community would now
37:07
work on a turn detection model that has that is adaptive or you
37:13
know, because if we can clone a voice in 10 seconds, why can't
37:17
we detect the turn settings of a person in 1000%?
37:22
Yeah, Well, yeah. Speaking of open source, there
37:24
is a model that came out and I think you were playing with it a
37:27
little bit, right. Which model was it?
37:29
Yeah. The Kokura.
37:31
Yeah, Kokura, I'm not sure if I'm pronouncing it right.
37:34
I'm I'm I'm not good with names. The audio of the real time API
37:38
isn't also so you know, yeah, no, but this, this open source
37:42
model, it blew me away and it was not you know, we've seen 11
37:47
labs, great quality, you know, amazing, amazing work for the.
37:53
Text to speech aspect, right? Yeah.
37:55
So this is text to speech, Yeah. It's not real time, yeah.
37:59
But it, it deserves a shout out in this in this episode because
38:05
it's it's really amazing. Like they managed to get really
38:10
good performance from 82,000,000 parameters.
38:15
So all these models were in the billions.
38:17
And these guys just say like, OK, we have here a model that's.
38:22
So if I run it on my MacBook, I get five times the speed of real
38:26
time out of this text to speech. So it can generate five times as
38:31
more speech than it has the time for just on a MacBook, you know,
38:38
and they claim to to, to reach over 100 times if you have like
38:42
a proper GPU. But it's it's that's liberating
38:46
because you're basically democratizing the, the, the text
38:51
to speech because everyone now can download the model and
38:54
generate speech playing with it. Yeah, for free.
38:57
Yeah, and that's a big move. So I'm wondering if 11 laps is
39:01
sweating at this time before we wrap?
39:06
Up, I want to talk a little bit about like how you think about
39:10
flows with voice AI agents, specifically comparing them to
39:15
something like the token web usage or Slack bot that you can
39:21
interact with. We heard from Paul that it's
39:24
really difficult to do all this in real time.
39:26
The latency starts to get really high and it's not quite feasible
39:31
yet. Yeah, so.
39:33
Feasibility is a large worth, but there are there are
39:37
definitely things you need to rethink.
39:41
I still remember the first time I had the the agent running on
39:45
my laptop with some tools and I was like, oh, I'm so amazing.
39:48
I I made this voice agent work. It will definitely be great.
39:53
And I started talking to it and it called the tool.
39:58
I was really happy, but then it was just that silent and it was
40:03
awkward. I was like, Oh yeah, you tool.
40:07
Because you don't know if it's working.
40:09
You you may might have known because you were looking at logs
40:12
and you were seeing it. But if it was me on the other
40:15
end of a phone call, I wouldn't have known.
40:16
What's going on. And are you still there?
40:18
What's going on? Yeah.
40:19
So it would just like. Stop talking because it was
40:22
executing the tool because it was the logical next step to do,
40:27
because I prompted to do that. You know, if, if the user says
40:29
this, call the tool, it was waiting on the tool output and
40:33
then it was reading out the tool output.
40:35
You know, it's like, hey, I just got this tool response and it's
40:38
saying this and and it was like, oh, that's an awful experience.
40:43
You know, it was truly awful. And I was like, this is not
40:46
going to work. And then then we knew that we
40:48
need to rethink these things. So there are a few tricks you
40:51
can do. You know, it's even if a model
40:55
calls a tool, it can do it can. So going back to the text, it
41:02
can do text and a tool call audio is the same.
41:05
It can do audio and a tool call. So you can prompt it to say, you
41:08
know, if you call a tool, you know, tell that you are doing it
41:12
and maybe explain what the parameters were in the in the in
41:16
the tool call. That can kind of buy you a bit
41:18
of time. But the other side is, of
41:21
course, make the tool really quick.
41:23
So don't add tools that take long in a real time
41:26
conversation, you know, then it's just better to offset.
41:29
Then you're just saying like, Hey, that that thing you want me
41:32
to do? It's it's being initiated.
41:35
You know, you will get a process message or when it's done I will
41:39
send you an e-mail or a text. So you separate the workflow.
41:43
So don't do stuff real time that cannot be handled at real time
41:48
speeds. So that's like the first, first
41:51
realisation where we're like, OK, only fast tools we're going
41:53
to keep. And then we're going to offset
41:55
these jobs, you know, where it's like, OK, the actual ordering of
41:59
the food, that is a job. And we say, like, we're going to
42:02
order you food. That's it, You know, You know,
42:04
they're going to call the pizzeria, and then they leave
42:07
you on the line until their pizza's done, right?
42:10
So it will be really weird. So yeah.
42:13
So that's kind of the same with the real time voice.
42:15
And yeah, a second one is you need to rethink flows in a way
42:21
where you need to avoid spelling.
42:25
So if I say, hey, my name is Flores, you know, get me my
42:31
medical record, you know, to come back to the medical
42:34
discussion and it can then spell my name on 20 different ways.
42:41
You know, it can even detect like a normal English word, but
42:44
it could be really awful at at getting my spelling right.
42:47
And that's essential for the next step.
42:50
So you could kind of work around that.
42:53
So maybe let's take a more an HR setting.
42:55
So it could ask me like, Hey, what team are you in?
42:59
I'm like, hey, I'm in the AI team and I say, OK, what's your
43:02
name? And then it already has a short
43:04
list of names. So it has like 25 options.
43:08
And then I say Flores. And then it's precise because
43:11
then it knows from all these names, Flores looks the most
43:15
like this spelling and then it can map.
43:18
And so redoing that workflow are now you are overcoming the
43:23
spelling issue. So it's not a limitation.
43:26
So where we started, where we're like, are we, is it viable?
43:32
So yes, things are not viable, but there are workarounds and
43:37
just we need to be more creative.
43:39
And I'm, I'm not sure if if spelling will ever be something
43:43
that will be solved from. Yeah, well, you saying that
43:46
brings up a. Really cool company that I've
43:49
seen called Canonical AI and they basically are doing product
43:53
analytics for voice AI agents. And one of the things that they
43:57
do is they see where these voice agents are falling flat.
44:00
And it gives you, as the creator of the agent, the ability to
44:04
click in and recognize. We have a blind spot when people
44:09
start to say their names or we have seen that there is for some
44:16
reason on this flow, a lot of users disengage after the agent
44:23
comes back. And so you get to recognize out
44:25
of the way that people are interacting with your voice
44:29
agents. Where is it not working?
44:32
Because it could be, again, it took too long, or it could be
44:36
it's just giving the wrong answer or it didn't respond that
44:40
it was thinking. There's so many X factors going
44:43
back to the initial part of this conversation that you want to
44:47
try and find where these flows are breaking down or where
44:51
customers and users of the agents are not having the best
44:55
experience so that you can try and tweak the flow a little bit
44:58
more. Yeah, that's.
45:01
Super valuable, you know, especially at this point where
45:04
nobody really knows where these things break.
45:07
The more insights and especially real time insights, if you're
45:11
able to follow your age in real time, you can react fast and you
45:16
know the pain points and you know how to improve or think
45:19
about if you're a call center. And you all of a sudden are
45:22
having 20. Or. 50 or 500 calls that are
45:28
hanging up after the same step. And really a call should be a
45:32
minute, but now it's like there's an average of three
45:35
seconds for this call. You want to be alerted of that.
45:38
And it's real time insights to say, hey, something's going on
45:41
here and it's like your data dog or your Prometheus in software.
45:47
Yeah, it's it's. It's it's just it needs to be
45:49
there if you're building software products to have these
45:51
real time insights. Yeah, I think it's a it's a
45:54
amazing company. So one of these insights we
45:57
actually gained from looking at these logs was actually that it
46:03
was overconfident, you know, if people would mispronounce or mis
46:08
say something so it would never ask the question, can you repeat
46:14
that? You know, and it's such a normal
46:17
thing to do for humans, you know, if they're uncertain, but
46:20
it's exactly the same as we have a text like they're
46:23
overconfident and they would never give you a no.
46:27
And the same is with text. So I'm I'm really, I'm really
46:30
looking forward to the first audio model that will what they
46:34
will tell me like can you repeat that or it just says, huh, yeah,
46:38
that's the next frontier, you know, can you repeat that model?
46:42
That will be good. All right, that's it for today
46:49
A. Huge shout out to the process
46:51
team for their transparency because it is rare that you get
46:54
companies talking about their failures, especially companies
46:59
that are this big in the AI sector and really helping the
47:03
rest of us learn what they had to go through so painfully.
47:09
Sometimes a mention that they are hiring.
47:13
So if you want to do cool stuff with the team that we just
47:16
talked to and even more, hit them up.
47:18
We'll leave a link in the show notes.
47:20
And if you're a founder looking for a great design partner on
47:24
your journey, then I highly encourage you to get in touch.
47:27
We'll leave all the links for all that good stuff in the show
47:30
notes.