Latest / Agentic Conversations (formally mlops.community) / Voice Agent Use Cases

Transcript

Voice Agent Use Cases · Agentic Conversations (formally mlops.community)

0:17
There's a challenge with controls where since there are
0:20
so many controls, if you have like a developer team within a
0:24
company, product focus company, do this, then often what I see
0:30
with customers is that if you give too many controls and if
0:33
they exactly don't know how to sort of set it up, it is like,
0:36
oh, this just doesn't work right.
0:38
So I think there's always a tension between like
0:40
preconfiguration, right, and giving more flexibility.
0:45
But if you give more flexibility, then for example,
0:48
sometimes people are like, OK, why the latency is so high for
0:52
speed generation? Because, you know, we maintain a
0:55
certain buffer size to make sure that the the speed generation
1:00
has enough context, right, to be prosodi aware to to have the
1:06
right tone, maintain the consistency tone and higher the
1:09
buffer size, you will have like higher latency.
1:12
But now if you let's say, let them reduce the buffer size, the
1:16
speech quality sort of goes down this time.
1:18
And then they're like, OK, yeah, you're complaining, right?
1:21
So I think that is a general challenge.
1:23
What is the right abstraction? Right.
1:25
So we should also talk about general abstraction of agents.
1:29
So many of the companies building orchestration, like it
1:34
VAPI on the tech side, launching these are meant for like tools
1:43
to be built by software engineers, like you build agents
1:46
over them. But like I see, let's say in
1:48
customer support, customer support is more like a cost
1:53
Center for a company, which means you'll have like less
1:56
number of engineers there. And the, the, the real person or
2:00
the real customer is the operations leader there who is
2:03
basically managing the workforce there and one student end to end
2:07
operation. But like the current interfaces
2:10
of our existing orchestration systems are not meant for them,
2:13
right? And that is where like we have
2:16
recently received a project called 11 labs for support by
2:19
Sierra Decagon, where we define these interfaces for
2:22
non-technical sort of authors to, to define the behavior of
2:26
agents, right? So even before AI agents, you
2:30
will see what, what would I was in Amazon.
2:32
And if we have let's say 100K support agents across the world,
2:36
many of them are not full time, right?
2:38
Like they can come and they can leave at any point, right?
2:40
Which means there's not really an opportunity to train them on
2:43
specific support. So customer support always had
2:46
these like pages of documents called Sops and every time you
2:50
call someone, they say, oh, give me a moment to look up your
2:53
information. They're literally reading that,
2:55
right? That way this person is reaching
2:57
out for refund. What should I do?
2:58
I should look for information about Demetrios, then I should
3:02
check his eligibility for refund, his past sort of
3:05
purchase behavior, blah, blah, blah.
3:07
And then that that takes time, right?
3:09
So these SO PS were written by the operations leaders or
3:13
managers or customer support managers and they would use that
3:18
to for human agents to follow them, right?
3:20
And then the second mode of sort of interaction between them
3:24
would be that once the human agents are sort of following
3:27
those Sops, sometimes they would not follow them and they would
3:32
break the rules or be non compliant to that.
3:35
And then the operations manager has mechanism to listen to calls
3:39
and then give them feedback, right?
3:43
So I think my thought process is that if we need to make like
3:45
agents used by non tech authors, which are which are largely
3:51
which is most of the population who would be using agents and
3:55
they still need autonomy on the behavior of agent, right?
3:58
What an interfaces we need to build.
4:01
So I think on customer support specifically, can I keep the
4:05
interface same, right? So instead of human agent, can I
4:09
have an AI agent? But let's say if I'm the support
4:11
manager, can I keep the interface same where I still
4:14
define SOP's now, right? Just like I used to.
4:17
But now I can go very very detailed and long because I
4:21
don't have a human constraint of not being able to read a long
4:25
documents, right? Like all large tables, right?
4:29
So I think that can change, but like the regular interface can
4:33
be the same. And then the second is just like
4:35
I was providing feedback to human agents, can I also sort of
4:39
provide feedback to AI agent that oh, you provided wrong
4:42
information there or you sort of broke this rule or you did not
4:47
follow the SOP or sometimes the SOP is not clear enough.
4:50
So the agent got confused. Even humans get confused.
4:53
How do we take that feedback and sort of iterate on the
4:57
specification and procedure. So I think that is quite
5:00
interesting that like how do we sort of keep the interface of
5:04
interaction between sort of domain experts who may not be
5:09
software engineers, same as how they were sort of working with
5:13
human agents, right? And and if you were to do that,
5:17
then there's a lot more heavy lifting to be done, right?
5:20
Which means continual learning, which is pretty hot topic right
5:23
now that how do we make these agents improve based on natural
5:27
language feedback, right? So we can talk about some design
5:30
patterns for that, how we are doing it, how companies are
5:33
doing it. So yeah.
5:35
Why do you feel like voice is harder than chat?
5:39
Yeah, so even chat is not easy, right?
5:41
Like chat has not been solved and I've spent more time doing
5:45
chat than voice to be honest. And I don't think like first,
5:50
there are no real benchmarks for like end to end dialogs.
5:54
Tao bench is 1 but it is so much more easier than the real world
5:58
situation. So for example.
6:01
Why is that? Just because you have so many
6:03
random sounds or you exactly different accents, that type of
6:07
thing. Yes.
6:08
So I think first, if we start with chat dialogues and
6:11
dialogues are often multi turn. So longer a dialogue, there are
6:15
the chances of failure increases, right?
6:18
And then for chats, the challenge is that how does the
6:22
agent recover, right? If you change your mind, you
6:25
change your intent, you said something that was unclear, I
6:29
misunderstood, right? The agent misunderstood.
6:32
And then you know, what are the self correcting abilities?
6:36
What are the abilities to ask right follow up question at the
6:38
right time? Because if it's not the right
6:41
follow up question at the right time, people have this
6:44
historical baggage of not trusting agents, right?
6:46
Like the the AI agents or chat agents, right?
6:49
So which means that since they don't always trust you have to
6:54
be very precise, like when you should ask a follow up question
6:57
to for clarification, right? When you think you misunderstood
7:01
them and like, you know, pivot. So these variations are present
7:06
in the text stack and now you add voice to that, right?
7:09
Voice is OK. Where are you sitting?
7:12
Like most people are not in such a studio setting, right?
7:14
Like wired often, you know, especially in like and, and,
7:19
and, and another important factor is that voice, the
7:23
distribution of how many, what percentage of population
7:26
interacts with support or any of these countries are just through
7:30
voice is vastly different between US and Europe and let's
7:34
say India, right? Like in India, probably 75% of
7:38
conversations, customer conversation happen on voice,
7:41
right? While that number could be
7:43
around 30 to 35% in the US, right?
7:45
So, so the difference is very stark.
7:47
And now when most people are calling from phone, they are
7:50
usually outside, there is background noise, there could be
7:53
multiple speakers present. How do you sort of filter that
8:00
people don't talk about enough about transcription people when
8:03
when voice agents are there? I think people want to hear like
8:07
the most beautiful voice from text to speech.
8:10
But in terms of reliability for specific domains like customer
8:14
support, you need very accurate transcription, right?
8:18
So for example, if you call me and I ask you for your order
8:23
number, your e-mail, or your name, and if I get any of that
8:27
wrong and I don't have mechanisms to correct that or my
8:31
transcription is not super accurate, then there's no way my
8:35
dialogue or the conversation will succeed.
8:38
It just fails. So now, like we discussed in
8:41
chat Agents, there are few degrees of freedoms where it
8:44
could fail. And that significantly increases
8:47
its voice because of let's say transcription errors, background
8:51
noise, multi speaker diarization, turn taking is a is
8:55
is still a open problem. Yeah, how do you, because I I
8:59
heard about with turn taking that you can't just say a number
9:05
of how long you wait or if X amount of time has gone by, you
9:11
know. Oh cool, now I can talk.
9:14
So there's like probability, yes.
9:17
Is it a whole separate model that you're using?
9:19
Yes. Tell me about.
9:20
That yes. So you can have a separate turn
9:23
taking model, you know, based on transformer architectures
9:26
fine-tuned for turn taking tasks where you collect data.
9:29
The data should have enough variations in terms of like
9:32
different accents but different like pace and intonation.
9:36
Let's say I speak very fast, but someone else could take longer
9:40
pauses, right? Yes.
9:42
So having all those variations across all accents and
9:47
languages, I don't think those rich data sets exist today in
9:50
voice. And then like training turn
9:53
taking based on that large part of turn taking has moved to
9:56
these neural models. But I do feel, and we can talk
9:59
more about that, that the the old school ways of extracting
10:03
voice features like, you know, the the the pitch, the RMS
10:07
energy, right, like from your voice and using those features
10:11
and using like simpler heuristics over them.
10:14
I think that is still helpful. Like what?
10:17
Yeah. So using these features to
10:19
determine when to stop, like you know, when the user has stopped,
10:22
Correct. So largely the turn taking
10:24
models are transformer based architecture, but I do see like
10:28
a hybrid as well, right? So because if you look at turn
10:32
taking models, most of them take speech and text both has input.
10:37
So imagine in a cascaded world, you speak something, then I
10:41
transcribe it, then I pass the speech as well as text.
10:44
It already adds a bit to the latency.
10:47
But let's say you know, in some cases you are your features of
10:54
voice are strong enough, right? Like where these acoustic
10:58
features are with a very high confidence, I can say that like,
11:01
or Demetrius has just stopped speaking.
11:03
Then I don't even need to like wait until the transcription is
11:07
over right to to to detect the turn.
11:10
So I think this also helps the hybrid approach also helps in
11:14
reducing latency and to end. But when you're running those
11:18
models, it's two separate models you're running, right?
11:20
Yes, yes. But I think the the the old
11:23
school extracting acoustic feature is fast enough compared
11:26
to the neural models. And you have something set up so
11:30
that the acoustic models are going to be the ones that always
11:34
override. So if that says, hey, we're
11:36
done. At a very high sort of
11:40
confidence threshold, right? Like otherwise you still sort of
11:43
use neural models for large number of use cases, right?
11:46
So I think that is one pattern to somewhat reduce latency,
11:50
right? Like through turn taking and
11:52
also have more confidence like using both these approaches.
11:55
But it's still an open problem. There are not enough good
11:58
benchmarks for turn taking open source, something I want to work
12:02
on once I have more time and then there are not enough good
12:05
open source models. I know Pipecat has done some
12:09
work with Smart Turn V2 model where I think they created this
12:12
interface to publicly collect conversation data and then use
12:18
that to fine tune our turn trainer turn taking model.
12:23
But it is often very hard to get data sets from like real
12:29
distribution, right? For because when I'm talking to
12:31
customer support, more often than not, you will not have
12:34
access to that data, right? So many of these conversations
12:39
on which this these models are trained are either synthetic or,
12:43
you know, by some of the data collection companies and they
12:46
probably pay users to do that. But like when when both of us
12:50
are talking and then I know what the purpose is, it could change
12:54
things, right? Like subconsciously, I feel that
12:56
is a very interesting area that like how do you sort of collect
13:01
this more natural data and use that to sort of tune your turn
13:07
taking also, it depends on domain a lot, right?
13:10
So the distribution of how people talk in customer support
13:14
versus talk the same person talking to a friend versus
13:17
talking to a customer support agent versus, you know, maybe
13:24
talking to even AI agent, right? Like the pitch and annotation
13:29
can change, right? So I feel like fine tuning that
13:34
for domain data is is very useful.
13:37
And I think that is still possible because when we are
13:39
working with some enterprise customer and if you say that,
13:42
OK, you know, we can have turn taking specifically for your
13:45
customer support. There's a certain pattern, let's
13:48
say most of your customers are in the US and there's a general
13:52
pattern to that sort of conversation, the noise levels
13:56
versus let's say someone sitting in like, you know, India or in
13:59
London, right? Like so fine tuning definitely
14:03
helps on specific data set. So I do believe like when we
14:06
don't have great benchmarks here, creating internal company
14:10
benchmarks on real data is a lot more powerful than, you know,
14:15
just benchmarking on open source because these data sets have the
14:18
challenges I was just talking about.
14:20
All right, so this episode is brought to you by the good folks
14:23
at ML Flow, the open source platform for developers who want
14:27
to build production ready AI applications.
14:31
Enhance your AI applications with end to end AI
14:34
observability, all in a single integrated platform with ML
14:38
Flows Gen. AI capabilities.
14:39
You can evaluate AI applications using a suite of built in or
14:43
custom judges, visualize trace executions and agentic
14:48
analytics, and continuously monitor evaluations, all while
14:53
tracking every run in one place. Ship better agents faster.
14:58
You know that's the name of the game.
15:00
Get started at mlflow.org. We had Zach in here from Sierra
15:06
a few months ago and he was talking about how they're
15:10
running a constellation of models.
15:14
You've seen that that's very common in your eyes.
15:16
Yes, yes. Can you break down how many
15:19
models and what what that constellation looks like?
15:22
Yeah, so that is true and we can talk about constellation of
15:26
models in different contexts. So one of the example was like
15:30
you know for for turn taking was like I said like you know one
15:35
could be a simpler acoustic features based model.
15:38
Second is neural right now you go to LLMS, right.
15:40
That is a big challenge because in the easiest thing in chat is
15:44
that somehow there is a, there is a higher tolerance for
15:49
latency where you are chatting with something and there's like
15:52
dot, dot, dot and thinking, right?
15:54
But in voice, that is, if that is complete silence, people
15:59
would just drop the call, right? And I think there are like more
16:01
historical reasons for that because voice agents or voice
16:06
chat bots traditionally were like so dumb that people
16:08
generally don't trust them. So now if you're putting like 2
16:12
seconds of pause, they'll be like, OK, I don't think I'll get
16:15
anything out of it and they'll disconnect, right?
16:18
So, but there's also a need from our customers to use the best
16:21
models, right? Like if Cloud Opus is there or
16:24
Sonnet is there, they still want to use that as an intelligence
16:27
layer. How do you do that?
16:29
And I feel like there's something in between the like
16:34
dumb cascaded way of just chaining things together and
16:38
like speech to speech, where you come up with like more
16:41
intelligent patterns of cascaded architecture or whatever you
16:44
call it. For example, let's say when when
16:49
two of us are talking and you ask me a hard question, right?
16:52
Which requires me to think more, right?
16:55
I would not go completely silent after you asked me, right?
16:59
I would often either ask you to give me a moment, right?
17:04
And let's say if it's taking more time, I'll be like, hey,
17:06
Dimitri, as I apologize, right? But give me 5 more seconds,
17:10
right? And that's better.
17:12
But even better is that you asked me something, let's say
17:15
about voice agents, right? And you asked a very open-ended
17:18
question. I had to go go and then do a
17:21
deep research to give you a comprehensive answer.
17:24
But can I keep the conversation going right?
17:27
Can I start with something very basic about voice agents, what
17:30
they are, where are they used, right?
17:32
And for that, do I need the most expensive model?
17:34
No, right. So I think what we often see
17:39
working with our customers, but also in general is like, I can
17:42
have a like a smaller model, right?
17:45
That is for these more cursory conversations, right?
17:48
A high level chit chat or are basically like turns that
17:53
require low intelligence right where these models keep you
17:57
engaged. And then they they do delegate
18:01
the the more intense tasks in background to A to a more
18:05
expensive model so that now there's no complete silence.
18:09
I have a model which is trying to be helpful and at some point
18:14
if it doesn't have enough context of what you're asking
18:16
about, then it can ask you to sort of wait and you know, still
18:21
keep the conversation going while the background tool or
18:24
background more expensive model returns.
18:26
So this is a very common model and now you can have like more
18:29
than two models in this, right? So I think this is 1 very common
18:33
use case of using multiple models.
18:36
Second is what I have seen is in customer support, specifically
18:42
that and at Amazon, we we went back and forth multiple times
18:49
where there was 2024 was very confusing where like you see
18:55
these great model releases from GPD or cloud.
18:58
Then you chat with them like they are just like amazing.
19:01
But then on on task oriented dialogue tasks like customer
19:04
support, where I don't really care about the overall general
19:09
intelligence of the model, but like I want higher reliability
19:12
on those tasks. There they would like often like
19:15
the prompt based approach or a react style looping method would
19:18
often fail. So then we would have like a
19:21
constellation of models again, right?
19:22
Like where for certain tasks, let's say tool calling cloud was
19:27
excellent even in like late 2024.
19:30
So I would probably rely on Haiku, right?
19:33
But like response generation is something where I want to have
19:36
more control because I'm building models for Amazon's
19:40
customer support, which is in across so many languages impacts
19:44
probably 2 billion plus interactions.
19:46
I cannot like let it loose on Haiku and just prompt the heck
19:49
out of it, right? So there I would use like a
19:52
smaller sort of fine-tuned model for response generation tool
19:56
calling. I will get the best from Haiku.
20:00
So I think that's another sort of reason to use a constellation
20:05
of models, yeah. I wonder how many is too many,
20:12
if there ever is too many. Yes.
20:15
And too many shouldn't be too many in my opinion because very
20:19
hard to sort of update them, improve them.
20:24
And so that was a challenge. And there also we went back and
20:30
forth even in early 2023 where we were like OK, we should given
20:35
that LLMS are here and we were already using a constellation
20:38
before pre LLM because generative capabilities were not
20:42
there. So for intent detection, you use
20:44
like you fine tune a bird style classifier for response
20:48
generation. You really don't use generative
20:50
models in production then, right?
20:52
We would have these like ranking models where you still use the
20:56
understanding ability of bird style models, but to keep the
20:59
dialogue contained and not not say anything like we define for
21:04
each domain, let's say refund from our historical transcripts,
21:07
we only see of 500 types of like responses that are possible,
21:12
right? Given that it's a constrained
21:14
domain and long tail, it's fine. We can transfer it to a human,
21:17
right? And we asked the model to
21:20
basically understand what the customer is saying and then like
21:23
pick the right template and fill that template with high
21:26
demetrios, right, right. And I can put the placeholders
21:29
and choose the template, right? And then we were using like a
21:33
bunch of models together and then we were like, OK, LLMS are
21:36
here, we should not, they are intelligent.
21:38
They should be able to do sort of multiple things together.
21:42
So what we did was our first iteration was fine tuning Flan
21:48
and Mistral on on our own data for multiple tasks, which was
21:53
action prediction. These days it's called tool
21:55
calling, but that back then response generation, intent
21:58
detection, dialogue state tracking, which is still very
22:03
common. So and we would have like
22:07
bespoke models for each of them and we tried to consolidate
22:10
them. And then you will see that like
22:12
there's an interference, like if one task sort of if the model
22:18
regresses on one task, right, then I add, let's say more data
22:21
to the post training mix for that.
22:23
And then it ends up impacting my other tasks as well.
22:27
Yeah, it's a vacuum mole. And then, you know, of course,
22:29
you have to still release stuff in production.
22:31
So like we went from a like constellation of models to a
22:37
multitask to a constellation, but like a smaller
22:40
constellation. And I do believe in many
22:44
production system, I haven't seen one model doing everything,
22:48
especially for like more complex situation.
22:51
If it's like a small to medium sized business, largely AQ and a
22:54
use case, couple of tool calls, right?
22:57
I think. I think just one model is just
22:59
fine. But then what you're seeing for
23:03
the various models in the constellation of models, what
23:08
are the different use cases of the different models?
23:10
You mentioned a deep research. I imagine there's OK, you kick
23:14
off some tool calling, maybe you want to go and grab some data
23:17
from somewhere. Yes, those are all pretty
23:20
obvious ones. Are there others that are
23:22
missing? It's just not deep research.
23:28
You can deep research is the most common use case like doing
23:32
that. But sometimes they are just APIs
23:38
that take longer invoice agents. If you see demos from different
23:42
companies, they'll be like, oh, you know, in 500 milliseconds,
23:46
we do everything right. But in real world, you are
23:48
deploying it in production. The APIs themselves take a
23:51
couple of seconds or sometimes one second, right?
23:53
And so it it is not a constellation of models, but
23:56
often you also use these models to sort of hide the latency of
24:01
those like more expensive tool calls or legacy APIs, right?
24:07
Yeah, yeah. Other constellation or other is
24:11
like retrieval, right. If you still use retrieval
24:14
today, for example, I still need to come up with a quick answer.
24:20
So I will, I will have like a fast retriever which is like,
24:24
you know, largely keyword based or grab based to give you a
24:27
quick answer and then you know, more expensive retrieval that
24:31
sort of does its job like more like deep research.
24:35
And so this is something where you would experience it as the
24:38
voice agent saying, OK, let me talk to you about that more.
24:43
Yes. And then boom by the time it
24:44
finishes that. Phrase yes, exactly or or give
24:47
you something more useful than that right.
24:49
So like, like, like if you ask me what are voice agents, I
24:52
think smaller models are still, if I just do like grab on my
24:56
documentation and search for voice agents, right?
25:00
And then I will still have a reasonable answer to give you,
25:03
right? So I, I just don't need to ask
25:05
you to wait. I can give you something And
25:08
then I'm like, hey, Demetrius, do you want me to go deeper?
25:10
But I was already going deeper in the background, right?
25:13
And then fetch the response for you.
25:15
So by the time you say yes, I get enough time to to to
25:19
basically like, you know, use the response from a deeper deep
25:23
research or a more expensive retrieval to to self.
25:27
And that's where you're masking the latency.
25:30
Yeah, that's like latency masking.
25:31
Yeah, yeah, yeah. Yeah, that's awesome.
25:34
OK, So what else? I know there are so many things.
25:37
That and then there are other examples of like having
25:40
constellation for example in support.
25:43
Most of my examples are going to be from support because.
25:46
You're knee deep in it these days.
25:47
So it makes sense, for example, like if you just do like the
25:52
cookie cutter way of building agents where you put a knowledge
25:55
base and some tools, right? At best it's going to match the
25:59
performance of a level 1 customer support, right?
26:02
Which is like usually not the best quality in a given
26:05
organization. Every organization has level 1
26:08
and level 2. Level 2 are probably more like
26:10
permanent employees, domain experts, right?
26:14
And no matter what company it is today, it's so hard to beat
26:18
their performance even today, right?
26:20
Yeah. So one of the ideas we were
26:22
like, OK, how do we make these agents smarter, get closer to
26:26
this level? Can we sort of learn from their
26:29
trajectories, right? Like how are they pivoting in a
26:32
conversation? How are they sort of taking
26:34
decisions on exceptions and not what are they saying to
26:37
customers that is more reassuring that the customer
26:40
doesn't disconnect, right? And how do we sort of fetch that
26:43
context and then also like, you know, give that as an example to
26:48
our agent to sort of follow that.
26:51
And so for that, we had like another model that basically
26:54
processes them because these conversations look very similar,
26:59
right? But just based on like one or
27:01
two values of like let's say age of the account or location of
27:08
the account, the policies could be very different.
27:10
So the model can make mistakes. So we had sort of another sub
27:14
agent to like fetch these conversations that are high
27:17
quality within similar context from these expert agents.
27:22
Give it to an LLM to see if there's something it has to
27:25
learn from them, right? Pass it to as a context to the
27:28
main LLM, right, or our main agent.
27:32
And that was another sort of multi agent setting where it was
27:37
helpful. So you're doing this
27:38
dynamically. It's not like the training, no.
27:41
Yeah, you, you and we did end up training that because if you the
27:46
conversation that you sort of retrieve like these trajectories
27:50
of conversation, the top five are so similar, right?
27:54
When you retrieve them that you would even we would not be able
27:58
to find which is the correct one and which is the according to
28:00
the policy and the SOP, right? These domains are very
28:03
compliance or SOP heavy, right? So you still need like some kind
28:08
of like fine tuning over that to make it understand the
28:12
difference between the correct and the wrong 1 based on some
28:15
minor details, let's say the, the, the loyalty status of the
28:20
customer or the tier of the customer or their credit usage
28:23
or whatnot, right? And for that, I think you need
28:28
to still sort of do some kind of a preference tuning to make the
28:32
model understand those like very specific differences.
28:35
Yeah, there's variables, yes, are only understood by the
28:38
subject matter expert. Exactly because like two of our
28:42
conversations could be very similar, but then in one of the
28:46
conversation, probably the location is different or or some
28:51
attribute is different and that makes all the difference, right?
28:53
Whether I should be giving you refund or not giving you a
28:56
refund, right? So if you don't have like
28:58
something that that learns to have this sort of discrimination
29:03
and out-of-the-box, I haven't seen anything.
29:04
Working, yeah, that discretion is so.
29:07
Important. Yeah.
29:08
And for some reason these level 2 agents have that discretion.
29:11
Uh huh. Right.
29:13
And I think it's still an open problem.
29:14
How do we sort of learn from their behavior and make the
29:19
agents like the current state of agents, which are at best level
29:22
one, get closer to that? It is also like context capture,
29:26
right? Like these days, agents are
29:28
largely working on internal knowledge bases and tools,
29:32
right? But the judgement of like these
29:36
experts, right? A lot of that information is in
29:39
their heads. But do you think they know why
29:42
there? Like do you think the experts
29:44
know why and they can explain if they were asked to?
29:49
Yes, and that is something I'm very passionate about that.
29:51
How do we capture that data, right?
29:53
Like I don't think they log it anywhere, but like with voice
29:57
becoming so much better, right? Like, you know, can you ask them
30:00
to like just talk about their reasoning of doing that and like
30:04
make them log at least some of them, right?
30:06
And then sort of close that gap through that information, but
30:10
also like observing their behavior and sort of emulating
30:14
that. I think that's a very important
30:16
area of applied research where how do we sort of get this
30:20
information that's there in their head or through their
30:23
experience, either observing their past trajectories or, like
30:28
you said, explicitly asking them.
30:30
But making it easier for them to be able to say that because
30:33
agent handle time is a big sort of constraint.
30:37
If I'm a human support agent, especially an expert one, the
30:41
amount of money I make would be would depending on how many
30:44
calls I take in a day, right? Which means if I'm spending more
30:48
time in like recording my rationale, then like it's.
30:52
So I think that the incentive structure has to change their or
30:56
it has to be a more natural way of like capturing that feedback.
30:59
Yeah, you almost have to give them as much time or money
31:04
compensation for doing this as if they were and.
31:08
There's also like a huge tension within the operations org in a
31:11
company and the org that is building AI agents to make those
31:15
operations more efficient. Because for operations org, if I
31:19
want to like use the expertise of these humans, it increases
31:24
their handle time, increases their cost, at least in the
31:27
short term. While without this information I
31:31
would not be able to make my agents better, so there is quite
31:35
a back and forth I've seen across different places.
31:37
Yeah, there's a little bit of a catch 20.
31:39
Yes. You know, what always fascinates
31:41
me with voice is how much more willing I am to give extra
31:47
context. So when I interact with someone
31:53
through voice, I will say and explain much more than if it's
32:00
with text. Yes, I feel that is like that is
32:02
the real unlock in my opinion. Then like at a very high level,
32:08
people saying like, oh, voice is the most natural way of
32:11
communication. Of course it is, but how does it
32:14
help it help? It manifests in exactly what you
32:16
said because it is a more natural way.
32:19
I can speak more, I can provide more context, right?
32:22
I can correct you more right. While my patience level when I'm
32:26
just chatting is like so much low.
32:27
I will just type minimum amount of words and that is going to be
32:32
ambiguous. The agent is going to get
32:34
confused and like, but in voice. As long as you don't like
32:37
frustrate the heck out of me, I'm happy to provide more
32:41
context to you. Yeah, yeah.
32:42
And just naturally I would say that I provide it without being
32:47
prompted extra because I want it's as I'm thinking through it,
32:53
I'm also talking through it. Yes, yes.
32:55
Normally when I will write with text, yeah, I'm, I'm thinking
32:59
through something, but I'll think so much faster.
33:03
And then I'll type out the summary.
33:05
Yes, exactly. And same for like voice other
33:08
applications, right. This is just sort of chat or or
33:11
real customer conversation. But the amount of time in let's
33:15
say customer support, how much time these associates spend in
33:19
writing an e-mail, right, That is a significant part of the
33:24
handle time. Sometimes as much as they they
33:27
spend like helping you out, they have to like do some compliance
33:31
stuff, add information, frame an e-mail, right?
33:35
And I think that part has also been like, you know, agents or
33:39
LLMS have helped quite a bit. And now with voice that becomes
33:42
so much more easier, right? Like if you have to frame the
33:45
e-mail you just talk, or if an LLM pre populates the e-mail
33:49
based on the summary of conversation.
33:51
If you need to correct you can just say it right.
33:54
Yeah, yeah. And so the thing that I've
33:57
noticed is that I am starting to get really lazy when it comes to
34:03
typing. And I think that's the wow
34:05
moment that a lot of people have with whisper flow is when you
34:09
can say, oh, no, I didn't mean to say that.
34:12
Or when it automatically formats with bullet points.
34:16
That's incredible because normally with the dictation,
34:22
it's really shitty. And when you say, oh, I didn't
34:24
mean to say that it will write it.
34:26
Will write that exactly and that is where I feel like this sort
34:29
of another area which I haven't personally sort of explored but
34:34
like which which can get important is like fusing the the
34:38
ASR and the LLM layer, right? Why most people don't like that
34:42
in production is because they want to have full control over
34:44
their LLM, right. But for applications like this,
34:47
right, where I don't want it to transcribe verbatim everything
34:50
I'm saying, it needs to be intelligent enough to have the
34:53
context that like, OK, whenever I was like, you know, back
34:57
channeling or sync or stop for a moment or set something
35:02
irrelevant in between, it shouldn't be transcribing that,
35:05
right? And so those models are getting,
35:08
you can literally like fine tune an open source model that can
35:13
take like multimodal input speech and text both to be able
35:17
to like basically fuse the 1st 2 layers, right?
35:20
So I know there are companies and some customers doing that to
35:25
manage latencies, but I think these personal use cases, right,
35:28
where you don't want transcription to be like
35:30
completely verbatim, you also need to understand the context
35:34
of the situation before you, let's say, send an East e-mail.
35:37
So you can basically bake the ASR and the intelligence layer
35:40
together. Yeah.
35:42
I hadn't realized that, but makes a lot of sense.
35:45
I think there's this wise AI startup called Ultravox.
35:49
They used to be called Fixi dot AI in CI.
35:51
Yeah. They follow this approach.
35:53
Really. Yes.
35:54
Yes. They're the first one from what
35:55
I know that hard because I remember back in the day they
35:58
were trying to do like a Ling chain competitor.
36:02
Yeah. No, I think now they are like
36:04
from what I know from my last couple of things change very
36:08
fast these days, but they're like a voice agents platform.
36:11
But they're like general thesis was that like, you know,
36:15
somewhere in between the speech to speech and like complete
36:18
cascaded is where we basically fuse 2 components, right.
36:22
So make it less cascaded, right. But again, right, there's a
36:25
spectrum, then you lose some control, right?
36:27
Yeah. Yeah, yeah.
36:29
What other use cases do you see a lot of with the voice agents?
36:34
Customer support, inbound sales, outbound sales, there are
36:39
companies that are building over us for outbound sales as well.
36:44
And inbound is much easier because see, if you look at the
36:47
sales cycle, right, like there are different types of
36:51
customers, like some are like strategic, like key accounts,
36:55
right? Where maybe you want to deploy
36:56
your best salesperson, right? But then there's a long tail,
37:00
right? And maybe eventually you might
37:03
still want a human there, at least with the current
37:05
capabilities, but the initial basic conversations, right?
37:08
Like if you see your LinkedIn and I see my LinkedIn like there
37:11
would be GTM folks from some startups reaching out, oh, we
37:15
are launching this product, are you interested, happy to set up
37:18
a call with our founder or whatever, right.
37:21
Like I think that initial call or initial e-mail, that is a
37:26
very common use case with with voice agents or text agents
37:30
today where the the first part of the outbound sales cycle is
37:34
like can be handled by that which is a long tail of
37:37
customers. Don't you think that would piss
37:39
people off though? If they get on a call and then
37:41
it's like, oh this is just a fucking voice agent, get out of.
37:45
Here and that is why I feel like inbound is more common.
37:48
Inbound is like when customer is reaching out to you that I want
37:51
to use your product and I want to do some kind of a lead
37:53
qualification that OK, is Imatrios the right customer?
37:58
Instead of having them fill out a form, yes, get them on a call
38:02
and you just. Right.
38:03
So that is a very clear one. Outbound, I know a few startups
38:07
which are specifically companies built for outbound sales agent,
38:10
they are building over us and they seem to be doing quite
38:13
well. So I'm pretty sure they are like
38:16
they have found a segment of customers who are just fine with
38:20
this early calls being with voice agents, especially with
38:23
how natural the voice has become.
38:25
And to be honest, like it was never about talking to, at least
38:30
for me talking to a human or to an AI.
38:33
I'm fine talking to an AI if I have enough trust that it's
38:37
going to solve my problem, right?
38:39
Just speaking to a startup founder where they are solving
38:44
this for plumbing, right? Where and I've seen that problem
38:47
myself and friends, you are looking for a plumbing
38:50
contractor today. You go on the website, you
38:53
submit your information, you ask a code and then a couple of days
38:56
later someone reviews that and gets back to you, asks for you
39:00
the best time to talk and. But I want that like thing to be
39:04
fixed today, right? Especially plumbing have some
39:07
serious problems. Exactly right.
39:09
So I don't think that works. So for plumbing, for example, it
39:13
is like so much easier now. And that's a great use case in
39:17
my opinion, where like if I can basically get a code
39:21
immediately, right? And it could basically find.
39:25
And I think plumbing also has this problem of like
39:28
seasonality. So if there is like a layer
39:31
above plumbing, which is like, because I many cases I don't,
39:35
I'm not fixated on a contractor. I want a contractor who is
39:39
charging me like reasonable and is available today, right?
39:42
So something that can basically make calls to contractors on my
39:46
behalf, submits that what I'm willing to pay for it and then
39:50
like get me something like tonight or tomorrow.
39:53
Right. It reminds me of there's these
39:56
folks who will run Google ads for like landscaping businesses,
40:02
but they don't have a landscaping business.
40:05
What they have is when somebody clicks on there and says I want
40:08
to get a quote. They then take that lead and
40:12
they'll sell it to an actual landscaping business.
40:14
Yes. And then booking and reservation
40:16
is like super sort of common use case if you are in a restaurant
40:20
and if I'm an authenticated user, if you get a call from me
40:24
to book a reservation, as long as you are sure that like I am
40:27
the person calling, do you care if I if it's an AI agent on my
40:31
behalf or it's me? No.
40:32
No. So booking from the customer
40:36
side, yes, yes, yeah. Booking any appointment?
40:39
Yes. Any appointment which means in
40:42
future I do imagine to have like once the whole authentication
40:47
layer, the payment part is is taken care of.
40:50
And I know companies like Visa and others are building a layer
40:54
for that for like payment authentication with agents where
40:58
I do have ability to authorize my agent to take XYZ actions on
41:02
my behalf. It could be shopping, it could
41:04
be making a reservation. And I think once, and I think it
41:08
will come like very soon and then I can have my
41:11
representation, right that can make these calls, book these
41:14
appointments. Because today what happens is
41:16
like I call especially in SF reservation, in restaurant, they
41:21
don't pick up phones. So I think this solves both
41:23
sides of the problem, right? The the user side, but also the
41:27
vendor side or the restaurant side, because there is one
41:30
person who is also serving but also taking calls.
41:33
They did not pick up the calls, right.
41:35
But if you have it's, it's very simple, right?
41:37
Like I know how many people are reserved for today, how many
41:41
sort of spots I have. And then, you know, I could, I
41:45
could basically book a reservation automatically or
41:49
worst case, if that information is not there yet, right?
41:52
I could still like ask the person to call you back, right,
41:56
so that I don't have to call back right when they are
41:58
available. So in both cases, I feel like
42:01
this significant like time we are spending and we are like so
42:05
unproductive doing this, that this should already be solved
42:08
now. Yeah, dude, this is great.
42:11
What else do you want to talk about?
42:13
What are some other things? I know you mentioned you rattled
42:15
off a ton at the beginning and I can't remember everything
42:18
because you said so much. I think the whole like cascaded
42:22
versus like speech to speech debate, right?
42:24
That is the most common question that comes from customers.
42:29
Also customers who are probably like more abstracted out or at
42:34
like leadership levels where they don't want to get into the
42:38
details, but they clearly feel that like, OK, when they talk to
42:41
like speech to speech APIs, it sounds cool.
42:44
Why do we need to have this like complex orchestration?
42:46
Why can't one model do everything right?
42:49
And yeah, so I was talking to in an informal setting with some
42:55
open AI researcher and I think they are still sort of working
43:01
on finding how do you sort of keep the speech quality good
43:05
because the model has to be smaller to to, to to have the
43:09
latency. Right.
43:10
Right. It has to be fast.
43:11
But then then we already know based on scaling laws right now,
43:15
right? A bigger model is better, right?
43:18
So I don't know, like I think that tension would always exist.
43:21
But more than that, even if these speech to speech models
43:25
keep getting better, right? There's always, even with the
43:28
latest, let's say, release of Opus, right?
43:32
It will not satisfy me 100% of the time if I'm enterprise,
43:36
which means like in some cases, I need to like have ability to
43:41
fix it, right? Sometimes it could just be
43:44
prompt or maybe adding like tools or maybe using a different
43:47
model, right? Like we talked about
43:48
constellation of model. So I think this whole like
43:51
cascaded architecture gives me that flexibility.
43:54
As long as we figure out, like we talked about that there are
43:58
different layers of users for voice agents today.
44:01
There are like non-technical business owners who want to
44:04
automate their like concierge experience, right?
44:07
Like taking bookings and stuff like that.
44:10
What is the interface for them? They want us to do all the heavy
44:13
lifting, no customization needed, right?
44:16
Then there's a layer below which is like customers who are small
44:21
enough but have few engineers, right?
44:23
And they want a little bit more of control right there.
44:27
It gets like, you know, we do offer some controls, but I think
44:30
it's still up for debate what is the right amount of control,
44:33
right? Because if it is like all the
44:35
control, then there are just too many knobs to change.
44:38
And then if you don't do it right, then your experience is
44:40
bad and then you can turn. It's witchcraft.
44:43
It's a witchcraft after that. Right.
44:45
So I feel like the number of knob, the the biggest challenge
44:48
of Cascade is not that you can't make it work like a real time
44:52
speech to speech in terms of voice quality, but with better
44:55
intelligence. It's more about the number of
44:58
knobs and the effort it takes to do that.
45:01
And that is why I feel like we see that more and more where
45:04
customers want to control the text stack and like offload the
45:08
voice orchestration to us. That is getting very common.
45:12
And I think there's a spectrum between pure dumb cascaded,
45:16
which is just chaining them and the speech to speech.
45:19
And there are different patterns like we spoke about, you can
45:22
fuse the ASR plus your LLM layer.
45:27
You can fuse the ASR plus turn taking layer, right?
45:32
They can be the same models. They don't need to be different,
45:35
right? Then you can have the TTS,
45:40
right. A couple of weeks back, we
45:42
released this expressive mode TTS models.
45:45
So, so far, TTS models were already so good, but like they
45:50
were not trained or fine-tuned for long conversations and long
45:54
conversations. I mean like 5 to 10 tons, which
45:57
is pretty standard in customer support or other areas.
46:01
So these models were just trained on, OK, this is the
46:04
text, right? With these emotional cues,
46:07
right? Like laughing or whisper
46:09
generate a speech based on this, right?
46:12
But like real world application of these models, if you look in
46:15
more enterprise setting, which is not on the creative side of
46:18
things is like I want to make sure that when I generate a
46:22
speech, it has context of my conversation so far, right?
46:27
Like all of a sudden it should not start laughing when the
46:31
conversation was going quite serious, right?
46:34
And so far in this pipeline, all that offloading was being done
46:39
on the LLM, right? Where LLM is responsible to
46:42
basically generate a sentence in a way that it is, it is it would
46:48
not laugh, right? Either use the right emotion
46:50
tags with that or or or keep the language that way that the TTS
46:54
model knows that this is not a happy moment, right.
46:57
But now with this conversational TTS is nothing but that that
47:02
just you train it in a different way where you pass it the final
47:06
utterance that it needs to do, but also pass historical context
47:10
of the conversation. And it's super important in in
47:14
domains like customer support, like booking sales, for example,
47:18
right? Like so with some of these
47:21
design in patterns where these TTS models getting more
47:24
conversational, like, you know, taking full context of the
47:27
conversation rather than that immediate turn ASR models and
47:33
LLM, you can sort of fuse them together having like this
47:39
foreground and background approach of model that you want
47:43
to use like expensive model, the best model, but also want to
47:47
keep it natural. So have this like small model
47:50
that is you can call it a masking model that is not as
47:54
dumb to just say that like say that give me a moment every
47:57
time. It's smarter than that, but less
47:59
smart than the big model. And so I feel like there are
48:02
enough design patterns in between, right, which could
48:06
basically make it sound as good as speech to speech, but with so
48:11
much more flexibility that when an enterprise customer says
48:14
that, OK, why I'm seeing hallucination at XYZ or, you
48:17
know, my, there's an outage on like, you know, Google Gemini
48:22
and I want to like, now change the model.
48:24
Now how do you do that with like speech to speech?
48:26
Like what if I'm using a speech to speech API from Gemini or
48:31
open AI and there's an outage? What do I do there?
48:33
You're screwed. I'm screwed, right?
48:36
So I feel like for, and that's my personal opinion, that like
48:40
for some time now, right? This, this somewhere in between
48:45
is the right approach, right? Not the old cascaded way, right?
48:49
Just chaining them. It doesn't work.
48:50
You will have so many edge cases that it would end up sounding as
48:55
dumb as the old chat bots, despite improvements in ASR and
48:58
TTS. So you can based on your
49:01
appetite, right? Like are you OK?
49:03
Do you need to, you have a small enough use case that you can use
49:07
open source model. You don't need to rely on, let's
49:10
say cloud. And in that case, can you fuse
49:15
the ASR and, and the LLM and, or maybe the foreground model, the
49:19
smaller model, you can fuse with the ASR, the bigger model can be
49:22
separate and that can be accessed as a tool called,
49:24
right? That is, that is one way of
49:26
doing that. Second, how do you sort of have
49:31
a more sort of conversational TTS?
49:35
You can potentially also combine the turn taking with this.
49:38
There are multiple permutations and I feel like the right
49:41
developer platform and we are also sort of improving on that
49:44
side would be at least for the core developer layer giving
49:48
these options right so that they don't making it easier to
49:51
implement and experiments with these design patterns.
49:54
I think I feel like most platforms are like had a long
49:59
way to go there. Yeah, right.
50:00
Yeah, I haven't heard that from anyone.
50:02
Yes, yes. So I feel because it it is non
50:06
trivial, right here we talked about these ideas.
50:09
It's easier said than done, right?
50:10
Because I also do like like implement them or like, you
50:14
know, in this like whole async event loop, like making that all
50:20
these events tie up together, right, exactly at the same time,
50:23
right. It is, it is non trivial.
50:25
So I think there's the dev platform layer need to provide
50:29
this flexibility. And then probably once people
50:33
experiment with these permutation, then you know, they
50:36
can choose, choose within the spectrum, not the cascaded and
50:39
the speech to speech. There are options in between
50:43
that give them enough trade-offs between, let's say, control
50:47
versus quality, reliability versus quality.