Latest / Agentic Conversations (formally mlops.community) / Voice and Language Tech // Catherin Breslin // Coffee Sessions #129

Transcript

Voice and Language Tech // Catherin Breslin // Coffee Sessions #129 · Agentic Conversations (formally mlops.community)

0:00
Hi, my name is Catherine, Reston.
0:02
I founded a company called, Kingfisher laughs consultancy,
0:05
which helps companies who are building voice and language
0:08
technology, and I love cup of coffee.
0:11
And I, my favorite is a black Americano.
0:15
We were talking with Catherine and we're talking all about like
0:19
speech recognition and machine learning with speech.
0:24
What are your top takeaways from this?
0:26
Yeah, for me, it's a quote. I don't know very well when I've
0:29
kind of Avoided. It's like almost a completely
0:32
different flavor of data science.
0:34
So it's interesting to hear like right from someone that's tired
0:39
like PhD level and run into work at some of the biggest
0:41
organizations in the world doing yet massive applications.
0:44
So that's quite cool. I think getting under some of
0:49
those suspicions about how these things are pieced together was
0:50
quite interesting. So like the fact that yes
0:53
something like an exercise is like lots of different models
0:57
back-to-back and you just are to Fathom the About Engineering in
1:00
place. That's really quite cool.
1:02
I think. Yeah, one thing that I liked
1:04
that she mentioned was how back in the day when she started on
1:09
all this, how it was very much its own thing.
1:13
And a lot of the tools you could not use, if you were doing stuff
1:18
in other areas of machine learning, it wouldn't cross over
1:21
to this type of machine learning.
1:24
But now with the advancements in just how, in her eyes, what it
1:28
seemed like is that we they were coming together very much so and
1:32
so you can use a lot of the same packages.
1:34
You can use a lot of the same tooling and that seemed really
1:38
nice because I always mm kind of dunking on how there's never
1:45
going to be this machine. Learning stack, like the modern
1:49
ml stack and that I think is because there are so many
1:55
different use cases that you have in so many things that
1:57
you're optimizing for so many different kinds.
2:00
SLA. Is that you're looking for all
2:01
that good stuff and with this it almost was like a little bit of
2:07
a pattern interrupt because she was talking about how different
2:10
it is from when she started till now on the tooling and it feels
2:17
like the tooling is starting to become more friendly to this
2:21
type of machine learning. So let me just give a quick bio
2:25
on who Catherine is and then we can jump into this full podcast.
2:29
With her. So Katherine is a
2:32
machine-learning scientist and she founded Kingfisher Labs.
2:36
She did her doctorate working on automatic speech recognition at
2:41
Cambridge and then she has extensive experience working in
2:47
automatic speech recognition, natural, language,
2:50
understanding, and human computer dialogue systems and
2:54
she worked at places like Toshiba research and then Also
3:01
Amazon where she was one of the teams that launched Alexa which
3:06
is awesome. Awesome stuff.
3:08
So we dig into that. You can bet that we are talking
3:13
a ton about what the experience at Amazon was like when they
3:18
launched Alexa and how that matured over time and we might
3:23
as well just get into it. I think last but not least if
3:26
you have not subscribed to our newsletter that is killing it
3:29
right now. Now, we've got some amazing news
3:31
letter writers. And so, if you want to stay
3:33
up-to-date on all of the cool stuff that's happening, in the
3:36
ml Ops ecosystem, the ml Ops Universe, check out our
3:40
newsletter, we've got three different ones, one a weekly one
3:43
that rounds up all of the stuff that we're doing as a community,
3:46
a monthly one that gives background on and a little bit
3:50
more thoughts, deep thought and shows the best stuff that has
3:55
come out in the broader ml, Ops ecosystem.
3:58
And then a third one, that is Just all about the best threads
4:03
that we've had in slack. So, jump on that, I'll leave a
4:07
link to the description below. And without further Ado, let's
4:11
talk with Catherine. All right.
4:18
So Catherine, welcome to the ml Ops Community podcast.
4:24
This is very exciting. I know that we were just talking
4:26
before I hit record about how you don't necessarily Sara Lee,
4:30
get to talk about all the cool ml up stuff you've done.
4:33
You mainly get asked about all of the research stuff today is
4:36
going to be totally different. We're going to do something that
4:40
is a little bit atypical, and before we get into that, I think
4:45
it's probably cool. Just to hear a little bit about
4:48
your background. Like, tell us where did you grow
4:50
up? hey, well, thanks very much for
4:54
inviting me on my background is I guess very Not, not not hugely
5:02
exciting. I have lived in the UK, or my
5:04
life. I know this is a very
5:07
international field. So a lot of people have very
5:09
exciting backgrounds. Mine is a lot planer.
5:12
So I've lived and worked in the UK since I was as young.
5:16
I grew up partly in the south of the country party up in
5:20
Yorkshire, don't have Yorkshire accent anymore, but I lost that
5:24
when I moved down south again, But I've always been interested
5:29
in math and computers physics, earn, and all of that.
5:31
Those subjects all the way through school.
5:33
I went to University to study engineering and I ended up
5:37
studying engineering and computer science as joint
5:39
honours, which proved to be a really great introduction and a
5:43
really great foundation for machine learning, which is, as
5:46
you know, so, a combination of sort of engineering and maths
5:49
and computer science. So, this is how I started on the
5:53
path to machine learning. That was quite a while back now.
5:58
So machine learning wasn't as popular as it was as it is
6:01
today, but I still thought was really interesting field and a
6:04
specifically interested in some of the things that, you know,
6:08
people can do, but computers couldn't, and I was curious
6:11
about how can you get computers to do some of these things.
6:13
Like, like, understand speech things that children can do with
6:16
ease, but we can get computers to do with any accuracy.
6:20
So, I decided to go from there and study study this subject
6:24
more. And I ended up doing a PhD.
6:26
In automatic speech recognition here at Cambridge, which is
6:29
where I still am. And so what was the idea of this
6:34
like, or how did it go from doing, this PhD to then starting
6:39
to get more and more in-depth into speech.
6:43
Technology. Did you realize right away that?
6:47
Wow, I can now start talking to computers or what did that
6:51
trajectory look like I think yeah, it might.
6:56
PhD was about trying to improve speech, recognition, technology
7:00
and how to make it better. So, I studied that for a few
7:02
years and then I just stayed in the field, it was really
7:07
interesting and it combines a lot of interesting aspects, a
7:11
lot of signal processing at the time.
7:13
A lot of maths, a lot of computers programming, all of
7:17
that stuff mixed together. So I think there's a really
7:19
interesting and challenging field and I could see that
7:22
although we weren't there at the time and we're a lot better.
7:26
Now, in terms of speech recognition performance, I could
7:28
see that this is technology that really could actually help
7:31
people if we get it to work and so I could see that it also as
7:35
well as combining these challenges of the subjects I
7:37
loved. It was something that had a real
7:39
practical applications that I could see see coming along.
7:43
So that was really what drove me into carrying on in the field.
7:46
And from there I've spent my career in different aspects of
7:50
soft voice and language technology.
7:54
So one thing that was interests me about this space is like it's
7:59
not one. I really used for my work as
8:00
much, but I imagine a lot of the Technologies and tools and
8:03
things that are used by very different to the kind of
8:06
standard, like, SQL database pipelines, like all that stuff.
8:11
So, yeah. Could you just elaborate a
8:13
little bit on kind of challenges around Technologies and toolings
8:16
that maybe some people aren't familiar with?
8:19
Yeah, I think that the tooling has evolved an awful lot over
8:22
the past decade or two that I've been in the field, there is in
8:27
the same way that a lot of machine learning has become more
8:29
accessible to people. There's been a lot more tools
8:32
and libraries available. I think the same is true in the
8:34
speech world, but you're right that a lot of the tools that I
8:40
started using when I was earlier in my career were sort of
8:43
hand-built and custom-made for the field because we were
8:47
dealing with large. G.i. files similar to how maybe
8:50
computer vision has the same challenges with with large video
8:53
and image files because large audio files had challenges and
8:57
storing them and accessing them, sharing them around a lot of
9:01
those. He's now because we have a lot
9:04
more tools for dealing with these large files and the ways
9:09
that we need to deal with them are in place now but these are
9:12
evolved over the time and I think there's been over the past
9:16
five years, perhaps more. More of a standardization across
9:19
more machine, learning Fields, where a lot of the same tools
9:22
and techniques are coming together now.
9:24
So, whereas computer vision and voice technology were disparate
9:28
Fields, 10 years ago, a lot of the, the tooling and that's
9:31
driven a lot by the fact that the models underneath the hood
9:34
of becoming similar, the sort of ways that they used to becoming
9:38
similar that they're being integrated together in new and
9:41
interesting ways. And so the the knowledge of
9:43
being shared a lot more between the different fields and there's
9:46
been a much more of a Alignment between the different machine
9:50
learning people's is that it's like a similar Trend reflected
9:55
in the skills that are. So because I imagine like he
9:58
can't go out and get a generic data scientist again because I
10:01
don't think many people have worked with voice data and and
10:03
audio processing things like that.
10:05
So how do you find the kind of? Is there a skills barrier?
10:08
Is it easy to pick up or jeju-do for a certain flavor of person
10:11
when you go out during for kind of these kind of DeSantis Yeah,
10:16
I think it's really important to have some sort of grounding in
10:21
Signal processing or language or engineering maths or something
10:26
like that, which is a little bit more in-depth than you might
10:29
find another other places you really speech, has a lot of
10:34
quirks and a lot of lessons that you learn through your career.
10:38
I guess about how speech works and some of the challenges and
10:42
the things involved in it and so just knowing Unsafe things as
10:46
you is good foundation to be able to work from.
10:52
So I think it is opening up in the way that other machine
10:57
learning fields are opening up because the tools are getting
10:59
more more advanced and more standardized and easier to use
11:02
and they're developing nicer ways for people to interact with
11:06
them. But fundamentally, I think you
11:07
do need somebody in order to build sort of speech recognition
11:11
systems. You do need some specialist
11:13
knowledge about how speech works.
11:15
And how, how to put the system together to get the best out of
11:18
it. Hmm.
11:20
And when you look at the trajectory of when you started
11:23
until now, and everything that has happened and the ease and
11:30
the developer experience really that that change that has
11:33
happened. And then you look at where we
11:35
are now, where are some developer experience hiccups
11:40
that you still feel like are there and you wish wouldn't be I
11:46
think that is a small number of toolkits around for doing
11:54
working with speech data. You've got the general ones like
11:58
tensorflow. I torture, you can build up
12:00
speech models in those. But there are a handful of sort
12:04
of custom speech recognition tool kits, some of which have
12:08
quite a long history so they are quite old.
12:12
And that means that, you know, people have built up a lot of
12:15
knowledge about how to use them and sometimes they're not as
12:18
well, documented, as they might be, or you need a real expert to
12:21
come and use them. So I think one of the challenges
12:24
that people have, I see companies who would like to work
12:27
on this technology and they find these toolkits and they really
12:30
struggle to work with them. So I think that that is one of
12:32
the hiccups. Although in general, the field
12:35
is moving towards a better developer experience.
12:37
I think there's still a little bit of a way to go there and
12:40
this is work. That is happening.
12:41
I think. Our take of these toolkits and
12:43
working on them to bring them, make them more accessible to
12:46
people. But still at the moment I feel
12:48
like they do need some specialist expertise to be able
12:51
to use them. That's one of the bigger bigger
12:55
hiccups I see. But then on the other hand there
12:59
is a lot of sort of practices and things that come from
13:02
outside of the voice field that we've brought into the into the
13:06
voice technology which I think has made it easier.
13:09
So there are, you know, a lot of Or purpose Frameworks for
13:14
building up models and running jobs on a distributed cluster.
13:17
Those sorts of Frameworks can be really just brought into the
13:19
speech field and adopted. So having these combined with
13:23
some of the tool kits that we use can lower the barrier.
13:27
So just why you talking about me thinks the say I'm a data
13:31
scientist and working for like an organization that's got the
13:35
budget was, maybe it's a perverse and the next big thing
13:38
that the we've decided the value lies in.
13:41
Yeah. Right speed Assessing what we
13:43
want we want to tackle that for our use case, maybe it's
13:45
localization, maybe it's a different dialect or something,
13:49
I don't know. But where do we start?
13:53
I what's your advice that in that person that goes right?
13:56
Is a whole new world. Yet, do I do first and where do
13:59
I go? So that there are a few levels
14:03
of software that you can use. So I was just talking right now
14:06
about the toolkits for building speech recognition which is like
14:09
getting right into the weeds into the details and building
14:11
your own. System.
14:12
But on the other end of the spectrum, there are Services out
14:16
there cloud services which do speech recognition for you.
14:19
Can just stream your audio up, get the results back.
14:21
Those are often the best place to start because they have great
14:24
systems. They're actually really good
14:26
accuracy for the general use case the easy-to-use a
14:29
relatively low cost to try out a few things.
14:32
So until you've actually sort of spend some time proving out your
14:36
products and being sure that it's the right thing for you.
14:39
These are great place to start. So the big Companies have these
14:43
are a few other companies as well, that that have their own
14:46
speech recognition services that you can use.
14:48
So that's where I would. Usually advise people to start
14:50
is use was there what's relatively cheap and easy and
14:54
then you can prove your product without having to worry about
14:57
the technology because building that voice technology yourself
15:00
in house is an expensive undertaking and not want to go
15:03
into if you're not really confident about the value is
15:06
going to bring Mmm, I love that prove out the use case before
15:10
you overcomplicate things. Exactly, So, you went on to work
15:18
at to Siva research, Toshiba research, if I can pronounce it
15:22
correctly, and can you give us a bit of the story around how that
15:27
happened? So she for research is also here
15:29
in Cambridge, which is where I started my PhD, they actually
15:33
help fund some of my PhD and they're already close links
15:36
between Toshiba and the engineering department here in
15:38
Cambridge. And so that was a relatively
15:41
easy, easy transition. I knew that they were looking
15:43
for people. And just as I was finishing my
15:45
PhD, which was lucky for me, because that was like, 2007,
15:49
when there was this Global Financial crash and a lot of
15:52
people stopped hiring. So, it was all of a sudden we
15:56
machine learning jobs. Weren't that comment at the
15:58
time, maybe even less common in 2007.
16:00
So, I was really lucky to be able to move into in stuff even
16:04
carry on working. There that is.
16:06
So then what I'm really interested in is some of the
16:10
challenges that you faced while you were working there and going
16:14
from that and working in a research lab and the transition
16:19
into working in the industry. E.
16:24
So I think there are some real differences between industry and
16:30
Academia. That takes some getting used to
16:32
if you move between them. But one thing I would say is
16:35
like there are some research Labs that are closer to Academia
16:39
and some which are further away from Academia Toshiba was a nice
16:42
place to start because it was much closer to Academia than
16:45
some of the other places. I've worked some of the places I
16:49
work later in my career have been even further away from
16:52
that. 12. But there are some real big
16:56
differences. I think when you move into
16:58
industry, so if you're in Academia and you're researching
17:02
a topic you often, your job is to figure out how to model
17:06
something better. How to come up with those
17:08
machine learning models. That do better on the, on the
17:10
task that you're looking for often, you have the data, you
17:14
have a data set, ready, prepared, or you might have to
17:17
go and collect a data set that you're building a sort of static
17:20
data set and you're trying to see how you can get.
17:22
The best performance on a particular task and that's what
17:25
a lot of academic research. Does that sort of proves out,
17:28
you know what, models we can use and how we can use them and you
17:32
take them into industry. The job is slightly different
17:35
because you don't necessarily have the data.
17:37
They're ready and cleaned up for you to use.
17:40
You have to figure out what's the best data source and how
17:41
you're going to collect it and what cleaning up?
17:44
It needs and things like that. You have to focus a little bit
17:48
on the building the models, which are going to get out into
17:51
production. How can you It's better for your
17:53
users. How can you look after the
17:55
models that you've put out into production?
17:57
Because you can't just put them out into production of get about
18:00
them, they have problems and bugs and, you know, go wrong
18:04
sometimes. So there's there's a different
18:06
mindset, I think. And thinking about how you can
18:08
take machine learning and apply it to a business problem as
18:12
opposed to an academic problem. I think that's the real
18:14
transition, which is tough to make in the first years, after
18:17
you leave Academia. Sorry.
18:20
You said, they're kind of want to get on.
18:22
The A and stuff but just that bit around this Boggs and like
18:26
taking care of models in production, what does that look
18:28
like for speech? They are like, like, how do you
18:31
unit test that kind of stuff? So you obviously you can unit
18:37
test the bits of your code if you know what the code is.
18:41
But one of the difficulties of a machine learning model in
18:44
production, is that you don't know how it's going to behave,
18:47
so it depends on the data that you've got.
18:50
So you can only really a Jersey testing models.
18:53
So you take a big data set and you see how well, what the error
18:57
rate is on your data set, before you push something out, and has
19:00
it improved from the previous iteration of the model.
19:04
So you have to look at adding in sort of accuracy testing, as
19:07
well as unit, testing and integration testing and all of
19:10
that other kind of lovely stuff as well.
19:14
Yeah. Now that makes sense it.
19:15
So yeah, hint of it there. Be one of the I suppose the most
19:19
commonly known speech recognition tools that people.
19:22
We used a today's Alexa, right? And so you were involved with
19:26
that. What was the and just linking
19:28
back to the difference between like industry and Academia was
19:32
that another step down the chain towards like production kind of
19:36
Powerhouse like getting models out the door was that still very
19:39
researchy? What was the culture like there
19:41
and how did that go? It was yeah another step away
19:44
from from sort of the very research-focused I joined Amazon
19:48
just before Alexa was launched and I was in a lucky Shouldn't
19:52
be able to see it launched and to see how it became mature.
19:57
And so that was quite a journey figuring out what we need to do
20:00
to take this machine learning products and really make it
20:03
robust and production worthy. Because this is something that
20:07
there are a handful of machine learning models in production in
20:09
the world. But at that time but not a lot.
20:13
And so we didn't really have best practices to lean on.
20:16
We didn't really know how to do a sort of inventing, some of the
20:18
stuff as we went along. So it was real.
20:22
A challenge to see how you could take this, this great idea.
20:26
This machine learned, it was a sort of a pipeline of different
20:29
machine. Learning models that strung
20:30
together to make Alexa and how to look after those and how to
20:33
improve them, how to debug them, how to maintain them, and how to
20:38
make them better over time, Japanese entertainment, like War
20:41
Stories of horror stories from that time, they're allowed to
20:43
share none that are allowed to share a.
20:49
Yeah. But there's some good ones that
20:50
you can tell us offline but the Thing that I'm wondering about
20:54
is when you were looking at this because the launch of Alexa and
20:59
I'm just remembering like the launch of Siri and how bad it
21:03
was when it was launched. And because we have such a high
21:07
bar, for what we expect from the machine to be able to do.
21:13
And when you use a computer specifically, like, let's just
21:17
take a calculator, for example, there's no way that you're going
21:21
to think. Like the answer that this
21:24
calculator is giving me could be wrong, or it might not have
21:27
understood what I typed in because it's a human error, if
21:32
it gives me the wrong answer, it's because I did something
21:34
wrong, not because of the computer and then all of a
21:37
sudden, you shift that to something like Alexa.
21:40
And it doesn't understand the way that I'm saying, something,
21:44
or it doesn't even hear me. When I say, hey Alexa, there's
21:46
so many things that can go wrong, right?
21:49
And so you saw and you were On the team having to change the
21:54
maturity of Alexa or just incrementally, make it better
21:59
and better and you talked about the fact that you needed to get
22:04
it more mature. What did that process look like
22:07
like, how did you evolve it? So, Alexa launched in 2014.
22:14
It was just a few years after ciri had launched and like, say
22:19
Alexa was different because it was a standalone device.
22:21
I had no screen either. So we had to be very thoughtful
22:26
about the answers that Siri were that I like sword gift, because
22:29
there was no screen to back up, what you were saying to give a
22:31
link to the web or to, you know, click here for additional
22:34
information or anything like that.
22:36
So people do tend to trust system that talks to them.
22:39
We're very true. Think of things that look human
22:42
or sound human. So there's those factors to take
22:45
into account when we were thinking about this, but when
22:50
Alexa launched, possibly don't remember that far back, but he
22:55
launched in the US and it had a handful of features but nowhere
22:59
near the feature set that I had. Now, so started with some very
23:03
headline features like playing music.
23:08
Asking the time setting timers, you know, various things like
23:11
that and new features were added over time.
23:14
It didn't even launch with shopping on voice shopping,
23:18
which is obviously, you'd imagine that Amazon being a
23:20
retailer first and foremost, that would be a prime use case
23:24
for them, but no shopping wasn't even there in the beginning.
23:28
So, it was taken from a system which had just a few features
23:33
and it was a Beta release with just a limited.
23:37
Release in the beginning to sort of try it out turned into a big
23:40
success. I think it was done, there was
23:43
nothing like it on the market. So it was unsure how it be
23:46
received and I think it's really changed the way that people
23:50
interact with with voice technology now and people have
23:53
it in their minds for the time, it was brand-new.
23:56
And so over time we took it from a system, which had a few
24:00
features added more features added more languages.
24:04
So it came to Britain shortly in Germany a year.
24:07
Laughter and that spread to other countries later and on to
24:11
other devices as well. So one of the things I worked on
24:13
was how do we get a lesser to work on the Fire TV which is
24:16
Amazon's TV plug in the plug into your TV and you can watch
24:20
streaming video through there. So we take an Alexa to new
24:23
devices as well over the series and we did that sort of slowly
24:26
and carefully always having a lot of testing in place, to make
24:30
sure that it was working the high bar on accuracy than.
24:34
So you had to be sure. If not two, Eating the current
24:37
performance of you don't want to make the performance of a music
24:42
playing worse by introducing shopping on that on the device.
24:46
So, this will be this balance between adding new features and
24:49
keeping the accuracy of the current ones.
24:52
That's was a slow and steady be careful.
24:54
And I can only imagine you you asked for Alexa to play your
24:59
favorite song and it orders you pair of pants instead.
25:03
Yeah. So those are things and I mean
25:07
We've had, we had Curtis North would I think is his last name
25:13
on here before and he is the Mind behind clean, clean lab,
25:19
and hit. He worked with Alexa and the
25:22
speech. And one of the things that his
25:25
tool was trying to predict was how many times people would ask.
25:30
Hey, Alexa. And it wouldn't respond, and he
25:36
was talking about how Difficult. The problem that is just because
25:39
you don't know like how are you supposed to know?
25:43
Because Alexa is not going to tell you oh I heard hey Alexa
25:46
and I didn't respond right exact so.
25:49
So I imagine there's all kinds of crazy challenges like that
25:52
and I think before we dive into some of the other challenges
25:55
that you faced around like these pipelines in these models, it
25:58
would probably be good for myself and Adam to.
26:01
I imagine to know about what exactly is behind a virtual.
26:07
Voice Assistant. Like what does that entail?
26:10
What does that even look like, I don't even know.
26:13
Yes, I there are several voice assistants in the world.
26:16
So Alexis one of them mentioned Siri the other companies have
26:20
their own versions and there are some small sort of custom
26:23
business, to business assistance out there.
26:26
But I think they will have a lot of similarities between them, so
26:29
you talking about something that you talk to you with your voice
26:32
and it you ask it a question or ask it to do something and it
26:35
will come back and Respond to you will do that thing.
26:38
And I'm short period of time. So we have a pipeline of
26:43
different machine learning models under the hood which
26:46
which make this happen. So the first thing in there is
26:48
the speech recognition, so that's going to listen to your
26:51
voice and transcribe the words, you said hopefully in a
26:55
relatively accurate way, so that it can pass it down stream but
26:59
this is obviously if it can make mistakes and some of those
27:02
mistakes might get past past Downstream and then we have to
27:06
do. Understanding the language.
27:08
So there are many different ways somebody can ask for something
27:11
so I could say play me some music or I could say I want to
27:15
listen to music two, very different ways where the
27:19
assistant should should play some music.
27:21
So we have to understand, you know, which of the actions does
27:24
to use a want. So we have to analyze the
27:28
language to classify it and decide which action should be
27:30
taken. We also have to pick out any
27:33
relevant keywords. So if I ask for, What's the
27:36
weather in Rich. I should get the weather and
27:38
Cambridge as opposed to if I ask, what's the weather in
27:41
London? So, we need to know there's
27:42
important keywords in that someone has asked about, as
27:45
well, so that we can return the right result.
27:50
Then sometimes, you know, the user doesn't give enough
27:52
information in the first time or you have to ask a question back
27:55
to clarify. So, there's some sort of a
27:56
dialogue component in there as well.
27:59
Often, some are more complicated than others dialogue is very
28:04
hard and so, like trying to As the kind of dialogue is always a
28:08
good idea at the moment but there's often a dialogue which
28:12
you know asks back you know which city did you want the
28:14
weather for or who do you want to listen to what sort of Music
28:18
would you like to listen to? And then some questions are more
28:23
complicated than others playing. Music might be quite easy, but
28:26
if I ask a factual question you know maybe the system has to go
28:29
out to Wikipedia and figure out what the answer is that can be
28:32
an entire machine learning and modelling itself sort of
28:35
question answering system. And then we have the final bit,
28:39
which is the voice back. So the synthetic speech, which
28:41
is going to speak the answer out to you, and synthetic speech is
28:44
another machine learning model. So you've got speech recognition
28:47
is or language, understanding dialogue question-answering and
28:50
synthetic speech, this pipeline, five different machine learning
28:53
models all chained together. Hey Lazlo here, if you're
28:58
serious about envelopes, you hit subscribe right now.
29:04
So I had a this is rubbish but I had a really years ago the quite
29:09
a large percentage like over 10% of queries that went to end up
29:14
with a bit of Mechanical Turk type back in where to get food
29:18
is that it depends what you mean by Mechanical.
29:23
Turk there are not all questions are answered by a machine
29:27
learning component. Yeah, sometimes you do just want
29:31
to be sure that, you know, the answer will be Ice and say there
29:34
are as in any I think most production systems evolve in
29:37
language. Do you have some canned answers?
29:41
I would be very surprised if a voice assistant didn't at least
29:44
have some sort of canned answers or if there are popular
29:47
questions that come up there, are people figuring out like,
29:50
how to answer them, because one of the things that happens in a,
29:54
in an assistant setting is, you know, events happen in the
29:58
world. And people ask about them and
29:59
you might not be prepared for them, you might not have the
30:01
answers. So, R can be, you know, world
30:05
events that happen or you know, a celebrity dies.
30:08
And suddenly you have to have that information available to
30:10
you and you can't update your machine learning models quick
30:13
enough. So most language systems, I know
30:16
our hybrid really even even Beyond virtual assistants
30:22
anything that involves sort of language, understanding,
30:24
typically is a hybrid system of some rules and some some machine
30:29
learning there. But no, no human.
30:32
In the background typing, the answer, That Alexis.
30:35
So, what is the things that voice assistant does?
30:39
Think voice assistants try? Not to very quickly and so, I
30:43
don't think a person could answer quick enough.
30:47
Really. So to see, if I remember, I was
30:52
told was that when an exit doesn't understand you, that
30:56
audio is sent directly to a call center, and then when you repeat
30:59
it, you're connected to that person that gives you the answer
31:02
kind of thing. That's, that's Like they
31:04
transcribe a river. That was the kind of Brewer.
31:05
And I thought that makes sense. Actually.
31:08
Yeah, I thought, but it couldn't be fast enough, right?
31:11
That's the thing. I think Kathryn you're saying,
31:13
is that but when it says, sorry, I didn't hear you.
31:15
That boys a little bit of time. So.
31:26
Okay. It's different, quite tight and
31:28
pivot engine. And you talk about other
31:29
devices. So they sit right.
31:31
Let's say it's Just did an extended family or over and like
31:37
what a cousin starts, the conversation about social media,
31:41
listening to everything you say on your phone and sending
31:45
adverts at you. What's your answer?
31:47
How do you get a chip on to that conversation as an expert?
31:52
So I do not think that social media websites are o, are
31:55
listening all the time? I think the processing power
31:58
involved in listening and analyzing everybody's
32:01
conversations. This is too much.
32:04
For very little value like the actual value, but there have
32:08
been studies. People have done of internet
32:11
traffic and in someone's house showing that actually this data
32:15
is not being sent around the internet.
32:18
I just don't see it being a valuable thing to do.
32:22
Yeah, I saw nothing that about models that could doing that
32:26
listening actually, which is much more powerful.
32:28
But yeah. Yeah.
32:29
Okay. Yeah.
32:31
So I'm wondering about this. This pipeline of models that
32:36
you're talking about and the speed that you're talking about,
32:41
how are you able to get it to go through all of that at the rate
32:48
that humans are able to have the patience for?
32:52
So usually I think the the speech recognition part is the
32:57
bit, which is the slowest and that takes Most of the time when
33:04
you're talking about interacting with the pipeline like this, so
33:08
what most people will do is they will stream the audio into their
33:12
speech recognizer as soon as a person starts talking.
33:15
And so when you are, as you were talking, the speech recognition
33:18
system is decoding what you were saying.
33:20
And then by the time, you have finished talking very shortly
33:23
after that, that the result can be available the transcription
33:27
and if you don't do that and see if to wait until someone's
33:29
finished talking and speech recognition systems, There's a
33:33
trade-off between run how long it takes and the accuracy so you
33:39
can make it run much faster than real time, but lose accuracy.
33:43
So you want to play a little bit as much time as you have.
33:46
So that's why you start streaming up the audio as soon
33:49
as person talk. So you've got a little bit about
33:52
real time factor of one to be able to do the speech
33:55
recognition. And then the rest of the steps
33:58
are much quicker. I think.
33:59
So you can You know, the language understanding can be
34:04
very quickly built this big models offline.
34:07
They can, they can run very quickly.
34:09
The potentially, you know, going looking things up and databases.
34:13
If you have to look for answers, can take time, sometimes the
34:17
similar technology runs in call center, sometimes where you are,
34:21
you know, maybe interacting with a big Mainframe and you caught
34:24
your bank and say, can I have my bank balance?
34:26
Let's go to go into a query to a database that's going to take a
34:29
few seconds. They will play what we call it
34:31
here. Constable sounds in your ears.
34:32
To tell you that like the audio equivalent of the spinning
34:36
cursor will play those in your ear to tell you that they're
34:39
still listening to you and that they're doing something.
34:42
And so there are tricks like that which you can use, most
34:45
voice assistants don't use days, they really try and respond
34:48
quickly but in other scenarios that the back end just isn't
34:52
responsive enough for that. So there's a lot of engineering,
34:57
a lot of trying to architect these systems to be as fast as
35:01
possible. And really like have the, the
35:06
trade-off between speed and accuracy at the right level so
35:10
that you get the best performance and the quickest
35:12
Tantra time response. So just injected to get your
35:16
fill then what are you kind of most excited about in the kind
35:20
of space at the moment then like what's on the Horizon?
35:23
Research or even industry things you've heard about what?
35:27
Yeah what kind of makes you think all that's going to be
35:29
quite cool and it lands. So I think there's A lot of so
35:34
we know how to build. I think a good speech
35:37
recognition system. If you have data for the speech
35:40
recognition system, you want to build you can you can usually
35:44
build a getting that data might be tricky.
35:46
But if you do, then you can wait, can you explain that a
35:50
little more like the data you need?
35:53
Just from people talking to the speech recognition system.
35:56
Yeah. Is that from like scouring news
36:00
or just watching YouTube for A long time.
36:03
Well you have to be obviously careful of the licenses of the
36:06
data that you're using. So only some data is available
36:10
for commercial. Use for you have to pay to
36:12
collect your own data, but we know how to take data of people
36:17
speaking and build a fairly good robust, speech recognition
36:21
system for a particular domain. So we might take people talking
36:25
in the course and build a legal transcription system or you
36:28
know, people talking on the trading floor and build a
36:30
financial speech recognition. We can we can do those which
36:34
work for specific scenarios. There are lots of languages in
36:38
the world where you don't have access to a lot of that data and
36:41
even collecting, it would be hard.
36:43
So there's a lot of work on what's called low-resource
36:46
speech, recognition low-resource, voice, technology,
36:48
where you're trying to build these systems for languages,
36:51
which just don't have the resources of English, or Spanish
36:54
or Mandarin, or some of the big languages.
36:57
And I think that's really interesting because people we
37:01
learn with far less. Data than a speech recognition
37:04
system. Really needs right now.
37:05
So there's there's some indications that we could train
37:08
these things with less data than we currently need.
37:11
So some of the biggest speech recognition systems are trained
37:14
with, you know, hundreds of thousands of hours of audio data
37:18
or even more than that. It's a lot of audio data that
37:21
that's more than a lifetime's worth.
37:24
And so you see if a small child can learn to hear in a few years
37:28
that's far fewer hours than we throwing out a speech
37:30
recognition systems right now. So I think Already interesting
37:35
field of research and I'm looking forward to see what that
37:37
guy's, you know, how can we build these systems with less
37:40
data? And then I think the next
37:43
another really interesting area which is becoming much more
37:48
feasible now. But now that the underlying
37:51
machine learning models across a lot of machine learning fields
37:54
are becoming more standard more similar to each other.
37:57
There is much more scope for building pipelines, like a voice
38:00
assistant, but, you know, other lines.
38:03
Another one would be speech to speech translation.
38:06
So you've got speech input machine translation and speech
38:09
output. So it's another kind of
38:11
pipeline, you can build, our you can imagine bringing computer
38:14
vision into these things as well, and more language
38:16
understanding. And there's a lot more
38:18
combinations of things we could build.
38:20
So I'm really looking forward to see what that goes in the future
38:22
as well. Oh yeah, that's cool to think
38:25
about and I know that my wife. So my wife is from The Basque
38:29
country and Basque is one of Languages that doesn't have a
38:34
lot of what you're talking about.
38:38
Like there's just not that many people that speak Basque in the
38:40
world. There are not that many books
38:44
that are written in Basque, etcetera, Etc, even though it is
38:46
one of the oldest languages and so so yeah, it's 6,000 languages
38:52
in the world I think, in many of them are dying out because they
38:56
have very few speakers and you need a community to keep them
38:59
going. And there are some where
39:01
governments of countries Countries have invested in
39:04
technology and in language learning to keep these languages
39:07
alive but others that they don't have that investment and they
39:10
are just disappearing which is a shame.
39:14
Yeah. And how, yeah, that makes
39:18
complete sense that the government's would want to
39:21
continue this because the other piece that you said about speech
39:25
to speech translation, or what was it speech to speech
39:29
translation? Yes.
39:30
Yeah. That is Something that I think
39:33
about a lot with my neighbor who is in her 90s and she only
39:37
speaks German and so whenever I talk to her, I get off my phone
39:41
and having the phone listen, and then I read what it says and
39:44
then I kind of like try and talk to her back.
39:47
So yeah, who knows where that will go though?
39:51
Yeah, exactly. This, so many people with this,
39:53
this could benefit, so many people who don't, even, even if
39:58
technology can help with all of this, there are there so many
40:00
languages in the world that I think it can really help.
40:02
He's communication between a lot of people in different ways just
40:07
interested, even like most about research in that.
40:10
But have you seen out in the wild?
40:13
Any like, weird and wonderful applications the like sort of
40:17
layman like me, might not be familiar with that.
40:19
We kind of bring think of anything like that weird ways to
40:22
do interesting ways to use. These kind of models to achieve
40:26
a result I think a lot of the, the way is to use, it wouldn't
40:32
come as a surprise to you necessarily so things like
40:35
speech to speech translation, you know, meeting transcription,
40:39
which is something I've worked on in the past, which is always
40:42
very popular. Can you transcribe a meeting and
40:45
find the notes from it? What I think is really
40:46
interesting is, you know, the way that people take these, you
40:49
know, somewhat generic pipelines and find specific niches to put
40:53
them in. So, for example, you might have
40:59
Legal Fields. There are a lot of meetings, a
41:01
lot of chats. It's very much driven by by
41:05
Spoken Word and so there's a real Niche there for voice
41:10
technology to do something. That might not be a you might
41:13
not think is useful because you're not familiar with the
41:15
legal world and it wouldn't be useful in another place where as
41:19
you know a voice assistant might be really great for you know you
41:22
know you know Factory that's a noisy Factory floor and people
41:26
have their hands busy doing operating machinery.
41:29
And so voice, interaction is really useful therefore
41:32
specifically about the. So I think taking these on
41:34
finding the niches that they work in really well, is more
41:37
interesting than the, the ways that you put them together. and
41:43
actually when you're saying that it is so true how there are
41:46
those note-taking apps and I've seen I'm just trying to remember
41:51
the name but one thing that I think about with what you're
41:54
talking about and then meeting transcriptions and specifically
41:58
when it comes to very very Niche things like when I try and
42:05
transcribe these podcasts, I always have a hard time because
42:08
we talk about things that the The understanding of the model,
42:13
it doesn't get there from just like, if we use a regular model
42:18
and so I effectively have to go and train that model or I have
42:23
to keep telling it. No, I said, ml Ops it means ml
42:27
Ops. That's how you should write it.
42:28
And and, but one thing that I'm wondering about in these, like
42:33
Niche situations, if you have a meeting that is for a niche
42:36
thing, like your software design, and then, You say, all
42:41
right we have these action items and then it automatically
42:45
transcribes, the meeting and then it gives you the action
42:47
items and it gives you a nice brief summary.
42:49
And it gives you something that is very tangible that you can
42:53
take away that you didn't have to have someone there as the
42:58
Scribe the whole time. Yeah, that's definitely
43:02
something that it's really interesting especially in some
43:07
of these fields like the legal world.
43:09
I think where they have a lot of meetings, there are other other
43:14
fields as well, I'm sure where they rely very heavily on, You
43:18
Know, spoken meetings for other than written communication.
43:21
But yeah, you have to be careful not to lose these things because
43:25
it's really easy to have a meeting and feel like you've
43:29
decided on things and then come to, you know, You later mom
43:32
realized that everybody sort of Forgotten what's going on.
43:35
So I think helping make people more efficient and log, some of
43:39
these action items definitely something which is very valuable
43:42
in some fields and then it automatically gives you a
43:44
reminder 5 days afterwards and ask you how you're doing on that
43:48
task. Yeah, be nice.
43:49
If it did the task for me, then, yeah, that even better, I like
43:53
that. So I'm if I type Learners out
43:59
there, people are going to get started in this space.
44:01
Is that what's like the Titanic dataset for speech?
44:06
Like what's the first toy problem?
44:08
People get started on. There are a few problems trying
44:17
to think if I can remember. So there's there's data set
44:21
called timet steai MIT, which is about phone.
44:26
Mm recognition. So I don't know if you know what
44:28
a phony mess but it's like a unit of sound.
44:30
So words are made up of phonemes.
44:33
So the word cat is at three phonemes know what bat but ah,
44:39
so one problem in speeches, you know, breaking words down into
44:43
phonemes and recognizing those. So that's a relatively old data
44:47
set though, but if you are very interested in just trying
44:51
something up quickly, having a look at that.
44:53
It might be might be a good place to look.
44:56
But people often look at these as switchboard, which is
45:03
conversation, telephone conversations between Pairs of
45:05
people. I think these people were given
45:08
topics to talk about. It's a very well-known data set,
45:12
and if you have seen a few years ago, there were some headlines.
45:17
I think, out of Microsoft about how speech recognition had
45:20
become superhuman and could do better than human performance,
45:24
and that was based on results. From this data set called
45:27
switchboard where the the system that Microsoft had was
45:32
performing on a par or perhaps slightly better than what we
45:36
expect. The human transcription accuracy
45:39
to be on that, that data set. So that's another, which is very
45:44
well known in the field. But speech recognition is
45:49
actually one of the fields. We just had a lot of shared data
45:54
sets over the years. So Even from, when I started
45:57
back in the early 2000s, there is, you know, share data sets,
46:02
which were distributed around the world, and people figured
46:06
out results on these compare results.
46:08
Across research, papers worked in collaboration with other
46:12
research institutes, to find the best systems on and those have
46:16
stuck around. And then there's a series of
46:18
them over the years of getting gradually harder and harder
46:21
starting from, you know, something like reading
46:25
newspapers. Each all the way through to, you
46:27
know, meeting conversations and even harder than that and then
46:30
our into other languages. So there's plenty of data that's
46:33
out there for speech recognition that that are available but
46:37
probably the one that is most famous as switchboard but it is
46:44
relatively smaller than some of the more recent data sets as
46:47
well just, which is a good place to get started on but but then
46:51
it gives you a little like the entry point again to other other
46:55
datasets So I want to switch gears because you went out, you
47:00
started Kingfisher Labs, what is the problem?
47:04
You're trying to tackle there. I did sigh Kingfisher Labs is a
47:09
couple of years old now I found it.
47:12
The is consultancy which I started because I have been
47:16
working with mostly startups now who are people who want to use
47:22
this technology and you know, need some expertise or some
47:25
advice. To get started.
47:27
So work with a variety of different clients who do
47:30
different things in the voice technology field.
47:34
I a lot of companies do get started with the off-the-shelf
47:38
solutions and then they want to save realize what their product
47:42
could look like. And they've got a little bit
47:44
further on their thinking, and they want to build the systems
47:46
in house and because they're not quite right that the
47:49
off-the-shelf ones, or, you know, they simply want to build
47:52
up their own IP inside. The company, they might have One
47:56
or two machine learning people working with them already and
47:59
some of that expertise and you know Real World Knowledge from
48:03
building these systems, they're interested in the technical side
48:07
of blood and the systems or else the management side, how to who,
48:11
to hire, how to interview them, how to organize, their team,
48:15
effectively, and that side of things as well.
48:17
And how to how to communicate between machine learning
48:20
scientists and the business teams, nice the translation
48:25
layer. Yeah, yes.
48:28
So so yeah. And work with the companies who
48:30
have, you know, been through that stage of looking through
48:32
the off-the-shelf solutions and figured out that that's not the
48:34
possible them in the end and want to take it further than
48:37
that. Excellent.
48:39
Are you hiring? I personally, am not hiring but
48:43
plenty of my clients are hiring machine learning scientists and
48:46
machine learning particularly ml Ops there is you know an awful
48:51
need for for this particular intersection of engineering and
48:54
machine learning right now. So I know people hiring hiring.
48:57
Thanks a lot. Awesome to hear.
49:00
So if anyone would like to get a job or try and find out about
49:05
that, can they reach out to you? I think you're in slack.
49:08
So, or is there a better way to connect with you?
49:13
So I'm on Twitter and Linkedin. Are we gonna find me that site
49:18
on LinkedIn? Catherine Breslin and on Twitter
49:20
in? My handle is Catherine.
49:22
Be UK? Excellent.
49:25
Yeah we'll leave the links to those in the description.
49:28
This has been awesome. Catherine, thank you so much for
49:31
enlightening us on the workings behind when I talk to my
49:37
inanimate objects in the house. Fantastic, I do hope you
49:40
realize. Now just why we need all these
49:42
envelopes practices and Engineers to make this.
49:46
It'll work because it's very complicated system.
49:49
There's a lot of complex engineering fighter.