Latest / Agentic Conversations (formally mlops.community) / MLOps at the Age of Generative AI // Barak Turovsky // #169

Transcript

MLOps at the Age of Generative AI // Barak Turovsky // #169 · Agentic Conversations (formally mlops.community)

0:00
I'm Barack Turovsky. I'm executive in residence at
0:02
Scale Venture Partner, a leading enterprise, BBC Cypher focused
0:08
on AI and I I'm not a big connoisseur of coffee, I just
0:16
prepare a cappuccino using a MSN.
0:23
Welcome back to the MLBS Community Podcast.
0:25
I am your host, Dmitry Osten. Today I'm flying solo.
0:29
I just got done talking with Barack and wow, what a amazing
0:34
person to chat with. He is just full of so much
0:37
wisdom and it is incredible because he is one of the
0:41
originators in this space. We could say he worked on the
0:45
Google team that created the translate that we all know and
0:48
love. I know specifically I used that
0:51
tool so much when I was learning.
0:53
Danish and got dropped in Spain and had to fend for myself.
0:58
So that was my best friend back in 2008 I think.
1:02
And he talked today to us about how he was using machine
1:07
learning and specifically what we could call today as a large
1:11
language model, but basically just transform architecture on
1:15
the translate. App that you get from Google and
1:21
how they created that and what he thought about it and what he
1:26
called the first boom of AI and what repercussions that had.
1:31
And then he just went into his now what he's calling the second
1:36
boom of AI. He has an incredible framework
1:39
that he goes through as to when you can be looking at using
1:43
large language models when you shouldn't be using them how he
1:46
views. The use of them right now and
1:49
what? He thinks is the best uses of
1:53
them. I mean, there was so much
1:55
incredible information here. It was just like.
1:59
Yeah, I loved it because he comes with a product mentality.
2:04
He was a product manager at Google when they created the
2:08
Tpus, which is a funny story that he shares in talking about
2:12
that whole experience about how they said, well, we just need to
2:15
create better hardware if we want to make this work.
2:17
And then they went out and did it and invested hundreds of
2:20
millions of dollars to make that actually happen without knowing
2:23
if it was going to work or not. That is wild to me.
2:26
And he was at the forefront of that.
2:28
He was the product manager on. Some of these incredible use
2:32
cases, so I loved that. But then we got into how he
2:36
thinks product and machine learning engineers can work
2:41
Better Together, and more specifically, what you need to
2:44
know in the ever changing paradigm of working with machine
2:48
learning today. And how you can set yourself up
2:52
for success when you are working with product departments or
2:56
product personas. I loved it.
2:58
I had a tonne of incredible takeaways and I really
3:01
appreciate Barack for coming on here.
3:04
So let's just get right into this conversation.
3:07
In case you haven't real quick, it would mean the world to me if
3:12
you just share this episode with one friend.
3:15
We're starting to increase our reach a tonne and it seems like
3:19
people are sharing. So I'm going to keep asking.
3:22
I'm going to keep asking you to share this with one friend.
3:25
Because it is awesome just hearing how you all are enjoying
3:31
these episodes and liking them. I am going to put a little
3:34
disclaimer because I know we have been talking about LMS
3:37
quite a bit lately. This episode is no different,
3:40
but we are trying to find that happy medium.
3:42
It's not going to jump only to LLM's all the time.
3:46
We are still talking about traditional ML OPS use cases.
3:50
I guess we can call it traditional even though it's
3:52
only like 2 years old. You know what I mean?
3:54
And so. Now share it with a friend.
3:58
If you liked it, mean the world to me.
4:00
Let's jump into this great to have you on here.
4:11
I'm super excited because there are so many people that I think
4:16
follow you and enjoy your content on LinkedIn and on
4:20
social media, and you're one of those people that actually has.
4:24
Been in the trenches. You've done a lot of great stuff
4:27
in the field of machine learning and you were just talking before
4:30
we hit record about how your expertise really lies in
4:33
product. But I think you're a little
4:35
humble and you've got a tonne of expertise and you've also been
4:40
around for a while, so I'm excited to dive into this
4:43
conversation with you. Where I really want to start is
4:45
just from the beginning. Where did you realise that you
4:49
wanted to get into this whole tech thing?
4:52
Yes ohh on tech. Well it's it's a very long story
4:56
I first of all the mattress, thank you so much for inviting
4:58
me. I'm excited to be here on this
5:00
podcast and obviously it's very important topic to talk about
5:03
not only how AI will transform many industries but I also
5:07
believe it will transform envelopes and DevOps industry
5:10
tool where people will need to gain new skills and transition
5:15
to new skills. And I'm very excited because I
5:18
believe we are in a second coming over I and I'm very feel
5:23
very lucky and humbled that I was part was in the central part
5:29
of both of them. The first part of the eye was in
5:31
2015, 2016 when Google did the breakthrough of basically being
5:38
able both on hardware and software side to be able to use
5:41
what we call digital networks, what we now call large language
5:44
models. So AI at scale because it is all
5:47
that was purely theoretical or let's call it academical, you
5:49
couldn't use it on a very large datasets.
5:52
And Google specific Google Brain team that they work very closely
5:55
with basically proved that it's possible both on the software
5:59
and on Google developed a custom hardware called Tensor
6:02
Processing Units to do that. And the first very first product
6:06
that was using that scale was Google translates the product
6:10
that I LED. So I mean I will extremely
6:12
humble and proud of being part of that amazing journey.
6:16
As a side note, if people want to read a little bit about the
6:19
history, both technical history and overall history of the I,
6:22
there is still a great article from New York Times called The
6:25
Great AI Awakenings appeared comment everyone to read.
6:28
It's a really great article. Maybe we can even send the link
6:32
to people. Yeah we'll put it in the shadows
6:35
a part of the podcast. So that's basically the first
6:38
coming of AI. But I would say this forthcoming
6:41
in 2016 it really exploded and that created all this
6:44
revolution. But to some extent this
6:46
revolution was limited to big tech companies like Google, like
6:50
Meta, like Microsoft and others Amazon and all the stuff around
6:56
like stand and speech. And Google did a lot of things
6:59
about query understanding and ads etc.
7:01
And I did a lot of this work across understanding speech,
7:04
understanding speech, understanding search queries,
7:07
matching queries to ads, etc. All this was amazing work,
7:12
amazing achievements. But there were limited the use
7:15
case for limited. Now with this to some extent I
7:18
think even open a I was surprised when they put this
7:21
clever UI on top of those large language models and they were
7:25
obviously because there are out of that are much less bound by
7:29
constraints that Google or other big companies have.
7:32
But now the interest and applications of last language
7:35
models are going to be democratised.
7:39
They're going far beyond, I would actually say, and we'll
7:42
talk about it, they will transform a lot of industries.
7:45
Some industry will be transformed faster, some slower,
7:48
some use cases will be transformed faster, some slower.
7:51
But eventually what will happen is that I believe that we will,
7:55
the will truly go into AI being the new electricity, They'll
8:00
transform almost every aspect of our life.
8:02
So as I said, I've worked in this space for for more than 10
8:09
years. In the end was part of both
8:11
Waves or both Cummings. So very excited about that.
8:14
If you talk about my tank background, I started in in the
8:19
military and Israeli military many years ago doing what it's
8:22
called the I. But now at that time nobody
8:25
called it this way and worked on the range of companies across
8:28
startups and big companies across advertising and commerce
8:33
and payments. But last 10 plus years I mostly
8:36
focused on AI. Yeah.
8:39
And so you were at Google when they released that, what you
8:43
were calling the first wave, right, of AI and you saw how it
8:48
affected Google. Can you explain that whole
8:51
movement and what? What happened?
8:54
Like where your eyes opened as soon as Translate came out and
8:58
what was the internal buzz? There were people asking what
9:02
else can we add this to and what did that look like?
9:04
Yeah, it's a great question. So it's a it's a fascinating
9:07
story. So the girl networks were a
9:11
talk. By the way, AI exists for many
9:13
years. I mean some people going back to
9:15
Turing test and as I say, it's a great Great Awakening article.
9:19
It's a great article. I highly recommend people to
9:22
read. But on a high level a lot of
9:24
academics including academics that were at Google came to
9:27
Google Translate and said hey, why you're not using gift neural
9:32
networks And we ask them what's your data set?
9:34
They're like, oh we have amazing results like what's your data
9:36
set 10,000 sentences. You're like well our datasets is
9:40
like on an older system let's call it machine learning.
9:43
One of those statistical machine translation was like in good
9:46
languages like Portuguese or would be like single digit
9:49
billions. So like can you system run it?
9:52
No. And the results were not great
9:54
if you use your networks on 10,000 sentences.
9:57
So it's not doesn't help. Then at some point Jeff Dean who
10:01
is kind of I think considered the father of modern AI, he
10:05
started to look at that and got interested in that and he came
10:07
to our team and said I think it's possible and engineering my
10:11
engineering. At that time she was my boss and
10:13
then he became my peers. He basically was sceptical
10:15
mostly because of the production aspects of it.
10:18
He said it would be too slow and Jeff Dean basically asking what
10:21
would it take and he said new hardware and Jeff said you'll
10:27
make it happen and that I mean it's unbelievable.
10:30
It's actually gives a lot of credit to Google.
10:32
Google invested upfront without no clear youth case yet on
10:37
monetization even though it was obvious that Google will
10:39
monetize it. Google invested $130 million
10:42
upfront. It's a public information to
10:45
develop what's now called custom hardware and the reason for that
10:49
when we started looking at the results.
10:51
But in 2015 there also software breakthrough with Google Brain
10:55
that they develop a system that could process all those billions
10:59
of training sets at scale. But it was extremely slow.
11:02
Just to give you a sense of how slow it was, it was 100 X slower
11:06
in production versus our production systems.
11:08
So let's say if translation would take, I don't know end to
11:12
end one second, it will take 102nd, It was unusable system.
11:15
That's why Google had to play both sides of the model and also
11:19
with the hardware part and effectively developing custom
11:23
hardware that initially Google Translate was the first
11:25
customer. But and it was unbelievable.
11:29
The excitement, to your point about excitement, excited that
11:32
decides that so many people were so excited that we
11:35
conservatively estimated it will take us three years to launch
11:39
the first language. We launched I believe 20
11:43
languages. We finished 20 languages and
11:45
nine months. So we did it in 1/4 of a time.
11:49
A because obviously Google is amazing company is amazing
11:51
engineer. But also the excitement about
11:53
this literally transformative moment was unbelievable.
11:57
It just another kind of funny stories.
11:59
This article, the great Awakening, because we all
12:02
understood that we're working on something so transformative for
12:05
the industry, not only for Google, we actually allow the
12:08
reporter to sit under embargo. He could not publish anything
12:11
for six months. He would literally sit in the
12:14
productization meetings that I would lead when we would talk
12:17
about how we produce every tonne of details, how you do it and
12:20
how you launch it in a product that had billions of users.
12:23
So it was super exciting project and as I said, we had to
12:27
overcome latency issues and quality issues and hallucination
12:31
is all the stuff that we are talking today that still exists,
12:35
but we had to overcome them and it was amazing, right?
12:39
Wow. So that is incredible.
12:40
Can you tell me a little bit more about the experience of
12:43
building a product at that scale?
12:45
A and the challenges that you went over but also.
12:49
These challenges are very topical for today what we're
12:52
talking about. So how did you overcome things
12:55
like hallucinations? Yes.
12:57
So it's a, by the way, first of all, you'll see, you know,
13:00
literally hit the nail on the head.
13:02
I think what we experienced right now is this height.
13:05
It's also, as I said, it's very positive height because now the
13:08
Elms or this kind of AI is democratised, but people mostly
13:12
toying and playing with it. And there is a big difference
13:15
between toying and playing and doing something cool and
13:19
launching it at scale to billions and billions and
13:23
billions of users and making sure that the product is not
13:27
only useful but also you know, not offensive etcetera,
13:30
etcetera. So there was a lot of work
13:33
including building supporting ML systems to build these
13:37
combinations. Again conversations happen.
13:39
It will continue to happen in in AI.
13:44
And by the way, it's also back to the use cases and I will talk
13:47
about my framework. In some use cases hallucination
13:50
are very important to deal with. For example use case of service
13:55
that I will talk a little bit. It's a use case that I believe
13:58
you need high, very high accuracy and very few
14:01
hallucinations. And also we need to make sure
14:03
you reduce defensiveness etc. Especially if you're talking
14:06
about the company like Google or Facebook etc.
14:09
The other use cases and translation one of them.
14:12
Or let me give you an easier use case.
14:14
You're writing a draught of an e-mail and use the user will
14:17
probably review it before you send it in those particular
14:20
cases, even if you have collision, nation is the
14:23
division of work between human and machine.
14:26
Is that machine? Is this console what I call
14:28
fluency, meaning you know, good writing, good acquaint writing?
14:33
Because that's in many cases takes a lot of time from people
14:37
and then it might be fine and maybe that's actually a good
14:40
time to talk a little bit about my framework on that front.
14:43
Because I think that's kind of my framework was actually
14:47
developed as part of my work in the Google Translate and also
14:50
all the discussions especially in recent months, whereas all
14:53
those massive listeria that all search will be disrupted
14:57
tomorrow and markers will take over Google search with open AI.
15:00
So a lot of those things it was very quickly the industry proven
15:05
that that's not the case but and I was very enjoying see that I
15:09
was right but maybe I will, I can also expand why kind of look
15:13
at the the, the framework to kind of explain that a little
15:17
bit. Yeah, I love the framework.
15:19
I think you were what you showed in the article that you wrote.
15:24
It matches up so much with a lot of this stuff that we've been
15:27
talking about and how basically how severe the downside is is
15:33
really what you want to look at if you're trying to use AI,
15:36
because you can't allow for any kind of hallucinations if the
15:40
downside is huge. But if you're just as you said
15:44
in the article, you said, well, if you're just creating a poem
15:48
or some music lyrics. If it housemates, that might
15:51
actually make for a better poem. Who knows?
15:53
Yeah, it might be a feature. Yeah, exactly.
15:57
Exactly. So yeah, maybe get into the
16:00
framework a little bit more. That would be helpful.
16:02
OK. So as I say, the framework was
16:04
developed a based on a lot of like 10 years of experience
16:07
working on a lot of use cases including search and others, but
16:09
also all those massive stadiums that a lot of especially
16:12
investors would call me and ask me, hey, is Google that And you
16:16
know, you search completely disrupted because of open AI.
16:19
And basically I try to explain to people and people say, oh,
16:23
that's very helpful. So on the file level, I believe
16:25
one of the biggest misconceptions on the people
16:28
look at what the language models can do.
16:30
They're very excited about what they call fluency and fluency
16:33
just means in milingo, fluency means how suddenly people
16:37
discover that machines could talk like human in many cases
16:40
very eloquent and very confident.
16:42
Human meaning like if you interact with, you know open the
16:46
eye with PT or with BAR with new goal generator, I you see how
16:51
eloquent and how consonant and how polished as the answers are
16:56
and that's great. It's a huge technological, but
16:59
it's important to understand that it does not mean that very
17:03
confident talking means that you're highly accurate.
17:06
You put bits, a completely different dimension, you could
17:09
be very confident and very eloquent and very polished and
17:12
very inaccurate. And by the way, investors
17:15
usually know those people very well.
17:17
It's a very common behaviour in some people, which is basically
17:24
it's like human trait. You are very constantly in and
17:29
yeah, constantly and very Polish manner talk about any topic in
17:34
the world. If you ask a question you don't
17:36
know an answer, you just make shit up on the spot.
17:39
And and because you're so confident, because your delivery
17:42
is so smooth and so amazing, people tend to get carried away,
17:47
especially in topics they don't understand like AI.
17:50
And that's a very dangerous outcome regardless of the
17:52
intent. It could be con artists that
17:54
want to defraud you of your money and you get carried away.
17:57
We have tonnes of examples including recent ones like FTX.
18:01
Or it could be a persons that is not a con artist but he really
18:05
believes in his own thing. It doesn't mean it's true.
18:08
It could be completely wrong. He just believes in it.
18:11
Again, maybe FTX could be that example.
18:13
I mean, you know, send back that trade talks about his innocent.
18:17
He just believes that he's right, right.
18:19
So you never know. But at the end of the day, the
18:21
outcome is pretty bad regardless.
18:23
So I think what's really important when you look and
18:26
approach and applicability of large language models is to look
18:28
at the plots, the use cases from those two by two grid, maybe
18:33
even three by two by three grid and basically put on one end you
18:38
put high fluency versus low fluency.
18:40
The other way you put higher pressure lock low accuracy and
18:43
you say OK, so what's most important for this use case, and
18:48
I'm specifically using an extreme examples, when you write
18:53
a greeting or write children books or composing music or
18:56
writing a poem or writing a science fiction, what you really
18:59
really need is a good story. In some cases you might not even
19:04
need accuracy, it's actually might be a feature that you
19:08
don't need. I made-up story anyway.
19:10
But at the very least there is no right or wrong answer, right?
19:13
So accuracy is not that critical.
19:15
When you look at the use cases more into a search site, for
19:19
example, you're asking a query that will use as a supporting
19:23
data for important personal business decision.
19:26
Meaning it will include involving money or time or both.
19:32
What you really need the end of today's high accuracy.
19:34
Yes. And some use cases like I want
19:37
to buy a dishwasher or I want to book an hotel in Paris for my
19:42
next vacation. I need some explanation.
19:45
I need a story. But at the end of the day if a
19:47
good story comes from the expense of kayak accuracy,
19:49
that's not going to help. You really, really, really need
19:52
high accuracy here. And the last thing that
19:56
represented by colours and my framework is basically what I
19:59
call low stakes versus high stakes, meaning what is?
20:03
What is the risk of? What are the consequences of
20:06
making a mistake? So let's look at maybe a bit
20:08
more practical examples. For example, writing a business
20:13
memo e-mail that is kind of yellowish in this case.
20:16
Meaning what I mean by yellow is like it's not that like it's
20:19
it's. I don't care that the e-mail
20:21
will be inaccurate, I do care. But the good thing is that me as
20:25
a user, unless I'm very dumb, I will probably not send an e-mail
20:28
that was generated by large language model just like that.
20:31
I will double check for accuracy and what I will do is that that
20:35
will be division of work between machine and human.
20:38
That is very reasonable human machine health.
20:40
We write a good story because writing a good eloquent story
20:43
takes a lot of time and effort for most of people.
20:46
People are usually very good stock, they're struggling to
20:49
make it a good story. The same about creating
20:51
business, presentation, etc. So I will take.
20:54
Pause right there for one second because I want to explain to
20:57
those that are just listening. We have this XY coordinates and
21:03
on the left side you have low accuracy.
21:05
And then on the right side you have high accuracy and then on
21:09
the bottom you have low fluency and on the top you have high
21:12
fluency and basically everything in the left upper.
21:17
Quadrant are green. Use cases.
21:21
And that means that you don't need super high accuracy.
21:25
You do need higher fluency, though.
21:27
And these are things that Barack was just talking about, like
21:31
writing a poem or writing science fiction, where story is
21:34
important there. But as you start to go over and
21:37
you need more accuracy, that's where things become more yellow,
21:42
and the further to the right you get, so the higher accuracy that
21:46
you need. That's when things get read.
21:49
And so you were just about to go into some of the other?
21:51
Cases. Feel free to jump into some of
21:53
these red use cases where you do need high accuracy like the
21:57
travel recommendation bookings or that because I would guess
22:01
also the difference between yellow and red because then
22:03
yellow you might actually need high accuracy too.
22:06
But the good thing is that user could double check for accuracy
22:10
because the use case even at scale, meaning even if you need
22:12
to write millions and billions of business presentation or
22:16
business memo emails, you still will Division of work will be
22:21
right me a good story idle double check for accuracy and
22:25
even if you need to adjust some of the fact, it could provide a
22:30
productivity boost of up to 70% which is huge because as I said,
22:34
eloquent writing takes time and effort for a lot of people.
22:38
Now on the red side, that's where it becomes very tricky.
22:42
Any search use case, it's physically impossible to put a
22:45
person behind every search query and validate it.
22:50
That's not possible. And if you ask users to validate
22:54
every search query, then it's not that different than what we
22:58
are doing in search today. And therefore if we go to the
23:02
next slide, when I basically create those two circles in blue
23:05
and red, basically I clearly if I grossly oversimplify, I
23:10
basically say that use cases that are created or workplace
23:14
productivity. Use cases Improving productivity
23:17
of creators, meaning writing, composing music, writing a poem,
23:22
writing children's books or improving what I call walk white
23:26
collar productivity, writing business memo, e-mail, creating
23:30
a business presentation, creating a marketing asset,
23:33
creating stock images, etc. Those use cases are much better
23:37
fit for large language models. Because of those use cases, you
23:41
either have what's really important in the story and you
23:44
can basically users can validate the facts while the use case
23:49
that I call them information. Slash decision support use cases
23:52
that are roughly search use cases.
23:54
Like give me an answer your machine.
23:57
Give me an answer which hotel I should book in Paris.
24:00
Give an answer what appliances I should buy?
24:02
What insurance should I buy? Or give me an answer for a very
24:05
important supporting data for very important business
24:07
decision. Those use cases in my opinion
24:10
will take much longer. Those are longer term use cases
24:12
for alms. I would even argue that people
24:15
will not be considered stable and ready to adopt the use cases
24:20
of search using yellow lamps. At least right now people will
24:24
focus more on those creative workplace productivity use case.
24:29
So that's a high level my framework.
24:33
You also have this idea of. It's not only.
24:37
Hey. Let's have a travel
24:41
recommendation or a hotel that we're going to recommend that
24:44
you go to it. Right now a lot of people are
24:47
talking about Ohh. Yeah well you'll get the
24:49
recommendation and then you'll be able to book it right there.
24:53
But what happens when you don't really trust the booking agent
24:57
either And maybe you book it, but it books for two children or
25:00
books for two adults and really you wanted you have it's just
25:04
you going. That kind of thing is also very
25:06
scary and it's hard to. Debug those pipelines.
25:11
Apps that look at Apple don't say lessons that I would even
25:13
not talk about taking actions. And my great example is what
25:17
happened with voice assistants and making those purchases and
25:21
voice assistants like Alexa because, you know, we were in
25:25
those scare of new paradigm shifts that will disrupt search.
25:29
Yeah, like in 2015, 2016 when Alexa launched Voice Assistant
25:34
and said, hey, you now can buy, you know, your toilet paper and
25:38
your toothpaste or whatever using Alexa.
25:42
And there was no technological challenge with doing that
25:44
because Amazon, they have 150 million Prime members at least
25:48
in the US where they know they can literally predict when
25:51
you'll buy next to space because you order it for them.
25:55
And there is no technological issue to understand that asking
25:58
for a toothpaste and they know which one.
26:00
But guess what? And Google.
26:01
Google got cared about it and invested a lot in Google
26:05
Assistant and Amazon invested and Google invested and
26:08
Microsoft invested in Cortana and Facebook invested and Apple
26:11
invested. Everybody invested in all those
26:13
devices. And there is hundreds of
26:14
millions of devices. But those devices people don't
26:17
use them for what they call high stake use cases of let me buy
26:21
something using voice, which is very close to alerts, right.
26:24
People use it for setting an alarm.
26:26
People use it to play music, people use it for loss taking
26:29
these cases. So my feeling is that forget
26:31
about just like can I buy it because an Amazon, you could buy
26:34
it very easily and it will be even shipped to you.
26:37
I think the paradigm shift of trusting a machine for something
26:41
that will wolves time or money or both people are not there
26:45
yet. And some of it is technological
26:47
challenge, but some of it like in Amazon and ecommerce, it's
26:51
not a technological challenge, it's a user adoption challenge.
26:54
So I think those areas will be adopted much slower.
26:57
If you ask for prediction, people will go broadly.
27:00
Let me try use machines or LLM for creative use case.
27:05
Let help me writing a review, help me write a note, thank you
27:09
note to the teacher, help me write a memo, help me write a
27:14
presentation. And when you start comfortable,
27:16
more comfortable, comfortable, that could take three to five
27:18
years, then I will start saying, OK, now I trust your machine.
27:22
Be my personal assistant. Give me an answer what what
27:25
restaurant I should book. Give me an answer, what
27:27
insurance should I buy, etcetera, etcetera.
27:29
That's my feeling, how it will go in terms of consumer
27:32
adoption. And there's obviously
27:33
technological challenges of hallucination, where you're
27:36
talking about high stakes, highly accurate use cases.
27:40
Yeah, my favourite thing to do with Siri literally is saying
27:45
where are you? And when I lose, my phone is
27:48
like, hey, where are you? And that's basically what it is.
27:52
And like you said, setting alarms or maybe playing music,
27:55
but not much more than that. And I'm not sure what you're
27:59
saying is, yeah, in a few years we'll have that adoption because
28:03
people will feel more comfortable using these large
28:05
language models. I'm kind of swinging the other
28:08
way that the more I use them, the less confident.
28:11
I feel in doing more with them. And so I I wonder if other
28:17
people are feeling that way too. But, and that's I'm just
28:19
curious, that's the matters. Is it for use cases that are
28:22
more creative use case or those are more use cases of
28:25
information seeking? More creative use case even.
28:28
Creative. OK, OK.
28:29
Interesting. Yeah.
28:30
Like hey, generate something for me or.
28:33
Help me summarise this. Help me transcribe this,
28:38
whatever it may be. I I guess I hold them to a very
28:42
high bar and I maybe right. By the way, I would consider
28:45
both you and I not very representative users at scale
28:49
because like we are still very advanced users, right.
28:52
We are early adopters of everything.
28:54
So yes, I mean if you ask me starting to the paradigm of
29:00
search will probably take five to seven years to get to sizable
29:04
scale. I think other topics like
29:06
creative and working with you will be faster, but again, it's
29:10
still will take time, yeah. The good folks AT1DB Weights and
29:16
Biases have decided to sponsor this podcast and they wanted to
29:20
mention that Andrew Yang and the Deep Learning AI team have
29:24
taught and inspired thousands of our fellow machine learning
29:28
engineers. This is why they're absolutely
29:30
thrilled to announce Weights and Biases partnership with Andrew
29:34
and the Deep Learning AI team on a new course.
29:38
All about training, evaluating and debugging generative AI
29:43
models. Super important topics that we
29:46
like to talk about here a tonne We're living through a real
29:49
landmark moment in generative AI and this is a great opportunity
29:54
to learn about how you can either join or hone your skills
29:58
that you already have. Evaluating and Debugging
30:01
Generative AI is an interactive course where they'll walk you
30:05
through training and sampling of different diffusion.
30:09
Models, evaluating and fine tuning looms and how you can use
30:14
weights and biases to make it a lot easier all along the way.
30:18
It's free to enrol and you can check it out now.
30:22
We'll leave the link in the description, or you can just go
30:25
to Weights and Biases, Slash Gen AI ML OPS course and sign up
30:32
there. If you want to chat with the
30:35
course creators and instructors, they're hosting an hour long
30:38
workshop for you. To go through all the questions
30:41
that you may have on April 15th, that is way out of date for
30:48
August 15th. Once again, if you want to check
30:50
out the course, which I highly encourage you to do, and we'll
30:55
leave that link in the description, but it is one.
30:57
DB dot me slash genai Dash ML OPS Dash course.
31:06
So go ahead and check it out and let's get back into the pod.
31:10
So there is something that you you kind of talked about there.
31:13
On one hand, everyone needs to remember that infamous blog post
31:18
that is you are not Google. But on the other hand, I would
31:21
love to hear what you as a product owner, what you think
31:26
about when you are looking at these new applications that are
31:30
coming out. And as you mentioned, there's a
31:32
big difference between making a Twitter demo like an auto GPT.
31:36
It blows up and gets a tonne of stars.
31:38
But then when people actually use it and try and use it in
31:41
production, it's like dude, this is not working.
31:45
And that can go for any kind of app right now or any use of
31:52
these MLM's and how we can mitigate some of these big
31:56
questions and trade-offs that we have, one being costs, that's
32:01
huge, right? And then the other one that you
32:02
mentioned is latency. And so if you are trying to do
32:06
this at scale. And really have a battle
32:09
hardened bulletproof LM OPS. Pipeline, what are you looking
32:15
at and how are you thinking of that?
32:17
And especially also just to add another question on to the
32:21
infinitely long question, how do you see other companies doing
32:26
this? Well, first of all I think it
32:29
goes back to my framework is that people should be realistic
32:33
what can be achieved and start with your skills that I would
32:35
call them easier than harder because cases, because if you're
32:38
trying to shoot for the stars and say I'll replace Google
32:41
search right, that's very difficult.
32:44
LM might take 10 years to get there because there is always
32:47
you know the Pareto principle that in productizing something
32:50
you sometimes spend 80 to 90% of their work to to optimise for 5%
32:57
of the use cases. So for example reducing
32:59
hallucinations you might it might take literally 10 years to
33:03
get from Nike percent accuracy from 90% to 95 and that might
33:07
not be enough because in search of 1999% accuracy.
33:10
On the flip side, if you need to create, if you start with
33:13
creating use cases, you know, writing a movie script, you
33:16
might be totally fine with 80% accuracy, right?
33:18
And 20% hallucination is less of an issue.
33:22
So I think first of all you need to be realistic what use cases
33:25
you should focus on, right? That's the first thing.
33:27
And then there are still tonnes of topics that you mentioned.
33:30
One of them, back to the use cases, is that latencies and
33:34
cost believe. Guess what, Even on those you
33:36
should focus on either use cases because many use cases latency
33:40
is less of an issue. It is a huge issue in search
33:43
because people expect instant answers, so you need very high
33:47
accuracy means probably means larger models, bigger costs and
33:50
you need very low latency meaning and very very high
33:54
freshness meaning. In search, you need super fresh
33:57
results. Otherwise there's a tonne of
33:59
search queries that are influenced by by fresh results.
34:03
And guess what? Fresh results have much more
34:05
abilities to either being hallucinated or being influenced
34:09
by both or whatever. If tomorrow, I don't know, Trump
34:11
was indicted or Trump announced he will running for president,
34:15
you can have tonne of polls that will completely pollute your
34:17
results, right? And it's not even hallucination,
34:20
it's for real something, something not not fully track.
34:27
If you go to use cases of I don't know, please create a
34:30
draught e-mail for me a you probably can wait, maybe maybe
34:34
even an hour you can wait. So latency is not as rigid as I
34:39
said, accuracy results are not that critical etc.
34:44
So the first and foremost I think classical role focused on
34:48
users that are achievable and then even on those use cases are
34:52
work because you need to make sure also even the point of
34:58
open-ended and Lulu based input. It sounds very cool.
35:02
Let me allow people to say whatever they want.
35:05
But what I discovered that my many years, for example working
35:09
on speech, that our language is extremely rich and it's awesome
35:12
but it also extremely open-ended and ambiguous.
35:16
And I always give people this example of, you know if you were
35:20
a witness of a crime scene, you will you call to a police
35:23
station. They like describe the describe
35:26
the suspect like, well, I I cannot describe.
35:28
I cannot draw like don't worry. Let me bring you a police sketch
35:31
artist. He will help you roles as a
35:34
person and then you suddenly discover he's asking you these
35:37
questions that is, you know, 15 types of eyebrows and 15 types
35:41
of years and 15 types of eyes and and forty types of noses.
35:45
And then it becomes very complex.
35:47
So in many cases when you give people a lot of choice to
35:50
express themselves, it's awesome.
35:52
But it also complicates the product solution.
35:55
In many cases you actually need to provide people easy to use,
35:58
either toggles or you know, UI choices.
36:02
You know, like shorter versus longer story, visual versus
36:07
textual story, casserole versus professionals.
36:12
So a lot of the things can also be simplified.
36:14
There are still tonnes of things to be figured out there.
36:18
But make big advice to people and I think we can also talk
36:20
about the industries where I believe that will be disrupted.
36:23
Focus on use cases that are lens are good fit today versus use
36:29
cases. Ellms could be a good fit
36:31
tomorrow because there's potentially a very long road to
36:34
make it happen. I love that, and there is one
36:37
thing that you said in your article that I want to harp on
36:39
for a moment, and that is around the idea of having.
36:44
The user feedback and really where you're going to create a
36:47
mode. If you are creating some kind of
36:49
tool or a use case, maybe it's people within their companies
36:53
and the whole company is giving them pressure that they need to
36:56
start using these LMS. And they have to make sure that they're
37:00
powering up all their coworkers with some large language model
37:04
capabilities or whatever it may be.
37:06
But you were talking about how the real delta is going to be
37:09
created when you can get the human feedback when it is built.
37:14
Into the workflow of the application.
37:18
So that may be if we're going back to this.
37:21
Text creation example. That may be that every once in a
37:25
while you just have to create some kind of text or you have to
37:30
change the text. It's underlined like Grammarly
37:33
for example. Maybe they say, look, this isn't
37:36
correct, but we're not going to give you the answer that we
37:38
think is correct. We're going to ask you just to
37:41
change it. And then because you change it,
37:43
now Grammarly has a more robust data repository that they can
37:48
use and call on and train their models with better.
37:51
Yeah by the way there is simple way to look at that.
37:53
You could also suggest a change and you learn from adjustment
37:56
that people do. There are multiple ways to do
37:58
it, but definitely feedback is very important angle here.
38:02
Yeah, yeah, exactly. And so that is kind of the idea
38:05
that you're making is that the feedback is going to be the
38:10
crucial aspect as we move forward.
38:12
So whatever you are creating, try to make sure that you can
38:16
have that feedback loop inside of whatever it is you are
38:20
creating. Yep, absolutely.
38:23
O That's super cool. Now there is a a few other
38:26
things that I wanted to go down the road of which is.
38:31
One being. These different?
38:35
Industries that you feel like are going to be affected the
38:38
most. I get the feeling that we are
38:41
both very aligned on the idea of it's not going to be self
38:45
driving cars. That's not where we're going to
38:46
see LM's make their biggest impact.
38:49
And for that case, it's none of those bubbles that were in red
38:54
that are basically as you go along the high stakes access the
39:00
higher stakes, the less LM's are going to become useful in the
39:05
short term, correct? And you you think that later on
39:09
give it whatever. 5-10 years it will eventually get there.
39:14
You have no doubt my feeling. Again it it was highly depends
39:17
back to your point how I think it will be a progress.
39:20
People will start trusting machines more and more on this
39:23
creative or workplace productivity skills and if they
39:26
build this a trust or a poor or whatever you call it, then I
39:31
think you might be able to go there, to go there and go to
39:37
those use cases. And remember, there's also there
39:40
is also the aspect of technical aspects that needs to be done.
39:44
Yeah, that's such a great point. So the I guess the the really
39:49
interesting thing that I'm wondering about are what are the
39:51
interest industries in your mind that are going to be?
39:55
Disrupted the most in the next few years or that you are seeing
40:01
being disrupted right now. I mean, I think we've all seen
40:04
that copywriters are getting a huge level up.
40:06
It's not necessarily that they're being completely
40:09
replaced, but they are being able to use a lot more tools and
40:14
they have a greater advantage now and they're able to get more
40:17
done. Same with coding.
40:19
If you are using Copilot, that's an awesome example, but what are
40:24
some other I mean I guess. Law taxes.
40:27
I know I've I've seen a few really great use cases of a
40:31
friend of ours, friend of the pod Hyannis who came on here and
40:35
he was talking about how he works at this company called
40:37
Digits and they are using large language models to analyse
40:42
people's taxes and how they do the taxes and suggest to a human
40:48
who is a tax professional. Hey, maybe you can do this to
40:52
save some money on your taxes and then that tax professional
40:56
will say yes or no on it. Yep.
40:59
And you touched on three out of four that I wanted to talk, but
41:03
I will kind of price. I'm the first one I think is by
41:06
far the first one in the short term is entertainment.
41:09
It's already happening and that's not surprising because
41:11
entertainment is for good or for bad, always are running after
41:15
the shiny new things or shiny new technology.
41:18
I mean again, you probably know that ecommerce and Internet and
41:22
credit cards at first. As the first adopt, early
41:25
adopters were supporting industry, but that's kind of a
41:29
part of the entertainment industry And on on the bad side
41:33
they always were forced to sue like Napster example.
41:36
And we see exactly the same thing.
41:38
You'll see deep fakes and different artists trying all
41:43
those things because it's also easy to try and put on the sleep
41:47
tight. You see, studios are basically
41:49
suing the platforms or asking them to remove like give.
41:54
They called Drake and all other things that is already happening
41:58
and it's fascinating topic, so I will probably publish an article
42:01
about it. It's fascinating optic to look
42:04
at not only because of entertainment as an interesting
42:07
angle, but also because it's in many cases shows the way where
42:11
things could evolve because they're always the first to
42:15
adopt new cool things that are very you know top of mind for
42:18
people and also you know try to stop it.
42:21
So it's very interesting. The second one is a big it's not
42:24
really an industry. Some what I would call a
42:28
collection of use cases and I call them use cases that are.
42:32
Customer interactions, use cases, sales, marketing,
42:35
customer service, etc. Those include almost every
42:40
vertical. So you mentioned legal.
42:42
It includes legals. Includes, you know, it includes
42:45
transportation like airlines. That includes utilities like.
42:51
You know cable companies et cetera.
42:53
It includes healthcares, includes insurance, everything,
42:55
right. I believe it will impact every
42:58
company. I would actually say more than
43:00
that that I believe that the disruption of customer facing
43:06
interactions will impact almost every industry across the board,
43:09
financial services, banks, etc. And the reason for that is that
43:14
I believe that within three to five years any company that has
43:17
sizeable, let's say more than 1,000,000 customer base.
43:21
Would need to make their entire knowledge bases like their
43:24
internal knowledge bases. For example if you call American
43:27
Airlines and ask why am I eligible for refund if my cycles
43:32
cancel or you're calling you know you Internet provider
43:36
asking my Internet not working. Basically what customer service
43:40
are doing today is they're basically understanding your
43:43
intent and trying and creating an internal knowledge base
43:46
trying to give you a cookie cutter response.
43:49
I believe that if you make those knowledge bases accessible and.
43:52
That was a lot of language models across all channels.
43:54
Emails, documents, chats, even calls.
43:57
For example. You're probably familiar with
43:59
Google Duplex technology that Google used to book appointments
44:04
that people could not differentiate whether humans
44:06
speaking. Companies that will not embrace
44:09
this new paradigm will face the risk of being disrupted by
44:13
competitors, either existing or new.
44:15
That will deliver Not only. Fraction of a cost, it could be
44:20
50, much lower cost, it could be 40 to 50% cost structure.
44:24
But I believe it would be much better customer experience
44:27
because you will leverage generative AI language models.
44:30
Because think about it, if today you call customer service and
44:34
saying why my Internet isn't working and it's relatively
44:37
poorly trained in many cases offshore customer service guy
44:40
who understood you in 10 more or less query the database or
44:44
knowledge base and give you a cookie cutter response that
44:46
could be too technical for average.
44:49
Emerson's large language models could easily you can easily ask
44:52
him. Gives this knowledge, summarises
44:55
knowledge based on knowledge bases, Summarise it, makes sense
44:57
of it and rephrase it. For a non-technical person it's
45:02
also very good. CHPT 4 is very good on
45:04
understanding human emotions to some extent, that person is
45:07
upset or scared etc and find a way to calm down.
45:12
Basically used phrase that will come down a person etc.
45:15
I believe we'll see much better customer experience if you use
45:19
large language models and lower costs.
45:21
So I believe that will be. As far as the biggest
45:24
disruption, I now have this kind of a slogan that all those
45:28
companies that have sizeable customer base need to focus on
45:33
Chegg Between your competition or your chat, your competition
45:37
will change GPU. And finally, you mentioned
45:40
according its encoding will be disrupted and it's probably very
45:43
relevant to your audience. And finally, I also believe that
45:47
education is being disrupted as we speak.
45:51
As you probably should you be before especially is amazing and
45:54
passing standardised test, it immediately impact public
45:58
companies that focus on that like check or person and I think
46:02
we just in the beginning of that.
46:04
But I would say back to my point that customer, customer have
46:08
interactions, it's almost every industry that will be big focus
46:12
for 11. Yeah, that's fascinating to
46:16
think about because I know you think about these apps such as
46:20
Twilio and Twilio feels like they're in a perfect spot to
46:26
help others considering what they do, just if they add some
46:31
kind of GPT feature to it. So it makes it really easy for
46:34
people who are already using the Twilio features, then they can
46:39
go and add that on top. And so I think we don't need to
46:43
get into it because it feels like that's probably for a
46:46
different discussion when it comes to, do you create a whole
46:50
new tool or do you do what hopefully Twilio will do since
46:54
they already have the distribution, they already have
46:55
the customer base, you just throw on some chat GT.
46:59
Or something like that. But it's not that easy.
47:02
Not on the tools and not on the process.
47:04
Because if people think it's a new offshoring era, let's fire
47:08
100 people in US or Europe and fire inside of machines instead,
47:13
like we hired in India. That's not that you will need
47:16
different tools to deal with hallucinations.
47:18
So we need different skill set. For example, you need more
47:21
people that do data cleanup. Etcetera and data processing you
47:25
will need probably people that will be higher and customer
47:28
service agents to build with exceptions the things that high
47:31
stakes for example giving refund or monetary aspects.
47:35
So the entire process and toilet needs to be reengineered.
47:38
So it's not as simple as like oh let's just yeah, let's just
47:42
pretend that we added ChatGPT on top of Twilio, other tools and
47:46
let's call it a bit. I think we are very far from
47:50
figuring out what how it will look like.
47:52
It's a It's a big. Amount of It's a big change.
47:55
Yeah. Well, it is interesting you
47:57
mentioned that the the data side is going to be so important and
48:00
it feels like the data engineers are going to be even more
48:06
important if they weren't already an integral piece of the
48:09
whole MLS or machine learning lifecycle.
48:13
The data engineers are going to be so integral in this and maybe
48:17
you can speak a little bit to that and what you see happening
48:20
there. Yeah.
48:21
So that is actually 2 aspects to it.
48:23
So I mentioned the first aspect. So the most interesting
48:27
questions that we still don't have an answer is that will the
48:31
field of 11 field dominated by. Huge and mostly proprietary
48:35
models. Or it will be split or maybe
48:40
even dominated by more custom models that are built on top of
48:43
open source models that are custom customised and fine-tuned
48:49
for specific industry specific use cases or even specific
48:52
company. And it's not.
48:54
I mean, I don't know the answer to it.
48:56
Like the first three months it was like, you know, open the
48:58
eyes eating the world and everybody would try it.
49:00
But as I said, they would try. And then when people go to
49:03
scale, they just realise a it's expensive and there is issues
49:07
with privacy and security and proprietary and you name it.
49:12
And then suddenly we see something that we, I knew
49:15
forever that like opening it doesn't have any big mode
49:19
because a lot of those research came out of Google.
49:21
Google does have mother. You probably saw this articles
49:24
that. We don't have a mode and neither
49:27
does Open AI. There's a tonne of companies
49:29
that develop open source models that any industry, you know,
49:34
entertainment industry could take an open source model and
49:37
put their own training data, especially if it protected and
49:42
do something with it, something interesting.
49:44
So if that happens you will see a huge impact on Elopes and
49:49
DevOps to basically all the tooling and the vector database
49:53
etc. Find that data prepared
49:56
preprocessor, store it in the right embedding manner or
50:00
vectorized manner to make it available for for training.
50:05
So that's one angle that I think it's relatively highly dependent
50:09
on how you which models will be used.
50:13
And my feeling it will be there will not be 1 dominant approach.
50:17
It will not be all open AI or Google or all Microsoft, it will
50:22
be split. Some companies will use custom
50:24
models. Some companies will use
50:27
proprietary big models because those models are expensive.
50:30
The 2nd aspect is that I think it's very important is what I
50:35
call data retrieval. So I was talking about that you
50:38
need to connect your internal. Knowledge bases to loans.
50:44
It doesn't have to be that need to train it every time or fine
50:47
tune it, but you definitely need to make sure it's accessible
50:50
through some kind of indexing or some kind of internal enterprise
50:53
search engine. And that is also totally known
50:58
trivial task that requires to some extent envelopes and
51:02
DevOps, different type of tasks. So it's becoming closer and
51:06
closer to data science and finally, I believe that.
51:10
In many cases other lamps will be focused much more on handling
51:14
what I call front end. They will handle customer
51:17
interaction in a sense. They will understand the intent
51:20
of the customer. That will also generate the
51:22
output. What we're told of tools on the
51:25
back end that some of them might be ML based or even LLM based or
51:29
some could be just classical data retrieval that is also very
51:33
complex. How do you call your private and
51:36
public sources of information? Internal knowledge bases, I
51:41
don't know Reddit again in my example of how do I reset, I
51:44
don't know router, you might have information internal
51:47
information, but if it's a complex issue, maybe even shared
51:51
on Twitter, on Reddit, what people are saying how to deal
51:54
with this and that could be another option to help people.
51:57
So you need to query a lot of databases, which is very hard
52:02
grungy work and have some kind of logic what kind of
52:05
information you pass to LLM. So will do the reasoning and
52:08
summarising and basically. Finding was the right answer to
52:11
give to the user all the stuff I think will basically those
52:16
smaller internal models, whether I like milk based or not or LLM
52:22
based or not. We'll send a lot of this back
52:24
end and that would require a lot of work and I think Milos and
52:29
DevOps, their skill set needs to be much more what I call
52:33
friendly. They would need to be much more,
52:36
much deeper understanding there. A strength and also limitations
52:42
of elements and making sure the systems, internal systems are
52:45
designed in a way that will make the make lens be more efficient.
52:51
And that is such a great point and this idea of it's the front
52:55
end but it's not necessarily the back end.
52:59
We're taking the input and doing something maybe with some LMS
53:02
and then you pass it to the back end and you potentially can use
53:07
some kind of rule based system. But as long as it's passed
53:09
through that first LLM and then when you're bringing it back,
53:14
you are maybe passing it again through another LM and Yep,
53:19
yeah, yeah, I I love that idea. So that there's so many great
53:25
things that I've learned from you in the last hour chatting.
53:27
I've got one more question if you will allow me.
53:30
I know we're a little bit over on time, so hopefully somebody
53:34
isn't waiting for you on the other end of this call.
53:37
The thing that I'm wondering about as a product leader, how
53:42
do you recommend to the machine learning engineers to work with
53:48
product and work with the teams to better position themselves
53:54
for success? Yeah, great questions.
53:57
I would look at that from 2 angles.
53:58
One is on the technical side, which is probably closer, really
54:01
try to be on topples as much as you can.
54:04
There is, I mean, the head is spinning even for me.
54:07
There is a tonne of stuff happening every day on like
54:10
what's going on with models. But there is good, there is a
54:13
lot of good articles that even right now the leaderboard.
54:15
And by the way, if needed, I can send a link the leaderboard that
54:19
compares leading proprietary and open source models and their
54:25
performance and they even show. How they measure it.
54:28
I think understanding those aspects and looking like where
54:31
different models come ahead and where they have limitations.
54:34
I think understanding that angle will be very critical because it
54:37
will start and people start understanding where the
54:40
limitations are. Also understanding it beats the
54:43
basics of how looms work, especially the concept of
54:46
embedding or vectorized database that the how they try to predict
54:52
the next word because that will give people an idea when they
54:55
start thinking about. The data from internal knowledge
54:59
bases, from internal system needs to be potentially stored
55:02
or it is prepared to be ready to be used, consumed by looms and
55:06
more efficient manners. That's important.
55:08
And on the other angle, I think on the product side, the more it
55:12
can be product and client professional and try to
55:15
understand and ask a tonne of questions and even challenge the
55:17
product managers about the use case.
55:20
Because, as I said, charities are still not hype about, oh,
55:23
let's just connect. Let's just take some kind of an
55:26
input and call. The GBT 4 and give some some
55:29
response on that. That's our application.
55:32
That's not an application, that's a demo.
55:34
Start syncing as a user like, OK, here is my user and how does
55:38
that? Is it a demo for 5% of my users?
55:41
So it's actually applicable to many users.
55:44
I mean, I always encourage and maybe some of my product
55:48
colleagues will not like it because they sometimes want to
55:51
pretend that they know everything, but like nobody has
55:54
a monopoly on good ideas. Right.
55:58
Good ideas will come from everyone and bad ideas could
56:01
come from everyone. So it's important to try to ask
56:03
questions and understand the use case and product or business use
56:06
case as much as you can. Those two says, I think will
56:09
make you much more efficient, professional.
56:13
Brock, this has been fascinating.
56:15
I really appreciate you sitting down and chatting with me about
56:18
this. I learned a tonne.
56:19
And of course, if people are not following you on LinkedIn, I
56:22
highly encourage that you share all kinds of great wisdom and
56:28
up-to-date information on everything that we just talked
56:31
about today. So I think we're going to end it
56:33
there. Thanks again.
56:35
Loved having you. It was a pleasure.
56:37
Thanks a lot