Latest / Agentic Conversations (formally mlops.community) / GenAI in Production - Challenges and Trends // Verena Weber // #224

Transcript

GenAI in Production - Challenges and Trends // Verena Weber // #224 · Agentic Conversations (formally mlops.community)

0:04
Hold up, wait a minute. We gotta talk real fast because
0:07
I am so excited about the MLB's Community Conference that is
0:12
happening on June 25th in San Francisco.
0:14
It is our first in person conference ever.
0:17
Honestly, I'm shaking in my boots because it's something
0:19
that I've wanted to do for ages. We've been doing the online
0:23
version of this and hopefully I've gained enough of your trust
0:26
for you to be able to say that. I know when this guy has a
0:30
conference, it's going to be quality.
0:32
Funny enough, we are doing it. The whole theme is about a I
0:37
quality. I teamed up with my buddy Moe at
0:40
Colina who knows a thing or two about AI quality and we are
0:45
going to have some of the most impressive speakers that you
0:50
could think of. I'm not going to list them all
0:52
here because it would probably take the next two to 5 minutes,
0:56
but just know we've got the CTO of crews coming to give a little
1:01
keynote. We've got the CEO of Ucom
1:05
coming, We've got Chip, we've got linens, we've got the whole
1:09
crew that you would expect. And I am going to be doing all
1:13
kinds of extracurricular activities that will be fun and
1:17
maybe a little bit cringe you. You may see, hear or see me
1:22
playing the guitar. Just come.
1:24
It's going to be an awesome time.
1:25
We'd love to have you there. And that is again June 25th in
1:30
San Francisco. See you all there.
1:33
Hi everyone my name is Marina and I work as a self-employed
1:37
generator for IT consultant or also Chief AI Officer on demand.
1:43
So basically I help companies going on the AI journey
1:47
transforming into an AI on centred or AI integrated
1:53
business. So I so this starts with
1:56
architecture and the AI strategy going over to governance and
2:02
also helping them with integration and adoption.
2:06
And yeah, I usually have my coffee black.
2:10
So no sugar, no milk, just black.
2:15
MLP's community We are back for another podcast.
2:18
I'm your host Dimitrios and today what a conversation with
2:22
Marina. If there's one thing and one
2:24
thing only that you can take away from this conversation, it
2:27
is the cliche of starting with a problem.
2:32
Don't start with some technology that you shoehorn into figuring
2:37
out if it will solve your problem.
2:39
Marina really harped on this and harped on how when she has been
2:45
the most successful in her career it is because she has
2:48
been problem focused and problem oriented and then gone out to
2:52
find what solutions there are to solve those problems.
2:56
As opposed to scouring Twitter or papers and all the newest
3:01
stuff that comes out in this AI world and then looking for a
3:06
problem to link it up with. I know it sounds like it's
3:10
something that we talk about quite a bit and it is easy to
3:15
say it, but it is hard to live by it.
3:18
Brina made sure to make it super clear how important it really
3:23
is. And then a few other pieces that
3:26
I will give to you without spoiling the whole episode.
3:30
She talked about how one of the problems that they were having
3:33
when she was working on the Amazon Alexa programme or
3:38
working at Amazon on Alexa I guess is another way of putting
3:42
it. They were having problems when
3:44
they were retraining the model and they wanted to make sure
3:49
that if they retrained the model or if they fine-tuned the model,
3:53
that all of the classic utterances.
3:56
Basically, Alexa, play your favourite song of the week, you
4:03
don't want to update the model and then all of a sudden that
4:06
doesn't work for half of the user base because that would
4:09
create a horrible user experience and probably a lot of
4:12
support tickets. So she recognised this and they
4:17
went out and they went looking for a solution to this problem
4:20
and just so happens that another team at Amazon was working on
4:24
something very similar. She breaks down how they did
4:27
that and how they managed to make sure that when updating
4:30
models it doesn't just totally skew all this stuff that it's
4:35
learned in the past. I had a blast talking to her.
4:37
We got into all sorts of cool subjects, even a little bit of a
4:41
ski trip that she went on, which you will hear about right now in
4:43
the beginning and how she has been tackling the
4:47
entrepreneurship from the slopes.
4:50
I loved it and I will say before we jump into the full
4:54
conversation, thank you to everyone.
4:57
I am so humbled by these support that we have been getting for
5:02
the in person conference that we're doing on June 25th in San
5:05
Francisco. It is incredible.
5:07
People are talking about flying into San Francisco from around
5:10
the world that just like it melts my heart.
5:15
I don't know what else to say. It is so cool to see the support
5:19
if you want to get a ticket already, We just released the
5:23
early bird tickets. It's at aiqualityconference.com.
5:29
We got to give a shout out to the folks at Coletta who are
5:34
helping us and we are partnering with to make this the best
5:37
conference that you probably will ever go to.
5:41
I want to do some cool things and I'm accepting requests.
5:44
So if you have ideas on how we can make this a whole lot of
5:48
done. I've already thought about
5:49
setting up different jam rooms potentially.
5:52
Joe Reese has been contracted to be DJ I I want to have some fun
5:59
bikes that we can ride around. Right now I'm literally going
6:02
into a conversation with a community member, Wes, who owns
6:06
a coffee farm in Columbia. So maybe we'll ship up some
6:10
coffee beans special for the occasion.
6:12
There's so much stuff that I'm excited to do.
6:15
Feel free to reach out to me if you have any suggestions.
6:18
But June 25th, I'll see you all in San Francisco.
6:21
And thank you so much for this report we've had thus far.
6:24
Let's get into this conversation with Brenda.
6:33
Alright, so you were just skiing in Italy.
6:36
Where were you skiing? It's a place called Cervinia,
6:39
which is basically on the border to Switzerland and you could see
6:43
the Matterhorn but from the Italian side, so that was pretty
6:46
cool. It sounds so incredible.
6:49
And what were you doing there? Give me the whole breakdown
6:52
because I love this idea and I really want to do something
6:55
like. This, yeah.
6:56
So this whole thing was it was called Sky Person of Influence,
7:01
which is basically an entrepreneurial ski trip.
7:04
So you go on a trip with we were like 45 peoples or 45
7:09
entrepreneurs and you go skiing, you have a little bit of content
7:13
sessions in the evening. But yeah, most of all you, you
7:17
get a lot of chances to speak to other people what their
7:21
challenges are, what how they're solving them.
7:24
And yeah, this whole thing was organised by Daniel Priestley.
7:29
So he is a very successful entrepreneur who has built a
7:34
couple of businesses already and now does does actually coaching
7:38
for other entrepreneurs. So maybe you've heard about Key
7:41
Personal Influence, which is like a quite a successful book
7:46
that he's written. And yeah, he just organises this
7:49
trip because he likes it and it was a super awesome experience,
7:53
I have to say. Yeah.
7:55
And over the people that were there working on.
8:00
And it was a pretty mixed group. And so there was like one guy
8:05
who had built a business that build a software for
8:11
bookkeepers. There were two women who were
8:14
building a community of bookkeepers and they're also
8:17
doing training. So basically teaching them how
8:21
to make a six figure business out of their bookkeeping
8:25
services. There were people in in the
8:28
cybersecurity field, people are building apps to track your
8:33
location, which then helps you to submit your texts.
8:37
Declaration because you need to show that you're not spending
8:40
too much time outside the country.
8:44
Hit was really cool. Like, he opened this whole
8:46
event. Like, yeah, let's not only ski
8:50
down black runs, but let's also have black run conversations.
8:55
So really, not just, you know, talking about the superficial
8:58
things, but really going deep and what are your current
9:01
challenges? What are the problems that
9:03
you're facing? Yeah.
9:05
And and this is really what what happened then during the
9:08
challenge we did have a lot of black round conversations, as we
9:11
called it. It's pinch foots.
9:13
Really cool. Yeah.
9:15
Yeah, I saw someone doing this with data engineers.
9:20
They got a bunch of data engineers together and they
9:23
said, OK, we're going to have a bit of like a conference, but
9:28
not really a conference like a retreat, I think is a better
9:30
word for it. And we're going to get 40 or 50
9:34
data engineers together, hand-picked.
9:36
We're going to go to Switzerland and we're going to all stay in
9:39
the same hotel. We'll go skiing during the day
9:41
and in the afternoon, evening we'll be hanging out.
9:45
But we'll also be sharing stuff about what we're doing and what
9:48
we've learned over the years. And we have a few speakers,
9:50
etcetera, etcetera. A friend of the pod, Joe Reese,
9:53
was there. And that's how I heard about it.
9:56
And I was like, Oh my God, how do I get an invite to that?
10:00
That sounds like a blast. And then I started thinking
10:02
about it and I was thinking, we should do something like that.
10:06
And potentially, since ski season's over and I don't have
10:09
patience to wait until next ski season, I was thinking about
10:12
Ohh. We can go to Portugal and do it
10:15
but surfing instead of skiing. Yeah, actually everyone was a
10:20
really, really energised after the wind and we started talking.
10:24
Ohh, we should do it in summer again.
10:25
We don't want to wait until next year when Daniel does the ski
10:28
trip again, but let's have something in summer.
10:30
Yeah. And there were actually a couple
10:32
of people who do a similar thing for, for their audience, for
10:35
their clients. And yeah, yeah, yeah, it's a
10:39
really nice idea. It feels like it could be a
10:43
really cool way to bond with people and socialise and then
10:46
learn a tonne that you wouldn't necessarily learn from a talk at
10:52
a conference that you get to go deeper.
10:55
Not only about, you know, what you learn conceptually and
10:59
intellectually, but it's also this kind of energy that you
11:02
take with you, this inspiration, this sense of possibilities, um,
11:09
that changes how you feel about yourself, how you see the world.
11:12
And I think that is probably also something you don't take
11:15
away from conference. Yeah, yeah, for sure.
11:20
So you've had quite an awesome background in general.
11:24
I want to get into what you're doing now with coaching and with
11:28
your AI consulting stuff, but 1st we should probably start
11:32
with what you were doing over the past like 5-10 years well.
11:38
I mean, I started originally with statistics, so really
11:41
classical master in statistics. So back then actually there
11:44
wasn't a lot of data science machine learning programmes
11:47
happening yet. But luckily I did have some
11:50
courses in data science and machine learning during my
11:53
master. And then you kind of enjoyed
11:56
that much more because it was more trial and error, less based
12:01
on assumptions and then decided to to just stay with that field
12:07
and back at the time. So I was doing my masters degree
12:11
in Berlin already and really liked the city, didn't want to
12:14
move anywhere else. So I was trying to find a data
12:17
science role in Berlin, but at the time you wouldn't believe
12:20
that now. But there was hardly any company
12:23
hiring data scientists, not even Sarandos.
12:27
Orlando was just hiring in Dublin, so I didn't want to move
12:30
to Dublin. I can understand why exactly.
12:34
No, no. Hate to all The Dubliners out
12:36
there, but Berlin is an awesome city and I wouldn't want to
12:39
leave if I was living there either.
12:40
Yeah, and also the weather in Dublin is is harsher than it is
12:43
in Berlin. And Berlin can be harsh already.
12:46
Yeah, that is really where where my limit is, yeah.
12:51
But then luckily I found a data scientist role at Deloitte.
12:54
So basically consulting company that at that time had a
12:58
specialised unit for data engineering and data science,
13:03
which was cool because you could try a lot of different things.
13:07
So I tried everything from time series, anomaly detection, NLP,
13:14
computer vision, everything. Yeah.
13:19
And then I mean at the time also data science was really kind of
13:22
more things that people would try if they were very
13:25
innovative, but it was not at the centre of attention as it
13:28
now. So basically it was a lot of
13:31
proof of concept stuff that we did, but not that much moving
13:34
into production or going further than that.
13:38
So then I was looking for for a job where I could go over this
13:42
proof of concept stage, spend a little bit of time at at the
13:47
digital, then went to Mobility E which was part of eBay at the
13:51
time, because they then had way more data.
13:56
Because I realised that the more classical and older companies
14:00
didn't have that much data yet where you could really work with
14:06
they were really. Or a paradox, I should say.
14:09
They were kind of trying, you know, because it's always easier
14:13
if you build something from the ground up, right?
14:15
And then you can build this data-driven thing right into the
14:17
company. That's always easier compared to
14:20
when you have to make this transformation.
14:24
You know what's funny about that, which I have been thinking
14:28
a lot about lately, is that you have the companies who started.
14:35
So you have those, those typical companies that you're talking
14:39
about in 2018, they weren't considered advanced and they
14:44
were still trying to go through that digital transformation
14:46
journey and it was very slow and they weren't really getting on
14:50
the bandwagon. But then you have the tech
14:53
forward companies that were doing that and they had to make
14:57
a lot of design decisions in 2018 and 2019 on how they were
15:01
going to build their stack, which today feels like OHH, they
15:07
have to support legacy tools and they have to either get off of
15:12
that legacy or they, if they're contributing to open source,
15:17
they have to be like the main contributor to that open source
15:19
project because they made that decision in 2018 when that's all
15:24
there was. But now in 2020 or there's much
15:27
easier ways of doing things. And so if you were one of those
15:30
companies that in the 20 tens was very slow to move and now
15:35
you're finally getting on board with it, you can potentially
15:39
have a much easier Ave to getting things into production.
15:43
And the easiest example of this is just like look at what LM's
15:48
did and they just exploded and got everyone into the AI spear
15:53
super easy. But I'm even thinking about all
15:56
the infrastructure that you need and how that decision like I I
16:03
always come back to this conversation that I had with a
16:06
guy that works at Pinterest and he was saying, yeah, in 2016 we
16:10
started our ML journey And so we had to create infrastructure
16:14
around machine learning in 2016, which we are continuously
16:19
updating and we are continuously like trying to see what's the
16:23
ROI if we shift off of this tool, do we need to continue
16:26
supporting it and all of that stuff.
16:27
But I digressed that was a little bit of a tangent.
16:31
So I didn't wanna take you off course there.
16:35
No, absolutely. And I mean that actually brings
16:37
me to my next step. So I I spent before I became a
16:42
self-employed consultant. I spent almost four years as a
16:45
research scientist at Amazon Alexa and for a natural language
16:50
processing device or I mean virtual assistant, Alexa is
16:53
actually pretty old, right? They're like 80 years or
16:57
something, but now I think so. So that is super odd.
17:00
If you think about what happened alone in in the last two years
17:03
in that space and also there right when they started the
17:08
whole field was in a different state.
17:11
So what what their one of their main challenges now is actually
17:15
you know working through all that legacy and and being able
17:19
to to adopt all these new technologies that also keep
17:23
changing at such a rapid pace. That's so true, yeah.
17:28
And were you all using? I imagine it was all using Sage
17:32
Maker for this, so or is it a homegrown?
17:38
Yeah, it's a lot of homegrown tool that we were using.
17:41
I mean, of course it does rely on AWS at the end, but I didn't
17:45
use like this. There's normal, say, Jamaica
17:48
interface that that other people are familiar with.
17:51
So Amazon actually likes building their own tools.
17:53
And within Alexa, yeah, I think the only tool that everyone
17:58
would know that we used was Slack, but it was so only after
18:02
a while. So when I when I joined, Amazon
18:05
still had their own tool, which was called China but really
18:08
sucked. So he went.
18:09
Should he be moved to Slack? So tell me more about Amazon and
18:15
working there and what were some of the key challenges you were
18:17
working on? Yeah, so I was a research
18:20
scientist in in the team based in Berlin.
18:23
So we were responsible for Natural language understanding
18:27
models for German, French and then later also for English and
18:32
Great Britain. So our, yeah, our work was
18:36
basically divided between two parts.
18:38
So one part was the really maintenance operational staff
18:41
where you had to retrain models and deploy new releases, updates
18:47
of the model and you know sometimes also stuff like making
18:51
sure the training data is all right.
18:53
So really more things that were on, on the maintenance side.
18:58
But then on the other hand, we really had also research
19:02
projects where we were, yeah, trying to find ways how we could
19:08
improve the model or improve the processes around how we work
19:13
with the model and basically automating processes around
19:18
model deployment or improving the model itself.
19:21
And that was pretty cool because how that was different from
19:25
previous data science rules that I had was it has much more, it
19:30
had much more of an academic component.
19:32
So we were really reading the newspapers and then also going
19:37
to academic conferences. And part of our goals was also
19:41
to publish our own papers, which I really enjoyed.
19:45
Nice. And of course, yeah, I got to
19:47
work with really cutting edge technologies or all the deep
19:50
learning stuff that usually in in other companies you of course
19:54
don't work with because it doesn't make sense for them.
19:57
But since in Alexa machine learning is a core product,
20:01
right. And you really they, yeah, you
20:04
really invest a lot of time and effort into improving their
20:08
product. So that was cool.
20:10
What were some of the papers that you read and then you felt
20:12
like? Ohh, let's try this and see if
20:14
it will make a difference. Do you remember?
20:18
I mean usually it worked the other way around.
20:20
So usually we we would start looking at the problem.
20:24
So what are the problems that we really need to solve that are
20:27
holding either aspect or are holding our or I really
20:32
negatively impacting the customer experience.
20:35
So I mean one of the approaches that Amazon is you work
20:38
backwards from the customer, right, And you always work
20:40
backwards from the problem. So usually we define the problem
20:45
and then you go looking for the papers.
20:47
It it hardly was the case that we looked at the paper and then
20:51
thought, OK, this is also a problem that we could solve.
20:55
But actually there was one one example indeed.
20:58
Um, it was a paper that was published by a WS, so one of our
21:02
colleagues basically and they looked in.
21:05
So the paper was called Positive Congruent Training and it looked
21:10
into how you can minimise the negative flips between updates.
21:15
And a negative flip is basically so if the previous model
21:19
interpreted a training instance correctly but then the new model
21:25
interprets it incorrectly, you have a negative flip, something
21:28
you don't want. And that is particularly the
21:32
case if you have a model that's running in production because
21:36
you want to keep the user experience constant.
21:39
And and this is especially important for Alexa, right?
21:43
If there is like important requests or utterances as we
21:46
call it, and then you retrain the model and all of a sudden
21:49
that doesn't work, it's pretty annoying for the user, right?
21:52
And also for us, because then we get a high severity ticket as
21:56
though we had a very complicated process of making sure that
21:59
after every retraining the model still works on these very
22:03
frequent utterances. So basically we applied this
22:07
technique of adding another term in the loss function to prevent
22:14
these negative flips from happening during the model
22:17
training and not after. It feels like that is still so
22:23
relevant today because how many times have we heard people talk
22:27
about how? Some random open AI update
22:32
happens and now my prompts don't work as well.
22:34
Yeah, now. And that potentially could be
22:38
one of the things that might help in that case.
22:42
Yeah, and it's also not an easy one to sort of right.
22:44
I mean, even with this technique you can prevent some negative
22:48
flips, but it doesn't mean you you prevent them all.
22:52
As you were talking about that, I wonder if you had some kind of
22:58
understanding of the data of like the distribution of what
23:02
people asked Alexa and there's got to be a tonne of common
23:07
questions that people are asking, right?
23:08
And that it's almost, I imagine there's like a bell curve.
23:11
And did you, did you look into that and make sure, OK, the
23:16
majority of questions we can never let those go awry and all
23:22
the other stuff, let's try to minimise that.
23:24
But if it's that bell curve, I want to make sure that all these
23:29
really important questions that people are asking continuously,
23:32
we never get that. Like we have to really test to
23:37
make sure that isn't going haywire.
23:40
Yeah, absolutely. I mean there is really like a a
23:45
set of utterances or or requests that are super frequent, um that
23:50
we really also had to make sure work perfectly all the time.
23:55
And and then there's this very long tail where you have a lot
23:59
of things that are very infrequent and but still you
24:03
have a lot of these. Continuing on the theme of just
24:07
like what you were doing in your day-to-day while you were
24:10
working on Alexa, are there any other challenges that you went
24:14
through as you were working on that and you were like, whoa,
24:17
this is a really novel way of solving this problem that we
24:22
encountered. I mean there were a couple of
24:25
things, right? But I think I don't know if it's
24:28
necessarily totally novel, but it was also very specific setup.
24:32
But I think 11 project that I really enjoyed and that was
24:36
really cool was kind of making the model more robust to small
24:44
changes in the input utterance. So what we saw there is that
24:48
even small changes like the user saying please well, so could
24:53
actually make a change in prediction of course not
24:57
something you want, right. So what we did, and there's also
25:01
a paper about it, so that that's why I can talk about it this.
25:05
And we trained a T5 model on on these utterances with a small
25:11
variation and basically had that generate more of such variations
25:16
and then use that in our training data to to make the
25:20
model more robust against these small variations.
25:24
And that was pretty cool, yeah. Which again, like now, is so
25:30
obvious for anyone who has spent 5 minutes prompting and trying
25:34
to understand how to prompt better or what words are going
25:40
to be taken differently when prompting.
25:43
O if you prompted then you know that by saying please or by
25:48
throwing in, changing up the word order, it's going to
25:51
drastically, sometimes drastically, sometimes not so
25:53
drastically change the output. And it feels like you were
25:57
discovering that a little bit before the rest of the world.
26:02
And so the way that you solved that was by creating synthetic
26:06
data with this T5 model and then using it as more robust data for
26:12
the training. Is that what I understand?
26:15
Yes, exactly. Yeah.
26:16
And also part of it for testing, yeah, to see if we actually make
26:22
it more robust. It seems like you were doing
26:23
that for training data, but have you also thought about doing it
26:26
for training data for fine tuning training data?
26:30
Ohh yeah I mean if I speak about training data it is fine tuning.
26:35
It's not pre training. Yeah, sorry I I forgot.
26:38
To mention. OK.
26:40
Yeah, we. I mean because usually what what
26:44
we were working on were this was this fine tuning step because
26:47
you don't do pre training all the time, right?
26:49
You do that once, and then every update is just a fine tuning
26:52
step that you do. So you were constantly just fine
26:55
tuning the last version of the model?
26:58
Yeah. Yeah, interesting.
27:00
And don't you get into a little bit of hot water with like,
27:04
what? One of my favourite words in
27:06
this space. Like catastrophic forgetting.
27:11
Yeah, I think for Alexa it's fine because you have so much
27:15
training data and it's not like you're changing, you know?
27:20
I mean, everything that Immortal still needs to know is still in
27:23
the training data, in the fine tuning set, and it's it's there
27:27
to a large extent. So we didn't have that problem
27:32
that much longer. Ohh, that's awesome.
27:35
OK, well that's good to know. And so then the the other piece
27:40
last part about working on Alexa and then I want to get into some
27:43
of these different topics that I I know you brought up.
27:48
It was around like the training data and ensuring the quality of
27:51
that And so you had the synthetic data.
27:54
Were you manually checking that out to make sure that it was OK?
27:58
And if it's all audio data, it feels like it's much more time
28:05
consuming to label it and quality check it then if it's
28:11
text, right? Or is that just my false
28:14
intuition? Ohh, no, you're right probably.
28:17
I mean, I'm not a specialist in analysing audio data, but I
28:20
would assume so. Alright, real quick, some words
28:23
from our sponsors Ellis and then we'll be right back into the
28:25
programme. Are you struggling with building
28:29
your Retrieval augmented generation AKA RAG applications?
28:33
You are not alone. We all know tracking
28:37
unstructured data conversion and retrieval is no small feat, but
28:42
there's a solution available now.
28:44
Introducing Soliz Cloud Pipelines from the team behind
28:48
the open source vector database, Melvis.
28:51
This robust tool transforms unstructured data into
28:55
searchable vectors in an easy manner.
28:58
It's designed for performance. Zelis cloud platforms pipelines,
29:04
Zellis Cloud pipelines also offer scalability and
29:08
reliability. They simplify your workflow,
29:11
eliminating the need for complex customizations or infrastructure
29:15
changes. Elevate your rag with high
29:18
quality vector pipelines. Try Zilis Cloud Pipelines for
29:23
free at soliz.com. You can find the link in the
29:26
description. Let's get back into this
29:28
podcast. But yeah, I mean, the way Alexa
29:31
worked when I was there, I mean, I don't know if if they changed
29:34
the system in the meantime. So I want to be clear about how
29:37
it worked when I was there. So you basically transcribe the
29:42
audio into text and then you sent the text to the natural
29:45
language understanding model. So all our input that we got was
29:48
text, so we could basically just analyse the text.
29:52
And to the to your other point about and whether we check the
29:56
quality of the generated data, yes, we did and and we did
30:00
actually you know some some kind of exploratory analysis, but
30:05
checking in this kind of these kind of things And then based on
30:09
that we derived some heuristics and methodology methodologies to
30:13
clean up the generated training data because you definitely
30:17
couldn't use everything that came out of the T5 model, but we
30:21
had to put some filtering mechanisms in place.
30:25
So this feels like a good segue into one of these themes that we
30:29
wanted to talk about, which was like shifting from unimodal to
30:34
multimodal. And when you said like, ohh
30:38
yeah, we would, we would transcribe the utterances into
30:43
text and then send the text to the model, I instinctively was
30:47
like, yeah, of course that makes total sense.
30:50
I should have known that already.
30:52
I feel a little bit like an idiot since I didn't.
30:54
Know. No, no, but no, actually not.
30:56
I mean there were also, yeah, I mean that's that's just, I
30:59
think, yeah, that's the fact that Alexa is really old, right.
31:04
That's that's why the system is built that way.
31:06
But there were projects that, yeah.
31:10
Or there were also ideas where we thought, OK, maybe we can
31:12
just make this audio too. Intent or domain.
31:15
Yeah, yeah. But that's that's just it's an
31:20
old system that takes time to to change.
31:23
And then of course, it also has its, its advantage is if you
31:27
break it down in steps, right? Yeah.
31:30
Yeah, the pros and cons and the trade-offs for that.
31:34
And my mind is racing on like what trade-offs you would look
31:37
at in those cases. And at the end of the day, it
31:41
probably comes down to how accurate it is and how fast it
31:46
is. Yeah, yeah.
31:48
And also how easy to maintain it is, right?
31:51
Yeah, because I I think this maintenance part is a little bit
31:56
easier if you break it down into steps and also if something goes
32:01
wrong right you you it's easier to to fix it.
32:06
Yeah, the debug ability. So as you're thinking about the
32:13
shift from unimodal to multimodal, what are some things
32:16
that come to your mind? Yeah, I mean it.
32:19
It's just been, I think, groundbreaking, right?
32:23
That now you can't just input almost everything you think you
32:27
can think of into into these large language models.
32:30
And it it's it's just making the application space so much
32:35
bigger. I mean, just creating a
32:38
marketing video or putting in some lines of text, you get a
32:41
video. I mean, that's amazing, right?
32:44
It it's just crazy to see that as we just spoke about how, you
32:48
know, things used to be divided in step by step things and you
32:51
had these models that were specialised on one modality.
32:56
And we saw this happening with large language models too, right
32:59
where they were at first only focused on tax, but now we're
33:02
really with Gemini, you can have video, audio, text and so many
33:08
different modalities. And it's it's just amazing
33:12
because you can basically use it for everything for a much wider
33:18
variety of tasks. Yeah, and and also you can get
33:22
more clear on the prompts too, which is or get more specific.
33:27
I think it's if you can give a photo and give the text, It's
33:34
much more accurate in my eyes. Yeah, yeah, maybe, yeah.
33:39
And then I think, yeah, one example that I saw was pretty
33:42
cool, where you basically upload a photo of and a scene in a book
33:50
and then the model gets you the right pages from the book or you
33:55
upload basically a photo or or you just put in how you solve a
34:00
problem, a physics problem, and then the model says, ohh and
34:04
that's where you went wrong and this is how you need to do it.
34:06
So I think also the applications and how we can use this
34:09
technology in education and self learning is has really expanded
34:15
through this multimodality. And so when you're working with
34:19
companies and you're seeing these different ideas, I can
34:23
imagine that you, if you're anything like me, you're like
34:26
wow, there's, you know, we could do this and that and there's
34:29
this super cool cutting edge way of utilising the newest model
34:35
when in reality that may be over engineering it.
34:38
And what you want to do is just get some kind of a MVP out that
34:43
doesn't need to use all the newest cutting.
34:45
Edge. Absolutely.
34:46
I mean, I'm a big proponent of not using the cutting edge stuff
34:51
if you're just starting out. I would never recommend anyone
34:54
who's just getting started on their AI journey to, umm,
35:00
immediately start with the news staff unless they have a use
35:04
case that absolutely requires it, right?
35:08
But I am, and I actually just recently gave a talk at a meet
35:11
up where I really pointed out that you should not start
35:18
thinking about the solution and then figure out which problem
35:21
you can solve with the solution. But you need to start with the
35:24
problem, define the problem clearly, and then find solutions
35:29
for that problem. And I always encourage people to
35:32
start with a very simple solution and and for every
35:36
problem you can find a very simple solution, it just maybe
35:40
doesn't perform as well, but you can find a simple solution.
35:42
And then from there you know go to the more complex ones and
35:46
always benchmark it against the simple one and then compare what
35:50
does it cost you to go to that more sophisticated solution and
35:55
what are the the performance gains that you get.
35:58
And oftentimes, this increase in performance does not justify the
36:02
cost and it's also not just about performance.
36:05
Also, like, do you have the capacity to maintain all that,
36:09
that stuff, right? Because it becomes way more
36:12
complex to update systems once you have a deep learning model,
36:17
right? And so there's a lot of factors
36:19
that you actually need to consider when you when you pick
36:22
that solution, not just, yeah, is it really cool, is it really
36:26
fancy? Do we want to try it out?
36:29
Yeah, the classic engineering. Exactly.
36:33
I know. I know it seems so obvious and
36:35
so trivial, right? It it seems like what?
36:39
But yet, I still see people getting carried away with it.
36:44
Yeah, you're. Echoing so much of what past
36:49
guest was talking about, we basically spent 30 minutes
36:52
talking about that exact same thing.
36:53
Like hey, just figure out what your problem is and get some
36:56
good metrics around that and then try and push those metrics
37:00
as far as they can go with the most simple solution.
37:04
Because as and this was Sam that came on and Sam said I've never
37:09
heard an engineer say that. I wish I would have made the
37:12
system more complex. Exactly, yeah.
37:17
And yeah, there's so much you can achieve with simple
37:20
solutions already. And even, you know, I mean
37:25
techie BT and we all know that now, but there there were models
37:28
before Tetchy BT, right, that are already, you know, they can
37:32
solve a lot of problems that the majority of businesses have
37:35
right now, right. And and also I think people
37:38
sometimes forget what CHPT is actually trying to do, right.
37:42
It's trained for conversational language understanding.
37:46
So if your problem is not about conversational language
37:48
understanding, if it's just a simple classification problem,
37:52
then start with a classification algorithm and not.
37:56
I mean, of course check CPT can give you an answer and it might
37:58
even be correct, but it's over engineering the problem.
38:02
And OK, if you're just doing 100 classifications every five
38:08
months, maybe it doesn't matter. But once you want to scale and
38:11
once you have more traffic on that model, it's it's way too
38:14
expensive. But yeah, it doesn't scale and
38:17
it doesn't make sense for. That's my opinion.
38:19
I don't think it makes sense to use tech EBT for classification.
38:23
Doesn't make any sense. That's so.
38:25
True, even though I heard people doing.
38:26
It. But I still think it doesn't
38:28
make sense, yeah. It's.
38:31
It's one of those ones where, you see, I remember we had the
38:36
ball from Angelis come on, and he was talking about how they
38:40
spent six months putting together a classification model
38:44
and in two weeks after Chat Tribute came out, they started
38:50
using that and they were benchmarking it against their
38:53
homegrown model shot. GT was just beating it all over
38:57
the place and they were like, ohh, no.
38:59
So in that case it's a little bit like hmm where is the where
39:06
is the over engineering happening?
39:07
It's good that they were honest enough and.
39:12
What? What's the word I'm?
39:13
Looking for humble enough to recognise, OK, this model that
39:18
we worked on for six months? We can throw it out and now chat
39:21
TPT is much better. But I think for every one of
39:25
those stories you've probably got another 10 stories of what
39:28
you're talking about where chat should be.
39:30
Team may be overkill, yeah? And then also the question is
39:32
what do? You compare it to, right?
39:33
I mean if they build like a model based on TF, IDF and maybe
39:37
some manually engineered features and then compare it to
39:40
to ChatGPT, maybe not the best comparison right?
39:43
I think what you should do then is take a bird model, get a bird
39:47
in bedding and then build a classification layer on top and
39:50
compare that to check GPT where you then already move away from
39:55
manually engineered features. But you you use the technology
39:58
of deep learning, right? You get these embeddings you use
40:02
a lot like transform our model or transformer based model and
40:08
then use that for your classification.
40:09
Because otherwise you're also a bit comparing several rules to
40:14
super sophisticated AI model, which yeah, maybe is is.
40:19
Of course they're not not comparable in that sense.
40:23
Whereas all those tools. That are highlighting how
40:26
they're using Burt. This is you don't see those out
40:28
there. Everybody's highlighting how
40:30
they're using ChatGPT and AI. It would be funny to for someone
40:34
to zig where everybody else is zagging and safe.
40:38
We're using the non state-of-the-art.
40:40
Yeah, and I mean. OK.
40:42
It also depends a little bit what, what are the capabilities
40:44
that you have in your company, right?
40:46
And of course it's super easy to use.
40:49
Everyone can build like a quick proof of concept with that and
40:54
and that is great. And if you don't have like a lot
40:57
of traffic on on the model and OK maybe it's a viable solution.
41:01
But if you really need to build someone something that needs to
41:05
scale, then probably you need to look into other options, yeah?
41:12
Yeah, 100%. So this has been awesome.
41:16
Verena, I really appreciate you coming on here and talking to me
41:19
about all the the good stuff that you've been up to.
41:22
I want to mention to everyone that I have been reading your
41:25
newsletter diligently and I appreciate you doing it.
41:29
We'll leave a link in the description for anyone else that
41:32
wants to get smarter by getting a little bit of you in their
41:35
inbox every week. I also appreciate what you are
41:40
doing when it comes to coaching women.
41:42
Can we talk about that for a minute?
41:45
Yes, absolutely. I'd love to, yeah.
41:48
And so. So that is a topic that's dear
41:50
to my heart and and basically all started when when I read
41:54
that more than 50% of women leave their tech career at the
42:00
midpoint of their career. And part of that is because they
42:04
feel like they need to prove themselves more of their gender
42:07
because of their gender, because they feel like they're not
42:09
giving the right opportunities to grow and they basically just
42:14
give up because they don't see a future in the industry.
42:17
And of course, women are totally underrepresented in tech roles
42:22
and we need more women in in tech roles to really make sure
42:28
that the technology that is being developed is, yeah, it's
42:32
more representative of the whole population.
42:34
And So what I do is I coach women to.
42:39
Yeah, basically take control of, of the things that they can take
42:42
control of. Yeah, because we can't change
42:45
the industry from being male dominated to being balanced
42:49
immediately. But what we can do is we can
42:52
adopt certain strategies to work on our confidence, be more
42:56
visible, be more strategic about our career growth, asked for the
43:00
things that we want and go after the growth opportunities.
43:05
Deal with the stress and overwhelm that, that, yeah,
43:09
sometimes come with these challenges and just developed
43:14
routines that support us in thriving in this environment,
43:18
even though it comes with certain challenges.
43:20
Yeah, the way. That we met, we should probably
43:23
mention that we were on a panel together in one of the MLBS
43:26
community meetups in Berlin, and it was a panel all about having
43:33
more underrepresented voices in tech, I think, and women in
43:39
tech. It was in collaboration with the
43:42
Women in Tech or Women in Data science in Berlin.
43:44
Because in tech, I think girls in tech.
43:46
That that's it. And so I was on that panel,
43:51
which my wife told me it was a very stupid idea right before I
43:55
left to go to Berlin. She was like, you were about to
43:58
say something that will get you cancelled it and hopefully
44:01
they're not recording it. I said, well, luckily I got out
44:04
of it unscathed and I really appreciated a lot of these ideas
44:07
that you were sharing, especially around this.
44:12
What you just said, like there's no reason we should have people
44:16
fleeing from, especially like females fleeing from tech,
44:21
because if anything, we need a lot more in tech to
44:24
counterbalance all of. I think right now there's like
44:28
I've been seeing, there's a little bit more female audience
44:31
of this podcast, which I love seeing, but we could say
44:34
probably like two or three listeners are female, OK?
44:39
Yeah. Well, I definitely appreciated.
44:41
That you were on the pound because it's not something that
44:44
you know we can solve on our own and we need allies.
44:47
We need you know and and also this newsletter is not directed
44:51
at women only. It's really about every man
44:55
that's interested in what are the challenges that women face
44:58
in these environments and they're interested to support
45:01
them. Right.
45:02
And I I think it was really cool that you were part on on of that
45:05
panel. Yeah.
45:07
I always feel a little bit. Awkward.
45:08
When we're organising the conferences and I'm trying to
45:13
get more female speakers, we're just speakers from
45:16
underrepresented groups. It doesn't have to be female.
45:19
Yeah, and they're more but. Having to like.
45:23
Specifically say that to people. I I never really figured out how
45:27
to say it and I always kind of felt very weird being like, ohh,
45:33
no, we can't have you talk because you're not from
45:37
underrepresented groups and people like the men would get
45:40
like why are you biassing against men And it's like, no,
45:44
just want to have space. And I'm I think that for me at
45:49
least there's the benchmark that I'm going for and all the events
45:52
that we're doing is trying at least to get 50 percent. 50%
45:57
feels like that's saying wow, yeah, it's great, but it it's
46:02
definitely hard. But it it also is like, it's not
46:07
outrageous, you know, I mean given that we only have.
46:10
Like 20% of women in tech roles, it is quite hard to meet that
46:15
mark. So and I think it's also quite
46:17
unusual to see that. Umm, yeah.
46:20
But people appreciate it. They definitely do.
46:22
Like people will write on the feedback forms that we have and
46:27
they will say like, it's cool to see so much diversity.
46:30
And so even though you don't necessarily, it's not like I'm
46:34
not going to be there in all the messaging and the marketing and
46:38
be like we get 50% underrepresented groups to
46:41
speak, people still notice and that's cool.
46:45
It makes me feel good because it's a lot of work to do that
46:49
and honestly I think. It's also one way to making
46:52
females more comfortable speaking at these events because
46:56
it's it's always weird when you when you feel like you're the
47:00
only female person in the room and so it's really nice if you
47:04
go to A to an event where this is different.
47:07
Yeah, the truth is. And I I.
47:10
Firmly believe this, that there are so many incredible people
47:15
that are doing such cool stuff from these underrepresented
47:18
groups that it's almost like that's my job is to go out there
47:21
and find them. It may be a little bit harder,
47:23
but that's why we call it work. That's what I got to do.
47:27
So yeah, and it's cool that you. Do that because I tend to find
47:32
that people from underrepresented groups are just
47:35
a little bit more quiet. They're shy, You know, they
47:38
don't necessarily feel safe in these environments.
47:41
And that is why they don't feel comfortable to be visible.
47:45
They don't feel comfortable, you know, going after these speaking
47:49
opportunities. But then when they get asked and
47:52
you know, you even give them this, this nice environment of
47:55
5050, then it can be a good entry point for them.
47:59
And then maybe they find out, oh, it's it's not that bad after
48:03
all. Maybe.
48:03
Yeah, I should do this more often.
48:06
Let me just. Clarify.
48:08
I try for 5050. I'm not gonna say I hit it all
48:10
the time because now you're gonna hold me to that.
48:13
I really try really hard. But the I can't hit it always.
48:18
I get it. I get it.
48:20
It's. Hard.
48:21
Always. Yeah, but it's not.
48:23
Impossible and I. Definitely strive for it because
48:25
I think it's important. So Brenda, this has been
48:28
awesome. I appreciate you coming on here
48:30
and chatting with me about all this.
48:31
I had a great conversation. Me too.
48:33
Thanks for having me this. Was really fun.