Latest / Agentic Conversations (formally mlops.community) / Managing Data for Effective GenAI Application // Anu Arora and Anass Bensrhir // #215

Transcript

Managing Data for Effective GenAI Application // Anu Arora and Anass Bensrhir // #215 · Agentic Conversations (formally mlops.community)

0:00
I had all my name is is as associate partner at Quantum
0:05
Black. I mean speaking about the black.
0:09
I like my coffee and I drink it three times a day.
0:13
In the morning, latte in after lunch and espresso, and at night
0:19
I drip coffee and I'm very big coffee enthusiast.
0:25
My I'm underwater. I'm a principal data engineer
0:29
with Quantum Black, and that's how I take my green tea.
0:35
Welcome back to their Melbs Community podcast.
0:38
I am your host Dimitrios and today we are talking with Anu
0:41
and Annas about how data engineering is still hard.
0:47
That is right folks. Despite the text to SQL models
0:51
that have come out, it is not any easier.
0:55
And that is what we get into today.
0:58
Specifically, we go through some practical and understandable
1:03
examples of how these folks are seeing Gen AI being used in the
1:10
wild. But the challenges that they are
1:13
encountering when they are seeing these use cases go from
1:20
MVP to actual in production. You know, we love that in
1:26
production. I mean, there were some
1:29
incredible points that were hit on here in this conversation.
1:34
The key example that I love was when they spoke about how ETL is
1:42
now something a little bit different.
1:44
The Genai or ETL of unstructured data is still very much a work
1:52
in progress, and particularly when you have to look at things
1:57
like data quality, data quality. When you're grabbing a PDF and
2:02
you're trying to just get that data out of the PDF, how can you
2:05
measure that data quality? That's much different than if
2:09
you're tracking event like an event captured by hitting a
2:14
button on a website, right? So that was fascinating.
2:18
The other piece to that is when you are dealing with something
2:22
like a use case like a rag, how do you update documentation?
2:28
How do you update your rags so that the documents are saying
2:33
the most up-to-date information possible and you make sure that
2:38
that up to date information is what is used in your RAG?
2:44
So you can think about something as trivial as saying like an HR
2:49
policy has been updated from saying you have Now you've gone
2:55
from a European vacation policy of 30 days a year to American
3:00
vacation policy of two days a year and and those two days are
3:06
expected to not be without Slack or e-mail on.
3:12
And you're have you have your little rag going with your HR
3:16
chat bot, and your HR chatbot is grabbing from the European
3:20
policy. It's not grabbing from the
3:23
American policy. What do you gotta do?
3:25
Well, turns out you thought you deleted the documentation where
3:30
it had the different policy on it, but you didn't delete it
3:34
everywhere. And because of that it just it
3:36
just makes this vector database that you're using in absolute
3:40
landmine because the LM will grab or it will get information
3:45
that is potentially outdated. So keeping all of this stuff up
3:49
to date is incredibly important and not as simple as we would
3:55
think. There was all kinds of other
3:58
gems in there. I'm going to let you all listen,
4:00
figure it out for yourselves. If you enjoy this episode, leave
4:04
a review. It would mean the world to me.
4:06
And tell one friend we've got to give a huge shout out to our
4:12
partners, Quantum Black, in this thank you so much for being part
4:17
of the team, helping the MLA's community be what it is.
4:20
Let's get into this conversation now.
4:29
The fun part begins for we wanted to get into this
4:33
conversation. I know because there's a lot of
4:36
talk about MLM's and how useful they are and all that good
4:41
stuff, and it feels like the data engineers have been left
4:45
behind. They've been forgotten about.
4:47
But in all reality, in my eyes, I think the data engineers are
4:54
more of the hero than ever before.
4:57
And so I want to discuss a little bit more about the data
5:02
engineering landscape now that we have the LM's out and there's
5:06
this LM craze and just we're now calling it a I.
5:11
It's not ML anymore. It's all AI.
5:13
We kind of shifted over to that, which I think is kind of funny.
5:17
But today with us to talk about this, we've got I knew and and
5:21
us and we should start with a bit of background anew.
5:26
Can you kick it off? I know you mentioned you've been
5:28
working at Quantum Black for the last seven years and you've been
5:32
a data engineer for 13. So give us a bit of a breakdown.
5:36
How did you get into tech? Hi everyone my name is anus.
5:40
I'm based on London Ohh Tech. Indeed a right I mostly my role
5:46
revolves around helping clients in building their scalable data
5:49
pipelines altogether and capability building and defining
5:53
their data architecture. It's been close to seven years
5:57
here, so the I really enjoy working with data and solving
6:00
the problems that are there with the data itself.
6:02
Like you were talking about, right?
6:03
LLM everybody talks about LLM, not many people talk about data,
6:07
and that's where you give a competitive edge overall.
6:11
I started my journeyasa.net developer long back, but I moved
6:14
into this data Courier when everybody was talking about
6:17
Hadoop and big data long back. And then that's why my journey
6:20
has started overall and I'm going to be more than happy to
6:24
talk about this. Yes, it feels like you're prime
6:29
and ripe for this conversation. So, And what about you and us?
6:32
How did you get into this whole field?
6:35
Ohh look, I think it's such as probably 12 years ago.
6:39
It has always started with data while measure was in data and AI
6:43
and I've been doing data all my life.
6:45
So I think that's the, I think the journey was on the, I think
6:51
it were different industries but I think what I'm doing now
6:54
Quantum Black, I'm concentrated more on the financial
6:58
institutions. So what I do is that I help
7:01
banks and insurance to get into Guy and Gini and basically put
7:08
products in, in production, right.
7:10
I think 1 aspect of it is how to know the enterprise tackle all
7:15
of the beat engineering with all the different architecture and
7:19
technologies and different maturity levels.
7:22
We're happy to be here today. I think subject is very close to
7:26
my heart also. All right, folks.
7:29
So we know we've got these looms out there.
7:32
And before we get into how Willms are affecting data
7:37
engineering, let's talk a little bit about how data looks and how
7:44
data engineering looks in this new paradigm, in this new shift
7:48
that we have. And maybe on us you can kick us
7:52
off. No, very good.
7:54
So I think as you know like data engineering has I called stepped
8:01
over is a subject is very important to basically get a
8:06
data and give it to a model and then get your size from the
8:09
model. So it always has been important
8:12
from the normal reporting side to the AI when we have that
8:18
prediction models and all of it. And now I think it's way more
8:22
important now with Jenny, right, because previously we were
8:27
touching on structured data and we had solved it.
8:31
We we found solutions to measure the quality, to build pipelines
8:36
and basically to do lineage on the data and to have
8:40
predictability about the output. Now I think with Genai, I think
8:46
data quality is way more important because we are
8:49
basically touching and using structured data.
8:53
I think also the output is also touching a lot of potential
9:00
users in the end. So I think this engine has been
9:03
important. It's still very important now
9:05
and I'm happy to to say also that I think now that
9:13
engineering is also being empowered by the AI.
9:16
So it's not that it is important now, more important, but I think
9:21
now it's also that a lot of our job back couple of years back
9:25
now, it's been very easy or easier using AI.
9:31
Just adding to that as well, like we were talking about here,
9:34
right? Overall on this LLM aspect of it
9:37
and also the UI, UX, right, like like let's take a step back when
9:42
we are talking about data overall, right.
9:44
If you're talking about one company versus another company,
9:47
data is something that's going to be a key differentiator for
9:50
you, which is going to give you a competitive advantage because
9:53
that's what something which is owned by you.
9:55
Other than that, when you talk about LLM or you talk about your
10:00
UEX, right, BLM, everybody is using it as of today.
10:04
UI, UX, you can build your UI but think of an example of
10:07
Instagram, Snapchat or any feature that comes in.
10:10
In one application. You can see where the other
10:13
companies might be, might take inspiration and that feature can
10:16
also pop up in that application. But there's another thing that
10:19
is very important which is data so that can improve the customer
10:23
experience and engagement for you overall.
10:25
Like if you're thinking of an example for retail aspect,
10:28
right? For a retail, you can have a
10:30
customer. Your experience can get more
10:33
personalised if the data is going to be used there. 100%,
10:37
yeah. One thing that you mentioned in
10:39
US, which I think is fascinating, is how we're now
10:43
dealing with so much more unstructured data and how
10:48
unstructured data all of a sudden, well, for a lot of us,
10:52
all of a sudden, for others, it's like it's unstructured data
10:56
has been my bread and butter for years, right?
10:58
But for a lot of us, all of a sudden, it's like we need to
11:01
learn so many new things when it comes to unstructured data.
11:05
A lot of people had to learn what a vector database was and
11:08
how to chunk different things, or how to parse out these, or
11:13
even just like ingest PDF's and what does that look like?
11:16
And I don't know if there's optimal solutions yet out there
11:20
because it is still such a new field for all of these different
11:24
pieces. But what do you feel like?
11:27
Are some of these new paradigms that we've had to learn about
11:32
because of this rise of unstructured data?
11:37
No, very good. So I think it instructed data
11:40
like I've been through the time like 2000, tens of 2008 where
11:46
data leak was the new thing and the promised land of data leak
11:49
was, hey, we can basically store videos and audios and and and
11:54
make sense of it all. But I think until a couple of
11:58
years back, nobody was putting the new structure days into the
12:01
day today because we didn't have the tools to understand the
12:06
video and get insights from it, right.
12:08
So I think, yeah, now I think the Titanic is being used at its
12:15
fullest potential because then we can store, process and use
12:20
unstructured as structured data. So I think that component has
12:24
been with the enterprise for the last 10 to 15 years.
12:27
So that there's nothing to change.
12:29
I think the new things here are all the new technologies
12:34
basically that the enterprise and looking into using now.
12:38
But I think the good news is that they don't have to throw
12:41
anything away. Everything they have is still be
12:43
valid, but I think there are some probably two or three
12:47
additions they need to add to their stack, right?
12:51
So I think #1 is virtual databases because basically if
12:54
you want to build a rack, you basically need to store vectors
12:58
into a database. So I think for the people who
13:04
are basically want to be serious about it, selecting the right
13:08
vector database and using it is key.
13:11
I think that the other aspect is also integration with the other
13:15
Dems because basically there are multiple archetypes offline and
13:19
them online, a bit hybrid. So I think integration with the
13:23
new component then with something new.
13:26
The third bit is a bit the data management that we were doing
13:32
with structured data. Now I think that the structural
13:38
change now is that beforehand we're connecting to our table
13:41
and we would get the insights right and we would also be able
13:45
to measure the quality, recency and all of that.
13:47
Now imagine that we're connecting to a mail server,
13:52
we're connecting to document storage and we're connecting to
13:56
video storage systems. So how do we get that data?
14:02
How do we transform it? How do we store it and how do we
14:08
measure the quality, right. So for example, on the PGA, how
14:11
do we measure that input? There is right on, not always
14:15
recent on that. So I think these are the new
14:17
challenges. Luckily the data management
14:20
companies have, I mean did they have also like discovered items
14:25
recently, but I think they quickly designed and like new
14:31
solutions basically like any 2 bit of data catalogue that you
14:34
can ask it questions and then it will generate a SQL query for
14:39
you or basically you can use the alum and measure if the PDF has
14:45
PYD data, right. So I think these are the new
14:48
things that we see data management companies
14:51
introducing. And if I want to summarise, I
14:55
think nothing to throw away the what stage apiece have been at
14:59
investing on on data management or DTG tools.
15:04
It's still very valid that probably 3 additions that I
15:09
think my clients are looking at #1 is virtual databases.
15:13
I think it's key. And the good thing is that we
15:15
have many solutions of the show or on top of databases they had
15:19
previously #2 is integration with 11.
15:23
How to Figuring out how to make sure that data privacy is there.
15:27
How to make sure that data OPS is there to to send data and
15:31
receive data from the edit them. And the third bit is data
15:35
management tools like the Etos, the data catalogue, the data
15:39
quality tools that were very good at measuring structured
15:42
data. And now we need to look at how
15:46
to how to do this simply within structured data.
15:51
There's something fascinating there that I want to hit on
15:54
because there was actually, there was two things that jumped
15:56
out at me from what you just said.
15:58
And it's almost like up until recently, the data lake was
16:04
where you would put your unstructured data.
16:06
And it was like almost a place where unstructured data would go
16:10
to die because you would throw it there and you would think, we
16:13
need it, we need it, we're going to do something with it.
16:16
But you wouldn't actually ever do something with it.
16:18
And then, yeah, now this came out and it's like, oh, well,
16:22
actually, maybe we can make use of this.
16:23
Now let's figure it out. And then the other one which I
16:27
think is fascinating to ponder about is how we have this new
16:31
paradigm of ETL. And the ETL before where it was
16:35
like how many times did someone click on something that isn't as
16:39
relevant when you're dealing with LMS, it's more how can we
16:43
ingest a PDF and make sure that we can look and get the tables
16:48
correct. And so data quality in that
16:51
regard is much different than the data quality and the regard
16:55
of like are these in dollars or in euros?
17:01
Why is it on the table? It says it's in dollars, but
17:03
really it should be in euros. That's like a different type of
17:06
data quality than what we're thinking about when we're
17:09
ingesting a PDF and we're not able to correctly parse it out.
17:13
So with that in mind, like when it comes to quality, how do you
17:19
think about these things on you? How do you make sure, How can
17:22
you, like, assure that the data quality is still there as you're
17:25
ingesting all of these new forms?
17:27
Or maybe new is not the right word because new is hard to say
17:31
these days, but and these different types of data.
17:37
Yeah, I think that's a very good question.
17:39
Maybe in continuation to that also, I'll just put in one
17:42
thing. There's also a misconception
17:44
about that. You just put PDF in L and you'll
17:46
get an answer. PDF is could kind of be huge,
17:48
right? You have to break it down into
17:50
chunks. People feel like, oh, I've got
17:51
the data, just put everything in NLM and you'll get an answer.
17:54
So there's a lot of preprocessing that needs to
17:56
happen in the background as well.
17:58
You break it down into chunks and then otherwise your cost is
18:01
going to go so high with LLM. Now coming to your question on
18:05
quality, right again, quality, it's got a lot of respect to it
18:09
right when it comes to your data.
18:10
So when you're doing a preprocessing, you have things
18:13
in place where you cross checking there's no missing
18:16
venues in there. Again when you've interpreted
18:19
the data and loaded the data from PDFs, you ensure that you
18:23
have a prompts written which is going to basically validate the
18:26
inputs that you have. Overall going to ensure that
18:30
you're not entering or adding any.
18:32
The quality and privacy is is kind of going also another hand
18:36
in hand which is cross checking you're not using any sensitive
18:38
PII data and if your data is consistent as well.
18:43
So there are a lot of tooling available in the market, I'm not
18:47
going to go there. But again, you can have a
18:50
prompts written in general that can help you in that case, and
18:53
also some other enterprise tooling that is also integrating
18:56
quality in place so that when you're reading that data, those
18:59
cheques can be also applied. Now I I think, I think also if
19:06
you look at the data quality, right.
19:09
So now what's important is that you need to look at the data
19:13
quality as an input, as a prompt and also as output, right?
19:19
So I think I'm not going to talk about the news, but I think
19:22
every week I'm hearing about that shut the box saying things
19:27
that are not correct. Yeah.
19:31
So I think there are many examples.
19:32
I think this week we heard that a good story about like I bought
19:38
about like a channel that company has gone through.
19:41
So I think here we need to be a bit holistic about data quality,
19:45
but we need to look at the input, we need to look at the
19:47
prompt, we need to look at the output and the output.
19:49
As I said that addressing the issues, there's also electricity
19:52
and lots of new concepts, right. And I think also just for the
19:55
data quality. So I think this is a very
19:58
important subject because I had given specifically for example.
20:01
So one of my client is an insurance.
20:03
I basically they were very happy because what they have done is
20:06
that it's a client. Trans business, so basically
20:10
with big contracts with like business contracts, what they
20:13
have contracts written interacts with some covers and some
20:18
details about it. So typically what they were
20:21
looking for to do is with an LLM basically we have a claim and
20:28
then we'll look at the claim and we'll check whether it's covered
20:32
by the policy or not, right. So to help a bit, the, I mean
20:37
the insurance agent to make a quick decision because sometimes
20:41
these contracts can be handed off pages depending on the size
20:45
of the business and everything. So they did a PLC, very happy,
20:48
it works fine. They have checked the value
20:51
stick and they're happy. Let's go to production, right.
20:54
But then I think when we were looking at the production,
20:59
basically the question was, hey, these contracts, they're
21:03
basically scanned and put into a document storage, right?
21:08
How do we make sure that that contract has deleted?
21:11
Because basically if we're helping that agent instead of
21:16
reading 100 page and figuring out what to do, what if we
21:21
uploaded the all we use in the previous version like the two
21:24
years version where that cover wasn't there, the implications
21:28
are huge, right. So I think then we set back on
21:30
the client and figure out how to make sure that the input data is
21:34
recent is correct and is specific to that particular
21:39
customer, right. So I think data quality once the
21:43
wave of do a PUC, let's check the value of our use case.
21:47
Now it's getting worse. He used to take that you see and
21:51
take it to production. And I think as I said, we need
21:54
to look at the three layers of the tree steps, the inputs, the
21:58
prompts where we have the chart history, where we have are
22:01
basically the the context setting and also the output is
22:05
key to make sure that more to production Is is a bit secure,
22:10
yeah. Angry.
22:12
And I think because of these issues, data engineering is not
22:15
going anywhere. I think it's gonna get more
22:17
important now. Yeah, I agree with you 100% on
22:21
that one. That feels like a what every
22:25
data engineer knows, because if it were going somewhere, I think
22:30
the data engineers workloads would be lightened quite a bit.
22:34
But if you know data engineers or you or you are data engineers
22:38
and everybody else out there that knows data engineers knows
22:41
that it's been the opposite. Like every day time.
22:44
Every data engineer I know has been working their ass off since
22:49
these elements came out because it's just like there's so many
22:52
different things that you need to start thinking about and
22:54
knowing about now and tail being covering for.
22:58
Like us, you're talking about this idea of alright cool, we
23:01
have this MVP that works, but then when you go to production,
23:04
there's all these small details that you need to be thinking
23:08
about where you need to be thinking.
23:11
How are we continuously updating this so that if anything
23:15
changes, like somebody upgrades their plan, we make sure that
23:19
this upgrade gets added to the context.
23:22
So all of that context, how are we making sure that it is
23:26
continuously minute by minute or second by second, depending on
23:31
how real fast we want to make it?
23:34
How are we making sure that gets up to date?
23:36
Yeah. Ohh, I think like I agree with
23:39
you, like I think they just need is going nowhere.
23:41
I think it's going to be more and more important and I think
23:46
also the Daily News are also trying to adapt with the new
23:51
business requirement and try to basically use these tools also
23:55
to help them move quickly, right?
23:58
Yeah, basically no complex and other challenges.
24:02
Exactly, exactly. Because now that we have these
24:05
different areas that we are cognizant of and the data
24:09
engineers knowing that, OK, well, if this is possible for
24:14
insurance claims, could it be possible for the work that I'm
24:17
doing too and how can I make my life easier?
24:21
And so I know there has been a lot of speculation on how LM's
24:24
will affect things. Like we've seen a lot of text to
24:29
Sequel models comes come out. And I kind of laugh about those
24:35
because it's like every data engineer knows that writing the
24:39
SQL query isn't necessarily the hard part in their job.
24:44
But I would love to hear what are some areas that you have
24:47
seen or think that these LM's could be used in the data
24:53
engineering sphere. Maybe I can add from few
24:56
elements and then as you can pitch in, of course I'll talk
25:01
about the topics. And then of course there are a
25:02
lot of guardrails around it which is regulations gonna be
25:05
taking place. But let me start with a couple
25:07
of things when it comes to writing the data pipelines
25:10
altogether. Right.
25:11
You can use a generator to help you with the data pipelines
25:16
altogether. So which can help in stitching
25:18
the code altogether. The other things could be unit
25:20
testing, which is another important criteria where you
25:23
generate the code altogether. There unit testing can be
25:27
helpful Where Jenny I can help in the word in in my scenarios,
25:33
I've seen it in synthetic data. So when you need like a
25:36
synthetic data that needs to go in your integration testing,
25:40
that mock data can also be generated based on that
25:43
scenario. I've not used it, used it but it
25:47
can also help further on classifications of your PII data
25:51
which is tagging and also doing the data cataloguing altogether.
25:54
But having said that, you have to put a lot of guardrails
25:58
around it. There's an IP infringement.
26:00
You have to be very careful. You can't just copy paste the
26:02
code and you say OK, my work is done.
26:04
So and with European AI Act also coming up, there are so many
26:09
regulations that we have to be very careful with also when
26:11
you're using Gen AI, do you think that is?
26:15
Yeah, no, no, that's like we had that percent.
26:16
So I think this is a great era because I think the the Lambs
26:20
are also making decision near job easier on some aspect.
26:25
So I'll take also like ingesting the alright, so imagine that you
26:29
have a lot of content and previously like just to give you
26:34
for example on like on a like for his upper bags and shoes, we
26:40
work a lot of scanned documents and PDFs.
26:43
So I think previously to go and get the exactly the data that we
26:47
need that PDF you had to draw a metrics and pinpoint the pixels
26:53
on which to get that data. I think it has evolved with time
26:57
with natural language processing, but I think now it's
27:01
as easy. Of course with the right guard
27:03
was as easy as plugging in the PDF with the different methods
27:07
and as I told them to get the insight or summarise or get what
27:12
the value that you want, right. So I think that's the data
27:14
ingestion. I think #2 which is that it just
27:17
cataloguing, right? So basically enterprises, they
27:22
have a huge data catalogue and in order to navigate it and to
27:26
figure out what table goes where to join, how to join.
27:30
I've seen a client also experimenting with like instead
27:37
of yeah as you said, like writing a query in natural
27:41
language like say hey give me the last 10 elements sold in
27:46
this branch in that area. And basically for new users, for
27:52
users who don't know how to write the SQL either needs help
27:55
and needs support. I think the system can help tell
27:58
you which tables to look at or even help you create the query
28:03
or get you the value. Either of that data.
28:06
I think we also like tagging data or showing you where
28:13
there's a potential data quality issue.
28:15
I've seen, I've seen some general, we have little Quantum
28:19
Black, a tool that basically help you look at the table or
28:24
multiple tables until you where are the potential data quality
28:28
issues and to try to help you fix them traffic also the the
28:35
lineage, right. So I think also data elements
28:39
can help you find out where the data is coming from, right.
28:44
So I think look, I think it's a pleasure to be live in this year
28:50
right now to see that basically engineer can also have the
28:54
engineers move faster. I don't think better them can do
29:00
all that decision hear does or can be basically replace right,
29:05
if this is the right word. No, I think, I think it will
29:08
help support. It's not Accelerate, it will
29:11
help materialise the work of decision here quicker.
29:16
And maybe another thing to add here is Ohh element model.
29:21
They take no for an answer. They always have an answer for
29:24
you. It can hallucinate and give you
29:26
some answers that you might not even which is not even true.
29:29
Maybe the data does not even exist.
29:31
But I'll give you an answer. So you have to be very careful.
29:34
You can't just say blind, you can go ahead as it has mentioned
29:36
and it's not going to replace you.
29:38
And of course the guardrails and regulations are in place which
29:42
can you can use where the data needs to go.
29:44
Where is the LLM hosted? They're going to be important
29:47
comments anywhere. Yeah, I I was thinking about
29:50
something along the lines of, but it feels like a lot of times
29:54
potentially you don't necessarily need an LM for this.
29:59
And I think these days it's it's easy to be like, ohh, yeah,
30:03
let's throw some AI at it. And you know, it's like you do
30:08
you need a I for that? Because I was thinking, ohh,
30:10
yeah, well, what about if you could figure out like the how
30:14
the data has been transformed and the provenance of the data?
30:17
And then I was thinking about it a little more and I was like,
30:19
you don't need AI for that. So I'm going to change gears a
30:22
little bit because we were talking about PPI a little bit
30:24
and tagging that. What do we feel like when it
30:30
comes to LM and working with LM's from the data engineering
30:34
side? How do you think about PII and
30:38
data privacy? I know we've, we've spoken a lot
30:41
on this podcast about how you really have to be conscientious
30:48
of where you're sending the data, especially if it's just
30:50
going out to ChatGPT and how ChatGPT is is using that data,
30:55
even if they say they're not using it.
30:58
I was just reading a paper actually a little while ago last
31:03
week from these folks out of university in Prague, and they
31:08
were talking about how all of the different large APIs leak
31:13
the data into some way shape or form.
31:17
They're leaking that data, so you can't.
31:19
Even if they say that they're like bulletproof, it doesn't
31:22
matter because it's being leaked and so you can't really trust
31:26
it. How do you look at data privacy
31:29
and PII in this new paradigm of large language models?
31:33
So in terms of privacy, right, it is the data that you have
31:37
within your company, you can control it.
31:38
So one, I always have this question for my clients whenever
31:42
we are building applications or anything, right?
31:44
Do you really need to feed in the actual PII data into your
31:47
model? And 9095% of the times it's no,
31:51
you can anonymize it. You don't have to send a
31:54
customer ID or a customer number of bank card details, right?
31:57
You can anonymize it, need that information in your model and
31:59
that's where you can use it. So I think it's always about
32:04
before you even use them. And rather than throwing all the
32:06
information to the model itself, take a step back and think, do
32:09
you really need to feed in the sensitive data in the 1st place.
32:12
So majority of the times you what you would typically do is
32:16
you anonymize the data altogether, so you don't even
32:19
send that data in in that particular sense in the 1st
32:22
place. So you do that tokenization,
32:24
anonymization or hashing or whatever which is going to be
32:27
the most secure mechanism. Ohh for this depending on the
32:30
type of data that you're gonna put in Ohh that's going to be
32:33
the point. Number one, you don't send the
32:35
data in the 1st place and do those guardrails altogether
32:38
while making a call but five personally we still need to send
32:41
that data and so on and so make sure where is your LLM hosted
32:45
like now you can also have country specific right?
32:48
You may be data from 1 country doesn't need to go to an LLM.
32:52
Think about where that that LLM is hosted, where the data is,
32:56
where it's going to be going and also ensuring the authorization,
32:59
the excess of that. At the end of the day like
33:02
application does not have flexibility that it can download
33:05
all the data altogether. So you also take care of that
33:07
authorization aspect which we have not talked about yet, but
33:10
access management to ensure that only a certain set of people
33:13
have access to that set of data if required.
33:17
No effect. Also whenever there is something
33:20
new, a lot of people are worried about it, right?
33:24
So I think with with the recent McKinsey like resources that we
33:31
have done a lot of the companies we have discussed with like OHH
33:36
thinking like more than 70% of thinking that Junior right has
33:39
introduced or will introduce some data risk, right.
33:44
So that needs to be considered right.
33:46
So I think this is the same story all over again like when
33:49
we had cloud and everybody was saying hey, how can I be sure
33:52
that my data is going to be seen by somebody else or my data is
33:56
not leaving or is it secure. So I think we need, it's a
34:01
maturity level that enterprise will basically give a trust or
34:06
trust that process. But I think go in there, there
34:11
are a lot of different like archetypes or different
34:16
solutions that at least my clients are looking at beautiful
34:21
banking rates. So I think as Andrew said,
34:23
there's one we don't really need to send anonymized data to the
34:27
other them because basically you can put all the our property
34:31
data into a vector database that is hosted within our premises
34:36
and will use the Genai within them just to interact and
34:40
retrieve that data, right. So, and even though the
34:43
retrieval of that data, we can make sure that on the vector
34:46
database, we don't have any pussy, right and we use the
34:49
context, what we use the client information audio on the front
34:54
end. So that's I think one of the
34:56
series anonymized data #2. There's potentially also hosted.
35:01
And so I think now with the introduction of Michael and Lamb
35:06
and many offline Adams, a lot of enterprises are testing.
35:12
Of course, I think there's always the question about the
35:16
GPU or the accuracy versus the the leaders on the market, but
35:21
for some applications they are. I think there's no question that
35:24
the data will leave the premises of the company.
35:28
And I think there are also this affected the third bit with
35:32
which is a bit the, the, the, the hybrid approach to as as I
35:35
said previously which like 11 is for just interaction, but the
35:40
data is hosted 100% within the premises of the of the
35:45
enterprise, right. So I think there's different
35:47
archetypes, many players, artists and different things.
35:52
But I agree with you that it's a maturity level.
35:55
I think as we have gone through this with the cloud where nobody
35:59
is trusted to send data elsewhere, now it's more like
36:05
everybody uses the cloud, right. So I think, yeah, yeah, I think
36:09
somebody tweeting level, we need to get into good time.
36:13
Yeah, that's an interesting point where you draw the
36:17
parallel between the cloud. And I can see how back in the
36:20
day, people were very afraid of saying, like, wait a minute, I'm
36:24
going to go and I'm going to take everything that I own and
36:28
just give it to somebody else. And you're going to assure me
36:31
that it's all gonna be OK. And what kind of assurances can
36:35
you make that it's just going to be OK You're just going to give
36:37
me your word on that one? OK Yeah.
36:39
And I'm. I'm going to go and spend a lot
36:41
of money to do that. Let me see about that.
36:44
So there there does feel like it and there are some parallels.
36:49
It feels a little bit different though, because we don't really
36:53
know what the LM's are doing. We don't really understand
36:57
what's going on. Yeah, it's a little bit more
37:00
like, OK, I can open a I can tell me that they're going to do
37:04
everything in their power to make sure that nothing happens,
37:08
right? But then until we fully
37:10
understand the idea of what is actually happening in the LM and
37:15
if anything is going on behind the scenes, then I I think we, I
37:23
I feel like people will still be more hesitant and it is more of
37:26
that maturity level that you're talking about in us.
37:28
Like we're going to over the next couple years.
37:31
We're going to be able to see where things fall short and then
37:36
we're going to be able to create whatever we need to create to
37:40
make sure that everyone is very, very confident and we can put in
37:45
our little sock 2 compliance. That even though I'm using this
37:49
API, it everything is still all good.
37:52
I still conform to my sock too, and it's it's all kosher, right?
37:58
So thinking about this whole idea of how do we move fast, How
38:05
do we take advantage of this technology that's out there.
38:09
But at the same time, like how do we not shoot ourselves in the
38:12
foot and be one of those companies that is in the news
38:17
for having a gaff or selling a car for $1.00 or talking bad
38:23
about its own company, all those different ones and us that I
38:27
know you, it's funny, you don't even have to tell me which one
38:30
you are referencing. I can just like think of three
38:33
or four because every week a new one comes up and it's it's like,
38:38
ohh yeah, those poor folks, they just tried to run fast and it
38:43
ended up shooting themselves in the foot.
38:45
So how do you all think about like balancing that innovation
38:49
and trying to make things work with these LM's but at the same
38:55
time having this data protection and having these guardrails in
38:58
place and and maybe it's that you're thinking, OK, well maybe
39:04
for very high stakes stuff it's off limits.
39:08
Or maybe we try and just do things internally in the
39:13
beginning and until we see that we have enough reps internally,
39:17
then we can bring it to the external world.
39:19
Even though as soon as you put something external, you know
39:23
it's gonna be like there's so many more malicious actors just
39:26
because they want to have something to post on Twitter for
39:29
no other reason than that they wanna be like look at what this
39:31
chat bot told me. So how do you all think about
39:34
that? Like that balance right there?
39:37
Look up, I think that the different ways you can do that.
39:41
So I can talk about examples of some companies or what they have
39:46
done. So I think one approach was to
39:49
say, hey, this is a new technology.
39:52
This is moving really fast and we don't know how much it's
39:56
going to cost us down the line. We don't know like how much the
39:59
value will be. So that's the one thing.
40:02
Let's take the use cases. Let's privatise them.
40:07
By values, what will be the impact by feasibility?
40:11
With feasibility we have also the risk, right?
40:13
And I think when you do that exercise, most of the time, like
40:18
at least some of the bank's banking clients I'm working
40:22
with, we try to do something either internally, right.
40:26
So let's say it's hour chat box that we used to ask questions
40:34
like how many days do I have leave a year.
40:37
And you limit a bit intent, right?
40:39
You start with the intent, you limit it, and then you expand it
40:43
but with the control, with the control environment.
40:49
I think the other way to do it is basically to to launch a
40:55
product with a limited set of users just a bit, fail fast and
41:01
then build build upon it until Because I think there are
41:05
probably 3 things you need to guarantee. #1 is that you
41:09
basically know that this would provide value to you.
41:13
So number one you to validate the value.
41:15
Two you need to validate that you master the technology and
41:18
you can basically specify the intent, how the guardrails and
41:23
make sure that the output is clean.
41:26
The top bit is also to master the costs because I think a lot
41:31
of times we say that the general I will help views a bit the
41:35
costs of doing XY&Z but I think we need to a lot of companies
41:41
need to prove that right. So I think a lot of especially
41:45
the backlights I'm working with even though many banks have
41:49
launched some chat box, but if you go there, it's see there you
41:53
can see the it's either they started previously with the
41:57
internal beasts and they visited the the technology and the risks
42:02
too. They have launched this
42:04
previously but less limited set of customers so they can test or
42:09
potentially all potentially. What they have done now is that
42:16
they really limited the the, the intents of what they launch
42:21
online to some specific actions, right.
42:24
But of course I think one important piece is is anyhow you
42:29
need other than putting the gather rails, is that you need
42:33
to add some human touch in the end to validate a little bit
42:37
some of the outputs, right? Because I don't think everything
42:43
it shouldn't go out and should be autonomous right?
42:45
I think they should be a lot of control over the outputs at
42:49
least for the for the maturity level.
42:51
Well, here we are in within this couple of years.
42:55
Yeah, adding to the points that industry just mentioned, right,
42:59
like summarising in the context of these three things, right,
43:02
your privacy, quality and access like this is going to be very
43:06
important guardrails that you have to apply.
43:08
Things are still evolving. I'm not saying we are there
43:11
there yet, but still these three components, depending on your
43:14
the risk factor of your application, you make sure that
43:17
your quality, privacy and access is taken care of.
43:21
I think that's where you will help as a client or as a
43:25
customer can help a little more in going in that journey.
43:28
Other thing of course is a country regulations we are
43:30
coming up like European AI act, right.
43:32
Other countries are also going to come up with those
43:35
regulations, which is going to drill down on the list of things
43:38
that an application need to be careful with.
43:40
That is going to be similar to what we had for GDPR earlier,
43:43
right on the AI application that's going to be also the
43:46
guard rails that will be enforced at the country level.
43:49
But as of now if you don't have those things, make sure that
43:52
based on how the risk factor for application is, you take care of
43:55
the privacy fixes and overall your ohh this these three
44:02
components that I talked about here.
44:05
Yeah, it feels like there's almost, so I really like that.
44:08
It's like a a matrix of the risk factors, costs and how easy it
44:17
will be to implement so that you can recognise.
44:21
Should we be putting this or should we be spending our time
44:24
on this first of all and Next up like if we do decide to spend
44:29
our time on it, what's the likelihood of success here?
44:32
And then also having those feature flags in place so that
44:36
you can quickly test it with a small group and if it gets out
44:39
of hand with a small batch size of 100 people, you can recognise
44:45
that. Definitely not gonna be showing
44:46
this to 100,000 people like let's go back and see and and
44:50
also and then limiting that ability of what it can do and
44:54
putting those guardrails on on the front end.
44:56
So how the person that's interacting with that in this
45:00
case the HR chat bot, how that person can speak to the HR
45:04
chatbot and then what the output of the HR chat chat bot is too.
45:07
So you can't say, hey give me my my holidays package in a poem in
45:15
the style of, you know, Bob Dylan or something, which would
45:20
be great, but it's like ohh, do we need, do we need that right
45:23
now? The other piece that I thought
45:25
was interesting that you mentioned in US and I'd love to
45:27
dig into this a little bit more, is like people feel like when
45:32
you implement LM, it's going to reduce the cost a lot of the
45:35
time, but it's not always a given that thing the cost will
45:41
go down. So can you explain that a bit
45:44
more and? No, no, very good.
45:45
So I think while while to say by that is that basically if we
45:50
choose case and we have seen a lot of risk cases whose engineer
45:54
I you need to you need to understand what is delivery you
45:58
are trying to optimise. Is it basically tried to serve
46:02
millions of clients and is it to retrieve information quickly or
46:08
is it something else? What's the leave we are trying
46:10
to optimise? I basically for each of these
46:12
use cases we need to get a dollar value, right?
46:16
But because each of these, and I'm talking about the
46:20
enterprise, about the back, about insurance, in order to get
46:23
this into production and answering the questions about
46:26
privacy, regulations, all of it, it's going to cost money, right?
46:30
So I think before you get into something that costs a lot of
46:36
money, you need to understand what's the value of each use
46:39
case, what is the feasibility and then to take what is has the
46:44
highest they do and a highly feasible with less risk, right.
46:48
And you start with that, you try to pilot it, you launch it, you
46:54
get the value out of it and then you understand you will
46:57
basically master the cost. You understand how much it's
47:01
costing you. Because I think a lot of people
47:04
now are looking at the build they receive from Julianne
47:08
asking questions to if we also need your people to be trained
47:15
on how to through this use cases.
47:17
So I think within that you guarantee that you are limiting
47:21
the risk, you are getting value and at the same time learning
47:25
how to do more complex, more high stake use cases.
47:29
So I think for me this is critical because I think, I
47:32
mean, I always like to go back as a model.
47:35
I like to go back into history and to see where companies have
47:39
filled in the past where they say, hey, we have this machine
47:42
learning, OK, machine learning, let's build a model that's gonna
47:47
do this, finds it, and then they go, they build it.
47:50
There's no uptake from business because where's the value,
47:53
right. So I think now with the hype, a
47:57
lot of companies are trying to build a new things.
48:01
So for example, on that example, I was telling you about a
48:04
company trying to build layer that's going to generate a SQL
48:10
query for you, understanding the data catalogue and the data.
48:14
I think the company has spent a lot of money to do it, but now
48:18
data management companies are offering that on top of their
48:22
database, on top of their databases, right.
48:26
So now it's becoming, so you invest a lot of money, but a lot
48:30
of companies are offering that off the shelf, right?
48:32
So I think starting. If a company wants to start
48:35
tomorrow, they need to look at all the laundry lists of use
48:41
cases. What is the?
48:42
Never tried to put a dollar value into each 1A feasibility,
48:47
prioritise biometrics and then go ahead by the highest value.
48:51
High visibility right test it, see the value of it, see if
48:56
there's any uptick from business and then skip from there.
49:00
God, that's so right, man. And that's so, so hard to do on
49:04
that strategy part to think about.
49:07
I could spend a lot of time and effort creating this because
49:13
it's not on the market right now or I could wait six months and
49:17
see if it's going to be on the market because I'm probably not
49:22
the only one thinking about this.
49:23
And so that question right there, it just, it eats at me so
49:29
much considering that you have to have like this crystal ball
49:37
where you have to recognise like her, OK, we could go and spend a
49:41
lot of time and energy in this or we could just wait and what
49:45
are you gonna just wait on? And then in six months, what if
49:48
it's still not there and you say, you know what, we waited
49:50
six months, it's still not here. And then in another six months
49:53
it is there. And so you're like, dah, weird.
49:58
But so that is is so hard. So anybody out there that's
50:01
making those kind of decisions, I have a lot of respect for
50:04
them. Yeah.
50:06
And yeah, I I I heard one example like there was somewhere
50:09
I read where people were talking about if I can just message chat
50:13
in a video gets created out of it.
50:15
So a lot of ideas are being thrown like that in the market,
50:17
which is like I just explained. And then I get a complete video
50:20
for three minutes 4 minutes on the topic.
50:23
I think this has been lost a couple of days ago, right.
50:25
Is it? Yeah.
50:26
Yeah. See the things are moving.
50:29
Yeah. Exactly.
50:30
Yeah. Yeah.
50:31
Soon enough that'll be a reality.
50:34
And that's so wild to think about.
50:35
The wild. So this is incredible.
50:38
Well folks, I am so thankful that you came on here and you
50:43
were chatting with me all about the data engineering aspects of
50:47
this LM world, and it's always a pleasure to get to catch up with
50:52
smart people like yourselves. Thank you very much.
50:54
Dimitri is a pleasure.