Latest / Agentic Conversations (formally mlops.community) / A Decade of AI Safety and Trust // Petar Tsankov // MLOps Podcast #218

Transcript

A Decade of AI Safety and Trust // Petar Tsankov // MLOps Podcast #218 · Agentic Conversations (formally mlops.community)

0:00
Hi, my name is Better Jankov. I'm the CEO and Co founder of
0:05
Lattice Low AI and I drink my coffee.
0:09
So it's not a special kind of coffee.
0:10
I drink espresso, but it's interesting how I drink it, so I
0:13
always get up and drink my first coffee very fast.
0:17
I think of it like, you know, 2 minutes and then immediately
0:21
then jump on the second one. So that's how people recognise
0:26
me if the way I drink my coffee. What is up, MLP's community?
0:31
This is your host. For the next hour, I'll be
0:35
taking you on a little bit of a journey.
0:37
My name is Dimitrios and it is a pleasure to be here with you
0:40
talking with Cedar today about all of the things that go into
0:46
AI Trust. Wow, what a deep dive.
0:49
And I've got a few key takeaways before we get into this
0:52
conversation that I want to bring up and highlight for you
0:56
so that hopefully they stick in your mind as we cruise followed
1:00
by them in the conversation. The first one is how AI research
1:05
has traditionally been about optimising one metric, and that
1:09
metric has been accuracy. But now when we take these
1:13
models into production, there's so many wrenches that can get
1:18
thrown into the system and so many X factors that we need to
1:22
be cognizant of. And that is what Peter has been
1:26
thinking about day in, day out. How to really make your system
1:32
bulletproof, make it robust, make it all that it can be.
1:38
I want to reference a song right now, but I'm not going to.
1:41
The other thing that I will bring up as a highlight that I
1:44
felt like was pretty spot on was how you cannot decouple models
1:52
from data. It is almost like a trope that
1:56
we've gone over time and time again.
1:59
But your model is only as good as its data.
2:03
And he was speaking about how in lattice flow, the product that
2:09
he has built, they are very, very interested in everything
2:16
that happens upstream from the model because you're not able to
2:23
create the best systems without being able to go upstream and
2:28
make sure that the data is solid.
2:30
O if he had this really cool feature that he spoke about
2:35
where you can see the model blind spots so that you can
2:40
create a more robust system. And with lattice flow you can
2:45
test which data it just does horribly.
2:50
With my motto, basically, how can I figure out where my model
2:54
is doing bad? So then I can go back upstream
2:58
and give that motto what it needs when it comes to data so
3:02
that it won't whip it so hard. This was an awesome
3:06
conversation. We got into all things trust and
3:09
a I I trust I think is the word of the day.
3:14
And I really appreciate Lattice Flow coming and being a huge
3:20
supporter of the community. If you are interested in this
3:24
conversation, hit up Peter. And as always, if you like it,
3:30
share it with just one friend or leave us a review.
3:33
It would mean the world to me. Let's get into conversation now.
3:42
We have to start with the fact that you, the last time we spoke
3:49
to you, were in Switzerland. Now I think you're not in
3:51
Switzerland. That's a good idea.
3:53
So we started expanding a little bit.
3:55
So we did recently opened a second office in Sofia which is
4:01
it's just appearing in Bulgaria which is where I'm from and it
4:05
was you know not a random decision, so it was quite
4:08
strategic. So one of my cofounders actually
4:11
Martin Michev, so he has a really, really big initiative
4:14
here in the region. He raised 120 million to
4:18
basically bring world class science into Eastern Europe and
4:22
he's doing this quite successfully here.
4:23
So you know we're thought well that's a great idea, you know it
4:27
Lattice. So I we have very cool deep tech
4:29
topics. So you know, we have to have an
4:31
office right next to this structure.
4:33
So basically, we're now literally on the same floor with
4:36
all the researchers in the research labs and they're
4:39
working on these topics. And have you seen any tangible
4:43
outcomes already? Not yet, I will say.
4:48
Still the vast majority of machine learning team, you know,
4:52
skills. There is something that we based
4:54
in Zurich which is just you know, purely by realising that
4:59
Zurich is one of the for me the best place in the world to build
5:03
deep tech teams, especially focusing on machine learning.
5:06
So not this kind of outcomes yet, but we hope that's coming
5:11
very soon. So again, strategic, so it's an
5:13
early investment and this will also come in the future.
5:16
Yeah, we'll talk to me about Zurich because I know that
5:18
you've had some experience working with the very famous IT.
5:25
I don't even know how to pronounce it like ETH Zurich or
5:28
ETH. ETH ETH zero.
5:30
Yes. So I mean, I got into Zurich
5:33
after, so I did spend some five years in the States, you know,
5:37
to study, to focus on computer science.
5:38
And then I did want to come back somewhere in Europe.
5:42
They're not that many good options.
5:43
So it was kind of had to choose between London and go to
5:46
Imperial, Imperial College London or Zurich, but, you know,
5:51
fantastic places to study and everything.
5:54
But Zurich wanted me as a city, you know, just very well
5:56
organised. And then when I got to Zurich
5:59
and they did my master PhD focusing on security,
6:03
reliability, I did have a pretty the clear focus to go back to
6:08
the US. I was kind of missing this, not
6:12
randomness, but yours is a little bit wild.
6:14
So there's like a broad spectrum of opinions and everything.
6:17
So there everything is extreme, right?
6:19
And like growing up also in Bulgaria where she also has
6:22
quite a lot of extremes. So I was missing a bit of this
6:24
in Switzerland, but again, Switzerland is just amazing in
6:28
terms of building teams. So right now I I'm kind of stuck
6:31
there which ability to also commit to Sofia, so that's also
6:35
nice. Yeah, yeah, yeah, you.
6:38
You wanted more chaos that you couldn't get in Switzerland.
6:42
But if you got, if you grow up like this, I think, I think
6:44
that's actually important also for entrepreneurs, you know, for
6:47
creating new things. So you do need this kind of
6:50
randomness, you know, crazy ideas that are floating around.
6:52
So they do help kind of, you know, I think they do they, they
6:57
do help, you know, come up with like the next big thing, so.
7:00
Yeah, yeah, the diverse opinions, definitely.
7:04
You think outside the box and talk to us about AI safety.
7:08
I know you've been working on it for a while.
7:11
Yeah, almost 10 years now. So it's been a while.
7:12
So, yeah, so a safety broadly or trust a trust I prefer better is
7:18
the umbrella term. So the main story kind of how
7:22
everything happens in the in the safety space I would say we have
7:27
seen basically a whole decade of really having AI research being
7:31
focused on improving accuracy. And this was driven by all these
7:36
golden datasets like image that for example, you know where you
7:40
would kind of fix the data and just get the models to get to
7:43
make them as accurate as possible.
7:45
And that's what's has been the focus for decades actually even
7:48
pretty pretty deep learning. You know, we were trying to
7:50
optimise this and kind of then what happened is well, we
7:54
figured out how we can really get highly accurate models like
7:58
on this once you fix the data and then everybody got super
8:02
excited to say, ohh, that's great.
8:03
So you know, we have super, you know, we can build, you know,
8:06
vision, speech, language models are superhuman.
8:09
So now we're going to crack all the problems around the world
8:11
and you know this build this super intelligence models to
8:15
solve everything And that's what also safety kicked in because
8:19
it's a very big gap. There's a very big gap of you
8:21
know building something that works on your data in the in the
8:24
lab to something that you actually then deploy to solve
8:27
our mission critical tasks. And it actually works well
8:30
reliably in these new environments.
8:32
So that all these topics kind of reliability meaning that it's
8:36
not just accurate on your data but also as you know the
8:40
environment changes or something else changes the most kind of as
8:43
graphs and learn these kind of goods features to to solve the
8:46
task in these environments, you know then that's of course
8:48
becomes a big topic because the world changes in the deployment
8:51
environments are different safety also.
8:54
So, you know, so a lot of these models, especially if you deploy
8:57
them to so mission critical tasks, which in my view is where
9:02
the big value of AI is, right. You know, we don't want to be
9:04
solving, you know, tiny problems.
9:06
We want to solve problems in, you know, in business, in
9:08
medical and financial services and so on.
9:10
Then a lot of those models, they've affect people's lives.
9:14
So then you know, how do you make sure that also the
9:16
decisions don't harm people. So kind of all these kind of
9:20
hitting everybody and and so on so.
9:23
So if I'm understanding this correctly, it's very much about
9:27
how we've been optimising for one metric.
9:30
Accuracy example yeah the main thing that kind of the data set
9:34
was fixed and then you're optimising kind of kind of make
9:37
how to make it highly accurate meaning that your models are
9:40
giving you the expected outcome on this data than to really
9:44
going beyond this right so. Now that we don't have the fixed
9:49
data sets anymore. Yes.
9:51
So it's it's a whole different, whole different set of
9:55
challenges and even space, you know, I think if you talk
9:58
broadly about trust expands beyond the speed, reliability of
10:01
these models. It's also about ethics.
10:05
Are there biases that we don't want to have?
10:06
I mean, we are about 27 February now, so we are.
10:11
My feed has been full with a lot of anger from the world about
10:16
the Foundation models that are popping up so.
10:19
Ohh the good old. I guess you know following SO.
10:22
Yeah, that have been produced. This image generator has gone
10:26
awry and people are up in arms about it.
10:30
So that's definitely there's some things there and and I
10:35
think that's almost like a good case study as to trying to make
10:39
things too politically correct and too safe in a way like so
10:45
that you over optimise or you almost like overfit for these
10:51
other types of like diversity metrics.
10:56
It's especially I would say when you talk about like using other
11:00
models which is something that we see like more and more
11:04
remaining that's you know it used to be that's a you know,
11:07
machine learning was this task where you you know a company or
11:10
whatever it is, you would gather your data and start start
11:13
playing in the lab and building models.
11:16
But now things are transitioning to 3rd party models.
11:19
So you have a bunch of good models and you know you start
11:22
building applications on top of them.
11:24
So they don't have the foundation models to do some you
11:26
know they could be some model custom specialised models.
11:29
And like in the case with the you know the Gemini in the then
11:33
like if somebody has built this model and you lack some
11:37
transparency on how this model was trained, what kind of you
11:41
know, biases and everything was was was used to produce it.
11:45
Then I think people would appreciate if they know if they
11:49
expect this, but if this is coming unexpected then that's
11:53
really you know, a no go for I think many, many users around
11:56
the world. And this of course contributed.
11:58
I would say a loss to this whole AI safety space because this is
12:02
now you know not about a company deploying a model that's you
12:07
know doesn't have the right kind of accuracy and under delivers
12:10
in terms of some business metrics.
12:12
You know, this is about end consumers that are playing with
12:15
these models and just pure disappointment of what's going
12:17
on. Yeah.
12:18
And that goes back to that trust piece where it's we can't even
12:22
trust it to generate an image that is.
12:27
Yeah, well that's actually correct.
12:29
How can we trust it on these very, very important problems
12:34
and life changing issues like you were talking about?
12:38
This date. And so how do you feel like
12:41
those types of things can be combated?
12:46
Is this just being more public, being more out front with it?
12:51
If there are signs in these models that are generating
12:55
different types of biases and making sure that you get in
12:59
front of it, or is it? Try not to have that happen at
13:03
all. Yeah.
13:04
So maybe I'll take a step back and try to capture like what is
13:08
like what is the main problem in a in a way in the AI safety and
13:12
trust reliability to to address. And while I do, I do think these
13:17
like models that are making quite a lot of news today
13:20
because again because they do touch the end consumer and
13:23
everybody can see the mistakes and you know, get really upset
13:26
about this. There's still the vast majority
13:30
I would say in terms of like pure problem to solve is a big
13:34
part of it is in supervised machine learning, classical
13:37
supervised machine learning models where you would build
13:41
custom models to solve really really well given task you know,
13:45
could be in the bank, could be in medical, could be another
13:48
application. And I would say that while it's
13:51
really interesting also to solve the, you know, the biases and
13:54
the security and other products in the generative models and
13:57
we're sure it's something you know, big topic to address there
14:00
is right now also we do see this from kind of from an industrial
14:04
point of view, a very big push to also so the safety,
14:08
reliability issues in the classical setting because a lot
14:13
of these classic kind of classical so and why this
14:16
happening. So the reason is that, so when
14:18
you do build these custom models, supervised models to
14:21
solve a specific task, they're very useful because they
14:25
actually they drill on autopilot, you just run them and
14:27
then they solve something end to end, just like I have to read
14:30
out but then kind of optimise my performance as a human.
14:34
So they're really, really valuable and very often also
14:40
very business critical because they solve you know very, very,
14:43
very important tasks and still reliability challenges.
14:47
There are just tremendous meaning that you know we're
14:50
talking about like definitely more than majority of the models
14:54
just not making it into production because of because of
14:57
these issues, so. For for, it's.
15:00
It's funny you say that. One thing that I've been
15:02
noticing is that there is so much interest in generative AI
15:09
that it's almost like the traditional ML systems have been
15:13
thrown out the window and people that are doing traditional mill
15:19
have kind of like taken a back seat.
15:21
I have a friend who is hiring for ML engineer and they aren't
15:27
doing any type of generative AI and he said it's so hard to find
15:33
someone that wants to work on the problems that we are working
15:36
on because everybody is trying to work on generative AI
15:41
problems these days. Yeah, I think that's true.
15:44
So that's we also see it, you know, people want to work on
15:47
specific topics, you know, especially the ones that are
15:49
right now very, very trendy. And I mean don't don't get me
15:53
wrong. So you know, there's definitely
15:54
very high value. These are really impressive
15:56
models. They're very useful but it's not
16:00
to underestimate also where the value is.
16:02
And so typically that's I think especially when it comes to new
16:07
technology. So it always at some point but
16:09
it was down to the value, right. Like why am I doing this, why am
16:11
I investing hundreds of millions and billions like what am I
16:14
getting in return. And the reality is that indeed
16:18
you know that's where a lot of the value is and it will be you
16:21
know in the next in the in the coming years.
16:23
So I think that's what's kind of slowly not come back, but you
16:26
know, there will be this kind of realisation that's, you know, we
16:29
really have to grasp this technology for also for
16:32
supervised machine learning and figure out how to well to either
16:36
reliably deploy these models because otherwise we will just
16:40
miss out so much, you know, if we don't create this problem.
16:43
Yeah. Now, there is this feeling that
16:47
I've had that because of the explosion of generative AI and
16:52
specifically like ChatGPT and Sam Altman going and testifying
16:56
in front of Congress and the Hugging face CEO, I can't
17:00
remember his name, but he also went up there and next thing you
17:03
know, it's like AI is on everyone's minds.
17:09
Yeah. Do you feel like that has just
17:12
blown up the idea of trust and safety when it comes to AI?
17:18
Absolutely. Yeah.
17:18
So there's definitely contributed a lot to kind of
17:21
developing this this area again because once you reach the end
17:25
consumers then there's I mean it's literally all over my feed
17:29
now like all these mistakes, issues, biases that are popping
17:33
up which was you know that would not be the case that my LinkedIn
17:38
or Twitter feed would exceed, would be filled with biases
17:41
because you know defective inspection model for some
17:45
semiconductors is has a particular bias that's not
17:47
something that people would be posting about every day.
17:51
But awareness was raised. It definitely also reached what
17:56
we do see is a lot of these models like high stake mission
18:00
critical models deployed in enterprises also supervised
18:03
models. So definitely management and
18:07
executives are also very aware now of the of the challenges and
18:11
that they would very explicitly require you know more
18:15
validation, more trusts again especially when you go to 3rd
18:19
party models. We maybe interesting here to
18:22
give a anamorphic create example, we recently completed
18:25
actually such a third party such an assessment for a third party
18:28
model used by Migros bank which is one of the largest Swiss
18:32
banks. And you know what they do is
18:35
they're using externally build models to provide just a much
18:40
better car leasing service. So you can predict what would
18:43
the value of a car be in the future and give the best
18:46
possible terms to the user so that they can ultimately, you
18:49
know, leave this car and use it. And so this is all great and
18:52
nice. So you know, they can use this
18:54
now massive data sets to build these models and offer this
18:56
service. But ultimately this model is,
18:59
you have to understand it's responsible for managing
19:03
hundreds of millions of Swiss friends.
19:06
So you know, how do you make sure this works right?
19:08
And that's always has been the big challenge at this level.
19:12
How do you how, how do you first get it to work really well and
19:15
how do you also get off give this trust also to other people
19:20
to the to rely on that this model would work would work as
19:23
well. And I would say this kind of
19:26
general topic of just gaining trust in the in the models and
19:29
in the in in new applications has been around at the
19:34
Borderlands for for quite a while.
19:35
That's actually to a large extent also how Lattice lawyer I
19:39
started. So we initially did not start
19:43
with this idea to, you know, build the company.
19:45
We were just a bunch of research people that's, you know, worked
19:48
on the safety verification back then and we had good success in
19:53
building some of the scalable frameworks in this space.
19:56
And then there was a this was back in 2015, sixteen.
20:02
So quite quite a long time ago when?
20:04
Everybody was still discovering that the brightness and
20:07
challenges around these models. And even back then a lot of
20:10
these large companies like Boeing, Siemens, Bosch, I still
20:13
remember they would pick us in our our, our emails, research
20:15
emails. And they would ask, well, you
20:17
know, this wave is coming. You know, right now we build
20:21
classic software systems. We have processes, we have
20:25
standards, we know how to ensure that they work, but you know
20:28
there's something coming. So you know soon that there
20:30
would be a big transition in the industry where we would be
20:32
relying on just building these massive models on data and we
20:36
have no idea. So and this would be business
20:39
critical by definition, like if you want to play, you know, if
20:43
you want to be part of this next generation of businesses, you
20:45
have to, you have to be understanding how to get a grasp
20:49
on this technology. And a lot of them were asking
20:51
exactly these questions, Well, how do I make sure it works?
20:54
How do I get it to work now? What is the process?
20:56
What standards do I follow? And that's where it kind of also
20:59
hits me as a researcher. Well, that's a pretty big
21:02
transition. You know, we better get
21:06
something, you know, get it, get it, get better, get started
21:09
working on this and, you know, deliver some solution for these
21:11
companies. There is just a little side note
21:15
that I wanted to mention when you were talking about how
21:19
Twitter and the consumers are getting to Red Team with all
21:24
these models. It reminded me, I think I read
21:27
somewhere that you were you were doing red teaming before red
21:31
teaming was actually a thing, and you did you write a paper on
21:36
red teaming? So we we did a, I mean we did
21:39
safety competition, so competitions on AI safety, that
21:42
was back in 2020, so before red teaming.
21:45
And then we did participate also on some classical regime there
21:49
was a very nice competition organised by the White House on
21:52
this topic. So yeah.
21:55
So bringing that back around to what you're talking about and
21:58
this transition of going from, OK, I understand how my software
22:02
systems work. I know that things are
22:06
predictable, especially in these big enterprises.
22:10
But then you start sprinkling data and predictive modelling
22:13
into it and it's like, UMM, I'm not so confident in what the
22:19
output is. You were red teaming and you
22:24
realised, ohh, you know what, maybe we can almost.
22:27
It's almost like the white hat hackers I guess is the
22:31
inspiration And you said I think we can figure out a way to show
22:36
people how robust and how strong and how well put together their
22:40
systems are. Is that kind of the catalyst for
22:44
the start of lattice flow? Yeah, so maybe to give a bit
22:47
more background. So we were really working on
22:49
this kind of very. So how did it all start?
22:52
So after so people are building in research all these accurate
22:56
models. And then at some point, but
23:00
several smart researchers from Google and Stanford, they were
23:04
trying to understand how these models actually work.
23:06
You know, how do they, what do they learn?
23:08
How do they make decisions? And there were, there was a very
23:11
easy way to kind of play these exercises.
23:13
You tweak with inputs and you see what happens.
23:15
Then actually for a long time they thought there's some bug
23:17
because you changed a little bit and then output like just goes
23:20
through the roof and it's really could be completely different
23:23
which is totally unexpected if you have a model that is around
23:26
97% accurate and then suddenly it's behaving so oddly.
23:30
So but it turned out you know that's a that's a that's a
23:32
challenge and there's it's it's there's properties to check
23:36
beyond beyond accuracy And then everybody started working on
23:40
this topic or how do I ensure robustness of the models.
23:43
And then that's when we also got into this topic and we what we
23:48
started working on is on the, I was on the far end of deep tech
23:52
like how do I even, how can I prove even basic properties
23:55
about neural networks. And this is typically how kind
23:59
of people deal with complex systems.
24:00
You know, we don't know how they work.
24:01
Even if software is software, you know, if you have a, you
24:03
know, large code base, you don't know what's going on.
24:06
But what you can always do is you can express some simple
24:09
properties of something that you expect that the model does and
24:13
then you can verify them and even in some cases
24:16
mathematically prove them. So we started with this was the
24:20
original entry research. And then a lot of the industry
24:23
was actually really curious because they would read the
24:25
papers and then it would say, well, I have a mathematical
24:28
proof that your model is working.
24:29
And then I go, Oh my God, that's great.
24:31
Now I have the request of my on my models, which is not entirely
24:36
the case as there's, you know, other issues around this.
24:40
You know it's you cannot fully really verify, you cannot fully
24:43
specify what the model should be doing.
24:45
That's kind of the whole point of machine learning.
24:48
You're learning very complex tasks that you cannot specify,
24:51
you cannot code, you have to learn them from data.
24:53
So, but that's kind of how how everything started and then you
24:56
know industry was interested on how do you get this to work and
25:00
that's how we kind of got into the space and then started I
25:04
would say this whole journey of learning what does AI safety and
25:08
reliability really means in an enterprise, in an industrial
25:12
environment. So, which is what's quite
25:14
different from this kind of standard robustness testing that
25:19
that research was doing at the time.
25:21
Yeah, let's dig. Into that a little bit more
25:23
because it does feel like there is a gap and there you have the
25:28
research side of things and you being able to say like I can
25:31
promise you, your model is doing what it should be doing.
25:34
And then the industry where as you mentioned, there's so many
25:38
different X factors. How do you see that delta and
25:43
what are some ways that you can bring that trust AI into the
25:48
industry? Yeah.
25:50
That was a really fascinating journey in the company because
25:53
we started from the robustness angle and then we got, I would
25:58
say, broadly speaking, So industry was highly influenced
26:01
by research at the time because, you know, I was really a, I
26:04
meant, you know, Europe. IMO papers and industry would
26:08
pick on specific topics in research like robustness,
26:12
explain ability, you know, oh, that's going to solve all my
26:14
problems. And we did get a bunch of those
26:16
requests like, you know, hey, gonna come here and I'll get it,
26:19
verify my mother's correct. And then you go and you said
26:22
wait, that's like far away from me when working, you know
26:24
there's all sorts of other challenges like the data is, is
26:28
very different from what you have in production.
26:30
You know the labels are wrong. You know there's biases in the
26:34
data. There's you know, on this kind
26:35
of date your model completely doesn't work.
26:37
Like I'm not proving anything here.
26:39
First you have to go back fix all these challenges on the data
26:43
in the model set and then you can then we can talk about even
26:47
deploying these models. And that was, it was a journey
26:50
that the industry was going through as well, because they
26:53
would want to kind of eager to deploy something that would not
26:57
work. And it's very unclear what went
27:00
wrong and what you should do to even fix these models.
27:03
And then kind of at the same time in parallel also.
27:06
But at the company we were expanding capabilities of the
27:10
product to ultimately build something that's across the AI
27:14
lifecycle because you do need to address data quality, you do
27:17
need to address how do I validate the model to find where
27:21
one model is failing to kind of close this loop and iteratively
27:25
improve and get better models. So it has been, I would say
27:28
joint journey with a lot of bunch of interesting and excited
27:32
companies that want to go in production and you know really
27:35
building the practical side of it of what you need to do to get
27:39
these models to work. And you're talking about some
27:44
serious issues that we've seen time and timeout with.
27:51
Making sure that your system is running properly and making sure
27:54
that you're thinking through each piece of the system as
27:57
you're talking about there. Thinking through these
28:00
challenges. It is so crucial.
28:05
Like how how can I make sure that the data I have or the data
28:08
the whoever is building the model has access to is the same
28:14
data that is out in production? How can we make sure that the
28:18
labels and the data is clean and it's actually useful data?
28:22
How can we just make sure that the people that are messing with
28:25
the data have access to that data?
28:27
There's so many what you could like quote unquote called data
28:30
OPS challenges right? And then that data preparation.
28:35
Creating a platform seems very smart because it's upstream
28:41
issues and the problem that you were trying to go in at you
28:46
can't hit the question of is my system robust?
28:52
Can I explain the model output if you don't solve those
28:55
upstream issues first exactly? Exactly.
28:59
And what you also uncover once you get deeper into this is that
29:03
you also people purely speaking you cannot disconnect these kind
29:07
of problems. For instance, one thing that we
29:09
do is ability to find what we call model blind spots.
29:13
Like have built a model gives you some aggregate performance.
29:17
How do I find on what kind of data performance is really,
29:20
really bad? And this you can do in the
29:23
various ways to and so on. And this like this particular
29:27
problem you cannot detach from for example, data curation,
29:30
because very often what happens is, you know, maybe it's a bias,
29:33
maybe it's lacking data, and then you go back and just fix
29:36
your data set. So you just cannot, for example,
29:38
what this tells you. You cannot separate the data
29:41
work from model work one model validation.
29:45
So it's kind of these are tightly connected things and
29:47
that's why you have to put them under one umbrella so that you
29:51
could comprehensively solve this.
29:53
And that's I think very, very also important to notes O that's
29:57
so. Cool.
29:58
So basically you can see and I think I saw in a meet up that we
30:03
did in last summer where you were showing off some of the
30:08
features of this model blind spot where you can say look
30:12
across this distribution of data, the model performs really
30:17
bad. What is going on there?
30:18
Let's try and debug that. And so it gives you that insight
30:23
into being able to debug your system much more.
30:26
And then I'm guessing you can be like, Oh well, it's because we
30:29
don't have the proper data. Maybe we can go and either
30:34
gather that data or we can synthetically create that data,
30:37
like do all of those different things right Exactly.
30:40
Exactly. And you don't want I'm going to
30:42
be connecting these things. And what's really interesting
30:45
about specifically like this as you mentioned model blind spots
30:48
is that this is a just it's it's a quite fundamental problem that
30:52
people often talk about the unknown unknowns right.
30:55
How do I find out about biases that my motto has that's
31:01
influence performance and I didn't even think about them So
31:04
it's very easy if I have some idea in mind and I easier to go
31:10
and check it but how do you kind of also automate this task of
31:15
discovering those. So that's that's really
31:16
interesting and very important also from a purely from a more
31:22
basically from any mission critical use case you can think
31:24
about, you know you do need to be doing this kind of analysis
31:28
before before going into production, yeah because.
31:31
It's almost like you're red teaming before opening up the
31:35
floodgates and you're making sure that you can try to get
31:41
you're you're covering your back before you let it out there and
31:45
then see what happens. Almost.
31:48
And you can never be like 100% sure, but at least in this case
31:53
you can see more of your you're like stress testing the model
31:58
and your stress testing your ability on the system to be able
32:01
to say, OK, we did not realise there was a blind blind spot
32:06
here, let's go and work that out before we update this model or
32:11
push it into production. Yeah.
32:13
And what's also very interesting is that kind of traditional
32:18
trending on the in the whole spaces that's again it's monos
32:22
are moving away from you know I built my model myself and I'm
32:26
going to use it to hey somebody else build this model and I'm
32:29
going to use it now. So how do you kind of then
32:32
bridge this gap? So how do I get to ensure that
32:36
whatever data was used is good, That's, you know, there's no
32:39
model blind spots, that there are no biases, that best
32:41
practises were followed. And that's where I would say a
32:45
lot of the work on AI safety reliability is on framing this
32:51
problem. And how can we ensure that
32:53
there's a baseline of best practises that are always used
32:56
for certain kind of applications.
32:58
And then that's where we kind of all the standards are now that
33:02
are being defined, are really attacking this.
33:04
So we did recently had a very nice event at the boss at the
33:09
World Economic Forum on this topic and we saw this and, you
33:12
know, other people and that's basically what's what's what
33:15
they aim to do. And you know, how do you define
33:18
systematically what are the general properties that you need
33:21
to check on data? Because then this helps you
33:23
solve this problem. You know, I can, it's a third
33:26
party. If I'm a vendor giving my models
33:28
to some other people, I can just say that look, I apply this,
33:32
whatever, ISO 50 to 59 and then I followed all the best
33:36
practises. So that's why I claim that my
33:38
model should be OK. So anybody that is putting out a
33:44
third party model provider like we just saw yesterday or two
33:48
days ago, Mistral came out with their API, right?
33:51
And making sure that we can trust that API there in some
33:58
sense should be that it's just kind of right.
34:01
Right now we're just taking it at face value like, OK, this
34:04
model is good, I guess, let's evaluate it, Let's see if it
34:07
works with my application. And then let's put it out into
34:11
the public, let's put it into production and see if it
34:14
doesn't, like, totally mess up. Yes.
34:16
Yes. So this would definitely not,
34:18
you know this this try and error would not work for like if
34:20
you're for any of the mission critical use cases.
34:24
So this is going to change and there's kind of two parallel
34:27
tracks that work that are happening on one end.
34:30
You have the legal side I would say, which is what's the USA ACT
34:34
is trying to do in Europe is how they even define the classified
34:37
use cases on what needs to be adhering to, you know, stricter
34:42
standards, quality, you know, quality standards.
34:45
And then that's one part of it, the legal aspect.
34:48
And then the other part is, well, now that I am actually one
34:51
of those, you know, mission critical use cases, what steps
34:55
do I actually take to kind of show others that I have followed
34:59
best practises. And that's where standards come
35:02
into play where you know they would list all the potential
35:06
risks, potential possible mitigations or maybe if there's
35:09
no mitigations, you know, that's something that you have to tell
35:11
to whoever is going to use this model that's, you know, that's a
35:14
problem. So if you're concerned with it,
35:17
that's, you know, just as you have to live with it and so on.
35:20
All this space is going to be now actually being structured to
35:25
provide this kind of basic level of trust so that it's not just,
35:29
hey, here's a model, go and play and maybe it works.
35:33
Yeah, this is not a very, not a very good idea.
35:37
I mean correct me. If I'm wrong, but it feels like
35:39
it is not that transparent at all.
35:41
Like what you're speaking about sounds awesome.
35:44
It sounds very far away from where we're at.
35:47
Ohh, yes. So we we uncovered this just a
35:49
few months ago when we when we did the first assessment for for
35:54
for for the Swiss bank. And so we did take the latest
35:58
ISO standards. They're quite nice.
36:01
You know they defined a bunch of properties, but the gap between
36:04
the high level requirements discussed in these standards to
36:08
the actual technical objects you have to be running on these
36:11
projects, you know on the data and the model is very very big
36:14
up O thinking about. They would all talk about data
36:19
has to be representative, it has to be relevant and all these
36:23
things. But what does this even mean?
36:26
So there was a lot of back and forth with well you know these
36:29
actually things, these these are quite generic so we can check
36:32
those but you know we have no idea what relevance means.
36:34
So then you go to the bank in this case and say hey what, what
36:38
data is relevant to you and then you kind of encode those.
36:40
So that's going to take quite a few I would say iterations and
36:46
you know until this is becoming something that's everybody can
36:49
do right now it it because quite a lot of know how basically to
36:54
bridge this gap because you're just so white that you cannot
36:56
just read and implement well. It does feel like that is a case
37:01
for using open source large language models, because you can
37:06
control them a little bit more, but even still, you don't know
37:09
how they were trained. And there may be those unknown
37:13
unknowns that are just kind of hanging out in dormant, latent
37:17
space. And I'm sure you saw that paper.
37:20
I think cohere put it out your anthropic put it out where it
37:23
was like, oh, there's just a this stuff that can happen
37:28
because in latent space if you touch it all of a sudden it's
37:32
like this bomb that just went off in your model.
37:35
And I'm doing a very poor job of explaining the paper, which I'll
37:40
try and find later, but the the ship idea.
37:44
There is. Like you don't have the
37:46
understanding of how it was trained and how that model was
37:49
built. And because of that you don't
37:53
have full understanding at what is possible and capable with
37:57
that. Are you thinking about trying to
38:00
retroactively do things like that where, OK, we're never
38:04
going to be able to work with every open source model provider
38:09
like you're not going to be able to?
38:11
Well, you would probably love to work with Meta and Minstrel and
38:15
whoever else comes out with their next open source model,
38:17
right? But in the case that you can't
38:21
do that, is there the possibility of being able to pop
38:24
in a model and then test it after the fact?
38:28
Something exists, others know. So there would be certain things
38:30
that you could, you know, test just in a black box manner.
38:34
Meaning that you know, like robustness would be an example
38:37
of this, right? I can, I can test some of those
38:39
properties. But if you talk about, let's say
38:42
then data, relevance, representativeness, you know,
38:45
and I have no clue what data this model was trained on, well,
38:49
good luck. You just don't have the enough,
38:52
you know, knowledge, knowledge, knowledge to to run these
38:55
cheques. So I think open source.
38:57
Of course that's useful because it does give you some
38:59
visibility, but it's not sufficient and in a way that
39:03
that's a big includes the need for doing these kind of
39:06
additional assessments and validations.
39:09
Because I mean, just look at open source code itself, right?
39:13
It's not like people would not blindly trust open source code.
39:17
In fact, actually one of the big challenges for the industry was
39:21
you have all these great open source projects and every
39:24
developers love it. So it's just imports, compile
39:27
and run. But how do you know that means
39:31
that you're now relying on 3rd party developers pushing commits
39:34
and you have no idea what kind of vulnerabilities you will be
39:36
importing. So that was also the software
39:38
space that was, you know, open source was not a solution, it
39:42
was just something that was useful because it accelerates
39:45
accelerated developers. But still there has to be all
39:48
these additional tooling and systems that would need to track
39:52
vulnerabilities across versions and so on.
39:55
So actually builds reliable software, so models will not be
40:01
any any any difference here. Yeah, it exactly.
40:05
I'm not trying to make the case for you.
40:07
Grab an open source model and then you're good.
40:09
It's almost like the open source model.
40:12
You can play around with it more or have a little bit more
40:16
insight into it and then O it into your system and have more
40:21
control. But I hear you with like, yeah,
40:26
you can have more control, but at the end of the day, if
40:29
they're not giving you the data on what that model is trained
40:33
on, then good luck. You can fine tune it all you
40:37
want, but you're not going to be able to really get down to the
40:40
nuts and bolts of it. Yeah, and.
40:42
What's also interesting is so I mean often you would not want,
40:44
so for all of these companies data is the IP.
40:48
So you don't want to just then put up your data and say, hey
40:50
look, that's what it is. So then this whole domain of
40:56
third kind of independence assessments come into play and
41:00
that's in fact how it worked with the Swiss bank.
41:03
So it was a third party vendor building these models that they
41:06
felt did not go and publish all the data to the bank because
41:09
then the bank had just take, you know train their own models.
41:11
And so then we had access to the vendors, data, training code,
41:16
everything, so we could plug them in, run the cheques and
41:19
then deliver this to the bank. But we are kind of an
41:22
independent provider that's guessed this access and
41:26
ultimately ensuring that IP in data in particular is is
41:30
isolated from the from the consumer.
41:32
So that's definitely something not something to underestimate
41:35
in terms of kind of dynamics and how this would, how this would
41:38
work. Does it make sense?
41:40
Yeah. And.
41:41
Thinking about that a little bit more, it's you're not talking
41:44
about a third party provider as like the traditional what I
41:49
would think of it as a third party provider as like Open AI
41:53
or Anthropic. You're thinking of a much more
41:55
specialised third party provider that is building out a machine
42:02
learning model that is specific for this leasing use case that
42:06
you're Speaking of. Exactly, yeah.
42:09
And it's less of like a gigantic LM that you can grab and it's
42:15
open AI's version and good luck figuring out what's going on
42:18
there and working with them on on that.
42:22
Which I can imagine makes it just even more difficult to have
42:26
trust in your systems if you're using something like an off the
42:31
shelf open API instance or just the API.
42:35
Correct, Yes. So, yes.
42:37
So there's still, I mean there are interesting cheques, we're
42:41
actually we are, we have been looking into this also in
42:43
collaboration with some of the research team.
42:45
It's at ETH basically. How do you kind of compile some
42:49
of the cheques in the AI act? But again, you many of those you
42:54
just cannot run unless unless you have access to to
42:57
everything, which is the case today to make up for many of the
43:00
models. And you feel like safety will be
43:09
only taken seriously when there's the regulation that is
43:14
put around it. Just like you have the sock too,
43:17
and you have the ISO and all of that fun stuff.
43:21
Or is it like because there's a lot of money on the line and
43:24
because somebody needs to be able to cover their ass,
43:27
especially if it's dealing with serious amounts of cash, like in
43:31
this bank example, it's already at the forefront of discussion?
43:36
Yeah, that's the other case definitely.
43:38
So again, when we were in Davos, actually we did have one of
43:41
those vendors coming to speak and he was up front.
43:44
He said, look, I don't care about the regulation.
43:47
I care for two things. You know I'm building these
43:50
mission critical models have to make sure they work because
43:53
otherwise my customers would lose a bunch of money.
43:57
And 2nd, I have to instil trust in whoever is using these models
44:02
because otherwise they would just never put, they'll never
44:05
use them. So that's really where we do and
44:07
I think that's very healthy dynamics to observe on the
44:10
markets because you don't want stuff that are purely regulatory
44:14
driven. I think that's not overly
44:16
exciting. So you know, people do realise
44:19
that you know there is, there is real, real need for ensuring
44:23
this stuff works. There's real needs to
44:25
demonstrating this to others And then that's why I've always said
44:29
that, you know, you have to focus on getting things to work
44:32
before encoding it is to standards and regulations
44:34
because it should not be backwards.
44:36
So what standards and regulations should do is again,
44:39
they should kind of go and put these baseline.
44:41
Once you understand how things should work out, this is the
44:44
baseline for all these applications in this domain.
44:48
We want to make sure that these things are checked.
44:51
That makes sense. The second thing that the
44:53
regulations are very important is also kind of the whole red
44:56
lining of use cases. Because I mean we cannot ignore
45:00
that this is super powerful technology and you know, you can
45:04
do just bad things like mass surveillance, you know, do we
45:07
want this in Europe? Probably not.
45:10
So I think that's kind of always here to be my position.
45:13
Focus on redlining things that we agree on and that's good to
45:17
do. For everything else, just gonna
45:20
follow what the industry already needs just to put this baseline
45:24
to make sure that some vendors are not slacking off by ignoring
45:27
these things. It is not a surprise that money
45:34
drives this innovation much more than regulation does.
45:38
Yeah. And regulation in one way, shape
45:40
or form is, is almost a little bit behind.
45:43
And then you spoke about some of these very vague regulations
45:48
that are out right now. And yeah, I'm wondering if you
45:51
feel like there are certain regulations that are actually
45:56
very useful and that you like to see, like the surveillance
46:01
warning could be an example of that.
46:03
Or maybe there's other ones around the robustness that you
46:06
were talking about. Yeah, so in.
46:08
Terms of regulation so I have less I mean I've read them so
46:13
this is a real document. So I have there are still very
46:16
good I think also typically again not an expert but these
46:19
get set outs in courts to see what did what did this mean is
46:22
it this or that and then after a few you know that's why you
46:24
always refer to cases to show how things were interpreted.
46:27
So I think that would definitely be a big iteration on of this
46:30
kind to concretize a lot of them beauty in the in the on the
46:34
regulation side I am interested in the standards.
46:37
I think they are very useful because these are you know, more
46:41
down to earth technical documents that's you know risk
46:43
real technical risks and mitigations and those are
46:48
already do provide useful frameworks.
46:51
So they do exist. Many of them are.
46:53
So I mean we were talking about here for example ISO 50 to 59
46:58
which tackles data quality, there's a 24, zero, 44 that is
47:03
about model robustness. There's stuff on the model
47:06
governance and they're well thought out.
47:08
So I think they they would they would definitely you know take
47:12
be adopted in some way and there are concrete enough so that you
47:16
know experts like can take them and start implementing the
47:20
cheques and so on. So yeah, and so the the.
47:23
Difference there is like the regulation is just the
47:26
government saying we need, you have to do this by law.
47:30
The certification is where you say I'm going to jump through
47:35
this hoops so that I can show and I can prove that I've
47:39
thought about different areas where my model can fail.
47:43
It's not by any means a must, but it is something that I want
47:47
to do so that I can get that money and I can get that
47:50
contract. Yeah, and there.
47:51
There would be. So I think the regulations would
47:53
require some of the standards to be, you know that you would need
47:58
to be certified for certain things, but we're not there yet.
48:01
So I think that will still take, I mean at least two years based
48:05
on if you look at the AI Act, so that's you know April, so two
48:08
years afterwards if it happens on time.
48:11
So there will be some time until this this takes takes a bit.
48:16
And what, what were the two certifications you mentioned
48:19
2024, ISO 20, yeah there were, there is one.
48:22
It's 5259, so that's a very useful standard for data
48:27
quality. They sold 40,001.
48:30
This is about AI. AI defines how to do API
48:33
management system effectively. So these are the ones that we
48:37
see most kind of discussions around those and things that
48:40
would likely become something that's implemented by the
48:45
industry going forward. And what is the 4000 like?
48:49
What is it around? How to do API management?
48:53
Yeah, so. This is this is about like just
48:56
properly documenting like what models exists, how do you
49:00
upgrade models? So defining the kind of it's
49:04
around the process of managing models as opposed to drilling
49:08
down and then checking a particular project.
49:11
Like, is it actually saved? You know, is it?
49:15
Does it comply with specific data quality and model
49:17
robustness cheques. So that would be the different
49:20
set of standards that would apply for, for this, for these
49:23
cheques and these. Certificates.
49:25
They aren't new, are they? It's not like they just came out
49:29
this year. Yeah, well.
49:30
They they've been, they've been in the in working in the
49:32
workings for several years now. Some of them are very, you know
49:38
close to final or final. So this is all happening
49:41
basically right now. Incredible.
49:44
OK, yeah, because it does feel like I've heard these being
49:50
talked about before, but I didn't realise that it is not
49:54
quite finalised and it's something that people are
49:56
actively thinking about. They're trying to incorporate
50:00
just to get ahead of the curve. So as soon as it is finalised
50:03
you can go and get that certificate and say look, we our
50:07
thinking about this and we're actively working towards it and
50:11
so that boom, you have the certificate.
50:13
Yeah. The thing is because even if
50:15
they're not final, it still gives some methodology to it.
50:19
So let's say for this you know vendor for the bank.
50:22
So if we just say, well look I I thought of five ticks, I think
50:27
they're good. So that's why the zero in the
50:29
bank should trust me. I think that's not a very good
50:31
argument, right. So you don't want to rely on
50:34
something that's more more globally acknowledged and
50:37
accepted by invite people. So even if not fully final,
50:40
they're still variable. Yeah, it it's reminds me of like
50:44
certified organic food and you know, you want to make sure that
50:48
you're hitting the certifications on that or any
50:52
kind of sort of certification, it makes complete sense.
50:55
So does lattice flow help in that regard?
51:00
Like are you actively building into the product ways that it
51:07
makes getting certifications easier?
51:10
That's a good question. So this is something that's
51:12
indeed actively happening because we did start with this
51:17
general framework for improving data quality and model
51:22
performance, safety based on intelligent workflow or
51:26
prefixing data and model issues. But now now that's kind of the
51:30
market is also shifting towards using third party models and so
51:33
on. This whole validation piece is
51:34
becoming really key and these things are very tightly
51:38
connected because again because they should be, it cannot be
51:42
that you know to get a model to work you have to do these 10
51:45
things and then standard talks about something completely
51:47
different that would be no sense.
51:48
So there are basically now these things are being qualified to
51:52
best practises into these standard standards and it's
51:55
becoming kind of packaging, those kind of packaging the
51:59
cheques that you would normally do S&M engineer or it's a data
52:03
scientist to improve your project.
52:05
It becomes as kind of cheques that you would package them to
52:07
show to somebody else as opposed to acting on them to fix certain
52:12
kinds of issues. So that's definitely something
52:15
that is evolving and we are let's say very actively also
52:19
looking at it and playing, playing, playing on this market,
52:23
so. Basically being able to expose
52:26
and. Show off.
52:29
What you're doing in a way so that it's not behind closed
52:32
doors and it is very visible for others and getting that
52:36
transparency into the workflow, Yep.
52:39
So that we can go back to the whole idea of, yeah, how are you
52:43
going to trust something if you don't have these transparency
52:47
pieces? Yes, and that.
52:48
Also helps a lot of communicating across different
52:50
levels in the organisation because it's one thing to talk
52:53
about, I don't know, data biases and labelling errors.
52:57
The engineering wrong. It's another thing to, you know,
53:00
go to CIO or the chief risk officer in the bank and then
53:05
talk about these topics. So this is kind of it's because
53:09
this standard, these are standard frameworks in ways that
53:11
the industry already works. You know this is putting these
53:13
structure so you could actually communicate between these
53:17
different levels and that was. One question that is popping up
53:22
in my head is how much are you seeing engineers jumping on
53:28
board with this type of stuff versus the compliance and risk
53:32
versus any other stakeholders? Yeah.
53:36
So complacent risks Not yet. So this is still very, very
53:39
early. I would say again this is not
53:41
something that's developed now. So with your focus on engineers,
53:47
so you know that is slow is used by engineers to get their models
53:50
to work O but if they there is this that was a transition going
53:54
on on the markets and that this is also becoming a markets on on
53:59
its own but it's not definitely not something that's kind of
54:04
fully developed yet so. Yeah, just give it time.
54:10
Let Air Canada keep selling, no refund policies and Chevy keep
54:15
selling their cars for a dollar and all these mistakes.
54:18
Gemini keep generating the Black Forefathers and you'll be fine.
54:25
Man. Compliance and risk.
54:26
We'll want to get into the conversation before you know it.
54:29
I think so. Yeah, I think I think this year,
54:31
this year 24 would be. I think last year 20/20/23 was
54:35
really a lot about the awareness of this challenges.
54:38
No. Now this year 24 is a lot about
54:42
delivering value with these models as well as kind of making
54:45
progress with this validation that you could entrust in
54:50
instilling trust in the in the models.
54:52
How far do you? Think we can push the whole deep
54:56
learning approach to like more mission critical applications?
55:03
Yeah, I think that's a really. It was also something that we
55:05
actually discussed in Davos now with the roundtable of experts,
55:09
because there is certainly things that we can make
55:14
progress. Also all these topics around
55:16
like ensuring safety, reliability for you know,
55:21
specialised custom models that solve kind of in control
55:24
environments. I think that's definitely be in
55:27
the realm of things that we would grasp.
55:29
And now the question is how can we make this systematic and you
55:33
know, have this widely deployed for for everybody.
55:37
I think the main limitations are quite well known, you know,
55:40
still like deep learning is handling the long tail, you
55:43
know, it's very hard to handle the edge cases, get the model to
55:46
generalise, to work universally well across your distribution.
55:51
All the challenges around like I can tweak them, tweak my inputs
55:56
and get the model to output something is a fundamental
55:59
thing. So this will not disappear
56:01
magically, no matter what kind of safeguards guardrails you put
56:05
around these models. And that was also a comment that
56:08
one of the participants, his name is Apostol Vasilev, he's a
56:13
a security expert at NIST. He made it, he made a really
56:16
smart observation that they liked and he said he kind of
56:19
brought an analogy from traditional security worlds
56:22
where normally you would want to have a secure design.
56:25
So something in cryptography for example, is known as information
56:29
theoretic guarantee. So you know that if you
56:31
implement this, it works, and then securing the system boils
56:34
down to just implementing it correctly, meaning you don't
56:37
have bugs. And that's very important
56:39
because some of those are design issues.
56:42
So you can you can get the bottle to outputs no matter
56:45
what. So you cannot just safeguarded
56:48
by plugging in another model or something else.
56:49
You need to do something fundamentally different so that
56:52
it is the design is, you know, has a good foundation and then
56:56
you can solve it. So in general, so there's a very
57:00
big difference and that's important.
57:02
So to basically differentiate benign environments where you're
57:05
deploying against something in controlled environment where
57:07
people are not trying to attack you versus something in a fully
57:10
adversarial environments where I have full control and I'm going
57:13
to do whatever it takes to break your model.
57:15
And these are things that would not be solved with current
57:18
approaches. And that's important because
57:21
again, these are the risks that we have to know, you know,
57:24
there's no solution and then we just assume them.
57:28
We're done deployed. If only it could all be benign
57:32
environments and rainbows and flowers.
57:36
Tie dye would make everybody's life a lot easier, huh?
57:41
For. Sure.
57:42
Yeah. Excellent.
57:44
Well Peter, this is really been informative for me on the Trust
57:50
AI front. I know that you're knee deep in
57:52
it and I appreciate you teaching me a tonne about it.
57:55
Thanks. Thanks.
57:56
For having me the messages.