Latest / Agentic Conversations (formally mlops.community) / EU AI Act - Navigating New Legislation // Petar Tsankov // MLOps Podcast #271

Transcript

EU AI Act - Navigating New Legislation // Petar Tsankov // MLOps Podcast #271 · Agentic Conversations (formally mlops.community)

0:00
Everyone So my name is Petter Petr Tankof Co and Co founder of
0:04
Flatis Low AI and I do like my coffee non regulated because I
0:09
like to make it exactly the way I like it.
0:13
What's going on, people of Earth?
0:15
We are back for another Envelopes Community podcast.
0:17
As always, I'm your host Dimitrios.
0:20
And today we got talking all about the AIEU Regulation Act.
0:28
I find it fascinating because there is a narrative that is
0:32
going on which is all regulations bad, the EU Act is
0:36
stifling innovation, etcetera, etcetera.
0:39
And here what we see, Pedar came on.
0:43
He talked about what we need to know if we are doing any kind of
0:49
AI within the EU and how we can action all of this legal jargon,
0:59
the words on a page into tests for our technology.
1:06
And you know what we're going to do just in homage of today's
1:09
conversation, all about regulation, we're going to kick
1:12
it off with a little song from the 90s.
1:17
You may know this one from good old Warren G and Nate Dogg.
1:21
You. Regulate any stealing of his
1:23
property. We're damn good, too.
1:25
Regulators can't be any geek off the street.
1:30
Let's do it. See you all on the other side.
1:35
Mount up. It was a clear black night, a
1:38
clear white moon. Warren G was on the streets
1:40
trying to console some. Search for the E so I can get
1:43
some phones. Rolling them around, chilling
1:45
all the young. Just hit the east side of the
1:48
LBC on a mission trying to find Mr Warren G.
1:51
Seen a corporal of girls. Ain't no need to tweak or you
1:54
search. No.
1:55
What's up with 213? Last time we talked about how we
2:14
were optimising for accuracy. This time I really want to get
2:17
into how we've been optimising for capabilities, especially
2:22
with these large language models.
2:25
But then there's this sneaky thing that kind of came up and
2:30
blindsided a few folks who may or may not have realised it was
2:36
coming. But that is the whole EU act and
2:40
the AI regulation. So I wanted to have you on here
2:44
and just really get into AI safety, AI compliance, AI
2:49
governance, all of that fun stuff because I think there's a
2:52
lot to unpack here and what it means exactly.
2:57
Like this hopefully will be the definite guide for if you are
3:00
trying to do AI in the UEU in 2024 and 2025.
3:06
As long as the current legislation stands, then you can
3:12
come and listen to this and get the best breakdown and hopefully
3:17
the best understanding to know where to go and what to do.
3:21
So last time we spoke about a decade of AI safety, right?
3:25
Let's change since our last discussion.
3:28
Good point. So I'm glad, glad to be back on
3:31
the podcast. So last time when we spoke, that
3:33
was just about 6-7 months ago, I believe.
3:36
It feels like it's been several years, but time was very fast in
3:39
our space. And last time it was very, very
3:42
mixed state in terms of AI safety.
3:44
I think there was still a lot of confusion like safety was such a
3:49
such an abstract topic. So people are still worrying
3:52
about the wrong things, like worrying about existential
3:55
periods of AI, not quite clear what's going on with regulations
3:59
and so on. And lots of things have changed.
4:02
I mean, the on, on all aspects. So on kind of on the more
4:05
practical aspect, companies are a lot of organisations right now
4:10
are really hands on implementing AI governance.
4:13
So I have been to multiple industry conferences where you
4:17
see companies realising that they go from kind of from 5:00
4:20
to 5:00 AI applications to 100 over like a period of, you know,
4:24
several months, which is crazy. And everybody's realising we
4:28
have to put this in order. So let's, let's start, you know,
4:31
making sure everything is well, at least understand what
4:33
solutions we have in place. Let's make sure we can classify
4:37
which ones are important for the business, which ones are not
4:42
kind of what are the risk if there's regulation, you know, if
4:44
there are applicable regulations like the EUAI Act that came into
4:48
force since August, but first, so just last month.
4:52
So you know, when then how, how do we demonstrate compliance?
4:56
You know, how do I pick the models that would actually pass
4:58
these cheques? So this is very healthy to hear
5:01
this kind of progress because it's not about, you know, that's
5:04
AI would destroy the world tomorrow.
5:06
Let's stop training models globally.
5:09
So this is now, these are really now the, the practical applied
5:13
topics that you want to be thinking about.
5:14
And that's, that's really good to see.
5:17
I like that you call that out, how there was so much fear
5:20
mongering, especially, yeah, in February even, and people were
5:25
being called to the Supreme Court or to Congress to testify
5:29
in front of Congress. And they were talking about how
5:32
we need open source or oh, we're going to put a ban on a certain
5:36
parameter of model training size or whatever.
5:38
It was also outrageous and now it's changed a tonne.
5:45
I was even funny enough just reading through the State of AI
5:48
report and it talked about how these model providers have
5:52
switched from this whole AI doom scenario type of narrative to,
5:58
uh oh, now we got to go out and make money.
6:00
And so we're going to not talk about how AI is going to take
6:05
over the world or any of that kind of stuff.
6:08
And I find that that really funny.
6:10
But you said a few things that I want to pick apart.
6:14
What is companies that you've seen going from 5 to 100 models
6:20
and recognising this can get out of hand very quickly, so they
6:25
need some kind of governance process in place.
6:28
I want to talk a bit about what governance processes you've seen
6:31
that have been working. But also I want to mention, we
6:35
had a guy on here probably around the same time that you
6:41
came on last and he was talking about how he was working for
6:44
Yandex, that Russian Google type thing, Google search.
6:49
And they had no idea out of the thousands of models that they
6:54
had in production, which models were doing what and which ones
6:58
actually were affecting the business.
7:01
So he coined the term, they had zombie models in production.
7:06
And one of his biggest projects was to go out there and take
7:11
down a lot of these models that weren't doing anything because
7:15
people were afraid. They were literally afraid like,
7:19
ah, I'd rather just keep it out there because I don't know if I
7:21
take it down if it's going to all of a sudden show up as a
7:24
$2,000,000 loss on the balance. Sheet.
7:28
Yeah, it's all clear. That's how it's being used and
7:31
what it's doing. But you better, better keep it
7:33
running. Yeah, yeah.
7:34
But it's really, it's really, it's also, it's really, I mean,
7:37
especially in the larger organisation, understanding
7:40
what's happening underneath. You just don't know and we do.
7:45
So unless you put this very centralised process in the
7:48
organisation where there's like designated team responsible for
7:51
this with more clear definitions about what is high risk.
7:55
I'm talking about high risk for the business.
7:57
Not, not necessarily, you know, with respect to regulations that
8:00
UAA act and kind of other criteria that's are also now now
8:04
becoming becoming relevant. And, and yes, so unless you have
8:10
these people to kind of go and dig and understand this, you,
8:12
you don't, you don't, you just don't know.
8:14
And that just exposes then the organisation, it's, you know,
8:18
high risks because on one end you have, I think everybody
8:22
really understoods the, the real business risk that this is not
8:25
just technology that you deploy, but there's users on the other
8:29
side that get affected. So I mean, there were already
8:33
several lawsuits or the typical things like buyers, wrong
8:37
information. I mean, if you rely on customer
8:41
support and the organisation gives the wrong information and
8:44
then I because of this, something goes wrong, why would
8:47
I not sue? You know, I would.
8:49
That's how that's how life works.
8:51
And that's kind of where I think this, all these problems, they
8:55
immediately got lifted very high up in the organisations because
8:58
it's, it's it has this massive impact, business impact when you
9:03
talk about risks. And yeah, so that's what then
9:06
requires you to to do this systematic process.
9:09
And what we have seen on top of this, because I think governance
9:12
for me is not governance for me, is that like the starting point,
9:16
like this is like you need to open your eyes and see what you
9:19
have in the organisation. But then the second big, the
9:21
immediate next big challenge is, well, what do you do about the
9:24
high risk ones? And we can later chat about the
9:28
EU Act as well. Like how do you actually assess
9:31
the risks or how do you assess compliance?
9:33
Like no, like these are now the topics that I see becoming very,
9:37
very relevant because you have to address them right now.
9:40
Suddenly, you know, you know, you know what you have in, you
9:43
know, you need to make sure they, they are compliance safe
9:46
and so on. And that's where we do see kind
9:49
of now kind of two that kind of dynamics 1 is we see like
9:54
individual teams in organisations, like adopting
9:56
specific, like AI applications could be identity management,
10:01
could be, you know, predicting prices, whatever it is.
10:04
And these people, they, they do realise that, well, this model
10:07
is critical. I have to make sure it works.
10:08
So let me make sure I don't get fired.
10:10
So it has to be checked, thoroughly checked.
10:12
And they actually are seeking out explicitly like going out
10:15
and seeking for experts to, to validate them.
10:18
But I think governors would slip everything upside down because
10:20
suddenly everybody will know, well, actually, these are the 50
10:23
important a applications in the organisations.
10:26
You have to go and check them. Like, no, you know, that's kind
10:29
of becoming top down now driven on how you how, how you address
10:34
risk at the organisation, which is also typically how you deal
10:37
with risk, right? You don't rely on like a
10:40
diligent employee or a manager to make sure that they do this
10:44
on their own behalf. That's kind of opportunistic.
10:49
It does feel like since it was so easy to get AI out there, if
10:55
you played it fast and loose and you put something into
10:59
production and you didn't think through all of the risks and the
11:04
governance, then it came back to bite you.
11:08
And I like that you mentioned that got escalated up very
11:13
quickly. And so now you're seeing teams
11:16
that are specifically built for this purpose.
11:19
And I would imagine this is more of, yeah, when you get to a
11:23
certain size of a company and you get to a certain type of AI
11:30
or use case, then you 100% need that.
11:33
Have you seen that be the case? Yeah.
11:37
So virtually any organisation at this point needs this.
11:41
So I, I wouldn't put like a size threshold, but I mean there's
11:44
some minimum size of threshold, you know, maturity of the
11:48
company where you would have that volume of solutions where
11:51
you would need this oversight. But we do see this being
11:56
definitely centralised because it requires specific expertise
12:01
like the understanding of risk management.
12:04
And we often also see that this becomes an extended function to
12:08
risk teams that are now also not purely dealing with kind of the
12:12
classic risks risk models, but also being tasked with and you
12:17
know, to on boards, AI engineers that capable people that really
12:22
understand also the technology so that they can think about
12:25
this and ensure that they set a process to, to, to, to validate
12:29
them. So that's kind of the two ways
12:32
we see it. But it's definitely, definitely
12:34
becoming a thing, like at any organisation.
12:38
I want to talk about a tool that you just open sourced recently
12:42
and props to you all for the name.
12:46
I think it. Yeah.
12:48
So it's comply, right? But it's comply, yeah.
12:53
Exactly. Compile.
12:54
Compile Yi. Yeah.
12:55
What is it? Yeah.
12:57
So complies what it is. So the reason we started this is
13:03
to address the big challenge with the high level regulations.
13:07
So right now there's a lot of fear around specifically this is
13:11
for the EOI Act. So the first is the first is the
13:14
first compliance centred evaluation framework for the EOI
13:18
Act for language models. So what this means, you can take
13:21
any language model and you can, you can run the evolution
13:25
framework and it will give you scores that can create scores
13:27
between zero and one to show you how compliant the model is
13:32
against the specific principles defined in the EU AI Act.
13:35
So this is what it is. And what we had to do to make
13:38
this possible is so I don't know if you, if you read the EU AI
13:43
Act, it's very, very high level. It does have a lot of technical
13:47
terms as well in like inside the text, but it's very high level.
13:51
And this is was really scaring a lot of people because you don't
13:55
you, you suddenly have a regulation that you have to
13:59
comply with. If you don't comply with,
14:01
there's like serious fines like €35 million or 7% of revenue,
14:06
but you don't know even how to show compliance.
14:08
You don't know you have models, you have no idea how well they
14:11
are against the EOA Act. So that was, you know, to
14:14
address this, you had to basically bridge this gap.
14:17
You have to take the regulation, the high level regulatory
14:20
requirements, then map those into, translate those into
14:24
technical requirements. So something that's, you know,
14:27
that's actually actionable that you can measure and that's
14:31
effectively what we did. So mapping the requirements, the
14:33
high level requirements to technical requirements and then
14:36
each technical requirements on its own is mapped into concrete
14:40
evaluation, you know, algorithms and benchmarks that you can run
14:44
and then measure, measure the numbers.
14:46
So that was what the effort was. And I think the reason this got
14:51
a lot of attention is that, you know, this was really the first
14:54
time we also in addition to the project, once we implemented
14:57
this, we evaluated many of the available, the public models and
15:02
there's lots of leadership boards online that talk about
15:05
performance capabilities, you know how well these models are
15:08
performing. But is it the first time that's,
15:11
and we understood how well the performer gets the the UAA act
15:14
and that was kind of now interesting to see like well,
15:19
how do they actually perform? Now, does this work for
15:24
fine-tuned models? Also, can I throw this on top of
15:28
whatever fine-tuned model I've created?
15:32
Absolutely, yeah. So this is you can integrate.
15:34
So right now it is for language models.
15:36
So if you want to do some other modality that would not work,
15:39
but it doesn't matter. So you could take, yes, as long
15:42
as you can plug in the model and you have an kind of an inference
15:45
API to to run it, then you can then you can do it.
15:49
And yeah, so that's that's why also how how we were managed to
15:53
evaluate the models from open EI, Entropic, Mistro, Alibaba
15:58
and kill, kill the other ones. Yeah, it was fascinating that
16:02
some of the models they just have said, hey, we're not going
16:07
to play in the EU because of the EU Act, right?
16:10
I think Llama is one that I am that comes to mind right away.
16:16
And it's that the Llama 4 O 5B model or Llama 3.2 basically was
16:23
not released in the EU, right. And so do you feel like this is
16:28
a way to combat that and maybe skirt around it?
16:33
Yes, I think, I think that that's ultimately what you want
16:37
to address. I think it's not surprising
16:39
that's, you know, vendors that are training the models are
16:42
worried about deploy here because if you have this strict
16:47
rules, but they're actually they're not actionable, meaning
16:50
you don't know if you comply or not.
16:52
You're just, you know, it's, it's just opaque.
16:54
So you want to you want to stay out.
16:57
So in a way, the reason of this work is that to show that well,
17:01
kind of let's break, let's let's just not not not adopt this like
17:04
traditional narrative, you know, regulation is bad.
17:08
You know, this is like, you know, some things has to be
17:11
regulated. If it's powerful technology, you
17:13
have to regular, you know, you have to put some, some baseline
17:16
safety and so on. So, but let's make it
17:19
actionable, right? Let's, let's make sure that any
17:23
organisation can actually evaluate their models, can
17:25
understand against which principles the models are
17:29
performing well, against which ones, you know, not not so much
17:32
and where effort should be placed.
17:34
If you have this kind of actionable benchmark evaluation
17:38
frameworks that can guide you what to do, I don't see, you
17:42
know, that's not really scary. This is just giving you
17:44
direction, directions on how to improve your models.
17:47
And if you look at what are actually the, you know, the kind
17:50
of the core principles and requirements that you need to
17:53
satisfy for compliance, Well, they are very, very meaningful.
17:58
There's, you know, these are not things that's generally AI
18:00
vendors would not want to comply with.
18:03
You know, we're talking about things like toxicity or, you
18:06
know, making problems. These are useful things that you
18:08
know vendors want to have those, but you also need to have the
18:12
proper benchmarks to evaluate it.
18:14
Well, one thing that I was thinking about, as you're
18:17
mentioning how vague the EUAI Act is, and really the whole
18:24
part of this work is making it actionable.
18:26
How did you take the vague words of the EUAI Act and then create
18:31
something that is actionable? Yeah.
18:34
So that was let me summarise one year of effort into a few
18:38
sentences. Now there were a lot of people
18:42
involved also just just to be clear and to acknowledge all the
18:45
contributors. So with this was we also worked
18:48
with ETH Zuri. So there were very, very bright.
18:50
Some of the brightest PhD students working on AI safety
18:53
were involved in this effort. Also Insight, which is one of
18:56
the software based research institutes created in
19:00
partnership with ETH and EPFL. So you know, they're also
19:03
actively participated. So it was a very broad effort.
19:05
We'd also, we had people with legal degrees or legal
19:09
backgrounds that did also have an AI background.
19:13
So you do need this kind of mix of expertise to, to make this
19:17
possible. So it was, you know, really
19:19
going through the acts, finding all the, the technical terms
19:22
kind of to understand what is the intent behind these
19:25
principles and then mapping them to the common technical
19:29
requirements that are that where we do, you know, there's
19:31
actually real tangible work in research that can be, can be
19:35
applied. So this will be, you know,
19:37
things like robustness, fairness for cybersecurity that would be
19:42
prompt leakage attacks and and so on.
19:46
So this was really what's the, the, the first phase of the
19:49
effort was, and you know, it was a kind of a, a, again, a, a big
19:54
collaboration. And right now what's actually
19:57
interesting is the, the EOA office.
20:00
So we are also very closely, you know, this is not something we
20:03
have been in contact with the EOA office and the European
20:06
Commission about this effort. So they're, they have been very
20:10
positive. You know, we're really, really
20:11
welcoming this work because they realise they also have to do it
20:14
because otherwise they themselves are getting a lot of
20:17
criticism that you're. Putting something out that is
20:20
not actionable. So you've been steeped in this
20:23
for the past, I don't know how long.
20:25
So and when you say terms like the EUAI branch and does XYZI
20:32
have no idea what that means, like the EUAI branch, is that
20:37
how many people are we talking here?
20:38
What is their profile? I know there's some lawyers,
20:41
there's some technical people in there.
20:43
And this is, I also know that this legislation goes through
20:48
many phases and then it gets passed and then this happens and
20:54
maybe it gets appealed. So can you just break down the
20:57
whole law system and branch of government that we're dealing
21:03
with in the EU here and what that looks like?
21:06
Yeah, I probably cannot. I don't, I don't fully
21:12
understand it myself either. But we do have a, you know, we
21:15
do understand which are the core offices within the European
21:18
Commission that we do, you know, that there's a public port house
21:21
where you can contribute, could contribute this work.
21:24
And the way they work is again, by setting up this like special
21:28
working groups with external experts that are capable to kind
21:33
of fill in these gaps that they would not do on their own or
21:35
that they cannot do on their own, meaning that you don't
21:38
want, for example, the EU cannot create a benchmark and endorse
21:42
like this is the benchmark for compliance.
21:44
You know, this is not the intended goal for the, for the
21:48
EOA Act. The goal is to outline, you
21:50
know, to have the legal document, outline the principles
21:53
and then through the help of these working groups, translate
21:56
it, make it actionable. And then, you know, other
21:59
organisations would provide like us would be providing
22:01
benchmarks, evaluation frameworks to demonstrate
22:04
compliance. So this is so it's, it's really
22:08
there are key, I would say that there are key people within the
22:12
office that are, you know, driving these initiatives.
22:16
OK, Yeah. And what type of personas are
22:18
you dealing with in these different branches?
22:21
Because that always fascinates me.
22:22
Is it like a school teacher that decides they want to do some AI
22:27
on the side or are these folks very well versed in AI and and I
22:32
would imagine law too? Yeah, I would say that I mean it
22:36
is the one that we have been in contact with so very well well
22:39
versed in, you know, law. So not, not not, you know, don't
22:43
imagine like engineers that are writing Pythorch code and
22:46
training models. So but this is also needed.
22:50
So I also don't want to be you know, I think that's that, that
22:54
that does make sense because ultimately it is that's the
22:57
goal. You have to have the the end to
22:58
end framework that makes sense from a legal point of view.
23:02
And they're also think, you know, many things we also
23:05
acknowledge are non-technical, right?
23:07
So there are many things that go beyond this.
23:09
So kind of what were we can contribute and help is exactly
23:14
on the technical aspects, how they interpreted them and how do
23:17
you benchmark them, because that's that's really kind of the
23:20
key. And that's that's where we've
23:22
been having our core focus. And so now when you look at
23:26
these different models, the large language model providers
23:29
have anything stick out to you as far as, oh, I wouldn't have
23:34
expected that model to be failing on this compliance issue
23:38
or this toxicity, whatever it may be.
23:41
Yeah, there were a few interesting findings I would
23:44
say. So overall it was the results
23:47
were not kind of terrible. So the, you know, the models did
23:51
perform fairly OK on many of the principles.
23:54
So that was kind of first positive thing that's, you know,
23:57
it's, we're not, we're not in a situation where models are
24:00
really collapsing and dying across the, the, the, the
24:03
benchmarks. But what was very visible is
24:06
first, that definitely a lot of the models are heavily optimised
24:10
purely for, for capabilities and performance, because they
24:13
perform very, very high scores on capability benchmarks.
24:16
But on some of the compliance benchmarks, you do have a, you
24:19
know, big gap. So you're talking that's, let's
24:22
say if it's, you know, on the complex benchmarks, performance
24:24
drops to 0.5 or 0.6, which is really, really, really quite
24:28
low. Another very interesting thing
24:32
was that there, there were some kind of differences if you look
24:34
at close and open models that actually made sense.
24:38
Like if you look at so the ACT has one of the principles is
24:41
that like cybersecurity capabilities, meaning that you
24:45
don't want to leak Proms or other to put other guards and
24:48
defences built into the models. And if you look at the close
24:52
vendors like open AI, for example, so on tropics, so they
24:56
have really spent efforts to make sure that their models are
25:00
secure, which makes sense because you just have an API,
25:03
you don't know what's inside. So you you have to kind of
25:05
protect what's you know, what's behind what's behind this API.
25:09
But then if you look at the open models, like Mistro models, they
25:13
didn't perform so well because which kind of makes sense?
25:16
Like why would you invest so much time to protect the model
25:19
because it's open, it's out, you know, if I have white box access
25:21
to this model, I can make it do whatever I want anyway.
25:25
So that was kind of surprising, but also logical and another I
25:31
think logical thing that we saw for some of the benchmarks that
25:35
have very direct PR kind of implications.
25:38
The models were very optimised, specifically toxicity.
25:41
So because all the model vendors, everybody tried to get
25:46
them to do some, you know, gets aggressive and talk yeah
25:51
nonsense and things like this. This was just basically what the
25:54
media was capturing at the beginning when the models were
25:56
coming out. And they have definitely,
25:58
everybody has spent a lot of effort to optimise because the
26:01
benchmarks perform, you know, the models perform very well
26:04
across these benchmarks. And I, I want to doubt.
26:06
I, I think that's actually very kind of positive sign overall,
26:11
because this means that whatever we are kind of the vendors, the
26:15
AI vendors care about and are aware that, you know, this is
26:18
bad, we have to fix it. We have before we release, they
26:22
actually are, have been able to make solid progress.
26:26
So that's very positive. And that's that's kind of means
26:28
that it's just a matter of kind of providing this global
26:33
visibility to our across all the principles for the ACT and all
26:36
the regulatory requirements. If they have the visibility on
26:39
how things perform, they would be able to also make progress.
26:42
So that's overall very positive to see as a trend.
26:46
Yeah, it's almost like you can't improve what you don't measure.
26:49
So if you're able to measure it, then the improvement will
26:53
happen. But I I also wonder when you're
26:56
talking about these benchmarks, what are some examples that
27:02
you're running the models through?
27:05
Yeah. So these are of course specific
27:07
depending on what whatever the requirement is, but it's let's
27:10
say if you're looking at toxicity as a simple example,
27:14
that could be that would be, there's one on also available
27:17
benchmarks instead of prongs that kind of try to trigger the
27:21
models to output toxic, you know toxic outputs.
27:25
And then it's a matter of just running, running the prongs,
27:27
getting the outputs and then using a toxicity classifier.
27:31
There's also plenty of those from pegging face and other
27:33
vendors to assess like how toxic is the outputs and then that
27:38
makes it then measurable like pretty specific requirements.
27:43
But that's again just one example.
27:45
And many of the other technical requirements that we had to
27:49
assess were really challenging, I would say, meaning that the
27:53
benchmark themselves are very limited.
27:55
So if you think about things like copyright, for example, so
28:00
the model should not be using copyrighted contents to for
28:03
training and you know, you know the examples from, there's many
28:06
lawsuits now from New York Times and other, other organisations
28:10
that are pursuing this. But it's very hard to test this
28:14
because you just don't have a global overview of all the
28:17
copyrighted content in the world so that you can go and assess
28:20
the models. So ultimately what you end up
28:22
doing is, and that's what the current benchmarks do, they
28:25
would take a set of copyrighted books and then you can do, you
28:28
know, specific if you're limited on what you check.
28:32
So what this means is that because it's not the benchmark
28:34
itself is not complete. It's.
28:37
Yeah, you just don't have a complete knowledge.
28:39
So you would have to either rely on other people checking if
28:42
their own copyrighted content has been part of it as a way to
28:44
police this or. Yeah.
28:47
So it's not clear basically how to handle this.
28:50
And there's multiple such benchmarks that's just kind of
28:53
hard, hard to make them fully complete.
28:56
But at least this is a good first starting point that's you
29:00
can get like for and for the basic.
29:02
You know, for the obvious things you could already evaluate and
29:05
see how the models perform. Well, it feels like that could
29:11
fall flat too, specifically with the copyright, because you as an
29:18
end user of a model do not have access to the training data a
29:23
lot. Of the time yeah, but you you
29:25
can test so if you have the model you could still there are
29:28
methods to test if the model has you know if it produces like
29:32
very close to almost like verbatim specific contents and
29:37
there's ways to kind of trigger trigger the models to to
29:39
generate this then you can it's very high confidence.
29:42
And then if you go to chords then you know I don't know how
29:46
they would set up this. Maybe there there would be some
29:48
way to force the vendors to provide access to this data.
29:52
So and that's maybe to take us a side remark is still globally an
29:57
open point. Like how would how would you how
30:01
how much would actually AI vendors expose for compliers and
30:05
general like risk assessments? Because a lot of as you said,
30:09
you know if you just explore the training data that becomes very
30:11
easy. I can just kind of grab the
30:13
training, you know the search and and try to find if if the
30:17
day if they're using like my specific operated content, but
30:22
they would have to expose and they would not want of course,
30:24
because that's very often the core ID kind of thing that you
30:28
want to protect. So but one big trend that we do
30:33
see right now kind of beyond the AI vendors also in generally how
30:37
where the world. Is going so Garter, is it it's
30:40
explicitly actually teaching or kind of mandating procurement
30:43
departments in the contracts. So if you're, let's say you're
30:46
buying an AI solution in the terms to have explicit terms
30:52
that would allow for white box assessments of the models,
30:56
meaning that let's say if I'm buying a solution to identify
31:00
like to read, let's say identity cards or some other, you know,
31:03
critical, critical task, then I would need, you know, that you
31:07
may want to put this in the contracts so that you could go
31:11
and validate properly. And I think that is a way to go
31:15
because you cannot, yeah, you just need white box assessment
31:18
for someone to to conduct like really thorough validation of
31:21
these models. Yeah, unless you are a vendor
31:26
that is using one of these large models in the background and
31:31
then how are you going to, it's like, well, open the eye or
31:34
Anthropic doesn't give me white box access.
31:37
So how am I going to give that to you?
31:40
Yeah. That will be tricky how how to
31:43
resolve this. So this is I think still this
31:45
will take, you know, maybe we when we do another podcast in 12
31:48
months, I think we'll. Get more clear.
31:51
We'll see where things go, but this is one of the kind of
31:54
tension points now that we see because definitely organisations
31:59
demands or the the demands, I mean they don't need the rights
32:04
just to inspect the training data.
32:05
They just need proper access to validate that the models would
32:09
perform if they're used for a business critical use case in
32:12
the, in the organisation. And kind of one way to do it is
32:16
you go to the, you know, you can, you almost have to force
32:19
the vendors to do it and they they are more willing to do it
32:23
with third parties that have limited access.
32:25
But ultimately that will have to be resolved in a kind of more
32:31
invasive way for the vendors, which is possible by the way.
32:34
So you could, so, you know, you don't necessarily have to expose
32:38
like everything to whoever is buying the solution.
32:41
So you could, these are, I mean, it's not like you would go and
32:46
manually start reading the parameters and reading the
32:48
training data. So in any way you're, you know,
32:51
automating the, the benchmarks, the test that you're running.
32:54
So there is a, when we're, we're looking into this kind of how do
32:57
you deploy a specific kind of solution that would run all the
33:02
validation locally at the vendor and then only expose the
33:06
relevant results for compliance or risk or whatever it is only
33:09
that got exposed to the to the client.
33:11
So that's also another way, but you would again have to be, you
33:15
will still need to get the vendor to actually deploy your
33:18
piece of kind of checking software to explore these
33:22
results. Yeah.
33:23
I wonder if we're going to see some type of SoC 2 or SoC AI
33:28
type of certification that will come out and that becomes the
33:34
norm. The norm.
33:35
Like we have SoC two, yeah. Yeah, definitely.
33:38
We do see also for governance. Now there's specific ISO that
33:42
are coming up, but there's still to be determined which ones
33:47
would be the main ones because there is the reality there's
33:49
more than 100 different standards.
33:52
ISO along has more than 25, so it's over.
33:56
It's overwhelming, so you need to have bits.
34:00
Simplify this a bit. All right.
34:02
Well, the EUAI Act definitely defines different categories and
34:09
I'm wondering if you've seen any specific gaps in these EU
34:15
categories like cybersecurity, resilience or fairness or the
34:20
copywriting? So that's a good point.
34:22
So there are things that are just, there's no not like not
34:29
not non checkable, but almost like not, not not well defined
34:32
way how to check. And then we can argue like
34:35
whether then it makes even sense to have this.
34:38
So this would be an example would be the principles around
34:41
interpretability explain ability of the models.
34:44
Well, there's not well defined way to actually technically do
34:48
this. So how are then, how are we then
34:51
going to really properly benchmark and show evidence for
34:53
this? So these are the areas where
34:56
kind of it's inevitable that you need to do like close iteration
34:59
between the high level and the low level just to make sure that
35:02
these things match well together.
35:05
And I think that's a lesson that also future similar efforts,
35:09
regulations and so on will have to be developed in some closer
35:14
way because right now it was just done in a waterfall way.
35:17
It's like, OK, let's define the high level.
35:20
Hopefully this can translate to something meaningful in the
35:23
technical layer. And that's, and there are some
35:26
gaps meaning that not everything is translatable.
35:29
So I think in the future that would be done ideally
35:32
differently. So I think they were just
35:34
rushing basically to it. You know, it's a matter of time.
35:38
Also, you cannot do like everything in a such a short
35:40
period of time. So that's one example where, you
35:42
know, things were just basically could not be translated in a
35:46
meaningful way. And this is now communicated.
35:49
So we'll see what the, the reaction would be on the other
35:52
side. Then in terms of the other
35:54
principle, So I, I mentioned some of them like cybersecurity,
35:57
we saw some gaps there across closed open models.
36:01
We saw that in general, one general trend trend is that the,
36:05
the, the benchmarks themself, the compliance benchmarks are
36:08
just more limited. Like if you look at capability
36:11
performance benchmarks like there has been a lot of efforts
36:15
to define those and these are very good at differentiating how
36:18
good different models are. That's not the case for
36:21
compliance benchmarks. Like if you look at the fairness
36:23
actually most of the model have, most of the models have almost
36:27
equal scores. So that doesn't necessarily mean
36:31
that the models are equally fair.
36:33
It could be just the benchmarks are not complete enough to
36:36
differentiate and show how how the models differ.
36:40
Oh, fascinating. So basically the benchmarks did
36:45
not have, there weren't enough. Samples of.
36:48
Benchmarks, yeah. And it wasn't robust enough.
36:51
And so it's almost like that when the working group gets
36:55
together and starts to do some work.
36:59
And for this EUAI act, that's one of the things that they can
37:03
do. They can dig into absolutely
37:05
more robustness for these these different benchmarks.
37:09
Yeah, exactly because it's, it's in a way it's, it's a, the
37:12
effort goes on two layers. So 1 is just to translate them
37:15
into like what do you need to check?
37:17
Let's say it's robustness, cybersecurity and so on.
37:22
And then the bench. And then you map this to
37:24
specific like benchmark like or evaluation method doesn't
37:27
necessarily have to be a static benchmark.
37:29
It could also be something that you dynamically generate by kind
37:33
of looking at the responses of the model.
37:36
And then the effort needs to be also at that level kind of to
37:39
make them comprehensive enough. And I do think that a lot of the
37:42
focus now would shift towards this because now that there is a
37:47
global realisation that is not just about capabilities, but
37:50
also about safety and compliance, now there's quite a
37:54
lot of effort also on that layer.
37:57
So progress will be made eventually.
38:01
And do you think the safety and compliance is more of an
38:05
afterthought? Like you still have to, you have
38:09
to know that it's capable of doing something before you go to
38:13
the safety and compliance step, right?
38:16
But I think a lot of times, like you said, so much is put on the
38:21
capability. Can we get there?
38:24
And then safety and compliance gets shoved in at the end and
38:29
trying to like, OK, we, we got there, we can see that it works.
38:33
Let's now cross our T's and dot our IS.
38:37
Yeah, I think that I mean, especially it doesn't make sense
38:40
to push this to the to the end. It's like, I mean in code
38:46
security is the same issue, right?
38:47
You don't just write code and then start.
38:49
So you do this from the start. Typically you don't do it at the
38:52
end. It's much more efficient, much
38:54
more meaningful. So and that's ultimately the
38:56
goal here if if the evaluation frameworks, the benchmarks are
39:00
readily available. So you could kind of quickly see
39:03
how you perform then, then it makes sense to do this early on
39:06
because it does affect it, it it does affect how you build the
39:10
model. So you don't want to kind of
39:12
have it ready and then spend like another six months trying
39:16
to reoptimize it in a magical way also for compliance because
39:19
you have to do it. So I think, and I, I think it
39:23
would be ultimately it would be done very early, like what early
39:28
on, meaning that these benchmarks would run early, but
39:31
they just need to exist. And kind of that's the key point
39:33
now. Well, now it does exist.
39:35
So there's no need to, you know, there's no, no need to ignore
39:40
this basically. So it's it it makes it much
39:43
easier for vendors to plug this in.
39:45
And how do you see people using this?
39:47
Is it just that they will plug this in to their test suites or
39:51
their evaluation suites and have it as one extra layer of
39:56
evaluating their whole system? So that's AI guess a more
40:01
complicated question here because it depends like what do
40:04
they want to achieve, right? And you earlier relation also
40:07
like safety and compliance, like what do they actually care
40:10
about? So because a lot of this and
40:13
ideally these things are very well aligned, meaning that so
40:17
compliance basically the regulatory compliance aspects.
40:20
So this just makes certain some things mandatory.
40:22
So typically you want this to be a subset.
40:25
Of the general broad things of safety, things you care about.
40:28
So this would be kind of the things that we all globally
40:31
aligned as we want and you can once like whatever else you care
40:34
about for the business or whatever, for whatever
40:36
motivations. So then, yeah, so this is kind
40:41
of one key point, like what is what?
40:42
What do they want to achieve? So if you do fall into a
40:45
category where you must be compliant, then it's kind of
40:49
easy answer then you know, you just have to plug this in.
40:52
And then how the way you would use it is, is very early on that
40:56
would become not part of as important as the capability
41:00
performance metrics because it's a must do thing, right?
41:05
So that's kind of one key, one use case.
41:07
And here we do expect that both vendors, but also the companies
41:10
that would be organisation that would be auditing ultimately
41:13
these vendors, they would also rely on, they would have to rely
41:17
on such benchmarks because let's say you want to validate an
41:21
application for compliance, you just cannot do it unless you,
41:25
unless you run such a benchmark to get the, to get the real
41:29
technical metrics that's kind of demonstrate compliance,
41:32
technical compliance with respect to the ACT.
41:34
So that would be one use case and kind of the auditors
41:38
themselves, but developers as well, I think they, they would
41:42
also use them for the things again that they care about.
41:45
So again, and we do see this already being a priority, like
41:50
toxicity is already a priority because you don't want to deploy
41:54
a benchmark that provides toxic outputs.
41:56
And now this, this kind of gives you now more comprehensive
42:00
evaluation benchmarks to to pick from to to kind of make an even
42:05
better safer models internally. And do you fear over fitting on
42:13
these benchmarks and not really getting a good understanding?
42:18
Because now that there are benchmarks and people do know
42:23
that regulators are going to be using the benchmarks or or
42:26
auditors potentially are going to be using the benchmarks.
42:29
It's there's an easy way to counteract that without actually
42:32
getting to the root of the problem, right?
42:34
Yeah, that's the it's, I mean it's a challenge also for it
42:40
also determines how you're building these evaluation
42:43
frameworks. So I have seen several, I'm not
42:45
going to name like the providers of these benchmarks, but
42:48
basically they will not allow you to run the benchmark more
42:51
than once. So what is this?
42:53
So you do, you don't need to make sure that, you know these
42:55
benchmarks are dynamic enough so that you cannot overfit like
42:58
this. So it's a kind of responsibility
43:01
on both sides. Also, the benchmarks need to be
43:04
smart enough to not allow you to overfit to those.
43:08
Otherwise you have to either hide them and do some other
43:11
things to ensure that organisations are not
43:14
overfitting to to to those because you could if you have,
43:17
if it's a static benchmarks, I mean, that's not a very hard
43:22
thing to to solve them. It's just it's.
43:25
Been happening for the last two years, right?
43:27
And I think a lot of the sentiment that I've been seeing
43:31
out there is that benchmarks are really kind of bullshit because
43:35
of that, because models just can get over fit or all of a sudden
43:39
it's used in the training data and boom, they pass all
43:43
benchmarks. And then when you give it some
43:47
similar questions but that aren't on the benchmarks, they
43:51
fail horribly at it. Yeah, yeah.
43:53
That's why they should not be only static.
43:54
So they have to really be dynamic.
43:56
Dynamic like looking white box, looking at the model, trying to
44:00
understand like where to go next to to evaluate it.
44:03
And this would kind of lift the barons and makes it make it much
44:07
harder to overpit to them. You have to do this, yeah.
44:11
The other thing that I was thinking about is, do you
44:15
foresee a world where there are different tiers of compliance
44:20
for this? Or let's say that I really want
44:25
to be sure that everything cheques the boxes.
44:29
So I'm going to, I'm going to use the full package and throw
44:33
everything at it. But then there's another use
44:35
case where I want it to comply, but like I want it to be C level
44:41
compliance. I don't really necessarily need
44:43
to make sure that it passes with flying colours.
44:47
Or do you think that it's just kind of going to be 1 size fits
44:50
all? Compliance is compliance.
44:53
Yeah. So that's a tricky 1.
44:55
So I do, I do. I do see, definitely see.
44:57
Compliance is a Boolean thing. That said, the benchmarks
45:00
themselves, the evaluation framework does give a score,
45:03
right? So it can tell you you're robust
45:07
0.85%, you know, that's like specific score you get.
45:11
So you could kind of optimise to some level and kind of the
45:16
question would be then, well, who would set the thresholds and
45:18
what thresholds would be accepted as compliant?
45:20
Once, once you give a threshold is Boolean, you're either below
45:22
or above the threshold. And then you can talk about like
45:25
being more compliant, meaning that you're you, you probably
45:28
given like way beyond kind of the threshold so that you're
45:32
very compliant, right. So that's an interesting
45:34
question. I, I, I don't have an answer on
45:36
how would exactly how would we exactly set the thresholds.
45:41
It is in a way very interesting that the EUA ACT is a very, it
45:46
is really a deep tech regulation.
45:48
It is a technical topic where you actually, I mean, just look
45:52
here at the discussion, we're talking about benchmark
45:54
thresholds and so on, which is kind of unusual for this, for
45:58
this, this, this, this type of regulations.
46:01
So I expect that there would be a kind of a commonly understood
46:06
evaluation frameworks and threshold that's basically would
46:09
keep you safe or keep it compliant.
46:13
But I, it's not very clear yet how exactly that would be
46:17
defined. So I, I imagine that would be a
46:19
most likely a group of the, you know, the organisation that
46:23
would be providing this compliance service, you know,
46:25
they would have to commonly agree on what's doesn't get sued
46:29
for good. Because if you don't get sued
46:31
for, you know, being incompliant, then it's then it
46:33
then it's kind of OK. Yeah.
46:37
But we're not there. We're not, we're not there there
46:39
yet. So, you know, like this is now
46:40
the first time the bench, the evaluation framework was
46:43
created. I mean, this is really the first
46:44
technical interpretation. So at least we have some scores.
46:48
And now we can think about it's, you know, it's very good.
46:50
We can, we're actually talking about, you know, we can talk
46:53
about these things like how much is, how much is enough now that
46:56
it's concrete and measurable. Yeah, It's bridging that gap
47:00
between the law jargon and what we can implement and the actual
47:06
technical possibilities. No.
47:10
And so how does this reflect in what we're doing and what we're
47:13
building? And so it's it, it's nice that
47:15
you came out with benchmarks to be able to say, look on these
47:19
levels that the EU Act has mapped out specifically.
47:24
We have certain ways that you can test for that.
47:29
Exactly, yes. And this is really just to maybe
47:33
kind of abstracted a bit more. So this is really a very, I
47:37
would say, general problem for AI risk.
47:40
And so it's not just the EOAI Act.
47:42
This is just one example, like business risks as well.
47:47
Like you have specific requirements for the model.
47:49
So these tend to be very, very high level.
47:51
And you need to go through this process on mapping them down to,
47:55
to the technical layer. And then like Comply is really
47:58
just one instance on how you solve this, how you take the
48:02
EOAI Act and translate it to technical requirements and you
48:06
know, could create evaluation methods then underneath to, to,
48:08
to measure these technical requirements.
48:10
But it is kind of a problem that you see, you know, for different
48:13
regulations, for different standards for custom business
48:15
requirements. And there things get even more
48:18
interesting actually, and more in harder in a way, because then
48:22
maybe for what we did here with the with the act and what's the
48:25
EU is Commission is doing with these working groups, the code
48:28
of conduct. You know, you can actually sit
48:30
down and do this mapping. But imagine then how you would
48:34
be validating arbitrary models for custom business requirements
48:38
where you don't have this mapping.
48:39
So how would you actually go about doing this?
48:42
Like you would need to have like this expert AI validation teams
48:46
somehow or whatever. Face it, technology would exist
48:50
in the future that would be able to provide this mapping tool to
48:54
validate models properly. You mentioned this right now is
48:58
just for LLMS. Do you see it having a world?
49:02
Because, as the Internet likes to say, everything's going
49:08
multimodal. Do you see it going into the
49:12
multimodal dimensions next? Yeah, for sure.
49:15
So just I guess my answer is stay tuned, but yes, for sure.
49:21
So I mean was the technical interpretation is done, then the
49:26
good thing is this doesn't have to be repeated because you can
49:30
basically then extends the mapping from technical
49:33
requirements to benchmarks or to enter evaluation methods that
49:36
support this multi model or large vision models or whatever
49:40
the type of model it is. So this is kind of the the
49:44
harder part has been done the thing that actually takes one
49:46
year and now it's more about extending it to its additional
49:50
technical capabilities to support more modalities.
49:54
And you have to do this. That's that's clear.
49:56
But we have to keep it kind of more limited in scope.
49:59
So we could we could release something tangible already.
50:03
Before, quote unquote, traditional ML, how does this
50:09
reflect like the EUAI Act? How does that reflect and
50:13
specifically maybe these benchmarks or the way that we
50:16
can take the law jargon and translate it into technical
50:22
actions? What does that look like, if
50:24
any? Have you thought about it?
50:26
Mean for like classic like supervised machine learning like
50:29
how do? You yeah, like a fraud detection
50:31
model or a recommender system or whatever type of thing.
50:34
This also applies there as well. So if you look at kind of more
50:37
broadly at the act, so that you have the, because you have the
50:40
different risk categories for reading, you have high risk,
50:42
high risk. Also there's a bunch of
50:44
requirements, regulatory requirements that do apply.
50:48
There are also covering the same thing this, you know, need to
50:51
adhere to the same principles. So fairness, robustness and so
50:55
on. And then for those, the same
50:58
process applies actually. So you just need to then you
51:01
have a model that's could be just let's say, predicting
51:04
salaries or being used for recruiting software to hire
51:09
candidates. So you would need to then show
51:11
the same, go to the same same principles and then implement
51:15
the same technical requirements, but for for those models.
51:18
So actually what what we did in this work is not just look at
51:22
the general purpose AI sections that talk about this foundation
51:26
models, but look at all the regulatory requirements for high
51:30
risk AI systems and the GPIGPAI models as well.
51:37
And then kind of we took the union of all the requirements
51:40
and then mapped those to technical requirements.
51:42
And then the complete concrete implementation of these
51:45
technical requirements was done for language models, but you
51:48
could do it for the other supervised machine learning
51:51
methods as well, because these methods they do exist as well.
51:56
Yeah. So for example like a
51:58
recommender system, how would that look?
52:01
Yeah. So I mean, very simple example
52:03
would be let's say one of the principles is around robustness.
52:06
So then there's very well defined methods for testing
52:11
robustness for recommendation system.
52:14
So for example, like small change to input should not
52:18
should not dramatically change the output.
52:20
So this you can start testing and getting a concrete score on
52:24
how well the how how robust is the particular model.
52:28
So very similarly again, just you need to do different type of
52:33
methods than than the ones who used for for the generative
52:37
models. Yeah, excellent.
52:41
Well, when it comes to risk, basically AI risk and risk
52:48
mitigation, what do you see as the differences between the two
52:53
of those? Yeah.
52:55
So AI risk in general. So this means that's that what's
52:59
how could this impact in the context of an enterprise
53:03
organisation? So what are the possibilities?
53:06
You know, there's legal risks. We saw this like you deploy a
53:09
model is biassed somebody sues sues sues you for for this
53:15
misinformation. So generally again, back again,
53:17
these are lawsuits that could trigger reputation risks are
53:20
also toxicity was a particular principle that really affected,
53:27
you know, affected the reputation.
53:29
These companies that are ultimately losing their
53:31
evaluations, their evaluations go down.
53:33
Compliance risks is also that isn't a subset of the general
53:37
risk. That's one specific risk that
53:39
you need to be careful with, but I see general risk mitigation is
53:44
a kind of broader topic that covers, you know, all the
53:46
possible including the custom business risks that could be
53:50
that could be impacting the organisation.
53:51
Compliance is only a specific subset that's more well defined
53:55
with more well defined regulations and so on kind of
53:57
this is the commonly agreed thing that we need to satisfy.
54:01
And then in broader risk mitigation, we can add whatever
54:04
else is also could be a potential risk for for for us.
54:08
So for example, one of the AI assessment that we did for one
54:14
of the Swiss banks, they this was classic model that predicts
54:17
prices for cars like specific example, then that was not a
54:22
particular, you know, compliance, compliance risk.
54:25
But of course, it's very, very important that this model
54:28
predicts well accurately the prices because this could affect
54:31
directly affect the business and results loss of money, loss of
54:35
money. So this is kind of custom risk
54:37
that for this model within the within the organisation.
54:41
Yeah. It's almost like you need to be
54:43
able to translate the business requirements into the potential
54:50
risk that you're willing to take on.
54:52
And that's very hard. Yeah.
54:55
Well, it's hard because you need to have also the like the domain
54:58
knowledge for the specific use case.
55:00
Like what does this mean? Right.
55:02
So it's, it's a challenging thing and that's what we need to
55:08
deal with. So ultimately.
55:10
Yeah, it's funny because it's so hard just to translate the
55:14
business requirements into the technical requirements.
55:18
And then also having that other side of, OK, what are the risk
55:24
factors here? So thinking and playing 3D chess
55:28
in a way. Yeah, yeah.
55:30
Because then you, you just need to really understand details
55:33
like in this case, you understand about, OK, this
55:36
price, these predictions effects, the, you know, effects
55:40
you can lose money basically. So then what does this mean?
55:43
Well, then I have to evaluate the performance and kind of
55:45
bounce the gap of the, of the error.
55:49
But then how do you, so you don't keep thinking, well, how
55:51
do I reliably measure the error? Well, then I need to have a very
55:53
good representative data set. So then let's be out a specific
55:57
standard that applies about data quality and kind of check mark
56:01
the data. Otherwise I cannot even have a
56:05
reliable model performance evaluation.
56:07
So you have to go through this creative process that would,
56:11
given all these things that I have done, I have high
56:13
confidence that this is going to be OK.
56:16
So this is something that needs to be more streamlined.
56:21
I would say it just right now it just doesn't exist.
56:23
Like you can say, well, this model can result in XYZ, then
56:27
how do you even get started? Yeah, it's very, very hard
56:31
topic. So you hinted at what you're
56:35
thinking about when it comes to creating the different
56:40
benchmarks or the comply for multimodal.
56:45
What else are you thinking is next for you?
56:48
Yeah. So one big topic is given that
56:51
down now this is available publicly available.
56:54
So definitely working directly with the AI, the big AI vendors.
56:58
So I'm actually, we're, we have, we're starting meetings next
57:02
week to, you know, help them run the framework, explain the
57:06
results so that that's, you know, they're they're clearly
57:11
unambiguously interpreted. And then kind of to calm, not to
57:15
calm the vendors, but just to, to, to, to show them basically
57:18
how you can also optimise the model for compliance.
57:21
So that will be 11 big effort internally.
57:23
That's kind of non-technical. This is just outreach to these
57:26
vendors to to help them out. And the second one is the second
57:30
immediate which step, which has been ongoing is also more
57:34
closely working with the specific working groups within
57:37
the EUAI office that are doing this, because they are like
57:42
literally taking this as a first step, first draught of what what
57:46
they need to do. And ultimately, you know, to
57:49
solve this problem, you need to you see, you know, we are aware
57:53
that we need to refine the the, how the mapping was done.
57:55
We need to expand the benchmarks.
57:57
So it's kind of now going into this next level effort as it's
58:01
not just internal within the organisation that we
58:05
collaborated with, but it's kind of at the level of the EU
58:08
Commission trying to make sure that it's properly, properly
58:11
mapped out. And this is very short term.
58:13
So the output of the code of conduct is I believe April next
58:17
year. So there's literally a couple of
58:19
months until more official interpretation have to exist.
58:23
So they're working on the very, very tight timelines.
58:25
So we're, you know, doing our best also to support in this.
58:29
So if we can use Llama next year in the EUI am going to be
58:35
thanking you for that and if we can't, I will blame it all on
58:39
you. I take it as a challenge.
58:41
Let's go.