Latest / Agentic Conversations (formally mlops.community) / Cracking the Black Box: Real-Time Neuron Monitoring & Causality Traces

Transcript

Cracking the Black Box: Real-Time Neuron Monitoring & Causality Traces · Agentic Conversations (formally mlops.community)

0:00
But yeah, so I don't see, I don't see the regulations as a
0:03
barrier. They're like, if you get it
0:05
right, they're there reflecting what good practice should be
0:07
anyway. So deal, you know, think of it
0:09
like that. These are the expectations of,
0:11
of, of, of people of good systems that integrate with
0:15
society. So we get that right then we
0:16
should, we should we'll be be able to move, move along a bit
0:19
quicker. Let's talk.
0:27
Regulations, compliance, we're going to make it fun today and
0:32
hopefully help all of the engineers listening understand
0:37
what is important. You have the hidden sexiness
0:41
that lies under the surface of compliance and regulations.
0:47
Yes, so be careful listener. You may fall in love after this
0:51
episode. Why and what have you been
0:56
focusing on in the regulatory space recently, Mike?
1:03
We've been pretty much solely focused on the beast that is the
1:08
EUAI Act, which is a piece of horizontal legislation.
1:13
So it's kind of supposed to be applied across the whole, you
1:16
know, the whole territory of the EU, regardless of what business
1:19
you're in or what your activities aren't.
1:21
And it's kind of the tip of the iceberg because the the ACT is
1:26
like a piece of legislation. So it lays down, this is what
1:30
you got to do. Basically all these are these
1:32
are things we expect you to achieve the standards really to
1:36
not get into any trouble. But that sent me the start of
1:39
the story because kind of hidden away in dark rooms around Europe
1:44
for many years of being lots of experts who've been working on
1:47
what's known as harmonised standards.
1:50
So this is the kind of implementation layer under the
1:53
act and there are thousands of people that have been involved
1:57
with that process over many years.
1:59
And they've been writing these these standards, which actually
2:02
are full of real gold. They tell you if you need to
2:05
comply with like the most stringent rules are in a system
2:09
that's supposed to be biased, unbiased and fair.
2:13
OK. That's everybody acknowledges
2:14
that's somewhere that you'd like to get to, but what does it
2:18
really mean in practice? So you've got all of these,
2:20
these underlying documents that give you a real kind of paint by
2:24
numbers nearly kind of walkthrough of what you should
2:27
focus on in terms of your systems and your models and the
2:31
sorts of things which would, if you can, if you can match the
2:35
standards, you've got a presumption of conformity with
2:38
the ACT. So our product basically is
2:40
reverse engineered against the EU Act, but actually it's
2:43
reverse engineered against this whole raft of standards,
2:46
harmonize standards which sit below it.
2:49
OK. So if I'm understanding this
2:50
correctly, you're saying there's the EU Act, which is very fuzzy
2:55
and interpretable by lawyers. It can go many different ways,
3:00
but there's also a group of folks across Europe that are
3:05
trying to say here's like numerical and kind of tangible
3:11
ways that we can put this legislation into practice and we
3:17
can cover our ass. So that if for some reason it
3:22
seems like our AI is acting bias, we can say, well, we
3:29
followed all of these steps that we were told are best practices.
3:36
That's right. Yeah.
3:37
So you've got this, this phrase called presumption of conformity
3:40
in this world, which is if you do, if you have actually, if you
3:44
can demonstrate that you've followed the standards, then you
3:47
will automatically be complying with the act.
3:49
So all the risk teams and big businesses, they, you know, get
3:52
very hot under the collar about the risk of all this stuff like
3:56
regulatory fines and reputational damage and all that
3:58
kind of stuff. So in terms of how insider ML
4:02
OPS team or a dev team who are putting this stuff, trying to
4:05
get it into production for business value, you've really
4:08
got a road map. As long as you can follow the
4:11
instructions in the standards, then you don't have to stray
4:15
from that. Essentially, if you follow it,
4:17
then your risk team would be happy because you've got this
4:19
presumption of conformity. There's still some
4:21
interpretation from those standards.
4:23
It's not it won't apply clearly in for every single case that
4:27
they could possibly be. So you have to sort of interpret
4:29
it and say, all right, well, this is my reading of how it
4:32
would apply in our particular case.
4:34
So there's still work to do with the risk team and to kind of
4:36
define exactly what these testing evals and sort of
4:41
monitoring systems need to do. But the gap's pretty small.
4:45
So, you know, most people who are familiar with the space
4:48
would be able to, you know, spec out what those tests should be
4:52
to to make sure that you conform.
4:54
But yeah, so it's kind of like a, I think it's just like gold
4:58
really. Nobody really knows about that.
4:59
You just find these standards. I mean, one of the things is, is
5:01
that they are generally you got to buy them.
5:05
So if they're not like they're not like freely available
5:07
because they get produced by standards organisations like the
5:09
ISO or here in the UK by BSI British Standards Institute,
5:13
which it kind of they license out all of their their no hack
5:17
in these in these documents. And you mentioned words like
5:23
evals and observability. How does that way that they
5:29
interpret observability differ from like me in my head, we're
5:35
just observing the systems and we're seeing is the traffic
5:40
going through, is it not? Are we predicting things?
5:44
Are we not? How are the predictions coming?
5:47
Are they drifting etcetera, etcetera.
5:50
Like what kind of observability do you think about too?
5:56
It's more about a kid. So I think there's like the the
5:59
starting point for most engineers who are doing
6:03
observation tasks are we want to make sure our systems work well
6:08
all the time and we're aware of the failure modes and we kind of
6:13
patch for that and we work that in and try and fix it.
6:15
Whereas from the perspective of complying with regulatory
6:21
demands like the EUAI acts and there's loads of others in
6:24
different places, they've got different flavours, they tend to
6:26
all have the same types of themes in them.
6:29
They're coming at it purely from a risk perspective.
6:32
So the idea there would be right.
6:35
We've done an audit of our inventory, our all our AI
6:39
systems and the models inside them.
6:41
And we compared that to, let's say you're deploying a system,
6:44
it's the EU. We then check it to see whether
6:47
any of those systems are in fact classified as high risk because
6:52
there's there's prohibited systems, not supposed to use
6:54
them at all. Then you've got the high risk
6:55
ones and then you've got literally everything else and
6:57
everything else kind of gets a pass.
6:59
It's like, well, if it works, it's fine.
7:01
It's not going to hurt anyone. So the ones in the high risk
7:03
category tend to link to whether you can be discriminated against
7:10
your basic human rights. So you you're not getting a loan
7:13
because you're, you know, you're this type of person rather than
7:16
that type of person. OK, so that would fall foul of
7:18
it. Or if it's a system which is
7:20
already needs some kind of conformity from a Product Safety
7:25
angle. So if it's AAI system that's in
7:28
a car, in a lift, in a toy, in some machinery, then
7:32
automatically it's fast as high risk because it already needs to
7:35
prove it's safe without an AI element.
7:38
With an AI element, you've got to then apply the same kind of
7:41
stringent rules to it. So all of the thinking around
7:45
observations then becomes what's risky here and therefore what
7:50
have we got to track? So if you take the example of
7:53
like a, a loan application that might be biased to some
7:57
applicants, the risk is, is that you are potentially going to cut
8:02
across that human, you know, those, those human values
8:06
measures. So then you identify, OK, we've
8:09
got a system here which is going to be caught by the ACT.
8:12
It's going to be caught by the ACT because it's potentially
8:14
could be unfair to certain people.
8:16
So then you've got to build your monitoring tests and evals
8:19
around trying to catch that. So from a, you know, the, the
8:23
status quo is, does it work well, IE we don't, we don't want
8:27
a bias model. But in this case, the demands of
8:30
making sure that it isn't biased go up several notches so that
8:33
you'd have to kind of engineer to a much higher level of
8:35
granularity. And on that particular point,
8:39
our kind of our secret source with our technology is really
8:41
around in mechanistic interpretability.
8:45
We're getting inside models when they're running inference time
8:47
and we're pulling out those causal chains of decisions so we
8:51
can have a kind of ground truth of what's actually running in
8:54
the model when it spits out an output to say, oh, you, you get
8:57
a loan or you don't. And then we build tooling,
9:00
observation and and testing tooling on the back of those
9:02
rule rule traces. So this may be a bit naive, but
9:09
if I am sending everything to a model provider like Anthropic or
9:17
Open AI, am I not offloading the risk on to them?
9:23
Or is it like, Oh no, they're a black box system and should be
9:27
treated like so and so it is much more dangerous if I have
9:32
them as part of my flow. You inherit whatever their
9:38
regulatory risks are, but you have the disadvantage that you
9:43
can't see it. Sure, they'll produce model cars
9:46
for all you know, for all their models.
9:47
And you can do some due diligence around that and do
9:50
some testing and some drag race type testing with these kind of
9:54
metrics, but they are only input output type tests.
9:58
So, you know, that's pretty sophisticated that world, but it
10:01
won't necessarily comply with the the highest demands of the
10:05
EUAI Act. So in those cases, the risk
10:07
assessment around that when you're planning that system
10:10
might be OK. We don't want to inherit a
10:14
closed commercial model where we can't get access to the
10:17
internals. So we use an open weights model,
10:20
OK. So it, you know, you've then got
10:22
a kind of performance decision to make.
10:24
If we get an open weights model, we have to make sure that it's
10:27
going to do the job well. And there's often a trade off
10:29
between those things. What's the risk of the system
10:33
falling foul of the compliance and the regulations and what are
10:36
the downsides to all of that versus what's the performance
10:39
hit we might take if we have to use an open weights model?
10:43
But you know, there's a, there's more and more people that are
10:45
kind of considering there because a gives you more insight
10:49
into the internal operations of the model, get more control over
10:52
it. And you can, you can get more,
10:56
more flexibility around fine tuning sort of rag based
10:59
systems, that sort of thing. So it for LLNS in very high
11:03
stakes situations like defence, financial services, healthcare,
11:07
there's a kind of a move, I wouldn't say wholesale, but
11:10
there's a growing movement to take the open weights models
11:15
because they've got this level of control and you can then
11:19
prove to the regulators what you're doing with them to a
11:22
much, much higher, much higher level of granularity.
11:26
And so I remember back in the day when we used to call it
11:30
machine learning, there were some folks that would come on
11:35
the podcast that were in the financial services world and
11:39
they would talk about how every time they released a new model,
11:45
there was a whole rigmarole process that they had to go
11:49
through to get it approved. And then there were things that
11:52
they could do once a model was approved and it was able to go
11:56
out into production, but like if they needed to retrain it or
12:02
things like that. But I'm wondering if and how
12:07
that world has changed since the advent of LLMS. Not a lot for
12:14
financial services. That's my my understanding.
12:17
I mean, we've got 2 target markets, financial services and
12:20
defence and we're having early conversations with clients with
12:24
in both of those sectors the in the UK.
12:29
So I can speak more more specifically around the
12:31
situation in the UK. There is has been a tendency to
12:40
lean on and use more linear systems like gradient, gradient
12:44
boosted models, random forests and that kind of thing.
12:47
And they and they've become very sophisticated at doing things
12:50
like fraud and credit decisioning and that kind of
12:52
stuff. But they are inherently more
12:55
explainable and interpretable because they are not, you know,
13:00
opaque models. So they're more linear.
13:02
So they so basically there's been sort of a branch in that
13:05
decision way back to say, OK, well, the compliance tells us
13:09
that we have to be able to stand behind the decisions to a
13:12
certain level of understanding. So this whole class of models,
13:15
like all big learning is off the books because it's just won't
13:18
ever give us that. So they've gone this other way
13:21
historically and have now very good and have made those models
13:23
very, very, very sophisticated and do their job very well.
13:27
But even. Yeah, but there's now kind of a,
13:31
I think a, a drive to sort of say, OK, well, we're, we're
13:36
missing out here. We're we're, we're leaving
13:38
things on the table, leaving value on the table.
13:40
If we had a better model, then, you know, we should be using it.
13:43
So let's try and make that work. And the regulations in that
13:47
space there say sound, sound like a bit of a geek now.
13:51
SS1/23, which is the PRA Bank of England Prudential Regulatory
13:58
Authority, we've basically got a whole piece around explain
14:01
ability and they're saying lime sharp input output kind of
14:05
feature relevance type metrics aren't really good enough.
14:09
You're getting 'cause really, you know, get a variety of
14:12
correlation type insights, but you're not getting causality.
14:15
So you don't know where and when these things might what's going
14:18
on inside. So yeah.
14:20
And because of that same argument around using language
14:23
models, I think I mean, they, they are used, but I think
14:25
people, the risk teams are very cautious about at where they're
14:29
being deployed back office stuff, passing a load of PDFs
14:32
and doing some structured output into some of the part of the
14:35
system kind of OK, you know, it's just kind of just admin
14:39
type processing. But if it gets close to the
14:42
regulatory sensitive areas like customers and bias fairness,
14:46
that sort of thing, then then these they get more highly
14:50
scrutinized. Yeah, I was fascinated in the
14:56
financial sector back when LLMS came out.
15:02
I spoke to one of the lead machine learning engineers at
15:06
Angel List, and he was telling me about how he was using them.
15:12
And he came on the podcast and he was like, dude, LLMS are
15:16
amazing. They're much better than any of
15:18
the models that I built myself. And it's way faster to get into
15:23
production because I just have to hit API versus me figuring
15:27
out the whole platform side of things.
15:30
But I was kind of looking at him like, oh, so are you just like,
15:36
how are you dealing with the compliance side of this?
15:39
Because I know that it's in the financial sector and it it does
15:44
feel like that is dangerous area.
15:48
You're kind of like walking on thin ice the whole time.
15:50
Yeah, Yeah. If it's a driver, I think a big
15:52
driver is this sort of race dynamics is I guess boards and
15:57
people who make investment, investment decisions around IT.
16:01
They are in a race aren't they against the next guy and the
16:04
next girl. You know, we're kind of if you
16:06
were overly prudent and you just didn't invest R&D into this, you
16:11
might end up it could be a company killer.
16:13
So the boards are really kind of, I think quite scared around
16:17
we need to be in the race, but we need to do it in such a
16:19
measured way that we're very aware of what the risks are and
16:25
the compliance risks and just the business risks are falling
16:28
behind. So they're, I mean, they've been
16:29
recruiting, I think quite a lot, especially in the UKA lot of a
16:31
lot of financial services businesses have been beefing up
16:34
their kind of responsible AIAI ethics, AI committee type staff.
16:39
Because historically it's not necessarily a kind of horizontal
16:44
decision team. You know, you've got your risk
16:48
team, business team, your IT or devs team or whatever.
16:53
And the AI challenge in that sector kind of sits across all
16:56
those things. They've kind of been recruiting
16:58
to form forums which can get the expertise into shaping what do
17:04
we invest in, what risks are we happy to take and what kind of
17:08
goals are we setting and the progress against those gold
17:11
verse, you know, vis A vis their competition.
17:17
All right, y'all real quick, let me talk to you about hyperbolic
17:20
GPU Cloud. It delivers NVIDIA H1 hundreds
17:23
at $1.69 per hour and H2 hundreds at $1.99 per hour.
17:32
And this is with no sales calls, no long term commitments or.
17:36
Hidden fees. You can spin up one GPU or scale
17:39
to thousands in minutes with VMS, bare metal clusters and
17:44
high speed networking. You've also got attachable
17:47
storage and you only pay for what you use.
17:51
Save up to 75% less than legacy providers.
17:55
And Oh yeah, by the way, you need steady production grade
17:58
inference. Well, you can choose dedicated
18:01
model hosting with single tenant GPUs and predictable performance
18:07
without running your own hardware.
18:10
Try now at app dot hyperbolic dot AI.
18:15
Let's get back into this show. Yeah.
18:17
Talk to me a little bit more about this stakeholder alignment
18:20
and how you've been seeing technology help in that regard,
18:27
because it does feel like what you're trying to do is help
18:32
bridge a people issue with tech in a way or just make people's
18:38
lives easier to communicate against certain standards.
18:44
Yeah, it's probably the biggest challenge we have I think is
18:47
because there isn't an established sales process
18:51
because it's not really in what we're selling is not really in a
18:54
category yet. So we're trills trying to work
18:57
out who are all of those key decision makers and how do we
19:02
corral them into a decision making group.
19:05
It's not easy. And I think the main, the main
19:07
challenge is historically the business leads in the, you know,
19:12
inside a business, there is a business need.
19:14
We want to invest in an A system to make it happen.
19:21
And they would deal with the tech team and then scope it out
19:23
and they build it. And then later or at some point
19:26
later, the risk team would come along GRC team and start to, you
19:31
know, be seen as the fund police basically and say, oh, no, you
19:35
know, prove me to, you know, prove this model against these
19:38
metrics. And then the scope of the
19:39
project could shrink or there's a load of tech debt that builds
19:43
up because you've got to start wrapping everything in loads of
19:45
additional tests. And if you change any component.
19:49
So they, I don't think historically the cultures
19:51
between those teams have been like, let's just go down the pub
19:53
for a couple of beers at the end of the day.
19:54
They kind of, that's my take on it anyway.
19:57
So we're trying to, we're trying to work, how do we, how do we
19:59
get into that? How do we get into that world
20:02
and make these people love each other dearly?
20:06
So what, what can we give both, both parties something that is
20:09
helpful to the other? That's basically been a
20:11
strategy. So like for the, for the, for
20:14
the tech teams, we're trying to produce materials which explain
20:18
in absolute definite detail you will need eventually to make a
20:22
system of this type. Comply with the following and
20:26
this is exactly the walkthrough in the regulations, which means
20:30
why you've got to do that. That's the kind of the, the
20:32
justification for including that particular thing in your testing
20:36
regime or in the product, basically, or, or in the service
20:39
and then tell them you, whilst it's all very boring, you're
20:44
going to, when you show that to your risk team, they're just
20:47
going to throw their arms around you because the problem that the
20:50
wristing got is exactly the other way around.
20:52
They come into the and they don't really understand the
20:55
nuance of the tech, but they do understand what they systems
20:58
have to achieve. So we kind of give the other
21:01
party the secret sauce, basically all, all the keys to
21:04
the Kingdom. And then when they both meet up,
21:07
then they've got this shared ground, even though they might
21:10
not even kind of internalise all the details, they're armed with
21:15
the right information to say, well, that we put these two
21:18
things together and then we're going to get where we need to
21:19
go. Because I think the, the
21:23
frustration with building stuff that doesn't get into production
21:27
is like nearly taken as taken for granted.
21:30
Oh yeah, we're working on something and we got X amount of
21:33
projects on and this many are in prototype and this many got this
21:36
far and this many got this far. And it's like a, can we can we
21:39
actually get it through and push it to production?
21:42
So everything which doesn't make it, which is still got a valid
21:45
business case is, you know, some cost, isn't it?
21:49
It's apart from what you might learn all the way.
21:51
It's like kind of waste and redundancy.
21:54
So if you can get rid of as much of that as possible and get your
21:57
prototype to kind of production conversion rates, you know,
22:00
higher, then everybody's going to be happy because you're, you
22:03
know, as somebody, as builders, you see your stuff, you build,
22:06
go out in the world and do good things and you don't have it
22:09
knocked down or rescoped as the project goes along because of
22:13
this kind of ugly spiky compliance issue that keeps
22:18
keeps coming in. Well, it does feel like those
22:22
two sides of the business or these different stakeholders are
22:26
almost like at odds with each other and there's different
22:30
incentives there and different goals.
22:32
So they're going to have that friction, but I would love if
22:41
you can walk me through since I've never been in one of these
22:47
companies that is in a highly regulated space and putting AI
22:52
into production in a company like that.
22:56
What is it that the technical team now needs to do?
23:01
I know you mentioned like having a certain set of tests
23:05
understand it. I I'm guessing there's like
23:08
special kinds of documentation that hit criteria, but can you
23:12
give me like a concrete use case and then concrete needs that
23:16
they would need to perform? Yeah, OK.
23:20
We've got a a project which we're scoping at the moment in
23:25
the defence sector, which is for an autonomous surface vessel.
23:33
OK. So it's going to it's like sub
23:35
the classes under 24 metres. So these are kind of near shore
23:40
doing survey, you know, whatever they're doing, they're out
23:43
there, there's nobody out on board and they've got to not hit
23:45
anything. That's the main thing they've
23:46
got not hit. So to scope that project on day
23:51
one, we have to understand what the domain of operation is.
23:59
Once you spec that you then spec what are the prevailing
24:04
regulations that this system it once it's been deployed, needs
24:09
to comply with, of which there are several.
24:12
There's the Maritime and Coast Guards regulations, which have
24:15
got AI sections in them. There's the MO DS internal
24:20
version of that. There's rules of the sea.
24:23
There's lots of. So all of these things are, if
24:26
there were people on that boat or on that vessel, they would
24:29
have to comply with all these regulations.
24:30
You take the people off and there there's it's autonomous,
24:33
it still has to comply with all of these things.
24:36
So sensing visibility. To help me understand.
24:41
Because it's if there were people on the boat, they would
24:45
need to wear life preservers and they would need to have horns
24:50
that they honk. But if it's autonomous, what is
24:55
it on the level of compliance that they need to be doing
25:00
besides just not hit something? I, I'm not, I can't, I'm trying
25:05
to connect those dots, you know, like the code needs to comply
25:10
with all of these different regulatory bodies.
25:13
Idea of what autonomous code should look like.
25:17
No, it's more about what are the, what is the, what are the
25:22
capabilities of the vessel, what's it trying to do.
25:24
So then you specify it's sort of domain of operations.
25:27
So this when we deploy this autonomous vessel, we are going
25:31
to follow operate within a certain area.
25:34
We are going to report our status back to, you know, some
25:38
reporting station every X whatever seconds, minutes,
25:41
whatever, we are going to alert other sea faring vessels to our
25:47
presence. We are going to scan through the
25:50
following sensors, radar, LIDAR, sonar, we're going to ingest GPS
25:58
data of other vessels. So it kind of you, you, you work
26:01
out what it's going to do. So it's very much just like
26:03
speccing any other system. It just happens to be the system
26:05
is a boat on the sea. And then once you know what your
26:09
domain of operation is, you then isolate off the subparts of that
26:12
system. So you say right for the bit
26:14
which is going to be tracking the GPS of other vessels, like
26:19
what what does that sub component of the system do?
26:23
What are the autonomous bits and how do those autonomous bits
26:26
relate to the prevailing regulations?
26:30
So as doing all of that scoping work, you need to do that right
26:33
at the beginning is because it literally leads the rest of the
26:36
project in this case because it's such a sort of safety and
26:40
security focused project. So it's really that kind of
26:43
mindset is that you have to do all of that scoping and
26:47
translation of the regulations and then you map what your
26:52
automated systems are doing. Now some of this stuff is all,
26:54
you know, you've got lots of electronics, electronics on
26:56
boats anyway. So there's these things are
26:58
covered by safety cases and safety tests and certifications
27:02
and inspections. So you're just mapping that kind
27:05
of world onto what autonomous or AI or a gentic type
27:11
functionality within that. But that's the kind of world
27:14
compared to like build something.
27:16
Is it a good model? Does it?
27:17
Is it great at predicting AB or C and then ship it and just
27:20
check it works all the time. When you get into these high
27:23
risk scenarios. It literally is a wholesale
27:25
change mindset change that you don't build a damn thing for.
27:30
At least the first third is all scoping checking compliance work
27:34
in kind of scoping to fit the to the rules?
27:37
Yeah, because you could end up burning a whole lot of money if
27:42
you don't go that way. Yeah.
27:46
And I suppose our our business is predicated on, on the fact
27:50
that whilst there aren't that many systems which get caught by
27:54
the EUAI Act at the moment, it is the gold standard because
27:57
it's people have been working on it since the late teens.
28:00
There's thousands of experts that have kind of fed into this,
28:02
this harmonised standards library that sits below it, that
28:07
even if you're not caught by it, if you are taking a belt and
28:12
kind of a gold, yeah, copper bottomed approach.
28:14
Like we, we don't want unknown unknowns cropping up in our risk
28:20
as a business. So we want to ship the most
28:23
responsible and trustworthy system we possibly can.
28:27
Then you would need to hit these sorts of standards anyway, in
28:30
which case it then starts to make the working practice more
28:34
like I just explained, loads of scope and loads of checking.
28:37
So our business is trying to take that pain away.
28:38
Basically, it's like we can plug into all these different rules
28:41
and regulations and then the metrics can map quickly to all
28:46
of these, all of these very these specifics.
28:48
And then when you're building, you know that you're going to be
28:51
compliant to these higher standards without all that pain.
28:55
Cool. Talk to me more about the
28:57
product then, is it? I come to the product and say
29:00
I'm building an autonomous boat and what do I need to know about
29:08
or how? Like how do I interface with the
29:12
product itself so that it can give me the right regulation and
29:16
map to the standards I need to hit on?
29:20
OK, so the the 1st it's easy is the integration.
29:24
We give you an SDK and that hooks into your deep learning
29:30
model. So you've got to basically have
29:32
your own open weights model that you can get that you basically
29:36
develop yourself or you've got access to an open weights model.
29:39
So the SDK has got some boiler plate in there that just hooks
29:41
in when you run that model. We then pull out the inference
29:45
data, so that kind of raw vector style data, which then depends
29:51
on the deployment. So lots of these higher risk or
29:54
high risk scenarios, they don't want, you know, they want
29:57
complete, they want their hands completely around the data
30:02
governance piece. So no date they want any, any
30:05
data missing. So our product road map, we
30:07
don't do this at the moment, but it's on the road map is it's
30:09
basically completely SAS deployable solution.
30:12
You put it in your world on your client turn, integrate it with
30:16
your models. The data doesn't go anywhere.
30:18
It just is processed by our kit in your world.
30:20
But at the moment we're not there.
30:22
So the inference data comes to us with that rule inference
30:28
data. We then do a couple of data,
30:30
proprietary data transformations which get it.
30:33
Basically it's data, it's a information minimisation is the
30:36
first step. So we squeeze out all of the
30:38
noise and hang on to the signal. So every time the model runs, we
30:42
get an inference log for that particular output, store it.
30:46
We then do this information minimisation and then we end up
30:51
with a trace, basically a causality trace of what's
30:54
happened inside. And that's basically a
30:57
representation of what's happened inside that deep
30:59
learning model when it ran on that particular time.
31:03
You can then run it, you know, in your test 10,000 times.
31:06
Then we'll end up with 10,000 cases.
31:07
And then we start to do post processing on those to analyse
31:11
them, to work out against certain metrics what is good and
31:17
what is bad. So you would end up the goal for
31:19
a user is to define like a golden set in this this system,
31:24
when we build it, normal and expected use looks like this.
31:28
So inputs like this, we expect the outputs to look like this.
31:32
That's gone through, you know, other types of like accuracy
31:35
testing you might have done. And once you're happy with that,
31:37
then you'd run a load of test cases.
31:39
We then you hook our kit in, we drag out the data and then we do
31:43
different analysis over the top to find out whether they are
31:47
robust, accurate, unbiased, whether the transparency
31:52
standards are there, that kind of thing.
31:53
So it depends what what is the specific metric that we're
31:58
testing against basically. And then that becomes your
32:05
basically you end up with an in profile set and an out profile
32:07
set. So normal use look at the trace.
32:11
Fine, this one's fine because it's in profile.
32:14
It's great. Let it go next one comes through
32:16
right. There's something wrong here.
32:17
What's that? Oh, injection for a open weights
32:20
LLM, There was something weird with the input that meant that
32:23
the internal activations in the model just were squiffy.
32:28
And then we can classify what squiffy looks like for certain
32:31
things. And then we can say, oh, that
32:32
looks like it's a prompt injection.
32:34
So we can hold the process at that point.
32:37
So in a live deployment, we'd be able to gate that output because
32:41
we say that red flag, there's some.
32:44
Something up here because it's. Our profile, yeah.
32:47
So that's the. Kind of data.
32:48
Bit and then we've got a SAS front end platform which does
32:53
dashboarding basically, you know, load up your models,
32:56
configure your tests and then report on them in, you know, in
33:01
in graphical format and hang on to all the source logs.
33:05
So the that's how it works. And then your question was about
33:08
how do you tune that to the specific needs of various
33:12
regulations. We're building it to fit with
33:15
all of those requirements for the EUAI Act and the difference
33:19
between those and let's say NIST, which is the American AI
33:23
risk management framework. It's more, it's more in that
33:27
case, it's not the law. It's like recommended best
33:29
practice, but there's so much crossover because they've had
33:33
the same people work on what are the principles here, accuracy,
33:36
safety, security, They all crop up in the same, in the same kind
33:39
of ways that it's only a small tweak very often to to make you
33:43
know what, what is robustness in the through the eyes of the EUAI
33:47
Act? What is robustness through the
33:49
eyes of the NIST AIR FM? They're very, very similar.
33:53
So little tweaks. So these are kind of like a
33:55
layer that sit on top. Yeah.
33:58
So. Basically, to summarise,
33:59
there's. Modules like if you want to
34:01
comply with like, you know, the the FCA in the UK as a financial
34:06
regulator, they got a whole sort of suite of your system must do
34:10
this and must not do this. Basically we would build
34:13
building modules so that they, as a user, you just choose what
34:16
you need and you just plug them in.
34:19
Yeah. What?
34:20
What I didn't? Understand is how you have all
34:23
of this rich data and then you have this compliance or these
34:29
modules as you were calling them.
34:31
How do you map those two together or how do you connect
34:35
those two worlds? OK, so the.
34:39
Thing around like the bias as an example, the.
34:43
Law I'll paraphrase, but. The law basically says when you
34:48
do a risk assessment of your system, if you think that, so we
34:52
go back to the lone example that your system could be biased
34:56
against your basic human rights, which is you're being
34:58
discriminated of because of your sex, gender, race, whatever.
35:02
Then that would then that system would fall foul of the the the
35:06
bias requirements in the EUAI Act.
35:10
In practice, what it means is you have to build a system which
35:13
can identify when bias might be a creep into your system.
35:18
So how'd you do that? Well, if you're using a deep
35:21
learning model as a classifier in some part of your system that
35:26
can be tested. So this is a separate part, you
35:28
know, we can do this or the client can do it.
35:30
You could use a fairly standard assessment to say, OK, is this
35:35
potentially biased? Are there anything is a cause
35:38
for concern around this model as it's operating at the moment?
35:41
If it's yes, then our systems can then go and do this
35:44
internals analysis, which then can report any event, any
35:48
individual output from the model which is out of scope.
35:52
So you have your in scope. These are all normal and
35:54
unbiased because you've you've got a golden set which we can
35:57
kind of get a kind of a like a, a thumbprint, you know what I
36:02
mean? It's like a fingerprint of that.
36:04
So we know the profile of what what normal looks like.
36:06
And then if we have that model run and do a loan application
36:09
decision where compared to what the normal profile looks like
36:16
around these features, yeah, we then have something which is out
36:20
of scope that indicates that that model has now fallen foul.
36:24
So you have to have this logging at that kind of granular level
36:27
because you'll risk management of that system says we must be
36:32
aware to be compliant with the app.
36:34
We must be aware every and any time that the model could be
36:38
biased. So the only way to do that is
36:40
actually to get that information at inference time and make this
36:43
calculation against whether it is potential, you know, whether
36:46
it's in scope or out of scope. And if it's out of scope, then
36:50
if that output goes as far as the customer, you have to save
36:54
all that stuff. And because you never know when
36:56
it's wrong or could be wrong, you've got to save the logs for
36:59
every time that system runs in production.
37:03
Yeah, every time. For every model.
37:04
Because you need that audibility right to.
37:06
Backtrack. Yeah, exactly.
37:09
So that order. Having that chain of custody all
37:12
the way back down to the internal activations inside the
37:14
model is necessary because you will have internal audit, which
37:18
is like your third line of defence in financial services.
37:20
This example who are like, they have nothing to do essentially
37:24
with the operations of the business.
37:25
They walk in internal police, right, We're going to test your
37:27
systems. They need to get access to this
37:30
stuff. External regulators need it for
37:33
either doing investigations or you report this kind of stuff to
37:36
them. Usually in aggregate or, and
37:40
probably most pertinently, any end user who's a citizen who
37:45
gets an output of one of these systems has got the cause for
37:48
contestability and redress. So if they think I'm not sure
37:52
this is right, I feel something's wrong here.
37:54
I'd like to ask a question. So the comparison with GDPR
37:59
would be a subject access request.
38:02
Send me. All the data you have on me.
38:03
Because, you know, I just don't like what you've got.
38:06
And then you can have, you can ask them to remove it.
38:09
You've got the right of removal under GDPR, so in the.
38:13
EU AI actually got. The same thing you can contest
38:16
it and to contest it, the only way to contest it is to get all
38:19
the information and have it presented back to you in an
38:21
explainable format. So it's no good as a as a bank
38:25
who's denied me a loan, sending me a load of logs with vector,
38:30
vector matrices or whatever and go there you are.
38:32
That's why that's why we could give you like you have to then
38:36
build systems which explain in that particular case, what were
38:41
the features in the model, what was their particular, you know,
38:44
characteristics of them as the, the input data, basically why it
38:48
was right or wrong. So you have to build our load of
38:50
explainability kit. There's no, no good just having
38:52
the raw information. You have to then take it to this
38:55
explain ability level. Yeah, so the login, the login is
38:59
like. The is where it all starts.
39:01
You have to be able to grab all, grab everything, otherwise you
39:03
can do everything else. And because you never know when
39:05
something might be wrong, you've got to save it all.
39:09
Are you creating? Different data sets or more data
39:15
or less data depending on the use case or that module that
39:18
you're talking about. Are you logging different
39:21
things? Are you doing it differently?
39:24
Or is it that you just capture all the data you possibly can
39:28
and then depending on the module you're applying it to one or the
39:31
other so you can do so as? Long as you, as long as we save
39:34
those traces, like you get the raw inference data, that's kind
39:37
of step one. Then we do this data
39:39
transformation into our own proprietary kind of data
39:42
structures. Then we save those.
39:44
So we don't need to do the next bit if the client doesn't want
39:48
that. If they say all we want to do
39:49
here is have a complete system of record.
39:52
So every time the model runs, we just want to save it.
39:53
And we what we know that if we get a complaint from a customer
39:57
or a regulator puts a request in to see XYZ, we can then pull
40:01
that from storage. Then we can run the analysis
40:05
over it. Yeah.
40:06
So they don't necessarily have to run all of this stuff in real
40:08
time as like a monitoring thing. It can be saved, stored.
40:11
I mean, this is a decision for the client in line with their
40:14
risk management approach. Really.
40:16
So yeah, you could like the dumb braces approaches that you would
40:18
have everything monitored live. With reporting on all of these
40:22
sensitive areas live and gating the answers live, that's a huge
40:26
overhead and big engineering challenge to to make that work
40:31
or you take a risk based approach.
40:34
Why is it a huge? Engineering overhead.
40:38
Just well, you might have models that are running, I don't know,
40:41
hundreds of 1000 times a day. So the amount of data which has
40:44
been collected and then processed.
40:46
No, that would, that would be great for us because it's a huge
40:48
amount of work and therefore a very, very big contract.
40:51
So, but it's not, it's not intermantable.
40:54
I think, you know, you have to optimize quite a lot of the data
40:57
stuff because otherwise it would just become overwhelming quite
41:00
quickly. So that's part of our of our
41:03
value proposition, I guess is to be able to make it more
41:05
manageable than it would be if you tried to build something
41:07
like that yourself. Dude this is fascinating to.
41:13
Me because it is a whole world that I do not play in at all.
41:17
And I know living in Europe I just hear people complaining
41:22
about the EUAI Act because back. In the day it's.
41:27
Like, oh, we're not going to open source models to Europe.
41:29
That's just a pain in our ass. And so we didn't get, I think
41:32
like Llama 3 when it came out, it wasn't open sourced in Europe
41:36
or something. And so I.
41:42
Have seen. That side of it, I haven't seen
41:46
that in depth side of it that you're talking to me about right
41:50
now, which I feel like is the. Assumption, you know, I'm coming
41:53
for like a free marketeer and I always think like, you know, put
41:55
regulations anywhere that's just a pain and you want to just let
41:58
the market decide. But I kind of think that because
42:01
this technology, this current wave AI has is hit like society
42:07
and and and has become a very immediate thing that they think
42:11
about it. It's like the socio technical
42:13
thing, you know, it's not just technology and it's not just
42:15
society. These two things are kind of
42:17
blending together quite tightly and quite quickly that in that
42:20
space, it's probably it the regulations are just reflecting
42:24
or they should just reflect what?
42:27
Everybody wants, right? I don't want to be in
42:31
interacting with a system which is going to be difficult to
42:34
understand and could be unfair to me or could disadvantage me.
42:38
Or if I wanted to ask why I got an answer from it, it was unable
42:42
to do that. I just think it's common sense
42:44
that whether you call it regulations or whether you say
42:48
this is just the expectations of stakeholders like, you know, the
42:52
users and the systems owners and the companies and all the rest
42:55
of it, then we should aspire to be able to make these things
42:58
work like deterministic software systems.
43:00
Like it's really hard because they're not nondeterministic and
43:04
you will live in a world of probabilities really.
43:06
But we should be able to not shy away from that as a challenge as
43:10
what we're doing is why our company exists.
43:12
We're trying to attack that black box issue and try and make
43:14
it as transparent and explainable and as kind of
43:17
deterministic as possible so that if you have this trust,
43:21
that's then. Kind of accepted.
43:24
Like, oh, we can. Prove things to everybody.
43:26
Then the adoption rates for this technology speed up and the
43:30
compounds. What we might be in danger of
43:33
now is like having all this promise and all this investment,
43:36
but because you can't cross this trust bridge, we're all
43:40
everybody's on one side being a little bit cautious about it.
43:43
And then you don't get the the kind of widespread adoption and
43:46
then you can't compare the benefits, which you know, should
43:49
be amazing for the, you know, medical research and you know,
43:51
all the, all the big problems of the world could be attacked, not
43:54
men necessarily solved quickly, but you can make some great
43:56
progress on lots of them. But you're not going to if the
43:59
you can't bring everybody with you.
44:00
So you need this trust to get in there somehow.
44:03
And you know, what we do is only a tiny part of that.
44:05
There's lots of other things around AI literacy and other
44:08
policies like that that that that will help.
44:11
But yeah, so I don't. See, I don't.
44:12
See the regulations as a barrier.
44:15
They're like, if you get it right, they're there reflecting
44:18
what good practice should be anyway.
44:19
So deal, you know, think of it like that.
44:21
These are the expectations of, of, of people of good systems
44:25
that integrate with society. So we get that right then we
44:27
should, we should all be be able to move up, moving on a bit
44:31
quicker. Yeah, we can't.
44:33
Have no hacks building our systems around here.
44:38
No shortcuts, right? No.
44:42
Well, dude, this has been great. Is there anything that you
44:46
wanted to talk about that I didn't ask you about yet?
44:50
No, I've really enjoyed that. I think that was like other
44:52
right kind of level in terms of, you know, we're we're deeply
44:55
technical. So my Co founder, the 10 year AI
44:58
research career PhD solely focused on AI trust, reasoning,
45:04
explain ability. So he's kind of been in deep,
45:06
deep, deep in that world for a for a long time.
45:10
So we could have done more kind of technical bits, But really I
45:13
wanted to have the opportunity to kind of fly the flag for why
45:16
regulations and compliance is a good thing.
45:20
And that that it's, it's basically a force, force
45:24
multiplier for good systems. If done right, it should be seen
45:28
as like kind of a wall to try and hop over or dig underneath
45:30
or get right on the sides. OK, real quick if you happen to.
45:46
Find yourself in the South Bay on March 3rd.
45:49
We're going to be taking over the Computer History Museum for
45:53
our Coding Agents conference. That's right, we're organizing.
45:57
Another. Conference despite me telling
45:59
myself so many times after we did our AI quality conference
46:03
that I would never do another conference because of the stress
46:05
that it caused. Somehow I seem to have broken my
46:08
vows because the pole was too strong.
46:12
I. Have been.
46:14
Falling in love with all of these new ways to code and use
46:19
coding agents so I wanted to organize a place where we can
46:22
learn from the best in the game at how they are getting the most
46:28
out of them. Some of the notable speakers
46:30
that we have already announced are Sid, friend of the Pod and
46:34
Co creator of Claude Code, Harrison Chase, the founder of
46:37
Lang Chain, Thomas Reamers, the founder of Graphite who just
46:40
sold it to Cursor, good old Dex, the founder of Human Layer and
46:46
the dude who popularized the term context engineering and
46:50
harness engineering. And last but not least, someone
46:53
I consider a very close friend, Mikhail Erik is going to be
46:57
doing a workshop. He's a Stanford lecturer and
47:00
actually he was a past Co host of this here podcast that you're
47:04
listening to, you know, before he got all famous and stuff.
47:09
Come join us. It's intimate by design.
47:11
There's only 450 people that we're going to let in to the
47:15
room to try and keep the signal as high as possible.
47:18
We've got early bird pricing happening until February 1st, so
47:23
I'll see you there.