Latest / Agentic Conversations (formally mlops.community) / MLSecOps is Fundamental to Robust AISPM // Sean Morgan // #257

Transcript

MLSecOps is Fundamental to Robust AISPM // Sean Morgan // #257 · Agentic Conversations (formally mlops.community)

0:00
I'm Sean Morgan, chief architect at Protect AII.
0:03
Take my coffee black, sometimes Americano, but black coffee
0:07
works just as well. What is going on ML OPS
0:11
community? We are back for another edition
0:14
of the ML OPS Podcast. As usual, I am your host
0:19
Demetrios and talking with Sean today.
0:22
We're talking all things security, not just that boring
0:25
type of security, not that DevOps type of security.
0:28
We're talking ML OPS type of security.
0:31
And Sean's coming at us. There's a whole other community
0:35
out there. It's like our sister community,
0:36
the ML SEC OPS community. So if you vibe with anything
0:40
that's going on here or you want to know how you can be more well
0:44
versed in your security when dealing with machine learning
0:48
and AI, I highly recommend you go and check out the ML SEC OPS
0:54
community. Feels like they may have gotten
0:56
a little inspiration from somewhere familiar on the name,
0:59
but I love it. I appreciate it.
1:02
We tend to talk about security and security vulnerabilities in
1:07
the ML OPS community. These folks talk about it all
1:10
day and all night. And he brought up one security
1:14
vulnerability around Ray that I'm not going to spoil anything
1:18
right now. I'll let him talk about it.
1:21
But if you're using Ray, just make sure you ain't got that
1:24
shadow Ray going on. Let's get into this
1:26
conversation. And as usual, if you enjoy this
1:34
episode, it would mean the world to me if you can share it with
1:37
just one friend. All right, real quick, I want to
1:39
tell you about our virtual conference that's coming up on
1:41
September 12th. This time we are going against
1:44
the grain and we are doing it all about data engineering for
1:49
ML and AI. You're not going to hear rag
1:52
talks, but you are going to hear very valuable talks.
1:56
We've got some incredible guests and speakers lined up.
2:01
You know how we do it for these virtual conferences.
2:04
It's going to be a blast. Check it out right now.
2:06
You can go to home dot ML, OPS dot community and register.
2:13
Let's get back into the show. So basically what I was just
2:23
saying is I was at Cubecon in Paris a few months ago and it
2:28
felt like every other booth or every 3rd booth was security
2:32
related. And for some reason when we get
2:36
into ML OPS as opposed to DevOps which is very cube carn focused.
2:41
But this year as a little aside, if it was everything AI because
2:47
it's the hottest thing out there right now, right?
2:49
But for some reason, when we get into ML OPS, it is a little bit
2:56
thrown on the wayside, I would say, And I don't know why that
3:01
is. You've been talking to a tonne
3:02
of people and you are spreading the good word about ML security
3:08
or there's the dev SEC OPS and then there's the ML SEC OPS.
3:11
I would say yeah, Why do you think that is?
3:15
Why do you think people forget about security for the MLOP side
3:18
of things? Yeah, I mean, there's, there's
3:20
probably multiple factors there. I think, you know, the pace of
3:22
innovation and AI machine learning kind of dictates that
3:26
you build towards, you know, accomplishing things quickly.
3:30
And you know, security is, is often one of the first
3:32
casualties when, when that is a requirement of the organisation.
3:35
Unfortunately you end up paying the, you know, the price later
3:37
down the line. But I, I certainly agree with
3:40
the sentiment that, you know, in MLOP's security is not ingrained
3:45
in the same way that it is in dev OPS.
3:47
And that's part of the message that we're trying to share with
3:49
the ML SEC OPS community is about how you can integrate
3:52
security into every phase of the model development so that, you
3:56
know, at the end of the process, you're not simply trying to, you
3:59
know, slap some security on a model that's about to get
4:01
shipped into production. Yeah, that's so good.
4:04
And what are some ways that you've seen things go haywire?
4:09
Yeah, I mean, there's all different ways.
4:11
So there are the kind of traditional threats that exist
4:14
in DevOps, but also exist in ML SEC OPS.
4:17
So specifically I'm, I'm thinking about like the supply
4:19
chain here. There are all of the library
4:22
dependencies that you're pulling in.
4:23
You know, machine learning and AI is, is foundationally built
4:27
on the open source libraries that we all use.
4:29
But there's more with machine learning, there is the data sets
4:32
that you're pulling in and the foundational models that you're
4:34
pulling in. And you've gather these all from
4:37
repositories that that make it very challenging for the end
4:41
user to really, you know, attest to who made them or how they
4:45
were made or, or scans for vulnerabilities.
4:48
And it just increases the the attacks surface.
4:50
So it it actually in in practise is is more difficult to secure
4:54
the machine learning a life cycle, but but can be critically
4:58
just as important if not more important.
5:00
Yeah, It feels like going back to your first point, it slows
5:08
things down considerably. If all of a sudden now you're
5:11
thinking about, cool, I have X number of data sets and how can
5:17
I ensure that all of these data sets are not going to make me
5:24
regret using them, right. And nine times out of 10 or like
5:28
9.5 times out of 10, it's really not malicious at all.
5:33
And, and you may not even know. This is another thing that I
5:36
would love for you to speak to because I know that there are
5:39
people doing this right now in the LLM training world where
5:45
they're buying Common Crawl domains and then they are
5:50
putting malicious stuff on there because they know that the LLMS
5:53
are going to be trained with those Common Crawl websites.
5:58
And so you've got to think about how do I even combat this and
6:06
how do I, like you said, produce business value, but at the same
6:12
time, don't commit a huge faux pas and do something that could
6:17
really screw stuff up. Yeah, for sure.
6:20
And I would say data sets, you know, kind of as you're calling
6:22
out here are some of the most challenging ones to confidently
6:25
say that this data set is secure because as you mentioned, you
6:28
know, these are terabytes in size, often more.
6:31
And, and to say that there is no poison data or no one that you
6:35
know, snuck something in the common crawl, it is quite
6:37
difficult to do. But at the minimum, what you
6:40
should do is you should attest to how you're creating your
6:42
models and what data sets are going into them in a way that
6:45
allows you to remediate the problem if it become, you know,
6:49
it surfaces in in literature that, hey, this data set
6:52
actually had a problem. Are you able to go find which
6:55
models, you know, you created with that and and and take
6:57
appropriate actions? Is is just kind of the first
6:59
step of a lung, you know, number of layers of the fence that you
7:02
should likely do within ML SEC OPS?
7:05
Yeah, that lineage it and it's so funny because it's not, it's
7:11
not top of mind until something goes wrong and it's the same
7:15
thing. I remember my first job in tech
7:17
was AI was working for a password management company and
7:21
the only people that wanted to buy password management tools
7:23
really were folks who had problems with password sharing
7:29
that went wrong. It's not like it's not, you
7:32
know, the normal people were like, yeah, share my passwords
7:35
or I just use 123456 and it's all good.
7:39
I've never been hacked. It's the ones who have been
7:42
hacked or who have had things leak that have problems and then
7:46
they recognise the value of security.
7:49
And the SEC OPS and the other piece on the data, which I find
7:55
fascinating is that it could be totally malignant for so long
8:01
and then out of nowhere it can come up if there's something
8:05
hidden in that poisonous data or that toxic data, whatever you
8:09
want to call it. I don't know what the technical
8:10
term is for like tainted data that you've trained your ML
8:15
models or AI models on it. It might come up later and you
8:20
don't recognise it and then you have to go sifting through that.
8:24
I just feels like a nightmare. Yeah, it, it, it's certainly a a
8:28
risk, you know, for enterprises, but as as well as hobbyists, you
8:31
know, that are doing machine learning.
8:32
So with respect to, you know, what you can really do about it
8:36
there, there there's different avenues of defence that you can
8:39
put in. You know, I, I know there was a
8:41
talk with ads Dawson from Co here.
8:43
There are runtime protections that you can put, you know, at
8:45
the inference endpoints and things along those lines.
8:48
But the the kind of core principle behind ML SEC OPS is
8:52
that there's no one fix to this problem there.
8:54
There there's a life cycle that that happens as machine learning
8:57
developing is done. And at every step within that
9:00
process, security needs to be, you know, front and centre or at
9:04
least a consideration and, and how you're going about obtaining
9:07
data sets or or moving models productions or red teaming them
9:11
are, you know, various different steps throughout that.
9:15
Yeah. And you did mention there's this
9:18
whole life cycle. How do you see it and what are
9:23
the different security measures you could take for each step
9:27
along the way? Yeah.
9:31
So I, I think that there is the the model builders which often
9:37
you know, can come compose of a different team than the ones
9:41
that actually bring it to production.
9:43
But within that area there there's all types of security
9:48
considerations that you should should make.
9:49
There is the, you know, scanning of static code the same way you
9:53
would do in dev OPS, but also, you know, taking that a step
9:56
farther and scanning your notebooks for for for API keys
9:59
or anything else that may have that type of ability to leak
10:02
this type of sensitive information when you commit it
10:04
to a repository. You know, from there, there is
10:07
the pulling in a foundational models, which is, you know,
10:10
critical with an LLM OPS today. And there's a lot of different
10:15
nuance involved in that. So their first step would be,
10:19
you know, pulling from a organisation or, or, or a person
10:24
that you could attest to actually built it.
10:27
And that's something I'd like to talk about a little bit later is
10:30
around, you know, open source tooling in order to sign your
10:33
model to say that, you know, Meta created this model or, or
10:36
some other organisation. And then the second aspect of
10:39
that is, you know, how are you pulling in the model itself?
10:42
When we look at, you know, models that are up on Hugging
10:44
Face, it takes some time for the actual models architecture to be
10:49
added to say like the Transformers library.
10:52
And so when the state-of-the-art models come out, you know,
10:54
often, you know, if you're using the Transformers library, you'll
10:57
see kind of the use remote code equals true functioning
11:01
parameter. When you pull a new model and
11:05
and you need to understand, you know, what exactly are you doing
11:07
there? You're basically saying there is
11:09
a repository that exists out there.
11:11
I am going to load that, you know, Python module, whatever it
11:14
is, because I trust this source and you know, that's acceptable
11:19
if you know, the source is actually the one that put it out
11:21
and it's not some type of, you know, attack where they're
11:24
pretending to be someone else or or or something along those
11:26
lines. So we've seen this in the wild
11:29
that there's good examples we can talk about, you know, people
11:31
pretending to be different organisations on ugly face.
11:34
And I think that the call out here is that the same types of
11:37
attacks that exist on NPM or Pi, Pi, they, they exist on model
11:42
repositories too. And so you need to be cognizant
11:45
of that and aware throughout the process.
11:48
OK, so that's the new AI world. You've piqued my interest.
11:57
I want to go down this tangent of people impersonating
12:01
companies or different organisations.
12:05
And then I also want to go down the road of what do people need
12:08
to think about with the traditional ML route.
12:10
First, let's get into the juicy stuff.
12:12
What happened? Yeah.
12:13
So I mean you'll see this actively on Hugging Face there.
12:17
There's published research with a number of sources here and
12:19
majority of them are our security researchers kind of
12:22
poking holes into the the model that people are using.
12:25
And so if you go on Hugging Face right now, you know, as of the,
12:28
the recording today and, and you look up Meta Llama, you know,
12:33
Llama 3.1, you'll, you'll see the official model that comes
12:37
from meta. But if you go and you look up
12:39
Facebook Llama, the organisation instead of Meta Llama, that's
12:42
actually a, a security researcher that's put up a, a
12:45
separate model that's been patched for a security issue to
12:48
prove this type of attack. And so often, you know, we hear
12:51
from users that just say, you know, well, if you're pulling
12:54
established models, like what are you really concerned about
12:57
from an attack vector here? It's not that simple.
13:00
There's very, you know, sophisticated and and clever
13:04
ideas you can do in order to get people to, you know, copy code
13:07
on the Stack overflow, pull the wrong model, and execute some
13:10
bit of arbitrary code on your system.
13:14
Dude, we're lucky that it's the nice folks that are trying to
13:20
poke holes in this. I imagine if you get the black
13:22
hats, there's probably, they're probably out there and you can
13:25
probably know stories more than I do.
13:28
But it's funny that it's like we're all so nice to each other.
13:32
We're just poking holes in it just to show that there are
13:35
holes. And when you get the black hats
13:38
coming in, that's what we know. Yeah, we really.
13:41
And again, like that's the hard part about security is that it
13:46
feels like people don't take it seriously until they've got
13:50
their company data held at ransom or until things are
13:55
leaked. And then you have to go back and
13:57
say, OK, we should probably invest in this.
13:59
And even if they do invest in it, they probably aren't
14:02
investing enough. Look like snowflake just got all
14:05
their data leaked, right? That was fairly recently.
14:08
And that caused whole world to hurt for a lot of companies.
14:11
And so hi, it's, it's one of those ones where whatever you're
14:16
investing in security, it's not enough.
14:19
And I. Strongly agree.
14:22
You know, it's one of those things where it's easy to kick
14:24
the can down the road, but but it's really about investing in
14:27
the, you know, products and, and services that are utilising for
14:30
security, but also within your team and upskilling them and
14:34
educating them about the risks and how to manage that and how
14:37
to mitigate that and, and to implement these processes
14:39
throughout the process. And and so earlier to your
14:41
question about where do you implement these, these set these
14:45
steps, you know, we talked about model development, but you know,
14:48
the next part of that is you know, you're moving these in the
14:50
model registries and then you're packaging them up into inference
14:53
containers or are using inference servers and like the
14:55
threats continue to persist throughout the process.
14:58
And and what kind of threats are there?
15:00
Yeah. So for inference servers in
15:02
particular, what you are exposing is, is API endpoints
15:07
for you know users that you think should be using it as well
15:09
as users that that shouldn't be. And so if you look at some of
15:12
the the recent CV ES around, like Pytorch inference server
15:16
or, you know, MLOP, slightly different, but also has API
15:19
endpoints, there is this ability to, you know, compromise the
15:24
server itself. You can do things such as, you
15:27
know, running payloads within the server, but also like
15:30
network traversal. So like, you know, I'm able to
15:31
compromise the server that's, you know, feeding the model to
15:36
the user. And then I can move across and I
15:39
can move across to say, the user profile that's able to make the
15:42
models, which you know, by definition has access to a, a
15:45
treasure trove of data in order to do that.
15:48
So these personas that that do get targeted by machine
15:52
learning, you know, attack vectors have very elevated
15:56
privileges generally and and it makes them a very juicy target
15:58
for the black hats that you that you're talking about.
16:02
So you got any good stories? So there's been several models
16:06
within Huggy Face that that, that we've been reported and,
16:09
and taken down. You can kind of look at some of
16:11
the threat research. There was a pretty widespread
16:16
number of, of organisations that were being name squatted on,
16:19
which I talked about before, everything from SpaceX 23andMe
16:23
where they were uploading these type of models.
16:25
And, and specifically, I believe the 23andMe one had the ability
16:29
to basically scoop your environment credentials and push
16:32
them to a server. So I mean, what payload these
16:35
models can run is pretty much off to the the creator of them.
16:40
At the end of the day, these are binary artefacts that are meant
16:42
to execute something. There's nuance to the file
16:45
formats that that enable different things.
16:47
But yeah, it's it's an active threat in the wild for sure.
16:52
Well, it also what you're talking about feels a lot like
16:58
the end of the supply chain that you had mentioned.
17:05
And so it's all right, cool, we've got the model.
17:08
Now we're going to go and do stuff with that model and put it
17:10
out there. And, and there's the other side,
17:13
which if you're looking at the data side of things as you were
17:18
talking about, all right, where am I getting this data?
17:20
How do I know that it's good? What are some other pieces
17:24
around that? Because maybe you're importing
17:27
frameworks. I think I heard about something
17:29
where they were saying the one of the frameworks that you could
17:33
import was tainted or people were downloading something off
17:36
of Pie Pie and it was turned out to be totally malicious.
17:39
Yeah. So there was one might have been
17:42
a year ago or so for PIE Torch actually, where there was a
17:46
commit branch where you could see that they were going to be,
17:49
you know, using another import as as a dependency of Pie Torch,
17:53
but they had to get claimed the name for it.
17:55
So somebody actually saw that ahead of time, they saw that
17:58
they were going to be adding a new dependency.
18:00
And then they actually went up onto π Pi and they and they
18:05
claimed the name and they put up a malicious package there.
18:08
Luckily, I think it only affected the nightly builds of
18:11
of π Torch. But I mean, nonetheless, plenty
18:14
of people install the nightly's and it was caught before, you
18:16
know, a full release of π torch had had happened.
18:20
But yeah, I mean, the, the, there's really a lot of ways to
18:24
make this this happen. And, and in a space when open
18:26
source is so critical for AI machine learning development,
18:30
you know, that's a great thing that really enables this pace of
18:33
progress. But we have to make sure that we
18:35
secure it. And, and it's a, it's not just
18:38
the security team's job to do that.
18:40
It's it's throughout the life cycle of being cognizant of
18:43
that. Yeah.
18:44
And how do you generally go about it?
18:46
Because I can imagine you, you and all the other security folks
18:51
are seen as the party poopers and you want to like just rain
18:56
on our parade all the time. Especially, as you so aptly
18:59
mentioned, you kind of got to move fast.
19:03
You want to be able to prove business value with your ML or
19:09
AI projects. And a lot of times your ML
19:12
projects are in exploration mode.
19:15
It's not super clear if you've got that ROI there.
19:18
You got to put it out and you got to test like you want to get
19:21
something out as quick as possible to know if it actually
19:24
moves the needle on metrics. But then you have to juxtapose
19:29
it or you have to fight against the security team who's saying
19:34
you can't just put that out there.
19:35
What are you doing? So how do you navigate those
19:39
waters of like pulling people back or letting them run freely?
19:45
Yeah, I, I, it's a great question.
19:47
I think it's funny that, you know, the, the assumption is
19:49
that I'm from the security world.
19:51
So, so generally I, I was a data scientist, I used to work in
19:53
semiconductors. I used to do research on, on an
19:55
algorithm. So I know the pain.
19:58
I, I know the pain of, you know, what you're doing.
20:01
Is it experimental work or perhaps you know, you finally
20:04
really land that model that that is driving the business value?
20:08
And then, you know, for someone to come in and be like, well,
20:10
what was your security process? You know, like, can you, you
20:12
know, give me a, a rundown of everything that went into this
20:15
model is definitely cumbersome. And So what I think we need to
20:19
do is, is, is meet in the middle on these type of tools that, for
20:23
example, don't require, you know, much action or change from
20:28
the end users. So there are, you know,
20:30
different capabilities where you can have basically, you know,
20:35
artefacts stores that have been kind of vetted or a walled
20:39
garden of models that have already been approved by the
20:41
security team perhaps. Or you can put, you know,
20:44
proxies between the end users and hugging face so that the end
20:49
user, the machine learning developers process doesn't need
20:51
to change. And instead, you know, you can
20:54
still give those kind of guarantees with, you know, zero
20:56
code changes to the to the general process.
20:58
So that's one example, but I think meeting in the middle is
21:02
really a key part on on how to get these teams to collaborate
21:04
well. And are you seeing folks like
21:10
yourself understand or be better suited for positions like this
21:16
because you have the knowledge of what the life cycle looks
21:21
like from a data scientist perspective, From a, maybe it's
21:25
AML platform engineer or someone who understands what it takes to
21:29
productionize it, but they also understand the creation of the
21:34
models aspect. Or is there a different persona
21:39
like APM that can step into this role and, and constantly be
21:45
recognising where there's threats, where there's
21:47
vulnerabilities or if more questions need to be asked?
21:50
Because it's, it's less about like, or from what I understand,
21:53
right, is it's less about we can't do that.
21:58
It's more about hey, let's make sure that we daughter eyes and
22:02
cross our T's before we do that. Yeah, 100%.
22:06
And so to the point of the question there, you know, what
22:09
persona do I think is is best to run this?
22:12
I think it varies. I think you kind of aptly call
22:14
out that someone with a, you know, data science or machine
22:17
learning engineering background certainly has the context into,
22:21
you know, how iterative this process can be.
22:24
You know, where the data sources are coming from, the models are
22:26
coming from. And that enables them to really
22:29
understand, you know, what points in the ML OPS life cycle,
22:32
you know, should the security be implemented in?
22:36
The flip side of that is, you know, something that we called
22:38
out is it's not just unique attack vectors, you know,
22:40
adversarial inputs, prompt injections.
22:42
There's also the traditional supply chain and the libraries
22:46
and, and other, you know, containers and things that
22:48
you're pulling in, in which case, you know, a traditional
22:51
Absec persona may have a really good grasp there.
22:54
So we've seen everything. There's a kind of a growing, you
22:58
know, push to these kind of AI security directors or, you know,
23:02
team leads that that own this. Ultimately, I think it's about
23:06
assigning ownership to someone and that's someone kind of owing
23:10
how they're going to go about, you know, facilitating their
23:12
team to do this. Do you have an area or a piece
23:18
of that supply chain that tends to give the most headache.
23:23
Yeah. So I think that today the
23:26
easiest issue to call out is those model artefacts that, you
23:30
know, when you typically work in an enterprise and you want to
23:32
pull down like a binary artefact, there's, there's
23:34
generally a process for it. It usually gets scanned, it gets
23:37
approved by IT. But because of the pace of, of
23:40
machine learning, like you generally just go to on your
23:43
face or in your site, who, who is, you know, performing a very
23:46
valuable service here. But at the end of the day, you
23:48
need to understand what you're doing.
23:50
You're, you're grabbing a binary artefact and you're pulling it
23:52
into your enterprise and running it.
23:55
So that's, that's one that I would say causes the most issues
23:59
and most apparent issues for sure.
24:01
But when it comes to processes like, you know, verifying that
24:05
models are being signed and things like that, I, I think
24:09
that that's coming on the horizon.
24:10
We're starting to see this growth within like open source
24:15
tooling and then OSSF and Linux Foundation where people are
24:18
really pushing towards making this an easier process for you
24:21
to sign your models to verify that they're coming from the
24:23
source that you think it is. And that's where I see, you
24:27
know, a current problem that may be alleviated soon.
24:30
So very much looking forward to. That and are there other open
24:35
source tooling out there that can help scan for
24:39
vulnerabilities or help you like we were talking about with the
24:43
data if you recognise or helping that whole data process to rip
24:48
out different data sources if need be and clean the data
24:53
sources or or any other mix of making sure that you have all
25:00
those i's dotted and T's crossed like we were talking about
25:04
before? Yeah.
25:06
So, so I'll specifically call out kind of the model artefacts
25:08
and then the the trained models and and what you can kind of
25:10
scan there. There are tooling, some really
25:12
nice tooling within Pickle Scan, which is designed to scan the
25:16
default Pi Torch serialisation format.
25:19
There is model scan that also supports Tensorflow formats,
25:22
Caras formats that have these other known model abilities.
25:26
And then as you go about creating the models there,
25:28
there's open source tools like Garack and LLM Guard where you
25:31
can actually, you know, scan these models for
25:33
vulnerabilities. Look for things like
25:35
hallucinations or, or other impacts that perhaps poison data
25:39
training had on your models when it comes to scanning data sets
25:43
itself. It, it is very much an academic
25:46
problem. I used to work on a, on a
25:48
research programme for DARPA around this type of aspect.
25:52
It, it is very difficult. It is very difficult to say that
25:54
that one sample within a, you know, TB file is going to coerce
25:59
a, you know, false information or fake news or something along
26:02
those lines. So I think to your point, like
26:05
is that, you know, where the headache is within the data set.
26:08
It could be, it could be down the line.
26:10
And so we need to make sure that we're, you know, keeping that
26:13
data lineage and pulling from trusted sources and, and making
26:16
sure that, you know, copyrights are in compliance and all these
26:19
other issues that that happened not just from poison data, but
26:22
just from the sources that you're pulling from in order to
26:25
get a model with efficient accuracy.
26:28
OK, so I know you guys are doing a bunch of stuff at ML SEC OPS
26:33
and you've created a whole community around it.
26:35
I think in fact, that's how we met.
26:39
I met somebody from the community in the Databricks
26:44
conference last year and also our good friend of the pod in
26:48
the ML OPS community, Diego Oppenheimer.
26:50
He's very vocal about security vulnerabilities and when it
26:54
comes to ML, and he had mentioned you all to me, and so
26:59
it's cool that you're on here. What are you doing exactly
27:03
though in the community? What are some things that if I
27:07
went there and I wanted to poke around, what would I find?
27:11
Yeah, I mean, you would find training material.
27:14
So we have certificates that you can take and actually, you know,
27:16
validate that your team is is aware of these types of issues
27:20
and they understand kind of exactly how these can be
27:22
addressed. We also have blogs, publications
27:26
and community resources such as Slack channels where you can
27:29
actually connect with other members of the community that
27:33
that are taking this kind of security first mindset and and
27:36
there's a lot to be learned there.
27:37
So happy to hear you know the words getting around.
27:41
That's super cool. Have you?
27:44
What are some stories like? I know that with the MO OPS
27:47
community, I get lots of stories of people asking questions or
27:51
learning stuff or getting jobs, whatever it may be.
27:54
What are some stories that you've got from the community
27:56
and maybe that's them sharing that they were able to stop a
28:00
security vulnerability or learn about something.
28:05
What does that look like? Yes, some of some of my
28:07
favourite shareings within the community are when people
28:09
actually creating new libraries for these type of defences.
28:13
So you mentioned, you know earlier what kind of open source
28:16
tooling is in there. One of the kind of defences for
28:19
LLMS is known as Visual LLM. And I saw a post first for that
28:24
maybe six months ago in the MLOP Slack, Slack channel.
28:28
And since then I've been, you know, following it actively and
28:30
it's a really great programme. So amongst the, you know, shared
28:33
research and, and, you know, ideas that people are, are
28:36
sharing within those channels. I I'm a huge fan of people
28:39
sharing their open source work and, and socialising that.
28:43
Yeah, that's. Super cool to see that,
28:45
especially if folks are sharing the open source work and then
28:49
they get others from the community who jump on board and
28:53
then you can all go and create something really cool together.
28:58
Exactly, Yeah. And that that's kind of the
28:59
goal. So yeah, sure.
29:01
Nice. And have you seen where someone
29:08
completely drops the ball? Like, what are some potential
29:11
consequences if you just throw all inhibition to the wind?
29:16
And you say, yeah, like this security stuff is cool and all,
29:20
but we don't need it. And then it comes back to bite
29:23
him in the ass. Yeah.
29:24
So I think probably the best example of that I have is within
29:27
Ray. So Ray distributed processing,
29:30
you know, machine learning framework, great framework has a
29:33
user interface and that user interface itself has no
29:36
authentication. It is very easy to kind of spin
29:39
up within your local environment and easy to think that it is
29:42
simply, you know, a tool for you and nobody else.
29:46
What we've seen is there was some, several reported
29:48
vulnerabilities with Ray and then since then there's now
29:52
active reports of Ray vulnerabilities where people are
29:55
scanning the Internet to find these user interfaces which
29:59
they're able to attack and, and do privilege escalation and
30:02
everything along those lines. So I'd say, you know, issues
30:05
like that where you, you think that the framework itself, you
30:08
know, as hundreds of maintainers, everybody's paying
30:10
attention to the security and you can trust it.
30:13
They always have, you know, some other aspect of them that that
30:16
likely is getting looked at less, such as a user interface
30:19
or a web interface. So it it's very important that
30:22
we look holistically up at what these kind of libraries are that
30:25
you're pulling in and how you're using them.
30:27
Tude and Ray's got like $100 million company behind it for
30:32
sure. And remember the last round that
30:34
any scale did was really big and that is wild.
30:40
I had no idea. So anybody out there that's
30:41
using Ray, look at your user interface and what can they do?
30:46
What is the patch or? Yeah.
30:48
So if if you read the documentation thoroughly, you
30:50
know they will specifically call out how you can set up the user
30:53
interface so that it can only be contacted by your local host.
30:57
The problem is, you know, not everybody reads the
30:59
documentation thoroughly and and so many people use Ray that this
31:02
actually manifests as as an open problem.
31:05
And it actually was known as kind of the 1st attack on AAI
31:10
workloads that is being actively explored in the wild.
31:12
It's not a shadow ray. You can, you know, do a bit of
31:15
research on your own, but there's been several
31:17
publications about it. So yeah, it's not just, you
31:21
know, small libraries that that don't have the funding behind
31:24
it. It can be the large critical
31:26
infrastructure pieces as well. And again, you know, it's
31:30
important that every team member, everyone who may want to
31:33
spin up a user interface, is aware of what type of security
31:37
risks can exist. I will give it to him.
31:40
Shadow Ray is a great. It is.
31:43
I really like that. That might, that might have
31:46
something to it. We might need to make a shirt or
31:48
something for that shadow ray. So what are some other ones?
31:52
What are some like? Are there common vulnerabilities
31:55
or flaws in other projects or open source tools that I may be
32:00
using besides the ones that we're talking about with, like
32:03
the models being impersonated or the model organisations?
32:07
For sure. So ML flow is is probably one of
32:09
the most actively. Heavily used.
32:13
Yeah, heavily used and also one of the most actively, you know,
32:17
calling out systems, especially the open source version for
32:20
vulnerabilities. Now the data Bricks team is
32:22
fantastic about patching those vulnerabilities and get them
32:24
updated. But if you just search for CV ES
32:26
for ML flow, you'll see a number of different ways in which you
32:29
can, you know, say log a model to the registry and instead
32:35
actually write a file to their disc.
32:37
And then once the files written to the disc, when they get a
32:40
load up a bash shell, you can actually make a remote code
32:43
shell for the attacker. And and at that point you're
32:45
completely compromised. So there's numerous MO flow,
32:51
other abilities, specifically on the server itself.
32:54
That's where we see a lot of these coming from.
32:56
When you have these exposed endpoints, that is where you
32:59
know, hackers really, you know, make their money is when there's
33:03
an easy means to access something.
33:06
You know it helps the machine learning developer, but it also
33:08
helps the malicious actors. It's that low hanging fruit we
33:12
always love to talk about. Yep, for sure.
33:16
The attackers are no different than us.
33:18
It's like how much can I or how little can I work and how much
33:22
can I get from that little amount of work, right?
33:25
So absolutely, I mean, they're they're, they're not in the
33:27
business of doing more work for the same result.
33:29
So yeah, that they're looking for the ways to attack in.
33:33
So explain that a little more because it feels like it could
33:36
be generalised where it's on the server, it's the endpoints on
33:40
the server, So what else? Or how can we generalise that
33:44
just to make it so that in case maybe there's some, we're not
33:48
using ML flow, but we're using another tool and we want to make
33:51
sure to cover for those vulnerabilities also?
33:54
Yeah. So I would say one of the
33:56
primary things you need to do is, is, is you know, continuous
33:59
or frequent patching like these vulnerabilities do get found.
34:03
The company that I work for actually runs above bounty
34:05
programme where we pay researchers to find these type
34:08
of vulnerabilities, but as well as the maintainers to fix them.
34:11
And So what you'll see is that these often are identified and
34:15
they do get fixed. But updating your ML flow server
34:17
is not something that you know, is a a common practise, you
34:21
know, throughout the week. And so once these
34:24
vulnerabilities are known was the CVS are published, everybody
34:27
in the world knows about it or can know about it.
34:29
So at that point you have a vulnerable server that the
34:33
entire in a Black Hat community knows is vulnerable.
34:37
Getting those servers patch is critical.
34:40
So you want to make sure that you're you're doing that
34:41
appropriately. Yeah.
34:44
I guess the key is that if you find the vulnerabilities, if the
34:49
payout for that is nice, then you're not going to be thinking,
34:54
all right, well, let me just do something illegal and try and
34:59
get more money than I would get from exposing this
35:02
vulnerability. Yeah, for sure.
35:04
I mean, I think bug bounty programmes have had a measurable
35:07
impact on, you know, a number of people that may have went down
35:09
the wrong pathway in life and found a way to make a a very
35:14
sizeable living, you know, reporting uses as white hat
35:16
hackers. So I think for sure that's been
35:19
a big impact. Yeah, I'm glad.
35:21
I have no idea how to even start with this because then it's just
35:26
like one less thing that I have to worry about.
35:29
I would get, I would go down rabbit holes and it would not be
35:32
good. So.
35:34
At least get a temptation for sure.
35:36
Yeah, it is. It is definitely a temptation.
35:39
We've been talking a lot about model vulnerabilities and what
35:42
happens when the models out there.
35:43
And that is very machine learning.
35:47
But then there's just like the data stuff that's happening
35:52
before this. And I can imagine there's a
35:55
tonne of vulnerabilities there which may or may not play into
35:59
the ML part part of things, but they're also just as important,
36:03
right? Yes, for sure.
36:05
I mean, you know, the, the models are a function of the
36:07
data that that trains them. And so when you look at, you
36:10
know, the, the data op space and you look for, you know,
36:12
distributed scanning of, of personally identifiable
36:15
information or, you know, bits of samples of information that
36:19
may be incorrect. It it absolutely leads into, you
36:22
know, the, the rest of the process.
36:24
So I, I would say data OPS is, is, you know, within the ML SEC
36:28
OPS domain, it does go end to end.
36:30
It does have specific challenges with respect to scale and and
36:36
how you are going about protecting that.
36:37
But that's something that, you know, I'm really excited to see
36:40
where that goes in the next, you know, five years or so.
36:43
Because simply, you know, legal entities that are now, you know,
36:48
coming after model foundational model training and, and, and
36:52
fine tune models for copyright infringement and stuff are
36:55
starting to put this emphasis that, hey, you know, the data
36:58
lineage that you put into it, as well as the process that you go
37:01
through and vetting it is coming into the spotlight.
37:03
And, and nothing makes people you know pay attention more than
37:06
either illegal challenges or security breaches.
37:09
So, so good. That is so true.
37:13
Oh man. So any before we go like any of
37:20
your own stories on how you potentially learn the hard way
37:26
and what what you learn from that first of all and then what
37:31
happened. Yeah.
37:33
So, so one thing that I would say, you know, back when I used
37:36
to kind of consider, you know, security aspect with respect to
37:40
the data that I was pulling in or the models that I was using
37:42
was when it came to inference time, I would typically, you
37:46
know, be looking for infant servers or even, you know,
37:49
package containers that were ready for this process.
37:53
I, you know, I at the time, I wasn't very much interested in,
37:56
you know, the distributed inference that that that goes on
37:59
with an enterprises. And So what I found is that, you
38:02
know, those containers that you pull, you know, from cloud
38:05
vendors or or other may not be as secure as you think.
38:09
And So what I would recommend is, is you know, at any time
38:12
that you can to scan those containers for vulnerabilities
38:16
within the libraries contained in them.
38:18
You often don't need a novel system for this.
38:20
There's a lot of open source tooling like trivia and others
38:22
that can scan containers. And you need to understand that
38:25
that aspect of the integration as well.
38:28
And and so in my past I I had worked as someone who assisted
38:33
with enterprises deployments. So then to recap, I think one of
38:39
the biggest things and the easiest things that we can start
38:42
doing right now if we are not is scan everything.
38:47
Yeah, take 0 standpoint. Yeah, that feels like if you're
38:51
not doing that and maybe you think you are, I feel like
38:54
there's probably a lot of times where, because, as you're
38:58
saying, there's not really clear ownership on this.
39:03
Sometimes you think it's been scanned potentially, or you
39:08
think that, yeah, after I hand this off, it will get scanned,
39:12
That's what the security team is for or whatever.
39:14
And so a lot of times I can imagine, again, this is my
39:18
little fairy tale fantasyland, which we learned earlier isn't
39:23
the most accurate because I thought that the data was a base
39:25
headache. But in my little fairy tale
39:27
fantasyland, it feels like it can be organisational issues
39:34
just as much as the technical issues.
39:36
And if you're not communicating, and if you don't have that odour
39:38
saying this is what we need to do every time, or here's the
39:44
process we need to go through, then you can let things slip
39:49
through the cracks. Certainly.
39:51
Yeah, you know, it's, it's very easy to to assume that this it's
39:54
somebody else's job. And so when you have that clear
39:57
owner, they can not only take, you know, responsibility for
40:01
making sure that that the process is vetted and running
40:03
smoothly, but also to make sure that it's not, you know, too
40:07
much of A burden that the pipelines that are being
40:10
utilised for deployment or, or registry models into your model
40:14
registry include these scans without the need for every
40:17
single user to have to go through the process of scanning
40:20
it themselves, just simply when they do the.
40:22
Find process, it happens behind the scenes and I think that's
40:25
ultimately where you wanna move your organisation to.
40:28
So #1 rule scan everything number two rule.
40:32
Well number one rule have somebody that owns this number
40:35
two rule scan everything. Create the processes to scan
40:38
everything. Number three rule.
40:40
If you're going to use hugging face, maybe think about what
40:44
were you saying use proxies for it.
40:46
That could be a better. Word that can that can do the
40:48
scanning for you at intercept the traffic and and hugging face
40:50
does a great job of scanning models.
40:53
They they're continuously improving that they they have
40:56
antivirus scan they have the pickle scan that I mentioned
40:59
earlier. So you know, beyond just go
41:04
looking for solutions to solve this for you, go to the hugging
41:07
page, fade hugging face page itself, look at the banners,
41:10
check for how the scans were performed and what the results
41:13
were, which is often a step that that most people kind of forget.
41:17
Yeah, I can't tell you how many times I've just downloaded
41:19
something. Didn't look at anything.
41:21
Just in like, Oh yeah, this is it.
41:22
That's what I want. All right, cool.
41:24
Yep. Happy and paste yes.
41:28
In hindsight, it is incredible that nothing has happened to me.
41:34
Let's knock on wood. And that's what I imagine a lot
41:37
of people that are listening are thinking that same thing too.
41:41
Yeah, for sure. And you know, security is one of
41:43
those things that you you have to get out of.
41:45
You don't want to wait until something bad happens and, and
41:48
you want to, you know, find a way that it is it overly
41:51
burdensome. So there's processes that we
41:53
talk about on MLSAC OPS about how you can integrate this
41:56
security without it being overly burdensome.
41:59
Yeah, yeah. So last key takeaway, if you're
42:02
using Ray, make sure that you are not doing what was it
42:06
called. I can't.
42:07
Remember the name Shadow Ray? Yeah, having you.
42:09
Yeah, what a good name. How did I forget that?
42:11
That is so true. Do not do Shadow Ray.
42:13
If you are using Ray, just look up Shadow Ray and make sure that
42:16
you're protecting against that. Dude, this has been great.
42:18
Sean, I appreciate you coming on here.
42:20
Yeah, of course. Thank you very much for having
42:22
me, Demetrios.