Latest / Agentic Conversations (formally mlops.community) / Guarding LLM and NLP APIs: A Trailblazing Odyssey for Enhanced Security // Ads Dawson // #190

Transcript

Guarding LLM and NLP APIs: A Trailblazing Odyssey for Enhanced Security // Ads Dawson // #190 · Agentic Conversations (formally mlops.community)

0:00
My name is Add Starset for me. Well, I guess Adam.
0:04
I'm a senior security engineer at Kyle here.
0:07
I will see my coffee anyway I can get it.
0:10
So I moved to North America. I was kind of like addicted to
0:14
the gasoline style coffee you have here.
0:16
You know, which literally just gets the ball rolling.
0:19
But I I've become like a I went to Mexico recently.
0:22
I've become a bit of a snobby connoisseur.
0:24
I I'm really into my Aeropress. My, my, my Espressos by Moser of
0:31
Sucker for cortado. So you know, if it's a early
0:35
morning, yeah, of course I'll go or yeah, anything.
0:39
I I love coffee. I'm a big fan.
0:41
Anything dark roasted is perfect.
0:45
What's going on everyone? Welcome back to the ML OPS
0:48
Community Podcast. I'm your host Dimitri Osen.
0:50
Today we are talking with none other in the ads Dawson who gave
0:55
me a deep dive master class on everything security when it
1:00
comes to dealing with large language models and how your AI
1:03
systems not only large language models I should say Also this in
1:08
a way applies to predictive ML also but he just talked about.
1:15
What you need to be cognizant of when you put these AI
1:19
applications into production from a security standpoint.
1:23
He's done a lot of work on the Oauth system which he talks
1:26
about and if anyone wants to go and check that out or if you are
1:30
doing anything with with AI right now, you should probably
1:34
go to. Lmtop1010.com LLM topten.com
1:43
because that gives you a bit of a checklist of everything that
1:46
you need to be cognizant of when you are creating these AI
1:51
systems so that they do not have any wild vulnerabilities.
1:55
We go into some of the wildest vulnerabilities that he's seen
2:00
on the job. We also talk about what he
2:03
thinks is a potential danger. That is not being talked about
2:09
enough and also how to bridge this pipeline between the
2:13
securities world and the mill world and what we need to be
2:17
thinking about as mill engineers when we are looking at our
2:22
systems and making sure that they are safe and secure.
2:25
Last but not least, this was probably my favourite part of
2:29
it, and it's at the end of the episode when we talk about data
2:33
poisoning and his friend Will, who did.
2:37
As an experiment, he went and bought a bunch of domains on the
2:41
Internet that he knew Common Crawl was going to crawl.
2:45
And I'll leave it at that, because that is what's called
2:49
foreshadowing. All right, I hope you all
2:52
enjoyed it. And if you do, make sure to
2:54
share it with a friend. That would be the best and
2:58
coolest thing that you can do for this podcast.
3:02
We'll see you on the flip side. Take care.
3:11
So my buddy Diego Oppenheimer is he used to own or he's the
3:19
former CEO before they sold to Data Robot of Algorithmia.
3:24
And since like 2020, he's been on a kick, always talking about
3:29
how important it is to think about security and security
3:33
vulnerabilities. And he literally sent me.
3:38
About 3 days ago, something that he called out back in I think
3:42
2021, which was like hey everybody, you might want to be
3:47
careful about just Python packages import or pip install
3:52
this package or that package and he sends me this which says
3:56
hundreds of malicious Python packages found stealing
3:59
sensitive data and I imagine you've seen that.
4:03
So I think we we can kick it off there like that's probably like
4:06
the. The 101 that we can start with
4:11
like the pip installs, be careful with your pippin stalls
4:14
yeah 100% and so on. It's like supply chain
4:17
vulnerabilities is it is its own, is its own realm super
4:23
important in any kind of security like right.
4:26
Traditionally, no we we care about Opsec.
4:28
But I feel like there's definitely a gap between
4:31
security and ML OPS or however you want to traditionalist that
4:35
that sometimes there's a security guy like we don't know
4:40
the balance community is super intelligent, like building
4:43
models and stuff. And that's something I don't
4:45
understand because that's not my background.
4:47
But so it's about understanding from a security perspective.
4:51
Be like, hey you're actually building on Python code.
4:53
OK Like what are you actually dependent on?
4:55
Because like, yeah supply chain vulnerability and sustain dust.
4:59
Although there's arguments about whether they're good, useful or
5:03
bad, like absolute key pillars. And yeah, especially if you've
5:06
got sensitive information, for example, like you if you go like
5:11
sense if you're training on training on private data, then
5:14
yeah, like you could be hosting a day.
5:16
So it's like all about like safeguarding your environment
5:20
and like it completes also supply chain for sure.
5:24
Yeah. So that is the way that we just
5:26
like dive in off on the deep end into this kind of conversation
5:30
that I am expecting is gonna be, it's gonna scratch my own itch
5:34
because I also am very. Interested in the whole security
5:39
and privacy, but mainly security and vulnerabilities space
5:44
because I got my start working for a password management
5:48
company and basically I started like dipping my toe into sharing
5:53
things on LinkedIn in 2018 and all I would do is reshare ohh
5:57
this company got hacked or this this happens.
6:01
So I kind of knew all about the bad shit that could happen, but
6:05
you've been in the game for a while and I think it's worth.
6:09
A little introduction, You're currently working at Cohere you
6:12
are a security and vulnerabilities ninja.
6:15
You're doing that, but you also are doing things with OWASP and
6:20
OWASP made its rounds around the community and if anybody hasn't
6:24
seen that project and what you're doing there, we'll leave
6:27
links to it in the description. But can you give us a bit of
6:31
background on a your work at Coherent and then what OWASP is?
6:34
Yeah, thanks very much. Yeah.
6:36
So I I'm part of the security team.
6:38
OK, here obviously I guess within our security realm we
6:42
have Standard Red Team and Blue team as you would call.
6:44
It kind of emphasises this is my first gig an NLP company, which
6:49
is super exciting. So I get to be part of this
6:51
incredible kind of journey and shifting landscape that we're
6:55
seeing at the moment. But yeah, primarily I'm in
6:58
charge of looking after our application security stock as
7:01
well as our true like our models integrating and the perfect way
7:04
for me to get into that space. We're actually being a part of
7:08
the project. The title is always top ten LMS,
7:13
the applications, the ladder part of that applications is
7:16
important because it basically defines it's not vulnerabilities
7:21
or risks about LMS although it does touch on it, it's to
7:25
abroad, sorry broader scope of how do you safely and securely
7:30
implement those models into your environment.
7:32
So it takes like traditional ways of like application
7:35
security like supply chain securities we discussed earlier
7:39
and how to really develop those models and the projects we just
7:44
actually complete the 1.1 which was start of October and we're
7:49
continuously developing open community And that's been kind
7:54
of a great, great, great story and great chapter for me because
7:58
it's really helped me learn from you know Emma outside and kind
8:02
of kind of like trying to bridge that gap that I feel that we
8:06
have between security and ML alignment, yeah.
8:11
Yeah. What are the things that you
8:13
feel like are different just in the work that you've done in the
8:16
past versus now working with a large language model company?
8:21
Ohh, right. So I mean to be honest like the
8:24
there's there's a few kind of vulnerabilities that may
8:27
overlap. You've got your kind of like for
8:29
example like Dos for example. But how something could like
8:33
something like denial of service could relate to like shadow
8:36
modelling. I feel like it's it's it's
8:39
really, it's helped me understand.
8:41
But also there's a completely different realm because you have
8:44
traditional kind of vulnerabilities, but then you
8:46
also have a separate layer because applications themselves
8:50
inherently have security risks. Those are in.
8:54
But then you're putting an intelligent model on top of your
8:58
application which has has access hooks, you know all these kind
9:03
of realms and effectively that's built.
9:05
That's like spreading your ecosystem, which is spreading
9:08
your attack vector, which when you know when you're threat
9:11
modelling that is giving you increased risk and so many other
9:15
kind of guts and areas that you need to fulfil, especially on
9:18
the model itself. Obviously you care about the
9:20
model quality, because if the model is Paul quality, that it
9:24
might have a subsequent action on the amount of permissions or
9:27
access it has, right? If you've got a vulnerability in
9:30
like how the model can access or pull something like let's say
9:32
like a plugin. If the model quality is not
9:36
there as well, then it may also be susceptible.
9:38
So I feel like there's two parts of it.
9:40
You've got LM application security, but then you've got
9:43
the actual LM secure itself, right?
9:45
Ohh, I see what you're saying there.
9:47
It's like, hey this. LM has this access, but if it's
9:52
not locked down, that access being open can get hacked from
9:59
something else or just by the LM itself doing things without even
10:03
realising that it's doing it. Yeah, like if you've got, if
10:06
you've got like like the application has got
10:08
vulnerabilities or you've got something real simple, let's say
10:11
like it has access into a private database instance or
10:14
something that it shouldn't do which is holding private
10:16
information. But then that LM itself is also
10:19
vulnerable, the prompt injection, which is like another
10:22
kind of the taxonomy. Then effectively you're like you
10:25
have two layers of that not trusted because the model is is
10:30
self not trusted because it's vulnerable to prompt injection.
10:32
But then also you've got the other gap in the environment,
10:35
right? So yeah.
10:38
So, so this is the other piece and it's funny that we're
10:41
talking today because it feels like it's come up in a few other
10:44
conversations that I've been having today.
10:47
And so getting you on here now as the specialist, I get to kind
10:51
of pry and learn what some of the best practises are.
10:56
And really some people were mentioning how if you are fine
11:01
tuning an LM on private sensitive data, then all of a
11:05
sudden that is now open. For everyone to go and have
11:11
their way with and potentially there's a lot of prompt
11:14
injection problems that can come from that or just even data
11:19
access. Like, yeah, we spend so much
11:22
time making sure that the right people have the right access to
11:25
the right data. And then later you just throw an
11:28
LM out there in the wild and boom, anybody has access to it.
11:31
So have you seen good ways of how that is being combated?
11:36
Yeah, I think there's also like a there's a lack of the taxonomy
11:41
between that kind of thing because say a prompted action
11:43
can lead to like Server Side Request Forgery, Cross Site
11:46
Request Forgery, there's there's so many kind of vulnerabilities
11:51
on the LM itself which can then spread into that.
11:53
But yeah, you're exactly right. Are you kind of guess?
11:58
I kind of go back to like the, you know, the layer 8 or
12:01
sometimes the joke about how humans are your biggest element
12:04
of weakness when it comes to security.
12:06
If you've got like a if you've got a LM which is also forming
12:11
that bad practise and you've got problems.
12:13
So I like to think what I like to do is from myself, I do a lot
12:18
of like red teaming. So I like to look at something
12:21
like an external perspective and I kind of threat model which is
12:25
basically assessing and evaluating risk and then working
12:29
from there. So you kind of looking at your
12:30
attack vector and seeing like what your, what your, I guess
12:33
your gaps are and then trying to apply mitigations where you have
12:37
the highest amount of risk and so forth.
12:39
And can I go back to that securing layers.
12:41
So yeah, if you're doing like a private beta in your training on
12:45
private data or you're tuning on that, first of all you want the
12:48
lineage. So where is that data coming
12:50
from? Like what kind of access?
12:52
Like if you're pulling from Slack, let's say let's check the
12:55
token and say like, does it have the permission to pull from
12:58
public channels or private channels, All this kind of
13:01
stuff. Because then if you have a
13:03
prompt injection then you could be leaking something from a
13:06
private channel, which inherently is private because it
13:10
has something in it which you don't want the greater or the
13:13
incident, right? Ohh, such a good point.
13:18
So don't don't give your keys to the Kingdom and then have to
13:22
give something access to be able to pull that right, I guess.
13:25
Yeah, and when you're talking about data lineage, it's it's on
13:29
the training part or it's also in the output part, like when
13:32
the LM is outputting it and they're saying here's the
13:35
reference if it. Yeah, takes a box.
13:39
Yeah. Or or.
13:40
How does that look like? What it that that's it.
13:43
That's a really great question and honestly I I think it's
13:46
twofold. So first of all, like, it's
13:48
almost it. Depending on what you're
13:50
training for and your use case, I feel like sometimes it's
13:53
almost impossible to have that entire lineage unless it's so
13:56
closed and so small the training data that effectively you know
14:00
if you've got something like Open Crawler, you've got no
14:02
chance of like being able to be able to find that.
14:05
But I guess the key is regular kind of benchmark.
14:10
Well I want you search benchmarking cause it's
14:12
different in Mops but insecurity like you know, I guess like
14:16
regular red tea mix sizes and ensuring that you're logging and
14:20
analysing and interpreting the data that goes in and out.
14:23
Because effectively the trust boundary between a client and a
14:27
model is bidirectionally let's say like you don't really, you
14:31
can't trust what's going in and you can't trust what's going
14:33
out. So there's a way that you can
14:35
measure that and analyse that then that's effectively going to
14:38
be able to help you to understand where now citations
14:41
are coming from or where you've got trends.
14:44
But they're also having that that lineage to be able to
14:47
potentially go back and be like, OK, we trained and released the
14:51
model of this week and now we're getting a bunch of content which
14:54
is unsafe and let's kind of like almost like a backup and restore
14:58
kind of method like let's look back and then look and try and
15:01
find what was actually added between those dates.
15:04
So I know this sounds very convoluted, but kind of a high
15:08
level that is able to. If you can provide that then you
15:11
know it gives you that kind of lineage and auditing.
15:14
But again, it's it's hard based on your use case.
15:18
I think I'm going to steal this phrase that you just said, which
15:22
is bidirectionally unsafe. That might be my next album
15:26
name. I'm primarily a network guy, so
15:32
bidirectional is is, yeah, it's just embedded in my brain.
15:36
But yeah, like again, kind of going back to the the threat
15:39
modelling, if I was to perform a traditional threat modelling, if
15:43
I was used like Might or Atlas framework or something like
15:45
that, then I would start start sketching, you know, the
15:50
environment and then look at trust boundaries.
15:52
And that's where you can kind of really evaluate that risk
15:54
because like say, between a client and a model, it's
15:58
inherently not trusted both ways.
16:00
Because like I say, yeah, that's the whole point, right?
16:04
Well, when you're talking about red teaming, I think there is
16:06
something else that comes up where red teaming means one
16:11
thing when you're dealing with a model, and then it means another
16:14
thing when you're dealing with secure systems.
16:17
And a little bit of a tangent that I just remembered.
16:20
I wanted to point out where you were saying, yeah, that the
16:23
cliche is that you're. Your weakest link is the humans
16:26
that are part of the system. And I just remember that going
16:30
back to the password management company I worked for, we would
16:34
always laugh because the meme there was that your passwords
16:38
are 12345 and that doesn't cut it, you know.
16:42
And so this is, it's very much that where you're looking at.
16:48
Your system. And you have the red teaming of
16:52
the models, which is trying to get the model to do something
16:56
that it shouldn't do, I guess. Or, and really this is in the
17:01
reinforcement learning with human feedback, where they're
17:05
trying to make sure that the model isn't going to tell you to
17:08
go and. How to kill yourself, right?
17:11
And so I've heard that described as red teaming.
17:15
But then when you're talking about red teaming, I think I get
17:18
the feeling that you're talking about a whole different type of
17:20
red teaming, which isn't necessarily.
17:23
Figuring out how to not produce toxic output from the models,
17:28
yeah no you're absolutely right. And I guess there is a twofold
17:32
or a multi angle that you can kind of look at it.
17:35
I would say because yeah, effectively we do perform and I
17:40
have performed red teaming on our model itself.
17:42
So you know like trying to you know whether you call it like
17:45
jailbreaking or you try from that angle but then also if it's
17:49
susceptible to that or it may be set table, then you also like, I
17:53
don't know like position right teaming itself.
17:55
The problem with security is that there is actually no
17:59
automated workflow like standardisation which normally
18:03
you can, you've got your traditional like web app
18:05
scanners. That's not everything you should
18:07
rely on when it comes to application security, but you at
18:09
least have a basis. Whereas when it comes to like LM
18:13
application security, it's so specific because you may have a
18:16
model that doesn't even have plugins for example.
18:19
So like why would you try you try attacker plugin.
18:23
So it's all kind of customised and there really is a huge gap
18:27
in the market right now. If you want like I want to do a
18:29
starve LM, red tea ring, yeah. But yeah, so specific like say
18:35
yeah, this traditional attacks that you can see a bunch of like
18:39
gaol breaks and stuff. But then also like say you want
18:41
to be monitoring for toxic citations, toxic content.
18:46
But again, like, you know, if you're talking about red
18:48
teaming, let's say toxic citations for example.
18:52
If you're doing that threat modelling and you're trying to
18:55
on security and layers, then why not take the huge mitigation off
19:00
the LM itself by actually like putting something like, you
19:03
know, like some kind of safe proxy in front of your model,
19:06
right? If you're building a model
19:08
application and it's pulling citations to be able to produce
19:11
outputs for me and you to tell us what weather is tomorrow,
19:14
what all this kind of stuff, just a simple kind of security
19:18
layer that that's one element you could do something like
19:20
that, which is then controlling where it can access.
19:23
Which is kind of what I was trying to lead to earlier when
19:25
we're talking about having that potentially potentially
19:29
unintelligent like brain function.
19:32
But then also like what can it actually pull and access right?
19:36
Yeah. OK, so.
19:39
I have to ask this now because you've seen a lot of shit.
19:43
What's the freakiest vulnerability with with respect
19:47
to LM? Because I'm sure if you could
19:49
talk all day about all the mutability you've seen.
19:52
Like what's something that you've seen with respect to LM's
19:56
and maybe the the top ones or the OR the the top one that just
20:01
stands out. Like, dude, I could not believe
20:03
this one. Yeah so if I if I give a shout
20:07
out it's embraced the right should check out some of this
20:10
stuff. It's it's so a lot of these like
20:12
vulnerabilities are published online once they you know
20:15
mitigated. So that's a that's a great
20:18
reference that I would suggest CC one of the people in that
20:22
space. But yeah I've seen one of the
20:25
things I started looking into this realm because I'm a bit of
20:27
a nerd and a loser. I love to read about lies and
20:30
stuff and I use that for my own kind of knowledge and I kind of
20:34
learn from that. So I get a two way from it.
20:36
But yes, I've seen, I've seen things like prompt injections.
20:42
So for example like the you've got the client and the model,
20:45
the client says to the model, hey summarise this website and
20:49
this website is some kind of website, it's my own.
20:52
And on that website I might have a GIF or an I've got some kind
20:55
of bogus information. But underneath I've got like
20:58
hidden markdown elements which contain like I guess like let's
21:02
say malicious instructions, let's call it.
21:04
So it's got like a gaolbreak on it, or it tells the model to do
21:08
XY. The model reach out pulls the
21:11
citation, is vulnerable to prompt injection.
21:13
So you've got kind of two realms of that's where your problem is.
21:18
Pulls the instructions. It starts, you know, outputting
21:22
the outputting the response to the website like the legitimate.
21:25
It'll literally then perform the takeover and actually start
21:28
performing the instructions and then those instructions have
21:31
then used other plugins to pull information from other websites
21:35
or post information so like it could then make some other kind
21:40
of authorised. So like, let's say you're
21:42
malicious instructions say delete my GitHub repository
21:47
secrets, you have the plugin installed with GitHub on the
21:51
same model. Boom goes the GitHub sends a
21:53
POST request. See you later like like that was
22:00
really like a huge eye opener for me, because it's one of the
22:04
first. It might not be like the
22:06
highest, there's a bunch of other stuff, but it's definitely
22:09
kind of you realise how many moving pieces you have and how
22:13
many, how that could be so severe.
22:16
And it's not only that, but the. It's that.
22:22
You're completely out of the loop.
22:25
And it has so much autonomy and so much agency, no pun intended
22:30
with these agents that it can go so far and do so much without
22:34
you being anything involved, correct?
22:38
Yeah. Like APIs have so much access
22:41
and inherently aren't secure. You put something which is like
22:44
vulnerable, susceptible to like being manipulated on top of it,
22:47
then and which has like full administrator privileges and
22:51
full access. Then yeah, you've got like,
22:53
yeah, it could be really, it could be sticky.
22:57
So it sounds like just the basic basic, basic thing to do is just
23:02
make sure you're not giving those LLM's enough or any
23:08
privileges, or just like the minimum that it needs it.
23:12
Really, look at that, revise that.
23:15
I think I said if you don't mind, I'll use another one.
23:17
So you know, we talked about like the LLM security itself and
23:21
we talked about like toxic citations.
23:23
One thing that you might not like consider which is toxic,
23:26
but it is toxic, right. So I've seen this is quite a
23:30
trending thing at the moment. So OK, your client A I'm client
23:34
A, your client B, client A goes to the model and says hey like
23:40
like I'm having some problems with this MPM package JavaScript
23:44
JavaScript library. OK, it you know, the other model
23:48
tries to help me and then that information then enters the
23:53
actual, I guess like what we would call the data of the
23:56
model, right? That MPM package doesn't exist
24:00
at the moment, let's call it bogus package.
24:03
So then you come to the club, comes to the model for help
24:06
later and and he's like I'm having problems with this
24:08
JavaScript thing or you know I'm having this problem or help me
24:12
with this code. Model tells you to you know
24:17
reference five packages, for example, one of them is my
24:21
malicious package which inbetween your session.
24:24
I've then gone and posted a malicious backdoor inside a
24:29
JavaScript package and then you're you're pound right if
24:34
you're the. Yeah, easy.
24:36
I mean like that is obviously a bit hard, that's harder, but it
24:40
is realistic and it has been done like these, these
24:44
situations are like realise it's not something I'm like this is
24:47
what could happen. It's like I've seen
24:49
exploitations of this in the wild and that's like crazy.
24:52
But the thing about that is also kind of knowledge.
24:54
Is that like you? You would never.
24:57
How would you know that your model is like, hallucinating on
25:02
like like weird packages Because like, you don't know all the
25:05
packages. Like how do you, how do you like
25:07
how do you frame that and be like, so like is this Yeah, it's
25:12
crazy. Yeah, actually it's so funny
25:14
that you mentioned that one because I had heard that and I
25:18
thought it. I didn't realise it was that it
25:20
was planted or someone was training the model to talk about
25:24
a certain package and then they would go and create that
25:27
package. I thought it was like, ohh, a
25:29
model kept hallucinating this package name and then somebody
25:32
was like, well shit, I'm going to take advantage of that also
25:34
into the. Yeah, and then it's sweet.
25:37
Now I've got this, really, I've got this package that I'm just
25:40
gonna create whatever I want from it and people are going to
25:44
get download my package because the model is recommending it and
25:48
we just take whatever the model outputs and be like, all right,
25:51
sweet. And especially on that level
25:53
where you're saying is so true that.
25:56
I think it's human nature in a way, because we're so used to
26:00
dealing with calculators that when you type in a number.
26:04
Times another number on a calculator and you don't know
26:07
what the answer is, but you look at it on the calculator, you're
26:10
like, Yep, that's it. It's a big number.
26:12
And calculate, I'm going to trust 100%.
26:15
That's right. But now all of a sudden it's
26:17
like, Oh yeah, this may or may not be right.
26:20
And we gotta, it's going to be a lot of work to figure out if it
26:23
is right or if it's hallucinating.
26:25
Like might as well just take it at face value.
26:27
And so who hallucinations traditionally aren't really a
26:33
security problem that aren't traditionally dollar security
26:36
problem right. But yeah you're right.
26:37
Sorry I I forgot to mention that that's how it was identified I
26:41
guess because the model is telling you to use bogus package
26:45
and you're like hey that doesn't exist and like I have an idea
26:48
like let's create myself like a botnet army right.
26:52
But also one thing to also mention is over reliance which
26:56
is part of like I guess this is this is not like a it's a it's a
27:01
hard one. But over Alliance itself is a is
27:04
a vulnerability and you are seeing people using models like
27:07
you said all the time like the calculator kind of comparison is
27:11
that people are using models all the time but may not have like
27:15
proficient coding usage to be able to identify that.
27:17
And that's where you have like that huge gap.
27:19
Like you seen companies which have had like the information
27:24
leaked because they've input something like source code into
27:28
the model and it spits out So what else?
27:30
But then as an LLM application developer, you have a
27:33
responsibility as part of security is to ensure that that
27:37
information that is entering is scrubbed.
27:39
It is sanitised from inputting the model, which is another
27:43
vulnerability itself, right? Because yeah, I guess as you're
27:46
you're, you're wanting to protect your stakeholders of
27:48
that model and by doing so that's, you know scrubbing,
27:51
interpreting that data and making sure that sensitive stuff
27:54
isn't from client A and client B is entering the model really.
27:59
Yeah. And that really is the case for,
28:01
hey, I mean, I think that's a great case for Coherent being
28:05
like, hey, we're bringing the model to you and you don't need
28:09
to worry about 20 other businesses using this model.
28:13
It's just yours. Yeah, exactly right.
28:16
So yeah, that that's definitely seems to be a gap, which is,
28:20
which is truly awesome. But yeah, there's definitely you
28:24
date your input again because it's untrusted trust boundary.
28:28
Yeah, you don't. You can't trust it.
28:30
But you also don't want to pull it in because it may be
28:33
malicious or it could be unintentionally malicious.
28:35
You I guess when you talk about projection, you've got direct
28:39
prompt injection, but you've also got indirect prompt
28:41
injection like so. It's like, why is it telling me
28:44
that I didn't ask to? And yeah, exactly.
28:47
Right. Exactly.
28:51
Yeah, but now that you gave me this answer, I guess I'm gonna
28:54
use it like that. You have to be very you have to
28:56
have some high morals to. Not all of a sudden it spits out
29:00
the secret key and you're like, well, calculate calculates it
29:04
so. Yeah, Yep.
29:08
Let's go to Vegas. Maybe we're yeah, we're on so.
29:13
Ohh, that's awesome. And how do you see this moving
29:16
forward? Like, is this going to be
29:20
continuously just like getting gnarlier and earlier as we move
29:26
forward? Because we're going to start
29:28
seeing more and more production applications being built with
29:31
this and then more vulnerabilities because the
29:33
surface area is so wide and that attack vector is so large.
29:37
Is that what you're thinking or do you feel like there's also
29:41
the companies that are coming out with this are starting to
29:43
recognise? Hey, this is a really serious
29:47
problem and we see these ways of getting ahead of the pot.
29:53
Yeah, yes, you're right. I think it's both.
29:55
I think like you know companies are wanting to companies do cat
30:01
like they they care about their customers and do they like
30:03
that's what customers are caring about Because I feel like this
30:06
space is becoming more well known.
30:09
I think when you know AI is been going for a long time, but this
30:13
shift in like our, you know our landscape is like and The thing
30:18
is they are rapidly developing at the moment.
30:20
Hey, Laszlo here. If you're serious about Emma
30:23
Lops, you hit subscribe right now the problem with security
30:32
and adapt adopting it into LM applications because there's so
30:36
many LM applications which are there right now which don't
30:39
inherit any security controls. There's such a like a race I
30:43
guess in, you know, AI itself between between companies that
30:48
moving so quickly is so hard to adapt when you're trying to
30:51
apply security, but you're also trying to produce the best
30:54
product so that you're definitely trying to.
30:56
We're in a bit of an arms race, as it were, but you know, you're
31:00
definitely trying to trying to meet both ends.
31:03
But yeah, there's definitely, definitely a need.
31:08
And I think these attacks and things will just start to become
31:12
crazier and crazier and Malia and you know, you got open
31:16
source models as well, right? Which are fantastic, yeah.
31:18
But then that also is increasing the attack vector because you're
31:22
going to have so many stakeholders, clients, they're
31:25
going to be using applications published on, publishing public
31:29
repos, things like that. So, yeah, it's, it's, there's
31:33
definitely, definitely a shift in landscape, but I feel like
31:35
there's definitely red teaming foot for LMS itself or LM
31:41
applications. It is like say, as I was saying
31:44
earlier is a custom thing right now and it's so hard to identify
31:49
and it's so hard to standardise that into your pipeline as you
31:52
were even like you like CICD for example.
31:55
There's so many things you can do, but then it is so specific
31:58
that there's definitely a need for that as well.
32:02
And you mean it's so handmade because of the different use
32:06
cases or just because of the different systems that each
32:09
business has or the different models like what makes it so
32:12
custom? Yeah, I'd say both because some
32:15
models have been trained on X data.
32:17
So theoretically, like there may be situations where you you're
32:22
not, there's no concern about what it's.
32:25
It's a hallucination about something that is public because
32:29
you've only trained on private data, for example, or or like or
32:33
getting something like prompt injection.
32:35
You know, like kind of knowledge like that was talking about
32:37
earlier where he's pulling malicious instructions on the
32:39
website. If you don't have like if you
32:41
don't have an application or an agent which has like rag which
32:45
is able to pull out citation, then that's not a risk to you.
32:48
So therefore implying that in your Red team means, I would say
32:53
it's not as worthwhile in that case because that that's what
32:57
I'm trying to mean by the custom kind of thing like say you
33:00
you've got different different use cases, different data that's
33:03
been trained on different, different features or plugins.
33:08
I guess it's kind of the word that most people are familiar
33:11
with, whether it has access to the Internet or not, whether it
33:13
has access to private data stores.
33:16
For different apps, yeah. Well, so that's another
33:20
fascinating piece is like, how do you see security changing
33:23
depending on the different? Use cases or maturity of your
33:29
LLM application. Like if you're using a RAG, how
33:31
does the security need to change and what do you need to be
33:34
thinking about versus just like chatting with chat with your
33:38
data type thing? Well, I guess the chat with your
33:40
data potentially is going to be using a RAG, but then just like
33:44
a off the shelf messing around with a chat bot LM, yeah, I I
33:49
think there's there's definitely an opening of taxonomy now.
33:53
I feel like like we within the iOS projects, we were definitely
33:56
able to get like a building base.
33:59
It was the projects themselves. Even the same for traditional
34:03
APIs or mobile applications. It's not like a settings there
34:07
and that shouldn't be like the Bible that you base on because
34:10
like I said when you've got when you're actually like assessing
34:14
risk and you've got something custom like a web application,
34:17
then you may not be susceptible or vulnerable to certain
34:20
elements. So therefore those aren't
34:21
adoptable to you. But I feel like in the in the
34:25
OSS project we would definitely first able to kind of get like a
34:28
standardisation and kind of build that taxonomy.
34:30
But one thing I think from like you know from vendors or whoever
34:35
this good you know, these vendors of this of this space
34:38
needs to identify that taxonomy between even like bud brownies,
34:42
right. We've seen like 11 or NLP
34:44
companies on bug bounties right now like defining that taxonomy
34:48
of how prompts injection could also lead to like cross site
34:51
scripting which is you know traditional API security
34:54
problem. So there's definitely,
34:56
definitely an open space for that.
34:58
Role so for the uninitiated. The SEC OPS people out there
35:05
like OWASP, what it is, what the?
35:09
Sorry. What the project that you did
35:12
with it is and go deep into that.
35:14
Yeah, I was top 10 for. Yeah, yeah, cool.
35:18
So is a is an open foundation kind of well known in in the
35:23
security community. They have things like what's
35:27
known as the ASVA, which is the application security
35:30
vulnerability. Something a tip?
35:35
Ohh, nice. What's the important one?
35:36
Yeah. So yeah, the last last actually
35:39
is something but they're basically they have a a set of
35:42
frameworks which is to aid developers and engineers on how
35:47
to build like secure software. So what we did from that is that
35:52
I was a foundation or framework has multiple projects for
35:56
different things like you've got traditional web applications
35:58
which could be REST APIs, GRPC for example, but then you've
36:02
also got a mobile applications. Well, what we did here is that
36:05
we tried to develop an L top 10 for LLM applications, which is
36:11
taking the elements of building a secure application, whether
36:15
that's like something like a web app or something, but then say
36:19
additionally laying that LM on top of it, right?
36:21
So like you've got your traditional things like, I don't
36:25
know, denial of service, which is like a traditional like web
36:29
application scare. You want to make sure you have
36:31
things like rate limiting in place, but also with your model.
36:35
I guess that same element element applies where you don't
36:39
want your model to become available for lack of rate
36:42
limiting. But then also the lack of rate
36:45
limiting on like a model could also relate to things like
36:49
shadow modelling, which also then you've also got other
36:52
problems. So what we try to do is try to
36:54
build on top of that. So some of the, I guess some of
36:58
the top ten which is like just I can open standard, no standard,
37:02
but it's more of a reference for people and how to build secure
37:06
LM application. Is that like, hey, these are
37:09
like the top things you should look out for based on what we've
37:12
seen, what we've know, what we can interpret and what we're
37:15
kind of worried about based on those, based on those things.
37:20
Ohh, nice. So we've we've published what
37:22
seems like a top ten, which is effectively really simple as LLM
37:26
topten.com. It's effectively just 10
37:30
vulnerabilities which we deem the highest.
37:32
They were voted by the community, it's the highest and
37:34
we have our submissions. We continue to develop.
37:37
That list is not hard set, you know there will be removals
37:41
editions as there is with every iOS top ten, but there's also
37:44
other spaces, other white papers, things like that.
37:47
But it's just like a free resource and community through
37:52
people still contribute and it's been very honestly I've learnt
37:56
so much. We have people from all kinds of
37:59
backgrounds, communities where they have no experience like an
38:05
emo OPS ninja, whether they're security person like me and just
38:08
like totally entering this like crazy world or vice versa.
38:12
So it's really building or bridging that gap, which is
38:16
great. And so it is a fully open
38:19
project and it is community driven and it's LM top ten, like
38:25
10 or T10, yes, good point top 10.
38:32
OK, one, zero, yeah top and that.
38:35
And that's awesome because anybody can go.
38:37
So for me, I'm messing around with my system and tomorrow I
38:42
realise ohh shit like I just found a big vulnerability when
38:47
dealing with agents or something like that.
38:49
I can go and submit that to the project and let people vote on
38:54
it, yeah, Yeah. So example like excessive agency
38:58
is actually one of the entries. So what I'll have like if you go
39:02
into the actual vulnerability itself.
39:04
I forgot to mention the project is also accessible from
39:07
oauth.org which is like the main framework website as it were,
39:11
but CSA to navigate straight to it.
39:13
But you'll have an introduction about basically this is what it
39:16
is. Then there'll be like common
39:19
examples of the vulnerability. So this is like if you don't
39:22
apply no secure practises that are listed in the introduction
39:27
or the description then this is kind of vulnerabilities or risks
39:32
you can see based on XYZ. Then it will have like this
39:36
example attack scenarios. So like how someone would do
39:38
something like that like you know we talked about like the
39:42
client A&B earlier how you could control that in those kind of
39:45
security layers and then how to prevent select steps which are
39:49
recommended to you based on how to mitigate those threats.
39:53
And then there's a few reference links in there where there's
39:55
been like a public right of about whether it's like it's an
39:59
exploit in the world that we've seen or seen someone wearing a
40:02
white paper on how to apply that best practises to that.
40:05
And they're gonna ability No problem explaining that well.
40:08
But yeah, it's basically like a TLDR of what these
40:12
vulnerabilities are and how they're exploited and how to how
40:16
to, how to defend against it. It's basically.
40:20
The minimum. That you need to have in place
40:24
if you're building a system with.
40:26
Yeah I yeah, I think like if you if I kind of go back to the O
40:31
top 10 for web applications, separate projects that is
40:35
foundationally used by pretty much any developer secured
40:40
people which is building who are building a web application based
40:43
that's where they start. That's their basis that's like
40:46
their checklist. You see like a lot of compliance
40:49
frameworks being like do you comply with the oath top ten
40:52
which means that you've assessed and mitigated risk across those
40:55
top ten. You shouldn't just leave, you
40:58
should be limited to that top 10.
41:00
But that's kind of like it's become like the de facto
41:03
standard or the benchmark that you should be reaching, which is
41:07
what we're trying to achieve with the top ten LM
41:11
applications. And.
41:15
What are some things that you would like to see changed within
41:17
that project? A great question.
41:20
So what? The way how we come in triage
41:23
and the work is we deal with catalogues, so we're on GitHub
41:27
as well. They're repositories on GitHub.
41:30
I won't give you the link for that because it's crazy.
41:32
Well if you go to like GitHub and put iOS top ten or use a
41:37
search engine or whatever, we we we deal, we we work in GitHub
41:41
issues. So effectively like we have a
41:44
bunch of preterm labels. I'll raise an issue with that,
41:46
like an enhancement, a recommendation or just something
41:50
that I'm concerned about, whether I'm like, hey, I don't
41:54
think it's fun, abilities, really relevant as like this.
41:57
Or I don't think we've covered certain scenarios in here.
42:01
And we have what's known as like a core team where there's
42:04
certain people attributed and dedicated to those individual
42:07
vulnerabilities. They will then kind of work with
42:10
you and develop that. So I've actually submitted it to
42:13
you myself, Repository I'm, I'm kind of my beliefs.
42:18
I'm also a fan of getting, you know, maybe streamlining our top
42:23
ten and, you know, a few additions to that based on some
42:26
other kind of vulnerabilities I've seen.
42:28
Um, So yeah, I I think this continued adoption, like it's
42:34
still early, it's still young like a welcome anyone to come
42:37
and come and contribute. It's a it's a great opportunity,
42:41
everyone's welcome. But also like you know we have
42:44
an open Slack channel where we have an actual main channel.
42:48
We just discuss about whatever it may be.
42:52
We have regular biweekly meetings and then each
42:55
vulnerability also has its own like Slack channel itself.
42:58
So if you want to talk something really specific about prompting
43:01
junction and you want to like go to town, then maybe that's a
43:04
place like discuss it. So just continue development and
43:09
if we could branch out from something else apart from the
43:12
top 10 where that's kind of like something like the ASVA which is
43:15
like almost like a developer checklist, a bit more in depth
43:19
than that would be great. So just continue to support one
43:23
thing myself as well is that as a security person when you're
43:29
kind of getting insecurity or one thing that I did from Red
43:31
Teaming is that I worked on like CFS or captured the flags and I
43:36
love what I did was I hosted a lot of like sandboxed vulnerable
43:40
applications myself. Try to like hack them, learn
43:43
from them and then apply that in real world scenario.
43:47
There's enough that at the moment there's a gap which I'm
43:50
wanting to fill is we have like no vulnerable LM application as
43:56
part of a learning resource. Like a sandbox?
44:00
Yeah, right. Yeah, yeah.
44:02
Some people may have heard of like OSG Shop Shop for example,
44:08
which is something like that, which would be awesome.
44:12
Yeah, that makes complete sense. So that people can get that
44:15
experience of messing things up without actually having the best
44:19
things up. Yeah, you like get get your
44:21
hands dirty with prompt injection rather than trying to,
44:24
you know, stay in a safe, controlled manner.
44:26
Whether there is like a flag where you have to achieve
44:29
something so you can kind of learn from it.
44:31
Have you seen the game? Gandalf, My friend David, I
44:35
have. Yeah, yeah.
44:36
So yeah, I I know the guys at Lucara, they're awesome.
44:41
Gun Belfie. Super cool.
44:43
I also have fun with the wizard guy.
44:45
I'm very, I said you want one? I got into.
44:49
I found that. Well, I did find it.
44:51
I heard it, heard about it, kind of just in the community.
44:55
And it's great, something like Gandalf, but incorporating, I
45:00
guess I would say like the iOS top ten that we've listed, not
45:04
just prompt injection itself, that is kind of, that's what I
45:08
was kind of making. Yeah, it also feels like there's
45:11
bigger scope for the whole system as opposed to just like
45:15
interacting with the. LM It's like what what is the LM
45:20
being hosted on, or how is the inference coming and where's the
45:24
vulnerabilities there and what does it have access to?
45:27
And all of that would really help round out the experience so
45:32
that you can prompt Gandalf to delete your GitHub repo or
45:37
things like that. Yeah.
45:39
So that would be great. I like CC vulnerable web
45:43
applications all the time. So like security development or
45:46
cloud, things like this will go, for example, where you can like
45:50
spin up a container on a laptop or something and you can hack
45:54
away at it. So there's a yeah here is great,
45:58
Gandalf is awesome. I encourage anyone to give it a
46:01
go. It's good fun.
46:02
It's also a great learning opportunity as well.
46:05
Yeah, we just had the LM's in production virtual event and
46:10
David came during one of the breaks and he walked me through
46:14
the different levels and so he was there and obviously like I
46:18
wasn't doing that good and he was giving me hints along the
46:22
way. But it is fun, man.
46:24
It is. You can spend a lot of time
46:27
injections. One thing that is fascinating
46:30
when it comes to prompt injections and like a prompt
46:33
injection attack. Is that one of the other guests
46:38
on here? And so friend of the pod?
46:40
We could say Phillip Carter, who built out the tool the LM
46:46
feature at Honeycomb, his current job, which is like a
46:52
observability platform. He said that when they notice.
46:58
Prompt injections. It's very clear because all of a
47:02
sudden you're getting a shit tonne of prompts, like way more
47:06
than the average out. You know, it's it's just like
47:09
boom. 50 prompts per second type thing and they're trying every
47:14
they're throwing everything they can at it.
47:16
So it was it was kind of easy to combat.
47:19
But he was like yeah, it's very common too.
47:23
It's not something that you think.
47:25
Ohh yeah. Who's actually going to try it?
47:27
Inject prompt, inject this. People are out there and they're
47:30
trying. Especially if you have a free
47:32
tier that allows you access to the LM?
47:36
Yeah, no, absolutely. And I saw that you going to say
47:40
I went to left column this year with the with the going from the
47:43
camera which was great. And as part of the AI village
47:47
where we built the taxonomy and did the GRT event which was kind
47:51
of like I guess the TLDR was that it was like a public open
47:58
offering for anyone at Defcon to come and kind of like Red Team,
48:01
a bunch of models. Nice.
48:03
Yeah. Which was great because it also
48:05
was an experience for the if there's a I companies actually
48:08
take away that information and kind of use it so they they have
48:12
that was great. But yeah I guess as well like
48:15
you say that kind of goes back to the whole reason why we did
48:18
the top ten LM applications. It's because yeah you wanna be
48:23
careful if you're allowing free tip people to send like shifty
48:27
prompts a second that's a real concern.
48:30
That's something like you know you should be longing monitoring
48:33
that so you're able to it was identified that but you can use
48:36
that to your advantage. There's tonnes of people trying
48:39
to like you know, trying to do prompt injection on models right
48:42
now to get it to do XYZ, to get it to give me like you know,
48:45
your grandma's e-mail address, like all these things.
48:48
There's so many people trying to do this right now, but as a as a
48:51
company you should be using this to secure your platform further.
48:56
You learn from like from attacks to be able to then defend
49:02
before. That's the whole point of red
49:03
teaming right? Like you're, you're effectively
49:05
attacking something to take away defences and implement
49:09
mitigations across what you've been able to successfully
49:11
exploit. Yeah, yeah, that's.
49:15
I think I remember Philip saying, like, yeah, that was
49:17
kind of an easy fix because we just.
49:20
Made sure that if all of a sudden they're doing like 10
49:24
prompts per millisecond, we cut that off real quick.
49:27
Yeah, yeah, exactly, yeah, exactly, exactly.
49:31
And traditional REST API, web application security rate
49:35
limiting, things like that. So many other security lights
49:38
you can implement even at the user level, right?
49:40
If you've got depending how like your free tier and premium
49:43
subscriptions are and all that kind of stuff.
49:46
How much do you feel like like how much overlap is there where
49:50
you're like, hey, this is just traditional security stuff in a
49:54
new flavour that now we're adding it to an LLM?
49:57
And how much is there? Like, oh, this is really new and
50:00
I never would have thought about it.
50:03
I'd say, I'd say like, I think that honestly, I think at the
50:06
moment like I think 1/3 of it is the new stuff, but it does
50:10
honestly go back to like traditional like security and
50:13
layers, see really does and like assessing that risk, applying
50:18
that security in layers. Because yes, effectively you've
50:22
got a very smart intelligent model label to do all these
50:25
things. But if you surround and you
50:27
secure the environment and the ecosystem around it then bridge
50:32
you so much risk and you mitigate so much threats just by
50:35
doing that itself. Like that should be your
50:37
foundation when building any kind of application whether it's
50:40
like a web, a a mobile app. Ohh, I'd not be.
50:44
I would say it's pretty about a third, you know like trade like
50:47
example training data that's completely like an LM shindig,
50:51
that's like you know that's traditional website has no
50:54
relation. But again, like traditional
50:56
security, I seen, you know, I've seen for example, where let's
51:03
say you open crawler data set, right?
51:05
It's pulling all this data set. One thing I don't if you like
51:10
like aware of it, some people are like actual purchasing
51:13
domains which were part of open crawler.
51:16
So all these bogus domains on the Internet, right?
51:18
And they're adding a bunch of bullshit.
51:20
Yeah, just like. And then yeah, there's some.
51:24
It's like data poisoning or something.
51:26
I think it's called, yeah, yeah, data poisoning.
51:27
Yeah it is. I actually, I can get the name
51:30
of it in a second. But yeah, example like simple
51:34
data poisoning, I mean that's more relevance to the fat, like
51:37
if you're pulling, you want to make sure what you're actually
51:40
pulling that data. But again, there's specific
51:43
security controls you can do around that, but yeah, for sure.
51:47
Yeah, that that is so funny that you mentioned that that people
51:50
are just going out there. They're saying, you know, it
51:53
would be great if we don't even have to deal with.
51:57
The whole security measures that someone puts into place and we
52:01
get to the model before anybody because we're in the training
52:05
data and the training data is exactly.
52:09
So that was actually what he was really bugging me.
52:12
So I had to check out. I did a talk on it a while ago.
52:14
There's a talk at Defcon from my friend Will.
52:17
He's he's fantastic. He's called Split View Data
52:21
poisoning. And then I actually did a talk
52:24
at the local OS Defcon kind of meet up.
52:27
I actually built a new image on my GitHub repo which was kind of
52:32
proud of it was I spent a lot of time during that.
52:35
But yeah, like you know it's actually uses the main
52:39
monitoring for known common and public training data sources.
52:42
Purchase a bunch of expired bogus domains I think will spent
52:45
like 13 bucks and like literally have control over thousands of
52:49
it. Like you and Jack like let's say
52:51
you put like cap means in there. You've got a model that's like
52:54
training on like cat memes which is like you can use.
52:57
You can see the use cases where it could be malicious or you
53:01
know, just like bogus. Like you know if it's a
53:04
competition, like that's also one thing that I am personally
53:09
worried about right? Like if if I want to beat you, I
53:14
know what you're training on. Maybe I dislike for a bunch of
53:18
light cat memes in there and like, you know, throw you off.
53:23
That's very nasty, but. Yeah, dude, because you're never
53:28
gonna find that. I mean it's so much effort to go
53:31
through the all that data that you've got, especially on these,
53:35
you know, like the 180 B Falcon model that's trained on
53:40
everything that anyone's ever said in this universe, pace bin,
53:45
all that kind of stuff, right? Yeah, exactly.
53:48
And damn, so. If you've got.
53:52
Ohh shit. Tonne of data poisoning
53:55
happening and you know how to prompt it to get that.
53:59
Then all of a sudden, ohh, there's a vulnerability in my
54:01
model. Yeah, cash money.
54:04
Yeah, and you don't even. The funny thing is that.
54:08
It's like a. It's a latent vulnerability
54:12
almost because it's not there until somebody that knows about
54:15
it goes, ohh, hey, I think you trained all my data and then
54:19
boom, they can capture that in a way, 100 said, yeah, that's what
54:24
was kind of alluding to it, like if you could keep a lineage of
54:27
that, right? And the only way I guess to do
54:29
that is to perform internal housing, internal red teaming
54:33
experiences regularly on like a pipeline basis for you to try
54:38
and find that earlier. So you can mitigate that you can
54:41
go back and be like, you know, we weren't seeing this, we
54:44
weren't seeing this from the release of this model, but we're
54:46
seeing it from the release of this model.
54:48
What did we change or what did we add between that could help
54:52
us kind of then go back and drill down into that?
54:56
Yeah, where could this have come from since then?
54:59
All of a sudden you find wills websites that they're happy and
55:02
yeah and then you're like, ohh that makes.
55:05
Sense. Yeah, perfect do.
55:08
But yeah, the kind of shit that you could just put and the next
55:13
Lama model all of a sudden is open sourced and nobody knows
55:16
about it because it's in latent space that nobody's touched.
55:19
And then somebody that does, the one or two people that do know
55:22
about it, they can take advantage of that and there's a
55:25
huge vulnerability there make make this even naughtier and be
55:28
like, OK, so you know the example I gave you earlier about
55:31
what it like deleted the GitHub repo.
55:33
Yeah, like let's say let's like hit the markdown elements in the
55:37
web page. How are you going to even detect
55:39
that? Because like then you that makes
55:41
it even more. It makes it even more difficult
55:43
for you to be like identify that if you've got hidden markdown
55:47
which has got instructions or instructions BS in there or BS
55:51
in there in general, but like rag is pulling that, then else
55:57
it makes it even harder. Like it might not be a captain
55:59
in front of you. It could be something like, you
56:03
know, like you see things like drive by downloads, all right,
56:05
you don't see that from like the face value.
56:07
It's what's underneath. As well, yeah, one layer deeper
56:11
because you're looking at it and you're like, I see a cat meme
56:13
and the metadata is telling the model something completely
56:16
different. Yeah, exactly.
56:18
I, one of my other good friends, Rob, wrote a book called Not
56:23
With a Book with a sticker. I don't know if I want to read
56:26
it, but it's a great book, the kind of the subjects of the
56:30
book. I guess it kind of goes back to
56:32
like toddlers where they were doing AI driving and you slap
56:36
like a bit of tape in front of like a three and make it one of
56:39
the. Changes that, changes speed
56:42
based on that. So yeah, you definitely got like
56:46
inherit and I guess indirects, indirects, kind of prompt
56:49
injection techniques because there's different ways of being
56:52
able to hide stuff or whatever, right?
56:55
Or yeah. Yeah, one thing that we used to
56:59
talk about when it came to model monitoring was an example of a
57:05
self driving car and it recognising stop signs, but then
57:09
all of a sudden it snows. And yeah.
57:12
This stop sign has snow all over it and it direct prompts
57:15
injection. Yeah, exactly.
57:17
It's like, wait, was that the model that changed or was it the
57:21
real world that changed and so you had that data drift or just
57:24
that unknown data set that it hadn't been trained on, but.
57:29
Dude, yeah, fascinating stuff. I mean when it comes to this
57:34
idea of. Being able to.
57:38
Put toxic data out there for LM's to go and get trained on.
57:43
That is a really interesting one that I hadn't been thinking
57:46
about. I knew about data toxicity.
57:48
I just didn't realise. How?
57:52
Predominant that could potentially be now that we're in
57:55
this world of. Creating language models with
57:59
all the data on the Internet. Yeah, and like, yeah, spinning
58:03
up a bogus lane is you can automate that.
58:06
You can make it super cheap. But yeah, it's not, it's not
58:10
like super in depth to do that, right?
58:13
If someone, if someone has the drive to do it, you know.
58:16
That is so Wild Well. Adds I appreciate you coming on
58:19
here, man. This has been enlightening for
58:22
me. Maybe I will ask you to pop in
58:24
every once in a while and give us just a security update.
58:28
It would be my privilege. Thank you so much for having me
58:31
as an awesome opportunity. I I thank you guys.
58:34
I was actually reading some of your blogs and the community is
58:37
awesome and join the slight group so always open to chat.
58:43
Thank you so much Robin. It's been awesome and I
58:45
definitely will be calling you as well.
58:47
I'll be stalking. Here, I love it.
58:50
That's the power of the community, man.
58:52
And so anybody out there that is interested or tangentially
58:55
interested in getting into the security world?
58:59
There is a great project out there and it is at
59:02
lmtopten10.com. You can check it out and we'll
59:06
end it there. Thanks so much.
59:08
Thanks everyone. Have a great day.
59:14
Hey, this is Mike Delbalso, Cofounder and CEO of Tecton.
59:19
ML OPS Community is the best way to stay in the loop on the
59:22
latest ML OPS news and best practises.
59:24
It's also a great way to connect with experts and get support
59:27
from an amazingly helpful community.
59:29
Subscribe and stay in the loop.