Latest / Agentic Conversations (formally mlops.community) / Fraud Detection in the AI Era // Rafael Sandroni // #301

Transcript

Fraud Detection in the AI Era // Rafael Sandroni // #301 · Agentic Conversations (formally mlops.community)

0:00
I'm Rafael Sondroni. I'm the founder of the Guardian
0:04
AI is AI security company. Usually I mix Brazilian cough
0:09
with Italian mocha, so that's the best combination.
0:13
But I also I like the espresso. We'll be back for another
0:17
Envelopes community podcast. I'm your host, Dimitrios, and
0:20
today we're talking with Rafa all about the security vectors
0:24
that you got to watch out for when it comes to building apps
0:27
or just using LLMS. I particularly enjoyed the moment
0:33
that we touched on traditional fraud detection that he's done
0:37
at places like New Bank, New and New.
0:41
Shout out to their engineering blog.
0:43
And then we contrast that with these days, the kind of fraud
0:48
that you can do when it comes to LLMS and if you have AI systems
0:54
in place. Let's get into this conversation
0:56
with my man Rafa. I know you were working on Siri,
1:04
and I know that you were doing stuff that was to help Siri
1:08
understand other languages, but Siri's been around for a while.
1:13
It's also been pretty wild how the evolution of Siri has been
1:17
happening. And you posted some stuff that
1:20
caught my eye on LinkedIn about like how to make assistance, AI
1:26
assistance better. Things that you need to like
1:28
keep in mind when you're creating AI assistance, what are
1:34
some of these things that you've recognized as best practices
1:37
that you can share from your time doing that?
1:40
And I know Apple has some pretty strong NBA, so you don't need to
1:44
get into specifics, but just what are some key learnings?
1:49
OK, So what I understand is that with the LMS, the the APIs and
1:55
So what is it's much easier to everyone to create AI
2:00
assistance. And based on my experience, I
2:04
don't know, Siri is the one of the like top five AI users in
2:08
the world. It's a huge problem to make sure
2:12
that everything is going like a testing, make sure that the
2:16
response have a is like a kind of quality, but also the
2:21
orchestration like a depending of the what AI assistant is
2:25
this? So you need to orchestrate,
2:28
integrate with APIs. You need to make sure that
2:32
everything is integrated test disintegrated because I don't
2:37
know, it makes you need to make sure that you have contracts
2:40
between the integration, the tasks as well.
2:42
So you can see recently using a mix of AI and human your task.
2:48
Yeah, I also like the task is one of the most important part.
2:51
Like if you are creating some value for people, you need to
2:56
make sure that the AI the this AI assistant is like working
3:00
well, you don't have kind of regression.
3:04
So I work at at city on some components to make sure large
3:08
rails to make sure that you don't have a kind of answers
3:12
referencing like a kind of profanity and competitors.
3:16
And So what, and if you think about, I don't know, a bank
3:20
using this is a regular regulatory markets that have a
3:25
kind of regulatory as well. You need to make sure that the
3:29
answers of this AI is you have a kind of control, right?
3:34
Oh yeah. So we're capting tests, the
3:36
integration make sure have a kind of contract and also
3:40
controlling like I have like gradual rails or controlling the
3:44
output is like then a good start point I feel.
3:49
Like Siri is the ideal abstraction layer that I want an
3:54
agent to play at or an assistant because it can do anything on
3:58
the phone and it can go into the apps.
4:01
I wish it could go deeper into the apps, right?
4:03
But I guess that's just each app has to make things discoverable
4:08
for Siri. And so there's this question, do
4:12
I want to have my own AI assistant inside of my app, or
4:16
do I want to have Siri be the AI assistant inside of my app and
4:21
that type of thing potentially, or what different apps have to
4:27
grapple with all the time. You were doing the guardrails
4:30
before guardrails were cool and before there was these huge
4:33
blunders. So it's funny to think about if
4:36
you didn't do your job well, Apple could have been the Air
4:41
Canada. It could have had that brand
4:44
awareness that no brand really wants.
4:47
But then I know you went and after you working at Apple, you
4:51
went and worked for one of my favorite engineering blogs of
4:54
all time, which was Nubank, right?
4:56
And Nubank is awesome because of how transparent they are in
5:02
their engineering blog. I think they're, I say this a
5:07
lot, but I don't say it enough. The Nubank Engineering blog.
5:11
I've learned so much from them and they are so transparent that
5:17
I'm thankful for that. What were you doing at Nubank?
5:23
Yeah. So it's a little bit complex
5:25
because I spent almost three years over there and I had the
5:29
opportunity to work together with the part of this thing.
5:34
But I, I guess the important part the, the thing that I like
5:38
at New Bank is like a sharing like like dollars.
5:41
So you have a lot of, let's say, workforce meetups internally to
5:46
make sure that everyone is in the same bar.
5:50
And my, my first experience at New Bank was interesting because
5:53
I was part of, let's say a new feature internally, you as you
5:58
are building a personal finance assistant for New Bank users.
6:03
And it was part of my startup my that I founded years ago.
6:09
And you had the opportunity to create the basis of this AI
6:14
assistant. And the funny that the funny
6:16
part, I was new bank decided to, to buy one of my computer,
6:22
computer stores like a personal files management app.
6:26
And it was part of the due diligence and they're sending
6:29
internal tools and so well, so it was a interesting part
6:33
building this AI assistance, I guess now is part of the new Bay
6:37
Gap, helping people to have savings, controlling the
6:40
savings, have a better planning, financial planning.
6:44
And but after that, it was part of like a fraud team.
6:48
Fraud team like was basically the same.
6:51
And the challenges was like, I guess.
6:54
Maybe you? You saw this blog post about the
6:58
real time machine learning for machine learning models for
7:00
fraud, I guess everything tech and bank, you have this such
7:04
challenge like understanding blocking fraudsters in
7:09
transactions online or like onboarding for new customers.
7:13
And yeah, I had the opportunity to like create the tools like
7:17
doing retraining models automatically for to understand
7:21
and locate the models with like a new patterns and also
7:26
improving this the performance side this model.
7:28
So I has I have the design interesting bug post about this.
7:34
Yeah. And that was all traditional ML,
7:36
right? So you went from, yeah, with
7:38
Siri, you were working on traditional ML or you were doing
7:42
the guardrails or you were doing software engineering.
7:46
What do you feel like the evolution was there between
7:49
those two? Yeah, it's a kind of funny to
7:53
say traditional ML because I don't when you say MLP, I
7:58
believe it's traditional ML, right?
8:01
But yeah, but I believe everything is connected, right.
8:05
Probably in a level of maturity of AI, you need to implement
8:10
traditional ML, you need to go out of APIs and implements a
8:16
kind of some of those internally to have more control.
8:20
So not only the lens to use open source, but also I don't know if
8:24
you need to watch Rails, probably the best way to go to
8:28
traditional NLP like a cost fire using small language models and
8:33
implement this. But the challenges is like
8:35
around not on the AI, but the soft engineering like how to
8:38
deploy. Yeah.
8:39
So I I believe that everything is connected.
8:43
It's easier right now to use. We start using AI, right.
8:47
But I guess the challenge is the same of the four or five years
8:50
ago. So if you need traditional ML.
8:52
So the chart is the same all right now.
8:56
Talk to me about what you've been learning about recently.
9:01
Yeah, so this is the the interesting part of this talk,
9:05
but I left Apple months ago to start something and I, I started
9:10
learning about the ICQ to how to make sure that every AI
9:14
application AI assistant is like it have it's protected guys like
9:19
fraudsters, hackers and so on. So I started with the wasp top
9:26
thing for LMS. So later I joined as a neighbor to understand more
9:30
the challenges and it's a huge problem to be honest.
9:34
Everyone that I'm talking about like a banks, larger fintechs,
9:38
healthcare companies, legal, we start-ups local like everyone is
9:43
like implementing, but it's a kind of challenging stopped
9:47
because you need to make sure you need to some people like so
9:51
basically. They want to get hacked and then
9:53
they want to put a password manager in place.
9:56
Yeah. So basically so depends of the
10:00
future of the the company. So I guess the banks and the
10:03
financial markets more like aware of these risks.
10:06
So, so they they face the same challenges on the softer side.
10:10
So probably the AI, they are taking care of this.
10:14
But that's an interesting part. I would say that you have two
10:19
components of to make sure that to implement like secured
10:25
standards on AI. So the first is you need to
10:29
implement the AI following a zero trust approach.
10:33
So make sure that you have the permissions, the credentials and
10:38
the integration is safe. But after that you need to do
10:43
like a kind of task security task.
10:45
So write a teaming the same way that you have like a Bing task
10:48
in the software side. So to discover the
10:52
vulnerabilities, understanding these vulnerabilities like you
10:56
have the opportunity to fix on your implementation or implement
11:00
use word rails. So the the second part is a way
11:04
to following the zero trust approach.
11:07
Well, you don't believe that you don't believe that the LM is 100
11:12
safe and you can blame it to just guard rails, not one for
11:15
the security, but also let's say to prevent like a kind of
11:20
hallucinations, a kind of off top.
11:24
And and so and the secured part of the guard rails is very
11:28
challenging because these guard rails you should learn you
11:33
should this model should know what partners what's what's like
11:39
a prompt ejection methods that have more effectiveness.
11:44
So to block these attempts. Yeah, so overview of the ICQ, I
11:49
would say like a starting on the AI application.
11:52
So have to this confidence. So as you're looking at these
11:56
different vectors that LMS can be basically corrupted on, and
12:02
you're thinking about guard rails, you're also thinking
12:04
about zero trust. First of all, I would love to
12:07
understand what you mean by zero trust for the LLMS. What does
12:11
that actually look like in practice?
12:15
The same approach like when you implement the software.
12:18
So make sure that you don't have the, you have the access just
12:22
for that function for that X for that reserves.
12:27
So yeah, it's the same thing. So if you have AI assistant that
12:31
have access to APIs and other tools like making sure that the
12:36
scope is very defined at the access, the permission is like
12:40
very defined. Basically it's like approach
12:42
that's the developers the the dev team can follow when, when
12:47
and just in the implementation. Yep, they said it's.
12:52
Like it's, it's more like a mindset.
12:54
And trying to limit the scope, which is a hard thing to do.
12:59
Or it's a very fine line because with the LMS, they generally
13:04
perform better the more context you give them, and so you want
13:07
to give them as much context as possible as long as it is the
13:13
right permissions. And so trying to figure that out
13:17
and trying to architect your system in a way that is going to
13:21
know who has permission to what can get a little tricky.
13:25
Have you seen good ways of doing that?
13:28
What do you mean about the permission here?
13:30
So I guess it would be like who has access to what data?
13:33
Who has access to what documents, that type of
13:36
permission? And then because you don't want
13:40
an LLM to be able to it maybe it has access to every database or
13:47
every channel in Slack. But if I'm asking a question, I
13:51
don't have access to every channel in Slack, right?
13:55
And then all of a sudden the LLM does.
13:57
And so I'm able to find out things that I am not Privy to.
14:03
Yeah, So this is a part of making sure like the fighting
14:06
what's the most important data to share if they I so you don't
14:12
have to share out of the your database with the eyes and and
14:16
maybe you don't need to share off out of your table data table
14:20
for with AI. So it's complex, but I guess is
14:26
is following this mindset and being more like a restrictive
14:31
about the data. And then what?
14:35
So we were talking about the different attack vectors that
14:38
you've been seeing, and one of them was jailbreaking.
14:42
One of them was red teaming. I think people have seen a lot
14:45
of examples of those on the Internet.
14:47
What have you been thinking about when it comes to the
14:50
security vulnerabilities and then making sure that guardrails
14:54
work? Like how do you architect the
14:56
guardrails so that they are more effective?
14:59
I guess the the most challenging vulnerability of AI is the the
15:04
prompt injection. So we can pass some contacts, do
15:08
a kind of social engineering with AI and then I don't know,
15:13
some have access to some data. The system prompt will be like
15:17
the first step to understand more dynamic of AI or AI agent.
15:22
I believe like implementing the right gradual rails to prevent
15:30
that to have, let's say malicious inputs that can affect
15:34
your LM or your AI agents like the chain of LMS you'll be,
15:40
you'll be like the best practices.
15:43
So that's what I'm saying seeing right now in the industry.
15:48
So it's not the finally, it's a kind of mitigation, it's like a
15:51
one it's not 100%, but you can eliminate a good part of
15:55
fraudsters or or malicious users that can try to expose your EI.
16:01
So and usually this was your real.
16:04
So you you implement this as like a Euro time is you are
16:08
implementing the orchestration and it's working real time
16:11
blocking some some patterns. And also if you have application
16:17
on that, that have access to the Internet or documents or data
16:22
that you are not the owner, you should make sure that you can
16:27
you should like a filter and make sure that you don't have an
16:30
indirect injection. So there's an example very
16:35
interesting as an example from the conference coloured Black
16:39
Hat of when a guy trying to the Microsoft Copilot that was
16:45
configured through summarize emails.
16:49
And basically it it open vulnerability.
16:53
It said, I external person can code send an e-mail to this
16:59
corporation and then use our gap.
17:01
So every e-mail could be like a summarize it.
17:04
And in the last step, let's say replied with the AI agent
17:09
depending of the topic and this external user standard e-mail
17:13
with a prompt injection hidden bit by the human eyes, but like
17:18
a using encoding and colours. But the I code understand and
17:22
they it is starting sending phishing emails for every use
17:28
e-mail in the user base. So it's a example.
17:31
So if I have access to external Rd., you should make sure that
17:35
you don't trust on the the prompt instructions.
17:40
Don't put your word rails on the system prompt user use like a
17:44
independent models, machine learning models to filter and
17:50
detect this kind of let's say Potters.
17:53
So it's it's very challenging again, because it's not 100%.
17:58
You can like it's just a mitigation.
17:59
So you can be like 100% safe, but so.
18:02
Let me see if I understood that Black Hat example, because that
18:06
is a new one that I haven't heard before, where if someone
18:11
is using Copilot and it's in their e-mail, then you can get a
18:17
phishing e-mail that has white on white text.
18:22
So white text on a white background that the human eye is
18:26
not going to see, but the copilot will see and it will
18:30
think, oh cool, I can summarize this.
18:33
And when it summarizes it, then that goes into the LLM and it
18:37
does an indirect prompt injection, which says basically
18:41
send an e-mail or summarize this e-mail and send it back to XYZ.
18:47
And so if you don't have two things in place, one, just the
18:51
human touch that needs to OK the send or return reply e-mail, or
18:59
two, that you have some kind of a external independent critic
19:05
that is judging what the copilot LLM call or agent is doing.
19:13
And then it's saying, hey, look, here's all the information.
19:17
Here's what I did. Is this correct?
19:20
Here's the prompt, here's what I've done, here's what
19:24
information I got. And then you have that judge
19:28
that says yes or no, this is correct or this isn't.
19:31
You might want to go back to Step 1.
19:33
So you could get into a whole lot of trouble if that is not
19:37
what you're doing and how you're architecting it.
19:39
So I've I've seen that in a way that type of design pattern
19:43
where you have almost like this catch all critic at the end and
19:47
you can throw everything into it and you can say looking at all
19:51
of these steps, this is what I've done.
19:53
What do you think? Is this the right way of doing
19:56
it? Yeah.
19:58
I guess the point is that looking for the you have a judge
20:01
and the name is a judge and the the same vulnerability that you
20:06
have in the LM we have on these LMS that judges I guess.
20:10
The solution that I'm saying these are like a using
20:12
traditional machine learning classifiers and that you don't
20:17
have the same vulnerability of the LLM is popped by LLM and
20:22
then we can mitigate don't have don't expose another channel,
20:26
another another space, OK, so. You're saying don't even use
20:29
another LLM because then you might just get this cascade of
20:34
prompt injections. Yeah, don't use another LM to
20:38
future malicious inputs because this another LM can suffer and
20:43
can follow these malicious inputs as well.
20:46
And what are some other ways that you've been seeing security
20:50
vulnerabilities or some real life examples of things that are
20:53
happening in the wild I working? With some, some fintechs
20:58
understand the challenges and like validating these business
21:01
status. So validating some ideas around
21:03
the ICQ it there are some companies the fintech's working
21:07
building AI assistant on the WhatsApp and these AI assistants
21:11
can like can do transfers, bank transfers, can pay bills and so
21:16
on. So we have basically like AI
21:18
assistant to have access to the your bank account or in other
21:23
your scenarios, you have your bank account managed by AI on
21:29
the WhatsApp. Yeah, the trick part is that I
21:33
saw my interesting example recently with some hackers
21:36
trying to override the file they are of the transfer for example,
21:43
or override the account balance that you have in your account.
21:47
So, so this is in your real life, you actually don't have,
21:54
you don't have a lot of problems because you, this scenario that
21:59
I'm explaining, like design startups have, let's say
22:03
infrastructure as a banking in the core.
22:06
So we have another kind of validation.
22:09
But that's why it's important to follow like that zero trust
22:14
mindset because we can like defend and protect this
22:18
malicious behaviors on the softer part, not the AI as well,
22:22
but it's important because out as well.
22:24
So you can like you have, you can have the AI trying to answer
22:30
some something that is following some bias and talking about
22:35
competitors and generation kind of repetition or risk for the
22:39
company. So.
22:43
I'm trying to think how that looks and what the benefit would
22:47
be of having me be able to do my banking through WhatsApp.
22:52
And so I guess the benefit for a end user like myself, I'm using
22:57
WhatsApp and I say, oh, send 50 bucks to Rafa and then it knows
23:03
which Rafa I'm going to send it to and it will send it.
23:07
Or I tag Rafa as a person and then it has your phone number
23:13
and my contacts book and it will then send it to you.
23:16
But it feels like there's so many things that can go wrong in
23:20
there because it could just hallucinate a different Rafa or
23:25
a different phone number or whatever.
23:27
There's so many vulnerabilities that I wonder if the small lift
23:36
that we get from being able to interact with our bank account
23:40
on WhatsApp is worth the gigantic risk that it brings.
23:47
Yeah, I guess is it that's the challenging of the developers,
23:52
the ML engineers nowadays? If so, you should have, you
23:56
should follow the, the tasks have the kind of high quality
24:02
essentially to make sure that it doesn't happen.
24:05
And I believe that you're learning how to create this kind
24:08
of application, right? Like a disk, a mission creature
24:11
application. Yeah.
24:13
That's a very, very challenging, but I guess very interesting
24:16
problem to to work on. Yeah.
24:19
It does feel like we're still exploring where and how we are
24:23
going to interact best with AI assistants, AI agents, what the
24:29
ways that the public prefers to use them.
24:33
And like you said, maybe for a banking app, it's not the best
24:39
option. But potentially for a food app,
24:41
it is the best because we can just text the food app and say
24:45
I'm hungry, I want something in the next 30 minutes.
24:49
And it gives me 5 different suggestions.
24:52
And then I can just say, yeah, I go with suggestion too.
24:55
But if it's for a banking app and it's send $20 and next thing
25:00
you know I sent $2000 because there was some kind of a mess up
25:05
or a hallucination, then it's a lot more risk than if I get
25:09
pizza or pita so. What I'm seeing you have this I
25:14
don't know you have AI bank assistant like how you can talk
25:18
with for the is that I have so depends.
25:21
I, I believe that there is a profile specific customer
25:25
profile for this, for this application.
25:28
And so it's a risks, it's riskier, but I don't know, I
25:31
believe that the market's going from this side.
25:34
I don't know looking for aeristage fintechs building on
25:37
this space, but later, you know, actually years, I don't know
25:41
like the banks could have a new channel as the same of the
25:44
mobile. So we have this lag assistance,
25:48
banking assistance, and then they try to follow the fintechs
25:53
and the and build the mobile application.
25:56
Make sure that like the UX nicer, everything is is motor
26:01
and maybe the next wave you'll be like the having IAI helping
26:06
people to like make it transfer, understanding the expanse and so
26:11
on. It's a mis mis critical
26:13
application, more critical than asking for ordering your food.
26:18
But I I believe that there is a value of the this application
26:22
has have value from the some customer profiles.
26:27
Yeah. And when you have worked on
26:30
traditional fraud detection systems and then you think about
26:34
these new ways of fintechs having fraud vectors, how do you
26:42
weigh these different areas and, and vulnerabilities?
26:45
And like, how do you think about traditional fraud?
26:49
And almost, I don't know what the new one would be called, but
26:53
I'm going to coin it, I'm going to say new age fraud.
26:56
What do you look at? Is it different?
26:57
Is it the same in your mind? How do you view those two?
27:01
I believe that with AI, both of parts of a fraud, the fraudster
27:05
and the let's say the bank, it's like it they have superpowers.
27:10
So it's the fraudster have like the tools to create a lot live
27:15
in his face and the fake documents and so on for them.
27:19
It's becoming easier to to bypass a fraud system and and
27:24
also internally inside the fraud, a bank, a fraud thing.
27:29
I guess you have the tools, but it's a challenging moment to
27:32
looking for this new kind of AI. But also we have the fraud
27:37
around the AI like an interracial.
27:39
So it's a kind of fraud as well. So basically you have a more
27:45
vulnerabilities around. We have the fraud with
27:47
superpowers, with AI superpowers trying to figure out like what
27:53
is the vulnerability that they can they could like follow and
27:56
deep dive. And I don't know it, it's a
27:59
challenging moment if you look for the cybersecurity as well.
28:02
So not only define it on the financial space, but the the
28:07
cybersecurity in general. So it's a challenging moment
28:11
because they think you are have seen increase of incidents,
28:17
security incidents. That's because of the
28:19
fraudsters. The hiker have more powers right
28:22
now, they can automate, speed up the process that we spent one
28:25
month with some minutes with AI. Yeah.
28:28
And you're able to just test where the vulnerabilities are so
28:31
much quicker. I believe that the fraud, the
28:35
same dynamics of a fraud in the that you know right now and all
28:41
days you see this dynamic in the AI as well.
28:44
But I guess the challenge would be this almost the same.
28:47
So we have some fraudster trying to injection, doing prompt
28:51
injection and other side you have the large rails or other
28:55
reserves to protect them, learning the new patterns and so
28:59
on. I don't know, maybe human the
29:01
loop as you have in the fraud operation as well.
29:05
Maybe you should have this human in the loop in the AI operation
29:10
as well. Yeah, you make a great point
29:13
about how many different levels there are that fraudsters can
29:18
play at because it can just be falsifying documents or now
29:23
these days you can clone voices with just like 5 seconds of
29:26
voice. And so you can do social fraud
29:29
and get information by doing those like social hacks.
29:34
And then on the other hand, you have the LLMS themselves and how
29:39
they can be vulnerable. And you have the new systems
29:43
that we're putting into place. Like we talked about how, OK,
29:46
now we're talking to our bank through WhatsApp there.
29:49
There's a whole bunch of vulnerabilities that come along
29:51
with that. Or you have this super powered
29:55
fraudster who can use AI to their advantage and scan for
30:00
security vulnerabilities or fraud vulnerabilities.
30:05
Are there specific design patterns that you've been seeing
30:10
that are now becoming very common and something that is
30:17
like table stakes for if you're trying to deal with AI fraud on
30:25
any of these levels? So I believe that's knowing by
30:30
first hand if the vulnerability that a fraudster can attack.
30:36
It's important. So doing the Reddit teaming,
30:40
testing your system, doing a ping test on software and red
30:44
team in general, I guess it's it's a much have should do it
30:50
before. So if you have a Mitchell
30:52
critical critical obligation and so on, that's the best way.
30:56
So you can have the first hand knowing the vulnerabilities and
31:01
then trying to protect this, implementing the the protection.
31:04
I believe it's it's approach that is becoming Mccommo and
31:08
also as well like the gauge rails implementing the the
31:12
correct gauge rails for the obligations like the painting of
31:16
the use case depending of like the the probation on the risks
31:20
that's the you like to accept or not.
31:23
All right, So what other kind of things have you been seeing out
31:27
there in the wild? AI agents.
31:29
Actually, there is a recent paper report from OS for about
31:34
the vulnerabilities in the AI agents.
31:38
And it's interesting because when you have a lot of as many
31:43
LLMS stop with each other, it's become like more complex to make
31:48
sure to control LLM is not predictable and we have a lot of
31:52
LLMS working together. It's even unpredictable.
31:56
And and the problem is that way in the AI agents obligation, you
32:00
can let's say with accommodation of tokens like say attack can
32:05
broke the first word, but the context, the information didn't
32:09
kind of noise can follow through the next steps.
32:12
And also depending of the steps, if I have some documents data to
32:17
access externally, one simple, well, let's say one prompt
32:20
injection that was received in the first step can affect, I
32:24
don't know the the last steps, you know, but it's more like a
32:29
general is not example, a specific example.
32:32
But it's interesting because you can have a kind of noisy going
32:37
forward for each step. More LMS, more yeah, yes, more
32:41
LMS you have in your system make more protection.
32:45
We should have to make sure that you don't have this information,
32:48
just noise information going forward.
32:50
Yeah, it's almost like one prompt injection can get
32:53
propagated throughout the whole chain of agent calls and you
32:59
have to make sure to architect your agents in a way that like
33:03
you we were talking about before.
33:04
Potentially it's not even a another LLM call critic at the
33:09
end. It is something that is a bit
33:13
stronger and more rules based type of guardrails that you
33:17
have. I'm also looking at the
33:19
owasp.org page right now on their top 10 vulnerabilities,
33:26
and I find it fascinating that the first one is broken access
33:30
control, because I think a lot of folks are trying to figure
33:37
out how to get agents to act on their behalf.
33:40
And this just feels like a WASP is calling it out early and
33:46
saying, hey, look, you can try and have the agent act on your
33:50
behalf, but there's a lot of potential that you're going to
33:53
have some access control issues here.
33:58
Yeah, this is the challenge, because there's a balance
34:01
between autonomy and the security, right?
34:04
Yeah. This is the challenge part.
34:06
I don't know, you have been seeing a lot of value from agent
34:11
that do your steps by itself. But I guess this is a part of
34:15
the zero trust approach that you should follow.
34:20
So let's make sure that's like you have the right permissions,
34:23
is restricted for that task, for that scope and so on.
34:30
Yeah, these are things you as a machine learning engineer going
34:37
into this world of security, it feels like you had some
34:41
experience when it comes to the fraud detection side of things,
34:45
but now you're having to learn a lot about, I'm sure, the
34:48
security side of things. And are there pieces that you
34:53
wish you knew when you started going into this world?
34:56
What are some things that potentially you can just share
34:59
as a nugget of insider knowledge that took you a while to figure
35:05
out, but now that you know you're very happy that you do?
35:09
I would even classify that a little bit further and say what
35:12
are some things you've learned in the last three months that
35:17
you did not realize as a machine learning engineer, but now you
35:21
see as like you're getting into the security world?
35:26
Yeah, the first thing is that fraudsters and hackers, they are
35:30
like they have a lot of crap TV. So even as a machine learning
35:36
engineer, you're building an AILM and So what you don't know
35:41
what's the vulnerabilities. So you know, there there is like
35:43
a oh, someone can use base 64 and encoding a kind of method AB
35:51
or C, but when you put in it in the road, like a lot of new
35:57
steps you didn't figure out like that people can do this.
36:01
And it's very wild. And one of the most the learns
36:06
that I did recently is joining cybersecurity engineers with
36:10
understanding talk with these guys and learning their
36:15
perspective about AI security. So it because it's different as
36:20
AI machine engineer or AI research is different is that is
36:24
another they have another opinion is so let's say when you
36:29
have when you do run teaming, they're not taking care of just
36:33
the LM, but the whole assistant, for example.
36:38
So it's important and also about the protection the graduated
36:43
them, they don't it's not looking just for the LM, but the
36:47
whole system as well. So they access the provision.
36:50
So and yeah, I guess the the insight that I have recently is
36:55
they have another perspective. It's more broader about the ICS
36:59
thing in general. I'm researching about what the
37:04
guard rails security guard rails data, it's important.
37:09
So as like we are building large rails, you should have data to
37:15
understand what kind of and methods that's like hackers are
37:19
using or should or could use to to bypass IEI or to use virtual
37:24
rails. Data is important and if you
37:26
look for just for the open source data that you have is not
37:29
enough and. Interesting.
37:31
Wait, say, tell me more about that.
37:33
So it's like you should a try to red team your own system, pen
37:37
test your own system and thinking about the system as a
37:40
whole, trying to break into it in any which way possible, not
37:43
just the LLM. But then if you're only doing
37:48
that, you don't know what you don't know.
37:50
And when you put something out live, it's going to get hit by
37:54
the very creative hackers that they are.
37:57
And so is it like you should get a third party to try and pin
38:02
test also? Or is it that you should like,
38:06
how do you get that data if you don't have it and you don't know
38:09
that you don't know? Yeah, you sure do choose.
38:13
I don't know if you have my open source model that was trained on
38:16
this data or you have, I don't know a tool provider that work
38:23
is works on this, that collect the data, filter this data to to
38:28
train like the graduates models. That's because it's very
38:32
difficult to create the graduals by internally by itself.
38:36
So it can use a lens as well, but it's it's vulnerable
38:40
vulnerable as well. And also to create a small
38:44
language models, you should have data.
38:46
So it's a challenge just like a cat and hat the but the
38:51
challenge is, is like it should what the best way here to
38:54
collect data is to work together with the cybersecurity thing,
38:57
the red teaming to understand the patterns and then implement
39:02
this this solution. Yeah, it makes sense and I
39:07
believe, I don't know. You, you have more, you see more
39:13
open source open size and open open source model around AIC QS.
39:19
You don't have many tools and and models on this space.
39:24
Let's say some models that you can, you could use as a wage
39:27
rails and then make sure that everything is OK.
39:31
So it's part of 1A path that I'm looking for.
39:35
Like to to create more a kind of open size around it.
39:40
Yeah, almost like the trick there is that as soon as you put
39:44
out a data set that can help you understand where there are
39:49
attack vulnerabilities, that data set is no longer valuable
39:53
because it needs to be behind closed doors.
39:58
It needs to be something that is almost not shareable or not
40:01
public. Yeah.
40:04
Do you mean the data right? Yeah.
40:07
Yeah, but it's a challenge you know of collecting data,
40:10
collecting new partners, learning and then you'll be
40:13
dating is constantly. So like almost like this as we
40:18
did in traditional ML and as you did a ton on retraining fraud
40:23
models, you're in a way continuously collecting data and
40:29
then retraining these LLMS. But you're not retraining them in
40:35
the sense that you're not going in, you're not going to train or
40:39
fine tune the LLM. You were just updating your.
40:42
Maybe it's your prompt or maybe it is the LLM calls something
40:46
like that. Yeah, I believe that's a good
40:50
point. It's iterating around learning
40:52
and iterating and improving, right.
40:55
Yeah, it's almost half process. So you can inspire on the
40:59
traditional ML and and use it in the in the AI application LLB
41:05
applications.