Latest / Agentic Conversations (formally mlops.community) / Robustness, Detectability, and Data Privacy in AI // Vinu Sankar Sadasivan // #289

Transcript

Robustness, Detectability, and Data Privacy in AI // Vinu Sankar Sadasivan // #289 · Agentic Conversations (formally mlops.community)

0:00
Hi, my name is Vinush Shankar Sadashivan.
0:03
I am a final year PhD student at the University of Midland.
0:07
Currently, I am a full time student researcher with Google
0:10
DeepMind working on AI jailbreaking.
0:14
So today we'll be discussing more about the hardness of AI
0:18
detection and redeeming of these AI models, especially with the
0:23
focus on generative models. And yes, I generally don't do
0:29
not drink coffee. And if I do get coffee, I go for
0:33
latte. What is happening, good people
0:36
of the world? Welcome back to another ML OPS
0:38
community podcast. I am your host Dimitrios.
0:41
And today we get talking about red teaming and also how to
0:48
jailbreak those models. Vinu did a whole paper and PhD
0:55
on being able to identify LLM generated text.
0:59
So we talked about watermarking and if it is a lost cause.
1:07
And don't think I forgot about you, we've got a song
1:10
recommendation coming in hot. Good old Tim Maya, Jimmy Maya
1:16
from Brazil. OK.
2:16
So let's start with why watermarking is so difficult.
2:23
And you basically told me or you didn't say it, but I read
2:28
between the lines in our conversation before we hit
2:30
record, which was all of that stuff that you see where you can
2:35
turn in a piece of text and it will tell you how much
2:41
percentage is AI generated versus not.
2:44
That's kind of bullshit. Or what?
2:49
OK, so I wouldn't say it's bullshit, but I would say
2:53
something which we should be not completely relying on.
2:57
So we had this paper where we were researching on different
3:03
kinds of detectors. So to start with, watermarking
3:07
is just one kind of detectors that we are looking at, which
3:10
has been predominantly a famous method of waterworking.
3:16
Waterworking has been there forever.
3:17
Like it's it has been there for images and for text from a very
3:21
long time. So for text earlier we used to
3:24
put like say a spelling mistake in between text and save the
3:28
spelling mistake keeps repeating it say a watermark or like a
3:32
double space or the pattern of the spaces or the punctuations
3:36
or these kind of things could be like a watermark.
3:39
But now things have changed AIF blooming people have made new
3:45
method for watermarking. So what I'm seeing is
3:49
watermarking is a really good technique, but the problem is
3:52
been there are attackers in the setting which you're looking at
3:56
then it by not be as effective as we think it is.
3:59
So, so in the paper which we looked at, we look at for
4:04
different kinds of detectors and one of the most important one is
4:09
watermarking, which also really works well.
4:12
And the other kinds are using a trained detector, which is what
4:16
I think most of the detectors out there are using right now
4:20
because language models are not yet completely watermarked.
4:23
Not all of them are. So it's just basically a
4:26
classifier where you give an input, it just says it's like a
4:29
dog or cat. It just says AI text or not an
4:32
AI text. And the other one is zero shot
4:36
detectors where you don't have to train a network but you just
4:39
use a network to somehow look at the statistics, say that look at
4:43
the loss values and say if it's less loss value it is probably a
4:46
part of age generated text. If it was not a low loss value,
4:52
it is human generated text because generally AI text
4:55
quality is higher so the loss values are lower and hence it's
4:58
AI text. And retrieval is another method
5:02
where you store all the AI generated text in a database and
5:06
then basically search given a candidate text if that text is
5:10
present in the database or not. So we look at all these kind of
5:15
detectors and we break them empirically in two ways.
5:20
One is to make AI text look more like human text and one the
5:24
other way is to make human text look more like AI text.
5:27
So type 1 and type 2 errors both.
5:30
And we also show between fundamental trade off of these
5:35
detectors and we show that as language models get bigger and
5:40
bigger, detection gets harder because if you think intuitively
5:45
about it, language models are highly capable when they get
5:50
bigger. They can easily mimic the style
5:52
of human writing if you give it the relevant instructions to it
5:57
or even enough data. So if I give like a longer text
6:00
of, OK, this is Donald Trump the way of writing text or talking.
6:04
And if we ask it to mimic with a lot of data given to it, it
6:07
probably would mimic it very well.
6:09
And it gets harder as language ones get even bigger.
6:13
So that's one of the other concerns.
6:14
So what I'm asking is just one of the tools which we are less
6:19
in the paper and we show it's easy to break them if an
6:22
attacker really wants to. So the take away from the paper
6:25
would be if someone wants to really attack something, then
6:29
excess, no foolproof technique right now.
6:32
And our theory shows that there will not exist anything like
6:36
that. So it's really good to have one
6:38
layer of security like watermarking for now just to
6:43
tackle cases where like people are directly using AI text out
6:47
of an AI model and we can detect such text very well using
6:53
watermarking. But if I really want to remove
6:55
it, it's. It's easy to remove that.
6:57
Now what are the cases that folks A would want to know when
7:03
it is only AI generated text and then B when someone would want
7:08
to be adversarial and not let the person know that they are
7:13
only using AI generated? Text, Yeah, yes, that's a good
7:18
question. So the cases where I wouldn't
7:21
want to show that it's an AI text with when I'm a student or
7:25
when I'm going to do plagiarism, I'm summoning my assignment, but
7:29
I used ChatGPT to write my answers.
7:32
So in that case, I wouldn't want my professor to know that I used
7:36
AI to submit my assignments. That's one major scenario which
7:41
people are looking at right now because that's the lot of the
7:44
caches where the lot of money is.
7:46
So all the leading text detection tools basically focus
7:49
on that for plagiarism purposes because there's a lot of money
7:53
there. And the other case where you
7:58
want to really detect is again, plagiarism would be 1 case.
8:01
It's like a min Max game here. So students want to make it look
8:05
like human text, but professor want to actually make it look
8:08
like AI text if it actually was AI text.
8:10
And the other case would be like spamming, phishing, this kind of
8:13
attempts where you might be actually talking with a chat
8:16
agent who is not a human. They might be scripted to fool
8:20
you into some scams and it's easier for them to scale up
8:25
these scamming items if they have access to AI, which is
8:29
really dangerous. What if there's an AI model
8:35
later on which is like so natural it it kind of words,
8:39
text to speech and they are making like they're basically
8:44
simulating a call centre talking to multiple people parallely
8:48
trying to scam them and make a lot of money out of it.
8:51
So here they would really want to make AI text or AI speech or
8:58
whatever morality it does look like more human like so that
9:01
humans don't detect it. But still, they get to do
9:05
whatever they want to do the adversarial objectives they have
9:08
without getting caught. Now you mentioned there were
9:13
ways that you broke these different settings or the
9:17
detectors, and one way was making AI generated text look
9:22
more human. Was that just through the
9:24
prompt? No.
9:27
So what we do is so that is one of the methods which you can
9:31
actually do. But as we discussed before the
9:35
conversation, the evolution of the systems, it has become
9:39
harder to do that. So now I think the models are
9:45
well fight tuned to somehow imprint the AI signatures
9:49
better. So it's harder to actually use
9:52
AI input prompts to just change the detection affect the
9:58
detection. Well.
9:58
So I was recently trying with all these tools, Gemini ChatGPT
10:02
and Anthropics cloud to see how actually you can give prompts to
10:07
make them look like less AI text.
10:11
So I can't really make a study out of it because it's very hard
10:14
to do manual prompting on like thousands of texts myself and
10:19
then do something. But what if you got out mainly
10:22
was commonly for most of these AI models was generally if I
10:30
give it prompts like convert passive text to active text or
10:36
or something like use simpler sentences, don't use like longer
10:39
sentences avoid and or punctuation and things like
10:43
that. If I give such such features
10:47
which models use generally to write very high quality text and
10:51
make them try to be little low quality by using, OK, use less
10:57
of which vocabulary like how humans would stop using a lot of
11:01
punctuations and write shorter sentences and things like that.
11:05
I get to make them break sometimes, but it's hard.
11:09
So I've been noticing this because I tried them out
11:12
sometimes with some gap in the timeline.
11:15
So I every say few months I tried this out and I find that
11:19
it's getting harder to do that. So the way which we did in our
11:24
paper and which aligns with the theory of a paper is basically
11:28
use a paraphraser. And why we did that was to get
11:34
our attack, the empirical attack which we do to be in line with
11:37
our theory. And one other reason is because
11:43
also we wanted to attack watermarking and one of them
11:46
attack matters with the watermarking.
11:48
The the first AI watermarking text paper, the AI text
11:52
watermarking paper showed was to change the words, just replace
11:55
words with another word. So that's the first naive attack
11:58
you would think of. So given an AI candidate text,
12:01
can you do minimal editing to it by changing the words to add
12:05
some synonyms? Or.
12:07
Change some of the punctuations made like add words like and or
12:11
and things like that to make minimal edits in terms of the
12:15
edit distance to see if I can break it or not.
12:21
But what isn't basically like you saying control F for all the
12:27
instances where Delve is in there, and then you replace all
12:32
of the Delve with another, OK, let's see.
12:36
And then it passes with flying colors.
12:40
So that would be the case for some other detectors, but it's
12:43
not the case with watermarking. So watermarking is quite robust
12:46
to it. So if you really want to attack
12:48
watermarking by just changing words, you might want to change
12:53
almost like 50% of the words in place, which is going to be a
12:57
very hard task, right? Because you can't change a lot
13:00
of the words in, even given a sentence with 10 words, you can
13:03
hardly change like two or three words in place without affecting
13:06
the quality or the meaning of the sentence.
13:09
So that's where paraphrasing is really important.
13:12
So basically a given a sentence, I can completely change the
13:15
structure of it. I can make your active voice
13:17
sentence to a passive voice. So it completely flips, changes
13:21
the structure. I can join different sentences
13:25
with and and ORS and things like that using a paraphraser, which
13:30
was not the case with if I simply changed different talking
13:33
sort of words in the passage. So I think to explain that back,
13:37
well, it's important to go into how watermarking works, how the
13:43
the first text word marking Pepper did it was it?
13:47
It's a simple algorithm, but but it works really well given.
13:52
So you're writing a passage. Think of it like this way.
13:55
You're writing a passage. You start with writing a passage
13:59
or an essay about the dog. So you say the dog was playing.
14:03
This is how a human would write a text.
14:05
You wouldn't think exactly how to pick the words, you just want
14:08
to pick the words so that it makes sense.
14:11
But how the watermark AI would think is OK?
14:14
I started with the word word the.
14:17
Now for the next word, I have to only pick words from 25,000 of
14:23
the words out of the 50,000 words I have access to in my
14:25
vocabulary. So suppose I have 50,000 words
14:31
in my vocabulary. The AI would partition that in
14:34
the two halves they call 1/2 the Red List and the other half the
14:39
list the green list. Now the AI would focus on always
14:44
picking the word from the green list, so eventually when it
14:48
writes word by word it would try to make most of the words in the
14:52
passage coming from the game list.
14:55
The what a human would do is it a human does not know what a Red
14:59
List and green list is, though he or she might end up writing
15:04
taking almost 50% of the words from the Red List or the green
15:08
list, right? And what I what this watermark
15:12
AI model would do is it will end up having a passage which is
15:14
mostly having the green list words.
15:18
So this is the watermark. So the AI model knows given a
15:22
passage, it has 90% green words in them and 10% red words in
15:27
them. So it's very highly likely they
15:30
are AI text. But given a human text, OK, it's
15:34
around 50% red words and 50% green words.
15:38
Now there are more advanced versions of this, which makes it
15:42
better in terms of quality. So what they would do is instead
15:45
of having a hard partitioning of Red List and green list, because
15:48
sometimes when I write the word Barack Obama, the word Obama
15:52
might be in Red List, but I want it to be green list because I
15:55
wrote the word Barack before. I want Obama to happen after
15:58
Barack. So what they would do this, I
16:02
would make this reckless green list partitioning dynamic.
16:06
So every step I would choose a new word my Red List and green
16:10
list will keep changing and that would change based on the
16:15
previous word which was written. So if I had written Barack for
16:18
the next word, the green list and Red List will be determined
16:21
by that word Barack. So it would be modeled such a
16:25
way that Obama will be in the green list mostly something like
16:28
that. So this begs the question, all
16:33
of the model providers or all of the model creators need to have
16:40
this inside of their models for it to be valuable, right?
16:44
And you also kind of all have to be on the same page on how
16:49
you're doing it so that then when you have some kind of
16:53
watermarking detector, it can detect if it's AI generated
16:58
content, I guess you could take all the different ones at each
17:03
flavour of model has its own flavour of watermarking.
17:06
And then the detector could just have all of these different ways
17:10
that it can detect it. But you mean to start at the
17:15
model level. And if the model providers
17:17
aren't doing this, then you can't detect it.
17:21
That's a really great operation. That's one of the major
17:24
limitations which watermarking models have right now.
17:28
I'm glad you pointed it out. So but if if open air has its
17:36
model watermark and Gemini does not have or vice versa it it
17:40
does not make any sense because these are using models and
17:43
attacker if they want to use the model, they would go for the non
17:46
watermark AI model which exists out there.
17:49
And other main concern which I don't know which no one is
17:51
talking about right now is we already have open source models
17:56
which are released which are not watermarked.
17:59
So I have Llama 3.2, the larger models I can always download and
18:06
keep them in my hard disk. They're pretty good in doing
18:08
whatever I want right now for scamming and things like that or
18:11
plagiarism even. I can always use them for
18:14
writing AI text. At at least to what good quality
18:18
we think is good right now maybe I mean at at least at par with
18:23
humans. So it's also crazy that we are
18:27
still trying to do watermarking. OK, I understand in future it it
18:31
might be more powerful. Watermarking might be something
18:34
which we want, but to some extent we have done the damage
18:37
already. Someone who needs to do the
18:40
damage in future could still do a lot of automation with the
18:43
open source models which are already out there, which is not
18:46
watermarked, right? And also if all these AI
18:50
companies come up with different watermarking schemes, it might
18:53
be harder for detection in the future like you mentioned.
18:57
So they all always have to be on the same page, someone
19:00
regulating them how to watermark and how to do detection and
19:04
things like that. Because say in the future they
19:06
are like 10,000 AI companies and you don't know what is coming
19:10
from there. You might be waking a lot of
19:12
computer on detecting which came from there and looking at the
19:16
prominence of the AI text, which might be hard.
19:20
But there are like technical limitations to this problem
19:25
which we haven't addressed yet. Yeah, the genie is already out
19:29
of the bottle. So what are you going to do on
19:32
that? It does feel like especially
19:35
with the SEO generated content from AI models that you can see.
19:45
I don't know if Google does or doesn't punish you if you throw
19:50
up a bunch of different blog posts at one time and you're
19:56
just churning out AI generated content.
19:58
I think I read somewhere on their SEO update that if it is
20:02
valuable to the end user, then you're not going to take a hit
20:05
on your SEO score. But you have to imagine that
20:09
there is a world where they're looking and they're seeing, hey,
20:13
this is. 90% AI generated, if they can figure that out, they
20:20
would want to. But right now it's almost like,
20:23
yeah, maybe they can figure it out a little bit as you're
20:26
saying that if they watermarked things and if everyone was on
20:31
the same page with watermarking, then it would be useful or we
20:36
could potentially see that. But at this point in time,
20:40
that's not happening. Yeah, Yeah, that's true.
20:44
So then what else about watermarking before we move on
20:47
to red teaming? My other favorite topic?
20:49
Yeah. So I I think I was coming to the
20:52
attack which we were doing in the paper to remove water.
20:56
So the really important thing with the text watermarks that
21:00
access right now is this dependence on the previous word
21:05
which it had sampled. So if I had sampled Barack, I
21:08
want to sample Obama next. So I might have a random Red
21:12
List or green list for the next word, but with I would increase
21:16
the probabilities of sampling the word Obama so that it is
21:20
sampled. That's one thing so that the
21:22
text quality is not affected. And the other thing is the green
21:26
list and Red List is now dynamic so it changes with every word
21:30
depending on the previous word as the IT is basically the seed
21:33
to the random number generated to for partitioning the
21:35
vocabulary to Red List and green list.
21:39
So the way watermarking works is basically this.
21:43
And if you want to attack it, if you just think of it, if I
21:45
change one word in the middle, it might affect the red listing
21:49
green list of the next word, right?
21:50
Because that might not be ending up in the red dot.
21:53
Green. Word before that.
21:56
But the problem is if I do just one word, it only changes the
21:59
red dot green list for the next word.
22:01
And so if I need to completely change all the number of green
22:05
words in it, I have to essentially make a lot of edits.
22:09
But what if I rearrange these words?
22:12
Then the structure of this, the ordering is completely disturbed
22:15
and the Red List of being list is completely random now.
22:19
So that's what happened. When you rewrite a sentence, you
22:22
generally don't try to preserve the exact ordering of the words,
22:26
but you write them so that it's essentially rewritten.
22:30
So if I have say, a longer sentence, I can probably slap
22:35
the sentence A and BI could probably say B&A even within AI
22:40
could change to grammar maybe, or I could change active voice
22:43
to passive voice and so on, or even synonyms and things like
22:46
that. So the attack which we look at
22:49
is using an AI model itself to paraphrase the AI text.
22:53
So ideally you would want the AI paraphrases output also detected
22:57
as AI text because it is again AI, not a human.
23:02
So what we observed was if you use a paraphraser model to
23:06
paraphrase the AI text, the output you get is mostly
23:10
detected as human text for a lot of this detection techniques.
23:16
But one of the most robust technique is watermarking.
23:19
It is still robust, likely to paraphrasing a little bit
23:24
because the current paraphraser have not trained to do this.
23:29
So if we give really good manual prompts to the paraphraser, we
23:33
could actually do it in one shot.
23:35
But what we ended up showing was something called recursive
23:38
paraphrasing, where the AI text which you paraphrased once would
23:42
be again given back to the paraphraser to paraphrase once
23:45
more. So it will be paraphrased twice.
23:48
And then you can keep doing this multiple times with whatever is
23:54
whatever you want based on the strength of the watermarking.
23:57
So we find that just after two rounds of paraphrasing,
24:00
recursive paraphrasing, the watermarking schemes accuracy
24:04
goes below 50%. So we just see that two rounds
24:08
of paraphrasing is enough for breaking the watermarking
24:14
algorithm. And this is what we essentially
24:17
show in our theory too. So the theory goes like this
24:21
suppose if you have like a distribution of text which is
24:24
basically AI text plus human text or which is a subset of
24:28
that right? So it can be anything.
24:30
So given a sentence A, passage AI can also look at another set
24:37
B, which is essentially all the sent passages similar to A in
24:43
meaning. So even if I take something from
24:45
B, I'm OK to replace A with that, right?
24:50
But the problem is for watermarking in the set B.
24:53
Suppose there are 100 passages which I can replace A with the
24:56
sentence the passage A with for a watermarking agent.
25:00
I can't say out of all the 150 of them are watermarked, because
25:04
if that's the case, it's likely that a human writing a passage
25:08
would be 50% coming in the watermarked.
25:11
Label. So I I have to make sure that
25:14
the likelihood of a human writing a passage with similar
25:16
meaning is less because it has to be watermarked.
25:20
You know, for that out of the 100, I have to say, OK, one or
25:23
two of the sentences are only labeled as watermarked and
25:27
others are not. Still the the false positive
25:30
rate, which is a human writing a text and detected as AI text is
25:32
one person or two person, which is quite high.
25:35
But OK, let's be, I mean, lenient on that.
25:39
Just say, OK, one or two of the 100 texts are labeled as AI
25:44
watermark text. But now the problem is, is easy.
25:48
So I have the first sentence labeled as AI watermark.
25:51
What if I use a paraphraser, which is say really good and I
25:55
hope from the first text to one of the random 100 text that very
25:59
likely I'll drop onto a text which is not watermark.
26:02
Now because the watermarking was designed in such a way that it's
26:06
likely to drop into another text which was not watermark.
26:09
So if you look at this is a trade off.
26:11
If I try to increase the watermarking strength to
26:14
parafacing, I have to increase this number one out of 100 label
26:17
as watermark to say 10 out of 100 label as watermark or 50 out
26:20
of 100 label as watermark. But if I do that, I'll end up
26:24
making a human falsely accused of plagiarism with a higher
26:28
chance. So it's essentially a theory
26:30
between type 1 and type 2 error if we use this kind of detection
26:36
systems. So this is what our theory shows
26:39
you. So our theory says even for the
26:41
best detector that can exist out there, we upper limit the
26:45
detection performance of this model using the distance between
26:49
the distributions, which is jargon and we don't need to go
26:52
into that. But essentially, yeah.
26:56
But essentially for the best detector out there, we are not
26:59
claiming that's watermarking. But even if something that's
27:01
better than watermarking, the best detective that can access
27:04
theoretically out there is upper limited by a quantity which we
27:11
characterize in our paper, which is still not 100% reliable.
27:16
And the performance of that still has a trade off with
27:19
respect to true positive rates and a false positive rates which
27:21
is the type 1 and type 2 errors. So you have to make it go down
27:27
on one of the errors to make it better on the other error.
27:31
So that's the major highlight or the major results which we show
27:34
in the paper. So following up on that,
27:39
paraphrasing is something which is really considered a good
27:42
attack right now and lot of the leading test detection tools
27:45
like Turnitin have been using methods to deal with it.
27:50
The what they are doing in the the recent blog post they had
27:53
mentioned was to use a paraphrase detection tool which
27:57
tells you if an AI text was paraphrased or not.
28:00
If it was paraphrased you can say it's an AI text or not and
28:04
the problem is if you do that you are ending up hurting your
28:07
true positive rates. Yeah.
28:09
Well, yeah, that's exactly what it sounds like is that you have
28:12
this upper and lower bound or the right and the left side of
28:15
the spectrum. And the more that you go on to
28:18
the right side of the spectrum, then you're going to get these
28:21
false positives and. Exactly.
28:25
You can't really win no matter what you try and do.
28:28
Exactly, so if more money isn't detecting innocent kids as
28:34
plagiarized, OK, you can choose to make money like that.
28:37
But if most money is OK as an AI detection tool, I would probably
28:42
want to say that OK, I caught more students plagiarizing,
28:47
maybe that's more efficient for them.
28:50
But in in. But it can actually be ending up
28:52
having bad reputation for them falsely accusing students.
28:55
So it's if that's why they need to make.
28:58
But the real question is, do we really want to use this for such
29:04
strict plagiarism detections? Because I think as we go ahead
29:10
and as these technologies come up, we have to find a way to use
29:13
them collaboratively for our work because they improve our
29:17
production instead of replacing us or something like that.
29:20
It's, I believe it improves our production and we have to learn
29:23
to use them as a tool instead of using them completely for
29:28
plagiarism. So yeah.
29:32
That you were talking about there for a second, you were
29:34
mentioning how you feel like it's becoming more and more
29:38
difficult to make it make these models write more like humans
29:45
and they're almost being trained or they're being red teamed to
29:51
not write like a human and have their own distinct AI voice.
29:57
Yeah, yeah. So I think this is just a
30:00
speculation. I'm not sure what actually goes
30:04
into the training of this models.
30:07
So, but yeah, from what it looks like, I think there has been
30:11
recent steps taken by this AI giants to make the models more
30:20
easy to be detected. It could be either that or it
30:23
could be the AI detection companies make doing a better
30:26
job at making their detectors. It could be either one of them.
30:31
But I feel also in at some point of time in watermarking was
30:37
introduced, some of these models text quality actually went down
30:40
a little bit. So while I was seeing some of
30:43
the comments on Twitter, I saw sometimes people were saying, is
30:46
it me or just is it just me or everyone else finding that
30:50
ChatGPT text quality has gone out a little bit.
30:52
I'm not sure if it's because of these kind of trainings added on
30:56
top of it. It could be that it could be
30:59
some other safety alignments which they have added on top of
31:01
the model which actually trades off on the performance.
31:05
Yeah, it's a side. Effect.
31:06
It's an effect of it. So there's a trade off in
31:09
everything. If you're tried to make
31:10
detection better, you have to trade off on the text quality or
31:13
on the OR on the type one or type 2 errors of detection and
31:16
things like that. But I think, yeah, that my
31:20
speculation is models could have been fine-tuned to make
31:24
detection more possible because recently garments have been
31:28
pushing these tech chains to have watermarkings embedded at
31:32
them. DeepMind recently released their
31:35
watermarking paper on nature and Metal already has been trying
31:41
has been all in the game for the image watermarking and things
31:44
like that openly. I'm not sure what the scenario
31:48
is, but I think all these companies are pushed to have
31:52
watermarking in them. So they are trying to possibly
31:56
do potentially they're doing something which helps detection
31:59
better. Yeah.
32:01
But these are some of the trends which I think which I've been
32:06
noticing, which is again a speculation, but I think 1
32:12
common features with all these models have been there have been
32:14
trained to have really good text quality.
32:17
So that could be another side effect of it.
32:19
They tried to make use little very poetic words and poetic
32:24
devices, things like that to make text look very nice,
32:27
ornamental words and things like that.
32:30
So if we give some instructions, it was earlier very easy I think
32:35
in my experience. So I think that's because of the
32:37
training which these models have been gone through to make
32:43
detection far more easier. So recently, if I tried to make
32:48
it OK, I just say, OK, make it sound like how Donald consists.
32:51
Still, the text which comes from the model still is dated as a
32:55
text, which was not the case earlier when I had checked like
32:59
a year back or so. So yeah, potentially it's either
33:03
the models being trained to do that or the detection tools are
33:07
improving with time. Yeah.
33:10
OK, so let's take a turn for red teaming and just give me the lay
33:14
of the land on how red teaming has changed over the years
33:18
because of all of the models getting better.
33:22
I think there's a whole lot more people that are red teaming
33:26
models, whether they are getting paid to red team them or not.
33:30
Everybody loves to think about or loves to be able to say I got
33:36
ChatGPT or insert your favorite model to say this or to do this.
33:41
It's almost like a badge of honor that we can wear on the
33:43
Internet. So what have you seen over the
33:46
years and how is it different? Yeah.
33:50
So I think it's like a very recent advancement which we've
33:54
been seeing in AI, people trying to jailbreak, people trying to
33:58
do red teaming. But I'd like to see more of it.
34:01
Like it started off with people and then it got went to like
34:05
automated things because the effort which people have been
34:07
putting is almost all similar because it's just manual
34:12
attempts, trial and error, some insights which you get from the
34:15
model, the feedback you get, you put put it back to the
34:18
prominence and I treat the process.
34:20
So it started off like that. It it was called do anything
34:23
Now, Dan. So there was a page where people
34:27
were compelling different ways to jailbreak these models.
34:30
You could either write manual system prompts or write manual
34:34
input prompts to make the model thing that you are innocent, you
34:39
are not going to do something harmful.
34:42
So some of the I think classic examples are where the question
34:46
is you have to make the model say how to make a bomb, which
34:49
for some reason you can find it on Google, but you don't want
34:52
your model to output. But I never thought about that.
34:57
Yeah. But I mean, yeah, sure, it's
35:00
still a good objective to keep in mind.
35:03
But yeah, I mean, you can just take it as a toy example for
35:06
now. We don't want the model to talk
35:08
about something, OK? It's totally cool.
35:10
We just don't wanted to talk about it.
35:12
And how do we do that? Because bomp can come in
35:16
different contexts, right? We can't just use a word
35:18
filtering algorithm. Bomp could be something
35:20
explosive, or I could also say, OK, that was BOM, maybe to say
35:23
that was like a fantastic thing or something like that.
35:27
I don't know in what context you would be using it unless I
35:30
understand what it is. So I can't just use a string
35:33
matching algorithm. See.
35:35
OK, if there's a bomp in it, I don't reply to that.
35:37
That's not going to work. Even the word screw, if I say
35:41
I'm going to screw that, it might be OK, but if I'm saying
35:45
I'm going to screw you up, then it could be something offensive.
35:49
So there are things which comes in context.
35:51
So it's not to just give context of things.
35:55
It's not easy to do word filtering in most of these
35:59
cases. And also if you have to make
36:00
like a list of a blacklist of these words, it's going to be
36:03
really large and you can't end up doing that, especially when
36:06
these models are getting multilingual.
36:08
Now it's hard to make maintain a list of words which you want to
36:11
filter on. But the thing which people used
36:14
to do was to write manually craft the systems where they say
36:17
my grandmother was sick or things like that, and I have to
36:20
make like a magical portion for her or say it's for my school
36:25
project. I really have to get the good
36:27
grade. It's it's funny.
36:30
It's exactly how you would try to fool a human to get them to
36:33
answer something. Saying, acting, working on the
36:36
sympathy aspects, emotional aspects offered and making the
36:40
models say since they are human aligned somehow it ends up
36:44
breaking it because that's how they were trained.
36:47
Also they were safety aligned with human values.
36:50
So probably it is expected that they break to these methods
36:54
where human would break an average human or abila average
36:58
human would break based on the way in which you be trained it.
37:04
So that was initial evolution of the taming, like where it
37:07
started, where people showed OK, there are this manual techniques
37:10
where I can write prompts to break them, which started off as
37:13
a prompt engineering trick where people initially use prompt
37:16
engineering to improve models performance.
37:18
But now they started using it to break models.
37:21
But now it's more like an iterative process.
37:23
So you give it a question, it it with a prompt, it does not
37:28
answer it. You refine the prompt eventually
37:30
but you essentially get not much signal.
37:34
It ends up saying I can't answer your question but it is really
37:36
not a good signal for you. But you end up doing Thailand
37:39
error to reiterate and re refine your input from such that the
37:45
model somehow breaks into some later it iteration of your
37:50
attack. But from there it has advanced a
37:53
lot more. People have come to a point
37:55
where it is more automated now, which is more dangerous because
38:00
then the attacks get scalable and do much more harm than what
38:04
it could do when you were mad in the system prompting.
38:07
So The thing is what the models have been doing after, So what
38:15
attackers have been doing after that.
38:16
There was this paper last year which was from Andy Zoo where
38:21
they introduce an algorithm called GCG, which is essentially
38:26
a gradient based algorithm. So what they do is if I have a
38:30
question how to make a bomb, I can add random 20 tokens after
38:35
that and then use gradients to optimize that to select like a
38:40
random set of 20 tokens, basically the suffix tokens
38:44
which makes no sense, but then added that as the suffix to the
38:48
input. How to make a bomb?
38:49
The model just breaks, so it doesn't make any sense to us,
38:52
but the model somehow interprets that and breaks to it.
38:55
So this is something which was really expected if we come from
39:00
machine learning adversarial literature because machine
39:03
learning models which an essentially neural networks can
39:06
be broken with some input perturbations.
39:09
This is well known strategy in the adversarial literature.
39:14
But the thing was it was hard for language models because the
39:18
way they work is very different from the traditional machine
39:21
learning model which we have used.
39:23
Because in tradition, machine learning, people were mostly
39:26
focused on computer vision and continuous data, where you have
39:29
say images, which is like a different kind of data where you
39:33
have continuous data, right? It's just pixels between value
39:36
zero to 255 or between -1 and one.
39:39
It's just continuous data. But for text, when you come to
39:41
text, they're just discrete data, which is just say 50,000
39:45
tokens. So you just say this talker is
39:48
255 or maybe 50,000 or something like that.
39:50
So it's they're discrete. So The thing is when you do
39:53
gradients, take gradients on this with respect to the input.
39:58
Since the input is discrete, it's really hard to take
40:00
gradients on them because they're not defined at all the
40:02
points. So the attacks were not
40:05
effective or most of the times when we tried to do attacks on
40:09
language models. But this recent work did
40:11
multiple tricks or multiple hacks to make that work out.
40:15
And this is the end result where you add random jargon random
40:20
suffix. To end of the question, it will
40:22
break. And it's just random words that
40:25
if so, like I can have my prompt and then I can just think of
40:29
random words. Or is there specific words that
40:34
you throw on to the end? Or just specific letters?
40:37
Yes, it looks random to us, but to the model it does not say
40:40
specific words according to the model, just optimized to make it
40:44
break. But how does the attacker know
40:47
what the optimized words are? Yes.
40:51
So one is if you have access direct access to the model, you
40:54
can optimize the model with taking gradients of the model
40:59
and updating the words. The other method which they show
41:02
is something called transferability which is again a
41:05
non phenomenon in adversary reputation.
41:08
What you do is you have access to say 3 or 4 open source
41:12
models. You find the suffix which breaks
41:14
all of them. Right now, if you use this
41:18
suffix to a new model which you'd never had access to, it
41:21
might end up breaking to that. So that's transferability.
41:24
And they show that this suffixes can be transferred to a new
41:27
model. But you can train use Llama to
41:29
train for the surfaces and use that to big chargeability.
41:33
And so it doesn't matter. It's not about the training of
41:38
it, it's really about the model architecture.
41:41
Exactly. So it's one thing probably
41:43
because most of these models are transformer based, that's one
41:46
thing. And the other thing is the data,
41:48
most of these models end up using very similar kind of data.
41:50
The structure is same. So the way you can break them is
41:56
similar. Probably that's why I think
42:00
transferability works because in lot of the cases which I have
42:02
looked in in language models, transferability works quite
42:05
well. I think it's because of the
42:07
underlying similar text which was used for training.
42:11
And so this is 1 vulnerability we could say how are the model
42:17
providers combating it? Yes.
42:20
So one thing is again to use automated red teaming to combat
42:25
this. That's one way.
42:28
OK, so let's go again in, in a sequential way, the evolution of
42:34
the different systems too. So the first, this particular
42:38
method which I discussed you is just adding a random jargon,
42:41
right? So if you look at here, the
42:44
quality of the text goes bad. So one easier way to do it, I
42:47
look at the input prompts, text quality.
42:49
If it's really bad, I just don't answer to that.
42:51
I just say I don't understand your question.
42:53
So that's one way, which is why latent later attacks came to
42:57
improve the readability of the suffix that they actually make
43:01
text which is readable and do the similar attacks.
43:06
So the models can't really detect based on the quality of
43:08
the text. So what models end up doing
43:12
there is one thing is chain of thoughts.
43:16
The other is to have something a llama guard where they train use
43:21
a copy of the llama model and make it attain like a
43:24
classifier. So it is trained to see if this
43:27
input is harmful or not right? So a pre trained llama model is
43:31
taken and it is trained to just do a classification task where
43:35
it is given a set of harmful Proms and a set of not harmful
43:38
Proms and made to give a label 0 or one if one it's harmful.
43:42
IF0 it's not harmful. And you can also add the
43:46
training data set of this classifier.
43:49
These kind of adversable Proms, which we designed earlier to
43:52
make them robust to this kind of attacks.
43:54
But again, if you use AI to defend AI, this the AI in your
43:59
pipeline, if one of them breaks, all of them breaks.
44:03
Yeah. And and 'cause this is all on
44:06
the input, right? There's none that are happening
44:09
on the output. Or is there also another filter
44:12
that's happening on the output in case it goes through the
44:16
input catch up? So there are multiple ways these
44:20
systems work. Some work just on inputs,
44:22
because if you want to really save on the compute time it's
44:25
better to look at the input. But if you have the board the
44:29
the capacity to look at both outputs and inputs, that's the
44:33
best method. So Rama guard ends up looking at
44:36
output and input if it has a capability of doing that.
44:40
And some methods even look at the internal activations of the
44:43
model. And there are detectors which
44:46
are trained to see if my if I have a layer of transformer and
44:49
if I look at the activations of different neurons of them, if
44:53
say a set of these neurons activated more, it's probably
44:57
harmful prompt. So there are even methods which
44:59
even look at the internal activations of this models to
45:02
see if it's the harmful text or not.
45:06
So it's like basically the dark side of latent space.
45:10
They can know where that is, the back alleys, and they can say
45:13
like, hey, if you're travelling in these back alleys of the
45:16
latent space, you're probably up to no good.
45:19
Exactly. So they call it circuit breaking
45:21
where you try to make the model understand if it's it's
45:26
activations are towards the dark side of it and then just break
45:29
it off right there and stop the generation.
45:31
And just that's why sometimes you have seen the models just
45:35
stop generating some things. They're probably uncertain about
45:39
it or they just are just the circuit is just broken so that
45:42
they don't continue it because it's probably going to something
45:44
harmful. Wow.
45:47
And have you seen because I know you're doing a bunch of red
45:51
teaming right now for deep mind, right?
45:53
And I can only imagine that you've been having fun with all
45:57
of the models, not just deep minds I and so have you seen
46:02
different ways that certain models are stronger and and
46:08
certain models are weaker? So I think that's again a red
46:15
teaming is to me it's somewhat broad term because in my last
46:22
work which was published at ICML, we look at red teaming
46:26
algorithm where we find an automated algorithm to make
46:31
again readable suffixes, which makes the model but in a fast
46:35
way. So the GCG algorithm which I
46:37
mentioned to takes like 70 minutes for optimizing the
46:40
suffix which is really large. So what we did was our attack
46:45
would take like one GPU minute to take the model and that what
46:49
would be published at ICM. And in that particular paper we
46:53
just proposed the algorithm. But the capability of this
46:56
algorithm is tremendous because we since it's fast and from an
47:02
Academy sitting back then with less GPU's, we could try out
47:05
different attacks. So one is jail breaking which is
47:08
one kind of red teaming I would say it which exist.
47:12
Other we do is something called hallucination attack, where we
47:15
change the prompt such that the models end up hallucinating much
47:18
more. Right.
47:20
And the third one is something with privacy attack where we
47:23
attack the prompts such that the existing privacy attacks
47:27
performance is boosted. So for example, there's
47:30
something called membership inference attack where you want
47:32
to see a given say I give got a text passage from Harry Potter
47:36
and ask if it was part of your training it or not.
47:39
So there were attacks to do that well, but let me.
47:44
Get it was so. Yeah, we can't really give a
47:50
guarantee on that right now, but I am definite Most of these
47:54
models used Harry Potter for training them.
47:57
They actually generate sometimes verbatim exact text from Harry
48:01
Potter. Yeah.
48:04
Oh, that's classic. Yeah.
48:05
So these models, So what we did was the third attack was privacy
48:11
attack where we attack the input such that the more the
48:13
performance of the privacy attacks improves.
48:15
So there are different kinds of attacks which you can think
48:18
about and people have been only looking in jailbreaking for a
48:20
teaming, but we have noticed it really depends upon your
48:23
training. So LAMA is one really good open
48:26
source model which I know is good at jail but resisting to
48:30
jailbreaks. So when you come compare it to
48:33
other methods, models like mistrial from model or Vicuna or
48:45
Llama and things models like these, we find that Llama is
48:48
quite robust to jailbreaking attempts.
48:52
So is tragic BP on Claude or Gemini, but Claude does really
48:59
well in terms of defense most of the times.
49:03
They have a lot of safety filters and chain of thoughts
49:05
going in the background, which I understand is why they are very
49:10
good at it because they're mainly an AI safety research
49:14
organization. So they are focused more on
49:16
safety. I'm assuming they have put more
49:18
compute on making them better at safety.
49:22
But yeah, coming back to open source models, Llama has been
49:26
really good at not jailbreaking. So we found that it's easier to
49:31
jailbreak llama when you ask to make it to generate fake news.
49:35
So Llama has a vulnerability where it generates more of fake
49:38
news. It's easier to make them
49:39
generate fake news when compared to other models.
49:42
And one thing to note here is that this is just jailbreaking
49:46
attacks. And all these models have been
49:48
fine-tuned to be resistant to jailbreaking attacks.
49:51
But we when we move to hallucination attacks, we find
49:53
that llama is equally breakable when compared to all other
49:56
models for hallucination attacks.
49:58
So the hallucination attack is essentially we had a suffix and
50:01
the model ends up saying, OK, eating watermelon seeds is
50:04
dangerous for you. You might even end up dying
50:05
eating watermelon seed or walking in the wardrobe.
50:08
You might end up dying if you walk in the wardrobe.
50:10
So things like that. So we have actually examples
50:14
where Lama ends up doing this after an attack.
50:17
And it's crazy how it works because it really depends upon
50:21
how you fine-tuned your model. If you did not, if you forgot to
50:24
find your model to be robust to hallucination attack, your
50:28
chances are gone. So once you deploy the model,
50:32
it's out there, people can attack to make it help me more
50:34
make misinformation out there. So yeah, it really depends on
50:39
your training, how your training is done.
50:41
So when comparing these open source models to the production
50:44
models which are already out there and not open sourced
50:47
closed models out there, I think they have been extensively
50:52
fine-tuned to be robust to these kind of techniques.
50:56
And also they are in academic setting at least we actually let
51:00
them know this attack is there. We are going to publish them so
51:03
they get time to adapt to it if it's a really important attack.
51:07
And also these companies have these programs where you were
51:13
earlier asking if they're paid to attack or not, where they
51:15
have like a bounty program where you red team them and find
51:20
vulnerabilities and report to them.
51:21
And the model is actually trained to make them better on
51:23
that. So they are actually doing good
51:26
red teaming research to be well ahead in the game before someone
51:31
else outside their organization breaks it.
51:34
So I think as this gets like how the open source community grew,
51:40
if the red teaming community grows very large and it outgrows
51:44
the community that the companies have, it might be harder for
51:48
companies to cope up with the red teaming approaches which
51:51
would exist. But it we could make it little
51:56
more harder for the attackers by adding this kind of differences
51:59
which the models which these companies use right now.
52:02
But yeah, again, it's the same problem which existed in
52:05
detection. I believe.
52:07
It's not easy to have a complete solution for jailbreaking
52:11
because if you look at it, the definition of jailbreaking is
52:14
not fully clear to us. What are the kind of question we
52:17
are not supposed to answer to the?
52:19
If we do not know that and we we don't know how to take the model
52:22
for that, That's the fundamental problem which we are looking at.
52:27
If we can define the problem, how do we find a solution which
52:30
is well defined? So we need to define the problem
52:34
first, which is very ambiguous because the context changes and
52:42
the scope of harmful questions changes and things like that.