Latest / Agentic Conversations (formally mlops.community) / ML Security: Why should you care? // Sahbi Chaieb // MLOps Coffee Sessions #51

Transcript

ML Security: Why should you care? // Sahbi Chaieb // MLOps Coffee Sessions #51 · Agentic Conversations (formally mlops.community)

0:16
Welcome back to another edition of our ml Ops coffee session
0:21
podcast. Today we are joined by none
0:23
other than Chavi and I brought him on this show because I
0:29
wanted to Talk about the article that he wrote some time ago now
0:34
and it all about machine learning systems and security of
0:41
these machine learning systems for those of you that have been
0:45
listening for a while, you know, that security is something high
0:49
on my list of priorities. When it comes to machine
0:52
learning and bat and a few other things, but we are getting into
0:58
security specifically today. I will tell you that if you are
1:02
not in our security and privacy channel on the MLS Community
1:07
slack and you enjoy these kind of topics, I guess that is the
1:11
first important piece if you enjoy or think that they are
1:15
important, then jump into that channel because there are smart
1:19
people sharing all that they can about security and privacy.
1:23
And one of the great pieces that has been shared in, there is the
1:27
article from shabby. So with me, Always is Vishnu,
1:32
welcome shabby and I'm excited to talk to you about this.
1:35
Maybe we can start just by getting a bit of background
1:38
information about yourself. Yes, thanks a lot, Jim Pearson
1:44
mission of our having me here. I just yeah, want to thank you
1:48
also for all the work that you're doing with the dma looks
1:51
Community, it's really a community that I have that.
1:56
I love. It's there is so much content,
2:00
good content out there. A lot of experts, sharing really
2:04
interesting things. A lot of strengths that we
2:07
talked about in the community. You always bringing superstars
2:13
in your mirror. Meetups and so thanks a lot for
2:17
all of your work. Yes, to present myself.
2:22
So, I'm I'm sorry. So, I'm a data scientist, I
2:28
started working like six years ago, six years ago, in start-up
2:33
in Paris, so we were actually a bit early.
2:38
So we started trying to automate Machine learning and also work
2:43
on putting the models in production and explain this
2:47
models like six years ago. And after that, now I working at
2:54
SAS so it's a much bigger company and with a lot of
3:00
powerful tools, also to manage this, this data science word in
3:06
order to have the best platform for for data scientists.
3:11
Awesome. So I think the logical question
3:15
here is. Why did you write this paper?
3:18
What inspired you to write the article?
3:24
Yeah. So let me start with a story
3:31
like all data scientists experience this kind of this
3:36
kind of things. It's what I call the lawyer.
3:41
Put your finger into production kind of situation.
3:44
So it's not directly related to security but yeah, it happened
3:50
in my early days as a data scientists I worked on on a huge
3:56
project so it was with International customer.
4:01
So we spent a lot of time on this project.
4:06
Working with the customers we went abroad.
4:08
It was in China. The customers game.
4:11
Two friends to work with us and we did a really good job to to
4:18
actually have the best model in terms of performance and all
4:23
this work was I was finished. You have my boss who came to me
4:29
and you say great, now you put it into production and that's
4:35
where you start to think about all the things that maybe you
4:40
could have. On differently if you want to
4:43
put your model into production. So maybe you used some, some
4:50
some data that you want to query in real time but you are, your
4:56
mother would be offline. So this this doesn't work you
4:59
made the super complex, architectures of models,
5:03
combining in some link different models and it's too difficult
5:07
also to to deploy it. So there are A lot of choices
5:13
that you make when you're building your mother's choices
5:17
that you do differently, if you had the visibility about, what
5:22
are you going to do and what are you going to put in production?
5:28
And I think that security is, is one of this, this choices that
5:36
you could do differently. If you're aware of the the
5:40
implication Of what to do so recently.
5:43
So I started to think as a data scientist.
5:46
Of course I want to have the best performance for my mom's.
5:50
So most of the time I'm going to fetch a lot of data in order to
5:56
have more data from different parts.
5:59
And so it's a it can be for example scrapping some some
6:05
weight websites using some open data.
6:11
Government and using Apple trained model from from from
6:16
GitHub. And actually when you think
6:18
about that each form of these choices is actually a security
6:25
threat. It's as soon as you go and fetch
6:32
something from the from the internet.
6:36
So some external data or modems, there is a risk.
6:41
So that's this, this kind of data could be malicious or could
6:46
contain actually some some, some something that that is the
6:53
threat for the security of, of the model.
6:58
So that's how I started to think about the subjects.
7:01
And at the time, I I tried to look about the different kind of
7:06
threats that we could have in machine learning and And I don't
7:11
find actually the source where I could find this, this whole list
7:18
of threats. So I decided to do this research
7:23
and and to look at to look to look at this subject and I
7:29
discovered that actually, it's a research area.
7:33
So that's why we don't have so much content about this subject.
7:39
But in research, Um what you have is a lot of really bright
7:44
people but what they doing is like designing attacks in order
7:51
to attack some machine learning models and others designing
7:57
defenses against against these attacks and then it's a cycle
8:03
because another guy is going to come and he's going to design
8:07
another attack that we going to break this defense.
8:10
So it's kind of a game but actually there is nothing that
8:15
could help me as a data scientist working on the
8:18
business project customer in order to make my model safer and
8:24
have a clear idea of what's happening.
8:27
So that's how the idea of this article came.
8:30
I did some research and that under that / writing this piece.
8:35
Well, I love how you went from needing it.
8:39
First of all, I love the. Now, let's put it into
8:42
production because I think everyone can relate to that and
8:45
that's probably half of the people in the ml Ops Community
8:49
are there because they've had that very same experience, and
8:53
so it's good to hear that. You had that in.
8:56
You were a little bit discombobulated with.
8:58
Well, what do I do now? And oh, maybe I shouldn't have
9:01
optimized for those things that I optimize for.
9:03
Maybe I was like making things a little over engineered here.
9:07
So it's really nice that you were able to to see that and
9:12
learn from it. And then the security part where
9:17
you realize that there's only or the majority of what you're
9:21
reading out, there was research and it applied to research, it
9:24
didn't really apply to your specific use case in situation,
9:29
where you needed to have tangible things, you could do to
9:32
make your model more secure. So that is just, it's and it's a
9:37
great way. You're solving this pain Point
9:40
by learning. Going through the different ways
9:44
of how you can make this model secure on your own.
9:48
And so and then you became quote, unquote expert on it.
9:52
So I love that journey and maybe we can talk now about some of
9:59
these reasons why we should even care about keeping our models
10:05
secure. I imagine that a lot of people
10:08
out there, especially data scientists.
10:10
Are thinking in the way that you used to think and they're just
10:14
trying to create the best model with the best accuracy score.
10:18
And not really thinking about production yet and everything
10:23
that comes with production. And so maybe there are a few
10:25
things, I know in the article, you talked about like, what are,
10:30
what are things we should care about, or why should we care
10:33
about these different things? Yeah, so when we are talking
10:38
about machine learning security, first, we have to be aware that
10:44
machine learning systems are actually software pieces, so
10:50
they're going to be. So in terms of security, you're
10:56
going to have the software security cybersecurity part, but
11:00
in addition to that, you're going to have some specific.
11:05
Security threats for your machine learning model.
11:09
So there is some part that is already solved or not.
11:15
That's hard. But you have a lot of resources
11:18
or already that that is available out there about this
11:23
service software part, but not that much about this, this
11:27
specific machine learning machine, learning part.
11:30
So when we're talking about machine learning models, The
11:37
kind of threats that we're going to have it's it's it could be
11:42
about data or the model. So the model actually is
11:48
something that you build out of data.
11:51
So as soon as your model is subject to attacks, it's
11:56
actually the data that is used which is a threat as well.
12:01
So the kind of threats that we are going to have is Data
12:06
extraction. So the ability actually to
12:10
extract some specific data from your model.
12:15
And we know that this is this could be a really problematic in
12:21
terms of privacy first and foremost.
12:26
So about about privacy, it's could be another type of problem
12:32
affecting the model by actually Corrupting it, making it
12:40
generate false predictions. So this could be, this could be
12:45
used by by harmful people, for example, to avoid some filters.
12:52
For example, let's say, let's say on social media, there is a
12:55
fake news detector. So by by manipulating your
13:01
input, in some way, we're going to avoid about that that
13:04
feature. And there is also the problem of
13:09
the model stealing actually just by providing an API so that
13:18
somebody can query your model and get a response by doing
13:23
that. A lot of times, we are able
13:25
actually to make a copy of this model.
13:29
And this this is a problem, for example, if actually your
13:34
service is a paid, Service because some people can copy
13:39
your model and make it make it free or use it use it without
13:44
without 2 or it could be a competitor that that could steal
13:48
your model as well. So there are different kind of
13:54
traits related to models and data.
13:56
And each each each kind of threat is, there is going to
14:00
have some, some sensitivity. And I think that Of course, not
14:07
all the projects that were we work on every day.
14:12
Our may be sensitive sensitive models, but it's, it's something
14:19
that we should do for each project.
14:22
It's to think about it and maybe assess the risks of this is
14:28
Mommy's. Yeah.
14:29
That makes a ton of sense. Exactly.
14:31
And so this framework that you've provided for security,
14:35
And the different threats that exist.
14:38
It's something that sounds very similar to what we discussed
14:41
with Diego open Hammer the CEO of algorithm Mia who joined us
14:45
to talk a lot about what I'm Elsa cops looks like with their
14:50
platform and how they've been thinking about this challenge.
14:55
How do you as a data scientist now now that you know, some of
14:59
these challenges, and some of these threats that are posed to
15:04
machine Learning System, Eames how have you changed your
15:07
development practices to mitigate those threats during
15:11
the course of model development and production ization?
15:14
Yeah. So when we talk about security
15:19
so we talked about assessing Inca, assessing the risk of our
15:24
models. So I think that each one in the
15:27
organization can have his drone in avoiding this these problems.
15:33
So me as a The scientist, I have to be aware when I build my
15:38
model, if there are security threats associated with it.
15:43
So if I'm going to fetch data from unreliable sources, this is
15:48
a security threat. If I'm going to fetch a model
15:53
from Models, ooh, there are different models that were where
15:58
people can can go to to use some free training model.
16:03
I have to make sure that it's a reliable Versus source as well.
16:07
Because if this model is corrected, my model is going to
16:10
be corrected as well. So there is during how during
16:17
the modeling in information of how I am going to to use the
16:24
data and to expose my model as a data scientist, I should.
16:28
I should know where are the threats.
16:32
And then you will going to work with a machine, a machine
16:35
learning engineer with, with, with your manager.
16:40
And at each stage, there could be some some responsibility.
16:46
So, here are the, there is a another unanswered question,
16:51
it's the accountability who is accountable for that.
16:54
But, for example, as a, as a product owner, should be able to
17:01
To evaluate the different risks associated with the with this
17:06
model. So working with data scientists,
17:09
we know where the risks risks and we can associate may be some
17:15
sort of some importance to these risks, and then, as a manager
17:20
and decision-maker, I could decide if I put the this model
17:24
in production as these, or if I do some tweaks, tweet that often
17:29
come actually in, The cost of accuracy or or other things that
17:35
I have to like they have to drink.
17:38
Got it, got it. So it sounds sounds very logical
17:44
which is, you know, as a as a professional it's incumbent upon
17:47
you to really understand some of the day-to-day threats that may
17:51
emerge. Everybody has to play a role in
17:53
that and then to also work as part of a larger team to really
17:56
take security in as a, what's the term?
18:00
I'm forgetting the term but almost as like a first
18:03
consideration to make that as part of your development
18:07
practices. And that makes a lot of sense.
18:09
One of the things I really enjoyed about your article that
18:11
I would love for us to cover are some of the ways that we can
18:15
actually secure models on a, on a more almost tactical level.
18:20
You mentioned things like differential, privacy and
18:23
homomorphic encryption. Could you talk to us a little
18:25
bit about what some of those, you know?
18:27
Obviously, you know, I would love to you know I'd love to as
18:31
a data scientist. Be completely responsible with
18:33
all of my security practices and you know, as you said models ooh
18:37
We're looking at other external sources and take that into
18:40
account. But assuming I've done all of
18:42
that. What are the other tools in the
18:44
toolbox that, you know, you covered in the article?
18:47
And could you give us an introduction to those?
18:50
Yes. So long.
18:52
Let's go for each one of the threats that you mentioned.
18:56
We have some defenses that exists for our produce.
19:01
This kind of threads. So yeah, when we, when we Talk,
19:08
for example, about the data extraction, the problem, yet the
19:13
fact that for large models we were able to extract data.
19:18
This could affect. Yeah.
19:19
Very, very large language models.
19:23
For example, this kind of this kind of models are very subject
19:28
to this kind of trades. For example, you take there is
19:33
the key algorithms used, for example, in Gmail birth.
19:38
Composer. Some researchers have shown that
19:42
by prompting the model, the model with the right words.
19:48
We're able actually to extract very specific data, personal
19:54
data. So they were able to extract
19:57
name phone, number email address physical address of a real
20:03
person from this kind of hops. So luckily the Press Yeah.
20:08
Luckily for us these Engineers they also worked at Google.
20:13
So Google is doing the the right things in order to secure this,
20:19
this this kind of models but the risk the risk is there.
20:23
And in order to avoid this problem in enlarged, in large
20:29
models, we have to understand what what what the problem why
20:35
do we have? Who are we able to This this
20:39
data. It's because actually the model
20:41
is so complex that it's able to memorize some specific details
20:48
in the data. So so what we can do in order to
20:55
to have a defense against that is one technique that is called
20:59
the differential privacy. Is to constrain put the
21:03
concentrate on the model in some way in order to actually to
21:09
control how much it's going to memorize data.
21:14
So what then for us, we're going to have some knob called Epsilon
21:20
and depending on where I put the knob and able to decide whether
21:26
I care about privacy first or accuracy first.
21:31
So it's going to be a choice in order to choose where I'm going
21:36
to put this business. So this is differential privacy
21:41
but there are books. Ari before we jump to the next
21:45
one. Do you have any other stories or
21:48
occurrences like this? Google one where data was leaked
21:52
in that fashion? Do you have any other ones?
21:55
An example of his much resale? More recently Microsoft to
22:03
recently released co-pilot for the table.
22:07
So this this so it's enable you as a as a default developer.
22:13
To have Smart, suggest some suggestion when you're coding.
22:19
So first thing, some people did is just write API key equals and
22:27
see what the model output and there you're going to have of
22:32
course some some some some serious privacy and security
22:36
problem because yeah it can output some some real Keys,
22:41
sometimes 8 outputs Also some headers of websites with, with
22:46
names inside it, but the thing is, copilot is trained on public
22:52
data. It's all open source, but you
22:57
have, you also have some some personal details and sensitive
23:01
data in public public date. So yeah, so it's a yeah year the
23:13
VP That you must exist. Also in the energy t to the
23:18
model developed by open air. So in this kind of kind of
23:23
models you're going to have this this kind of, right?
23:27
Yeah, but there are other defenses.
23:31
The so for each type of model you're going to have some some
23:35
defense, one of the most known threats is adversarial examples.
23:42
The fact by Tweaking, some, some inputs, for example, it could be
23:47
images or, or, or texts you you going to make the v-model
23:54
output, another, another prediction.
23:58
So we talked about the digital spam.
24:04
It could be, it could be in spam filters or in fake news
24:07
detectors in social media, use also have some harmful content
24:14
Filters. And if, if your model is not
24:18
secured actually it's it's possible to to design some some
24:25
inputs that it's going to get through this these filters.
24:30
And the problem here is that the only defense is actually two to
24:37
do it before the attacker to know, to know which kind of
24:41
transformation is going. To do and actually train your
24:45
model with the data of motivation in ordered that your
24:49
model is so that your model is more robust to these kind of
24:54
attacks. So I guess those these things
25:02
I'm just wondering as you're talking about, this are these
25:06
because you're in France and you probably have heard and looked
25:11
at the new proposed regulation from the EU and how they talk
25:18
about this robustness. Do you feel like this Kind of
25:21
strategy will help those who now have to all of a sudden take
25:26
this new regulation into account.
25:30
Yes, exactly. So the new regulation, actually,
25:36
what's good about it is that it puts a framework and it defines
25:40
console what's a sensitive AI application, because that's what
25:46
we're talking about. It's it's really this kind of
25:50
systems that that should be robust and that we should be
25:55
sure that they're not, they're not The value-based such attacks
26:00
and you were regulation defines Defiance sensitive application
26:07
as something that is harmful psychologically, or physically
26:15
to people. So yeah, assume that we Define
26:20
it as such for example, if if some people are able to put some
26:29
harmful content on social media. Yeah, this is, this is this,
26:34
this is harmful and then it's it shouldn't be shouldn't happen or
26:39
you're going to have some fines Etc when we're talking about
26:44
autonomous vehicle. Yeah.
26:47
Could be physically harmful so it's a sensitive system as well.
26:52
So it's yeah. It's I think it's a, it allows
26:58
us to, to better to Define this kind of sensitive applications
27:03
as well. So there's something else that I
27:06
wanted to talk about that. You mention And it is the
27:12
dangers of Federated learning. And I know that Federated
27:17
learning is quite popular these days in the community, so maybe
27:21
we can go down that route and why that's different than just a
27:25
normal machine learning. Yep.
27:29
Yes. So, Federated learning is, yeah,
27:35
for just a very quick introduction is about.
27:40
Let's say this. You get to learn so it's the
27:43
ability to use the data that is going to be in different
27:50
locations and use it in order to train one model based on all
27:57
these states without aggregating all the data on in the in the
28:02
same location. So it's used for example in
28:05
smartphone we take time to read or for iPhones.
28:11
There is a very some Federated learning that is used.
28:15
So, in theory, if I'm a hacker and I have a smartphone,
28:21
actually, I'm able to put anything I want in this device,
28:27
I have total control over the data here on this, on this
28:32
device. And this data is going to be
28:35
used in order to learn a model. So this is, this is actually a
28:40
threat. Federated model because a hyper
28:43
able to inject any data he wants for the final for the final mode
28:51
and it's even more sensitive because Federated learning is
28:55
also used a lot in the medical areas between the between
29:02
hospitals because we have patients data in different
29:07
hospitals and we want to use all of The data for in all the
29:15
hospitals around the world in order to have the best the best
29:20
AI in order to detect some some disease, but we don't want to
29:26
put all the data in the same place and it's it's difficult to
29:31
share data between between hospitals.
29:33
The Federated learning is is a solution.
29:37
But for medical data, we have to be really Hyper secure when we
29:44
are aware manipulating these data.
29:46
But luckily I think that the brightest people right now are
29:50
working on this subject. And that's why we have some
29:54
really interesting techniques the defenses that were designed
29:59
for Federated learning. We can we can talk about how
30:04
morphic encryption. It's, how actually you ensure
30:10
that Encrypt your data in some way that you ensure that via the
30:18
direct with the the computation that you do on encrypt the data,
30:23
you going to to detect the results from it and and and
30:30
unencrypted in order to get the final result.
30:32
So it's doing the computation encrypted data.
30:36
There is also other methods such as security Thirty computation.
30:42
So there are different different techniques designed mostly for
30:47
Federated learning, but we have to also to be aware that.
30:50
It's still a research area and it's improving every day.
30:54
So it's it's we don't have yet the perfect framework that you
31:02
have different framework with with, with some advantages and
31:08
some problems this week. Yeah, I know that's that's a
31:12
great answer. And really highlights, I think
31:14
in Federated learning there's a lot of excitement about
31:17
Federated learning and a lot of different Industries because I
31:19
think, you know, the challenge is, is that a lot of people see
31:23
data as a strategic asset and getting unlocking, the value of
31:28
data has been a challenge because of that and Federated
31:32
learning has been posited as a solution to that.
31:35
But what you're highlighting is it's not necessarily A Panacea
31:40
when you Sitter the security challenges that are involved.
31:44
That's I think that that's a great Point.
31:46
As I think I'm word there Panacea.
31:50
I've been studying word of the day, huh?
31:52
There we go. Yeah.
31:54
It is so natural. Demetrius just blew up my train
32:02
of thought trying to reconstitute it and I think
32:06
listening to your descriptions of some of the challenges that
32:10
are inherent it to model development and security
32:12
challenge inherent the mall development.
32:14
It seems that, you know, there's almost this process of model
32:18
interrogation. Right?
32:19
Which is, you know, in some ways, similar to what we call
32:23
interpretability, but it's a little bit more directed in the
32:26
sense that, you know, for example, what these Google
32:30
researchers did to unpack, the smart composed models was a very
32:35
focused exercise. That wasn't just dependent on
32:38
one analysis, like interpretability You know,
32:41
interpretability I think people are using shop scores or lime
32:44
and very, you know, specific techniques and interrogation.
32:48
There's almost, you know, an entire process and to me, I
32:53
think, when I hear you go through it it seems like there
32:56
are 33 almost three legs to the stool of model interrogation,
33:01
right? You have your operational
33:02
component where there are different teams security and all
33:05
engineering data science that can help us that there's a
33:08
regulatory piece, right? Which is the Euler's and cells
33:11
or saying have you answered these questions in the third
33:14
stool or third leg to the stool? That I feel like I haven't had a
33:17
good grasp on is the tooling around this, right?
33:21
Which is what are the tools that allow us to interrogate models?
33:24
And to answer some of the security practice challenges
33:28
that we face that we just talked about?
33:31
Where do you think the current state status of tooling is and
33:35
are there any tools that you found useful in terms of
33:38
ensuring model security? Yes, And what's exciting about
33:42
this is that if you have asked this question when I wrote the
33:46
article, I think there were none but a few months later, there
33:51
are some some tools that have emerged and actually, we're
33:57
starting to see something really interesting in some, some of the
34:04
biggest companies, where you have this, this kind of
34:09
sensitive systems. And these companies are now
34:12
building some teams that they call a, I read teams.
34:17
So, actually these things it's coming from the, the security
34:23
area or gaming as well. So, we have the red team and The
34:26
Blue Team blue team is focusing on defensive strategies and
34:32
ratings are focused on trying to attack their own systems in
34:38
order to be aware of the risks and too, To defend better at the
34:42
end. And so you have some are red
34:47
teams in some big companies like Microsoft for example and
34:51
Facebook. And these teams just started to
34:56
develop some tools that they are open sourcing right now.
35:00
So one for example we have the Facebook team that development
35:08
ugly. So this is actually a tool.
35:10
Tool that allows to to be more robust to to adversary attacks.
35:18
So it's a tool where they try to to do try a lot of augmentations
35:25
on on the input data in order to make the model more, more robust
35:31
to this kind of attacks you have Microsoft to develop the tool
35:37
Cara called counterfeit which It's interesting because the
35:42
idea is to evaluate the threats of of your model.
35:47
So it's based on this Matrix of states that that we just
35:54
discussed and it tries to look at these different threads in
35:58
order to evaluate the risk of of the model.
36:02
So this kind of tools we didn't have sour for some months ago
36:07
but it's the tools that that that we need in order to to to
36:11
make to make our model a more secure.
36:14
And if you look at it it's actually it's like the same
36:18
story again because we had the data scientists and then with
36:23
more tools we switch to a machine learning Engineering in
36:29
order to be a more production focused and for security.
36:34
So it's the same thing that's happening because we say it's
36:37
it's a research area. But as soon as We have tools
36:42
that are developed, we're going to have maybe a more profiles
36:47
who are able to to take pieces, to take this and to work on the
36:52
the security of this kind of models.
36:57
That's awesome. It's really, really promising to
36:59
hear that. There are so many of these tools
37:01
that are that are coming out. Now, especially I think a lot of
37:06
the ways that, you know, if you look at the history of sort of
37:09
ml Ops to date so far, it's been, you know, you had ml
37:14
happening and do uniquely talented Enterprises, like
37:18
Google Uber Pinterest cetera, then people who built those
37:22
systems came out and started an ellipse to a tooling companies.
37:27
Right. And then that has enabled it for
37:30
all other companies to be able to adopt same.
37:32
Best practices that other companies develop more easily
37:35
and it sounds like what you're saying is a similar sort of
37:38
thing is occurring and I don't security where the leading
37:40
Enterprises are developing some new best practices those
37:43
professionals. Hopefully we'll come out, start
37:45
companies or create knowledge that others.
37:48
Then we'll be able to access to be able to implement ml
37:52
security. Best practices, does that sound
37:54
that sound like a fair summary, Yeah, it's sad.
37:58
So I think it's exactly that. I'm just, that's how how science
38:02
work. And that's why we have, we have
38:05
a research and I think that this domain is really exciting
38:11
because of the end is the fusion of to drink domains.
38:15
She's Ai and cybersecurity. And yeah, when you, when you
38:21
look at the research paper and problems, it's yeah, sometimes.
38:27
You really have to dig into the inner functioning of the modern,
38:31
to the mathematical implication in order to design some.
38:37
Some tools like that? Yeah, it's very interesting to
38:43
see. Yeah.
38:43
I think you highlighted it because you had it.
38:46
Exactly right. Which is it's not just that
38:48
there's this research happening in the Cyber in ml world, you
38:52
know, security for ML that's cyber security.
38:55
Itself is vast. Field.
38:57
And it feels like there's so much, you know, with the
39:00
advances in cryptography and what that advances, you know,
39:03
just in terms of thinking about the threat points for different
39:08
cyber security systems. And in fact, using machine
39:11
learning for cybersecurity, it's it's crazy to think about all
39:14
the different, you know, Innovative that you had.
39:16
Sir. Yeah.
39:17
It's all these different intersections that are
39:19
happening. Yeah, that's true.
39:22
So yeah, we been hearing about machine learning.
39:27
For cybersecurity. But yeah, the cybersecurity of
39:33
machine learning is actually completely completely different
39:36
and it's a subject on its own as well.
39:41
So I wanted to just mention something that was posted in the
39:47
security and privacy channel on Slack.
39:52
From Diego actually, who we also spoke to about this a few weeks
39:57
ago. And when we talked to him, he
40:00
mentioned how vulnerable or how, how bad of a practice it is to
40:05
just go and download anything off, pie, pie, and bring it on
40:09
your jupyter notebook. And then just, you know, import
40:11
whatever from this on pie, pie and and how this article that he
40:18
shared talks about how python developers are being targeted.
40:22
With malicious packages on Pi Pi.
40:25
And so that's just something that I since we are talking
40:28
about this and the greater ecosystem and ways that you can
40:33
potentially be vulnerable. I wanted to mention that anyone
40:38
who wants to dig into it more, we can link to that article in
40:41
the description on and or you can go into the security and
40:45
privacy Channel. But so my next question for you
40:49
shabby is more along the lines of Synthetic data.
40:53
And I've heard a bit about synthetic data and I'm wondering
40:57
where you stand on it. Maybe you can just give us your
41:00
two cents and how you feel. This can be helpful or is it not
41:07
really that important? What would it be used for?
41:11
Is it a tool that we can use as data scientists to help us
41:15
create more robust systems? Yeah, so, okay to data so far.
41:23
Mainly, what you want to generate using generative
41:28
generative models. So what could be images?
41:33
For example, we see a lot of use cases or to, to make to make
41:39
some art like making some some paintings that resemble, some
41:47
some painter. It's it's also used in music.
41:50
Now, we have some Generation of music.
41:54
But when we talk about the security it's permit not
42:00
directly but synthetic data but data aggregation.
42:05
The difference is that what you're going to take is actually
42:10
the your your input and try to do a lot of transformation on
42:16
this inputs. So we're going to stand
42:19
synthesize example that your pigmenting some existing Data in
42:23
order to make it more more robust set to 22 attacks.
42:30
So by flipping it, if it's images rotating it for a fourth
42:36
text, we're going to introduce some, for example, characters,
42:43
some different two, different words that could be used, so,
42:47
it's more data argumentation, but yeah, I'm trying to think.
42:52
If data synthesis could be used as well, but right now, I don't
42:57
have examples of it. Yeah.
43:02
So that makes sense on the, on the synthetic data standpoint.
43:07
I think just to kind of bring this whole conversation full
43:10
circle, you know, we've talked about so many different new
43:18
privacy methods. So to speak or new sort of
43:20
security defense tactics. So You potential threats for
43:26
data scientists to take into account as they develop models,
43:30
you know, as you sit here in 2021, how do you think what is
43:35
your prescription on? Where do you think?
43:37
You know this whole question of ml Security will be in 5-10
43:44
years. I'm kind of curious how you see
43:45
things look like going forward. Will we have, you know, almost
43:51
automated security Security practices the same way that we
43:54
have another sort of like infrastructure as code or do you
43:57
think that, you know, there's a different way that this will
44:00
develop? Yeah, yeah.
44:04
I think that's actually what you when you look at the pink.
44:09
Big companies that actually reflects where most of the
44:14
companies are going when they're going to mature.
44:18
Today by creating this kind of Red team's.
44:23
The idea actually is even broader than just the the model
44:29
security because there are so much thing at stake.
44:33
You have the the Privacy the security but also the buyers of
44:40
the algorithms. So all of these things you have
44:44
to ensure that that the two models are always secure and
44:50
robust to this to this guy. And of this kind of threats,
44:55
especially with, with Emma, lots and automating how our models
44:59
are put into production tomorrow.
45:04
I have a, I have a robust algorithm that is in place but
45:09
that is scrapping. Some, some kind of comments from
45:12
a website one day or another a competitor or a malicious
45:17
person. That person is going to inject
45:19
some some data out there and they have to be aware That if
45:23
this happens, what's the implication for for, for my
45:27
company. So I think that's we're going to
45:31
see more and more of these teams in other companies.
45:36
So some teams who are really focused on this machine,
45:41
learning security, security parts for the moment, this I
45:48
think it's quite some scientific skills that they are needed most
45:53
because it's still research area.
45:55
But as we have seen with data science with more and more tools
45:59
that you're going to see, I hope that you're going to be
46:04
development. We would we would have maybe
46:08
some more software software oriented personas and skills
46:14
that could also take on this. This this kind of this kind of
46:19
subjects. So I love that you're Brothers.
46:23
Because when you look at different companies that are
46:27
heavily relying upon machine learning for their bottom line,
46:32
and you need to know the implications of what it means to
46:39
have data poisoning or these different threats that are
46:42
coming at you. You need to have that very clear
46:45
because that could be a heavy, heavy hit to your company, and
46:51
the bottom line, like we're Talking about the last thing I
46:54
want to ask you which is a bit of a tangent, but I would love
46:59
to go down this little rabbit hole with you as you were
47:03
writing this. And as you were trying to figure
47:06
out, what are some ways that my models are vulnerable, what are
47:12
some threats to them? How can I fix these threats?
47:16
Where did you encounter the biggest pain points?
47:20
What were some parts where? Maybe it was in implementing
47:24
some of these security measures, you realized it was very
47:26
difficult or maybe it was just in, finding the information on
47:32
how the model is a threat. And then having to try and lock
47:37
down your model in that way. What were the pain points that
47:41
you encountered? Yes.
47:43
Oh I think is when I wrote the article it's we can't allow the
47:49
content about about security was not that available.
47:53
They think that this is we're seeing more and more content out
47:58
there. Luckily for for us.
48:03
So I think one of the biggest problems is maybe the level of
48:12
science that you need in order to Understand in order to
48:17
understand the attacks and the defense's.
48:21
Because it's really linked to how the, how the model and the
48:26
Learned is done. So it's I think, yeah.
48:32
For now you want to have this, this kind of level of
48:35
understanding of your of your models in order to understand
48:39
how these work. So maybe this this could be as
48:45
well as an obstacle for hackers because in order to design some
48:52
some some new methods maybe there are easier ways to attack
48:57
some machine Learning System than to design the whole new way
49:01
to to corrupt, a great indecent. So it's a it's a good news from
49:11
from one point of view. But what what I also liked when
49:17
I started to make researches with with especially this new
49:22
tools that are coming, you have now some some there is a tool
49:28
that that was released by indeed, I think which is
49:33
nonviolent that would allow anyone to actually try to hack
49:40
machine learning the machine learning model, so it's an
49:43
environment made for us. Attacks in order to to have some
49:48
hands-on experience and try to understand how much working.
49:56
So now this is I think a good area and it's good that we start
50:00
to see this kind of this kind of tools.
50:04
There is some problem, another things that that is no
50:08
organized. I think by Microsoft, it's
50:12
actually a challenge. It's so it's like a good
50:16
challenge for data science. But here, it's not about making
50:20
the best performance. It's about attacking an existing
50:25
model trying to divert it and to make some first area attacks
50:33
against this model and also in defense trying to to to best
50:38
defend, some model and making it too, robust to progress to these
50:42
attacks. So I hope the You're going to
50:45
see more and more initiative like this, because it's, it's
50:49
with such innocent initiatives that we're going to have more
50:53
and more people aware and knowledgeable about how this
50:57
works. Yeah, and that's one of the
51:00
reasons why I love talking to people about security at this
51:03
point in time because it feels like it is one of those pieces
51:07
of machine learning. That is a little bit, put off to
51:11
the side or like you talked about at the beginning.
51:14
Who owns this problem? Who is the one that is
51:18
responsible for the security piece.
51:20
And I really liked what you said at the beginning, on each person
51:23
has to do their job, whatever that may be.
51:27
And you really need to look at trying to make your piece of the
51:32
puzzle as secure as possible. And then once it's handed off
51:36
the next person is also trying to make their peace as secure as
51:40
possible so that you don't have any problems in the future and
51:43
you don't have these acts. So with that being said, this
51:48
has been an excellent conversation.
51:51
I really, really appreciate talking with you Savvy.
51:53
It has been a long time coming. I know we wanted to do this
51:57
months ago and we finally got it done.
52:00
Hopefully everyone enjoyed it. If you are out there still
52:03
listening the way that you can show your support is by giving
52:07
us a like, or commenting or subscribing or whatever.
52:11
The cool kids do these days for the podcast.
52:14
And YouTube videos. That's all we got for today.
52:17
Thank you, everyone. Thank you shabby.
52:19
Thank you Vishnu and the word of the day is, what was it?
52:23
Pan Sia, Panacea, and can I see it?
52:27
There we go. Hopefully, you all learn
52:29
something.