Latest / Agentic Conversations (formally mlops.community) / Detecting Harmful Content at Scale // Matar Haller // #246

Transcript

Detecting Harmful Content at Scale // Matar Haller // #246 · Agentic Conversations (formally mlops.community)

0:00
My name is Mattar Haller Matara is with guitar.
0:02
I'm the VP DNA I here at active Fence and although I don't I
0:07
have three kids, I don't drink caffeine.
0:08
I drink decaf black tea with milk.
0:13
What is up MLA's community? You are back for another podcast
0:16
with your host AKA me Dimitrios. I'm gonna be real with you
0:22
before you go any further. If you are one of those people
0:24
that does not like hearing about the dark shit in the world, do
0:27
not listen to this episode. If you get triggered easily, we
0:30
go into some depths and the Thrones of the Internet, that
0:38
horribleness out there and how AI plays its part in helping to
0:44
make sure that there is not as much horribleness on the
0:48
Internet. But our walked us through these
0:51
very high stakes AI use cases on keeping hate speech and all
0:56
kinds of nasty stuff off the Internet and how her company,
1:02
Active Fence is playing its part.
1:05
I'm going to leave it at that. I'll let you decide if you want
1:08
to go forward or not. Let's get into it.
1:12
If you enjoy this episode, you know what to do.
1:15
Share it with a friend who has the ability to listen to this
1:21
type of stuff or give us some feedback.
1:25
Spotify's got a cool thing where you can give us a few stars.
1:29
We'd love that. Alright, let's get into it.
1:39
I'm going to start it with this. We met because you were
1:43
presenting at the Tectonically conference, the virtual
1:45
conference, and there was a tonne of amazing talks.
1:49
But by far the one talk that stood out to me just because of
1:53
the novelty and the good that you are doing was your talk.
1:58
And so I want to explain, I thought, well, first I have to
2:03
have you on here and I want to explain a bit of what I remember
2:06
of that talk. And then you can maybe fill in
2:07
the blanks of what it is that you're doing.
2:10
And so you are using AI to make sure to, I think it's you, you
2:17
called it AI safety tech. And so you are making sure that
2:22
when there is hate speech, when there is stuff that we just
2:25
don't want online, you can detect that and make sure that
2:31
it is cleaned up. So that's like first part.
2:36
Can we start there? I know I just gave a
2:38
gigantically broad overview of it, but now I'll let you fill in
2:41
the details. So I don't know what talk you're
2:42
talking about. My talk was about dinosaurs.
2:44
You're like. I was way off.
2:47
Wait a minute. So yeah, that's right.
2:51
So my talk was about using AI for trust and safety and how an
2:59
active fence we are an AI trust and safety company and what that
3:05
means and how it's done and why it's a super hard problem, but
3:10
one that is also incredibly impactful and touches us, all of
3:15
us every day. I think one thing that you did
3:19
go into pretty in detail in that talk, and we'll leave the link
3:23
to that talk in the description for anybody that just wants to
3:26
go directly to that before they come and listen to this.
3:29
But you mentioned why it is so hard.
3:33
And you kind of went over all these different use cases on
3:36
whoa, wouldn't you think that this should be OK or not OK?
3:41
And there's a lot of ethics involved.
3:43
There's a lot of times where I was saying to myself, like,
3:46
yeah, I think that's OK. And then it's like, ohh, you do,
3:50
well, maybe you want to think twice about that And so.
3:55
Yeah, it's so first of all, I think it's like, besides the
3:58
fact that it's hard, it's also really, really important, right?
4:01
So like, who cares? Like, why even trust in safety?
4:04
Why is this even an issue? And I think trust and safety
4:08
isn't a new concept, right? Like, if you think about what
4:12
happens online, so online harm can lead to offline harm, right?
4:15
I think like for me, the most like visceral and jarring
4:20
example of that was what happened to January 6th with
4:23
like all of this, like, you know, this misinformation we
4:26
were tracking prior to elections and that and that we were
4:29
notifying platforms about to take down and what that led to
4:32
with the capital rights and so forth.
4:34
So there's many, many examples that we think about where online
4:37
online harm can sort of leak offline.
4:39
And so much of our lives are online.
4:41
And also, I mean, as a mom, it's something that I like care about
4:45
deeply. I told my daughter like she's
4:48
she's 9 and all she's asking about all the times the cell
4:50
phone and I'm like, maybe when you're 35.
4:54
But at the end of the day, like the this is something that's
4:57
like it touches all of us, right?
4:58
Whether it's comments on, on posts or chats or videos that we
5:04
see or images that we share. It's like our lives are online.
5:08
And although trust and safety isn't new, it's now, I think
5:11
after sort of these sort of big events that we've seen, like I
5:14
said, January 6th and also some really well known trusted safety
5:18
fiascos that we, that if you think back, they'll, they'll,
5:22
they'll come to mind. It's now it's finally something
5:24
like really, really being discussed, right?
5:26
Like we'll see in the media and even like regulators and
5:28
legislation and the Digital Safety Act and, and even users
5:33
now care about this, right? It used to be that like, it
5:35
wasn't even something that us as users would think about when we
5:39
went online for a platform. But now like we're exposed in a
5:43
lot of ways to a lot more hate speech or a lot more bullying
5:47
or, you know, just really vile content.
5:50
And it could be just because oftentimes content moderation
5:53
just isn't good enough and also because it's an extremely,
5:57
extremely adversarial space. And so we can talk about why
6:02
it's important to companies and to users, You know, it gives a
6:06
competitive advantage. You don't you don't want to be
6:08
exposed to that. But like you said, it's also
6:12
really, really hard. It's, well, can you break down a
6:16
few of the use cases because I know we're, we're talking a
6:18
little bit high level, but exactly what is it that some of
6:22
the things that you're doing so we can get a better sense of why
6:27
this can be so hard? Yeah.
6:29
So actions, we are in the content moderation space.
6:33
We don't do the moderation right.
6:35
We we flag it for the platforms. We give every item we have an
6:38
API, customers can platforms can send their content to us and
6:43
then it gets a probability of risk.
6:44
And what do we mean when we say content?
6:46
This content can be audio, video, images, text.
6:51
The text can be comments, they can be chats, they can be
6:53
usernames, they can be descriptions, they can be
6:55
titles. It can be podcasts.
6:57
It can be short. The space of online stuff is
7:00
huge. And we deal with all of that.
7:02
And not only do we deal with that, but also the text can be
7:05
in many, many different languages.
7:07
And there's many, many, many different cultural contexts,
7:10
right? Calling someone a dog in one
7:13
language versus another language has very, very different things.
7:17
And if we're gonna, if we're gonna be trying to keep users
7:20
safe, we need to be aware of that and how slang changes and
7:23
so forth. So it's difficult because it's
7:27
a, it requires sort of subject matter expertise, right?
7:30
You need to know the language, you know, the slang, you need to
7:33
know in images, you need to know, is this logo of a hate
7:36
speech group? Is this logo benign?
7:39
Is this just like, you know, it needs it's, you know, it's
7:43
constantly changing. So unfortunately there's new
7:47
child abuse content every day. There's new hate speech groups
7:49
every day. And, and also the bad actors,
7:53
like they're, they're not stupid, right?
7:55
They're bad, but they're not. And so they're trying to evade
7:57
detection. And so they'll use phrases,
8:00
they'll use lead speech, they'll try to obscure stuff.
8:05
So that's, I mean, it's a very outfit.
8:08
I brought it down a bit from the high level.
8:09
It's like very concrete in terms of like the media types and the
8:12
challenges. Yeah, exactly.
8:14
And so you're getting all kinds of input data that is basically
8:21
coming through the API and you're saying, yes, this looks
8:25
good. Or ooh, maybe like, do you give
8:28
a scale of how potentially harmful this could be and then
8:32
you send it back to the people that sent it to you?
8:35
Yeah. So what we have is basically we
8:37
have something that we call a policy management.
8:40
And so it's sort of this UI where clients can come in and
8:42
they could basically decide what for them is a violation of
8:47
policy. So when a child plot and the
8:49
platform for children, I don't want any nudity, I don't want
8:52
any swearing, I don't want any bullying.
8:55
And so I can go in here, I could say, OK, here's a model for
8:57
bullying, here's a model for swearing, here's a model for
9:00
nudity. This is what my policy violation
9:01
looks like. And then I send in my content
9:03
very API. And then we basically fan it
9:05
out. We have our models that run and
9:08
then we can provide a risk score.
9:09
Like what is the probability that this item is risky?
9:13
It's not severity, it's probability of risk.
9:15
And then basically clients can come in and they have a UI and
9:19
they have moderators and they can say, ohh, like I any item
9:22
that's like, you know, more than 80% probability of risk, I want
9:25
to automatically remove it. So they have these codeless
9:27
workflows. They can just set it U any item
9:29
that's like less than 50, I don't even want to look at it.
9:31
And then there's this sort of sliver of window where I say,
9:33
OK, these are things that require an expert to look at.
9:37
And the idea, the idea being is that we care about not only the
9:41
users well-being, but also the moderators well-being, right?
9:44
And we don't want to expose them unnecessarily to harms, right?
9:49
So we can catch the things that are highly probable being risky
9:52
and just remove them. Then we've spared them from
9:55
like, this gory, awful stuff. Yeah, that's, I actually heard
10:00
about that back in the day on how people that were doing
10:03
content moderation for Facebook. It was like menial things 80 or
10:08
90% of the time and then they would be clicking through and
10:12
like whatever, boring, boring, boring, and then bam, just
10:16
getting something that would rock your socks off and give you
10:20
nightmares for the next two weeks.
10:22
And so it feels like if you can remove that completely, that's
10:26
very valuable. And now going back to the idea
10:30
of the models that you have, you're allowing someone to
10:35
choose which models. And I'm assuming when we say
10:38
models here, it's like policy models or is it machine learning
10:41
models or AI models there? A little bit of all of the
10:46
above. It depends on these cases,
10:49
right? So our system needs to be able
10:51
to handle thousands of requests per second at very, very low
10:55
latency. If you think about it, things
10:58
are coming online all the time. The only way to deal with this
11:02
massive amount of content at scale is with models that are
11:06
able to do the work faster and more efficient and with less
11:09
like mental harm than humans. And so we do what we have to
11:15
transform our models. We have ensemble models, we have
11:20
every everything under the sun we use where our big focus is on
11:26
ensuring that it's also very cost effective and very
11:28
accurate. And as much as we can, we want
11:31
to like win on both. And usually we do OK.
11:35
So the idea is, is fascinating to me when it comes to all the
11:40
different models that you're using, making sure that they are
11:43
fast and they're accurate. Because usually, especially
11:45
these days with Transformers, it's like you kind of sacrifice
11:48
one for the other, right? You've sacrificed speed at the
11:52
maybe it's not the most accurate, or you want accurate
11:55
and it's not super fast. How do you think about that
11:59
tradeoff? We don't always use like, you
12:02
know, the most accurate model that's like the slowest and
12:05
heaviest, but sometimes that's what's necessary and we need, we
12:10
also try to sort of enable our customers to be part of that
12:14
conversation. Like what's more important to
12:16
you, right? So for example, in for some use
12:18
cases, keywords are fine, like they're totally good.
12:23
And we have a keyword matching tool that's completely
12:26
self-service. You can like put in the keywords
12:28
that you want, put in the even the emojis that you want,
12:30
because in a lot of cases, like for example, in drug trafficking
12:34
and emojis are super important in child pornography also emojis
12:40
are very, very important. If you think about it, this is
12:41
like a very evasive field, right?
12:43
And so we have a tool that basically enables clients and
12:47
also our subject matter experts to sort of put in the emojis or
12:51
the keywords or whatever in the languages that are, that are
12:53
relevant and then run, run with that.
12:56
And so, and that's super fast, super cheap, super effective.
13:00
We can do different workflows. And so we could do something
13:03
like, you know, if this triggered this, then let's run
13:07
that and let's see what, you know, something heavier picks it
13:10
up. So we have different ways of,
13:12
of, of handling that. But at the end of the day, also
13:15
for some models, like a video model, then there needs to also
13:19
be an expectation of like this isn't going to be instantaneous,
13:21
right? Or an image model that's taking
13:23
an image apart and breaking it up into components and looking
13:26
for, for logos and known bad actors and victims and all these
13:30
things. It's, you know, let's set an
13:32
expectation of how long something that can reasonably
13:34
take and at what price. One thing that I do remember
13:38
from your talk was how hard the cultural references are.
13:43
And so how are you making sure not only from like the subject
13:48
experts? So I guess you have to have
13:49
somebody that understands the cultural references and can then
13:54
translate that into the models. How does that happen?
13:58
Yes, so attack defence, we have, we're very, very fortunate to
14:02
have a very large team of experts of subject matter
14:08
experts, of intelligence analysts.
14:09
And they're experts in child safety, in hate speech, in, you
14:13
know, they know about like a vegan neo-Nazi group in
14:16
Argentina. Like they're like, they know all
14:19
these sort of like edge cases. They know the newest groups, the
14:21
news. And we work really closely with
14:23
them. And so when we're developing a
14:25
new model, the first thing that we have to do is we have to say,
14:27
OK, what is the policy, right? What is in and what is out?
14:31
Because until, if you don't have a very, very clear policy, then
14:34
that's going to influence everything downstream, that's
14:36
going to influence your data labelling, it's going to
14:38
influence your valuation and it's going to influence you when
14:41
you reach production. And so we work very, very
14:43
closely with the policy manager manager and also with these
14:47
experts to define what's in and what's out.
14:50
And then, you know, we go out, we, we have the data OPS
14:53
operation here that can collect the data, whether it's from like
14:57
real traffic or from forums or even generate the data, right.
15:02
So we can in some cases just. Going to ask.
15:04
Yes. So when so one thing that we do
15:09
is we, we generate data either manually or with the help of an
15:15
LLM friend. And the reason that we do that
15:20
is to combat bias exactly from the things that you're talking
15:23
about. And but in order to identify
15:26
that we even have bias, it's not enough to sort of just look at
15:30
precision recall just now. We had a model that we were
15:32
working on. I think it was sex solicitation,
15:37
but it might have been something else.
15:38
I can't remember where we looked.
15:42
And we're like great precision recall, amazing.
15:45
And then we ran our functional test.
15:47
And functional tests is something that I highly, highly
15:51
recommend for anyone that's building a model.
15:54
And what are these? What are functional so?
15:57
A functional test. What this does is it basically
16:00
enables targeted identification of model weak spots.
16:04
What that means is that you basically take subsets of your
16:08
policy, or subsets of your target definition if you will,
16:11
and you write tests specifically for those things.
16:16
So for example, for hate speech, um, you can have reclaimed
16:22
slurs, you can have hate speech expressed using a slur, right?
16:25
These are sort of two sides of a coin.
16:28
And you want to have a test that basically ensures that it's, you
16:31
know, if I say I'm proud to be a member or I'm the freshest in
16:38
town, these should always be negative, right?
16:41
This is like a set of, you know, a bunch of examples of reclaimed
16:45
slurs that I need to make sure that my model doesn't mess it
16:48
up, right? Because I, that's like one
16:51
subset of my target definition. If I say you are just umm, to me
16:55
or I hate or dirty, right? These are slurs.
16:59
And I'm going to have a bunch of those examples.
17:01
And I want to make sure that my model is consistently performing
17:04
and flagging those as negative. And I have a huge Bank of
17:08
functional tests. And then, you know, precision
17:10
recall is important, but I want to look at how I perform on
17:13
these because then I can say, oh, great, my model is amazing,
17:15
except I'm always, I'm always failing, for example, and
17:18
harassment, bullying, they're always failing on like wishing
17:20
someone cancer, like we just couldn't get that.
17:24
And you need to catch that, right?
17:25
And so sorry. My, my threshold for like things
17:29
that like make me uncomfortable is like not normal after working
17:32
here. So like go, sorry.
17:34
Really fine. It's just I was just thinking
17:36
like, I would never want to get into an argument with you
17:39
because you probably have the best disses in the world.
17:43
You've seen all of the worst kind of did.
17:47
I come work here because of that or did I?
17:51
Ohh, I didn't think about that part.
17:52
Yeah, so sorry. I I digress.
17:55
Where were you that wasn't? Saying so, so those are the
17:57
functional tests, right? And so, and so then what?
18:00
So, and also with this helps you is like looking at over time,
18:04
right? So like you have a new model or
18:06
we're in an adversarial space, the world changes and suddenly
18:08
we're like falling apart on a specific type of functionality.
18:13
And then what we can do is, OK, so we've, we've discovered, you
18:16
know, a functionality that we're bad at.
18:18
So what do we do? So we can go out if we're like
18:21
bad on many functionalities and we can just go out and get more
18:24
data, fine. But if it's a specific
18:26
functionality, then we can generate more data to address
18:29
that particular bias that we have, right?
18:32
And so and that can be done manually or like I said with
18:36
with like Gen AI techniques. So, so many questions from this
18:41
and I love what you're doing and how much depth you're going into
18:45
your LM friend. It's not your run-of-the-mill
18:48
Luma 3I imagine because those have been handicapped, they're
18:52
not going to spit out all kinds of generated data that is going
18:55
to help you in your use case, I would imagine.
18:57
Or or are you creating your own model?
19:00
That is highly toxic we could say.
19:04
So, so, so two things. One is a lot of times it's
19:08
really, really helpful to use the LM actually to generate the
19:12
benign side of things, right? So a lot of times bias is
19:16
actually like, if your negative examples are all like slurs
19:19
about a particular ethnicity, then you want to add data that's
19:25
like shows that ethnicity in a positive light.
19:28
And so that is easy to do with any element.
19:30
Like give me some examples of, you know, of texts about
19:34
whatever, about, you know, some religion or whatever, and it'll
19:37
just give you and, and those, those are good and, and those
19:40
are reasonable. And I'll say that one advantage
19:44
of having subject matter experts working with us and having teams
19:49
that part of what we do is we do red teaming for big foundational
19:55
models is that we also know how to cause even the best of models
19:59
to behave breakdown. Nice.
20:03
Yeah, very cool. That's always fun trying to
20:05
trick them, and getting to do that as a job I find very
20:09
interesting. Yeah, the other piece.
20:12
That I wanted to go down is OK. So you've got all of this data
20:17
or you've created this model. I can see various scenarios on
20:24
how this can become not that useful very quickly.
20:29
Whether it is new terms that are coming into vogue, we could say,
20:34
or you have a lot of data that was generated or the model was
20:39
created and it's almost like this snapshot in time for up
20:43
until a certain date and then it has the cutoff.
20:45
So the new terms come in and then you can't use it and so you
20:50
have blind spots there. Are you continuously retraining
20:53
to make sure that OK? And how does that process look?
20:58
So one thing that we do is first of all, like before we deploy to
21:01
production, I think we do a lot of back testing.
21:02
We look at it on, you know, you know, productivity that we had
21:06
and we do. And we sort of look at the even
21:08
even on unlabeled data, we'll look at distribution of scores,
21:11
right? You want to see and you want to
21:12
look at things. If you have things that are
21:14
labelled, then we look at things that flipped, right?
21:16
And then we say, OK, does it make sensitive flipped?
21:18
Are we better and with our flip or worse with our flip?
21:21
And then also just look at for on unlabeled data, like what's
21:24
my distribution, right? If I have a new model, we're now
21:26
things are like much, much, much more positive.
21:28
Unlikely, right? We're much, much, much more
21:30
negative, also unlikely. So that's like before we even
21:32
head out into the world. But obviously your model hits
21:36
production and sometimes you fall apart right away and
21:40
sometimes you fall apart later. But at the end of the day, like
21:42
you're not the world is tough and are not forever.
21:45
And So what we do is we actually have an auditing process.
21:49
And so we have trained analysts that will sort of do audits of
21:54
our models in production based on real production data.
21:58
And also some clients also share with us, you know, enable us to
22:02
to sort of see their their moderator decisions.
22:04
And so we have like real labels from how our model is
22:08
performing. And then we have sort of our
22:09
policy managers and analysts look and say, you know what,
22:11
there's a trend I'm seeing here now, like somehow you're always
22:15
missing that when they're talking about like food, it's
22:19
actually not hate speech. And this actually happened to
22:22
us. Like we had a food delivery
22:24
company and we had a hate speech model because they didn't want
22:26
people. It was like a chat or review
22:28
system or whatever on their, on their website and our model went
22:33
to production and suddenly we were flagging like, I hate
22:36
Indian food or like Chinese suck or all these things.
22:38
It's like people are talking about food, like they're not
22:40
talking about about the people. And then so these sort and then
22:45
and then we're like, OK, now we need to go into like a targeted
22:48
like model improvement session, retraining and so forth.
22:51
And then and then bring it out and we this constant.
22:55
So we're sort of at the same time that we're developing new
22:57
models and sort of coming up with new sort of policy areas
23:01
cover. We're also like retraining our
23:04
models and ensuring that we're U to date based on sort of looking
23:07
at how the model is actually performing in the real world.
23:10
And again, we'll also look at the distribution of scores in
23:12
the real world, right? We don't need labels to be like,
23:14
hmm, there's drift here, like something is going on, Yeah.
23:18
And so so basically it's me choosing as a customer when I'm
23:23
hitting your API. How many different models and
23:28
protocols I want to be looking at my data from?
23:33
And so for each one of these, I'm assuming that it's not just
23:37
like one model each. Maybe you have an ensemble of
23:39
models for certain types of use cases.
23:43
Yeah. Can you break that down?
23:44
Absolutely. So one example is like I was
23:47
saying for hate speech or for terror, terror, terror, these
23:52
examples. So we know how to look for
23:54
specific components of like different objects or different
23:58
markers within the image. And then basically those go into
24:02
a train ensemble model that's able to give what is the
24:03
probability. So for example, we have logos so
24:06
you know, ISIS and, and, and so on his bad ends.
24:09
So and you know, otherwise we have faces there, so known bad
24:14
actors. And this, you know, this is
24:16
relevant not only for for terror, like did you know that
24:20
there's also logos for child pornography studios?
24:23
Like that's those are other logos that we can look for.
24:25
And so in many different situations, we basically can
24:31
break apart the image, look for known victims, look for nudity,
24:34
look for, you know, things that are indicative of child
24:38
pornography, hate speech, terror and so forth.
24:41
So the the experts here are involved in the curation of
24:48
data, making sure that you have robust data when you're training
24:52
these models. But then there are also getting
24:54
involved in that like 50 to 70% that gets flagged as a maybe or
24:59
is that just getting thrown back to the your clients as like you
25:04
might want to check this out we do content.
25:05
Moderation in terms of like showing what like what is the
25:09
probability of risk, but we don't have moderators, right?
25:13
That's that's the client. But what we do is we do internal
25:16
audits for ourselves to ensure that our model are still
25:19
performing like how how they should be.
25:23
So that like 50 to 70% that that's not that's out of my
25:28
game, right? That's like goes to the client.
25:29
They can do what they want. I can recommend to them, right?
25:31
Like I can recommend, I can say, look, there's a 90% probability
25:35
that this child pornography. But if they don't, if they look
25:37
at it and they say not by me or they don't want to that that's
25:40
them. But I do have on my end, I'm
25:43
going to be auditing that and and using it to improve my
25:46
abilities to detect. I was going to say, I was going
25:48
to sort of say that like all this we're talking about just
25:50
like regular content that we see online or whatever.
25:53
But right now with Jenna, I like the the space has changed,
25:57
right? And so if it used to be that
26:00
like making high quality bad content or like malicious
26:06
content, you could do it at a low scale to make it high
26:09
quality, or you could do it at a very high scale and make it
26:12
really low quality. Like now you can do really,
26:17
really high scale and really high quality malicious content,
26:21
which just means that the need for AI is that much stronger
26:25
because the only way to do it at scale is with AI.
26:28
And The thing is, is that for us, like, like, just like we've
26:33
been the experts and we've built a lot of expertise in pundit
26:37
moderation like we can now we're sort of transferring that to AI
26:40
safety. And how much of this also is
26:43
just trying to detect if it is a I generated or do you not care
26:49
about that at all O? That's for our space.
26:52
That's less of an issue, right? Because child pornography is
26:56
bad. Whether it's real or Jenny, I
26:59
like, I don't want that on my platform.
27:01
I don't want to be exposed to that.
27:02
Same thing with terror. Same thing with, you know,
27:04
misinformation. Same thing with hate speech.
27:05
It doesn't matter. It's bad and so that means that
27:09
at the end of the day, like regular UGC, like user generated
27:12
content and AI generated content are pretty similar in a lot of
27:15
in the space that we're at, which means that we can leverage
27:18
our experience and be the experts here as well.
27:22
Basically taking everything that we know from the trust and
27:24
safety space and just transfer learning it.
27:29
And so for both of them, you need the sort of like multilayer
27:32
detection and you want to mitigate in different places in
27:34
in the life cycle. And and so there's also accept
27:40
that there's sort of A twist here, I guess where with AI
27:45
generated content we can enter sort of the the chain, the food
27:50
chain earlier on. How's.
27:52
That so. If for user generated content,
27:57
we can catch it on the platform side, right?
27:59
Like once it's come to the social media platform or the
28:02
gaming platform or the dating platform, that's where we're at
28:05
and we can help them find it. But with with sort of with, with
28:10
AI generated content, we can actually be at the source right
28:14
where the where these Jenna models are being trained.
28:18
And actually we have partnerships with Cohere and
28:21
with NVIDIA specifically for that to sort of help improve the
28:27
models and red team and so forth.
28:28
And already at the source. It's not it's obviously not
28:32
enough because besides the big foundational models, there's
28:35
also open source models and even our nation models can, you know,
28:39
make mistakes and so forth. And so we need to be at both
28:42
places. But we're moving U sort of in
28:45
the food chain with with Gen AI, with NVIDIA actually, what we
28:51
don't know if you've heard of guardrails.
28:53
Yeah, yeah, yeah. Nemo.
28:55
Yep. Nemo guardrails.
28:56
Yeah. Nice.
28:57
So you can actually through any more guardrails, you can hook up
29:00
to our API with an API key and have our model sort of on your
29:07
prompts and on your outputs to ensure that they're safe.
29:10
Ohh, that's cool. So then it's like that added
29:15
confidence that you're going to not be having anything that you
29:20
don't want coming through. You know.
29:24
One thing that I was thinking about on this was like, are you
29:28
just using Cohere models off the shelf or is everything that
29:32
you're doing super fine-tuned for these different use cases,
29:36
so? Cohere is actually we're not
29:40
there are there are client of ours.
29:42
So we're helping them make their models safer.
29:45
So, and a lot of the things that we learn with our work with them
29:49
are useful for us to understand sort of how to, you know, how to
29:53
make safer, safer models. And so that's, that's like a
29:59
different kind of partnership with NVIDIA.
30:02
They're using our our models. Our predictive models are in
30:06
order to provide risk scores on input and outputs and Nemo
30:08
guardrails and and then in house, we use a wide range of of
30:14
solutions of general solutions if we need to generate data and
30:18
so. The the solutions are on like,
30:22
or let me take a step back. It's like one part of the
30:26
pipeline is generating the data and then another part is having
30:30
that machine learning or AI model or a lot of them.
30:34
You maybe have the computer vision model that will give you
30:37
the text that's in the meme that is potentially harmful.
30:40
And you've also got the model that's going to tell you if that
30:45
text is good or bad and give it that score.
30:47
And you've probably got a few other models in there.
30:50
None of these are like you being able to send a send any of that
30:57
information to like an off the shelf model, I would imagine
31:00
because it has to be very, very specific to each one of these
31:04
use cases and like built from the ground up.
31:07
If I'm understanding it correctly, yes.
31:10
Correct yes, that is exactly right.
31:13
You oftentimes you'll see like open source models for like
31:18
finding hate speech or finding this or whatever and what we
31:21
found is they can't they can't work at scale right They're
31:24
going to fall apart at the skills that we're running out.
31:26
There's no way they are inconsistent.
31:29
If we'd like to take it back, revise that policies important.
31:31
They're inconsistent in their policy.
31:33
They don't get the nuances right.
31:34
Like if we look at all of our functionalities, they fall apart
31:38
where we take the integrity of our data and then negative or
31:42
labels and then, and then you know, as a like a, as the
31:46
integrity of our models, like extremely, extremely seriously.
31:49
These are important decisions. These are important things that
31:51
you know, that we're that we're doing.
31:53
And so the quality, the data quality and then the model
31:57
quality and the performance at scale and production is saying
32:01
that we take very seriously. And for the performance side, I
32:07
often wonder about this, like if you're using transformer models
32:11
or looms, how are you making sure that there's not
32:15
hallucinations happening? And because it seems like you
32:19
have to be very confident of the accuracy and sometimes with
32:25
transformer models, they can just say whatever they want,
32:28
especially if they don't know or they've never seen this
32:31
particular type of hate speech. So is there is that where you're
32:37
talking about like you're constantly monitoring for the
32:40
drift to make sure that whatever you've labelled, you're double
32:45
checking and seeing if it is true.
32:48
If it's not, how does that breakdown?
32:51
Yes. So, um, so with any model, we
32:55
want to look at the functionalities.
32:56
And so like if there's a new type of hate speech that we
32:59
didn't know before, then the policy manager, you know, we'll,
33:01
we'll say, hey, there's this new functionality that we have to
33:04
make sure that we cover and we can catch it proactively.
33:06
If we don't catch it practically because something there's like a
33:08
new slang term or a new, you know, a new style or everything
33:14
is the same in the hate speech, But the data is a type of data
33:16
that we've never seen before, right?
33:17
Like food reviews or book reviews or whatever.
33:20
And for whatever reason, we didn't test on that data and
33:22
suddenly we fall apart. And so that's why we're
33:25
constantly auditing and, and like looking at improving and
33:28
we're retraining a lot to ensure that we're sort of non drifting
33:33
and that, you know, our performance stays stays high in
33:36
terms of LM's. We're not using currently LM for
33:40
inference because at the scale and at the cost that we need to
33:44
perform at, it just doesn't make sense.
33:46
We need to make sure that our clients and us, we all lose an
33:50
arm and a leg in terms of like being able to predict and to
33:53
give value at such a high scale. And so we use LMS intensively
34:02
beforehand, but at prediction time we need much more robust
34:05
solutions. Yeah, that makes sense.
34:07
And it's totally like it's what you would expect almost, right?
34:14
You don't mean because the LM's are a much slower and be a
34:20
little bit less trustworthy on the output, They can hallucinate
34:24
or do whatever. I would hope that either you
34:28
knew some kind of trick and you had some special magic pill or
34:33
silver, silver bullet or you're doing that, which totally makes
34:38
sense. Now the auditing process for me
34:41
feels like something that is very rigorous.
34:44
Can you go into that a little bit more?
34:46
And what the whole audit looks like and how then you make sure
34:53
to like continuously stay at a very high accuracy.
34:57
Yeah. So in addition, so even before
35:00
the auditing, that in addition to the functional test they
35:02
talked about, which I think are like really, really critical, we
35:05
also do other types of tests. So we have a go, no go test,
35:10
which basically what we say is this is like a set of data that
35:13
we can't afford to mess up on, right?
35:15
Like these terms are always positive and these terms are
35:20
always negative. And if we fail on them, then
35:23
that's a no go for the model, right?
35:25
This is like a data set that basically is like go, no go.
35:27
Can this model go to production based on these?
35:29
Because I can never say have like hello world or I love you
35:34
be hate speech like that won't work or high, right?
35:36
There's things that are, and so there's that, Um, and we also
35:41
have a false positive rate test. So what we do is we'll take our
35:44
model and we'll run it on data that is supposed to be
35:47
completely benign and also on data that's like borderline.
35:51
And then we can see like what is our false positive rate?
35:54
So for example, for hate speech, we can run it on like on Harry
35:58
Potter on some book for hate speech text, and there shouldn't
36:01
be any hate speech in there. And if there is, then we can
36:04
analyse the situations where it did get flagged, like what are
36:07
these false positives? And then understand our model a
36:09
bit better and understand, OK, like why is it flagging this and
36:11
what can I do? And sort of retrain again for
36:15
child pornography or child abuse, we can run it, you know,
36:18
for, for the benign data, we can just run it on like Google open,
36:21
set, open image, like, you know, it's just like a dump of images
36:24
or whatever. And then like make sure that
36:27
like you don't get, you know, a positive score on like a picture
36:29
of a keyboard or whatever. And then but we can also run it
36:33
on images that are like, maybe that are like a little trickier
36:36
for the model, like Sports Illustrated, imminent swimsuits
36:39
on mother's kissing babies, breastfeeding people on the
36:43
beach, toddlers and diapers, things that are like not season,
36:46
but they they do have, you know, like bare body and children and
36:51
you know, children on a playground, things like that.
36:53
And then you want to make sure that like none of those are
36:55
flagged either. And sort of and they're sort of
36:57
more towards the border and like more difficult for the model.
37:00
And so these are things that we do, you know, production or out
37:04
or like before, you know, after retraining to ensure that we're
37:07
like still like at keeping our quality height in terms of like
37:12
once we're in production, then like I said, we can do a couple
37:17
of things. We can, or a few things we can
37:19
just look at our distribution of scores, right?
37:20
And sort of start to notice, like, are we drifting?
37:23
Like are our scores suddenly like much, much lower or much,
37:26
much higher? Because probably like at the
37:28
scales that we're talking about, the world isn't suddenly
37:30
becoming really friendly, right? Like people are still being
37:33
mean. People think so that's just
37:35
where they're changing. There's something going on as
37:37
much as. We would like the world to be
37:39
becoming more friendly. That's usually not the answer.
37:43
And also. We have like we'll do audit
37:45
specifically targeted for customers.
37:47
Like let's look at like really, really high scores.
37:50
Let's look a little bit of scores in the middle.
37:51
Let's sort of look at, you know, the hardest is defined false
37:54
negatives. But we can also sample within
37:56
the din and be like, OK, things, you know, with this keyword or
37:59
this sort of things like let's look at all of them and see what
38:02
the distribution of scores were. And are these negative ones like
38:05
really negative. Like for example, you know, we
38:08
have, we used to have something like.
38:10
One thing that we look at is like token overfitting.
38:12
So, um, I like black coffee or I hate black coffee or I love
38:19
Chinese food or I hate Chinese food.
38:21
We want to make sure that like those aren't being flagged.
38:24
And if they are, then we need to find ways to sort of combat
38:28
that. And we can look for that also in
38:29
production data to make sure that you know our functional
38:32
tests are still valid in production.
38:37
OK. So and then you have the audits
38:40
that you'll go through and there's the false positive,
38:43
there's the go, no go, there's the functional test.
38:46
And then once it's out in the wild, you're just looking at it
38:49
and you're seeing, did we overfit for some things?
38:52
Let's look at this distribution score, let's see where we're at.
38:56
Do you also have integration tests and like unit tests and
38:59
these kind of? Things, Yes, yes, sorry, that
39:01
goes without saying. That's like the four Yes, of
39:03
course, yes, we do all that get into a test We have we also
39:08
before our model hits production, we do load tests.
39:11
We want to make sure that we're still able to like that we
39:13
haven't changed anything so that you know, we can still meet the
39:15
load. We look at the price like what
39:18
instances it's on. So we have a lot of heavy
39:20
engineering work that we do before a model like ever sees
39:25
the light of day. And also the functional tests
39:27
and can and the guano does test. They're also run on production
39:31
data, right? Like while the model is out in
39:33
the wild, like just, you know, it's not, they're not only done
39:36
like when we've been trained, they should be done constantly.
39:40
And also the audits is that we have our policy managers and we
39:42
have analysts that are going in and sampling data and looking at
39:45
and looking at the data and saying like, is this a false
39:47
positive? They'll look at the high scores
39:49
and say, OK, this is a mistake. This is a mistake.
39:51
And then they go and they sort of summarise it for us and say,
39:54
hey, guess what? Every time someone talks about
39:56
food, you're flagging it. I keep bringing up food because
39:58
like, it was like the first, like big.
40:00
Fiasco that we had. So the, the process that I would
40:06
love for you to walk me through is how does it look like when
40:13
the data scientist has whatever hate speech model, one that
40:20
they're going to be using very specific, whatever it may be,
40:24
and they're like, this is good. I'm ready to bring this to
40:29
production. I'm ready to are you throwing it
40:33
over the fence, giving it to somebody?
40:35
Do they run with it? How does that look to like take
40:38
it from the potentially Jupiter notebook if they're playing in
40:42
that and then now being served as an API at very low latency?
40:47
So what we have is we are, we serve our models, we use
40:52
Kubernetes, we are on, we're, we're on a WS and our, but
41:01
basically our inference pipeline is our inference infrastructure
41:05
is like is we, we, we built it ourselves.
41:08
We were looking at all kinds of different vendors and we decided
41:10
that in order to really keep costs low and at this with at
41:13
this scale, we didn't want to sort of pin overhead for every,
41:16
for every inference. It just didn't make sense.
41:18
And so we were also able basically to write at least for
41:23
a text models right now. And this thing that we're still
41:25
working on, on this infrastructure basically to have
41:27
sort of like a no code deployment workflow for our data
41:31
scientists. So once they've for like
41:34
architectures that were like that we know and have been low
41:37
tested and we're very comfortable with, there's like a
41:38
very easy path for them to basically just deploy to
41:41
production. And the data scientists, they do
41:44
that. I strongly believe that people
41:47
should write code for production, including data
41:49
scientists. And I think that sort of like if
41:52
your output is a Python notebook, then you lose a bit of
41:56
the ownership. And even though the data
42:00
scientists are not the ones usually waking up at night phone
42:02
calls, right? That's like the engineers.
42:04
But it's theirs. They own it.
42:06
They know it. They're familiar with it.
42:08
It's part of their ownership. And so it's not that they sort
42:12
of like just do the research and then kind of like kick it over
42:14
and say, hey, someone else needs to take this.
42:18
Yeah, you're preaching to the choir.
42:20
The production code, we actually have a channel in our Milos
42:25
community, Slack, that is called Production Code.
42:28
And it is 100% that just like talking about how we can make
42:32
sure that we're writing more and better code, production code,
42:38
especially those who are data science scientists or have data
42:42
science backgrounds and understanding the benefits of
42:45
that will take you so far. Absolutely.
42:49
And so the the OPS team is like very focused on writing the
42:54
infrastructure so that that can happen seamlessly, but the
42:58
ownership and put in pushing it there and it needs to be needs
43:01
to lie also with the data scientists.
43:03
Yeah, it's fascinating too to think about.
43:05
OK, you created almost like this no code solution so that it
43:11
eliminates the friction that the data scientists have once
43:16
they've once they have the confidence that like, alright,
43:19
this model is going to be better.
43:21
And if passed all the cheques that I needed to pass, then they
43:26
can push it and say, alright, cool, now let's go.
43:29
And they have that ownership. They can take it across the
43:32
finish line. I will.
43:33
Say that it's not completely seamless, it's not completely
43:36
smooth. We're constantly learning.
43:37
Like we're still building, we're a startup.
43:41
But yes, that part works. Yeah.
43:44
There's other parts, yeah, that are fun and that you're probably
43:46
digging into and I can imagine, I, I can imagine a tonne.
43:51
So there there is so much good stuff that we talked about here.
43:57
I think the thing for me that is one of the hardest problems that
44:03
you're dealing with is a tonne of languages, a tonne of
44:08
cultural references, trying to make sure that you are not just
44:14
like taking all the fun out of the Internet.
44:19
Because there I, I, as you mentioned, you could overfit and
44:24
now somebody says hello world and boom, that's flagged.
44:27
Yeah. And so how can you find that
44:30
proper medium that is not taking the fun out of it, but and just
44:36
like flagging everything, but also not being too lenient and
44:40
not flagging the things that definitely should be flagged.
44:43
Yeah. So I mean.
44:44
This is a well known business problem, right, where you have
44:46
sort of this balance between precision recall and a lot of
44:50
times it's also if it's a client question, right, like how
44:53
sensitive, how sensitive are you?
44:55
So if you're a child, a child's platform, then you probably want
44:58
to err on the side of like high recall, right?
45:01
And even at the cost of precision and you'd want to like
45:03
over flag and yeah, take some of the fun out because at the end
45:06
of the day, I don't care if they're limited to like 12
45:08
words, they can type like that's fine.
45:10
At least there's yeah. Versus if you're a dating
45:14
platform, you're gonna be a little bit more tolerant of
45:16
like, you know, some a little bit more exciting talk.
45:19
And so it's a business question. We have our policy.
45:24
And like when we're building them, like I said, we're like
45:26
very, very clear about our policy.
45:27
The policy manager is like, works very closely with our data
45:30
scientists. He's at all of their dailies.
45:33
And anytime there's a question, there's like, it's, it's, it's
45:37
an ongoing conversation because sometimes the policy will say
45:40
something and then we'll be like, yeah, what about this
45:42
simple? Like, is this, is this not in
45:43
the labelers are always asking and we're always asking.
45:45
So it's sort of this like constant conversation to make
45:49
sure that at least in terms of our policy, our data and our
45:51
data quality, our labelling is like very, very high quality.
45:54
And that's also that we can that we, you know, communicate to the
45:57
client like this is our policy. If you're more sensitive than
46:01
you can add a keyword, right? So if, for example, our hate
46:04
speech policy, reclaimed slurs are not hate speech.
46:08
So if I'm saying like I'm proud to be a, then that's not hate
46:14
speech, it's reclaiming, right? Or don't call me a that's not
46:18
his speech. That's like victim, like, you
46:19
know, asserting themselves. But for some platforms, any slur
46:23
is not OK, like I don't want you to use that word at all.
46:27
And so that's that has nothing like my model will not is not
46:32
going to be like tuned to that particular, you know, sensitive
46:35
level. And so for that we have our
46:37
keyword tool. Feel free add the keyword and
46:39
describe how it is and then just add it to your model and then
46:42
like that, that'll be the race zoo level.
46:44
And then if you want, you can take all the fun out that way I
46:47
have some level of fun. And so it's again.
46:50
And then also clients can set their own threshold, right?
46:52
So I give the probability of risk and then if you're like
46:56
very, very, very sensitive and you don't want anything, you'll
46:59
set that threshold really, really low.
47:01
And then you'll make sure that anything that's like a even a
47:04
little bit probable of risk will get flagged.
47:07
And if you're not sensitive then just set your threshold really
47:10
high and go have all the fun you.
47:13
Yeah. Well, it does feel like the main
47:17
thing that you're probably doing continuously is labelling data.
47:21
Is that safe to say? Like you are you or the teams?
47:26
I would imagine like it feels like 90% of this happens at the
47:30
data labelling phase to make sure that whatever goes through
47:33
the pipeline and is then on the output gets flagged or is is
47:40
caught. So yeah.
47:41
So I think there's two things that make our solution our
47:43
models are three things that make our solutions, our models
47:45
so powerful. The 1st is really the subject
47:49
matter experts and the intelligence analysts and the
47:51
fact that we, we really know this world, right?
47:53
Like we know how to build a policy.
47:56
We know what it means to be a speech, what it means to be
47:58
harassing building, what it means to be suicide, self harm.
48:00
Like we know this really well and.
48:02
In different languages, right? And different cultures, yes.
48:04
Many and we also have cultural context.
48:06
So someone calling someone a dog in one language is not the same
48:09
as calling them a dog in a different language.
48:11
And so we're sensitive to that and we know we know how to do
48:14
that. And what that means is that our
48:17
data labelling quality is our data quality is very, very, very
48:20
high because it's so linked to our intelligence analysts and to
48:24
our policies. And we're also very aware that
48:27
like that's what gives us an edge, right?
48:28
Like our data is our gold, right?
48:30
Like this is what like we, we've invested a lot of this and this
48:34
is what it's, you know, it's what our models are built on.
48:37
And then the other part of that is that we just killer engineers
48:40
and killer data scientists that are able to take this, you know,
48:44
this data and turn it into something that's able to make a
48:47
real impact in the world at scale.
48:49
There's it's two parts of the puzzle, because you can have all
48:53
the data in the world that is very high quality data, but if
48:56
you can't operationalize that with a machine learning model,
48:59
then it's as good as nothing, right?
49:03
Right. So that's like the raw
49:05
materials. We need to know what to do with
49:06
raw materials. That's why I said that there's
49:07
like a lot of open source models out there and oftentimes they
49:10
fall apart of the data, right? Like they can throw like a
49:11
really big cool model at something and then it will run
49:14
at scale with all in production. And also, it's based on poorly
49:18
labelled data or inconsistent data and it doesn't matter, like
49:21
how amazing the algorithm is. It makes sense, yeah.
49:24
Well, yeah. And it seems like the other
49:27
piece, for better or worse. Like I can imagine it is hard
49:32
being some of these subject matter experts because you have
49:34
to go into the depths of like the worst parts of the Internet.
49:39
Yes, so where we take the well-being of our labelers and
49:45
of our experts very, very seriously.
49:48
We have a psychologist on staff. We have like well-being
49:52
programmes. This is something that's not
49:55
taken lightly. Like I said, we care a lot about
49:57
this. This is another reason like
49:59
putting AI in the loop here. It's not only because it's like
50:03
really it's the only way to combat this, right.
50:05
Like there's, there's the sheer scale of things like you can't
50:08
do it manually. But it's also because we want to
50:11
offload and sort of focus human energy only on where it's like
50:16
going to be most important and most effective.
50:18
Uh-huh. Yeah, the questions and that
50:20
burden, it makes a lot of sense. So incredible.
50:27
But thank you so much for coming on here and talking to me about
50:30
this. I've, I am thoroughly fascinated
50:33
by it and I think you're doing so much great work.
50:37
And as a father of two daughters, I know I told you
50:40
this before, but I'll tell you that now that we're recording, I
50:42
am so grateful that people like you and Active Fence exist
50:47
because it is just the I know the Internet isn't going to
50:53
always be like 100% safe, but it does feel like at least you're
50:59
trying to do something to make it a little bit safer.
51:02
Thank you. It was great being here.
51:04
I was really excited to get to speak to you and thank you for
51:07
inviting me and I hope I get to keep doing great work.
51:12
Important. Excellent.