Latest / Agentic Conversations (formally mlops.community) / Building Threat Detection Systems: An MLE's Perspective // Jeremy Jordan // MLOps Podcast #134

Transcript

Building Threat Detection Systems: An MLE's Perspective // Jeremy Jordan // MLOps Podcast #134 · Agentic Conversations (formally mlops.community)

0:00
Hey, I'm Jeremy. Jordan.
0:02
I work as a machine learning engineer at Duo security, and I
0:06
don't drink coffee but I do enjoy tea.
0:09
My favorite is the Tazo passion team, big fan of that.
0:15
Welcome everybody to another ml, Ops Community podcast.
0:18
I am your host Demetrios. And today, I am joined by none
0:23
other than the incredible co-host Vishnu.
0:26
What is happening my man? It's your favorite.
0:29
I'm back. You know it.
0:34
So we had the luxury, the pleasure of talking with Jeremy
0:40
today, which is incredible because I have been hounding
0:44
this guy. I, and when I say hounding him,
0:46
I reach out to him probably once every couple months to get him
0:49
onto the podcast and that has been happening for two years and
0:54
finally he caved in. So it whenever someone is down
0:59
on their luck and you feel like somebody just rejected you
1:03
remember, no does not mean, no, it just means not.
1:06
Now in this context in other contexts.
1:08
No means no. All right, let's just keep it
1:10
straight. So what do you think, dude, it
1:13
was a great conversation Jeremy. Awesome.
1:15
If anybody does not know about his blog I recommend you check
1:18
that out right now. Pause this and go and read some
1:21
of the stuff on his blog because he's an incredible writer.
1:25
What were you take aways? Yeah, I mean, for those of you
1:27
who don't know Jeremy, Jordan. He is a machine learning
1:30
engineer. Do a security right now was
1:32
previously at a company called proofpoint study Material
1:35
Science at NC State and is overall an extremely prolific
1:40
machine learning systems and ml Ops blogger.
1:44
She has blog. Jeremy, Jordan .me, Jeremy.
1:47
Jordan that me as like incredible content when it comes
1:50
to actual practical considerations around how to
1:54
build machine Learning Systems topics on testing topping on an
1:57
advanced models topics when the infrastructure as code, all
2:00
kinds of stuff and that was really the reason that we wanted
2:03
to bring them on the podcast. I actually, yeah.
2:07
Just as a little Accolade side note, he is a maintainer for pi
2:11
torch lightning to write. Doesn't really talk about that
2:14
much. But he does it.
2:15
So just another thing to add to that list.
2:19
He's incredible. He's a gourmet dinner of it,
2:21
right? Yeah.
2:22
So it's great call out. I really enjoyed it.
2:24
I've been a fan of Jeremy's work for the last two years,
2:27
actually, like a lot of what he's written.
2:28
Has been extremely useful for me, on my email journey.
2:32
And I think what I really enjoyed during our conversation
2:34
was talking about rules engines, funnily enough, I think a big
2:39
part of what I think I learned in this conversation from
2:43
hearing about, Jeremy was How he thought about rules as an
2:47
addition and a hedge against the unpredictable behavior of ml
2:51
systems, that lack of predictability is kind of a
2:55
feature, not just a bug, right? That's the whole point of these
2:57
probabilistic systems and how he thinks about design, using rules
3:02
engines as sort of additions and hedges around those systems.
3:06
I thought that was a fascinating area that I'm definitely going
3:09
to apply at work. What about you?
3:10
Yeah, I loved it. And so for me, what is
3:14
Interesting. Is that it really feels like
3:18
there is a clear pattern that we have been seeing with some of
3:22
these great in the field. That is ml Ops and that pattern
3:28
that I can see. Now that we've interviewed, I
3:32
don't know how many of them hundreds is that so many use
3:38
writing as a forcing function to learn about where they have
3:43
holes in their own. Understanding of something.
3:46
And so if you are not writing, I've said it before.
3:49
I think I tweeted out something to wear.
3:50
I posted on LinkedIn how important writing is for your
3:54
own development. But if you are not writing, this
3:56
just is another great example as to why he goes into it in depth,
4:01
as to how beneficial it is for him to write.
4:06
And for him to see that he doesn't understand something if
4:11
he cannot rearticulate it in writing, So that was my key
4:17
takeaway. I loved it.
4:18
Vishnu I know you write these days a little bit more.
4:21
Not as much as I wish you wrote. Yeah, I still think you have
4:25
like five pending blog post for the envelopes Community blog.
4:28
But what do you can't win em? All right.
4:31
And do I still have plenty? But I'm trying to trying to be
4:33
more prolific. Think Jeremy.
4:35
There you go. So, let's get into this full
4:38
conversation with Jeremy. And while you're at it, if you
4:41
like this, if you feel, if you feel that you're getting any
4:44
value, You out of this conversation, you know.
4:47
It would be huge. What would be huge?
4:48
Vishnu, like, subscribe and review on YouTube.
4:53
Your favorite podcast app wherever you can is that right?
4:56
Demetrius. Leave a review.
4:58
Also I think that was kind of in the yeah.
5:01
You said that right? So I did reviews is huge
5:04
comment. If you're on YouTube all that
5:07
stuff is immensely valuable for us as we keep growing the
5:12
podcast, one thing that Vishnu and I set out to do Do at the
5:15
beginning of this year which is crazy to think that we have hit.
5:19
This goal is. We said we were going to get to
5:22
x amount of listeners by the end of the year per episode.
5:26
And I don't think Vishnu has looked at the analytics in a
5:29
while but we hit our number. And so I'm probably going to be
5:32
writing a blog post about it at the end of the year just to tell
5:36
everyone where we're at as far as numbers and keep a little
5:38
transparency on it because I think that is so cool and it is
5:42
huge. That being said, let's get into
5:47
the conversation. Dude, it's great to have you on
5:56
here and I want to jump into this directly by talking about
6:00
the norm Kampf which is I think still short form for the normal
6:06
conference. And from what I understand is it
6:09
is very similar to an idea that our buddy Flavio who's also a
6:14
Community member wanted to create in 2020 where it's like
6:20
and he wanted to call it the boring comp not by any means is
6:23
your conference boring. Carol, I'm trying to make here
6:28
but what I think is interesting is that there's a lot of hype
6:31
everywher in machine learning and Ai and every day there's
6:35
more of it but you're trying to highlight the stuff that works.
6:39
And can you just give us a bit of an explanation on what you're
6:44
doing and why you're doing it? Yeah, that's exactly the, the
6:47
conference started as like a joke.
6:50
Tweet by Vicki boy kiss. And, you know, she tweeted like,
6:54
um, Go start a conference about like not all the sexy type stuff
6:58
but like you know the boring practical stuff that makes the
7:02
world run and that tweet got so much like positive reception on
7:07
Twitter. Everyone's like yes please
7:09
actually do this and you know we had a group chat going like well
7:14
should we actually do this? So yeah.
7:19
One thing led to another, we decided to make it a real thing
7:22
and you know, Host this conference we have A great
7:26
lineup of speakers on all sorts of fun subjects and you can
7:31
check that out. I like norm conf.com and it's a
7:35
free virtual conference that we're hosting December 15th
7:40
that. Yeah, just like wanted to have
7:42
kind of a refreshing take of practitioners talking about the
7:46
nitty-gritty day-to-day work that, you know, sometimes gets
7:50
overlooked in lieu of all the, you know, hype that we see every
7:54
day. And you're talking, right?
7:57
You're giving you're giving a speech or presentation.
8:00
Are you giving the keynote? What's your name?
8:03
I think he's giving the keynote which is like the bright Jade
8:07
that. Yeah, she this and so it
8:10
actually is short not for normal conference but Vicky has this
8:14
newsletter and kind of brand around like Normcore technology
8:19
and so like Norm comp is kind of like that the Normcore
8:21
conference. So very much, you know, fitting
8:24
into her. Her brand there.
8:25
And I think it also helped people like quickly like that
8:28
read her newsletter which is great.
8:31
And you know, you should read it and I latched onto like what
8:36
this conference would be about. But yes, I am giving a talk.
8:40
I am going to give a talk about how rules engines can work
8:46
together with machine learning and you know, I think the common
8:49
Trope is like, yeah, you deploy here stick to start that gets
8:53
you going and then you replace it all with Learning and I've
8:57
learned kind of working in cybersecurity like that, replace
9:01
it with machine learning, doesn't usually happen.
9:03
It's usually more like we have a rule system, and then we're
9:06
adding machine learning on top of it and they work together.
9:10
I love that topic Demetrius. Just something you want to add
9:13
there. Oh, you men.
9:15
Okay. Alright, perfect.
9:16
Yeah. Because because this is actually
9:18
a topic. I'm very much interested in.
9:19
You know, how rules engines and machine Learning Systems can
9:25
Play nice together. You know, I think I saw your
9:27
slides from the threat detection talk that you did that, you
9:31
know, in a in a, in another ml Ops Community Demetrius it's
9:36
chips unnamed and a lot of community to throw it back to a
9:40
previous meme. So confusing, but whatever.
9:45
Yeah, so my first question here is like When a rules engines,
9:52
good, and when are they not? Let's start there, because I
9:55
feel like this is going to take up next few minutes.
9:58
Definitely, so, especially working in cybersecurity, like,
10:03
one of the main tasks that we work on his threat detection and
10:08
being able to detect threats, so that we can then stop them from
10:11
causing harm to users. And a lot of times when like we
10:16
first learn about the threat, we don't have a ton of data on that
10:19
threat. Want to provide protection
10:22
immediately, like using the domain knowledge that we do
10:25
have. So rules are great for encoding,
10:28
our domain knowledge or like understanding of a given threat.
10:32
Before we have this huge labeled data set that we can go and
10:35
train a machine learning model. And there's like the landscape
10:41
of cyber security is always evolving because we get B like
10:44
at this cat and mouse game. We get better at detecting
10:46
threats. The people these like threat
10:49
actors make money. When you know, they can
10:53
successfully attack somebody and you know fish their credentials
10:57
or, you know steel yeah. Like get access to accounts so
11:03
when we get better at detecting and blocking those threats they
11:07
get less Revenue. So been they're incentivized to
11:10
implement new threats and vulnerabilities so that they can
11:14
increase their revenue. Like you start actors commonly
11:17
run this like a business and they, you know, Like any good
11:21
marketing campaign, they like calculate metrics on, you know,
11:25
how successful are that actor him like their campaigns doing.
11:30
And and so, you know, if there is a decrease in their
11:32
efficiency, they're incentivized to then go find some new
11:35
loophole or new social engineering attack.
11:39
So yeah, the landscape is always changing and we have to be able
11:42
to like adapt to those new threats before we always
11:45
necessarily before we necessarily have a bunch of data
11:48
to go train a model to detect that Rhett got it so I can
11:53
totally see how this problem is very difficult, but to make sure
11:56
that I'm understanding your sort of saying rules engines are very
12:01
helpful for encoding. What we know about the problem
12:03
and what we know about the problem often outstrips the
12:07
amount of labeled data, we have, right?
12:09
Like if a new thread pattern emerges, you know that as a
12:12
practitioner before, you have a ton of labeled data to be able
12:14
to train a model. And so you go and Implement a
12:17
rule. Is that right?
12:18
What I'm hearing there? Yeah.
12:21
And so that's one side of where the rules can be really good.
12:25
The other side is say, we have both rules and machine learning
12:29
models that we have deployed, our machine learning model,
12:33
might introduce some false positives or we might start
12:36
blocking benign content and that's bad.
12:40
You know. Great customers complain about
12:42
that. It's really easy to quickly
12:44
remediate that with a roll and you can do that a lot quicker
12:48
than it, takes to like okay let's add this.
12:51
This to the label data set, and then maybe we need to go collect
12:53
similar examples and label that and then kick off a model
12:56
training, run that you guys just like a longer process.
12:59
And, you know, ideally you're doing both so we can improve the
13:03
model and reduce those false positives as well.
13:06
But in terms of quickly reacting, a rule engine is, you
13:10
know, much easier and much quicker.
13:14
That makes a ton of sense. My sort of basic question here
13:18
is, how do you implement a rule? Is it a sequel function?
13:22
Is it a python function like what is a rule in your world,
13:25
and in your systems? Yeah, that's that's a really
13:29
good question. It's kind of depends on the
13:33
environment. So I've worked at a couple cyber
13:35
security companies. I'm out Duo and a lot of our
13:38
hero sticks right now. Look like we would try to make
13:44
it very flexible to Define so it's effectively like a python
13:47
function and then if you can write a python function that
13:52
adheres to, you know, a certain convict like set of conventions,
13:57
Mentions of how we like input and process the data then, you
14:02
know, we can take that and deploy it alongside another set
14:05
of rules and you know run that in a distributed fashion. okay,
14:11
so you have these rules implemented, sort of using
14:14
python functions as part of this current system that we're
14:16
talking about here, you know, this it's it's I'm asking these
14:22
questions because this is actually something that I'm
14:23
thinking about it's a real problem and I'm kind of trying
14:25
to face, you know, in terms of, you know, how do we implement
14:28
the rules engine and, you know, as a practitioner as a machine
14:33
learning practitioner, I immediately jumped to a certain
14:35
problem that I'm focused on saying, this makes a ton of
14:38
sense for a rules engine and the pushback that I got Is that
14:42
well, how do you manage an ever-increasing number of rules?
14:47
How do you weight them? And how do you make sure that
14:50
overall the system doesn't get unwieldy right?
14:53
And I was like, okay you guys are way over complicating this
14:57
but I don't know how to answer that and I want to ask you that
15:00
question. Like how do you manage rules
15:04
sprawl? Yeah, that's a good question and
15:10
On that particular. Like, on that note, particularly
15:13
I think one good practice, especially like in my world of
15:17
cyber security. We want to make sure that the
15:21
rules that we are defining our very high Precision.
15:25
So we can effectively say like, for when we're trying to detect,
15:29
something you can just say, like, if any role fires, then
15:33
we're going to flag it as malicious and block it.
15:37
So you don't want to have to deal with like interactions
15:39
between Rolls of like, oh well, this rule is kind of noisy.
15:42
So I have this other heuristic that, like, tries to Tamp it
15:45
down, if we can instead, you know, have this Union of Suite
15:51
of very high Precision rules, that's usually like the most
15:56
productive outcome. It does limit your ability to,
16:00
like catch everything, because we're like, you know, optimizing
16:05
for precision can harm recall. But I think that's also where
16:09
Machine learning can complement things really well.
16:12
Is that machine learning can like kind of learn from the or
16:15
big labeled set kind of generalize better patterns and
16:20
uncover threats. That like we're being missed by
16:24
these high Precision roles and then you can use, you know,
16:28
okay, machine learning model flag.
16:30
This is malicious but our rules didn't catch it.
16:32
Was it actually malicious? If so can we update a rule to
16:36
then have better coverage for that.
16:38
So like the two systems can Can work together quite nicely.
16:41
That was the exact question that I wanted to ask.
16:43
Next was, how often are you updating or reviewing the rules?
16:50
Yeah, it's a continual process and cyber security because you
16:55
know the threat landscape changes and evolves so much.
16:58
And there's always more to be, you know, there's always more
17:04
like detection capabilities that that we can offer.
17:07
So it's an iterative part of the process of like having good.
17:13
Observability over like, okay, what what rules are these
17:16
content flagging? Sorry, what content?
17:19
These rules flagging and, you know, are they still performing?
17:25
Well or are we missing things? Do we need to using our
17:28
knowledge of the domain? Do we need to add new rules to
17:32
get better coverage of evolving or new threats?
17:37
I had a quick question there as you bring that up.
17:40
It sounds like you. So I'm actually there such that
17:43
Martin Fowler post on rules engine, as there is on
17:45
everything. There's a bunch of all of us.
17:47
Okay. And it's really helpful.
17:49
It's a really nice short post that he has and that that
17:54
behavior that you're describing of how rules interact with
17:58
another one another, he calls chaining, right?
18:01
Yeah, which I'm not sure if that's like an industry standard
18:03
term, but you know what? Use are sort of telling me here
18:07
is that you your system, doesn't engage in chaining.
18:12
It's something that you try to do to try to keep these rules.
18:14
Pretty separate. Is that correct?
18:15
What I'm hearing there? We're not saying it's, it's
18:19
mostly correct, I think it like, There is a case where we have
18:27
like, will Implement a heuristic?
18:28
That is a bit noisy and then we'll add kind of like a
18:30
suppression layer on top of that passion.
18:34
That is like, we might have a general rule that we write that
18:39
applies to all customers and or like that were all users that
18:43
were protecting. But then we might have like some
18:47
suppression logic, that depends on, you know, the previous
18:50
behavior of that user or customer Zone.
18:55
There there is in some cases that like, kind of suppression,
19:01
I think it. Yeah.
19:02
Just kind of like depends on the environment.
19:06
Yeah, I mean, I think, you know, it's funny, we're sitting here
19:09
in this is an ml Ops podcast and we talked about generative.
19:13
It generative, AI, deep learning.
19:15
And here we are in 2022. Talking about the finer points
19:18
of rules engines. So, you know, maybe as much as
19:22
things change, they always stay the same but I think this topic
19:25
is very interesting because, you know, The way that you described
19:31
rules being sort of front end almost you know for your for
19:36
your machine learning model which are in the back end
19:38
catching more complicated threats.
19:40
That to me is speaks to the question of design and the
19:43
design of machine learning products and systems, which is a
19:47
really important component of overall, like ml systems
19:52
development. It's something that's under I
19:56
think address right now, right? We have a lot of guidelines.
19:58
Lines on how to build a model, how to put it into production
20:01
now, but we don't really have a lot of guidelines in terms of
20:03
like how to design an ml system, how to how the fail-safes that
20:06
rules can provide. And that's why I'm so interested
20:08
in this topic, right? They help prevent the kinds of
20:10
Harm's and they help mitigate the kind of risks that we see a
20:13
lot of times that machine Learning Systems have.
20:16
And so, I think, you know, I don't know, and I don't know if
20:19
you find it. Similarly, interesting in that
20:21
realm, but I think I remember, you know, one part of your talk
20:23
that you sort of mentioned something similar in that
20:25
regard. Yeah, definitely.
20:27
I. So I have, like, I really
20:31
enjoyed design thinking, I actually, when I was in college,
20:36
was pursuing a minor in Art and Design, just because I really
20:39
love design thinking and have always been really interested
20:44
in. Like, yeah, how do we design, as
20:46
machine learning products? So, like, you know, you hit that
20:50
virtuous cycle where, you know, you get data and that improves
20:54
the model and then better model means more.
20:57
Jim more data and that like, there's that, I think Nirvana of
21:00
like the data flywheel that can cause a positive feedback loop
21:05
and designers really influential in that.
21:08
And so I love thinking about this area and specifically in
21:12
cybersecurity. I think it's interesting to
21:17
think about how we design our, like, detection systems and, and
21:22
how we can protect users, because I mentioned earlier,
21:25
like, you know, The rules for example, and for blocking models
21:29
as well. So if you have a model that's
21:31
going to block content, all of that needs to be super high
21:33
precision and that's because we don't want to impose a lot of
21:38
friction to users. Like you know, if you're trying
21:41
to log in to do your work today like and you're blocked, that's
21:46
that's very annoying and might lead to you actually just like
21:50
turning off all the cyber security protections that we
21:53
might have. So we need to be low friction
21:55
and easy to use. So, I can like enjoy getting
21:59
those protections and so, you know, we can block in that
22:05
really high Precision regime but then there's still like you know
22:08
some content that's like in this gray area.
22:10
It's like we're not really sure that it's benign but we can't be
22:14
so confident that it's malicious that like we feel comfortable
22:17
blocking it. And so I think it's a really
22:19
interesting area of like how do we provide our users safety in
22:23
that gray area? And I given that talk I gave a
22:28
couple of examples. So one I started my career proof
22:33
point it was there for about three years which is like a
22:37
lingerie email security provider and so we're very focused on
22:42
things like phishing malware delivery.
22:45
And one of the products that I mentioned as like this
22:49
interesting design application in that gray area is so say we
22:54
have a link in an email that We're not kind of like, it could
22:59
be suspicious, but we're not confident in blocking it,
23:03
instead of just letting the user open that link in their normal
23:06
browser, like proofpoint, has this product called isolation
23:09
browser, which still lets you visit the webpage, but you know,
23:14
can prevent you from doing like some like accidentally leak in
23:17
your password or other other things like that.
23:21
So it's like instead of completely blocking the user
23:23
from visiting that URL, we can place them in an environment.
23:27
Ain't that they can still like visit the page, but it's a
23:29
little more lock down just to, you know, for their own
23:32
protection and safety. And then, you know, similarly,
23:36
now I'm at Duo security which we've largely focused on access
23:40
security. So we started with two-factor
23:42
authentication. We now have a whole Suite of
23:44
products, like SSO password list, authentication, all the
23:48
all that fun stuff. So when area where we're kind of
23:53
trying to design for that gray area, No, we're not confident,
23:58
walking on login attempt, but we don't necessarily feel
24:01
comfortable, just like letting that through because there's
24:04
some area of risk associated with that.
24:06
As we have, this concept of risk-based authentication, where
24:11
depending on the risk that we evaluate associated with a given
24:15
login attempt, we might restrict to the set of factors that you
24:19
could use to actually, authenticate and login.
24:23
So, you know, if we have an authentication that looks risky
24:26
Gee, maybe, we don't allow you to get like an SMS two-factor,
24:30
authentication code and maybe we require you to use like yubikey
24:34
or touch ID to authenticate. So kind of like modulating the,
24:40
the amount of friction that we provide in the login attempt
24:44
based on the risk associated with that.
24:48
That is really cool. And I just want to ask one
24:51
question there, which is the notion of risk and confidence,
24:55
right? I think I've seen a lot.
24:57
A lot of work recently from the economic machine learning World
25:01
about confidence estimation. Uncertain learning and uncertain
25:05
Realms cybersecurity is a highly uncertain realm from a machine
25:09
learning development perspective.
25:11
How do you approach certainty and risk estimation?
25:15
You know, when the output isn't just 0 or 1, Yeah.
25:20
And I think that gets back into like you know, okay so we're not
25:24
designing for binary outcomes. If like allow or block, we're
25:27
designing for this like gradient of possible outcomes and when we
25:32
are blocking, we want to be very sure.
25:35
So we're going to like tuner thresholds to be like very high
25:39
Precision that we feel confident that we can block on that
25:42
content. And then in this gray area, I
25:45
think the best case scenario is Is we're not really sure.
25:50
So we went to like try to modulate friction, but we can
25:54
also observe like was the user able to login.
25:57
Like we made them use a more secure factor for logging in and
26:02
they pass that they actually log in and that's useful feedback
26:07
for us. So we can kind of like continue
26:09
to fine-tune our system or when we increase the friction.
26:13
Did they like you know where they unable to login?
26:18
And it's a proxy because it doesn't necessarily tell us that
26:22
like oh that was a malicious login attempt but it's like a
26:26
useful signal in combination with other things that we can
26:29
kind of continue to improve this system over time by collecting
26:35
data. And kind of like observing what
26:37
happens in that middle ground state.
26:42
Hey, Lazlo here. If you're serious about ml Ops,
26:45
you hit subscribe right now. I want to change gears real fast
26:51
because you wrote an incredible blog post.
26:54
Recently on testing machine learning and testing models of
26:58
value. The difference between testing
27:00
models evaluating models and how to test different models and
27:04
machine learning. And so I want to dive into that
27:08
because I find this topic fascinating in its own, right?
27:13
And it is something that people have come on here to talk about
27:16
a few times and it also comes Zup quite a bit in the ml Ops
27:21
Community slack so much so that we created a whole Channel
27:24
around testing and it's not exactly like software testing
27:30
and you write all about this in the blog post can you just give
27:34
us a quick overview about the blog post and then we'll dive
27:37
into some specifics. Yeah definitely.
27:40
So this blog post was largely inspired by a paper that
27:46
co-worker had sent me on like behavioral testing.
27:48
And He models and reading that paper, I was like, you know, it
27:55
resonated a lot with me that I was reading through that was
27:58
like, yes, yes, yes. This, we should be doing this
28:02
and it kind of got me thinking in that area of, you know, okay.
28:07
Like what what is effective testing look like for machine
28:11
learning and as I was thinking through that, I was trying to,
28:15
you know, look at, you know, how do we test software where
28:18
they're like, the best? Practices that we've established
28:20
over Decades of developing software and which of those like
28:26
can map to testing machine learning models as well,
28:31
thinking about like, no, Andre cut by these.
28:33
Like, you know, software 2.0, like, you know, now, it's all
28:37
the logic is in these models. How do we test that logic and
28:42
do? So in a way that isn't just
28:45
like, oh, I made that accuracy. Your Precision number.
28:50
Go up. So, this is a good model of not
28:53
ship it. But we like, you know, how do we
28:56
actually get a deeper understanding of how, how that
28:59
model is performing in a variety of scenarios?
29:02
So obviously, I'm I feel like with all of the hype these days
29:09
around foundational bottles, it would be interesting to talk
29:12
about testing those foundational models, because it feels like
29:15
and this has happened before where we've been talking to
29:18
people about Tomatoes on the podcast and how there's such a
29:21
large surface area that things cover and we are learning as we
29:25
interact with them. And it's very hard to test those
29:30
unless, you know, something that I don't have.
29:33
You thought about how you can test these large language
29:36
models. yeah, it is tricky cuz when I think about testing, I am
29:45
often thinking about like what are the scenarios in the product
29:49
or in the end application that we want to like understand and
29:52
characterize that behavior. And there's not like really one
29:58
application of these Foundation models.
30:00
I think that's the whole point. It's like, okay we're we have
30:04
this Foundation model. This large language model that
30:06
can be applied for, you know, tons of different tasks and we
30:10
can amortize As the cost of training, this huge model over
30:15
many, many different applications.
30:17
So I do think it's tricky and don't have a magic answer on
30:22
like you know, how how do you necessarily like comprehensively
30:26
test that I've seen more papers coming out where it's just like
30:30
the answer is to have a bunch of different evaluations of like
30:35
enumerating. Here's all the different ways
30:37
that you could apply this model and you know data sets to To
30:42
evaluate those models on and I mean that seems like a
30:46
reasonable approach. It's better than you know just
30:49
looking at you know. What's the loss function for
30:53
that large language model like you trained on?
30:55
It's like you know giving you a better peek at you know what are
30:59
the downstream performance characteristics that you might
31:03
care about. But I think it's something that
31:06
we'll continue to, you know, figure out as these models are
31:11
applied in More and more settings.
31:13
Yeah and you did mention something that I also want to
31:17
just call out which I thought was fascinating when it comes to
31:21
the difference between model evaluation and model testing and
31:25
how the evaluation will cover metrics and plots which
31:29
summarize the performance on a validation test or on a
31:34
validation or test data set and then the model test involves
31:39
explicit checks for behaviors that.
31:41
That we would expect the model to follow.
31:45
So it's almost like an that instinctively makes me go to
31:50
like to monitoring right and monitoring it has become a very
31:55
big business in ml Ops and when you look at the differences
32:02
there do you feel like that should be an Like encompassed by
32:07
a monitoring tool or is that something that you're doing
32:10
beforehand and you're just making sure that when you're
32:13
creating this you go through the right checks.
32:16
Yeah. It's an interesting question.
32:18
I tend to think of like the model testing side is analogous
32:24
to like unit tests. Where, you know, you you have a
32:28
code base, say, software, 1.0, you have a code base and that
32:33
code base is evolving. We're making changes to It.
32:36
So you make a change that break something.
32:38
Ideally, there's a unit test that starts failing and and so
32:43
like you know you make your change, you put up a PR you see
32:47
the pr failed because it's unit test failing or maybe run the
32:50
test locally and that unit tells test kind of tells you exactly.
32:55
This is the part of the code that you changed that is now
32:59
like breaking logic that we care about and you can kind of get a
33:04
targeted way go back. To your code and update that to
33:07
make sure that, you know, we can make all those unit tests
33:10
passed. And as when I think can
33:12
sometimes get missed if we're like, now, looking at software
33:16
2.0 and we're just looking at evaluation metrics, like the
33:19
underlying logic that's baked into.
33:23
These models might be changing and the behavior can change in
33:27
some ways that, you know, might be hidden from just looking at
33:31
the the metrics themselves. So if there's like Behavioral X
33:36
that that we care about. I think it's important to be
33:39
able to like try to wrap that into a test.
33:41
So, when we knew now ship a new model update, like and that's
33:45
where I think the tests are very useful is like it.
33:48
Making that decision of like, okay, we train a new model, we
33:51
think it's better than the, you know, existing model.
33:54
But we want to understand, like, when we deploy this model, is it
33:58
like what's going to happen? So being able to try to like,
34:02
you know, take slices of that logic and Evaluate and like our
34:07
test that individually helps us get like a better picture of,
34:11
you know it's kind of like report card of like here's what
34:14
the model is good. At here's what the model is not
34:16
good at. One question I have here relates
34:22
to how you apply some of these ideas into your day-to-day, you
34:26
know, work and and in your, you know, in your day-to-day
34:29
practice shooting of machine learning, right?
34:32
You know, the paper that you referenced, you know, checklist
34:37
was it, I think testing system that's reference there.
34:39
You know, we now have Reckless. You know, there's all kinds of
34:43
new packages tools and suggestions that come from
34:47
Academia frequently. Here's the new thing you should
34:50
be doing, you know, for your system.
34:52
And I find it as a particular, very overwhelming, right?
34:54
It's hard enough to do the basics.
34:56
It's hard enough to have a good perspective on what's the
34:58
sensitivity and specificity I need for the task.
35:00
Let alone incorporate new kinds of sophisticated testing and
35:05
other sorts of practices that people are correctly.
35:07
Suggesting we do, how do you approach?
35:10
You know, making sure you're focusing on the fundamentals and
35:13
adding some of these newer more cutting-edge techniques in your
35:17
work in your day-to-day. I Duo.
35:20
Yeah, that's a great question. And I think the answer is like,
35:25
all those like really great suggestions.
35:28
Like, they don't always make it to what we're actually using.
35:34
And the that testing blog post was really interesting because a
35:40
lot of the blog posts I write to kind of like learn about a given
35:44
area, that one was a little unique in that like, you know, I
35:48
read that paper and it was really more of Like thinking,
35:51
like, how could this be done? You know, what would like an
35:56
ideal setup look like for like, controlling like having reliable
36:03
robust, rollouts of machine, learning models.
36:05
And like, what should we be doing?
36:08
But like, when I wrote that blog post, that's not what I was
36:11
doing at the time. I'm still not necessarily
36:13
following everything in that, it's been cool because I wrote
36:17
that blog post and then I think I, you know, had some people
36:22
reach out that like you know, hey this was really interesting,
36:25
like we're actually starting a company around this and it's
36:29
cool to like you know, put an idea out there in the world.
36:32
And you know it all like all the credit goes to their like papers
36:37
are the authors of that paper that just like got me thinking
36:40
about this and, you know, I just wrote down some thoughts and put
36:43
up pretty figures and, you know, shared and shared it around, but
36:48
it's cool to like for people to put About in the world is like
36:52
here's an idea of what things could look like in other people
36:55
kind of like take that and build the tools that are necessary to
36:59
actually be able to use that in practice.
37:03
It's like that saying the cobbler has no shoes or what's
37:07
that saying? Is it something like that?
37:09
The am I going to know? Is that a saying?
37:13
Only another language. The emperor has no clothes.
37:17
No, it's not that one. It's Well, whatever it's more
37:21
like do as I say, not as I do and I swear there was like the
37:26
Cobblers wife has no shoes or something.
37:30
I don't know. I'll look it up and get back to
37:32
you on that but it sounds like the the phrase like don't trust
37:37
a skinny Chef like yeah, along those.
37:40
Same lines. That is also a great one.
37:44
We found the saying no I swear the cop.
37:50
I know you wanted to ask Demetrios about Jeremy's writing
37:52
and if you like this is a good segue in considering we just
37:55
talked about this blog post. Take it away.
37:58
Yeah, exactly. And the Trope or the I just
38:02
Googled. It is the Cobblers children have
38:05
no shoes. It's an actual saying I did.
38:09
Not just make that up. It's been around for a while.
38:12
So to do you all understand what it means though, it's basically
38:17
like the cobbler who is a shoe? Ooh maker doesn't spend the time
38:22
and attention in his own household to make shoes for his
38:26
kids and kind of feels like that it's like yeah you you put that
38:31
blog post out there and it's very aspirational and ways but
38:34
then you're not doing everything but you see what is the
38:38
important piece around it? So segueing that into your other
38:43
blog posts and other writing that you do, how common is this?
38:47
Are you doing this often? Where you All right about stuff
38:51
just so that you can start to implement it or maybe you
38:54
recognize that. Hey, it's here.
38:56
Now I'm going to implement it into what I'm doing at work or
39:00
how does that process even look. Yeah, it's a good question.
39:04
And like I mentioned that, that that testing ml blog post is a
39:09
bit unique in that area. Like, most of the time I'm
39:12
writing a blog post to learn about a given area.
39:17
So I recently wrote went on tear for him because I like the
39:22
beginning of the year. I was like, you know, I need to
39:25
know this technology better. I use it a lot.
39:28
I do /, like rolling out infrastructure.
39:30
Now that I've learned it, but I use reading as a learning tool
39:35
and I think this started because like I started my journey into
39:40
machine learning after graduating, from from
39:43
University. I got my degree in Material
39:45
Science but that I was It's kind of like seeing all the cool
39:50
stuff happening. Machine learning and my Material
39:53
Science. Research was like, sort of
39:54
trending in that direction I started doing why like
39:56
computational research, like, simulating molecular structures
40:00
and, and observing different, you know, physical properties of
40:05
that. So I was like, turning in that
40:07
direction and I was like, machine learning.
40:09
This stuff, is way too cool. Like I want to be doing this.
40:11
So after I graduated I did a self study too kind of like
40:15
build up the foundation and like make my way into the Old and
40:20
because I was doing a soft study like there wasn't tests or
40:24
anything to check and make sure I was understanding things well
40:28
and I found it really useful to write a blog post and kind of
40:32
like force myself to teach myself like my audience.
40:38
Most of the time when I'm writing is like myself six
40:40
months ago and forcing myself to kind of like teach and capture
40:47
those moments like right after I learn them.
40:48
Them. Yeah, I think when you're
40:50
learning a given a new subject, it takes effort and you know,
40:55
it's like, doesn't always come right away.
40:58
And, you know, you're thinking on this or like trying to piece
41:00
things together and then usually like they're a couple insights
41:04
that help make things click. So I tried to capture that in my
41:08
writing as well because then I you know say you know I you know
41:14
don't use terraform for a while and you know my skills in that
41:18
atrophy. Can always come back to that
41:19
blog post and like very quickly, get back up to speed on on the
41:24
technology. So I use writing as a learning
41:28
tool and have found that when I force myself to write, I very
41:32
commonly will identify like gaps in my understanding like when
41:35
you have to try to teach something and make, you know, a
41:39
cohesive narrative about a given subject.
41:42
You realize like, oh wait, I don't understand this as well as
41:45
I thought I did and then you have to dig deeper and go.
41:48
Back and revisit that. I like the idea around it being
41:52
a checkpoint for you so that you always can come back to that be
41:56
like oh, yeah. Alright, cool.
41:57
That's where I was. Yeah, yeah.
42:00
And like some of the subjects I've written about like the
42:04
terraformed blog post. For example, there's so much
42:06
great content about terraform that.
42:10
I when I was first, you know, like deciding, okay.
42:14
I want to like, you know, really spend the time to know this
42:16
technology. Well, I wasn't necessarily
42:18
Planning to write a blog post, but I decided to not, and I was
42:24
thinking, you know, there's so much great content out there
42:27
like what's my blog post going to add.
42:30
But I I wrote one and I think the thing that adds is it's in
42:35
my voice. So I get the most benefit from
42:38
that is because you know, then two years.
42:40
Now, I can go read something in my own words and that's always
42:44
going to like make. It's always seemed like the most
42:48
efficient way. A to relearn something, if it's
42:51
in your own voice and like, you know, you frame the ideas in a
42:55
way that like makes sense to you.
42:58
Yeah, the way I think about it is, you know, I use writing very
43:01
similarly and it's kind of like the Amazon mindset of be the
43:04
first and best customer for your own product, right?
43:06
Like that's what a tab, that's what I was.
43:08
Almost radio us. The first and best customer for,
43:10
you know, distributed computing and all the cloud services and
43:13
it's the same way with great writing.
43:16
So I'm just a regular Jeffrey business over here.
43:20
What I wanted to ask is you know, you have these great blog
43:24
posts, you know which I have been reading for going on two
43:27
years now. I'm kind of curious.
43:29
Where do you post them? How do you go about getting the
43:32
word out? I just maybe you do a lot, maybe
43:34
you don't just curious. Yeah, so I post them, I like
43:40
host my own blog. It's powered by ghosts which is
43:44
an open source blogging platform started by some some developers
43:49
at WordPress like left and started ghost.
43:53
So I use ghost and I host it myself which I like because that
43:57
gives me the most like control, like over how that information
44:02
is presented. Like, when I was starting medium
44:05
was just entering The scene and a lot of people are publishing
44:08
on there, but I'm knowing what medium has turned into it.
44:15
Like, you know, there's this flood of content especially a
44:19
lot written by beginners. I'm glad that I know decided to
44:25
just start my own blog and it's you dodged, you know.
44:28
Yeah. And now I have like my little
44:32
corner of the internet that I can just, you know, publishing
44:35
control what that looks like. Like, I think there is a second
44:39
part to your question, though. In terms of just where you
44:43
distribute, the content is it. Oh yeah.
44:45
LinkedIn Twitter. You active on those places kind
44:48
of curious. Yeah.
44:50
So usually, I will just like right up a blog post and then
44:54
tweet a link out like, hey, you know, I've been learning about X
44:58
and I wrote a blog post on it and, you know, sometimes I get
45:03
share it around. I also like Do get a fair amount
45:07
of organic traffic to my site just like people do Google
45:11
searches. It can be cool.
45:13
Sometimes like my old roommate was going through the Georgia
45:17
Tech online Master's program in machine learning and who was
45:20
learning about, I don't know, like variational autoencoders
45:23
and so he was just like Googling it to, you know, do his homework
45:27
and like learn about it. And my blog post, like, showed
45:30
up, it was like one of the top results.
45:32
Like wait, I live with that guy. So it can be cool like to hear
45:38
about those scenarios where I guys, I think actually, one of
45:40
my other friends sent me a screenshot because he was
45:44
Googling variation autoencoders. And my blog post like was
45:48
occurring above the Wikipedia page on variational encoders,
45:53
here's like, wow, this is so cool.
45:55
It's like yeah, that's awesome. So it's a mix of just like
45:59
tweeting out on link once I published as well as people are.
46:03
Googling this stuff all the time and like I get it.
46:06
Getting traffic from that and you know it is valuable.
46:09
If you didn't even have to do any SEO or anything specifically
46:13
and it is landing above the Wikipedia page.
46:17
I think, I think ghosts does some SEO under the scenes.
46:21
I don't know about but yeah, I'm not I'm not like doing any of
46:26
that. That's incredible.
46:29
That was just going to ask a very quick question, very quick
46:31
question. Which is what has been your most
46:33
popular blog post? It, has it been to beat the
46:35
variation? There's one like which the one
46:37
that got the most traffic. That's a good question.
46:40
I saw I have analytics on the site but I like very rarely
46:44
actually look at them, so I'm not sure I would guess.
46:51
The variation. Autoencoders one is probably up
46:53
there. Now I'm kind of curious I'm
46:57
probably probably look follow through Google analytics and
47:00
see, you're leaving all kinds of advertising revenue on the
47:03
table, man. You could just throw some banner
47:06
ads on there Like It's 1999 and get all kinds of.
47:11
Yeah, yeah. But like I get a ton of value
47:15
out of that block already and I just from for me like the Of
47:21
writing and learning. But also all the people that I
47:23
meet that like, you know, come across my blog and reach out,
47:27
you know, I think that's like how we got connected.
47:30
So I get a ton of value already from it and I like keeping it
47:35
ad-free, I was actually. So I used ad blockers on on like
47:41
Chrome or whatever. Yeah, do motionless I think.
47:45
Yeah. But I don't have out blockers on
47:48
my phone and there is one time I'm when I was like, scrolling
47:52
through one of my blog posts, and I went down to the bottom
47:54
where like, I have a discus plugin for comments.
47:57
And I saw that they were inserting ads, like above the
48:01
comment section. And I was like, so embarrassed
48:04
because I didn't know like, in my browser, you know, I didn't
48:08
see any of those ads and like, of cool, just, I don't know.
48:11
It feels interior to have those ads being shown.
48:15
And so I figured out there was a way to like disable it, but it
48:19
was like nested behind Three different panels within
48:22
discusses settings. Oh man.
48:25
Well yeah. I think it's been what like I
48:29
reached out to you in 2020 to try and get your on the Pod and
48:32
you were like I don't have anything to talk about and then
48:35
I kept seeing you put out these blog posts and I'm like, come on
48:38
man. We'll just talk about your blog
48:40
doesn't matter. Like I'm sure you have a lot to
48:42
say and you've been learning about a bunch of cool stuff.
48:45
And finally, now I'm going to Pat myself on the back because
48:48
it was a lot of persistence on my side, it was Was far too much
48:52
persistence, but I found out that this is your first podcast.
48:57
I know in my heart that it is not going to be your last
49:01
Jeremy. It's been awesome, getting to
49:04
interview you and getting to learn from you.
49:06
It's so cool to see what you're doing.
49:10
How you're doing it? And I love the work that you're
49:12
doing and putting out there. So keep it up, man.
49:15
And then everybody that is listening, please go and sign up
49:19
for normal comp. If it is free, there's no
49:22
excuse. Not to it is happening next
49:25
week. I think no to we have to cember
49:28
15th, December 15. That means we gotta do some fast
49:32
turnaround on this episode that we're editing because it was
49:36
going to take longer than two weeks but we're going to make
49:39
sure that people can hear this by next week so that we can get
49:44
this out and everyone can know about normal Kampf.
49:48
Thanks dude, this was cool. Thanks for having me on.
49:51
I enjoyed the conversation and yeah, great.
49:54
Great to for this podcast to be my first podcast experience.
49:59
We are thrilled ago. We finally got to interview.
50:02
Thanks for joining Jeremy. Yeah, it's hopefully we did not
50:05
let you down and it was all that you were expecting it to be a.
50:09
I know you were a little nervous because you wanted to prep and
50:12
podcast. You don't need to prep.
50:14
That's the beauty of it. You just get to chat with us?
50:17
Yeah. Well it was a great
50:18
conversation. I enjoyed it.
50:20
Right on, man. Right on, man.