Latest / Agentic Conversations (formally mlops.community) / The Challenges of Deploying (many!) ML Models // Jason McCampbell // MLOps Podcast #149

Transcript

The Challenges of Deploying (many!) ML Models // Jason McCampbell // MLOps Podcast #149 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm Jason mccampbell. I am the Chief Architect at
0:03
Wallaroo. And I start my morning with a
0:06
cup of coffee every day. A little bit of cream, or little
0:09
bit of, maybe coconut milk. Almond milk in it.
0:14
Welcome back to another envelopes Community podcast.
0:18
I'm Demetrios your host. And today, I am joined, by none
0:22
other than Abby. What's going on a be fantastic
0:27
and not any meetings today, but Tomorrow is like back-to-back
0:32
meeting but I'm glad we got this conversation today because I
0:36
will take a few minutes to wrap my head around everything that
0:39
we serve learnt or so cool. Go very different perspective
0:44
on. Oh, so true.
0:46
I am going to go into this with the disclaimer that I am a
0:52
complete and utter Noom when it comes to deploying machine
0:58
learning on the edge. Or doing any kind of Ops doing.
1:02
Anything on the edge, will just say that.
1:05
I knew very little about it, and Jason schooled us so hard.
1:10
It was so cool to listen, to some of these challenges that
1:14
you have to think about, when you are deploying machine
1:18
learning at the edge. And I want to hear your key
1:21
takeaways first and then, I'll give you what I thought were my
1:24
highlights. Porto's in is device, tree
1:28
living for me, it was always our trainers and, you know, to lower
1:33
level devices, also, grew more. So because have been a part of
1:36
like the risk fire community. And most of the discussions over
1:41
there are compilers him to be able to break it down for us and
1:46
talk about the economics of operations of, like, how do you
1:51
take the data in one place and how do you reduce the The
1:55
deployment costs by should be going XYZ City and if so what
2:01
would be the cost of managing a team over their death is so
2:05
fascinating for me because that's a perspective will never
2:07
heard before. I was very new to this kind of
2:11
conversation, these kind of challenges as well.
2:15
Oh my God the deployment cost was a huge one for me too and
2:18
especially when he talked about the different vectors that you
2:21
need to keep in mind. How much data are you taking in
2:24
how fast does it mean to be? And it made me realize how much
2:29
we take for granted when we are doing anything with a cloud and
2:34
the standard way that we're doing machine learning for most
2:38
people is that and there is just so much that you do not need to
2:42
think about whether that is or like you were Saying, how heavy
2:48
is the model that you're putting out there?
2:50
What kind of security are you looking at?
2:52
And all of this there just brings out so many more
2:57
vulnerabilities. And speaking of security, I've
3:00
Loved when he got into the different ways that you can have
3:04
problems. When deploying on the edge, when
3:07
it comes to security and just basically sending data around
3:11
and making sure that are you doing it Federated, are you?
3:15
You just keeping it everything on there or liked what he, what
3:19
did he say? It was like, are you creating a
3:23
one way street? So that there's only one place
3:27
one device that all of the data can go to, and you have
3:32
everything else firewalled, I was really appreciative of how
3:36
he just broke all of that down, and then the other piece that I
3:40
loved and we both mentioned this after we hit and recording.
3:45
Around how we never thought of doing anything on the edge as
3:50
more than just having a few and Reno devices and playing around
3:54
like toys, maybe making some robots.
3:57
And pretending like we work for Boston Dynamics, but this really
4:01
like this is some crazy stuff that he was talking about when
4:04
it comes to 5G and the towers and just like the constraints
4:10
that you have to think about are so much different in the
4:13
challenges are really. Exciting.
4:15
And so hopefully everybody enjoys this.
4:17
It's something that was super new and different for me, I
4:21
imagine for those who play in this and live their day-to-day
4:26
with doing AI on the edge or ML on the edge or just ml Ops on
4:30
the edge, whatever you want to call it, whatever your flavor of
4:34
naming convention, you like, aha, it's probably not going to
4:38
be the most in-depth at Jason is incredible.
4:41
He was able to go super in-depth with while still keeping it.
4:45
Very very understandable for us that know, very little about it
4:51
and I think that everybody's going to get something out of
4:54
it. Unless you are working day in
4:58
day out on this, then you may or may not know this kind of stuff
5:02
and they had a lot of people in the community to talk about
5:05
fidgeted lightning but this would be the first time he
5:10
leased for a lot of people who work as to assign this to be
5:14
able all to think about how to or what kind of application does
5:20
retreated, learning actually happen machine learning thing
5:23
for any sort of em adopts Engineers to be thinking about
5:26
the new challenges as well, and the entire scope of the field.
5:32
But level can be go to, how much can we break down either
5:37
problems and stuff? So, it will give everybody.
5:40
I think a very different perspective as well.
5:43
So I will Will encourage everyone if you are out there
5:47
and you like this and you enjoy the ml ABS at the edge, we have
5:52
a whole channel in our ml Ops Community, slack dedicated to
5:56
this. It has been a little bit
5:58
inactive for the last six months so go in there.
6:02
Give it a little love. See if everybody still about and
6:05
say hi, maybe you can spark up a little bit more interest and
6:12
Conversation engagement around that and before we jump to the
6:17
full conversation, I wanted to mention that Wallaroo was kind
6:20
enough to sponsor us for this episode and I've got all kinds
6:24
of cool call to actions for you from them.
6:27
They have a free Community Edition, that we are going to
6:30
leave a link to in the description and they also have a
6:33
simulated Edge AI tutorial. So if you want to go deeper into
6:38
this and you like what you're hearing, go check that out.
6:41
Out and huge. Thanks to Wallaroo.
6:45
Go. If you're doing stuff at the
6:46
edge, like, you might as well check them out because they are
6:50
on it. Jason I feel in very good hands.
6:54
If Jason is taking care of me and he's the architect behind
6:57
the water of platforms. So let's jump into this
7:01
conversation, like And subscribe and all that fun stuff and leave
7:03
us a review. Mr. Jason I want to know about
7:13
your background because you came from the semiconductor world.
7:16
Not exactly ml world. But I know there are a few
7:20
parallels. Can you give us a bit of an
7:23
idea? How you broke into this field?
7:27
Well, so back when I was in school.
7:30
Well, alright. So I guess I'm dating myself
7:32
here. When I took AI in school, that
7:36
was optimization, algorithms, that was Brute Force.
7:40
Search, it was rule based systems.
7:41
There was no deep learning neural Nets, were these tiny
7:45
little toy things and it was very unsatisfying.
7:52
Those techniques work really well.
7:53
If you can build a mathematical model of something So if you're
7:58
talking about navigating from point A to point B, I can make a
8:01
mathematical model that says, I'm optimizing for distance or
8:04
for travel time, or energy usage.
8:08
But I, you know, mathematical model of this is a cat or this
8:11
is the word hello. So I was dissatisfied.
8:16
I was young let down but I have always liked automating things
8:20
and I like to look tronic. So I went into semiconductor
8:22
design and it turns out those AI techniques work really well.
8:27
Well, because you can take a collection of transistors and
8:29
say what, here's a mathematical description of it.
8:32
It's an and gate or it's an or gate, and here's the timing
8:35
properties, and so on. So that's great.
8:38
We can synthesize a billion Gates into a chip and we can run
8:42
all these nice algorithms on it, but I've always wanted to keep
8:46
one foot in that. I've always been attracted by
8:48
nights, I followed the industry and I had needed to get back to
8:52
startups. And when the Wallaroo
8:54
opportunity came along Man, I jumped at it and from that
9:00
standpoint. It's all high performance
9:03
Computing. Its how do you take an algorithm
9:05
whether it's in ml model or, you know, a semiconductor
9:10
optimization algorithm. How do you run it on a lot of
9:14
data using as Little Hardware as you can or maximize the
9:18
hardware. So, you know, there are
9:20
parallels there, but it's been sort of a roundabout trip.
9:26
H fisting and how to go about you know building this expertise
9:32
and combining Edge and emerald together or were those some
9:37
sterk common things and some of the common challenges between
9:43
each So I guess inch is always been fascinating to me.
9:51
Like, even as a kid, I always wanted to have a computer where
9:54
I could talk to it and answer questions and my kids, you know,
10:00
they're used to having voices in assistance around.
10:03
They can ask the weather. That was complete sci-fi when I
10:06
was a kid or having the robot clean, the floor stuff like
10:10
that. So that's all sort of the fun
10:12
Edge types of things. And then we had We've had
10:16
clients that have come to us and said, okay, this is great, we're
10:19
deploying models. Hey, but we also have to deploy
10:21
over here. Can you help us deploy their and
10:24
was one of those of like, well, that sounds interesting.
10:27
Yeah, let's talk that. Look, tell me more about it.
10:30
What are you doing on? And there's such a huge range.
10:33
It's really fascinating. This thing that brings me to the
10:37
next question, which is, what do you consider edge of you say,
10:43
you know, there are so many devices out there, how do you
10:45
define it? So that, yeah, that's a good
10:49
question, that's a huge range in.
10:50
It depends on who you're talking to.
10:53
I've had the privilege of talking to people are doing
10:57
everything from microcontrollers, where you
11:00
have, you know, 12 a couple megabytes of memory on them.
11:06
Up to more desktop-class processors are more x86 22,
11:11
small kubernetes clusters where you have gpus and you're
11:14
processing. A lot of video footage to even
11:17
well, we're deploying in a data center but the data centers
11:20
halfway around the world and we don't have the normal access to
11:25
it. So to be able to set the stage
11:30
microcontrollers, require a whole different set of tooling
11:34
than Cloud deployments and the rest.
11:37
So, you know, that's tiny ml domain and I tend not to get
11:43
down to that level. Usually what I look at is say
11:46
raspberry plot pie class devices.
11:49
You know, a couple hundred megabytes gigabytes of memory up
11:53
to clusters of CPU servers. Say if you have a retail outlet
11:59
in your trying to capture metrics on the number of people
12:01
within the checkout Q, you may have a couple of servers in the
12:05
back. Groom with GPU cards, monitoring
12:08
the cameras? All the way up to.
12:12
Hey, I need to deploy these models in China in Asia, in
12:19
Africa and Europe. And On top of that, I can't
12:23
bring the data back because it's personal information or is IP
12:27
protected or something like that.
12:28
So you end up having an edge location.
12:32
That's a 10-cent cluster in China.
12:37
But legislation in the government rules, say that's
12:42
personal information. You can't leave this
12:43
jurisdiction. So then you and segmenting your
12:46
your network and your deployments around the world,
12:48
even if it's a data center, it still ends up looking like an
12:51
edge deployment. Oh well.
12:53
So this reminds me of back in the day when I used to be in
13:00
sales and I talked to a guy who is working for a health company
13:05
and he Was asking us. If our tool back in that day was
13:09
thought science, the now-defunct science or IP that company and
13:14
he was asking you know we are all over the globe and one of
13:19
the things that I'm having a particularly hard time with when
13:23
it comes to our data platform. Is that data from Canada cannot
13:29
leave Canada. So if we have some data
13:30
scientists were trying to create a model of machine learning
13:35
model but they're based in the UK, they cannot touch any of the
13:39
Canadian data and I imagine that's what you're saying.
13:43
When you talk about like 10 second, Cloud being considered
13:47
Edge, Absolutely, yes. Yes, if you're talking about
13:52
health information, if you're talking about personally,
13:55
identifiable information, it, you know, he could even be for a
13:59
website or a game type of experience, but if you get down
14:02
into some of these details about a person, you either you can't
14:08
send it out of the jurisdiction or you don't even want to go
14:10
through the legal headaches, of what data can I transport.
14:14
So what's in Canada? Stays in Canada, was in China.
14:18
Stay He's in China and you do all the processing there but
14:23
then what are the real interesting bits is?
14:25
So how do you know if your model is working?
14:27
Well, how do you monitor it without being able to bring the
14:31
data back, huh? Well this is yeah, this is
14:36
something fascinating because I would just consider that like
14:41
there's a specific unit or part of the company that works on
14:48
that. And they are privy to that
14:49
information. I wouldn't consider it like an
14:53
edge deployment where it's almost like.
14:56
When I think about it Edge, I think about being a little bit
14:59
removed as opposed to being inside and being able to work
15:05
from it. From any thing that you need
15:09
when you're in Canada and you can launch, and you can do
15:12
everything inside of Canada as opposed to doing it remotely
15:17
from the UK, or somewhere else. Else.
15:19
But I am going to preface this with.
15:22
I honestly have no experience with Edge Computing, or how the,
15:27
the laws work around this personal identifiable
15:30
information. And so, I'm excited for you to
15:34
enlighten me real fast. Well, so that is certainly a way
15:39
you could do it. If you, if the number of
15:41
companies or countries rather or jurisdictions is small enough,
15:46
you may certainly have a team in each of those those in those
15:50
locations. Yeah.
15:51
Okay. If you're talking about every
15:55
different country that you're serving, you can get to the
15:58
point, where will I have a team of five data scientists in and
16:03
Ops folks? And I have to serve the entire
16:07
country or the entire world, rather, you know, it doesn't
16:11
scale, but a flip side of that would be Okay, I'm want to
16:16
deploy my models in to say a science lab into a research lab,
16:20
and there may be a rack of servers.
16:24
There you have plenty of compute capacity and if I'm publishing a
16:28
model that runs on a microscope or some other instrument, I'd
16:34
like to see that observability data back, but the researchers
16:38
May well be saying there's no way you're sending data about my
16:43
research off to your Cloud. That's my IP.
16:48
We have strict security requirements here.
16:51
So, again, it has to stay in that.
16:53
The same thing would be true of an advanced Factory or
16:56
semiconductor Fab or any of these types of places where you
16:59
have really valuable IP, and you have third party, Models coming
17:04
in and you really want to be able to run those models, but
17:09
the data scientist would like to get the data back to.
17:12
So that's an interesting challenge that that we run into
17:16
is okay. So how do you enable that this
17:18
also makes me think you have like this decentralized teams or
17:23
on the place my question would be how do is consolidate the
17:27
information or the learnings from let's say one of the teens
17:30
in one place to another place because let's say your training
17:35
motto on a deer for like a particular, Country or a
17:42
particular company where you can take the data out.
17:45
Let's say you were serving some other client on a very similar
17:48
use case in certain other country.
17:50
How do you consolidate that information from this place to
17:53
that or do you or can you it's a tough problem?
17:57
I tend to think. Yes we can it's something we're
18:00
that I'm actively looking on that all of us are working on
18:03
right now and there's parallels to how do you train data if you
18:08
can't bring it out. So there's techniques for
18:11
Federated learning where you can push the algorithms out to it.
18:17
Maybe the edge it maybe data, center edge, type of cases, do a
18:22
partial training, bring the weights or the other information
18:25
back. Do some reconciliation and you
18:27
do multiple round trips of that. So you can now train the model
18:34
without having to aggregate all of the data back to a central
18:37
location. So what I'm really interested in
18:40
is how do you do the equivalent? Lent of a Federated
18:42
observability, where you want to do drift analysis you?
18:47
We want to do, anomaly detection and all this.
18:49
So can you push these algorithms out to the edge saying, to the
18:55
research lab statistically, summarize the data and bring
18:59
those summaries back such that you're not revealing, any
19:01
specific IP but in enough detail where you could look at multiple
19:06
deployments of this model? And aggregate the results.
19:10
So now you can do things like even something like cameras in a
19:16
retail store or traffic cameras or something, and you can look
19:20
at the performance of the model in aggregate.
19:23
You could zoom in say let me just say take this one
19:25
intersection or this one store or even let me look at this one
19:29
camera. Well this one camera the
19:33
classification rate dropped by 20% why?
19:36
Oh well somebody stuck a Valentine's display.
19:38
Up and there's a red light shining into the camera and they
19:42
can't see anything now. And so you can start doing all
19:46
of these neat analytics. If you can bring at least some
19:50
of the results back and sometimes they may not be
19:53
sensitive, you can bring everything back.
19:54
Sometimes you don't have the bandwidth, so you bring back
19:57
samples of it. Sometimes you statistically,
19:59
summarize it, and bring the results back.
20:01
But so you can keep learning and optimizing and then push out
20:05
improved models. I've got a real anyone from when
20:10
we had a meet up in San Francisco and I cannot for the
20:13
life of me, remember the company that one of these guys that I
20:16
talk to is working for, but the whole concept was around Cameras
20:23
being in a store like your typical supermarket and then you
20:29
get charged. When you pick things up and it
20:34
automatically gets sent to your your account, the bill, as you
20:40
leave, it gets sent to your account and then you know, all
20:43
right. Cool.
20:44
I picked up these five items and you don't have to do any check
20:47
out nothing like that. And I just remember one of the
20:52
adversarial. Attacks that happened was,
20:54
people would put their coat jacket above their chest or like
21:01
out over the line of sight to whatever they were picking up,
21:05
and then they would pick something up.
21:07
So that the camera, never detected that they actually got
21:10
it. And then they would just put it
21:11
into their pocket and then leave.
21:14
So he had to figure out how to properly defend against that.
21:20
And I think it was something along the lines of checking to
21:22
make. Sure, if none of the stock had
21:27
been depleted also, and so that is another one of those common
21:31
things where you would like to see what exactly is going on and
21:35
why are we losing money? If we are.
21:37
And but the main question coming back to the conversation at
21:41
hand, what I was thinking about when you were talking about,
21:45
this is your doing all of this. Retroactively or can you do it
21:52
also in real time and just have a little bit more little bit
21:57
larger of a window with that real time window that we talked
22:00
about like it's still batch but it's very, very fast like that.
22:04
Well, you can certainly do it with streaming so you can do it
22:08
near real time. But y'all the limitation is one.
22:12
Do you have a network connection, big enough to send
22:14
it back? So can you get the images back
22:18
or can you get It Whatever the inference inputs are and the
22:21
results. And then the other issue is,
22:24
unless you have ground truth data, which you probably don't
22:29
because it's you had ground truth results.
22:31
You wouldn't need an inference model.
22:32
You just know the answer. So then the observability is
22:37
based on how does it compare to Prior baselines in looking at,
22:43
as at a statistical distribution and you can say, well, my
22:48
classic Asian rate of Shoppers in line or of cars, or cats
22:55
versus dogs in my images has dropped, or it has shifted.
23:00
And then you say, well, now there's something I need to go
23:02
investigate, but you would that sort of case would be harder to
23:06
pick up what you'd probably pick up is And I have Shoppers going
23:11
through and they're buying less, or we, I have some who are
23:14
buying very little and now, I'm seeing a deviation in what
23:18
people pay for versus what my inventory status over.
23:22
And now you need to dig in and say, what on Earth is going on.
23:26
Oh, I get it. Okay, so watch it for showers
23:30
and footage. Yeah.
23:32
And you start to see the patterns Yes.
23:35
Yes. Or how many people did something
23:37
and they all they didn't pick something up.
23:39
Wait a minute. Is that right?
23:40
That was a lot of people who pick stuff up and and didn't
23:43
they actually put it in their cart?
23:44
Oh, they put it as a note and yeah, that's a cool attack by
23:48
the way? Yeah, it is, of course you're
23:50
gonna you gotta figure that every which way it can be
23:54
hacked. People are going to try Yes, in
23:58
the answer can't be we'll just put more cameras so you can see
24:01
it from all the different angles because then your cost is go off
24:04
and it's not worth it right, you work.
24:06
But they're coming from something which is basically a
24:08
few GB which is raspberry to like a couple of CPU servers.
24:13
Think you have such varied devices in such varied levels of
24:19
the devices that you working with.
24:20
How do you maintain statistical? If you tried to annotate of Dia
24:25
in these Federated, It works for is everything being collected on
24:30
the same kind of devices and then being compiled together.
24:33
Oh, so that's that's a very interesting question.
24:37
It depends a lot on the application.
24:41
In my experience, usually, the variation is across applications
24:46
so you may be deploying a given model on something like
24:50
Raspberry Pi class. Hardware, you're probably using
24:53
similar hardware for that model across the fleet of them, but
24:59
then you may have a different application where you're doing
25:02
video processing and it's running on large x86 processor
25:08
with a GPU. And that application now needs
25:12
to be deployed there for that that level of processing.
25:17
But that means that the Ops teams have to then deal with it.
25:21
So you know again it's going to depend on the application but
25:25
you're bringing back the standard sort of metrics of CPU
25:28
time, memory usage in and so on plus you're bringing back maybe
25:34
the full observability stream, maybe your sampling 1% or some
25:38
fraction of a percent of It, maybe even want to adjust that
25:43
observability in specific cases where you see something starting
25:46
to go wonky and you say well, you know, let's hope that to 10%
25:50
in the end. Let's get a few more samples for
25:52
this particular model. She is the queen of emeralds
26:00
that work and this is your daily dose of analogs.
26:05
Well, if you are dead serious, about the ml Ops you have to
26:10
Immerse yourself in the emeralds content, the best way to do it
26:15
is to subscribe to the email Ops Community podcast.
26:19
All great people are here on this podcast soon, good luck and
26:24
keep learning. So, speaking of the Ops teams,
26:35
and this whole way that you're bringing it about When it comes
26:43
to architecture of ML on the edge or ml Ops at the edge,
26:49
whatever you want to call it, how do things differ?
26:53
Because I can only imagine it's a lot more complicated.
27:00
Yes. So particularly for the Ops
27:03
folks Because now, of course you're dealing with limited
27:08
network connectivity. Certainly some cases you have no
27:10
network connectivity but in that case, you're baking the model of
27:14
a software, everything in, and it's in a firmware update and
27:19
it's just leave it alone. And that's great for some
27:21
applications. But ml models, AI models are
27:25
different beasts and software. They're really a function of the
27:30
data. They were trained on.
27:32
And we're asking these models to Tease out some very subtle
27:36
patterns in the data. So there's a lot of background
27:39
noise and relatively smoke signal.
27:41
And sometimes those patterns are going to be very consistent over
27:47
time. If you're running a model
27:49
detecting, some aspect of a jet engine, they've characterize
27:52
that jet engine Every Which Way and it's going to be pretty
27:56
consistent. But if you're talking about
27:57
teasing out patterns of behavior for humans and traffic in all
28:03
sort of the real world, those patterns shift over time and
28:07
sometimes the background patterns that you're trying to
28:09
discriminate change over time. So that the models have to be
28:12
very dynamic. So one of the interesting
28:16
aspects to me for Edge is we already want to do this in Cloud
28:21
deployment, we want to be able to deploy the model continue to
28:25
monitor it redeploy and keep updating so we can optimize the
28:28
behavior and keep it performing. So how do we do that at the
28:33
edge? Which means how do we enable
28:37
relatively rapid update? So the models not necessarily
28:41
with the software but how do we push those out there and And how
28:45
do we bring the data back? So that the data scientists on
28:49
the team? Can analyze the models and
28:52
understand if they're still performing.
28:53
They know whether do do that. Retraining.
28:57
And it's interesting because there's actually a lot of
28:58
parallels to software and that yo old DVD players.
29:04
They're not connected to the internet there.
29:07
You've never update the software.
29:09
The inputs are fixed set of buttons, the remote control with
29:13
the discs, and that's about it. It.
29:14
But as soon as you take software and stick it on the internet,
29:17
you have all of these different changing patterns, some of the
29:20
malicious and those malicious attacks are changing over time
29:24
and you'd better have a very Dynamic method of updating your
29:27
software because you will be getting hammered from different
29:30
ways. I mean, there's probably not
29:32
completely robust and models of the same, whether it's just
29:36
natural changing patterns of cell phone use on a cell phone
29:39
tower, or actually, malicious attacks.
29:42
Where people are discovering new ways of Yo, grabbing something
29:45
off the shelf and walking out of the store or turning on your
29:50
toaster. That's hooked up to the Internet
29:52
and we're turning on your whole as smart home as I saw I think
29:57
in one she's case where somebody went on vacation for a weekend
30:03
and then they came back and they're wholesome smart home was
30:05
taken over by by somebody and can't remember the exact
30:09
specifics of that story but it was something around those lines
30:12
so I can totally see. That and it leads me to the next
30:17
question is, how are you looking at as a specifically?
30:20
Like there's the adversarial attacks or there's attacking the
30:24
machine learning models themselves.
30:27
But then how are you looking at security in these edge cases to
30:33
make sure that not only is the model doing everything that it
30:36
should be doing. But then taking it from the Ops
30:39
perspective and making sure that everything is going as it should
30:43
be going. Yeah, so security is really
30:47
fascinating to me. I don't dig into that very much
30:52
from my perspective, that's deployed the models and try to
30:55
get the data back but then of course there's the piece of how
30:57
do you get the data back securely it?
31:03
That's an interesting challenge because it's going to.
31:06
It depends a lot on the application of how exposed is it
31:10
to the internet? Is it completely exposed like
31:13
it's designed? To interact with it or is it
31:16
something where you could set up something like an overlay
31:19
Network where all of the software systems on that edge,
31:23
device can only see one target back at the cloud infrastructure
31:30
and Cloud all of the data back there and you don't let anything
31:32
else into the firewall. Can people play with it?
31:37
You know, if it's something like a voice assistant in my house
31:41
it's not getting attacked too much except you know my kids.
31:44
He's banging on it, but or something gets into my internal
31:47
network. But if it's something out, you
31:49
know, in a store where people can start, now poking at the
31:53
Wi-Fi connection and everything, you have to deal with all a
31:56
whole lot more headaches. One of the challenges that our
31:59
to is, if you have models running on an edge device from
32:05
different sources, Say you have a scientific instrument in the
32:09
operator changes a setting. We want to build a swap to a
32:12
different model, but if those models come from different
32:16
sources because we're starting to see that as a trend where you
32:21
don't want to make the model, you want to open it up to
32:23
competition. Will now, you also have to make
32:25
sure that what the model is, observing and sending back home,
32:31
had better go back to its actual home and not somebody else.
32:37
Right? You don't you know if my company
32:39
makes a model, I don't want my observability data going to my
32:42
competitor for them to see. So now you end up having to
32:45
bacon to the models will, what is my what is my return
32:48
destination? Where what am I allowed to send
32:50
back and where do I send that and do it securely.
32:54
There is one thing that I wanted to ask real fast about baking or
32:58
not baking. We had about a year ago, Luke
33:02
Marsden on the MLS Community meet up and she talked about the
33:07
Was in cons between baking models into your containers.
33:10
Sounds like just about everything you're doing your
33:12
baking them in, we can but actually my preference is to not
33:16
bake them in. It would be more you leave the
33:19
container running and when you need you hot swap the model out
33:23
and you let the software keep going because the last thing you
33:26
want to do is disturb the Software System that's running
33:28
on it. Yo one of things I've looked a
33:31
lot at is 5G cell towers so you they'll have something like the
33:37
one type of towers that we've been looking at have a 10
33:39
millisecond timing Loop. Whether optimizing some acts
33:43
some aspect of how the spectrum is divided up among the
33:48
different clients on the network.
33:50
And you may have a model that's optimizing for ultra-low
33:54
latency. Or you may want to have a model
33:57
optimizing for streaming, throughput for video types of
33:59
things and you want to be able to swap those models in
34:04
relatively short timeframe. But you don't want to disturb
34:09
the timing Loop and tear down the whole container and redeploy
34:11
and everything. You just want to hear what
34:14
switch I'm optimizing for streaming now and just keep on
34:17
running true. So let's, let's come back to
34:21
your work at while true. I'll try a bit because I that's
34:24
something else. I'm sorry, so, fascinated by
34:27
which is you are working with a lot of use cases from cyber
34:32
security to iot to real estate to manufacturing to Ed Tech and
34:36
it's like that is a lot of Wait, or that is a lot of bread that
34:43
you are working. Are the observability challenges
34:47
saying in all of these fields, or how are they different,
34:52
especially like when you're working on these others?
34:58
Yeah, so so good question. Is he right?
35:01
We It. I love it because I get to touch
35:04
a lot of different sort of use cases and actually reminds me of
35:09
the old ads for BASF, we don't make whatever we make it better.
35:17
There are no chemicals company. I believe so My job here, my job
35:24
at Wallaroo and wohlers job. In general is not to teach the
35:29
domain to any company, the data scientists know, their real
35:34
estate domain or they know oil exploration or they know any of
35:38
these sort of aspects Our job is to provide, the plumbing is what
35:44
our CEO says we're plumbers. The idea is, how do we enable
35:50
the Ops folks in the data scientist to deliver more value
35:55
by automating the lower level? The Redundant pieces, how do we
36:00
take this model and deploy it in a production form that has load
36:04
balancing? It has replication and so on,
36:08
it's highly available or so, it's packaged up and you can
36:11
deploy in a relatively A small piece of resources and then how
36:15
do we provide the tools that let the experts in their domain?
36:21
Understand how the models performing?
36:22
So then what it comes down to is the big differences.
36:26
Are you talking about structured data?
36:28
You know, if you're talking about predicting a house price
36:32
or something yelled, and that's I bring that up because that's a
36:37
demo, that's that we have in the free edition and you can see
36:41
this The house price predictions for the Boston area and the
36:46
trend over time. And you can see shifts in it and
36:49
it's a rather old model. Yeah, it's laughable.
36:53
It's a, Well, if it's over a million dollars for a house,
36:55
would pay a million dollars for a house in Boston.
36:58
Well, okay it's hard to find a house less than a million
37:02
dollars in the Boston area. I suspect but dad's a structured
37:07
continuous variable. If you're looking at
37:09
classifiers, you can do the same.
37:12
I'm paying and whether they classification is images.
37:17
Looking at vehicles or images, looking at Shoppers or defects
37:21
in parts. Coming off of assembly line, you
37:24
can end up the experts that can come up with the criteria that
37:28
they're interested in monitoring for their domain.
37:32
And same thing with NLP models, you can look at what are the
37:36
results here. What are my metrics that I care
37:39
about? And now let's monitor that for
37:42
And let's look for anomalies on a case-by-case basis.
37:46
Is that what you're looking for? Abby, those are large Fields.
37:49
How do you build expertise, or does it / expertise as well?
37:54
Also, when it comes to hiring different people because what
37:57
happens is, let's say you're working on an LP challenges,
38:01
over a period of time. You tend to have this Edge over
38:04
other companies that are working in an appeal, it's a because
38:09
you're also doing subsequent research work because As you
38:12
also know, these are the few patterns that I see coming up
38:16
over and over again. And let invest a little bit more
38:19
research work. So we can solve these problems.
38:23
My question was more so because you would have very different
38:28
kinds of data with everything. So how do you build expertise
38:35
and how do you hear in that kind of like Market edges?
38:39
Well, when you're working in so many things, we certainly end up
38:43
having to build up a lot of expertise in these different
38:47
types of domains and really. Yeah as we get customers we
38:51
talked with them and we really try to understand what they're
38:55
working on and what their problems are.
38:59
But there's actually a fair amount of commonality when you
39:01
talk about the different classes of models, whether you're
39:04
talking about structured data coming from sensors on
39:08
equipment, or it could even be latitude and longitude of a
39:12
house versus images versus NLP models.
39:17
And then it's a matter of Okay, Wonder we understand the basics
39:22
of NLP and processing there. So now let's really dig in and
39:27
understand the specific cases of what you're trying to do.
39:30
Are you looking at understanding sentiment analysis, in brokerage
39:35
reports from the SEC? Or are you looking at Tweeds?
39:38
Trying to understand what an adversary is doing, based on
39:41
social medias, the stuff like that and because you're right,
39:44
there's so many different details, but I consider that one
39:47
of the best parts of the job is being Old work with experts in
39:51
their domain, they understand what they're looking for in is
39:54
let me help you understand your challenges when it comes to
39:57
deploying models and then getting the data back and and
40:00
how do you want to see this? So one thing that is awesome and
40:06
you do not have to say any names that you don't want to say, but
40:09
I just want to hear about one of the last thing is that you were
40:11
working on because as I mentioned this whole field is
40:15
completely foreign to me. So maybe you can take us through
40:19
like what was one of? Last challenges you had to
40:21
overcome and tell us a story about it.
40:24
Yeah so I can't say any names but one of the projects I've
40:28
been working on for you know most of the time I've been here
40:33
is how do you integrate into a 5 G cell system you have?
40:40
So I just so I had didn't really know anything about telecoms in
40:44
terms of this, but I'll 5G tower has a lot of software.
40:48
It used to be, this was Dedicated Hardware Hardware has
40:51
gone so fast. You can virtualize this as lots
40:54
of different software systems. So, we're talking big, big
40:59
Hardware, multiple 25 GB Network links and you can virtualize a
41:05
lot of the optimization of this. So now you have really tight
41:09
like 10 millisecond, type control loops, which means your
41:13
ml inference. Budget is maybe a millisecond.
41:17
And when you start thinking about that in terms of Speed of
41:20
light, you can't go very far. You hit has to be pretty close
41:24
to the tower. So, okay, we need to turn around
41:28
inference times in. How do you get the software
41:31
architecture? Right?
41:32
Because just a simple HTTP connection.
41:36
If you re-establish a connection, each time is often
41:39
on the order of five to seven milliseconds.
41:41
So, well, that's not going to work.
41:43
You need to keep the connection open you.
41:45
How to get the data in and out quickly.
41:49
You might be able to do Json and parse it.
41:51
You might have to use a binary format, you know, these are
41:55
solve problems. We know how to do that and then
41:57
there's the thing of. How do you get that much data
41:59
back into aggregate it? Because you think about how many
42:02
towers are in the metro area while it can be thousands.
42:05
So how do you flood all the these data back in continue
42:10
monitoring that? And this is this is an ongoing
42:13
discussion. This is not solved but the use
42:16
cases are fascinating because You have the really tight timing
42:22
optimizations and then there's also a higher level
42:24
optimizations where they're trying to optimize on second or
42:28
a couple second time period across multiple cell sites.
42:33
And how do you shut down some of the capacity to optimize power
42:37
because I didn't realize it, but a big cost of cell towers is how
42:40
much power they consume. So if you don't need, but what
42:43
can you reduce power usage there.
42:45
Wow man, this is awesome. That is So, so many challenging
42:50
problems there. Wow.
42:52
Yeah, and it certainly, this is not something that I or, you
42:56
know, the whole team or even going to solve.
42:57
This is lots of people in the industry but as an engineer and
43:03
they're just a nerd in general. Understanding the complexity of
43:07
what it takes to have a phone where I can hand it to my kids.
43:11
They find you can watch Netflix for a little bit.
43:14
Wow this is amazing. So killer wow.
43:18
Yeah so This speaking of all these different use cases and
43:24
we've gone over them. But I'm just thinking about some
43:26
in my head and there are those Telecom use cases.
43:30
You also have, like, I remember when I was back again, working
43:33
with do science, we were talking to a company that was doing
43:37
stuff on the edge because they were a shipping company.
43:40
And so, I feel like there's probably a lot of use cases
43:43
around shipping and just being out at Sea and not having
43:46
connections. Then you have any kind of I
43:51
would imagine, like the What are they the we're talking about
43:56
Healthcare and I bet there's stuff when it comes to like
44:00
pacemakers that. I don't know if they're actually
44:03
doing machine learning on the pacemaker.
44:06
I would doubt it. Maybe I have no clue again.
44:09
I'm very much a novice in this field and that's why I'm glad
44:13
I'm learning from you but then like just science are indeed.
44:16
You have manufacturing is probably another huge drug, huge
44:19
one preventative maintenance that is I miss any there.
44:26
You hit a lot of big ones. It's almost unlimited and you
44:29
can even you go into say oil, exploration or mining or
44:32
something. And yeah, this sort of things
44:35
that they're trying to optimize and you know, if only barely
44:40
touched on it, in terms of when they're asking about, how do you
44:43
do, how do you know, can we do this?
44:45
Can you manage it? 5G is a big aspect there, where
44:49
you can do a 5 G private deployments.
44:51
Now, you can cover large areas in, give something like Wi-Fi.
44:56
Activity were Wi-Fi would be harder to spread out over many
44:58
square miles or even if you're talking about, you know, there's
45:04
semi-autonomous warehouses where you have robots driving around
45:08
and you just need to optimize how they're they're running.
45:11
You don't want to put too much compute power on each robot.
45:14
That's expensive. So, you know, they get into the
45:17
trade-offs of what you want to run locally and what do you send
45:21
back over Wi-Fi or 5G or something like that, to a
45:24
central server? In terms of coordinating the
45:27
robots and you have local control also for accident
45:30
avoidance. So it feels like there's a few
45:32
different vectors that you're looking at.
45:34
When you're deploying, you're on the edge and one is how much
45:39
compute you can actually have on the edge and the other is that
45:42
speed at which you can get the data.
45:44
The connectivity are there other things.
45:47
And how do you like way those trade-offs?
45:49
Just, as you were saying about the the warehouse, I remember,
45:52
when we spoke with Someone from ocado Technologies and they have
45:58
that exact same thing like they're looking at a gigantic
46:01
warehouse and they're using computer vision with they're
46:04
different, they're different Pickers is what they are.
46:08
They so it's like those games that you play in carnival, you
46:11
know, they have their their different hands that they go and
46:14
pick things out of in the warehouse.
46:16
And so when you're looking at these different vectors, how do
46:20
you look at the trade offs and Recognize, what is the optimal
46:28
way of doing it is. So, yeah, that's really
46:30
interesting there. There's so many different
46:33
vectors you can optimize because it if you battery-powered, it
46:37
can even be. How much power is my inference
46:40
going to take, if I run it locally versus if I use a
46:45
network connection to transmit the data off in, it could be
46:48
something like, well, if it's, if the data are tiny, That's a
46:51
little bit of power to transmit it.
46:54
So send it off if it's in image and I'm doing a little bit of
46:57
filtering and makes a lot more sense to filter it locally in
47:01
send-off, only a small fraction of the images that actually need
47:05
to be processed but then you also end up with all of the
47:09
networking, trade-offs of well, for one would be just bandwidth.
47:13
If your video is heavy, it takes a lot of data.
47:18
So if I'm monitoring cameras in a Retail store.
47:23
That model may not be profitable, the economics may
47:26
not work, if I have to provision enough bandwidth to stream it
47:29
all the way to my cloud provider.
47:32
So I need to bring the compute closer and then there's also the
47:36
latency if it if there's a timing Lou but there's tight
47:40
requirements whether it's user experience, or 40 control
47:43
system, can I get the signal there and back in?
47:47
Can I get it there and back with enough nines of reliability that
47:50
the system stays up? Well, Oh, I can't, you know, I
47:55
can't go 200 miles away. I don't have enough time
47:57
bandwidth or it's not reliable enough.
48:00
So let me bring the compute closer to the problem.
48:03
There's so many different vectors there to optimize and
48:07
where, how do you make the economics work?
48:09
Where computers really cheap in a data center but if it's too
48:14
far away, so I'll bring you a closer.
48:16
Can I afford to do that? Can I afford to go stick some
48:20
expensive servers in a warehouse where I don't have IT staff to
48:24
manage them and you know, now my support costs are higher there.
48:28
So does that still make sense? So it really feels like all of
48:35
these challenges are super specific to this type of AI at
48:40
the edge, right? Like these questions we take for
48:46
granted when we're just grabbing something off of AWS.
48:49
And we know that we have infinite power, infinite
48:52
compute, infinite storage, all of these different challenges
48:56
that we We would never encounter if we were working at the edge.
49:02
It's fascinating to me. Oh, it is, it's we have a little
49:07
bit of that. And we've all gotten the cloud
49:09
bill, or our managers gotten to the cloud.
49:11
Dylan come down and said, what did you do to fix it?
49:15
What have you done? So cloud is great.
49:18
You can scale until you can't afford it but but yeah, then you
49:23
have all these other tray also like why can't scale.
49:25
I have this much. You know, this one's Computing,
49:28
this much memory and all of the services have to share that.
49:32
Figure out how to get it done. I love the fact that you talked
49:35
about economics of operations, because as we are gathering
49:38
mode, Dia, by the minute, it is one of the most important
49:42
questions and maybe even for people who have their deer on
49:45
the cloud because not all of the DIA is relevant or stuff, maybe
49:50
stops being relevant after a point of Time, how do you know
49:56
for the cost of deploying these models?
50:00
And how do we think about how much they are do?
50:03
We keep On a premise. Good question.
50:06
We struggle with this the same question.
50:08
So, Aw of my job is how do we lower the cost of deployment and
50:14
cost of deployment? Is everything from how much
50:17
bandwidth does it take to get an optimized new version of a model
50:22
out if it's a whole container that maybe GB potentially?
50:28
If it's a bloated container, that's a lot of transmit time.
50:31
If you can get it down to just deploy a small model bundle that
50:35
can be megabytes or tens of megabytes, perhaps.
50:39
So but really to me the biggest cost of deploying a model is
50:43
typically the human, the Ops Talent, the data science talent.
50:48
And how do you make those people more efficient?
50:51
So that they're really generating as much value as
50:55
possible and not having to mess around with, you know, with all
51:00
the little details and I tend to think of that is the same thing
51:04
that we saw with kubernetes and all the other data center.
51:08
Optimization tools is when you are able to take your team and
51:13
suddenly suddenly over a couple of years, be able to manage 10x
51:17
more servers. People didn't say well, great.
51:21
I'm going to slash my op staff. They said, give me 10x more
51:24
servers or really what they said was.
51:27
Wow. Now suddenly a bunch of my
51:29
applications are cost-effective when I go from twenty dollars an
51:34
hour to two dollars an hour to 20 cents an hour.
51:38
So give me 30, x more servers, go hire people.
51:41
This is awesome and we're seeing the same thing with and all in
51:45
general. And I think we'll see it with
51:47
edgware. You started off with huge
51:50
Industries, like high frequency trading on Wall Street and
51:54
online advertising where they could hire phds by the bus load,
51:58
and do everything full custom and of a cost a hundred million
52:00
dollars. That's a good deal.
52:03
Most companies aren't going to be able to justify that.
52:06
So if you can get it down to a million dollars to deploy a
52:08
model, you open up more applications.
52:11
If you get it down to ten thousand dollars, a model to be
52:15
able to for a year to be able to deploy it, run it optimize it,
52:20
you know, grow and nurture this model and your farm of models,
52:24
you know? Now this team is increasing
52:26
their abilities, their generating more value for the
52:29
organization, you know. Now you have even more models
52:32
that people go. Hunter deploy.
52:36
I love that. That is so true with the idea of
52:39
getting the Ops cost down and then you're not all of a sudden
52:43
saying okay great. Now we don't need our hops team
52:47
because our costs are so low. Again we subsidize it is some
52:52
other way it's like know. Now we want more of this because
52:55
we are seeing that Roi and we are seeing the potential and I
53:00
think that's crucial like being able to tie the machine
53:03
learning. Projects that you have in the
53:07
products that come out of your, your team that deal with machine
53:11
learning tying that back to revenue.
53:13
Is something that we hear time and time again when it comes to
53:16
this podcast, like so many people have said it and actually
53:21
funny enough. Somebody wrote a paper on it
53:23
about how many times we ran Rave about it and in this podcast
53:27
specifically because it is so important, it is like one of
53:31
those, it's Paramount and a lot of times we can get get a little
53:35
bit distracted and sidetracked with the shiny new toys, but if
53:39
you're not driving Revenue, at the end of the day, your You're
53:45
just playing around. It's like if it's not actually a
53:49
useful toy. Absolutely, at the end of the
53:52
day, somebody had you have to pay the salary somehow there has
53:55
to be a reason. Yeah, you're grounded.
53:57
It can be fun if you're not doing it, then you get to the
54:00
point. You're like, wait a minute, this
54:01
isn't going to last forever, is it?
54:03
Yeah, it just it turns into like, how much of a cost center
54:07
are these 70 data scientists that are on payroll and how How
54:14
can you justify keeping them if you aren't actually?
54:20
They're not pulling their weight or even is not only the data
54:23
scientist that can be, the data Engineers, the data, anybody, or
54:27
the machine learning Engineers. Whoever it is, if all of these
54:30
projects that are coming out of the team that is supposed to be
54:35
delivering some machine learning capabilities are not actually
54:40
hitting the mark, you've got problems.
54:42
So I know I'm preaching to the choir and Jason has been great
54:46
talking to you, man, in case anyone has not already.
54:50
Meanwhile, the roof I would highly recommend they check it
54:53
out and they get to see your skills in action with that
54:57
platform will leave all of the information to everything you
55:00
need to know about while the room in the description.
55:03
And that's about it for us. Thank you so much.
55:06
I'm a fan of the Pod. This has been awesome.
55:08
I enjoyed it. Hi, this is meanwhile Eric.
55:12
I'm the founder of comment on data Innovation and confetti I
55:16
and this is a quick Public Service Announcement.
55:18
If you're tired of having nightmares, Is about the machine
55:21
learning models. You have in production, then I
55:23
encourage you to subscribe to the ml Ops Community podcast,
55:26
it's where all the cool ml Ops kids hang out and learn how to
55:29
tame their ml models.