Latest / Agentic Conversations (formally mlops.community) / [Exclusive] Tecton Round-table // Get your ML Application Into Production

Transcript

[Exclusive] Tecton Round-table // Get your ML Application Into Production · Agentic Conversations (formally mlops.community)

0:00
We are back and this is a very special session we've got today.
0:06
You know how we like to do it. Bring it real, real hot and
0:11
heavy with this ML in production stuff and we've got some hard
0:17
hitters with us today for this event.
0:20
It's a very special round table session, as we like to call it
0:24
here. And what I wanted to do is try
0:28
and recreate some of the magic that I have seen happen when I
0:32
have visited the Tecton office in San Francisco and New York.
0:38
And every time I've been there there's been this like random
0:44
conversations that will be happening at the water cooler
0:47
you could say. And since there are so many
0:50
incredible engineers that work at that company, it inevitably
0:56
teaches me a tonne. And so I thought, well, why
1:00
don't we try and get some of these awesome engineers onto a
1:04
call together and set a very, very high expectation of having
1:08
it be one of these kind of conversations.
1:12
So enough of me blabbing. I might as well just bring on
1:17
some of these engineers that I am Speaking of and make sure
1:21
that you all know who they are. First, I gotta bring on my
1:27
German brethren. From the CTO desk of Tecton,
1:33
we've got Kevin Stump. Where you at, man?
1:36
Where you at? There he is, numero Uno to come
1:40
on to the stage. How you doing, dude?
1:42
Don Katya, I'm doing very well. Thank you.
1:46
It's a pretty disgraceful that I still don't speak enough German
1:49
to be able to converse with you right now in it.
1:52
Next up we've got Isaac, Where are you at?
1:56
I said. You are coming in as the
2:00
consulting architect from Tecton.
2:03
And how you doing today dude? Ohh and we can't hear you first.
2:11
First one of the day, I get the feeling there's gonna be a few.
2:15
Technical all the way there was. Nobody had to do it.
2:18
Somebody had to be that person. So Next up, I'm going to have
2:23
Eddie join the screen. Eddie, dude, what's going on?
2:27
Solutions architect at Tecton. Great to have you here.
2:31
Yeah. Thanks for having me.
2:33
And last but not least, we've got Derek coming on.
2:38
Derek, for everyone out there, is a product manager at Tecton
2:42
and has helped me immensely thinking through products and
2:46
how to properly look at ROI and I want to get into that so much.
2:53
But first, we all came here because getting ML models into
2:59
production, it's been it's a tale is as difficult as the end
3:04
of time, right? It's been difficult to get these
3:07
models into production for a while.
3:11
And does it have to be I guess is the first question, I think
3:16
it's still is even though there are all these MLM's out there.
3:20
But for the ML stuff like we want to talk about these
3:25
challenges, that is is happening with what I've heard some people
3:30
call classical ML, traditional MLML, one point O, whatever you
3:35
want to call it, whatever your flavour is.
3:37
I would love to hear from whoever wants to jump in first.
3:44
What are some of these challenges when it comes to
3:48
getting your traditional classical mill into production?
3:52
Yeah. So in my role as a solutions
3:55
architect, I talked to a lot of different folks.
3:58
We're in a pre sale site so there's a wide variety of
4:01
different customers that we get to talk to.
4:05
My opinion, by far the biggest challenge is the way different
4:08
organisations are set up to actually try to deploy models
4:13
and and what I mean is typically have data scientists on, you
4:17
know, one end of the floor, developing models, training,
4:21
building features on their laptop.
4:24
With God knows what language and doing what they're doing and
4:27
then they unfortunately throw the recipe for the features over
4:30
the fence to some or MLE team that's probably understaffed
4:35
that now has to professionalise. That recipe and make these
4:40
features go live and this is difficult like in a bad setting
4:44
like these batches features. But then we have like these real
4:48
time features, features that are coming up the stream that
4:50
increases complexity. And I, you know, just to see
4:54
these two different teams and it happens that every organisation,
4:57
I think there's a lot of friction there to get these
4:59
models into production. If you're really lucky enough, I
5:02
think to have the data available to train your models on and ID.
5:06
Yeah. You already jumped, I think, to
5:09
the point where a data scientist may have access to the relevant
5:12
data in the organisation, have a model that's actually a good
5:15
quality and whatnot, and now that you absolutely you're
5:17
excited about throwing over the wall to an ML engineer, Data
5:20
engineer, software engineer. What have you.
5:23
But I think you even to get to that point you first they do
5:27
make sure that he collect the right data and that he give
5:29
access to the right data and that makes it easy to to find
5:32
it. And then organisation, obviously
5:34
depending on the size of the organisation that is and almost
5:37
the the regulatory requirements of the industry, the companies
5:40
and etcetera, that can already be the first pretty gigantic
5:43
hurdle to overcome. Yeah, Yeah.
5:47
Like the question or the framing of like.
5:50
Why is it still hard? Right, we've been.
5:52
People have been trying to build tools to make this easier.
5:54
We've been doing this for a long time.
5:57
Measure, reflect on like you know that.
6:00
I think the best we can aspire to is.
6:02
You know it's as hard as like a software project, like software
6:06
is so hard, right? Like putting any real good
6:09
engineering product out there in the world is is an endeavour.
6:12
Now I. Obviously machine learning, like
6:16
I think as a project manager I'm trying to make this type of
6:18
thing easier. That probably be my goal if we
6:21
could be like at least as easy as you know, regular software
6:25
engineering project, but just as like Eddie and Kevin are saying.
6:29
You have these additional things that come in.
6:31
Yeah, we saw diverse skill sets and also all those access to
6:36
data, yeah, extra complexities that come into play.
6:39
So I don't think it's ever going to be easy, but obviously we try
6:43
every day to make it more approachable.
6:45
I wonder actually if it could be easier than deploying software,
6:48
because with like today, it's definitely harder than it is in
6:52
most organisations to. Train and deploy ML models if
6:56
you compare it to how easy it is for them to develop and deploy
6:59
software applications. But if you think about it, I
7:01
think that. Solution space.
7:03
So let's say the the flexibility of the different types of
7:05
solutions that you may want to implement in software is a
7:09
larger space than the slightly more constrained space of metal.
7:13
And so maybe we can set the bar even higher than that and say,
7:16
hey, we should get to a point where it's not just as easy as
7:19
playing software or as hard if we will, but actually
7:22
significantly easier. I think, Kevin, you mentioned
7:25
like you need to have the data in place.
7:27
I think we're in a good spot now because a lot of companies have
7:30
been spending 5-10 years. Getting their data in the right
7:34
space and now they're saying how do I get business value out of
7:37
that data and and that that is going to be some of these
7:41
traditional model as well as operational model.
7:43
So as I work with customers, we're in that kind of Inflexion
7:46
point. We're working with that.
7:49
This case into product and where we see kind of we're building
7:53
out project plans associated to get that use case and still that
7:57
single largest like pole in the tent is around kind of setting
8:01
up those data processing pipelines and and actually
8:04
developing that on SQL or Pyspark or other code.
8:08
So when I see like what is the largest bottleneck, it's
8:10
actually having inefficient feature development workflows
8:14
that on that tape go from your analysis and design phase to
8:18
actually writing the features and putting that into the
8:20
production. So how do we actually make that
8:23
workflow in that lifecycle much more efficient?
8:28
Love that. Well, it it's interesting what
8:31
you're saying. There, I say.
8:32
Is how we've been building these data foundations for a while and
8:40
I would love to hear from you and or Eddie on have you seen
8:46
just like messy sprawls and if So what was the worst one and
8:51
tell us who it was. No, don't call out anybody by
8:54
name obviously, but like because there's that rationale of, yeah,
9:00
we should have our, our stuff together by now.
9:04
But potentially there's also another rationale where it's
9:07
like, yeah, it's been 10 years of us just like accruing tech
9:10
debt or data tech debt. Yeah, I mean.
9:14
I I do think we've coalesced on like a base set of tools that
9:19
we're going to use to develop our data pipelines.
9:22
Like 10 years ago, you may have decided to build this
9:24
application with something called Storm or or something
9:29
else, right? Yeah, I think we've landed on
9:31
maybe Flying Spark, Streaming Spark.
9:34
No play like we know the tools that we're gonna be using.
9:37
But I still like even. Once we get into production,
9:41
there is so many things that can happen upstream.
9:44
That can be challenging from like a data perspective, but
9:46
just to give you an example of working with customer this week.
9:50
Feature was being refreshed on a daily basis, was working fine.
9:54
All of a sudden that there's no features being developed or or
9:58
being created on daily basis, what happens?
10:00
OK, the upstream ETL now decided to run 2 hours later than what
10:04
they were used to. And OK, we have to catch this
10:07
now. So I feel like you know the
10:09
tools data is in somewhat shape, but there's still so many,
10:13
there's so many points at which things can get messed up that it
10:16
makes it challenging to really get these features and models
10:19
into production. Ohh, that's a great point.
10:22
So Speaking of that, I think there is an awesome question.
10:26
I mentioned it. I kind of like hinted at it,
10:29
foreshadowed it if you will, for you, Derek, and talking about
10:34
how much infrastructure cost you need and how many places that
10:39
things can go wrong. As Eddie was saying, how do you
10:42
even evaluate if you're getting positive ROI from all of these
10:48
different costs and from all of this time that you're putting
10:51
into making sure that things are working like they should be,
10:54
What does that even look like? Like what's your framework
10:56
around that? My favourite story around this?
11:00
In terms of evaluating the ROI as the ML was second to some
11:05
folks, it's a long time ago now. So I think it's any sensitive ID
11:09
but something like the Uber pricing team and 2015 or 2016
11:14
maybe and they had come up with this like really sophisticated
11:18
deep learning model that like really like optimised like the
11:23
price of the conversion rate. Now it's people can play,
11:26
requesting rides. They were, you know, obviously
11:29
there Jane, generating more business.
11:30
You're requesting Pride, so they have this awesome model, but it
11:34
costs like. $0.50 per additional ride generated, so they did all
11:39
that math right. They're like well we we spent
11:41
$0.50 of like additional compute on this deep brain model and
11:44
then we get additional right. Which might make sense in North
11:49
America, where their profit margin might be a couple dollars
11:52
a ride, but in, you know, some place like India where their
11:56
profit margin might be like $0.10 a ride, it didn't work.
11:59
And and so this team which built an awesome model and ultimately
12:03
ended up not rolling it out because there were some places
12:06
where the additional value they got of that out of that was not
12:10
covering their infrastructure costs associated with it.
12:14
Yeah. So, yeah, that's like a team
12:17
that actually did really well where you were able to finally
12:20
calculate your return. So it's like something that is
12:24
hard to do and usually falls in the realm of like a, B testing
12:27
with your rolling out a change to your application.
12:30
And then also even harder, I think actually is getting a good
12:33
grasp on much what's the cost of it.
12:35
And even if you, you know, take out like a development costs,
12:39
yeah, there's a lot of different infrastructure that adds up into
12:42
this. So, as Eddie was saying.
12:44
Like all your CL processes and then you have potentially like
12:48
your databases that are powering your real time applications and
12:51
then the other services maybe like your model serving.
12:55
Infrastructure. And so there's really no way
12:58
around it other than you need good cost attribution.
13:01
You have to make sure you have all your tags in the right place
13:04
so that you can go back and say this use case, you know running
13:08
it for a week cost me this amount of money and then when
13:13
you have that in place you can go back and say OK it was this a
13:15
good return or not. And now there is also a lot by
13:19
your industry this type of problem where your ML is really
13:23
expensive. On the running side is more like
13:27
a BBC problem where you have, you know, you know, millions of
13:31
users, maybe hundreds of millions and especially like
13:34
recommendation systems are really expensive to run.
13:37
And the the marginal value there is a little bit lower versus
13:40
when we look in like the Fintech space or would be the space you
13:44
know every like. Guy caught fraudulent
13:47
transaction you get saves you a lot of money and so those were
13:51
cases are a little bit more like obviously worth it not to be as
13:53
as careful. But if you're looking at things
13:57
like surge recommendations and how big consumer app is, where
14:00
it's, it's definitely impactful for the business if you can
14:03
finally attribute the cost of running the systems.
14:07
Now, what tooling would you recommend Derek for somebody who
14:10
doesn't have the most amazing cost attribution in place right
14:13
now? Like where?
14:13
Where should they start? Yeah, I mean I'd say I'm a
14:19
pretty big fan of the WS cost Manager, right.
14:20
It actually or whatever your five provider is because there's
14:23
going to be really disparate set of tools.
14:25
So you want to make sure that you have all that attribution
14:28
and obviously I have to show a tecton.
14:31
We really do try to give you granular insights into the
14:34
individual features that are running and how much it costs to
14:37
produce those. That's just one piece of the
14:38
puzzle, I'll say. There's both like standing the
14:40
model in front side and so really you need to like kind of
14:43
collate all this data. You have actually Lightweight
14:46
pipeline that. Together all this cost data so
14:48
you can get that total report. And.
14:52
So I said, yeah, it's really about having that full view over
14:55
your infrastructure as opposed to an individual one.
14:58
But it also comes down to how you architect your system so
15:01
that you're actually are able to attribute costs to different
15:05
systems, like if you're. If you just have one big
15:09
pipeline that runs all your features, you never know how
15:11
much an individual and costs and saying look like you're using a
15:15
shared model server across different applications and it's
15:17
hard to attribute it. So it also comes down to how you
15:20
architect your system. I think it's probably the
15:22
highlight also. I mean the cost attribution to
15:25
the infrastructure is important, but also I like what you said
15:27
there about what the value is from a use case perspective like
15:31
that. That's maybe on the business
15:32
side of the house, that's so much infrastructure really, but
15:34
I think you also have to understand that.
15:37
And oftentimes that's like the hardest part, right, Like for
15:39
something like fraud detection, I think it's pretty obvious.
15:41
But for many other use cases that we saw that ad over many
15:45
years ago with no platform like you wanted to, ideally just a
15:48
very hard objective number of these are the additional dollars
15:52
that this use case as to the bottom line, but very often it's
15:54
just not the case. If it's a safety prediction or
15:57
maybe it's just end customer, end user delights, like how are
16:01
you going to measure that in terms of dollars and it's.
16:05
The you know you'd have to follow the user over many, many
16:08
weeks, months and then somehow try to isolate that one
16:11
variable. It'll be basically impossible.
16:14
And but even if you don't have a heart objective number how it
16:18
affects the bottom line, I think oftentimes it's it's still
16:20
extremely important to just showcase to those different
16:23
teams, well, this is how much infrastructure you're actually
16:25
spending and then they'll see, OK, well this is.
16:28
Obviously way more expensive than the value that my model
16:33
could ever create for the business, even if they don't
16:35
necessarily up heart number. And so I think measuring the the
16:40
data on surfacing it to the poker involved becomes crucial
16:44
no matter what, because as Drucker said, you can't manage
16:47
what you don't measure so good. Alright.
16:50
So ROI is a huge one. I think the, the employee that
16:57
can understand how to bridge those two gaps between like the
17:00
business side and the technical side is worth their weight in
17:03
gold. And to be able to understand
17:05
that it is brilliant, we got that.
17:09
Let's keep it cruising though and let's talk a little bit.
17:12
Like Eddie, I'm going to throw this one over to you is looking
17:18
at different sized customers, right?
17:20
You've been working with small, big ones and you've undoubtedly
17:25
seen hurdles when it comes to getting these models into
17:28
production. Is it a difference where you're
17:32
going from? Like if you've got a team of of
17:35
1, like that one unsung hero or maybe a group of 10 versus a
17:42
whole platform team that's servicing the needs of 70 data
17:45
scientists, that type of thing. Like how did those two sides of
17:50
the spectrum different, different differentiate
17:53
themselves? I think there's challenges on
17:55
both sides. I.
17:57
Just at the top of the mind, I feel that the larger, perhaps
18:00
more advanced teams just have a lot more requirements that you
18:04
have to try to. Meat.
18:09
And I think that makes it a bit challenging, right?
18:11
There's security requirements, there's SLA requirements,
18:14
there's uptime, and there's there's a whole host of things
18:16
that have to be taken into account versus a smaller,
18:19
scrappy team that you could be perhaps a little more nimble
18:23
with. Might be easier to get things
18:26
done there, but you know there's this challenge on both sides of
18:32
the customer profile. I'll I'll add to that which is
18:35
like I think what Eddie is is saying is that when you just
18:39
have you know a handful of use cases that are running on your
18:43
platform, it's a lot, it's a lot clearer what you're building
18:45
for. OK, we need to meet the specific
18:47
requirements of this like fraud case and so that that helps draw
18:51
box of what you need to work in versus when you usually when
18:56
they're the larger organisation it usually means more diversity
18:59
of use cases. And so even though you have more
19:01
resources to bear and it's like you can invest in this ML
19:03
platform, you also have to balance the like.
19:06
Very different needs of the recommendations team versus the
19:08
fraud team, which is the pricing team and I think that's where
19:13
you know. You can.
19:15
If you try to do all those in one system, sometimes you can
19:19
shoot yourself in the foot. Yeah.
19:20
Thank you for articulating that. So welter there we go.
19:23
You just pass the ball over to Derek will clean up on it.
19:28
So the the other thing like when it comes to that depending on
19:35
the sizes, what do you feel like are certain infrastructure
19:41
pieces that you are you must have in place before you can
19:44
even think about putting some XML into production.
19:48
Like I know and and we should probably go and specify what was
19:54
it. Someone told me the way like the
19:57
speed at which you need those decisions to be made, right?
20:00
We should specify like are you doing batch or are you doing it
20:05
super real time ohh or super batch and what kind of
20:10
infrastructure? Needs to have there and maybe
20:13
Kevin all kicked this one over to you and everybody's nodding
20:16
their heads, so I'm sure everyone has some kind of
20:18
thoughts on it. Yeah, let's maybe start with a
20:21
super batch as you just coined it.
20:24
You guys think no matter what type of e-mail you apply, the
20:27
batch real time would not like you always have the need to
20:31
generate training data and then later on you need to have the
20:34
need to actually generate prediction data.
20:36
And what you need to do is if you want to be able to generate
20:40
training data, you need to be able to look at what the world
20:44
looked like at any given point of time in the past because
20:47
that's what you train on. And if you don't do that?
20:50
And you introduce the risk of data leakage, label leakage,
20:54
where your model just trains on some data that it sees from the
20:57
future. And so no matter what, I'd argue
20:59
that one of the most important things to start with is ensuring
21:02
that we actually have a clean record of data somewhere in the
21:06
data warehouse, somewhere in your data lake that preserves
21:10
what the world looked like at any given point of time in the
21:13
past. As far back as you may want to
21:15
trade your models on. And the simplest case that
21:18
basically just means they take like snapshots from your
21:20
operational database. Once a day in the road into a
21:24
different partition on your data lake or in your data warehouse.
21:27
Or have you more a little more advanced?
21:30
You may want to actually deal with these immutable event logs
21:35
of data so that you're not redundantly snapshotting all the
21:39
data at night, but you're actually just appending to an
21:41
immutable and only log of data whenever a change happens in
21:46
your operational system. To make that a bit more
21:49
creative, your credit card company whenever transaction
21:52
comes in or a transaction state changes, you would just actually
21:57
append a new row to the data to the table that you have in your
22:00
data lake. If you deal with this incredible
22:04
event logs versus just snapshotting all of the
22:07
operational transactions that you maybe have in your database
22:10
at any given point of time at midnight.
22:12
So that's that's step one like having the centralised data
22:16
warehouse data lake where you store all the historical data so
22:19
that the data scientist can actually go there and trade
22:24
their models. And that comes to the second
22:26
piece, which is what I mentioned earlier, actually making it easy
22:31
for the data scientists in a secure way to access the data,
22:35
sure that it's audit log, make sure that they have actually the
22:38
right permissions to access it for they're getting the use case
22:41
and the different industries have different regulatory
22:44
requirements. Well, that's where I'd start.
22:46
And then you should have most things in place from a data
22:50
engineering perspective to train a model and then use it to make
22:54
batch predictions. Obviously outside of just the
22:58
data domain, there's also the modelling domain.
23:00
So you want to give your team an environment where they can
23:03
actually train their models, and that could be as simple as
23:06
spinning up open source Jupiter notebook.
23:10
Or you could go and use one of the other awesome notebook
23:14
providers like maybe X or Deep Knowledge or any of these good
23:17
solutions so you have a standardised way for data
23:20
scientists to actually train their models.
23:24
And before I continue and talk about maybe more advanced use
23:28
cases and more advanced infrastructure, anything else
23:31
that anybody here wants to chime in?
23:34
No. Perfect cover.
23:36
They're leaving it all to you, man.
23:37
It's all on you now. Thank you.
23:40
No, no, yeah. I think you hit the nail on the
23:41
head. I think just.
23:44
From my perspective, when I've done these architecture reviews,
23:47
if I see folks trying to drive features off of their OLTP right
23:51
their online database, I think red flags come up.
23:56
You did definitely need this kind of separation of this
23:58
online system with the system, your data lake, your data
24:01
warehouse, whatever that you can actually compute and Dr these
24:04
features from. So like that's one basic
24:06
separation. Or distinction that you need in
24:09
college. You're making it started.
24:11
Like somebody who does not have in place, what tool would you
24:13
recommend that you use? Ohh you know I think there's
24:18
countless 5 crown, you know something that could be
24:21
basically take the snapshot off of the OTP and just land it into
24:25
DVT or something into like us three or your data lake or data
24:30
warehouse. Yeah, I think 5.
24:32
I think that's a wall. It's a wall solved problem at
24:35
this point, it just it hooks on there yet.
24:38
Cool, and you've got the infant place to do these batch use
24:40
cases. But batch is great for lots of
24:44
use cases. Aware you don't need to make
24:46
real time decisions that affect your end user and the bulb.
24:51
Like say if you wanna do lead scoring and make it easier for
24:54
the sales team to figure out how would lead so they go after or
24:58
you want to do churn predictions to figure out who do I want to
25:00
send a a reward e-mail to you then these batch use cases are
25:04
fantastic. But whenever you want to
25:06
directly and in real time to track with the end customer
25:08
advice, say detecting fraud on the credit card immediately, or
25:12
recommending a certain item, or giving them an insurance price
25:16
quote or anything like that, then batch by itself is simply
25:20
not good enough anymore. MAD.
25:22
And so that's when we see most organisations actually put
25:25
streaming infrastructure in place like having your Kafka
25:29
cluster set up or just use Kinesis on it US.
25:32
Or partner up for the provider like Confluent just to make it
25:35
as trivial as possible to complement of your batch data
25:38
infrastructure with a streaming layer that provides real time
25:42
data, real time events to all the different downstream
25:44
consumers. And one very important one of
25:46
those would be typically the ML application or the the feature
25:51
platform The feature. More upstream, which is
25:53
continuously processing data coming from the stream and
25:56
making the feature calculations immediately available for real
26:01
time predictions. Isaac, I see you man.
26:04
I see you wanna say something. No?
26:07
Ohh yeah. Yeah, I just say like one of the
26:09
challenges that we see often when people are going from batch
26:12
batch to either batch and online inference or streaming online
26:16
conference is the change how you actually processing data with
26:19
batch and batch. Oftentimes you're you can get
26:21
away with kind of a trunk and reload type processing framework
26:25
or on a regular basis you're scanning all of the data and
26:29
then writing that into kind of a feature store and running.
26:34
Data as you start working with streaming or are you start
26:37
building kind of pipelines are saving online infrastructure
26:41
instead of processing and scanning the old data, you just
26:44
want to scan data that has changed within the last time you
26:47
ran a job or and that introduces the kind of a lot of data
26:52
engineering problems that many data scientists aren't familiar
26:55
with. So as we are working with all
26:57
our customers to kind of move from what we what was
27:00
traditionally batch batch to these kind of up the maturity to
27:04
streaming and online. We see some of the some
27:07
roadblocks around kind of changing kind of that or moving
27:10
up that learning curve to understand kind of how to
27:13
process and handle change to capture which goes into other
27:17
problems like how to handle it, arriving data, etc.
27:21
Would you recommend this? Kind of like the simplest way to
27:23
getting started? If somebody doesn't quite have
27:25
the streaming entry in place yet, but they've got their whole
27:27
batch stack, figure it out. They have an operational
27:30
location, have an operational database.
27:32
What should be the next step to without?
27:36
Premature optimization. Putting some infrastructure in
27:39
place in order to get access to it.
27:41
Fresher features and therefore more accurate predictions.
27:44
I think it really depends on kind of the sophistication of
27:48
your tool you can of your team and the kind of where you want
27:52
to excel in on they. One of the kind of first steps
27:56
would be, hey what is it a streaming architecture and
27:58
that's where you kind, I would lean towards like Kafka or kind
28:02
of a surface like consulate or. I mean there are kind of a lot
28:06
of services that you can go out and kind of.
28:08
Again, confluent or even kind of moving to something like effect
28:13
on who, what, where. It's kind of is able to abstract
28:15
some of the complexity away from on those challenges I wanna
28:20
throw. In even before you actually like
28:25
make the jump to streaming, one of the cool parts with these
28:28
animal models is you can actually simulate what would
28:31
your potential performance gains to be before you go and do all
28:35
that additional infrastructure. So assuming that you did what
28:39
Kevin said earlier and that you have good cast state of the
28:43
world data. You can run your model
28:47
evaluation pipeline assuming that you have shared.
28:50
So let's just like assume every time a new record comes in, I'm
28:53
instantly able to use that for inference.
28:56
I could run my. Area under the curve, like
29:00
across. Yeah, type measurement.
29:03
Understand. Hey is this a a 1% or a 20%
29:06
improvement in my model accuracy compared to when I'm only
29:10
updating my, you know, tables like one today and my features
29:12
one today and then? Hopefully like that.
29:16
Going back to Rai conversation, it doesn't tell you exactly what
29:19
the impact on user is gonna be, but it should tell you of like
29:22
well this is this is really worth me moving into that
29:25
streaming world because it's like has a really clear impact
29:30
on my model performance. And for sometimes you actually
29:32
might learn that it doesn't depends on just like the
29:35
velocity of your data and the nature of your problem.
29:38
There's something else that I was thinking about as you all
29:40
were talking about this, and it's potentially you've seen
29:45
this happened more than you anyone likes to admit, where you
29:49
set up your batch and then you set up your streaming, and they
29:52
don't really live that cohesively, like they aren't
29:57
able to talk to each other that well.
30:00
And so you end up having almost like a wall between the two.
30:04
How do you go about fixing it so it's more like a superhighway
30:10
than a wall? And what does that look like in
30:13
practise? And maybe, Eddie, I'll throw it
30:15
over to you because I see you bobbing your head the most.
30:18
Yeah. So I I think when it comes to,
30:21
you know, we've been talking about streaming features kind of
30:23
these online use cases and then supposed to be batch use cases.
30:29
And and this is how I interpret the question D I I think the
30:33
biggest problem I see there is something called training
30:35
serving skew. Where you have these real type
30:38
systems. Delivering features to your
30:41
online system, right? It's, you know, the thing that's
30:44
powering the fraud model or actually doing the inference
30:47
with the recommendation. And that's all well and good,
30:51
but then you have this other. And wants to Hwy going in the
30:54
opposite direction to keep the analogy going, which is how do
30:57
you actually drive the training data to build the model?
31:01
And that training survey skill is kind of a universal problem.
31:04
You almost have two different data pipelines.
31:08
Hey, I'm Vishnu. I'm a data scientist at first
31:10
hand and I definitely think that you should subscribe to the ML
31:14
OPS Coffee Sessions podcast. It's the best podcast out there
31:18
to stay on top of what MLP's actually is, to talk to the true
31:21
thought leaders in the space. And oh by the way, Demetrius is
31:25
absolutely hilarious. What a weird guy.
31:27
You should definitely subscribe to the podcast.
31:35
You have your batch written in SQL and you have your real time
31:38
written in Java or something like that.
31:40
And then how do you make sure that the features are actually
31:42
the same? That's a very challenging
31:43
problem. I see that for any organisation,
31:46
big or small, regardless of use case.
31:48
I think that's like #1 problem that folks have trying to get
31:52
into production. How do you solve it?
31:56
I I think Tecton does a very elegant job of solving this.
31:59
Obviously I think we we, we you know the fact that we can
32:03
leverage Spark to actually compute the real time streaming
32:07
features as well as the batch features and then kind of have
32:10
this lab architecture to write to both the online and the
32:13
offline store. I think that's probably the
32:15
easiest, best way to do it, but I'm sure there's other ways to
32:19
handle that problem. Sure.
32:20
Is what do you, what do you guys think?
32:21
Like there's obviously batch features, there's three features
32:23
we've talked about, but then there's also real time features
32:26
that are actually executed at prediction time rather than pre
32:29
computing them necessarily from the street.
32:33
From what do you guys have seen in your tuition, how do you
32:37
describe kind of the relative importance of those three
32:39
different categories of features?
32:42
Yeah. And I think it's important to
32:43
define what the, what the real time feature is, maybe for books
32:48
that might not be familiar with our nomenclature.
32:49
But basically in these type of data that she only knows the
32:53
type of inference like we're doing a fraud transaction,
32:56
you're trying to score that, you know the merchant, you know the
32:58
amount of the transaction, you only know that if the Democrats.
33:02
Of those use cases, I think the real time features are super
33:06
important for that type of use case.
33:08
Yeah, I I would go so far as to say that for many use cases your
33:13
real time data is actually like the most.
33:16
It impacts all and so you know everything about search right.
33:20
One of the most uneconomical ML use cases.
33:24
You know you're like you're the literal LIKE search terms that
33:26
you put into the query are life most important things for
33:30
getting you good results. And maybe the second most
33:33
important thing might be like you're like are you in the US or
33:38
are you in Canada, right. That's going to influence what
33:40
results are useful for you and and then the rest of it right
33:44
when we start getting into like. Maybe what is my?
33:49
What type of queries I've done in the past and that's what you
33:52
get into back end streaming and like those we know can have like
33:55
a really big impact on the performance of your search and
33:58
recommendation results. But I think our actually
34:01
secondary to the value that you get out of that immediate
34:04
context and depends on the exact use case.
34:08
But I think that shouldn't kind of underestimate that that real
34:12
time payload. Yeah, I think in relative
34:15
importance is so use case and model specific.
34:18
It's hard to make streaming versus batch first real time.
34:21
I would say real time at depth play probably the most common
34:24
that you see up on within almost every single model is there's
34:27
there's going to at least some type of online model.
34:30
You're seeing some level of real time.
34:32
I think what they posed some unique challenges especially
34:35
from the data governance perspective.
34:37
There's a lot of times, unless you have, like if you just show
34:39
a feature platform, you're pushing this logic into kind of
34:43
the model code itself. Under your coupling this in your
34:45
model code or have this maybe decouple it as as another step
34:50
in your workflow. But then you're maintaining the
34:53
kind of log this code in haphazard way within the model
34:56
base, so it's easy to make mistakes within that Python
34:59
logic and it's hard to reuse that logic again again.
35:03
So I think it poses some unique challenges that you don't
35:05
necessarily see from batch or streaming, which are usually
35:08
kind of decoupled outside of the model code itself.
35:12
How are the perils of it? And now one thing that I was
35:15
thinking about is when you're looking at this like 0 to one,
35:23
we could say, or, yeah, basically getting up, getting
35:27
going versus you're going and now you want to scale and now
35:32
you want to go to the moon, per se.
35:35
How do those differ in what you all have seen as far as, oh,
35:42
well, just getting up and getting going.
35:43
You need to be thinking about these things.
35:45
And this is the first stuff like get everything out of the
35:49
Jupiter notebooks and get working.
35:52
And then when you're up and running and you need to scale,
35:56
how do you go from one to N autos and feel confident about
36:01
that, right? I think the easy answer is to
36:04
have a repeatable process to get these data pipelines into
36:07
production. If it takes you months to do a
36:11
data pipeline, OK, that's for one feature set you can't really
36:16
scale. It takes you that long to get
36:19
one feature pipeline in place. So what does Pebble process?
36:23
Yeah. Sorry to jump in there because I
36:26
probably should preface this too, is like, what is that like,
36:31
Eddie, you spend all day looking at people's stacks, right?
36:35
What are some red flags when you're looking at stacks like
36:38
specific things where you're just like, OHH, that's going to
36:41
be hard to do that potentially it's different technology that
36:45
they've got together or like Kevin was saying, there's some
36:47
data access that you have to worry about or there's just like
36:51
the data cleaning in that pipeline.
36:53
What does that reproducible pipeline that takes a month look
36:57
like versus one that takes a day or even like is happening 20
37:03
times a second? Yeah, you know what?
37:07
I'm building out these architecture diagrams.
37:09
I think the more boxes and arrows the the infrastructure
37:14
diagram has, the more red flags I get.
37:19
You don't get extra points for more.
37:21
Boxes, and I think it's exponential.
37:23
Every every arrow that you see the other diagram just
37:27
exponentially. You make things more complex.
37:29
So if I see two different theatre warehouses, if I see two
37:32
different streaming queues, If I see you know, different
37:37
applications consuming or two different ways to generate
37:41
training data. That that just becomes difficult
37:44
and that's sort of the reputability comes into the
37:46
picture, right? How can I do the same way?
37:48
Because as soon as you have success with one model, guess
37:52
what your business partners, your data scientists are going
37:54
to want to ratchet up and build more.
37:58
So you need to have out of the gate a sane infrastructure and
38:02
assembly to get these pipelines up and running.
38:05
And when I look at those repeatable processes, I mean
38:08
there's multiple steps in those you kind of have your analysis
38:11
phase, you have your gathering requirements.
38:14
You author the features, test them, put them into production.
38:17
I'd say like the most common pitfall I see in that end to end
38:21
process is not investing enough time in that design and
38:25
requirement gathering phases and being able to look at not just
38:30
your holistically, not just at your first use case, but maybe
38:33
your next set of use cases. Kind of building out a logical
38:37
model, really focusing on what is the granularity of your data,
38:40
what are the key entities that you're going to be working with,
38:43
and what do I want to have. In each table within my online
38:47
store. And then being able to kind of
38:49
build that out overtime is going to make sure that you're not
38:52
kind of overfitting your features or overfitting your
38:55
overall model to that first use case which is allow you to scale
38:59
as you move forward. And then the other key piece
39:03
would be to focus on kind of again that requirement
39:06
gathering, understanding what is the list of features that I need
39:09
for my model? What align on, what are the the
39:13
needed requirements of that? What is the future definition?
39:16
Associate with them and then actually go out and build that.
39:20
This doesn't help in a couple of ways.
39:22
One is it's not allow you to have those frank conversations
39:25
around, hey, this feature on it may.
39:28
Say I may require 2-3 weeks to develop it, but it's 150 in
39:34
future importance as we look at ROI and we've not be worth
39:37
including this feature on the first iteration of this use
39:42
case. Let's try to get this in the
39:43
production, see if we're going to realise value without it.
39:46
And then the benefit of MP's platform and and a future
39:49
platform is you can iterate quickly and then the other piece
39:53
around that is just making sure that you can change code as you
39:58
move forward. One of the things we see
40:00
counterintuitively is a lot of times migrations of existing use
40:03
case take longer than like a net new use case.
40:07
And oftentimes that's because folks want it.
40:10
They're trying to reverse engineer the requirements from
40:13
kind of the existing legacy coat, which might, to Eddie's
40:16
point, have a million boxes and tonnes of arrows that are all
40:19
interconnected. And then two a lot of times if
40:23
you're migrating and just having this code handed to you, you may
40:27
not have the confidence to change logic.
40:30
On to kind of allow kind of best practises or follow some good
40:34
design scenarios which may again add cycles as you get a use case
40:39
introduction. This is like a super challenging
40:41
problem to me, just to you know, I see this almost telling ticket
40:46
repeatable process, right? But I've seen sophisticated
40:49
teams, big and small. They they have a beautiful way
40:53
to build badge features on top of their data warehouse.
40:56
Requirement comes in to now have streaming features, requirement
41:00
come in, comes in that, OK, now we have data in S3 we need to
41:03
incorporate and that's just, it makes it very difficult to
41:07
continue to augment your future story.
41:08
I think it's even challenging for us to talk, Tom, to build a
41:10
generic multipurpose tool that can solve all these different
41:14
use cases. I mean we've see everything and
41:17
it's still challenging. It's a challenging problem
41:19
solved for sure. One thing that's made me think
41:21
of is like when you have all these different use cases and
41:25
various different data pipelines that would not.
41:28
The idea of reuse and sharing comes up.
41:31
I'm curious how you guys think about the importance of feature
41:34
sharing, data pipe and sharing between different use cases.
41:37
I I mean so when I'm talking to folks in like a pre cells world,
41:41
particularly like. With directors and you know,
41:46
C-Suite folks, they they want feature reuse.
41:48
They don't want their teams reinventing the wheel.
41:52
To drive features in practise. What I see though, is that
41:56
features are very near and dear to data scientists.
41:59
And. Even if the squad this coat
42:03
features calculated, it's just a tiny bit different than the
42:05
other one. Data scientists want to use
42:06
their feature, you don't want to reuse the existing feature.
42:10
So I think, I mean I think a couple of years and I think
42:14
that's just what I've seen across the board.
42:16
Yeah, it takes from customer leadership to kind of online
42:21
setting some future governance. Best practises align on a
42:25
feature definition, even have like the concept of data steward
42:28
applied to the ML world. So that you can have like if you
42:34
say I want to know what is an active customer or what is
42:37
transactions in the last 90 days, I can kind of use that
42:41
that one feature versus having nine different flavours of that
42:45
same feature. And I think 1 requirement if you
42:47
ever want to accomplish sharing and reuse in an organisation as
42:50
you need to establish trust because you live in a data
42:54
scientist identifies another feature owned by someone else
42:57
and they think this thing is amazing.
42:59
How? How will they trust that this
43:02
feature is still gonna be around and it's not gonna be broken
43:05
down the road? Because maybe the use case that
43:08
it originated and they decided to just make some minor tweaks
43:11
and then suddenly you've got this downstream model
43:15
performance outage. And so it's extremely important
43:18
to somehow have a plan in place to actually establish trust for
43:22
these feature pipelines, data pipelines where the ultimate end
43:27
user is gonna be able to. Rely on the fact that if an
43:31
upstream gene change happens, they're gonna be informed and
43:35
they'll get the opportunity to maybe just duplicate duplicate
43:39
the entity as it is in place. And they're not just gonna wake
43:44
up days, weeks, months later to just some model.
43:47
That's not the point anymore. And now you need to find the
43:49
needle in the haystack as to what actually broke it to begin
43:51
with. O that's awesome.
43:54
If I can just reflect back what I think I understood from that
43:58
is the idea here is. You've got.
44:03
Potentially a scenario that gets created where a lot of different
44:07
people are creating. More or less the same features,
44:12
and they don't. Want to use other features that
44:16
are basically the same thing because there's that lack of
44:19
trust there. Or maybe that's a little bit of
44:22
arrogance, but let's just go on the lack of trust thread.
44:25
Let's pull on that a little bit more because if it's not theirs
44:29
and they don't own it, they don't know if it's going to be
44:32
around tomorrow. And if their model is built with
44:34
that, then they're not going to feel confident putting that
44:40
model out into the wild if it's built on somebody else's
44:43
features and they don't have the proper.
44:46
Quality control and justice processes around those features
44:52
staying around. Is that.
44:55
Am I understanding that correctly?
44:57
Yeah, that's. You wanna feature reuse,
44:59
etcetera actually take place or data pipeline reuse to it.
45:03
It's not even isolated, just features and machine learning
45:06
applications. You have to have the right tool
45:08
in place to ensure that. You know, see I ski pipelines or
45:13
any other automated processes in the organisation, it has to be
45:16
automated. I don't think it can rely just
45:18
on human recognising that there is another dependency.
45:21
Recalling it out and sending a slacker e-mail message has to be
45:23
automated. You have to have such tooling in
45:25
place in order to identify these other dependencies and then
45:28
inform them, get the permission or just duplicate basically that
45:31
data pipeline. Otherwise I think the the
45:35
sharing would only happen if there is basically just no other
45:39
way for somebody to recreate the pipeline.
45:42
To get access to the data. Otherwise they'll always opt to
45:45
just create a redundant copy, which gives them a greater Peace
45:49
of Mind. Yeah.
45:52
Yeah, it's such a good point. The thing that I'm going to take
45:57
some questions from the chat, I see one coming through from
46:00
Steven, my man. Do you have a way?
46:03
To monitor the quality of different features that data
46:07
scientists might use And how does that work?
46:11
Is that a question specifically around like what Tecton does or
46:14
is it a general? I think it's, I think it's
46:17
around what you're doing because yeah the do you have a way to
46:20
monitor the quality of different features that yeah, I mean.
46:23
I think you can take it as a How do you do this in Tecton?
46:27
Type question, yeah. So I think this goes a large
46:32
amount towards the trust thing that Kevin was just talking
46:35
about. I the.
46:39
Two different things I hear people talk about when they say
46:41
NFC jerk quality. And one.
46:46
Is is it like, is my feature literally broken and the second
46:50
one is like has it subtly changed over time?
46:54
And people will get really interested in that like has it
46:56
steadily changed over time problem really that's not the
47:00
the why and share the issue. 90% of like data quality problems
47:06
I've seen in ML, both in my time I touched on as well as when I
47:09
worked on a different pricing system to lift was that you're
47:15
like something just breaks upstream, you're an event schema
47:19
changes, some pipeline runs at a different time as I was saying
47:24
and now like. My teachers aren't there.
47:28
Attacked on, we have a system that tries to make capturing
47:32
these really like seemingly obvious broken cases as easy as
47:37
possible. So every time we compute a
47:39
feature, we're generating some basic profile of, like.
47:43
You know, obviously it was how many nulls did you have?
47:45
How many zeros, what was like the mean, men, median, Max, that
47:48
kind of stuff. So we're capturing some basic
47:50
statistics from any feature that's computed and then we also
47:54
apply what we call as validations on top of that.
47:57
So is that, are all these basic statistics within an expected
48:01
range? And even more obviously, it's
48:04
like, let's say a column is now just missing, then we're going
48:07
to fire off an alert to the owner of that feature that says,
48:10
hey, something broken. And what we've heard from our
48:13
our customers is like Tottenham makes up for maybe some derelict
48:21
data quality monitoring upstream and their stack.
48:24
So really again like something else happened in their ETL
48:27
pipeline and. Like maybe there are a lot of
48:31
consumers of that data and organisation, but nobody's
48:33
really watching it properly. And Tecton helps them catch.
48:36
Yeah, my eye, yeah, transactions table is broken because we have
48:41
these basic alerts in there. And you know, just it's really
48:46
important for now because you know, it's one thing if your
48:48
dashboard is broken, you know the director is going to load it
48:52
up and they're gonna send off an angry e-mail.
48:54
But in the MLS case, if your features are broken, then the
48:59
quality of your applications are still great immediately.
49:01
So that's why it was like really important for Tecton to include
49:03
that out-of-the-box. Last one that we've got coming
49:07
through the chat that I want to ask you all, and this one is
49:12
such an awesome question in so many different ways because it's
49:16
basically saying hey. I'm new to this world.
49:19
If you were in charge of a couple of applications without
49:23
even the basic user event tracking setup, and you wanted
49:27
to get a recommendation system up in order, what are the?
49:34
Need tabs? Nice tabs that you would go out
49:38
there and. Get as far as tools and ways of
49:43
thinking about it too. And that wasn't exactly the
49:47
question I'm paraphrasing, but I love the question so much I
49:49
added my own flavour to it. So who wants to start that one
49:55
off? And then we can go around the
49:57
horn because that's great. And just to reiterate, it's like
50:01
you're coming from nothing, but you wanna your end goal is like,
50:05
hey, I want to have a recommender system up and
50:06
running. What do I do?
50:09
How do I like incrementally get there?
50:12
Yeah, no, I guess error this one.
50:15
The first thing I do is, like you know, separate the product
50:18
goal, which is to recommend relevant products to your users
50:22
from the implementation of it. All I had to say, like you you
50:26
might not be. I'm almost ML model for this.
50:30
You do need that data, so I would focus on the data side and
50:34
obviously. Power at that time and covered
50:37
it like, but maybe you have a simple heuristic which is like
50:41
let me I'm just gonna recommend like the top three purchase
50:44
items in the last week. And if you go with that type of
50:48
approach then you can get a lot of those fundamentals in place,
50:52
which is going to be like getting all those blinds and
50:54
everything and without having to start to go into.
50:58
The additional complexity of actual machine learning models.
51:03
Now obviously that is the place where you want to go to to
51:06
maximise your performance, but I think that's something that can
51:09
be can be layered on later when you want to add the additional
51:11
like 20% gain. Yeah, I would definitely start
51:14
with the data. If it's, you know, just be
51:16
myself and I doing this, I would definitely make sure I have my
51:19
data in order and I personally would just use like an off the
51:23
shelf recommender API from one of the big clouds probably stuck
51:27
there. And then, you know, once I see
51:29
some lifts there, I think definitely tuning your own
51:31
model. Is the end goal.
51:33
That's where you will get the best returns, but I would kind
51:37
of leave that down down the road a bit.
51:40
Yeah. My my last call out on Rexis and
51:43
specific specifically is would be to start with batch.
51:47
You can get really really good performance with Rexis just with
51:50
kind of a batch data pipelines running maybe on a weekly you
51:54
daily, monthly basis and then move to and once you have enough
51:59
scale to get a return on real time move to real time with
52:03
Rexis use cases and in particular that lift from batch
52:07
to kind of handling real time on the handling.
52:10
Some of those cuts are problems with recesses, engines is a is a
52:14
large leap. So again, start, move up that
52:19
crawl, walk, run slowly, Make sure you don't skip a a step and
52:24
jump to real time. That said, I think building it
52:28
using real time is now easier than it's ever been before.
52:31
And like there's more off the shelf tooling now that we can
52:33
use to make it easy to retrieve the relevant candidates from say
52:39
an embedding store, which could even be as simple as your
52:42
postgres. And then narrow it further down
52:44
using some philtres and then narrow it further down into some
52:48
racket using machine learning. Like there is enough blueprints
52:51
now online and enough managed software solutions out there now
52:56
where you can really just stitch together.
52:57
The two different marriage solutions if and you have a
53:00
pretty flexible and powerful real time recommendation system.
53:05
That said, still start with a juristic to get something out
53:08
there, but it's not as crazy of an undertaking anymore to build
53:13
something in house. You have just a team of one or
53:15
two folks. Then it would have been to even
53:17
two years ago. Yeah.
53:19
So I'm going to try and like throw all of it together now and
53:24
I think it's, it's brilliant A looking at the business side of
53:27
things, you're going to want to be able to quantify what you're
53:30
doing as you go along, so you can get more buy in from the
53:35
other side of the house where the company, right.
53:38
So if you're doing what Eddie's talking about, or Isaac's
53:41
talking about where you're going gradually there, you want to
53:45
make sure that, hey, can we actually prove that this is?
53:50
Being valuable to the business. And how do I prove that?
53:54
Alright, cool. I can show that there's some
53:57
incremental stuff here. And then if you wanna go all in
54:00
and say we've got something now, let's really make it much
54:05
better. Then you can and you ideally
54:09
will have that buy in because you've been able to prove it
54:13
out. And Kevin, I gotta ask man, like
54:15
the last thing you said with two tools, 2 managed services you
54:20
could get there. Besides Tecton, what's the other
54:23
one that you would pair it up with?
54:25
Like what would you say? Have the planets database.
54:29
And if that's already part of your stack, like you need a, you
54:32
need to play somewhere you can run your model, and that can be
54:34
as simple as honestly using Sage Maker Model Survey.
54:38
Excellent. I think that is a great place to
54:42
end it fellas, this has been awesome and all these tech tone
54:47
guys gotta go get back to work. They gotta go keep building out
54:52
the feature platform. Thank you all for joining us and
54:55
if anyone wants to chat with the Tech Tonne crew, we'll leave all
54:59
the information. I'll throw it all in the chat
55:01
right now. And yeah guys, this was awesome.
55:04
We definitely went above and beyond my expectations, so I
55:09
appreciate that. Hey, this is Mike Delbalso,
55:16
Cofounder and CEO of Tecton. ML OPS Community is the best way
55:21
to stay in the loop on the latest ML OPS news and best
55:24
practises. It's also a great way to connect
55:26
with experts and get support from an amazingly helpful
55:29
community. Subscribe and stay in the loop.