Latest / Agentic Conversations (formally mlops.community) / Let's Continue Bundling into the Database // Ethan Rosenthal // MLOps Coffee Sessions #131

Transcript

Let's Continue Bundling into the Database // Ethan Rosenthal // MLOps Coffee Sessions #131 · Agentic Conversations (formally mlops.community)

0:00
Hey everyone, I'm Ethan. I am an AI engineering manager
0:03
at square and I drink just regular old.
0:06
Black drip coughing. Mike welcome man.
0:10
You are our guest host of modern today.
0:13
It's great to have you here, man.
0:14
Thank you for having me. Soho, we just got off of an
0:19
interview with Ethan Rosenthal. He is working at Square.
0:23
He's doing some really cool stuff and I thought it was
0:26
timely around the blurb that you just wrote.
0:30
About large language models. He was talking about how they're
0:32
using large language models. All that stuff was awesome.
0:35
Then we drove into a little feature store conversation and
0:38
his hopes and dreams for feature store / Monitoring Solutions.
0:42
What were your takeaways? First of all?
0:45
He didn't super. It's like a privilege to talk to
0:49
someone who's so, so such a clear thinker, but also kind of
0:54
like, so deep in all of the technology and the
0:56
organizational things that are really needed to get ml into
0:59
production, So is a great convo things that stood out to me, the
1:04
relationship between the AI engineers and the product
1:06
managers. It's something we don't talk
1:07
about very much, but you brought it up and he was like you got to
1:10
get this, right? If you don't get this right then
1:11
either the you're not focusing on the business problems in the
1:14
right way or that PM's are not going to understand the tech
1:19
appropriately to help make the right decisions.
1:22
The other thing that streaming database is what is their future
1:25
role in things like ml monitoring and feature stores
1:29
thinking about, Out some of the like that the tech implications
1:33
for the future. Stack of ml is really
1:36
interesting, so I thought it was a great combo.
1:38
Super, he's just a sharp guy and you guys haven't back here
1:42
again. And again, what about you do?
1:45
Would, what did was set out to you?
1:47
So for me, I actually really liked this idea that you
1:51
presented. It's a little bit off, the
1:54
beaten path. There was, of course, everything
1:57
that you said just now about Ethan and I knew you were going
1:59
to call. The product manager piece being
2:02
annexed product manager. Right?
2:04
You're like yes we need more credit earlier but but one thing
2:11
that I liked you talking about in and bringing up the point of
2:15
is the trade-offs going from very specific model use cases to
2:20
like and using many different models to one model that you're
2:24
just throwing everything at and how you have to keep in mind now
2:29
and kind Of level up and see things from a different place in
2:33
the business. So that was a huge.
2:36
Takeaway for me because I hadn't heard a that talked about like
2:40
that transition from using hundreds or thousands of models
2:44
for very specific things to using bigger models that you
2:49
just throw. The idea is like, instead of
2:51
having like even an Uber as an example, we had one model that
2:55
for every single city, right? We had.
2:56
So we have like six hundred models, we were managing and
2:59
that was just for us. Use case a for ETA prediction,
3:02
but what you thing was talking about is not just bringing all
3:05
of those kinds of models together.
3:06
But bring multiple models that do different things together
3:09
into one big model. Yeah.
3:10
And then the question is, okay, now we want to launch a new
3:13
version of this model, it's doesn't only affect one use
3:16
case. It affects all these other use
3:17
cases to. So now we have to make
3:19
trade-offs across like areas of the product and that leads to
3:23
new and interesting discussions. So which is I'm sure that's
3:27
going to come up again and again.
3:29
Yeah. And it makes Smee think that's
3:30
probably why his his love or a little bit of wear his love for
3:35
monitoring stems from here. You can tell that he is.
3:38
I mean, he, he tried to build and 2019 his own monitoring
3:42
solution and or he did build it. And then he ended up joining
3:48
Square. So that seems to me like a
3:51
little bit of a precursor for now what he's doing.
3:55
So that's awesome. Let's get into it with Ethan
3:59
without further. Or do we're going to do it.
4:01
And by the way, if anyone out there is listening and they have
4:05
not signed up for our newsletter that we just referenced
4:08
throughout this episode, you can do so at MLM stock Community or
4:13
in the link in the description and we would love it if you can
4:17
give us some feedback or stars or follow a like all that
4:23
whatever you can do to let the algorithm know that this is
4:26
where it's at. And this is a cool podcast, we
4:29
super appreciate that. At now, let's jump into it.
4:38
All right, let's get this. Let's get this crackin.
4:43
Ethan, Mike both of you. It's awesome to have you both
4:46
here and Ethan, especially like I want to dive into what you're
4:50
doing at Square. I want to know everything that
4:53
there is to know about your team, the way that you all are
4:57
operating and a bit of your history on that.
5:00
I know you mentioned that you're on the fraud team and now you're
5:03
doing new stuff. So let's just start with like
5:06
what team? You on good question.
5:10
Yeah, so I'm on a team called the conversations team.
5:13
We're like this small little team.
5:15
We built a product called Square messages, which is a messaging
5:19
hub for square, businesses to be able to communicate with their
5:22
customers. So you know, maybe you let's say
5:26
sign up for an appointment at the hair salon you might get a
5:29
text message, confirming your appointment.
5:32
You can then respond to that text message and that message
5:35
will go into like all these Text messages go into one place where
5:39
the business owner can read through them and respond to
5:41
them. So text emails, other things
5:43
like that. Yeah, so my team builds that
5:46
product and then I run a small team of AI Engineers who add.
5:52
Let's call them like smart features to this.
5:54
So we have a chatbot. So if you want to, let's say,
5:58
reschedule an appointment that you have scheduled, you can talk
6:02
in natural language to this chat Bots and it will, you know, work
6:05
with you in order to Schedule which saves the business owner
6:09
from missing out on sales and things like that.
6:12
We also have a feature called suggested replies.
6:16
So you know how like Gmail will suggest a response for you to
6:20
send back to an email. We do the same thing for
6:23
business owners so that they can quickly respond to their
6:26
customers. So you can imagine lots of
6:29
natural language processing, you know, language models which are
6:33
pretty hot right now and other sorts of things like that.
6:36
We work on. And Ethan, we you have when you
6:40
work on a team like this, it was it was interesting to me how you
6:43
describe it. You said we'll work on some
6:44
smart features and and we work with a bunch of teams who
6:48
describe themselves like that as well.
6:49
It's not just like, I'm the fraud team and I detect fraud.
6:53
It's I'm on this product team and I'm trying to improve the
6:57
product and a whole number of ways.
6:59
And anybody on my team might end up working on it.
7:01
Any of, you know, that in many different opportunities we have
7:04
to improve the product is that, So one is that accurate, is that
7:08
how you think about it and then two is that something like like
7:12
of a trend that you're seeing where people are moving more to
7:15
just being like the AI engineering crew on a product?
7:18
And we work on a lot of different ways to improve it
7:20
rather than just this one really high value use case.
7:24
Yeah, I guess it probably depends on your team and I'm
7:27
guessing it depends on your relationship with product
7:30
managers, right? So I think there's definitely
7:34
been teams that I've worked on where let's say, Kpms have
7:38
features that they want built. And so you have let's say a lot
7:41
of inbound requests to the i-team and it's the i-team's job
7:44
to kind of solve for these things.
7:47
The, the flip side of this is that, you know, often
7:51
non-technical stakeholders might not know what Ai and machine
7:54
learning are capable of. And so, then the, the NL team
7:58
needs to come up with these ideas themselves and my team
8:01
kind of sits in the middle somewhere.
8:03
So there's a kind of obvious business use case.
8:07
Is where we want to inject some smart features.
8:09
But then there's also lots of kind of Greenfield areas where
8:14
we are coming up with the ideas of what we can do.
8:16
And then we, then we work with product people, we work with
8:19
designers and things like that. In order to understand can we
8:21
actually build out a product or feature out of these crazy ideas
8:25
that we have? That's pretty fascinating how
8:30
that relationship with the product.
8:32
Managers is so important because you may be in AI ml engineer on
8:38
a team and your product manager may just not get it and it's
8:42
probably pretty hard for you to be able to get the same type of
8:48
impact. It makes me makes me just think
8:51
about like how important the product manager role is on these
8:56
types of products where it hasn't been necessarily like the
9:00
primary Persona, we're talking about my speaking like a true
9:04
product manager. I have a product So I was like
9:08
bring this up it's usually important.
9:11
I think it's probably easy for engineers and data people to
9:16
kind of scoff at product managers and things like that.
9:18
But I work with a guy I work with a great product manager
9:21
right now and it is so freaking helpful compared to other
9:26
scenarios that I've been in where you don't have great
9:28
products support. But it's I'm sure Mike that
9:30
you're aware. It is very hard to find product
9:32
managers that are able to like, I think product is hard enough
9:35
just for regular regular. ER, quote software Engineers,
9:39
but then to start to work with, with data people.
9:41
Ml people, it's just even harder to find to find good people.
9:44
It's their super crazy. Yeah, we're hiring PM's and
9:49
tecton, but ya know, it's so hard.
9:51
Like it's because you need a date.
9:55
You need a certain level data science, intuition, you need a
9:58
certain level of engineering. It's not just, you need to be
10:00
technical somewhat. Technical you need the data
10:03
science intuition as well. So that scope sound the
10:06
population, And a big way. And the thing that I always look
10:12
for in a product managers just that they're really, really,
10:14
really smart and they can make good decisions and so that isn't
10:17
even bigger filter. That's why it's so tough.
10:20
So it's good that you're working with the right person right now
10:24
got lucky on this team. Yeah, that's awesome.
10:27
So you insinuated at it I want to know because especially
10:34
because Mike just wrote something in the mega Ops.
10:37
Newsletter that we have that went out a day or two ago.
10:42
Large language models. Is it all hype?
10:45
Where do you see them playing into your workflow and how like
10:50
you actually get work done? Yeah.
10:52
So we fall on use them, we train them, we deploy them and they're
10:59
pretty, pretty incredible. I don't know.
11:01
So I think, I think we're kind of going through, I don't know,
11:04
an inflection point or something like that.
11:07
We're lots and lots of applications of natural language
11:11
processing which might have been, you know, you might have
11:14
had different bespoke solutions for different tasks.
11:17
Let's say, question-answering or text classification or things
11:21
like that. These can kind of all be solved
11:23
by large language models and possibly by the exact same large
11:27
language model. And so we it's kind of a
11:31
constant question on our team. How much How many different
11:35
let's say use cases of a model? Do we want to fit into a single
11:39
model? Yeah.
11:40
Nice. You know there's there's
11:42
benefits there but then like this is the envelopes podcast,
11:45
right? Like there's like operationally.
11:48
It's maybe easier to have one model but then and maybe we'll
11:51
talk about this later. Like, if you want to start
11:52
automatically, retraining this model and you have multiple
11:55
tasks that it makes life a lot more difficult.
11:58
But it's it's a very fun area right now and I feel like
12:02
there's, you know, it's It's unclear exactly how much we can
12:06
we can shove into single models and be successful with, but
12:10
yeah, that's interesting. Just to clarify, i d made it
12:13
sound like I wrote something that said, it's all hype and
12:16
that's not the, it's just that you just throwing you under the
12:20
bus word and it's a way that's not at all what I was saying.
12:22
But yeah that's that's fascinating.
12:25
I think was one thing, you know, playing around with like G PT 3.
12:29
That I've noticed I've been messing around with it to
12:33
generate For Dolly and stuff like that.
12:35
Just looks like a side thing but, you know, it gets things
12:39
totally wrong, pretty frequently.
12:41
And so, this whole notion of prompt engineering is really, I
12:46
believe it's going to be like a very huge thing and it's kind of
12:48
just, you know, something that we just so new that we don't
12:51
talk about it too much. But the my my realization is
12:55
that like it's pretty easy to have just like totally wacky
12:59
like outputs from these large language models and from a
13:03
product Manager perspective that makes me think about like what
13:06
use cases is something like this at the current stage of this
13:10
technology applicable to. And I think the types of use
13:14
cases are ones where mistakes are not really high cost and so
13:19
I wonder if that's something that you've your team has been
13:22
thinking about like when we're figuring out where we can apply
13:24
this stuff. Let's focus on the ones where if
13:27
we get it wrong it's not that big of a deal as opposed to like
13:30
I don't know something where, you know it's some fraud thing
13:33
or whatever. Four million dollars.
13:35
Is that the kind of conversations you're having
13:36
internally like how are you, how are you taking this new
13:39
technology and then figuring out internally?
13:42
Like how do we apply? What are the big questions?
13:44
You're figuring out. Yeah, yeah, so that's those are
13:47
great points. We, you know, we can put guard
13:50
rails in place, right? So you know like right now we
13:54
suggest a response so a string of text that we think that the
13:59
that the merchants can send back to their customer and we A bunch
14:04
of filters. After that point in order, to
14:06
make sure that we're not showing suggestions that are obviously
14:09
dangerous or bad. So, for example, we don't want
14:12
to leak Pi, I into the suggestions that were
14:16
generating. And so we have detectors for
14:18
that and we make sure to suppress any, any text, that
14:21
might contain pii and things like that, you know, models are
14:26
trained off of bias data at. So we don't want to start
14:29
manifesting that those biases as well, but At the end of the day,
14:34
you know, this isn't, this isn't perfect.
14:36
And I think one big question for us in the team is how do we want
14:41
to deal with the human that is in the loop of our system?
14:44
So I mentioned that we have a chatbot that chat bot is
14:47
basically fully automated if the chat bot, can't handle
14:50
something, then it alerts the business owner to have them, you
14:54
know, swoop in. But for you know, if we are just
14:57
suggesting a response to say it's kind of okay if it's not
15:02
great, right? You know nobody's gonna To lose
15:04
lots of money. Like I used to work in
15:06
recommendation systems and, you know, I worked for like a, like,
15:10
a beauty products company and it's like, all right, like if
15:12
you get suggested the wrong lipstick, like you're going to
15:14
be okay, but then, as you mentioned for fraud use cases,
15:18
you it's much more important. And so I think that how much you
15:22
rely on the human in the loop is a way to kind of navigate.
15:26
How much you're willing to allow the for like a see how
15:31
conservative you have to be with your models.
15:33
The and these are the just a ring back to the thing we're
15:36
saying before, this kind of understanding of the cost of a
15:39
false positive or something like that is something that is very
15:44
important for the product manager to be able to think
15:47
through with a really deep intuition.
15:49
And so that's one of the things we're always evaluating.
15:53
Like do you get it? Really?
15:54
Because it's a whole new domain of dimension of like, of
16:00
product, understanding and Technology, understanding that
16:02
you have to That really affects the, it's a primary factor in
16:07
the end product experience. And and so like how I think like
16:13
how the product teams and like your team work together is, is
16:18
kind of like this new thing and sounds like you guys are
16:21
figuring it out, but a lot of people are are still just there.
16:25
Like so like we don't even have a product manager, who knows
16:28
anything about this, but they're taking it, they're taking
16:30
Ethan's course, right now, it's something like that.
16:32
What? What advice would you have for
16:34
other folks in there? Kind of like, I'm an ml engineer
16:38
and I got a product manager who doesn't like was not an expert
16:41
at this stuff. How can I work really
16:44
effectively without person? Yeah, I I think one key pieces,
16:50
you know, stepping out of your narrow domain and trying to
16:53
explain things in the context of let's say non-technical
16:57
stakeholders. And so you know, one piece is
17:00
defining your metrics in terms of business metrics, as To model
17:04
metrics, you know, I most like 99.9% of the people in the
17:08
world, don't know what the area under an Roc curve is you know
17:11
at so like you just shouldn't even be telling somebody that I
17:14
it's not to say that you shouldn't try to teach people
17:17
things. So I used to work on the risk
17:19
team at square and lots and lots of people across the broader
17:24
risk, Oregon, understand what precision and recall curves.
17:26
Look like because these are metrics that they kind of have
17:29
to live and breathe, but not all teams are going to be like that.
17:32
And so I think, you know, I think, I guess it starts with
17:36
metrics probably and it's sometimes hard for ML people to
17:39
be able to think through those product metrics.
17:41
But I think how your model interacts with the full
17:46
ecosystem that it's in not to get too like a thought leader e.
17:51
But I think that that's like, you know, over the next five
17:53
years. Like that is where the real work
17:56
is involved. You know, it's getting easier
17:58
and easier for us to build models deploy them and
18:00
everything else. But actually, you know,
18:02
understanding Interaction in the ecosystem is it's a lot of work.
18:07
We can't really get around it. So you know I was going the
18:10
direction of like how do we get? You know, I'm a product manager
18:14
who understands the business but not the technology to be able to
18:17
work you know better with the technical people but also you
18:21
know your point is like the technical people need to
18:24
understand if they're going to be affected.
18:26
They need to understand the business at the end of the day.
18:28
Both sides. Both sides have to meet each
18:30
other in the middle. Yeah, that's important.
18:33
Yeah. Reminds me of when we talked to
18:35
harness who worked at this company called digits and he was
18:42
saying how his whole team like they get together with the front
18:46
end and designers a lot, just to know what kind of things they're
18:52
looking for, and they're interested in and then if
18:55
they're changing things, they also get that those signals and
18:59
they get updated beforehand or ideally like there.
19:03
All on the same page when things are being changed.
19:06
So that in case anything breaks because of a change, they know
19:10
about it and it doesn't break because they get that
19:13
premonition. And so it's a little bit like
19:16
that. Like they have to be able to
19:19
speak to each other in a way and what you're saying, makes sense.
19:22
It's kind of follows that same line of thought, where you want
19:26
to be able to speak to a product manager.
19:27
You want the product manager to be able to speak to you.
19:30
And so, how can you both find that?
19:34
That overlap in the Venn diagram understanding.
19:38
Yep. Exactly.
19:39
So I feel like everybody has different vocabulary terms for
19:42
the same things, right? That's so even just a great on
19:45
that. I mean, I, you know, even a
19:47
machine learning, you have statistics, you have computer
19:49
science, they have the same terms for the same, you know,
19:52
inference versus prediction and everybody interprets these
19:54
things differently. And so, even just dealing with
19:58
the vocabulary as hard sometimes and and Ethan you touched on
20:02
like some of Analogs concerns around this.
20:05
And you know that's one of the things I wanted to ask you
20:07
about. We're so Saudi was just saying
20:11
to make sure this was just saying.
20:12
Hey, we publish this ml Ops Mega Ops newsletter the other day,
20:15
and one of the things that I wrote a little snippet and it
20:19
was about like, hey, there's large language models and I was
20:23
talking about like, the text to image stuff, particularly, but
20:26
you know, this stuff's really useful, and we're super early
20:28
days, and I think the large language stuff is farther along.
20:33
Long in its maturity and adoption and Industry than like
20:36
the text to image stuff, but that it's still early.
20:39
And so while we have all these well-known ml years of ml, Ops
20:44
best practices that have accumulated in ml Ops Community
20:47
where somebody can go and take a course on this is how this
20:51
company or this company runs, you know their decision Tree in
20:54
production and this is who does what and stuff like that.
20:58
It's my understanding that how to do that with large language
21:02
models. Has is still very early, both in
21:06
what people's responsibilities are and what are the best
21:09
practices for training and retraining and monitoring the
21:15
stuff in production and stuff like that.
21:17
Are you dealing with? So now coming from a fraud team
21:21
to this team. Are you dealing with
21:23
fundamentally new ml W challenges and what kinds of
21:27
challenges are presented if so yeah, I think that We're
21:33
definitely dealing with new challenges.
21:35
I think one of the big challenges is trying to fit lots
21:39
of use cases or tasks, whatever we want to call this in to the
21:44
same model and so that's interesting you know like a lot
21:48
of I'm sure that for people who have maybe a more robust ml app
21:53
setup or something like that, they've built out a framework or
21:57
an abstraction layer around how their models are used and maybe
22:01
they have mapped. You know this model can be used
22:04
for this use case or something like that.
22:07
And we might version We Wait version an entire model
22:11
artifact. But if you start to combine
22:15
multiple tasks into like one model, then you can imagine
22:21
that. Like, you now have multiple use
22:25
cases that are mapped to the same model.
22:27
You might want to only update portions of this model.
22:30
So, like fine-tuning is a popular.
22:32
With language models you might only be fine tuning a portion of
22:36
a model, maybe it's just a Transformer head that pops off
22:39
of a large backbone. And so how do you start to
22:42
version these like, like these components?
22:45
These artifact components that end up, you know, being combined
22:48
into one thing? How do you, how do you version?
22:52
Let's say, like, pre-trained embeddings as opposed to
22:56
Downstream tasks and how do you. And if you are updating these
22:59
embeddings, then you now have this cascading effect, we have
23:02
to update all the Downstream tasks.
23:04
And so there's a part of me that's like like this is just so
23:07
painful you know, like life is easier if you just have like
23:10
simple models for each task. Although if you're deploying
23:15
simple, models for every single different tasks.
23:17
And there is like an infrastructure overhead to
23:19
deploying, all these things as opposed to deploy one model.
23:22
But it's like the moment you change.
23:24
Anything is when, you know, the moment you update a model is,
23:27
when life gets hard and there's some cost to, there's always
23:30
some benefits to centralization and some Stu it.
23:33
And it's and so it sounds like you're kind of like hitting some
23:37
of those trade-offs. We dealt with some similar
23:41
things at Google used to be a product manager on the machine
23:45
learning team for Google's ad stuff to shoot to choose which
23:49
ads. You see when you type something
23:51
into search and we had that we went through this like big
23:55
sounds very related to what you were saying.
23:57
This big change from having a lot of models where either would
24:01
be a model per country and a Model / desktop, or mobile kind
24:04
of thing to the one Mega model and it was like, you know, now
24:09
we just have this one model, maybe you have a different
24:11
versions of it to manage, but there's just this one big model
24:14
and we don't have to worry about like a thousand different models
24:17
that were managing. And who were we actually had the
24:19
thousand different models and they went through the similar
24:22
thing of, like, centralizing things.
24:24
The benefit of that centralization was.
24:27
Yeah, there's less models to manage, but secondly, the model
24:30
can kind of like learn better. Like that just has more data so
24:35
it can learn things that it can apply to one region that it
24:39
learned from another region and if you set the model up
24:41
properly, you can actually just have that the model learn like
24:46
region specific things. But we had to do a lot of like
24:49
careful construction of the models in those times.
24:52
This is kind of like pre deep learning for for a lot of the
24:55
stuff that we did a Google. How do you so?
24:59
So in the world that you're in, when you think about Out like
25:03
we're going to have this one thing that's going to be useful
25:05
for a variety of different tasks.
25:08
What kind of work do you have to put in to manage the kind of
25:12
like, what data can be affected can affect learning the what
25:15
kinds of learnings and how the behavior can be different across
25:19
various tasks? Does that question?
25:21
Make sense? Yeah.
25:22
Yeah. No.
25:23
I mean I think it's you know, that's one of the issues, right?
25:26
Is that your tasks can affect other tasks?
25:28
And so when you have a single model I can just focus on
25:31
metrics for that model but Now if I am training a model on
25:35
multiple tasks, that maybe this new model is better for certain
25:39
tasks, but worse for other tasks and so kind of making sure that
25:45
like I guess you're a taking a portfolio approach to model
25:48
Improvement or something like that ends up being important.
25:53
But I think it's also just a trade-off from like a
25:56
operational standpoint around. Like, you know, is it actually
26:00
worth it to try to? Fit this task into this other
26:04
model or would life just be easier if we train this task
26:07
separately. I do think that especially for
26:09
natural language processing. The ability to do like few shot.
26:12
Learning is extremely appealing for large language models.
26:16
So we have a number of these. You want to explain what that is
26:20
for everybody. In case it's kind of new.
26:21
Yeah yeah. So like a classic natural
26:25
language processing task is to do classification on a piece of
26:28
text and so typically you need some labeled data in order to
26:32
Last fight that text and it can be expensive to collect this
26:36
data. So we have humans annotate lots
26:39
and lots of data points for us that we can train supervise
26:41
problems on there. But the idea of a few shot or
26:45
zero shot learning is that you might only need a couple labeled
26:49
examples for a model to be able to learn that classification
26:53
tasks and large language models have shown that they're pretty
26:58
good at learning from very few labeled examples and so Not only
27:03
does that reduce cost because you don't have to send out a lot
27:07
of data for labeling, but it reduces a lot of operational
27:10
headache around collecting quality labels, and things like
27:13
that. And all of a sudden now, you,
27:15
you have opened up, you know, the ability of your model to
27:18
start to solve lots and lots of different tasks.
27:22
Yeah. It's a what came to mind as you
27:25
were kind of talking through some of the trade-offs that you
27:28
have to make is that, you know, in any team that's deploying
27:33
models, there's there's rarely just as one number that they're
27:37
optimizing for it's rarely just like oh we're just increasing
27:42
revenue and that's all we care about because you know maybe
27:45
your increasing Revenue short-term Revenue but you're
27:47
increasing unsubscribe rate and you know it's going to kill you
27:50
in long-term revenue or something.
27:51
So every business As their own. Like, people ultimately are
27:53
looking at a number of a number of metrics within their use case
27:59
that that there's always conversations between all the
28:02
different people in the team around.
28:03
Like how do we really trade these things off and what is
28:07
kind of an interesting? It sounds like it's an
28:09
interesting new challenge that your team is dealing with and
28:12
presumably other teams in your position.
28:14
As these trade-offs are not what's the right way to say it
28:19
intra team. But now you have Coughs that.
28:22
You know, we're evaluating this new model but it's going to
28:26
affect all this other teams, use case works.
28:28
Now because it's this model affects all these other teams
28:31
different like all these different use cases at the same
28:32
time and so that really requires someone to be able to make
28:36
trade-offs at a little bit of a higher level where you're able
28:39
to say. Okay well we're getting this
28:41
capability like I'm just making these use cases up alike.
28:44
I'll replies to the businesses or to the customers are more
28:47
accurate but the like the troubleshooting language Kind of
28:52
like less accurate. Now, are we okay with that
28:54
change? Is that are you seeing like the
28:57
kind of level at the trade-offs that you have to make a change
29:00
now as well? Yeah, I mean to be clear we are
29:03
we are early in our in our journey was early.
29:07
So like we have combined some things but we have like not
29:11
everything lives. Not all of our use cases yet
29:13
live in a in a single model. But but yeah.
29:16
I think like that's that's totally true where you have to
29:19
now Yeah, not just navigate the difficult metrics for your
29:24
single model, which as you said are, you know, could involve,
29:27
you know, well has this metric gone up?
29:29
You know, constrained by the fact that this other metric has
29:32
not gone down below some level, but now we have this across
29:36
tasks and so and then and that's not so bad for across tasks, but
29:41
you can also Imagine just even for an individual, even for an
29:45
angel individual task, let's say that we are generating text, we
29:49
might be generating text better for certain Sets of our
29:52
population than other subsets. And so and each time you retrain
29:56
your model that the distribution of your performance across your
30:00
population can change. And so how do you make sure that
30:05
you're kind of maintaining sufficient performance for?
30:09
Let's say protective subsets. This is Skylar, I lead machine
30:16
learning and health rhythms if you want to stay on top of
30:20
everything happening in ml. Ops subscribe to this podcast
30:24
now. So I want to change gears a
30:31
little bit because you've written some incredible blog
30:34
posts and one of which picked up quite a bit of steam and I want
30:39
to jump into the The unbundling / bundling and this one
30:45
specifically is all about the unbundling of airflow.
30:48
I think is what it was that you and correct me if I'm wrong.
30:52
And I say I just wanted to say something on that blog post as a
30:56
way to just tear up my own blog post.
30:58
So there you go. That's it.
31:00
And so I'm I'm wondering when it comes to the unbundling, Andre
31:06
bundling argument especially on this building performance.
31:11
Databases which is what your blog post goes into right.
31:16
Where do you stand on it? Now have things changed or you
31:20
still. You still gave me just summarize
31:22
the point for everybody as well. Yeah, that's probably good.
31:25
So I think I still agree with what I wrote a couple months ago
31:28
but that's you know asked me a couple bucks and I'm sure
31:30
that'll change. But the main argument of the
31:33
blog post was that there's kind of a couple different areas of,
31:37
let's say an elapsed tooling or data tooling nowadays.
31:42
Let's say feature stores, machine learning, monitoring
31:46
tools, and I guess metrics layers, which I admittedly don't
31:50
know much about. And I feel like they're also
31:52
just worse defined than the other two tools.
31:54
But the point of the blog post was that each of these different
31:58
kind of specific use cases are all kind of generalizations of
32:02
the same idea. And that presumably with modern
32:07
streaming databases we can kind of build out those tools as
32:11
Opposed to needing specific vendors for this.
32:14
Not to say that we don't necessarily need vendors for
32:17
this but yeah. And so the overarching idea was
32:22
that like if you take something like a feature store or
32:25
monitoring machine learning models, there's kind of like two
32:28
key components to this. So one is that you need to be
32:32
able to serve up kind of aggregated Statistics quickly in
32:37
real time. So, for a feature store, maybe I
32:39
need to know. You know, how many transactions
32:43
has this Merchant processed in the last 60 seconds?
32:47
Maybe that's a feature, that's interesting to me and I'm using
32:49
that in order to detect fraud or something like that.
32:52
And so your future store needs to serve up that number very
32:55
quickly. And so the feature store might
32:58
be consuming streams of events and then we are aggregating
33:01
those events into a single number that we have to serve up
33:05
fast. Machine learning monitoring.
33:06
Same idea, it's hard to talk about the performance metrics of
33:10
a single mom. Prediction, but you might look
33:12
at, you know, what is my models accuracy over some time period?
33:18
Let's say like a trailing like one week accuracy, perhaps.
33:22
So that's one piece, is that you have to be able to serve up
33:25
these aggregated statistics fast, the other piece is that
33:28
you typically want to be able to understand what were those
33:31
statistics values at previous points in time.
33:34
So, for a feature store, maybe you want to train your model off
33:37
of historical data and your historical data needs to be.
33:41
You know, maybe you need to know for that transaction.
33:43
That was processed a year ago, how many transactions had that
33:47
Merchant processed in the prior, you know, our and so.
33:54
So to support both of these systems, it's really really
33:58
complicated. It's like very, very painful
34:00
from infrastructure standpoint. I'm sure bike is as well aware.
34:04
There's a tire companies that do this at their very necessary.
34:08
But I in my blog post, I talked about this, Like these new
34:11
streaming databases like for example materialize where you
34:17
kind of get this support out of these databases.
34:20
And so when I'm kind of putting out to the world as is maybe we
34:24
can start to build these sorts of tools on top of these
34:27
databases and maybe it'll work. Maybe well I've been playing
34:32
around but nothing nothing big yet so I'll pause there.
34:36
Yeah. I think it's totally could work
34:37
and it's one of the things that we've had to be like and as
34:41
we've been building tacked on we're keeping a close eye on on
34:46
all of this street. Like streaming is a really big
34:48
use case for feature stores because it's one of the hardest
34:52
things to get, right? And so it's it's not a thing
34:56
that we see as hey now you can replace your features door
35:03
completely with this thing. But it's more like, hey, the the
35:06
data processing engine for a pretty big Subset of features
35:14
feature use cases, may be replaceable by some, you know,
35:20
like we may have like the features for plug into streaming
35:25
database. What we found though is that
35:28
they're all really early so we haven't.
35:31
We haven't like really like we wanted to this is kind of just
35:35
like a little bit like what's happening in practice right now.
35:38
We want to use them because it just the promises.
35:40
So So awesome and your job easier.
35:44
Yeah. I mean, like I can't tell you
35:47
how because we take on a lot of complexity for our customers.
35:50
Our customers come to us and they're like, hey, streaming
35:53
features kills us. So can we use your feature store
35:57
and just make it way easier for us, right?
35:59
And like, you know, square is an example of that.
36:03
And and so we hope we out them do that.
36:07
However it's it's not like streaming.
36:11
Solved now streaming is still just as hard.
36:14
We just have the ml wrapper around and a lot of management
36:18
and simplification stuff that we provide.
36:20
So it's a much simpler API for data scientists behind the
36:24
scenes, we got to deal with all the complexity and so we would
36:26
love if there's a really nice streaming Engine That Could
36:30
support like these use cases, but to meet and I think they're
36:35
getting there and so I think very soon we're going to see for
36:38
a lot of these use cases that kind of engine be.
36:41
Be able to be useful for a lot of feature story use cases, but
36:45
a lot of the like if someone if you're listening to this from
36:47
like, you know, a big tech company.
36:49
Your company with like a really large-scale.
36:52
We've been also trying to use explore these for those use
36:57
cases and the kind of like latency and scale needs is I've
37:04
been a little extreme for those in a way that has pushed the
37:08
streaming data bases outside of their like Comfort zones.
37:11
It's kind of early to apply those to the to the larger
37:14
companies, but it's going to get there and we're really
37:17
optimistic about it. So a so we're, you know, we're
37:20
keeping an eye on them and I think it's a really good point
37:22
like not a not a ton of people have not a ton of people know
37:26
about the streaming database stuff and so we don't get asked
37:29
about it too much but but I'm keeping a close eye on it
37:32
because I want to bring that into tecton.
37:35
Yeah yeah, I mean, it's ironically the yeah, they're
37:38
probably not great for for larger.
37:41
Companies that have stringent features were requirements.
37:44
They're probably great for much smaller companies.
37:47
Like, I, you know, I used to work at some eCommerce startups
37:49
in New York and we kind of had our own very hacky way of
37:54
serving some features in production.
37:56
With like one day latency let's call it.
37:59
They get updated once a day and you load everything into memory
38:02
and it would have been great to to use it like a streaming
38:07
database there but probably the smaller companies might Do not
38:11
know what feature stores are yet.
38:12
And so it's, I think I think eventually, right?
38:17
Like, eventually feature stores, will become more unanimous and
38:20
then possibly for the for the lower end use cases the
38:25
streaming databases will be able to solve for that.
38:27
Yeah, and what we find is even for the like the really lower
38:30
end, use cases, you don't even need streaming often or you
38:34
don't need. So like a lot of the cool stuff
38:36
on streaming databases is someone will say yeah.
38:39
But I could just use like Like my any database.
38:42
I could just have batch features only and I'm fine.
38:45
So, so what happens is machine, learning teams, go through.
38:50
This is how we see it. Like machine learning teams, go
38:52
through this, kind of like, journey of maturity and they
38:55
want to start simple and and then there they grow to, you
39:00
know, doing what like block is doing today and and along that
39:06
Journey they don't really know what they're requiring.
39:07
You know if you're at stage 1 you don't you're not thinking
39:10
about here. What am I?
39:11
Requirements going to be at stage 2.
39:13
And and so that's kind of a funny thing because we're asking
39:16
people to choose, what's the technology you're going to?
39:19
You're going to need when you're three stages ahead because you
39:21
want to you're making your plans for three stages from now.
39:24
And and so part of part of like the the kind of interesting
39:28
challenge that people deal with in the ml op space as how do you
39:32
adopt the tools that will grow with you along the way?
39:36
And, you know, we think about that when when building tecton
39:40
but I think It's I think the point that you're making is
39:43
like, hey, there's another stage of like, where's there's a nice
39:46
technology fit for a certain level of sophistication?
39:49
And, you know, that's making the simplifying things for people in
39:52
that stage but it's also making love like story for people.
39:56
It's just adding another system where it's like, you know the
39:58
thing you got to know of like how it fits in with other
40:00
things. Yeah and I think the my hope
40:04
would be that these streaming databases can replace having
40:07
multiple systems and so you know, like I Lee your feature
40:12
store and your model monitoring, can all be done in one place and
40:16
like and then that way, also, if your other data lives there,
40:19
then you're able to kind of connect both your modeling
40:24
workflow, and your analytics back to features that are used
40:27
and your models performance and things like that.
40:30
So, like, one of the things that I write in the in the blog post
40:34
is that like I don't want to send my data to some tool.
40:37
I'd rather the tool come to my data and so Now, when I think of
40:41
like a machine learning monitoring tool, like, you know,
40:44
how do I, how do I calculate the performance of my machine
40:48
learning model? It's very rarely something as
40:49
simple as like precision and recall, it's going to be some
40:53
complicated business metric. So maybe let's say it's like
40:56
fraud, maybe we are conditioning or events on certain things,
41:00
having happened and maybe we want to wait our precision and
41:03
recall by some loss value and a bunch of other things and so you
41:08
know like to do that. Typically you write like a big
41:11
Charlie snowflake query in order to calculate that.
41:13
But then in order to get that very complicated metric into,
41:18
let's say a machine-learning monitoring tool.
41:20
If you want it to be served up, in if you want to be able to
41:23
monitor this in real time, you might have to then reconstruct.
41:25
All of that same logic in order to set it to that tool or
41:30
feature stores. So if I'm calculating really
41:33
useful features in my future store, ideally, I'd like to be
41:36
able to run some analytics queries that are able to use
41:39
those features and Is so absolutely.
41:42
What sometimes happens is. You first have to query
41:44
Snowflake and then you create a feature store and maybe in
41:46
Python you were joining these things together and ideally my
41:51
analyses and like my production like features and and monitoring
41:56
values and things like that are, are all coming from the exact
42:00
same process. Yep.
42:01
And I think like, there's this trade-off that you're
42:04
highlighting, which is if you, if you choose a tool that is
42:10
very July's sometimes it comes with some level of like siloing,
42:15
right? And then you kind of locked into
42:17
this tool and the benefit of that tool might be like, hey,
42:21
it's got the, these really amazing work flows and to end
42:24
that are specialized for your use case, but then it kind of
42:26
like keeps all your stuff in that world and then when you're
42:30
in the analog, when you're building a real-time dashboard,
42:32
your normal analytics tools, like don't integrate with it or
42:35
whatever. And so you give up on a lot of
42:37
like standardization and centralization and and then the
42:40
other Option as hey, keep everything.
42:43
In this one standard I like, you know, let's use snowflake for
42:47
everything kind of thing at all. The tools can plug into it, but
42:51
you may be missing out on. Imagine it's a monitoring tool.
42:54
All of the predefined monitoring metrics that you would get.
42:57
And then, and then you're stuck like rebuilding all these things
42:59
in managing all the definitions of these metrics and stuff like
43:01
that. So, so I one thing that we think
43:04
a lot about, and I like that, you're highlighting this.
43:06
This trade-off is, how do you get the best of both worlds,
43:08
right? How do you, or how do you like?
43:11
Not have the negatives of either a world where you know, you want
43:14
to avoid siloing, so you want to the tool to plug into as much as
43:18
possible. Your existing data set, but you
43:21
want to also get like the cool stuff that I tool provides which
43:25
is all the, all the metrics are all the workflows or that the
43:29
the part of the experience that makes life more productive, more
43:33
effective for the say, the data scientist, or whoever.
43:36
And and certainly like, you know, streaming Databases
43:43
provide like a new point. They're right.
43:44
Because it helps you like achieve, some of the technical
43:47
things that both worlds provide in a little bit of a much more
43:51
unified way, which is really nice.
43:53
Yeah, dude. So Ethan.
43:57
This is the beauty of the internet.
43:58
I got to ask before we go because we're starting to get
44:01
down on time. What is Rosenthal data and why
44:06
is that not your thing anymore? What happened there?
44:10
So, See before before I came to square.
44:13
So I joined square like right before the pandemic.
44:16
It so, early 2020 before that I was freelancing.
44:20
So I was like freelance data scientist, machine, learning,
44:24
engineer, building some things for people around the country,
44:27
which is very fun. And then, I got this idea
44:32
because I, you know, I kept building things for people and
44:34
then you if you build out like a production machine learning
44:36
model for somebody, you just hand over this like giant pile
44:40
of code and An infrastructure to them, that if they don't have
44:43
any in-house people, you know, they're like thank you.
44:45
And then I guarantee, you know, I try to write good code, but
44:49
it's going to break at some point, all these systems break.
44:51
And so, at the time, I was like, oh man, there needs to be a tool
44:55
to be able to monitor how machine learning models are
44:59
performing in production. It's so I tried very briefly to
45:04
start a start-up doing this. So, I spent about six months
45:07
built a POC, I got really hung up on this data.
45:11
Issue of like, how do I store large volumes of data and join
45:16
them together and calculate these aggregated statistics
45:18
like, this is why I'm like, so interested in the data side of
45:20
things. But yeah, I basically, I
45:23
couldn't find any customers and I decide to call it quits.
45:25
And then, I took a job and the pandemic, it and I was really
45:29
thankful to have a W-2 when the pandemic and so will, then I was
45:33
that? Yeah.
45:34
How, how do you see the monitoring space now?
45:37
Because it feels like in what 2019 it.
45:41
Not as populated as it is. And now, I would say it's one of
45:44
the most crowded spaces, so you obviously weren't the only one
45:48
having that idea in 2019. Yeah, I mean, I can't tell if me
45:53
writing this this blog post about, like, let's replace
45:56
monitoring tools and streaming. Databases is just me being
45:58
bitter, but I'd like to think that this is an actual
46:02
constructive, you know, comments to the community.
46:05
But but who knows. I like, I'm glad that there's
46:08
lots of companies doing it. I think it's super important.
46:11
And we have a contract at Square.
46:13
I my team doesn't use them but but block does have a contract
46:16
for the model monitoring tool and like I still stand by the
46:20
original idea that like it's it's important for us to be able
46:23
to bother these bottles at. It's non-trivial.
46:26
It's like, you can't just use data dog for this.
46:29
At the same time I kind of just wish there weren't so many
46:33
tools. There's like the whole Space of
46:35
ml Ops is so fragmented. And you know I just I just don't
46:40
like Integrating with things. So even though I'd love to just
46:45
ask you, if we have another minute just about your time
46:48
freelancing. What was that?
46:49
Like, I've never freelanced. Where did you find jobs?
46:53
What was the good part? What were the good parts of that
46:55
about it? What, what was really tough?
46:58
Yeah, so network is key. I would say.
47:00
So thankfully I had worked at and quit at a number of
47:04
companies by that point. So I had a good network of like
47:07
former co-workers to lean on and so Where I guess I got a lot of
47:12
my business but it's like it was really exciting, right?
47:18
So I was really focused on. I want to build production
47:21
machine learning models for people and so that's where I
47:23
tried to narrow my business and I got to do it and so it was
47:26
like, really fun just kind of swooping into different areas
47:30
and having to get up to speed really fast.
47:32
You know, like I feel Bayesian models for this company that was
47:35
optimizing ad spent on Facebook. I built like time series
47:38
forecasting, models for like, A podcast analytics company like
47:42
all sorts of things and what people come with to you, if hey
47:45
I need the specific model built or what would it be like just
47:48
help us figure out what to do with data.
47:50
Sometimes they come to me with a specific model Sometimes they
47:54
come to me with specific ideas and I probably like 80 percent
47:58
of the time would turn them away and I'd say look like you don't
48:01
even have a data warehouse like you don't even know what your
48:03
data is showing you. Like this idea that you have is
48:06
a terrible idea for like, you know, I would, you know, couch
48:09
my feedback a bit better than that but A lot of people just
48:12
like we're not ready for machine learning yet because it's like,
48:15
as we all know is just so much freaking work and so but some
48:20
people had very good ideas and I would just come in and build it
48:22
for them. I think the better if you can
48:25
get a well-defined project then you can charge like an entire
48:30
project fee for it, which is a great way to decouple your time,
48:33
from your money. I got very obsessed with, like,
48:35
decoupling my time for my money, but if a business just doesn't
48:39
really know what they need and they, Kind of just want you
48:41
working there for 10 or 20 hours a week.
48:43
Then get billed hourly which is fine but it's all I learned a
48:48
lot about like the like risk-reward trade off.
48:51
So if you, if you're willing to try to take on the risk of just
48:54
charging a fixed fee for something, then it can be a
48:56
large reward. But you can also get hosed and
48:59
it seems like choosing the right.
49:01
Be really good at choosing the right clients is a very
49:04
important skill. Yeah.
49:05
That's the beauty of it. Is that you go, you don't have
49:07
to take the job. You can fire clients.
49:09
I never had to do it, but You know, every client you signed a
49:12
contract and you can just say, all right, you know, 10 days.
49:15
Notice I'm out of here, so that helps ideally don't do that
49:21
well. So now you are teaching and I
49:24
would love to finish on you talking to us about your course
49:27
that you're doing and how that came about.
49:30
Yeah. So I am an Adjunct professor at
49:32
nyu's engineering school. So I'm teaching a course with
49:36
friend of mine, iacopo, who I think has been on this podcast.
49:39
He must have been on here. For yeah, for sure a few times
49:43
like many years. Yeah, iacopo wrangled me into
49:46
doing this. So he Co taught the course last
49:49
year and so we're co-teaching this year.
49:51
So it's an intro to machine learning for financial
49:54
engineering. And so it's kind of like, Like
49:59
basically all the way from what is machine learning to actually
50:04
deploying some models in the cloud with a bit of a financial
50:07
engineering color to it. And so yeah.
50:12
So what we're focusing on is, you know, theory is good to get
50:16
a bit of a foundation, but I think that building out
50:19
practical skills for people is a pretty important thing.
50:22
You know, like especially that would days, you don't even have
50:25
to know, math anymore, we have you know, like deep learning.
50:28
Berries that take derivatives and everything else for you.
50:30
And so while we touch on some of the theory, I think it's just,
50:34
you know, linking all the pieces together and building up that
50:37
mental framework of understanding.
50:38
Like, what are the inputs to my model?
50:41
What are the outputs to my model?
50:42
How do I actually measure the, like, the models performance and
50:45
Effectiveness? That's what what I'm hoping,
50:48
people will take away from it and whose lose it for Ethan.
50:50
Is it for people who know a little bit or is it for people
50:52
who are just like, hey I've never done data science stuff at
50:55
all and I'm interested. So these are jurors in financial
50:59
engineering students and they said their masters students and
51:03
they know python that they know linear algebra and things like
51:05
that, but they and most of them seem to know a bit of data
51:08
science, but they might not might not know anything about
51:11
machine learning and then especially don't know anything
51:14
about deployment or kind of the system side of things.
51:18
Well, I heard they don't like hard classes at NYU.
51:23
So if you want to keep your job better, not, I saw them are
51:27
volt. And so don't make it too
51:31
difficult on those poor students.
51:33
Yeah, it's the warning flags are very clear.
51:40
Well, do this been awesome, man. Thank you for coming on here and
51:42
sharing your views and teaching us.
51:44
A few things about what you're doing over there at square.
51:47
And yeah, I think we'll just end it there.
51:51
Yeah, thank you. Thanks so much for having me.