Latest / Agentic Conversations (formally mlops.community) / Zipline Roundtable episode: Building Real-Time ML Systems with Zipline + Chronon

Transcript

Zipline Roundtable episode: Building Real-Time ML Systems with Zipline + Chronon · Agentic Conversations (formally mlops.community)

0:05
We are live Welcome everyone to this incredible day of good old
0:12
fun. I see in the chat that some are
0:15
asking are we live? Are we not, Is it happening?
0:18
I just want to confirm. Tell me in the chat you can
0:21
actually see us. We are full on in the session.
0:26
We've got a, there's the, there's the, that was me talking
0:31
in the background. So it should all be live.
0:36
So we've got a jam packed day for you.
0:38
The next hour we're going to be talking to some incredible folks
0:41
that I want to bring on to the stage right now.
0:43
We're going to be doing it in a really cool format where each
0:47
one of the participants today is going to give a bit of
0:50
background on what they do and what they're up to.
0:54
And then we're going to have a nice big round table discussion.
0:59
So I'm going to start bringing people on to the stage.
1:04
First up, I see on my screen, we've got Nick.
1:07
Where you at Mick? Hey, there he is.
1:10
Hi, nice meeting you all. Yeah, great to have you here and
1:15
all. I see we got Jermaine.
1:17
Hey, dude, what's going on? We've also got Ben behind.
1:23
There he is. How you doing, Ben?
1:28
So you guys all are doing some amazing stuff at your respective
1:33
companies. I know that we had said let's do
1:38
some different ideas here where you can share and give everyone
1:43
context on what you're working on.
1:45
Are there any takers on who wants to share first about what
1:50
they're doing? I can't start I guess.
1:55
There we go, Ben. I like it.
1:58
So I was going to put like a slide together and then I was
2:01
like, I don't know how to describe it, but so at Depop,
2:06
we've been actually migrating over to zip line for the past
2:11
few months. And as part of that, as you
2:13
imagine from like a marketplace, the big problem is the matching
2:18
problem. So we've mostly been working on
2:20
the ranking model around that and forming the data set,
2:24
getting all the features up and ready.
2:28
Yeah, as you could go through that.
2:30
I think like the big piece around ziplining Kronon has
2:33
always been how do you fit into like a ziplining Kronon model?
2:38
And then how do you kind of bring back that orchestration in
2:43
like the correctness aspect into your existing feature set and
2:46
what you like existing model look like?
2:48
So a lot of time just kind of conforming back to you what
2:53
expectations look like and having a lot of fun with that.
2:57
All right, And men use cases you're working on?
3:01
Ranking models, retrieval from cases, the virtual side that
3:06
we're getting over to. Excellent.
3:09
All right, Jermaine, hopefully I'm pronouncing that.
3:13
Right, it's pronounced guerman. No worries.
3:16
Hello everybody. Way off dude.
3:18
I was going to go with German, but I was like, there's no way.
3:21
Very common. No worries.
3:24
Yeah, nice to be here. Thanks for having me.
3:27
Just to give you a big bit of background about myself, So I
3:29
work at Intuit Credit Karma. I joined Credit Karma before I
3:33
was part of Intuit in early 2017, like the very early stages
3:38
of his feature store and recommendation system.
3:40
So I've been here for a little over 9 years at this point and
3:44
I've been mainly like on the data and ML realm on the online
3:48
platform side of things primarily.
3:50
And I, I started out, it works on like the online model
3:53
selection and scoring service and like handling model
3:55
deployments and refreshes and serving at scale.
3:59
And that system grew from a handful of new models 1/4 to
4:04
hundreds of refreshes weekly and like hundreds of models total.
4:08
And then I also worked on the custom build feature store that
4:12
we had. So we started out, we had like
4:13
an in house sharded my sequel and we, we had a very like small
4:18
feature platform, feature store at the time, there wasn't really
4:22
a concept of feature store in early 2017.
4:24
So just kind of crazy to think about.
4:26
But yeah, we, it blew up and we know we, we scaled it.
4:32
We had to scale. We have to change things up.
4:34
We migrated to Google big Table and data Flow and that worked
4:37
out very well for us and evolved into like a custom built feature
4:40
platform with mature data pipelines.
4:44
But yeah, we've been just continuously evolving our custom
4:46
setup. And like I shifted my focus to
4:49
future platform exclusively. And the landscape recently
4:52
especially has changed so much for the future platform and
4:55
future stores. And there's so much tooling and
4:57
so, so many products contrast to even though like a few years
5:01
ago. So we decided to.
5:04
Do. A gap analysis and our aim was
5:07
mainly to continue improving our future platform, but sometimes
5:10
the best way to do that is just to look outward.
5:12
So we just wanted to learn me primarily what's out there and
5:16
see, you know, where we can improve from the things that
5:19
exist today. And ultimately that culminated
5:22
with like Cronon becoming our next evolution after a bunch of
5:25
talks, a bunch of, you know, PO CS and trying things out.
5:29
And thankfully our, you know, company leadership is very
5:31
encouraging support us to enable us to do this.
5:33
So it's it's been a really fun journey and we're in the very
5:36
early stages of Cronon adoption here, but things are looking
5:40
great. OK.
5:41
I'm going to have a lot of questions for you later.
5:43
But before that, Mick, you want to give us some background
5:46
context? Yeah.
5:49
Hi everyone. My name is Nick.
5:51
I am now at Open AI. We're working on classical
5:55
machine learning feature platform.
5:57
Previously I was at Stripe working on LED the ML training
6:02
with Flight. Yeah, happy to be here.
6:07
Very cool. So now what I, what I need to do
6:14
is I need to start with what your systems looked like before
6:22
you started using, what you're using today and what you're
6:26
going with like give me the kind of like run, craw, walk type of
6:33
thing or how was it? How is it and how has that
6:39
evolution happened? And maybe, Mick, I'll start with
6:43
you, because why not? Sure.
6:48
Yeah, Opening, as you know, we, I work in a product side,
6:52
infrastructure for the product side.
6:55
So we're not training LMS here, but the capacity crunch is real
7:00
across the organization. So there's definitely value for
7:03
classical machine learning models, and that's where I think
7:06
Krenon really shines to help us with these standard use case
7:10
test that industry has been solving for the entire, you
7:13
know, for a decade at most. And I think, yeah, Krenon is a
7:17
great fit for starting with, you know, crawling when people are
7:21
actually not using classical ML in in our product side of
7:27
organization because people were able to use LLM to push through
7:32
classical problems in the sense of prompt based asking for
7:37
classifier response. And over time, as people got
7:42
used to that, there's actually not a lot of appetite to
7:45
actually adopt classical models because as you all know, it's
7:49
actually hard for you to actually maintain end to end
7:51
pipeline and maintain your model life basically ML OPS, right?
7:56
And we didn't have a team for that.
7:58
So to get started with Kernan Ashley, a good place, it's using
8:03
it as a way for people to extra to export offline data and
8:10
analytics into online serving. So basically we just use Kernan
8:14
batch to calculate using the expressive features and
8:18
aggregation semantics that Kronon provides.
8:21
And readily people can actually query that on an online path.
8:25
And this is like, if you squint, this looks like ML in the sense
8:30
that we have features, but then the rule, it's actually just,
8:33
you know, human encoded. So we provide the features, but
8:36
the it's still heuristic based. So that's how we got started.
8:40
And over time, I think Kronon really shines with, you know,
8:46
real time features and online, offline consistency, right?
8:49
So with the real time part, I think we just, we started with
8:53
Sora, which is open AI video AI generated video content platform
9:00
and critical to that it's recommendation system.
9:03
And that's where Kryon helps us to quickly spin up this real
9:08
time features that otherwise would have taken us a lot more
9:11
time in within within a month or so we were able to get it
9:15
started. Excellent.
9:18
All right, Thanks for that. I'm going to come loop back
9:20
around to you. I realized I've been keeping one
9:23
of our panelists in the dark. Raj, where are you?
9:28
Get on here. There you go.
9:29
Sorry about that. We have a surprise guest.
9:35
Hey, there he is, Raj. And everyone.
9:38
How you doing dude? How are you working?
9:40
I interrupted my mic. No, it's perfect timing.
9:45
We've been just talking about what your system looks like
9:48
before and now what it looks like now and maybe going over
9:52
some of these hard challenges that you've had as you're
9:57
evolving your system to what you have now.
10:00
So I know that's a very broad question.
10:02
Let's just tighten it up by saying, hey, what did your
10:06
previous system look like before Cronon and Raj, since you're new
10:11
here, I'll pick on you. Ah, OK.
10:14
So I was listening to the stream from the other one like Gemma
10:19
didn't introduce it well. So I, we started off our feature
10:24
platform in 2017, 2018 ish land majorly.
10:32
We are big, we are kind of big on GCP.
10:36
With that we kind of scaled our legacy features from around
10:40
53000 count to 50,000 count now. So overall it has been a journey
10:46
and like you know with the growing business with what we
10:51
did is mainly you can say it's kind of Lambda style feature
10:55
platform if not completely Lambda where we do have separate
11:00
UL pipeline system where you have a complete batch pipeline
11:05
for creating features and also streaming pipelines for creating
11:08
features. But as such, we have overall
11:11
gone through a phase of rapid expansion and during that phases
11:17
we have accumulated a lot of debt.
11:20
As one does. Yes.
11:21
Yes. And and it's it's a really good
11:25
setup for us to actually try out Kronon because obviously a team
11:30
of four world can't compete with the open source value which is
11:34
which has been developing from like, you know, almost 780 years
11:39
similar to as similar to what we have been doing too, right.
11:43
Overall, Yeah, this we have done some POC's that has been a
11:47
really great journey as such. Excellent.
11:51
All right, Ben, I'm throwing the ball over to you.
11:54
What's your system look like beforehand?
11:56
And really tell us about the good, the bad and the ugly.
12:02
Give us all The Dirty secrets because I know there it's not
12:07
all roses and tie dye. I feel like I'm a really bad
12:11
person to answer this. I joined EPOP like 6 months ago.
12:14
So then it's like, how do you historically look back?
12:18
You've only done. The good.
12:20
But I I think like a good reference point for what has
12:25
changed in, in just that six months since we started moving
12:27
over is the separation of like features from training data and
12:35
what you generate offline and like use online.
12:39
Prior to the migration, we we would have features that feature
12:44
views that like there was nothing actually unifying that
12:46
into a training data set that was often pushed out onto the
12:49
product teams has a responsibility for them, which
12:54
is pretty difficult for like large backfills.
12:56
How do you start chunking? How do you start, you know,
12:59
filling in gaps when you do have problems?
13:02
And since we've started to move over the envelops, the the
13:05
platform side has started to own more of that.
13:08
And there's been like a responsibility shift in what do
13:12
product teams own? Where do they spend their time
13:14
and where does that focus go? And that's really been like the
13:18
six month shift at least that at the doc.
13:22
Yeah. And maybe you can go a little
13:24
deeper on what some of these capabilities are with like that,
13:28
that specific issue, you know, and saying like what were you
13:33
trying to unlock there and how has it worked out so far?
13:37
Yeah, I think the the big unlock is like correctly frames is
13:42
probably like orchestrating correctness and orchestrating
13:45
training data. So when you write a feature, you
13:49
know, you have like a 30 day window, that means that you have
13:51
a 30 day dependency on some, you know, upstream source.
13:55
How do you correctly know that you're backfilling?
13:58
That is like one small step of just like how do I backfill that
14:01
one feature? Usually ranking models are
14:04
pretty wide. Like say you'll have like 300
14:07
features. Now how do you generate the
14:09
training set with all 300 features at once to come out of
14:13
that and then orchestrate that training set is fully complete
14:17
off of all of the dependencies over that full, you know,
14:20
everyone having their own window and like alignment requirements.
14:26
That was all like each feature individually would get kind of
14:30
wrapped in a in a feature view and you can orchestrate that
14:34
separately. But the moment you combine it
14:35
and the moment you want to like incrementally add features, all
14:38
of that was still being managed by the product teams.
14:42
So what we kind of like really wanted to push forward and shift
14:45
to was the more the platform owned that incremental to like
14:48
change that incremental shift in every training set, the quicker
14:53
the product teams could kind of iterate and adopt and test new
14:57
features. That was I'd say like the real
15:00
value in the shift out of the product teams back onto the
15:04
platform, but also they're just not owning as much.
15:08
So now they can just focus on the actual task, which is like,
15:11
how do you, you know, get people to match all the products that
15:15
they actually want to match over and not necessarily how do you
15:17
solve like backup problems? So I love digging into some of
15:23
these pain points because it really exposes where there's
15:28
friction. And maybe we can talk about what
15:33
other pain points have been solved with Cronon and what pain
15:37
points you were going into. I'll throw this out to anybody
15:42
to who wants to take it. Are there pain points that
15:46
you've had in your mind? And you were like, look, I'm
15:49
going to try and resolve this. And how has it gone so far?
15:56
I could grab this one. Too.
15:58
All right, keep it going. Keep it going while the others
16:01
are talking. So recently, like during this
16:04
plan, we have embeddings that get generated.
16:08
And this is a really good example where prior to the
16:11
actual migration, we couldn't put the embeddings inside the
16:14
feature store because it's cost concerns, They were online cost
16:18
concerns and offline cost concerns.
16:20
But that means that your training data, your features
16:23
that you generate are clearly like misaligned with what you're
16:27
going to serve online. Like your, your embeddings are
16:29
not being sourced out of your feature store.
16:32
They're completely sourced out of something else that gets
16:34
orchestrated separately. We have we're like really
16:38
intentional like embeddings get sourced from the feature store.
16:41
Every single thing that you want to derive off that comes off
16:43
this feature store, the training data that dumps out the
16:47
embeddings is now sourced from a single source.
16:50
The online values also. So I think like because costs
16:55
can be cut, because you can save so much, you can also help like
17:00
resolve those really complex orchestration problems that
17:03
occur both of the line. From our perspective, so one of
17:08
the things on our side is because we have like a fully
17:10
custom system. I think both of us, Raj and I
17:13
mentioned like there's a lot of maintenance overhead as a result
17:17
and just like maintaining custom tooling.
17:19
So that's something we wanted to move away from and kind of like
17:22
find an open source community, ideally open source platform
17:25
with a lot of like, you know, really solid adopters, just
17:29
proven at scale. The Cornyn has and so that
17:35
that's one of the things. The other one I would say is the
17:38
feature definitions is ensuring feature correctness.
17:41
I think that with a structured DSL, it is significantly easier
17:46
to make sure that the features that you create are great
17:51
features and you know, point of time correct.
17:54
Basically it's construct that's front and center in Kernan.
17:58
But whereas if you have your own DSL, your own kind of logic,
18:03
it's pretty easy to get wrong, especially if you have a lot of
18:06
like free form of future definitions going on where
18:09
you're trying to kind of like hand roll point in time joint
18:11
correctness. Generally you get it right.
18:14
But if you get it wrong, it's you have to have really good
18:18
monitoring in place and be able to catch it.
18:20
And so we really like that aspect of Kernan where it makes
18:23
that a lot easier. I can add one more point to
18:27
that. So we over the time we have
18:31
built our batch pipelines first and then built on top of that we
18:35
have built streaming and real time pipelines.
18:38
And one of the things which has which grown and has really
18:42
solved for us is a unified authoring system.
18:45
So we had different authoring platform or different authoring
18:48
style for batch and different authoring style for or
18:54
streaming. But now it kind of is unified,
18:56
which kind of makes it easier for the data scientists down,
19:01
down the lane like because they don't have to interact with two
19:04
different things. They don't have to think on,
19:06
hey, should I just put this, should I put this on here in
19:09
this particular style or should I put it here in this particular
19:13
style? Now it's kind of unified.
19:16
That's the from the user experience side, from the data
19:19
science experience side, it has been like, you know, really good
19:24
overall experience per SE. Eliminating that friction.
19:29
I like it, Mick. I see you want to say something.
19:32
Yeah. I think besides all the points
19:36
about kernel economics and how it fits with existing workflows
19:40
that other people talk about and fit with existing, you know ML
19:44
like requirements of their previous system.
19:47
I think just from the scale perspective that we are starting
19:51
with, I think we evaluated an open AI across multiple
19:55
solutions whether we have like in house counters that people
20:00
can write their own fling job. I think I mean fling really
20:03
suffer from the. You know, a large bootstrap time
20:06
when your time window it's really large and Kern on just
20:09
with this Lambda architecture, it's able to effectively address
20:12
that. We also have in house so like
20:16
OLAP system where the real time events are interested and people
20:20
have the benefit of dynamic queries that are able to get
20:23
more expressive features. But Kern on strikes a balance
20:26
between, you know, the expressive sequel.
20:29
But then with that, so like more restricted expressions, you're
20:34
able to materialize this and actually serve that with much
20:38
smaller latency compared to dynamic queries issue to O labs.
20:43
And so I, I think with that considering from skill
20:48
perspective, that's something that really compels us to, to
20:51
use Krenon. And the other part it's I think
20:54
the team and AT open source is absolutely world class and I
20:58
think they really understand this problem.
21:00
Nikhil and the team has been thinking about this for so long
21:04
at really granular details. So yeah, I think we fully trust
21:10
them and it's has been super collaborative.
21:13
So I'm definitely advocating for open source just because, just
21:20
because of the founders and and the engineering team there.
21:23
That is a glowing recommendation.
21:26
I like that and you spoke a little bit about this stack.
21:31
I like how you gave me some frame of reference on it, but
21:35
maybe you can enlighten me a little more on how the how
21:40
Cronon fits in the overarching stack and where it plays.
21:49
So it's a part of data platform. So we have different standard
21:55
data products. I would start with just like how
21:58
I understand Cranon in our system and fits with overall
22:01
architecture. So basically because it's Lambda
22:03
architecture you have a batch side and you have a streaming
22:06
side and so from the batch side it fits very much without in
22:11
house Spark system. And on a daily basis, this batch
22:17
part of Lambda architecture, it's being so-called compacted
22:21
or we create the historical sort of like aggregations from from
22:26
historical up to like midnight UTC and then we export this to
22:31
our KV look up. And then on the real time side,
22:36
of course, the source here is real time Kafka events where we
22:41
have flank that continuously consume from these events and
22:47
write out pre aggregates for on the streaming features.
22:51
And we push this to AKV store. And then during the serving time
22:55
when the client comes in, let's say we want to get feature for a
23:00
particular user. Krannon serving layer knows how
23:04
to pull in from the batch side store that has been stored that
23:08
has been exported from the batch spark and then from the KV store
23:13
which has the pre intraday pre aggregates and merge them
23:18
together together to get the full features for for the real
23:22
time 1. Awesome.
23:25
And how about you guys, maybe somebody else on here, how does
23:31
it look? Is it very similar to that?
23:32
Is it different? It's very similar in our side
23:35
also. That's, that's the beauty of
23:36
Karnan is it's, it's like it's portable.
23:38
I think the difference on our side is the KB store.
23:41
I mean, we're in GCP, so we use Big Table and we have a lot of
23:44
experience with big Table and we really like big Table.
23:47
So it's nice that it's natively supported within Karnan.
23:52
But aside from that, yeah, the the batch and streaming
23:54
pipelines very similarly, you know, Spark and Flank and we
23:57
have a similar, you know, serving layer that is able to
24:00
pull from both of those and serve those features
24:03
architecturally before Quran or system was pretty similar.
24:07
So that kind of made it easier to adopt and that we also have
24:10
like batch pipelines and streaming pipelines, except in
24:13
our case we were relying more heavily on big query and data
24:16
flow as opposed to Spark and Flank.
24:19
But the fact that it's like cloud agnostic and also pretty
24:23
similar is a really strong point for us, especially because we're
24:28
part of Intuit now and Intuit is on AWS.
24:31
So the cloud interoperability gives us a lot of advantages and
24:35
kind of future potential integrations that make this a
24:37
lot easier. So it it felt more like it
24:42
wasn't really a huge architectural shift, more like a
24:44
natural evolution of the system that we were working towards and
24:47
had a lot of this similar capabilities kind of in progress
24:50
in flight already. So that's kind of how we started
24:55
to move towards Kurnan. For us, I think it's pretty much
24:59
the same. I think there is there's always
25:01
like some slight difference inside of some part of it.
25:05
So for us, it was like our on our data warehouse side, we have
25:08
unpartitioned tables. They're all like clustered
25:11
overtime stamp columns instead. So there's like some shifts on
25:15
the orchestration work and like how you declare dependencies
25:18
that weren't like naturally fitting.
25:20
The zip line team was there to help on that.
25:23
But otherwise, yeah, Wesley follows on the same, you know,
25:28
construct around it. I think it's worth, it's worth
25:31
pointing out that the system, it's very elaborate, It has a
25:35
lot of moving parts, but the fact that it actually applicable
25:40
to many organizations, I think that speaks highly to the the
25:44
right love abstractions that Kunan open source is getting
25:47
this right. So yeah, I'm very, very happy to
25:51
to to to onboard Kunan. I mean, it was without zip line,
25:56
without ONS, open source team help, it would be really
25:58
difficult to get started by ourselves.
26:01
And maybe when I said like we were able to set up a real time
26:07
streaming within one month that is actually on the foundations
26:10
of having the bats running for a while.
26:13
But otherwise I think it is really hard to to to get
26:18
started. Yeah.
26:21
I would love to know as you are look through some of these
26:27
different pieces in the stack. I know that when I was with the
26:32
Cronon folks in Seattle a few months ago, they were talking
26:36
about ways that they're incorporating like the LLM
26:40
capabilities into what do on a day-to-day basis.
26:45
I don't want to just be like, hey, you think you could use AI
26:49
for some of this or like coding agents on this.
26:52
But maybe we could take a little bit of a broader step and think
26:55
about what are different use cases you see in the future
27:02
using something like Cronon or zip line and what are the like,
27:09
what are the major use cases and how are those if at all from
27:14
what you are doing or have been doing in the past?
27:19
I think with the agents and generous AI, I think we have a
27:26
lot of questions about that getting started with Kernan
27:28
coming from open AI. And I think thinking about
27:32
Kernan adds, you know, a tool that agents can actually
27:36
deterministically get this and I'm doing it at a much cheaper
27:40
cost. I think that's it's an endearing
27:43
aspect of standard future engineering.
27:46
Speaking from experience and knowing a colleague at a large
27:49
financial fintech company, they are using agents to orchestrate
27:56
a lot of, you know, user facing experience, but they do rigorous
27:59
back test and they use Cronons as a part of that to actually
28:04
feed as input to the agents. And having Cronons which fetch
28:11
these let's say last K features or last K items that agents can
28:15
reason about context of the world in a recent history that
28:20
they still use Krunan just because it's unifies both the
28:23
classical machine learning models and the agent take work
28:27
stream that actually wants reproducibility and ability to
28:29
back test. Brave to think about what the
28:35
future might hold or make a prediction on something we have
28:38
no idea about. I can chime in a little bit like
28:42
from the just feature definition and just set up Phase.
28:46
I think using agents makes it a lot easier in terms of like you
28:53
don't really have to manually write feature definitions that
28:56
much anymore, especially if you have the structure DSL like
28:58
Kernon provides. And that if you give it enough
29:04
contacts and you know enough examples.
29:06
And also there's a Kernan skill that exists that we've been
29:10
using. We started using pretty
29:12
recently, but it's been working pretty well where you know, you
29:15
can free form text and you'll be able to generate pretty good
29:21
feature definitions. Maybe it needs some slight
29:23
tweaking back and forth, I mean update documentation.
29:26
But ultimately I think we can get to a state where we don't
29:29
really have to write like Kern on DSL manually at all or almost
29:34
at all. I think that's a really
29:36
promising feature that will make feature iteration and just
29:40
creating features a lot easier. Yeah, you got it.
29:46
You got to love that. Take a little bit of that
29:48
painful road work out of it. So I've got a question from the
29:53
chat that's coming through on Gabriel's asking, do you feel
29:57
Cronon has a large learning curve?
30:00
How easy was the adoption attack on their?
30:03
Like what's the developer experience like?
30:05
How does that work? And I think one thing that is
30:10
worth noting for specific tools like this is there's you all are
30:18
consumers or users of the tool, but then you have your consumers
30:22
of that tool. So it's like there's two aspects
30:24
of it. And I know that makes it
30:26
particularly difficult to build for that type of this area
30:31
because you have maybe other personas that aren't necessarily
30:35
as strong and software engineering as you that want to
30:38
also leverage these tools and things that you think are
30:42
blatantly obvious maybe to a data scientists aren't as
30:47
obvious. So all that out there, what's
30:52
the developer experience like? And Raj, you're here.
30:56
So I'm going to throw you on the under the bus and ask you.
30:59
Yeah, the part with part I like is here we zip line made a lot
31:07
of this easier. They have encapsulated or like,
31:10
you know, abstracted all of this as such, like interacting,
31:15
creating jobs from just your feature definitions and then
31:20
even running backfills and seeing the impact of those new
31:24
feature definitions and evaluating those that if you
31:28
were to do with direct open source, you have to you have to
31:31
create multiple jobs and then like, you know, bring them all
31:34
together so that you iterate on them.
31:37
And that one piece that those kind of pieces, zip line folks
31:43
have made it easier with their encapsulations and the the top
31:47
level standardization, which they have done on top.
31:50
And that that kind of eased a lot of learning curve, I would
31:55
say. So now even if you give any
31:57
coding assistant these docs and all that automatically just
32:03
makes it happen for you and also in a very easy fashion.
32:07
So that's that's something helped us sell it to our
32:11
downstream consumers as well while we were doing this thing
32:15
that because there are two aspects to it, right?
32:18
There is one piece is that we as engineers have to like, you
32:23
know, once they alter the features, we have to maintain
32:26
those and then we have to keep the systems running.
32:28
The second piece is we have to make it easy for the underlying
32:33
downstream data scientists to actually try out the new, try
32:37
out new experimentations or try out new features and then alter
32:42
something and then like, you know, make that particular part
32:45
very easy, right? So that's where this one has
32:49
really helped us out. I think there's a lot of life
32:57
the the part that comes with Cronin and zip line is the IT
33:00
works offline. So it works online side prior to
33:05
the move. There's, there's usually some
33:07
work around like some feature that you generate offline, tack
33:12
onto your DAG offline and now like you use it to your train
33:15
over. But then you have to go
33:17
production either because you see some lift offline and it's
33:21
just a ton of work to do. Like you're orchestrating like
33:23
multiple model calls online. Some of them you have to make
33:28
sure that they're both getting updated around the same time.
33:31
So I I think that the offline online like you just know it's
33:35
going to work right away. Geese is such a, it just brings
33:40
like so much benefit to the the product side that they're
33:43
they're pretty happy with it immediately when they like
33:46
figure out that they don't have to go productionize every single
33:49
feature that they go and generate because they just kind
33:51
of know it's going to work once they flip the switch on.
33:54
I think that is like kind of like alleviates any of the the
33:59
mental overhead that they might have feared early on.
34:04
So the question about like large learning curve and how easy was
34:08
the adoption, I would say definitely like not to sugar
34:12
coat it it I mean it, it definitely has a pretty steep
34:15
learning curve. And I think adoption and like
34:17
adopting it is the biggest hurdle probably.
34:21
And the porn on like system itself.
34:25
It is a like a, a brand new system.
34:27
You have to learn it and you have to understand how it works.
34:30
And it's a continuous process. You know, you're going,
34:33
especially if you have a insurer system now that you understand,
34:36
understand very, very well. And now you're going into this
34:39
brand new system that you have to continuously learn about and
34:42
understand the insurance and outs of that too, that that does
34:45
take time and you have to reason and think about, you know, how
34:49
am I going to set it up and align it to my organization and
34:52
how we're doing things. You know, every company does
34:55
things a bit differently. So there is that the steep
34:58
learning curve. But having said that, I think
35:01
working with the zip line folks helped us significantly.
35:03
We had a lot of back and forth with them and also like a lot of
35:07
like features that we needed that open source kernel didn't
35:10
have. We were able to collaborate and
35:12
that helped us a lot. 1 of the biggest ones is like the GCP
35:15
integration was I think non existent at the time that we
35:18
were looking into Kernan. And so as a plan folks helped us
35:21
a lot there. And then a couple other things
35:25
also like orchestration through the zip line hub is a lot easier
35:29
compared to running your own like Airflow and you're trying
35:35
to figure out DAG dependencies and all of that.
35:37
That kind of gets attracted away from you.
35:39
So it makes it makes it a lot easier to set up and also just
35:42
collaborative as a learning experience.
35:48
Go ahead mate, sorry. Yeah, Yeah, I think the I think
35:54
those mentioned before, but would like agents helping people
35:57
to write code because now I think for my team doing both
36:04
turn on and also we have like flight as a, you know, I'm out
36:09
like a straighter system. But people really don't write
36:11
code these days anymore and they don't look at the artifacts at
36:16
all. So I think agents are pretty
36:20
good with the reasoning about Krenon.
36:22
But I think the key here is perhaps education for your user.
36:27
I think people are sometimes stuck with the idea of maybe
36:32
like textbook examples of like maybe Kaggle competition that
36:36
you have a feature set and you just do like super expressive
36:39
and like Python third parties and like 10 times
36:43
transformations of the features to be able to to get it to feed
36:47
to your training. But I think when people think
36:50
about, I think educating people that this is, you know, real
36:53
time features and there's actually a lot that you can get
36:56
from real time events and just a standard query.
36:59
So like the thing that I always tell my users like, OK, imagine
37:02
that you have an events analytics like user event
37:06
streams. Kernan is actually a SQL probe.
37:10
Like you express your SQL expression.
37:12
You only have that you put this probe into that event stream and
37:16
Kernan will be able to get you features in real time and that's
37:19
going to help people too. Once they understand that have a
37:22
good mental model around that, I think it's easier for them to
37:25
express to the agents to get what they want and the agent can
37:29
then learn how agent will be smart enough to to express what
37:33
user wants once they have the right mental model in
37:35
synchronous language. Yes, sometimes man it is so
37:40
wild. It's those simple unlocks where
37:43
if you just understand how to speak to the agent and explain
37:47
it, give it what it needs, and it's like, wow, that was not
37:51
very difficult from my part, but I use the right words in the
37:55
right sequence and I have a whole new world in front of me.
37:58
Let's keep it moving because there's some good questions
38:00
coming through in the chat and we only have a few minutes left.
38:05
I would love to know two things. I'm going to start with what
38:08
other feature stores or feature platforms did you all evaluate,
38:12
if any? I know that.
38:15
I think the Intuit Credit Karma guy said that you already using
38:20
one and you went off of that during that migration.
38:24
Did you look at anything else that was open source?
38:27
I know there's a few out there and then also, so get it ready.
38:35
What are the bad parts about this?
38:39
We had one person in the chat set saying hey, we just have all
38:43
the pros and no cons. This it's a little SUS, so start
38:48
thinking about that too. Hit, hit me.
38:51
Raj, I see you're off mute. What other things did you
38:55
evaluate? Let's start with that while you
38:57
were making that. Migrate.
38:58
Well. For us the constraint space is
39:01
quite large because we are operating in finance space,
39:06
right? It's, it's fintech, right.
39:07
So one of the biggest things is that we wanted an on Prem
39:12
solution. We didn't want to evaluate
39:14
anything with where we have to move the data altogether because
39:19
that introduces a different, it introduces a different
39:24
complication altogether moving around data.
39:27
And even even if you have all these what do you call security
39:32
set things in place, it kind of makes the whole space itself
39:37
very complex to operate in. So, so we, we were evaluating
39:42
options where we can bring in the whole technology and then
39:46
host it in our infrastructure itself.
39:48
And that's one of the reasons why, like you know, Cronon had a
39:53
lot of points as such Cronon and we wanted to see if we can
39:59
begin. Along with an open source built
40:03
on build something on open source so that we can also make
40:06
it work for our ecosystem. If, if something we are
40:09
diverging from from it. And also we do wanted to have
40:13
like, you know, say, hey, a support group for that, like,
40:17
you know, I'm making things move faster.
40:19
And that's that's the reason why like, you know, grown on and zip
40:23
line together, they added a lot of points.
40:25
They got a lot of points, but we were looking at other options as
40:29
well. But majorly all of them fell
40:31
apart in on this particular thing that we wanted to bring
40:36
that technology and put it. That's where we were giving a
40:39
lot of points to. So yeah, that was fine.
40:44
And you got pain points also, right?
40:49
One pain point which I do want to talk about is this system.
40:54
It's, it's really good from the end user perspective where
40:57
you're actually getting the right definition saying hey, if
41:02
I want a feature aggregate from this point in time to last 360
41:06
days or something, right, you're getting it.
41:08
But underneath you have 3 ecosystems operating with each
41:14
other and they are talking to each other at with precision.
41:19
That makes it extremely difficult to maintain and
41:24
operate as such right? Like I mean I maybe not maintain
41:29
but if debug ability gets extremely tough, if something
41:33
goes wrong, right? If any of these systems at some
41:38
place falls apart, your final definition is going wrong and
41:44
figuring those pieces out like, you know, you need to really
41:47
understand how they're operating and how they're hand shaking
41:50
perfectly so as to so as to actually make sure that thing
41:56
which is running in production is right.
41:59
If something goes wrong, you need to know exactly where you
42:02
have to look at so that you can make things come back to normal.
42:08
So the the whole ecosystem, we are adding a lot of guard rails
42:13
for things such things not to happen.
42:17
But overall, you need to understand that these three
42:21
systems have to work together to give the final product.
42:24
And that's that's where a lot of fiction lies.
42:28
I'm deadly biased. So I, I want to be transparent.
42:31
And when you said like, you know, I only say about positive
42:33
things, but I, I was from, I was from Stripe and we were a big
42:39
user and big fans of Krunan. But of course, like we had to do
42:43
due diligence. I think we definitely looked, we
42:45
looked at actually in house, other in house solutions.
42:48
And as we look more at the computational model and how it
42:53
is actually going to stand a test of scale and we believe
42:56
that Krunan does. But of course, as I also want to
42:59
be balanced in that there are like efficiency challenges with
43:03
Krunan, it can be expensive. And I think the team and we
43:07
worked with them aggressively to build different models,
43:10
different non models in our models, different computational
43:13
scheme that can can optimize for things.
43:17
So there's definitely rooms that some trade-offs that you can
43:21
that you can strike to to get better efficiency.
43:27
Are these things like data freshness and that type of?
43:31
Yes. So I think it's a flank the
43:34
operational ability of flank states or like the streaming
43:37
part, like how big the job would be compared to the.
43:42
So maybe technically, I think on the serving side, if you are
43:47
want to do more work on the right side, you get a lot
43:52
cheaper read side for us in recommendation system, you a
43:58
single call to Kronan usually fetch a large number of
44:02
candidates, right. I think Ben might know this well
44:05
too. But and basically, I mean for
44:08
maybe the the catalog or like the the the universe of things
44:13
that we want to just like it's not too high.
44:15
It's like maybe medium sized criminality, but the number of
44:19
users that comes in is a lot higher and the number of fan out
44:23
of the things that want to fetch, it's a lot higher.
44:26
So maybe 1 user we actually serve 1000 ads at the beginning
44:30
of the recommendation pipeline. So in that case, the read side
44:35
if it's using so like a traditional.
44:39
So like when I was saying that Kernan is merging pre aggregates
44:43
of multiple times time bucket together because it wants to
44:47
give you a real time features. Then your read actually scales a
44:54
lot with like that fan out to serve this recommendation use
44:58
case. Then.
44:59
So to strike the balance here is that you actually do a lot more
45:03
in flank side. You keep track, you keep time.
45:07
So like a timer so that you can't evict out.
45:10
Basically be the challenge here is that it's a rolling window
45:14
aggregates, right? That means that the flank always
45:17
have to be active and keep writing out and publishing
45:20
things and compared to if we want to just go on the, you
45:27
know, pre advocates come apart in the naive or like the the
45:31
basic Krannon features. Flink actually just write out
45:36
the so the 5 minutes window, a one minute, 1 hour window that
45:40
they cares about. And after that it's it doesn't
45:42
have to care about trying to evict or like keep the state
45:45
anymore. And it pushed the responsibility
45:48
to the reader side, But that comes with the cost.
45:51
So I think this is some there's a recent development, but I
45:54
think it actually allows for not to scale a lot more.
45:57
There's definitely more efficiency challenges, but at
46:00
scale but anything the team it's definitely are actually working
46:05
on this. But but for medium scale use
46:08
scare setting Kronon. It's definitely a great fit.
46:12
That's awesome. Yeah.
46:15
I wanted to talk more a bit also like about the pain points and
46:19
just, you know, be transparent about like some of the negatives
46:22
that we've we've gone through. So again, it's going back to the
46:26
adoption, like I would say the adoption, not only is it like
46:29
deciding that you want to adopt kernel, but also as you're
46:32
actually in the middle of the adoption and like planning
46:35
things out, you're, you're probably going to run into some,
46:38
some challenges and, and some, some things you have to
46:41
reconcile in terms of like how your current system operates and
46:43
how kernel wants to operate. Or you have to modify Corona to
46:48
kind of align to more towards your system, which is a
46:51
challenge. But like a couple examples I'll
46:54
give is like one of them is a feature versioning strategy.
46:56
I think Quran was a bit lacking in that department when we were
47:00
doing the POC. And I think the support for that
47:02
is improving because I think a few clients, including us, kind
47:07
of wanted to push for a stronger versioning strategy.
47:10
And again, the SIP line folks helps a lot there.
47:12
And there's good support for versioning in the SIP line fork,
47:16
but it that's just like one of those things or like if you want
47:18
to modify features in place, for example, or like, yeah, that's
47:23
going to be a challenge. And there's just some like
47:25
quirks and some things you kind of have to think through.
47:28
The other thing is in terms of performance in the POC, we were
47:33
stretching the engine engine quite a bit because we chose
47:35
like a really large use case, like one of our largest use
47:39
cases at Kurta Karma and a specific like expensive
47:42
aggregation that we like to do a lot of a credit Karma.
47:44
And it was kind of stretching, stretching Karma a little bit.
47:47
And we, we saw a bit of like red flags in performance.
47:49
And I was like, I don't know, like I was not looking too
47:52
great. We worked with the, again with
47:54
the supply folks here and that ultimately culminated into the
47:57
SKU free union join. Nikhil gave a really good talk
48:00
about it a few months ago, but it improved the, the join
48:05
performance by a huge factor like 5 to 10 X for us.
48:08
So it, it's like depending on your data sets, depending on
48:12
your use cases, what you like to do as a company, you will run
48:14
into those challenges. But the, the nice thing about
48:16
like having this community and having more people that use the
48:21
same platform is that other people will have these
48:24
challenges and the platform will improve as a whole over time.
48:27
So you're you're, it's going to be stronger and stronger as
48:29
opposed to like working in a silo within your own company on
48:31
a custom system. If you adopt like a framework
48:34
that a lot of other companies are using and struggling through
48:37
certain things and improving the platform, ultimately everybody
48:39
will be better off for it. Such a good point.
48:43
And you don't have that problem where it's like, Oh yeah, that
48:46
engineer that just left knows how to do that thing.
48:49
Exactly. Yeah, yeah.
48:52
Which we've all like. The reason we're laughing is
48:54
because we've all run into it and it's just like, no, the
48:57
documentation sucks on this. Why is this not working?
49:02
And it's so vague. Hopefully now with agents
49:07
writing our documentation, it's not going to be like that
49:10
forever, but who knows? Anyway, Ben, what you got for
49:14
us, you want to bring us home? We've got about one minute left,
49:17
you're on the clock and then we'll wrap it up.
49:20
Sure. Like the hard part is usually
49:24
during the adoption of there are some really like good data
49:28
expectations that sit inside of Cronon.
49:30
And for sure not every company aligns with those expectations
49:35
like, oh, your, your data is partitioned.
49:37
You, you know, your schema from the Kafka topic aligns with the
49:41
schema of the offline table. We we hit those during the
49:46
adoption period. And either there's like breakout
49:49
strategies of OK, go and like renormalize your table in a
49:53
stage inquiry within Cronon or push back onto the as it blind
49:57
side of like, Hey, we need help. And then they'll come and fill
50:01
the like gap of, OK, here's how you get support on time
50:04
partition tables. I I think that is like the real
50:10
that that alignment is, is pretty difficult to, to start
50:12
with. And it's like a one time cost.
50:15
And then the moment that it's over like you kind of you got
50:18
the good expectations that were kind of built in inside of
50:20
Cronan it within your order. Folks, this has been extremely
50:27
helpful for me. I really appreciate you all
50:30
coming and doing this and being very transparent with
50:33
everything. If anyone has any more questions
50:37
for you, I will direct them to hit you up on LinkedIn.
50:41
I think we're all on LinkedIn and if not, then scour the
50:46
Internet and find them on X or wherever it may be.
50:51
And we're going to drop in a link to, I'm going to try and
50:54
find Nikhil's talk that was wrapped.
50:58
And we will be sending everyone a follow up e-mail so that if
51:03
you want to go deeper down the rabbit hole, you can.
51:07
And you can see a few of these assets that were referenced.
51:12
So until next time, thank you all for doing this.
51:16
This was a blast. I appreciate you and I'll see
51:19
you later.