Latest / Agentic Conversations (formally mlops.community) / Powering MLOps: The Story of Tecton's Rift // Matt Bleifer & Mike Eastham // #209

Transcript

Powering MLOps: The Story of Tecton's Rift // Matt Bleifer & Mike Eastham // #209 · Agentic Conversations (formally mlops.community)

0:01
Hold up. Before we get into this next
0:03
episode, I want to tell you about our virtual conference
0:07
that's coming up on February 15th and February 22nd.
0:11
We did it two Thursdays in a row this year because we wanted to
0:16
make sure that the maximum amount of people could come for
0:20
each day since the lineup is just looking absolutely
0:25
incredible. As you know, we do.
0:27
Let me name a few of the guests that we've got coming because it
0:32
is worth talking about. If you got Jason Louie, we've
0:35
got Shreya Shankar, we've got Drove who is product Applied AI
0:40
at Uber. We've got Cameron Rook Wolf
0:42
who's got an incredible podcast and he's Director of AI at Riba
0:46
Engine. We've got Lauren Lockridge who
0:50
is working at Google also doing some product stuff.
0:54
Ohh, why is there so many product people here?
0:56
Funny you should ask that because we've got a whole AI
1:00
product owners track along with and engineering track and then
1:04
as we like to, we've got some hands on workshops too.
1:08
Let me just tell you some of these other names just for a
1:10
moment, you know, because we've got them coming and it is really
1:15
cool. I haven't named any of the
1:16
keynotes yet either, by the way. Go and check them out on your
1:20
own if you want. Just go to home dot ML OPS dot
1:23
community and you'll see what we've got.
1:25
Tungi, who's the lead researcher on the deep speed project at
1:29
Microsoft. We've got Holden who is the open
1:32
source engineer at Netflix. We've got Chai, who's leading
1:37
the AI platform at Uber. You may have heard of it.
1:41
It's called Michelangelo. Ohh my gosh.
1:43
We've got Fazan, whose product manager at LinkedIn, Jerry
1:47
Lewis, who created good old Lama Index.
1:51
He's coming. We've got Matt Sharp, friend of
1:53
the pod, Shreya Rajpal, the creator and CEO of Guardrails.
1:59
Ohh my gosh, the list goes on. They're 70 plus people that will
2:03
be with us at this conference, so I hope to see you there.
2:10
And now, let's get into this podcast.
2:13
And Matt, I'm a product manager at Tecton and I like my coffee
2:18
black made with an Aeropress and some fresh hand ground coffee
2:23
beans. I'm Mike.
2:26
I'm the chief architect of Tecton and I usually make my
2:30
coffee with a in a pour over and I just drink it black.
2:37
Welcome back to the ML OPS Community Podcast.
2:40
I am your host for the day, Dimitri OST, and we've got the
2:46
Tecton team with us here, Mike and Matt, What a conversation.
2:51
We went through their history where these guys worked before
2:55
they jumped into Tecton, and both Mike and Matt have been at
2:59
Tecton since pretty much the inception.
3:02
They've been around for four or five years and they've been
3:07
working on a very challenging problem.
3:10
For those of you that do not know, Tecton is a feature
3:14
platform and it helps you decouple your code from your
3:20
feature engineering. It also does a whole lot more
3:24
and what I was excited to get into with these guys today was
3:30
how they have evolved the product over time and also how
3:36
these days because of technology advancements they were able to
3:42
make the product much more lightweight and not have it be
3:46
dependent on Spark which has been as they noted something
3:51
that was a little bit of a hang up in the last couple years.
3:55
So because Spark is such a beast, they noticed that not
4:00
everyone wanted it and they wanted to see, can we go out
4:02
there and can we build something for those people that don't
4:07
necessarily need to be doing all kinds of big data.
4:12
It goes in line with the ethos that we heard from the folks at
4:17
Duck DB on, and actually we got into Duck to be a little bit
4:21
because I think under the hood Tecton is using a few different
4:26
Duck DB tricks and trades. So I hope you enjoy this
4:31
conversation with Matt and Mike's a huge shout out to Tech
4:35
Tone for sponsoring the episode. And if you liked it, you know
4:39
what to do. Share it with one friend.
4:41
We'll be seeing you on the other side.
4:50
I got to call it out. Me SIM, you put the form that I
4:55
asked you guys to fill out before this conversation for
4:59
your job and your bio. It says.
5:02
Michael Eastman is the chief at Tecton.
5:05
Ohh, I've missed a word. That's actually his official
5:10
title. Yeah.
5:12
It was recently announced that he would just be our chief.
5:14
And everybody just calls me chief.
5:16
Yeah, no, that was a I accidentally ordered there.
5:23
But it makes for such a better title.
5:25
Man, I'm yeah, I'm expecting you to change that on LinkedIn now.
5:29
Official Titles Big Dog. That's Matt's official title.
5:33
So we've got Big Dog Matt here, the Ohh Man product product
5:41
manager. What is your official, so your
5:42
big dog product I guess Matt and then?
5:47
Yeah, that's what it says on my on my resume.
5:50
I'm a product manager. I guess my official title is a
5:53
group product manager. There's a group involved these
5:56
days as we've expanded our team a bit.
6:00
So that's been fun. And yeah, I've been Pmsing at
6:04
Tecton for almost four years. I got like two months till my
6:08
four year anniversary. Pretty exciting.
6:11
Wow. That is crazy.
6:14
Stock fully vested then, I guess.
6:17
Yeah, right. Four years, actually.
6:21
That's incredible. That's the important metric.
6:23
So yeah, four years and almost I realised I was introducing
6:27
myself to someone recently. I've almost hit a decade now of
6:32
just working on ML platforms, which is wild.
6:35
Might be even cooler than the the four years attacked on
6:38
things. So coming up on that, still in
6:41
my final year, but it's exciting.
6:44
Hence why you get the name Big Dog PM.
6:47
Yeah, that's how I got it. I know that before you were at
6:51
Tecton, you did some work at Twitter, right?
6:53
And this was early days, obviously four years ago.
6:57
Yeah, similar situation just at Twitter.
7:00
I came in in 2017 and was hired to be like the first I CPM on
7:06
their ML platform team. And so was in the early days of
7:10
like figuring out kind of what is an ML platform in the 1st
7:14
place ever. Crazy story to tell about this
7:17
actually. So I joined.
7:19
I joined in 2017. The team ends up going to an
7:25
off-site in Boulder, Co to figure out like what does our ML
7:30
platform need to look like? How should we kind of Orient the
7:33
teams? Like we had done some stuff, but
7:34
we were kind of, you know, really putting a stake in the
7:37
ground there of like, this is what our ML platform work is
7:40
going to look like and like the day that we get to Boulder.
7:46
Mike Del Balso, our, our CEO and Jeremy from Uber released the
7:53
Meet Michelangelo blog post. Ohh.
7:55
And in it, they're like, Ohh, yeah, we built this thing called
7:59
a feature store and we're all, you know, we're in like diagram
8:02
mode on the whiteboard and we're like looking at it and we're
8:05
like wait, that's a great idea. We don't have that in here.
8:08
And so then we like Resch the whole thing.
8:10
We put a feature store inside of it.
8:12
We staff A-Team. It becomes one of my first
8:14
projects. And so it was kind of crazy then
8:17
to come full circle from like they they're blog posts spawned
8:21
all the work that I did at Twitter.
8:23
And then just a few years later I came and joined them to work
8:28
on it over at Tecton. So was a quite full circle.
8:33
I was expecting you to say and then Elon showed up and.
8:37
I I was. I was gone long before that.
8:42
Although I'm sure that was an interesting thing to watch from
8:47
the inside, or at least have, you know, heard stories from
8:49
friends and such. But yeah, I left.
8:52
I left in 2020, so, so long before the Elon phase.
8:58
No. How are you a user of Twitter
9:00
and is there any, like, sentimental value?
9:03
Do you feel like he's butchered your future platform?
9:06
All that work you did? I like tried to remain kind of
9:12
neutral or optimistic and see what would happen.
9:14
I feel like the thing that finally upset me though was the
9:17
rebrand. Like the the Twitter to ex was
9:20
like, this was a great brand. Like, I identified with it.
9:23
It was like really weird because I was in San Francisco and I was
9:26
walking down the street and like, you know, there's the
9:28
famous, like, Twitter sign going down the corner of the building
9:31
and it's just blank. And I think that one kind of hit
9:34
hard. I was like, you know, I don't
9:35
mind you. I don't mind you like changing
9:38
things up, breaking the product, all that stuff.
9:40
But like, the name had to go that one.
9:42
That one hurt me emotionally. Ohh.
9:47
Classic. Alright, so back to the Chief, I
9:51
gotta ask you a few questions here.
9:53
You've been in the software engineering game.
9:56
So basically Matt comes to you with all kinds of requirements
9:59
or you get you guys probably liaison quite a bit with what
10:06
you're building and how you're building it.
10:07
And before Tecton, you were at Google, right?
10:11
Correct me if I'm wrong there misam.
10:12
Yeah, that's right. I was at Google for about 7
10:15
years before I came here. Wow.
10:18
Wow. Seven years there.
10:20
It's also incredible working on web search, right?
10:23
Yeah, it started out on the indexing team and so was working
10:28
on a project. That while we were.
10:31
Trying to do kind of like what a web browser does when it's
10:34
rendering a document, like getting layout and things like
10:37
that, so that we could extract signals from it.
10:41
Spent about half my time on that and then moved on to the web
10:45
server team while I was working on three different things, but
10:49
like experiments was one of the big ones.
10:51
And now you are at Tecton, you've been there for quite a
10:57
bit too, you're. Coming up on five years in a
11:00
couple of months, yeah. There we go.
11:03
How's your role changed over the last five years?
11:05
What have you been working on like?
11:07
What's the journey been there? Yeah, well, so my actual title
11:11
now is Chief Architect. It's not just Chief So more of a
11:16
kind of like organisational focus, trying to figure out, you
11:19
know, what direction we should kind of evolve the architecture
11:22
of the product in and, you know, trying to kind of harmonise the
11:27
work between the different engineering teams that we have
11:29
now. But yeah, when I started, I was
11:32
like the 1st engineer here. So the work was quite different,
11:35
just was, you know, trying to get something out the door up
11:39
and running so he could, you know, try it out with customers.
11:44
But yeah, it's been a pretty fun journey so far.
11:47
So now you're thinking more about the bigger picture.
11:50
And I imagine things have gotten much more complex and you've
11:54
been there since the beginning. So you've seen it.
11:56
You have a lot of context and as the complexity grows, you're
12:01
like one of the key points where people.
12:04
You're one of those people that understand how things work, I
12:08
would imagine. Yeah, that's the idea.
12:11
You could say that, yeah. Yeah.
12:15
But yeah, trying to kind of, you know, I feel like we've done a
12:18
lot of, you know, as we've expanded and you know, different
12:23
markets and trying to support different customers, we've sort
12:25
of like grown out a lot of functionality and sometimes kind
12:29
of ad hoc or haphazard way. And so you know, I've spent a
12:32
lot of time trying to figure out how we can kind of actively
12:35
reduce the amount of complexity we have in the products and
12:38
figure out how to like keep supporting those different use
12:41
cases. But, you know, in a way that has
12:43
like a simpler implementation so that we can, you know, continue
12:47
to build cool new stuff without, you know, being super bogged
12:50
down by what we've done before. So give everyone that's
12:55
listening a bit of a refresher on what exactly Tecton does.
13:00
I know in 2020, especially when the community was first
13:04
starting, it was hard to go a week without hearing someone
13:08
talk about a feature store. And so there was a lot of
13:13
questions about it because I think of that blog post that you
13:18
were talking about, Matt, that had come out.
13:20
And when Michelangelo was being examined by everyone else, they
13:25
realised, OK, there's some here when it comes to decoupling
13:30
features from code and the other models and all that.
13:33
So Tecton has also evolved, though it's not just a feature
13:39
store, I don't think. Now, can you break down what
13:43
what it does for us these days? Yeah.
13:47
So to to kind of tell this story a bit here, it's interesting
13:51
because like back in 2020, I think if you were kind of in the
13:55
like niche ML OPS type of community, you are aware of what
13:58
this was. But when I joined I, so I joked
14:02
that like when I joined I, you know did everything except
14:04
product management and then kind of just moonlighted as a PM on
14:07
the side and such as the nature of like a super small company.
14:10
And so I was doing early on like a bunch of our demos and sales
14:15
calls and there was but a fair number of them that would come
14:19
in and be like, I have no idea what you guys do.
14:22
I just heard that the Michelangelo guys were starting
14:25
something. So like, tell me what it is.
14:26
And so I'm like, OK, like, let me tell you about this product
14:30
that you didn't even know existed in the first place in
14:32
the category that you've never heard of.
14:34
And so I'd have to really start from scratch in the early days
14:39
of the feature store. It's a very different world at
14:41
this point. And so really the way that I
14:43
pitched it was like, hey, you have all of this raw data that
14:47
you're trying to use to make automated decisions in your
14:50
business, whether it's like detecting A fraudulent
14:53
transaction or making recommendations or, you know,
14:56
close to home use cases for me, like ranking someone's Twitter
14:59
home feed. And in order to get that data to
15:03
that model, it has to be turned into features.
15:05
And then it has to be delivered in two contexts, which is at
15:09
training time. Offline.
15:10
People need to be able to get the values of the features as
15:13
they were any given training event.
15:15
So like, hey, this credit card transaction, we determined it's
15:19
fraudulent at the time at which someone was making the
15:21
transaction, what were their feature values?
15:24
And that's a tricky problem to get right.
15:26
But then also hard is getting those same feature values to
15:30
your model at inference time, all at low latency.
15:33
And what makes this really hard is that, like, it's kind of this
15:36
place where data science and software engineering suddenly
15:39
meet, which have traditionally been a bit of like different
15:42
worlds. Like data science is, you know,
15:44
very scientific. It's very experimental in
15:48
nature. It's kind of the Wild West.
15:49
Pip install whatever you want, figure it out, that's totally
15:52
fine. But the world of software
15:53
engineering like had decades of best practises built up with
15:57
DevOps and CICI D and people thinking about scaling and
16:01
latency etc. And it turned out that like when
16:04
you finally merged the two of those, it didn't go quite as
16:07
well as everybody had hoped. And so there was a lot of room
16:11
there for people to like figure out the right technologies that
16:14
we're going to merge like data science into the software
16:18
engineering realm. And so that's kind of like that
16:20
like key part where the feature store comes in And so we are
16:23
building this out. But what was weird is like at
16:25
the time we didn't know what it needed to be like.
16:28
It was almost like the people on the calls were like we don't
16:30
totally know what this is going to turn into, like we're solving
16:32
some problems, we're working with customers.
16:35
But as kind of time went on, we realised that like it really had
16:39
to do with more than just kind of the online offline storage
16:43
and serving component, which was what it was kind of like
16:47
evolving to mean canonically was like ohh, you know, you have an
16:51
online store, you have an offline store, you serve values
16:54
at training and inference time. But what we found when we talked
16:57
to a bunch of customers who are just having problems is like
17:01
they also had a tonne of problems with actually like
17:04
building an orchestrating and running the feature pipelines
17:07
themselves. So like the streaming
17:09
infrastructure to calculate these values and near real time
17:12
or the orchestration to backfill them or even we started to find
17:18
customers that were like all of my logic needs to be executed at
17:21
request time and I need to be able to manage that.
17:24
And so like there was a whole category of like how do I build
17:27
features and engineer them, There was the storage and
17:29
serving component. Then obviously, like monitoring
17:31
became a natural kind of progression to this, like, OK,
17:34
now need to monitor for skew and data quality.
17:37
And then you know, as we helped more and more big organisations,
17:40
there's also like this huge collaboration component of like,
17:43
hey, how do I share features that another team uses, How do I
17:46
have access controls on this whole system?
17:49
And so I feel like for a while we kept trying to call it a
17:52
feature store. And internally we were like,
17:54
this isn't like checking out anymore because then other
17:56
people would say feature store and they meant something
17:58
different, like is it a feature store?
17:59
Is it, is it something else? And so we finally bit the bullet
18:03
a few years ago and we're like all right, we're going to put a
18:06
stake in the ground. It's a feature platform and the
18:08
feature store is like the core storage and serving component
18:11
which is you know several different solutions for.
18:13
But really what we find is like the feature platform is the
18:16
thing that ends up like really unlocking a lot of the like real
18:19
time AI at these organisations. Yeah, because the feature store
18:26
really just makes people think like, ohh, that's like a
18:29
glorified database. Yeah, right, exactly.
18:32
I can just do that in my database where I don't why do I
18:35
need a whole new product for that.
18:37
But really what I'm hearing you say is there's a lot of other
18:41
hard parts to this. Whether it's orchestrating these
18:45
features and their pipelines or if it's monitoring that the the
18:49
features are actually doing, they're supposed to be doing or
18:52
it's serving them at very low latency, which I know was always
18:57
something that became problematic when you wanted to
19:01
get to, yeah, real, real time as somebody, as some people call
19:06
it. You know, like not just that
19:08
every five seconds, real time or like when you're talking about
19:13
real time, I guess you guys know best, like what is real time to
19:16
you? What does that even mean for
19:18
you, you know? Yeah.
19:20
And really like that's, it's funny you said like there's
19:23
other hard parts, but it's almost like the hardest parts
19:25
actually lie you know in the rest of the system, at least in
19:29
in what I found, I sometimes say even just internally like the
19:33
storage is almost an implementation detail.
19:35
Like it's like it's there so that we can cash values
19:38
effectively and serve them faster.
19:40
But like if I didn't need a intermediate storage layer then
19:44
I wouldn't use one. And even in offline case,
19:46
sometimes we actually don't, which is interesting.
19:48
But like really the, you know, I'm a PM I think in jobs to be
19:51
done, it's like you have some feature that you're trying to
19:53
express or like what's the user is, you know, average
19:57
transaction total in the last one year.
20:01
And you have some requirements like I need to know that value
20:04
as of to your point, like 10 milliseconds ago.
20:07
And I need it served, you know, to my model that's, you know,
20:10
running in my production environment.
20:12
Like you just want to express here's semantically what the
20:15
feature means, here's what my requirements are, and then you
20:18
want the system to worry about. Like ohh OK, there's an
20:20
orchestrator to figure, you know.
20:22
To schedule these and to backfill them.
20:24
And there's a storage layer that's going to help cash these.
20:26
And you know, there's a real time layer to finish computing
20:30
them. But like, you don't want to
20:31
worry about that stuff. You don't want to think in terms
20:32
of like the components and the architecture.
20:35
You're like, I'm just trying to express a thing and when and
20:37
where I need it and how. And then the system takes care
20:39
of kind of getting that all done.
20:41
So I almost think like feature store is just like increasingly
20:44
kind of like a misnomer. Like it feels very like you
20:46
said, like it's indexing on a component or the underlying
20:51
architecture instead of like the problem.
20:53
That is really what people have. Yeah, you don't want to worry
20:56
about the architecture. Let the chief know.
20:58
Deal with the chief. We'll figure it out for you so
21:01
we have him for. Ohh, Classic.
21:06
And I know you all been working hard to update things and there
21:10
was a big announcement. I feel like that came out when I
21:13
did the Apply conference a few months back.
21:16
But since you've also had newer announcements, can you update me
21:21
on what is like the newest stuff you've been doing?
21:25
Yeah, totally. Yeah.
21:27
OK. It off here and Mike can can
21:29
chime in. So historically Tecton has been
21:33
a fairly Spark centric platform and that kind of evolved as a
21:39
consequence I think of the types of customers that we worked
21:43
with. Like you know a lot of that like
21:46
early adopter category was made-up of like these larger
21:50
organisations that had super real time requirements.
21:53
A lot of them ended up being like Spark shops and were very
21:57
you know proficient and Spark. And so we indexed on Spark as
22:00
like a key technology to start inside of Tecton.
22:04
And then we started exploring like the data warehouse space
22:06
and how we would integrate there.
22:08
And like quickly, quickly started realising a couple
22:11
things like 1 when we went out and talked to a bunch of the
22:14
market. Like Spark only really works for
22:16
maybe about half of them and it's kind of either just
22:20
overkill or unfamiliar for the other half.
22:24
But we also realised that like we didn't want to build like all
22:27
these versions of Tecton that was going to be like there's,
22:29
you know, Tecton on Spark and then there's going to be, you
22:31
know, Tecton on Snowflake and you know which version of Tecton
22:34
do you want to get. And so we wanted to design
22:37
something that was like, OK, what's like the one solution
22:39
that's going to work for all of these organisations.
22:42
And so something that we built and released recently that we
22:45
were super excited about is a built-in compute engine detect
22:49
on that we call Rift. And it's entirely Python native.
22:53
So you can run Python And Pandas transformations whether you're
22:57
building like a batch feature, a streaming feature, a real time
23:00
feature, it's super performant. We can do like cost efficient
23:05
aggregations over millions of events in a very large time
23:09
range, which is 2 typically super hard.
23:12
It requires like no external infrastructure.
23:14
So like you can literally hop into a hex notebook or a deep
23:17
note notebook or even a local Jupiter notebook, pip install
23:20
Tecton and do all of your feature development there with
23:23
Python as your only dependency there.
23:25
And then it plugs into data warehouses too, so you can plug
23:29
into your snowflake. It'll actually like push down
23:31
compute to your warehouse to do like initial stages of
23:34
transformations and then kind of follow that up with additional
23:37
logic. And what's cool is it
23:40
dramatically simplifies for a lot of customers like what it
23:43
takes to get started with the feature platform.
23:45
It's a lot more in their wheelhouse.
23:47
The iteration speed is like a lot faster for them.
23:49
Like it just it snappier when you don't need to start a Spark
23:54
job on some attached cluster. If you, you know, have a whole
23:58
company designed around like making Spark, not a giant pain
24:02
to manage, then like, you know it can be all right.
24:05
Or if you're like a Spark shop, right and you're like we're
24:08
really deep in Spark and we know how to manage it, then great.
24:10
But like, if those things aren't true like you, you don't want to
24:13
be diving into it unnecessarily. You're like, look, I'm a, you
24:17
know, I'm a Python shop, I'm a data warehouse shop, like I just
24:20
wanna pip and saw something and be off to the races.
24:22
And so that's what we've been working on.
24:24
That's what our chief here has been helping design out for us.
24:29
And yeah, we finally launched it into private preview in
24:33
November, racing towards a public preview.
24:35
Already got a few customers on board and and go into production
24:39
and we're super excited. And there's one thing I want to
24:42
say before me, some jumps in, which is you mentioned before.
24:47
I feel like it was the understatement of the year of
24:50
when data scientists had to do software engineering stuff or
24:55
even data engineering stuff back in the day.
24:57
That is a recipe for disaster almost 10, nine times out of 10.
25:03
And what you're talking about here, if I'm understanding it
25:07
correctly, as you're saying, look, we know it can be painful
25:11
to use Spark if you're not used to using Spark.
25:16
So we want to just get that out of the way and we want to come
25:20
to where you are instead of you having to come to where we are.
25:24
Yep, that's. Yeah, that's exactly right.
25:26
And to add to that, it, I think it wasn't even just a matter of
25:29
like people not having prior experience with Spark.
25:33
We were also kind of asking them to use Spark for use cases that
25:37
didn't make a whole lot of sense.
25:40
So in a lot of cases, you know some of our, you know, some of
25:44
our customers would be dealing with data sets that were like a
25:47
few 10s of gigabytes and we're you know spinning up these giant
25:51
Spark clusters in order for them to process those data.
25:55
That's just gonna make a lot of sense and it wasn't worth the
25:57
extra complexity that Spark comes with.
26:00
So what are the big design principles of like this new
26:04
piece of the product we have is just like kind of choosing the
26:07
appropriate tool for the job and and not going overboard?
26:12
So I'm very close to Amsterdam. I live right outside of
26:16
Frankfurt and Amsterdam is like 4 hours away and I noticed that
26:21
there was gonna be duck DB con and I was looking at who was
26:23
speaking and I feel like somebody's name on here on this
26:27
call popped up. Was it you?
26:30
What? Is it true?
26:31
Is it the same person? That's that's, yeah, it's the
26:34
same person. Even though it says Chief
26:36
Architect instead of Chief, that's that's still.
26:39
You we'll get that fixed. And so, is that what's going on
26:44
under the hood? Are you running Duck DB and is
26:47
that how you're able to do things?
26:49
Still fast, but not having to be reliant on Spark.
26:53
Yeah, that's a big piece of it. So part of what Tecton provides
26:57
is sort of this kind of built in library of queries that we kind
27:04
of combined with the user query in order to prepare you know
27:09
their feature data to be materialised for online or
27:12
offline serving. So we've used DB for all of that
27:17
and it's been a really great experience so far.
27:20
We've we've really enjoyed using it was really quick to get going
27:23
and we've been impressed with the results that it can get with
27:27
limited resources. So yeah, I'm I I am going to be
27:29
giving a talk about that at Duck on the I'm looking forward to
27:33
meeting more users library. It's been really great so far.
27:36
Are you going to be in Amsterdam?
27:39
Yeah, coming to answer for that. All right, Looks like my choice
27:44
has been made. I wasn't sure if I was gonna go,
27:47
but if you're gonna be there, I guess I have to go now and I'll
27:50
be able to say hi in person. That's super cool.
27:53
So tell me more about what's going on under the hood with
27:57
Rift. Yeah.
27:59
So like I mentioned part of it is Doctor B and then we I I
28:03
would say really the the most underlying piece of everything
28:06
is that we sort of shifted everything so that it was all
28:10
the everything is built around Arrow.
28:12
And the reason we chose to do it this way is because we had part
28:16
of the design goal was that we really wanted to be able to
28:20
easily integrate with a bunch of different warehouses and you
28:24
know different data engineering libraries like pandas or polars
28:28
or even Duct B if you know our customers end up wanting to do
28:32
their feature engineering in SQL.
28:34
So we've built this thing out in sort of a modular way where we
28:38
use Arrow to exchange data between the different stages.
28:42
And so it's typical query would look like maybe the customer
28:48
sets up data source which is a snowflake table and then they
28:54
give us a transformation that they want to run in order to
28:57
produce feature data from that table and then they want to you
29:02
know serve those features through our online serving
29:04
infrastructure. So what we would do because we
29:07
were first run the query and snowflake potentially applying
29:11
philtres or projections in the snowflake query based on what
29:15
the user's asked us for. If they're only interested in a
29:17
subset of the data or something like that, that gets streamed
29:20
back to our job and then we pass that as like an arrow data set
29:25
and deducted B where we can do aggregations on it depending on
29:30
what the user configured. And then that gets passed out of
29:33
duct. BB is another data set and then
29:35
that's where we upload it to our online database and then later
29:39
when you know later on when they come to query that, then we can
29:42
serve it out of that online database with like reliable low
29:45
latency. So that's kind of like the batch
29:49
workflow. We also have a separate
29:52
interface where people, you know have like a stream of data that
29:55
they wanna create features from. We have like an HTTP interface
30:00
where they can just send us a row of data at the time at a
30:02
time and we can apply similar transformations to what I was
30:06
just describing for the batch workflow.
30:09
And so I know that when I've looked at other engineering
30:14
blogs a lot of times you'll see like the Kafka, Flink, kind of
30:21
pear come U. And I imagine that you guys
30:25
thought long and hard about the architecture of this and the
30:29
design. What are some design decisions
30:32
that you were a little bit sceptical of making, but now
30:37
that you've done it and you've seen the fruits of the labour,
30:41
you are like, oh, I am going to do this nine times out of 10
30:46
next for the next whatever, 100 times that I do this right.
30:50
Yeah, I think you know one big one is that rather than you know
30:54
using kind of a purpose built stream processing engine like
30:58
Flink, we've gone for this kind of simplified approach where you
31:04
know our stream ingestion, it's actually just stateless.
31:07
So you know, in contrast to flake where it has to, you know,
31:10
depending on what types of aggregations you're trying to
31:12
do, you have to, you know, potentially maintain these very
31:16
large query states which can lead to a lot of operational
31:18
issues. Hey, this is Mike Delbalso,
31:22
Co-founder and CEO of Tecton. ML OPS community is the best way
31:27
to stay in the loop on the latest ML OPS news and best
31:29
practises. It's also a great way to connect
31:32
with experts and get support from an amazingly.
31:34
Helpful. Community subscribe and stay in
31:36
the loop. Ingestion time we all of our
31:47
processing is totally stateless. So you can philtre things out or
31:50
you can do projections but no like aggregations at
31:52
materialisation time and then what we do is we just actually
31:57
do the aggregation observing time and so that the benefit of
32:03
that is that like the it's it's very simple they're like a lot
32:07
fewer things to go wrong. You can't run out of disc space,
32:10
older streaming checkpoints, you don't have to worry about, you
32:14
know, the latency of the stream query like committing A
32:17
checkpoint. Basically the critical path from
32:21
like data going into being available at serving time is is
32:26
just dead simple. And I think that served as super
32:29
well and it's made it, you know, not just possible but actually
32:33
like relatively easy for us to you know operate this managed
32:37
service in a way that's like meets our customers rely very
32:41
reliability goals which are pretty high.
32:44
I mean we, you know, we have customers that are using this
32:46
for things like you know, fraud detection for you know, credit
32:52
card transactions and things like that, you know, use cases
32:55
where they want it to be you know, online nearly all of the
32:57
time. So that was a super important
32:59
design goal for us. The downside of that of course
33:03
is that like if you get super large aggregations, it can, it
33:06
can start causing performance issues.
33:08
And so we're currently actually working on a project to kind of
33:15
do those what we call compaction.
33:18
So if you're aggregating over a large time series, we don't, you
33:23
know, do the the aggregation and materialisation time, but we'll
33:27
sort of ingest the data later on we'll come back and do the
33:30
aggregation and replace it in the database.
33:33
And so you can kind of get the best of both worlds that way.
33:36
You talk about how if you have larger aggregations, what I
33:45
guess what is large in your eyes, define large and what that
33:50
looks like. And I don't want to be the guy
33:51
that says because I know in every talk that has ever been
33:56
given at a conference you always have one person that will raise
34:00
their hand and be like, so how does this scale?
34:03
And I don't want to be that guy right now, but I'm going to be
34:05
that guy. What is large and what is not
34:09
large? Yeah.
34:10
So I mean I guess the typical like engineering answer to this
34:14
question is always it depends and it does depend in this case,
34:20
I mean in terms of. So there's a couple of different
34:22
issues where you might imagine there would be, you know a
34:25
couple different dimensions where you might have trouble
34:28
scaling. One is like the amount of data
34:31
that has to be aggregated for a single feature.
34:34
I think with our current architecture, without the
34:36
compaction, we're doing pretty well aggregating a single series
34:41
of maybe a few 10s of thousands of data points and potentially
34:45
even in the low hundreds of thousands pass that that can
34:49
start to be a bit of an issue. I mean, of course it depends on
34:53
the latency budget that that particular use case has.
34:58
So you know, 100,000 is like plenty for a lot of use cases.
35:02
But for, you know use cases where people have more than
35:04
that, we're working on this compaction feature to address
35:06
that. And then another dimension that
35:11
you might be concerned about is like the total size of the data
35:15
set that you're trying to transform into feature data for
35:18
materialisation, kind of like in the batch use case.
35:22
And it's a little bit early days for us with this, like we, you
35:26
know, we're still improving. Preview we haven't done, we
35:28
don't have like a whole lot of experience.
35:30
But what we found so far is that Duck D actually does Duck DB and
35:35
Arrow do surprisingly well with large datasets.
35:40
They are much more memory efficient uh then like pandas
35:43
would be for instance. So I think a lot of people
35:45
probably have you know, the experience of like trying to
35:48
load U like a 5 GB data set in the pandas and then they find
35:53
out like it's actually exceeded, you know the 32 gigabytes of
35:57
memory they have on their laptop.
36:00
Arrow and Ducted B are both more efficient about the memory they
36:04
use. And then they also have ways of
36:05
processing larger than memory datasets where you kind of just
36:08
go through a chunk at a time. And so the combination of those
36:12
means that we're able to handle pretty large datasets like
36:17
better than what you might think of your used mostly in the
36:20
pandas. We also make it like pretty easy
36:23
for people like vertically scale.
36:24
So you know, materialisation time, it's, you know, choose an
36:28
EC2 instance that you want. You can scale it up really big,
36:31
it can process a figure them out there and we find it works for
36:35
honestly the majority of use cases.
36:36
I don't know if you saw D there was a really good blog post from
36:40
I wanna say you as a product manager of Big Query.
36:42
It was called like Big Data is Dead.
36:44
That came out somewhat recently, basically arguing that like you
36:47
know, big data has been the whole wave and that's where
36:50
everything's going, etcetera. But really it's like, you know,
36:53
a very small percentage of companies that have truly large
36:56
data and instead the vast majority of people even, you
37:00
know, on a large single note instance can actually process
37:03
most of what they need to. And there's other like clever
37:06
ways of being able to to optimise this.
37:09
Like Mike's been talking about and you know, we split up, split
37:12
up, backfill ranges intelligently.
37:16
We, you know, do optimizations like he's talking about with
37:19
compaction to make things simpler along the way.
37:22
And so I think what we find is like it actually turns out for
37:25
the vast majority of use cases, you can get super far scaling up
37:28
on a single node. And so it's not worth like all
37:31
of the additional complexity that needs to be introduced by
37:34
like large scale distributed processing.
37:37
Yeah, I think if you go and look at that blog post, so you kind
37:40
of identifies one terabyte as sort of like the maximum size of
37:46
like a data warehouse that you would expect to find in most
37:49
companies. And if you look you can actually
37:51
from Amazon, you can buy EC2 instances that have 24 terabytes
37:56
of memory in them now. So the the size, you know limit
38:01
for vertical scaling these days is pretty large compared to what
38:05
lots of people actually have in their data warehouses.
38:08
Yeah, actually I think that was the speaking about Duck B that
38:13
was the creator of Mother Duck like the managed.
38:16
That's right. TB he he went from Bigquery and
38:20
he he started to do Mother Duck and he was talking about that
38:24
how? Because we just had him on
38:26
recently, like whatever a month ago to this podcast and he was
38:31
mentioning how it felt like for a while.
38:37
If you weren't over engineering things for all this big data,
38:41
then were you actually an engineer?
38:43
You know? And so it was almost like you
38:45
were getting peer pressured into doing more engineering than you
38:50
needed to because you were expecting to get this influx of
38:54
data or you were expecting to be working with gigantic amounts of
38:57
data. And then when push comes to
38:59
shove, the data scientists and the people that are actually
39:02
using and consuming this data, maybe they don't even want that
39:07
much of the data, they just want like the last weeks worth of
39:10
data because of the freshness constraints that they have.
39:14
Yeah, I personally, I'm super happy that that blog post is
39:18
around and it's getting a lot of traction because I think like
39:22
you know, it's kind of one thing to be able to look at the
39:25
performance of these systems and come to the conclusion yourself
39:28
like, well this will be adequate for what our customers need.
39:31
But like you also have to be able to convince people that it
39:35
will actually be adequate. And so the fact that there's
39:38
sort of this growing consensus around kind of simpler single
39:43
node systems, you know, led by people like Mother Duck, I think
39:48
has been super helpful for this, you know, this new product to be
39:52
successful for us. That's so.
39:53
True. Really.
39:55
I think combining that with this kind of stream architecture that
39:58
Mike talked about has been like the biggest factor in us
40:02
simplifying the product. And it's kind of a similar thing
40:05
of like you could easily over engineer the problem and be
40:08
like, yeah, we're going to, you know, run a Flink cluster under
40:11
the hood and you're going to worry about, you know, managing
40:15
and provisioning streaming infrastructure.
40:17
Etcetera. And like it turns out, they're
40:18
like the majority of what people need is like a dozen or so
40:22
aggregations. And really what they want is
40:24
like they want very limited operational overhead.
40:28
They want to be able to just use like Python And pandas to run
40:30
transformations and then they needed to be performant to their
40:33
like latency or scale. Which generally means like hey,
40:37
I have 100 millisecond budget to be able to get features back,
40:40
but I need to be able to have an aggregation that spans like you
40:43
know, the entire lifetime of an account maybe.
40:46
And so indexing much more like we'll just make that as simple
40:50
as humanly possible. I think has been kind of you
40:53
know, retrospectively looking at it, one of the the better design
40:57
decisions along the way. That's I think kind of the
41:00
mixture of that with Rift with kind of you know simplifying the
41:05
like away from the large scale distributed processing when it's
41:09
not needed. Like that's been dramatically
41:11
helping us lower the complexity of the product.
41:14
And Matt, you know how every engineering blog posts, when you
41:20
talk about creating some kind of a product, there's like here's
41:25
our fundamental principles or the guiding lights that we were
41:29
looking at and our constraints and that kind of thing.
41:32
Did you have principles as you were putting this together and
41:37
you were going through the product creation phase?
41:40
Yeah, definitely. One was the the simplicity and
41:45
getting started like I said, like it can't, it can't take
41:47
more than a pipe and salt tacked on for a data scientist or an ML
41:51
engineer to start to be productive with this.
41:54
If suddenly you're like, ah no, you're going to need to like
41:57
have some sort of spark provider external compute that we plug
42:01
into and you're going to need to know how to use XYZ in order to
42:04
do this. Like that immediately would be
42:06
kind of a no go. Yeah, more skills.
42:10
And exactly right. Like it needed to be like look
42:13
if if you know Python, like we're we're good to go.
42:16
You know you can start using the product.
42:19
So that was one for sure like testing and iteration speed is
42:23
another thing that we really wanted to to optimise for.
42:26
So kind of in that same vein like you should be able to work
42:29
in whatever environment makes sense to you.
42:31
And that goes like the principal Mike was talking about, like
42:33
let's meet people where they are and let them choose the right
42:35
tool that makes sense for them as opposed to saying like, ohh,
42:38
you know, I know that to work in deep note or hacks.
42:41
But now you're going to need to learn how to use, you know,
42:43
Databricks notebooks as your core development environment and
42:46
you're going to have to use Spark, etcetera.
42:47
It's like, hey, if you work in hex and you like to use Pandas,
42:51
sick. If you guys are a data warehouse
42:54
diehard shop and you want to be using stuff like SQL for all of
42:57
your feature engineering, that's awesome too.
43:00
Like we can make that work. And so like meeting people where
43:03
they are and working inside of their environment and
43:05
simplifying that testing and iteration loop.
43:08
That was also like a big element of I think what we were trying
43:11
to go for as we, we built out this solution.
43:14
And then like it said kind of in that same vein like we really
43:18
wanted this to just be like 1 product.
43:19
Like we didn't want to have all these different versions of Tech
43:21
Tonne built on different underlying compute technologies.
43:24
And so we needed to find something that was like modular
43:27
and flexible enough that whether you're using Spark or Pandas or
43:31
Snowflake Sequel or whatever else like that fits nicely into
43:35
the platform. And that kind of goes to some of
43:37
like the architectural decisions that Mike talking about of like
43:39
standardising on the Arrow format etcetera and also
43:42
allowing us to evolve because you know today it might be data
43:47
warehouses or everyone's in on pandas.
43:49
But like the the industry's going to keep evolving.
43:52
This isn't going to be like the last time that customers come to
43:54
us and like want something else baked into their feature
43:58
platform to be made really simple and to fit with the tools
44:00
that they're using. And so we wanted it to be able
44:02
to kind of to kind of grow with that.
44:04
And so I think that's where we hit like a good best of all
44:06
worlds. And you know, we talk about a
44:08
lot of this and contrast this part, but actually how we
44:10
designed the system was like we wanted them to both kind of live
44:15
harmoniously under one roof. So like we still have a lot of
44:17
customers that for very good reason use Spark.
44:20
Like we have customers that are generating, you know, 90
44:23
terabyte training datasets. And what's interesting is that
44:27
like even within an organisation, like there's teams
44:30
that use Spark and then there's teams that don't and there's
44:32
teams that use the data warehouse and then like there's
44:34
teams that don't. And so it's not even just like
44:36
OHH, some customers will use one or the other, but it's like in a
44:39
single organisation you might have like different data
44:42
scientists or ML engineers choosing different technologies.
44:45
And we want all of those to like work together to go into a
44:49
training data set. If you need to scale up and use
44:51
Spark, then cool. If you don't, then like use
44:53
Rift. And so we designed it all to
44:55
just like live really nicely under one roof out the Git go.
44:58
So it's one product and under that product you just reach for
45:02
the tool that you need and they all work together.
45:03
And that's what really like let's your data science team
45:07
collaborate effectively together.
45:09
Yeah. To to add to that, I would say
45:11
that like kind of ironically I think the existence of the Spark
45:15
version of the products really enabled us to make a lot of
45:18
product decisions to simplify Rifts.
45:20
Because there was always an option to say like you know, if
45:23
you know we can, there's a tradeoff here where we could
45:26
make the product easier to use, but maybe like somewhat limit
45:29
the maximum scale that it can get to in this circumstance
45:33
without circumstance. And by having Spark available,
45:37
it made it a lot easier for us to say, OK, if someone hits
45:40
that, they can, there's this kind of escape hatch to go to
45:43
Spark. And so we can kind of focus more
45:47
on the like 90 or 95% use cases without worrying about the last
45:51
5%? I love that because I in my mind
45:57
it seems like a lot of times there's the battle between
46:01
simplicity and flexibility. And it almost is like you were
46:05
able to get the best of both worlds there and say we're going
46:09
to go for simplicity, but we're also going to make it very
46:12
flexible. And if you hit that certain
46:15
scale, then we can bring in the big guns and we can go and get
46:19
even more firepower behind you. Something we talk about
46:22
internally, you know Kevin, our CTO and Head of Product, him and
46:27
I talk about this a lot though like I think as a product it's
46:30
really important to make like the simple things simple but the
46:33
hard things possible. And so if what you're doing is
46:37
like you know, working with small to medium sized data sets,
46:42
then it should not be hard at all for you to get started and
46:46
to be able to you know, engineer your features, scale up
46:50
appropriately, etcetera. And but if you're really going
46:53
to the most extremes where like I said, we have some of our
46:56
customers there like that's got to be doable.
46:58
And like there might be incremental flexity that you
47:00
accept as you do that necessarily, but like that
47:03
complexity, you can't then like spoil the easy cases in your
47:07
product or any. And so I think part of that is
47:09
like giving that optionality and knowing that like you know, you
47:13
can kind of progress with people through that complexity by
47:15
allowing them to like choose the tool that makes sense and like
47:19
voluntarily take on that complexity as their use case
47:23
requires it. So you ready for the part of the
47:29
show? This is a new segment that I'm
47:32
gonna call. Is he serious right now and or
47:38
even better I'm gonna call it is this guy on drugs.
47:41
And the questions that I am going to ask you now may seem
47:46
like they come from an A drug induced stupor, but I've heard
47:50
murmurs and I want to know if you all have been encountering
47:55
this out in the wild because it's your day in, day out.
47:58
One of these questions comes from features, and looking at
48:04
features and seeing that in the data world, features could.
48:10
Maybe they're not that far removed from what data analysts
48:14
use as KPIs. Have you ever seen features and
48:19
KPIs being shared or used in almost like the same way?
48:24
What does that look like? Explain that.
48:28
I was expecting an even crazier question, so I'm I'm fully.
48:31
Prepared Excellent. But yes, definitely we see some
48:38
kind of cross pollinating in that world.
48:41
And so this kind of comes in in two different forms.
48:44
One, sometimes we'll see like hey I want my data analysts to
48:47
be able to like create features or I want like the stuff that
48:51
they've already created to be able to be used as features.
48:54
And then the other way is like hey I've created all of these
48:57
features inside of my feature store.
48:59
But like these could be really important business metrics for
49:01
us. How do I get that like you know
49:03
back available inside of my analytics systems.
49:06
And so something we've done for the latter that we published
49:10
recently is actually the ability to like take any of your offline
49:13
feature store data and publish it back into your warehouse.
49:16
And so it shows up just like as standard snowflake tables for
49:21
example. You can find all of your
49:23
features there and tactile and takes care of all of the, you
49:26
know, orchestration and publishing of all of that
49:28
feature data. And so that's one way that we
49:29
kind of like get the features back in the analytical world so
49:32
that they're available in both systems and then analysts using
49:36
UH, using a feature platform. That one's interesting, like one
49:40
that's important is like or like.
49:42
One element that's important is definitely allowing them to
49:44
engineer with things like Snowflake SQL and what they're
49:47
familiar with. But I think there's also room
49:50
for feature platforms like lower the barrier of entry and make it
49:54
even easier to work inside of the platform, etcetera.
49:57
Some things that we do is make it really easy to take an
49:59
existing snowflake table and just be like this is my
50:02
features. So we can get those really in
50:05
really easily like that. But I would say like it's it's
50:09
in the realm of like things that we think about and things that
50:11
we get asked about quite a lot. I do think that there's like a
50:15
there's good overlap in that you want like features and business
50:20
metrics available in both systems.
50:22
But I think that there's also enough divergence that like you
50:25
don't want it to be the exact same system.
50:27
Like there's a, there's a good reason why.
50:29
Like analytics teams have their tools and data science and data
50:32
engineering teams have their tools.
50:34
Like they hit on different requirements and different kind
50:38
of like interfaces that those teams are familiar with.
50:40
And so I don't think you actually want them to like
50:43
converge to one tool. I just think that you need that
50:45
data to be really easily shared between them.
50:48
Yeah, it's like 1 hop away as opposed to being the same thing.
50:53
Yeah, so I gotta think of a better name for this.
50:56
This one is gonna be the part of the show.
50:59
This is the segment that we call.
51:01
Can you get anymore hype? And you know, it was coming, but
51:06
how are people, How are you seeing people use LM's when it
51:10
comes to feature stores and are they creating features with LM?
51:15
I know there is actually one guy, summat I think that I
51:19
follow on LinkedIn who's always talking about LM and recommender
51:23
systems. And so it feels like there's a
51:25
lot of potential there. But again, you're in the wild
51:30
day in, day out. Is anyone actually doing stuff
51:33
in production with this? That's so good question.
51:37
Definitely saw that one coming because I think if you're in the
51:40
data space at all you're you're getting hounded with this
51:45
regardless of what what team you're on or organisation.
51:50
So yeah, for a while like, you know, we ourselves had to figure
51:52
out like what does this story actually look like?
51:54
Like, where do feature platforms fit into this space?
51:58
You know, do we use LLM to be like creating features?
52:01
Like maybe people are like giving human readable
52:03
descriptions of features and were then like in the same vein
52:07
as what I talked about before, like we worry about all the data
52:09
engineering under the hood and you worry about just like
52:11
telling us what your feature is. So we explored that path a bit,
52:16
and I do think there's actually some, like, real, real fruit
52:19
there. That's like English to feature
52:21
basically instead of like or text to feature.
52:25
Yeah, pretty much. Because like we we have a giant
52:26
repository of, you know, feature definitions and associated
52:31
descriptions and things like that.
52:32
Like it's not unreasonable that you could go from a human
52:36
readable description of a a feature to all the, you know,
52:39
data engineering pipelines needed to make it happen.
52:42
But I think the most interesting thing for me of how they fit
52:45
together is like if you like zoom out and you kind of look at
52:49
what these systems are doing right.
52:51
And like, you know, traditional predictive ML, you have data
52:56
that is getting turned into like relevant information that then
52:59
goes into some model which is making a prediction, which is
53:02
then being used to affect the application in some way.
53:06
And as we kind of integrate WLMS into software applications, it's
53:10
kind of fundamentally the same thing.
53:12
Like you have relevant information, that relevant
53:15
information needs to be turned into a format that's that can be
53:18
digested by an LLM. You take that, you put it into
53:22
an LLM, it gives you back some you know results, you know in
53:26
most cases and traditional ML it's like some prediction here
53:30
it might be like some generative text and then that goes into
53:32
your application and changes the behaviour and so like there's a
53:35
tonne of parallels. It's like the LM systems and I'm
53:39
sure you've talked all about like the the RAG design with
53:42
like a retrieval augmented generation.
53:45
So like you have some data instead of turning it into
53:49
features like you're turning it into embeddings and turn instead
53:52
of loading it into a key value store, you're loading it into
53:55
some vector database. Instead of, you know, writing
54:00
some rule on some like prediction that you get back,
54:04
you're like taking back all of that text and you're like
54:06
inserting it back into your application to show to a user.
54:09
And so like a lot of the core components of this are all the
54:12
same. Like at the end of the day, you
54:14
have data that's information it needs to go into model.
54:17
The results of that, that will need to go back into your
54:19
application. And so I think there's a tonne
54:21
of overlap in like the fundamental problems of how do
54:24
you productionized this, how do you integrate it into an
54:26
application and how do you get that relevant information like
54:29
think like feature stores, getting features for a model.
54:32
It's really not that different to like, get relevant business
54:34
data to an LM so that if you say like, you know, if you're a
54:38
travel website and the person's like, hey, I need to figure out
54:44
where I'm gonna go next summer, it's like you want that LM to
54:47
have relevant information about that person, what they've been
54:49
doing, like where they've been before.
54:51
And like, that's honestly the same role that feature stories
54:54
play today. I think they'll play that same
54:55
role in kind of these rack systems in getting that
54:59
information there. And then I think also there's
55:03
another interesting part that I've been looking at where we do
55:07
a lot of like these real time feature pipelines where like you
55:12
know, at request time someone might need to like run some
55:15
logic to compare an existing transaction amount to a
55:18
historical one. And I actually think there's
55:20
some really cool things we can do right in that space with like
55:23
kind of like managed prompts, where like you're storing your
55:27
prompts as code, but then at retrieval time you're kind of
55:30
hydrating in relevant information from your feature
55:32
platform. So you're like, ohh like tell me
55:35
all this useful information about the user, then take those
55:38
values, inject them into these prompts, then take that prompt
55:41
and call out to an LLM. Then get that LM response, parse
55:45
it, send it back. And so you then just have one
55:47
API where you're like, you know, I'm putting in some like user
55:50
query here and the feature platform is taking care of all
55:53
of that. And where you used to have like
55:55
hardcoded features, you might actually have like dynamic like
55:58
prompts that you're then feeding into LMS. Ohh, I like that.
56:03
Yeah, I've definitely seen that as far as a pattern goes with
56:08
prompts, you need to give that context.
56:10
Like the more context you can give the prompt, the more that
56:14
you're going to get a a valid answer or useful answer, right?
56:18
And so, if you can give context around how many times did this
56:22
person look at this flight in the past 24 or 48 hours, or
56:26
where else have they travelled to and they've given these stars
56:31
on Google. All of that context is going to
56:33
be really useful and not is features right there, like
56:37
everything that I just said those.
56:38
Are all things. Right, so I.
56:41
And it's really the same thing. Like you're taking that data,
56:43
you're putting it in the model, you're taking those results or
56:45
doing something with it. Like it was almost funny.
56:48
Like, looking back, they're like, oh, that's actually so
56:50
obvious. If you look at the system at
56:52
first, you're like, well, how did these fit together?
56:53
And you're like, wait, it's like kind of the same thing with that
56:56
word, doing, like, you know, maybe the individual components
56:59
are now a little bit different along the way.
57:01
But like there's no fundamental difference in taking data,
57:04
giving it to a model, taking the results and changing the
57:07
behaviour of your application. Incredible.
57:10
Well, fellas, I guess I've got one last question for you before
57:15
we go. If I'm out there and I'm
57:17
thinking about what you're going to be doing next, where what's
57:24
on the road map and what do you want to sink your teeth into?
57:29
I guess as a PM I can kick this off.
57:32
So certainly a lot of this LM stuff that we're talking about
57:36
here, we're you know actively making investments into this
57:39
area. So stay tuned with more to come.
57:43
That's definitely a big area for us.
57:45
I think continuing to make this Rift product successful and
57:48
lower the barrier to entry. Like I think you know we're
57:51
still in the early days of a future platform.
57:53
There's a lot of people out there that are trying to figure
57:56
out how to infuse their products with real time decisioning,
57:59
whether it's with an LLM or whether it's with, you know,
58:03
traditional predictive ML types of applications.
58:07
And I think we want to make it really easy, make it possible,
58:12
allow these teams to make their applications way smarter.
58:15
And Rift is going to be a big part of us doing that.
58:18
And then I think it'll be a lot of fun for us to expand beyond
58:23
just managing features to managing really those entire
58:27
real time decisioning pipelines, everything from like submitting
58:30
comes to LM's to running multistage ranking for
58:35
recommendation systems. I think that's where Tecton is
58:37
going to continue to go in the future.
58:39
Curious, Mike, if you have any thoughts around this?
58:42
Yeah, I mean definitely I have a lot of focus on Rift right now.
58:46
I mean that's you know, kind of one thing to be developing
58:49
something you know within the company and but you know of
58:52
course when you get users onto it, they find all sorts of
58:55
things you might never have thought of that break.
58:57
And so you know, focusing a lot on making sure that you know we
59:01
make that successful. Another one I'm excited about
59:03
and Rift is part of this, but we also have some other things in
59:07
the works is like making it easier for people to actually
59:10
try out Tecton. So historically it's you know
59:13
we've had a fairly involved like people have to talk to sales
59:16
folks in order to get their hands on the product and we're
59:21
we're focusing on making it a little bit easier for people to
59:23
kind of test drive things. So I'm I'm excited for that in
59:27
the next year. Yeah, it feels like Riff lends
59:30
itself nicely to that, just the pip install.
59:32
Tecton is a much different story than, Yeah.
59:35
So we'll talk to me about your Spark cluster and yeah, exactly.
59:40
How about you, Dimitrios? What are you excited about in
59:42
the next year? Ohh man, what am I excited
59:47
about? The main things that I think my
59:51
focus are on right now are virtual conferences and having
59:57
awesome experiences as you all know, whether that is Tecton
1:00:01
apply that we're going to be doing pretty soon here or it's
1:00:05
the AI in production conference that I'm putting together or it
1:00:10
is. Any other slew of conferences
1:00:14
that I can con people into letting me be a part of?
1:00:19
That is what I'm excited about. But then I'm thinking about
1:00:25
doing an in person conference. It's just a little bit scary
1:00:28
because I could totally go bankrupt if I do it wrong, you
1:00:33
know, like that's the big part on it.
1:00:36
Yeah. It's not like a virtual
1:00:38
conference where, all right, nobody shows up, no big deal.
1:00:41
If nobody shows up, I've got a whole warehouse rented or
1:00:45
whatever. And and I do wanna do it
1:00:47
differently. If I do a conference, I don't
1:00:49
wanna do a conference. I wanna do like a hullabaloo or
1:00:53
something like that. I want to do an actual festival
1:00:57
type vibe as opposed to a conference type vibe.
1:01:01
So people in costumes and all that stuff but for.
1:01:05
The Coachella of Data Conferences.
1:01:08
Yeah, yeah, that's that's kind of it.
1:01:10
It's just, yeah, it's really like I gotta, I gotta work up
1:01:14
the courage for that one. So we'll see if in Q3 or Q4 that
1:01:18
actually happens that materialises or not.
1:01:21
And in the meantime the other thing like my main focus is the
1:01:24
virtual conference. But the other thing that I've
1:01:26
been focusing on that has been an absolute blast because I've
1:01:29
learned a tonne about what people are doing in the wild is
1:01:33
the surveys that we do. So we do surveys and then I kind
1:01:35
of racked my head against the wall or I hit my head against
1:01:38
the wall for few months, try and gather data and understand the
1:01:43
data and really parse it out and then I write a report on it.
1:01:48
So that's something new that I did last year with the LMS and
1:01:53
production survey and report that came off the back of it and
1:01:58
we're doing it with the evaluation survey right now.
1:02:00
So how people are evaluating their systems And so there's
1:02:06
it's like it's a painful process, but it is exciting
1:02:09
because I learn and those are the two things that I've got on
1:02:12
my mind. That's cool.
1:02:15
If you if you pull the trigger on your conference festival,
1:02:19
send me an invite, I'll be there.
1:02:21
There we go. I'll bring, I'll bring you some
1:02:22
costume. Yeah, Chief, you gotta come with
1:02:25
a headdress and all of that. I'm expecting big costume from
1:02:28
you. That might be seen as it might
1:02:32
not be politically correct, it might be appropriated
1:02:36
scholarship. You have to think about that one
1:02:39
a little bit. Yeah.
1:02:41
Well, yeah, we'll we'll figure out a different costumes, right.
1:02:44
Well, so guys, this has been awesome.
1:02:47
I really appreciate you coming on here and getting to chat with
1:02:50
you again. And of course, like hopefully
1:02:55
when we meet in person again, we will get to make some music
1:02:59
because I have fond memories of the last time that we were
1:03:02
hanging out in person making music.
1:03:04
People probably don't know this that are listening, but Matt is
1:03:07
an expert drummer and you give him a table talk, he'll make it
1:03:11
into a drum. Even so, pairing that with my
1:03:14
poor guitar skills is a recipe for disaster.
1:03:17
But that doesn't mean we don't do it.
1:03:19
You are underselling yourself. I think some good jam sessions
1:03:24
are for sure in our future. Excellent.
1:03:27
Well, fellas, I'll talk to you later.
1:03:29
This is awesome. Good catch it up, Dean.
1:03:31
Yeah. Thanks for having us.
1:03:36
Pops, you have to immerse yourself in the ML OPS content.
1:03:41
The best way to do it is to subscribe to the Inner OPS
1:03:45
Community Podcast, so good luck and keep learning.