Latest / Agentic Conversations (formally mlops.community) / Streaming Ecosystem Complexities and Cost Management // Rohit Agrawal // #302

Transcript

Streaming Ecosystem Complexities and Cost Management // Rohit Agrawal // #302 · Agentic Conversations (formally mlops.community)

0:01
So I, my name is Rohit. I work at Tecton.
0:04
My title is director of engineering.
0:07
I essentially just lead a few engineering teams.
0:09
And for coffee, I'm a huge fan of Cometeer.
0:12
Actually. I like my coffee like basically
0:15
how easy and simple I can make it.
0:17
So I started using Cometeer back in COVID when like all the cafes
0:20
had shut down and I became a user then and then I was just
0:25
since then, like I think it's one of those COVID things that I
0:29
just stuck probably maybe the only thing.
0:31
And I still drink Cometeer essentially.
0:34
People of Earth today we get into it with my man Rohit all
0:38
around streaming data. This real time low latency use
0:44
case that can be associated with traditional ML.
0:50
Now I want to tell you a quick story before we get into it
0:52
because over the holidays about two months ago, I asked everyone
0:59
in the community what they wanted to see for the new year.
1:03
And many of you said I want to see some traditional ML stuff.
1:07
It feels like we've been talking about the AI and LLM booms too
1:12
much. So this episode right here is
1:16
for your viewing pleasure. And if you are listening on
1:23
podcast stations around the world, we've got a recommended
1:27
song, recommended listening, which comes from a fan, Amy or
1:34
AMI, because I've been asking folks what their favorite music
1:40
is when they join the community. And this one is a gem that I did
1:50
not know about. It's Gibran Alkoser and the song
2:01
is called Idea 10. Testing production, that's how
2:32
we're running with this. Basically this conversation came
2:35
about because I was on a call with you and Kevin, the Co
2:41
founder of Tecton and you guys were explaining to people all
2:47
about streaming and the situation that the data
2:52
streaming and streaming ecosystem looked like.
2:55
And I thought, man, that is such a cool topic for a podcast.
3:03
I would love to talk to you more and dive into it.
3:06
So we should probably just start with like, what is your
3:09
day-to-day? You're working at Tecton.
3:11
What are you doing there? Yeah, absolutely.
3:14
So I joined Tecton about like four years ago.
3:17
So I joined as an IC. We were pretty small back in the
3:20
day. So I worked on pretty much
3:22
everything, but I focused mostly on the real time pieces.
3:25
So for example, streaming, as you mentioned, a lot of
3:27
challenges around that. My role now has changed a little
3:30
bit. So I am more in like a
3:32
managerial role right now. So I managed 3 different teams
3:36
that focus on like streaming data, batch data, online
3:38
inference, offline inference, etcetera.
3:41
So kind of like the core data flows and infrastructure that
3:43
Tecton provides as part of its like core product.
3:47
So in in short my days, a lot of meetings, but yeah.
3:53
We should probably stay for everyone because we can't take
3:56
it for granted that everyone knows what Tecton does.
4:00
Yeah, absolutely. Yeah.
4:01
They're one of the original OGS feature platforms.
4:06
So a lot of dealing with your features, transforming your
4:09
data, making sure that you can do use cases like fraud
4:14
detection, recommender systems, loan scoring.
4:18
I mean, are there other stuff that you see people using Tecton
4:22
a ton for? Also for like insurance claim
4:24
processing is also like a big use case for us.
4:28
Recommenders are like in many different shapes and form and
4:30
then broadly fraud and risk applications I would say is the
4:33
biggest like use case for us. Yeah, yeah.
4:36
That those use cases that you just need really fast responses.
4:41
Yeah, yeah. So that's why we wanted to talk
4:44
about streaming, man, that leads us right into the whole
4:47
streaming ecosystem, like Breakdown.
4:51
What is it? What does it look like right
4:53
now? What are some things that have
4:56
changed since I was on that call with you like a year and a half
4:59
ago? I can't even remember.
5:00
Yep, yeah, absolutely. So I think like when we look at
5:04
customers, most of them have, you know, invested in like a
5:10
streaming solution which can like screen data at low latency.
5:13
So for example, this is like Kafka, Kinesis, etcetera, but
5:19
that's kind of like a data delivery mechanism, but they
5:22
kind of fail when it comes to using that data effectively for
5:27
applications. So a lot of app companies we
5:30
talked to have like data in these Kafka Streams, can use the
5:33
streams, but then they fail in making like machine learning
5:36
models or even like analytical use cases use this data
5:40
effectively and so. Wait, why is that?
5:44
So yeah, I mean, I'm going into it.
5:46
So I think like fundamentally the the ecosystem is extremely
5:50
fragmented in terms of like several different tools and
5:53
technologies. And so you have to string
5:56
together a bunch of these things, hire exports for each of
5:59
these things. And then once you can even do
6:01
these things, it's not a one time cost to actually build it
6:03
out. But these systems need like
6:05
resiliency and reliability. And so it's like an ongoing
6:09
effort to basically maintain these systems.
6:11
So the simplest solution we see is people have like a Kafka
6:14
stream and they would have a stream processor, which can
6:17
either be like a Spark or a Flink, and then they basically
6:20
connect that to then storage like could be key value stores,
6:23
could be offline storage like Iceberg, and then build a
6:27
serving layer on top of it for applications to consume this
6:29
data. And each of these different
6:32
steps actually requires like different skill sets.
6:34
So building a team to actually do all of these things is very
6:37
difficult. And the tools inherently that
6:39
are available from the ecosystem are also like fairly difficult
6:44
to actually use, especially if you're trying to like go from
6:47
zero to 1. I think like if you have a very
6:49
large team and you're running like Flink at let's say, you
6:53
know, Facebook or meta scale, you know, it's maybe somewhat
6:56
manageable. But for like smaller companies
7:00
and teams that just want to still leverage the real time
7:03
data the ecosystem doesn't really have like these simpler
7:06
tools available for people to use.
7:09
Yeah, and I know that Dynamo DB plays a huge part in this.
7:14
Where does it plug in? Is it on that serving layer?
7:17
Yeah. So Dynamo is basically storing
7:19
records coming in from a stream processor and then making it
7:22
available for high throughput, low latency serving.
7:25
So essentially something needs to write to Dynamo and something
7:28
needs to read from Dynamo. And that writer and reader is
7:32
something that each company kind of built it their own way.
7:35
And even maintaining that is a challenge because you need to
7:39
think through like what is the schema which favors like read
7:42
latency versus like write latency?
7:44
And which one is more important to you and how do you evolve
7:47
that over time? And all these are like
7:48
challenges that people who are building all these pipelines
7:51
have to think through very deeply when they're building
7:55
these systems. Yeah, because you're trying to
7:58
just keep things super fast, as we said, and make sure that your
8:02
use cases like the loan scoring or the fraud detection or
8:06
recommender system, whatever, is just blazing fast.
8:10
And so you need to think through wherever you can shave off
8:14
milliseconds. Yeah.
8:16
And there's this also additional dimension of like cost because
8:20
so I, I always think of like these are two sides of the same
8:23
coin, which is how fresh do you want your data and how much are
8:27
you willing to pay for it, Right.
8:29
And ideally, like, they should be like In Sync with each other,
8:32
meaning that like, if I want data at like millisecond level,
8:36
like granularity, I'm willing to pay more money for it.
8:38
And if I am OK with, you know, an hour worth of like freshness
8:42
and I'm willing to pay less for it, right?
8:44
And I really want a system that has the right levels of like
8:47
knobs build in which can give you this like trade off and you
8:51
can like actually play with it. For a lot of like systems that
8:55
we see in the real world, like this is not present, meaning
8:57
that like they either have a system that's built for like
9:00
very low latency and very, very and there's a very high cost of
9:03
like the system or it's like the complete, you know, on the other
9:08
side where the cost is relatively low, but the but the
9:12
system cannot afford like, you know, freshness.
9:14
That's pretty low that that's required by applications.
9:17
And most companies have like a very like a very varied set of
9:23
use cases. Some need very low latency, some
9:25
are OK with slightly higher like latencies, etcetera.
9:28
And so I think having a system that can manage the cost for
9:32
that freshness is also very important as opposed to like,
9:35
you know, building it like the simplest analogies.
9:37
Like, you know, people are not just looking for like a Ferrari
9:41
because it's, it's just like fast, but it's also very
9:43
expensive. And then like what people what
9:46
really want is that like they want a system that's like fast
9:49
on some days when they need it to be, but can also like act as
9:53
like a Ford SUV when you want to go to the grocery store and buy
9:56
some milk and eggs. Yeah, if you're with your family
9:59
and that makes total sense because you have different use
10:02
cases that have different needs and different freshness and
10:06
different variability of how much you're willing to pay for
10:09
that. So for each instance of these,
10:14
you want to be able to configure it to those different
10:19
constraints that you have. Yep, absolutely.
10:22
And the other thing that I was thinking about because you were
10:26
talking about how folks are using Flink or Spark, I think I
10:32
remember Duck DB being plugged in here in some ways.
10:37
Can you talk to me about that? Yeah.
10:39
So on the batch side of things, we do provide a solution called
10:44
Rift, which uses Duck DB as an alternative for some of the
10:48
batch pipelines. OK.
10:49
So that's kind of like the equivalent of like Spark batch,
10:52
right. But on the streaming side of it,
10:55
we we don't use a dot DV there. We have our own like stream
10:58
engine there essentially that we use.
11:00
Yeah, that makes sense. OK, so basically that is the
11:05
scene you have set is that there's a bunch of disparate
11:10
pieces to the puzzle. Mostly everyone builds one size
11:16
for all of their use cases in their organizations, and they
11:21
don't think about how to service each individual use case
11:26
through. And then you've got reliability
11:31
or maintenance of these pipelines and of these platforms
11:36
that come into play. How do you see teams dealing
11:40
with that maintenance? Yeah.
11:43
I mean, honestly, it's like very challenging because a lot of the
11:46
times the teams that are building these are sometimes
11:48
like machine learning engineers or people who are more focused
11:51
on like the data layer of it. But the production experience of
11:55
like maintaining the system typically falls to a different
11:57
team. And this, this silo between the
12:00
teams often is a huge challenge for a lot of our like customers
12:04
that we see. And on top of that, like it's
12:09
users care about like end to end reliability, right?
12:12
As opposed to like whether my Flink went down or my worker
12:15
went down, etcetera. And they were looking for
12:17
solutions that can offer end to end reliability across the
12:20
entire pipeline. And that's something that's just
12:22
very, very difficult to build because a lot of the on call
12:25
production team things of like one service in the individuality
12:29
of it and a lot of the machine learning teams care about like
12:32
the end to end reliability of the entire pipeline.
12:34
And this mismatch often is like a huge concern for a lot of our
12:38
like customers that we see. So wait, this means basically
12:42
the SRE who's getting pinged at 3:00 AM is getting pinged for
12:47
one particular piece like the Flink went down.
12:50
Exactly, Yeah. The machine learning engineer is
12:54
thinking about, well, I've got Kafka and I've got my flank and
12:58
and how are they looking at it? That's not just like that one
13:01
service went down. Correct.
13:02
Like for example, if let's say like your flank went down in or
13:06
like a Spark Structure streaming engine went down, then probably
13:09
no messages are being backed up in like Kafka.
13:12
And now the machine learning applications are seeing like
13:15
high freshness basically like they're not processing any more
13:18
real time data. And so the repercussions of each
13:21
of these are like so far and beyond that, a lot of the, if
13:26
you're building a solution that requires stringing together a
13:28
bunch of these disparate tools, then it's like a never ending
13:32
game of like trying to build end to end reliability because it's
13:35
just very difficult to do that for like individual components.
13:38
And then you start to see that's where you can make the case for
13:42
a lot of money being burnt on fire.
13:47
Yeah. I mean, we constantly come
13:49
across customers who are spending like, you know, 10s of
13:52
thousands, if not hundreds of thousands on like just simple
13:55
like streaming pipelines. And I'm just talking about the
13:58
infrastructure cost of it. There's the other cost of it,
14:00
which is like the SRE cost of it and the operational cost of it
14:04
on the day-to-day basis. But just like pure AWS infra
14:06
bill for all of these pipelines is like pretty high and like
14:11
optimizing it is very, very difficult given their stack.
14:15
Have you thought about different best practices on how to
14:18
optimize that besides like, OK, just plug in Tecton here, right?
14:22
I imagine that you have seen teams that are doing this well,
14:27
and what does that look like? Yeah.
14:29
I think like to be honest, a lot of like newer age technologies
14:32
that are coming in are actually aimed at bringing down like the
14:35
operational and one time cost of setting these things up.
14:39
I do think it requires a departure from just like a
14:43
mental thinking model of like I want to use what Google and
14:47
Facebook are using, which typically could be Flink
14:50
etcetera. And I think it requires you to
14:51
be open to some of the newer technologies that are coming in.
14:54
So I'll give you one example where we're seeing a lot of like
14:57
streaming solutions not use like local disks attached to nodes,
15:02
but use S3 as like a storage service and, and stream all that
15:07
data through S3 essentially and kind of build an object storage.
15:11
So one good example of is, is, is like warp Stream, for
15:14
example, which kind of like is, is that a Kafka on S3?
15:19
Basically they got recently acquired by Confluent.
15:22
And we're seeing a lot of like these newer age, like real time
15:26
operational systems that are truly building for like the
15:29
cloud and leveraging object storage.
15:31
And that's, I think that's a huge way that huge level that
15:34
customers can pull and see like where are we using cloud
15:37
resources effectively in a stack or not?
15:40
And is that just because it's much cheaper to do it through
15:45
S3? It is cheaper but also like the
15:49
IT gives you elasticity. So I think like, the biggest way
15:51
you can think of it is you can start to scale compute and
15:54
storage independently as opposed to, for example, if you are
15:58
using like disks that are attached to your server, then
16:02
every time you add a new server, you're also adding more disks to
16:04
it. And so it's very difficult to
16:06
decouple the two, whereas some applications may require more
16:09
compute. And in those cases, you're still
16:10
ending up paying for like more storage.
16:12
And I think the cloud gives you this elasticity of like being
16:15
able to scale different like layers independently.
16:18
And it leads to like not only reduced cost, but also just like
16:21
reduced operational overhead. Yeah, that is cool.
16:26
What other tricks? Because I hadn't heard about
16:29
these guys that just got acquired from consulate.
16:31
What was their name? Warp Stream.
16:33
Warp stream. Yeah, fascinating.
16:36
Yeah, I think that there are other like in other interesting
16:40
tricks that like we see some of our customers use, for example,
16:43
like a a big cost of like streaming workload is typically
16:48
checkpointing, which is like how frequently are you writing to
16:51
storage in case you're streaming application, in case you're
16:54
streaming pipeline failed, you can recover from the latest
16:56
checkpoint. And you know, typically the more
17:00
you are checkpointing, the more you're paying for this
17:04
checkpoint it cost. But the benefit you can get is
17:05
that if you let's say it went down, you can you can pick back
17:09
up from the most recent checkpoint, which maybe just be
17:11
a few seconds ago, right, Because you're checkpointing so
17:14
frequently. And I think for a lot of our
17:16
customers at the default checkpointing settings for a lot
17:18
of these streaming applications is just like very, very high.
17:21
And they're often surprised to see just like the cost when the
17:24
bill comes up. And for a lot of our customers,
17:27
like we can just look at like it's just like one of those
17:29
things where when most customers ask us like, why is my streaming
17:34
though so high for my existing stack?
17:36
Like the first place we would look is like how frequently are
17:39
you checkpointing to disk, etcetera.
17:40
And I think like a lot of our customers are just surprised by
17:44
like, OK, I didn't even know about this.
17:48
And what is it that the majority of the time you don't need to be
17:52
checkpointing as much? So I think like it's, I think
17:55
checkpointing is usually like very important and it should be
17:58
done. It's a question of like, do you
17:59
want to do it every second, every 10 seconds, every 30
18:02
seconds, every minute, every hour?
18:04
And just the default settings for a lot of the streaming
18:06
workloads is like very aggressive checkpointing.
18:10
And if even if you can tune it from the one second to let's say
18:14
10 seconds, and if your application can allow that level
18:17
of like recovery point objective, then you can save
18:21
like pretty significant costs for a lot of our users.
18:24
Yeah, that feels like a huge difference from one second to 10
18:28
seconds. Yeah.
18:29
I mean, as I said, like it's a level of like how much cost are
18:31
you willing to pay for how much freshness and these knobs are
18:35
present in some systems. But you need to know of like
18:37
what these knobs are and how to tune it effectively.
18:40
Otherwise you will. You might just end up spending a
18:42
lot of cost for workloads where you don't require that level of
18:46
like freshness. Well, it goes back to your whole
18:49
thing at the beginning where you were saying when you have to
18:52
patch together so many disparate tools, you have to have people
18:56
that understand and are experts in all these tools so that they
19:01
know, oh, it's possible for us to just checkpoint every 10
19:05
seconds. Yeah.
19:06
And they also need a reason about the end to end pipeline
19:08
and understand what this one thing will lead to the end to
19:11
end pipeline because most application teams are concerned
19:14
about that as opposed to like this other objective for
19:17
example. And it's just a very hard
19:19
problem to solve. I know you talked about the way
19:24
that you look at the shape of the data as being a lever that
19:27
you can pull. And so it is this picture that
19:31
you're painting in my mind is just like, there's so many
19:34
stakeholders involved. And that just makes it way more
19:40
complex because at any time you have to throw more bodies at a
19:43
problem, it's going to increase the complexity of the, the, the
19:48
solution in this case. But then you've got like like.
19:54
I imagine the data scientist isn't thinking about the schema
19:58
as much for how to make it just fully optimized for speed.
20:06
Yep, exactly. Like I'll give you, I'll give
20:07
one example where we came across a customer who was like storing
20:12
protobuff messages inside the message stream, right?
20:15
And well, it made sense because the team that was like putting
20:18
data inside the stream was dealing with protobuffs and they
20:22
were like doing it that way. However, like on the consumer
20:25
and we were having to deserialize all these protobuffs
20:28
into messages that we can like bars and ingest in our like
20:31
store ingest in the storage and serving solution.
20:34
And when you spoke to the data scientist, like they just did
20:38
not want to reason about like why do you want us to approve us
20:41
in the 1st place? And it was just like an
20:42
abstraction level of thinking where they are, which they are
20:45
like not used to. They are used to operating a
20:47
little bit like higher level of abstraction, which like
20:49
completely makes sense. But they are either forced to
20:53
like reason about like much lower levels of abstraction or
20:57
they just have to deal with like the performance hits that they
21:00
are like getting and it just seems like they have to pay a
21:03
tax one way or the other with a lot of the existing systems of
21:06
BSc. Yeah, what a great point,
21:10
because they're just going to do whatever is the easiest.
21:13
It's like water flows downstream and is trying to minimize the
21:19
friction that it encounters. And so if that is the only
21:24
solution that they know works, and they're probably going to
21:27
take that upon them. And, and it makes me think also
21:30
about how important it is on the organizational side for these
21:35
different stakeholders to be talking to each other and to
21:38
understand each other's goals and know that, all right, the
21:42
data engineer is setting this up for me.
21:45
So what, like, talk to me more about what the data engineer can
21:50
do, right? Like if I'm a data scientist, I
21:53
am diving into data engineering, which probably isn't my strong
21:57
suit, but it's going to help me understand what these folks are
22:02
able, capable of of doing, or it's going to help me understand
22:09
what levers these folks have to pull.
22:12
Yep, exactly. And like, for example, we see
22:14
cases where there are some people who don't care about what
22:17
is inside the screen as long as they treat each of those
22:20
messages like a black box basically.
22:22
And their job is just to make sure that these black boxes are
22:25
flowing through the system. And there is some other team
22:27
that like cares about what's inside the black box and what
22:30
how that affects like the end to end system.
22:33
And we see like this again, again, again about like how it's
22:38
kind of like this is not my problem, it's your problem, It's
22:41
not my problem, it's your problem.
22:42
But at the end of the day, it's like hurting the application
22:44
that's actually needing this data.
22:46
Yeah, and that's not even dimension like going upstream
22:48
where the data is created. Yep, exactly.
22:51
Trying to figure out like how if things change way upstream then
22:56
it it's a domino effect all the way down.
22:58
Correct. Yeah.
22:59
And we see that like we see that when like we'll see like the
23:03
application team basically saying that like, hey, we're
23:06
seeing like events that are not fresh.
23:09
It's, we were used to seeing events about like a second ago
23:12
and now we're seeing events that are like an hour ago, right?
23:15
And debugging that just takes like so much effort sometimes
23:18
for a lot of our customers. Which is that is it the
23:20
application that's processing the screen that's having delays?
23:23
Or is it the application that's putting messages inside the
23:26
screen that's having delays? Is it the consumer application
23:29
that's having delays? And so it's kind of like this,
23:32
you know, highway where like, even if like one place is like
23:35
stuck, it can affect like backstream the entire thing.
23:37
And figuring where the exact problem is can often be
23:40
challenging. Yeah.
23:43
And you have to go and then be this detective in a way.
23:48
And the more tools that you're using, the more headache.
23:52
Yeah, it's just, it just gets the problem just gets like
23:55
exponentially difficult because because you now need to check
23:57
like all the different tool interactions.
24:00
And now suddenly if you're using 2 tools or four tools, that's
24:03
now like 8 or 16 different interaction points you need to
24:05
check. I feel for all those folks out
24:08
there that are dealing with this on a day-to-day basis because it
24:12
sounds painful. What else you got for me?
24:16
What other tips and tricks and and just like streaming
24:20
ecosystem stuff. I mean, I love the fact that I
24:24
am very green in this field and you live in it.
24:32
And so just like simple stuff that you're telling me is
24:35
blowing my mind. Yeah.
24:37
Like another thing that we see is like how much data, how much
24:40
data retention is there in your stream?
24:42
Like for example, streams are meant to typically contain like
24:45
recent data, but recent itself is like subjective.
24:48
Like you might keep a day's worth of data in your stream, or
24:51
you might keep like 7 days worth of data in your stream, or you
24:54
might keep like 30 days worth of data in your stream, etcetera.
24:57
And you can just keep extending that.
24:59
And again, like you need to think effectively about like how
25:03
much data you want. So let me back up a little bit.
25:07
So a lot of our customers have to like stitch together
25:12
streaming as well as batch data because what they want insights
25:15
on is like, yes, they want insights on what happened in the
25:18
last five minutes, which may be present just by looking at the
25:21
stream data. But they also want to know
25:23
insights about what happened in the last like 7 days or 10 days,
25:26
right? Which may include looking at,
25:28
OK, I'll look at the stream for the last like maybe a day.
25:30
And then I look at my batch data because my batch has data for
25:34
like the last six days, right? And so I think one effective
25:37
thing is like deciding what's the cut off point between what's
25:41
stored in batch and what's stored in stream, right?
25:43
And so one way to lower cost could be that like I'm only
25:46
going to keep like the last two days worth of data in stream.
25:50
And then anything beyond that up until like history, it's all
25:53
going to be in my batch data, or you could say that like another
25:57
deal with the batch data and my application just needs like
26:01
statistics up to 30 days. And I'm going to keep all that
26:03
within the stream itself. And each of these decisions that
26:06
you make affect like cost in like pretty substantial amount
26:09
of like ways. So for example, the one easy way
26:13
we like help a lot of our customers is if you had a system
26:16
that can effectively handle batch and streaming data and
26:21
make it available to the application where the
26:23
application doesn't care about whether it's coming from batch
26:25
or stream, Then you can like really start to arbitrage based
26:29
on what you want in the stream and what you want in the batch,
26:31
right? And you can start putting lesser
26:34
and lesser data in your stream, like only the recent relevant
26:37
data and push more stuff to batch.
26:40
But it requires you to have like a very advanced, I would say
26:43
like data engineering pipeline, which can really be able to
26:47
combine this batch and stream. They are coming in effectively.
26:51
So for example, we can operate with like even, you know, a few
26:56
hours of data in the stream as long as, as long as data beyond
26:59
that is coming from the batch, which can, you know, really
27:02
reduce cost for users if they are used to storing like 30
27:06
days, seven days, 10 days worth of data in the stream.
27:09
Dude, so do you see folks creating two different systems
27:17
for the batch and the streaming? Yeah.
27:19
So, so it's actually pretty like so interesting.
27:22
So what we see is that like if let's say a machine learning
27:25
engineer, data scientist said, hey, I need like five data
27:29
points. One is the recent transaction
27:31
history for the last five minutes, one is for the last
27:33
hour, one is for the last 10 days.
27:36
And 1 is like a lifetime sum of it essentially like the total,
27:39
right? The first 2 are like more recent
27:42
and like the engineer to build like a streaming pipeline for
27:46
that. And then the other two are like
27:48
more historical like 7 day or lifetime and they would build a
27:51
separate batch pipeline for that, right.
27:54
And then these completely are like disparate pipelines and the
27:59
the data scientist would have to write transformations etcetera
28:02
separately for each of these pipelines.
28:04
And then they would then like use that in the machine learning
28:07
model and separately train and like serve based on that.
28:12
Having these separate pipelines is often a huge challenge for
28:16
like customers in terms of like duplication of efforts,
28:19
infrastructure cost as well as like operational maintenance of
28:23
each of these pipelines. There we go with that
28:26
operational maintenance. Again, that is something not to
28:29
be underestimated. Yeah.
28:30
I mean, because this is because the cost doesn't just stop at
28:33
like the day you launch this in production.
28:35
Like that's kind of like the beginning of it actually, where
28:37
most people actually feel if you spend like let's say three to
28:40
six months and you put this in production, your work is done.
28:43
But like in the in reality, like that's kind of like the starting
28:46
of the work. And then beyond that, to
28:48
actually run this in production is actually the hardest part.
28:51
Yeah, that's where you get those bills.
28:53
Coming in and all of a sudden. You're like, what the hell are
28:55
we spending all this money on? Yep, exactly.
28:57
Yeah. The other thing that we're also
29:01
seeing is a lot of the streaming ecosystem players are also
29:04
maturing quite a bit and they're offering more and more services
29:07
within them. Like a good example of this is
29:10
if you today had a pipeline that was, you know, taking stream
29:14
data and dumping it into S3 or Iceberg, etcetera, actually like
29:20
Confluent is offering like managed services for that.
29:22
So you can just have like stream and just say dump it to like
29:26
Iceberg and they have this new service called like table flow.
29:29
And so we're seeing that like a lot of this like fragmentation
29:32
in the industry where maybe there was some vendor for like
29:36
just producing stream messages and like serving them out.
29:38
There was another vendor for like putting them in like object
29:42
storage. Maybe there was another vendor
29:43
for putting them in online storage.
29:46
We're seeing like some consolidation within that, which
29:48
is effectively raising again, like the abstraction on which
29:51
you need to think of. So for example, as I mentioned,
29:54
like Confront is offering now a fully managed service to
29:57
basically take your stream and bump it into like iceberg.
30:00
And you don't even have to think about like how data is going
30:03
from your screen to iceberg and what would happen there,
30:05
etcetera. I do think like these managed
30:08
services are expensive when you think about like the cost, like
30:12
the one time cost of it. But if you truly are able to
30:15
factor in the operational cost of it, I do think like these
30:18
services make sense. And we're increasingly seeing
30:21
this across like for example, even within Deer Bricks, even
30:24
within Confluent and even like Snowflake, for example, is
30:28
offering some of these services. So for example, if you have a
30:31
Kafka stream and if you want the records dumped into a Snowflake
30:34
table, they're offering like minus services for that.
30:37
And then obviously they have the entire store Snowflake stack
30:40
beyond that. So we're seeing like effectively
30:42
a lot of these vendors provide more end to end solutions and
30:47
take on the complexity of like stringing these tools together.
30:51
And I do think like if, if you're like a service, if you're
30:54
operating a service, if you're operating a company where you
30:57
are dealing with a lot of these disparate tools, then using some
31:00
of these managed services by different vendors could be like
31:03
quite useful for you. We see that all the time in the
31:07
ML OPS community Slack. So many folks will come through
31:10
and ask the questions and it's, it's almost like, yeah.
31:15
And, and the question is phrased in some shape or form of hey,
31:20
I'm using XYZ managed service, be it Sage Maker, be it data
31:23
breaks, be it whatever. But it's so expensive.
31:27
And I'm thinking about what we can do to make it cheaper.
31:31
And inevitably someone will chime in on the thread and say,
31:36
OK, you can use this open source alternative, or you can try and
31:42
roll your own in this way shape or form.
31:45
But don't forget the cost of the humans that are going to be
31:49
maintaining that. And so you think it's cheaper.
31:53
Maybe on paper your cloud cost go down, but then your manager
31:58
or your manager's manager is looking at it and saying we just
32:01
spent less on the cloud cost, but we're spending way more in
32:04
headcount. Yeah.
32:06
And that's truly one thing that we see.
32:08
The other thing we see is that a lot of companies want to focus
32:12
on like product velocity, right. And so for example, if like even
32:15
if you set all this thing up, but you are anticipating that
32:19
like I need to build you know, three times or four times as
32:23
many features and models etcetera and large time in
32:26
production. Is this going to be a bottle
32:28
like in terms of me launching these new models and like
32:32
pipelines in production, which is really a problem because then
32:36
it delays your ability to go to market with these products
32:39
quickly. And so far we see this across
32:43
the industry as well, where sometimes people just come to us
32:46
because they feel like, hey, I know this, I can run this on my
32:50
own. I can even hire people and I can
32:51
do all this stuff. But as a business, I just don't
32:54
want to focus on things about these things.
32:56
And I just want to focus on product velocity and being able
32:59
to launch the best products out there to to the end consumers
33:03
and not have to worry about whether this product will get
33:07
delayed because some other team will not start this on their
33:10
road map and will require more resources to maintain these
33:13
things. I saw a blog post the other day
33:16
from fly dot IO talking about how they created GPU offerings
33:23
because they're, you know, like this cloud.
33:26
And they basically said we were wrong about the whole GPU
33:29
offerings and it was a lot of hard work and we aren't going to
33:33
continue with it. They're not discontinuing it,
33:35
but they're not really advancing forward.
33:37
And there was this nugget in that blog post that I thought
33:42
was the coolest phrase I've heard in a while.
33:45
It stuck with me. It said a very useful way to
33:50
look at a startup is that it's a race to learn stuff.
33:57
Yeah, that makes a lot of sense, Yeah.
33:58
It's just a race to fail and fail and then learn what works.
34:02
And so that's exactly what you're saying, Like it echoes
34:05
this product velocity. How fast can we learn?
34:08
Yep, if this product works or it doesn't.
34:11
And yeah, exactly. And you want to remove as many
34:13
bottlenecks, that's like slowing down your learning curve as much
34:16
as possible. And like probably the only like
34:19
bottleneck should be how quickly are you getting feedback from
34:22
customers. But beyond that, you don't want
34:24
to slow down your process of delivering value to customers.
34:28
Yeah, because that's in your hands.
34:30
Exactly, Yeah. Yeah, yeah, The other stuff
34:32
maybe it's a little bit more difficult to get that feedback
34:37
or that that loop coming back. So it makes sense, man.
34:41
So all right, cool. Well, that is a, that's a solid
34:45
one. I mean, the managed service one
34:47
too, just how different products are going into different spaces
34:56
always fascinates me because it feels like everybody's trying to
35:00
become the one platform like the Databricks.
35:04
And so in five years, all of a sudden you can do everything
35:08
that you can do on Databricks on conflict.
35:10
Now I I don't know if you have thoughts on that one.
35:13
Yeah. I mean, I think a good example
35:15
of this, for example, is like Iceberg.
35:17
And I think like Iceberg is on track to become like, I think
35:21
like what GitHub is to code. I think like I will be to data
35:24
basically for any organization essentially like it'll be
35:26
decentral place in which everything is stored.
35:29
And like all the vendors, it could it be data bricks, could
35:32
be Snowflake, could be confluent, essentially they have
35:34
to read and write from that, but then everything is like stored
35:37
there. And we are seeing more and more
35:39
of these services becoming agnostic to sort of like these,
35:44
these implementation stuff and focus more on like how can they
35:48
be more like compatible with each other, especially when, for
35:51
example, storage is not moved to like a common solution.
35:56
And I think we will see more of these things where you could
35:59
more easily transition from like 1 vendor to the other because
36:04
you don't have to think about how the data is going to be
36:07
migrated from 1 vendor to the other, which used to be like the
36:09
biggest challenge. Like if you're storing data in
36:12
one vendor, just migrating that to the other vendor without
36:15
losing it in a secure way. I think there's going to be a
36:18
thing of the past. And if all of this data ends up
36:22
being stored in like iceberg, etcetera, or like an open
36:25
standard formats on the cloud, then switching from 1 compute
36:30
vendor to the other is, I think going to be significantly
36:33
easier. Wow.
36:36
And maybe it's not even that. OK, we're on data breaks now.
36:40
We're going to migrate to Snowflake.
36:41
It's like for this job we use Snowflake and for this.
36:45
Exactly. Yeah, because at the end of the
36:47
day, it could just be that if you took a 10,000 foot overview,
36:50
could be like there is some compute job that spins up, reads
36:53
data from Iceberg and writes it back to iceberg.
36:56
That job essentially could be a data brick spark job.
36:59
It could be a snowflake query. It could be like a big query
37:03
something it could. Be their duck.
37:05
Etcetera. And I think we're already
37:07
science starting to see the early signs of this where a lot
37:10
of companies are now mandating that we want all of our storage
37:13
to be in iceberg because we don't want to be in a vendor
37:16
lock in with like any particular vendor among data storage.
37:20
Man, that's wild to think like the analogy of Iceberg is to
37:28
data what GitHub is to code. Yeah.
37:31
I mean, effectively, if you go to any organization today, like
37:34
by and large, mostly there might be several teams, like several
37:38
different engineering teams, could be UI teams, front end
37:40
teams, back end teams, etcetera. They're probably all storing
37:44
their code on like GitHub and they're all searchable and they
37:47
have like common mechanisms. They may have like different
37:49
build processes etcetera as well like that building code,
37:52
etcetera. But they're all effectively
37:54
reading from GitHub and they're like making changes to the repo
37:57
there. And I think we're we're I think
37:59
like code was maybe ahead of it's time and data is maybe
38:02
catching up now. But I do imagine a world where
38:05
like we don't have to think about like is this data stored
38:09
in one vendor on? Is this data stored in a
38:12
different vendor? The data is just stored in
38:13
Iceberg and you can read write from anywhere basically as long
38:17
as you have a standard data format and standard catalog
38:19
essentially. Yeah, well, that's the dream.
38:22
And then what it's like the data would be specialized it for its
38:29
specialized use cases like oh, I've got this generative AI use
38:33
case where I've got this fraud detection use case and it's
38:37
doing like you said, it's spinning up some kind of a
38:39
compute and. Yeah.
38:42
I mean we're already seeing this for like structured data.
38:44
I think it's kind of centralizing on this.
38:47
I think unstructured data and different modalities of data
38:50
like video etcetera. We still have some room to go
38:53
there. But I'm fairly certain for like
38:56
structured data as well as like unstructured embeddings data, we
39:01
are very, not very far from a world where we will see a lot of
39:05
organizations to store everything in iceberg.
39:08
Damn, yeah, that's very cool to think about and what that
39:14
unlocks. Yeah.
39:15
And I think like, for example, like the Databricks acquisition
39:18
of like Tabular when Databricks was themselves actually building
39:21
Delta, which was a different format than like Iceberg.
39:25
I think it's a great indication that like even probably the
39:29
vendors themselves are seeing where they need to head to and
39:32
where the bug is going. The first thing that comes to my
39:36
mind is this is happening with no like formal committees, no
39:42
standardizing bodies. It's just happening because
39:47
people see that this is the way forward.
39:50
Yeah. I mean, like a lot of the
39:52
iceberg like community came from like several different
39:55
companies. I think they're like maybe 10 to
39:56
20 companies that are operating, but I think they're all
40:00
realizing that we have to work together to achieve this
40:03
solution or else we're good end up in a world where it's
40:08
possible that like 90% of the world is like using iceberg, but
40:11
we have our own storage format and they don't want to be in
40:14
that world like where they're like on the silo somewhere else.
40:16
And so when, when there's enough momentum in this, this space,
40:21
which I do think there is enough momentum in the space, a lot of
40:24
people want to join the momentum as opposed to like, you know.
40:28
Stay away from it or like run against the tide.
40:31
Now let's go back to something that you said earlier, which is
40:34
folks are understanding that they are not necessarily in a
40:41
position that they need to be like Google or be like Facebook
40:47
and they don't need to be built for this gigantic amount of
40:51
scale. Talk to me more about that.
40:53
Yeah. I think like the I think like
40:55
the biggest example that's in front of our like faces is dot
40:59
DB. And I think like in a world
41:01
where, you know, let's say Google comes up with like big
41:05
query and like snow, there's like snowflake that has data
41:09
breaks and there's Presto spark. A lot of these like very large
41:12
scale batch data processing solutions.
41:15
They typically came out from like very large companies
41:18
because they needed it and they were kind of shoved down
41:21
everybody else's throat that this is basically what you have
41:23
to use, right? Even if I don't have that much
41:26
data or even if I don't have that much scale, I still need to
41:29
use the system and deal with the complexity of like using the
41:31
system. It's like, you know, if I wanted
41:33
to go from like San Francisco to New York instead of taking
41:37
commercial flight, I need to fly like a fighter jet and sort of
41:40
like take deal with the complex. It's like just not something
41:43
that's like near, but you know, some people may need it.
41:45
And so that's why now I have to use it.
41:47
And I think like the recent popularity of like Duck TV is
41:50
like a good example of it, right?
41:51
Where people are saying that like, yes, I mean like these
41:54
Snowflake Big Query solutions make sense for like some kind of
41:57
workloads and not all kinds of workloads.
41:59
And I need to look at like the workloads that I need to support
42:03
and whether they fit better in Duck TV or Snowflake and you
42:07
know, Big Query, etcetera. And we see this as well where a
42:10
lot of data pipelines don't need like very complicated
42:13
distributed multi parallel like batch processing solutions.
42:17
They're actually significantly simple to build on, like simpler
42:20
tools. We've seen that.
42:23
I think we've seen this in the batch world and I think we will
42:28
see this in the streaming world as well where people will kind
42:32
of like look at systems that are significantly simpler than Fling
42:37
or Spark streaming, etcetera and look at like building these
42:41
systems on top of such primitives.
42:44
Streaming. I think it's a little bit early
42:46
where there maybe there is nothing as popular in streaming
42:51
as what dot DB is in the batch world.
42:53
And I think that's a wide gap there that there are some
42:56
companies that are like addressing that, but nothing has
42:58
broken out as as as much as dot DB has in the bad side.
43:02
And I think once we enter that world, I think it'll be
43:05
interesting to see how people are choosing like simpler tools
43:09
like dot, DB, etcetera, versus like more complicated solutions,
43:12
both across batch as well as stream.
43:16
Dot DB is known for its user experience and I think people
43:20
love that. The simplicity of it or just
43:23
like the developer experience too on top of it is magical.
43:28
And why do you think it is that in the streaming world we
43:33
haven't seen that equivalent yet?
43:35
I just think like streaming is slightly more difficult problem
43:38
to solve. Especially I think you need to
43:42
think about like ongoing state much more than you would need to
43:46
think of need to think of in a batch world.
43:48
A batch workloads are typically a little bit more ephemeral,
43:51
like you can spin something up, computer and then throw it away
43:54
because you have the results. Streaming is more of an ongoing
43:57
thing where you need to continuously keep maintaining
43:59
state and that might itself grow over time, etcetera.
44:02
It's just a little more difficult problem to solve.
44:04
But I'm pretty convinced that like, we will see something like
44:07
that show up like in the next 5 to 10 years, just possibly or
44:11
possibly even sooner. And I've heard a lot of folks
44:16
talk about how streaming is a necessity and everything
44:22
eventually will go to streaming. Do you believe that?
44:30
I, I don't believe that, to be honest.
44:32
I think maybe I have a little bit of a contrarian view.
44:35
I don't see a world where people need to always only have
44:40
streaming data and everything within streaming.
44:44
The batch processing systems are getting very, very powerful and
44:47
dealing with like large amounts of like data.
44:50
The streaming systems are very good at getting like the recent,
44:53
recent data, right? I'll give you like a database
44:56
analogy, which I think like may make this slightly simpler,
44:59
right? So for example, we've always had
45:01
like transactional databases like Postgres, my sequel,
45:04
etcetera. And we also had like analytical
45:07
databases, like for example, Snowflake Big Query, right?
45:10
And analytical databases look at like fast historical data very
45:13
effectively. And the transactional databases
45:16
are very good for like changing one draw or changing one data
45:18
point, etcetera. And in the database community, I
45:21
feel like there was this time when people felt that we can
45:25
just have one system do it all like, and they had like this
45:27
system called H tab hybrid transaction analytical
45:31
processing. So they can have one system that
45:32
can do both like they can do transactional as well as like
45:35
analytical. And these systems did come out
45:38
actually. However, it turns out like these
45:40
systems are, they can do a mediocre job at both analytics
45:46
as well as like transactional. And the buyer on the other side
45:49
is looking for a best in class solution for analytical
45:51
workloads. And they're looking for a best
45:53
in class solution for transactional workloads.
45:56
And they're not OK with like a solution that does a mediocre
45:58
job at both, right? And so I feel that even if like
46:02
streaming, even if we get to a world where like everything can
46:05
be done in streaming, I'm fairly certain that like there will be
46:09
parts of it that are like more mediocre in performance and
46:12
overall these experience than for batch.
46:15
And so I, I do, I do think that we will be probably entering a
46:19
world where each of these systems will exist and they will
46:22
just be significantly more powerful doing what they do best
46:26
essentially. Yeah, just leaning in to their
46:30
strong suits. Exactly, Yeah.
46:34
Is there anything else that we didn't touch on that you want to
46:37
talk about? I think the one thing that we're
46:41
seeing interestingly that I think we may see more is bring
46:46
your own cloud. And what I mean by that is a lot
46:48
of times people have this like negative feeling about vendors
46:52
because a lot of vendors want all the data processing to
46:55
happen within their account. And a lot of customers are like
46:59
you're dealing with sensitive data and I don't want to dearly
47:01
my account with the Warp Stream acquisition be seen Confluent
47:07
now offering BYOC solution or bring your own cloud so your
47:11
data can still live within your account.
47:13
Red Panda is doing BYOC and they're doing very well with
47:16
that. We're seeing database all
47:17
already offer like a BYOC solution.
47:20
Snowflake historically has not done that.
47:22
They've kind of been in the world of like all everything
47:25
live within Snowflake and give me all your data and I will
47:27
store and process everything within my account.
47:30
I think screaming solutions, I think as well as batch, I think
47:33
will probably more move towards the BYOC solution.
47:39
And I think that will make for a lot of companies accepting
47:43
vendors even more easy and a little bit less challenging than
47:49
it is right now. So the BYOC is just like the
47:55
vendor goes to wherever you are. Yeah, exactly.
48:00
Like basically like the vendor will deploy their stack in your
48:03
cloud account instead of like asking you to move your data in
48:08
their cloud account essentially. Yeah, that is a trend that makes
48:14
a lot of sense to me. Yeah.
48:16
I think, like historically, vendors have been a little bit
48:17
opposed to that, but I think they're realizing that the
48:20
gravity is where the data is, and it's easier to move their
48:24
compute systems and to move any vendor's data into anybody
48:27
else's account. Exactly.
48:30
It's so hard to get that data to go somewhere else just because
48:33
of the sensitivity of the. Data.
48:36
Yep, exactly. Yeah.