Latest / Agentic Conversations (formally mlops.community) / Accelerating Multimodal AI // Ethan Rosenthal // #242

Transcript

Accelerating Multimodal AI // Ethan Rosenthal // #242 · Agentic Conversations (formally mlops.community)

0:00
My name is Ethan Rosenthal and I'm a member of technical staff
0:04
at Runway ML and I take my coffee black, no sugar drip.
0:11
Welcome back to the ML OPS Community podcast.
0:14
We are back. I am your host, Dimitrios.
0:16
For the next hour, we're going on a bit of a journey with Mr
0:20
Ethan Rosenthal. He is a returning guest.
0:22
And boy, oh boy, he does not disappoint.
0:25
I'm going to give him an award today.
0:27
It is the first of its kind. It is the most buzz word of 2024
0:33
and that is because he introduced what he is calling
0:37
the multimodal feature store. He also talks about the
0:42
differences or or should I say the tension between engineers
0:48
and researchers and giving it researcher the flexibility to go
0:53
off and create and make magic but also making sure that that
0:59
magic can be productionized. He's come U with a few tips and
1:04
tricks for us. I thoroughly enjoyed this
1:07
conversation. I feel like he's got a great
1:10
viewpoint of what MO engineers are going through and especially
1:15
for his use case, working at Runway Mill is a fun one because
1:19
they're doing so much cool stuff with generative AI and those
1:23
foundational models that it puts him in a very unique position
1:29
and he's learned a tonne and he articulates his learning very
1:32
well. O Sir, Ethan Rosenthal gets the
1:36
award today for the most buzzword of 2024, and we're
1:42
going to get into this conversation.
1:44
Ohh yeah. By the way, if you enjoy the
1:47
podcast, you can engage with the app you're listening on in many
1:53
different ways to let that recommender system know.
1:58
Let's get into it. O, you were out square slash
2:10
block and you were talking about working with LM's back in those
2:14
days. And this was very early in the
2:17
LM boom, I think maybe before Chad GBT.
2:20
This was like, it was right after the Stable Diffusion stuff
2:25
got big and so but then before Tetchy, BT and so, yeah, in
2:32
hindsight, it seems like I had amazing foresight by working on
2:36
language models, but I just happened to kind of like fall
2:39
into that work at the time. And how did you fall into that?
2:44
I was working on a different team at Square and like wasn't,
2:48
wasn't as happy on that team. Kind of wanted to do something
2:50
different, wanted to do something a little more AI or
2:54
something like that, you know, do a little more deep learning
2:57
and they're just happened to be a team opening.
2:59
Talked to the team, ended up there.
3:01
I was like, oh, like I don't really know anything about LP,
3:03
but you know, I used to do recommender systems so I know
3:05
what embeddings are. So like this should be fun.
3:08
Same thing. Yeah.
3:09
And then like a year and a half later, Chat TPT comes out.
3:12
That was like, ohh, wow. OK, now, now everybody's
3:14
interested in this. Yep, Yep.
3:17
You're the OG. You're the thought leader in the
3:19
space you've been doing. It since before one year of of
3:23
work. Prior to that, therefore, the
3:25
OJ. Yeah, you're like, I was doing
3:28
this during the AI winter and that's how far back I go.
3:34
But then you moved on. It's funny that you mentioned
3:37
like Stable Diffusion. You moved on to the creators of
3:40
Stable Diffusion to Runway, and now you're doing all kinds of
3:45
fun stuff. Can you breakdown what you're
3:46
doing these days? Yeah, so I in January I, I left
3:50
square and and moved to Runway. Yeah, one of the creators of
3:53
Stable Diffusion and Runway. Like I think most people know of
3:57
Stable Diffusion for generating images, but I think Runway were
4:01
one of the first people all the way back at the start of 2023,
4:05
you know, like 20 years ago in AI years, they, they had one of
4:10
the first kind of generative video models.
4:13
And so that's, I think what Runway is, is most known for.
4:16
And kind of our one of our hottest, hottest products that
4:20
we have is around generating video.
4:22
So you can write some text and generate a video from that.
4:26
You can take an image and then animate that image, you know,
4:31
turning it into video. And then we have a bunch of
4:33
other other tools to do interesting things, interesting
4:37
creative things with video. Yeah, and for those who haven't
4:41
seen the demos, those would always be so much fun for me.
4:45
Because you could see how much thought the product people at
4:50
Runway were putting into the user experience.
4:53
And it's one of those products that for me, it feels like, wow,
4:57
OK, this is actually a good use of AI.
5:00
This is something that is quite helpful for a video editor
5:06
because they can now type into a prompt box or the context box
5:12
and say, OK, this video, but without such hard colours or
5:17
without the street lamps on the street when people are walking
5:21
down, and it will automatically remove those.
5:26
Sometimes I think better than other times, but still the idea
5:30
was there and so that captured the imagination of a lot of
5:32
people. Yeah, I think that's, you know,
5:35
it's kind of hard to thread the needle between like AI hype and
5:39
actual value and stuff like that.
5:40
Yeah. So I think one of the things I
5:42
liked what I was looking for new places to work was that it, it
5:46
seems like, you know, this runways tools are actually
5:49
valuable to all sorts of content creators and people who just
5:55
generally work in like the creative industry.
5:57
And I think it helps that like the companies founded by
6:00
creatives, like the I do, as you mentioned, like the UX is, is a
6:05
part that I really like. You know, it's, it's fine to
6:08
like write prompts all day, but I, I really hope we're all not
6:12
prompting 10 years from now. I think it's, it's more
6:14
interesting when like we can use the full capabilities of our
6:17
computers to, to do things. So, yeah, like one of the older
6:22
tools that Runway built was this, this green screen tool.
6:24
So, you know, instead of if I want to like crop a person out
6:28
of every frame of a video, you know, back in the day, you'd
6:31
have to go in by hand and draw a lasso around them and crop them
6:34
out of every frame. But instead, we have this nice
6:36
tool where you can kind of like draw a mask or like paint a
6:40
brush over the areas that you would like to be cropped out.
6:42
And then, you know, with the quote magic of AI, you can now
6:46
just do this on the whole video and disappears.
6:49
There's no prompting involved. It's just kind of using your
6:52
creative tools like you would expect to.
6:55
Yeah, so wonderful. So basically you were like,
6:58
alright, I spent a lot of time at a company that jumped on the
7:03
crypto hype, changed its name to Block and now I need to find
7:08
something that is not chasing that hype and I think Runway has
7:11
done a great job of that. I will say they are doing some
7:15
cool stuff. And now your day-to-day though,
7:18
you went from working with LM's to are you still working with
7:22
LM's? Is it diffusion models?
7:24
What is your day-to-day look like these days?
7:26
It's kind of a little bit of everything and like, yeah, to,
7:31
to be clear, like I, I do some modelling work, but I also like,
7:35
if I spent all day staring at lost curves, I think I'd be a
7:39
little sad. On the flip side, like if I'm,
7:43
I'm like writing, writing code and just waiting for unit tests
7:46
to run all day, then, then I also get a little sad.
7:48
And so I, I try to sit a little bit in between.
7:51
And so, yeah, we definitely have diffusion models, language
7:55
models, all, all sorts of stuff like that here.
7:57
But I've been spending a lot of time on kind of creating nice
8:03
training data sets and organising our training data and
8:07
things like that in order to, you know, help help the
8:11
researchers, like help to kind of scale what they're doing,
8:14
train on larger data sets, make sure that we have interesting
8:16
diversity in our data sets, and kind of incorporating new and
8:21
interesting inputs in into these models.
8:25
Well, I know how cumbersome that can be.
8:27
Hopefully those researchers appreciate you and.
8:30
Yes, they they're, they're very appreciative, very nice.
8:34
Yeah, excellent. And the last piece that I will
8:37
say about Runway, which is also something, it's an incredible
8:42
feat. It all runs in the browser and
8:46
so it's not like a a that you download to get this AI magic.
8:51
It's all happening in the browser.
8:52
We had Brennan on here who is your colleague, coworker.
8:56
He talked to us a lot about how you are using Kubernetes and in
9:01
the infrastructure of Runway. And so if anyone wants to listen
9:05
to that, I highly encourage it. It was fascinating talk, but now
9:09
I want to talk with you about a little bit of a buzzword bingo
9:13
that you were mentioning before we hit record, and it's all
9:16
about what you're calling like a multimodal feature store.
9:20
Can you break down that idea for me?
9:22
Yeah, like before everybody rolls their eyes, because I know
9:25
that that's a lot of cheesy buzzwords, I guess what that
9:30
means. So let's say we'll start with
9:32
multimodal. Multimodal, honestly, actually
9:35
like with kind of kind of a top of the moment right now, you
9:39
know, you, you look at GPT 4 O that came out what last week,
9:44
two weeks ago, and we see that all sorts of different modes are
9:48
now being incorporated in AI. And So what I mean by mode is
9:52
like we have images, we have videos, we have audio, we have
9:56
text. And I think like historically
9:58
these have all been treated somewhat separately.
10:00
So, you know, when I was back at Square, I worked on language
10:03
models and everything we worked on was just with text.
10:07
We might incorporate kind of like tabular information in the
10:11
language model, but at the end of the day, in order to feed
10:13
that model, you convert all of your information into text and
10:18
then train a model off of that. But now that we have, you know,
10:23
like nowadays, we have all sorts of inputs into our models.
10:26
So you can send an image to ChatGPT and ask it about it.
10:31
We can send video to Google systems.
10:34
You can talk to GPT 40 now. And so I think on the input
10:39
side, we have lots of different modalities that we want to feed
10:43
into the model. But then also on the output
10:46
side, we have different modalities that we want to
10:48
generate and how it is. And so text to speech, we are
10:53
generating speech from that. We can generate text, but then
10:57
all the way to like Stable Diffusion and things like that,
11:00
we can generate images and generate videos.
11:03
And like, I feel like if you had asked me this six months ago
11:07
about like, you know, is multimodal AI important and
11:09
things like that, I definitely would have rolled my eyes.
11:11
I have now joined a company that does this.
11:13
And so I've probably spent like the last couple of months
11:15
drinking the kool-aid, but but that's OK, cool.
11:18
It's tasty. And like, it does seem like this
11:22
is where everything is going. And I think that people do see
11:25
that there are, you know, benefits to learning and like
11:30
learning from and producing different modalities.
11:33
So you start to get some kind of like cross modality learning and
11:36
things like that. But anyway, yeah, so that's
11:39
multimodal, right? But then the other side is is a
11:42
feature store. So I think actually the last
11:45
time I was on this podcast, we talked about feature stores.
11:49
Probably. Yeah, and.
11:50
You wrote you had a blog post about feature stories, right?
11:54
I did back in the day. I had a couple.
11:56
One was about really liking them.
11:59
We had a great one at Square feature Store for tabular data
12:03
and I was like this is fantastic.
12:06
But then I also, I think kind of blog posts saying that feature
12:11
store companies were a little silly and that we should
12:13
probably just replace them with good streaming databases.
12:17
So it's a. Then I've brought on somebody
12:20
from a feature store company. You did, you guys.
12:22
Talk. It was a fun, fun cohost for you
12:24
to have for that conversation. I'm surprised that you agreed to
12:29
coming back on here. And you probably thought I was
12:32
gonna ambush you with like, state stability AI founder
12:36
Ahmad. And you're like, dude, come on,
12:38
surprise me. Though you'll be nice to me this
12:42
time and then I'll come back on 1/3 time and then you'll then
12:44
you'll get me. So, So yeah, I think most people
12:47
like even feature stores babies still a bit of a buzzword, but I
12:51
think people are somewhat familiar with them nowadays for
12:54
tabular learning and things like that.
12:55
We have classification and regression models where we, you
12:58
know, want to compute a whole bunch of features about our data
13:02
and kind of use these for both training and inference in a
13:05
performant manner. And so I think this is hard
13:08
enough for tabular data. You know, if you have features
13:11
like like back when I was at at Square, you might have a feature
13:14
that's like, you know, what percentage of payments have been
13:18
blocked by, you know, from this merchant within the last 60
13:22
seconds? We want to use that information
13:24
to power a fraud model. So, you know, if this merchant
13:27
tends to have a lot of payments that have been blocked recently,
13:30
then perhaps they are they are riskier than others.
13:33
And so kind of aggregating these, these events and things
13:36
like that and turning them into features and serving them for
13:39
inference that that's already difficult.
13:42
And now we get to the world of like multimodal AI and we have
13:48
lots of different modes that we have to deal with to incorporate
13:51
into our feature store. And so for for this, I think
13:55
that like inference is perhaps a bit less important.
13:58
We're not, you know, dealing with like very fast streams of
14:02
events that we have to quickly aggregate.
14:04
Like we might be for like a fraud application or for, you
14:08
know, like DDoS attack applications and things like
14:11
that. But instead we have, let's say,
14:15
let's say we're training a model to generate video.
14:18
We might have a bunch of videos, perhaps unsurprisingly, in our
14:22
data set. And so now our feature store
14:25
needs to 10 videos and like videos are really big and
14:30
they're slow to decode, they're slow to download and things like
14:34
that. And so we might have videos, but
14:37
then we might have also other information about these videos
14:40
that are relevant for training. So as an example, we might want
14:44
to know what is the the width and the height of the video or
14:47
like what is the resolution of the video because that might be
14:50
informative for our model. And so you have kind of this
14:53
high variance between the different columns of your data
14:57
set that you're training on. So one column might contain
15:00
videos and another column might just contain you know how many
15:04
pixels are in the video. And storing all of this data and
15:11
being able to query this data and then being able to philtre
15:16
it and then in particular being able to use this data during
15:21
very large scale training jobs is is totally nontrivial.
15:26
So and there's, it's kind of like an unsolved problem right
15:30
now, I would say. So when you're talking about
15:34
these multimodal feature stores, it's primarily for the training.
15:39
And when we think about feature stores in their past iteration
15:44
or what I imagine a lot of people understand them for these
15:48
days is for serving and specifically like low latency
15:53
online serving, right? When you need that fraud
15:56
detection or you have a recommender system or whatever
15:59
it may be, you want to get something out real quick.
16:03
That's when you're using feature stores.
16:04
But what you're saying is, OK, well, for a multimodal feature
16:08
store, this is for training because we have so much
16:12
unstructured data and structured data that can go together and we
16:17
need to be able to understand both of those.
16:21
Yeah, yeah. Like I think that for the former
16:24
case, I think it's, I think the fast inference is the hard part
16:29
of feature stores for tabular data and things like that.
16:33
But you also need to have the training component as well for
16:36
like let's say unimodal or like regular feature stores.
16:40
And the reason is that you don't want this skew to occur.
16:44
You know, you want to make sure that you're the data that your
16:46
model was trained off of matches the data that is being fed into
16:50
the model very inference as best as possible.
16:52
And so ideally, like in most feature store companies do this,
16:55
they try to kind of maintain this parity between the training
16:59
features and the inference features and the training
17:01
features. You know, those are, they're not
17:03
so hard like to store them. You know, we have formats like
17:07
parquet and things like that nowadays where we can store some
17:10
of these large datasets. But then the hard part is making
17:12
sure that inference matches that, Although I guess it's also
17:15
they also have to kind of like backfill data when you add new
17:18
features and other sorts of things that are that are fairly
17:20
complicated. And the data sets tend to be
17:22
quite large for the training data sets.
17:25
But you're right that for what I'm talking about now, this is
17:28
primarily focused on the on the training side of things.
17:32
And that's, that's also because most of the inputs to these
17:36
generative AI models, most of those are provided during
17:40
inference. They're not kind of like they
17:43
don't, there's not that many inputs.
17:45
You know, like if you think about what are you inputting
17:47
into chat, TBT, maybe some text, maybe like a system components
17:52
of a system prompt and maybe an image or something like that.
17:56
But you're not, it's not like we're dealing with thousands of
17:59
inputs to the model during, during inference.
18:02
And so yeah, the inference side is much easier, but then the
18:05
training side becomes much harder.
18:08
Things like AI feature stores. I really like that difference
18:11
that I hadn't thought about before.
18:14
Where when you are dealing with traditional, quote unquote,
18:18
traditional mill, you have all those different inputs and you
18:22
have a lot of them. And then you also have that low
18:25
latency that you need to be hitting when it comes to LM's or
18:29
just these foundational models that we're dealing with.
18:32
Now you've got the prompt and that's kind of it.
18:35
There's a lot of iterations on how you could do the prompt and
18:38
what you do. Maybe it's with voice and maybe
18:40
you're giving different chain of thought, chain of reason,
18:43
whatever you want, but it's still kind of just the prompt,
18:46
right? And it's not like you're giving
18:49
all of those data points into the model and saying, OK, this
18:54
is it. Is it fraud or not?
18:56
And so that's why I'm a big proponent of saying, yeah, OK,
19:00
you have your LM slash foundational model use cases and
19:05
those are totally different than your traditional use cases that
19:10
you would get from classical mill.
19:12
And so it does, it does feel like that.
19:16
I still think the, I think this is fascinating when it comes to
19:19
the multimodal feature store. Then with the can you just
19:23
breakdown the benefits that you would see from that training
19:28
perspective? Like if you have that, what are
19:32
the benefits? Yeah.
19:33
So I think, yeah, one piece is searchability, right?
19:36
So I think like with tabular data, like lots of people have
19:41
Snowflake or data warehouses like that.
19:43
And if all of your data lives in warehouse, then you can write
19:46
some SQL queries and you can calculate some aggregations of
19:50
your data, you can answer analytics questions, you can
19:53
segment your data and things like that.
19:55
I think that when you're dealing with kind of this unstructured
19:59
multimodal data, it becomes a little bit harder to do that.
20:02
And so like for example if I want to like search over a data
20:08
set of images. For like some semantic quantity
20:11
like a, I don't know, find all images with cats or something
20:14
like that. To do that, like we can do that
20:16
nowadays, right? Like we have to 1st calculate
20:19
embeddings like vector embeddings for all of the
20:21
images. And then maybe we run some sort
20:23
of nearest neighbour search over this.
20:25
But that's, you know, that's a much more complicated, I would
20:28
say than just running a SQL query in your warehouse.
20:31
And so that ends up becoming a component where maybe you want
20:35
to have a vector search that sits on top of your, of your
20:38
feature store where your where your training data lives.
20:42
And you need that to kind of be performed because maybe I want
20:45
to, maybe I want to fine tune my data or maybe I want to find
20:49
certain subsets. Maybe I want to ensure there is
20:52
sufficient diver like semantic diversity in the data that I'm
20:55
training off of and so as a result, if I break that down,
21:02
how do I run vector search? I might need to 1st calculate a
21:05
bunch of embeddings related to the data that I have.
21:09
And like you can think of that as as doing feature engineering,
21:13
like we're taking our data and then we're calculating a feature
21:17
from it. But in this case, the feature
21:19
that I'm generating is an embedding.
21:21
So it necessarily takes a little while to do this, especially on
21:25
a big data set. So like the computation pieces a
21:29
bit more gnarly than running some kind of like streaming
21:32
aggregation like you would with a, a regular feature store.
21:36
And then this also means that my future store needs to support
21:39
storing things like vectors. And then I also need to be able
21:43
to run run vector search over it.
21:44
And so like, that's, so that's an example from the querying and
21:50
curation side. There's also like basically like
21:54
the analogies of just straightforward feature
21:56
engineering. So as an example, like I think
21:59
at the start of 2023, Runway had a paper called like about our
22:03
model Gen 1, which is like a video to video model.
22:06
Take a video and then you can write with some text and maybe
22:10
take an image and kind of convert the video into into a
22:16
style that matches the image that you've provided or convert
22:19
the video based off of the text that you've provided.
22:24
And so in that paper, I think text can be converted into an
22:32
embedding with a clip model I believe is what we used.
22:35
And then the video to video model that under the hood uses
22:40
this model called Midas, which does depth estimation.
22:43
So based on an image or based on a video, you can estimate what
22:46
is the kind of depth, you know, how far away is every pixel from
22:50
the camera. And so this starts to kind of
22:54
get into the fact that a lot of these, you know, image and video
22:57
generation models, they're not just one model.
22:59
It's not like like I think we often think of, you know, cheap
23:02
PT3 and GT4 and things like. That as gigantic model.
23:06
These single models, but instead, you know in this
23:08
example, this video to video model it, it consists of you
23:11
know there's a diffusion model in there there is a like there's
23:15
the clip model for converting both images and text into
23:19
embeddings. There is a depth estimation
23:21
model. And so now if I want to, if I
23:24
want to train this model, there's two ways that I can do
23:28
it. I can take all of those
23:30
different models. I can take my diffusion model
23:32
that I want to train, I can take my CLI model, I can take my
23:35
death estimation model and I can take all of them.
23:37
And I can kind of calculate these features on the fly.
23:41
So as I'm training my diffusion model, you know, I get I get a
23:45
video and on the fly I calculate the depth of every every pixel
23:50
in that video. But alternatively, I could
23:53
precalculate all of this and I could store this information in
23:58
my feature store O I could take all of the all of the texts that
24:05
I might have and I could precalculate CLI embeddings and
24:08
store that in my feature store. And so it's the same idea as
24:13
doing feature engineering on the fly.
24:15
You know, maybe like as a simple example, we often normalise
24:20
features before they get input into our model.
24:23
And so that is, you know, taking a feature and then like
24:25
transforming it in some way. And so, you know, I could do
24:29
that where I I take text and on the fly I converted into an
24:32
embedding or I can precalculate this and store this in my future
24:35
store. And so, but then you can imagine
24:38
maybe I want to use a different model other than clip.
24:41
Maybe I want to use some other model that converts text into
24:45
embeddings. And now I can start to store all
24:51
of this information in my feature store and give all of
24:55
this, give all of these different features to the
24:58
researchers where they can start to choose what they want to use
25:01
when training models. And they can start to run
25:03
experiments based off of all of these different precomputed
25:07
features. So you're giving them new data
25:12
sets in a way. Or or new features built on top
25:16
of the data sets. Yeah, OK.
25:20
And again, this has nothing to do really with speed.
25:26
The pre computing doesn't. It's going to be faster, but
25:30
that's not really why you're doing it.
25:33
Now instead it's, it's more that we, we can now have a canonical
25:37
place where all of this information lives.
25:40
And you know, the same way that like a, a data warehouse becomes
25:45
the canonical place that everybody gets basically solves
25:51
their analytics needs. But we can now have a canonical
25:54
place where we have large training datasets with all sorts
25:57
of features and information against them that can then be
26:01
used by downstream researchers for running all sorts of
26:04
experiments. And do you think that this is
26:09
going to be solved by traditional feature store
26:13
companies or is this something that feels like it's going to be
26:17
like a new database that comes out of nowhere and says, yeah,
26:21
we do vectors. And we also do a little bit of
26:23
because I've seen people doing like semi structured data or
26:28
this I I feel like I've seen some people.
26:31
My friend Cody Coleman, I don't know if you know him, but he is
26:35
doing coactive. And one of their value props is,
26:38
hey, let's be able to search video just as easily as we can
26:43
search a data warehouse. And so I wonder if because of
26:48
the opinions that the feature stores have, the traditional
26:52
feature stores have taken. My guess is that there is not a
26:58
large amount of overlap between the implementations of kind of
27:04
quote traditional feature stores and these more unstructured
27:09
multimodal feature stores. There is some, but as we talked
27:13
about, I think a big part of the traditional feature stories is
27:17
around the inference side of things.
27:20
And that's that's the expensive side of things too.
27:24
And a lot of that is not here and it's not relevant for for
27:28
the like for these kind of multimodal feature stores.
27:32
And the last point that I want to make on this before we move
27:35
on to something else is talking about large scale distributed
27:40
training and how this in your mind can help.
27:44
Yeah. So I think like, you know, when
27:48
you run jobs to train AI models nowadays, especially if you're
27:53
dealing with kind of multimodal AI and things like that, you are
27:58
going to be running your job on many machines.
28:01
Each machine often has many GPUs, and among the different
28:06
GPUs you might be running your job with multiple data loader
28:09
workers. So like you might be
28:12
parallelizing, you basically are massively parallelizing requests
28:17
to your kind of training data set.
28:21
And so, you know, like when the data set that you have is small,
28:25
you can kind of just like, let's say I'm running a job on two
28:29
different machines, like I'm, I'm training a model on two
28:31
different machines. If your data set is small, you
28:33
can just download the whole data set on both machines, and maybe
28:37
you train on half the data set on one machine and half the data
28:41
set on the other machine and and that's it.
28:43
And things are kind of easy, but if your data set is and like to
28:48
be fair, this is what I did when I was at Square.
28:50
So we were dealing with text data and the text data, we had a
28:55
lot of data, but in the grand scheme of things, it was
28:57
relatively small. And so we could just download
29:00
the entire data set to disc and then we could train a model off
29:05
of that data set. But when you have like videos
29:09
and images and embeddings, these get quite large quite fast.
29:13
And it is like totally intractable to download that
29:18
entire data set on every single machine that you are training
29:22
your model off of. And so instead you need to be
29:26
able to download portions of your data set and you want to
29:30
really only download the portions that are relevant to
29:33
training your model. And so like I think we talked
29:36
about this before, where maybe I want to have different embedding
29:40
models from my data set. I don't necessarily want to
29:44
download every single embedding that I've calculated.
29:46
I probably only want to download the ones that I care about and
29:49
use those as input into my model.
29:52
And so like one way to solve this problem is you have you
29:57
have your feature store that we've already talked about and
29:59
then you have a totally separate system that creates training
30:03
data sets for you. So maybe you query the feature
30:06
store, you generate your training data set, and then you
30:09
have your models trained off of that training data set.
30:12
And so like this is kind of like in like classical machine
30:18
learning world. Maybe I have to query database,
30:21
but at the end of the day, I create my like my design matrix,
30:26
like my ex in Sikit learn parlance.
30:29
I create my ex and my SO like my X matrix and my targets that I
30:33
want to train my model off of. And so maybe it's one services
30:36
job to create those training data sets and then you're models
30:39
trained off of this. But again, if the data that
30:42
you're dealing with is really, really big, it could be wasteful
30:45
to have to 1st create training datasets and then train your
30:49
model off of these training data sets.
30:50
And so in an ideal world, you can.
30:54
Your feature store can solve for kind of searching and querying
30:58
and storing all of your data, but it also solves for
31:02
performance only pulling down batches of that data while you
31:07
are doing a large scale training job.
31:09
And So what you need there is you need to be able to kind of
31:12
query your feature store and only return specific rows and
31:16
columns that you care about for training your model.
31:20
And you need to be able to do this kind of in a in a
31:22
distributed fashion where certain portions of my data are
31:26
sent to one machine, certain portions are sent to the other
31:29
machine. And so yeah, I think in an ideal
31:32
world you've you can solve for both large scale training and
31:36
all of the other requirements of the feature store all in one.
31:40
OK. So the whole value prop there is
31:44
that since these data sets are gigantic and I only want a
31:48
sliver of it and I don't even that sliver is going to be too
31:52
big for me to train on my one or two machines.
31:55
I just want the data that is relevant to me when the training
31:59
is at a certain point. Once it gets past that point, I
32:03
want to pull down new data that is relevant to me and etcetera,
32:07
etcetera. That kind of keeps going until
32:10
the training is done. Exactly.
32:12
Yeah. And so like, yeah, when, when
32:13
doing deep learning like this, you, everything, you know,
32:16
stochastic gradient descent, everything operates on batches
32:19
of data. And so you want to be able to
32:21
kind of stream batches from the from the feature store and only
32:27
kind of process these these batches that are relevant for
32:30
you and ignore everything else. That makes a lot of sense.
32:35
OK dude. Wow.
32:38
All right, real fast, I want to tell you about the sponsor of
32:40
today's episode, AWS Tranium and Influencia.
32:43
Are you stuck in the performance cost trade off when training and
32:46
deploying your generative AI applications?
32:49
Well, good news is you're not alone.
32:52
Look no further than Aws's Tranium and Influencia, the
32:57
optimal infrastructure for generative AI.
33:01
AWS, Tranium and Influencia provide high performance compute
33:06
infrastructure for large scale training and inference with
33:10
LLM's and diffusion models. And here's the kicker, you can
33:15
save up to 50% on training costs and up to 40% on inference cost.
33:21
That is 50 with A5 and A0. Whoo.
33:26
That's kind of a big number. Get started today by using your
33:29
existing code in frameworks such as Pytorch and Tensorflow.
33:33
That is a WS, Tranium, and Influencia.
33:37
Check it out. And now let's get back into the
33:39
show. So unreliable feature store.
33:45
This is cool. I I think we could get sued
33:51
calling it a feature store just because so many people out there
33:55
have one idea of what it is. We got to think of a different
33:59
name. But I do like this idea and I do
34:02
think that it can make life a lot easier for the researchers
34:08
and perfect segue into like dealing with researchers and
34:13
making sure that when you're working with them, you're able
34:17
to both let them run wild, but also get something for
34:23
production. And have you found any tricks on
34:27
how to do that, How to make sure that researchers are actually
34:30
producing for production, or at least once it gets past a
34:34
certain phase, they understand what the productionized means?
34:40
Yeah, I mean, I, I don't have any silver bullets, but I have
34:44
lots of thoughts. And so I think something that I,
34:49
you know, I a lot of my experiences drawn from my, my
34:52
team at Square, but I've seen similar things here at Runway.
34:56
You know, I, I feel like there are a lot of people or there's a
35:01
lot of thoughts out there around how to solve this and how to,
35:04
you know, drive more value out of, out of researchers, data
35:07
scientists, machine learning engineers and things like that.
35:10
And I think oftentimes and probably these people have been
35:13
on this podcast, there are lots of tools that are sold to try to
35:16
do this as well. And I think and I, I have used
35:21
some of these tools and I've tried to use many of the tools.
35:24
And I think that one thing I often see is like a wall that
35:28
people throw their models over. So there are tools that are
35:32
really designed to help researchers iterate quickly on
35:36
what they are doing and give them maximum flexibility.
35:39
But at the end of it, but then in order for that researcher to
35:43
kind of deploy their model and things like that, they might
35:47
need to kind of like, I don't know, package their model up
35:51
into a script or something like that.
35:52
So like a lot of these tools, I would say oftentimes like the
35:56
the deliverable of the researcher to the quote, real
36:01
engineers, you know, the the team that's involved with
36:03
production. Amazing is that the researcher
36:05
needs to serialise their model to some artefact and then maybe
36:09
if they're lucky they can hand off a script like a Python file
36:14
and and hand. That off to another team.
36:17
Exactly. Yeah, that's when they e-mail
36:19
it. Sometimes they put it on like a
36:20
zip disc and send it over. Yeah.
36:22
And the problem that they have with this is that this often
36:28
ends up like, meaning that the researcher kind of has to
36:33
reinvent from scratch their model every single time.
36:37
O all of their models are just like one big long script that
36:42
they might hand off to somebody else.
36:45
And, and all they know about is the script and like the, the
36:49
details of which libraries are run during inference and like
36:53
information about the Docker container and everything else
36:55
like that, that that is kind of, you know, abstracted away from
36:59
the researcher. But I think that by having to
37:03
hand off a script, this prevents people from being able to reuse
37:07
code. And so like when I was at
37:11
Square, we we had a single code base that did both training and
37:17
inference. And so we had one code base with
37:20
a lot of shared code and a lot of libraries like internal
37:24
libraries that we had been that had been built up over time.
37:27
And we were able to reuse code that we had written during both
37:32
training and inference. And I think that like if you
37:36
talk to a lot of like conventional software engineers,
37:41
this idea of having a single code base that does training and
37:44
inference feels maybe a little bit bad.
37:46
Like we should have, we should have code that is for training
37:50
and then we should have code that is for inference and
37:52
everything in between needs to be like serialised or something
37:55
like that. Yeah, it's like staging in dev.
37:58
Exactly. Staging and prod, Yeah.
38:01
But I think the problem with this is that if you don't get to
38:05
reuse code, you don't get to enjoy all the benefits of like
38:08
conventional software engineering.
38:10
And so by having a single code base to do all of this and you
38:15
get to reuse your code, reuse your code, you get to keep
38:18
things nice and DRY. Do not repeat yourself.
38:21
You get to write tests. If I'm just handing a script off
38:25
to somebody and they're going to run that in production, then I'm
38:29
not going to bother to write tests.
38:30
But I can, if I have a shared code base, I've already built
38:34
out CID on that code base. I can write tests and things
38:37
like that. And I think that like, and then
38:40
as well, well, like the the trade off of this is that by
38:46
having this singular code base that other people are working
38:48
in, you now need to write code that doesn't break other
38:52
people's code. And so if you want to kind of
38:55
modify something in the training pipeline, you don't have the
38:58
full flexibility to do that. Instead, you need to make sure
39:00
that your modifications don't break other jobs that are using
39:04
that code for training. But I think, and this can slow
39:07
people down, but I think that like the the benefits of working
39:13
in a shared code base end up outweighing the the the cons or
39:18
like the the potential slowdowns of limiting some flexibility.
39:22
I know that this was a long a long ranty response, so maybe
39:26
I'll pause there. Well, I guess the other side of
39:29
it is that people probably think, OK, but my researchers,
39:36
for better or worse, they probably aren't thinking about
39:43
how to optimise the code or how to create quote unquote
39:48
production code. They're just thinking about how
39:50
to get whatever it is in their head out and try and make it so
39:55
that model works and maybe there is a certain metric that they're
39:59
trying to get as high as possible.
40:03
And then it's somebody else's job to figure all the other
40:07
stuff out. And when your asking your
40:11
researcher to now figure out like alright, make sure that
40:16
your conscientious of not breaking things when you're for
40:21
working on stuff, maybe you gotta go and figure out these
40:26
unit tests or integration tests. And you also gotta start using
40:30
git and start doing things that you haven't been doing for your
40:36
whole career or that you haven't liked to do.
40:40
And now I I really think you put it perfectly.
40:43
It potentially can slow people down, but the benefits outweigh
40:47
the cons there. Yeah, I, I think they do.
40:51
And like there's ways to help people with this too, right?
40:53
Like if you I don't like, and I think I have a lot of empathy
40:57
for the researchers. Like I, I went to grad school, I
41:00
wrote really, really bad code when I was in grad school, I
41:03
wrote all MATLAB code. It was it's, it's hideous, just
41:07
like any old code that you've written should be like you
41:10
should always like. If your old code doesn't look
41:12
hideous, then it means that you haven't learned anything since
41:14
you wrote it. But the I eventually learned
41:19
over time various like, I don't know, for lack of a better term,
41:22
best practises that software engineers use.
41:25
And not all of these best practises are applicable to
41:28
iterative research environments, but I think some of them are.
41:32
And I think one of the hard things, one of the hard things
41:35
for me and probably one of the hard things for researchers who
41:38
are in a certain environment is if you don't have any examples
41:41
to build off of. So you know, if you tell
41:44
somebody, hey, here's a an empty GitHub repo, please write unit
41:51
tests like please put all your modelling code in there write
41:54
unit test setup Cici did build a docker container and all of
41:58
these other things, then that's going to be you know they're
42:01
gonna have a hard time but if there's.
42:03
Already a writer? Get a blank page.
42:07
Go and write a book. Exactly.
42:09
And they're gonna reinvent the same book or something, Yeah.
42:14
But like, yeah, it's just a lot more overhead to think about it
42:17
all the way through. Then, if you're editing
42:19
something else that's already on the, there's already something
42:21
on the page. Ah.
42:22
Yes, sorry I cut you off. Yeah, no, I think it's a great
42:25
analogy. Like it's much easier to edit,
42:28
it's much easier to refactor than it is to write something
42:32
from scratch. And so if you have a code base
42:34
that already exists, that is already running tests and
42:37
already has examples of this is how you write a test for this
42:40
type of thing, then it's a lot easier to just add your own test
42:44
on top than it is to build out an entire testing suite from
42:48
scratch. And so there's still, there's
42:52
still the fact that some people just don't want to do this.
42:54
You know, I've worked with people where they're like, this
42:57
shouldn't be my job. And I, I generally don't like
43:01
working with those types of people.
43:02
I don't know, I think it's, I think it's good to kind of like
43:06
work at the boundaries of what you think your job should be
43:10
because like it smooths the interface between people on, on
43:14
different teams. And so, you know, if you, if
43:17
you're a researcher and you have a bit of like interest in the
43:21
production side of things, that little bit of interest can go a
43:24
really long way because if you kind of learn how to talk like
43:27
an engineer, then you can communicate with them better and
43:31
you can get stuff shipped faster.
43:32
And, you know, look, the converse is true as well that,
43:35
you know, engineers who understand a little bit of the
43:38
modelling side of things, they're going to be able to work
43:40
a lot better and like understand requirements a lot better from
43:43
the researchers. And I think that like code ends
43:46
up being the best way for everybody to communicate with
43:49
each other about this stuff. Hmm.
43:52
And so aside from this mono repo idea, are there other
43:58
infrastructure pieces that you've seen that have helped a
44:01
tonne when it comes to researchers and engineers
44:04
playing nice together? Yeah.
44:07
I think like on the on the training side of things, there
44:11
is a lots of tools to like that will claim to kind of like train
44:17
your model very nicely in the cloud or like serve your model
44:21
very nicely on GPUs in the cloud and things like that.
44:24
And I think that again, some of these tools end up running into
44:28
the issue where you need to kind of package up your code.
44:32
Like either you need to like use some weird like domain specific
44:35
language to do this, you need to write like an early YAML file in
44:39
order to do this, or you need to kind of rewrite.
44:42
If I have an existing code base, I need to rewrite everything and
44:46
turn it into a script. And with my, I don't know my
44:51
requirements dot TXT and my Docker information, I need to
44:54
like convert all of that into into this tool, which isn't
44:57
great. And So what I really like is if
45:01
you can still work within Docker world, like most, a lot of
45:06
training under the hood happens in Docker.
45:09
A lot of inference happens in Docker.
45:11
And so I think from like for the researchers, the, the hard thing
45:16
can be that like, you know, if I want to train a model in the
45:19
cloud, I might need to like build an entire Docker
45:22
container, like upload that from my laptop and then wait for it
45:26
to be downloaded on different machines.
45:28
And all of this is, is very painful.
45:30
But I think like the, the closer that experience can get to
45:35
basically just being like Docker with a whole bunch of caching
45:37
involved. Like the, the closer my
45:40
experience is to writing code on my laptop in a shared code base,
45:44
then hitting run when I run a model.
45:47
But if that happens to run in the cloud on lots of GPUs and
45:50
like that, that is the ideal experience is that I can kind of
45:54
sit in my code base and write code how I like to write it, you
45:59
know, using whatever ID I want. But then like with low latency
46:05
abstract away running things in the cloud and like, I think
46:08
there's some new, new companies that are doing this in
46:10
interesting ways, like modal and bowl plan and, and people like
46:15
that. But like, and this is kind of
46:18
what we did at Square. We had you have a shared code
46:21
base, you write code on your laptop.
46:25
And then we had like a tiny little CLI where we package up
46:30
everything into a Docker container, ship it up to the
46:31
cloud and then run a job on GPUs.
46:35
And it was a bit slow because we weren't great at caching things,
46:38
But I think the better you can cache and the faster that that
46:41
experience becomes, the better things are.
46:45
Yeah, because you know it is not going to work trying to get that
46:48
researcher who barely wants to learn Git to learn Kubernetes.
46:53
God no no. Hey, it would be nice if nobody
46:57
had to learn it. It's just.
46:59
Those people who do know it, it's nice when only they are the
47:01
ones doing it. Somebody told me like, ohh,
47:05
yeah, there's this whole generative DevOps movement and I
47:09
made the joke like, alright, well I'll believe it when these
47:12
MLM's have Kubernetes fluency, you know, or like YAML fluency,
47:16
I think is what the way that I put it in there like it's
47:18
coming, it's coming. Still am not so bullish on that.
47:23
And I'm not sure if it's going to ever happen, but hopefully
47:26
I'll be proven wrong. Yeah, I, I, I It would be nice
47:30
if we have something other than Kubernetes before the model,
47:34
like the AI models have to learn Kubernetes.
47:38
That's a different way of thinking of it.
47:40
Just get rid of Kubernetes and then you won't have to worry
47:42
about it and teach an AI Kubernetes or YAML.
47:46
I like that. My my other very brief rant is
47:50
that I did this at Square and I I replaced all the YAML with
47:56
with Python code. And I think that every like I
47:59
hate YAML and I think that the more you can like we replaced
48:03
all the emo with pedantic models like kind of data class, like
48:06
configs where you get nice type hints and validation and
48:09
everything else. And I think that like the this
48:13
ended up paying dividends as well because I feel like what
48:16
often happens is you have like maybe engineers who start
48:20
building out all of this crazy custom parsing of YAML files
48:24
because the YAML becomes the interface between the researcher
48:28
and the engineers. Because the engineers like, all
48:30
right, as long as you can get it into YAML, then you know, I can
48:34
like run your training job or serve your model.
48:36
And then the researchers like, well, actually I need this like
48:38
weird custom logic. And instead of just being able
48:41
to write Python code to do that, the engineer now has to write a
48:44
custom parsing of the YAML file in order to implement the logic
48:48
that the researcher wants. And now you have this like these
48:50
weird YAML transpiler and it's a mess and I think people should
48:54
just write Python code so. Yeah, first the researcher
48:59
spends probably 5 to 10 hours researching YAML and how to how
49:04
Kubernetes even works and what Kubernetes is.
49:07
And then they're like, actually, you know what, Fuck this, going
49:11
to try and get somebody else to do it or they try and figure
49:14
something else out. So yeah, that that is a classic
49:19
one. Man, that is so funny.
49:20
The last piece on this from my end is how are you structuring
49:26
teams so that you have the researchers and you have the
49:29
engineers playing nicely on that organisational front?
49:34
Yeah. So we we have the kind of cringy
49:37
title of member of technical staff for all of the people, all
49:41
of the technical people at Runway, which I feel like is
49:44
like the new, the new AI company thing to call everybody this.
49:47
And you're cool. Yeah, it's the flat.
49:50
Yeah. And like 2010, it was what, flat
49:53
management? So you didn't have a hierarchy.
49:57
Something. Yeah, yeah, yeah, yeah.
50:00
So, so maybe that's what it is. But you know, the argument is
50:03
that boundaries between teams should be fuzzy and people
50:07
should feel free to work on what they want to work.
50:09
That said, we still do have teams.
50:11
And so we have like we have a large team of researchers.
50:16
We have a kind of solidly back end software engineering team
50:21
and front end software engineering team.
50:22
And then I sit on the machine learning acceleration team that
50:26
sits in between back end and researchers.
50:30
And so this is kind of. Like your acceleration.
50:32
Not engineering, but acceleration.
50:35
Huh. And wait, what is the goal or
50:39
like what kind of KPI's do you have or OK, RI, guess what, what
50:43
are the metrics that you care about?
50:45
No, none. I mean, we're a.
50:47
Startup. So I think, you know, with
50:52
artistic leadership too. That's the other piece.
50:55
Exactly. I think things change too fast
50:58
to set like quarterly, quarterly KPI's and things like that.
51:04
But generally I feel like the team that I'm on that I'm on is
51:09
really warranted around kind of speeding U the research both
51:14
like training as well as in France for the researchers.
51:16
And so you know, the, the faster they are able to run
51:19
experiments, the faster they're able to train their models, you
51:23
know, the better that their model scale when training all of
51:26
that side of things we work on. And then on the inference side,
51:30
you know, the GPU's are, are very expensive.
51:32
I don't know if you know this, but they, they cost a lot of
51:36
money and so you want to be able to, you know, you want them to
51:38
go Burr, you want to use them as much as possible.
51:41
And so on the inference side, you know, the faster we can run
51:44
inference kind of the more models we can pack on the GPU
51:46
and things like that, the the better we can kind of utilise
51:50
our our capacity and stuff like that.
51:53
Is it all homegrown stuff that you have?
51:57
No, we we have a giant mixture of everything.
52:02
O nice, nice. Well, this has been fascinating,
52:07
man. I really appreciate this idea of
52:10
the multimodal feature store and it's something that I'm going to
52:13
chew on for a bit. I think especially when it comes
52:16
to thinking about a how it enables the large scale training
52:22
and the distributed training and getting the data exactly when
52:25
you need it. It reminds me of like the Toyota
52:27
just in time model and you don't want to create the car before
52:32
somebody has demand for that car.
52:34
You don't want the data to be downloaded until there is
52:38
somebody that has demand for that data or a model needs that
52:41
data for their their training run.
52:44
And the searchability on the features of a multimodal feature
52:50
store also seem fascinating. And then being able to segment
52:54
your data sets and and then give researchers different options
53:01
for the flavours of datasets. I think that's a cool way of
53:04
looking at it too, because then they can hopefully get more out
53:09
of the same data almost I guess is what it would be.
53:13
It's like you can squeeze the lemon a little bit more and get
53:17
more juice out of it. Exactly.
53:19
There's like that that Google Paper, you know, everybody wants
53:23
to work on models like, but nobody wants to work on the data
53:27
part of it. I don't remember what the title
53:28
is, but it's, you know, it's the same idea that, you know, the
53:30
model is this tiny component, but then like the, you know, the
53:34
quality of the data that you're training off of is, is hugely
53:37
important. It's just typically a lot less
53:39
fun than, you know, futzing around with like neural Nets and
53:43
attention and things like that. And so I think that the the more
53:46
fun we can make it or you know, more realistically, the less
53:50
painful that that we can make expecting and understanding your
53:55
data set and deciding what your model trains off of the kind of
53:58
the better, the better results that we've got.
54:02
Yeah. And I also like this what you
54:05
were talking about on how to accelerate the researchers, like
54:07
you're the researcher enablement, department or
54:10
acceleration department. And that is a really cool role
54:13
to have. So you get to think about, for
54:16
these researchers, what is the best structure and what is the
54:20
best way that we can make sure they are utilising all the
54:24
resources we have to their maximum capability and we are
54:28
getting as much as possible from them and hopefully nobody's
54:33
overspending on GPU's in the meanwhile.
54:36
Never, never optimal spending the whole time.
54:41
Excellent. Well, thanks Ethan.
54:42
This was great man. Thanks so much.