Latest / Agentic Conversations (formally mlops.community) / Foundational Models are the Future but... with Alex Ratner CEO of Snorkel AI // MLOps Podcast #139

Transcript

Foundational Models are the Future but... with Alex Ratner CEO of Snorkel AI // MLOps Podcast #139 · Agentic Conversations (formally mlops.community)

0:00
Hey, I'm Alex, I'm one of the co-founders and CEO at snorkel.
0:03
A, I led the snorkel project at Stanford prior to spinning out.
0:07
I'm also on faculty, University of Washington.
0:09
And I think that the prompt question was, what kind of
0:12
coffee do I drink? So my, my embarrassingly
0:15
stereotypical Silicon, Valley answer is actually actually use
0:21
a kind of flash frozen coffee called comments here.
0:26
It's awesome. If you if you want to check it
0:29
out and basically they shipped these little capsules that you
0:32
run under hot water and just plop into a cup of cup of hot
0:37
water. And and that's that's what I
0:40
drink because it's a I wish I was sophisticated enough to have
0:44
the master the art of making it myself.
0:46
But that's, that's easy and fast.
0:49
And it's it's pretty delicious if you want to check it out.
0:51
So that's my, that's my coffee, answer.
0:55
Welcome back everybody. This is another edition of the
0:57
MLA Ops Community podcast is I am your host to be treat, host,
1:02
and I am here with none, other than the most, incredible
1:08
co-host. Abby, what's going on?
1:10
Abby? How you doing?
1:12
Wonderful. Just finished my workout came
1:16
back to crash. I'm absolutely nothing better
1:21
than recording. An intro about a podcast that we
1:25
did a week ago. We've had some time to let this
1:28
one sink in and our special guest today, was none.
1:32
Other than the CEO of snorkel. Alex, he also has written some
1:37
pretty incredible papers. You knew him from his papers
1:41
first before snorkel, right? I mean, he's been around.
1:46
Yeah, I've got a friend who joins nor coup in 2017 18, so I
1:53
had a chance to read some of his papers and resist serve seeing
1:57
their open source work from the outside, I wasn't really
2:00
involved in all of that stuff, the sort of like keeping track.
2:05
Yeah. Today when we talked to him and
2:08
in this episode I had so many huge takeaways.
2:12
I love the fact that he is starting to think about
2:16
Foundational models and how ml Ops and these gigantic models
2:21
are going to start playing together.
2:22
It's becoming one of the biggest Buzzy words of the day or of the
2:28
year. I think the foundational models
2:30
everyone is excited about because of their potential but
2:34
there was a lot of stuff that we dug into.
2:36
We got into the papers we got into just his view on the world.
2:40
What were some of your takeaways?
2:42
So I think for me first was our discussion It's around Ensemble
2:47
modeling, which was how to combine different Waters because
2:50
he was obviously using we supervision, but he was also
2:54
starting to combine it with other deep learning techniques.
2:57
So, that was quite interesting for me.
2:59
The other is more. So his mention of how to
3:05
actually use foundational models because I feel like I've been
3:09
somebody who's been slightly skeptical, I mean all that GPT,
3:13
3gpp, for all that stuff is great.
3:16
I've always felt like it was excellent as a high from because
3:18
of the predictions that are being that good are.
3:21
It's like, so for high-stakes situation, we can't actually
3:24
sort of use that one. So true give a very interesting
3:28
perspective. It was like, let's use them as
3:30
base models and do feature engineering on top of it for use
3:34
case, specific scenarios and that makes perfect sense.
3:38
That's so true. And the way that he talked about
3:40
some of their clients using these models shows that it's not
3:44
all hype there. Is value being derived from
3:47
this. And so I'm excited as to where
3:49
it goes and what we're going to see in a year or two years from
3:53
now and how much of the market is going to be using these type
3:58
of models as opposed to something totally in-house or
4:02
the smaller models. So this is the new frontier and
4:06
it's always staying interesting. But option would be very
4:10
interesting to see ya, it's going to be really cool as time
4:15
moves forward. So we got to give a huge
4:17
shout-out to snorkel for sponsoring this episode.
4:20
They are incredible. I mean, I've Loved snorkel even
4:23
before they sponsored this, I think they're doing some super
4:27
cool stuff and if you are interested in anything that we
4:32
talk about in this episode definitely hit them up.
4:36
You know what their website is already but will leave a link in
4:39
the description snorkel dot Ai and I think we might, as well
4:43
just get into it a be, any last-minute thing. is that you
4:46
want to say before we jump into the conversation, give us a
4:49
comment on He's part by ever, very using us.
4:53
Give us all the good ratings and really good bones.
4:56
Why not share this with a friend while you're at it?
4:58
That would be incredible. All right, let's see, let's get
5:02
into it. Young'uns around.
5:10
Yes, there's there's like a mini Nursery because we got, we have
5:14
an eight month old and then my nephew is here.
5:17
Also, and you want me to go try valid or real or no.
5:21
No, I shall don't worry. Don't worry, it's all good.
5:26
I have the kids around too so that's just life these days
5:32
kits. More time to read about, we see
5:35
revision and active learning, right?
5:40
It's great Demetrius. How old are yours?
5:43
So I have a four year old and she just got sick.
5:47
And so she's just basically is constantly coughing like
5:51
continuously coughing and so my wife and I have not slept in the
5:55
past like three days because of that.
5:58
So, my wife just went to bed at 6:30.
6:01
Here, she just texted me like I'm going to bed.
6:03
Ed. I was like, oh my God. 6:30.
6:07
All right, that's some texture to give this advice in but take
6:11
a spoonful of Honey, sprinkle a little bit of black pepper in it
6:15
and then give it to her, that would just take away the cough.
6:19
Oh yeah, twice's we've been given her the honey, but not the
6:22
black pepper part so I'm gonna, I'm gonna put that to the test.
6:26
Yeah, I'm going to steal it also because we've got yeah, I have I
6:29
have two kids. The two-year-old just started
6:32
preschool. So Just a constant, she's the
6:34
disease Vector. She brings new coffee each week
6:37
and then it goes through the babies and, you know, parents
6:40
grandparents. And so it's just been constant
6:42
sickness here to, and lots and lots of coughing.
6:44
Lots of coffee. That's all it is, man.
6:48
They come up with. I like how you say that?
6:50
It's a new cough every week. It's literally, my daughter was,
6:55
she's been off and on coughing for the past six months.
6:58
And every time that she goes down, she literally is like a
7:02
sine wave like she Just is down and then back up coughing.
7:05
And then she recovers right in time to get another cough.
7:09
So anyway, we didn't come here to talk about that.
7:13
Abby gave you a prompt or asked you a question before, and I
7:17
said, save it because I want to hit record before we get into
7:22
that. Abby, you want to jump into it.
7:24
Yeah, one of my questions face. Basically you've been pursuing
7:28
week, supervised learning. But why week supervised learning
7:32
oversee, some They're sort of transfer learning or Active
7:35
Learning. Well, that's an awesome awesome
7:38
way to dive in. So I think that at a high level,
7:42
the the broadest interest that has kept me busy, for the last
7:47
almost eight years, which sounds a little not at all depressing
7:51
to me. Although it sounds a little
7:52
wonder the dimensional if you think about it in one way, but I
7:55
think about another way it's a there's just lots of occasion.
7:59
Yeah and there's lots of explore around around data and what we
8:02
think of as data-centric AI. So I think top-level concentric
8:05
Circle, there's And it's not an either/or, but there's this
8:09
split that we make between model Center development where you
8:13
might think of data being relatively fixed.
8:15
You're iterating on the model, that's what I often think of as
8:16
the kind of kaggle era of machine learning or the imagenet
8:19
era, where you just, you know, what a lot of us started with.
8:22
If you look at five plus years ago, every single Paper was
8:24
about features, our algorithms are bespoke model architectures,
8:27
that's what people were doing out in Industry.
8:29
If they were actually working on a new problem and the data was
8:32
someone else's job, it was something outside of the kind of
8:35
ml developer's responsibility. Be right.
8:38
And so this broader set of things that that has been
8:40
capturing Maya. Now, many others interests,
8:43
which is super exciting and and the snorkel team, since back in
8:46
2015 is this idea of broadly data-centric methods where
8:50
you're you're, assuming that the model class can be Auto
8:53
selected, or you can kind of pick a standard class.
8:55
That was kind of the forward-looking bet.
8:57
That's not not too forward looking anymore.
8:59
Given given how things have converged in such an exciting
9:01
way out in the open source, and what your instead of doing is
9:04
really iterating on the data labeling, it shaping it.
9:07
Helping etcetera. So within that there are all
9:09
these methods that you can view from the data Centric
9:12
perspective and I'll rephrase them.
9:14
I own simplistic way, but, you know, you've got Active Learning
9:16
which is essentially saying, let's be smart about where we
9:20
look on our next iteration of developing or labeling data.
9:24
You've got transfer learning, which is, let's see if we can
9:27
kind of reuse something from a prior data set or model to kind
9:31
of jumpstart the performance. And then you have things like,
9:35
weak supervision, which I would phrase is how Be radically more
9:38
efficient and Scrappy around how we actually do the labeling once
9:42
we're looking at something, so you can kind of see.
9:46
Obviously my way of phrasing. It is going to lead into the way
9:49
we view it, but we do these as very complementary techniques.
9:52
They're all they're all tools in the toolkit and it's worth
9:55
noting. The kind of at least the
9:56
cautionary note I get first, which is that each one of these,
9:59
you can find instances of them being pitched as magical, silver
10:02
bullets. And those kinds of free energy
10:04
machine pitches. Don't really help, real
10:07
Practitioners. That's our, that's our strong
10:08
perspective. After eight years of working in
10:10
the space, Active Learning has been around for decades, amazing
10:14
progress in the field of a grad student.
10:15
I get to code by a u-dub who just published a paper on active
10:19
learning this week supervision actually and some cool results,
10:21
but it's not a silver bullet. Even if a little theoretical
10:25
intuition is that deciding where you need to look next in the
10:31
data is as is often as or more difficult than actually deciding
10:36
on the decision, boundary that Trying to build and say,
10:38
training a classifier in the first place.
10:40
In other words, intuitively if you knew exactly where the model
10:42
needed more signal, that's as harder or harder of a problem
10:46
than actually training the model.
10:47
The first place, you don't know what you don't know.
10:49
So to speak. And so these are all really
10:52
helpful techniques but they're not silver bolts.
10:53
And our view is that and this is what we both put our research
10:57
around but also build around our commercial platform, snorkel
11:00
flows that they all work together.
11:02
So in our conception of the data-centric iterative workflow
11:06
each iteration Or saying how can I label or otherwise improve
11:10
the, the quality of my training data and active learning guides
11:13
you to where to look to go next. Where should I label by hand?
11:18
Where should I label programmatically, or with week
11:20
supervision? Where should I debug or or edit?
11:23
And they're a whole bunch of ways to doing that.
11:24
But that's the kind of the, where do I go next?
11:27
Then there's the, how do I label do I go through and kind of
11:31
Click one data point at a time in the kind of Legacy manual
11:33
labeling approach which can be appropriate and He is part of
11:38
the process in tough problems but then the week supervision
11:41
question that we've been working on for years, is can I be more
11:43
efficient? Can I write functions at labeled
11:45
attic? I even use things like
11:47
Foundation models, or large language model.
11:49
So take a wild. Guess we'll get into in our
11:51
discussion today and that gives a hint of how can we not?
11:54
Yeah. How can we not that gives a hint
11:56
about how we think about about fitting it into this data
11:58
Centric workflow, we can talk about why but basically how can
12:02
we use all these more efficient ways to label the data also more
12:06
maintainable, Always things that look more like labeling your
12:08
data with code. So you can now edit and change
12:11
and share and reuse and govern which you can't do with big
12:15
piles of manual labels and then transfer learning we think of as
12:18
just a way to kind of raise the Baseline that you're starting
12:21
with. Right.
12:22
So whether it's classical transfer learning or it's zero
12:25
shutter. Few shot methods, which surprise
12:28
surprise will again take us back to Foundation models.
12:31
If we, if we dive in there, that's the kind of okay, how can
12:33
I start with a baseline performance?
12:37
Rather than starting from scratch, that's actually why
12:40
myself. And my co-founder Chris is, was
12:43
part of starting up the Stanford center for research and
12:46
Foundation models, like, likes that Foundation model term
12:49
because we think of the foundation's, their kind of
12:51
sturdy generic foundations, then you still have to build your
12:54
specific custom building or house on top, but you want to
12:57
start with with those strong foundations.
12:59
So to summarize transfer link, think of it, as kind of your
13:01
warm, start versus your cold, start your kind of Base, Most
13:06
problems are not magically solved by transfer learning.
13:09
If they're complex and non-standard, right most
13:12
problems are not solved by zero. If you shot learning when
13:14
they're actually kind of tough. High-value bespoke problems on
13:17
custom data sets custom objectives out in the real
13:19
industry but they're great. It's a great way to start.
13:22
Active Learning is kind of the, where do I go next to label, or
13:26
improve my data. And we supervision is, how do I
13:28
label my data in a more efficient, and maintainable way.
13:32
So a little bit long-winded but that's how I see them all
13:34
fitting together and and and playing playing nicely together.
13:38
And there are two ways to go from here.
13:40
One is we directly jump into the foundation of what it was but I
13:43
had fewer questions about your papers.
13:46
So I'll ask you the questions about papers and then we can go
13:49
into detail on foundation and Waters.
13:51
So, one of the things I was reading your paper on Nemo and
13:56
you were talking about labeling heuristics and how to allow
13:59
users to participate or inject their knowledge into the
14:02
labeling process. Can you talk a little bit more
14:05
about them and Alice thought you thought?
14:08
That we were going to like, do pleasantries and say hello and
14:12
go through like, what's your background?
14:14
We just jump right into him and I think I need to get to do a
14:17
proper intro. We hit no to hardest-hitting
14:21
topics. I can imagine sick kids at home
14:23
from preschool disease vectors and then straight into paper
14:26
interrogation. I love it, I love it.
14:28
I'm Alex. By the way, I think there's our
14:30
intro but let's let's dive right back into.
14:32
This is the fun way to do it. So the name of paper, this is
14:34
from a student to get to work with University of Washington
14:37
Chang. You who's Awesome.
14:38
This is actually what I was referring to when I said that
14:41
kind of recent work on Active Learning plus programmatic
14:44
supervision. So, the idea at a high level in
14:48
Nemo is its kind of building on this, this snorkel programmatic
14:51
supervision line of work, where you start by you label data by
14:56
trying to write a function rather than just labeling
15:01
individual data points. And this function, we usually
15:04
call labeling function labeling heuristic.
15:06
But sometimes we very at the names, And make the academic
15:09
work, more generic. But I'll refer to as a labeling
15:13
function here, which is our standard terminology.
15:14
And the idea there is like you said, it's a way of injecting
15:18
domain expertise, so I can. So let's just focus on that
15:21
part. This is kind of a snorkel idea
15:23
and then we'll talk about where Active Learning comes in in the
15:25
Nemo work. The basic idea here at one, way
15:28
to motivating, is actually how we started the snorkel project
15:32
or one of the motivations is this awesome project through
15:35
DARPA that we were funded under DARPA simplex?
15:37
And they've had, they paired us with what they call, smees or
15:40
subject matter experts. So, we were working with some
15:43
genomicists accent, collaborators, and Stanford.
15:45
Some say they have do some awesome stuff on, pediatric
15:47
genomics, that actually has International impact so really,
15:50
really exciting. Motivating projects.
15:53
Unfortunately I can't solve the common cold in two to four year
15:56
olds, like we were talking about at the beginning that they for
15:59
some for some random person that just drops in on this DARPA
16:04
working with smiies and the genomics, or than omics.
16:09
If you were just a little bit outside of the box, it's like,
16:13
wait a minute. Is this some like role play a
16:16
game that I don't know about or archaea?
16:18
Yeah. Exactly.
16:20
How do we get here? Well, I guess I get the
16:21
hard-hitting questions like sending up flares.
16:23
There's the random other pieces but now I'll Arc back to our ml
16:27
Ops and The Eccentric AI B. Which is that, a lot of what we
16:31
spent time doing. This is when the data center
16:33
stuff started to hit us in the face.
16:35
Bulk of the project was not building our fancy machine
16:37
learning models back that in 2015 2016.
16:40
That's a we kind of thought we should be doing is ml
16:42
researchers. Most of the time was spent
16:44
labeling and relabeling and debugging training data and
16:48
error modes. The model and all of it consists
16:50
of sitting there with a smiie saying, okay, I'll label this.
16:54
But you said this. Why is that wrong?
16:56
What features are we missing? What things are we missing?
17:00
And then okay, let's label this data and the really interesting
17:02
thing we did sessions a week is that as they were labeling data,
17:06
these collaborators were talking about why they're relating to
17:09
data. Well, I see this, this negation
17:12
here and I see this kind of phrase so I know it's this kind
17:14
of thing versus that kind of thing.
17:17
But then, all of that information was getting thrown
17:19
away and not passed on to the machine learning model.
17:22
All we were giving the machine learning model was just the XY
17:25
tuples. The here's a data point.
17:27
Here's the label. So a lot of what we've been
17:30
pushing on for many years now, you can view it almost as just
17:34
saying, let's not be egregiously wasteful with this labeling
17:38
process. I mean, this API of all that you
17:40
pass to the machine learning model, is a data point that a
17:43
label for supervised learning has been Table for the field.
17:47
It basically has let people do their model Center of
17:50
development and and not even think about the data and all the
17:52
domain expertise that goes into it which for a period in the
17:55
space. I would argue it's been amazing.
17:57
It's led to this this gold rush and boom of model progress, but
18:02
if you think about it from a real-world sense, why are we
18:04
playing like twenty or Twenty Thousand questions with a
18:06
machine learning model rrrr sneezed?
18:08
The people who are labeling the data, they know, all this
18:11
information, you know, that I'm going to label it here because I
18:13
see this this key phrase. I'm going to label this way
18:16
because I see this Pattern or reference it with this external
18:19
resource or whatever it might be and then we just throw it out
18:22
and we ask the machine learning model to try to statistically
18:25
reinforce it. Why not?
18:26
Just give it to the model. So that's a little bit of a
18:29
long-winded Arc of saying, what is this idea of a labeling
18:31
function? This programmatic, supervision
18:34
or weak supervision? It's that we can label training
18:37
sets often 10 to 100 times more efficiently measured in time
18:42
taken to do the labeling by just giving more information labeling
18:46
with Functions or heuristics are patterns rather than just by
18:51
hand and accepting that it's going to be a little messier
18:54
than individual data labels, and we're gonna have to clean up
18:56
that messiness algorithmically, which is the weak supervision
18:59
part. So I'll take a brief pause there
19:03
and then I can get to the Nemo paper, which I'll give the quick
19:06
bit there which then ties into Active Learning, which is
19:08
basically saying okay, that's the going back to where we
19:11
started. That's the that's the how do I
19:13
label? Let's get richer information
19:16
from. It that are experts who are
19:18
labeling the data in the form of a labeling function rather than
19:21
just click click click, but still, we can also be smarter
19:25
about what data we show those people to label next.
19:29
So regardless of whether you're just clicking and labeling in
19:32
the Legacy away or you're writing a labeling function to
19:35
give richer information, you can still be smarter about where you
19:39
look next to build that. These lately functions don't
19:41
just come in a vacuum, they come from looking at the data and
19:44
saying, oh, I should, I should label it this way or Etcetera.
19:47
And so the name of paper combines the two and says, let's
19:49
both be smart about how we label and where we label on each turn
19:53
of the, of the, of the of the crank.
19:59
Hey, I'm Vishnu. I'm a data scientist at first
20:02
hand, and I definitely think that you should subscribe to the
20:05
ml Ops coffee sessions podcast. It's the best podcast out there
20:09
to stay on top of what ml Ops actually is to talk to the
20:12
truth. Thought leaders in the space, if
20:14
you definitely subscribe to the podcast. great resting and this
20:23
But this also ties up with another thing.
20:25
You talk about The Eccentric AI in something, which has been my
20:28
personal experience, which is often the model you pick,
20:31
whether it's for classification or regression problem or even
20:35
for labeling, it depends very heavily on the gear itself.
20:38
And in your original snorkel paper that you published in
20:42
2017, you said there, there are two directions from here.
20:45
One is, we figure out an end-to-end system for their
20:49
programming for textual data and the others.
20:51
We look at different Modalities including images with your
20:55
detector, would it be fair to say weak supervision works very
20:59
well for textual data and may or may not work.
21:02
I don't know the research on that one but does it work
21:05
equally well for other modalities?
21:07
So we started in text because we're we're big nerds around a
21:13
lot of the really exciting problems and natural language
21:16
processing and knowledge extraction which is a lot of we
21:19
worked in, but there's such a gold mine.
21:20
I mean, I can draw it on about this to about all this.
21:23
Information that's kind of more and more accessible than ever
21:26
before. In one way I won't repeat the
21:27
kind of I don't know, Market stats on data availability, we
21:31
know there's a lot of data out there, the more accessible than
21:33
ever before but it's also the knowledge and that data so
21:36
inaccessible. So a lot of what we do.
21:38
We started with we do today is all about buying tagging,
21:40
extracting pulling structured, usable data out of unstructured
21:44
data. Whether it's a bunch of
21:45
scientific papers back in the Stanford days or whether it's a
21:50
multi hundred page contracts in a bank or, or or a Insurance
21:53
Agency, Etc. So that's where we started.
21:56
But image has been a big focus and my co-founder at this normal
21:59
company, Perlmutter all PhD are the big applied.
22:02
Focus was on image and video, and I think we've actually
22:05
published more papers such on the medical side.
22:08
Couple nature, calms papers on image and video and we
22:10
supervision there than we even did on text, which is a weird
22:15
artifact, but it is true. And so, we're actually really
22:18
excited about progress there and actually that takes us You one
22:23
thing that we've been teasing, but I don't want to talk about,
22:26
which is foundation, I'll stay, which is how Foundation models
22:29
we see fitting into this, this picture.
22:32
I'll just preview that a lot of the image use cases are
22:37
accelerated by these Foundation models as well.
22:39
So one way that we'll get to, I want to take a top-down, but
22:44
this idea of a labeling function, you can kind of think,
22:46
maybe in more simple ways about all in textile, look for a
22:49
pattern, or I'll look for a key phrase or something like that,
22:52
but it both to image that might seem harder.
22:54
How do I do that over pixels? There are answers to that you
22:57
can. You can run feature detectors or
23:01
object, detectors to give you building blocks that you can
23:03
then write labeling functions over that some work we publish
23:05
in 2017, you can use metadata that maybe textual or
23:09
structured. But you can also now use
23:12
labeling functions. Based on Foundation models that
23:14
allow you to kind of right here. A sticks, right?
23:18
Labeling functions over image using some of these great
23:22
either. Image based or multimodal
23:24
Foundation models that give you these kind of powerful ways you
23:27
can literally just write a natural language prompt to write
23:30
a labeling function over image. Now, given these amazing models
23:33
and so the foundation models play a big part of that about
23:36
that strategy as well. Which is why we've been seeing
23:38
some exciting image results and are rolling that out also.
23:41
Now, on the, on the, on the company side, I love that.
23:46
So one thing that I think foundational models do not get
23:50
enough credit for is the use cases around Enterprise, like
23:58
actual things that make money, right?
24:00
Like you hear a lot of people saying, oh yeah, foundational
24:04
moms are great. If you want to make a cat riding
24:07
on a rocket going over the moon, that's cool.
24:10
And it just gives you some picture or it's doing some,
24:13
like, Runway ml. Is doing some insane things when
24:16
it comes to video. But when it's like, but let's
24:20
get down to business and the adults in the room, still need
24:23
to predict fraud. You can't do that with the
24:25
foundational models. Right now, I think you have a
24:28
bit of a different opinion like Enterprise use cases when it
24:32
comes to Foundation or models. Can you break some of those down
24:35
for us? Yeah, I think it's a great
24:38
Prelude because there's incredible excitement and
24:40
incredible real value and magic in these Foundation models in
24:45
the way that they've gotten scaled up.
24:47
And yet we see such a gap between that and real Enterprise
24:50
usage. And so the fact that Gap exists,
24:53
it is is very real. From what we see, right?
24:56
We work with a bunch of top US Banks, and government agencies
25:00
and insurance Healthcare tell Cam across a lot of verticals.
25:03
And in a lot of these settings where there's High business, Or
25:06
Mission or patient value. High value also goes with
25:10
stricter, constraints and and and and greater consequences and
25:13
governance structures Etc. You don't see Foundation models
25:17
being used and so let me take a step back and give a our, our
25:22
view on this from experience in these, in these use case
25:25
settings and all characterized kind of to gaps here around the
25:30
challenges of adaptation and deployment.
25:32
And those two things adapting Foundation models to these to
25:35
the challenging. Some use cases in that have high
25:38
value and Enterprises and solving the deployment
25:42
challenges of, how do I actually serve a foundation model in
25:45
production, quick preview answer is that most Enterprises count
25:48
and don't think they will be able to, for a couple of years
25:51
for not just cost and latency, but also governance reasons.
25:56
So, how do we solve those adaptation deployment
25:58
challenges? So, let's talk about the
26:01
adaptation one first, and I can start with an example.
26:05
But one way of viewing is that if you have all All these kind
26:07
of generative created exploratory, use cases.
26:09
I want to generate that the cat on a rocket ship, going over the
26:12
moon or I want to generate some copy text that I'm going to edit
26:15
Foundation models, can do amazing things right out of the
26:18
box in this kind of human, the loop kind of procedure, one way
26:21
of thinking about it is, there are looser, accuracy,
26:24
constraints there because you're using as part of the creative
26:28
process. Not as something that has to
26:30
spit something out with a certain High bar of accuracy,
26:33
right? Versus your augmenting humans.
26:36
Abilities as opposed to replacing it.
26:38
Yeah. And it's super exciting.
26:40
And I'm personally a big fan of on the amateur side of that,
26:43
those Loops I've been playing around like everyone else with
26:45
these models but that's very different from the kind of
26:49
automation or what we had more technique called discriminative
26:53
versus generative use cases. Especially in complex settings
26:56
where you you have to achieve a high accuracy bar and also you
26:59
have to do it in a setting. That is not the same as the
27:03
data. These models are trained on on
27:04
webtext, right? So that I'll get, actually, let
27:09
me give an anecdote really quickly.
27:10
I was playing around actually early on with the stable.
27:13
Diffusion to get a little snorkel logo.
27:16
You can see this one is a snorkel ago.
27:18
Wearing a space suit for one of our internal hackathons, but I
27:21
was saying, okay, what kind of generative aii these days with
27:24
Foundation models do here? So I had an awesome experience.
27:27
It was really exciting. I went through about 30 samples
27:29
until I got something that looked pretty cool.
27:31
I couldn't get the octopus, wearing the snorkel underwater.
27:34
Maybe that was too far outside of the Support distribution but
27:38
I got a really cool image. So from a creative generative
27:42
workflow that was a big success, but if you think about it from a
27:47
kind of production automation, or discriminative modeling
27:50
perspective, one success, after 30 tries is like a 3.3%, kind of
27:55
hit or accuracy rate, right? That's abysmal.
27:59
So how do you get from one to the other?
28:02
And how do you kind of bridge that gap, which is both about
28:05
tuning, the accuracy and then doing it on?
28:06
A bespoke use cases. We should always be very
28:09
careful. I think about drawing analogies
28:11
between Ai and humans given how how, how dumb AI still is, and
28:15
how we're still struggling to solve very simple challenges and
28:18
real production, use cases, but one analogy is the kind of
28:21
generalist to specialist if you have someone who's learned to
28:24
read on Reddit and the internet. You wouldn't expect a human to
28:29
be able to just suddenly reach multi hundred page contracts or
28:31
analyze very technical fraud or the network Telecom data right
28:35
off the bat, they would need to have some.
28:37
So true adaptations some tuning, right?
28:39
So, this Gap from generalist to specialist, and from kind of
28:43
loose, accuracy, constraints in the gender of world to high
28:46
accuracy bar, in the discriminative world.
28:49
That's this kind of adaptation Gap and the standard way of
28:53
Crossing It Is by fine tuning. This is an old transfer learning
28:56
concept, right? You just fine-tune the model on
28:59
labeled training data. And once again, you're back to
29:02
this data Centric, AI problem. You can do prompting and zero
29:05
shutter fuchsia techniques, but most studies showed it and our
29:09
internal results also, corroborate that if you want to
29:12
get on these very tough problems to high accuracy.
29:14
Usually you're just going back to find sitting on labeled
29:16
training data probably less than if you were starting from
29:19
scratch. But still for these bespoke
29:21
problems quite a bit. So our view is surprise surprise
29:26
that adaptation. Is adaptation Gap.
29:28
Once again comes back to data-centric development.
29:31
You're not like when you want to do this adaptation Gap, you're
29:34
not going another way to think about it.
29:35
You're not going into the foundation model and if we can
29:38
some of the multi-billion parameters, I mean there there's
29:41
some people work on patching and stuff that's going on but you're
29:44
kind of leaving that. You're fine, tuning your
29:46
adapting and that's all about labeling editing developing a
29:50
data. Once again, so you've got this
29:53
big adaptation Gap. That a lot of people that's at
29:55
least how we frame it across in this.
29:57
Go From generative to to discriminative, or prediction
30:00
Automation, and also Crossing from kind of generic web text
30:04
where all these models are trained to custom settings, or
30:08
specialist settings. And then the other one that I
30:10
think a lot of people underestimate is this deployment
30:12
gap which is that I talk to one of our customers, a large top
30:16
three US bank and they said that they were talking.
30:19
They were thinking about submitting to their model risk
30:22
management, committee about getting G, PT, 3 and production.
30:24
And they likened to an art project.
30:27
And a a Don Quixote like tilting at windmills activity.
30:31
So this was their pessimism that when they would happen given the
30:34
they're still even challenge is getting modern deep learning
30:37
models kind of through to production and honestly there
30:41
are cost barriers. These things are very expensive
30:44
to serve their, our latency barriers.
30:46
They take time, we're going to chip away at that.
30:50
One of the biggest ones is when you also throw in governance
30:52
because we are just beginning to understand these large
30:56
Foundation models, actually Ali doing really exciting things on
30:58
the academic side, where it's almost like a return to Natural
31:01
Sciences world where you're kind of poking and prodding
31:03
empirically rather than just reasoning formally.
31:08
But yeah, I think Enterprises are justified in saying, hey, we
31:11
have to understand a bit more before we just kind of serve D,
31:15
PT 3 to customers, even if we can, solve the cost challenges.
31:18
So you've got these adaptations gaps.
31:21
How do we do the data center development to adapt these
31:24
models for these very specialist, high accuracy bar
31:26
settings, And you've got these deployment challenges around,
31:29
how the heck do we actually put this into production, the next
31:32
year or two or three in these large Enterprise settings?
31:35
And so this is something we're, we have an opinion as you could
31:40
guess, and we're actually about going to be announcing what we
31:44
call our support in platform for data, Centric Foundation model
31:47
development, and the basic idea here is first of all, for the
31:51
adaptation, all that fine tuning that again, is where we've been
31:55
working for years about, how do you Make data labeling and
31:59
training data development as rapid as possible, through
32:02
programmatic, techniques and active learning, and
32:03
transferring, and all these things.
32:06
And the deployment Gap we solve, by actually, using these
32:10
Foundation models to help us label data, which we can then
32:14
use to train smaller models that fit into our deployment
32:16
Paradigm. So 11 cool set of it.
32:19
We're going to put it'll be off soon a case study on some public
32:23
data that we can really share the full results.
32:26
It's like a hundred way. Classification problem on
32:28
contract so still even simpler than some of the ones you see in
32:32
production but a little bit more representative.
32:34
And the cool thing is if you take a baseline of fine-tuning G
32:36
PT 3 on a bunch of manual labels to get to a high accuracy bar
32:41
using our approaches. You can use less than 1% of the
32:44
ground truth labeled data and actually train a model that is
32:49
over a thousand times smaller and hundreds of times cheaper to
32:52
run to actually go to production.
32:54
So think of it as you're using these Big bulky generalist
32:58
Foundation models to help accelerate data center
33:00
development of smaller, specialist deployment models
33:03
that are both more accurate with lesson but also far cheaper and
33:09
again this may sound like it's too good to be true but it's
33:12
really just, it's really that generals to specialist divided,
33:15
right? In an Enterprise deployment
33:17
setting. We're not looking to have this
33:18
gigantic, generalist created machine that we can query.
33:21
For anything, we're looking to solve 1 Task on very custom
33:26
data. Very accustomed objectives.
33:28
With high accuracy and repeatability and ability to
33:32
apply proper governance. And so what we're trying to
33:35
build is we're aiming to be that bridge between all this
33:38
Foundation model goodness and these actually Deployable models
33:42
in these production settings. If that makes sense.
33:45
I love that. It's like this offspring of the
33:49
foundational model or the the child of it in a way and you've
33:53
got your mini Station remodel almost but it you're echoing.
33:57
Something that I was just talking to a buddy Danny
33:59
yesterday about. And he had worked at Google for
34:04
like eight years. He was working with researchers
34:07
on trying to figure out ways to get their foundational Manos,
34:11
like finding ways to put them into products, right?
34:15
That was one of his main things was syncing between the
34:18
engineers and the researchers to find different ways to get those
34:23
foundational models in. The different Google products.
34:26
And one thing that he talked to me about was he was saying how
34:31
with these certain types of models you have to be okay with
34:36
that, lower accuracy score. And you also have to be okay
34:41
with the idea of The risk of error or cannot be high stakes
34:48
like what you're putting this into play for, it cannot be high
34:51
stakes because the accuracy score is so low.
34:54
So you can't ever, you can't do it where it's like, like we were
34:57
talking about with these fraud Mount front detections where it
35:00
could really mess things up. And so he made the comment of
35:04
like, yeah, social media, you could throw it in there because
35:07
if somebody sees the wrong post, whatever, it's not really
35:10
ruining anybody's day, just because one wrong Ang post or
35:14
whatever. So now it feels like though,
35:18
what you're talking about is much more.
35:20
Let's figure out how we can take these gigantic models that are
35:25
covering so much surface area. And then take the pieces that we
35:30
like extract those pieces that we like use the foundational
35:33
model to train those pieces that we like and then we can have
35:37
that higher bar and we can put it into much more high stakes
35:42
settings. Yeah, exactly.
35:45
If said said even better and more succinctly.
35:48
Actually, we may share this contact because I also had a Dan
35:53
from Google research, who worked in this area, come by the lab a
35:56
couple years back to give a talk.
35:57
And no, I won't do full names on podcast but later, we can talk.
36:03
Yeah, we definitely heard about this taking a step back, just
36:07
even without my well at least about my metaphorical snorkel
36:11
had a long because I'll keep my literal hat on right now.
36:14
Now, I desperately need a haircut but strict orders from
36:18
the wife. Just keep your hat on whenever
36:20
you can especially on video. That's gonna go out onto
36:24
YouTube. Yeah.
36:27
Yeah. I think in general thinking of
36:29
these settings where I often think of it yet, like, it
36:32
doesn't have to be perfect accuracy or your kind of
36:34
competing against a no Baseline. I think, in general, these are
36:38
great places to fly. Any type of machine learning,
36:41
especially these Foundation models where Where they're not
36:45
tuned and very flexible but not at this kind of like sa accuracy
36:49
SLA bar. So I think that's a good
36:52
strategy to keep in mind in general.
36:53
Right. This is why a lot of ml projects
36:55
in think are really successful and things like like let's take
36:58
our work. We've done the medical field
37:00
with a lot of radiologist at Stanford and VA Healthcare.
37:02
We did stuff around triaging rather than automation, we said,
37:05
okay? Like let's let's triage a cue
37:07
that's backlog because the Baseline is just nothing.
37:10
It's just the doctors read The order they came in.
37:12
So any It is better versus. Okay, let's take on full
37:16
responsibility of automating. What a doctor does, which is a,
37:20
is a big consequential bet to make.
37:22
And I still think we have a ways to go there.
37:25
So, I think in general, it's a great kind of divide to think
37:27
about. It's definitely what we see for
37:30
foundation models to write. If you again, think of these are
37:32
these big generalist generative models that are amazing and what
37:37
they can do, but they're not fine-tuned or specialized to get
37:40
amazing accuracy on some specific task.
37:44
In fact, even making them kind of better on General as tasks
37:47
often requires fine to me, like there's the flan results that
37:50
came out from Google recently. And what do they do?
37:52
They find to it on all these all these tasks.
37:55
So actually you see in the broader space of course I would
37:59
say this, I'm talking my book, I'm sticking to the the
38:02
data-centric stick, but even in building these generals models
38:06
you see a lot of data centricity in the thinking, right.
38:10
Everyone's kind of converging on more or less the same
38:12
architectures. There's obviously The amazing
38:14
engineering work that goes into it, but you look and see, like,
38:16
look at some of the models that come out, this is it.
38:18
Look at the stable, diffusion release.
38:20
You look at some of the one of my colleagues u-dub, blood with
38:23
Schmidt published, some interesting work with his team
38:25
on clip models. Open air Whispers.
38:29
Another one that has some good details in the data.
38:31
All these models of plan, they all the one of the biggest
38:34
factors of innovation is how you curate the data going into
38:37
training them. So even in getting these
38:39
Foundation models built, we're seeing prize the data, they're
38:42
trained on the curation of that. Very stable, diffusion, they had
38:45
a separate semi-supervised model where they labeled data to look
38:48
for beautiful images and then bootstrap, this model to some
38:51
select down to a training Corpus of nice looking images, right?
38:54
So you even in just building a foundation walls themselves,
38:58
you're already seeing data Centric iteration data center
39:01
development being key. So we're just saying, okay, now,
39:04
when you want to take that next step of actually putting them
39:07
into a production setting, a specialist setting.
39:10
Once again, it often comes down to the data both to fine-tune
39:13
them, but also, To be this kind of bridge into a Deployable
39:16
artifact that you actually can put into production today, for
39:19
interesting. And this is one thing I often
39:22
think about especially because as a developer, the foundational
39:25
models are very interesting. You kept retreating what?
39:28
And somebody spending up the money and giving it to you to
39:30
play with. That's all cool.
39:33
But when it comes to, hey, are these the best models to work
39:38
with for actual business problems?
39:40
I am often questioning Is it the right way to just be able to
39:46
increase the number of parameters and just optimizing
39:48
it optimizing further? Should there be better models or
39:51
should we be putting more research into better models?
39:54
That are generally optimized for these kinds of solutions.
39:57
Given we've done already done a lot of work in meta learning as
40:01
well as generating mottos. Yeah, I think I mean I think
40:05
we're going to see progress from from all kinds of directions.
40:09
Right? So I think and you tell me if I
40:12
misunderstanding that some of the points you're making but
40:15
we're I think we're going to kind of see chipping away on one
40:17
end up making these models more specialized and making them kind
40:21
of better for subdomains, right? Like a legal and medical GPT
40:25
whatever it might be. And then you know, on the
40:28
frontlines of kind of trying to get these these productionize
40:32
what we're trying to do. Do is just find good ways to use
40:35
them now, and actually, going back to what you were asking
40:38
about on our first question. The first question about a week
40:40
supervision and this idea that is underscored a lot of our work
40:43
of labeling data programmatically and then being
40:45
able to combine multiple and perfect signals, which is kind
40:48
of the Crux of the thinking there, that's part of how we
40:52
got. So excited about found it
40:53
Foundation models in this way of using them, because your, right,
40:56
as a developer, What Foundation model should I use what's
41:02
appropriate? First of all there's there's an
41:04
exploding Universe of them, right?
41:06
And especially in the open source now, which is
41:09
fantastically exciting and then now on top of it but you have
41:12
all these different ways of prompting them right?
41:14
Which is I mean of course you have fine-tuning you a few shot
41:17
learning now. But now the zero shot technique
41:19
of prompting is there's a new weird little prompt sentences.
41:23
You have to write every single day coming out.
41:25
And so how do I actually know develop the prompts to kind of
41:28
It's the right information out. Well in our week supervision
41:31
formalisms, that's just another type of labeling function.
41:35
So and this is actually one of the features were releasing.
41:38
Their actually kind of three that we call one is fine tuner.
41:42
So just we're adding support and snorkel flow to fine-tune some
41:45
of the newest breeds of of foundation model.
41:49
So if you actually can ship a foundation model to production,
41:52
now we can let you fine-tune that with this programmatic data
41:55
approach that we support and have built.
41:58
Alt. And then we have these two other
42:00
features called warm start and and prompt Builders.
42:03
So warm start is basically just the kind of what I was talking
42:06
about transfer learning as a kind of a bootstrap Baseline
42:09
using Foundation models with zero shot techniques.
42:12
Just Auto label your data, as a first pass on, one of these
42:15
Specialists production problems. They're not going to get you up
42:18
to the accuracy level. I'll but the simplest of
42:21
problems. Like for example, we were using
42:23
this with am A large Bank, who's a customer on an anti money
42:27
laundering and know your customer problem where they had
42:29
to pull hundreds of pieces of information out of these big
42:33
piles of customer documents. So some of the simple things,
42:36
like, what's the name, or what is the date on this contract or
42:38
things like that? Actually just using this kind of
42:41
warm start, zero shot approach could get pretty accurately,
42:44
which is exciting and you kind of expect the bulk 80 80 plus
42:48
percent. It gave a kind of jumpstart but
42:51
it was nowhere near accuracy level.
42:53
So you can kind of Basically, warm start vs, cold, start your
42:57
problem, get some of the low-hanging fruit labeled.
43:00
And then you can use programmatic labeling, right?
43:05
These labeling functions to rapidly label the rest of the
43:07
data up to Quality in this iterative approach that we
43:09
support and snorkel flow. But now you can also write what
43:12
we call Prompt labeling functions which is basically
43:16
writing domain specific prompts for these various Foundation
43:20
models to try to label parts of the data set and then the best
43:24
part is It all of the week supervision formalisms.
43:26
We built for years. They're all about okay, how do I
43:28
combine all that signal? Into my final training set,
43:32
final model so that you as the developer don't need to worry
43:35
about, hey, should I be using, you know, this gbd3 or Bloom?
43:39
Should I be using this prompt or that prompt?
43:40
You just kind of dump them all in and snorkel takes care of
43:43
modeling and integrating them which is that's how it kind of
43:46
fits a little in the weeds. But that's how it fits nicely
43:49
into the the kind of paradigms have worked on for years.
43:52
All about, how do I my opportunistically use all the
43:56
signal I have at hand for labeling in a more efficient
43:59
way. And in this rendition Foundation
44:01
models are just another tool in the toolkit and immensely
44:04
powerful one. But it also an imperfect and
44:07
tricky one that you have to to in an engineer and is not going
44:10
to work magic out of the box on your specific problem.
44:14
So why don't you just throw it in with all the other kind of
44:17
signals in this data center workflow?
44:18
That's kind of the approach we take.
44:19
And we've been saying seeing it work pretty well for customers
44:22
even even in early beta. So what you're saying is
44:26
basically use it as a baseline model the way we use Autumn oh
44:30
photos to sort of first see the Baseline accuracy and then we
44:34
build a model on top of it and see.
44:36
Okay let's do feature engineering on top of it.
44:39
Yeah start with a baseline because that's the kind of
44:41
concept of Foundations. And then do this iteration on
44:44
improving the data. It could be by writing
44:47
engineering specific prompts to kind of fine tune, the output of
44:50
the foundation models. It could be by writing other
44:52
labeling functions of just say hey if See, this pattern labeled
44:55
this way, using that kind of specialized domain knowledge and
44:58
then put it all together. That's what snorkel does for you
45:01
and turns it into a clean training set that you can use to
45:05
train or fine-tune any model that you want for deployment.
45:09
I see, very interesting. Now, I'm going to ask you one
45:12
more question. One more piece of snorkel that
45:14
really excited me, which is basically, you now have your own
45:18
hosting infrastructure snorkel, float, how does it compare
45:21
against gcp? Because I know your focus was
45:24
building integration as a service platform.
45:27
Yeah, so we deploy and kind of connect with all the cloud
45:31
environments and all the main ml vendors.
45:33
And so the key for us is we just want to support this
45:37
data-centric workflow. And so we connect to whether
45:40
it's a Vertex or Sage maker Azure or a home built machine,
45:44
learning modeling solution. We just want to get you through
45:47
this data Centric workflow on whatever platform, and we
45:51
actually run the gamut there to. We have our snorkel flow Cloud
45:55
solution that is that we host. But we also do a lot of customer
45:58
bpc on-prem deployments actually because a lot of our customers
46:02
are working on very critical problems with sensitive data.
46:05
That's part of why it's so difficult for them to get it
46:07
labeled. And it's part of why the GP
46:09
decrease of the world don't work out of the box because it's this
46:12
private custom on-prem data, very critical.
46:16
So because of that we're very flexible in the deployment
46:18
environments and then just because, I mean, the data
46:20
science space is amazing in terms of the amount of tools,
46:22
whether open source vendor that are out there.
46:24
And so we think interoperability is a First class design
46:29
principle for any platform and our beliefs, our high-level
46:32
stick internally. And with customers, in terms of
46:36
our design principles building, our platform snorkel flow has
46:39
always been, we do want to provide comprehensive support
46:41
for a workflow and in particular this data-centric workflow, so
46:45
over 62 percent of our customers, use us for kind of
46:49
end-to-end development of applications.
46:50
Start with raw data and get to some model and they do that in
46:53
snorkel flow. But we also want to make interop
46:56
as easy as as possible. So if you want to label your
46:58
data and snorkel flow and then go train a model in your own
47:01
custom environment and do some models education there and then
47:04
come back to do edits on your data.
47:06
We try to make that as easy as possible to So last question for
47:11
you from my side, you all came out of I know I got a buddy who
47:16
just went over to Chris, raise Venture Capital firm, /,
47:21
incubator /, everything machine, you all came out of there.
47:26
How was it? Can you give me some background
47:29
on what that was like you and how it inspired you on what
47:31
you're doing now? Chris is a, Chris is an
47:35
extremely unique. Imagine is he's one of the one
47:37
of our, the actually, I can't give him too much credit on
47:39
public record little. I can't give him that much.
47:42
So I would say he's one of the smartest people I've ever had,
47:44
the fortune of working with. He was my advisor back at
47:46
Stanford and he's a, it's a pretty incredible place.
47:50
So this was back in the quaint old days.
47:53
It was just just the lab Noah Factory.
47:58
But, I mean, yeah, I could rattle down for a whole nother
48:01
hour on things that I Learned from from Chris and from that
48:05
environment and continue to learn working with him.
48:08
But I think maybe just a couple things here to tie it back to
48:13
the themes we've talked in one of the things that I think was
48:15
pretty unique and is still with that group that both resume with
48:19
me, which is why I gravitated to working with Chris.
48:22
That I also took away was the importance of especially the
48:26
field of machine learning where there's so much people turning
48:29
up papers, their new fancy methods everyday.
48:31
There's lots of, there's lots of shiny things floating Round
48:34
right, Chris and the people in his lab or Geniuses at those
48:37
pieces as well. The snorkel team the sub team
48:40
that spy out. We made our contributions on the
48:42
theory and algorithms side, but the importance of just getting
48:45
your hands dirty with real use cases and and that Anchor Point
48:50
being good on the principles. The fundamentals, the formalisms
48:54
but also pushing yourself to spend time just sitting with
48:57
that genomicist or a clinician or whoever is actually trying to
48:59
build something and actually taking the weeks months years to
49:03
To just see that through we never would have started on
49:06
data-centric AI if we hadn't been doing that because we were
49:08
thinking, okay, we're going to work on X fancy machine,
49:10
learning modeling technique in our users kept saying, okay, no
49:13
we're stuck on the data, we just need to label the data That's
49:16
What's blocking us. Do you have any solutions there?
49:19
Took us long enough but you don't actually I mean it sounds
49:21
trite but I think it's true in this field, it's very appealing
49:24
to Because of how we set things up, it's wonderful for progress
49:28
in some ways, but it's very appealing to just download a
49:31
benchmark data set and try to build the fanciest model to get
49:35
the score to go up or to build the best cats on rocket ships.
49:38
Forcing yourself to actually work on real production, use
49:43
cases. While also doing your hard work
49:45
to study. Avi those textbooks that you
49:48
asked about in the background that we recognize a couple of
49:51
them that I'm, I will not confirm or deny.
49:53
If I've read every single page of the ones you'd reference, but
49:55
like learning the fundamentals. The pairing that with just
49:57
getting your hands dirty with real problems, real data.
50:00
That was one thing I took away and that I think Chris.
50:04
And that whole group, that whole machine is very good at
50:07
balancing. The two because if you just have
50:09
to, I mean, if you just hack, if you just look at real problems,
50:11
you just hack away without trying to abstract and formalize
50:15
and and pull on lessons from everything in the academic and
50:19
and I'll world we built up. Then you also won't go the right
50:21
direction, but if you just polish nice, abstractions,
50:24
without getting your hands dirty.
50:25
So it's it's that unique combination of forcing yourself
50:27
to do both. We still try to do that with the
50:30
company. We try to be very Hands-On with
50:31
customers with these cases, not just throw something over the
50:35
wall and that's something that's one of many things.
50:37
But one big thing I took away from that group that I'm hoping
50:40
your friend is, is going to take away and be excited about as
50:43
well. Incredible man, this has been so
50:46
cool. Alex, I appreciate you coming on
50:48
here and expanding my vision of the power of foundational
50:53
models. And what comes next in the
50:58
marriage of Additional models in ml Ops.
51:00
Hopefully, this has been an absolute pleasure, and I want to
51:04
thank you again for doing this Demetrius Abby.
51:07
Thank you so much for the awesome conversation.
51:10
Definitely one of the best I've had because we started right
51:12
away with the hard-hitting questions and got in the weeds
51:15
and and they are pulling back on the, I think, the two things
51:18
leave it there, I think we talked plenty about it but at a
51:21
super high level really simple. Things one data, surprise
51:24
surprise and and developing data is at the center of everything
51:27
in Ai and it It is even more so the case and that'll continue to
51:32
come to Bear went with Foundation models, come into the
51:34
scene and second, there's a bunch of really exciting and
51:37
really under fulfilled opportunities which were
51:40
pursuing, but there's lots of room there to kind of bridge.
51:43
These Foundation models with real use cases, existing real ml
51:47
Ops infrastructure and approaches.
51:50
That's what we're excited to be announcing.
51:52
There's lots of stuff to do there though.
51:53
So I hope other people kind of focus on that because the
51:56
generative exploratory use cases are awesome and they're There
51:58
should be a bloom of creativity there that that's going to be
52:01
awesome. No doubt.
52:03
But bridging it to these real production use cases is of is a
52:06
massive white space right now. And so there's going to be
52:09
exciting stuff there and it was awesome to get to talk about
52:12
today. Yes, I love it right on dude.
52:17
Hey, Lazlo here. If you are serious about ml Ops,
52:21
you hit subscribe right now.