Latest / Agentic Conversations (formally mlops.community) / The Birth and Growth of Spark: An Open Source Success Story // Matei Zaharia // MLOps Podcast #155

Transcript

The Birth and Growth of Spark: An Open Source Success Story // Matei Zaharia // MLOps Podcast #155 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm a taser. Herria, I'm co-founder and chief
0:03
technologist that data breaks. I'm also see us professor at
0:07
Stanford, and I don't drink coffee, but I love tea.
0:12
Hello. And welcome everyone to a very
0:14
special. Emma loves Community podcast.
0:17
I'm your host Demetrios and I am joined today by none other than
0:21
Vishnu was going on my man. It's been too long Demetrios.
0:26
Why has it been so long since you've had me on a podcast?
0:28
I mean, you've been busy. Uh dude, do you want to tell
0:31
everybody what you've been up to?
0:33
Well, if you're interested in healthcare claims which I don't
0:35
know why you'd be but I am Iran. Recently ran an online course on
0:39
how to use claims data insurance claims to actually you know,
0:43
power different Healthcare use cases and stuff.
0:45
So I have been busy. You're right.
0:48
Yeah, but I love that. You are actually shipping
0:51
something, you know, while some people are just sitting around
0:54
talking on podcast, others are going out there and creating
0:58
true products. And getting it out to the world.
1:01
So I appreciate that about you and I you know, you said you
1:05
were going to do it, right? And you actually went and did
1:07
it. I mean you have made a lot of
1:09
claims over the years about writing a lot of blog posts for
1:13
the ml Ops community. And I think you've authored to
1:18
maybe 12 to shoot a 22. Let's do so, I was a little
1:22
skeptical when you told me that that was actually going to
1:24
happen, but I mean, you proved me wrong man, and I'm proud of
1:28
it. Thanks man.
1:29
Now let's talk about Matt Tay. Let's get into this because it's
1:34
their what a conversation. I mean, I knew this guy was
1:40
incredible but I was not expecting this level of
1:44
incredibility. What were some of your key
1:46
takeaways? And then I'll give you mine,
1:48
honestly, I think I kind of had two big, takeaways number one,
1:52
it's so clear that Mathias, somebody that you know, Kind of
1:58
has created like an almost like an intellectual Village, you
2:01
know, as a Stanford I data breaks.
2:03
Just like clearly bounces ideas around.
2:05
A lot has a lot of wide-ranging conversations is thinking big
2:08
and I think it's just a lesson for all of us to, you know,
2:11
continuously sharpener ideas by sharing it with others.
2:13
However, that might be take advantage of the smart people in
2:17
our lives. So that's sort of one thing.
2:19
I think the other thing that I learned from him was just
2:21
understanding that translational Gap.
2:23
You know, there are a lot of Open Source projects out there
2:25
that never really, you know, Used by tiny people and I never
2:28
successfully commercialized, right?
2:30
And data, Bix data Briggs has been a part of to not just like
2:33
major ones, but like industry Legends, right?
2:35
And ml flow is bar not to mention other open source stuff
2:38
that they've also been promoting and sort of commercializing.
2:42
And so I think it was just interesting to hear how he
2:44
thinks about shipping with users building with people.
2:49
You know, to kind of get that early traction whether it's an
2:52
open source project or a or a product.
2:55
So those were sort of my takeaways What were yours was so
2:58
good and I'm going to just say yes and to everything you said
3:03
because it is so true. I mean, for me, I didn't know he
3:06
was a rock band fan, I found that one out, so if anybody's
3:10
looking for a Christmas present for meant, a get him rock band,
3:14
and preferably the Beatles addition because that was one of
3:17
the best ones. And the thing that I loved was
3:22
when he started talking about these large language models, I
3:25
mean, sadly, he did use the The term llm Ops which I'm trying to
3:30
make sure doesn't actually become the thing.
3:32
But I it's got more momentum than I can handle.
3:35
It's all the input on my shoulders and I can't do it.
3:38
So I'm a cave and start saying it sooner rather than later.
3:43
But he talked about how much of The operational cost and the
3:50
resource intensive act, it is to actually create a large language
3:57
model. And then if it's not that good,
4:00
you've got something on your hands that you can just like
4:03
make incrementally better or make like astronomically better
4:08
by any means. You got to go back to square
4:10
one. So you kind of have one shot at
4:13
it and you go and whatever it comes out with your stuffed.
4:17
Duck with it. And when he was talking about
4:20
some of the work he's doing at Stanford with potentially
4:23
smaller models that have access to databases that they can pull
4:28
from and the ability for you to update those databases and then
4:33
the large, the small language models are able to pull from
4:37
those. I thought that was fascinating.
4:38
I love getting into that. And I really just, I mean,
4:43
honestly, I just loved seeing the way the guy thinks and The
4:47
things that she is looking at and how he analyzes things,
4:53
there's a reason that the company is doing from what the
4:58
rumors have said. And potentially his PR teams
5:01
going to make us cut this out, but I'm going to say it anyway.
5:03
We've heard on the street, they're doing like a billion in
5:06
revenue and so I make sense, man, I mean he took spark and it
5:11
was nothing. He talked to a lot of users and
5:15
it became the Behemoth that It is.
5:17
And I actually if you want to talk about using terms, I have
5:22
been trying to make this cool check out this term when
5:26
something. Usually a piece of technology
5:31
gets sideswiped by another Peach of piece of technology.
5:35
I just call that it gets he duped and Mets.
5:38
A single damn? Didn't lie?
5:40
Hadoop to dupe? It's pretty good.
5:43
It's pretty good one. So hopefully ml Ops is not going
5:48
to get he duped anytime soon for those who are listening in, this
5:52
is your first time tuning in, I will remind you that we have got
5:57
a newsletter, we've Actually, we've got three newsletters that
6:01
you can join. So jump in on that, we've also
6:04
got the slack community, that is a vibrant Place.
6:08
We've got the local, meetups all around the globe and the virtual
6:12
meetups and what would mean the world to us?
6:16
What would mean the world to us Vishnu, if the listeners would
6:19
just subscribe? Oh, dude, there's your college.
6:26
A cappella group is coming out in five.
6:28
But I do right now a little taste on this Earth, why let's
6:33
get into it before. All of a sudden the watch rate,
6:37
just dropped it plummeted right there before we get to that day
6:42
and everybody's just dropped off, dude.
6:44
Alright, that's it. Here's Matt say, I'm going to
6:53
start. I really want to get to know
6:56
Matt a the person because I feel like you give a ton of talks and
7:01
you are known throughout the community for your intellect and
7:08
the tools that you've created. But I would love to just start
7:12
with like how you come up with ideas.
7:15
Yeah, great question. Yeah.
7:17
I think I don't know. I mean I think a lot of it is,
7:20
you know just talking to a wide range.
7:22
Of people and and, you know, trying to figure out a what,
7:26
what challenges are they facing, what I what are the things that
7:29
excited about, you know, with with them out and putting some
7:33
of those ideas together. So I'm pretty fortunate to be
7:38
able to see, you know, like both the academic research world.
7:42
And people, they're thinking about like, what's a really new
7:45
research idea? And, you know, just companies
7:48
that are actually trying to use ml to like make something happen
7:52
through. To see the database extent.
7:53
Of course, there's also dataworks own perspective, you
7:56
know, using ml internally for various things so I think just
8:00
like hearing about problems across.
8:03
A wide range of people helps you helps you synthesize sort of new
8:07
ideas and beyond that. I mean I also like, you know,
8:11
I'm definitely a big believer in like going for a walk and like
8:15
you know I'm taking time to think about things and like
8:18
really like you know kinda asking about the next level
8:20
question like, you know what's That's a better way to do this.
8:24
Yeah. So when you want your ideas
8:26
challenge to do, you go to that's very easy all you know my
8:31
co-founder is all the folks at database we have a culture of,
8:35
you know, debate and you know, raising questions.
8:38
So it's very straightforward but yeah, bouncing bouncing ideas
8:41
off folks helps a lot. Yeah, I love this idea of that.
8:46
You're speaking of trying to get to the next level of questions
8:50
and finding those blind spots that you have.
8:52
Have and, like, the, the Shower Principle, I find is one of
8:57
those funny ones where you think about something really hard,
8:59
right? And then you forget about it,
9:02
take a shower and that's when the answer comes to you.
9:04
I love it when that happens because it feels like you don't
9:06
have to do the hard work that you've already put it in.
9:10
But can you talk to us a little bit about spark because that's
9:15
almost like machine learning lore by now and I would love to
9:19
hear House Park went from being an idea that you were bouncing
9:26
around with people to actually becoming a full-fledged product
9:30
and being used by a ton of people.
9:33
So spark started out what I was a PhD student and at that point
9:39
I was really interested in this kind of data center is scale
9:43
Computing that was happening, mostly at web companies are the
9:47
time like Google you know Yahoo was it was a big company
9:50
Microsoft. Often so on and they were, you
9:53
know, mostly doing things with the web indexing, the whole web
9:56
and then doing stuff on top of it.
9:58
And there were you know in like right before I started my PhD,
10:03
there were these open source projects to do mapreduce like,
10:06
like Hadoop that had started out.
10:07
And so more companies were thinking about it and I thought,
10:10
you know, collecting collecting data isn't actually that
10:14
expensive. The cost of storage is pretty
10:16
low and there are so many things that produce data that like
10:20
Evie. Company every scientific lab and
10:23
so on, will want to use. So I thought, hey, many more
10:26
people will care about this, like just programming was with
10:29
large-scale data and so I wanted to learn about it.
10:33
And I, I worked at the beginning, I actually met a lot
10:37
of the users over the open source tools.
10:39
I collaborated with them on some projects so I can boost things
10:44
there. But I also quickly realized
10:46
that, you know, these tools work really Limited in the kind of
10:51
Jen's a corridor and they were good for building a weapon.
10:53
That's why they weren't good for running.
10:55
You know a more interesting algorithm like machine learning.
10:59
And the interesting thing is as soon as ever and collected lots
11:02
of data and organize that they wanted to run all these
11:05
applications, you know, no one stopped.
11:06
I like, okay, we we've downloaded and like organize the
11:10
whole web but all we want to do is like surcharge, you know,
11:12
they wanted to do other stuff with that.
11:14
So it was very clear early on that people wanted to do a whole
11:18
lot more algorithms on it. And then then with spark we
11:23
identified like to use cases that that people couldn't do
11:27
well. One one was machine learning
11:29
with with iterated, algorithms like is Judy and then the other
11:32
one was interactive ad hoc queries and we built an engine
11:37
that was you know, primarily for those two things at forced
11:40
overtime, it expanded to doing, you know, all the other like
11:44
large scale on this batch processing stuff.
11:46
You can do. And the other reason it became
11:50
very closely. Tides, do machine learning is at
11:54
Berkeley where I was a student, you know, we didn't have huge
11:57
data sets. We didn't have like I'm sorry,
11:59
I'm this who are riding a giant website or anything like that.
12:02
So I was looking for people there who actually wanted to do
12:05
large-scale Computing and I was in a lab sitting next to machine
12:09
learning people and they all wanted to try, you know, is
12:11
that? So, a lot of the early users
12:13
were doing large scale machine learning there.
12:16
That's also, I mean, since you brought it up, one thing that I
12:19
did want to ask you about was The crew and Berkeley and
12:23
correct me if I'm wrong. But is it a lab?
12:25
Yeah. What was it like working in that
12:29
crew? What was that whole experience?
12:31
It was a great to go by thinking in my ear.
12:34
When I started, my PhD ton of students came to Berkeley
12:39
actually, like a lot of the people I met at all the grad
12:42
school, you know, visit days and other places ended up coming
12:45
there. I don't know why.
12:46
Maybe the weather was nicer there that year when we visited
12:49
or something. But you know, a lot of them And
12:52
and so we had a big group of friends from the beginning and
12:56
you know, there were all the other folks around meeting in
12:58
the lab before. And one cool thing about this
13:01
lab is it was so the Berkeley faculty who are involved.
13:08
There were a lot of people who work on computer systems and
13:11
databases but they also they were interested in looking at
13:14
how machine learning interacts with those basically.
13:18
So they also put in Michael Jordan's.
13:21
Mike, Jordan is, you know, a big machine learning researcher and
13:25
they had us all, you know, kind of sit in the same physical
13:28
space and they actually like created a big open area where we
13:32
would play like rock band, you know, in the evenings and stuff
13:35
like that. Like it was a really nice space
13:37
and you got to meet these people that you wouldn't have met
13:40
normal eat in, you know, in schools where they're just like
13:43
on different floors and you know, you like never run into
13:46
them. So I think that created a lot of
13:48
cross-pollination and people people became friends But people
13:52
also, like thought about research ideas across these
13:54
things, that's how the machine learning people like when they
13:57
when they saw I was working on spark a bunch of them came to me
14:01
and said, hey we want to try some of this stuff.
14:03
So yeah. Wait do you play any
14:06
instruments? No I don't.
14:08
I did play Rock Band. I wasn't right at it and yeah,
14:13
yeah, yeah. Rock band ingredients for you.
14:18
I love the description of this sort of open and freewheeling
14:22
sort of cross pollination that you're talking about in a lab.
14:26
I feel like that's how, you know, so many of the iconic
14:29
environments that have, you know, seated, you know, great
14:33
ideas, actually operate. And I'm kind of curious, you
14:36
know, how do you keep that sort of culture of cross-pollination,
14:41
you know, part of data bricks, you know, as it's grown to be a
14:44
much larger organization but it's still Still pushing out,
14:47
you know, very forward-thinking products.
14:50
Yeah, that's a great question. I think.
14:52
So, one thing is that internally we have a very open and
14:57
transparent culture like basically, you know, like the,
15:01
you know, are the CEO like basically, like shows everyone
15:05
in the company, you know, the same, the same kind of things
15:08
that he'd show to the the board basically.
15:11
So so there's a lot of like, discussion of all the things
15:14
going on and also It time you know like like someone designed
15:20
something or proposes something or whatever by default, you
15:23
know, you're supposed to send it to a mailing list that
15:26
basically, like every engineer is on, everybody Rock manager
15:30
and like anyone else can sign up for that list too.
15:32
So you can easily like send out a thing and then have, you know,
15:36
like you know VP of like sales at a region looking at it and
15:39
have like, you know, me looking at it and who knows who else?
15:44
So it does encourage everyone to hear about things that Are going
15:47
on and and comment on them beyond that.
15:51
I think having you know especially for like specific
15:55
topics having you know off sites or like sort of many Retreats
16:00
for each team can help like it's very different from sitting
16:05
around on zooms or you know like just you seeing a few people
16:09
once in awhile and sending dogs around.
16:12
So we tried to encourage that and yeah I did.
16:17
The bigger thing is just to make sure that everyone has a chance
16:20
to like, hear about all the pieces and provide, you know,
16:23
their view on it and it's not uncommon to have like a leave a
16:27
CEO, you know, send an email and then have like other people's
16:29
they are, I kind of disagree with this, you know, here's why.
16:32
And then have a bunch of debate, all right, so if I'm
16:37
understanding this correctly, you win, you were at Berkeley
16:42
created spark. And you were hanging out with
16:44
some ml researchers and they said, you know what?
16:46
But that would be great for what we're working on and then you
16:50
decided to create a company from it and you brought this culture
16:55
into the company and you recognize that you needed to be
16:58
pinging ideas off of people as much as possible and testing
17:02
your assumptions. Now the actual formation of
17:06
spark really is something that I'm wondering like where did you
17:11
go from there? Once you start a date of bricks
17:13
in those first days, tell us about that was what was that
17:16
like Yeah, so, I mean, so before we started the company spark
17:21
had, you know, it was around as an open source project and, you
17:26
know, it quickly went from like, I used to know all the users.
17:29
There are like, you know, like three companies using it and a
17:32
couple people at Berkeley and I would go and visit them every so
17:35
often. And like, oh, that's help them
17:36
out because they were trying out.
17:37
This this research project to it went to a place where like, I
17:41
didn't know all the users and even better.
17:44
Like I didn't have to answer all the questions on the mailing
17:47
list because like people answering each other's
17:48
questions. So outside was not cool.
17:52
When did that happen or how fast?
17:55
I think all this happened within about two years, maybe three
17:59
years. Yeah, because we forced open
18:02
source said in 2010. And then we started the company
18:05
in 2013 and like, you know, as we were starting at it, this
18:08
community out already, you know, began forming.
18:11
So, it was pretty fast. It was definitely, you know,
18:13
like, the right thing at the right time because everyone was
18:16
Excited about large-scale ML and data analytics and you know,
18:21
compared to programming and all the other tools that was it was
18:24
just much easier. And it was also designed to
18:27
support like libraries and an ecosystem of packages.
18:31
Like so you can just download and mix things where, as the
18:34
other stuff wasn't. Yeah.
18:36
So so we saw the community forming and then, you know, we
18:40
also realize that like, really get this to the next level and
18:44
have it used, you know, beyond deck from Ernie's and so on it
18:47
needs accompany backing it because all, you know, all the
18:50
companies in this space where felt like maybe somewhat
18:53
threatened by this thing. They didn't, they didn't want to
18:56
adopt and we actually went and tried to convince them to
18:58
adopted and they said you know not right now like and help us
19:02
with like our other system instead I really threatened well
19:06
because it's a sort of you know they don't have like full
19:11
control and understanding of what's going to happen with it
19:13
and it kind of competes with the existing But our psyche I knew
19:17
Ben it's sort of this unpredictable like kind of thing
19:21
in your own map. You made all these plans and now
19:24
you know you there's this other thing that's coming around and
19:28
honesty. It is hard to tell right in
19:29
their shoes. I would have said, yeah, it's
19:31
hard to tell how big will this thing be?
19:33
You know, will it actually work? Well what if it's one of them
19:36
bogs, right? If it's like really hard to make
19:38
it, you know, stable and reliable.
19:40
So it's a little bit scary but having a company helped helped
19:45
like really Make the open source project, awesome because we can,
19:49
we could hire people to contribute to add full-time and
19:53
also I think it helped other companies, decides to take that
19:56
jump and invest. And, and you know, join the
19:59
community and contribute in some form.
20:01
And once there were enough of them, of course, that would just
20:03
happen on its own. So you had spark and data Briggs
20:07
has also created, you know, ml flow, which is sort of, you
20:11
know, another sort of seminal open source project.
20:14
I'm kind of wondering, can you tell us a little bit about the
20:16
ways? At the spark story in the ml
20:18
flow story were similar. I mean given that ml flow or,
20:22
and and different. Right.
20:23
You know, given that I'm a full came out later.
20:25
Did it have like a lot more adoption more quickly from those
20:28
big companies that were originally more, you know
20:30
resistant. I'm kind of curious just how
20:32
they played out. Yeah, I mean I would say yeah
20:34
definitely one difference is when you launch an open source
20:39
project from a company that's backing it and is you know like
20:42
looking for collaborators and so on early on it's a lot easier to
20:45
convince people. All that this will be something
20:48
solid that they can kind of bad on and and design you know there
20:51
are ml off sponsors and so on or around.
20:53
So so that helped a lot. It's much easier where it says
20:57
research project where they say well you know it's a bunch of
21:00
grad students. Like maybe they're, you know,
21:03
they're like smart and so but who knows what they're going to
21:05
do in the future and who knows how solid this is in production.
21:09
But there are a lot of similarities.
21:11
I think, one of the biggest ones I kind of hinted at this before
21:16
was Defying, the need for an ecosystem and sort of a standard
21:21
for certain things. So, for example, with ML flow
21:24
like we we looked at the MLF space and we said there's no
21:29
there's no open source solution that anyone can just kind of
21:31
extends that support for the ml tools.
21:35
Like, for example, if I run one over the cloud ml, you know,
21:40
like platforms, like Sage maker it's, you know, it's nice, it's
21:44
well-engineered, but if I'm developing A new framework.
21:47
Like I'm developing the next bite or two or whatever.
21:50
I can to get, you know, Sage maker to integrate closely with
21:53
it unless like someone at Amazon decides to spend Diamond, do it.
21:57
So we wanted something that people could could extend where,
22:02
you know, everyone benefits from it.
22:04
And we thought there is going to be some open source.
22:07
You know, if there was an open source thing with that property,
22:11
it would be, it would be successful because it's in
22:13
everyone's interest. User is one the thing with Great
22:17
Integrations and also developers of either commercial or open
22:21
source projects wanted to work. And really the contribution we
22:25
can make is trying to figure out just the definitions like what
22:28
are you know, what is metric, what is a model?
22:31
What is an experiment like this and get everyone to agree on
22:35
some common apis and concepts for these?
22:38
And then you've got a really useful platform for users.
22:42
So that was that's the way we started and that's partly.
22:46
Inspired by what we saw in spark, which is in spark.
22:50
As I was saying, there were all these tools for, like, writing
22:53
distributed applications. Like, Hadoop mapreduce, you
22:56
know, various streaming tools, all kinds of stuff like that.
22:59
But no one, like just wants to write a completely new
23:03
application from scratch that, no, that no one else did people
23:06
want to run? You know, libraries packages
23:09
like existing algorithms, right? Like the first thing you do in
23:12
most programs is you import a bunch of packages?
23:15
So, for I can even though it was cool, that you could write say
23:19
the stupidest game means and like a few lines of code and in
23:22
map aggressor and Spark, no one actually wants to ride it.
23:25
They want to just call a library for like, a great
23:27
implementation. That's, you know, numerically
23:30
stable and like fast and all that stuff.
23:33
And it wasn't easy to make libraries for the other
23:36
Frameworks are in spark. We designed it.
23:38
So you can have you can just call a function and pass in like
23:42
your spark context or your data frame or whatever and they could
23:46
change. More computation on to it.
23:48
And then when you run the program, the whole thing,
23:50
execute sufficiently, like all the functions that have been
23:53
chained together, go into one execution plan and the system
23:57
optimizes it, and you get really good performance.
23:59
So now you can import a lot of stuff.
24:01
So we put a lot of stuff in the built-in libraries, but they're
24:04
also really great. You know, libraries outside,
24:07
like all the data sources, all the ml, libraries and Spark, all
24:11
the, you know, like geospatial and domain-specific things out
24:15
there. And that just hadn't existed
24:18
before and it just makes sense from a programming point of view
24:21
to really encourage composition makes a ton of sense.
24:25
And what I want to kind of Zoom double-click, whatever you want,
24:30
whatever term you want to use into, is the aha moment that you
24:34
had 4 ml flow because retrospectively, a lot of the
24:38
things that ml flow defined, you know, they now make sense, they
24:42
are industry standards. Right?
24:44
You know, in terms of what we think about a metric or a
24:46
pipeline line, you know, these are Concepts that make sense
24:49
going back in time. What was the moment where you
24:52
knew that those sort of Concepts and ml flow would make sense and
24:58
get sort of the buzz that they did?
25:01
Yeah. Great question how.
25:02
So we added things, you know, one by one to the project and as
25:06
we added a new concept we we wanted to make sure we really
25:10
validated with with, with potential users and like really
25:13
think through all the use cases. So, you know, it goo.
25:16
Over time. So so for example, I remember
25:18
like, you know, we started out with like, we started out mainly
25:23
with experiments and like multi-step projects you could on
25:27
and basically doing dragging across the rows.
25:29
And then soon after we added the model registry, which is kind
25:34
of, I mean it's a common concept now and everyone like, you know,
25:37
either has one always adding some version of that but it's a
25:40
basically it's a it's a place where you can have version
25:43
models and you can have something that looks like a
25:45
pulley Quest where you can Say, you know I have a proposal for a
25:48
new model but other people can view at, you know, automated
25:52
like processes. Like automated Bots.
25:55
Can also review it and comment on it and then like you only
25:58
approve it if that passes. So for all of these things, you
26:02
know, we talked we try to talk to, you know, many different
26:06
users at different kinds of companies.
26:09
At the very start, it was hard because we didn't, you know,
26:11
have anything. But once we launch the, the
26:14
basic ml flow there were there were users.
26:16
Both on database. So, like customers that you can,
26:20
we can easily talk to and then open source.
26:22
And we try to talk to people at very different stages like some
26:26
of our customers were small startups where it's like, you
26:30
know, one or two, machine learning, people who are
26:32
building, like, say their product recommendation and and
26:35
the whole company's like 10 people, some of them were huge
26:38
Enterprise. As with many machine learning
26:41
teams. And like, you know, each team
26:43
like some teams are in production for years already.
26:45
Some things are just starting Ting and usually, if you, if you
26:49
float, you know, some some idea or some designed by a bunch of
26:53
these and do something that like seems to be good for, you know,
26:56
like three, four or five of them that are pretty diverse.
26:59
It's likely to be good for, you know, many, many other folks, as
27:03
well. The other thing we did is we try
27:05
to keep the number of Concepts low and to keep things simple.
27:09
So, I got away at stuff and that I think makes it easier for
27:13
people to learn as well, and that's something you have to be.
27:16
There's something about it because it's very tempting to
27:18
just add a new things. Yeah, once we have that loop, I
27:22
think given the diversity of customers and also of Open
27:26
Source users, we could work with it.
27:27
It was, it was much easier to tell like, you know whether
27:31
something is a great idea or not, as much as I want to dive
27:35
into ml flow right now, I will mention to the listeners, we had
27:40
Corey and been on the podcast before and they're some of the
27:45
lead maintainers of That it's episode number 103.
27:48
If you want to go and listen to a whole hour of just ml flow
27:51
talk, what I want to get into. Now mad say is I mean and don't
27:56
take this the wrong way, but do you listen to DJ?
27:59
Khaled at all, you don't strike me as the person that knows or
28:02
listens to him much. I mean, I know who he is.
28:05
Yeah. But all right.
28:06
So, I feel like you're for a bit of the DJ Khaled of the ml world
28:10
because you just start dropping a hit after hit.
28:14
It's like and another one, and another One.
28:17
And so, what I want to know now is where's your head at?
28:20
What's the next hit? Mmm.
28:24
Yeah, I think yeah, I mean doubt there are a few things were
28:27
doing. So I'll mention so some data
28:30
Break Stuff, we're working on. And then also some stuff that my
28:34
research group at at Stanford's been doing so a database like
28:39
one of the things we're very excited about is really tightly
28:44
integrating ml. With data and it comes in two
28:49
ways. Hey, Lazlo here.
28:51
If you are serious, about Emma laughs, you hit subscribe right
28:54
now. So one obvious one is when you
29:01
build and drain them? Well, you drain it on data.
29:04
So like, you know, obviously like we you know, we try to make
29:07
that easy. But another really interesting
29:10
thing that we realized is a huge amount of ml Ops is in some
29:14
sense data analytics. So basically you're running the
29:18
model, maybe you're running different versions of it and
29:20
you're passing in, you know, like you're passing inputs and
29:23
it's creating predictions and then later you want to analyze,
29:27
you know, how did it do. So maybe later you have you have
29:31
feedback. Like for example you showed
29:33
someone a product on the web page but they didn't click it or
29:38
they didn't click it right and that that feedback comes later
29:41
after a few minutes and so there's a lot of data analytics
29:44
to combine like you Go to analyze all the events that
29:49
happened after your model came out.
29:51
And then maybe join them with other events and combine them
29:54
and get a great picture of what's Happening.
29:57
We've built for example, our model monitoring tool, like
30:00
basically you know, can put all these events from your model as
30:04
it's turning into a table and then you can use all the data
30:07
warehousing and analytics features, we have to analyze
30:10
that table and we've built kind of a unified data and model
30:15
Observer. Learning toolkit where you know
30:19
whether it's I can input stable that's generated upstream or an
30:22
output stable like that's coming out of your model.
30:26
You can do things like check for shifts in distribution duties
30:30
like delayed kind of feedback calculations where like you see.
30:34
Okay I started this was happening but then you know
30:36
here's what actually happened later and basically get a bunch
30:39
of operators to hook them together and you can also take
30:42
all these and get a like a nice dashboard or Or automated alert
30:47
based on, you know, the entire pipeline.
30:49
So that's like one thing I don't have.
30:51
That makes sense. But we call that lake house ml
30:54
or just in general, like integrating data with ML and
30:58
treating them as, you know, like they're both basically passes
31:02
that like, you know, produce data and you want to be able to
31:05
analyze and monitor that and measure different things about
31:08
it. And a bit of a tangent there
31:10
because I heard it's a good story before we jump into the
31:15
other cool stuff that you're doing.
31:16
Doing, how did you come up with the name data lake house?
31:19
He likes it soon as we started database pretty much.
31:22
You know, we let people do data science and like data
31:26
transformation and biplanes but we had people who wanted to, you
31:30
know, treat their data Lake as as a warehouse and just you know
31:35
connect bi tools to add like Tableau and power bi.
31:39
And so in like you know, enable like anyone who can use SQL to
31:43
basically Quay that data. So we always As you know, we
31:47
wanted to tell this story of like its not just a bunch of
31:50
files you as you work on them, you can organize them, you can
31:53
create, you know, tables and schema is and and do like sort
31:57
of Rich data management on there.
31:59
You can enable these these data warehousing workloads and it's
32:03
like really interesting because it means you don't have to copy
32:06
that data across lots of systems and manage very different
32:09
systems, but we had trouble like really, you know, coming up with
32:13
a term for this. I think when we started, we had
32:16
We actually use terms like virtual data warehouse or just
32:20
in time data warehouse and so on and none of them like really
32:24
stuck. And then one wandering, the the
32:27
engineers here who is, you know, who's good at naming things had
32:32
this idea and like some meeting or some Doc about lake house.
32:36
He said, hey, what you? You got this Lake you you want
32:40
to claim it does? Warehousing stuff do.
32:42
But it's really supposed to be more and it's supposed to be
32:45
like, nice if I die, neither. All of those things like who
32:47
wants to spend time in a warehouse or whatever.
32:49
So so why not do lake house and that that really stuck and you
32:53
know, we ordered a bunch of stuff about it.
32:55
I think it's it may be a term that like, you know other people
32:58
were also thinking about about it, it quickly became like an
33:03
industry standard thing where like if you look at Amazon or
33:06
Microsoft or like Google Cloud, now, they all claim.
33:09
They, you know, they offer some kind of lake house but I think
33:12
again, it's one of these things where like it's awesome to, you
33:15
know, have a lot Out of people kind of, you know, crunching on
33:19
the table more like hearing about it.
33:21
And then, you know, someone may come up with this, with this
33:24
cool idea and sort of propagate, it, such an underappreciated
33:28
skill to have the naming conventions.
33:32
Oh yeah. There's some people are really
33:34
great at naming, so good to figure out who they are and ask
33:37
time after. Yeah, I mean, I hope that
33:40
engineer got there. Got the raise that he disses her
33:42
for that. Yes, that's awesome.
33:45
I kind of want to to, you know, one of the things that is coming
33:49
up as a theme in this conversation, is it clearly,
33:52
there are a lot of great people that you had the privilege of
33:54
working with being taught by teaching.
33:57
And one of those great places that you're at now is Stanford
34:01
and Stanford, consistently has an outstanding sort of machine
34:05
learning profiles. So many experts are there in
34:08
our, in our notes. One of the things that Demetrius
34:10
and I noted was it's almost like the YC of machine learning where
34:14
it's like, consistently putting out in Seeing ideas.
34:17
We've had actually some of your PhD students on our podcast.
34:20
Like Cody, Cody Coleman who was on an earlier episode and I kind
34:24
of just want to ask, you know, from your standpoint.
34:26
What is it? That is so unique about the
34:28
environment and the culture of Stanford and the computer
34:31
science department, that allows this sort of flourishing to
34:34
occur. Yeah, I mean, I think it's a,
34:36
it's a, it's a great department overall, like, you know, great
34:39
set of faculty and it's going that reputation over time.
34:43
So it's able to attract a lot of great folks too.
34:46
To to add, you know, as it's changing.
34:48
Of course, one really unique thing about it is that it is so
34:52
close to Silicon Valley. So, you know, a lot of people
34:55
from industry will, you know, will come, visit and Doctor
34:59
people. So it's like, it's like super
35:01
easy. Like it even I mean, you know,
35:05
in other universities, like I would reach out to two people at
35:08
companies and say, hey can I can I come meet you and like present
35:12
about like this research project or whatever?
35:14
And they'd say, yeah, you know, come down in this.
35:16
They're like, oh yeah, I'd love to hear about it.
35:18
Actually. I'll come down to campus Because
35:20
you know, it's a nice place. I want to like, let's have lunch
35:23
now. That's have coffee.
35:24
So it's just so convenient for that.
35:27
And so, as a student there, you can easily hear from, you know,
35:32
people are huge companies that are doing stuff.
35:35
You can hear from other people from, you know, from people at
35:38
startups you've got a lot of venture capitalists nearby who
35:42
you know, my sense is like they will called call.
35:45
Like I'm ever Every student and you know ask them what they're
35:48
doing all the time, it's one of the things that they're trying
35:51
to do so. So it's just easy to get a lot
35:54
of perspectives and I also think the network of you know alumnus
35:58
I and and people like otherwise associated with it is is really
36:02
great. So yeah it's just I think it's
36:04
a, you know, it started out, you know, forming this community and
36:08
it's also physically like very convenient for everyone to go
36:12
and like, you know, to go chat with you about.
36:16
Of that makes a ton of sense. I mean, that campus is like
36:20
gorgeous and anybody would want to spend a lot of time there on
36:23
a personal note. Did you ever see things that
36:26
you, you know, going into your shorter, like PhD or going into
36:30
your sort of research Journey that you would be, you know,
36:33
sort of so translational and your career, right?
36:35
You'd start a company and that company would kind of spin out.
36:38
I like all these different products like dick was that
36:40
something that you always envisioned or is that something
36:42
that maybe you saw more of at Stanford given the culture.
36:45
Yeah. When I went It into my PhD.
36:47
I wasn't necessarily thinking to to start a company or to be a
36:52
professor. Really I just you know, I wanted
36:54
to like I said it's a great opportunity to you know learn
36:57
about cool things and explore you know some some new things
37:01
for a while you can you can kind of work on whatever you want for
37:05
like five or six years but I wasn't really sure what I want
37:08
to do. After I just wanted to like find
37:10
you know, exciting things to work on, you know, and hopefully
37:13
like have some impacts to them, you know, we At Berkeley like we
37:17
started, you know, we were, we were all at Berkeley, there were
37:20
some people, you know, like launching companies actually
37:24
both my my advisors have done companies before that were, you
37:29
know, that were successful in various ways.
37:32
They got a bunch of funding Scotch drinkers company.
37:36
Nasira was was acquired by VMware to do their virtual
37:39
Network infrastructure. So, so, you know, they had some
37:43
connections to that world, but it wasn't like, Is it wasn't
37:47
like everyone's looking around to build a company or they were
37:50
just interested in what are cool, you know, technology
37:54
changes that are happening. What are, what are opportunities
37:57
to do something interesting and also like what would actually be
38:00
useful to people. But yeah, I happened to pick,
38:04
you know, an a that you know, made a lot of sense to focus on
38:08
which was this large-scale data intensive Computing, all the
38:12
trends were aligned for this to become a really big thing and
38:15
you know what? The big things missing for it,
38:18
it was easier, programming interface as I'd so everyone
38:22
thought, I'd only that companies can build this, but it turned
38:25
out you can build better interfaces and and we did,
38:28
right? And so, within a few years, we
38:30
had like, you know, basically anyone who can, you know, who
38:33
knows even a little bit of programming and maybe their main
38:37
job is something else can use some python, or some are or some
38:41
SQL with spark and, and on computation on like, hundreds of
38:45
servers. And you know, people were very
38:47
excited about that and about sort of democratizing access to
38:51
these things. And so it became even bigger,
38:55
you know, potential set of people who would work with it.
38:59
Yeah, and I think again with the company, you know, and when we,
39:02
when I started the project, I wasn't like, said on, you know,
39:05
figuring out something that would be good for a company, but
39:08
we saw, there's this operate, we did all of us on the founding
39:12
team cared a lot about, you know, real world impact and like
39:16
help. People do new things and we saw
39:20
we thought, hey, the best way to do this is to actually like, you
39:23
know, we start a company around this and like really be able to
39:26
fund it and learn from the most serious users.
39:29
You know, not not just like smaller groups that were talk to
39:32
us but like you know, go after the hardest ones and figure out
39:36
like what it takes to help them do things with data and machine
39:39
learning got it. I have a question that just came
39:43
to me. It's not on our pre plans Set of
39:46
questions for what is the one thing that you emphasize or one
39:52
piece of advice that you give your grad students or to the
39:56
people that you Mentor, that others don't or don't enough?
40:01
Like what's the one thing that you uniquely emphasize?
40:04
Hmm, that's a good question. I don't know because because
40:07
faculty tend to say a lot of things so I guess quite possible
40:10
that anything I said someone has said, I think Yeah.
40:18
I mean one thing is like really, you know, put yourself in the in
40:23
the shoes of like a user of the technology or of your system and
40:27
like, really try to understand what they're thinking about and
40:31
you know what their challenges are.
40:33
So, so I've told a lot of students, you know, if you
40:37
really want to understand like this type of thing you're
40:39
working on, like, whatever whatever it is.
40:41
I like maybe it's like, you know, Vector databases or like,
40:44
you know, like we parallel do. I'm learning like something like
40:48
that. Going like try to find someone
40:51
who you know who needs to do that thing as their job and like
40:54
see what the challenges are and there might be like totally new
40:58
things you hadn't thought about and that that no one in the
41:02
research Community thinks about and their same thing applies
41:05
outside research, right? If you don't you know, if you're
41:08
developing a product but you don't like fully understand.
41:11
You know what it is to use it or like what actual things are
41:14
people trying to use that for you.
41:17
You'll miss out on some opportunities.
41:20
So I think that that ability to that opportunity to like see the
41:25
world from a different perspective and then go back to
41:28
your perspective and like you know use all the knowledge you
41:31
have and all the skills you have is is important.
41:35
So basically just invite people to play Rock Band for a little
41:39
bit and see if they're open to telling you what's on their
41:43
mind. Yeah, that doesn't work.
41:45
Yeah. Yeah, we're just do that thing
41:48
hands on my go, you know, you're developing a, you know, like a
41:52
web server that you think is like faster or whatever like or
41:55
more, reliable, go build some websites and see you know what
41:59
happens when you do that or whatever it is.
42:01
So you hinted at this earlier that you have a few things that
42:05
you're excited about one is on the data brick side and the
42:08
other is with the research that you're doing.
42:10
Can you go into a bit more of that?
42:12
Yeah, I mean, we've been doing a bunch of different things in my
42:15
go, but one thing, I'm really excited about where we also have
42:18
some an open source project is basically building, you know,
42:23
complex applications using large language models.
42:26
So using language models as as a building block in a more
42:31
sophisticated by blind and we have this, this project DSP,
42:36
demonstrate search predict that lets you build applications out
42:40
of LMS plus other systems, that return text and the Big one were
42:47
looking at is storage and retrieval systems so you can do
42:50
things like you can build a, you know, a question answering
42:54
system that when you ask a question about say, Wikipedia
42:58
and it does multiple searches over, I'd like, as it reads it
43:01
each article and then searches for other ones and then it
43:04
returns an answer to your question.
43:06
And basically, the reason I'm excited about it is you don't
43:10
normally, like if you just use lme's on their own they are you
43:15
know, they They often like can understand a bunch of
43:18
instructions and produce pretty good outputs, but it's hard to
43:21
control the quality and if you got something that's only like
43:25
70% to 80% of the time working and the rest, it's not, it's not
43:30
great for a lot of applications is good for some things but not
43:33
for everything. But in this project we're
43:37
putting the llm sort of in in a box where like we tried to give
43:42
it a very specific task. And we also have ways of No in
43:47
improving, reliability, around that, like ways of, you know,
43:51
generate finding all the mistakes that did and then
43:53
fine-tuning it on to fix those or calling at multiple times or
43:58
calling multiple models or whatever.
44:00
And so, it's just one building block in this pipeline.
44:03
You create that overall can be, can be more reliable and can do
44:07
things that Dell LM alone, couldn't and building an app by,
44:12
you know, programming this kind of Pipeline and calling existing
44:15
LMS is just so much easier and less expensive than trying to
44:19
drain one from scratch and then like realizing okay now it's 90%
44:23
correct but it's still dark. Not quite correct when I spent
44:26
like you know huge amount of time training it so yeah I'm
44:29
excited. It's a pojo like let's let's use
44:32
it as just one building block in a bigger application.
44:36
I like to say that lme's hallucinate more than your
44:39
average college student and so the Thing that I'm really
44:47
interested in. I mean, it sounds like you don't
44:49
want to go down the path of training.
44:52
Your own large language model and having that for data bricks,
44:57
as like a service, how do you see large language models in?
45:01
Like the new Revolution? That's happening right now in AI
45:05
fitting in with the work that you're doing at data brick.
45:08
Yeah, I know. Don't get me wrong.
45:09
I think it is super important to be able to train your own model.
45:13
And this is definitely something that we do.
45:15
We help people Oil train them but I also think when you're
45:19
building an application like training is a pretty is a useful
45:25
tool. But it's a pretty like, you
45:27
know, course, sort of tool wear like you it takes a lot of time
45:32
to use it and then if it doesn't work out like, you know, it's
45:35
really hard to improve the quality beyond their.
45:38
So if you I do think a lot of that's involved in getting you
45:43
know, great applications of these Involves you know other
45:47
pieces after like a good example.
45:49
Does this blog post where someone like reverse Engineers
45:53
basically the API calls into into GitHub copilot?
45:57
And they point out that, you know, what each time you call
46:01
copilot you actually send a lot of information about the context
46:06
in your program. Like what all the file names?
46:08
What? All the classes, what are some
46:10
functions and things here? And it's strange to take that
46:14
context and then produce a completion.
46:16
Based on that. You basically you can do a lot
46:19
more like and two ends to improve an application and this
46:22
is where the ml Ops come in. You know, you might think like
46:25
weight without lme's. You don't need ml Ops anymore.
46:28
You just need to like talk to it but I think you are going to
46:31
need like llm Ops basically or whatever you want to LM L op.
46:36
So whatever you want to call it, that, you know, let you use this
46:41
within a bigger application and experiment with the whole
46:45
pipeline of Stuff in and out of it.
46:47
So I just think it's important to think about both of these.
46:51
Yeah yeah this is fastened and you said something there.
46:53
The is so crucial that I hadn't thought about.
46:57
It's like you don't really this, you don't get two chances or I
47:02
mean, you can have infinite chances if you have infinite
47:05
resources and in cash to burn you can just keep creating large
47:10
language models. But like you said the output or
47:14
the large language model that you created.
47:16
Wait. It's what you get and then to
47:18
make it better. If it's not that good of the
47:21
base model or whatever to make it better is really hard.
47:25
Yeah. And so it had something that I
47:27
hadn't even thought about, as far as the ones who are
47:32
training, those models it is a bit of a risk and really want to
47:36
make sure there's so many things that can go wrong when it comes
47:39
to training those. I mean yeah.
47:42
Yeah. So it's as like, you know, MLA
47:44
research, I'm really excited. It about these models that are
47:47
very powerful out-of-the-box. But as an engineer thinking of
47:50
like, okay, if I wanted to hook the like, you know, improve my
47:54
startup. So I never liked conversion rate
47:56
or something with these things, as an engineer.
47:58
I want to have, you know, way more levers, like way more tools
48:01
so I can, like, really control. What's what's happening with
48:04
them? Exactly, exactly.
48:07
So right now, if you were out there and you were going to
48:11
start a company, What kind of company would you start?
48:15
Hey, yeah. It's a great question.
48:18
I think, I mean, there's a lot of excitement about machine
48:22
learning now so it only more than, you know, I mean, there's
48:25
always been a lot with deep learning but I think especially
48:28
now with whatever and being able to try out like Chad g b d and
48:32
you know talk to it is kind of funny but that like very simple
48:37
change in programming interface from like a little text box on
48:42
in Stark GPD too. To, you know, a chat thing, like
48:45
really got a lot of people to think about it.
48:47
So I mean, I think The the yeah I actually think you know, one
48:56
of the most exciting things is like looking at application
48:59
areas where these models can help and then developing
49:02
something specialized for that. And then getting this sort of
49:06
flywheel of, you know, as you build an app.
49:09
Like the first step that uses LMS to do whatever, like, you
49:13
know, whatever AI you're working on.
49:15
You got a lot of data and experience with that.
49:18
Then makes it hard for others to catch up.
49:21
Because, you know, when you, when you start a company, you
49:23
always want to have like something that you learn over
49:26
time, some Advantage you get where, like, you know, over time
49:29
and becomes more valuable. And I think ml provides this
49:33
ability to develop, like, if you were the first to do a certain
49:36
thing and you executed it well it will just get better, you
49:39
know, faster than someone else who can get started.
49:42
So that's the kind of space I would look for.
49:44
And I would assume that the technology around these things
49:48
is pretty fluid and they'll be new.
49:51
New types of Technology coming out.
49:53
But, you know, there are other things you can do to if you can
49:55
build like a better GPU, like, you know, go for it.
49:58
You know, that's, I think a ton of people would want that, you
50:03
know, there are many infrastructure things.
50:05
You can also build one question. I had hearing you talk about the
50:10
LMS and you know what, challenges or opportunities.
50:13
They pose is really about the institutions that have created
50:16
these lme's, right? You know, the Google research is
50:19
the open, my eyes and Go brains, whatever.
50:22
You know, we have seen over the last 10 to 15 years, kind of an
50:26
interesting emergence of corporate research Labs.
50:30
You know, that have tremendous amount of talent and are really
50:33
developing, sort of cutting-edge products and research, and they
50:37
have a symbiotic sort of interesting relationship with
50:40
the university departments, right?
50:42
Which have traditionally been more of the bastions of
50:44
research, you know, from your standpoint.
50:46
How do you think about the emergence of those corporate
50:49
research Labs, you know? No and how they are shaping the
50:53
future of a I like where do you see that going over the next
50:56
five to ten years from your standpoint is that sort of a
50:59
productive model where research or how do you think about that?
51:03
Yeah, so it's very interesting. I think what's happened recently
51:08
is that people discover that there are some ml approaches
51:13
where you can, so in scale at it, so in lots of resources,
51:18
lots of data and actually got better is also They went to, you
51:22
know, there's they're just trying to throw in a lot of that
51:26
and and do these large-scale experiments and build these
51:28
Large Scale Models. But of course, it's less
51:31
accessible to everyone else and one of the things that had made
51:35
machine learning and deep learning progress so quickly why
51:38
is that many people could you know could work on these
51:42
problems and there were like you know, all these I tens of
51:44
thousands of like, you know, researchers at universities and
51:48
who you know who are trying. About new things and then
51:52
someone finds the thing that works and everyone Builds on it.
51:54
So I think that's actually maybe going to slow down with these
51:58
big companies, which is a bit of a problem for the field but I
52:02
think it also means that if someone discovers a much cheaper
52:07
way to do some of these things. Like all the research Focus will
52:10
fix will shift that because more people are able to play around
52:15
with their dried. So so I think, you know, there
52:18
is a possibility that like will will end up It's a new way of
52:21
doing the staffs that is that it's cheaper.
52:25
I'll just mention one things with the LM specifically,
52:28
because if you look at, if you compare them with computer
52:31
vision, it's pretty different in computer vision.
52:34
You don't need like a huge data center to drain a very good
52:38
model. The kind of, I mean, of course
52:40
they get better with more parameters and stuff, but we
52:42
there the quite good with, you know, a reasonable size and I
52:47
think with lme's part of the reason you need that is because
52:50
the model It encodes and sort of memorizes whole lot of knowledge
52:55
about the world, right? So like when you talk to Chad
52:57
GPD at can answer topics and all kinds of random stuff like every
53:01
sports team, you know, like every video game like everything
53:04
that's been written up somewhere, but it has to like
53:07
remember all that info. And so even though it's large a
53:11
lot of the parameters are just hosting that.
53:13
So I actually think it's quite possible that you can decouple
53:18
like the knowledge storage from The sort of like language
53:23
processing or like Computing that's happening and end up with
53:26
a small model. If you take, if you have a
53:28
smaller, neural network, plus the ability to create just a
53:32
search index, basically like a database or something with your
53:35
knowledge, you might be able to produce equally good answers and
53:38
that's one of the things that like my group has been working
53:42
on of, you know, is to try to make this happen at least for a
53:45
specific task. So I would say it's not crazy to
53:50
imagine a world. Worth you know much smaller
53:52
models plus you know, search or like basically memory lookups
53:56
instead of let's unlock as a floating Point operations, that
54:01
makes so much sense to me because also again, going back
54:04
to the operational overhead and the resource overhead that you
54:09
need for these large language models, if you can do it with a
54:11
smaller model and just have that context there for it when it
54:15
needs it. Yeah, it seems like it's a much
54:18
more obvious play. Yeah, exactly.
54:21
The other aspect that's like really important for building a
54:24
system. Is if you separate these out,
54:26
you can also update the knowledge, very cheaply.
54:28
You don't have to retrain. Like like a great example is you
54:33
know, like let's say the model memorize that the president of
54:36
the u.s. is like Joe Biden. But then after the election it
54:39
someone else you don't have to wait, like sea monster.
54:42
He trainer you just go, you know, upload the latest, you
54:46
know, Wikipedia article on that thing and it finds it so well.
54:50
Yeah. Yeah, that is killer and that
54:52
makes complete sense. So last question I've got for
54:55
you man. When you look around and you
54:59
look at the ecosystem, what are some companies that tend to stay
55:04
on your mind or and or you greatly respect?
55:09
Yeah, there are a lot of great companies.
55:11
And in this space I would say like, you know, obviously and
55:16
you know in the past few years, I could opening, I has had a
55:20
huge amount of impact on, you know, getting more developers to
55:24
think about using deep learning so So that's been cool to see
55:28
and I think, you know, whether like some people they say there
55:32
is a, it was sort of unintentional that gvd would
55:35
become so popular and so on and probably exceeded the
55:39
expectations. But they've done, you know, a
55:42
great amount of stuff to get a lot of people to drive these
55:46
approaches. So so that's been pretty cool.
55:49
I mean, I think it's also great to see a lot of ml Ops and ml
55:54
infrastructure companies that are successful.
55:56
Well, like like text on, for example, who, you know, is it
56:00
has really invested in, you know, in features stories and
56:04
you know the many companies and the monitoring space, many
56:07
companies in the space of like robust machine learning.
56:12
So it's it's cool to see those and I would also say probably
56:17
infrastructure probably like basically data labeling, an
56:23
acquisition companies like Label Box and scaly.
56:25
I have also enabled A lot of the ecosystem.
56:28
I'm trying to look for ones who are like, you know, people are,
56:31
you know, thinking of things basically very differently
56:34
because of them. And I think things like open
56:36
a.i. and and label box and scale.
56:38
Just make it easier for more teams to say.
56:40
Yeah. You know, I'm going to try doing
56:42
this, you know, potentially like expensive thing of getting a lot
56:45
of people to look at something and figuring out how to manage
56:49
that. Do you still feel like there's
56:50
room for innovation in the ml up space?
56:53
I think there is. Yeah, I think especially Well,
56:57
because the the ml applications are changing.
57:00
So the algorithms are changing so quickly.
57:02
I think there is, I think it's still, you know, kind of
57:06
complicated you have to hook together a bunch of different
57:09
things. And if someone figures out a
57:11
better way to do, at least, I express, if I Cap locations, I
57:14
think it will help. So I think all these spacers,
57:16
like data is similar. I think they can get simpler
57:20
over time, and if people figure out how to do that, like that,
57:24
it'll be. If you make it simpler, you also
57:26
make it So we'll do a much larger set of users.
57:30
So you actually have a ton of new people that basically expand
57:35
the market. So I think that's going to
57:37
happen. Yeah, that's a what an absolute
57:40
pleasure, man. Thank you so much for coming on
57:42
here. I think this is a perfect place
57:44
to end it. Great.
57:45
Thanks so much for having me Pop's.
57:50
You have to immerse yourself in the amylose content.
57:54
The best way to do it is Is to subscribe to the email Ops
57:58
Community podcast soon. Good luck and keep learning.