Latest / Agentic Conversations (formally mlops.community) / ML Scalability Challenges // Waleed Kadous // MLOps Podcast # 154

Transcript

ML Scalability Challenges // Waleed Kadous // MLOps Podcast # 154 · Agentic Conversations (formally mlops.community)

0:00
Yeah, my name is Mallika this. I'm the head of engineering here
0:03
at any scale. A company that is the behind the
0:06
open source project Drake and believe it or not.
0:09
I really don't drink coffee. I maybe I'm the last person in
0:12
Silicon Valley, who doesn't, but I'm much more of a cold drinks
0:15
type of person. And I try not to use anything.
0:19
That's going to clap my mind. Welcome back to the envelopes
0:23
Community. Podcast everyone.
0:25
I am your host Demetrios. And today, I'm joined by none
0:29
other than Than Abby, what's happening?
0:31
A be a lot of stuff. Quite a lot of interesting stuff
0:37
so much good stuff, right? We just got off of a
0:40
conversation with my doctor. I'm not even gonna say mister
0:44
because this man is a certified doctor while lead caduceus and
0:49
he is leading engineering at any scale.
0:51
If you all do not know him, he's one of my favorite people to
0:55
talk to. I've had the joy of introducing
1:00
Him two times before for the apply conference and both times
1:04
that he gave talks, I was blown away.
1:06
So this is one of those rare moments where we got to have him
1:09
on the podcast and he just talked about.
1:11
I mean, for me it was non-stop scalability.
1:15
Coming back to. Okay.
1:16
Scalability is a really hard problem.
1:19
We need to solve it and it's not as hard as it was 20 years ago.
1:23
I mean, he's got a very decorated career, he was working
1:27
at Google, he then went to Uber and now, Now he is heading up
1:31
engineering at any scale. Abby, what were some of your key
1:33
takeaways? The first power is where he is
1:36
calculating the cost very quickly.
1:39
I couldn't wrap my head around other that's numbers which is
1:43
just amount of compute that we are now looking at with all of
1:48
these large language models for both training as well as infants
1:52
and their unit economics of all of that time.
1:56
So I thought that was really fascinating the part where he
2:00
Even if you're able to get a device of this capacity, he
2:06
honestly cannot do it. Even if Amazon gives you like
2:09
the who once you read to, yeah, it 100.
2:12
She's still need for a 100's and even at that rate, you're going
2:16
to need to link those machines up together and it is no easy
2:19
feat. This is something that we've had
2:21
the folks from run a, I come on here and talk about this problem
2:25
to it is not a trivial problem when it comes to the scale.
2:29
Al and I find the big question here in my mind to be what I
2:35
asked at the end where it's like, we see a very, very strong
2:39
movement towards more data and more parameters and just bigger
2:43
models, right? And then I see on the other
2:45
hand, where it feels like, there are data engineering folks that
2:50
are trying to say that big data is dead and we need to think
2:54
about this small data again. And so I think those are
2:58
juxtaposing Or I see them as being to juxtaposing views and I
3:05
find it fascinating. I liked to hear while leads
3:08
commentary on that because I asked him that very same
3:11
question and of course I think we all know where he lands on
3:15
that. He's working at any scale and
3:16
they're very much trying to figure out scale.
3:19
There were so many good parts to this too though, if you will
3:23
allow me to mention the moment where we got into, just the vide
3:28
gritty of the models. And how what he was blown away
3:32
by was how it just the attention is all you need paper blew
3:36
everything out of the water and all of a sudden, everything that
3:39
he had been playing with back in 2013, 2014, it was no longer
3:43
relevant. He's been through the trenches
3:47
when it comes to machine learning and ml Ops.
3:50
Because right now, he's building a tool that basically helps you
3:54
take away those ml Ops burdens. So we got to hear what Rays
3:58
doing and who their services. We also got to hear a little bit
4:01
about any skill and I think it's worth us jumping right into it.
4:06
Abby, I know you got a few more things to say, though.
4:09
Before we do I was gonna put you on the spot now and say, do you
4:13
have any word reviews for today? I didn't but hold on, I can pull
4:17
some up, this is awesome. That's a great idea.
4:20
We definitely needed to read out more reviews and give me just a
4:25
second while I pull this up, let's see how fast my internet
4:27
goes. It's always great when people
4:29
write At us reviews, especially because we like the funny ones,
4:34
we like the ridiculous ones and we like the ones that kind of
4:39
talk, a little bit of shit. Okay, so here we go.
4:43
Here is a recent review, interesting discussions covering
4:48
rapidly developing field. I'm a senior ml engineer deals
4:52
with a lot of ml Ops related items.
4:54
Since we don't have a dedicated role for that on our team, sorry
4:58
to hear, I feel for you this This show and the slack
5:01
Community, join that if you're not in it and the newsletter, if
5:04
you are not subscribed to that, have been great resources for
5:07
inspiration and staying up to date, on the constant evolution
5:12
of tools and best practices. It's very useful to hear from
5:16
other practitioners as well as to navigate this landscape
5:19
together. Huge shout-out to Senor.
5:26
Go, Alec GK, I think that's the same.
5:29
Awesome. We appreciate all of the reviews
5:32
and we would love it. If you all leave one, it would
5:35
mean the world to us. And if you leave one and say
5:38
something funny, it's even better when we read it out.
5:40
Alright, that's all we got. Let's jump into this
5:43
conversation with Wally need Walid.
5:53
I'm going to come out the gates hot and I gotta know what's up
5:59
with any skill. Not sponsoring the ml Ops
6:01
Community, where we at man? Really, that's how you're gonna
6:04
stop this. We're gonna take this to me.
6:05
Curious of all the places in the universe.
6:08
Look at this, you know what? I don't even have any schwag
6:11
from any scale. I have some arise Cup right
6:15
here. They're sponsoring the community
6:16
thinks I got like a tecton backpack.
6:19
I've got can fix you in a be up there.
6:21
Worry, there we go. At least some of Swag, I would
6:24
think that that hundred million in funding could go at least
6:27
towards them. A lobster immunity just a little
6:30
bit but you know, we're going to be getting the message to meet
6:38
again. There we go.
6:39
See if we can before I get the feeling that you didn't think
6:43
you were actually going to come on here.
6:44
Did you you thought I was playing?
6:46
You know, that? That's really excited about
6:50
coming in here. I mean you and I have talked at
6:51
apply twice and You know, I was really excited and actually,
6:55
like, I think Jules when like, a developer relations person, kind
7:00
of reached out and said, hey, this guy called Demetrius is
7:02
after you definitely. And I'm like, yeah, I kind of
7:07
reached out and, you know, they could for us to chat.
7:10
There we go. Man, I know I've been putting it
7:12
off because one thing or another.
7:14
And finally, we are here, and I'm so excited.
7:17
Only, you get to come on here. Only you Walid?
7:20
That I have so much respect for. And I am so thankful.
7:23
Thankful that you are coming on here.
7:25
I mean, you've you've got a very decorated career as we mentioned
7:28
in the intro and you are just like one of those guys that I
7:32
look up to in the mL of space because of the time that you've
7:36
had in the game, maybe it is worth us talking about.
7:40
You started at Google. Can you give us a bit of
7:42
background of what you were doing there or maybe even before
7:45
that when you were doing your studies?
7:48
Yeah, yeah. So back in Australia, where I'm
7:50
from I was doing my PhD and you know, His machine learning in
7:54
the mid-90s, you know, back then it was what was called the
7:57
second AI winter. You know, machine learning was
7:59
just kind of this tiny little corner of wider.
8:03
Artificial intelligence, think not blossoming.
8:05
Like now, I remember it being a nice email, 2002 and had like,
8:09
300 people there. And now you hear about things,
8:11
like, new reps, or those types of conferences that have like,
8:14
6,000, 7,000 people, huge. So the lot smaller than and back
8:18
then, it was like the older more information theoretic based
8:21
approaches like decision trees and that type.
8:23
Type of stuff they were just coming on.
8:24
This was the dawn of like Bayesian staff and support
8:27
Vector machines which I now have gone out of fashion but still
8:30
have their uses. I still remember in 2005
8:32
actually I was working around attended Europe's and the
8:37
funniest thing about new rips was there was almost no neural
8:40
papers are actually more biological neuron papers than
8:42
they were like artificial neural networks.
8:45
So it's funny how things turn and switching history.
8:47
But, you know there's also kind of recurring themes.
8:50
Like I remember I was doing the experience when PA Dan was doing
8:53
hyperparameters tuning, and it was just taking too long, so I
8:57
did the math on. How long will it take for me to
8:58
finish my phds and decision trees applied to claim series?
9:02
It was like, I was going to take six months of experiments.
9:04
So I just kind of said, you know what, I'm gonna hack together 17
9:07
boxes and kind of write some really happy at the time.
9:11
Perl scripts to kind of distribute the workloads across
9:14
their 17 machines. And yet, it turned out, other
9:16
people found those machines useful.
9:18
And I think it was like five or six phds built on that little at
9:21
the time, what we used to call Beowulf Clusters.
9:24
Yeah to do the processing and then you know I did a couple of
9:27
postdocs we didn't those postdocs.
9:29
I focused on again machine learning for speech and then for
9:32
robots. So using computer vision to
9:35
recognize victims in disaster sites.
9:37
It took me a while. It took me like 16 years, but I
9:39
worked out that I wasn't a very good academic, and so I kind of
9:43
decided it was time to go work somewhere else.
9:46
So I ended up joining Google and working on maps, and that was a
9:50
really, really fascinating experience.
9:51
Again, you know, it's kind of crazy.
9:53
Start to see some some very, very recurring trends.
9:56
Like so I've been working on processing GPS traces for a
10:00
small area and then a Lois, I'm not have to do it for like the 8
10:03
million miles of the United States of America.
10:06
And so you have to scale up. This tiny little algorithm that
10:08
you wrote for like one block or two blocks to like nine million
10:12
square blocks and we started to apply some machine learning
10:14
there. Honestly, you know, I had a
10:16
little bit of battle scouring from doing my PhD, so I didn't
10:20
even want to touch machine learning for 2000.
10:22
And You know, for like a decade after I did it, but then I
10:26
started to really come back and I ended up managing a computer
10:29
vision team based in Pittsburgh actually, and, you know, they
10:32
just changed them. They were an acquisition and
10:34
doing some really cool stuff with computer vision.
10:36
But back then it was the old-fashioned stuff of like,
10:39
what's the distance between the eyes and cannibal the classified
10:42
one day? I'm talking to them.
10:43
It says what's up with this neural stuff?
10:45
This is a back in 2012 and it's like, oh man.
10:49
This thing is for real. I said, you know, I was very
10:50
skeptical at the time I was like Man, this is like the third
10:54
time. When is it going to work this
10:55
time? And then they started to tell
10:56
me, dude, it's getting super human results.
10:58
There's old algorithms were working on and there's are
11:02
cutting it anymore, compared really planning based approaches
11:05
until she was really funny to me the way things come and go.
11:08
You know, at the time, I couldn't believe it.
11:10
Deep learning the third time was the time.
11:12
So by now, it's about 2012, 2013.
11:16
And you start to see things like the alphago victory in 2016 in
11:20
New York. A, this is for real and that
11:22
point I moved to Ubah and again at the very Forefront of the
11:26
machine learning Revolution and I was kind of like the Elder
11:28
Statesman of Rim like machine learning a trooper for various
11:31
reason. I ended up not being part of
11:33
machine learning group that kind of advocated across the entire
11:37
organization across maps. And everyone else to really have
11:40
you, but take Cutting Edge, machine learning like deep
11:43
learning, seriously, and clothes from Acquisitions of some.
11:45
Some really interesting companies and then about two
11:48
years ago, funnily enough, I joined any scale in a scales
11:51
company behind the open source. I called re and re funnily
11:55
enough it's like closing the loop.
11:56
I was going back to my PhD again where I was hacking those 17
12:00
boxes together and going okay let me try to do what I did in
12:04
my PhD and you know I knew I was onto something.
12:06
When the fans of my love my laptop started spinning up
12:10
because it was actually using all of the hcp used and actually
12:13
making machine get hot. It's like this Tech is pretty
12:16
ruled. Cool.
12:17
Then when I pointed it to actually run on like 60
12:21
machines, it was, you know, Within 50 percent of theoretical
12:24
maximum, after three hours of hacking and it got, this
12:27
technology is really cool. First of all, it's been, it's
12:29
been crazy to see the Revolutions in machine learning
12:33
over that time. But also some of the recurring
12:35
themes like scalability. But, you know, looking at what's
12:38
happening. Most recently with our lens.
12:40
I've never seen anything like this before in those in those 30
12:43
years, right? I haven't seen us cross over,
12:45
you know, machine learning cross over into the mainstream, the
12:48
way it has most recently, but the interesting thing again with
12:52
LMS and Lisa super huge, you know, 175 billion parameter
12:56
models is It's just it's going to be a scalability problem
13:01
again and forgive me. Like, it's just kind of like
13:03
this. This recurring weep of, like we
13:05
get more computational power and then we use it to do even
13:09
crazier stuff, which needs more computational power.
13:12
And it's just like, it's fascinating to see these Loops
13:14
go on and on in history. So quick, one for you just about
13:17
re that I was thinking about before we started and we hit
13:21
record did a video invest in Ray or was it, how is that?
13:26
It might just making that up, I'm not sure.
13:27
ER, I don't think well I'm not sure I can get back to you on
13:30
that. My, my focus is on the
13:32
technology, not the business side of things.
13:34
Yeah, that's hence. Why we haven't gotten sponsored?
13:37
Because if it was up to you I imagine he would have been
13:40
already done. So dear.
13:42
Dan you got? You gonna get stuck on this
13:44
again. Images will cut that out, don't
13:47
worry. So anyway, you're gonna have to
13:53
pull out your credit card and tap the screen real fast and So
14:00
no getting back to this, incredible career that you've
14:02
had and what you've done. I mean you went from doing
14:07
machine learning in University where I imagine you could read
14:11
every single paper that came out related to machine learning and
14:16
I were very up on the times and then you hit this certain
14:21
scaling issue and then you went to maps and you saw oh my God
14:24
there's a scaling issue. Again I have to take something
14:27
that My programmed for a few blocks and bring it up to
14:30
something that now is going to power a lot of people's
14:35
directions and then go into Uber and being like, you know what?
14:39
We need this, we need deep learning.
14:41
There's so many interesting questions that I have in funny
14:44
enough last week, we were talking to some of the team from
14:47
Uber, who is in charge of the ml education at Uber and it seems
14:54
like it's been very instilled in the culture.
14:57
Which I imagine was thanks to your doing in part.
15:01
One of the things that I think about is what are some of those
15:03
use cases that you were thinking about with deep learning back in
15:07
those days. So I think it really extends to
15:09
everything, you know, back in those days.
15:11
It was also trying to collect imagery of particular places in
15:14
cities so we could have like, super accurate Maps.
15:17
We using deep learning to improve location quality and we
15:20
even implemented crazy features like accident detection.
15:23
So we would look at the sensors on the phone and see if we saw
15:26
like you know, a cool Asian or saw something that looked like a
15:29
collision and then we would kind of have a call center, is call
15:31
someone and say, hey, were you in an accident?
15:33
Is everything? Okay, so, you know, we were
15:36
using it from everything, from census to location, Improvement,
15:39
to trying to extract information from imagery.
15:42
It was, it was, it was very much.
15:43
And that before you start to talk about things, like we need
15:46
to predict etas across the entire country, how does that
15:49
work? And, you know, we kind of have
15:50
to do crazy stuff to do with scalability.
15:52
Like we had a separate model for each City in terms of the The
15:58
ETA estimation. And so, you ended up with, like,
16:00
seven hundred models that you had to manage one per City for
16:03
the correction and stuff like that.
16:05
And so again, the same kind of things came up, in terms of
16:07
scalability mean, just the huge amount of data that is used and
16:13
it was real world data in a very, very privacy aware way.
16:16
You know, everyone was opting in, we were very transparent
16:19
about it but just the real-time nature of it as well.
16:22
And the funny thing with they were is like, you.
16:23
Look at how many companies have come out as we were at that
16:25
time, you know, in terms of companies, He's like tecton and
16:30
and it was right. You guys like the database's
16:32
company like Carniceria came out of them.
16:34
Cadence came out of them. Already baseball came out of
16:37
that. Pretty?
16:38
Yeah, yeah. So it was really, it was kind of
16:43
weird to be living in that zone where there were all of these
16:46
cutting-edge things to do with real-time data processing, to
16:50
do, with dealing, with complex systems.
16:53
Even some of the crazy stuff we did with like monitoring our
16:56
systems to notice anomaly. Detection is a really good bro.
16:59
A blog post about a system called Argus that a colleague of
17:03
mine friend Val worked on. There was literally looking at
17:06
millions of signals to look for anomalies.
17:08
It's just kind of a really fun time because what we were was at
17:11
the edge of was historically, most of the startups had been
17:14
purely in the digital space, right?
17:16
A Google is a Google search engine, right?
17:18
And you can search for how to buy a car but you don't actually
17:20
have to interact with a car. As I let you go was distinct
17:23
thrust and there were two characteristics about it.
17:25
There were important. Number one, it was in the
17:27
Physical world and in the physical world, the data is just
17:30
so complicated. And nuanced measures that you
17:34
really have to use something like machine learning to deal
17:36
with it, the huge amount of data.
17:38
And then the second thing was all real time, like you have to
17:40
respond to people like super quickly and all of those kinds
17:43
of things. And as a result, it led to this
17:45
blossoming of opportunities that were from Uber at that time.
17:48
And so, yeah, it was kind of, really easy and interesting to
17:51
see that very quick, question over here because I work and
17:55
very similar to me. So we weren't doing it.
17:57
Cars but we were doing it for like motorbike Riders, which is
18:01
like connecting the same to you position.
18:02
The tea calculating the ETS and you had all of this, the person
18:07
is turning left versus right? And you have to predict their
18:10
driving behavior and all the fun stuff.
18:12
Yeah. But how did you deal with the
18:16
infrastructure challenges were there?
18:18
Because like the company that was working with, at that time,
18:22
it was one of like, Consulting clients.
18:25
We had one basic problem, which I was trying to solve, which was
18:28
the data labeling problem. We had so much to do and not
18:32
enough people to basically be able to annotate the data.
18:36
How did you decide for us? It was very much like we didn't
18:39
really have the budget to go buy an external tool Serene.
18:43
They hired me and we were sort of putting on like a deal.
18:47
How about like point of view decide?
18:52
Let's this solution in-house versus.
18:56
Let's go look at other That are already building solution and
18:59
you. Let's keep our core expertise,
19:01
which has, I don't know, like, the maps and all of that stuff.
19:04
I think at the time, like you gotta remember, it was started
19:06
in like 2009 or something like that.
19:09
Like, you know, and I joined in 2015, I think by then.
19:12
So this was before all of these companies.
19:14
Like I said, you know, it's really these companies are
19:16
spinning out of yerba rather than us going and bringing them
19:20
in so much of the time. It was even if we could buy an
19:22
option, it could buy like some commercial option, it usually
19:26
wasn't good enough. And we Our needs of scale were
19:28
just so much phenomenally high like at any one time.
19:31
They're 23 million drivers on more than 15 million.
19:34
Trips a day, the scale of our data was just something that
19:37
other systems, just could not handle.
19:39
It literally, was that case. So we had, we had no choice.
19:42
We had to roll our own the culturally, but also is kind of
19:45
a like, roll your own culture, but I think even if we didn't
19:48
have that culture, we didn't have much choice.
19:49
This predates things like say Jamaica I remember for years
19:53
into my time at through but I'm like oh they now say to maker
19:56
has okay. That's interesting, the bit late
20:00
and you'll see that, you know, things like Michelangelo came
20:03
out of yerba because we have to build our own infrastructure and
20:07
especially no more recently, it has been switching from like the
20:10
decision trees, you know, XG boost world for things, like ETA
20:13
estimation 2D PTA estimation and that's caused like another
20:17
tectonic shift like how do you coordinate gpus and all that
20:20
kind of things. And ironically so I left boober
20:23
took a bit of a break with a non-profit and then during any
20:26
scale and when I Joined any scale as like wait, you know,
20:30
Uber is one of the top users of Ray and right now converted
20:33
their systems to use re and it is entire machine learning
20:37
infrastructure. Now, runs on top of rate,
20:39
including the things that I was working on before and it's again
20:42
it's because of this need for a massive scale to these companies
20:45
like Reba. And they also are huge
20:48
contributors to Kafka. I think, right there is one of
20:51
the biggest contributors. Yeah, there's some vessel
20:54
interesting architectural choices that were made it about
20:56
not all of them. I would say is correct, but when
20:58
is, you know, kind of like focusing too much on these kind
21:01
of asynchronous, event messaging buses, but that's kind of like
21:04
the engineering me coming out and then I would much more
21:06
prefer in many of those cases to use things like a PC.
21:09
So I think we went a little bit calf, get crazy internally, to
21:12
be honest about it. But at the time, it's very hard,
21:14
it's very easy to kind of start small.
21:16
In find yourself in a mess of scale, without kind of, you
21:20
know, back when you have 10,000 cars kafka's, fine.
21:22
And then all of a sudden, you know, like we're probably doing
21:26
more traffic on capita than any other company in the world.
21:28
I guess, we better contribute to khafre the end and make it more
21:30
scalable and robust. That happened a few times at
21:33
Weber as well. Wait, we adopted some open
21:36
source architecture and I was like, yeah, we're pushing this
21:39
well beyond where it's supposed to be right after and even
21:42
things like the mono repo, we have to innovate again.
21:45
You know, there's so much code at a place like Cuba that, you
21:47
know, we had to push some of the limits of get just to get it to
21:50
scale to the number of Engineers that we want it to work with.
21:55
Dude comes back to scale. Always yeah, schauf cost
22:00
anything One quick question on this, I mean it's a Labour
22:04
office right now because we there's a level of maturity
22:09
which the envelopes stays has achieved in like these previous
22:13
three, four years to the extent that there are so many options
22:18
out there. But for a company that still
22:20
starting and I think we're probably going to see more
22:25
problem areas that we're slowly going into and or without a
22:29
tease of data that we're going in.
22:32
How could any sort of machine, learning data size company that
22:35
is probably starting today have, would you suggest that they
22:40
think about their infrastructure and architecture choices?
22:44
Is it a good idea to look for commercial tools to build
22:49
in-house or to be able to go open source and then sort of
22:53
build it on top of yours. And how do you make that
22:56
decision? Which is, this is a good team,
22:58
we should put everything on this.
23:01
Because I mean, with a lot of open source project, this
23:04
usually happen to maintainers to get tired or they switch jobs.
23:07
And there's new Patisserie putting the effort on to like
23:11
the maintenance of that and it depends a little bit on what
23:13
your company is, what the focus is.
23:15
I mean, the other thing is like, to build a world-class and I
23:18
love seeing is not, not easy either.
23:20
I mean, how many people does it take to build like the team to
23:24
support your infrastructure and all of that kind of stuff
23:27
companies like uber? We're blessed because we had
23:29
like, 50 that we had like Data scientists ml engineers and ml
23:34
infrastructure engineers and each one of those was separate
23:36
Rogue right and you know maybe we had 400 data scientists,
23:41
maybe 200. Engineers and 100 ml
23:43
infrastructure Engineers but not every company can afford to have
23:47
100 ml infrastructure Engineers, right?
23:49
So I think, at this stage, if you're trying to build it from
23:52
scratch is probably the existing systems are not.
23:54
It's probably not worth it in most cases, unless your core
23:57
business is machine learning. Unless your core business is, we
24:00
want to be, you know, if you Want to be snorkel or open a I
24:04
or someone like that. Yeah, makes sense.
24:06
But I think, for most companies that make sense to go off the
24:08
shelf, the tricky things is you want something that's flexible.
24:12
You want something that scales and that's when it gets a little
24:14
bit tricky because, you know, some of the existing Solutions
24:18
out there, they don't usually combine that flexibility and
24:21
ability to scale. Some have one some have the
24:22
other. Right?
24:23
So you know I think that kind of like the least scalable end is,
24:27
you know, you buy like a machine with a chibi using it and then
24:30
you run SK. Learn on it, while you're
24:32
running, like, whatever your favorite algorithms are and I
24:34
can see a certain place and you just have one person who's
24:37
trying to maintain that machine. But obviously that doesn't scale
24:40
at some point. You're gonna have to go to
24:41
multiple machines and if you have to kind of rewrite your
24:44
entire stack, for scalability, at that point, you're going to,
24:48
you're going to have some performance issues.
24:50
And the reality is most people hit those kind of scale
24:52
limitations faster than they think they will.
24:55
And that there are very, very big limits to kind of that
24:58
vertical scaling of, you know, let's just buy bigger machines.
25:01
It's also not very cost effective and doesn't work with
25:04
some workloads like like serving right?
25:06
Like you can't have a big beefy machine running 24/7 at $10, an
25:11
hour or whatever, right? Like that's just
25:12
cost-prohibitive. So I think what I'd advise
25:16
companies to do is to really look for something that allows
25:19
their developers to transfer from development, to scalable
25:24
serving, and scalable infrastructure pretty quickly.
25:27
And also allows them to kind of take Um, use their favorite
25:33
python, libraries and whatever they've done.
25:35
So they can just continue that and really look at how you can
25:38
do that in a simple way. And, you know, so that's what
25:41
I'd look towards right now. It's really interesting to see
25:43
many of the open source libraries hitting in that
25:45
direction already still early days but like, pie torch has,
25:50
you know, f s DP and DP. And again, you know, those large
25:53
language models are really pulling us in that direction of
25:56
like build for scalability from day one.
25:58
When you look at some of the existing Solutions, some of them
26:00
like skates, Say to make it definitely has a scalability but
26:03
is it flexible enough? Demetrius going back to my talk
26:06
where I talked about Second Generation and third generation
26:09
ml systems. Most of the systems that are out
26:11
there are very fixed function pipeline there.
26:13
Like you have a monitoring stage and training stage.
26:17
Well, what did you want to close the loop?
26:19
What if you want to have semi-supervised learning the
26:21
minute they did. You want to do something
26:23
slightly out of the ordinary cut and gets hard.
26:27
And so that's why the flexibility and kind of moving
26:30
towards something. That feels more.
26:31
Like normal programming and is just like a bunch of annotations
26:35
that you would put on your normal rate, programming normal,
26:38
normal programming, you know, that's the way that I think
26:40
things should hit. So you kind of touched on it but
26:44
I would love to go a little bit deeper into this idea of your at
26:48
a start-up and you're good, or at least you think you're good.
26:52
And then in a blink of an eye, you've scaled to a point of no
26:57
return and you realize, oh my god, what have we gotten
27:00
ourselves into that spot right there between being good and
27:05
then overloaded with scale, a, how does it happen?
27:09
What have you seen to? Bat that.
27:12
And how do you look at it? Like what are some signs of a
27:15
starting to potentially hit the limit of where we should beat?
27:19
Well some some businesses, in some of our customers kind of
27:22
heat scale from day one? So for example let's say the
27:27
here, you're trying to build a geographic processes company
27:30
called Descartes labs and they do like geographic information
27:33
processing for the whole world, right?
27:35
So you and you're applying algorithm to a small patch of
27:38
the world and then will you just have to process the entire
27:41
globe? Say you're trying to find
27:43
rubbish patches in the ocean or something, right?
27:45
So you'll try this little model in a single area and then you
27:47
got to cover the rest of the 70% of the world, world service
27:51
area, to find all those other rhombus rum rubbish patches.
27:54
So in some cases you know, you're going to have scale from
27:57
in at number one, in other cases, it's scaling, as you hit
28:01
more customers and that one's, you probably have a little bit
28:04
more time to do and other cases. Again, you know, another
28:07
example, where, you know, from day one is if you're doing a
28:09
large language models and you have like 175 million
28:12
parameters, scale is part of your equation, right?
28:15
And that's why companies like open Ai and so on a building on
28:18
top of rate, which is this open source, scalable solution,
28:21
sometimes you get a little bit of warning that the typical
28:24
pattern is, oh, I could this is this was trying on my local
28:26
machine. I can fit everything on my local
28:28
machine. So you have a transition, when
28:29
you go from your local machine to a virtual machine and then
28:33
the next transition is from small virtual machine to a big
28:36
virtual machine and then and then you just kind of have to
28:40
bite the bullet at that. So we start with this small
28:44
machine, your local laptop, maybe then you move to a bigger
28:47
virtual machine and then from that bigger virtual machine, you
28:50
kind of get a bigger but, you know, maybe you realize then
28:54
that you've got to do something funky and people come up with a
28:57
key Solutions. Like I don't know, splitting the
28:59
model geographically or maybe hacking together a python script
29:03
that does half of a, you know, maybe just trying to go, oh my
29:06
goodness are. Maybe I can use air flow to do
29:09
this and all of a sudden they're having to Kind of do all of this
29:12
custom hackery and when you starting to do that, custom
29:15
hacker, you're kind of like okay it just gets harder and harder
29:19
as the scale gets bigger and bigger.
29:21
Hey Lazlo here, if you're serious about Emma laughs you
29:24
did subscribe right now. Right.
29:32
Now, that's a thing you quickly mention which is sounds.
29:36
These biggest largest large language models.
29:40
Were certain put, he's in the bay.
29:42
Is that something? We're at least that's the way
29:47
how I feel, which is most of these.
29:49
And most of the companies where you could still have a human in
29:53
the loop, whether that be like an education, preparation kind
29:56
of company for could it be copywriting company?
30:00
Or maybe a few others. All of those might actually very
30:04
quickly transition to using these elements and generated.
30:08
What is, what would be the big things for them to think about
30:13
when it comes to scaling or when it comes to capturing this game?
30:18
Yeah. So let's I mean, on the one
30:20
side, like I said, earlier on, I've never seen anything like LM
30:23
before on the other side. I think there's barriers to LOLs
30:28
that people really need to think about, right?
30:31
And the first of those is cost, like I don't know if you want to
30:34
serve a fine-tuned model from from open from open a, I was
30:39
looking at this if you want to use like the Vinci which is the
30:41
most complex model It's 12 cents for a thousand tokens and that
30:47
1000 tokens is like 700 words, right?
30:50
So can you imagine if we create cost 12 cents 4441 output,
30:54
right? Like that's kind of crazy town
30:55
numbers, right? So before you start getting only
30:58
overexcited about lme's, just realized that cost is are like a
31:02
major major issue. We have some estimates that
31:05
we've seen that, you know, a Google query costs. 1/10 2/100
31:09
is much as an llm query of these kinds.
31:13
Right? And then you say well that's
31:15
okay. I'm not going to use one of
31:16
those commercial Partners. I'm going to roll it myself.
31:18
And then you go, wait, how big of these models and you go?
31:21
Well 175 being permanence about 350 geek.
31:24
Also let me go get a 350 Geographic scope.
31:26
Well, good luck, finding that, right?
31:28
The biggest you can find find in theory is a 100's with a DGP,
31:33
you know, 80 gigs. So that means best case if you
31:36
can convince Amazon, to give you an a 100, right?
31:40
You're looking at for gpus just to serve the model, right?
31:43
So, So rolling it yourself is also not an option in the sense
31:47
that it saves you, or you, I mean, imagine taking your
31:49
machine learning model and now having split over 16 gpus, you
31:53
need some kind of system to kind of get a coordinated as well.
31:57
So even though large language models are extremely promising,
32:00
I still think we have to think about the cost and the scale and
32:03
that's still the area where I think there's a lot of
32:05
challenges and you know, thinking back to your previous
32:09
podcast with Alex right now and he was pointing out that you
32:12
know, cost is an issue to It is an issue but also the governance
32:16
issues, right? And I think that's where it's
32:18
really interesting because dealing with all of those issues
32:20
can be very hard. So, I think where the industry
32:23
is going, is this trying to work out how they handle this?
32:26
And how you do with customization over a large
32:28
model, fine-tuning kind of works.
32:30
But the problem is again to deploy a fine-tuned model, is
32:33
either very costly or very complex the function models.
32:37
If you're going to, I'd still think there's a lot of really
32:39
interesting work on things like patching or layered neural
32:42
networks on You have a foundational model, that is
32:45
served. And then you somehow patch it
32:46
after the fact, but it's still kind of in the lab and it's
32:49
still pretty tricky to do. So, this is a conversation.
32:53
I was having today, which is what's a good balance between
32:56
fine-tuning forces, prompt internally.
33:00
And one of the answers, I was talking to our David.
33:04
Do we heart basically stability. And he said, well, when you do
33:09
own the model itself, finally news.
33:13
An option, you're accessing the model to an API.
33:16
I'm so basically the only hate, the only thing you can do on top
33:22
of that is basically prompted today.
33:24
Which again brings all of these questions that you talked about
33:28
which is governance and to say, getting out, how do we deal with
33:32
outliers? What should be an expected
33:35
behavior and accountability, like the legality side of
33:40
things. And so I mean, I still think,
33:42
you know, this is Is maybe taking a historical approach.
33:45
The one concern I have with these Lads language models is
33:48
nobody knows what's going on inside there.
33:50
And actually, you know, there's a really good professor at
33:52
Stanford called, Noah Noah Goodman.
33:54
He's kind of one of these new Neuroscience / machine learning
33:58
people. And he's actually using the
34:01
tools of Neuroscience to study. What the hell is happening in
34:04
these in these date models? Noah Goodman you said, yes, I
34:08
know a good is doing this really work, which is to apply the
34:10
techniques of Compton like Neuroscience to understand what,
34:14
what's actually going on in these 175 begin parameters.
34:18
That's how complex they got. It is a little bit of work,
34:21
that's kind of interesting retrieves.
34:23
Sure. Is that if you find chain you
34:24
can actually shrink your model to something a little bit more
34:27
handleable, like six billion parameters.
34:29
But again, Kind of its on this border between research and
34:32
practicality. And it usually takes like, 18
34:36
months to two years for those problems to kind of be sorted
34:38
out, depending on the level of complexity and sometimes much
34:41
larger. So, I think we're at the very
34:44
beginnings of this. I think the key thing is just to
34:47
be ready when it comes. And yeah, this is, this is a,
34:50
I'm surprised, like I was reflecting, you know, the
34:53
original attention is all you need.
34:54
Paper is from 2017 and it's like, I've never seen anything.
34:59
A thing like this every every you know recurrent neural
35:02
networks basically wiped out LS TMS are wiped out CNN's elect up
35:07
against that kind of sort of on the edge just hanging on by the
35:10
skin of their teeth. Right.
35:11
This type of Revolution doesn't happen very often but so you
35:14
kind of have to design your infrastructure from day, one to
35:17
be capable of supporting whatever it is that's coming
35:20
down the pipeline. And one thing we know about it,
35:23
for sure. Is there going to be big?
35:24
Like the smallest model people talk about is Six billion
35:28
parameters, 10 billion parameters 22 billion parameters
35:31
still gigantic. Yeah still gigantic you may be
35:35
able to fit them on a single machine with like a cheapie use
35:38
but you know still going to be complicated to coordinate those
35:41
H-E-B used to do things together.
35:43
There is some really interesting work about coming from Berkeley
35:46
about a system called Alpha. That is looking at how to slice
35:49
this, these systems slice, these complex systems to run them
35:53
across multiple gpus and multiple machines.
35:56
Uh-huh. Nice flavor.
35:57
Pretty cool. Demo online where they're
35:58
serving a 175 Boolean parameter model.
36:01
Well, so there's two questions that I kind of want to ask and
36:07
not sure which one is more interesting but I'll go, I'll go
36:10
with the spicier one. I'll say these reference about
36:15
sponsorship again, Demetrius it. You're going to hang up.
36:18
These are gonna slam the laptop down.
36:21
There he goes. So no actually, I think about
36:25
the movement that's happening. Eating right now, in deep
36:28
learning. And basically a lot of apps that
36:30
are being built on top of G, PT 3 or chat GPT, right?
36:35
And how it has in one regard empowered.
36:38
Many people to just hit the open a.i.
36:41
API and leverage machine learning.
36:46
But then you talk about price and you think about the skill
36:50
ability when it comes to that. Like if all of a sudden now
36:54
these apps that are being built on top of Many eyes API are
36:59
getting billions of queries per second.
37:02
It's gonna be like, what is your business model you're going to
37:06
need to know your business bottle?
37:07
Real fast, right? Yes.
37:09
Yeah, absolutely. I mean the unit economics, I
37:11
still don't think people have sat down and worked out the unit
37:14
unit economics of these open API calls again you know up to 12
37:18
cents per query for a fine-tuned model.
37:21
Actually, there was an answer to your earlier.
37:22
Question on me. You can you can now find
37:24
shooting with with something like up.
37:27
Nyko here has a similar interface.
37:29
You can kind of upload your data and just say hey fine-tuning on
37:32
this. It doesn't work very well.
37:34
Uh-huh. And I suspect it's not quite as
37:36
soon as I know like it existing Theory.
37:38
I don't think the practice is there but yeah you can actually
37:42
find chewing these models Demetrius.
37:44
Well, as your actual original question, sorry it slipped my
37:46
mind. Where was I going with that very
37:48
thing on the plane? It's basically what what I was
37:55
wondering. Save question.
37:58
No more sponsors have no more. I promise I will not talk about
38:03
that anymore until the end. So he's the real thing that I
38:07
was thinking about was when we're looking at doing that and
38:11
then if you factor in that we're going to be trying to
38:14
potentially linked up a bunch of these different models and for
38:19
certain use cases, we're not just going to want one.
38:22
We're going to want it together. Do you foresee that when you are
38:25
looking at the scalability? Shoes and you're seeing right
38:28
right now. There's a bottleneck if I had to
38:31
guess, I would say that the price is going to come down or
38:34
people are going to get more creative on how they are
38:38
chaining together models or using less data or what is your
38:42
your view on that. Yeah.
38:44
So the prices will come down and I think part of that is a shift
38:48
towards more vanilla infrastructure and, you know,
38:50
there's a lot of work going on in that front deep speed, for
38:53
example, is helping to make things simple.
38:55
But there's a lot of work on me. Compression.
38:57
But, you know, the funny thing is, you know, again, that stuff
39:00
is going to take time. It takes, like I said, two to
39:02
three years for this stuff that's kind of a little bit
39:04
researchy to be good enough that you can trust it.
39:07
I think what will happen is, people will find, you know, I
39:10
was thinking back to again, this is where history is a really
39:12
great teacher. If you look at the history of
39:15
speech recognition, what happened is people picked off a
39:19
really few very lucrative use cases of speech recognition and
39:24
then made that into the market. So you have to examine People's
39:27
one was medical doctors, giving dictation.
39:30
And the other one was the legal industry and those are both very
39:32
high value. So I think what we'll see is
39:35
high-value use cases will be first cab off the rank and then
39:37
it will kind of cascade downwards into the rest of the
39:40
industry as people learn how to use these things more,
39:43
effectively, as things like model compression.
39:46
And what you can do with model compression without breaking
39:48
things, you know, there's even things out there like Flex gen
39:53
which is a really interesting technology for reducing the
39:55
price for batch. Cases where they take all of
39:58
your examples and kind of try and fit it on the GPU and do it
40:02
like once every so often. So really looking at things like
40:05
the latency throughput distinction, to find ways to
40:08
make deep learning and these batch, I think a lot of the
40:11
complexity comes from doing it in real time.
40:13
If their use cases that are batch oriented, they might be
40:16
first cab off the rank. So, what I'll see is, you know,
40:18
what? I predict is there's there's
40:20
going to be a few areas where they're either very high value
40:24
or there's a trade-off that you can make to really Really
40:27
distinguish yourself. And by the way, what you're
40:29
saying about, you know, chatting, you know, putting
40:32
together, lots of different systems that's happening to.
40:35
So, there's a really great system called Lang chain and
40:38
open source project. That's exactly about kind of
40:41
well, I'm going to do these three stages and then I'm going
40:43
to pre-process it and then I'm gonna send another crew to open
40:46
a. I and then bring that back and
40:48
then do this, right? I think the other thing to
40:49
realize is not all of it is that complicate.
40:52
So you know, large language models are huge, but the first
40:54
stage is embedding and what can do it.
40:57
Just with him Betty, you know, in terms of taking doing things
40:59
like documents such as a company called Victoria, that's doing a
41:02
combination of large language model embedding plus Vector data
41:06
base to give like text search results, perfect, and they doing
41:10
some very interesting things there so we can, you know, it's
41:12
not like you have to adopt shatter apt or G PT 3 wholesale,
41:15
you can do bits of it or just use the embedding part of it.
41:18
And then, finally, I think there's a secret Source in chat
41:22
GPT that people don't realize, which they have to think about
41:25
very carefully. Which is, it's not a pure
41:30
supervised Learning System. It's actually a reinforcement
41:33
learning system. So, what happens is, you have
41:36
humans review the output, there's thousands of people
41:39
employed, I think there was a report saying, you know,
41:41
thousands of people in Kenya going through, looking at chat,
41:43
gbt things and kind of manually annotating them just to make
41:49
sure the Chutney PT produces like very good results and
41:52
that's to deal with the governance issues that we talked
41:54
about earlier, but, you know, I'm having Working at how
41:58
smaller companies will do that. Again, it's going to be, there's
42:00
there's these challenges, people are going to think that they can
42:03
get good results. Like check TBT, but they're
42:05
ignoring this huge component that only got company like,
42:09
open-air can do unless, you know, hopefully they will be
42:12
open source data sets that would allow other companies to do it
42:15
as well. Well, so that's the big question
42:17
that I had next is, will there be enough open source models out
42:21
there for the rest of us, if we don't want to give a percentage.
42:27
Integer of our monthly budget to opening I or is it going to be
42:30
opening eyes the next AWS gcp? Type thing?
42:34
Well, I think they were at this. I mean, I think they are such
42:38
for starters. There are lots of these large
42:41
language models and to name probably the two most.
42:43
Well-known one is Bloom from the big science project and then the
42:47
other one is opt 175 B, which is from metal, I think.
42:52
So, both of those are kind of steps in people moving towards
42:55
open source data set. Open source, training
42:58
algorithms, deep speed is, you know, you can train your own
43:00
model and deep speed. Now, I think there's going to be
43:03
a very strong force towards open source for these types of
43:06
systems again, because of the need to fine-tune and to modify.
43:10
And I think ultimately companies are not that comfortable sharing
43:14
their data with like open-air. Oh, so you want all of my
43:17
internal customer services messages to give that to an
43:20
external company could process. When they've already proven
43:23
that, you know, there are accidentally causing copyright
43:25
issues and there's Ready. Suits around them.
43:28
So there's I think it has to go in the direction of Open Source.
43:31
There's also a really great companies that are really
43:33
helping us to push that, which, like hugging face for example,
43:37
which is putting itself at the middle of this particular, kind
43:40
of movement. So will come.
43:41
They will take longer. And again, the tricky part is
43:43
the chat, gbto, gbd3. You know, it's just like this
43:46
very straight. I mean, it's not very
43:47
straightforward, but it's a very traditional machine.
43:51
Learning trading model or catchy PT let's not put those two
43:55
things in the same bucket, right?
43:56
They not discerning. Again, because of this are lhf
43:59
and I don't think the are. I mean there's already an open
44:02
source project trying to do. They are lhf the reinforcement
44:04
learning with human feedback stuff, which is really the
44:08
secret sauce to chat apt, but that's going to take some time
44:12
and that takes some money to actually run like, you know, a
44:15
you know essentially at a human Borg data center that's doing
44:20
these labeling things as well as well.
44:22
And then speaking of open-source I think we got a touch on real
44:25
little bit and what Is going on with Ray because I looked before
44:30
we got on this conversation and it's got like, 24,000 GitHub
44:33
Stars, right? It is loved by the people.
44:36
I would say, that's not a modest amount of GitHub Stars, it's
44:41
pretty good. And you also mentioned
44:44
beforehand, that open a.i. uses it.
44:46
You mentioned that there's all kinds of different users of Ray
44:50
when it comes to like what it is.
44:53
You've been hinting at it for this conversation.
44:57
Ocean and especially when we're talking about scalability
45:00
issues, but now it's probably a good moment.
45:02
Just to explain real clearly what it's doing and what the
45:07
value at is there. Yeah, so the way I think about
45:10
raise its three things. Number one, it's a very clean,
45:14
very cute and it tation based modern interface for python
45:18
distributed computing. Right.
45:20
And you know that's I think one of the reasons that people
45:22
really love it, they get to choose their own libraries, but
45:24
they don't have to give up anything.
45:26
They don't need to kind of Compile, like special versions
45:28
and ship them and do all that kind of stuff.
45:31
It's just python it leaves. Um, you know, so that's part.
45:34
One part two, is that lives on like a better really intelligent
45:37
relatively complicated, cluster order scheduler, that you give
45:41
it a cluster of machines and it distributes the workloads
45:44
written against that Ray API to all others different machines.
45:47
And then finally it's a it's an ecosystem of libraries that sits
45:51
on top of that very clean pythonic API for your favorite
45:54
machine. Learning tasks, reinforcement
45:55
learning hyperparameters. Our optimization deep learning
45:58
training. But now optimized to be trained
46:01
in a scalable way across dozens of machines or hundreds of
46:03
machines or even thousands of machines.
46:05
And that's why it's not just opening, I who uses us, it's
46:08
companies like uber instacart Shopify Spotify, probably
46:13
forgetting half a dozen other ones, but Ray is used a lot.
46:16
If a company is large enough to build its own, kind of internal
46:20
machine learning toolkit, that's linked in all of the different
46:22
ways. But there's another way people
46:24
use Ray, which is just doing, like, Very simple parallelism
46:29
that helps you to get things done.
46:31
So, you know, in some cases let's say you're doing demand
46:33
prediction for like ten thousand different products and you got
46:37
to build 10,000 different models.
46:39
That's very hard to do with existing systems and it's more
46:42
or less trivial in re because you know, you kind of folk the
46:44
data out and you just have 10,000 tasks and there's 10,000
46:47
tasks can run on 100 machines. I have two tasks each, all of
46:51
that detail about, you know, once we've converted it to
46:53
10,000 tasks. What do you do is handle for the
46:56
person by, right? Ray, another example is, we see
46:58
a lot of companies doing like NLP, but they want to run NLP on
47:02
like, 50 million documents. And so Ray just gives you a very
47:06
nice tool kit for batch inference and those types of
47:08
things. So it's not only just the people
47:10
using it for your ml infrastructure.
47:12
Those big companies that I mentioned earlier, it's also
47:14
these data scientists and machine learning people want to
47:17
do a little thing, more flexible.
47:20
We, which one I'd like a cute interface that allows them to do
47:22
those massively parallel. Another example is back-testing
47:25
rank, like we all do dementia. National something like that,
47:28
you know, it stopped production and now you need to test on 50
47:31
years of data. That's a real pain in the neck
47:33
generally to do re can be the toolkit to kind of help you do
47:37
that. And of course, we also we're
47:39
still testing, we have a hosted version of Ray called any scale,
47:42
that we also make available to some of our customers.
47:44
If you're interested in that, feel free to reach out to me,
47:47
you know, we're still in the early stages of that.
47:49
But really, it's the power of raised, just something that I
47:52
fell in love with, right? Like I said, you know, before I
47:54
joined the company, I used it to recode It's on my PhD and I was
47:58
like, really impressed because I made it.
48:00
You made parallel and distributed computing, which are
48:03
historically, like, some of the most difficult problems in
48:06
Computing accessible and easy, which I really enjoy.
48:09
Go on then, tell us what the value add for any scale, is,
48:13
what do you get on top of Ray? I mean you sold me on Ray.
48:16
Now I want to know what's what's?
48:17
I need to go there too toasted? Yeah.
48:20
What are you hosting? Ideally, you know, we can't take
48:24
on all of your ml Ops workload but most of it, you know,
48:27
D-League is automated now because a lot of the
48:29
complexities of the scalability questions.
48:31
So hopefully reduces your ml Ops workload and that's why we're
48:35
seeing a lot of interest from small companies.
48:37
Like five-person startups that want to process 20 terabytes of
48:40
data. We also have like a very nice
48:42
development environment called workspaces.
48:44
That helps you to really make that transition easy.
48:47
Using your favorite tools, like Jupiter or Visual Studio code.
48:50
And we also have all of the things you need for Enterprises
48:52
like security, no public IP addresses cost management.
48:57
All of those types of things that you need like in a real
48:59
business, not just for experimentation.
49:01
If you don't have your own team of Mo infrastructure, Engineers
49:04
completely. So when it comes to the idea of
49:08
just having massive amounts of data, I think there is an
49:12
obvious move towards that. As you're mentioning, there are
49:15
quite a few companies that are going in that direction, and
49:18
they've got so much data in there.
49:20
Just not, there's no turning back, right?
49:23
And then you have, and then what I've seen is, there's this Other
49:27
movement that I would put the duck DB under or the mother duck
49:34
group. And they're trying to push this
49:37
theme of big data is dead and so I wonder how you reconcile that
49:42
is. It something that you see, okay?
49:45
For certain use cases, you can't help but just have an incredible
49:50
amount of data. And then for other use cases,
49:53
you don't necessarily need to bring on all that operational
49:58
cost just like I think when we look at the idea of going real
50:01
time it's not necessarily something that you have to do.
50:05
It has a value add for certain use cases but it's not like you
50:10
should do it. Always because of the
50:12
operational overhead that at this point in time it brings to
50:17
the table you can and in some cases that's enough, right?
50:20
And then you can also look at foundational models.
50:23
There's another way to take a shortcut there so you can do
50:25
zero shot or fuchsia. Learning or do you prompt
50:29
engineering as a beamer suggesting earlier?
50:31
But the thing that kind of makes me come back to more data
50:34
eventually winning possibly with human review or human
50:38
annotation. Right.
50:39
Problem is the more data you add.
50:41
This is kind of the lesson of the last 30 years.
50:44
We all thought it was going to be like, more clever, algorithms
50:46
like decision trees and everything else.
50:49
But you know there's a phase shift once you get past.
50:52
I mean like the way I think about it is again thinking back
50:55
to you know in 2012 when I was like why are these deep Learning
50:59
Systems better than these kind of hanshin?
51:02
Fake feature engineered things as like maybe a human in their
51:05
lifetime sees 10,000 faces? These machines, look at 100
51:11
million faces. Is it all that surprising that
51:14
they're better than human performance?
51:16
And you know, people have done analyses that Shone like the The
51:19
data you had for a lot of these large language models, the more
51:21
accurate they get. So given the desire for more
51:25
accurate models. And in many cases, the value
51:28
comes from that accuracy. I think that tendency will be
51:31
more towards data. There is another alternative
51:33
which is human annotation or human wisdom.
51:36
And there's all sweet of approaches are active learning
51:38
semi-supervised, learning labeling functions, Alice nor
51:41
call, you know, all of those kinds of things, but they're not
51:44
cheap. Either, you can get a bit more
51:45
value of them, but no think about what it takes and why
51:48
there are startups like Scale that I that specialize in
51:51
labeling so either you do it more data, if you want high
51:54
accuracy or you do it with more human intelligence.
51:57
And personally, you know, I've always been I think for me, I
52:00
think historically, we've been like the robot model of machine
52:04
learning, you know, where data in and something comes out.
52:06
And I'm really looking towards the transition more towards what
52:08
I call the cyborg model, which is a hybrid of human and machine
52:11
working together. Yeah.
52:12
Neither of the two parts is easy though, right?
52:15
And that's why there's movements towards things like machine
52:17
teaching and so on, I think, That's the other strand.
52:20
So it really depends on your the night, maybe for some of the
52:22
domains you're interested in, you can bring human expertise to
52:26
help you make the problem easier.
52:27
But again, the painful lesson of the last 20 years in machine
52:32
learning for me is data usually wins and I wish it was any other
52:36
way because we know humans are humans.
52:38
Don't need that much data to perform as well as they are, but
52:41
that's the current state of the art, right?
52:43
Like that's just the reality. I wish it wasn't the case.
52:46
I love clean cute algorithms that we Stan, what's happening?
52:49
What the hell is happening inside them?
52:51
But deep learning is just one, it's just very hard to help.
52:53
Perform one, just a prank question.
52:56
And I think you've seen an oath models over your time, working
53:00
in the field, do you think there's a chance we could go
53:04
from, like, fine? We are at peace, deep learning
53:09
models right now, which do not really have a good understanding
53:13
of cause and effect. I think that's one of the key
53:16
factors as to why they need so much.
53:18
Data. Do you think there's a
53:20
possibility that maybe in the next five years?
53:24
We may be working in that direction where we can be figure
53:28
out the cause and effect, which is basically serve adding
53:32
causality to earn models. And then for me, not eat catfish
53:35
Tito together. This is a question that, you
53:38
know, has dogged Ai, and machine learning for 30 years, I am
53:41
really excited about some of the newer symbolic approaches.
53:44
Like, I really like the idea of we can represent some things
53:48
like Dream back when I was at boober is it should be that the
53:53
car and robot should be able to read the California vehicle code
53:57
and somehow take all of these rules and converted into a
54:01
process. And it can, it can take but
54:03
that's just not how it is at the moment.
54:05
And so, historically through Computing, it's either being the
54:07
neural approach of the symbolic approach.
54:10
I think all of us are looking for the fusion, the neuro
54:12
symbolic approach, but nobody knows how to do it.
54:15
And I wish I was confident that it was coming in the next five
54:17
years but I still Dildo. I think it's a bit further away
54:20
than that. The other thing is, this is
54:22
other form of analyzing the world which is statistics and
54:26
probability. And you know, I wasn't been
54:28
interested in kind of combinations of the Bayesian
54:31
approach with deep learning and some of them have a flavor of
54:34
that but it's just it hasn't come together.
54:36
The way that I think we were hoping.
54:38
Right and machine learning is full of dead ends of the things.
54:42
We thought were good ideas, but didn't play out in practice.
54:44
So, you know, I'm really interested in the question, but
54:47
I think it's technically a lot harder.
54:48
Order. And there's also not just
54:50
academic technical question, but it's almost like a communities
54:54
question. There's the symbolic approaches
54:56
in the neural approaches, and they're both kind of have a
54:58
little bit of skepticism in each unit, each other and remembering
55:01
back to again. You know, nips nips, had almost
55:04
new neural papers and everyone thought neural networks were a
55:07
dead end. There's also that cultural
55:09
barrier to overcome, which is not a small thing in academic
55:12
and research communities to be honestly I think there's a few
55:15
thought leaders like I discussed no a Goodman earlier.
55:18
People, like Zubin Gary money and so on that.
55:20
I really looking at ways to fuse these two models, but it's I
55:25
think it's a lot longer than we think.
55:27
So, coming back to building on top of these models.
55:30
What have been some of your favorite apps or different
55:35
developments that you've seen because there has been such an
55:38
explosion, right? And I imagine you've seen a lot
55:41
of cool stuff, whether it's inside of these companies or
55:44
outside, just the what I lovingly referred to as toy.
55:48
A Oops, let me another there, might not be toy ABS, but
55:51
they're still trying to find their footing and it's like, oh,
55:54
check out what we can do really easy now.
55:56
So what have you seen that? You really liked and it stood
55:59
out to you. So, I'm really interested in
56:01
this idea of like, when you look in public media, it's always a,
56:04
i versus human, okay? But I always liked these ideas
56:08
that like, I'm or AI plus human in flavor.
56:11
I think one of the really interesting applications is in
56:13
painting. So right now, when you do stable
56:15
diffusion, you have to type in like something like Just the
56:19
same things but you can't provide like a sketch as input
56:21
and I'm really interested in Things by like imagine that you
56:24
were like not very good at drawing but you could draw a
56:26
rough image and then you could give it to something like stable
56:29
diffusion and it could turn into a better image.
56:31
I think that's really interesting.
56:32
I'm looking forward to the time when Adobe like neural stuff is
56:37
just a feature built into Adobe Photoshop because then it's just
56:41
part of our normal workflow, right?
56:43
And this is why I like this idea of impending, you know, there's
56:45
a little part of the image. I kind of most of the The image
56:49
looks good. I sketch that out.
56:50
That's okay. Let me just delete this little
56:51
bit in the middle and have the a I correct that, that idea of
56:55
idea. Generation.
56:56
This is for me, that the coolest thing about the coolest use of
57:00
chat apt is the use of it for human enhanced idea generation.
57:04
So you can do things like, hey, I have to give a talk to this
57:06
group of people. What should I talk about?
57:08
And it'll list a few ideas? For means a?
57:11
Oh, that one looks interesting. Tell me more about your ideas in
57:13
that space. And so really, if this human
57:16
machine augmentation that I really I am personally excited
57:20
about rather than this kind of adversarial type of AI versus
57:24
human philosophy. That the media sometimes gets
57:27
stuck in. Do you no longer have an excuse
57:29
for any kind of writer's block or any kind of artistic block
57:33
whatsoever because you can just plug it into this and get so
57:37
many options. I think that's the coolest thing
57:39
is that when you hit a block, you can go and talk to the
57:44
machine and it will help you through and it will not only
57:46
like help you through with one option.
57:48
In it will give you 20 different options and you can explore
57:52
those different options in Layton space which I also find
57:56
fascinating. Yeah, absolutely.
57:58
Walid dude, this has been incredible man.
58:02
I knew it was going to be fun. I did not think it was going to
58:05
be this fun. I appreciate you coming on here.
58:07
This seems like a perfect place to end the podcast, Demetrius
58:10
and happy. I just want to say thank you for
58:12
your time today. I had a blast as well.
58:14
One quick thing, the race Summit is coming up in September, but
58:18
there's Call for proposals out. So that if folks are interested
58:21
in participating in the upcoming race Summit, please don't
58:24
hesitate to look us up at radar IO and we look forward to seeing
58:28
your proposals. I know I'm actually a little bit
58:31
good proposal. Oh, yeah.
58:33
There we go. I think, you know, the questions
58:35
were interested in are really about scalability and how do you
58:38
build scalable flexible machine Learning Systems in our previous
58:42
speakers, have been people like Predator base, cohere, you know,
58:46
these types of companies that are really pushing the
58:47
boundaries in Terms of how you make flexible and scalable
58:52
machine Learning Systems. I was hoping you were going to
58:54
ask me to host a panel or something while we don't realize
58:58
I'll see what I can do. Demetrius lots of let's talk
59:01
about promises from you. Hmm, I'm asking a lot of you
59:06
man, but this has been great. I really appreciate it and
59:10
anybody that's out there. Also, you want to like is any
59:13
scale hiring. I guess we are hiring me looking
59:16
for software engineers. And especially people in
59:20
developer productivity and Cloud infrastructure.
59:22
So we'd be very very happy to hear from applicants.
59:25
I think it said, hi, the job's, not over any, any standard in
59:30
the description job? Yeah, yeah.
59:32
Don't worry, control worry. Yeah, for sure.
59:35
You get out there. Go high applied to work with
59:39
Wally and that would be awesome. I can't think of a better place
59:42
to be working and and working on these tough problems of
59:44
scalability, man. I mean, you've said it so well.
59:48
Self over the last 30 years. It's been something that has
59:52
really been pounded at your doorstep.
59:54
So it's cool to see that you're still at it.
59:57
After the this time trying to do it.
1:00:00
I mean, it's awesome. Yeah, and still having fun.
1:00:03
There we go. That's the fun part that that's
1:00:05
the best part. All right, dude.
1:00:09
Hey I'm Vishnu, I'm a data scientist at first hand and I
1:00:12
definitely think that you should subscribe to the ml Ops coffee
1:00:16
sessions podcast, it's the best podcast.
1:00:18
Cast out there to stay on top of what ml Ops actually is to talk
1:00:22
to the truth, thought leaders in the space.
1:00:24
And oh, by the way, dimitrios is absolutely hilarious.
1:00:27
What a weird guy, you should definitely subscribe to the
1:00:30
podcast.