Latest / Agentic Conversations (formally mlops.community) / MLOps vs ML Orchestration // Ketan Umare // #183

Transcript

MLOps vs ML Orchestration // Ketan Umare // #183 · Agentic Conversations (formally mlops.community)

0:00
And my name is Caitlin and the CEO of Union.
0:04
I intermittent fast so I have coffee black every morning.
0:09
Just say a shot of espresso with hot water.
0:14
Welcome back to the MLM Community Podcast.
0:17
I'm your host, Dimitrios. Today we're talking with Kate
0:19
Tan and we have a special guest, host Stephen, who is not here
0:24
with me to do the normal banter of the intro, but he jumps in
0:28
about 2345 minutes into the conversation.
0:31
You will notice that he is there asking questions.
0:34
It's very special because he is a user of flight, he works at
0:40
Vault, the delivery service, and we actually did a podcast.
0:44
Specifically with him recently, so go check it out.
0:47
And he is an avid user of the flight platform and he built the
0:53
mill platform at Vault with flights.
0:56
So I thought it was only right that we get him to come on here
0:59
and chat with Katan and really dive into some of these
1:03
questions that we had with Keaton.
1:06
Ohh man, this is like his third time on and I enjoy talking to
1:09
him every time he comes on, especially because he.
1:15
Talked about some things that he didn't necessarily talk about a
1:18
lot in the first two times on which are how he feels building
1:22
a startup. You know he left lift after
1:26
building out the platform there and he decided to start union,
1:31
which is what? Light is based on and he really
1:35
talked about a building union being a company builder.
1:40
And then he also talked about how they differentiate between
1:45
what features need to go into Flight, the open source project,
1:50
and what features he feels needs to go into Union.
1:53
I thought it was fascinating looking at the differences there
1:56
and how he felt and how he prioritises one over the other
2:00
because flight has this gigantic community.
2:03
Around it, it's doing some great things and a lot of people are
2:06
asking for certain features and Keaton has to field these
2:10
features and say is the product going to go in that direction?
2:13
Are we going to invest resources into the open source product
2:18
having these capabilities or? Are we going to create these
2:22
capabilities inside of Union? And the long and short of it
2:26
was, was that when he's thinking about creating?
2:30
Features inside of Union, it is because they are more enterprise
2:36
use case features. I can't knock them for it.
2:39
You gotta get those enterprises happy.
2:42
They're the ones that are paying the bills, so I understand how
2:45
that works. This conversation was great,
2:48
though, because it's rare that you get.
2:51
Someone who is an avid user of a tool alongside of the creator of
2:56
the tool to get a little banter going back and forth.
3:01
Hope you all enjoy this podcast with Ketan and my co-host Steven
3:08
And if you like it or you know someone else that would like it
3:11
would mean the world to me if you share it with a friend,
3:14
Let's get into it. And a huge shout out to Union
3:17
and Ketan for being a sponsor of the ML OPS community.
3:20
They've been one of the first sponsors that we've ever had.
3:24
And so it is from the. Bottom of my heart.
3:30
Thank you. For anyone that is looking to
3:32
get some of them pipelining tools, hit up Caton.
3:35
Just tag them in the slag or check the link in the
3:39
description. Get yourself started with
3:41
flight. It is fully open source.
3:43
Or if you're that enterprise use case, go ahead and drop them a
3:47
line and tell them you want to give them a boatload of money.
3:49
See you. Katon man, this is your third
3:58
time on the show. I don't know of many people that
4:01
have been offered. The ability to come back, I know
4:07
one time, but two times. So this is a huge accomplishment
4:12
and I'm humbled. We can only reserve it for
4:15
someone like yourself, man. It is an absolute honour, as
4:19
always, to have you on here and today.
4:23
I want to talk about so much. I mean one of the last times
4:26
that you were on here we. Went.
4:29
Down a huge rabbit hole, talking all about orchestration.
4:32
I want to go back down that rabbit hole and hear what's
4:35
changed since the last time you were here because.
4:38
Large language models came out and that boom happened.
4:42
But I also want to know like. We talked a lot about air flow
4:47
and how airflow isn't enough. Last time I want to retouch on
4:51
some of that stuff and I was doing a little bit of research
4:56
for this episode and I I went back and I listened to some of
5:00
the snippets from the last time that we talked.
5:02
So I may put you on the spot and hold you to some of the stuff
5:06
that you said, but she had these points before the hand.
5:10
Dude, I'm gonna call out the elephant in the room, which is
5:14
you've had a company now for 2 1/2 years.
5:17
You are a founder. You're full on startup founder.
5:20
How's it feel? That's a great question.
5:23
I actually, I was talking to my wife the other day and I told
5:27
her he may not work, he may fail, but I still would be
5:31
happy. At 50, when I look back, I'll be
5:34
like, hey, I did something cool. I tried building a crazy
5:39
product. I tried building a company.
5:42
I tried building a community IIH like 10 years.
5:45
During that year, you know, it's like I was it Steve Jobs or Jeff
5:55
Bezos. I think regret minimization and
5:57
I think probably I feel like I realised very great like I
6:01
literally think that we especially in the tech industry
6:06
we sometimes go in the last decade I've seen excess and it's
6:14
very easy to get caught in that excess and when I started the
6:17
company I told my wife you probably not gonna make any
6:20
money and and high. Expectations Crazy, that is.
6:25
That is nuts. But I did started to make money.
6:29
I started to actually try and do something really different and
6:35
it it took guts and I had to dig deep and I was able to so and
6:42
I'm continuing to and which is fantastic.
6:44
I still want to come in every day and do more, which is why I
6:47
look forward to and unfortunate to have done this.
6:52
Also. Of course, I've been very lucky.
6:55
Yeah, I I think this is this is the best experience of my life.
6:59
Dude, so you set the expectations high with your wife
7:03
so that it wouldn't be too hard to break them, huh?
7:07
I'm imagining you guys have made some money and so she's already
7:11
probably very happy for you now. Yeah, we we we worked for 10
7:17
years. So we're we're OK, right.
7:19
We, you know, that's the other thing, we live with minimal
7:23
stuff and that's you're happy. But yeah, when I look back I
7:26
realise that many people are still stuck and that's how we
7:29
hire. Do we're like if you just come
7:31
to me and say like, hey, it's all about competitions like
7:33
startups start for you? Now that's yeah, go find a job
7:37
at one of the big Fang companies and there are great jobs.
7:40
Again, nothing against that. And that's the right thing for
7:42
you to do in the right stage of for like, you should know it.
7:45
If you know it, don't do it for the money.
7:47
Do it for having a fun ride. Yeah, so you're on this ride
7:52
now, man. You've built quite the community
7:55
with. Flight and.
7:59
I want to hear about how that has been because me as a
8:01
community guy and especially in the ML space, what have you
8:06
noticed since open sourcing flight back in the day and then
8:11
getting crazy ML engineers like Steven to come and start using
8:15
it? And all this in between?
8:19
Yeah, this is like the Rocky cutscene right now.
8:22
You're in it. Oh my It's like companies need
8:32
money to do almost anything. Communities don't need money.
8:36
Come come. Communities need passion and
8:38
love. If you join the slack and still
8:40
ask a question, I will still be answered and you'll probably see
8:44
me answering all the time. And that really, I don't know
8:49
what we did. What we did is we did not build
8:52
a toy in open source. We did not lie to the community.
8:56
We consistently tried to improve it and we tried to help where we
9:00
can. Yes, there were people who had
9:02
different expectations and those people did not join the
9:05
community. But most of the people who
9:06
stayed on hopefully, I've been happy and I've, I've talked to
9:10
them consistently. They're some of the earlier
9:13
folks and and they're still using.
9:16
They're still happy they're still looking forward.
9:18
They still want to move on and and it's it's growing and it's
9:22
going from one company to another and the best sign of the
9:27
community that I have had is when a person leaves a company A
9:32
and then goes to be like hey I missed this product, Can you
9:35
please add this to this company and and and that that's been
9:38
happening consistently now right a lot of times.
9:42
So and it's slower growth probably because it's organic,
9:47
but it's more rooted growth and the growth that sustains.
9:53
Yeah, definitely. I mean I can talk from my point
9:56
of view here is that when I joined flats like flights, the
9:58
slight shadow, actually it was like it was very nice and you
10:01
know, people had a lot of empathy as well and like you
10:03
know, people were taking the time as well to explain things,
10:07
even things that might be simple for you, Kate, and for example,
10:10
you know, still take the time to explain it.
10:13
The the flight community has a lot of empathy and is also
10:16
understanding. To be like, yeah, there are
10:18
people behind it and. And yeah, they they are trying
10:22
to build something. So yeah, and then then what we
10:25
realised is the type of people come in right now it's, it is a
10:28
range right? People with played with people
10:31
who have not played with Kubernetes.
10:33
People who you know experts in certain fields like Mill but do
10:38
not know anything about data. Everyday people who are data
10:40
ideas we're trying to work out. And so it's a huge mix, like
10:45
some people have never even tried Docker, right?
10:47
And then they'll come and ask us like how do we do it.
10:49
So we ended up building like certain things like image spec
10:52
that was essentially to answer or help folks who do not know
10:57
this and still get the maximum out of it.
10:59
The what we have learned is that it community you don't choose
11:02
right. It's not like, again, a company
11:04
you can choose who you work with in a community you don't
11:08
initially, and probably you don't want to because you want
11:12
that diverse. If that misses, our people are
11:16
getting one-dimensional feedback and that one-dimensional
11:18
feedback will potentially put it in the wrong place.
11:22
Is that it's an echo chamber. Yeah, that's such a good point.
11:26
I mean one thing that you mentioned there is that people
11:29
are coming from every angle and one angle that they're coming
11:32
from is the data engineering angle and I imagine.
11:37
They're very much thinking, hey, I'm gonna run this ML
11:42
orchestration just like I do my regular ETL.
11:45
What happens then? Yeah, it's fun.
11:51
Then they say where does the backfill button for like OK hold
11:54
on, what's the backfill? Ohh, why is that?
12:00
There are no time, always existent.
12:02
And I'm like OK, time is relative and time is not.
12:06
You turn into the philosopher Kate John.
12:09
Yeah. All of a sudden, like, shit.
12:11
I didn't know I had to go down that road.
12:13
I'm gonna have to put on my white beard.
12:16
And to start, yes, nobody. Absolutely right.
12:20
Actually, I still keep on finding people who expect it to
12:27
do, let's say what else wrote there or something else does and
12:30
we're like, but is it? It is its own product, right?
12:33
You have to give it its own, its own merit.
12:36
Like for example you cannot just ship like massive well models
12:41
that are being trained and pipelines to these products.
12:44
Similarly you probably cannot do time on on flight.
12:49
That's we because we think of time as just a parameter.
12:53
We think time is is an axis. Especially in the world you're
12:56
back testing your your running azoff scenario if you're
13:03
checking your hypothesis. When you're checking your
13:05
hypothesis, you wanna be able to play with anything later.
13:08
Die and play with data. I'll play with Martin, David I.
13:11
Yes, Cleveland Chord. So that and that's our first,
13:16
that's like the user first philosophy that we have like and
13:21
the user is experimenting in 90% of scenarios when they're
13:24
testing and building any product that uses eminent data because
13:29
you don't know if this is ever going to reach a customer.
13:32
When I first talked to a data scientist on machine learning, D
13:35
or even a data engineer was building a product nowadays that
13:38
reaches a customer. And then what is the likelihood
13:41
that a customer really see it and it's 5050 in the beginning
13:45
and then as you prove it, it becomes higher and higher.
13:49
Uh. So that's a different modality
13:51
of operation and I think we are not.
13:55
We have seen massive, massive by origins in the data and ML
14:00
communities and that is sad because ML communities cannot
14:05
survive without data. Many of the data engineers that
14:08
exist today will have to start building data products in my
14:11
opinion, and every product with that, when Mill is infused, it's
14:16
actually getting data. It's used somewhere.
14:18
So you have to start relying like it's not OK to go and
14:21
delete a table and just backfill it, right?
14:24
That's not OK from any point of view because you've probably
14:26
made a decision that changed the outcome.
14:30
Let's say for example, you go to a bank and it denies your loan
14:34
and then next day you go to the same bank and you talk to them,
14:38
the customer service and they say, hold on, I can see the loan
14:41
is approved. You're going to be angry.
14:44
What do you want people to prove your point.
14:46
And so when you go back and they'll be like, OK, let me go
14:48
back to the system and check it out and you turns out that there
14:50
was some data corruption in the top and you deleted it and and
14:54
you don't know what it was. And now you're like, OK, I made
14:57
some bad decisions in my life and you can't even regret about
15:01
it. So that's how I think about it.
15:05
Like we all have to start thinking in a little different
15:08
modality and it'll take time, right?
15:10
This is the reality of the world and that means we have to keep
15:14
on shifting that mics. Turns out that the model that
15:19
was powering your loan scoring was actually ChatGPT, and it
15:23
hallucinated. Yeah, it gives you a different
15:27
answer every time. So just keep applying and you
15:30
may get lucky. Yeah.
15:31
Ohh yeah, that's that's dangerous.
15:36
I don't want to be in a banking system that does that.
15:39
Yeah, totally. Well, you said something really
15:41
good there. That I think is worth noting
15:43
too, which is. A lot of these ML products.
15:48
They have. I think 5050 is a very generous
15:52
number to give people. When you're exploring and you're
15:55
trying to figure it out, you're just trying to see what's going
15:59
on. How can we hopefully get some
16:02
value from this? But you have to really push the
16:07
limits of what you think is possible and go down.
16:12
Dead ends that you don't realise are dead ends until two or four
16:15
weeks later. Yeah, I'll tell you an example,
16:18
but also the 50 fifties from the user, from the ML engineers
16:21
perspective or the data scientist perspective, they
16:23
think they're going to have to talk to them again, nothing.
16:28
I get it. But it's like, you know, the
16:29
optimism in a human. But the reality is that there
16:33
are five other data scientists at the same time working
16:36
potentially the same problem from a different axis.
16:39
There was a data scientist who built this amazing model, which
16:42
was, you know, Ottomans were hot.
16:45
It's OK, we'll go. And it was very good, extremely
16:50
complex. So it was hard, but like you
16:52
know, it's it showed value in some limited testing, but to
16:57
pressurise it takes a while. He went down the rabbit hole and
17:01
it did not prove any value, absolutely nothing, right.
17:04
So production. So yeah it's is like it's not
17:11
even 50 at that point. We were 7030 almost all of us
17:14
like ohh this seems amazing. We looked at the world and what
17:17
a genius insight and it seems like it but it proves that it
17:20
did not work. Like another example that I
17:23
actually typically gives that I worked on as well.
17:25
The first things is real time traffic right.
17:28
So in Etas we we were trying to use real time traffic.
17:36
ETF's are estimated time of arrival that lift would give
17:39
when you want to get it right and you were like think traffic
17:44
will play a very, very critical role here.
17:46
Now they can have because everywhere there's a lot of
17:48
traffic and that will change the ETS dramatically and we can
17:53
bring down our embassy and Maes and so let's try that out and we
17:58
how do you prove it like it kind of simulated a little bit.
18:01
I tried it out and maybe so but real time traffic is a big
18:05
challenge. Yeah you have to get data real
18:08
time processing analyse it. It was all quote, some of the
18:12
code was AC because the routing engines are in C viewer like
18:16
loading things and this takes like six months of effort to do
18:19
this stuff. And then that time we didn't
18:21
even have slight like thing. One of the reasons why it's like
18:24
kind of works with any languages because of that stuff, right?
18:28
You're scared in the beginning, yeah.
18:30
And what it turns out is that thankfully we didn't invest in
18:35
real time M as I've heard many times, because firstly, the
18:41
speed at which you go causes a lot of aberration.
18:44
Exactly. Think about it.
18:46
There is one car that stops that we have a sample for and we
18:51
think there's huge traffic jam, but potentially turns out that
18:54
that was just a top light or just like you know, that person
18:58
decided to stop for a. Little drop it off there.
18:59
Yeah, whatever. And you don't want to change
19:03
like the ETS. It's just dramatically
19:05
fluctuate. The ETS up and down.
19:06
It doesn't matter. So you have to put a damper.
19:08
You have to put like a bunch of other models.
19:10
And it turns out the hourly predictions for a week are so
19:14
good that the bias adjustment is what the traffic does only like
19:19
a little bit of addition so that it improves the tail problems.
19:24
You don't really cause a huge deviation in this.
19:26
And so that's what we found. And you're like, OK, people are
19:29
saying let's do real time and this is the most real time kind
19:32
of scenario that you can think of, like fraud, right.
19:35
Even in fraud, if one person does a fraud, you don't want
19:38
immediately to react to it like saying that, all right, this
19:40
everybody else who's doing like everybody's known should be
19:43
rejected. Saying, no, dude, that guy was
19:46
probably a one off case, right. So you want to wait for a little
19:50
bit, buffer it up, damper it out and then.
19:52
So 10 minute travel is what we found pretty good.
19:57
So things like that, like you know you you learn only when you
20:00
try and you don't know the real outcomes till you till you
20:04
really put it in production. Then I have to think about that.
20:07
So I'm actually working on a project too.
20:09
Lots of that now. So maybe I'll be like we don't
20:12
need real time e-mail here. No, try try smart, try go, go
20:16
faster and you know and then speed it up.
20:18
It's easier to prove value every time you go up, and at some
20:21
point you'll be like, hey, the next step is so hard that the
20:24
ROI on it is like. I like, it's our quantifiable so
20:28
well, but also I hear about, like, you know, from all meetups
20:32
and everyone, everyone is always like, oh, I walk through all The
20:34
Time Machine learning all the time.
20:36
And then you always ask the question like, yeah, but what do
20:38
you mean what do you call with real time?
20:39
You know? And usually it's like 15
20:42
minutes, 20 minutes and you're like, well, let's.
20:47
It's it's very hard to do. You have to think about signals,
20:51
right? How noisy are these signals?
20:53
And then you fancy again. I'm sure there are some signals
20:56
like, but I even tick tock doesn't do it like real real
21:00
time because they have to gather enough sample to kind of affect
21:04
the rate. But I probably should get
21:07
somebody from TikTok. I know somebody I can connect
21:09
you, so he can totally come clean and and share.
21:14
Man, well, I, you know, I was thinking about how many
21:18
different edge cases you'll have to and it just.
21:22
Comes back to this idea that you're over engineering
21:25
something that is not going to give you that big of a benefit
21:31
at the end of the day. So how much time are you going
21:34
to put into that engineering? When the benefit that you get
21:39
out of it is not related in any way, it's like maybe the benefit
21:44
is half or a third of the amount of time and effort and energy
21:49
and resources you push into this.
21:51
But it's almost like a badge of honour that we can wear and say,
21:55
like, you know, I'm the one who got that real time shit figured
21:59
out and so now I can go give talks all around the world on
22:02
it. Yeah.
22:03
No, I think there are really cool cases in which real time
22:06
works like, again, don't get me wrong, like pricing and so on is
22:09
just like, but what ends up happening?
22:14
Is that when there is really that need general purpose
22:17
infrastructure is not. The D5 choice right is what
22:22
you'll realise because did they are very customer like everybody
22:25
has these custom needs it one of the things that we found is a
22:29
bit surveying and features survey and all of that.
22:34
The really, really big teams said like, hey, we don't want to
22:37
use this stuff. We want to build our own because
22:38
we have huge performance requirement.
22:41
That being said, there are parts of it that do, which is where we
22:44
put our most efforts and right where we found, like back
22:47
testing or training or collecting data or evaluating
22:52
and so on is almost always the case and you need it.
22:57
But like surveying feature sharing, for certain people it
23:00
can be very very hard just because their needs are like.
23:04
A bespoke. And maybe that will eventually
23:08
all that should be. Maybe everything it solves
23:11
everything is. And and it brings everything
23:14
together, but. As far as I can tell, not yet.
23:19
But how do you balance everything out like because I
23:22
mean, you know, the community would be like, hey, I want that.
23:25
And you know, they'll be like it's really important for us,
23:27
like if you don't build that, we don't use flight.
23:29
But then it might be something extremely specific as well.
23:32
So how do you balance those? Ohh how about this jumping into
23:36
community I think. Fantastic question that have
23:39
been. Lot of people actually have come
23:44
and asked us two things. One is.
23:51
Dusk fusion or. Running really really, really
23:55
fast workflows on flight. And.
23:59
And and and and some other things are like another person
24:02
that's like really long training job.
24:05
It's a distributed, get full observability, let's say metric.
24:09
Sorry. And and I'm going to take these
24:12
two examples. I think we can do both.
24:16
The problem is if I do any of those.
24:19
Anybody who is installing flight will.
24:22
Right. And this is how it is.
24:24
They will love it. They want that effect, but it's
24:26
not going to be easy to install. It's very hard to minimise the
24:31
total number of dependencies that you can take on a product.
24:35
And the reason why you want to minimise is because and
24:39
installation for a large company like.
24:43
LinkedIn, Spotify and so on. They are able to directly have
24:46
10s of engineers, but a small company has almost no straight
24:51
really like maybe three. And if I tell them now you need
24:55
Kafka or you have to go and buy MSK and then they go, OK, I
25:00
don't want to introduce this new thing into a stack literally.
25:04
Another one would be, you know, into observability like the
25:08
first one we we were actually. We will, we are picking, we are
25:13
and that and that's where we started thinking OK how do I
25:17
keep improve or reduce the surface area of what you need so
25:21
that microbial people can get value and that's like.
25:24
And where can we like take away the management overhead?
25:28
For most people, that's. And that's how we do a trade
25:32
off. We are like we literally cannot.
25:35
Even. And The thing is, we also want
25:37
to do it in a very open source site way.
25:41
Because flight is part of the destination, we I don't own it.
25:44
We we don't really go and add like hooks in there.
25:48
Saying that now it goes to Union cloud or something that's we
25:50
don't do that helps you have to immerse yourself in the MLPS
25:55
content. The best way to do it is to
25:58
subscribe to the Inner OPS Community Podcast.
26:02
So good luck and keep learning. The way we do it is we listen to
26:12
a lot of people and we honestly tell them that.
26:15
We agree, some of this is really cool.
26:18
Here is an alternative that you guys go and use and we have
26:21
actually told that before and in some cases we say that doesn't
26:25
exist, but they're what like the same versioning and everything
26:28
with like the performance let's say off like of super fast
26:31
running task and we're like oh you realised what we have to do.
26:35
We have to engineer the way that container startup we have to
26:38
bring in hard containers, we have to.
26:42
Right, certain core that's QA and and like some queues etc.
26:47
Distributed queues, you have to do the reader elections like
26:50
hundreds of things underneath and we can build it right.
26:52
We have built this in the past. But will that improve your one
26:58
use case? Yeah, what about the 95% of the
27:00
use cases that are there today on the profile, right.
27:03
Adds complexity to that. So that's why we built a plug in
27:05
architecture and so on so that we can keep on extending these
27:08
things. And so eventually we hope that
27:11
we can do these certain things. In those cases where somebody
27:15
comes and says I want this, and then you say, all right, you
27:17
know what, actually, it's probably better if you use
27:20
Airflow 1.10 for that. Yeah, I think it scales better
27:25
than two dot. So I'm just joking.
27:27
Underhanded chap. Yeah, No.
27:32
For everybody out there that is on air flow 1.10, we feel for
27:35
you and you're you've got yourself into some fun.
27:40
But before we talk about airflow 1.10, I I did want to stay on
27:43
this idea of like union versus flight for a minute and.
27:48
Think about it seems like you've clearly set aside, hey, we are
27:52
going to use flight for the majority of service area that we
27:56
can cover and we're going to add these plug-in models so that we
27:58
can continue to extend it. And when people come to us in
28:01
the community and they want new features, they can get that.
28:05
And whether it's us giving it to them or it's us creating a plug
28:09
in with another tool that can give it to them, we're going to
28:12
make sure that that's possible, especially if we hear it more
28:15
than once or twice in the same day.
28:18
And. Then for those really hard jobs
28:22
that you are thinking about that are going to take a lot of
28:26
engineering and you're thinking that companies.
28:30
May not have the resources or the power that human power to
28:36
throw at it. But it is a very high high
28:41
quality problem and if someone could solve that for them.
28:45
They would be very happy. And so it's worth you creating a
28:48
managed service around or it's worth you creating some kind of
28:52
paid product around it because you know that the companies that
28:55
do have those problems that don't have the 10s or 20s of.
29:00
Or 30s. However many engineers to throw
29:03
at the problem, they're going to be able to say, OK, you know
29:07
what, we'll give you some money for that.
29:10
Yeah, I'll give you another example.
29:12
In that world is like almost everybody has asked us.
29:15
Build our back into flight the flight it ships with, so.
29:22
There's an SO wall of shame I don't if you know by the way
29:25
basically and we were ourselves using a product many products
29:29
like internally at Union and we have like.
29:32
Ohh, I have to pay for Enterprise just to get so this
29:36
is crazy. What world are we living in?
29:40
And leadership as a so in in like that and I've told you this
29:43
in the past, it causes a lot of I tell the team like I don't
29:46
know why. Because everybody will ask like
29:50
I cannot get key clock working. I cannot get all zero Google
29:55
assist on Google. So like it's terrible.
29:57
It doesn't even have an authorization server, like
29:59
people don't realise. So we have to build an
30:01
authorization server and that has vulnerability problems,
30:03
right people? Because people will not change
30:05
some of the codes and we're like OK.
30:08
You have to know what you're getting into, right?
30:10
This is complicated stuff. This is this is potentially.
30:15
Security hazard. And so you should understand
30:18
what you're getting into, and they're able it, right?
30:21
And that is the reason why we ship with us.
30:23
So there's like you should be secure.
30:26
But please go and you long thing like you just you know we then
30:29
give two or three but people expect that we can work with
30:32
every single product Cognito Azure LDAP, it's unlimited
30:37
number of services right and. Then people will.
30:41
Most of the folks would say, hey, can we get?
30:45
Are back and they're like. There is no standard even for,
30:48
albeit, there's nothing. Think about the crazy amount of
30:52
madness that I'll have to go through, because even just to
30:55
support and it will not be working because you have to go
30:58
and pay this random providers. And whenever you are facing with
31:03
such a thing, it better to go to.
31:06
Potentially illicit service which knows what they're doing
31:10
right like and this is where we can offer and you guys can
31:14
create it an open source that's absolutely open, but it's super
31:18
hard to explain to the community how they have to.
31:22
Super. It's it's completely not lack of
31:25
trying, right? Trust me, it's not lack of
31:27
trying itself. Yeah, but I asked this question
31:30
already. I've been guilty to ask the
31:32
airbag question on the flight. See if it you thought, no, no,
31:37
no, no. I know it's everybody.
31:39
And I'm like, I tell me the standard, let's come up with a
31:42
standard and then let's build it, right?
31:44
Like, what if there is no standard how I'm going to build
31:46
it exactly? Yeah.
31:47
But even like unfolds, internally it's like, yeah?
31:51
And it doesn't work across right?
31:53
Because many people are running in multi cluster mode, like even
31:55
if you use the cooperative oppa kind of stuff that's.
31:59
So it's yeah, it is I I have not found a single solution, but
32:02
maybe I'm just naive and and this is a community.
32:05
Please contribute. That's what I say.
32:08
PR's union does ship with Iraq. We know how much work it took
32:13
just to make our bed possible. You know the the set of things
32:18
that we wanted to do was. He within a company that could
32:21
be a financial infra team. They could be a a a data infra
32:25
ML and for Emma product right? And some may be dealing with
32:29
very. Private data like customer data
32:32
and you have to keep them secure and that's where we provide RBAC
32:35
and we take it all the way down to a single word.
32:39
And this takes a lot of effort. So it's possible.
32:42
We know you can do it with light.
32:44
We just don't know what service we actually have to connect with
32:47
some other services, pay them money to that.
32:50
You're doing all of that, folks. And So what are some of these
32:54
other high value features that you have incorporated into?
33:00
Union on off the back of flight, right?
33:03
That aren't really proper for flight because of the
33:07
engineering work and the amount of time you need to put into it.
33:11
And also again. The overhead that you will need
33:16
to ship flight with. Yes I'll tell you a bunch like
33:23
maybe a non trying to steer but the and I'll give you a reason
33:26
behind each one which is first time ever probably open source
33:31
company is going to say this but so flight supports multi cluster
33:36
but it's not dynamic multi cluster you have to configure
33:39
the clusters because if you think about it most people are
33:41
running a single cooperative cluster.
33:43
And that's why. That worked for 80% of the
33:47
cases, and we've tweaked a single cluster to almost the
33:51
maximum. We know companies that are
33:53
running hundreds of thousands of workflows and power which is and
33:59
each of them having hundreds of tasks of millions of tasks.
34:03
An hour on flight and it works absolutely fine in its scales.
34:07
OK, in there also we decided to go like a single binary for the
34:11
absolute beginner, but you can scale out to multiple pieces and
34:14
so on. But in Union what we found is
34:19
that. You were talking to users and
34:21
they're. I don't get GPS, I guess what do
34:24
I do? Can you give me GPS?
34:26
I'm like, I mean I have to do a GPS myself, but I know GCP has
34:30
GPUs or Oracle cloud has GPUs et cetera.
34:34
So what we do in union is you can bring a Kubernetes cluster
34:37
or any cloud and connect to union or within a cloud and some
34:41
people are like, hey we want to separate out different projects
34:44
and. In GCP or certain things, and
34:48
all of that is supported out-of-the-box on the union
34:49
where you can bring in clusters, clusters have their life cycle
34:53
and you can drain them, etc. Because we inverted the model
34:58
out of the need, our first goal was to actually get.
35:02
Yeah, you need to work on laptops.
35:04
And that's that's what we started.
35:06
So we had like 4 laptops and you can connect four laptops and
35:09
workload to be sent to four laptops dynamically.
35:13
The second one was of course RBAC.
35:15
I told you the reasons SSAB configure the SSO, so you have
35:18
to do the wrong thing because we pay for it.
35:20
It's not free, but we can figure it.
35:24
Lord is. Another thing that when we were
35:28
talking to people, we realised everybody wants to get deep
35:31
visibility into what's happening.
35:33
Like let's say you're training a model and you're like is my GPU
35:37
getting saturated? Is my CPU getting saturated and
35:40
never getting saturated and? And So what we call that is task
35:46
level monitoring. So you can go to an individual
35:48
task and in unit allows you to dig deep into it and it gives
35:52
you all kinds of metrics like how much storage are you using.
35:55
Now if you realise yes this can this be done in open source, I
35:58
think people can do it, but you'd have to host a different
36:01
set of monitoring infrastructure.
36:03
And this is like, again, I can add it and people were like, OK,
36:06
you're putting data or whatever. Yeah, we don't do data, by the
36:11
way. And the other other thing that
36:14
we realise why this is important to think about is we are not
36:19
designed APM Systems today to solve these problems for our
36:23
millennial and I'll tell you how.
36:26
Data Dog with red all of these appeals system.
36:30
It's not one, it's almost everyone of them.
36:32
Charge based on the cardinality. Cardinality is based on number
36:36
of metric unique values. Pipelines that experiments are
36:43
all unique. And at scale in that company,
36:47
already hundreds of or thousands of experiments in a short period
36:52
of time. And they die away, they go up,
36:54
they don't exist. But because of the retention
36:57
policies within the system, you will get charged at the end of
36:59
the month for every single experiment you ran.
37:03
And actually many types. My often what I said to people
37:08
is like, I would love to do a business where I am paid for
37:12
observing you do something. That's a PM for me.
37:15
You just come and observes, you do something and it's like,
37:18
alright, here you go, Amazing. I'll pay you $60 million.
37:23
Sadly, I'm not that company. Nobody pays me for to do that.
37:29
But so we thought, this is actually a massive problem, so
37:32
let's solve that. So the TLM is included in Union
37:36
Cloud for free and we've designed it in a way that the
37:39
cost of operating it for us is not.
37:42
Enormous. So we can observe individual
37:46
things at a very, very low cost. The secondary thing is we
37:51
realise, OK, you do this at an individual level.
37:54
But when there are platform, there is like Stephen, right,
37:56
he's he's offering a service he wants.
37:59
To know across this company who is.
38:03
Using our GPUs. And then because you know that
38:06
would be may come and ask like OK yesterday we spent
38:09
$1,000,000, can we identify what we did and is there an ROI for
38:13
doing it. So you need a top level like a
38:16
dashboard. Again this is observability.
38:19
So that gets added on it like it allows you to drill down by a
38:22
project over and workflow figure out who's running what, which
38:26
where you're spending are you. The other interesting thing
38:29
through our many years you have found is that most people over.
38:34
Calculate the amount of memory or CPU et cetera that they need
38:37
and end up using 50% or lower than that.
38:40
That is 50% savings on your tip. And we show you right off the
38:44
bat like, hey, here's a 50% optimization chaps, go change it
38:48
as you will see if eventually you are automatically into some
38:52
of these things. And that's why we're really
38:53
capturing some of this stuff. Yeah and and so these are three
38:58
features along with that we are working on how deep lineage and.
39:05
Shedding of artefacts, etcetera, that work.
39:10
And and that's about it. Like we we have few other
39:13
smaller features which we manage everything for you.
39:16
So we have kind of built. Like including, Like most people
39:20
don't get it. You don't have to manage
39:22
Kubernetes cluster. You don't even have to know
39:24
you're running cooperative cluster.
39:25
You don't have to roll you're running machines or scaling
39:28
them. You just tell us pointers to an
39:30
account and we manage all of. At any scale.
39:35
That's our goal. Boom.
39:37
That's so cool. Yeah, it's really cool.
39:39
It's inspiring. Like, especially like, I felt
39:42
seeing a couple of times when you were talking when, you know,
39:44
like the data dog observability or like using resources like
39:48
crazy in a data scientist, they're like, yeah, I won,
39:51
there's a hundred, you know? Yeah, it's.
39:57
Like, I want all of them, yeah. And doing it uniformly across
40:01
every cloud, across every deployment is a super art
40:04
challenge. And that's what we've been
40:06
trying to solve. And we're doing it securely,
40:07
making sure there's no security availability.
40:11
It's our chatting and we think once you start using it, there's
40:15
no way you're going to go back. Well dude, there is one thing
40:19
that I wanted to finish off with which is.
40:23
We started or in the beginning of the conversation we were
40:27
talking about how data engineers come with their certain wants
40:33
for what? Flight offers and how mill
40:37
engineering is a bit different. I get the feeling since ChatGPT
40:42
came out, there's a lot of. The LM's that have come out and
40:48
how that's different than quote, unquote classical ML.
40:51
What have you seen and what do you think?
40:53
What are your thoughts on that that have been folks that have
40:57
had a chat with her like hey, we are building an NM
41:00
infrastructure in IT, Why? What do you mean by that?
41:04
Like, let's break it down. You're taking a model?
41:08
Sure, that's a checkpoint from the past and you're fine tuning
41:12
it. And that's what most people want
41:14
to do. If you are thinking of fully
41:16
training a model from scratch, I question whether that's the
41:22
right thing to do. In this day and age where Llama
41:25
Two is open source, I unless you have something really found a
41:31
great insight which most companies would not have, I
41:35
would not do such a thing. And then it mostly fine tuning.
41:42
And if you're doing fine tuning, it's still entertaining.
41:46
The hard part is there are two hard parts here, or three parts
41:50
which are common. Take data, the most important
41:54
part. Take your data and get it into a
41:56
format that can be used. Because you're going to change
41:58
the prompt, right? So then and then send it to the
42:02
through the training cycle. Uh, update to your prompts or
42:05
maybe even send to multiple training like you might have to
42:08
do retraining to answer questions, et cetera, and then
42:12
evaluate. You should evaluate before you
42:15
deploy and then the final part is deployed trade and this is
42:21
that I'm editing infrastructure now let's go to classical music.
42:24
You get data. Father Martin evaluate, they
42:29
deploy OI. Don't think it differs.
42:33
The big difference is that all models are best, and so you
42:38
basically they're hungry for cheese.
42:41
But you should also wonder whether the biggest model is the
42:46
right model for they depends on their problem domain and I don't
42:51
know how to prove it but to try so.
42:56
So what we do and we looks like you're being we have an elastic
43:01
Lugin which essentially allows you to train models on multiple
43:07
GPUs, could be on one node on multiple nodes and it uses like
43:12
the torch run which is the default that you see everywhere
43:15
and it automatically scales it out and so on.
43:18
And also there are plugins for types, so you can create, you
43:23
know back to our state, although that's and so on.
43:25
So we can optimally use the GPU, that's what you need.
43:28
You just need that. There's no way to improve
43:31
performance of training by putting any kind of cash or
43:35
anything in between. You have to use raw networks so
43:39
that GPUs have to communicate with themselves and using the
43:42
best back end like nickel or or EI and then communicate with
43:47
each other if they're on on a node and they have to be placed
43:50
up. This is all infrastructure
43:53
orchestration, yeah, yeah. And then you want to do a
43:57
checkpoint day. So we did this.
44:00
That's why we added hugging face as an integration with flight
44:03
now. So if you people install
44:04
Transformers it has a flight callback.
44:08
You then you will be able to get metrics of all the execution as
44:12
well as checkpointing. So you can just simply say
44:15
adding phase checkpoints like to automatically checkpoint your
44:17
code. So you can start using Spark
44:19
machines if you want to train it, bring down the cost is
44:23
consistent, and then recover the training.
44:26
You can do all of those things and it's all available.
44:29
Or you can use a. If Ray is what you want to use,
44:33
you can use Ray. We don't pick a choice.
44:36
We do not see a massive user rate if you are.
44:42
Most models can be fine-tuned unlike 8A1 hundreds which is 1
44:48
node and you put that on one Lord.
44:52
The the net interconnect is very efficient between them right?
44:56
It's on them device. And what you need to really need
44:59
to do is you need that TLM, the task level monitoring to make
45:02
sure you're saturating your GPU and make sure you're saturating
45:06
your CPUs on those machine to process.
45:10
If you do that, you will get the maximum out of your process, but
45:14
other than that it's the same step, right?
45:16
And then this is what we do for vision models, this is what we
45:19
do for speech models, this is what we do for classical Xgboost
45:23
models. Like doesn't matter, it's the
45:25
same stuff. Now, yes, there are some special
45:28
plugins and they exist. Yeah, I guess for me the only
45:31
thing that is missing there and that I always think makes.
45:35
Dealing with large language models.
45:37
A bit different than traditional quote unquote, traditional
45:40
e-mail is the whole prompting. I think the hardest part is
45:42
evaluation. Yeah, that's that's a great
45:45
point. Yeah.
45:47
Those two those two things just prompting it and then evaluating
45:50
it, it's a huge task and it's not necessarily, I mean I guess
45:56
evaluation you could kind of look at is.
45:59
Accuracy, but not quite. And so, how do you even know?
46:06
Yeah, that's it. Yeah, but if you hallucinate,
46:09
how do you know your it is hallucinating?
46:13
But you have to. It's like as hard a problem as
46:17
searching a needle in a haystack, right?
46:18
Language in the haystack. Here is the Internet, maybe
46:22
there is some document that says that is right.
46:26
So that's not your problem domain.
46:27
Is interesting right? You can if your problem domain
46:30
is limited, you can evaluate on that problem domain.
46:34
And I think Hugging Face has been doing some really amazing
46:36
work on this side, like creating the leaderboard and so on, and
46:39
it's expensive. They have the money.
46:42
So it's really, really amazing stuff that they're.
46:45
Doing well, do you think do you think because I've heard like
46:47
they consensus being using LM's to evaluate.
46:52
Prompts for you at Allens. Do you think it's something
46:56
that's we're gonna have more in the future?
46:57
Like I I tend to talk about it by someone from Google and they
47:00
were like, yeah, we tend to use elements to evaluate our
47:03
products and. And the roofing, I guess my only
47:06
thing is probably you need a benchmark still somehow right?
47:11
Like if it evaluated once you create a benchmark.
47:15
Almost I don't know about manually or some somehow which
47:19
creates that like like get your golden set.
47:22
Yeah, you say like use my watermark.
47:25
If I'm hitting my watermark and improving on it, I'm most likely
47:29
improving. But again, it's most likely it's
47:31
not 100% and it depends on your application, right?
47:35
If you are using this to again administer loans, don't.
47:40
Maybe not. Maybe help somebody in Slack to
47:46
answer a question and you're putting a caveat over there that
47:50
I am a bot. I may be wrong.
47:53
So it's amazing application of it, right?
47:56
And like, whenever there's what I found is whenever there's a
47:59
human in the loop, the applications really tend to shy
48:05
because you're you're what. This is the promise of Mill.
48:09
And that's what we are doing. We are augmenting a human.
48:12
We are reducing this stupid stuff that you'll get sorry to
48:16
buy or and just give it to you. Like here's my four things.
48:19
Can you pick one for me? An exam?
48:21
Right. And then and then the human does
48:23
the job and based on the context that they already have, which
48:27
may not be written now and then, you know then I feel.
48:31
But besides that, going to your original question, I think the
48:34
infrastructure is still the same, serving is still the same,
48:37
there are fantastic work has been done, like suddenly I feel
48:41
quantization is working really well, can really freeze all of
48:47
these modules and shrink them to work on whatever hardware
48:51
nowadays, It's amazing so. Well, dude, Katon, this has been
48:57
awesome, man. I appreciate every time that you
48:59
come on here. And today did not disappoint.
49:03
So I think we're going to end it there.
49:05
Ben, thank you. Thank you for having me.
49:07
It's always fun. And thank you for having
49:09
Stephen. You get extra brownie points for
49:10
that. Thank you.
49:17
Hi, this is Michael Eric. I'm the founder of Pametan Data
49:21
Innovation and Confetti AI and this is a quick public service
49:24
announcement. If you're tired of having
49:26
nightmares about the machine learning models you have in
49:28
production, then I encourage you to subscribe to the ML OPS
49:31
Community Podcast. It's where all the cool ML OPS
49:34
kids hang out and learn how to tame their ML models.