Latest / Agentic Conversations (formally mlops.community) / Kubernetes, AI Gateways, and the Future of MLOps // Alexa Griffith // #294

Transcript

Kubernetes, AI Gateways, and the Future of MLOps // Alexa Griffith // #294 · Agentic Conversations (formally mlops.community)

0:00
My name's Alexa Griffith. My title is I'm a senior
0:04
software engineer at Bloomberg. I love whole milk, so I just do
0:09
regular coffee and milk. I'm a savage.
0:12
I'm weird. Welcome back to the ML OPS
0:15
Community Podcast. I am your host Demetrios.
0:19
And today there was one gigantic piece of this conversation that
0:24
I cannot stop thinking about. And that was when Alexa
0:26
mentioned how useful it has been for her to have mentors and be
0:33
on teams that could choose problems with real business
0:38
value. And how that has translated for
0:41
her in her career to working on a whole lot of open source
0:46
projects because they were able to properly champion for the
0:53
need to work on these open source projects and tie it back
0:58
to how it would help the companies that she has been at.
1:03
I love hearing that. I loved talking with Alexa.
1:06
Let's get right into it. And for those of you that are
1:13
listening in on good old Spotify or Podcast Land, we've got a
1:21
song for you today. And that is keeping in theme
1:26
with Japan because Alexa just got back from there.
1:30
I want to play you this song by air is called Lost in Kyoto.
1:38
No, sorry. It is called Alone in Kyoto.
1:43
And Air, if you haven't heard of them, they're from the 90s, but
1:47
really like they're timeless. They are just timeless.
2:06
Let's get into this podcast with Alexa.
2:10
Wishing you all a very special day.
2:15
Hope you enjoy. But tell me about Japan, because
2:48
I really want to know you were skiing there.
2:50
Yeah, it was awesome. It was crazy.
2:53
There was so much snow, over 3 meters of snow.
2:55
I mean, it didn't stop snowing there.
2:57
I went to Hakuba actually, which is right outside of Tokyo, but
3:01
it was amazing. I know this echo is pretty
3:04
popular, but yeah, it was an amazing trip.
3:07
I think Tokyo is really great too.
3:10
We went to Seoul as well, so I hopped around for a bit.
3:13
I was gone about two weeks, so it was a long trip, but it was
3:17
so amazing. Yeah, I know that you just got
3:20
back and you are still having fun with a little bit of jet
3:23
lag, which is always nice. I found a trick for when I do
3:27
the trips across the pond to go back and visit family.
3:31
And if you wear compression socks, then the blood doesn't
3:37
like coagulate in your feet. And apparently by keeping the
3:42
blood flowing, it helps in some magical way combat jet lag.
3:48
Wow, that's amazing to know. OK, yeah, I'll keep that in mind
3:51
for next time for sure. But that's my problem is that I
3:55
didn't wear those socks. You should have worn the
3:58
compression socks. That in a lot of water
3:59
apparently helps a ton, and then seeing sunlight in the place
4:05
that you're at as early as possible also helps.
4:08
But. It's a good tips.
4:10
Yeah. So you've been in tech for a
4:13
while. I think that you've got a really
4:15
cool story. And actually what I want to
4:18
start off with is that you've, you've written a lot about your
4:23
tech journey. And one thing that you wrote
4:26
about was using Airflow in a past job, right?
4:29
And using Kubernetes and Airflow.
4:31
And I am fascinated by what to this day do you remember as
4:39
being like the biggest pain when it comes to Airflow and using
4:44
airflow. It's funny you asked that
4:46
because it has been a while and we were using airflow like we
4:52
were running it ourselves, building ourselves, pulling it
4:54
ourselves and our data scientists were building a lot
4:57
of dags, dags with it. I remember that we also built a
5:00
few pipelines for the data scientists as well.
5:02
I was on a data science infrastructure team, so that was
5:04
part of something that we own. And I my first task was getting
5:11
logs to show an Airflow so that people could view the logs of
5:14
their jobs. And basically we just had logs
5:16
going into a database and then I was pulling from that database
5:19
and showing them on a screen while the the task was running.
5:23
So I think there were a lot of operational things that we had
5:27
improved and there were a lot of things that improved within
5:30
Airflow as well. We had like a fork of it at the
5:32
time too. What I remember the most was
5:35
just it would crash a lot. It couldn't handle a lot of
5:39
workloads running and spanning at once.
5:41
Now this was right when I started, like I said, so maybe
5:45
there there was a resource thing or maybe tuning the resources
5:48
was also an issue. But I think our scheduler would
5:51
kind of crash out a lot at what we were trying to do, which also
5:55
begs the question, like maybe there was a better tool for, you
5:58
know, the, the large fan out that we were trying to do as
6:03
well. But I remember that at the time
6:05
was an issue. It was really great to start off
6:08
writing about airflow for me because as you mentioned, I, I
6:12
have AI would say atypical, but I feel like a lot of people now
6:16
in tech are a little bit atypical too.
6:19
I didn't have ACS degree. I had a chemistry degree.
6:23
I actually studied computational chemistry.
6:25
I did research in computational chemistry, and so that's kind of
6:29
how I jumped into the world of software engineering and a way
6:34
to learn and digest that information.
6:36
I started writing about it and that's kind of how that came
6:40
about. And it was really great that my
6:42
first project was Airflow, because at the time so many
6:46
people were also interested in how other people are using
6:49
Airflow and the problems we're having, how to best set up a
6:52
DAG, how to structure it and make the best use of it.
6:55
So it was really great. It was a really great
6:58
opportunity for me that led into a lot of other things.
7:01
Yeah, it's, it's interesting that you say like, oh, maybe
7:04
this wasn't the right tool for the job.
7:06
But back in the day when you were using it, I think it was
7:10
kind of the only tool for the job.
7:12
And there wasn't this maturity that you see now.
7:15
There's all kinds of pipelining tools, whether you're doing ML
7:19
pipelines or you're doing data pipelines, you have a lot more
7:23
options. But just four years ago, even
7:26
you didn't have those options. And that's what spurred.
7:29
I think a lot of people like yourself had experiences where
7:33
Airflow is crashing or it wasn't working in the way that they
7:36
wanted. And then they went and they
7:37
started their own companies because of that.
7:40
And so you see now the maturity around the pipeline space.
7:45
That's so true. I mean, I remember at some point
7:48
we started to discuss, OK, now should we look into coop flow
7:52
because coop flow was becoming popular?
7:55
Should we look into Argo workflows?
7:57
I mean, we used Argo CD at the time, which is a great tool to,
8:00
I mean, so nice to use. I remember when they implemented
8:03
Argo CD, it was like, wow, this is amazing that we can manage
8:06
our deployments and see all of our resources in such a great
8:09
way. So, yeah, Argo, it was such a
8:11
hit and still is. But yeah, I, I think we started
8:15
to kind of think about other things that we, we had invested.
8:17
And now I don't know if they're still using it or not.
8:19
Maybe they have changed. But yeah, it's right.
8:22
I think even I've been in tech for, I think five years now or
8:26
so. I, I, yeah, coming up on 6/5 or
8:30
six, who's counting? But I think it's sometimes easy
8:34
to lose that context of how much has changed.
8:36
So I'm thinking now there are other workflow tools, but then
8:39
you're right, they weren't as mature as they are now.
8:42
And how quickly that changes. Crazy.
8:44
Have you played around much with Argo workflows?
8:48
Personally no. We have a team that manages Argo
8:51
workflows but I I have some for jobs and it's super useful and
8:58
from my understanding pretty easy to use.
8:59
Similar to Argo CD. It has a similar layout from
9:02
what I understand, but personally no, I haven't used it
9:05
so much. Yeah, 'cause I I always wonder
9:08
about the different pipelining tools that are out there.
9:12
At the end of the day, you've got these dags.
9:16
And I was just on a conversation earlier this week with a guy who
9:20
was talking about how basically everything is a graph and
9:24
everything is a workflow in one way, shape or form.
9:28
Whether it is a technical workflow or it is a business
9:32
procedural workflow or flow chart.
9:35
You can look at it and and he mentioned that usually the
9:40
business procedural flow charts are much more complicated than
9:44
the technical dags because he was like, yeah, and his name is
9:48
Alex, Alex Milowski. And he was saying how a
9:53
technical DAG in general, you get like 3 or 4 steps.
9:58
It's like, I wanted to do this and then do this and then do
10:01
that. Going back to the original idea
10:03
on why I was mentioning this with Argo workflows is that you
10:06
have different folks with different backgrounds and needs
10:10
that will come to pipelining tools with their ways and
10:17
opinions of doing things. And Argo workflows, I think
10:22
folks who are coming from a dev OPS background really tend to
10:25
gravitate towards that. And then you have the air flows
10:29
or the Dagsters in the prefix and mazes of the world and folks
10:33
who are data engineers can understand those or like those a
10:37
little bit better. And then you have the cube flows
10:39
and the meta flows and Xenomels of the world and folks who are
10:45
like ML engineers kind of vibe with those better.
10:47
And so you get all the all this space to play with your
10:52
pipelining tools that you like. And this is completely
10:56
forgetting about the pipelining tools of the like no code low
11:01
code pipeline tools. Let's just take those out of the
11:03
picture. But it's fascinating to think
11:07
about that and how each background lends itself well to
11:13
one type of tool or one type of space that you'll get into.
11:18
Yeah, that's so true. I think you see that across a
11:20
few different things, especially in the AIML world where you have
11:24
the infrastructure people creating things and then also
11:28
the AIML engineers using it. So even, you know, I think they
11:31
favor or AIML engineers favorite Python.
11:35
Typically it seems like infrastructure engineers
11:37
favorite Go or YAML configs. And the difference is in the how
11:40
you interact with the APIs or the tools for sure, I've seen
11:45
that quite a bit. Have you found ways that you see
11:51
the interactions between those like 2 personas do better or
11:56
worse? Because there is a very much
11:59
like opinionated side for the infrastructure folks versus the
12:04
opinions of the ML and even data scientist.
12:07
That's almost like very removed opinions and workflows and what
12:11
they need to get done. But I wonder if you've noticed
12:15
things that have helped bridge the gap between these different
12:19
personas. I think you're exactly right.
12:21
It's they are user of our platform and but we're both
12:26
engineers. So it's an interesting it's a
12:29
it's an interesting interaction. And I think there's a lot of
12:31
work now and it's not exactly always easy, but to abstract
12:35
away Kubernetes and those may the things that what's a paw
12:40
with a container? What's you know, I mean, and you
12:42
can get really like, what's a virtual service?
12:44
Do they really need to know that and trying to extract as much
12:48
away as we can from them. I think that that seems to be
12:51
the move, just be able to clearly tell you based on the
12:55
three different types of errors that you get in your YAML spec.
12:58
Like if there's an error, like what does it actually mean for
13:00
the person that deployed it, who maybe doesn't really know much
13:03
about Kubernetes? Like what action do they need to
13:05
take? Because there should be some
13:07
action they need to take, something wrong with their
13:09
service they can fix. And trying to bridge that gap is
13:13
something we've been working on a lot that I found quite
13:15
interesting too. And at first my reaction was,
13:18
what do you mean they don't need to know about Kubernetes?
13:20
You know, like from our side, like of course they need, they
13:22
have a, they're running it, they have a container.
13:24
But I mean, you see it more and more and, and in general,
13:27
Kubernetes has so many abstractions on top of it to
13:30
build all these tools. I mean, that's what a lot of
13:33
Kubcon is a huge Kubernetes conference.
13:36
That's a lot of, you know what it is a ton of different tools
13:39
that people have built on top of Kubernetes to make it easier to
13:41
run in all these different ways. So I think that seems to be kind
13:46
of what we're working on is trying to make sure they don't
13:48
have to need to see all that stuff.
13:52
Speaking of Cubecon, you spoke recently right at Cubecon in the
13:56
Cubecon AI Day. Can you talk to me about what
13:59
you what your presentation was on?
14:02
Yes. Recently I gave a keynote at
14:04
Coup Con for the Envoy a Gateway.
14:07
It's a new open source project that the engineers from
14:10
Bloomberg and Tetra have been working on together.
14:13
Again, it's an abstraction layout basically on top of the
14:17
Envoy Gateway project, which is a server proxy that is used in a
14:21
lot of our or is used in our inference services to manage
14:26
things like traffic rate, race, request it like laws and
14:31
monitors traffic flow. So it's super useful for us.
14:34
We use it. And when all these G and AI
14:38
models are coming out, there's a lot of different complications
14:41
that come with them compared to the inference services in the
14:45
past. Like they're way larger, they
14:48
use tokens. So for those reasons, they have
14:52
a different set of problems that need to be solved.
14:54
So no longer do we want to rate limit a service based on a
14:59
request, but will be really useful as to be able to rate
15:02
limit a service based on the number of tokens, because that's
15:05
the unit of work that we're trying to trying to distinguish
15:09
against there. So that was a really cool
15:12
experience for me because I never talked in front of that
15:16
many people before. And it was really cool to
15:19
introduce a new, a really, really interesting new open
15:23
source project as well on behalf of, you know, all the engineers
15:26
that have worked together at Tetra and and Bloomberg.
15:28
So I'm super excited about that. Yeah, it's always fun.
15:32
And especially when I think I heard a stat that 95% of people
15:37
would rank public speaking as a greater fear than dying.
15:43
Oh. My gosh, really?
15:45
Yeah. OK, that's funny that yeah, it
15:48
is for a lot of people, that's true.
15:50
And I was nervous, but in an excited way.
15:53
So talk to me about the difference or how Caserv and the
16:00
proxy that you set up, what is it called?
16:03
It's envoy. Envoy.
16:04
I gateway. Yeah, Envoy AI gateway, which is
16:07
different than Envoy, right? There's an Envoy, and then
16:10
there's an Envoy AI gateway. Yeah, Envoy AI Gateway is the
16:14
new project that builds on top of Envoy Gateway.
16:16
It basically just adds some more features that are specific for
16:21
these LM Gen. AI models.
16:23
OK. And how does that interact with
16:26
K serve? Like where and how does the
16:29
stack look with those two? Yeah, yeah, for sure.
16:32
So I mean, I've kind of assume that most people know what
16:36
Kubernetes is here, but it's these are all open source
16:40
projects that are built on top of Kubernetes, which helps you
16:43
to run your services at scale and to manage them.
16:46
So Envoy is a service proxy. So every, every request that
16:51
comes into your cluster or clusters is routed through can
16:55
be routed through Envoy. So this helps specifically with
16:58
things like, you know, traffic control, traffic management,
17:01
observability, it can do request routing.
17:05
So things like that about managing the request and how it
17:08
moves through the system. But yeah, so K native is I'll
17:13
just mention that also it's another tool or a building block
17:15
of things we use. So K native serving is the the
17:19
main project within K native that we use.
17:21
Also open source. It's great for helping to run
17:26
applications without, with this idea of being serverless, you
17:29
know, without having to worry too much about the
17:31
infrastructure. So again, it's a it's a
17:34
abstraction on top of Kubernetes with the goal of just making it
17:37
easier to set up and run a service that's serverless.
17:41
That by serverless, you basically mean something that
17:45
can provision resources dynamically doesn't really care
17:48
about so much about what server these resources you don't have
17:51
to specify that it figures it out for you.
17:54
So it makes it easier running. It's all to make things easier
17:57
to run because if not, you have all these YAML files with all
18:00
these configs, you can be copy and pasting them and you're
18:03
setting up all these resources to make sure everything runs.
18:06
I really like running coup cuddle tree.
18:08
There's a tool called coup cuddle tree.
18:10
And from that what you can see is a tree of the top resource
18:14
you have because we in Kubernetes, all these different
18:16
resources like deployments pause and there's a few more depending
18:20
on what you're running. And it can show you all the
18:22
different resources that are made and you can go into the
18:25
namespace and like see what the configurations are and what
18:27
they're doing. I find that super helpful for
18:29
like understanding these types of things.
18:31
But basically these tools help you to automatically set up all
18:35
these things so you don't really have to worry about it.
18:36
So they're all making some assumptions that most everyone
18:39
needs these configs. And usually for most tools that
18:43
if you need something more specific, well, you can add that
18:46
or change it as well, but not so you need to know and specify
18:50
everything yourself. But K native is so that that's
18:53
the instruction to help you just to get things running in a
18:56
serverless way. It helps you with auto scaling,
18:59
so scale up and scale down your services based on things like
19:03
usage. And also to scale to 0, which is
19:06
something we actually use quite a bit because, you know, GPU
19:10
resources are limited. So some services, if they can,
19:13
they can use this thing called scale to 0, this feature.
19:16
And if you can make a setting like, oh, if my I haven't gotten
19:20
any request in an hour or two, then scale to 0 and let someone
19:24
else use that GPU resource instead of taking it up without
19:27
being active. Some can't use this, you know,
19:30
but when you can, it's a quite useful tool to have.
19:33
So yeah. So all that to say, those are
19:35
the building blocks of Caser K serve is another abstraction
19:40
that simplifies actually running AI models, AI services, so
19:45
inference services themselves. So basically what it is is you
19:49
can have this really short YAML. It's like this big, if you're
19:52
using a model out-of-the-box or some some AI like a llama or
19:59
something, they have support for that.
20:00
And what you can do is you can just easily get it running in a
20:04
serverless way. So that's what's nice about it.
20:06
So you don't have to worry about all these configs.
20:08
The goal is that you don't have to have this whole team that
20:10
knows everything about how to run AI models.
20:13
They can focus on other things like building the platform as
20:15
well. But yeah, so it helped, it makes
20:18
it a lot easier to run these services.
20:21
OK, so so basically the gateway is on the front end.
20:25
As the traffic comes in it can dynamically provision and then
20:29
you have the K native which helps provision.
20:33
So it sees like, oh, Gateway is saying that we need more
20:37
resources, provision those and then K serve is built on the K
20:43
native in that 01 of the resources that we need is to be
20:49
able to ping this model. And the model might be a large
20:53
language model or it might be some kind of a random forest
20:57
model. It doesn't really matter in that
20:59
way. Is that what I'm understanding?
21:01
Yeah, yeah. So, so K service specifically
21:04
like K native can run just serverless services no matter
21:08
what, but K service specifically for AI models, ML models, things
21:12
like that. Exactly.
21:12
So it has a lot of out-of-the-box tooling specific
21:16
just like Envoy AI gateways specific for AI models.
21:19
So is Caserv. It has a lot of out-of-the-box
21:21
tooling and features that are specific to what you need for an
21:24
AI model. So it can it can support like
21:27
easy configs for running a lot of out-of-the-box models like it
21:30
easily sets that up or you can have a a custom predictor is
21:34
what we call it. You can like write your own
21:35
predictor. So we just basically spin up the
21:38
service. You get an endpoint, you can hit
21:40
it, and we have this unified API which is really useful because
21:44
all these different model providers can have different
21:47
access patterns about how to hit the model.
21:49
SO1 really nice feature of Caserv is that no matter which
21:52
model you're using, you can always use the same unified API
21:57
and an extension of this or on the AI gateway is an extension
22:01
of this. It also as one of the three MVP
22:03
features has a unified API. So if you're trying to reach a
22:07
model in bedrock or you have something on Prem, it shouldn't
22:11
matter. You'll have one unified API
22:13
going to the Alan Bay AI Gateway and it will it will under the
22:17
hood direct the traffic with the correct the correct structure to
22:22
wherever you're you're using your model.
22:25
Oh, nice. Yeah.
22:27
So that that was actually one of the other questions that I had
22:30
is like, does this only allow you to use as on Prem services
22:35
or open source models? Or can you also dynamically send
22:40
some stuff to the open AIAPI and then maybe you want to send some
22:44
stuff to Anthropic or whatever. If if you do have your own
22:48
services and your own open source models, you can send it
22:51
to those two. Yeah, so Envoy AI gateway allows
22:55
you to be able to run cross cloud like hybrid cloud and and
23:00
that's one of the big features and one of the reasons that the
23:03
problem arose as well. It's because all these different
23:06
cloud providers have different ways of accessing their system.
23:09
But yeah, I mean case of itself is, you know, similar to
23:11
something like stage maker Google vertex if and in cloud
23:16
you can definitely run K serve as well if you're running if you
23:19
want to manage your own, you know, infant services.
23:22
I mean, it's definitely helpful if you don't want to use one of
23:26
these cloud providers if you want to, maybe if you're worried
23:29
about cost savings and you want to manage something yourself, a
23:32
lot of people can use it on cloud as well.
23:34
But I mean also there are these products for inference running
23:37
infant services that are also very similar, have the same
23:40
goal. So the Envoy AI Gateway is and
23:48
just stick with me here because I know I'm a little slow on it.
23:51
But no, no, you're. Good first time I'm I'm really
23:54
digging into it and I really like these AI gateways.
23:58
It's not the first time that I've heard of it, but I do find
24:02
that it is a problem folks have, especially when you get rate
24:06
limited so easily. If you're using external
24:09
services, you want to have some kind of a fall back plan.
24:13
And so it's almost like Envoy AI Gateway is an abstraction out
24:19
and you can throw whatever endpoint you need underneath
24:24
that. So whether you're using a Sage
24:26
Maker endpoint or a Vertex endpoint or AK Serve endpoint,
24:31
you can link those all up to the Envoy AI Gateway and it will
24:37
figure out where the request needs to go depending on what it
24:40
is. Yeah.
24:41
Yeah, exactly. Yeah.
24:43
It has a unified API where you can easily specify, you know
24:47
what you need to and it'll auto route for you, which is great.
24:51
It's all about just making it easier to run and easier to
24:53
manage. And yeah, you start to see
24:55
patterns. Like I said, everything's kind
24:57
of starting to be built on top of Kubernetes and built on top
25:00
of these other tools. Always the goal to just make it
25:02
easier to run and not have to worry about the infrastructure
25:05
so much. I mean there are a lot of also
25:08
really cool or really interesting problems brought on
25:11
by these large language models and gin AI systems as well.
25:16
The model size are so large that downloading the model it takes a
25:18
long time as well. So one interesting problem that
25:22
K service starting to work on is the model cash being able to
25:25
cash models and not have to download them every time a pod
25:28
sucks up for every pod. So that's something like little
25:31
things also that will be super helpful like GP working on GPU
25:35
utilization as well, of course, because GPU's right now are
25:39
resource limited. So a lot of these issues around
25:43
running and getting enterprise, enterprise AI up and running,
25:47
being useful and being optimized is something that we're
25:51
definitely working on. Yeah, that cold start is it's
25:56
like you can't while sometimes, yeah.
25:59
And you can't really base services off of that if you're
26:02
like, Oh, well, yeah, just come back tomorrow and maybe it'll be
26:06
running. We'll see.
26:08
Yeah, I mean it's wild how large models have gotten in such a
26:13
short amount of time, how much more they're able to do as well.
26:15
I mean, I think what was it? Llama 3.1 just came out and it
26:20
has over it can take over 128,000 tokens.
26:24
I mean it has a also has a crazy storage size, right?
26:28
Like it can't fit on one node. So yeah, it needs at least 2
26:34
notes around, which is also makes it more distributed and
26:37
you have to tackle those problems as well.
26:39
So I mean, I think it's great. There's just new problems and
26:43
things are changing a lot, but it's a very fun space to work in
26:46
as well. Yeah, well now talk to me about
26:48
Celery. What is that?
26:51
Celery yeah, it's been a while. That was my first open source
26:55
contribution. So it happened because there was
26:59
a little silly bug that we needed to fix, but it happened
27:02
from working on Airflow. I remember they didn't a lot.
27:05
It was something like they did. It was, I think salary is I
27:09
don't know, I don't know if everything was wrong thing, but
27:13
basically it's a part, a component that Airflow uses.
27:16
And there was I was trying to set up a connection like a
27:21
private connection with Google Cloud.
27:23
And it was something that just wasn't supported yet fully.
27:26
And they just said all these options don't allow like options
27:29
A through C don't allow and like my options started with the
27:32
letter C or something. So I got my first open source
27:35
contribution by saying allow this, you know which I mean,
27:38
that's how it is, but it's it was a cool feeling to have my
27:42
first open source contribution. Now I have I'm involved in open
27:46
source way more, but it's really cool to make an impact and have
27:51
other people view your code from other companies.
27:53
That's really cool too. So that was my first attempt
27:56
into that. Now that you've had so many
28:00
issues accepted or PRS merged in the open source realm and you've
28:05
done a lot of work just like as a core contributor, right?
28:09
What are some things that you would tell yourself back then,
28:15
or as you were earlier on in the game, about contributing to open
28:20
source? I think what's really nice about
28:24
all of the open source work I've done is that it had a business
28:27
value, like a very clear business value.
28:30
And that makes it a lot easier. And I think I got a bit lucky
28:33
with that because like going to Bloomberg, my goal wasn't to
28:37
work on open source, but it, it's a great part of it, you
28:40
know, it, it's amazing that we can do that.
28:42
And we use it every day here and everything we're doing, it's
28:46
features that we actually need, you know, So I think that that
28:50
makes it, I say, if you, I'd say if you really want to work on
28:52
open source, I love learning on the job.
28:54
I think hobbies and side projects are great for, for the
28:57
fact of learning. But when you learn on the job,
28:59
it's, and you're using it and production and you're deploying
29:02
it. I mean, that's the best way to
29:04
do it if you can. But I think if you really just
29:07
want to get involved in open source and that's the goal of
29:09
yours and you can't find a job that does that now or you for
29:13
some reason you you won't. Yeah.
29:15
I think just, you know, most communities are super accepting
29:20
and there are a lot of labels for good first issues.
29:24
And if it's something you're really interested in, a good, a
29:26
good first issue. And a lot of them also have
29:28
community meetings, you know, once a month, bi weekly.
29:32
So if you want to, you know someone to help you on your PR,
29:35
it's probably good to go to those and get to know people as
29:37
well. But I mean, if you're an open
29:39
source, you're also someone typically who is, you know,
29:44
really a supporter of open source and everyone's coming
29:47
together and contributing. So I've found that my experience
29:50
with people in the open source community is very positive.
29:54
Going back to what you said, when it comes to having a clear
29:58
business value, what are ways that you found you can
30:06
demonstrate that or champion for that to make it very clear to
30:10
whoever you need to that this is the right problem to be working
30:14
on, that this problem has business value and that you
30:19
should be spending your time on it?
30:22
Again, I feel a bit lucky in the space I'm in because usually
30:25
it's pretty clear, like because Gin AI has come, because LMS are
30:31
so large, like we need this stuff, you know, it's very
30:34
apparent that we need it. So I think, you know, from a
30:38
resource perspective, it's very apparent from a usability
30:40
perspective, there's a lot of room for growth in this area.
30:44
So I feel like that's been a little bit easier, but something
30:46
I think about quite a lot that I've seen, especially if I think
30:50
about my first job, because it was a smaller company and there
30:54
were just a few principal and staff engineers, but they were
30:59
like rock stars. I don't mean like in a like a
31:01
coder rock star, but they were just great.
31:02
They're really good and they dug in and they're really good at
31:05
finding business value problems. I think that is a key skill.
31:10
And I thought a lot about some of it is luck.
31:13
Sometimes you fall. I think, for example, in my
31:15
first company, I fell into the best team.
31:18
I had the best mentor that I could have possibly gotten.
31:22
I mean, I feel extremely lucky and grateful for the opportunity
31:25
that I was given because I didn't, I didn't really know
31:27
coding that well. Like I, I did some Python stuff
31:30
in school, but I, I got that experience and because they're
31:33
so willing to help me and because they were so good at
31:36
finding business value problems and then giving it to me because
31:39
to help me grow, I mean, it made like a world of difference.
31:43
I, I think sometimes if I would have been on another team or at
31:45
a different place, would it have been the same?
31:48
But I think seeing them work, it made me think a lot about this
31:52
topic. And I don't know exactly what
31:55
the answer is, but I would like to.
31:59
Well, the the idea of like being able to sniff out key business
32:04
value problems and having a good eye or nose for that is so
32:11
enlightening. As you say it.
32:12
It's like, yeah, that is a skill right there.
32:16
Yeah. And I do think something I try
32:19
to be clear on when we're designing something, what are
32:21
the requirements. And if you don't know, then you
32:24
should find out to make sure it's worth your time.
32:27
I think what I have tried to do, and again, I still would like to
32:31
get better at this skill, of course, but what I've tried to
32:33
do is at least know, try to figure out what's not a good
32:36
business value, you know, there. I think there's some clear signs
32:39
that maybe it's not. So how many people are going to
32:42
use this versus like how many people is this generating any
32:46
money for us or what purposes is served?
32:48
I mean, depends on what the product is.
32:50
But like, do you need that new button?
32:52
Is it, have we asked them, is it actually useful?
32:54
Is that something they would really use?
32:56
I think getting customer or client or user feedback is super
33:01
helpful as well, because I think the worst thing you can do is
33:05
make something that no one cares about.
33:08
Yeah, it's a waste of time, it's a waste of money, and it's super
33:11
demotivating. Yeah.
33:13
And then you go and try and like justify what you've been doing
33:17
for the past three or six months.
33:18
Yeah, yeah. And the worst thing too is also
33:21
like working really hard on something no one cares about as
33:25
well. So I think knowing where to
33:27
manage your time and your energy is super important.
33:30
And just making sure, as much as you can, that it's going to be
33:33
something really useful. I also wonder because there are
33:38
certain things that maybe, you know, they're very important,
33:43
but also if you don't properly evangelize them with your
33:48
stakeholders, then it could still fall flat on your face.
33:52
We had this guy Stefano on here like a few months ago, and he
33:58
talked about how he built the most incredible ML platform ever
34:03
with everything that you could ever warrant, all of the top
34:07
features that he read in every blog, you know.
34:10
And then he said we released it to crickets.
34:12
None of the data scientists wanted to use it because we
34:15
didn't properly go out there and evangelize how good it was for
34:20
them to start using it. So they all kept just doing
34:22
their own thing in their own ways and not coming on the
34:26
platform that he spent all this time making absolutely
34:29
incredible, you know, and having all the the features that you
34:33
would want and you would think are almost like table stakes or
34:38
everybody must need them. And he fell flat on his face
34:43
even after. Doing that, yeah.
34:46
And I think another quote I really like is, and this is was
34:51
the model of the engineering engineering department at the
34:56
time when I was there in my change was as simple as
34:58
possible, as powerful as necessary.
35:00
So I think one thing to speak to that is to get something out and
35:04
then iterate on it so that, you know, it doesn't have to be
35:07
perfect. And not saying that this was,
35:10
but just one point that that made me think of is that it
35:12
doesn't need to be so perfect. Just get something out that's an
35:15
MVP and be able to start iterating on it and get people
35:18
using it and get feedback. Because I do think a lot of
35:20
times we think we know best about what people want, but
35:24
sometimes it's just not true, you know, especially as
35:26
engineers making a platform for another engineer.
35:29
I think we, we think even more, oh, we know what they would
35:32
want, but sometimes it's just not true, you know, and like,
35:35
like you said, like sometimes people need to adopt it or like
35:39
halftime or maybe maybe it wasn't exactly what they're
35:42
hoping. But another point to that that I
35:44
think you bring up a lot too, is that not only did I have and
35:47
still have really great mentors, but also having someone who
35:50
champions you and make sure that your work is out there and known
35:53
and publicized is very important.
35:55
I mean, I do a lot of, you know, self public publicization as
36:01
probably like you do as well. You know, like we're very public
36:03
people. We're posting a lot.
36:05
And like, does that help me? Yeah, I think it does a lot
36:07
actually. I mean, I think, I hate to say
36:10
the the loudest person in the room gets heard.
36:12
I mean, that's not always true, but you need to your name needs
36:16
to be known or like the project needs to be known and put out
36:19
there. Like, I'm sure there's some
36:21
like, amazing apps, but if they don't have any marketing, maybe
36:24
they don't have any users. So you don't have any users.
36:27
You can have the best app in the world, but what does it really
36:29
matter, you know? Yeah.
36:31
Like first time founders focus on product and second time
36:34
founders focus on go to market basically.
36:39
Yeah. So I mean, yeah, it's super,
36:42
super important. The whole marketing part of it,
36:45
marketing yourself is super important as well.
36:47
Like I for example, I create a a brag doc for myself.
36:51
I got this idea. I'm forgetting the the tech
36:55
influencers name. She was big on Twitter a while
36:59
ago, but she created these zenes like these comic book magazines.
37:03
And it's an idea from her. I have to look up who she is.
37:07
But to write a brag doc about yourself.
37:09
So every quarter I do it. Every quarter I have certain, I
37:13
have certain like I have certain sections about like
37:16
presentations, like general work and I summarize all my work
37:19
every quarter and I give it to my manager or whoever's doing my
37:22
review. And the feedback I've got is
37:24
that that's also super, super helpful.
37:27
Just on the topic of evangelizing yourself and new
37:30
products as well. I think it's it's really
37:31
important in tech. I need to do that just so that
37:35
on the days that I feel down, I can open that up and be like,
37:38
look, I'm I'm not as much of A failure as I'm making myself out
37:42
to be. Yours will be pretty awesome
37:44
too. You do a lot of stuff.
37:45
So yeah, it's good to remind yourself.
37:47
It's good to contract, you know? And then it makes writing
37:50
because sometimes you forget what did I do?
37:52
But having having a record of it is super nice.
37:55
Yeah, it's, I think there is a part of human nature that you
37:59
get into these slumps and having something like that and going
38:04
through something like that is very useful.
38:06
I heard even a story of like, John Lennon and The Beatles were
38:12
in the recording studio and Lennon was down and wasn't
38:16
didn't want to record anything. But this was after a whole, I
38:21
think, like week Bender. But as it happens, after a week
38:25
Bender, you're kind of depleted and he's not wanting to record
38:28
anything. And then people started playing
38:30
him some songs back or reading him lyrics back that he wrote.
38:34
And he got like, oh, yeah, I wrote that.
38:36
Oh, yeah. Maybe I do have.
38:37
Maybe I am a OK songwriter, you know?
38:40
So even the best of us, it can happen to us.
38:42
And having something like this is quite useful.
38:46
It's like just to get your spirits high again and then get
38:49
you back on track. Yeah, for sure.
38:53
But going back to this motto of what was it only as powerful as
38:57
necessary and as simple as possible?
38:59
Yeah, as simple as possible, as powerful, as powerful as
39:02
necessary. That is such a great motto to
39:07
like live by because it's so easy to over engineer stuff.
39:11
It's so easy to grab the shiniest tools and before you
39:15
know it you're way in over your head.
39:18
And. You're maybe this is going to
39:22
take a little longer than expected because the scope has
39:25
creeped. Exactly.
39:26
Yeah, yeah, that's super difficult, but at least starting
39:30
with that principle, I think helps a lot.
39:32
Like I said, get your requirements, understand what
39:35
you need, and then try to grow from there instead of trying to
39:37
solve this huge problem in the beginning.
39:40
I think that's applicable to a few different scenarios.
39:44
Yeah. And I'm thinking about the
39:45
requirements and also how you're mentioning like things that are
39:49
red flags when you're starting a project that can be almost like
39:55
early warning signs that this might not have as much business
39:59
value as you think. And and I'm wondering if you
40:04
have any more there's like it does the team that's creating
40:07
it, is it bigger than the actual people that will benefit from
40:10
it? That is a huge red flag.
40:12
Yeah. So that's something like how
40:13
many users are wanting this feature?
40:16
Like how many people have we asked that how, where did it
40:20
come from? Who actually wants it?
40:22
Like what's the cost of running it?
40:23
And that's what I also mean by requirements, just like what it
40:26
what's the criteria for like why do we need this?
40:29
What purpose does it serve? So yeah, I think if one person
40:34
says, oh, hey, it'll be really nice to have this.
40:37
I think I have some questions about if it's worth my time.
40:40
But if it sometimes it's obvious that it, it is worth your time.
40:43
You know, sometimes it's obvious that we definitely model caching
40:46
that's obvious to everyone, you know, But other times, do we
40:49
need this button? I don't know, maybe, maybe not.
40:53
Yeah, yeah. I, I wonder about like anything
40:57
related to GPU's and like saving GPU time or saturating the GPU
41:02
is more like getting the most out of the GPU's.
41:05
That feels like something that is, yeah, we should go and work
41:09
on it. But again, you should come at it
41:15
from this. I almost like a discerning eye
41:19
to be able to ask questions and recognize from what you're
41:25
getting if it's actually as valuable as folks are making it
41:28
out to be. Yeah, exactly.
41:31
So and there's a lot of different ways to do things.
41:33
Like for a good example of this in Caser, we started with, we
41:37
started adding some of the open AI protocols like chat, chat
41:39
completion. There's more, but we just
41:42
started with that and we started supporting a a few different
41:45
type of task auto routing, what kind of tasks you want.
41:48
And then for the GPU thing, there are a few different ways
41:51
that you can manage GPU is working together.
41:54
And I know that right now they're working on a solution
41:56
that uses two of those because those are maybe the two most
41:58
popular, the two that are most needed.
42:00
So I think that approach is also pretty good.
42:02
You don't need to do everything at once, at least just get
42:05
something that people can start using.
42:06
And then I really like using chat, the chat endpoint here.
42:09
Now I would really like to use this one as well.
42:12
And I think that's that's super helpful.
42:14
How much are you going to the non-technical stakeholders that
42:19
are on the business side of the house and having conversations
42:21
with them about what they may want since like with the advent
42:27
of LLMS, it opens the aperture of who's using the AI solutions?
42:35
Yeah, I, I mean, that's a good question because I feel like I'm
42:38
pretty low on the stack as a platform engineer.
42:41
We discuss a lot with the AI and Mel engineers, but we do have
42:44
product people that we talk with regularly.
42:47
And I know they are really good about getting all of that
42:50
different feedback and kind of funneling it to us in a way to
42:54
create products and features. So for us, I believe that's the
42:57
way it works at Bloomberg. It's really cool because even a
43:02
lot of the management and product people are super
43:05
technical. They, they grow a lot in
43:08
Bloomberg. Bloomberg's really great at
43:09
keeping people. I know a lot of a lot of tech in
43:13
tech, a lot people move around quickly.
43:14
It's just so common, right? But I'm super impressed with
43:17
that Bloomberg as there are a ton of people who've been here
43:20
for 10 plus years, you know, so they've moved up and around in
43:25
Bloomberg. But yeah, so that's one
43:26
interesting part about Bloomberg.
43:27
But as far as how we get it, it's usually through product,
43:31
which I do like that, you know, streamline way of doing things.
43:35
But I do agree. And there's like a whole
43:37
philosophy around this about going to the users and asking
43:40
them what they want. And I think that's a really
43:42
helpful and I think that's the way to go.
43:44
You should always be doing that. You know, if you're not doing
43:47
that, it's kind of worrisome, I think.
43:50
Yeah. Again, red flags.
43:52
That is something you might want to look out for because you're
43:56
you're potentially not finding the highest levers or the the
44:00
biggest levers that you can pull.
44:02
Yeah, I think if you can also creating a personal, like a
44:05
personal relationship with say the MAIML engineers that are
44:08
using our platform to go sit with them.
44:10
And just like, hey, if you have someone that you would consider
44:13
like a super user that's really using it a lot, always in the
44:15
support chat, like show me how you're using it.
44:18
And then maybe they'll be like, Oh yeah, by the way, I, I don't
44:20
know what this button is or, Oh yeah, by the way, I always have
44:22
to do this every time. Maybe you would pick up on
44:24
things. I think that is also a really
44:27
good way to figure out what are people's patterns and using your
44:30
tool and what are the pain points.
44:33
Because they might not say in a survey and they might forget it,
44:35
or they might not say it in the chat when they're doing
44:37
something. But by sitting with them and
44:40
observing with them, you could figure it out.
44:42
Yeah, I am just probably, I would say like 90% of the tools
44:48
that I use, I am never like blown away by the experience.
44:54
But that doesn't mean that I go and I explain what could be
44:59
better about the tool to the people that are creating that
45:02
tool. And I am assuming that that's
45:06
like the majority of folks, right?
45:07
Because it takes so long to explain like why this is a pain
45:12
and how big of a pain is it for me?
45:14
And so then usually I'm like, whatever, I'll I'll do something
45:18
else with my time. Yeah, you can pick and choose
45:21
your battles. Yeah.
45:22
And so you go in and sitting with someone, you almost like
45:26
force them to show you what is painful.
45:28
Because if if I had somebody that was right next to me and
45:31
watching me do my thing, I'm sure I would be very vocal.
45:34
Yeah, same. Yeah, kind of.
45:37
It's a bit more intimate in the way, like it makes you feel more
45:40
comfortable, you know, than just entering something on a survey.
45:44
It takes more time, but it's more personable.
45:48
The higher, yeah, higher leverage, 100%.
45:51
The, the other thing that I was going to ask you about is when
45:55
you look at the stack and you think about the traditional ML
46:01
use cases and almost like tabular data, low latency, that
46:06
type of stuff that you are creating a platform for or
46:10
you're helping folks serve those use cases versus the new LLMAI
46:17
world that you're creating a platform for or helping usher in
46:22
those use cases. Where do you see them diverging
46:26
and where like we talked about the Envoy AI Gateway and that's
46:31
specifically for an LLM use case, right?
46:37
How do you see those two worlds playing together in a platform?
46:43
Like does the platform support all of it and almost have like
46:46
this sprawl? Does the platform have pillars
46:51
that depending on what the use case is, you kind of plug into
46:55
the platform over here or over there?
46:57
Like how did, how do you visualize that or look at it?
47:02
So a few things. I think we're trying to make the
47:06
platform as easy to use as possible.
47:08
And at the end of the day, you're just deploying a model
47:11
somewhere. And we try to make sure the user
47:14
doesn't have to care too much. Like I was saying, trying to
47:17
abstract away a lot of those Kubernetes concepts as much as
47:20
we can, try to make sure the user doesn't care too much about
47:22
what or where. Like why should they care if
47:25
it's on Prem or in a cloud? It's just running where there's
47:28
resources, you know? So for AI models across hybrid
47:32
environments, we try to create a platform that's very seamless.
47:36
You don't really know or care where it's running.
47:39
And that's the whole, you know, point of serverless and like
47:43
being able to run hybrid cloud, just be able to use resources
47:46
for different things. As far as different products
47:49
running on Kubernetes, I would say training would be like a
47:52
different tab in our UI, you know, and where does it differ?
47:56
I think it differs in infant services are long running jobs
48:00
and things like training jobs usually are not depends on what
48:04
you're training and what you're doing.
48:06
So the way these jobs run, you're still deploying something
48:09
to Kubernetes though. So the basics are still the
48:11
same. I think some of the smaller
48:13
features are different, like we have a model registry.
48:16
So where you can save your models, pull versions of them
48:19
just by a click of a button and have different change different
48:22
configs slightly, have different different artifacts.
48:25
So I think that's something that is needed in inference that
48:29
maybe isn't as needed in training or some different
48:32
features like that. That's kind of where I see them
48:34
diverging. Did that answer your question?
48:37
Yeah. And I like this idea of like the
48:41
things that are the same or that whether you have a gigantic
48:45
model or you have a very small pruned model or a fine-tuned
48:49
model or a distilled model or a random forest model, it's a
48:54
model and it's going to be sitting on some kind of
48:59
Kubernetes service. Like you just want the model.
49:02
You don't really need to think about how much traffic can that
49:07
model handle, what the actual like where it sits, which cloud
49:12
is it on, is it on Prem, all of that stuff.
49:15
I really like the idea of like, let's abstract away everything
49:19
that we can so that if you have a data scientist that is putting
49:25
a model out there, they know they can just get that model out
49:29
there and their job is to create the best model and then the rest
49:34
is taken care of for them. Yeah, exactly.
49:36
And that's why the unified API is so nice too, because we give
49:39
you an endpoint and the endpoint will look very similar no matter
49:44
where you're running it. And the what you will use to
49:48
predict or chat will be so similar and the same format no
49:52
matter where it's running. And so that's a really good plus
49:55
of these tools, I think. Do you distinguish at all
50:00
between like the models that are for LLM worlds, you can distill
50:07
models and, or maybe you prune them or you do things to make
50:12
them smaller or maybe you have an ensemble of models and you
50:16
have the gateway and whatnot. And so there's things that you
50:20
potentially want to do to LLMS that you wouldn't necessarily do
50:24
to traditional ML models, like if you're training those models
50:28
like you said and you have the model registry and there and
50:32
maybe there's feature engineering happening in the
50:34
traditional ML world. So do you, do you distinguish
50:37
there too or is it just like you were mentioning where these are
50:41
different tabs on the platform depending on your use case?
50:45
So for that, So what your infra service is running, is it
50:48
running this model or that model that's just a config or 1 little
50:53
part of the small YAML that you're putting into the infra
50:55
service. And then the chaser will make a
50:58
lot of assumptions underneath the hood.
51:00
So you don't have to write all these big things about their
51:03
specific to that model, how it runs.
51:06
Like even like what what's the port that gets the metrics?
51:09
Like we, we know Tensorflow is on this port and another models
51:13
on this port. So we'll automatically open up
51:16
port or, you know, be able to pull from it if you're, you're
51:18
getting metrics. So a lot of things like that,
51:21
it's just you need to tell us the keyword or key config one
51:24
thing and then we do the rest. But that's in the YAML itself.
51:26
So that's within the tab, the inference product itself.