Latest / Agentic Conversations (formally mlops.community) / How Hera is an Enabler of MLOps Integrations // Flaviu Vadan // Coffee Sessions #115

Transcript

How Hera is an Enabler of MLOps Integrations // Flaviu Vadan // Coffee Sessions #115 · Agentic Conversations (formally mlops.community)

0:00
I'm originally Romanian my name is Flavio, it's pronounced and
0:03
like fle, various process version.
0:05
I'm a senior software engineer Dinah Therapeutics a biotech
0:08
company based in Watertown. Massachusetts I work on all
0:11
sorts of infrastructure name of things that Dino, I take my
0:13
coffee black. I sometimes go for espresso or
0:17
like long americanos, depending on the day and what's available?
0:20
I'm not that picky. I do love coffee, though.
0:22
I'll enjoy decaf from time to time.
0:26
Flavio. We had another great
0:29
conversation, man. I mean, this is it's what we do?
0:33
Yes. Flavio vidhan senior software
0:36
engineer at Dino Therapeutics, who previously studied
0:39
bioinformatics and computer science at the University of
0:44
Saskatchewan. Joined us today to talk about
0:48
what he does at Dino Therapeutics.
0:50
And what I really enjoyed is, I'll let you talk about all the
0:54
cool Gene. Staff.
0:55
But I thought the second half of the conversation where he walked
0:58
us through what their machine learning platform, has to look
1:01
like as an internal Innovation engine.
1:03
Because at the end of the day, they're not exposing machine
1:06
learning to the outside world. They're using it internally to
1:08
make their sort of virus Discovery efforts better.
1:12
I thought that was so cool and the way that they designed the
1:14
platform to enable that would you think?
1:17
Yeah, I mean I knew you were going to love it.
1:19
Let's be honest. This is like the perfect Venn
1:23
diagram of your likes, you know. No, machine learning biotech,
1:27
great. I'm in.
1:29
So I really appreciate that. I, I like how he broke down for
1:34
us and especially me because I think you already knew it, but
1:38
just the nuances of what they were doing.
1:41
Exactly. Biotech is not my strong suit so
1:44
he was able to dumb it down a little bit after I asked a few
1:47
times and had to tell them to pump the brakes a little bit,
1:50
but that was, yeah, that was crucial.
1:53
I mean this You knows what he's talking about.
1:56
They built an incredible platform.
1:58
He talked about what they did and design decisions, the
2:02
trade-offs that they made with that.
2:04
And let's get into it before we get into it, though.
2:08
If anyone out there is listening to this and you think you know
2:12
how we could make this show better, let us know.
2:15
We would love to hear your feedback and it would mean the
2:18
world to us. If you could just tell one
2:21
person on the internet about this here.
2:25
Episode, if you enjoy it, tough love.
2:31
It's great to have you here, man.
2:33
Thanks for having me. Super excited to talk to some
2:35
biology ml, ml Ops. This is a good stuff.
2:40
This is like visnu's, wet dream right now because it is biology
2:44
and ml off. I was gonna say it.
2:46
I was going to say, but you said it doesn't make it worse, but I
2:50
said it, it actually does because it is that it's that
2:54
much more obvious. That everyone.
2:57
But now I'm thrilled. I'm thrilled to have you on here
2:59
flower. I have been waiting for this
3:01
episode for a while as listeners.
3:03
Likely know, I work at a Healthcare company.
3:05
I'm deeply passionate about Health Care by attack.
3:08
And whenever we have a guest on that, that works on, I'm a
3:10
lobster machine learning in that context, it's like it's like
3:14
it's like a lot of fun. It's a lot of fun for me.
3:16
Yeah, so thank you for joining us.
3:18
Yes, upon the feeling is mutual. We should probably get started
3:22
with just giving us a breakdown of what time.
3:25
I know does and where do we start with?
3:28
Like what is that? What is your role?
3:31
What is the whole like, download?
3:35
Yeah, we can we can definitely start there.
3:37
So at Dino were pioneering an MLA.
3:41
I powered approach in the field of gene therapy like that's the
3:44
tldr. So we use ML and lab
3:47
experimentation approaches to invent and Test new ways to
3:50
design, gene therapy vectors, which are essentially transport
3:53
mechanisms that can help take. Gene therapy to a specific type
3:57
of cell. And in short a gene therapies of
4:01
medical modality for fixing underlying genetic disorders and
4:07
the current, the current best approach is to deliver genes
4:11
into the body. Using these vectors derived from
4:14
harmless, natural viruses, call Davies which stands for a dino
4:17
sociated viruses. And you know, these naturally
4:21
occurring a vis-a-vis, don't Target specific cells organs and
4:25
And tissues really effectively and they're often, you know,
4:28
ineffective, because the body recognizes them as foreign
4:30
agents and it just neutralizes them.
4:33
So I'd, I know we're focused on solving this Grand Challenge to,
4:37
you know, realize the full potential of gene therapy, which
4:40
will make it possible to deliver the service to every cell in the
4:43
body. And to do this, we have an
4:45
amazing team of molecular synthetic and synthetic
4:48
biologists protein, Engineers, gene therapy, scientists working
4:51
alongside software Engineers, computational, biologists and
4:54
machine learning. Experts to design and test these
4:57
approaches that we can reconstruct to develop and
5:01
enabled. The machine guided design of
5:03
gene therapy vectors. And in my role, I'm one of the
5:06
software Engineers working on ML Ops, devops core infrastructure
5:11
and compute and currently exploring some of the and
5:15
working through some of the MLA I approaches that combine these
5:19
things for in silico design of viruses.
5:22
So when you say I have to say, Gina, I study bioengineering in
5:27
college, I studied by engineering, and I took a lot of
5:31
Biotech courses and that was one of the best descriptions of gene
5:34
therapy. I've heard actually, sorry to
5:37
Beatrice. I had to I had to get in there
5:39
and say that for all the listeners be all right.
5:40
For really really sophisticated study bio engineering in
5:44
college. I got a few more questions that
5:47
visitors are always like. Oh, that's simple.
5:50
I want to know a little bit about because they're I've heard
5:53
of crisper and I've heard of In editing.
5:56
I'm guessing this is not anything to do with that, right?
6:00
It's a little bit different. Yeah, so I think the simplest
6:06
way to think of it is like crispr can do exactly what you
6:10
mentioned them and that's you know, Gene editing but what
6:13
we're working on is the transport mechanism.
6:15
So getting the crisper based therapy to a specific organ
6:19
tissue and cell type, okay? And so then the other piece is
6:26
Is this how I've heard about like, kidney transplants, or
6:32
actually. So, it's a little bit wild, but
6:35
stay with me here. I've heard about using Pig
6:38
organs and then putting them into human bodies and they don't
6:43
get rejected because of certain proteins that they use.
6:48
It. Does this have anything to do
6:49
with that in short? Not really.
6:52
All right, I just want to make sure because I saw a radio I
6:56
have or I heard a radio lab episode when it was talking
6:59
about. I'm I'm blanking on the name of
7:01
the protein but anyway the the long and short is that, no, it's
7:08
not. Not up your alley.
7:10
You do something a little bit different.
7:12
It's mainly just with putting these these genes inside of the
7:17
body and making sure the body doesn't reject them.
7:19
Yeah. So for us, the immune rejection
7:22
as Mike has to do with the actual virus itself.
7:26
And for the, for this delivery path that it has to go through,
7:31
all right? And so then the machine
7:32
learning, part comes in where you're deciding what genes,
7:37
you're putting into the body and that's all done outside of the
7:40
body, I guess, actually, the ml part comes in designing the
7:45
virus sequence itself. Hmm.
7:48
Okay, yeah, makes sense. Now, how are you guys like
7:54
you've been around for a while? And you've been doing machine
7:57
learning for a little bit. What is, what does it look like?
8:01
Can you give us a breakdown of? The the deployment?
8:04
Is it mainly scientists and the subject matter experts that are
8:09
doing the machine learning? Or are you also getting just the
8:13
data scientist off the street to come and help?
8:16
Yeah. So maybe I can give you a
8:17
perspective of what is the role of MLA Dino and then, you know,
8:22
kind of go from there and build towards who are the people
8:26
behind Building the platform and who are the users of this
8:29
platform that enable the use of their mouths, specifically for
8:31
Designing sequences. Yes.
8:35
Is that okay? Cool.
8:39
So we'll start by being good, ml practitioners and talk about
8:43
baselines. So like I mentioned, there's
8:46
this naturally occurring set of aavs or do you know these
8:50
viruses that kind of work fairly well in gene therapy, but they
8:55
lack certain features that make them actually, you know, even
8:58
more useful. So, I like some can only
9:00
transport the therapy that's Limited, in size, for example.
9:03
So like there are literally physical constraints of how big
9:06
The thing, like you said Dimensions, like the crisper
9:08
thing, like you have something inside this capsid, it's limited
9:11
in size physically. So of course, you have to
9:19
ultimately take these design and did these verses that were
9:22
design and sell the Cohen have them, you know, since I have
9:24
them be manufactured in synthesized and you need to have
9:27
them fold properly, into the right shape and and other things
9:30
like that. And some of these these Target
9:34
specific cell types but they're not great.
9:36
And there are just a lot of high chances, actually, for
9:39
off-target binding, which means this transport Vector goes to
9:42
some unintended destination in the body.
9:45
And while they don't trigger strong, immune responses, some
9:50
level of immunity can develop, as we had already discussed and
9:53
this requires higher doses. But then you risk side effects
9:56
with this higher dose. So they're all of these, like,
9:58
constraints and limitations that have to be satisfied because
10:02
like I said, we're ultimately making something physical in,
10:05
sorry. Certain to jump in there real
10:08
fast. When you say higher doses, you
10:10
mean like is it a pill that somebody's taken as an injection
10:14
and what is the higher dose of? It's literally a higher number
10:19
of these viruses. That is injected into some some
10:24
Morgan. Let's do some organism cool?
10:26
Yeah. Yeah.
10:28
So speaking of it, that this universe of possible a these is
10:33
actually immense because like there's so many types
10:36
Possibilities of Designing these things.
10:38
So, any approach used to address these challenges must address
10:42
this incredible. Tough incredibly tough problem
10:44
of how do you narrow down this universe of possibilities and
10:49
how do you identify naturally occurring?
10:52
How, how do you modify naturally-occurring cavies to
10:54
give them their desired properties, and to be more
10:58
precise. There's a part of this AV
11:01
sequence that we generally work with that has about 700 30
11:05
characters. It's like possible characters,
11:09
and there are 20 of these amino acids, the molecules that
11:13
actually make the sequence. So ultimately have like 20 to
11:17
the 730 possible configurations and sister so many of them, you
11:22
know. How do you, how do you choose?
11:24
And there are many approaches to generate all of the combinations
11:28
but then you're limited by lab protocols, for example, which
11:31
takes you back to this idea of physical constraints.
11:34
So you can simply like, Pokemon catch them all in.
11:37
Try it out, you know, within the set of possible combos.
11:41
There are some things that won't work simply by virtue of like
11:45
protein, Dynamics and whatever. And I know we've published work
11:49
to Showcase how we can increase the efficiency of these
11:51
generated a of he's using ml, which helps us model this
11:54
relationship between the sequence we generate and the
11:57
associated function and ultimately guide our search
12:01
towards a of these that are not only viable physically, but they
12:04
also have optimized characteristics like Targeting
12:06
specific top cell types. So the tldr is that ml allows us
12:11
to encode this extreme level of complexity into some
12:14
mathematical representation, that's much more effective than
12:17
simple, search to generate a lot of variance, based on
12:20
heuristics. It's an interesting theme
12:22
actually that I think you're touching on which, you know, we
12:26
had on Jesse Johnson. It's an absolute.
12:29
I refer to a lot use the is the VP of data engineering at the
12:33
dewpoint Therapeutics and in that episode and another episode
12:36
Here, we had Life Sciences speakers come in.
12:39
There's almost this way that machine learning through its
12:43
ability to Hoover up, large amounts of data and find
12:48
patterns is helping us understand biology in a way that
12:54
is not possible for us. Currently, right.
12:56
It's almost like the way that we think about biology right now is
12:59
very analog, right? In the sense that we think
13:02
about, you know, discrete elements, we think about, you
13:04
know, amino acids on a protein. Mmmmm right, individuals
13:07
individual units, but the reason reality, it's much more of like
13:12
a entropic space than the way that we think about it.
13:16
And that's where machine learning.
13:18
And in fact, that biotech has massive amounts of data are
13:21
very, very compatible. So I love hearing stories like
13:25
this and from companies like yours that are, that are
13:27
tackling this challenge, because it feels like we're on early
13:32
steps of really starting to understand biology, much better
13:35
than we ever had. For now that we're getting tools
13:38
that allow us to do that. And with that, what I wanted to
13:42
ask you a little bit about is you know we talked about what
13:46
dinos trying to do, right? In a gene therapy around
13:50
sophisticated stuff and you told us about how machine learning
13:53
helps do that better than let's say competitors or the way that
13:57
other people might try and crack this problem.
14:00
I want to understand what does that process of machine learning
14:03
deployment. Look like, what does that
14:04
process of production eyes? In machine learning look like in
14:08
a research oriented environment like Dino, right?
14:10
It's not like, you guys are going to have a product tomorrow
14:13
that's on the market. That's not the way biotech
14:14
works. So, how do you think about
14:17
production izing machine learning at Dino first?
14:21
I'll say that. I very much agree with the first
14:23
perspective, you outlined. I think I've heard this, I think
14:26
I've heard someone in some presentations, a, that biotech
14:29
is in the era of Commodore 64, or the equivalent of that era.
14:33
And I told that's very, very Very nice perspective.
14:39
So when it comes to production izing ml, a dino specific
14:42
culture, you have to think about ml Ops, right?
14:45
Like, how do you take a model and how do you store it?
14:49
How do you allow practitioners to to use it for Designing these
14:52
viruses? Where do you store all the
14:55
associated metadata and things like that.
14:57
So when it comes to, you know, this regime of ml Ops, I have a
15:01
hunch that almost every organization that uses them L
15:03
and is concerned with these types of issues that you've
15:05
outlined. Anu follows primary, you know,
15:08
different practices all over the place and they're primarily
15:11
Guided by some set of principles and you know some of these come
15:15
from see ICD engineering like monitoring feature, an artifact
15:19
storage metadata management and other things like that.
15:23
And I know we basically do the same and we only we certainly
15:27
only follow a subset of these mlx principles out there and now
15:33
I die. No it's a Emma loves is
15:35
primarily concerned with Facilitating access to model
15:38
training rather than deployment. Well at least at least for now
15:43
and you know practically speaking.
15:44
The this domain of mL of the dinos concern with like the
15:49
design of interfaces that support data scientist in
15:52
training any model implemented in our code base.
15:54
For example, for specific biological problems, you know
15:57
also like these models can be implemented in torch Sai kit or
16:02
something as custom as numpy exclusively.
16:05
And that's only the The starting point really because there's
16:07
also access to tomatoes once they are trained and access to
16:11
the associated artifacts and datasets metadata about what
16:14
Transformations and training, hyperparameters parameters were
16:16
applied and things like visual visualizing the training
16:19
process. So, once the Practical table,
16:22
Yeah, quick question about that actually.
16:24
It's because now I feel like I'm starting to understand how Dino
16:27
thinks about machine learning. So you're saying going back to
16:30
the problem again. The opportunity here is to use
16:33
machine learning to learn how to better design.
16:36
Davies. And what you want to do then is
16:38
encourage as much modeling is possible.
16:41
You want to throw a ton of machine learning models, a ton
16:44
of experimental approaches and really Empower ml scientists.
16:47
And data scientists to take a bunch of data try models and
16:52
then be able to test those models.
16:54
You're not necessarily thinking as much about maybe something
16:57
like you know, scale or deployment in the same way.
17:01
That like an e-commerce company is you're thinking a lot more
17:03
about the proliferation of models.
17:05
Is that correct? Yeah, I say it's an accurate
17:07
assessment specifically because we don't have external facing
17:11
apis or something like that. So to us it's more about
17:14
facilitating in enabling Independence for experimentation
17:18
in this for the biology problems that we have and facilitating
17:24
the access to the train models. Once they are trained and their
17:27
use in specific applications. That is really cool because I
17:33
think we're sometimes what we hear on our Podcast is about
17:38
machine learning as an external driver of value creation, right?
17:43
Where like, you know we will have Wayfarer someone like that,
17:46
you know, effectively serving machine learning model and
17:49
recommend it recommendations, right?
17:50
But what machine learning is doing for Dino is it's an
17:54
internal Innovation engine, right?
17:55
It's allowing you guys to build better products internally, and
17:59
I think that's such a interesting contrast.
18:01
I think where I would love to understand is, is, you know, For
18:07
our practitioners that are thinking about the engineering
18:09
trade-offs that you're making, I think a lot of them and I'm
18:12
certainly sitting there and saying, okay, it sounds like you
18:14
really need an internal platform to what extent have you designed
18:18
this yourself and to what extent did you guys go with external
18:22
existing best of breed or rest of class prop platforms, right?
18:25
I mean we talk a lot about tool adoption.
18:27
And I'd love to understand what your approach is there, as you
18:29
build your internal platform. Yeah.
18:32
Actually, this is a great segue to the idea that I wanted to
18:35
outline outline. Access to what happens once but
18:39
practitioner trains a model. So you know, once a dino
18:43
specifically want someone trains or it.
18:45
No, it is able to train a model. This story transitions into one
18:49
that is primarily focused on in, on Integrations that facilitate
18:54
other responsibilities of ml Ops at Dino.
18:57
So at Dino specifically Integrations are core to the
19:02
value production pipeline because this through these
19:04
Integrations that we have the opportunity T to buy multiple
19:07
disparate domains like monitoring the training models,
19:11
access to storage mechanisms and things like that.
19:14
Piper parameter tuning or whatever and access to for
19:18
example, High scale, training and many models.
19:20
So Vishnu you mentioned scale for US.
19:23
Skill is really trained, many, many models in parallel, all on
19:28
like, for gpus or something like that.
19:30
So ml Ops overall is the glue of everything that may be needed to
19:33
facilitate the practice of ml Within Context.
19:37
Hi, this is mihail Eric. I'm the founder of pun intended
19:40
Innovation and confetti Ai. And this is a quick Public
19:43
Service Announcement. If you're tired of having
19:46
nightmares about the machine learning models, you have in
19:48
production and I encourage you to subscribe to the ml Ops
19:51
Community podcast, it's where all the cool ml Ops kids hang
19:54
out and learn how to tame their ml models.
19:57
It's really interesting. I think the way that I think
20:00
about some of the challenges and machine learning our data, To
20:07
compute and evaluation right? And you know from the standpoint
20:12
of like just to it's a really rough heuristic by the way,
20:15
there's so many more so much more there.
20:16
That I know that dimitrios does as is probably sitting there and
20:19
thinking like what the heck but when you think like a lot of
20:23
times as a machine learning practitioner, you're sitting
20:26
there saying do I have the data kind of train a model and then
20:29
after I turn the model it can I just test it right?
20:31
And I'm kind of curious for you guys that.
20:33
Can you walk through like if I'm a data scientist, It's like to
20:36
the extent that you can disclose like, what is my interaction
20:39
with data? What does it access later look
20:41
like, how do I then access computer?
20:43
How do I then translate data to the compute layer and then what
20:47
does evaluation look like look, can you walk through those three
20:50
things? Yeah.
20:51
Yeah that's what I can definitely offer you a more
20:53
practical perspective. Do to help you develop an
20:57
understanding of how we created basically ml a platform at Dino.
21:01
So I mentioned how ml Ops is concerned with the design of
21:04
interfaces to facilitate access. S2 training for instance and I
21:08
should be clear that we primarily use Python just like
21:11
to kind of set the tone of how these things can look like you
21:14
can imagine, whatever I will say in a python context.
21:18
Now practically speaking, if you are a user of this interface,
21:22
all something that allows you to train a model, all you really
21:26
exposed to is some function. Call that takes one or more of
21:30
your model class specifications like a torch model that is
21:34
stored in our code base and without you Specify a single
21:37
configuration object that takes that the offers you, the
21:40
opportunity to control characteristics.
21:42
Such as epochs datasets data set Transformations, where they've
21:46
set comes from how to aggregate gradients when you use
21:50
distributed, give you training optimizers and things like that.
21:53
And there are many defaults we use to facilitate this access so
21:58
that users don't have to repeatedly do the same thing and
22:02
you know, only support them in only specifying a small number
22:05
of Perimeters and once you hit play.
22:08
So does so to speak there. Some Association that's made
22:12
between the model or the collection of models you submit
22:16
that, were that did you submit with the right?
22:18
Training infrastructure. So we have multiple of these
22:21
like vertical stacks of training infrastructure, that support,
22:25
training specific model types that are users submit, so some
22:29
models are dispatched the training, mechanisms, dedicated
22:32
to torch numpy or psyche, it for instance, in the case of
22:36
Something like torch, you can also use things like distributed
22:39
training, for example, and in the case of numpy, we run custom
22:42
Training Loops. So that each of these objects,
22:44
each of these different Frameworks has its own
22:47
independent needs satisfied. So to say, and whatever
22:50
libraries eventually used to training, your search model, it
22:54
all ends up on our go workflows which, which is a product that
22:57
has been extremely enabling a dino because it allows us to
23:01
scale operations almost to Tomos whatever degree we want to
23:05
provide. Dan.
23:06
Of course, that we satisfy some of the limitations in exposed to
23:09
such as Hardware access and all of this ultimately ends up on
23:14
kubernetes of course, which is literally the work for Workhorse
23:17
of everything. Almost everything we do, a dino
23:19
in the computational realm and our go for ML has specifically
23:23
enabled us to construct numerous independent Integrations, kind
23:27
of going back to the integration idea who that provide value by
23:30
taking advantage of the flexibility.
23:32
We have with kubernetes to run basically anything we want.
23:36
We run our own storage model storage mechanism, built on top
23:39
of Google Cloud firestore. So models are saved
23:42
automatically. When users train them, we have
23:44
an integration with aim or aim stack where we visualize
23:48
experimental and production training runs.
23:51
Actually I'll mention that aim stack allows us to push metadata
23:53
to it as well which is great because we can see associations
23:56
between some of the biological context with the model directly
24:01
in this platform. And for distributor training, we
24:04
provide access to multiple do Views by horror of odd and
24:09
finally, you can specify for example, what metrics you want
24:12
to track? And you know those are generally
24:16
fat from this catalog. That torch metrics offers us
24:19
which are again automatically tracked a name and automatically
24:23
aggregated when performed in a distributed training regime and
24:27
the glue that all of that unifies, all of these elements.
24:30
All the way from the intervals that I mentioned to distribute a
24:33
GPU training and monitoring is the wrong is done to Close,
24:37
which is an SDK for Argo. That Dino released last October,
24:41
okay? Before we get into Hera and
24:44
yeah, all of that good stuff. What does the team look like?
24:49
Because there was a paper that came out.
24:52
I think it was like a week ago or two weeks ago, and it's been
24:54
doing its rounds on LinkedIn and it showed all it was like the
24:58
ultimate Venn diagram for ML Ops.
25:00
I imagine you've seen it well, link to it in the show notes if
25:03
anybody hasn't seen it, but it sounds like there's a lot Going
25:06
on how many people are working on is like the whole company
25:11
aimed at this machine learning thing, or is it like you have a
25:16
separate subset of like a few devops people data, Engineers
25:20
machine learning Engineers, the data scientists and you all are
25:24
doing this crazy ml stuff of off to the side.
25:27
What does that look? Like I'm sitting here smiling
25:30
while you're asking this question Demetrius because what
25:34
you described as you know, a breakdown on Team of devops and
25:37
envelopes and things like that. That's the ideal situation for a
25:41
lot of organizations. We are not in that position yet,
25:46
so our responsibilities are kind of mixed and like, that's why,
25:51
you know, in my introduction I mention how I like I work on
25:54
core compute and an MLs doubts, and whatever is like, there's
25:57
been many hats to wear in an organization of our size.
26:01
So I'd say, you know, we have a fairly small Engineering Group
26:04
and everyone has their own special has ations a few of my
26:09
colleagues and myself work on the mL of side of things devops
26:12
and they've experienced overall. There are some folks who are
26:16
more focused on the data side. So how do we store data and
26:19
provide access to data? And then there's a group of
26:24
scientists, primarily who most of them come from Academia who
26:28
are primarily concerned with developing, the methods that
26:30
enable us to design, a these, that take in all of these
26:33
methods independently, they can be independently and things
26:37
like, jupyter, notebooks on vertex, AI on gcp, but once they
26:41
have to be integrated With everything else that we provide
26:45
in this like pipeline, we have for design, then they work with
26:50
us, Engineers to we collaborate on constructing.
26:54
All of these, like Integrations that are necessary for
26:58
introducing a new method into the pipeline officially Source.
27:02
A, and we have this protocol because of course, anything we
27:05
make kind of going back to the idea of physical constraints.
27:08
We have to do all sorts of QC and things like that before we
27:11
introduce something. So there's a If it's a highly
27:14
collaborative environment in which all of the group's
27:17
actually work together, but when it comes to what you asked, Dev
27:20
Ops forces have also serve whatever.
27:22
It's like many hats wearing, you know, we were many hats in the
27:25
Engineering Group. Yeah, typical startup,
27:28
basically. Yeah.
27:29
Yeah. So that's a lot of fun.
27:32
I do notice that you are leveraging, our go workflows
27:36
which I feel like has been coming up more and more.
27:39
And I wonder why if you ever Looked at trying to use Cube
27:45
flow and why, if you did look at Cube flow, why you chose are go
27:50
workflows because it seems like a lot of people are going
27:53
towards our go workflows these days.
27:55
Yeah. So I'll maybe I'll give you a
27:57
perspective of why kubernetes, and why are go and then we can
28:00
go ahead and talk about Hera and how that facilitates access to
28:05
these types of resources. So, I'm not sure how it could
28:10
be. Red is popular but it strikes me
28:13
that People are still not super familiar with extent of exactly
28:16
what it does. So briefly, it's an open-source
28:19
platform that supports management the management
28:21
scaling and deployment of containerized applications.
28:24
So as long as you can isolate your dependencies into a
28:26
container generally talker of your application or workload can
28:30
run on kubernetes, so it offers Dino the necessary flexibility
28:34
of running almost anything and are almost arbitrary scale which
28:41
allows us to it also, Facilitates resource
28:44
accessibility gives us the opportunity to move fast as
28:47
deployments can happen quickly and at high frequency, I'm a big
28:51
fan of kubernetes but also have the knowledge that one the one
28:56
that that one of the downsides of kubernetes is this steep
29:00
learning curve that practitioners have to go through
29:01
to understand the intricacies of it.
29:04
However, I think that that learning curve is absolutely
29:07
worth it given that kubernetes is part of CN CF, which is the
29:10
cognitive Computing foundation and I'd argue That when you know
29:15
what people here called first they likely think of kubernetes
29:18
and what's incredible about CN CF is that any project almost to
29:23
my knowledge any project that's hosted by CN CF integrates very
29:27
well with kubernetes. So again kind of going back to
29:31
the idea of integration. What's amazing is that if you're
29:33
a clock first company like Dino that uses kubernetes, you're not
29:37
only adopting kubernetes but you're basically adopting CN CF
29:41
in anything that's hosted by CN CF can now use be used to
29:45
deliver value to your organization by integrating
29:48
multiple projects into a single unified stack and you have
29:52
access to a huge diversity of ml Ops tools like khatib, for
29:55
example, for hyper parameter tuning in, case are for model
29:58
serving. And I attended que con last
30:02
year, and it also some presentations about things like
30:06
data set, caching directly one-on-one kubernetes.
30:09
So it's kind of this reasoning that was used when we adopted
30:12
our go workflows and we knew we needed, you know, we knew we
30:15
needed a workload engine, and we looked at airflow, prefect and
30:18
other things like that, but none of them have really the
30:21
potential to enable these second order effects that I've
30:24
described that kubernetes does. And for instance, I will, you
30:28
know, I believe a float. Offers a kubernetes executor and
30:31
V2. However, if you have the
30:32
opportunity to construct a platform that quote, unquote
30:35
speaks kubernetes through our, go and uses exactly the same
30:39
constructs as kubernetes does, I personally see that as a huge
30:44
opportunity so it kind of goes into this idea of how did how
30:50
did her our come about and and Q flow?
30:52
And you know we can definitely talk about that next.
30:54
Let's do it. I want to get into.
30:56
I want to get into hearing about Hera it's For those who are
31:00
listening were definitely going to add in the show notes, a link
31:03
to the original blog post from Dino about what Hera is so that
31:09
you can read more about it. I know Demetrius and I
31:11
thoroughly enjoyed reading it, but for the sake of the
31:14
listeners flop, can you take us through?
31:17
What exactly that project is? Where it came from, and how you
31:19
worked on it. Yeah, yeah, absolutely.
31:22
So I'll take you all the way back in time when we used to
31:26
flow and I'm using, you know, this language to like this
31:29
showcase how fast things move in the MLS world because like, you
31:34
know, back in time and kill flow was is a really a young project
31:37
anyway so he used to flow but we had some challenges associated
31:41
with observability specifically especially as they pertain to
31:44
image pullback offs if you're familiar with that term like
31:47
you're trying to pull an image and you cannot find it and
31:50
things of that kind and as and also access the gpus in addition
31:54
the queue for pipelines as decay or kfp of also known as P.
31:59
Turn out to have some, some of these God chose that caused some
32:02
confusion among the team, especially things like,
32:05
sterilization, that allowed that was used to create parallel
32:10
jobs, the decorators that had to be used.
32:12
And all of the parameters that people had to set, and it didn't
32:15
really these characters didn't really support our experimental
32:19
Independence. And as I was mentioning to you,
32:22
Vishnu like that's one of the primary aims we have in
32:24
developing this ml platform. So you know people People who
32:29
knew more about kubernetes and observability paired.
32:32
A lot with people who work with cute flow, in order to debug
32:36
issues associated with the use the edge of Q flow.
32:39
Now, Q flow is actually a very nice product, for a lot of
32:42
reasons that your recent guest Ryan Ruslan, actually actually
32:46
touched on in his episode. And I must say that I agree with
32:51
the perspectives you exchanged in that episode, because yes
32:54
Argo does feel a bit more comfortable to devops and Emma
32:56
to the devops in ml Ops community and Some degree is
33:00
challenging for data scientists to be told, you know, now we're
33:02
using this thing called Argo. Not Q flow now, but I do think
33:06
there's a nice balance to strike, which is what Hera aims
33:09
to the aims to provide. Actually, you may know how Argo
33:14
is configured through, you know, llamo and basically pseudo,
33:17
programming 3mo files, which is extremely error-prone and
33:20
doesn't really support Independence unless you're
33:23
actually familiar with our go. It's hardly testable and a dino
33:26
were way more comfortable with python So we use this SDK called
33:31
Argo workflows DSL. If I recall correctly, that was
33:34
available at the time to create some chord.
33:37
I know were closed on our go deliver value to our go were
33:40
close when we after we initially adopted adopted that project and
33:45
kind of just check it out to see what we can do with it.
33:47
And as we tried more and more of these workflows, we found that
33:51
it didn't really support the independence to the degree that
33:54
we sought. We still needed to work
33:56
together, to create data science, workflows, which is,
33:58
which is But it provides An approved inefficient at specific
34:02
points. Also to get around some of the
34:05
missing features, we have to understand the actual
34:07
implementation of this DSL to know, kind of what inner feels
34:10
to modify. So we can provide access to
34:12
gpus, for instance. But you know once we prove the
34:16
value of our grow more flows which was the important element,
34:20
we looked at other options to kind of provide access to it and
34:25
we found cooler and I should be clear that these are amazing.
34:29
Easing. Sdks is just that they did not
34:31
work specifically, in dinos context and color was good at
34:35
the time. But the parallelism in
34:36
dependency model design involves using lambdas in some convoluted
34:40
ways, and it also interacted with Cube CTL, which is the CLI
34:43
of kubernetes. And we know we had some security
34:46
concerns and around that specific topic.
34:50
So, we wrote Our Own after a few trials, design Hera and
34:55
collaborated with the Argo Folks at Intuit to open source it
34:58
under our Project labs and the primary goal from the Star Wars
35:02
and it will always be to empower the team at Dinah to be
35:05
independent and in the hope and the hope of open sourcing was to
35:08
empower other organizations face with a similar technique similar
35:12
technique choices. Like, you know, using article or
35:14
close to have the same degree of Independence and facilitate
35:18
access to our go. I love that story.
35:21
I think it's a great example of how an engineering team that is
35:26
trying to meet, you know. Crucial business constraints in
35:30
a number of ways. Right around the flexibility and
35:33
the efficiency. And effectiveness of the
35:34
platform that you're trying to build, has to take multiple
35:37
approaches sometimes right? And prototype and some things
35:40
fail, some things don't work out but that's the whole point
35:43
behind trying to, you know, solve the problem, the right
35:45
way, right? Not just the quickest way and I
35:48
think one question I have here is, we are as a community.
35:52
There's a ton of energy around open source, and I think this is
35:55
an interesting opportunity. You guys are start up your
35:57
fast-moving. You have a lot of Like business
36:01
needs to solve like, how did you guys make time to open source?
36:04
This is Project. Like, how did you communicate
36:06
that to your stakeholders? Like how much work was it just
36:09
so that other people other practitioners?
36:11
We're at companies where they want?
36:12
We want to push an open-source model for some other projects
36:14
like how can you walk through your experience so that they can
36:18
understand the level of effort there?
36:20
I think there are two core characteristics that that
36:23
engineered specifically have to be aware of when they would like
36:25
to open source a specific project.
36:29
Get that they worked on at the company and it goes to goes to
36:33
this idea of development empathy which is something that I'm a
36:37
big proponent of. You really have to understand
36:41
who you're communicating to communicating with specifically
36:45
when it comes to design of projects and open sourcing.
36:50
So one of the core risks, your you will be I was face-to-face
36:55
with specifically from was leaking Any sort of private
37:00
information about Dino as an organization through the, on
37:03
open-source effort to address some of this.
37:07
I generally recommend finding Champions and supporters who are
37:12
sort of, you know, kind of to say, on your side to push for
37:16
this effort and actually speak for the opportunity that an open
37:21
source project presents in terms of recruiting, for example, and
37:25
things of that kind. So you have to position your
37:27
pitch very, you know, in a very Retailer manner for the odd, for
37:32
your audience in, in, and in and in a context, like, dynasty was
37:36
the legal team. So, we had to work together to
37:38
understand, what are the risks were actually exposing ourselves
37:41
to what are the implications for?
37:44
What are the practical implications for for engineer?
37:47
So who's going to maintain it? How much time does that require?
37:50
How are we going to take? External externally contributed
37:54
code and integrated into our own practice in a safe way?
37:59
And who's going to be responsible for that kind of
38:01
thing. And and what is the overall, net
38:04
benefit for Dino as an organization?
38:06
So you really have to be thoughtful in the way you
38:10
position your pitch. I think it's all about that and
38:13
finding support from perhaps more mature engineers and more
38:16
engineer. More experienced Engineers,
38:18
great advice. I think it's actually really
38:20
interesting to hear that your company it was likely legal
38:24
team, right? That was sort of the Core State
38:26
stakeholder here and I think it's a good tip for Engineers,
38:29
you know, including for me to kind of think from that legal
38:31
perspective, understand, you know, I never really learned all
38:34
that much about some of the software licensing stuff, right?
38:38
And I think it's a good tip that you just gave to all our
38:42
listeners to me or on how to navigate some of the issues
38:45
here. And you know as much as I would
38:48
love to keep this conversation just going and going and going,
38:51
we are up against time and I just want to thank you for that
38:54
for living. I got one more nice.
38:56
That's one that I wanted to ask really, hold on.
38:59
Hot hot, it's not done yet, hold your horses, I just want.
39:04
So I wanted to go into something because you mentioned earlier,
39:07
if love how you're following certain ml Ops practices and
39:11
certain ones, you kind of threw out the window or you, you
39:14
decided not to follow and I was thinking about which one's those
39:20
were like, which ones are you deciding to follow and why?
39:24
And then, which ones have you put to the side and said, not
39:27
right now. And maybe it's because Cuz of
39:29
the use case, or maybe it's just because of the Manpower or woman
39:33
power human power. What is that like look like I
39:37
think whenever someone so I must first acknowledge that.
39:42
I don't think it was a conscious Choice like I don't want to come
39:45
off as I knew everything ahead of time and like I designed it
39:48
very intentionally or something like that.
39:50
It was more of, let's take an organization need and solve for
39:55
that problem right now as it is, and then we're going to adapt
39:58
and live. Take small steps and solve
40:00
problems as they come up and then I realized that.
40:03
Hey, we're we've actually cut some Corners in this like from
40:07
this collection of em llp's principles and practices, then
40:10
we only focused on a subset of the, of the core principle.
40:14
So if you know this, like infinite sign that says, you
40:21
deploy a model, you? Sorry, you collect data, train a
40:24
model test, the model, deploy model monitor the model revert,
40:28
or like any, Keep going in this Loop for us.
40:31
It was mostly about what are the current problems of the orgasm
40:35
mentioned and for us it was train the model, sorry exit
40:40
facilitate access to data, train the model store, the model and
40:44
then use it for a bottle of biological problem because our
40:48
experiments change from time to time data changes as collections
40:51
come back. And things of that kind, we were
40:54
primarily focused on like a half of this infinite sign for now
40:58
and as our science advances our practice advances as our
41:02
infrastructure advances, we may end up in a in a world where
41:06
they know this full infinite sign is going to be fully
41:09
satisfied, but until then were generally just aiming to solve
41:12
for organizational and business issues and only provide access
41:16
to the things that people care about and we know are going to
41:19
be used for otherwise we end up building things that are like
41:23
nice but they're not necessarily immediately useful and You know,
41:29
rather than over engineer build in small increments and then
41:33
learn from that is probably the best thing we've done so far.
41:37
Awesome, strong fundamentals there.
41:39
All right, now we can clear it. We're going to cut it there.
41:42
Thank you flop. This has been awesome and I for
41:44
your time. Thank you so much.