Latest / Agentic Conversations (formally mlops.community) / How to Avoid Suffering in Mlops/Data Engineering Role // Igor Lushchyk // MLOps Meetup #55

Transcript

How to Avoid Suffering in Mlops/Data Engineering Role // Igor Lushchyk // MLOps Meetup #55 · Agentic Conversations (formally mlops.community)

0:17
Igor. It is a pleasure to have you
0:19
here. And for everyone that does not
0:24
know Igor, he's a data engineer. A taejun Correct me if I'm not
0:29
pronouncing that, right? No one knows how to pronounce
0:32
correctly, especially from exports.
0:33
So, because I've heard a lot of versions from Odeon to Aegean to
0:38
like some smoothies. But let's take the idea again.
0:41
There we go. And what happened is that I
0:48
somehow came across something that you were doing, and if
0:51
anyone wants to go to Igor's profile on LinkedIn, I'll throw
0:54
it in the chat right now. He, you just radiate.
0:58
It ml Ops from your profile. I don't know how you do it but
1:04
instantly what I saw your profile on LinkedIn.
1:07
I was like dang. This guy seems like he's got
1:10
some knowledge. We should probably get him on a
1:12
meet-up and I think I reached out and shortly thereafter.
1:16
He were like sure give me a little bit of time but I'll do
1:18
something. Yeah, I've tried my best man,
1:22
because I'm not everyday person. Whom I wanted to Reach out like
1:29
comes to me for himself because I wanted to reach out to you
1:32
like he Demetrius maybe we can speak about something like I
1:36
like, wow, I was thinking about this and here's here's here,
1:40
that's awesome. Yeah.
1:41
So it's pretty incredible what you're doing.
1:43
I think we should probably get a bit of background info before we
1:48
even start on the topic of today, which is a philosophical
1:54
topic of how to avoid suffering. Comes through and allow ups and
2:00
database and data engineering and machine learning
2:03
engineering. That's the long title.
2:05
The short title is just how to avoid suffering and I would love
2:12
to know how exactly you got into Tech and what was your journey
2:17
like to becoming a data engineer?
2:20
Hmm, it's very complicated, journey and complicated story
2:24
because first time I had gotten into that kid was Mid-90s, I
2:29
guess the I was small kid and I was playing on my mother
2:33
mother's PC at here like Institute in game, like Prince
2:39
person. I still remember the game.
2:41
It's awesome. And basically, all my childhood
2:45
to talk to his old books like basic for children, something
2:49
crazy's know, like throughout the scalp as attachment, but the
2:55
to be serious, I've started the programming When I was learning
3:00
by informatics because my background is not computer
3:03
science. I was doing like for very long
3:08
time, like my masters and some something on my bachelor
3:12
studies. I was doing by informatics,
3:14
like, genomic assemblies, genomic, annotations to cetera.
3:17
And there, I needed to learn something.
3:20
And because by hand, like you can do, like, eco-products.
3:24
You can't model anything. Like, just, like, with your, I
3:27
don't know, like, Like resume information you can but you need
3:30
to provide some results. And there I've started to work
3:34
with the such ecosystem like statistical ecosystem by car.
3:40
I mean, that's awesome to my and like for statistical
3:44
programming, for visualisations, ETC.
3:48
And I mean verb is very awesome. Practice like vehicle doctor but
3:55
but afterwards like other words, they felt that are like it's
4:00
awesome but it's super black box.
4:02
Sorry, it is Black Box. It has some methods implemented
4:06
and yeah, you can use them. You can to some extent, even
4:10
extends, mm, but it's not super flexible.
4:13
So all the my PhD teases. I was trying to do with spice
4:18
on. There are like it was It's just,
4:23
I wasn't down the road. Two thousand summers are like to
4:27
solve that by me, but these it is, I didn't finish, so I'm
4:30
happy. Happy me, I didn't finish it.
4:33
Yeah, I've realized 22 students at their first death for doing
4:37
PhD in to be super romantic person because I mean, because
4:41
it's does not give you super big value.
4:44
So, for example, like some romantic Emeritus in, like
4:47
thinking about that, you're contributing a lot of into
4:49
science, especially in that like, Rotational biology science
4:53
is actually a very conservative one.
4:54
Yeah and I guess yeah 2000 after after I've dropped my PhD
5:01
studies. I've started doing I mean I
5:05
started programming basically I don't know how it happened but
5:11
at that time I had title like back and engineer like and this
5:18
is all but almost all the my Career.
5:22
I was doing just, I was working just with data.
5:26
So even on my first project we were writing like web analytic
5:30
tool like from for making an element analysis of by blogs
5:34
Etc. And even there, we didn't amend
5:37
our own small nosql database. I mean, like it's Superfly.
5:41
I know is it's a say? It would be super fun allowed to
5:50
say that it was no secret. The base.
5:52
But yeah, it was persisting Data initiative is the way which we
5:56
want to sort of bicycling to be reading it.
5:58
It's blasphemy on the SQL databases man.
6:03
At that time, I didn't know almost anything but I started
6:06
writing it. I didn't know anything about how
6:08
it should be working, especially because my background is wasn't
6:11
computer science and like, whoo! Wow, I need to rise the sun
6:15
like, okay, I need to read how she should be working.
6:18
It's actually what is well-liked right ahead, lock is
6:21
sidetracked. Why does it need to be?
6:23
Does it need a seat? Or does it not need Aces?
6:26
The stage for our about this sharding, like all the stuff I
6:31
needed to learn at what at one time.
6:33
Basically, without having all the background in computer
6:38
science, I would say it was so challenging and into a super
6:41
hard. And after yeah, after that,
6:46
after that, I just go on like this and other projects.
6:52
I did a lot of them in consultancies, but recently,
6:57
last last couple years in before I again, and in this for again,
7:02
inside outside, and I'm working mostly like data Engineers
7:06
national office. So I'm helping companies to
7:12
choose their pipelines, which machine learning models and
7:14
data. Well, I know that you all agent
7:19
have a dedicated like Ml Ops Team and I want to get into that
7:24
in a little bit. But you said something in this
7:27
intro that I find fascinating, which is you had to learn a
7:32
million things at once and figure out the best practices,
7:35
right? Do you have any recommendations
7:37
on how to best do that? Because it feels like we're very
7:42
much in a state right now where a lot of people are coming into
7:46
ml Ops from different areas and they need to learn a million
7:51
things because the System is so large and try to pick up some
7:54
best practices along the way. So, are there any tips that you
7:59
learned in that section of your life that you could pass on a
8:03
lot of tips? I mean, I also also, I was
8:07
teaching Python Programming after onion after like this,
8:10
like after learning everything by myself, I decided.
8:15
Okay. Maybe I need to transfer my
8:17
knowledge or more all my tips to like friends of mine and
8:22
Everyone who needs that. So, and I also was a teaching
8:27
programming, like Python Programming flight for two years
8:29
in a row like locally in Ukraine.
8:34
Yeah, so tips. And then from that time, I have
8:38
just wanted. If you need to build something,
8:41
you don't use huge Frameworks when you don't understand how
8:44
they, how they do work. Try to understand everything
8:48
from smaller pieces and like, like, for example, you You have
8:52
like in web development, you have industrial like ready
8:58
Frameworks like for example in Python jungle where everything
9:01
comes with with it and you can small that Frameworks.
9:05
So from from my in my opinion, it's better to use small
9:09
Frameworks and to build everything by pieces.
9:12
Like these errors things like with routing with authentication
9:16
such an aura and you will understand how does it work like
9:21
after that. When you become old lazy or just
9:24
lazy or so like you know, like sure you can use ever since what
9:30
you want. But when you're learning, don't
9:33
hesitate and to use small pieces and to don't hesitate like to
9:38
look inside. How does it works?
9:40
Like because well even when I was interviewing so I wasn't I
9:46
wasn't looking at like tools or with which personally had
9:51
experience I was asking about Insight like about inside
9:56
functionality of tools, how they are working.
9:59
How person does understands there's an instant like Python
10:03
language or how person understand how data flows should
10:09
you should look like inside. So that's, that's my
10:13
recommendation, like to work with small pieces and to look
10:18
inside of them, like, try try to turn on your careers.
10:22
Basically, because you need to push yourself to dig in inside
10:28
of the tools to understand how they do work, why they work like
10:33
this and search for ourselves. So, if I, I can disseminate that
10:37
for a moment, it seems like there's on one hand, you took
10:42
your knowledge and you tried to, I like how you said that you
10:46
tried to transfer it to others and that I've heard is a great
10:50
way of also solidify. Buying what, you know because
10:53
you'll see where the holes are. And then on top of that, you
10:57
were digging into these pieces that I really like how you say
11:02
that. Don't start with the gigantic
11:04
framework that does. Everything, especially when you
11:06
don't know how it works later. When you are an old hat, then
11:11
you can get into it more and you can say, well, this is going to
11:14
make my life significantly easier, so I'll do it then.
11:17
But now let me just figure out how these tools.
11:22
Actually work and how they played together and what is
11:24
underneath the abstraction layer so that I can become a better
11:30
programmer. I can have a deeper
11:32
understanding in case something does go wrong.
11:35
Yeah, yeah, yes. It's totally like hundred
11:39
percent to it as and especially we teach them the others.
11:43
I mean, first time like first year, When I Was preparing
11:47
lectures, I like it. It was topic like which I New
11:52
bike, my heart and I was eating it every day and say to her.
11:55
But every time when I Was preparing to doctors, I was
11:58
thinking about how to explain this white use case.
12:02
How to explain cop, doesn't work.
12:04
And I was prepared. Like, I was spending one, full
12:07
day for one hour lecture, and every lecture I was getting
12:12
like, I wouldn't say stupid but question, but very simple
12:17
question to, which I didn't have correct.
12:19
Answer at one time, it was Carson on the after that, they
12:27
leave on time to us embarrassing.
12:28
But I learned like very positive.
12:31
If I don't know something, it's better for me to say, I don't
12:34
know to look to try to find something exciting like next
12:39
time or in, I was like a sexual. So, a little bit off topic here,
12:44
but, nobody's, nobody's given a shout-out to your background.
12:49
And I want to just mention that your background is, Is classic,
12:53
especially because I think a few weeks ago, I put on LinkedIn or
12:57
something. It was, like, explain the ml Ops
12:59
ecosystem in a gif and somebody put that gif of the house on
13:05
fire and it was like, or it was like a meme that it was like
13:08
everything's fine here, we're all good.
13:11
And so that's the kind of confidence.
13:14
We all have and ml ABS ecosystem at the moment.
13:17
Let's shift gears into what you're doing now with ML Ops and
13:22
I think it would be useful for us to get a bit of background on
13:28
how your team is structured. What you all are doing, because
13:31
I know you have a dedicated platform or a dedicated ml Ops
13:35
Team, I think is what you call it.
13:37
Can you dive into that a little bit more?
13:40
Yeah, but I might be a, I will give you a bit more history
13:45
here. So, yeah, because Italian, we
13:47
just started to have dedicated amount of steam just recently.
13:51
Like, last couple months I would say, like, it was officially
13:55
shaped official because before we had initiative, like, with
13:59
people who like, who were doing it, but we have this Team inside
14:06
of, for another big initiative like which we call center of
14:12
data infrastructure sexual. So because because we need to
14:19
keep this team, very close to infrastructure itself.
14:25
So why it is? Maybe it's a habit of topic.
14:29
But I mean, I want to explain how to avoid all these fire
14:34
behind. Me are felt to however, after a
14:38
couple months later, okay, this is fine.
14:39
This is just small fire Center as a reason, very nice talk from
14:46
former head of data engineering cats.
14:48
Like, I don't, I don't remember his name right now, but it was
14:51
about a variety of engineering, and he had this infinite Loop of
14:56
sadness. When in Four Corners, he has
14:59
like business data science data, engineering, and we can place
15:04
here. Also my oops.
15:05
And infrastructure. Everyone's them of disconnected
15:09
especially when you have them like in their own domain Silas
15:14
essential and one of them. Every one of them disconnected,
15:16
like in Communications and you have mass, you have fired.
15:20
You don't have this fire, have huge fire like dumpster fire.
15:24
I really like his. The, his example is dumpster
15:27
fire. So this is why, when we were
15:30
starting this initiative, like, with data engineering, can we
15:33
use? I'm a lofts We like we were
15:38
pushing to be close to infrastructure and we were
15:43
designing it with thinking of how to connect us with data
15:48
scientist because in that engineering can resume and
15:53
obvious, you have a couple steaks stakeholders and
15:56
unfortunately in very rare cases, you have take color like
16:01
a business. But in most cases you have
16:03
stakeholders that your data scientists Our clients, internal
16:07
clients and you need to have very close communication with
16:11
them. Like I did not like about every
16:14
day, but every week you need to like catch up with them.
16:18
Like to share your experience, they need to stay with you.
16:22
Is there pain? Thanks because you need to solve
16:24
the brain so and basically we we have not not, not big team, I
16:32
would say we have team of five and we Compose a team of on that
16:39
bigger group of data infrastructure.
16:42
Like Engineers, not like infrastructure Engineers, but
16:46
Engineers who know who knew how our data infrastructure works.
16:51
And also data scientist like technical one.
16:56
So take us through this evolution of what you've been
17:00
working with in the past and I know that you mentioned to me
17:05
before Or if you had you rolled your own right?
17:09
And then eventually, you got to a point like where you said we
17:13
need to transfer over to open source because the it's mature
17:19
enough, or what does that look like?
17:21
Can you give me a bit of background?
17:23
Yeah. So as we were nice blog posts I
17:28
didn't pack about about our infrastructure, which we had so
17:33
you can Google it so. It was about our internal ship
17:38
scheduling system, so we will try to find it a new Post in
17:42
chat there. If it was describing that in
17:45
basically due to some constraints, which, which we had
17:50
previously like three years ago, it was built in-house scheduling
17:55
system, like with some look at air flow and how it was working,
18:00
but it was built from ground like with our own Island.
18:06
Which our own bags Etc. And it was the Portage has
18:09
passed by one person. I mean, but it was doing it
18:13
works, it works. And that's why it was adopted,
18:18
why it happened? It happened because at some
18:20
point all the data initiative were, I don't know where we're
18:27
starting that it was just peeled see like proof of concept and it
18:30
was a huge mistake for the people who were working with.
18:35
Why? Because when you trying to build
18:39
system, which interconnected with different parts, let's say
18:44
we have data engineering core machine learning engineering.
18:48
Okay, let's let's dive into machine learning, you have
18:51
system which connected with different parts.
18:53
This infrastructure that's see ICD pipelines.
18:56
That's some artifact Erza cetera.
18:59
So that's draining system for machine learning models.
19:02
You cannot start thinking about like pussy, See you already need
19:06
to have in your mind, thoughts about that walk how it should be
19:11
working? What is production-ready or how
19:15
it should be working in a year? I mean, I'm not saying about
19:18
different parts, but I'm saying about constraints or
19:22
requirements, like does it need to have fairly availability,
19:26
does need to be full tour. Loved all around redundancy,
19:30
Etc. So what load it should maintain
19:33
so and also like Here in jobs, describing like technical Parts.
19:40
But huge problem also that when you start something and you roll
19:46
out, like some puc products to people who is hungry for doing
19:53
analytics. Yes sir.
19:54
Okay here are puc product. Do analytics like you can do
19:57
like like great features great your models, they start working
20:03
with it and the And the eventually you end up with
20:08
something, which while they adopted were, we with rich
20:11
people, like overly, they learn how to work with it.
20:15
And immigration from that homegrown system or like puc
20:20
like-minded to something very super production, ready?
20:25
We will be very painful because of the adopted practices.
20:28
They wrote some code for like flick heavy coat to cover.
20:33
Some use cases. You say to cover some bags, Etc.
20:37
So that's a huge point right there.
20:40
If you don't plan for production from the beginning, then it's
20:46
going to be painful. Trying to go from POC to
20:49
production. Is that what I'm understanding?
20:51
Yeah, correct. Yeah.
20:53
Especially in data, especially in data world.
20:55
You need to forget about POC here.
21:00
You need to think about products and the even when you're doing,
21:05
Doing Busey, it's need to cover all use cases, which you need to
21:09
have here. I'm saying about technical, use
21:13
cases, and also about the use cases for end-users.
21:16
I mean, are like, I mean, this, my opinion personal maybe
21:20
someone will not agree with me, but, When you working in data
21:24
world like an imaginary world, forget about puc puc.
21:28
Can we can work on your laptop to check.
21:31
Is it good for this environment on my laptop and the solve our
21:35
you need, you want to check updates or for new Mo flow UI or
21:40
whatever. But forget about that you see,
21:45
like this abbreviate your life through letters puc in data and
21:48
my world. I love that idea.
21:51
That's a great quote too. So we've got some questions
21:55
coming through the chat and I will just take a moment.
21:58
If you mentioned everyone, feel free to throw questions in.
22:02
This is all of us here so I will get to them as they come.
22:08
Ron is asking a question, Ron. I was trying to warm them up a
22:12
little. I was going to get here, but I
22:13
like how you dive into it and get right to the point.
22:16
Ron does not like to beat around the bush.
22:18
He says, what tech stack does your team use now?
22:22
And how do You handle data lineage.
22:25
Who did lineage unfortunately was deadliness.
22:30
I cannot say a lot here. Why?
22:34
Because something is under and in India so much about tech
22:38
stack, for sure. So because I mean, long story
22:44
short, again is the on-prem company and it says this why we
22:50
constraint with some choices but For our data rules.
22:55
We do use open source tools like HP platform.
22:59
Like course on data works on data platform, spark.
23:03
And the biter cop, installation to search for like, processing
23:08
data for like ad hoc queries the cetera for orchestration and for
23:16
controlling, like, when scheduling of machine learning
23:19
models, like framing company learning modules, I guess, we,
23:24
It's over the industrial standards but it's airflow.
23:27
I mean, it's not a secret and for machine learning like meta
23:34
data, storage or for tracking. I mean, it's where it's very,
23:40
very hard topic with me because we, we, we've started to use a
23:45
mofo long time ago when it was the boot on it.
23:48
Whether we just start like was presented by date of weeks.
23:54
But with our experience which we have that I can say is I'm also
24:00
maybe it's good on and super good and data boots Cloud.
24:04
Maybe. But for use case on, when you,
24:08
when you have a installation, when you need to support
24:13
different like ecl's, like access to release, like a
24:16
different role based access control.
24:18
Like when you know, you want to see Visualization and metrics
24:25
not based on the ground of, like, of every model.
24:28
But on base of experiments, which for, I mean, it's not,
24:31
it's not super nice to you. Need to you need, you need to
24:35
add new functionality, but you like, especially with you, I
24:40
etcetera. I'm not complaining about API
24:42
that special, especially we see, are you were restricted?
24:45
Because it's you already have already compiled, react, huge,
24:50
react, application, and you super restricted with Like, I
24:54
mean, I'm like, from my point of view, it's not super nice for on
25:00
premise, two relations. That's something that I've heard
25:04
from a few other providers, where it will get you 80% of the
25:10
way there. And then, if you want to add
25:13
that extra 20, you're going to suffer.
25:16
So yeah, definitely, I don't back to the how to avoid
25:19
suffering that is probably one choose choose.
25:23
Wisely but it seems like you chose back in the day and now
25:29
you're regretting it, I'm saying I'm writing about you because
25:36
just because personally me I was not thinking about all these
25:41
cases if I were thinking about all these cases I may be able to
25:47
I would might different challenges but at that point of
25:52
time. First, I didn't want to reinvent
25:55
the wheel, honestly. So, and even now.
25:57
So I will say that I'm going to or we are going to teach
26:02
something. So I'm is better for me to work
26:04
with officially supports or bike supported by Community open
26:09
source tool and try to enrich it somehow like with our internal
26:14
tools or with homegrown to society or so.
26:18
For example, a mofo has super nice and super rich API which we
26:22
still can use. So we still can use it in
26:24
different projects and different aspects of cetera.
26:28
But yeah. So I would say how to avoid
26:31
suffering in. Don't start with very inventive
26:34
grills, check experience, which I mean you're not the only one
26:38
with this problem, especially in our roles, we have a lot of
26:43
companies, which try to solve the same problem.
26:45
So try to reach to the community and try to use tool, which
26:50
solves 80% of you unfair, Use cases and maybe 20% restraint
26:57
verse percent. You will be suffering but just
26:59
will not be like 100% suffering that.
27:04
And if anybody else has any any of those stories, let us know in
27:09
the chat of using one tool and it gets you 80% of the way
27:13
there. And then that last 20% was a bit
27:17
of a headache. The next question we're getting
27:20
from carsten is is your Work constrained because your
27:25
stakeholders are data scientists.
27:27
Do you have to make compromises? Because of that we've talked
27:31
before about how it can be a pain to use notebooks.
27:36
So would you force the data scientists to write Python
27:40
scripts or are there any other friction points like yours here?
27:46
I have multiple answers first I don't feel myself constrained
27:50
who uses pairing Call this as a data scientist say because data
27:56
scientists is that I mean, okay, I think that this scientific
28:02
data science is my clients with saying like, if if I can have
28:07
analogy like of consultancy and like, Consulting business here,
28:12
I'm I'm, I will say that here. I have my own startup in which
28:17
I'm, I'm pretty new to taking part in sir and we have clients
28:21
like the design T so and And it's not constrained.
28:23
So I need just, I need you to understand their pain and how to
28:30
help so they're pairing. And that's from nice.
28:33
Because personally, me like hello almost engineer or data
28:37
engineer. I don't know what to create.
28:41
I mean, I can great platform. Like, anyways, I'll say it
28:45
state-of-the-art platform but if it doesn't solve the problem of
28:48
data scientist who need to train model too, Work like do a lot
28:54
model to check Matrix to serve model or what hair really is.
28:58
It will be piece of shit. Sorry kids.
29:02
Well and so that begs the question you must be working
29:06
closely with the data scientist. How does that interaction
29:11
happen? Yeah.
29:16
Except except that. We have like very I mean it's
29:23
That we have channels like in Matamoros.
29:25
That's another course. Like I said, sure, but we have
29:26
Jeff where we constantly floating about everything,
29:31
especially release the release of related and code related.
29:35
But also we have different environments sessions with
29:38
different. I would say, it's different
29:41
level of representativity. So we have alignment sessions
29:45
with data scientist. Also, we like in short one, and
29:49
it's just between team leads. Sarah.
29:54
Also, we have alignment sessions.
29:56
So for broader group, where we share our progress.
30:00
I mean, we share our progress, like, they'd Engineers Machinery
30:05
engineers and they discuss their own topics.
30:09
Sometimes it's super boring for me sometimes it's not, it is
30:14
not, but yeah, so and this Loop needs to be.
30:19
We need to, we need to Have this Loop all the time like open and
30:25
we need to communicate also when I don't know how it happened,
30:30
but it's super nice. Because when something is not
30:34
working data science is the science.
30:38
Everything directly to some Engineers who like, who is
30:42
responsible for that feature or with cool.
30:44
They were taking about like this feature cetera and the
30:47
supervisors because just after like after release something
30:52
broken, And we were speaking on standard like yeah, I mean we
30:55
know this is it is broken. Maybe we need to fix it like a
30:58
colleague of mine says yeah. So yeah, I've already had the
31:01
talk with data scientists like yeah I know I'm going to fix
31:04
this when the super nice. I mean just speaking speaking
31:08
speaking this is that's there's not there's not another answer.
31:14
Yeah. So it's that's really the closed
31:18
feedback loop or the the short feedback loop.
31:20
And then also I like what You said that there are, I can't
31:25
remember the name now, I had it in my flight but the the
31:28
different sinks that you have on the different levels.
31:32
What did you call it? The alignment sessions that I
31:35
think is huge too. It's very interesting to think
31:38
about. So we've got another question
31:40
coming through the chat from Daniels, shout out to Daniel,
31:44
he's saying Jack handle on, VP of data, at Reddit, had a great
31:49
quote, the other day where he talked about how.
31:52
So, Some providers have pressured to grow their
31:55
offerings. Horizontally and start spreading
31:58
to new things. Yes, totally one example of this
32:02
might be weights and biases expanding from experiment,
32:05
tracking to hyperparameters sweeps.
32:08
Yes. Again, especially because they
32:11
just got that round of funding. Just curious if you have any
32:15
thoughts on how to think about this kind of thing in build
32:18
verse by decisions, users by decision.
32:23
What does sorry? I'm in Me Maybe.
32:25
Yeah. Basically like when you have a
32:30
company and you buy them for one piece of the puzzle by, right?
32:37
Yeah. And then they start to grow and
32:39
they start to take up more of the space because they're
32:42
getting pressure probably outside pressure from investors
32:45
or they're just they want to try and be like any company and make
32:50
more money, make more sales which is lately respectable, how
32:55
do you look at that when you're analyzing the build verse by
33:00
decision I found on clearance from to - honestly soon because
33:06
When you hear, if I understand correctly.
33:08
So when you, when you pushed to like grow, and you push to
33:15
deliver different different tools, Etc.
33:19
Okay. And I have, I don't know why,
33:22
but I have company on my mind, this date of weeks.
33:26
It has very nice, very nice product like this part, and if
33:31
it's a provides, very nice cloud-based.
33:35
A Like solution of his part but also under the why, but it is
33:40
pushed to deliver different as they buy products.
33:44
And sometimes they are gue, sometimes they are not good.
33:47
So overall I would say that overall they want company is
33:53
pushed to deliver something, then I guess they have good
33:57
reason for this, but I begged them, like, please check
34:02
offering check quality as Thatcher have Closed loop,
34:07
communication Loop is community, Etc?
34:10
Yeah, yeah, completely. Now, one thing that I like to
34:15
know is about this idea of when to build and win to grab and
34:23
open source tool and if I understand correctly, what you
34:28
all were doing, was using a lot of or you were basically
34:33
building a lot of your own and then you Started to say, hey, it
34:38
seems like the ecosystem is mature enough.
34:43
Now we can use some open source tools.
34:46
Is that how it worked or did you go from open source from the
34:48
beginning and just try and build on top of that Island?
34:52
I will not say that. We were, we were writing for
34:58
open source to be mature enough. I mean, because honestly, when I
35:03
joined when I joined I didn't we already had a very mature, very
35:08
mature, open source ecosystem. So I mean, I'm not fully aware
35:15
about all the decisions which were way before before before me
35:20
it's a train but went to, I'm, I'm finished.
35:25
Open source. I will say, when you when you
35:29
can and when you don't have any legal Regulatory Compliance
35:34
restrictions and go with open source.
35:37
And if you can also return to open source because adjust in
35:41
that way, we can have good Open Source Products, Google's open
35:46
source, ecosystem. And yeah, so I is that.
35:49
That's an interesting one and I know there are a lot of people
35:54
that are very Very deep in open source or very big Advocates of
35:59
it. But I'd love to hear your take
36:00
on why you feel. You would rather have open
36:03
source than either building your own or buying something.
36:09
Yeah, I mean, we can even buy open source.
36:12
Pi means there is huge because there are big company which grew
36:17
on open source. Let's say data Britax connect
36:22
like Kafka constant. Sorry.
36:24
Yeah. If their product like consent,
36:28
this is Texas State Farm. I'm not as is not entertainment,
36:32
but I'm saying because there is companies which offer Open
36:35
Source Products, like we support, but I would say, hmmm,
36:44
always, if you can, if you don't have restrictions, always try to
36:49
go with open source. Just because there is open
36:53
source, is supported by Huge number of people like I mean and
36:59
this number can be like 10 times 100 times more than your team or
37:03
your company and they should they share just knowledge and
37:08
experience in that products. And you can be smart as enzyme
37:13
or you can make their stupid mistake which will be spotted by
37:18
some other person like in an open source community.
37:23
Yeah. Do you feel like you need to
37:26
have a certain maturity or a certain number of Engineers to
37:33
be able to do something like that, though?
37:35
Hmm, about maturity. No, I wouldn't say That's Unique
37:38
to be. You need to wait 10 years after
37:41
that's trying to commit to open source, or to like to resolve.
37:47
Some of her, like box in your own private open source tools,
37:51
you need to write Because some of us are open source tools and
37:55
striding. I mean, they need to help like
37:59
in even inexperienced help like for fixing boxing documentation,
38:03
or somewhere else. Like and the second part of
38:09
passion, I might be, I might forget, but you were asking if
38:13
we need to give why we need to give back?
38:16
Yes, well, no. I was thinking more along the
38:18
lines of a company that might not have the engineering.
38:22
Sources. And so is it wise for a small
38:27
start-up? That's trying to move fast to go
38:31
with open-source still in your opinion, or should they?
38:36
Yeah, my something I mean it's person cause that if I would say
38:46
to go with open source because you need to understand what you
38:49
need to buy. So Very very simple example,
38:54
like with micro services or with the database service
38:58
microservices, you don't need to start directly with micro
39:01
Services. You need to start with your
39:03
monolith and to grow grow and to see what parts needs to be
39:08
separated. This thing is database, you
39:11
don't need to start directly with the no sequel or with
39:16
Hadoop cluster. I mean start with small village
39:18
Liberation database check, patterns strike.
39:22
What Our growing essential outgrew, your database and Etc.
39:27
So for for buying commercial supported products, you also
39:33
need to have that use case, but you need to outgrew some small
39:38
Open Source Products, or some widely adopted promise.
39:42
I mean, maybe, maybe it can save a lot of money, I guess, I it
39:50
can save a lot of money if you buying, for example, Some
39:54
database from Oracle or whatever.
39:55
But everyone's favorite database provider, everyone's favorite
40:01
database provider Oracle, but honestly, you need for paying
40:09
for something you need to have use case for the to and
40:14
basically you need to out-group, your own resources and here and
40:19
I mean, your own skills which you have In your company or in
40:24
your team. Mmm, that's a great point.
40:26
So, let's talk about this process of you translate
40:30
transitioning and I know they're there was something that you
40:36
were. You mentioned to me and it's
40:38
like finding the correct process in the transition from and
40:45
scaling up right and how to find that that process.
40:50
How to know when You're doing something right when you're,
40:55
you're transitioning from homegrown or you're building on
40:58
top of Open Source and how can you tell what your true north
41:02
is? Is it just by talking to the
41:04
data scientists and making sure there's that feedback loop we
41:07
talked about earlier? Or are there other ways?
41:11
Hmm. No, sir.
41:12
There is no other race who, you know?
41:14
But I mean there's no other words.
41:17
So you need to speak visit the Sciences.
41:19
Why? Just because, I mean, Is there
41:23
is migration strategy. It has two names.
41:27
I don't know why it is called green, red or green black green
41:31
strategies actually. I mean the whole idea of it.
41:35
You you have your alt installation which does
41:39
something and meantime when it does something, can you maintain
41:42
it? And maybe adding new features?
41:44
You are building new installation but this one old
41:49
green and this one bladder red diamond.
41:52
Doesn't matter. So I mean there is slight
41:54
difference between black and red red.
41:56
So and why while you building it?
42:00
And it's not it is not fair project and it can last for a
42:03
year or two years old, I don't know like for sure to block will
42:07
have lasting more than two weeks.
42:09
So and why are you building you need to communicate with your
42:12
stakeholders data scientists, like what features you need to
42:16
migrate from old one to the new one and the what and especially
42:22
what does it? For all, just take all this on
42:25
for you to get a like to have feature parity between all the
42:28
installation new installations because maybe some features, or
42:31
some use cases, or some functionality will be slightly
42:35
different, but it will be feature, parity, subsequently.
42:42
And just after getting feature, parity between two installations
42:47
you can have some official point in time when you switch off your
42:52
oldest Ation sexual. I mean, there is the there are
42:56
possible different ways by switching station called
42:59
installation because you need to make sure that new is working
43:03
correctly that introducing saying results saying data, Etc,
43:07
you need to compare so like all the sermons but they just one
43:12
thing which you need to assure first, this feature, parity
43:17
that. And that's how you can achieve
43:20
just by communication with data. Dentists data analysts or
43:25
whomever else like your team needs to have buy clothes.
43:31
Look like you've rolled out new feature or some like Technical
43:36
and non-technical already test detest that on new installation
43:40
and also Nation check that yeah that's the case.
43:42
It's cool and afterwards you can be sure the tires indicate that
43:47
I think is a brilliant point that you made about the future
43:51
priorities and how Ow, you need to know what to take with you
43:55
and what to leave behind. And when you take it with you,
43:58
it shouldn't just be this lift and shift.
44:00
It should be an upgrade, right? I've heard people talk about how
44:05
they don't believe in just lifting and shifting.
44:08
They think that whenever you're going to actually migrate you
44:13
should migrate for the better. And I'm sure for sure, even like
44:19
to sound are like API or UI Remains the Same.
44:23
But honestly, you need to invest in a bit more sources into
44:29
implementation, just to be sure that the next guy who comes
44:33
after you will save it in Specialties.
44:35
Yes, because of that. Yeah, we got a question coming
44:40
through and then I want to ask you about.
44:43
So I'll ask this question. And then later, I'll ask you
44:46
about some more stories because I'm sure you have a lot of them.
44:49
And I'd love to hear what went wrong.
44:53
That's always fun to talk about. Yeah.
44:55
But let's hit this one first. How do you know?
44:59
If one of your models being served in production is working?
45:03
Well, or needs to be retrained. I guess this was because the
45:07
monitoring piece. What are you using for
45:10
monitoring? This is another huge part of the
45:14
puzzle. So how are you keeping tabs on
45:16
these models? Basically, a lot of vestments.
45:21
Do you also? Piggybacking on the monitoring
45:25
infrastructure. Which was build by were
45:29
brilliant infrastructure engineers and it's super bored
45:34
myself. Maybe for most of you guys but
45:38
for me to use scrapping and we just visualizations with methods
45:41
on my dashboard. Interphone you find that to be
45:47
good enough because this is one thing that I am fascinated about
45:51
is if Ami theists and Griffon. De are good enough for the data
45:55
space and because you see a million tooling companies coming
45:59
out right now? Well, not a million but there's
46:02
a lot that are popping up like mushrooms that are specifically
46:06
monitoring for data for. I mean, hear me speaking not
46:12
monitoring data but we are speaking about monitoring our
46:16
models on production comes are served.
46:18
Yeah sorry. Did I say yeah?
46:20
Exactly. I mean yeah.
46:22
You friends and monitoring data. Because once you monitoring
46:26
models and Productions is basically we have a couple more
46:29
efforts to monitor. Okay, not a couple but you have
46:32
different two, different groups of monitoring this monitoring
46:37
like web service itself and all the hardware metrics of server
46:42
which which we have assessments of I guess it's super boring
46:45
style because there is all the blog posts, Etc.
46:50
But basically you scrubbing was permitted to do so. the metrics
46:53
are, how does it perform a search for our health checks or
46:59
even if you if you use some service Discovery example,
47:03
console you can use this Bruin tool or I guess from Netflix we
47:08
have Erika like service cover to which already know if there's
47:13
his house and if it's rank is a Torah and second stuff which you
47:19
need to monitor with your Wait, there are also models.
47:23
That's how does it performs like interference?
47:28
Does it decorate or if it's not equal 8?
47:31
And till this moment, grandfather is still good for
47:38
having. I mean, maybe it's lagging a
47:40
bit, but to compare that interference of new model is
47:47
better than old one etcetera. It still can represent.
47:51
So it becomes A bit messy when you have, what you want to, when
47:56
you want to describe are very want to see huge number of
48:01
services. How does they perform on stage?
48:04
Crying different data, centers different machines?
48:07
Like even the same model, but which was deployed in two
48:11
different data centers. Like, you only can have very,
48:15
very clumsy like you, I such as well.
48:18
Yeah, but it still doesn't does it.
48:21
It's job, and do you have Automatic re triggering,
48:26
retraining of models set up, we still working.
48:31
This is so we just, I mean, let's let's clarify because
48:40
different people or different kinds of things or different
48:44
companies, understand different stuff with automatic with rigor
48:50
so what can be nasty? Medical taking.
48:53
It can be just training as a model, so and it's all or it can
48:57
be trying the model and deploying it to production.
49:01
So trains model is to make it magically.
49:06
It's kind of easy to do with state of state of the art or
49:11
orchestration to like air flow for example because when you
49:14
have dark, which does this, you need just to release the code or
49:19
release your data cetera. So and it's already Scheduled
49:23
and it will be automatically retrigger.
49:25
Even even if you don't have when you don't have scheduled the
49:30
scheduled time for training, you still can trigger with airflow
49:35
API or Air Force here after matically.
49:39
But if he's speaking about deploying it to the production,
49:45
I mean, our part of stuff still needs to be still needs to be
49:49
done. If we still yet to discover How
49:52
its should be done, and it's in our space.
49:55
It's very interesting. And super hard, I would say
50:00
challenge because because financial institutions, they're
50:05
super regulated, we need to have different compliance.
50:09
For example, different people the same person cannot own
50:14
Entire deployment or entire infrastructure.
50:17
So, if I'm building something, I cannot own, its deployment has a
50:20
transom. And that's, that's which we need
50:25
to obey. And there is numerous of
50:28
different different restrictions, which you can.
50:32
So we need, I guess for us, it's to get to be discovered and we
50:36
need to understand how to do it correctly because everyone wants
50:40
like to have just one button, click it and modern production.
50:43
This is getting life traffic Etc.
50:46
But unfortunately, no, there should be some like some
50:49
friction added. Yeah.
50:51
Yeah it's interesting. You say that too.
50:53
There is a lot of talk about that and it still is not clear
50:59
if that's the best way of going about it, right?
51:02
Like it is nice like you, say everyone wants that but as
51:07
you're mentioning is, especially in the financial space you have
51:10
regulations and you can't just do that without Without keeping
51:14
those regulations in these different has different laws in
51:19
mind. So that is a very important
51:23
point to keep in mind. Now, if initial initial sphere
51:29
is no, he's not super regulated. If we compare even with
51:32
Healthcare, yesterday was speaking.
51:34
This friend of mine from from Health Healthcare and he has
51:37
much worse situation because the because of explain ability, I
51:42
mean, just imagine if Same goes wild beasts his model which
51:46
predicts like someone health condition and it's much worse
51:52
problem. And I'm, I mean, I'm willing to
51:57
say that I'm super happy that I'm not in health care, but it's
52:01
very nice problem and super big problem to tackle also nice.
52:07
So, a quick question before you give us a war story from Jack,
52:11
he's asking what kind of Hardware your Using for the
52:14
on-prem model training. Mmm.
52:21
Yeah. However, for on time training,
52:25
we don't have any special Hardware.
52:27
So I guess that usual machines machines with 32 cores for some
52:34
sequences. I don't remember how much rum
52:38
but I guess around. Bigger than half.
52:41
Their, I don't know. The other members are just using
52:44
machines with with the serving machines.
52:47
It's a bit, it's a bit harder because serving machines you
52:52
need to understand it. Machine learning machine,
52:56
learning program model prediction, that's Fuel.
52:58
And so as as are you need to have and you cannot serve more
53:07
models than or I mean you cannot see ya.
53:09
You cannot serve Models is more than your piece of course on
53:13
your machine. This out there because because
53:16
everyone wants to have very fast interference like request,
53:20
scoring to response request responses with.
53:22
So basically you have cpu-bound, see is cpu-bound process with
53:31
ram also. The reason there is not so big
53:35
house. A yeah, there is no constraints
53:37
with rum like you need to get all the for a massage chair.
53:40
Because requirements or there because you need just to have as
53:44
much from as the you have models, as total size of the
53:52
models to be sterilized and can be placed in from.
53:55
So, answering your question, also about serving which you can
54:01
pull out, of course, a lot of cars.
54:03
Yeah. Now let's hear it man.
54:06
The moment everybody's been waiting for.
54:08
Give us some Of your War Stories, whatever, or horror
54:13
stories. Even what are some things that
54:15
you've learned along the way to finish out this Meetup horror
54:21
stories? I wasn't.
54:24
I mean it's not super Horror Story about this.
54:26
Share sharing of experience guys don't do context switching
54:33
because this context, switching key constant constant context.
54:36
Switching you can do mistakes, for example, removing the data
54:40
From the hdfs skip trash and as far as checking what data you
54:44
did remove. So don't do this.
54:49
Because from that stuff, comes all your from Context switching
54:55
and without a, without focusing on some stuff which needs to be
54:59
delivered. You can, you are doing a lot of
55:01
mistakes. Also, it's our internal joke,
55:06
which I want to I want to fix some somehow, but I'm not
55:12
allowed because I mean, because of because of some because of
55:20
some constraints but device is naming of your products, guys.
55:24
Because we have router, which is not rather but server but it's
55:29
called server. And we have Gateway which is not
55:31
Gateway but it's route dys vizier namings.
55:38
Sounds like a lot of fun. And so you have to automatically
55:42
train your mind to think router is actually server and Gateway
55:48
or gateways server and router is not a.
55:50
Yeah, yeah, yeah, yeah. We always ran down the mines and
55:54
the, but it is fun. So, like, Percy our failures,
56:00
which we have even cannot force, you cannot recall them.
56:08
Why? Because I wouldn't say that
56:11
Sergeant is that company, which allows you to make a lot of
56:15
mistakes, but you can make mistakes.
56:19
So, and he, I mean, if you making mistakes and you're not
56:23
being punished for that, so you just okay, okay?
56:26
Then I can improve myself and this is why I'm, this is wife
56:31
for last few years, for sure. I did a lot of mistakes,
56:34
especially with immigration and especially, with communication,
56:37
because I'm super optimistic person and, like, oh, Okay.
56:40
Now we can be great and afterwards, feeling, right?
56:42
Think after I do like, a couple months Etc.
56:46
But that's part, which I'll do like about audience.
56:50
And it's, I mean, you can make mistake is Forgives, I mean, to
56:55
some extent, but not every totally scored by these
56:59
different mistakes that you've made.
57:02
They've passed through your mind, and you can't remember
57:04
them now. Yeah, yeah.
57:06
And that's also examples that I did all of them.
57:11
Well, I really appreciate you taking the time to sit with us
57:15
and talk and go through this process and really talk a bit
57:19
about everything. We bounced around a lot.
57:21
So thanks everybody for joining us and coming along for the
57:26
ride. But you gor.
57:28
This is it man. I really appreciate it.
57:30
And for all of those that are listening, I didn't even put a
57:36
link to our slack but if you want to check out the slap, just
57:39
go to Olaf's doc Community or if you're listening in the future
57:43
then you can check the links below and that's it.
57:46
I will see you later. Thanks again and Gore.
57:49
Thank you. The matter is.
57:50
Thank you guys for listening. It was very nice experience.
57:53
See y'all?