Latest / Agentic Conversations (formally mlops.community) / I Am Once Again Asking "What is MLOps?" // Oleksandr Stasyk // #308

Transcript

I Am Once Again Asking "What is MLOps?" // Oleksandr Stasyk // #308 · Agentic Conversations (formally mlops.community)

0:00
Hello, I'm Sash. I'm an engineering manager at
0:02
Synthesia. And to take my coffee out of the
0:04
machine and then drink it, folks.
0:09
Welcome back to another ML OPS community podcast.
0:11
I'm your host Demetrios and Sash schooled me to so many things in
0:16
this upcoming conversation. He has truly done ML OPS in many
0:23
different places. Right now he is creating the ML
0:26
platform and as you probably know the requirements and
0:31
expectations are quite different when you're creating a video
0:35
generation type of product versus what he did at his last
0:39
job around chemistry and creating an MO platform for a
0:44
chemistry LED MO initiative. He broke down the different
0:50
parts of this and where he has seen initiatives fall flat.
0:55
He also talked through how much of A cultural thing this is, not
1:01
just the technical thing, which we harp on here many, many
1:04
times, but this really felt like the man has gone through the
1:10
trenches. He has some scars and he is
1:13
sharing his wisdom with us. Hope you enjoy.
1:17
And if you are listening on one of the podcast player platforms,
1:22
not on YouTube, I've got a little treat for you because all
1:26
these folks that are joining the community are giving me some
1:29
incredible recommendations when it comes to music and I get to
1:35
pass that on to you. This next song is from a band
1:39
called The Voids and the name is Leave It in My Dreams.
2:22
I literally just posted something that was more or less
2:26
a meme saying like pour one out for the homies who thought they
2:31
could vibe code their way into $1,000,000 product.
2:34
And it really strikes me as funny how much I'm seeing folks
2:42
talk about how now you don't need to be a software engineer
2:46
to become a millionaire with a SAS product, because you can
2:52
vibe code your way into a SAS product and all that fun stuff.
2:56
Yeah, sometimes I guess people have to learn the hard way.
2:58
Yeah, but even I I do respect that There's a lot of great
3:04
things that you can now do if you understand software
3:07
engineering. I I have 100% agree.
3:11
To me almost feels like, you know when the vibe coding has to
3:14
stop. That's the that's the key skill
3:17
vibe coding is all good, but you need to understand when what was
3:21
generated is unmaintainable. And that's when you kind of say
3:25
it's, it's enough of the vibes for now.
3:28
Getting too many vibes. Yep.
3:31
I got to stop with the vibes. Bad vibes.
3:33
Bad vibes. Yeah, that's so funny because
3:37
for me, I I laugh because when you watch videos of folks saying
3:42
like, oh, I'm just going to vibe code it, The prompts are very
3:46
much coding prompts. And if you have known nothing
3:49
about coding, you can't prompt your way into that type of
3:55
thing. Like you're prompting with very
3:58
coding specific language. Even if it is in English, it
4:02
still is like you. Describe the algorithm.
4:05
Yeah. Or describing the exact things
4:08
that you want and understanding what you need and the
4:11
requirements there. It's it's pretty difficult in my
4:15
eyes to be able to do that. And then the whole reason that I
4:19
came up with this was because there's the meme going around
4:22
someone on Twitter that said something along the lines of
4:27
help me, I'm under attack. Since I've been sharing about
4:30
how I vibe coded my product. Now people are like D Dosing it
4:36
or they're finding security breaches and I don't know what I
4:40
created and I don't know how to make it more secure.
4:46
Some problems should not be solved with code, I feel.
4:49
And and you know, vibe coding kind of implies that you're
4:52
writing code and you've got, you know, it's text in, text out.
4:55
And that's what developing software is all about, right?
4:59
You just write some symbols and the computer interprets it and
5:03
outcomes value and most importantly, Benjamin's.
5:06
But in reality, like you know, sometimes you go aha, maybe we
5:10
should solve this as infrastructure, or maybe we
5:12
should just actually move the file over here and I'll make
5:14
things a lot simpler rather than brute forcing it with with wipe
5:17
coding, sometimes you don't need code at all and just a
5:20
spreadsheet will do. Yeah, or you have to think if
5:23
someone has no idea about security best practices and.
5:30
They forget to ask. They're, yeah, they're putting
5:33
their secrets everywhere, or they aren't even thinking about
5:35
secrets and they aren't even thinking about the most basic
5:39
common things that you learn as you're going through the trials
5:45
and tribulations. You you're going to learn the
5:46
hard way. Yep.
5:47
Yep, Yep, I it reminds me once I asked a lawyer.
5:51
You know, you know, in in a very sick way, I'm kind of interested
5:55
in law, you know, like, how hard is it just for me to do like,
5:58
you know, cheeky little bit of lawyering without any of the
6:02
background, any of the experience, any of the
6:05
education, Like, what could, what could possibly go wrong,
6:07
Right. Yeah.
6:09
You know, just read the book. It's like, you know, just like a
6:11
giant document the law, right? And you go, yeah, there we go.
6:17
And I'm sure the lawyer was very fond of that question, but yeah,
6:20
I was like, yeah, give it a shot.
6:22
And just said end call immediately.
6:27
Yeah, why would you ever need to pay a lawyer or a software
6:30
engineer when you have ChatGPT absolutely vibe code or vibe
6:35
law, vibe defend yourself, all of that.
6:37
So. We are on another hand, on
6:39
another hand, I think the Brexit is allow it's, it lowers the bar
6:44
for people to actually explore. Like there is, there is
6:46
definitely a light side too. I think not, not to just the,
6:48
you know, completely shit on vibe coding.
6:50
Like I think it gives people the opportunity to do to, to
6:53
actually get into stuff because like last thing you want to do
6:57
is be a gatekeeper. Obviously like no vibe coding
6:59
for you because you need to have four years in uni to, to study
7:02
how well NN of n ^2 is, you know, O of n ^2 and all the
7:07
complexities. But this way, you know, people
7:10
can get stuff going quite quickly, even though it might be
7:12
wrong. I think guidance is always
7:14
needed. But the fact that people are
7:18
actually enabled to do this, that's, that's almost a
7:20
completely different world to live in.
7:22
Sorry, I just wanted to bring a little bit of light to the
7:25
situation. Yeah, we got to steel man it
7:27
because like I said, there is some incredible stuff that's
7:30
happening with it. I think it just needs to be
7:33
taken with a pinch of salt and like you're saying, the ability
7:38
to explore and hopefully peak your interest.
7:43
And then you go deeper down the rabbit hole and you're now
7:46
becoming more and more familiar with when you should do things
7:50
when you shouldn't or how to explain yourself.
7:52
How to know when this vibe coding is going off the rails?
7:55
I think exactly yes. The key?
7:57
Knowing when to stop and ask for help.
8:00
Or knowing when to hire someone that can actually do it.
8:04
Yep. Oh, but that, that's an even
8:05
bigger, that's an even bigger situation.
8:07
Vibe coding aside, knowing when to when you need to hire someone
8:10
that's a. That's a huge thing, isn't it?
8:13
Yeah, Vibe hires the ChatGPT, the Test Driven Development
8:19
doesn't tell you, hey, you should probably hire someone
8:23
now. That's classic.
8:25
But dude, I wanted to talk to you and I know that you've also
8:29
been interested in chatting because you have this idea that
8:34
ML OPS is more important now than ever.
8:36
And so it's great to hear that because in a way because of all
8:43
the vibe coding, because of all the AI hype and you don't hear
8:47
about how important it is to have these foundation and.
8:52
Foundational pieces. Yeah.
8:55
The best practices, the idea of whether it's Gen.
8:59
AI that you're putting into production or it's traditional
9:02
ML, you still need the those foundational pieces that are
9:07
hopefully going to be a part of everyone's project.
9:11
And so let's just start with that.
9:13
I know you wrote an awesome blog post like 2-3 years ago all
9:18
about what is MLA. It still holds in my opinion,
9:22
but I could definitely write another one.
9:26
Exactly so. So let's just hear your thoughts
9:29
on this and then we can dive into it.
9:32
So the blog post specifically was incepted in my head.
9:37
There was a scenario, I was given an MOPS team.
9:42
I got told go do MO OPS and now and that was happening.
9:45
And in the meantime, there was another department who was kind
9:48
of on the side and they're saying, oh, we would also like
9:50
an MO OPS team. And it was definitely not
9:54
something that we could both hold.
9:55
So it's definitely time to hire. And the interesting question
9:58
was, and to be fair, this was me quite new to MO OPS, etcetera,
10:01
etcetera. There's more details in the
10:02
article behind it, but the the interesting question was what is
10:06
the skill set? What do we put on the job
10:08
description of, you know, you need to have XY and Z to be
10:11
envelopes And boy, oh boy, I sat there and stunned.
10:15
You know, I don't have an answer to that because if you list
10:18
everything, you're never going to find that person.
10:22
Like good luck. So the the the blog post kind of
10:26
reflects upon the idea thinking about.
10:29
So I myself, big confession, I don't have a PHDI don't have a
10:32
researcher background, which I sit on a chair of imposter
10:37
syndrome and I see all these people with their chevrons and
10:40
their masters and their professorships and but on the
10:45
other hand, I can claim that I got some things working in
10:47
production. I definitely know pods are
10:49
failing and I got them fixed and I looked at logs and I know
10:51
tracing and I know Grafana is pretty sweet.
10:54
So I have that to support me. So I imagined that the skill set
11:01
of an envelope engineer kind of takes the idea of AT shaped
11:07
skill set. So you have one long or deep
11:10
specialization in something and kind of divides it.
11:12
So one of the legs becomes your foundation where it came from,
11:17
be it either a software engineer or a researcher.
11:22
And it kind of drives you towards the other side.
11:24
So you start off with solid soft engineering background and then
11:29
you get told you'll make some models in prod and get them
11:32
served. And then you kind of dive into
11:35
that world and the media get hit with a cross key validation and
11:37
you go, where are these names? I don't understand.
11:40
But slowly but surely, hopefully you're surrounded by people who
11:43
can help you learn that this is a model, this is a feature, this
11:47
is drift, This is what inference does.
11:50
And then you can see, ah, I can see there's a problem here in
11:53
scaling. Let me just do some micro
11:54
service magic and make that scalable, right?
11:58
And there's vice versa, which is I think almost just as beautiful
12:02
because the combination of these works really well together is if
12:05
you're a researcher and you go, right, I've got super cool model
12:08
in my trip to notebook. I get my 99% accuracies.
12:11
This is super sweet, but boy oh boy, how do I give it to my
12:15
customers? You can't just walk around with
12:17
your laptop letting people use it to influence your model,
12:19
right? So you got to you got to, you
12:22
got to do the deploys, right? And then you go Google it.
12:25
How do I deploy things? And then all of a sudden dockers
12:28
and Kubernetes and everything starts falling onto you and you
12:31
kind of go, all right, but then what?
12:33
I think the beauty is and when those two skill sets 2 little
12:36
pie shaped engineers or ML Opsers meet and that's when the
12:40
real magic happens because they can support each other, right?
12:43
There's also arguably 1/3 little leg between them knowing you end
12:46
this year. Please, this is a family
12:49
friendly challenge. Is the actual domain
12:55
specialization right? So for example, yes, you have AI
12:58
and yes, you have software engineering, but what you're
13:01
doing, right? Are you doing chemistry?
13:04
Are you doing computer vision? Are you?
13:06
Doing. Finance like so that's the
13:08
little, little, little thing that is the genuine, you know,
13:13
domain expertise. So you can argue that like, you
13:15
know, it could be super duper knowledgeable about the domain
13:18
and then you can kind of go out there.
13:20
So like you can stretch this analogy left, right and centre,
13:23
obviously. But like the main idea is that
13:24
when you say ML OPS, you kind of go, what does that actually
13:27
mean? Like who, who's this ML OPS
13:29
Unicorn that can do all these things together?
13:32
But in my opinion, it's not about who, it's about the
13:35
strength is in the team because then you have economy of scale,
13:39
you have the sum is getting part of the part of the some, some of
13:44
the parts the whole sorry and and like that's when the real
13:49
beauty happens. It's about your 10X environment.
13:53
Yeah. The other role or special
13:57
specialization that I would think about is a data engineer
14:00
and being able to get the data to where you need to go.
14:03
And, and this is, it's interesting that you're talking
14:06
about it in this way because it's very much in my eyes,
14:11
something that you would potentially need more of these
14:15
specialties in the traditional ML world.
14:21
Do you feel like now in the Gen. AI world you can have a bit less
14:27
or is it just that it's different?
14:30
Well that's a good question. I do have some opinions about
14:34
this current Gen. Gen.
14:36
AI world, but what I do think the real winner is is money
14:40
making because money making is a much more important thing now
14:46
because you want to get that stuff going faster.
14:49
Yeah VCs, please check out my cool video that I posted on
14:52
LinkedIn, right, yeah, you have to sacrifice some things you
14:57
have to basically right and it's it's it's it's pragmatism at the
15:01
end of the day, right. So when it comes down to yo, by
15:04
the way, did you know data is important?
15:07
Like that is a foundation, like how to put it's, it's
15:12
essentially tag that you have to pay that pay down at some point.
15:15
And if you don't pay down, it's going to be bad news bears.
15:20
And then I think I think you could definitely take steps
15:24
towards it. But as you said, as you just
15:26
mentioned, data as aside from ML OPS.
15:30
Oh yes, my, my, my personal opinion.
15:33
And it's almost like a mantra that I always hold in the corner
15:38
of my heart and my head that like can't have AI without good
15:40
data. It's just that sometimes you get
15:43
lucky enough with data to get going, but other times you have
15:48
to like, you know, pay down the debt and get better at it or get
15:52
better data, essentially. You can MVP something really
15:57
quickly and it goes back to what we were talking about with vibe
16:00
coding. You can MVP something by vibe
16:03
coding your way into the MVP and having some one shot or zero
16:10
shot prompting. And you're using AI with your
16:14
product and it's just hitting an API.
16:16
And then when you want to go to a deeper level of production,
16:20
which you want to validate first, you don't want to get
16:24
there. And I, I imagine you've seen
16:27
this meme where it was something along the lines of, well, my
16:32
product has seven concurrent users, so I guess we need to
16:36
switch to Kubernetes now, yes. And so you don't want to over
16:42
engineer before you have to really and.
16:45
Absolutely. Yeah, yeah, yeah.
16:46
Inside of you, there's 2 rules. One of them wants to do the
16:49
right thing, the other one wants to do the thing right.
16:51
Yeah, exactly. And the thing that is, is
16:58
constantly going through my head is like, OK, well, you NVP it,
17:05
you get something out as quickly as possible, you test it.
17:10
And then once you see right on, there's there's legs here, let's
17:15
go for it. Then you're doing what you're
17:17
talking about very much like we're building the team with
17:20
their specific specialities so that we can leverage each piece
17:27
of the puzzle. And I'm just thinking back to
17:31
what you told me before we hit record and that is ML OPS is
17:35
more important now than it's ever been.
17:39
And so I I want to explore that idea a little bit more, like why
17:42
now more than before. Vibe coding, vibe organization
17:50
building. If you did, if you will allow me
17:53
to expand the analogy. And the question is, what is ML
18:00
OPS at the end of the day? Nitrous, right?
18:02
What are those questions? I'm sure we get asked that every
18:04
day, but I'll, I'll, I'll, I'll put this straight here.
18:07
Rather than being mysterious about the it's, it's no
18:12
different than essentially DevOps, but with extra things,
18:15
right? So you have to understand that
18:17
you need to break down the walls.
18:19
You need to be able to think about your feedback loops in
18:21
your organization. It's, as you said, validating
18:25
what you're doing, right? The more organizational barriers
18:28
are put in place, the worse the time you're going to get in
18:31
terms of getting your feedback faster.
18:34
And that is 10 times as true for ML because now you have a little
18:40
bit of, you know, research going on over here and then a little
18:44
bit of production going over here.
18:46
And if you let that grow into two separate things, what you'll
18:51
end up with two different companies essentially. 1 is
18:54
publishing papers. The other one is saying, yeah,
18:58
we've got a model that we made originally, but we haven't
19:00
updated it since. So envelops to me is making sure
19:05
that, well, it's not about making sure, but essentially
19:09
trying to undo that to, to create tunnels, to create
19:13
bridges, to create windows, but ultimately to melt away the
19:16
distinction completely. Even though sometimes it's very,
19:18
very hard because researchers got the research and you know,
19:22
not everything in production is about ML.
19:26
But what's most important is your value stream map from an
19:31
idea in terms of I would like to predict something to does it
19:36
actually give us the money, right.
19:38
And then the shorter you can make that a feedback loop, the
19:41
better off your business will be for sure.
19:45
So it's about being able to iterate on your ideas, being
19:48
able to validate them and as close to the customer as
19:51
possible, essentially, right? And then make those feedback
19:53
loops tighter and tighter and tighter and tighter.
19:58
Bring teams together, Melt the concepts of.
20:02
Research and engineering if you need to, depending on the skill
20:08
sets you have, you might need to have a projectorization step.
20:11
You know, you might need to rewrite some code for sure
20:14
refactorings, you know, maybe the Kubernetes is managed by
20:18
DevOps, right? But you can try and build your
20:20
organization towards in such a way where the process is
20:24
actually streamlined, right? So this is kind of my spiel in
20:29
terms of what I think Envelops is about, is about enabling your
20:33
business to actually iterate on your machine learning
20:35
efficiently. Yeah.
20:38
Efficiently and quick. And I think that you point out
20:44
something that is inherent in these projects, especially the
20:53
bigger the company gets, is how you need to break down those
20:58
walls between the functions. And you, you want to look at it
21:03
like we've said it for so long, but it does feel like it needs
21:09
repeating. And that is recognizing this is
21:13
just as much cultural and company culture organizational
21:19
as it is technical. And so even if you have the
21:23
coolest tech, but if each team is siloed, like you said, you're
21:29
going to end up creating two different companies where you
21:32
have one model that's been in production and hasn't been
21:35
updated for a while. And then meanwhile you have
21:38
another team that is just creating research papers.
21:42
Yep, indeed. And recognizing that this is
21:45
this is where the vibe organization building has to be
21:48
kind of like, you know, put on pause, you can go, right, OK,
21:51
now we're, we might think of a reorg, we might think of this,
21:54
we might think that. But all of these little
21:57
decisions come from, you know, sometimes arbitrary creation of
22:03
a team, even right? Say like, oh, we need to build a
22:05
team that does this right. And it's little by little, these
22:08
little actions and these little decisions that can solidify this
22:13
difference, essentially. Yeah, that was where I wanted to
22:18
go next was the idea of the organization and the team
22:23
structure and what you've seen work well and not work well when
22:30
it comes to platforms, platform engineering.
22:33
Do you need, is it the ML platform?
22:36
Is it a data platform? Is it just a platform engineer?
22:39
And how organizationally does that look?
22:44
Are you embedding teams? Are you centralizing these
22:47
platforms? Are you trying to do the hub and
22:50
spoke model like? Have you tried at all?
22:53
Each of those questions are almost worth the podcast in
22:55
themselves. I'll, I'll start with I'll start
22:59
with what I think works and what doesn't.
23:04
I think what's important is the skill sets that you hire.
23:12
And this is very hard to speak about what's good and what's
23:15
not. But imagine let's say you just
23:17
only hire back end devs and only hire front end devs, right?
23:22
If you just keep going like that, what does your
23:25
organization look like after? If you like, it's extrapolating
23:29
time, right? You have a whole bunch of people
23:31
who can do this and a whole bunch of people who do that.
23:34
And then hopefully you have graph KL to solve the hard
23:37
problem between the right. And then I, I see this almost
23:42
like equivalent, but you don't just have back end in front of
23:45
devs. Now you have AI researchers and
23:46
you have this and you have DevOps and you have that.
23:48
So I think, and this is where I try myself to be as I don't like
23:54
to use the word like full stack dev and all that kind of stuff
23:57
because like I'm crap at front end.
23:58
I'll, I'll put my hands up, right?
23:59
I do try, but like I get laughed at every time my friends see
24:04
what I do. I have a classic back end in
24:06
front of the scenario. But what's more important is
24:10
that even if you might not have the skills on the other side,
24:14
right, what you must at least have is the empathy, the empathy
24:19
to see things from that side, right?
24:21
What are the problems that the front end does face when you do
24:25
stuff in the back end, right? That's the first step, to be
24:27
able to actually work together, right?
24:31
Same thing with research versus product engineering, etcetera,
24:34
etcetera. Like you need to understand,
24:37
hey, like researchers value their freedom to be able to
24:39
experiment things. And that's very important
24:40
because you need to drive that innovation because if they're
24:43
not researching, no, they're not, they're not really doing
24:47
their job. So you need to enable that.
24:50
This is where the platform stuff comes in.
24:56
Hard to say in terms of what, what's what's good and bad.
24:59
I did ask one of my good friends that I work with, what is Emma
25:04
Lops, you know, one of those, one of those philosopher after
25:06
questions and his answer I still hold close to my heart, which
25:11
is, you know, it could be anything.
25:13
And every organization is different.
25:15
And I think there's just so much context in there is that like,
25:18
yes, you can kind of vaguely see like this is this is these
25:22
things you should do these days and you shouldn't.
25:23
So I've already kind of described them.
25:25
But when it comes down to actually practical things, what
25:28
do you actually do, right? How do you actually organise
25:30
your teams? Yes, sometimes you need people
25:33
who like, you know, look after infra and maybe call them MO
25:37
infra people, right? But what's more important, I
25:41
think is as a person who is in charge of hiring and building
25:45
this organization. So you really, really, really
25:47
need to know your people well, what their skill sets are, what
25:51
they, what their preferences are and understand this dynamic.
25:57
Because if you then just kind of go on, here comes an MO platform
26:00
team and they will do this and here comes a research team and
26:02
you'll do that. You just miss so much nuance.
26:05
Like that kind of goes out the window.
26:06
And it's, it's very easy to get things wrong that way.
26:10
So like you have to kind of be very careful and do step by step
26:12
and definitely don't be afraid of our, of our reorganization
26:15
just in case. Because like, you know, you
26:17
might need to change things. Sometimes when you build a, a
26:21
Lego house, you do need to like go back a few steps, right?
26:27
In, in terms of what not to do. It's also kind of hard to say
26:33
because like sometimes things generally just work well even
26:36
though they don't represent best practices.
26:39
Yeah, they shouldn't, but somehow it's working.
26:41
Yeah, yeah, yeah. It's, I'm sorry, I don't have
26:45
any clean answers here. Something, something, it
26:49
depends. And please my consultancy fee
26:52
for that? Are there specific things that
26:56
in your experience haven't worked?
27:00
I don't have any glaring examples of like, please don't
27:02
ever do this kind of thing, but I can definitely say like this
27:06
could be improved, right? I think it's which skill sets
27:12
you inject at which stages that can improve interaction.
27:18
So let's say some like to give another non envelopes actual
27:21
example is that, you know, somebody wants to make a micro
27:24
service and they need the database, right?
27:27
Do they make a ticket to DevOps to say, please, can I have a
27:29
Postgres instance, blah, blah, blah, blah, blah.
27:31
And then DevOps go, oh, that's nice, Please join the join the
27:34
queue, right? And then we'll see you in 20
27:36
days, you know, and you just want a database, right?
27:40
But imagine you're enabled to write the Terraform to actually
27:42
create your database, right? That little nuance there can
27:49
like unlock massive, massive innovations.
27:54
So when you have two teams trying to do something together
28:01
and you have enough skill set to kind of cover all of these gaps
28:06
and people with enough initiative to actually follow up
28:08
on that, then you can definitely have things progress.
28:14
Because sometimes it's almost not necessarily the things are
28:18
complicated to achieve, it's that there'll be a blocker here
28:21
because somebody just doesn't know how to do something right
28:23
and the right people are not in a conversation.
28:25
So end of the day, the answer here is probably have the right
28:29
people, the right skill sets and the right conversations.
28:32
Oh, I like that because you're looking at things in this
28:36
dimension of time, also in the maturity and as it matures, you
28:42
want to make sure that the right person is there.
28:46
And I also like the idea of understanding through empathy
28:51
that you called out. I know that one of one of my
28:56
friends, Chad Sanderson, talks about this a lot and how when
29:02
software engineering folks will make changes to data, it effects
29:09
so many different things downstream.
29:12
And it's usually the data engineer that has to deal with
29:15
that. And there's products being built
29:20
by the analysts and by the ML engineers or the data scientists
29:24
that are not working. And so they are putting pressure
29:29
on the data engineer like, hey, why?
29:31
Why did this not work? Like my model is throwing all
29:34
kinds of errors now. What is happening?
29:36
My heart goes out to my data friends, they'll probably be
29:38
watching this. Exactly.
29:40
I have, I have all the empathy in the world for them.
29:43
That's and it comes down to what you're saying, that if you can
29:50
empathize with what is happening so far downstream just as much
29:57
as an ML engineer can empathize with the software engineer for
30:02
why those changes need to continuously be happening, then
30:06
maybe there's some way that you can create a very low friction
30:12
idea of when changes are made they get double checked on
30:17
downstream. Yep, and you never know how much
30:20
time you might save someone else by just doing a little check.
30:24
Oh my God, that is such a great point.
30:28
You just think about all those wasted hours because of some
30:33
change that Oh yeah, this event, we don't need that anymore, do
30:37
we? Yeah, let's get it out of here,
30:39
meanwhile. And sometimes it could be.
30:43
So that's an example of YAGNI, right?
30:47
I ain't going to need it. For people who don't know, Don't
30:54
repeat yourself is also one of my favourites is just somebody
30:57
trying to do the due diligence and refactoring something and
30:59
saying, yo, let's let's let's just abstract this together.
31:04
And then downstream team wise, downstream time wise, downstream
31:11
code wise, for example, all of a sudden you go, Oh my God, we're
31:15
coupled now and we can't do XY and Z because reason, because
31:18
reason, because reason because this drying has happened.
31:22
You just got into a marriage you had no business being in.
31:27
Sorry, Romeo and Juliet, your great, great, great great
31:30
grandparents were at war and that's just how things are.
31:33
That's how it works, and you probably don't even realize it
31:38
until two hours later. Or five days later when you
31:42
recognize because this, because this, because this.
31:45
And then you track down the root cause and you go, uh oh.
31:48
As the victim. Unfortunately, as the as the the
31:50
culprit, you might not be in the company anymore.
31:55
That's the classic scenario is that hey, wait a minute, why did
31:59
this? Oh, I should start looking for a
32:03
job too now. Let's cheer up.
32:06
Let's let's, let's let's talk about something fun.
32:09
Yeah, exactly. Let's talk about the the good
32:11
stuff too. But that is a great point.
32:16
It's not necessarily all doom and gloom.
32:17
It is talking about this so that hopefully it becomes more and
32:21
more commonplace to recognize that it is good practice.
32:25
If you can figure out when changes are being made upstream,
32:30
how can you track those changes so that downstream products are
32:34
not affected by it? Absolutely.
32:35
Yeah. Sometimes all you need is a
32:36
cheeky little test in CI to shift things left and make
32:40
people aware. Yeah, that shift left, Yeah,
32:43
that's a, that's such a good point.
32:44
And the even the ideas on like it's not only traditional ML
32:51
that is affected by this, because if you have this data
32:54
and you're creating Gen. AI products from it, you can
32:59
still get into the same scenario.
33:00
And I think that's why the idea of the ML OPS being more
33:06
important now than ever and hopefully bringing those
33:10
barriers down is part of what we're talking about today.
33:17
So this is, this is fun. Actually, you mentioned to me
33:22
before we hit record that we're almost living in this post agile
33:26
post dev OPS. Why do you think that?
33:30
Oh, that's oh, I feel quite strongly about it because in a
33:36
way, as I grew up in my career, it was all about agile good, dev
33:41
OPS good. And these buzzwords and we
33:43
almost religiously cult like followed it.
33:46
So like, but this is agile, but this is dev OPS, right?
33:49
And you know, maybe sometimes it worked out, maybe you didn't.
33:55
Maybe sometimes you go a hit in the face with a capital agile
33:58
and you had to go work out story points.
34:02
DevOps, not so much. But I think, I think with DevOps
34:04
it's, it's a little bit more sad than painful because you kind of
34:08
go, oh, DevOps is just terraform and AWS.
34:12
Cool. But what I've experienced is
34:18
that people kind of deny people. People see agile as only ever
34:24
existed as capital A. They've never experienced a good
34:27
agile with a small A and there were some blog posts about like
34:32
agile dead yo, you know, nice and nice and click baity.
34:36
But in my opinion, the spirit still lives on.
34:39
Like I actually worked with one quite a senior engineer one day
34:41
who who was very much anti buzzword across the board.
34:46
But when I talked to him about AJ, he just said, like, you
34:48
know, I don't get this agile thing.
34:49
It just means go do the thing, right Literally describe this,
34:52
go do this thing and I'm like, you know what like that's that's
34:55
literally it, right like don't mess around with this.
34:58
Definitely don't waterfall for sure right, but go do the thing
35:03
just needs a little bit of feedback on the way back and
35:05
there you go. That's all Agile is right, Just
35:08
your very frequent feedback loops, right.
35:12
But when I say post agile, I see, I see people saying that
35:18
first and foremost agile is dead, but also saying we don't
35:21
need any meetings, right? Just delete all meetings and
35:25
like have one sync once a week and say how are things?
35:28
They're not delivered yet get them delivered, right?
35:31
That's you, that's your meetings.
35:33
I'm blocked on XYZ. Yeah, exactly.
35:36
So in a way, it's kind of like this rejection of all these
35:43
ceremonies, like no stand ups, no Sprint plannings, like Sprint
35:50
plannings. I have very little defense
35:52
towards that, right? But sometimes it's like no to
35:56
Jira and I look at them and go, yeah, you know, I can't say I
35:58
love Jira. Ain't nobody, ain't nobody here
36:01
going to say stand up to say I love Jira.
36:03
I'm sorry Atlassian if you're watching this, but Jira, they
36:06
know. Just a little bit of a sidetrack
36:09
on that. Once I'm one of the ML engineers
36:12
that works at Atlassian, came into the community and
36:14
introduced themselves and said yeah I'm working on JIRA and
36:17
then put in parentheses. Sorry everyone so they know.
36:23
Fair enough, but when I look at like my my recent team begged to
36:28
get JIRA because we had nothing, right?
36:33
How do you track work? How do you represent your
36:37
process discipline without some sort of can banish like tracker,
36:43
for example, right? You can do it in the whiteboard
36:45
if you want. You can literally type it in
36:47
Slack every day. Every day I did this, this is in
36:50
progress, this is to do and this is completed, right?
36:53
But then why not just use Jiro, right?
36:55
Like it's up to you how you configure it.
36:58
But it's almost kind of like, you know, this big reset, like,
37:00
oh, no, no, no, no, no, right. We're going to we're going to go
37:03
super fast because we're just not going to do all those things
37:06
that people did, right? So I think it's kind of like
37:09
this revival, but the heart of it, right?
37:12
The heart of agile still lives on because it's not like like
37:17
you don't waterfall things, right, Which is like the main
37:20
other alternative. So like if you don't waterfall
37:22
things, that just means like you do small things quickly.
37:25
And it's just the size of these things that varies, right?
37:28
But the end of the day, like I think this post agileness is
37:31
mostly about rejecting the capital agile.
37:35
It's Capital A agile. You think it's because there's
37:39
too much friction that comes with the capital a agile?
37:42
Absolutely. So first of all was yes, but
37:44
it's, it's the survivors of this friction that kind of, it's
37:49
almost like the, the agile apocalypse has the capital, a
37:52
agile apocalypse has happened. And now we see all the mammals
37:55
that survived the, the, the, the meteorite and they're all kind
37:59
of like it's a whole new world now, right?
38:01
We're not doing agile, right? We're not right, even though
38:04
like, you know, life goes on, right?
38:08
But I do find it quite interesting because a lot of the
38:13
concepts are reinvented, just under different names.
38:16
And I remember, I remember speaking to some senior
38:18
engineers back in my life where they just said like, you know,
38:20
things go in cycles. You know, this was like, you
38:23
know, Docker containers were a thing back in the day.
38:25
They were just called something else.
38:26
Like it's just things keep going.
38:28
It's very similar to DevOps. DevOps obviously is, is a
38:32
different impact because yeah, you know, DevOps equals AWS and
38:37
and Terraform, but DevOps could just literally equals
38:43
reorganization, right? You could just say like, it's
38:46
not quite right. These teams should work like
38:48
this, right? Boom, DevOps, you know, like out
38:51
of nowhere. Or you know, how about we hire a
38:55
platform team and not call it DevOps?
38:56
That's already a step forward, right?
38:59
Like this concept of platform teams, it's just the same thing.
39:01
It's just different names. So it's not like concepts
39:06
disappear, right? You don't go, oh, DevOps is
39:09
dead, so we're not doing DevOps, you know, but you know, feedback
39:13
loops still exist, right? And when once people figure out
39:16
that oh God damn, these feedback loops are good, right?
39:19
There you go, there's your DevOps culture or feedbacking,
39:21
right? You know, so it's, I think it's
39:25
more of a case of people having had enough of buzzwords and new
39:32
people come in and they are not familiar with this whole
39:36
education almost back in the day, like because as I said at
39:39
the start, like, you know, it was almost like, you know, pray
39:42
to the agile gods and pray to the dev OPS and this is the way
39:45
things are and that means good. At least that's how I felt,
39:48
right? And now we have this new
39:49
generation here like, you know, DevOps for those boomers, you
39:52
know, like you know, you know, we have new stuff, right?
39:55
We just get, we have. Vibe Coding.
39:57
Yeah, vibe coding and getting shit done, you know, so it's
40:04
I'm, I feel like I'm getting to that stage of that senior that
40:08
I've spoken to so many years ago about how they said like, you
40:11
know, I've seen all this before and slowly but surely I'm like,
40:15
oh God damn, I'm I'm starting to get the same feeling like I
40:18
don't. Like it's, it's obviously
40:20
slightly different, but I'm kind of like, you know why?
40:22
Like, this was a thing already, you know?
40:26
And like, you know, sometimes it's just a case of learn from
40:30
history, if that makes sense. Well, have I got news for you on
40:37
the hype train man, You want to come into a area where there is
40:42
more hype and buzzwords than any other?
40:45
Area. Oh yes, Oh yes.
40:48
I know just where to point you to, but at the end of the day,
40:53
the idea or the concept of how do we create really fast
41:00
feedback loops? Seems like it is a huge piece or
41:05
thread that you're pulling on right now.
41:07
And you're saying, hey, look, I don't care if you are a full
41:10
stack dev, if you're a data scientist, if you are a data
41:14
engineer, how can you create as small of iteration loops or as
41:20
fast feedback loops as possible to get that in place so that you
41:27
can know you're going in the right direction?
41:30
Whether we call it DevOps, whether we call it agile,
41:32
whether we call it ML OPS, it doesn't really matter.
41:35
We can call it LLM OPS if you want.
41:37
That's another one of those buzzy words.
41:40
But at the end of the day, what you, what you really want to do
41:43
is try. And I really like how you said
41:45
it. And so I'll iterate, reiterate
41:48
it. It's breaking down those
41:51
communication barriers and recognizing how to get folks
41:58
talking when they need to be talking in the right ways.
42:03
Yeah, and having like important conversations earlier.
42:06
Yeah. And whether you do that with
42:07
JIRA or you do it with click up or you do it in Slack, it
42:10
doesn't really matter. It's whatever works for you and
42:15
hopefully you and your team and your company have found ways to
42:20
lower friction. I know there is probably an AI
42:26
project at every company internally that is trying to
42:31
figure out how can we use AI to get rid of JIRA.
42:34
And I can almost guarantee you that every company over 500
42:38
people has an internal team working on that.
42:41
And I can just about guarantee that nobody's been successful so
42:46
far. Am I quitting yourself into a
42:48
whole new JIRA? Yeah, you and you would think
42:51
like you, you kind of imagine oh, it can't be that hard,
42:57
right? It's got to be, can we just get
42:59
AAI note taker to hang out on all of our stand ups and can we
43:05
get an AI whatever? Can we like install a Slack bot
43:10
so that it's Privy to all of our Slack conversations?
43:14
And then can we have it read Jira or click up or whatever And
43:19
hopefully it can just start doing what it needs to do.
43:24
That is always, you think it's going to be that easy?
43:27
And then you talk to people who have tried it and it's like,
43:30
yeah, there's, there's a few problems with that.
43:32
Approach. There's always The new plan
43:37
survives the first contact with enemy or getting punched in the
43:41
face. The enemy always has a say.
43:44
Yeah. And but this world that you
43:47
described prior to this problem is that like it's almost like,
43:50
you know, if we could just use AI, we could all live like the
43:54
the people on the Wally ship just sitting there, you know,
43:58
barely like, oh, we'll just use AI to not even say our stand
44:01
ups, right? Or just reach into the brain,
44:04
right? But yeah, sometimes, sometimes
44:08
you just have to do the work and that's what makes you stronger,
44:12
right? Like it's over reliance on the
44:15
vibe coding again, When does the vibe coding end?
44:18
I think that's that's a cool title to the to the talk.
44:21
When should you stop vibe coding?
44:22
Yeah. Well, it it's funny because
44:26
there's the product folks who are probably thinking it can be
44:31
done. We can get rid of JIRA, we can
44:33
get rid of this friction if we have a better approach and we
44:38
try and think AI 1st instead of bringing AI into Jira, How can
44:44
we make it so that our agile, for lack of a better word,
44:49
approach is now going to be AI 1st?
44:53
And I was talking with my friend Flores about this on one of the
44:58
podcasts that we did a month or two ago, and he was saying,
45:01
because they tried the whole, hey, let's bring an LLM to Jira,
45:06
we've got this Jira agent. It's going to not only
45:10
understand the context and everything and understand where
45:14
we are with projects, but once certain things get updated and
45:18
checked in and the PRS get merged, they're going to just
45:22
update the Kanban board itself. What they found is that the LMS
45:28
do not understand the context on these projects because when
45:34
you're a human writing for another human, you write as the
45:40
OR the least amount of information as possible so that
45:45
the other human can understand because you're on the same
45:48
wavelength. You've been working on this
45:49
project maybe for the last two weeks or two months, who knows?
45:52
And so you don't need to go and explain each little ticket and
45:55
what you've done and how you've updated it.
45:58
That makes it very hard for the AI agent to understand what the
46:01
hell is going on. And so I asked him.
46:03
I said, you know, well, what would you do differently?
46:05
Would you try and create some kind of knowledge graph and
46:08
slack so it can get all of these more tapes?
46:13
Yeah, of course man. Like why?
46:17
This is a problem that for sure can just be solved.
46:20
By how else would you get paid? Yeah, You know what, this
46:24
project needs more AI. That's what we need on this
46:27
project. So if you throw them a few more
46:31
LLM calls at it, it's going to work, right.
46:32
And he said, I think about this a lot.
46:35
And what I would do is I would try to probably have each
46:39
engineer be interviewed by LLM note taker and they get asked
46:48
all the questions for all the context on the project so that
46:52
it can now be in the loop. And he was saying this is
46:56
something that we think about a lot is that are we trying to jam
47:02
AI into an existing product or are we trying to reinvent the
47:06
workflow with AI because it is now fundamentally changed?
47:15
11 in this in this AI hype. Obviously everybody started
47:18
doing it. I was just like can't can't even
47:21
tie your shoe laces without AI now can you?
47:24
And to be honest, again, it's genuinely very cool, right?
47:29
And even I have some like, oh, can I like?
47:32
So just a little hobby aside, I'm very, very deep into
47:35
Warhammer and I'm like, can I just AI some Warhammer, You
47:38
know, almost doesn't matter what it is, but like, it would be
47:41
cool to element this and do that.
47:43
But at the end of the day, just kind of sit down and go right
47:48
now. I'm just going to do the human
47:49
thing. Like, you know, just just just
47:53
buy scripts, man. Yeah, just buy scripts.
47:56
What did I just see? Some someone wrote a blog post
48:00
that I was reading this morning and he was talking about how
48:03
more LLM cause or more work doesn't mean better work.
48:09
And just because you are creating the illusion of doing
48:15
more by prompting something and then getting output.
48:18
If the output is mediocre and at the end of the day you have to
48:21
do a lot of work on that output or scrap it all after giving up
48:29
and saying I don't think it's going to get there, then that is
48:33
not progress. That sounds like exactly like my
48:36
project last night. I was just talking about writing
48:38
this parser, trying to vibe code as far as I can.
48:42
I'm just realizing I have to stop, you know, and just do it
48:45
the proper way. Yeah.
48:49
And so it's, it's funny, just going back to the idea of, hey,
48:54
can we use AI to help us breakdown friction, help us
49:01
breakdown these communication gaps that there may be or help
49:09
us create faster iteration loops.
49:13
That's an interesting one to think about.
49:16
But the the thing that it comes back to in my eyes is like you
49:23
can't substitute doing the work. That's true in a way you can.
49:30
I was just thinking, is it just the case that our brains are so
49:35
much more multimodal than any LLM could possibly be?
49:39
That like there's no LLM ING like as you say, like, you know,
49:42
how do we use AI to solve our organizational problems?
49:45
Like, well, do you need to have like a little or pat on your
49:49
shoulder that kind of watches your everyday work life and kind
49:52
of goes, oh, there's a problem there.
49:54
I think, you know, a little Clippy that suggests have you
49:56
considered reorganization and bringing these teams together?
50:01
But first of all, how the hell are you going to train it?
50:06
Like how do you find the data to actually tell her that this is
50:08
this is what should be done. On the other hand is yeah, the
50:13
multi modality is going to break.
50:14
But as you said yourself, sometimes you just have to do
50:17
the work yourself as a human. Yeah, how are you going to give
50:20
it that reward function? How are you going to create
50:22
evaluation sets? School, right, So I'll I'll see
50:27
you tomorrow. We're going to make a new start
50:29
up. Surely going to get a lot of
50:31
funding for this idea. I think we could go pretty far
50:34
with this, even though it's complete vaporware and it will
50:38
never work, but it sounds like a really good idea.
50:42
Oh yeah. It would sell to at least a few
50:46
companies and you would have that overarching LLM that can
50:52
see into all of the different channels and see when folks are
50:55
getting stuck and then suggest you should talk to John from the
50:59
platform team. Might work, we never.
51:06
Know we're never going to know until we've I've.
51:08
Got it. Yeah, that's true.
51:09
Yeah. We got to just get that MVP out
51:13
real quick. So before we go since I, I think
51:17
I would be doing a disservice to not talk about like problems
51:22
that you've seen in in going to production with ML and AI and
51:27
maybe you have some confessions. We started doing these
51:30
confessional stories in the ML news, ML OPS newsletter that
51:34
goes out and it's hilarious what people are writing in with.
51:38
They are talking about how one time you know that they took
51:43
down the Etsy homepage because the recommender system they just
51:47
kind of like poured over some code from one of the smaller sub
51:50
pages and it exhausted the CP us and all this low level AP is
51:57
were just getting bombarded. I wish I had something something
52:00
as dramatic like my the the ones I can think of, the especially
52:06
ML OPS related were only as bad as oops, the machine's gone, let
52:11
me reboot it. So please use a new IP address
52:14
like it's unfortunately, it's very, very mundane in terms of
52:19
confessions. Obviously very painful for
52:21
researchers, right? But not as not as epic as
52:28
Impacts. Yeah, another one that I heard
52:31
was from Flavio talking about how for 18 days straight, their
52:36
recommender system was recommending the same item to
52:39
every single person that went on to the web page.
52:43
And they did not catch it until 18 days later and they realized,
52:48
Oh yeah, we may have lost out on a couple $100,000 worth of
52:53
products sold. Actually, I do have a chemistry
52:55
1. So in pandas world and all that
53:03
tabular goodness, very favorable to these things.
53:06
Call them lovely. We call them lovely.
53:08
So you've got your labels and you got your values right.
53:12
And when you jump in across, you know, umpteen micro services
53:17
here and there, it's, you know, you better hope that those
53:21
labels and values in the right order.
53:24
And in chemistry, it's beautiful because you you have no idea if
53:28
this is good or not, right? You're like this compound.
53:30
Like I have no idea what this compound is and this number for
53:33
this property. Yeah.
53:35
Like, you know, how acidic is this random company?
53:37
Never. You're not even a chemist, you
53:39
know, you barely know. Oh, I said that to you probably,
53:42
you know, something like lemons, right.
53:44
So you kind of go this number kind of maybe OK.
53:49
But like, you know, you never know.
53:50
So imagine like, you know, you're going to say like a
53:53
million of them, right? And then you go, please, please,
53:56
please micro server, micro service hops do not change the
54:01
order of these because if you do, nobody, nobody will ever
54:06
know. So of course, of course there
54:08
were some instances where like, you know, through these umpteen
54:12
hops, through all these micro services, pop comes out the
54:14
answer for like all of these compounds for all of these
54:17
properties. And you know the numbers, we
54:21
don't know if they're in the right order or not.
54:23
And at some point the chemist kind of like something's not
54:26
right because they, they obviously know if this should be
54:28
acidic or not. But the way for them to prove it
54:32
is that they'd have to like reverse engineer the AI to kind
54:36
of say, like, you know, this is what the predictor should
54:38
actually be. And they go like, this doesn't
54:39
align. And then one of the software
54:41
just kind of goes, oh, we've, we've, we've sorted it by
54:46
accident when we should have been sent.
54:49
Oh no. Yep, so like that's definitely
54:53
pretty painful, as you can probably imagine in terms of my
54:58
current place, it's all this video generation and the AI
55:04
problems themselves. Oh man, they're it's almost kind
55:07
of like you hope that something goes wrong because their results
55:10
are absolutely hilarious. Like literally today I've seen
55:13
the wrong skeleton being applied for a render and somebody just
55:17
got Chad Chin and it's almost like, you know, why don't we
55:20
just ship that as a product? Just these random creative
55:25
hallucinations. Yeah, exactly.
55:27
Yeah. That's the premium community
55:31
that you can sell. They get access to all of the
55:35
the ones that you don't use. Yeah.
55:38
I think one last message I do have is that we talk about
55:42
envelopes and organizational practices, but software
55:48
engineering best practices are just as applicable.
55:51
So imagine pre envelopes almost kind of like pre DevOps things,
55:55
but like continuous integration, continuous delivery.
56:00
These things aren't just Jenkins, right?
56:02
So a lot of the times people go, do you do CICD?
56:05
They go, yes, we have GitHub actions.
56:10
So I'm a big fan of kind of returning to thinking about what
56:14
continuous delivery actually is. And that is so crucial in terms
56:19
of implementing these feedback loops.
56:21
This is all about the shifting left.
56:23
This is all about thinking about when, when is the sweet spot for
56:27
this automation, right? Like obviously not straight away
56:29
because you have to prove things, but the same time if you
56:31
leave it for too long, you'll grow way past that moment.
56:35
Now this might be a whole nother podcast, so I don't want to
56:37
delete things too much, but I feel like that's almost kind of
56:41
like the foundation of all of this, right?
56:43
Like, you know, it doesn't matter if it's MO Lobs or DevOps
56:45
at this point or or whatever, but like these things have their
56:49
place and obviously like, you know, doing them too early could
56:53
be very damaging just as much as doing them till it could be very
56:57
damaged. Have you seen signs that it may
57:02
be too late or it may be too early?
57:05
Well, it's, it's never too late as they say, but I've seen signs
57:09
of of the luck. So in a way, if you have some
57:19
code that isn't tested in the production like environment for
57:22
too long, it might not even be integratable.
57:25
It might even just basically say like, we can't even ship this
57:28
because the effort of actually productionizing it is way too
57:31
high. And that's how you know, OK, we
57:36
need to do something here. This is.
57:38
And this is where shifting left is such a key activity here.
57:42
And can you explain what shift left means for you?
57:47
Well, first and foremost, what does the shifting left mean to
57:49
begin with, what is left and what is right?
57:51
So imagine right is production, left is your head and then hands
57:57
and then keyboard and then commits and then pushes.
58:00
So shifting left, hopefully you can visualize it by some sort of
58:05
thing initiative check test that is moved closer to your head.
58:12
You know, the most immediate best shift left is like, you
58:14
know, you talk with someone and you say, let me rubber ducky an
58:17
idea with you. And that gives you a way to say,
58:21
wait a minute, this was a bad idea all along or this is a good
58:24
idea all along, right? Next thing is you write some
58:28
code and you have type checking. Type checking is one of the next
58:34
shift left things that can happen.
58:35
You know, bad type. Please don't do this.
58:36
Please do that, right? And I think usually what happens
58:41
is that the further right you go with, the more and more value
58:45
derived from these checks, for example, right?
58:47
So like at the end of the day, you go production, go, where's
58:51
my money? And either people start going,
58:54
yes, this is a great product, or you go, oops, oops, it didn't
58:59
work out right. So this is like the rightest,
59:02
the most right check you can possibly have, right?
59:05
So the, the idea is kind of like shift as many of these checks as
59:10
to the left as possible. But obviously like some of them,
59:13
you have to pay in cost of times like the best, the most, most
59:16
practical examples is integration tests, right?
59:18
There's this nice seesaw of how much do you test versus how much
59:22
time it takes. You obviously don't want to
59:24
implement the most end to end test possible, which takes three
59:26
hours. And then every time you want to
59:28
push the smallest change, it's a 3 hour CI job and you kind of
59:31
go, well, at least I know it works, right.
59:35
So you have to be very careful in terms of balance of value
59:38
versus versus time. And you can extrapolate that
59:44
it's not just integration tests, it's either conversations or
59:48
it's anything further down the right.
59:52
Let me try and think, sorry, I left my head.
59:54
But like or, or for example, some sort of scientific check,
1:00:00
you know, you can say like, you know, this model produces some
1:00:03
something that's not good. You can't quite always check the
1:00:05
NCI for example. It's interesting that you talk
1:00:09
about that because I've seen it relayed as the closer you get to
1:00:16
production, the more expensive it is to catch a problem.
1:00:19
Yes. And the most expensive is when
1:00:24
it's in production. And as you start going left, it
1:00:28
gets less and less expensive. If you catch that problem, if
1:00:32
you catch it right at its source, that's great.
1:00:34
You can quickly iterate and hopefully fix it.
1:00:37
If you catch it downstream, then it's going to be a little bit
1:00:41
more expensive. Sometimes things are not as easy
1:00:43
to catch as well. So, so there's that dimension as
1:00:46
well is that, of course, if you could catch the fact that
1:00:49
nobody's going to buy your product and you've just spent
1:00:52
like 10 billion in AWS bills, if you can write a unit test for
1:00:55
that, that would be some unit test.
1:00:57
That is the product that we need to create.
1:01:00
We need to vibe code that product in new existence because
1:01:05
I can tell you what if we could unit test whether or not someone
1:01:09
is going to buy your product and sell that as our magic snake
1:01:13
oil. Talk about a billion dollar
1:01:15
company. Oh yes, easy.
1:01:19
Yeah. And and it's cool too that
1:01:22
you're talking about shifting left because that idea I've seen
1:01:25
a gain more popularity. I'm actually going to be hosting
1:01:28
the shift left data conference. And so it's bringing the idea
1:01:34
and the fundamentals of shifting left, but to your data pipelines
1:01:40
and your data CICD. And whether or not your data is
1:01:46
it has analytics built on top of it or it has ML built on top of
1:01:50
it. That same idea of can we catch
1:01:52
problems at their source, but also how do we balance one,
1:01:57
which is I don't want to have to have three hours CICD check
1:02:02
every time I make the smallest change verse.
1:02:06
I don't want to just not have anything and be Willy nilly and
1:02:09
kind of pray every time I put something out there.
1:02:13
It depends. Yeah, well, what in your eyes
1:02:20
would be different when it comes to shifting left for code versus
1:02:28
shifting left for data? Oh boy, oh boy.
1:02:31
Once again, I, my heart goes out to my data friends.
1:02:36
So recently enough, I had to do quite a lot in terms of setting
1:02:41
up an entire data platform and that just gave me so much
1:02:49
respect for that area. The best thing you can say is
1:02:54
that it's a lot harder, right? A lot harder because code is
1:02:59
stateless. First and foremost, data is
1:03:02
state. And that in itself is already
1:03:06
super hard. Like it's I, I kind of struggle
1:03:09
to explain it from first principles, but please trust me,
1:03:13
data harder than code. So because of that, I think like
1:03:19
how, how do you test data? Well, you can have data
1:03:21
validation and integrity tests and all that kind of stuff, but
1:03:26
implementing that and running it and operating that like, you
1:03:29
know, that is so much harder than for example, testing if
1:03:34
you're my inference works. That's great, right?
1:03:37
Yeah. And also like the the amount of
1:03:41
the sheer amount of possible knock on effects.
1:03:42
Because if you're going to dive into data like as a holistic
1:03:48
view to kind of like take it from bird's eye view, right.
1:03:53
If you're going to start thinking about, you know, here's
1:03:55
my source. This is the entry, right?
1:03:58
And this once again comes down to how much value do you want to
1:04:00
derive from your test is like how far down the stream of all
1:04:03
the possible ways your data can go?
1:04:05
Are you going to test things and the nature of your data?
1:04:09
Like, you know, a lot of the times we speak about data, we
1:04:12
implicitly assume tabular, right?
1:04:17
My current, in my current place, I don't have such luxury.
1:04:21
There is tabular metadata for sure, but boy, oh boy, the
1:04:24
majority of the actual data is media.
1:04:26
So testing that is even harder because you could imagine like
1:04:32
working with media is not the same as, you know, having
1:04:35
something that you can fit a lot of in in the container.
1:04:39
So I'm sorry to give you like a very hand WAVY answer, but like
1:04:48
shifting left in terms of data just seems so much more harder
1:04:50
than than in code. In code, it's, I don't want to
1:04:53
say easy, I forgot, absolutely not.
1:04:56
But it's a lot more easier to visualize because you can kind
1:05:01
of say this is right or this is not right.
1:05:02
You know, your unit test passed because 2 + 2 does not equal 5.
1:05:08
And you kind of say red unit tests, right?
1:05:11
And you can kind of extrapolate an extrapolate, extrapolate
1:05:13
right with data. It's just so many different
1:05:19
cauliflower level fractal problems that it can expand to
1:05:23
that I'm finding it hard to visualize.
1:05:26
Sorry. Maybe I just suck at data.
1:05:28
Maybe that's. No.
1:05:29
It's also true that with code it's a much more mature field,
1:05:33
and these ideas have been iterated on for.
1:05:39
I've got a guest. Yeah, what a 2D for anybody
1:05:43
that's not watching. We have a dog that just jumped
1:05:46
on screen. Really good looking dog.
1:05:50
So Data is 100%. I forgot where I was going with
1:05:58
this, but yeah, one thing is for sure.
1:06:02
Yeah, the dog I'm I'm looking at the doc socho.
1:06:05
So one thing is for sure though, Data has had the idea of
1:06:11
shifting that for much longer. So it's almost like this mature
1:06:16
idea, or it's much more mature than the idea of shifting left.
1:06:21
Did I say data? I meant.
1:06:22
Code. So one thing is for sure, code
1:06:29
shifting left is a much more mature idea because it's been
1:06:33
around for so long and data shifting left is a newer idea,
1:06:38
and so it almost is. It needs those iterations and it
1:06:41
needs diverse minds to come at it and try and attack the
1:06:45
problem from different ways so that it also can become a bit
1:06:49
more mature. And in my own experience, if I
1:06:52
say that like when you just tack, try and tackle that from a
1:06:55
code perspective, like that's when things just stop lighting
1:07:01
up quite as neatly because it's, it's almost like a different
1:07:03
mindset you have to have as well.
1:07:06
That's good. Yeah, that's a very interesting
1:07:09
one.