Latest / Agentic Conversations (formally mlops.community) / The Ops in MLOps - Process and People // Shalabh Chaudri // MLOps Podcast #146

Transcript

The Ops in MLOps - Process and People // Shalabh Chaudri // MLOps Podcast #146 · Agentic Conversations (formally mlops.community)

0:00
Hi, my name is Caleb. I am head of customer success at
0:03
Union aii and I drink coffee. Maybe too much.
0:09
Six seven cups a day all before noon, but it is getting out of
0:15
the house. It's with Keurig craft just to
0:20
kind of, but if I have time to enjoy it'll be, it'll be a
0:23
French press all grind myself and kind of make it myself.
0:26
But again, I got any some time for that and then every now, Oh,
0:30
that I'll indulge you to Starbucks, you know, flavored
0:33
thing. Maybe a couple times a month.
0:36
Yeah, coffee out of here. He just thought that we're live.
0:42
Welcome everybody. I am Demetrios and this is the
0:46
ml Ops Community podcast today, I am joined by none other than
0:50
Abby, how you doing today? I be Wonderful.
0:54
There's so much excitement that is going on in Tamil opsec
0:57
entity so it's fascinating. You want to jump right into the
1:02
review that I promised you I would read, right?
1:06
I know you want to. But first let's talk a little
1:08
bit about Sheila because we just got done talking with him and
1:13
the conversation was so cool. It was a whole different angle
1:16
than we usually take. What were some key takeaways
1:19
from you? I think for me what was quite
1:22
interesting is I've Heard Em adopts being described from the
1:27
perspective of dear scientist, from the perspective of the
1:29
ancient there, but I've never really understood it from like a
1:33
customer success perspective where he's working right now and
1:37
feel like he brought such a unique way of looking at the
1:41
problems of working to some of the challenges that I feel was
1:46
like almost precious. I loved his idea about setting
1:52
expectations, but was very much like you don't.
1:59
You don't want to say that you can do something, if you are a
2:04
vendor in this space, and then six months later, or a year
2:09
later, you have to go back to the drawing board and
2:13
everybody's frustrated because you haven't been able to get
2:15
anything into production, you thought that this team was going
2:20
to give you a value-add and its really taken up a lot of time
2:22
and you haven't seen that value. And so then it's going to leave
2:26
a bad taste and the team's mouth and He was very upfront about
2:31
figuring out. What are the winds that we need
2:34
and how can we assure ourselves of these winds?
2:37
And so I loved that angle. I really, I mean, of course,
2:42
he's based in Seattle. I think.
2:43
So, he's very customer focused, because I think everybody in
2:47
Seattle has been indoctrinated into the Amazon way of doing
2:51
things. And we want to give a huge
2:54
shout-out to Union speaking of which, because they were kind
2:58
enough to Answer the community. They've been an incredible
3:01
sponsor this whole past year and I just talked to him, they're
3:05
going to be a sponsor again this next year and that is so cool.
3:10
I want to thank them if anybody is looking at flight.
3:14
If you were looking at trying to do like ml workflows at scale,
3:19
check out flight, you can spin it up really easily.
3:22
We used it in our ml Ops engineering labs hackathon and
3:27
you can just get your hands on it.
3:29
That's the open source version and then Union is the paid
3:34
version, the Enterprise version we could say.
3:36
And shout out to them too, because they just crossed 3,000
3:39
stars on GitHub. Congratulations to the team now
3:42
let's get into what you have. Definitely been waiting for
3:46
Abby. I know you want to hear the
3:48
reviews for, all right so we've got one.
3:52
I'm just going to read you one and I'm going to put the call
3:56
out for everyone. That is listening in case you
4:00
would like us to read your review on this podcast.
4:05
Leave us a review on a podcast and net doesn't matter.
4:09
If it is bad or good funny, or actually the funnier the better.
4:14
So I want to start us off with one from mr.
4:17
Nomadic coder, shout out to you and huge.
4:20
Thanks for the five star review. He says very mucho.
4:25
Mel Ops there is so much ml of stuff in this podcast that it
4:29
fascinates me as to how I did not know about ml.
4:33
Ops prior to 2021 for those who do not know.
4:36
Ml Ops is basically Cloud, native ml engineering, AKA let
4:41
us reinvent the ml wheel. Every year design choices aren't
4:47
easy in the mo Ops World, and this podcast gives you the right
4:52
context to hone in your decision-making.
4:55
Source. Thank you, mr.
4:58
Nomadic coder. And let's get into this
5:00
conversation with shelob. Huge!
5:02
Shout-out to Union. Check them out.
5:03
Will leave a link in the description.
5:11
She loved, let's set the scene you and I we met back in the day
5:15
when you were working for algorithm.
5:16
Yeah, I know that wasn't your first job and you were shaped a
5:21
little bit. When you were working there, I
5:23
know, Diego has a very much customer Centric view of things,
5:28
but can you give us a bit of backstory on your journey?
5:32
Yeah, yeah, that's right. We did meet when I was at all
5:35
grid me and that was my first machine learning that role and a
5:40
lobster at that to and kind of my journey to, you know, there's
5:45
a kind of a pre algorithm Mia and post algorithm Ian type of
5:49
Journey. So I came into algorithm
5:51
uniform, a cloud, migration background, I was working for a
5:54
company called Sky. Tap, we were doing Cloud,
5:58
migrations was migrating books from on-premise to the cloud and
6:00
really working with logic. Enterprise's.
6:03
It had been following algorithm for a while just AI am L.
6:06
And I'll also is everywhere and it was just an area of Interest
6:09
mostly because machine learning just as you guys know by its
6:12
nature is just so complex. It's so functionally dependent
6:17
on various roles adoption and riding Dahlia.
6:21
Machine learning is definitely a challenge.
6:24
So those are all the the challenges that attracted me
6:27
towards algorithm and I decided to make the move over there.
6:30
And then since then you know I'm now a union doing a similar role
6:34
customer-facing. Customer success driving value
6:36
for Enterprises but focusing more on orchestration versus
6:41
what I did over at Mia, okay? So there's so much that I want
6:45
to talk about. I know one thing that we
6:47
mentioned is that we're going to get into the people and process
6:52
of in the labs and as you said, that is a strong focus of yours.
6:58
I think about one thing that has stood out since we chatted and
7:04
not only that first time, but we chatted about a week ago, and we
7:06
were really talking about the mess that things are these days.
7:14
And some will say, we have mihael who joins us quite
7:18
frequently on this podcast and he says, that's to be expected
7:22
when you're in new category. How do you feel like how does a
7:26
company? Find success with the current
7:32
state of affairs that we are in right now.
7:34
Yeah yeah. And and I completely subscribe
7:38
to the, this mass is expected like this is a this is a sign of
7:43
things or working for a new market, but for companies to be
7:47
successful, you know I look at it from two perspectives.
7:50
Honestly, my background is more from a, you know, a software
7:54
vendor provider for machine learning.
7:57
Solutions and less so from within a company but I've worked
8:00
with a lot of companies in getting value out of the mouths
8:03
of that perspective as well but I think some of the challenges
8:08
that I think come from this is it when a space is this new, it
8:14
is an opportunity its category creation, right?
8:16
As is up to the different vendors and the folks that are
8:18
there, practicing machine learning and building
8:21
applications and solutions based on machine learning or just call
8:24
it AI enabled Solutions. The The complication that
8:28
happens is folks, get to Define what ml Ops means to them or
8:32
what various parts of the machine.
8:35
That's so true. That is so true.
8:38
So different vendors will come in with a different perspective.
8:40
And you know, software vendors are considered to be at least
8:46
they should be knowledgeable about the whole Space.
8:48
So I think, I think what I really feel for is the buyers
8:52
like and again, I'm looking at as a software vendor and say,
8:56
there's a, you know, Ha VP level for AI Solutions, whatever that
9:00
title might be especially if they're not, you know, deeply
9:04
experienced on the space. There's just hearing all these
9:08
different sometimes conflicting messaging on what an ellipse is
9:13
and how it helps customers and and and I think it just
9:16
complicates the landscape. But again it is all intended as
9:19
part of a newly emerging category.
9:22
It's still relatively new. So that's the one for me that
9:26
surfaces On top of the list. So when you start engaging with
9:30
large Enterprises, or what I call folks, that are doing mli
9:35
reasonable scale. And again, that's not my term.
9:38
It's a it's a term. I don't know who I attribute
9:41
that to but ml at massive scale. And I'm Ella reasonable scale,
9:44
massive scale being The Meadows and the Twitter's of the world
9:46
have that stuff figured out its, the more that are not Cloud
9:51
native or a, I needed companies that are building those
9:53
Solutions. So for those types of companies
9:56
that become specially, Aging when they're hearing, is
9:58
conflicting messaging. And I think it's, it is vital
10:03
for folks, to go in with a holistic, understanding, and not
10:06
conflicting messaging. Very interesting in this.
10:11
So if inspires me to ask two questions, the first question is
10:15
more so around, you know, whenever most of most of our
10:18
machine learning models are sort of tied up to subside in
10:21
accuracy measures like Precision recall.
10:25
But most of those do not serve in any sort of way that we
10:29
present for the company is looking for which would be
10:33
something around the key place. So how do you sort of tie the
10:36
two measures together or how do Think about time these two
10:40
measures together in this second is you have multiple
10:43
stakeholders. We are priorities are changing
10:46
every single time whenever somebody gives instructions
10:49
ending their specifications that come in and then within 23
10:53
months things change. So how do you work around that
10:56
process? So I think yeah so I think
10:59
reading it's if when were thinking about business value
11:05
and just outcomes from machine learning, The traditional
11:12
technology Centric API such as recall that accuracy and
11:15
everything else. I think I think it's appropriate
11:18
when we're experimenting and kind of trying things out but I
11:21
think in the end there needs to be a very clearly defined Roi
11:27
from from the application or solution or the outcome that is
11:31
being enabled by Machine learning and it even starts with
11:34
because machine learning the right approach was a lot of
11:36
times, it may not be the best in the most.
11:38
Straightforward approach to solving a business problem.
11:41
So understanding the business problem why is that an important
11:46
business problem to even talk about or solve for?
11:48
How do we know it's an important business from problem?
11:50
How do we know what success is about?
11:53
Is this problem and then going a layer down and say is machine
11:57
learning the right way to go about solving this business
11:59
problem, are there simpler ways to do it?
12:02
And if machine learning is indeed the right way to do it.
12:05
You know, how do you know, have you experimented internally?
12:09
Just to just to, at least get an initial hypotheses that this,
12:14
this technique machine learning, gets you to that, solving that
12:17
business problem. And and the ROI is really
12:21
measured. In terms of what the business
12:23
outcome is, is it driving an increased number of Click
12:27
throughs, or is it driving less children or, or whatever?
12:31
The business kpi is, are tying. The time the machine learning
12:35
models outcomes with those business kpis by first doing a
12:39
business case assessment, even if it makes sense to go down
12:42
that path, there is something That I find interesting about
12:47
what you're talking about because it is I don't want to
12:50
say it's common sense but I will say that a lot of people think
12:54
about this but not a lot of people actually picks acute on
12:57
it and so where have you succeeded?
13:01
Where others? Fall short?
13:02
Yeah, so this goes to the other part of the question, in terms
13:07
of multiple stakeholders with different functions.
13:10
So it's making sure that again, putting Maya software Your hat
13:15
on when I am interacting with various stakeholders at a mid to
13:20
large Enterprise, you want to make sure that you're talking to
13:24
various functional roles within the organization.
13:27
So not just, you know, a chief data officer or someone from the
13:32
IT devops team. We also want me talking to the
13:34
product teams, like the the chief product Officer Down
13:38
depending on size of the company, but someone will own is
13:40
the, the the end product or solution.
13:45
I will have benefits from the machine learning models that are
13:47
enabling those business outcomes.
13:49
So, having conversations with those product managers product
13:52
leaders and breaking them into the conversation around.
13:57
How are you thinking about machine learning?
13:58
What is, what are you remember? Laughs needs and, and they might
14:02
not have that answer on their own, but they're able to lease
14:06
the provide the right contacts in terms of how important and
14:10
and what are the business outcomes?
14:12
The company of those product teams are looking for.
14:14
So, I make sure that I'm talking to various stakeholders within
14:18
an organization and not just focusing on that one individual
14:21
that has been asked to stand up an MLs platform.
14:26
Yeah, it really comes back to making sure that you have done
14:31
your due diligence, right and talking to the right people and
14:35
considering how Abby mentioned there are so many people that
14:38
get involved, you want to get opinions from all of these
14:42
different stakeholders. Now, as you are looking at it
14:46
from the, the tooling perspective or a vendor
14:50
perspective, there are a lot of different tools out there and I
14:54
wonder how you look at it. Like, How do you get your
14:58
solutions to work alongside of everything else that's in and
15:03
like the Legacy? Yeah, equipment that teams bring
15:07
to you. Oh my gosh.
15:08
Yeah. So that you know that's that's a
15:11
that's a huge Challenge. And I think something that
15:15
really comes to the fore front working with these types of
15:17
companies. And I think what becomes
15:19
important is you obviously want to have a perspective on the
15:23
capability that your ml Ops tooling and solution enables
15:26
Also, if you're doing observability monitoring your, a
15:29
feature story and it does only matter kind of we're in the ml
15:32
lifecycle you fit and obviously you have a very old fully clear
15:37
understanding of how your solution, kind of, latches into
15:40
the old ml end-to-end tooling and life cycles.
15:44
But at the same time, you have to have a holistic understanding
15:47
end to end from what an effective machine learning
15:52
pipeline is, and you need to be able to provide some subject
15:55
matter expertise. In terms of a reference
15:58
architecture, you know, I'm still waiting for like a
16:01
canonical kind of machine learning pipeline model type of
16:05
to emerge in. But but how does your solution
16:08
fit with other tooling up and downstream in the chain and the
16:13
pipeline? And, and based on the customers
16:16
needs which one of your reference architectures patterns
16:22
emerges, as specially beneficial and helpful for that customer.
16:26
So So we're just going to summarize you wanted to see, you
16:28
got to understand the whole intimate life cycle and not just
16:30
your product. You can just be focused on
16:32
observability or monitoring and say this is, you know, if you do
16:35
this XY and Z will happen because it won't, it had their
16:38
whole ml pipeline in end-to-end, tooling has to be successful.
16:43
So having a knowledge but also providing the customer best
16:45
practices and subject matter expertise.
16:48
So one thing, I mean, if you do understand the whole lifecycle
16:53
it's only obvious sometimes you get biased.
16:56
But certain tools because you see them working much better as
16:59
compared to some sort of company, which has been using a
17:02
certain tool for quite a few years.
17:05
So how do you work in tandem with a client or that has
17:09
already been using some sort of legacy to but not let's say
17:12
there's better open source tool out there for sure.
17:15
Sure. Yeah.
17:17
So I'll kind of a starting step that I take with with all
17:23
customers is just a build a mental model of what this
17:27
customers it and data science maturity levels are so you know,
17:31
from the data size perspective, how big is their data science,
17:33
team are the data and all teams tightly coupled.
17:36
Sometimes they're the saying okay if that's the case, how
17:40
many models are in production? How many are ready to go in
17:42
production? Because based on these this
17:44
criteria, the customer will think about machine learning
17:47
Machining operation. He's very differently, so kind
17:50
of to link to that have. So that's what the design side
17:52
of things. I like an IT and devops.
17:54
Alrighty, and devops tightly coupled with a IML data science.
17:57
Teams within the customer is a kind of a, you know, still a
18:00
throw over the wall type of Paradigm.
18:03
And then how engaged are the business stakeholders executive
18:07
need management Frontline, you know, who's driving this?
18:10
And then last thing I consider is like, what is the, what is
18:13
the mindset? Are they comfortable from a
18:16
cultural perspective? Because think about devops
18:18
Devops was around a cultural manifestation as much as the
18:22
tooling. So this customer, you know,
18:25
understand holistically from a system thinking thinking
18:28
perspective that every every aspect is intertwined and are
18:32
they comfortable with failing? Any machine learning is
18:34
experimentation at the end of the day, are they comfortable?
18:37
Or are they hoping for like a big being success at the end of
18:39
this deployment? Because that likely won't
18:41
happen. So anyway, so I kind of take
18:44
that model and I going to the customer and then Abby to your
18:46
question it, It changes how I look at various customers
18:51
whether they have Legacy tooling, whether they're
18:53
starting from scratch and they have nothing in place.
18:55
But building this kind of mental model by going through these
18:59
questions, helps inform that approach and strategy.
19:03
What that particular customer for interesting because you
19:06
mention, do you have to look at their entire life cycle in their
19:09
maturity level signal that? So one area where I see a lot of
19:15
companies started struggling with is, is basically Thinking
19:18
about cataloging solution and some companies do Surf, use some
19:23
tools others, try to build it in house.
19:26
So how do you think about? Let's say we are working with a
19:30
certain customer in their, they're stuck at that very first
19:34
level of problem which is they don't have any sort of catalog
19:36
in solution and place better copy cataloging in this example
19:40
Wars just in general. I think there is there is a
19:45
definitely a deficiency today. In terms of an unbiased
19:50
perspective, which a lot of times is filled by Consulting
19:53
companies, but like an unbiased perspective that can be tailored
19:57
specifically to the needs of that customer in terms of even
20:01
saying no our solution does not work well for you or suggesting
20:05
other solutions that might be a good a good fit for that
20:09
customer based on their needs. So, you know, there's a there's
20:12
a couple of things I was picking up in your question.
20:13
Here, there's a build versus buy for all in a lot of countries
20:16
have to solve for N. And when it comes to the time of
20:20
sure, I recommend that they, you know, procure XYZ catalog
20:24
solution or go build one. I think fundamentally.
20:27
And this is true, not just for ML software.
20:30
In general, you know, is is the reason you want to build
20:35
something, is if it's your core competency and as a company,
20:39
your business performance relies on building that capability and
20:43
that outcome. So, in most cases in most
20:46
companies, Building a kind of a catalog tooling or solution is
20:51
not a kind of a central need. We're in most cases, my
20:57
recommendation would be to Outsource and procure, those
21:00
types of things provided, it, fits those customer needs, and I
21:04
didn't a Blog, you will come into play when there is that
21:08
level of scale trait in terms of uses and models.
21:12
So, by that question, I'm assuming the customers already
21:14
there, in which case, I kind of talked about weight, you know,
21:18
Before the market cataloging like it.
21:20
How do you know your models are working?
21:22
And and, you know, they're solving the business problem
21:24
need that you have. So not a cut and dry answer.
21:27
That's why every customer is so different.
21:29
It's just, it's just where they're coming from and where
21:32
they're going. A lot of times help Define what
21:37
their solution should be built, pursued by or if they're buying.
21:40
What the specific solution will be.
21:42
The last thing I'll say is again it comes down to it, we did it,
21:46
Albert Mia, we're doing here, you know, Ian where we build
21:48
Partnerships with tooling other MLF stooling providers where,
21:53
where either we have Integrations that are
21:56
predefined, or we have a tightly coupled stack, that we find
22:00
where we're able to go to a customer and say, based on your
22:03
knees or your profile as a small customer or medium customer.
22:07
This type of tooling should work for you based on what we've seen
22:10
in the industry because of X Y and Z is Skyler.
22:16
I lead machine learning it. Health rhythms if you want to
22:20
stay on top of everything happening in ml.
22:22
Ops subscribe to this podcast now.
22:30
So there's something that you are saying here.
22:34
Let me see if I can abstract away.
22:37
What we've been talking about for these last couple minutes,
22:41
which is almost like Everyone should be thinking about if they
22:47
are starting along their Journey or if they are in the middle of
22:51
their Journey, first one is who are the people that I'm
22:55
servicing, who are the people that are the stakeholders?
22:59
And what do they care about? Figuring that out and having a
23:02
clear definition of what your use case is and who it's
23:05
touching, right? Right?
23:07
That is something that you mentioned needs to be very clear
23:10
and then the maturity levels and basically how you You or your
23:15
organization works, and what the teams look like, what the actual
23:22
execution process looks like. Whether that means that you've
23:27
got a team of a bunch of different skill sets, or you've
23:30
got the one team, and they're tightly coupled with another
23:34
team, etc. Etc.
23:35
And figuring out those pieces will get you so far along the
23:42
journey. Yeah, no that's a that's a good
23:46
summary. And again, that's a starting
23:49
board rate. It's, it won't guarantee you
23:52
success with Hema Labs, but it puts you in a much more, a
23:56
higher probability of getting a meaningful outcome out of it
24:01
build or buy solution that you might entail.
24:04
Yeah, that's awesome. So, I know there is the people
24:09
aspect that you wanted to talk about and really like setting
24:12
people You up for success and how for you, it's not around the
24:18
tooling and tooling is just one aspect and this is something
24:23
that we harp on quite a bit in the community but we can never
24:27
harp on it too much because there have been so many VC
24:30
dollars that have gone into tools and hear a lot about the
24:33
tools and not as much about this process and the people and I
24:37
think that is what a lot of people.
24:39
When they talk about, I saw a great like post just recently
24:44
and how it was saying that ml UPS is not ml + Ops.
24:50
It's really devops + ml. Yeah.
24:53
So if you really look deeply at devops devops was so intertwined
24:59
with the people in the process. What are your strong thoughts on
25:03
this? Because I know you've got a few
25:05
Yeah. So there's a devops angle
25:07
honestly, and kind of Hema Labs is maybe an iteration of that or
25:12
if you do a Venn diagram, there's different ways to
25:13
represent it. But I think the ml specially and
25:19
again, this is probably not news to the community, but in terms
25:23
of analysis it was so much more Dynamic, right?
25:25
Then traditional software. So because you have all these
25:28
different Dynamic aspects of machine learning in the first
25:31
place, just by the nature of it. How its iterative experimental
25:34
Oh, but on top of that, there are different stakeholders that
25:38
are coming in, but sometimes conflicting perspectives.
25:42
So, you know, I talked about the product teams that are Building
25:45
Solutions that will utilize machine learning as a technique
25:49
to make that product or solution successful.
25:51
So getting their perspective, understanding what they want to
25:53
get out of that. We talked about that, the other
25:55
aspect kind of comes into the data or ml teams who owns that
26:01
ecosystem, that tooling. Who's driving kind of building
26:06
the models and kind of iterating on those which is the
26:09
organization that has a perspective and stay there.
26:12
So that's another stakeholder group and then it and devops.
26:16
So Similar to kind of what used to happen, parade, cic devops
26:22
tooling microservices days here. We've got that again proverbial
26:27
wall, where where it and devops teams aren't necessarily on the
26:31
same page with what product teams are trying to accomplish
26:34
or what data science, teams are trying to do.
26:37
So they have such different perspectives and what you want
26:41
to make sure is that again I look at as an envelope sender.
26:44
But again you can take the the other view of The inside a
26:48
company. But you want to make sure that
26:50
every one of those functions has a seat at the table.
26:53
And and that they are, they are representing their needs and and
26:59
kind of what they are trying to get out of the solution.
27:02
Let us built or cured, but make sure they have a seat at the
27:04
table and they're defining very clearly kind of the outcomes
27:07
they need. So let me give you an example.
27:09
Like ml Ops ml Ops is you don't need a lobster.
27:12
If you have a handful of models and you know, you're still
27:15
perlina journey and a laughs just by definition.
27:17
About driving or a five-year-old.
27:20
Definition is I want to have confidence in my email system to
27:24
run reliably predictably in a scale.
27:26
So making sure that all the different functional areas.
27:30
That I talked about that, they are not able to provide their
27:35
perspective and their requirements in terms of what is
27:37
real reliability mean. What does predictability mean?
27:40
What is that scale? So reliability generally as an
27:44
SLA or or some kind of business - focused kbi attached to it
27:49
which might confer product teams are different, seems
27:52
predictability, you'll see ICD is like a good indicator,
27:56
predictability. Like how how confidently can you
27:59
execute on your when your machine learning iterations?
28:02
And then at scale, you know, what is, what kind of P close
28:05
are you? Are you expecting?
28:06
And that's what kind of cloud comes it.
28:08
So you've got it that Lobster. Now maybe at the scale angle,
28:11
you've got data, size teams, talking about some of the
28:14
predictability aspects, reliability better.
28:17
Might be coming to a product or these might come from different
28:19
team members, but making sure that you kind of have that on a
28:23
dialogue with these different functional owners on those
28:27
various aspects of machine learning operations initiative.
28:34
Yeah. The communication needs to be so
28:35
clear and it reminds me of a conversation.
28:38
I think that we had with hannes one thing that he was talking
28:44
about was how his machine learning team even consults with
28:50
their UI, ux designers and they have conversations quite
28:55
frequently because they want to make sure that everybody's on
28:59
the same page. And if the designers are
29:02
changing a Tonor, they're changing a call to action or
29:07
miniscule things that we probably would think are
29:10
miniscule. They do have Downstream effects
29:14
and especially when you're looking at the models that
29:16
people are building with those changes in mind, maybe there's
29:20
something different that can happen because of that change.
29:24
And so, they want to be aware of it.
29:26
And then they also want the UI ux designers, to be able.
29:31
To come to the machine learning engineers and say, hey, you know
29:35
what, if we do this, do you think that it would change and
29:39
they can suggest ideas? So that open line of
29:42
communication is killer? Oh 100%.
29:45
Yeah that's a good one. Because again if you're doing
29:49
some kind of real-time inference the the usability of what the
29:53
user sees, when that prediction is being generated like it, the
29:56
ux team has to go to help Define that and how do you, how do you
30:00
make that user experience? Such that the the limitations of
30:05
what the model might be able to do or the machine learning
30:10
pipeline is a hold that has to be either?
30:13
Mast? Or accentuated by the ux so that
30:16
the user has that seamless experience.
30:19
And that's the whole point of analogs ml, so, yeah.
30:22
No, that's a very good point in terms of ux playing a role to,
30:25
dude. So I know you've got some
30:29
incredible stories. I want to hear some of them of
30:32
like, what are some engagements that you've had?
30:35
What are some and you don't have to mention any names unless you
30:38
really want to? But give us some scenarios that
30:41
you've seen and how things were over.
30:44
Come, when there was challenges that you hit, you know, the
30:47
first one that's coming to mind is going to go with this one.
30:49
Might not be the perfect example, but we'll go with the
30:52
one that's top of mind. This is working with a large
30:58
National Services Company, let's leave it at that and it's always
31:02
the financial services companies, that's all.
31:05
He's how it goes. Yeah, yeah.
31:07
Financial Services biotech, any of these guys are like a
31:10
logistics. You know, operational resource
31:13
type of use Cases, they're at the Forefront.
31:15
They're doing a lot of stuff. Yeah, for sure.
31:17
Yeah. So this was more of a
31:20
traditional financial services company in that, you know, there
31:23
were sold her lie there and I'll journey and AI Journey building
31:28
a. I knew able solutions, kind of
31:30
both responding to Market needs and kind of internally.
31:33
What folks were asking for. And what's interesting is, you
31:38
know, the Mandate came down likely from board down to say,
31:42
you know, we need to find a look at AI tables.
31:44
Oceans to cut down on costs or drive more revenue and promo
31:48
products. And what was interesting is, you
31:51
know, we were part of an RFP and that's how some of these large
31:54
companies. Do a lot of the ml Ops or
31:58
enterprise software engagements and the company was embarking on
32:03
a, you know, VI enables strategy while migrating to the cloud at
32:08
the same time. And this there's a A certain,
32:14
this is a good example of what there's this is a step-by-step,
32:18
this is a foundational. Machine learning, is a
32:21
foundational, you build on your success of prior engagement.
32:26
So kind of being comfortable working in the cloud and
32:30
understanding containerization having a sufficient knowledge of
32:33
kind of kubernetes. Because that obviously is a very
32:37
common solution. Is powering lot of solutions,
32:42
other machine learning Solutions, But there are the
32:46
reason I bring that up and be the reason why that was
32:48
interesting is, what could go wrong.
32:50
We were able to write exactly that.
32:53
We were able to say this is not the right time for a solution of
32:58
like ours to be coming in to solve this need that has come
33:04
down come from the board down and and he need to kind of take
33:07
a step by step charity. And these are steps you should
33:09
take. So I don't want to go into too
33:12
much detail because I don't give away.
33:14
The customer was. But but again, the point being
33:17
that, these are very intertwined and step-by-step oriented
33:23
outcomes where different team members have to have a certain
33:26
level of sophistication to be able to build something that
33:29
works for the whole organization from a mobster respected.
33:31
So, dude for me, it sounds crazy.
33:34
That anyone would think that could work like that strategy
33:39
because you can just jump onto the cloud and instantly you're
33:42
getting your feet wet. You're fully submerged in
33:45
machine learning. And feels like, if you asked any
33:49
engineer out there, they would know.
33:51
And they would be able to tell you walk before you can run.
33:55
Yeah. Are the people that are making
33:57
these decisions how and how are these decisions being made?
34:01
How do people get into this level of leadership?
34:06
And then they think, like that this case, it was a, you know,
34:09
with all the fintech solutions out there for this company, it
34:12
was a little bit of a panic. Nick response to say there's all
34:16
these fitdeck players out there that are capturing market share
34:19
because of these amazingly seamless experience.
34:22
As we use those are in terms of opening a bank account or
34:26
getting each other's or whatever the case might be.
34:27
So I think it was a little bit more of a panic response say
34:30
we're falling behind their Roadie market, share.
34:33
And but but the organization wasn't ready for my people in a
34:36
process perspective or even tooling technology.
34:39
Yeah Dad lot of companies strike now especially like Legacy
34:43
company. There's trying to jump on the mo
34:46
Ai and packing yeah because they realized that their newer
34:50
companies are coming in taking their Market smaller startups.
34:53
In fact, yeah, Dad to have the technical expertise that these
34:57
companies do not respect of having a bigger infrastructure
35:01
and bigger teens. How can these companies sort of
35:04
get to this place with just what needs to be done to be able to
35:09
innovate quickly while also not over-promising?
35:13
Yeah. Yeah, no, that's a like a
35:16
million-dollar question. So, the funny thing is, you
35:19
know, when I, when I was outside, AI am L is like, oh my
35:22
gosh, look at least great Solutions being built that we
35:25
see in the media day and night. But when you go to these
35:29
Enterprise companies, or Legacy companies or even some like
35:33
technologically for companies, it is it was surprising to me to
35:37
see how simplistic some of these animal models were, right?
35:40
So despite the spite, the Splashy headlines room.
35:44
See out there, most companies are doing boring mundane,
35:48
relatively low sophistication of machine learning and it is still
35:55
an adoption curve, that, that kind of has to hit its peak Peak
36:00
point. So that's kind of an interesting
36:02
anecdote about shifts. Before these Legacy companies.
36:04
I think, I think it's goes back to kind of what we started
36:07
talking about. It goes to kind of the mindset,
36:10
culturally, are you aligned have?
36:13
You had Have you had good success from your devops
36:18
initiatives and efforts when you roll them out?
36:21
What did you learn? Have those learnings being
36:23
institutionalized in the company because I think that's a good
36:28
initial indicator of companies that have successfully kind of
36:32
made the devops move and, you know, doesn't matter when they
36:34
did it. And those companies have kind of
36:38
been through a similar type of process and iteration.
36:42
So making sure that they have Background and they've learned
36:45
from an and then that at some point can inform if machine
36:48
learning efforts will be successful.
36:51
The other thing is again, making sure that they've done enough
36:56
experimentation ml offices. Wait, I'm a laughs just by
37:00
definition. Again is for scale for or large
37:03
animal systems. But companies need to spend a an
37:09
outside is amount of time in terms of experimenting with With
37:14
the business problem. Do they have the data for that
37:16
business problem, based on the machine learning technique?
37:18
That needs to be applied? So my questions and find a
37:22
conversations focus on understanding their insights
37:25
from that Journey. What have you done in terms of
37:28
experimentation? And how do you know that ml is
37:30
the right technique? How do you know this, you know,
37:33
machine learning technique of the right technique?
37:36
And and only then should we be talking about kind of how do we
37:40
add scale and predictability and reliability?
37:43
Liability to it because that's a different dimension of
37:46
sophistication in a challenge. So for legacy companies, I
37:52
really make sure that I am spending a lot of time.
37:56
Making sure those Basics have been thought through by the
37:59
company and and and they might not have all the answers, but
38:03
they're putting an effort into it to drive an understanding.
38:06
And if they don't have that, It would be, I think it would be
38:11
crazy for any. And all Spenders with nothing
38:13
for us to kind of sell a solution to a company, that is
38:16
not culturally ready. That is not procedurally.
38:18
People lies ready and what's going to happen is a year or a
38:22
year and a half from now that. Right?
38:23
It's going to fail. Yeah.
38:25
And everyone's going to be disillusioned by the ml Ops
38:27
Meander. Yeah.
38:28
Yeah. You're setting yourself up for
38:30
churn and so you have to it's very short-term thinking now
38:35
when you've gone into these different companies and You have
38:39
seen what they're doing. What are some common pitfalls
38:43
that you've noticed across the board that when you are asking
38:47
these questions that you asked you tend to get answers and you
38:51
realize roof this might be a little bit of a red flag or it's
38:56
something that they don't think is that big of a deal but you
38:59
know, from experience it has been, you know, some of the
39:02
common pitfalls. I would say.
39:07
Again not having broad representation of process
39:11
functions in in the conversations, we talked about
39:15
that Biz not having a business case, understanding why they're
39:18
doing why are they are embarking on this solution or approach?
39:23
We talked about that the other was I would say it is when It
39:29
sometimes it's when you ask these questions you'll surface
39:34
things like we're again, they're buying their buying a mlrs
39:42
capability just to check off a box a lot of times and you can
39:46
kind of tell what do you guys do?
39:47
Such conversations with different team.
39:48
Members have differing opinions on on what it means for them to
39:54
build that capability or use our solution.
39:56
So I think pitfalls So I think it's kind of similar to what we
39:59
talked about. It's not having the right
40:00
stakeholders on table. Those stakeholders not having a
40:03
common understanding of what they're trying to do and what
40:05
they're trying to accomplish with ML.
40:06
Arms are mln General lack of just business.
40:09
What does success mean way mean to them in terms of and then one
40:15
thing we haven't talked about is, are wise, you know, a lot of
40:17
folks will talk about oh this is too expensive to run from
40:21
animal's perspective, or too expensive to build from a no
40:23
mobs perspective. I think that's the wrong
40:26
dimension of conversation. It should be more around, what
40:29
the ROI is. There's different ways to cut
40:31
down costs and costs. A lot of times are just, just a
40:35
function of the infrastructure DWIs UCP as your spend a lot of
40:39
time for training or close but but you can cut those costs down
40:43
by just utilizing heterogeneous on spot instances for example.
40:47
But what customers are only focused on the cost equation of
40:51
of ml Ops again either building or buying and they're not
40:55
thinking about an hour wipe respected that That is also kind
40:59
of a maybe a yellow flag. I've not a nut bread red flag
41:02
because I just means that, you know, they'll have some rude
41:07
surprises and learnings later on, if they're not thinking in
41:12
terms of Roi also wanting a lot of Engineers just want to build
41:16
it. Right?
41:17
Rick and product. Managers often want to do like
41:21
little given the dates. Let's make sure that we are able
41:25
to have some revenues coming in with this.
41:28
So what the real is, it has been visible to me because of being
41:34
really specific in, the emerald research engineering industry,
41:37
where the things are quite obvious which is there.
41:40
So very defined use case. Let's say is there's an
41:43
e-commerce company in the already have customers and
41:47
they're trying to build better predictions and that case, it's
41:50
quite obvious for the e-commerce companies.
41:52
But for there are so many other companies that do have different
41:57
stakeholders. Those that do have different
41:59
priorities. Now, how do you as a vendor go
42:03
in and have these conversations with all of these people who
42:07
come from different departments because you said, you have to
42:10
talk to people from different parts of the organization and
42:14
make sure that they're on the same page with each other.
42:17
Let's say they know how big his everybody has their own
42:20
preferences or everybody has their own version of how the
42:23
product should go. Is that a vendor shop to bring
42:26
everybody on board or should you go?
42:28
Act like guys discuss with each other come back to us.
42:32
When you have a clearly defined really cool that you want us to
42:36
work on I think vendors. So you need a champion at any
42:42
customer. You need a champion at the
42:44
Frontline engineer. Technical level you need a
42:46
mid-level and a business. If if that's in place that's
42:49
ideal kind of situation to be able to bring all the
42:54
stakeholders together and kind of have that conversation of
42:58
Elite eight that conversation. If if there are, if those
43:01
Champions are not identified or in place, then it's going to be
43:04
really hard for a vendor to dry those types of conversations
43:08
because we can't vendors, can't bring the right people to the
43:11
table. If the right people are at the
43:13
table, we can we can facilitate that conversation.
43:15
It's okay for engineers to focus on the technically oriented
43:19
outcomes. Make salsa so big of a Formula
43:21
One car, right? So the engineer that's building
43:24
the Formula 1 car is focused on those.
43:26
Very technical aspects of Car. Whereas but the Formula One car
43:33
by itself is kind of a, it's not an ml pops type of constructs of
43:40
a a racing team is an oval office concert right here doing
43:43
it at scale. You're doing it.
43:45
Repeatedly, there needs to be more outcomes there, there's
43:48
business owners that, oh, there's financial interest with
43:50
the team to be successful. There's advertising stuff that
43:52
needs to happen. So so I'm just use that example,
43:55
that it makes total sense for engineers to focus on.
43:58
Those very technically oriented outcomes of kpis but because
44:04
Emma Ops is production izing and ml system, but you can build an
44:08
ml system in isolation. That does every part of the ml
44:11
lifecycle, but it's not an at scale system that that's, that's
44:16
doing inference, that any kind of at any kind of scales, but
44:19
when you start moving the scale, you got to bring all those
44:21
functional stakeholders in there.
44:22
So, I got to work a little bit more on that now, Z, but the
44:25
idea is that it heat up engineers Should be rightfully
44:29
focusing there and it's not the vendors job or even the edge of
44:33
years to only think in business terms, that will be, it will be
44:37
a stronger engineer. We can do both, but I would have
44:40
machine learning Engineers applied, which e-learning
44:42
Engineers, it devops folks that I connect and engage them to the
44:48
right level of their counterparts on the customer
44:52
side of thing. But then, let individuals from
44:55
my team's. Am I shooting engineer Drive?
44:58
If that conversation with his or her counterparts on that side,
45:01
so then it kind of becomes a synchronous and folks are doing
45:05
things on their own and we're kind of getting all that
45:08
feedback back together putting a big picture and saying, okay, a
45:11
mr. Or Miss customer you know,
45:13
either you're too early for this or the success that you're
45:16
trying to derive from your envelopes.
45:19
Objective might not be as attainable because of x y z
45:23
reasons. Think about this and that so you
45:27
know, whatever outcomes that come out.
45:28
At the end of the day you're kind of just like a party
45:31
pooper, huh? You come to these guys and you
45:35
say look that whole idea of machine learning and AI that you
45:39
have on your PowerPoint. Yeah.
45:41
Let's be realistic. You're not even in the cloud
45:43
yet. Get ahold of yourself.
45:46
Yeah, I mean, so either of you that I can have that
45:49
conversation today or I can have that a year from now after I've
45:52
spent and they spent tons of Dollar hours.
45:54
Yeah, totally. I'd smart obviously you want to
45:57
do what's right for the cost? A, and if what's right is you
46:00
are not ready for this. And that's what's right.
46:03
But most times, you know, we'll try to structure so that we make
46:08
the make their effort and initiative bite-size to say, why
46:12
don't you start with this component?
46:16
You know, with I'll use an example for Union, right?
46:19
We've got flight, which is our open source orchestration
46:22
solution. Then we're building Union Cloud
46:24
as a managed offering on top of that and injured in cloud is a
46:26
lot of Enterprise features. And In and there's a price tag
46:29
associated with it. Whereas flight is open source.
46:32
So we talk to customers all the time to say, listen, Laura
46:36
sophisticated, orchestration platform.
46:38
You might not be quite ready for the union scale just yet, but
46:41
why don't you do flight? You know, you can run it
46:43
locally, you could run it on a small old Foster leave as have
46:47
and just experiment with that and we'll support you, you know,
46:50
as an open-source Community to be successful with that.
46:53
And then we'll move you to communion Cloud.
46:55
Yeah, you know as much as dollars would be nice, if the
46:58
End goal is about making that customer successful, but
47:00
whatever is right for them at that time.
47:02
So, let's switch gears a little bit because one thing, which is
47:06
fascinating to me for for somebody who's never worked in
47:09
customer success. I am curious, were fascinated
47:14
you to this field. Just overall Indian how the
47:17
transition happened to machine learning.
47:19
Yeah, you are love. I love customer success because
47:23
it is one of the few functions that that touches every point of
47:27
an organization. So You're dealing with product
47:29
engineering sales support in and it gives you a very good
47:34
perspective of whole will company.
47:36
I love, B2B platforms, B2B infrastructure.
47:39
Because I think there's there's like these compounding effects
47:42
from B to B infrastructure IV to be platforms it because you
47:46
know, you make Engineers successful and more efficient
47:50
and effective and then those of Engineers go and build services
47:54
and solutions and software that makes an end consumer out there.
47:58
You know, successful as well. So I just love the Dual kind of,
48:01
you know, nature of benefits that come from B2B
48:06
infrastructure. So, focus on that, what I love
48:08
is the the people in process acts aspect of complex
48:11
Technology Solutions before that.
48:14
With the cloud prior to that when I was at Microsoft is doing
48:19
stuff on SQL Server everything. But, but what always always been
48:23
a common thing for me is how how do users?
48:28
Credible, long-term measurable value out of software,
48:34
enterprise software, B2B SAS Solutions.
48:37
And there isn't anything more complex today than machine
48:40
learning in terms of getting value out of it and something
48:43
that is prone to failure. So I think it's like the
48:46
ultimate Challenge in terms of, you know what what attracts me
48:50
to this taste. Both in terms of customer
48:53
success. I am L food.
48:54
Interesting, so the next part of question for people who are
48:57
listening Ting to this podcast on Spotify, iTunes for.
49:02
So how do they get into this rule of customer success with
49:06
working with the machine Learning Company?
49:08
Where are the kind of skills that they need to come up with
49:12
what we take? The little second question
49:13
first. So kind of folks coming into
49:15
canals Emma. Laughs is the is Omar music,
49:20
perfect, but like it's a very good route into ml because about
49:25
jobs is fundamentally a software.
49:28
Eric it devops type of Challenge.
49:31
And and, you know, obviously there's the data science aspects
49:34
to it as well, but but from an MLM is respect.
49:37
That's a good entry point folks on a cloud.
49:40
And and so it doesn't success respective.
49:42
I think it's you, you need to be, you need to have a business
49:47
hat. He'd have a technical.
49:48
Had you need to be a good communicator.
49:50
Used people skills. So folks, that naturally
49:53
gravitate towards those types of activities.
49:58
At ease. I think find customer success
50:02
interesting. And if you have that type of
50:07
again, if you gravitate to those types of things, customer
50:10
success, makes sense. But so yeah.
50:13
If you want to be, if you want to be technical, you can get to
50:16
the nth level of detail. I have such a hard time, finding
50:19
customer-facing, machine learning Engineers.
50:20
So, you know, if there's a, if there's a machine learning
50:23
engineer that happened that is that has the right communication
50:26
skills and can Engage The On deeper level to understand.
50:31
Why are you opting for this tooling?
50:32
Why are you picking this design Choice versus that?
50:36
So people that ask a lot of questions people that
50:38
communicate. Well, but I can also be
50:39
technically proficient. Those people do well, it
50:43
customer success, especially on. Yeah, and I'll type of solution.
50:46
So there is one thing that I want to touch on before we jump,
50:51
and that is around the different sizes of companies and working
50:56
with different. Teams and juggling and however,
51:02
the capacity it is that you're juggling the ideas of all these
51:08
stakeholders where the needs and the wants of all these
51:12
stakeholders. And I know that we had
51:13
previously mentioned, what a beast, it is to work with the
51:17
government. I thought working with a big
51:19
Enterprise company was bad, you schooled me and told me that
51:23
that's nothing compared to working with the government.
51:26
Can you break down the Differences in working with
51:29
these and how you tend to go about the different styles of
51:35
engagements. Yeah.
51:39
Yeah. I mean on a complexity scale
51:40
government sits at the top Trump's off.
51:43
Yeah. So if ya had exactly, so what do
51:47
the bigger challenges it? Again, this comes out to people
51:50
and process again because technology is technology and and
51:54
it's rare. That technology is a limiting
51:56
factor for for a company to get value out of whatever that
52:01
Technology Solutions adopters, usually process of procedural
52:04
and government. For example, complicates the
52:07
process Procedural aspect way more than say you know a 30%
52:11
startup where decisions can be made quickly and they can they
52:14
can iterate quickly. So it's it's so much about the
52:18
devops kind of Journey that we went through as an industry that
52:23
it took us like a decade to just transform that mental model in
52:27
the mindset. That is what I kind of used to
52:30
inform some of the ml off shelters and outgo part of
52:33
overcome them. So working with the large
52:36
customer like a gun. Man, you want to make sure that
52:39
you have complete Clarity on who the decision maker is.
52:43
Who are the right players who is influencing and informing the
52:48
different stakeholders and just making sure that you are staffed
52:53
and Italy. And from a, from a resource
52:56
people perspective to engage in those long cycles and long
53:00
decisions of conversations and design choices, they're very
53:02
deliberate design choices and and sometimes it's a good thing,
53:05
sometimes it's overly bureaucratic.
53:07
Akin but we just want to kind of approach that differently and
53:11
kind of have different expectations and and frankly who
53:15
followed their process. Yeah.
53:17
Can we double click on that real fast?
53:18
Like how do you not lose your mind by things?
53:21
Just going so slow? Well, the deal is likely going
53:27
to be so gigantic that you just kind of just focus on the cash.
53:32
They, I know, I'm being a little facetious of that answer.
53:36
Could I think you actually hire, you put a different type of
53:41
person to work with that customer and New View.
53:45
You know. Even from a account management
53:47
perspective is different, but the technical stakeholders, I'll
53:50
be working with their counterparts.
53:53
It is 8. It was a different type of
53:55
individual who can sit through. Literally, I'm sad through eight
53:59
hour long sessions over three days to come up with a design
54:03
choice on a system. And now granted this was a very
54:06
complex system military application.
54:09
So you know, there's tons of there's about twenty six tenders
54:11
involved so because it's like that, some of those, some of
54:15
that procedural step by step approach makes sense, but you're
54:19
right. It is it is daunting.
54:21
So you want to have the right person who's able All to engage
54:26
at that intensive level of dialogue, the customer and just
54:31
go in with the right expectation.
54:32
Quality traits of this person are Oh, someone who's curious
54:37
someone who is just you know intensely focused on making sure
54:44
that you know, we're not just saying yes to a decision because
54:48
it's convenient some of those courageous and one of the back
54:51
hole in someone who can who can go deep and Broad if needed.
54:56
So it's again it's those people kind of qualitative skills then
55:01
end up mattering more because again from a technology
55:03
perspective. Well, you What have a better?
55:07
A greater sense of Legacy, Technology Solutions.
55:10
Could then the most people would a lot of government applications
55:16
are the flower, but most of them are not so you won't be able to
55:18
make sure that you can. You can have some of those
55:21
conversations as well. And I imagine you also want to
55:24
be patient you've had stats like quality number one.
55:29
I can imagine if you go in with those expectations though and
55:32
you know that it's just going to be a long process and you're
55:35
not. Expecting to get anything done
55:37
anytime soon. Then it will be surprising if
55:41
you do get something done within years, as opposed to months.
55:44
And so, oh, yeah, yeah, so yeah, swimming a nine-month cycle is
55:49
is kind of where you want to start by terms of getting things
55:52
done. Wow, but what makes a person in
55:56
this role successful is being able to incorporate and design
56:03
around design a solution. Round the customers limitations,
56:09
limitations, whether that be in terms of infrastructure, in
56:14
terms of internal processes that they have, they might have
56:19
internal tooling that they've built that they need to
56:21
integrate with or as with some of the more you know, Cloud
56:26
native newer companies, you know, there's a there's a
56:28
clearly and pretty common stack that's available, it's usually a
56:32
WSU sleepy. There's some opt-out from an SEO
56:35
perspective, But for large Enterprises, there's some
56:37
tooling that they have internally that you have to
56:39
incorporate and desired around that.
56:42
So being able to drive conversations that help surface
56:46
those customer limitations and then being able to work
56:50
collaboratively with the customer to design a solution
56:54
that works well in spite of those limitations that they
56:58
have. Which again could be policy,
57:00
security policy process. You know one example I'll give
57:04
you is. Some of the larger customers
57:07
don't want you managing their infrastructure resources, so
57:10
whether kind of eks or even even, you know, just kind of
57:14
doing scaling, which is what you need to do as a developer
57:17
solution and they don't want you to have like direct access a lot
57:21
of time. So they're BBC.
57:23
And in for a lot of customers, it's valid.
57:26
I mean you don't want to tell them.
57:28
Listen, everybody's running in the cloud, don't worry about it.
57:30
It's for them. Their need is such that, it's
57:32
valid. So that's a limitation where
57:34
they don't want you out. Assessing their infrastructure
57:38
in their me, PC, for example. So how do you design your
57:41
solution that can still deliver that high levels of service?
57:45
Reliability and all the value of the customer gets out of it
57:49
while incorporating that limitation of, in this case, a
57:53
security limitation, I love this idea of like sussing out where
57:58
the limitations are and then figuring out how you can still
58:02
be successful with those limitations finding them.
58:06
Early and hopefully finding them quickly.
58:11
And then being able to say now we're going to do x y z because
58:17
of this limitation that we have or because of these demands that
58:23
we need to keep in mind. So dude, well this has been
58:26
awesome. Thank you so much for
58:28
enlightening us about how you doing the what you're doing.
58:32
Why am L Ops is so much based around the people of the
58:35
Assesses and why it is so important to focus on these
58:40
intricacies of people and process when it comes to doing
58:44
machine learning. I appreciate you coming on here.
58:47
Also, thank you for having me. Evan not recommend to a podcast,
58:53
sem and lead engineer to keep you think I'm definitely not
58:58
wanting other Engineers to have all this relevant information
59:02
and I will not give my name is Justina.
59:04
So definitely I do not recommend this podcast.