Latest / Agentic Conversations (formally mlops.community) / Machine Learning Operations — What is it and Why Do We Need It? // Niklas Kühl // MLOps Podcast #137

Transcript

Machine Learning Operations — What is it and Why Do We Need It? // Niklas Kühl // MLOps Podcast #137 · Agentic Conversations (formally mlops.community)

0:00
So hi, I'm Nicholas. I actually have two jobs on the
0:03
one hand, I am working as a managing consultant in the area
0:07
of data science for IBM Consulting.
0:10
And on the other hand, I'm leading a small research lab
0:13
called the applied AI in Services lab with eight
0:16
researchers. Where we do a lot of research on
0:18
the interface of Academia and Industry and where we try to
0:21
bridge that Gap and use novel approaches from Theory to apply
0:25
them to practice, I rarely drink coffee on.
0:30
Only once a week and then it depends what the occasion is.
0:33
So if it's a social event, I'll have a cappuccino.
0:37
If it's only to get some work done, it's just black coffee,
0:43
welcome everyone to the ml Ops Community podcast, this is your
0:47
host Demetrios. And I am here today, with Abby.
0:51
We just spoke with Nicholas who wrote some incredible papers
0:56
that you probably have seen. They went and they had Their
1:00
moment on social media. If you are at all involved in
1:04
the ml Ops World, you probably saw these.
1:07
The paper is called machine learning operations, overview or
1:12
ml, Ops overview, definition and architecture.
1:16
Abby, what were your takeaways I love this conversation, but one
1:21
of my favorite things from this conversation with his focus on
1:25
the product side of things, where he said Emma loves is not
1:28
just about the technical side of things which is the package of
1:30
the Tooling in the Frameworks. But you have to think about the
1:33
entirety which goes from the first point when we are really
1:38
thinking, if there's scope for this project and if there's
1:42
going to be a return on investment for this project, So
1:46
good. Yeah, and his definition of ml
1:49
Ops, I've Been Looking for clean definition.
1:52
I love what he said. So keep your ears out for that
1:56
when you're listening to this because I could not put it in a
1:59
better way. I think he articulated what it
2:02
actually means to be doing ml Ops.
2:05
By the way, before we jump into this full conversation, I want
2:09
to mention that we have three different newsletters.
2:12
You can subscribe to on our website if you are not already.
2:15
D. Also it would meet the worked
2:17
was if you give us a rating and leave a review for the podcast.
2:22
Yeah. Some of the reviews that we've
2:24
gotten thus far, have just been incredible.
2:29
I will read one of them to you right now.
2:31
Abby, it is the worst podcast that I still come back to.
2:38
So thank you, whoever John 750 for is.
2:43
I appreciate that rating. Keep them coming.
2:45
And we read them. All means the world to us.
2:54
Nicholas, we should start with a little bit of how you came to
2:59
get into ml because I know it's been a winding road for you.
3:03
I want to hear The quick summary before.
3:05
We actually jump into the incredible paper that you all
3:08
wrote. Thank you.
3:10
So ml, not ml Ops, I mean ML and then m/l Ops give us the yeah
3:16
you can make it as flourished as a hobby.
3:19
You've got started in the field the entire field.
3:22
Yes absolutely. No, so actually, it was, I think
3:25
back in 2014, I just finished my, my master studies in
3:29
industrial management and engineering.
3:32
And I was looking for an interesting PhD topic and in
3:36
2014, Machine learning was a very, very promising field.
3:41
But still a lot of interesting applications have not been done
3:44
yet. So, there were a lot of
3:46
opportunities to work on that, and I was quite open to
3:51
exploring something new, of course, it's your PhD, right?
3:53
So Of three and a half years to work on that.
3:56
So you'd rather do something that you that you love.
3:59
So I thought I'll definitely want to use machine learning as
4:03
a methodology and on the other end, then I looked for an
4:05
interesting application and I found one.
4:07
So I started working with Twitter data.
4:10
Lot mining, Twitter data building models, predicting,
4:13
customer needs in Twitter data and this was basically my my PhD
4:18
and three and a half years went over crazily fast.
4:21
And after that, I was At this important stage, where some will
4:26
continue doing a postdoc other will go into in, others will go
4:29
to industry. But at that time, I already
4:31
started my own Lab at my Institute.
4:35
So, I continued and I found many more interesting applications of
4:39
machine learning. And from that on, I started
4:41
building my own team. The team got larger at a later
4:44
point. I joined IBM as a data
4:46
scientist. So from there on a lot of new
4:51
avenues started, but it actually all started.
4:54
At one conference where I was confronted with the idea of
4:58
using machine learning as a methodology for for Designing
5:03
Innovative products, and that really made sense to me at that
5:07
time. Incredible.
5:09
And so, you got to see, just the amazing examples of how Humanity
5:15
can be incredible people by scraping a bunch of Twitter
5:18
data. I'm sure did you a did review?
5:24
A lot of Twitter data as you can imagine.
5:26
And as you put it correctly the vitreous quite a few interesting
5:31
tweets among them. Yeah, the only thing better
5:34
would probably be if you scraped a bunch of Reddit data then yes,
5:39
I always the next product. So actually my product started
5:42
with scraping Reddit, Facebook, and Twitter, but Facebook closed
5:48
down its API, while I was developing and the Reddit API
5:51
didn't work as well. So in the end, it was just left
5:53
with Twitter. So I got my fair share of Reddit
5:57
posts and Facebook post as well. And did you create any products
6:01
you mentioned that you you created some products in the lab
6:04
were they to help companies? See if there was buying intent
6:09
on Twitter was it something like that?
6:12
That's exactly the correct direction.
6:14
So what I did I build an artifact of product which I call
6:18
it need mining. So what it did is it actually
6:20
mind Twitter data for customer needs and I did that for
6:25
different domains. I think one of the most
6:26
interesting back at that time, were electric vehicles.
6:30
So I wanted to support in better understanding why our people are
6:33
hesitant to using EVS is that the range is the price, the
6:39
convenience, the charging infrastructure, stuff like that.
6:42
So I wanted to support actually both Society but also industry a
6:45
little bit with that to gain more understanding on why are
6:50
people not using V's. Why are we potentially not
6:53
addressing? They're their needs.
6:55
So the product is called neat Mining.
6:58
And we're there any conclusions on why people didn't want TVs?
7:01
I mean now it feels like they're much more accepted, right?
7:04
Especially Society is embracing them but back in those days and
7:09
when was this 2016 2014 to 2017? And so what was the conclusion
7:15
there? I mean, most of the stuff is
7:19
fixed by 2022. I'd say, but back then, it was
7:24
missing charging infrastructure arranged that works for most of
7:30
the people in their daily lives. But not if they want to drive on
7:33
holiday especially not if they are long if their computers,
7:37
which have to go quite far distances together with a
7:41
missing charging infrastructure, that was quite an issue.
7:44
But I mean, this was these were on a map.
7:46
At a level, the main needs. But if you looked on the the
7:50
highest frequency, it was always charging stations, not working
7:54
either. Technically not working, or they
7:57
were technically working, but you're charging card or your
8:00
charging airports, not compatible.
8:02
And this was a big issue in 2015, 2015, 2015, 2016.
8:07
Because there's a lot of there were yet to carry with you 40
8:11
different charging cards of different charging providers and
8:14
then check which one would work with Charging station at this
8:18
works. A lot better as of today.
8:20
I can tell you from my own experience.
8:22
Incredible. Yeah, I mean, there's still some
8:24
of those complaints, right? Especially if you're doing a
8:28
long road trip, but let's get into that Tech a little bit.
8:32
You were using NLP to figure out what people were saying and
8:37
extracting insights from that. Can you experience breakdown
8:40
like, how you did that? Absolutely.
8:43
So there are two parts of that. The first part is identifying
8:46
Whether a tweet contains a need regardless of what that precise
8:50
need is. First of all, it's all about
8:52
identifying need tweets. So tweets containing needs and
8:56
then there's the second part and that part it's more interesting
8:59
to Cluster the needs. Once we know that there are
9:01
needs contained and of course this is all based on natural
9:04
language, processing actually very basic, very simple stuff
9:08
back at that time, right? Things like tokenization
9:11
removing stop words, things like that.
9:13
A little bit of semantic, pre-processing.
9:17
I Burnham's things like that, but really not to advance to be
9:21
honest and already work pretty well in identifying the needs
9:25
identifying the tweets that contained the needs and then
9:27
after that I tried out different things like unsupervised.
9:31
Learning for clustering, the needs into different buckets.
9:34
Having pre-trained models that they that and some things work
9:38
better. Some were not so good.
9:42
I think, if I would do it again now, with, with the foundation
9:45
models that you have at hand, right?
9:46
Now, with all the pre-trained models with things like like
9:49
birds I think the results would be much much better, but to be
9:53
honest at that time, it was more of a proof-of-concept.
9:55
Can we do this at all? Can we drive Innovation with
9:59
this automatable? Scalable approach?
10:03
Because previously people would go over to the customers
10:06
manually through interviews with them.
10:09
Do focus groups, surveys and it was all very manual and I was
10:12
like, wait but people willingly Express their needs on
10:16
platforms. Twitter like Reddit like
10:17
Facebook, why don't we just use that data?
10:19
That is available where we have so much larger set of data,
10:25
right? So, that brings me to the next
10:29
question which is it seems like a lot of work.
10:31
You were doing was around consumer consumer, customer
10:34
Behavior prediction. Was that a niche that you picked
10:38
out for yourself? Or were there some techniques
10:40
that sort of interested you to go into these particular kinds
10:43
of applications? We are as there are other
10:45
applications As well. Like adversarial and all of that
10:48
stuff was actually coming up around 2017 against super sort
10:52
of a big thing as well. Yeah, excellent question.
10:57
So this has a little bit to do with with the history of the
10:59
Institute. Where did my PhD?
11:01
Because they are first and foremost interested in
11:04
Innovation. So how can we drive Innovation
11:08
often times in an interdisciplinary Manner and it
11:12
was not really important. What kind of tools that you use
11:14
to drive that and I was among the first ones were actually
11:17
used machine learning in that group.
11:20
So this was kind of the angle that I had and this meant Okay,
11:25
if I want to design better product, if you have a design
11:28
thinking mindset, then obviously, you need to
11:30
understand your customers. And then obviously, this is the
11:33
data that you will look for is firsthand data generated by, by
11:39
users and potential users. So this was the angle that I had
11:42
back then, nowadays. I'm more open, to be honest.
11:46
So we do work a lot with data from from Factory trees, with
11:50
iot data, to be honest. It's not a single specific
11:58
industry or field right now, but this is where I started
12:01
initially. I see.
12:04
So that brings me to one more question on this, which is given
12:08
you say you were the first one from your lab.
12:10
He was working machine learning and particularly in these kind
12:13
of applications. How is everything easier?
12:16
Because it hard to figure out the management side of these
12:19
things because that's one piece, which is still broken, which is
12:22
the communication between machine learning systems, and
12:25
the management that is that these companies, because there
12:29
are sometimes quite some expectations of what machine
12:32
learning can achieve for, Us. And what cannot?
12:36
Excellent. So luckily I was not the first
12:39
but among the first one. So there were two colleagues who
12:41
did this as well. So we were all struggling at the
12:44
same time and to be very honest with you, I think there were no
12:49
expectations at all back at that time because of course, machine
12:54
learning research was already a big Community, but it was mostly
12:58
driven around developing new architectures, developing new,
13:01
new algorithms, it hasn't, it had less focus on the
13:05
application. Ian's side, although this did
13:07
already happened in the industry, research was more
13:10
focused on the algorithms, so in the end talking to potential
13:14
customers talking to companies who could use that that product
13:19
they had actually no expectations at all.
13:21
They were just like, okay, if it, if it helps at even a tiny
13:24
bit, it does help us write. Because, as of now, this is an
13:28
entirely manual process. Now, looking at my Consulting
13:33
work today at IBM, the world has changed So now, the expectations
13:38
are quite high in when it comes to what can or can machine
13:42
learning do. And, and this is something
13:45
that's really important and great for mentioning that Abby.
13:48
Is that you communicate? What are realistic expectations
13:52
when it comes to ml products, right?
13:55
Mmm, and this is not easy, typically.
14:00
Well, especially with the current hype around machine
14:05
learning and I'm Sure, people probably see stuff that's like,
14:08
wow. I created this image out of text
14:11
and then they extrapolate out a little bit more and they're
14:14
like, you know what, we'll just throw some AI at it and that
14:17
will solve all our problems. Absolutely, because the tools
14:22
become so convenient more and more like, and I mean that's
14:26
also a great park, right? That that's awesome.
14:28
I mean, this is a technology that that a lot of people should
14:31
be using and experimenting with. I mean, this is great at
14:33
hopefully helps with the adoption of ml products, but in
14:37
the end it's exactly what you just said, right?
14:39
It raises some expectations that are sometimes hard to meet.
14:42
And as I work a lot with German smes small medium Enterprises,
14:48
it's your you either me to do Two types of people, write the
14:52
rotten are so hyped about that and think that ml products can
14:55
solve all the problems, regardless of the data that they
14:59
have, and the other ones are highly skeptical.
15:03
So it's always interesting to work with these people and to
15:07
get realistic expectations. So, what we were also wondering,
15:14
let me just grab this before. We jump into the paper, I wanted
15:18
to ask about, like, One thing that comes up a lot in the ml
15:26
Ops Community is budgeting and when you're doing production
15:31
machine learning at scale, there's a lot of times that
15:35
people will try and figure out ways to bring down the cost
15:38
because it seems that when you're putting machine learning
15:42
into production, whether it is the cost of the engineers that
15:45
you need or it's just the AWS spend, it's expensive.
15:50
Have you looked at that and figure it out, like How there
15:55
are ways or tricks or special secret sauce, that you may have?
16:01
That's actually one of the main challenges that we are facing
16:05
right now because I mean we will get into that, right?
16:08
But if you really want to excel at ml orbs, then this requires
16:14
different roles, right? And an interaction of these
16:17
different roles, to be really successful to really
16:20
operationalize ml products. Now, on the other end, this
16:24
means a lot of costs because you might need five people, six
16:28
people 10 people, 50 people and then On top of that, of course,
16:32
are the computational resources that are required.
16:36
And this is really an issue right now, because before we
16:39
start a project, we need to convince the customer that it's
16:43
worthwhile, exploring that and depending a lot on what
16:49
experiences they already made with animal products, some are
16:52
positive, some are negative. They want to know what the
16:55
return of investment is. And this is really interesting.
16:59
And to be honest, this would Something, maybe four different
17:03
podcast, but it would be really interesting to drive more
17:06
research in that area. How can I be really good at
17:09
estimating? The value of M L products and a
17:13
very early stage. How can I make this tangible?
17:16
To the customer? What we typically do is at this
17:21
is part of the ml Ops Paradigm saying, whatever.
17:25
I bid, even, it's even if it's a tiny proof-of-concept, you build
17:29
it in a way that it's going to be.
17:31
Be productive scalable from Day Day Zero on.
17:36
And in my experience was really, really helpful is show something
17:42
shiny at the very, very early stage.
17:46
So the customer re can see where this is going becomes hooked,
17:51
even if your model is not working properly yet, but
17:54
showing something early is really, really helpful.
17:57
And I mean, that tools as of now which helped us in doing these
18:01
things. Rapid development cycles and in
18:03
my experience, this really helps then to secure the budget that
18:06
is needed to really build a successful mL of steam and then
18:10
in the end a successful ml product.
18:14
I love that, okay? This is something really
18:16
interesting that even the high interest, High interest debt
18:20
paper. I think that's one of the most
18:22
popular papers in the field. Giving.
18:24
It was quite early and ahead of its time with all the dead that
18:28
accumulates from deploying, these Machine learning models
18:32
have production in the changes that happen when it comes to the
18:35
data sources data dependencies and all these things.
18:38
So that brings me to your paper right now.
18:42
You researched quite a lot of Articles, talked to quite a lot
18:45
of people but everything kind of process for you.
18:48
Like and what inspired you to work on this paper?
18:50
Specifically, Yeah, so first of all, it's a team effort, right?
18:56
So they're two point two more people that should be mentioned
18:58
here. Dominic and and Sebastian, so
19:01
the three of us, we are the co-authors of that of that
19:03
paper, doesn't humble, man right there, making sure that if we
19:08
know everyone gets to others, you can't take all the credit
19:12
for it, but they didn't. I think when I reached out to
19:15
them to see if they want to be on the podcast, they didn't
19:18
respond. So for now we're going to say it
19:20
was all you, but Like that. You say, yeah, no, no, there
19:25
were two other people. Let's not forget that, of
19:28
course, we internally coordinated who will do the
19:30
interview, but ya know us. So, yes.
19:34
Apart from me being humble, to be honest.
19:38
I so I'm a bit of an of an oddball, in both words, in the
19:43
academic space because I also work at a data scientist at a.
19:46
Large American company at IBM and on the other hand, for my
19:50
fellow data scientists, I'm a little A bit odd because I do
19:53
research and teaching at a large German University.
19:57
But to me, personally, this is my dream job because I get to
20:00
know what's currently happening in Industry a.
20:02
But on the other hand, I also have the tools and the theories
20:05
that I think are often times really helpful in addressing a
20:07
lot of these. A lot of these problems and Emma
20:11
knobs for me was a phenomena that we did use already in
20:15
projects in terms of its basic principles, but to me it was
20:21
never really clear. You're, what is it all about?
20:26
I mean, what are its core principles?
20:28
What are the processes? What is the architecture?
20:30
I had a rough understanding of that, but it's not like if you,
20:33
if you would have asked me, it's not like I could have told you
20:37
immediately, right? So this was something.
20:39
I was not really satisfied with more more.
20:42
The academic of me, less the consultant and then luckily.
20:49
So I knew that already. I want, I knew I wanted to To
20:53
write the paper on Emma, laughs. And I knew that I wanted to do
20:56
interviews to do a literature study and the to review and
20:59
combine all of these three. And then I met, one of my calls
21:04
has Dominic and he was actually searching for a master's thesis
21:09
and he wanted to do something on envelopes so I thought.
21:11
Okay well this is perfect. This is the perfect match.
21:14
Let's let's do it together and that's how it all all started.
21:18
And then we first dug deep into Cha and surprisingly we didn't
21:24
find a lot. I mean, of course, they are, is
21:27
a lot of literature on ML Ops, but when it comes to giving you
21:30
a good introduction, defining principles roles, components,
21:35
architectures, their books, which are 300 pages long but no
21:40
one can read that or will read that, especially not novices who
21:43
just get into the topic and I would consider myself a novice
21:47
even even now, right. So I think that was that was
21:50
something that I wanted to do. Do and from the, from the
21:54
literature that you saw, I think back.
21:57
But when we started, I think none of the papers was
22:00
peer-reviewed, right? So they were all on archive or
22:06
other other other service, right?
22:07
Where you you would find open science.
22:10
And I thought this was really interesting, not that they're
22:13
not to say that they're less work because of that.
22:16
But that just showed that Academia was lagging behind in
22:20
in researching that phenomenon. Which even motivated me more.
22:25
Okay, this is now, the best reason to start investigating
22:28
that and after reading all the literature and a lot of that was
22:31
helpful, we even thought. Well, but there's still a gap.
22:33
We still expect. We as now speaking, as
22:36
practitioner, we experience ml orbs from a lot of other
22:39
different aspects. There are a lot of things that
22:41
are not mentioned in literature, so we definitely need to talk to
22:44
people who do that on an everyday basis, either from tech
22:48
companies, or from customers of these tech companies that use
22:52
these Tools. Yeah, this was the setup for for
22:55
the study. Yeah.
22:56
You had to set the record straight in a way and I think
23:00
you all did such a nice job with it.
23:02
Because when it first came out, I literally saw it everywhere.
23:07
Whether I was looking on LinkedIn, or on Twitter or in
23:10
our ml, Ops Community slack. It was being shared like
23:14
wildfire was just going off and so I think it resonated it
23:19
really like it hit a vein with a lot of the practitioners out
23:22
there. As you put into words things
23:24
that we had intuitively been feeling and knowing but you were
23:29
able to articulate it. And so for that I think you all
23:32
did a great job and I'm wondering now that you did that
23:36
and do you have a definition of what ml Ops is?
23:40
What what do you? How do you like to say?
23:42
When someone comes to you? And they say, what is Mo Ops?
23:44
Because I have my definition but I'm still like trying to figure
23:48
it out to be honest. And so I think the, I would love
23:52
to hear what Yours is. Yeah, the I sent first of all,
23:55
thank you so much. I really appreciate that.
23:58
And I think part of the the success if you want to call it a
24:02
success was that I had the outside perspective in not being
24:06
too deep into these these areas. And I think that that really
24:10
helped. Now, regarding your question on
24:13
a definition. As you can imagine, this was one
24:15
of the things we discussed the most when writing that paper.
24:21
So we use a lot of I'm discussing bed and I mean in the
24:24
end you come up with the final version, but there were so many
24:27
other versions of this of that definition.
24:30
In the end, we try to have one specific focus in that and that
24:35
is let me try to put it in words without reading out the
24:39
definition. But every, every machine
24:42
learning and ever should result in a machine running product,
24:47
and this should be the ultimate goal.
24:49
So there always needs to be a product in the end that is Then
24:53
maintained that is continuously run continuously developed
24:57
improved, but that should always be the goal and everything,
25:01
everything that Emma loves does as a paradigm.
25:05
It should support that claim, it should support.
25:10
The process of developing robust, operate fully,
25:16
operationalized machine learning products.
25:20
Now, this is not the definition but this is the goal.
25:22
I think if you have that perspective on it, then a lot of
25:25
other things come into place because then suddenly you're
25:28
going to ask yourself. So what do I need in order to
25:31
accomplish that? Then suddenly we can talk about
25:34
the principles required, the roles required.
25:38
How do I instantiate these principles at the interface of,
25:44
of the of the role and off the architecture?
25:47
And so for me that's it. Develop is it's a paradigm for
25:52
Ali developing ml products. Hey everyone, my name is aparna
26:00
founder of arise and the best way to stay up to date with Emma
26:03
lops is by subscribing to this podcast.
26:10
I love that it's starting with that end State and realizing we
26:14
want to get here whenever we're doing any kind of machine
26:17
learning. The goal is that we actually are
26:20
extracting value from these machine learning models, and
26:23
it's the Trope. Now, the common Trope is that it
26:26
stays on somebody's lap top and is in jupyter, notebooks
26:29
forever. And that's why I think a lot of
26:32
people resonate with the idea of like production izing machine
26:36
learning. And so I really liked and I She
26:39
ate the fact too, that you are not specifically calling out.
26:45
Like oml Ops is these set of tools or these set of pipelines
26:49
that you set up. But also there is that
26:51
organizational level involved in the processes that you need.
26:56
And I'm really like I think that a lot of times myself and in the
27:01
end of Ops Community we sometimes get wrapped up in the
27:04
tooling and we make it we turn ml Ops into A technological
27:10
problem, but it's not only that. How did you navigate?
27:15
That idea of 0ml Ops is just the set of, you know, you set up
27:19
these pipelines. Use a few of these off the shelf
27:22
tools or grab yourself some vertex, AI or Sage maker and
27:27
you're good. Yeah, that's a tune excellent
27:33
point. So of course we could have gone
27:38
that way. And if you read the paper, you
27:42
will, you will see a few names of tools being dropped like
27:46
Humanities redhead openshift. Things like that, right?
27:51
But for for me what was really important to have a
27:56
socio-technical perspective on that?
27:58
And this is because again The Institute I'm coming from has a
28:01
focus on Innovation and it always says we cannot only look
28:05
at technology. We have to look at the
28:06
socio-technical systems. Right?
28:08
We have to look at the combination.
28:09
Nation of Technology, people and organizations and only if we
28:13
have this holistic perspectives on new phenomena, we can really
28:17
understand them because of course technology is not
28:19
isolated. Technology will always be used
28:21
by people within organizations and only by having having that
28:25
view. You can really, really
28:27
understand that. So where we start the project it
28:30
was really important for us to not have a strong focus on
28:33
tools, but always think on a meta level what are so we do
28:38
have these tools. What are the shared principles
28:42
Behind these tools? What do they have in common?
28:45
Even when we did the interviews, right?
28:47
We had the transcript of the interviews.
28:50
We had to manually code. All of this, this data.
28:53
Even looking at that, at all of these, these answers from from
28:56
the experts within the field. We always try to abstract from
28:59
that and say okay but what is the underlying meta level here?
29:04
What can we what can we say? Because What You observe in a
29:07
lot of papers, is that? The Technologies come and go
29:11
even the precise ones. But what we want to do is to
29:13
have an understanding on what is the the general idea of the
29:17
Paradigm? What can be still the case even
29:20
in 5 or 10 years when they are new players emerging when old
29:25
players are not there possibly anymore, right?
29:28
This was always what we were, we were aiming at.
29:30
But to be honest, we had to force ourselves a lot and not to
29:35
mention too many tools not to be specific to for example Cube
29:39
ETS. But just try to abstract from
29:43
from that on the one hand, but on the other hand also, to keep
29:46
in mind the organizations and the people that were using it.
29:50
Because that's something that I actually observe on a daily
29:53
basis in my job, as a data scientist, consultant is that
29:57
our customers, they might only have a few data scientists.
30:01
And then this thing, a data scientist has to be the software
30:05
engineer as to be the machine learning engineer.
30:09
And this is really hard, and for, for an organization.
30:13
First of all, they need to understand that an mL of spared
30:16
I'm requires multiple roles in order to function well, but this
30:21
is something that is not necessarily known to the
30:25
organizations and we need a shift of expectations there.
30:33
One thing that you mentioned earlier of what is doable for a
30:36
single data scientists and that in order to be successful We do
30:39
require the monks, others different, different roles with
30:44
distinct, distinct characteristics and goals.
30:49
Wait, so when you are doing these interviews, the user
30:53
interviews, please tell me you were recording them and then
30:56
putting them through the tool that you created to figure out,
31:01
aren't that great? You didn't think of it, man, you
31:06
should include me in this project, I would have given you.
31:09
All kinds of insights. But it brings up this this idea
31:14
which I love and you are, are speaking of.
31:17
And it's almost like if we take a step back, I find it
31:20
fascinating because Abby and I just recently interviewed Simon
31:26
who talked about how and he's been doing machine learning
31:29
since I think like 1994. If I remember correctly, Abby
31:32
might be able to fact, check me on that, but he was talking
31:36
about, we asked him one of the questions that I think Abby
31:39
asked. was around what a team composition should look like,
31:43
what we should be thinking about when we are starting our
31:46
projects for machine learning and he came back, very sternly
31:52
and said, the only thing that matters in the beginning is you
31:56
figure out the ROI of this machine learning Endeavor and
31:59
then you can figure out how many data scientists you need, or how
32:02
many data Engineers you need machine learning Engineers,
32:05
whatever you need, you will figure out because you can See
32:09
what the data is like, what? All that is going to amount to
32:14
after you recognize how much money it's actually going to
32:19
make? Yeah, I can relate 100% of that,
32:24
from a practitioners perspective because of course, you need to
32:29
make sure what is the return on invest that, that, that you will
32:33
receive with an MLA product? And we talked about that
32:38
earlier, right? How do How can I do a good job
32:41
in that? How can I be precise at
32:43
calculating a reasonable hour? I but then on the other hand,
32:46
also to make sure that I can assign the required resources to
32:50
what is needed in order to reach a tower.
32:53
I so it's kind of a chicken chicken egg problem.
32:55
So, from a practitioners perspective, I would agree
32:58
saying that. Of course, budget is everything.
33:01
Because if we don't have the budget, maybe there's only one
33:04
person which has to fulfill the seven rules.
33:07
That would be required in theory.
33:09
For a successful envelopes project, but that's then, just
33:12
the case. Now, the luxury that I have
33:15
right now, is to lean back and say well, but from an academic
33:18
perspective, of course, these are the required roles and this
33:22
is what the architecture look right.
33:24
But putting it into practice. I can fully fully fully agree
33:27
with them. And that's why I think typically
33:30
the most successful projects that we did in the realm of
33:34
envelopes were not the first ones with that same company,
33:38
right? So we would Will you start with
33:40
small ones with the company showing some successes showing
33:43
that they would actually receive that Roi that we promised
33:47
earlier and then a full-blown ml Ops project like we would
33:51
describe it in the paper would be the fourth or the fifth
33:55
project at that stage. And then we would have the
33:58
distinct roles and then all of that worked out.
34:01
But of course this is kind of the reality T.
34:04
Check on these on these problems.
34:06
And again, I have the luxury of sitting here and saying well,
34:09
Look at that paper and that's how it should be done in theory.
34:13
This is interesting because I was actually looking at the
34:16
figure that you've given in the paper which is basically
34:18
end-to-end envelopes architecture.
34:20
And one of the things I realized at least in my personal
34:23
experience working in the industry is a lord people just
34:25
expect one person to know everything.
34:27
So I came from like a research background and everybody was
34:31
expecting. Hey, come dude, engineering and
34:33
everything and that is still an expectation, which people having
34:37
more and more companies are now trying to Define this new rule,
34:40
which is basically called Full stack Yes, like this somebody
34:45
even if they can't do everything but at least they are a viewer
34:48
of the entire process and to end.
34:50
So I just wanted to ask you a little bit about that one which
34:54
is you said there are different versions when it comes to
34:57
working with these teams and their different stages.
35:00
So could you break down these stages with me and talk about
35:04
what kind of rules are the companies.
35:06
Look at these SME companies are looking at when it comes to
35:09
different stages of the project and how does the team increase
35:12
in size? Isis after different levels.
35:15
Yeah, so definitely, I mean, this depends a lot on the
35:20
organizational setup of the of the Data science, seems AI teams
35:27
whatever they called in the respective company.
35:32
One typical phenomena that we observe is there is a designated
35:37
data science unit Center of excellent or Center of
35:40
competence and then they work together with with the business
35:45
units and some cases they even have satellite data scientists
35:48
sitting in the business units and then you have a core Center
35:51
or the center of data science and in the first Paige.
35:55
It's all about showing success. You don't have time to weekly to
36:02
think about the different roles, although you should you should,
36:04
but it just doesn't work. So you need to show some ml
36:09
products which actually generate value, and this is the first
36:14
success criteria from from from my point of view, right?
36:17
So you typically start with a few data scientists and I'm just
36:20
going to call the data scientist.
36:22
What they do is they are data. Tist.
36:24
Then a male engineer then as the engineer, their the data
36:27
engineer, the devops engineer in the software engineer, but all
36:30
within one, one person. And then if you were successful
36:34
with your first ml products, then at some point in time they
36:40
get more headcount. And then, of course, the
36:42
question is, who is that new role going to be right, who's
36:46
need the most data scientists. And I, another a male engineer,
36:50
now, the software engineer and this is one of the most
36:52
interesting phases of these of these Again, from an
36:55
organizational perspective to say, how do you prioritize now,
36:58
different roles and how do you, how do you do the setup for a
37:03
successful data science team? And now again I didn't do an
37:09
empirical study on that. So this is only my, my personal
37:11
experience. But at some point you have
37:13
reached the level where all the gears are are working right
37:18
where all the roads are collaborating where standards I
37:22
established where workflows are published.
37:25
And then, of course, this at this stage, your products become
37:29
scalable. They become fully fully fully
37:32
usable product. And that's interesting stage,
37:35
but to be honest, every it's really hard for me to say, when
37:38
is this stage where everything is working, flawlessly and
37:42
what's that? What's that breaking point,
37:44
right? What's the amount of people you
37:47
have? This is really hard to say, but
37:49
it's interesting. It's really interesting to look
37:51
at these different stages. I would agree.
37:54
How do you define success for different kinds of designs
37:58
projects which is. So the key is given the fact
38:02
once the project has been deployed in production is a, is
38:05
it just hate the model works? And it predicts whatever you
38:09
want to predict. Let's say it's a housing thing
38:11
in it has done their job or let's say after we start
38:15
deploying it in production in there.
38:17
A lot of things get involved, especially when we start working
38:20
with streaming data and as such. So it takes a lot of time before
38:23
we can monitor. Clarinda to all of that part
38:26
which is the later stages of managing, the modern dealing
38:29
with the drift and all of that. So at what point do you start
38:33
looking and defining your model as a successful model or the
38:37
project is successful in saying okay now let's move to the next
38:40
stage and we've done what was asked of us?
38:45
That's that's highly individual as you as you might expect, and
38:51
there are a lot of discussions on that depending on the
38:53
business unit, you working with the customer that you're working
38:56
with. For some, it's only performance
38:59
right? In a supervised machine,
39:01
learning model. It's, can we continuously
39:03
achieve that performance in? I don't know what identifying a
39:07
novel, he's predicting prices, whatever things like that, and
39:11
it's only performance-driven. I think it's debatable whether
39:16
that's, that's meaningful, but of course, companies think in
39:19
kpis, they think a numbers. And that's, that's the way that
39:23
they approach it. So let's that's one way that you
39:27
can do it, but to me a lot of ml products also deliver value on
39:32
other on other areas, for example, they give you
39:35
meaningful, insights, whatever that means, right.
39:38
I mean, this is, this is, this has a lot of perspectives to it
39:42
and it's hard to say but it definitely Definitely gives you
39:45
insight into your own operations even though if it might not
39:49
reach the performance you want to aim for it did give you some
39:51
interesting insights and most of all it is a very good reflection
39:56
of your business in many cases because because it does show you
40:00
what data do you have? How valuable is that data?
40:03
Even if you're embedded product in the end turns out not to be
40:07
successful, this overall process, that we also go through
40:10
in the paper, the workflow reveals a lot.
40:14
Insights on your data, quality on your current state or as a
40:21
company when it comes to data driven products.
40:24
So, it's really hard to measure success with single metric and
40:32
it really depends to be honest in the end on the, on the
40:36
personality of the business stakeholder.
40:38
Because some do understand that and they say, hey, if we get
40:42
insights into a few anomalies He's in our data, this is
40:47
immensely helpful. While others say is hey hey we
40:50
need a performance of, I don't know, F1 score of 90% And if we
40:56
below that that's terrible and if we're above that, that's
40:59
awesome. So I think it's really hard to
41:04
say but I think that's a discussion.
41:06
We should have. What makes ml product
41:08
successful? Is it only a single performance
41:11
metric or is there more to it? So then would be really
41:13
interesting too. Explore further.
41:16
Also research wise, that's your next paper.
41:19
There you go. Here we go.
41:20
So, speaking of next papers and all that good stuff, I mean,
41:24
what are you up to now? You've been doing this paper or
41:28
you, you did this paper. Now, you mentioned that it is.
41:31
It's being peer-reviewed, and hopefully, in a few weeks, maybe
41:35
a few months, it's actually going to be published and put
41:39
out. But then you have other stuff
41:41
going on like your day-to-day, which involves Cheating in
41:45
triathlons. I hear you like to skip the
41:48
course, and then win. What is going on there.
41:51
Give us the breakdown on that story.
41:52
On the triathlon. Okay.
41:56
So dimitrios, of course, required a fun fact about me.
42:00
So I entered that I, that I cheated at a triathlon, not
42:04
knowingly, to be honest. So it's actually, it's actually
42:07
a short story. So I went to, I think it was
42:10
like, my first triathlon and I was late right here, have to
42:14
bring your Bike and get into your swimming gear, things like
42:17
that. I had to find a parking spot.
42:20
So I was really really late for my cue what that meant is I
42:23
missed the entire introduction where they tell you.
42:27
Okay, now we swim here then we biked there then we run there.
42:30
So I missed that entire pod and I just came at the very second
42:34
where the people jumped into the water, that's when I entered and
42:36
I also jumped in the water. I did my swimming, I did my
42:40
biking. And after the biking, I felt
42:42
well that went really, really well.
42:44
I Didn't have any GPS or stuff like that.
42:46
So, I was just just doing the Bike Tour.
42:48
I change into my running gear started running, and then I
42:52
recognize that I'm I'm there's no one else there, right?
42:56
I'm, I'm really the top guy, no time.
42:59
I did feel, I did feel. I mean, of course, it was
43:04
exhausting. But I was like, is this
43:06
something feels off? So, I walk into the stadium.
43:09
The people were applauding the announcer said, this is
43:13
incredible. Look, who is coming breaking
43:16
World Records? This is his sister's his first
43:18
triathlon and he already makes first place.
43:21
We were going crazy and that point, I felt.
43:23
Okay. Something is wrong.
43:25
I mean, there's something something went wrong here but
43:28
you didn't stop and say it, then you cross the finish line under
43:30
that guy because I ran through the stadium and the celebrations
43:36
were later as rights because all people had to get in.
43:39
So I did this together with a friend of mine and he came in, I
43:42
think 20 minutes later or so and He was like, wait, you're
43:45
already here and typically when we were training he was he was
43:48
the more athletic one. So I.
43:51
So when he told me is that, in fact, the instructor said, you
43:54
have to go to the bike Robin Hood.
43:56
Two times. Not one time and then change it
44:00
through the running gear. So, but I didn't hear that.
44:02
And I didn't have any GPS, and well, I thought it was, alright.
44:06
So I was very ashamed after that because obviously I Then I then
44:09
cheated and at some point I had to tell them and they figure it
44:12
out. I think themselves at some
44:14
point. So, I'll be honest with us, did
44:18
you still pick up the metal and the trophy?
44:21
No, to be honest, I when I went home, I still do have a
44:23
screenshot of of the placement, but I was actually very ashamed
44:28
after that. So there was actually my first
44:30
triathlon experience. And yeah, apparently first and
44:34
last that was the last, to be honest, but I did cheat.
44:38
So I feel bad about that and now it's Public's.
44:41
Yeah, you've been banned from any sanction triathlons, now.
44:44
Do they think you're on EPO or something?
44:47
You're doing drugs like lands. But anyway, aside from cheating
44:51
in triathlons, what else are you up to these days?
44:53
Yeah, I mean of course, ml Ops is not the the only piece that
44:57
I'm that I'm interested in but what always interests me, is
45:00
this combination of Academia and Industry?
45:02
So what tools methodologies, what knowledge is do?
45:06
We need to bridge these two worlds of people working on the
45:11
best algorithms in the field, but on the other end on the
45:14
people, Seeing it in practice. And one thing, I mean there are
45:18
multiple things that I'm working on.
45:19
I think there are, there are three ones that I find most
45:24
interesting and I'll try to be try to be quick about them.
45:28
What is actually inspired by Andrew NG, which is called
45:33
data-centric a. I don't know if you guys heard
45:36
about that, but I think it's again, it's kind of a paradigm.
45:40
It's, it's a really, really great idea.
45:44
Sure? Because what they say is in the
45:46
past, we had a very modern Centric approach on building ml
45:49
products. So, someone gave us data, we've
45:53
got the data from somewhere and then we started to build the
45:56
best possible models on that. And somewhat, it's were 0.1%
46:00
such points better than the others and we did competitions.
46:03
But that's not the way that we should think about it and it's
46:07
actually something that we also put into the paper.
46:08
And I think it's it connects well with with ML Ops because it
46:12
says, you have to look at the holistic picture.
46:15
Don't take the data for granted. In some cases for factory.
46:19
It might be easy to Simply Implement an additional sensor
46:23
and a stage, where it would give us a lot of insights.
46:25
So think of that holistically, right?
46:27
Not only think about what you can do in the model part.
46:30
But also think about what you can do in the data part, even if
46:33
it means. And I know this is strange for
46:35
computer scientists, but even if it means going into the field
46:37
and changing a few things around, and I think if you have
46:41
this, this, this mindset, this change is how Your products are
46:45
in the end because sometimes it only does require one additional
46:48
sensor here or one additional camera here and it would give
46:51
you all the things that that you need.
46:53
So I think this is a very promising field of research and
46:57
the data center Ki not necessarily that you knew need
47:02
new methodologies there. But that you just make sure that
47:05
people working on animal products have this this
47:07
Viewpoint, not only doing hyper parameter optimization to the
47:11
last bit. But also thinking about the,
47:13
the, the overall path. I don't know.
47:15
What are your thoughts on that? Is that?
47:16
So yeah that you think about. Well I think that's the beauty
47:20
of this industry, right? Because we get to work.
47:24
So closely with the business side of things, the machine
47:27
learning is so tightly coupled to actual business outcomes and
47:31
then like you said, you can actually go into the field and
47:36
Tinker with things to see if you can produce better data.
47:39
So that just opens up. So many different Avenues on How
47:45
you can solve problems and for engineers who enjoy solving
47:48
solving problems? It's the world.
47:51
Is your oyster, absolutely. Absolutely.
47:54
Yeah, yeah. That's that's so true.
47:57
Maybe one, one more thing that, that I really like, and this
48:02
really applies to the, to the Emma products once they are once
48:07
they are in place. So, in the past, I have been
48:09
like to and I'm simplifying here.
48:12
But there have been two types of people.
48:14
One say, Okay, all the Amana models at some point of time
48:17
will take over all the jobs in the world.
48:20
They will, you know, become the best predictors and every task
48:24
that we have out there and on the other hand, people saying ml
48:28
products or more General. A I will never reach that that
48:33
stage and I think their arguments for both cases but
48:39
what I do observe now in both research and Industry is is a
48:44
collaboration of human agents and machine learning agents.
48:50
And this, this phenomenon of human AI teams is really
48:54
interesting because there's so many design choices that you can
48:58
make and how to present predictions to humans.
49:02
How do you miss inject with these predictions?
49:04
How do we explain them things like that and I could talk about
49:07
that for hours, but I think one of the most interesting or
49:11
important aspect that I myself learned in that.
49:15
Is that we do not need to build for performance, but we need to
49:19
build for complementarity because what you do see is that
49:24
humans have different strengths than the machine learning
49:28
products, but right now typically, the machine learning
49:32
products are built in a way to be good at everything but I
49:35
don't think that's the way we should have brought approach
49:37
this, this hybrid intelligence but we should say, how can we
49:41
make the best out of these two entities?
49:44
And And if we start specifically designing a product for
49:48
complementarity, I think that would be a huge, huge change,
49:53
right? Because we did some.
49:55
We did some studies where we can show that the the complementary
49:59
team performance. So, the performance of both
50:02
entities together is better. If we aim for that
50:07
complementarity because what's interesting is that in many
50:09
cases you have an AI product which recently, which is the
50:12
performance of 90% Why would you need the human in that stay
50:17
right, because the human on paper might reach?
50:19
I don't know on 85% but so if you simply combine them in many
50:24
cases, we still the the top that we reach is still this 90% of
50:29
our of Rai product. But if we design for
50:32
complementarity, now we can use the strengths of both our
50:35
entities and maybe come even above what the top AI performs
50:39
of what the top Yuan performance is and I think this is really
50:42
interesting, really promising and what I really love About
50:44
that is this idea of machine and human working together.
50:48
Collaboratively. On a task instead of saying, oh,
50:52
the machines will take over or the machines will never take
50:55
over, and I think that's great and I think there's there's a
50:58
lot of stuff to be done to understand better how this
51:02
complimentary Works, what our principles that make it work.
51:06
So that's something I am really motivated.
51:10
About interesting case about two years ago, I was Applying for a
51:15
PhD in almost similar which was are doing some work around
51:19
assisted living but sort of combines machines but humans to
51:23
serve help us lead better lives and that comes a lot of the
51:27
applications were in Warsaw and medical health care in such
51:30
working with depressions are working with autistic patients,
51:33
all of that anyway. So one will question, I'll ask
51:36
you on that one. What does a day-to-day in
51:40
Nikolas life look like? It's very interesting because
51:44
typically, there's not not the day-to-day, but I do have two
51:47
jobs. Right on the one hand, i'm
51:49
again, managing consultant data science for IBM Consulting.
51:52
So I do Consulting work. On the other hand, I'm, I'm
51:56
leading a research lab at the calls for service research
51:59
institute, where we have eight full-time researchers.
52:03
So I'm doing customer work, I'm doing teaching and doing
52:11
research some Edmonds. Stuff, obviously, but that's
52:15
what makes my days interesting. Because it's, that makes sure, I
52:18
think if I was only a researcher, it would be too
52:22
boring. And I would fear losing that
52:24
grip to reality and Industry on the other end.
52:28
If I would do only Consulting work, I wouldn't really feel
52:32
feel satisfied as well. So my typical day is the mixture
52:36
of doing lecturers consulting with phds and then having Some
52:42
workshops with with customers and Deputy this, this works
52:47
really nicely because I can use a lot of knowledge that I
52:50
generate in the research world. But on the other hand, I also
52:54
know what the problems in the real world are and what we
52:57
should approach next. Yeah so I mean I'm funny that my
53:02
dream job for someone who's around you.
53:04
What is one lecture that they can drop in given by you?
53:07
If any do you mean like like a, like a public?
53:12
Lecture a public lecture could be a university.
53:16
I mean people's sometimes, just walking if somebody wants to
53:20
attend fun of you talks, where did they go other than other
53:23
than our podcast? Obviously, that's a good
53:27
question, so because I do come come around quite a lot.
53:31
So, if you're in South, Germany and you're close to close to KI
53:35
T, this is typically the spot where hold lectures.
53:39
So, if you're at in, in karlsruhe, which is a beautiful
53:41
town in, Germany, then then feel free to drop by at Khe.
53:47
Come to a lecture or even if you don't come to a lecture, happy
53:51
happy to say hi if you come by my office.
53:53
All right, so that's probably the, the physical place where
53:56
you could be. But apart from that, you can
53:59
reach me via LinkedIn, Twitter, and the typical channels.
54:02
There we go. Don't say too much.
54:04
I'm close to South Germany. So, I'm right outside of
54:08
Frankfurt. So I may drop in on you, see,
54:10
we're now almost welcome last. Last question that I've got for
54:14
you is what I find fascinating right now.
54:16
There's so much advancements. And so, in re in the research
54:21
field, it feels like things are going very fast, and they're
54:24
coming out with new different papers every week, I know we've
54:32
talked to some people about this and how back in the day in 2019,
54:36
you could keep up with all the Deep learning papers that came
54:40
out, because there was maybe one For a week or one per month and
54:46
now you definitely cannot just because they're coming out so
54:49
quickly. And you also see all kinds of
54:52
wild stuff happening with foundational models and then
54:55
you've got the different piece which is you're in the industry
55:00
and you see what people need? How do you look at that gap
55:04
between, what is coming out and The Cutting Edge of research,
55:08
and then what the companies actually need and Potentially
55:13
without sounding too cliche. Like how do you feel about the
55:17
foundational models that are coming out?
55:19
Do you feel like those are going to be able to fill some of the
55:22
gaps that we have when doing machine learning in the industry
55:26
and not just be for Creative types.
55:31
Excellent question. So let me answer the first part.
55:35
So this is actually one of my main challenges in my in my
55:40
daily job because The amount of papers, even if you only look at
55:45
peer-reviewed ones is too high for a single person, even for an
55:50
Institute to, to read to comprehend, to reproduce.
55:55
It simply doesn't work anymore. So that's something that I'm
55:58
struggling with a lot is to keeping up with the pace.
56:02
I mean, it's a positive thing that so much is happening, but
56:06
for an individual, it's really hard and can be very, very
56:10
overwhelming. So this is Something I really
56:13
honestly struggle with keeping up with it and it does happen
56:16
sometimes that people in a conversation mention, some topic
56:20
that I've never heard about. And for them, it's, it's
56:23
completely clear what that is because that's their Niche.
56:26
That's a gap, but it's becoming really hard.
56:30
So, I don't, to be honest, you were not asking for a solution
56:34
but I don't have one. It's really hard.
56:36
And I, I do follow a lot on what's posted on LinkedIn and
56:40
Twitter. But if that's the best way to
56:42
go. Phds approaching me with with
56:44
papers. I mean that's how I do it but
56:47
you want to get a little bit of real fast.
56:49
I'll tell you I saw a foundational model, I think it's
56:53
using GTP 3 and it is summarizing papers that come
57:00
out. And so it's using machine
57:02
learning for this exact problem. Which I find a little bit meta
57:06
but sorry I cut you off. Hey, no, you did not.
57:09
No, I mean that would be the answer, right?
57:11
I mean here we have it the Of the models are going to solve
57:14
all of our problems. But now that's probably not
57:16
going to be the case. So so coming to to phone
57:21
foundational models. I do believe that this is
57:27
exactly what is needed. But to be honest more in the
57:31
industry than in research, I think in the in the research
57:35
field still there going to be a lot of specific models.
57:38
I mean also an industry there will be a little of specific
57:41
models but I personally really appreciate that.
57:45
We have this trend towards more generalizable models that can
57:50
then be used on a lot of lot of different tasks.
57:53
As of now, I'm having a real hard time on saying whether this
57:58
will be successful across all different areas.
58:01
And I also don't feel in the position to say, whether the
58:06
current ones are are having the best effects, but overall that
58:13
something I'm really, really interested in actually that's
58:15
something we do together with, with IBM research as we speak
58:19
right now, because we're trying to build found foundational
58:21
models for climate disasters. So we do believe in that but as
58:27
of now it's really hard to say, if we will succeed, although I
58:31
do hope for us being successful in that, best of luck to you,
58:36
man. This has been awesome.
58:37
They give us. Thank you so much for coming on
58:39
here and telling us all about your life and your paper that
58:45
you wrote and you're cheating habits on your triathlons.
58:48
Everything has been awesome. Dude, this is so good.
58:54
Thank you guys. Really appreciate it.
58:56
Thank you Abby. Thank you very much.
58:57
Was thank you for having me and this is Skyler.
59:00
I lead machine learning and health rhythms if you want to
59:03
stay on top of everything happening in ml.
59:05
Ops, subscribe to this podcast now.
59:10
Now, now, now now now Now.