Latest / Agentic Conversations (formally mlops.community) / Packaging MLOps Tech Neatly for Engineers and Non-engineers // Jukka Remes // #322

Transcript

Packaging MLOps Tech Neatly for Engineers and Non-engineers // Jukka Remes // #322 · Agentic Conversations (formally mlops.community)

0:00
You can put the we can extend on this, this later, but basically
0:04
you can you can put the like the expectations for for things on
0:09
our platform and then you can take it to the ML operations
0:13
side. And then you can also follow up
0:16
on on the like how those expectations are met.
0:19
So. So I'm Yukar, Ms. I'm a senior
0:27
lecturer of ICT and AI at Hakka Haley University of Applied
0:31
Sciences, and also the founder and CTO of A to AI.
0:36
And yeah, mostly I drink tea. We.
0:39
Were just talking about open source platforms.
0:42
You mentioned that is very, very top of mind for you, almost
0:47
passionately top of mind. Why is that?
0:52
Well, yeah, I actually I'm a quite late entry to the like
0:57
actually contributing to the open source.
0:59
I've been mostly using those. But then we like found this new
1:04
life for the this platform that we created that at Silo AI,
1:11
Yeah, for for envelopes, a certain type of like open source
1:15
platform. And now like I've been working
1:20
since Silo in, in academic setting and then like I've, I've
1:26
been in the research world before and there I, I really,
1:31
when I started things, there weren't these kind of like
1:34
stacks available. So I'm kind of coming the full
1:37
circle back to, to the research world where which would benefit
1:42
from envelopes. And then this like having the
1:46
open source platform which we can develop for that purpose
1:50
specifically, then it's now my one one of my contacts that I'm
1:55
living out. Can you explain to the listeners
1:58
what was Silo doing and why did you need to create AML OPS
2:03
platform? Oh yeah.
2:04
So I actually, yeah, I turned silo originally.
2:08
I see, because Silo was like, now it's acquired by AMD and,
2:13
and the reality is might be different, but back when I
2:15
joined it in 2020, just just after the COVID time started, it
2:23
was a consultancy largely. And, and I, I joined as a, a
2:28
solutions architect. So I did, I did a client cases
2:34
and big and small and many of those actually that I ended up
2:40
working with, with envelope. So there was like this demand at
2:47
that time to build practices for the envelopes as well as then
2:52
the platform settings. And then we also like then
2:57
launched as one phase of Silo to have this like technology areas.
3:02
And I also took up on leading the cloud AI and envelopes area
3:07
that we had at Silo. And it then entailed like both
3:11
competence development because we had like people with very
3:16
various backgrounds at Silo, especially coming from, from
3:20
academia was a very, very strong, like deep research based
3:24
AI and kind of data science type of like skills, but varying
3:28
levels, levels of engineering related to, to to the AI and,
3:32
and MLN. Then then partially we, we
3:38
started like putting effort into the competence development
3:41
having like information sharing sessions, workshop workshops,
3:46
like all kinds of things, visiting lectures and such.
3:49
But we also developed, started developing that the because we
3:54
had a certain amount of like internal R&D activities at
3:57
Shiloh even at that time. So we started developing in my
4:02
team then this platform based on open source components and the
4:11
IT was that it's quite like loosely, it's kind of
4:17
integrated, but quite loosely. So we are basically we're
4:22
collecting those kind of like open source technologies that
4:25
made most sense. And we're going to extend that
4:29
and make the like the platform for certain kind of like client
4:34
cases so that we can, we can for, for, for those clients that
4:40
completely, for example, outsource the ML development
4:43
that we would have like a platform to do the development
4:47
on and and deploy the models and such.
4:50
But then also regarding this kind of like cases where we have
4:54
clients who are bit more mature or even really mature and they
4:59
want to build their own platforms or parts of the like
5:02
MLF setting, then we can like leverage for example, even just
5:06
part of that platform set up that we have for for those
5:10
purposes. So, so that was kind of the
5:13
origin origin of the of the platform work in in itself,
5:17
yeah. So classic story.
5:19
You have AI researchers and these ML researchers that don't
5:24
have as strong of engineering chops, and they probably want to
5:28
see their models and their work get into production.
5:32
So you create a platform to help that process.
5:36
You mentioned something there that I want to hit on, which is
5:41
you not only created the platform, but you created more
5:44
than the platform, the processes around that.
5:47
Can you explain what those were? Too, well actually not
5:51
internally that much. We didn't like standardize
5:54
internally that much. We, we pretty much focused on
5:57
the on the platform development itself.
6:01
But of course, like in the client cases that there was like
6:06
like what would be the work flows and and these kind of
6:09
things for for for various clients.
6:15
Take me back to that moment when, especially internally,
6:18
when you're saying, OK, we need to start building an ML
6:22
platform, where, where do you even start?
6:24
What do you go with and why is that the place to start and then
6:29
fan out from? Well, it was like based on the
6:32
kind of the demand that we were facing.
6:34
So it was very practical in that sense that it had to do with the
6:38
like what we thought that we would need when engaging the
6:40
silo clients at the time. So we would need the technical
6:44
platform specifically. Yeah.
6:46
Which piece of the technical platform?
6:48
I mean, is it a? Like end to end like so we we
6:53
basically like incorporated, which is still like the extent
6:56
that we got at that point that's high, low.
6:59
We, we got the like the pipeline orchestration and then the model
7:05
management, so supporting both training and development and the
7:12
model and metadata management and for the experiments and and
7:18
then the deployment, but also included some, some components
7:22
for the for the monitoring part. So that was the kind of the
7:25
intent. And we, we, we covered the whole
7:27
whole extend end to end from all the way from training to
7:32
deployment and operating the models.
7:35
Now I'm curious, did you also do stuff with the data?
7:41
No, we didn't actually go that way then ultimately.
7:43
So yeah, that was one idea that we would extend to that
7:47
direction, but never got to that.
7:49
So the kind of the, the reality at the, at the silo as in any
7:56
like this sort of like fast evolving and fast growing
8:02
company was that the direction changed and somewhat like how we
8:06
do things. And so so and like new things
8:09
came along. And then we we kind of like did
8:15
develop the platform forward in, in this R&D collaboration that
8:21
we had with international consortium a bit.
8:25
But then then we didn't that much yet anymore like put the
8:30
R&D efforts there, but there was actually separate like R&D
8:34
entity within silo established. And so the kind of the most of
8:38
the R&D went the kind of like a bit to other directions.
8:41
So, and ultimately this also led to the the possibility and the
8:47
idea that let's open source the platform because the within that
8:50
consortium like University of Helsinki, there was like
8:56
interest for that also Fraunhofer Institute in Germany
8:59
and and some of the companies involved with the with that.
9:05
And so if I'm understanding this correctly, you saw for the
9:09
external clients of Silo, you needed to create some way that
9:15
these folks could standardize and get these models into
9:19
production. And so you stitched together a
9:23
few open source tools to make sure that you were landing on
9:27
these key pieces within the ML development like life cycle.
9:33
And after you had done that, you recognized, well, since this is
9:39
pretty much a full platform in itself, why don't we take what
9:45
we've built and put it out there into the world?
9:49
Yeah, exactly. And then of course I continue
9:51
with that because then we could involved the University of
9:55
Helsinki has continued developing certain things on top
9:58
of that actually in the research like context because they have
10:03
been making a set up, extended set up with that platform.
10:08
We are soon like also merging that effort into the into the
10:13
main repo of the platform in the but they are like they have
10:18
included support for this high performance computing or
10:21
supercomputing in the in the centres that are available in
10:25
Finland for for research use. And we are still like actually
10:28
continuing to extend on that. And then I have now developed
10:33
like continued that in the in the university context myself
10:38
with the students to to work on the on the on the project.
10:43
So tell me what use cases this supports, because I imagine it
10:49
is certain types of models that are being built or it is certain
10:53
types of of use cases that you're supporting or is it
10:58
across the board you're trying to help support all different
11:03
models? I haven't like yet myself looked
11:08
that much into the like how the components keep up with the
11:11
large model like context. So have have they like as far as
11:17
I understood from the University of Helsinki that the because we
11:21
have basically cube flow and then we have MO flow and then we
11:24
have case from the cube flow echo ecosystem.
11:27
So they are pretty generic, right?
11:29
So, so you can basically do anything, any anything with
11:34
those. But then like how practical it
11:37
is or like how efficient this it's to like do the trainings or
11:45
do do deployments with with those regarding the last models,
11:49
that's for example, one one thing that it seems to have been
11:52
tested by University of Helsinki, but that's one a new
11:55
thing that I I'm looking into with them.
12:00
But yeah, but in general, it's pretty much meant for like to
12:05
cover all kinds of like cases. Of course, like often like if I
12:11
go present to some, especially certain fields of research as
12:15
they, they, they bring up some like more like, I don't know if
12:21
they are exotic or H, but perhaps like a bit more rare
12:24
cases. Like what about the Federated
12:26
learning here and, and, and this kind of like set up.
12:31
So but the basis, I mean, that's very generic.
12:33
But then then of course, like it needs like extra, extra work to
12:38
perhaps cover certain kind of cases.
12:40
So. Since we're we're talking about
12:42
researchers getting a bit more platform jobs or using an ML
12:49
platform, have you pondered the value that an ML platform brings
12:58
to a researcher? Yeah.
13:00
So like like I said, actually I need to comment still on on the
13:03
previous one that the you you can of course like run any
13:07
bigger workloads as well with the, with any like TPO requiring
13:11
ones. And of course like with the,
13:14
with the kind of the additions that we are now going to as a
13:17
supercomputer environment that that has to a lot to do with
13:21
leveraging the GPU super clusters that are available like
13:25
the AI factories, even even the like the clusters that are
13:29
within the scope of the AI factories that the European
13:33
Union just founded and which are being launched at the at the
13:37
moment. So, yeah, so the intent is to go
13:40
to that direction that we can also trend large models also
13:44
with this platform. But yeah, so the value of the
13:48
platform. So in research, so like I said,
13:53
I, I, I was like, I was working pretty early on, like before the
13:59
whole even not just before the deep learning times started, but
14:04
but before the data science kind of like era started as well.
14:07
I was, I was at the University Hospital of all in in Northern
14:10
Finland. And it was like that time
14:13
analytics and MLI was running certain pretty ready made
14:20
algorithms through all like data from brain imaging through
14:26
through those. And there was like a lot of like
14:29
within that research, a lot of like variation as to like what
14:35
kind of like parameters do you use with those analytics tools
14:41
and like what kind of like filterings of data or other like
14:47
data preprocessing you, you need to have there.
14:50
And then when I need, I had myself, for example, that kind
14:53
of like set up where you have this different options and you
14:58
need to like somehow orchestrate running a very big set of like
15:04
different kind of experiments, basically, or analytics jobs
15:09
with different kind of preprocessing.
15:12
Then then like handling that, those, all those combinations
15:16
and the results therein or even like creating the running of
15:20
those. Then, then I, I, I ended up
15:22
doing that without any of the tools that I had so far.
15:25
So that that's a lot, lot of the kind of the basis of my
15:29
appreciation for the current stacks because it was really,
15:33
really tedious and, and, and I see a lot of like opportunity.
15:36
I think. I'm not sure if you are familiar
15:39
with KNIME. I I guess that's yeah, yeah,
15:42
yeah. But I guess it has like found
15:43
its place like in in bioinformatics at at least and,
15:46
and in some. So KNIME was kind of like the I
15:50
think the first one emerging at that time when I was doing this.
15:53
And after that then came a lot of like different things.
15:57
But the, yeah, the even the Python like of course it was
16:01
used a bit, but it was it, it wasn't like that in that big of
16:04
A use, especially in the in those fields that I was in yet
16:08
at that time. So, so the kind of this, this
16:14
bringing the consistency to complicated research processes
16:20
like increasing the quality. We are supposed to trust the
16:23
information to signific like results that we, we published
16:28
from there, right? So they, they are supposed to be
16:31
the creme de la Creme of like all the information that we have
16:34
in the world. Even though I'm not sure like
16:36
how much fake science there is also nowadays with but but but
16:41
still like the, the quality criteria are very high, right?
16:44
So you if you are doing the complex stuff then you need
16:46
something to deal with the even executing that complex stuff but
16:51
then making sense of the complex stuff and the results.
16:54
This is a fascinating statement. How the reproducibility of
17:01
researchers work potentially comes into question if you do
17:07
not have a standardized way, or at least a way to actually go
17:14
and put that model into production.
17:18
If it's just research that's being done in the lab, then you
17:21
write a paper on it, that's great.
17:23
But there could be some problems when you go down the line and
17:29
you say, all right, now let's actually make this something
17:32
that the industry can use. Yeah, and even even like even if
17:36
we stay in just the science, then like the kind of how well
17:44
can we assess like this goes back to the again, like more
17:47
generic things than and rather ML alone, but it reflects on the
17:51
ML as well because there are there are reviewers that need to
17:56
be able to review the results. And then if it's much more
17:59
credible and it's much more efficient to to review if you,
18:03
if you can share everything. So you can like share nowadays,
18:09
of course, like there's more and more like you are able to share
18:12
at least the experiment setup code and parameters and these
18:16
kind of things. The data is a bit tricky given
18:19
what kind of like constraints might might be related to that's
18:22
used, but it's even like better if you can like share it in
18:27
exactly the same way as you run. So like you can, you can have
18:31
the full pipelines and the full experiments like as such and
18:35
somebody can replicate that. Then it's much more like
18:39
accessible in, in, in in some sense.
18:42
But yeah, like you said, the also the kind of like taking the
18:45
research, research results into, into anything, any, any
18:50
practical use. Then if there's that sort of
18:52
like applied research angle that the, the next step would be the
18:56
commercialisation or the productisation, then there's a
18:59
huge gap if you if you cannot take it like I'm, I'm always
19:03
myself like reflecting. For example, I was working at
19:06
Nokia technologies for a while and then I, I, I saw there and I
19:10
saw at IBM which at which I was working after the Nokia that OK,
19:15
yeah, there are this both of these big corporations that have
19:19
the research centres, but the research is actually a lot in in
19:24
many cases, like even there in that sort of corporate threat
19:28
thing disconnected from the kind of the product development.
19:33
So they are kind of like different, all the different
19:36
stages. So perhaps they are like lifted
19:39
and sifted in in certain manner. The results then that the
19:42
product side and taken further there, but there could be much
19:46
smaller transition to actually commercialise.
19:49
And I, I think now with the advent of the OR even it's not
19:53
the advent anymore, but the, the coming of the regulation like AI
19:58
act and, and this kind of things, then it's if you need to
20:02
have all this like probability of that you have done your.
20:09
Do things when developing the model.
20:14
So how, how can you prove that if you don't start that from the
20:17
research stage, if you if you try to catch up with that in the
20:20
in the in the product development phase?
20:23
Yeah. I don't know how it was at IBM,
20:26
but I know that a friend of mine, Danny, was working at
20:30
Google from, I think it was 2015 to 2020.
20:35
And his whole job was to figure out how to take the research
20:42
that was being done in the ML sphere and plug that into
20:46
products that Google had. And so he was working closely
20:50
with the researchers and anything that was coming out
20:53
that seemed promising, he would go around to different teams and
20:58
say, you think we could use this?
21:00
You think we could maybe plug this in here?
21:02
Would it be useful if? And then he would sit with the
21:04
PMS and try to figure out if there was products there.
21:08
Yeah, that's also like 11 perhaps like aspect like how,
21:13
how what is the driver? I'm really for like this sort of
21:17
like kind of unlimited like innovation, if that's possible
21:23
and and kind of researching like something that is not
21:25
constrained by the by the by the business.
21:29
But then then then yeah, you have like this kind of like a
21:31
bit like tech first kind of approach there, right.
21:34
So that it's not necessarily like addressing, you're kind of
21:39
calling it the other way around because the the results might
21:43
not be based on the real world problems, right.
21:46
And then then they are not addressing exactly some some
21:49
problem. Yeah, and sometimes you can
21:51
create something very cool that people don't figure out is
21:55
actually valuable until five years later, like the
21:58
transformer models. And it's not even you who
22:01
figures out really how to productize that and bring it to
22:05
market. So it it's a fascinating 1.
22:07
And he was also the same guy that told me when ChatGPT came
22:11
out, he was like, yeah, I remember when Transformers were
22:14
at Google and I was trying to figure out ways to put them into
22:20
different like Google Sheets and figure out if we could add them
22:24
to Google Docs and, or the translate app that they had.
22:28
So it's a fascinating 1 to think about.
22:31
Now I, I also wanted to touch on this platform that you're
22:36
creating it a bit because of what you have under the hood.
22:42
You mentioned that you have Q flow, you've got some ML flow
22:48
and then you've got Kserv. Yeah.
22:51
And then Prometives and Crafana, basically those are the kind of
22:54
the five things we were supposed to like extend on them.
22:56
It's it's completely possible because it's it's pretty like
23:00
simple, but it's packaged together.
23:02
And I guess that's the kind of the beef of the whole thing in
23:05
terms of like making it easy to use, so.
23:08
And so is it? Is it another abstraction layer
23:10
that sits on top of these different tools, or is it that
23:16
these tools now can just really easily be deployed as one
23:23
cohesive unit? That that's kind of like the 11
23:27
major thing because I, I think that the like playing around
23:32
with the platforms line and, and, and, and trying to build
23:38
them like some separate set of like instruction, like according
23:44
to some separate set of instructions and then manually
23:48
that's, or, you know, like perhaps ultimating that, but in
23:52
a custom way, it doesn't make sense like for for many users.
23:55
So first of all, it's it, it might be like tedious, even for
24:00
some people that deal with the infra and deal with the
24:03
platform. But if it's a new thing to them,
24:05
then let, let alone let's say that we have that university
24:09
student or, or like researcher from any field that just wants
24:13
to run the ML. So like, if they if they want to
24:16
take this kind of like set up into use and they are not, they
24:19
are not, they don't have like access to some cloud service as
24:24
the, for example, universities not often have and not all, all,
24:27
all the like in the companies is even have then then this kind of
24:32
helps with that. But the kind of the philosophy
24:37
originally also around this would be that then then you
24:39
would like develop this kind of like more of a pipelines on top
24:46
of that. And of course, like when we are
24:49
like using queue flow as a basis, then of course, like then
24:53
you, if you're where you have only queue flow pipelines or
24:56
queue flow there, then you end up defining them in the way that
25:00
you define things for Q flow with the DSL and, and, and, and
25:04
that kind of things. But that was the kind of the
25:05
idea that we and one could provide like this pipeline
25:11
templates in a sense that that would be the kind of then the in
25:15
a sense that more integrating things that you would have like
25:18
the end to end pipeline for all the, all the different stages
25:22
and using all the different parts of that.
25:24
And you could aggregate this like library of pipeline.
25:27
So it's a, it's a bit similar as what you can have, for example,
25:31
with the WSH maker and Azure ML, right?
25:34
You can define pipelines there and then the whole team or
25:38
community or whomever can share them and use them as a basis
25:41
easily to launch their own things on top of that those
25:46
platforms. So that was kind of the
25:48
philosophy here that we have pretty loosely coupled services.
25:51
So those services are not that much yet at least like
25:55
integrated within the platform more.
25:56
What is more is the kind of like intact package that integrated
26:01
package in the sense that it you can install them easily to put
26:05
in place of course, like this work then that my students at my
26:11
university has been have been doing on top of that like with
26:15
Git repositories that and CICD pipelines to to go with this or
26:19
any other platform that that's kind of like extra, extra layer
26:24
also there so. Well, you've given me a lot of
26:29
confidence in the youth when I hear about AI researchers using
26:34
Git and CICD. That is amazing first of all.
26:39
Yeah, I'm not sure like how many actually use that that I I would
26:42
guess that many many use some notebooks just as a on from the
26:48
corner of their own laptops hard drive.
26:51
But the. Yeah.
26:52
But this this idea also is something that makes a lot of
26:58
sense to me in that I can, as a researcher, go and grab a
27:03
pipeline off the shelf and not have to worry about standing up
27:08
these different tools and taking the time to figure out why did
27:14
installing Cube Flow just crashed my whole system.
27:18
You know, like it is a much easier type of workflow if I can
27:25
now go and grab these pipelines off the shelf.
27:29
They come with everything that they need.
27:32
And I don't know how it how it works, the chicken or the egg.
27:35
Is it that I get the pipeline and by using the pipeline, it
27:38
installs everything, Maybe it terraforms out everything that I
27:41
need as part of that pipeline. Or do I do I get everything I
27:46
need and then I can start using the pipelines?
27:49
Yeah. That's more like the latter one.
27:51
Yeah. So currently, so it's not like
27:54
that it would provision it any place that would be kind of like
27:57
extra thing to do as well. But the kind of the installer
28:00
that we have for the the platform is in the open source
28:04
like we have from the silo times the Google, Google Cloud like
28:10
manage Kubernetes like how to deploy it there.
28:14
But the, the main kind of installer with the open source
28:17
plat version of this platform is such that you can, it's
28:20
basically running with the kind so the Kubernetes on, on Docker
28:26
or in Docker. And so then we you, you can
28:31
basically like provision the platform in, in your own
28:36
computer as we are doing actually that there's a lighter
28:39
weight with just the key flow pipelines there or you can do it
28:43
to basically any machine. So, so we are for example using
28:47
the virtual machines that are available in the Super Computing
28:51
centre or this idea centre of science at at Finland for that,
28:54
but you could basically put them anyway.
28:57
So, so that that, that that kind of like need to make them
29:00
pretty, pretty generic in that sense.
29:02
That has like been the approach that we have been using so far.
29:06
But then yeah, after that, then with this kit related stuff that
29:12
the students have been doing there, we are kind of like
29:16
providing tooling with which you can set up like in a sense like
29:25
configuration, configuration related, like hit repositories
29:29
and then ML project related repository.
29:33
So you can establish this configuration repository for, it
29:39
doesn't perhaps make that much sense for one person, but you
29:41
could establish that for the whole team of researchers or
29:44
data scientists or whomever. And then the tool, with the tool
29:48
you can, you can set up the projects for the individual
29:53
people or per project. So those are the kind of the ML
29:59
repos and and those link back to the configuration repos.
30:04
So whenever you have like some configuration that you need to
30:06
update that everybody can just like basically pull much of the
30:10
changes to their own repositories.
30:12
For example, where do the common platform instances reside?
30:17
So that if the endpoint addresses are changing or, or
30:20
this kind of things are, are changing.
30:22
So you can you can pull that updates to to all, all the
30:26
repositories that the the actual ML developers are are working
30:31
on. And then the tool also setups
30:33
the CICD pipeline. So, so you can kind of like
30:36
hideaway kind of this sort of like platform interaction or
30:40
platform API interaction stuff behind those Cicds pipelines.
30:46
So it like you can configure what kind of like CICD workflows
30:52
and what kind of like file structure, like a directory
30:56
structure you want in the in the repo.
31:01
But after you have that and we have like a default of course,
31:05
like to go with that, then it just setups that for you.
31:09
And then you can start introducing your Python code or
31:12
whatever to the different steps of the, for example, the ML
31:17
pipeline. And then when you introduce that
31:19
and if you commit to certain branch, it like compiles the ML
31:23
pipelines within the CICD pipelines and then deploys that
31:26
to wherever you your platform instances are that are that are
31:30
connected to that branch in the Git.
31:32
So this is kind of the way. It's basically like you're like
31:37
saying this is. A self-serve productionisation
31:45
tool, yeah, yeah. But of course like you can use
31:50
that for like I said, also do the research.
31:52
So, so after that, you would have in the research context
31:55
also or in in in company context, if you don't have some
32:01
cloud platform to use or even if you do, you can set up the
32:07
platform easily to some environment or environments
32:12
instead of different instances of that.
32:14
And then you can work with the as usual with the ML kit repo,
32:19
but like focusing on the kind of the contributions that you want
32:23
to do the to the ML part and not to worry too much about the to
32:28
the engineering. And what are some of the
32:32
pipelines that you've been seeing folks set up?
32:35
It's also training pipelines. Yeah, Yeah.
32:38
Of course like with the we have only like the example pipelines
32:43
at the moment. So, yeah, so those cover the
32:46
training paths. But also in those you you can
32:49
have also like as part of the same like pipe, like you flow
32:53
pipelines, you can have the deployments.
32:58
But the the idea like especially originally was that you you
33:03
could like launch whatever. So you can launch just
33:05
separately the training pipelines, you can do the
33:11
deployments as a separate things and so forth.
33:14
So. Now I understand why you were
33:18
saying it's also the ML part, not just the productionization,
33:22
which makes a lot of sense. OK, so now there is a bit of a
33:32
juxtaposition that we talked about before we hit record where
33:38
you mentioned teams desperately need ML OPS.
33:46
But it's not until there's this pain that they feel.
33:56
And so you, you in a way, are hanging out in a world where you
34:02
can be accumulating tech debt and accumulating tech debt and
34:08
not necessarily have to bring on certain practices or platforms
34:15
to help you get rid of that technical debt until you hit a
34:19
certain level. And then you're crossing the
34:24
chasm in a way. And I was just talking to a
34:29
friend of mine on another podcast how AI built this about
34:33
this very same thing, how there's Eno OPS is very hard as
34:37
a practice. And so it inevitably will turn
34:41
people away from it because it is so much work.
34:47
But you get to a point where you can't deny that work and you
34:52
absolutely need to do it. And so if you are a company in
34:56
the beginning stage and you have not hit the pain where you need
35:02
to figure your shit out, you can kind of live ignorant and
35:07
blissfully. Yeah, I guess like I, I, I, I
35:12
think like a nice comparison that I heard was to like soft
35:16
software DevOps like just recently that now it's a kind of
35:22
like a pretty, pretty common stuff in the, in the software
35:26
development itself. It's it still wasn't like 10
35:28
years ago or so, but with the envelopes, even though there has
35:31
been like several years. So already with that, although
35:35
I, I, I think it's the even the awareness, let alone then the
35:42
kind of the adoption they are pretty still like not perhaps in
35:47
infancy anymore, but at at low level.
35:51
Well, as you said, you can like, one can go like go about your
35:56
business, like go about your things like ignorantly and, and
36:01
and such. And then you at certain point
36:04
like end up catching up that or needing to catch up with that
36:08
debt. And I, I guess like in research
36:10
actually that's a bit like perpetual like.
36:19
Thing because there there is certain kind of like perhaps
36:22
like threshold that there that there's a kind of this like like
36:27
we discussed like this standing chasm between the research and
36:33
then the kind of the utilization of the research results in some
36:37
product context. Yeah.
36:39
And the product hitting a certain scale and then once it's
36:43
at that scale, then you start to have to make the product
36:48
velocity go faster. And so to get that velocity that
36:52
you need, you are figuring out all right, well, let's implement
36:56
some processes around this. But also, like, I, I think it's
36:59
just for example, if we consider the lineages and, and the, the
37:05
tracking of the results, then I, I think, oh, I, I suspect that
37:10
the, now the AI act might actually like force to consider
37:18
this. That we talked about like
37:20
previously, that how can you actually, how can the utiliser
37:25
of the like, for example, models done in the research, if they
37:30
are utilising those in a, in a commercial setting, like how,
37:34
how can you like really comply with the regulation necessarily
37:38
unless you have the kind of the whole story behind those models?
37:43
I'm actually a bit sceptical about that like this, some of
37:47
the foundation models, like how, how will that turn out in, in
37:50
terms of the AI act? Because there you have, OK, you
37:53
have model cars and you have certain information there, but
37:55
does that ultimately suffice? So that might be this kind of
38:00
like traceability of like how things have been done, like one
38:03
factor that will actually lead to more envelopes adoption in
38:08
the research stages as well. But I don't know what kind of
38:11
incentive that is to say, academic research then like
38:15
perhaps there will be some funding instruments guiding
38:18
ultimately that the if you want to get the funding for the for
38:22
the academic research, then you should also have some sort of
38:26
like you should have somehow deal with these things as well.
38:30
I don't know. Well, yeah, you already see it
38:32
with at the time of recording this Llama Ford just came out
38:36
this week. And I think there's a asterisks
38:42
in the whole release notes that say, Oh yeah, this is freely
38:47
available for anybody except for if you're in Europe, then you
38:49
really have to be careful with it.
38:51
And we're not going to take any or we just almost want to step
38:57
away from that whole shitstorm and having to put our model out
39:02
there open source and then potentially get sued or have the
39:05
regulations beat down on us. So in that regard, yeah, it's,
39:11
it's a little bit of you need the practices, but you also need
39:17
the transparency to show what you're doing.
39:21
If you really want to put it out there with the EUAI Act.
39:25
I, I think a lot of people are unclear on exactly what you need
39:29
to do because it is written in language and it is not written
39:32
in code. And so the engineers that are
39:35
creating the actual ML and AI or the researchers that are
39:41
creating it are sitting around thinking, OK, how does how does
39:46
this apply to what I am actually making?
39:50
Yeah, and I, I think it's, there's still a lot of like as
39:53
far as I understand it now there's many things like up to
39:59
both like interpretation as well as pending like definition.
40:05
So even some parts of the. Oh, some some certain certain
40:09
things related to AI act, they are coming along only after a
40:13
certain while. And, and as I have let myself to
40:16
understand from people that understand the, the
40:21
relationships of like standards to the regulations also that the
40:27
probably the certain standards will be actually defining the
40:32
many of the technical details ultimately for those.
40:35
But that's also like longer process in a sense because like,
40:42
yeah, but there, there are many of those standards already.
40:46
So those so those will be probably like being leaned on
40:50
when, when these things get actually interpreted or set in
40:54
more set in stone. I have been working with the
40:57
regulations in, in, in the medical field because that
41:02
that's the besides the software and, and MLT third kind of like,
41:09
oh, more like domain specific, like part of my, my whole
41:12
career. So, but not to only to certain
41:16
degree with the regarding the models in, in that context
41:19
because it was like earlier, but now, now I'm now I'm working
41:25
with the AI actor because that's the then the partially the
41:33
context of of the company that I have been founding lately.
41:38
What are you working on these? Days yeah.
41:41
So besides this like envelopes things that I'm those are like
41:46
lastly like this kind of like basic envelopes related and for
41:50
example envelopes related DevOps things that those I'm doing in
41:54
in the in the academic context. But then, then I've been, I have
41:59
now been founding this this new company A to wave AI.
42:03
So there we are putting something more on top of the
42:07
envelopes. So yeah, we are basically like
42:10
connecting the the operation or AI operations to both like from
42:16
the training and as well as the deployment side to the like how
42:21
do you how do you guide and lead the AI project so that they
42:30
actually provide both value as well as them are compliant with
42:38
the regulation. So yeah, that's that's what we
42:41
are building product for so. So and how do you build a
42:45
product that fills that need? Because it feels like it is
42:51
inherently a a service, not necessarily a product.
42:57
Well, it's a yeah, we are. Well we are building basically a
43:02
digital service, but it's a product development.
43:06
So we are not the, it's not consulting, right, so.
43:09
And so you're helping folks that are using AI and ML understand
43:16
how the AI act in the EU has actual repercussions for them,
43:25
Is that it? It's more like that you can, you
43:29
can put the we can extend on this, this later.
43:33
But basically you can, you can put the like the expectations
43:37
for for things on our platform and then you can take it to the
43:41
ML operations side. And then you can also follow up
43:45
on on the like how those expectations are met.
43:48
So, so we are providing like in, in a sense certain type of like
43:52
missing management layer on top of the envelope.
43:57
So connecting, connecting the kind of the needs that for which
44:02
the AI features or AI processes should be built for.
44:10
So and it doesn't concern only the AI, but the also the broader
44:14
business needs so that you actually have some return of
44:19
investment or from the AI process.
44:21
But this is actually something that we we saw like both at Silo
44:28
with people and as well as then before that with with some of
44:32
the people that are involved with the company that did often
44:37
the projects fail because of that.
44:39
So they are a bit disconnected the from the.
44:43
So it feels to me like what you've created is almost a
44:49
connector between the PRD or it is in a way it's a very strong
44:58
and battle tested PRD for the ML efforts because you're helping
45:04
spec out what is needed legally and for compliance reasons and
45:10
then for business reasons. And so you were saying that the
45:14
reason that a lot of different projects or or ML products
45:21
failed was because they didn't have that insight into building
45:26
out everything that needed to be built out on the compliance and
45:32
different stakeholders? Yeah, Perhaps not before because
45:35
of the compliance, but before because of that like they are
45:39
not like necessarily building exactly towards what this needed
45:43
by the business. So there's a broken phone.
45:45
So everybody are working in their own corners of the
45:51
organization and and we are trying to change that.
45:55
So yeah. Yeah, I've seen that.
45:59
I've heard that many times on this podcast.
46:02
It's a broken phone or you're not really each, each expert is
46:09
trying to optimize their little silo and they're not looking at
46:14
it as a bigger product. And so you have someone who is
46:18
trying to optimize the accuracy score of the model and later
46:23
they realize actually I'm building a model that doesn't do
46:28
what needs to be done. Exactly, Yeah.
46:32
And so you're, you're coming at it from a different angle.
46:35
I think then I've seen and heard other folks talk about coming at
46:40
it from which is you're building this middleware in a way that is
46:46
helping the business folks, basically every stakeholder
46:50
because there are so many stakeholders in this process.
46:54
You're helping them all add their issues or their
46:58
requirements to the document or to this software that you've
47:03
created so that at the end of the day, when a product pops out
47:09
the other side after that birthing journey, you have the
47:13
right product, not something that has been optimized in a
47:19
cave without any feedback for for many years.
47:22
And you know exactly what has been happening there.
47:25
So you also like keep track of all the things that happened.
47:28
So you have the like constantly the understanding like how it
47:35
have it has come to be. So you're also tracking the kind
47:39
of the operations there and then you have the confidence to
47:43
report for audit purposes or you have to also the confidence to
47:47
actually take it to production, right, Because the business
47:51
decision makers don't need to be afraid anymore because they can
47:53
see that. OK, yeah, we are here.
47:56
We have done this, it fulfills this and this kind of things.
48:01
So yeah, we're good to go. Now is isn't this what most
48:05
project management software is trying to do?
48:08
And why do you feel like it falls flat like a JIRA or a
48:10
click click up? Is I think they are like applied
48:14
on a certain level. We have a broader ambition there
48:17
to like really cover all the stakeholders in a way that makes
48:23
sense to them. Of course, like you, you, you
48:27
could like it's again like a big same thing as with the envelopes
48:31
air, yeah, you could build your own platform.
48:34
Yeah, you could like use some tool or perhaps like you could
48:37
do like use some individual tool from from someplace.
48:44
You, you can for example track experiment information in in
48:48
queue flow as well. But then you often people want
48:51
to use ML flow more more to that even when they have the queue
48:55
flow. So you everybody could like
48:58
build whatever like ways of applying things.
49:03
But what we are trying to do is like productize that so that it
49:07
you would need to do that of course, like that's the whole
49:10
value proposition of the product productization there to begin
49:14
with that you don't need to have like this hacks or like like
49:19
custom things that you end up like also then needing to that
49:24
much maintain. I see I OK so the the whole
49:31
thing here is that because I I was trying to figure out and
49:34
play it back in my head. Why is JIRA not good enough for
49:38
this? Or why is click up not good
49:40
enough for this? But you're saying not only are
49:43
you gathering requirements, which is what you could
49:46
potentially be doing in JIRA or click up, you're plugging into
49:50
the platform so that it is tracking the experiments and it
49:56
is helping document each step of the journey in this whole ML
50:02
product life cycle. So that it's not like you have
50:05
to go into JIRA and try and figure out a way.
50:08
Are you uploading ACSV file to your Sprint to showcase, Say?
50:14
Here were all my experiments, and here's the one that we're
50:16
going with. Exactly, Yeah.
50:18
You're, you're natively building it into ML Flow and you're
50:21
saying we've got this connection with ML Flow.
50:24
So if you want to go and then figure out what the experiments
50:28
were, you can click into it and see all of that inside of ML
50:33
Flow. Well, you can like see the like
50:38
I, I, I think it's a still different like use that if you
50:42
want to have like ML flow for hosting the models and such that
50:46
that's like as part of the envelopes that that's a
50:49
different thing. But with our platform, then it's
50:51
a kind of additional thing to which you integrate and then you
50:56
put those kind of things there though that kind of like
51:00
metadata there that the different stakeholders need to
51:04
see there in terms of like lineages and and then the
51:12
performances and and and hitting the targets and this kind of
51:16
thing so. It took me a minute to fully
51:19
wrap my head around it, but I love that idea of you have this
51:29
buddy that is taking the journey with you to help make sure that
51:33
you're documenting everything that you're doing.
51:36
But before you you even get that far, you have all of the
51:40
stakeholders gather requirements and plug it in so that you're
51:44
building the right thing. And then when you do take that
51:46
journey, you are automating a lot of this documentation life
51:54
cycle and making sure that that metadata gets put in the right
51:58
place and then it's not lost. Or you have to go back and try
52:01
and sift out two years later and figure out why exactly did we
52:06
make that decision and have that model out there in 2023,
52:11
etcetera, etcetera. Exactly.
52:13
Yeah, yeah, yeah. Cool.
52:16
Is there anything else that you want to hit on that we didn't
52:18
talk about? I'm not sure perhaps this covers
52:24
pretty much what I have been at and and I, I just want to say
52:28
that the of course, like with the there are other options like
52:33
to the open source platform that I was talking about.
52:37
Like I, I think as far as I've looked at the flight, it covers
52:42
certain bits, perhaps even a bit more like approachable manner.
52:49
It doesn't necessarily cover everything as far as I
52:51
understand. So, but the perhaps the
52:53
deployment side, but the, I guess the like if the, if you
52:58
are interested in the, in the, in the platform, please come try
53:02
it out. If you are even more interested,
53:06
come make a contribution, join the, join the project.
53:09
We are pretty early on with the with the open source one, but I,
53:12
I think there's a good stuff going on there.
53:14
Then for example, the software DevOps related part that's
53:18
agnostic. So even though we have this
53:21
Kubernetes in, in Docker kind of set up for the queue flow and ML
53:25
flow and and so forth. Then the I think the the, the
53:31
DevOps part where you can set up the Git based CICD operations
53:39
for your ML projects that's applicable even more widely.
53:42
So, so there's a different kinds of things and of course, like if
53:46
you happen to be in in Finland or interested in the, or needing
53:51
to use the AI factories and the supercomputing, we are extending
53:55
to that, like I said. So now in Finland and later on
54:01
perhaps in Europe, you can, you can utilise this to launch jobs
54:04
into the supercomputers. It was actually interesting to
54:08
hear about the one of the first like AI factory launches that
54:11
was in Finland because the they claim that the it's a in this
54:21
European HPC consoles on the at least with this low Mitsuba
54:28
computer that we have in, in, in in Finland.
54:31
As part of that, the capacity now going forward in, in
54:37
relation to this AI factor is it should be free for all small and
54:41
medium size like companies in, in the union, which is quite a
54:46
tall promise, I think. But the, you know, I, I think
54:49
it's very like a test for the kind of the product development
54:53
part, this capacity apparently available when you need to train
54:57
the models. So, so perhaps in the future you
55:03
can, you can utilise that capacity through through this
55:06
platform. Free GPUs for everyone.
55:09
That's what we like to hear. And big clusters of them, so you
55:13
can really do the large model trainings as well, so.