Latest / Agentic Conversations (formally mlops.community) / ML in Production: A DS from Ubisoft Perspective // Jean-Michel Daignan // MLOps Podcast #151

Transcript

ML in Production: A DS from Ubisoft Perspective // Jean-Michel Daignan // MLOps Podcast #151 · Agentic Conversations (formally mlops.community)

0:00
Arizona should be no data scientist at Reserve.
0:03
I am drinking mostly or chocolate.
0:05
Hello and welcome everyone. We are back with another every
0:08
obscurity podcast. And I am here with none other
0:13
than a be my name is dimitrios on.
0:16
We are your hosts for today and we have the pleasure of talking
0:20
with Jean-Michel, the data scientist.
0:24
Working on the machine learning platform that Ubisoft.
0:27
I particularly enjoyed this. This conversation because a
0:32
zombie cell came from the data science background and he is now
0:39
focusing on the platform. So he has the understanding of
0:43
how data scientists want to do things and what they need.
0:46
And also, he gave a few hot takes on what they need to do
0:52
when it comes to machine, learning platforms.
0:54
But then I really, really liked it because he was very honest.
1:00
About. Hey, here's our journey.
1:02
We're starting from this place. We're trying to bring people
1:06
onto the platform in a very non-invasive way.
1:09
So he talked about Merlin their platform, their SDK that they're
1:12
creating and they're not forcing anyone on the team to use
1:16
Merlin, which I thought was fascinating, because sometimes
1:19
people are a little bit more heavy-handed and then you get
1:22
data scientists who are disgruntled.
1:24
And so, There was that piece of it but then there was the other
1:31
side, which is also just as cool.
1:35
They had the idea of working with recommender systems and
1:40
figuring out a strong Roi of machine learning.
1:44
So they were able to do what we talked about all the time on
1:47
here. Tie back there, machine learning
1:50
products to the business, use cases, and the amount of money
1:55
that it It's going to make the business, so, you know, you're
1:58
sitting pretty, if you can say, your machine learning product is
2:04
making or saving the company millions of dollars.
2:08
So what were some of your big takeaways Abby?
2:11
I felt like he was quite trust her at one of the things I loved
2:15
from this conversation, with his focus on how to build things
2:19
quickly, because this is one thing we talked about a lot in
2:23
the day size World, which you need to sort of have been
2:26
Inspecting delivering value and building something.
2:30
And I felt like he surf clarified in the most simple
2:34
words possible. I also loved one.
2:36
We went on a little tangent and of course with the news.
2:42
Now of course I'm not going to remember the name.
2:44
I remembered it when we were recording what is the name of
2:46
that company? That just got a bunch of funding
2:48
for assets for video games and something like Mario that's it
2:53
so scenario, Gigi got funding. NG and we went on a little bit
2:57
of a tangent on how they're using generative AI potentially
3:02
as a different use case for machine learning and AI within
3:05
Ubisoft. But he was pretty steadfast.
3:08
He was like, yeah, we're really focusing on the stuff that, you
3:12
know, costs or saves. Yeah, it has money.
3:16
It has Roi so that all that fun stuff with generative AI.
3:20
That's great. But we want to like focus on the
3:24
actual money making stuff a lot. If you run your markers for more
3:27
than 24 hours. Yeah.
3:31
She was like yeah shit. I totally forgot that he said
3:36
that so they're watching they're definitely watching their give
3:39
them a lot of leeway but they're also watching those data
3:43
scientists to make sure they don't take too much anyway.
3:47
Awesome conversation. There are some incredible blog
3:51
post that Jean Michel has put out and I I will leave the links
3:57
to these blog post in the description in case anyone wants
4:00
to check them out. We touched on just pieces of the
4:04
blog post. So I encourage you to look at
4:07
them and before we jump into the conversation.
4:11
Abby, you know what we need to do?
4:15
We need to tell people about if he's slack Stars. so, I was
4:24
going to say is No, I wasn't going to talk about the
4:29
sponsors. So before we jump into the full
4:32
conversation, I do want to mention if you find any value in
4:36
the ml Ops Community podcast. It would mean the world to us.
4:39
If you write us, give us some Stars leave a review.
4:43
It's always fun for us to read out, some of the reviews that
4:48
you all are leaving and you make fun of Demetrius in the reviews,
4:54
he'll love it. Yes, that actually yeah I get
4:57
some sick pleasure out of that but the other piece is we just
5:02
ask like if you like it, why not tell one friend just one friend
5:07
about it because the more that this spreads the more ears it
5:10
gets into, I think the more it is going to help the whole
5:13
industry around doing machine learning.
5:16
So that is my ask of you. Hopefully you enjoy it and we
5:20
will see you after the show. No, we won't.
5:24
We'll see you right now. When Get into this conversation.
5:27
So that is my ask and that is all we got, all right?
5:29
I'll talk to you all later. You've got a rich history, so I
5:39
kind of want to go into that. I want to start there with all
5:42
this stuff you've been doing. Before we dive into your most
5:46
recent projects at Ubisoft. Can you just give us the tldr of
5:51
how you got into this business? Yeah, sure.
5:54
I'm To perform this off like 400 years ago.
5:58
Physically integrator transversal team that is working
6:01
on data and Telemetry into L to be low platform, but my
6:05
background is definitely not on machine learning.
6:08
I don't have a computer science degrees, I really related in
6:11
2015, from a master's degree in physics and Engineering was
6:15
really focused around energies in all these things.
6:19
And I started to work in the area of data to apply it on
6:22
smartphone application, Internet of Things.
6:25
That's why I work from different company or Research Center in,
6:29
France from the cea to the EDF. It is like it's a big energy
6:33
supplier in France and I work on the other subsidiaries, you
6:36
know, in England, in in this job, I started to work on of
6:40
course, smart application, like smartphone application like
6:44
forecasting energy consumption and this kind of application but
6:47
I started or soup to work on proof of concept.
6:50
A related more to some aspect like building locomotor system
6:54
for an e-learning platform. Doing some analysis of job by
6:57
data, for nuclear plant analyzing transcript of
7:01
discussion between a client and a customer services.
7:04
It was pretty wide because it basically was contact by
7:07
businesses. Internal part of the business to
7:10
present proof of concept, Based on data machine, learning and
7:13
testing Technologies. So, it was really based on that
7:16
and during my work at EDF, we were doing a lot of a cathode.
7:20
So we were trying also to do, I get on on other company, to take
7:24
a good This has to see the kind of data that people have access
7:28
and can play on, and I participate to work at an
7:30
Ubisoft. So I started to network, we will
7:32
be soft keyboard. And when my contract finish, we
7:35
will my former company. They have a job opening at
7:39
Montreal, and I really wanted to go back in North America, so
7:42
plus like a perfect fit for this job application.
7:45
So, are you? Apply?
7:46
And that's it. You have been riding a turn and
7:50
I love the way that you've been breaking it down because you
7:53
have this background where you're coming into a Hobbs from
7:56
the, the other side we could say where there's a lot of people
8:01
that are in your same position. I think there's a lot of people
8:03
that are thinking about ml Ops, but they're coming from
8:06
different angles, they don't have that traditional software
8:09
mindset. And one thing that I think about
8:13
is, when you are working on an ml platform team, as someone who
8:20
has more of a data science background, what has helped you
8:24
succeed, In this position. So from my best, big diva to
8:30
Beyond partisan and every platform team.
8:31
So it really trying to work with developers like to fit their
8:36
2005 under language understand of tools work.
8:39
Like your kind of a team's work when you test Technologies.
8:42
We work with ML fluid mostly to do for example like a model
8:45
versioning. So I'd like to three years ago.
8:49
We did like an evolution from the data science perspective and
8:52
from the developer perspective and how much to the coast to
8:54
integrate this kind of tool It's, I think it's ready to
8:57
communicate with the people that building the platform and trying
9:00
to find your needs to answer a specific question.
9:04
She needs. What is really important and I
9:07
think also, it's really important to understand the
9:10
landscape. It's pretty may see that there
9:13
is a lot of tech of tools, Etc. So really trying to now it to
9:17
your needs and starting small. I think it's pretty important.
9:21
Just having the basics that can answer.
9:23
Our needs is really important. I will Ali-A like quote, some
9:27
people from Netflix that I met during a conference that was
9:31
seeing when you build an animal platform, you should not build
9:33
it based totally on your data sense needs but on the business
9:37
needs a very so yeah. To to bring the right features
9:41
that can really solve an issue on the business side.
9:44
You mentioned the basics. What for you are the basics,
9:47
like, focusing on. But from my perspective, it's
9:51
having an API call when you want to serve them.
9:54
Others, like it's not too complicated to make the
9:56
integration. That is pretty standard in our
9:58
case, we really feed the stand out of Ubisoft and online
10:01
services of Ubisoft to call our API from the data science side.
10:06
I think it's really having the ability to have shown your
10:08
models look Matrix. When you do some tests or nine
10:11
pipeline, I think it's pretty important and having also access
10:15
to different kind of machine and really dress it up.
10:18
I've been to be pretty easy. You don't need to Select
10:21
executor right models of your machine that you want to have,
10:24
which we just have a small machine on medium machine and a
10:26
large machine. We did make some abstraction on
10:29
Technologies and if new technology come out to just
10:32
update, what's under the hood. And after, you know, you can
10:35
sign in for data scientists, should be like very easy to
10:39
manage on this side. So as you think about your time
10:44
in this position, what are some things that you've done or how
10:48
have you succeeded? Where you feel like Like others
10:51
have failed potentially and succumb, to the downfalls of
10:56
being a data scientist in a software, engineering World, let
11:00
like an Ubisoft or more General though.
11:01
I think more General is what I'm after I want to know.
11:05
So so we'll be very honest with you.
11:07
I'm not really a deep learning guy for me and can receive more
11:11
than four highest risk allocation and like trees and
11:14
more trees, guys, I would say. So I'm really going fast.
11:17
I think on my iteration to be in murders and to be like It was
11:20
generating. I'm more focused on my dad and
11:23
doing like fancy architecture, even it fine.
11:26
If I find it very useful. So I think for me doing a
11:30
prototype it's pretty fast. So I think on that I can like
11:33
I'm not trying to do the best model in the world and trying to
11:37
be like good enough, like just having like some kind of medium
11:41
expectation. Just to really reduce a wizard
11:44
time-to-market even if it's maybe not the right wording on
11:48
that but we need to go fast and drink.
11:50
Prototype pretty fast. I think it's come out from my
11:52
previous jobs where we will be doing a lot of prototype and
11:55
wanted to show like the value of an application pretty quickly
11:59
and having a prototype that can work, very easy.
12:01
So I think for me, it's really not to focus on what is trendy
12:05
like you're doing like Transformer.
12:07
Also. The planning Etc but keep it
12:10
simple. Building a prototype putting in
12:12
production, the important part. And after you can iterate and
12:15
building something more complicated, All right, so you
12:19
said something that I find fascinating, which is trying to
12:22
have the right abstraction so that if anything changes under
12:27
the hood, you can hot swap it out and the data scientists will
12:31
never know the difference when you are building for data
12:35
scientists. What are some things that you
12:38
try to keep in mind? So from our perspective that our
12:43
team has also really what I'm seeing because I am dancing with
12:47
people that buildings are platform and the ui's of bottles
12:50
that we have a top of our platform.
12:52
I think reads to keep this less click possible, for example, in
12:56
our borders that everything is available that nobody's is too
13:00
much constraint, and from the tech side is really maybe the
13:05
low maintenance. What can be luminance to manage
13:08
example? Think it's really what drivers
13:10
using mail flow. It was pretty simple to install
13:13
a pretty low maintenance and it work, pretty great.
13:15
It's not perfect. Like, for example, user
13:17
management of get all the experiment of everybody so it
13:20
can be maybe a little bit tricky in will be, we need some
13:23
Evolution on that. But I think really a I think
13:26
something quick and documentation or so I think it's
13:29
a it's kind of important but even if it's not enough, but you
13:33
need to document because if you have a question of someone that
13:35
is using your platform using, okay, if you want details we've
13:38
got this tutorial. We've got this project and so
13:40
on. Building this kind of a small
13:43
things that can help data scientist to use.
13:45
Our platform is a really important in building.
13:48
This layer is really important it in and I think I think data
13:51
scientist like co-worker user that can give you feedback is
13:53
also pretty important on this aspect.
13:57
Yeah. That that power user is huge and
14:00
I also think about just designing for data scientists in
14:04
general. And what the difference is there
14:08
are because as you mentioned, I think, Its more common than not,
14:13
that a data scientist does not have a software engineering
14:16
background and it's become a meme now.
14:19
And it's something that we always talked about in the mobs
14:23
Community, but how comfortable data scientists are in notebooks
14:28
and then they're so comfortable that they want to bring them
14:31
into production, right? And so when you're thinking
14:35
about these different design principles, do try to bring
14:41
bring software engineering to the data science team or do you
14:45
just say no? That's too.
14:48
That's too hard to work. We're not even getting into that
14:50
as a whole can of worms that workin open and we would rather
14:53
not Yes, sir. As I said, I don't have this
14:58
background in software engineering.
15:00
So for example, are doing you need test integration tests.
15:02
For me, it's pretty new. And I have like some kind of I
15:05
will not sit hot-headed discussion with Dave.
15:08
And my teams that push me to do you need test.
15:10
But I don't see release a point because you will not fix my
15:12
problem. But I think at some point when
15:14
you want to share like function, like for example, we have a lot
15:18
of people that that doing like separate files on their
15:22
pipeline. We will also function to compute
15:24
Matrix lock things. This helper function, if you
15:27
want to share with people, you need to wrap it doing unit test
15:31
because it's helpful document, it help.
15:33
If someone want to modify your code you need to modify your
15:36
unit tests and so on so it can kind of break the deployment.
15:40
So on my side I'd switching from notebook to production is
15:44
definitely you can just like exporting dot by file and put it
15:48
in production like data. I will say it's okay but we
15:51
monitor 8 that the food won't take too much time if people
15:55
want to do Optimization. We show them a sample project.
15:57
We say, okay, we doing like that, I'll feature engineering
16:01
or machine learning modeling. So it's definitely to share
16:04
practices on that. But after we don't want first to
16:08
much like to do, you need tests. And so on, we are more focused
16:11
on testing the scalability of your pipeline and we are AV user
16:14
of spark. And I think, for me, put your
16:16
focus to test that your ETA else on your machine learning by
16:19
paint can and do 10 of 2 times of 10 times.
16:23
The volume expected of, for example, I love players that
16:26
come in on your games. I think it's more important that
16:28
during unit tests on your on your function in, you should
16:31
focus on that. From my experience, you've been
16:35
working in pipelines for quite some time.
16:38
I mean, in your previous role, as well before coming Ubisoft.
16:43
Can you take us through your, the ain't went quite like that.
16:47
You're working on and what are the functions?
16:50
Like, sure. So I know I may be a trapeze.
16:55
I'm really working on the next door recommendation, like
16:59
in-game store augmentation when you want to buy Cosmetics or
17:02
some other element for your game.
17:04
So really my process is really to take the event of telemetry
17:08
because we tracking like the transaction, some specific
17:12
action that ring in your games 33 to take this events, building
17:17
some videos on that building some Target to predict.
17:19
Like, for example, the next item to buy and after that, I'm just
17:24
building a model that can Undo this prediction.
17:26
So I'm using tools like a spark to do all this.
17:29
This modeling bar on Jupiter, like testing some new mothers.
17:32
And after that, I'm going to production.
17:34
So in this case, I'm just putting on my code that need to
17:36
run in a pipeline in my GitHub repository and when I can deploy
17:41
it on an air flow deck, that is running regularly and updating
17:44
the recommendation that ourselves into game.
17:47
So that's I would say the modeling part and after that
17:49
there is a monitoring pipeline part where I'm building a
17:52
pipeline that can just monitor the prediction, evaluate the
17:55
quality give this prediction versus previews prediction in.
17:59
There is also this fact to test your model.
18:01
So I'm doing some A/B Testing. I'm doing a pipeline that can
18:04
split my play population and test like course of treatment
18:07
and the control and doing all the analytics API which it's
18:10
really likely, they are to live in it.
18:13
You have all this morning part around the doing the the data
18:16
processing in the machine learning, which you have also
18:18
tour the discussion with the client.
18:20
That was the features like to give developers that want to
18:22
have this talk Malaysian so you can propose new idea of
18:26
modernization, are you can pop some idea of tests.
18:29
You are also in short to do some aggression on the data side,
18:31
because our data stack is evolving a lot in the pasture
18:35
for your. So you change your data storage
18:38
Etc as so kind of part of all this pipeline I'm kind of Like
18:42
we'd say we for data scientist, enough teams, we are alone to do
18:46
all of that on other team. That using our platform, they
18:49
have analysts that engineer will be a developers that adds them
18:54
to process the event and build the features.
18:56
But on our side we really are. I would select the full stack
18:59
data scientist that doing all this part, but we working with
19:02
close partner, also on the analysis, side to get some
19:05
feedback. And so on, where is seriously
19:07
Allah. And I am rich were people who
19:10
had the intent experience. That's where a lot of people
19:13
don't get, which is like a lot of work.
19:16
I ate think there's a difference now.
19:19
We're more and more data scientists have started
19:21
deploying the models in production by at least
19:24
classically till 2020. What one of the major sins was
19:27
you built the model and then you just give it to the mle.
19:31
Yeah. And now more people are sort of
19:32
going over there. My question on this thing,
19:36
particularly was, obviously, your meat building and
19:39
maintaining your ETFs. What are the major problems?
19:42
Seems that you have think you're doing maintenance for these
19:46
things. We're do things break for you.
19:48
So for me, reads data quality, my main problem and data
19:52
freshness. I think it's human struggle that
19:54
we get because we are dependent of people that collecting the
19:57
event and put it it accessible to us for process it when we
20:02
sometimes we have maybe some kind of blind if there is an
20:04
issue and sometimes it's the ml side that thing up.
20:07
There is something wrong in your prediction.
20:09
You are your accuracy is dropping so you can The people
20:12
say, oh there is something wrong.
20:14
So it really could double-check everything having on the suspect
20:17
to not trust. Blindly your data provider feel,
20:21
it's the most important part. But for my perspective in what
20:24
I'm seeing on those of companies in that you deserved, it is
20:26
pretty good. In terms of data, we've got a
20:29
piece of that office that give a lot of tools like to
20:32
orchestrated yell to collect the event and we do all the
20:36
processing. So I think we have observers
20:39
kind of food that I control, and from the CEO, it's important.
20:42
And for him to have a digital culture.
20:44
So we have plenty of tools, but sometimes you need to do all the
20:47
glue but to connect everything and work.
20:49
Not in a silo, a tree discuss with your data provider and so
20:52
on. So when it comes to the
20:55
organization and especially talking about data quality, yet
20:59
who owns that. Have you taken on the idea of
21:01
like consumers and producers and do the data scientist get pinged
21:06
at 3 a.m.? If the model goes Rogue, how
21:09
does that whole process? Look look.
21:11
So from the side, On the definitely on call because we
21:15
have like Safeguard we've got two fullbacks.
21:17
That is a prediction by default with the most popular items on
21:20
the store. So even if there is no
21:23
accommodation at our updated, you still have something that is
21:26
pretty strong liking for a strong base like so that's okay.
21:30
But after that is really for you to have user right monitoring.
21:33
We don't have something by default left.
21:35
Final people that sending features and your platform that
21:38
detecting. If there is an issue, we've got
21:40
people that working on that for sure.
21:42
Sure. But on my side, I really prefer
21:44
to build some function in our platform.
21:46
We've got something that we could mail in is d.k., that is
21:48
over helper function, that we share with our clients in for
21:52
us. And we trained to put our
21:53
knowledge, in it, that they can be used by data centers, but
21:56
also analyst of people at the beginning of the pipeline in
22:00
term of data processing. But we always trying to build
22:03
our own. I left and I think having this
22:05
mindset to build is on alert system and just trying to show
22:10
what is important for us. It really can help platform to
22:13
evolve like ml platform or data platform in the right way
22:17
because you show what is needed in terms of monitoring because
22:22
For monitoring, you can monitor plenty of things, but what is
22:25
really important is what user really need anything.
22:28
Trying to prototype, your own monitoring is very important on
22:31
the suspect. And so are you monitoring from
22:34
the system level? And then the data level and also
22:38
the model level quality, I'm going to Triangle my data that
22:41
I'm producing and sending to our platform and just checking the
22:44
freshness. Checking a midday something on
22:47
the distribution of the raw data.
22:49
I trained also to check the training set when I merge
22:52
different data sources with pretty small block that I'm
22:54
building. And also for the monitoring and
22:56
have a specific pipeline that is just checking the quality of
22:59
production and so on. So it's really simple.
23:02
I don't I don't have the the pretension to have building a
23:05
crazy thing. Stuff, it's pretty old, sail
23:08
manually. No way are.
23:10
There is no stand up by paying for monitoring, but I really
23:12
building that for the platform after opposite.
23:15
Okay, we need to win it out. This kind of aspect we need to
23:17
Matassa drifting of the Matrix, the freshness of the metrics
23:20
Etc. Because, as I say the platform,
23:24
you sometimes, it's important to reduce the expectation to not
23:27
think that because you have a team dedicated to be low
23:30
platform, they will do everything.
23:32
Out of the box or they will do like what you saw on Tech blog
23:36
manager makes it. Our it's really important just
23:39
to some time taking the true that you get at your disposition
23:43
and I Kings of things and just show what you want it.
23:45
Yeah, it's really important. From our perspective, we're
23:47
doing a lot that in my team's but user of wearing also, I King
23:52
platform. Like for example, we've got on
23:54
one side of an ETL scheduler, that can process all the event.
23:59
And platform that doing another are for instance, that launching
24:03
the bike lane and some of our user just triggering the airflow
24:06
by plane from the material provided processing.
24:10
The event. It's not a features that we are
24:12
allowed, but they can do it and they just like their way on it.
24:16
And I think it's important to give them the Liberty to do it.
24:19
But at some point, would take this needs to really feature of
24:23
the platform to the people listening.
24:25
You're probably going crazy right now because I've done a
24:29
poor job of asking you about what the use cases and what kind
24:34
of scale were talking about what basically can you give us some
24:37
numbers and a little bit of nitty-gritty details on what
24:40
exactly you're doing everything that you can share.
24:43
I imagine there's some stuff that's confidential but break
24:46
that down for us because from what it sounded like, from what
24:50
I call, it's recommender system, but I imagine there's other use
24:54
cases or not from me. I'm walking Muslim locomotor
24:57
system because it's fall. Companies, that will save the
25:01
project that has the most important to hell, why not go to
25:04
nervous at this point because it's pretty simple to implement.
25:07
You get good result when you to summon a disease, but we've got
25:11
client that using our platform to do like, Cheeto detection or
25:15
friend's recommendation in our PC client.
25:18
We've got also, people are doing recommendation but not in store
25:22
in game. Like, for example, just density
25:23
using our platform for their next songs will come under
25:26
today's, like think your use case originally we We were
25:29
really focused on personalization.
25:31
But now, with the mesh of teams, we train to able to like player,
25:34
safety year project of players report project that are bringing
25:37
mayor, other people are in the data office.
25:41
But on my guest all Commander, like it's pretty important to
25:44
have that as a release of a game and because like when we
25:47
releasing an Essence kid games, you need to have talked moderate
25:50
day one. And it's really important
25:51
because it's median of players, but it's not to secret on that
25:55
we get million of players so you can expect to get like me.
25:59
Ian of Welcome addition that need to be computed.
26:01
Maybe they are not useful because like, you know, doing
26:04
some question games, it really depends of the kind of games and
26:08
all this factor, which we need to be ready at some point just
26:11
to to in personalization in your games.
26:14
So, You've got the recommender system.
26:17
You also mentioned that you using air flow as a dag.
26:20
Your scheduling stuff you have that ETL or elt.
26:24
I can't remember which one you said, but I get them confused
26:26
anyway so it doesn't really matter.
26:28
It's so the the thing that I'm interested in though, you
26:32
mentioned you kind of hinted at Merlin.
26:35
And this SDK that you've got and you've built on top, it's like
26:38
an abstraction above all. Love what you're doing below
26:42
which is these Yells. And the airflow is that right?
26:47
Or how does it look not Treaty of giyanti, all seats.
26:50
So developer of our platform that building saw this Vapor or
26:54
function, or its different modules that can help you.
26:57
For example, to send a message via slack to send a message to
27:00
teams to send a message by email.
27:03
So you can visit every message that you want.
27:04
There is also a modules around A/B Testing like to split
27:08
population. If you want to build a new
27:10
players in your pipe and you want to split them when you do,
27:12
we'll be testing if you want to do.
27:15
A significance, significance. Test also have a function that
27:18
we wrap based on our experience. There is a data quality module
27:21
that check the features quality, which will you set of a function
27:25
that can interface with some module of our platform.
27:27
Like, for example, with a villainous Decay, if you want to
27:30
update your fullbacks for your application, you can do it in a
27:34
programmatic way. It's just a simple little better
27:37
function and also you can interface with other elements of
27:40
the online services of Ubisoft. Like, for example, check the
27:43
catalog of items for Also, store in-game store, check what is
27:47
available. I'm Emmanuel Mason machine
27:52
learning engineer, and stripe and author, building machine
27:55
learning, powered applications. And if you don't want your
27:58
machine learning models to explode, you should subscribe to
28:02
this podcast. We have a function that it's a
28:16
wine and function when you can just get this information.
28:19
So it's really more like experience sharing that give you
28:23
the ability to interact with the platform but not only, but
28:27
really for data scientists to do his job at Ubisoft, because we
28:31
kind of targeting to the zoo machine learning platform at
28:34
Ubisoft at some point. So, having this kind of
28:37
interaction with also, observe part of the online world of
28:41
Ubisoft is a really important for us, but you're not forcing,
28:44
the The data scientists to use it or, you know, officer forcing
28:47
anyone to use, it know what we want is for you and you know,
28:50
source project. So people if they want to add
28:53
new function, new modules, they can do it or they open a request
28:56
and they are able to enter in scan of the description with
28:59
developers. Are you should do you need
29:00
tests? Are you should do all these
29:02
things but yeah we've got friends in particular that
29:05
working in Paris Odie. Develop a model around
29:08
collaborative filtering. So really tell out for Commodore
29:10
system but as they bringing some specific algorithm on the
29:14
application, You really want to share it with people and select
29:17
that everybody can can use it, but it's kind of this process
29:22
are really need to have like manager of director to push this
29:27
kind of initiative, led to contribute.
29:28
And at Ubisoft, I think we have a pretty good ecosystem in time
29:32
of machine learning and data scientist that are really
29:35
pushing our platform and people to contribute to it.
29:39
So what it looks like to me is You're Building serve like this
29:44
infrastructure. For other DS, itís the company
29:47
to sorry using to be able to derive more insights and one of
29:51
the major areas of work has been recommendation systems.
29:54
I also wanted to take a little bit of a and talk about the PS
29:58
storage. I'm assuming you would have
30:01
massive amounts of the M9 tireli sure where the odious schema
30:06
looks like, do you use the conventional table structures or
30:10
kind of deer storage systems to use for Cloud platform is really
30:16
structure with S3, we just taking the files from the abyss
30:20
of that ecosystem, just copy them on S3 and having after a
30:24
group that is doing the glue is a files and rebuilding those the
30:28
table. So it's kind of our data storage
30:31
strategy. Currently, we are calling that
30:33
at the beginning of the video store, it's not a feature starts
30:36
with the collection of storage and data base, to be honest.
30:39
But now that we are merging, we was of teams.
30:41
They have some video store. I think I share it presentation
30:44
of one of our Our colleagues, the debile feature store for
30:47
detection and we really want to bring this feature, stop bricks
30:52
on the platform having a decent one on it.
30:55
It's pretty hard work to be honest.
30:57
From the, a cool that I have from people that doing this part
31:00
of the platform to be defeatist or And discussing, we was our ml
31:04
platform on the other companies. Seems that yet the adoption,
31:07
It's difficult for this concept of feature story, but I think at
31:10
Ubisoft, people are starting to understand what we want to do
31:14
and how useful it could be able to share features cross game.
31:17
For example, when we building were all Commodore system on the
31:21
Assassin's Creed games, we using data from previous Assassin's
31:24
Creed, also, just to cover models that can take that in
31:27
account. As having this concept of
31:29
features, driving standard features between games are, it's
31:32
pretty important and I think it's going to be our future.
31:36
I have heard feature stores are a pain in the ass.
31:38
Are you doing gun or if you work with badge?
31:41
Or are you doing color or quit? Stripping Force?
31:46
It's definitely my banner in batch because for my use case of
31:50
Ingham storage batch, it's perfect to all to start but we
31:55
already sent Lee a released the live production video.
31:57
So it's not like there is a streaming then computation of
32:00
features in life but it's more the game client that sending
32:04
information of the Position. And after that, the model is
32:07
processing. This event from building this
32:08
video, on the flight and Computing, but really one of the
32:12
expected to teams are building our feature store data storage
32:16
strategies to have the ability to in life of building features
32:20
are in the pipeline, but it's kind of in the difficult, but
32:23
there is like streaming strategies, attribute software
32:26
on data processing, any. We are part of this discussion
32:30
but it's a, it's a long way. I think to add some point to go
32:33
to the ceiling, but we can still Bring great value by doing
32:37
Bachelor predictions with so it's pretty good at some point
32:40
but definitely it will unlock new capabilities for machine
32:43
learning to ever streaming ability.
32:46
So there is one thing that you mentioned in your recent blog
32:51
post that came out talking about.
32:54
Well it didn't come out yet but I've been privy to see the draft
32:56
and I think by the time this actual episode comes out it will
33:01
be out. So I'm going to act like it's
33:03
come out already and you mentioned that one thing you
33:07
should do when you are working with an ml platform which I
33:11
found. Hilarious is basically lower
33:14
your expectations of what the ml platform is going to be.
33:17
Can you explain yourself a little bit for us?
33:21
Yes, it's all on the suspected sweetie because we've got data
33:25
scientist. And some that are really that
33:27
no, there wouldn't commissioning world's the new technologies.
33:30
And so people are really expecting at some point that the
33:34
main platform will do. Everything out of the box and
33:37
it's pretty difficult. And sometimes we've got like
33:40
needs that are all I want to use this Tech but they don't see it
33:44
to do what and maybe you can do the same thing without the stick
33:48
sometime it. What is important?
33:49
That animal platform, we not bring also taken the world, it's
33:52
not scalable in terms of cost installment maintenance and
33:56
really it's only bring what is really needed because we have
34:00
periodization and so on. So I think for me, it's
34:04
important as an email platform need help or so I decide to
34:08
design what is needed on your platform and get feedback on
34:11
that in a few having power user. It's important enough case when
34:15
we built the machine learning platform teams, we will have two
34:19
data scientist. Always on the platform as a
34:21
power user, but now we have power user also in France, or
34:25
the people in Canada. So, I think having their
34:28
feedback is important, but they should be like, no other
34:31
expectation. And for sure, I think we could
34:35
have been faster on some aspect. I mean like going full managed
34:38
service I like using such maker or end to end and we'll have
34:41
something that can run pretty fast.
34:43
I've been living prediction a few months above extra but from
34:47
my perspective is good to start with that.
34:49
And I think it will quote a real Cooper from the climate up on
34:52
the taxes, on that, you can start with that.
34:54
And after are, you can go more deeper in some custom things.
35:00
But for us as we have 10 of Diaper or so, with the life
35:04
cycle of a video games. So we don't have always or the
35:07
deadlines that we need the features for tomorrow sometime
35:10
you get six months when you have to build it and for example, the
35:13
left prediction, our main client was people that building the new
35:17
Just Dance games. So the deadline was really at
35:20
the reserve a games. I think it helped would take
35:22
time to start with managed service to do buck and after
35:25
switch to a more accustomed way, even if from my perspective we
35:30
should go just full manage but I I'm a data scientist.
35:33
So I just want everything the quickest better.
35:36
It's really a different mindset. How different are the features
35:39
for different kinds of games to painting on.
35:44
I don't know. Like, you might have plenty of
35:47
games. I noticed one of those Tom
35:49
Clancy's one, but I do know a lot of metal.
35:52
Yes, of comes the features. You mean, like the new features
35:55
in the models of the features in the features and the
35:57
recommendation system style particular.
35:59
So I also, once again, I moved like Electra guys, so I'm just
36:04
doing like, for example, on the, on the game, like Rainbow Six
36:06
Siege, you are going, for example, to focus, really on how
36:09
people are using this kind of character in the games because
36:13
like, each character has disabilities.
36:15
So and each character has its own cosmetic, so it makes sense
36:18
to just mix. All people are playing each
36:21
character and just building a conversation with that, we're
36:24
gonna give an Assassin's Creed. There is like, plenty of
36:26
features or the kind of a point that you are using, because
36:29
you're really building your character if you want to play.
36:32
The female of the male character.
36:34
Also it's important in you're in this kind of Butters.
36:37
So it's free to adapt to the data processing in features
36:42
engineering to that, but the logic is kind of the same
36:45
process so that our for specific day, you look at the what has
36:49
been by on this specific day in which is doing join on that in
36:52
just to adapt to the kind of game.
36:54
So for example, on the game like just dance, like the features
36:56
are kind of different because it's a different models and
36:59
different games but to it's Yuri.
37:02
Need to adapt with, I think, what's good?
37:04
Attribute subjects that we've got franchise.
37:06
So, what you, build 41 games, you can be pretty sure that you
37:09
can reuse it on the next games, out of the box.
37:12
Like, for example, on Assassin's Creed ogc in essence, get Vera.
37:16
The processing can be really similar on the way to process
37:19
the event and it really helps you to be able to quickly
37:22
approve of concept that can go in production.
37:24
And after you can iterate and really build a specific model
37:28
for your application. One of the one we're in central
37:32
questions which is do you play games?
37:35
That's why I was definitely less playing games before to come to
37:38
be served but for sure and playing games currently.
37:42
My favorite one is Marvel snap for sure.
37:45
It's a card games that you can play on your phone.
37:49
Definitely it's kind of my drug for the past two months as we
37:52
can see also and building after column that to build better,
37:56
that I bet that you be so very like Assassin's Creed games.
38:00
I really like this idea of historical games like a PG and
38:04
I'm playing the different games. Have a huge catalogue of games
38:08
that I buy in the past. We need to catch up on that.
38:10
I will tell him playing like different game.
38:12
Is it is it a prerequisite to work at Ubisoft that you are a
38:16
gamer Not really, it's not a prerequisite, but I think it
38:21
really helped a lot when you building features that are close
38:24
to the gameplay. I think you need to, like, you
38:27
got to know what a big ready to play some games, to check your
38:30
data checks for tracking Etc, but there is like a lot of data
38:34
scientist that tune playing too much game, like people doing
38:36
machine learning that don't do it too much game, because there
38:38
are more specific aspect that are not ready to the game
38:42
directly. So I figured on that Ubisoft is
38:45
pretty open. It's really based on your own.
38:48
It's a virus treating. One of the papers around a very
38:52
similar thing, which is basically the very a couple of
38:55
deer scientist who were using play data to be able to
39:00
understand clear personalities and to be able to model their
39:04
behaviors. Is that something that you do
39:07
inside Ubisoft as well? And does it help?
39:11
There is teams yet. There is teams are doing this
39:14
kind of analysis at Ubisoft. We've got what we call it.
39:17
Lean user research laboratory that doing this careful analysis
39:20
to detect different. Kind of profiling, the game
39:23
people that doing the coil exploration, that of people that
39:25
doing more crafting. So, there is teams are doing
39:30
this very fond of analysis and what they do it, we AB sometimes
39:34
detect bugs, or so, for example, on some Mission the day,
39:38
something wrong people are stuck and they don't know why, because
39:40
there is maybe some bugs at some points, which I'll show you
39:43
this. Kind of teams are doing that and
39:44
so that office is giving tools for this teams to beat that.
39:47
Golden analytics on that in me and busted that office.
39:51
So I'm helping more on the machine learning side that but
39:54
their input is really important because it can help to reduce
39:57
the feature engineering for a specific models.
39:59
If they show some new trends like for example, a new items is
40:03
really trendy on a shop. You should maybe knowing this
40:06
information on your on your computer system.
40:09
So I wanted to take this in a bit of a different direction.
40:13
Now, I was thinking about how you mentioned earlier.
40:17
Earlier how your tree guy and you like the random forest and
40:21
all that, there is a lot of Buzz right now, as you know is when
40:25
it comes to deep learning and all the diffusion models, are
40:28
the foundation models, and how do you think about or how do you
40:33
juxtapose the two? Like you are now building tools
40:37
for the data scientist? I'm sure some of the data
40:40
scientists that you work with are really hyped on the new deep
40:45
learning stuff that's happening. It and there.
40:47
Probably asking you. Hey what about?
40:50
Can we get some G PT 3 or Chad gbt support for this?
40:56
And then we could build this feature into be.
40:57
So cool that how do you weigh the two?
41:00
Because on one hand you have a very clear use case with the
41:04
recommender systems. And like you said, it's easy to
41:07
calculate the ROI on that and then you have this more
41:12
Innovative pie-in-the-sky type thing with the way.
41:17
In learning, feels like it's going and where potentially, it
41:21
could be in a few years or or longer or shorter depending on
41:26
who you're talking to. So, how do you take those
41:30
trade-offs into consideration? Yellow.
41:33
On this one. We are.
41:35
So I'm not looking for my team's.
41:36
It's more my perspective from that, but it's really, we give
41:40
them a platform with some GPU. If they work, it will be D with
41:46
the wheel of this kind of Detective and this kind of GPU
41:49
and machine on that. But definitely, there is people
41:52
in a new piece of that wound to operate, this kind of models or
41:57
ji bu you can run stable, diffusion on it.
42:00
That's, that's should totally fine if they want to use it but
42:03
at the end if you want to serve you output, you can really use
42:06
the classic online services way. Our API is pretty standard.
42:10
It's user-based. So you can actually serve I will
42:14
say from the classic online services way but if they want to
42:18
have a flask API that running was instituted like running like
42:22
24-hour and so on they can do it.
42:24
I will not advise him to do it because it's good for need for
42:27
the usage. We try to monitor it and we will
42:29
see it, but we are open to use this kind of mothers.
42:32
And it's really more critical, because there is also the themes
42:36
that working on this aspect on generative AI.
42:39
Yeah, I can leverage. That is kind of our end that is
42:42
doing the link between the Academia and Industry it working
42:45
on this topic and the You want to put a type things and maybe
42:49
at some point we can switch to us are not little.
42:53
So like someone that doing a lot of talk around NLP at left for
42:56
school, Benson syringe is in guy.
42:59
Build some interesting tools. And yeah, there is different
43:03
needs on the suspect in for show, investigated that and how
43:07
we can integrate it. But there is no I will say a
43:10
real use case to be put in production yet on the suspect.
43:15
So it's really for us putting in predictions.
43:17
Something. So it's just like building
43:21
something that doing some drawing in the games.
43:24
It's maybe about what we want to tackle, but Wu knows if there is
43:26
some ideas around that that can be cool.
43:29
Our platform can be open to that but we need to hear this case
43:32
around. Yeah, I was thinking about
43:34
because I've seen a lot of these stable diffusion Photoshop
43:38
plugins, or blender plugins. And if some of your artists or
43:43
if people working on the creative side and I think I just
43:46
saw What is it scenario? .G G, they just got a bunch of
43:51
funding and they create assets for games and they're using.
43:55
I think they're using stable diffusion on the back end, but
43:58
don't quote me on that one and yeah, I think it's the right
44:01
decision. Yeah, so it's almost like I was
44:03
just thinking about how you have this recommender system, which
44:07
is one that is. Again, like you said, it's
44:10
almost like more of the traditional ml in quotes and
44:15
then you have some of this other stuff and especially When it
44:17
comes to gameplay and the mapping out of new characters
44:22
are new assets and trying to make that process faster.
44:26
I was wondering if you have thought about servicing the
44:28
needs with some of the generative Ai and what you're
44:33
thinking about, when you're creating the data platform or
44:36
the ml platform that will help those teams.
44:41
So, you know, because if you need to see, we are more Focus
44:44
to what can be served at the play Oliver than more Ado, tools
44:48
that can help you build video games.
44:50
It was of teams that working on this aspect, there is a good
44:55
team into Technology Group. That is another part of Ubisoft.
44:59
I think there is a paper that we build around the detecting bugs
45:02
in your code that you that you push Etc.
45:04
So they are more focus on this aspect but it's that kind of the
45:08
foggy area or you can use the name El platform and a kid.
45:11
For your needs are on this aspect but I'm on date at some
45:16
point. Should be the kind of the email
45:17
platform that we refocus first at the player level, what we can
45:21
do to the players so it could be game.
45:23
Played could be around monetization on the other
45:27
aspect. But to after that, bringing new
45:29
use cases for the up to the to the developers that want to have
45:33
these features available in. We need to adapter on that.
45:37
That happens when you start getting some of the characters
45:41
to be powered by chat to be T and you can have conversations
45:45
with them. Yeah, will be the fun part.
45:48
So do this has been awesome. Abby, did you want to mention
45:51
anything before we sign off? And this is quite interesting
45:56
Becky's. You have a very clear use case
46:00
that you've been working on. There's so much group of the
46:03
number of things that you can do at a gaming company and because
46:07
I was sort of like, interviewing at unity and be high.
46:11
Peace, very similar teams, they have like a team working art and
46:15
on Emily jeans and stuff. But to be able to serve the
46:22
customers, one of the things I wander on that side is about the
46:28
volume of the data that you have, and how much data do you
46:34
keep in Courage? And when you decide to discard a
46:39
particular data pick, He is any person that's playing the game,
46:44
will more likely keep on playing the game.
46:47
So when you decide, okay, now this data is not relevant for
46:51
us. Good question.
46:54
So definitely for sure if a player doesn't want for us to
46:58
keep the data, we've got all the GPR registration so they totally
47:02
out of of detail on this aspect. But it really part of the
47:06
mothers that you are building. Like on my case, I'll be needing
47:10
a recommendation based on the data on the last seven days of
47:13
transaction because I think it's too good head of about ticking
47:17
in finding the right like tendency on the on the last week
47:22
of that. And I'm always only updating
47:25
accommodation for people that play on the last month or last
47:30
three months have to limit the cost because we saw that with
47:33
this approach, we can basically catch like a big amount of the
47:37
transaction, it in, it's good enough for us.
47:40
So it's really depend on them of cost and we currently trying to
47:43
see it because as you know, like with the recession and so on, we
47:46
need to reduce cost. It's very important for us.
47:49
So we trying to also to monitor our or user of our Form using
47:54
our platform in how much of accommodation deluding for so
47:57
bad sharing and maybe ask them after a few weeks.
48:01
A you should maybe reduce because we see the number of API
48:04
calls. We sing how much Rose you
48:06
updating. It doesn't match really.
48:09
Also you should maybe reduce in, trying to apply some strategies.
48:12
And we're trying to show the good example like to soak it in
48:14
your project. This is our best practice and
48:17
it's very late by all these clients.
48:20
So maybe you should follow it in terms of of cost saving
48:24
Excellent man. So this has been so cool.
48:28
Thank you so much for coming on here, and thank you for writing
48:31
these blog posts and being so transparent with your journey
48:34
and what you all are doing at Ubisoft.
48:36
I know I've appreciated a ton because you get to see the real,
48:41
basically, the real process of how someone goes about building
48:47
a machine learning platform. And so that not a lot of people
48:51
are sharing and I and I thank you for that.
48:56
It's my pleasure and I really hope that we can continue to
48:58
discuss about what we are doing. Because we have the means of our
49:01
teams in Paris. And in Montreal laughing, the
49:03
doing pretty cool stuff and they want to publish it outside or
49:06
so. Yes.
49:07
And for anybody, that was wondering, I asked John michell,
49:13
what something he is proud about is, and, you know, some people
49:17
say there, they had a child, a very proud about that or I
49:21
climbed Mount Everest, he said, Said, I've been running multiple
49:25
machine learning pipelines in production for the last four
49:27
years. I don't know who it's so it's
49:31
not all fun and games when you're too in this machine
49:34
learning production stuff yet. Yeah dude I love that.
49:39
I had a nice laugh when I saw it.
49:41
So thanks again man. It's been great.
49:44
Having you, thank you. Hey I'm Vishnu.
49:51
I'm a data scientist at first hand and I definitely think that
49:55
you should subscribe to the ml Ops coffee sessions podcast.
49:59
It's the best podcast out there to stay on top of what mlms
50:02
actually is to talk to the true thought leaders in the space.
50:05
And oh, by the way, dimitrios is absolutely hilarious.
50:09
What a weird guy, you should definitely subscribe to the
50:11
podcast.