Latest / Agentic Conversations (formally mlops.community) / MLOps #28 Continuous Evaluation & Model Experimentation // Danny Ma - Founder & CEO at Sydney Data Science

Transcript

MLOps #28 Continuous Evaluation & Model Experimentation // Danny Ma - Founder & CEO at Sydney Data Science · Agentic Conversations (formally mlops.community)

• 0:01
  So without further Ado, I'm going to say welcome to the
• 0:04
  Meetup. Danny ma thank you for joining
• 0:06
  us. It's been a long time coming.
• 0:08
  So, very happy to have you here. Thank you so much for having me
• 0:12
  to meet. Yes, it's been really a pleasure
• 0:15
  to just kind of have a chat with you and also to share some of my
• 0:21
  experience with everyone who's watching.
• 0:24
  I hope you guys can get something out of it today.
• 0:28
  If anything, I just want to demonstrate Demonstrate how all
• 0:31
  you really need to be relatively successful in this field is just
• 0:35
  to have a lot of passion for what you're doing.
• 0:39
  And to kind of stick through the, there's going to be
• 0:42
  sticking points, there's going to be struggle, there's going to
• 0:45
  be really difficult things to try and Tackle but just have the
• 0:49
  great mindset and persistence to go through it trust in the
• 0:53
  process and you will definitely make it.
• 0:56
  Oh, that's awesome. Yeah, the inspiration that I
• 0:59
  needed? My Thursday morning.
• 1:02
  So of course, let's jump into it.
• 1:06
  And I just want to ask first off, how did you get into data
• 1:10
  science? And then how did you make the
• 1:12
  transition over to machine learning engineering?
• 1:16
  So yes, one of my one of my tag lines before was that I've had
• 1:21
  pretty much almost every single data role in the industry.
• 1:26
  So Gone In from when I was doing internships, I was doing data
• 1:29
  The entry. So I was manually just typing
• 1:32
  stuff into Excel. Then one fun right?
• 1:34
  Yeah. Oh it was it was amazing.
• 1:36
  My typing skills went up but not much else and then from there I
• 1:40
  kind of I joined as a data analyst as a graduate I started
• 1:45
  working on campaign analytics did a little bit of predictive
• 1:48
  modeling. Did some did digital analytics
• 1:52
  worked really closely with great dbas and daughter engineers?
• 1:56
  Then I started working within a proper data.
• 2:00
  And for all I was doing machine learning predictive modeling and
• 2:05
  then once I built my first models then I was also tasked
• 2:08
  with putting them into production and that was my first
• 2:11
  glimpse into some of the machine learning engineering techniques
• 2:15
  and some of the tooling that we use currently and yeah and in my
• 2:20
  latest role I've been I've moved over into the real dark side and
• 2:27
  yeah I've just been working fully on the cloud.
• 2:30
  Working with the data scientists to try and production are some
• 2:32
  of their models and also working with in a very challenging data
• 2:37
  engineering space as well. I think, yeah, a lot of people
• 2:42
  usually think of machine learning, engineering and data
• 2:44
  engineering as kind of two separate beasts, sort of, but
• 2:48
  then once you work on larger end-to-end projects, you see
• 2:52
  that they go hand in hand, you almost need knowledge of both to
• 2:56
  be able to deliver things end-to-end.
• 2:58
  That's such a great point. And that is something that we've
• 3:02
  had Dan on here. Another, Dan not you, I'm news
• 3:09
  that he, but he danced, Sullivan.
• 3:12
  He talked to us basically about how interesting it is, and how
• 3:17
  important it is for a data scientists, who know a bit of
• 3:21
  data engineering and have that like tool in their tool kit to
• 3:25
  be able to draw on when they need it.
• 3:28
  And what he said that I I thought was really nice was that
• 3:33
  it's like having, you know, going from a hacksaw to a
• 3:39
  chainsaw when you know, that kind of information and you can
• 3:43
  have a bit of a better idea of what did engineering can give
• 3:47
  you, it will give you that kind of that power up in that level
• 3:51
  up that you can look forward to. So, I love everything that you
• 3:57
  say, I want to let you get into it, I know you.
• 4:00
  Some stuff for us before you jump into that, though, I just
• 4:04
  wanted to ask about this production izing and when you're
• 4:11
  going into the different tooling, are you working mainly
• 4:14
  with like, third-party tools? Are you building it all
• 4:16
  yourself? How does that look?
• 4:19
  Yeah. So am I my first foray into
• 4:23
  machine learning engineering within the bank.
• 4:25
  A lot of the tools that currently we use extensively
• 4:28
  like the The cube flow, ml flow, all those things didn't really
• 4:32
  exist back then, so we would be manually built a lot of those
• 4:37
  two lines within the bank, which is really fun.
• 4:39
  I think. Yeah, I was one of the lucky
• 4:42
  ones who had the opportunity to actually go from like almost
• 4:46
  Ground Zero and build everything from scratch with in there.
• 4:49
  Wow. And it was it was quite fun
• 4:53
  quite challenging because I personally didn't come from a
• 4:55
  software engineering background so it was super challenging for
• 4:59
  me. I Picked up nearly everything
• 5:00
  that I know. Now, on the job, I had my
• 5:04
  education was in Actuarial studies.
• 5:06
  So mix of business and statistics, so had a reasonable
• 5:10
  grasp of the maps and the different, like some, some
• 5:16
  thinking probabilistic thinking and things like that.
• 5:18
  But outside of that, I didn't really have much more to offer
• 5:23
  from from a technical software perspective.
• 5:25
  I like to think that I'm okay at solving problems.
• 5:29
  Oops, so every time I look at a code thing, it's literally just
• 5:32
  so, how do I solve it? Where do I find the information
• 5:35
  to solve it? Oh, I should.
• 5:38
  Oh, I can ask people who have done this before, who are doing
• 5:40
  it at the highest quality from that.
• 5:42
  Seems like a good idea, which is why we're here with you today.
• 5:48
  I'm fine. So again, that can only share
• 5:52
  what? I know.
• 5:53
  I don't think I'm the best at what I do.
• 5:55
  But I I love teaching others. So hopefully that comes out the
• 6:00
  Nice know. That is an incredible statement
• 6:03
  before Cube flow and ml flow were even widely adopted you
• 6:08
  were building your own tools and that had I'd say it's probably
• 6:16
  stretching it to say we're trying to build the entire stack
• 6:19
  or anything but uh I think we were we built most of the data
• 6:25
  ingestion, working with different systems, optimizing
• 6:28
  the sequel and running. I mean, all that sort of stuff.
• 6:31
  The feature engineering, we I think the first models that I
• 6:34
  built the way used in production were actually done in our, so we
• 6:38
  dock arise that deployed it on one of the servers.
• 6:42
  And there wasn't there wasn't any monitoring proper
• 6:46
  monitoring. So we would literally just
• 6:48
  looking at like big Grand tabs, making sure that nothing was
• 6:52
  failing. Well, hacking around the
• 6:54
  networks, actually send you an e-mail, something went wrong?
• 6:57
  Yeah, it was, it was good fun. All the Mike, the dirty, the
• 7:01
  dirty hockey work. They can see when some of the
• 7:05
  machine learning teams are starting to think, about how to
• 7:08
  production. As some of the things I've gone
• 7:11
  through, most of it and she'll come out, the other end, shout
• 7:15
  out to anybody still using our. Oh yeah, ours are is great.
• 7:21
  I, I always suggested people who come from an analytics
• 7:27
  background or a more familiar with.
• 7:29
  Data and Analysis, maybe statisticians as well.
• 7:34
  I actually went python first because I thought when I was
• 7:37
  studying, everyone said All You Need To Learn Python because
• 7:39
  pythons pythons the best and then I'm like, okay cool.
• 7:43
  I learned the python. So I start aligning the python
• 7:46
  then in my first, actual data science roll.
• 7:49
  Everyone was using our that was the only thing that was
• 7:51
  supported in the bank. So yeah, we go back and learn
• 7:55
  that. Yeah, I was, it was fun.
• 7:58
  I found after learning one thing.
• 8:00
  I would just keep translating back and forth between the two
• 8:04
  languages, and then every now and then, when you use a lot of
• 8:07
  data, you just always like well II, usually default back to
• 8:12
  sequel. So all the group buyers and all
• 8:16
  those other window functions, they're mostly the things that
• 8:20
  are in other languages are just translated versions of the SQL.
• 8:23
  So yeah. Awesome man.
• 8:27
  So now can you walk us through? What you have prepared and I
• 8:32
  know you've got some cool Tech going on on your side yet.
• 8:35
  It's you've got some great screen sharing.
• 8:37
  Look at that. How cool is that?
• 8:39
  There he is fried on. Okay, all that, I'll let you
• 8:43
  have it. All right, as you're so awesome.
• 8:48
  Thank you so much, thanks for everyone who has joined us.
• 8:53
  So for today in the Meetup I thought I'd try and make some
• 8:56
  slides. So it feels like a real meet up
• 8:59
  as I'm presenting, but feel free to jump in with any questions
• 9:02
  and I mean Demetrius will kind of have like an ongoing dialogue
• 9:06
  whilst I'm presenting this. So think of it as almost like
• 9:09
  we're doing a knowledge sharing where this is one on one but you
• 9:12
  guys are watching. So I think for today What I
• 9:15
  wanted to focus on one of my passions, especially around the
• 9:19
  data in the machine learning is experimentation.
• 9:22
  So I wanted to help you guys understand from an ml Ops
• 9:25
  perspective and just more in general for the machine
• 9:28
  learning. What do we mean by experiments
• 9:31
  within this space? And then from there kind of
• 9:34
  tires on to the Glorious Target variables.
• 9:39
  So, most of my experience has been in supervised learning.
• 9:43
  I like to think that most of my, Pieces in that space as well.
• 9:47
  So I'll talk through some of the ideas that you can use that.
• 9:52
  Most artists scientists will be using to generate their target
• 9:54
  variables and how it has an impact especially from an ml Ops
• 9:58
  perspective. What Downstream impacts does it
• 10:02
  have what sort of Upstream considerations should you also
• 10:05
  have when you're trying to generate your target variables
• 10:08
  from the data? And that also then flows into
• 10:12
  your business applications of your machine learning.
• 10:15
  Models and how you would run actual experiments in
• 10:19
  production. Or even when you're doing it
• 10:21
  before production, you're trying to do your proof of Concepts,
• 10:25
  we'll try and keep it really relevant and simple with very
• 10:28
  simple ideas. And I think we're going to try
• 10:32
  and do like a prediction of someone who purchases something
• 10:36
  that's versus someone who doesn't so nice and nice and
• 10:39
  simple. We can also add live and have
• 10:42
  different. What was that sir and you Useful
• 10:45
  also, Mmm. Yeah, I think, yeah, if we can
• 10:50
  try and focus on those problems which cover like 80 to 90% of
• 10:54
  what we do in the industry, it will give you guys a much better
• 10:57
  idea of how you can apply it to these problems which will most
• 11:01
  likely show up at work. And then kind of give you some
• 11:06
  build up your intuition for how you can apply it for different
• 11:08
  problems. Great.
• 11:10
  So that's that's the summary for today.
• 11:13
  The cliff notes will probably summarize this in like a tldr in
• 11:18
  the post or something. So now let's talk about
• 11:21
  experiments for ML Ops. It doesn't have to be ml Ops.
• 11:24
  It could just be like machine learning in general.
• 11:29
  So when we think of ml Ops experimentation, you have your
• 11:35
  ml flow and Cube flow and some of these Technologies where you
• 11:38
  can They've got inbuilt experiments where you can run
• 11:42
  with different hyperparameters. You can also try different
• 11:47
  algorithms, as well. Maybe you will have different
• 11:52
  processes for doing your feature engineering.
• 11:56
  And, of course, you can also change your input variables of
• 12:00
  the labels and the target to determine what you're going to
• 12:05
  predict and what your machine learning algorithm will actually
• 12:09
  And so that the hyperparameters, it's not sure if you guys are
• 12:15
  familiar with all the different ways that you can, choon, your
• 12:18
  machine learning models. But essentially you have your,
• 12:21
  you have grid search where you manually input some different
• 12:25
  values in a grid and your algorithms can kind of run
• 12:29
  through each one train those models and then you'll do model
• 12:34
  selection by choosing the best, the best model with the the
• 12:37
  lowest error. Or the highest Roc or different
• 12:41
  metrics like that. In the same way, you can also do
• 12:45
  random search so very similar to The Grid, but you initialize
• 12:50
  with random points. So, if we think of this as the,
• 12:52
  the Grid or sorry, I didn't realize that my head is blocking
• 12:56
  that box. I'm going to do so say there is
• 13:02
  this grid here and if you do the grid search you kind of get
• 13:07
  these dots in between so they're very defined in terms of the
• 13:11
  search space and you might you if you're lucky enough, you
• 13:16
  might find the local minimum or maximum where you trying to find
• 13:20
  that best model but probabilistically.
• 13:23
  It's more likely going to be in these random areas It generally
• 13:28
  Won't be in those set hard criterias for the
• 13:31
  hyperparameters. So what we like to do in
• 13:34
  practice is actually kind of Imagine This is the grid and
• 13:38
  will just randomly kind of do this and computationally.
• 13:42
  It's relatively cheap because you can parallelize it quite
• 13:46
  well. So in general we would we would
• 13:50
  typically go with random search as your best like your Occam's
• 13:55
  razor, the simplest Choice, what you?
• 13:58
  Could do all soon. So we can also do, I think in
• 14:03
  some of the Technologies, now we have Bayesian hyper parameter
• 14:07
  tuning as well. Where it kind of does a
• 14:10
  sequential search for the best set of hyperparameters and each
• 14:17
  subsequent Run for the hyperparameters will search
• 14:21
  within a space which has a higher likelihood of having that
• 14:24
  maximum or the minimum to optimize Model.
• 14:29
  So I can probably talk about this for a bit longer, but in a
• 14:31
  nutshell, had Proprietors. And the way you choose them, you
• 14:35
  can you can think of them as experiments.
• 14:37
  So they different methods as well as the different values
• 14:40
  that you use to optimize. And in the same way, we can also
• 14:46
  do it for algorithms, so you guys must have heard of your,
• 14:50
  any of the deep learning the tensorflow that care ass, D, PT
• 14:54
  3 or different, things like that.
• 14:57
  I don't know if we can use. That within the thing anyway,
• 14:59
  but there's there's all these different algorithms that you
• 15:01
  could use for your machine learning tasks.
• 15:04
  There's deep learning you could have just your regular logistic,
• 15:07
  regression something simple. And any of the other operators
• 15:11
  that we have within those machine learning Frameworks, we
• 15:15
  can also use. So maybe if you're trying to
• 15:19
  predict a certain problem, you can just swap out some of the
• 15:22
  these different algorithms and compare them and you choose the
• 15:26
  best model that pops out the other end.
• 15:29
  And now, when you think of the feature engineering, usually
• 15:33
  machine learning will always have the input data.
• 15:37
  And we use that to learn the different relationships between
• 15:41
  that data input. And the target variable, we can
• 15:45
  also update and change and experiment with different types
• 15:48
  of features and information that we're passing to these
• 15:51
  algorithms, so we can have different steps within the
• 15:56
  machine learning pipeline too. Generate these features I'm not
• 16:01
  familiar with. Oh, I'm not sure who's familiar
• 16:03
  with the different pipelines, like sklearn or spark a live or
• 16:09
  different things like that. But you can definitely create
• 16:12
  different pipelines to create your features so you can
• 16:15
  experiment with maybe you might have a call it feature set one,
• 16:22
  where you just have really simple features and then maybe
• 16:24
  feature set to you have things where you might have Emil
• 16:28
  combinations or various different things and
• 16:32
  Transformations log Transformations.
• 16:35
  Maybe add some features together, all these different
• 16:38
  things that you can experiment with.
• 16:40
  These can also be considered as experiments with in machine
• 16:43
  learning. Can you go into work?
• 16:44
  A polynomial combination is a bit more.
• 16:47
  I yeah. So that one that literally just
• 16:50
  like came off the top of my head but it's think of.
• 16:54
  So I'm going to draw down here. So, I draw down here so imagine
• 16:59
  you have your features, you might have.
• 17:03
  Let's go column A B C D. So if we're going to do like a
• 17:08
  polynomial transformation let's just say we're going to do a
• 17:12
  quadratic for example. So maybe if you're going to do
• 17:15
  that you might have a feature which is a squared B squared, c
• 17:21
  squared, maybe D scope. Maybe you have other things like
• 17:24
  a be so combinations of the two. AC ad Etc.
• 17:31
  Maybe you have cubics as well so you can have a cubed but these
• 17:37
  things oh sorry I drew off the screen but these things it's all
• 17:42
  down to experimentation some of these work with different
• 17:45
  algorithms in different ways as well.
• 17:48
  If your features are still correlated it still doesn't
• 17:51
  work. You still want uncorrelated
• 17:53
  features when you're doing your machine learning.
• 17:56
  So it's a, there's I think this is more of like the art
• 18:00
  component of machine learning. Hmm.
• 18:03
  It could you give us like an example of this in the real
• 18:07
  world? We could say, when you're
• 18:08
  actually doing it and putting it into practice?
• 18:10
  Yeah, Mmm. So the one that I've seen like,
• 18:14
  the textbook example of this is trying to do and like a house
• 18:17
  price prediction. I'm trying to remember the exact
• 18:22
  example, but let's say you're trying to break Boston house,
• 18:25
  prices something. and, They could try and multiply some of
• 18:31
  those features together to try and get a better representation
• 18:35
  at different points of the space.
• 18:37
  For example, maybe some of those really expensive houses.
• 18:43
  If you multiply out the number of cars that you have in the
• 18:46
  garage, how many toilets you have all of those things in some
• 18:51
  sort of combination in, might provide you some more insight
• 18:55
  and you kind of get these I'm trying to, I'm trying to find
• 18:59
  the right word for it would be kind of like nonlinear data as
• 19:07
  opposed to just looking at the number of rooms, the number of
• 19:11
  toilets, the M2 of something trying to get different
• 19:14
  combinations. Might give you a better
• 19:16
  understanding of the space. And this is in a nutshell, what,
• 19:20
  some of those deep learning algorithms do anyway, so kind
• 19:23
  of, look at all those different combinations and blow it up into
• 19:26
  higher Dimensions. So, Can learn more from it.
• 19:29
  But the downside is there a little bit harder to explain and
• 19:32
  understand exactly what's going on?
• 19:34
  Yeah, yeah. So think of this as like the,
• 19:38
  the manual process, which most of those deep learning, all the
• 19:42
  different layers, kind of do that in this nonlinear space as
• 19:47
  well. Awesome, thank you for the call.
• 19:52
  That's right. Yeah, so features are very
• 19:55
  important. So of course they're the thing
• 19:57
  that your machine learning classifier, learns from this.
• 20:02
  So let's say for example we have these ABCD features maybe in
• 20:07
  another experiment. You might have features from
• 20:11
  like e through to Z or Z. What do you guys say Z?
• 20:16
  Was it. I figured we're done because I -
• 20:19
  yeah. Yeah, but for example, you can
• 20:23
  try different experiments with more or less data.
• 20:26
  Maybe you want to try and run other examples where you're
• 20:30
  going to remove some correlated features.
• 20:32
  You run some other processes to remove the those correlated
• 20:36
  features. Maybe you remove some with lots
• 20:40
  of null values or things with high.
• 20:42
  Cardinality you can try one hot, encoding versus log odds and
• 20:47
  coding versus Has I think there's another type of encoding
• 20:51
  using Fisher information criteria, split which is pretty
• 20:55
  cool as well. But yeah there's all these
• 20:57
  different things that you can do to try and massage your data
• 21:01
  into a different group of features that you feed into your
• 21:04
  machine learning algorithm and traditionally, this is what data
• 21:10
  scientists would be doing but now that we're going into a
• 21:14
  world where you kind of yes, your data scientists can kind of
• 21:17
  live in a sandbox and Do all this stuff and and find the best
• 21:23
  thing that what from that small blast radius proof of concept.
• 21:28
  But then the question would be how long will it take to put
• 21:30
  that into production? And it's from from my
• 21:34
  experience, it's kind of we're moving towards a space where if
• 21:38
  you have all your data in one place and in the right
• 21:40
  production pipeline, all your sandboxing and your
• 21:43
  experimentation should be done within this production
• 21:46
  framework. So then if you need to really
• 21:49
  Get into production. You almost just need to flip a
• 21:52
  button, turn it on and then we'll start running.
• 21:56
  Yeah. Completely like what you would
• 21:58
  expect it to do, right? You don't want it to be, you
• 22:01
  don't want to be held up by this, because you have to change
• 22:04
  everything from one environment to the next or it just doesn't
• 22:10
  seem efficient that way. So having it be, where you're
• 22:15
  doing all of this, with the end game in mind, It seems like the
• 22:20
  right way to go about in it's logically and from an efficiency
• 22:26
  perspective, there's, you can't really argue against it.
• 22:30
  The only issue is probably at the, in the current scenario,
• 22:34
  not too many companies have the maturity to actually have all
• 22:37
  their data in one place. Have their data engineers and
• 22:41
  machine learning engineers and data scientists working together
• 22:44
  with the business to actually get all these business processes
• 22:48
  to Read off some of these machine learning outputs.
• 22:51
  It makes it really difficult to actually tie it all together.
• 22:55
  So I think it's a maturity thing, it'll come in time that
• 23:00
  and that I think we've heard so much on this Meetup.
• 23:04
  And so I just want to kind of derail you real fast.
• 23:09
  We'll get right now because this yeah.
• 23:12
  Is there some things that you've seen that?
• 23:16
  You could say the maturity is not there because of this or
• 23:23
  that like how Can if I'm a company, how can I become more
• 23:27
  mature? Mmm, so I have to think of my
• 23:33
  words carefully in case I inflamed someone I think a lot
• 23:39
  of the time where I've seen some businesses really struggle with
• 23:43
  this on one hand, usually that really large.
• 23:46
  So there's just lots and lots of silos.
• 23:49
  The the production data means different things, for different
• 23:53
  people, for some, it could In lives in Excel, spreadsheets
• 23:56
  that get delivered on a regular Cadence for others.
• 23:59
  It could be, it's only spark for others.
• 24:02
  It's only Kafka or streaming or whatever.
• 24:05
  So there's I think it's a little bit of a mismatch in between the
• 24:10
  levels of data more like, it would be data.
• 24:16
  Engineering know-how across the different groups of within the
• 24:21
  business and there's usually For large businesses.
• 24:25
  There's never really one overarching strategy.
• 24:29
  You see? Lots and lots of like each.
• 24:32
  Each group will have their strategy, but there's nothing
• 24:34
  that really ties them all together and even if there is,
• 24:37
  that's usually like a huge transformational thing that just
• 24:40
  takes years and years. And by the time you get close to
• 24:44
  delivery who knows, like maybe the leadership has changed and
• 24:47
  the new guys that come in, or then they could just change
• 24:50
  everything because they don't like it.
• 24:52
  Yeah. And yeah.
• 24:55
  It's kind of crazy when you think of it and from like a cost
• 24:59
  perspective because these these processes cost money like huge
• 25:04
  amounts of money. There's all this, you see the
• 25:06
  cloud transformation. The just moving from one
• 25:08
  provider to another. Even on premise is they usually
• 25:12
  multi-million dollar projects, like two to three years time and
• 25:17
  they're just massive monoliths. Just moving very slowly as well
• 25:21
  but they're difficult if you have lots of data and yeah.
• 25:25
  Yeah. And I thought it was insightful
• 25:28
  to how you said not only is it the teams like these data
• 25:32
  science, teams data, engineering teams, but also the business
• 25:36
  side to everyone needs to be looped in, right?
• 25:39
  So that you're getting the right information, you're getting the
• 25:42
  right data, and you're actually solving the right problems.
• 25:47
  Mmm. Yeah.
• 25:49
  Almost 99 times out of 100. When if you've seen some machine
• 25:53
  learning project that misses the Mark it's nearly always down to,
• 25:59
  they weren't solving the right problem or that they're trying
• 26:01
  to build a model for something that they couldn't tie it back
• 26:04
  to the business objective or they could have measure
• 26:07
  something. It's always the case it's never
• 26:10
  really about oh, your machine learning model was off by Point
• 26:14
  1 AZ 0.18, you see is quite Judgment .00, 180c or whatever
• 26:20
  it is, it's always down to how you apply it and just their
• 26:24
  know-how and experience to kind of avoid some of those.
• 26:28
  Pitfalls is the most important thing.
• 26:31
  It's awesome. All right.
• 26:32
  So a little bit of a tangent there, I'll let you get back to
• 26:35
  what you were talking about. But thank you all for
• 26:38
  entertaining by my questions. Okay, so we're still talking
• 26:44
  about experiments for the machine learning.
• 26:47
  So we talked about the hyperparameters, the the
• 26:50
  algorithms, as well as the features that you can experiment
• 26:53
  has. But one of the most important
• 26:55
  things that you should be experimenting with is what we
• 26:59
  call the labels or the target variable and this is purely
• 27:04
  within a supervised learning space.
• 27:06
  But if it usually you think of this as the, the column that you
• 27:10
  do not share with your Learning classifier.
• 27:13
  Because if you give it to it lines exactly what it is.
• 27:17
  We call that feature leakage, label leakage, but usually
• 27:21
  you'll have a data set that looks to be like like this, this
• 27:24
  is all your columns in your rose.
• 27:27
  And what we want to do is probably hold out that final
• 27:31
  column or whatever and you can you can think of that as your,
• 27:37
  your why, you know, in the maths when people go like there's a
• 27:41
  model that has X. And then you try and fit the
• 27:44
  model, the f of x equals the Y. And then you try and build that
• 27:50
  mathematical model to try and make that prediction in the same
• 27:53
  way, the machine learning is doing exactly that.
• 27:55
  Where it sucks in visting and then tries to predict this
• 27:59
  thing. And that's that's like machine
• 28:03
  learning in a nutshell. That's all it really does.
• 28:06
  Yeah, it's not hard. No, that's totally
• 28:09
  understandable. Yeah, I think, yeah.
• 28:13
  Maybe too many people think of machine learning, is this
• 28:15
  amazing Ivory Tower thing where you've got like crazy people
• 28:19
  hacking up all sorts of things. It's really difficult.
• 28:23
  I think for this community for the ml Ops Community, it's
• 28:25
  probably less like that but it's definitely not a science really
• 28:29
  has this mythical All nature from probably on LinkedIn and
• 28:35
  all these different places where people just think data
• 28:37
  scientists are really amazing. I'm here to almost debunk that
• 28:41
  and to say not this is you can anyone can learn the stuff you
• 28:44
  just you just want. You just need to want to learn
• 28:48
  it. Yeah, it's a systems goes a long
• 28:51
  way. Yeah, definitely.
• 28:54
  Okay so now I'm going to talk about how you can update and
• 28:59
  change these labels. And Target variables.
• 29:04
  This might be a little bit abstract but bear with me.
• 29:07
  But when we think of the target variable usually in let's say
• 29:13
  the data set, let's say this is your y Target variable and we're
• 29:18
  literally just going to go whether something is a positive
• 29:22
  or A negative. Generally you see this as either
• 29:26
  ones or zeros, And the way. So you guys might have
• 29:34
  experienced kaggle competitions, where you always get given the
• 29:38
  target variable that you're trying to predict in practice
• 29:42
  working as a data scientist or machine learning engineer, you
• 29:47
  usually never get this. You have to figure out some
• 29:50
  business logic to generate it and nearly I would say your
• 29:57
  nineteen ninety percent of the time.
• 29:58
  This is purely sequel. Some variant of it, some Sparks
• 30:03
  collar, or something, something that you're using the data.
• 30:06
  It can also be done in an hour or python or any other language,
• 30:11
  but majority of the time you can do most of the label generation
• 30:14
  within pure SQL. You'll like the tones are I've
• 30:19
  had to build labels. We had to join.
• 30:21
  Lots and lots of different data sets and tables from different
• 30:24
  databases handle some of the Effective and expiry dates for
• 30:32
  some of those slow-moving dimensions and all these
• 30:35
  different data modeling aspects of getting the data rate, which
• 30:41
  is it's probably its own separate topic as well.
• 30:44
  But just, yeah, trust me when I say getting this label right in
• 30:51
  practice is the hardest thing but it's also the most important
• 30:56
  thing you could have a really great label and just through In
• 31:01
  a lot of really good data or even average data, and it's
• 31:04
  still outperform something, where your label is totally
• 31:06
  wrong. So it's super important to get
• 31:09
  this, right? And I believe it's teachable,
• 31:14
  you don't need to have actual hands-on experience to learn how
• 31:18
  to build this label, but it does help of course.
• 31:21
  So, in saying that some of those, some of the things that
• 31:26
  you have to keep in mind for this target variable, is I just
• 31:30
  list them. I'm out here is timing bias and
• 31:33
  drift imbalanced classes, and also how you split out your
• 31:37
  train, validation test data sets.
• 31:40
  So let's talk about timing first.
• 31:43
  So I'm just going to rock this guy out.
• 31:45
  So when we think of timing, let's go one of the simplest
• 31:49
  things is probably. Let's try and predict the the
• 31:52
  probability of someone doing some event.
• 31:58
  And usually that's why people stop.
• 32:00
  So let's say the car insurance is really popular.
• 32:04
  So people would like to say, oh, let's predict which one of our
• 32:07
  customer base is going to lapse in this period or something.
• 32:12
  So you get you get this. This y variable is just this
• 32:16
  thing where you get people of 1 or 0 again of whether they
• 32:20
  elapsed. But you should be asking
• 32:22
  yourself when do they laps when when during the year, when when
• 32:30
  is this 1 and 0 actually set? And all these other questions
• 32:35
  which they, there's no real statistical Concepts or
• 32:41
  technical Concepts. It's literally just logical
• 32:43
  thinking So that's another thing for anyone who doesn't have the
• 32:48
  math and science background, you kind of don't need it to start
• 32:52
  applying some of these things and thinking through some of the
• 32:54
  problems. So what we like to think of as
• 32:57
  practitioners is usually okay, what's the probability of some
• 33:01
  event happening in the next Mike and period so this could be 30
• 33:11
  days or tea time or whatever. But when we Of supervised
• 33:17
  learning problems in general. It's much easier to think of it
• 33:22
  within a Time bound problem. So for example, when we when we
• 33:30
  try and run these things in production, you want your model
• 33:33
  to kind of give you out? Oh the, this is the probability
• 33:36
  of Demetrius lapsing on his car insurance account in the next 30
• 33:42
  days or something like that. So then when you, when you use
• 33:45
  that information to actually do something about it, then you can
• 33:49
  have some really actionable Insight so if I know Demetrius
• 33:53
  is going to lapse within the next 30 days, That and he maybe
• 33:57
  has like a 90% probability of doing that.
• 34:01
  What should I do? Maybe I could be more aggressive
• 34:04
  and I offer him 50% off just because I want to keep this
• 34:06
  business. Maybe you just give someone you
• 34:10
  call him just to catch up and see how he's feeling if if
• 34:14
  everything's. Alright, and then try and get a
• 34:15
  good sale out of him. There's all these different
• 34:18
  things that you can do from a, it's from a marketing
• 34:20
  perspective. But these things, when you, when
• 34:24
  you think about the machine learning, Me or my light has,
• 34:27
  which tough sorry. So it's a little bit dark, but
• 34:30
  these things from a machine learning perspective, a super
• 34:33
  important because an especially from the target variable because
• 34:38
  you need to think about how it's going to be used.
• 34:40
  Once you build the model. So timing is super important, I
• 34:46
  think one of the key things is just being able to know the
• 34:52
  timing where you'll be able to action some prediction.
• 34:56
  So within the marketing space, 30 days might be okay, let's say
• 35:00
  you're working in heavy industrial Machinery.
• 35:03
  Maybe, you only get one minute or less to actually do
• 35:07
  something. If you're trying to predict a
• 35:09
  catastrophic failure for example.
• 35:13
  Yeah, I think this is where machine learning gets really
• 35:16
  interesting. You can apply it in many places,
• 35:18
  but it's definitely not a one-size-fits-all for all
• 35:22
  problems. Such a good boy.
• 35:27
  Hmm, do you have any other questions so far?
• 35:30
  In terms of the timing? Well, I think it's just it goes
• 35:34
  back to that whole business value, right?
• 35:36
  And feel free. If anybody else in here has any
• 35:39
  questions drop them in the chat but for me it always comes back
• 35:44
  to that. Like yeah asking the timing of
• 35:48
  when an event is going to happen.
• 35:50
  Then gives you Insight on what you can do about it from the
• 35:53
  business side of things, whether it's the marketing thing or it's
• 35:57
  okay, there's going to be this catastrophic event in the
• 36:00
  factory. So we No, just to shut
• 36:03
  everything down or we know to do X and potentially evade that
• 36:10
  that big catastrophic event, right?
• 36:14
  Yeah. And yeah I think from a
• 36:18
  practitioner point of view it's you could you could build the
• 36:22
  best model but if you're if the thing that you're predicting Is
• 36:27
  not is not the right thing and not applicable.
• 36:30
  Then it kind of doesn't matter. Doesn't matter how good your
• 36:33
  deep learning and Hyper parameter optimization skills
• 36:36
  are if you, you can't use it to actually make a decision.
• 36:41
  Yeah, I did this. So funny.
• 36:43
  You say that because Dan, Sullivan my reference before,
• 36:46
  mentioned the same thing. How he one of his biggest.
• 36:50
  I asked him about a war story and he said, one of my biggest
• 36:53
  war stories was just, I spent months on a model.
• 36:57
  You know, to to get and getting it to be the best and went back
• 37:02
  and the people that actually were using it.
• 37:04
  For business value said we can't use this what this doesn't do
• 37:08
  anything for us and so he was just like they didn't care what
• 37:13
  what did he say, like they don't care what my f score was but he
• 37:18
  was so proud of his F score. That was so so good.
• 37:22
  He was like, why is this is great and then he goes and gives
• 37:25
  it to the business side of things they're just like, What
• 37:27
  we can do with this. Like I have I've seen a few
• 37:31
  meetings in my in my time where that data scientists the really
• 37:36
  smart people like with phds and really really intelligent.
• 37:41
  And they're trying to explain an Roc curve to a senior executive
• 37:46
  and you just see that the glazed look like they're just that
• 37:50
  they're gone and they just have no idea what what they're
• 37:53
  talking about and they kind of dismiss them.
• 37:57
  Mark. And then the whole point of that
• 38:00
  meeting was to try and explain the results of the model or
• 38:03
  anything that they were using and the the executives just kind
• 38:07
  of came out of it with like do these guys really know what
• 38:10
  they're doing like. So the whole point there is
• 38:14
  really to be able to as a data scientist, I guess to be able to
• 38:19
  show your worth, in boil, it down into this like simple to
• 38:25
  digest. Hey, we You can tell when
• 38:28
  someone is going to churn or we can tell the probability of this
• 38:33
  happening within the next X days.
• 38:35
  So we suggest that you do this about it or we suggest marketing
• 38:40
  takes a deeper look into it and puts them into a sequence or
• 38:45
  different newsletter, whatever it is.
• 38:48
  Mmm. Yeah.
• 38:49
  And it's nearly all the machine learning has to be about
• 38:52
  application. Definitely sweet.
• 38:56
  So what do we got next? Yeah, so timing is very
• 39:00
  important. I could probably, if you have me
• 39:03
  on another time, I can go through all the different
• 39:05
  methods that we can try and try and do the timing in a
• 39:09
  statistically valid way. But that, that will take me two
• 39:12
  hours that by itself. But another thing that we can
• 39:16
  talk about is the bias and the drift and this kind of that the
• 39:21
  two separate Concepts. But when I think of bias You
• 39:25
  can. It's also linked back to the
• 39:27
  timing. So if you think of that that
• 39:33
  lapse the insurance lapse example, if you, let's say you
• 39:37
  choose the date of the actual event for the the date where
• 39:42
  you're going to use for all your feature generation, because
• 39:47
  another thing to keep in mind is let's say your date of your
• 39:50
  label. Is here and let's say 20 2001
• 39:56
  2001 when you choose that as your label date you have to get
• 40:02
  features from the past. You can't get features from the
• 40:05
  future. That's that's a no-go.
• 40:09
  You can't predict what you can't.
• 40:11
  You don't have feature data essentially, but the next thing
• 40:15
  for the bias is also very related to the timing because
• 40:18
  let's say, Demetrius is this example and he's elapsed.
• 40:22
  And he lapses on that date. He lapses on like 2020 Jan 1st
• 40:29
  if you use that as your 1 or 0 metric like that one or zero
• 40:33
  label across all your customer base.
• 40:36
  What are you predicting? You're predicting the
• 40:38
  probability of someone like seeing today.
• 40:45
  And that's, it's not really usable is that and the you,
• 40:52
  you're bound to get a lot of noise as well, because maybe
• 40:56
  you'll pick up all the different activities that Demetrius was
• 40:59
  doing before that laps event. Maybe you won't.
• 41:03
  I've seen other ways that people try and generate this date for
• 41:07
  the for that training example, also, which is different.
• 41:10
  So let's say I probably shouldn't go into this.
• 41:14
  Much because this is part of the to our topic that we could talk
• 41:17
  about, but you're just kind of everybody wants to.
• 41:20
  You can check out his usual Diana.
• 41:22
  Yeah. And he goes way into the weeds
• 41:24
  there. And yeah, essentially bias is
• 41:30
  more around bias in terms of when something happens.
• 41:35
  So I guess it's also tied to the timing as well, but when we
• 41:39
  think of the drift and the bias together, you can kind of think
• 41:44
  of it. Like let's say there's a
• 41:45
  timeline and this is say it goes all the way from 2008 up to
• 41:52
  2020, depending on what? Do you want to include in your
• 41:58
  machine learning problem? Do you have any?
• 42:03
  Like let's say the data from 2008 up to 2012.
• 42:09
  This is let's say, GFC period. Do you think that will be very
• 42:14
  different to post GFC period and then from 2020 to 2025 to do you
• 42:21
  think they'll be some covid related data?
• 42:24
  That's baked into your context and whether that will also
• 42:28
  impact your prediction tasks as well.
• 42:31
  And these are more, they're very external context things, but
• 42:36
  when you think of people's behavior in the data or the
• 42:40
  different things that you're trying to predict if it's a
• 42:42
  consumer Behavior pattern or a banking or retail or anything
• 42:46
  like that, of course, these economic factors will definitely
• 42:49
  play its part. If it's Machinery, maybe it's
• 42:53
  not in such a large time factor, that may be the Chair of the
• 42:57
  seasons, the rainfall humidity, all these sorts of things might
• 43:03
  play a factor as well. So these are things that you
• 43:06
  kind of overtime and domain experience within the different
• 43:10
  areas as a machine learning practitioner.
• 43:13
  You have you develop stronger intuition for when these might
• 43:17
  impact the actual problem, but when you think of it, these are
• 43:21
  Tech. These are not technical issues.
• 43:23
  These are thinking issues their logical problems.
• 43:26
  He's so just having the knowledge of how you might
• 43:29
  handle some of these situations. Yes, that might be technical,
• 43:33
  but knowing that these situations might occur because
• 43:36
  of the scenario and the context, that's business, knowledge,
• 43:43
  Okay, so biased adrift very important as well.
• 43:48
  The third thing to think about is imbalanced classes, so, in
• 43:53
  kaggle, or some of those different machine learning
• 43:56
  examples, normally, you see, most of those examples will have
• 44:01
  one to one negative to positive ratio.
• 44:04
  This rarely happens in practice. You might get, if you're really
• 44:09
  unlucky, you might have Have 0.001% of positive cases, and
• 44:15
  the rest is all - cases. So it's a real rare event
• 44:19
  prediction and there are different ways to handle
• 44:24
  imbalance classes in terms of how you would do with the target
• 44:29
  variable. So there are different things.
• 44:32
  I'll just list out a few different techniques here, but
• 44:34
  you can always downsample the majority class.
• 44:38
  So let's say, let's say, One zero zero zero.
• 44:41
  One percent is a little bit to over.
• 44:43
  Let's go. Let's imagine that it's 10% is
• 44:48
  positive and 90% is - to get it closer to that one-to-one ratio.
• 44:55
  You could potentially down sample.
• 44:56
  This 90% And you just take what is that 10 divided by 90 or you
• 45:03
  try and make it match? One to one with the Positive
• 45:06
  case, another example or another Technique, that is very popular.
• 45:12
  These days is using something called smoked.
• 45:17
  I forgot what it actually means but it's kind of up sampling.
• 45:21
  The minority class and kind of using the statistical
• 45:27
  distributions of the data to generate fake data.
• 45:31
  So you can also learn from it and it kind of it does the other
• 45:34
  way around. So your oversampling the
• 45:37
  minority cost. There's there's other
• 45:40
  techniques, which is so, these are kind of simple techniques.
• 45:45
  The more advanced technique is to actually update the logic
• 45:49
  itself. So Force more positive, cases
• 45:53
  into your data set. So and that ties in really well
• 45:57
  with this train, validation split.
• 45:59
  So what a lot of people think of Trane, validation split, right?
• 46:05
  So maybe I'll ask Demetrius to find out.
• 46:09
  You have you ran an examples where you get, you just get
• 46:11
  given a data set and how would you use, how do you use that to
• 46:16
  split your training validation and your test sets, what sort of
• 46:20
  techniques would you use? I actually will defer to the
• 46:27
  practitioners in the chat on this one because I'm not sure I
• 46:33
  have any good answer. Does anyone in the in the chat 1
• 46:37
  and let us know what you think. Might be a difficult difficult
• 46:45
  question. We'll see if anything comes
• 46:48
  through for now. Otherwise that yeah, I'll
• 46:55
  probably keep barreling on but yeah, but it's not until
• 46:59
  something comes through a random, we got 10.
• 47:02
  Bingo. So nearly nearly all the time.
• 47:08
  People would just split this into 60 2020 or different
• 47:13
  things. Maybe you might not even have a
• 47:15
  test set. You might only have a training
• 47:17
  in a validation set. Maybe you do k-fold
• 47:20
  cross-validation where Have split your data set into K
• 47:24
  Parts. You train it on K, minus 1 and
• 47:27
  then you validate it on the remaining pot.
• 47:30
  There's all these different techniques that you can use to
• 47:33
  kind of improve your model prediction by kind of learning
• 47:38
  from the data as much as possible in practice.
• 47:42
  This doesn't always work as well as always in practice.
• 47:47
  What we tend to see is that. So, We let me, let me go back to
• 47:54
  what the training set and the validation set and the test set
• 47:57
  actually does what are the functions in machine learning
• 48:01
  from a purely theoretical perspective.
• 48:03
  So, when you think of the train set.
• 48:08
  So let's just, let's just say, we've we just randomly split the
• 48:11
  data. So I've got my train set here,
• 48:14
  validation set, and the test set here.
• 48:16
  So I think we're all in agreement in that this test set,
• 48:20
  we just kind of Leave at that we don't do anything.
• 48:25
  And for the training set this is where we kind of we want to do
• 48:31
  all the different Transformations.
• 48:33
  The feature engineering build, all those pipelines, and then
• 48:36
  eventually run the machine learning algorithm on this
• 48:39
  training set. Whether it's validation set
• 48:42
  comes in, is I'm not sure for what people do these days.
• 48:48
  But when I was learning we would always run our algorithms and
• 48:53
  use the validation set as an early stopping criteria.
• 48:57
  So you can think of the algorithm just keeps running and
• 49:00
  running on this train set but after each iteration it would
• 49:03
  actually go and test that data or test that model on the
• 49:08
  validation set and return. You symmetric say AC for example
• 49:14
  and you can I said it, you can make your training routine,
• 49:18
  continue until that asymmetric kind of stops improving.
• 49:23
  That's why we call it early stopping.
• 49:26
  So that is what what I used to apply and I would still apply.
• 49:31
  Now I don't know what people do. Now they've gone all this crazy
• 49:34
  stuff that goes on. But when you think of the
• 49:37
  training set and the validation set and the test set, there are
• 49:42
  some logical components to that. Station set and the test set.
• 49:47
  So let's say. Most people when you think of
• 49:52
  these training validation and test slits, it all comes from
• 49:55
  the same data set. So you just get this block of
• 49:58
  data, I could a data frame and then you randomly split it in
• 50:02
  practice, if you randomly split data across different time
• 50:07
  periods, you could inherently learn something alone.
• 50:11
  Some sort of information from the future and use it to predict
• 50:14
  the past. And one of the ways that we
• 50:17
  avoid doing that is to do. I'm out of time, training,
• 50:23
  validation and test plate. So going back to this GFC
• 50:26
  example maybe we would use maybe that's not a good example if we
• 50:32
  if we have another if we have another timeline so let's say
• 50:37
  this is just 2016 through 2019. For example maybe we would use
• 50:45
  the first two years for training data.
• 50:47
  So mopped, 2018. And then we split the last year
• 50:52
  into six months and six months, maybe and this is assuming that
• 50:58
  you don't have funny things going on throughout the year, if
• 51:01
  you have funny things going on throughout the year, maybe it's
• 51:03
  safer to actually go at a year rate for your training and your
• 51:08
  validation or maybe training can have more data for your
• 51:11
  validation in your test. Should be the same split size
• 51:15
  and when we think of the training, validation and test,
• 51:19
  when you think of the Distribution of the data.
• 51:23
  What what I was always taught was your validation and your
• 51:26
  test set. So, these two guys, so this
• 51:30
  validation and that test, they must come from the same
• 51:34
  distribution. Otherwise, you have no idea what
• 51:37
  you're optimizing over. And that's like a it's some
• 51:43
  could consider it a technical thing, but it's almost just The
• 51:46
  Logical. So if you're, if you're
• 51:48
  training, your machine learning algorithm and you're iterating
• 51:51
  over the training set and you select the best model with that
• 51:55
  early stopping criteria. Based off that validation set,
• 51:58
  you really need to make sure that whenever you're stopping
• 52:01
  it, it matches to the thing that you're going to test on which is
• 52:04
  the test set So, I know it's a, there's a few mental hops in
• 52:13
  there, but it's all very sequential.
• 52:16
  So, do you have any comments on that Demetrius?
• 52:21
  I was just seeing that Lazlo was put some stuff in slack.
• 52:25
  I don't know how long ago. It was, I guess it was a while
• 52:27
  ago last. Oh so just seeing this.
• 52:31
  Because I've been monitoring the chat here but not not the slack
• 52:35
  so I'll say, I guess this will be catch.
• 52:38
  Up then he's just mentioning that when he worked in finance.
• 52:44
  If the data is on premise then it is in Sample.
• 52:48
  You can only do out-of-sample with data, that doesn't exist
• 52:51
  yet. That was his thing when he was
• 52:55
  also my dance. I don't remember where that was
• 52:58
  from. Maybe you have a better memory
• 53:01
  than I do. Lazlo.
• 53:03
  If you want to have a say or what you were mentioning.
• 53:07
  Yeah, I think. Yeah, ensample, it's still
• 53:11
  definitely. You can still use in Sample data
• 53:15
  I think from my, from my experience, I've always used
• 53:18
  this out of out of time testing because I'd always be running
• 53:23
  these models in production to try and predict something ahead.
• 53:27
  Which I think that's what Leslie.
• 53:30
  Leslie Leslie Leslie. Yeah, yeah, yeah.
• 53:35
  So for yeah, for out of time, it's very useful.
• 53:38
  If you're trying to predict something in the future,
• 53:41
  sometimes for machine learning models, all you need is to try
• 53:45
  and understand and predict what happened in the past and try to
• 53:49
  say, oh, in the past we could predict fairly is from this
• 53:54
  point in time or whatever. Had excellent so-and-so metric,
• 53:59
  which is all good. Also, I think the out, Time and
• 54:03
  join the predictive modeling for Behavior.
• 54:07
  You really do need this out of time.
• 54:09
  Testing, I know that a lot of people just kind of do, they
• 54:13
  just bash it all into the same data set, which it does work,
• 54:18
  but it there are implications in how testable it is.
• 54:22
  So you can only the only way to test it is to run it in a live
• 54:27
  environment and then see what happens.
• 54:31
  And we I can talk about that. At four days also but there's a
• 54:34
  whole forward testing regime that we can do with our models
• 54:37
  to make sure that they're doing the right thing before we even
• 54:40
  go live and that's even when you have your training validation
• 54:46
  and test slits, there are still more things that you can do to
• 54:50
  validate that. It's going to work before you
• 54:52
  actually go live with it and potentially risk pissing off
• 54:56
  some customers or something. Yeah.
• 54:58
  Totally. So I want to be respectful of
• 55:02
  your time, I know Just hitting the top of the hour.
• 55:06
  Do you have two minutes or three more minutes to wrap it up for
• 55:10
  us and give us. Yeah, I can say I'm more than
• 55:14
  happy to go for as long as as long as you want.
• 55:16
  I think I've literally I'm up to slide 3 of 7 or something.
• 55:22
  So I just ended up talking way over time.
• 55:25
  I am must be respectful of your time also.
• 55:29
  So I think this has been great. This has been super exciting and
• 55:35
  interesting and for me, of course, like always when I'm
• 55:38
  watching your stuff, I've learned a ton and it all that's
• 55:42
  also me, understand things. You have an ability to put
• 55:46
  things into a very simple Manner.
• 55:48
  And it's, again, it goes back to that distilling down these very
• 55:53
  complex Concepts to share to people who aren't necessarily
• 55:58
  doing this day in and day out. And you're able to do that.
• 56:02
  I'm one of those people that is not in the trenches, right?
• 56:06
  But I understood what you were talking about and I learned a
• 56:10
  ton from it. So maybe we do the the to our
• 56:15
  Deep dive on your YouTube channel.
• 56:18
  And yeah have, yeah, I think, I think it's really cool for
• 56:22
  everybody that is out there and has not checked out Danny's
• 56:26
  YouTube channel. Go have a look at it.
• 56:28
  If you want to learn more, he's doing all kinds of cool stuff
• 56:32
  with Learning data science and obviously you've seen for up
• 56:36
  until now he's got an ability to teach and he has a lot of
• 56:41
  knowledge that he is very generous imparting unto the rest
• 56:45
  of us. So is there anything else that
• 56:50
  you want to say, just before we go?
• 56:52
  And to kind of wrap it up? I would say probably for, for
• 57:01
  the ml Ops community, I know there's a lot of focus on
• 57:05
  tooling and model serving and working closely with the data
• 57:10
  scientist to make sure that they're that all the processes
• 57:13
  and operations are all good. What I would like to stress for
• 57:17
  all the machine learning Engineers out, there is two kind
• 57:20
  of challenge yourself to really understand where the outputs are
• 57:24
  being used. Especially in most places, you
• 57:30
  don't really get a chance to determine What business
• 57:33
  processes? It actually changes or the
• 57:36
  different decisions that it gets to make, unfortunately, but it's
• 57:40
  definitely nothing is stopping you from learning about it and
• 57:43
  getting that contextual understanding of where your work
• 57:46
  is being applied. So I think that's super
• 57:50
  important, and very valuable it also it also helps when you're
• 57:55
  writing stuff in your resume and you can say, oh my outputs help
• 57:59
  deliver x million of dollars of revenue for Mark, Initiatives or
• 58:03
  something. But yeah, I think just be as
• 58:08
  curious as possible. Don't be intimidated by the data
• 58:11
  scientist, talking about all these statistics and maps and
• 58:14
  whatever. Most of it, just boils down to
• 58:16
  Pure logical thinking work closely with the data
• 58:21
  scientists. When they're doing the
• 58:22
  experimentation chances are, they're not as technical as you
• 58:26
  are. So there are definitely tons of
• 58:29
  things that you could share, that will help make their lives
• 58:31
  easier. And Hopefully, they have lots of
• 58:34
  things that they can teach you as well from the statistic side
• 58:37
  and all those other things that we talked about today.
• 58:41
  But I think yeah, just be as collaborative and as curious as
• 58:45
  possible and believe in yourself that you can learn all of it
• 58:48
  because if I can learn it, you guys can learn it too.
• 58:52
  Was such a good point. It is, it comes back to that
• 58:59
  producing real business value, right?
• 59:01
  We're That's what we're going for here and I think you put it
• 59:06
  wonderfully because that will give you a level up.
• 59:09
  If you're always thinking about the business, need at the
• 59:11
  business value that you're going, for your going to be
• 59:15
  playing in a different field than just.
• 59:17
  Okay, what's my accuracy score? How can I predict this or how
• 59:22
  can I put this model into production?
• 59:24
  If your machine learning engineering, you're just focused
• 59:26
  on the pipeline or you're focused on getting things out?
• 59:29
  Yeah, that's great. But is it something what Is this
• 59:33
  model going to be doing once it's out.
• 59:35
  And so really conversing with the people that are more on the
• 59:38
  business side to make sure that you are producing the most
• 59:41
  value, because maybe a little cross-pollination every now.
• 59:45
  And again, you realize that there's something very simple,
• 59:47
  you could be doing that produces even more value.
• 59:52
  Definitely awesome value is the word that's it.
• 59:57
  So you have given us a ton of value today, teaching us all
• 1:00:01
  about this stuff. I really thank you again and for
• 1:00:05
  anyone who doesn't follow you on LinkedIn, get out there and
• 1:00:08
  follow Danny ma he's dropping all these bombs on us all the
• 1:00:12
  time, every day and for everyone else out there.
• 1:00:17
  Thanks for watching, thanks for sticking with us. and we will
• 1:00:21
  see you next week for more ml Ops meetups Awesome.
• 1:00:29
  Thank you so much for having me. See you guys and girls it
• 1:00:34
  everybody.