Latest / Agentic Conversations (formally mlops.community) / Scaling Similarity Learning at Digits // Hannes Hapke // Coffee Sessions #122

Transcript

Scaling Similarity Learning at Digits // Hannes Hapke // Coffee Sessions #122 · Agentic Conversations (formally mlops.community)

• 0:00
  Hi, Buzz. I'm from Portland, Oregon.
• 0:02
  The coffee City. Normally, I like to drink my
• 0:06
  coffee as a latte, double stop. But right now, I take a little
• 0:11
  break from it and I switch to green tea.
• 0:14
  I got a ton of takeaways from this conversation.
• 0:16
  First one has to be hands down data hygiene man, like the
• 0:22
  importance of data hygiene and how honest was able to map out
• 0:26
  what good data hygiene looks like.
• 0:28
  Not only From the machine learning perspective.
• 0:31
  But also from just like the software engineering
• 0:35
  perspective, from a design perspective, from the back end,
• 0:38
  engineering all the way data Engineers, everything that was
• 0:42
  mind-blowing to me, I loved it. What about you and who are we
• 0:45
  talking to you? By the way, I didn't even say
• 0:47
  harnesses name and what he does and where he's working, and I'll
• 0:50
  get into that. I'll get into that, I'll give
• 0:52
  the intro and then I'll let you get my takeaway.
• 0:54
  So, Thomas hopko. First off.
• 0:56
  Awesome guy, incredible guy, he's the first MLS.
• 1:00
  Engineer at digits where they are building Financial tools for
• 1:03
  the future. I mean, really, you guys gotta
• 1:05
  listen to this episode in here about it.
• 1:06
  And at digits, he started, he started and sort of laid the
• 1:10
  foundation for their ml Ops approach and their machine
• 1:12
  learning approach as a company. He prior two digits has worked
• 1:17
  in a number of Industries, including retail, Healthcare and
• 1:19
  Erp companies. And he's actually the co-author
• 1:22
  of to machine learning books. One of which we talked a lot
• 1:24
  about building machine learning pipelines, which is published by
• 1:27
  O'Reilly. You should go check it out and
• 1:29
  the S being an LP and action for Manning.
• 1:32
  So that's hottest incredibly impressive guy.
• 1:35
  And I really, really, really enjoyed two.
• 1:39
  Takeaways I'll say number one. I loved his point about how
• 1:42
  thoughtful design supercharges machine learning and vice versa.
• 1:46
  It's such an interesting point that honestly it's the first
• 1:50
  time that we've had someone on podcast and like 100 episodes
• 1:53
  come on and say that. Yeah.
• 1:55
  And then the second takeaway that I had was really Around the
• 2:00
  twenty one minute Mark hannes walks through the entirety of
• 2:05
  their machine learning Pipeline and the and how they implemented
• 2:09
  it. And like what all the elements
• 2:11
  are what the alerting looks like with tooling looks like.
• 2:13
  And I'm like if anybody is like building machine Learning
• 2:17
  System, they should just like cut that clip and like listen to
• 2:19
  it and like rights print tickets based on their team.
• 2:22
  But yes right sprinting, it's based on what he describes the
• 2:26
  features as and you're like, okay, build a learning into
• 2:29
  slack, fine. Right.
• 2:30
  We're going to put that as a Sprint ticket.
• 2:31
  I thought that was such a cool and like complete elucidation of
• 2:35
  like, what a machine Learning System was.
• 2:36
  So, this is a great conversation.
• 2:39
  We specialize in great break down real fast.
• 2:41
  What you mean by Design superpowers or design?
• 2:47
  Supercharges machine learning. Yeah.
• 2:49
  Yeah. I mean, machine learning in a
• 2:53
  product, is kind of a double-edged sword, right?
• 2:56
  Because it can make the product a lot more useful.
• 3:00
  But it depends on assumed and strictly defined behavior from.
• 3:07
  Users right? Think about all of the ways that
• 3:11
  YouTube, for example, the delays YouTube's recommendation,
• 3:14
  algorithms are gamed by unsavory actors, like Neo-Nazi groups and
• 3:18
  whatever. That's an example, where like
• 3:20
  users are exploiting poor product design around the
• 3:24
  machine learning algorithm. But the other way is true as
• 3:26
  well, right? Like, you know, with other kinds
• 3:28
  of recommendation systems like maybe tick-tocks or something
• 3:30
  like that. People find really great
• 3:33
  thoughtful content because the way that the system app is
• 3:35
  actually designed Promotes effective use of the algorithm
• 3:39
  to find things that people really like.
• 3:41
  Now depends on the way you think about Tick-Tock.
• 3:42
  I just use that an example. I'm not saying it literally but
• 3:46
  that's, you know, I'm on Tick-Tock now, you know, and I
• 3:48
  know you are waiting for you to stitch one of my videos.
• 3:52
  I'll leave a link to that in the description in case anyone wants
• 3:55
  to come join me. I get like five watches per
• 3:59
  video and I thought talks algorithm was supposed to send
• 4:02
  stuff out. I guess my stuff.
• 4:03
  Isn't that good though? So every come on man, come and
• 4:06
  Me out everyone starts somewhere, but yeah, that's how
• 4:08
  that's to me how design and and machine learning go together,
• 4:12
  right? They're really, they're really,
• 4:15
  really a powerful pair when when used effectively well yeah, I
• 4:19
  liked how he was talking about when they are thinking,
• 4:23
  basically, the way that the design team and the Machine
• 4:26
  learning team interact is fascinating to me, and it feels
• 4:29
  like they've really, they're really advanced in that way and
• 4:32
  there's a lot of benefits from coming together and Having
• 4:36
  multiple touch points and just talking about like having the
• 4:39
  design team being literate in machine learning stuff that
• 4:44
  they're doing like, and how they, the machine learning team
• 4:47
  will sometimes go back and they'll teach the design team
• 4:51
  different concepts. So that they understand what is
• 4:54
  possible. But then when he was talking
• 4:58
  about, how they baked in explain ability into the tool so that
• 5:01
  they wouldn't be getting as much garbage data or that the end
• 5:05
  user could And why they were getting certain recommendations.
• 5:10
  I thought that was awesome. That was it's just super cool to
• 5:12
  see. I mean, he goes into so much
• 5:15
  detail and he's so transparent about how and why is it, did
• 5:19
  everything I thought this was awesome because of that, like
• 5:22
  it's rare. I feel like people are so open
• 5:25
  when they come on here and he was very much from the get-go,
• 5:28
  like these things need to be talked about.
• 5:30
  We need to be addressing these problems, and I want to help get
• 5:34
  the word out about it. So, Hannes might have one like
• 5:38
  my favorite human of the Year award.
• 5:40
  He was great. And I think we should let the
• 5:42
  listeners get into it. Let's do it.
• 5:48
  I think we have to start with your book.
• 5:51
  Like, give us a breakdown. How did you even decide to
• 5:56
  create such a beast and like, tell us about that process?
• 6:00
  And why? Well, I don't know about this,
• 6:04
  her beast, but I think any technical book is about East
• 6:06
  because like while you write the book anything behind the scenes
• 6:09
  is changing new versions of coming out and you're like
• 6:11
  constantly rewriting chapters. So that qualifies it - piece for
• 6:15
  every technical author. Why that book it started out of
• 6:21
  the need of like, having a concise into end overview about
• 6:26
  like how to implement machine learning, machine learning Ops
• 6:30
  for both companies. So, like, starting with, like,
• 6:33
  what is the data ingestion through the data?
• 6:36
  The model training model, validation Amanda serving part
• 6:40
  back in the days of like, the serving part.
• 6:42
  Like ten civil servant came out and that sort of like they
• 6:44
  started to standardize the ways of how we deploy machine
• 6:47
  learning models. And then liked, I think tensile
• 6:51
  load data validation came out and we're sort of like there was
• 6:53
  a tool to analyze the data and then we have like, luckily your
• 6:57
  more tools like great expectations and others.
• 7:01
  But what was missing was the glue code of like how do we
• 7:06
  connect? Checked, all the different
• 7:07
  tools. There was like, tentacle turns
• 7:09
  from came out at some point, but it was this piece missing
• 7:12
  between tens of the data validation, schema generation,
• 7:15
  and then principal transformed. And so, the original goal of the
• 7:21
  of the book was like, how do we connect on the different pieces?
• 7:24
  And what would be, what, how would the pipeline look like?
• 7:28
  Well, then one year right into writing this book, Google open
• 7:32
  source, their internal tfx platform.
• 7:35
  Basically the Between those different different tools.
• 7:40
  And then I was in the scenario of length.
• 7:42
  Let's rewrite this whole thing and that's why we focus solely
• 7:46
  on tfx, because that was the platform at the time.
• 7:50
  Going out of coming out and I think it's for anybody who's
• 7:55
  interested in getting in the understanding of like, how can I
• 7:58
  put those pens together? How can I consistently trained?
• 8:01
  My machine learning models. I hope this book is a good
• 8:03
  introduction, and overview and show.
• 8:06
  Showcasing up. Like, how to do this with one
• 8:08
  tool? And how much have you thought
• 8:11
  about going back and rewriting it?
• 8:14
  Now after a few years like, or how, how do you fight the urge
• 8:18
  to not like, have to update every Edition.
• 8:22
  Every time, you're like, oh man, now I wanna, I want to add that,
• 8:25
  that thought came up maybe 30 minutes after we hit submit and
• 8:32
  the final draft or writing, there's like, a time frame
• 8:35
  between yourself major Final draft and then they do all the
• 8:38
  magic work behind the scenes until you actually get the hard
• 8:41
  copy. And I think Google released like
• 8:45
  an in major minor version upgrade like some some cool
• 8:48
  features an event like, damn it. Yeah.
• 8:52
  So cool to add this feature but at some point you just have to
• 8:56
  you have to stop because they're like some people saying hey
• 9:01
  missing this component like I think in probability for
• 9:04
  validation component landed after we It such a cool
• 9:07
  component to test machine learning models before you
• 9:10
  actually release them into the Royal.
• 9:12
  But there's, if you continuously update the model or update
• 9:16
  another model, but then book, then it's a moving Target in
• 9:20
  this really hard to get the book out.
• 9:23
  So sad with like printed copies, it's really hard to keep them to
• 9:29
  the, to reduce the time frame between like last edit.
• 9:33
  And the print, there's always Delta there.
• 9:37
  Yeah, makes complete sense and I'm wondering, like when you
• 9:43
  were thinking about writing the book because there is some piece
• 9:46
  of it that you can abstract away and does not have to focus on
• 9:50
  the tooling, right? And then there is another piece
• 9:52
  of it, where it is purely tooling, and it's purely code.
• 9:56
  And like you said it has to change as the like when tfx
• 10:01
  whatever 3.0 comes out, then. Oh, now everything.
• 10:05
  That we put in the code in the book is no longer valid, or so.
• 10:10
  So, like, what are in your mind? Some of those pieces that are
• 10:16
  the, the former where it's like, or sorry, former ladder.
• 10:19
  Yeah, the, the parts that you can abstract away that are not
• 10:23
  this specific code Snippets. I think as a specific before,
• 10:27
  Emma, laughs is sort of the thinking about how to get how to
• 10:30
  go from raw data to a model and how to serve that model, I think
• 10:35
  we Regardless of what will you use at the end, there's a mental
• 10:38
  workflow of like how you deal with your machine learning model
• 10:41
  and the corresponding data, for example, the concept of a meta
• 10:45
  data store. It doesn't matter what what
• 10:47
  store you use or like what format or what orchestrator you
• 10:52
  used to Orchestra, do machine learning by plants.
• 10:54
  The fact is that you think about the orchestrator to coordinate
• 10:58
  your machine line? Prevents the times, did you
• 11:00
  think about like The Meta data store, what's the value of
• 11:04
  setting up this? I think those are the pieces.
• 11:07
  I would love to for the folks to take away from the book that
• 11:10
  they may be the the specific tool.
• 11:13
  We making maybe the version We pinned for this book is sort of
• 11:17
  like outdated but the concepts and the sort of like a the the
• 11:25
  concepts and the mental thinking around like how do we actually
• 11:28
  structure machine learning Ops in a company or in, for any
• 11:32
  large project? I would hope that this The
• 11:35
  takeaway for the book and just as a side note, we went back
• 11:38
  after the release and we updated all the example, code to tf-x
• 11:44
  1-0, do the review. They have a pin version.
• 11:48
  And so we support this, unfortunately, it's not that
• 11:50
  printer. Copy that I'm glad.
• 11:53
  I'm glad you guys got to do that.
• 11:54
  You guys got to do that. And did that with the companion,
• 11:56
  code for the book? One question, I have you pointed
• 12:01
  out this, this mental model of a pipeline where You know, you
• 12:06
  have to think about from the very raw files, that, you know,
• 12:12
  are are used. As you know, the source
• 12:16
  information all the way to the model output like that, that the
• 12:19
  mental model, there's a pipeline.
• 12:20
  And so, I started off as a machine-learning engineer and
• 12:23
  I've been learning more about like traditional Concepts and
• 12:25
  data engineering, right? We're in data engineering, I
• 12:28
  think a lot of this notion of like pipeline development,
• 12:31
  starting with cron jobs, it, Facebook in 2010, you know,
• 12:36
  That, that literally, you know, I'd be owed everything was just
• 12:39
  a bunch of, you know, shell commands and stuff that I think
• 12:43
  data engineering. Really like, got the whole
• 12:48
  software engineering industry, thinking about this, mental
• 12:50
  model a pipeline or this design pattern of a pipeline.
• 12:53
  And I'm kind of curious, you know, from your standpoint.
• 12:56
  Like, how did you weave in to the book?
• 12:59
  Some of the concepts of data engineering that are really
• 13:02
  essential to doing machine learning engineering.
• 13:04
  Well, how did you approach? That sort of crossover.
• 13:08
  That's that's a really good question.
• 13:10
  I think, when I, when I started looking into machine learning
• 13:13
  operations, specifically coming from the data sign of data
• 13:17
  scientist world or from the mirror machine learning world.
• 13:19
  I started really appreciating a patch of being not fully
• 13:24
  understanding the concept at the beginning, but then sort of like
• 13:26
  overnight unpeeled the onion at every layer.
• 13:30
  I got deeper and deeper. It felt like.
• 13:31
  Wow, this isn't a magic tool. This is a magic tool.
• 13:34
  And to your point is like This is probably today's version of
• 13:37
  the Cron job. Everybody's using some sort of
• 13:39
  like a pipeline to Le Petit beam, airflow, whatever.
• 13:43
  But those those tools are actually keeping the machine
• 13:46
  learning world, I guess. Alive.
• 13:48
  Like that's that's what I would say.
• 13:49
  Like in the book itself, every T of X component is based on
• 13:56
  Apache bean and we stressed in the book of like how can you
• 14:01
  write your custom components? How can you write?
• 14:03
  Like we only show like People being cold.
• 14:06
  And I think that's out of the scope of the book to actually
• 14:09
  introduce the art, go beyond the introduction to Apache Bean, but
• 14:13
  if you want to ride your own components, and each component
• 14:17
  is sort of handling terabytes of data.
• 14:20
  Like, this is a good approach, like think about the data in
• 14:23
  such a way that you can paralyze things and do you point of like
• 14:28
  maybe back in the day, was cron, jobs now, it's Apache beam.
• 14:32
  We use data flow at digits. That's like It's not just about
• 14:37
  training the model. I think training the models
• 14:39
  actually, everybody has talked about this like how can we
• 14:43
  design the architecture and there's so much things around
• 14:46
  it, which we need to setup and tuned to actually get those
• 14:51
  pipelines to work. And the core part of that is
• 14:53
  like, how can be paralyzed the data ingestion, the
• 14:56
  pre-processing, maybe towards an efficient training.
• 15:00
  So when I one thing I would also like to for folks, to take away
• 15:05
  from the book and we're What will you use as like the
• 15:09
  pre-processing of the data is a and you benefit from this?
• 15:14
  Once you deal with like terabytes of data on your legs
• 15:16
  to shuffles through, like you want you to run multiple
• 15:19
  iterations over your data. But then, the big win there is
• 15:24
  like, if you take those graphs, you generate from that
• 15:27
  pre-processing and you attach this to the later produced
• 15:30
  model, that is an ultimate solution against Training
• 15:35
  Services. You like that's what we do a
• 15:38
  digits. Like when we progress our data
• 15:40
  we automatically generate a pre-processing graph and then we
• 15:44
  will ship the model to production.
• 15:45
  We connect those two pieces, the pre-processing graph, plus the
• 15:49
  train model, and then all the sudden, all the changes we make
• 15:52
  during our training run trickle through automatic view to our
• 15:56
  production deployment. I mean I'm taking a second to
• 15:59
  process what you do. What you just said there
• 16:01
  Demetrios you look like you have a question about that?
• 16:03
  Yeah I think that's a really interesting.
• 16:06
  Sure thing, one that I hadn't heard before, like with this
• 16:08
  training serving skew because that is something that comes up
• 16:12
  quite a bit. And I know that I've heard like
• 16:16
  a feature store. For example, is something that
• 16:18
  will combat that but I hadn't heard of this pre-processing
• 16:21
  angle and maybe can you just go a little bit deeper into how
• 16:26
  that looks? Yeah, maybe an example.
• 16:29
  Yeah, it totally. So the typical example is like,
• 16:35
  let's say you have a feature you want to normalize And you train
• 16:40
  your, you have some values in your living official store, or
• 16:45
  like in your, in your raw data, big data sets.
• 16:49
  And so, you have a mean and a standard deviation given on
• 16:54
  that. So in your, let's say you use a
• 16:57
  pre-processing tool like tents will transform, the tool itself
• 17:02
  does two passes over the data. And so that's one pass over the
• 17:05
  data to determine. What's my mean what's The
• 17:07
  deviation for this particular feature and then it does another
• 17:11
  path of the data to actually formalize those values.
• 17:14
  And so in that process, it generates a post processing data
• 17:19
  set that data set, you can then use to retrain your models or to
• 17:24
  train your models. So in that case, every time you
• 17:27
  pull a batch, you don't have to process the data anymore, so
• 17:31
  that happens once. And if you iterate over the data
• 17:33
  set multiple times, you see some pre-processing time there.
• 17:37
  But the other Oh cool benefit of that is you generate a graph
• 17:42
  from based on all the pre-processing computation.
• 17:45
  So meaning that the the graph contains of like here's a raw
• 17:50
  value, how do you manipulate that Rob value to get to the
• 17:53
  normalized version? And that this is a normalized
• 17:57
  value and that is generated during the training process?
• 18:01
  And then when you deploy that machine learning model, that
• 18:03
  model receives the raw data. But before it actually hits the
• 18:08
  model for the prediction, it has to go through the pre-processing
• 18:10
  graph. So then you would subtract the
• 18:14
  mean, do you mean normalization values or whatever you want to
• 18:18
  do in that moment? And that is a really good
• 18:23
  approach to like keep the training in sync with the CERN.
• 18:27
  And if you use a machine learning pipeline, then this
• 18:30
  happens, We don't even today. We don't even think about it
• 18:33
  anymore. So we have like in our case that
• 18:35
  digits we pre processes our unstructured data in various
• 18:42
  ways. But if you if somebody makes a
• 18:45
  change to the pre-processing step that triggers a reader,
• 18:48
  retraining of the model that triggers a redeploy of the model
• 18:51
  and that changed goes out hand in hand.
• 18:54
  I have this happening in the past, there was working with
• 18:58
  another data scientist. I would suppose once before, The
• 19:01
  machine learning apps that they despise was once before the
• 19:04
  model itself, we had a little glitch in the communication.
• 19:09
  I forgot to mention that as I go, I change, the bedding is
• 19:12
  Generation here and then all the sudden we deploy the model.
• 19:17
  So I want garbage for a couple days and or we saw the garbage
• 19:21
  immediately but it took a couple days to actually figure out what
• 19:24
  actually what happened here and then after we realize that mmm
• 19:28
  that pre-processing is out of Sin from the From the internal
• 19:32
  of the model. So in if you go by those
• 19:35
  connecting those pieces, you basically prevent those
• 19:39
  scenarios from the get-go. I think this, this step of
• 19:45
  machine learning engineering around, curating, your data
• 19:49
  structuring your data and applying the same software
• 19:51
  engineering Precision that you do to model training to data
• 19:56
  pre-processing. It's something that we're not
• 20:00
  just starting to Emphasize in Industry applications like
• 20:04
  yours. It's also something I'm starting
• 20:06
  to see in the actual education of ml Engineers now.
• 20:10
  So for example in the full stack deep learning course that's
• 20:13
  running right now. I'm sure some of the people in
• 20:15
  the community and on this podcast or listening to it or
• 20:17
  are following along with that course.
• 20:18
  There's a big emphasis on like some of the data parallelism
• 20:24
  Concepts that you've talked about some of the emphasis on
• 20:26
  like ways to process data. Not just, in terms of tricks to
• 20:30
  improve. Training, but ways that actually
• 20:33
  enable you to scale your machine Learning System more
• 20:36
  effectively, right? And I think this, this is kind
• 20:38
  of, what is the concept that you're getting to their.
• 20:41
  The question I have is, I see your point about the
• 20:45
  pre-processing and I see your point about how you involve it.
• 20:48
  How do you, what tools, do you actually use?
• 20:49
  How do you actually use the continuous integration?
• 20:52
  Continuous deployment? What kinds of testing, do you
• 20:54
  have for the code itself to make sure that, you know, these
• 20:59
  changes happen in a continuous? Is fashion and you don't need a
• 21:02
  lot of human involvement. Yeah.
• 21:05
  Very good question. I think you touched on a very
• 21:07
  important point. I think the last couple years,
• 21:10
  we made this transition to adopt a bunch of software, engineering
• 21:14
  principles in them in the data science and field.
• 21:18
  I don't know if that is, what is the foundation now, from for
• 21:21
  Emma UPS, I hope more of those software Concepts coming to the
• 21:25
  whole data, science, environmental and data science
• 21:27
  thinking, like I see. Luckily more towards you, for
• 21:33
  example, version notebooks. Like I initially that's a really
• 21:37
  hard thing to like, how you make a change in a notebook and then
• 21:40
  it's hard to dip them. How does it look like an hour
• 21:45
  and our site. So we try to remove humans as
• 21:49
  much as we can. So do walk through an example,
• 21:55
  here we have a pipeline set up the pipeline has like an Point
• 22:01
  to get the data from so that could be like static files.
• 22:04
  In our case it's like a big query, a database.
• 22:08
  We have the there's a there's a statistics generation and a
• 22:14
  schema generation that happens automatically we use 10-spoke
• 22:16
  extended. That's a freebie we get from
• 22:18
  those components. We have the code for the
• 22:22
  transform. That is pure python code.
• 22:25
  We have to use in this case to actually generate the graph, we
• 22:28
  use tensorflow ups for this. That allows us to go on to
• 22:34
  combine those pre-processing graphs with the with them
• 22:38
  trained model but it doesn't have to be the case.
• 22:41
  There's a new component coming out in or has been released
• 22:45
  where you can use pandarus, do the Transformations and that
• 22:49
  code can be tested. So you can have like pure unit
• 22:51
  test and say like hey this is my raw data.
• 22:53
  This is my expected outcome of that transform function, that
• 22:59
  model training. We do in Liberty, we do some
• 23:03
  testing around that as well. Like what do we expect from the
• 23:07
  architecture? The core pieces of young pitcher
• 23:10
  and then the model analysis and that's sort of like highly
• 23:13
  standardized like you have like you have a Max break, you have
• 23:16
  your test data going in that in itself is like is automated and
• 23:22
  then we write custom components to go beyond this.
• 23:25
  Like for example, we have like qualitative data sets and
• 23:29
  quantitative data sets with running them against Every model
• 23:32
  and then we compare them against the last trained or deployed
• 23:34
  model and see the dip. And that generates a report for
• 23:38
  us showing, what's the Improvement in this case?
• 23:42
  And then we push this model into the registry and when this
• 23:46
  happens we get a message in slack say like hey here's a new
• 23:49
  model available. It's not deployed at this point
• 23:51
  because then somebody that's the one human touch Point data
• 23:56
  scientist. Go xion.
• 23:57
  Looks at all the reports be automatically generated, And
• 24:01
  there's a check. Is that is it something we
• 24:04
  expected? Is it better?
• 24:05
  Is it worse? Does like a interpretation of
• 24:08
  those results and if that happens then we make another
• 24:14
  commit as her. I forgot to mention at the
• 24:16
  beginning, those pipeline runs are triggered through CI
• 24:19
  systems. So you have a bunch of trigger
• 24:21
  trigger points like currently we have to make a commit to a repo.
• 24:25
  So that is all all code reviewed to actually trigger the pipeline
• 24:29
  itself, we're looking into. You automatically triggering
• 24:32
  this. Once we have X number of new
• 24:34
  samples reviewed from another data pipeline, but that's
• 24:39
  currently in development. But then once the model is
• 24:42
  produced and registered with the model registry and the data
• 24:45
  scientist has looked at it, then we make another commit and say
• 24:49
  like okay now we want to deploy this model and in that case they
• 24:53
  did, if they decide to just confirms the model, the idea
• 24:56
  every model regenerate out of gets automatically videos on
• 25:00
  confirms That uid confirms the configuration but that's all in
• 25:03
  configuration file. Commits this to repo and then we
• 25:05
  have this guy system which is like figuring out.
• 25:08
  OK, what has changed what version of the increasing number
• 25:11
  of instances machine times Etc. And then handles the deployment.
• 25:16
  So nope, we basically no human can deploy models like manually.
• 25:23
  Everything has to go through a code pieces.
• 25:26
  So I'm fascinated by how you've architected this and where you
• 25:31
  put the human checked in and why you ended up saying, all right,
• 25:36
  everything is going to be automated until it gets to this
• 25:40
  point where the data scientist is going to look at it.
• 25:43
  Like how did you say at this point is when we want the data
• 25:46
  scientists to go look at it and they say it's good then it gets
• 25:49
  pushed out and everything gets put into its database and it all
• 25:54
  gets that. Lineage that Tracked back to it
• 25:57
  but there's it seems to me like there's only one touch point for
• 26:01
  a human in the loop. Right?
• 26:02
  And why did you decide that one touch point is where it is.
• 26:06
  Hi, this is mihail Eric. I'm the founder of pun intended
• 26:10
  Innovation and confetti Ai. And this is a quick Public
• 26:13
  Service Announcement. If you're tired of having
• 26:15
  nightmares about the machine learning models, we have in
• 26:17
  production, then I encourage you to subscribe to the ml Ops
• 26:20
  Community podcast, it's where all the cool ml Ops kids hang
• 26:24
  out and learn how to tame their ml.
• 26:26
  Jules. I think it's this touch this
• 26:29
  it's at this point in time because then we have the model
• 26:33
  fully trained me up the model fully evaluated and the reports
• 26:38
  generated to do that. Extend that we can actually have
• 26:42
  a conversation about the model improvements.
• 26:45
  Like we have a number of pipeline runs where we saw minor
• 26:50
  improvements and we bailed on deploying that machine learning
• 26:55
  model. And then that sort of like a
• 26:59
  group discussion will sell at, hey, be serious.
• 27:01
  Improvement of like less than half a percent or something.
• 27:04
  Do we actually going to go through and spend the
• 27:07
  engineering efforts to backfill or data that would that has been
• 27:12
  also across as we massively improved over the last couple
• 27:15
  weeks? Initially was always a big
• 27:18
  undertaking. So now the burden to re get new
• 27:21
  moms out as much Nest but that's always sort of like, in that
• 27:25
  moment we can Discuss the trade-off between, is it worth
• 27:29
  while to deploy this model or potentially, even discover a
• 27:33
  model which which performance is that decreasing?
• 27:37
  And then we're like, okay we don't want to do this, but I
• 27:40
  know earlier point in time we have all the reports available
• 27:42
  and like all the evaluations and that is probably also the last
• 27:47
  moment before we would create I don't want to say harm by but
• 27:51
  chaos and like maybe mess up some data so that it's like, In
• 27:56
  my mind, as I was like the ideal Point, like we're ready to go,
• 27:59
  but like you're right before the rocket launch, but we're you
• 28:01
  bail on the rocket launch personally, I like that.
• 28:05
  So can you talk to us about the updates you did with the
• 28:09
  backfilling of the data and how you change that?
• 28:12
  Like what was the pre and post? What does that look like?
• 28:15
  Yes, I think I have two digits. We do a live with like a
• 28:21
  different machine learning like in a typical standard machine
• 28:25
  learning case where you do like Computer models and like you
• 28:29
  take every data record and you would run over the entire data
• 28:33
  set again and just recompute everything and in our particular
• 28:39
  case and then maybe we need to talk a little bit about like
• 28:42
  what Y is like this case a little bit different.
• 28:46
  We we not just want to re not just want to recompute
• 28:50
  everything. So we know we don't do pure
• 28:54
  classifications. We do like similarity Machine
• 28:56
  learning. And so, in that case, we have to
• 28:58
  take every record and then generate an event in for this
• 29:01
  and then recompute every embedding to against all the
• 29:03
  other embeddings, clearly that doesn't scale for like a fast
• 29:07
  growing startup. So in that case, those backfill
• 29:11
  processes we had to make some tweaks to do to improve this,
• 29:15
  this speed up to recompute once we ship a new model version and
• 29:21
  so a couple of things will be changed is like the way we
• 29:24
  ingest the way we generate. Those buildings in the first
• 29:27
  place, that's something we highly paralyzed back to your
• 29:29
  point, which news I click making things parallelizable is like,
• 29:33
  so that was the key there. And then when we compute our
• 29:38
  nearest neighbors for our similarity, assessments, we
• 29:44
  heavily optimized, the way we find nearest neighbors for our
• 29:49
  transactions, that I find similar transactions, but given
• 29:51
  transactions. And then the way we, WE Post
• 29:54
  process that So that was like a massive undertaking by the
• 29:59
  amount team, the last couple weeks and all the improvements
• 30:04
  in that process trickle down to all the machine learning Apso
• 30:08
  because that backup process is now much easier and also fully
• 30:13
  automated. We can just basically register a
• 30:16
  new model in our registry that down down process system picks
• 30:20
  it up and then handles everything out of battery from
• 30:22
  like inferring all the nettings and recomputing all the nearest
• 30:25
  a So before we go further, I do want to take the opportunity to
• 30:29
  ask for our listeners benefit. What is it that digits actually
• 30:33
  does? What's the product you guys
• 30:35
  build? What is it?
• 30:36
  You do and, and how did you kind of get their totally linked?
• 30:41
  So digit is a, is on the mission to build like a new set of tools
• 30:46
  or the next generation of Finance schools.
• 30:49
  We're able to achieve this by like using machine learning or
• 30:53
  learning left. Right?
• 30:53
  And center. And we have a number of machine
• 30:55
  learning. And we sort of like sort of like
• 30:59
  summarize them as we call it like a living model.
• 31:03
  So what that means is like we ingest the books of small
• 31:06
  businesses and then for the we Empower and countenance and the
• 31:11
  business owners through the insights from digit.
• 31:14
  So, normally, when you're a small business, you have an
• 31:17
  accountant and the company comes into your books, once a month,
• 31:20
  classifies them, and I gives you a Pinot.
• 31:23
  But by the time, the accountant actually takes a look at it.
• 31:26
  The books already updated because you might have hired
• 31:28
  somebody else or you might have had another major expense in
• 31:30
  your pianos and they're always laughing, right?
• 31:33
  And so what we what we're doing it, did you just like making
• 31:35
  this real time? We look at the books, make a
• 31:40
  bunch of decisions and working together with the accountants.
• 31:43
  I think that's really critical of like it's not like we were
• 31:47
  placing the accountant, it's like we're enabling.
• 31:49
  They comes to be faster and to make decisions in real time for
• 31:53
  the business owners and that's something we do through.
• 31:56
  So we read the unsorted data, we get from the banks or like their
• 32:00
  accounting platforms. Ingest them process them with
• 32:04
  machine learning extract additional information, annotate
• 32:07
  the information or like find similar.
• 32:09
  Transactions, suggest category, suggest vendors things like
• 32:12
  this, to make all the things which the accountants are the
• 32:17
  business owners need a realtor and accessible.
• 32:20
  So instead of waiting, until the end of not of the month, you can
• 32:23
  go every hour and you get an updated view of.
• 32:25
  Like how is it, is that right now?
• 32:28
  I just think that, I mean, Dimitri is very small business
• 32:31
  owner. You've understood these kinds of
• 32:33
  tools, you tell me how you feel about it.
• 32:35
  Well, I was laughing because I was like, oh my brother is super
• 32:38
  lazy and he's an accountant. I gotta tell him about this tool
• 32:41
  who he's gonna love it. So is we have a we have a tool,
• 32:48
  it's called digit boost and that is heavily used in machine
• 32:51
  learning because the accountants keep doing their work in but
• 32:55
  digits boost his base. Watching the back, so it helps
• 32:57
  them save automating time. It helps them.
• 33:00
  Save time by automating. There were close, but at the
• 33:02
  same time, it's sort of like, watching that back.
• 33:04
  So if they categorize something or a sign of injured to
• 33:07
  something, and we have machine learning models to figure out
• 33:10
  was like, hey, this is looking odd like you the ten times in
• 33:13
  the past, he said, like, this should be, I don't know venmo.
• 33:16
  And now you saying this is PayPal, and we'd spend the
• 33:19
  middle of notifications that I hate.
• 33:21
  Are you sure about this? Or we can even we can go the
• 33:24
  other way around like that. Me.
• 33:25
  And I'm just like, hey, we think this is demo for you and then
• 33:29
  they can just say sure, I'll take that.
• 33:32
  So it's like a it's a perfect use case.
• 33:35
  I think it's a perfect use case for machine learning.
• 33:37
  It's a little bit underrated use case, but everybody thinks about
• 33:40
  self-driving, cars, and, and cool computer vision Tech but
• 33:45
  every small company runs on an accounting books.
• 33:48
  Everybody has your books, otherwise they get in trouble
• 33:51
  with the local tax agencies, right?
• 33:55
  And Perfect. Use case for machine learning.
• 33:58
  It does have an interesting twist.
• 34:03
  Every classification is highly subjective and I think it's
• 34:05
  solving this with machine learning is super interesting
• 34:09
  and I'm super excited and working on this problem.
• 34:14
  I mean, first off, I just think it's such an interesting
• 34:17
  application. Like, as you said like, you
• 34:20
  know, like cash management counting, like, working for
• 34:23
  anybody, who's worked at a start-up, you know?
• 34:26
  Like the way that like, you know, and that's just one flavor
• 34:29
  of small business and its high gross and whatever the heck
• 34:32
  else. But like for every business,
• 34:34
  there's all this like Arcane stuff that goes into like how
• 34:36
  you actually manage your cash, think about finance and it has
• 34:39
  an implication on everything, right?
• 34:41
  And so the idea of taking that and bringing it from, I don't
• 34:45
  know like the mid-twentieth century of like accounting and
• 34:48
  the way that it worked with like what is it credit on left?
• 34:51
  And on the right is something like that.
• 34:52
  I don't even know to to Really applying, you know, advanced
• 34:56
  Machine learning is it's super interesting.
• 34:58
  And, you know, I looked at the website for the company.
• 35:00
  I encourage everybody listening. Check it out.
• 35:02
  It's a really, I feel like once you look at the website, which
• 35:06
  will put in the show notes, you'll get a much better
• 35:08
  understanding of like, wow, this really makes sense for why, you
• 35:12
  know, machine learning can be used to solve this problem now,
• 35:16
  with and I think I love you. Go ahead.
• 35:18
  I think that it's hilarious that now you all are probably doing
• 35:21
  better than ever because of that bill that just got passed in the
• 35:24
  US where they're Higher whatever. 80,000 new IRS 80
• 35:30
  billion dollars for the IRS to. Yeah that's it.
• 35:33
  Eighty billion dollars enforcement.
• 35:34
  So everybody go go get digits and make sure you're not messing
• 35:37
  up. Yeah, this is free.
• 35:42
  So you don't have to well, this is game over.
• 35:47
  Where do I sign up? Yeah, the question I had though
• 35:51
  is you were the first time I'll engineer there You were the
• 35:55
  first person to start implementing machine learning or
• 35:59
  I'm just kind of, I'm so curious, like I've been the
• 36:01
  first data scientist, you know, at my company right now.
• 36:04
  And so, I have some understanding, but for a company
• 36:07
  that is so premised on the notion that like machine
• 36:09
  learning is, you know, one element of our secret sauce.
• 36:12
  Like, can you talk us through like what the first few months
• 36:15
  of your job were like what kinds of things were priorities for
• 36:18
  you? And how you spent day-to-day?
• 36:20
  Yeah, I have to admit. I'm yes.
• 36:23
  And the first She's learning engineer and the first person
• 36:26
  who's like totally focused on machine learning but I'm
• 36:28
  standing on the shoulders of giants.
• 36:30
  There's a there's a team. There's an amazing team of
• 36:33
  Engineers. They have been there for 20 plus
• 36:36
  years, they developed amazing data engineering infrastructure.
• 36:42
  That was already there when I arrived.
• 36:44
  So, when I spoke to the founders Geffen, Wayne they said, like,
• 36:49
  hey now is the right time to do machine learning because we
• 36:51
  spend all the time to get the data in place.
• 36:53
  So, in The previous jobs when I showed up, we talked a lot about
• 36:58
  machine learning, but then when I arrived, it was like, oh, we
• 37:00
  need to give it in place and you clean the data.
• 37:02
  We need to do this business. This was already in place, so
• 37:06
  there's like the team. The core team had previously
• 37:11
  founded in worked on crashlytics precludes.
• 37:14
  Got I think God bought by Twitter, Twitter ever sold this
• 37:17
  to Google and so the core team had like a wealth of knowledge
• 37:22
  about how to do data and Knowing at the scale of Google and this
• 37:26
  little Twitter. So for me, it coming in as a
• 37:30
  machine-learning Engineers, like this is the ideal place and I
• 37:35
  because, like the data is in a really good shape.
• 37:39
  They have amazing data engineering principles so I can
• 37:43
  come in and just put the tiny cherry on top of them.
• 37:45
  Amazing pie, like to really build like models.
• 37:49
  I can fully focus on building the models and building the
• 37:52
  infrastructure around how we We keep those models updated how we
• 37:57
  train them, how we actually use machine learning to the vast
• 38:00
  extent. What does that look like when a
• 38:04
  company has their data in good shape?
• 38:07
  Like can you show a tell us what some of those green flags were
• 38:11
  as you came in and you're just like whoa this is really nice.
• 38:14
  Yeah, I think it comes out in a couple things so you can ask
• 38:20
  questions around the data and somebody just like hate, just
• 38:23
  write it. Write a query for this.
• 38:24
  That's right. It's not like he's in some
• 38:27
  unstructured text files and we need to parse them first in that
• 38:30
  sort of stuff. You have a team who is highly
• 38:34
  educated in terms of like how to process the data in a parallel
• 38:38
  fashion. Like they were already using
• 38:42
  Apache Bean Google Cloud. Dataflow for two plus years.
• 38:46
  So there was like a whole wealth of knowledge I could tap into of
• 38:50
  like how do we batch process something?
• 38:53
  How do we stream? Of things.
• 38:56
  So in that moment, we only had to build the connector pieces
• 38:58
  between the machine learning models and the existing
• 39:01
  pipelines already. So it's the pipelines were
• 39:04
  already in a good shape. They often had like basically
• 39:12
  models in place already, who would do some work.
• 39:16
  What we then replaced with like deep learning models or that's
• 39:20
  always a good good point of like, if you already have like a
• 39:24
  base, Model present. Then it's a good moment to do
• 39:28
  like deep learning. Give it a soft, like try out a
• 39:32
  fancy model and see does it actually boost your accuracy
• 39:35
  rate? And in this case, I when I
• 39:38
  showed up, they had already done all the all the statistical
• 39:41
  machine learning models, all the approaches that already
• 39:44
  happened. And in that case, I was like,
• 39:47
  okay. Now I can actually bring in my
• 39:48
  expertise and like how to build deep learning models and how to
• 39:51
  build machine line pipelines to keep them fresh and updated.
• 39:53
  And so on. So we don't Aunt, in a lot of
• 39:57
  places, when machine learning Engineers are being higher than
• 40:01
  they started with a baseline models.
• 40:03
  And here I showed up and they resigned mom's priority system,
• 40:07
  That's why I'm every day I come to work.
• 40:10
  I think I am standing on the shoulders of giants because they
• 40:13
  that team is just amazing. That's so cool to hear.
• 40:16
  I was listening what you're saying and taking notes in my
• 40:20
  head, so three things. First data, structure data in
• 40:24
  good shape right was good Tools around it.
• 40:26
  S a good perspective on how to do data processing because I
• 40:30
  feel like people really Vibe with that because I think a lot
• 40:33
  of people they are like, okay, we can Get our data structured
• 40:37
  for our first iteration. But updating the data.
• 40:39
  How do you do that, right? And then the third piece is just
• 40:42
  a team that's educated on these Concepts right and just knows
• 40:45
  how to think about it and I think it's easy to point.
• 40:50
  Variety of reasons, we have like the team itself is highly
• 40:55
  interested in machine learning and machine learning is not a
• 40:59
  siloed department at digits. We work in conjunction with like
• 41:05
  back and enjoy. Here's data, Engineers, funded
• 41:07
  engineers and designers, and I think the connection between
• 41:11
  machine learning and design is incredibly critical.
• 41:15
  They usually happen in siloed departments, and so they may be
• 41:20
  be once a year at the Holiday party and that's when some
• 41:24
  knowledge transfer happens, but in our digits, we work with
• 41:29
  designers working on the team. So when we push out a machine
• 41:32
  learning model, then we have like a conversation with a
• 41:35
  designer. Like, okay, how does it actually
• 41:37
  affect the design and vice versa?
• 41:39
  How does the design affect the model and how does the model
• 41:41
  affect the design? And because we have seen this a
• 41:45
  previous jobs when I push the model out and that Loop was
• 41:50
  missing that connecting pieces missing.
• 41:51
  So we spend months and months and months to train like a
• 41:54
  perfect model and then it was misunderstood by the us over the
• 41:58
  user not understanding what the implications of the modeler in
• 42:01
  the click the wrong buttons and the generated coverage data and
• 42:04
  like all of a sudden you're like aw damn it.
• 42:06
  And that's sort of like in my mind is often the one of the
• 42:09
  core pieces. Why machine when he fails is
• 42:12
  sort of like this integration with design and how do you real
• 42:15
  users interact with a machine? Like Mom's what are the real
• 42:18
  world consequences once the monitors are because like any
• 42:21
  system adopts to changes within the system and it's the same for
• 42:26
  machine learning. That's like a.
• 42:28
  Once you start predicting something people starting how
• 42:30
  can I tweak the system? Like, how can I turn this into
• 42:33
  my benefits and the other thing? Went I think it's odd.
• 42:38
  What we should talk more about like in the whole ml Ops
• 42:41
  Community, I think that the real machine learning happens once
• 42:45
  the model is trained for the first time.
• 42:48
  Like scaling machine learning models is a big, big deal and I
• 42:52
  think we still don't talk enough about that like it's not just
• 42:55
  about serving a machine learning model.
• 42:56
  But like scaling the the usage of the machine.
• 43:00
  I'm also recently, we we have a tremendous growth at digits.
• 43:04
  Like we already, we ingest Millions of transactions, every
• 43:10
  new transactions every day. Like this, like it's a perfect
• 43:13
  place for machine learning Engineers like it's small, it's
• 43:15
  like being a really good team but we have a lot of data.
• 43:19
  And so we often see like interesting things around like
• 43:24
  we have the model model deployed.
• 43:25
  This is all on auto scaling. We can we can scale those
• 43:28
  operations but then also we need to scale our pipelines in
• 43:31
  conjunction with our machine learning models in to get the
• 43:33
  optimal throughput. There that is a We interesting
• 43:37
  question right now, and I there's a lot of learnings on
• 43:42
  our end up like, how to scale machine learning how to get an
• 43:47
  optimal throughput through the systems after the most point?
• 43:53
  I had a quick quick dot here that I wanted to share first
• 43:56
  off. Are you guys hiring on the
• 43:57
  machine learning side because this is like a running
• 43:59
  advertisement. This is like, yeah, it's like a
• 44:03
  gradual rat. I long 45 in a glass door at for
• 44:06
  working at ditches, all positions from and back-end data
• 44:12
  engineering, machine learning anything.
• 44:14
  That's also. Okay, cool.
• 44:16
  Alright. Well, everyone now, you know
• 44:17
  what a higher and I know where to look to get hired.
• 44:22
  The second thing, And say is I think something I've been
• 44:26
  thinking about in my head as kind of been the evolution of
• 44:30
  our industry from working on ML models to products to systems.
• 44:36
  Right? And I think you know, between
• 44:40
  You know, twenty fifteen to Seventeen, right?
• 44:43
  Like maybe 18, like everything was all about like working on
• 44:46
  like fancy models, right? Like just putting out a new kind
• 44:50
  of deep learning model, or something like that, right?
• 44:52
  And now between 18 and maybe around 20 22 or so, like, we've
• 44:57
  started to think a little bit more like, okay.
• 44:58
  Well, what does it take to actually build an amount
• 45:00
  product, right? And I think we're still thinking
• 45:03
  about that and now with some of the more, some of the
• 45:06
  Integrations with data, engineering, some of the, some
• 45:09
  of the changes in data engineering.
• 45:10
  Themselves and software, engineering like the systems,
• 45:13
  the systems level engineering that you can do not just a
• 45:15
  Google and Facebook can do with oodles, and oodles of dollars
• 45:18
  and Engineers, but everybody can do it now, right?
• 45:20
  And that's what's really exciting to me is to see that
• 45:22
  and like, digits is an example of that.
• 45:25
  Absolutely, I think miss the machine learning field.
• 45:29
  We have this struggle right now than a lot of things for had to
• 45:32
  learn. A lot of things were sort of
• 45:34
  like marketing driven was like, Hey, we're in AI company, right?
• 45:39
  I think Like just because you produce a machine learning
• 45:42
  model, doesn't make it any I company.
• 45:45
  Like when you need to get to the systems point of the what you
• 45:48
  said like then I think we can say like yes this is an ml
• 45:52
  German company or like we're data-driven or like machine
• 45:55
  learning powered or something like this.
• 45:57
  But we have to get to the point of systems of the system's
• 46:00
  approach and just training. A machine learning model, just
• 46:04
  deploying this doesn't make it a system, it does make a system
• 46:08
  when like in Control Systems approach, like typical systems
• 46:13
  engineering. Like you have a black box, it
• 46:15
  gets an input against an output and then you have a control
• 46:18
  parameter in for our machine learning systems, that control
• 46:21
  parameter is a feedback loop. You need to give your users a
• 46:26
  way to give you feedback around your machine Learning System at
• 46:31
  any given point in time. So, for example, we have like in
• 46:34
  the product we have a tool which is sort of like, making
• 46:38
  recommendations of like We think this is from in your co-captain.
• 46:45
  The users can click on a little button and then see, why did we
• 46:49
  make that suggestion? So we built a machine learning,
• 46:52
  explain ability right into the product and it shows you like
• 46:56
  something where you really expect from sharp or shape.
• 46:59
  Sorry, I'm always mispronounce this cool.
• 47:02
  I'm like why in this particular spring?
• 47:05
  Why did L machine? What did our machine learning
• 47:07
  model fire on? and there's like directly
• 47:11
  integrated into the product and if people have a problem with
• 47:14
  that they can just click on a little button, we get a
• 47:16
  notification and we going to analyze the particular tool and
• 47:20
  we look at all the actions, use a users do based on our
• 47:24
  recommendations and that is the feedback loop only critical and
• 47:27
  all the thinking about them Ops So just piggybacking off of that
• 47:33
  design decision that you took to say, we're going to have
• 47:36
  explained ability built into our machine learning model.
• 47:39
  I imagine that is because of the conversations that you have with
• 47:43
  the designers. And I'm wondering if you have
• 47:46
  any more examples of how conversations with designers has
• 47:52
  changed the way that you have built machine, learning products
• 47:56
  have to admit. The design team is amazing.
• 47:58
  Like we have Wonderful designers and they have a very, very
• 48:02
  strong interest in machine learning.
• 48:04
  That's that is the first ingredient to like a really cool
• 48:07
  collaboration that the machine learning folks have.
• 48:10
  It is have an interest in design.
• 48:11
  I think this is happens naturally, like just because
• 48:14
  your eye is like working with you and your like, whenever you
• 48:17
  see new designers on this automatic appeal to like it.
• 48:21
  Check it out. And so, it's much harder for the
• 48:24
  design team to have an interest in the machine learning
• 48:26
  operations, under the use case of machine learning, but I think
• 48:30
  That happens automatically for the for the folks on the team.
• 48:33
  So that's really great. We will never have to drag them
• 48:36
  in the conversations. They're already like they're the
• 48:39
  part of the decision process and the conversations in terms of
• 48:42
  the actual conversations. I think that these conversations
• 48:46
  going both ways, like from the machine learning side, we have
• 48:51
  conversations around what type of feedback loops do we need?
• 48:56
  To make our machine learning model successful.
• 49:00
  And then the designers come back was like, hey, I think this is
• 49:03
  how we could integrate this into the ground like we coined the
• 49:08
  phrase of like we have this phrase of a living model and the
• 49:13
  design is basically like lifting the covers and like we give
• 49:19
  people a sneak peek into the brain of like what's actually
• 49:21
  happened behind the scenes photos of a model.
• 49:23
  So and those conversations Going both ways like from Informatica
• 49:28
  in all sizes, like, this is what we need and the designers of
• 49:31
  like hey this is what we could write.
• 49:33
  But also there's another thing. We're like, let's say you have
• 49:36
  an onboarding flow. The designers could come in with
• 49:39
  questions of like, hey do we actually need to ask this
• 49:43
  question into it during our onboarding or could be predicted
• 49:47
  in some way. We might not be always correct.
• 49:49
  But like, can we make this onboarding flow simpler?
• 49:52
  And this those are sometimes questions begin from the design
• 49:55
  team. And Then sometimes the answer is
• 49:57
  yes. And sometimes the answer is no,
• 49:59
  and its really good conversation and this is something what I've
• 50:02
  seen in. Other companies, sort of like
• 50:03
  missing where the designers are, not even aware of like what
• 50:06
  machine learning can provide to the design itself?
• 50:10
  But did you have to do specific, like data literacy, or machine
• 50:13
  learning literacy to get them up to speed?
• 50:16
  Or were they just interested in? They started coming to meetings.
• 50:20
  Where you all would interact it? Sorry.
• 50:24
  It's a little bit of bonus and so We are determined to we work
• 50:27
  and what we call workspace and The Works being that covers a
• 50:31
  particular feature. And in this feature would like,
• 50:33
  or in this works for him. We are paired a bunch of machine
• 50:36
  learning folks up here at with a designer and back-end and
• 50:39
  front-end NGO. So every time we meet we like we
• 50:41
  get a realistic view a couple times a week.
• 50:45
  Just on the side note link I think digits are some cool
• 50:47
  things around like how we organize things and how we work
• 50:50
  remotely but only that. So in that point we always have
• 50:55
  Like multiple touch points during the week when we actually
• 50:58
  get to talk to the designers and the designers, can we might talk
• 51:02
  on the side about like something machine learning related in the
• 51:05
  design is like, hey, wait a second, can be used to design,
• 51:07
  so that happens naturally. But then when we bring up a new
• 51:11
  new Concepts, like for example, similarity learning, what does
• 51:15
  it actually mean for the product?
• 51:17
  Then sometimes we have like in brown bag sessions and the
• 51:21
  designers always part of that. And they would learn about like
• 51:25
  This is, how does that actually work?
• 51:27
  And then this would like a natural conversation happens
• 51:29
  afterwards. Like how can we actually use
• 51:31
  those things in in the product? And so, those brown bag sessions
• 51:36
  happen at other companies, as well.
• 51:38
  Totally, but they usually just happen within the machine
• 51:41
  learning team to educate other machine learning engineers.
• 51:44
  And then they have to wait for the next holiday party to
• 51:46
  actually convey this message to to the design team.
• 51:51
  So this is so good man. And it really shows me the power
• 51:55
  of like that hybrid team layout. And how if you just get the
• 52:01
  ability for these teams to be able to interact in a nice way
• 52:06
  together, it's going to create beautiful things and it you're
• 52:10
  going to see pieces that you as machine learning engineer,
• 52:14
  you're very focused on subset of problems and when someone else
• 52:18
  comes with a different perspective, I'm sure.
• 52:21
  There's been a lot of times where they've opened up your
• 52:23
  eyes to how you can use something differently, or
• 52:26
  potentially you're teaching them, and then they come with
• 52:29
  questions. Like could we do this?
• 52:30
  Like you were talking about with the predicting the onboarding
• 52:33
  flow. So I love hearing about that.
• 52:36
  I also, I'm really like, I think that you have stated it so
• 52:43
  clearly the importance of like data hygiene and not just the
• 52:48
  machine learning level. Like if What you were talking
• 52:51
  about? That foundational level is not
• 52:53
  there, it doesn't matter how good of a machine learning team.
• 52:55
  You have, you're not going to be able to go places because you
• 52:59
  need that data hygiene on the very, very basis of the the
• 53:04
  whole pyramid or hierarchy of needs, right?
• 53:08
  So, I got one last question for you man before we jump off and
• 53:13
  it is a little bit of a rapid fire round.
• 53:15
  We could call it. What was the last book?
• 53:18
  You read? Because I see in the background,
• 53:20
  there's nothing but Riley books on your bookshelf, so you only
• 53:26
  see half of the books table. So there's a some yeah.
• 53:31
  None non already performed as well.
• 53:34
  What? So, what was the last one you
• 53:36
  read, though? O'Reilly or not?
• 53:37
  It doesn't matter as a technical book.
• 53:41
  I think what's currently on the nightstand is like, the atlas of
• 53:44
  the eye, is a really great introduction about like what are
• 53:49
  the consequences of In learning. Or in in parts, you might not
• 53:55
  have expected like environmental consequences, for example, may
• 53:58
  be consequences. Yeah.
• 54:00
  Like consequence, don't like politics democracy.
• 54:05
  Highly, highly eye-opening book. Like I highly highly recommend
• 54:10
  this this book and On the non on the fiction side been, if they
• 54:21
  hadn't carried a book for me with me for the last 20 years,
• 54:25
  just by unpacking in a box. The other day I found like a
• 54:30
  Fury from Salman Rushdie like before all the news happen.
• 54:36
  But that's basically on my nightstand is a fiction book
• 54:39
  right now. Incredibly nice.
• 54:42
  Last one for you. How do you want to be
• 54:44
  remembered? Ooh, that's a good one.
• 54:53
  Hmm. That's a good one.
• 54:57
  I would love to like I don't know something around like His
• 55:01
  jokes are City but they were funny, like they're odd, but
• 55:05
  funny that's something I could get behind.
• 55:09
  That's that is mine too. Well honest, man, this has been
• 55:14
  eye-opening and I appreciate the transparency.
• 55:18
  And the level of just openness that, you've been able to get to
• 55:22
  with us. I love when were able to pick
• 55:24
  someone's brain and there, they tell us everything.
• 55:26
  You broke down your whole stack, maybe later on, we will create
• 55:29
  some kind of diagram. The listeners out there and have
• 55:34
  it like what your stack looks like and and I just think it's
• 55:38
  fascinating. This idea, there's two huge
• 55:41
  things that I loved one was the data hygiene and the other was
• 55:45
  like have a baseline model in place before you go and you do
• 55:49
  your deep learning like really cool, Cutting Edge stuff.
• 55:52
  Make sure that you have your bases in check, so this has been
• 55:57
  awesome. Visually you got any last?
• 55:59
  Final, thoughts. Closing thoughts?
• 56:01
  People. Go work at digits, man.
• 56:02
  Go work at this. That's all I got.
• 56:05
  Is it? That's it.
• 56:06
  We'll leave the yeah, exactly. We'll leave that careers page
• 56:11
  four digits in the description in case anyone wants to go and
• 56:14
  apply. It is a good place, indeed, and
• 56:18
  if anybody has questions around the stack and how we solve
• 56:21
  problems, feel free to reach out, we happy to answer them.
• 56:24
  That's another thing about machine learning, you can, I
• 56:26
  think you can be really open about, like, what is actually
• 56:28
  happening behind the scenes because even then, it's To
• 56:31
  replicate the things for folks because you don't have the data.
• 56:35
  So it's like yeah. And there's a lot of like
• 56:38
  learnings we can share with other people and they can apply
• 56:41
  them to their business problems. Because at the end of the day,
• 56:44
  we trying to solve business problems, not like writing pens
• 56:47
  papers, and we have to share information.
• 56:51
  So feel free to reach out. Yeah.
• 56:55
  Last call to action for everyone.
• 56:56
  Listening, go and check out his book.
• 56:59
  Building machine learning pipelines.
• 57:01
  And we will also leave a link to that in the description.
• 57:05
  That's all we've got for today. Thank you for having me.
• 57:07
  Was that was a blast.