Latest / Agentic Conversations (formally mlops.community) / MLOps Meetup #36: Moving Deep Learning from Research to Prod Using DeterminedAI and Kubeflow // David Hershey, DeterminedAI

Transcript

MLOps Meetup #36: Moving Deep Learning from Research to Prod Using DeterminedAI and Kubeflow // David Hershey, DeterminedAI · Agentic Conversations (formally mlops.community)

• 0:02
  We got a great Emma lap speed up for you and that is our intro.
• 0:13
  Thank you very much. I appreciate it.
• 0:16
  Excited to be here. There we go.
• 0:19
  I don't think you're going to get an intro quite like that.
• 0:25
  Add any other meet up that you guys can't tell you the last
• 0:29
  time I've been Marinated, so I appreciate it.
• 0:33
  Alright let's, let's get real. Let's get started.
• 0:36
  It looks like we got people here.
• 0:38
  Welcome everybody. In the in the house we have
• 0:43
  David as you know, he is coming here for a little chat today,
• 0:50
  and we're going to be talking about some great stuff.
• 0:53
  I'm excited about how we're doing this.
• 0:57
  But first, before we even Anything, you know I got the
• 1:01
  announcements to make first one is, if you're not in slack, what
• 1:05
  are you doing? Get in it.
• 1:07
  Get in our slack Channel, you will not hear more of me,
• 1:10
  serenading you but you will have lots of stimulating
• 1:15
  conversations happening and all kinds of good questions that are
• 1:19
  being answered, when it comes to the whole ml Ops Universe.
• 1:26
  We could say so Hopefully, you can jump in it.
• 1:31
  We have the invite link. I hope it's not Chris.
• 1:35
  I just want to mention that. One of the invite links is
• 1:39
  broken. Hopefully, this is not the
• 1:40
  broken one. A lot of people have been
• 1:42
  complaining about that one because we pasted the invite
• 1:46
  link all over the Internet and slacks, had this invite link
• 1:49
  will never expire and then it expired.
• 1:53
  Now, I know we could I could just play guitar We could
• 1:58
  actually sing this whole meet up, but David I think you got a
• 2:04
  lot of good stuff that you want to tell us and I don't know how
• 2:07
  well your voice in tones over the internet I'm sure you don't
• 2:12
  want it I promise so man it's great to have you here I think
• 2:19
  everybody heard the intro but if they don't know who you are
• 2:23
  you've got some street cred. You working at determined Ai and
• 2:26
  today we are going to to be talking about moving deep
• 2:30
  learning from research to production with determined Ai
• 2:35
  and Cube flow. Anytime that we talk about Cube
• 2:38
  flow, people get really excited. I want to I want to start before
• 2:47
  we talk about any of that where I normally start from and just
• 2:51
  give us a break down. How did you get into Tech?
• 2:55
  Yeah, I took a ride. I ended up.
• 2:58
  So I started doing aerospace engineering because I like the
• 3:00
  airplane still do. That's my undergrad, but I ended
• 3:04
  up doing was like, writing code, the whole time.
• 3:07
  I was doing aerospace engineering and at some point I
• 3:09
  realized I liked the code or than the aerospace engineering
• 3:12
  so and the following that along a path to machine learning at
• 3:17
  the time, still out of interested in seeing if we could
• 3:20
  make planes that fly themselves better but then I steered just
• 3:25
  to machine learning was my passion and love it.
• 3:28
  I mean I got into the space at Ford and an awesome job.
• 3:31
  They're working in the Deep learning space.
• 3:34
  I'm just like a huge wide range. Of applications was really lucky
• 3:37
  to be able to touch a lot of different things and had a lot
• 3:40
  of freedom to really like learn and grow different types of
• 3:42
  models and through that just ended up focusing, a ton of time
• 3:49
  on the infrastructure, supporting machine learning.
• 3:51
  So just found out how much there is and how much need there is
• 3:56
  which I which is why this community exists.
• 3:59
  And started chipping out problems one by one there and
• 4:04
  ran into determined in the process of that and ended up
• 4:06
  here. But yeah, so it's been a it's
• 4:08
  been a path. I sort of been following the
• 4:09
  wind's really taken me through this but I am pretty happy with
• 4:14
  where I'm at now. So cool.
• 4:17
  So let's here's how I kind of structured, the next hour in my
• 4:22
  head and you tell me if this works for you, I wanted to talk
• 4:26
  about what determined AI is real fast, just so everyone can have
• 4:29
  a bit of a background. I wanted to talk about a little
• 4:32
  bit about Cube flow and what that's doing and just general
• 4:36
  chatting on Cube though, then talk about how these Two Worlds,
• 4:39
  Collide Cube flow and determined and then the idea of research
• 4:44
  machine learning to Action. Yeah, and if we have any time at
• 4:50
  the end, just some general questions whoever is with us
• 4:53
  right now. Feel free to ask any questions
• 4:57
  you have in the chat or throw them in the Q&A section or if
• 5:01
  you're really feeling lucky or like, you want to get involved,
• 5:05
  you can unmute yourself and ask away.
• 5:08
  So I'll try and be a little bit on top of my game to make sure
• 5:14
  if anybody wants to ask anything.
• 5:16
  You can, and you can just raise your hand and then I'll let you
• 5:18
  go for it. So, let's start with determined
• 5:22
  Ai. And can you just give us a
• 5:25
  breakdown of what it is? Yeah, so determined is a open
• 5:30
  source, deep learning train platform.
• 5:32
  So unpacking that a little bit, we're talking about deep
• 5:36
  learning training. I mean, what we focus on is
• 5:38
  really the experience around building and training models and
• 5:44
  so we provide a Little tools to make that process a lot easier
• 5:49
  and also let teams and just data scientist scale of what they're
• 5:53
  capable of when they train models.
• 5:55
  And so when you install determined the first thing you
• 5:59
  do, is it installs on some sort of compute cluster that cluster
• 6:02
  can be your laptop that can be a GPU box you got somewhere, it
• 6:05
  could be a large-scale hundreds and hundreds of gpus and then
• 6:11
  you start describing the right machine learning models the same
• 6:14
  way you have before pie. Torch tensorflow.
• 6:17
  Yeah where people are comfortable and then determine
• 6:21
  will take that and you can do things like just run training.
• 6:26
  You can run distributed training, we make it really easy
• 6:28
  to scale up to large-scale distribute training.
• 6:31
  We have tools for hyperparameters, search based on
• 6:34
  the OSHA algorithm that was invented by one of our
• 6:39
  co-founders and you know all of those come together to make it
• 6:43
  really easy to take models. And train the more efficiently,
• 6:47
  train them faster, train them better and all the while keep
• 6:51
  track of the work you're doing and and all that kind of thing.
• 6:55
  So I've got some some demo materials available for this
• 6:59
  whole talk that we can get into. So I'll be able to show it to
• 7:01
  you in a minute, but that's right.
• 7:03
  Yeah, three. So let's just, let's focus on
• 7:07
  this because I know a lot of times it gets a bit murky when
• 7:12
  people want to compare tools. Yeah, you give us an idea of how
• 7:17
  determined AI is different than some more standard artifact
• 7:23
  storage. Like ml flow is the first thing
• 7:24
  that comes to mind. Right?
• 7:26
  Yeah, so that's a great comparison and I guess I'll
• 7:31
  start by saying no front when at the other piece of being a train
• 7:34
  platform is just like we don't really mess in the data space.
• 7:37
  We're not really talking like Pachyderm are friends of us that
• 7:40
  work Upstream that kind of thing, right?
• 7:42
  And then Downstream we don't really do that.
• 7:45
  Inferences space, very much. So we're not hosting models, run
• 7:48
  serving models, like Selden, we're not doing batch inference
• 7:50
  for not spark provider, that's your stuff, right?
• 7:52
  So really in training and then, you know, comparing ourselves to
• 7:55
  something like ml flow is a good comparison or you can think of
• 7:59
  something like weights and biases I think is similar in
• 8:01
  that space. That let you track artifacts and
• 8:04
  track your training and things like that.
• 8:07
  They are both tools to get a little bit, technical that give
• 8:10
  you like apis. That you add to your code, you
• 8:13
  say, track this metric test, this metric track.
• 8:15
  This artifact then they all get uploaded to some server
• 8:18
  somewhere and track it, right? And so you just run your
• 8:21
  training code with them and then they track this stuff and some
• 8:25
  of those tools. Also provide utilities to
• 8:27
  actually run your code determined is a slightly more
• 8:32
  holistic solution in the training space we get a little
• 8:36
  bit more intimate with your code.
• 8:38
  We ask you, what is your model? What is your Optimizer?
• 8:40
  What is your, how are you loading data?
• 8:44
  And then we end up I'm taking that and actually optimizing it
• 8:49
  for the cluster for you. Letting you run sophisticated
• 8:52
  experiments without you having to rewire things and so and also
• 8:56
  I should say the artifact, tracking the metric tracking.
• 9:00
  Keeping track of what I prime was using that stuff all of this
• 9:02
  sort of falls out when you tell us all that stuff.
• 9:04
  So we're the ones that actually, you know, say train the model
• 9:08
  for some number of steps but then we track all the artifacts
• 9:10
  associated with that automatically and it's not need
• 9:15
  that Add in extra tracking and stuff like that.
• 9:16
  We just thought I'll falls out of working with determine nice.
• 9:21
  So, one question that I think is good to note, too.
• 9:24
  Is like, where are the boundaries between what your
• 9:27
  tool, determined? AI does really well and where it
• 9:31
  works smoothly with other things around it.
• 9:35
  Yeah, so the sorry just to preface this a little bit.
• 9:39
  I know that right now with like DVC came out with a few months
• 9:43
  ago now with like 1.0 right And some people were asking in
• 9:47
  slack. How it seems like you're moving
• 9:50
  into ml flow territory? Is it worth it to use both of
• 9:54
  these or should I just choose one or the other?
• 9:57
  And so, I'm wondering like, how do you do you balance that
• 10:01
  situation? Yeah, that's a great question.
• 10:04
  I think you could use ml flow and determined like you could
• 10:07
  add all those API calls in. I think you'd find out it was
• 10:10
  pretty redundant when she got there though.
• 10:12
  By the point where you're training your models with
• 10:14
  determine all the stuff You want to track an ml flow is
• 10:17
  automatically going to be tried for you and determined and are
• 10:19
  gonna be able to see it. So while you could interface ml
• 10:23
  flown determined, I think I'm a like determined is really a
• 10:26
  pretty overlapping solution. And I'd be surprised that that
• 10:29
  was something that you ended up actually wanted to do when she
• 10:31
  got there. You know, on the other ends I
• 10:36
  think there's a lot of clean room for Integrations and that's
• 10:38
  some of the stuff we're going to talk about today.
• 10:40
  Whether it's an inferencing system, we provide, you know,
• 10:43
  API level access to the results of Training so that you can
• 10:46
  grab, you know, a model that you train them determine and
• 10:50
  instantiate it for whatever kind of inference you want to do,
• 10:52
  right? And so that's like a really
• 10:54
  smooth transition to some Downstream piece, which is
• 10:56
  making it super, super easy to take the results of a training
• 11:00
  run and use that good inference somewhere.
• 11:03
  And similarly, Upstream, you most people are getting their
• 11:06
  data to their models, just writing python code and we don't
• 11:10
  really get in the way of that. So whatever tools are using be
• 11:13
  it, DBC, or pachyderm. And we have good examples where
• 11:17
  you can specify things like data set versions in a config file
• 11:21
  while you're training a model, and then that all gets tracked
• 11:24
  with it alongside all the other metrics that you were attracting
• 11:27
  a determine so you can keep track of data stuff.
• 11:29
  But we aren't versioning providers.
• 11:31
  We just, you know, make it easy to interface with those version
• 11:34
  providers. So let's move on to cube though.
• 11:39
  Okay, and there was a great thread that came up the other
• 11:43
  day in the questions answered. It's part of slack.
• 11:48
  And it was saying, is cute flow dying and I want to know your
• 11:52
  take on it. Yeah, you follow me.
• 11:55
  I think and I've only interfaced with that Community, a handful.
• 11:59
  I should say up front but from my perspective I think It
• 12:03
  suffers a little bit because of how many like they manage that
• 12:06
  project some extent like it's a whole bunch of manage sort like
• 12:10
  individual projects and some of them have grown and flourished.
• 12:15
  And continue to grow and flourish and thrive.
• 12:17
  In particular, like I am a huge fan of give the pipeline's.
• 12:20
  I think they've done an incredible job of bringing
• 12:22
  workflow management to Python and machine learning.
• 12:26
  I think that's an interface that ml people can enter, like can
• 12:29
  work with and understand, and it makes sense to them.
• 12:32
  They're writing python to do it. That's an awesome component.
• 12:36
  And other components that are mature things like TF job in a
• 12:39
  job. Like those are really good
• 12:40
  building blocks and they solve the problem at the beginning of
• 12:43
  just like, how can we actually run machine learning?
• 12:45
  You flow, right? That said, there are other
• 12:47
  components that, like, have some ups and downs.
• 12:51
  I have like, I never want to bash on things because there's a
• 12:54
  lot of good work, but like, things like khatib is like, kind
• 12:57
  of tricky to use. Like I've never had a
• 12:58
  particularly good experience using their High parameter
• 13:00
  tuning, but it just so that's a long-winded way to say.
• 13:05
  I don't think it's dying. I think that there are excellent
• 13:08
  components. I think they come along at
• 13:09
  different Paces. I think if you find yourself in
• 13:12
  a hole of trying to get khatib to work, you might say, like,
• 13:14
  what is this? Cute fluffy.
• 13:16
  But in reality there's you know awesome work being done on stuff
• 13:19
  like pipelines that make it a really sweet tool.
• 13:21
  So great answer. So anybody that's listening if
• 13:24
  you have any strong thoughts on that, throw it in the chat or
• 13:28
  throw it in the comments because I love hearing about Cube flow.
• 13:31
  I think it's just such a beast like we know and there are so
• 13:35
  many different components and some parts are flourishing.
• 13:38
  Some parts not so much and it's great to hear your take on it.
• 13:43
  What are some of the things that?
• 13:45
  You hope for as far as a future with determined Ai and Cube
• 13:52
  flow. Like, how do you see those two
• 13:55
  becoming more solidified? Yeah.
• 13:58
  Yeah. So I think it's important to
• 13:59
  address up front that there's overlap between determined and
• 14:03
  some of the components of keep flow.
• 14:05
  So, you know, in terms of running training, you know, we
• 14:09
  are very similar in that concept like your MPI job.
• 14:12
  Your your TF job, what? Maybe we let you run, you know,
• 14:19
  models train models distributed that sort of thing in
• 14:23
  kubernetes. You know, we announced Community
• 14:24
  Support last week and so very similar overlap there.
• 14:30
  I would say like the thing that I think determined as a lot
• 14:33
  better is bringing that to a data scientist level.
• 14:37
  If you're just a data scientist and he was Doc, you Flo you're
• 14:39
  going to run into like having to learn kubernetes which I've been
• 14:42
  through that process before but I don't recommend it to most
• 14:44
  people who it's not a very fun thing to have to learn just for
• 14:47
  the sake of training models and similarly khatib, you know
• 14:50
  that's another piece where we provide High parameter tuning
• 14:54
  and you know I think frankly we do a better job of it and so
• 15:00
  there are pieces that we arrived with is what I'm getting at.
• 15:03
  So when you're close sorry if you wanted to finish that feel
• 15:07
  feel I'll wrap up real fast that said I think plugging into some
• 15:11
  of the excellent things they have pipelines like getting more
• 15:14
  intimate with that. Providing better apis so that
• 15:16
  you can instantiate determine runs and just, you know,
• 15:20
  building making it as easy as possible to use tools like
• 15:24
  pipelines and use their serving tool like deploy models.
• 15:27
  I think there are really clean interfaces that we could build,
• 15:30
  that would make it so that by, as soon as you write a model and
• 15:34
  determined, it's ready for serving in something like,
• 15:37
  Selden for instance, right? And that's, that's really cool.
• 15:40
  I want I want to explore and make that a better integration.
• 15:43
  Yeah, that does sound very nervous.
• 15:45
  Nice. So let's talk about that.
• 15:48
  We jumped ahead a little bit for what your vision for the future
• 15:52
  is. But let's talk about where it's
• 15:54
  at right now and how does determined interact with Cube
• 15:58
  flow at the moment? Yeah I got some demo material.
• 16:04
  I'd be happy to show off but right now you know there's a
• 16:09
  couple different ways. The first is something like you
• 16:12
  flipped pipelines. We've got API level access.
• 16:15
  And when you eat anything about workflow management solution, I
• 16:19
  think of, you know, like what apis can recall, the do really
• 16:21
  cool things, right? And so by providing clean API,
• 16:25
  as to do things like take a model from the GitHub repository
• 16:29
  and do distributed training right with the tournament,
• 16:31
  right? And so now, you can set up cute
• 16:34
  flow pipelines that can trigger distribute training jobs in
• 16:38
  determined. And if they're running weekly,
• 16:40
  not now I'll send you have a solution for retraining.
• 16:43
  New data comes in every week, you need to Restrain a model
• 16:46
  using Q flow pipeline. Determined provides the right
• 16:48
  apis to be able to programmatically do that and do
• 16:53
  it at scale or do hyper a tune that kind of thing, right.
• 16:55
  So that's one piece is just like by being a pi provider for
• 16:59
  pipelines. You can do really really cool
• 17:01
  things. Another side is something as I
• 17:05
  said like surveying we provide tools and this is again a pi
• 17:10
  contracts where you can We have them all registered you so you
• 17:14
  say you know, I'm naming this model, this is my question
• 17:18
  answering model, you can upgrade that model.
• 17:21
  You can maintain versions of that model, but then we have an
• 17:24
  API contracted so Downstream if you have a serving solution it
• 17:29
  can you can quickly push out and say update to a new version of
• 17:32
  this model and it will hook in s determine for that model.
• 17:36
  Load that model from determine and instantiate it.
• 17:38
  And you can use it without having to rewrite anything you
• 17:41
  just It's a bumper version, you know, and it's, you know, those
• 17:47
  are both, I think clean clean things already that you can do
• 17:51
  combine in this duel. So as far as like, let's say I
• 17:57
  want to find out which model has been trained.
• 18:00
  And what type of models they are, is the cube flow metadata,
• 18:06
  the way to go for that one. And determine does determined
• 18:09
  have a way to record. This metadata?
• 18:13
  Yeah determined is frankly I think a much easier solution
• 18:18
  than keep flow for tracking. That metadata when you train a
• 18:23
  model or say something in the model, registry the whole
• 18:26
  history of that training is title on with it.
• 18:29
  So every metric lost curves evaluation metrics is tracked.
• 18:34
  What hyperparameters you used? What machines?
• 18:35
  Had trained on one. It happened the state of the
• 18:38
  code when you trained it, if you're using a data version, Any
• 18:42
  provider like what version of the data used?
• 18:44
  Like all of that is automatically tracked as tide
• 18:47
  along with the models. You can access.
• 18:49
  It via API is like, it's all, it's all there.
• 18:51
  And so, whereas in Cube flow, you might need to engineer your
• 18:55
  own solution to snapshot, this stuff to upload, metrics to S3,
• 19:00
  or wherever you may be storing things, right?
• 19:02
  And determine when you train a model, when you version a model
• 19:06
  that stuff is frozen and accessible at all times.
• 19:10
  So let's just Take a step back because since we are chatting a
• 19:14
  little bit about model, Registries, can you talk to us
• 19:17
  about what a model registry is and why it's so important to
• 19:21
  have that? Yeah, so it sounds almost dumb
• 19:27
  when he talked about a mile registry like but there's this
• 19:31
  like actual chord difficult problem of I built this model.
• 19:35
  I was doing research. I'm trying tons of different
• 19:38
  hyperparameters, I'm modifying the model every five minutes.
• 19:42
  It's kicking off all these jobs. I get to this point where I
• 19:46
  think this is a good model, right?
• 19:48
  And then I don't want to use that in production somewhere,
• 19:51
  right? A lot of times the people bring
• 19:54
  into production is a completely different team.
• 19:57
  Like give it an ml engineering team are in ml Ops Team or
• 19:59
  whatever it may be. And so suddenly you need to
• 20:02
  coordinate. Like how do I physically, how do
• 20:05
  I hand off this model to this other team?
• 20:08
  And there's a lot that goes into it.
• 20:09
  It's the actual version of the code to instantiate the model.
• 20:12
  It's the right set of Weights, you probably want some way to
• 20:16
  track metadata associated with it, in case their scores be able
• 20:19
  to interact programmatically. So, at its core, this suddenly
• 20:24
  goes from, like, what sounds like, it should be a pretty easy
• 20:26
  problem to like, I'm spending weeks and meetings and emails,
• 20:30
  trying to like physically figure out and help this person.
• 20:33
  Instantiate my model and load my weights.
• 20:35
  And so, instead of training new models, or working on this now,
• 20:38
  your researchers spending time, all day with the ml engineering
• 20:42
  team getting figured out and then it gets worse because she
• 20:46
  tried to then build a new version of the model.
• 20:48
  And instead of just like that person being able to grab the
• 20:50
  new version. They have to go through this
• 20:54
  whole stupid cycle. Again of, get the weights figure
• 20:57
  out where to host them. Where do I put it?
• 20:59
  How do I instantiate? The new version is the same as
• 21:01
  their new environment like it's a pain right?
• 21:04
  And so with the mall registry comes in and does is just
• 21:07
  basically provide a clean interface for that hand off when
• 21:10
  you train your model and determined as I said everything
• 21:12
  gets tracked, right? You're the code that creates the
• 21:15
  model, the weights metadata, all of that right?
• 21:19
  And so by saving it to the mall registry.
• 21:21
  Now you Can say to your ml engineering team to get my
• 21:25
  model. My model is called
• 21:26
  question-answering asked to determine for that model.
• 21:29
  It doesn't just tell you where the stuff is but it will
• 21:31
  actually like instantiate that model for you and so you can use
• 21:35
  it in a downstream system caveats, it's python-based right
• 21:39
  now like we don't have connectors to plug into your C++
• 21:42
  Solutions and whatever may be right but at least for some
• 21:47
  simple things like running pipelines or batch inference or
• 21:52
  Serving Solutions, like seldom. It's, you know, you call an API,
• 21:56
  ask for a question answering model and you get it and
• 21:58
  instantiate it and you can get a version of it and you don't need
• 22:01
  this whole horrible handshake of how to do it.
• 22:04
  Hmm. Yeah, that makes sense.
• 22:07
  That seems like it will make life a lot easier to and it's
• 22:11
  one of those things that we talked about a lot here.
• 22:14
  How back in the day, the whole reason that devops came to be
• 22:17
  right was you don't want to throw something over the fence
• 22:20
  and have two teams taking Care of something?
• 22:23
  Yeah, it can be merged into one. And so I understand that's a
• 22:27
  little bit of what you're trying to do.
• 22:29
  There is make sure that it's not this whole process to get these
• 22:34
  models out there. Yeah.
• 22:36
  Start serving them as quickly as possible which I think is really
• 22:39
  what it feels like so many different companies.
• 22:43
  Right now are trying to do in this space is the pain is very
• 22:48
  clear for us, right? The pain is that it's taken too
• 22:51
  long. To get the models out there.
• 22:53
  Like I can't tell you how many people I've talked to on this
• 22:56
  Meetup. That, that is what they're
• 22:59
  trying to figure out how to make that time, break it from six
• 23:04
  months to two weeks or however long, even from two weeks to
• 23:08
  just, you know, a few minutes. So let's let's go into what you
• 23:15
  wanted to show us. Can you give us a quick demo
• 23:17
  real fast? And then if I don't know how
• 23:20
  long you budgeted for the demo? Hopefully we can talk a bit
• 23:23
  about the whole idea of getting from research to production
• 23:27
  also, and yeah, yeah, I can do it pretty quick.
• 23:31
  I think I should be able fired off. 33, let's do it.
• 23:36
  All right, so I just want to start with this exact story were
• 23:42
  just telling so right? This is my version of what
• 23:47
  without the tools you doing research?
• 23:49
  Kind of looks like you build your Bert Squad model.
• 23:53
  For a question answering in pie torque or actually.
• 23:56
  I'm using hugging face here, right?
• 23:59
  You're right. A model, you can train it.
• 24:02
  You write out MET Lyrics all that cool stuff, right?
• 24:06
  And as you go like, you can train models, you can see good
• 24:10
  results, but you're going to run into a handful of things.
• 24:13
  So, you know, I tried to kill model, it's getting decent
• 24:16
  scores. I can test it.
• 24:19
  I can, you know, ask some trivial things run against it,
• 24:24
  see what's going on? Maybe I can tweak some type of
• 24:28
  parameters train again you know like iterate on this thing
• 24:32
  eventually come up with a good model and this is going to work.
• 24:38
  Okay. For you but you're just get
• 24:40
  stuck. So it's things like you know,
• 24:42
  I'm saving checkpoints to this checkpoints directory but I
• 24:46
  probably actually want to name. Those checkpoints something has
• 24:48
  to do with the hyperparameters and I somehow else I want to
• 24:50
  like the metrics along side of them and all that.
• 24:53
  And so the first thing that comes up is she just you need
• 24:55
  some way. If you ever have any hope of
• 24:57
  going to production to keep track at a real core level of
• 25:01
  your work and so you something like ml flow, weights and
• 25:05
  biases, determined whatever. It may be, right?
• 25:09
  And you need somebody like, you finally can keep track of this.
• 25:13
  And so, you know, for this talk, let's talk about determined.
• 25:17
  I'm just going to jump to doing this exact same thing and
• 25:20
  determined so I promise you before determine runs very
• 25:25
  similarly once again we write a hugging face Squad model and
• 25:36
  kick this off and determined. And now we end up with something
• 25:39
  that looks a little bit more like this.
• 25:41
  So, It's going to do the same thing.
• 25:43
  It's going to train that same model of fixed type of
• 25:45
  parameters for a while, but when you run into sharpened, you have
• 25:50
  really comprehensive tracking of what's going on.
• 25:52
  So you have your model code, it's tracked, you can see where
• 25:57
  your checkpoints are being stored and have clean access to
• 25:59
  that. You can look at what
• 26:02
  hyperparameters were used to train this model, you have all
• 26:06
  of the curves you care about whether it's your F1 score or
• 26:09
  just you know, whatever it may be.
• 26:12
  Your metrics. And so this is the first steps
• 26:16
  production is like I actually have a lot of stuff.
• 26:18
  I know where this model is, I can use it.
• 26:21
  Like, I have a clear history of all the work I was doing and and
• 26:26
  so now you can do things such as say, you know, this was called
• 26:31
  notice was experiment to and determined, we can ask determine
• 26:35
  to get experiment to grab the checkpoint.
• 26:37
  Top checkpoint load that model. And we can use it to do
• 26:43
  inference. And so, you know, I say this to
• 26:46
  say, instead of having to keep track and messily work on your
• 26:49
  checkpoint code and keep a spreadsheet of all the metrics
• 26:52
  and all of that. You now you have a solution to
• 26:55
  cleanly keep track of all of the work you're doing.
• 26:59
  And you know, do inference in some sort of programmatic way
• 27:04
  instead of some ad hoc way where you're taking these models in
• 27:06
  notebooks and tossing them around.
• 27:11
  So the cool part about this though is it's not just you know
• 27:18
  as I said it's not just models that you pass around and ad hoc
• 27:21
  instantiate. We're giving you a programmatic
• 27:23
  way to do it and so what that means is suddenly you can do
• 27:28
  things like build scripts to do inference from some specific
• 27:32
  version of a model. So the other thing that
• 27:36
  determine provides as I said before is this registry and so
• 27:39
  suddenly, we can build a script that says, if I've got data in
• 27:43
  some path take grab some model from determined.
• 27:46
  Determined is located here, grab some version of the model and
• 27:49
  then let's going to do is actually like Loop over that
• 27:51
  data and score. So this is a really standard
• 27:55
  batch inference script, right. But instead of having to ad hoc,
• 27:59
  figure out how to do the code and upload my model that's three
• 28:03
  somewhere and someone else downloads.
• 28:04
  It's Where I have clean apis, same ones.
• 28:07
  I was using here to do it and if you're familiar with you
• 28:13
  pipelines at all like if you have a script that operates on
• 28:15
  some input from our parse and then creates outputs, you've got
• 28:19
  a one step jump to do say run this kind of thing in cute flow
• 28:24
  and so if we jump to cube flow we can this is my like boring
• 28:30
  version of this Pipeline. And so what's going to do is
• 28:32
  create a shared volume Download some data from S3 to that volume
• 28:36
  and then just call that script on that data, right?
• 28:41
  And if we tell it where determined is similarly, the
• 28:48
  name of the model, we want some version of the model, I have two
• 28:52
  versions here and where the data is suddenly you're talking about
• 28:56
  kicking off a script that you know and with people you can do
• 28:59
  this like run programmatically, you can trigger it from some
• 29:03
  trigger you can That Hawk, you can run it on some timeline,
• 29:06
  right, but it's coordinating a really reproducible way to do
• 29:12
  this as your model versions iterate.
• 29:14
  You know, all you're looking at is popping in a new version
• 29:18
  here. So if you've got someone in your
• 29:20
  research team iterating, creating better models, it's
• 29:23
  really easy to still use the same exact code to do inference.
• 29:28
  And you know, this experience is not the ad hoc collaboration.
• 29:34
  But you have this real clean dividing line between someone
• 29:36
  that's working in determined doing, you know modeling but
• 29:41
  maybe they can go and also kick off some hyperparameters, search
• 29:44
  to improve their model. They're doing all of the
• 29:46
  research stuff that they want to do to iterate on this model.
• 29:49
  Make it better try new datasets try new models and then behind
• 29:54
  the scenes, The Ops Team can take that and easily create
• 29:59
  scalable ways to do batch inference, right?
• 30:03
  And so this is really the core of what I wanted to show off
• 30:05
  that is just like this is a queen easy.
• 30:09
  Handoff to have your research over here and determine you get
• 30:12
  to do really cool stuff. You could scale up to Cluster
• 30:14
  scale and train models fast the easy.
• 30:17
  But with the right apis, you can make scalable ways to then do
• 30:22
  that and keep flow supplement that for inference Looks like a
• 30:25
  match made in heaven. Happy answer questions about
• 30:29
  this too. I know we flew through that but
• 30:31
  I have, we can dive into any code here to if people are
• 30:34
  curious about it. Yeah, for sure if anyone has
• 30:36
  anything that they want to say, throw it in the chat, I am
• 30:41
  seeing one from before that was asked, and it was airflow verse
• 30:47
  Luigi. Verse Argo verse ml Flow versus
• 30:50
  Cube flow which orchestration tool to use win.
• 30:56
  Frank, it is a good question. I've worked with a bunch of
• 31:00
  them. I have examples in one of our
• 31:02
  determined repositories for working with air flow and Argo
• 31:06
  in cubes of five points. I've heard of something called
• 31:11
  cooler that I'm thinking about building a example with later
• 31:14
  this week that is meta over all of those.
• 31:18
  I think they serve similar rules the thing.
• 31:20
  I like about Cupid pipelines is, and we can actually look at it.
• 31:25
  The way you declared people pipelines is just with python.
• 31:28
  Like, at least it's one of the ways you can do it and I just
• 31:31
  think it feels a lot more natural for a data scientist to
• 31:34
  be able to like, at my core. That's what I like to do, right?
• 31:37
  I want to write more python code and I want to eat less like Long
• 31:41
  demo files, the describe what I want to do here, feels the same,
• 31:45
  but I think the execution standard is a little bit
• 31:48
  cleaner. When you, blow air flow has a
• 31:52
  lot of plugins for doing like data transformation of stuff
• 31:54
  like that, or its keep flows a bit more geared towards running
• 31:58
  machine. Learning training type jobs.
• 31:59
  So, yeah. So another question is coming
• 32:02
  through here in the chat and it's asking when using Cube flow
• 32:05
  or determined, how can you deploy the model through CD
• 32:08
  tools like Jenkins? Yeah, it's a great question.
• 32:12
  So, I've looked into this for a couple of different ways or like
• 32:18
  even Argo has a CD there, something called Argo City that
• 32:20
  I've used before. One option is to have these cute
• 32:25
  flow pipelines, get kicked off by by some sort of c.t. to like
• 32:30
  Jenkins, right? So you push new code, say it's
• 32:33
  new versions pipeline or something like that, it kicks
• 32:36
  off this Pipeline and I have another example here.
• 32:41
  Which we in touch at all, but it's something like, you know,
• 32:45
  this is a pipeline that grabs a model from GitHub trains, the
• 32:50
  model checks, if it's better than the previous version of the
• 32:53
  model, and if it's better, it deploys, it was Selden, right?
• 32:56
  And so, this is exactly what you're saying.
• 32:57
  Say, you push new a new version of model code, to some
• 33:00
  repository, say, you push a new data set to someplace.
• 33:03
  Say you push new deployment code, right?
• 33:06
  Suddenly you can use your Jenkins or whatever.
• 33:11
  You may be using for CD to trigger this pipeline, run, all
• 33:16
  of this stuff and push that new model to the ployment.
• 33:19
  And, you know, I think that is a feasible step to actually
• 33:25
  delivering some sort of CD, or cic D to this machine learning,
• 33:29
  which is always been kind of hard for for a lot of people.
• 33:33
  Yeah, yeah, for sure. So there's another question.
• 33:38
  It's a yard M's asking in the chat how Is determined connected
• 33:42
  to cube flow in question. So in this specific case the
• 33:51
  batch inference, there's you know, this Cube flow pipeline is
• 33:56
  just running these steps. This step is actually calling
• 33:59
  those determined apis. It's in fact, this step is this
• 34:03
  exact script that I had open here that just says reach out to
• 34:09
  determined get the model. Called question-answering get
• 34:12
  version 2 of that model, it then, you know, loads and
• 34:15
  instantiates it and then you can use it.
• 34:18
  And so, You know, when you think about a pipeline like this, all
• 34:23
  its really doing is, we wrote a script that tells cute flow how
• 34:27
  to talk to determine and load a model and use them all over
• 34:31
  batch inference on some data set.
• 34:34
  And, you know, to some extent all Integrations in software are
• 34:38
  just good apis. And that's why I think we've put
• 34:41
  out here All right, mister researcher.
• 34:46
  Are we going to talk about a little bit of this?
• 34:48
  Now let's let's get into. I know that you've done some
• 34:52
  research with deep learning and I read up on your website where
• 34:58
  you talk about how your photographer and you also talked
• 35:01
  about some Atari game that you put some deep learning to was
• 35:04
  that it? Should I get that correct?
• 35:06
  Or was it just a random game? It was back in back in grad
• 35:12
  school actually. I did some work, there's a paper
• 35:15
  called feudal networks, for hierarchical reinforcement.
• 35:18
  Learning, the Google put out. Also paper, great concept.
• 35:24
  And so I put a lot of work in to try and replicate that paper and
• 35:29
  horribly fail, just couldn't do it.
• 35:32
  I find with a lot of Google papers that they have a lot of
• 35:34
  Secret Sauce that makes their things tick and if you don't use
• 35:43
  their secret sauce, you can't get it.
• 35:44
  So reproducibility and machine learning is really important.
• 35:48
  I found that out the hard way by pouring my heart and soul and to
• 35:50
  try and reproduce paper, but I love the space.
• 35:54
  I mean, I think it's just a blast working in the
• 35:55
  reinforcement learning space. It's, you know, we still had
• 35:59
  determined haven't seen a lot of people bringing that into like a
• 36:03
  real production system or setting, but it's a fun fun
• 36:06
  space. Well, let's talk about like,
• 36:09
  what components you feel are needed to Do inference in
• 36:14
  machine learning and how can we structure different models for
• 36:20
  that machine learning inference. Great question.
• 36:24
  Yeah. Inference and it's so I want to
• 36:28
  be clear that this varies a lot model the model which is part of
• 36:32
  the difficult things with machine learning.
• 36:35
  So funny enough, I actually think that deep learning is
• 36:39
  easier a lot of times to do inference with And that's
• 36:43
  because the nature of it working frequently on unstructured, data
• 36:48
  solves, one of the problems like one of the hardest hardest
• 36:51
  problems in inference. And so what I mean by that is to
• 36:56
  do deep learning inference, a lot of times you just need to be
• 36:59
  able to instantiate the model somewhere and have it be able to
• 37:02
  accept data and so you know instantiate the model can mean a
• 37:06
  lot of things you generally need code.
• 37:08
  That creates the model you need weights like you're trained
• 37:12
  weight. Still a Load them in and
• 37:14
  potentially some other number of artifacts with something like
• 37:18
  you know language model you might need the artifacts that
• 37:21
  actually instantiate your vocabulary stuff like that,
• 37:23
  right? And so that stuff is all easy to
• 37:28
  track with something like determined because when you're
• 37:30
  training the model it's like the same stuff you use to train a
• 37:32
  model ends up getting used to do inference and so if you can, if
• 37:36
  we know how to train your model, we also to some extent can do
• 37:40
  the same stuff for infants. But the upshot is that it's
• 37:44
  lovely like take an image and do inference, all you need to do is
• 37:46
  upload a plain old image. And what comes back is
• 37:49
  inference, I think inference gets a lot more complicated when
• 37:55
  there's a lot more data pre-processing that happens.
• 37:57
  That is, by far, I think the hardest part of inference.
• 38:01
  If I had to guess it's why people rant and Rave about
• 38:05
  feature stores whenever they come up it's because and this is
• 38:10
  that same thing we talked about before and You threw a real.
• 38:12
  MLS problem is when you have researchers that are trying out
• 38:17
  new features that are building, you know, python code to
• 38:20
  transform their data into the perfect column that will then go
• 38:22
  into their model to build a model better.
• 38:26
  The code that transforms stuff from SQL and two features here
• 38:30
  and then some python that changes it and then you change
• 38:33
  your model and you say, hey, look, I got a great new model
• 38:35
  trying to take that set of things and set it up.
• 38:38
  So that on live huge. Gail inference data that same
• 38:44
  set of Transformations happens and also trying to have like
• 38:46
  you're completely different team reconstruct that impossible
• 38:51
  impossible and so that's a whole set of problems that I think the
• 38:56
  people tackling feature stores like power to them.
• 38:59
  I hope they figure out a good way to make that scalable and
• 39:02
  find a good General solution for a lot of Enterprises.
• 39:07
  And I'm glad that we're in the Deep learning space and don't
• 39:09
  have to bet that. It's a slightly easier easier
• 39:11
  set. Problems.
• 39:13
  So long way to do that is to say in my happy little bubble.
• 39:17
  If you have your code, your weights, a good way to
• 39:21
  instantiate all of that, you're pretty much good to go.
• 39:25
  Well, tell me and forgive my ignorance on this, but I have
• 39:30
  two ideas of in ml researcher, right?
• 39:34
  One is the person that is at a university that is trying to
• 39:39
  create some crazy algorithm Or some new Cutting Edge thing.
• 39:44
  Then you have the other one and I kind of separate them in my
• 39:47
  head. The other one is this person
• 39:49
  that is at a company and they're trying to prove out a business
• 39:53
  value or they're researching. If this is really supposed to,
• 39:58
  if this problem is going to get like fixed by using machine
• 40:02
  learning, is that correct, or are they the same one in the
• 40:06
  same. When we talk about ml
• 40:07
  researchers? I don't think they're not the
• 40:11
  same. They're like different Levels of
• 40:12
  research but I still use that shorthand of research because I
• 40:16
  think the core Loop that a lot of people that are building a
• 40:19
  male models do looks a lot more like research in the way
• 40:24
  classical sense than it does. Like the production moves that
• 40:28
  were used to with like a software engineering type of
• 40:31
  role, right? And I mean like you're
• 40:34
  iterating, you are changing your code all the time.
• 40:37
  You don't commit every time. Like you know, commit new code,
• 40:41
  every time you make A small change to your model.
• 40:43
  You're really like dinking and dunking around until something
• 40:46
  good comes out, right? And that feels to me a lot like,
• 40:51
  like research. And so some of and I think
• 40:54
  that's one of the tensions when you try to push, you know,
• 40:58
  software engineering on that set of people is their core Loop of
• 41:02
  what they're doing isn't really very similar to software
• 41:05
  engineering once you get over the wall and that good model
• 41:08
  exists, then it starts looking a lot like software engineering,
• 41:11
  you need to integrate You need to deploy it.
• 41:13
  You need with some obviously big corks.
• 41:16
  That sometimes you have gigabytes of things you're
• 41:18
  moving around and yeah. Yeah.
• 41:20
  Right. So to get back to the original
• 41:25
  question a little bit, those are different kinds of researchers.
• 41:28
  There's the researchers that are completely trying out new
• 41:30
  architectures. And a lot of times that is never
• 41:33
  with a has no goal of getting to production, you don't care if
• 41:36
  you can do production, as long as you can get the metrics out.
• 41:40
  I've read enough repositories of at code that it's very obvious.
• 41:43
  That a lot of them have no intention of ever putting
• 41:45
  production behind that, but I still think the people of the
• 41:50
  companies are doing some variety of research and it creates that
• 41:56
  core tension of Y can be hard to get to production sometimes so
• 42:00
  can you talk to us about the different steps that you'll see
• 42:04
  from research to production? And also the idea that you just
• 42:10
  said, like some code It's never going to go to production,
• 42:14
  right? So how can we prepare our code
• 42:16
  better or how can we better be ready for when we do, want to
• 42:21
  get to production? Yeah, I think there's a POC
• 42:25
  phase that happens a lot like for everybody, no matter where,
• 42:28
  like, even the most mature organizations where it's like I
• 42:31
  see this new paper came out really promising, like we just
• 42:34
  need to get it running and start data set.
• 42:36
  Like, can I get this running against my data set and see if
• 42:39
  it's like, worth our time to invest in?
• 42:41
  A completely new model to see if we can work out a few extra
• 42:46
  percent of this system that we have.
• 42:50
  And that's a lot of times the code that you don't care about,
• 42:53
  light it on fire, when it's done.
• 42:54
  Like, as long as you get the answer.
• 42:56
  Is this worth my time or not? It's good enough, right?
• 43:00
  And then, you know, it's kind of a funnel, you get down to the.
• 43:03
  Yeah, this is worthwhile. We need to think about what this
• 43:06
  is going to look like. We need to do things like yeah
• 43:10
  sure if you're trying to speed this up or we want to do high
• 43:13
  for parameter to need to build a better model.
• 43:15
  And it's at that point that I think in reality as a ml,
• 43:19
  scientists need to start thinking a little bit better
• 43:21
  about how you're structuring your code, determined takes a
• 43:25
  pretty opinionated stance on how you structure code.
• 43:28
  And I think it is, The first like it is that first step to
• 43:32
  moving production, we asked you to literally say, like here's my
• 43:36
  model, here is my Optimizer, here's how I'm loading data.
• 43:38
  As I said before, right? And that by doing that by
• 43:43
  keeping some level of organization on your code,
• 43:47
  you're setting yourself up for success in the future because it
• 43:49
  provides like a standard that you can then translate to a
• 43:52
  production system later. So talking a little bit more
• 43:56
  about development. Do like a lot of times you'll
• 43:58
  see one good Model come out by a whole bunch of hand tweaking or
• 44:01
  stuff like that. The next step that I tend to see
• 44:04
  is people that start wanting to retrain models all the time,
• 44:07
  right? They get new data, they're
• 44:08
  grinding out, new data, they have a huge labeling team there.
• 44:13
  An autonomous vehicle company that's got cars driving around,
• 44:16
  you know, the model is not tweaking a whole bunch but they
• 44:18
  this pouring in faster than you can imagine.
• 44:21
  And so this is another thing where you want some tool to
• 44:24
  programmatically be able to run that training.
• 44:28
  That huge scale on new data every day and push that result
• 44:32
  to somewhere and this is another thing.
• 44:35
  You know that I think something like determined really has a
• 44:37
  good role in which is like providing the tools and apis so
• 44:41
  that however you want to do large-scale recurring training,
• 44:45
  you have you know and that's like a super production model at
• 44:48
  that point that you're like really retraining and that sort
• 44:50
  of thing. And so where do you?
• 44:53
  Oh sorry, good work. I'm just wondering where you see
• 44:57
  this in this This pipeline. Where do you normally see people
• 45:00
  getting stuck? Yeah, so it varies a lot on the
• 45:06
  maturity of an organization. I think there is this initial
• 45:11
  thing where people get confused in this, like, jump from pocs to
• 45:17
  like, oh crap. I kind of need to build models
• 45:19
  that have some hope of getting production, right?
• 45:22
  And there's a lot of stuff to blame.
• 45:25
  There have been repositories that have come out recently, or
• 45:28
  detector on twos and like, that are really good and set up
• 45:31
  already for success. But if your Going to research
• 45:34
  code and training models that way like literally take you
• 45:37
  someone's research code, figuring out how to plug in your
• 45:39
  data set, and go, let me tell you, it's a pain to get that
• 45:42
  kind of code to like, you need to literally rewrite a model of
• 45:46
  times, right? And so, and it's not fun work,
• 45:49
  that's not enjoyable. Work that people get excited
• 45:51
  about training new models. That would be excited about
• 45:52
  refactoring code, right? And so, I just think it's some
• 45:56
  core like that, initial push from like I'm in the POC phase.
• 46:00
  My goal is to show that this works too.
• 46:03
  I'm in a production phase, I can still train models but I need to
• 46:07
  do it in a somewhat reliable way where I'm using tools to track
• 46:10
  it and I'm keeping track of all my artifacts and to be clear
• 46:15
  like, this is where I think tooling can come in, like,
• 46:17
  determined tells you how to structure your model and then
• 46:20
  tracks everything for you. Right.
• 46:22
  Same like ml flow, at least like gives you really, really easy
• 46:26
  apis to keep track of all of your work.
• 46:28
  Like, let you have a slightly more reproducible stance on like
• 46:32
  how you're doing this? Of.
• 46:34
  And so this is one of the places where I really think there's a
• 46:36
  space for for tooling to help bridge that Gap.
• 46:39
  So that when people like to help people understand what it looks
• 46:42
  like to build production-ready models.
• 46:46
  Great point. Last question, I got for you and
• 46:49
  then I'll let anybody in the chat, throw what they want and
• 46:54
  ask away, but I want to know once you've gotten to this
• 46:58
  maturity and you've created something.
• 47:02
  We could say your a little bit farther along this path.
• 47:05
  What are things that you've seen?
• 47:06
  Pop up that you're not necessarily thinking about in
• 47:11
  those first phases like that POC to just getting it out there.
• 47:18
  You know. I think the most well-documented
• 47:21
  and one that I don't want to dwell on but I like it really is
• 47:24
  is just like your data changes up and Heath you eventually
• 47:27
  right always happens. Data drift is like the thing
• 47:30
  that happens to everybody's models all the time.
• 47:34
  And at its core, what that means is, you need to have tools to
• 47:38
  solve that in place. That means Downstream, you need
• 47:43
  tools to identify it happening. Like when you're whatever you're
• 47:45
  doing for in French, need to be able to find it quickly address
• 47:48
  like identify exactly what it is.
• 47:50
  It means Upstream you need to be able to figure out how to label
• 47:54
  new data. Fix it solve the problem and in
• 47:58
  the middle was determined you need to have the tools that when
• 48:00
  all of those pieces are firing you can.
• 48:03
  Quickly really quickly, retrain that model on the new data that
• 48:08
  you identify. Did you need to label?
• 48:09
  Because Downstream you identify what was wrong, right?
• 48:13
  And, you know, I feel weird even saying that that's the thing
• 48:16
  that people don't know about because it feels like these
• 48:17
  days. Everybody knows about it but
• 48:20
  it's still B people all the time and things that you have to be
• 48:23
  ready for is when you know when shit hits the fan, how are you
• 48:26
  doing? It?
• 48:27
  It's also important, it's so important and I think we talked
• 48:31
  about that quite a bunt, a lot in the In the slack channels.
• 48:36
  And I'm a big proponent of it. Especially this whole monitoring
• 48:40
  peace. That it feels like what is the
• 48:44
  way to do it? How do we go about it?
• 48:46
  It's so much more complex than just monitoring software, right?
• 48:50
  Because of the data and because of the model and you have to
• 48:53
  check for drift, and I like this idea of saying you, if you're
• 48:57
  monitoring, you want to be able to have a very clear path and go
• 49:00
  back and fix it really quickly. Rather than just A drink.
• 49:04
  Okay drift is happening. Let's be able to know where it
• 49:08
  started or why it started and get it fixed.
• 49:11
  So that we don't have to suffer through that for a while without
• 49:17
  actually needing to know and looking at some of the most
• 49:21
  mature organizations, understand this.
• 49:23
  The one that always comes to my mind is the autonomous vehicle
• 49:25
  companies where they see new soft science.
• 49:28
  They see news road conditions, they see something and they have
• 49:31
  active learning set up so that while they're training.
• 49:33
  They could, they can get like, in the middle identify new
• 49:37
  stuff. They test their cars all the
• 49:40
  time to see and identify where they need to collect, new data
• 49:43
  and label new data. And they have really
• 49:45
  sophisticated pipelines to quickly, retrain models at huge
• 49:48
  scales, right? And you know, I think they're
• 49:52
  good Beacon to look for words of how people are really solving
• 49:56
  this issue at skill right now. I think you are muted.
• 50:06
  Thank you. That's always whatever.
• 50:10
  I see the last question. Coming through here, in the chat
• 50:12
  and make nav, is asking what are the most underrated topics
• 50:19
  regarding deploying? Machine learning models in,
• 50:22
  production, in your opinion, Yeah, we briefly touched on it
• 50:29
  but the thing that I don't think people talk about enough is how
• 50:33
  hard it is to recreate your data in production.
• 50:39
  As I said, I think for deep learning we're gifted or pretty
• 50:43
  lucky that sometimes you can just upload an image and get an
• 50:45
  answer back. When you are talking about
• 50:49
  complicated transforms before your data is ready getting that
• 50:53
  to happen. Yeah, at all is hard like just
• 50:57
  recreating that side of transforms as hard, let alone
• 50:59
  recreating it in a production grade way where if you're
• 51:03
  getting data and you need it to be ready and transformed a
• 51:06
  fraction of a second later so you can make a real-time
• 51:08
  prediction. I think without a doubt that's
• 51:11
  the thing that people have the hardest time.
• 51:15
  Identifying which is actually I think the hardest part of this
• 51:18
  production ization of malls, do you have a solution or you just
• 51:22
  going to leave us with that problem?
• 51:24
  Mmm. So just feature stores are real,
• 51:29
  like, people. I don't think people have a for.
• 51:31
  So I think when people first think about feature stores, the
• 51:35
  thing that comes to their mind is a place where you can store
• 51:39
  all of your features and then reuse them for later.
• 51:42
  And that's a small part of the value prop.
• 51:44
  But the real value prop, is providing clean pipelines to
• 51:47
  take features that you use for training and use them in your
• 51:52
  Downstream system. And so, He's out there doing
• 51:55
  that. The one I think of the most is
• 51:59
  something like tecton AI. It's relatively young but made
• 52:02
  by the people who made Michelangelo and then there's a
• 52:06
  couple of open, opsworks house one that they market and then
• 52:10
  there's an open source offering called Feast which is a Go Jack,
• 52:14
  Google collaboration and I know Cube flow is talked about work,
• 52:18
  integrating the feast. So those are all tools that
• 52:21
  start to try to address that problem which is it's a real
• 52:25
  hard one for sure. So last one, I see.
• 52:29
  We got two minutes left. I'm going to try and squeeze it
• 52:32
  in here. Gonzales, asking in my
• 52:34
  experience managing data science teams.
• 52:38
  Usually they're not software perficient like developers
• 52:43
  having this in mind. How do you see the adoption of
• 52:46
  tools such as determining to float by data scientists?
• 52:50
  Yeah, I agree. I mean I say this In Florida
• 52:55
  State as a data scientist, who probably should not be allowed
• 52:57
  to develop software that, that I completely agree.
• 53:02
  And so that's why I think determined is like a good
• 53:06
  solution for this problem. Frankly, you're letting me put
• 53:09
  on my sales have for a second here and I apologize in advance.
• 53:13
  We built determined to be a tool that data scientists want to use
• 53:17
  like they write in Python, they describe information about
• 53:21
  models and optimizers and like the stuff that they think About.
• 53:25
  And then they can use that to not worry about how it gets
• 53:28
  happens. Like they don't need to worry
• 53:29
  about machines and infrastructure and all of that,
• 53:33
  that August provisioned, as long as they say, here's my cool
• 53:36
  model training on a gpus and then determine goes and joins
• 53:39
  money gpus whereas something like Cube flow is not like that.
• 53:43
  Like you follow if you want to train a model and a gpus you've
• 53:46
  got to write a spec that's a whole bunch of kubernetes and
• 53:53
  that's not like I have yet to The data scientist, most
• 53:56
  organizations in the world that like is super excited to have
• 54:00
  thread, whole bunch of kubernetes.
• 54:01
  Yeah, mole to be able to train their model and so determined
• 54:06
  Feels Like For Me, fills the gap of like being an accessible for
• 54:11
  data, scientists tool to do model training and stuff like
• 54:14
  that. And then, you know, I see it as
• 54:17
  building, the right ways to hand that off, to people, who are
• 54:19
  software proficient, like your software leaning, people who
• 54:22
  maybe aren't Being on the coolest deep learning models,
• 54:26
  but they can talk machine learning and they can use tools
• 54:30
  like our model registry to interface with the work that
• 54:32
  those data scientists are doing. And let's just finish it up
• 54:37
  here. I know that determined does have
• 54:40
  an open source components. Can you explain that real fast?
• 54:44
  I know a little bit but essentially all of determined
• 54:48
  everything that I briefly showed you today is open source, you
• 54:52
  can install it. Yeah, right now, it's available
• 54:56
  on GitHub. Determined AI / determined.
• 54:59
  And essentially, the whole platform, we do have an
• 55:03
  Enterprise Edition. It has an oppressed security
• 55:06
  features and stuff like that, but none of the all of the core
• 55:09
  machine learning stuff that data scientists use that.
• 55:12
  All of it is open source so easy to get going with installs and
• 55:17
  AWS gcp, kubernetes on-premise, whatever you like wherever
• 55:23
  possible. Awesome, dude.
• 55:25
  David it's been so good talking to you man.
• 55:26
  I really appreciate you coming here.
• 55:28
  Showing us what's going on answering these questions that I
• 55:31
  had in my mind and talking about cute flow with us.
• 55:35
  Yeah, it's been fun. Really enjoyed it.
• 55:37
  Thanks for having me. Thank you everybody for sticking
• 55:39
  around. And if you have anything else to
• 55:42
  say to David, he is in the slack.
• 55:46
  So get him at David, her, she just tagged him and ask him
• 55:51
  questions. Please do look.
• 55:53
  To it.