Latest / Agentic Conversations (formally mlops.community) / Making MLFlow // Lead MLFlow Maintainer Corey Zumar // MLOps Coffee Sessions #103

Transcript

Making MLFlow // Lead MLFlow Maintainer Corey Zumar // MLOps Coffee Sessions #103 · Agentic Conversations (formally mlops.community)

• 0:00
  Here's what I think Cory's got experience and insights and it's
• 0:05
  so obvious that he's dropping bombs without even meaning to,
• 0:09
  you know, it's like he says things and it took me a minute
• 0:13
  to play back what he said and in my head and then I'm like, whoa!
• 0:17
  That was really deep. Yeah.
• 0:19
  And he didn't even he did it without even like cracking a
• 0:23
  smile or anything, just straight up to the point.
• 0:25
  Just give it to you straight and I loved it.
• 0:28
  And yeah, he sleeps with the stuff.
• 0:30
  Under his undersell. Oh man, this is this is this is
• 0:32
  piecemeal. No seriously.
• 0:35
  I mean he's been working on it. So for those who do not know, we
• 0:38
  just got done. Interviewing Corey, who is one
• 0:41
  of the lead maintainers of ml flow, which I can imagine.
• 0:46
  Most everyone has heard of by now.
• 0:48
  And we talked a lot about that and what it's been like for what
• 0:51
  he just mentioned, the better half of a deck half of a decade,
• 0:55
  sorry the better half of a half of a decade if that.
• 0:58
  Yeah could get any more confusing tell me.
• 1:00
  So, five years in layman's terms, he's been chugging along
• 1:06
  with Emma flow and we were originally going to do it with
• 1:09
  him and great friend and common guests of the Pod Ben Wilson,
• 1:16
  but been sadly, his internet went out while we were filming
• 1:21
  or recording and so, he wasn't able to come on and we just were
• 1:26
  with Corey, this whole time, but Corey was a champ.
• 1:30
  And he was giving us insights and of insights, for the both of
• 1:33
  them. What was your favorite take
• 1:35
  away? At one of the most really astute
• 1:40
  points that he made that I thought was just yeah, very one
• 1:42
  of his mind blowing type moments where we were talking about, you
• 1:47
  know, how does how does an open source project that's affiliated
• 1:50
  with like an Enterprise organization like a commercial
• 1:53
  entity? How do you kind of walk that
• 1:55
  line right? Where there could be competing
• 1:57
  factors between the two men? And in a sense, there's
• 2:00
  different principles at play in the two types of organizations.
• 2:03
  And he just, you know, said the basically as long as there's a
• 2:07
  line that very often, there's alignment between what a user
• 2:10
  will want, even like an open source capacity and an
• 2:14
  Enterprise offering write something that somebody actually
• 2:16
  pay for and I think that having the fact that Emma flow has
• 2:20
  struck that balance. Right has really found that
• 2:22
  sweet spot. I think speaks to like an
• 2:24
  incredibly successful project, right?
• 2:25
  From a product Market, fit standpoint.
• 2:28
  If you want to use that terminology.
• 2:30
  I mean, once you get that, it's it's like the best of all the
• 2:32
  worlds, right? You have something that can make
• 2:34
  money, but also something that delivers a lot of value to
• 2:36
  people, right? And I don't even get anything
• 2:38
  better than that. Yeah, yeah.
• 2:40
  People willing to pay for it. That's the beauty of it and
• 2:44
  that's really true. How he said we're trying to find
• 2:49
  that alignment between what people want in our Enterprise
• 2:55
  desires and our Enterprise like needs.
• 2:59
  Also So for making money and so I think that was a great point.
• 3:04
  The other point that I loved was how he talked about each
• 3:08
  individual piece of ml flow. They wanted to be something that
• 3:13
  was as easy as possible to set up and you didn't need to set up
• 3:17
  the rest of Mo flow just to get started using that piece that
• 3:21
  you're looking for. So it's like such easy barrier
• 3:26
  to entry, there's like no barrier to entry really.
• 3:28
  You just get This little part first and then you say I kind of
• 3:32
  like it, maybe I'll go a little bit deeper and then before, you
• 3:35
  know it you've got this end-to-end tool which we talked
• 3:38
  about, we went into like how do they view how especially does
• 3:41
  Cory view an end-to-end tool? Which I thought was very
• 3:44
  insightful and how he wants to look at ml flow as an end-to-end
• 3:49
  to land where they position themselves and what he thinks
• 3:52
  about in that space. So those are some keys.
• 3:57
  Yeah, well while maintaining the extensibility, right?
• 3:59
  I think that's that's the Point that you've brought, which is
• 4:01
  fantastic. It's like end to end, but you
• 4:03
  can still trapped up how you like it, so you can kind of take
• 4:07
  your pick your adventure, right? Yeah.
• 4:11
  I'd like that idea. Pick your own adventure, like
• 4:13
  the old books that we used to read back in the day.
• 4:16
  So is your own adventure ml flow?
• 4:19
  That should be some kind of game.
• 4:22
  Somebody gets to be on a t-shirt that's that should be on their
• 4:25
  t-shirts, are there we go. So this This is what we're going
• 4:32
  to get into. Let's talk with Corey.
• 4:35
  And before we jump, I want to mention that, if you are not
• 4:40
  subscribed already, we would love it.
• 4:41
  If you do, give us some subscribe and like or follow or
• 4:46
  a rating or a comment feedback, all of that.
• 4:50
  Does a huge amount for us. It is very helpful so we'd love
• 4:54
  to see you participate. We're also giving away a free
• 4:57
  book in the comments so if You fill out a comment, I will reach
• 5:02
  out to you, potentially and send you a free book.
• 5:06
  That's all we've got. Let's chop jump into it with
• 5:09
  Cory. I really want to start out what
• 5:15
  that is. Just like the origin story or an
• 5:20
  origin story of ml flow. I feel like that's the only way
• 5:23
  to start out right here with you two on So core you want to kick
• 5:28
  it off. Like how did this idea come
• 5:32
  about? Absolutely yeah.
• 5:35
  So I joined data bricks when ml flow is still an alpha project
• 5:41
  and there was this identification of a gap in the
• 5:46
  ml UPS space or a lot of platforms that were out there.
• 5:50
  We're either very specific or opinionated to a particular
• 5:56
  framework. So you had platforms or
• 5:58
  tensorflow models. You had platforms for Pirate
• 6:01
  swords models. You had Platforms that were good
• 6:06
  metadata tracking for model parameters and performance.
• 6:09
  Metrics other platforms that we're focused on model
• 6:12
  deployments. And there really wasn't a whole
• 6:15
  lot of glue between these various tools at their didn't
• 6:20
  really exist a platform that anybody could just pick up and
• 6:23
  use and would be compatible, but the variety of tools and
• 6:28
  software that they needed, and the sort of end-to-end ml Ops
• 6:33
  life, Go. So, when I came in that, that
• 6:36
  problem had been recognized and small team of nine developers at
• 6:40
  data, brakes, were working on building out this one-of-a-kind
• 6:46
  first and class open source platform for the end-to-end
• 6:50
  machine learning lifecycle built on these open apis.
• 6:53
  And it was, it was really fun to kind of watch and coming in the
• 6:57
  door. I didn't really know what open
• 7:00
  source development was about or kind of How you got a project
• 7:04
  off the ground. So there's this really awesome
• 7:06
  opportunity to see like, okay we're what's going to happen
• 7:10
  when you launched a version 0.1 and where's it going to go and
• 7:17
  adoption and the growth of the platform over time is really
• 7:21
  going to exceed anything. That I could've imagined been
• 7:23
  telling me and what is it situate us?
• 7:27
  Like, where were we with spark with spark already a big player,
• 7:33
  what Are we looking at there? Like yeah.
• 7:37
  Oh yeah. So Spark by that time had
• 7:39
  certainly been really widely adopted as a core underpinning
• 7:43
  of data versus business. You had platforms like EMR and
• 7:47
  other competitors offering and manage spark, if spark Community
• 7:51
  was was thriving with hundreds of, pull requests a month.
• 7:56
  And yeah, it was, it was a really mature, really huge
• 7:58
  project, and I'll flow was one of the kind of first, major data
• 8:03
  Brooks. The following the model of
• 8:04
  spark, but not really wasn't just an extension of the spark
• 8:10
  ecosystem and of its own tool. But we took a lot of lessons
• 8:15
  from the spark open source, community and growth, and kind
• 8:20
  of have that we saw that play out.
• 8:22
  As we develop them all blow, you know what, we're some of those
• 8:25
  lessons and absolutely. So one key Insight is that
• 8:32
  especially in me. Machine learning but also an
• 8:35
  ETL. There's just this huge ecosystem
• 8:38
  of tools and requirements. The different users might have,
• 8:43
  and instead of trying to purpose build solutions for every single
• 8:48
  Niche tool, when you use case, it's often easier to create a
• 8:52
  really extensible, and coherence core API, and then let
• 8:57
  developers bring their own workflows to that API.
• 9:02
  So the initial As I understand the initial versions of spark,
• 9:07
  it was really centered around a set of in wasted interfaces.
• 9:12
  For example, in the Emma flow tracking API, let you do with a
• 9:16
  small handful of things log of parameter, logging metric, set
• 9:21
  of tag, create a run and create an experiment and we didn't even
• 9:25
  have searched at the time and that was that was kind of it.
• 9:28
  We had a model specification was basically what version is this
• 9:32
  model? And like where's the The
• 9:34
  software environment, the Condi animal defined and that was
• 9:38
  pretty much an awful models and something similar for projects.
• 9:42
  So that was a real core Insight that it was important to get a
• 9:47
  project off the ground and start getting feedback.
• 9:49
  Start getting usage without trying to boil the ocean from
• 9:53
  day one. I so I'm a, I personally am a
• 9:58
  huge, huge fan of ml flow. I mean, you as a practitioner, I
• 10:00
  think it's a fantastic piece of software and Some sense in my
• 10:05
  mind like one of the big like a success story of Open Source and
• 10:09
  I'm a lobster, you know, it's like something that really
• 10:11
  people use a lot and I think has done a good job of building sort
• 10:15
  of abstractions that people find very useful.
• 10:18
  And in a sense like, you know, open source is something that a
• 10:20
  lot of envelopes. Companies, think about, in
• 10:22
  organizations, it's become so core to how a lot of groups and
• 10:25
  making about deploying machine, learning, tools and
• 10:28
  infrastructure. I'm curious, you know, what have
• 10:30
  been some of the Core, abstractions and principles that
• 10:35
  that have you think of made Mo flow.
• 10:37
  So successful, that may be other groups can learn from.
• 10:40
  Yeah, first of all. Thank you so much for the high
• 10:44
  praise and it sounds like you've got your hands dirty with the
• 10:48
  platform and tried it out. And that's just the cone really
• 10:50
  appreciate it. I think that one of the, the
• 10:55
  core abstractions that we found the most success with is the is
• 11:00
  this extensible tracking component.
• 11:02
  No, it's just such a challenging problem in the ml up space, to
• 11:08
  build a model, iterate on that model and actually know that
• 11:12
  hey, I'm making substantial improvements here.
• 11:14
  The model that I trained today was better than the one that I
• 11:17
  trained a week ago and there's a lot that goes into that, you
• 11:21
  have parameters, performance metrics, and those can be kind
• 11:27
  of arbitrarily complex. And we're looking at ways that
• 11:30
  maybe we go beyond scalar method.
• 11:32
  For example, you have all sorts of artifacts.
• 11:36
  So the model source files at the source code that was used to
• 11:39
  build that model. Model objects, themselves
• 11:42
  software environments. You have some metadata around
• 11:46
  okay, who actually built this model with system did come from,
• 11:50
  what? What version of a particular
• 11:52
  platform if you're running on data bricks, for example, and I
• 11:55
  think solving that problem has really driven a lot of growth
• 11:59
  than a mall flow. We do get a lot of usage, Of
• 12:02
  these other components that the model registry animal models and
• 12:05
  projects. But I think without tracking and
• 12:07
  having that that API that, like I said, started small but grew
• 12:12
  over a period of time. But I think that's just
• 12:15
  fundamental and important the system and a lot of other
• 12:18
  Integrations tie back to it. So actually, there's some
• 12:22
  interesting that you mentioned there in this goes more into
• 12:25
  like product development and with open source, I know it can
• 12:29
  be very difficult for people to And a, how many people are using
• 12:34
  their tool and be how they're using their tool, what they
• 12:37
  like, without going in there and surveying people, each time,
• 12:43
  they want to bring out a new feature or do something
• 12:46
  different with the tool. And how did you get around that?
• 12:49
  Like, what was the feedback loop from the users?
• 12:53
  Like and how were you getting metrics on?
• 12:57
  What they were using what they liked what they didn't like?
• 13:00
  Yeah. Awesome question.
• 13:02
  It's kind of funny. You mention surveys because we
• 13:04
  have made pretty liberal. Use of surveys specially in the
• 13:07
  early days. It's been a while since we did
• 13:09
  one. But after launching the the
• 13:11
  Alpha version and it within the next two or three months of
• 13:15
  development, we sent out a survey and asked what do you
• 13:19
  like about this platform? What are you using, what could
• 13:21
  be improved and what maybe don't you find useful or what do you
• 13:25
  find confusing and that actually helped in those very early days
• 13:28
  to kind of identify What components resonated and where
• 13:32
  we need? Maybe polish things up after
• 13:36
  that though. One thing that's very unique
• 13:38
  about them, all flow as it came from from data bricks this
• 13:43
  Enterprise platform for data science and machine learning.
• 13:48
  And as a result as we integrated them all flow of data bricks, we
• 13:51
  have this awesome stream of information about usage from our
• 13:55
  customers. So we gave them the same set of
• 13:58
  components that we're giving. A open source users with largely
• 14:03
  comparable functionality and we got to say, hey, how are, how
• 14:07
  are hundreds or thousands of customers using these features,
• 14:10
  and was the adoption? Look like, and that was also
• 14:13
  super useful for us. I think that there might be as
• 14:15
  kind of a lesson there, that open source growth can really be
• 14:20
  powered sometimes by a third party platforms, that decided to
• 14:25
  adopt these kinds of tools and for us as I guess.
• 14:30
  It was designed with that. In mind, we built an all flow
• 14:33
  partially the start of the open source Community, but partially
• 14:35
  Disturbed, data breaks customers.
• 14:36
  And that really, I think helped with product development.
• 14:42
  I think that's a really interesting point that you bring
• 14:44
  up, is this being when I hear ml flow, like, in my mind I
• 14:49
  immediately make that Association to date of bricks,
• 14:51
  right? And and there's kind of this
• 14:53
  interesting Dynamic here, right? Because we're talking about a
• 14:55
  fundamentally open source project that, you know, when we
• 14:59
  think open source, we think for the good of the world, you know,
• 15:01
  volunteer-led. It's just totally free and open.
• 15:03
  But then there's this connection to like a very commercial
• 15:06
  entity, right? Leg for profit money, making
• 15:08
  machine and some sense. So I'm curious How how do you
• 15:13
  think about sort of straddling that fine line between, you
• 15:15
  know, in some sense what might be perceived as competing, you
• 15:19
  know, principles right in some sense and how does that, you
• 15:23
  know, guidance of the development of the project?
• 15:26
  Absolutely. It's something we think about a
• 15:28
  lot and our stance has changed over time.
• 15:31
  So in the early days, when that first year or so of them all
• 15:34
  full development, it was all about open source, open source
• 15:36
  would get the features before data bricks, got the features
• 15:39
  that was that was extremely important.
• 15:41
  And over time and of what we consider to be a feature has has
• 15:46
  somewhat changed. So when we talk about features
• 15:50
  now at data breaks its that's a data bricks feature.
• 15:52
  That's an ml flow feature. And oftentimes the distinction
• 15:57
  is, well, is it a core piece of functionality for any data
• 16:03
  scientist or machine learning engineer out there?
• 16:07
  Anybody might benefit from this or is it specific to some
• 16:10
  tooling or Workflows that we see it at data bricks, maybe it's an
• 16:15
  artifact of how our platform is is structured and that usually
• 16:19
  tells us where we invest the effort.
• 16:23
  So, do we invest in spending time filing pull request against
• 16:28
  open source, publishing rfc's for the community or do we
• 16:32
  invest in the kind of getting down to it and making tweaks and
• 16:37
  improvements in the difference product and we're still very
• 16:40
  much at this point. Focused on trying to open
• 16:44
  source. I'd say as much as we can that
• 16:46
  we think is useful for the the broader ml Ops Community, for
• 16:50
  making sure that the customer needs and requirements or
• 16:54
  addressed quickly. So, it's a fund balance for a
• 16:56
  small and Nimble team, and it's so far so good, I'd say, but I'm
• 17:01
  sure you had a community. Might have any thoughts, or you
• 17:03
  guys might have any feedback. I like it.
• 17:07
  I mean, I don't, I don't again, it's dissociation, but I don't
• 17:10
  feel like I'm being pitched. Data breaks.
• 17:12
  Every time I use the tool, right?
• 17:13
  That would be the Shameless way to pursue the the collaboration.
• 17:18
  Yeah, absolutely. Yeah.
• 17:19
  That's the. That's the goal.
• 17:22
  So as you're looking to kind of build out a new platform and
• 17:28
  start a new project, what are some major questions that come
• 17:32
  to mind being on the ground floor of something now?
• 17:36
  Yeah, I mean one that actually is very related here is this
• 17:39
  question of, you know, Going back, maybe even stepping back
• 17:43
  at a meta level is like OSS as the, the kind of the
• 17:46
  go-to-market, right? People a lot of ml Ops groups
• 17:50
  organizations, do pursue that as their strategy, but then there's
• 17:53
  other models to write up just like freemium or kind of sort of
• 17:56
  closed Source kind of opencore. I mean, there's a really
• 17:58
  different different ways to kind of pursue that.
• 18:02
  And, and, you know, I'm a flow, is purely open source, right?
• 18:04
  So, I'm definitely curious to know how, you know, why.
• 18:08
  That was the way that why, that was the path forward shows.
• 18:11
  Why that was the the right go to market for something that's
• 18:14
  still was like a date of bricks spin-off, right?
• 18:17
  I mean an Enterprise company spin-off.
• 18:18
  Yeah, actually it's funny. I think about when I think about
• 18:22
  the open source tools in the ml Ops landscape ml flow is one of
• 18:29
  those ones. I think Emma flow in Cube flow
• 18:31
  are and airflow funny. It's all the the flow brothers
• 18:36
  or sisters that cancer flow, sensor flow to.
• 18:40
  But there are some very bad examples of doing open source
• 18:46
  where it feels like this is not really open source for being
• 18:49
  open source. It's open source for trying to
• 18:52
  get customers but ml flow doesn't really feel like that to
• 18:56
  me. Mo flow feels like it's a much
• 18:58
  bigger Beast than that. Like maybe it started as that.
• 19:03
  But now it's like you opened Pandora's Box and you can't put
• 19:07
  anything back in it. And so yeah, going back to me,
• 19:10
  Kyle has And like, how did that? That look and the execution on
• 19:16
  it was brilliant in my mind, appreciate that.
• 19:19
  It was definitely intentional. So we didn't kind of overextend
• 19:23
  ourselves and open source and create this widely adopted
• 19:28
  platform by accident. But we then have to maintain and
• 19:31
  serve it was very much our goal to build out a set of standards
• 19:35
  that we felt could capture what it is to be a machine learning
• 19:40
  platform. And And I'm all life cycle tool
• 19:43
  and and deliberately to do it in open source that way it
• 19:47
  benefits, both citizen data scientists and other
• 19:51
  organizations but it also benefits adopters, Enterprise
• 19:55
  adopters of the platform that decide to offer it as a service.
• 19:58
  So it benefits data bricks to have ml flow exist as a standard
• 20:02
  that other people know about it exists as a an entry point into
• 20:07
  the em all up space that other people can go and say, hey I'm
• 20:10
  looking for an elapsed. Form what's out there?
• 20:13
  Oftentimes, one of the first platforms they'll see is an all
• 20:15
  flow. And then they find that a beta
• 20:17
  bricks is a great place to use about flow and and that way
• 20:23
  you're both contributing to the space.
• 20:24
  You're establishing thought leadership.
• 20:26
  You're helping make machine learning applications better and
• 20:30
  you're also driving some number of customers and driving some
• 20:35
  adoption for data breaks. So it's a win-win and that was a
• 20:39
  lesson from the spark model. If spark started as a as an open
• 20:43
  source tool data breaks wasn't really even going to be a
• 20:48
  for-profit company. As I understand it in the early
• 20:51
  days it was all about just building out and Barks and
• 20:54
  offering it to other organizations, just making ETL
• 20:58
  and data management, all that better.
• 21:01
  And obviously, it didn't quite go that way, but that was the
• 21:04
  initial Mission and Ethos. So, I think that pattern seems
• 21:09
  to work. 242. If you for three years, I guess.
• 21:14
  It depends on how you evaluate Delta and other projects.
• 21:16
  But so far, I think it's there's a proven track record.
• 21:21
  You have multi multi-billion dollar company that just
• 21:23
  randomly fell out of out of an open source project that is so
• 21:27
  is a nice thing to find you know, under the rainbow or
• 21:29
  something. It is.
• 21:32
  Yeah and it's from the sound of it was not expected.
• 21:36
  So interesting kind of on the topic of Of Open Source is like
• 21:43
  a fascinating concept just in general in software development.
• 21:46
  You know, there's this idea of vanity metrics, right?
• 21:49
  Like GitHub stars is one of the most obvious ones where hey, the
• 21:52
  more stars, we have more successful.
• 21:54
  We are as a project and yeah, I'm not super convinced.
• 21:57
  I think the people that are really in open source like don't
• 22:00
  necessarily buy into that is the main metric of success.
• 22:04
  When you think of something like I'm a flow, what is your idea of
• 22:07
  like a successful open source project and how do you measure
• 22:10
  your success? S one of the main metrics, it's
• 22:13
  a great question that we're actually still trying to refine.
• 22:15
  So when we give, for example, of a Public Presentation of what is
• 22:19
  ml flow or what are the updates demo flow.
• 22:22
  Oftentimes we will cite vanity metrics is it makes us look good
• 22:26
  it's like hey we're closing in on 11,000 GitHub Stars you know,
• 22:30
  we have over a million downloads a day all that kind of fun stuff
• 22:34
  because that gets people excited says hey this is this is a
• 22:37
  widely used project. Now when we're actually managing
• 22:41
  A community and building out the next generation of the platform.
• 22:46
  For example, those are the things that were generally
• 22:49
  looking at. So, typically the focus is on
• 22:54
  how many customers and on the database side of things.
• 22:58
  But also how many open source organizations are adopting this.
• 23:02
  So we get really excited about logos on the on the website
• 23:06
  people reach out to us by email and say, hey my organization X.
• 23:11
  Our of people is using this platform and we're doing these
• 23:14
  things that's and that gets us really excited because it says,
• 23:18
  hey, we're the, we have this diverse set of use cases and
• 23:21
  these different workflows and collaborative settings, where
• 23:26
  tools being is being used. And I think that's really
• 23:29
  motivating and shows that were on the right path, similar thing
• 23:33
  for data, Rich. Customers that customer growth
• 23:35
  is really important in the adoption there because that's
• 23:37
  often a much stickier than a one-off pie pie.
• 23:41
  Loved her example. So those are super good metrics,
• 23:44
  we actively monitor, new issues, and new poll requests.
• 23:48
  It's something we're trying to get better about making sure
• 23:51
  that we follow through and continue to address those.
• 23:54
  I think we're on the right track there, but that's an excellent
• 23:56
  signal. If people are actively filing
• 23:58
  issues, it means they've spent more than five minutes, trying
• 24:01
  to use this thing and if they're filing pull requests, that
• 24:05
  usually indicates an even more substantial level of investment.
• 24:09
  So, when I think about mfo, one of the things that I think about
• 24:14
  is just how easy it is to set up and get your hands dirty with
• 24:19
  it. And I feel like it is very much
• 24:21
  that first taste of the in the lobster drug.
• 24:26
  It's like the gateway drug into Mo UPS in a way, because it's
• 24:30
  just a pip install away, right? And I'm wondering what the
• 24:35
  fundamentals or what are some of the pillars that ml flow is
• 24:38
  built on. When it comes to the design
• 24:41
  decisions that you made because you can always try and have
• 24:45
  everything and have your cake and eat it too.
• 24:48
  But at the end of the day, you have to make some hard decisions
• 24:51
  when you are implementing this. So what were the things that you
• 24:56
  try and always come back to or still?
• 24:57
  Are I imagine like do you have a North star or the vision
• 25:02
  statement or things that you tell yourselves?
• 25:06
  Definitely. Yep.
• 25:07
  So, in terms of ease of use, one of the kind of key obstacles
• 25:12
  when somebody setting up a platform, is just getting all of
• 25:14
  those dependencies rainbow that set up.
• 25:16
  So, for example, you could imagine a world where in order
• 25:19
  to use and all flow for the first time I need to install, my
• 25:23
  SQL and spin up a SQL server or something like that.
• 25:27
  And connect my ml for tracking server, which I have to run
• 25:31
  using a separate process. Spinning up a flask server at
• 25:35
  the Those two things together. Yeah that would probably take
• 25:38
  some people out of time. One really nice thing about the
• 25:44
  corporation and all flow is it's designed to be compatible with
• 25:48
  any host that has a file system. You can just go out there and
• 25:51
  start logging things to the this file based representation of ml
• 25:56
  flow storage and it requires no setup at all.
• 25:58
  Like I'm also knows how to read and write files and that's just
• 26:01
  awesome very similar thing with some of the other components.
• 26:06
  Ponents like ml flow models. A model is just this really thin
• 26:10
  wrapper around some native ml framework.
• 26:14
  So if you managed to install tensorflow or you managed to
• 26:17
  install Pi targe, you can start saving them all, flow models
• 26:20
  without requiring any additional dependencies.
• 26:23
  So keeping a very thin set of software requirements that
• 26:28
  actually have to be installed and set up and configured to get
• 26:30
  started with the platform is really important.
• 26:34
  Another thing that we come back, To time and time again.
• 26:37
  Is that each of these components of them all flow, the tracking,
• 26:41
  the models, the registry, they should all stand as their own
• 26:44
  pillars to use that word. Again, they user should be able
• 26:48
  to leverage each of these components, totally
• 26:50
  independently of the other ones so that you're not requiring,
• 26:54
  that everybody install everything and figure out how to
• 26:58
  get everything set up every time they want to accomplish a
• 27:01
  particular task, and that I think has been successful as
• 27:04
  well. Let's, let's people.
• 27:06
  Oil kind of adopt tracking or address adopt models, or adopt
• 27:09
  the registry that organization without having to reason about
• 27:12
  the rest of the ecosystem. We obviously like them too and a
• 27:15
  lot of people start with one and they get somewhere else over
• 27:18
  time. But I think it's it lowers the
• 27:22
  threshold for adoption. That's crazy because I think
• 27:26
  about Cube flow and I think how it's a little bit different than
• 27:29
  that. And a lot of people have talked
• 27:33
  about the headaches that cute flow brings and you see Kind of
• 27:36
  happening now with K serve how is is doing that?
• 27:40
  I think in the past it's always been you've always been able to
• 27:44
  only use select pieces of cube flow but it's I definitely don't
• 27:50
  think it was part of the fundamentals of the project that
• 27:54
  allows you to just. All right we want to use this
• 27:57
  one little piece. Let's make it as easy as
• 27:58
  possible to get up and running with that, and then later we can
• 28:01
  grow and, and blow it out. Sorry Caillou.
• 28:06
  I'll cut you off, man. What were you going to say?
• 28:07
  No, no, no. Please know that.
• 28:08
  I think that. That's a good question.
• 28:10
  Keep going. Keep going to be traced.
• 28:12
  Well, I didn't really have a question for say around this,
• 28:15
  that's just fucking shit on to flow.
• 28:19
  One of my favorite pastimes. It's funny in the earlier days,
• 28:25
  I think we've come back to it a couple of times because people
• 28:27
  have asked a, should we spend time making sure that I'm also
• 28:31
  integrates well with good flow. We've done a little bit of
• 28:34
  prototyping and the It seems the time and time again that whoever
• 28:38
  is diving into. It comes back and says man that
• 28:40
  was hard to use it's just purely due to the volume of set up
• 28:45
  getting that the zero to one problem I guess is truly an
• 28:49
  issue there. One of the things I wanted to
• 28:52
  call out in terms of ease of use is going to developer
• 28:54
  ease-of-use. That's something that Ben's been
• 28:57
  doing a little work on and I think is also really important
• 28:59
  and keeping a community healthy and and making sure that you get
• 29:03
  good contributions without requiring somebody to spend Five
• 29:06
  hours figuring out how to develop on your platform.
• 29:09
  So been, maybe that's something you want to speak to.
• 29:12
  He's he left us, he said I'm done with this.
• 29:15
  I'm out of here, I know I was talking to him on slack.
• 29:20
  He said it's power went out and he'll try and join in a minute.
• 29:23
  Well, that'll do it yet. Apologies is still showing up on
• 29:26
  my guest list here. So I figured I was wondering why
• 29:29
  he was not video but that'll do it.
• 29:30
  Why wasn't talking? Yeah, I hope he gets the power
• 29:34
  button. So quiet, man.
• 29:36
  Why? So yeah, I mean, we can we can
• 29:43
  definitely get to that when he comes back or surely if he comes
• 29:46
  back, if his power ever comes back on.
• 29:49
  But actually, I wanted to kind of go back to one of the things
• 29:52
  that you said, Cory earlier, about sort of the pillars, write
• 29:56
  these kind of like, in a sense I'm a flow is like, multiple
• 29:59
  projects in one, right. And there's I can have each of
• 30:01
  these independent pillows. The artifact, you know, kind of
• 30:04
  store the model store. Like kind of the tracking
• 30:06
  component and I think it's best that you mentioned that in a
• 30:10
  sense they should remain, or they should be able to exist
• 30:13
  independently. Because that's like, personally,
• 30:16
  that's how I that was. My gateway drug into demo flow
• 30:19
  was, I was just looking for something better than tensor
• 30:20
  board. I was like, this is not fun to
• 30:22
  use and that's when I first adopted ml flow is like a hey,
• 30:26
  this is significantly better. I'm able to, you know, actually
• 30:29
  use this without running weird commands.
• 30:31
  The so that was my entry point into in the kind of the broader
• 30:35
  ecosystem and I In that sense, it's been like a pretty
• 30:37
  effective strategy for forgetting people to try
• 30:41
  something out. See, they like it, you know,
• 30:43
  kind of make each of these components really strong on its
• 30:46
  own and then have that be the enabler into the next broader
• 30:50
  set of tools that they can adopt.
• 30:52
  So they can, he's kind of easing people's kind of path into the
• 30:55
  project or do think is really well done.
• 30:58
  Appreciate that. Yeah, it's it's nice for users.
• 31:04
  It's also nice for developers and maintainers in order to it
• 31:09
  creates more robust platform to, in order to test the platform
• 31:12
  you can test each of its individual components and in
• 31:16
  relatively hermetic and isolated environments and that's helped
• 31:20
  us I think build out a production grade platform and
• 31:23
  open source as well. So there's some benefits to both
• 31:25
  sides. Whoo.
• 31:28
  Coming through the other side of that which is you can try and
• 31:35
  eat up too much of the work flow where you can try and do too
• 31:38
  much and become this end-to-end tool that is opinionated and
• 31:42
  maybe doesn't do things for certain use cases or whatever.
• 31:47
  How do you look at that and like making sure or I mean there are
• 31:50
  some pieces of ml flow that seek to be this end-to-end tool and
• 31:57
  and it's like Like a, I mean not quite but it does follow in the
• 32:03
  same vein of a data, robot or h2o or something like that.
• 32:07
  And so, what are the ways that you're thinking about best in
• 32:12
  breed or full on one tool to rule them all?
• 32:17
  We we've also been going back and forth on that for the last
• 32:21
  four years, to my opinion on this.
• 32:26
  And that the opinion of others might differ is We really have
• 32:30
  to realize that vision of being a complete end-to-end polyps
• 32:35
  platform, in order to win market, share be most useful, to
• 32:40
  users data. Scientist machine learning
• 32:42
  engineers and we incrementally move further and further along
• 32:47
  in that Journey. But we're still not totally
• 32:49
  there places where I think we should go.
• 32:53
  Our kind of upstream, and downstream of the existing Apple
• 32:57
  platform you have - like data ingestion and management feature
• 33:03
  ization and integration of features with models on the the
• 33:07
  upstream and I don't think we've touched and then Downstream you
• 33:10
  have model monitoring see ICD improvements.
• 33:16
  For example, a notion of web hooks which exists in the
• 33:18
  database products but doesn't exist, open source.
• 33:22
  And then more, comprehensive model monitoring.
• 33:25
  These are all things that are really important.
• 33:29
  For users and and I think they're growing as a problem
• 33:33
  space. So as users and customers, get
• 33:36
  familiar with them all Ops, and maybe they've had success using
• 33:39
  tracking and using our deployment functionality and and
• 33:43
  projects in schools like that. Then saying, hey, we're running
• 33:46
  into this problem, where we, we don't know how to version our
• 33:51
  data and make sure that we can see exactly what Theta was used
• 33:57
  to build a model or Having trouble efficiently passing
• 34:01
  queries to our model because the volume of data is too large, or
• 34:05
  once a model hits production, we're not sure if it's
• 34:08
  performing sort of well enough, but I do think we need to reach
• 34:12
  those areas, but Demetrius going to, as you pointed out, it's
• 34:16
  important to have good apis. Here that are inclusive.
• 34:19
  That capture most of the work flows and that's the part that
• 34:22
  sometimes takes a long time. Yeah, we're dealing with nascent
• 34:26
  spaces with a lot of these problems.
• 34:29
  And one thing that you point out there that I think has got to be
• 34:33
  such a fascinating Evolution, that you've seen just unravel
• 34:38
  before your eyes is the ml op space mature with the ml Ops 2.
• 34:45
  So it's like as ml flow is maturing the overall space
• 34:49
  because we are so new right now is also maturing.
• 34:53
  And so you're seeing these different use cases.
• 34:55
  And people are hitting new bottlenecks that I'm sure to or
• 34:59
  Three years ago they weren't hitting because of the way that
• 35:03
  they were looking at these problems.
• 35:05
  Yeah, you nailed it it's exactly the case and it's an awesome
• 35:10
  opportunity at the same time. It can be a little bit stressful
• 35:15
  as a platform developer because all of a sudden, you know,
• 35:17
  you're trying to maybe vet something and you don't see,
• 35:22
  maybe the there's a clear product Market fit or that
• 35:25
  enough people are asking for it. And then six months later,
• 35:27
  people are beating down the door saying, hey, We need this
• 35:29
  feature set and like, okay, what what tools are out there, and
• 35:33
  then each of them is purpose-built and opinionated
• 35:36
  and think, okay? Abstracting over these as hard.
• 35:39
  And even asking you to individual users, should have
• 35:42
  what do you need out of this? A good 10 15 different
• 35:44
  responses. So it can be, it can be tricky
• 35:48
  which is why maybe we've moved more slowly than I think
• 35:51
  sometimes the community wishes you would have into these other
• 35:55
  areas but we're working on it and I would imagine we'll make
• 36:00
  some progress there soon. Hey, I'm Vishnu.
• 36:03
  I'm a data scientist at first hand, and I definitely think
• 36:06
  that you should subscribe to the ml Ops coffee sessions podcast.
• 36:10
  It's the best podcast out there to stay on top of what ml Ops
• 36:13
  actually is to talk to the truth, thought leaders in the
• 36:16
  space. And oh, by the way, dimitrios is
• 36:19
  absolutely hilarious. What a weird guy, you should
• 36:22
  definitely subscribe to the podcast.
• 36:24
  You mentioned at, can I do go back to one of the things you
• 36:26
  said earlier around your opinion?
• 36:28
  Is that The the ultimate vision for a mail flow, should be more
• 36:32
  end-to-end like are there all these other parts of a stack
• 36:35
  that made you guys haven't addressed yet in and there's a
• 36:38
  way to kind of hear that and think, oh, that's because you
• 36:40
  guys want to own everything and like lock people in to the
• 36:43
  platform as much as possible, which is, you know,
• 36:45
  understandable from kind of like a data breaks.
• 36:48
  I'd totally makes sense. But there's also an alternative
• 36:50
  way to look at it which is that, you know, this is driven more by
• 36:53
  usage, like people's needs that it is by like a top-down
• 36:58
  corporate design. Fire to like have people
• 37:00
  all-in-one platform, right? That's actually people, when
• 37:03
  given the choice, don't really want to have to pick everything
• 37:07
  themselves, right? That really, they want to be
• 37:09
  able to just say, hey here's my One Stop Shop for every single
• 37:13
  part of the mo life cycle. And I'm curious how you think
• 37:15
  about that trade off. Yeah, it's nice.
• 37:18
  That those objectives aligned so well in terms of what to invest
• 37:23
  in. So what's better for the data
• 37:27
  scientist and machine learning? Near is is often better for a
• 37:31
  business to the fact that the most database customers are
• 37:37
  asking for end-to-end tools and most open sores users are asking
• 37:40
  for end-to-end tools because it's just so hard to stitch
• 37:44
  together a bunch of different platforms and deal with API
• 37:46
  version and changes and behavioral nuances is awesome.
• 37:51
  So we aren't really dealing with this kind of tugging of the
• 37:54
  heart and different directions. And ultimately, that means we
• 37:57
  just get to roll up our sleeves and And it's all problems and
• 38:00
  know that we're doing the right thing for both sets so it's cool
• 38:03
  that remains aligned and I hope it continues.
• 38:06
  Yeah. What do you guys think my sense
• 38:08
  is if I could just jump in as I think that You know, to put it
• 38:14
  as bluntly and as kind as possible, I think people tend to
• 38:17
  be especially developers, we tend to be kind of lazy like
• 38:19
  when given the choice we do like flexibility but like at a
• 38:23
  certain point you don't want to spend too many cycles thinking
• 38:26
  about things that someone has already solved.
• 38:28
  So I think that the broader trend is toward end to end.
• 38:33
  And I think that a lot of groups organizations do recognize that,
• 38:35
  which is why they're all spear heading in that direction, right
• 38:38
  data. Robot H2O Sage maker mean, they
• 38:40
  all kind of want to own that stack as much as possible in
• 38:43
  addition. Fact that of course, it's more
• 38:44
  commercially lucrative to have people use.
• 38:47
  You can charge them for every part of the stack rather than
• 38:49
  just, you know, the 11 tool they're using.
• 38:52
  So I think that you always have people that want to stitch
• 38:54
  everything together because they like, you know, they're
• 38:57
  libertarian and don't want to get, you know, they don't want
• 39:00
  that, they want to stick it to the man as much as possible.
• 39:02
  Like, you don't you dare tell me how to do everything but I think
• 39:05
  that for the bulk of users if you can give them something
• 39:09
  that's like really good, they'll forego some flexibility to To do
• 39:14
  that, you know? I mean if you can solve 80% of
• 39:16
  the 85% of their problems, that's a hot take.
• 39:21
  I like that. Tweet that one, tweet that.
• 39:23
  Yeah. All the libertarian ml engineers
• 39:26
  at exactly. And I think you see that
• 39:33
  especially with people who are coming into the space and not,
• 39:37
  they're coming in a little bit new and they just want to figure
• 39:40
  out how can I get from 0 to 1 as quickly as possible?
• 39:43
  And so that's why you do see a lot of success around.
• 39:47
  I think about Sage maker and how much people complain in the ml
• 39:52
  Ops Community slack about how Sage major can't do this and
• 39:55
  save maker, can't do that. But at the end of the day
• 39:58
  they're still using it. So they chose to use it and then
• 40:02
  it gets them 80% of the way there and that other 20%, they
• 40:07
  have to work hard on and so it does make sense that you would
• 40:12
  want to try and do that. And I Like this idea of the end
• 40:16
  user and the business values are very aligned.
• 40:18
  So you can try and Tackle them both and at the end of the day,
• 40:22
  if the end user doesn't like it, they're going to vote and
• 40:26
  they're going to show you and you're going to realign and
• 40:30
  figure things out. Exactly and if the end users
• 40:34
  complaining about it, so, you know, they're complaining about
• 40:37
  it and open source and an Enterprise context and we have
• 40:42
  yet to find a moment where it's at.
• 40:44
  Hey, the, the ask in the needs and the workflows are totally
• 40:48
  orthogonal or opposed, we're not at that point yet.
• 40:53
  And I hope we never get to it because that that becomes
• 40:56
  tricky. Yeah, like how do you how do you
• 40:58
  justify to accompany the leadership that something is
• 41:01
  important. Then It's nice that we don't
• 41:02
  really have to, and just one one thing to build off of what
• 41:06
  nature cetera, which I think was a really, really good point.
• 41:09
  It's, you know, ml Ops ml tooling is still like a
• 41:12
  relatively new space, right? And so there's every year,
• 41:15
  there's this new influx of people coming into the space and
• 41:19
  people that come into the space today.
• 41:21
  Compared to people like the oh jeez, they came into this face,
• 41:24
  you know, five five plus years ago, but months ago.
• 41:27
  But by months ago, right, I was going to people that, you know,
• 41:31
  if we go really, really Far back before you know people that were
• 41:33
  still writing their own gradients before tensorflow and
• 41:36
  all that stuff. You know, these folks have
• 41:40
  because at the time there were not these tools, right?
• 41:43
  These tools didn't exist, they had to patch together, their own
• 41:46
  Solutions and developers. Once they have a system that
• 41:49
  works for them we'll just kind of stick with those systems
• 41:51
  because like I don't want to learn a new tool like this is
• 41:53
  what I'm gonna keep doing. This is what I did before and so
• 41:55
  you know I definitely have these biases as well.
• 41:57
  We're like yes sometimes a new tool come along and I just don't
• 41:59
  really want to use it because this is already something I'd
• 42:01
  like figured out. Kinks of my like crappy or
• 42:03
  solution, but like I know that solution you what I mean versus
• 42:07
  people that are just coming in today a month ago, you know,
• 42:11
  next month, these people are don't have their kind of
• 42:14
  preconceived notions about how things should be done, right.
• 42:17
  And they don't have their own existing tooling that they
• 42:19
  already want to use. And so it's much easier to kind
• 42:21
  of own their own. Their mind share a little bit
• 42:23
  more, and kind of indoctrinate them in the way of like an
• 42:25
  end-to-end system, you know, like in ml flow or or Sage
• 42:29
  maker, right? Definitely so.
• 42:34
  Oh, go ahead. Yeah.
• 42:36
  At the end of the day, we're just purely.
• 42:42
  I think in an egalitarian sense in the open source side of
• 42:45
  things trying to do what's most helpful for for practitioners.
• 42:50
  So if that means that, you know, you've been successful with your
• 42:53
  tools and stack and workflows for five years, and you don't
• 42:57
  need anything else. I know I said, I wouldn't
• 43:00
  probably take on the effort switch platforms.
• 43:03
  Just for the heck of it, right. I don't think we expect that
• 43:05
  other people will either But as, as use cases in needs evolve, I
• 43:09
  think there does become this poll of platforms where the
• 43:16
  developments being done for you. In contrast, the continued
• 43:20
  stitching and globbing on additional components to your
• 43:24
  existing stack and you can do both and maybe it's more
• 43:29
  appealing to go with the devil, you know initially but I think
• 43:32
  it would be a Time Savings for a lot of folks to consider After
• 43:36
  that, that job, even if it might take slightly longer and it
• 43:40
  really saves a lot of time in the future.
• 43:42
  Yeah, it's a great question and something we look at a lot.
• 43:45
  All right, Cory, I want to jump into some Community sourced
• 43:48
  questions because I put it out there on slack and on Twitter
• 43:51
  that we were going to be chatting with the mo flow to you
• 43:54
  and we had quite a few people respond which is awesome.
• 43:58
  And shows you how stoked people are about this product.
• 44:01
  And and just how big the ml flow Community is First one that I'm
• 44:07
  going to give a shout out to Matt Cain.
• 44:09
  This one got a ton of upvotes. We gotta do this one first
• 44:13
  because I want to make sure we get to it.
• 44:14
  It's a little bit long-winded but I'm gonna read it off to you
• 44:17
  and tell me if you need any clarification afterwards, Mass
• 44:22
  asks, or says, I'm curious about if they've ever considered
• 44:27
  adding a project abstraction separate from the current ml
• 44:33
  project thing that experiments Runs and models say underneath
• 44:38
  for instance the project might be my cool stock prediction
• 44:42
  project with an experiment underneath it called SDM and
• 44:48
  another called extra boost. Oh sorry list is the mm no list
• 44:55
  and actually boost the goal would be to time multiple
• 44:59
  experiments together under one umbrella as if you had some
• 45:03
  production model and you wanted to use it.
• 45:05
  Experiments to experiment on that model, the lack of this
• 45:11
  kind of abstraction always bugs me.
• 45:13
  This is how I think about projects and we've sort of
• 45:16
  hacked it together by trying a model named to an experiment
• 45:21
  name and using that shared name as a project by bundling them
• 45:25
  together but it's not perfect. Absolutely, it's a fantastic
• 45:29
  question. We're aware of the corresponding
• 45:31
  GitHub issue with the 30-plus, thumbs-up and votes, and all
• 45:37
  It's actively maybe this community members will be really
• 45:40
  excited to know that we are looking at that space though.
• 45:44
  Not maybe not exactly with the solution that some folks might
• 45:50
  have had in mind. So introducing and of a new
• 45:54
  layer of abstraction or contents level, there is kind of a big
• 46:00
  move. So we have, you know, we have
• 46:02
  the Theta Brooks implementation of them all flow.
• 46:04
  We have a lot of third-party E implementations a journal is a
• 46:07
  big adopter. So rather than a kind of making
• 46:10
  that cord to the the ml flow API.
• 46:13
  For example, we're thinking that it might be really nice to just
• 46:17
  let users search and filter their experiments based on
• 46:21
  pre-existing data like right. Now, this we've had this
• 46:24
  long-standing limitation where all you can do is list
• 46:26
  experiments. And in terms of search
• 46:29
  capabilities, you can't filter them by name.
• 46:32
  You can filter them by tag things like that.
• 46:35
  You can imagine representing a project as a sort of unified or
• 46:40
  consistent tag on a collection of experiments, and if you want
• 46:43
  to pull all the experiments with a particular tag, then you can
• 46:47
  go ahead and fire off a search query, their UI or API and pull
• 46:52
  in all of the experiments corresponding to that project
• 46:54
  tag. That also increases flexibility
• 46:57
  in my opinion because say you have an experiment that might be
• 47:00
  shared by. I don't know multiple projects
• 47:03
  for example. I can't think of it.
• 47:05
  Stays off the top of my head but you could tag the experiments
• 47:08
  with multiple projects and then perform the search that way.
• 47:12
  So, that's one place where we're definitely invested in making
• 47:15
  improvements, and the community should see something within the
• 47:18
  next six to eight weeks and hopefully, that makes some folks
• 47:23
  happy. And if that if that doesn't
• 47:26
  happy to chat about of them, have a GitHub issue and see what
• 47:29
  we can do to adjust, super cool. Yeah, shout out to Matt for that
• 47:34
  one and I've got some more. I mean I've got a few more
• 47:38
  actually. So are there any plans of
• 47:40
  implementing our back and multi-tenancy features in ml
• 47:44
  flow? And and this is me speaking.
• 47:47
  Now you said how much like how big the ecosystem has become 4
• 47:53
  ml flow. You have the data breaks, you
• 47:54
  have the Azure, does that make it really hard to start
• 47:59
  implementing bigger pushes or bigger bets in certain
• 48:02
  directions? If that's a good question, I
• 48:06
  wouldn't say that and of the platform growth and better
• 48:11
  brakes and a dry mouth has, made it hard for us to make big
• 48:14
  improvements. It's just made us.
• 48:16
  I think a little bit more. Deliberate about what we
• 48:19
  actually end up prioritizing. And once once we've recognized
• 48:21
  that something is important. I think our velocity is still
• 48:25
  pretty good. Our ability to iterate there.
• 48:28
  So when it comes to things like like our back and multi-tenancy
• 48:32
  and kind of those Only Enterprise features.
• 48:34
  We could build something like that and open source at all flow
• 48:40
  and I don't think anybody would have major qualms about it or
• 48:44
  try to block that our goal. I think right now with respect
• 48:49
  to that class of issue should be to make it as compatible as
• 48:55
  possible with industry standard tools that people use for for
• 49:00
  our back and multi-tenancy not necessarily to build an
• 49:03
  opinionated. Model and then off low, but to
• 49:06
  make sure for example, that it works with an engine X reverse
• 49:09
  proxy. We users have had some issues
• 49:12
  with that in the past. I think we should invest there
• 49:14
  and make a deliberate effort to make sure that workflow is like
• 49:17
  that or compatible that also might extend within plug-in
• 49:20
  layer where people can plug in their own, their own off and
• 49:24
  their own sort of notion of user identification.
• 49:27
  Haven't seen the overwhelming need for that yet, but it's
• 49:31
  definitely something that we could consider.
• 49:33
  Fascinating to think about how we were just talking about the
• 49:36
  idea of what to eat up and or what parts of the stack to take.
• 49:44
  And then here, which parts of the stack to hand off to a third
• 49:48
  party. And so it's clear that you
• 49:52
  thought deeply about this and you're you're constantly
• 49:55
  thinking about what should we make our priority.
• 49:58
  What should we make? What we want to fix and what can
• 50:03
  we hand off? And what can someone else do
• 50:05
  better? Because if we do build something
• 50:09
  that is opinionated about this inevitably, it's going to make a
• 50:14
  lot of other use cases or certain people potentially, like
• 50:19
  it's going to end it's going to alienate certain use cases.
• 50:24
  So that's not like Let's keep going with these questions.
• 50:29
  When will you implement login features to the experiment page?
• 50:35
  I think that's a ditto on the the previous answer there.
• 50:40
  Yeah, Austin and user management and we would love I think to
• 50:45
  have a conversation about what a layer with layer of abstraction
• 50:51
  might look like and and I'm all flow.
• 50:53
  I don't think we've taken The opportunity to have that
• 50:58
  dialogue with the community yet and we've certainly triage, the
• 51:01
  asked multiple times and Revisited whether or not it's
• 51:04
  something you want to invest in but getting more specific and
• 51:08
  into the weeds and figuring out exactly what the requirements
• 51:10
  are. Seems like a good exercise to do
• 51:13
  immediate plans to do that, but we're definitely open to
• 51:15
  discussing how to make the integration of third-party off
• 51:20
  Easter sweet. So What is the challenge that
• 51:27
  most companies or clients you've worked with face when
• 51:31
  operationalizing ml flow in their stack?
• 51:35
  What's been one of the hardest things?
• 51:38
  And we're challenges and what's been one of the most frequent
• 51:41
  challenges This is where I wish that and we're really here to
• 51:46
  speak from the customer perspective because he's super
• 51:48
  close to the metal on that. But he's doubled up enough
• 51:52
  information. And I've talked to enough
• 51:53
  customers that they have a sense of kind of what some of those
• 51:56
  challenges are. So a lot of it is is not so much
• 52:01
  infrastructure related, which is good.
• 52:04
  It means we're doing our job. It's how do I actually, like,
• 52:08
  most effectively use these tools, like I'm also tracking
• 52:12
  and ml flow Structure by workflow as going back to what
• 52:19
  we talked about in the beginning ml flow tracking is kind of a
• 52:21
  very simple abstraction API for logging a bunch of stuff and
• 52:25
  querying that stuff but how you decide to log that stuff and and
• 52:32
  lay it out and query it later can really there.
• 52:36
  So you have some customers that doing time series forecasting
• 52:41
  for example that say you know, hey we have Have a hundred or a
• 52:46
  thousand different categories of product or in the being sold in
• 52:53
  a variety of locations. And so we're going to build a
• 52:55
  model for each of those. We're going to log them all
• 52:57
  separately in a going to retrain them every hour.
• 53:01
  And then all of a sudden that's oh crap.
• 53:05
  I'm having trouble figuring out what was trained when because
• 53:08
  it's just a mass of data, that's like that all flow.
• 53:11
  I'm having trouble operationalizing.
• 53:13
  These models gonna need the pole like another, the best model for
• 53:17
  each category back and then write all this code to do it.
• 53:20
  So they kind of, they end up locking themselves into some
• 53:23
  complex workflows and through, I wouldn't say a fault of the
• 53:27
  tooling, but because it's open and an opinionated customers
• 53:33
  often struggle with what's, what are the best work flows within
• 53:35
  that platform to adopt? And that's something that we're
• 53:38
  hoping to make easier for the time series use case, and Again,
• 53:43
  I wish Benton, we're here to talk about that.
• 53:45
  But then also, for General ml Ops workflows, introducing, some
• 53:49
  more opinionated Concepts that we've recently published a
• 53:53
  mouthful pipelines, are FC in the open source community.
• 53:58
  And the goal of a pipeline is to be substantially more
• 54:01
  opinionated about what it is that you should be logging and
• 54:03
  how you should be accessing it and basically create an even
• 54:07
  happier path to production. I have a little bit of a
• 54:09
  different question from the community so you know One of the
• 54:14
  maintainers of the project and obviously a lot of fantastic
• 54:17
  work that you've done and I'm sure a lot of fantastic things
• 54:20
  still to come, but, you know, this didn't always have to be,
• 54:24
  didn't have to be this way. Like there could have been a
• 54:25
  different set of universes where you did other things.
• 54:29
  So, my question is, I guess my question on behalf of the
• 54:32
  community is, if you weren't working on ML flow, what would
• 54:35
  you be working on? Would you want to be working on?
• 54:39
  Yeah, me personally that's a great question.
• 54:42
  I really enjoy and a web service, architectures
• 54:47
  distributed systems and things like that.
• 54:49
  So if I didn't have this glorious opportunity to explore
• 54:52
  that and in open source but then we also within data bricks and
• 54:56
  I'd probably be architecting and code monkeying around the the
• 55:01
  Enterprise side of things and that I enjoy that kind of work.
• 55:05
  I like building platforms. Open source is awesome and it's
• 55:09
  super cool to have that opportunity to engage with in
• 55:13
  open source community and build out this tool using systems
• 55:19
  engineering and platform design considerations, but I think
• 55:22
  ultimately yeah that's that's the interest of - is platform
• 55:26
  development nice. When you think about different
• 55:30
  projects that have been built on ammo flow or not projects, like
• 55:35
  using ml flow. I think about it.
• 55:37
  Jake's like on top of it, melt flow or things that take
• 55:41
  inspiration and use ml flow as a foundation.
• 55:45
  Have you seen any that you really like?
• 55:47
  I remember that probably like two years ago.
• 55:51
  One of the community presentations that we had when
• 55:54
  we were first. Starting was this framework, it
• 55:57
  was like a scaffolding around ml flow called Hermione and it
• 56:01
  really made ml flow a bit more intuitive for a certain subset
• 56:07
  of Users, have you seen stuff where you're like?
• 56:10
  Wow, that's a really cool thing to do on top of it, like some
• 56:13
  kind of app or something that you you like and you want to
• 56:16
  give a shout out to. So it's a great questions.
• 56:20
  We've seen a whole bunch over the years and I think
• 56:23
  unfortunately we haven't seen ones that are heavily adopted
• 56:28
  over a period of time that have had our radar.
• 56:30
  So I'm hesitant to shout out any particular thing worrying that
• 56:33
  might State Myself by about the two or three years.
• 56:36
  The This point, but there have been all sorts of cool blog
• 56:39
  posts about. Hey, we used ml flow to do this
• 56:41
  so we built an extension layer and some of those those projects
• 56:46
  have fed contributions back into the ml flow fluent API surface.
• 56:51
  So we've had big improvements to our search abstractions and to
• 56:55
  our run management and stuff like that as a result of people
• 56:58
  going out and building their own software on top.
• 57:01
  So I think I'm going to shout out anything.
• 57:02
  It's the collective folks that have decided, hey, We build
• 57:07
  something useful. We want to contribute it back to
• 57:09
  the mainstream and then you'll get that emerged in.
• 57:14
  Thank you so much. It's been really helpful for the
• 57:16
  growth of the project would that does feel like another this is a
• 57:22
  proxy metric, maybe, and it's very hard to quantify, but
• 57:25
  knowing, if you have a healthy product or a healthy community,
• 57:30
  that is a great metric to look at, like, how many people are
• 57:33
  creating things on top of what you've built, Is much more
• 57:37
  powerful in my eyes than GitHub Stars.
• 57:42
  You're absolutely right Demetrios and I think it's a
• 57:45
  great signal for us moving forward.
• 57:47
  So there's that also metrics like how many different
• 57:51
  projects? Embed ml flow is dependency.
• 57:54
  We had to look at that more. So I appreciate the guidance
• 57:56
  there if you're absolutely right.
• 57:58
  All right, I've got a few rapid fire questions for you before we
• 58:03
  finish. I mean there was one question
• 58:05
  was kind of for Ben, but maybe I'll ask it for you before the
• 58:08
  rapid-fire questions, just to fake you out a little bit.
• 58:11
  Let's see if you have any opinions on it.
• 58:14
  How do you feel, or how do you think about feature stores and
• 58:17
  ml flow and using the two of those together?
• 58:21
  Fantastic question, that there it's a Synergy that's important
• 58:26
  and needs to happen. So on data bricks, for example,
• 58:30
  there's a feature store offering if there are some in some ml
• 58:33
  flow Integrations, I think that there's more To be done there,
• 58:38
  and that feature management. It is core to the development
• 58:44
  and production ization of ml models.
• 58:47
  And as a result it's very much within ml flows domain.
• 58:51
  So we're looking at it, we're trying to figure out what the
• 58:53
  affections might look like. And I think people that need
• 58:57
  this kind of functionality should have their needs met
• 59:01
  sometime in the next year or so would be my guess.
• 59:04
  Yeah, we're definitely interested.
• 59:06
  All right. Right now, the lightning round
• 59:08
  that we've all been waiting for. What was the last bug you
• 59:12
  smashed in? I was just saying production.
• 59:15
  Yeah, I was gonna say in production, but then I thought,
• 59:17
  no, just what was the last book you smashed?
• 59:19
  Just give me, give me the last one definitely.
• 59:23
  So we smashed some pretty brutal performance issues with Emma,
• 59:27
  full tracking on sql-based stores, which is, which is
• 59:31
  pretty critical. We were able to take logging
• 59:34
  performance for a batch of And metrics from something like a
• 59:39
  thousand of these to 30 seconds down to about less than a second
• 59:43
  or two seconds. So that was that was really
• 59:48
  recent and really important for performance tracking.
• 59:52
  What was I mean, take away from that.
• 59:55
  What did you learn in? Tell us absolutely that pushes
• 59:59
  much as you can and to SQL because that's just such a
• 1:00:03
  heavily optimized ecosystem with. 50 plus years of
• 1:00:08
  investment as you're not going to do better trying to Wrangle
• 1:00:10
  Your Own Thing Upstream. Wow, okay, now last book, you
• 1:00:15
  read last book, I read. I've got it sitting on my desk.
• 1:00:20
  This philosophy of software design, I've had my classic and
• 1:00:23
  they gave it to us. Actually, when I joined the
• 1:00:25
  company and probably should have read it sooner, oh nice.
• 1:00:28
  Well, I'm going to do something because I saw somebody else
• 1:00:31
  doing this and I thought it was awesome.
• 1:00:35
  We're going to give out That book to someone that comments in
• 1:00:40
  the YouTube page, let us know what you think about and we will
• 1:00:45
  give a free copy of that book to you.
• 1:00:47
  So give us some comments. What you think of this
• 1:00:50
  conversation. Now, what piece of Cory is that
• 1:00:55
  apply to us as well? Does that apply to us as well?
• 1:00:57
  Because I don't have that book. Can I get a comment like or
• 1:01:02
  comment on this? All right, man.
• 1:01:05
  But you gotta do it with your Fake Avatar or your Twitter box,
• 1:01:09
  or whatever set that up right now.
• 1:01:11
  Yeah, exactly. So our Quarry, what piece of
• 1:01:19
  technology are you bullish on? That might surprise people.
• 1:01:22
  It doesn't have to be coding technology.
• 1:01:24
  It can be like iot of your house, or whatever.
• 1:01:27
  But is there something that you think it can be coding?
• 1:01:30
  It can be whatever. I don't think kubernetes would
• 1:01:34
  surprise anyone but is there something?
• 1:01:37
  That you're particularly interested.
• 1:01:39
  These days are you've gone down a few rabbit holes and you want
• 1:01:41
  to see more of happening? Don't say crypto, please don't
• 1:01:45
  think so. And if T is no, not that Cliff
• 1:01:54
  this is going to be boring but and I just want to continue
• 1:01:56
  promoting as much as possible, but that pipelines for from Alan
• 1:02:02
  for data, I think our Super useful.
• 1:02:06
  And we've seen general-purpose orchestration tools for for
• 1:02:10
  pipelines in the past and things like air flow and in others that
• 1:02:14
  have gotten a ton of adoption. And it's super useful.
• 1:02:16
  But I think there's a special characteristics in the canal,
• 1:02:19
  space pipelines, can really help with and I'm excited to see how
• 1:02:22
  that plays out the very much in the Wheelhouse.
• 1:02:24
  Pretty much in the same space. And I'm thrilled about it crypto
• 1:02:28
  school too. But I don't think I'm really an
• 1:02:30
  expert enough to talk about it. Not so bullish on that.
• 1:02:33
  Now, that makes me very Excited because I've got my friends over
• 1:02:37
  at Zen ml. Who would be stoked to hear you
• 1:02:40
  say that and they are working in that space.
• 1:02:43
  Currently right now last question for you and then we're
• 1:02:46
  going to roll. How do you want to be
• 1:02:49
  remembered? I would love to be remembered as
• 1:02:52
  the person that helps everybody from the citizen data,
• 1:02:55
  scientists to the Enterprise, do em, all ops better.
• 1:02:59
  And I hope that folks feel like they're on that Journey with me
• 1:03:04
  and with us, The team and I hope that we can continue to make
• 1:03:09
  them happy and deliver what they need.
• 1:03:11
  Excellent answer. Love it man.
• 1:03:13
  Much progress, thank you for coming.
• 1:03:15
  This has been awesome. Thank you so much for having me
• 1:03:19
  on I hadn't realized. Yeah, thanks so much, Demetrius
• 1:03:23
  me. Hal so much fun.
• 1:03:25
  I'm up close in. Good hands in good hands.
• 1:03:27
  This is this has been very, very inspiring appreciate that.
• 1:03:32
  There was a question, but it's like, what's the roadmap look
• 1:03:34
  like for a mail flow in the next whatever?
• 1:03:36
  But I'm sure that everybody that is an awesome question, I will
• 1:03:40
  also in 15 seconds, we are working on a public-facing
• 1:03:44
  roadmap V2. So we wrapped up our first ever
• 1:03:47
  public facing roadmap. Epic 75 percent contribution
• 1:03:51
  rate from the community on that which is great.
• 1:03:53
  We're doing it again and just one other thing to shout out as
• 1:03:58
  we're hoping to expand the set of maintainers over time from a
• 1:04:00
  flow. Don't want this to be a data.
• 1:04:02
  Rich driven project lookout for opportunities there as well.
• 1:04:07
  Please get involved and help us. Execute awesome.
• 1:04:10
  How do you incentivize people to get involved?
• 1:04:15
  Three books, or ebooks? Pretty bug spray questions from
• 1:04:18
  the Ops Community? We do have also t-shirts and
• 1:04:22
  mugs. So if that's a feeling your
• 1:04:25
  contributions will be noted and release notes that are published
• 1:04:28
  and on the website on. On Twitter and GitHub.
• 1:04:33
  So you get some of the personal branding side of things as well,
• 1:04:37
  which is nice.