Latest / Agentic Conversations (formally mlops.community) / Why You Need More Than Airflow // Ketan Umare // Coffee Sessions #109

Transcript

Why You Need More Than Airflow // Ketan Umare // Coffee Sessions #109 · Agentic Conversations (formally mlops.community)

• 0:00
  Hey, my name is Katie Tom and I have been having coffee recently
• 0:06
  with oat milk. I love oatmeal classes.
• 0:10
  George we got Kate on whom are on today and I'm going to
• 0:14
  confess to something before we even start talking about that
• 0:16
  awesome chat that we just had with him.
• 0:19
  I've been going into the forest recently and where I live
• 0:23
  there's wild raspberries everywhere.
• 0:26
  So basically if you see any raspberry around my Mouth right
• 0:30
  now, that's because I've been on a diet of strictly raspberries,
• 0:35
  maybe it's going to be a new fad.
• 0:36
  It's the raspberry diet. Stay healthy.
• 0:39
  Everybody out there and each raspberries, dude, it's amazing.
• 0:44
  I'm every time I get one, it's like, ah, so good and I love it.
• 0:50
  This is the first time that I've ever found something like this
• 0:52
  and it's awesome. So anyway, we talked with Kate
• 0:57
  on today, we talked all about like, Basically orchestrators in
• 1:01
  the ml space air flow flight, what it looks like.
• 1:06
  What were your thoughts on it? Do you have any takeaways?
• 1:09
  Very knows. We're good.
• 1:10
  He knows his stuff in the space. It sounds like they're providing
• 1:14
  a framework to make sure you get things right in the mo workflow
• 1:17
  area. Lot in the way of the same on
• 1:19
  the analytics, it's rare for the sort of the same sort of level
• 1:22
  of depth to be going into how you orchestrate machine learning
• 1:25
  at scale. Yeah, that's so true.
• 1:27
  I love how he talked about. How Things can get messy a lot
• 1:31
  quicker than you would expect. And I feel like anyone who's
• 1:35
  ever gotten to a significant amount of scale knows that and
• 1:39
  they're just like saying, yeah, you're preaching to the choir
• 1:42
  man. I totally understand what was I
• 1:46
  can't remember his exact words but it was like something around
• 1:50
  how you know you start with something simple and it's very,
• 1:54
  very tiny deployment and then quickly I can build and now you
• 1:59
  start. You've got these dependencies
• 2:01
  and next thing, you know, what was it?
• 2:05
  I think it was 600 pipelines or running together at flight and
• 2:09
  lift and you can't really tweak any of them because you don't
• 2:13
  know what is dependent on the other and things break all the
• 2:17
  time. And it's just a headache to try
• 2:19
  and sort out where it's broken or why it's broken, or if you
• 2:22
  change something, if you update something it it can break.
• 2:26
  And I feel like that is In the as you're starting out you think
• 2:33
  about but it's not really in your reality, right?
• 2:37
  It's not something that you're like, oh well, that's not really
• 2:39
  a problem that I'm facing right now, but I would like to keep
• 2:42
  that in my mind and the back burner and then, like you said,
• 2:46
  you can just wake up one day and boom, it is your reality and you
• 2:49
  gotta go and figure it out. So it's cool that he has been
• 2:53
  thinking about that the other part that I loved.
• 2:55
  And I want to hear what you think about the signal idea that
• 2:59
  he had Yeah, no, it sounds. I mean, trying to get
• 3:02
  reinforcement learning. Its production is a famously
• 3:04
  difficult to the problem. It came like requires.
• 3:08
  You can't do the same sort of offline validation to the same
• 3:10
  quality that you can with deep learning because you need to be
• 3:13
  actually making a decision based on the inputs on what your
• 3:16
  models, how your models acting? So there's a whole new unique
• 3:20
  set of technical problems to be solved there.
• 3:22
  And if flight sort of distinguish themselves based on
• 3:25
  that sort of feature set, then I think they could probably go go
• 3:28
  really far, probably the next Big thing for machine learning
• 3:30
  to innovate it. I found my man, the CEO and
• 3:38
  founder of the team that brought you flight.
• 3:42
  Union aii is back with us again. We've also got George here.
• 3:46
  How you all do and what's going on Caton?
• 3:49
  It's been a long time. This is round two now.
• 3:52
  What's happening? Yeah, I'm looking forward to it.
• 3:56
  Last time was a lot of fun, I don't think they're see, this is
• 4:00
  going to be better so yeah, thank you for having me.
• 4:03
  This is what I tell all the guests to say when we first
• 4:06
  start. So it brings up the anticipation
• 4:10
  and excitement. So this is cool because George,
• 4:14
  I know you've got a little forte in the dark arts of pipelines
• 4:19
  and so we wanted to bring you on here.
• 4:22
  Kate on basically Ben's 24 hours a day, thinking about
• 4:28
  pipelining, and pipelining for data engineering and pipelining
• 4:31
  for machine learning, since you've been on here caetano, a
• 4:36
  lot has changed with Julian. I mean, you guys weren't even
• 4:40
  let's give a little context last time you were on here.
• 4:42
  You were still working at lift, like flight was an open source
• 4:47
  project that you were debating. I remember we finished the call
• 4:51
  and I was like you think you'll ever start a company around
• 4:53
  this? And you're like, I don't know,
• 4:54
  maybe it's kind of a lot of work fast forward.
• 5:00
  Two years in the future, you've got your own company.
• 5:03
  You are just killing it. And so let's talk about what is
• 5:07
  going on and what has changed since we last chatted I started
• 5:12
  a company but I guess it. I'm reading a book 0 to 1 by
• 5:20
  Peter Thiel. And by interesting, I was just
• 5:22
  of the chapter where they were like, most individuals should
• 5:27
  not be starting a company because the power law is are
• 5:30
  against them and which does make sense when it's like the chance
• 5:36
  of Success Through starting a company with a Chance of Success
• 5:39
  Through going, you know, Looking at a much more successful
• 5:43
  Endeavor, that exists already. That's cute.
• 5:47
  So you do much better off working at a company.
• 5:49
  So the question is, why did I start a company?
• 5:57
  I have been working for about 15 plus years and sadly.
• 6:06
  And I walked across the different fields, high frequency
• 6:09
  trading banking. No mapmaking, Logistics, right?
• 6:16
  Sharing, and cloud. And, and across all of these.
• 6:20
  I've kind of done this. While solving the real problems,
• 6:25
  I have been always had to solve some infrastructure related
• 6:29
  issue all the time and that has always been around this.
• 6:32
  Like not really clearly defined as pipelines.
• 6:36
  Like, we, we only think about pipelines the data engineering
• 6:38
  sense, but they exist everywhere.
• 6:40
  How do you think you get a easy to machine when you request
• 6:43
  forward, it goes through a freakin big Piper.
• 6:46
  What happens when you order and Amazon not like order goes
• 6:49
  through a massive Pipeline and because why Plans are a very
• 6:54
  natural way for humans to think about it is just another lie
• 6:59
  that they're great organizational tools to actually
• 7:02
  really, really allow you to scale, folks, independently.
• 7:08
  And so when I saw that, and when I saw what's happening with
• 7:12
  machine learning, There's a big gap in the way we build
• 7:19
  software. Just traditionally think about
• 7:22
  databases. Let's take an example mongodb,
• 7:26
  If I tell you I just started a new company that just means a
• 7:30
  new database. It's good.
• 7:32
  But give me two years will be better and you probably going to
• 7:36
  be if I put enough resources at that number of people with good
• 7:39
  experts I can build a really good database given time but
• 7:45
  let's take a machine learning model, 2019 December and maybe
• 7:51
  doordash in lift trying to Target the customers and 2020.
• 7:56
  March, what does models don't work anymore and the reason why
• 8:01
  is it, is it something fundamentally changed within
• 8:04
  those companies? Now, but what impacted was
• 8:08
  something outside of the control of the company was environmental
• 8:12
  political factors, right? Like the NGO world was going
• 8:16
  crazy with the pandemic. And how do you react to this?
• 8:21
  You have to sometimes change your assumptions concept curve
• 8:25
  you know we say model so are sometimes you have to change the
• 8:29
  data sometimes and these changes have to happen rapidly and if
• 8:33
  you try new things to kind of deliver this product, very
• 8:37
  quickly, and what I said, at least in my opinion, My point of
• 8:43
  view is that. Software and and machine
• 8:48
  learning products are data-driven, Machinery.
• 8:50
  Products are extremely a kind of diametrically opposite and if
• 8:54
  they are diametrically opposite, the existing set of
• 8:56
  infrastructure that we have for solving software tooling is just
• 9:02
  not well. Suited for machine learning,
• 9:04
  it's actually wrong because you have not taken this fundamental
• 9:08
  assumption that the things are going to change, constantly
• 9:12
  constant flux. And the moment that assumption
• 9:14
  kicks in, you're like, oh, Order whatever we did.
• 9:17
  For the last 10 years is not going to work as it.
• 9:20
  We have to really rethink and I was like that excited me.
• 9:23
  That was like, that is a challenge and somehow we are we
• 9:28
  are very early when we are very early 10 years ago if I asked
• 9:32
  you would you need all this devops and you would be saying.
• 9:35
  Now when I say when you started AWS and you had all these
• 9:40
  on-prem and I was there, lots of on-prem cluster than it, will
• 9:44
  you go to cloudy and say now, And the reason is because that
• 9:48
  transition period is the hardest because you don't know which
• 9:52
  side is right and so on. And we are on my opinion, we are
• 9:55
  in that transition period. And I was like, let me take that
• 9:59
  transition period, let me take a my own stab at how this world
• 10:03
  could be shaped up for solving such problems.
• 10:06
  Wait so you said software and data infrastructure are
• 10:11
  diametrically opposed right specific even machine learning
• 10:15
  It comes to machine learning dive into that a little bit more
• 10:18
  because that's an interesting take.
• 10:20
  I like it. Yeah.
• 10:23
  It's like diametrically opposite like an extreme statement and I
• 10:28
  love to make made an extreme statement here.
• 10:30
  But what I'm saying is that the way the let's see the software
• 10:37
  evolves, you know, machine learning product has a different
• 10:40
  life cycle as compared to a traditional Software, write it
• 10:46
  in software is stateless. Very usually, when you think
• 10:49
  about things like databases Services, gearing somebody like
• 10:54
  making a payment pretty stateless.
• 10:57
  You know, we know what we want to achieve.
• 11:00
  There are databases or transactions of distributed
• 11:02
  systems hard stuff. I'm not, don't take me wrong,
• 11:06
  I'm not saying we have solved it completely but with kind know
• 11:10
  how to do this stuff. We've built a lot of experts in
• 11:13
  this field. You like a lot of kids, but on
• 11:15
  the other hand, most companies are if let's take a look at who
• 11:23
  are the companies who have actually delivered fantastic
• 11:27
  machine, running products to their customers, Google.
• 11:32
  Maybe Facebook quickly tails off.
• 11:35
  This is some Amazon and Microsoft and maybe a little bit
• 11:37
  of, you know, the jail, two companies, but most of the
• 11:42
  companies have not been able to do this as in a very effective
• 11:45
  efficient, and repeatable way. And the reason and oftentimes,
• 11:51
  even if it is deployed, its fragile gets broken.
• 11:54
  And that's because The nature as I talked about is different,
• 11:59
  like we are constantly in a flood like, like I know teams.
• 12:03
  This is what they did. They would deploy a piece of
• 12:08
  machine learning product and then they would immediately say,
• 12:10
  okay, you've done this, can we move to your new project that
• 12:16
  just throw a note when you project you have to continue
• 12:19
  working on it probably because immediately a pandemic was
• 12:22
  happening and all your models is screwed.
• 12:26
  Sorry, you got to babysit this bad boy.
• 12:29
  Exactly. And, and this is, and the way we
• 12:33
  have done data science today, or machine learning being taught,
• 12:37
  this is not part of the curriculum.
• 12:39
  This is not how we do. We kind of jump from problem to
• 12:42
  Problem, the, the fantasy of like going, you know, just
• 12:45
  throwing data at solving about, which may be fine.
• 12:49
  For some people, some people are Builders.
• 12:52
  And then some people have to maintain this stuff.
• 12:54
  So, so that. And how do you bring This as a
• 12:57
  philosophy, as a other infrastructure piece is what I'm
• 13:01
  interested in and the helps people do this.
• 13:04
  Amazing. So I want to jump in to the idea
• 13:11
  around Pipelines and I love. This is how you talked about how
• 13:18
  there's pipelines everything basically is a pipeline.
• 13:21
  It's very easy for us, humans to understand pipelines and to
• 13:25
  really wrap our heads around it. There is a pipelining tool or an
• 13:31
  orchestration tool that we all know and love airflow, right?
• 13:36
  And I think everyone's played around with it, it has a very
• 13:40
  data engineering spin. To it and then there's what you
• 13:46
  all are doing it flight and you're really trying to take it
• 13:51
  to the next level specifically for machine learning.
• 13:56
  What I would love to just chat about for the next.
• 14:00
  However, long is where does airflow fall short?
• 14:04
  Why is like is using to like flight and air flow together?
• 14:10
  Is that too many tools wood. Would you want to use them both?
• 14:15
  Should it just be flight? Like, what are your thoughts and
• 14:18
  your vision? Really on how that looks, as we
• 14:21
  move forward and we go from this transition from the on-prem to
• 14:25
  Cloud, but just in the machine learning world.
• 14:29
  Yeah, good question. You can imagine.
• 14:34
  I'm, I've been debating this question for a while you with
• 14:36
  lot of people, there are people in Within the industry that
• 14:43
  thing. If so should I take this some
• 14:48
  other books, this should exist? There are some people who just
• 14:53
  love their flow. I am politically right camp, but
• 14:58
  not really politically, right? I am more fundamentally knowing
• 15:02
  the ground reality. Yeah.
• 15:05
  So If I am a company. That has existed for a few
• 15:11
  years. Three years.
• 15:15
  I have potentially already have are for running.
• 15:19
  In the company, right? Maybe some couple other tools
• 15:23
  but like that potentially have. Now, the cost is, should I
• 15:29
  migrated all the things I've done from airflow to like a new
• 15:34
  shiny tool, and And what do I get out of it, but that's the
• 15:40
  real reality of it. And In some cases.
• 15:46
  Specifically with ETL, you don't eat too much you'll get
• 15:50
  something good but it's a new way of thinking if we do some of
• 15:54
  the stuff you to rewrite the code and so on, if you just want
• 15:57
  to do exactly the same stuff that you were doing with their
• 15:59
  flow and just migrated over to anything, you probably are
• 16:01
  getting very little specifically if they're floodwaters.
• 16:07
  But you are definitely want to do more computed, then system
• 16:13
  your body, double machine learning products.
• 16:17
  Airflow lacks over there and and here's like, I'll give you a
• 16:20
  bunch of examples. We will, we will not do a full
• 16:24
  debate between what are the differences and so on, if
• 16:27
  boring. But like really, let's try jump
• 16:31
  to the sense of like, what we are trying to achieve, right?
• 16:35
  As I said if, if we take one that that statement that I said,
• 16:39
  if we, if you agree that if you learning products are a constant
• 16:42
  flux constant change and are different from Products.
• 16:48
  And I'm going to put it a little software product category, even
• 16:51
  though they are really not, but let's put for a minute, in that
• 16:55
  case, then you need tools that allow you to do quick.
• 16:58
  I trips it has to happen quickly.
• 17:00
  It has to be cannot be like, oh, let me, you know, coordinate
• 17:04
  with the world and like, get everybody to use control flow
• 17:07
  2.0. And then then to go and deploy.
• 17:09
  It just doesn't work. That's not gonna work.
• 17:12
  I need to give that power to the user.
• 17:17
  To push down the power to the user.
• 17:19
  That means they should be able to change whatever, they feel
• 17:21
  like try something new, get an output and potentially not that
• 17:27
  outputs are gonna work. So try again, try again and try
• 17:31
  again, right? This modality is not supported
• 17:34
  by our phone is just it's just the wrong tool for that.
• 17:37
  It was never. Secondly, I think we in the same
• 17:44
  scenario when you're trying things out, certain things are
• 17:47
  going. So I don't want to redo those
• 17:48
  things again, and again. And you need the ability to
• 17:52
  reuse results from the previous Also, we probably you've you've
• 17:57
  talked a lot about feature engineering and future stores
• 18:00
  and all that, right? I have a slightly different,
• 18:04
  take on this right. Features are feature stores are
• 18:07
  very important in production but which is George have a lot of
• 18:10
  problems in the early early days of adoption cycle.
• 18:15
  That's because you don't know what features you want, but you
• 18:18
  somebody else, grab some features and that's where the
• 18:20
  feature store Discovery and all that comes into play.
• 18:23
  But even those may not be materialized.
• 18:25
  We why should I materialize? All the features of the word,
• 18:27
  like, I'm not going to use them today, so let's not materialize
• 18:30
  that that's DNA, the materialization, but even then
• 18:32
  Discovery is a What? And and you may want to reuse
• 18:36
  parts of like the future generation pipelines and that is
• 18:41
  very, very hard to do with them. So because of its intrinsic
• 18:44
  assumptions on context. And so on another piece is like
• 18:51
  invocations of how the invitations happen and how
• 18:55
  different people can work. One of the one thing that I'll
• 18:58
  tell you that happened within teams, Pricing at lift was
• 19:05
  supply. Chain elasticity to learning,
• 19:09
  basically, when how this is, this is how we control such
• 19:15
  pricing, or we we have little control such facing and it you
• 19:23
  need a bunch of models. You need to know what the
• 19:25
  current supply chain elasticity one, other number of Rivals, how
• 19:28
  to remove can I intend device that can help a bunch of
• 19:31
  different. Is of, controlling the market,
• 19:33
  really doing balance to the market, and there was a team of
• 19:37
  seven or eight people. They had about five models, 600
• 19:43
  Pipelines. All working together to build
• 19:48
  like this, like five models are actually better production.
• 19:51
  At that scale. It's things are getting reuse.
• 19:55
  Things are getting moved around. Some pipelines are deprecated
• 19:59
  some pipe and renew and all of them exist.
• 20:01
  They cannot delete the code because they don't know if they
• 20:04
  will want to revive it again some parts of it.
• 20:07
  And so this complexity, how do you handle this in effort?
• 20:09
  Just doesn't work. It is one Team, 600 bike racks.
• 20:14
  80 other teams you know you can imagine the complexity and
• 20:19
  multiply that spotify's here is just like whatever 25,000
• 20:23
  pipelines thousand, Repose thousand, repos GitHub repos,
• 20:29
  having pipeline code in them and constantly changing and just
• 20:36
  that's the reality of companies that are dealing with machine
• 20:39
  learning products and and data products at scale, right?
• 20:42
  And this this end Other than this quickly happens, you have
• 20:46
  no idea when this is going to happen, you start off with like,
• 20:48
  oh no, I don't need all of this. I mean, just one machine and
• 20:51
  click. I'm good at.
• 20:52
  This is all good and if you have the optimism in the company, we
• 20:55
  should all be optimists in those in that way, that the company
• 20:59
  will improve and immediately you have to Repose and the two will
• 21:02
  become 4. And if it follows a power law,
• 21:05
  just what I mention it immediately, it escalates it has
• 21:08
  hundreds of people and again that's where it's not really
• 21:12
  like shooting. So in terms of the core of
• 21:16
  structions, you implement what really differs from like prefix,
• 21:20
  airflow dike. So you've got kind of your, your
• 21:23
  individual functions. I think the task in your world
• 21:26
  and you have your whole pipelines.
• 21:27
  Something like like jobs is the same the same set of
• 21:30
  abstractions, really the same sort of core structure that you
• 21:32
  can everything to or am I completely believe mistaken?
• 21:36
  It is the affection. One thing that we changed in the
• 21:40
  abstraction logic. Is that the abstract is the
• 21:44
  smallest The task as you said a task in our case is like a we
• 21:49
  derive ideas from function program.
• 21:51
  It's nothing's coming hamadan, right?
• 21:54
  Is a monad. What a monad means it exists on
• 21:57
  its own but that means you can run attachments on and in Flight
• 22:02
  when you go you register things, you can just go hit a button and
• 22:05
  you can you can run the task Walter.
• 22:07
  So like let's say and secondly, the task itself is version.
• 22:11
  So you have all the versions of how they have changed, so you
• 22:14
  can go back. In time and you can run older
• 22:17
  ones you're tracking on what input trigger for older ones and
• 22:21
  so on. And the last part is that they
• 22:24
  can be arbitrarily complex. Now, this is a completely weird
• 22:29
  way of saying it, but I'll give you an example.
• 22:32
  The typical way when you look about think about in airflow,
• 22:35
  that you're running a job. Let's say you're running a By
• 22:40
  constant and in airflow, the python script will run on the
• 22:45
  worker, right? You can run it on the work.
• 22:48
  Now you'll say, oh I want to run a spark up and it won't run on
• 22:52
  the water I have to when I get control in that task a kickoff,
• 22:57
  something in some place here, model relative degrees lost all
• 23:03
  tracking, all this like data flow, or whatever.
• 23:06
  You have to like kind of pass your context to this arbitrary
• 23:09
  system and make it work. And now you were doing Sage
• 23:13
  Baker training ordered like you want just triggering things
• 23:15
  externally and that's one way of thinking we think about it
• 23:19
  differently. Think that this this task
• 23:21
  modality can be arbitrary complex.
• 23:24
  That means when you get triggered, you may be a simple
• 23:27
  python script that runs whatever you want for dinner.
• 23:31
  You could be a distributed spark job that runs on multiple
• 23:36
  containers, but it within the same modality, the
• 23:38
  expressiveness, the code writing, everything is the
• 23:41
  Saviour. You write python code by sport
• 23:43
  coat and it becomes a spark cluster, right?
• 23:47
  And then the same code goes to Sage maker or data breaks.
• 23:51
  And so, the modality in, in this is, this is a very small
• 23:54
  difference. In terms of.
• 23:55
  It looks like it's ux which actually core in trenched into
• 23:59
  the system for this, the system had to be designed with specific
• 24:03
  points in place and I will give you an example.
• 24:08
  I, we want it to give the capability of between users to
• 24:13
  run arbitrary, like the spark jobs on either database or EMR
• 24:18
  or qualities. Kubernetes, being free care of X
• 24:23
  in here, where you have to pay in our Psychology major
• 24:28
  kubernetes cluster, speak three to run in there.
• 24:31
  So we wanted to give that capability and And doing that,
• 24:37
  just by this baton passing is very, very hard specifically if
• 24:43
  they want to right spot code directly within their
• 24:45
  consumption butter. And we also wanted like if you
• 24:47
  write a panda's data frame, we should just get it as a Spartan
• 24:50
  of, it really complex idea. I'm curious.
• 24:53
  Mmm, and so good. I'm really curious.
• 24:57
  We thought about integrating more heavily with like machine
• 25:00
  learning specific packages. So, I was thinking the other
• 25:02
  day, it's only be quite nice to be able to do is to Set up a
• 25:06
  project up front from so like some hyperparameters tuning
• 25:10
  something like opportunity and then you could just offset your
• 25:13
  success criteria for each step and then spin off a new Run for
• 25:17
  the sort of lexan you might want to do.
• 25:19
  So if this works, try something else Etc and typical data
• 25:23
  engineering, sort of workflow orchestration tools, don't do
• 25:25
  this, but you make your tools specific for machine learning
• 25:28
  workflows. I just wondered if that's on
• 25:29
  your road map, potentially. Yeah, we are doing so already
• 25:33
  like, for example, if you write I thought Source like it or not,
• 25:37
  you can ask packages, you get auto check point if you of
• 25:42
  course, the important thing but checkpointing is very key.
• 25:46
  It's like a very arid. If you think about pipelines
• 25:49
  have pipelines are checkpointing, but take.
• 25:52
  You cannot check point within a task unless, you know, that the
• 25:57
  task is running our training tasks and and training tasks
• 26:00
  cannot really be easily split into multiple workflow steps is
• 26:04
  just doesn't work correctly. It's not a farmer it's like it's
• 26:07
  not the right abstraction. So yeah, we support introduce
• 26:11
  checkpointing without the users, having to know.
• 26:13
  And if you do retries within the retries will automatically pass
• 26:16
  the checkpoint between the retrace.
• 26:18
  Similarly, when you actually output a model, we understand
• 26:21
  the model and we understand the context very output.
• 26:24
  So, for example, if you are on a GPU and you have to put the
• 26:26
  model, now, you want to store the weights in a device
• 26:29
  invariant way. So that when you actually load
• 26:31
  it on a CPU which just just work without you having to think, oh,
• 26:35
  Oh, let me do on CPU or GPU over yet.
• 26:39
  So these are kind of some of the features including if you output
• 26:43
  a t.f. tensor. The right way is not pickling if
• 26:47
  tense, the right way, is to actually convert it into
• 26:49
  approach. It's got above representation
• 26:52
  and serializing in and sending it to the next part of whatever.
• 26:55
  If you want to store it, these are interest intrinsic
• 26:59
  assumptions that the system knows about right?
• 27:02
  And and, and it was the way we have layered it is that it's
• 27:06
  fully accessible. All of these things could keep
• 27:08
  on extending for forever as things change and evolve within
• 27:11
  the machine learning ecosystem. So yeah, so I was not going to
• 27:14
  go into Epic features, but thank you for putting me in that
• 27:18
  direction. No, no, it's yes.
• 27:19
  The songs that. Yeah, there are and there are
• 27:23
  people, who are, we are actually working with a team that's
• 27:26
  coming out too. So flatter rayon flight
• 27:30
  essential because right is an orchestrator, IT orchestrates.
• 27:34
  Your think about it as a meta layer on kubernetes.
• 27:38
  It allows you to run, random scheduling on kubernetes, and
• 27:41
  managing, one of the reasons why because we are working with like
• 27:45
  teams from Spotify shocked, if I threat ra figuring out okay, I
• 27:49
  want to use re who brings a tray cluster.
• 27:53
  I just the user just wants to use Ray, can you imagine Supply
• 27:57
  brings up the right plaster and coating multiple steps.
• 28:00
  It actually initially we are doing single step is one red
• 28:04
  cluster but across multiple steps were going to be reused.
• 28:08
  The same rate cluster within the lifespan of our.
• 28:11
  And this is these are like things because you know that Ray
• 28:14
  has a plasma store, it can reuse the data that it has stored.
• 28:17
  So all of these are intrinsic assumptions that you need for
• 28:20
  machine learning and that's it. Into flight.
• 28:24
  So so who do you think of your kind of met core Target uses?
• 28:27
  Is it kind of Enterprise level companies, where they might have
• 28:29
  a machine learning specific work, filtration tool, and then
• 28:33
  a separate kind of tool design for more analytics, Focus,
• 28:36
  functionalities, or do you think it can work for across the
• 28:38
  spectrum of different sizes? It can work for both, right?
• 28:43
  That's absolutely like like, for example, Spotify users it
• 28:46
  across, for this tax rate at lift.
• 28:50
  Many, many analytical work load. And the things that we still use
• 28:54
  their flocks of our Legacy stuff was on our floor lift many, many
• 28:58
  workloads run outside, including some, the analytical side, some
• 29:01
  even from the offside. For example, what we did is we
• 29:06
  built an air flow to flight. Operator flight has an API.
• 29:10
  Always Was API first 15 years ago, like this today, if you
• 29:15
  have first, so we built an operator that just goes and
• 29:18
  triggers into flight just like as an aw service.
• 29:21
  That's open source. Now it's available, be delayed
• 29:25
  in open sourcing sugar and we think we want to do that with
• 29:31
  everybody. If the effect would be open to
• 29:33
  it, you would love to work with them.
• 29:34
  If Dexter would be open to it, you would love to work with them
• 29:38
  today with the used to be a tax on flight also.
• 29:41
  Ian but I don't know. Yeah.
• 29:52
  Like like if they want to hand off the more compute intensive
• 29:56
  workloads to flight that. Absolutely right.
• 30:01
  Which should work. So if you are organization like
• 30:04
  is using A simple like some simple workloads on and they're
• 30:10
  not compute intensive, they're just simply like running queries
• 30:13
  externally and you're happy with their folks continue using that
• 30:16
  and then you get two examples, where you want to run
• 30:20
  distributor trainee, some kind of gpu-based workloads and you
• 30:25
  want to use like all the versioning, all the Primitive.
• 30:28
  You can adopt flight just for that workload, and they should
• 30:30
  should work seamlessly with your existing workflow.
• 30:34
  And I don't think It's and it's necessary that you have to read
• 30:38
  all your in at the moment at least and if then if you decide
• 30:45
  that oh I want to kind of go to one, it could be your choice.
• 30:50
  So that brings up a great point that I was getting a because it
• 30:54
  seems like like you mentioned earlier, things get messy quick.
• 30:58
  You don't even realize it. And all of a sudden it's like,
• 31:00
  oh my God, we got 25,000 pipelines.
• 31:03
  We don't know how to debug these pipelines, we don't know who's
• 31:05
  dependent on. What?
• 31:07
  And when you're building these and say, you're starting at that
• 31:11
  foundational level and then things start to get more and
• 31:15
  more, and all of a sudden, you're needing these gpus, and
• 31:18
  you're needing a bit more compute intensive, and you're
• 31:21
  talking about, well, airflow is great.
• 31:23
  If you're just starting out and it's great for that base layer.
• 31:27
  And then when you want to start getting, if you want to hand
• 31:30
  off, some of this more compute, heavy stuff to flight, you can.
• 31:35
  What does the evidence? Evolution in your eyes look like
• 31:38
  because you mentioned well you can go fully over to flight.
• 31:42
  You can just keep some of them in air flow, but then I feel
• 31:45
  like you still, if you are trying to mix and match like the
• 31:50
  tasks in flight and then the airflow air flow like pipelines,
• 31:56
  you're still going to run into that problem of things being a
• 31:59
  mess and you can't really debug that easily, right?
• 32:02
  If you are already using airflow, you're used to Handing
• 32:08
  off work to some other system that's not new.
• 32:11
  That is, that's how you do it in a way.
• 32:14
  That's probably what you should do with.
• 32:16
  Therefore, if you do start doing computer intensive stuff within
• 32:19
  your are protester, You by. Don't have to tell you guys that
• 32:21
  you probably will not end up in the right.
• 32:25
  And and and so I don't think that increases complexity by
• 32:28
  handing over work to another system that handles that
• 32:31
  complexity better and for the parts that does increase
• 32:35
  complexity. You could trust you You could
• 32:38
  use, for example, like completely for those parts but
• 32:44
  if you have ETL workers that are running are friendly.
• 32:47
  Yeah, at the end of my ETL pipeline, I want to run one
• 32:50
  training cycle. Boom, add one operator and it's
• 32:53
  like calling such Mater or calling any other system that
• 32:57
  that's how it should be. Like, in the simplistic way from
• 33:01
  an airflow point of view, if we're.
• 33:03
  And and so, for example, at lift all the spots, Just to run a
• 33:08
  flight. So we actually had the airflow
• 33:12
  spark operator for spark, you would just hand it over to
• 33:15
  flight and does it make a difference if it gives you an
• 33:17
  API that says, start a job, wait for a job like pole.
• 33:22
  And those two APS is what you need with F1.
• 33:24
  So, it's pretty cool. Those are available today for
• 33:27
  pipelines or tasks, complex pipelines, all of that like with
• 33:32
  in flight, and so you can easily hand it over.
• 33:36
  And I don't think it should look any different than bad for
• 33:38
  whatever other things you want. He is the queen of amongst at
• 33:43
  work and this is your daily dose of among herbs.
• 33:47
  Well, if you are dead serious, about the envelopes, you have to
• 33:51
  immerse yourself in the envelopes content.
• 33:54
  The best way to do it is to subscribe to the email Ops
• 33:58
  Community podcast. All great people are here on
• 34:02
  this podcast soon, good luck, and keep learning for I was I
• 34:07
  was there. I thought George might have
• 34:09
  another one he wanted to jump in but it looks like I was getting
• 34:12
  excited, I've seen that, you've done, you've done some nice
• 34:15
  work. Just very recently in terms of
• 34:17
  better, visualization of the sort of thing.
• 34:18
  You'd be looking at in a basic science pipeline so support for
• 34:22
  markdown rendering various box plots and so on.
• 34:25
  But I keep being frustrated by all these projects to get like
• 34:27
  90% of the way there. But as soon as you get to the
• 34:29
  point, you can just render HTML your into a lovely space or put
• 34:33
  some little bit of run a bit of a joke, jupyter notebook.
• 34:37
  What did he say, Mel? Then whatever bit of python,
• 34:39
  whatever python package, you want to make use of your freeter
• 34:42
  to go. And then you're in the perfect
• 34:44
  sort of Lego blocks. And Roo of I can like, take my
• 34:48
  favorites from each part of the different bits of data,
• 34:50
  engineering and a science and put them together.
• 34:51
  As I wish I was just wondering. Is that something you'd you to
• 34:54
  do in on the line? Is it just harder technically
• 34:56
  than I think it is. Why is that not on the way yet?
• 35:01
  Which part of we did we do have visualization?
• 35:03
  Sorry I didn't catch that part which part is the same.
• 35:06
  So as far as we can tell so far it's marked down and like some
• 35:12
  specific breakdowns of like dataframe statistics and so on
• 35:16
  but not to the full HTML rendering It is full HTML
• 35:20
  rendering, actually, we've just added like our team, only added
• 35:25
  a few. In the way we, it's open to the
• 35:28
  community to add a bunch, right? This is where like adding a lot
• 35:31
  of these small to HTML. It takes a time, right.
• 35:35
  Like so there are, and the thing is, I have to convince a lot of
• 35:43
  people to open source. The work that they've done one
• 35:45
  this stuff, right? It's right.
• 35:48
  Is part of the sound. It's completely open, so people
• 35:52
  can do whatever they feel like and not open source their work,
• 35:55
  right? It does not.
• 35:56
  Like, I know, there are vertex AIT Creations that are running
• 35:59
  in that. Some companies that are their
• 36:01
  integration like people have done pandas profiling, which we
• 36:04
  are actually 0, which we also did, but like some other stuff,
• 36:07
  right? So it gives you a canvas.
• 36:09
  It gives you a HTML canvas. That's what it gives you how you
• 36:14
  come up with that HTML. Is like, there's a simple
• 36:16
  plug-in that you write tomb and it should be able to generate
• 36:20
  it. We are slowly and surely adding
• 36:22
  as we can within the core team win Union, but we are at Weston
• 36:28
  tiny team and we we are doing like 15 other things.
• 36:32
  Like we're building, CD 4 ml, we are doing like my external
• 36:35
  signal approval workflows, we think.
• 36:38
  If you want to do, you know, let's say some sort of amazing
• 36:43
  like multi embedded sore or even a simple time slit experiment.
• 36:46
  How should he do that? You should be able to like the
• 36:49
  claw. Wait for something, great for a
• 36:51
  signal and then go to the next deployment and go to the next
• 36:54
  level and so on and we want to code all of this up within this
• 36:58
  one framework which is want this to be the toolkit that data
• 37:01
  scientist now and learns love. So can we just double down on
• 37:05
  that you saying using using fight for our L deployment?
• 37:10
  Because that sounds like a whole, like, very specific
• 37:13
  technical set of problems and very interesting ones.
• 37:16
  So what do you have done? Will knock.
• 37:17
  Knock. Knock, just had a like let's
• 37:19
  see. Keep our eye out for a minute,
• 37:21
  but even for doing Simple models, like what happens?
• 37:23
  Usually you have something running in production and you
• 37:27
  want a new model to be a be tested with it.
• 37:30
  And the way you do it as let's say you want to deploy it to
• 37:33
  some sort of machines and you want to say okay ten percent of
• 37:36
  the traffic should go here or you want to do the traffic
• 37:38
  shaping happens outside of like that doesn't happen in Flight
• 37:41
  like assuming you are using something like on y or Selden or
• 37:45
  like a bunch of different things that does that shape a but you
• 37:48
  need things to Roll that you need like the brain that saying
• 37:51
  that. Oh, let's push that out.
• 37:54
  And you want one place where you go and see all of them
• 37:56
  happening. So we are building that inside
• 38:01
  and the way we are building it more general purpose.
• 38:04
  So for example, we think about it like signals.
• 38:08
  So for example, you you run a pipeline the pipeline enters
• 38:12
  into a state where it's like, okay, I'm waiting for output,
• 38:15
  signal input signal from an external system.
• 38:18
  It could be, Completely automated.
• 38:20
  So let's say you're doing data drift detection or model drift
• 38:24
  or like some sort of drift detection based on the outputs
• 38:27
  or like maybe even doing the output monitoring may be a
• 38:29
  ground. Truth comes in really quickly,
• 38:31
  you want to check if your new model is performing correctly,
• 38:34
  you could send a signal saying a Bart or roll back or move
• 38:38
  forward to 50%. And so, our goal is one day.
• 38:41
  This should be fully automated just like we do it develops,
• 38:45
  right still far away. So we are building the base.
• 38:49
  Infrastructure to power such a thing.
• 38:52
  Similarly, that same concept to be used for labeling.
• 38:55
  Same concept, abuse for like a bunch of different things like
• 38:57
  semi-supervised lately. And so you're building that into
• 39:00
  the cold air. Yeah, I didn't really get that
• 39:03
  the signals. Can you explain that again?
• 39:05
  Because that sounds really cool, but I'm not sure.
• 39:07
  I fully wrap my head around it, so let's take that as a so
• 39:12
  workflow. As we, we call workflows in
• 39:15
  flight, but like, that's, it's a pipeline, right?
• 39:17
  And in the pipeline, you know, you want to to split your
• 39:20
  deployment into three steps you want to say, 25%, if things go
• 39:25
  well 50% if things go. Well, what a very simple
• 39:33
  scenario, right? How do you do that?
• 39:35
  What is how do you tell the system that things are going
• 39:38
  well? This is no way to externally
• 39:43
  interact with a pipeline that's in progress.
• 39:46
  So the way you do it is you will deploy three parts.
• 39:48
  One pipeline that that's 25%, then you manually trigger
• 39:52
  another pipeline that does the 50%.
• 39:55
  And who passes the data in between that context has to be
• 39:57
  passed in transferred and this complexity increases, right?
• 40:01
  And what we think about this is yeah, do 25%.
• 40:05
  And then say, I want to expect a signal.
• 40:08
  Now, the signal could be a You coming and hitting a button
• 40:12
  saying, I approve it to go to 50% perfectly, normal close to
• 40:17
  50% another again, it goes and wait for a signal and now a
• 40:21
  person comes in and says 50% 100% may go to a another option
• 40:27
  and this is where I'm saying, the future future would fully
• 40:30
  automated, you get data metrics out of this.
• 40:35
  You see not always down to this available immediately but let's
• 40:39
  assume in this case it is you Look at it, you're like oh my
• 40:42
  accuracy has improved. Its actually improving in
• 40:44
  production with this model, go to 50%, automatically 0 accuracy
• 40:50
  is degraded, go back to 25%, things like that, you could
• 40:54
  code. These complex ideas in the and
• 40:57
  and and have experiment happening in production, all the
• 41:01
  time, right through your same workbench that you started
• 41:06
  trying out with that's our goal. So this starts like deviating,
• 41:10
  right? From the typical pipelining
• 41:12
  stuff that people have been taking last charge.
• 41:17
  There is always always The crucial moments.
• 41:19
  I'm going to, I'm just going to change the Wi-Fi.
• 41:22
  It's actually worth it at this point.
• 41:24
  Yeah, so with with that human in the loop soar process this is
• 41:28
  something that really really interest me.
• 41:30
  Normally there's a hacky work around, you can kind of get it
• 41:33
  working by like dropping a file in S3 at the right times.
• 41:36
  You have some sense of waiting for it.
• 41:37
  We have to create your own separate UI.
• 41:39
  Is that sort of natively? Acid in Flight, it's not yet,
• 41:43
  it's coming in the next month. If you join our, I definitely
• 41:48
  recommend people to join the bi-weekly sings.
• 41:50
  This is all in Linux Foundation, open sink.
• 41:53
  Yeah, we we are will be showing the ux marks for the so even a
• 41:56
  you I will be supported for all of this data Valley within the
• 41:59
  pipe and stuff. But of course, everything and
• 42:02
  flight, as always runs through an API, so you can build your
• 42:06
  own amazing, you eyes and power it, however, you want to power
• 42:11
  This externally. So like I think the way we have
• 42:14
  been talking to a few companies, I can't name all the names of
• 42:17
  these but one company, for example, a Spotify again name.
• 42:21
  So Spotify was use this for their royalties payouts, right?
• 42:24
  Like this. That's absolutely okay.
• 42:27
  It doesn't need to be purely machine.
• 42:30
  Learning, the climate come as an example, but in this case, they
• 42:33
  want to see the price and then hit a button and boom, all the
• 42:37
  payouts happen and things like that.
• 42:40
  Right? We would love to power help
• 42:43
  companies power, semi-supervised learning.
• 42:46
  Yeah, that's something that I think about a lot too.
• 42:48
  And I know many people are doing like, I've seen an interview
• 42:53
  people who are doing phds on this stuff and it's just like,
• 42:56
  where does the human touch the loop?
• 42:59
  Like, what part of the loop? Do you put the human in, right?
• 43:03
  And if you're going to automate it, how are you making sure that
• 43:06
  you're automating it in the way that is going to be the most
• 43:10
  beneficial? And if fully understands, and so
• 43:14
  when you're putting something like this together, with flight,
• 43:17
  have you either a seen people? Because I imagine everyone
• 43:21
  that's using flight is giving different signals.
• 43:25
  We could say, or saying this is the signal that we think is the
• 43:29
  best here. Have you seen some really cool
• 43:31
  examples of these signals being used?
• 43:34
  Or do you feel like there is a best practice already put into
• 43:37
  place besides the one that you mentioned before, where it's
• 43:40
  Like if we see the 25% looks good, then someone just click
• 43:46
  the button and it goes to 50% to 75%.
• 43:49
  That said, whilst people are attracted a manual intervention
• 43:55
  and are timed intervention. That means wait for five days of
• 44:01
  project. Like what were those are core
• 44:04
  intrinsic but why build? The platform in a way that
• 44:13
  doesn't allow the Automation in the future, but it's like, what
• 44:17
  comes first? Sometimes it's pretty easy.
• 44:19
  Does it? There is the infrastructure come
• 44:20
  first or does the solution come first?
• 44:23
  Usually, the solution doesn't. And we've seen this, right?
• 44:28
  Automated in multiple ways in devops, so, why it?
• 44:32
  Why should that not be done? And that's where we think we
• 44:35
  have. We're building the
• 44:35
  infrastructure. We are going to open it up.
• 44:37
  The API is open. You can try multiple and and
• 44:40
  within Union. And that's why we started Union
• 44:42
  ml codify. Many of these practices into
• 44:45
  Union ml as a higher level or top of flight where we are
• 44:49
  working with Partners. We are working with all the
• 44:52
  folks who do monitoring and so on.
• 44:53
  And and our goal is we could build a way to do this automated
• 44:58
  one. Click.
• 44:59
  Go for it constantly. Yeah, every 10 minutes,
• 45:02
  something is getting trained and automatic and we used to do that
• 45:04
  actually live every 10 minutes is to deploy New Market to
• 45:07
  production which is kind of like science fiction.
• 45:10
  I feel now. For many people, but we did it.
• 45:13
  We did it with the same practices.
• 45:14
  We put, we had a lot of checks, all those checks would pass and
• 45:18
  we'll get alerted to the. They did not pass and the model
• 45:21
  would not go. So yeah, KS test and all kinds
• 45:27
  of tests to figure out if things are okay or no again this is
• 45:30
  2016 so we did that in 2016. Yeah, so you also mentioned
• 45:39
  something really interesting that I would Touch on which is
• 45:43
  Union verse flight. And what does that look like?
• 45:48
  Like I know people probably come into and are very familiar with
• 45:52
  flight but now Union has launched.
• 45:54
  What exactly is Union? It's on top of flight.
• 45:57
  What are the benefits? Why would I?
• 45:59
  Yeah. Stop using flight and and and
• 46:02
  use Union. Is it just a managed service on
• 46:04
  top of it? What does it look like?
• 46:06
  Yeah, don't stop using fight if you're using it.
• 46:09
  We And I, you know, this is a hard reality and we I treat open
• 46:15
  source as a sank space and this is this is how it should be done
• 46:21
  when I'm talking about open source.
• 46:22
  I should not be talking about Union and I'd like, we're not
• 46:25
  right? So now I'm putting my union hat
• 46:28
  on as a CEO with different you Sherry for yeah, exactly.
• 46:33
  Like a Tax Advisor, whatever you got to say.
• 46:36
  Okay, now I'm putting on my Get blurred in many other places,
• 46:41
  his lines get blurred and it's kind of like confusing and I
• 46:44
  don't want those Vines to be blood.
• 46:45
  Union exists in two modes right today.
• 46:50
  Today Union number one is the top contributed to fly.
• 46:55
  We don't own the trademark to flight.
• 46:57
  We don't own the rights to flight.
• 46:59
  We don't own anything. Just like flight is a standalone
• 47:03
  entity. We assert dominance really
• 47:09
  through our contributions. We just lead the way by
• 47:15
  contributing the most amount of code.
• 47:17
  Like all these features are all getting built into flying.
• 47:20
  We are not building in any Union.
• 47:21
  These are all four flights. And yes, everybody who else to
• 47:26
  take advantage. I know.
• 47:27
  There are companies that are, at least three to four companies
• 47:29
  that are built vertically on top of light, and they are valued
• 47:32
  more than union today. So, no, no way.
• 47:38
  Happens. It's fine.
• 47:40
  You know, but but that's like you know that's how outstanding
• 47:44
  products are built by people like that.
• 47:45
  They should be built in a way that they like will remain.
• 47:49
  There are companies getting value from it which is just
• 47:51
  amazing. It's great for them, it's great
• 47:53
  for the community. That so that's number one role
• 47:57
  of Union. Number two role of Union is that
• 47:59
  we over the years, these like over to your brother, we've been
• 48:05
  people have tried to launch and tight.
• 48:07
  Even today is not that easy to get started with.
• 48:10
  And that saddens me because of the complexities of people
• 48:14
  deployment infrastructure and in dresses and like they can
• 48:19
  security and the right ways of doing security and like things
• 48:22
  like that. And those are very very crucial
• 48:24
  and key things that But sometimes overlooked and that's
• 48:29
  where Union is. Second motor comes when we want
• 48:32
  people to be very efficient and effective in utilizing flight
• 48:38
  but get there quickly and the third final motive where Union
• 48:42
  comes in is we actually as I said, we are trying to codify
• 48:45
  the best practices. To deliver a fantastic machine,
• 48:51
  learning organization and an mL of solution on top.
• 48:54
  I Machinery. Orchestration is a superset of
• 48:56
  ml Ops, right? It's required to achieve a more
• 48:59
  logs, but it's not a drops, right?
• 49:01
  That's where the confusion lies with maybe go, but we are
• 49:05
  building the bunch of Emma logs, practices, that directly.
• 49:10
  Like if you use Union ml, for example, and then what we are
• 49:14
  saying is this a big problem. We talked about features,
• 49:17
  Serving you. There's a big problem, the
• 49:20
  feature transforms and model prediction code in battery.
• 49:24
  So when you're training it usually batch data set and that
• 49:29
  code is different and then once you go to production, the
• 49:32
  prediction gourd is completely different many times and these
• 49:36
  small differences caused a lot of problem.
• 49:37
  We want to eliminate that, we just want to get rid of it.
• 49:42
  And so how do you do that? That's not no way.
• 49:44
  But to create like a compressed practices.
• 49:47
  Has experience and we think good intentions are not enough.
• 49:51
  So we are writing all of it in code and so that's what union is
• 49:55
  that's the third practice and and over a period.
• 49:57
  We are building layers within Union, it will take time and we
• 50:04
  think we that's how we expand the market from.
• 50:09
  Folks are comfortable with flight which is engineers and
• 50:13
  data scientists. And machine learning Engineers
• 50:14
  who are okay with coding and know a lot about 10, like Python
• 50:20
  and Java and Scala and all these things, whatever they are
• 50:22
  coding, but we want to expand it to you, no more.
• 50:29
  I lied engineers and the scientists who, who do not
• 50:33
  really care about Kate, and so on, and for that, that's where
• 50:36
  you didn't exist. That's the third mission.
• 50:38
  That's awesome. And that makes a lot of sense.
• 50:42
  I mean the data scientists who probably are getting their feet
• 50:47
  wet with air flow then want something that is going to be
• 50:51
  more up their alley that goes into the machine learning and
• 50:55
  and if you have something that is easy to plug in like you
• 50:59
  said, I think that's a lot of a lot of this right now and a lot
• 51:03
  of companies in this space are trying to crack that nut.
• 51:05
  How can we make this easy and lighter weight?
• 51:09
  Because when Start dealing with infrastructure, especially
• 51:13
  Legacy infrastructure. And when you start dealing with
• 51:15
  data, and there's the security issues around data.
• 51:19
  There's the pii around data. There's just so many variables.
• 51:23
  And so many dangers and security risks, or just risks in general.
• 51:28
  You really have to be take the necessary precautions and so it
• 51:32
  makes complete sense. Yeah.
• 51:34
  And that's how we are like the Stewart.
• 51:35
  I'll tell you because the see me is now open.
• 51:38
  We had a security vulnerability. Inflight about two and a half,
• 51:42
  three months ago, we patched it in four hours and we deployed it
• 51:47
  to all 35 of the top companies are using flight within two
• 51:54
  days. This is not possible in a purely
• 51:57
  open source. Being it wouldn't be possible
• 52:01
  without having funding in Union, right?
• 52:03
  So that's where we think our role is we are stewards of this
• 52:08
  open source. We have the, we have to help all
• 52:14
  our users. Make sure that they are secure
• 52:16
  and in turn, their customers are secure, right?
• 52:20
  That's our duty. I start to think about, like
• 52:23
  most of my work has just been evaluating different tools and
• 52:25
  exactly this little space and I think of things in terms of the
• 52:28
  like time to approve a concept, I what point can I just show
• 52:30
  this to somebody and say like this is what it could be useful.
• 52:33
  And how long from that point onwards is it take for the
• 52:36
  adoption to actually worth that like learning curves.
• 52:39
  So so you're Go from are floated Acts or ever to fly or something
• 52:42
  similar to that. How how many is it weeks?
• 52:44
  Is it months? At what point does that
• 52:46
  transition become worthy? Even if the second tool is
• 52:49
  definitely better, does it? Take yes, a quarter to hit that
• 52:53
  point and like, who knows what the ecosystem will look like
• 52:56
  when we get, but it sounds like you're on top of both.
• 52:59
  Both those parts of the process. This is where Union Cloud comes
• 53:04
  in today. Number one, is to speed up that
• 53:07
  that if you are again, Might sound like a sales pitch and I
• 53:12
  want to say that I tell if I'm wearing a flight hat, that's not
• 53:15
  what this is. But if you are trying out and
• 53:19
  and you want to Go to it quickly.
• 53:22
  Just don't don't have to set up going to unions out.
• 53:25
  Like it's just night one-click, really?
• 53:27
  And if the data still resides in your own V PC.
• 53:33
  So that way the data gravity is not lost, we take care of the
• 53:36
  hard lifting, right? Looks like getting things up and
• 53:39
  running and so on which is infrastructure stuff.
• 53:42
  And we think, for example, some of our one of our users is using
• 53:47
  in three different clouds, their data, plane's control.
• 53:51
  By the same, one flight, One Union, Cloud flight, plus other
• 53:57
  stuff, right? And, and, and, and they don't
• 54:00
  have to think about, on other users, using on-premise and
• 54:02
  cloud like, and migrating their stuff, what lives?
• 54:07
  And this is happening. And so, yeah.
• 54:10
  So that's how we build this system is essentially to make it
• 54:14
  really, really easy for you to start and achieve your goal.
• 54:18
  Your goal is also to move. Let's set of cloud native
• 54:20
  system. And those are the places where
• 54:22
  we are right, fit? If you are happy with, like,
• 54:26
  running things on one or two machines, which many companies
• 54:29
  are, we probably are not the right fit in with us, you have
• 54:32
  to use kubernetes, right? And all you have to want to use
• 54:35
  Google, it is, right? And we've been working with
• 54:37
  large companies, like large companies, are in the transition
• 54:42
  phase to cover it is. So this is an early time for
• 54:46
  them and and and they're partnering with us as a
• 54:49
  functioning and we have I'm still whether the users are no,
• 54:53
  we have time. We try to tell them what could
• 54:56
  be the right practices and how could you get.
• 54:59
  So yeah, we are clear of essentially of infrastructure
• 55:01
  partner. I mean, speaking of kubernetes
• 55:05
  and let's probably go back to your flight hat, not Union hat,
• 55:11
  but everything about using sound great it didn't sound like a
• 55:14
  sales pitch to me so that was two thumbs up.
• 55:16
  I think it's awesome. And the value props there.
• 55:19
  It's very true and like what George said is so crucial for an
• 55:24
  engineer to bring on a SAS product and get value out of
• 55:28
  that. How fast can you get value out
• 55:30
  of that? And how quickly are you able to
• 55:33
  then transition to doing more impacting?
• 55:36
  Work is so important for anyone to do the company, right?
• 55:41
  Like whatever you're doing that, that and the ability to write
• 55:46
  well, and like, explain the way that you're doing things.
• 55:51
  And Impact other Engineers. Those are two life hacks that a
• 55:56
  really smart friend of mine told me and I had never thought about
• 55:59
  it that way and it's just like boom that gives.
• 56:02
  Yeah, you really valuable you just become like so valuable to
• 56:07
  the company right away. Yeah, yeah.
• 56:11
  People today in the open source command they are, they sometimes
• 56:14
  struggle with like setting up a real grass or whatever and we
• 56:17
  helped even in that case we have them.
• 56:19
  But you know if it's sincere Requested them.
• 56:22
  Maybe start off with the cloud. If you want to migrate migrate
• 56:25
  to the open source, it's okay. But get any impact out, the more
• 56:29
  quickly you get an impact out. The more, you're able to push
• 56:34
  that much more within the organization, your respect and
• 56:37
  your value becomes higher. And so it's not about running
• 56:40
  infrastructure, it's about creating impact for your users.
• 56:44
  That's it. So I got to ask man, we did like
• 56:48
  the last two podcasts. George was on one a week ago and
• 56:54
  then we also had one with this guy Ryan and it was all about,
• 56:58
  we talked about Cube flow and Cube flow pipelines.
• 57:01
  And then we also talked last time George was on, we talked
• 57:04
  about ml flow and since that recording ml Flow came out with
• 57:08
  pipelines. How are you looking at these as
• 57:11
  opposed to what flight is doing? I mean I know Cube flow
• 57:15
  pipelines, is very kubernetes native, it's also got its pains.
• 57:20
  It's got It's downfalls. But are you feeling like you're
• 57:24
  not even in the same ballpark? Like what you're doing is very
• 57:28
  distinguished from Cuba pipelines?
• 57:31
  Yeah, great question. I have to think, because It's
• 57:36
  great that you flow pipelines is creating the awareness.
• 57:41
  And explaining the need for a system like this.
• 57:47
  Sadly. Probably I'm gonna regret this
• 57:52
  sales but they are doing it, not in a way that's very user
• 57:56
  friendly and user focused and that causes people to
• 58:02
  potentially go away from the system.
• 58:09
  We we've taken like so there are 10 Again.
• 58:14
  Always, you can start with first principles that take a radically
• 58:17
  different approach it. That's what we decided to do.
• 58:19
  We decided like What if you have a blank slate?
• 58:22
  Then if you have kubernetes as a thing like that.
• 58:25
  So the reality it's becoming more and more and more really
• 58:27
  like it's getting penetrated hugely then how would you do it
• 58:32
  differently? That's what we did.
• 58:36
  And that's how we have written the software instead of tacking
• 58:39
  on stuff onto an existing piece of software.
• 58:42
  Like for example, when you're running a sage raker job, should
• 58:48
  you be running a pod to start a sage maker shop?
• 58:51
  I don't think. So that's like, if you have run
• 58:55
  this at scale, Too many parts is not great for collaborating with
• 59:00
  that and I'd like a very very high especially fast churn rate
• 59:03
  and that part is not doing anything.
• 59:04
  It's expensive. What is that part is now?
• 59:07
  Can you put that part on a spot machine?
• 59:08
  You cannot because you're tracking something else.
• 59:10
  So all this is just not the right way of building it, we've
• 59:14
  built. So if you run a sage make a job
• 59:17
  within fight. For example, it doesn't come
• 59:19
  apart, it runs directly from the in July.
• 59:21
  If you run a snowflake query, which may run Forte is or hours
• 59:25
  or minutes or whatever. If you Bigquery, if you don't
• 59:28
  know, I think I carry whatever. They all get run directly from
• 59:32
  the core engine, which means the extremely lightweight, you won't
• 59:38
  pay a single penny for running a snowflake to AC to should not be
• 59:44
  coming. And we know, you know,
• 59:49
  theoretically Point 2 CPUs is okay, but it doesn't it look
• 59:53
  like we we run 40 million containers, 200, Million
• 59:57
  containers, like within the flight, flight cluster itself.
• 1:00:01
  And at that scale, we know what, all things break all the end.
• 1:00:04
  And those we have like we've learned from them and codify
• 1:00:07
  those principles into the system.
• 1:00:10
  And so you, we know it scales, we know it reliably scales to a
• 1:00:15
  very large number and you won't have to redo, but it comes at
• 1:00:20
  some penalty and we constantly keep on improving that.
• 1:00:22
  So we've added concepts of back and plug in.
• 1:00:24
  And what that also does is improve the That experience.
• 1:00:28
  So when you're running a typical job and I would highly recommend
• 1:00:31
  right look at kubernetes, spark on, flights Park on flight and
• 1:00:36
  set my car on flight or a and fly.
• 1:00:38
  It doesn't look like you're like You doing there for like things.
• 1:00:42
  Very much like something else. It looks like you're just
• 1:00:45
  writing your code which runs locally.
• 1:00:47
  If you don't have a it runs on one machine runs locally and now
• 1:00:51
  you say, oh I want for workers. It will get the for workers and
• 1:00:54
  create that, you know, the MPI connection between them so that
• 1:00:58
  they can all work together and that's how it should be.
• 1:01:00
  Should be Complexity as you needed.
• 1:01:04
  And for example, spark to should be the same way, I think.
• 1:01:07
  I know so many people use fire weather, you don't have the
• 1:01:10
  right data sets. We use Park, just for fun, we
• 1:01:14
  have 10 gigabytes of data sets and using Spike, and I'm like,
• 1:01:17
  don't use it, it's too much complexity, but make your system
• 1:01:20
  so that you can skate to it. So write it on one.
• 1:01:23
  Just one node runs park on the one node local modem, and if you
• 1:01:27
  want to scale, add one thing and that becomes hundreds of notes,
• 1:01:31
  and it has worked Yes. How you been finding your
• 1:01:37
  current work going going. So low.
• 1:01:39
  Compared to the days back back, with, in bigger companies.
• 1:01:43
  Oh I'm just dropping sets now. Huh?
• 1:01:49
  Yeah like constantly focused on one problem sometimes you get
• 1:01:54
  into your Echo chamber chamber and I try not to do that.
• 1:01:57
  I try to be realistic. I ask hard questions to the team
• 1:02:01
  but I am here is a lot of stress to be honest with the current
• 1:02:09
  economic situation and so on but it's it's fun it's like it's
• 1:02:15
  what I would do if given a choice again.
• 1:02:18
  If I also delicious knows I had a baby last year like it's 11
• 1:02:24
  months and I had a baby before the penalties are 33 year old
• 1:02:28
  and a 7 month old, and a company.
• 1:02:31
  So a lot of Interesting challenges over there, but I
• 1:02:36
  would still not make a different choice.
• 1:02:37
  I would this, I truly believe in this problem and I think whether
• 1:02:44
  it's me or not somebody's going to solve this problem.
• 1:02:47
  Why don't we take a crack at it? Mmm, actually, it's very
• 1:02:52
  noticeable. Even when such an open-source
• 1:02:54
  package, doesn't come to dominate or maybe it does often
• 1:02:57
  other packages, learn, from the API decision choices that they
• 1:03:00
  made in very, very constructive ways.
• 1:03:01
  You see that various orchestration engines have sort
• 1:03:04
  of consolidation roughly the same design behind the scenes
• 1:03:07
  about the completely different, but they all have the same kind
• 1:03:10
  of style of abstraction, that makes it very easy to move from
• 1:03:12
  one to the other. It feels like the whole V
• 1:03:14
  ecosystem, grows together and learn from the mistakes and
• 1:03:16
  successes of the other packages. So, you could elderly, want me
• 1:03:19
  or not. Yeah, we learn from airflow to
• 1:03:22
  be honest. And I said this in the past to,
• 1:03:24
  I think airflow got one thing right, which is to connect to
• 1:03:29
  other pieces. That goes, operators idea that,
• 1:03:32
  that idea is key is crucial. So it's not a workflow engine.
• 1:03:37
  So some people have a little work to ensure.
• 1:03:38
  No, it's not, that's not the goal of it, the goal of it is to
• 1:03:42
  let you orchestrate different things now.
• 1:03:44
  Yes, you can do it through workflows, that's one thing, but
• 1:03:47
  if you the typical workflow here, which is off their shelf
• 1:03:49
  like now, Maybe a couple other. You have to write all of this on
• 1:03:52
  your own and the user experience for that is not going to be the
• 1:03:56
  same. It's gonna be like a large
• 1:03:57
  something. Then right now, the state that
• 1:03:59
  does something hasn't have a failure State, it's going to be
• 1:04:01
  bad. We want to take all the
• 1:04:03
  complexity of it. So yeah, we learned that from
• 1:04:06
  effort to be honest. And yeah, but we also feel some
• 1:04:11
  of our things have been fired. Like, I've told you this the
• 1:04:14
  past, we have had checked with the queue flow pipeline steam
• 1:04:16
  before they think they existed because B existed.
• 1:04:19
  Well before that, That we have. I think our API in Python is
• 1:04:24
  this pretty is kind of I don't want to say this but like a lot
• 1:04:29
  of people are getting inspired by that but we are the only
• 1:04:33
  probably within this space that allows multi-language and people
• 1:04:36
  have not seen what can be done with it but I am excited.
• 1:04:39
  Hopefully within this year you'll see something really
• 1:04:42
  really fun. Like you can write Java Scala
• 1:04:43
  Python and just kind of like work together.
• 1:04:47
  So dude, so awesome to hear all of this.
• 1:04:50
  I'm excited for what you're doing.
• 1:04:52
  I always have been a fan of flight, I also love that we did
• 1:04:57
  the hackathon. If anyone has not seen the
• 1:04:59
  results from the hackathon that we did with flight, you can
• 1:05:03
  check them out in the video, we will put in the description
• 1:05:07
  because there were some really cool projects that came through
• 1:05:10
  and we ended up flying the winners out to the Toronto.
• 1:05:14
  Machine learning ml Ops World Conference, that was fun.
• 1:05:19
  Got to meet Bunch of the participants.
• 1:05:22
  We got to do that. Kate on, I want to finish up
• 1:05:25
  with a little bit of a lightning round.
• 1:05:27
  If that's cool with you do that. Some people, it's a lightning
• 1:05:33
  round. That is not so lightning.
• 1:05:35
  So, you know, we like to joke. It's like, I ask you these deep
• 1:05:40
  questions than expect a three-word answer.
• 1:05:42
  All right, here we go. The last book you read.
• 1:05:46
  I'm reading. Zero to one.
• 1:05:49
  That's right, you're reading zero to one.
• 1:05:51
  You gotta watch out with quoting, Peter teal because he's
• 1:05:53
  gone off the rails but it's a. Oh, what's that one?
• 1:06:04
  It's so it's another controversial topic potentially
• 1:06:07
  but really interesting. It's like, you know, this guy
• 1:06:10
  who learned how to use old-school military.
• 1:06:16
  Like 200 year old 100 year old military technique.
• 1:06:19
  And apply them to companies to really like streamline
• 1:06:24
  operations and the company really achieve its goals.
• 1:06:28
  Right. Make the that's why the art of
• 1:06:30
  action. Pretty nice.
• 1:06:32
  I like it so far, so if we ask meals or anybody at flight,
• 1:06:39
  you're not a dictator. Are you?
• 1:06:42
  They would limit quite on the other side.
• 1:06:44
  Say like I am, I'm an operator. I still write code.
• 1:06:49
  I love writing code, sometimes ideally since the code because
• 1:06:53
  of the other thing but I am an operator.
• 1:06:55
  I think the best way to lead is lead with Thousand Cuts in the
• 1:07:00
  front. So, excellent.
• 1:07:03
  All right. Next question.
• 1:07:04
  I've got for you, what was the last bug that you smashed?
• 1:07:11
  Let me see. Oh, yes, last bug.
• 1:07:14
  I don't remember. I remember the security bug that
• 1:07:18
  like was two and a half months ago.
• 1:07:19
  Yeah, that needed the code. I was like, what?
• 1:07:25
  What a hole and it was amazing. It was like I was able to
• 1:07:29
  reproduce it and was able to get a secret out of system without
• 1:07:34
  really without being a big hacker or anything.
• 1:07:37
  So I was like, oh my God, this is dangerous.
• 1:07:39
  And then we were able to patch it within four hours.
• 1:07:41
  So so what was the big learning from that?
• 1:07:44
  Oh my God. Security bugs are the scariest.
• 1:07:47
  That's one and second. They use this everywhere.
• 1:07:51
  So, so be careful when you software open source software,
• 1:07:58
  that's one thing also open source, software needs to words,
• 1:08:04
  That have given Linux needs people to constantly fix, and
• 1:08:07
  really, you know, push that stuff through.
• 1:08:09
  And we are thankful to all the companies who contribute to
• 1:08:13
  meaningful open source and let companies be safe and secure.
• 1:08:18
  That is so true. So what's a piece of software
• 1:08:21
  that you are? Bullish on.
• 1:08:23
  That might surprise people. Mmm.
• 1:08:30
  I webassembly. Oh yeah, yeah, I think I am
• 1:08:37
  enthused by it in probably my team will tell you I think there
• 1:08:41
  are ways we could do magic in the next couple years.
• 1:08:46
  I've been diving deep into it interesting but it's really even
• 1:08:51
  within the ML and data work. It's gonna it's going to terms
• 1:08:54
  like what we do. Interesting.
• 1:08:56
  Oh I like that prophesizing. Next we'll have you on next
• 1:09:00
  time. We will have you just talk about
• 1:09:03
  that for the whole hour? Yeah, when it fails, probably
• 1:09:06
  that. I look like, I feel like how
• 1:09:09
  wrong I was but but like, you know, I thought you said you've
• 1:09:12
  seen the python on in the browser, right by the, what is
• 1:09:17
  it? Patrick lightning or nope.
• 1:09:18
  I like an accordion. Yeah, that's it by step.
• 1:09:21
  And that's true. Yeah, yeah, yeah, yeah.
• 1:09:28
  Some of the core. So, okay.
• 1:09:32
  It's fine. It's like it's really.
• 1:09:35
  The properties are really amazing.
• 1:09:38
  Last one for you man. How do you want to be
• 1:09:41
  remembered? I wanted to be remembered by my
• 1:09:49
  my kids for us that I love loving father, right?
• 1:09:52
  Like that's number one, my wife and then in the world as an
• 1:10:01
  engineer who was not afraid to try something very, you know,
• 1:10:05
  the first principle, but not just going with the stream, kind
• 1:10:09
  of finding my own back. That would be, that would be
• 1:10:15
  wonderful. So, and yet having the right
• 1:10:17
  path in the end would be amazing.
• 1:10:20
  Kate on the Rev? Oh, I like it, man.
• 1:10:24
  I appreciate you, dude. I appreciate you coming on here.
• 1:10:26
  This is awesome. I love talking to you and I look
• 1:10:29
  forward to our webassembly chat that we're going to have, and
• 1:10:33
  absolutely. Yeah, this is the girl.
• 1:10:38
  That's going to be fun. So hopefully before that, we can
• 1:10:41
  have some other chats we can dive deeper into some other fun
• 1:10:44
  stuff we are doing for sure. Dude, so everyone if you've
• 1:10:49
  stayed with us this long and you have not checked out Union yet,
• 1:10:52
  I'm going to put on Kate on sales hat for a moment and tell
• 1:10:55
  you to go check it out. We'll leave the description in
• 1:10:58
  the link below. We appreciate everything that
• 1:11:00
  you're doing for the ml Ops Community.
• 1:11:02
  Kate's on that is another thing that I will say thank you and
• 1:11:05
  the union team. It's awesome.
• 1:11:07
  Some to have you all as sponsors and it's great to see the work
• 1:11:10
  you're doing. So that's it.
• 1:11:13
  George, you got any last words, man?
• 1:11:14
  Just like so much for the chance to us.
• 1:11:16
  Awesome foot program questions. You've also the move very well.
• 1:11:19
  I have more. I'm going to check it out more
• 1:11:20
  thoroughly and it might be might be worth the adoption cost.
• 1:11:24
  Ya go. Thank you.
• 1:11:26
  All right, there we go. Sweet.
• 1:11:30
  See you all later.