Latest / Agentic Conversations (formally mlops.community) / MLOps Coffee Sessions #14 Conversation with the Creators of Dask // Hugo Bowne-Anderson and Matthew Rocklin

Transcript

MLOps Coffee Sessions #14 Conversation with the Creators of Dask // Hugo Bowne-Anderson and Matthew Rocklin · Agentic Conversations (formally mlops.community)

• 0:16
  Well, welcome everyone for another episode of coffee
• 0:19
  sessions. I am one of the hosts David
• 0:21
  Aponte and I'm here with my co-host Demetrius Brinkman to
• 0:24
  meet you and say what's happened and everybody I'm talking nice
• 0:27
  and soft because it is late at night where I I'm at right now
• 0:31
  but I could not miss this chance to talk with our two wonderful
• 0:35
  guests. Yes, we have two very special
• 0:39
  guests with us from coiled and my mother.
• 0:41
  They might reduce themselves, tell them a little bit about who
• 0:43
  they are. So we have with us.
• 0:44
  Hugo bound Anderson and we have Matthew Rockland, Matthew, Rock,
• 0:48
  and can we start with you? Tell us a little bit about
• 0:50
  yourself. Yeah, sure.
• 0:53
  Thanks for having me, it's robbing us.
• 0:54
  So, yeah, my name is Matt. I've been a software engineer in
• 0:58
  the open source python space for a long time.
• 1:00
  So there is like, numpy and pandas, cycler, and Jupiter, and
• 1:02
  that space. I mostly think about scalable
• 1:05
  and parallel Computing. So, how do we Scale out that
• 1:08
  existing ecosystem the projects and I do most of the work in a
• 1:12
  project called desk or have been one of the lead maintainers the
• 1:14
  last five or six years ever since I got started and then
• 1:19
  more recently human eye and a few other folks start company
• 1:22
  coiled which provides you know hosted Solutions around dusk.
• 1:27
  That's a guess what we're talking about today.
• 1:30
  Excellent in hyuga by yourself. Tell us a little bit about
• 1:32
  yourself. I will and I just love to thank
• 1:34
  you both and thank the entire Community for having having us
• 1:37
  here today. And I love that.
• 1:40
  The matrices is slut slightly Whispering.
• 1:43
  There's a certain hypnosis to use his tones that I enjoy a lot
• 1:47
  more problem that you were moving voice here.
• 1:49
  Exactly. Exactly what a lot of dynamic
• 1:52
  duo. I.
• 1:56
  Yeah, so as Matt said, we found a coiled earlier this year,
• 2:00
  where we're building a scalable products for data scientists,
• 2:05
  and people doing machine learning, essentially hosted
• 2:09
  deployments and that these type of things to make it easier to
• 2:11
  get all of your data signs up and running in clusters on the,
• 2:15
  on the cloud and on-premise and all of these types of things,
• 2:18
  one of the many reasons I am attracted to the work.
• 2:22
  We're doing is because really interesting a lot longer The
• 2:25
  barrier to entry to data science and analytics and machine
• 2:29
  learning my previous Incarnation.
• 2:31
  I was doing this in the education, space at a startup
• 2:34
  called could data Camp prior to that.
• 2:36
  I worked in academic, research and cell biology and and math
• 2:39
  and biophysics. And I saw the, I think a state
• 2:45
  of data tooling for basic research, which required a lot
• 2:50
  of work to lower the barycenter. I think all the products and the
• 2:53
  tools are being developed very well.
• 2:54
  But it was it that it was very difficult to access them.
• 2:57
  And we've seen a lot of work happening there in the past five
• 2:59
  years since I, since I came to the dark side, as I say, of
• 3:03
  Industry away from basic basic research.
• 3:07
  So, that's pretty much an intro to me.
• 3:09
  If I do, find it interesting that we're speaking to the with
• 3:12
  the ml Ops Community today because one of the things we're
• 3:16
  working on is taking a lot of the devops out of out of data
• 3:19
  science. And what I mean by that
• 3:22
  essentially is to be Practical data scientist these days.
• 3:26
  There are so many moving Parts you need to do from all the
• 3:29
  software engineering stuff to the to the devops.
• 3:32
  And if you want to do that as a working data scientist, more
• 3:34
  power to you, but I think a lot of data scientists want to get
• 3:37
  their analyses up and running and don't necessarily want to
• 3:41
  get too involved with the kubernetes and Docker
• 3:44
  containerization, and these types.
• 3:46
  So that's kind of a brief idea of what with thinking about.
• 3:49
  I'm excited to chat about that more with you all.
• 3:51
  Yeah that's awesome. You guys have quite the
• 3:53
  backgrounds. So It's just to hear how that
• 3:56
  led you to your passions, I guess dealing with distributed
• 4:00
  computing and making things easier for data science has to
• 4:03
  do work. Where did that inspiration come
• 4:05
  from? I would love to hear from each
• 4:06
  of you. Just some of the you know the
• 4:08
  history the background of how you got to where you are.
• 4:10
  I would love to answer that by deferring, come out.
• 4:16
  Yeah, that's a good question. There's like a history here in
• 4:19
  like the python lure, right? So walk back.
• 4:22
  Five years ago, python was used. Everywhere for data science,
• 4:26
  machine learning sitting on pipe and has scikit-learn tensorflow,
• 4:29
  pie, torch were on the scene, maybe Fiona was there early on
• 4:34
  and all of them was great on a single machine.
• 4:35
  But just like, did not scalable for machines or even to like
• 4:38
  multiple cores are out of memory data.
• 4:40
  Since I was working at Anaconda at the time, an anaconda and how
• 4:44
  does like a for-profit python company and I kind of said,
• 4:47
  like, this is a structural challenge to the ecosystem.
• 4:50
  That's, let's fix that in the same way to kind of fix
• 4:53
  packaging with conda at least. No packaging.
• 4:55
  The care about libraries aren't just my phone.
• 4:57
  That's right. So we built that.
• 5:00
  Ask and ask was actually originally designed to be like a
• 5:03
  parallel numpy. We thought, look let's just
• 5:05
  paralyzed numpy, the whole other stack will, come on, come on top
• 5:09
  of that. That wasn't the case.
• 5:10
  You'd like, do lots of Social Work, lots of cultural work and
• 5:14
  we end up sort of massively contorting to ask to be this
• 5:17
  thing that could be snuck into all these other libraries.
• 5:20
  So, my ordinance or is actually much more about talking to The
• 5:25
  software Engineers, who wanted to support other user
• 5:27
  communities, but after seeing that leverage was actually
• 5:31
  really exciting and really satisfying to affect, you know,
• 5:34
  whole domains of Science and old domains of Industry through all
• 5:39
  of these other software libraries.
• 5:41
  So, you know, we helped numpy paralyzed itself and I'll
• 5:45
  certainly lots of, you know, cancer researchers or
• 5:47
  genomicists or, you know, climate scientists are solving
• 5:50
  their problems that scale and now that kind of impact has just
• 5:53
  like really fuel tank. a lot of us in the sort of desk and Pi
• 5:57
  data development communities, That's awesome.
• 6:01
  It sounds like you I mean both Hugo as well.
• 6:04
  You mentioned that you studied I guess in The Sciences as well.
• 6:06
  So you have, you know, scientific Computing was
• 6:09
  essentially what you guys are working on, right?
• 6:11
  So you know, processing a lot of biological data and maybe some
• 6:16
  of that it requires, you know, some of those, I mean we're kind
• 6:18
  of get into some of the challenges of why it's hard to
• 6:21
  do distributed computing. And in particular why python it
• 6:24
  can sometimes be just such a pain in the ass with a with
• 6:27
  paralyzing things. But yeah, you're right it.
• 6:29
  Maybe it's because Maybe even easier.
• 6:31
  Now, I would plan has always been the easiest language, but
• 6:34
  that's again for all the conversation.
• 6:35
  Maybe. Yeah.
• 6:36
  I would love to hear more about that but yeah.
• 6:38
  Sorry. Hugo.
• 6:38
  How about yourself? Tell us a little bit about how
• 6:40
  you got interested in the space and I know you've been in the
• 6:42
  education seen for a while and I actually that's how I heard
• 6:45
  about you when I was in grad school and I was in grad school,
• 6:48
  I was you know, utilizing data came to really get me some
• 6:51
  hands-on experience with things and I got some really great
• 6:53
  projects glad to meet you. I've got to say that he wrote
• 6:57
  before we got on to this. Podcast.
• 7:02
  David, every time we would say, yeah, we're going to be with you
• 7:05
  go. They would give his best Aussie
• 7:07
  accent. Here we go, we gotta hear it.
• 7:09
  Then it's Hugo bone Anderson. I can't do it a lot as you, you
• 7:16
  know, about the curse is yours is so rich when I love you and
• 7:19
  that's, yeah. And it's me, I mean, I there's
• 7:23
  the old joke, right? That Charlie Chaplin later in
• 7:25
  life, and to the Charlie Chaplin look-alike contest and kind
• 7:30
  There will be a day when I am not no longer the best me
• 7:34
  impersonator. But you mentioned something
• 7:36
  really, really important about. I think access to tools and
• 7:42
  skills and my background in. You're right, I did work.
• 7:45
  In basic research, I also worked with a lot of biologists.
• 7:47
  I worked in a research institute with hundreds of research
• 7:50
  biologist who were generating a lot of data and they didn't have
• 7:54
  the tools or the skills to actually analyze them.
• 7:58
  And at that point, I realized that addition to basic research,
• 8:01
  I could play at least a small role in enabling researchers to
• 8:06
  have access to the skills and tooling that will allow them to
• 8:10
  do, do their work and feel nourished in in doing their
• 8:14
  work. And one side of this is
• 8:15
  education as it's become really apparent particularly with the
• 8:19
  large amounts of data being generated.
• 8:21
  Now, after the height of big data has died down and the size
• 8:24
  of models that we need to build in some places these days and to
• 8:29
  be clear small data, and Models is, like, I always encourage
• 8:32
  people to figure out how to optimize their pandas code
• 8:35
  before going going and using using Dallas cried and do you
• 8:39
  need, you know, plot your learning curves, do you need all
• 8:42
  all this data? Because the answer may be no but
• 8:43
  if you do, there has been a serious lack of Enterprise
• 8:48
  tooling to do. So I think a lot of the OSS
• 8:51
  stuff plays an incredible role for individual contributors, but
• 8:56
  for organizations at large, who are adopting Open source
• 9:00
  software at scale. Now we need and assure a really
• 9:04
  fat long tail of of companies to support these Endeavors.
• 9:08
  And that's one of the several origin stories of coiled as
• 9:11
  well. You know, Matt Matt half jokes
• 9:13
  that he didn't necessarily want to start a company but there
• 9:17
  were so many requests and such a need that it made sense for us
• 9:20
  to do this at this point in time I think.
• 9:24
  Yeah, it's is working in a drug Discovery company.
• 9:26
  At the moment, I deal with Scientists all the time and it's
• 9:29
  an interesting dynamic. You know kind of mixing in
• 9:31
  science. And I want to see maybe
• 9:33
  engineering or technology because the rate at which they
• 9:35
  move you know, software Engineers are so focused on
• 9:37
  speed of deployment, right? Getting things out introduction.
• 9:40
  Quickly while researchers, you know, they wrote the road is a
• 9:43
  little bit all over the place but that's so important.
• 9:45
  Especially for machine learning and data science.
• 9:48
  It is largely experimental, you know.
• 9:49
  And and one of the challenges that we deal with, you know,
• 9:53
  that I am, we're often talking about is how to you know
• 9:56
  minimize the the errors that happen from taking these
• 9:59
  resource ID. He has and getting them into
• 10:01
  production and not only for production, but being able to
• 10:03
  utilize the right tools for the right jobs because a lot of the
• 10:06
  times you end up making your own ad hoc solution, which is what a
• 10:09
  lot of data scientist end up being right and of you know,
• 10:11
  doing whatever it takes to make something work.
• 10:14
  But one thing I want to, I wanted to highlight and it's
• 10:16
  something that I kind of implicit in some of the things
• 10:18
  you guys are saying. It sounds like there is a close
• 10:20
  relationship between best practices and tooling that
• 10:23
  somehow, the right tooling can enable actually some right
• 10:26
  practices and I wanted to ask you guys a little bit more about
• 10:29
  the what are you? Thoughts on just that that
• 10:31
  relationship? Do you think that that's there,
• 10:33
  or is that are they two separate things?
• 10:37
  Yeah, that's a good point. Maybe I'll agree with you in two
• 10:38
  different ways. Like I think as a software
• 10:40
  engineer first want to do something 20 times before.
• 10:43
  I realize, honorable the library around that right and so
• 10:46
  software to a certain extent in codes best practices s, what I
• 10:51
  would say is software, also allows you to communicate best
• 10:55
  practices, who have no idea about those practices, rest like
• 10:59
  the data side is the real time recording have no idea the best
• 11:01
  practices around security, right?
• 11:04
  That's not something that like they're going to have this
• 11:05
  Factor. Is around.
• 11:07
  We're going to add a bunch with some software to communicate,
• 11:09
  those practice to other people with software, or more likely
• 11:12
  with services with other systems.
• 11:15
  And that's really, you know, where companies like, welcome in
• 11:17
  the sort of software like best comes in, you know, people don't
• 11:20
  know how to do a distributed join on GPU data frames, right?
• 11:25
  It's hard. It was like, five different
• 11:28
  software packages you can sort of throw together to make that
• 11:30
  happen. Yeah.
• 11:34
  Well you know when the things that I was doing some research
• 11:36
  and reading about coiled and I took some things that I thought
• 11:39
  you guys really cared about like to highlight them.
• 11:41
  So one of them is data literacy. Let's talk about data literacy.
• 11:44
  Why is that important to you guys at coiled?
• 11:47
  Well, I think it's probably more important further up the funnel
• 11:53
  so to speak. I do think that the type of
• 11:55
  people who are using dark already are relatively data
• 11:58
  Savvy and and data literacy that data literate, pardon me, but I
• 12:02
  do think it's incredibly important to it almost in God's
• 12:05
  best practices around reasoning about data and understanding
• 12:09
  data. And I do think we also need
• 12:13
  literacy around distributed compute.
• 12:14
  And what I mean by that is even for Or it's on a need-to-know
• 12:18
  basis, but when you think about different stakeholders in an
• 12:21
  organization with respect to data and distributed compute,
• 12:24
  you have team leads, you have individual contributors.
• 12:26
  You have ICS, you have certain people at sea level and oppose
• 12:30
  what? I mean, by distributed computer,
• 12:31
  literacy or data literacy is they need, they know what they
• 12:34
  need to know to do their, do their jobs, with respect to
• 12:37
  these Technologies, and I think everyone needs to be becoming
• 12:41
  more data literate. And if you're in an organization
• 12:43
  that does distributed compute, you need to have a certain
• 12:46
  amount of That literacy in order to do your job and maybe it's
• 12:49
  minimal, or maybe it's getting getting more and more one other,
• 12:54
  and this is slightly tangential, but it's about building data
• 12:57
  products example, I think about, is it, let's say, we have our
• 13:02
  chief people officer or a head of HR who uses machine learning
• 13:06
  product to, in their hiring flow.
• 13:09
  Right? Let's say at the top of the
• 13:11
  funnel, right? Do they need to know about
• 13:14
  k-fold cross-validation? Probably not.
• 13:16
  But They do need to know about the difference between false
• 13:18
  positives and false negatives, as opposed to just pure
• 13:21
  accuracy, right? And confusion matrices, and
• 13:24
  understand the bias is implicit in this type of stuff.
• 13:27
  Now, in terms of building best practices into products or
• 13:30
  product. Like that could actually inform
• 13:32
  any user of this along, the lines of what's happening there.
• 13:36
  So that kind of speaks back to this idea of encoding best
• 13:39
  practices in software and products as well.
• 13:43
  Because the danger is in building products that can do
• 13:46
  like Data stuff at scale. We may be equipping people to be
• 13:49
  dangerous as well. So having best practices in
• 13:52
  education built into products in that respect, I think is is key
• 13:56
  that harks back to Matt's point of individual contributors who
• 13:58
  maybe don't understand like data scientist, who don't understand
• 14:02
  the ins and outs of security and authentication and data access
• 14:05
  and that type of stuff if you're going to build that into the
• 14:08
  product. So the IT people maybe have the
• 14:10
  access to that and understand what they need to do there.
• 14:12
  But abstract away all the details that the individual data
• 14:16
  sign. Just doesn't need to know.
• 14:17
  I think that's, that's best case scenario.
• 14:21
  Yeah, that's awesome. So there's this trade-off here
• 14:24
  right where you want to, you know, broaden access, you want
• 14:28
  more people to have access to, not only compute, but the right
• 14:30
  tooling to utilize that but then you also want them to know only
• 14:34
  what they need to know. Right.
• 14:35
  You know there's not you know, like a data scientist has too
• 14:38
  often, wear a lot of hats nowadays.
• 14:39
  You know there's been a lot of toxic had in the past around you
• 14:41
  know being a full stack data scientist.
• 14:44
  But sometimes it in my opinion, it's actually kind of
• 14:46
  productive, you know. Like I don't need to know, you
• 14:48
  know, a lot of these things and I could be spending.
• 14:51
  Time elsewhere. You know that opportunity costs
• 14:53
  of me working on making my job work in a kubernetes cluster
• 14:57
  versus just you know doing what I know how to do and I think
• 15:00
  that there's a balance there. How do you guys think about that
• 15:03
  balance between enabling data scientist to have a lot of
• 15:07
  power? Right?
• 15:07
  You know, powerful compute but also use that wisely because
• 15:10
  there's I'm sure this comes concerns from like an SRE about
• 15:13
  you know, resource utilization and underutilization over
• 15:16
  utilization. Things like that.
• 15:19
  Yeah, a lot of that. Is this about visibility right?
• 15:21
  Can we surface, what's going on to the user?
• 15:24
  So they understand that things are faster slow and why they're
• 15:26
  faster slow, can we surface things to ID?
• 15:29
  So they're aware of, you know, how utilizes their cluster
• 15:33
  should they be putting controls in place can be surface things
• 15:37
  to in person, whoever was paying for stuff.
• 15:39
  You know how much money did we spend parsing CSV?
• 15:42
  Files last month, right? How much would we save?
• 15:44
  It was their star k, right? And so I think, It's a coupling
• 15:49
  that power with visibility with observability.
• 15:51
  Might be the term that this community response to I think is
• 15:55
  critical and those are really fun to like, it's actually like
• 16:00
  tasks like famously has this dashboard which people love,
• 16:03
  it's very, very Visual and the secrets that dashboard was
• 16:06
  designed, not for users. It was designed for me as I was
• 16:09
  building, algorithms in Dusk and thinking about your sticks,
• 16:12
  like, I needed that to make things good, but now as a result
• 16:16
  like that, ability goes to everybody and that really, I
• 16:21
  think elevates the level of understanding of everybody
• 16:25
  that's touching that system Right?
• 16:27
  And so that's that's some that I think a tool can do tool can
• 16:29
  teach you as you use it. Yeah.
• 16:32
  And maybe and I don't know if this if you were thinking about
• 16:34
  that as well but you know when you're actually utilizing these
• 16:37
  things in production or whatever you're doing your work, it's
• 16:39
  important that the monitoring process also has just as much
• 16:42
  care, right? You're thinking about what's
• 16:44
  going on, not only usage. But, you know, we've talked
• 16:46
  about this before when you have a model in production and you
• 16:49
  know, you're monitoring its prediction values over time to
• 16:52
  see if drift occurs, or if your features are just going all over
• 16:55
  the place and That that to me is not just about finding bugs, but
• 16:59
  about, you know, again, increasing or reducing the bus
• 17:02
  factor, I guess, sharing knowledge.
• 17:04
  You know, making sure that more people are included in this
• 17:06
  process, is something we're going to talk about as well,
• 17:08
  where there is various stakeholders in a data science
• 17:11
  team, right? And there needs to be and maybe
• 17:14
  we can even just kind of get to that to think about tracking
• 17:22
  best practices, experimentation because historically like, Well,
• 17:27
  I usually machine learning. I'd like I'd be riding down,
• 17:30
  like by hand, like my scores and then like putting it in a Google
• 17:33
  sheet or something, something like that, you know, and seeing
• 17:37
  the rise of course, a lot of companies have built their own
• 17:39
  internal platforms. Now, to deal with this, this
• 17:42
  type of stuff, but we do see the rise of, you know, weights and
• 17:45
  biases and Neptune Ai, and these types of things to try and solve
• 17:48
  this for a broader broader broader class of people.
• 17:51
  I think that that type of thing. Such a burgeoning discipline is
• 17:55
  really interesting in key. Hey.
• 17:57
  Absolutely. Yeah I mean the this whole
• 17:59
  Community is really revolving around some of that because it
• 18:02
  involves a lot of key players. Are you have the data aspect
• 18:05
  which are crisis own set of practices the modeling which
• 18:08
  requires its own set of practices the infrastructure.
• 18:11
  And and then there's the monitoring aspect of test,
• 18:13
  there's a lot of moving Parts in this, you know.
• 18:15
  And and I do think that's why it's this is especially
• 18:19
  important for the community to think about this because I don't
• 18:22
  think one person or one company can necessarily have all the
• 18:25
  answers to that and it's great to hear how Different people are
• 18:28
  kind of dealing with the same sorts of challenges, maybe
• 18:31
  sometimes at a different scale but we're learning how to kind
• 18:34
  of what, you know, collate what actually works in these
• 18:37
  situations and communicate them and kudos to all of you for
• 18:40
  creating a community around it, because historically, a lot of
• 18:42
  this stuff is siloed in in large, powerful organizations,
• 18:46
  right? So actually having these
• 18:48
  discussions in an open space in, this is awesome.
• 18:51
  Yeah, that's what we're all about.
• 18:52
  Let's get into, you know, some of the things around you know
• 18:55
  the data science. What makes it so difficult one.
• 18:57
  We all know that with more data and more complexity, you know,
• 19:01
  data science at scale can be very challenging, right?
• 19:03
  You have lots of data requires lots of set up infrastructure
• 19:07
  and then even the expertise to maintain and make sure all that
• 19:10
  process is running. Why, you know, why is this
• 19:13
  something that's more prevalent? Now, I know that you were
• 19:15
  mentioning, you know, when you were doing some were kind of
• 19:18
  doing things at Hawk, it's kind of sloppy, it's messy, you're
• 19:21
  all over the place, but now it's becoming much more.
• 19:24
  You know, procedure. There's a lot more process which
• 19:27
  I think is Is actually good but it's also important to balance
• 19:30
  like the you know that that freedom to experiment and play
• 19:34
  Target. You want to say something.
• 19:37
  Yeah, I was just going to let Matt speak more to this with
• 19:41
  respect to what we're doing at coiled and what he thinks about
• 19:44
  it, a task but one of the reasons I just want to say it's
• 19:46
  slightly provocatively but one of the reasons I think data
• 19:49
  science is so difficult is because it isn't real.
• 19:51
  It doesn't data. Science isn't a single thing.
• 19:54
  It's an amorphous blob of like ad hoc techniques and tools and
• 20:00
  methodologies and questions that have kind of been thrown into
• 20:03
  this giant bucket so like it isn't a row.
• 20:06
  I mean Someone doing distributed machine learning like deployed
• 20:12
  doing like a real-time servicing recommendations or whatever it
• 20:15
  is is very different to someone doing analytics and building
• 20:17
  dashboards and that type of. Yeah, yeah yeah.
• 20:21
  So the other thing I think that's relevant here is that
• 20:25
  it's evolving so quickly. The tooling is evolving so
• 20:27
  quickly the value. It's delivering is evolving so
• 20:30
  quickly and the methodologies are so keeping keeping up with
• 20:34
  that is one of the biggest biggest.
• 20:37
  Challenges. And that's why we're really
• 20:39
  interested in building tools that are Nimble and agile enough
• 20:42
  and ergonomic enough to allow data scientists to do it.
• 20:45
  Use it from their own environment and use the entire
• 20:48
  open source ecosystem, that they're comfortable using.
• 20:52
  Yeah, simple jump in on that. So I think data science or
• 20:56
  machine learning. Generally, exploratory machine
• 20:59
  learning list those call. It is distinct from other kinds
• 21:01
  of computing because it is exploratory, right?
• 21:04
  People are trying they're doing something that has not been done
• 21:06
  before, right there, looking at data.
• 21:08
  They've never seen before, and they're using techniques, this
• 21:10
  will techniques computer techniques interpersonal, social
• 21:13
  techniques in order to explore something new.
• 21:17
  It's all about novelty there. In order to do that, they pull.
• 21:21
  Out all the stops right? Then I just use one tool.
• 21:24
  They're going to use, you know, their news, Jupiter for a bit
• 21:28
  and I use an Excel spreadsheet for a while.
• 21:29
  They might make some phone calls, right?
• 21:31
  They might use a GPU amazing CPU, they might use an fpga, you
• 21:35
  can use whatever they can grab at hand to solve their problem.
• 21:39
  And that I think is really what what typifies what exemplify is
• 21:43
  for the data science, exploratory, work flows, The
• 21:46
  challenge. I think that we see into the
• 21:48
  envelopes Community or like the scalable Computing Community is
• 21:51
  that when you're used to using every tool at your disposal from
• 21:54
  switching to excel to Jupiter, to a GPU to an fpga to a phone.
• 22:00
  Replicating that same experience on a cluster in the cloud or on
• 22:04
  some, you know, Cloudera machine.
• 22:05
  You got somewhere else is extremely hard, right?
• 22:09
  You're in for like a world of pain or world of Lowered
• 22:11
  Expectations when you realize, oh, it's not my laptop, right?
• 22:15
  I don't just have one hard drive as my software is now under the
• 22:18
  containers. What's the soccer thing or, you
• 22:20
  know, I can't just like, spin up a QT application to go explore
• 22:24
  the stuff, I've got to do everything through the web, wait
• 22:26
  a minute, how to web proxies work.
• 22:28
  And so it's really that experience.
• 22:29
  I'm trying to To recreate the sort of the laptop experience
• 22:35
  which I think is just is just painful in general.
• 22:38
  And yeah, it just thinking about the way I saw problems and how
• 22:42
  messy it is in convoluted. And oh gosh, yeah that you're
• 22:45
  right. That is part of the challenge.
• 22:46
  It's just like, you know, allowing their, you know, data
• 22:50
  scientists freedom to solve problems in whatever way they
• 22:52
  see fit. And you're right, sometimes it
• 22:54
  could be something super simple. And I learned that actually when
• 22:56
  I first was because a former to being a prior to being Machining
• 23:00
  engineer was working as a data scientist.
• 23:02
  And I was one of the, my takeaways from that experience,
• 23:05
  is that being a data scientist being a problem.
• 23:07
  Solver, that's essentially what you do your bit of a generalist
• 23:10
  and you kind of where a lot of those hats.
• 23:11
  And I like that, I like the complexity of that, but I can
• 23:14
  also see from an operations maybe point of view.
• 23:16
  RIT point of view, how challenging that could be
• 23:19
  because you want to, you want to extract as much value from that
• 23:22
  right for the business. But, you know, it's like you
• 23:26
  want to do that but you also want to let it be free and
• 23:28
  enable some of those things. Yeah.
• 23:30
  So, so let me dance, was really designed to, to pretend that
• 23:35
  your cluster of machines was like your laptop to a certain
• 23:38
  extent, we try as much as possible to make all of those
• 23:41
  sort of physical machines. You have look, kind of like your
• 23:44
  local python process and we tried very hard to hit python
• 23:47
  apis. Make everything look, really
• 23:48
  familiar and sensible. What we found though is that
• 23:51
  there's other set of problems, the sort of it set of problems.
• 23:54
  That's what we're kind of coil is trying to solve, but it's all
• 23:56
  about that sort of bringing the exploratory.
• 24:00
  Three local experience to larger and larger scale and trying to
• 24:04
  hide or pretend that these other challenges don't exist.
• 24:07
  When in fact, they very much do this right, finding the right
• 24:10
  set of abstractions. Well, one thing I'm super
• 24:13
  interested in is open source and I know that you guys are very
• 24:16
  passionate about open source. I know we have to community are
• 24:19
  as well and one question I have in because you guys are kind of
• 24:22
  working in the spaces, how can a for-profit company compliment?
• 24:26
  OSS open source software, There's like 20 answers to that
• 24:32
  question, but maybe I'll jump in first.
• 24:37
  So then if you look generally, You look historically, you know,
• 24:47
  maybe previous technology, Stacks, maybe like SAS an oracle
• 24:50
  large companies, bought a large technology stack and a bunch of
• 24:54
  support around that stack. And they, you know, we're happy
• 24:56
  for a while and then eventually, you know, the open source came
• 24:59
  in and just blew away the technology Stacks.
• 25:01
  You know, dude, came in Python came in come to says, this is
• 25:04
  great. This offer is way better, we
• 25:06
  don't the pay for it. Awesome but when they threw out
• 25:10
  those large companies, you know, SAS or Well, whatever, they also
• 25:14
  threw out, a lot of other things that are attached to the
• 25:17
  software, they threw out, you know, like a support contract,
• 25:21
  they threw out a system to distribute the software they
• 25:24
  throughout, you know, paying the developers who are actually
• 25:27
  fixing bugs or relationship with those developers who are fixing
• 25:30
  bugs. And I think what we're seeing is
• 25:32
  all of these sort of Open Source.
• 25:33
  Companies are fitting different parts of what was sort of Left
• 25:37
  Behind from these large sort of software monoliths.
• 25:41
  The providing support is very easy.
• 25:43
  The providing funded development.
• 25:45
  They're providing you managed services or hosted versions of
• 25:48
  software like what coiled knows with desk and that I think is a
• 25:51
  very, very clear benefit for everybody.
• 25:55
  Additionally, I think we have this sort of, we have these sort
• 25:58
  of two communities, these sort of business communities and we
• 26:02
  have these are open source development communities.
• 26:04
  They actually don't know how to talk to each other, or there's
• 26:06
  no way that like Ford or GM is an open up a contract with the
• 26:11
  pandas developers. I thought like, Not, they're not
• 26:13
  a legal entity that you can really engage with.
• 26:16
  So you need this sort of interstitial group that can that
• 26:19
  can talk to both groups. I think commercial open source
• 26:21
  companies can serve, that intermediary role is as much a
• 26:24
  business role as it is a social one.
• 26:28
  I'll build on what's what Matt has said by first, stating that
• 26:35
  the question you asked is one of the most important questions of
• 26:38
  now hands down, and to kind of signal that.
• 26:42
  I'll quote, mats heard me quote brand before, but Brian Granger
• 26:45
  said at the second, Jupiter Khan is a great talk.
• 26:47
  I think, I think it's online that we're and as a physicist,
• 26:51
  he of course, framed as a phase transition, he said we're going
• 26:53
  through a phase transition in which no longer previously.
• 26:58
  Institutions had individuals using open source but the phase
• 27:03
  transition is going to a point where organizations and
• 27:06
  Enterprise are adopting open source at scale, right?
• 27:10
  And you look, I mean, I mean, big examples, involve Walmart
• 27:13
  and Capital One and JPM, see several of which actually used
• 27:18
  us for what for what that's worth.
• 27:21
  But what I think, what that speaks to is the fact that we do
• 27:25
  need to figure out how Enterprises can adopt OSS as
• 27:29
  successfully as possible, that signaled a variety of challenges
• 27:34
  there. That for-profit companies can
• 27:36
  help to solve I also and I haven't, I don't, I haven't
• 27:39
  developed this also been interested in your thoughts on
• 27:41
  this, but I do get the sense besides a few edge cases.
• 27:45
  OSS tools are really good for ICS but they don't necessarily
• 27:50
  W. Yeah, individual contributors.
• 27:53
  Yeah. Sorry.
• 27:54
  So they don't necessarily solve all the large-scale
• 27:57
  institutional challenges so if you think of An organization
• 27:59
  like a network, right? It's sold.
• 28:01
  OSS is really good at solving for the nodes, but not
• 28:04
  necessarily for all the all the edges, right?
• 28:07
  And so if you want to solve for edges between it and data
• 28:10
  scientists management and data, scientists IT and management.
• 28:15
  You do need in this framework. At least one way to solve that
• 28:19
  is for-profit companies. Providing support training but
• 28:22
  also Building Products to solve for these types of challenges.
• 28:26
  I'm picturing the graph. I see it in my head right now.
• 28:30
  One thing the edges man, focus on the edges.
• 28:34
  One thing that I was just going to mention on this is, you know
• 28:37
  if we look at the last 10 years some of the biggest companies
• 28:42
  have been these open source companies besides what just
• 28:45
  happened recently with snowflake, right like you have
• 28:49
  some gigantic companies and I was just reading about article
• 28:54
  on medium out. Try and link to it in the show
• 28:57
  notes about basically, like how open source is eating the world.
• 29:00
  It's taking over in a way and really, I think that, that shows
• 29:06
  the viability of it, it shows that it is something that is
• 29:10
  people preferred in a way. And so, to make money off of it,
• 29:15
  I find really interesting with the companies, how they're
• 29:20
  deciding to create a business plan around it.
• 29:23
  And then, since it is open source, anyone, Come and create
• 29:26
  some kind of business plan around it.
• 29:28
  And I know I saw a funny quote somewhere on Twitter where it
• 29:33
  was like. Yeah, Amazon getting into your
• 29:37
  your like what was it Amazon getting into your business?
• 29:42
  No, ah man, I'll have to find the quote.
• 29:45
  It was like Amazon is stepping into your field of business or
• 29:51
  becoming a competitor. And then there is the part where
• 29:54
  it's like, Amazon, becoming a competitor with your own open
• 29:59
  source technology, right? Like that.
• 30:02
  Maybe brings up an interesting point.
• 30:03
  You're talking about, you know, open source software is eating
• 30:05
  the business world but also the business world is eating open
• 30:08
  source software at the same time, right?
• 30:10
  And like, you know, maybe numpy and tensorflow are different in
• 30:14
  some ways, right? Numpy is maintained by a
• 30:17
  community of volunteers who've done that for the last decade,
• 30:20
  many decades actually. And there, you know, academics,
• 30:23
  they've got, I've got an honesty, there's like a bunch of
• 30:25
  Transparency there. There's such a flow which is
• 30:28
  maintained by Google. They're school project managers.
• 30:30
  Super successful project has a lot of great good for Humanity.
• 30:35
  How do we as a community Now respond to that right is open
• 30:39
  source software and the future just going to be Google and
• 30:42
  Amazon fighting over, who gets to acquire, whatever small
• 30:45
  startup that was just looking to get acquired open source
• 30:48
  technology or is this community-based thing where
• 30:51
  everyone knows and trusts the software, right?
• 30:54
  There's no so much money. Around this Community Asset
• 30:57
  that. Now we need to be sort of aware
• 30:59
  of Open Source, maintainers need to be aware of the business
• 31:02
  challenges in the business, motivations behind some of these
• 31:06
  decisions. And if we want to sort of make
• 31:08
  the rules going forward, we have a relatively narrow window and
• 31:10
  weirder to make those rules. Now, something that we need to
• 31:13
  think about a little bit. If we care about the sort of
• 31:16
  community nature of Open Source, maybe we don't write maybe
• 31:19
  there's different different levels open source, you may care
• 31:21
  about some versus others. Well, I think we do and I think
• 31:24
  it comes down to some Sing segmentation, right?
• 31:26
  Because you're absolutely right. Tensorflow is quote unquote,
• 31:30
  open source, its governance model one that we think of and
• 31:35
  resonates work with us as people in the open source Community.
• 31:38
  Maybe not so much. So I think it's important to
• 31:41
  have these have these distinctions and be intentional
• 31:43
  about them. And one thing is somebody just
• 31:47
  brought up in in actually our slack was how in real open
• 31:56
  source project or really well run.
• 32:01
  One is we're not only one person makes money from that project,
• 32:07
  right, where it can create an ecosystem of different companies
• 32:11
  that are thriving from that. So that's another thing.
• 32:15
  If If it has the scope and it's big enough, and I instantly
• 32:19
  think of something like Cube flow.
• 32:21
  This is where, where we were talking about.
• 32:22
  It Cube flow is not only the maintainers that are arguably,
• 32:28
  not making any money, but then you get all these other people
• 32:32
  into the ecosystem and there's lots of companies that can be
• 32:36
  supportive of the technology and they are then there's companies
• 32:43
  created out of this. Yes.
• 32:46
  Really challenging. You know as I'm thinking about
• 32:48
  some of the, you know, the points you guys are making like,
• 32:50
  it's yeah I can't quite wrap my head around it and I, you know,
• 32:53
  I would be if I was a, you know, a business and I had to think
• 32:56
  about some of these things. It'd be hard.
• 32:58
  There's a lot of trade-off to consider what one question I
• 33:01
  have around. This is like how, you know, and
• 33:02
  this is maybe just more my curiosity, how does like, how do
• 33:06
  you encourage more people to contribute to something and to,
• 33:09
  you know, ensure quality. Because I know that that, that
• 33:13
  can be, you know, difficult when you're Working with a lot of
• 33:16
  different people. How does that?
• 33:18
  You know, I maybe maybe Matthew can speak a little bit more to
• 33:21
  that, weird ask, how does that work?
• 33:22
  You know, how do you make sure that things stay high quality
• 33:25
  that are working really well? But yeah, if you could, you have
• 33:29
  to be thoughts on that. I mean, one might also across
• 33:35
  the question. Like, how do you run a business
• 33:36
  successfully, right? There's like, a ton of answers
• 33:39
  to that question as a bunch of different philosophies around
• 33:41
  the question. I'm not sure there's like a
• 33:43
  single a single answer, right? I think we ran that asked very
• 33:47
  much like an interpersonal thing, so like we try to get
• 33:50
  companies engage, we try to get their maintainers trained up.
• 33:53
  We tried to find incentives for them to keep being around.
• 33:57
  We, you know, it's a community how do you build communities?
• 34:00
  There's a, you know, ask I think, you know, yeah, it's all
• 34:03
  right. Seat belts on this, too.
• 34:05
  It's not. There's no clear answer to this.
• 34:06
  That's a longer conversation, I think.
• 34:09
  Yeah, yeah, fair enough. I do that.
• 34:12
  There are a few things. I've seen, daus daus, do and
• 34:14
  scikit-learn. Do among other packages really
• 34:16
  successfully, in terms of at least getting developing you,
• 34:22
  the new contributor space and awareness among new users.
• 34:25
  That it is possible to contribute and not just be a
• 34:28
  user and that that comes down to documentation comes down to like
• 34:32
  inputs and Community buildings. Sprint's.
• 34:34
  I mean it's not not scalable but that's how these things, how
• 34:37
  these things start and then simple things like on an issue
• 34:42
  tracker having tag for you know this is a good issue for a
• 34:46
  first-time contributor to or you know, work on the documentation
• 34:51
  to become more familiar with how we deal with pull requests and,
• 34:54
  and that type of stuff. I think opening the space of
• 34:57
  contributions to a far wider. Wider audience is really helpful
• 35:01
  there. So I wanted to ask A fast
• 35:04
  question from Dan ger, like, who is a mutual friend of all of us.
• 35:10
  We all know him. He was wondering, like, do you
• 35:14
  see Collide operating within the desk ecosystem?
• 35:20
  For example, will you be developing proprietary
• 35:24
  extensions or open sourcing all of the core Tech?
• 35:29
  Yeah. So this is a question that we
• 35:32
  think about when making open source companies, what is, how
• 35:34
  do you monetize something? I think I'm most companies start
• 35:39
  like ours. We sell infrastructure that
• 35:41
  surrounds desk, right? So if you go to coil that I owe,
• 35:45
  you can sign up for cloud.google.com cloud and you
• 35:49
  can get running, you know, on a hosted system really easily as a
• 35:53
  bunch of features we could talk about there which are not desk,
• 35:56
  but their peripheral to desk, it's all the infrastructure that
• 35:58
  you need. In order to get up and running
• 36:00
  quickly. So that's, that's not
• 36:02
  competitive with desk, right? That's something that's sort of
• 36:04
  on the side and that's it, thanks for the open.
• 36:08
  It's like a nice model for open source companies, what happens
• 36:12
  sometimes. And we'll see if this happens
• 36:13
  with coiled is competitors show up.
• 36:16
  They offer the same thing when Amazon shows up and offers
• 36:19
  hosted. Ask how do we respond right?
• 36:22
  And so the the lever that we have at Amazon doesn't have is
• 36:24
  control over desk itself. And so you want to see, you
• 36:28
  know, how can you sort of you Your asset to be competitive
• 36:31
  without strangling, the community, and without holding
• 36:34
  things back. And that's like a very nuanced
• 36:38
  thing to try to do. And we've seen you a companies
• 36:40
  like Mongo debt mongodb do this, we've seen, you're confident do
• 36:43
  this. And that's, that's an open
• 36:45
  question that we're going to see what happens.
• 36:46
  But we are starting, I think most companies start very maybe
• 36:51
  perhaps naively but my even the altruistic we're not trying to
• 36:54
  sell that we're trying to sell support around dusk to make it
• 36:57
  much more successful. We believe that.
• 36:59
  His death is easier to use, you'll make money in the future
• 37:03
  and on top of that, I think that has the added benefit of feeding
• 37:06
  back into the dots Community. I mean the more tools and more
• 37:10
  products and services around that mag task easier to use the
• 37:13
  more people. Hopefully, that will be using
• 37:16
  dust because well, so we're also growing the entire dust
• 37:19
  community and family which is really exciting.
• 37:23
  This is a great time to start talking about desk itself.
• 37:26
  I know we've had previous episodes, so forgive us.
• 37:29
  If you guys have heard this already, you guys know desk, but
• 37:31
  I do know some people aren't. We're going to dive a little bit
• 37:34
  deeper into what desk is and what we're trying to do, what
• 37:38
  coiled is doing with that. So let's start with what is
• 37:40
  desk. Matthew, can you just kind of
• 37:42
  break it down for someone that never heard of it?
• 37:45
  Sure. Yeah.
• 37:46
  So Basques original goal was to paralyze.
• 37:51
  Other popular python libraries like numpy, pandas and
• 37:53
  scikit-learn. And provides a different
• 37:58
  libraries but which scales out to a single machine for large
• 38:02
  memory data sets or onto a cluster of machines.
• 38:05
  So if you want to run your pandas code on, you know, 20
• 38:07
  terabytes in the cloud tasks, like to do that.
• 38:11
  Internally, what we found. So, in order to build that
• 38:14
  system, we had to build a relatively sophisticated
• 38:16
  Dynamic, task scheduler. And I'm not sure how much in
• 38:19
  death we want to go. There was an engine inside of
• 38:21
  desk which runs parallel code and it turns out lots of users
• 38:26
  said, hey, the system that you built, the paralyzed pan is an
• 38:29
  Umpire looks great. Actually don't care.
• 38:30
  I just want the internal engine please because they're building
• 38:33
  their own, very different thing. And so now about a fair,
• 38:37
  majority of desk use is actually just using that core system.
• 38:41
  And so, that core engine of parallelism ends up powering,
• 38:45
  lots of other libraries actually boost prefect, blazing BB.
• 38:49
  There's lots of other systems that are built on top of desk,
• 38:51
  as well, as lots of you go to any hedge fund in New York City.
• 38:55
  They'll be like running some weird trading strategy on top of
• 38:58
  desk. So Das is also sort of a
• 39:00
  low-level general-purpose library for parallel Computing.
• 39:05
  It's all in Python is sort of all the perils and you need to
• 39:08
  build out cool stuff without any of the Means that maybe like, a
• 39:12
  Hadoop or mapreduce framework, or spark.
• 39:13
  Framework would come with so little bit lower level.
• 39:18
  Awesome. That's a great answer.
• 39:19
  And can we break down some of those Concepts in that for a
• 39:21
  piece of data scientists that are less familiar with
• 39:23
  distributed computing? Well, what is parallelism?
• 39:26
  And concurrency, first of what is, what is parallelism.
• 39:28
  How would you define it for someone that maybe has heard
• 39:31
  this term? As it's, you know, kind of been
• 39:33
  hearing it thrown around but what should they know about
• 39:36
  parallelism? There's a I recommend people
• 39:44
  take a look. If you go to YouTube and you
• 39:47
  look for Amish barn, raising it's actually a great example of
• 39:51
  parallelism so there's this time lapse time, lapse videos.
• 39:54
  Yeah. So people who are not in the
• 39:56
  United States the Amish are a community United States or very
• 39:59
  pastoral they sort of live. Like you'd expect people living
• 40:02
  in there with 1800 1700's. I think, once you get a whole
• 40:06
  Community people together and they build a barn in a day and
• 40:11
  if you look at these videos, you will see Hundreds of people
• 40:14
  working together towards one outcome.
• 40:17
  They're building a barn together.
• 40:19
  And it's actually really interesting to think about.
• 40:21
  If you had 100 people building something, how would you
• 40:26
  instruct them on how to construct that Barn as
• 40:30
  efficiently as possible. Now if you're one guy, you know
• 40:32
  great you're going to go build this poster about this poster
• 40:34
  with this post, you know, you know, put some paneling on it.
• 40:37
  You're going to build a roof and put things on - 100 people is
• 40:41
  like a scheduling problem. You got like a foreman, how are
• 40:43
  you going to direct everybody to do all these different things?
• 40:46
  That's parallelism right. You've got all of these
• 40:48
  resources, you've got this thing, you want to do.
• 40:52
  How are you going to plan things together?
• 40:54
  It's not quite like that of technical definition.
• 40:55
  I understand it. Some reaction.
• 40:57
  I really like that one, actually.
• 40:59
  Yeah, that you have to. Maybe share will share the link
• 41:01
  for that at that video. You mentioned a video or is
• 41:04
  there something in my eye? Look for Honor.
• 41:07
  You just go to YouTube and do a search for Amish barn.
• 41:10
  Raising time-lapse nice. You will not Of one also, I got
• 41:14
  a text. This is fun things there.
• 41:17
  Yeah, there's one thing. Is there to like what if a
• 41:18
  worker gets hurt, right, how do you replace it?
• 41:20
  Oh, that's resilience, right? How do you like, you know, do
• 41:24
  you all take a break for lunch? Like, oh, that's downtime,
• 41:27
  right? There's like there's lots of
• 41:27
  different things you could think about with that analogy.
• 41:29
  It's probably love that. I love their what.
• 41:31
  So, why is it so hard, though? You know, I think some people I
• 41:35
  know that when I was first learning about some of this
• 41:36
  stuff, I just the idea of just, you know, taking some big work
• 41:41
  into making it spreading into. It'll pieces of work.
• 41:43
  Okay, that makes sense. But then why is it so hard to
• 41:45
  put things back together? Why is it so hard to schedule
• 41:48
  these things? And in particular with python,
• 41:50
  why does python need a whole new tool to enable that or maybe
• 41:54
  some other tools as well? Yeah, I mean there's you can you
• 42:00
  can split the, the difficulty into two parts, maybe one is
• 42:03
  like constructing that plan, right?
• 42:05
  Great about 100. Carpenters, I've got to Mason's.
• 42:07
  I've got these three people who are sending food around.
• 42:10
  How do I use a people, effectively, to build this barn?
• 42:13
  And you can imagine building a schedule.
• 42:15
  We've got to go talk to your nearest foreman and they'll talk
• 42:17
  to you a lot about scheduling and a lot about, you know,
• 42:19
  making plans to build stuff and that's an interesting challenge.
• 42:24
  Sending everything out logistics Administration.
• 42:27
  That's a, that's a challenge. There's a second challenge of
• 42:31
  how do you actually interact with the hardware that you need
• 42:33
  to interact with? How do you interact with your
• 42:35
  network? How to interact with compression
• 42:37
  libraries, how to interact with communities, how to interact
• 42:39
  with all those sort of nuts and bolts of how our computers are
• 42:43
  hooked together today. And that's much more incidental,
• 42:47
  right? And that maybe like the less
• 42:48
  fun, but maybe more pragmatic, part of parallel computing.
• 42:54
  You could have done all of this before, right?
• 42:56
  So the previous systems to dusk so there was spark, no spark
• 43:00
  wasn't to be quite as flexible as you would want.
• 43:01
  It kind of built one thing if you want to build a barn maybe
• 43:04
  we're maybe going to build a house.
• 43:06
  Spark was great, fun to build a barn or a car or like a shed or
• 43:10
  like a rocket ship, or Rumba sparkly, but wasn't quite right.
• 43:15
  The other option was to go way lower level or something, maybe
• 43:19
  like, MPI or zero. And Hugh were to manage sockets
• 43:22
  yourself maybe with like threads and q's.
• 43:25
  If you're sort of like, been a computer scientist, you can have
• 43:28
  to think about different kinds of ways, of plugging, all these
• 43:30
  pieces together task is maybe like the Tinker Toys that like,
• 43:34
  have all the common patterns you're going to want, but don't
• 43:38
  force you to think about, you know what, what B should send
• 43:41
  across this wire in order to signal to this other computer.
• 43:44
  That is other computers. Down.
• 43:46
  And how do I make sure that? Like, if that wire goes down
• 43:48
  during that time we're going to send a recent, a message.
• 43:51
  It's a desk is handling. Its increasing level of
• 43:54
  abstraction, little bit to make it.
• 43:56
  I think easy for most sort of people who are proficient in
• 43:59
  Python, but it's not raising abstraction so high the Europe.
• 44:03
  It like the sequel database level where you can no longer
• 44:05
  really build the barn, you wanted to build it the way you
• 44:08
  wanted to build it. I don't know if anyone else
• 44:12
  caught that, but that said, if you want to build rocket ship,
• 44:16
  or you want to build a Roomba, but I actually usually use the
• 44:21
  term. Electronic cat talk about this
• 44:23
  stuff, but the python people do weird stuff.
• 44:27
  It turns out right by that you will do lots of weird things and
• 44:31
  yeah, the creativity knows no bounds.
• 44:35
  Love that. So A couple of things I don't
• 44:37
  want, I'm going to kind of move a move a little bit ahead but
• 44:40
  you mentioned before that desk has the ability to work really
• 44:43
  well. Sorry I'm just going to She's
• 44:50
  Ms. Yeah, so it allows you to work nicely with other API.
• 44:53
  So it integrates well with other tools that allowed to date
• 44:55
  scientist familiar with, but it also did that allows you to do
• 44:58
  some more complicated. Things like Dynamics, Dynamic
• 45:01
  task scheduling. Can we talk a little bit more
• 45:03
  about that and why that's why that's useful for data
• 45:06
  scientists? One second.
• 45:10
  There you go. I think you have to unmute.
• 45:16
  Oops. You're muted map. oh, Yeah,
• 45:30
  we're good. Sorry about that.
• 45:31
  Weirds or sell my apologies? Everyone no no Spence was great.
• 45:36
  People really want to know about Dynamic task scheduling them?
• 45:44
  Yeah. So again so dance to these high
• 45:46
  level API is no pipe as learn beneath.
• 45:49
  A this engine that engine is a dynamic distributed task
• 45:52
  scheduler. What that means is that you can
• 45:55
  tell that ask it, got this python function, it's my own
• 45:58
  function. Go run that somewhere for me.
• 46:00
  Please He's right actually got 100 of those so good.
• 46:02
  Run them was whenever you where we happen to have space and it's
• 46:06
  a task will run those python functions, could be any
• 46:08
  function. Could be a tensorflow train
• 46:09
  function, it could be something, he wrote and we're going to run
• 46:12
  it on the hardware. You give us Thing might add
• 46:16
  something more complex, you might add data dependencies, we
• 46:18
  may say okay we'll get after you finish loading up the CSV file
• 46:22
  one when you parse it and then when you train it then once all
• 46:27
  of those things are drinking want to try combining it with
• 46:29
  some sort of diagnostic later order.
• 46:33
  And so, you might have more complicated graph.
• 46:36
  They want to compute that has dependencies between those
• 46:39
  different tasks. And so now you start thinking
• 46:43
  about like, well, where's it at a living?
• 46:45
  How do I move it across a wire? How to make sure that I reduce
• 46:48
  that a movement? I want to keep track of where
• 46:49
  all those big pieces of that are some of the small pieces of data
• 46:51
  to them. What happens?
• 46:53
  If a worker goes down. So you get sort of some
• 46:55
  interesting complexities there. Then you may add another layer
• 46:58
  onto it. Wait a minute, like what if I've
• 47:01
  got a new file coming in every second?
• 47:04
  How do I then like update my graph on the fly in real time
• 47:08
  and keep pushing out some other results?
• 47:10
  Also in real time now you're going, In a real time to share
• 47:12
  an image to a scheduling and so that's what desk provides.
• 47:17
  So sounds like the problems that is trying to solve.
• 47:19
  We've kind of been getting at some of them as well.
• 47:20
  Not only increasing speed by paralyzing things, right?
• 47:23
  But also increased reliability, because of the fault tolerance,
• 47:27
  right? So, one worker goes down.
• 47:28
  Another one, could carry that load.
• 47:31
  And why is this in particular important for data science?
• 47:35
  I mean, it seems like, there's like how often will a data
• 47:39
  scientist run into the need to, you know, use these sort of
• 47:42
  tools to have, you know, things run faster to have that
• 47:45
  reliability do jobs. Spheal that much is that
• 47:47
  something that you guys see as a big problem?
• 47:51
  Yeah it's it's not often a problem.
• 47:55
  Unless you're operating on the cloud with cheap cheap
• 47:58
  instances, using a condom and pricing or Smart, Pricing our
• 48:01
  preemptible, nodes often. Those nodes can go away.
• 48:05
  And so that's something you care about, right?
• 48:06
  If you can support resilience, then you get to use cheap
• 48:09
  machines. You could just sort of fit in
• 48:13
  within the rest of the cluster, all the little holes.
• 48:17
  So, real quick, before we finish up, it sounds like you mentioned
• 48:21
  spark before but it sounds like desk is a little bit, not only
• 48:24
  more pythonic but more lightweight.
• 48:26
  Why is that? Why is that a plus?
• 48:28
  Do you think for from from your site Yeah, there's a bunch of
• 48:33
  reasons. One, it's like easy to pip
• 48:35
  install if you want to write, is included by default Anaconda.
• 48:39
  It's everywhere. It's like, these are easy to get
• 48:41
  up and running. We talk about clusters,
• 48:44
  actually, most asked users just use a laptops right there, just
• 48:47
  got by that. By default basket has been a
• 48:50
  thread pool. Two important tasks ends of the
• 48:52
  thread pool runs things on there.
• 48:54
  It's very accessible. I think individuals is a, is it
• 48:57
  really much easier place to start?
• 48:59
  You can grow up to running on the large wheels on a
• 49:01
  supercomputer Ders. Where you often?
• 49:03
  Just start with a little small engagement.
• 49:07
  I think what's maybe more important where we sort of more
• 49:10
  and more pressure to be lightweight was among different
• 49:13
  Library authors and so Das gets often integrated into other
• 49:18
  packages that and that they are usually very, very picky about
• 49:23
  how how lightweight tasks can be which is a really big piece.
• 49:26
  We have an idea. S in s, learn Facebook's profit
• 49:32
  Library, XG boosts. Prefect, there's lots of things
• 49:36
  included, a stud, or at the bottom, there's dozens of
• 49:39
  libraries that all and great with task.
• 49:42
  So this is maybe up to really big difference between spark and
• 49:44
  ask inspark. They're Reinventing a whole data
• 49:48
  science stack, right? They've got spark that are
• 49:51
  frames instead of pandas, they've got Emma live, instead
• 49:53
  of scikit-learn there have got, you know, some graphics
• 49:57
  Graphics. They've got all these different
• 49:58
  libraries. They're building on top of
• 50:01
  spark. The engine desk isn't an
• 50:03
  ecosystem task is like a little bit of parallelism that were
• 50:06
  injecting and to eliminate existing reason.
• 50:11
  So you know, that's just part of the broader python community of
• 50:16
  libraries. We're not trying to reinvent
• 50:18
  everything. We're very lightweight and urges
• 50:22
  a we can do that. And as a result of being
• 50:25
  lightweight, we all of that leverage of all those other
• 50:28
  people building, awesome, libraries, And as real, that's
• 50:32
  actually much more like a community solution, right?
• 50:35
  You don't use, just ask, you used ask plus, 50 of the
• 50:37
  libraries, which are maintained by thousands of people around
• 50:40
  the world, and we're not in one company were in thousands of
• 50:43
  companies, right? And that's, that's like a
• 50:45
  really, a cultural difference between tasks and most, other
• 50:47
  open-source sort of scalable Computing, things we grew out of
• 50:50
  the community, rather than trying to build a community
• 50:53
  around it. And I'll build on that by
• 50:56
  saying, it's important to think about the barrier, to entry for
• 50:58
  tools, like tasks and Spark and I'm not talking about You know,
• 51:02
  really technically Savvy, engineering, devops E-Type data
• 51:05
  scientist. I'm talking about the adoption
• 51:07
  of python has been immense over the past, at least, 55 years
• 51:12
  particular, the data science based.
• 51:13
  So there's a whole new breed of analysts and learning python who
• 51:17
  until recently went building their models in Excel, right?
• 51:20
  If they're, if they need to do some sort of distributed
• 51:23
  compute, we're doing potentially hundreds of thousands.
• 51:25
  If not millions, millions of people right in the coming
• 51:28
  decade, if they're going to do distributed compute that.
• 51:31
  I've learned python after going from Excel to the pythian.
• 51:35
  Do we then want to ask them to go to something, like, spark,
• 51:38
  where they get thrown jvm errors.
• 51:39
  When they do Imports or do we want to meet them where they
• 51:42
  are, and of course, park has its own Pros as well.
• 51:45
  Don't get me wrong, but do we want to meet them where they
• 51:47
  are, and have apis, as close to the tools?
• 51:50
  They're already using which The Dusk a data frame, for example,
• 51:53
  mimics and the dusk array, mimic some with respect to Patterson
• 51:57
  numpy respectively. And also the fact that it, you
• 52:00
  know, for the most Most part runs python code and the pandas
• 52:03
  and respective numpy code. In the back, end means the
• 52:07
  actual execution mirrors, the mental model of what they have
• 52:12
  of how the gods actually works. I think it does open the entire
• 52:15
  space to a whole new breed of users which I'm really excited
• 52:18
  about. As this has been a really
• 52:24
  awesome conversation. I just want to close with a
• 52:27
  question. What are you?
• 52:28
  Where do you guys see as the future of desk?
• 52:31
  Where do you think things are going to be heading?
• 52:33
  Do you? And I, you know, I really I
• 52:35
  don't want to take away from the point.
• 52:36
  You go about, you know, making things easier that is so true.
• 52:40
  But do you see more of that you what's in store and what can we
• 52:44
  be prepared for? Your music.
• 52:51
  Sorry, Matt, it's probably my fault suspense again.
• 52:54
  This is Ben's again we're gonna meet I'm not intentionally
• 52:59
  getting myself. I suspect when they're playing a
• 53:02
  prank. No no no, I'm eating it because
• 53:05
  of the echo. I'm just trying to like Olivia
• 53:07
  decker1 someone else. Oh my apologies, no worries
• 53:11
  about it. I know we're so good I can
• 53:12
  resolve that. Yeah I think task is good as
• 53:16
  expanding two directions I think.
• 53:19
  So, horizontally were received, lots of other domains starting
• 53:23
  to pick up desk, right? So that's been really a common
• 53:27
  in federal services. In government in geosciences,
• 53:31
  we're seeing a lot of growth and Life Sciences.
• 53:32
  We're seeing a lot of growth, and a lot of times there's an
• 53:34
  election single growth and other fields.
• 53:36
  And so, you'll see a lot of other libraries starting to add
• 53:38
  bath support in other domains of haven't culturally been in
• 53:41
  before, and that's super exciting.
• 53:44
  I think also what we're seeing we're seeing a growth of smaller
• 53:48
  and smaller groups who want to use, it should be Computing and
• 53:51
  so I think we're going to start seeing a lot more accessibility
• 53:53
  and should be Computing. And that's where really what
• 53:54
  we're focused on in coiled. It's making so that anyone from
• 53:57
  the largest companies to the smallest companies have access
• 54:01
  to scale without knowing all of those challenges that we've
• 54:05
  talked about during this during this session, right?
• 54:08
  And that's the thing we're that accessibility barriers going to
• 54:10
  lower a lot, really? Got them.
• 54:16
  Same here by you Hugo. Some final thoughts.
• 54:19
  I think so. I agree with everything Matt has
• 54:21
  said. I also I look forward to the day
• 54:23
  which could be five ten years away where task is taught at
• 54:25
  Junior High. I think they are very great.
• 54:29
  Yeah we are entering the space where the pythons apply data
• 54:31
  stack is going to be taught a junior high's like ideally
• 54:34
  sooner rather than later and if people want to think about
• 54:37
  distributed compute a foundational way to do that in
• 54:39
  the pi data. Ecosystem is dusk.
• 54:43
  So I'm really excited for that and we're doing what we can.
• 54:46
  Boiled to help with that mission as well.
• 54:50
  I do, I would love anyone in the community to let us know how
• 54:54
  they what they've used. Ask for we run live streams
• 54:57
  science Thursday. So if you have a called a story,
• 55:00
  please ping us to come on that but I definitely would have to
• 55:03
  find myself. If I didn't let you know that
• 55:05
  we've just launched our Cloud product and made an announcement
• 55:09
  with respect to funding. So please do check out cloud.com
• 55:12
  oil.i, oh, and will include some links in the show notes as well.
• 55:16
  We'd love your feedback there as well, everyone.
• 55:22
  Sweet Demetrius. When you close us out here,
• 55:25
  sweet, soft voiced. I just want to say that that
• 55:29
  barn-raising story. It stressed me, the fuck out,
• 55:33
  man, thinking about that. I'm just thinking about how I
• 55:39
  can barely coordinate with my wife to make dinner in the
• 55:42
  kitchen. How can a hundred people,
• 55:45
  coordinate raising a barn together?
• 55:48
  That is nuts? So that's Off topic.
• 55:52
  But this has been a very stimulating conversation.
• 55:55
  I appreciate this so much. I was this was like, you blew my
• 56:00
  mind on so many levels from the barn-raising to just the
• 56:03
  open-source chat and then yeah, teaching teenagers a little bit
• 56:09
  of python, why not? I like, but I like it all.
• 56:13
  So hopefully well we will be seeing each other again next
• 56:18
  week and we'll be talking more with you all.
• 56:20
  This has been an Incredible talk.
• 56:23
  Really thank you again. Thank you so much.
• 56:29
  Alright guys, have a great evening.
• 56:31
  Thank you for listening everyone for listening and another coffee
• 56:34
  session. It's great to have you guys.
• 56:37
  Bye. Bye.