Latest / Agentic Conversations (formally mlops.community) / Open Standards Make MLOps Easier and Silos Harder // Cody Peterson // #234

Transcript

Open Standards Make MLOps Easier and Silos Harder // Cody Peterson // #234 · Agentic Conversations (formally mlops.community)

• 0:00
  My name is Cody Peterson. I'm a Senior Technical Product
• 0:03
  Manager at Voltron Data and I generally do not drink coffee.
• 0:10
  Yeah, actually, they Coke is usually my go to caffeine.
• 0:15
  What is up, ML's community? Welcome back to another episode.
• 0:19
  I'm your host, Dimitrios, and today we're talking with Cody,
• 0:23
  who broke down the Ibis standards.
• 0:26
  He talked to me about why open standards are a good idea, what
• 0:30
  exactly Ibis is trying to do. We got down and dirty on the
• 0:34
  data layer, so if you're looking for one of those LM talks, this
• 0:38
  isn't the one for you. If you are little more data
• 0:42
  engineering inclined, this is right up your alley.
• 0:48
  Let's get into it. And for those who are just
• 0:53
  joining us for the first time, we would love it if you leave
• 0:59
  some feedback, drop a star on Spotify, subscribe on YouTube,
• 1:05
  or you know what else is amazing?
• 1:08
  If you share this with a friend, all right dude, so you started
• 1:19
  cutting your teeth at Azure ML. Can you explain to me what you
• 1:23
  were doing on that product? Yeah, sure.
• 1:26
  So I joined Microsoft and working on Azure ML out of
• 1:30
  college as a product manager. I worked there for just under
• 1:34
  four years and worked on a bunch of different teams.
• 1:37
  Started on the data team at some point, went over to the ML
• 1:40
  training team on the inferencing team and it was actually on a
• 1:43
  dedicated Milos team for about two years.
• 1:46
  O yeah, working with customers, helping them deploy their end to
• 1:51
  end machine learning systems on Azure Mill and really learned a
• 1:54
  tonne through the experience. Do you feel like the Milos or
• 1:59
  Mill life cycle? Let's just go with like the old
• 2:03
  school mill Predictive Mill as they call it, Is it
• 2:08
  standardised? Is there a standard way of doing
• 2:11
  that? Because someone literally just
• 2:14
  told me online the other day they were trying to pick a fight
• 2:17
  with me saying like all this mill OPS stuff, it's all
• 2:19
  standardised now and there's best practises for that and I
• 2:23
  had to kind of like ask. Is there though?
• 2:26
  I'm not sure. I don't think so.
• 2:30
  I think there are definitely some common threads, you know,
• 2:33
  version control your code. There's still ongoing debates
• 2:36
  about whether you should use notebooks and production, but
• 2:39
  it's such a broad discipline, just like software engineering
• 2:42
  that it's hard to say, it's hard to give a damn dogmatic response
• 2:46
  of you need to do this in this situation.
• 2:49
  I think the answer is often. It depends.
• 2:51
  You can have a lot of best practises for certain, like
• 2:54
  verticals. If you're doing recommender
• 2:56
  systems or I don't know, linear regression or whatever, There's
• 3:01
  a lot of existing content you can go pull from.
• 3:04
  But yeah, I think it's very hard to say that it's standard.
• 3:06
  It's going to depend a lot on what you're trying to do.
• 3:09
  That is a great point and I think doesn't get made enough.
• 3:13
  Depending on your use case, you have more history and more
• 3:21
  companies that have been able to leverage that type of machine
• 3:25
  learning. So you can see more content, you
• 3:28
  can see it feels more mature, like a recommender system is a
• 3:31
  perfect example of that. Are there things that you
• 3:34
  learned from your days at Azure Mill about Milos that you feel
• 3:40
  like you still carry with you today?
• 3:43
  A tonne of it. It's.
• 3:45
  It very much taught me about the role of data in kind of modern
• 3:50
  software systems and justice data systems.
• 3:53
  It taught me a lot. Like the data layer that you
• 3:57
  gotta have? And what exactly do you mean the
• 4:01
  role of data? Yeah.
• 4:03
  So you in these ML systems, you typically need to update them
• 4:06
  overtime you're getting new data in.
• 4:09
  How are you managing that data? How are you versioning things?
• 4:12
  If you're in certain industries, you have to be able to go
• 4:15
  through audits and say why did we deny someone's, you know,
• 4:20
  application for something And you need to be able to point
• 4:23
  back to the data and show that it was fair.
• 4:25
  Lots of lots of places where data is very important in these
• 4:28
  systems. It's all of that transforming
• 4:33
  the cleaning, the access, even. All of that is so messy and I
• 4:38
  think a lot of Times Now when people are dealing with LMS, all
• 4:42
  of that kind of gets abstracted away.
• 4:44
  Sometimes There still is the need for a data engineer and
• 4:49
  more and more as you want to go and do more advanced use cases,
• 4:54
  you recognise that what you thought you could get away with
• 4:58
  and not having to learn that data stuff, all that really
• 5:02
  hairy data problems, you can't get away with it as much as you
• 5:07
  thought. But in the traditional ML
• 5:09
  systems, like you said, it was so much at the forefront and it
• 5:12
  is so much at the forefront. Do you think that the LM world
• 5:18
  loses that, or makes it so that you don't have to think about
• 5:23
  that as much? I think it's still important,
• 5:26
  maybe as important and it's it's still going to depend what are
• 5:29
  you doing with your alms, what kind of data are you running
• 5:32
  them on. One of the big learnings I've
• 5:35
  had kind of towards the tail end of Azure ML and since is really
• 5:38
  the role that database technology has to play, it's
• 5:41
  kind of interesting seeing these vector databases.
• 5:43
  And you know the, the founder of Lance, TB gave a talk recently,
• 5:47
  I think a data council that was like we probably will see these
• 5:50
  things converge and databases are adding vector indices or
• 5:54
  vector search natively vector databases or adding more
• 5:57
  traditional database data types and things.
• 6:01
  And yeah, there's a lot I think the Milos creativity can learn
• 6:04
  from the database community and maybe vice versa.
• 6:07
  LMS and Genai is kind of going everywhere.
• 6:11
  Yeah. And so now you've been spending
• 6:13
  a lot of time on the ebase project, as I was calling it,
• 6:19
  and you mentioned that's not what it's called.
• 6:22
  Can you explain what exactly what exactly is Iris and what
• 6:27
  have you been doing with it? Yep.
• 6:29
  So these days I work as a product manager at Voltron Data,
• 6:32
  focused exclusively on the Ibis project named after the bird.
• 6:37
  Not the HH chain. Not the hotel chain.
• 6:40
  There's actually a good list somewhere, the lead maintainer
• 6:43
  put. Of all the other different Ibis
• 6:45
  things that are out there, the hotel is one of them.
• 6:48
  But yeah, the Ibis project was started long time ago, actually
• 6:51
  in 2015 by Wes McKinney, who was the creator of Pandas, cocreator
• 6:56
  of Apache Arrow, and also one of the cofounders of Voltron Data.
• 7:01
  And there's a famous or infamous article he wrote called, I think
• 7:05
  Apache Arrow and the 10 Things I hate about Pandas.
• 7:08
  And it kind of listed some of the some of the issues and
• 7:12
  longstanding gripes with pandas that he is kind of the creator
• 7:15
  had. And a lot of those have been
• 7:17
  solved through, you know, pandas is still very much alive and
• 7:21
  well. But Ibis kind of came along to
• 7:23
  take a different approach and say what if instead of tightly
• 7:26
  coupling our dataframe API to the execution engine.
• 7:30
  So pandas for instance is very tightly coupled to numpy and
• 7:34
  kind of its own way of doing single threaded in memory
• 7:37
  execution. What if instead we compile down
• 7:40
  to say SQL and sent that off to a database back end or some big
• 7:45
  distributed system and let that handle the execution?
• 7:48
  An Iris just handles the API, so it's the kind of origin of the
• 7:53
  project back then. Wait, so say that again because
• 7:57
  I'm not sure I've fully understood.
• 7:59
  So basically the versatility of Ibis is because it can just be
• 8:07
  boiled down to Sequel and then you can send it to wherever you
• 8:10
  need it to go. Yep, and not just SQL.
• 8:15
  So today Ibis project supports 20 at least 20 back ends.
• 8:20
  These are things like Duck DB and Oars and Snowflake and Big
• 8:24
  Query and click House and pyspark and dask and actually
• 8:28
  pandas itself as well. And the way it does that is by,
• 8:32
  you know, it's a data frame API. You write your data frame code,
• 8:36
  there's some sort of internal intermediary representation and
• 8:40
  then that gets converted into back end specific code.
• 8:43
  Over most of these systems it's actually just SQL.
• 8:46
  For pandas it's pandas code, for Polish it's polars code, and for
• 8:50
  Dask it's Andas Pullers and ask are the three dataframe ones,
• 8:53
  but the rest are all SQL back ends.
• 8:56
  It compiles your code to back end native code, sends it off,
• 8:59
  and lets the back end system deal with all the computation
• 9:03
  and data management and all that good stuff.
• 9:07
  O What was the life of the data scientists or analytics
• 9:14
  engineer? I guess is another person that
• 9:16
  would probably be playing in this before or not using Ebis
• 9:22
  and using Ebus. Yeah, so the thing I saw a lot
• 9:27
  going back to Azure ML was this problem of data scientists
• 9:30
  throwing their code over the wall to yeah, data engineers or
• 9:34
  mill engineers. So they might write some often
• 9:37
  pretty bad pandas code or inefficient pandas code, and
• 9:40
  they might be pulling say they're working with Snowflake.
• 9:43
  They might pull like a 1% sample down to their local computer,
• 9:48
  iterate on their pandas code, and then say, hey, this works
• 9:51
  great, go make it run efficiently on Snowflake.
• 9:54
  Then you get the memes worked on my machine.
• 9:57
  Yep, exactly. And with Ibis instead of doing
• 10:00
  that, you actually you go into Ibis, you connect to Snowflake
• 10:04
  for instance, and you just work with your data there with your
• 10:07
  API. You could still pull your data
• 10:10
  locally and work with it and dude or pullers if you wanted to
• 10:13
  do that to save on costs or whatever.
• 10:16
  But kind of regardless of which back end you use to scale it up,
• 10:19
  it's the same code. So instead of throwing that code
• 10:23
  over to the wall over the wall to the engineer, you just head
• 10:27
  it off and say here's my Ibis code.
• 10:29
  It already runs on Snowflake natively, so you can just go put
• 10:34
  this in production or make it more efficient if you need to.
• 10:37
  And so one thing that I wanted to talk about was this idea of
• 10:41
  like SQL versus data frames and how do you look at that?
• 10:45
  What are some insights that you've gained from almost like
• 10:49
  living in both worlds, I feel. Yeah, I definitely started in
• 10:54
  the completely Python data scientist data frame world.
• 10:58
  In Azure ML, there was pretty much never any SQL.
• 11:02
  My first interaction with SQL was actually trying to look at a
• 11:07
  database of metrics that we had and you know, trying to do like
• 11:10
  product analytics. And I got very frustrated pretty
• 11:13
  quickly with SQL. I did the, you know, select,
• 11:16
  star, download as CSV and I just use pandas to like look at the
• 11:20
  data instead. So that's kind of how I started.
• 11:23
  Yeah. I joined DBT Labs for a bit
• 11:25
  under a year and released all this other side of like people
• 11:28
  who would never think to use Python And data frames and
• 11:31
  really prefer writing SQL code in a query editor or in DBT of
• 11:36
  course. And it was very interesting.
• 11:38
  So there's definitely this two world problem in data of people
• 11:42
  working completely in SQL and people working completely in
• 11:46
  Python. And I think there's a lot that
• 11:48
  both sides can kind of learn from each other.
• 11:51
  There's a lot of overlap, of course, people use both.
• 11:53
  But yeah, and itis itself you know, one of our taglines is it
• 11:58
  provides the performance and scale of modern SQL, of the
• 12:02
  flexibility of Python. So really bringing those best
• 12:05
  practises from and performance from SQL databases to Python
• 12:09
  dataframes. Yeah, and so Avis is a
• 12:13
  completely open source project, right?
• 12:16
  And it allows you to almost abstract a layer above to make
• 12:23
  sure that the engine you can choose whatever engine you want
• 12:27
  off the back of it. And you have that flexibility to
• 12:32
  say what your system is going to look like, probably inheriting a
• 12:37
  lot of what your company already is using and what you're used to
• 12:40
  using as you mentioned. How have you seen Ibis evolve
• 12:45
  over these couple of years that you've been working on it?
• 12:48
  See, I've been on Voltron Voltron Data for one year and I
• 12:52
  became aware of Ibis about a year before that and kind of
• 12:55
  watched it evolve it well drawn data.
• 12:58
  It's despite being around for about 9 years, it was kind of
• 13:02
  contributed to on and off just as an open source project and
• 13:05
  really in the last 2 1/2 to three years under Voltron Data
• 13:09
  stewardship, it has really taken off and become very, very
• 13:12
  useful. So it added Duck DB as a default
• 13:15
  back end which just gave you a out-of-the-box really good local
• 13:18
  option. It's added all the things you
• 13:21
  would expect in a dataframe library for like reading CSV
• 13:24
  files, reading a per K We actually support Delta Lake now
• 13:28
  and it's just become really rock solid as well.
• 13:31
  The engineering team did a very large refactor for the recent
• 13:35
  nine dot O release that really stabilised a lot of the
• 13:38
  internals. Yeah, it's cool to see and as
• 13:40
  you mentioned it is a, it's a completely open source project.
• 13:44
  It has its own governance, it's not owned by Voltron Data or any
• 13:48
  one company, and that's really what we aim to be as an open
• 13:51
  standard That ideally instead of like creating your own dataframe
• 13:55
  library that only works on your engine, if you have a QL engine,
• 13:59
  you can just come and pretty easily adopt Ibis and make it
• 14:03
  work and give your users a delightful data frame API on top
• 14:07
  of your database. So there's one thing that you
• 14:11
  said before we hit record, which was open standards are a good
• 14:15
  idea. What about working on this
• 14:19
  project has made you see things that way?
• 14:23
  Yeah, they're a good idea really on both sides of the equation.
• 14:27
  So they're good for customers or users of these open standards.
• 14:32
  And the reason there is #1 efficiencies that you get.
• 14:35
  So in the past, if you wanted to go from, well, there weren't as
• 14:40
  many options. But if you wanted to go from
• 14:41
  like Pandas to Pyspark or various other platforms to each
• 14:45
  other, there was a pretty heavy cost in like moving that data
• 14:49
  and also learning two different data frame abstractions.
• 14:53
  Today though, with Apache Arrow for instance, that you can
• 14:57
  seamlessly convert between pandas and pullers and ducts B
• 15:00
  and Justice choose the interface that you prefer.
• 15:03
  An arrow. Can you break that down for me?
• 15:05
  Because I've heard it thrown around a lot and I want to know
• 15:09
  exactly what it does. Yeah, so patchy arrow.
• 15:12
  As I mentioned it was Co created by Wes McKinney and a bunch of
• 15:15
  other people, A lot of them maintainers work at Voltron Data
• 15:18
  as well. But again, it's a fully open
• 15:20
  source project under the Apache Foundation, has contributors
• 15:24
  from a bunch of different organisations and it's basically
• 15:27
  a spec for an in memory, in memory data format that's used
• 15:33
  by a lot of beta systems now. So Duck DB uses it as their data
• 15:37
  format, Ollers uses it as their data format.
• 15:41
  Pandas is increasingly taking a dependency on.
• 15:43
  It has some amount of dependency on it and basically what it
• 15:47
  allows you to do is if you're building a new data system
• 15:50
  instead of inventing your own in memory data format, you just use
• 15:54
  Apache Arrow and all of a sudden you can communicate with all
• 15:58
  these other data systems pretty seamlessly.
• 16:01
  So if you're going from duck TB to pullers, you don't need to
• 16:05
  like convert. You know how your float and ints
• 16:09
  and all the memory layouts are? They're just the same, You just
• 16:12
  go through Apache Arrow, but it's like a zero copy
• 16:15
  conversion. Very cool.
• 16:17
  OK. Sorry, I totally cut you off on
• 16:21
  the answer that you were giving. I sadly can't remember the
• 16:24
  question that I asked them when I cut you off because I got
• 16:27
  distracted on the Apache arrow side there was something along
• 16:30
  the lines of open standards being a good idea.
• 16:34
  Right. So they're they're good for
• 16:36
  users because they tend to provide more efficiencies.
• 16:39
  So rather than being locked into one specific vendor with their
• 16:44
  own memory format and their own data frame API, if we have an
• 16:48
  open standard that has buy in from a bunch of different
• 16:51
  vendors and different products out there, it allows you to
• 16:55
  easily kind of switch between them and choose the right tool
• 16:58
  for the job. And this kind of dovetails into
• 17:00
  the idea of a modern composable data system was McKinney gave a
• 17:05
  talk about that recently at the I believe Data Council, which
• 17:09
  was very good. So shout out to that yeah.
• 17:11
  The idea is like kind of going back to our Milos thing.
• 17:14
  It's you don't necessarily want to build a very rigid system for
• 17:21
  data or for mill. You want to be able to swap out
• 17:24
  components when the need arises. So if you want to switch from a
• 17:28
  batch processing or a batch updating ML model to a model
• 17:33
  that's updating in real time with like a streaming data
• 17:35
  system, you don't want to necessarily re architect
• 17:38
  everything. And open standards kind of bring
• 17:40
  us closer there, where you know you can just use the same APIs,
• 17:44
  use the same interfaces and data formats, and seamlessly switch
• 17:48
  between whichever product works best for your scenario.
• 17:53
  So I've been thinking about standards a bunch, especially
• 17:55
  because we're doing the AI Quality Conference and one thing
• 17:58
  that we plan to do at the conference is create standards
• 18:04
  for AI quality. And specifically like you have
• 18:09
  your industries and then you have your use case specific
• 18:12
  standards. Have you seen things that have
• 18:15
  worked as far as making sure that the standard you are
• 18:20
  creating doesn't just become one more standard that now a
• 18:26
  project? If it wants to plug into this
• 18:29
  little area over here on the map or these different tools, cool.
• 18:33
  It has to abide by this standard, like the Ibis
• 18:37
  standard, and then if it wants to go and play over here on this
• 18:40
  side of the map, it has to abide by a different standard.
• 18:44
  Yeah, it's a very difficult problem.
• 18:46
  People, when they hear about itis often bring up the the XKCD
• 18:51
  of you have 14 standards. Somebody says, let me introduce
• 18:54
  another one. We actually use that in most of
• 18:56
  our slides because yeah, it comes up all the time and I
• 19:00
  don't have a great answer. I would say that part of it is
• 19:02
  just making sure you get buy in from a diverse set of
• 19:06
  organisations. Like I don't think Apache Arrow
• 19:09
  would have worked if it were full drawn data didn't actually
• 19:12
  exist at the time. But just as an example, if it
• 19:14
  were only built drawn data pushing on Apache Arrow, you
• 19:18
  know and trying to get it adopted everywhere that would
• 19:20
  know if it worked. But it has buy in from a bunch
• 19:23
  of different companies and a bunch of different organisations
• 19:25
  and West and a bunch of other people were kind of facing the
• 19:29
  same problems at the time, got together and decided on this
• 19:32
  standard. And a lot of it's just yeah
• 19:34
  getting that buy in across the ecosystem and across
• 19:37
  organisations and popularity to some extent like even looking at
• 19:42
  Open A I I think open AI has almost created a standard in
• 19:46
  their REST API alms today and it's there's then like slight
• 19:50
  differences if you try to use Anthropic or Mistral or
• 19:53
  whatever. And yeah it's I don't know is
• 19:56
  someone going to come along and like create a standard API for
• 19:59
  these things. I know there's some open source
• 20:01
  projects trying to do that and it's just it's hard.
• 20:03
  You really have to get by in and just gain popularity over time I
• 20:06
  think, and eventually becomes a standard.
• 20:09
  Yeah, that is brutal because it is constant work getting that
• 20:18
  buy in, recruiting people to come on and you have to have
• 20:23
  that network effect. Otherwise it's just not going to
• 20:26
  work, is it? Yeah.
• 20:29
  And especially because like if we're talking about the Ibis
• 20:32
  project, if it only had out-of-the-box two or three
• 20:36
  different things that you could use with it, it wouldn't be as
• 20:41
  attractive as the 28 plus that you have now.
• 20:45
  Yep, definitely. And it then becomes partially A
• 20:48
  maintainability and scalability issue.
• 20:51
  You know, we we do have contributions from outside
• 20:54
  organisations. Alibaba's been contributing
• 20:57
  stuff. Oilers actually contributed
• 20:59
  their back end back in the day, single stories contributed a
• 21:02
  back end. So it's growing.
• 21:05
  But to to maintain that kind of momentum we need, we need a
• 21:09
  bunch of contributors from a bunch of different places.
• 21:11
  And you know, going from like 20 back ends to say 100 back ends
• 21:15
  is quite the challenge. Unless it really does become
• 21:19
  that well adopted open standard then that's where we're kind of
• 21:23
  trying to get to next just get buy in from different
• 21:25
  organisations, get them to kind of maintain their own back ends
• 21:29
  and Justice use Ibis as the front end for their data
• 21:32
  systems. Yeah, that could get so messy,
• 21:36
  yes. Especially if there's one or two
• 21:39
  people that are trying to maintain all of that.
• 21:41
  Yes. So we have a great group of what
• 21:43
  are we at like 6 or 7 full-time engineers working on a Voltron
• 21:47
  data bunch of people in the community as well.
• 21:49
  So it's fairly well funded, but yet to get to that next level we
• 21:53
  need a contributors welcome if anyone wants to get involved in
• 21:57
  open source. PR's open.
• 21:59
  Totally. So you you spoke about the front
• 22:01
  end and I think Voltron data is one of the back end, right?
• 22:06
  And we were having a nice conversation about this
• 22:08
  beforehand that Voltron Data has an interesting like open source
• 22:14
  play it. It's not like Open core what
• 22:17
  did, what did you call it? We've been calling it open
• 22:20
  periphery. I don't know if that's a term
• 22:22
  that we're going to try to really push, but yeah, it's
• 22:25
  Voltron. Data is partially able to be in
• 22:28
  such a unique position because its founders have such deep open
• 22:32
  source backgrounds. So I think I mentioned Wes
• 22:35
  McKinney was one of the cofounders and he was the CTO
• 22:38
  for a while, but Josh Patterson, CEO had worked on Rapids Eye and
• 22:43
  Cootie F and all that stuff which NVIDIA.
• 22:45
  So really pushing there open source software and most of the
• 22:48
  engineers are pretty deep in open source like the itis
• 22:52
  engineers had worked on Desk and Apache, Arrow of course and
• 22:55
  pandas and bunch of different projects.
• 22:58
  And so we have, we have a very heavy open source culture but
• 23:01
  part of what allows us to do this is kind of the business
• 23:04
  model. So we're selling, you know, we
• 23:07
  are a vendor as well. We are selling a distributed GPU
• 23:11
  database for you know, petabyte scale, very, very big data
• 23:15
  problems. And part of the problem and
• 23:19
  opportunity with that is it's not for most people, 99.9% of
• 23:26
  people are never going to need an engine that runs on, you know
• 23:30
  many different GPUs and does your analytics for you.
• 23:33
  And so the the founders and PCs as a product is faced with the
• 23:38
  problem of like, OK, how do we get an interface to this thing,
• 23:42
  but we don't want to just like invent our own data frame API
• 23:45
  that oh 1% of people are going to use.
• 23:49
  So instead we use Ibis. And if you're working with small
• 23:53
  local data, you can use ducts, B and pullers.
• 23:56
  If you're working with medium sized data, you can use Click,
• 23:59
  Callous and Snowflake and Pyspark.
• 24:02
  And if you do need Theseus, if you are in that kind of top tier
• 24:06
  of data needs, then you can use CCS as well.
• 24:09
  So yeah, this is the interface for Thesis, but it's also
• 24:13
  attempting to be an open standard and it's also why it's
• 24:16
  important that it is a self governed open source project.
• 24:19
  It's not owned by Voltron Data. It just kind of happens to be
• 24:23
  the interface for Thesis and happens to be where a lot of the
• 24:27
  Ibis engineers are employed. To thank our sponsors of this
• 24:31
  episode, Weights and Biases elevate your machine learning
• 24:35
  skills with Weights and biases. Free courses including the
• 24:39
  latest editions, Enterprise Model Management and LM
• 24:44
  Engineering Structured outputs. Choose your own adventure with
• 24:49
  wide offering of free courses ranging from traditional Milos
• 24:53
  LLM's and CICD to data management and cutting edge
• 24:58
  tools. That's not all you get to learn
• 25:01
  from industry experts like Jason Lewis, Jonathan Frank, Shreya
• 25:05
  Shankar and more. All those people I will 100%
• 25:11
  vowed for. They are incredible friends of
• 25:13
  the pod enrol. Now to build better models,
• 25:16
  better and faster and better and faster and just get your
• 25:21
  education game on. Check the link in the
• 25:23
  description to start your journey.
• 25:24
  Now let's get back into the show.
• 25:29
  Who are a lot of the people that you are seeing using Voltron
• 25:33
  data because as you mentioned that scale is gigantic and so
• 25:40
  I'm guessing it is for very big companies.
• 25:45
  It is. We do also sell basically
• 25:49
  enterprise support for open source and that was what we'll
• 25:52
  try and Data has had kind of before thesis launch.
• 25:55
  So if you need help with like Apache Arrow or Ibis or
• 25:58
  Substrate and that's something that we're on Data offers.
• 26:01
  And I don't know if we have, well one of the companies that
• 26:05
  we work with is Duck TB Labs actually.
• 26:07
  So we collaborate with them pretty closely like they're
• 26:10
  Apache Arrow usage and things for Ibis as well.
• 26:13
  As I mentioned, DB is the default back end.
• 26:16
  But then, yeah, for Theseus, we see very large customers with
• 26:22
  terabytes or petabytes of data and who also have a lot of GPUs
• 26:26
  that they're able to configure. It's you can't just like have a
• 26:31
  bunch of GPUs. You also have to have the
• 26:32
  networking set U, you have to have really fast storage.
• 26:35
  It's a very like almost a HPC style problem of how you
• 26:40
  configure all this stuff. And we also largely partner with
• 26:44
  companies. So I think the big public one
• 26:46
  right now is HP, HP E has their own kind of cloud product and
• 26:53
  you're able to embed Theseus in your own product and kind of
• 26:57
  sell it sharing with full drawn data effectively.
• 27:03
  Excellent O. When you look at how to choose a
• 27:07
  data system, what are some key considerations that you think
• 27:10
  about? Yeah, one of the first ones is
• 27:14
  whether you're doing batch or streaming.
• 27:16
  Streaming is kind of, I think it's been a bit of a joke that,
• 27:19
  you know, this year will be the year of streaming data, but it
• 27:23
  really is like kind of on the rise, slowly but surely.
• 27:27
  But yeah, what are your kind of latency requirements?
• 27:29
  Does the data need to be happening in real time or is
• 27:32
  once a day, once an hour, every 10 minutes, fine.
• 27:35
  Then I look at kind of data size.
• 27:38
  Are you dealing with something that you can work on on your
• 27:40
  laptop or a Modest Eyes VM? And if you're in that kind of
• 27:46
  almost less than one terabyte range, given how how big cloud
• 27:50
  VMS can get today, I would recommend something like Really
• 27:53
  Dude or Oars? Just a good single node OLAP
• 27:57
  engine. You definitely want to avoid
• 28:00
  doing anything with distributed systems unless you really have
• 28:04
  to. Once you get past that kind of
• 28:07
  one terabyte threshold, you might want to start looking at
• 28:10
  sparks and treinos and click houses and those kind of
• 28:14
  systems. So I'd go back to your
• 28:16
  streaming, then what is your data size?
• 28:18
  And then questions around like governance and do you need also
• 28:22
  like what does your organisation use often you might not be
• 28:25
  choosing that, you might just be locked into like well we use
• 28:28
  Snowflake or we use data bricks and that's kind of what you have
• 28:32
  to use. Have you seen it?
• 28:35
  Use cases where people do have a certain set of lock in, but
• 28:42
  they're able to almost like slide into something else quite
• 28:46
  easily because of Ibis now. Yeah, I just lets you kind of
• 28:52
  try out different things. I find the use case of like,
• 28:57
  yeah, taking a some of your data and Justice working with it
• 29:00
  locally in duct B very appealing.
• 29:01
  See if you're kind of locked into Snowflake, but you want to
• 29:04
  try out something more elaborate and just do it with the local
• 29:09
  sample of data, you can do that with DUTV and then go back and
• 29:12
  deploy it. See, it's definitely something
• 29:14
  that we see and it's also something back to our point
• 29:17
  earlier on SQL versus data frames.
• 29:19
  If if you're using, we heard this about Druid recently.
• 29:24
  SO1 user has like Druid and they don't want to think about like
• 29:29
  Druid SQL and all the nuances there.
• 29:31
  So they just use Ibis on top of Druid, they're using a data
• 29:34
  frame. And so to some extent it comes
• 29:36
  down to preference. If you have a SQL only system
• 29:40
  and you don't want to use SQL, you want to use a data frame,
• 29:42
  Ibis is a really good choice for.
• 29:44
  Ohh so this is fascinating. So basically it's like I have
• 29:49
  this sequel only set U, but actually I feel more comfortable
• 29:53
  in Python. Can I just slap Ibis on top of
• 29:58
  my SQL only setup and get the same results?
• 30:01
  And there are a lot of cases where kind of this repeating
• 30:05
  story of some new database vendor comes around and maybe
• 30:09
  they have a niche or you know, they're just really fast or they
• 30:12
  have good governance models or whatever, but they're SQL only
• 30:15
  interface. And that's fine for a long time.
• 30:18
  But as you get more and more customers and more and more
• 30:20
  users, you're going to get asked like, hey, I want a Dataframe
• 30:23
  interface on top of this. And then they have to ask the
• 30:27
  question, do we develop this in house?
• 30:29
  Is this something we had expertise for?
• 30:31
  How much time and effort does this take?
• 30:33
  In a lot of cases, Ibis might just already support it or be
• 30:37
  very easy to add support for. So yeah, if you have a SQL only
• 30:41
  interface, there's a very good chance Ibis already works with
• 30:44
  it. And if it doesn't, there's a
• 30:45
  very good it's. It's pretty easy to add and then
• 30:48
  yeah, you just have a pretty fully featured data frame
• 30:51
  interface on top of your what was a SQL only engine.
• 30:55
  So where do you want to take Ibis next?
• 30:57
  Where are you seeing things going?
• 31:00
  We are doing a bit of boring stuff, so just like working on
• 31:03
  stability. Ibis has an interesting story of
• 31:07
  it started as a mono repo and then it kind of split out into
• 31:10
  having repos for back end. We're actually back in a mono
• 31:14
  repo right now. So those 20 plus back ends are
• 31:16
  all in the Ibis model repo and we probably want to split that
• 31:21
  out again. So getting to the point where
• 31:23
  like I don't know, Snowflake or Click House could just own their
• 31:26
  repository in their back end in their own GitHub organisation or
• 31:30
  wherever they want to store that.
• 31:32
  Theseus actually is in a private gold drawn data repo.
• 31:36
  So that's a kind of external back end right now that we might
• 31:39
  try to use as the model for doing that.
• 31:42
  But to get there, we have to do a bunch of, you know, setting U,
• 31:45
  testing, how will these external back ends run the gigantic test
• 31:50
  suite that we have for Ibis? How will they have a stable
• 31:53
  interface that they can use? How will we like version them
• 31:56
  together? There's a lot of like pretty
• 31:59
  technical questions there, so that's kind of the boring stuff.
• 32:01
  More exciting, we're working on actually an Ibis mill package.
• 32:05
  So that kind of takes the idea of Ibis and brings it to machine
• 32:09
  learning, data preprocessing, pipelines.
• 32:13
  So yeah, so if you. Because there's ohh interesting
• 32:17
  since it's so desperate and you have so many different ways to
• 32:22
  preprocessed data. It's like Ibis is going to
• 32:25
  create the standard around that too.
• 32:28
  Yeah, so we the idea behind Ivis smell is kind of the same idea
• 32:33
  behind Ibis of a lot of, you know, psychic learning.
• 32:37
  Pandas basically make the assumption that your data fits
• 32:40
  in memory and is running in some single threaded thing.
• 32:44
  But what if I want to run my data preprocessing pipeline
• 32:47
  right on snowflake or right on pipework?
• 32:49
  Or of course there's like Spark ML and things.
• 32:53
  But can Ibis just provide a standard preprocessing interface
• 32:57
  on top of these 20 plus back ends.
• 33:00
  So then you can like run it on Duck TB locally to do your
• 33:04
  experimentation and run the exact same pipeline on whatever
• 33:07
  snowflake, Theseus, click house Trino.
• 33:11
  See how we're we're launching that soon, which will be
• 33:13
  exciting. It's kind of already public and
• 33:15
  out there, but not something we've hyped up too much.
• 33:19
  Yeah, you can think of it as like a a kind of a psychic
• 33:22
  learned pipeline. But again, that will run on any
• 33:24
  of these back ends. But you're not playing in the
• 33:27
  like pipelining abstraction layer.
• 33:31
  The not like the work castration like DBT DAXTER prefect level.
• 33:35
  On airflow, yeah, not, not. Just the just the like.
• 33:41
  You can already use Ibis for your preprocessing code.
• 33:44
  You could write a lot of this yourself, but part of it is like
• 33:48
  reusing. If you run a standard scaler or
• 33:51
  something, you need to save that standard deviation and whatever
• 33:55
  the other number is to be able to reuse it when you're like
• 33:58
  doing retraining. So there's like some ML specific
• 34:01
  stuff. One hot encoding, of course like
• 34:03
  that's. It's something you can do in
• 34:05
  Ibis natively, but it's a little tedious to write, so ISML just
• 34:08
  wraps it for you and has a nice little one hot encoder, stuff
• 34:13
  like that. Are there things that the
• 34:17
  community has asked for that you really want to get into
• 34:22
  production or you really want Ibis to be doing?
• 34:25
  And it's not quite there yet, but you have it in mind and it's
• 34:29
  going to be something that you're working on besides these,
• 34:32
  the Ibis ML, which I think is probably along those lines, but
• 34:35
  is there other stuff that you're thinking about?
• 34:38
  There's a few things. We get a lot of requests for
• 34:40
  back ends that to our conversation earlier, it's hard
• 34:43
  for, it's hard for us to just say yes and put in the effort to
• 34:47
  add every new back end that gets requested.
• 34:49
  And that's where kind of scaling is.
• 34:50
  An open source project would be very helpful, but having people
• 34:54
  like contributing a back end is a very good way to for people to
• 34:57
  get started. So there's kind of a backlog of
• 35:00
  back end requests there. Another one that comes up all
• 35:04
  the time that's very technically challenging is cross back end
• 35:08
  joins or working in data across different back ends and it might
• 35:13
  not be something that we ever do.
• 35:15
  It is something that we've had a prototype for that our lead
• 35:19
  engineer doesn't like to talk about because it's doesn't want
• 35:23
  to support it. But you know, like if you have
• 35:25
  data in Snowflake and data induct BI just want to join
• 35:28
  these two tables and have Ibis like go figure out how to
• 35:31
  efficiently do this. And yeah, it's not really what I
• 35:34
  this is for, but it is a cool. Sounds.
• 35:37
  Makes a cool demo. Yeah, and it is magical that
• 35:40
  keeps coming up with people and. Well.
• 35:43
  Maybe one day. When you say it like that, it
• 35:46
  sounds ridiculous that that's not possible.
• 35:50
  OK. Again though, as you just
• 35:52
  mentioned, that's not what Ibis is for.
• 35:54
  Potentially there's scope for another tool to come in there
• 35:58
  and say we can do that for you and that's a whole another open
• 36:03
  source project because I can imagine there are some sticky
• 36:08
  situations that you can get into when you're dealing with that.
• 36:11
  And so I don't know. You'd have to make the case as
• 36:14
  to why that should be done by Ibis and not some other tool.
• 36:20
  And that is something that like Trino kind of does with its like
• 36:23
  Federated. You can add different data
• 36:25
  sources and then Trino kind of figures out how to go and do
• 36:27
  that. So yeah, it does make a cool
• 36:30
  demo. It does come up a lot.
• 36:31
  I don't know, We'll see. It might not be something we
• 36:33
  ever be able to do. Yeah, not at the moment.
• 36:37
  Well, I know you have a hot take on LM's.
• 36:41
  I want to finish with that and what your thoughts are, what
• 36:44
  your feelings are on the whole AI movement.
• 36:47
  Yeah, I feel like I kind of joined.
• 36:50
  So I joined Azure Mill in 2018, which I feel like was at the
• 36:53
  tail end of the last AI hype cycle with computer vision and
• 36:57
  things. And kind of the end of my hot
• 37:00
  take is I feel like we're at the peak.
• 37:03
  Maybe I shouldn't say this, but I feel like we're at the peak of
• 37:05
  Ahye right now in this cycle, open edge to their announcement.
• 37:09
  It really feels like the forefront of LMS is saturating
• 37:14
  at the performance level and my hope is that we see kind of the
• 37:20
  hype side of things die down and see the actual usefulness ICU,
• 37:25
  because in the years that I worked at Azure Mill, I saw a
• 37:27
  lot of real mill applications and the mill side of things
• 37:32
  weren't necessarily that complicated.
• 37:34
  It was, you know, Xgboost or you know, maybe a neural network,
• 37:38
  but pretty standard stuff. The hard part was figuring out
• 37:41
  how to alley that in a real business scenario.
• 37:45
  So I'm hoping we see fewer shiny kind of look at this AI tool
• 37:53
  changing the world products and more things that actually
• 37:56
  improve our lives. And yeah, I'm I'm hoping, you
• 37:59
  know I think LM's are very important and going to be used
• 38:02
  in real scenarios. But I'm I'm personally a little
• 38:05
  sick of the hype that we've seen over the past, almost a year now
• 38:10
  or over. Well, I think that 100% shows
• 38:15
  that you are very much a product person because you're instantly
• 38:20
  thinking, OK, how can we incorporate this into the
• 38:23
  business to make the business more valuable or make it better
• 38:27
  for the end user? And a lot of the stuff is hype.
• 38:32
  Without that basis, it's just like, wow, look at the cool
• 38:35
  stuff that we can do with this now And you have to go back and
• 38:39
  say, alright, but how can a company incorporate that into
• 38:42
  their product to make something that is more useful for that end
• 38:47
  user? Yeah, it's very cool.
• 38:50
  You know, one of the first things I did with her all
• 38:53
  networks is like detect cats versus dogs.
• 38:55
  And that's very cool, right? I can put in an image and say
• 38:58
  whether it's a cat or a dog, but what do I do with that?
• 39:01
  You know, Do I put it in security cameras and like, it's
• 39:05
  now telling me whether there's a cat or a dog at my door?
• 39:07
  I don't know. I.
• 39:08
  Think yeah, I think LM are heading in that way.
• 39:11
  We've seen all the cats versus dogs demos.
• 39:14
  Now we have to see the what do these actually get used for in
• 39:17
  production? Yeah.
• 39:18
  I'm excited. I think that every a lot of
• 39:22
  times what you get. Is that alright?
• 39:25
  Well the hello world of the LLM use cases are chat with your
• 39:29
  data and be able to now your never going to have data siloed
• 39:36
  or if you need to ask questions. You don't have to bother your
• 39:39
  manager, or your manager doesn't have to bother you and that
• 39:42
  sounds great. It's just a little bit harder in
• 39:46
  practise and I don't know if you've found that, but just
• 39:50
  chatting with your docs, AI rag, whatever it is internal docs
• 39:56
  bot. I haven't found it that useful
• 40:00
  yet and maybe it's because the rag is not set up properly,
• 40:04
  maybe it is because it hits a limit or it's just sometimes
• 40:10
  that it's it's not. The best way that I'm looking to
• 40:13
  understand this information is through like asking a question
• 40:17
  and having to be very pointed with the question that I asked.
• 40:20
  You know what question that I need to ask in order to get the
• 40:24
  information so and that that probably is on me, but.
• 40:29
  Thinking about the announcement from Open AI and talking about
• 40:33
  how now we've got voice and vision all in an LLM, I saw
• 40:39
  something saying, all right, well, this is great, but how can
• 40:42
  you incorporate it into your company?
• 40:45
  What are some tangible ways that you can make this part of your
• 40:50
  product starting tomorrow? If you can hit an API and now
• 40:53
  you've got all these capabilities, what are you going
• 40:56
  to do? Are you going to create a voice
• 40:59
  bot or someone calls and complains from your customer
• 41:04
  service? Are you going to throw an Open
• 41:06
  AI agent on them? And I'm really curious to see
• 41:11
  how that all works out and what happens there.
• 41:15
  Yeah, I think it's very difficult and I did play around
• 41:18
  a lot with chatting with your database and things like that
• 41:21
  and doing rag. We actually have a, it's called
• 41:24
  Ibis Bird Brain. It's a project for doing just
• 41:27
  this and it's not something that we're contributing to, but it is
• 41:30
  out there if anyone wants to take a look and it's very hard.
• 41:33
  And I don't think that adding a voice layer necessarily helps
• 41:37
  with some of the fundamental problems that you touched on,
• 41:40
  which is like that one just gets things wrong or doesn't quite
• 41:44
  understand at a human level. You know what you're trying to
• 41:47
  ask. And the biggest issue I've seen
• 41:50
  is really in chaining. So if you're trying to make
• 41:53
  multiple LLM calls and do something that's more agentic or
• 41:56
  whatever, the second you get one thing wrong, you kind of go
• 42:00
  start going down the wrong path and it's very hard to correct
• 42:03
  that. And that's where I'm a little
• 42:05
  sceptical that we're going to get to a level of model
• 42:08
  performance that solves that problem.
• 42:11
  I don't know if that problem can be solved with a system with
• 42:14
  yeah, like better rag or better, you know, kind of guardrails and
• 42:18
  prompting will be very interesting to see.
• 42:20
  But I I hope that, you know, as I said, the hype kind of dies
• 42:24
  down and we start to see like, OK, how can we really make this
• 42:27
  work in practise? And it'll probably take a while.
• 42:32
  It's so hard to think about the hype dying down every other
• 42:36
  week. It's just like ohh, this new
• 42:38
  model was released. Ohh, now you've got this.
• 42:40
  Now that and it just gives the Internet and social media a
• 42:46
  whole lot of fodder to hype it up again.
• 42:49
  And it's almost like it's a little bit tiring.
• 42:52
  I know that as soon as Open AI announces something or Google
• 42:56
  announces something, that's basically going to be my news
• 43:00
  feed. So I try and stay off of social
• 43:02
  media for a few days and let it settle and die down a bit and
• 43:08
  ideally I don't have to deal with it as much.
• 43:11
  Yeah, it's we'll see. But it's definitely it's my news
• 43:15
  feed as well and it does get it gets old, I guess.
• 43:18
  Yeah. And there is another piece that
• 43:21
  you mentioned there that I guess could be used a lot with the
• 43:28
  world that you live in. Have you played around much with
• 43:31
  the text to SQL models? Yes.
• 43:35
  So that is largely what Ibis Bird Brain did.
• 43:38
  So we as Voltron data, we get asked by customers, you know
• 43:42
  what are you doing with this AI stuff, Bunch of questions.
• 43:45
  And I was partially tasked with like going and figuring out our
• 43:49
  answers and we saw basically 3 use cases for looms with data.
• 43:53
  One is generating synthetic data, which is actually very
• 43:56
  promising and cool. You kind of just get into like
• 43:58
  cost issues. If you want to generate a
• 44:01
  million or a billion data points like that's costly depending on
• 44:05
  how you're doing it. The second one is yeah, LMS
• 44:09
  rating analytical code and so that could be like Ibis code or
• 44:13
  pandas code which it does OK with.
• 44:15
  But it turns out LMS are actually pretty good at writing
• 44:17
  SQL. So if you're just doing like
• 44:20
  basic questions and you're feeding in the schema and maybe
• 44:24
  a description of your data, GPT 4 level LM's to a pretty good
• 44:29
  job of text to SQL. The problem is as soon as you
• 44:32
  start getting more specific and asking like vague questions.
• 44:35
  Because if I say, you know, what was our an example we used to
• 44:39
  use a lot was like what's your revenue for selling shirts at
• 44:42
  some apparel company? And it's like, what do you
• 44:45
  consider a shirt, Do you consider tank tops a shirt?
• 44:47
  Do you consider, you know dress shirts within the shirt carry,
• 44:51
  you know, a bunch of different stuff and an LM is not going to
• 44:54
  figure that out for you? And the third use case, which is
• 44:57
  actually pretty interesting too, is using LMS within subroutines.
• 45:01
  So that's like if you have a bunch of reviews for your
• 45:05
  product and you want to have an LM turn that into a rating from
• 45:10
  like zero to 10. Or you know, basically sentiment
• 45:12
  analysis. But LM's are so generally you
• 45:16
  can also do like translation from English to Spanish.
• 45:19
  You can do structured output. So like turn this BLOB of text
• 45:24
  into some structured JSON. So I think we're going to see a
• 45:26
  lot of use cases there. I think text to SQL will be more
• 45:32
  of a gimmick or just a very makes sense to have on basically
• 45:37
  any product where you can just type something in English and it
• 45:40
  will give you the sequel. But I think going deeper than
• 45:43
  some very surface level questions it breaks down.
• 45:46
  But there's definitely gonna be use cases with melons and data,
• 45:48
  and it will be interesting to see.
• 45:52
  Excellent, Cody. Well, I appreciate you breaking
• 45:54
  down, Ibis for me, walking me through what that looks like and
• 45:59
  all the work you're doing on it. And this has been great, man.
• 46:03
  Yeah. Thank you so much for having me.
• 46:05
  It's been awesome and great to meet you.