Latest / Agentic Conversations (formally mlops.community) / How Data Platforms Affect ML & AI // Jake Watson // #207

Transcript

How Data Platforms Affect ML & AI // Jake Watson // #207 · Agentic Conversations (formally mlops.community)

• 0:00
  So my name is Jake Watson. My job title is Principal Data
• 0:07
  Engineer and my coffee is is is bourgeois Espresso.
• 0:12
  Welcome back to the ML OPS community.
• 0:14
  I am your host, Dimitrios, and hoping that you are having a
• 0:19
  magical day full of surprises, and beautiful surprises at that.
• 0:25
  Ohh, stop the tape. Before we get into this next
• 0:29
  episode, I want to tell you about our virtual conference
• 0:33
  that's coming up on February 15th and February 22nd.
• 0:37
  We did it 2 Thursdays in a row this year because we wanted to
• 0:42
  make sure that the maximum amount of people could come for
• 0:46
  each day since the lineup is just looking absolutely
• 0:52
  incredible, as you know, we do. Let me name a few of the guests
• 0:56
  that we've got coming because it is worth talking about.
• 1:00
  We've got Jason Lewis, We've got Shreya Shankar, We've got Drove,
• 1:04
  who is product Applied AI at Uber.
• 1:07
  We've got Cameron Wolf, who's got an incredible podcast and
• 1:11
  he's Director of AI at Rebuy Engine.
• 1:13
  We've got Lauren Lockridge who is working at Google also doing
• 1:19
  some product stuff. Ohh, why is there so many
• 1:21
  product people here? Funny you should ask that
• 1:24
  because we've got a whole AI product owners track along with
• 1:29
  an engineering track. And then as we like to, we've
• 1:32
  got some hands on workshops too. Let me just tell you some of
• 1:36
  these other names just for a moment, you know, because we've
• 1:39
  got them coming and it is really cool.
• 1:42
  Haven't named any of the keynotes yet either, by the way.
• 1:45
  Go and check them out on your own if you want.
• 1:47
  Just go to home dot Milos dot community and you'll see.
• 1:51
  But we've got Tungi, who's the lead researcher on the deep
• 1:54
  speed project at Microsoft. We've got Holden, who is the
• 1:58
  open source engineer at Netflix. We've got Kai, who's leading the
• 2:04
  AI platform at Uber. You may have heard of it.
• 2:07
  It's called Michelangelo. Ohh my gosh.
• 2:09
  We've got Fazan, whose product manager at LinkedIn, Jerry
• 2:13
  Lewis, who created good old Lama Index.
• 2:17
  Ohh, he's coming. We've got Matt Sharp, friend of
• 2:19
  the pod, Shreya Rajpal, the creator and CEO of Guardrails.
• 2:25
  Ohh my gosh, the list goes on. They're 70 plus people that
• 2:29
  will. Be.
• 2:31
  With us at this conference. So I hope to see you there.
• 2:36
  And now let's get into. This podcast today we're talking
• 2:39
  with Jake all about data platforms.
• 2:41
  I love his articles. I'm going to just come right out
• 2:45
  and say it. I'm a huge fan of what he's done
• 2:48
  when it comes to building data platforms and what you need to
• 2:51
  know as you're building out your data platforms.
• 2:54
  He's gone deep into everything from the architecture, how you
• 2:59
  can add and maximise value and then all the pieces that come
• 3:03
  with the data platform that coincidentally or not so
• 3:08
  coincidentally have a lot of overlap when it comes to your ML
• 3:13
  platform. So these are things like data
• 3:15
  obs, data modelling, data pipelines, my God, data
• 3:19
  transformation, data quality, data governance.
• 3:22
  Can I say data one more time? Let me try and sneak that in
• 3:26
  there. Data security obviously and
• 3:30
  organisational pieces like just people, teams, hierarchy of
• 3:34
  teams and how you make that happen.
• 3:37
  So this conversation centred around all of these posts that
• 3:41
  he's done and things that he's been seeing out in the wild and
• 3:45
  what has been most effective. I almost said effectful, pretty
• 3:50
  sure that's not a word. What has been the most effective
• 3:53
  as he's been out there doing his thing as the principal data
• 3:58
  engineer. Hope you enjoy and if you can do
• 4:01
  one thing and one thing only, We would love it if you share this
• 4:06
  podcast with one friend yours. Talk to you soon.
• 4:16
  Well, Jake, it's a pleasure to have you on here.
• 4:19
  I am very excited because we get to talk all about the data
• 4:22
  platform today, how that interfaces with ML platforms,
• 4:26
  really what I am calling 2024, the year of the data engineer,
• 4:32
  which feels like they were a little bit forgotten in 2023.
• 4:37
  I don't know if you felt that too, but the AI hype and the LM
• 4:40
  hype, it made people think that like these jobs are almost
• 4:45
  obsolete I think. And those that have been in the
• 4:49
  pits digging trenches knew that they weren't going anywhere.
• 4:52
  If anything they were gonna get more valuable.
• 4:54
  And so I've been seeing a tonne of demand in the community for
• 4:59
  topics like this that are going to help upskill and get people
• 5:03
  that even if they are further down the line on their LM
• 5:06
  journey, they still need that good old data.
• 5:10
  They needed to be transformed, they needed to be modelled
• 5:13
  correctly and so who better to talk to than you about all this
• 5:17
  good stuff? Thank you.
• 5:19
  I'm glad to be here. I'm happy.
• 5:21
  I'm looking forward to it. Fun chat, just getting into the
• 5:24
  weeds. Yeah, I definitely feel like we
• 5:27
  did feel that the hike was all consuming, but now I'm starting
• 5:32
  to see increasingly people asking me how do I get this
• 5:35
  element thing working, app scale and with our data etcetera,
• 5:40
  etcetera. So yeah.
• 5:44
  Exactly. So let's give everyone a bit of
• 5:47
  background on you. I know you from your blog and
• 5:49
  your sub stack. If you wanna whatever, like
• 5:52
  newsletter, blog, whatever sub stack is these days, yeah, yeah.
• 5:55
  And the work that you've done on there.
• 5:57
  You have super comprehensive blog posts on how to build data
• 6:02
  platforms, the reasons for different components.
• 6:04
  I want to get into all of that, but I think it's worth talking
• 6:07
  to people about what you do day in, day out.
• 6:10
  Yeah, that's absolutely fine. My title is Principal Data
• 6:15
  Engineer. I work for the Oakland Group who
• 6:18
  is a data consultancy. Yeah, even my title can be a bit
• 6:23
  misleading in that respect as you could say it's simply data
• 6:27
  consultant as I tend to go work through a variety of as you've
• 6:31
  usually starting with the client when they haven't even got
• 6:34
  anything bit built and it's a blank sheet of paper.
• 6:36
  So trying to architect, design that gather requirements, making
• 6:41
  show that that they're going on the right path.
• 6:43
  OK, so you wrote a really good blog post that I want to dive
• 6:47
  into that is all about the data platform foundations, and I'm
• 6:52
  wondering what inspired you to write that post.
• 6:57
  I might have described it a little a little bit in my intro
• 7:01
  post when I did it, which was basically data platforms can end
• 7:06
  up being you know, dozens of different elements and different
• 7:12
  and lots of talking with lots of different people as well.
• 7:15
  So you usually need a really large support network in your
• 7:18
  company to make it stand together.
• 7:22
  Just a platform builds and keep supporting it as well and
• 7:25
  maintainable, which can be a whole different kettle of fish
• 7:27
  caught. It can be quite easy to build a
• 7:29
  POC and another toll thing to actually keep it running and in
• 7:33
  production as well. So I wanted to sort of get that
• 7:37
  across about all the different parts and and also try to
• 7:42
  explain there's lots of different varieties as well.
• 7:44
  There's not just your modern data stack, but might have 20
• 7:49
  different vendors all attached to it, which I know is a gross
• 7:53
  overwhelming thing. But it might just be a database
• 7:57
  doing some transformations, taking some data, transforming
• 8:02
  it, and then doing some outputs on that and getting some
• 8:06
  insights from that. And it could be any WET, it
• 8:08
  could be something as simple as that, or it can be something
• 8:10
  very complex where you've got your Ubers, Airbnbs, where
• 8:13
  you're building your own custom pieces of software to do things
• 8:18
  that no one else is doing in the industry.
• 8:20
  Yeah. So one thing that I really liked
• 8:22
  is how you basically showed both sides of the spectrum and you
• 8:26
  said, look, here is what it can be if it is the simplest of the
• 8:31
  simple and maybe there's a one person band that is trying to
• 8:37
  upkeep this whole platform and then here's what it can look
• 8:40
  like in the most messy of situations.
• 8:43
  And so I in your day-to-day, what's more common, what do you
• 8:47
  feel like you're seeing a lot of as far as patterns when it comes
• 8:52
  to the platform? It's somewhere in between.
• 8:55
  So as I mentioned, I work with quite a lot of enterprises that
• 9:00
  are that might that right that that can either be standards
• 9:05
  only or they might be quite far along on the journey.
• 9:07
  So it can vary. They usually got a couple of
• 9:11
  people on their data teams, so it's rarely just one or two.
• 9:14
  But I have worked with those with those people and also
• 9:17
  sometimes people that don't even have a data team yet and they're
• 9:21
  still trying to figure out how they can they can use all this
• 9:26
  about how can they actually get you know make some data-driven
• 9:28
  insights all all the way to sort of like large large 5000 man
• 9:34
  teams on UH that have like lots of different micro services
• 9:38
  going along where they're punting data from here and there
• 9:40
  and everywhere it's sort of like thing in between.
• 9:45
  So I I'd I'd say I was trying to say the average what I suppose
• 9:50
  the average data team I will be just plastic two pizza team.
• 9:55
  It's sort of like it can be difficult it's quite it's it can
• 9:59
  be not too bad to scale to that. It gets quite difficult to
• 10:02
  actually scale beyond that because you then have to start
• 10:04
  to place it into teams and then it starts and then you have to
• 10:07
  have another level of the hierarchy on that And then you
• 10:11
  need to sort of like think about maybe you need to split up parts
• 10:14
  of your platform. So he so different teams coming
• 10:18
  well like work autonomously. Thomas Lee at speed as well,
• 10:21
  he's starting to almost get into sort of like the data fabric,
• 10:24
  data mesh sort of landscape. So it can be quite difficult to
• 10:27
  move on from there and some people just saw like I'll just
• 10:30
  build another data silo and deal with it that way.
• 10:34
  And one thing that is clear to me is that you talk about how
• 10:41
  there is that data mesh that's coming up.
• 10:43
  You also speak about like the organisational side of things,
• 10:48
  right? What you're talking about right
• 10:50
  there is very much, it's not a problem so much scaling on the
• 10:54
  side of the technology. It's more how are we going to
• 11:00
  have our teams be responsible for the data platform now?
• 11:04
  What are we going to look at as far as a hierarchy of the people
• 11:08
  that are working on the data teams?
• 11:10
  Are there ways that you've seen this be successful?
• 11:13
  I mean, I know data mesh is a buzzword.
• 11:15
  It feels like something. I've seen people talk about how
• 11:18
  data mesh is dead in 2024 and it was very lofty or is a very
• 11:26
  lofty goal. I don't know if it really caught
• 11:30
  on as much as people were hoping for.
• 11:32
  Unless you have other thoughts on that, I would love to get
• 11:34
  your take on that. But as far as the organisational
• 11:37
  side of things and how people can architect that, have you
• 11:41
  seen success in different patterns there?
• 11:45
  And I'd I'd say there's been a couple of success stories but it
• 11:51
  it it is as you say it's probably been few and far far
• 11:54
  between them people expected. I think it does come back to the
• 11:58
  people element that people usually when it came to data
• 12:02
  mesh because they have people issues, but people issues to
• 12:06
  solve are quite hard. So it will take a few years for
• 12:08
  them to solve that, and it's usually people that have
• 12:12
  actually that. I've already been on the Data
• 12:16
  mesh track but didn't quite realise it at the time.
• 12:20
  Is working in one large public sector organisation that we're
• 12:24
  almost doing sort of the data mesh but didn't realise they
• 12:28
  were doing it where they were having for each of their that
• 12:34
  their lines of business where they were building their own
• 12:37
  platforms and organisations with like Essential Governance
• 12:42
  playing underneath and managing all data governance and billing.
• 12:48
  So I think a couple of organisations were already on
• 12:51
  the path but didn't realise it. But I think it's I think it's
• 12:54
  going to be a slow burn and it doesn't it doesn't fit a lot of
• 12:57
  organisations as well. Yeah, I think you need to be.
• 13:00
  You need to be alive organisation to truly benefit
• 13:03
  and you need to be highly distributed by nature.
• 13:07
  So if you're a company that's globalised and has lots of
• 13:10
  offices and lots of different countries and are companies like
• 13:15
  insurance firms that by nature they're split up.
• 13:18
  You know you've got get home insurance, car insurance and
• 13:21
  they'll be having a name, may have their own, each have their
• 13:24
  own data teams then it can make more sense there as opposed to
• 13:29
  someone that already has a centralised somewhat working
• 13:33
  well data team already and doesn't need to go too much in
• 13:36
  that they might like. And there is some things you can
• 13:39
  do sort of that on the sort of like on the way to measure mean
• 13:44
  comes back to that. There's the classic data Mart
• 13:48
  modelling thing that's been around since the 80s.
• 13:51
  Uh-huh. Yeah, we're splitting up your
• 13:54
  warehouse by domain and that can get you a long way towards a lot
• 13:59
  of your pain to solve in terms in terms of that.
• 14:02
  It won't be like a fully distributed system, like a data
• 14:05
  mesh. It would have like separate
• 14:07
  DevOps systems, which might still mean you can only style so
• 14:11
  far with the data map structure. So if you are hitting SO you
• 14:14
  might want to explore that and see if you're still hitting the
• 14:16
  ants with that before trying other methods as well.
• 14:20
  And there's so many places that I want to take.
• 14:22
  This one is really along the lines of different the
• 14:28
  structures for the the data platform.
• 14:33
  Are there design patterns that you have seen that have worked
• 14:38
  out well and are easiest to implement or that have been
• 14:43
  useful and it's almost like your go to recommendation?
• 14:48
  So I've been working a lot in the lake house and I've been
• 14:54
  quite, I've been quite a big fan of what I think what database
• 14:58
  called the medallion architecture.
• 15:01
  It's also been adopted by many people in streaming and like
• 15:04
  you've seen a bit with people that are building Flink if
• 15:08
  you've come. Across the, the, the.
• 15:11
  Stream processing library which can do also like Spark can do
• 15:16
  batch and real time streaming and that can reduce the sort of
• 15:21
  like overhead and amount to a learn if you want to do both
• 15:26
  batch and real time. I'm starting to see more and
• 15:28
  more people slowly wanting to have stream streaming because
• 15:32
  they're like because they use are starting to demand real time
• 15:36
  updates and sort of like notifications.
• 15:38
  You know they've been using those Uber apps and Netflix and
• 15:41
  getting notifications in real time and says I want our company
• 15:44
  to have a bit of that too as start demanding it to their data
• 15:47
  team. So they've got to start figuring
• 15:48
  out how do we get real time, and one of the easy ways is to get
• 15:54
  good enough. Um, streaming is through
• 15:58
  something that can offer both batch and real time, like
• 16:01
  Databricks and Snowflake. I know I've been offering it
• 16:04
  too. I'm not going to try and be too
• 16:06
  much of A shrill to. One or the other?
• 16:11
  Yeah, Actually one thing, it's funny you mentioned that because
• 16:16
  one thing that I was talking to a friend about the other day was
• 16:19
  how so many companies will start with one, be it snowflake and
• 16:25
  then they kind of hit the maxim that Snowflake can handle and
• 16:30
  then they'll have to go to data bricks for a data lake.
• 16:33
  And so they end up paying both of them millions of dollars and
• 16:37
  they they're like there's caught of you a better way than this.
• 16:40
  And it's like we're using them boat.
• 16:42
  We can't figure this out. I mean I can't.
• 16:47
  I mean the difference from each company.
• 16:48
  So I can't offer like say a silver bullet on that and that
• 16:53
  things. I know there is a cost
• 16:56
  implication. We reason both.
• 16:57
  They can cost a lot of money at scale which means some people
• 17:01
  start moving back into open source.
• 17:03
  It's it. Which is probably a lot of
• 17:07
  reason why the likes of Netflix and Airbnb, and I think we're in
• 17:10
  a recent blog about Instacart, remove some of their stuff back
• 17:14
  from managed services to run in themselves.
• 17:19
  They can save a lot of costs like I think they move from AWS
• 17:22
  Kinesis to Kafka and Kubernetes for example to save.
• 17:27
  But you know once you start running those servers and then
• 17:31
  paying overhead on top of that for that managed services
• 17:34
  because that those managed services extra costs can start
• 17:37
  to add up to actually a number of engineers that you can just
• 17:40
  hire to to run that service itself.
• 17:44
  But yeah, I don't know how often that sort of light comes up in
• 17:49
  terms of they they tend to be like big high maturity companies
• 17:55
  that can sort of do that sort of thing.
• 17:58
  So one piece that you have on the data platforms that is
• 18:02
  almost like another thing that I feel like a lot of people talk
• 18:06
  about incessantly is the data modelling piece and how
• 18:11
  important it is that you get that right for the rest of the
• 18:16
  downstream effects, right? Can you talk a bit about how you
• 18:19
  see that? Yeah, I mean it's it's a, it's
• 18:23
  a, it's a tricky one with data modelling because it comes back
• 18:28
  to that you're trying to get stuff done as quickly as
• 18:31
  possible. So it's one of those things that
• 18:33
  can be left on the way side until it becomes a massive
• 18:36
  problem and it becomes absolute pain to sort later.
• 18:40
  It had it it. I think the one problem is at
• 18:44
  the moment is if there's no one golden data model to use every
• 18:50
  every organisation will have something slightly different.
• 18:52
  All might have multiple data models in that where for example
• 18:57
  machine learning it's classically you want a one big
• 19:00
  table so you can put that into a matrix and pass that into
• 19:05
  Psychic learn or pytorch or Tensorflow.
• 19:08
  But say for business intelligence, Power BI and
• 19:12
  Tableau, but for star Schemer, Kimball dimensional modelling
• 19:16
  with facts and tables. So you might end up running
• 19:20
  multiple things and then you might be doing log analytics
• 19:25
  which is sort of like it's all in JSON and Elasticsearch and
• 19:28
  mongo DB. So you can end up with like
• 19:31
  multiple different models and and that can make you hard to
• 19:34
  sort of like make a single source of truth on that and to
• 19:40
  match that all together. So I wouldn't say I've done
• 19:42
  multiple, but be flexible, don't try so you might.
• 19:46
  I find it works better if people are a bit more flexible and not
• 19:49
  try and have one golden type of data modelling technique of
• 19:55
  saying ohh I have everything has to be king or everything has to
• 19:57
  be one big table you're going to find you might have to be
• 20:02
  flexible there and sort of like fit it best to your use cases
• 20:05
  and tooling that you're using is probably the best bet I've got.
• 20:09
  Is the best advice I can offer on that one really.
• 20:13
  Yeah. And then how do you deal with
• 20:15
  the sprawl that inevitably comes from that, right?
• 20:18
  With with that the I think that is being an increase in call to
• 20:24
  bring back the good old days of enterprise modelling which is
• 20:30
  more it's it's sort of like a it's it's escaping me is the
• 20:37
  conceptual modelling which is sort of like getting back to
• 20:42
  what is a customer, what is a product and can we possibly have
• 20:47
  a golden data set for customer and product if purpose which is
• 20:52
  a long and hard road to go down. Hey everyone, my name is Aparna,
• 20:57
  founder of Verizon, and the best way to stay up to date with Emma
• 21:00
  Lops is by subscribing to this podcast.
• 21:05
  And another problem with that is conceptual modelling was born in
• 21:08
  the 80s, so it was back in the days of Walter Roll and the
• 21:12
  People. And you'll find that a lot of
• 21:14
  the people that preach the good word of conceptual modelling are
• 21:16
  still doing waterfall. So there needs to be a bit of
• 21:20
  give and take where you need to take conceptual modelling into
• 21:24
  the days of cloud and agile where you're trying to split up
• 21:27
  the works. It doesn't become, I'm going to
• 21:29
  spend three years just to make one golden customer table and
• 21:34
  work out what the hell a customer is and also but also
• 21:39
  not turn it into a a mess of a sprawl where we're never going
• 21:42
  to work it out, we don't care. And now we're just gonna make 50
• 21:46
  versions of the customer table and no one knows actually how
• 21:49
  many customers we have it in our company.
• 21:52
  So it's it's trying to find that balance.
• 21:55
  So you can bring the fast, you can bring bring some value, but
• 22:00
  you might find that you have to make compromises in that.
• 22:02
  Yeah. And one other piece that you
• 22:04
  talk about a bit is the data quality and how basically you're
• 22:09
  making trade-offs when you are dealing with all these different
• 22:12
  kinds of data models. Inevitably, I think you're going
• 22:15
  to have to say how can we keep the data quality high and have
• 22:20
  that integrity in our data modelling.
• 22:22
  I I think that I've I've read a few couple of people that say
• 22:26
  you can solve everything with just enough tests.
• 22:30
  I I'm I'm I'm a little bit sceptical of that approach.
• 22:34
  There's there's always more potential tests you can write
• 22:38
  because it's very rare you have 100% coverage because data
• 22:43
  because data quality tests are queries against your data
• 22:47
  warehouse and if you're using the light to Databricks and
• 22:50
  Snowflake you'll get charged per query.
• 22:52
  So the more data quality test you make, the more it will cost
• 22:55
  you not to take in all the maintenance costs and all that.
• 23:00
  So data quality is kind of sort of like linked to classic
• 23:04
  technical debt in software engineering where the more the
• 23:10
  accrues, the harder it can be to make changes because you know
• 23:14
  that if you make any changes you can introduce data quality
• 23:19
  issues and you need data quality tests to give you some
• 23:23
  confidence in that. So I know it's been called by
• 23:27
  like the likes of Monte Carlo sort of like Day to Debt where
• 23:32
  it's and the more data that you accrue the harder it is to sort
• 23:37
  of like change at pace in an in an agile manner.
• 23:41
  O There is that balance of data quality can cost you a lot of
• 23:46
  money to do, right, But it can also cost you not to do.
• 23:50
  It costs you not changing and dribbling value for the business
• 23:54
  and get and and the more value you give, the better your
• 23:57
  company's going to be. And so you break down a lot of
• 24:00
  different pieces when it comes to the data platform and
• 24:03
  especially in these different the post, you go in depth on all
• 24:07
  these different pieces, right? And is there a hierarchy that
• 24:11
  you feel like you can't get to, be it data quality before you
• 24:16
  have data modelling done or you can't worry about your data
• 24:20
  transformations? Or is it all weighted and valued
• 24:23
  equally? How do you look at that type of
• 24:26
  thing as you're building out your data platform and you're
• 24:30
  trying to make it, you're trying to build for the future too?
• 24:34
  It really depends on how fast you want to move and deliver
• 24:37
  value. Like if you're doing a quick and
• 24:39
  dirty OC MVP, depending on which terms you the fur, um, then you
• 24:49
  might not concentrate too much on getting modelling.
• 24:51
  You're just trying to prove a thing can work and we'll and
• 24:54
  we'll deliver value rather than getting hung up on making robust
• 24:59
  tests on robust modelling. But however if you're doing it,
• 25:02
  if you're working on a critical reporting service, accounting
• 25:06
  safety reports, things that where lots of money of our lives
• 25:11
  and can you know lives change can matter on that data, then
• 25:16
  you want robust, you want robust quality on that and we've
• 25:21
  modelling it's sort of like if you are if you know you're going
• 25:24
  to end up working on a rather big system that's going to have
• 25:26
  to scale out then trying to buy into that model in early on can
• 25:30
  reach you much return. So if you can do that, it's
• 25:33
  recommend to do that. So I think the, I think the
• 25:37
  answer it it you've got to. So I take the user requirements
• 25:41
  and sort of like fit it to what sort of works and then also
• 25:44
  build a plan to say right, we're going to come back to that and
• 25:49
  at a later date if we've proved that X works then we'll go to Y
• 25:52
  and sort of like keep our technical debt down there.
• 25:56
  There is also a monitoring element as well to this, so you
• 26:01
  want to monitor how well your data platforms do it.
• 26:04
  So if it isn't becoming as robust as you want, you want to
• 26:07
  actually concentrate on getting your technical debt down.
• 26:10
  So if you have feeling like you're getting too many errors
• 26:12
  and people aren't trusting your data platform, then you need to
• 26:14
  get more robustness into that with data quality.
• 26:18
  And if you're feeling that the changes are not you're not
• 26:21
  making fast enough change, then your data model is holding you
• 26:24
  back, either in performance or how quickly you can make those
• 26:27
  changes. Then you may need to look at
• 26:30
  remodelling your data that works better in terms of performance
• 26:35
  or rate of change. So as I'm looking at this and
• 26:41
  I'm thinking about a data platform and having a strong
• 26:45
  data foundation in place, one thing that I realise is that
• 26:49
  before you get a data foundation up and running, it's really hard
• 26:54
  to talk about doing any kind of AIML and I imagine you've seen
• 27:00
  that a tonne. And so how have you felt like
• 27:06
  there's been success when it comes to bolstering AIML on top
• 27:13
  of a data platform? What are some things and design
• 27:17
  patterns that you potentially have seen that have been useful
• 27:20
  in that regard? There's a couple of different
• 27:23
  ways that like we we. A common one is you have like
• 27:28
  I've seen a completely separate data science team which has its
• 27:32
  own pros and cons which might build out their own ML platform
• 27:37
  which is quite common out there in terms of data platform.
• 27:44
  What I find is what I find often useful is you you do want that
• 27:49
  fundamentals in there so you want to have some trust in your
• 27:53
  descriptive analytics before you start your inferential
• 27:56
  analytics. So you can build on top of them
• 27:59
  with a with a solid foundation that you know your pipelines are
• 28:04
  working. You know the dates you're your
• 28:06
  outputs and this is of high enough quality and that sort of
• 28:12
  makes it easier as you sort of like it's you're not so much
• 28:16
  building the plane while flying it too much.
• 28:19
  With that and it comes back to I think I mentioned in one of my
• 28:24
  books, I like the Pyramid of Hierarchies of the Data Science
• 28:27
  Hierarchy tree, which is fairly common, which is you want to
• 28:30
  get, You want to get the the foundational analytics in place
• 28:34
  before you then start. You wanna look, you want to
• 28:37
  first look back back at your historic data and know that's
• 28:41
  good before you can move on to start forecasting into the
• 28:44
  future and start making wisdom and knowledge from that.
• 28:49
  But I have seen some success with like data science seeing
• 28:53
  sort of like go away and creating their own solution and
• 28:56
  sort of like build their own thing.
• 28:58
  Umm, which is kind of interesting.
• 29:00
  It's sort of, it's like it probably go against most
• 29:05
  classical data practitioners which we just talked about that
• 29:08
  hierarchy insight. I suppose what that would give
• 29:11
  them is a bit more autonomy in that so there's more, there's
• 29:18
  more risk in them getting of getting the wrong outputs
• 29:20
  because they don't know the data it's as well and they're on
• 29:23
  their based on their solid foundations, but they have more
• 29:25
  autonomy to work and sort of like build fast, which can be
• 29:28
  quite useful when you're just trying to get something out
• 29:31
  quickly. Yeah, that that's interesting
• 29:35
  because it does feel like there is a very strong tendency for
• 29:40
  companies once they hit a certain maturity to have the
• 29:43
  data platform team, the ML platform team.
• 29:46
  And then almost like the data scientists are the users of the
• 29:51
  ML platform and it's like they're the customers for the ML
• 29:54
  platform team. And so the ML platform team goes
• 29:57
  and they are making sure that those data scientists are are
• 30:02
  taking care of and getting what they're needing.
• 30:04
  Yeah, which can be a I think. I think we have a fully.
• 30:11
  I don't think the the community at large has actually properly
• 30:16
  figured that out as well. There's there's quite A and it
• 30:21
  can be quite tricky because data access is one thing.
• 30:23
  I know the the common one is data scientists want access to
• 30:26
  the raw data, which can be a whole minefield to get through
• 30:32
  for political purposes because that data might be extremely
• 30:35
  sensitive for whatever reason. And they're very and companies
• 30:40
  are very worried to getting it out there.
• 30:43
  There's there's also the case of I think that there's sometimes
• 30:47
  it can be sometimes people go too early to the raw data big
• 30:50
  thinking that they're they can just use the raw data and figure
• 30:55
  it out themselves. When actually you need to step
• 30:58
  back a bit. Have a look at the business
• 31:00
  logic of what an analytics team have created and then sort of be
• 31:05
  able to apply that to your and mod machine learning models and
• 31:13
  if that makes sense So so you're not trying to run, you start not
• 31:17
  trying to run before you can walk and sort of like fully
• 31:20
  understand your data estate and the download your domain you're
• 31:23
  in before you start just hacking the raw data.
• 31:28
  Because you can end up wasting so much time just trying to
• 31:31
  recreate what the business intelligence team has already
• 31:34
  created because it's quite common to create features that
• 31:38
  are very similar to what a KPI is.
• 31:41
  Basically, models kind of end up just being a collection of KPIs.
• 31:46
  Ohh, interesting. Yeah.
• 31:48
  So don't reinvent the wheel. If you are looking for features,
• 31:52
  maybe the BI team has already created them with.
• 31:54
  The KPI it's. It's one thing I try to tried to
• 32:00
  look at the existing work done so far rather than new things.
• 32:04
  Just that, try to find those low effort, high value things when
• 32:07
  building when feature engineering and rather than
• 32:11
  trying to create your own complex calculations just to
• 32:14
  create a feature, try and find the low hanging fruit in that
• 32:18
  sort of sense. Wow.
• 32:21
  OK, then. Wow.
• 32:23
  OK. Yeah, I hadn't thought about
• 32:25
  that, but that feels like I trust your opinion on it.
• 32:29
  I think that it feels right, you know.
• 32:32
  I don't know how controversial that assumption is to be fair.
• 32:37
  But I had I I even came across one block a few years back where
• 32:40
  someone was sort of like where there then there might be some
• 32:44
  future where we're both where both business analysts and data
• 32:48
  scientists are using feature stores which might be a very.
• 32:52
  I think it's not something really done.
• 32:54
  So it's very controversial, but if you know where because at the
• 32:58
  end of the day you're building, you're trying to build
• 33:00
  interesting insights either via features or KPI and and maybe
• 33:05
  his feature stores the one place where data scientists and
• 33:09
  businesses can come together. Are we just calling the same
• 33:13
  thing by different names? When you have an analyst calling
• 33:16
  it a KPI and you have a data scientist calling it a feature,
• 33:20
  but the root of it is is really the same.
• 33:24
  Yeah, it can be. As I say, there are times when
• 33:27
  it can be different in terms of because it can be it it is.
• 33:31
  It can definitely be expressed differently in the outputs
• 33:33
  because a model or prefer, you know, a model wants, you know
• 33:38
  wants integers, preferably in their matrix or decimal numbers.
• 33:42
  They don't want text, whereas business intelligence end users
• 33:46
  want text. So there there are sometimes
• 33:48
  differences like that because you know an end user will want,
• 33:52
  say you know traffic like report where they want green, red,
• 33:56
  amber, but a model will want 123 and 01/2 or something like that.
• 34:03
  Yeah. And one thing that I think both
• 34:07
  ML and data engineers can agree on is they have a shit tonne of
• 34:13
  pipelines happening all over the place.
• 34:16
  Like pipelines are just the go to and we talked a little bit
• 34:20
  about air flow before we hit record and we've had a lot of
• 34:25
  people on here to talk about four ML, how airflow quickly can
• 34:30
  be a bit of a mess and so it might not be the best choice.
• 34:35
  I'm interested to hear your take a like data pipelines, what do
• 34:40
  you see them as when it comes to the data platform?
• 34:43
  How do you look at them and then be like how do those then feed
• 34:48
  data into the ML platform and what does that synergy look
• 34:51
  like? I think it it cannot be yeah I I
• 34:57
  could definitely see how air flow can run out of scale
• 35:03
  especially for ML because it can FF is it basic element he's
• 35:08
  creating a DAG and it and and honest like selected to create a
• 35:11
  like a lovely loop of you know you get the output data a lot of
• 35:15
  them all algorithms like to create that loop of where it's
• 35:19
  almost doing a force feedback you could say like so yeah and
• 35:25
  it just it it is actually quite painful it can be quite painful
• 35:29
  to scale as well to in terms of manage and it's sort of the and
• 35:33
  I could imagine someday so I just want to get away from
• 35:36
  managing an air flow pipeline rather than you know there wants
• 35:40
  to be building new features making their model better
• 35:43
  getting more insights or their customers rather than managing
• 35:47
  an airflow instance. I I know there's there there is
• 35:53
  sort of more sort of interesting alternatives that might suit um
• 36:00
  data scientists to sort of like I know this keep keep flow which
• 36:05
  is sort of like more aimed at ML.
• 36:09
  Yeah, yeah. I mean, there's a there's a
• 36:11
  tonne, right? There's cube flow, there's
• 36:13
  flight, there's XML, there is. And then there's the data
• 36:17
  engineering ones you've got like Mage and Daxter and Prefect and
• 36:22
  all that fun stuff. So there's a plethora of them.
• 36:25
  Yeah, I think prefect for example, because I've used that
• 36:28
  in the past as well. That was that works much better
• 36:31
  for and now there's a bit more flexible.
• 36:35
  The API is easier. It's easy to it's sort of like
• 36:39
  it's it's it's had it's it's a very much a system that's sort
• 36:43
  of like has tried to learn from flames and then it's and airflow
• 36:48
  itself was sort of like done that idea.
• 36:50
  It's sort of like I think people have realised it has been
• 36:52
  upgraded a bit over the last few years.
• 36:55
  It's it is. Again, he can really depend
• 36:58
  because you can use it. You can get away with using air
• 37:01
  flow to a certain extent depending on what how many, you
• 37:03
  know, how many models you want to produce and uh, what kind of
• 37:07
  models you want to produce if you are.
• 37:09
  If you're not feeding the data back to the start, it's much
• 37:12
  easier to build a pipeline because as I say that it's all
• 37:18
  about building a directed acyclic graph, you know, dial
• 37:22
  and they and they prefer to just go one way.
• 37:24
  They don't want to be doing things like looping back on
• 37:26
  themselves. Yeah, that's a great point.
• 37:30
  Well, Jake, this has been awesome, man.
• 37:31
  I appreciate you coming on here and teaching me a little bit
• 37:35
  more about data platforms and of course 2024, It's all about
• 37:40
  them. Data engineers, I'm gonna call
• 37:42
  it right now or in January, we're gonna be hearing a lot
• 37:46
  more about how vital it is, especially as these LM projects
• 37:50
  run up again the data problems and they're going to be called
• 37:55
  in the data engineers all over the place.
• 37:56
  Yeah, yeah. I've heard it's sort of like
• 37:59
  there's a lot of talk about intelligent data platforms,
• 38:01
  which is sort of like mixing generative AI and ML and data
• 38:07
  and all into one place. So that almost sort of like a
• 38:10
  symbiotic relationship where you are using a, you're using using
• 38:15
  AI to make your data platform better, but you're also using
• 38:18
  your data platform to make better AI and ML.
• 38:21
  Yeah, you see a lot of text to SQL, large language models
• 38:25
  coming out because of that. So we'll see.
• 38:28
  Hopefully there's. It looks promising.
• 38:30
  It seems exciting. Let's see where the future takes
• 38:33
  us, huh? Yes.
• 38:34
  Just make sure you have the right data for it.
• 38:36
  Have high quality, robust data point.
• 38:43
  This is Skyler. I lead machine learning at
• 38:45
  Health Rhythms. If you want to stay on top of
• 38:48
  everything happening in ML OPS, subscribe to this podcast.
• 38:52
  Now. Now.
• 38:55
  Now. Now.
• 38:58
  Now. Now.
• 39:03
  Now.