Latest / Agentic Conversations (formally mlops.community) / Data Quality = Quality AI // AIQCON Panel

Transcript

Data Quality = Quality AI // AIQCON Panel · Agentic Conversations (formally mlops.community)

• 0:10
  So hey, everybody, this is data quality equals AI quality.
• 0:14
  And today you have four of the best people in the space to be
• 0:17
  able to talk to you about this. We had conversations beforehand
• 0:20
  about this. And I got to tell you, they have
• 0:22
  some of the best opinions that I've heard in the space, A space
• 0:25
  I've been in for probably close to a decade now.
• 0:29
  And so this is a treat in the sense that you get to hear all
• 0:32
  four of them at once, which is really awesome.
• 0:35
  So to blanket start this, we're just going to start with what
• 0:39
  data quality means to you and what the definition of that
• 0:44
  actually is in its applications to AI quality.
• 0:48
  So, Maria, I'll start with you. Hello everybody.
• 0:53
  Thank you for coming and it's a great question.
• 0:57
  What is data colouring? It means different things to
• 1:00
  different people in different scenarios.
• 1:02
  I actually want to state of a failure case.
• 1:05
  I'm sure you've heard of this expression garbage in, garbage
• 1:09
  out, right? So that's the situation I
• 1:12
  absolutely want to avoid. And so that starts with data
• 1:16
  quality. Yeah.
• 1:22
  So data quality for me, it looks like it, it needs to be
• 1:29
  available on certain metrics and then those metrics need to be
• 1:34
  monitored. And then if you're able to
• 1:39
  monitor on a certain metrics of data completeness, data
• 1:42
  accuracy, data validity, timeliness of the data, that's
• 1:46
  sort of really important into building like a good monitoring
• 1:51
  structure and then yeah, makes sense.
• 1:52
  Yeah, makes. Sense.
• 1:55
  Joe, let me ask you what happens when you ignore data quality?
• 1:59
  You get the current state of affairs in most companies.
• 2:02
  So yeah, I love it. Dad, what happens when you
• 2:13
  ignore data quality? Yeah, hard for me to say
• 2:17
  anything much different than Joe.
• 2:19
  I think there's a few things that start to happen.
• 2:22
  The big one is, is that you start to run into this issue at
• 2:25
  scale where it becomes very, very difficult to understand why
• 2:31
  models are failing, why dashboards are showing the wrong
• 2:35
  information, why our AI programme is not taking off the
• 2:39
  ground. One of the really common
• 2:41
  indicators of data quality not being implemented from the very
• 2:44
  beginning, or at least early enough, is that incidents are
• 2:48
  happening. And then when you go to an
• 2:49
  executive sponsor within the company and say, hey, we should
• 2:52
  fix this, they say show me the ROI of investing in data
• 2:55
  quality. That that.
• 2:57
  That's how you know that something has probably broken
• 2:59
  down at some point. Can I ask the audience a
• 3:02
  question here? Who works at a company?
• 3:07
  How many do you? How many of you feel like your
• 3:09
  data is pristine and awesome and OK?
• 3:14
  Otherwise you wouldn't really be there's.
• 3:15
  Not a lot of hands like. Maybe OK, it's just just yeah,
• 3:19
  just checking. Thank you.
• 3:21
  Thank you for the anecdote show. I I want to branch off that
• 3:24
  actually because which said about ROI, I've worked on a lot
• 3:27
  of predictive projects where it was easy to measure that right.
• 3:31
  You have some regression score or something that it or like
• 3:33
  time to value or even in today's world time to first token where
• 3:37
  it's easy to say the faster that this gets to the user, the
• 3:40
  faster they get a recommendation, the higher ROI.
• 3:43
  But the infrastructure right is different in today's world,
• 3:47
  given that we are now with the generative kind of generation to
• 3:51
  be a little punny there. Pushkar, what do you think about
• 3:53
  that? Yeah.
• 3:55
  I think it's, it's first of all, most important to be able to
• 3:59
  bring all of the structured and unstructured data into one place
• 4:01
  like a data lake, lake house so that you can build some sort of
• 4:06
  monitoring on everything together.
• 4:10
  And then I, I see data quality in like 2 perspectives.
• 4:14
  One is the data, data world and then the second is the ML world.
• 4:18
  So all the ingestion transformation that that
• 4:20
  happens, I see it lying in the data world.
• 4:24
  And then once they are converted into features, how do we use to
• 4:29
  use them to build models and then do predictions that sort of
• 4:32
  like lies in the in the ML world in my perspective.
• 4:34
  So building like a data lake where we can get everything
• 4:41
  together and then defining a set of metrics that work well for
• 4:46
  the structure data and then some, some of them also work for
• 4:49
  the unstructured data, like data validity, data completeness,
• 4:54
  timeliness, all those kind of things, which, yeah, so I think
• 4:59
  that that's really important. Absolutely.
• 5:02
  I'm curious, does anybody here have a Horror Story about data
• 5:05
  quality that they can tell something where, I don't know,
• 5:09
  there was a mismatched scheme or something didn't work right or
• 5:13
  maybe a migration or something. All right, Chad, I see you over
• 5:16
  there. Come on, tell us.
• 5:17
  Yeah. I mean, I, I probably have, I, I
• 5:21
  could write pages worth of data quality issues that I've
• 5:23
  encountered in my life of all different varieties and and
• 5:26
  sorts. So I can tell too very quickly.
• 5:29
  So I used to work at a freight tech company called Convoy that
• 5:33
  unfortunately for the world no longer exists.
• 5:36
  Thank you COVID. But we are very heavily model
• 5:39
  driven and we had a couple models that were incredibly
• 5:44
  important to the business and one of them was our pricing
• 5:46
  model. And the way that the pricing
• 5:48
  model worked is that it would essentially based on data we
• 5:51
  were getting from our freight marketplace, which was an
• 5:54
  auction, it would predict a range.
• 5:56
  It would say this is what we think the ceiling of this
• 5:59
  particular shipment is going to go for in an auction and this is
• 6:01
  what we think the floor is going to be.
• 6:04
  And then depending on that range would say, well, you know is
• 6:06
  this within margin or sort of out of band for margin for us.
• 6:09
  So that model was really, really important.
• 6:12
  Otherwise we'd be bringing shipments into our marketplace
• 6:14
  and we'd actively lose money trying to fulfil them within the
• 6:19
  actual auction environment. And we had two really major data
• 6:22
  quality incidents within a period of about 3 months.
• 6:25
  One of them came from a very simple schema change where a
• 6:28
  junior software engineer didn't understand that a feature,
• 6:32
  ironically called is dropped was actually used as a feature
• 6:35
  within the model. They dropped it just because
• 6:37
  they're like, Oh well, our application's not using this is
• 6:40
  dropped column anymore and it blew up our entire data pipeline
• 6:43
  and nobody really knew exactly why for at least a few days
• 6:47
  until we did a root cause analysis.
• 6:49
  I swear I've made that mistake before.
• 6:51
  Everybody has, so many people have.
• 6:54
  And then, then the second is a is is a lot more subtle and I
• 6:57
  think more nefarious in some ways where we had a new feature
• 7:01
  that was being launched and this pricing model was trained on the
• 7:04
  data from our auction. And this behaviour within the
• 7:07
  auction was all human generated, right.
• 7:09
  So it was a person going in and making a bid and then adjusting
• 7:12
  their bid and then taking the shipment and so on and so forth.
• 7:16
  And a very smart brilliant product team decided to build a
• 7:20
  feature called auto bid. And auto bid meant all a a truck
• 7:24
  driver had to do was turn this feature on and a machine would
• 7:28
  incrementally bid down in increments of $5.10 dollars,
• 7:33
  $50.00. So they bid lower and lower,
• 7:35
  lower. And what that meant was after a
• 7:37
  certain period of time, we had thousands of machine generated
• 7:41
  bids in our marketplace and that was the data our pricing model
• 7:44
  was being trained on. So that was a multi $1,000,000
• 7:49
  incident that took place over a period of about three months.
• 7:53
  Wow, it's, it's interesting the power of compounding error like
• 7:56
  that when a generated error gets used again and how it compounds
• 7:59
  that's it's specifically something that I feel like a lot
• 8:02
  of people have run into with now the scope of generative models
• 8:06
  in their use cases today. Speaking of scope, I'm curious
• 8:09
  in, in enterprise, there's AI feel like a different set of
• 8:12
  requirements. There's a lot of PO CS going
• 8:15
  out, right? There's a lot of oh, I built
• 8:17
  this small rag app, but where I built this agent.
• 8:20
  But at the same time, there's not what I've seen at least a
• 8:24
  tonne or a plethora of enterprise ready use cases.
• 8:27
  So I'm curious, what about data quality gets you closer to that
• 8:32
  enterprise readiness? Maria, do you have an answer for
• 8:34
  that? Yeah, I'm afraid I don't have a
• 8:38
  perfect answer for that, but I do know that question.
• 8:42
  I think there is a gap that exists both in terms of the
• 8:47
  precise understanding of what does it mean and also from a
• 8:51
  compliance perspective, what is it compliant and even the
• 8:56
  definition of SLA, right? It's the first token arrival
• 9:01
  satisfied the SLA, right? And how about guard railing,
• 9:05
  hallucination, misalignment and so on and so forth, right?
• 9:09
  Is 99.9% error rate OK? Because human being makes errors
• 9:14
  too, right? We talk about agents, it's
• 9:18
  really hard to get to 100% accuracy rate.
• 9:21
  So combine all of these factors, I believe there will be a new
• 9:26
  definition of enterprise readiness and there will be a
• 9:31
  first wave of early adopters or companies that are savvy and
• 9:38
  that are contributing to, you know, this this new standard of
• 9:44
  procedure per Southeast. But personally, I'm really
• 9:47
  excited to see that happening and actually would love to hear
• 9:51
  your thoughts on, you know, what does it mean in the era of
• 9:54
  generative AI by enterprise ready?
• 9:58
  Enterprise is an interesting one.
• 10:00
  I wrote about this last week on my blog.
• 10:02
  I felt like there's there's really two types of data
• 10:05
  practises going on. There's there's product land
• 10:08
  where you're making, you're using data to make products,
• 10:10
  right? And there's also enterprise land
• 10:14
  where traditionally data is used in a back office IT function.
• 10:17
  And so I think when we're talking about enterprise, it's a
• 10:19
  fascinating 1 because it's, I mean, a lot of enterprises I
• 10:24
  see. And this is some of the biggest
• 10:25
  companies around to small enterprises and everything in
• 10:29
  between. But it feels like data is an
• 10:30
  absolute dumpster fire. And like people are companies
• 10:34
  are still barely able to do BI at this point, right?
• 10:37
  So the question I always ask when I go around the world is
• 10:40
  would you be willing to throw a large language model on top of
• 10:42
  your corporate data set as it exists today to get quote
• 10:45
  insights? And my second criteria is would
• 10:48
  you be willing to bet your job if this doesn't work?
• 10:50
  I think we asked us to let in Australia last year.
• 10:52
  Yeah, I think one person who is working on a large language
• 10:55
  model raised their hand and the others are like, no, I'm
• 10:57
  definitely not going to do that. So I mean, that's kind of the
• 10:59
  state of affairs. I mean, I, I've had this moniker
• 11:01
  for a while. I mean, a lot of companies can
• 11:03
  barely do BI, but now they, they, The thing is there's a lot
• 11:06
  of, there's a lot of motivation to want to throw AI at the
• 11:09
  problem. So I, I think there's a huge
• 11:11
  disconnect between, you know, sort of the incentives and the
• 11:14
  outcomes of what we're about to see.
• 11:17
  There's not a board or ACEO who, who isn't, who's going to admit
• 11:21
  like, Oh yeah, we're not going to go down AI right?
• 11:23
  You'd look like an idiot. Everybody wants to say in their
• 11:25
  board report, right? Right.
• 11:27
  At the same time, knowing what you all probably know about most
• 11:31
  data sets in companies, I I think you can make the the leap,
• 11:34
  right? It's generative AI.
• 11:35
  It's more like degenerative. AII guess would be a better way
• 11:38
  of putting it, but that's sort of the state where we are.
• 11:41
  I think it also applies to some subsets of predictive modelling
• 11:44
  as well. Oh, it always has.
• 11:45
  It always has. It's always been the data
• 11:46
  quality piece has always been there.
• 11:49
  It's just that now it feels almost like it's emphasised
• 11:52
  because of its impact, its outstanding impact.
• 11:54
  You know, I, I think it was Pushkar, I think it was Pushkar
• 11:57
  Chad who was talking about the differences in predictive versus
• 12:00
  generative and like how that's changing the definition of data
• 12:05
  quality and model quality. Like we talked about model
• 12:09
  quality being something maybe like system quality of time to
• 12:11
  1st token, but data quality being completely different.
• 12:15
  You want to talk about that? Yeah, sure.
• 12:18
  So when I see data quality for the predictive side and then the
• 12:23
  generative side, I think a lot of it applies in like the
• 12:29
  intersection of of both of them. But there are elements of
• 12:33
  predictive AI, the data being used to do predictions for
• 12:37
  predictive AI, those are like very, very different like
• 12:41
  emails, call transcripts, Those are some of the things that we
• 12:44
  look at, right? So how do you identify tone?
• 12:46
  How do you identify like some kind of action that is needed?
• 12:51
  So those kind of things I think are additional to what you would
• 12:56
  eventually, you know, look for when you're just looking at the
• 13:00
  generative, the the predictive side.
• 13:02
  Yeah, yeah. Like a comment on that is that
• 13:05
  I've seen a lot of people try to use LLMS to do things like query
• 13:08
  intent modelling. And I, I mean, I don't, I don't
• 13:10
  know about y'all, but like six years ago I was doing that with
• 13:12
  XG Boost. You know, you come up with four
• 13:14
  categories and it's just multi label classification, right?
• 13:17
  And so it seems like it's almost like we stuff too much towards
• 13:20
  the LLM. What do you think, Chad?
• 13:22
  Yeah, I, I mean, I, I think that that's, that's definitely true.
• 13:26
  I mean, we're seeing a lot of sort of products and vendors
• 13:29
  these days that are trying to do, you know, traditional sort
• 13:32
  of machine learning as if it was an AI function.
• 13:35
  And that's definitely not the right way.
• 13:37
  But to, to sort of push Car's earlier point, I, I think, and
• 13:41
  to Joe's earlier point as well, I think that there are so many
• 13:45
  companies that are trying to start thinking about investing
• 13:48
  heavily in AI and AI quality, but their ability to do data
• 13:53
  quality on relational data sets is, is horrible.
• 13:56
  And that is like way easier. It's 10 times easier than
• 14:01
  thinking about data quality for unstructured data.
• 14:04
  And there's a whole host of issues, even the most basic ones
• 14:07
  possible, like a schema changed for a data pipeline that I used
• 14:12
  and it broke me. And these data teams really have
• 14:15
  no way of preventing or being proactive or even knowing that
• 14:19
  these issues are coming, let alone more of the complicated
• 14:23
  data quality issues relating to like semantics.
• 14:26
  So something changes in the data and it doesn't have anything to
• 14:29
  do with the schema. Like, you know, we're updating a
• 14:32
  column called, you know, date time dot now to date time dot
• 14:37
  UTC, right? If you've worked with data
• 14:39
  significantly, like, you know that that's going to cause a
• 14:41
  massive issue for you. Always use Unix time right
• 14:44
  integer, integer, Unix time every time.
• 14:47
  Exactly. But but most companies don't
• 14:50
  even have a great way of understanding that these issues
• 14:53
  have happened, let alone preventing them.
• 14:55
  So there's sort of this whole murky territory of dealing with
• 14:59
  data quality even in structured data land.
• 15:03
  And I think trying to make that jump without getting your
• 15:05
  foundations in place is pretty, pretty dangerous.
• 15:08
  Joe, you talked about anti patterns.
• 15:09
  That sounds like one of them, you know, treating unstructured
• 15:12
  data like structured data. Do you have any other?
• 15:15
  You talked about some of those anti patterns.
• 15:17
  Do you want to number some of them for us?
• 15:19
  I. Think that's a good one.
• 15:25
  Where do you even begin here? I mean, a structured data I
• 15:30
  found is always the hardest data set to work with, right?
• 15:32
  Who's is anyone that felt the same way?
• 15:34
  Like it feels like unstructured data is usually easier, but then
• 15:37
  structured you're you're sort of held victim to your own
• 15:40
  circumstances of the data sets of Zach, how you doing?
• 15:44
  So yeah, other anti patterns. I would say obviously not having
• 15:49
  observability early and often on your data sets is a big one.
• 15:52
  Not knowing what you have is a big one.
• 15:53
  But but again, this is most companies.
• 15:56
  So now you're having to go retrofit all this stuff in and
• 15:58
  now you get to atone for your past sins.
• 16:00
  I would say also the other big cardinal sins so that Chad can
• 16:03
  speak to is just not addressing your your issues as far upstream
• 16:07
  as possible, right. That's a huge anti pattern.
• 16:10
  I think all too often right now we're sort of on the receiving
• 16:12
  end of it, but you're the receiving end of a sewer.
• 16:15
  So you can try and clean your water there or you can make sure
• 16:19
  things are better upstream, but that's your call, not mine.
• 16:23
  So hey, what do? You think?
• 16:25
  Yeah, I think I'm a little bit more optimistic that my fellow
• 16:30
  panellists here. I actually think, you know,
• 16:34
  we've the entire industry has gone through digitization.
• 16:37
  So I do think the data is, you know, stored somewhere, and
• 16:41
  that's a good starting point. One thing I've noticed
• 16:45
  repeatedly that people overlook or misunderstand the lack of
• 16:51
  precise understanding of the data.
• 16:54
  And that is so critical. Everything starts there.
• 16:58
  So often people just lump it together is like, this is my
• 17:03
  data set or structure data versus unstructured data.
• 17:06
  But there's just so much more to it.
• 17:08
  My suggestion is really to take a much closer and much more
• 17:12
  granular look at your data set, right?
• 17:15
  What's dynamic, what's factual, What's slow changing?
• 17:19
  What's human labelled with subjectivity?
• 17:22
  What is factual? When you have a more precise
• 17:25
  understanding of all this data at a column rows level, then you
• 17:30
  need to give them the the proper treatment and and the treatment
• 17:33
  is going to look very different, right.
• 17:34
  So it's the classic divide and conquer.
• 17:37
  If you put everything together, it's going to look really messy
• 17:40
  and you feel like you can never get it right, right.
• 17:43
  But if you look at OK, this is my dynamic data, I'm going to
• 17:46
  measure latency, right? I'm going to measure, you know,
• 17:49
  completion. I can't make it synchronous,
• 17:52
  right so on and so forth. And you know you're good there.
• 17:54
  And if this is slow changing factual type of data, I don't
• 17:58
  need to worry too much about it, right?
• 18:00
  Then I measure change. It shouldn't be changing too
• 18:02
  much. If you change drastically,
• 18:03
  something went wrong. Yeah, so I see over, over again,
• 18:07
  people overlook and try to kind of summarise it, like just put
• 18:11
  all of these issues into one giant basket and then they feel
• 18:15
  overwhelmed to come up with the right solution.
• 18:17
  So take a closer look and really break it down.
• 18:21
  Yeah. Actually, just to sort of
• 18:23
  comment on that because I think that's super relevant.
• 18:26
  This is a, this is a metadata problem, right?
• 18:29
  And it's a metadata problem that's existed in data for the
• 18:31
  past 50 years. You know, cataloguing is a huge
• 18:35
  component of data management. Like what is my data?
• 18:38
  Where is it from? What does it mean?
• 18:41
  What are the semantics? Who are using it?
• 18:43
  How is it changing over time? There are whole companies built
• 18:45
  on just storing the metadata for your Snowflake database.
• 18:48
  There are a whole companies built only around this.
• 18:50
  And so I think applying it in the context of unstructured
• 18:53
  data, structured data AI, like it's a thoughtful exercise that
• 18:57
  folks need to undertake. One of the activities that data
• 19:00
  scientists spend the most time on relative to anything else is
• 19:04
  just figuring out what the data means and where it comes from.
• 19:07
  And if you don't have sort of an automated system in place to
• 19:10
  help you with that, then ultimately you end up relying on
• 19:13
  the goodwill of whoever the producers of the of the data are
• 19:17
  to tell you accurately what this information means.
• 19:20
  And what I find is that there's an enormous amount of
• 19:23
  institutional knowledge that gets embedded in the engineers
• 19:26
  that are responsible for creating these data sets the
• 19:29
  data scientists depend on. And if you don't do a good job
• 19:31
  documenting that institutional knowledge, then like you're
• 19:35
  asking to to be subject to data quality issues in the future.
• 19:38
  Yeah, that's that's really interesting.
• 19:40
  We talked about two getting data scientists and machine learning
• 19:43
  engineers that you know as the canonical sense of the two
• 19:46
  terms, like getting them to work closer together and how it's
• 19:51
  more important that the people creating the systems and the
• 19:54
  people that are creating the data and the people that are
• 19:56
  cleaning the data all work together.
• 19:58
  Do y'all have any like tips or practises or methodologies or
• 20:02
  anything that you know of that gets that group of people to
• 20:06
  work Better Together? Yeah, I can take that.
• 20:11
  So what I have seen is like implementing things at the
• 20:16
  platform level is is sort of always the best way implement as
• 20:21
  much as you can at the platform level and then make frameworks
• 20:24
  available. So like data quality in the data
• 20:26
  pipelines, right. How do we implement that?
• 20:28
  So bake in operators in, in Airflow or whatever your
• 20:31
  orchestration tool is like Great Expectations or any other data
• 20:36
  quality related tooling and, and then make those available
• 20:40
  generally to to the data scientists so that they can,
• 20:45
  when they're building those, their data pipelines, they're
• 20:47
  able to utilise those and then make their lives easier because
• 20:51
  they don't really like working with Terraform and other like,
• 20:55
  you know, tools like that. So making their lives easier is
• 21:00
  something that I think we should go by.
• 21:03
  I'm talking from platform ML, platform person's perspective,
• 21:09
  yeah. I If you see a data scientist
• 21:13
  deploying Terraform, you let me know because I want to hire
• 21:16
  them. That person is multifaceted.
• 21:21
  Chad, do you want to go? You have some, Yeah, sure.
• 21:24
  I think, I think there's a lot of things that you can do to
• 21:29
  better improve sort of the relationship between the
• 21:31
  producers of data and the consumers of data.
• 21:35
  The sort of foundational, the model that I use is that data is
• 21:38
  very similar to a supply chain. And in a supply chain, you've
• 21:42
  got a producer and the producer is generating some raw input and
• 21:46
  then that input is transformed many times.
• 21:48
  And then at the very end of the process, you have a product.
• 21:51
  And I think that's true for the data pipeline as well.
• 21:54
  You have a raw data source, you have some, you know, ETLELT
• 21:58
  mechanism, multiple steps of transformation.
• 22:00
  And then at the end you have a data product and AI might be one
• 22:03
  of those data products. And in the supply chain world,
• 22:05
  what's super important is this idea of end to end supply chain
• 22:09
  visibility so that each person who's responsible for a
• 22:13
  transformation understands how that transformation is going to
• 22:16
  impact the owners of the data products downstream.
• 22:19
  If you don't have that visibility in the awareness,
• 22:21
  you're operating in a black box, right?
• 22:23
  You make a change, you might end up blowing up someone who's
• 22:26
  running an AI system, or maybe not, and it's totally fine.
• 22:29
  It's more or less impossible to know.
• 22:31
  And so one of the technologies I'm a huge advocate of is data
• 22:35
  contracts. The data contract is basically
• 22:38
  starting to apply an API to your data.
• 22:41
  And that's not like a literal API.
• 22:43
  It's an API in the sense of like.
• 22:44
  There are certain expectations and Slas that you have for your
• 22:48
  data assets. You always expect the schema to
• 22:50
  look a certain way. You expect the contents of the
• 22:52
  data to look a certain way. There's data quality rules that
• 22:55
  need to be enforced. We would always expect like 1000
• 22:58
  events to be omitted over this time period and never five.
• 23:02
  And the goal is to have that check be as close to the
• 23:05
  producer as humanly possible. So if the failure happens, you
• 23:08
  can communicate that to all the other contracts downstream.
• 23:12
  And now your data supply chain is connected by contracts and
• 23:14
  not these disconnected people. So that's what I recommend.
• 23:17
  And it's something that some of the larger companies in the
• 23:19
  world are starting to implement across their entire data
• 23:23
  ecosystem, and it's very effective.
• 23:24
  Joe, what do you think about that supply chain?
• 23:27
  I like supply chain. Supply chains are good.
• 23:29
  It's pretty cool. I think also to solve the
• 23:32
  problem of making everyone work together, like trust falls and
• 23:40
  group activities are good. I'm sort of kidding, but not.
• 23:43
  Yeah, the supply chain thinking is interesting.
• 23:45
  I actually come from a supply chain background.
• 23:47
  My background's in Lean. So it's interesting to see how
• 23:50
  many principles from Lean are actually adopted and you know,
• 23:53
  starting in software now and data now AI like this is an OPS
• 23:56
  conference, right? I mean, that gets its, its roots
• 23:58
  and lean. And so I think if you, if you
• 24:00
  actually a good thing to go study, if you all really want to
• 24:03
  read a great tome is the goal. It's a supply chain book, I
• 24:08
  think written in the late 80s or 90s, actually the 80s.
• 24:11
  It was it still holds true today.
• 24:13
  I would say that this is an awesome place to start.
• 24:16
  That'll give you a lot of the reference you need.
• 24:17
  But it's about continuous flow, reducing bottlenecks, right?
• 24:21
  Collaborating with people, single piece workflow.
• 24:23
  I think if, if the industry can master these principles, it
• 24:26
  actually goes a long way towards reducing defects.
• 24:28
  The way you get into errors really is the way we do it now,
• 24:30
  which is you lob things over walls, right?
• 24:32
  And, and supply chain. This is this is the old way of
• 24:35
  doing stuff. This is mass production, right?
• 24:38
  Accumulate a bunch of stuff in batch, which is anti pattern and
• 24:41
  lean. But this also results.
• 24:42
  And now you have to go and rework 1000 possible things
• 24:45
  versus one where you can catch the error and fix it.
• 24:47
  So I think adopting those principles would go a long ways
• 24:50
  towards, you know, I think fixing data quality, which, you
• 24:53
  know, people like Chad are working on and, and, and others,
• 24:55
  but it, it's, it's really a process question more than
• 24:58
  anything. If you could read, go back and
• 25:00
  read supply chain, like Deming, for example, in the 60s, he
• 25:02
  already called out that 98% of the problems are usually process
• 25:05
  related. It's not people.
• 25:07
  It's just that was not on my bingo card but I loved that.
• 25:11
  I'm not sure who had that on their bingo card.
• 25:12
  Supply chain tome this is what. Happens when you.
• 25:14
  Invite you to What do you think, Maria?
• 25:18
  Yeah, I agree with everything you said.
• 25:21
  But there is a human element and that's what I call holistic
• 25:25
  ownership, right. Even I work at some of the
• 25:27
  largest companies and of course there are different departments,
• 25:31
  different functions. And my philosophy it was always
• 25:37
  be really take ownership of it and not toss it over the wall
• 25:41
  and feel like, oh, I'm done with my job.
• 25:43
  Let me give you a simple example.
• 25:45
  So of course, developers need to log right activities,
• 25:48
  consumption, metering, so forth, right?
• 25:51
  And eventually it becomes billing to customers.
• 25:54
  And and if there was a mistake, either you overcharge your
• 25:58
  customer, which is not nice. And when they find out they're
• 26:01
  furious. And two is you, you under bill
• 26:04
  your customers and there's, you know, massive revenue loss,
• 26:07
  right? So I actually got these two
• 26:09
  teams together. They actually never met, right.
• 26:11
  On this side, you're creating these logs and supply chain.
• 26:16
  Five points later, dollars came out of there and I could just
• 26:19
  see and there were issues. Of course, it's never perfect.
• 26:22
  And the developer fell so bad, right?
• 26:25
  That because it was, you know, and then I feel like he's taking
• 26:29
  his job from a different lens. Like the work I do like every
• 26:32
  piece of data because if sometimes it feels meaningless,
• 26:35
  right? I'm logging all these things.
• 26:37
  We're talking about billions and billions of events on a minute
• 26:39
  business, right? They're just like, whatever,
• 26:41
  right? And now he's like, OK, I'm going
• 26:43
  to take my job seriously. And as I start talking, start
• 26:46
  communicating, do a monthly review together, do error
• 26:49
  injection together. Like so really have the sense of
• 26:52
  holistic ownership and really understand where your data is
• 26:56
  going to deliver and, and end to end.
• 26:59
  Yeah. Well, I think that just about
• 27:03
  does our time, so I just want everybody to thank these awesome
• 27:05
  panellists for coming and sharing their expertise.