Latest / Agentic Conversations (formally mlops.community) / Cleanlab: Labeled Datasets that Correct Themselves Automatically // Curtis Northcutt // MLOps Coffee Sessions #105

Transcript

Cleanlab: Labeled Datasets that Correct Themselves Automatically // Curtis Northcutt // MLOps Coffee Sessions #105 · Agentic Conversations (formally mlops.community)

• 0:00
  My style is whatever coffee is available.
• 0:02
  If I'm tired and I need to be awake.
• 0:06
  Thank you so much for joining us.
• 0:08
  This is an episode. We've been doing waiting to do
• 0:10
  for a while and we're very excited to have you.
• 0:13
  Join us on the MLS podcast? Thanks.
• 0:16
  Yeah, glad to be here. You know, I think the first
• 0:19
  place I want to start and Demetrios.
• 0:21
  I want to get you going on this too because I know you have a
• 0:24
  lot of thoughts on data-centric. AI is, you know, I know you
• 0:29
  gotta A lot of thoughts. My first question is, what is
• 0:33
  the difference between ml Ops and data-centric a, I like just
• 0:37
  starting right there like what is the difference between those
• 0:40
  two terms just so we can start this conversation from a clear
• 0:43
  Point? Yeah well I think the terms are
• 0:45
  still being, you know made up but I'll give you my take and
• 0:49
  and based a lot on where we are today clean them.
• 0:52
  So I think of ml Ops is is what you expect anything that's in ml
• 0:56
  operations and that can involve the model, right?
• 0:58
  Because in ml Have both the model and the data.
• 1:01
  So I think of ml Ops is a superset.
• 1:04
  Also, data-centric is kind of a different thing.
• 1:06
  Mmm, Ops is more of like an industry standard type term.
• 1:09
  There's some papers and Academia now that'll reference ml Ops and
• 1:12
  I think there was just a paper that came out that was like
• 1:14
  here's you know, the landscape of ml ups but that's kind of new
• 1:18
  and the the academic realm. Most of data Centric, AI is
• 1:22
  going to be more on that academic realm and Industry.
• 1:24
  And I think that's because folks, like, Andrew being are
• 1:26
  pushing it and they're both in Academia and industry.
• 1:29
  But, Animal opposites, like more industry pushing it.
• 1:32
  So you're going to get different flavors there.
• 1:35
  Another big thing in danger, Datacenter Ki, that's yet to be
• 1:38
  done, but speaking more to the Future, because I think the
• 1:40
  future will determine how these two terms separate themselves
• 1:43
  out, isn't data-centric AI. There's a lot of work yet to be
• 1:46
  done to show. Theoretically, why is using data
• 1:49
  in some circumstances and focusing on improving data,
• 1:52
  actually? A more effective solution for
• 1:54
  approaching some machine learning problems.
• 1:56
  And I'm happy to chat more about that.
• 1:58
  But one example, that may surprise People is that you can
• 2:00
  actually show for doing machine learning with noisy data and
• 2:04
  noisy labels. You can actually show that
• 2:06
  several methods that work by altering the data but not the
• 2:10
  model actually outperformed methods where you alter the
• 2:13
  model and happy to chat more about that.
• 2:15
  But there's a bunch of I would think of just data Centric, AI
• 2:18
  is more of, how do you fix up the data in order to improve the
• 2:20
  AI Pipeline and then m/l offices?
• 2:22
  You can do anything. You can change the model.
• 2:24
  You can have better infrastructure better,
• 2:26
  paralyzation, Etc. That's definitely the clearest
• 2:29
  answer I've gotten to date. What do you think Demetrius?
• 2:32
  How am I supposed to follow that one?
• 2:34
  That was so good. I was just going to talk some
• 2:36
  shit on people saying like, oh well data Centric AI.
• 2:40
  It's the new marketing buzz word that you hear all about because
• 2:43
  people want to talk about how it falls in line with this idea of
• 2:49
  data scientist spend too much time, cleaning their data, they
• 2:52
  need to look at the data more. We need to focus on our data
• 2:54
  more and it for me data-centric a Just one of those things that
• 2:59
  is more of a buzzword, but when I hear Curtis say that I I'm
• 3:06
  like, yeah I go along with data-centric a I know.
• 3:10
  All right. Yeah, I can get down with that.
• 3:12
  It sounds really good when you put it like that.
• 3:16
  What what I like about what you just said there Curtis is I
• 3:21
  think it's this idea of data Centric.
• 3:23
  AI is really really pleasure as your words is about How data
• 3:28
  solves machine, learning problems, right.
• 3:31
  Not just thinking of it as like an incidental, sort of component
• 3:35
  of a machine-learning pipeline which is been the way that I
• 3:38
  think a lot of work maybe from like 2015 2020 really thought
• 3:41
  about data. But now thinking about it as
• 3:43
  like very much a starting point for the solution and you know
• 3:47
  that has coincided with, you know, in the ml Ops World, a lot
• 3:52
  of attention to data pipelining how to setup data pipelines, and
• 3:57
  how to set them. Up in a way that, you know, kind
• 4:01
  of respects the quality of data and focuses on that.
• 4:04
  And so, starting with that point, what I'd like to ask is,
• 4:09
  how did you realize this problem of data, quality in machine
• 4:13
  learning manifesting, what was your personal experience or your
• 4:16
  War story that made you want to get into this realm around data
• 4:20
  quality? Yeah.
• 4:22
  For sure. A quick tidbit on your first,
• 4:25
  your first comment and just one way for folks who are new to
• 4:28
  data-centric AI to think about the field and how to think about
• 4:31
  what it would like, how do I get started?
• 4:33
  Like, what is data-centric AI, what are we even talking about?
• 4:36
  Just to get started you can just think about something really
• 4:38
  simple like a mirror its neighbors.
• 4:40
  So if you're from the ml space, if you've ever done k-nearest
• 4:43
  neighbors that's really just data-centric AI.
• 4:45
  I mean there's no there's no actual like big model, right?
• 4:48
  And you have to store some representation of distances
• 4:50
  between to make things more. But you could also not you can
• 4:53
  literally just have a data set and and that's all you've got.
• 4:56
  And then you have some new point and you're just like, what's the
• 4:58
  data near me? What's the label of that predict
• 5:00
  that? And that's like a very
• 5:01
  data-centric approach. So just to sort of help ground,
• 5:06
  you know, how you can think about how the two are similar
• 5:08
  and different with that, in mind.
• 5:11
  One thing to like jump in real, fast is with data-centric a i
• 5:16
  for some reason in my head it was always associated with like
• 5:21
  her vision problems. Am I the only one who thinks
• 5:24
  that or is that something that is precedent?
• 5:29
  Like is it not more a computer vision thing or does
• 5:33
  data-centric a I cover all of it?
• 5:35
  Because I guess it does and especially the way that you're
• 5:38
  explaining it, it totally does. Yeah, covers all of data.
• 5:43
  If it's data, it's it can fit in data-centric Ai and a one-way
• 5:48
  Under the Umbrella of the day. Yeah.
• 5:52
  Okay, okay, let me don't let me derail you from business.
• 5:55
  Great question, with my ridiculous ones.
• 5:58
  Go ahead. Yeah, I'll say one thing and
• 6:01
  then maybe just a reminder of the question both for myself and
• 6:04
  for the audience, but the sort of one final important way to
• 6:07
  think, I think, at least for me what helps me is there is no
• 6:10
  learning. Without some way to learn and
• 6:13
  something to learn. So there is no machine Learning
• 6:17
  Without both a model and data, there is no human Learning
• 6:20
  Without, You know, a brain and some information to learn from.
• 6:24
  So all learning exists in the contest, context of both, a
• 6:27
  learning model and also data. And so you know that the the
• 6:30
  both fields are going to be highly important to the field of
• 6:33
  machine learning. I think we just got our quote
• 6:36
  Demetrios. There's a the quote for this
• 6:39
  episode of you already coming in hot Curtis, there is no way to
• 6:42
  learn without something to learn and some way to learn.
• 6:46
  And I think that's a great summary of what we do in machine
• 6:50
  learning overall, is really think about those two pieces to
• 6:54
  go back to my question, it was really about what experience or
• 6:59
  War Story. Got you into this field of data
• 7:03
  quality testing, Eccentric. AI was it that made?
• 7:07
  You want to focus more on that? Yeah, you bet.
• 7:12
  So from the beginning so that this immediate team short answer
• 7:16
  is I built the cheating detection system for MIT and
• 7:20
  Harvard in particular for their online courses.
• 7:23
  You can look it up, it's called The Cameo see améo Cameo
• 7:27
  cheating detection system and we observe something and then what
• 7:31
  we observed was that people on edx courses were creating
• 7:34
  multiple People accounts. And on one account, they were
• 7:38
  just clicking show answer. And then on another account,
• 7:41
  they were clicking, you know, submit answer and they were
• 7:44
  doing it on these two accounts with the same IP address within,
• 7:47
  like 3, S 3 minutes of each other sometimes, within three
• 7:50
  seconds of each other, and they were doing it over and over and
• 7:53
  over and earning certificates. And so, when I first started at
• 7:57
  MIT, I'll rewind, how did we get there?
• 7:59
  So why was I building that? Why, is that okay?
• 8:02
  You know, who cares? So, I'm from Earl Kentucky.
• 8:05
  I grew up in Kentucky. My dad was a mailman.
• 8:08
  His dad was a mailman, his dad was a mailman.
• 8:12
  So if I had been, you know, a mailman, I would be a fourth
• 8:15
  generation mailman. My mom is, she works in a call
• 8:18
  center and don't imagine like a big farm.
• 8:21
  Imagine you know what, you got, what you might expect from rural
• 8:24
  Kentucky, like, you know, small Stone House at its problems and
• 8:28
  all that. And so this is sort of how I
• 8:30
  grew up and there wasn't a lot of opportunity.
• 8:32
  But one way that I created a lot of opportunity and the US was we
• 8:37
  have the best educational system at least how should, I know, I
• 8:40
  didn't grow up anywhere else, right, but I thought it was
• 8:42
  awesome because I have a really nice life today and I didn't
• 8:44
  when I was growing up and the way I did that was just to do
• 8:47
  well in school. So just imagine that motivation,
• 8:49
  that's where I'm coming from. Okay.
• 8:51
  And now I, finally get to MIT and it's been, you know, this
• 8:54
  big exciting Journey getting there and every day was better
• 8:57
  than the day before and there's always more opportunity as more
• 8:59
  getting more and more exciting. And finally I know what I'm
• 9:01
  going to do for my PhD I'm going to use machine learning and AI
• 9:04
  to help people. And education and most
• 9:06
  importantly to democratize education to make it available
• 9:09
  to everyone in the world by being the first researcher at
• 9:12
  edx. And that's what I was.
• 9:14
  I was at MIT and I was working with Isaac, shuang and Sanjay
• 9:17
  Sarma. He's the vice-president of open
• 9:19
  learning MIT and Isaac strong, was the senior associate
• 9:22
  director of of edx and open learning.
• 9:25
  And so I'm working with these guys, and how do we make
• 9:27
  certificates? How do we first?
• 9:29
  Get them to be accepted by the world.
• 9:31
  So everybody in the world, can, you know, get an education from
• 9:34
  MIT or Harvard? Word even if you're from you
• 9:36
  know some small place in Africa if as long as you have an
• 9:39
  internet connection so that was the goal and then you find out
• 9:42
  you make a discovery you're going through all the tracking
• 9:44
  logs of you know literally millions of students and you
• 9:47
  find that people are just cheating to get these
• 9:49
  certificates and then you find out that they're taking a
• 9:51
  certificate that they cheated on.
• 9:53
  And and it says, now that they have seven certificates from MIT
• 9:57
  and they go in, they get a job and then they, they can't do the
• 10:01
  job because they cheated to earn the certificates.
• 10:03
  And now someone else who's from, where I'm from I'm safe from
• 10:05
  rural, Kentucky or safe from, you know some poor place in
• 10:08
  Africa, they apply for the same job, you know, six months later
• 10:11
  and they don't get it and they don't get it because even though
• 10:14
  they actually earn those seven certificates, someone else
• 10:16
  cheated and ruined it for them. And that to me was like the
• 10:19
  worst thing ever, like, think about my background and how much
• 10:21
  opportunity means to me, and we're trying to create this.
• 10:23
  And now, it's you just ruined that for someone.
• 10:25
  And so I was very, very motivated to solve this problem
• 10:28
  of cheating detection. Now, what does that have to do
• 10:30
  with your question? What has to do with your
• 10:31
  question? Is that cheating detection is
• 10:33
  literally just a noisy label. It's literally just an error and
• 10:36
  data. If you think about data, as you
• 10:38
  have a bunch of certificates, the question is, which of those
• 10:41
  certificates are the errors, which of those certificates, are
• 10:44
  the ones where someone cheated. So, first, I wrote some very
• 10:47
  simple pipelines, like filters. Like I mentioned, same IP
• 10:49
  address, you know, a bunch of filters to get data, right?
• 10:53
  But my data was noisy, I missed people.
• 10:55
  And I had lots of false negatives people.
• 10:57
  I just didn't get because I had very tight thresholds.
• 10:59
  We don't want to accuse someone of cheating if they didn't
• 11:01
  cheat. Right.
• 11:02
  That's a very bad thing. So I had to have tons of false
• 11:04
  negatives. I think people who I just wasn't
• 11:06
  catching. So then I was like, okay how do
• 11:08
  we solve this? Well, I was a PhD student in
• 11:10
  machine learning so I naturally I'm going to try to take Ai and
• 11:13
  machine learning approaches. And at this time, it was 2014
• 11:16
  and there was no multi-class solution for machine learning in
• 11:19
  noise with noisy. Labels, it just didn't exist.
• 11:22
  And so I start applying machine learning, algorithms and
• 11:24
  training them on this noisy data and nothing worked.
• 11:27
  And I there literally wasn't a solution.
• 11:29
  And so this was very frustrating because I have this problem, I'm
• 11:31
  very motivated to solve and I've, you know, shared why it's
• 11:34
  really important. Important for the world and I
• 11:35
  just can't solve it with machine learning and a sudden I had a
• 11:38
  big realization and the realization was if you want to
• 11:41
  actually train AI machine learning models on really messy
• 11:44
  noisy. They labeled data, especially
• 11:46
  for human-centric problems problems, where you really want
• 11:49
  to help people. And it's for people people's
• 11:51
  data is messy and people's data is really noisy.
• 11:54
  And so, we needed a framework in machine learning to be able to
• 11:57
  deal with messy and noisy data. And I spent the next six years
• 12:00
  doing it and I can share more about where what we did at
• 12:03
  various companies. Like Google and Amazon along the
• 12:06
  way before clean life existed? Yes, please.
• 12:10
  How was what? What a story man?
• 12:13
  Yeah, I know bringing it all the way from Kentucky to edx do now
• 12:19
  like, what Google and Amazon are doing.
• 12:22
  I love that. Like, keep it coming.
• 12:24
  I just I'm just gonna sit here and listen, even I I really
• 12:30
  appreciate you. Taking us through your story.
• 12:31
  There, the one, you know, Demetrius what, this kind of
• 12:34
  reminds As me of is that Cody Cody Coleman's episode that we
• 12:37
  did like a couple months ago, it was it's like very similar in
• 12:41
  the sense that you guys have these amazing personal stories
• 12:44
  that you channeled into motivation that then turns into
• 12:47
  solutions to problems that a lot of people face.
• 12:50
  So it's obvious what you want to know, something crazy you cry
• 12:54
  just with Cody. I just I just went on a walk
• 12:57
  with him two days ago. No way.
• 12:59
  All right. Yeah.
• 13:02
  He's one of my best friend's. He's absolutely.
• 13:05
  Fantastic guy. Yes of course this shit up.
• 13:08
  It's like, you know. Yeah, look about her movie until
• 13:10
  you can't believe it, right? We were roommates at MIT.
• 13:14
  We were also lab mates at MIT and he's one of my best friends
• 13:17
  from MIT, amazing gadgets that not surprise me.
• 13:20
  Why does that? It doesn't surprise me at all.
• 13:23
  And I'm going to get the, I want to option the movie script for
• 13:26
  this. I want to do when you guys, when
• 13:28
  you guys to sell both of your companies for billions of
• 13:31
  dollars or merge them or whatever.
• 13:32
  I'm going to option that movie script and Just it's going to be
• 13:36
  FYI. Curtis Vishnu talks about
• 13:38
  writing blog posts and he hasn't gotten around to doing it so
• 13:42
  optioning scripts. I mean that is it's now it's a
• 13:47
  lot but we'll see. Well I'll believe it when I see
• 13:49
  it but anyway you were talking about how you saw through and so
• 13:55
  this like you had the vision and then basically what happened
• 14:00
  from over those last six years. Yeah what?
• 14:03
  Yes What did that look like? Yeah.
• 14:07
  Totally and Vishnu is that work for you, too?
• 14:10
  I saw you, you looked ready to let something, okay.
• 14:14
  Let's hear all your stories, your take on the last six years.
• 14:16
  And how you went from people cheating at Harvard on this test
• 14:20
  and noisy labels to where you are now.
• 14:22
  I like, is that people cheating it at Harvard?
• 14:25
  They also cheated else elsewhere but I don't mind how that was
• 14:29
  praised and I say I say that as a I say that as a someone with
• 14:33
  MIT Roots were always, you know, having a good time in each other
• 14:36
  down the street. There, lots of great, great
• 14:39
  people at both places, but anyway, so so yeah.
• 14:44
  So so the first problem was, how do you detect false, negatives
• 14:49
  and data? So that's something called pu
• 14:51
  learning and that stands for Positive unlabeled learning and
• 14:55
  what the idea of that is, you you have data and some of it is
• 14:59
  positive, positively labeled. So call that someone say and a
• 15:04
  cheating detection data set, that would be someone who
• 15:06
  cheated. So, you know, for sure somebody
• 15:08
  is a positive but you don't know for sure that someone didn't
• 15:12
  cheat and that's actually, if you think about our Judicial
• 15:14
  System that's how it works, right?
• 15:16
  If you know, if you're on the stand, your assumed innocent
• 15:19
  until proven guilty, the idea is that you never know for sure.
• 15:22
  Sure that someone didn't do something.
• 15:25
  So we can't assume that they did something.
• 15:28
  We assume that they didn't do something.
• 15:29
  And that's actually, if you think about it, a very logical
• 15:32
  reason, for why we assume innocence until guilt, you have
• 15:35
  to provide enough proof to substantiate that.
• 15:38
  Someone is indeed guilty and so, you can only do that in One
• 15:41
  Direction. And the idea of that is, you
• 15:43
  never really truly have the labels of Innocence, which is
• 15:47
  kind of interesting to think thing to think about, especially
• 15:50
  for people who work in machine learning in the legal space, He
• 15:52
  difficult to train a machine learning algorithm to know if
• 15:55
  someone's guilty or not. That's why we don't, we haven't
• 15:57
  replaced judges and it's unlikely that we replace the
• 16:00
  Supreme Court with a i anytime soon.
• 16:02
  And and this is a very logical explanation for why that's
• 16:05
  difficult but given that. So we'll the first thing that we
• 16:08
  did is we had to solve this P, learn from P learning problem
• 16:11
  and that requires estimating a single noise rate, what is the
• 16:14
  probability of a false negative? And that's just one noise rate.
• 16:16
  There's another noise rate, the probability of a false positive
• 16:19
  and so we then generalized our solution there.
• 16:22
  To the full binary case of identifying false negatives and
• 16:26
  false positives just for binary multi classification and I'm
• 16:30
  presented some of that stuff at you.
• 16:31
  I the papers call drink pruning, and you can check it out.
• 16:33
  It's pretty theoretical it was early days of grad school.
• 16:36
  And I had to, you know, go pretty deep in the math, but I
• 16:38
  think it's, well, formulated for people who are in the field
• 16:42
  Beyond then, it was a similar time to that when I went to
• 16:45
  Facebook and I was working in Ian, lagoons group, folks, might
• 16:49
  recognize him. I saw him like four times.
• 16:53
  I barely worked with them, I think on paper, he was my
• 16:55
  manager. But my real manager was Elon
• 16:57
  burrow and absolutely fantastic person in the early days.
• 17:01
  She was one of the first female research scientists at fair and
• 17:05
  this is Facebook are research not Facebook engineering and we
• 17:09
  were in the New York office and they had a really interesting
• 17:12
  problem. The problem was, how do you
• 17:15
  avoid bias in up and down votes on comment rankings?
• 17:20
  So if you have imagined that Donald um, post something
• 17:24
  that's, you know, highly offensive.
• 17:27
  But because you're you're affiliated with a similar
• 17:31
  political. You say, you're a republican,
• 17:33
  you might want to upload it just because like, you want to
• 17:35
  support your party, which is a very reasonable thing to do.
• 17:38
  Even though the actual content of the message might be
• 17:41
  offensive. So, how do you that's an issue,
• 17:43
  right? Because now, you have people who
• 17:44
  are upvoting content, that's actually not content that they
• 17:47
  would have vote if not for some bias.
• 17:50
  And so what we ended up doing is, is Of it almost as a label
• 17:54
  error. Right?
• 17:55
  So how do you detect these sort of biases?
• 17:57
  And I started to realize, like, whoa, I mean, this is what I'm
• 17:59
  doing and cheating detection but now we have a gigantic
• 18:02
  corporation that needs to do this on petabytes of data, like
• 18:06
  literally trillions of data points at scale and there's no
• 18:08
  solution for this. So what do they do?
• 18:10
  They hired an MIT PhD student to do an internship on it, right?
• 18:15
  And I was like, wait a minute, this is, you know, this is what
• 18:16
  I'm doing like as my as my PhD, but this is clearly a problem
• 18:20
  that needs to be solved. Well, the next summer I interned
• 18:22
  Ended up going to Amazon. And I did the exact same thing
• 18:25
  at Amazon. They had an issue with the Alexa
• 18:27
  devices where they have no idea when they wake up, or when they
• 18:31
  don't wake up. So how would you know?
• 18:32
  For example, if your say your Amazon and you've got your Alexa
• 18:36
  dice, a devices all over the world, like, hundreds of
• 18:38
  millions of them, and people are saying, you know, the keyword to
• 18:41
  get it to wake up and they just don't wake up because they don't
• 18:43
  hear the keyword. Then how would you ever get that
• 18:45
  data? Amazon doesn't have that data.
• 18:48
  They don't have it. So how do they know what their
• 18:51
  device is false? Rates are they have no way to
• 18:54
  estimate that and and so it doesn't make does the problem
• 18:58
  make sense. It makes total sense.
• 18:59
  And I'm just, I'm just literally sitting there being like that
• 19:02
  literally sounds like an impossible problem because
• 19:04
  there's no way you'd know something didn't wake up,
• 19:06
  because there's no way you'd get the log from the device.
• 19:10
  So, how did you do that? Like, is there such a thing?
• 19:13
  Like, what do you do? We solve that problem using
• 19:16
  clean lab. So this is very early days of
• 19:18
  clean lab. And at this point, I had
• 19:19
  starting, I started building out the open source repo.
• 19:21
  Fun fact the The only reason why I originally did clean lab is
• 19:24
  open source. I think open source has a great
• 19:26
  model and obviously I have a business around it.
• 19:28
  So I clearly really love open-source.
• 19:30
  I think it's a good idea. But the initial reason I got
• 19:33
  into open source was not because I was like a soup in super open
• 19:36
  source Guru. I actually knew very little
• 19:37
  about it at the time I got into it because people didn't believe
• 19:40
  that you could actually just have data find errors and
• 19:43
  itself. Like the idea of that seemed
• 19:45
  like black magic to people. So how do you convince people?
• 19:48
  You make it runnable in one line of Code online and then people
• 19:51
  start to believe you. You see, that's what that's why
• 19:54
  I did it. Literally people were just like,
• 19:55
  what is this nonsense? Like, I've never heard of this,
• 19:57
  it's a new field. How are you just going to invent
• 19:59
  a new field? As a grad student, like that's
• 20:01
  total nonsense and I got a lot of pushback.
• 20:03
  And so we started just releasing everything online and then the
• 20:06
  push back, you know, quieted and we got a lot different reaction.
• 20:10
  So the reason I share that is because I'm building out this
• 20:12
  open source package. Meanwhile, I was doing an
• 20:15
  internship at Amazon and the open-source package, what it
• 20:18
  did. Is it, estimated these noise
• 20:20
  rates that I mentioned earlier, right?
• 20:22
  So a It's false positive and false negative rate and
• 20:25
  specifically what that is is it's you have data and it's
• 20:28
  labeled as 0 but it actually should have been labeled a one
• 20:31
  or you have table that's data. That's labeled 21.
• 20:33
  But it should have been labeled a zero.
• 20:35
  And one thing that we do is, you know, I clean lab is we tell
• 20:37
  you, what's the fraction of data.
• 20:39
  And we do that by estimating this joint Matrix of noisy
• 20:42
  labels and true labels. And that's just going to be if
• 20:45
  it's binary a 2d Matrix. And then, you know, the off,
• 20:48
  diagonals of that will be the flipping rates and the diagonal
• 20:51
  of that Matrix will. Be sort of your consistency
• 20:54
  rates which are like the things that are labeled one and they
• 20:57
  actually should be and then the things are labeled zero and
• 21:00
  should be. So that's kind of hard in an
• 21:01
  audio you know, environment that to see it.
• 21:04
  But just imagine you got a 2d Matrix in your head and you're
• 21:07
  trying to estimate those off diagonals, because that's your
• 21:09
  noise rates, your false - false positive, okay?
• 21:11
  So to the Amazon problem, so at Amazon, that they don't what
• 21:15
  they don't have and they need to estimate is they need to
• 21:17
  estimate the probability that something was labeled a 0 but it
• 21:22
  should have Been a one. Okay, so the rice got it real
• 21:27
  quick so you just said their consistency rate and flipping
• 21:30
  rate, right? And that's the way to think
• 21:32
  about a noisy data set, is like those two factors.
• 21:35
  Let's say, adding up to one or something like that and just
• 21:38
  like the loosest way to think about it, right?
• 21:40
  And what you're trying to do with those with the Alexa
• 21:45
  examples is try to estimate. Like, what's the actual flipping
• 21:48
  rate of the data being wrong? Is that correct?
• 21:50
  Yeah, totally. And the only caveat to what You
• 21:52
  said is that the in this Matrix the whole Matrix sums to 1.
• 21:56
  So it's like a joint distribution and then if just a
• 22:00
  fun fact of how to think about the problem, if the diagonal of
• 22:03
  that Matrix sums to 1, then you have no air at all in your data.
• 22:09
  God meaning that all the off. Diagonals are zero which means
• 22:12
  all the noise rates are zero, which means nothing's flipped,
• 22:14
  so no air in your data. Got it.
• 22:18
  Okay, interesting. And another thing to keep in
• 22:21
  mind is that this is only one formulation of the problem.
• 22:24
  We do several and I've spent time on several, this
• 22:27
  formulation is one called class conditional noise, and it
• 22:31
  assumes that every, you know, any class can be flipped any
• 22:35
  other class at some set constant rate and you're trying to
• 22:38
  estimate that concentrate. And that's, that's really useful
• 22:40
  because like the probability of a, if you see you have an image
• 22:43
  data, set of a bathtub being flipped, to a lion is very low.
• 22:47
  Like, you're not going to have many images that are images of
• 22:49
  bathtubs with the label line that's really unlikely.
• 22:53
  But you are, it is very likely that you have images that are
• 22:56
  labeled tub, you know, you know, there's all sorts of tubs.
• 22:59
  There's like buckets, that's a tub and but it might be labeled
• 23:02
  bathtub or vice versa or a lion might be mislabeled as tiger.
• 23:07
  That's highly likely but it's highly unlikely.
• 23:09
  That microphone is mislabeled as vegetable or something and so
• 23:13
  you can see how these flipping rates emerge And then there's a
• 23:18
  sense. Yeah, and there's also like all
• 23:21
  sorts of other noise where you have instance, called instance,
• 23:24
  dependent. So that would mean, depending on
• 23:26
  the image. You so say you had an image and
• 23:28
  it has a very cat-like dog then because it's a cat like dog.
• 23:32
  You would say what the probability of it being
• 23:34
  mislabeled as cat even though it's a dog and that's another
• 23:37
  approach and then there's also just data noise in the data that
• 23:40
  has nothing to do with the labels like out of distribution
• 23:43
  all stuff we do but let's focus on the solution for Amazon so
• 23:46
  the way that we were able to approach that yeah and By the
• 23:49
  way, one of the reasons we couldn't publish this as if it's
• 23:51
  kind of funny, it's really hard to convince the massive company
• 23:55
  to publish its noise rate in a like, you know, a device that's
• 24:00
  sold to to hundreds of millions of people to, you know, to
• 24:04
  openly public. No, that's not great.
• 24:07
  So it's not easy to convince Administration to say, hey yeah,
• 24:11
  let's post. Let's let's publicly share how
• 24:14
  wrong our devices are so. But we, here's how we estimate
• 24:18
  the noise rate we use Clean lab to find the one of the
• 24:21
  diagonals, okay? And then we also use clean lot.
• 24:24
  If you can find errors with something, you can also find
• 24:26
  good data with something. There's always some ranking
• 24:29
  metric. So what we do is we can find a
• 24:30
  quality score for every example and then you can find examples
• 24:33
  examples that are high quality and that gives you an idea of
• 24:36
  giving out of the things that are labeled one and the things
• 24:39
  are labeled zero. What fraction of those are
• 24:41
  likely labeled correctly as well.
• 24:43
  And so now you have three numbers, you have the false, you
• 24:45
  have the false positive rate, the true positive rate, the true
• 24:49
  Great. And that's all you need to
• 24:51
  compute the last one, which is the false - false negative rate.
• 24:54
  So if you have three things that you can estimate because you can
• 24:56
  observe them and in total that Matrix, some someone then you
• 25:00
  can always estimate the fourth. And so that's how you can
• 25:02
  estimate something that you actually just don't have
• 25:05
  because, you know, all the other cases of how the model performs
• 25:09
  on the data. And so even without the data,
• 25:12
  you can estimate how the model would have performed.
• 25:14
  And that's, that's kind of, its kind of brilliant.
• 25:16
  And so that was the approach that we took.
• 25:17
  We just couldn't share how bad You know, thanks performed
• 25:20
  publicly. That was, that was a downside of
• 25:22
  the research. So just so I understand because
• 25:27
  this is kind of blowing my mind here right now and I'm sure it's
• 25:29
  blowing the mind of a lot of our listeners right now.
• 25:31
  I don't know about you Demetrius, but you're saying
• 25:36
  that when you talk about doing it in one line of code, right?
• 25:40
  I want I want to go to that example.
• 25:42
  Right there. Or now, that you've been working
• 25:44
  on clean lab and maybe this is a clean transition clean clean.
• 25:50
  If I have, let's say, I have, you know, data set that I've
• 25:54
  worked with which is chest x-rays, right?
• 25:58
  And the chest x-ray data set 2014 from the NIH.
• 26:00
  It's filled with data errors, right?
• 26:04
  And I was tired trained, a machine learning model on it and
• 26:08
  there are a couple classes in there for different diseases.
• 26:11
  And you know, let's say it's pneumonia and some other lung
• 26:14
  disease and I have some results where I've trained the model and
• 26:18
  I want to use clean lab in this example, how can clean lab know
• 26:22
  just based on the models performance that particular
• 26:26
  examples are more likely to be You know, falsely labeled or
• 26:32
  incorrectly labeled? The can you walk me through how
• 26:34
  clean lab implements that? Yeah, totally then, and this is
• 26:39
  a good thing about having a company.
• 26:40
  That's, you know, half open source, have SAS product, is
• 26:43
  that I can share things and that was pretty necessary for us.
• 26:46
  We have three is three co-founders that have been
• 26:48
  mighty. We all did our phds there and we
• 26:51
  care a lot about being able to educate and share.
• 26:54
  So part of the idea of founding a company that has roots in open
• 26:57
  source, is it gives us the ability to just talk and answer
• 27:00
  questions like This instead of most CEOs in this position would
• 27:03
  have to give you some kind of and Ira, truly don't mean this
• 27:06
  in an offensive way they have to do their job but they would give
• 27:09
  you a BS answer. They wouldn't actually tell you
• 27:11
  how their product works, right? That's terrifying but we can
• 27:14
  actually do that. And so here's a cool.
• 27:17
  So I'll just share so very briefly one in sight.
• 27:20
  We there's obviously more advanced things that we do in
• 27:22
  this as product and I'll leave those where they like.
• 27:24
  But here's a really simple intuition, you have a model and
• 27:29
  it has produced a predicted bility.
• 27:31
  Okay, you've trained on your data set and now you're going to
• 27:33
  run an inference passed on a new example.
• 27:36
  You have some new example some test set or something or some
• 27:38
  held out data and you run your model on the it's been trained
• 27:41
  on your training data which has noise in it by the way and now
• 27:45
  you run a new example through it and that example is a mislabeled
• 27:49
  example so you have a real label for it it's held out data so you
• 27:52
  actually have labels don't think of it as test data you do have
• 27:55
  labels for this and it's held out and now you run a new.
• 27:58
  You run example through your model that's been trained.
• 28:01
  Noisy data set and it produces a probability distribution and the
• 28:04
  probability. Let's say that it says for say,
• 28:06
  the thing is labeled dog, but it says, like, with 1.0 confidence.
• 28:12
  Okay? So it's like a soft Max with,
• 28:13
  you know, literally all of the weight on cat.
• 28:16
  Okay, then you and your head, you might just be thinking like,
• 28:20
  that seems surprising, right? Like, this thing is, I have a
• 28:24
  label for it and the labels dog, and I threw it into my model and
• 28:28
  my model spit out, 100% probability.
• 28:30
  A of being cat, you know? Like if this model learned
• 28:33
  anything at all, then something is amiss.
• 28:36
  And so that's just like, a really easy way to think about
• 28:38
  how you might find an error. Like, literally, that's seems
• 28:41
  pretty like one line of code, right?
• 28:43
  You just did it in friends pass. And then you just measure is the
• 28:47
  probability of the given label like way off, you know, is it
• 28:50
  way different than the probability of the ARG Max?
• 28:53
  So that's one way you can think about it.
• 28:54
  The question is and this is where it gets more complicated.
• 28:57
  How do you know what the threshold is?
• 28:59
  And what I mean specifically is okay.
• 29:01
  What if it's .5 dog .5 cat, what if it's .3 cat .72 up?
• 29:07
  Like how how do you know the threshold?
• 29:09
  And so there's a lot of proaches to that and a lot of smart
• 29:11
  things you can do. One way is you just can look at,
• 29:15
  you know, random the 90th percentile of all your cat
• 29:18
  examples and other ways to take the mean.
• 29:20
  Another way is to take an approach that's entirely
• 29:22
  different and like fully model the entire joint distribution,
• 29:25
  and then you can, you know, sum up the diagonals and do all
• 29:28
  sorts of fancy stuff. But that just gives you like a
• 29:30
  really clear idea, you have a data set, you can pass it
• 29:33
  through a model that's been trained on it.
• 29:35
  And then you can take the, you know, whatever the probabilities
• 29:38
  are and say, hey, look, this this thing is labeled dog.
• 29:40
  But like the probability of dog is really low and they're
• 29:42
  probably of casually High. It might be a cat.
• 29:45
  And that's just an easy way to think about finding the players
• 29:47
  with a model, I see. So it's looking.
• 29:50
  So it's really looking at where models make mistakes and using
• 29:54
  that to understand what that false positive false negative
• 29:58
  rate is, is that First airing, this is Skyler.
• 30:01
  I lead machine learning and health rhythms if you want to
• 30:03
  stay on top of everything happening in ml.
• 30:06
  Ops, subscribe to this podcast. Almost it's a super Fair summary
• 30:11
  but there's one really important caveat.
• 30:12
  It's not just where models make mistakes.
• 30:14
  This is the, this is the model Centre point of view.
• 30:17
  It's also where the data make mistakes, where the data made
• 30:20
  mistakes, and the way that you can think about this in the
• 30:23
  model is embedded, two things, one is noise because the model
• 30:28
  is not good enough and the world.
• 30:29
  You can think about that noise is its noise where if you had
• 30:33
  perfect data and you train them all, you still wouldn't get 100%
• 30:35
  accuracy on all test data. If you did, that would be
• 30:39
  incredible for your problem and great for you, you've just, you
• 30:42
  know, you have a business and you should probably build that.
• 30:45
  But, in general, most people don't get 100% test accuracy.
• 30:49
  And so, whatever the noise is, right?
• 30:51
  So say you get 95%, then your model has like five percent
• 30:54
  inherent, you know, their words for this, I lied.
• 30:59
  Tarek. If you've heard of Elliott's
• 31:00
  work and epistemic uncertainties, Alia torque is
• 31:03
  like label noise, and epistemic is like model noise.
• 31:06
  So that's as epistemic. This is, this is intrinsic noise
• 31:10
  in the model but then there's also a noise from your data so
• 31:13
  things that are mislabeled and bad data.
• 31:15
  And the model is going to learn that to you trained on that
• 31:17
  data, it's going to learn that noise.
• 31:19
  So if say 5% of your, you know, cat examples are mislabeled as
• 31:23
  dog. Then that means that when your
• 31:24
  model is producing predictions, 5% of the, if it's a good model,
• 31:28
  five percent of the time, It's going to produce wrong
• 31:31
  predictions. It's going to produce, you know
• 31:33
  the wrong label for those dog examples and so you can
• 31:36
  literally think of it that way your models making mistakes on
• 31:38
  its own intrinsically. That's called epistemic
• 31:41
  uncertainty, epistemic noise. And it also is modeling because
• 31:45
  it's trained on noisy data. It inherently models that to it
• 31:48
  models the noise as well so that's called epistemic
• 31:50
  uncertainty. Now the kicker the trick is you
• 31:54
  have to disambiguate epistemic and aleatory uncertainty out of
• 31:58
  one single model. And you cannot do that without
• 32:01
  assumptions because the models prediction, you know, it's
• 32:04
  literally just making a prediction.
• 32:05
  So you have to make some assumption and then use that
• 32:07
  assumption to disambiguate. And if you remember earlier, I
• 32:09
  talked about my Matrix, where you'll get the flipping rates of
• 32:12
  every class to every other class that's an assumption.
• 32:15
  And you can use that assumption that to fully solve this problem
• 32:18
  and fully characterize label Noise.
• 32:20
  We also make some other assumptions like out of
• 32:22
  distribution assumptions and also you can do instant space
• 32:25
  assumptions where you say like there are groups of are.
• 32:27
  There's a, you can go crazy here.
• 32:29
  Crazy for a podcast but you get the idea.
• 32:33
  No, that's a brilliant, brilliant, extra explanation of
• 32:37
  where I may have glossed over the crucial detail.
• 32:40
  And I love that point about the difference between epistemic and
• 32:43
  Alia tarek uncertainty that really helped me understand what
• 32:46
  it is, what conceptional you're trying to do and now I
• 32:50
  understand what makes clean lap, particularly powerful is that
• 32:53
  you've embedded a lot of the ways of working with these
• 32:56
  Concepts into code, so that people can start Art to do this
• 33:00
  with their models and their data and by making it open source,
• 33:04
  the code can get even better, right?
• 33:06
  And people can contribute and help you guys.
• 33:08
  Expand your functionality, is that a fair thing to say?
• 33:12
  Yeah, for sure. Okay, yeah, there is definitely
• 33:14
  the idea of what it looks like and can you just break us break
• 33:19
  it down? Like what do we have?
• 33:22
  Is it some kind of python wrapper?
• 33:24
  Or is it something I need to install?
• 33:27
  How does it actually plays? With my environment.
• 33:32
  Yeah, for sure. So, you have some data set, it
• 33:34
  can be represented, however, it doesn't matter.
• 33:37
  So here's some fun facts. So, clean lab works with any
• 33:40
  model and it works with any data as long as you can, run a model
• 33:44
  on it. So as long as it's like ml data,
• 33:46
  let's say so it works with any ml data and then it works with
• 33:50
  any model and that's cool, right?
• 33:53
  So that means if we invent. So right now Transformers are
• 33:55
  hot. LS teams were hot 10 years ago,
• 33:59
  so, So if you built a whole company around Ellis TMS, your
• 34:02
  company might be in a really bad spot, you know in 2020 if you
• 34:07
  built it in 2010 and so it's the same idea, right?
• 34:11
  You don't want to. So you know we built a company
• 34:14
  around this. We don't want to make we don't
• 34:16
  want to be in a bad spot and 2030.
• 34:18
  And so one way we can guarantee that is by building a company
• 34:20
  that works no matter what the model is that's invented in the
• 34:23
  future. So if something much better than
• 34:25
  Transformers comes out, we want to make sure we work with that
• 34:28
  too and Way that we do that is we work with the outputs of a
• 34:32
  model but we don't care which model generated them, it's
• 34:36
  different than a markovian assumption, but you can think of
• 34:38
  it like a markovian assumption. We don't care where it came
• 34:41
  from. We only care about the now what
• 34:43
  are the models predictions? It doesn't matter what the model
• 34:46
  is and it doesn't even matter what the data is and reality.
• 34:49
  There's actually a lot of data like very specific data
• 34:51
  assumptions and it different data distributions are totally
• 34:54
  different and there's all sorts of weird stuff that happens.
• 34:56
  But if you have an appropriate model for that data set, You can
• 34:59
  abstract that away make it more of an engineering prong problem
• 35:02
  and less of a theoretical problem and just take those
• 35:04
  model outputs. Okay, so the answer your
• 35:06
  question concretely. Now you have these model outputs
• 35:08
  their predicted probabilities and you have your labels some
• 35:12
  noisy labels and then you're going to input those two things
• 35:15
  into one line of code and it's going to spit out things for
• 35:18
  you. Like, what are all the label
• 35:19
  issues? You know, in that data set and
• 35:23
  or it'll it'll spit out, we do a bunch of stuff that's there's a
• 35:25
  that's one, there's a package called clean learning says, A
• 35:29
  cute if you do machine learning but you want it to be clean,
• 35:34
  right? You want to train on cleans data
• 35:36
  machine? Learning with clean data, we
• 35:38
  call it clean learning and that's a class and you can
• 35:40
  actually just pass in like a, you know, sklearn model directly
• 35:44
  into that class and pass in your data and it will just work or
• 35:48
  you can use the function at the functions directly where you can
• 35:52
  just pass in like your model outputs and that way you can
• 35:54
  work with any model at all. But yeah, that's mainly how it
• 35:57
  works. I love that summary.
• 36:00
  I think that's a great question Demetrios because it got us to
• 36:03
  the nitty-gritty of what it actually feels like, which is so
• 36:06
  important with I think tooling right now in machine learning.
• 36:11
  Right? This stuff is moving very
• 36:13
  quickly. You know, the fact that we are,
• 36:16
  the fact that you're talking about the concepts that you are
• 36:18
  around, how to think about data feel very different than the way
• 36:21
  that we taught about data and modeling like five years ago,
• 36:24
  right? And with the speed at which
• 36:25
  things move having tools, that feel easy and Well with your
• 36:29
  existing workflow without too much like you know I guess you
• 36:32
  could say friction is really important and it certainly
• 36:34
  sounds like clean lab is that way.
• 36:38
  I think one of the questions I had coming out of this is you
• 36:42
  know as you can probably tell I've worked a lot in them in the
• 36:47
  medical context, right? Where noisy errors are both
• 36:51
  incredibly costly and Incredibly common.
• 36:53
  Right? Nose noisily labeled data sets.
• 36:55
  We have a lot of variance in the way that medical professionals
• 36:58
  are trained. You and on this data quality
• 37:03
  question, did a quality problem. I kind of have a question around
• 37:06
  like how ml engineer should think about it.
• 37:09
  And my question is, is this something that can be just
• 37:13
  solved by a tool or a framework or some kind of, you know, new
• 37:19
  widget like clean lab not necessarily to to denigrate it?
• 37:23
  But just like a tool like that or is this something that I you
• 37:26
  think ml engineer should take more of a Systems level approach
• 37:29
  to and kind of say like, hey, you know, the same way that you
• 37:33
  think about like, you know, web systems architecture or
• 37:36
  something like that. In a different software,
• 37:38
  engineering field machine learning Engineers should start
• 37:40
  to specialize in thinking about Clean Machine Learning Systems.
• 37:46
  Let me know if that question made sense because it's just
• 37:48
  makes perfect sense. Okay, cool.
• 37:50
  Yeah, it makes so much sense that our entire business model
• 37:52
  is the answer to your question. Alright, don't hit me nice.
• 37:56
  Yeah. So, Let's let's first, let's
• 38:00
  start. Here's a good thing to think
• 38:02
  about. If somebody says that they have
• 38:04
  built a something that works 100, it does everything.
• 38:08
  We do a lot of things automatically, so we can find
• 38:11
  label issues for you automatically.
• 38:12
  It's pretty cool. And we can even provide
• 38:14
  suggestions for Solutions automatically, but that's not
• 38:16
  where it ends. It's not just 100% everything
• 38:19
  automatic. You press one button and that's
• 38:21
  it. You're done.
• 38:22
  If you have someone, whether their academic or they're in
• 38:25
  business or their entrepreneur, you know, and they say I have
• 38:28
  built in AI solution and it just does everything and no human
• 38:33
  ever needs to be involved. Well then the question is, why
• 38:36
  are you involved, right? Why don't you just let the a I
• 38:39
  run off and you know, and just do everything all on its own.
• 38:41
  Like, why do you even need to be involved?
• 38:43
  The reality is there are very few AI solutions that don't
• 38:46
  still require some kind of human input.
• 38:49
  Like if you look at, you know, Dolly, it's incredible, it's
• 38:52
  really incredible, G PT 3, it's incredible.
• 38:54
  But also why do they need a team of like 150 people?
• 38:58
  Build those things, that's where the human is involved.
• 39:02
  So it's not like these things are building themselves or
• 39:04
  they're just like and now there's even bigger teams that
• 39:06
  are working on these things because they've been shown to
• 39:08
  have viable business model. So there's always a human in the
• 39:11
  loop. So, that's just the most
• 39:12
  important thing to remember. It's easy to forget, you know?
• 39:14
  Yeah. These models are doing a lot of
• 39:16
  really cool things. I think there was like, some
• 39:18
  article yesterday about somebody who is working at Meadow or
• 39:21
  something and they created the accidentally created.
• 39:24
  You know, I'm talking about Vishnu, is this the one, the
• 39:26
  one, the guy, where the goo Engineer was saying that the AI
• 39:31
  is sentient. Is that the one you want?
• 39:32
  Yeah. He looked like, he was like
• 39:34
  yeah, yeah, and I like got fired or quit his job or something
• 39:38
  terrible happened because of this, but he was like, oh no.
• 39:40
  I am working with an AI and it's sentient and everyone's like,
• 39:43
  this is this is a lot. Yeah, yeah.
• 39:46
  And exactly talking about, right.
• 39:48
  Yeah, and I think what we can create, we eventually can create
• 39:51
  things that give the appearance of sentience and maybe even
• 39:54
  indistinguishably. But just keep in mind that it
• 39:57
  was created by a person. And that person created
• 39:59
  constraints. And the, the more you constrain
• 40:01
  something, the easier it is to make it appear sentient within
• 40:04
  the context of those constraints and that's just kind of a rule
• 40:07
  of thumb. Okay, so with that in mind so to
• 40:09
  answer your questions. So your question is about, you
• 40:12
  know, can you can you just have something like clean lab just
• 40:15
  you know, fix everything and then it ends there.
• 40:17
  And and so that's, that's really important to think about.
• 40:20
  So what the open-source does is it will, it's prescriptive,
• 40:24
  okay, just like a doctor, it tells you but it doesn't make it
• 40:27
  all go away. You know, it tells you hey, you
• 40:30
  need to take this medicine, you need to do this but it doesn't
• 40:33
  like take the medicine for your data, okay?
• 40:35
  So it'll tell you what the issues are.
• 40:37
  It'll tell you the quality of your data set, it's really
• 40:39
  helpful but it's just prescriptive.
• 40:41
  It's it doesn't change the data set.
• 40:43
  If you want something that will automatically fix your entire
• 40:45
  data set for you, with one button, click, good luck.
• 40:48
  But we'll get you really close. Okay, so we have something
• 40:51
  called clean Lab Studio. That's our SAS product.
• 40:54
  It builds on top of the open source.
• 40:55
  Adds a bunch of fancy infrastructure.
• 40:57
  Trains the model. For you and makes it so that
• 40:59
  it's a no code solution. Now, I just sound like, you
• 41:01
  know, a CEO pitching but like check it out, it's actually
• 41:03
  really awesome. But here's why it's useful and
• 41:06
  white answers your question. What the what it does is it also
• 41:09
  helps you fix the data set where as the open source helps you
• 41:12
  find. So the open source is
• 41:13
  prescriptive and then our whole SAS business model is like, hey
• 41:18
  people have seen that clean LAB Works but now they want to, you
• 41:21
  know, very quickly and easily be able to actually just fix the
• 41:23
  whole data set, not just know what's wrong with it and so to
• 41:26
  do that, you do need a human in the loop.
• 41:28
  You need To take a look. We provide suggestions of what
• 41:31
  we think all the corrections are and we provide, we have an AI
• 41:34
  and it runs and then it checks in.
• 41:36
  It it guesses. Okay, this is what I think.
• 41:38
  All the true labels are, here's what all the errors are, every
• 41:40
  all the data is ranked and we do all this for you automatically
• 41:43
  and that's really cool. But we still suggest, you know,
• 41:45
  you should have someone take a look.
• 41:46
  And so one of the ways that we help you is we take data
• 41:49
  Centric, AI approaches to make things more human Centric.
• 41:54
  And so the way that works is if you know which data point is the
• 41:57
  most likely to be a A problem. Then you want to show that to a
• 42:00
  person first so you don't waste their time and then you can know
• 42:03
  the second most likely problem and you show that to a person s.
• 42:06
  And so the cool thing that are SAS product does is it lets you
• 42:08
  very quickly go through only the most important stuff that might
• 42:12
  be wrong with your data set and we also will give you an idea of
• 42:14
  what we think. The right answer so you do as
• 42:16
  little work as possible and that so that answers your question,
• 42:19
  the question being, you know do we still need people to be
• 42:22
  involved in the answers? You know, definitely if you want
• 42:25
  the highest quality for many solutions and that's An example
• 42:29
  of how you can use data Centric, Ai and data Centric approaches.
• 42:32
  So that when you do involve people, you make it as easy as
• 42:35
  you can for them, they still should be involved.
• 42:37
  But what we can do, is use Smart methods to make it so that you
• 42:40
  only spend like, you know, an hour cleaning a dataset, that
• 42:43
  normally would have taken ten thousand hours and that's the
• 42:46
  goal and that's what we do. Brilliant.
• 42:49
  So there's there's something that There's something that you
• 42:55
  said when it comes to Quantum Computing and how one of the
• 43:01
  investors is or one of clean Labs advisors and investors is
• 43:08
  dealing with Quantum Computing. What does that have to do with
• 43:11
  anything? Yeah, totally.
• 43:14
  So I will just rephrase and when you say dealing with Quantum
• 43:18
  Computing so so we're talking about Ike, he's known as I But
• 43:24
  formally, Isaac shuang. And he was the first person to
• 43:28
  realize meaning, to actually have a physical on Earth, man of
• 43:32
  manifestation to build the first quantum computer.
• 43:36
  So, this guy built the first working quantum.
• 43:38
  Computer singles are not just dealing with it.
• 43:41
  He's the guy behind. Yeah, so he wanted Computing,
• 43:45
  it's fair to say, he invented the first real realization of a
• 43:48
  quantum computer on planet Earth.
• 43:51
  And there were a few groups doing it simultaneously.
• 43:53
  And And so let's give credit to everybody but he was one of
• 43:56
  those groups and maybe maybe this is within, they're all
• 44:00
  within a week but he's credited that way and and rightly.
• 44:03
  So he's been doing really, really top-notch research in
• 44:06
  Quantum Computing at MIT for the last like 20 years.
• 44:09
  He also wrote the book on Quantum Computing.
• 44:13
  Isaac Tong, if you just look him up, you'll figure that out
• 44:15
  pretty quickly. So why is that relevant?
• 44:18
  And this is really cool. So the way as relevant as they
• 44:21
  approach in particular, that was used to summon.
• 44:23
  Held in Mr. Quantum Computing. And we don't have to like, get
• 44:28
  into the details, but the way that you do Quantum Computing is
• 44:30
  you have a noisy, you have noisy information, and then you're
• 44:33
  trying to do some sort of computation on it where you are
• 44:37
  in physically inferring, what is the true signal from some noisy,
• 44:41
  you know, non-binary, 01 signal? Where you have to observe
• 44:45
  something hit with a laser and then you have all.
• 44:47
  There's a lot of air in that process.
• 44:49
  And so one of the key Solutions, and if you just look up in Mr,
• 44:53
  computing In Quantum Computing online and you'll you'll see
• 44:57
  that the biggest problem actually with the field is as
• 44:59
  you have more qubits it's harder and harder to understand what is
• 45:03
  the noise and what's actually what are you interpreting
• 45:06
  versus? What's the reality of what's
• 45:07
  happening on the machine. And so the way that you solve
• 45:09
  this, as you estimate was called a noisy channel, the noisy
• 45:12
  channel is what perturbs the actual computation to What You
• 45:16
  observe. And if you can estimate the
• 45:18
  noisy Channel, then you can see. Okay, what did I observe?
• 45:21
  And then I can use the noisy Channel and I can actually
• 45:23
  reverse Engineer and find out what really was the computation
• 45:26
  that happened, and that's how you do Quantum Computing.
• 45:28
  That's like the key to being able to do Quantum Computing.
• 45:31
  Is you have to be able to understand what's really
• 45:33
  happening versus What You observe and observability.
• 45:36
  Okay, so this this all sounds like crazy, you know, you see
• 45:39
  where it's going. Alright?
• 45:40
  So so it sounds like, you know, just a little bit wild at first
• 45:43
  but then, you know, here's where it's going.
• 45:45
  So we said, all right, let's take the whole model of how
• 45:48
  Quantum Computing is done and let's apply it to a data set.
• 45:51
  A data set has What You observe. That's just the Data, right?
• 45:54
  You observe that data but the real question is, what would the
• 45:57
  data have been if it didn't have out of distribution examples?
• 46:00
  If it didn't have label errors, if it didn't have all this
• 46:03
  noise, that's causing you to not be able to train your model.
• 46:05
  Well, right? And so that's the question and
• 46:08
  so what we do is we compute that same noisy Channel and we have
• 46:12
  our observation of the data and we can use that noisy Channel.
• 46:15
  Meaning, what was the noise process that created this noisy
• 46:18
  data to unveil to uncover what the true data was and then you
• 46:22
  train on that true stuff? And that's how you get like a
• 46:24
  better model or that's how you find out, which of your
• 46:27
  annotators is making the most mistakes or whatever you want to
• 46:29
  do, and that's how it works. So you can actually use the same
• 46:32
  ideas from information theory in order to solve this data problem
• 46:36
  and me and Ike work together to invent what's called competent
• 46:39
  learning, which was this field that underlies clean lab and we
• 46:42
  worked on at MIT together on this for about six years.
• 46:47
  I guess the question that I wanted to ask after that is, you
• 46:51
  know, going back to where you're talking about Alexa and the work
• 46:56
  that you were doing and Amazon and Facebook like just work
• 46:59
  around confident learning. What was the result that made
• 47:02
  you realize that you really had something that worked right?
• 47:07
  Like I'm kind of curious. What your aha moment was there.
• 47:09
  Yeah it was. There were a few so they were
• 47:13
  they were Stepping Stones. I thought I would be faculty.
• 47:16
  I was doing a PhD, we were building a new field, it was
• 47:19
  really cool. I had some good contacts at the
• 47:21
  media lab that you know, Likely would have been fruitful or
• 47:25
  let's say may have been fruitful and I thought that would be a
• 47:27
  cool cool thing to do in the next step of my life.
• 47:30
  I had some really good friends who I had been working with
• 47:32
  along the way, Jonas, Mueller, and Anisha.
• 47:34
  That all ye who had been publishing papers with.
• 47:36
  We had, we were nominated for the best paper in Europe's for
• 47:40
  our work with clean lab. We if you go to label airs.com
• 47:44
  you'll see the 10 most commonly cited machine learning data sets
• 47:49
  and actually the test sets that we Benchmark on and they're all
• 47:52
  the errors in them. And this is groundbreaking
• 47:55
  because like truly, because now you could actually see that all
• 47:58
  the test sets of the field of machine learning that we
• 47:59
  Benchmark on, they're all broken.
• 48:02
  And so basically we had a lot of results, a lot of excitement.
• 48:05
  So why did we, why do we choose to do a business and what was
• 48:09
  sort of the key stepping stones that convinced us?
• 48:12
  We really need to do this. This is a viable business and
• 48:14
  it's a good idea. So I'm rephrasing your question,
• 48:17
  but that's because it sets me up and it sets me up for the
• 48:19
  answer. And so the answer is at first
• 48:22
  I'm at MIT and I'm building this Detection system and I realized
• 48:25
  am. The only way I can do this is to
• 48:27
  fix the data set because I don't even know like any of my false
• 48:31
  negatives. I, you know, I have tons of them
• 48:33
  and how do I estimate those? So then I'm like, okay, if I
• 48:36
  want to build an AI am L solution that like really helps
• 48:39
  people, I have to do this. Confident learning thing, I have
• 48:41
  to have something like clean lab so then I'm like, okay, clean
• 48:44
  lab seems pretty useful. Well, then, I went to Facebook
• 48:46
  and I was like, okay, now we're doing the same thing on
• 48:48
  petabytes of data, like literally trillions of data
• 48:50
  points and clean lab is really useful.
• 48:53
  And then I went and then I but still, like I didn't realize I
• 48:56
  didn't think of it as a business because I was so focused on the
• 48:58
  problems and then I was in the theory.
• 48:59
  I was a grad student, my head was down, I got to get a PhD.
• 49:03
  Then after that, I went to Amazon and I mentioned that use
• 49:06
  case after Amazon, I went to Oculus research, I spent two and
• 49:09
  a half years with oculus research, part time, I worked
• 49:11
  with Richard Newcomb, who's the director of science for, for, I
• 49:15
  guess, now, whatever metas new version is of oculus research
• 49:20
  and it keeps changing, but Richard Newcomb is a Ask that
• 49:23
  guy. He was one of the creators of
• 49:25
  Slam in terms of making slam, operational and useful.
• 49:28
  Generally a he made a startup That Was Then acquired by Oculus
• 49:32
  and that's how he became director suit, he's like this
• 49:34
  British guy super fun, super smart guy also to work with
• 49:36
  bikes, like 100 miles a week. Just a really cool guy and I
• 49:40
  worked with him for two and a half years and that was pretty.
• 49:42
  It was pretty instructive and my career and seeing how does
• 49:46
  someone direct a very large AI team?
• 49:49
  Not just being on the researcher, but how do you build
• 49:50
  these things? And how do you build a team?
• 49:53
  And that was helpful and pivotable from me, pivotal
• 49:56
  pivotal for me. Then what do we do there?
• 49:59
  We use clean lab to be able to clean and create some of their
• 50:02
  data sets. And so that was like what this
• 50:04
  is really useful again now for Oculus.
• 50:06
  Well the next summer I went to Google and this is when I
• 50:09
  started to really realize, okay, I need to I just needs to be a
• 50:12
  business at this point. Clean lab open source, was
• 50:14
  getting pretty fleshed out. It was useful.
• 50:16
  You could run things in one line of code and I ended up
• 50:19
  implementing a version of clean lab a simpler version in I'd of
• 50:24
  Google. And if anybody works at Google
• 50:25
  and they're listening to this, you can still see it.
• 50:27
  It /. Google 3, / 3rd Dash party /,
• 50:30
  clean lab. And you can that's and we use
• 50:32
  this at Google to fix the OK, Google and hey, Google data
• 50:37
  sets. So if you've ever said that to
• 50:38
  an Android device the way we. Yeah, that's how we clean that
• 50:41
  data set. Yeah, he's clean lab.
• 50:44
  And so, then I'm like wait a minute.
• 50:45
  So that's another gigantic data set across like 50 million
• 50:48
  examples and then we got some testimonials from Google, some
• 50:51
  of their Engineers, you know, they sent us.
• 50:53
  Testimonials and they were like, hey if you guys, you know, have
• 50:55
  a company around this like we here's a successful pilot.
• 50:59
  Meanwhile, we got some reach out from other companies in the
• 51:02
  labeling space saying, hey, would you like to join our team?
• 51:04
  You know, can we acquire this thing?
• 51:06
  And I'm starting to realize, okay, you know, this is their
• 51:08
  people paying attention and not just from an academic
• 51:10
  perspective, this is a problem that pretty much every major
• 51:13
  tech industry needs to solve. If you don't know three trillion
• 51:17
  dollars is the estimated cost of the us alone of bad data.
• 51:20
  And we have a tool that we have shown and Worked out performs
• 51:24
  other tools to find bad data. And so it's like well this is a
• 51:28
  gigantic market and we have a solution for that market and
• 51:31
  then more and more people started reaching out.
• 51:33
  And then before we had the team that was happening, organically
• 51:36
  from the research at MIT and then we had a best paper
• 51:39
  nomination that nerves which brought more folks.
• 51:41
  And then we did well in the data Centric, AI competition and they
• 51:44
  just there was more momentum. And then it was it was something
• 51:46
  I actually could not do and I was doing another startup, it
• 51:49
  was really cool started, but used clean lab to clean emotion
• 51:52
  data so that you could Train an emotion detection algorithm and
• 51:55
  that way, if you're doing like a sales call or a hiring call, you
• 51:58
  could actually say what's the metadata for every frame in
• 52:01
  terms of their emotions? That's really useful for doing
• 52:03
  sales, it's extremely useful, but it's very hard to train
• 52:06
  because emotion data is noisy. So we use clean lab for that and
• 52:10
  then I was like, what? Why am I building a startup that
• 52:13
  uses clean lab clean lab is the startup and that was sort of
• 52:17
  the, the final pivotal transition.
• 52:20
  And we yeah, that was the plot twist.
• 52:22
  Yeah. Dude, so that's one thing.
• 52:26
  Oh, go ahead. It was there solid on the finish
• 52:28
  with, It's the final. It was gonna, I was gonna do
• 52:31
  like a j.r. Tolkien final sentence.
• 52:33
  That was how it came to be. So there's one thing that stands
• 52:40
  out to me, though, in all of the history of clean lab is you've
• 52:44
  been working with huge companies?
• 52:47
  I mean, the biggest companies around, right?
• 52:49
  Especially in Tech. Does this still have?
• 52:53
  Have as much utility for the small data sets as it does for
• 52:59
  the Googles and the Amazons. Yeah it totally.
• 53:02
  It works for any data set. So it's the reason why we wanted
• 53:06
  to work with big companies from the start is because nobody else
• 53:09
  could do that. And so, if you're building
• 53:11
  something, if you're entering a market, it's a good idea to say,
• 53:14
  like what can you do that? Other people can't do.
• 53:17
  If you look right now, if you just search online data quality
• 53:19
  for AI, it's like 100 companies like everyone is branding
• 53:23
  themselves. Equality for AI.
• 53:25
  But where show me, show me a company that does label quality
• 53:29
  and it's just like, not a thing. Well, that's what we do at clean
• 53:31
  lab will show me a company that can do label quality for data
• 53:35
  that's already been labeled at massive scale like that.
• 53:39
  These aren't keywords that you see as often and so we realized
• 53:41
  that was a differentiator for us.
• 53:43
  And so we wanted to work with key Pilots where we were fixing
• 53:47
  labels, specifically. We can also do tons of stuff in
• 53:49
  the data space out of distribution.
• 53:50
  A distribution examples, but we wanted to focus on thing that
• 53:53
  differentiates Us and something where we do extremely well,
• 53:56
  where we know we have one of the best Solutions and that was just
• 53:58
  the main reason but it works for any dataset.
• 54:00
  So you've inevitably had to have seen some really exciting things
• 54:07
  happen since clean lab launched. I guess what are some huge
• 54:13
  success stories or just things that you didn't think we're
• 54:17
  possible that potentially became possible or things that you
• 54:23
  stay. With you because it's been so
• 54:26
  astounding. Hmm, that's a good question.
• 54:32
  I would say the whole journey in a sense, this is feels that way.
• 54:36
  If you, if you just I I take a big bird's eye view for
• 54:39
  questions like this. So I just think about where were
• 54:41
  we ten years ago? So, I, when I was sitting in
• 54:46
  when 2016, I'm an intern. I'm a very green.
• 54:50
  Eyed excited in turn on like 20, 24, or 25 or something at this
• 54:55
  time. And I'm sitting inside of And an
• 54:59
  office room with Geoff, Hinton and Ian, laocoon and some other
• 55:03
  Facebook AI researchers. And I really didn't even realize
• 55:07
  at the time just how, how much of an experience that was
• 55:11
  because I was pretty young and I'd read a few papers, but I
• 55:13
  didn't know like, you know, these are some of the biggest
• 55:16
  minds of our time. I didn't know they were about to
• 55:17
  win the Turing award or the Nobel Prize for computer
• 55:20
  science. And so, this is an experience I
• 55:23
  was in and then Geoff, Geoff Hinton, he shared with us one.
• 55:28
  Label error. Okay, so we're 2016 just to give
• 55:31
  you an idea of the time and space and he shared with us.
• 55:35
  One label are and Ian mcewan's data set and he was actually he
• 55:38
  was genuinely excited about that a because he's you know, proving
• 55:41
  a friend, you know, their friends and he's like, hey haha
• 55:45
  I found an error in your data set but also be I think that
• 55:48
  that, you know, that's just representative if you have one
• 55:50
  of the greatest minds of our time, in our field, who's
• 55:53
  excited about one error in a dataset.
• 55:56
  And now we're finding, you know, millions of these things.
• 55:58
  Scale and we can do them generally.
• 56:01
  That is exciting. When I just take a bird's eye
• 56:03
  view and I'm like, okay, we've built now a business around
• 56:05
  this. That actually does this at scale
• 56:06
  for anybody, and if you just go to label airs.com, you can
• 56:09
  literally see millions of them. Many of them in the same data
• 56:12
  set that. Geoff Hinton was finding an air
• 56:14
  and that's that's pretty cool. Hell yeah, that's incredible.
• 56:21
  Well, I want to end with some rapid fire questions for you if
• 56:24
  you don't mind. Sure.
• 56:26
  First one is what? What was the last book you read?
• 56:31
  Last book I read was the bizarre and the cathedral which is it's
• 56:37
  an Open Source book. It's from some of the original
• 56:40
  folks who created if, you know, Linus Torvalds who created Linux
• 56:45
  and it's the perspective of why open source as a business, can
• 56:49
  be a good model. We do both SAS and open source,
• 56:52
  but it's a pretty cool book and I highly recommend it for anyone
• 56:54
  who's interested in open source. Okay, I like that and as they
• 57:00
  say, Big fan of open-source. I am going to have to check that
• 57:03
  out. Next one, I've got for you is
• 57:09
  What was the last bug that you smashed?
• 57:12
  I know you were up lat late last night was that smashing bugs or
• 57:17
  no numbers slashing. We had a, some, some good
• 57:22
  opportunities and they were good enough.
• 57:24
  I wanted to stay up for him, but let me think the last bug I
• 57:28
  smash, you know, last night there was a there was I so I
• 57:32
  mean I'm in California sometimes you'll open the door and there
• 57:34
  will be some flies that come in, you know, and it was a nice day
• 57:38
  and Wow, buzzing around. You know that experience?
• 57:42
  You're like trying to do your work and there's some fly
• 57:44
  buzzing, it doesn't matter. It doesn't matter where you are,
• 57:45
  you can be in a penthouse or you can be.
• 57:47
  And, you know, tent somewhere. There's always going to be flies
• 57:51
  and it's one thing that unites us, no matter, no matter what
• 57:53
  your position in life, you all have to everyone has to deal
• 57:56
  with flies so I totally smashed that fly.
• 57:59
  All right. I like you took that one a
• 58:03
  little bit different direction than I was expecting but I like
• 58:06
  it. So, You as a business owner,
• 58:11
  that is selling into the ml space.
• 58:13
  What is a piece of ml marketing or MLM Ops marketing, ml,
• 58:20
  tooling marketing that you have thought about putting out or
• 58:26
  your team has thought about putting out or you've seen put
• 58:29
  out that major eyes roll. Hmm.
• 58:33
  Roll like, this is like, I mean, I, I think I already mentioned
• 58:36
  it, but when I see just data quality for a Just in general, I
• 58:40
  use, I even use we even use that sometimes at clean lab.
• 58:43
  Like I think if my current Banner on LinkedIn says data and
• 58:45
  label quality for AI. But then we put we put automated
• 58:48
  around it just to emphasize what we do differently.
• 58:50
  But in general, I don't like that phrasing.
• 58:52
  It's just that people have gotten used to it, but I think
• 58:54
  it's so it's so generic. And if we are all saying the
• 58:57
  same thing, how does anyone know what anyone's doing?
• 58:59
  So when I see that it kind of makes my eyes roll because
• 59:01
  there's no differentiator and it's just like, what exactly are
• 59:03
  you talking about dude? I think that's one of the
• 59:07
  biggest problems right now and This space is that so many of
• 59:12
  the tooling companies are saying the same thing, but taking
• 59:16
  drastically different approaches to getting it done.
• 59:20
  It's like, oh, the outcome is the same.
• 59:22
  And so, that's what we're pitching.
• 59:24
  You the because I think all the marketers on the teams are like,
• 59:27
  they've gone through. The what is it?
• 59:29
  The things to be done marketing school, where it's just like,
• 59:33
  no, you don't want a hammer and a nail, you want a picture on
• 59:36
  your wall and so, Pitching you the picture on the wall?
• 59:40
  Not the hammer and nail. That gets you that picture on
• 59:43
  the wall, right? So the idea here when people say
• 59:47
  oh yeah we make your ml faster or we give your data quality
• 59:52
  assurance and that kind of stuff.
• 59:53
  It's like yeah but how and then it just creates a very cloudy
• 59:59
  and mystified space. And people have to really search
• 1:00:03
  around and dig really hard to see if what the tool that
• 1:00:07
  they're looking at. Does what they're looking for
• 1:00:10
  and if it does it in the way that they need it to be done.
• 1:00:14
  And so I'm fully onboard with what you're talking about there.
• 1:00:18
  Like, 100%. Now, this next one.
• 1:00:23
  Did you have anything there? Sorry, I would just say the
• 1:00:25
  biggest. No, it's okay.
• 1:00:27
  I just think the biggest detriment for the field.
• 1:00:30
  And I mean an industry like for businesses and ML and AI
• 1:00:34
  companies is that there are probably 90% of them are
• 1:00:38
  companies that Claim to use Ai and there's no real clear,
• 1:00:42
  explanation of what is the AI and then you find out that
• 1:00:45
  they've downloaded app, I torch model or a hugging face model
• 1:00:48
  and they're running it through a you know a generic Sage maker,
• 1:00:51
  you know, tool that's already been set up for them and that's
• 1:00:54
  the total sum of their a is just a black box tool that anybody
• 1:00:57
  else could download for free. And when you find that out and
• 1:01:00
  then it's called me. I company, I just it's not
• 1:01:02
  necessarily a bad thing, you know, like people will do
• 1:01:05
  business where they do business and not all business has to like
• 1:01:08
  come out of It'll research, you know, out of a university.
• 1:01:11
  It's okay. It's just that it's confusing
• 1:01:13
  for users and for customers what exactly the value is that
• 1:01:17
  they're paying for, they don't know that they're paying for its
• 1:01:21
  potential. That some companies they're just
• 1:01:22
  paying for something they could download online and that's not
• 1:01:25
  made aware or clear to them. I think that's a problem from a
• 1:01:28
  business perspective, and from an honesty perspective with
• 1:01:31
  customers and users. If you're going to be in AI
• 1:01:33
  company, ideally, and this is not the world, we live in, but
• 1:01:36
  in an ideal world, you would be paying for it.
• 1:01:38
  Patience and Ai and that's why you're actually paying for
• 1:01:41
  something. But and that would also Drive
• 1:01:44
  Innovation and AI which would be good for the world.
• 1:01:46
  But, you know, we don't live in an ideal world.
• 1:01:47
  We live in a, you know, a capitalist one.
• 1:01:50
  We do and it's Cutthroat out there.
• 1:01:52
  There's sharks out there so they gotta eat last one.
• 1:01:58
  Well, What piece of technology? Are you bullish on?
• 1:02:03
  That might surprise people besides clean lab.
• 1:02:06
  Obviously, that wouldn't surprise anybody.
• 1:02:08
  I don't think. Yeah, I'm bullish on a lot of
• 1:02:12
  things. I think long-term, I am pretty
• 1:02:16
  bullish on Quantum, but it's I think it'll take longer for like
• 1:02:19
  the last 30 years they've been saying 10 years, right?
• 1:02:22
  So, but I've seen the work that's done internally, and I
• 1:02:26
  think that it will be fruitful when combined with a classical
• 1:02:29
  computer. I'll just leave it at that
• 1:02:31
  because I think I can tell you got another question.
• 1:02:33
  Damn, that leaves me wanting to know so much more, but we'll
• 1:02:37
  save it for the next one. Last one is Curtis.
• 1:02:40
  How do you want to be remembered?
• 1:02:44
  I want to try to empower people. I don't actually mind how it's
• 1:02:49
  how I get there, but one way is to build a business that makes a
• 1:02:53
  I work for real-world human problems and that's clean lab.
• 1:02:58
  I make some music and I hope that that helps people and its
• 1:03:01
  own way. I tried to do a lot of stuff
• 1:03:04
  back home, I'm from Kentucky. And there's if you if you think
• 1:03:07
  about the field of MLM businesses, you know, a lot of
• 1:03:11
  that. What's that?
• 1:03:12
  It's earned and the revenue and what's created by that and the
• 1:03:15
  opportunities tend to go to the cities and not much of that
• 1:03:19
  usually goes back to places like Kentucky.
• 1:03:21
  So one thing I did is I hired our chief of staff for example
• 1:03:24
  from Kentucky and she works remotely there and just any way
• 1:03:28
  that you could sort of give back.
• 1:03:30
  So just in general and empowering people, creating
• 1:03:32
  opportunity, I tried to you know validate certificates and edx we
• 1:03:36
  did that at scale. We ended up validating literally
• 1:03:40
  tens of millions of certificates and I just want to keep doing
• 1:03:43
  stuff like this until the day, I die.
• 1:03:45
  How do we use technology and the latest Innovations and build new
• 1:03:49
  ones that can Empower people at scale?
• 1:03:52
  That's what I'm here to do. Incredible answer.
• 1:03:55
  You guys hiring? Yeah, we're hiring, we just, we
• 1:03:59
  filled most of our positions will do another big hiring
• 1:04:02
  Sprint next year, but if anybody's interested in devrel
• 1:04:06
  developer relations, that's a pretty awesome job because you
• 1:04:09
  can it's sick because you basically build your own
• 1:04:14
  profile. If you like social media, you
• 1:04:15
  can, you know, use your use the job to your advantage to build
• 1:04:20
  your own following while also helping us out.
• 1:04:22
  So we're hiring for that role and also, you know, some, some
• 1:04:25
  Engineers, if you're a great front-end engineer, Reach Out,
• 1:04:28
  All those tick stalkers out there Curtis up.
• 1:04:35
  So sweet, man, this has been really cool.
• 1:04:38
  How many people did you catch cheating at edx?
• 1:04:41
  Do you have a number? Yeah, so we did it.
• 1:04:43
  We gave a lower bound because it was published.
• 1:04:46
  So, and Academia, you have to when you say something, you
• 1:04:49
  know, you need to back it up and it needs to be falsifiable, but
• 1:04:52
  we said that it was a minimum of one percent and some courses it
• 1:04:56
  was much higher was has high as 10%.
• 1:04:58
  Not true anymore. Now they're running lots of
• 1:05:01
  cheating detection algorithm so I highly recommend that you
• 1:05:03
  don't cheat. You're probably gonna get banned
• 1:05:05
  but in the early days you would have like ten percent of people
• 1:05:09
  just blatantly cheating and not learning, which was really sad.
• 1:05:12
  Yeah, that's a bummer dude. This has been so insightful.
• 1:05:16
  I thank you for coming on here and schooling us to all of this
• 1:05:21
  and now I will never look at data Centric, AI the same way
• 1:05:26
  again. I'm just going to come Lean.
• 1:05:29
  I thought it was marketing bullshit when we got on this
• 1:05:33
  call and now I'm a little bit pot into it.
• 1:05:36
  And as long as I am able to reproduce what you said, maybe
• 1:05:43
  like 1/10 of the ability of what you were able to say.
• 1:05:48
  It means, I think that I'm cool with saying, data-centric Ai and
• 1:05:53
  not feeling like cringe-worthy afterwards.
• 1:05:59
  Totally. Yeah.
• 1:06:00
  It's pretty it's pretty useful stuff.
• 1:06:03
  Sweet man. Well, thanks again.