Latest / Agentic Conversations (formally mlops.community) / The Future of AI and ML in Process Automation // Slater Victoroff // MLOps Coffee Sessions #64

Transcript

The Future of AI and ML in Process Automation // Slater Victoroff // MLOps Coffee Sessions #64 · Agentic Conversations (formally mlops.community)

• 0:16
  Hey, everyone, and welcome to another episode of ml Ops coffee
• 0:20
  sessions. I'm Vishnu and I'm joined here
• 0:23
  as usual by Demetrius. A Demetria happening.
• 0:27
  Everybody. Are you are you at home?
• 0:30
  No traveling right now I am really a jet lagged so if there
• 0:35
  are any mental gaps that happened during this session,
• 0:40
  you know why? Yeah, Demetrius is quite the
• 0:45
  jet-setting traveler these days but we're glad to have him back
• 0:48
  on the podcast, back at home settled in his studio.
• 0:51
  And today, we are joined by a very exciting guest Slater,
• 0:55
  Victor off the CTO and founder of Code data and Slater today is
• 1:00
  going to talk to us about a couple topics that I'm really
• 1:04
  interested in. And I think the community at
• 1:05
  large is really interested in, in terms of unstructured data,
• 1:10
  which many of us are in settings, where data is
• 1:13
  actually, you know, not neatly tabular or or cleanly Define,
• 1:18
  but we're somehow supposed to machine learning on it.
• 1:20
  Let's say image or text and also multimodal machine learning.
• 1:25
  There are rarely cases where all the data that you want.
• 1:29
  Want is in the same format in the same context and Slater is
• 1:32
  going to talk to us a little bit about what that actually.
• 1:35
  What the research in that field looks like how indigo is
• 1:38
  thinking about production, izing it and tell us a little bit more
• 1:41
  about how Indigo production is. Machine learning.
• 1:43
  Thanks for joining us later. Thanks so much for having me.
• 1:45
  I'm really excited to be here. Yeah I'm excited for what's
• 1:49
  going to be a great coffee session but first off before we
• 1:51
  get to the ml up stuff that you know everybody in our community
• 1:54
  here is a lot about. I hear that you're an MMA
• 1:57
  fighter or had a Make sure I'm actually huge UFC fan.
• 2:01
  Kobe was my nice big enough. Biggest favorite fighter, I just
• 2:04
  want to show an MSG actually a couple weeks ago.
• 2:06
  Tell us about that. Yeah, absolutely and not, you
• 2:08
  know, full disclosure I don't fight right now, you know.
• 2:12
  So this is, this is kind of Slater of a past life.
• 2:14
  But growing up, you know, I did a lot of martial arts for kind
• 2:18
  of several years. I did you just karate and I did
• 2:22
  taste to do, and I did some Aikido as well.
• 2:26
  And, you know, it was something that was always kind of
• 2:28
  important. Peace for me, right?
• 2:30
  But never something, you know, with school that I was able to
• 2:32
  really devote myself to in a serious way and, you know, all
• 2:37
  through high school, I was doing a lot of these very academic
• 2:40
  extracurriculars. And for various reasons, I ended
• 2:42
  up taking a gap year in between high school and college.
• 2:47
  And my main goal for the Gap, your was get out of my head,
• 2:51
  right? What can I really lean into
• 2:52
  that? Will help me do that?
• 2:53
  And martial arts was kind of a really clear thing.
• 2:55
  You know, it's been something I've been doing for my whole
• 2:58
  life and it was great. Dig in a serious way.
• 3:01
  So, you know, there was one of the things that I did during my
• 3:04
  Gap year was that I trained, you know, pretty heavily and MMA.
• 3:07
  I actually went to China for two months to study martial arts.
• 3:11
  And I did a couple of amateur fights which were which were
• 3:15
  great. It was really fun, you know?
• 3:17
  And then and then I learned when you go to an engineering school,
• 3:20
  very few people there are willing to get in a fight, you
• 3:23
  know I was always taught is sort of like a friendly thing
• 3:26
  Engineers. Just just as a rule here.
• 3:29
  If I'd of getting hit in the face so you know, it's been it's
• 3:32
  been a little while since I've fought but I would actually my
• 3:36
  wife is a third-degree black belt in Tae Kwon Do and so that
• 3:41
  was sort of a club that we were in together sort of our first
• 3:44
  year. But you know, we're both retired
• 3:46
  at this point. That is a that's got to be like
• 3:50
  the world's most fun hobby. I got to say, it was fun
• 3:53
  interesting Hobby and it's enough for any of you who are
• 3:56
  thinking about writing in with negative feedback to this
• 3:59
  podcast. I do not.
• 4:01
  Yeah, life will come kick your ass.
• 4:06
  Okay. All right, so we got a flavor of
• 4:09
  your journey into MMA and now I'd actually love to learn a
• 4:12
  little bit about your journey into machine learning.
• 4:15
  You know, you started in Dakota a couple years ago.
• 4:17
  Now in the earlier days of machine learning, tell us about
• 4:20
  how you got started with that. Yeah, so I mean, I would love to
• 4:24
  say that it was a meticulously crafted plot and, you know,
• 4:27
  every step was extremely strategic, but nothing could be
• 4:30
  further from the truth. So I was very lucky, a to fall
• 4:35
  kind of backwards into a very interesting machine learning
• 4:38
  problem at my first internship, which was roughly trying to
• 4:42
  classify videos. According to this big taxonomy
• 4:46
  of learning information was like, 212.
• 4:48
  So you know which topics in biology.
• 4:50
  Does this video teach you? I didn't actually realize it was
• 4:53
  machine learning problem. At the time I ended up getting,
• 4:56
  you know, lucky with an approach and, you know, my IP ended up
• 5:01
  being, you know, part of this patent that got got well for
• 5:04
  this, you know, really, really interesting kind of overall
• 5:06
  system which was cool. And at that point, obviously,
• 5:10
  you know, I knew everything there was to know about machine
• 5:12
  learning and I went back to school and I told one of my
• 5:14
  professors as much and I'll never forget these words this.
• 5:17
  Was in 2012. And I said, the war is over,
• 5:20
  deep learning lost, right? And I was.
• 5:24
  So I was so completely, sure of myself, right?
• 5:26
  I'm starting sophomore year and then I started doing kaggle
• 5:29
  competitions with the person that would go on to become my
• 5:32
  co-founder. Alec Radford, and it was really
• 5:35
  interesting because for the first six months, I was feeling
• 5:39
  really good about my traditional techniques, right?
• 5:41
  And when I say traditional techniques, right?
• 5:42
  I mean, this is tf-idf, right? This is logistic regression,
• 5:46
  this is svm's, this is No, like keyword features you know,
• 5:50
  meticulously tweaks here and there with you know, interesting
• 5:54
  weighting schemes and stuff and I was pretty good at that stuff.
• 5:58
  But deep learning really starting to come true score for
• 6:02
  right. You know, 2012 obviously that's
• 6:04
  a really really important year, you know, Alex and it sort of
• 6:06
  came out and you know, we started to do these increasingly
• 6:09
  ambitious capital projects, where we were dealing with text
• 6:12
  and dealing with images and you know, I'll never forget, there
• 6:16
  was the point where just the He's deep learning techniques,
• 6:19
  raced ahead, right. It felt very very tangible.
• 6:21
  The second we kind of God, you know, a modern CNN up on one of
• 6:24
  these problems. And at first, I didn't want to
• 6:27
  accept it and I went into all of these different research arcs
• 6:30
  and I, you know, II did novel research write.
• 6:33
  And I really, you know, kind of tried to push the limited field
• 6:35
  and I succeeded in that, and I got some really great results,
• 6:40
  but I was not capable of meeting any of the deep learning
• 6:43
  techniques. Right?
• 6:44
  And so over the course of the next three months, after that
• 6:47
  first, you know, You know, loss, if you will, I do Vin.
• 6:51
  I tried to beat these techniques and I eventually realized that I
• 6:55
  couldn't why I couldn't, and I decided it was a lot more
• 6:58
  productive to instead figure out how you could take this
• 7:00
  technology and make it accessible and make it sort of
• 7:04
  Enterprise grade. Got it and that and from that
• 7:10
  Indigo was born. Is that correct?
• 7:12
  That's exactly right. You know you know and you know
• 7:15
  it started as exactly. That's exactly, right.
• 7:19
  Got it. And so I want to talk a little
• 7:22
  bit about what indigo's actually doing.
• 7:24
  Absolutely you guys call yourself.
• 7:27
  The unstructured data company. Interesting tagline, can you
• 7:31
  tell us? What do you how you define
• 7:34
  unstructured data and what Indigo actually provides in
• 7:37
  terms of making that data more useful.
• 7:40
  Absolutely. And I'll roll back the clock a
• 7:43
  little bit right to wear that just ended in kind of craft that
• 7:45
  art because I think it's highly relevant the keeper.
• 7:50
  That we recognize when it came to using deep learning in an
• 7:55
  accessible. Way was training data
• 7:57
  fundamentally, right? You know, it requires far too
• 8:00
  much, right? It is to inaccessible, you know,
• 8:02
  you don't have a hundred thousand examples for Relevant
• 8:04
  problems. We were very, very early to a
• 8:07
  space that has sent us come to absolutely dominate the field of
• 8:10
  ml, right? Which is transfer.
• 8:11
  Why. And you know, so we just just
• 8:15
  for some reference. I think, most people believe
• 8:17
  that transfer learning with Large scale language models and
• 8:21
  text kind of started with Burt, which was 2020, something like
• 8:25
  that. We had our first large-scale
• 8:28
  language model for text, you know, in language using
• 8:33
  transform learning deployed to production in 2015.
• 8:38
  And you know, it was an RNN, you know, Transformers didn't exist
• 8:41
  yet but you know. So we've been we've been at kind
• 8:43
  of that that Art and Science aspect for quite some time.
• 8:46
  But the thing that's really interesting is about the
• 8:48
  implications of Transformer. Right.
• 8:50
  Whistling that very, very broad field.
• 8:51
  You know, I would kind of argue with this point, you know, if
• 8:54
  anyone isn't using transforming their kind of behind the curve.
• 8:56
  It's just such an amazing effective technique.
• 9:02
  However, right, it, introduces some interesting constraints,
• 9:05
  right? Is that actually, it's so easy
• 9:08
  to build these new models, actually, then kind of holding
• 9:11
  them in memory. And, you know, how do you
• 9:13
  provide, you know, a thousand different models that you might
• 9:15
  need for some, you know, broader workflow that becomes a really
• 9:19
  difficult. Cult and tricky problem.
• 9:21
  And then the thing that's really interesting and Powerful is when
• 9:23
  you can then train these production quality models with,
• 9:27
  you know, 100 200, you know, 500 even data points, right?
• 9:31
  You transform it from something that takes, you know, research
• 9:34
  team and months to something that you can do, you know, in,
• 9:38
  you know, an afternoon or, you know, a week, if you're busy
• 9:40
  right. And that really is the core of
• 9:43
  our platform, right? Is this training harness, if you
• 9:47
  will, that allows People importantly to create sort of
• 9:52
  these arbitrary unstructured workflows right based on this
• 9:57
  kind of very very Cutting Edge ml.
• 10:01
  Got it. That makes a lot of sense I'm
• 10:03
  excited to dig into that. One thing I did want to mention
• 10:06
  is I think it's so cool to hear from you this historical
• 10:09
  perspective for how and how it can be sort of contextualizes
• 10:13
  your journey through ML. And what, even what you do it in
• 10:17
  the cone. Now I think you know, there's
• 10:20
  just so much to learn from how you know, machine learning was
• 10:23
  done prior to the invention of just a large-scale neural
• 10:27
  networks because I think there's a lot of techniques that are
• 10:29
  helpful. Like, you know, I think I think
• 10:33
  some things that are really, really interesting is
• 10:36
  understanding which techniques have become redundant and which
• 10:38
  techniques are now more effective, right.
• 10:41
  Because it's had, its had that approach on both sides right.
• 10:43
  On the one hand, write all of these insane like stemming and
• 10:46
  limitation T techniques, right? Absolutely.
• 10:49
  Don't use them like terrible waste of time now, right?
• 10:52
  Modern techniques. It they'll just take care of
• 10:53
  that, right? You know, on the flip side, when
• 10:56
  you look at a lot of the indexing techniques that we use
• 10:58
  for search engines, those are now amazingly useful.
• 11:00
  Ville because suddenly you can generate inputs for them that
• 11:03
  are, you know, ten times better than you could before
• 11:05
  representing data in a richer way than you could ever imagine.
• 11:08
  Right? So I think you see, both of
• 11:10
  those right where these traditional techniques are now
• 11:13
  you know, 10 times more powerful and in other places, you know,
• 11:16
  maybe more on the traditional design side where they become
• 11:19
  less useful, but it's still helpful to understand what kinds
• 11:24
  of problems you can run into bright.
• 11:25
  I think it's helpful context in any case.
• 11:28
  So I feel like there's Been all these advances in this space
• 11:34
  especially with unstructured data.
• 11:37
  And it feels to me like the adoption of unstructured
• 11:44
  Technologies like has remained low.
• 11:47
  Do you have any ideas for why that is, is it just still people
• 11:52
  trying to figure out some very common problems that are, that
• 11:58
  can be solved with technology and it the word hasn't gotten
• 12:02
  out about it or is it because they're even at step 0.
• 12:06
  And they're trying to figure out how to make machine learning
• 12:11
  actually valuable for the company.
• 12:14
  Also, as a really, really good question and maybe just to, I
• 12:18
  realize, I didn't answer the issues earlier, question about
• 12:20
  defining unstructured data, right?
• 12:22
  So for me right, unstructured data is, you know, everything
• 12:25
  that's not rows and column. So it's text its audio, its
• 12:28
  image, and it's every combination.
• 12:30
  I was right. So that means it's video right?
• 12:33
  It's documents which are really interesting when we can talk
• 12:35
  about and you know all of these other kind of different
• 12:38
  multimodal pieces. Importantly, I would say
• 12:40
  unstructured also encompasses where you've got those
• 12:43
  unstructured sources that are perhaps and joined with some
• 12:47
  structured metadata. So maybe you've got, you know,
• 12:51
  when author on the PDF document for instance, I'd say that
• 12:54
  that's still an unstructured use case or not structured document,
• 12:58
  okay? So then to talk about why Why,
• 13:00
  why do you do option is tough, right?
• 13:02
  I think that there's there's really two key pieces and the
• 13:06
  most important one I think, is this really big knee and
• 13:08
  Technology, right? Is when you look at, there's
• 13:12
  very clearly easier techniques in ml, right?
• 13:15
  When you look at, you know, if you have kind of really
• 13:18
  rigorously laid out forms for instance, you can make a
• 13:21
  template that is heavily Reliant upon X Y position there, right?
• 13:25
  And it's going to work really, really well for that, very
• 13:28
  particular kind of document. Now, the problem is That
• 13:30
  technique is just really fundamentally different from
• 13:34
  what you would need to put in place.
• 13:36
  If you were going to, for instance, do sort of a more
• 13:38
  sophisticated large-scale language modeling technique that
• 13:41
  actually you know uses context and takes advantage of these
• 13:44
  modern advances on that same document.
• 13:47
  And that's actually one of the really big rubs is when you look
• 13:50
  at the vast majority of people in this space, right?
• 13:53
  Is they are not necessarily very sophisticated on the GPU and
• 13:59
  infrastructure side of things. Right.
• 14:01
  And their ability to actually tackle the and then it's
• 14:04
  actually funny because there's multiple layers it, which
• 14:06
  unstructured is defined. So there's unstructured, you
• 14:10
  know, sort of the broad if you're in the ml space.
• 14:12
  I think the definition I just used up on structured is very
• 14:14
  normal. Like that's just what that term
• 14:15
  means within the unstructured space, if you will, there's a
• 14:20
  further delineation of structured versus I'm
• 14:22
  structured, that's kind of what I was just talking about where
• 14:25
  you then start to refer to these forms that have a Static X Y
• 14:29
  position, that's them. Actual data and then
• 14:31
  unstructured becomes sort of long form contract type
• 14:34
  documents, right? Anything that just really
• 14:36
  linguistically, you know, heavy right?
• 14:39
  That really relies on that language information and you
• 14:42
  know, obviously the nomenclature is super confusing because
• 14:46
  you've got that weird nesting definition to it but it's You
• 14:53
  know, people have kind of sliced off parts of the really, really
• 14:56
  large unstructured umbrella that are addressable.
• 14:59
  And when you look at the advancement of those
• 15:00
  Technologies, I mean adoption is going absolutely Bonkers, crazy
• 15:04
  bananas. I mean that stuff is deployed
• 15:05
  really really heavily and I think the other thing is that
• 15:09
  when you look at things that really happened on the
• 15:11
  instructor side on these language pieces, those use cases
• 15:14
  have historically been so valuable and so difficult to get
• 15:17
  after you tend to see them in these really verticalized
• 15:19
  companies, right? So you'll see a company that
• 15:21
  just does lie. Analysis.
• 15:23
  Right. And only a super successful
• 15:25
  company using these techniques right?
• 15:27
  But it's not a generic unstructured data company,
• 15:29
  right? Because it took them, you know,
• 15:31
  two years to build that model that does that thing.
• 15:34
  And, you know, I think that since then, there has been a
• 15:37
  really big shift, people are starting to realize, oh, wait a
• 15:39
  minute, I don't have 150 problems, right?
• 15:42
  I have one problem that is, how do I could do unstructured
• 15:45
  stuff? Right?
• 15:47
  And it was actually really funny after.
• 15:49
  We did our big announcement clarify, they and then came out.
• 15:52
  And as you know, you know, the other unstructured data company,
• 15:56
  or whatever, they call themselves like two weeks later.
• 15:59
  So it was interesting where you seen it addressed in a lot of
• 16:02
  part pieces and now because the technology is kind of advanced
• 16:06
  so far, that it is significantly easier to take up some of these
• 16:09
  modeling techniques and enough people have seen these really
• 16:12
  dramatic success has people like, oh wait a minute, you
• 16:14
  know, this is actually kind of one problem set that you can
• 16:17
  address with a unified technology stack.
• 16:20
  Yeah, that makes a lot of sense and just so I make sure I'm
• 16:24
  understanding correctly. What you're kind of saying is
• 16:26
  unstructured data, is this realm of non tabular information?
• 16:31
  Let's just call it that, right? And it's an image audio, you
• 16:35
  know, text, any kinds of things like of that nature, which is a
• 16:39
  lot of use cases, and in a lot of companies, you know, the way
• 16:43
  that such unstructured data has been solved and that you create
• 16:46
  a business value with this is by focusing in on some narrow part.
• 16:50
  Of that question, right? So something that comes to mind
• 16:52
  for me, it is a company like Flatiron Health, where they were
• 16:55
  working with electronic medical records that were scanned PDFs.
• 16:57
  And they basically just built a whole architecture of human
• 17:01
  abstractors that turn those PDFs into, you know, sort of more
• 17:05
  structured information that they could do machine learning on and
• 17:07
  there's countless stories of that in a number of different
• 17:10
  Industries. When you say Technologies, can
• 17:13
  you tell me a little bit more? Specifically, what particular
• 17:15
  things that you believe, you know, that you're talking about?
• 17:18
  Is it machine learning modeling? Is it, you know, more data
• 17:21
  engineering, what kinds of things are you talking about
• 17:24
  there? Yeah, so I mean I'm primarily
• 17:26
  talking about right now really the modeling piece of things,
• 17:28
  right? So when we talk about, for
• 17:30
  instance, what we were using in this dovetails, a bit into the
• 17:32
  multimodal EI piece, right? We actually use these sort of
• 17:35
  Next Generation, I don't know. Like, mutant X-Men language
• 17:39
  models, I might call them, right?
• 17:41
  But the idea is that when you look at documents and this is
• 17:45
  really, it's not only true of documents documents, just
• 17:48
  happened to be, you know, an Using Lee valuable use case and
• 17:51
  like, everyone knows what it is. You might at first think that a
• 17:55
  document is text and you would be wrong.
• 17:59
  We thought documents for text and we were wrong.
• 18:01
  That's how we know. And the way you find that you
• 18:05
  are wrong, is that if you look at someone who say processes
• 18:08
  invoices for a living or reads contract for a living, you take
• 18:11
  your text and you OCR it and you show them, you know, the OCR of
• 18:15
  that document, right? They can no longer do their job.
• 18:20
  And I don't mean their job is now difficult.
• 18:22
  I mean literally without that visual formatting their job is
• 18:27
  now ambiguous. Right?
• 18:28
  That document actually loses much of the information that is
• 18:31
  key for interpretation, right? And that's something that I
• 18:34
  think people tend to overlook right?
• 18:36
  They think oh if I'm just good enough at language modeling I
• 18:38
  will eventually figure it out. Absolutely not just not how it
• 18:41
  works if you want the most obvious example of this, think
• 18:44
  about a table that has numbers and nesting right?
• 18:48
  There is no amount of language model.
• 18:50
  Saying that will account for the differences in read order and
• 18:54
  nesting that you got there, right?
• 18:56
  And, you know, not not even to mention things like Rich text
• 18:58
  and embedded images, right? This is like the tip of the
• 19:01
  iceberg, but that's why and you seen through this big Resurgence
• 19:05
  now in some of the top area in the some of the top Labs like
• 19:09
  Microsoft and Amazon, right? Where people are using these
• 19:12
  multimodal techniques. So the idea is you start with
• 19:15
  this language, modeling back and obviously the language
• 19:17
  understanding is really critical.
• 19:20
  Rather than just thinking about linguistic aspect, you find ways
• 19:23
  of encoding, sort of the local visual information.
• 19:26
  Write the position on the document, write the position of
• 19:29
  the document potentially in some larger bundle, right?
• 19:33
  All of these additional pieces of information and you get, I
• 19:36
  mean, not just radically more effective models, right?
• 19:39
  I mean, it just works way way better, but you start to see
• 19:41
  this really interesting glimpse of the future of ml where, you
• 19:44
  know, traditionally Texas, NLP, right?
• 19:47
  Vision is computer vision, right?
• 19:49
  And when you think about a lot of of the most compelling cases,
• 19:51
  like, you know, self-driving cars, you know, anything with
• 19:54
  video, right? Even if you're talking about
• 19:56
  simply blending metadata and text right in customer chap
• 20:02
  surface. Right, you know, really basic
• 20:04
  stuff like that. We don't really have the tools
• 20:05
  to do that today. But these multimodal techniques
• 20:08
  allow you to kind of take a language modeling back-end.
• 20:11
  Tweak them in sort of these Point ways of technique that we
• 20:13
  use is called Model surgery. It's a little bit Niche.
• 20:16
  You know, opening I developed it in the DOTA but there's other
• 20:19
  techniques as well. Reproaches.
• 20:20
  That's just what we use to add in this additional sort of
• 20:24
  structured metadata which is just it's very very interesting
• 20:29
  and you know it's been really effective for us.
• 20:33
  Really helpful example with the with the documents.
• 20:35
  Because I completely see what you're saying, they're right.
• 20:37
  It's not just a bag of words, it's actually their structure
• 20:41
  that's there that are hard. That's just right now, you know,
• 20:44
  our technology can make it hard in terms of actually applying
• 20:48
  that structure in machine learning context.
• 20:49
  And, you know, one of the fundamental things was
• 20:54
  challenges that I hear in sort of what you're saying is again,
• 20:57
  the modeling side, we have an ability to deal and make use of
• 21:02
  more. Then we have before, right?
• 21:04
  In more creative ways. We have more techniques.
• 21:07
  We have more powerful networks, but as challenge that I still
• 21:11
  see in my day-to-day, and I think a lot of people see is on
• 21:14
  the data side of that, right? Which is how do you go?
• 21:18
  It's, you know, it's cool to hear that, you know,
• 21:20
  unstructured data can be used in machine learning context more,
• 21:22
  but we still need to set it up. You know, you know, in
• 21:26
  somewhere, that's not just an S3 bucket or in something that's
• 21:29
  not just, you know, a loose collection.
• 21:33
  Section of files right there has to be some sort of data, schema,
• 21:37
  or some sort of structure applied to that at least so
• 21:39
  that, you know, perhaps for some analytical reasons or just
• 21:42
  because that's more efficient, right?
• 21:45
  And so this kind of gets us into the question of, like, ETL and
• 21:48
  data Engineering in the context of unstructured data, which we
• 21:52
  want to do more, you know, sort of machine learning on and you
• 21:56
  were telling us before we started this conversation, but
• 21:59
  those are some areas that your experience, a lot of challenges.
• 22:02
  Can you tell About how the data engineering side of some of this
• 22:05
  unstructured data from the challenges you experience and
• 22:08
  how you solve them. Absolutely.
• 22:10
  And I've even I'm going to jump as far as to say that modeling
• 22:16
  is simply not the limit anymore, you know?
• 22:20
  We we have models to radically effective right?
• 22:23
  You know, and I think that while there are obviously people doing
• 22:26
  great work, there is very much, not a limiting factor and I
• 22:28
  think the data is more than anything, the one big factor and
• 22:32
  I mean that, That in a couple of actually really important ways.
• 22:36
  And this comes down to to these questions of ETL and how you
• 22:39
  handle your data. And in a lot of ways, why
• 22:41
  unstructured data is such a challenging ETL problem, right?
• 22:46
  And let's use documents again as an example.
• 22:48
  Right now, you've got a scan document and you have what,
• 22:52
  let's just take a very simple kind of workload example.
• 22:56
  You're going to OCR it, right? So you use some OCR engine say
• 23:00
  OCR engine a right and then say, Train some classification model
• 23:05
  that says oh you know this is one of three document types
• 23:08
  right? And then say each one of those
• 23:10
  different document types it gets fed to an extraction model like
• 23:15
  a custom named entity recognition model.
• 23:17
  And then that populate some payload that is structure,
• 23:19
  right? So you know, really pretty
• 23:21
  simple, you know, very relatively straightforward here
• 23:25
  is what is really tricky about it?
• 23:26
  If you were to think about it in a standard flow, right?
• 23:29
  If I start by labeling, Labeling some data for that last Model
• 23:35
  that extraction model. And then I find that the OCR
• 23:39
  engine I'm using is not very good and I want to change o CR
• 23:42
  entrance. Normally the way you store that
• 23:46
  label will not allow you to reuse the label right?
• 23:49
  Because the text is change. The read order is changed right?
• 23:51
  Figuring out what someone actually labeled you actually.
• 23:55
  You can't really do it in a traditional flow.
• 23:57
  So, not only do you have to re trained all of your models, but
• 24:01
  you have to relabel all of your data and this kind of dependency
• 24:06
  mismatch, right? This is something that you see
• 24:09
  all over the place in unstructured data and because
• 24:12
  unstructured, Data. You can't effectively put strong
• 24:15
  constraints around your distributions.
• 24:17
  You can't really change them. And so, an example of how you
• 24:21
  can help that problem kind of get around that problem is
• 24:23
  thinking about sort of the most flexible possible way you can
• 24:28
  represent your information. So for instance, what we do here
• 24:32
  in this case is say okay when someone labels a particular
• 24:36
  value rather than say we're just going to index into the string
• 24:40
  of that document, right? You have both the string of
• 24:42
  that. Community and the position of
• 24:44
  that information because again, it's multimodal information,
• 24:47
  right? And then what that allows you to
• 24:49
  do is if you ever switch OCR engines, right?
• 24:51
  You can actually then reconvert that label back into the text
• 24:54
  format that, you know, that's that's what you need for
• 24:56
  training. And so that's a kind of simple
• 24:59
  example, right? But you you see this?
• 25:02
  I think all all over the place and you're going to see it a lot
• 25:04
  more when you come up with this notion of changing the way that
• 25:08
  you want to do supervision, right?
• 25:12
  And I'll kind of Like throw one other notion out there that I
• 25:15
  think is really key in all of this.
• 25:17
  And this is something that Microsoft has been championing,
• 25:19
  a lot recently called machine teacher and that's the idea.
• 25:23
  And, you know, I kind of said modeling is not a limiting
• 25:25
  factor. Then what is the limiting
• 25:27
  factor, right? And even frankly, you know,
• 25:29
  transporting models today, they can learn on incredibly small
• 25:31
  amounts of data, right? However you need pretty good
• 25:37
  data, right? And importantly, you just need
• 25:39
  consistent data and it turns out this is the thing that's really,
• 25:42
  really hard for Right. Is that the way that we tend to
• 25:45
  start to these ml problems? You know, if you think
• 25:48
  classification, would binary classification of Twitter
• 25:50
  sentiment, right? They're actually very far from
• 25:53
  how humans think about the problem, right?
• 25:56
  And historically, you know, we see 95 percent agreement, for
• 25:59
  whatever on binary Twitter sentiment, and we're like we're
• 26:01
  voting, right, have more people vote at it, right?
• 26:03
  But it never really gets better than that because the problem is
• 26:07
  actually not with people disagreeing.
• 26:09
  The problem is, with the framing of Twitter sentiment as Binary
• 26:13
  positive - right like that framing will only get you so far
• 26:18
  and then you think about more interesting framings because, I
• 26:20
  think everyone sort of parses the sentence the same way.
• 26:23
  So you think you can kind of move in the other direction and
• 26:26
  make some of that reasoning? Explicit have people be like,
• 26:28
  oh, you know, here is the, you know, whatever parse tree.
• 26:31
  And I'm actually using to read through this sentence, these
• 26:33
  positive aspects. These are the negative aspects
• 26:35
  sort of like aspect-oriented sentiment right in a basic, kind
• 26:38
  of way, but thinking of that, as a sort of exclusive supervision
• 26:42
  that he used, would provide and you know, so yeah, I think
• 26:46
  that's a really interesting Trend and it has some
• 26:49
  interesting implications for ETL Yeah, I think this whole this
• 26:56
  particular quote that you have around like changing the way you
• 26:59
  want to do supervision. I think that that that that's it
• 27:02
  that's a huge lesson. I think that I'm seeing happen
• 27:05
  right now and I think it's funny how this happens.
• 27:07
  You know I think the way that happens in companies, it's not
• 27:09
  you know somebody saying okay guys we're going to go from you
• 27:12
  know supervised learning to unsupervised or something like
• 27:15
  that where people are applying like rigorous definitions to the
• 27:17
  kind of technique they're using. They're saying hey these are the
• 27:21
  fields that we More confidence and ease of use, we have less
• 27:23
  confidence and we need to get more data for the stuff that we
• 27:26
  have less confidence. And how do we do that, right?
• 27:28
  What are the techniques that we can find modeling side?
• 27:31
  What are the things that we can do from a data collection and
• 27:33
  sort of cleaning side? That's the way that these
• 27:35
  changes happen. Sort of incrementally at least
• 27:38
  in my experience. And yeah.
• 27:42
  Yeah. No.
• 27:42
  I was they absolutely happening for mentally.
• 27:44
  I think, I think I love the fact that you mentioned the people
• 27:48
  don't use supervised and unsupervised consistently.
• 27:50
  I think it's absolutely correct. Wrecked.
• 27:52
  And I actually think it's a problem with the terms, right?
• 27:55
  A right. So, it makes it sound like they
• 27:57
  were alternatives to each other. They really not write any modern
• 28:01
  system. You're doing supervised learning
• 28:03
  on top of unsupervised, learning.
• 28:04
  That's just, that's just how it works, right?
• 28:07
  They sort of just fundamentally different functions, right?
• 28:11
  Yeah. I like to think of it more in
• 28:13
  terms of like a data spectrum is like you want to have you know
• 28:16
  this Bedrock Foundation model of a lot of unsupervised data just
• 28:21
  like here's what the word. Looks like, and then you've got
• 28:23
  some sort of proxy tasks, right? You know, I call sort of silver
• 28:26
  standard data, things that are probably directionally correct.
• 28:29
  Or, like, summarization is a great example of, like, you
• 28:31
  can't come up with a good summarization metric.
• 28:33
  But you can say, this is a better summarization, we're
• 28:35
  summarization. And it's sort of this, this
• 28:37
  pyramid right of less and less data as you get closer and
• 28:40
  closer to the actual problem that you're trying to solve and
• 28:43
  everyone knows and practice like, you're always dealing with
• 28:45
  less gold standard data than you want.
• 28:47
  But you can you can like pad that out lower on the pyramid.
• 28:51
  I I think that is a good segue into what I wanted to talk about
• 28:55
  next which is synthetic data in the role of synthetic data in ml
• 29:00
  Ops and you had mentioned, let me just grab my notes, you said
• 29:04
  this, before we started recording, which I thought was a
• 29:07
  fascinating term. Some people look at synthetic
• 29:10
  data as this perpetual motion machine.
• 29:13
  I love that mental imagery, but explain to everyone what exactly
• 29:20
  that means for you. Absolutely.
• 29:23
  And and I just have to remember which demon, I'm thinking of, I
• 29:29
  think it's Maxwell's demon, but there's so many of them, you
• 29:32
  know, it's hard to. Yeah, it is.
• 29:34
  Maxwell's demon, right? So this is a famous physics.
• 29:37
  Thought experiment, and I promise it's relevant.
• 29:40
  And the idea is, you know, the second law of Thermodynamics
• 29:43
  obviously is very important. The idea that you can't just
• 29:45
  kind of make free energy out of nothing to, you know, make it
• 29:49
  very, very kind of simpler. You know, things tend Towards
• 29:52
  entropy without external energy and the idea is that okay?
• 29:56
  Say you've got to joined containers, right?
• 29:59
  That have a gas in them, right? And now, the way that
• 30:05
  temperature Works vaguely is that fast particles, you know,
• 30:09
  that's higher temperature, slow particles, that lower
• 30:11
  temperature in these particles, you know, these are gas
• 30:13
  molecules. So the way the, the idea goes is
• 30:16
  say, you had a little a demon Maxwell's demon that, then
• 30:19
  operated this door in between the two Chambers and Always let
• 30:22
  the fast things in one way and always let the slow ones and
• 30:25
  others write and say. And the idea is basically then
• 30:29
  because you created this temperature separation, right?
• 30:32
  You can get work out of that. And so you kind of created this
• 30:35
  energy from nothing. So it is, how is how have you
• 30:38
  done that? What is the problem with that?
• 30:39
  And the thing? And the, the answer, which it
• 30:42
  turns out is super fascinating and interesting, right?
• 30:45
  Is that the information processing done by that demon?
• 30:48
  When it looks at the particle and tells whether It is fast or
• 30:53
  slow that actually has an energy cost right to make that
• 30:57
  decision, right? And that is how it balances out
• 31:00
  and you don't actually get it for free, but one of that, the
• 31:04
  really key pieces is that they show that energy and information
• 31:07
  are, you know, there's an equivalence there and the same
• 31:10
  way that you can't create information out of nothing.
• 31:12
  You can't create energy out of nothing or are, you know, vice
• 31:14
  versa and I think that's why I love the analogy of the
• 31:17
  perpetual motion machine with synthetic data.
• 31:19
  And to be clear, you know, synthetic data has Some really
• 31:22
  important uses and we'll talk about those, you know, after
• 31:25
  this. But, you know, I think people
• 31:28
  importantly sometimes have this notion that because synthetic
• 31:31
  data looks to us like real data that you must be able to by
• 31:35
  creating more and more synthetic this synthetic data improve.
• 31:39
  You know, the models understanding right, say, Hey,
• 31:41
  you know, make me more chairs. Feed the chairs back in, is more
• 31:43
  training data, but it turns out the only thing that you end up
• 31:46
  doing is reinforcing what the model actually believes, right?
• 31:49
  There's no net new Information added in that process and it's
• 31:56
  an important but actually a pretty subtle point and and the
• 31:58
  same kind of fundamental issue. Actually plagues a lot of a lot
• 32:02
  of areas, right? If you think about the generic
• 32:05
  Active Learning task and this is something a lot of people aren't
• 32:07
  aware, of active learning has actually failed to reject the
• 32:11
  null hypothesis, which like let that sink in for a second,
• 32:15
  right? Is like, no one can actually
• 32:16
  prove definitively that Active Learning Works.
• 32:20
  It's like we've all seen the papers right through tons and
• 32:23
  tons of active learning papers, right?
• 32:24
  And you can get active learning to work on any data set, right?
• 32:28
  And it's this really weird Paradox and it turns out that
• 32:32
  actually the thing that's really key is the way that you set
• 32:34
  Active Learning up on that data set, what I would call human
• 32:38
  overfitting, right? Where you see what happens on
• 32:40
  the data set and then you tweaked your hyperparameters,
• 32:41
  you tweak your Active Learning, right?
• 32:43
  That is the information flow, that is the overfitting and
• 32:46
  that's why you can get it to work on any use case, while at
• 32:48
  the same time, you cannot not violate, you know, you cannot
• 32:51
  reject the null hypothesis, you know, just with a generic Active
• 32:55
  Learning approach. So Just so I understand that
• 32:59
  real quick. You're basically saying I kind
• 33:04
  of see what you're saying with the information energy.
• 33:06
  You can't just can't just make it about us than are right and
• 33:08
  it has to represent. Something has to cost something,
• 33:11
  but on the active learning piece, you're basically saying
• 33:15
  that by just You know, by feeding pieces of data that we
• 33:21
  know that machine learning model is failing on.
• 33:25
  We're engaging in a form of human overfitting.
• 33:29
  Well, let me let me there's an important line in between the
• 33:32
  things that, right? Because you said, we know it is
• 33:34
  failing on, right? And that's actually what's most
• 33:37
  important is, you don't know that in an active learning
• 33:39
  context. If you learned that and say, it
• 33:43
  has failed on this example, then it's actually not active
• 33:45
  learning anymore, right? Then you just labeled the
• 33:47
  example. Right.
• 33:50
  And that's actually Chi Delta, right?
• 33:52
  Is that an active learning we must rely on this proxy for was
• 33:55
  wrong. Uncertainty sampling, obviously
• 33:57
  is the most common one, right? And uncertainty sampling will
• 34:00
  sharpen all your decision boundaries.
• 34:02
  It will not tell you if you have the wrong decision boundary.
• 34:06
  And it turns out that, you know, in practice, as often as you
• 34:08
  want to sharpen a decision, boundary, you want to change a
• 34:10
  decision boundary and you know, obviously you know, you can't
• 34:13
  generically fix that. But at the same time for any
• 34:17
  given problem, you know, given problem will have a reasonable
• 34:20
  trade off between sharpening and, you know, exploring and you
• 34:24
  find that for the given problem, like it will work for that
• 34:27
  problem. Just not generic, got it.
• 34:31
  It's is helpful. You know, I think one of the,
• 34:34
  one of the most helpful things for me in my machine Learning
• 34:36
  Journey was like, taking a theoretical machine.
• 34:39
  Learning course that I really didn't do very well in, but
• 34:41
  really thought me taught me in terms of how to think about, you
• 34:45
  know, distributions and modeling, you know what it is.
• 34:48
  Actually trying to model, right? And what sort of information it
• 34:52
  is that you're trying to gather in order to then, you know,
• 34:55
  predict in the future. And I think what I'm hearing
• 34:58
  from you, you know, as a lot of intuition around that.
• 35:02
  And what I would love to do is talk about how that intuition
• 35:05
  plays out in production systems, right?
• 35:09
  Because I think, right where the challenge of challenge happens
• 35:12
  is these systems to get to scale.
• 35:14
  You know, you know, you're dealing with millions and
• 35:16
  millions and millions of data points, you're dealing with so
• 35:18
  many. Be different.
• 35:19
  You know, features that people start to lose that underlying
• 35:21
  intuition. And then they just say, okay,
• 35:23
  the next model, refresh, we'll just do better, right?
• 35:26
  We'll do better on some metrics and then, you know, ten months
• 35:29
  from now, we look back and say, wait the data drifted, the model
• 35:32
  drifted and our metrics are down, right?
• 35:35
  And so can you talk a little bit?
• 35:37
  You know, it could be in terms of synthetic data or could be a
• 35:41
  little bit in terms of multimodal data or multimodal
• 35:43
  machine learning. How have these lessons around
• 35:47
  thinking about Information and data played out at Indigo and
• 35:53
  what were some examples? Yeah.
• 35:55
  So and you know, there's there's obviously, you know there's a
• 35:57
  couple of things wrapped wrapped out in there, right?
• 36:00
  I think one of the things that's really key that we recognize is
• 36:03
  that The user with the best understanding of process is the
• 36:09
  person best situated to train them.
• 36:11
  All right. And this is something that a lot
• 36:13
  of people Miss. I think it's not usually how
• 36:15
  organizations are set up is on often.
• 36:17
  It is the data scientist responsibility to teach the
• 36:20
  model. Absolutely and I think actually
• 36:23
  the theoretical ml course that you talked about.
• 36:26
  Actually I think that's much closer to what I would assert is
• 36:29
  for the proper role that a scientist as a property to
• 36:32
  scientist, I think is going to be most effectively deployed
• 36:35
  designing It's deciding. Okay, what metric is actually
• 36:39
  important for us to track in this context?
• 36:42
  I think that the world where a data scientist is primarily
• 36:45
  feature engineering or doing hyper parameter optimization
• 36:48
  again. Modeling is just not the
• 36:49
  limiting factor anymore and I think that that is less and
• 36:53
  less. I think in some cases that is
• 36:57
  still worthwhile, I think that it is becoming a lot rarer, I
• 37:01
  think, in most cases data scientist end up, doing a
• 37:03
  disservice by focusing on that. And so we much more instead we
• 37:09
  say, okay look the goal is to engage the subject matter expert
• 37:11
  as much as possible. Get as much information into the
• 37:13
  system as possible, right? And then step the data scientist
• 37:16
  up to basically maintain that system, right?
• 37:18
  Be the architect really of how this plugs together, right?
• 37:23
  And you know, that's that's important.
• 37:24
  Separation, just thinking about who has, what information at
• 37:26
  what point? I think another piece and
• 37:29
  actually, I'm going to kind of point to Teslas version of
• 37:32
  active learning here because I think it's a clever way they get
• 37:35
  around that, that Knocks an active learning, right?
• 37:38
  Because still it is useful. You still do often want to sit
• 37:42
  through, you know which labels are used?
• 37:44
  You know, which examples are useful to label and not and you
• 37:48
  but you have to add information into that system, because as
• 37:50
  we've established, like, generic active learning, it just won't
• 37:52
  work. And the way that Tesla does this
• 37:55
  is that they actually have a human in the loop Active
• 37:57
  Learning. Not in the way.
• 37:59
  You think what, you know, of course, a human labels, it
• 38:01
  Downstream, but they have a human operator decide which
• 38:05
  Active Learning can. Candidates actually get labeled
• 38:08
  which sort of interesting. Right?
• 38:09
  You know we think of this notion of like a weird example as like
• 38:12
  we're to the model but what they have found and you know, from a
• 38:16
  theoretical perspective you inject information into process
• 38:19
  by doing this in quite an effective.
• 38:21
  Way they say, humans are much better at evaluating, what is a
• 38:24
  weird example and what is not and and you know obviously
• 38:28
  Tesla's had quite a lot of success doing this.
• 38:30
  So I think that's that's strong evidence that they're correct
• 38:32
  and I think there's good theoretical justification for
• 38:35
  why that's a good at so, I wanted to touch on something
• 38:41
  that you also mentioned which was Data flows and infra flows
• 38:47
  and like our sorry info information flows.
• 38:51
  And it seems like you, you kind of were talking about that where
• 38:55
  you want the right person? Doing the training, is that what
• 39:00
  you were hinting at when we mentioned that before hand.
• 39:03
  Yeah, so that's, that is another really key actually, the term I
• 39:07
  used earlier that I think is very Useful here is human
• 39:10
  overfitting, right? And I think this is an
• 39:13
  information flow that people don't think about very often.
• 39:16
  It's why it is one of the key reasons.
• 39:17
  I think information flows are such a helpful way of thinking
• 39:20
  about ml systems is that when you as a user, look at the
• 39:26
  result on your test set, right? And in some way, change your
• 39:30
  model to improve that, that is an information flow, right?
• 39:34
  That is overfitting, even though it wasn't done by a machine,
• 39:38
  even though you put a human in the The middle, right?
• 39:40
  It is still over fitting in the same way.
• 39:41
  Now, it happens suddenly, right? And it happens over dozens and
• 39:45
  dozens of generations, but I mean, we've all been there,
• 39:47
  right? Oh, dear tested, it 3.4 tested
• 39:49
  3.5 tested 3.6, right? And and you know, I think no, no
• 39:53
  place. Can you see this more clearly
• 39:54
  than, you know, the way that by do, you know gained imagenet
• 39:57
  back in the day, right, where they were literally, just, you
• 39:59
  know, bending over backwards to try to get, you know, what was
• 40:02
  it, five or ten extra? You know, Ted test set runs a
• 40:05
  day and obviously, you know, it had a dramatic impact but You
• 40:09
  know, we set things up in a very kind of definitive One Way flow
• 40:14
  and, you know, we think it's, you know, at the same time, it's
• 40:16
  very important to focus on quality of the data and
• 40:20
  consistency of the data because, you know, models will not fix a
• 40:27
  broken process. They will only replicating
• 40:30
  working process. This has been a really really
• 40:36
  interesting discussion. I think because of the
• 40:38
  observations that you've made about how data needs to be set
• 40:42
  up. And where I really love to take
• 40:44
  this conversation next isn't talking about the specifics may
• 40:48
  be an example case. How Indigo has done this?
• 40:50
  I think a lot of our listeners in the community, you know, I'm
• 40:53
  sure that they're listening to this and I know I am I'm
• 40:55
  listening to what you're saying and I'm thinking, okay, how do I
• 40:58
  manage my own system based on the lessons?
• 41:00
  I'm learning. Right?
• 41:01
  And what am I doing that? Maybe I can change.
• 41:03
  And I would love to kind of make that a little bit more tangible.
• 41:06
  And talk about links above what, one of an example of having to
• 41:09
  close doing this. So maybe we can start with like,
• 41:12
  you know, what does a typical client for Indica usually look
• 41:15
  like maybe in terms of Industry size or use case?
• 41:19
  Yeah. So you know, we're Fortune 100
• 41:22
  kind of be FSI firms. You know we do we do broader
• 41:25
  stuff than that. But say, contract analysis or
• 41:29
  something along those lines, right?
• 41:30
  Her actually was take annual statements, you know, just maybe
• 41:34
  more folks will be familiar with those.
• 41:35
  They're slightly visually Rich documents where folks alike.
• 41:38
  Here's how things work fear. And the idea is that the
• 41:40
  customer says, hey, you know, we've got some tasks, some
• 41:42
  extraction that we want to do from this, right?
• 41:46
  And, you know, we want to find someone to help us do this
• 41:48
  extraction and we say, okay, great get us two hundred of
• 41:51
  these, right? We're going to help you actually
• 41:53
  label these two hundred that becomes sort of the experiment
• 41:55
  to set up. And, you know, show you how, you
• 41:59
  know, how well this works, right?
• 42:00
  But, but fundamentally the idea is they get the product in
• 42:03
  there. Hands, right?
• 42:04
  They label the 200 examples in they, you know, kind of label.
• 42:07
  Say, here is the information that we want extracted model is
• 42:09
  automatically trained in the background and then sort of Pops
• 42:12
  to life and turns that motion of data extraction into this, you
• 42:16
  know, AI augmented approach. You know, it's it's highly kind
• 42:21
  of composable and configurable. So, the idea is, you got the
• 42:23
  model as an API, right? You've got this interface for
• 42:26
  the labeling, right? You've got upstream, and
• 42:29
  downstream data hooks, right? You can plug these into message
• 42:31
  queues. So, you know, folks will do
• 42:33
  things Things like grade, you know, like a contract in
• 42:37
  Salesforce inject it back into their Salesforce, you I write
• 42:41
  and then maybe they do a correction of the extraction in
• 42:43
  Salesforce, but then they're sending it back to into code to
• 42:46
  do the actual model update. Okay, got it.
• 42:51
  So it's like basically you know, I'm a subject matter expert as
• 42:55
  you put it at the bank, I've gotten access to indigo's.
• 42:59
  Dataset, I take my annual statements, I drop them in.
• 43:01
  I do some labeling and voila, you know, I have sort of a model
• 43:05
  that's sitting in the background but the sort of the API that I
• 43:08
  interact with is basically just taking some particular inputs in
• 43:11
  the future. You know whatever fields in that
• 43:13
  annual report that I'm interested in modeling and gives
• 43:15
  me the outputs that I want. Yeah you know exactly and then
• 43:19
  you obviously you've got Lot of configurability for exactly how
• 43:22
  you want that process to flow, okay.
• 43:24
  And, and when you say like configurability in terms of the
• 43:27
  process, does that mean that you know?
• 43:29
  Okay. Now I'm the I'm the, I'm the
• 43:31
  analyst that has you know, another batch of 1,000 reports
• 43:35
  that I wanted then, you know do the sort of extraction on how
• 43:40
  does that how do you guys usually recommend that process?
• 43:42
  Because I think it's interesting.
• 43:43
  We talk a lot about production ization.
• 43:45
  This is production, ization of like a machine learning model
• 43:47
  solution. Yeah so it's actually the way.
• 43:50
  That we kind of refer to it as staggered look.
• 43:52
  So the idea is that there's not really a distinction for us in
• 43:57
  the way that we operate, right? So you load, you know, even it
• 44:00
  might be easier to think, think you loaded a million documents
• 44:03
  in it, the star right and say it's million documents.
• 44:06
  You're going to be adding a million documents to that every
• 44:09
  year, right? Because that's, I think that's
• 44:10
  more typical to have folks. Folks, working though.
• 44:12
  There's some bandwidth that they've got process at, okay?
• 44:15
  You start, you pick it up, you start labeling, you're like,
• 44:17
  okay, this is the stuff that I want to extract, right?
• 44:20
  And then, you know, as that as you kind of continue working
• 44:23
  through the queue, right, that data is automatically fed into a
• 44:26
  model that is helping you label faster and faster faster, right?
• 44:30
  And then you know you've got configurability for like hey you
• 44:33
  know for this field whatever just like send it through.
• 44:36
  I don't even want to look at it right and the corrections
• 44:39
  generally will then get fed into a staging model.
• 44:42
  So the idea is at some point you say okay like this is good
• 44:45
  enough for us to work production, whatever this is
• 44:46
  like 80% faster than it was before we're just Going to start
• 44:50
  using this kind of cut this Hazard as a production time
• 44:53
  stand. Then the corrections, just as
• 44:55
  you go they all get fed into the staging model, which is it
• 44:58
  starts as a copy, but then, you know, learns its additional
• 45:00
  information and then you sort of opt-in to an advancement.
• 45:03
  So that's why we call it staggered Loop closed, loop
• 45:06
  terrible idea for a thousand reasons.
• 45:08
  You do not want that data feeding back into the model, you
• 45:11
  know, it will screw your model up and then there is no method
• 45:14
  of recovery and everyone is sad. And so that's why that's we kind
• 45:18
  of use that and not not to mention the fact that Was like a
• 45:20
  compliance nightmare. So people then opt into the
• 45:23
  advances or sometimes we'll say, Hey, you know, we added a
• 45:27
  thousand data points, it didn't improve what happened?
• 45:29
  Oh hey, you know, you know Johnny has been labeling wrong
• 45:33
  for the past month kind of thing.
• 45:35
  Okay. So I'm starting to understand
• 45:37
  then how like your subject matter expert is empowered via
• 45:42
  machine learning and this AI augmentation right for their
• 45:45
  workflow and how that machine learning model sits in the
• 45:48
  backend and is gradually sort. The improved without them
• 45:51
  needing to write a single line of, you know, sklearn or
• 45:54
  tensorflow or whatever that makes sense.
• 45:56
  But in this workflow you know I can't imagine that every user
• 46:01
  that you work with is particularly sophisticated with
• 46:04
  machine learning and think through something like data
• 46:06
  drift or think through something like you know maybe you know
• 46:10
  they're dealing with annual reports in 2019 and 2020 the
• 46:13
  format changes. You know those kinds of breaking
• 46:15
  changes happen all the time and in machine Learning Systems in
• 46:19
  my context else. Our claims changed formats from
• 46:21
  Pair, two pair, how do you guys deal with that?
• 46:24
  Yeah. So they don't have to deal with
• 46:25
  it at all. You know, when I was describing
• 46:26
  the staggered Loop, right as it goes in through production.
• 46:29
  So one of the things that's really key is that older
• 46:33
  techniques, if you're relying really heavily on some aspect of
• 46:36
  those formatting, they were just they break, they break hard when
• 46:41
  things. Good actually modern techniques,
• 46:43
  you don't have that issue. Nearly as much, you've got a lot
• 46:46
  of historic variability. Where you start you you know,
• 46:49
  obviously you never Want to train on just one variant of a
• 46:51
  document that's like, yeah. I know, folks do that especially
• 46:54
  with templated stuff but that's exactly why you don't want to do
• 46:57
  that is because they drift and just breaks things.
• 46:59
  So, you know, we're talking 200 documents, we're talking 200
• 47:02
  random documents from the past, you know, decade, right?
• 47:05
  And then that gives you a huge amount that, you know, when you
• 47:08
  see a new format, it's not going to heartbreak, right?
• 47:10
  You do see data Drift But that turns it into sort of this
• 47:12
  gradual thing over time, which then means that by incorporating
• 47:16
  it into your staging model over time.
• 47:18
  It's not a problem right? You know so long You then, you
• 47:20
  know, you can decide how often you want to look at that stage,
• 47:23
  Remodel and flip it over to production, because that's
• 47:25
  what's going to capture the data drift folks.
• 47:28
  Usually don't need to do it for like years because it starts.
• 47:33
  But, you know, I mean, it takes five minutes if you want to, we
• 47:37
  find folks. Often do it will like monthly or
• 47:39
  quarterly or something? Okay.
• 47:43
  Got it. So basically what you're
• 47:45
  describing is almost that the in decode tool encapsulates, a
• 47:49
  process in terms of thinking about data into models into
• 47:53
  model versions and improvements deployments.
• 47:57
  Yeah, I love that. What you were talking about
• 48:00
  earlier, and that idea of like models, won't fix the broken
• 48:04
  process, they're only going to replicate a working process, got
• 48:11
  it. Yeah.
• 48:11
  And I think this is really Is really is a nice lesson for us
• 48:14
  to all kind of coalesce around because a lot of times what I
• 48:16
  think in the community we talk about is process, right?
• 48:20
  And how the process differs that companies given the needs, you
• 48:23
  know, compliance could be a need security, could be a need or
• 48:25
  just, you know, our product is a bit more complicated than
• 48:28
  something else. And that's what makes the
• 48:30
  process of building and tuning and, you know deploying machine
• 48:34
  learning models hard. Yeah.
• 48:35
  And I think there's just maybe one one other note there which
• 48:38
  is that I think a lot of people a lot of times people don't
• 48:41
  recognize that The cost of making a bespoke change to your
• 48:45
  ml model. And this is, this is exactly the
• 48:47
  case, right? If you did anything at all to
• 48:50
  your model, that was specific to the prior version, right?
• 48:53
  Any specific ETL process, right? That breaks really hard, right?
• 48:57
  And then when you've got the new version, everything falls apart.
• 48:59
  So that's one of the things that's really, really key.
• 49:02
  Is that the way that our process works, you are not allowed to do
• 49:05
  that, right? You're not allowed to apply
• 49:07
  custom ml. Sorry custom like ETL processes
• 49:10
  to this workflow, right? You can do You know and pricing
• 49:13
  for like what gets presented to the human but that's the only
• 49:16
  way that you actually can allow this sort of automatic upgrading
• 49:20
  to happen, right? Otherwise you need a data
• 49:22
  scientist to look at it every single time, you know, when you
• 49:24
  look at these systems where you've got, 1,500, regex, has
• 49:27
  you're like, oh, well, works really good for X.
• 49:29
  It's like, yeah, but no one knows how it works, and if
• 49:31
  things change, you will never recover it.
• 49:35
  So is it fair to say that your perspective on the modeling
• 49:38
  question is too, kind of almost put the model in like a lockbox
• 49:43
  and say like don't touch this unless you know, the data flows
• 49:47
  are kind of clean. Yeah, I think that's absolutely
• 49:50
  right. I think that, you know, people
• 49:52
  often think about model understandability and explain
• 49:54
  ability is like I should be interrogating model weights.
• 49:57
  I think that's the absolute wrong way to think about it for
• 50:00
  a hundred different reasons. I think the best way to think
• 50:03
  about it is that these These are programs that we are commanding
• 50:07
  and directing with data. And so the explanation is that
• 50:10
  you are looking for, are in your data, they're not in your model.
• 50:13
  And, you know, in the model, you're just chasing Phantoms,
• 50:16
  right? When the data actually has the
• 50:18
  answers, that's really cool. I really like that way of
• 50:21
  thinking about programming data rather than programming models
• 50:24
  because at the end of the day, that's Tim that's the real
• 50:26
  input, right? It's not this, you know whether
• 50:29
  your layers are, you know, they have you have nine versus 10, it
• 50:33
  doesn't really make a difference.
• 50:34
  Difference. Unless your data really is is,
• 50:37
  you know, was correct as consistent as you put in it goes
• 50:40
  yeah it goes back to the whole idea of synthetic data and how
• 50:42
  just throwing more synthetic data at?
• 50:44
  It isn't going to fix the problem, you know, although you
• 50:48
  may think it will know. It's I mean there's this joke,
• 50:52
  right? Which is if you have 24 hours to
• 50:53
  build a model, right? How do you do it?
• 50:56
  How do you build the best model? You spend 23 hours, labeling
• 50:58
  data and one our building. I'm all right.
• 51:01
  And that's the answer and I think a lot of people like
• 51:03
  they'll spend Like years, trying to figure out how to get out of
• 51:08
  labeling, 50 data points and it's Bonkers to me that's a
• 51:12
  great Point, honestly. Also, because as a
• 51:14
  machine-learning engineer I have been feeling lately that I have
• 51:18
  been criminally under exposed to data engineering.
• 51:21
  And like now I'm putting all my time like learning the history
• 51:24
  of databases and like learning like how different like Data
• 51:27
  Warehouse Systems in there because like that's where all
• 51:29
  the action actually is like this notion of modern data stack and,
• 51:33
  you know, I completely We agree with your notion that the
• 51:35
  modeling part, it kind of feels like, you know, it feels like
• 51:40
  the, you know, the the the outside there has been limits,
• 51:43
  not the limiting factor for us, that's how I talk about.
• 51:45
  It was the limiting factor for several years ago.
• 51:48
  Today. The models are so sophisticated
• 51:51
  and capable that data is our limiting factor, right?
• 51:54
  Yeah. And you know I don't even want
• 51:56
  to go far, go ahead and get dressed.
• 52:00
  It's funny though, too. Because nura EPS is packed.
• 52:05
  At like conference gets a lot of attendees and it feels like the
• 52:11
  shift is happening especially as more and more people are talking
• 52:15
  about it. We've interviewed quite a few
• 52:16
  people on here who are trying to scream from the mountaintops
• 52:19
  that this is the thing. And then you have Andrew Yang
• 52:22
  who's obviously making a lot of waves talking about these kind
• 52:26
  of things but there's something inherently interesting.
• 52:31
  It's like a dog with a bone data scientists and Eels.
• 52:37
  Yeah. I mean, it's a, it's very
• 52:39
  rewarding and very fun to watch the number go up.
• 52:42
  But I do, I was really, really impressed with opening
• 52:46
  isomerization work recently. They were kind of like jumping
• 52:49
  on board with this notion of like, hey look what did we
• 52:51
  create? It was a new way of doing
• 52:53
  supervision and it was awesome. That's really cool.
• 52:57
  I'll have to check that out. I have one last statement to
• 52:59
  make. And one last question, I think
• 53:01
  statement being, I think, you know, We are we talk a lot in
• 53:05
  the community about machine learning engineering and how it
• 53:09
  differs in a similar software engineering, right?
• 53:12
  And you know, if you think about it, I think, you know, maybe the
• 53:14
  future of this whole thing, looks like your software
• 53:17
  Engineers continue to deal with code and machine learning
• 53:19
  Engineers are approach data programmers, right?
• 53:22
  Where they are thinking about their applying the mindset of a
• 53:26
  data scientist in terms of interrogating data and thinking
• 53:28
  about data. But then using the powerful and
• 53:31
  sort of expressive tools that software Engineers have
• 53:34
  available And those kinds of tools are coming out now, you
• 53:36
  know, as this whole production ML and I'm a large movement
• 53:39
  moves forward. Yeah, I totally agree.
• 53:42
  I think the data scientists of the future is, you know, in
• 53:46
  data, architect first even make sense.
• 53:50
  Yeah. They're people that are
• 53:50
  designing data intensive systems with a strong for the
• 53:53
  statistics. Yeah.
• 53:55
  Yeah. Because the statistics in like,
• 53:57
  how the training clothes and how the information flows.
• 53:59
  I mean that is that is just as important as what database
• 54:02
  reviews, right? Right?
• 54:05
  So we're coming up on time here. This has been a great
• 54:07
  conversation and I think just the last question, I'd really
• 54:10
  like to ask is, you know, with your wealth of knowledge, your
• 54:15
  experience like building sort of data intensive systems in a
• 54:19
  number of different contexts using unstructured structured
• 54:22
  data. And with where you see the field
• 54:23
  going, what would you tell a machine-learning engineer in
• 54:27
  2021, who has sort of a mountain of data?
• 54:31
  But it's kind of trying to understand like how to go about
• 54:34
  Building A system that can be that can be robust scalable and
• 54:39
  reliable. Like what advice would you give
• 54:41
  to that person? In terms of what the focus on
• 54:43
  Focus their time on? I would say that the number one
• 54:47
  thing to focus on is making sure you've got a good problem
• 54:51
  definition and what I always say is people want ml to be like a
• 54:55
  genie that shows up and it's like in Oracle it does
• 54:58
  everything perfectly even putting aside to the fact that
• 55:01
  that's not how it works. Imagine it did work.
• 55:04
  That way. How would you know that that's
• 55:07
  always what I tell people like how would you know that it did
• 55:09
  that, right? So you need to start with good
• 55:12
  consistently labeled data, right?
• 55:14
  And then the second thing that I would just say is like recognize
• 55:18
  that these are really big enterprise-grade, you know,
• 55:23
  difficult systems to put together at sort of the very
• 55:26
  edge of what is possible from a complete perspective, right?
• 55:31
  You cannot do it as an individual person, in an
• 55:33
  Enterprise, right? Maybe like it.
• 55:34
  Startup kind of demo thing, you know, perhaps but it's
• 55:37
  recognizing that it is not trivial that folks, will, you
• 55:42
  know, a demo is really, really easy to stand up, but you know,
• 55:45
  we see folks take three years from demo to production and
• 55:50
  those are the lucky ones because they get to production, right?
• 55:53
  Like, look at the failure rates, like the failure rates are real,
• 55:56
  they are cautionary, tales and just you need help to do it.
• 56:01
  Well, I would say and so look for help.
• 56:04
  And there's a lot of help out there.
• 56:06
  I'll say as well. Awesome fascinating man.
• 56:11
  This talk has blown my expectations way out of the
• 56:17
  water. I don't know what I was coming
• 56:19
  in here with but it has gone above and beyond that is for
• 56:26
  sure and I thank you Slater. There were so many gems in here
• 56:30
  and I'm so happy that we got to geek out on this stuff.
• 56:34
  It won't. Is something that I was not
• 56:36
  expecting and next time hopefully I will not be as
• 56:41
  jet-lagged and I can hold my own a little more but I think Vishnu
• 56:46
  did an awesome job fishing. You got any final comments
• 56:49
  before we sign off. I mean, I just want to say thank
• 56:52
  you again Slater for joining us to everyone in the community.
• 56:55
  I would say, like, you know, these are really interesting
• 56:59
  lessons about unstructured data. Your experiences.
• 57:01
  In terms of like making data useful to machine learning
• 57:04
  systems, that I would love to hear your thoughts on, on the
• 57:06
  slack in the discussions channel in the MLS questions, answers
• 57:09
  Channel and a quick shout-out to Slater's Twitter has website
• 57:14
  blog, you can find him online Slater, is there anything else
• 57:18
  you'd like to say to the community in terms of where they
• 57:19
  can find you or what to talk to you about?
• 57:21
  I was lucky enough to grab Slater dot website and so I'll
• 57:24
  try to put a blog post up here soon.
• 57:27
  No, I mean well thanks so much for having me.
• 57:29
  It was a total pleasure like right back at you, you know,
• 57:31
  comes the tons of fun when I and you know, jet lag, sometimes
• 57:36
  make conversations more interesting.
• 57:37
  So you know, I hope it's as good as you know what you look back
• 57:40
  at it. But yeah, awesome.
• 57:42
  This has been really cool, man. I appreciate it.
• 57:45
  Everyone that is still sticking around checkouts later stuff.
• 57:49
  Give us a like subscribe. Add a comment or a review
• 57:53
  wherever you are listening in on.
• 57:55
  And we'll be back next week for another coffee session.
• 57:59
  Thanks y'all.