Latest / Agentic Conversations (formally mlops.community) / How Sama is Improving ML Models to Make AVs Safer // Duncan Curtis // #307

Transcript

How Sama is Improving ML Models to Make AVs Safer // Duncan Curtis // #307 · Agentic Conversations (formally mlops.community)

• 0:00
  Duncan Curtis, the SVP of Product and technology here at
• 0:02
  Summer and the way I like my Coffee, is a triple espresso
• 0:06
  with three shots of caramel. Boom, we are back for another ML
• 0:11
  OPS Community podcast. I'm your host Dimitrios and
• 0:13
  we're talking data again. What do you know, it's 2025 and
• 0:19
  we're still talking about data. We just can't not talk about it
• 0:24
  when it comes to AI and ML. And Duncan is knee deep in the
• 0:30
  data world. He talks to us a lot about how
• 0:34
  to think through your data, your data collection, what that I
• 0:38
  even looks like, things that can be gotchas when it comes to your
• 0:44
  data and your data strategy and robustness.
• 0:50
  Shit, I'm saying a lot of buzzwords.
• 0:52
  Let's just get into this conversation.
• 1:00
  What are you working on these days, man?
• 1:02
  Because you've got so much cool stuff that you've been doing.
• 1:06
  Give me a breakdown of your day in, day out.
• 1:09
  Yeah. So we're really focused on
• 1:13
  empowering AI across some of the largest enterprises in the
• 1:18
  world, whether it's in autonomous driving, whether it's
• 1:21
  in generative AI, the the latest model releases, we've been
• 1:25
  working through on those. And So what I'm constantly
• 1:29
  thinking about is we've got our own workforce, a full time
• 1:32
  workforce and platform. And so I'm just living and
• 1:36
  breathing the how can I both anticipate what's coming next?
• 1:41
  But then how can we also really get the benefit of the human in
• 1:46
  the loop? And so the reason I say that is
• 1:48
  that, you know, so I've seen the evolution of data annotation for
• 1:53
  AI really, you know, the whole core of AI being what you put in
• 1:57
  is what it learns. It's kind of like a kid where if
• 1:59
  you only teach them like sci-fi books, they only know about
• 2:02
  sci-fi topics. And if you never talk about
• 2:04
  economics or something, they're never going to know about it.
• 2:07
  And it started like a long time ago where you've got things
• 2:11
  like, hey, we want to know if there's a dog or a cat in the
• 2:13
  picture. And so you'd go through, you
• 2:15
  know, 10,000 images and that's a dog, that's a cat, that's
• 2:18
  actually a Buffalo. That's not, not what you want
• 2:20
  and what you're, you're interested in.
• 2:22
  And the thing that's interesting is especially as the annotation
• 2:25
  can get more and more complex. So if I, I think of some of the
• 2:28
  stuff we're doing now in autonomous vehicles where you've
• 2:30
  got a LIDAR capturing millions and millions of points every
• 2:36
  second or every frame, you've got multiple cameras, you've got
• 2:39
  radars and ultrasonics and you're getting someone to draw
• 2:46
  and capture the information that's within that and, and
• 2:48
  really codify it for an AI to understand.
• 2:51
  The thing is the human intelligence in the loop is and
• 2:55
  the human in the loop, but I like the eye to also stand as
• 2:58
  the intelligence. It's not about how do you draw?
• 3:01
  Well, even though that is a skill, the actual intelligence
• 3:05
  that you're capturing is, Oh, well, I recognize it's about the
• 3:08
  scene. Oh, I realize that's a car.
• 3:10
  That car, even though it went behind a truck, is the same car.
• 3:13
  And I know that because I can see the little, you know,
• 3:16
  Unicorn sticker on the bumper. Or I can, you know, there's all
• 3:19
  these areas where I'm trying to look at.
• 3:21
  How do we marry up the tooling and technology side that goes
• 3:26
  with the people side for capturing what's important for
• 3:29
  AI and trying to minimize the other work that can easily get
• 3:35
  lumped into it? So, you know, even things like,
• 3:38
  hey, do we allow the AI to, or how do we get the AI to say take
• 3:42
  the first pass on annotating and guessing?
• 3:45
  You know, in most cases it's right, But you know, not, not
• 3:48
  certainly not 100%. How do you you're oh, that car
• 3:51
  with the Unicorn bumper sticker? It doesn't think about the
• 3:53
  Unicorn bumper sticker and it goes, Hey, that's car one.
• 3:55
  And now it's car two since it drove behind a truck and I
• 3:58
  didn't see it for a little while, but it's a lot faster if
• 4:02
  that car is already drawn and you can be like, oh, hey, here's
• 4:05
  where you messed up it. That's actually still car number
• 4:07
  one. That's the same car.
• 4:08
  And so that's a way that we can enable that intelligence to be
• 4:14
  captured at a fraction of the time.
• 4:17
  Yeah. Now the idea of where to plug in
• 4:22
  the human is always fascinating to me.
• 4:25
  And I'm sure you think at lengths about that because it's
• 4:28
  not like you can just haphazardly throw in a human
• 4:33
  step in the pipeline since it can get very expensive.
• 4:37
  And that probably is arguably the most expensive part of the
• 4:40
  pipeline, right? It is in for the vast majority
• 4:43
  of industries. It's the most expensive thing.
• 4:46
  Some specialized industries have more expense in the data
• 4:48
  collection, like if there's very expensive machinery or other
• 4:51
  things involved. But you're right, like just in
• 4:53
  general, it's very much the most expensive part.
• 4:56
  Now, when you do talk about say, let's say, like large language
• 5:00
  models, that the new most expensive part is the cloud
• 5:04
  training, just because the data size they're using is huge, but
• 5:07
  they're also not having humans look at it.
• 5:09
  And that's why we still see problems with like ChatGPT and
• 5:12
  Gemini where you've you're using the Internet at with no, with no
• 5:17
  human filter, which is all of it.
• 5:21
  Exactly. Oh, yeah, yeah, definitely.
• 5:24
  Yeah. We've seen some definite public
• 5:27
  stories around what what sort of happened on that side.
• 5:29
  But yeah, so when we think about where the human goes in, there's
• 5:32
  like a few places and you're right.
• 5:35
  How do I, how do I do it in the minimal way to get the maximum
• 5:39
  benefit to the AI model where we can capture the most
• 5:42
  intelligence. And so there's even steps before
• 5:44
  say like labeling in terms of like how do we pick the right
• 5:47
  data? Because if I go like driving and
• 5:50
  for a really long time, let's say like near hours and hours,
• 5:56
  if I or you think about your commute to work, if you, you
• 5:58
  know, back in the days, you know, pre pandemic when everyone
• 6:00
  was driving to and from work every day of that drive,
• 6:04
  probably only 10 seconds was actually relevant.
• 6:08
  Most of it is you're in your lane, there's a car in front of
• 6:11
  you, car beside you, nothing's happening.
• 6:13
  But when someone cuts you off, maybe something unexpected
• 6:16
  happens. That's actually that like that's
• 6:18
  the, that's the really interesting part.
• 6:21
  And so how do you find those pieces in, in those, in that
• 6:25
  data? So that instead of labeling an
• 6:27
  hour together an hour to get 5 seconds of use, you're actually
• 6:31
  only grabbing that 5 seconds, maybe let's call it 10, maybe a
• 6:34
  little bit around it. And then you can do that again
• 6:37
  and again to, to get that maximum data usage of data value
• 6:42
  to the AI. And so you can use not just in
• 6:45
  that labeling pot, but it's also in what we call curation, where
• 6:48
  you take a larger data set and you want to look at it.
• 6:51
  I want to look at things that are like, how do I find things
• 6:53
  that are interesting? How do I find a good
• 6:55
  distribution of classes? Because and it's not just
• 6:57
  classes, but events. So things like we've seen, you
• 7:00
  know, in the in not just self driving, but across across AI
• 7:06
  tech for years, there's been a lot of bias in the data.
• 7:10
  You know, back in the day of the old back from my video games
• 7:13
  career when we were building, we're taking Fruit Ninja from
• 7:17
  the phone and putting it on on one of the consoles they had
• 7:22
  they had a sensor where you'd go in front of it and do your
• 7:24
  little fruit slicing with your hands.
• 7:28
  It it couldn't pick up some of the people in the office who
• 7:31
  were of color. Oh wow.
• 7:33
  Why? Because it turned out that the
• 7:35
  team who had built it had been a bunch of, you know, white dudes
• 7:39
  and they'd use themselves as data collection to train it on.
• 7:41
  And so it performed poorly both on children and on people of
• 7:45
  color. And so it was like that that I'm
• 7:47
  just trying to say it can be exacerbated in many different
• 7:50
  use cases. Obviously, a lot of people like
• 7:52
  to talk about the current ones with autonomous driving, but
• 7:54
  this is not a new problem. So we can help in curation where
• 7:57
  you're looking at, you know, do you have let's talk about self
• 8:00
  driving cars or autonomous vehicles?
• 8:02
  Do you have enough buses and cars?
• 8:05
  And what about bikes? And what about skateboarders?
• 8:08
  Do you? But then you can also dig in as
• 8:11
  long as you're adding the metadata or have a way to
• 8:13
  generate it. Do we have enough representation
• 8:15
  across different attributes, height, skin tone, a lot of
• 8:20
  different things that can make sure that you're you're, you're
• 8:23
  going to see that and also goes to not just those, but things
• 8:25
  like weather conditions like bright sunny days, sun glare,
• 8:29
  snow, night time rain. There's a lot of different
• 8:33
  attributes that can really make sure that you want to get good
• 8:37
  distribution to have as best of a chance that you've caught the
• 8:40
  edge case before you see it in production.
• 8:42
  Yeah, finding all of the anomalies is always the fun
• 8:48
  part, I would imagine. But then this robust data set is
• 8:53
  really where the work comes in and trying to think through all
• 8:57
  of the different pieces too there is.
• 8:59
  Yeah. And it's a mix as well because
• 9:01
  you it's not just raw collected data, real world data, you've
• 9:07
  also got synthetic data as part of the mix.
• 9:09
  I know you know, for example, you know, there's a reason why
• 9:13
  Waymo has been around for you know what 1314 years and is
• 9:17
  while they have deployed and have now I think they were just
• 9:19
  said they've passed 50 million rider driverless miles or rider
• 9:23
  only miles, which is an amazing milestone.
• 9:27
  They're they haven't stopped and they're not deployed everywhere
• 9:30
  because the amount of edge cases and simulation you want to do is
• 9:35
  is huge. Because yeah, I mean.
• 9:38
  Yeah, and I, I remember seeing some, you always catch those
• 9:43
  videos that pop up on and do their rounds on the Internet.
• 9:46
  And one was where there was a horse and carriage and it wasn't
• 9:53
  clear because the training data was never with a horse and
• 9:58
  carriage. So again, that's like you'll get
• 10:00
  these anomalies pop up and the model has no idea what to
• 10:04
  classify it as. I that's a great example, yeah.
• 10:08
  You would say, you know, and you couldn't look back at your
• 10:10
  training data is like, what representation of those classes
• 10:13
  do we have? Like is that actually something
• 10:15
  like have we seen four horses? Maybe we've seen 1000 horses,
• 10:19
  but how many carriages or horses and carriages together have we
• 10:23
  seen? One that I like is a relatively
• 10:25
  simple example, but it can it can show that, you know, in
• 10:31
  another way. Another aspect is go back a few
• 10:35
  years and we've got even if you've got good distribution of
• 10:39
  classes, let's take skateboarders for example.
• 10:41
  So OK, we've got the concept of skateboarders.
• 10:44
  It's a human. They've got a little flat thing
• 10:46
  under their feet. They can travel up to, let's
• 10:49
  call it 15 mph maybe in most cases I've.
• 10:53
  Taken you as a KMH guy. I am, but you're converted.
• 10:59
  Make sure that I speak both now and bilingual metric and
• 11:03
  imperial. So, but if we're doing, if
• 11:05
  you're doing up to maybe 15 mph and then this weird thing
• 11:10
  started happening. There was these things that
• 11:12
  looked a lot like skateboards, but they could do 40 or 50.
• 11:16
  And so if you training behaviors on a certain class type and
• 11:20
  you're like, great, it's a human standing upright on a flat board
• 11:24
  and maybe the handle thing, but you're not going to really
• 11:26
  notice that too much. And it's to say going on a
• 11:28
  sidewalk or on a cross street and you're like, OK, well, it
• 11:30
  can probably only do about 15, so I don't need to slow down
• 11:33
  right now. Something you get on tearing
• 11:35
  along at 40 or 50, I mean, that's a recipe for disaster.
• 11:39
  And so being able to identify those and then quickly create,
• 11:43
  reinforcing the model to say, hey, this is a new class type.
• 11:46
  There's actually scooters, a whole different thing and it's a
• 11:49
  really different and we'll, we'll and models can learn the
• 11:51
  difference, but it needs to know who to learn the difference.
• 11:54
  Exactly. You got to retrain it.
• 11:56
  Yeah, it feels like you just got to go collect a bunch of data
• 12:00
  driving around India whenever. Oh man, there's there's a reason
• 12:05
  that no one has even tried to deploy there, man.
• 12:07
  That is like you're looking for like difficulty challenges.
• 12:10
  I'm like, oh, like Rd. rules. What are Rd. rules?
• 12:13
  No, but it somehow works. I remember I got I when I was in
• 12:20
  India for a while, I bought myself a scooter and in the
• 12:24
  beginning it was the most adrenaline inducing thing that I
• 12:29
  could ever do, go out on my own. And for some reason I decided to
• 12:32
  go out on my own with two other friends on the back of the
• 12:35
  scooter because it's India and went in Rome.
• 12:38
  And I remember that by the end of my stint there, there was I
• 12:45
  was passing camels and elephants and whatever else.
• 12:48
  The rickshaws, the don't even talk about the cows that would
• 12:52
  just randomly pop up. Yep.
• 12:54
  And it was no thing to me like on my own internal model had
• 12:58
  been assimilated everything. And I realized, OK, cool camel.
• 13:03
  Yeah, just another day. Yeah, but now you know how
• 13:06
  camels move. Now you know that camels, like
• 13:08
  first time you drive by a camel or an elephant that can crush
• 13:11
  you, you just see like you're like what's going to happen?
• 13:14
  Where is now that you've seen 1000 elephants and you're like,
• 13:17
  oh, it's not going to turn around and crush me.
• 13:18
  It's just, it's going to ignore me because I'm one of 500 in
• 13:21
  this minute that's going to pass it.
• 13:23
  It's going to ignore it. But I really like the fact that
• 13:26
  you, your mental model had to change.
• 13:28
  And that's a really good analogy for how these driving models
• 13:32
  need to adapt and change. And by the way, there's even
• 13:34
  differences regionally between driving styles where you get
• 13:38
  like, not just there's also laws that are different between state
• 13:41
  to state in the US, for example, to Europe to to Asia.
• 13:46
  There are lots of different, lots of different law changes
• 13:49
  that you need to respect built into the model.
• 13:52
  But also people just drive differently.
• 13:54
  You know, you think about, you know, drivers in New York and
• 13:57
  how aggressive they can be. Or you get, you're like, hey,
• 14:00
  make sure that you're driving fast.
• 14:02
  If you're here in California, I know that if you're over in the
• 14:05
  far left, you better be doing at least 10 or 15 over the limit
• 14:09
  because otherwise you're like blocking traffic.
• 14:11
  And so, you know all these like regional differences.
• 14:14
  As well. Exactly.
• 14:15
  Those come with the culture. One thing that I wanted to touch
• 14:18
  on was something that you've lost over earlier and how you
• 14:22
  think a lot about what's coming next.
• 14:25
  And I'm sure you've got some stories of this, like what I
• 14:31
  would coin forward combat compatibility failures.
• 14:36
  And so one story that I heard recently from a friend, I'll
• 14:40
  kind of kick it off and then see what you have in your bag of
• 14:43
  tricks. There is a friend told me he
• 14:46
  spent so much time hacking together all these ways to make
• 14:52
  the context window bigger when ChatGPT first came out.
• 14:57
  Only two in six months have all of that work go right out the
• 15:02
  window as larger context windows became the norm.
• 15:06
  Yeah, that's yeah, that's a great example.
• 15:09
  I'm just trying to think of some that definitely had several of
• 15:13
  those experiences where you're like, hey, let me prepare for
• 15:16
  what I think the next technological or the current
• 15:19
  technological challenges are. And then my God, now I don't
• 15:23
  need to. I think a good, a good example
• 15:26
  for that would have been say Sama's fit for NLP.
• 15:30
  So text based work. So we do a few languages that
• 15:33
  were primarily in East Africa with English as our primary
• 15:37
  language. Yes, we've got some local
• 15:39
  languages as well, but most companies in the NLP or text
• 15:44
  space, natural language processing space, we're really
• 15:47
  looking for you know, 120 languages, something I really
• 15:51
  distributed. And our model where we've been
• 15:54
  focused on, we believe the opportunity is equally, sorry,
• 15:57
  talent is equally distributed, but opportunity is not.
• 16:00
  It means that we've had a very strong presence in Nairobi and
• 16:03
  Kenya as well as in in Gulen, Kampala nearby.
• 16:08
  But it means that we've had a much more primarily English
• 16:11
  based workforce. And so scratching my head around
• 16:16
  like, hey, building plans for where do we want to have if we
• 16:20
  want to continue to expand into that market, do we?
• 16:23
  Because we believe it's a good fit for people who are coming
• 16:27
  out of poverty as trainable. They've got good, good English
• 16:29
  skills or good education level skills, but aren't finding jobs.
• 16:34
  How do we fit that to the, to the AI economy and the work
• 16:38
  available there? And is that an area we want to
• 16:40
  expand into? And so plans upon plans of like
• 16:44
  where in the world we would need to open up new centers in order
• 16:48
  to tap into new, new groups, similar to what we're, we're
• 16:53
  doing in East Africa to really make sure that we're, we're
• 16:56
  keeping true to our mission and not just not just doing
• 16:59
  randomly, but really doing it purposefully.
• 17:02
  And then Gen. AI came around and we were like,
• 17:05
  Oh no, oh, it's all text based except that 8590% of the core
• 17:10
  models in the world English. So in terms of a fit for our
• 17:14
  existing workforce and our like all of our training pipelines,
• 17:17
  our sourcing pipelines was an amazing fit.
• 17:21
  And we're like, OK, so all of that planning, totally waste of
• 17:25
  our time. OK, totally, totally not needed
• 17:27
  at this stage. So Yep, that's a fun example.
• 17:31
  How do you look at really specialized jobs or data that
• 17:41
  needs to be annotated by very specialized or experts and how
• 17:47
  do you think through the whole thing of, hey, this is a very
• 17:52
  high skill that is needed to be able to understand this?
• 17:57
  It's not just like me that can to annotate that data.
• 18:02
  I have to go through years and years of training to be able to
• 18:04
  annotate it. And So what are some things that
• 18:08
  you've been looking at doing in that realm?
• 18:12
  Yeah. So I think that's, it's a really
• 18:14
  interesting space and I think we've seen that big trend, you
• 18:18
  know, sort of within LO Ms. as they got through, you know, as
• 18:20
  the, as the, let's call it, you know, education level of the LO
• 18:24
  Ms. has continued to, to progress.
• 18:26
  Is that we're now seeing a lot of the requests being within,
• 18:30
  let's call it hyper specialized domains where.
• 18:33
  So I, one of my anecdotes that I love to tell the story of is one
• 18:37
  of my most amazing product managers.
• 18:40
  I've got Patrick. He's actually a data scientist
• 18:44
  by training originally. And so he's worked as a data
• 18:47
  scientist, has a advanced mathematics degree and was
• 18:50
  looking through some of these tasks.
• 18:51
  And he goes a dude, I can't do these, like I'm not at the right
• 18:55
  level for, for doing these. And so you, you're talking about
• 18:58
  like really standing into esoteric requirements.
• 19:02
  And so we've seen some of our competitors in the space and
• 19:06
  some of the other players actually build professional high
• 19:10
  domain expertise networks that do that.
• 19:13
  It's not something that we've chosen to do with summer only
• 19:16
  because we're with our mission, we're more focused on how can we
• 19:21
  take, how can we grab more generalized human intelligence
• 19:24
  and some domain expertise. But that level of esoteric
• 19:30
  knowledge, while I think it's really valuable for that
• 19:34
  knowledge to be captured into LLMS for those particular use
• 19:39
  cases, I'm seeing that rapidly accelerate and continue to go.
• 19:44
  And as I've seen in the past where you look at something like
• 19:47
  we've now got Metasam, for example, they can segment
• 19:51
  anything because of all the work that went into that.
• 19:53
  I think that this is to, from my perspective, this is a trend
• 19:57
  that the data will be collected and then that work is going to
• 20:01
  go away. And So what we're looking at is
• 20:05
  what's the next area where we're going to still need human
• 20:11
  intelligence captured, but it's not going to be of that domain
• 20:15
  specificity level because I don't believe that the scale is
• 20:18
  going to be while it's large at the moment for if you look at
• 20:21
  each individual domain, it's quite small.
• 20:24
  Even though it's a well paid, you know, well paying gigs, for
• 20:27
  example. And so for us, this goes back to
• 20:30
  your question of you as the technology changes, what's
• 20:34
  changing for you? I'm looking more say like a
• 20:36
  genetic AI experiences where we're now trying to help provide
• 20:40
  feedback loops to models that are creating a plan and then
• 20:44
  trying to execute steps of a plan by like, hey, OK, so if I
• 20:47
  want to, you know, book you a vacation, you've given me some
• 20:51
  dates and a location. OK, well my plan is OK, let's
• 20:55
  figure out where that is. Let's figure out what you like.
• 20:58
  Maybe it's we wanna book Aruba in the summer and okay, well I
• 21:04
  know you like hiking, so let's see if there's good hiking
• 21:07
  trails. I'm gonna, so I'm gonna go out
• 21:09
  to these different websites and bring that information back and
• 21:12
  create a calendar, generate some hotels nearby that, oh, maybe
• 21:16
  you're vegetarian. So I want to make sure I go to
• 21:19
  websites and can specify that. And so it's going through a lot
• 21:22
  of complex steps to get to the result, which is a really
• 21:25
  amazing capability. But one of the things that
• 21:27
  Gentic AI is being very poor at is when it does a very serial
• 21:33
  set of steps, it does them, It does them one after the other.
• 21:36
  And if it goes off the rails, it finds it very hard to either a,
• 21:40
  know that it went off the rails or B, recover from that.
• 21:43
  Now, I'm not saying that that won't be fixed in the future,
• 21:47
  but that has a lot more breadth and a lot more of the kind of
• 21:51
  what we're talking about edge cases that we were talking about
• 21:53
  earlier in autonomous driving. Think about the edge cases that
• 21:55
  we manage as a human using different tools every time.
• 22:00
  I mean, every time they update, you know, the UI on, on, on your
• 22:04
  social, on your, whatever social media you use, it's a new
• 22:08
  learning experience where you're like, OK, well, I used to go
• 22:10
  here to do this and now I've got to take a look at the screen and
• 22:14
  try to figure out how do I do this now?
• 22:17
  And so you even had to recognize at the beginning, the owner of
• 22:19
  the UI has changed. And so we believe that that's an
• 22:22
  area that is going to be very rich for a long period of time
• 22:27
  as the world and tools that LLMS use is going to continue to
• 22:31
  evolve. So the segment anything is a
• 22:34
  good example of where we've gotten gone through a domain,
• 22:38
  we've gotten a capability and it's pretty solid.
• 22:41
  And so the amount of additional research in that area or
• 22:45
  additional work, unless it's pretty highly specialized could
• 22:49
  be solved by segment anything. So you can take the segment
• 22:52
  anything model and just be like, hey, I'm a startup.
• 22:55
  I'm going to use that instead of trying to label millions of
• 22:59
  images to start from. Now, if your startup is doing
• 23:01
  something really specialized, that's probably not going to
• 23:04
  work. But you're you're going to be
• 23:05
  able to use that segment anything model as a basis and
• 23:08
  you can you can fine tune it from there.
• 23:12
  I see. So it's not saying segment
• 23:14
  anything. Version 5 is going to be so good
• 23:17
  because we're just going to start seeing it get better and
• 23:20
  better. No, I'm saying.
• 23:22
  That it's, it's already replaced a set of like human annotation
• 23:27
  work and AI capabilities that used to be something that there
• 23:30
  was a large, a large amount of interest in.
• 23:32
  So it was like, Hey, can you help me build a model that can
• 23:36
  segment this camera video so that I can figure out what's in
• 23:39
  it? Let's say for a security use
• 23:41
  case or for a like you wanted to do security cameras and be able
• 23:44
  to detect when people were like a sort of like a ringing
• 23:46
  doorbell where you're like, oh, I want to know if a person's
• 23:49
  walking up the path. Well, you can segment anything
• 23:52
  as a, as a base model and you know, maybe something like a
• 23:55
  Yolo and put them together and you can actually have something
• 23:58
  out-of-the-box running like pretty quickly.
• 24:00
  Whereas before the that those capabilities came out and were
• 24:03
  made available, you would have to stop from scratch and put up
• 24:07
  some video cameras and try to collect a lot of data and then
• 24:10
  try to have a whole load of annotation done to get to a
• 24:14
  place that's, yeah, probably maybe a little bit better than
• 24:16
  Sam is to start with, but for a lot more cost.
• 24:19
  Whereas So what I'm trying to say is that almost specialized
• 24:23
  like LLM domains, I think it's going to be the same where we're
• 24:27
  capturing that information and it's going to get encoded into
• 24:30
  these LLMS. And then that's going to be the basis that you
• 24:33
  can work from. So that field is not going to be
• 24:38
  around in the same scale that it is at the moment for an extended
• 24:42
  period of time. Yeah, I, I can see that.
• 24:45
  And I do like this idea of the technology not being the barrier
• 24:51
  to getting something up and running.
• 24:53
  And you really can get super far with what we've got right now.
• 25:00
  So you can go and create a product and there's probably the
• 25:07
  ability to use segment anything in your product like for this,
• 25:11
  for example, this Ring camera, but it there's an.
• 25:16
  There's an interesting thing there is that I love that it
• 25:18
  creates so much more growth potential for start-ups where
• 25:24
  it's like I can pull something together.
• 25:26
  I mean, I could like think about my like university and college
• 25:29
  days. I could be like, oh dude, a
• 25:31
  hackathon over like a 48 hour hackathon with a couple of
• 25:34
  mates. I could, we could put together a
• 25:36
  Ring doorbell suite. Like not a problem like this
• 25:39
  would be amazing, but the counter side to that is, is it
• 25:44
  going to be differentiated? So you're, what you're saying is
• 25:48
  that these technologies will enable new products and the way
• 25:52
  you put them into products is going to be the differentiation,
• 25:55
  but the technology is not going to be the competitive
• 25:58
  differentiator for you because someone else can see your
• 26:01
  product and do it in 48 hours as well.
• 26:04
  Yeah, the. Product and the UI and the user
• 26:07
  experience, all of that is really what we've been seeing
• 26:11
  like it going back to these examples of using agents.
• 26:15
  I think everyone is starting to wake up to the fact that it's a
• 26:21
  lot more than just detect. It's the agent.
• 26:26
  Like how do we interact with the agent?
• 26:28
  What is exactly the best way to make it work?
• 26:31
  So I want to give you. A hyper relevant like I think
• 26:34
  one of the best examples of where design just absolutely
• 26:39
  crushed it over the last couple of years.
• 26:42
  Do you know how long generative AI has been around LMS?
• 26:45
  Basically a bunch of years. Yeah, exactly.
• 26:48
  Transformers came out in what, 2017?
• 26:50
  That paper. 2017 so they've been around for ages.
• 26:53
  The one thing that changed was when you know, Sam and open AI
• 26:59
  stuck a new style of interface on it and turned it into a chat
• 27:03
  like experience and Boo the whole thing took off.
• 27:06
  But the technological change between the sort of ChatGPT
• 27:10
  model that we were using there and what they had six months
• 27:13
  before that and what others had at the time, it wasn't the core
• 27:17
  technology that was the differentiator.
• 27:19
  And I see a lot of these model comparisons where you're looking
• 27:22
  at like benchmarks in different, you know, skill sets and things
• 27:26
  like that. And while I will actually say
• 27:29
  that, yes, we've kind of found out for ourselves and not just
• 27:33
  within my technology teams, but across different teams and
• 27:36
  people I work with, like my marketing department loves
• 27:39
  cloud. It's like the absolute best for
• 27:42
  when you're like doing, wanting to do writing or you're doing
• 27:44
  LinkedIn posts and you're like, hey, I know what I want to
• 27:47
  write, but I want it to be better and I want to take less
• 27:50
  time doing it. The results you get out of cloud
• 27:53
  versus ChatGPT is actually significantly better.
• 27:56
  But in a lot of domains, the tech, the differences are very
• 27:59
  minimal. And it's more about how does it
• 28:02
  integrate into your life. They're like the voice interface
• 28:05
  that you can do with Chad. DBT is just phenomenal.
• 28:09
  One of my, I love it. It was a while ago when that
• 28:11
  first came out, one of my PMS, Ed talked about how his commute,
• 28:17
  he used the voice interaction to get like an extra hour of work
• 28:21
  done. And so he'd get in, get into the
• 28:23
  office and he'd be like, and whether it was professional work
• 28:27
  or his own personal stuff, he would, he would be like having
• 28:30
  active conversations and get through it and then just do a
• 28:33
  dump like when he gets to his destination and be like, all
• 28:35
  right, well, I've already like, I've already got lined up.
• 28:38
  I've either got my research ready for a new product areas
• 28:41
  I'm looking at or I've gotten, I've got my calendar sorted for,
• 28:45
  for, you know, my kids stuff on the weekend, whatever, whatever
• 28:48
  it wants, Yeah. So we had this guy on here a few
• 28:53
  weeks ago named Kenny, and he was really talking about data as
• 28:57
  the greatest bottleneck that we have right now in the AIML
• 29:03
  world. I'm assuming you feel kind of
• 29:07
  the same way You're tackling a similar problem, right?
• 29:10
  Like why do you think that data is so much a bottleneck right
• 29:14
  now? It feels like it's still hard to
• 29:17
  go and explore and work with data, right?
• 29:21
  And it's, it's not a new idea. It's not.
• 29:24
  Data has been hard for a long time.
• 29:26
  I think the why we're seeing it as a greater, even greater
• 29:29
  bottleneck than it's been in the past is the volume of data where
• 29:34
  needed to consume is getting exponentially larger.
• 29:38
  So if I think about like the, the petabytes of data that the
• 29:43
  LLM models are being trained on, it's, it's phenomenal.
• 29:47
  And so when you're, that's where we're starting to see.
• 29:51
  And I'd also say the economic impact of, of things like, you
• 29:55
  know, open AI where you saw, you know, Reddit actually licensed
• 29:59
  their data set to, to Google and they change their tones because
• 30:02
  people had just been in the past, people had just been
• 30:04
  scraping Reddit and sorry, allegedly scraping Reddit and
• 30:09
  using it to train their models. But their tones were very clear.
• 30:13
  And now they're like, but this is actually a hugely valuable
• 30:17
  asset. And that was right before they
• 30:19
  IP owed. And so it was a way for them to
• 30:23
  realize that they're a data they like the, the data that they had
• 30:27
  generated or the users had generated was such a phenomenal
• 30:31
  product that could be unsold. And we're only kind of touching
• 30:35
  like I know we've talked about like the open Internet has
• 30:39
  basically been has, is has been scraped.
• 30:42
  I'd say there's a lot of sub areas like Reddit that are
• 30:45
  getting smarter about their data and licensing it instead.
• 30:50
  But then you've also got the huge data stores that are in
• 30:54
  private companies where maybe interactions with their apps,
• 30:57
  interactions with their services, and that data was
• 31:00
  never collected in a way that was intended to be used like
• 31:05
  this ever again. And so it's, you're ending up
• 31:08
  with this phenomenal data engineering challenge as well,
• 31:11
  let alone the scale and size of data, but also just like, Oh my
• 31:15
  God, it was collected how you did what with it?
• 31:18
  How am I even going to like make this machine readable or, or
• 31:22
  have a human look at this or, or do any of that?
• 31:24
  So there's multiple, I think threads that are making this a,
• 31:27
  a big challenge. Yeah, I, I.
• 31:29
  Remember hearing a Horror Story back in the day of a consultant
• 31:33
  that went in and was saying, OK, we need this data.
• 31:35
  And they were like, yeah, yeah, we got that.
• 31:37
  It's in this database. And she was saying, but it's not
• 31:41
  in this database. I don't see anything since, like
• 31:45
  2011. And they said, huh, that's
• 31:49
  weird. And so someone went and checked
• 31:50
  it out and apparently one of the connectors broke and no one
• 31:54
  knew. Yeah.
• 31:55
  And no one was checking on it. And so they did not have that
• 31:59
  data since 2011. Yeah, well, I mean you.
• 32:02
  Also think about things like the public sector or, or very large,
• 32:05
  slow, highly regulated industries, healthcare and
• 32:09
  others where I had a friend reach out to me the other day
• 32:12
  who was talking about a transformation project they
• 32:14
  were, they were working on. They were taking a system from
• 32:17
  Cobalt to a more modern system. And they're like, we're trying
• 32:21
  to find Cobalt engineers to help us understand how to interact
• 32:25
  with the system to be able to make changes or to at least
• 32:29
  migrate away from it. And it's it the, the wealth of
• 32:34
  hilarious, like, foibles that are going to occur as we go
• 32:38
  forward. Is, is it's.
• 32:39
  It's really difficult as the world modernizes.
• 32:41
  Yeah. Cobalt engineering is back in
• 32:44
  high demand. Exactly.
• 32:45
  Literally, I'm sure like I want to.
• 32:47
  I would love to see the, the, the people either coming out of
• 32:50
  retirement or deciding to pick up that skill, that niche skill
• 32:53
  because of the, the high demand. And like, it's, it's, it's stuff
• 32:56
  that like when I did, did my degree 25 years ago, was an old
• 33:00
  technology and you're like, oh, you guys aren't using C++ and
• 33:05
  you're like, you're not even using C, you're using COPO.
• 33:08
  Wow. I mean, waiting for those
• 33:10
  assembly systems to come back out and we'll just do some, you
• 33:12
  know, roar machine commands. And I'm like, wow, that's
• 33:16
  incredible. So, so talk to me about Samma
• 33:19
  and the product itself, because I feel like that's a great that
• 33:24
  I I want to know exactly what it does and where you're playing in
• 33:28
  cool. So we're really in the business
• 33:32
  of getting clients really highly valuable data sets to drain
• 33:38
  their AI. Now we do things that we don't.
• 33:41
  We have a platform that we use internally.
• 33:43
  It's available, but it's not, we're not in the business of
• 33:46
  take our platform and go, you know, go train a workforce or go
• 33:51
  use your, your other stuff. What we're really here to do is
• 33:54
  we consult with companies at the very early stages when they're
• 33:58
  looking at AI problems and we say, hey, let us help you
• 34:02
  understand AI. I mean, a lot of companies we
• 34:04
  work with, while we do work with a lot of the high tech
• 34:06
  companies, which is great, there's also a big set of people
• 34:09
  who are going through what I'm calling like the AI digital
• 34:11
  transformations or it used to be cloud was the last one.
• 34:14
  And now you're getting to AI where you've got a lot of
• 34:17
  industries that are like, hey, I'm getting a lot of pressure
• 34:20
  from my board. I need to use AI.
• 34:22
  I genuinely think it could be useful for me.
• 34:25
  I can kind of grok how it could be used, but I've got no idea
• 34:28
  how to go from here to there or what order should I do it in.
• 34:32
  So, you know, we talk with people and say like, Hey, what
• 34:34
  are your, what are your top pain points?
• 34:36
  What are your business objectives that you're trying to
• 34:38
  achieve? What are some of your ideas?
• 34:40
  Give us, you know, do workshops with them.
• 34:41
  What's your top ten, top 50, whatever.
• 34:44
  And then let's talk to you about like, and what data do you have,
• 34:47
  by the way, like, as you said, like check on that system.
• 34:49
  Did it stop recording in 2011? Like, oh, that's going to be an
• 34:53
  awkward conversation. And then really understand, like
• 34:56
  help them understand like an ROI basis, Like how, like what are
• 34:59
  the problems with your, your, your company's experience now?
• 35:01
  Is it, is your company like all in on AI and everyone's like,
• 35:05
  Hey, let's go that way. So you can just start on maybe
• 35:08
  the biggest best project and, and go that way.
• 35:10
  Or are you, are you someone who might be a director of AI who's
• 35:15
  who's, or maybe a brand, brand new director of AI at a company
• 35:19
  that's maybe more archaic and set in its ways?
• 35:22
  And what you really need is I need some quick wins, guys.
• 35:25
  Like give me something that I can do in a couple of months
• 35:27
  that's going to get us a win and show some business value.
• 35:31
  And then we can maybe work on to those bigger ones.
• 35:33
  Or maybe we choose projects that have parts of the company that
• 35:37
  are all under one division because it's very easy to align
• 35:41
  everyone to do it. So we work with people there.
• 35:43
  We also work with them on their data collection strategy on,
• 35:47
  hey, how are you going to get the data if you don't have it?
• 35:49
  How do you take what data you end up with, find the right data
• 35:53
  within it. So that curation that I
• 35:54
  mentioned before for and yeah, we do the labeling and
• 35:56
  annotation and there's some, some clients that that's all we
• 35:59
  do for them, you know, as part of our legacy business.
• 36:02
  That's, you know, a large part of part of what we do, even down
• 36:05
  to like, hey, do you need us to help like fine tune models or
• 36:07
  build models like for you and deploy them?
• 36:10
  We do all of that. So the way I like to think about
• 36:13
  what we do is we're really here as a partner on your AI journey,
• 36:17
  depending on what you need. If you're more advanced and you
• 36:20
  just need, you just need data labeling and you need to, you
• 36:23
  need someone to, who's seen this across your industry who can
• 36:26
  help you build what, what we call quality rubrics.
• 36:28
  Like how do you think about what's a good annotation?
• 36:31
  What's a poor annotation? Or do you need to expand
• 36:34
  further? Do you just, you've already got
• 36:35
  your data, you know the problem that you're solving, but you've
• 36:38
  got too much data and you don't have the money to do it.
• 36:40
  Like how can we help you with a mix of automation, curation and,
• 36:44
  and labeling strategies to help you get the most value and make
• 36:47
  sure you've got good class distribution and avoiding
• 36:50
  biases, like having a, a bias plan in place so that you've
• 36:54
  got, you know, you've got a methodical way to go about that.
• 36:57
  Or we go all the way up to, hey, look, I've got a business
• 37:00
  problem and I know I want to use AI.
• 37:03
  Could you help me along? And maybe we don't do absolutely
• 37:06
  everything that you need along the way, But being that we've
• 37:09
  been in the, in the industry for so long, over 15 years, we've
• 37:12
  been doing this is that we can help you.
• 37:15
  We've got partners that we can help with.
• 37:16
  If you've got something that doesn't fit within our
• 37:18
  wheelhouse. What are some?
• 37:20
  Cool business problems you've been solving recently?
• 37:24
  Oh. That's a, that's a good one.
• 37:27
  I'm going to give a slightly older one only because I, I find
• 37:30
  it to be absolutely fascinating. Elephant butts.
• 37:36
  So one of the projects we did a little while ago was it was in
• 37:41
  East Africa and we were helping to track elephants to help with
• 37:48
  anti poaching. And So what it turns out is that
• 37:52
  the back of an elephant is a fingerprint for elephants.
• 37:57
  So they're all unique. And so we were solving a problem
• 38:01
  of how do you, like, we want to use AI to track elephants.
• 38:05
  And it turns out the best way to do that is by having camera
• 38:09
  footage or whether you're using drones or whether you're using
• 38:11
  fixed cameras, having a mechanism that can detect
• 38:15
  elephant butts and then compare them.
• 38:18
  The hands are just giving your your comment about India and
• 38:21
  driving past elephants before I thought that'd be a good fun one
• 38:24
  for you. At a larger scale, though, I
• 38:28
  think that some of the interesting problems are
• 38:32
  meeting. I'm just trying to think about
• 38:33
  it. A specific example where we had
• 38:35
  a client who's has I'm just trying to think is getting to
• 38:40
  the stage of validation with their model.
• 38:43
  So they're, they've now got like good capabilities, but they're
• 38:46
  suddenly realizing the scope and scale of validation and how pure
• 38:53
  manual annotation like they've been doing for a while is not
• 38:57
  going to cut, cut it for the scale they need to get to and
• 39:01
  trying to find the the right balance.
• 39:04
  And not just balance, but the right approach and methodology
• 39:08
  in order to validate across large volumes of data with the
• 39:13
  human in the loop at the right points.
• 39:15
  And for the most part with humans in the loop.
• 39:19
  As opposed to like just doing the basic math where if they
• 39:22
  continue to do say let's say single frame full annotation
• 39:26
  from scratch in a validation use case, it was just going to be
• 39:30
  exorbitantly expensive and it just didn't make fiscal sense.
• 39:34
  And so managing through with that partner and talking through
• 39:37
  the strategies of like, OK, well, what if instead of looking
• 39:40
  at individual frames, we can look at larger sequences and we
• 39:44
  can look at like medium to long length sequences.
• 39:47
  So that there's a lot of technologies you can use where
• 39:49
  you could say only annotate frame one and frame 20 and or
• 39:54
  frame one and frame 50 and you let AI do the interpolation or
• 39:58
  extrapolate out further for you. There's a lot of technologies
• 40:01
  that can help reduce the sort of.
• 40:05
  Cost per annotation while keeping that human intelligence
• 40:09
  high, high in the mix there. So as well as strategies as I've
• 40:13
  mentioned around like what data do you choose to validate?
• 40:15
  How do you pick the right things?
• 40:17
  How do you have good distribution goodbye.
• 40:19
  Like so that you're not just annotate everything and then
• 40:22
  hope or at least then observe what you did annotate, but be
• 40:26
  proactive. Like there are tools that can
• 40:28
  let you be much more accurate in your prediction of what's in
• 40:31
  this data. So that you're saying this is
• 40:34
  statistically the better data for us to go after.
• 40:36
  And yes, we'll do a check afterwards to make sure that we
• 40:38
  were right or to see where it it finally ended up.
• 40:41
  But much better approach than saying large scale and take a
• 40:46
  look after what's. Yeah, so.
• 40:49
  You know, one thing that I hear a bunch of people talk about and
• 40:53
  it's a great topic to noodle on is really the ROI of different
• 41:02
  projects. Because for many of these use
• 41:07
  cases I'm sure that you deal with, you have to make a strong
• 41:12
  case or the your champion inside of the company has to make a
• 41:17
  strong case for what they're doing and why they're doing it.
• 41:20
  Even if the board saying we need AI, we got to be AI first.
• 41:25
  All of that, you still don't want to just create some kind of
• 41:29
  AHR chatbot that six months from now or a year down the line, you
• 41:32
  realize has absolutely no value for the company, right?
• 41:38
  So are there ways that you've looked at, maybe there's rubrics
• 41:42
  like you were saying, or maybe there's just different scenarios
• 41:46
  that you've walked someone through that will help
• 41:50
  illustrate the different ways that you can find business
• 41:54
  metrics to tie your AI and ML journeys to, I think.
• 42:01
  Amazing topic. I absolutely love it.
• 42:03
  This is something I think about all the time, especially
• 42:05
  engaging with clients. And a lot of it Hawks back
• 42:08
  actually to a lot of my training at Google where I do.
• 42:12
  That was my first role while I'd been a CEO and executive
• 42:15
  producer before that, it was my real like introduction to
• 42:17
  Silicon Valley product management and seeing the like.
• 42:19
  Oh, cool. And what a lot of it comes down
• 42:22
  to is MVPS, how and proofs of concept.
• 42:25
  How can you minimally show I get a minimal viable product that
• 42:30
  shows value in that like as you mentioned, what business metrics
• 42:35
  do we want to move and how can we show we can move it?
• 42:37
  And I'll give, I'll give an example that we had a worked
• 42:42
  with a client who wanted to produce a, I'm going to change
• 42:45
  the industry just because of the client.
• 42:48
  But they wanted to have an externally facing LLM that could
• 42:53
  process, let's say, let's say insurance claims.
• 42:56
  So they had an internal team that normally you'd either jump
• 43:00
  online and use a chat bot to do, or you would, or you would, or
• 43:05
  you'd call up and they wanted to make it self-service and do it
• 43:08
  all, all externally. And they were terrified because
• 43:14
  of the bot. Sorry, not terrified is not the
• 43:17
  right word. They've seen a lot of examples
• 43:19
  where there has been public failures and has led to
• 43:23
  reputational like damage for companies and that is worth like
• 43:27
  so much more. That is the, the cost to your to
• 43:30
  your company valuation could be phenomenally large.
• 43:35
  So one of the ways we we talked through with them was we said,
• 43:38
  well, look, what if we work together and you build this, we
• 43:42
  can help you with that. Why don't you deploy it with
• 43:45
  your current internal stakeholders first?
• 43:49
  So you build it, you deploy it, you can see how much it
• 43:51
  accelerates their work. So you're gonna drive some
• 43:54
  business like your ultimate goal was cost reduction and perhaps
• 43:58
  an improved customer experience, maybe both of those your two
• 44:02
  business goals. So while you've got this amazing
• 44:05
  workforce who's got a lot of experience already, why don't we
• 44:08
  deploy it with them or percentage of them and be able
• 44:11
  to compare how does this group versus the group that's not
• 44:14
  using this tool perform? Are they faster?
• 44:16
  Are they able to to clear more tickets?
• 44:18
  Are they able to provide better customer experience or is it
• 44:22
  actually degrading customer experience?
• 44:24
  Because yeah, yeah, what what does your business really care
• 44:27
  about? And you're doing it internally,
• 44:29
  meaning that they're seeing it on their screen, can copy and
• 44:32
  paste it and put it into the, they can use it within the tool
• 44:35
  and but they have to approve it before it goes to the, the end
• 44:39
  user. And that may, and it was a great
• 44:41
  way for them to get very comfortable with it.
• 44:44
  And it, it also revealed that it the journey to get to where they
• 44:48
  needed that model to be externally.
• 44:50
  It's still not released externally and they're driving,
• 44:53
  they've been driving business value for over six months, maybe
• 44:56
  closer to a year now. But it, it showed them, oh, I
• 45:00
  thought it was going to be so easy to be good to go out there,
• 45:04
  but it's not. And it can be really scary.
• 45:06
  But they're already driving business value from a much
• 45:09
  faster proof of concept. Because by the way, when when it
• 45:11
  first came through, there was a lot of problems with it, which
• 45:13
  is, which is fine because the humans were able to catch it and
• 45:16
  get that feedback loop, but it was able to show value quickly,
• 45:19
  which was important.