Latest / Satellite image deep learning / State Of The Art Object Detection

Transcript

State Of The Art Object Detection · Satellite image deep learning

• 0:05
  SPEAKER_01: Hi Isaac, welcome to the podcast.
• 0:08
  SPEAKER_00: Hi Robin, I'm very excited to be here.
• 0:11
  SPEAKER_01: It's a pleasure.
• 0:11
  SPEAKER_01: Let's begin with where you work and what you do.
• 0:16
  SPEAKER_00: So I'm a machine learning research engineer at RoboFlow,
• 0:19
  which basically means that I spend my time trying to build new
• 0:23
  types of models that
• 0:25
  our various users can use to better solve their different
• 0:28
  computer vision use
• 0:28
  SPEAKER_00: cases.
• 0:31
  SPEAKER_01: All right.
• 0:31
  And we're here today because there's one particular model that
• 0:33
  you've been working
• 0:34
  SPEAKER_01: on,
• 0:34
  SPEAKER_01: RF Detter.
• 0:35
  SPEAKER_01: Do you mind telling us a bit about that?
• 0:37
  SPEAKER_00: Yeah.
• 0:38
  So RF Detter is the new state of the art for real-time computer
• 0:44
  vision in terms of
• 0:46
  SPEAKER_00: Cocoa,
• 0:46
  SPEAKER_00: and it is the overall state of the art
• 0:50
  for object detection as benchmarked by our dataset that
• 0:54
  measures ability to
• 0:55
  SPEAKER_00: transfer to downstream use cases,
• 0:58
  SPEAKER_00: diverse downstream use cases,
• 1:00
  SPEAKER_00: beating out models that are orders of magnitude slower with much bigger
• 1:06
  SPEAKER_00: pre-training.
• 1:09
  SPEAKER_00: I say new,
• 1:10
  SPEAKER_00: not because it's just recently released,
• 1:13
  SPEAKER_00: but in fact,
• 1:14
  SPEAKER_00: the first version was released almost like eight or nine months ago now.
• 1:18
  SPEAKER_00: But we're constantly improving it.
• 1:23
  SPEAKER_00: We're constantly releasing new things on top of the model.
• 1:26
  And the model just a couple days ago was accepted to ICLR,
• 1:31
  which is a big step for us.
• 1:33
  SPEAKER_00: And...
• 1:35
  Now it's a kind of an official peer reviewed thing that is new,
• 1:38
  which is cool.
• 1:41
  SPEAKER_01: Congratulations.
• 1:42
  SPEAKER_01: So was your objective to make something that is,
• 1:44
  SPEAKER_01: as you said,
• 1:45
  SPEAKER_01: smaller,
• 1:46
  SPEAKER_01: faster,
• 1:47
  SPEAKER_01: and also the top performing model?
• 1:50
  SPEAKER_00: Yeah,
• 1:51
  we wanted to make something that people could use cheaply in a
• 1:56
  lot of different
• 1:56
  SPEAKER_00: contexts without sacrificing accuracy.
• 2:03
  SPEAKER_00: And specifically,
• 2:05
  SPEAKER_00: we wanted to build something that just was going to work really,
• 2:08
  SPEAKER_00: really well for people in a lot of different contexts.
• 2:13
  SPEAKER_00: Data context as well as hardware context.
• 2:15
  SPEAKER_01: So how did you do it?
• 2:22
  So there are,
• 2:23
  for the object detector,
• 2:24
  there are kind of two main things going on.
• 2:28
  SPEAKER_00: One is we leverage kind of these big or bigger pre-trained vision models,
• 2:38
  SPEAKER_00: such as Dynamo V2 in our paper we also talk about.
• 2:41
  SPEAKER_00: a bunch of other experiments we did with other Foundation models,
• 2:44
  SPEAKER_00: such as,
• 2:45
  SPEAKER_00: I think it was SigLip using SAM2 as a backbone,
• 2:49
  SPEAKER_00: and also CAEV2.
• 2:55
  But just to basically go in and have the model start knowing a
• 2:59
  lot about the world.
• 3:00
  SPEAKER_00: So if your model knows a lot about the world to start with,
• 3:03
  then it needs less from you in order to understand what it is
• 3:06
  that you're talking
• 3:07
  SPEAKER_00: about and what it is that you want to detect.
• 3:11
  SPEAKER_00: And then for the object architecture,
• 3:12
  SPEAKER_00: the other major aspect is most of these papers,
• 3:17
  SPEAKER_00: well,
• 3:18
  SPEAKER_00: I guess all of these papers,
• 3:22
  SPEAKER_00: use a fixed architecture.
• 3:23
  SPEAKER_00: So they assume that there's one architecture that can work super,
• 3:26
  SPEAKER_00: super well in a variety of domains.
• 3:30
  SPEAKER_00: And that's probably true to a large extent, but...
• 3:38
  SPEAKER_00: If you're really,
• 3:38
  SPEAKER_00: really going for that high-performance edge,
• 3:42
  SPEAKER_00: in fact,
• 3:42
  there's no reason to assume that the same neural architecture
• 3:45
  is going to work best
• 3:48
  SPEAKER_00: for many,
• 3:49
  many tiny objects distributed across a high-resolution image
• 3:53
  versus any one object
• 3:55
  SPEAKER_00: where you're just trying to figure out what class it belongs to.
• 3:58
  It seems rational to think that different architectures would
• 4:01
  do better for those.
• 4:05
  So the other major aspect with respect to object detection of
• 4:10
  the RF data work was
• 4:11
  SPEAKER_00: to introduce a neural architecture search method,
• 4:14
  where we go in and we actually simultaneously train many
• 4:19
  thousands of different
• 4:22
  models and then we can choose which architecture is the best
• 4:26
  for you uh after and
• 4:29
  we do this in a very very efficient way so it only takes like
• 4:31
  three or four times
• 4:33
  the uh the train budget to actually create a custom
• 4:36
  architecture uh for you so just
• 4:42
  make sure i understand so actually when you do a training bar
• 4:44
  of data you're
• 4:45
  SPEAKER_01: actually doing neural architecture search is that right
• 4:48
  SPEAKER_00: In the paper, we do neural architecture search.
• 4:51
  SPEAKER_00: We haven't released the neural architecture search engine to open source,
• 4:58
  SPEAKER_00: but we intend for it to soon be a feature on platform.
• 5:04
  SPEAKER_01: Right, okay.
• 5:05
  SPEAKER_01: So this is specifically for using the RoboFlow platform.
• 5:09
  SPEAKER_00: Yeah,
• 5:10
  our goal is to make it so that the RF that are open source is
• 5:15
  the best fixed
• 5:16
  SPEAKER_00: architecture available.
• 5:20
  SPEAKER_00: period, you know, including models that are orders of magnitude slower.
• 5:25
  And that the,
• 5:28
  but that you can do even better if you actually use the
• 5:31
  platform.
• 5:31
  SPEAKER_00: So we can make some money from the thing.
• 5:34
  SPEAKER_01: Right, okay.
• 5:36
  And,
• 5:36
  but the public models,
• 5:38
  they also are the product of neural architecture search then?
• 5:43
  Yeah,
• 5:43
  the public models are the result of neural architecture search
• 5:45
  against the COCO dataset.
• 5:47
  So the public models that do very well are applying RFID or
• 5:53
  Neural Architecture Search to COCO.
• 5:56
  SPEAKER_00: And in our benchmarks in the paper,
• 5:58
  we talk about what it is to apply RFID or Neural Architecture
• 6:02
  Search to every
• 6:03
  single one of the hundred datasets that we use to judge
• 6:07
  downstream performance of
• 6:08
  SPEAKER_00: object detectors.
• 6:09
  SPEAKER_01: Right,
• 6:10
  SPEAKER_01: okay,
• 6:10
  so it was quite a complex setup to run all these searches
• 6:15
  against 100 target data
• 6:17
  SPEAKER_01: sets.
• 6:19
  SPEAKER_00: Yeah, we trained many, many thousands of models.
• 6:24
  SPEAKER_00: We feel like we... Actually, we were very, very thorough about evaluating
• 6:35
  SPEAKER_00: downstream performance and models in general.
• 6:38
  SPEAKER_00: So, for example, YOLO26 dropped, I think, last week or the week before.
• 6:42
  SPEAKER_00: And, like, by the next day, we had trained...
• 6:51
  SPEAKER_00: every YOLO 26 size on RF100VL,
• 6:54
  SPEAKER_00: which is our data set from NeurIPS for this transfer we'd trained,
• 6:59
  SPEAKER_00: every YOLO 26 model on RF100VL.
• 7:02
  SPEAKER_00: So we had trained something like 500 neural nets overnight,
• 7:08
  SPEAKER_00: which may be more than Ultralytics trained when they actually...
• 7:16
  SPEAKER_00: Within one day,
• 7:16
  SPEAKER_00: we may have become the largest trainer of YOLO 26 models,
• 7:19
  SPEAKER_00: period,
• 7:19
  SPEAKER_00: because we wanted to be very,
• 7:20
  SPEAKER_00: very thorough about our analysis.
• 7:24
  SPEAKER_01: Right, OK.
• 7:25
  SPEAKER_01: And so what do we end up with?
• 7:27
  Do we end up with something that looks radically different from
• 7:30
  what is done in the
• 7:31
  SPEAKER_01: past,
• 7:31
  SPEAKER_01: or is it very much a refinement?
• 7:36
  You mean in terms of the types of networks that emerge from
• 7:39
  neural architecture search?
• 7:41
  SPEAKER_00: Yeah, it's super interesting.
• 7:46
  So it's the same basic architecture as LWDeader,
• 7:57
  but there's some kind of key differences.
• 8:00
  SPEAKER_00: For example...
• 8:03
  SPEAKER_00: Well, there's some, like, kind of subtle structural differences.
• 8:05
  SPEAKER_00: For example,
• 8:06
  SPEAKER_00: like,
• 8:06
  SPEAKER_00: LWD introduces this idea of,
• 8:08
  SPEAKER_00: like,
• 8:08
  SPEAKER_00: row major window detention in a VIT backbone,
• 8:10
  SPEAKER_00: which is a mouthful,
• 8:12
  but it's kind of this just interesting memory optimization with
• 8:15
  respect to how to
• 8:16
  SPEAKER_00: do window detention a little better.
• 8:18
  SPEAKER_00: And they use four windows,
• 8:21
  SPEAKER_00: and Neural Architecture Research discovers that in most cases,
• 8:25
  SPEAKER_00: the optimal is two.
• 8:28
  SPEAKER_00: We find that we can actually...
• 8:34
  SPEAKER_00: Other debtor models use a fixed number of queries.
• 8:39
  SPEAKER_00: So a debtor works by you encode your image with the backbone,
• 8:43
  SPEAKER_00: and then you have queries that cross-attend to the image hidden state.
• 8:49
  SPEAKER_00: It's almost like an encoder-decoder transformer architecture in general,
• 8:54
  SPEAKER_00: but you don't have the autoregression,
• 8:58
  SPEAKER_00: and it's not pure transformers.
• 8:59
  SPEAKER_00: You got some, like, convolutions and other stuff happening.
• 9:03
  we find that we can actually drop queries after training to
• 9:10
  speed up inference,
• 9:12
  which is novel.
• 9:13
  SPEAKER_00: I don't think any other papers talked about that.
• 9:15
  And what this does is it actually encodes statistics as to how
• 9:21
  many objects are
• 9:22
  SPEAKER_00: likely to be in an image from your specific data set.
• 9:28
  SPEAKER_00: So we find that data sets with fewer objects
• 9:34
  SPEAKER_00: fewer objects per image after neural architecture search tend to prefer
• 9:38
  SPEAKER_00: architectures that have far fewer queries,
• 9:40
  which gives you a meaningful speed up over architectures that
• 9:44
  only operate with a
• 9:45
  SPEAKER_00: fixed number of queries.
• 9:45
  SPEAKER_00: There's a lot of fun stuff going on.
• 9:55
  SPEAKER_00: RT debtor showed that you could drop decoder layers
• 10:03
  SPEAKER_00: In modern tender training,
• 10:05
  SPEAKER_00: you have multiple layers in your decoder,
• 10:07
  and you are simultaneously applying losses to every single one
• 10:11
  of the layers.
• 10:11
  SPEAKER_00: So the idea is that each layer is acting as an iterative improvement,
• 10:15
  as opposed to just optimizing such that the last layer
• 10:19
  representation is good.
• 10:22
  RTDeditor showed that you could drop decoder layers and get
• 10:28
  nice performance.
• 10:29
  SPEAKER_00: And we showed that, in fact, you could drop all of the decoder layers.
• 10:33
  SPEAKER_00: So you're not even doing this, like, cross-attend part anymore.
• 10:36
  SPEAKER_00: It's just basically a VIT forward pass.
• 10:39
  And that participates in it,
• 10:41
  which is a meaningful differentiation versus the rest of the
• 10:45
  field.
• 10:46
  SPEAKER_00: But...
• 10:48
  SPEAKER_00: Yeah, and then we do other things in the paper.
• 10:50
  SPEAKER_00: We also introduced novel segmentation head architecture,
• 10:52
  SPEAKER_00: which turned out to be very effective,
• 10:57
  SPEAKER_00: especially when combined with neural architecture search.
• 10:59
  SPEAKER_00: I don't know if that answers your question.
• 11:03
  SPEAKER_00: I kind of forgot what the question is.
• 11:04
  SPEAKER_01: Yeah, so I'll invite you all.
• 11:06
  But it's interesting that the neural architecture search is
• 11:09
  kind of steering you in
• 11:10
  SPEAKER_01: maybe novel directions or taking things to the extremes,
• 11:14
  SPEAKER_01: which people just hadn't done themselves.
• 11:18
  SPEAKER_00: Yeah,
• 11:19
  SPEAKER_00: I mean,
• 11:19
  it's interesting because the way that you normally develop
• 11:22
  these models is you do
• 11:24
  SPEAKER_00: just a massive amount of sweeps.
• 11:26
  SPEAKER_00: A lot of papers will do a huge amount of sweeps.
• 11:32
  SPEAKER_00: And so finding these little details for fixed architectures
• 11:44
  SPEAKER_00: means meaningfully increasing your sweep size,
• 11:46
  SPEAKER_00: which hugely increases the cost of running the experiment.
• 11:52
  SPEAKER_00: But if you just kind of throw it into a neural engine,
• 11:54
  SPEAKER_00: like a neural architecture search engine,
• 11:55
  SPEAKER_00: you don't do that.
• 11:57
  SPEAKER_00: They're all trained at the same time.
• 11:59
  So your cost of developing these models is many orders of
• 12:02
  magnitude lower,
• 12:03
  which lets us pop out these kind of interesting objects that
• 12:05
  otherwise would not be
• 12:06
  SPEAKER_00: found.
• 12:08
  So how do you think this changes the approach people might need
• 12:11
  to take if they're
• 12:12
  SPEAKER_01: working in specific domains like medical or satellite,
• 12:15
  where maybe the data just doesn't even look like anything that
• 12:18
  is being pre-trained
• 12:19
  SPEAKER_01: on,
• 12:19
  SPEAKER_01: and now you also have this option of different architectures?
• 12:23
  SPEAKER_00: Yeah.
• 12:24
  SPEAKER_00: Yeah, I think they're pretty complementary.
• 12:30
  SPEAKER_00: So one thing is that R400VL has...
• 12:38
  SPEAKER_00: different subcategories.
• 12:40
  SPEAKER_00: So we've got like an aerial subcategory,
• 12:41
  SPEAKER_00: we've got medical subcategory documents,
• 12:43
  SPEAKER_00: a couple of them.
• 12:45
  SPEAKER_00: And so we can do basically a differential analysis where we say,
• 12:50
  SPEAKER_00: how do different model families perform on different subcategories?
• 12:55
  SPEAKER_01: Sorry, just to clarify, that's your data set.
• 12:58
  SPEAKER_00: Yeah, yeah, sorry.
• 12:59
  SPEAKER_00: RF100VL is our data set.
• 13:02
  SPEAKER_00: We also open source NeurIPS 2025.
• 13:08
  SPEAKER_00: SAM3 benchmarked on it, which was a nice validation for us.
• 13:11
  Like,
• 13:12
  our phone is in table two of SAM3,
• 13:14
  which was actually a really nice moment.
• 13:20
  SPEAKER_00: Yeah,
• 13:21
  SPEAKER_00: so you can look at how does Define do in aerial imagery,
• 13:26
  SPEAKER_00: or how does YOLO 11 do when you're looking at flora and fauna?
• 13:35
  SPEAKER_00: And we find that RFDudder does particularly well on...
• 13:41
  SPEAKER_00: medical and aerial and I think documents, if I remember correctly.
• 13:47
  SPEAKER_00: These things that are more different from Coco and other...
• 13:54
  SPEAKER_00: standard pre-training, such as Objects365.
• 13:58
  SPEAKER_00: And the idea is that the DynoV2 backbone just has a lot of knowledge,
• 14:03
  and we're training in a way that we are attempting to be
• 14:07
  deliberate about minimally
• 14:09
  SPEAKER_00: erasing that knowledge during our object detection pre-training.
• 14:13
  SPEAKER_00: So our data has a couple of pre-training stages.
• 14:15
  SPEAKER_00: One is the foundation model pre-training.
• 14:20
  SPEAKER_00: which is not done by us, which is done by, in this case, Meta.
• 14:24
  And then we do object detection level pre-training,
• 14:27
  for which we use Object365.
• 14:28
  SPEAKER_00: For segmentation,
• 14:31
  we also used SAM2 to provide masks for every object in
• 14:37
  Object365 and use that as
• 14:39
  SPEAKER_00: pre-training for segmentation,
• 14:40
  SPEAKER_00: which was an interesting thing,
• 14:41
  SPEAKER_00: which helped.
• 14:45
  SPEAKER_00: And then we do Cocoa Fine Tunes.
• 14:47
  SPEAKER_00: And in the NAS case,
• 14:49
  SPEAKER_00: our neural architecture search is a weight-sharing NAS,
• 14:51
  so we actually can pre-train all of the different sub-networks
• 14:57
  at the same time so
• 15:00
  SPEAKER_00: that we never actually make an architecture decision until,
• 15:03
  SPEAKER_00: like,
• 15:03
  SPEAKER_00: we've done our pre-training,
• 15:04
  SPEAKER_00: we've done our fine-tuning on your data set,
• 15:05
  SPEAKER_00: and then we do the architecture,
• 15:07
  SPEAKER_00: which is kind of novel.
• 15:11
  SPEAKER_00: But yeah,
• 15:12
  so we do a differential analysis and we find that we do well on
• 15:16
  data sets that are
• 15:16
  SPEAKER_00: meaningfully different from kind of that standard COCO regime.
• 15:20
  SPEAKER_00: And there are a couple interpretations of that.
• 15:22
  SPEAKER_00: Like one,
• 15:22
  SPEAKER_00: as you said,
• 15:23
  like data being more likely to be in distribution as we do
• 15:27
  these large scale
• 15:28
  SPEAKER_00: retrains.
• 15:30
  SPEAKER_00: But there's another aspect, which is...
• 15:34
  SPEAKER_00: data may not be explicitly in distribution,
• 15:36
  but the features learned during a large-scale pre-train may
• 15:40
  turn out to be more
• 15:41
  SPEAKER_00: generic,
• 15:43
  SPEAKER_00: meaning that they transfer better to out-of-distribution stuff.
• 15:49
  SPEAKER_00: And then there's some really interesting analysis on...
• 15:53
  SPEAKER_00: Man,
• 15:57
  SPEAKER_00: there's so much cool stuff coming out about how to hold different...
• 16:04
  SPEAKER_00: visual free training strategies in terms of,
• 16:09
  SPEAKER_00: like,
• 16:09
  the web SSL paper talking about how does Dyno V2 strategy scale
• 16:15
  versus something
• 16:16
  SPEAKER_00: like CLIP in terms of OCR recognition was,
• 16:18
  SPEAKER_00: like,
• 16:18
  SPEAKER_00: fascinating.
• 16:20
  SPEAKER_00: All this to say,
• 16:22
  SPEAKER_00: you can have some unexpected,
• 16:24
  there's some evidence that some unexpected stuff can pop out as
• 16:26
  you start to reach
• 16:26
  SPEAKER_00: certain scales.
• 16:28
  SPEAKER_01: Mm-hmm.
• 16:30
  yeah um i hope that was uh yeah i mean i think it's really a
• 16:37
  novel area uh that
• 16:38
  i've not been that i've not seen in the past right this kind of
• 16:41
  task specific
• 16:43
  pre-training coupled with very large scale steering us towards
• 16:47
  quite optimized uh
• 16:49
  architectures and potentially different pathways depending on
• 16:54
  what you actually
• 16:54
  SPEAKER_01: want to do with the model
• 16:56
  SPEAKER_00: Yeah.
• 16:57
  SPEAKER_00: There's one major downside of this type of approach,
• 16:59
  SPEAKER_00: which is actually why we invented the NAS stuff,
• 17:04
  SPEAKER_00: or for,
• 17:05
  SPEAKER_00: you know,
• 17:05
  SPEAKER_00: RF data's ridges of NAS,
• 17:07
  SPEAKER_00: which is that people,
• 17:11
  SPEAKER_00: when they do these large-scale trainings,
• 17:14
  SPEAKER_00: don't have us in mind.
• 17:18
  uh like they're not training these models with the idea that
• 17:23
  they're going to run
• 17:23
  on a jetson nano at high resolution for example like they so
• 17:27
  they tend to train
• 17:28
  basically larger models and that's not great because uh if we
• 17:33
  want super super low
• 17:34
  latency or alternatively really really good cost efficiency we
• 17:39
  have to figure out a
• 17:40
  SPEAKER_00: way to work with those large models um
• 17:44
  SPEAKER_00: So for example,
• 17:45
  SPEAKER_00: the smallest,
• 17:47
  SPEAKER_00: we spent a lot of time looking for basically just what is the smallest,
• 17:52
  SPEAKER_00: highest performing pre-trained VIT we could find.
• 17:59
  SPEAKER_00: And we ended up with a couple options, including Dynav2 small.
• 18:06
  SPEAKER_00: But LWDetter,
• 18:08
  SPEAKER_00: which is,
• 18:08
  SPEAKER_00: again,
• 18:09
  SPEAKER_00: the paper that we started from,
• 18:11
  SPEAKER_00: uses VIT Tiny as their backbone for their small and nano models.
• 18:17
  So you've got,
• 18:20
  I think it's like a 4x higher parameter count using the small
• 18:25
  than the tiny.
• 18:25
  SPEAKER_00: And it's a slower model.
• 18:29
  So the question becomes,
• 18:31
  OK,
• 18:31
  how do you make use of a 25 million parameter model
• 18:36
  SPEAKER_00: on a nano time series.
• 18:40
  Like,
• 18:40
  how do you get that to run as an object detector backbone in
• 18:43
  two milliseconds?
• 18:45
  SPEAKER_00: And the...
• 18:47
  Turns out you can just kind of throw it into the neural
• 18:49
  architecture search engine,
• 18:50
  SPEAKER_00: and it figures out,
• 18:51
  SPEAKER_00: okay,
• 18:52
  SPEAKER_00: well,
• 18:52
  SPEAKER_00: actually,
• 18:52
  SPEAKER_00: you can just run this at a lower resolution,
• 18:55
  SPEAKER_00: and that works,
• 18:57
  SPEAKER_00: which is pretty convenient.
• 19:00
  SPEAKER_00: So we end up with...
• 19:04
  like 30 million parameter models that are as fast as like 2
• 19:07
  million parameter
• 19:08
  SPEAKER_00: models from other families.
• 19:10
  And one of the things that they tend to do is run at lower
• 19:14
  resolutions or use a
• 19:16
  SPEAKER_00: larger patch size.
• 19:17
  SPEAKER_01: So I want to understand.
• 19:18
  SPEAKER_01: So the optimization approach,
• 19:20
  SPEAKER_01: is it good at finding smaller,
• 19:21
  SPEAKER_01: more performant models or can it also go the other way and say,
• 19:24
  SPEAKER_01: this is a bigger,
• 19:25
  SPEAKER_01: higher performing model with other trade-offs?
• 19:31
  SPEAKER_00: It can go both ways.
• 19:33
  We find the biggest gain from using NAS over fixed architecture
• 19:38
  so far in our
• 19:39
  SPEAKER_00: experiments has been on these kind of like smaller things.
• 19:45
  SPEAKER_00: But I don't think that's...
• 19:49
  SPEAKER_00: Well, there's some interesting papers talking about scaling NAS methods.
• 19:55
  SPEAKER_00: From my perspective,
• 19:56
  I think that's basically just because the way we defined the
• 19:58
  search family has more
• 20:01
  SPEAKER_00: options in this smaller range.
• 20:03
  So I think we could run a larger search and presumably get
• 20:06
  higher performance in
• 20:08
  SPEAKER_00: that higher range as well.
• 20:09
  SPEAKER_01: Right.
• 20:13
  SPEAKER_00: And we also throw like some larger,
• 20:14
  SPEAKER_00: we do a Dyna V2 base backbone for the detector in the paper as well,
• 20:19
  SPEAKER_00: which...
• 20:21
  SPEAKER_00: scales super well.
• 20:23
  SPEAKER_00: We end up,
• 20:23
  that ends up being the first real-time object detector that
• 20:29
  claims to beat 60 map
• 20:31
  SPEAKER_00: on Cocoa.
• 20:32
  SPEAKER_00: Whether or not you could actually get that with other approaches,
• 20:34
  SPEAKER_00: like maybe no one published it before,
• 20:36
  SPEAKER_00: but to the best of my knowledge,
• 20:37
  SPEAKER_00: this is the first published thing that is like,
• 20:40
  SPEAKER_00: you know,
• 20:41
  SPEAKER_00: it's like 17 milliseconds and over 60 map,
• 20:43
  SPEAKER_00: which is fun.
• 20:46
  So the final result is basically a family of related models at
• 20:50
  different sizes with
• 20:51
  SPEAKER_01: different sort of performances and,
• 20:53
  SPEAKER_01: uh,
• 20:55
  SPEAKER_01: requirements,
• 20:57
  SPEAKER_01: uh,
• 20:57
  SPEAKER_01: capabilities.
• 20:58
  SPEAKER_01: You've got object detection.
• 21:00
  SPEAKER_01: You said segmentation as well.
• 21:04
  SPEAKER_00: Yeah.
• 21:05
  SPEAKER_00: Yeah.
• 21:05
  We built a novel,
• 21:06
  uh,
• 21:06
  segmentation head for real-time data models,
• 21:09
  um,
• 21:09
  kind of inspired by,
• 21:11
  uh,
• 21:12
  SPEAKER_00: Based off the mask dyno approach,
• 21:17
  SPEAKER_00: but they're using a hierarchical backbone and we're not,
• 21:20
  SPEAKER_00: so we had to come up with some novel stuff there.
• 21:23
  This is kind of a more mundane detail,
• 21:30
  but we did a lot of work on memory optimization.
• 21:35
  SPEAKER_00: That train time specifically, so the model...
• 21:39
  SPEAKER_00: The new release that came out,
• 21:41
  SPEAKER_00: I think,
• 21:42
  last week for ArtBetter makes segmentation training like three
• 21:45
  to four times less
• 21:46
  SPEAKER_00: memory intensive for no change in performance,
• 21:51
  SPEAKER_00: which is fun.
• 21:52
  SPEAKER_00: That's good.
• 21:54
  SPEAKER_00: In fact, the gradients are exactly the same.
• 21:55
  SPEAKER_00: There's some tricks that make it so it just takes less compute.
• 22:00
  SPEAKER_00: Right.
• 22:01
  SPEAKER_00: Which is really fun.
• 22:03
  SPEAKER_00: I had a lot of fun building that.
• 22:04
  SPEAKER_01: Yeah.
• 22:05
  and then obviously reduces the cost of fine tuning that model
• 22:09
  you can use a smaller
• 22:10
  machine yeah it takes it from like requiring 40 or 50 gigs vram
• 22:14
  to like 10 which is
• 22:15
  really cool wonderful uh so that's instant segmentation is it
• 22:20
  yeah yeah okay um i'm
• 22:25
  SPEAKER_00: we're we're looking at key points and um
• 22:30
  SPEAKER_00: I would like to do semantic segmentation as well.
• 22:34
  SPEAKER_00: From my perspective,
• 22:36
  SPEAKER_00: we would like to be providing very,
• 22:39
  SPEAKER_00: very,
• 22:40
  very high performance Apache 2 frameworks for kind of all these
• 22:47
  standard tasks.
• 22:48
  SPEAKER_01: Right.
• 22:49
  SPEAKER_01: Yes.
• 22:50
  SPEAKER_01: So similar to another framework that's not Apache 2.
• 22:53
  SPEAKER_00: Yes.
• 22:54
  SPEAKER_00: Very similar.
• 22:55
  SPEAKER_00: Fantastic.
• 22:55
  SPEAKER_00: And hopefully higher performance.
• 22:58
  SPEAKER_01: Well, thank you so much.
• 22:59
  SPEAKER_01: It's been really interesting technical deep dive.
• 23:02
  SPEAKER_01: What's next then?
• 23:03
  SPEAKER_01: I think you've hinted at a few things.
• 23:05
  SPEAKER_00: Yeah.
• 23:06
  SPEAKER_00: So you engage with the GitHub repo.
• 23:14
  SPEAKER_00: And I am sure you noticed that we haven't really been super...
• 23:19
  SPEAKER_00: I respond to issues,
• 23:20
  SPEAKER_00: but I don't think many code changes happen that frequently.
• 23:26
  SPEAKER_00: That's the big downside of being a research engineer.
• 23:28
  SPEAKER_00: I don't really write that much code.
• 23:36
  SPEAKER_00: And the other people on my team,
• 23:37
  SPEAKER_00: we're all...
• 23:37
  SPEAKER_00: I mean,
• 23:40
  SPEAKER_00: we're...
• 23:43
  SPEAKER_00: The paper,
• 23:43
  SPEAKER_00: we have five co-authors,
• 23:46
  SPEAKER_00: three of whom did...
• 23:51
  SPEAKER_00: all of the training.
• 23:53
  SPEAKER_00: So very, very, very small team working on this.
• 23:57
  Of course,
• 24:00
  we have wonderful,
• 24:00
  wonderful support,
• 24:01
  but ultimately there's a lot of work to do.
• 24:03
  So we...
• 24:05
  The thing that I'm looking forward to the most is we hired a
• 24:09
  new open source...
• 24:14
  SPEAKER_00: open source research engineer,
• 24:15
  SPEAKER_00: I think is his title,
• 24:17
  SPEAKER_00: who's one of the key developers behind PyTorch Lightning.
• 24:21
  SPEAKER_00: And the first thing that he's doing,
• 24:22
  SPEAKER_00: and I don't know if you've noticed,
• 24:26
  SPEAKER_00: but things are changing,
• 24:28
  SPEAKER_00: and hopefully significantly for the better.
• 24:30
  We're actually getting devoted dev time to making sure that
• 24:34
  that repo progresses in
• 24:36
  SPEAKER_00: a nice way,
• 24:37
  SPEAKER_00: which is super,
• 24:38
  SPEAKER_00: super,
• 24:39
  SPEAKER_00: super exciting.
• 24:39
  SPEAKER_00: And I am...
• 24:44
  SPEAKER_00: I am,
• 24:44
  SPEAKER_00: you,
• 24:45
  SPEAKER_00: you,
• 24:45
  SPEAKER_00: you,
• 24:45
  SPEAKER_00: you,
• 24:46
  you tagged me in an issue or I think a pull request this
• 24:48
  morning and I'm planning
• 24:49
  SPEAKER_00: on tagging him instead.
• 24:52
  SPEAKER_00: I don't know.
• 24:54
  We're figuring out how to work,
• 24:55
  uh,
• 24:56
  like what the proper work dynamics are.
• 24:58
  SPEAKER_00: But I,
• 24:58
  SPEAKER_00: you know,
• 24:59
  SPEAKER_00: this is a,
• 25:00
  SPEAKER_00: this is a first for me,
• 25:01
  SPEAKER_00: like dealing with this,
• 25:02
  SPEAKER_00: um,
• 25:02
  SPEAKER_00: large scale of,
• 25:03
  SPEAKER_01: uh,
• 25:05
  SPEAKER_00: of an open source project.
• 25:06
  SPEAKER_00: So it's a massive learning experience.
• 25:07
  SPEAKER_00: And I just, um, I think, uh,
• 25:11
  it's i am interested in deferring to people who know more about
• 25:15
  how to do this than
• 25:16
  i do yeah yeah so obviously a lot of attention for this model
• 25:20
  performance license
• 25:22
  etc lots of people coming and creating feature requests finding
• 25:26
  things that need
• 25:27
  addressing and you've been heads down on the paper and aversion
• 25:30
  too so totally
• 25:31
  SPEAKER_01: understandable that it's been hard to get
• 25:34
  to all those issues and yeah more hands on deck to address them
• 25:37
  that's great to
• 25:38
  hear so uh and also you did mention that uh there's going to be
• 25:42
  an integration with
• 25:43
  the roboflow platform itself do you mind talking a little bit
• 25:46
  more about that yeah
• 25:48
  i don't know how much uh i can say specifically but we're
• 25:52
  planning on um there's
• 25:55
  SPEAKER_00: going to be a lot of uh
• 25:58
  A lot of performance improvements coming from a variety of
• 26:01
  directions in the next
• 26:02
  SPEAKER_00: couple in the near future.
• 26:08
  SPEAKER_00: One that I'm particularly excited about is a lot of these...
• 26:15
  SPEAKER_00: I think basically all papers advertise certain low latencies,
• 26:19
  SPEAKER_00: including ours,
• 26:20
  SPEAKER_00: that are
• 26:22
  SPEAKER_00: not what you get out of the box.
• 26:24
  SPEAKER_00: So we intended to be kind of more reproducible in that we also released a
• 26:36
  benchmarking framework called Single Artifact Benchmarking that
• 26:38
  attempts to make it
• 26:39
  SPEAKER_00: so you could,
• 26:39
  SPEAKER_00: okay,
• 26:40
  SPEAKER_00: here's one file that contains everything about your model,
• 26:44
  it accepts an onyx graph and it will give you the latency
• 26:47
  accuracy number like it
• 26:48
  it it just gives you the result and there's no subtlety about
• 26:52
  it there's no no
• 26:53
  caveats um but the latency numbers we compute with that and if
• 27:01
  you just run the rfd
• 27:02
  SPEAKER_00: repo uh it just runs in pure pytorch and you don't you don't get the same
• 27:07
  SPEAKER_00: performance which is kind of frustrating um
• 27:13
  SPEAKER_00: And that's basically true of most research repos.
• 27:16
  It's true of most open source things because the context in
• 27:20
  which you can do these
• 27:21
  SPEAKER_00: optimizations,
• 27:22
  SPEAKER_00: such as TensorFlow RT,
• 27:24
  is a little bit more sophisticated than is easily pipeline-able
• 27:31
  for a lot of users.
• 27:31
  SPEAKER_00: So there's a new...
• 27:44
  release people are working on within RoboFlow to make our
• 27:47
  inference package much,
• 27:50
  SPEAKER_00: much,
• 27:50
  SPEAKER_00: much,
• 27:51
  SPEAKER_00: much faster.
• 27:51
  SPEAKER_00: And that'll soon start to be able to actually,
• 27:57
  SPEAKER_00: you should be able to just,
• 27:59
  SPEAKER_00: you know,
• 28:00
  SPEAKER_00: easily one-liner get kind of the advertised performance,
• 28:04
  SPEAKER_00: not only on GPU,
• 28:05
  SPEAKER_00: but on a variety of other hardware providers.
• 28:10
  SPEAKER_00: And I think that's something that I think is really important.
• 28:14
  SPEAKER_00: And I'm hoping that in the open source repo,
• 28:19
  we can basically take that same ease to work with
• 28:21
  infrastructure and make that...
• 28:28
  make that kind of the happy path for researchers who would like
• 28:31
  to translate their
• 28:32
  SPEAKER_00: work into some high performance artifact for downstream consumers.
• 28:35
  SPEAKER_00: Right.
• 28:36
  So really closing that loop so people know the performance they
• 28:40
  can get if they put
• 28:41
  this into a real world system and building all the
• 28:44
  infrastructure to enable the use
• 28:45
  SPEAKER_01: of it.
• 28:45
  SPEAKER_01: Well, that's really great to hear.
• 28:46
  SPEAKER_01: Obviously, it's on GitHub.
• 28:49
  SPEAKER_01: We'll put the link in the show notes.
• 28:51
  SPEAKER_01: Um, I know that there's a RoboFlow newsletter.
• 28:54
  SPEAKER_01: People can get updates.
• 28:55
  SPEAKER_01: They want to hear more updates.
• 28:56
  Is that,
• 28:57
  is that the best place to go or is there another platform as
• 28:59
  well?
• 29:01
  SPEAKER_00: Yeah.
• 29:01
  SPEAKER_00: The newsletter is, I think, I think good.
• 29:04
  SPEAKER_00: Um, we're also hoping to be more active in the Git repo.
• 29:09
  SPEAKER_00: So if you would like to engage,
• 29:11
  SPEAKER_00: I mean,
• 29:11
  SPEAKER_00: I,
• 29:12
  SPEAKER_00: I answer technical questions if you have them in that repo all the time.
• 29:17
  SPEAKER_00: Um, uh, hopefully that, that will be, uh,
• 29:21
  SPEAKER_00: more popping in the near future.
• 29:25
  SPEAKER_00: Yeah.
• 29:27
  SPEAKER_00: Brilliant.
• 29:30
  SPEAKER_01: Okay.
• 29:31
  SPEAKER_00: I'm very excited.
• 29:31
  SPEAKER_01: GitHub is the place.
• 29:33
  SPEAKER_01: Yeah.
• 29:35
  SPEAKER_00: Newsletter GitHub.
• 29:36
  SPEAKER_00: Yeah.
• 29:36
  SPEAKER_00: I think.
• 29:38
  SPEAKER_00: Yeah.
• 29:39
  SPEAKER_00: I'm pumped.
• 29:40
  I think it's a lot of,
• 29:42
  you know,
• 29:45
  I think you have to have kind of a base base,
• 29:48
  SPEAKER_00: model that you care about as a, as a platform.
• 29:52
  And then you can start making sure all the infrastructure
• 29:54
  pipelines in ways that
• 29:55
  SPEAKER_00: are really,
• 29:56
  SPEAKER_00: really respecting that model.
• 29:58
  SPEAKER_00: And I think we're, we're working on it and I'm excited for the result.
• 30:02
  SPEAKER_01: Well, it's really fantastic outcome.
• 30:04
  SPEAKER_01: Looking forward to seeing how it progresses.
• 30:07
  Once again,
• 30:07
  thank you for joining me today and until next time,
• 30:09
  take care.
• 30:11
  SPEAKER_00: Thank you.
• 30:12
  SPEAKER_00: Take care as well.