Latest / Satellite image deep learning / Geospatial Annotation with LabelMe and Segment Anything

Transcript

Geospatial Annotation with LabelMe and Segment Anything · Satellite image deep learning

• 0:05
  SPEAKER_01: Hi Kentaro, welcome to the podcast.
• 0:08
  SPEAKER_00: Hi Robin, thank you for having me.
• 0:11
  SPEAKER_01: Let's begin with where you work and what you do.
• 0:14
  SPEAKER_00: Hi everyone, I'm Kentaro.
• 0:18
  SPEAKER_00: I work...
• 0:20
  Someone might know me as W Kentaro on GitHub because I've been
• 0:24
  working on open
• 0:25
  SPEAKER_00: source projects for some time.
• 0:28
  And currently I'm working at the robotic startup in Japan
• 0:32
  called Mujin as a
• 0:34
  SPEAKER_00: computer vision engineer.
• 0:35
  SPEAKER_00: And I co-lead the vision team
• 0:39
  SPEAKER_00: where we deploy the vision system for warehouse and factory automations.
• 0:45
  SPEAKER_00: And as you guys might already know,
• 0:48
  the vision system requires a lot of image data or ground truth
• 0:52
  data that is very
• 0:53
  SPEAKER_00: useful for evaluation and also training machine learning models.
• 0:57
  SPEAKER_00: So on the side,
• 0:58
  SPEAKER_00: I've been working on this software project called LevelMe,
• 1:05
  SPEAKER_00: where we can annotate the image
• 1:09
  SPEAKER_00: And to teach the machine what machine should recognize in an image.
• 1:14
  SPEAKER_00: And I was sharing some of the updates on the X.
• 1:18
  SPEAKER_00: And then Robin found some of the updates on here.
• 1:21
  SPEAKER_00: And then here we are.
• 1:22
  SPEAKER_00: That's right.
• 1:23
  SPEAKER_01: I think you've been working on LabelMate for several years, right?
• 1:27
  SPEAKER_00: Yeah.
• 1:28
  SPEAKER_00: Actually, I started working on that when I was...
• 1:33
  I think the fourth year in the bachelor or maybe the first year
• 1:36
  in master when I
• 1:38
  SPEAKER_00: was in the Tokyo university where the robotics lab and then I,
• 1:44
  SPEAKER_00: yeah,
• 1:45
  SPEAKER_00: the point was,
• 1:48
  SPEAKER_00: yeah,
• 1:48
  SPEAKER_00: I remember like,
• 1:49
  SPEAKER_00: it was like robotics challenge.
• 1:52
  SPEAKER_00: There was like Amazon picking challenge.
• 1:54
  SPEAKER_00: And then we, as a lab, we participated in the challenge.
• 1:59
  And that was very early of the deep learning era where we were
• 2:03
  shifting from like
• 2:05
  SPEAKER_00: feature-based,
• 2:06
  SPEAKER_00: like a shift color histogram to more like convolutional net,
• 2:11
  SPEAKER_00: type of free convolutional net was the just coming up.
• 2:16
  SPEAKER_00: And then,
• 2:17
  SPEAKER_00: yeah,
• 2:17
  SPEAKER_00: we were,
• 2:19
  it was very challenging for this traditional method to
• 2:24
  recognize the object,
• 2:27
  SPEAKER_00: you know,
• 2:28
  SPEAKER_00: Amazon
• 2:29
  warehouse challenge sorry the challenge was like you need to
• 2:33
  detect the object or
• 2:35
  segment object inside of the shelf and then give the location
• 2:41
  so that robot can
• 2:42
  pick and then in the challenge you build a vision system and a
• 2:46
  robot system at the
• 2:47
  SPEAKER_00: same time and then we needed to
• 2:51
  SPEAKER_00: Yeah, build this like annotation tool.
• 2:54
  SPEAKER_00: And then I started building this and open sourcing it.
• 2:58
  And still I'm using this even in current work and also making
• 3:04
  some updates in public as well.
• 3:06
  SPEAKER_01: Amazing.
• 3:07
  So you basically created the tool because it didn't exist and
• 3:10
  you needed it and
• 3:11
  SPEAKER_01: you've been working on it ever since.
• 3:14
  SPEAKER_00: Yeah, exactly.
• 3:15
  SPEAKER_01: Fantastic.
• 3:16
  SPEAKER_01: And what's changed recently in the area of annotation?
• 3:20
  For instance,
• 3:20
  like people are now using tools like Segment Anything to speed
• 3:24
  up annotation.
• 3:25
  SPEAKER_01: Can you give me some context?
• 3:27
  SPEAKER_00: Yeah.
• 3:29
  SPEAKER_00: I think the couple of years later,
• 3:32
  SPEAKER_00: like earlier,
• 3:35
  I think Meta started publishing the new foundation image model
• 3:41
  called Segment
• 3:42
  SPEAKER_00: Anything Model,
• 3:44
  SPEAKER_00: SUM.
• 3:47
  SPEAKER_00: Currently, they released version 3.
• 3:55
  SPEAKER_00: After seeing that, I started integrating that features into Label.me.
• 4:02
  SPEAKER_00: The original meta published as PyTorch models,
• 4:07
  SPEAKER_00: so I also needed to export it to more easy to install
• 4:13
  way because I try to keep the Label.me as easy to install and I
• 4:19
  also compact binary to install.
• 4:22
  SPEAKER_00: That's one of the key features of this.
• 4:25
  SPEAKER_00: So yeah, that's like what I've been working on.
• 4:30
  SPEAKER_00: Then I also Label.me got updated.
• 4:32
  SPEAKER_00: So where you can, oh, sorry, maybe I didn't explain what some is.
• 4:37
  SPEAKER_00: So some is like where you can click the
• 4:42
  point of the object and then it automatically segments object
• 4:47
  and then recent
• 4:48
  upgrade in some three is you create the boundary box of single
• 4:53
  object and then it
• 4:54
  automatically segments a similar looking items or you can also
• 4:58
  give text as a query
• 5:01
  SPEAKER_00: and then you can detect object so that's the recent update
• 5:05
  SPEAKER_01: So lots of new ways of annotating data,
• 5:08
  SPEAKER_01: not just manually putting boxes around things,
• 5:10
  SPEAKER_01: but prompting a model,
• 5:13
  SPEAKER_01: you know,
• 5:14
  SPEAKER_01: visually or with text.
• 5:16
  SPEAKER_01: That's really fascinating.
• 5:18
  SPEAKER_00: Yeah, I feel like it is very...
• 5:22
  like similar trend as like a coding because cloud code and
• 5:26
  codex is generating code
• 5:27
  SPEAKER_00: and then we review it and it's similar thing like you prompt the query of
• 5:32
  annotation and then you verify and then tweak the sum of the
• 5:35
  annotation and then
• 5:37
  SPEAKER_00: both can be done in demo.
• 5:38
  SPEAKER_01: Right.
• 5:40
  SPEAKER_01: And do you know how popular those kind of features are?
• 5:43
  Is that something that is being picked up rapidly or is that
• 5:47
  still something that
• 5:47
  SPEAKER_01: people just experiment with?
• 5:50
  SPEAKER_00: You mean like this auto annotation?
• 5:53
  SPEAKER_00: Yeah.
• 5:53
  SPEAKER_00: Yeah.
• 5:54
  SPEAKER_00: I think the, I believe it is like becoming mainstream feature.
• 6:00
  So the,
• 6:03
  before some three probably,
• 6:06
  yeah,
• 6:07
  some two was the main like feature.
• 6:10
  SPEAKER_00: And then where you,
• 6:13
  can automatically segment single item by clicking the part of
• 6:18
  that object,
• 6:19
  SPEAKER_00: like a part of the human face,
• 6:21
  SPEAKER_00: and then it's automatically segment the face.
• 6:23
  SPEAKER_00: But it couldn't,
• 6:26
  it wasn't like you annotate a single face and then you detect
• 6:30
  all of the face in a
• 6:31
  SPEAKER_00: picture where hundreds of people are there.
• 6:34
  SPEAKER_00: And then the sum3 unlocked that feature to annotate
• 6:39
  like hundreds or thousands of objects as long as that is
• 6:44
  similar semantically
• 6:45
  similar that i think is completely changed the paradigm and now
• 6:51
  i think industry is
• 6:52
  shifting the direction to use more and more this air assisted
• 6:58
  feature and then
• 6:59
  SPEAKER_00: using human not like a qa as like checking the quality right so
• 7:06
  The machine is doing the heavy lifting and the human is there
• 7:09
  for verification.
• 7:10
  SPEAKER_01: That seems to be the same thing with coding these days.
• 7:13
  SPEAKER_01: Amazing.
• 7:15
  SPEAKER_01: We connected because I think you shared geospatial support in SAM.
• 7:18
  SPEAKER_01: What's the story behind that and what have you implemented?
• 7:21
  SPEAKER_00: And in the LabelMe measure version upgrade,
• 7:24
  SPEAKER_00: we started,
• 7:26
  I realized that I can support larger images and then notice
• 7:30
  that people are wanting
• 7:33
  this feature so that they can use LabelMe for annotating the
• 7:38
  satellite images and
• 7:39
  SPEAKER_00: satellite image tend to be really large.
• 7:42
  And that was kind of new field to me because I came from
• 7:44
  robotics background where
• 7:46
  SPEAKER_00: images are small for real time use.
• 7:49
  SPEAKER_00: And the,
• 7:50
  SPEAKER_00: yeah,
• 7:51
  SPEAKER_00: and that's when I started sharing like,
• 7:56
  SPEAKER_00: yeah,
• 7:56
  SPEAKER_00: the,
• 7:57
  SPEAKER_00: what we can do with label me plus satellite image.
• 8:03
  SPEAKER_00: Yeah.
• 8:03
  And that's,
• 8:04
  I shared on X and then Robin notice,
• 8:06
  oh,
• 8:06
  this kind of interesting.
• 8:08
  And the,
• 8:09
  another thing I noticed was I tried this text prompt in some
• 8:13
  three on satellite image.
• 8:15
  SPEAKER_00: Like there was like some of the, like,
• 8:19
  SPEAKER_00: pond or like a storage tank and then try to segment it,
• 8:24
  but didn't work very well because I think something was trained
• 8:28
  on like natural
• 8:30
  SPEAKER_00: images,
• 8:30
  SPEAKER_00: like taken from iPhone or normal cameras,
• 8:34
  SPEAKER_00: whereas it looks completely different in satellite image.
• 8:38
  And then I noticed that I need to support this like a bounding
• 8:42
  box based query.
• 8:44
  and then yeah it worked very well so i wanted to share that too
• 8:49
  today right so you
• 8:51
  had a few issues it was on your backlog you just needed to
• 8:53
  support much larger
• 8:54
  images and once you had that that unlocked uh supporting
• 8:58
  satellite imagery yeah
• 8:59
  exactly right and was it straightforward yeah i think once i
• 9:09
  realized that that is
• 9:10
  SPEAKER_00: needed because before like
• 9:12
  SPEAKER_00: When I always test image of my use cases, it was just instant.
• 9:18
  SPEAKER_00: So I didn't know, I didn't really pay attention.
• 9:23
  SPEAKER_00: So once I pay attention, yeah, it was kind of straightforward.
• 9:27
  SPEAKER_01: Yes.
• 9:27
  SPEAKER_01: Fantastic.
• 9:28
  So the features are supporting large images,
• 9:31
  but also the different formats?
• 9:34
  SPEAKER_00: Yeah, yeah.
• 9:37
  SPEAKER_00: Yes.
• 9:38
  SPEAKER_00: While working on this,
• 9:39
  I realized that the satellite image tend to be used like a TIFF
• 9:46
  format where I
• 9:48
  SPEAKER_00: normally test with PNG or JPEG normal images.
• 9:51
  SPEAKER_00: So,
• 9:52
  SPEAKER_00: yeah,
• 9:52
  I started supporting multi-channel image and then to normalize
• 9:57
  that visual so that
• 9:59
  SPEAKER_00: you can open it in the LabelMe to start undoing.
• 10:04
  SPEAKER_01: Fantastic.
• 10:05
  We're going to record a demo demo video showing the geospatial
• 10:08
  features after this.
• 10:10
  SPEAKER_01: Just to, I guess, recap, you know, what label me is.
• 10:14
  SPEAKER_01: So it's a desktop application that people can download.
• 10:18
  SPEAKER_01: It's basically written in Python, I think.
• 10:19
  SPEAKER_01: And then you can file that like an executable.
• 10:23
  SPEAKER_01: So people don't need to know any Python to actually use it.
• 10:26
  SPEAKER_01: Is that right?
• 10:27
  SPEAKER_00: Yeah.
• 10:28
  SPEAKER_00: So you can download from GitHub.
• 10:31
  SPEAKER_00: And if you are comfortable with command line,
• 10:32
  SPEAKER_00: you can install with PIP or UB,
• 10:36
  SPEAKER_00: like a Python install edge too.
• 10:38
  SPEAKER_00: But I also provide like a downloadable executable on the website as well.
• 10:44
  SPEAKER_01: Right.
• 10:45
  SPEAKER_01: And is it a paid product?
• 10:47
  SPEAKER_01: Is there a free tier?
• 10:50
  SPEAKER_00: Yeah, currently the...
• 10:53
  the application itself is the free trial for three days and the
• 11:01
  i provide this like
• 11:02
  easy to install app as like a paid product but the you open
• 11:08
  source version you can
• 11:09
  use for free and the the current like basic features are
• 11:15
  already yeah everything
• 11:17
  SPEAKER_00: there
• 11:18
  SPEAKER_01: Amazing.
• 11:18
  SPEAKER_01: All right.
• 11:19
  SPEAKER_01: We'll put the links in the show notes.
• 11:22
  SPEAKER_01: Like I said,
• 11:22
  SPEAKER_01: we're going to record a demo now,
• 11:23
  SPEAKER_01: but once again,
• 11:24
  SPEAKER_01: thank you for joining me today and sharing information about LabelMe.
• 11:29
  SPEAKER_00: Yeah.
• 11:29
  SPEAKER_00: Thank you.