Latest / Satellite image deep learning / TorchGeo 1.0 with Adam Stewart

Transcript

TorchGeo 1.0 with Adam Stewart · Satellite image deep learning

• 0:05
  SPEAKER_00: Adam, welcome to the podcast.
• 0:07
  SPEAKER_00: Yep, thanks for having me.
• 0:08
  SPEAKER_00: This is something that we've been talking about for a very long time,
• 0:11
  SPEAKER_00: and I'm glad we finally made it happen.
• 0:13
  SPEAKER_03: Yeah, and unbelievably, it's your first ever podcast.
• 0:16
  SPEAKER_03: I'm quite thrilled to have you on board.
• 0:17
  SPEAKER_00: First ever podcast.
• 0:17
  SPEAKER_00: Thanks for having me.
• 0:19
  SPEAKER_03: Pleasure.
• 0:19
  SPEAKER_03: All right, let's begin with where you work and what you do.
• 0:22
  SPEAKER_00: Yeah,
• 0:22
  SPEAKER_00: so I'm a postdoctoral researcher at the Technical University of Munich,
• 0:26
  SPEAKER_00: and I'm also the author of Torch Geo.
• 0:29
  So most of my time is kind of managing geospatial software
• 0:33
  libraries for machine learning.
• 0:34
  And for people that aren't so familiar with Torch Geo,
• 0:37
  what's the elevator pitch?
• 0:39
  So if you're familiar with Torch Vision,
• 0:42
  it's the same thing for satellite imagery.
• 0:45
  SPEAKER_00: If you're not familiar with Torch Vision,
• 0:48
  SPEAKER_00: basically in the PyTorch ecosystem,
• 0:51
  SPEAKER_00: you have PyTorch itself,
• 0:52
  SPEAKER_00: which has all the basic building blocks you need for deep learning.
• 0:56
  Torchgeo is all of the high-level abstractions specific to
• 1:00
  geospatial data.
• 1:01
  SPEAKER_00: So the data loaders,
• 1:02
  SPEAKER_00: the foundation models,
• 1:05
  SPEAKER_00: transforms,
• 1:07
  SPEAKER_00: and PyTorch Lightning modules for actually using all of these things in
• 1:10
  SPEAKER_00: combination.
• 1:12
  SPEAKER_03: Right.
• 1:12
  SPEAKER_03: So it basically is the toolkit you need to work with geospatial,
• 1:16
  SPEAKER_03: handles all those geospatial specifics,
• 1:19
  SPEAKER_03: enables you to do things like just load datasets,
• 1:22
  SPEAKER_03: train models.
• 1:23
  SPEAKER_03: Have I missed anything there?
• 1:25
  SPEAKER_00: And a lot of people ask, why can't I just use TorchVision?
• 1:28
  SPEAKER_00: It's all images.
• 1:30
  SPEAKER_00: You know,
• 1:30
  SPEAKER_00: TorchGeo is really designed for things like multispectral images,
• 1:34
  SPEAKER_00: hyperspectral images,
• 1:35
  which don't have three bands and don't work with TorchVision
• 1:40
  transforms or don't
• 1:41
  SPEAKER_00: work with TorchVision models.
• 1:43
  So it's really hyper focused on these kind of obscure image
• 1:48
  data sets where you
• 1:49
  SPEAKER_00: have to deal with things like projection and resampling,
• 1:52
  SPEAKER_00: aligning to
• 1:54
  raster and vector data sets automatically rasterizing a vector
• 1:58
  data set so there's
• 1:59
  a lot of complexity in geospatial data that doesn't exist in
• 2:02
  typical image data
• 2:03
  sets yeah those people don't know how lucky they are yeah all
• 2:06
  right and it's a very
• 2:08
  SPEAKER_03: popular project i mean what are the stats at the moment
• 2:11
  yeah so as far as i know it's i think it's safe to say it's the
• 2:15
  most popular
• 2:16
  geospatial machine learning library so there's three and a half
• 2:20
  thousand stars um
• 2:22
  over a hundred contributors from around the world we've had
• 2:25
  about half a million
• 2:26
  downloads in total since the project started so yeah it's been
• 2:30
  quite successful
• 2:31
  much more successful than i think any of us expected that's
• 2:33
  great to hear and
• 2:35
  recently there's been a few key updates maybe we want to just
• 2:37
  fire through some of
• 2:38
  SPEAKER_03: those
• 2:39
  SPEAKER_00: Yeah,
• 2:40
  SPEAKER_00: so as of yesterday,
• 2:42
  SPEAKER_00: at the time of this recording,
• 2:43
  SPEAKER_00: we just announced the Torchgeo organization.
• 2:46
  SPEAKER_00: So this is something we've been working on for over a year now.
• 2:52
  SPEAKER_00: Basically,
• 2:53
  SPEAKER_00: when Torchgeo started four years ago,
• 2:55
  it started off as my intern project at the AI for Good Research
• 3:00
  Lab at Microsoft.
• 3:02
  SPEAKER_00: And we made it open source.
• 3:04
  SPEAKER_00: We thought it'd be useful for the community.
• 3:06
  And we were actually really surprised that it turned out to be
• 3:08
  very useful for the community.
• 3:09
  A lot of people picked it up,
• 3:11
  started using it,
• 3:12
  started building other libraries on top of it.
• 3:16
  SPEAKER_00: And with all of that,
• 3:17
  SPEAKER_00: with the kind of competition over who has the best foundation models,
• 3:22
  SPEAKER_00: who has the best software libraries,
• 3:24
  SPEAKER_00: we've noticed there's a lot more competition than collaboration.
• 3:27
  SPEAKER_00: And one of the things that we wanted to do was make Torchgeo
• 3:32
  SPEAKER_00: even more open.
• 3:34
  SPEAKER_00: So it's already open source under an MIT license.
• 3:37
  We wanted to also have open and independent governance so that
• 3:40
  it no longer belongs
• 3:41
  SPEAKER_00: to Microsoft,
• 3:43
  SPEAKER_00: it belongs to all of you,
• 3:44
  SPEAKER_00: the Torchgeo contributors.
• 3:46
  SPEAKER_00: So this is something that took a very long time.
• 3:49
  Obviously there's a lot of people at different levels of
• 3:51
  Microsoft who need to
• 3:52
  SPEAKER_00: approve this,
• 3:53
  SPEAKER_00: different lawyers we need to talk to.
• 3:56
  SPEAKER_00: It was actually Microsoft's idea to set up this governance panel.
• 4:01
  So they suggested this minimal viable governance model that
• 4:04
  GitHub published,
• 4:07
  SPEAKER_00: and we decided to adopt that,
• 4:08
  SPEAKER_00: made a few changes here and there,
• 4:10
  SPEAKER_00: but otherwise set things up.
• 4:13
  SPEAKER_00: So there's two different things to be aware of.
• 4:17
  SPEAKER_00: There's the TorchGeo organization, which governs the TorchGeo project.
• 4:22
  SPEAKER_00: The Torchgeo organization is led by the Technical Steering Committee.
• 4:28
  SPEAKER_00: Currently, we have six people.
• 4:29
  SPEAKER_00: So there's me and Niels from Toom.
• 4:32
  SPEAKER_00: There's Caleb and Anthony from Microsoft.
• 4:36
  SPEAKER_00: And then there's Isaac from Werobots and Ashwin from Space42.
• 4:40
  So these are the people currently leading the entire Torchgeo
• 4:43
  organization.
• 4:45
  And then approximately the same set of people is also the
• 4:48
  maintainers of the Python
• 4:50
  SPEAKER_00: library itself.
• 4:52
  SPEAKER_00: There's a lot of other reasons we wanted to do this.
• 4:54
  So Torch Geo itself,
• 4:57
  the Python library,
• 4:59
  is not the only thing that we manage.
• 5:01
  SPEAKER_00: So we also have a Hugging Face community,
• 5:04
  SPEAKER_00: a Zenodo community,
• 5:05
  SPEAKER_00: where we upload data sets and models.
• 5:08
  We have our Slack workspace,
• 5:10
  obviously,
• 5:10
  where we have a code of conduct that we need to enforce.
• 5:14
  And we would also like to grow the Torch Geo organization to
• 5:17
  have multiple projects
• 5:18
  SPEAKER_00: in the future.
• 5:23
  SPEAKER_00: medium range future,
• 5:25
  I'm managing a few master's students who are actually working
• 5:29
  on GIS integration
• 5:31
  SPEAKER_00: for TorchGeo.
• 5:32
  So imagine loading up QGIS or ArcGIS and chatting with a vision
• 5:37
  language foundation
• 5:38
  SPEAKER_00: model and using it to automatically label your satellite imagery for you.
• 5:43
  SPEAKER_00: So that's something that's coming probably in the one year timeframe-ish,
• 5:49
  SPEAKER_00: long after all of the other stuff we're actively working on right now.
• 5:54
  SPEAKER_00: So that would also be something that they would govern.
• 5:58
  SPEAKER_00: So the current technical steering committee is those six people.
• 6:01
  SPEAKER_00: We do want to grow this in the future.
• 6:03
  SPEAKER_00: So if any of you are wondering, how can I take part?
• 6:07
  especially other libraries that are built on top of Torchgeo
• 6:10
  who want to have some
• 6:11
  SPEAKER_00: sort of say in backwards incompatible changes or release schedules.
• 6:17
  SPEAKER_00: It's quite easy to join all of our technical steering committee meetings.
• 6:21
  SPEAKER_00: They're all open to the public.
• 6:22
  Just join the Slack workspace and join the technical steering
• 6:26
  committee channel.
• 6:28
  In terms of actually onboarding people onto the technical
• 6:31
  steering committee,
• 6:32
  SPEAKER_00: we'll probably start slow and onboard more people one at a time,
• 6:36
  but we do actually want to grow the technical steering
• 6:40
  committee to have more
• 6:41
  SPEAKER_00: diverse leadership.
• 6:43
  SPEAKER_00: So we've got people from big tech companies and startups.
• 6:46
  SPEAKER_00: We've got people from academia.
• 6:48
  SPEAKER_00: We're really missing people from government labs.
• 6:52
  SPEAKER_00: So we would love to have people from NASA,
• 6:55
  SPEAKER_00: ESA,
• 6:55
  SPEAKER_00: DLR,
• 6:56
  SPEAKER_00: USGS,
• 6:57
  SPEAKER_00: NOAA,
• 6:57
  all of these other government agencies that actually provide
• 7:00
  the data and provide a
• 7:02
  SPEAKER_00: lot of the expertise for pre-processing and using the data.
• 7:05
  SPEAKER_00: We would love to have them as partners as well.
• 7:09
  It sounds like it's really setting the foundation for the
• 7:12
  future development
• 7:14
  SPEAKER_03: objectives and also just broadening the sort of engagement.
• 7:18
  SPEAKER_00: Yeah, exactly.
• 7:19
  SPEAKER_00: I mean,
• 7:19
  SPEAKER_00: if you think of it this way,
• 7:21
  SPEAKER_00: I do a lot of research on satellite imagery for agriculture.
• 7:27
  SPEAKER_00: And if it was only me making decisions,
• 7:29
  SPEAKER_00: we would only have agriculture datasets,
• 7:31
  SPEAKER_00: only have agriculture models.
• 7:33
  SPEAKER_00: But the more people we have from different application domains,
• 7:37
  SPEAKER_00: different experience levels,
• 7:39
  people who are only computer scientists who don't understand
• 7:42
  anything about
• 7:43
  SPEAKER_00: satellite imagery,
• 7:44
  people who are only remote sensing experts who don't know any
• 7:47
  machine learning.
• 7:48
  If you don't have a more diverse community leading things,
• 7:51
  you end up with
• 7:52
  documentation that only works for one audience or features that
• 7:56
  only cater to one audience.
• 7:57
  SPEAKER_03: Absolutely.
• 7:58
  SPEAKER_03: And is the provisional plan to have meetings periodically?
• 8:02
  SPEAKER_03: What do the committee actually do then?
• 8:03
  SPEAKER_00: Yeah, we'll have meetings once a month.
• 8:05
  It will probably be the first Monday of the month at 6pm
• 8:09
  Central European time.
• 8:11
  I think next month we're going to shift by a week just because
• 8:15
  of Labor Day in the US.
• 8:16
  SPEAKER_00: But other than that, yeah, we want to meet every month.
• 8:19
  SPEAKER_00: We want to have it open to the public.
• 8:20
  SPEAKER_00: Everyone should come and join and
• 8:22
  Just chime in and say,
• 8:23
  hey,
• 8:24
  I'm using Torch Geo,
• 8:25
  but this feature isn't working for me.
• 8:27
  SPEAKER_00: Or why don't you have this feature?
• 8:28
  SPEAKER_00: Or I'm interested in actually contributing.
• 8:31
  SPEAKER_00: How do I get started?
• 8:32
  SPEAKER_00: What do you need?
• 8:34
  SPEAKER_00: Questions like that are always open.
• 8:37
  SPEAKER_03: Okay, not long to go to that first meeting.
• 8:40
  SPEAKER_03: There's also some other changes coming.
• 8:44
  SPEAKER_00: Yes,
• 8:44
  SPEAKER_00: the biggest thing that we've been working on,
• 8:46
  SPEAKER_00: and this is also almost a year in the making,
• 8:49
  SPEAKER_00: I think we started pretty heavily in October.
• 8:52
  We really looked at the features that Torchgeo has and looked
• 8:57
  at the stability of
• 8:57
  SPEAKER_00: Torchgeo and we said,
• 8:59
  SPEAKER_00: Things are starting to stabilize,
• 9:01
  SPEAKER_00: especially as other software libraries build on top of Torchgeo.
• 9:04
  SPEAKER_00: They want some guarantee of backwards compatibility.
• 9:07
  SPEAKER_00: They want more deprecation cycles long term.
• 9:12
  So they want something close to a Torchgeo 1.0 release where we
• 9:16
  can guarantee some
• 9:17
  SPEAKER_00: level of stability.
• 9:19
  In terms of the features that we already have,
• 9:21
  we have most of the things that we want.
• 9:24
  But I think the one major feature that we're still missing is
• 9:28
  complete time series support.
• 9:31
  This is something that is lacking in almost all GeoML libraries
• 9:35
  at the moment.
• 9:36
  SPEAKER_00: I think the only shining example would be SITs from the R ecosystem,
• 9:40
  where they're basically focused solely on time series support
• 9:44
  and very good time
• 9:44
  SPEAKER_00: series support.
• 9:46
  So we tried to take inspiration from SITS and other libraries
• 9:49
  that have tried to do
• 9:50
  time series support and looked at how we could integrate it
• 9:53
  into Torchgeo.
• 9:55
  When I mentioned time series support,
• 9:57
  I'm talking about many different things.
• 9:59
  SPEAKER_00: So there's 1D time series support.
• 10:02
  So imagine you're working with a weather station or an air
• 10:07
  pollution sensor,
• 10:08
  and you want to forecast future levels of air pollution at that
• 10:11
  specific location.
• 10:12
  SPEAKER_00: So that would be the 1D example.
• 10:15
  In terms of 2D,
• 10:16
  TorchGeo has always done 2D,
• 10:18
  so that's quite well covered.
• 10:20
  SPEAKER_00: But we also want to support 3D.
• 10:22
  So imagine you have a location that's been damaged by some sort
• 10:28
  of natural disaster
• 10:29
  SPEAKER_00: or by warfare,
• 10:30
  SPEAKER_00: and you want to do change detection.
• 10:32
  SPEAKER_00: So you have two timestamps for one location,
• 10:35
  SPEAKER_00: two images before and after,
• 10:37
  SPEAKER_00: and you want to label damaged buildings or...
• 10:40
  SPEAKER_00: places that you should target for recovery.
• 10:43
  But also moving beyond the two timestamp,
• 10:45
  we want to support multiple timestamps.
• 10:47
  SPEAKER_00: So let's say you're doing something like crop type mapping,
• 10:50
  SPEAKER_00: where you've got an entire year's worth of Sentinel-2 imagery,
• 10:54
  SPEAKER_00: and you actually want to take advantage of both the time dimension,
• 10:57
  SPEAKER_00: but also the spatial dimension.
• 10:59
  SPEAKER_00: So that's the 3D case.
• 11:01
  SPEAKER_00: And then there's also the 4D case,
• 11:03
  SPEAKER_00: which is becoming more and more popular,
• 11:04
  SPEAKER_00: where you're targeting things like medium range weather forecasting,
• 11:08
  SPEAKER_00: now casting,
• 11:09
  SPEAKER_00: even decadal scale climate modeling.
• 11:13
  So here you have the X and Y dimension,
• 11:15
  but also the Z dimension for altitude.
• 11:18
  SPEAKER_00: and then, of course, the time dimension as well.
• 11:20
  SPEAKER_00: All of these are covered under time series support.
• 11:24
  We actually chose these specific applications to really
• 11:27
  highlight the fact that
• 11:28
  SPEAKER_00: TorchGEO is not only for satellite imagery.
• 11:31
  SPEAKER_00: Obviously,
• 11:31
  SPEAKER_00: most of the datasets,
• 11:32
  SPEAKER_00: most of the features that we add are targeted towards satellite imagery.
• 11:36
  SPEAKER_00: But we just merged our first weather forecasting model.
• 11:40
  We have many data sets for point data and a lot of other very
• 11:44
  interesting applications.
• 11:46
  SPEAKER_00: So that's everything that's covered.
• 11:48
  SPEAKER_00: Again, this is something we've been working on since last October.
• 11:52
  So there's a lot more work involved in this than any of us
• 11:55
  really expected.
• 11:57
  I started looking at some of the backend design of Torchgeo
• 12:01
  that I came up with
• 12:02
  SPEAKER_00: four years ago,
• 12:02
  SPEAKER_00: and I was like,
• 12:03
  SPEAKER_00: why did I do it this way?
• 12:04
  SPEAKER_00: This is so inflexible.
• 12:06
  SPEAKER_00: It's just not practical.
• 12:08
  So,
• 12:08
  for example,
• 12:09
  we used to use Archery for all of our spatiotemporal indexing.
• 12:13
  So we did keep track of the spatio-temporal bounding boxes of
• 12:17
  all of our files so
• 12:19
  SPEAKER_00: that we could sample them quickly and efficiently.
• 12:23
  The problem with that is that our tree lacks a lot of features
• 12:28
  that we need.
• 12:28
  For example,
• 12:29
  we want to be able to separately index space or time,
• 12:33
  not both at the same time.
• 12:35
  SPEAKER_00: So we did a very long and intensive,
• 12:39
  SPEAKER_00: basically,
• 12:40
  SPEAKER_00: software literature review on all the options available to us.
• 12:43
  SPEAKER_00: We looked at Artree,
• 12:44
  SPEAKER_00: we looked at Shapely's STR tree,
• 12:47
  SPEAKER_00: we looked at Geopandas,
• 12:48
  and then we also looked at the Stack ecosystem and tried to
• 12:52
  find something that was
• 12:53
  SPEAKER_00: a little bit more flexible,
• 12:54
  SPEAKER_00: more powerful,
• 12:55
  SPEAKER_00: and had a lot more features that we could use for this.
• 12:58
  SPEAKER_00: So we ended up switching to Geopandas.
• 13:00
  So I basically had to rewrite every single geodataset and
• 13:03
  geosampler from scratch
• 13:04
  SPEAKER_00: to work with Geopandas.
• 13:07
  SPEAKER_00: At the same time,
• 13:08
  SPEAKER_00: we're also looking at the data loaders,
• 13:10
  SPEAKER_00: how they load the data,
• 13:11
  SPEAKER_00: what data they're able to load.
• 13:13
  SPEAKER_00: We previously used RasterIO for all data loading.
• 13:16
  And RasterIO is great because it wraps around GDAL and anything
• 13:19
  that GDAL can read,
• 13:20
  SPEAKER_00: Torch Geo can read.
• 13:22
  But there's a lot of things that we want to be able to read
• 13:25
  that aren't in that list.
• 13:27
  So,
• 13:27
  for example,
• 13:28
  a lot of the weather and climate data is stored in NetCDF,
• 13:33
  HDF5.
• 13:34
  Technically,
• 13:35
  GDAL can read these things,
• 13:36
  but not the version that's on PyPy.
• 13:39
  So we started looking more and more at X-Array and at the
• 13:42
  moment I'm working on a
• 13:43
  pull request to actually replace RasterIO with Rio X-Array so
• 13:48
  that we can load
• 13:49
  SPEAKER_00: anything that RasterIO can load,
• 13:51
  SPEAKER_00: anything that NetCDF or HDF5 can load.
• 13:56
  SPEAKER_00: And have a little bit more flexibility.
• 13:59
  We're also planning on replacing Fiona with Geopandas so that
• 14:02
  we can drop some of
• 14:03
  SPEAKER_00: these dependencies that we no longer directly rely on.
• 14:06
  They become indirect dependencies where there's higher level
• 14:09
  libraries that wrap those instead.
• 14:11
  SPEAKER_00: Yeah.
• 14:12
  SPEAKER_00: So this will also help a lot for scalability.
• 14:16
  Things like Geopandas and Rio X-Ray have really good
• 14:18
  integration with Dask.
• 14:19
  SPEAKER_00: You can very easily parallelize them.
• 14:21
  SPEAKER_00: They scale up quite well.
• 14:23
  So there's a lot of very big backend changes that are very
• 14:26
  backwards and compatible.
• 14:27
  That's another reason why we wanted to make this 1.0 release
• 14:30
  where we say,
• 14:31
  SPEAKER_00: hey,
• 14:32
  SPEAKER_00: we're changing the major version number.
• 14:33
  SPEAKER_00: This is a very big change.
• 14:35
  SPEAKER_00: Expect things to break.
• 14:37
  We will be making release candidates before we make the
• 14:40
  official release so that
• 14:41
  people can test them and make sure everything is ready to go
• 14:44
  for the official
• 14:46
  SPEAKER_00: release.
• 14:47
  But basically everything related to geodataset and geosampler
• 14:51
  is probably going to break.
• 14:53
  SPEAKER_00: The rest of the library doesn't change so much.
• 14:55
  SPEAKER_00: It should be mostly backwards compatible.
• 14:58
  SPEAKER_00: We're also adding a lot of new features.
• 15:00
  SPEAKER_00: So for example, we just merged a new change detection trainer.
• 15:05
  So you can do binary multi-class and multi-label change
• 15:08
  detection now with TorchGeo.
• 15:10
  We've got a bunch of change detection models that we already
• 15:13
  have integrated with
• 15:14
  SPEAKER_00: that and we're expecting to add more as well.
• 15:17
  And we're working on an autoregression trainer that you can
• 15:19
  plug in for things like
• 15:20
  SPEAKER_00: air pollution forecasting or weather forecasting,
• 15:24
  SPEAKER_00: where you can integrate all these existing foundation models.
• 15:27
  SPEAKER_03: Wow, it's a really exciting set of features.
• 15:29
  SPEAKER_03: And I think, as you mentioned, kind of unique in the ecosystem as well.
• 15:34
  So it's partly about features,
• 15:36
  but also about saying now the foundations are there.
• 15:40
  SPEAKER_03: This is 1.0, and we're going to make some backward compatibility changes
• 15:44
  breaks but also guarantees for the future it's interesting i
• 15:48
  think i i talked to
• 15:49
  some of the folks from um pie torch lightning for example and
• 15:52
  one of their
• 15:53
  SPEAKER_00: philosophies was you know get your software to a point where it's stable
• 15:58
  SPEAKER_00: And then move on to the next big thing.
• 16:00
  SPEAKER_00: And Torchgeo is not going away.
• 16:02
  SPEAKER_00: We're going to continue to maintain it for the long term future.
• 16:05
  There's a lot of other features that we do want to add in the
• 16:07
  near future,
• 16:08
  SPEAKER_00: things like stack integration,
• 16:10
  SPEAKER_00: that they're not necessarily backwards incompatible,
• 16:13
  SPEAKER_00: but they're much needed features.
• 16:15
  SPEAKER_00: But we wanted to get it to a point where it's relatively stable.
• 16:18
  SPEAKER_00: where I as the main developer can take a step back and say,
• 16:22
  SPEAKER_00: I'm going to review PRs,
• 16:24
  but I want the community to really lead these PRs and do a lot
• 16:28
  of the work.
• 16:29
  SPEAKER_00: And while that's happening,
• 16:31
  SPEAKER_00: I can explore new things and say,
• 16:33
  SPEAKER_00: I want to target GIS integration for this low-code,
• 16:36
  SPEAKER_00: no-code environment for non-computer science experts.
• 16:40
  SPEAKER_00: This is one of the things that we've gotten a lot of
• 16:43
  know kind of i wouldn't say criticism but but jokes about where
• 16:48
  torch geo is the
• 16:49
  best library ever for a machine learning expert but for a
• 16:53
  remote sensing expert
• 16:54
  there's a lot of things in torch geo that are still quite hard
• 16:57
  to to wrap your head
• 16:58
  around what is a geo sampler why would i need it what is a
• 17:02
  pytorch data loader what
• 17:03
  SPEAKER_00: is pytorch lightning why do i have to learn all these new things
• 17:06
  SPEAKER_00: in order to do machine learning on data.
• 17:09
  SPEAKER_00: It's obviously much more complex than a simple scikit-learn workflow.
• 17:13
  So we wanted to take a lot of the abstractions that we have
• 17:16
  with PyTorch Lightning
• 17:18
  SPEAKER_00: and bring those to the remote sensing audience in a way that's much more
• 17:21
  SPEAKER_00: accessible.
• 17:23
  SPEAKER_00: Exciting.
• 17:24
  You briefly mentioned that Torchgeo is built upon by a few
• 17:28
  other libraries.
• 17:28
  Do you want to just list some of those and maybe talk about how
• 17:31
  it relates?
• 17:32
  Yeah,
• 17:32
  so the big one that I think a lot of people know about is
• 17:34
  TerraTorch.
• 17:35
  SPEAKER_00: But one that actually has even more stars is Geo AI.
• 17:39
  And then also the GeoBench benchmarking suite uses Torchgeo to
• 17:44
  curate many of its data sets.
• 17:46
  SPEAKER_00: Mm-hmm.
• 17:47
  SPEAKER_00: NASA's PyTorch Caney uses Torch Geo a little bit for data loading.
• 17:51
  So there's a lot of different organizations that are building
• 17:53
  on top of Torch Geo.
• 17:56
  One of the reasons we wanted this independent governance is so
• 17:59
  that it's easier for
• 18:00
  SPEAKER_00: those people to build on top of Torch Geo,
• 18:02
  but also we would love those people to join the technical
• 18:05
  steering committee
• 18:06
  meetings and eventually join the official technical steering
• 18:09
  committee board.
• 18:11
  SPEAKER_00: So we would love folks from NASA,
• 18:12
  SPEAKER_00: from IBM,
• 18:13
  from all of these other different institutions to join and help
• 18:17
  kind of drive the
• 18:18
  SPEAKER_00: future directions of Torchgeo so that we can collaborate.
• 18:22
  SPEAKER_00: We're not reinventing the wheel.
• 18:25
  SPEAKER_00: For example,
• 18:25
  SPEAKER_00: TerraTorch has a bunch of data loaders,
• 18:27
  SPEAKER_00: but what they really shine at is building on top of Torchgeo,
• 18:30
  SPEAKER_00: adding these higher level abstractions for...
• 18:34
  basically building a foundation model playground where you can
• 18:36
  fine tune and
• 18:38
  evaluate your foundation models much more easily than you can
• 18:42
  directly in Torchgeo.
• 18:43
  So we would love to take some of the more experimental features
• 18:47
  that TerraTorch has
• 18:48
  SPEAKER_00: and integrate them more directly into Torchgeo,
• 18:51
  but also offload many of the things that they don't want to
• 18:54
  maintain themselves so
• 18:56
  SPEAKER_00: that when there's big backwards incompatible changes to the data loaders,
• 19:00
  SPEAKER_00: we fix them,
• 19:01
  SPEAKER_00: not them.
• 19:03
  SPEAKER_03: Right.
• 19:03
  SPEAKER_03: So coordination will definitely benefit all parties there.
• 19:08
  One topic that's getting a lot of interest at the moment is
• 19:10
  foundational models.
• 19:11
  And I think I've seen an increasing number of foundational
• 19:13
  models being supported in Torchgeo.
• 19:15
  SPEAKER_03: What's your take on foundational models?
• 19:17
  SPEAKER_00: Yeah, so I'm obviously biased because I'm a foundation model researcher.
• 19:21
  So I've been involved in quite a lot of foundation model
• 19:24
  projects over the years.
• 19:26
  SPEAKER_00: The first one I got started with was SSL4EO Landsat,
• 19:29
  which was really the first foundation models ever pre-trained
• 19:34
  on Landsat imagery
• 19:36
  SPEAKER_00: and the largest Landsat pre-training data set ever created,
• 19:39
  SPEAKER_00: and I think maybe still to this day.
• 19:42
  SPEAKER_00: So we created 5 million Landsat images for Landsats 4 through 9.
• 19:47
  SPEAKER_00: We had several different foundation models of different ResNet and vision
• 19:52
  SPEAKER_00: transformer backbones,
• 19:54
  SPEAKER_00: different SSL techniques,
• 19:55
  SPEAKER_00: different generations of Landsat satellites,
• 19:59
  SPEAKER_00: different surface on top of atmosphere products.
• 20:02
  SPEAKER_00: We basically had 40 foundation models for Landsat.
• 20:05
  SPEAKER_00: And one of the things that reviewers said was,
• 20:08
  SPEAKER_00: What's the point of 4D foundation model?
• 20:10
  SPEAKER_00: Shouldn't you just have one foundation model?
• 20:12
  SPEAKER_00: Isn't that the definition?
• 20:14
  SPEAKER_00: And back in that time,
• 20:15
  foundation model was really just a big buzzword for any large
• 20:19
  model trained on a
• 20:20
  SPEAKER_00: lot of data,
• 20:20
  SPEAKER_00: usually using self-supervision.
• 20:24
  SPEAKER_00: When I joined the Technical University of Munich,
• 20:27
  one of the things that I wanted to do was we've got a lot of
• 20:31
  foundation models for
• 20:32
  SPEAKER_00: Sentinel-1 and 2.
• 20:34
  SPEAKER_00: We now have some for Landsat.
• 20:37
  There's still thousands of other satellites out there that
• 20:39
  we're not covering.
• 20:40
  We really want foundation models for every satellite,
• 20:43
  not just the most popular ones.
• 20:45
  SPEAKER_00: Yeah.
• 20:46
  SPEAKER_00: And one of the things that really intrigued me,
• 20:48
  one of the reasons I joined this lab was that there was a lot
• 20:52
  of really exciting
• 20:53
  SPEAKER_00: stuff going on with foundation models here.
• 20:56
  SPEAKER_00: So right when I joined, DOFA was already well underway.
• 20:59
  SPEAKER_00: So DOFA is the dynamic one for all model.
• 21:02
  SPEAKER_00: It's this neuroplasticity paper that people might have seen on archive.
• 21:07
  SPEAKER_00: Basically, the idea with DOFA is that you can encode
• 21:12
  uh use a use a dynamic hyper network to take the wavelength of
• 21:17
  each spectral band
• 21:18
  as input and dynamically generate the first layer of your
• 21:23
  vision transformer so
• 21:24
  that as long as you know the wavelengths of your spectral bands
• 21:28
  this single set of
• 21:29
  pre-trained weights works for every earth observation satellite
• 21:34
  ever launched
• 21:36
  SPEAKER_00: and that will ever be launched in the future.
• 21:39
  SPEAKER_00: So this is an extremely powerful,
• 21:41
  SPEAKER_00: flexible,
• 21:42
  SPEAKER_00: multimodal model,
• 21:43
  SPEAKER_00: really the first such multimodal model that existed.
• 21:46
  There's been a lot of other things since then that I've been
• 21:48
  involved with.
• 21:50
  SPEAKER_00: The Copernicus Foundation model,
• 21:51
  SPEAKER_00: which supports Sentinel-1,
• 21:53
  SPEAKER_00: 2,
• 21:54
  SPEAKER_00: 3,
• 21:54
  SPEAKER_00: 5P,
• 21:55
  SPEAKER_00: and Copernicus DEM,
• 21:56
  SPEAKER_00: where we're adding in a language encoder,
• 21:58
  SPEAKER_00: allowing us to support non-optical modalities.
• 22:02
  SPEAKER_00: Panopticon,
• 22:03
  SPEAKER_00: which takes a lot of the ideas of DOFA,
• 22:06
  SPEAKER_00: incorporates a lot of the ideas of Dyno V2,
• 22:09
  SPEAKER_00: and generally makes a more robust and more powerful foundation model.
• 22:14
  SPEAKER_00: We have DOFA Clip,
• 22:15
  which is a real vision language model where you can do
• 22:20
  zero-shot learning,
• 22:21
  SPEAKER_00: talk to your model,
• 22:22
  SPEAKER_00: and ask it to label your data for you without any fine-tuning required.
• 22:26
  SPEAKER_02: Mm-hmm.
• 22:27
  SPEAKER_00: So these are the things I've been involved with.
• 22:30
  SPEAKER_00: As you've noted, foundation models are the latest and greatest buzzword.
• 22:33
  SPEAKER_00: There's a lot of hype centered around this new concept.
• 22:37
  SPEAKER_00: Everyone wants to get into foundation models.
• 22:40
  SPEAKER_00: At the same time, the hype is obviously bigger than what it's delivering.
• 22:44
  SPEAKER_00: There's still a lot of people saying,
• 22:46
  you know it's great that you've spent millions of dollars
• 22:49
  training these gigantic
• 22:50
  foundation models on tons of compute tons of data but my random
• 22:54
  forest model still
• 22:55
  works quite well my my supervised unit still works quite well
• 22:59
  and i think the
• 23:00
  SPEAKER_00: things that are really missing in that discussion are the scientists so
• 23:07
  you know a lot of the the higher profile things i've seen
• 23:11
  recently are things like
• 23:13
  you know i tried i tried comparing um all of these foundation
• 23:17
  models to just a
• 23:18
  regular unit and i trained them on big earth net and unit
• 23:22
  worked better so
• 23:23
  SPEAKER_00: foundation models are useless
• 23:25
  SPEAKER_00: And what people don't realize is,
• 23:27
  SPEAKER_00: one,
• 23:27
  you're doing supervised fine tuning on one model and linear
• 23:33
  probing on all the
• 23:33
  SPEAKER_00: other ones.
• 23:34
  SPEAKER_00: And that's not a fair comparison.
• 23:35
  SPEAKER_00: You can still do supervised fine tuning on your foundation models.
• 23:40
  SPEAKER_00: And the real benefit is that you can...
• 23:43
  learn much quicker you already have a very good start so you
• 23:46
  can fine-tune directly
• 23:48
  from pre-trained weights instead of random weights that's a
• 23:50
  huge advantage for
• 23:51
  anyone who's not a machine learning expert but also the data
• 23:55
  set sizes that real
• 23:56
  scientists work with are orders of magnitude different than
• 23:59
  something like big
• 24:00
  SPEAKER_00: earth net which has a half a million images in it
• 24:03
  SPEAKER_00: So a lot of scientists I talk to,
• 24:05
  SPEAKER_00: they come to me and say,
• 24:06
  SPEAKER_00: hey,
• 24:06
  SPEAKER_00: I want to get into deep learning.
• 24:07
  SPEAKER_00: I have this huge data set that we've collected over the last five years.
• 24:11
  SPEAKER_00: It has over 100 images in it.
• 24:13
  SPEAKER_00: How can I get started?
• 24:15
  SPEAKER_00: And when you hear those numbers,
• 24:16
  SPEAKER_00: you say,
• 24:17
  SPEAKER_00: well,
• 24:18
  SPEAKER_00: huge means something very different to the deep learning community.
• 24:21
  SPEAKER_00: But now that I have these foundation models,
• 24:23
  100 images is actually sufficient for the first time for really
• 24:27
  fine tuning for
• 24:28
  SPEAKER_00: these very diverse applications.
• 24:31
  SPEAKER_00: So I do think that foundation models show a lot of promise.
• 24:35
  SPEAKER_00: I also think there's a lot of hype around them.
• 24:37
  I'm hoping that the hype dies down so that we can actually
• 24:40
  start collaborating.
• 24:43
  As you've noticed,
• 24:44
  there's a lot of promise of one foundation model to rule them
• 24:48
  all.
• 24:48
  SPEAKER_00: And in no domain is there one foundation model.
• 24:51
  SPEAKER_00: In language, in vision, in vision language, there's always competitors.
• 24:55
  SPEAKER_00: There's always people competing for the latest and greatest.
• 24:58
  SPEAKER_00: I don't think that's going to change anytime soon.
• 25:00
  But I'm going to sit here and eat my popcorn and enjoy the ride
• 25:04
  and just keep
• 25:04
  adding more and more foundation models to Torchgeo so that it's
• 25:07
  easy for people to
• 25:08
  SPEAKER_00: use them and easy for people to compare them.
• 25:11
  SPEAKER_03: Right,
• 25:11
  so really supporting all those investigations and making a bit
• 25:14
  of a hub where
• 25:15
  SPEAKER_03: people can...
• 25:15
  SPEAKER_00: And we also have a lot of these pre-training data sets in TorchGeo.
• 25:18
  We have things like Seasonal Contrast,
• 25:21
  SSL4EO,
• 25:22
  S1-2,
• 25:23
  and Landsat,
• 25:25
  Copernicus Pre-Train.
• 25:27
  SPEAKER_00: Big EarthNet,
• 25:27
  SPEAKER_00: you know,
• 25:28
  we have a lot of the data sets that people need to pre-train
• 25:30
  these models.
• 25:31
  We also have a lot of the data sets that people need to
• 25:34
  evaluate these models.
• 25:35
  SPEAKER_00: So Copernicus Bench,
• 25:36
  SPEAKER_00: SpaceNet,
• 25:37
  and we would love to add a lot more of these higher order meta
• 25:41
  benchmark suites
• 25:42
  SPEAKER_00: that combine all of these individual benchmark data sets.
• 25:46
  SPEAKER_03: I guess one last question on the foundation board,
• 25:48
  SPEAKER_03: as you mentioned about the linear probing versus fine tuning.
• 25:51
  SPEAKER_03: Is Torchgeo going to stay focused on one or the other?
• 25:54
  SPEAKER_03: Like, for instance, people interested in just using embeddings.
• 25:57
  SPEAKER_03: Is that something Torchgeo would get into or...?
• 26:00
  SPEAKER_00: So two questions.
• 26:02
  SPEAKER_00: On the linear probing versus fine tuning,
• 26:03
  SPEAKER_00: we do support both and we don't have any personal preference.
• 26:07
  SPEAKER_00: So for example,
• 26:07
  SPEAKER_00: if you use semantic segmentation task,
• 26:10
  you can tell it to freeze the decoder or freeze both the
• 26:15
  encoder and decoder and
• 26:16
  SPEAKER_00: only do linear probing at the end layer.
• 26:19
  SPEAKER_00: Or you can fine tune the entire thing.
• 26:22
  SPEAKER_00: So we're pretty flexible.
• 26:23
  SPEAKER_00: We want to remain flexible.
• 26:25
  SPEAKER_00: All of these are good use cases.
• 26:28
  SPEAKER_00: Really depends on how much compute you have, how much expertise you have.
• 26:31
  SPEAKER_00: Choose whatever works for you and use it.
• 26:33
  SPEAKER_00: We would actually like to make that more flexible in the future.
• 26:36
  SPEAKER_00: And I think that's something that TerraTorch has done quite well.
• 26:39
  SPEAKER_00: So depending on how we end up collaborating with folks from IBM,
• 26:44
  SPEAKER_00: we would love to make that even more flexible in the future.
• 26:47
  SPEAKER_00: On the topic of embeddings,
• 26:48
  a lot of people have gotten really excited about embeddings
• 26:51
  recently after the
• 26:52
  SPEAKER_00: release of Google's embeddings.
• 26:54
  SPEAKER_00: I think Presto also released their own embeddings recently.
• 26:57
  SPEAKER_00: Major Tom had embeddings long before both of those.
• 27:00
  SPEAKER_00: So I do think that embeddings are the future.
• 27:04
  SPEAKER_00: We are building embeddings for our own Copernicus Foundation model.
• 27:07
  So we want to have embeddings that not only incorporate
• 27:11
  satellite imagery,
• 27:12
  SPEAKER_00: but also incorporate things like ERA5,
• 27:14
  SPEAKER_00: which I don't think anyone is really trying to do at the moment.
• 27:17
  So embeddings are a really nice opportunity to take away a lot
• 27:21
  of the computational
• 27:22
  SPEAKER_00: load from domain experts so that they can rely simply on
• 27:28
  SPEAKER_00: a learned embedding,
• 27:29
  SPEAKER_00: a low dimensional representation of a location somewhere on Earth,
• 27:34
  SPEAKER_00: and focus more on the decoder side.
• 27:36
  SPEAKER_00: So there's a lot less compute required.
• 27:38
  SPEAKER_00: At the same time, embeddings are definitely in their infancy.
• 27:41
  SPEAKER_00: There's a lot of issues with the Google embeddings,
• 27:43
  SPEAKER_00: with all of the embeddings that currently exist.
• 27:46
  So just like there's a lot of progress that still needs to be
• 27:49
  made in foundation
• 27:50
  SPEAKER_00: model research,
• 27:51
  there's a lot of progress that needs to be made in embedding
• 27:53
  research as well.
• 27:54
  SPEAKER_03: Brilliant.
• 27:55
  SPEAKER_03: Well, thanks for sharing your opinion on those topics.
• 27:57
  SPEAKER_03: It's been a really fascinating conversation.
• 27:59
  SPEAKER_03: So much going on, so much potential, I think, still there.
• 28:01
  SPEAKER_03: Obviously, Torchgeo is on GitHub.
• 28:04
  SPEAKER_03: It also has a website.
• 28:06
  You mentioned the Slack channel being a place where people can
• 28:08
  go and connect with
• 28:09
  SPEAKER_03: the developers as well.
• 28:11
  SPEAKER_03: Any other places that people should look to connect and follow along?
• 28:15
  SPEAKER_00: Yeah, that's most of them.
• 28:16
  SPEAKER_00: I mean, you can email me, but I will eventually reply.
• 28:18
  SPEAKER_00: If you message me on Slack, I'll actually probably reply quicker.
• 28:22
  If you message me on GitHub,
• 28:23
  I might actually be on GitHub right now answering your
• 28:25
  questions.
• 28:26
  SPEAKER_00: So those are definitely the best places to reach out.
• 28:29
  SPEAKER_00: If you go to the GitHub,
• 28:31
  SPEAKER_00: which is github.com slash torchgeo slash torchgeo,
• 28:35
  SPEAKER_00: there are links to all of these other communities.
• 28:37
  SPEAKER_00: There's a link to join the Slack channel.
• 28:39
  SPEAKER_00: There's a link to join the Hugging Face community.
• 28:41
  you can join and participate in discussions and help drive
• 28:46
  future development of Torch Geo.
• 28:48
  SPEAKER_00: Brilliant.
• 28:49
  SPEAKER_03: Okay.
• 28:49
  Well,
• 28:49
  once again,
• 28:50
  thanks for joining me today and I look forward to connecting in
• 28:52
  the future.
• 28:53
  SPEAKER_00: Yeah.
• 28:53
  SPEAKER_00: Thanks for having me.