Latest / Satellite image deep learning / Building OlmoEarth: AI2’s Open Foundation Model for Satellite Imagery

Transcript

Building OlmoEarth: AI2’s Open Foundation Model for Satellite Imagery · Satellite image deep learning

• 0:05
  SPEAKER_01: Hi Joe, welcome to the podcast.
• 0:07
  SPEAKER_00: Hello, thanks for having me.
• 0:10
  SPEAKER_01: Let's begin with a brief intro to yourself.
• 0:13
  SPEAKER_00: Yeah, my name is Joe.
• 0:15
  I've been working in computer vision and sort of machine
• 0:19
  learning research for,
• 0:21
  SPEAKER_00: you know,
• 0:22
  SPEAKER_00: I don't know,
• 0:23
  SPEAKER_00: since about 2013 when I started grad school,
• 0:25
  SPEAKER_00: I guess.
• 0:27
  SPEAKER_00: I work now at the Allen Institute for Artificial Intelligence, AI2.
• 0:32
  I've been working recently on geospatial foundation models,
• 0:35
  which is why I'm here today.
• 0:38
  SPEAKER_01: Fantastic.
• 0:39
  SPEAKER_01: For people that are not familiar with AI2, what's its general remit?
• 0:44
  SPEAKER_00: Paul Allen started AI2.
• 0:45
  Paul Allen,
• 0:46
  one of the co-founders of Microsoft,
• 0:49
  started AI2 I think about 12 years ago.
• 0:53
  And he was interested basically in artificial intelligence for
• 0:58
  solving big real-world problems.
• 1:00
  I think it's something like AI for the world's biggest
• 1:03
  challenges or something like that.
• 1:06
  it's it's gone through a lot of different sort of phases over
• 1:10
  its existence but
• 1:12
  SPEAKER_00: these days it focuses a lot on
• 1:16
  SPEAKER_00: open ecosystem stuff,
• 1:18
  SPEAKER_00: so open language models,
• 1:20
  SPEAKER_00: open vision models,
• 1:21
  sort of training things in the open so people can understand
• 1:24
  how these things are
• 1:25
  SPEAKER_00: trained,
• 1:25
  SPEAKER_00: what kind of data goes in,
• 1:26
  SPEAKER_00: what kind of compute,
• 1:27
  SPEAKER_00: that kind of stuff.
• 1:28
  It also has a number of AI for science initiatives,
• 1:33
  so working on AI for cancer research,
• 1:35
  AI for
• 1:37
  SPEAKER_00: general scientific research, those sorts of things, paper discovery.
• 1:41
  And then I fall under kind of the AI for the environment,
• 1:47
  AI for climate push that AI2 is doing.
• 1:51
  SPEAKER_00: So as part of that,
• 1:52
  SPEAKER_00: there are a number of different projects that AI2 is working on,
• 1:56
  SPEAKER_00: but there is the climate modeling team,
• 1:59
  SPEAKER_00: which is just trying to do sort of standard climate modeling stuff,
• 2:04
  SPEAKER_00: but augmenting it using AI.
• 2:07
  SPEAKER_00: There's projects like Earth Ranger,
• 2:09
  SPEAKER_00: tracking sort of endangered species,
• 2:12
  SPEAKER_00: Skylight,
• 2:13
  SPEAKER_00: looking for illegal,
• 2:15
  SPEAKER_00: unregulated fishing,
• 2:17
  SPEAKER_00: and then Ulmo Earth.
• 2:20
  SPEAKER_01: So that's probably a great time to learn a bit more about Ulmo Earth.
• 2:25
  SPEAKER_00: Yeah,
• 2:25
  SPEAKER_00: so Omaha actually came out of all of these other initiatives,
• 2:29
  SPEAKER_00: basically things like Earth Ranger and Skylight.
• 2:32
  AI2 had a bunch of partnerships already with these external
• 2:35
  organizations that were
• 2:37
  SPEAKER_00: trying to solve various problems.
• 2:39
  SPEAKER_00: And one of the things that people kept
• 2:41
  SPEAKER_00: coming back to was this need for analyzing satellite data,
• 2:45
  SPEAKER_00: geospatial data,
• 2:47
  and the lack of expertise in a lot of these non-profits and
• 2:53
  NGOs that we were
• 2:53
  SPEAKER_00: working with in doing this kind of problem solving.
• 2:58
  SPEAKER_00: So the Olmour Earth Project started about a year and a half ago.
• 3:03
  SPEAKER_00: Some of the people who were working on Skylight,
• 3:05
  SPEAKER_00: actually,
• 3:05
  SPEAKER_00: because Skylight also uses Sentinel-1 and Sentinel-2 data,
• 3:11
  some of the researchers who were working on Skylight decided it
• 3:13
  would be a good
• 3:13
  idea to build a more general purpose foundation model for
• 3:18
  geospatial images.
• 3:20
  SPEAKER_00: They started the Old North Project and since then it's grown pretty fast.
• 3:24
  SPEAKER_00: I think we have about 20 or 30 people working on the project as a whole,
• 3:28
  SPEAKER_00: about six people working on the foundation model itself.
• 3:31
  SPEAKER_00: And the idea is basically,
• 3:33
  SPEAKER_00: you know,
• 3:33
  SPEAKER_00: build a really good foundation model,
• 3:35
  SPEAKER_00: just kind of like what everyone's doing,
• 3:36
  SPEAKER_00: and then partner with a lot of other organizations who have
• 3:40
  data,
• 3:40
  who have expertise,
• 3:42
  who have some problem that they're trying to solve in the real
• 3:45
  world.
• 3:45
  SPEAKER_00: So a lot of things like agriculture,
• 3:48
  SPEAKER_00: trying to understand what crops are growing,
• 3:51
  SPEAKER_00: how well they're doing,
• 3:53
  SPEAKER_00: wildfire risk,
• 3:54
  SPEAKER_00: trying to model what forests are in danger,
• 3:58
  SPEAKER_00: where resources could be allocated.
• 4:01
  And then a lot of smaller projects like mapping all of the
• 4:05
  ecosystems of the world or
• 4:08
  mapping where all of the mangroves are and how the mangrove
• 4:12
  forests are doing,
• 4:13
  SPEAKER_00: things like that.
• 4:14
  So foundation model at the core of everything and then on top
• 4:18
  of that all of these
• 4:19
  sort of partners that we have that we're working with to try to
• 4:24
  solve some of the
• 4:25
  SPEAKER_00: problems that they care about.
• 4:27
  SPEAKER_01: Amazing.
• 4:27
  So develop the foundation model and then utilize it in all
• 4:30
  these different
• 4:31
  SPEAKER_01: areas that you're working in.
• 4:33
  SPEAKER_01: You mentioned there's an awful lot happening with foundational models.
• 4:37
  SPEAKER_01: How does Olmo position itself?
• 4:40
  Does it position itself to do something different from what
• 4:42
  others are doing or is
• 4:44
  SPEAKER_01: it taking a different approach?
• 4:46
  SPEAKER_00: Well, yeah, we're trying to be the best, I guess.
• 4:50
  SPEAKER_00: I think one of,
• 4:51
  SPEAKER_00: you know,
• 4:51
  SPEAKER_00: AI2 has a number of strengths,
• 4:53
  but one of the things that has been really nice about AI2 is
• 4:57
  just the resources
• 4:58
  SPEAKER_00: that we have there.
• 5:00
  SPEAKER_00: They've really devoted a lot into this project in particular,
• 5:03
  SPEAKER_00: and this is something that is a challenge,
• 5:06
  SPEAKER_00: obviously,
• 5:06
  SPEAKER_00: in AI research these days.
• 5:07
  SPEAKER_00: You need GPUs to be able to do this kind of foundation model research,
• 5:11
  SPEAKER_00: and we've been fortunate to
• 5:14
  SPEAKER_00: have access to pretty good compute clusters.
• 5:17
  So we've been able to run a lot of experiments and try a lot of
• 5:20
  different things.
• 5:22
  SPEAKER_00: I tend to think of research,
• 5:23
  SPEAKER_00: honestly,
• 5:24
  as like throwing a lot of stuff at the wall and seeing what
• 5:27
  sticks and kind of
• 5:28
  getting lucky sometimes and then putting a lot of that stuff
• 5:31
  together and building
• 5:33
  SPEAKER_00: something really good out of all of these sort of random trials of stuff.
• 5:37
  SPEAKER_00: So in some sense, you know, we've been
• 5:40
  SPEAKER_00: Rob Markman- And I think that's something that isn't super common,
• 5:48
  SPEAKER_00: especially in kind of the academic research world.
• 5:50
  SPEAKER_00: Rob Markman- And especially,
• 5:52
  SPEAKER_00: you know,
• 5:53
  if we think about like what our partner organizations were
• 5:55
  doing before we started
• 5:56
  SPEAKER_00: working with them,
• 5:58
  SPEAKER_00: they definitely didn't have that kind of resource.
• 6:00
  SPEAKER_00: We also just have
• 6:01
  SPEAKER_00: a bunch of really talented people who have a lot of experience doing,
• 6:06
  SPEAKER_00: you know,
• 6:06
  SPEAKER_00: computer vision geospatial research work.
• 6:08
  SPEAKER_00: So we are,
• 6:11
  SPEAKER_00: I would say,
• 6:11
  currently positioned in the sort of free satellite lower
• 6:17
  resolution domain and that
• 6:20
  SPEAKER_00: we mostly work with Sentinel-1,
• 6:21
  SPEAKER_00: Sentinel-2,
• 6:22
  SPEAKER_00: and Landsat data.
• 6:23
  SPEAKER_00: So these are all
• 6:25
  satellites that have basically free data available that you can
• 6:29
  download.
• 6:30
  SPEAKER_00: Anyone can download.
• 6:31
  SPEAKER_00: I think Sentinel-1 and Sentinel-2 are from ESA and Landsat.
• 6:35
  SPEAKER_00: I think Landsat's from NASA.
• 6:37
  SPEAKER_00: You probably know this, right?
• 6:38
  SPEAKER_01: Yeah, exactly.
• 6:41
  um so so we're focused more on the lower resolution satellites
• 6:45
  um that's what we
• 6:46
  sort of pre-trained the foundation model on um and I would say
• 6:50
  other than that
• 6:51
  we're really trying to focus on um any tasks that our partners
• 6:55
  have indicated are
• 6:56
  important to them so uh we do do a lot of work in terms of like
• 7:00
  agricultural type
• 7:01
  SPEAKER_00: stuff a lot of land cover type stuff um
• 7:05
  SPEAKER_00: forest sort of problems, things like that.
• 7:09
  SPEAKER_00: But we really want to build something that's pretty general purpose.
• 7:13
  The other thing I will say is that there has been a lot of
• 7:16
  focus in the community
• 7:17
  SPEAKER_00: on embeddings,
• 7:19
  SPEAKER_00: and we tend to think
• 7:20
  SPEAKER_00: We think of ourselves as pushing in the other direction,
• 7:22
  SPEAKER_00: which is towards training and fine-tuning models for specific problems.
• 7:28
  SPEAKER_00: So instead of focused on getting good results out of embeddings,
• 7:31
  we're focused on building a model that we can bring in our
• 7:34
  partner's data and
• 7:36
  SPEAKER_00: fine-tune for their specific problem,
• 7:38
  SPEAKER_00: because we've just found that embeddings only get you so far,
• 7:42
  SPEAKER_00: and if you can fine-tune the full
• 7:44
  SPEAKER_00: pipeline on a particular task.
• 7:46
  SPEAKER_00: You can get really good results, even if they don't have that much data.
• 7:49
  SPEAKER_00: As long as you have a good pre-trained model,
• 7:52
  you can usually get better results fine-tuning than you can get
• 7:55
  out of the
• 7:56
  SPEAKER_00: embeddings.
• 7:57
  SPEAKER_01: Which isn't to say, you know, we don't want people using embeddings.
• 8:01
  SPEAKER_00: We will produce embeddings for you.
• 8:03
  SPEAKER_00: We have tutorials on how to get embeddings out of all more.
• 8:07
  SPEAKER_00: So
• 8:09
  SPEAKER_00: You know,
• 8:09
  I think embeddings provide a really good tool for fast
• 8:12
  iteration and for quickly
• 8:14
  SPEAKER_00: kind of getting results.
• 8:15
  SPEAKER_00: But if you really want sort of the next level in terms of performance,
• 8:19
  SPEAKER_00: I think fine-tuning is sort of the way to go.
• 8:21
  SPEAKER_01: Thanks for clarifying that.
• 8:24
  SPEAKER_01: And also it's good to understand,
• 8:26
  SPEAKER_01: you know,
• 8:27
  SPEAKER_01: where you think embeddings are position versus fine-tuning.
• 8:32
  I guess one of the areas people might utilize embeddings is in
• 8:36
  traditional
• 8:36
  SPEAKER_01: workflows or when they want to join with other kinds of data sources.
• 8:40
  The other approach I think you guys have taken is to actually
• 8:43
  include those data
• 8:44
  sources in the training so you have maps that are used in the
• 8:48
  pre-training,
• 8:49
  SPEAKER_01: etc.
• 8:49
  SPEAKER_01: Do you mind talking a bit more about that?
• 8:52
  SPEAKER_00: Yeah,
• 8:52
  SPEAKER_00: our pre-training sort of came from,
• 8:55
  SPEAKER_00: again,
• 8:55
  SPEAKER_00: this sort of long iterative process,
• 8:57
  SPEAKER_00: I would say.
• 8:57
  But Gabby is on the team who worked on a couple of different
• 9:05
  foundation models
• 9:06
  SPEAKER_00: before this,
• 9:06
  SPEAKER_00: but Galileo in particular,
• 9:08
  SPEAKER_00: right before Omorath.
• 9:10
  SPEAKER_00: So we started with basically this idea of building on top of Galileo.
• 9:16
  SPEAKER_00: And we tried to re-implement Galileo ourselves and basically failed.
• 9:22
  so what happened was uh we were trying to Galileo is a a latent
• 9:28
  masked image model
• 9:30
  um foundation model which means that you're trying to do
• 9:32
  reconstruction but you're
• 9:34
  trying to do like masked image reconstruction you're trying to
• 9:37
  do it in latent
• 9:38
  space so you convert the image into tokens and then you mask
• 9:41
  out some of the tokens
• 9:42
  SPEAKER_00: and you reconstruct them um
• 9:45
  but the target encoder so the thing that you're trying to
• 9:48
  reproduce is a standard
• 9:52
  like sort of target encoder in a similar way to like how dyno
• 9:55
  v3 or something works
• 9:56
  in that it gets updated during training too so with latent mass
• 10:02
  damage modeling
• 10:03
  latent MIM the target encoder gets updated via basically this
• 10:07
  rolling average of
• 10:08
  SPEAKER_00: your
• 10:10
  SPEAKER_00: encoder for the full training process.
• 10:12
  SPEAKER_00: So you have your encoder, your decoder, and then your target encoder.
• 10:17
  And your target encoder is this rolling average of your encoder
• 10:20
  from the full model.
• 10:23
  And what the rolling average is supposed to do is stabilize the
• 10:27
  model enough that
• 10:30
  SPEAKER_00: you're not trying to target this moving target.
• 10:34
  SPEAKER_00: in this reconstruction process.
• 10:37
  SPEAKER_00: But what we basically found was we couldn't get this to train well.
• 10:42
  SPEAKER_00: The model always basically figured out ways to cheat and kind of diverge.
• 10:47
  SPEAKER_00: And what that means is basically
• 10:51
  we're trying to reconstruct these tokens but the target encoder
• 10:55
  can produce sort of
• 10:56
  trivially easy solutions for the encoder to reconstruct and
• 11:01
  then the problem
• 11:02
  SPEAKER_00: doesn't get interesting it sort of collapses and you don't learn good
• 11:07
  SPEAKER_00: representations you basically just learn to
• 11:12
  produce some sort of basically positional encoding type thing
• 11:15
  or something along those lines.
• 11:18
  SPEAKER_00: There are various ways in which the model can sort of diverge.
• 11:21
  SPEAKER_00: So basically we tried to reproduce Galileo.
• 11:23
  SPEAKER_00: We failed.
• 11:24
  We went back to the drawing board and we were like,
• 11:26
  why is this collapsing all the time?
• 11:28
  We narrowed it down to our target encoder and sort of this
• 11:32
  moving target that we
• 11:33
  SPEAKER_00: were trying to reconstruct.
• 11:35
  SPEAKER_00: And that was too difficult of a process, we decided.
• 11:37
  SPEAKER_00: So we froze the target encoder.
• 11:40
  SPEAKER_00: and we used a very shallow target encoder.
• 11:42
  We actually just take our image and we project it through one
• 11:45
  random projection to
• 11:47
  SPEAKER_00: get it into token space.
• 11:49
  SPEAKER_00: And then we're trying to reconstruct that.
• 11:51
  SPEAKER_00: So we're not reconstructing in pixel space because we found basically
• 11:55
  SPEAKER_00: You can reconstruct in pixel space.
• 11:58
  SPEAKER_00: It tends to produce not quite as good of reconstruction.
• 12:02
  So we're reconstructing the token space and we're using this
• 12:05
  contrastive loss.
• 12:06
  SPEAKER_00: So sort of a class space lost in some sense,
• 12:09
  or you have a soft max over the match between the tokens that
• 12:15
  you're trying to
• 12:15
  SPEAKER_00: match together and the tokens that you want to be different.
• 12:19
  SPEAKER_00: And you're trying to optimize for that.
• 12:21
  but um we found basically that this frozen uh target encoder
• 12:26
  that was just this
• 12:26
  these random projections into token space works uh pretty well
• 12:30
  and it stabilized
• 12:31
  our training a lot um and enabled us to do a lot more sort of
• 12:35
  experimentation
• 12:36
  without worrying about collapsing all the time and this is
• 12:38
  something that gabby
• 12:39
  said too in developing galileo they saw that uh i think he was
• 12:45
  saying like one out
• 12:46
  SPEAKER_00: of every four runs or something would collapse um and
• 12:51
  SPEAKER_00: They had less of an issue with it,
• 12:53
  but it was something that we were seeing so often that we just
• 12:55
  had to kind of deal
• 12:56
  SPEAKER_00: with.
• 12:57
  SPEAKER_00: So then once we,
• 12:58
  SPEAKER_00: and we sort of dubbed this latent MIM light,
• 13:00
  SPEAKER_00: I guess,
• 13:02
  because it's kind of this lighter weight version of latent
• 13:05
  masked image modeling.
• 13:06
  SPEAKER_00: We're still
• 13:08
  doing mass-image modeling in latent space,
• 13:11
  but the target encoder is not quite as interesting.
• 13:14
  And then we have a bunch of other sort of tricks and things
• 13:18
  that we worked out in
• 13:19
  SPEAKER_00: terms of improving performance for the foundation model as well.
• 13:24
  It's really interesting to hear you talk about kind of the
• 13:27
  details of the pre-training there.
• 13:28
  It sounds quite different from just sort of fine-tuning a
• 13:31
  regular model,
• 13:32
  right?
• 13:33
  SPEAKER_01: It sounds probably even more of an art than that is.
• 13:39
  yeah I would say again you know a lot of what we're doing is
• 13:43
  just kind of throwing
• 13:44
  stuff at the wall and seeing what works but also pulling a lot
• 13:46
  from literature
• 13:47
  seeing what people are doing it is you know it always strikes
• 13:50
  me I come to
• 13:53
  satellite data from a computer vision background and I think of
• 13:56
  everything as an
• 13:56
  image and it's amazing how much satellite data is not an image
• 14:02
  at all you know you
• 14:04
  SPEAKER_00: can get so much
• 14:05
  uh so many good results just from thinking about the time
• 14:09
  series aspect and
• 14:10
  ignoring all of the spatial data you can just think you know
• 14:13
  one pixel look at how
• 14:14
  SPEAKER_00: it uh evolves over time and get really good results um that way
• 14:19
  SPEAKER_00: You have all of the multispectral information.
• 14:23
  SPEAKER_00: So it's not just kind of the RGB.
• 14:24
  SPEAKER_00: It's like, how does this spectral signature like change over time?
• 14:29
  SPEAKER_00: And I think in some respects,
• 14:32
  SPEAKER_00: especially with the data that we're dealing with,
• 14:34
  SPEAKER_00: which is low resolution satellite data,
• 14:37
  the spatial component is almost the least important part in a
• 14:40
  lot of these tasks.
• 14:41
  I think the most important component is kind of the spectral
• 14:44
  information and this
• 14:45
  SPEAKER_00: temporal information.
• 14:47
  SPEAKER_00: Coming at it from a vision background,
• 14:49
  a lot of things are different when you start to think about
• 14:51
  satellite data.
• 14:52
  SPEAKER_00: And this is an issue with pre-training.
• 14:55
  SPEAKER_00: You know,
• 14:55
  SPEAKER_00: because we're doing reconstruction of our masked image,
• 15:00
  SPEAKER_00: there's a lot of redundancy in the data.
• 15:03
  SPEAKER_00: If you have Sentinel-1 and Sentinel-2 data,
• 15:07
  SPEAKER_00: a lot of those patches are somewhat similar to each other,
• 15:10
  SPEAKER_00: and it's fairly easy to do
• 15:13
  SPEAKER_00: maybe some of that reconstruction,
• 15:15
  SPEAKER_00: or if you have multi-temporal data,
• 15:16
  SPEAKER_00: if you have one timestamp and then another timestamp a month later,
• 15:20
  that reconstruction process can get a lot easier because you
• 15:22
  can just look backward
• 15:24
  SPEAKER_00: a month,
• 15:24
  SPEAKER_00: see what was going on,
• 15:25
  SPEAKER_00: and then use that as a very strong prior.
• 15:28
  So a lot of the changes and adjustments that we did to our
• 15:34
  pre-training algorithm
• 15:35
  have to deal with the sort of unique challenges presented by
• 15:39
  satellite data as
• 15:40
  SPEAKER_00: opposed to,
• 15:41
  SPEAKER_00: say,
• 15:41
  SPEAKER_00: images themselves.
• 15:43
  um so that means you know handling the fact that we have uh
• 15:46
  these different
• 15:47
  modalities that are very strongly correlated with each other
• 15:50
  handling the different
• 15:51
  time steps that are also very strongly correlated with each
• 15:53
  other and basically
• 15:55
  SPEAKER_00: tweaking the reconstruction task in such a way to make it harder and more
• 16:02
  interesting such that we learn better features instead of just
• 16:05
  like kind of this
• 16:06
  SPEAKER_00: trivial task
• 16:09
  And there's other aspects to using these models which I know
• 16:12
  can be challenging for
• 16:13
  SPEAKER_01: people,
• 16:13
  SPEAKER_01: for instance,
• 16:14
  SPEAKER_01: like the preparation and normalization of the data itself.
• 16:17
  SPEAKER_01: Is that an aspect that you can tell us a bit about?
• 16:21
  SPEAKER_00: Yeah,
• 16:21
  this is something that we run into all the time with other
• 16:25
  people's models,
• 16:25
  SPEAKER_00: basically,
• 16:26
  mostly because we know how our model is supposed to be
• 16:29
  normalized and what data is
• 16:31
  SPEAKER_00: supposed to go into it.
• 16:32
  SPEAKER_00: But as part of writing the Ulmowarth paper,
• 16:36
  we did a very large evaluation task where we basically took a
• 16:40
  bunch of foundation
• 16:41
  SPEAKER_00: models that were in the literature,
• 16:43
  SPEAKER_00: all of the ones that we,
• 16:45
  SPEAKER_00: at the time,
• 16:47
  sort of recognized as being pretty competitive with the state
• 16:49
  of the art or just
• 16:50
  SPEAKER_00: like historical models that people cared about.
• 16:55
  SPEAKER_00: We took them all and ran them all on a standard set of evaluations.
• 16:59
  SPEAKER_00: And as part of this,
• 17:00
  we had to basically take all of these external code bases and
• 17:03
  force them into our
• 17:03
  SPEAKER_00: code base and get them to run on all of these evaluations.
• 17:06
  SPEAKER_00: And everyone uses a different normalization.
• 17:10
  SPEAKER_00: Are you normalizing by one standard deviation?
• 17:12
  SPEAKER_00: Are you normalizing by two standard deviations?
• 17:14
  SPEAKER_00: Are you zero centering or are you
• 17:16
  you know clamping between zero and one everyone does kind of
• 17:21
  different stuff with
• 17:22
  their input data and so we had to try to very carefully go
• 17:27
  through and figure out
• 17:27
  you know what these models were expecting and we we messed some
• 17:31
  of that up and
• 17:32
  we'll be the first people to sort of correct our papers when we
• 17:36
  do get some of that
• 17:37
  SPEAKER_00: stuff wrong I think like
• 17:39
  tomorrow a small archive update is happening because we got
• 17:43
  some of the print fee
• 17:44
  SPEAKER_00: numbers a little bit wrong.
• 17:46
  SPEAKER_00: It doesn't affect the overall story of the paper,
• 17:49
  but it sort of speaks to how complicated it is to get all of
• 17:56
  these models to work.
• 17:57
  SPEAKER_00: And then that's not even to say
• 18:00
  SPEAKER_00: Are you using Sentinel-2 data, you know, raw?
• 18:03
  Are you using it with,
• 18:04
  you know,
• 18:05
  what level of post-processing are you using it at?
• 18:09
  SPEAKER_00: There's all of these kind of,
• 18:11
  SPEAKER_00: and then everyone uses different normalization statistics too,
• 18:14
  SPEAKER_00: right?
• 18:14
  SPEAKER_00: So it's like,
• 18:16
  SPEAKER_00: if you calculate statistics during pre-training,
• 18:19
  SPEAKER_00: you have one set of statistics,
• 18:20
  SPEAKER_00: but then if you run it on
• 18:22
  SPEAKER_00: a dataset and that dataset is weird.
• 18:25
  SPEAKER_00: So we saw this a lot with MADOS,
• 18:27
  SPEAKER_00: the evaluation task,
• 18:28
  SPEAKER_00: which I think is like trash in the ocean,
• 18:31
  SPEAKER_00: basically.
• 18:32
  SPEAKER_00: But all of the input tiles of MADOS are ocean tiles.
• 18:36
  SPEAKER_00: And those have very different normalization statistics.
• 18:39
  So if you use the statistics from pre-training,
• 18:42
  you might get much worse results on MADOS.
• 18:44
  SPEAKER_00: But if you actually look at the statistics from just the MADOS dataset,
• 18:48
  you can re-normalize the data using the dataset statistics and
• 18:51
  then you get
• 18:52
  SPEAKER_00: different results.
• 18:55
  Our solution to this was we just tried to sweep as much stuff
• 18:58
  as possible.
• 18:59
  So everything that we could think of to try different versions
• 19:03
  of,
• 19:03
  we would sweep.
• 19:05
  SPEAKER_00: So for example, normalization statistics,
• 19:08
  um how we were doing pooling across tokens all of this kind of
• 19:11
  stuff um we tried to
• 19:13
  make sure that it didn't influence uh the results that these
• 19:16
  models were getting
• 19:18
  but of course if anyone out there uh sees their model in our
• 19:21
  paper and sees that
• 19:22
  it's getting worse results than they would have expected and
• 19:24
  and knows why um we we
• 19:26
  would love to get emails uh and we can make corrections to it
• 19:30
  but we tried to we
• 19:31
  SPEAKER_00: tried to have as as fair of a comparison as possible at least
• 19:35
  That's a really interesting nuance that you have to reprocess
• 19:39
  the data to better
• 19:40
  match the sort of normalization statistics that your model was
• 19:43
  pre-trained on.
• 19:44
  And I guess Donna is maybe one example of some of the
• 19:47
  challenges people might face
• 19:50
  SPEAKER_01: when using these models.
• 19:52
  SPEAKER_01: Another point you brought up was about evaluation.
• 19:55
  Do you think that we're approaching some sort of
• 19:57
  standardization on how we evaluate
• 19:59
  SPEAKER_01: foundational models?
• 20:03
  SPEAKER_00: It would be really nice if we were.
• 20:07
  SPEAKER_00: You know, I think this is like an XKCD comic, right?
• 20:11
  It's like,
• 20:12
  oh,
• 20:12
  there are 14 different standards out there and no one can
• 20:15
  agree.
• 20:15
  So I'm going to make,
• 20:16
  you know,
• 20:17
  the new one and then there are 15 different standards.
• 20:20
  SPEAKER_00: I think
• 20:23
  we are doing the best that we can and you know one of the
• 20:27
  reasons we ran all of
• 20:29
  these evaluations ourselves was because there isn't a standard
• 20:33
  set of evaluations
• 20:34
  that everyone sort of agrees on as like the right evaluations
• 20:39
  it also like it's
• 20:41
  SPEAKER_00: sort of silly to think of this as one monolithic
• 20:49
  SPEAKER_00: domain in that we only care about low-resolution satellite stuff.
• 20:55
  And a lot of big evaluation frameworks will include aerial
• 21:00
  imagery or high-res imagery.
• 21:02
  And that's not something that we train on and also not
• 21:05
  something that our partners
• 21:06
  SPEAKER_00: really care about that much for the tasks that we're doing.
• 21:09
  SPEAKER_00: But some other model, they might really care about high-resolution data.
• 21:13
  SPEAKER_00: They might not care about Sentinel-1 and Sentinel-2 at all.
• 21:15
  SPEAKER_00: So...
• 21:18
  depending on the domain there's going to be different
• 21:19
  evaluations that you care
• 21:20
  about we tried to basically span as much breadth as we could in
• 21:25
  terms of the
• 21:27
  different sorts of tasks and stuff in the evaluations that we
• 21:29
  were running but yeah
• 21:31
  SPEAKER_00: it is it's a challenging
• 21:34
  SPEAKER_00: thing.
• 21:34
  And I think basically the community is starting to slowly
• 21:38
  recognize that we
• 21:39
  SPEAKER_00: probably need something.
• 21:42
  SPEAKER_00: But I don't think, you know, like in computer vision, it was easy.
• 21:46
  SPEAKER_00: It was like,
• 21:47
  SPEAKER_00: all right,
• 21:47
  SPEAKER_00: ImageNet for classification and maybe like,
• 21:51
  SPEAKER_00: well,
• 21:52
  SPEAKER_00: Pascal VOC,
• 21:52
  SPEAKER_00: but it got too easy.
• 21:54
  SPEAKER_00: But then we had Coco and Coco was really good.
• 21:56
  SPEAKER_00: So Coco for detection and
• 21:59
  SPEAKER_00: and later on, you know, those sort of saturated.
• 22:01
  So maybe we got some other things,
• 22:03
  but there's no,
• 22:04
  you know,
• 22:05
  ImageNet for geospatial models yet.
• 22:10
  SPEAKER_00: There's, you know, GeoBench, GeoBench2.
• 22:13
  These are,
• 22:14
  I think,
• 22:15
  pieces of the puzzle maybe,
• 22:17
  but I don't think they are comprehensive.
• 22:21
  SPEAKER_00: And I don't think any evaluations that can really be comprehensive
• 22:27
  SPEAKER_00: Mm-hmm.
• 22:36
  SPEAKER_00: But yeah, it's a very hard problem.
• 22:38
  SPEAKER_00: We've been thinking about this.
• 22:39
  SPEAKER_00: We've been thinking about,
• 22:40
  can we release some sort of evaluation benchmark that would
• 22:46
  incorporate a bunch of
• 22:48
  SPEAKER_00: tasks from our partners because we have sort of land cover,
• 22:52
  SPEAKER_00: change detection,
• 22:54
  SPEAKER_00: single pixel tasks,
• 22:56
  SPEAKER_00: segmentation task detection.
• 22:58
  We have a bunch of different sort of stuff from different
• 22:59
  domains that I think
• 23:01
  SPEAKER_00: would be a useful resource.
• 23:03
  SPEAKER_00: I don't know if anything will ever be the one monolithic
• 23:06
  SPEAKER_00: evaluation.
• 23:07
  SPEAKER_00: But I think having a lot of different options out there is good.
• 23:12
  Yeah,
• 23:13
  let's hope that we can kind of stimulate that conversation and
• 23:16
  help move things forward.
• 23:18
  SPEAKER_01: As you say, there's a lot of kind of complexity to it and
• 23:22
  What are you even trying to compare a lot of the time,
• 23:24
  you know,
• 23:25
  needs to be clarified.
• 23:27
  SPEAKER_01: Another topic that's come up in,
• 23:29
  SPEAKER_01: I guess,
• 23:30
  the conversation recently is this comparison of foundational
• 23:33
  models to kind of
• 23:34
  SPEAKER_01: smaller and traditional models.
• 23:36
  And there's been a bit of kind of dialogue stating that
• 23:40
  actually foundational
• 23:41
  SPEAKER_01: models are not always much better.
• 23:43
  And they give examples of training small models and beta sets
• 23:47
  and the small model
• 23:48
  SPEAKER_01: achieves better performance.
• 23:50
  SPEAKER_01: Could you maybe talk a bit about that topic?
• 23:52
  yeah I guess this was uh something I don't know if you were at
• 23:55
  the were you at the
• 23:55
  iClear workshop on geospatial models um this was a topic that
• 24:00
  came up during the
• 24:01
  panel discussion there um and there were some there were some
• 24:07
  strong opinions at
• 24:09
  that workshop I would say uh I I think um you know we are so so
• 24:15
  for one thing we
• 24:16
  SPEAKER_00: are a very um
• 24:21
  SPEAKER_00: evaluation-focused field in some sense.
• 24:24
  So if one thing works better than another thing,
• 24:27
  just use that thing,
• 24:28
  I guess.
• 24:29
  SPEAKER_00: We're not saying everyone should use our foundation.
• 24:32
  SPEAKER_00: Okay, everyone should use Olmaworth for everything.
• 24:34
  SPEAKER_00: We're not saying everyone should use foundation models for everything.
• 24:36
  If you have a small model that works well,
• 24:39
  by all means,
• 24:40
  you should use it.
• 24:42
  SPEAKER_00: I think
• 24:44
  You should just try things and test out how well they work and
• 24:49
  use the best one for
• 24:50
  SPEAKER_00: your particular use case.
• 24:53
  SPEAKER_00: We have found that for our partners, we've done a lot of this comparison.
• 25:00
  Our foundation model works in general better than the sort of
• 25:05
  bespoke solutions
• 25:06
  SPEAKER_00: that they were using beforehand,
• 25:07
  SPEAKER_00: which are usually something like random forests or
• 25:10
  grading-boosted decision trees or something on top of some sort
• 25:13
  of extracted
• 25:13
  SPEAKER_00: features,
• 25:15
  SPEAKER_00: which are very powerful models,
• 25:16
  SPEAKER_00: but we can basically get a lot of benefit out of pre-training.
• 25:22
  SPEAKER_00: We have these strong features that we've pre-trained,
• 25:26
  and then the benefit of foundation models is you don't need as
• 25:31
  much labeled data
• 25:34
  SPEAKER_00: during fine-tuning.
• 25:35
  SPEAKER_00: I think if you have a ton of labeled data,
• 25:38
  SPEAKER_00: then you could train any particular model.
• 25:41
  It wouldn't really matter if it was a big model or a small
• 25:43
  model or whatever.
• 25:44
  SPEAKER_00: And in general, you know, most people don't have a lot of labeled data.
• 25:48
  SPEAKER_00: It's very time intensive to label data for these kinds of tasks,
• 25:53
  especially like some of the agriculture tasks people are like
• 25:55
  going on the ground
• 25:56
  looking at what crops are growing tagging them you know
• 26:00
  geolocating it like it's it
• 26:01
  can be uh require sort of real world um interventions to do
• 26:06
  some of this data
• 26:07
  SPEAKER_00: labeling so uh
• 26:11
  yeah if if you find that for your task a small model works
• 26:14
  better than a than a
• 26:15
  large foundation model use the small model and we have a lot of
• 26:19
  people where the
• 26:19
  foundation models work better so we're going to keep helping
• 26:22
  them use it but um
• 26:24
  we're not you know we're not trying to say that everyone should
• 26:29
  use this one thing
• 26:30
  for every single thing and there are other considerations too
• 26:33
  uh like Skylight I
• 26:34
  SPEAKER_00: think still uses
• 26:36
  a convolutional network-based approach because they need a
• 26:40
  little bit more
• 26:41
  SPEAKER_00: efficiency than our current models can kind of produce.
• 26:45
  SPEAKER_00: So there are definitely like efficiency trade-offs for some things.
• 26:50
  SPEAKER_00: There are a lot of different trade-offs to sort of consider for sure.
• 26:55
  SPEAKER_01: Yeah.
• 26:56
  SPEAKER_01: Thanks for clarifying your thoughts on that.
• 26:57
  I think it's probably needs to be included in the evaluations
• 27:02
  in some way,
• 27:02
  SPEAKER_01: but I guess that's part of it ongoing.
• 27:05
  SPEAKER_01: conversation there.
• 27:06
  I just want to sort of circle back to Alma and talk a bit about
• 27:10
  the sort of
• 27:11
  SPEAKER_01: usability and the sort of practical and ergonomics of it.
• 27:14
  SPEAKER_01: You mentioned that many of the people,
• 27:15
  SPEAKER_01: they've kind of got existing approaches,
• 27:17
  SPEAKER_01: maybe using kind of traditional models.
• 27:20
  SPEAKER_01: How are you sort of onboarding them to a foundational model?
• 27:23
  SPEAKER_01: What's your approach there?
• 27:25
  SPEAKER_00: Yeah,
• 27:25
  SPEAKER_00: well,
• 27:26
  SPEAKER_00: this is,
• 27:26
  SPEAKER_00: again,
• 27:26
  one of the great things about working at AA2 is just they're
• 27:31
  willing to dump a lot
• 27:31
  SPEAKER_00: of resources into building out infrastructure around
• 27:37
  SPEAKER_00: this kind of project as a whole,
• 27:39
  SPEAKER_00: which means we're not just building the foundation model,
• 27:41
  but the Ulmorth platform is this whole platform for using these
• 27:46
  models as well.
• 27:47
  SPEAKER_00: So it includes things like labeling data,
• 27:52
  SPEAKER_00: going in and actually like,
• 27:53
  SPEAKER_00: you know,
• 27:54
  either doing segmentation or detection or point labels or
• 27:57
  whatever you want to do,
• 27:58
  SPEAKER_00: finding a bunch of like tiles and look,
• 28:01
  SPEAKER_00: you can sort of use the interface to scroll through them over time,
• 28:04
  SPEAKER_00: see how things are changing.
• 28:05
  SPEAKER_00: So
• 28:06
  so you can label data in the platform and then you can or you
• 28:09
  can just import data
• 28:10
  SPEAKER_00: if you already have it a lot of our partners already have data
• 28:14
  you can validate existing data there's you know whole workflows
• 28:18
  for like assigning
• 28:19
  it to people and making sure that everything makes sense and
• 28:23
  then actually taking
• 28:24
  the foundation models that we've pre-trained and then
• 28:26
  fine-tuning it on data that's
• 28:27
  in the platform and these are mostly with like little wizards
• 28:33
  basically where
• 28:33
  you're selecting like what kind of task do you have what does
• 28:36
  your data look like
• 28:38
  things like that so trying to make it as straightforward as
• 28:41
  possible to get people
• 28:43
  SPEAKER_00: started
• 28:44
  all within basically this web platform where people can bring
• 28:47
  their data or label
• 28:49
  SPEAKER_00: data,
• 28:51
  SPEAKER_00: pre-trained or fine-tuned models,
• 28:54
  SPEAKER_00: and then also run the models to perform inference as well.
• 28:56
  So they can basically select areas to run inference and time
• 29:03
  ranges to run
• 29:03
  SPEAKER_00: inference over,
• 29:05
  SPEAKER_00: and it'll take their fine-tuned model and run it on
• 29:09
  uh these these windows and generate geotiffs as results
• 29:13
  basically you can view the
• 29:15
  results or you can a lot of people just want to download the
• 29:17
  results and use them
• 29:18
  in their own workflows um so we're trying to build out
• 29:22
  basically this platform uh
• 29:24
  where people can can do their full you know sort of workflow of
• 29:30
  geospatial stuff or
• 29:31
  at least part of their workflow again a lot of people uh you
• 29:35
  know that you generate
• 29:36
  SPEAKER_00: a map but that's not the end right
• 29:38
  SPEAKER_00: that's kind of where we stop.
• 29:41
  SPEAKER_00: But then people have to take that map and do something with it.
• 29:44
  So trying to make that available so people can download it and
• 29:48
  use it for whatever
• 29:51
  SPEAKER_00: kind of the rest of their workflow is.
• 29:53
  So we're trying to do that as one piece of the puzzle in terms
• 29:57
  of like connecting
• 29:59
  SPEAKER_00: with people who don't have any experience with machine learning,
• 30:01
  SPEAKER_00: who just want to like input their data and get results out of it.
• 30:05
  SPEAKER_00: And then on the other side, we have everything
• 30:08
  SPEAKER_00: is, you know, on GitHub, you can look through all of the source code.
• 30:11
  A lot of it is open source in terms of sort of freely usable
• 30:15
  available so people can
• 30:19
  SPEAKER_00: download the modeling code,
• 30:21
  SPEAKER_00: they can run the models themselves,
• 30:22
  SPEAKER_00: they can fine-tune the models themselves,
• 30:24
  and we try to provide a lot of tutorials and resources for
• 30:29
  people who want to dive
• 30:30
  a little bit deeper into the code base and actually use the
• 30:33
  foundation models and
• 30:34
  SPEAKER_00: kind of the native PyTorch implementations.
• 30:38
  SPEAKER_01: Right,
• 30:38
  SPEAKER_01: so full spectrum there,
• 30:40
  SPEAKER_01: you know,
• 30:41
  web user interface or people that don't want to code all the
• 30:44
  way through to,
• 30:45
  SPEAKER_01: here's the code and the weights and
• 30:47
  SPEAKER_01: You know, off you go and modify, et cetera.
• 30:50
  SPEAKER_00: Very ambitious.
• 30:53
  We've limited somewhat the number of people who are using the
• 30:56
  platform currently
• 30:57
  SPEAKER_00: and that we only have bandwidth to onboard a few organizations at a time,
• 31:03
  SPEAKER_00: basically.
• 31:05
  SPEAKER_00: working up more and more organizations, getting them in the system.
• 31:07
  SPEAKER_00: But if you signed up today,
• 31:10
  it might be a little while until you heard back from someone in
• 31:13
  terms of getting
• 31:14
  SPEAKER_00: access to the platform.
• 31:16
  SPEAKER_00: But with a lot of partners,
• 31:17
  SPEAKER_00: we're working one-on-one with them to get them into the platform,
• 31:20
  SPEAKER_00: to get their data in,
• 31:21
  SPEAKER_00: et cetera,
• 31:22
  SPEAKER_00: and make sure they have a good experience.
• 31:24
  So we're really trying to have a really good experience for the
• 31:27
  partners that are
• 31:28
  SPEAKER_00: working with us and slowly expanding that over time.
• 31:34
  SPEAKER_01: Yeah,
• 31:34
  SPEAKER_01: so there's a couple of other platforms,
• 31:36
  SPEAKER_01: obviously,
• 31:36
  SPEAKER_01: Google Earth Engine and Bansure Computer.
• 31:38
  SPEAKER_01: It sounds like this is going to be something a bit different now.
• 31:43
  uh yeah i would say i mean or obviously like google earth
• 31:47
  engine is this kind of
• 31:48
  behemoth um and they have a lot of uh resources available i
• 31:53
  think they are going to
• 31:54
  add in fine-tuning for their foundation model at some point
• 31:57
  maybe but uh it seems
• 31:58
  like you maybe will need to pay for it um you don't have to pay
• 32:01
  for it if you use
• 32:02
  our platform uh and you can download the model uh it's not some
• 32:06
  black box sitting
• 32:07
  SPEAKER_00: somewhere uh in the cloud so uh
• 32:12
  SPEAKER_00: And,
• 32:13
  SPEAKER_00: you know,
• 32:13
  SPEAKER_00: again,
• 32:13
  we're being sort of picky in terms of the organizations we're
• 32:17
  working with.
• 32:18
  We're trying to find organizations that we generally support
• 32:20
  the mission of,
• 32:21
  SPEAKER_00: but that means that we are providing all of the compute for free,
• 32:24
  SPEAKER_00: the platform is free,
• 32:27
  SPEAKER_00: et cetera.
• 32:27
  SPEAKER_00: So really trying to support the missions of these nonprofits,
• 32:32
  SPEAKER_00: NGOs,
• 32:33
  SPEAKER_00: some governments,
• 32:33
  I think like the Kenyan government we're working with to do
• 32:36
  crop modeling,
• 32:36
  SPEAKER_00: mapping kind of stuff.
• 32:38
  SPEAKER_00: So, yeah.
• 32:40
  So it's pretty much kind of a review process to get access to
• 32:43
  the platform at the
• 32:45
  SPEAKER_01: moment and the scope might evolve over time.
• 32:49
  SPEAKER_00: Yeah.
• 32:49
  SPEAKER_01: Fantastic.
• 32:50
  SPEAKER_01: So we can direct people to the Ulmowarth platform website.
• 32:54
  SPEAKER_01: Is that what it's called?
• 32:56
  SPEAKER_00: Yep, Ulmowarth platform.
• 32:57
  SPEAKER_00: If you just, I think it's alanai.org slash ulmowarth will get you there.
• 33:03
  SPEAKER_00: Or just search Ulmowarth, you know, with pretty good SEO, so...
• 33:08
  SPEAKER_01: Fantastic.
• 33:08
  SPEAKER_00: Sort of a strange name.
• 33:10
  SPEAKER_01: Yeah, I'll put the link in the show notes.
• 33:13
  Just to finish up,
• 33:13
  I know you had a .1 release,
• 33:15
  and I think you're working on a .2 release.
• 33:17
  SPEAKER_01: Do you mind just talking about what's been happening very recently?
• 33:20
  SPEAKER_00: Yeah, we had 1.1 come out, I think, last week, maybe two weeks ago.
• 33:25
  SPEAKER_00: And then, yeah, we have version 1.2 happening.
• 33:28
  SPEAKER_00: 1.1, we just got significantly faster.
• 33:31
  We're using,
• 33:32
  like,
• 33:32
  30% of the tokens that we were originally using,
• 33:35
  so the model got...
• 33:37
  got a lot faster in terms of pre-training and fine-tuning and
• 33:41
  inference time.
• 33:43
  SPEAKER_00: And version 1.2,
• 33:44
  we had a lot of people coming to us because a lot of people
• 33:46
  still want to use
• 33:47
  SPEAKER_00: embeddings.
• 33:48
  A lot of people came to us and said our embeddings looked weird
• 33:51
  because they did
• 33:52
  look weird because we were using sort of standard 2D syncos
• 33:56
  positional embeddings.
• 33:58
  for positional embedding in the image and when you run like PCA
• 34:05
  on the tokens that
• 34:06
  we were producing a really clear signal was this positional
• 34:09
  embedding that we were
• 34:10
  adding in so it produced these really stark artifacts in our
• 34:15
  embeddings and even
• 34:16
  though the embeddings worked well people don't trust something
• 34:19
  that has that
• 34:21
  SPEAKER_00: So we're using rotary positional encodings now,
• 34:27
  SPEAKER_00: rope,
• 34:28
  and that basically got rid of this really strong artifact that
• 34:35
  we were seeing.
• 34:36
  SPEAKER_00: We get better results, too, on many evaluations.
• 34:40
  SPEAKER_00: Again, it's tough, right?
• 34:42
  Like,
• 34:43
  you go up on one evaluation and then you go down a little bit
• 34:45
  on another evaluation.
• 34:46
  SPEAKER_00: It's like,
• 34:47
  SPEAKER_00: well,
• 34:47
  SPEAKER_00: how much do I weight Pastiz versus Eurosat versus Matos versus,
• 34:52
  SPEAKER_00: you know,
• 34:53
  SPEAKER_00: there's just SO2sat.
• 34:54
  SPEAKER_00: It's like so many different evaluations.
• 34:57
  That was one of the beautiful things about computer vision is
• 35:00
  if you got better on
• 35:01
  SPEAKER_00: ImageNet,
• 35:02
  SPEAKER_00: you got better across the board on basically everything.
• 35:05
  SPEAKER_00: And in geospatial stuff, that's not the case.
• 35:08
  If you get better on one task,
• 35:10
  it often means you're getting worse on other tasks.
• 35:13
  SPEAKER_00: We got better on a lot of stuff by adding rope in.
• 35:15
  SPEAKER_00: And that's version 1.2.
• 35:18
  SPEAKER_00: That will be out shortly, I think.
• 35:21
  SPEAKER_00: Probably,
• 35:22
  SPEAKER_00: I don't know how long it takes to upload the episodes to YouTube,
• 35:25
  SPEAKER_00: but I would say within the week,
• 35:27
  SPEAKER_00: we will have 1.2 on our album.
• 35:29
  SPEAKER_01: All right, exciting.
• 35:30
  So people will see,
• 35:32
  I guess,
• 35:32
  is there a paper there or is that just like a kind of code?
• 35:35
  SPEAKER_01: It'll be a tech report, yeah.
• 35:37
  SPEAKER_00: We'll upload a tech report,
• 35:38
  but it's just like we added rope and here are the experiments
• 35:42
  that we did.
• 35:42
  It's not a very impactful tech report,
• 35:45
  I would say,
• 35:45
  but interesting maybe.
• 35:48
  SPEAKER_01: Well, thank you for joining me today.
• 35:49
  SPEAKER_01: It's been a really interesting conversation.
• 35:51
  SPEAKER_01: I'll drop the links to the platform in the show notes.
• 35:53
  SPEAKER_01: And in terms of following updates from the team,
• 35:55
  SPEAKER_01: I guess LinkedIn or your own blog will be the best places.
• 36:00
  SPEAKER_00: Yeah,
• 36:00
  SPEAKER_00: we actually were just talking about how we don't have a LinkedIn,
• 36:03
  SPEAKER_00: but we maybe are going to make one.
• 36:04
  SPEAKER_01: Okay.
• 36:06
  SPEAKER_00: Yeah.
• 36:09
  SPEAKER_00: You got me on the spot.
• 36:10
  SPEAKER_00: We'll probably have an Old North LinkedIn at some point.
• 36:14
  SPEAKER_00: The Old North website is a good place for updates.
• 36:18
  SPEAKER_00: And if you're interested in the platform,
• 36:20
  SPEAKER_00: there's an email to send an email to on that platform website.
• 36:24
  SPEAKER_00: So yeah,
• 36:26
  SPEAKER_00: a few options,
• 36:26
  but stay tuned for hopefully some sort of more permanent blog
• 36:31
  or LinkedIn as well.
• 36:33
  SPEAKER_01: Fantastic.
• 36:33
  SPEAKER_01: All right.
• 36:33
  SPEAKER_01: Well, looking forward to all these updates.
• 36:35
  SPEAKER_01: Once again, thanks for joining me and see you later.
• 36:37
  SPEAKER_00: Yeah, thanks.