Latest / Satellite image deep learning / IceCloudNet and the PhD Journey

Transcript

IceCloudNet and the PhD Journey · Satellite image deep learning

• 0:05
  Hello,
• 0:05
  I'm Robin Cole,
• 0:06
  and welcome to the Satellite Image Deep Learning Podcast.
• 0:09
  SPEAKER_00: In this episode,
• 0:10
  I caught up with Kai Jegel to discuss his experience pursuing a
• 0:13
  PhD at the
• 0:14
  SPEAKER_00: intersection of machine learning and remote sensing.
• 0:17
  SPEAKER_00: We discussed his work on ICE CloudNet,
• 0:19
  a deep learning model that reconstructs 3D cloud structures
• 0:22
  from 2D imagery with
• 0:24
  SPEAKER_00: sparse depth measurements.
• 0:26
  SPEAKER_00: Data fusion and sparse machine learning are fascinating topics.
• 0:29
  SPEAKER_00: I learned a lot from this conversation, and I hope you do too.
• 0:32
  SPEAKER_00: If you're watching this on YouTube,
• 0:34
  please consider liking the video and subscribing to the channel
• 0:37
  as this really
• 0:37
  SPEAKER_00: helps to reach a wider audience.
• 0:39
  SPEAKER_00: Thank you.
• 0:40
  SPEAKER_00: Hi Kai, welcome to the podcast.
• 0:42
  SPEAKER_00: Hi Rob, thanks for having me today.
• 0:44
  SPEAKER_00: Looking forward to this conversation.
• 0:46
  SPEAKER_00: So today,
• 0:47
  SPEAKER_00: two-parter,
• 0:48
  so we're going to begin by talking about your career history
• 0:50
  and then move on to
• 0:51
  SPEAKER_00: your research.
• 0:53
  SPEAKER_00: Over to you for the history.
• 0:55
  SPEAKER_00: Thanks.
• 0:55
  SPEAKER_01: Yeah, so I give a bit of intro about my background.
• 1:01
  SPEAKER_01: So I've been in the kind of ML space for some years.
• 1:07
  SPEAKER_01: I worked,
• 1:08
  SPEAKER_01: like after my studies,
• 1:09
  SPEAKER_01: I worked in an ML ops startup as a software engineer,
• 1:13
  SPEAKER_01: but then was interested or motivated to apply the
• 1:18
  machine learning skills and techniques to more pressing questions,
• 1:24
  let's say,
• 1:24
  of our time.
• 1:25
  SPEAKER_01: So I was looking specifically to work on climate issues,
• 1:29
  SPEAKER_01: to tackle climate change with machine learning.
• 1:32
  SPEAKER_01: And then by a few coincidences,
• 1:34
  SPEAKER_01: I ended up in the Atmospheric Physics Lab at ETH in Zurich,
• 1:40
  SPEAKER_01: a group that works on,
• 1:44
  SPEAKER_01: tries to understand cloud processes
• 1:48
  SPEAKER_01: and climate modelling, etc.
• 1:50
  And clouds in general are one of the big uncertainties that are
• 1:53
  remaining in
• 1:54
  SPEAKER_01: climate modelling and also future climate projections.
• 1:57
  SPEAKER_01: So it's a topic that everyone is aware of,
• 2:00
  SPEAKER_01: but in the community,
• 2:02
  SPEAKER_01: it's quite a vibrant area of research and quite important as well.
• 2:08
  SPEAKER_00: So was that direction,
• 2:10
  was it based on your previous studies or just something totally
• 2:13
  new you wanted to
• 2:14
  SPEAKER_00: go into?
• 2:16
  SPEAKER_01: It's really completely new.
• 2:18
  SPEAKER_01: Before starting the PhD,
• 2:19
  I didn't even know that this kind of cloud topic was something
• 2:25
  that was actually important,
• 2:27
  SPEAKER_01: or I didn't know anything about microphysics and so on.
• 2:32
  I was mainly looking for PhD positions in this intersection of
• 2:38
  machine learning and
• 2:41
  SPEAKER_01: Yeah, and climate research.
• 2:43
  SPEAKER_01: And then I saw this call for applications and then I applied to it.
• 2:51
  SPEAKER_01: And it was also a pretty steep learning curve,
• 2:55
  SPEAKER_01: I have to admit,
• 2:56
  SPEAKER_01: like teaching atmospheric physics classes and so on as a TA,
• 3:03
  SPEAKER_01: knowledge was quite intense.
• 3:05
  SPEAKER_00: Okay.
• 3:06
  SPEAKER_00: So teaching was a significant part of the PhD program.
• 3:10
  SPEAKER_01: Yeah.
• 3:11
  SPEAKER_01: Yeah.
• 3:12
  SPEAKER_01: I mean, the first years for sure took up quite some time.
• 3:18
  Significant,
• 3:18
  I would say like 20 to maybe 30% of the time in the beginning.
• 3:22
  SPEAKER_01: And then in the later years, it was a bit less intense.
• 3:25
  SPEAKER_00: Yeah.
• 3:26
  And did you find your experience having worked in industry for
• 3:29
  some time as a
• 3:30
  SPEAKER_00: software engineer,
• 3:31
  SPEAKER_00: was that applicable?
• 3:32
  SPEAKER_00: Was it something that assisted you?
• 3:34
  SPEAKER_01: Yeah,
• 3:35
  SPEAKER_01: I mean,
• 3:35
  SPEAKER_01: of course,
• 3:35
  SPEAKER_01: there's a lot of coding involved and a lot of Python,
• 3:40
  SPEAKER_01: of course,
• 3:41
  SPEAKER_01: in data science machine learning.
• 3:44
  SPEAKER_01: So definitely these skills helped me and were transferable.
• 3:49
  SPEAKER_01: Being in a group of physicists,
• 3:51
  I think I could also help my colleagues sometimes with kind of
• 3:56
  my more technical skills.
• 3:58
  SPEAKER_01: So I think it was also a good symbiosis.
• 4:01
  SPEAKER_01: And yeah,
• 4:01
  I think it's just nice also to change a bit environments and
• 4:06
  around yourself with
• 4:07
  SPEAKER_01: people with different backgrounds.
• 4:08
  SPEAKER_00: Yeah.
• 4:09
  SPEAKER_00: And obviously when you do a PhD,
• 4:10
  SPEAKER_00: there's a lot of responsibility on you to set the directions,
• 4:13
  SPEAKER_00: set the objectives.
• 4:15
  SPEAKER_00: How did you find that having, you know, worked in industry?
• 4:19
  SPEAKER_01: Yeah.
• 4:20
  SPEAKER_01: I mean,
• 4:20
  SPEAKER_01: since I was a bit out,
• 4:22
  SPEAKER_01: like,
• 4:23
  SPEAKER_01: let's say,
• 4:24
  SPEAKER_01: like a new joiner to the field of atmospheric physics,
• 4:28
  SPEAKER_01: it took me some time to also understand,
• 4:30
  SPEAKER_01: like,
• 4:30
  SPEAKER_01: what is kind of the current state of the research,
• 4:33
  SPEAKER_01: what are actually interesting problems to work on?
• 4:37
  SPEAKER_01: I think I always came with the glasses,
• 4:38
  SPEAKER_01: okay,
• 4:39
  where can we use machine learning and data science to improve
• 4:45
  the status quo and
• 4:47
  just come a bit of a different angle than maybe other people in
• 4:52
  the group.
• 4:53
  SPEAKER_01: I think that helped a lot,
• 4:55
  just looking for inefficiencies and where can we automate
• 4:58
  things and so on.
• 4:59
  SPEAKER_00: Right.
• 4:59
  SPEAKER_00: Well, that's a great place to start talking about what you did then.
• 5:03
  SPEAKER_01: Yeah.
• 5:04
  Yeah,
• 5:05
  so I would like to talk about one project that I did towards
• 5:10
  the end of my PhD.
• 5:11
  SPEAKER_01: It's called Ice Cloud Net.
• 5:14
  SPEAKER_01: And as the name suggests, it focuses on clouds that consist of only ice.
• 5:21
  SPEAKER_01: So these are cirrus clouds, we call them in the jargon.
• 5:26
  SPEAKER_01: And they exist very high up in the atmosphere.
• 5:29
  SPEAKER_01: You all have seen them probably once.
• 5:33
  these are these feathery thin clouds up high up in the sky and
• 5:37
  those are yeah i
• 5:39
  would say specifically not so well understood yet so there are
• 5:44
  many microphysical
• 5:45
  processes and so on that are acting there and also one reason
• 5:52
  why it's hard to
• 5:52
  SPEAKER_01: study them is that there's not a good observational basis so it's uh um
• 6:00
  SPEAKER_01: On the one hand, it's difficult to get actual samples from there.
• 6:03
  So there are some campaigns that fly with airplanes into these
• 6:06
  clouds and measure
• 6:08
  SPEAKER_01: kind of the properties there.
• 6:10
  SPEAKER_01: But it's quite expensive and quite difficult.
• 6:13
  And also,
• 6:13
  yeah,
• 6:15
  it's much more demanding than for lower clouds,
• 6:18
  let's say.
• 6:20
  SPEAKER_01: And on the other hand,
• 6:21
  SPEAKER_01: there exist satellite data sets,
• 6:24
  SPEAKER_01: which I guess the audience is more interested in.
• 6:29
  SPEAKER_01: And,
• 6:31
  SPEAKER_01: There,
• 6:32
  SPEAKER_01: also,
• 6:32
  SPEAKER_01: we have the issue that on the one hand side,
• 6:35
  SPEAKER_01: we have passive satellites,
• 6:38
  SPEAKER_01: for example,
• 6:38
  SPEAKER_01: like Siviri,
• 6:39
  SPEAKER_01: which is a geostationary satellite,
• 6:41
  SPEAKER_01: or MODIS,
• 6:43
  SPEAKER_01: that just take 2D images,
• 6:45
  SPEAKER_01: let's say,
• 6:46
  SPEAKER_01: in different wavelengths,
• 6:47
  SPEAKER_01: like infrared,
• 6:48
  SPEAKER_01: but also visual.
• 6:50
  SPEAKER_01: And you get nice overview of the cloud dynamics,
• 6:54
  SPEAKER_01: because you get in,
• 6:55
  SPEAKER_01: for Siviri,
• 6:56
  SPEAKER_01: for example,
• 6:56
  SPEAKER_01: every 15 minutes,
• 6:58
  SPEAKER_01: you have an observation.
• 7:01
  SPEAKER_01: But the big problem is that you only have this top-down view,
• 7:05
  so you don't really get in or not much information about the
• 7:09
  structure of the cloud,
• 7:10
  SPEAKER_01: the cloud top height,
• 7:12
  SPEAKER_01: etc.
• 7:14
  SPEAKER_01: But if you also just look up in the sky,
• 7:17
  SPEAKER_01: the clouds are like a 3D dynamical system that's always evolving.
• 7:22
  SPEAKER_01: So it's quite important to also understand the vertical structure.
• 7:28
  SPEAKER_01: Right.
• 7:29
  SPEAKER_00: So not well understood.
• 7:30
  SPEAKER_00: Hard to get data on.
• 7:33
  SPEAKER_00: Exactly.
• 7:33
  Real data sources from above 2D,
• 7:35
  but we really care about the 3D structure.
• 7:38
  SPEAKER_01: Exactly, exactly.
• 7:39
  SPEAKER_01: And this actually, they exist.
• 7:41
  SPEAKER_01: Luckily, there exists also 3D data from polar orbiting satellites.
• 7:46
  SPEAKER_01: So, for example, MODIS is also polar orbiting.
• 7:50
  And I think with this satellite,
• 7:52
  many people are familiar with,
• 7:53
  or Sentinel.
• 7:54
  SPEAKER_01: But there's also satellites that carry active instruments.
• 7:58
  So that really sent down a laser,
• 8:00
  like a LIDAR or radar,
• 8:02
  and based on the kind of...
• 8:05
  SPEAKER_01: the backscatter, we can make assumption about the cloud properties.
• 8:10
  SPEAKER_01: And with these data products, we get really nice vertical information.
• 8:17
  SPEAKER_01: The big problem, however, is that it's very sparse.
• 8:20
  SPEAKER_01: As you can imagine, it's polar orbiting.
• 8:22
  SPEAKER_01: So you get very thin kind of curtains of data.
• 8:25
  So the spatial coverage is super low and also the temporal
• 8:30
  coverage is super low.
• 8:31
  SPEAKER_01: So to get data from the same
• 8:34
  kind of point you are the same like latitude longitude it takes
• 8:38
  14 days and so yeah
• 8:41
  not really any so you can only do like stationary studies but
• 8:45
  not really any
• 8:46
  temporal evolution which is yeah that's always evolving so
• 8:51
  right to study the
• 8:52
  SPEAKER_01: temporal evolution
• 8:54
  SPEAKER_00: Right.
• 8:54
  So your objective is to use this limited 3D data to augment the
• 8:59
  available 2D data
• 9:00
  SPEAKER_00: and get a much more holistic view of the system.
• 9:03
  SPEAKER_01: Exactly.
• 9:04
  SPEAKER_01: So the goal is,
• 9:05
  and this is actually the data set that I was looking for
• 9:08
  throughout my PhD because
• 9:10
  SPEAKER_01: I did a lot of studies on kind of with these active instruments,
• 9:14
  SPEAKER_01: but always the main limitation was that you can only kind of study the
• 9:21
  SPEAKER_01: the cloud at one point in time,
• 9:23
  SPEAKER_01: but you miss kind of the temporal history,
• 9:25
  SPEAKER_01: which is so important.
• 9:27
  SPEAKER_01: And then in other studies,
• 9:28
  SPEAKER_01: I augmented it,
• 9:29
  SPEAKER_01: for example,
• 9:30
  with RFI free analysis data and calculate trajectories with the
• 9:34
  RFI free analysis data.
• 9:36
  SPEAKER_01: But then you always have the problem
• 9:38
  SPEAKER_01: that RFI resolution is super high.
• 9:41
  In the model itself,
• 9:43
  you have different parameterizations that have a lot of uncertainties.
• 9:46
  So there's always this problem that we didn't have this kind of
• 9:53
  temporarily resolved data cube.
• 9:55
  SPEAKER_01: And with IceCloudNet,
• 9:57
  we are trying to close this observational gap and really create
• 10:02
  a 3D cloud
• 10:04
  SPEAKER_01: structure data set
• 10:06
  SPEAKER_01: with the time dimension, so basically for the data set.
• 10:12
  SPEAKER_01: Maybe I can show quickly how the data looks like.
• 10:15
  So I'll explain it a bit because I understand some people are
• 10:18
  listening to this
• 10:19
  SPEAKER_01: only and not seeing the video.
• 10:22
  SPEAKER_01: So here on the left side,
• 10:24
  SPEAKER_01: you see one sample image from Severi,
• 10:28
  SPEAKER_01: so a geostationary satellite that,
• 10:31
  SPEAKER_01: yeah,
• 10:33
  SPEAKER_01: just gather data in different infrared wavelengths.
• 10:36
  Here we see the 10.8 micron wavelength,
• 10:39
  but we also have some visible channels.
• 10:42
  SPEAKER_01: And you get this kind of image every 15 minutes.
• 10:46
  And just to visualize how kind of the polar orbiting satellite
• 10:51
  is covering this,
• 10:52
  SPEAKER_01: is that for one full day,
• 10:54
  SPEAKER_01: so we get actually 96 images from the geostationary satellite.
• 11:00
  SPEAKER_01: You only have one single overpass over a tiny
• 11:03
  kind of area in the geostationary satellite from the polar
• 11:08
  orbiting satellite.
• 11:09
  SPEAKER_01: So really only cover a tiny, tiny bit.
• 11:12
  However,
• 11:12
  we get these really nice vertical profiles,
• 11:15
  which you can see on the right side.
• 11:17
  SPEAKER_01: So here we can see, yeah, that we really have this vertical resolution.
• 11:22
  SPEAKER_01: We see a kind of here, an anvil cloud, we have a convective core.
• 11:26
  SPEAKER_01: So you, you see this somewhere somewhere.
• 11:30
  storms so you have this convective high towering core also a
• 11:34
  lot in the tropics but
• 11:34
  we can observe it sometimes in europe too and then this anvil
• 11:38
  that kind of gets uh
• 11:39
  pressed when the uh when the cloud hits the the top of the
• 11:45
  troposphere this is a
• 11:47
  very typical kind of cloud structure um that um yeah it's also
• 11:52
  changing now due to
• 11:53
  climate change because the yeah the temperatures in the
• 11:56
  troposphere are changing so
• 11:58
  SPEAKER_01: this is yeah like an
• 12:00
  SPEAKER_01: let's say, a hot research topic.
• 12:03
  SPEAKER_01: And...
• 12:05
  SPEAKER_01: But by only having the snapshots,
• 12:07
  SPEAKER_01: it's hard to understand actually how these Android clouds evolve.
• 12:12
  SPEAKER_01: And now I'm just quickly showing,
• 12:16
  and maybe afterwards you can talk a bit about the technical
• 12:18
  details of Ice CloudNet
• 12:19
  SPEAKER_01: if that's interesting,
• 12:22
  SPEAKER_01: what kind of the outcome is.
• 12:24
  Now on the left side,
• 12:26
  you see kind of the input to our Ice CloudNet model,
• 12:28
  which is a 2D model.
• 12:31
  SPEAKER_01: Siviri satellite images.
• 12:33
  SPEAKER_01: I mean,
• 12:34
  SPEAKER_01: in the model,
• 12:34
  SPEAKER_01: we patch it up and so on,
• 12:35
  SPEAKER_01: but now it's just like a full 10 by 10 degree snapshot.
• 12:44
  And the right side,
• 12:45
  you can see the vertical structure that we create with IceCloud Net.
• 12:49
  SPEAKER_01: So you really have the 3D structures.
• 12:53
  You can see anvils and you can see the development of the
• 12:56
  clouds in 3D and through time.
• 13:00
  SPEAKER_01: And with this,
• 13:02
  we increased the availability of the vertical cloud information
• 13:07
  by a factor of a million.
• 13:10
  SPEAKER_01: And we open source this data set.
• 13:12
  And so now researchers can use it to do studies on mesostatic
• 13:18
  convective system evolution.
• 13:20
  SPEAKER_01: errors or cloud interactions and a lot more.
• 13:24
  SPEAKER_01: And yeah, so this is kind of the project in a nutshell.
• 13:29
  SPEAKER_00: Fantastic.
• 13:30
  SPEAKER_00: So the 10 by 6 is because you've gone from a 2D to a 3D volume, right?
• 13:37
  SPEAKER_01: Yeah, not only because the...
• 13:42
  The spaceship,
• 13:43
  also the horizontal cover of the active satellites is very sparse.
• 13:48
  SPEAKER_01: So you don't, so maybe I go back.
• 13:52
  SPEAKER_01: So you basically have like one overpass every 14 days,
• 13:57
  SPEAKER_01: let's say,
• 13:57
  SPEAKER_01: or in,
• 13:59
  SPEAKER_01: I mean,
• 13:59
  SPEAKER_01: they're then getting in parallel,
• 14:00
  SPEAKER_01: but let's say per day in this kind of,
• 14:03
  SPEAKER_01: um,
• 14:05
  SPEAKER_01: 10 by 10 degree patch, you only have one overpass per day.
• 14:10
  So kind of you have also kind of horizontally the coverage is
• 14:14
  very sparse.
• 14:15
  SPEAKER_00: Yeah.
• 14:16
  So the 10 by six is from that slice on the right to the
• 14:19
  volumetric over the full area.
• 14:21
  SPEAKER_00: Makes sense.
• 14:22
  SPEAKER_01: Exactly.
• 14:22
  SPEAKER_01: Exactly.
• 14:23
  SPEAKER_00: Okay.
• 14:24
  SPEAKER_00: Yeah.
• 14:25
  I'm immediately wondering if a similar technique can be used
• 14:27
  for like LIDAR and
• 14:28
  SPEAKER_00: tree high,
• 14:29
  SPEAKER_00: other places where you have these kind of
• 14:31
  Point measurements almost,
• 14:33
  but another view that you could combine with it.
• 14:36
  SPEAKER_01: Yeah.
• 14:37
  SPEAKER_01: Yeah.
• 14:37
  SPEAKER_01: Yeah.
• 14:38
  SPEAKER_01: I mean,
• 14:38
  this one was like one big challenge also in the project that
• 14:44
  you have only these
• 14:48
  SPEAKER_01: data for supervision.
• 14:50
  SPEAKER_01: So you only have the...
• 14:53
  SPEAKER_01: Yeah, the vertical profiles are very sparsely available.
• 14:56
  SPEAKER_01: And in the end, you want to create this whole 3D structure.
• 15:00
  SPEAKER_01: So it took quite a lot of experimenting with the model,
• 15:06
  with how we choose the patches and a lot of data engineering
• 15:10
  until we got to the current.
• 15:13
  SPEAKER_00: Good.
• 15:14
  SPEAKER_00: And this was just one project you did within your PhD,
• 15:17
  SPEAKER_00: or was this kind of the cumulative output?
• 15:21
  SPEAKER_01: Yeah,
• 15:21
  SPEAKER_01: no,
• 15:21
  this was like the final project that I did in my PhD and kind
• 15:25
  of the one that I'm
• 15:27
  most proud of because I always had this problem in my PhD that
• 15:33
  I was missed because
• 15:34
  kind of the overarching topic was using machine learning for
• 15:38
  serious clouds and
• 15:39
  SPEAKER_01: kind of
• 15:42
  SPEAKER_01: working on improving our understanding of that.
• 15:44
  And I always had the issue that I only had these active
• 15:47
  satellite instrument retrievals,
• 15:49
  SPEAKER_01: but then no kind of temporal resolution.
• 15:54
  And yeah,
• 15:54
  as I said before,
• 15:55
  then I was using a lot of ERA-5 data or model output.
• 16:00
  SPEAKER_01: So we did one study,
• 16:01
  SPEAKER_01: for example,
• 16:02
  also on explainable machine learning where we tried to predict
• 16:04
  kind of the
• 16:07
  the Cirrus Cloud properties based on ERA5 data and then try to
• 16:11
  understand which
• 16:13
  meteorological and aerosol variables have the most impact on
• 16:19
  the Cirrus Cloud
• 16:21
  SPEAKER_01: property observations in different conditions.
• 16:23
  SPEAKER_01: So that was another project.
• 16:25
  SPEAKER_01: But I think for this kind of like satellite,
• 16:28
  SPEAKER_01: merging different satellite views and so on,
• 16:33
  SPEAKER_01: this was this last project and
• 16:36
  SPEAKER_00: Fantastic.
• 16:37
  SPEAKER_00: Let's hear about the solution then.
• 16:38
  SPEAKER_00: How does this work?
• 16:41
  SPEAKER_01: Yeah.
• 16:42
  So basically,
• 16:46
  it's a deep learning model that takes as an input the CVRI
• 16:55
  images and a bit of...
• 16:57
  SPEAKER_01: static metadata, like the location, the season and so on.
• 17:03
  And then we predict kind of the vertical,
• 17:07
  we predict the full cube that we saw before.
• 17:12
  SPEAKER_01: But then we can only supervise the model on
• 17:17
  SPEAKER_01: kind of where we have the data actually.
• 17:19
  SPEAKER_01: So we predict the full queue,
• 17:20
  SPEAKER_01: but then we mask it and we only have one kind of curtain,
• 17:25
  SPEAKER_01: one profile that we calculate then the loss on.
• 17:30
  SPEAKER_01: I can briefly show the kind of the model architecture that makes sense.
• 17:35
  SPEAKER_01: So on the left side,
• 17:37
  SPEAKER_01: you can see kind of the infrared,
• 17:39
  SPEAKER_01: the visible channels as an input,
• 17:41
  SPEAKER_01: the static metadata.
• 17:44
  SPEAKER_01: We have like a,
• 17:46
  SPEAKER_01: There's a little bar across the top.
• 17:50
  They might want to click the OK just to clear that so we can
• 17:53
  see what's going on.
• 17:55
  SPEAKER_00: OK, metadata, images.
• 17:58
  SPEAKER_01: You know, exactly.
• 17:59
  SPEAKER_01: And yeah, at the beginning, we experimented just with a simple unit.
• 18:06
  SPEAKER_01: I guess it's always a good starting point.
• 18:09
  And then in the output,
• 18:11
  we actually used and because it's kind of like a 2D to 3D translation.
• 18:16
  SPEAKER_01: So in the output, we used the
• 18:18
  kind of the channel dimension as the vertical dimension because
• 18:22
  then we only
• 18:23
  predict like one channel or like we have like the two different
• 18:26
  variables isolate
• 18:27
  the content and isolate the number concentration that we
• 18:29
  predict but then we use
• 18:30
  the channel dimension as the vertical dimension and then we
• 18:36
  create like these cubes
• 18:39
  SPEAKER_01: Then we use the masking and calculate the loss on the masked variables.
• 18:48
  SPEAKER_01: This is the first try.
• 18:50
  SPEAKER_01: I guess this is always the problem when working with convolutions,
• 18:57
  SPEAKER_01: both in computer vision,
• 18:59
  but especially also with satellite data that you always get
• 19:02
  this blurring effect.
• 19:05
  um if you if you use conf nets um and then we yeah just uh were
• 19:12
  inspired by some
• 19:13
  other work in uh um yeah different discipline like physics
• 19:18
  discipline uh where they
• 19:21
  had like sparse meteor showers or something um and then used to
• 19:25
  introduce this
• 19:25
  discriminator model so it's basically similar to the vae again
• 19:31
  so that you have
• 19:32
  SPEAKER_01: like a
• 19:33
  generator here which is our unit then we have a patch scan that
• 19:39
  then just learns to
• 19:43
  tell real from fake images apart and acts like as a structural
• 19:49
  loss to produce more
• 19:50
  SPEAKER_01: realistic
• 19:54
  SPEAKER_01: structures of clouds.
• 19:55
  SPEAKER_01: For example,
• 19:56
  SPEAKER_01: you have always this fading,
• 19:58
  SPEAKER_01: both in the horizontal,
• 19:58
  SPEAKER_01: but also towards the edges of the cloud,
• 20:01
  SPEAKER_01: which is not what we see actually in the physical world,
• 20:05
  SPEAKER_01: which is not physically consistent.
• 20:07
  SPEAKER_01: And using this discriminator helped a lot to produce better results.
• 20:13
  And yeah,
• 20:14
  then it's kind of just like the EGAN-like architecture that we used.
• 20:21
  and this this architecture presumably evolved to this point
• 20:23
  this wasn't v1 so sorry
• 20:28
  say again as in you you tried a few other like intermediate
• 20:32
  steps towards this
• 20:33
  SPEAKER_00: final architecture
• 20:35
  SPEAKER_01: Exactly.
• 20:35
  SPEAKER_01: I mean, yeah, we tried a lot of different things.
• 20:37
  SPEAKER_01: I mean,
• 20:38
  SPEAKER_01: in the beginning,
• 20:38
  SPEAKER_01: we only predicted like not because now we do like many vertical levels.
• 20:45
  SPEAKER_01: So like I think like 60 or so.
• 20:47
  In the beginning,
• 20:48
  we only did like two levels,
• 20:49
  like two type of different Cirrus clouds.
• 20:52
  SPEAKER_01: And then we did like the full vertical.
• 20:54
  SPEAKER_01: We first did the UNet.
• 20:56
  SPEAKER_01: Also,
• 20:57
  SPEAKER_01: even just using pixel-based approaches,
• 20:59
  SPEAKER_01: but then we saw using the spatial structure of the cloud in forms.
• 21:06
  SPEAKER_01: For example, what I earlier showed was this convective cores.
• 21:10
  SPEAKER_01: If you see the model in
• 21:13
  SPEAKER_01: I hope, I guess that's my interpretation of it.
• 21:15
  SPEAKER_01: What the model learns is you,
• 21:16
  SPEAKER_01: it seems like a conductive core,
• 21:18
  SPEAKER_01: then it usually,
• 21:19
  SPEAKER_01: it knows like,
• 21:21
  SPEAKER_01: it learns that there's usually like an anvil attached to it,
• 21:24
  SPEAKER_01: and then it kind of learns these spatial interactions.
• 21:28
  SPEAKER_01: That's kind of the interpretation,
• 21:30
  SPEAKER_01: I guess,
• 21:31
  why the spatial model works better than just using a pixel
• 21:34
  based approach,
• 21:35
  SPEAKER_01: which other people have done like some years ago already.
• 21:38
  SPEAKER_00: Right.
• 21:40
  SPEAKER_00: And what was the loss function?
• 21:43
  Yeah,
• 21:44
  so the loss function was we had the kind of composition of loss functions.
• 21:51
  So we just use like an L1 loss for the kind of loss for the
• 21:59
  masked input versus the
• 22:01
  SPEAKER_01: kind of the actual observations.
• 22:04
  SPEAKER_01: Again,
• 22:05
  SPEAKER_01: here,
• 22:05
  like the L1 works a bit better than the root mean squared error
• 22:10
  in the sense that
• 22:11
  SPEAKER_01: it produces less blurring.
• 22:13
  SPEAKER_01: And then in the discriminator loss, you have like a Wasserstein loss.
• 22:19
  SPEAKER_01: And then we combine this and have like an adaptive loss rate,
• 22:24
  SPEAKER_01: which we,
• 22:26
  because kind of like a similar setup was used in some seminal
• 22:30
  diffusion model papers.
• 22:33
  And yeah,
• 22:34
  I think it's always good to go and hug and face and check like
• 22:36
  what are,
• 22:38
  SPEAKER_01: kind of like state of the art models and computer vision doing,
• 22:41
  SPEAKER_01: which is,
• 22:41
  SPEAKER_01: I guess,
• 22:42
  SPEAKER_01: like the discipline in,
• 22:45
  SPEAKER_01: uh,
• 22:45
  SPEAKER_01: in machine learning,
• 22:46
  SPEAKER_01: that's closest to what we are doing in satellite data and see what we,
• 22:50
  SPEAKER_01: how we can do a transfer from these,
• 22:53
  SPEAKER_01: like a cutting edge AI research to our application research.
• 22:58
  SPEAKER_01: And yeah.
• 22:59
  SPEAKER_00: Yeah.
• 23:00
  SPEAKER_00: I wonder,
• 23:01
  can you give us an insight to the engineering approach
• 23:02
  considering you have like
• 23:04
  SPEAKER_00: quite a strong engineering background,
• 23:05
  SPEAKER_00: of course.
• 23:07
  SPEAKER_01: Yeah,
• 23:08
  SPEAKER_01: I have to say that this project,
• 23:11
  SPEAKER_01: I'm really fortunate that I could carry it out in the ESA PHY lab.
• 23:16
  SPEAKER_01: So my PhD was based at ETH in Zurich,
• 23:19
  SPEAKER_01: but this project itself was started at the European Space Agency,
• 23:25
  SPEAKER_01: where I had great collaborators.
• 23:30
  SPEAKER_01: Yeah, I mean, the engineering process was quite incremental.
• 23:33
  SPEAKER_01: So there was a lot of whiteboard session from data engineering,
• 23:38
  like how do we co-locate the different data sets together and
• 23:42
  then building up the
• 23:43
  SPEAKER_01: models incrementally and improving the outcome.
• 23:50
  SPEAKER_01: So I think there was...
• 23:53
  SPEAKER_01: I guess this is maybe what other people can resonate with.
• 23:58
  It's about taking existing approaches and then you have to
• 24:03
  refine it to your kind
• 24:06
  of specific problem set because usually it never works by just
• 24:09
  taking something.
• 24:11
  SPEAKER_01: And I think, yeah, there's no...
• 24:14
  SPEAKER_01: I think it's advisable to also use like battle-tested components,
• 24:18
  SPEAKER_01: like not writing too many things from scratch.
• 24:21
  SPEAKER_01: So we,
• 24:21
  SPEAKER_01: for example,
• 24:22
  SPEAKER_01: also use these ConfNext blocks,
• 24:24
  SPEAKER_01: which made quite a buzz,
• 24:26
  SPEAKER_01: I think,
• 24:26
  SPEAKER_01: in the computer vision community some years ago.
• 24:31
  SPEAKER_01: And just using kind of these components,
• 24:35
  SPEAKER_01: I think is really helpful than trying to invent everything from scratch,
• 24:39
  SPEAKER_01: even if that's a lot of fun.
• 24:41
  SPEAKER_01: But I think it just, yeah.
• 24:44
  avoids too many headaches on like the small nitty-gritty
• 24:48
  machine learning that you
• 24:50
  SPEAKER_01: still have anyway.
• 24:51
  SPEAKER_00: Yeah, of course.
• 24:52
  SPEAKER_00: There's always some challenges.
• 24:54
  SPEAKER_00: So this is really, really nice results.
• 24:57
  SPEAKER_00: You got a paper out on this work?
• 25:00
  SPEAKER_01: Yes.
• 25:02
  So currently the paper is on archive and it's hopefully
• 25:05
  accepted soon at AI for the
• 25:07
  SPEAKER_01: Earth Systems.
• 25:08
  SPEAKER_01: It's an AMS journal.
• 25:11
  um and also the data set is out currently one year but we are
• 25:15
  now extending it to
• 25:16
  10 years so maybe you can link it in the in the show notes so
• 25:20
  people are interested
• 25:21
  feel free to play around with data or contact me um if you're
• 25:26
  interested to use the
• 25:27
  SPEAKER_00: data or to learn more about it right and what's next for you then
• 25:33
  SPEAKER_01: Yeah, so I finished a PhD and now I'm transferring to industry again.
• 25:39
  SPEAKER_01: I will start in an Amsterdam-based startup called Dexter Energy,
• 25:44
  and they're doing renewable energy yield prediction and also
• 25:51
  price forecasts for
• 25:53
  SPEAKER_01: renewable energy prices based on weather and satellite data.
• 25:57
  SPEAKER_01: So I'm going a bit away from the climate, science, the physics,
• 26:02
  SPEAKER_01: and more to like energy applied machine learning,
• 26:08
  SPEAKER_01: still staying kind of sticking to the AI for climate change.
• 26:13
  SPEAKER_01: So yeah,
• 26:14
  I guess the renewable energies is a big part of solving the
• 26:21
  demand for more energy
• 26:23
  SPEAKER_01: and yeah,
• 26:26
  SPEAKER_01: exciting times ahead for sure.
• 26:28
  SPEAKER_00: Well, congratulations.
• 26:29
  SPEAKER_00: Very productive PhD.
• 26:31
  SPEAKER_00: Great, great work.
• 26:32
  SPEAKER_00: Fully open sourced as well.
• 26:34
  SPEAKER_00: Paper to look forward to.
• 26:35
  SPEAKER_00: Where's the best place for people to follow your updates?
• 26:41
  SPEAKER_01: So I would say semi-active on LinkedIn.
• 26:45
  SPEAKER_01: I guess that would be the place to follow me.
• 26:49
  SPEAKER_01: Yeah, I guess you can put it also in the show notes.
• 26:52
  SPEAKER_00: Will do.
• 26:53
  SPEAKER_00: Okay, brilliant.
• 26:54
  SPEAKER_00: Thanks for joining me and all the best for your new role.
• 26:57
  SPEAKER_01: Yeah, thanks, Robin.
• 26:58
  SPEAKER_01: It was great talking to you, and yeah, have a good day.