Latest / Satellite image deep learning / Uncertainty Quantification for Neural Networks with Pytorch Lightning UQ Box

Transcript

Uncertainty Quantification for Neural Networks with Pytorch Lightning UQ Box · Satellite image deep learning

• 0:05
  SPEAKER_00: Hello, I'm Robin Cole and welcome to the Satellite Image deep learning podcast.
• 0:10
  SPEAKER_00: In this episode,
• 0:10
  SPEAKER_00: I caught up with Niels Lehman to learn about uncertainty quantification for neural networks.
• 0:15
  SPEAKER_00: The conversation begins with the discussion on Bayesian neural networks and their
• 0:19
  SPEAKER_00: ability to quantify the uncertainty of their predictions.
• 0:22
  SPEAKER_00: Unlike regular deterministic neural networks,
• 0:25
  SPEAKER_00: Bayesian neural networks offer a more principled method for providing predictions
• 0:28
  SPEAKER_00: with a measure of confidence.
• 0:30
  SPEAKER_00: Niels then introduces the Lightning UQ Box project on GitHub,
• 0:34
  SPEAKER_00: a tool that enables rapid experimentation with a variety of uncertainty
• 0:38
  SPEAKER_00: quantification techniques for neural networks.
• 0:41
  SPEAKER_00: Model interpretability is a crucial topic,
• 0:43
  SPEAKER_00: essential for providing transparency to end users of machine learning models.
• 0:48
  SPEAKER_00: I hope you enjoy this episode.
• 0:50
  SPEAKER_00: Hi Niels, welcome to the podcast.
• 0:52
  SPEAKER_01: Hi Rowan, thanks for having me.
• 0:54
  SPEAKER_00: It's my pleasure.
• 0:54
  SPEAKER_00: Let's begin with where you work and what you do.
• 0:56
  SPEAKER_01: Yeah,
• 0:58
  SPEAKER_01: so I'm currently a second year PhD student at the Technical University of Munich
• 1:04
  SPEAKER_01: and I'm supervised by Professor Xiao-Shang Zhu and Jonathan Ember.
• 1:10
  SPEAKER_01: And yeah, I'm currently working on sea level rise related machine learning problems, I guess.
• 1:20
  SPEAKER_01: And I'm broadly interested in Bayesian deep learning,
• 1:23
  SPEAKER_01: genitive modeling and how these methods can be applied to Earth observation or
• 1:28
  SPEAKER_01: other data sources.
• 1:31
  SPEAKER_01: Besides,
• 1:31
  SPEAKER_01: I'm also quite passionate about open source contribution,
• 1:35
  SPEAKER_01: which I guess might be a bit of a topic in this session.
• 1:40
  SPEAKER_00: Naturally.
• 1:40
  SPEAKER_00: I think most people will be familiar with generative IAR at this point.
• 1:44
  SPEAKER_00: Do you mind giving us a quick idea of what Bayesian methods are?
• 1:48
  SPEAKER_01: Yeah, sure.
• 1:49
  SPEAKER_01: So,
• 1:51
  SPEAKER_01: I mean,
• 1:52
  SPEAKER_01: by default,
• 1:53
  SPEAKER_01: sort of the most of the neural network architecture and so on that we use and just
• 1:59
  SPEAKER_01: stay with supervised learning are deterministic in the sense that the output you
• 2:05
  SPEAKER_01: give it and the output that results through a forward path will be the same every time.
• 2:14
  SPEAKER_01: And then there are some methods that also keep a deterministic forward pass,
• 2:18
  SPEAKER_01: but maybe output some other parameters.
• 2:22
  SPEAKER_01: But by default, neural networks are not able to quantify uncertainty.
• 2:28
  SPEAKER_01: And this is ultimately something we would like,
• 2:32
  SPEAKER_01: because we want to have some sense of uncertainty in the predictions that our model
• 2:38
  SPEAKER_01: is giving us.
• 2:39
  SPEAKER_01: And one way to enhance neural networks to do this stuff is Bayesian methods.
• 2:48
  SPEAKER_01: And the rough idea is that instead of having just point-wise network weights,
• 2:54
  SPEAKER_01: you try to model each network weight
• 2:59
  SPEAKER_01: by a distribution most commonly by normal distribution and then there's several
• 3:05
  SPEAKER_01: approximations you need to take because in practice like doing that fully would be
• 3:09
  SPEAKER_01: computationally infeasible but as a result
• 3:14
  SPEAKER_01: you can then sample these weights and practically do multiple forward paths and you
• 3:20
  SPEAKER_01: get slightly different results every time.
• 3:22
  SPEAKER_01: And that gives you a notion of uncertainty that you can then compute.
• 3:30
  SPEAKER_01: And so yeah, there exists a range of methods, but ultimately that's their goal.
• 3:35
  SPEAKER_00: I'm quite interested to understand the difference with supervised methods,
• 3:38
  SPEAKER_00: where obviously if you have a binary classification,
• 3:40
  SPEAKER_00: it would say 80% probability class A or 20 B.
• 3:44
  SPEAKER_00: That already sounds like a kind of uncertainty.
• 3:46
  SPEAKER_00: How do the Bayesian outputs differ?
• 3:51
  SPEAKER_01: For classification problems, you have these most normally binary or other, just by using the softmax.
• 4:01
  SPEAKER_01: have a notion of probability.
• 4:05
  SPEAKER_01: There are several papers that these so-called pseudo probabilities are quite miscalibrated,
• 4:12
  SPEAKER_01: meaning that they sort of don't align well with the empirical underlying distribution.
• 4:20
  SPEAKER_01: So that's why in the classification setting,
• 4:25
  SPEAKER_01: there's still people that try to then go to different approaches to try to fix this miscalibration.
• 4:31
  SPEAKER_01: There's a whole range of methods that people have come up with.
• 4:37
  SPEAKER_01: And in the Bayesian sense,
• 4:39
  SPEAKER_01: you would then also try to get a distribution over different softmax outputs or so
• 4:45
  SPEAKER_01: in the sense.
• 4:47
  SPEAKER_01: So that's in the classification case,
• 4:48
  SPEAKER_01: but sort of in the regression case,
• 4:50
  SPEAKER_01: for example,
• 4:52
  SPEAKER_01: if you just stay with a single target regression output,
• 4:56
  SPEAKER_01: then you just get one output value and you don't have this softmax state where you
• 5:01
  SPEAKER_01: can interpret some sort of probability.
• 5:04
  SPEAKER_01: So there actually,
• 5:05
  SPEAKER_01: this becomes maybe,
• 5:08
  SPEAKER_01: yeah,
• 5:09
  SPEAKER_01: you really need to go to these sorts of methods to even get a notion of uncertainty.
• 5:14
  SPEAKER_00: Right.
• 5:14
  SPEAKER_00: So going from kind of pseudo uncertainty or no uncertainty to a well-defined
• 5:19
  SPEAKER_00: uncertainty distribution,
• 5:21
  SPEAKER_00: essentially.
• 5:22
  SPEAKER_01: Yeah.
• 5:23
  SPEAKER_01: And so, yeah, well calibrated uncertainties.
• 5:30
  SPEAKER_01: Although, I mean, this is also sort of a topic because when you start looking
• 5:35
  SPEAKER_01: into uncertainty there's unfortunately no ground truth uncertainty so for any given
• 5:42
  SPEAKER_01: example you can't really say oh well the output should be this and the right
• 5:48
  SPEAKER_01: uncertainty is this um because yeah that's that's just uh impossible to do and um
• 5:57
  SPEAKER_01: uh yeah similarly um
• 6:01
  SPEAKER_01: uh you kind of need to work also with other metrics to try to get a sense of how
• 6:06
  SPEAKER_01: well calibrated these are and most of the calibration methods for example they are
• 6:11
  SPEAKER_01: computed across a whole data set so even within that you might get a notion of the
• 6:18
  SPEAKER_01: performance of a model for a data set but then let's say during a during
• 6:26
  SPEAKER_01: production or when you're trying to predict a step for any particular sample that
• 6:31
  SPEAKER_01: you pass through,
• 6:33
  SPEAKER_01: you again don't really have that notion.
• 6:39
  SPEAKER_01: From a metric standpoint, it gets a bit more involved.
• 6:43
  SPEAKER_01: I think it's quite necessary, especially for Earth observation tasks.
• 6:48
  SPEAKER_00: I 100% agree.
• 6:49
  SPEAKER_00: This is probably a good time to start talking about what you've been working on specifically then.
• 6:54
  SPEAKER_01: Sure, yeah, I would love to.
• 6:57
  SPEAKER_00: Okay, so your project on GitHub Lightning UQ box, do you mind telling us what that is?
• 7:04
  SPEAKER_01: Sure, yeah.
• 7:04
  SPEAKER_01: So I first began looking into uncertainty methods during my master thesis,
• 7:12
  SPEAKER_01: and I sort of found it difficult to find sort of a range of different
• 7:18
  SPEAKER_01: implementation because from the machine learning literature,
• 7:22
  SPEAKER_01: by now there exists tons of methods.
• 7:25
  SPEAKER_01: And my supervisor at the time,
• 7:27
  SPEAKER_01: Eric,
• 7:29
  SPEAKER_01: so one thing he said that was a recommendation for a PhD was a good thing is how
• 7:35
  SPEAKER_01: quickly you're able to iterate through experiments in the sense that if you think,
• 7:39
  SPEAKER_01: oh,
• 7:40
  SPEAKER_01: well,
• 7:41
  SPEAKER_01: maybe this method should be interesting because of this property,
• 7:44
  SPEAKER_01: it might work well,
• 7:45
  SPEAKER_01: then it shouldn't take you long to just try that idea because otherwise you just
• 7:50
  SPEAKER_01: stuck in like long cycles.
• 7:53
  SPEAKER_01: So when I then began my PhD here I thought it would be good to have some sort of
• 7:59
  SPEAKER_01: common framework where you implement a variety of methods from sort of different
• 8:06
  SPEAKER_01: theoretical underpinnings with the exact idea that hey for your data set sort of
• 8:15
  SPEAKER_01: It didn't really take that long for you to just try ablation methods.
• 8:19
  SPEAKER_01: And so, yeah, it's sort of a motivation to make this a bit easier.
• 8:25
  SPEAKER_00: Okay.
• 8:26
  SPEAKER_00: So I can get a really tangible feeling for how this works in practice.
• 8:30
  SPEAKER_00: Does it take typical models and then add some extra capability to them?
• 8:35
  SPEAKER_00: Is it a totally new kind of model?
• 8:37
  SPEAKER_01: Yeah, so there are some approaches.
• 8:42
  SPEAKER_01: They just work on...
• 8:45
  SPEAKER_01: deterministic works.
• 8:47
  SPEAKER_01: So you don't really need to make any changes to the architecture.
• 8:50
  SPEAKER_01: But for example,
• 8:51
  SPEAKER_01: in Bayesian methods where you then try to model your weights,
• 8:55
  SPEAKER_01: not just point-wise,
• 8:56
  SPEAKER_01: but as a distribution,
• 8:57
  SPEAKER_01: you obviously have to make some changes to the architecture.
• 9:00
  SPEAKER_01: And with all of these methods that we've implemented so far,
• 9:04
  SPEAKER_01: we've tried to sort of keep it that you still have all the flexibility you want
• 9:09
  SPEAKER_01: with your network.
• 9:11
  SPEAKER_01: So
• 9:12
  SPEAKER_01: all these methods sort of just take a PyTorch architecture and then we'll make the
• 9:18
  SPEAKER_01: necessary changes under the hood,
• 9:20
  SPEAKER_01: for example,
• 9:20
  SPEAKER_01: to convert it to a Bayesian neural network or so.
• 9:23
  SPEAKER_01: So we sort of try to keep the constraints on the user as small as possible,
• 9:32
  SPEAKER_01: but at the same time,
• 9:34
  SPEAKER_01: hope to illustrate sort of how these methods work so people get
• 9:38
  SPEAKER_01: an understanding of them but aren't held back by having to implement a new page in
• 9:44
  SPEAKER_00: your network every single time they want to use one okay sounds good and if
• 9:49
  SPEAKER_00: somebody wants to use it what's what do they need to know i see you've got
• 9:52
  SPEAKER_00: lightning in the name so i'm assuming it integrates with the lightning by torch
• 9:56
  SPEAKER_00: lightning training framework
• 9:58
  SPEAKER_01: Yeah, exactly.
• 10:03
  SPEAKER_01: Fortunately, we're in an area where lots of research code is actually published.
• 10:10
  SPEAKER_01: When implementing,
• 10:12
  SPEAKER_01: we've been looking at a lot of these research implementations with the accompanying papers,
• 10:17
  SPEAKER_01: but often those are written specifically for that paper or project with lots of
• 10:23
  SPEAKER_01: additional code for visualizations and whatnot.
• 10:27
  SPEAKER_01: And Lightning is very nice in the sense that it gives you a certain structure in
• 10:33
  SPEAKER_01: which to implement methods.
• 10:36
  SPEAKER_01: And so I think from one side,
• 10:40
  SPEAKER_01: it makes it clear to understand,
• 10:41
  SPEAKER_01: well,
• 10:42
  SPEAKER_01: what's actually happening in like one training iteration when you train this or
• 10:46
  SPEAKER_01: that method.
• 10:48
  SPEAKER_01: And so because with Lightning,
• 10:51
  SPEAKER_01: then I would exactly know where to look for this sort of information,
• 10:55
  SPEAKER_01: like a big...
• 10:56
  SPEAKER_01: custom PyTorch implementation,
• 10:57
  SPEAKER_01: I might have to go through different files and see where exactly this is happening.
• 11:02
  SPEAKER_01: And the other is that it provides you so much useful properties for scaling,
• 11:10
  SPEAKER_01: like multi-GPU training and so on.
• 11:13
  SPEAKER_01: or logging.
• 11:15
  SPEAKER_01: And these are all things that for researchers,
• 11:18
  SPEAKER_01: you shouldn't need to spend like additional code or time to just try these things.
• 11:25
  SPEAKER_01: And so since Lightning gives you so much of this flexibility, we want to sort of fit in that framework.
• 11:33
  SPEAKER_01: Yeah.
• 11:34
  SPEAKER_00: Fantastic.
• 11:35
  SPEAKER_01: So that's sort of the motivation for Lightning.
• 11:38
  SPEAKER_00: Right,
• 11:38
  SPEAKER_00: so you need to be a little bit familiar with PyTorch Lightning,
• 11:41
  SPEAKER_00: but other than that,
• 11:42
  SPEAKER_00: it will look standard.
• 11:45
  SPEAKER_00: You've got lots of different techniques in there.
• 11:47
  SPEAKER_00: You have a few tutorials as well.
• 11:49
  SPEAKER_00: Do you mind just telling us what they cover?
• 11:52
  SPEAKER_01: Sure, yeah.
• 11:53
  SPEAKER_01: So the tutorials cover,
• 11:56
  SPEAKER_01: I mean,
• 11:56
  SPEAKER_01: in a sense,
• 11:57
  SPEAKER_01: a bit of,
• 11:58
  SPEAKER_01: well,
• 11:58
  SPEAKER_01: we hope to illustrate these methods,
• 11:59
  SPEAKER_01: but they also provide a bit of a sanity check to see how they work on like,
• 12:06
  SPEAKER_01: yeah,
• 12:07
  SPEAKER_01: these simple toy problems.
• 12:09
  SPEAKER_01: And so I'm not sure if it's useful to like share the screen or something for...
• 12:15
  SPEAKER_00: Yeah,
• 12:15
  SPEAKER_00: why don't you do that?
• 12:17
  SPEAKER_01: Okay.
• 12:18
  SPEAKER_01: Yeah,
• 12:18
  SPEAKER_01: so this is just the docs page where we try to give an overview over the different
• 12:25
  SPEAKER_01: methods that we've implemented so far.
• 12:28
  SPEAKER_01: But we're also looking to add new methods or have suggestions.
• 12:32
  SPEAKER_01: So yeah, very much looking for feedback.
• 12:35
  SPEAKER_01: And then on the tutorial side,
• 12:37
  SPEAKER_01: yeah,
• 12:38
  SPEAKER_01: in a sense for a lot of these concepts for uncertainty quantification,
• 12:43
  SPEAKER_01: like 1D regression examples are quite nice.
• 12:46
  SPEAKER_01: just because you can clearly illustrate things.
• 12:50
  SPEAKER_01: And yeah,
• 12:52
  SPEAKER_01: so they sort of all follow the same setup,
• 12:59
  SPEAKER_01: where maybe because we talked about some Bayesian methods before,
• 13:03
  SPEAKER_01: we can pick this one.
• 13:05
  SPEAKER_01: So we first tried to give sort of an intuition about, well, like...
• 13:12
  SPEAKER_01: rundown explanation of what this method does with some rough notation,
• 13:17
  SPEAKER_01: but we try to link to the papers to sort of give a more rigorous,
• 13:22
  SPEAKER_01: well,
• 13:23
  SPEAKER_01: derivation for the people that are interested.
• 13:26
  SPEAKER_01: And then, well, don't need the import statements.
• 13:31
  SPEAKER_01: And then basically we have to set up a small toy data set that you can also find
• 13:38
  SPEAKER_01: these types of data sets often in those machine learning papers where they
• 13:43
  SPEAKER_01: introduce such methods.
• 13:46
  SPEAKER_01: And then
• 13:47
  SPEAKER_01: yeah maybe to be also a bit more concretely sort of what a person or user would have to do.
• 13:55
  SPEAKER_01: So previously I said sort of you can bring your own model.
• 14:00
  SPEAKER_01: Here we just use a simple multi-layer perceptron with input some hidden layers and
• 14:07
  SPEAKER_01: some outputs and then each
• 14:11
  SPEAKER_01: uq method is implemented as a lightning module so for the people familiar with or
• 14:17
  SPEAKER_01: unfamiliar with like or familiar you basically for a lightning training you need a
• 14:22
  SPEAKER_01: lightning module
• 14:24
  SPEAKER_01: It's sort of your model,
• 14:25
  SPEAKER_01: and it describes what the model should do during training,
• 14:28
  SPEAKER_01: validation,
• 14:29
  SPEAKER_01: testing,
• 14:29
  SPEAKER_01: and then you need a lightning data module,
• 14:32
  SPEAKER_01: which sort of handles the data splitting and the data loading.
• 14:36
  SPEAKER_01: And those two combined with a so-called lightning trainer can then execute the training.
• 14:47
  SPEAKER_01: In this case, the spatial neural networks would take in the network that we defined above.
• 14:55
  SPEAKER_01: You can also use any optimizer you want that you say you fit or you can try out.
• 15:04
  SPEAKER_01: Then this parameter is for the spatial methods.
• 15:08
  SPEAKER_01: There was a recent paper last year where
• 15:11
  SPEAKER_01: that showed that even you can just make some layers in your neural network
• 15:18
  SPEAKER_01: stochastic or make them Bayesian and keep the other ones deterministic and you can
• 15:22
  SPEAKER_01: still get very good performance and so this sort of reduces a bit the computational
• 15:29
  SPEAKER_01: load that these methods can have so that's
• 15:32
  SPEAKER_01: something these can do and then you define how many samples you want to draw during
• 15:37
  SPEAKER_01: training and testing to train your methods and then you pass in the trainer and
• 15:44
  SPEAKER_01: exactly then you can execute training and see the prediction.
• 15:54
  SPEAKER_01: You can probably certainly tune these models in the way you want,
• 15:59
  SPEAKER_01: but we sort of just illustrate to sort of keep a base case.
• 16:03
  SPEAKER_01: And so now you can see that you actually have some uncertainty bands that come with your model.
• 16:12
  SPEAKER_01: Exactly, yeah.
• 16:13
  SPEAKER_01: And then that's sort of the format of all these different tutorials.
• 16:18
  SPEAKER_00: And I believe you have a remote sensing specific tutorial as well.
• 16:23
  SPEAKER_01: Ultimately,
• 16:25
  SPEAKER_01: we are of course motivated by Earth observation data,
• 16:29
  SPEAKER_01: but usually in that case,
• 16:31
  SPEAKER_01: just handling the data is a bit more involved.
• 16:34
  SPEAKER_01: but we're hoping to add more tutorials.
• 16:38
  SPEAKER_01: So this is one,
• 16:39
  SPEAKER_01: this might also be nice because recently there's been a lot of research in a field
• 16:45
  SPEAKER_01: called conformal prediction.
• 16:48
  SPEAKER_01: And conformal prediction is also a uncertainty quantification method that works on
• 16:56
  SPEAKER_01: just pre-trained neural networks.
• 16:58
  SPEAKER_01: So you don't need to change your architecture or anything.
• 17:00
  SPEAKER_01: And so the primary aim of that is
• 17:04
  SPEAKER_01: is looking at coverage in the sense that the prediction sets or integrals that your
• 17:10
  SPEAKER_01: model outputs should contain the true value with a specified value of alpha.
• 17:19
  SPEAKER_01: There's some,
• 17:19
  SPEAKER_01: this tutorial,
• 17:21
  SPEAKER_01: I think it's linked somewhere,
• 17:22
  SPEAKER_01: but there's some great tutorials for these conformal methods for people that are interested.
• 17:29
  SPEAKER_01: But for this tutorial, we chose the V645.
• 17:33
  SPEAKER_01: I'm not sure actually what the pronunciation is,
• 17:36
  SPEAKER_01: which is a classification data set that can be loaded with TorchGeo.
• 17:46
  SPEAKER_01: For conformal prediction, you always need a separate calibration data set.
• 17:52
  SPEAKER_01: We've just implemented that.
• 17:55
  SPEAKER_01: There's a total of 45 different classes.
• 17:59
  SPEAKER_01: In this case, we thought we could just illustrate again how you can bring your own models.
• 18:06
  SPEAKER_01: In this case, we use
• 18:07
  SPEAKER_01: a pre-trained Sentinel-2 model from this accompanying paper,
• 18:15
  SPEAKER_01: from the SSL4EO paper,
• 18:17
  SPEAKER_01: and train it with the TorchGeo trainer class,
• 18:22
  SPEAKER_01: or more specifically,
• 18:23
  SPEAKER_01: just actually fine-tune it.
• 18:26
  SPEAKER_01: So we've done that, so you don't have to do the fine-tuning.
• 18:30
  SPEAKER_01: But sort of the interesting metric to look at with conformal prediction is this coverage parameter.
• 18:37
  SPEAKER_01: And so here we see we get 0.79 coverage.
• 18:43
  SPEAKER_01: And now,
• 18:46
  SPEAKER_01: so practically speaking from the implementation standpoint,
• 18:49
  SPEAKER_01: it's again that we have this method.
• 18:52
  SPEAKER_01: You just pass in your model.
• 18:53
  SPEAKER_01: You specify the alpha value.
• 18:56
  SPEAKER_01: Commonly one is 0.1, which means that you have a desired coverage rate of 90%.
• 19:01
  SPEAKER_01: So over your data set.
• 19:06
  SPEAKER_01: Over all the examples that you have,
• 19:08
  SPEAKER_01: in 90% of the cases,
• 19:10
  SPEAKER_01: the true label should be contained in the prediction set that your model is giving.
• 19:15
  SPEAKER_01: And currently,
• 19:16
  SPEAKER_01: we just have a deterministic network that,
• 19:19
  SPEAKER_01: well,
• 19:19
  SPEAKER_01: it gives you just the highest softmax score as a class output.
• 19:24
  SPEAKER_01: And this wraps conformal method actually gives you a prediction set.
• 19:31
  SPEAKER_01: So it gives you a range of possible values.
• 19:34
  SPEAKER_01: And the idea is that you want to achieve high coverage,
• 19:38
  SPEAKER_01: but under sort of a minimized prediction set,
• 19:41
  SPEAKER_01: because you would always just output all classes.
• 19:44
  SPEAKER_01: You would have perfect coverage every time, but that's of course not very useful.
• 19:48
  SPEAKER_01: So yeah, it tries to minimize the set size.
• 19:53
  SPEAKER_01: We can use that and then we can see here that the coverage has indeed increased
• 19:59
  SPEAKER_01: substantially from 0.79 to now higher the coverage value that we wanted at least.
• 20:08
  SPEAKER_01: And you can also see that the set size has,
• 20:11
  SPEAKER_01: of course,
• 20:12
  SPEAKER_01: increased a bit,
• 20:13
  SPEAKER_01: but this sort of gives you an informed way of how you should construct these set sizes.
• 20:19
  SPEAKER_01: And we've also made an example.
• 20:24
  SPEAKER_01: So this is the input image.
• 20:26
  SPEAKER_01: The true label is harbor.
• 20:30
  SPEAKER_01: So if you actually look at the logit or predictions,
• 20:35
  SPEAKER_01: then the highest value that the model would actually predict is bridge,
• 20:39
  SPEAKER_01: which you might say might also be reasonable because these things might look like bridges.
• 20:46
  SPEAKER_01: But the prediction set that we get out with the reps method contains bridge, harbor, and ship.
• 20:53
  SPEAKER_01: So, we can see that by applying reps, the true class is actually then contained in the prediction set.
• 21:02
  SPEAKER_01: Yeah,
• 21:03
  SPEAKER_01: so it's sort of an illustration of a simple classification cast,
• 21:07
  SPEAKER_01: and we're currently working on some segmentation examples,
• 21:11
  SPEAKER_01: since a lot of tests in EO are segmentation or pixel-wise regression cases.
• 21:19
  SPEAKER_00: Thank you for that introduction and that overview of that particular tutorial there.
• 21:24
  SPEAKER_00: Why don't we just go to the GitHub repository page and we can leave people with that view there.
• 21:30
  SPEAKER_00: And in the meantime,
• 21:31
  SPEAKER_00: I'm also interested,
• 21:32
  SPEAKER_00: obviously,
• 21:33
  SPEAKER_00: to know what your ambitions are for this project,
• 21:35
  SPEAKER_00: what you would like people to do,
• 21:38
  SPEAKER_00: how you'd like them to contribute potentially.
• 21:41
  SPEAKER_01: Um, yeah, sure.
• 21:42
  SPEAKER_01: So,
• 21:43
  SPEAKER_01: um,
• 21:44
  SPEAKER_01: I mean,
• 21:44
  SPEAKER_01: to be like fully honest or transparent as in the recent,
• 21:48
  SPEAKER_01: like,
• 21:48
  SPEAKER_01: I think,
• 21:49
  SPEAKER_01: yeah,
• 21:49
  SPEAKER_01: very recently also discovered there's like other groups that are starting to work
• 21:53
  SPEAKER_01: on this sort of software packages.
• 21:54
  SPEAKER_01: So there's one called torch uncertainty,
• 21:57
  SPEAKER_01: and then also one from,
• 21:59
  SPEAKER_01: uh,
• 21:59
  SPEAKER_01: posteriors from,
• 22:00
  SPEAKER_01: uh,
• 22:01
  SPEAKER_01: uh,
• 22:01
  SPEAKER_01: from the newer,
• 22:03
  SPEAKER_01: um,
• 22:04
  SPEAKER_01: company,
• 22:04
  SPEAKER_01: a company called neural.
• 22:06
  SPEAKER_01: So it's definitely,
• 22:07
  SPEAKER_01: there's some movement in this space,
• 22:10
  SPEAKER_01: so to say,
• 22:11
  SPEAKER_01: which is overall,
• 22:12
  SPEAKER_01: I think that's really cool.
• 22:16
  SPEAKER_01: But obviously we've also done quite a bit of work and the overall aim is sort of
• 22:21
  SPEAKER_01: that we want to,
• 22:23
  SPEAKER_01: at this stage,
• 22:23
  SPEAKER_01: provide sort of a starting basis for people to see how one could implement all of
• 22:27
  SPEAKER_01: these methods.
• 22:29
  SPEAKER_01: And of course,
• 22:30
  SPEAKER_01: we want people to use these methods and see how they work for their problems and
• 22:36
  SPEAKER_01: how the implementations could be improved.
• 22:38
  SPEAKER_01: Because,
• 22:39
  SPEAKER_01: yeah,
• 22:39
  SPEAKER_01: if you just work with a couple of people to implement these methods,
• 22:42
  SPEAKER_01: you might not consider all the different ways that people might want to use this.
• 22:45
  SPEAKER_01: So, yeah, I'm really looking for people to provide feedback and try these things out.
• 22:52
  SPEAKER_01: Because, yeah, the spirit of this is very much open source.
• 22:58
  SPEAKER_01: so that, yeah, all people can make contributions or recommendations or whatever.
• 23:04
  SPEAKER_01: So,
• 23:04
  SPEAKER_01: yeah,
• 23:05
  SPEAKER_01: this is just,
• 23:06
  SPEAKER_01: it is sort of also explained on the GitHub page or laid out in the GitHub page a bit.
• 23:10
  SPEAKER_01: So, again, this has the overview of the methods.
• 23:14
  SPEAKER_01: And, I mean, the package is now on PyPy, so it can be just installed with PIP.
• 23:21
  SPEAKER_01: And, yeah,
• 23:24
  SPEAKER_01: Yeah,
• 23:25
  SPEAKER_01: I mean,
• 23:26
  SPEAKER_01: ultimately,
• 23:26
  SPEAKER_01: I think there's lots of different other methods that are interesting,
• 23:31
  SPEAKER_01: especially from the generative modeling side.
• 23:36
  SPEAKER_01: I mean, diffusion models, I saw a very hot topic for all different things.
• 23:40
  SPEAKER_01: But oftentimes, I think they're also just tailored for natural RGB images.
• 23:46
  SPEAKER_01: So I think there's some customizations that are necessary when working with EO data.
• 23:52
  SPEAKER_01: And I think those are also things that could be made more simple.
• 23:57
  SPEAKER_01: But yeah, I mean, there's tons of things out there.
• 24:01
  SPEAKER_01: And yeah, we just like if people want to try it out and provide feedback.
• 24:07
  SPEAKER_01: There you have it.
• 24:09
  SPEAKER_00: That's the place to go.
• 24:10
  SPEAKER_00: And if people want to follow your updates, which is the best platform?
• 24:15
  SPEAKER_01: Um,
• 24:15
  SPEAKER_01: I mean,
• 24:15
  SPEAKER_01: currently we just have the GitHub page that we make,
• 24:19
  SPEAKER_01: uh,
• 24:20
  SPEAKER_01: yeah,
• 24:20
  SPEAKER_01: that we,
• 24:21
  SPEAKER_01: uh,
• 24:22
  SPEAKER_01: follow,
• 24:23
  SPEAKER_01: uh,
• 24:23
  SPEAKER_01: or make all the updates to,
• 24:24
  SPEAKER_01: we're going to do a release kind of soon,
• 24:27
  SPEAKER_01: but currently we don't have any other platforms set up for communication,
• 24:32
  SPEAKER_01: uh,
• 24:32
  SPEAKER_01: et cetera.
• 24:33
  SPEAKER_00: Um, yeah.
• 24:36
  SPEAKER_00: And you personally, you're active on LinkedIn, Twitter, the usual places, presumably?
• 24:41
  SPEAKER_01: Yeah, so Twitter or email, you can definitely reach me.
• 24:45
  SPEAKER_01: I think that's also on my GitHub profile, so that's probably the easiest way to reach out.
• 24:52
  SPEAKER_01: Or also some of these other people that have been involved.
• 24:57
  SPEAKER_01: Mainly one collaborator from DLR, the German Space Agency, Nina Gottschling.
• 25:09
  SPEAKER_01: So we started this a while ago and we're hoping to continue this work.
• 25:16
  SPEAKER_00: Fantastic.
• 25:16
  SPEAKER_00: Well, thanks so much for the overview.
• 25:18
  SPEAKER_00: It's a really interesting and important topic, I think, model interpretability.
• 25:22
  SPEAKER_00: Best of luck with the development and your PhD, and I look forward to hearing updates in the future.
• 25:28
  SPEAKER_01: Awesome, yeah.
• 25:29
  SPEAKER_01: Thanks very much for having me.
• 25:31
  SPEAKER_01: Cool.
• 25:32
  SPEAKER_01: See you.
• 25:32
  SPEAKER_01: Bye-bye.