Latest / Satellite image deep learning / The FLAIR land cover mapping challenge
Transcript
- 0:05SPEAKER_00: Hello, I'm Robin Cole, and welcome to the Satellite Image Deep Learning Podcast.
- 0:09SPEAKER_00: In this episode,
- 0:10SPEAKER_00: I caught up with Nicholas Gonthier to learn about the FLARE Land Cover Mapping Challenge.
- 0:14SPEAKER_00: In this challenge, 20 centimeter resolution drawing imagery was used to create high quality annotations.
- 0:20SPEAKER_00: This data was paired with a time series of medium resolution Sentinel-2 images to
- 0:25SPEAKER_00: create a rich multidimensional data set.
- 0:28SPEAKER_00: Participants in the challenge were able to surpass the baseline solution by 10
- 0:32SPEAKER_00: points in the target metric,
- 0:33SPEAKER_00: representing a significant step forward in land cover classification capabilities.
- 0:38SPEAKER_00: The dataset is now being expanded to cover a larger area and incorporate additional
- 0:42SPEAKER_00: imaging modalities,
- 0:44SPEAKER_00: which have been shown to improve performance on this task.
- 0:47SPEAKER_00: Nicholas also provided important context about the objectives of the organization
- 0:51SPEAKER_00: running this challenge,
- 0:52SPEAKER_00: such as the need to balance model performance with processing costs.
- 0:56SPEAKER_00: This was a fascinating conversation, and I hope you enjoy this episode.
- 1:00SPEAKER_00: If you are watching on YouTube,
- 1:01SPEAKER_00: please like and subscribe,
- 1:02SPEAKER_00: as this will really help to grow the channel and reach a wider audience.
- 1:05SPEAKER_00: Thank you.
- 1:06SPEAKER_00: Hi, Nicholas.
- 1:09SPEAKER_00: Welcome to the podcast.
- 1:10SPEAKER_01: Hi, Robin.
- 1:11SPEAKER_01: Thank you for welcoming me.
- 1:13SPEAKER_00: It's a pleasure.
- 1:13SPEAKER_00: Let's begin with where you work and what you do.
- 1:16SPEAKER_01: Okay, so I'm a project manager in the innovation team at IGEN.
- 1:22SPEAKER_01: So IGEN is a national agency,
- 1:25SPEAKER_01: nonprofit one in France,
- 1:27SPEAKER_01: working about geographical and forestry information.
- 1:31SPEAKER_01: And so we are creating a lot of data for different policy,
- 1:36SPEAKER_01: public policy and policymakers,
- 1:38SPEAKER_01: mainly about agricultural information,
- 1:41SPEAKER_01: forestry,
- 1:42SPEAKER_01: ecological and land use.
- 1:44SPEAKER_00: Mm-hmm.
- 1:46SPEAKER_01: I'm more working about deep learning for observation and transduction.
- 1:51SPEAKER_00: Fantastic.
- 1:53SPEAKER_00: That's a nice introduction to the background.
- 1:56SPEAKER_00: And you've been involved in the Flare Challenge.
- 1:58SPEAKER_00: Do you mind giving us the background to that?
- 2:01SPEAKER_01: Yeah,
- 2:02SPEAKER_01: so in fact,
- 2:03SPEAKER_01: we create these very high quality data sets that we've called Flare about land
- 2:09SPEAKER_01: cover semantic segmentation in order to help us at Hygen to produce land cover projects,
- 2:16SPEAKER_01: especially products that can be deployed at country scale.
- 2:21SPEAKER_01: So it's now been used in the production pipeline at the Institute.
- 2:27SPEAKER_01: in order to have a very precise land cover.
- 2:29SPEAKER_01: And for that,
- 2:30SPEAKER_01: we have annotated a lot of different images from different regions of France in
- 2:34SPEAKER_01: order to have broad diversity.
- 2:36SPEAKER_01: And we then trained a supervised model on this data in order to help us to produce
- 2:45SPEAKER_01: this kind of mapping of the territory.
- 2:48SPEAKER_01: And we then decided to open both the datasets
- 2:53SPEAKER_01: and the code and the model in order to be able to reproduce.
- 2:58SPEAKER_01: And finally,
- 2:58SPEAKER_01: we organized two different challenges in order to encourage community,
- 3:05SPEAKER_01: more really computer vision and remote sensing community to work on our problem.
- 3:11SPEAKER_00: Fantastic.
- 3:12SPEAKER_00: So I think the first version of the competition and the dataset was 2022,
- 3:15SPEAKER_00: and then there was another version a year or two later.
- 3:18SPEAKER_00: What changed between those two versions?
- 3:21SPEAKER_01: Between the two versions,
- 3:23SPEAKER_01: we changed,
- 3:24SPEAKER_01: we added a new modality,
- 3:26SPEAKER_01: what we call a new modality in the second part of the dataset.
- 3:30SPEAKER_01: So the first dataset is composed of very high resolution images, parallel images.
- 3:36SPEAKER_01: that have been acquired by IGN.
- 3:40SPEAKER_01: So it's a 20 pixel ground level pixel with annotation made by hand and that have been verified.
- 3:48SPEAKER_01: So it's very high quality that we need to try and supervise model.
- 3:52SPEAKER_01: And on the second dataset,
- 3:53SPEAKER_01: we also add for each of the patch from these images,
- 3:57SPEAKER_01: Sentinel-2 time series from the same zone,
- 4:02SPEAKER_01: in order to bring more information about,
- 4:05SPEAKER_01: for instance,
- 4:05SPEAKER_01: phenology of vegetation,
- 4:07SPEAKER_01: or to be able to distinguish
- 4:09SPEAKER_01: some different classes that you can have between,
- 4:11SPEAKER_01: for example,
- 4:11SPEAKER_01: bare soil and agricultural,
- 4:13SPEAKER_01: because the behavior will be the same during the year.
- 4:17SPEAKER_01: And in fact,
- 4:17SPEAKER_01: the zones we provide are even wider than the small hourly zone in order to also
- 4:24SPEAKER_01: have context information.
- 4:25SPEAKER_01: Maybe you can know that if around you have a can of sea,
- 4:28SPEAKER_01: you can think that this is more sane than something else.
- 4:33SPEAKER_00: Right, so a high-resolution snapshot and then a time series at low resolution to...
- 4:38SPEAKER_00: Understand temporal signatures.
- 4:41SPEAKER_00: Exactly.
- 4:42SPEAKER_00: They're interesting.
- 4:42SPEAKER_00: And how do you exploit both of those image sequences in a model?
- 4:50SPEAKER_01: So we propose the baseline in this case to do this kind of fusion of information.
- 4:56SPEAKER_01: So we have first kind of classical units,
- 5:00SPEAKER_01: segmentation model for dealing with the high resolution IR images.
- 5:05SPEAKER_01: And then we use what's called a UTA.
- 5:08SPEAKER_01: So it's kind of units that are modified in order to have a temporal attention
- 5:13SPEAKER_01: system in order to compress and in the clever way,
- 5:17SPEAKER_01: the temporal information from the time series.
- 5:20SPEAKER_01: And we inject this information at different points in the main branch of the unit.
- 5:26SPEAKER_01: And we supervised the both branches,
- 5:28SPEAKER_01: so the one with the Arial part and the time series part,
- 5:33SPEAKER_01: with the high resolution annotation we have provided.
- 5:38SPEAKER_01: So that's the baseline we proposed.
- 5:40SPEAKER_01: And in the second challenge,
- 5:41SPEAKER_01: people try to do different kind of fusion elements,
- 5:45SPEAKER_01: try to remove some,
- 5:47SPEAKER_01: for instance,
- 5:48SPEAKER_01: cloudy images or try to only use very few images and also try to calibrate the
- 5:54SPEAKER_01: images based on the time series and everything.
- 5:57SPEAKER_01: A lot of things have been tried.
- 6:00SPEAKER_00: Fantastic.
- 6:01SPEAKER_00: And how do people do with their approaches versus the baseline?
- 6:08SPEAKER_01: The people bring,
- 6:09SPEAKER_01: I think if I remember well,
- 6:10SPEAKER_01: it was like something like almost 10 points in mean average precision in total.
- 6:16SPEAKER_01: So it was interesting things, and we will try to use them as much as possible.
- 6:22SPEAKER_01: In the second challenge,
- 6:23SPEAKER_01: we also add an extra thing,
- 6:25SPEAKER_01: the fact that we require user,
- 6:29SPEAKER_01: sorry,
- 6:29SPEAKER_01: participant,
- 6:30SPEAKER_01: to not use too much computer program.
- 6:33SPEAKER_01: Because, in fact, in the first challenge, we realized that some of the best methods were very costly.
- 6:39SPEAKER_01: And as we need to be able to provide data at a country scale,
- 6:45SPEAKER_01: so like a very huge amount of data,
- 6:49SPEAKER_01: we need to have efficient solutions.
- 6:52SPEAKER_01: So that's also one of the things we had in the contest.
- 6:55SPEAKER_00: Interesting.
- 6:55SPEAKER_00: And how did you specify the limitation there, was it?
- 6:58SPEAKER_01: We specified the limitation like,
- 7:01SPEAKER_01: on one review to have something like no more than twice the inference time from the
- 7:07SPEAKER_01: baseline model.
- 7:09SPEAKER_00: Right, so okay.
- 7:10SPEAKER_00: You set the hardware and said this is the expectations on how quickly it should run.
- 7:14SPEAKER_00: Okay, very interesting.
- 7:16SPEAKER_00: Okay.
- 7:16SPEAKER_00: I'm curious,
- 7:17SPEAKER_00: did you do any kind of investigation to see what was the most significant features
- 7:21SPEAKER_00: for particular classes?
- 7:22SPEAKER_00: Because you could imagine something like urban areas wouldn't actually matter to
- 7:26SPEAKER_00: have the time series,
- 7:27SPEAKER_00: but other like vegetation might.
- 7:30SPEAKER_01: So we remember that the vegetation learned a lot about the time series, but also the urban area.
- 7:38SPEAKER_01: We were surprised about that.
- 7:39SPEAKER_01: For instance, building by using Sentinel-2 time series get three points EOE improvement.
- 7:46SPEAKER_01: So the classes that have been the most improved are more the bare soil class,
- 7:51SPEAKER_01: in fact,
- 7:52SPEAKER_01: which should be more easily distinguished from agricultural land,
- 7:55SPEAKER_01: for instance.
- 7:57SPEAKER_01: And also vineyards have been also quite improved by this multimodal models.
- 8:03SPEAKER_00: Very interesting.
- 8:03SPEAKER_00: So there's some confused classes and having the time series actually helps to separate those two.
- 8:09SPEAKER_00: Okay, yeah, that makes sense.
- 8:12SPEAKER_00: And I understand there's also a third stage in the dataset and competition?
- 8:19SPEAKER_01: So, not in the competition, but in fact, we, so, this, providing this, no, sorry, I'll start again.
- 8:29SPEAKER_01: Organizing the competition forces us to have very clear baseline,
- 8:33SPEAKER_01: to have something properly visible,
- 8:35SPEAKER_01: and to be able to open both the dataset,
- 8:38SPEAKER_01: but also the code to be able to run and train model.
- 8:43SPEAKER_01: And then we also decided to publish in an AI conference in NeurIPS last year this
- 8:49SPEAKER_01: dataset in order to encourage academic people to work on our problems,
- 8:55SPEAKER_01: on our issue we have at IGN.
- 8:57SPEAKER_01: And for now, we are trying to think about creating a new version of the data sets.
- 9:03SPEAKER_01: Yes, we have three times more images with very high resolution.
- 9:09SPEAKER_01: So for now,
- 9:09SPEAKER_01: we have something like 800 kilometers squares annotated,
- 9:15SPEAKER_01: and we are aiming at 2,500 kilometers.
- 9:20SPEAKER_01: annotating and we will also add more modality or extra data so like sentinel 2
- 9:28SPEAKER_01: information but also sentinel 1 data on the same area right so i understand
- 9:34SPEAKER_00: expanding the modalities the number of sites presumably yes and the intuition is
- 9:39SPEAKER_00: that just with a larger data set you'll get better better models and better
- 9:43SPEAKER_01: performance exactly yeah what would you expect
- 9:46SPEAKER_01: And also,
- 9:47SPEAKER_01: and so,
- 9:47SPEAKER_01: yeah,
- 9:48SPEAKER_01: and we also released,
- 9:50SPEAKER_01: after the challenge,
- 9:51SPEAKER_01: we also released the baseline model with the weight so people can use them in order
- 9:55SPEAKER_01: to do their own things.
- 9:59SPEAKER_01: So they can fine tune them,
- 10:00SPEAKER_01: they can use them as evaluation,
- 10:02SPEAKER_01: as our initialization for training or some different things.
- 10:08SPEAKER_01: And we will do the same with the next version of the dataset for sure.
- 10:13SPEAKER_00: Fantastic.
- 10:14SPEAKER_00: I'd like to just talk a little bit about the process of creating these data sets.
- 10:17SPEAKER_00: Do you mind just at a high level saying how you go about it,
- 10:20SPEAKER_00: how you select where to sample data,
- 10:22SPEAKER_00: that kind of thing?
- 10:25SPEAKER_01: So, the idea was to have very diverse data sets, so we tried to sample data.
- 10:29SPEAKER_01: So,
- 10:30SPEAKER_01: it's only on the metropolitan areas,
- 10:34SPEAKER_01: so in parts of France and Europe,
- 10:36SPEAKER_01: because France also has a lot of different highlands in the Caribbean,
- 10:42SPEAKER_01: for instance,
- 10:42SPEAKER_01: or in the Indian Ocean.
- 10:44SPEAKER_01: And we will work on that later, for sure.
- 10:47SPEAKER_01: uh and uh then what we so we select a zone with different uh landscape with big
- 10:53SPEAKER_01: cities small cities uh with mountains and things like that and we also iterate on
- 10:58SPEAKER_01: that and for instance the next version of the set will have more diverse uh which
- 11:03SPEAKER_01: one again and we also this and then we uh make people annotating the different
- 11:10SPEAKER_01: classes the lan
- 11:11SPEAKER_01: covers after class.
- 11:12SPEAKER_01: And we also have a validation system that means that then a second person,
- 11:19SPEAKER_01: sometimes not always,
- 11:19SPEAKER_01: but sometimes at random,
- 11:21SPEAKER_01: will check if annotations are correct or not.
- 11:24SPEAKER_01: And if this is not the case,
- 11:26SPEAKER_01: we will go again on the annotation,
- 11:28SPEAKER_01: not to improve the quality of it,
- 11:29SPEAKER_01: because we aim quality of 95% accuracy on the human annotation.
- 11:40SPEAKER_01: It was very time-consuming and,
- 11:44SPEAKER_01: in fact,
- 11:44SPEAKER_01: costly,
- 11:45SPEAKER_01: but it's worth it because we were able to get a good AI model then.
- 11:51SPEAKER_00: Two questions.
- 11:52SPEAKER_00: On the selection,
- 11:54SPEAKER_00: did you go for an equal weighting on all the different classes,
- 11:57SPEAKER_00: or did you say,
- 11:57SPEAKER_00: this class is really hard,
- 12:00SPEAKER_00: or this location is hard,
- 12:00SPEAKER_00: let's have more data for that?
- 12:02SPEAKER_00: What's the strategy there?
- 12:04SPEAKER_01: At the beginning, we didn't do it like that.
- 12:06SPEAKER_01: We were more looking at the different regions of France because,
- 12:11SPEAKER_01: as we are a kind of mapping agency,
- 12:14SPEAKER_01: we already know where are the different,
- 12:16SPEAKER_01: let's say,
- 12:17SPEAKER_01: bioclimatic regions in France,
- 12:19SPEAKER_01: where we can find small cities,
- 12:21SPEAKER_01: more dense urban areas.
- 12:24SPEAKER_01: And then,
- 12:26SPEAKER_01: on the next version of the dataset,
- 12:27SPEAKER_01: we will focus specifically on some classes that are harder for the model,
- 12:31SPEAKER_01: yes.
- 12:33SPEAKER_01: We are able to say.
- 12:34SPEAKER_01: And for instance,
- 12:35SPEAKER_01: on the second challenge from Flare,
- 12:38SPEAKER_01: we decided to emphasize more about the validation because it was one of the hardest
- 12:43SPEAKER_01: things to distinguish.
- 12:44SPEAKER_01: And for instance,
- 12:45SPEAKER_01: we decided to put more validation class in the test set in order to select model
- 12:50SPEAKER_01: that will be able on this class.
- 12:51SPEAKER_00: Okay.
- 12:52SPEAKER_00: Yeah, it makes sense.
- 12:53SPEAKER_00: And in terms of determining which classes are hard,
- 12:55SPEAKER_00: did you see that it was always the same for the human and the model,
- 12:58SPEAKER_00: which classes were the hardest?
- 13:00SPEAKER_00: Or there are some classes that are easy for a model because it has a time series,
- 13:03SPEAKER_00: but hard for humans,
- 13:04SPEAKER_00: eh?
- 13:09SPEAKER_01: Try to think about it.
- 13:10SPEAKER_01: I think one of the classes that is...
- 13:11SPEAKER_01: So some of the hardest classes for machines are the classes like Brushwood and Plumland,
- 13:20SPEAKER_01: which are maybe easier for humans.
- 13:23SPEAKER_01: Also,
- 13:24SPEAKER_01: I think it also depends on which kind of photo interpreter you are,
- 13:29SPEAKER_01: because at Hygiene,
- 13:30SPEAKER_01: very high experts are able to distinguish different kinds of trees,
- 13:34SPEAKER_01: which is still a very hard task for machines,
- 13:37SPEAKER_01: for instance.
- 13:39SPEAKER_01: But for land cover,
- 13:40SPEAKER_01: I think it will be more about,
- 13:41SPEAKER_01: for machines,
- 13:42SPEAKER_01: especially about the difference between bare soil,
- 13:45SPEAKER_01: agricultural land,
- 13:47SPEAKER_01: things like that.
- 13:48SPEAKER_00: Right, okay.
- 13:49SPEAKER_00: Because presumably a human has additional context.
- 13:52SPEAKER_00: Yes, exactly.
- 13:53SPEAKER_00: They have a map.
- 13:54SPEAKER_00: Okay, that makes a lot of sense.
- 13:56SPEAKER_00: You mentioned about adding more modalities.
- 13:58SPEAKER_00: I think you mentioned Sentinel-1.
- 14:01SPEAKER_00: Presumably that's widely available.
- 14:03SPEAKER_00: But is there kind of a wish list of modalities that you would like to include in the future?
- 14:09SPEAKER_01: We also think about adding what we can call kind of spot images,
- 14:15SPEAKER_01: also satellite images,
- 14:16SPEAKER_01: but at one meter per pixel.
- 14:19SPEAKER_01: So it's like an intermediate between a sentinel at 10 meters and aerial images at
- 14:24SPEAKER_01: 20 centimeters resolution.
- 14:27SPEAKER_01: We also think about adding LiDAR information,
- 14:33SPEAKER_01: because for now,
- 14:35SPEAKER_01: so iGen,
- 14:35SPEAKER_01: we're also working on LiDAR acquisition for different land descriptions,
- 14:42SPEAKER_01: for instance,
- 14:42SPEAKER_01: like forestry mainly,
- 14:44SPEAKER_01: but not only,
- 14:44SPEAKER_01: also flow estimation.
- 14:48SPEAKER_01: And so we think that adding this kind of information,
- 14:50SPEAKER_01: like LiDAR,
- 14:50SPEAKER_01: which is a high-resolution LiDAR,
- 14:52SPEAKER_01: like 10 points per meter square,
- 14:55SPEAKER_01: could be interesting to add on the same region and to see if we are able to learn
- 14:59SPEAKER_01: models that can deal with different modality or try to leverage the different
- 15:04SPEAKER_01: information in the different modality.
- 15:06SPEAKER_00: That's a good point to segue back to the model side a bit.
- 15:10SPEAKER_00: as you add modalities you add number of channels does that indicate that different
- 15:14SPEAKER_00: kinds of models would be successful as you kind of increase the dimension
- 15:18SPEAKER_01: dimensionality in that way uh what we observe is the fact that for instance on a
- 15:24SPEAKER_01: let's say on the simplest case uh so in our case for the hourly information we have
- 15:29SPEAKER_01: of what we call five channel we have rgb infrared and also an elevation model that
- 15:36SPEAKER_01: we compute from photogrammetry based on the our images
- 15:40SPEAKER_01: And we have shown that adding the infrared increases the performance and also
- 15:46SPEAKER_01: adding elevation can increase performance.
- 15:49SPEAKER_01: So our clue is the fact that adding more modality,
- 15:52SPEAKER_01: if you do it on the right way,
- 15:54SPEAKER_01: can bring better performance.
- 15:56SPEAKER_01: Okay.
- 15:57SPEAKER_00: So that's even without changing the architecture of the model.
- 16:00SPEAKER_00: That is just showing that these features have useful information.
- 16:04SPEAKER_01: But I think for time series on the Flare dataset baseline we released,
- 16:10SPEAKER_01: we didn't add the time series as an extra channel,
- 16:15SPEAKER_01: but more like context information that will be processed by an auxiliary branch and
- 16:21SPEAKER_01: then injected in the feature space of the main model.
- 16:26SPEAKER_01: We did more like that.
- 16:29SPEAKER_00: You mentioned briefly to me offline that you saw the benefits of using transformer models.
- 16:35SPEAKER_01: Yeah,
- 16:36SPEAKER_01: on the first challenge,
- 16:37SPEAKER_01: we saw that the best performance was obtained mainly based on a vision transformer,
- 16:46SPEAKER_01: which like we can see in the other field of computer vision,
- 16:49SPEAKER_01: like in medical case,
- 16:50SPEAKER_01: natural images.
- 16:51SPEAKER_01: So it was not a surprise,
- 16:53SPEAKER_01: but it's interesting to see that some people were able to train this model on our
- 16:56SPEAKER_01: data set and to get very good results.
- 16:58SPEAKER_01: So now we're trying to incorporate that in our own pipeline.
- 17:03SPEAKER_01: and also to release a pre-trained model based on Vision Transformer.
- 17:08SPEAKER_01: One of the main problems is the fact that it's sometimes more costly.
- 17:12SPEAKER_01: And as I mentioned before,
- 17:13SPEAKER_01: as we are trying to produce large-scale data,
- 17:17SPEAKER_01: it could be a very clear problem for us.
- 17:21SPEAKER_01: And on the other hand,
- 17:22SPEAKER_01: some of the problems are the fact that Vision Transformer are based on patches from
- 17:26SPEAKER_01: the images.
- 17:27SPEAKER_01: Sometimes you can have kind of border effects on the patches.
- 17:31SPEAKER_01: That could be a problem for us also, and we try to resolve.
- 17:36SPEAKER_01: So maybe more model-like hierarchical vision transformer could be a solution.
- 17:41SPEAKER_00: Fascinating stuff.
- 17:43SPEAKER_00: Well, thanks so much for this introduction to the dataset series and the accompanying modeling work.
- 17:49SPEAKER_00: You mentioned that the third dataset will be out.
- 17:51SPEAKER_00: Do you have a date in mind for that?
- 17:55SPEAKER_01: Not yet, but I think it will be in 2025.
- 17:59SPEAKER_00: Okay, so early next year, hopefully.
- 18:02SPEAKER_00: Good stuff.
- 18:03SPEAKER_00: If people want to follow along,
- 18:05SPEAKER_00: updates,
- 18:05SPEAKER_00: both yourself and the activities at IGN,
- 18:08SPEAKER_00: which is the best platform?
- 18:11SPEAKER_01: Uh, it depends for the, um, more like the innovation part and all the AI model.
- 18:17SPEAKER_01: We are hosting a few data set and models.
- 18:21SPEAKER_01: Um,
- 18:21SPEAKER_01: again,
- 18:21SPEAKER_01: the face,
- 18:22SPEAKER_01: we have a page from IGN that then can share,
- 18:25SPEAKER_01: uh,
- 18:26SPEAKER_01: and for all the pollution parts,
- 18:29SPEAKER_01: uh,
- 18:29SPEAKER_01: it's the IGN website,
- 18:30SPEAKER_01: uh,
- 18:31SPEAKER_01: where we propose all the information I mentioned by about biodiversity,
- 18:35SPEAKER_01: about,
- 18:36SPEAKER_01: uh,
- 18:36SPEAKER_01: forestry,
- 18:37SPEAKER_01: uh,
- 18:37SPEAKER_01: agricultural information.
- 18:39SPEAKER_01: that are produced by all the institutes.
- 18:42SPEAKER_00: Okay, fantastic.
- 18:43SPEAKER_00: I'll put the links in the show notes.
- 18:45SPEAKER_00: Once again, Nicolas, thanks for joining me and I wish you all the best for the future.
- 18:49SPEAKER_00: Thank you.
- 18:50SPEAKER_00: You too.
- 18:51SPEAKER_00: Bye.
- 18:52SPEAKER_00: Bye.