Latest / EDGE AI POD / Revolutionizing Weather Forecasting with Acoustic Smart Technology

Transcript

Revolutionizing Weather Forecasting with Acoustic Smart Technology · EDGE AI POD

• 0:09
  Speaker 1: Okay, now we are going to the second presentation, and this presentation is titled Meet the Acoustic Smart Weather Station Aurora.
• 0:21
  Speaker 1: So the presenter is Jonah Bessings from CSCM.
• 0:27
  Speaker 1: He's an R&D engineer at the Swiss Center for Electronic and Micro Technology in Switzerland, so he's going to talk about the Aurora project, which I'm happy to be a little bit part of.
• 0:43
  Speaker 2: So, jonah, the floor is yours so thank you, thomas, for the nice introduction.
• 0:56
  Speaker 2: So I'm very happy to be here today to present to you Aurora, which is not a conventional weather station.
• 1:08
  Speaker 2: It's a weather station based on acoustics, and this all started with the TinyML Challenge 2022, which was co-organized by the TinyML Foundation to come up with a weather station that has no mechanical moving parts.
• 1:27
  Speaker 2: Now, why would we like to do that?
• 1:30
  Speaker 2: So there is a need for accurate and reliable weather information, both in remote areas, but also in resource-limited scenarios, and existing solutions, like the mechanical weather stations we have today, are expensive and, most importantly, they're prone to failure because they have mechanical moving parts.
• 1:54
  Speaker 2: We have one storm and all the parts are broken, and it's difficult to keep it up and running if it's far away from us.
• 2:02
  Speaker 2: Now, a dense installation of low cost and easy to install weather stations could improve this and could improve weather forecasting both on a spatial level, but also on a temporal level, and for these reasons, we created Aurora.
• 2:20
  Speaker 2: Aurora is a weather station based on acoustics that does wind and rain classification, it is maintenance free, it is easy to install and the inference happens entirely on the device.
• 2:32
  Speaker 2: With Aurora, we won the TinyML Challenge 2022, and since we were so much motivated, we are now co-organizing the follow-up challenges 2023 and 2024, to push the boundaries of this station further.
• 2:52
  Speaker 2: Now, today, I will first start giving you an overview of the design of Aurora, then I will move on to the data acquisition framework that we developed, followed by the machine learning process.
• 3:03
  Speaker 2: As well as giving some results, of course, the machine learning process.
• 3:05
  Speaker 2: As well as giving some results, of course, the most interesting part.
• 3:07
  Speaker 2: Then a demo where we show it in practice, and a conclusion and next steps.
• 3:13
  Speaker 2: So how does it work under the hood?
• 3:15
  Speaker 2: So we have a tiny ML board.
• 3:18
  Speaker 2: As you can see on the slide, it takes, with a microphone, information from nature, so it listens to the environment and, based on this, it runs a tiny machine learning model that then tells us what is the wind intensity, what is the rain intensity.
• 3:32
  Speaker 2: We also add another sensor, which is an environmental sensor, tracking temperature, pressure, humidity, just to give us more information about the weather.
• 3:42
  Speaker 2: However, this is not used inside the model, as we didn't see any benefit of giving this information to get wind and rain intensity.
• 3:52
  Speaker 2: What are the system components?
• 3:53
  Speaker 2: So we have a bill of material below 100 euros just to build a single prototype which consists of an MCU.
• 4:01
  Speaker 2: You see here an ESP32.
• 4:03
  Speaker 2: We use mainly an ESP32 for the data acquisition because in order to get data out of the device, we need a high data rate wireless communication.
• 4:13
  Speaker 2: The ESP32 has Wi-Fi, so it was an ideal candidate.
• 4:17
  Speaker 2: It's not the most efficient solution in terms of energy and that's why we are working on the second version, with a much smaller MCU just for doing the inference, but I'll talk about that later.
• 4:31
  Speaker 2: We also have an environmental sensor, we have a microphone, we have a Wi-Fi antenna, a 3D printed casing, a battery and some cables, which brings us to around 80 euros.
• 4:45
  Speaker 2: And then, optionally, we also have a solar panel and a mechanical weather station that we are using as a ground route, because this morning we heard that a real data set is still the best data set.
• 4:56
  Speaker 2: So we are really focusing on a real data set Now.
• 5:01
  Speaker 2: In order to get a real data set, you have to build infrastructure, and since for us it was very important to build get a real data set, you have to build infrastructure.
• 5:09
  Speaker 2: And since for us it was very important to build a real-world data set, we connected a mechanical weather station as you can see on the slide to our device and we gathered both the audio information but also the data from the mechanical weather station, so we send it via Wi-Fi to a central server where we do all the storage and, with this mechanical weather station, so we have a labeled data set.
• 5:32
  Speaker 2: Now what's inside the data set?
• 5:34
  Speaker 2: So the data set contains both audio information from the microphone 16 kilohertz sampling rate at 18 bits precision, and we chop it in samples of one second.
• 5:44
  Speaker 2: Then we also have the information from the environmental sensor and, if connected, also the mechanical weather station.
• 5:53
  Speaker 2: With this we built a realistic data set installed at the premises of our company, cscm, in Neuchatel in Switzerland, and we let it run now for almost one year, so since August 2023.
• 6:07
  Speaker 2: It's really a real-world data set.
• 6:10
  Speaker 2: So it's on the roof, it captures birds machinery that's active on the roof and which is activated in wintertime but not in summertime.
• 6:18
  Speaker 2: So really a real-world data set.
• 6:22
  Speaker 2: So we made this data set open access because we believe in the power of a community where we can work together on problems that we have today, and so we share it with the participants of the TinyML Challenge 23 and 24.
• 6:38
  Speaker 2: So in total we have more than 200 gigabytes of data, corresponding to more than 3,000 hours of training data.
• 6:52
  Speaker 2: But this data set does not come without any challenges, which we faced along the way.
• 6:58
  Speaker 2: So we are using a microphone and on the device, from the moment that we have the first droplet that's hitting the hood, we can detect that there is rain falling.
• 7:09
  Speaker 2: However, with a mechanical weather station, we usually have this tipping bucket that.
• 7:14
  Speaker 2: So, basically, when the rain falls it will fill the bucket and once it flips we get a signal and with this signal we can detect that rain has been flown and we can also know the quantity, because we know the quantity of the bucket.
• 7:32
  Speaker 2: Now, in order to know that it rains, it first needs to flip this bucket to get this signal.
• 7:39
  Speaker 2: However, with the acoustic solution, we already knew it from the first droplet.
• 7:44
  Speaker 2: So there is a discrepancy there which leads to uncertainty into the data set, and because of this we introduced safety windows.
• 7:54
  Speaker 2: So when it starts to rain, the beginning of this rainy period we have to remove, otherwise we made too much errors in the machine learning model.
• 8:09
  Speaker 2: Okay, so for the development process, this is quite traditional.
• 8:13
  Speaker 2: So we do all the training on a server.
• 8:16
  Speaker 2: So we do the training, the quantization, as well as the conversion to TensorFlow Lite Micro, which we then deploy on the TinyML board where we do the sensor acquisition, which runs on a real-time operating system that we developed at CSCM, which is small and is very portable to many different microcontrollers.
• 8:43
  Speaker 2: For the machine learning model we are using a convolutional neural network with around 27,000 parameters.
• 8:47
  Speaker 2: We have both binary and ternary variants.
• 8:51
  Speaker 2: So today we do classification into a binary level and a ternary level.
• 8:55
  Speaker 2: However, we are looking into more classes, which is work in progress.
• 9:01
  Speaker 2: We have a CNN with around four hidden layers and we designed this model from scratch.
• 9:07
  Speaker 2: So we didn't use any pre-trained models because we found out that it didn't work really well in our use case.
• 9:13
  Speaker 2: So we start really from one layer, two layers, three layers, until we faced overfitting of the model.
• 9:20
  Speaker 2: And in this way we did the design.
• 9:22
  Speaker 2: We test both with audio preprocessing and without preprocessing and what we found out is that for RAIN it's more higher frequency.
• 9:31
  Speaker 2: So there we apply a high pass filter prior to the data being fed to the machine learning model.
• 9:41
  Speaker 2: So for the results so on the left you can see the model and then for the results of the rain, two classes we have 97% of F1 scores, so we use the F1 score, which is the harmonic mean of precision and recall, because we have a real-world data set.
• 10:00
  Speaker 2: So it's highly imbalanced because usually, luckily, it's less rainy than we think, and so, due to this, we use the F1 score to assess the performance of the model.
• 10:13
  Speaker 2: For wind, we have 85% of F1 score.
• 10:18
  Speaker 2: So there we see that it's a little bit less performant than rain.
• 10:22
  Speaker 2: Rain is very clear sounds on the hood.
• 10:24
  Speaker 2: For wind it's a bit less controllable.
• 10:27
  Speaker 2: So also we see that results are a bit less good For the ternary classification.
• 10:33
  Speaker 2: For rain, we have around 84% of F1 score, and for wind, we have around 67% of F1 score.
• 10:41
  Speaker 2: For rain, you can see that most of the errors are made between the middle and the high class.
• 10:48
  Speaker 2: So it can well distinguish between no rain and rain, but once it starts to rain, we see that a little bit more difficulty in determining exactly how intense rain is.
• 11:02
  Speaker 2: So, in order to put this model onto an embedded device, we do also quantization.
• 11:06
  Speaker 2: So we use quantization-aware training, followed by integer 8 quantization, and what we saw is that we have maximally 2% of accuracy loss by doing so.
• 11:17
  Speaker 2: However, this leads us to a flash size of less than 30 kilobytes and a RAM size less than 200 kilobytes.
• 11:25
  Speaker 2: So let me show you a short demo.
• 11:30
  Speaker 2: So here you can see our device on the left and on the right.
• 11:38
  Speaker 2: You can see both results, from the binary model on the top as well as the ternary model at the bottom.
• 11:41
  Speaker 2: So let's start with the video.
• 11:43
  Speaker 2: So first we will do some rain measurements and then for wind so here you can see that no rain is detected.
• 11:52
  Speaker 2: So the confidence level of the class zero is selected.
• 11:57
  Speaker 2: Then it starts to rain, so we put water on the hood and then for the binary one it says that it's raining, whereas for the ternary one it says that it's heavily raining, and then after a while we reduce the intensity of the rain that is falling, and then we will see that on the top it still says that it's raining.
• 12:20
  Speaker 2: However, at the bottom we see that class one has the highest confidence level, and then once we really stop the rain, then it will say that there's no rain being detected.
• 12:35
  Speaker 2: Then we do the same for wind.
• 12:40
  Speaker 2: So for wind, we start also without any wind.
• 12:44
  Speaker 2: Then we start to generate wind, so in the beginning a bit stronger.
• 12:50
  Speaker 2: So we have yeah, it detects there is wind, even heavy wind, and then we will gradually lower the intensity, so now the intensity is lower, and so here you can see that the middle class is detected.
• 13:21
  Speaker 2: Good, this brings me to the conclusion.
• 13:25
  Speaker 2: So Aurora is a low, low-power, low-cost and easy-to-install weather station without any mechanical moving parts.
• 13:31
  Speaker 2: By listening to the nature and the surrounding environment, we are able to estimate rain and wind intensity.
• 13:38
  Speaker 2: Through tiny machine learning at the edge, and by open-sourcing the data set and co-organizing the challenges in 2023 and 2024, we really aim to push the frontier of what can we do, and we're really hoping for positive change.
• 13:57
  Speaker 2: So, but this is not a project which has been finished.
• 14:02
  Speaker 2: It's really in progress.
• 14:04
  Speaker 2: So we're also working on it ourselves.
• 14:06
  Speaker 2: So we are currently developing a PCB for the second generation of Aurora.
• 14:10
  Speaker 2: Today it's more an assembly of development kits.
• 14:16
  Speaker 2: Now we will go for a custom PCB where we will have more than one microphone.
• 14:20
  Speaker 2: We will integrate four microphones to further improve the performance, as well as adding other wireless communications like LoRa or BLE, to do the commissioning but also to send the result from the inference back to the network.
• 14:39
  Speaker 2: And we will also specialize on two operation modes one on the acquisition mode, which will be based on an ESP32, but then also on the inference mode, which will be based on an ESP32, but then also on the inference mode, which will be based on an STM32U5, to really improve the energy consumption.
• 14:56
  Speaker 2: We also aim to improve the quality of the ground.
• 14:59
  Speaker 2: Truth, as I explained in the presentation, the quality is not the best.
• 15:03
  Speaker 2: It's a good starting point.
• 15:05
  Speaker 2: We are also looking into how we can improve this and then finally also look into wind direction.
• 15:11
  Speaker 2: So today we can only do wind intensity, but of course wind direction is the next step and for this also the multiple microphones can help.
• 15:19
  Speaker 2: So I would like to invite you, if you want to collaborate or share ideas, please feel free to contact me, and very thank you for your attention.
• 15:38
  Speaker 1: OK, thank you so much.
• 15:39
  Speaker 1: So now we are going to Q&A.
• 15:42
  Speaker 1: If you have some questions for Jonah, you can.
• 15:46
  Speaker 1: Yes.
• 15:54
  Speaker 3: Thank you very much for the presentation.
• 15:56
  Speaker 3: So my question actually is more about the motivation of the work.
• 16:02
  Speaker 3: So, if I understood it in the beginning, estimation of the weather forecasting is something that we're looking at it on the long run.
• 16:14
  Speaker 3: Is there any specifications on what we need to predict from your model that could actually, at the end, help the weather forecast?
• 16:27
  Speaker 3: Do we have any specifications?
• 16:30
  Speaker 2: So we're in contact with a company working on weather forecasting, and so they have a clear view on what they would like to have.
• 16:40
  Speaker 2: What they told us is that actually, it's better to have multiple ground stations, which are less accurate, instead of having one that is very accurate, and this especially to guarantee a good spatial resolution of the of the weather forecasting.
• 17:13
  Speaker 4: Thank you for the great presentation and it's really exciting to see this evolve over time.
• 17:18
  Speaker 4: Two years ago it was just an idea, but now it's real.
• 17:21
  Speaker 4: Have you thought or I'm not sure, actually if you have already made the data set available the one that you have already because one of the biggest challenges we had when we launched this challenge was actually people didn't know how to gather the data.
• 17:35
  Speaker 4: There's no open data set, so have you thought about opening up the data for the people to use?
• 17:41
  Speaker 2: Yes, so the data set is open.
• 17:43
  Speaker 2: We have a dedicated website for that.
• 17:46
  Speaker 2: The data set is open.
• 17:48
  Speaker 2: We have a dedicated website for that.
• 17:49
  Speaker 2: So here on this website all the data can be found.
• 17:59
  Speaker 2: It's the label data set, so it's also the data set we use to generate this.
• 18:05
  Speaker 2: It's a real world data set so you can generate this, and it's a real-world data set so you can easily split it into training, validation and test set.
• 18:10
  Speaker 2: And so we're also looking into deploying the device in not just one location, as it is now, but also deploying in multiple locations, because weather can differ from location to location, to see what we can achieve more with that.
• 18:28
  Speaker 4: Thank you.
• 18:36
  Speaker 5: Hi.
• 18:36
  Speaker 5: Yeah, thanks for the presentation.
• 18:37
  Speaker 5: This is really cool stuff, I think.
• 18:39
  Speaker 5: Following on from the first question about the motivation, so you explained that this is like a low-cost solution that we can imagine sort of tiling the large areas using these devices.
• 18:50
  Speaker 5: I'm just curious just to get a sense for how much would it cost sort of the equivalent like tiling, of placing these devices to swap it out for something like something that measures wind speed and rainfall?
• 19:04
  Speaker 5: Is there a significant cost difference between this device and those sort of more classical sensor solutions?
• 19:10
  Speaker 2: So today I can show you some results on the mechanical station we use today, although it's a lot of just plastic and metal.
• 19:21
  Speaker 2: It's 80 euros for a mechanical weather station, which is today the same cost as we have for a single prototype, which is not.
• 19:27
  Speaker 2: It's really a collection of development kits that we did just to get something, but we are convinced that we can drastically reduce the cost to produce such a device, as well as the size, because today we still have Okay, we have this device.
• 19:48
  Speaker 2: We believe that if we just only focus on the inference on the device, a MEMS microphone is very small, so we believe we can make it much smaller as well, which also makes it easier to install it.
• 20:05
  Speaker 5: Yeah, so just to follow up on that, so for like the equivalent resolution, so you were measuring rainfall, heavy light and wind or no wind sort of heavy light.
• 20:16
  Speaker 5: Do you have an idea of what sensors just to measure those without machine learning, would cost?
• 20:22
  Speaker 5: Just as a comparison?
• 20:26
  Speaker 2: To have the same performance as a mechanical one?
• 20:29
  Speaker 2: Is that?
• 20:29
  Speaker 5: what you mean.
• 20:30
  Speaker 5: Yeah, yeah.
• 20:30
  Speaker 2: Yeah, it's difficult to say because today we are not yet there, so I know there exists some solutions as well based on ultrasound.
• 20:41
  Speaker 2: They are more costly and we really aim to do something which is low-cost.
• 20:49
  Speaker 2: And this is really just on the acoustics, and we hope to get better than we have before, but let's see how far we can go.
• 21:01
  Speaker 1: Thank you very much for your presentation, Just curious how sensitive is it to snow?
• 21:13
  Speaker 2: Unfortunately, we didn't have snow in the Châtel last winter, but I believe that snow is usually rather it's not that abrupt as the rain that's ticking.
• 21:26
  Speaker 2: So today I don't believe we can detect snow, but it might be that by integrating a small light sensor on top of the hood, we could see that if we yeah, I think this could help to try to differentiate between snow and no snow, because snow will block it.
• 21:50
  Speaker 2: So then we will be sure that there is something on the hood.
• 21:54
  Speaker 2: Maybe it can also be a bird, of course, but if it's there for a long enough time, maybe we can detect snow as well.
• 22:01
  Speaker 1: Thank, you Another question.
• 22:14
  Speaker 3: Hi, thank you.
• 22:15
  Speaker 3: Just a short question.
• 22:16
  Speaker 3: Maybe it's early in your development, but how portable do you think the model is?
• 22:20
  Speaker 3: If you train the model on your building and you take it outside here, do you think it would be reliable?
• 22:28
  Speaker 2: That's what we are hoping for, but today we didn't test it yet.
• 22:32
  Speaker 2: So this is indeed also something we would like to test.
• 22:37
  Speaker 2: If we install it in multiple locations, we can train in one and evaluate in the others and try to see to what extent it can improve.
• 22:46
  Speaker 2: But ideally, yes, so it should be independent of background noise and really focus only on the rain and the wind.
• 22:57
  Speaker 1: Yeah, Any other.
• 23:00
  Speaker 1: Okay, just a quick comment from me.
• 23:04
  Speaker 1: I think when we were starting this project I think three years ago it was a matter of understanding the challenges, especially in developing countries.
• 23:16
  Speaker 1: Because, okay, in here it's easy to find weather stations or weather forecast close by, and if it breaks, breaks, it's fixed quite quickly.
• 23:28
  Speaker 1: So it's something that people just think it's normal.
• 23:30
  Speaker 1: But but in most developing countries you don't have the luxury to have weather stations and if you have one, when it breaks takes six months, one year, to be repaired.
• 23:42
  Speaker 1: So this becomes a very good option in most of the developing world and these kind of solutions are very important because it helps, especially in most countries.
• 23:58
  Speaker 1: They rely on farming, so agriculture.
• 24:01
  Speaker 1: So if you have good weather stations, if you have good weather forecasting, it helps the farmers, especially when they are growing their crops.
• 24:04
  Speaker 1: So if you have good weather stations, if you have good weather forecasting, it helps the farmers, especially when they are growing their crops.
• 24:08
  Speaker 1: So we are still running this challenge in 2024.
• 24:12
  Speaker 1: We still have some issues to improve, but I think the first step we are there, with the help of CSAM, we have this data set that is open, it's public, everyone can use it.
• 24:29
  Speaker 1: We hope that we can get it on some platforms.
• 24:30
  Speaker 1: So if you have a platform and would like to have this data set there, it will be nice, because now our participants, our students, can easily reuse the data for further development.
• 24:42
  Speaker 1: But also, if you have ideas on how we can improve on this problem statement, please be in touch and we are happy to have more collaboration with the community.
• 24:52
  Speaker 1: So thank you everyone, and now I think we can go for break.
• 24:58
  Speaker 2: Thank you.