Latest / EDGE AI POD / Revolutionizing Space Technology with Edge AI and Satellite Autonomy
Transcript
- 0:00Speaker 1: Hello friends, welcome to my channel.
- 0:01Speaker 1: Today we will learn how to make a beautiful and easy to make cake with a simple recipe, without any ingredients.
- 0:13Speaker 1: Let's start with the ingredients 1 cup of flour, 1 cup of water, 1 cup of water.
- 0:29Speaker 1: Okay, fantastic, we are live and I'm here with David.
- 0:50Speaker 1: Danilo Pau is still connecting from ST, so as soon as he connects we'll bring him on board, but I really appreciate David coming on here from Eubotica.
- 1:02Speaker 1: David, do you want to introduce yourself?
- 1:13Speaker 2: Yeah, I'm David Maloney.
- 1:13Speaker 2: I am Chief Scientist at Ubotica.
- 1:14Speaker 2: Previously I was co-founder and CTO at Movidius and in fact we still have, for all of our kind of space platforms, a strong relationship with Intel, movidius and our former team, and there are several other people shown in the slides who are also ex-Movidius, who are part of the team.
- 1:33Speaker 1: Yeah, I noticed.
- 1:34Speaker 1: Yeah, it's quite a few ex-Intel folks, kind of a Movidius group there, which is pretty cool.
- 1:40Speaker 1: I used to work with Intel Movidius chips back in the day, so that's kind of interesting.
- 1:45Speaker 1: Let me see if danilo I see he might be available.
- 1:48Speaker 1: Let's see danilo, are you there?
- 1:49Speaker 1: He's on mute.
- 1:53Speaker 1: Well, we'll leave him in there.
- 1:54Speaker 1: Um, and, you know, david doesn't have his camera on, but it's for some some lighting issues.
- 2:00Speaker 1: That's fine, but david, I appreciate you.
- 2:02Speaker 1: Uh, coming out today and giving us a talk.
- 2:04Speaker 1: I think we're're going to talk about edge AI scenarios and kind of tiny ML or edge AI scenarios in terms of imaging and satellite systems.
- 2:13Speaker 1: Right, is that kind?
- 2:13Speaker 2: of the.
- 2:14Speaker 2: That's correct, jay.
- 2:16Speaker 2: That's what we've been involved in for the past number of years, probably starting back with a project that we started in 2017 with the European Space Agency, and it's kind of step by step has led to what we're doing now and our current satellite that's up orbiting six, seven hundred kilometers above our heads right now.
- 2:39Speaker 1: So, yeah, Wow, all right, cool.
- 2:41Speaker 1: Well, yeah, I mean, why don't we just kind of get into it?
- 2:43Speaker 1: I think Danilo will join as able and then also we'll keep an eye on the Q&A that's happening.
- 2:50Speaker 1: So for folks that are watching this on YouTube or LinkedIn or Twitch or where else on StreamYard itself, you can ask questions as we go, as David kind of walks through kind of the scenarios that they're solving with AGI and, yeah, we'll kind of take it from there.
- 3:08Speaker 1: I think it sounds good.
- 3:09Speaker 1: Let me bring up the slides and I think maybe Danilo and I will exit for a little bit and give you the stage and you can kind of walk through the story and we reserve the right to kind of jump in every so often with some questions absolutely, please do.
- 3:28Speaker 2: why don't you go?
- 3:28Speaker 2: Keep it interactive, okay?
- 3:31Speaker 2: So, um, the presentation is titled there's no cloud in space and in fact the first ai application that we ran uh was on, you know, cloud detection and removal, but we'll get into that in some detail later on.
- 3:44Speaker 2: So it's more than just a pump and you know what.
- 3:48Speaker 2: What we're doing at the moment is.
- 3:50Speaker 2: We're focused on space ai, which is basically doing uh ai inference at source on satellites, rather than than um transferring all the data to to ground and doing on the ground, and there are good reasons for doing that that I'll get into.
- 4:07Speaker 2: The team I've mentioned is where a lot of us are ex-Movidius.
- 4:12Speaker 2: I was the former CTO at Movidius, john Mitchell is involved, john Burke also, who's ex-Movidius, and then a number of other colleagues who we've worked on and off with in different guises over the years.
- 4:27Speaker 2: But essentially our DNA is really in Edge AI.
- 4:32Speaker 2: We were some of the pioneers in the area and that's really the skill set that infuses our eubotica uses or Eubotica.
- 4:44Speaker 2: So I mean, if you look right now, you know the image on the left there is a hyperspectral image with about five, five meter grant sampling distance.
- 4:55Speaker 2: So it means essentially pixel is five meters on a side, but unlike conventional images, it's divided into about 150 different wavelengths rather than just RGB.
- 5:11Speaker 2: So you have lots and lots of different planes.
- 5:13Speaker 2: So things get very interesting in terms of doing inference.
- 5:17Speaker 2: But the interesting thing is that rather than just seeing a color for each pixel, you effectively see a spectrum, so that allows you to infer things about what you're looking at that go just beyond.
- 5:29Speaker 2: You know what shape is it and what does it look like, and you could tell if there's a fire, you can tell if there's a methane bloom, you can tell if there's an algal bloom in the ocean, etc.
- 5:40Speaker 2: Plus, you could do all of the usual object detection that we know and love.
- 5:46Speaker 2: But essentially the incumbent method here is to basically beam those images as soon as you get over a ground station and eventually you get a notification saying there was a forest fire, whatever amount of latency ago, and that latency can vary.
- 6:03Speaker 2: In traditional systems, because two-thirds of the Earth is covered by water, it can be quite a while before you orbit over a ground station, and then you have other delays of the ground, so it could be literally many hours or in some cases, even days before you manage to get at data that you care about.
- 6:26Speaker 2: And obviously, if there's been forest fire, well you you know that's now a big burnt patch around that, at that area that you've looked at.
- 6:34Speaker 2: If you look at the forest fires that were that there were in Australia a few years ago that they were consuming about three football fields a minute.
- 6:45Speaker 2: So you can imagine so the existing systems are basically, you would rather, than doing the inference in orbit, you're doing the inference on the ground and you're transmitting that through the usual chain to a cloud installation.
- 7:03Speaker 2: You do an inference there and eventually your customer gets a notification.
- 7:08Speaker 2: And what we're doing is we're doing the AI inference directly on satellite.
- 7:13Speaker 2: So you go down to inference time in the order of tens of seconds and even through the transmit chain you're able to use, let's say, long-range, low-bandwidth satellite-to-satellite connectivity, which means effectively you're transmitting tens or hundreds of bytes rather than many, many hundreds of kilobytes for the raw image, and you can actually alert customers in one to five minutes, depending on where you are.
- 7:44Speaker 2: So it's a really game game changer in terms of getting timely information about what's happening on the ground.
- 7:50Speaker 2: Be it a forest fire, be it a ship that's in the wrong place or fishing illegally or whatever, you get very timely information.
- 8:01Speaker 2: So we get into some of the detail now about how this is being done, et cetera, and what we're doing here is really it's product, not pixels, that counts for customers.
- 8:20Speaker 2: The typical contracts with big satellite Earth observation companies tends to be based around data, but that usually involves costs for the customers.
- 8:33Speaker 2: So what we're trying to do is disintermediate the chain and provide product, not pixels, to customers.
- 8:42Speaker 2: So I mean, why bother with edge AI?
- 8:45Speaker 2: I guess the audience is familiar, but no, it's no harm to repeat it.
- 8:50Speaker 2: Big advantage is less power and generating heat, as the product of doing computation in space is actually a big problem, much bigger than on the ground.
- 9:04Speaker 2: So ideally there is kind of a power envelope of maybe 10 watts total that you want to stay below in in a um in a typical lower orbit satellite uh, bigger satellites obviously can have a bigger budget, but, uh, essentially the amount of power that you dissipate matters.
- 9:24Speaker 2: So you can't just fly a you know, a CUDA system in space.
- 9:29Speaker 2: Even if you could, because of the mass, you wouldn't be able to light it up because of the power issue.
- 9:34Speaker 2: Obviously you have the potential to consume far less bandwidth by doing the inference at the edge.
- 9:40Speaker 2: You have less latency and obviously you have more privacy, which, in the area that we live in, you have less latency and obviously you have more privacy, which, in the area that we live in, it's very important, not just for private individuals but for industries and for defence related reasons.
- 9:55Speaker 2: So I mean the kinds of applications that we're looking at here are things like maritime monitoring, the number of ships, image compression, vehicle detection.
- 10:07Speaker 2: It's very important to be able to georeference objects on the ground so you have some idea what way the camera is pointing, because you have a star tracker and you have to have the antenna pointed the right way, but still you need to be able to figure out the equivalent of a GPS reference for what you're looking at.
- 10:34Speaker 2: So that's very much part of a solution is figuring out the georeferencing and being able to interpret the image in some way, in terms of being able to provide a count of the number of vessels or whatever is also important.
- 10:47Speaker 2: There are many other applications, like removing clouds, because your typical earth observation image but 68% of it on average is obscured by clouds.
- 11:00Speaker 2: So obviously, if there's cloud, you can't do an inference on anything by cloud.
- 11:05Speaker 2: So obviously, if there's cloud, you can't do an inference on anything that's obscured by cloud.
- 11:08Speaker 2: So there's no point in transmitting the image or any inference product based on it so effectively, you can get a you know, a 3x uplift in terms of the effective bandwidth just by removing the clouds, and that was, in fact, the initial application that we looked at with the European Space Agency on FISA 1 back in 2020.
- 11:25Speaker 2: There's also lots of AI applications for the space economy itself so handling space, debris, robotics in space, health monitoring for satellites, for the solar panels, debris mapping and so on.
- 11:44Speaker 2: So there are lots of applications with that emerging space economy that will require lots of autonomy in space because otherwise, if you have a loop to ground, things are moving at high rate in space and you could have had some major incident by the time you're able to figure it out if you're doing everything right.
- 12:07Speaker 2: So everything is based on uh, um, intel and novidius, uh, bpus.
- 12:13Speaker 2: You know all that information is available um online.
- 12:19Speaker 2: I won't go into it too far, but essentially the the important point here is that we're using um you know, consumer off the shelf or cuts technology on satellites and there has to be work done to qualify it.
- 12:31Speaker 2: But it's exactly the same part and this is different from the way a lot of space hardware has been done to date uh, in that they would often have built bespoke chips and so on, and those bespoke chips are often lagging a long way behind the market in terms of the, the process, technology and software, etc.
- 12:51Speaker 2: So the using of the COGS technology allows you to basically build systems that are much closer to the market and are easy easier to maintain and so on, than some of these legacy systems that they've been playing, and your participation, as I mentioned, is well, well under the 10-watt limit that you have to consider in terms of maximum participation.
- 13:17Speaker 2: So I mean there are lots and lots of different events that can happen in space.
- 13:24Speaker 2: Essentially, you don't have the atmosphere to shield you, so you're worried about, you know, these high energy particles that can cause upset events, particularly in things like memory, but they can also call, cause latch up events.
- 13:38Speaker 2: So, essentially, though, traditional space technology would have used you know special structures and special, you know high, highly radiation tolerant technology and process technology to build chips in order to mitigate these, this issue, as soon as you're flying in space.
- 13:59Speaker 2: The approach that's taken with COTS is really to have one or two hardened elements in the system that are relatively low tech and what they're doing is monitoring that everything is okay, isolating your COTS technology from the rest of the system and they're in a position to reset you or if there's some kind of upset event.
- 14:20Speaker 2: So this is essentially what allows you to set event.
- 14:27Speaker 2: So this is essentially what allows you to run these applications in space, in non-cost technologies, to have essentially a wrapper that sits around the cost technology that is radiation hardened but it's not doing any of the processing, it's only monitoring system health and resetting.
- 14:41Speaker 1: So certainly when we did this, yeah, I don't want to cut you off here, just gently interjecting some questions based on what you've been talking about so far.
- 14:55Speaker 1: So you're using the kind of Intel Movidius platform off the shelf kind of AI acceleration hardware to do kind?
- 15:03Speaker 1: Of image recognition and things in the satellite itself.
- 15:06Speaker 1: So a couple of questions.
- 15:07Speaker 1: Sure One was what are you doing to sort of manage the like?
- 15:15Speaker 1: Are you producing the models and the system here, or what is Eubotica producing out of this?
- 15:22Speaker 1: You're not making the satellite.
- 15:24Speaker 2: We produce boards that go into satellites that are radiation tolerant, so the overall board is radiation tolerant.
- 15:34Speaker 2: And then we provide the software, and in many cases also the neural networks that are used to do the inference on board as well, plus the software necessary to deploy new software and software updates from the ground onto the satellite.
- 15:52Speaker 1: All right, so are you able to update the AI model, the model itself?
- 15:55Speaker 2: Yes.
- 15:57Speaker 1: From the ground.
- 15:58Speaker 2: From the ground, indeed.
- 15:59Speaker 2: So you can have multiple models running either simultaneously or in time series mode as well, depending on where you are.
- 16:13Speaker 1: I see Cool and I think, by the way, we have Dan couple of questions, Hi.
- 16:22Speaker 3: I just was wondering if image compression has something to do with generative AI.
- 16:30Speaker 3: I mean trying to achieve very high compression factors to transmit residual information and, you know, with generative capability to decode information.
- 16:47Speaker 2: if these are something to do within your absolutely it's part of it's part of the solution that we currently deploy, and one of the applications is image compression and you know you can get somewhere between 10 and 30x compression depending on the amount of, let's say, degradation in terms of the source image that you're interested or that you're able to tolerate when you want a machine consumable image.
- 17:22Speaker 2: If you're doing inference at the edge, then essentially you've got a bounding box, you'll get an object type and a GPS equivalent location and you get that.
- 17:37Speaker 2: You get a.
- 17:37Speaker 2: Going to have some issues in terms of getting the data off fast enough, particularly if you're in a polar orbit, because it could be some time between now and the next time you pass over a ground station when you can download the image at high rates.
- 17:58Speaker 2: I would say that in many cases, unless you're in an equatorial orbit, in many cases, unless you're in an equatorial orbit, you're typically going to be dealing more with AI inference product rather than actual pixels.
- 18:13Speaker 3: Cool.
- 18:13Speaker 3: And if I may, another question, peter, as implementer, as a guy that generates chips, I wonder if you like privileging more power consumption or energy efficiency, or both.
- 18:36Speaker 2: Both, really, because you have a hard limit of maybe 2 or 3 watts maximum for the whole board including everything, but you still want to be able to run modern networks.
- 18:49Speaker 2: So both absolute power and efficiency are both very important for two different reasons.
- 18:57Speaker 2: The first because you can't really, you know, actually removing the heat is a big problem in space.
- 19:05Speaker 2: Secondly, because you want to be able to, you know, because you have to live with that participation envelope, you still have to be able to get the same amount of work done.
- 19:18Speaker 3: Okay.
- 19:20Speaker 2: So it's challenging from both points of view.
- 19:25Speaker 1: Yeah, I think, Danilo I don't know if you brought up the generative AI issue.
- 19:29Speaker 1: I mean, using a transformer model here would require probably a different piece of silicon than the Movidius chip, since that's kind of a one-to-four-tops platform.
- 19:40Speaker 1: But that might be it.
- 19:43Speaker 2: Yeah, I mean, we're obviously looking at other platforms.
- 19:46Speaker 2: I'm just referring in the presentation here to the XE1 and XE2, which are already flying.
- 19:52Speaker 2: Xe2 is based on MyriadX, but we have other solutions in the pipeline.
- 19:57Speaker 2: But for many applications let's say, we have a space camera application that we're announcing, which, for that application, you're just interested in streaming video of a launch back to ground, maybe with some inference product associated with it, and you can probably continue to fly that as long as you have a supply of parts, and it's not.
- 20:24Speaker 2: It's not all necessarily about about performance either.
- 20:27Speaker 2: True, yeah, legacy, the Legacy, you know, flight Legacy counts for a lot of space applications, so it's quite different from, uh, from many other applications that might be more familiar.
- 20:40Speaker 2: Right, right, so should I continue on then?
- 20:44Speaker 1: Yeah, sure go for it, and if anyone has questions out there, put them in the chat.
- 20:49Speaker 2: We'll like to hear.
- 20:50Speaker 2: So interesting data point here is that when we started this project with the European Space Agency back in 2017, one of the things that we had to do was to test the part to see if a COTS device was going to be at all possible to fly.
- 21:08Speaker 2: So we actually went into the Large Hadron Collider Complex, the SPS line there, which is, I think, about 13 kilometers long, and we tested for about 40 hours continuously and we found that the upset rate was very low and that meant that it was a good candidate for doing the remaining phases of the initial PHYSUP1 project.
- 21:38Speaker 2: So you can see two members of our team deep underground there in CERN Geneva doing it.
- 21:43Speaker 2: Didn't see much of Geneva when they were there, but it was an exciting opportunity to basically visit the complex and do some real scientific work there.
- 21:54Speaker 2: So I mean what at the end?
- 21:58Speaker 2: You know we did a significant amount of work with the European Space Agency and what we found was that, in terms of you can see the bottom right hand corner there we're able to run Inception V3 with less than one upset event per day.
- 22:13Speaker 2: Three layer fully connected network was causing more upsets, but things like mobile SSD was again less than one upset per day.
- 22:25Speaker 2: So there is some dependency on network topology, but we've probably found there is.
- 22:31Speaker 2: You can select networks that will perform well, and the device as is, without any modification, was capable of doing the work, and I can provide references for those papers if people are interested.
- 22:46Speaker 2: So I mean, the whole idea here is to be able to do AI in space directly.
- 22:54Speaker 2: So we had a FISA one there if you see, it launched in September 2020, and that was doing cloud detection and removal.
- 23:01Speaker 2: So we did work with the University of Pisa.
- 23:04Speaker 2: They have trained a cloud detector and we were able to remove that cloud at source and just downlink tiles containing actual image data rather than images of the clouds.
- 23:18Speaker 2: And we've also had a product running on the ISS it's 2021 together with JPL, which is basically running lots and lots of different benchmarks in space on our hardware, and we have a range of other applications, including ship detection so on, that are currently running in space.
- 23:41Speaker 2: This was basically the assembly phase onto FISAT-1 of the original satellite.
- 23:48Speaker 2: This is what the size of the satellite is, so it's one of those large about the size of one of those very large boxes of corn flakes or whatever that you can get in the supermarket.
- 23:58Speaker 2: So quite small, with all the panels and everything on board that was assembled in Korea and was then shipped to Peru and French Guiana for launch.
- 24:09Speaker 2: This is the ISS.
- 24:12Speaker 2: So we've published work there, together with Hewlett Packard Enterprise and JPL in California, on running lots of these applications in space on the ISS and essentially monitoring them for performance and radiation tolerance, although it's a much more forgiving environment inside the ISS than it is on a LEO satellite, because there's some amount of shielding there to protect the astronauts.
- 24:41Speaker 2: We have a number of different products right now.
- 24:42Speaker 2: This XE2 is what I was talking about, which is in the current generation of satellites.
- 24:49Speaker 2: Xe1 flew on FESAT-1, and we have Cognizant Cam, which essentially is a myriad X based product, which is a launch camera that sits on the launch vehicle and gives you all those nice videos that people like my space, like SpaceX, produce on launch of their vehicles to keep their customers happy and investors happy, I guess, as well.
- 25:16Speaker 2: And we also have then pure software products like cloud removal and compression that go along with the boards.
- 25:25Speaker 2: So we've had a total of seven missions to date.
- 25:33Speaker 2: The latest one is Cognizant 6, which is a joint project together with OpenCosmos and that's orbiting right now.
- 25:41Speaker 2: It was launched on SpaceX transporter L11 on in Q1 of this year and it's launched, deployed and working, detecting ships and lots of different applications from about you know, six, seven hundred kilometers up, about six, seven hundred kilometers up Inside the satellite itself.
- 26:05Speaker 2: I mean, you have our board somewhere sitting in here in the stack.
- 26:15Speaker 2: You have all the comms links.
- 26:17Speaker 2: You have start.
- 26:17Speaker 2: You have to detect where the Earth's horizon is.
- 26:18Speaker 2: You have reaction wheels that allow you to orient the satellite in space.
- 26:21Speaker 2: You have a star tracker to figure out where, in what direction, you're pointing, because there's no point in having the cameras and the communications antennas not pointing at ground.
- 26:32Speaker 2: You have batteries, you have solar panels, etc.
- 26:36Speaker 2: So it's a pretty complicated piece of kit.
- 26:38Speaker 2: It's not just, you know, an inference board.
- 26:43Speaker 2: There's a lot of other technology that goes in there.
- 26:48Speaker 2: Kinds of applications that you'd be worried about are things like algal blooms, which can either clog all the filters in a big desalination plant if you're pumping water into it and cause you to have to take the desalination plant out of commission, or indeed you can get big fish kills.
- 27:07Speaker 2: In this case it was about 8 million fish were killed and I think the value of it was about 32 million euros, or probably 35-36 million US dollars in one event.
- 27:18Speaker 2: And these can happen anytime, and they typically happen in summertime, and obviously the higher the sea temperature is, the more chance there is of these things happening.
- 27:27Speaker 2: And knowing quickly that there's that there's a bloom going to happen means that you can harvest the fish early, otherwise they're just dead and you have to dispose of them in some other way.
- 27:39Speaker 2: Uh, other things are forest fires.
- 27:40Speaker 2: You can see there.
- 27:41Speaker 2: This is locations of forest fires all across the Amazon basin and, as I mentioned, you know, time is money and so on.
- 27:50Speaker 2: In that case, you could be consuming huge square kilometers of many square kilometers of forest every hour if you can't detect them.
- 27:59Speaker 2: The same kind of thing happens all around the world.
- 28:03Speaker 2: You can see places like indonesia, parts of africa, sri lanka as well I had a quick question.
- 28:11Speaker 1: We had a question in the chat from the shawl um question was has there been any usage for remote sensing for carbon monitoring?
- 28:19Speaker 1: I know you're going through a lot of use cases, but is that, yeah, this, this, uh, this, this satellite?
- 28:24Speaker 2: speaks to that point.
- 28:26Speaker 2: It's not quite carbon, it's methane, which is actually much worse.
- 28:31Speaker 2: It's about 4x worse than CO2.
- 28:34Speaker 2: And if you look at your typical methane leak, a single leak can cost you as much in 24 hours as 200,000 cars in a year.
- 28:51Speaker 2: So, again, plus, you've got the cost of the actual methane leaking out of the pipeline itself as well.
- 28:56Speaker 2: So that's definitely a motivating use case.
- 29:00Speaker 2: I'm not so sure if you can detect um carbon, carbon dioxide directly, but you know if you get, if you get lots of growth in terms of algae or whatever, or or in, indeed, trees or uh, or um, uh other vegetation, uh, you have way.
- 29:21Speaker 1: You have proxies for CO2 as well although not real-time, and you're doing all this based off of just kind of image pattern recognition, exactly.
- 29:31Speaker 2: And it's you know, because we're using a hyperspectral sensor, we're using a five-meter GSD, so it's five meters on a side per pixel.
- 29:39Speaker 2: You get a lot of information.
- 29:42Speaker 2: If you're looking at trees, you can kind of tell what the health of those trees is.
- 29:45Speaker 2: If you're looking at soil, you can tell if the soil requires nutrients or requires water and so on.
- 29:53Speaker 2: So it's actually a bit like you're looking at the earth with a spectrum analyzer.
- 29:58Speaker 2: Yeah, exactly, wild, all right.
- 30:02Speaker 2: So I mean, another application is in terms of detecting ships.
- 30:07Speaker 2: I mean there's been a lot of talk here in Ireland about Russian ships.
- 30:10Speaker 2: In the economic exclusion zone that we have, it's about 455,000 square kilometers, and you know square kilometers, and we have a very small navy.
- 30:32Speaker 2: It costs about 250 million euros to build one of these ships and then probably 25 million a year to keep it operating, and it can only cover a small amount of the economic exploitation zones.
- 30:43Speaker 2: There's obviously clearly a use case there.
- 30:44Speaker 2: It was a pretty interesting case in the past few months where the Irish Navy intercepted a cocaine shipment worth 150 million dollars from Colombia and they uh from colombia and uh, they, they, uh, um, they, you know there's, it's suspected that this is the tip of the iceberg in terms of the amount of drugs coming into uh, into ireland, britain and indeed the atlantic coastlines of france and portugal and uh and spain as well.
- 31:20Speaker 2: So this is an application where timely detection of ships is important.
- 31:26Speaker 2: Another one would be illegal fishing because, again, a lot of these economic exploitation zones come with restricted areas in terms of fishing and the current data.
- 31:39Speaker 2: I think it was a paper produced earlier this year which showed that about three quarters of the large fishing vessels in factory ships actually have their transponders switched off at any one time.
- 31:51Speaker 2: So the only way you can tell if they're spoofing their location or if they've got their transponder switched off as if you can validate visual- satellite imagery.
- 32:09Speaker 1: So that's it.
- 32:10Speaker 1: We have a question from glenn.
- 32:10Speaker 1: Uh, glenn, here, I'm just going to throw it up here.
- 32:11Speaker 1: Um, talking about the software stack are using tensorflow tiny ml.
- 32:14Speaker 1: I know you're using movidius.
- 32:16Speaker 1: Does that mean it's open vino or yeah?
- 32:18Speaker 1: Yeah, we're using.
- 32:18Speaker 1: Yeah, we're using open vino.
- 32:20Speaker 2: Uh, we have some of our own tools as well, but largely it's based on OpenVINO, because we're typically also running all the cameras and so on.
- 32:31Speaker 2: We're doing a lot of camera, isp and so on to condition the image correctly.
- 32:36Speaker 2: But yeah, essentially it's OpenVINO and we, or our customers customers typically be using TensorFlow or PyTorch as a as a front end for the, for the network design.
- 32:48Speaker 2: Got it Okay, so it's.
- 32:50Speaker 2: It's fairly pretty vanilla stuff.
- 32:53Speaker 1: Right?
- 32:54Speaker 1: Well, I guess that's.
- 32:54Speaker 1: The idea is that you're using kind of fairly off the shelf stuff but you're doing it in a very, I would say, harsh environment.
- 33:01Speaker 1: I think that's one of the things we see.
- 33:02Speaker 1: A lot is like it's one thing to kind of do it in the lab, but actually deploy these sensors in very harsh environments and make them work.
- 33:08Speaker 1: That's kind of what.
- 33:09Speaker 2: I'm working on.
- 33:10Speaker 2: It's challenging.
- 33:11Speaker 2: It's challenging.
- 33:12Speaker 2: All sorts of very interesting stuff can happen up there that is beyond your control and you're relying, really, on the succession of systems on board the satellite to recover from those air conditions, because you can get hit by a blast of solar wind, from a coronal mass ejection and so on at any point and you essentially have no protection.
- 33:36Speaker 2: In fact, the next section of the presentation speaks exactly to that point.
- 33:41Speaker 2: So I mean put a nice little quote from Lenin here, which is there are decades where nothing happens and there are weeks where decades can happen Maybe live in interesting times and all that.
- 33:54Speaker 2: So we're working with the European Space Agency on a mission called Vigil which is being deployed to the L5 Lagrange point and that's essentially a relatively stable point in space that gives you a kind of a third-person perspective of the sun.
- 34:11Speaker 2: So essentially you can be around the backside of the sun.
- 34:14Speaker 2: The sun takes 17 hours at least to complete a rotation and then the light from the sun takes several hours to reach Earth.
- 34:26Speaker 2: So essentially you can deploy an early warning system for space weather.
- 34:33Speaker 2: The motivating reason for this is in 1859 there was a thing called the Carrington Event where most of the wireless telegraphy in the world was knocked out.
- 34:43Speaker 2: Operators were getting electric shocks because essentially you've got long wires there and you've got the magnetic field induces a current in those wires and obviously if you're sitting connected to that you can get electric shock.
- 34:58Speaker 2: And more importantly now in the world that we're in now, if the Carrington event was to happen today and several of the big insurance companies have done an estimate that the cost of such an event would be about four times the cost of the the Covid pandemic.
- 35:18Speaker 2: So it's a big deal and it could take 10 years to unwind all of the details.
- 35:23Speaker 2: And there's real threat to humans as well, because obviously if you're flying in an aircraft and either the navigation systems go down or the aircraft's electronics gets disabled, you're in deep trouble.
- 35:40Speaker 2: So essentially we're working on this mission and it's 150 million kilometers from Earth.
- 35:46Speaker 2: The amount of link budget that you get and the amount of bandwidth that you're able to get back from the satellite.
- 35:52Speaker 2: We're talking about kilobits per second, so you're not going to be transmitting much in the way of raw imagery.
- 35:58Speaker 2: So the satellite currently, as defined, has about of raw imagery.
- 36:05Speaker 2: So the satellite currently has defined, has about it has a capacity for about 70 kilograms of worth of ballast to kind of balance the spacecraft.
- 36:10Speaker 2: So we're working essentially on a parallel project here where some of the space for the ballast could be freed up to actually do AI based event recognition and forecasting.
- 36:23Speaker 2: Oh go ahead.
- 36:24Speaker 2: Danilo, you had a question.
- 36:26Speaker 3: Yeah, sorry to interrupt you.
- 36:28Speaker 3: Can you comment a little bit on robustness against radiations, you know, and if you use any particular approach, also with respect to the fact that you accelerate AI?
- 36:42Speaker 3: Yeah, I mean.
- 36:44Speaker 2: I think I mentioned it earlier on around about seven or eight slides in.
- 36:49Speaker 2: We actually did radiation testing in a facility in Darmstadt and essentially you know raw on Myriad 2, it was at the time rather than Marriott X, which is a 28 nanometer bulk CMOS process, tsmc we were getting about one upset per day.
- 37:12Speaker 2: On networks like mobile net, some other networks, you could get more upsets per day, but essentially it's not like a one shotshot event where you know if you mess it, you're, you're toast.
- 37:26Speaker 2: Uh, you can essentially have and what we do have is a circuitry that's radiation hardened, that sits around the COTS processor and basically its job is to detect when something has gone wrong and reset the COTS processor.
- 37:42Speaker 2: So that's essentially the strategy there that allows you to use COTS in space.
- 37:47Speaker 2: So the reason for doing that is that it's way cheaper.
- 37:52Speaker 2: If you have to build a dedicated part for space, you're only going to maybe build 10 or 20 of them.
- 37:58Speaker 2: Even on a multi-project wafer, you're talking about hundreds of thousands of dollars just to get a few parts.
- 38:03Speaker 2: If you can, can use cuts parts and just solve the problem board level it's a game changer for the forward, for the industry.
- 38:11Speaker 2: That's the approach we're taking yeah, interesting.
- 38:15Speaker 1: Hey, there's another question here from the chat before we jump into the next topic.
- 38:19Speaker 1: Uh, just around ml ops, we talked a little bit about updating the models from the ground.
- 38:24Speaker 1: A question from Mohamed, curious about the process of updating models from the ground how is that managed in the context of space missions, especially for satellites with limited communication bandwidth?
- 38:33Speaker 1: Large model sizes.
- 38:35Speaker 2: Yeah well, I mean, I guess what you're trying to do is trim the networks down to the minimum size, even things like YOLO that we're using for many of the applications we have running in space.
- 38:46Speaker 2: We trim down the networks, you can compress them and so on.
- 38:52Speaker 2: So any of the usual tools that you would use to compress files, you can use those and that's part of our processing chain.
- 39:03Speaker 2: In terms of MLOps, I have a more detailed slide later in the presentation where I can speak to that.
- 39:12Speaker 2: But MLOps is definitely part of the solution.
- 39:15Speaker 2: Particularly if you've got multiple satellites in the constellation, you want to be able to learn across those devices and you want to be able to learn across those devices and you want to be able to fold what you've learned back into the data set and update the networks across all the devices deployed in the constellation.
- 39:35Speaker 2: So that's definitely part of it, but I can get into that more on this specific slide.
- 39:39Speaker 1: One more question before you jump into the next one too, on this specific slide.
- 39:42Speaker 1: One more question before you jump into the next one too, because we've got a few piling up here Does the edge nodes use ML inference at the edge or is collecting sensor points and sending information to the central server?
- 39:50Speaker 1: I think you might have addressed this one, but just to clarify.
- 39:53Speaker 2: Yeah, we're doing inference at the edge, the reason being that it's in space particularly if you're in a polar orbit, it could be hours or even days before you get over a ground station to be able to downlink the image.
- 40:07Speaker 2: So, whereas if you do inference directly adjacent to the sensor, you can use long range, low bandwidth inter-satellite communications to get that data down via the satellite network and the constellation.
- 40:25Speaker 2: So, essentially, you can get real-time updates and where, let's say, a traditional cloud based solution might take, you know, many, many hours or days to detect whatever kind of event, simply you just can't get the data down quickly enough, okay, okay.
- 40:45Speaker 2: So I mean one of the things that we're doing, which, for that mission, is because you're a long way away from Earth you're 150 million kilometers from Earth what you want to be able to do is to essentially auto label data, new data.
- 41:00Speaker 2: As you see it, nobody has actually deployed a mission of this type.
- 41:05Speaker 2: There are kind of similar missions for observing space, whether but nothing, at the L5 Lagrange point in many space applications, the first time you fly the application is the first time you're flying the sensor.
- 41:17Speaker 2: So you don't necessarily have enough level sensor data to start with to train your network.
- 41:23Speaker 2: So you need to figure out a way of bootstrapping this.
- 41:25Speaker 2: So there's two possibilities One, which is to essentially take an existing sensor and modify the sensor data so it looks like what you expect to see from the new sensor that's being flown on the mission.
- 41:39Speaker 2: And that's in fact what we did with Cognizant 6 and it worked very well.
- 41:42Speaker 2: So it meant that rather than, let's say, launch the satellite and then have to wait six months to get enough data to be able to start training the network, you can actually make a pretty good guess at how things should work from the get-go.
- 41:54Speaker 2: You can launch with the network and then you can obviously bootstrap from there and improve performance and improve performance.
- 42:03Speaker 2: So essentially, we've developed some technology in-house that allows us to triage data from the sensors that we have and to essentially we're able to fingerprint at the training stage constellations of features that light up in the network that correspond to particular subclasses in the training data set.
- 42:24Speaker 2: So that means that detection at inference time in the satellite.
- 42:31Speaker 2: You can then see okay, does this look like something I've seen before?
- 42:35Speaker 2: Does it have a fingerprint?
- 42:36Speaker 2: You know this neural network supervisor fingerprint that looks like what I've seen before.
- 42:41Speaker 2: If it hasn't, then you can essentially store that and you can essentially generate, you know, auto labels for that new class.
- 42:49Speaker 2: If, on the other hand, it's something that you have seen before, then you can decide to generate an immediate alert for it, saying that there's a big new coronal mass ejection happening and you better basically move the satellite orbits and so on on Earth or issue warnings to aircraft to get out of the sky because we think there's a big coronal mass ejection coming in 17, 18 hours time.
- 43:19Speaker 2: And the other possibility is that the data is just junk, in which case it's ugly and you throw it away.
- 43:25Speaker 2: So the good is the stuff that you do right now you send an update.
- 43:29Speaker 2: The bad is stuff that, well, you're not really now you send an update.
- 43:31Speaker 2: The bad is stuff that, well, you're not really sure what it is, but it's something novel.
- 43:33Speaker 2: You generate a label for it using your Lydian technology and you store that label and metadata and source data on, let's say, an SSD that sits or a similar mass storage device that sits on the satellite, or else you chuck it away, and triaging the data in real time is really kind of key for this kind of application where you're a long way from, let's say, the base station that connects to the rest of the network, but you don't want to miss anything.
- 44:05Speaker 2: So I mean this technology.
- 44:09Speaker 2: We're looking at a range of different platforms to support it, from things like Cognizant XE2, which is based on the Myriad X.
- 44:15Speaker 2: It's got two Leon 4s, one TOPS NPU and it's about, let's say, one and a half two watts envelope.
- 44:21Speaker 2: Alibaba has this board based on the Alibaba TH1520, which is based on RISC-V, four tops NPU and it's running two and a half watts.
- 44:35Speaker 2: And there's a similar rock chip, rk3588, which is based on a quad core, arm Cortex, with six tops and an NPU and two and a half three watts.
- 44:46Speaker 2: So there's lots of options there in terms of building a software system that can leverage all the usual Linux goodness that can potentially fly in this kind of application.
- 44:56Speaker 2: The main thing is to build the COTS tolerant subsystem around it that allows you to do that.
- 45:04Speaker 2: So the kind of things that we're doing here are we're doing real-time detection and streaming.
- 45:10Speaker 2: You can see here, this is a video frame.
- 45:11Speaker 2: We're running YOLO on it and we're doing real-time detection and streaming.
- 45:12Speaker 2: You can see here, this is a video frame we're doing.
- 45:13Speaker 2: We're running YOLO on it and we're running some additional post-processing on it and that's allowing us to track the size, the direction and so on of sunspots and coronal mass ejections on the sun and we can basically send back bounding boxes and intensity numbers or whatever as the scientists dream up that they want to see.
- 45:35Speaker 2: So we're working together with the Dublin Institute of Advanced Studies, who are solar weather experts in this area, and also with the European Space Agency, who have their own solar weather experts, to basically identify training data and detection strategies that we can use for tracking those CMEs and sunspots on the surface of the sun.
- 46:01Speaker 2: We've also done a lot of work because we're working across multiple sensors, so a bit like the sensors that we've been using hyperspectral sensors, in this case we're using X-ray sensors, ultraviolet sensors, conventional RGB and so on.
- 46:23Speaker 2: So you can essentially take, if you can georeference, everything that's on those different images, even though they're in completely different parts of the radio frequency spectrum, you can still combine weight and ensemble them to get good results.
- 46:39Speaker 2: And in fact that's the strategy we've been taking is to essentially use all of the available data to come up with and make heavy use of ensembles to get good predictions of what's going to happen on the surface of the Sun.
- 46:55Speaker 3: Do you see a scenario where the signals are processed or filtered based on language queries?
- 47:09Speaker 2: That's something that we've looked at.
- 47:11Speaker 2: I would say that we're at the beginning, really, of small language models, but it's certainly something that you could use to get better reports from these very far deployments of edge processing technology, but it's not something that we currently.
- 47:28Speaker 2: We have research going on in the area but we haven't got anything that we can deploy at this point in time.
- 47:33Speaker 2: But certainly you can imagine there being able to generate a human-readable report in text is going to require many orders of magnitude less bandwidth and still be more useful than it would be to transmit all of the source images from 150 million kilometers away.
- 47:52Speaker 2: So, yeah, you're correct that that's the way the market is going, but it'll be a while before we have something that we can deploy.
- 48:06Speaker 2: So I mean here we're basically looking across these three different instruments that are on board Vigil and it's.
- 48:11Speaker 2: You know we're using ensembles and lake fusion to detect with higher accuracy where these, where the sunspots are, than if we only used a single one of the instruments at a time.
- 48:23Speaker 2: And this is a particular example using the magnetogram for basically processing the information on MyriadX and again using ensembles to get a higher accuracy result than you might have just done using a might have just done using a single network and a single instrument.
- 48:54Speaker 2: This is basically auto labeling events using the Lillian auto labor labeler, so that we can basically take these events and catalog them where we haven't seen similar events in our training data set, which can be those training data sets for new instruments in these space applications.
- 49:21Speaker 1: Yeah, david, could I just jump in?
- 49:23Speaker 1: We had a question related to that from Ravi about data labeling and ingestion.
- 49:29Speaker 1: Question is is it real time on the edge or is edge device send data to land to train the model?
- 49:35Speaker 2: and then in.
- 49:35Speaker 1: So where are you doing the training exactly?
- 49:37Speaker 2: Where well, I mean right now, what we're trying to do is do the labeling at the edge.
- 49:46Speaker 2: I think ultimately we'd use like to use Tiny Train or something like that to do limited training and updates at the edge.
- 49:52Speaker 2: But right now it's not feasible to stream lots of raw data and then figure out at the other end what parts of it were useful.
- 50:04Speaker 2: So what we're doing is essentially using this leading technology to auto label new events at source, store them and then, when time allows and when we have enough labels accumulated for a new class, we can basically stream that back to Earth, compressed or whatever, and then update the training data set on the ground.
- 50:26Speaker 2: But yeah, I mean, ultimately it would be great to be able to do this stuff on the satellite itself.
- 50:33Speaker 2: But the big issue there is participation, right, because you only have a couple of watts probably to play with, maybe 10 watts or something on the high end, so you can't really do what people will be used to doing in more typical deployments.
- 50:55Speaker 1: Right.
- 50:57Speaker 2: But it's definitely a hot topic, something we're working on, this new paper called tiny train.
- 51:04Speaker 2: I can give a reference later on which from a team in Cambridge in in the uk, uh, that has, uh has that kind of technology, um, and we're looking at potentially adapting that for for this use cool um, and I think uh, um michelle asked a question there about mlops, so I mean right now, uh, if you look, I mean your edge deployment sits somewhere in here on the left hand side and you're basically doing data collection, ingestion, pattern analysis, labeling and data validation and so on.
- 51:41Speaker 2: So essentially, by pushing some of the data labeling process out to the edge here, you can potentially simplify and reduce the amount of labeling effort that you require in the grant, particularly if you can ingest a whole new class into your training data set and then you can basically go around the loop and eventually get to the point where you can deploy a new model on the upside.
- 52:10Speaker 2: So we know how to deploy models.
- 52:12Speaker 2: We have them updating in real applications right now for different networks and different applications.
- 52:21Speaker 2: But we're looking towards a future where you can do at least part of the data labeling in an autonomous way at the edge and to incorporate those new classes into your training data set.
- 52:38Speaker 2: So it's not a solved problem, but it's kind of heading in the right direction.
- 52:45Speaker 1: Cool.
- 52:47Speaker 2: Okay, and I think I mentioned already that we have this neural network supervisor.
- 52:54Speaker 2: So essentially we have these new data classes that can be added to the database.
- 53:02Speaker 2: You can check which, during the training phase, which, let's say, features within the network are being lit up and you can then essentially vote on those across all the different subclasses and you can then vote and you can generate a final decision as to what goes into the database in terms of a fingerprint for a new class and you can do this work at the edge for a new class and you can do this kind of you can do this work at the edge.
- 53:34Speaker 2: In fact, it's essential that you do it at the edge, so then you can communicate some statistical information about what that class looks like to the ground without having to transmit all the data.
- 53:44Speaker 2: In terms of auto-labeler functionality, we found a pretty good reduction in terms of the time and obviously dollars required to auto-label data compared to a human annotator.
- 54:07Speaker 2: You can stay pretty much flat on a very low gradient with the auto-labeler compared to human annotation and obviously putting lots of humans isn't an option at the edge.
- 54:19Speaker 2: So I think we've got a usable system from that point of view and I think, for anybody who's interested in reading further about this, we have a number of references on different papers that we're publishing on computational storage, on the triage subsystem, et cetera.
- 54:45Speaker 2: Cool, we'll put those up In terms of, I suppose, probably for this audience.
- 54:50Speaker 2: The interesting thing is, you know, what can we, what have we learned from the whole process?
- 54:56Speaker 2: And we've been able to build, you know, pretty good solar event forecasting within the 24 hour window that uses LSTMs.
- 55:07Speaker 2: That uses LSTMs and we've been able to use that to capture long-range dependencies.
- 55:19Speaker 2: We're able to use techniques like dilated compilations to basically get more computational context for the networks and we've been able to get pretty exceptional results in terms of, you know, 24-hour window, figuring out what is likely to happen in terms of development of a solar flare, which is kind of essential when, when, uh.
- 55:41Speaker 2: You know, the whole premise of this mission is to uh be able to have, uh effectively a sentiment that sits out looking at the sun, that is trying to predict and let people know on the ground what's likely to happen.
- 55:56Speaker 2: In terms of training these TCN models, we're able to basically get through networks we've designed ourselves.
- 56:13Speaker 2: We've been able to get very good performance and be able to efficiently learn and predict the behavior of these solar events within a 24-hour period Just a quick interrupt before we run out of time.
- 56:33Speaker 1: There's a question from Manos.
- 56:34Speaker 1: Want to know is this mission live or planned to launch?
- 56:38Speaker 1: And, if it's future, is the data you're using for inference provided by another satellite telescope?
- 56:43Speaker 1: What's?
- 56:44Speaker 2: the yeah, there are a number of solar observatories that we can use for data right now and the time window for this mission is in the next five years.
- 56:55Speaker 2: So things in space move pretty slowly.
- 57:02Speaker 2: So we have a number For people who want to get more detail.
- 57:07Speaker 2: There's a number of references there and I will share the slides afterwards so people can put a link in if they want more detail.
- 57:15Speaker 2: For one of the networks we've been using quite a bit is modified YOLOv5, for instance, for star detection.
- 57:27Speaker 2: We've done an AI based star tracker with very good performance and we've done an AI based Star Tracker with very good performance and we've used essentially a simpler YOLOv5 to basically do previously.
- 57:40Speaker 2: We've built kind of each part of the signal processing chain in the Star Tracker separately and what we found is you can actually get very good performance if you kind of tailor the YOLO model to do that and some.
- 57:55Speaker 2: There's some information there in terms of you know the YOLO p5 model and the proposed network versus you know various different versions and again, this, this will be in the slides for people who want more detail, and there's also something there in terms of the training strategy.
- 58:16Speaker 2: I realize we're just under the hammer now to finish up.
- 58:18Speaker 2: So anyway, basically, the lessons that we learned is the model.
- 58:25Speaker 2: Efficiency is key.
- 58:26Speaker 2: You can train the networks back to things that you can deploy, feasibly by cutting on the whole idea.
- 58:37Speaker 2: There is to cut the amount of memory requirements, because that's going to make up the bulk of the file that you need to upload to update the network.
- 58:48Speaker 2: We're using techniques like data pseudo-labeling to try and deal with the fact that we're dealing with new instruments and new scenarios that haven't flown in space before.
- 59:01Speaker 2: We're able to use transfer, learning and optimization to get better reductions in training time Although for this kind of application, I mean the network needs to work as well as it can.
- 59:12Speaker 2: But because the instruments haven't flown before, it's likely gonna have to be updated during the lifetime.
- 59:22Speaker 2: So it's not that you can kind of bake everything in before you launch the satellite, and resource management and energy efficiency are really key to being able to to fly these kind of applications in spacecraft at all, particularly given that uh, you know the the uh given the power envelope of a few watts cool sounds good we had.
- 59:47Speaker 1: We have um really appreciate it.
- 59:48Speaker 1: We had one more question from Ravi.
- 59:50Speaker 1: If we can squeeze it in there, I think he was just asking more about the architecture around GPUs and the Movidius.
- 59:59Speaker 2: Yeah, I mean there's lots of information on MiriDex and so on.
- 1:00:03Speaker 2: I mean, we're using off-the-shelf technology and we're using OpenVINO.
- 1:00:07Speaker 2: In terms of the training I training, we're using either TensorFlow or, I would say, in fact, more, pytorch.
- 1:00:14Speaker 2: Some of our partners and customers use TensorFlow, but generally it would be PyTorch, openvino, and it's pretty vanilla stuff.
- 1:00:22Speaker 2: All of the tricks for space or what goes around this, in order to make it possible to get it to work reliably and robustly, with enough off time in a harsh environment with lots and lots of other constraints upon you in terms of updating the network.
- 1:00:43Speaker 1: Yeah, fantastic.
- 1:00:44Speaker 1: Well.
- 1:00:45Speaker 1: David Maloney from Eubotica thank you so much.
- 1:00:48Speaker 1: Danilo Pau from STMicroelectronics, thanks for co-hosting here.
- 1:00:53Speaker 1: Really fascinating talk, and we'll post some stuff.
- 1:00:57Speaker 1: Come look at us on YouTube channel or on Discord or Twitch or LinkedIn, yada, yada, yada.
- 1:01:04Speaker 1: So we're all over the place, but this has been super helpful and really informative, really appreciate it.
- 1:01:09Speaker 2: Thanks very much for the opportunity to speak and for all of the interesting questions.
- 1:01:15Speaker 1: All right Sounds good Thanks everybody, bye-bye, outro Music.