Latest / EDGE AI POD / Bringing Generative AI to Your Pocket: The Future of Edge Computing

Transcript

Bringing Generative AI to Your Pocket: The Future of Edge Computing · EDGE AI POD

• 0:00
  Speaker 1: Welcome back Deep Divers.
• 0:01
  Speaker 1: We're diving deep today, folks.
• 0:03
  Speaker 2: Into some seriously cutting-edge tech Buckle up.
• 0:06
  Speaker 1: Yeah, today we're talking all about the move to bring generative AI.
• 0:12
  Speaker 1: You know that AI that can whip up like crazy realistic images or audio or even text and putting it directly on our devices like our phones, our watches.
• 0:21
  Speaker 1: All that stuff right at the edge, no cloud required.
• 0:23
  Speaker 2: The edge.
• 0:24
  Speaker 2: It's the new frontier.
• 0:25
  Speaker 1: So I think it's a good place to start.
• 0:26
  Speaker 1: Like, what exactly do we mean when we say Gen AI?
• 0:30
  Speaker 2: Right.
• 0:30
  Speaker 2: So you're probably familiar with regular AI.
• 0:32
  Speaker 2: You know, like when your phone recognizes your face to unlock, or when it predicts the next word you're typing Right, that's standard AI.
• 0:39
  Speaker 1: Doing its thing.
• 0:48
  Speaker 2: But gener Generative AI that's like next level.
• 0:49
  Speaker 2: Instead of just analyzing what's already there, it actually creates completely new, believable data from scratch.
• 0:51
  Speaker 2: Have you ever seen those crazy realistic AI-generated artworks online?
• 0:53
  Speaker 1: Oh yeah, those are insane.
• 0:55
  Speaker 2: Exactly.
• 0:55
  Speaker 2: It's that kind of creative power that we're talking about squeezing onto something like your smartphone.
• 1:00
  Speaker 2: That's Gen AI.
• 1:01
  Speaker 1: Wild.
• 1:02
  Speaker 1: So this isn't just some random like hey, let's try this idea.
• 1:05
  Speaker 1: This actually builds on a movement, right?
• 1:07
  Speaker 1: Oh yeah?
• 1:07
  Speaker 2: absolutely the tiny ML movement.
• 1:09
  Speaker 2: It's been doing its thing, showing us what's possible when you run AI models locally right there on your devices.
• 1:17
  Speaker 1: Tiny ML.
• 1:18
  Speaker 1: Yeah, okay Well yeah.
• 1:19
  Speaker 1: Like my phone being able to translate languages even when I'm on a plane with no Wi-Fi.
• 1:23
  Speaker 1: That's tiny ML.
• 1:24
  Speaker 2: That's it.
• 1:24
  Speaker 2: Or think about getting instant responses from your smart home stuff.
• 1:27
  Speaker 2: No lag, it's all happening locally.
• 1:35
  Speaker 2: Tiny ML paved the way and now Edge.
• 1:36
  Speaker 1: Gen AI is like the souped up next generation.
• 1:37
  Speaker 1: So, okay, I get that it's cool and builds on stuff, but I got to ask why bother?
• 1:42
  Speaker 1: I mean, why try to like squish these massive Gen AI models onto our little devices when we have those huge, powerful servers in the cloud?
• 1:49
  Speaker 2: Yeah, that's a fair question.
• 1:50
  Speaker 2: I mean, at first glance it does seem a bit like backwards, right, but listen, when you actually dig into it, there are some seriously compelling reasons.
• 1:58
  Speaker 1: Okay, convince me.
• 1:59
  Speaker 2: All right.
• 1:59
  Speaker 2: First off, think about independence.
• 2:00
  Speaker 2: Your devices wouldn't need to be tethered to the internet 24-7 to be smart or do all this creative stuff Off the grid gen AI, exactly.
• 2:08
  Speaker 1: And then there's the privacy angle.
• 2:10
  Speaker 1: All your personal data stays local.
• 2:12
  Speaker 1: It's not floating around in the cloud somewhere.
• 2:13
  Speaker 1: And let's be real, speed is king.
• 2:17
  Speaker 1: Processing everything right there on the device means no waiting for data to go back and forth to the cloud.
• 2:23
  Speaker 1: That's huge.
• 2:24
  Speaker 2: No lag, I like it.
• 2:26
  Speaker 1: And there can be some major cost savings too.
• 2:28
  Speaker 1: Plus the potential.
• 2:30
  Speaker 1: Oh man, imagine the possibilities when you can deploy intelligent AI across like billions of edge devices.
• 2:37
  Speaker 1: Right, that's a game changer.
• 2:38
  Speaker 2: Yeah, okay, the potential is huge.
• 2:40
  Speaker 2: So this isn't just some fringe research project?
• 2:43
  Speaker 1: Oh, absolutely not.
• 2:44
  Speaker 1: This is gaining some serious momentum.
• 2:45
  Speaker 1: Even the tiny ML foundation you know the pioneers of this whole small scale AI thing they just rebranded themselves, yeah, just this past November they became the edgy AI foundation.
• 2:56
  Speaker 1: That's a statement, a big statement.
• 2:57
  Speaker 1: It highlights the broader focus and this growing belief that edge gen AI it can seriously shake things up.
• 3:04
  Speaker 1: I mean, imagine our devices not just, you know, passively doing what we tell them, but actually solving problems and interacting with us in a way more natural way, like they're thinking, almost.
• 3:14
  Speaker 2: Almost.
• 3:14
  Speaker 2: It's this idea of local intelligence, right there in your hand.
• 3:19
  Speaker 1: So for today's deep dive, our mission is to really understand just how far this whole Edge Genii thing has come.
• 3:25
  Speaker 1: We've dug through a ton of research from 2022 all the way up to this year, 2024.
• 3:30
  Speaker 1: And we specifically focus on models that are actually running on real-world devices.
• 3:35
  Speaker 1: Right, We'll look at how they're doing, what they can achieve, but first let's paint a picture.
• 3:40
  Speaker 1: Where can we actually use Edge Genii?
• 3:42
  Speaker 1: What kind of cool applications could it unlock for us?
• 3:45
  Speaker 2: Okay, let's talk real-world impact.
• 3:47
  Speaker 2: One area where it's already making a difference is in assisting people with disabilities.
• 3:52
  Speaker 2: Imagine having like image or video captioning, or even the ability to just ask questions about what your phone's camera is seeing.
• 3:59
  Speaker 2: And all of this is happening right there on the device.
• 4:01
  Speaker 1: That's amazing.
• 4:02
  Speaker 1: So it's private and it works even if you don't have an internet connection.
• 4:05
  Speaker 2: Exactly.
• 4:05
  Speaker 2: It gives people, especially those who are visually impaired, a whole new level of independence and access to information.
• 4:12
  Speaker 1: And speaking of independence, what about personal digital assistants?
• 4:14
  Speaker 1: That's an area where edge gen AI could really shine right.
• 4:18
  Speaker 2: Oh, big time.
• 4:19
  Speaker 2: We're talking about assistants that are actually intelligent way beyond what we have now.
• 4:23
  Speaker 2: I mean, imagine your assistant like truly understanding your environment and responding in like milliseconds.
• 4:30
  Speaker 2: It's almost like having another person there.
• 4:33
  Speaker 1: Yeah, and the whole interaction could be so much more natural, like imagine real-time translation happening right on your phone, no server needed.
• 4:41
  Speaker 2: Exactly.
• 4:42
  Speaker 2: And speaking of our phones, think about the trend of under-display cameras.
• 4:45
  Speaker 2: Cool tech, but sometimes the image quality it's not quite there yet, right.
• 4:50
  Speaker 1: Yeah, it can be a little soft.
• 4:51
  Speaker 2: Well, edgegen AI could solve that.
• 4:53
  Speaker 2: Real-time image enhancement and super resolution happening right on the phone.
• 4:57
  Speaker 2: You snap a pic, boom instant improvement.
• 4:59
  Speaker 2: What's really wild is that they're starting to build this AI processing directly into the image sensors.
• 5:06
  Speaker 1: It's like having a pro photo editor built right into your camera.
• 5:09
  Speaker 2: Pretty much.
• 5:10
  Speaker 1: Now, beyond our personal devices, what about applications in like video surveillance or health monitoring?
• 5:16
  Speaker 1: I can see edge gen AI being huge there.
• 5:19
  Speaker 2: Oh, definitely Think about smart security systems, right?
• 5:21
  Speaker 2: Not only can they detect something fishy going on, but they can actually explain what's happening and suggest what to do.
• 5:27
  Speaker 2: All locally and in healthcare man.
• 5:31
  Speaker 2: The possibilities Edgegen AI could be helping with diagnoses, you know, making those interactions between patients and devices way more intuitive.
• 5:38
  Speaker 2: And again, all that sensitive data it stays secure and local.
• 5:42
  Speaker 1: Okay, that makes sense.
• 5:42
  Speaker 1: So I'm starting to get the why.
• 5:44
  Speaker 1: It's about giving power back to the users, making things more private and just making AI more accessible and immediate.
• 5:51
  Speaker 1: Right Got it.
• 5:52
  Speaker 1: So how are they actually doing this?
• 5:54
  Speaker 1: How are they shrinking these like massive AI models down to fit on our devices?
• 5:59
  Speaker 1: Yeah, our research looked at a bunch of different approaches and they kind of fall into these neat categories based on the specific tasks that AI is supposed to do.
• 6:06
  Speaker 2: Like specific jobs for specific AI right.
• 6:08
  Speaker 1: Right.
• 6:08
  Speaker 1: So first up, let's talk about visual question answering, or VQA.
• 6:11
  Speaker 1: This is all about getting a device to understand an image and then answer questions about it, like it's looking at a picture and having a conversation with you, yeah, like hey phone, what color is that car?
• 6:21
  Speaker 2: And it tells you.
• 6:21
  Speaker 1: Exactly phone, what color is that car, and it tells you exactly.
• 6:23
  Speaker 1: So one of the early breakthroughs in this area was something called mobi vqa.
• 6:27
  Speaker 1: This came out back in 2022.
• 6:29
  Speaker 1: The researchers were working with these existing really powerful vqa models, you know, like lx mert, xlx mert and vilti big names yeah, big complex models and they figured out ways to optimize them so they could run smoothly on mobile devices.
• 6:46
  Speaker 2: Like, what kind of optimizations?
• 6:47
  Speaker 1: Well, they did some clever things like early exit, where the processing just stops, if it's obvious the AI has the answer, or if doing more calculations won't really help.
• 6:57
  Speaker 2: So it doesn't waste time.
• 6:58
  Speaker 1: Right, and then they use this thing called question aware pruning, which basically helps the model focus its attention on the most important parts of the image, you know, the stuff that's actually relevant to the question.
• 7:09
  Speaker 2: So it's not getting bogged down by unnecessary details.
• 7:12
  Speaker 1: Exactly.
• 7:12
  Speaker 1: They even took these really accurate but super demanding models ones that usually focus on specific regions and adapted them to be more efficient by working on a grid system instead.
• 7:22
  Speaker 1: So when they tested these tweaked models on devices like the NVIDIA Jetson TX2 and a Pixel 3 XL phone, the results were impressive.
• 7:30
  Speaker 1: The latency, or the time it takes to get an answer it dropped like crazy, went from seconds down to just milliseconds, and they used less energy too.
• 7:39
  Speaker 1: All while keeping the accuracy pretty much the same.
• 7:41
  Speaker 1: This Mobi VQA it was like an early win that showed hey, we can actually do this, we can bring this complex visual understanding to mobile devices.
• 7:49
  Speaker 2: It's like that moment when you realize something huge is about to happen.
• 7:56
  Speaker 1: Right, okay, so then in 2023, researchers came up with something called the Bilaterally Slimmable Transformer, or BST, bst Catchy.
• 8:01
  Speaker 1: It's a mouthful Okay, but basically it's this framework for training, a single VQA model that can be shrunk down into smaller versions, these submodels, without having to retrain the whole thing from scratch.
• 8:12
  Speaker 2: Oh, that's smart.
• 8:12
  Speaker 2: So it's like having one master model that you can customize for different devices.
• 8:16
  Speaker 1: Exactly Different levels of processing power, no problem.
• 8:19
  Speaker 1: This BST thing.
• 8:20
  Speaker 1: They used it on models like Mencan, uniter and ClipFill and they were able to run some of these smaller MCN versions, like one that was less than 10 million parameters.
• 8:29
  Speaker 2: Tiny.
• 8:30
  Speaker 1: Tiny.
• 8:31
  Speaker 1: They got to run it on phones with those Snapdragon 888, dimensity 1100, and Snapdragon 660 chipsets, and the response times were super fast, like tens to hundreds of milliseconds.
• 8:43
  Speaker 1: This adaptability is key, right yeah, because edge devices come in all shapes and sizes.
• 8:47
  Speaker 2: Totally.
• 8:47
  Speaker 2: And then what happened in 2024?
• 8:49
  Speaker 1: Well, in 2024, there was a thing called TinyVQA, and this one was specifically designed for tiny ML hardware, you know, those super resource constrained devices, these researchers.
• 8:58
  Speaker 1: They were training this model for a really cool application.
• 9:01
  Speaker 1: They wanted to use it to assess damage after, like a natural disaster, using images from drones.
• 9:07
  Speaker 2: Oh wow, that's smart.
• 9:08
  Speaker 1: Yeah.
• 9:09
  Speaker 1: So they use this technique called knowledge distillation or KD, which is basically like a teacher, student and thing where a smaller model learns from a bigger, smarter one.
• 9:17
  Speaker 1: And then they did this thing called post-training quantization, basically shrinking those numbers down to save space and speed things up.
• 9:23
  Speaker 1: Clever and the results.
• 9:24
  Speaker 1: Oh man, this tiny VQA.
• 9:27
  Speaker 1: It was amazingly accurate for its size.
• 9:30
  Speaker 1: It was only 339 kilobytes Kilobytes Compared to a baseline model that was almost 500 megabytes.
• 9:37
  Speaker 1: It's like they shrunk an entire encyclopedia down to like a post-it note.
• 9:41
  Speaker 2: Yeah.
• 9:41
  Speaker 1: And they actually got this thing running on a GAPA processor, on a drone.
• 9:44
  Speaker 1: The latency was crazy fast only 56 milliseconds and used very little power.
• 9:50
  Speaker 2: Talk about efficient.
• 9:51
  Speaker 2: Okay, anything else from 2024?
• 9:53
  Speaker 2: .
• 9:53
  Speaker 1: Yeah, there's one more.
• 9:53
  Speaker 1: Researchers from Samsung developed a brand new transformer architecture and they paired it with this thing called Inteed Quantization, so specifically for mobile devices.
• 10:01
  Speaker 1: They tested it on a standard VQA data set and after they applied their quantization magic, the model size shrank to just 58 megabytes.
• 10:09
  Speaker 2: Impressive shrinkage.
• 10:11
  Speaker 1: Right, and it was crazy fast on a Galaxy S23 phone only 2.8 milliseconds to get an answer.
• 10:16
  Speaker 1: So there you have it Lots of cool research happening in VQA Getting AI to understand images and answer questions about them right on our devices.
• 10:26
  Speaker 2: It's like magic.
• 10:27
  Speaker 1: Now let's switch gears a bit.
• 10:28
  Speaker 1: What about image and video captioning?
• 10:31
  Speaker 1: This is where AI can actually generate a description, like a sentence, for what it sees.
• 10:36
  Speaker 2: Oh, like a fluffy cat is napping on a red blanket.
• 10:40
  Speaker 1: Exactly so.
• 10:40
  Speaker 1: There's this model called LightCap came out in 2023.
• 10:43
  Speaker 1: It's a super lightweight image captioning model made specifically for mobile.
• 10:47
  Speaker 1: It has all these different parts, like an image encoder to understand the picture, a concept extractor to pick out the important stuff, a thing to link the visual and the text information, and then this little language model called TinyBird.
• 10:58
  Speaker 2: TinyBird, it's all about being tiny.
• 11:00
  Speaker 1: Right.
• 11:00
  Speaker 1: And even with all those parts, it was still small only 40 million parameters, about 112 megabytes.
• 11:06
  Speaker 1: They tested it on a Huawei P40 phone and it had a latency of only 188 milliseconds.
• 11:11
  Speaker 1: Plus, it was super accurate with its captions.
• 11:13
  Speaker 2: Okay, so what about video captioning?
• 11:15
  Speaker 2: That seems way more complex, right?
• 11:17
  Speaker 1: Definitely.
• 11:17
  Speaker 1: But in 2024, there was this project called Tiny V2A and they focused on this even harder task of describing future actions in videos, specifically of mice, mice, mice.
• 11:29
  Speaker 1: They used this thing called a Video MAE Transformer, which is a mouthful and a language model called OPT, and they used all those techniques we talked about, like knowledge distillation and parameter efficient fine tuning, to make it all work.
• 11:43
  Speaker 1: The smallest version of their model, which still had 169 million parameters, ran in under a second on a Raspberry Pi, both a Pi 4 and a Pi 5.
• 11:51
  Speaker 1: So not just saying what's happening in the video, but predicting what might happen next.
• 11:55
  Speaker 1: Yeah, pretty impressive, especially on a Raspberry Pi.
• 11:57
  Speaker 2: Yeah, for sure.
• 11:58
  Speaker 1: Oh, and then there was Vela.
• 11:59
  Speaker 1: This one's a little different.
• 12:00
  Speaker 1: It's what they call a visual language model, or a VLM, and it's designed to run on the NVIDIA Jetson Orin, which is a pretty powerful little computer.
• 12:08
  Speaker 1: These VLMs are basically large language models that can also deal with visual stuff.
• 12:12
  Speaker 1: Yeah, so they can do image and video captioning, vqa, all that.
• 12:17
  Speaker 1: This Vela thing outperformed some of the best models out there, and they used 4-bit quantization to make it efficient enough for edge devices.
• 12:24
  Speaker 2: So, even these big multimodal models, they're finding ways to make them work the edge.
• 12:28
  Speaker 1: Yep, okay.
• 12:29
  Speaker 1: Moving on to another super cool application, let's talk about making our devices actually speak in a more natural way.
• 12:36
  Speaker 2: Text-to-speech, or TTS.
• 12:38
  Speaker 2: This is where things get really interesting.
• 12:40
  Speaker 1: Because a robot voice is not a good look.
• 12:42
  Speaker 2: Not anymore.
• 12:43
  Speaker 2: Back in 2022, there was this project called Nix TTS.
• 12:46
  Speaker 2: It's a non-autoregressive end-to-end TTS model that learned from a bigger model called VITS Nix TTS.
• 12:54
  Speaker 2: It's small, only about 5 million parameters and only 21 megabytes.
• 12:58
  Speaker 2: Now, it wasn't quite as natural sounding as VITS, but it was way faster on a Raspberry Pi 3B.
• 13:03
  Speaker 1: So sometimes a tiny bit of quality loss is worth it for a big gain in speed.
• 13:06
  Speaker 2: Especially for real-time stuff.
• 13:08
  Speaker 1: Oh, and there was another one from 2022 called Piper, a neural TTS system optimized for the Raspberry Pi 4, also based on VITS.
• 13:15
  Speaker 1: This one had about 15 million parameters.
• 13:18
  Speaker 1: So more options, more speed.
• 13:19
  Speaker 1: And then, if you need even more efficiency, there's Efficient Speech from 2023.
• 13:24
  Speaker 1: It's a non-autoregressive two-stage model.
• 13:26
  Speaker 1: It creates like a visual representation of sound and then uses this small vocoder called HiFiJAN to actually make the audio.
• 13:33
  Speaker 1: This one was super tiny only 1.2 million parameters for the acoustic model, and it ran on a Raspberry Pi 4B as fast as Fast Speech 2, which is a much bigger model.
• 13:44
  Speaker 2: So basically, amazing quality packed into a tiny model.
• 13:47
  Speaker 1: Pretty much.
• 13:48
  Speaker 1: And then there was Fast Stream Speech, also from 2023.
• 13:51
  Speaker 1: This one also used a two-stage approach, but added this cool streaming mechanism to make it even faster, especially on like smartphones.
• 13:58
  Speaker 1: It sounded as good as larger models, but they got it running super fast on a MediaTek Helio G35 processor.
• 14:03
  Speaker 2: Super fast.
• 14:04
  Speaker 2: So we've got the main TTS models, but don't forget about the vocoders, right?
• 14:08
  Speaker 2: That's the part that takes the sound features and actually creates the audio waves.
• 14:11
  Speaker 2: There are some cool projects out there, like TinyVocos and Bunched LPCnet, both designed for tiny devices that show you can even get great sound out of these smaller, specialized components.
• 14:22
  Speaker 1: So we've talked about understanding what our devices see and making them speak, but what about making what we hear clearer?
• 14:29
  Speaker 2: Ah, you're talking about speech enhancement, taking noisy audio and making it better.
• 14:33
  Speaker 1: Exactly so.
• 14:34
  Speaker 1: There's this cool model from 2024 called TransFilm.
• 14:37
  Speaker 1: It's an audio super resolution network for mobile devices.
• 14:40
  Speaker 1: It uses these things called transformer blocks and a U-net architecture which is known for doing great stuff with audio and image processing.
• 14:48
  Speaker 1: It takes low quality audio and makes it sound much better.
• 14:51
  Speaker 1: It's small about a million parameters, got good scores on audio quality tests and ran at a decent speed on a MyZoo 16S smartphone.
• 14:59
  Speaker 2: So imagine being able to clean up your voice calls or recordings right on your phone.
• 15:02
  Speaker 1: Yep.
• 15:03
  Speaker 1: And then, specifically for those super noisy situations like think industrial settings, there's this other model from 2024 that was designed to run on microcontrollers.
• 15:11
  Speaker 1: They took a smart speech enhancement system and adapted it to run on an STM32 H735 MCU, which is a tiny computer that could be used in like a safety helmet.
• 15:22
  Speaker 2: In a helmet.
• 15:23
  Speaker 1: Yeah, so this system has a part that detects emergency signals and then another part that's a DCRN, a deep, complex recurrent network for noise reduction.
• 15:32
  Speaker 1: They made a model really small, tweaked some stuff and used in date eight quantization to make sure it could actually run on that tiny microcontroller.
• 15:40
  Speaker 1: The result it actually worked and it performed as well as bigger models with low latency and low energy usage.
• 15:46
  Speaker 1: It's amazing what they can do with these small devices these days.
• 15:49
  Speaker 2: It's really impressive.
• 15:50
  Speaker 2: Okay, so let's talk about language understanding and translation, neural machine translation, or NMT.
• 15:56
  Speaker 2: This is all about getting AI to translate languages automatically.
• 15:59
  Speaker 1: And transformer models have been awesome at this, but they can be a bit resource heavy right, yeah, they can be.
• 16:05
  Speaker 2: So back in 2022, researchers developed Transformer DMB, which basically added some dynamic multi-branch layers to the standard transformer architecture.
• 16:14
  Speaker 2: Now, it wasn't the absolute fastest on a Raspberry Pi 4B compared to some even smaller transformer versions, but when they combined knowledge distillation with 8-bit quantization, they found a really good balance between size, performance and efficiency.
• 16:27
  Speaker 1: So it was a good compromise.
• 16:28
  Speaker 1: And then also in 2022, there was Hypoformer, which used a technique called hybrid tensor train decomposition.
• 16:35
  Speaker 1: It's a mouthful, but it basically helps compress and speed up transformer models, particularly for NMT, and by using knowledge distillation during training, they got it to perform just as well as a regular transformer on a Raspberry Pi 4B, but with way fewer parameters and much faster inference times.
• 16:51
  Speaker 2: All those techniques are crucial if we want real-time translation right on our devices.
• 16:55
  Speaker 1: Totally Now.
• 16:56
  Speaker 1: How about using AI to modify and transform images for enhancing privacy.
• 17:02
  Speaker 2: Yeah, so this is where Neural Style Transfer, or NST, comes in.
• 17:05
  Speaker 2: It's like taking the style of one image and applying it to another, or even changing an image based on a text description For privacy.
• 17:12
  Speaker 2: This is cool because you can use it to blur faces or change features while still keeping the rest of the image understandable.
• 17:19
  Speaker 1: So it's like an artistic way to protect people's privacy.
• 17:21
  Speaker 2: Exactly.
• 17:22
  Speaker 2: And in 2024, they developed a super lightweight NST model based on Gynet, specifically for anonymization.
• 17:30
  Speaker 2: This thing was tiny only 0.6 million parameters and when they quantized it to 8-bit it ran beautifully on a Raspberry Pi 4B and an STM32H743 microcontroller Low latency, minimal energy.
• 17:44
  Speaker 2: It was perfect for on-device privacy protection.
• 17:46
  Speaker 1: That's awesome.
• 17:47
  Speaker 1: Also, in 2024, some researchers created a really lightweight video NST model.
• 17:51
  Speaker 1: They distilled knowledge from a larger model and added this efficient feature transformation module.
• 17:56
  Speaker 1: It ended up being a tiny 2.67 megabyte model that could style transfer videos at almost 41 frames per second on those NVIDIA Jetson devices.
• 18:04
  Speaker 2: So real-time video anonymization totally possible.
• 18:07
  Speaker 1: And then there's text-to-image style transfer.
• 18:08
  Speaker 2: Oh yeah, this is cool.
• 18:09
  Speaker 1: There are these two projects from 2024, fastlab Styler and EdgeClip Styler.
• 18:14
  Speaker 1: They basically take a text prompt like make this picture look like a Van Gogh, and they apply that style to an image.
• 18:20
  Speaker 1: Edgeclip Styler specifically used a super efficient model for understanding the text to make it better suited for edge devices, though it did take about 15 seconds to do its thing on a Raspberry Pi 3B plus key.
• 18:32
  Speaker 1: So there's room for improvement there.
• 18:34
  Speaker 2: But it shows what's possible.
• 18:35
  Speaker 1: And then there's Mosaic from 2023.
• 18:37
  Speaker 1: This one's cool because it can apply different styles to different objects in an image, and it ran at a pretty decent speed on a Snapdragon 8 Gen 1 processor.
• 18:46
  Speaker 2: So you could have like one object that looks like a Picasso and another that looks like a Monet all in the same image.
• 18:50
  Speaker 1: Exactly.
• 18:51
  Speaker 2: Wild.
• 18:52
  Speaker 1: Okay, now what about directly swapping faces in videos or pictures?
• 18:55
  Speaker 1: Another potential privacy tool, right.
• 18:58
  Speaker 2: Face swapping yeah, definitely a hot topic, and researchers are looking into how to do it right on our devices.
• 19:03
  Speaker 2: Back in 2022, there was a study that showed you could actually swap faces in real time on a phone with a MediaTank Dimensity 1100 processor, and they didn't even have to specifically optimize the model for that task.
• 19:17
  Speaker 1: So the tech was already there.
• 19:18
  Speaker 2: Yeah, it's just a matter of making it smaller and faster.
• 19:20
  Speaker 2: And there's FinetGen from 2023 and SimSwap from 2024.
• 19:25
  Speaker 2: These were specifically designed for anonymization using face swapping.
• 19:29
  Speaker 2: Finetgen actually ran pretty well on a Kendrick K210 microcontroller, which is a seriously tiny device, and they got the smallest version of SimSwap running really fast on both a Raspberry Pi 4 and a STM32H743 microcontroller.
• 19:43
  Speaker 1: It's crazy how far they've come.
• 19:45
  Speaker 2: Right.
• 19:45
  Speaker 1: Okay, so we've covered a bunch of specific applications, but our research also found a whole bunch of other visual processing tasks that are making their way to the edge.
• 19:52
  Speaker 2: Oh yeah, tons of stuff.
• 19:54
  Speaker 2: We're talking super resolution to make pictures and videos crisper, in-painting to fill in missing parts of images, general image and video restoration, different types of image enhancement and denoising stuff happening right in the camera sensor with image signal processing or ISP, and even things like bokeh rendering, which creates those cool blurry backgrounds and photos.
• 20:16
  Speaker 2: There are so many different models being developed with different sizes and goals, but smartphones are definitely the most popular platform for all of this and the processing times.
• 20:25
  Speaker 2: They range from milliseconds to seconds.
• 20:28
  Speaker 2: There were even competitions like Mobile AI and AM in 2022 that really pushed things forward, especially in areas like super resolution and making it more energy efficient.
• 20:37
  Speaker 2: There was this one called TinyLUT that could restore images really well and it was crazy small and fast, even on a Xiaomi 11 smartphone and a Raspberry Pi 4B.
• 20:47
  Speaker 1: So many cool things happening in the visual processing world.
• 20:49
  Speaker 2: For sure.
• 20:49
  Speaker 1: And then there's the even crazier stuff like what about actually generating images?
• 20:53
  Speaker 1: Like creating them from scratch?
• 20:55
  Speaker 2: You're talking about text-to-image diffusion models.
• 21:02
  Speaker 1: This is where you type in something like a cat wearing a hat, riding a bicycle, and the AI actually creates an image of that Exactly, and that's where these big models like stable diffusion come in.
• 21:07
  Speaker 1: Now, getting something like that to run on a phone it's still tough, but there are some promising early results.
• 21:13
  Speaker 1: They've gotten it down to under 15 seconds on some of the more powerful mobile chips 15 seconds.
• 21:19
  Speaker 2: That's not bad.
• 21:20
  Speaker 1: Not bad at all, and researchers are coming up with all sorts of optimizations, new architectures trying to reduce the number of steps it takes to generate the image, models like BKSDM, edge Fusion, snapgen, snapfusion, mobile Diffusion they're all pushing the boundaries here.
• 21:34
  Speaker 1: Mobile Diffusion even managed to create images in like 0.2 seconds on the latest iPhone 15 Pro.
• 21:40
  Speaker 1: It's all about finding the right balance between model size, complexity, speed and image quality.
• 21:46
  Speaker 2: And what about running language models directly on our devices?
• 21:50
  Speaker 1: Ah yes, large language models, or LLMs, have been getting all the attention, but they are huge, right.
• 21:55
  Speaker 1: So the focus has shifted to these smaller, more efficient versions what they call small language models, or SLMs that are much better suited for running on our phones and other devices.
• 22:05
  Speaker 2: Yeah, these SLMs have way fewer parameters than the giant LLMs that power like chat GPTs, so they're much more realistic for edge devices.
• 22:18
  Speaker 1: Exactly.
• 22:18
  Speaker 1: And there are some really cool ones out there already, like Fi3, mini, mini, cpm, mobile, lm, mobile, lama, phonelm, and they're using all sorts of tricks to make them work well.
• 22:24
  Speaker 1: They're optimizing the transformer architecture, doing crazy quantization, sharing parts of the model to save memory and even sharing entire layers to speed things up.
• 22:33
  Speaker 1: Like PhoneLM can process 58 tokens per second on a Xiaomi 14 smartphone 58 tokens.
• 22:39
  Speaker 2: That's pretty fast.
• 22:40
  Speaker 1: Right Now, these SLMs.
• 22:41
  Speaker 1: They might not be as knowledgeable or as good at reasoning as the giant LMs yet, but there's still a big step towards having natural language understanding and generation right on our devices without needing the cloud.
• 22:52
  Speaker 2: Totally agree.
• 22:53
  Speaker 1: So we've looked at a ton of stuff today, from understanding images to generating speech to translating languages.
• 23:00
  Speaker 1: It's a lot to process, but there's some common threads running through all of this right.
• 23:04
  Speaker 2: Oh, absolutely.
• 23:05
  Speaker 2: A few key optimizations keep popping up again and again, Like first off, you've got to design efficient models, ones that are naturally smaller and require less computation.
• 23:16
  Speaker 1: Starting with a good foundation.
• 23:17
  Speaker 2: Exactly.
• 23:17
  Speaker 2: Then there's quantization, which basically means shrinking those numbers down, making them less precise, but way more efficient.
• 23:25
  Speaker 2: That's a huge one.
• 23:26
  Speaker 2: And knowledge distillation you know that teacher-student thing, that helps a lot too.
• 23:31
  Speaker 1: Yeah, it's like giving the smaller model a cheat sheet from the bigger one.
• 23:34
  Speaker 2: Exactly.
• 23:34
  Speaker 2: And then there's pruning, which is like trimming the fat off a neural network, getting rid of the connections that aren't really that important.
• 23:41
  Speaker 2: It's the combination of all these strategies that's really making edge gen AI possible.
• 23:47
  Speaker 1: Okay, so lots of exciting developments, but what about the big picture?
• 23:50
  Speaker 1: What are the trends?
• 23:52
  Speaker 1: Where's all of this headed?
• 23:53
  Speaker 2: Well, one thing is clear Research in edge gen AI is exploding.
• 23:58
  Speaker 2: There are more and more publications every year, with new breakthroughs happening all the time, and what's really important to remember is that we focused specifically on actually deploying gen AI on edge devices.
• 24:10
  Speaker 2: That's different from other research that might talk about edge AI in general.
• 24:14
  Speaker 1: Right, because it's one thing to talk about it and another to actually do it.
• 24:17
  Speaker 2: Exactly, and all those benefits we talked about earlier independence from the cloud, more privacy, faster speeds, lower costs, the ability to scale up AI massively all of that is driving this intense research, and it's opening up possibilities for applications we haven't even thought of yet.
• 24:34
  Speaker 1: Yeah, like the whole field is just exploding.
• 24:36
  Speaker 2: It is, and when you look at the research, it's clear that smartphones are leading the way.
• 24:40
  Speaker 2: They're the most popular platform, with over 50 different processors from six different manufacturers being used, and the big chip companies Qualcomm, mediatek, google they're all pouring money into developing new hardware that can handle these Gen AI workloads.
• 24:55
  Speaker 1: It's our race.
• 24:56
  Speaker 2: It is a race hardware that can handle these Gen AI workloads.
• 24:59
  Speaker 2: It's our race.
• 25:00
  Speaker 2: It is a race, and while most of the research is happening in visual processing, we're seeing more and more work in natural language processing and audio stuff too, so it's spreading Big time.
• 25:06
  Speaker 1: Right.
• 25:06
  Speaker 2: And you know, one thing that's really needed going forward is more in-depth analysis of how these models perform on devices in the real world.
• 25:13
  Speaker 2: Accuracy is great, but we need to understand latency, model size, storage space and especially energy consumption.
• 25:22
  Speaker 1: Because a model that drains your battery in five minutes isn't very useful.
• 25:26
  Speaker 2: Exactly.
• 25:26
  Speaker 2: We need to know how much energy it takes to run these things on a device compared to doing it in the cloud.
• 25:31
  Speaker 2: And then, of course, there's the environmental impact, which is something we need to think about too Good point.
• 25:36
  Speaker 2: And finally, there's the whole idea of agentic AI, giving devices the ability to learn, adapt and make decisions on their own, without needing the cloud.
• 25:45
  Speaker 2: Edge Gen AI could really push that forward.
• 25:48
  Speaker 1: Imagine having devices that are truly intelligent and proactive.
• 25:52
  Speaker 1: It's kind of mind-blowing.
• 25:54
  Speaker 2: It is.
• 25:54
  Speaker 1: So, to wrap this all up, it's pretty obvious that running these complex Gen AI models on our everyday devices it's not science fiction anymore.
• 26:03
  Speaker 1: It's happening right now.
• 26:04
  Speaker 2: And the progress in just the last few years has been incredible.
• 26:07
  Speaker 2: We're talking about faster, more private and more accessible AI for everyone.
• 26:13
  Speaker 1: And while there are definitely still challenges, the speed of innovation is just insane, both in how they're optimizing these models and in developing new hardware.
• 26:21
  Speaker 1: The future of Edge Gen AI it's looking pretty bright.
• 26:24
  Speaker 2: And that brings us to like the ultimate question.
• 26:27
  Speaker 2: Think about what it means to have truly intelligent generative devices all around us.
• 26:33
  Speaker 2: How will that change how we interact with technology?
• 26:36
  Speaker 2: How will it change our lives?
• 26:38
  Speaker 2: What new possibilities will open up when our devices can not only understand, but also create and act independently?
• 26:44
  Speaker 2: These are questions worth thinking about.
• 26:47
  Speaker 1: Big questions for sure.
• 26:48
  Speaker 1: Well, we hope this deep dive has given you a lot to think about.
• 26:51
  Speaker 1: Maybe you'll even explore some of this research yourself.
• 26:53
  Speaker 2: There's a lot to discover.
• 26:55
  Speaker 1: Thanks for joining us.
• 26:55
  Speaker 1: See you next time.
• 26:56
  Speaker 1: Deep divers.