Latest / EDGE AI POD / From Fragments to Foundation: The Sound of Progress in Edge Audio AI

Transcript

From Fragments to Foundation: The Sound of Progress in Edge Audio AI · EDGE AI POD

• 0:22
  SPEAKER_00: I think you are going to speak about uh industry applications and the deployments uh of Genai D edge.
• 0:32
  SPEAKER_00: So whenever you feel ready to share your presentation, please uh please do so.
• 0:38
  SPEAKER_00: And uh and the floor is yours.
• 0:41
  SPEAKER_00: Is it visible?
• 0:42
  SPEAKER_01: Thank you.
• 0:43
  SPEAKER_01: Thank you so much, Danilo.
• 0:45
  SPEAKER_01: Is my presentation visible?
• 0:48
  SPEAKER_01: Yes, it is.
• 0:50
  SPEAKER_01: It is.
• 0:50
  SPEAKER_01: Oh, okay.
• 0:51
  SPEAKER_01: Thanks.
• 0:51
  SPEAKER_01: Thank you so much.
• 0:52
  SPEAKER_01: Yeah.
• 0:53
  SPEAKER_01: Uh thank you, Danilo.
• 0:55
  SPEAKER_01: So um uh today uh so my name is Anir Ban, and uh I am representing Vipro Limited today um to talk about some of the real-world uh deployment and industry applications that we are seeing.
• 1:13
  SPEAKER_01: Uh this is the agenda uh for my presentation.
• 1:17
  SPEAKER_01: So some background on where uh we come from.
• 1:21
  SPEAKER_01: Uh what is the AGI uh relevance that we are seeing?
• 1:25
  SPEAKER_02: Sorry, Anirban.
• 1:27
  SPEAKER_02: We don't, I think you are maybe it's not another presentation you're showing because we just see your tile slide right now.
• 1:35
  SPEAKER_02: I don't know exactly if you want to try again.
• 1:42
  SPEAKER_02: Because I I think it's just the one that you have uploaded previously, but you're not sharing yourself.
• 1:49
  SPEAKER_01: Let me share that.
• 2:03
  SPEAKER_01: Is it visible now?
• 2:07
  SPEAKER_02: Um let me see.
• 2:09
  SPEAKER_02: No, I think it's still the other one.
• 2:12
  SPEAKER_02: I think you or maybe now.
• 2:13
  SPEAKER_02: Let's see, is that the one?
• 2:14
  SPEAKER_02: And then you just go full screen.
• 2:17
  SPEAKER_01: Yeah, table of table of content.
• 2:20
  SPEAKER_02: Yeah, table of content.
• 2:20
  SPEAKER_02: Yeah, and now you just need to go to full screen because we just have all your slides on the left.
• 2:29
  SPEAKER_02: And then I'll let you take it over.
• 2:31
  SPEAKER_01: I've gone to full screen.
• 2:36
  SPEAKER_02: No, it's not.
• 2:38
  SPEAKER_02: We are still seeing everything on the left to all your slides.
• 2:45
  SPEAKER_02: No, are you using different screens maybe?
• 2:57
  SPEAKER_02: Or when you just go to a slideshow.
• 3:05
  SPEAKER_01: Okay, let me share again.
• 3:33
  SPEAKER_02: No, it's not visible in the screen.
• 3:35
  SPEAKER_02: If not, yeah, we just have to go that way.
• 3:40
  SPEAKER_02: And if you just go and take slide show that it's showing your whole screen.
• 3:54
  SPEAKER_02: Right, Danilo, you can also see that, right?
• 3:56
  SPEAKER_02: It's not in full screen mode.
• 3:57
  SPEAKER_00: He is just uh I see no in no not in uh full screen, no, no.
• 4:05
  SPEAKER_02: Okay.
• 4:07
  SPEAKER_00: No, okay.
• 4:08
  SPEAKER_00: Any technical difficulty, Arniban?
• 4:12
  SPEAKER_02: Yeah, if not, yeah, we have to go that way, but it's not uh really what we want, but it's okay.
• 4:18
  SPEAKER_01: I'm sharing in full screen, but not sure uh yeah.
• 4:24
  SPEAKER_00: So maybe you have a multiple screen.
• 4:26
  SPEAKER_01: You can share the presentation that I uploaded.
• 4:30
  SPEAKER_02: Yeah, I can share that one, but that's the same.
• 4:32
  SPEAKER_01: No, I don't want that actually.
• 4:36
  SPEAKER_01: Uh without the video, it's fine.
• 4:38
  SPEAKER_01: I think we will continue.
• 4:39
  SPEAKER_02: Okay, then we go with this one.
• 4:41
  SPEAKER_02: Okay, can you move forward here in that slide?
• 4:43
  SPEAKER_01: And then I'll yeah, you can go ahead uh to the next slide.
• 4:50
  SPEAKER_02: Okay, so you want me to okay, okay.
• 4:53
  SPEAKER_02: I'll just follow you then.
• 4:57
  SPEAKER_01: Yeah, yeah, so uh this is the um agenda and then um uh some background on our uh team.
• 5:07
  SPEAKER_01: So uh for those um who are not aware, so Vipro uh is a leading global information technology consulting and uh software development company.
• 5:18
  SPEAKER_01: Uh we are present in more than 167 countries and we focus on different areas, including AI and um cloud computing and data analytics, for example.
• 5:29
  SPEAKER_01: So um I come from a background of embedded development and uh currently for the last five years working on AI.
• 5:38
  SPEAKER_01: Um, and I would like to acknowledge the bigger team uh behind this as well.
• 5:43
  SPEAKER_01: Yeah, we can go to the next slide.
• 5:46
  SPEAKER_01: Thank you.
• 5:48
  SPEAKER_01: So, to talk about the use cases, um so what we see uh are uh different trends in AI.
• 5:57
  SPEAKER_01: Um so there are advancements in the generative AI models uh with a lot of multimodal AI models coming in, and uh these uh models are driving new use cases as well as uh they are enabling improvements in the older implementation methods.
• 6:21
  SPEAKER_01: A lot of the implementation for multimodal AI uh happens on the cloud, but uh there are certain concerns with that in terms of mainly the data privacy part of it, where people may not be ready to share all their information to the cloud as well as the uh need for the availability of a high bandwidth network.
• 6:48
  SPEAKER_01: So, in some use cases, it is uh beneficial to have the inferencing done in an offline mode without a dedicated network connectivity.
• 7:01
  SPEAKER_01: So, uh, in terms of looking at um embedded devices, uh we looked at some of the different uh devices and we uh saw that like the printers and scanners is one category whether where there may be a lot of uh applications of multimodal AI.
• 7:22
  SPEAKER_01: Today the printers and scanners that we see are mostly passive, they don't uh basically uh process the content themselves, they struggle with a lot of uh inconsistent formats and layouts, and uh basically these uh formats are all unstructured.
• 7:45
  SPEAKER_01: Uh, a lot of the documents come in multilingual formats, in specially in enterprise context.
• 7:54
  SPEAKER_01: And uh with the traditional rule-based approaches and um even traditional deep learning techniques, uh, there is very limited reformatting support which is available.
• 8:10
  SPEAKER_01: So, with the coming of multimodal AI, um basically these uh passive endpoints can become more intelligent, and uh they can help in uh basically changing these unstructured formats into a more structured format.
• 8:28
  SPEAKER_01: Uh they have the power to translate the content uh from one language to other, and uh they have this special ability to understand both image and the text.
• 8:43
  SPEAKER_01: Uh we can go to the next slide.
• 8:47
  SPEAKER_01: Yeah.
• 8:48
  SPEAKER_01: So um we took one use case uh mostly on the document uh understanding part of it.
• 8:55
  SPEAKER_01: So a lot of uh multimodal AI models are available uh even in the open source.
• 9:02
  SPEAKER_01: For example, we have the Gwen 2.5 VL, uh which is a very popular model.
• 9:09
  SPEAKER_01: It is capable of handling text, images, and even other modalities.
• 9:15
  SPEAKER_01: Uh we have very uh special uh models like layout LM V3 from Microsoft, which is uh basically uh trained for document understanding.
• 9:27
  SPEAKER_01: Then there are other popular models like Donut and TATR from Microsoft for table understanding.
• 9:35
  SPEAKER_01: And these are the multimodal series of models, whereas we have another series more from like the diffusion models.
• 9:43
  SPEAKER_01: So we have a model from Flux, so uh they are very creative in uh generating new content.
• 9:54
  SPEAKER_01: We can go to the next slide.
• 10:02
  SPEAKER_01: Uh this is our uh solution approach that how um on the printers and scanners basically we can bring in uh these multimodal AI models.
• 10:12
  SPEAKER_01: So we have this input uh image or PDF formats, as well as uh we may have an user input that may be in the form of a prompt or it can be uh by voice directions.
• 10:28
  SPEAKER_01: So once this is uh acquired, we do some conversion, maybe we convert PDF to image, then we do a lot of uh pre-processing tasks like we do resolution conversion, we do page segmentation, uh, noise reduction, and so on.
• 10:47
  SPEAKER_01: And then pretty much we can use the visual language models directly.
• 10:55
  SPEAKER_01: Uh for uh optimized inference, we can do quantization and uh without much supervision, the VLMs are capable of uh interpreting the content, identifying key elements, and then uh restructuring the layouts.
• 11:13
  SPEAKER_01: Uh they can they are also capable of uh generating in the format that we desire, and then we have some post-processing activities where we clean up uh prepare in the format that we desire basically.
• 11:30
  SPEAKER_01: Uh, the interesting part of this is uh there is uh without even doing uh lot of fine-tuning of the base foundation models, we can get uh quite decent uh results.
• 11:46
  SPEAKER_01: So, primarily in our implementation uh we have used uh two models.
• 11:53
  SPEAKER_01: Uh so one is the Quen 2.5 VL model, uh, it excels in handling complex visual inputs, including images of different sizes and so on, while still maintaining a good uh uh linguistic performance.
• 12:11
  SPEAKER_01: Uh this model uh so we have tried with 3 billion parameters, 7 billion parameter, and so on, and um, the configuration that we used had a GPU RAM of 16 GB basically.
• 12:27
  SPEAKER_01: Uh on the other hand, we have also used the flux diffusion model.
• 12:32
  SPEAKER_01: Uh, this uh has a hybrid architecture of both uh multimodal and parallel diffusion transformer blocks, and uh it had a size of around 12 billion parameters.
• 12:44
  SPEAKER_01: Uh so this also needs uh uh VRAM of around 24 GB.
• 12:53
  SPEAKER_01: Uh we can go to the next slide.
• 12:57
  SPEAKER_01: So um here we have uh primarily two use cases.
• 13:02
  SPEAKER_01: Um is on extracting the tabula data and reformatting to the graphs.
• 13:11
  SPEAKER_01: So we can import a PDF document which has tables and other textual content.
• 13:22
  SPEAKER_01: So the VLMs are um uh quite skilled to extract the table information, and then once we have the tabular data, we can convert it into a different representation, say a pie chart or a bar chart, and uh then uh we can overwrite say the tabular data with this information.
• 13:46
  SPEAKER_01: So a lot of these creative uh things can be done uh with the uh documents.
• 13:53
  SPEAKER_01: Uh the next use case is more on image generation and modification.
• 14:00
  SPEAKER_01: So basically, we were thinking about uh greeting cards use case.
• 14:05
  SPEAKER_01: So with a textual prompt, uh we can um uh create a greeting card uh from the model, and then we can do incremental modification text uh steps by prompting the model.
• 14:22
  SPEAKER_01: So for example, here um uh the greeting card says happy new year 2023, and then uh with a proper text prompt we can modify the year to 2025.
• 14:34
  SPEAKER_01: So uh some of these modifications are done very simply.
• 14:39
  SPEAKER_01: Uh, then in the third step, we wanted to change the outer boundary of the image.
• 14:46
  SPEAKER_01: So we were able to do it with a simple prompt to the model.
• 14:52
  SPEAKER_01: You can go to the next slide.
• 14:59
  SPEAKER_01: This is another uh use case where uh we um uh do the text correction which I already discussed.
• 15:08
  SPEAKER_01: Then um coming to other use cases.
• 15:10
  SPEAKER_01: So, what we have seen is uh in a printer scanner use case, um automatically we can understand the scan documents and then classify them uh into specific categories.
• 15:25
  SPEAKER_01: Uh so this is uh real intelligence.
• 15:29
  SPEAKER_01: So in an enterprise context, it really helps that uh it can also trigger a notification that uh this document is intended for this group of people.
• 15:42
  SPEAKER_01: Then we have multilingual content handling ability, and um uh so these models can easily translate between different languages, uh, and this is very helpful in typical enterprises.
• 15:59
  SPEAKER_01: Another use case may be like adapting the layout for visually impaired users by increasing the font size, contrast, and simplifying the design.
• 16:13
  SPEAKER_01: Then in a scanner use case, we can say have uh some defects in the text, and uh all of those gaps can be automatically filled in by the model, and error correction can be done automatically.
• 16:30
  SPEAKER_01: We have seen other use cases like um uh when it comes to giving a print from a web page, like a lot of these advertisements and other content comes in.
• 16:42
  SPEAKER_01: So all of that can be automatically detected by the AI model and removed.
• 16:49
  SPEAKER_01: Yeah, we can go to the next slide.
• 16:51
  SPEAKER_01: Yeah, yeah.
• 16:53
  SPEAKER_01: Unfortunately, I think this um uh video will not run, but uh basically while we were showing the uh table extraction.
• 17:02
  SPEAKER_01: This is the text correction uh part of it.
• 17:05
  SPEAKER_01: Yeah, you can go to the next slide.
• 17:08
  SPEAKER_01: This is about changing the borders, and uh this is uh about changing the color of the bird by just a simple prompt to the model.
• 17:19
  SPEAKER_01: Uh it is able to update the bird color.
• 17:23
  SPEAKER_01: Yeah, you can go to the next slide.
• 17:25
  SPEAKER_01: Yeah, so this is some of the deployment aspects uh what we considered.
• 17:31
  SPEAKER_01: So uh the aim is to uh deploy it on the edge and the device, like we discussed, but uh these models still need uh uh considerable resources to run.
• 17:46
  SPEAKER_01: So uh first of all, we took a lot of steps in optimizing it.
• 17:51
  SPEAKER_01: Um, one step was basically downscaling the image.
• 17:55
  SPEAKER_01: So during the pre-processing step, we downscaled the image uh by reducing the resolution significantly so that it improves the image processing performance efficiency uh with some loss of uh accuracy.
• 18:12
  SPEAKER_01: Then these VLM models are very good in uh object localization and grounding.
• 18:20
  SPEAKER_01: So some of the tasks, such as accurately locating the objects within the images, using bounding boxes and point coordinates, so all of these can be done automatically for the VLMs, and as an application developer, we don't need to set up some complicated pipeline to do this.
• 18:40
  SPEAKER_01: So we use the power of the VLMs which are in built.
• 18:46
  SPEAKER_01: In terms of model quantization, um, we uh quantize the 7 billion parameter model using a 4-bit quantization technique, and uh we use the GGUF format for the model.
• 19:01
  SPEAKER_01: Uh, GGUF is a more efficient and flexible way of storing and using the uh models for inference and is designed to run well on uh consumer create computer hardware.
• 19:15
  SPEAKER_01: In terms of the diffusion model, we um did a lot of hyperparameter optimization.
• 19:23
  SPEAKER_01: So there are certain parameters for the diffusion model like case sampler and scheduler, and these can be tweaked for better performance by enabling better control of the sampling process, how the image gets generated, and uh by fine-tuning the sampling algorithms and parameters, uh, we could do a balance between the accuracy as well as the performance.
• 19:55
  SPEAKER_01: And last but not the least, we utilize the optimized library for Nvidia for the GPU to optimize the performance on the GPU.
• 20:06
  SPEAKER_01: So overall, like um the deployment configuration was based on an X66-based processor, an NVIDIA uh T4 GPU with a memory of around 16 GB.
• 20:20
  SPEAKER_01: Uh, all the mentioned steps that we took helped in reducing both the inference time and the memory footprint for deployment on typical printer or a companion computer SOCs.
• 20:34
  SPEAKER_01: So, um also the point of view that we have is um it may still take time to deploy such kind of models directly on the printer, but we also have the companion devices to the printer like the AIPC or the smartphone, which themselves are quite capable.
• 20:54
  SPEAKER_01: So some of these things can be orchestrated between both, and some things may also be done on the cloud.
• 21:04
  SPEAKER_01: Yeah, we can go to the next slide.
• 21:07
  SPEAKER_01: Yeah.
• 21:08
  SPEAKER_01: Um, so these are some of the challenges that uh still remain in terms of uh large document handling.
• 21:17
  SPEAKER_01: Um, so we need to do a lot of segmentation tasks and batch processing to manage the memory load.
• 21:26
  SPEAKER_01: Um, then though we did not uh really do a fine-tuning of these base models, but uh for the optimum accuracy, uh we would need to do uh fine-tuning uh per the use case, and uh that will help us in maintaining the performance.
• 21:45
  SPEAKER_01: Uh, when it comes to embedded deployment, we have to keep in mind the thermal and the power constraints using efficient scheduling and hardware acceleration uh to minimize the power consumption.
• 22:00
  SPEAKER_01: And last but not the least, the compute requirements are still high, but um, as the SOCs develop and we also get better methods of optimization, uh, I think the deployment on the target lower cost devices would be a reality.
• 22:22
  SPEAKER_01: We can go to the next slide.
• 22:25
  SPEAKER_01: So these are some of the conclusions and future directions.
• 22:30
  SPEAKER_01: So uh multimodal AI models are a transformative advancement, uh, especially if we look at document reformatting tasks in printers.
• 22:42
  SPEAKER_01: And by deploying such models directly either on the device or on a companion device, we can offer customers much better and smarter printing and scanning solutions.
• 22:58
  SPEAKER_01: Multiple innovative use cases can be accomplished by using these models, and uh this approach will set the stage for a new era of intelligent edge printing where both uh the content understanding and the deformatting can happen seamlessly at or near the point of output.
• 23:23
  SPEAKER_01: Yeah, so I think with this we uh come to the end of the presentation.
• 23:28
  SPEAKER_01: Yeah, and I would be happy to take some questions.
• 23:32
  SPEAKER_00: Uh thanks, uh thanks, Arniban.
• 23:34
  SPEAKER_00: Uh and uh yeah, your presentation uh was uh smooth uh and uh maybe we missed your video.
• 23:41
  SPEAKER_00: I don't know if you could find a way to share uh links to the video later on so that we can enjoy your demonstration.
• 23:50
  SPEAKER_00: Can you quickly comment about what the demo were uh uh about?
• 24:00
  SPEAKER_00: Sorry, can you repeat your question?
• 24:03
  SPEAKER_00: Um uh about what uh your demos uh were uh were about uh okay okay okay yeah yeah what was the video correct correct correct the demos were about some uh very simple examples for example from a document like uh we can do a table extraction, right?
• 24:28
  SPEAKER_01: So uh if you think of uh earlier times, right, we would have to use some very specific template-based approach for doing this extraction, or we would have to train a deep learning model using a lot of data, right?
• 24:45
  SPEAKER_01: But with this VA visual language models, the advantage is that a lot of the knowledge uh for these kind of tasks are already inbuilt in those foundation models.
• 24:56
  SPEAKER_01: So um, and they realize the both the imaging context as well as the prompts that the user gives, right?
• 25:05
  SPEAKER_01: So it can relate.
• 25:06
  SPEAKER_01: So when I say that okay, you extract this table uh and just uh output it into some CSV format, so the model itself has that capability to do it.
• 25:19
  SPEAKER_01: So this really uh would shorten the um uh development time for application developers, and um I think that is the one of the key areas to look forward to, and uh uh Arniban, and uh yeah, thanks thanks Arniban.
• 25:41
  SPEAKER_00: A curiosity also from uh a business perspective.
• 25:46
  SPEAKER_00: How do you uh how do you consider um about using a conditional model for a solution that you sell in the market?
• 26:03
  SPEAKER_00: After all, the foundational model uh, I mean uh you you adopt, but uh maybe someone else uh uh prepared, trained uh with with lots of data, and uh and and then you're trying to exploit, for example, for the purposes that you spoke about uh and uh and uh how how do you um manage such a solution which depends from foundational model?
• 26:33
  SPEAKER_01: Is uh is something that you consider viable, uh reliable, or do you see any any any counts about that uh yeah, so I think uh people have concerns on the foundational model, like what kind of data sets that it has been trained on, and so on.
• 26:57
  SPEAKER_01: So I think uh we should look at it uh use case by use case.
• 27:01
  SPEAKER_01: Um so uh I think the power uh comes from the data for AI.
• 27:07
  SPEAKER_01: Um and um just using the foundational models may not be uh appropriate for any commercial solution.
• 27:15
  SPEAKER_01: So we really need to fine-tune the data with uh uh whatever contextual data belongs to the enterprise, right?
• 27:25
  SPEAKER_01: And um ultimately the model may still hallucinate, right, at the end of the day.
• 27:31
  SPEAKER_01: And so we need to build in a lot of guardrails um so that if it really does, then it should have an uh exit path and it should uh give something which is more logical, right?
• 27:45
  SPEAKER_01: So I think a lot of the work with AI will um be with uh how you write the prompts, how you basically experiment with it, how you see what it is doing, and building some checks on top of it, right?
• 28:01
  SPEAKER_01: And uh still uh it would be a kind of a fuzzy system, meaning uh you can never predict that it would repeat the response every time, right?
• 28:10
  SPEAKER_01: But uh basically it still gives a benefit in certain use cases.
• 28:15
  SPEAKER_01: I think uh that is the balance we need to draw, like where it can be applied and where it cannot be applied.
• 28:22
  SPEAKER_00: So and uh and I like the you mentioned that that this technology can help uh, for example, visually impair the people.
• 28:33
  SPEAKER_00: Uh so you made me think that this uh great technology can uh do help uh to create new services and uh products and applications, but also serve uh uh for uh the needs uh serve the needs of uh people with uh less abilities uh and uh which can really uh enjoy the capability of GNI in uh in providing machine interaction at uh at the next level.
• 29:08
  SPEAKER_00: So thanks to underline also this possibility.