Latest / EDGE AI POD / Real World Deployment and Industry Applications

Transcript

Real World Deployment and Industry Applications · EDGE AI POD

• 0:05
  Speaker 1: Hello, arniban.
• 0:06
  Speaker 1: Good to see you, arniban, from YPRO Limited, and thanks to be with us, arniban, and to contribute to today, to the third forum of today.
• 0:21
  Speaker 1: I think you are going to speak about industry applications and deployments of Gen I at the edge.
• 0:31
  Speaker 1: So whenever you feel ready to share your presentation, please, please, do so and and the floor is yours thank you.
• 0:43
  Speaker 2: Thank you so much, danielo.
• 0:44
  Speaker 2: Is my presentation visible?
• 0:49
  Speaker 1: yes, it is it is okay.
• 0:51
  Speaker 2: Thank you so much.
• 0:52
  Speaker 2: Yeah, uh, thank you, danielo.
• 0:54
  Speaker 2: So um today.
• 0:56
  Speaker 2: So my name is anirban and I am representing vipro limited today um to talk about some of the real world deployment and industry applications that we are seeing.
• 1:09
  Speaker 2: This is the agenda for my presentation.
• 1:16
  Speaker 2: So some background on where we come from.
• 1:20
  Speaker 2: What is the AGI relevance?
• 1:25
  Speaker 3: that we need about it.
• 1:26
  Speaker 3: We don't.
• 1:27
  Speaker 3: I think you are.
• 1:28
  Speaker 3: Maybe it's another presentation you're showing, because we just see your tile slide right now.
• 1:34
  Speaker 3: I don't know exactly if you want try again, because I think it's just the one that you have uploaded previously, but you are not sharing your screen.
• 1:48
  Speaker 2: Okay okay, okay, let me share that.
• 1:50
  Speaker 2: Is it visible now?
• 2:08
  Speaker 3: Let me see.
• 2:09
  Speaker 3: No, I think it's still the other one.
• 2:11
  Speaker 3: I think you or maybe now let's see Is that the one?
• 2:17
  Speaker 2: And then you just go full screen.
• 2:19
  Speaker 2: Table of content.
• 2:20
  Speaker 2: Yeah, table of content Now.
• 2:21
  Speaker 3: you just need to go to full screen because we just see all the slides on the left, and then I'll let you take it over I've gone to full screen no, it's not.
• 2:37
  Speaker 3: We are still seeing everything on the left to all your slides?
• 2:41
  Speaker 3: No, no.
• 2:42
  Speaker 3: Are you using different screens?
• 2:42
  Speaker 3: Maybe on the left, to all your slides?
• 2:45
  Speaker 3: No, no.
• 2:45
  Speaker 3: Are you using different screens?
• 2:47
  Speaker 3: Maybe, or when you just go to slideshow.
• 3:01
  Speaker 2: OK, let me try.
• 3:01
  Speaker 2: Ok, let me share again.
• 3:32
  Speaker 2: Yes, screen.
• 3:36
  Speaker 3: Is it visible in full screen now?
• 3:38
  Speaker 3: No, it's not visible on full screen.
• 3:40
  Speaker 3: If not, yeah, we just have to go that way, like and if you just go and take slideshow that it's showing your whole screen video yeah right, danilo, you can also see that.
• 3:55
  Speaker 3: Right, it's not in full screen mode, he's just.
• 4:01
  Speaker 1: I see.
• 4:01
  Speaker 1: No, not in full screen.
• 4:03
  Speaker 3: No, no, Okay, no, any technical difficulty Arniban yeah, if not, yeah, we have to go that way, but it's not really what we want, but it's okay.
• 4:26
  Speaker 2: I'm studying in full screen, but not sure.
• 4:27
  Speaker 2: Yeah, Maybe you have multiple screens or maybe you can share the presentation that I uploaded.
• 4:30
  Speaker 3: Yeah, I can share that one.
• 4:32
  Speaker 2: But that's why do you want that actually, I without the video is fine, I think we will okay then we go with this one.
• 4:41
  Speaker 3: Okay, can you move forward here in that slide, and then I'll?
• 4:54
  Speaker 2: yeah, you can go ahead to the next slide, okay.
• 4:55
  Speaker 2: So you want me to?
• 4:56
  Speaker 2: Okay?
• 4:56
  Speaker 2: Okay, I'll just follow you then.
• 4:57
  Speaker 2: Yeah, yeah, so this is the agenda and then some background on our team.
• 5:07
  Speaker 2: So, for those who are not aware, so Wipro is a leading global information technology consulting and software development company.
• 5:17
  Speaker 2: We are present in more than 167 countries and we focus on different areas, including AI and cloud computing and data analytics, for example.
• 5:29
  Speaker 2: So I come from a background of embedded development and currently, for the last five years, working on AI, and I would like to acknowledge the bigger team behind this as well.
• 5:43
  Speaker 2: To acknowledge the bigger team behind this as well.
• 5:49
  Speaker 2: Yeah, we can go to the next slide, thank you.
• 5:51
  Speaker 2: So, to talk about the use cases, so what we see are different trends in ai.
• 6:01
  Speaker 2: So there are advancements in the generative ai models, with lot, lot of multimodal AI models coming in, and these models are driving new use cases as well as they are enabling improvements in the older
• 6:17
  Speaker 2: implementation efforts.
• 6:19
  Speaker 2: A lot of the implementation for multimodal AI happens on the cloud, but there are certain concerns with that in terms of mainly the data privacy part of it, where people may not be ready to share all their information to the cloud, as well as the need for the availability of a high bandwidth network.
• 6:47
  Speaker 2: So in some use cases it is beneficial to have the inferencing done in an offline mode without a dedicated network connectivity.
• 7:00
  Speaker 2: So, in terms of looking at embedded devices, we looked at some of the different devices and we saw that, like the printers and scanners is one category whether where there may be lot of applications of multimodal AI Today, the printers and scanners that we see are mostly passive.
• 7:27
  Speaker 2: They don't basically process the content themselves.
• 7:33
  Speaker 2: They struggle with a lot of inconsistent formats and layouts and basically these formats are all unstructured.
• 7:44
  Speaker 2: A lot of the documents come in multilingual formats, especially in enterprise context and with traditional rule-based approaches and even traditional deep learning techniques, there is very limited reformatting support which is available.
• 8:07
  Speaker 2: So with the coming of multimodal AI, basically these passive endpoints can become more intelligent and they can help in basically changing this unstructured format into a more structured format.
• 8:27
  Speaker 2: They have the power to translate the content from one language to other and they have this special ability to understand both image and the text.
• 8:42
  Speaker 2: We can go to the next slide.
• 8:45
  Speaker 2: Yeah, so we took one use case, mostly on the document understanding part of it.
• 8:55
  Speaker 2: So lot of multimodal AI models are available, even in the open source.
• 9:01
  Speaker 2: For example, we have the QN 2.5 VL, which is a very popular model.
• 9:08
  Speaker 2: It is capable of handling text, images and even other modalities.
• 9:14
  Speaker 2: We have very special models like LayoutLMv3 from Microsoft, which is basically trained for document understanding.
• 9:27
  Speaker 2: Then there are other popular models like Donut and TATR from Microsoft for table understanding.
• 9:34
  Speaker 2: And these are the multimodal series of models, whereas we have another series more from like the diffusion models.
• 9:43
  Speaker 2: So we have a model from FLUT.
• 9:45
  Speaker 2: So they are very creative in generating new content.
• 10:02
  Speaker 2: We can go to the next slide.
• 10:04
  Speaker 2: This is our solution approach, that how on the printers and scanners, basically we can bring in these multimodal ai models so we have this input image or pdf format as well, as we may have an user input that may be in the form of a prompt or it can be by voice direction.
• 10:27
  Speaker 2: So once, so once this is acquired, we do some conversion, maybe we convert PDF to image.
• 10:38
  Speaker 2: Then we do a lot of pre-processing tasks, like we do resolution conversion, we do page segmentation, noise reduction and so on, and then pretty much we can use the visual language models directly for optimized inference.
• 10:57
  Speaker 2: We can do quantization, and without much supervision the blms are capable of interpreting the content, identifying key elements and then restructuring the layouts.
• 11:14
  Speaker 2: They are also capable of generating in the format that we desire.
• 11:19
  Speaker 2: And then we have some post-processing activities where we clean up, prepare in the format that we desire.
• 11:28
  Speaker 2: Basically, the interesting part of this is that without even doing a lot of fine tuning of the base foundation models, we can get quite decent results.
• 11:45
  Speaker 2: So primarily in our implementation we have used two models.
• 11:53
  Speaker 2: So one is the QEN 2.5 real model.
• 11:56
  Speaker 2: It excels in handling complex visual inputs, including images of different sizes and so on, while still maintaining a good linguistic performance.
• 12:10
  Speaker 2: This model, so we have tried with 3 billion parameters, 7 billion parameters and so on, and the configuration that we used had a GPU RAM of 16 GB basically.
• 12:26
  Speaker 2: On the other hand, we have also used the flux diffusion model.
• 12:31
  Speaker 2: This has a hybrid architecture of both multimodal and parallel diffusion transformer block and it had a size of around 12 billion parameters.
• 12:43
  Speaker 2: So this also needs VRAM of around 24 GB.
• 12:51
  Speaker 2: We can go to the next slide.
• 12:57
  Speaker 2: So here we have primarily two use cases.
• 13:01
  Speaker 2: One is on extracting the tabular data and reformatting to the graphs.
• 13:10
  Speaker 2: So we can import a pdf document which has tables and other textual content.
• 13:21
  Speaker 2: So the vlms are quite skilled to extract the table information and then, once we have the tabular data, we can convert it into a different representation, say a pie chart or a bar chart, and then we can overwrite, say, the tabular data with this information.
• 13:45
  Speaker 2: So lot of these creative things can be done with the documents.
• 13:54
  Speaker 2: The next use case is more on image generation and modification.
• 13:59
  Speaker 2: So basically we were thinking about taking card use case.
• 14:04
  Speaker 2: So with the textual prompt we can create a greeting card from the model and then we can do incremental modification, steps, steps, by prompting the model.
• 14:21
  Speaker 2: So for example, here the greeting card says happy new year, 2023 and then with the proper text prompt we can modify the year to 2025.
• 14:33
  Speaker 2: So some of these modifications are done very simply.
• 14:38
  Speaker 2: Then in the third step, we wanted to change the outer boundary of the image.
• 14:45
  Speaker 2: So we were able to do it with a simple prompt to the model.
• 14:50
  Speaker 2: You can go to the next slide.
• 14:59
  Speaker 2: This is another use case where we do the text correction which I already discussed, Then coming to other use cases.
• 15:10
  Speaker 2: So what we have seen is in a printer scanner use case.
• 15:16
  Speaker 2: Automatically we can understand the scanned documents and then classify them into specific categories.
• 15:25
  Speaker 2: So in an enterprise context it really helps that it can also trigger a notification that this document is intended for this group of people.
• 15:41
  Speaker 2: Then we have multilingual content handling ability and so these models can easily translate between different languages, and this is very helpful in typical enterprises.
• 15:58
  Speaker 2: Another use case may be like adapting the layout for visually impaired users by increasing the font size, contrast and simplifying the design.
• 16:12
  Speaker 2: Then in a scanner use case we can say have some defects in the text, and all of those gaps can be automatically filled in by the model and error correction can be done automatically.
• 16:30
  Speaker 2: We have seen other use cases, like when it comes to giving a print from a web page, like a lot of these advertisements and other content comes in.
• 16:42
  Speaker 2: So all of that can be automatically detected by the AI model and removed.
• 16:49
  Speaker 2: Yeah, we can go to the next slide.
• 16:51
  Speaker 2: Yeah, yeah, unfortunately I think this video will not run, but basically, while we were showing the table extraction, this is the text correction part of it.
• 17:05
  Speaker 2: Yeah, you can go to the next slide.
• 17:07
  Speaker 2: This is about changing the borders and this is about changing the color of the bird.
• 17:15
  Speaker 2: By just a simple prompt to the model, it is able to update the bird color.
• 17:22
  Speaker 2: Yeah, you can go to the next slide.
• 17:26
  Speaker 2: So this is some of the deployment aspects, what we considered.
• 17:31
  Speaker 2: So the aim is to deploy it on the edge and the device, like we discussed.
• 17:38
  Speaker 2: But these models still need considerable resources to run.
• 17:45
  Speaker 2: So first of all, we took lot of steps in optimizing it.
• 17:51
  Speaker 2: One step was basically downscaling the image.
• 17:55
  Speaker 2: So during the preprocessing step, we downscaled the image by reducing the resolution significantly so that it improves the image processing performance efficiency, with some loss of accuracy.
• 18:11
  Speaker 2: Then these VLM models are very good in object localization and grounding.
• 18:19
  Speaker 2: So some of the tasks, such as accurately locating the objects within the images using bounding boxes and point coordinates, so all of these can be done automatically for the VLMs in it.
• 18:35
  Speaker 2: So all of these can be done automatically for the vlms and, as an application developer.
• 18:40
  Speaker 2: We don't need to set up some complicated pipeline to do this, so we use the power of the vlms which are inbuilt.
• 18:46
  Speaker 2: In terms of model quantization, we quantize the seven billion parameter model using a 4-bit quantization technique and we used the GGUF format for the model.
• 19:00
  Speaker 2: Gguf is a more efficient and flexible way of storing and using the models for inference and is designed to run well on consumer grade computer hardware.
• 19:13
  Speaker 2: In terms of the diffusion model, we did a lot of hyperparameter optimization.
• 19:23
  Speaker 2: So there are certain parameters for the diffusion model, like case sampler and scheduler, and these can be tweaked for better performance.
• 19:33
  Speaker 2: By enabling better control of the sampling process how the image gets generated and by fine tuning the sampling algorithms and parameters, we could do a balance between the accuracy as well as the performance.
• 19:53
  Speaker 2: And, last but not the least, we utilize the optimized library for NVIDIA for the GPU to optimize the performance on the GPU.
• 20:04
  Speaker 2: So overall, the deployment configuration was based on an x86 based processor and NVIDIA T4 GPU with a memory of around 16 GB.
• 20:20
  Speaker 2: All the mentioned steps that we took helped in reducing both the inference time and the memory footprint for deployment on typical printer or a companion computer SOC.
• 20:33
  Speaker 2: So also the point of view that we have is it will still take time to deploy such kind of models directly on the printer, but we also have the companion devices to the printer, like the AI PC or the smartphone, which themselves are quite capable.
• 20:54
  Speaker 2: So some of these things can be orchestrated between both, and some things may also be done on the cloud.
• 21:02
  Speaker 2: Yeah, we can go to the next slide.
• 21:06
  Speaker 2: So these are some of the challenges that still remain in terms of large document handling.
• 21:16
  Speaker 2: So we need to do a lot of segmentation tasks and batch processing to manage the memory load.
• 21:27
  Speaker 2: Then, though, we did not really do a fine tuning of this base models, but for the optimum accuracy we would need to do a fine tuning for the use case, and that will help us in maintaining the performance.
• 21:44
  Speaker 2: When it comes to embedded deployment, we have to keep in mind the thermal and the power constraints, using efficient scheduling and hardware acceleration to minimize the power consumption.
• 21:59
  Speaker 2: And, last but not the least, the compute requirements are still high.
• 22:04
  Speaker 2: But as the SOCs develop and we also get better methods of optimization, I think the deployment on the target lower cost devices would be a reality, we can go to the next slide.
• 22:25
  Speaker 2: So these are some of the conclusions and future directions.
• 22:29
  Speaker 2: So multimodal AI models are a transformative advancement, especially if we look at document reformatting tasks in printers, and by deploying such models directly, either on the device or on a companion device, we can offer customers much better and smarter printing and scanning solutions.
• 22:57
  Speaker 2: Much better and smarter printing and scanning solutions.
• 23:02
  Speaker 2: Multiple innovative use cases can be accomplished by using these models, and this approach will set the stage for a new era of intelligent edge printing, where both the content understanding and the reformatting can happen seamlessly at or near the point of output.
• 23:21
  Speaker 2: Yeah, so I think with this we come to the end of the presentation.
• 23:28
  Speaker 1: Yeah, and I would be happy to take some questions uh, thanks, uh, thanks, arniban and uh, and your presentation was smooth and maybe we missed your video.
• 23:41
  Speaker 1: I don't know if you could find a way to share links to the video later on so that we can enjoy your demonstration.
• 23:50
  Speaker 1: Can you quickly comment about what the demo was about?
• 24:01
  Speaker 2: Sorry, can you?
• 24:02
  Speaker 1: repeat your question About what your demos were about.
• 24:09
  Speaker 1: What was the concept?
• 24:12
  Speaker 2: of the video Correct, correct.
• 24:15
  Speaker 2: The demos were about some very simple examples.
• 24:20
  Speaker 2: For example, from a document, like we can do a table extraction, right.
• 24:28
  Speaker 2: So if you think of earlier times, right, we would have to use some very specific template based approach for doing this extraction.
• 24:39
  Speaker 2: Or we would have to train a deep learning model using a lot of data, right.
• 24:44
  Speaker 2: But with this visual language models, the advantage is that lot of the knowledge for these kind of tasks are already inbuilt in those foundation models.
• 24:55
  Speaker 2: So, um, and they realize the, both the imaging context as well as the prompts that the user gives, right, so it can relate.
• 25:06
  Speaker 2: So when I say that, okay, you extract this table, uh and just uh output it into some CSV format, so the model itself has that capability to do it.
• 25:18
  Speaker 2: So this really would shorten the development time for application developers and I think that is one of the key areas to look forward.
• 25:34
  Speaker 1: And Arnivan.
• 25:41
  Speaker 1: A curiosity, also from a business perspective how do you consider about using a foundational model for a solution that you sell in the market?
• 26:07
  Speaker 1: After all, the foundational model I mean you you adopt, but maybe someone else prepared the trainer there with with lots of data and and and then you are trying to exploit, for example, for the purposes that you spoke about, and and how, how do you manage such a solution which depend from foundational model?
• 26:32
  Speaker 1: Is is something that you consider viable, reliable, or do or do you see any cons about that?
• 26:45
  Speaker 2: Yeah, so I think people have concerns on the foundational model, like what kind of data set that it has been trained on, and so on.
• 26:56
  Speaker 2: So I think we should look at it use case by use case.
• 27:00
  Speaker 2: So I think the power comes from the data for the AI, and just using the foundational models may not be appropriate for any commercial solution.
• 27:15
  Speaker 2: So we really need to fine tune the data with whatever contextual data it belongs to the enterprise right, and ultimately, the models will still hallucinate right at the end of the day, and so we need to build in lot of guardrails so that, if it really does, then it should have an exit path, and it should give something which is more logical, right?
• 27:45
  Speaker 2: so I think a lot of the work with ai will be with how you write the prompts, how you basically experiment with it, how you see what it is doing and building some checks on top of it, right, and still it would be a kind of a fuzzy system, meaning you can never predict that it would repeat the response every time, right, but basically it still gives a benefit in certain use cases.
• 28:15
  Speaker 2: I think that is the balance we need to draw, like where it can be applied and where it cannot be applied.
• 28:25
  Speaker 1: And I like it that you mentioned that this technology can help, for example, visually impaired people.
• 28:33
  Speaker 1: So you made me think that this great technology can help to create new services and products and applications, people with less abilities and which can really enjoy the capability of a JNI in providing machine interaction at the next level.
• 29:08
  Speaker 1: So thanks to underline also this possibility.