Latest / EDGE AI POD / Revolutionizing Software Development with GenAI-Powered Edge Solutions with Anirban Bhattacharjee of Wipro

Transcript

Revolutionizing Software Development with GenAI-Powered Edge Solutions with Anirban Bhattacharjee of Wipro · EDGE AI POD

• 0:02
  Speaker 1: Transcriber's Name.
• 0:03
  Speaker 1: Reviewer's Name.
• 0:04
  Speaker 1: Time to switch to Arnibal by Seng Yong.
• 0:27
  Speaker 1: Thanks, arnibal, and I apologize to let you wait a little bit Then.
• 0:33
  Speaker 1: Now it's your time.
• 0:34
  Speaker 1: Your talk is Gen-I-based custom code system for AI PC and the floor is yours.
• 0:44
  Speaker 2: Thank you, danilo.
• 0:45
  Speaker 2: First of all, I would like to thank the TinyML Foundation for this wonderful session and happy to be part of this.
• 0:53
  Speaker 2: Yeah, it's like so just a brief introduction.
• 0:58
  Speaker 2: So basically I am working as a solution architect in the embedded software practice, part of Wipro engineering, and basically I come from an embedded and digital signal processing background and now very passionate about working in AI, ml related to embedded systems and to introduce to anybody who is not aware systems and to introduce to anybody who is not aware.
• 1:29
  Speaker 2: So Wipro is a leading global information technology and services company and we from Wipro Engineering are part of the group which work on engineering services and research and development.
• 1:38
  Speaker 2: For this work on the custom code assistant on AIPC, I would like to acknowledge a few of my team members, like Rupesh, deepak and Sagar, who are part of the Wipro engineering team, and also the Intel team who enabled us with their AI PC platform.
• 1:55
  Speaker 2: We can go to the next slide.
• 2:00
  Speaker 2: Basically, the problem, what we are seeing, is that today, with GenAI, there is a lot of opportunity and also hype, and there may be still certain doubts that how effective it is going to be, but in one area it is very clear that the things are changing very fast and that is the way that software development is going to happen changing very fast and that is the way that software development is going to happen.
• 2:28
  Speaker 2: So already we see an adoption of lot of commercial general purpose code assistance, namely GitHub, copilot, codeium and many other tools which are increasingly being adopted.
• 2:44
  Speaker 2: However, the approach that we are taking is that, though these general purpose code assistance would be good for some specific coding, but we see a need that many of the enterprises are sitting on a large code base of very proprietary code data, so they may not be very open to share it directly with many of these code assistants which basically leverage cloud-based and, at best, a hybrid approach.
• 3:16
  Speaker 2: So what we are proposing here is a device-based solution, like a complete edge-based solution, which is deployed on an AI PC and it utilizes the basically facilities provided by the hardware and runs in a very optimized way.
• 3:38
  Speaker 2: On that, maybe next slide, yeah.
• 3:44
  Speaker 2: So in terms of the enterprise software developers, few of the challenges which are seen are that the technology changes are very rapid, so people have to quickly learn a new programming language and they are having very stringent productivity goals, like what took six months previously now needs to be done in three months.
• 4:07
  Speaker 2: So how can this be done?
• 4:09
  Speaker 2: So this can only be done using certain intelligent tools in terms of helping you in the coding, helping you in debugging or optimizing the code which you have written manually, and also in basically understanding the legacy code base, which doesn't have the required documentation or design documents available.
• 4:35
  Speaker 2: Also, in meeting these challenges, there are certain expectations for any of these assistant tools, so these tools have to be very provide the results in very quick time with low latency.
• 4:52
  Speaker 2: People still don't want to pay the recurring cloud billing costs for, say, thousands of their employees in a large enterprise, and the accuracy of these assistants are very important, because if the results are not accurate enough, then these won't be successful.
• 5:13
  Speaker 2: Hence we see the need of a custom code assistant which can be tailored for every unique solution situation for our customers.
• 5:28
  Speaker 2: Maybe you can go to the next link.
• 5:29
  Speaker 2: Yeah, so this is the solution that we are building.
• 5:34
  Speaker 2: It is a complete web-based application and it can take in the inputs in terms of the source code files or the folders, and also the developer can paste the code.
• 5:50
  Speaker 2: In future, we are going to extend it to read like, say, github repositories and so on, and then we give an option to do whatever operation you want to do, like, for example, you have the code base and you want to add comments to that code base.
• 6:07
  Speaker 2: You can do that.
• 6:08
  Speaker 2: You want to create a summary of your code, you want to chat with your code and ask specific questions about specific functions?
• 6:20
  Speaker 2: So all those kind of features are supported.
• 6:25
  Speaker 2: How do we check how good this solution is?
• 6:29
  Speaker 2: So some of the key metrics that we are measuring here is basically how correct the output is or how useful it is for the developer.
• 6:40
  Speaker 2: Then what is the response time, or the throughput?
• 6:43
  Speaker 2: So all the responses should be in the order of milliseconds, otherwise it defeats the purpose.
• 6:52
  Speaker 2: How good does the tool understand the context and how user-friendly it is?
• 6:59
  Speaker 2: And some of the key parameters like, in addition to performance, is like what is the performance per watt?
• 7:09
  Speaker 2: Because increasingly the energy efficiency is also becoming very important.
• 7:17
  Speaker 2: We can go to the next slide.
• 7:18
  Speaker 2: So here we have shown a high level architectural diagram of our solution.
• 7:25
  Speaker 2: So mainly we have an UI on an input module which also does the pre-processing, and the data set we have used is a completely proprietary code base, which was our own code base.
• 7:41
  Speaker 2: Then we have the orchestration layer, where we use certain frameworks like LanChain or Lama Index and we use a mixed methodology, use retrieval, augmented generation, as well as we take some of the open source models like code llama, start button and code gemma.
• 8:06
  Speaker 2: So we have taken these models and we tried fine-tuning them and then we did a benchmarking and then selected the one which gave us the most accurate results.
• 8:18
  Speaker 2: One which gave us the most accurate results.
• 8:21
  Speaker 2: Ultimately we again used because we are using the Intel platform, so they have a framework for optimized inference on their platform, which is called OpenVINO.
• 8:32
  Speaker 2: So we converted the model into the OpenVINO format and optimized it on their platform format and optimized it on their platform.
• 8:45
  Speaker 2: The other important part is the human feedback loop.
• 8:49
  Speaker 2: So, based on the human feedback, we try to fine tune the model and improve the accuracy of the solution.
• 9:01
  Speaker 2: Maybe you can go to the next slide.
• 9:02
  Speaker 2: Yeah, so this gives a view of the Intel AI PC platform and what are the processing elements which are part of it.
• 9:11
  Speaker 2: So it consists of the Core Ultra series of processors.
• 9:16
  Speaker 2: The CPU itself has a deep learning boost accelerator which accelerates the AI tasks.
• 9:24
  Speaker 2: Then we also use the integrated graphics, which is the Intel Arc graphics as well, they have discrete graphics.
• 9:33
  Speaker 2: Also, mainly they have this Intel's neural processing unit.
• 9:37
  Speaker 2: So these neural compute engines do the compute acceleration for all the ai tasks and, in order to compare, so the cpu is very good for low latency and fast response.
• 9:53
  Speaker 2: The npu is good for overall efficiency, like it gives a decent throughput with good energy efficiency, and GPU is good for overall throughput.
• 10:06
  Speaker 2: So, depending on the use case, we have orchestrated our workload on the right processor.
• 10:16
  Speaker 2: You can go to the next slide.
• 10:17
  Speaker 2: Yeah, so here we give a summary of some of the key features that we are adding.
• 10:26
  Speaker 2: So one key use case we see is that a lot of the legacy code doesn't have proper design or documentation.
• 10:35
  Speaker 2: So if you want to automatically add the comments, so it can be very easily done through this kind of tool For a new developer who is coming on board to the team and is not having much idea, so the code summarization can generate an accurate summary of the code.
• 10:56
  Speaker 2: We can give instant code suggestions and auto-completions based on the current context when the developer is doing the coding.
• 11:06
  Speaker 2: And also the tool can help in terms of error detection, debugging, identifying potential errors and so on.
• 11:16
  Speaker 2: If we have a legacy code and we want to optimize the code, then the tool can give the proper suggestions regarding it.
• 11:27
  Speaker 2: Some of the use cases we see from our customers are like reverse engineering, the artifacts, so like the code is present but no requirement document is available, so whether we can generate the requirement specifications from the legacy code.
• 11:46
  Speaker 2: Another unique use case is like code conversion.
• 11:52
  Speaker 2: So we have, say, c code and we want to migrate it to C++.
• 11:56
  Speaker 2: Or we have a version of code in C++ 17.
• 12:00
  Speaker 2: So we have to migrate it to C++.
• 12:02
  Speaker 2: Or we have a version of code in C++ 17.
• 12:03
  Speaker 2: So we have to migrate it to C++ 23.
• 12:06
  Speaker 2: So doing it manually is quite cumbersome.
• 12:15
  Speaker 2: So these kind of tools can really hasten the process and go to the next slide.
• 12:23
  Speaker 2: Yeah, so what are the solution benefits of this Edge solution?
• 12:25
  Speaker 2: Primarily, it helps us in saving the recording cloud costs because this is completely on the device and there is no need to connect to the cloud.
• 12:35
  Speaker 2: It is a very secure solution because it runs on the local environment and ensures that there is no leakage of information, because the enterprise data is highly proprietary and confidential when once it is optimized to run on the underlying hardware platform.
• 13:04
  Speaker 2: And it is very efficient, provided that the workloads are properly orchestrated between the CPU, gpu and the NPU.
• 13:17
  Speaker 2: Due to the availability of the NPU, the overall power consumption can be greatly reduced.
• 13:26
  Speaker 2: And the Intel OpenVINO framework helped us in optimizing the model which is ultimately deployed on the underlying platform.
• 13:39
  Speaker 2: Coming to some of the business benefits, the developer productivity can be greatly improved because the time needed to understand the existing code base gets readily reduced.
• 13:56
  Speaker 2: So in especially systems programming, we have lot of firmware code which are very uniquely written and, for example, the bios code.
• 14:08
  Speaker 2: So for a new developer it's always very challenging and these kind of tools can greatly help them.
• 14:14
  Speaker 2: It improves the code quality, ensuring that the code is syntactically correct and optimized, leading to fewer bugs and much better performance.
• 14:28
  Speaker 2: These kind of tools can be readily integrated to the developer environments like VS Code and others environments like VS Code and others.
• 14:40
  Speaker 2: It can continuously learn from user interaction to provide increasingly accurate and relevant suggestions which are also personalized.
• 14:50
  Speaker 2: A lot of the enterprise companies have their own coding standards.
• 14:56
  Speaker 2: If you want to ensure adherence to those standards, then these tools can easily integrate those standard documents and ensure that we can talk about the code to requirements which was discussed and also, as I talked earlier, so the faster ramp up happens for new engineers, we can go to the next slide.
• 15:27
  Speaker 2: However, while implementing this solution, we saw that there are certain challenges, so many of the things which should work in theory, but when we actually implemented it, we saw that the accuracy is highly important and one technique doesn't work for every use case, because if the output itself is not accurate enough, then this tool can't be used.
• 15:55
  Speaker 2: Then user acceptance has to happen, because developers may be apprehensive about using these tools, and so only when they are trained and once they get used to it, then they understand the possibilities and the limitations of this tool and they can use it effectively.
• 16:18
  Speaker 2: Being a completely edge solution, and since we are running Gen-AI models, so there are certain minimum resource requirements.
• 16:29
  Speaker 2: So the disk space, or the memory at least, has to be in the order of gigabytes, and only then this kind of solution is feasible.
• 16:41
  Speaker 2: Yeah, so what we see is that the custom code assistant can really become a significant help in the field of software development.
• 16:56
  Speaker 2: By leveraging AI technologies, it will enhance the coding quality and provide real-time support to the developers.
• 17:05
  Speaker 2: So, whatever challenges are there, but the benefits far outweigh the challenges, making it a very promising solution.
• 17:16
  Speaker 2: Next slide yeah, so here we have some pointers on the future work that we are planning.
• 17:25
  Speaker 2: We are planning to support more programming languages.
• 17:30
  Speaker 2: Currently we are targeting systems programming languages like C and C++.
• 17:34
  Speaker 2: We want to improve the natural language processing engine so that it is able to understand the context and the background of the queries coming from the developers.
• 17:54
  Speaker 2: We plan to integrate this more with more IDEs and version control systems, and another topic would be how we can add up to the developer's coding style and preferences for the future.
• 18:08
  Speaker 2: Yeah, so that was all I had to share.
• 18:16
  Speaker 2: Any questions I'm willing to take.
• 18:26
  Speaker 1: Thank you so much, anirban.
• 18:26
  Speaker 1: Yeah, you touched a really important topic about the code assistant and I think many of us on daily basis are sharing and using the code, are sharing the needs of the code assistant and using them for increasing the productivity.
• 18:53
  Speaker 1: Now a question about that is can you share your experiences about measuring the productivity?
• 19:01
  Speaker 1: What would be a meaningful procedure to objectively measure the impact of such a code assistant in everyday life of the embedded programmer, the C++ programmers?
• 19:24
  Speaker 2: Yeah, I think that is a very important question, um, what you put.
• 19:27
  Speaker 2: So I think we are still exploring, um how best to measure the productivity.
• 19:34
  Speaker 2: Um, but to take an example, what I would say is that typically, like so, for example, we did some training on certain code assistance and basically what we thought is like we took some task, particular specific task, which maybe in a manual method, say, takes like two weeks, right, and then we challenged ourselves and saw that whether with the code assistant, we can do it in a reduced time frame, for example, that specific task, and if it is achieved, say the task of, say, two weeks, can be done within three to four days, then it's a real gain in the productivity.
• 20:28
  Speaker 2: So now, how do we scale that for multiple engineers and all?
• 20:33
  Speaker 2: So that would take a more detailed approach, but typically, like with certain training workshops on these coding assistants, you can at least get a feel that a particular task which could take two weeks now can be done in, say, three days or four days.
• 20:52
  Speaker 2: So that is the productivity gain you get, and that is by having multiple developers using these code tools as part of a training or a workshop of a short duration.
• 21:06
  Speaker 2: I think that is one way we are basically evaluating.
• 21:12
  Speaker 1: Thank you, Arne.
• 21:14
  Speaker 1: One question from Sandeep how much RAM is needed on the AIPC for the use cases presented?
• 21:25
  Speaker 2: Yeah, so typically these kind of AIPCs have at least 16 GB to 32 GB RAM, depending on the configuration.
• 21:35
  Speaker 2: So that is the configuration with which we have checked and what we saw also is that the GPU, the performance, is much better, definitely when we have a discrete GPU and the NPU is also performing quite well.
• 21:52
  Speaker 2: With the CPU only, the performance really is quite poor because it goes to the order of even minutes for any simple response.
• 22:08
  Speaker 1: Thank you so much.
• 22:09
  Speaker 1: Another question from my side.
• 22:11
  Speaker 1: I really like the topic because I'm a fan of the tools that really introduce automation and increase productivity.
• 22:23
  Speaker 1: It's also part of my job, so I'm wondering in your experience, what could you say about the acceptance rate sense that, for example, do you see more young professionals eager to be assisted by such facility, less seasoned, experienced programmers, or what other mix you are experiencing?
• 23:01
  Speaker 2: you are experiencing.
• 23:02
  Speaker 2: Yeah, I think the young developers are more interested and more ready to adopt these kind of tools because they want the shortest way right, whereas, like for us, who are more coming from the, with the baggage of having done manual coding and programming, I think maybe there may be an initial reluctance till we can see the benefit of it.
• 23:35
  Speaker 2: And I don't think there is a silver bullet that this kind of assistant tools can solve all your problems, but it can really improve your productivity definitely.
• 23:50
  Speaker 1: And, in such a case, for the people that is, for any reason, more reluctant or delay, procrastinating such such an adoption, in your opinion, what a a company employing such people should do to move really, uh, move forward the adoption.
• 24:17
  Speaker 1: So any, any, any procedure, any training, any special initiative you would envision to increase the steepness of the adoption rate?
• 24:33
  Speaker 2: Yeah, I think like say, boot camp or like a very short intensive training session, say one day, two, two day sessions, they really help because, first of all, when a trainer comes and shows you the benefit or helps you and in my experience it need not be like if you are a c, c plus plus programmer, you may also do a workshop in say java.
• 24:59
  Speaker 2: Okay, at least you see the benefit of the productivity gain that you are getting out of the tool right.
• 25:07
  Speaker 2: Once you are convinced, then, being the expert programmer that you are, I think you will find your own ways to get productivity in your work and find the right set of tools as well.
• 25:22
  Speaker 2: So what we have seen is that intensive training sessions of, say, one or two day duration they really work, like they first of all remove the inhibitions of the people and they see that, okay, this is at least working in, say, the java world and it should also be working in systems programming, for example.
• 25:44
  Speaker 2: So that is something what I have seen has worked.
• 25:51
  Speaker 1: Yeah, very good.
• 25:52
  Speaker 1: And last question, before to move to the next speaker I think we have still a couple of minutes, anirbal.
• 26:02
  Speaker 1: I think we have still a couple of minutes, anirban.
• 26:07
  Speaker 1: You know, generating C code, c++ code, make it fast running on the target, can be also helped by profile-driven optimization.
• 26:17
  Speaker 1: So I run the code, I get some profiling and then maybe the code could be improved, some kernels could be improved to make it faster or less memory demanding.
• 26:28
  Speaker 1: You know these type of things.
• 26:31
  Speaker 1: Is that part of, or could be a part of, the code assistant?
• 26:38
  Speaker 1: Or maybe it's not required by the inner processes of generating the code?
• 26:48
  Speaker 2: Yeah, I think.
• 26:50
  Speaker 2: In my opinion, it is something which you can add as an additional feature, but this is not the real problem that the generative ai can solve.
• 27:01
  Speaker 2: Uh, what we see from a generative ai perspective is like uh, you have a firmware code, okay like, which has been written, say, 15 years back.
• 27:12
  Speaker 2: A new developer comes to the team, has no idea on that code, uh, has, doesn't have any documentation because it doesn't exist, right, in that case, this kind of tool can actually give you a good summary okay, help you in understanding the code much better.
• 27:33
  Speaker 2: And these are the real use cases that can be solved.
• 27:39
  Speaker 2: The optimization part, I would say, is more like an add-on, because there are already sophisticated tools which can do that.
• 27:49
  Speaker 1: Thank you so much, anirban.
• 27:50
  Speaker 1: Thank you so much for your time, your contribution, which was really much appreciated by me and by the TPC committee.
• 28:03
  Speaker 1: Thanks again.
• 28:04
  Speaker 1: If you have time and there are questions in the comment chat, please consider to target some of them.
• 28:13
  Speaker 1: Otherwise, thanks so much again.
• 28:16
  Speaker 2: Thank you.
• 28:17
  Speaker 1: Danilo.
• 28:18
  Speaker 2: Yeah, thank you.