Latest / EDGE AI POD / Honey, I Shrunk the LLMs: Edge-Deployed AI Agents

Transcript

Honey, I Shrunk the LLMs: Edge-Deployed AI Agents · EDGE AI POD

• 0:05
  Speaker 1: Our next speaker, that would be, I would say, deep into the technical heart of Edge AI, and it's my pleasure to introduce Aruna Koulourou, from Dell Technologies, and her session is entitled Edge Deployed AI Agents Architecture Optimization and Security.
• 0:23
  Speaker 1: Aruna, architecture optimization and security.
• 0:25
  Speaker 1: Arona, the stage is yours.
• 0:28
  Speaker 2: Thank you so much.
• 0:29
  Speaker 2: Let me share my screen.
• 0:32
  Speaker 2: Can you see my screen?
• 0:55
  Speaker 2: Yeah, hello everyone.
• 0:56
  Speaker 2: I'm a workload AI specialist from Dell.
• 0:59
  Speaker 2: It's funny that Evgeny mentioned about the Dell AI laptops.
• 1:04
  Speaker 2: I do work for Dell and, though I don't work in that division, I actually work from the AI specialist division, focusing more on the enterprise deployments.
• 1:15
  Speaker 2: But AI agents have been very popular these days.
• 1:18
  Speaker 2: Let's actually talk a little bit about them.
• 1:22
  Speaker 2: I actually feel we are at a very pivotal moment in the evolution of this AI.
• 1:27
  Speaker 2: For years, ai has lived on the cloud, in the data centers, where powerful models ran behind the scenes, but today the whole paradigm is shifting.
• 1:37
  Speaker 2: Ai is coming closer to where the data is generated, on the devices in our environments, even in disconnected or low connected zones.
• 1:46
  Speaker 2: What we are talking about today is not just AI at the edge, but how these generative AI agents at the edge.
• 1:53
  Speaker 2: That's a very subtle but a very powerful distinction.
• 1:56
  Speaker 2: These agents don't just classify or recognize something.
• 2:00
  Speaker 2: They generate language, summaries, decisions and even plans of action, the reason.
• 2:08
  Speaker 2: They recall from past interactions, they choose tools to accomplish goals and they do it on-device, privately, securely and often autonomously.
• 2:20
  Speaker 2: This talk, let's actually focus a little bit more beyond the model compression.
• 2:23
  Speaker 2: Let's walk through some of the stack, as well as a couple of use cases as well and why we are here.
• 2:30
  Speaker 2: So Benedict Evans is a well-known technology analyst and a thought leader.
• 2:36
  Speaker 2: He spent years analyzing trends in tech, especially around platforms, software and AI.
• 2:42
  Speaker 2: In tech, especially around platforms, software and AI.
• 2:47
  Speaker 2: He's known for his sharp insights and widely followed presentations and newsletters that really explore how technology reshapes industries and societies.
• 2:52
  Speaker 2: His quote, actually, I thought, really captures that fundamental shift on how we interact with AI.
• 3:02
  Speaker 2: Old AI we called a model to do a task, like asking a calculator for an answer or something like that.
• 3:10
  Speaker 2: With the new AI, you collaborate with an agent, like working with a team member, who understands the context, adapts and helps you reach those goals.
• 3:19
  Speaker 2: It's not just about using AI anymore.
• 3:22
  Speaker 2: It's about working with AI as a partner that can plan, reason and act alongside you.
• 3:28
  Speaker 2: So let's quickly see what an AI agent is right.
• 3:34
  Speaker 2: Just a very quick introduction.
• 3:37
  Speaker 2: Like AI, agent is a system that can perceive its environment, make decisions and take actions to achieve a specific goal.
• 3:46
  Speaker 2: I know there are other considerations out there about the ethical considerations and everything, but in essence, that's what an AI agent is.
• 3:55
  Speaker 2: Agentic AI refers to AI systems that exhibit agency, which means that they can autonomously pursue goals, plan, adapt and make decisions over time, often in complex or dynamic environments.
• 4:13
  Speaker 2: So agents are separate.
• 4:16
  Speaker 2: When we talk about agentic AI, it's about bringing all these AI agents together to work on something autonomously.
• 4:23
  Speaker 2: Ai agents together to work on something autonomously.
• 4:25
  Speaker 2: So we used to think of Edge as a place to run small, pre-trained models.
• 4:30
  Speaker 2: These were typically lightweight classifiers or detectors, good for object recognition, sensor filtering, maybe the basic NLP, the outputs for deterministic and narrow like classify, detect, match.
• 4:45
  Speaker 2: But what's happening now is like this massive leap we are moving from inference to intelligence at the edge.
• 4:52
  Speaker 2: So in some of the demos we've actually seen some of the generative AI at the edge.
• 4:59
  Speaker 2: Here what we are talking about is generative AI.
• 5:02
  Speaker 2: Agents are a new class of AI systems that don't just answer, they explain, they don't just recognize, they actually generate and adapt and instead of just inferring, they actually plan.
• 5:15
  Speaker 2: So let me actually talk about an example here.
• 5:21
  Speaker 2: Imagine a drone flying over a wildfire zone.
• 5:23
  Speaker 2: It doesn't capture footage, it generates captions about terrain changes, summarizes dangers and plans the next survey route all on the device, without ever needing the cloud.
• 5:36
  Speaker 2: Or you can even think about a retail assistant running in store.
• 5:41
  Speaker 2: It watches shelves via vision model, detects what's empty and generates a personalized promo message for shoppers walking by.
• 5:48
  Speaker 2: All this is happening offline, with edge inferencing, retrieval and speech synthesis.
• 5:53
  Speaker 2: So how is it different, right?
• 5:57
  Speaker 2: So this is different because it's actually making those decisions at the edge itself.
• 6:02
  Speaker 2: Now we've seen what agents are.
• 6:08
  Speaker 2: I think there are three major enablers for this shift.
• 6:12
  Speaker 2: One is the miniature model foundation models.
• 6:15
  Speaker 2: We are seeing emergence of high quality models that can be quantized down to run on smaller devices.
• 6:22
  Speaker 2: Models like even Mistral, tiny Llama, gemma 2 billion can be compressed to 4-bit or even like 3-bit optimized versions.
• 6:31
  Speaker 2: They run comfortably in like 4 to 8 gig of VRAM.
• 6:35
  Speaker 2: This kind of specs you'll find on maybe Jetson, orion, raspberry Pi with swap or like even laptops right?
• 6:42
  Speaker 2: I've mentioned about the Qualcomm and Dell laptops, and the definition of edge is also slightly changing and the capacity at the edge is also changing and in some of the laptops the kind of specs we have is quite fascinating.
• 6:59
  Speaker 2: Right now I have another laptop which actually has like 16 gigs of GPU in it and, if possible, I'll show a demo or I'll actually show some of the screenshots I've got from the demo, I'm actually running like multiple of these models simultaneously as AI agents at the edge, and these AI agents actually also push that move towards more domain-specific models.
• 7:27
  Speaker 2: And we say domain-specific models, we want those models to be very small, trained on specific data, so that also drives smaller models to be run on that.
• 7:40
  Speaker 2: And then the second part of it is the agentic frameworks.
• 7:43
  Speaker 2: Having the model is not enough, right?
• 7:43
  Speaker 2: And then the second part of it is the agentic frameworks.
• 7:46
  Speaker 2: Having the model is not enough, right, you need to give it a structure.
• 7:48
  Speaker 2: Frameworks like Landgraf lets you define the data-based planning flows where the agent makes decisions and passes context from step to step.
• 7:57
  Speaker 2: And then we have Autogen that allows agents to use tools, memory and history in conversational loops.
• 8:05
  Speaker 2: And then Crew AI organizes agents into role-based teams like a planner, a verifier, an executor, really enabling that coordinated workflow across different agents.
• 8:18
  Speaker 2: These frameworks are the glue that turns a single model into an agent, intelligent worker or like a swarm of them working together.
• 8:29
  Speaker 2: Then the last one is the edge native runtimes and toolchains.
• 8:32
  Speaker 2: Deployment is now easier than ever with projects like O-Lama.
• 8:38
  Speaker 2: It's like a CLI and server interface or a runtime for running quantized LLMs locally with REST endpoints.
• 8:45
  Speaker 2: And we have Lamacpp, which is the optimized C++ runtime with GGUF model support, token streaming and flash retention.
• 8:55
  Speaker 2: And then you have LM Studio, a desktop app to run and chat with the local models, and WebLLM and WebGPU run LLM models right into your browser, even on a phone.
• 9:06
  Speaker 2: These tool chains make it possible to deploy agents on mobile devices, embedded systems or even into your browser tab completely offline as well.
• 9:16
  Speaker 2: So we have this agents in sync right.
• 9:25
  Speaker 2: So once we have all these multiple models, the agent communication gets very, very important, and the newest protocols which have been released in the last few months model context protocol allows for the AI agent to talk to all the environment and the tools around it.
• 9:44
  Speaker 2: For example, if an AI agent which we are talking about, the planning agent to talk to all the environment and the tools around it.
• 9:47
  Speaker 2: For example, if an AI agent which we're talking about, the planning agent wants to look at a disaster from a drone and it want to talk to a weather app to look at what's the weather next, to make the decision of what are the next steps, it could do that, or it has to send this message to so many people it can actually interact with other tools to do that.
• 10:06
  Speaker 2: Mcp allows a standard way of doing that.
• 10:09
  Speaker 2: Before mcp, um, there everything was like everyone had their own way to uh interact.
• 10:16
  Speaker 2: Um, most most often a json was used as an output and input to other agents, but with this one you have a standard way to interact with things.
• 10:27
  Speaker 2: And then A2A is like an agent-to-agent protocol from Google which enables, again, a standard way for agents to talk to each other in a standard way.
• 10:36
  Speaker 2: And you might have seen, lots of enterprises are actually releasing their MCP support for their tools so these agents can talk to them, and this is a very, very high level example of how multiple agents are interacting with MCP the diagram there.
• 10:56
  Speaker 2: And then we have generative agent.
• 10:59
  Speaker 2: Now that we have actually spoken a little bit about the shift.
• 11:03
  Speaker 2: What we're talking about here is not about a single model running on the device.
• 11:11
  Speaker 2: A true agent is more like a modular, intelligent system capable of perceiving, thinking, remembering, even deciding, and acting, all in a constrained environment.
• 11:22
  Speaker 2: Let's break down into the five essential traits for an agent right Perception this is where the agent actually connects to the real world.
• 11:29
  Speaker 2: It might be from the microphone, the voice camera, video or even a sensor like temperature or GPS this actually is your maybe like an audio transcription or image analysis, motion detection etc.
• 11:43
  Speaker 2: That can be done on the device.
• 11:45
  Speaker 2: And this local perception is what makes age agents situationally aware.
• 11:53
  Speaker 2: And then we have the generation.
• 11:54
  Speaker 2: This is the agent's expensive or the expressive cognitive power.
• 12:00
  Speaker 2: This can generate natural language response.
• 12:03
  Speaker 2: It can generate task summaries, decisions.
• 12:05
  Speaker 2: This is what makes generative and not just reactive.
• 12:22
  Speaker 2: And the third one is the planning agent.
• 12:25
  Speaker 2: This is the intelligent loop.
• 12:26
  Speaker 2: Instead of just operating one output, the agent might need to decide a sequence of actions, frameworks like Landgraf or Autoshock.
• 12:36
  Speaker 2: This is where reasoning happens deciding what to do next, not what to say.
• 12:54
  Speaker 2: Then we have the memory.
• 12:55
  Speaker 2: Edge agent isn't very useful if it forgets everything between the sessions.
• 13:01
  Speaker 2: Agents use a local elected databases like face.
• 13:06
  Speaker 2: Face is actually a light version, is not available, but if you don't compile a GPU version, it actually is a pretty light to store relevant context.
• 13:14
  Speaker 2: This lets them remember previous user interactions, learn from the usage and retrieve past data, even offline.
• 13:20
  Speaker 2: This memory actually makes them contextually aware and personalized.
• 13:25
  Speaker 2: And then we have the tool use, perhaps the most agent-like behavior.
• 13:33
  Speaker 2: Agents can call APIs, run scripts, query databases or control hardware, like maybe turning on the lights or activating a drone.
• 13:46
  Speaker 2: This is where frameworks like Autogen Shine, combining generation with tool invocation logic.
• 13:50
  Speaker 2: This makes them actionable.
• 13:52
  Speaker 2: Not just smart assistants, but autonomous workers.
• 13:55
  Speaker 2: Together, these five pillars from the complete agent architecture, they actually make that complete architecture.
• 14:03
  Speaker 2: This one is a few examples.
• 14:09
  Speaker 2: These are not prototypes or far future concepts.
• 14:14
  Speaker 2: These are deployable AI agents running on edge hardware.
• 14:19
  Speaker 2: Today let's explore a couple of these cases Autonom autonomous factory agents in industrial settings like manufacturing plants.
• 14:29
  Speaker 2: A vision transformer model can detect irregularities like misalignment or broken parts.
• 14:35
  Speaker 2: A local LLM maybe a Mistral 7B generates a defect report in natural language.
• 14:42
  Speaker 2: If needed, it triggers a robotic arm or files an alert for inspection.
• 14:48
  Speaker 2: This whole inspection process is automated.
• 14:54
  Speaker 2: It used to happen and this can happen without any calls onto the cloud or to a data center.
• 15:02
  Speaker 2: Agents are embedded in line, continuously learning from.
• 15:06
  Speaker 2: Failure.
• 15:07
  Speaker 2: Patterns stored maybe in light databases like maybe SQLite, are queried by vector databases.
• 15:13
  Speaker 2: And then rural health assistants.
• 15:16
  Speaker 2: Imagine like a health worker in a remote village.
• 15:18
  Speaker 2: They speak into a mic Patient has mild fever and a cough.
• 15:22
  Speaker 2: Last night gave maybe like 400 milligrams of paracetamol at maybe 10 pm or something.
• 15:28
  Speaker 2: The agent uses whisper for local speech to text.
• 15:31
  Speaker 2: Then quantized models like FEE summarize this into a structured clinical notes.
• 15:38
  Speaker 2: A local vector databases allows the agents to recommend protocols or verify previous symptoms, and the response is read aloud to the clinician using a text-to-speech, fully offline.
• 15:51
  Speaker 2: This setup brings contextual autonomous support to places where there's no connectivity, without compromising any privacy as well, and in disaster response drone, in emergency zones, a drone scans a flooded area, feeding images to some of the image or object detection, like maybe YOLO or any object detection tools, and then we can also use SAM to segment if required, and then that data or you can also send it to something.
• 16:26
  Speaker 2: What I've done in the demo is used Lava, which is a multimodal model, to really identify the scenario itself, not just the objects in the image.
• 16:35
  Speaker 2: This agent generates captions, like maybe floodwater has breached and cars stranded or something like that, and then it uses a local LLM to plan the path for human responders or other drones.
• 16:49
  Speaker 2: The agent can now log this to local memory, generate a visual map, overlay or broadcast a summary over maybe lower or ultra low bandwidth scenarios.
• 17:03
  Speaker 2: Again, here there's no delay, just actionable intelligence on the device itself.
• 17:11
  Speaker 2: The last one is the personalized retail advisor.
• 17:14
  Speaker 2: I spoke about this.
• 17:16
  Speaker 2: You can use a camera to track the stock values, like how much stock is there on the shelves?
• 17:25
  Speaker 2: Ai agent identifies what's missing and creates real time offers or stock requests.
• 17:33
  Speaker 2: The system can interact with shoppers using audio.
• 17:37
  Speaker 2: Would you like a discount or organic oaths, maybe?
• 17:42
  Speaker 2: These agents are powered by local Olamar runtime, imagine encoders and Whisper-based audio feedback.
• 17:51
  Speaker 2: It's a personalized, private and adaptive experience powered by Gen AI at the edge.
• 17:58
  Speaker 2: So these examples span different industries, but they share the same principles.
• 18:04
  Speaker 2: I wanted to pick a completely diverse examples, but the underlying it's the same.
• 18:10
  Speaker 2: It's a local inference, contextual generation, it's autonomy and privacy.
• 18:16
  Speaker 2: They'll demonstrate the incredible versatility of GenAI agents in different areas.
• 18:22
  Speaker 2: Then in agentic AI applications areas.
• 18:27
  Speaker 2: And then in agentic AI applications, data traverses through model context protocol.
• 18:33
  Speaker 2: It facilitates structured interaction between a language model and its evolving context, whether it's the memory tools or the environment.
• 18:42
  Speaker 2: And then we have the A2A protocol enables agents to communicate, delegate subtasks or coordinate workflows.
• 18:47
  Speaker 2: And then we have the A2A protocol enables agents to communicate, delegate subtasks or coordinate workflows.
• 18:50
  Speaker 2: And we have this shared infrastructure, which includes external APIs, tools, vector databases, file systems or even in buses.
• 19:01
  Speaker 2: And then I did not represent here, but I was actually thinking wherever you see those agents right Scout agent, planner agent and the communicator agent.
• 19:13
  Speaker 2: Each one should operate within its own isolated security zone, enforcing the principle of least privilege.
• 19:21
  Speaker 2: This segmentation actually ensures that if one agent is compromised, others remain unaffected.
• 19:26
  Speaker 2: Inside each agent we actually see the model context protocol.
• 19:30
  Speaker 2: This governs how the agent language model interacts with its evolving context, like previous tasks which it has done, or the memory or tools it accessed.
• 19:43
  Speaker 2: Maintaining the confidentiality and integrity of this context is crucial because it often contains sensitive information, for example customer profiles, intellectual property or a workflow state.
• 19:58
  Speaker 2: And then we have the A2A protocol Between agents.
• 20:04
  Speaker 2: We have the agent-to-agent handoff.
• 20:06
  Speaker 2: This is where agents communicate, delegate tasks or share partial results.
• 20:12
  Speaker 2: These handoffs must be tightly controlled and auditable to avoid data leaks and privilege escalation.
• 20:20
  Speaker 2: This layer is a potential vulnerability.
• 20:23
  Speaker 2: If a summarized agent or, like one agent, the scout agent, receives raw, undetected data from the researcher agent, it might unintentionally expose sensitive information downstream.
• 20:35
  Speaker 2: And then we have shared infrastructure and the risk surfaces.
• 20:39
  Speaker 2: At the bottom is a shared infrastructure that all the agents may access.
• 20:44
  Speaker 2: This includes the shared memory, where all the agents store and retrieve intermediate, state or long-term memory, and external APIs like web search, databases or internal services, and then tools like file processor, translations or calculators, and vector databases or file storages which hold embedding spaces or contextual data.
• 21:09
  Speaker 2: These components form a risk surface why?
• 21:14
  Speaker 2: Because they often cross boundaries.
• 21:17
  Speaker 2: For example, an API might log queries, a tool might behave unexpectedly or a vector DDB might be queried by unauthorized agents.
• 21:25
  Speaker 2: All these actually provide those barriers.
• 21:34
  Speaker 2: So far we talked about what agents do and how they're structured.
• 21:38
  Speaker 2: Now let's talk about how we get these large language models small and fast enough to run at the age.
• 21:46
  Speaker 2: So model optimization has advanced dramatically over the past 18 months.
• 21:53
  Speaker 2: We know about quantization.
• 21:55
  Speaker 2: That's the most impactful trick.
• 21:58
  Speaker 2: It reduces the bit of the model rates we all spoke about that.
• 22:02
  Speaker 2: I'm not going deeper into that.
• 22:04
  Speaker 2: But tools like GPTQ, awq and XL2 helps you quantize with minimal loss of performance.
• 22:11
  Speaker 2: This reduces memory by four to eight times and improves inference speed significantly a must for each device.
• 22:19
  Speaker 2: And quantization compresses.
• 22:21
  Speaker 2: But distillation actually learns.
• 22:24
  Speaker 2: A smaller model is trained to mimic the outputs of a larger model.
• 22:32
  Speaker 2: Phi is a great example of that.
• 22:34
  Speaker 2: It only has 1.3 billion parameters but punches about sweet because it's distilled from GPT 3.5-like data.
• 22:41
  Speaker 2: This technique helps you retain the knowledge while fitting the edge constraints.
• 22:46
  Speaker 2: This technique helps you retain the knowledge while fitting the edge constraints.
• 22:50
  Speaker 2: And then we have a LoRa or QLoRa, which are like fantastic for making edge friendly fine tuning practical.
• 22:56
  Speaker 2: Instead of updating all model parameters, you train and store only low rank weight deltas.
• 23:02
  Speaker 2: These are tiny, usually under 100 MB, and can be merged into a base model at inference time.
• 23:10
  Speaker 2: So, yes, you can actually do on-device fine-tuning which enables personalized or domain-specific agents without retraining them from scratch and then run.
• 23:22
  Speaker 2: So there's some runtime tricks, right Like the final layer of optimization.
• 23:26
  Speaker 2: Use GGUF model format, the standard in LAMAcpp, for optimized disk and memory handling, and enable flash attention too, to speed up the context window operations.
• 23:38
  Speaker 2: Use fused operators and quantaware token streaming for low latency interaction.
• 23:44
  Speaker 2: With these optimizations you can actually serve a 4-bit model at 10 plus tokens per second on something like Jetson.
• 23:59
  Speaker 2: So now let's explore one of the most powerful capabilities of the agents multimodal reasoning and tool.
• 24:05
  Speaker 2: Using Multimodal means, the agent understands more than the text.
• 24:11
  Speaker 2: It can actually take in images via encoders like SAM or MobileNet, or Variance and audio using the Whisper, cpp or Teams for speech to text.
• 24:25
  Speaker 2: It can also read sensor values.
• 24:27
  Speaker 2: Multimodal inputs make agents aware of the real world, not just what users say, but what's happening visually, physically, around them, and then generative output.
• 24:39
  Speaker 2: These agents aren't just interpreting data, they're gathering decisions right, creating captions for images tree fallen on the North Pole right.
• 24:46
  Speaker 2: Creating captions for images tree fallen on the North Road right.
• 24:48
  Speaker 2: Or summarizing events or commands, saying that flood detected in whatever sector, rerouting this particular drone, or responding verbally or visually via text to speech or UI.
• 25:03
  Speaker 2: And they do all this on-device.
• 25:06
  Speaker 2: Then we have the tool using behavior.
• 25:21
  Speaker 2: This is what separate agents from assistants.
• 25:23
  Speaker 2: Agents don't respond, they act Once they process the context and reason over it.
• 25:30
  Speaker 2: They can actually call APIs locally or on a LAN, trigger scripts or shell commands or actually control hardware, like maybe actuate a motor, toggle a sensor, send a LoRa signal.
• 25:43
  Speaker 2: This turns the agent from passive.
• 25:46
  Speaker 2: The demo itself.
• 25:59
  Speaker 2: If we have time, let's go there Now.
• 26:04
  Speaker 2: We actually you.
• 27:19
  Speaker 3: Thank you, I don't know if we can't hear you anymore.
• 27:22
  Speaker 3: I think you have some network issues, because you always go like silent and now you were off, but we can't hear you anymore.
• 27:25
  Speaker 3: Think you have some network issues because you always go like silent and now you were off, but we can't hear you anymore.
• 27:29
  Speaker 3: No, we don't hear anything.
• 27:44
  Speaker 3: Yeah, sorry everyone.
• 27:47
  Speaker 3: As Evgeny mentioned, he just chose that we are live and that just happened.
• 27:53
  Speaker 3: So let me know.
• 27:58
  Speaker 3: Okay, I'll do that.
• 28:06
  Speaker 3: I'll bring you on the other, okay.
• 28:11
  Speaker 2: Can you hear me now?
• 28:12
  Speaker 3: Yes, yes, we can.
• 28:14
  Speaker 3: Okay, we'll go with that, that's okay.
• 28:23
  Speaker 2: We've talked about the input, perception and output, generation and planning.
• 28:27
  Speaker 2: Now we reach a critical capability that makes age agents truly intelligent over time, and that's memory.
• 28:33
  Speaker 2: When we say memory, we don't just mean saving blocks.
• 28:39
  Speaker 2: We are talking about contextual memory that lets agents adapt to previous interactions, personalization that reflects the user environment or situation, and retrieval-based reasoning that enriches generation using stored knowledge.
• 28:53
  Speaker 2: And let's unpack this right like local vector databases To store memory.
• 28:59
  Speaker 2: Edge agents use vector databases that are on-device.
• 29:03
  Speaker 2: Like I said, face can be a good example.
• 29:06
  Speaker 2: Device.
• 29:14
  Speaker 2: Like I said, face can be a good example, for example, not a GPU-based version.
• 29:16
  Speaker 2: And then, for example, you can say summarize previous visits by patient diary, so and so.
• 29:19
  Speaker 2: The agent queries its FACE index and returns, records, notes or even like audio snippets.
• 29:25
  Speaker 2: This is how agents build contextual intelligence.
• 29:29
  Speaker 2: And then the retrieval augmented generation.
• 29:30
  Speaker 2: We pair vector memory with a local LLM using RAG, which stands for retrieval, augmented generation.
• 29:38
  Speaker 2: Tools like LAMA index and GPT cache make it possible to pull relevant memory chunks, feed them into a prompt, generate a personalized, up-to-date response.
• 29:50
  Speaker 2: This bridges the gap between static LLM knowledge and the dynamic user-specific context.
• 29:57
  Speaker 2: And then we have the personalization with memory and drag and place.
• 30:02
  Speaker 2: Agents can remember a username, history and preferences, maintain tone or style consistency, track repeated questions or issues.
• 30:13
  Speaker 2: This enables use cases like a retail agent that recalls a shopper's previous interests, and some of the light embedding models like Mini LM E5 Lite.
• 30:26
  Speaker 2: These are small, fast, highly accurate, perfect for edge inference.
• 30:30
  Speaker 2: All of them can run using sentence transformers or even in Lamacpp style environments with slight tuning.
• 30:38
  Speaker 2: In short, this memory layer is what separates a generic chatbot from a true agent, chatbot from a true agent.
• 30:53
  Speaker 2: And then we want to really focus on these agents, not just generating, but they need to plan sequence and act across multiple stages.
• 30:58
  Speaker 2: So let's explore the three agent planning frameworks that are incredibly relevant here.
• 31:05
  Speaker 2: Langraph again, it's a DAG based planning.
• 31:08
  Speaker 2: It lets you build agent flows like graph, where each node represents a step, a memory retrieval planning tool called validation.
• 31:17
  Speaker 2: You define the edges that describe how state flows from one node to the next.
• 31:24
  Speaker 2: You can loop, retry, verify or even branch based on outcomes.
• 31:30
  Speaker 2: This is ideal for multi-step tasks on agents.
• 31:33
  Speaker 2: An agent might first retrieve prior context from a vector database, then plan a response, then follow a tool for the API, then verify the result, because Langraph is local executable if it's edge workflows perfectly and similarly Autogen.
• 31:56
  Speaker 2: We spoke about Autogen and then Crew AI.
• 31:57
  Speaker 2: It actually provides role-based multi-agent teams.
• 31:58
  Speaker 2: Crew AI takes it further by letting you build a team of agents, each with its own speciality A navigator that plans the sequence, a verifier that checks the constraints, a responder that interacts with the user.
• 32:11
  Speaker 2: This is powerful for multi-agent orchestration on edge, especially when agents need to handle different modalities or responsibilities.
• 32:21
  Speaker 2: You can use this even on Jetson device by assigning roles to smaller models or tasks.
• 32:27
  Speaker 2: This even on Jetson device, by assigning roles to smaller models or tasks.
• 32:33
  Speaker 2: We have seen how powerful and autonomous these edge agents can become.
• 32:37
  Speaker 2: But this autonomy comes with the responsibility, especially when agents are making decisions locally, generating content and interacting with the real world.
• 32:42
  Speaker 2: Now let's talk about the governance auditing control for edge-deployed agents.
• 32:47
  Speaker 2: Encrypted local stores for all local memory, especially user data or logs, must be stored securely.
• 32:55
  Speaker 2: Use encrypted file systems or abstractions like OpenDAL, open Data Access Layer.
• 33:04
  Speaker 2: This helps agents store memory, retrieve data or action logs in a way that are actually tamper resistant and private.
• 33:09
  Speaker 2: Edge does not mean insecure.
• 33:12
  Speaker 2: In fact, local encryption is often more trustworthy than sending data on the network and then dual call, whitelisting and scoping, since agents can call tools like scripts, apis, hardware triggers.
• 33:26
  Speaker 2: We need strict control.
• 33:27
  Speaker 2: Define scope for what they are allowed to do.
• 33:31
  Speaker 2: Use whitelisting to restrict which tools are callable, under which conditions.
• 33:36
  Speaker 2: For example, a nurse assistant can generate reports but not modify medications.
• 33:41
  Speaker 2: You want functional autonomy, not free for all behavior.
• 33:45
  Speaker 2: And then we have the audit log that every agent decision tool, call and response should be logged, not just for debugging but for accountability and compliance.
• 33:58
  Speaker 2: These logs should be structured, timestamped and stored locally and optionally, if they can be synced to your secure hardware.
• 34:05
  Speaker 2: This enables auditability in regulated industries like healthcare, defense and finance.
• 34:12
  Speaker 2: And then the last one is the hardware-based test.
• 34:15
  Speaker 2: More-aged devices often come with hardware-based, hardware-backed security or hardware-based security.
• 34:22
  Speaker 2: That's the trusted platform module and arm trust zone, or encrypted SD cards or secure boot environments.
• 34:31
  Speaker 2: Use these to verify that the agent runtime hasn't been tampered with, that the models are signed, that memory is not leaked during execution.
• 34:39
  Speaker 2: This is especially important for mission critical or safety sensitive agents.
• 34:44
  Speaker 2: And then the emerging protocols.
• 34:46
  Speaker 2: We spoke about MCP and A2A on a standard way and a secure way to communicate, and there's also Cisco A&P, which is the AI network protocol for secure agent to agent communications, and IBM's work on FHE, that fully homomorphic encryption for private model inference, even on edge hardware.
• 35:09
  Speaker 2: These are building the secure fabric needed for decentralized AI systems.
• 35:15
  Speaker 2: And then let's take a step back now and look at the patterns right, the common blueprints of how edge agents are deployed in the real world.
• 35:23
  Speaker 2: These are repeatable designs that show up by any means.
• 35:24
  Speaker 2: These are not comprehensive.
• 35:25
  Speaker 2: These are someable designs that show up by any means.
• 35:26
  Speaker 2: These are not, like, comprehensive.
• 35:27
  Speaker 2: These are some of the patterns.
• 35:31
  Speaker 2: On-demand agents these agents are user triggered.
• 35:33
  Speaker 2: Think of a retail kiosk or an airport translator or a field assistant.
• 35:37
  Speaker 2: Right, they wake up and the user interacts maybe a button press, voice command or even a gesture.
• 35:43
  Speaker 2: They load models, respond intelligently and shut down or idle when needed.
• 35:50
  Speaker 2: These are energy efficient and suited for public facing or battery limited environments.
• 35:56
  Speaker 2: And then we have the event driven loops.
• 35:59
  Speaker 2: Here the agent is always listening to a sensor Maybe it's monitoring temperature, maybe it's morning temperature, motion or light or audio and a threshold is crossed, like a spike in the noise or an object detected.
• 36:14
  Speaker 2: The agent is triggered.
• 36:16
  Speaker 2: It then runs a pipeline like analyze, detect and act.
• 36:19
  Speaker 2: You see this in smart factories, security cameras, environmental sensors or something.
• 36:27
  Speaker 2: And then we have tool enhanced reasoning.
• 36:31
  Speaker 2: Some agents are not just responders, they are planners with capabilities.
• 36:35
  Speaker 2: They use local tools like, maybe, a calculator, gps, sql function, or like a file system or an external hardware.
• 36:44
  Speaker 2: They reason over context and decide when to use a tool, like uploading a report, opening a gate or sending a local message.
• 36:53
  Speaker 2: This is like true autonomy, where the agent thinks through actions and executes these local resources.
• 36:59
  Speaker 2: And then the swarm coordination.
• 37:02
  Speaker 2: This pattern is emerging.
• 37:03
  Speaker 2: Think of multiple drones, robots or devices that each run their own local agent.
• 37:10
  Speaker 2: They share sensor data, learn from each other or coordinate tasks all within themselves.
• 37:18
  Speaker 2: You will see this in agriculture or wildlife monitoring or disaster response, or even in military field intelligence.
• 37:25
  Speaker 2: Right, right, swarm agents act like distributed teams adapting to conditions, collaborating and improving what they're doing.
• 37:33
  Speaker 2: These patterns are not just theoretical.
• 37:36
  Speaker 2: They're being deployed right now and they help us design systems that are resilient to failure, fast to respond and flexible to deploy anywhere.
• 37:44
  Speaker 2: Um, how are we doing with the time like?
• 37:47
  Speaker 2: Is there time to switch for a live demo, or should I actually to respond and flexible to deploy in there?
• 37:49
  Speaker 2: How are we doing with the time Like, is there time to switch for a live demo, or should I actually go with the slides showing screenshots?
• 37:56
  Speaker 1: of Denver.
• 37:56
  Speaker 1: You actually exceeded the time by 10 minutes, yeah.
• 38:03
  Speaker 2: Okay, I'll quickly one minute.
• 38:06
  Speaker 2: I'll take one minute.
• 38:06
  Speaker 2: This is actually three different agents trying to take the input and what these agents are doing.
• 38:13
  Speaker 2: I had, like multiple agents here you can see for the scout agent, I have a model and a YOLO model and for the planner agent, I had a fee and a communicator agent, I have a Mr model and when I actually upload an image it actually generates a report like this this is the image I have uploaded and it each agent actually sends the information to the next agent.
• 38:39
  Speaker 2: But you can see it identifies the whole image, what's there.
• 38:42
  Speaker 2: This is the output from the YOLO and then it identifies the whole scene from the lava and it actually plans on what has to be done from the output of the scout agent and the last communicated agent generates a report to publish it to everyone, communicate of what is happening.
• 39:00
  Speaker 2: Sorry, I don't have time.
• 39:11
  Speaker 1: Thank you.
• 39:12
  Speaker 1: Thank you, aruna, for the very informative talk.
• 39:13
  Speaker 1: I mean it was really interesting this deep dive into the architecture and optimization of edge-deployed AI agents and also the demos, which are really interesting.
• 39:21
  Speaker 1: For the sake of time, I'm going to pick up again two questions from the audience.
• 39:25
  Speaker 1: One question from George Williams.
• 39:27
  Speaker 1: Of time, I'm going to pick up again two questions from the audience.
• 39:28
  Speaker 1: One question from George Williams.
• 39:29
  Speaker 1: Since agents are based on large or small language models, are they also vulnerable to hallucinations and are there ways to quantify this in agentic deployments and what are the best practices to reduce hallucinations at the edge?
• 39:41
  Speaker 1: Esp in resource-constrained hardware.
• 39:46
  Speaker 2: ESP in resource-constrained hardware?
• 39:48
  Speaker 2: Yeah, so irrespective of whether it's Edge or in the data center, agents are always domain-specific, right.
• 39:55
  Speaker 2: So we are not going on a very broad range of questions.
• 39:58
  Speaker 2: When you're answering them, asking them, you actually have a set of data which you train them for a very domain-specific purpose.
• 40:06
  Speaker 2: For example, when I used just the YOLO model for the demo, it was just giving me a random thing saying that it's a boating experience even when there's a disaster.
• 40:16
  Speaker 2: That's the reason I'm actually using that in combination with a LARVA model which has the holistic context.
• 40:21
  Speaker 2: So the domain specific models trained on specific data is the one way to reduce the hallucinations, to give the data which it is responding to.
• 40:31
  Speaker 2: So I assume there'll be less hallucinations by having domain specific models responding just to those questions.
• 40:40
  Speaker 1: The last question is by Vikash Kodati and regarding the vision to action slide like flood detection and rerouting are they already being offered as a service from Dell?
• 40:55
  Speaker 2: It's not a service from Dell.
• 40:56
  Speaker 2: Dell actually offers multiple endpoints, like the H-devices, ruggedized hardware to deploy in the disaster-like situations or anywhere, maybe on a truck or something.
• 41:07
  Speaker 2: We have had many use cases where these things are actually deployed on the truck right.
• 41:15
  Speaker 2: So for disaster, for defense, for agriculture all these were deployed on the truck in ruggedized environments.
• 41:23
  Speaker 2: And also the latest version of the laptops have quite a large GPU memory and all the models I was showing on that demo were running on one laptop, so and they're not like a very small optimized model.
• 41:39
  Speaker 2: So I was running with Mistral and YOLO and Lava and Fee all four of them on the same laptop at the same time.
• 41:48
  Speaker 2: So they're quite capable.
• 41:50
  Speaker 2: The models are becoming more powerful and the devices are becoming larger, so