Latest / AI Fire Daily / #64 Robin: Building a "SaaS-Noir" Jarvis in Cursor - How to Code a Voice-Native Desktop Agent Without Writing Code
Transcript
- 0:00So I had this moment recently. Yeah. A real moment
- 0:04of just, well, absolute awe. I was just sitting
- 0:06there talking to my computer. Right, like we
- 0:08all do these days. But I was not chatting with
- 0:10a normal web bot. I was commanding a customized
- 0:13desktop agent. It was literally building things
- 0:16for me in real time. That is always such a trip
- 0:19to experience. It really is. The mouse was just
- 0:21moving on its own. Code was executing. I did
- 0:25not write a single line of code myself. It really
- 0:28does feel like science fiction, doesn't it? That
- 0:30shift from typing to just speaking is wild. It
- 0:32totally changes everything about how we interact
- 0:34with machines. So welcome to this deep dive.
- 0:38We are incredibly glad you're joining us today.
- 0:40We really are. Today, we are unpacking a fascinating
- 0:43step -by -step guide. It shows you how to build
- 0:46a personal AI assistant. Basically, think of
- 0:48it as your own personal Jarvis. Exactly. We are
- 0:50going from a totally blind folder to a fully
- 0:53operational companion. And we are doing it using
- 0:56an AI tool called Cursor. And we are relying
- 0:59on just three massive prompts to do it. Yeah,
- 1:02which is crazy to think about. Building a custom
- 1:06desktop agent used to require an entire engineering
- 1:09team. Oh, absolutely now. It is entirely accessible
- 1:12right now to almost anyone. But, you know, before
- 1:15getting to the autonomous magic, we need a foundation.
- 1:18We have to lay down the building blocks first.
- 1:20Right. We cannot just jump straight into the
- 1:22voice commands. Exactly. So let's talk about
- 1:24the stack. The tech stack here is actually quite
- 1:26simple. You only need four core tools to make
- 1:30this entire system work. We mentioned Cursor
- 1:32already. Cursor is an AI code editor that essentially
- 1:35writes software for you. It handles all the heavy
- 1:38lifting of the actual build. Next, you need an
- 1:41OpenAI API key. Right. And an API key is just
- 1:45a secret password to access AI features. That
- 1:47is a great way to put it. That specific key connects
- 1:50your new assistant to GPT real -time too. Which
- 1:53is the engine giving it live voice capabilities,
- 1:55right? Exactly. You also need something called
- 1:57the EXA API that handles the live web searches
- 2:00later on. And finally, a framework called Electron.
- 2:03We will get to how that works in just a second.
- 2:05But first... You need a clean workspace to start.
- 2:08You start by downloading Cursor to your machine.
- 2:11You sign in and open their agents window. Then
- 2:14you just create a brand new empty folder. The
- 2:18guide suggests naming it Riley Custom Jarvis.
- 2:21This folder will eventually hold your full desktop
- 2:24application. Yeah, it becomes the home base.
- 2:26Then you grab your OpenAI API key. Which you
- 2:30just generate directly on their developer website.
- 2:32Right. Now, if you are familiar with tech, you
- 2:34already know this, but you must treat this key
- 2:37exactly like your bank password. Right, because
- 2:40if that key leaks out into the world. But people
- 2:42can rack up massive bills on your account. Very
- 2:45quickly. So keep it completely secure locally.
- 2:47Always. So we have our blank folder ready. We
- 2:50have our secure API keys locked and loaded. Now
- 2:52comes the first big prompt to the system. This
- 2:55is where the magic really starts happening. You
- 2:57do not just ask Cursor for a basic chat bot.
- 3:00Right. You go way back. You give it one massive,
- 3:02highly detailed set of instructions. You explicitly
- 3:06ask it to build a companion named Ricky. And
- 3:09you request GPT real -time to voice chat integration.
- 3:13Plus, you specifically mandate interruption support.
- 3:16Yes. Interruption support is absolutely crucial
- 3:19for a voice agent. Because otherwise it just
- 3:22talks over you. Exactly. It means you can cut
- 3:24the AI off while it is speaking. It stops talking
- 3:27immediately, listens and pivots. It makes the
- 3:30conversation feel truly human and natural. You
- 3:33also asked for an animated face on the interface.
- 3:35Right. It needs to blink naturally. It needs
- 3:37to show different moods. It has to lip sync to
- 3:39the audio. And you want a very clean user interface
- 3:41too. You ask for a primary companion window and
- 3:44an artifact panel. That artifact panel is vital.
- 3:47It separates the conversation from the actual
- 3:49data it fetches. You also request specific tool
- 3:52integrations in this prompt. Web search using
- 3:55that EXA API. Image generation capabilities.
- 3:59Plus, full support for mermaid diagrams. Right,
- 4:02and mermaid diagrams are visual charts generated
- 4:04directly from simple text. You put all of those
- 4:07complex requirements into one strong prompt.
- 4:09Then you just let Cursor do the actual coding.
- 4:12And you want to use the strongest model available
- 4:14for this step, right? Oh, definitely. The guide
- 4:17used GPT 5 .5 high for this initial run because
- 4:21this first build has an incredible amount of
- 4:24moving parts. Using Cursor is honestly like having
- 4:27a senior developer sitting next to you. It is
- 4:29like stacking Lego blocks of data. I love that
- 4:32analogy. You just describe the house, then they
- 4:34pour the concrete. That is a perfect way to visualize
- 4:37it. You act as the architect and the AI acts
- 4:40as the builder. How long does that usually take?
- 4:42It takes about 10 to 15 minutes to run. It has
- 4:44to wire up the voice model securely. Right. It
- 4:47is building the user interface entirely from
- 4:49scratch. It creates the animated face and connects
- 4:52all those tool systems. When it finishes, you
- 4:55have a working talking version. Your desktop
- 4:57companion can react to your voice immediately.
- 4:59But before we move on, I want to clarify one
- 5:02specific tool. Jared, what is up? Why exactly
- 5:05is Electron needed in this stack? Ah, well, Electron
- 5:08is a framework that packages web code into normal
- 5:11desktop software. It takes the web -based code
- 5:14Cursor just wrote and makes it native. Ah, so
- 5:16it packages web code to run like normal desktop
- 5:18software. Exactly. It makes Ricky feel like a
- 5:22real application, not just a browser tab. Okay,
- 5:24so, our concrete is finally poured. The first
- 5:28build is complete. But like any prototype, it
- 5:31is going to be rough around the edges. Yeah,
- 5:33the first test always reveals where the real
- 5:35work is. The voice chat and the basic animations
- 5:38usually work great out of the gate. Ricky blinks,
- 5:41listens, and answers questions fairly naturally.
- 5:43He can list out what he is capable of doing.
- 5:46And the mermaid diagrams usually render fine
- 5:48after a tiny adjustment. But other things will
- 5:51inevitably fail on the first try. In the guide.
- 5:54The web search failed entirely initially. Right.
- 5:56The EXA API key was simply missing from the underlying
- 5:59code. Two sec silence. Which is completely normal
- 6:02for AI coding. Your first build just proves the
- 6:04core application actually works. The second build
- 6:07is about making that system reliable. And here
- 6:09is where the strategy shifts. You don't just
- 6:11go in and tinker. Right. You do not fix these
- 6:13tiny bugs one by one. You give Cursor another
- 6:16massive revision prompt. You outline all the
- 6:19main problems together in one go. You tell it
- 6:22to fix the mermaid parse errors. You tell it
- 6:25to properly integrate the XI API key. You demand
- 6:29that both tools execute flawlessly every single
- 6:32time. This is also when you completely overhaul
- 6:34the interface. Yeah, the first version usually
- 6:36looks like a clunky prototype. You want to ask
- 6:39for a very clean 50 -50 layout. Face on the left,
- 6:43artifacts on the right. Simple. elegant buttons
- 6:46resting cleanly at the bottom. And you want to
- 6:48remove all those messy prototype looking rounded
- 6:51boxes. You have to make the assistant feel highly
- 6:54useful. Because cluttered interfaces make the
- 6:56AI feel much less capable psychologically. Exactly.
- 7:00But, you know, I have to push back on this approach
- 7:02for a second. Okay, lay it on me. Is it really
- 7:04effective to send one massive error correction
- 7:07prompt? I usually prefer fixing things line by
- 7:10line myself. A lot of developers do. Because
- 7:12if you change too much at once, things tend to...
- 7:14break. It definitely feels risky if you're used
- 7:17to traditional coding, but modern models actually
- 7:20prefer holistic context. Really? Yeah. When you
- 7:23give it all the errors at once, it sees the entire
- 7:25system. It understands how a broken search function
- 7:28affects the user interface layout. Right. And
- 7:31it can adjust the dependencies all at once. Exactly.
- 7:33It connects those dots much better than isolated
- 7:36piecemeal fixes. Next, you need to upgrade the
- 7:39actual output style. Because web search formatting
- 7:42is usually a massive problem initially. Oh, huge.
- 7:45The AI often tries to display search results
- 7:48in heavy data tables. And tables can feel visually
- 7:51overwhelming. They are really hard to read quickly
- 7:54on a screen. so you tell cursor to stream results
- 7:57as clean markdown instead markdown is a lightweight
- 8:01formatting style keeping plain text highly readable
- 8:05it streams those results directly into the artifact
- 8:07panel and the answer looks fast clean and incredibly
- 8:10easy to scan you also add specific voice commands
- 8:14during this massive revision a great one is programming
- 8:16it to respond to show me the menu That specific
- 8:20command opens a simple visual list of its capabilities.
- 8:23You also use this prompt to smooth out the mouth
- 8:25animations, because the first version usually
- 8:27just flaps the mouth open and closed randomly.
- 8:30It looks a bit like a cheap puppet. The revision
- 8:32makes it feel much closer to actual human speech.
- 8:35Better timing and smoother movements make a huge
- 8:37psychological difference. At this point, Ricky
- 8:40is becoming a very useful tool. He talks, he
- 8:43searches, and he presents results clearly. Beat.
- 8:47But let's clarify that formatting choice for
- 8:49a second. Why is clean Markdown preferred over
- 8:52data tables for search results? Oh, well, Markdown
- 8:56just flows better natively. Tables force your
- 8:58eyes into a rigid grid, which slows down reading.
- 9:01Right. Markdown keeps text flowing smoothly without
- 9:03feeling visually overwhelming. Exactly. Rapid
- 9:06readability is absolutely key for a desktop assistant.
- 9:09So now that Ricky looks good, we want him to
- 9:12act. We need to let him actually control the
- 9:14desktop environment. This brings us to a feature
- 9:17called computer use mode. Which is huge. It is.
- 9:20This is what lets Ricky interact directly with
- 9:22your operating system. He can open applications.
- 9:25He can click around the screen autonomously.
- 9:27He can type directly into a tool called Codex.
- 9:29And Codex is just a programming environment where
- 9:32AI interacts with your system. He can submit
- 9:35complex prompts entirely driven by your voice
- 9:37commands. But there is a massive frustrating
- 9:41problem in the first version. The dreaded approval
- 9:44loop. It is the worst. The system pauses before
- 9:47every single autonomous action it takes. It stops
- 9:50to ask for explicit confirmation, just to start
- 9:53typing. Then it asks for another confirmation,
- 9:56just to press the enter key. It might pause yet
- 9:59again before submitting a final prompt. It completely
- 10:01breaks the flow of having a natural voice agent.
- 10:04I still wrestle with prompt or diff myself, you
- 10:06know. It happens to all of us. Getting frustrated
- 10:08when my own AI tools constantly pause to ask
- 10:12for permission. It just ruins the momentum. It
- 10:14makes the tool feel broken, tedious, and heavy.
- 10:17A voice agent is supposed to be entirely hands
- 10:20-free. Right. If you have to click approve constantly,
- 10:22it defeats the purpose. So the fix here is another
- 10:26direct prompt to cursor. You tell the builder
- 10:28to find where that approval requirement lives
- 10:30in the code. You direct it to completely remove
- 10:33that blocker for trusted local actions. The assistant
- 10:36must be able to type and submit smoothly. It
- 10:38should execute everything immediately after your
- 10:40original voice command. We want the assistant
- 10:42to be safe, obviously, but constant confirmation
- 10:45ruins the entire user experience. It totally
- 10:49breaks the illusion of a helpful companion. You
- 10:51also need a major UI adjustment for this specific
- 10:54mode. That large 50 -50 window we built earlier
- 10:57gets in the way here. Right, you cannot actually
- 10:59see your desktop if Ricky is blocking half the
- 11:02screen. So you prompt Cursor to dynamically change
- 11:06the application layout. When in computer use
- 11:09mode, it becomes a small translucent overlay
- 11:11instead. It just sits quietly down in the bottom
- 11:14left corner of your screen. It features one clear
- 11:16button to return to the normal 50 -50 mode. This
- 11:20lets you see the full desktop while Ricky does
- 11:22the actual work. You speak, it acts in the background,
- 11:25and you stay in control. You do not have to babysit
- 11:27every single mouse click it makes. Quick question
- 11:30about removing those confirmation blockers, though.
- 11:32Is it a security risk to let the AI act autonomously
- 11:35on your machine? It is definitely a balance.
- 11:38You only remove those blockers for specific,
- 11:40highly trusted local actions. You keep strict
- 11:43safeguards in place for things like deleting
- 11:45files or sending emails. Got it. Safety first.
- 11:48But your trusted local apps get a fast pass.
- 11:51Exactly. You want speed and fluidity where it
- 11:53is completely safe to do so. Sponsor break. Sponsor
- 11:57break ends. So with that heavy friction removed,
- 12:00We can finally build a true workflow. We can
- 12:03chain these isolated abilities together beautifully.
- 12:06This is where we build the voice -controlled
- 12:08creative workstation. Specifically, a voice -driven
- 12:11thumbnail generator. Yes. You use your voice
- 12:15to generate and edit images entirely hands -free.
- 12:18Think about the old way. Opening an image tool,
- 12:21typing prompts manually, downloading files, and
- 12:24editing them again and again. It takes forever.
- 12:26Now you just talk to Ricky naturally. You say
- 12:29what you want. And he generates a thumbnail instantly.
- 12:31Then you keep refining and editing that image
- 12:33purely by voice. But there is a crucial UI rule
- 12:36required to make this work well. The guide refers
- 12:39to it as the grid system. Right. You have the
- 12:41AI display the generated images in a clean numbered
- 12:44grid. You show exactly three images per row on
- 12:47the screen. And the newest image is always automatically
- 12:51placed at position number one. Plus, every single
- 12:54image on the screen gets a specific visible number.
- 12:58This numbering system is what enables true voice
- 13:01editing capabilities. Without visible numbers,
- 13:04the assistant simply does not know which image
- 13:06you mean. With those numbers, you can speak incredibly
- 13:09naturally to the system. You can just say, add
- 13:12blue sunglasses to number two. Or you can say
- 13:15something like, pull up photo 13. The assistant
- 13:18keeps the entire visual history visible for you
- 13:20at all times. You can easily compare different
- 13:22versions and revert to previous results seamlessly.
- 13:25You can tell it, make three more versions of
- 13:27number one. And Ricky is not just a simple image
- 13:29bot either. Because of how we built the stack,
- 13:31you can combine tools in one single session.
- 13:34You can ask him to search the web for reference
- 13:36material first. He can summarize those findings
- 13:38directly in that artifact panel we built. Then
- 13:41he can generate a brand new thumbnail based on
- 13:43that specific summary. He can even draw a mermaid
- 13:45workflow diagram explaining his exact steps.
- 13:49All of this happens smoothly in one continuous
- 13:51conversation. It is honestly incredible to watch.
- 13:55Whoa. Imagine scaling to a billion queries just
- 13:58speaking to the corner of your screen while it
- 14:00builds a whole project. It really turns a simple
- 14:03chatbot into a real creative workstation. It
- 14:07saves an immense amount of manual tool switching
- 14:09and cognitive load. You just speak your thoughts,
- 14:13review the grid, and edit by number. It is a
- 14:16completely fluid, frictionless creative process.
- 14:20I do want to highlight one specific design choice
- 14:22there, though. Sure, go ahead. Why must the newest
- 14:25generated image always be at number one? Ah,
- 14:28well, you don't want to hunt through a grid for
- 14:30your latest generation. Right. Makes sense the
- 14:32freshest idea is always front and center for
- 14:34immediate feedback. Exactly. It keeps the momentum
- 14:36going perfectly. Let us zoom out for a second
- 14:39and recap this core journey. We started with
- 14:41a completely blank workspace folder today. We
- 14:44ended with a fully functioning custom desktop
- 14:46agent, a personal system that talks naturally
- 14:49and physically controls your mouse. It searches
- 14:52the web, edits images and completely bypasses
- 14:55the annoying approval loop. We built the entire
- 14:57thing with just three prompts and zero coding.
- 15:00It really proves that building a Jarvis is no
- 15:03longer a massive undertaking. Tools like Cursor
- 15:06and GPT Realtime 2 make it accessible to everyone.
- 15:10The first version you build will absolutely not
- 15:12be perfect at all. And honestly, that is completely
- 15:15fine. You start with the basic application structure.
- 15:17You test what works. And you use holistic prompts
- 15:21to fix what breaks. Then you keep adding the
- 15:23specific tools that matter most to your workflow.
- 15:26You can add email, deep research, reporting,
- 15:29or even Slack integrations. Because the real
- 15:31value here is not just having a fun toy chatbot.
- 15:34It is building a highly personal system to get
- 15:37real work done faster. We want to leave you with
- 15:40something to really think about today. If you
- 15:42were to build your own custom Ricky tonight.
- 15:45What hyper -specific, tedious task in your daily
- 15:47life would you automate first? Is it sorting
- 15:50through hundreds of daily emails? Researching
- 15:53competitors? Graping data out of endless Slack
- 15:55channels? Imagine delegating that entire burden
- 15:59to a secure voice in the corner of your screen.
- 16:01A voice you built yourself, perfectly tailored
- 16:04to how you think it'd work. Thanks for joining
- 16:06us on this deep dive.