Latest / Agentic Conversations (formally mlops.community) / Building Cursor: A Fireside Chat with VP Solutions Ricky Doar

Transcript

Building Cursor: A Fireside Chat with VP Solutions Ricky Doar · Agentic Conversations (formally mlops.community)

• 0:05
  How's it going? I'm good man.
• 0:07
  How you doing? I'm doing well.
• 0:09
  I'm doing well. I'm excited to be here.
• 0:11
  First of all, thanks for coming on here.
• 0:12
  Thanks for doing this. I'm super excited to chat with
• 0:15
  you about Cursor. It's pretty much every single
• 0:18
  developer that I know is either a die hard user of Cursor or
• 0:23
  they are playing around with it and getting a ton of value from
• 0:26
  it. You've been leading the whole
• 0:28
  solutions team. You talk with forward deployed
• 0:32
  engineers every day. That's like literally your job.
• 0:35
  And so I want to go over all of the cool stuff that you've been
• 0:39
  seeing out there in the wild. Yeah, I would love that.
• 0:43
  Sounds awesome. It's been a it's been a wild
• 0:45
  year. Definitely Cursor has been have
• 0:49
  having quite quite the year, quite the the the kind of year
• 0:52
  of growth and we've been talking to hundreds and hundreds if not
• 0:55
  thousands of enterprise companies at this point.
• 0:57
  That's huge, man. And Congrats on the huge
• 0:59
  announcement last week. I think everybody saw the number
• 1:02
  in the raise and had to do a double take.
• 1:05
  So that is very cool, very exciting.
• 1:08
  I know you've done a bunch of stuff.
• 1:09
  You were at Versail, you were doing, I mean you've been doing
• 1:13
  this kind of thing for ages. I know you're excited about
• 1:16
  what's going on in Cursor and maybe we can just go into this
• 1:20
  real fast with like what patterns are you seeing the most
• 1:25
  successful folks doing when your forward deployed engineers who
• 1:30
  are on the front lines are coming back and saying like,
• 1:33
  dude, you'll never believe how these guys are using Cursor.
• 1:37
  Yeah, yeah. I mean, it's definitely, you
• 1:39
  know, engineers everywhere are kind of learning how to build
• 1:42
  with AI. They have been for the past few
• 1:44
  years, but it's really kind of accelerating this past year.
• 1:48
  I, I think first and foremost it, it, it is good to recognize
• 1:52
  that working with AI is, is a new skill.
• 1:56
  It is a skill that is adjacent to engineering.
• 1:58
  It's not exactly engineering and just just like, you know, great
• 2:02
  engineers learn the right ways to think 1st and that's kind of
• 2:06
  more important than, you know, memorizing syntax.
• 2:09
  It's very, very similar in the AI space is there's kind of a
• 2:13
  new way to kind of think alongside these LLMS, Whether
• 2:17
  you're kind of working with AI in the foreground and kind of
• 2:20
  working as a pair programmer with AI or spinning up these
• 2:25
  agents in the background and kind of spinning up these tasks.
• 2:28
  Kind of the the best people that we work with on a day-to-day
• 2:31
  basis are the ones that are really, really good, but at
• 2:33
  breaking down large problems into bite sized tasks that the
• 2:38
  AI is far more likely to be able to accomplish rather than, you
• 2:42
  know, either going way too small or way too large and kind of
• 2:46
  running into these areas where AI hallucinates.
• 2:48
  So I think that that's probably the first and foremost skill we
• 2:51
  see that separates the, you know, power users from the
• 2:54
  people that, you know, in the early days struggle is can you
• 2:58
  understand the right size of problem to hand off to AI?
• 3:02
  And it's really all about how do you manage context and how do
• 3:04
  you build your prompts? Yeah, the context is so huge.
• 3:08
  I actually, if you stick around you'll see I made a video about
• 3:13
  context while I am here in Amsterdam enjoying myself.
• 3:17
  But is there like a rubric that you have looked through on how
• 3:22
  to separate what is too big and what is too small?
• 3:26
  Or is it just like trial and error throwing things at the the
• 3:28
  wall? It's, it's a little of both,
• 3:31
  right? You know, as these models get
• 3:33
  smarter, you know, this has been obviously, again, a huge year
• 3:36
  for AI. We've got seen tons of model
• 3:37
  releases throughout the year. You know, if we take like
• 3:40
  Anthropic as an example, you know, Claude 35 turned into 37
• 3:44
  turned into four turned into 4-5.
• 3:46
  There's also Sonnet, there's Opus.
• 3:48
  As these models get better and they get smarter, it kind of
• 3:51
  changes what they're able to accomplish.
• 3:54
  And you know, there's there's also no substitute for just kind
• 3:57
  of getting to know these models. These models have different
• 3:59
  levels of intelligence. They also tend to have, I mean,
• 4:02
  if you talk to engineers that work with these models on a
• 4:04
  day-to-day basis, they have personalities and what they're
• 4:07
  at and what they're bad at. And I wish there was some easy
• 4:10
  answer of just a framework of how big a problem should be so
• 4:14
  that the AI can accomplish it. But a lot of it is getting to
• 4:17
  know how good the AI is at the given task that you're working
• 4:20
  on and really kind of learning what, what size task can you
• 4:24
  know a model like GPT 51 codecs that just released recently
• 4:28
  solve that maybe GPT 4 or even GPT 5 couldn't solve, you know,
• 4:33
  two months ago. So a lot of this is, is just
• 4:35
  getting familiarity with the tools you're building with.
• 4:38
  And then you kind of build this internal barometer of, you know,
• 4:41
  what can this model accomplish and what can't did?
• 4:44
  Where does it often go wrong or or hallucinate?
• 4:48
  All right Randy, I got to take off my glasses because I need to
• 4:52
  see you for this one. I need you to see me for this
• 4:54
  one. What are some anti patterns?
• 4:56
  What are some ways that we shouldn't be using cursor?
• 5:00
  Yeah, it's a really great question.
• 5:02
  I mean, I think one of the things that we've seen that
• 5:06
  people struggle with is either they get extremely reliant on
• 5:12
  the AI and they forget that that a huge part of building AI is
• 5:17
  not replacing your brain to strategize and to understand
• 5:21
  your code base and to, you know, build the right, make the right
• 5:24
  architectural decisions as you're building out with AI.
• 5:28
  It's certainly extremely powerful, extremely helpful.
• 5:31
  You can ask a bunch of questions of your code base.
• 5:33
  You can use it to ramp up on code bases very quickly.
• 5:36
  But I think a lot of there's potentially can skip that.
• 5:39
  That first stage of do you know your code base extremely well,
• 5:44
  right? There's multiple ways you can
• 5:45
  use cursor. One of them is just ask mode.
• 5:48
  It means it's not going to write any code.
• 5:49
  It's just going to answer questions about how your code
• 5:51
  base works. And one of the downsides of AI
• 5:56
  is it can make you lazy. And as you get a little bit
• 5:59
  lazier, you start giving vaguer and vaguer prompts, You start,
• 6:03
  you know, pushing off a lot of the strategic decisions to AI as
• 6:07
  well. You know, there's a world where
• 6:08
  we might get to a place where models are good enough, where
• 6:10
  that's OK. But for now, it's it actually
• 6:13
  just lulls you into a false sense of security.
• 6:16
  You have to stay in the driver's seat.
• 6:17
  You have to make the right decisions architecturally.
• 6:20
  You have to drive the agent in the right direction.
• 6:22
  I think that's the biggest anti pattern we see, you know, across
• 6:25
  the board. That's so true.
• 6:27
  You get into this false sense of comfort and you keep offloading,
• 6:31
  offloading, offloading and then it'll it'll bite you right when
• 6:35
  you're not expecting it. Exactly.
• 6:38
  Well, since you said how are we doing, like you gave us a that
• 6:42
  anti pattern and you mentioned searching or getting up to date
• 6:45
  on a code base really quickly. I'm wondering about Greenfield
• 6:51
  versus brownfield and basically what you're seeing when you're
• 6:53
  going from zero to 1 versus what you're seeing when you're
• 6:57
  dealing with a code base that's 15 years old.
• 6:59
  Maybe you're still you got some stuff that's deprecated in that
• 7:02
  code base to the outside world, but internally it's not like how
• 7:06
  do you deal with cursor in those instances?
• 7:11
  Yeah, I mean, you know, obviously AI is super exciting
• 7:14
  in the Greenfield space. It's fun to make a prompt and
• 7:17
  just see, you know, something it created before your eyes.
• 7:21
  But I think this is also kind of one of the biggest advancements
• 7:23
  over the last year with Cursor, with other tools.
• 7:26
  But Cursor is phenomenally good at this is how do you take the
• 7:29
  context of your code base and use that to your advantage?
• 7:33
  This is also, you know, there's, there's discussions, debates in
• 7:37
  the AI space of what is the best way to use a code base
• 7:41
  correctly. Some tools, they only grep
• 7:44
  through the through the the code base as you're writing prompts
• 7:48
  cursor has found phenomenal success indexing your entire
• 7:51
  code base before you start asking any questions.
• 7:55
  So it, it indexes your code base, it creates essentially a
• 7:57
  semantic search. So if I say like, hey, I want to
• 8:00
  add authentication to this page, it can actually look throughout,
• 8:04
  you know, a massive code base, 500,000 miles, you know,
• 8:07
  millions and millions of lines and determine what area of the
• 8:10
  code base are most relevant to that prompt and pull that in.
• 8:14
  So one, the, the tools have got a lot better at brownfield
• 8:18
  because they can pull a lot of the surrounding context and
• 8:21
  actually have patterns to match off of rather than just purely
• 8:25
  building whatever they decide to build.
• 8:28
  And I think this is an area where both the the engineer has
• 8:31
  to be familiar with the, the space that they're building in,
• 8:35
  but also the tools can pull context from different, you
• 8:38
  know, codes, different files throughout the code base to have
• 8:41
  patterns to build on top of. So I think it's actually a
• 8:44
  little bit of a of a myth right now that AI is far better at
• 8:48
  Greenfield than it is at brownfield.
• 8:50
  That was true maybe last year when there wasn't a good ways to
• 8:54
  semantically search to quickly grip through your code base.
• 8:58
  But now with cursor, we have semantic search.
• 9:00
  We also have lightning fast parallel grip.
• 9:03
  As you ask these questions, the the most important part is if
• 9:07
  you can give AI examples to follow, right?
• 9:10
  You know, I have, you know, I'm trying to build out a new API or
• 9:14
  new infrastructure or a new, you know, whatever it is I'm trying
• 9:17
  to fix this bug or refactor this area, map it to other areas that
• 9:21
  you know are are are built well. And do you do that just like
• 9:26
  kind of give pointers, maybe check this file or I've seen
• 9:30
  this done well with this API or I want to create an API like
• 9:33
  this? Yeah, definitely.
• 9:35
  So I mean, specifically in cursor you can add different
• 9:39
  files and say like, hey, you know, I really love how this API
• 9:42
  framework is set up. I want to, you know, add a new,
• 9:45
  new endpoint to this API, have it map the the the architectural
• 9:52
  choices that have been made before, right?
• 9:54
  You can add a lot of these different things.
• 9:56
  This is also, you know, very important when you're building
• 9:58
  front end, kind of like, you know, changing front end pieces
• 10:02
  is you can tell the AI, hey, we have a very specific design
• 10:05
  system. Please make sure you're using
• 10:08
  the components, the colors, the design framework that we've used
• 10:12
  to build the rest of this application.
• 10:14
  A lot of building correctly with AI is making sure that you're
• 10:19
  minimizing the amount of arbitrary decisions that the LLM
• 10:23
  can make. It you know if if the arbitrary
• 10:27
  decision is right 80% of the time, but it has to make 100
• 10:30
  arbitrary decisions, you're gonna build something that is
• 10:33
  not quite correct if you can minimize those arbitrary
• 10:36
  decisions. And with brownfield it actually
• 10:38
  is easier to minimize those arbitrary decisions 'cause you
• 10:41
  have a code base to map onto. Yeah, of course you can say, Oh
• 10:45
  yeah, like this is an obvious example and I like how you said
• 10:48
  that with the examples. How have you seen the best folks
• 10:53
  maintain their context windows? Yeah.
• 10:57
  So I think there's there's a lot of, I mean, there's a lot of
• 11:01
  best practices. There's also a lot of like false
• 11:03
  myths about context. I think, you know, as we go
• 11:07
  throughout this year, people are learning more and more, but I
• 11:09
  think there was originally this myth that like the more context
• 11:12
  you give it, the better it's going to be.
• 11:14
  And if you just add as much context as you possibly can,
• 11:18
  you're going to get a better and better answer.
• 11:20
  There's a lot of research and a lot of studies that show that
• 11:23
  even these, you know, these models that have a million, you
• 11:26
  know, token context windows as they get to about 50% full and
• 11:30
  they get into sixty 7080% full, you're sending, you know,
• 11:33
  800,000 tokens to let's say Gemini or sonnet 4-5.
• 11:38
  There's actually a steep drop off in how, how you know how
• 11:42
  often these models hallucinate. So 1 is when you're scoping
• 11:47
  these tasks, make sure the only thing that is going into that
• 11:50
  context window is relevant to the task.
• 11:53
  One one of the most common piece of advice we give with cursor is
• 11:57
  when you start a new task, start a new chat window.
• 12:01
  There, there are a lot of engineers that might think, hey,
• 12:04
  I, I worked on something adjacent to this problem for a
• 12:07
  while in a chat window. And a chat window is essentially
• 12:10
  just a boundary for context. I want to start a new task and
• 12:15
  maybe this context is going to be helpful.
• 12:17
  And you, you end up staying in the same chat.
• 12:20
  You have this bloated context window.
• 12:22
  And it's really just a recipe for the AI to again, get
• 12:25
  confused, make an arbitrary decision based off of something
• 12:29
  you were doing before that is no longer relevant.
• 12:31
  So the the more pristine you can keep your context, actually the
• 12:35
  less, the better. It is as much context as needed
• 12:40
  without any superfluous context involved.
• 12:43
  And that is, that is also a skill you have to learn.
• 12:46
  But you know, just throwing the kitchen sink at a problem gets
• 12:50
  you worse and worse results. Just enough context as needed.
• 12:54
  I like that. The other thing that I like
• 12:56
  about what you're saying is let's not fool ourselves into
• 13:01
  thinking because there was a similar problem that we had, we
• 13:04
  can go down that path and continue with that context
• 13:07
  window or that chat. I wonder also, have you thought
• 13:11
  about or have you seen folks that will try to turn off pieces
• 13:16
  of the context when needed? Or is it just like that?
• 13:19
  Is too much time just start a new one?
• 13:23
  So I think starting, I mean, there's also ways you can
• 13:26
  summarize context. There's also ways that you can,
• 13:29
  you know, create kind of infuse context into a new chat window
• 13:34
  by making markdown files 11 really common way, which
• 13:38
  actually is now built into the flow of cursor is called plan
• 13:40
  mode. You're kind of building out a
• 13:43
  plan for maybe a bigger project or a bigger task and it will
• 13:46
  essentially build a markdown file of this is all the things
• 13:49
  that we need to do. And then you can use that
• 13:52
  artifact in a new chat as the starting point of these.
• 13:56
  These are all the tasks, all the decisions we made.
• 13:59
  Plan mode will also ask you, the engineer, hey, do you want to do
• 14:03
  X or Y? Do you want it'll it'll send you
• 14:06
  these multiple choice questions of different decision trees that
• 14:09
  you could go down. Yeah.
• 14:10
  And then you can use that as as your starting point.
• 14:14
  I think one really common, like easy to conceptualize way that
• 14:18
  context can cannot be your friend is it's also very common.
• 14:22
  Let's say you build out, you know, either you're refactoring
• 14:24
  something or build a new feature, whatever it is, and you
• 14:26
  run into a bug. And AI debugging is one of the
• 14:30
  most frustrating parts of learning how to build with AI
• 14:33
  'cause it can just rabbit hole down.
• 14:35
  The same problem thing that we see is if you try to solve the
• 14:40
  bug in the same context window that you created the feature, it
• 14:43
  has all this context built up that it was that whatever it did
• 14:47
  was the right thing to do. So it can actually make it
• 14:49
  harder for it to figure out what the bug is 'cause it is weighed
• 14:53
  down by all these other decisions that created the bug
• 14:55
  in the first place. Dude, like you're preaching to
• 15:01
  the choir right there. I feel like I have had that.
• 15:03
  Everybody in here who has played around and vibe coded for a
• 15:06
  minute has felt that now there is something.
• 15:09
  It's a little bit more of a nuanced piece, but I was just
• 15:14
  talking to a friend and they were like, yeah, last time I was
• 15:18
  using Cursor, there was a moment where I wanted to do something
• 15:22
  and I knew how I would implement it.
• 15:25
  And I have my experience, but what Cursor suggested was
• 15:31
  different. And so I'm at this crossroads.
• 15:33
  Do I go towards what Cursor suggested or do I go my way of
• 15:37
  doing it and why and what is going to be better?
• 15:40
  How would you recommend that we solve that kind of thing?
• 15:43
  Because it's like, I know that my way might work, but I also
• 15:46
  know that I don't know things like the known unknowns and then
• 15:50
  the unknown unknowns, right? And so now I'm looking at what
• 15:54
  Cursor's suggesting and I'm thinking, is that going to cause
• 15:57
  problems in the future because I totally don't understand what is
• 16:00
  happening here. Or is this the better way of
• 16:02
  doing it? Yeah.
• 16:04
  And I, I think this is kind of the biggest difference between
• 16:06
  like vibe coding on the weekend and building something fun and
• 16:10
  then building sustainable software in the enterprise.
• 16:13
  I would definitely say it's not really an either or, but I would
• 16:18
  not build anything for an enterprise solution that has to
• 16:21
  go out to maybe millions and billions of users that I don't
• 16:24
  understand, right. A huge part of this has to be
• 16:28
  like, I need to understand the solution that I'm putting out.
• 16:31
  And even if the AI wrote it, like the engineer is responsible
• 16:34
  for the PR that they shipped to production.
• 16:37
  You know, we also see at a lot of a lot of companies, you know,
• 16:40
  a bug will surface in production and someone's like, oh, what
• 16:43
  happened here? And you know, it's a very
• 16:46
  obvious feeling, but I don't know the AI wrote it.
• 16:49
  That's. Cloudflare today.
• 16:52
  I mean, yeah, we have the half the Internet down right now.
• 16:55
  I swear man, these AI just did that.
• 16:58
  But sorry, I didn't mean to cut you off, no.
• 17:01
  Don't worry, you know, the, the kind of best culture I've seen
• 17:05
  built around AI is that you can't blame the AI if a bug
• 17:09
  comes out. You have to be responsible,
• 17:11
  understanding what the code was and what was written.
• 17:15
  But that doesn't mean that you always have to choose your
• 17:17
  solution. It just means if there's a
• 17:19
  solution path that looks promising, but you don't fully
• 17:22
  understand why it chose that. I mean, one of the beauties of
• 17:24
  AI is you can say, hey, I would have done it this way.
• 17:27
  What are the pros and cons of my way versus your way?
• 17:30
  Right. You use AI to make yourself more
• 17:34
  informed about the choices that that are happening.
• 17:37
  You know, it, it is really enticing to to the, you know,
• 17:41
  get get into that flow of moving so fast and just building
• 17:43
  constantly and shipping code and writing lines of code.
• 17:46
  But you kind of, you have to, especially in the enterprise
• 17:49
  world, slow yourself down at key decision points, Inform
• 17:54
  yourself, learn, dive in, ask questions, understand, and then
• 17:58
  make the best decision path forward.
• 18:00
  Because especially if you branch off in 2 completely different
• 18:03
  areas, if you've chose the one that you understand less, it's
• 18:08
  just gonna make the next 20 steps harder.
• 18:10
  Totally, that's so true. So maybe you have a special way
• 18:16
  of dealing with this next problem that I've encountered
• 18:19
  where I have my rules, but some rules, you know, it's like
• 18:23
  playing the guitar and you have your scales, but scales are like
• 18:27
  a recommendation sometimes or scales can really make the sound
• 18:33
  not sound good. So I have like hard rules that I
• 18:36
  want the agent to follow and I have like recommendations and
• 18:41
  how do I distinguish between those and make it so that this
• 18:45
  rule like if for example, it's don't output a function longer
• 18:50
  than 21 lines of code and that's like a recommendation.
• 18:54
  If it really needs to be, it can be 23 but like keep it at 21
• 18:58
  ish. I can't say that in the rules
• 19:00
  like 21 ish. Yeah, it's a great question.
• 19:05
  I mean, I think this is one of the probably biggest learning
• 19:08
  curves of working with AI is just fundamentally you're
• 19:11
  working with a non deterministic system.
• 19:15
  And even if you write a rule, I mean, this is, you know, a huge
• 19:17
  frustration for a lot of engineers working across any AI
• 19:20
  tool. You could write a rule that is
• 19:22
  like this is the most important rule.
• 19:24
  Never, you know, start a new server, a new dev server, right.
• 19:28
  I think that's the most common one I see for front end
• 19:30
  developers is like never MPM run dev because I already have the
• 19:34
  the the. The host running I know where
• 19:35
  this is going do it. You know, one out of 10 times.
• 19:39
  I mean, this is actually a thing that a lot of AI researchers are
• 19:42
  currently working on. Is like, how do you build a
• 19:45
  model that is thoughtful and creative, but also closely
• 19:50
  follows instructions and is easily steerable and guideable?
• 19:54
  The honest answer is it's not a solved problem yet.
• 19:58
  There's no way to know deterministically.
• 20:01
  A lot of tools cursor has built this.
• 20:03
  There's this idea of hooks in Cursor where you can actually
• 20:07
  put deterministic hooks into the agent flow.
• 20:11
  So that at given moments, either right before your prompt sends,
• 20:14
  right after your prompt gets back, before the code's
• 20:16
  generated, after the code's generated, you can actually do
• 20:19
  deterministic things. This is not as creative as AI,
• 20:22
  but if you have to, you know, you know, let's say always check
• 20:26
  after code is written for certain things.
• 20:28
  You know, some, some companies, they'll check for copyleft data
• 20:33
  in in the code that is outputted.
• 20:35
  That is something you want to do in a deterministic hook rather
• 20:37
  than a rule. But if you have just a strong
• 20:40
  rule versus a soft rule, really the best way to do it today is
• 20:46
  it's like you're dealing with a, like a, a junior engineer,
• 20:50
  right? You can say in plain English
• 20:52
  like this is a rule to always follow like the most highest
• 20:55
  priority and it will follow it generally more than say like,
• 21:00
  Hey, I want it to stay generally under 21 lines of code.
• 21:04
  If it goes over, it's not a big deal.
• 21:06
  But like, please, whenever you can, try to prioritize under 21
• 21:09
  lines of code outputted that actually will work.
• 21:12
  It will. It will like loosely follow one
• 21:15
  and try to strongly follow the other.
• 21:16
  But again, models have different personalities.
• 21:19
  Some are hard to steer, some are easier to steer.
• 21:22
  Your Your mileage may vary. I often think about it as like,
• 21:27
  wow, wouldn't it be nice if we just had, you know, a volume
• 21:30
  knob that we could crank up or crank down?
• 21:32
  Because exactly what you're saying is so fuzzy in the
• 21:36
  language piece where it's like, yeah, this one hard rule, this
• 21:41
  never break this rule. And then on the other side, this
• 21:44
  is kind of a rule and you have to put that in your language and
• 21:47
  it can be interpreted in different ways every time at
• 21:51
  like every single turn, right? So. 100%.
• 21:54
  It's like, wow, sometimes I ponder in the shower.
• 21:59
  Couldn't we have just some knobs that will give me more
• 22:03
  steerability or more control in that regard?
• 22:05
  But also I understand that that kind of like defeats the purpose
• 22:08
  of this is so cool. We get to just talk to it and
• 22:11
  then it does it. And if you now are breaking out
• 22:13
  knobs for this, where does it end?
• 22:15
  So I don't you. Know, and this is like the
• 22:18
  original idea of like temperature.
• 22:20
  Yeah, exactly. When you're you're you're like
• 22:22
  talking to an API, it's like you can set that really high
• 22:25
  temperature and it's going to come up with novel and crazy
• 22:28
  ideas and it's going to be like this really creative partner to
• 22:30
  work with. Or you can set a really, really
• 22:32
  low temperature and it's like fairly deterministic to the
• 22:36
  point answers. But there is that Goldilocks
• 22:39
  zone of like, what? What's the point of working with
• 22:41
  an AI if it's like very, it's only going to tell me one
• 22:44
  solution every single time, even if it's a bad solution versus
• 22:47
  try to be creative, try to think of new things.
• 22:49
  Yeah. So I think this might be the
• 22:52
  last question for you, and it's one that I've been grappling
• 22:55
  with for a while. How much harness is too much
• 22:58
  harness? And how do you think about that
• 23:01
  when you're letting the agents do what they want?
• 23:06
  Yeah, when you say harness, 'cause like when when we say
• 23:10
  harness at Cursor, we talk about our agent harness and like what
• 23:13
  we do that you don't have to do in your prompt, that that's how
• 23:16
  we think of harness at Cursor. But I just want to make sure
• 23:18
  we're we're on the same page. We can go down that road.
• 23:20
  I was also thinking about, as I'm building my agent, how do I
• 23:26
  make sure that I am giving it enough freedom on that
• 23:30
  temperature gauge, but also not saying this is where you go,
• 23:35
  this is how you do it. And the opposite side of that,
• 23:39
  making sure that if something is very well known, it does it
• 23:43
  every time. Yeah.
• 23:45
  No, it's a great question. So I mean, I think we have a lot
• 23:48
  of experience building our own harness, which can can be very
• 23:50
  helpful for customers building their own agents.
• 23:53
  You know, in in the enterprise space that there is, you know,
• 23:58
  kind of as you're working. It's always been true that like
• 24:01
  data is very important. You should measure the outcome
• 24:03
  of these things as in production.
• 24:05
  You know what? What leads to better outcomes
• 24:07
  versus worse. It's almost more true in the AI
• 24:10
  space. Working with these
• 24:11
  nondeterministic LMS is when we build our own harness, right?
• 24:16
  Like you hear these like bench scores with LMS of how good are
• 24:22
  they at solving certain problems?
• 24:24
  Bench scores inherently aren't, aren't, are imperfect because
• 24:27
  you can gain bench scores. You can, you know, build things
• 24:29
  specifically for those things. But one thing that we noticed is
• 24:33
  like, you know, depending on what that agent is supposed to
• 24:37
  do, is it just supposed to output, you know, is it a
• 24:39
  chatbot is just supposed to like talk with you, in which case you
• 24:42
  can be a little bit loose with the harness.
• 24:44
  You can like give it some, some guidelines.
• 24:46
  Obviously, you know, most of the LMS you're going to be using
• 24:48
  under the hood have a lot of the safety guidelines already built
• 24:51
  in. So you don't have to give it too
• 24:53
  much direction in the harness, but like for something like
• 24:55
  cursor where it can do almost anything within a code base, we
• 24:59
  have to build a very extensive harness of like, how does it
• 25:02
  work with tools? You know, there's an edit tool,
• 25:05
  there's a search tool, there's a web tool, there's a like
• 25:08
  planning tool. And there you need like a very
• 25:11
  prescriptive harness of how do you work with these things?
• 25:14
  Especially like if your agent is interacting with a lot of MCPS,
• 25:17
  right? Those become MCP tools that you
• 25:19
  have to work with. And in that case, you do really
• 25:23
  want to kind of fiddle with that harness and test and see what,
• 25:26
  what gives you the best outcomes.
• 25:29
  And you know, like for us, our GPT 5 harness, like 2 months
• 25:33
  ago, we actually figured out we had a bug in it.
• 25:35
  It wasn't actually the best possible harness for GPT 5.
• 25:39
  And when we, the bench score went from like 45% to 65%.
• 25:44
  So it can make a huge difference to kind of optimize this
• 25:48
  harness, especially in this world where AI can output, it
• 25:55
  can hallucinate, and it can give you great solutions.
• 25:57
  Yeah, AB test that harness. I like it.
• 26:00
  Dude. Randy, thank you so much for
• 26:03
  coming on here. I know that this was something
• 26:06
  that you probably did not realize you were getting
• 26:09
  yourself into when you signed up for it.
• 26:12
  You've been really a great sport and I want to just like thank
• 26:17
  you from the bottom of my heart for jumping on here and sharing
• 26:19
  this with everybody on the live stream.
• 26:22
  Yeah, this was great. I I've been on a lot of fireside
• 26:25
  chats. I've never seen someone dressed
• 26:26
  this well for a fireside chat. This was awesome.
• 26:29
  Thank you for having. Me Next time, feel free to come
• 26:31
  in your best Halloween costume. I I, I didn't realize I should
• 26:36
  have dressed up. Yeah.
• 26:37
  You look great man.