Latest / Founder Mode / From AI Prototype to Production with Ankur Goyal

Transcript

From AI Prototype to Production with Ankur Goyal · Founder Mode

• 0:00
  The most important thing that I like to ask people is if I use
• 0:04
  your app and I complain, how much effort is it for you to
• 0:08
  take that complaint and turn it into an eval?
• 0:10
  And the best teams, it's like one or two clicks.
• 0:13
  If that the teams that are NGMI, they haven't even thought about
• 0:16
  the answer to that, right? And so, so I think that's really
• 0:19
  the goal. Like can you get to a point
• 0:20
  where it's two clicks from someone complaining to something
• 0:24
  ending up in your eval's? You're listening to Founder
• 0:41
  mode. The podcast where builders share
• 0:44
  what's really working instead of just pitching the dream.
• 0:47
  Today we're talking about something that sounds easy but
• 0:51
  turns out to be brutally hard, and that's going from your AI
• 0:55
  demo to production. Yeah, our guest today has spent
• 0:58
  the last decade building the infrastructure that makes
• 1:01
  scaling actually possible, and now he's applying it to help
• 1:04
  companies make their AI reliable.
• 1:08
  Yeah, I mean, we've all seen the hype, the the amazing 32nd demo
• 1:12
  that really hits, but then six months later it's still not
• 1:15
  live. That's why this episode really
• 1:17
  matters. It's about the systems feedback
• 1:19
  loops and boring details that make AI work for real teams.
• 1:22
  Kevin, we're seeing a lot of failure in AI implementation.
• 1:26
  The the MIT report that just came out, 95% of enterprise AI
• 1:31
  adoption has driven 0 ROI. What's your take?
• 1:34
  Yeah. I mean, you know, everybody's
• 1:36
  striving to now be in that 5% because they've all heard that
• 1:39
  the 95% and they're all looking themselves in the mirror and
• 1:42
  being like, which side am I on? And knowing that they're not on
• 1:45
  the good side more than likely. And so I think we've been
• 1:48
  fortunate we found a couple of the five percenters in our work.
• 1:52
  And I think what, you know, stood out to me when I read that
• 1:55
  is like, what were the things that sort of made you show up in
• 1:58
  that case? And typically it's a lot deeper
• 2:00
  work, a lot deeper integration, a lot more hands on, sort of
• 2:04
  like understanding the use case and then really thinking about
• 2:07
  it as a business and not saying, oh, I've got this cool UI widget
• 2:11
  or AI widget that I can apply to like make something see or
• 2:14
  something happen. And so for us, it's really been
• 2:16
  about finding something that's like materially hard and then
• 2:20
  spending a lot of time on it to deeply understand it and how the
• 2:23
  business sees it. Because today it's so easy to,
• 2:27
  you know, jump into ChatGPT and grab a doc, a piece of paper or
• 2:30
  just some data and ask it a question and it will give you an
• 2:32
  answer. Yeah.
• 2:33
  But then seeing that show up day in, day out, that's where it
• 2:36
  really the hard stuff begins. So, yeah.
• 2:38
  So I think the the real trick there is just figuring out like
• 2:41
  where the business value is and like being sure that you can
• 2:44
  build the thing that matters to sort of like actually achieve
• 2:48
  and land in that 5%, which is where you want to be.
• 2:50
  Yeah. I think it makes a ton of sense.
• 2:52
  And it's not about just participating in the AI hype
• 2:54
  cycle, right? It's doing things that drive
• 2:56
  business value. Totally.
• 2:58
  Anker's built across like multiple generations of
• 3:00
  infrastructure, I think from these structured data to
• 3:02
  unstructured search to AI agents and you know, done it quietly
• 3:05
  but at significant scale. Yeah, he's also one of the
• 3:08
  clearest thinkers, yeah, in the space.
• 3:10
  And he has no interest in selling you hype.
• 3:12
  So should we, should we bring Anker on?
• 3:13
  Let's do it. Anchor, welcome to founder mode.
• 3:26
  You've gone from databases to ML to AI, and now you're building
• 3:29
  honestly what's synonymous with eval with brain trust.
• 3:32
  Great to have you here. Excited to be here, thanks for
• 3:34
  having me. Yeah, Anchor, we're, you know,
• 3:37
  we, we do a little prep here. We kind of look at all of our
• 3:39
  guests backgrounds and the things that they've built and
• 3:41
  done. Super impressive.
• 3:42
  Your, your, your experience and your history.
• 3:44
  You've built across so many different layers of the stack,
• 3:47
  right, Like search databases, AI and now AI reliability.
• 3:51
  What's the common thread that kind of brought you here?
• 3:55
  Honestly it's databases. No matter what, I, I feel like I
• 3:58
  can't escape them or to me, everything looks like a database
• 4:02
  problem. We even like at Braintrust, we
• 4:04
  already built our own database called Brainstore.
• 4:08
  But you know, I, I think I've, I've always, I've always been
• 4:11
  fascinated by this idea that if you help someone kind of create
• 4:14
  a really good system that lets them capture data, work with it
• 4:19
  and then iterate with it, then they can build really good stuff
• 4:23
  in, you know, back in the day when I was working at MEMS
• 4:26
  sequel, it's now called single store, we helped a bunch of
• 4:29
  enterprise companies that were generating, you know, tons of
• 4:32
  data, like financial companies who if they could make trading
• 4:35
  decisions in 30 seconds instead of like 30 hours, they would be
• 4:39
  able to make a lot more money. Or, you know, industrial
• 4:42
  companies that were trying to get data from sensors and make,
• 4:45
  you know, better predictions about when things would fail.
• 4:49
  And it's kind of the same thing, I think in AI, the more
• 4:51
  effectively and the more quickly you can use data to improve the
• 4:55
  quality of your product, for example, like the much better of
• 4:57
  a product you build. Was there like a particular like
• 5:00
  pain from your past work where you're like, wow, it's it's
• 5:03
  evals is the point that's going to be the one for AI that's
• 5:05
  going to fix it? And they kind of triggered this.
• 5:07
  You know what? What is now?
• 5:08
  Brain trust. Yeah, you know, I I almost
• 5:10
  didn't want to start working on brain trust because I was so
• 5:13
  fatigued from this problem. So my last start up in PIRA, we
• 5:18
  built AI based document extraction and it was really,
• 5:22
  really hard pre ChatGPT. Now it's a much easier problem.
• 5:26
  But you know, back then it was a really hard technical problem
• 5:29
  and we supported a bunch of different document types for
• 5:31
  example like invoices, bank statements, insurance documents,
• 5:35
  etcetera. And we realized after Bert came
• 5:39
  out that we could train one transformer model to do really
• 5:43
  well at all of these things rather than train like different
• 5:45
  models for for each of them. But what we found is that every
• 5:49
  time we improved 1 document type, we ran the risk of
• 5:52
  regressing another one. So we might like improve the
• 5:54
  invoice performance and then break bank statements.
• 5:57
  Now you probably see where this is going.
• 5:59
  Our financial customers were not particularly excited about that.
• 6:03
  And so we had to get really good at doing evals.
• 6:06
  And as soon as we built evals, we went, I think we used to
• 6:11
  argue about model decisions for like 3 or 4 months at a time.
• 6:14
  We went to shipping a lot faster, and I saw that really
• 6:18
  transform how we built stuff. But then it became really hard
• 6:21
  to get data to do evals. And that's when we started
• 6:23
  figuring out, OK, how do we link together our observability stack
• 6:26
  to getting data to play with offline?
• 6:29
  That felt very weird at the time.
• 6:30
  And we had to write a bunch of scripts and build a bunch of
• 6:33
  stuff, but I didn't think too much of it.
• 6:35
  And then Figma acquired us, and I led the AI team there.
• 6:38
  And, you know, that was like right after ChatGPT came out.
• 6:41
  So we dropped everything that we were doing and we started
• 6:45
  thinking about, OK, how do we use LMS to make Figma's product
• 6:48
  better? And you know, funny enough, we
• 6:50
  ran into exactly the same problems.
• 6:52
  Like we kept building stuff and then another team would start
• 6:55
  playing with it and break it. And, you know, we, we ended up
• 6:58
  basically building the same tooling again.
• 7:01
  And after doing that twice, I was like really sick of this
• 7:03
  problem. So much so that I didn't really
• 7:05
  want to talk about. It and then you started a
• 7:06
  company to solve the problem. Yeah, And you know, this
• 7:10
  brilliant guy, Ilad was like, hey, you know, you, you kind of
• 7:14
  like built the same tools at both of these companies.
• 7:16
  But now it's not just you and a corner are working on AI stuff.
• 7:19
  There's a lot of companies like Instacart who are building AI
• 7:23
  products. Maybe other people have this
• 7:25
  problem. And I was like, oh, man, I'm so
• 7:26
  sick of, you know, working on this internal tool.
• 7:30
  But we talked to a bunch of companies including, you know,
• 7:33
  Instacart, Notion, Sapier, Airtable, Stripe, a bunch of our
• 7:37
  early customers, and they were like, yeah, you know, we're just
• 7:40
  starting to to figure this out. And that's how brain trust kind
• 7:44
  of came about. That's amazing.
• 7:45
  So you said that going from prototype to production in AI is
• 7:49
  is kind of an uphill battle. What do most teams
• 7:52
  underestimate? I think, you know, once you ship
• 7:57
  your product, you start to get feedback from users that you
• 8:00
  didn't expect. Honestly, that's true with any
• 8:02
  product. But in AI, what happens is, you
• 8:05
  know, someone uses your product. Let's say you're building, we'll
• 8:07
  keep using the Instacart example.
• 8:09
  Like let's say you're building a tool to help people construct
• 8:12
  their their grocery cart for the week.
• 8:16
  What happens is people will find things that don't work, they'll
• 8:19
  complain to you and then you'll realize like, oh wow, you know,
• 8:22
  the user is right. I need to go and edit my prompts
• 8:24
  or I need to go and like change, you know, the model I'm using or
• 8:27
  maybe try decomposing the problem what whatever it is.
• 8:31
  And so you try doing that and you play with whatever the user
• 8:34
  was trying to do and it improves that user scenario.
• 8:38
  And then you ship it and then another user complaints and
• 8:42
  says, hey, yesterday I was able to create, you know, my shopping
• 8:45
  cart, but now it's not adding bananas to it.
• 8:47
  You're like, Oh my God, what you know, what did I do?
• 8:49
  And basically, you know, when you're prototyping, you tend to
• 8:52
  have this razor focus on just a few examples or maybe one thing
• 8:57
  at a time that you're really looking at.
• 8:58
  And that's a really great way to make progress and iterate on an
• 9:02
  AI product. But when you actually ship
• 9:04
  something and you have a lot of users, suddenly you need to take
• 9:07
  into account kind of the collective wisdom that you've
• 9:09
  gathered over time. And I think that's that's where
• 9:12
  it gets really, really challenging.
• 9:14
  I think it can feel a lot like, you know, literally like playing
• 9:16
  whack A mole. Like you improve 1 user's thing,
• 9:18
  another thing breaks, you improve another thing, etcetera.
• 9:22
  And I think that's where teams start to to see the benefit of
• 9:25
  evals. You know, early on we had this
• 9:28
  heuristic that if someone hadn't shipped their product at least
• 9:33
  three months ago, it wasn't worth talking to them about
• 9:35
  brain trust. And we would often meet teams
• 9:39
  and they were like, hey, we don't need evals.
• 9:40
  Our product is great. My mom uses it.
• 9:42
  She really likes it. The models are great.
• 9:44
  You know, they're getting better.
• 9:45
  I don't need this. And then kind of like 3 months
• 9:48
  later, they'd come back to us a little bit hungover.
• 9:51
  Hey, are you guys still there? And I think that timeline has
• 9:57
  accelerated. We'll now understand that evals
• 10:00
  are quite useful, but the general, the general thing is,
• 10:05
  you know, is still there. The phenomenon.
• 10:07
  How do you help teams kind of move faster without losing
• 10:10
  trust? Like what's kind of the key
• 10:11
  thing, Like if you were going to like, you know, pick the one or
• 10:13
  two things that a team should do to say, you know, I want to move
• 10:15
  faster, like, you know, increase my velocity without like losing
• 10:18
  trust. And sort of like you said, what
• 10:20
  you built yesterday is not going to break tomorrow, kind of.
• 10:22
  Thing, yeah, I think there's really two things that you
• 10:24
  should get good at. The 1st is you should have a
• 10:26
  really good environment for doing iteration.
• 10:30
  We have something called the playground in our product that
• 10:32
  it's really simple. You can test multiple models and
• 10:35
  prompts or agents. You know more complex things if
• 10:38
  you want side by side with a bunch of data points.
• 10:41
  And you don't need to start by computing fancy numbers and
• 10:44
  scores like just running your prompt on a bunch of examples
• 10:48
  and looking at the output with your eyeballs is a pretty good
• 10:51
  way to start. And when you're, if you're like
• 10:53
  sitting in your terminal and playing around with something,
• 10:55
  you're only working on one example at a time.
• 10:58
  So I think just getting some basic tool in place that helps
• 11:02
  you do that. And then of course the 10
• 11:05
  examples that you start with will suddenly turn into like
• 11:07
  10,000. And then it's really useful to
• 11:09
  actually have some numbers to help you sift through
• 11:11
  everything. But having some kind of like
• 11:14
  iteration mechanism, I think that's really important.
• 11:16
  And then the other thing is having a really strong
• 11:18
  connection between what happens in production and what you're
• 11:22
  able to test. And I think This is why we've
• 11:25
  had this really sort of end to end experience from logging and
• 11:30
  tracing all the way back to evals in Braintrust from
• 11:33
  basically the beginning. It's funny, like one of our
• 11:35
  first customers was Airtable and we didn't have logging when we
• 11:39
  started. And there was an engineer at
• 11:41
  Airtable who basically created an eval.
• 11:44
  And then you see where this is going.
• 11:45
  You just kept adding stuff to it.
• 11:47
  And soon it became, you know, the biggest eval we had ever
• 11:49
  seen. And we were like, hey, what's
• 11:51
  going on? He's like, yeah, these are my
• 11:52
  logs from production, and I'm just using, I'm capturing them
• 11:55
  in braintrust so that they're in the same format as my evals.
• 11:58
  And that was a huge, you know, it's like, OK, yeah, that
• 12:01
  actually makes a lot of sense. Baby logging.
• 12:03
  Yeah. And so we, we, you know, we, we
• 12:06
  turned what he was doing essentially into, into logging.
• 12:09
  But having that that feedback loop I think is is just
• 12:12
  incredibly important. So you've been public about
• 12:14
  comparing models like opening eyes and and Anthropics.
• 12:18
  What is good model selection actually look like today?
• 12:21
  I think you know there's two analogies for it.
• 12:25
  For what models might be 1 is they might be like CPUs.
• 12:30
  And by that I mean like you can take the same set of
• 12:32
  instructions and pipe them through ACPU and you know, you
• 12:35
  ought to get exactly the same result, but different CPUs have
• 12:40
  different performance profiles. Like if you run, you know, Intel
• 12:45
  CPU from three years ago, it might weirdly be better at
• 12:48
  certain things, but likely not other things, etcetera.
• 12:51
  And then I think another analogy is databases.
• 12:54
  Again, I think a lot about databases, but every database
• 12:57
  speaks Sequel, but they speak slightly different dialects of
• 12:59
  Sequel. And some databases are
• 13:01
  dramatically better at certain use cases than others.
• 13:04
  And I actually think LLMS are a little bit more like databases
• 13:07
  than they are like CP US. Like the same English string
• 13:10
  sent to one LLM might produce like really different things in
• 13:13
  another. And kind of like databases, I
• 13:16
  think you need to do a bit of, you know, benchmarking and
• 13:21
  diligence when you decide which model or family of models is
• 13:26
  your best bet to solve a particular use case for some
• 13:28
  period of time. But then you need to focus
• 13:30
  almost all of your effort into making that model work really
• 13:33
  well. And I think the best teams that
• 13:36
  we work with, basically, if you were building a SAS product,
• 13:40
  let's say like 10 years ago, you might re evaluate your
• 13:43
  relational database every three years.
• 13:45
  Or if you're very, you know, advanced every year or something
• 13:48
  like that. If you think about AI,
• 13:50
  everything is compressed. So the best teams that we work
• 13:53
  with re evaluate their model choices maybe once every one or
• 13:57
  two months. And they re run a bunch of
• 13:59
  benchmarks and sort of understand, OK, well, it seems
• 14:02
  like now in GPT 5, GPT 5 Nano is like weirdly fast.
• 14:06
  And so I can suddenly unlock this user experience.
• 14:09
  I didn't think I could have done before.
• 14:11
  Let me go like play with that for another month or two.
• 14:14
  We do see teams that never re evaluate their model choices and
• 14:18
  you know that I think they often lose product market fit.
• 14:23
  And then we also see teams that spend all their time comparing
• 14:27
  models side by side and they never ship anything.
• 14:29
  So we kind of think this in between point is is what works
• 14:32
  best. How about like actually eval
• 14:34
  fatigue? Like, you know, once you get to
• 14:36
  scale, you're going to get, you know, obviously lots of evals
• 14:38
  and lots of data coming back. Like what's sort of the best
• 14:40
  practice to sort of understand what to look at, how deep to go
• 14:43
  and to make sure that you're sort of balancing this sort of
• 14:45
  like tuning versus like shipping and moving forward?
• 14:48
  Yeah, I think the short answer to your question is RM-RF.
• 14:52
  Don't be afraid to delete things.
• 14:54
  The long answer to your question is, I think my framework for
• 14:58
  evals is not to think of them as benchmarks, it's to think of
• 15:02
  them as time savers. So as humans, I think when we do
• 15:07
  evals, our job is to influence what an LLM is doing to better
• 15:12
  match the expectations that we have for what a user wants.
• 15:15
  That's it, right? It, that's our, that's kind of
• 15:17
  our, our, our, our role in, in, in this weird world that we live
• 15:21
  in. And let's say that we have like
• 15:24
  15 minutes a day to spend working on that.
• 15:27
  If you have, let's say you have one user and that user uses your
• 15:31
  product five times a day, so you generate five data points, then
• 15:35
  your eval might be you spending 15 minutes looking at what that
• 15:39
  user did five times, remembering what they did yesterday, and
• 15:43
  then maybe tweaking something so that you have a better shot at
• 15:45
  improving the five times they use the product tomorrow.
• 15:48
  Now let's say you have, you know, 500 users and they each
• 15:52
  use the product 100 times a day. You know, now you have 50,000
• 15:55
  data points and you just can't look at all 50,000 interactions.
• 15:59
  And I think the right way to think about evals is OK, I have
• 16:02
  these 50,000 examples. What are the five to 10 that are
• 16:07
  actually worth my time looking at in more detail?
• 16:10
  And I think if you think of evals as kind of like a
• 16:13
  prioritization function, then it, the answer to your question
• 16:16
  becomes kind of obvious. Like if you have too many evals,
• 16:19
  that means that you're just generating too many things which
• 16:21
  seem like a priority. And so maybe you more
• 16:24
  aggressively prioritize within that or maybe you remove things.
• 16:27
  I think the mistake is when people start to collect like
• 16:31
  50,000 evals and then they generate a bar chart instead of
• 16:34
  actually looking at the data. And they say, OK, now I'm, I no
• 16:37
  longer have time, you know, to look at the, to look at the
• 16:40
  example. So I'm just going to look at the
• 16:41
  bar chart. And I think, like with anything,
• 16:44
  right, with any example of anyone using data, that is not
• 16:47
  the right thing to do, right? You you posted this great line,
• 16:51
  I'll make sure I get it right. All agents converge on while
• 16:55
  loops with tools. What actually makes that design
• 16:58
  work? I think it is inherently what
• 17:02
  the LLM, what the LLMS do, right?
• 17:05
  Like if you think about stripping away all software
• 17:10
  abstractions and you know, really just thinking, imagining
• 17:14
  the most simple architecture that if it worked would be
• 17:18
  extremely simple. And if it doesn't work, it's
• 17:20
  only because the LLM is not yet smart enough.
• 17:23
  That would be a while loop of tools.
• 17:25
  And I think because LLMS are improving so quickly and because
• 17:29
  the model, the people who are improving, the LLMS are sort of
• 17:34
  adopting and building with this mindset, you can look at how
• 17:36
  cloud code is built. You can look at the Open AI
• 17:39
  Agents SDK, you can look at how ChatGPT works when you actually
• 17:42
  interact with it. It's all the same, you know,
• 17:45
  while loop with tools idea. That means that this
• 17:48
  architecture is the only one that really withstands the
• 17:50
  iterations that models make. Now of course the reality is a
• 17:54
  bit more complicated, like sometimes you need to think
• 17:56
  about state, sometimes you need to have one LLM delegate a task
• 18:00
  to another simpler LLM. But I think that arriving on
• 18:03
  these more complex designs with a purpose like hey, I used a
• 18:08
  while loop of tools and I got stuck because I need to delegate
• 18:12
  this task to another LLM for cost reasons.
• 18:15
  I think if you arrive on that from first principles by
• 18:17
  actually experiencing the problem, then it justifies
• 18:21
  investing in a more complex design.
• 18:23
  And then I think, you know, if you arrive on it with that, like
• 18:29
  from first principles like that, if the model becomes better or
• 18:32
  cheaper, then you'll also have no ego or no attachment to that,
• 18:37
  right? So if suddenly the, the, the
• 18:39
  assumption, which is that the model is too expensive to do a
• 18:41
  while loop of tools goes away, then you'll just revert back to
• 18:44
  the simpler design. So that's, that's just what we
• 18:46
  see. I think some of the best teams
• 18:48
  do. It was interesting, we were
• 18:50
  having lunch a couple weeks ago and we were talking about how we
• 18:52
  were solving one of our problems and kind of went back to a more
• 18:54
  deterministic solution. And you were like, I, I, I think
• 18:57
  you guys are going to flip on that a bunch.
• 18:59
  And I think this reminds me of that.
• 19:00
  And like, how have you sort of seen that?
• 19:03
  And like, you know, when I'm sure there's stuff in your
• 19:05
  guys's world where you're looking at that and you sort of
• 19:07
  say, oh, this time, you know, the LMS not smart enough.
• 19:09
  I need to kind of like coach it or make a more rules based kind
• 19:12
  of code. And then you end up saying, oh,
• 19:13
  now they got smarter. How have you, you know, do
• 19:16
  advise teams just constantly test?
• 19:17
  Is it like you have an eval framework work that sits on top
• 19:20
  of that or what would you sort of approach that problem in a
• 19:22
  more, like I said, first principles way?
• 19:24
  Yeah, I think there's really two things you can do.
• 19:27
  So the most important thing you can do is you should capture the
• 19:31
  thing that's not working as an eval.
• 19:33
  And the reason that's really important is anytime a new model
• 19:36
  comes out or anytime you have some kind of idea about how to
• 19:40
  improve things, you want to be able to re evaluate the
• 19:43
  assumption as efficiently as possible.
• 19:45
  So let's say like you're trying to build something in the
• 19:49
  healthcare space and you want to have something, you want to have
• 19:52
  a model look at patient's notes and you found that if the
• 19:57
  patient's notes are more than 3 pages long, the model gets
• 19:59
  totally stuck. I'm just making this up.
• 20:02
  And, and so you're like, OK, what do I do about this?
• 20:04
  Well, the first thing you should do is capture several examples
• 20:07
  of the model getting stuck so that let's say GBT 6 comes out
• 20:11
  tomorrow. You can click a button and then
• 20:13
  find out if it's still stuck. So make sure you do that now.
• 20:18
  To actually work around it, I think there's like really two
• 20:20
  things you can do. The first is you can hack and
• 20:23
  hacking works. And the second thing, which I've
• 20:26
  seen a lot of people start to do, and I think it's actually a
• 20:30
  pretty powerful method, is just to reshape the user experience
• 20:35
  around what the models can do today, with an eye towards
• 20:38
  enabling more as the models get better.
• 20:40
  If you think about some of the most popular AI applications,
• 20:43
  they are simply allowing the user to do things that from the
• 20:49
  outside, if you look at something like Cursor, you're
• 20:51
  like, Oh my God, they got so lucky because the models are
• 20:54
  really good at coding. Well, no one actually knew that
• 20:57
  until Cursor really became popular.
• 20:59
  And I think the Cursor team is pretty smart, right?
• 21:02
  They iterated like crazy and figured out like, oh, these
• 21:06
  things actually work really well.
• 21:07
  Let's actually figure out how to craft a user experience which
• 21:10
  highlights these things. So I think that mentality is is
• 21:13
  also really powerful. What is good observability look
• 21:17
  like in AI apps and and where are the where are most teams
• 21:21
  kind of blind right now? Yeah, I think observability is a
• 21:24
  bit of an overloaded term. And the reason is I I think it's
• 21:28
  really important to think about the business outcome that you're
• 21:30
  trying to drive by investing in observability.
• 21:32
  It's also expensive. And so it's important to think
• 21:35
  about like what am I actually trying to solve?
• 21:37
  So in the traditional software world, I think a really good
• 21:42
  product is Datadog. And the problem that you're
• 21:44
  trying to solve by investing in observability with Datadog is
• 21:47
  uptime, or, you know, uptime in performance.
• 21:50
  So, you know, we use Datadog and let's say someone is having, you
• 21:55
  know, the brain trust UI is slow.
• 21:57
  We rely on Datadog to go and figure that out.
• 22:00
  And I think, OK, you know, up time is really important to us.
• 22:04
  Therefore we invest in observability and pay Datadog a
• 22:06
  ton of money and you know it, it helps us achieve that goal.
• 22:11
  And AII think the purpose of observability is fundamentally
• 22:14
  different. It's quality.
• 22:16
  So an investment in observability is an investment
• 22:19
  in the quality of your application itself.
• 22:23
  And if you work backwards from that, you know, you ask, OK, how
• 22:27
  does observability contribute to quality?
• 22:29
  I think the most important thing is that it lets you build a
• 22:31
  feedback loop between what users are experiencing and what your
• 22:35
  developers and subject matter experts, product managers are
• 22:38
  looking at regularly. So the most important thing that
• 22:41
  I like to ask people is if I use your app and I complain, how
• 22:46
  much effort is it for you to take that complaint and turn it
• 22:49
  into an eval? And the best teams, it's like
• 22:53
  one or two clicks. If that the teams that are NGMI,
• 22:59
  you know, they don't they, they, they haven't even thought about
• 23:02
  the answer to that, right? And so, so I think that's really
• 23:04
  the goal. Like can you get to a point
• 23:06
  where it's two clicks from someone complaining to something
• 23:10
  ending up in your evals? We need to start a not going to
• 23:13
  make it like shame name and shame segment.
• 23:18
  How I mean to go into that one a little deeper.
• 23:20
  Interesting. Like so when you say like one or
• 23:22
  two clicks and that's like, hey, they report a bug because they
• 23:24
  shake the app or they put, you know, they click something on
• 23:26
  the UI to report a bug. And then behind the scenes
• 23:28
  they're sort of capturing that. Or they're like, are there teams
• 23:31
  that are actually taking user feedback and just saying every
• 23:33
  bug sort of captures the input or the sort of change and then
• 23:36
  just sort of makes a new eval for that?
• 23:38
  Yeah, I think there's there's some people who go overboard.
• 23:43
  So you asked this really good question about like what happens
• 23:44
  when I have too many evals and how do I interpret that?
• 23:47
  I think that one of the lessons I learned pretty early in
• 23:52
  traditional ML is that while you can be very liberal with the
• 23:58
  data that you use for training, you have to be very careful
• 24:01
  about the data you use for validation.
• 24:03
  And I think that evals are very much an exercise in applying
• 24:07
  human judgement and taste to an LLM, you know, and what it what
• 24:12
  it generates. And so I think at some point it
• 24:14
  is quite valuable to have a human curate the set of things
• 24:18
  that end up in your eval's. Now I think scale like dictates
• 24:24
  everything. If you have one user and your
• 24:27
  user uses the product five times a day, then you should look at
• 24:29
  every log and then say, oh, this one is fine, but this one didn't
• 24:34
  look so good. Let me add it to my eval's.
• 24:36
  As you start to scale, there are a lot of different things you
• 24:39
  can do. The thing that has worked really
• 24:42
  well for our customers is using online scoring.
• 24:45
  Online scoring lets you write code or use LLMS to look at your
• 24:49
  data and then figure out what is likely to be, you know,
• 24:52
  potentially bad and then and then save it for evals.
• 24:55
  And there's a lot of, you know, really interesting stuff you can
• 24:58
  do there. And then of course, the other
• 25:00
  signal of course is user feedback.
• 25:03
  So you can capture, you know, explicit user feedback like
• 25:05
  thumbs up or thumbs down or implicit, like are they shaking
• 25:08
  the phone or whatever. And that tends to be a good
• 25:11
  signal, but it's very, very noisy.
• 25:13
  So I think you know, using LLMS to post process stuff, applying
• 25:18
  curation with some human insight, all that is very
• 25:21
  important. What's next for evals, agents
• 25:23
  and AI infrastructure of the next few years?
• 25:25
  You know, so one of the weird things over the past couple
• 25:28
  years of building brain trust is that we work with teams that are
• 25:32
  building like the coolest AI products in the world, but our
• 25:35
  product has had like almost no AI in it.
• 25:38
  And you know, I feel like we, we're building like a previous
• 25:41
  generation product in in some ways.
• 25:44
  And the, the reason is that until recently, models have not
• 25:47
  been really good at looking at each other's work.
• 25:50
  I think like a good metaphor for this is a dog looking at itself
• 25:52
  in the mirror literally like if you give an LLM some of the work
• 25:59
  of another LLM, it will get confused and and sort of have a
• 26:03
  hard time understanding what is it versus, you know, the other
• 26:06
  LLM. Luckily for all of us I think
• 26:10
  Claude 4 GPT 5 and not yet Gemini 2.5 but hopefully.
• 26:14
  Listen up, Google. Yes, please Logan.
• 26:17
  Yeah, everyone fam, please. The model is clearly smart
• 26:21
  enough, but it's just not good at calling tools anyway.
• 26:25
  The most recent vintage of models are, you know, they have
• 26:29
  enough of an identity, if you will, that they're able to, to
• 26:31
  decompose this. And I think with advancements
• 26:34
  around reasoning and so on, we've just seen in our, in our
• 26:36
  evals that they're able to now actually meaningfully improve
• 26:41
  the work of another LLM. And so we've started unleashing
• 26:44
  this in our product. We just released something
• 26:46
  called Loop, which is an agent that's built into brain trust.
• 26:49
  It's now accessible as an MCP as well.
• 26:52
  And so you can use it, you know, in whatever medium makes the
• 26:55
  most sense. But it is incredibly good at
• 26:58
  actually looking at the work of another LLM and improving
• 27:01
  things. And we've seen great performance
• 27:03
  for things like automatically improving prompts, looking at a
• 27:07
  data set and figuring out, OK, what should I add to this data
• 27:10
  set to make it better? You know, looking at logs and
• 27:14
  trying to figure out what should I pull from these logs that
• 27:18
  Kevin's not already testing and, and help him.
• 27:21
  Yeah, exactly. And so I, you know, I, I think
• 27:25
  again, if you go back to what evals fundamentally are, right,
• 27:27
  it's like as a human, I'm helping to guide a model, do
• 27:32
  something that aligns with my or users expectations.
• 27:37
  The implementation of that until recently was very manual.
• 27:40
  Like you look at a dashboard with your eyeballs and then you
• 27:42
  click around and you're like, OK, let me like type 65
• 27:45
  characters to see if I can tell the model that I'm going to lose
• 27:48
  my job if I, you know, whatever and see if it changes things.
• 27:51
  And I think that that part of it is, is, is really going to
• 27:54
  change over the next year. It's the implementation of evals
• 27:58
  and creating the feedback loop from logs to evals is going to
• 28:02
  be very, very LLM driven. But I think we're still going to
• 28:05
  be guiding the taste and judgement that sort of goes into
• 28:08
  that. Where can people get involved
• 28:09
  and follow your work? So I'm on X slash Twitter.
• 28:13
  My username is ANKRGYL. As an Indian person, it's kind
• 28:17
  of hard to get your name, so I discovered when I was like 11
• 28:20
  that if I remove the vowels. That was available.
• 28:24
  And, and, you know, it's, it's actually worked well, it's, it's
• 28:27
  universally worked for me across every platform.
• 28:29
  So none of the other, you know, 1,000,000 people with my exact
• 28:32
  name. I figured it out.
• 28:34
  Yeah, exactly. Yeah.
• 28:35
  The. And what's the what's the brain
• 28:36
  trust website? Braintrust dot dev.
• 28:39
  Awesome. And then any offer anything you
• 28:41
  want to share with Founder Mode listeners for for Braintrust?
• 28:43
  Yeah, if you listen to the podcast and you try out
• 28:46
  Braintrust and you want a sweet deal, hit me up.
• 28:49
  You can reach reach out at ANKUR at Braintrust dot dev or hit us
• 28:54
  up on discord love. It all right, that was super
• 28:58
  clear and just like such a refreshing take on what's
• 29:01
  actually working in AI. So thank you for being here,
• 29:03
  Anker. Thanks for having me a lot of
• 29:05
  fun. That was one of those interviews
• 29:07
  where the answers really can help us and me tomorrow.
• 29:10
  So I think, you know, it's not just theory, but it's on the
• 29:13
  road map. Yeah, you're, I know you're
• 29:15
  going to call a couple people after this.
• 29:16
  What stood out to me was how Anker really broke down the way
• 29:19
  the models have evolved and how that has implicated his work at
• 29:21
  brain trust. And now he's solved this problem
• 29:23
  multiple times. It's been amazing.
• 29:24
  What were your top five, Kevin? Yeah.
• 29:26
  So I think databases are everything.
• 29:28
  I think this idea that evals need to become, you know,
• 29:32
  mandatory once you start to see regressions, the idea that you
• 29:35
  ship fast, but then have this closed feedback loop, LLM sort
• 29:40
  of need to be evaluated monthly, right?
• 29:41
  You know, in the old days you'd look at infrastructure every
• 29:43
  year or two. And now it's like LLMS are
• 29:45
  moving so quickly. You need to sort of like take a
• 29:47
  quicker look on a more, you know, frequent basis.
• 29:50
  And then finally this idea that as long as you're not using
• 29:52
  Gemini models can improve models and I Gemini will get there.
• 29:56
  Google, you know, they're grinding hard, but I think that,
• 29:59
  you know, was the big ones I saw today.
• 30:01
  Yeah, if this helped clarify your thinking on AI development,
• 30:04
  please share it with your team, especially the people that are
• 30:06
  thinking beyond the prototype. And if you're enjoying Founder
• 30:08
  Mode, like subscribe, drop us a comment, give us a review, and
• 30:13
  let us know your feedback. And that's a wrap.
• 30:15
  Because building AI is hard, but shipping it doesn't have to be.