Latest / CTO Lunches / Ep4 - Diana Pfeil on Evals

Transcript

Ep4 - Diana Pfeil on Evals · CTO Lunches

• Rishi: All right. Hi everyone. Welcome to the CTO Lunches podcast. I'm Rishi Malik here with Kendall Miller as always, and our special guest today, Diana File. Diana, would you like to introduce yourself? Diana Pfeil: Hello. Yeah, sure. I'm Diana File and I am the owner of Sunbeam Consulting, where I help companies implement evals for the their AI products. And I also do some mix of AI strategy consulting when people are stuck and don't know where to go with their Kendall: Can you before we keep going, can you talk just a little bit Rishi: Yeah. Kendall: about previous roles that you've had at Sunbeam? Before Sunbeam. Sorry. Diana Pfeil: Okay, yeah, I've been working in tech for a little over 20 years. I started out as a software engineer at Amazon. I think now it would be called an ML engineer. I worked on recommender systems. It's kind of the role that blew my mind and made me in love with data. And that's what I've been doing ever since: machine learning. I along the way got a PhD from MIT. I worked at two other startups, one called Kendall: No way, hang on, hang on. I just just really enjoying that you just sandwiched that in there and kept going. Okay, so keep going. Totally, no big deal. Just another Diana Pfeil: Ha ha ha. Kendall: just a PhD from MIT and then keep going. Diana Pfeil: Not that big of a deal. There's a lot of us. But I also was a travel tech startup as the CTO. more recently at the digital rights startup as a CTO. I've kind of gone back and forth between IC work in data science related and leadership roles. And most recently before Sunbeam. that's that's when I was at PECS, the digital rights company. I've also been at Honey, which is an e-commerce company that got acquired by PayPal a number of years ago, and I was reminded why I don't like large companies. So I'm just happily exploring the startup world and have kind of found a home with evaluations and AI. Rishi: That's awesome. Thank you. Sure. Rushi Malik, been a tech leader for decades at this point, founder of a find your next job and make sure you're ready to find your next job career platform called Hype Up, along with lots of other, you know, random things that I've done over the years, including Friday Deployment Spirits, a tech forward spirits bl brand that I run with Kendall. Kendall, over to you. Kendall: And I'm Kendall Miller. I'm working on AI in a box at a company called Deployed. and I also work with Rishi on Friday Deployment Spirits and we run a startup advisory group called Grow Big Advisors and a couple other things, but I'll stop there. And yeah, glad to go ahead, Rishi. Rishi: I just realized something, Kendall. We've had three guests on the show. not all the episodes have come out, but Corey told us how he has way more important people on his show than we do. And then we followed up with the fact that our Diana here and our last guest, Rick, who people see that episode with both have PhDs from MIT. what are we doing? How do we get such high-level people on this show Diana Pfeil: I told you there's a lot. Rishi: with us? Kendall: it's the it's just a PhD, Rishi. As Diana says, it's no big Rishi: Just Kendall: deal. Nobody's even really Yeah, I'm very impressed. It's that's that's amazing to me. Diana Pfeil: PhDs get Rishi: Yeah. Diana Pfeil: all bad rap. I think it's part of a lot of frustration when people hire data scientists or machine learning people. They tend to go down rabbit holes, they kinda get lost in all this stuff that doesn't matter for the business. It's like good and bad, I think. Kendall: Well, d so I was in academia for a while, and yeah, you you end up becoming the foremost expert on a topic that is this big that nobody r else Rishi: Mm-hmm. Kendall: probably knows about, let alone cares about, and but that's still pretty dang cool because to even find out where the edge of human knowledge is is kind of an impressive thing. But anyways. Yeah. Diana Pfeil: Yeah, it's fun to be able to dive into something insanely specific and push the frontier of it and feel like if you're you encounter a new problem you can like go there if you need to. But y I d you don't end up having broad knowledge necessarily. Or practical knowledge sometimes. Kendall: Well, speaking of broad or impractical knowledge, before we dive into evals I'm excited to get there, Rishi, what are you drinking? Rishi: I today am drinking a vodka yeah, Moscow mule made with mint from my garden that I had to go to battle with and tear out and this is my my victory drink after going through all the work to tear out this plant that I would actually classify as a weed. and took a lot of work. with Kendall: Okay. Rishi: rye vodka from from Berry Creek distillery. what are you drinking, Kendall? Kendall: well first of all, if when you conquer something, you put it in your drink, I'm curious what happens when you take out a wasp nest. But I just have a Pilsner today, that I'm Rishi: Mm-hmm. Kendall: drinking. I think it's a Stella. and I have a number of meetings left this afternoon. It's the afternoon. We're recording this in the afternoon, so the beer feels more appropriate than the morning ones that we've done in. Diana, what are you drinking? Diana Pfeil: Well, I feel like I didn't prepare for this enough. I wish I got a warning that I should have a cocktail, but I have a hint water with a bunch of ice in my cup here because I didn't want anything too bubbly. Kendall: Okay, we'll take it. Rishi: Nice. Excellent hydration. And to be clear, the reason you get a warning was just me completely dropping the ball on sending you the info. So I apologize for that. But cocktails on on on me for sure. Kendall: There we Rishi: well let's let's dive into evals. you know, I've I've heard you know a bit about them. I've played around with them, but think most people are probably somewhere on the spectrum of either super, you know, trying to use them in DAF, they're doing a lot with AI or have absolutely no idea kind of what that is or how it's different than anything that exists out there. So maybe we start with that, you know, just kind of what Neval is and then a little bit of a framing and context there for folks. Kendall: Well and and I wanna I wanna ask, my first job in tech was in QA. And so to me, evals of AI, all I hear is QA. And so yeah, what what are evals? How's it different from the way I think of QA QA? And and is it just is evals just like there's a human sitting in front of a chat box saying how many R's are there in strawberry and Give me a random number between one and one hundred. these kids that's obviously not what it is. I'm mostly sure. so yeah, explain like I'm five to start and then and then go. Assume some of us in this room don't have PhDs from MIT. Just I'm not gonna say who, because I don't want to embarrass the the at l Diana Pfeil: Ha ha Rishi: At least two thirds of the people in this room. Kendall: yes. Diana Pfeil: Okay, you can think of evals as QA for the AI age, I guess, but there's just a lot to it. It's more like a process and a state of mind, if you will. So I'll just explain maybe with an example too. So AI evals are tests to measure the quality of your AI application. but the Challenge with AI applications is a lot can go wrong. So let's say I'm building an agent that helps me book travel or helps customers book travel between for trips. So like I I chat, I put in my agent, hey, help me book a trip from Denver to somewhere in Greece, and then go. And so, unlike QA, where there might be one right answer for your system. There's a lot of right answers. who knows what this trip is going to be that this thing comes up with? And so AI evals allow you to answer: is my system working? What are the failure modes of my system? How often do those failure modes happen? If I change the behavior of my system, do I have any regressions? And It's extra hard to ask those questions of AI systems because they're built on top of LLMs, which are the craziest primitives. They're non-deterministic. There's infinite input, infinite output. So at a high level, it is kind of like tests, but and I can go into details like how they're different, but they kind of handle all of the extra stuff you have to deal with when you're building on top of LLMs as a primitive. Kendall: Okay, so so I have questions about this. And and yes, please keep going, even just in the explanation. But so if I'm buying a ticket, I you know, this is a When I was doing maybe don't AI, right? One of the things I talked about all the time is if I'm buying a ticket from Denver to Detroit, I don't under any circumstances want the model to decide that a ticket through Dubai is the best way to do that, right? and so that falls into the eval's world, it's it sounds like. And is the way that you're doing evals, well, I'm asking the question with Claude and I'm checking the answer with Gemini, or like, how are you actually going about this? Because when it's non-deterministic, ha you can't just use determinant. checks to check everything. So yeah, get get into some of the nitty gritty on on how this actually works. And I imagine there's gonna be a Jev conversation here too, but anyways, keep going. Diana Pfeil: Happy to talk about Jeff. Okay, so you're booking your ticket, you've asked for travel from Denver to Detroit, and you know that okay, a bad outcome would be if you get booked through Deep Dubai. But like, what are the all the other possible bad outcomes? Can you even enumerate those up front, like a software tester might do in the past? You cannot. So with evals, there's kind of a process. And this is a pretty immature field, I would say, but I the the process I follow is one that I think works really well, which is you start with this idea of error analysis. And what you do is you run, you instrument your system, so you're collecting traces, which means that you're kind of collecting what happened through every step of your agent. And then you generate either synthetically or ideally. sampling from your actual production user traffic, like a very diverse set of scenarios. You have to think through what are the sources of error and diversity in the kinds of requests that my system is handling. So maybe someone wants to c show up for math homework. Maybe the user doesn't speak English very well and that like the way that they're asking questions is really off. So you want so you sample and you can use an LLM to help you with this. a very diverse set of inputs and then you run that through the system and then there's this process of error analysis where you actually have to look at your data and you identify all of the error modes and failure modes of your system. And so you might notice that you're looking at a trace and it says, it's routing me through Dubai. That's so weird. And another one that's that says, I'm given this refund policy that doesn't exist. this agent just told me about, came up with this random refund policy and it's completely made up. And so you go through this process and you end up having a pretty complete list of the failure modes of your system. And then now you can create ways that you can check that those failure modes exist. They might be deterministic checks, like you might just be able to tell, is the trip journey like longer than this multiplier over the length, the distance between the two cities. Like that's a deterministic check that you could just run in code and it's fast. Or you might need a check that's an LLM judge. So it's subjective. It might be like, is this refund policy made up that so that might be a subjective check check. And what you have to do after that is you have to calibrate these LLM judges so that they're actually consistent and you can trust them when you run them against your traces in production later. But at the end of this, you have all your failure modes and you can think of your evaluation set as both a set of regression scenarios, like these are things that I expect that my system to handle well, and a system of like capability scenarios where I want to make sure that these errors that I found when I was looking through my agent aren't occurring again. So you end up with this nice suite of tests. And then let's say you you someone's asking, hey, Jev came out, can we swap out this like router model in our agent with Jev without breaking everything across all of our scenarios? You can make both improvements that way, with that w knowing that your error mode like your failure modes haven't popped higher than they used to be. Rishi: So that that all makes a ton of sense. for talk to someone, you know, like myself who's kind of from the pure software side of things, I did always have the mental model, these are regression tests or end-to-end tests, whatever you want to call it. but that can work on non deterministic output. Does the you know, with the work that you do with the, you know, the customers and clients that you have, do you see timing about when you develop this as being relevant? Right. A lot of us you have done T D or at the design and planning stage of a system, you're like, great, here's all the edge cases, here's the expected outputs and things that are gonna go wrong. I find for myself with L LMs that I don't really know all the wild concepts of what the output can be until they're actually running in production. So is this something you do in advance before you ship? Is it something you do after, or is it a I don't know, iterative improvement type of thing? Diana Pfeil: Yeah, unlike T D, it's not something you do before because you have to be able to run some kind of real scenarios through your system to do that entire error analysis process. It doesn't necessarily need to be in production. There are ways to generate synthetic simulations, at least so that you're somewhat comfortable of your best guess of what's gonna happen when you put this into the real world. But you do need to have some working version of the system. But it can be pretty basic. you can code up something pretty basic for a lot of agentic applications just after you put together your spec and your requirements. And so that's that's the point where you you start doing this process. Kendall: two questions. Why or did you have a follow up to that, Rishi? Rishi: Go ahead. I was gonna take it slightly differently. So feel free to jump in. Kendall: why there's there's two things I wanna know one, why can't I just say and LLM don't make stupid mistakes and y and then trust it to to make good you know like and like I I say that kind of tongue in cheek, but like won't these models just get better enough that like this becomes less of a problem as they get become better? I mean even if it's one in a thousand instead of one in a hundred, you know, why why are is this I understand why it needs to exist today. Do you think it's gonna need to exist forever? Do you think the models are gonna get better? And then the other question I have is can't I solve some very large majority of these problems by running the decision through another LLM and saying, was the decision that this LLM made a good decision or not? Diana Pfeil: I'll start with the last part. Can you run it through a model and ask is the decision this LLM made is a good decision or not? That is an LLM judge. So yes, you can run an LLM judge and say, hey, was did the customer get their question answered? Or whatever, what is whatever is a successful outcome of the product that you're building? And but if you use a really powerful model. in order to judge how good this is, you might as well have used that super powerful model for the entire agent. And yeah, maybe when costs of models go to zero and they're a lot better than they are, you need a lot less of this. And I can't like predict how that's gonna roll out. But I think some level of understanding being able to make changes to your system and figuring out if you have regressions or not or if you forgot something or something's off. I don't see that going away. Like I think that's always gonna exist where maybe you're optimizing one part of your system and it breaks down another part. Like you probably want to know and measure that and and like the models might get better, but you're often also still making subjective decisions. And so this whole process helps you also from a product perspective figure out what is good for my product and how do I encode that in my spec and then in my set of evaluations to make sure that I'm holding that standard for my product. Kendall: Okay. go ahead, Rishi. Rishi: Do you see thank you? D do you see trying to think of the best phrasing, the usage of LLMs in this way as being a step on a path to eventually something becoming deterministic, or is it always kind of LLMs? You know, and what I mean by that is you have an LLM do the actual work, you have an LLM judge to judge the output of the work, it's still by definition non-deterministic, you Could have another LLM judge, judge the first judge, you know, and it's LLMs all the way down. Or at some point, do you collect enough data by running these evals where you're like, great, now I'm gonna feed this into another LLM, but the output of this is deterministic static rules. Okay, I know enough about my system now, no I know enough about flight flight path to Detroit that after all of these non-deterministic aspects to it, I can now actually have a very codified set of rules that is gonna be in consistent and can be run just as a normal test. Or do you think it it stays as LMs all the way through? Diana Pfeil: I love this question. So I do think the one thing that will change is that LLMs as a primitive are so wasteful. They're so annoying. It's actually why I really like JEV, because for instance, a lot of agents might have a routing component. Like my travel assistant, someone might ask, Hey, look up a policy for this, or they might book a flight. And like you might wrote the route those to different tools, right? Our agent SDKs. have a set of tools that are available and there's a routing component where they're choosing which tool. And LLMs are a really inefficient, actually bizarre way to pick a cr across a set of options of tools to select. And something like JEV, which is removes a little bit of the the non I mean it's still like you still have a set of choices, but it's more It's more custom it makes sense more as a tool for routing in an agent. And I think there's gonna be more of these primitives that aren't just LLMs everywhere where you're like writing random prompts and hoping that they work that are gonna fit across scenarios. And so programmatically we're gonna be able to assemble just like primitives that tend to work better for us, which I think will remove a lot of the sources of error that we're currently seeing. But right now everything's built on top of LLMs, which is a little bit nuts. Kendall: Yeah, I think one of the interesting things about i you know, I I'm actually one of the more LLMs operate more like our brains than people give them credit, kind of people. You know, there's a lot of people who are like, It's not a brain, it's just in and okay, it's not a brain. I'm not arguing that, but yeah, go ahead. Go ahead. Diana Pfeil: They do. Can I interrupt you though? They they operate on our blank brain, but then we're building these systems and it's like a a function talking to a function via an LLM. It's like why would a function translate what it wants to an LLM to get data out of another function? Rishi: Mm. Diana Pfeil: That's so suboptimal. Like it should at the very least communicate through some latent, efficient representation, not like human language, which is Kendall: What yeah, one hundred percent. Well, and so so that's where I'm going with this is that in one of the ways that is my opinion that we operate very differently than LLMs is an LLM almost has to have the thought in full language before it's determined the outcome. Like it it's it almost writes the thing, processes what it wrote, and then makes the decision, right? It it's it's a very inefficient And as a person who is incapable of knowing what I'm thinking unless my mouth is moving, I know that my thoughts are not fully formed as they're coming out. Right. Like I'm I'm actually processing what I'm thinking as it's come you know, and and maybe I'm wrong. Like I don't I don't understand the internals of these as much as but like I when you're saying something, I'm nodding along, agreeing immediately. Before any words have been produced, even in my head, I'm I'm I'm not thinking to myself, yes, everything Diana is saying is correct. You're absolutely right, right? that's not what's happening. I'm just, yes, right. It's a binary, Diana Pfeil: Processing, yeah. Kendall: to your point, more like Jev, where it's this, it's this very quick decision because it doesn't need to go through the linguistic. The user is asking me this, and so here's the things that I need to you're right, like language models themselves, as brilliant as they are. are in a lot of ways inefficient. And well so so stay on the Jev thing for a second, if that's the case. I mean Jev is i gives us the and just as a description of Jev, I'm gonna give my dys description and then Correct me where I I get it wrong. But, instead of it being a language model that's going to say, Diana said this thing, here's the ways that I think she's right, and therefore I'm gonna say this thing in response, yada yada, it just says, you know, yes or no, or you know, some predetermined number of decisions that it can make and it assigns a probability to that and it and it goes. and so it's a much more efficient process, right? and so in fact, it's so much so that right now I think with Jev, you pay for The inputs to the model, but you pay almost nothing for the outputs, or maybe nothing completely. was it, was it, it's like, it's so cheap. Yeah, Diana Pfeil: Nothing for the output. Kendall: there's no reason for us to even meter it. That's kind of wild, right? So, like in in the world of evals, once you've fed enough information in, yes, you've paid to train it, but then like you're running near real-time intercept and decision making on a thing at basically free or actually free, like. This is the future of evals, right? Diana Pfeil: Yeah, but I think it's not that simple because from my under I don't have the full understanding of JEV because I haven't gotten to play with it yet. but I think you still have this notion of you have to give it all this context up front. Like let's say that you're having it decide is is this like is this trace appropriate or not? Did the user problem get solved? Did they get routed? Like did we recommend a really good activity for them to do in Detroit? Yes or no. You have to give Jev all of this context. hey, this person wants to go to Detroit. This is what who they're they are. This is how old they are. we're recommending them this bike trip. Yes or no, is this good or not? Like you still have a pretty like you still have to get that input and that context right. So It's more efficient and I think there's a lot of improvements to LLM w because the output is less crazy, but there's still a lot of this it's still subjective and still language models. You still might have to tune it. so yeah, I think it's the future, but it doesn't Kendall: No, I Diana Pfeil: like solve a huge problem n necessarily. It's a nice Kendall: Right. No, I I think you're right and I think I've oversimplified that. So that's that it's interesting what you're saying back, is that the the amount of input that still needs to go in is still extremely verbose even if what comes out is Diana Pfeil: Yeah, depending on how good the decision. it needs the context in order to give you yes or no where that probability is meaningful in some way. Otherwise it'll just like LLMs give you random stuff. It'll just guess make stuff up. Kendall: Yeah, which is is is a human problem too, right? Rishi, I need your advice on this thing, but give me twenty minutes because the context behind it is fucking nuts and then Rishi: Yeah. Kendall: at the end of it he goes, Yeah, that was a bad idea. anyways, Rishi, what were you gonna ask? Rishi: Well, so is that actually dovetails into a little bit, your your example is kind of great. Yes, I could say that that was a terrible idea, or no, it's a great idea, but I feel like so many aspects of things are not typically, you know, kind of that binary output. you know, when we're talking about Jeff, like that's the thing you get, it's like a you know, a list of probabilities essentially with whatever your your structure of output is. you know, and so I guess at higher level, you know, is this maybe it's the best practices or common mistakes type of thing with evals, but do people or should people be looking at this as a pass fail? It's a it is a test. Maybe it is, you know, an eval, but like it either passed or it didn't. or is that the wrong way to is that the wrong mental model to have around that and we should be thinking about them differently when it comes to evaluating the the efficacy of the test. Diana Pfeil: best practice for if you're building an LLM judge is a binary yes or no. And the reason is that it's already pretty tricky to draw a boundary between is this relevant, is this not relevant? Because there's a lot of fuzzy gray area. And so the binary decision both forces you as the person calibrating this judge to kind of make a decision either way rather than be wishy-washy. So actually like The LLM has more real stuff to go off of. There's real input being made here. And it's really hard to calibrate a judge when there's like a score between one and five. It's much harder to like have useful output. So even though it seems like you want some nuance in practice, it takes so long and so much effort to get a judge that's reliable even for a binary decision. That you don't want to complicate the process because now you're spending way too much effort on your evals. It shouldn't be the the bulk of your effort. And so the binary is the way to go. On the gem front, there is a way that you can get a probability out. Like you can ask a question and you'll get a score between zero and one. So there's like different ways that you can ask for a decision to be made. Kendall: Yeah. The well, so I I'm curious then okay, how how do I ask this? we need Diana Pfeil: Can I a can I add one thing to that, Rishi? Which is I don't Rishi: Of course. Diana Pfeil: maybe in your own work just playing with LLMs or doing prompts. my experience is definitely that if I ask the LLM to give me like give me the severity, is it like medium, high, or low? The medium is almost useless. Like I just I just never really learn anything from it. It's just much better to simplify this for this LLM. and give it fewer options. Rishi: Hmm. Diana Pfeil: Otherwise you really have to spend a lot of time prompt engineering the this much more aggressively so that your middle, like your severity medium is just as effective as your other two. Rishi: in in my head, what you're describing almost sounds like what you'd see in in circuits of hysteresis of in the middle ground it can shift either way, so you're not really getting good signal out of it. You have to push it by giving it, more information one way or another to get to more determinism. That makes Diana Pfeil: Mm-hmm. Yeah. Rishi: sense. That's fascinating. Thank you. Sorry, Kendall, you were you gotta say something before we go on? Kendall: No, and I appreciate the additional context. what I'm I am struggling without ask this question. In what world does it make sense to not do evals because the process of the evals is more expensive than the negative potential outcomes from the AI decision making? Diana Pfeil: If you're building your own personal project, probably not worth it to touch Evals. you don't really care about the quality. There's no legal action that's gonna happen. If there's like a policy that's recommended that doesn't exist, you're not gonna give yourself bad PR by having a terrible agentic experience. So it's really about impact of your and risk of getting something wrong for your product. how correct does it need to Kendall: So Diana Pfeil: be? How regulated is your space? what if you make a mistake, what is the impact on others or for your business? Kendall: So so how did how did this become the thing that you latched on to? why did this you've you've gone deep down this rabbit hole and you you you kind of got there through a little bit of a circuitous route helping with AI data, a number of different things. Why did this become the thing you latched onto? Like what about it particularly appeals to you? Yeah, I you know, I I said my first job in tech was QA. I don't think that was the case for you. So how did you end up you know, real interested in this particular idea? Diana Pfeil: I have zero affinity for QA. I I'm it's unfortunate that this is essentially testing that I'm now working on. So I really dislike that. But how did I get here? So I've been doing more general AI advising for a lot of clients, and they'll be building like a rag system or they have an agent, and they might want a sanity check, like, hey, Help us build this thing, is it working? Or my gosh, this thing d just never works. We cannot release this in the field. What's wrong? And so over those engagements, the biggest fire that people could not really figure out is the need for evals. Like it was always that their their agent or their system or their rag system, like they just evaluated it by by vibes. they would try a couple things and it's good enough. And they'd had no people just have no sense of how to build trust in these non-deterministic systems. So I helped a couple clients do that. And the reason why I like it is because it's actually a lot of data science work. You're calibr you're picking metrics, you're calibrating metrics. You're kind of like almost training LLM judges. You're doing like almost fine-tuning, but it's v And sometimes there is actual fine tuning that you have to do for to get the router right or whatever, and that's traditional machine learning, so I love that. And then sometimes you're calibrating these judges, but you have to do it in a very machine learning forward way, where you you set it up on a set of data and then you evaluate it on a held out set of data. Because if you calibrate it on all the data. then it's gonna be biased. It's just gonna have seen all the scenarios and you won't actually learn anything. And that's called generalization and machine learning. So I just feel like it's the new job of data scientists, these evals. Plus it's very product forward. And in my past, especially as a CTO, my biggest repeated point of friction was the fact that my engineering teams were building stuff that hasn't been validated with the user. And so I care a lot about product and like getting people to actually use the systems that you're building. And I feel like having an eval suite is the only way to do it with these like when you're building on top of L LMs. L LMs are a total mess. Like they make so many mistakes. So it's the only way to build a reliable system. Rishi: So I I I'm gonna I I'm gonna ask a question here and absolutely if this is just a terrible idea, just just you you can absolutely say that with no no qualms. You know, we're in a world where like d writing code is just trivial these days, right? It's it's like it's easy, you know, all these LMs can spit it out. and that's made it so the amount of code that's coming out is so large humans can't really review it. We're all doing agentic code review now of you know of all this stuff and it's you it's its agents all the way down. Is it a terrible idea to just have LLMs also generate the evaluate, also do the the analysis and build that structure all around it? Or is it something where, yeah, you can do it, but it's pointless unless you're going to have a human go through and do the calibration, do the review and make sure that what it's actually looking like looking at is, you know, sane. Diana Pfeil: just stepping back, we are all building kind of these black boxes and we're trying to figure out how to test them. And the ideal world is we have a combination of like very like type safe and good code, and so we eliminate a ton of errors that way. And we have like tests or evals where we can automate that and eliminate errors that way, and we feel completely comfortable just telling Claude to write all our code. The only problem is that unfortunately without that human review component where you look at your traces, it doesn't work because the LLM doesn't know the errors, it doesn't have the product sense to figure out what's actually a mistake. If it knew it wouldn't have made the mistake in the first place. And so you can automate a lot of the steps of the eval process. Like I mentioned, you can generate synthetic traces and an LLM can Claude can help you with all of that stuff. But there's some part of it that is not automatable. Like I've never seen it done. People try to do it and then you just get a lie that your thing works, but it doesn't work. Rishi: Yeah. Diana Pfeil: so right now it's still something that requires a human. And the like the way I explain why this is true is because co writing code is free, but our product manager hat where we judge whether this is a good product that people want. That's a different skill. You're not outsourcing that to Claude yet, right? this judgment, the product judgment is still something that we're doing ourselves. That's why we are directing our swarm of agents because we have to do some direction based on our product knowledge. So that's the analogous with these evals, you need a lot of product outvo knowledge to be able to determine what is a failure mode, what do I want this product to look like, what is a good interaction with my customer, et cetera. Rishi: That makes a ton of sense. Thank you. I I I appreciate that. should we so I guess a couple of things that we can we can talk through here. number one, you know, a little possibility of your your your view, your hot takes on where the industry is on evals and what people are doing, what people aren't doing. you know, I I think that'd be I think that's something that'd be super fascinating for folks. I guess before that, Kendall, did you have anything else you want to jump into? any other questions from what you were asking? Kendall: No, let's let's let's do that for now. I like I like that angle. Rishi: Excellent. And also say right before we jump into that, Dan, as well, with all your knowledge, if there's something you want to make sure people just that we haven't talked to about evals or about yourself and the the work that you're doing that you just want to make sure people understand, you know, feel free to to throw that out as well. So we just get that knowledge into the folks listening as well. Diana Pfeil: Okay, for hot takes, I'm not great at hot takes. do you guys have hot takes to kick us off? Rishi: I could, yeah. Kendall: On Evals, my hot take Diana Pfeil: Or at anything. Kendall: is yeah, I mean my hot take on Evals is that it is going to be LLMs all the way down. and we probably need a minimum of three to four to feel good about decisions, but we're gonna settle for one additional LLM judge. and and I think Jev is going to solve all of our problems. Rishi: It's almost like Jeff is sponsoring Kendall here. Kendall: it's not but so the the the w basically this is the world I lived in for the last year, with with maybe don't AI, right? Something needs to tell the AI maybe don't do that. And the things that Jev makes makes possible in that world at the speed that it makes it possible is really, really game changing and and I think we're very early in this, but I think it's it's gonna be more things like that. But I think this is the beginning of the that's how this should have been all along, not just outrageous verbosity, like let's create little loquacious machines. no, you gave it anxiety, you know, kinds of kinds of problems. Rishi, what's your hot take? Rishi: So my my hot take here is evals are probably the most important thing in learning from Diana here, the most important thing that a business can do to make sure their product is actually excellent, you know, you w when it uses AI. But almost no company is going to do it. And the companies that do it are gonna do a really shitty job of it. And it is just gonna be LMs all the way down and no one's gonna look at it and it's just gonna be it's gonna be worse than just AI slop because everyone's gonna have confident that they're shipping something good when it's completely misplaced and we're just consistently shipping bad things. Which for some early days now isn't huge, but like we were kind of talking about an earlier episode, when it comes to assistance or, you know, bookie travel or sending email or, you know, what it, you know, at some point when it's healthcare or like making, you know, HR decisions, it's gonna be a giant mess and tech collectively is going to be doing the wrong thing there. Diana Pfeil: Okay, I'm ready for my hot take. I like both of yours. Rishi: All right. Diana Pfeil: Okay. my hot take is that the lack of evals is the reason why everyone's agent prop projects get shut down. I think every company is like, hey, do something with AI, and people are like, okay, and they build some agent. And then it's like a mess and doesn't do what you want and has all these errors, and they're like Okay, we're gonna we can't deal with this right now. Let's deprioritize this. But if you could embrace this process and actually look at your traces, then you can build something that might be like the vision you have. It might be useful. Like you might be able to ship it to your customers and feel good about it. But right now a lot of projects just go into the garbage. It's like a lot of wasted time because People build these systems that don't work well enough to share in the public. Kendall: My my to piggyback off that, I think the the thought that I had was really good and gone. That's what I think. Thank God for the ability to edit things out. Diana Pfeil: And then Rishi: If you had an email for your thought processes, you'd have known. Diana Pfeil: And can I I have one more this isn't a hot take, really. It's just a take, which is I'm so excited about Jev because it's a different paradigm and we need it because agent-to-agent communication can't be LLMs. It's too inefficient. But I think that it's not like that great. Jev isn't that great. It has a lot of the problems LLMs have. You still have to calibrate your output. You can't just trust it. You're still need to gonna need to evaluate it. So but I think it's like a step towards a lot of classes of language models that we can start using as primitives. And I'm super excited for that, for like that to evolve. Rishi: I think that I think that's fairly hot hot take. I I personally saw Jeb and started playing around with it. I mean this was only yesterday, the day before, but I was like, I have to do what? I gotta do work here? No, no, no. This is the whole point of AI and LLMs. I don't have to think. What what's going on here? Yeah, exactly. Diana Pfeil: Yeah, Claude will tell you what your options are. People really like the laziness. It'll just fill in every blank. That's what's nice about it. Kendall: I think that when AI destroys humanity before the end of the decade, Jev is gonna let you live at least an extra month, Diana, just because of that hot take. So so I I appreciate that. And before we completely disconnect, tell us what your PhD was on. Diana Pfeil: My PhD was a mix of machine learning and optimization. So and I focused on air traffic systems in Aero Astro. So I helped predict when flights would get disrupted and then kind of reconfigure the airspace for optimal resolution of disruptions. Rishi: Yeah. Diana Pfeil: And just to th from your reactions that that sounded weird, but the the thing that I loved about it is that is that this Rishi: That sounded amazing. Kendall: It it's a son of man. Diana, I was writing poetry in college. That's that's why I'm amused. Sorry, finish your thought. the thing that you g finish your thought. Diana Pfeil: That sounds great. what I love is that this intersection machine learning is optimization. Like all of our LLMs are this gradient descent in order to better predict the next token of human language. It's just like all a huge optimization problem. And so I just see all of like this industry as a big machine learning optimization problem and I just love it also. I just love reading about All the developments from that perspective. Rishi: That's awesome. That's awesome. It is. Thank you very much for Kendall: Well, it's exciting. Yeah. Rishi: coming on here, talking to us, talking through all the the details with evals. for me, like I was saying earlier, I d I've learned a ton. I've in being transparent, I've clearly been doing the evals I've done wrong already. So I have homework from this episode to go go fix it and make it better. and I think that's amazing. So so thank you for all of your knowledge that you're willing to share with us. Diana Pfeil: Awesome, thank you. Kendall: I feel confident that every agent I have deployed in production is using evals exactly right because I have deployed zero of them. So thank you, Diana, for for joining us. It was it was great. and go look up Diana file if you're fiddling with AI agents and you don't want your agenc process to get shut down internally because you wanna build it right. So you just need some evals and then nobody will shut down your project. You heard it here first. Diana Pfeil: Awesome, this was fun. Rishi: Thanks for listening, everyone. We'll be back soon.