Latest / The Experimentation Edge / From chatbots to open-world agents at Microsoft: evals, go-live metrics, and copilot velocity
Transcript
- Ashley Stirrup (00:33) We're here with Mark Casalena, who is VP of Products, Core AI and AI Futurist at Microsoft. Ray leads the AI Futures team, a group focused on identifying emerging trends and developing next generation AI products. Earlier at Microsoft, he led products at Azure OpenAI, Azure Cognitive Services, Responsible AI and AI Studio. Before Microsoft, Marco spent over five years at Salesforce leading Einstein and the company's AI platform. So Marco, you're gonna have a lot of great stuff to talk to us about today and how AI is changing the world. Marco Casalaina (01:08) We certainly are in an interesting place. Ashley Stirrup (01:11) Welcome to the show, thank you so much. So this show spends a lot of time looking at how do you test new features and how do you learn about customer experiences and create the optimal customer experiences. How do you think about that with the products that you lead? Marco Casalaina (01:29) Well, I think in the AI era, we are at hyper speed on this stuff. mean, both our ability to create these features and our ability to bring them to our customers ⁓ is enhanced now. mean, consider, for example, one of our newer ⁓ products is the Copilot CLI. So you may have heard of GitHub Copilot, which was the OG, you know, the original code completion thing. But that's evolved into Copilot CLI, which is kind of like Claude Code that some people know. And I would say that it's... Ashley Stirrup (01:51) course. Marco Casalaina (01:59) pretty much reached parity now with Claude Code, but we're doing like two releases a day. I mean, it's insane here because we use Copilot CLI to help us build Copilot CLI. that's a thing, right? The velocity of this stuff is vastly increased. Ashley Stirrup (02:18) Yeah, and so that opens up all sorts of different questions on like, how do you measure? Is the feature I just rolled out adding value or not? What is value in an AI world? Because so often, like if it's a chat bot, was that chat a good experience or did the user leave frustrated? Those kinds of things can be very hard to measure. Marco Casalaina (02:38) Well, mean, on net, ⁓ in our business anyway, because we are largely an enterprise business, that is to say we sell to other businesses and not so much to consumers, in as much as we can measure it, we're looking at go lives. That is to say that they have actually gone live with this thing. And there are certain thresholds that we can see in the telemetry that indicate that, yes, they've gone live, that they have set this up in such a way that it's going to scale, is one indicator. or just overall token volume, like it suddenly increases by a factor of 10 or a factor of 100, I think they've gone live. Or they tell us, you know, lot of customers, because we have relationships with lot of customers, they're like, hey, we just went live for 55,000 employees or something. Okay, they're live, right? So that's key indicator for us of value, frankly, because in general, most companies will pilot this stuff first. And so they'll put out a pilot. Ashley Stirrup (03:12) Right. Marco Casalaina (03:29) And you kind of have to with AI features for some of the things that you mentioned, right? You have to figure out, is it acting in a grounded way? Is it responsible? Those kinds of things. And so by the time they get to the point that they're going live, they have generally vetted that this thing is actually working. Ashley Stirrup (03:44) And so I imagine that you're investing a lot in customer success and trying to figure out the best models in order to support customers on their journey. Because Go Live is obviously an important milestone, but hopefully they just keep expanding their use cases and their usage and they actually get the value out of it that they hoped for initially. ⁓ How do you think about that problem at the scale of company like Microsoft? Marco Casalaina (04:09) Well, at least in my group, Microsoft is, in a sense, many different companies rolled into one. We have our whole consumer business and Xbox and all that stuff, and then we have our enterprise business, of course. And I sit on the enterprise side. ⁓ And in terms of expansion, so I mean, what you see happening right now, for example, is, well, first of all, there's this movement happening from agents that were chatbots that answered questions. to agents that aren't necessarily chat bots and are doing stuff. Now, in Foundry, and Foundry is our tool for AI developers or our AI tool for developers, however you want to put it, in Foundry, we have a whole eval toolkit. And so you could do these evaluations on your AI. And we recently added a whole bunch of agent evaluations. And so the old evaluations were like groundedness and coherence and fluency. And these were things that were really made to measure, it answering? questions correctly and coherently and so on. But the new set of metrics are like tool call accuracy and tool call success rate. So a tool call is when an agent goes to do something, it actually goes and hits one of its integrations. We call that a tool call. And so increasingly we're evaluating those kinds of things. And also there are the task metrics, task completion, task adherence. So that is to say, did it actually do the task that you told it to do? and nothing else. And so those are evaluations that we use internally, certainly. I mean, we use these very same evaluations on our own first party products like Microsoft 365 Copilot. So a lot of folks have that. It's embedded into the office suite. And we're doing these evaluations every day. And there we do do A-B testing. So on the Microsoft Copilot product, there certainly are more A-B tests. with different models and different strategies and all those kinds of things. So we use them in our first party products and we offer them also as third party products for our customers to also use. Ashley Stirrup (06:05) Yeah, I was just thinking like, AI is moving so fast that what good looked like four months ago is very different than what good looks like today. And so I would imagine you need to be evolving your testing and you need to be helping customers evolve their testing. It's very possible that something they tried to automate four months ago is now much more possible and easier to do. Is that true? Marco Casalaina (06:30) Absolutely. mean, for example, we with our own Microsoft 365 co-pilot, we fairly laboriously put together what's called a model router. You know, you don't always want to use the same model for everything. And even if you use chat GPT, for example, they also use a model routing strategy where in some cases, if you make a simple request, it'll now route to GPT 5.3 instant as it's called. Whereas if you make a really complex response, it might go to 5.3 thinking, which is the reasoning model there. So ⁓ We, about a year and half ago, worked up this laborious model routing strategy to make that same sort of decision. But now we have that as a product. There is a model router and you could just use it, you know, and it can route across all these different families of models, Grok and Claude and GPT. So that's something that a year ago would have been a heck of an engineering effort and today is kind of a nothing burger. Ashley Stirrup (07:24) Yeah, yeah, that's interesting because that's like a whole other dimension is that not only is the model getting smarter, but we're building all this infrastructure around the AI models to just make the whole ecosystem work better. Marco Casalaina (07:37) Right, and that's a lot of it. You I mean, you think about these orchestration frameworks, it started with things like Lang chain and our own semantic kernel and has evolved towards things like what we are about to release the Microsoft agent framework for orchestrating multiple agents. But simultaneously, something else is happening and it's happening from here among other things, the copilot SDK. So, you you think about Claude code and copilot CLI, these tools that largely developers use to write code, but you don't. just have to write code with it. For example, I used Claude Code a couple of months ago to author a Word document, which is basically instructions on how to do this class that I was giving. So I didn't write code with it, I made a Word document out of it. These things are pretty intelligent. They can kind of make their own tools, right? So let's say traditional agents, if I wanted it to write a Word document, I would have to give it some kind of a tool that allows it to make a Word document, like by itself, it couldn't do that. But these things, they don't need that because if they don't have the ability to write a Word document, they're like, well, I'll figure out how to do that. I'll code it up for myself and then they'll just go and do it. So we recently released something called the Copilot SDK. And that's a layer on top of the Copilot CLI that allows you to use it as an agent orchestration system. And that is fascinating. mean, people, there's just this explosion of creativity lately where everybody and their brother, myself included, have built their own Co-pilot driven app, I guess. And it's really interesting what's happening there. Ashley Stirrup (09:12) Is that a little similar to what they're doing with open cloth? Marco Casalaina (09:16) It is in a way similar to OpenClaw, right? So in some sense, OpenClaw was one sort of manifestation of this in a fairly insecure way. But it is what I call an open world agent that is OpenClaw could do things that you didn't necessarily set it up to do. In general, mean, you still under any circumstance, like you would still have to integrate it to your email or your Slack or your Teams or whatever. that's not just gonna magically happen like it will just connect to your whatever system but in terms of things that can do can it right a powerpoint document can make a google slide or something like that in general most of the time they can kind of figure out how to do that for themselves open call also had an interesting aspect which was this shared memory and that in their case of massively shared memory that being multiple and this sort of website where they would collaborate with each other And the truth is that quietly in Foundry, we have also had the ability for a couple of months now to make a shared memory because we have this memory thing. You can turn it on for any agent. And in fact, you can point multiple agents at the same memory store. And so they have a shared memory. What that does is not well understood, not by anybody right now. And so I wouldn't recommend it for a production thing, but it's certainly interesting to experiment with it and open claw in general. It's an interesting experiment. don't think it's a product, but it's an interesting experiment that points us towards where AI will go. Ashley Stirrup (10:45) Yeah, I totally agree. feel like the AI coding is kind of a glimpse into the future for what the rest of us will be able to do in our job. Marco Casalaina (10:55) Right. Well, already, I mean, you know, consider Claude Cowork. If you haven't heard of that one, it's an interface that's now embedded into the Claude application and underneath it is Claude code. So it basically is doing the same stuff, but it doesn't look like code and it doesn't look like it's made to write code. It looks like something that you can use to interact with, you know, documents and spreadsheets and that kind of thing. And you could do the same thing. As I said, with Copilot SDK, you know, you could do all kinds of crazy stuff, build these apps on top of it. Ashley Stirrup (11:00) yeah. Marco Casalaina (11:25) And it has this unconstrained ability to innovate when it needs to, basically. Strange but true, yes. Ashley Stirrup (11:31) Yeah, sounds very exciting. So kind of circling back to how you kind of work with your customers and measure their success. I would imagine engagement across your customer base is kind of one of those important metrics. You talked about like task completion rates and things like that. But I'd imagine you're watching engagement pretty closely. Marco Casalaina (11:52) We are, but it's an interesting indicator, especially now. mean, in my group, again, my group Foundry is made for developers. And we have a web portal, the Foundry web portal. And in that web portal, we have playgrounds, and you can set up and monitor your models and things like that. So you can say, I want GPT-53 codecs now. And you can go into the web portal and build that. And then there's our documentation and stuff like that. Now, we discovered an interesting thing in December that caused us to change course. We discovered that something like 80 % of accesses to our documentation were coming from coding agents now. So whereas in the past it was humans, people were going and looking up the documentation at learn.microsoft.com. But now, particularly in our group, we found a lot of this was coding agents. And so that caused us to rewrite all of our documentation in such a way that it would be more amenable to agents reading them. So less florid prose. more terse get to the point kind of stuff, more specific code examples that were tested and vetted. We have an agent that does that stuff. like try the sample code on our site and make sure that actually works because it does get out of date very quickly and that kind of thing. So we are definitely measuring engagement. But then again, I mean, you think about the Foundry Web Portal. I mean, that's kind of like, well, that's kind of like maybe what a dating app is supposed to be. I've been married for a long time, so I never really used a dating app. You know, the idea of the way a dating app is supposed to work is you get in, you find your person, and you leave. You don't come back, right? So that's true also of the Foundry web portal. You get in, you find what you need to do, and then you're off in Codeland or you're implementing, and then you don't necessarily go back to the portal because you're past the playground stage and into the production stage. And so we're measuring engagement, certainly, and we want people to come and we want them to come back, but we don't necessarily want them to stay in the Foundry web portal so much. we would rather see other indicators of engagement like token usage, like go lives, those kinds of things. Ashley Stirrup (13:53) Wow, that's super interesting. I never, ever would have guessed that you'd have compared documentation to a dating app. That's pretty funny. Marco Casalaina (14:00) Well, it's not so much of the documentation as our web portal, it's true. And apparently dating apps don't really work that way. It seems like people are just in them forever. Ashley Stirrup (14:03) Yeah. Well, I met my wife that way so at least one example of coming out the other end. So yeah, that's super interesting. I mean that's kind of led into the next question a bit about unusual metrics and so just thinking about what you just said, it does cause you to change your perspective if you're optimizing for, you know, not a human anymore. Marco Casalaina (14:11) Did you? Alright, well... haha Ashley Stirrup (14:32) And as a marketer, I'm starting to do more and more of that so that you know When people ask questions of chat GPT or whatever chat bot they get a good answer I guess it's the same thing for you and then you know, you're ideally trying to minimize the amount of time they're spending there and you know maximize their token usage and hopefully the things they're creating on the other end Marco Casalaina (14:57) All right. mean, and then the challenge, think, with this market is also that some of the metrics that we use become outdated very quickly. I consider, for example, the original GitHub Copilot. I mean, when GitHub Copilot first came out, and at the time it was pretty impressive, you know, back in 2022 or 23, that it would complete your code for you. And the metric that we used at that point was accepted completions because it would complete your code for you. And you could either accept that or you could say, no, that's not right and reject it. Now in the copilot CLI era though, that doesn't really exactly apply anymore because the fact is, so for example, like two Fridays ago, I wrote a pretty chunky project in a morning. mean, this is a thing that would have taken me weeks to do. It was like a whole voice AI thing with multiple threads and all kinds of crazy stuff. And I mean, I kind of accepted all the code because really what the way that I rigged it was that I set ⁓ Copilot CLI to build it and then I also gave it ⁓ in this case I was using Vercell agent browser to verify it so I said okay after you finish writing the code go try it and See that it works and whatever doesn't work go fix it and go back and forth and back and forth and back and forth This is sometimes referred to as a Ralph loop But you know acceptance rate then Doesn't really mean anything anymore because I'm no longer sitting here accepting each individual little code change It is writing huge swaths of code ⁓ without my moderation per se and it's kind of going back and forth with itself. So those kinds of metrics which were super useful to us just three years ago are no longer the thing. Ashley Stirrup (16:35) Yeah, I've heard that the Ralph loop is a huge unlock. where basically you're only getting the output after the LLM has verified that it's actually completed its task. And so that allows you to check in a lot more code, much more comfortable. Marco Casalaina (16:53) That's right. And you also have to rig it in such a way that it can check its work. And so in my case, because I was building a browser application, that was relatively straightforward because there are things like Playwright MCP and Vercell Agent Browser, which I did use to help it do that, to have it automate a browser and kind of walk through it and that sort of thing. So that is part of the challenge, I think, in building these things is allowing them to test themselves. But also there's the evals. I, you know, to go back to what I was talking about earlier, when you're building an AI assisted application, not just building it with AI, but many of us are now building AI applications that fundamentally use AI and that makes them fundamentally non-deterministic. So where you used to have unit tests, you you could measure is this thing working or is it not? It was true or false, you red or green? Not anymore. And eval doesn't work that way, you know, and eval. So for example, I mean, there's that big system I told you about that I was writing, it was an agent that I wrote. Well, at first my eval metrics did not come out so good. So I'm like, my task adherence was at like 45%. So not great. But I had had my copilot CLI, I'd set it up in such a way. said, okay, re-iterate on the agent, rerun the evals. Iterate on the agent, rerun the evals. And it did that all night long. And I went to sleep and at 3.35 in the morning, it hit my threshold and I set my threshold at 90%. And that's the funny thing about evals is that as I said, they You're not going to hit 100 % most of the time. That's very rare. So you kind of have to pick a threshold and say, OK, this point is good. And in this particular case, I deemed that, somewhat arbitrarily, that 90 % was a good enough rate for this particular job. So yeah, mean, that's a weird dichotomy that I think AI developers have to deal with is that it's generally not going to be 100 % perfect ever. Ashley Stirrup (18:20) Mm-hmm. Yes. Wow, that's so fascinating. So it sounds like you're not just like tweaking the prompt, but kind of tweaking the whole design of the app you were building. Marco Casalaina (18:53) Sure, yeah. mean, was, I had made a pretty detailed plan upfront. Usually I use a tool like SpecKit ⁓ to build a pretty detailed plan that is documented, that's in the repo so that it can kind of follow that as a guideline, as an architectural guideline. But then, yeah, I mean, the iteration part is relatively new where you can kind of have these things self-verify. That's something that we didn't really do six months ago that now is quite common. Ashley Stirrup (18:57) Mm-hmm. Mm-hmm. Yeah. Yeah. Yeah. So interesting. Last week, Kelly Hall, a data scientist from Khan Academy presented about how they were, they started building on GPT-4, an AI tutor, even before it came out. So they had pre-release access to it. And they talked about the challenges of kind of building on top of a model that's moving and then they're tweaking their prompts and there's the eval stage, but then you really need to test it on real users. And I just imagine this whole approach has got to be constantly changing as the models get better and better and the ability to self-test gets better. Any advice for people on like how they, if they were starting today, let's say an AI tutor just to pick something like, Like how do you think about like how much testing do you really need to do? There's the eval side of it, which is kind of like QA, right? And then there's the, if it's a tutor, is it actually being a good tutor? Are students actually learning? know, that's a whole nother bar. And so like, how do you think about that kind of use case? Marco Casalaina (20:19) Right? Yeah. Well, I mean, you really do need to think about your evaluation thresholds, as I mentioned earlier, and what is your acceptable rate of error. For example, my daughter and I use CHAT GBT. She's a sophomore in high school, and she's in the honors chemistry class, which I took 30 years ago. And so my memory of it is a little bit not great. And so I had to look stuff up with CHAT GBT to help her answer some of these questions, because we work on her homework and study for tests together. And one day, it really only was. It gave me an egregiously wrong response where I kind of eyeballed it. was like, I don't know about that, man. That doesn't look right to me. And then it reversed it. You know, it was like, come on, ChadGVT, you're killing me here. But you know, one out of maybe a hundred times it does that. Is that acceptable in this instance? I guess it probably is here, right? You know, it depends on to what degree you have human mitigation available. Ashley Stirrup (21:05) Yeah. That's funny. Yeah. Yeah. Yeah. Yeah. Yeah, it really depends on your use case too. So I would say that, you know, like it's a finance app that the bar is pretty high. I know that Khan Academy, their bar is pretty high because it's, you know, teenagers and younger. And the last thing you want is either giving them bad information or even just giving them the answer, you know, because they're supposed to be encouraging them to think and helping them to think through the problem, not give them Marco Casalaina (21:23) Yeah. Ashley Stirrup (21:49) So I know they take all those things very seriously. Marco Casalaina (21:49) Right, but I mean, where we're going now, as I said earlier, mean, we are now kind of entering this era of what I call them open world agents, agents that are capable of doing whatever. You know, so you think about, I'm building a tutoring agent and all it does is tutoring. Well, but you know, what if you ask it, okay, ⁓ give me a Word document or a Google doc with a sample test on here, which is something that I actually often do with Chat GPT for my daughters like math or chemistry. Ashley Stirrup (22:15) Mm-hmm. Marco Casalaina (22:18) I'll make it generate me a document and print it out and give it to her as a sample test and then she'll try that and study with it. Well, I mean, what if you tell it, mind me, Bitcoin? So one of the things you do need to consider now as the agents become more more powerful, more capable of even working out solutions to things that previously they were just fundamentally unable to do. Now you have to consider Ashley Stirrup (22:27) Right. Marco Casalaina (22:40) out of domain evals. That is to say, mind me some Bitcoin and make sure it doesn't really do that. You know, we've had the notion of out of domain for a while and those were mostly like responsible AI sorts of things that is hateful or sexual or abusive or whatever language that you just didn't want the models to handle. But now there's another class of these things. Like I said, mind me some Bitcoin. There's nothing hateful or abusive about that. Like that's okay to say that. Perfectly safe for work. It's just not something that I... Ashley Stirrup (23:03) Right. Yeah. Marco Casalaina (23:07) want my agent to do unless I'm specifically making a Bitcoin mining agent, I guess. So you do have to consider that too, is that now you have to consider in what ways do you limit the capabilities of these otherwise unconstrained agents. Ashley Stirrup (23:12) Right. Yes. Yeah, yeah, it's a crazy world. We're almost out of time, so I'll ask you one last question. if you were talking to a friend who's either a product manager or an engineer and easy to get overwhelmed with everything we've just been talking about today and the rate of change, how would you advise them on how to keep up without getting overwhelmed? Marco Casalaina (23:47) Well, I actually did just have this conversation yesterday with a guy who's studying computer science at UCSC here in California. And he asked me just that question exactly. And I said, the thing you need to do now, as I've kind of alluded to in numerous points of this podcast, is master the art of evaluation. This is a little bit of a black art. There is no one way to do it. There's no one metric that's right. I mean, just as you know, Ashley, in your own business, There is no one metric that Growthbook pursues when you're doing things like A-B testing and stuff like that. It's a different metric for different of your customers, for different products and those sorts of things. And the same applies to AI. And today it's not super well understood. A lot of people don't really know how to do that. know, data scientists can do it in a sort of a technocratic way. or user researchers can do it in a sort of a sociological way. But it's still evolving. The tools are evolving, the methods are evolving, what we measure, how we measure it is evolving. And so if there's one way to really get your head around this stuff, mean, you really want to major in evals is what I said to him yesterday. But moreover, and more generally, mess with everything. You know, here on this computer I'm talking to you on, I have cloud code, copilot, CLI, have warp, I have chat GPT, I have every AI thing that you can think of on this computer. And I use them all for different things, Gemini, you know, I use them all. And I use them in my business life and when I can in my personal life. And I'm always looking for a reason to use them. Just the other day, for example, I went to go schedule a meeting and I said, you know what? I know that. Microsoft 365 Copilot now has the ability to do some scheduling. Let me try doing that and see what it can do. And it was able to do some stuff and not other things. It was able to create an event, but not delete an event, for example. But sometime in the future, we'll be able to delete an event also and have fuller control. So you want to try it and you want to keep trying it ⁓ as much as you can, because that's really the only way. You you don't learn to ride a horse by reading a book. So the only way you're going to do this is to actually get some first-hand experience with it. Ashley Stirrup (26:01) Yeah, and the thing I really liked about your example is finding use cases that you can apply it to your everyday life. Because otherwise it feels like another job on top of all of our busy lives. But you can find use cases that are a little bit of a stretch, but not too much of a stretch, right? Because otherwise you could go down the rabbit hole and spend a long time trying to build something that doesn't quite work out. But it can be. Marco Casalaina (26:24) And you know, mean, so one example of that recently, you know, my daughter was in the high school musical and her teacher sent out this flyer with all of the rehearsals on it. There were like 45 of them, I think. So I scanned in the flyer and I gave it to Gemini and Chachi Petit and Claude. And what I wanted it to do is add it to our family calendar. We keep a Google calendar, which is our family calendar. So we don't make conflicts and stuff like that. Well, Gemini didn't do a great job at all. Did nothing at all. ChatCVT was able to read it, but it also hallucinated a bunch of rehearsals that weren't there. Again, this was in December, so that was two models ago. ⁓ Claude did actually a really good job. It hallucinated like five of them, but other than that, I was able to point those out and like, those don't exist. Other than that, go. Gave me a big ICS file that I was able to import to the calendar and we were off to the races. Now there will come a time in the future when that'll be a lot easier. I will scan the thing in and I'll say, put this in the calendar. Ashley Stirrup (26:58) Yeah. Yeah. Hmm. Interesting. Yeah. Marco Casalaina (27:20) and it will just do it and invite my wife and daughter and that'll be the end of that. So I wasn't able to do that yet, but I will try that again the next time that opportunity arises 100%. And I'll try it again with like all three of those things with chat, GPT, Gemini, Claude. Yeah, I mean, it's funny because they, you you say the definition of insanity is doing the same thing twice and expecting a different result, but in today's world, that's what you have to do. Ashley Stirrup (27:31) Yeah. Yeah, I love that. I love that. Because a month later, it might actually work. Well, thank you so much. This has been just a fascinating conversation. I already knew how much the AI world is moving quickly, but I think you provided us with a real window into it from somebody who's right there at the core of it. So I really appreciate your time today. Thank you so much. Marco Casalaina (28:02) Thank you for having me.