Latest / The MonkCast / Why AI Agents Are Blowing Up Your Telemetry Bill with Nikhil Mungel
Transcript
- speaker-0: Hello and welcome to this Red Monk Conversation. My name is Kate Holterhoff. I'm a senior analyst at Red Monk. And with me today is Nikhil Mangel, the head of AI R D at Cribble. Nikhil, thank you so much for joining me here on the Monk Hat. Awesome. All right. So let's just start off with some introductions here. Tell me about yourself and what you do with Cribble. speaker-1: Excited to be here. Thanks for having me, Kate. Sure. So at Cribble I have ⁓ the very fun job of leading all AI research and development. So I ⁓ I lead engineering management as well as product management ⁓ when it comes to AI research at Cribble. So I have a few teams that I support which ⁓ are building agentic products that simplify the lives of IT practitioners and security practitioners. And then ⁓ I also lead a research lab team where we train our own models that can work on really high throughput streaming telemetry data. I that's what Triple does as a company. ⁓ that's what we specialize in. So yeah, we'd love to share more about what we do and, you know, where ⁓ where the industry is going with ⁓ AI as it applies to telemetry and observability and security use cases today. speaker-0: Okay, so we are definitely gonna be digging into that. I I figured we kind of start this conversation around some of the things that you've written. So I am specifically excited to have ⁓ someone who's an expert on like FinOps ⁓ in the agent space on the show because ⁓ I mean I I I think token spend we can all agree is top of mind. It's a become a real pain point for a lot of folks. ⁓ and you've written about this, ⁓ in the in the context of the first agent that your team has shipped. So can you walk us through what happened there and and what you actually saw in terms of That sort of larger question for our conversation of of ⁓ telemetry. speaker-1: Sure. Yeah. So ⁓ it's ⁓ I mean, yes, tokens and token costs are a pretty hot button topic, you know, right now in August of twenty twenty six. ⁓ it's token costs also like you know keep dropping compared to a few years ago. But then token use is going up quite a bit. I mean, models today and use cases that are powered by, you know, LLMs today are much more complex. They're much more longer running, and they tend to consume a lot more tokens than some of the some of the use cases that we used to have, you know, back in twenty twenty two and twenty twenty three. So as an aggregate, companies are spending a lot more on tokens today than they were a couple of years ago. ⁓ and I think it's pretty valuable. I mean, I feel like tokens are adding tangible, tangible benefits. ⁓ we, you know, at at Cribble we lean a lot on so there are there are two ways in which we where we use Asians. One is baking Asians into the product itself. that helps our customers and our users a ⁓ a much more smoother experience when it's ⁓ using the product and when it's you know taking taking the value out of ⁓ w whatever crypto deployment is configured to do. And then on the other side we also use AgenTech software development and AGNT research for the stuff that we do internally as well. So when we are training new models, we have a research agent that helps us And when we are writing code, obviously we use Agenc coding tools like plot code and cursor and ⁓ OpenAI Codex as well. So I would love to would love to share a ⁓ a personal experience where we had this runaway agent that accrued a massive token bill. ⁓ and I'm pretty sure a lot of listeners are gonna have to. ⁓ they're they're they're they've they've been in similar situations, I feel like. speaker-0: Yeah, let's let's hear about it. speaker-1: Yeah, yeah. So one of ⁓ one of one of the teams that I support is our AI research team where we, you know, train we have a little ⁓ GPU setup here in San Francisco where I'm talking to you today from, and we run ⁓ experimentation and we run research experimentation where we train new models that can operate on high throughput telemetry data. Now these models are a little different from L LMs, since LLMs can ⁓ or you know, AI models, multimodal AI models can ⁓ intelligently come up with ⁓ answers to the questions you're asking, et cetera. In the world of telemetry, it's a little bit different because you're dealing with a lot of really high throughput streaming logs. So you have terabytes of logs that are being processed by Kribble every single hour for every customer. Sometimes it's tens of terabytes every single hour. So using LM like Clot or, you know, using a model like Clot or using a model like GPD five six is not really feasible for those sort of ⁓ volumes of data that we deal with. So we design our own models to do that. And we have a research agent that runs in a loop overnight. So we run, you know, we ask it to run several experiments and then you know it's able to go and it's able to ⁓ run these models, you know, run run based on an objective function and you know gauge the loss ⁓ overnight and then in the morning when the researchers come in, they're able to see ⁓ the progress that was made overnight. So The objective function of an agent is really, or when it's running in a loop, is a goal that you give it, and then it optimizes for that goal and you know uses that goal to really train one of these models that we have. So a few months ago, or a couple of months ago, we had ⁓ this Asian that was going off and you know, training a model, and its objective function was to b to keep thinking more, right? I mean the way somebody had written that experiment was it's like the more you think you know, the better the output is gonna be. So the Asian just every call it made to an LLM, ⁓ was like, ⁓ cool, like this is this is great. You know, I'm gonna do more of this. And in the morning when the researcher logged on, we found out that it had accrued an enormous ⁓ token bill that I'm not gonna share ⁓ the exact number, but it was quite a bit. And ⁓ that was there was a big shock. And ⁓ ever since then we've been a little bit more thoughtful about, you know, how we structure these experiments and also put in some guardrails in terms of how ⁓ how many tokens a single experiment can consume before it ⁓ raises or you know taps a human researcher on the shoulder and asks it whether it should continue. speaker-0: Wow. Okay. Such a good example of why guardrails are becoming more and more important today. ⁓ all right. And so, you know, you you're this is just one example I suspect of many. I I'd love to hear ⁓ some of the trends that you're seeing around this, I guess. So, like, okay, are companies that you're speaking with using AI and agents to address ⁓ observability ⁓ in in practice, you know, like how is it that AI has shifted the way that you've approached this ⁓ more recently in in our sort of agentic present speaker-1: Yeah, I think so the way I think about this is in two parts. First, there is a lot more software today than there was like two years ago, right? Because of agent decoding and because how ⁓ approachable and easy agentic coding tools have made for teams to produce software. Teams have been producing a lot more software ⁓ you know, in the last ⁓ in the last few years than they had historically done. I mean a a team of five can now ⁓ build extremely complex systems, whereas before before agentec coding took off, it would probably take you, you know, fifty engineers or fifty developers to do the same thing. And because of that, because of the growth in the amount of software that has been put out there, the operational footprint has also grown sufficiently complex. So in a way, you don't have an you don't have an option but to rely on agentec SRE tools, agentec observability tools. to really understand how your application and how your software is behaving in production. So one one one example that comes to mind is how so if you if you have any software system that has agents within it to you know deliver user functionality and user value, these agents are running ⁓ many times in a in in in a window of time compared to a human operator or a human SRE would have done. And they're producing a lot more they're producing a lot more data. They're hitting the microservices or you know any any server side compute that you may have a lot more. And it's just creating a massive increase in telemetry that we have been seeing across the industry. And in order to make sense of all this telemetry, you need to ingest it, you need to store it, you know, you need to do all all the best practices that you have have been doing so far. And ⁓ in order to reduce the burden on your operational teams, your SRE teams, your IT ops teams, and your DevOps team and your DevSecOps teams, you have to lean on agentec ⁓ tools to, you know, take away some of that burden from the other side of the house as well. speaker-0: Right. ⁓ yeah, we've been tracking the amount of code that has been published recently and ⁓ it's staggering. ⁓ okay, so you you've also written a little bit about agent access, ⁓ specifically and arguing around ⁓ cost ⁓ being a reliability metric. So we've seen more attention on token spend. ⁓ I'm I'm thinking about the Linux ⁓ foundations. They just launched their tokenomics foundation, for instance. ⁓ I am not hearing panic yet, but from your perspective, is this becoming a bigger deal for folks like CFOs? speaker-1: Yeah, I I think I think it is. And if you if you again split ⁓ split split token expenses between what your internal employee workforce is using to drive productivity and you the first thing you want to separate that out from the token spend that your SaaS is driving to deliver value to your customers, right? So if you take an average software engineering company or a SaaS company, you have the SaaS that it produces Which is sold to customers and you know it's a usually a high margin business. And then you also have developers and product managers and employees and staff who are using tools like Cloud Code to deliver and build that SaaS. There's been a lot of focus on like the internal token usage right now, because that's that's something that you have. There's typically no bound on it, or you there's a very high bound on how many tokens a an individual employee can use and, you know, ⁓ to deliver value. So that's that's one that's one part of the equation that we can get into. ⁓ but the second part is your cogs or you cost of good sold that your SaaS is accumulating when you're delivering value to customers. What's one one unique thing that you know we have maybe it's not even that unique, but one thing that we have seen happen in the last couple of years is customers have demanded that vendors support agentic flows within their software. Nobody wants to use Software one point ⁓ where you have to manually configure software and like go through like six or seven different configuration screens and like look at docs and stuff like that, right? Everyone wants an easy button to set up whatever it is that they're doing, right? Maybe it's creating a ticket on Jira, maybe it's, you know, setting up an observability pipeline and cribble, whatever it is that we want to do. Everyone wants an easy button and a genetic flow. Have are you are you have you have you seen that as well? I speaker-0: It's funny. I was just ⁓ talking about click ops today, ⁓ and and sort of how it is that agents can best ⁓ use these infrastructure tools. ⁓ is it that they are it's just through like a Terraform something, or is is it that everything's gonna be through API keys now? Or ⁓ you know, I think it was like nine months ago everybody was talking about like, ⁓ I I need ⁓ an agent to buy me a plane ticket. I'm gonna give it access to my browser. So I I do feel like there is this ⁓ It's kind of an open question, ⁓ right now, but I I absolutely am am seeing it as like something that we're we're grappling with. I I I guess I just I am not sure where it's landing at this point. speaker-1: percent, right? And when you when you really try to unbundle, you know, how much tokens your SaaS is consuming to deliver value to your customers, that's when things start getting real really tricky, right? And that's the stuff that CFOs care a lot about is like, hey, are you running a high margin business or are you running a low margin business? Really that's where it comes down to. And when you when you look at when you look at tools out in the or SaaS products or you know just any products out there in the market today, I've seen like two two ⁓ types of billing models that seem to have been ⁓ catching on. One is some sort of a crowd-based model. So if you look at something like Notion, you know, you purchase AI credits that then you can use for generating, you know, documents, summarizing documents. So every agenda flow that you use within the product consumes some sort of credits. And that's one way to ⁓ pass on some of those costs back to your customers and not end up in a low margin business. And then on the second hand, I've also seen tools Offering a more outcome based ⁓ billing model. So, you know, if you have an agentic SRE tool, you may charge customers based on the number of incidents it closes or you know, the the how quickly it is able to close an incident or an investigation, regardless of the token use, right? So in that case, companies or or like SAS vendors are now incentivized to drive the same outcome using as few tokens as possible, or using a smaller model instead of a frontier model. for you know some of those tasks. So there's a bunch of a lot of a lot of ⁓ r R D these days is happening around how do you drive down marginal costs for delivering value to customers. Smaller models have also gotten really better at a lot of tasks. So you don't really need the beefiest model for every task. Right. And there are so many different model architectures. And you're you you can have like an Oracle architecture where you have a smaller model that's running a task and then it consults with a much smarter model as it as it needs. And you can have ⁓ a pretty tangible ⁓ effect on token costs if you use one of these patterns to drive down costs when you're operating a SaaS business. speaker-0: That sense. ⁓ I guess I'd be interested in hearing how you're pivoting ⁓ these observations and and I guess just the way that you are handling research right now, ⁓ toward governance and and some of the things that, you know, a lot of the leadership are are concerned about around these these ⁓ subjects. So yeah, what is what's your general stance towards ⁓ governance ⁓ when it comes to agents? speaker-1: I think it's we are so early in the game right now. And government's just about to take off in a really big way. I mean it kinda reminds me of like the early cloud days, you know, like fifteen years ago or something where ⁓ companies were trying to rein in, you know, cloud usage and I I don't I don't remember which CFO it was, but somebody interviewed them and they're like, Hey, how do you use any cloud? Do you use do you have any cloud exposure? He's No, we don't use cloud at company and then they serve the the the employee population and they had like a hundred cloud accounts, right? So there's just a lot of shadow use. There's a lot of shadow ⁓ agent use, there's a lot of shadow MCP use, which is fine because a lot of businesses right now are trying to turn around and deliver an ROI on all this token spin that we just talked about earlier. So people are okay, enterprises have been okay to move a little fast and you know, maybe bend some certain rules somewhere to, you know, just lock in those productivity gains. But we've just slowly started to see the shift happening between let's move fast and drive value. Well, the value is there. You know, the values are for everyone to see. If you use AI, you're you're more productive than if you're not using AI. So it's fast, it's quickly moving to a place where governance is becoming super important. And though the the successes that I've seen with good governance, it just starts with creating a good policy, right? I mean, you can't govern if you don't have policy. So successful Enterprises or enterprises that been successful in governance have been successful establishing a policy that makes sense for for for for the employee workforce. And it's best done at the department level. You know, like the the governance policy for an IT team is going to be different from that for an engineering team. It's going to be different for a marketing team. But having that policy is like the first step. And once you have that policy, then you know, there are there's there are a few SaaS tools available for you to established governance where you can, you know, work with a variety of vendors to ⁓ get that get that out there. And there are many strategies to govern as well. Happy to go into those. But that's where I'm seeing the shift right now. You know, there's it's governance not only from a and so w what what do what do most of these policies include? You know, there's a cost component. So like in how much are we allowed to spend, there's an access component, like what sort of data you're allowed to access. And then there is a ⁓ a what I like to call a blast radius where, you know, if you're using an agentic software, an agentic tool in a production system that has a much larger blast radius, right? If something goes wrong, it might take down a lot of your production instances, like it might delete data from databases and, you know, might suffer data loss, et cetera. Whereas if you're running an agentic coding assistant locally on your computer, the blast radius is limited, right? To just your local machine. Right, you're not gonna be deleting data for ⁓ that you have stored for your customers. So different policy makes sense in both those different settings. speaker-0: Right. I'm glad you brought up the locally running models too, because you had mentioned small models, but I I I feel like what we're hearing a lot of is this interest in having these models, yeah, just running off of ⁓ you know, private machines. I mean, this is this has always been a a a big deal for especially highly regulated ⁓ industries, ⁓ ones that have historically needed air gapping and things like that. So, and you know, frankly the security issues right now are ⁓ you know, deeply unsettled and unsettling. So ⁓ yeah, so ⁓ th this is ⁓ you know, certainly aligning really well with with some of the things that we've been following. and ⁓ so I I guess what I'd want to hear more about then is is how observability and telemetry ⁓ come into that. Like what is I guess you know, what is your stance right now in in terms of like how you're handling this at Kribble? speaker-1: Sure. We so we we help customers capture, you know, your LLM traces, your ⁓ your your token routing traces, you know, your how how your endpoints are engaging if there are like shadow MCPs running on your computer, how your cloud security posture exists, right? If you have like configured a bunch of resources on ⁓ a hyperscaler like AWS or G C P. So we do all those sort of things and you know collect all of that data for you and then The beauty is that we are able to route that data to whatever destination you choose. So it could be a sim that you already have. It could be a ⁓ a a just an S3 bucket that you just want to store data in and you know, look at it for audit auditability purposes or investigative purposes a few years down the line. ⁓ that's where that's where we've really ⁓ seen a lot of ⁓ observability and telemetry use cases come to light. And one of the one of the products that my team launched last year, end of last year, is called Cribble Guard, which enables you to redact, scan and redact sensitive information from your telemetry data, which is kind of pretty ⁓ relevant these days, right? I mean, like so do you remember when back when you know we had like the first Chat GPT moment three or four years ago at this point, everyone was very hesitant about the sort of information you should be putting into Chat GPT ⁓ or Claude. And now everyone is pouring their innermost thoughts in chat bots. gets logged, you know, like it it gets logged, it gets exposed in compliance APIs, especially if you're using a chat subscription that's provided by your employer, then it's 100% getting logged. If you're directly purchasing it yourself, then maybe it's not getting logged. ⁓ but one of the cool things is that when it does get logged, you don't wanna you don't want that personal private information to be stored anywhere. So you can use a product like Cripple Guard to scrub it out and remove sensitive information from your telemetry logs, right? So if somebody puts in their social security number or, you know, some private health information in there, guard can be used to scrub it out. I will say though, it's not going to scrub out your innermost thoughts. So those will still be there in text. So, you know, you should be careful about what you put in ⁓ when it when you're using an employee sponsored chatbot product. But ⁓ that's those are some of the use cases that we have seen. speaker-0: Okay. And some of the ways that I have heard you frame this, ⁓ in in your writing around the subject has been with guardrails. ⁓ and I feel like we've kind of alluded to this already here. So ⁓ talk to me about the role of guardrails in all of this. Like what what does that look like? speaker-1: Yeah, so guardrails really help you enforce policy, right? So you want to enforce somewhere in the chain. And there's multiple places where you can do it, right? You can do it at like the token gateway routing. So ⁓ and just to just for just for clarity, a token a token gateway or an AI gateway is a single choke point that can be deployed into an enterprise or you can use an open product like open router. And what do you want to do is you want to route all AI usage through that gateway. So that gateway is or the right model, you know, of the right intelligence based on the task. It can fall back between multiple model providers. So if one model provider is not available, it can go to the next model provider. And most importantly, it can enforce policy. It can enforce these guardrails. So if a particular agent or a system or a person is using more tokens than they should be using, the gateway can enforce that policy and start relimiting them, or maybe degrade them to a lesser capable model, you know, if they've used their quota for a more intelligent frontier model. So it can do all kinds of interesting things, right? Diff depending again, based on the policy that you want to enforce governance around and guardrails around. and then when it comes to ⁓ privacy and you know deep sensitive data, which is something something that I spend a lot of time thinking about, guardrails enable you to prevent sensitive information from getting into places where you don't want it to get into. Because if it does escape something, ⁓ or if it does escape into or g leak into, you know, destinations or, you know, surfaces where it's not supposed to, then you risk being out of compliance. You might be, you know, moving out of ⁓ or you might be in violation of a SOC two compliance or, you know, a an ISO compliance or a FedRAM compliance control, right? So you have these compliance controls and you want to make sure that you're adhering to those compliance controls. And that's where guardrails are really, really important. Something else I'll add to that is Software one point zero, which is you know what I like to call software that was ⁓ all the systems and enterprise tools that people used before the advent of modern agentic coding and ⁓ agent development, used to just have a bunch of buttons and you know, ⁓ radio buttons and check boxes and you know, a few fields, et cetera, where it was a very very much an on rails experience. Software today has a lot of free form text inputs, right, like a prompt box. Where you just go in and can just describe, ⁓ I'm looking for so and so information and I'm looking for ⁓ y you might go into your business analysis system and you can ask questions like, Hey, show me ⁓ why did this particular customer start using more of my product or you know, why did this customer churn and those sort of things. And even that text that you input into these boxes tends to have extremely sensitive information, which now you have to protect, right? Because it all gets locked, it you know, gets it gets into telemetry data, right? It it it ends up in anything that your users type into a prompt box in your product is going to end up in a telemetry pipeline somewhere. And if it goes in the wrong destination, then you're gonna be in violation of a compliance control. And that's why guardrails have become even more important in HNTIC in the in the world of Asians. speaker-0: And in terms of cost, we've talked about tokenomics here, but it also feels like this could be a solution around telemetry cost as well. So is there a way that gateways are also ⁓ serving a similar function there? speaker-1: Yeah, a hundred percent. I mean gateways do, you know, all kinds of really clever things these days. ⁓ you can't ⁓ one of the one of the cool things that I've seen gateways do is obviously enforce rate limits. So you can only consume you everyone has a quota and that quota can be on an individual basis, it can be on a team basis, it can be on department basis, right? And that's something that ⁓ you know usually the CFO or the CIO will determine what what makes the most sense in terms of an investment standpoint. And then totally an interesting phenomenon when you give your users the ability to pick a model, your user is always going to pick the most expensive model, right? Regardless of the task that you're doing. Because you everyone feels that the thing that they're working on is, you know, the most important thing in the world. And who wants to use a less capable model, right? So you're gonna pick the the the best one. And I like to call it model anxiety, right? Where you're you whether you You're just like always with the most expensive model and accruing token costs. And ⁓ really good token gateway products out there are able to abstract that away from your users. So the token gateway will be responsible for gauging how complex a query is and routing it to the the model that's ⁓ the smallest model that's capable of addressing that query. Removing that burden the decision burden from users to select a model. Right. So you just you just you just fire off a query and cursor, for example, does this really well when you use cursor for writing code. They launched a feature recently which is called AutoRouter, where you don't you don't have to pick a model, right? It'll do it for you and you can just focus on the task that you have in front of you. And that is something that I've seen drive down costs quite a bit on the consuming side. speaker-0: Yeah, I agree that that's where the future is heading. ⁓ I think having the option of choosing your own model can be valuable, but ⁓ ultimately who wants to make that decision for every little thing? ⁓ and then I kick myself when I'm using Claude and I accidentally use Fable or something for like a a dumb little query, like 'cause it was just already that's what I had it set up for. Yeah. All right. So I I wanna get back in this security issue, which I I you know, I feel like we've kind of been circling around anyways. I mean, and so much rest on that. ⁓ and so you've been involved with this organization called the Open Cybersecurity Schema Framework. Can you talk about that? I I you I I I learned about it from ⁓ some of your writing on the subject and ⁓ I I thought it was fascinating. So yeah, what is that? speaker-1: Yeah, OCSF is a or you know, it's it's called OCSF and it's a format for capturing and storing telemetry data so that it can be used by security products like a sim product or ⁓ you know, like a number of security products. And it was it was a it was a standard that was born ⁓ out of ⁓ a few companies like Amazon and Splunk. ⁓ it's only like three or four years old. And The adoption of OCSF has been phenomenal because it's such a robust such a robust format. So yeah, I've been I've been ⁓ lucky to be involved with the OCSF community. I've you know, I've ⁓ we had a a dinner at RSA ⁓ just a few months ago and you know there's a very welcoming community, you know, if people are interested in contributing to OCSF or, you know, being a part of the ⁓ being a part of the group that, you know, develops that standard, you know, there's a call of Tuesday mornings ⁓ that you know, I occasionally will dial it too. But yeah, it's a it's a it's a format that standardizes logs, right? Or telemetry information, if I had to put it ⁓ put it in a simple way. So you have logs and telemetry information that's coming in from microservices, you have it coming in from endpoints, like my computer, for example, you have it coming from networking equipment. And every appliance, every equipment, every service, every application has a slightly different way of emitting telemetry data, right? And OCF Sanitizes that. It sanitizes that into like dozens of different ⁓ types and ⁓ it there's a ⁓ a structure that it brings to the process. So when you store data or when you convert your logs and your telemetry data to OCSF, it's it all looks the same. Like when you're looking at firewall logs, it doesn't matter if they came from a Palo Alto appliance or if they came from a Cisco appliance, they're gonna be structured in the exact same way. Which has a bunch of Downstream, right? I mean, if data is structured in a particular way, it can be stored in a way that's really fast to retrieve it. You know, you can index it, you can optimize storage, you can reduce duplication. There's a lot of interesting things that you can do from an engineering standpoint when you're when you're handed structured data. And again, the the reason why this is really important is the sheer volume of data that we deal with when it comes to telemetry, right? I mean, enterprises are creating petabytes of telemetry every single day. And then when there's an incident, when there's an investigation, you want to be able to pull insights from petabytes of telemetry pretty much instantaneously, right? Because there's an operator, you know, you're you're you're you you got paged, maybe there's a security incident and you're going into whatever tool you're using and you're like, well, show me what happened. And the system now has to comb through petabytes of information to give you that answer, right? I mean, if you think about it, one way that I like to explain observability and telemetry to people is It's an information compression problem, right? You're trying to go from petabytes of data to like a kilobyte of data that can be presented to a human being, right? You're you're trying to go from all these logs into like a sentence. Like tell me what happened. Like, you know, which service is failing right now, or you know, where's the intrusion? ⁓ and in order to do that, you need all these optimizations that OCSF brings you or OCSF brings to the game and ⁓ you know. storage systems can then take advantage of the structure to save that data and store that data and index it and you know pre-process that data so that it's fast to retrieve that data. ⁓ I'll I'll put in the plug by saying one of my teams has created this awesome product called Copilot Editor, which can be used to transform data into OCSF. So you can create if you have high throughput streaming data that you want to convert into an OCSF format, you know, be the latest version or you know even we support like several versions. it will automatically convert that data into O C S F for you. And it's an open format, right? It's in open cybersecurity format. So you can ⁓ convert it to O C S F and then you can send it to whatever destination of your choice and then you can take ⁓ advantage of what O C S the O C S F ecosystem brings to the game. speaker-0: Yeah. I mean that just makes so much sense. speaker-1: So I can't stop talking about it. speaker-0: No, I I w I actually want to hear a little bit more about it. ⁓ I and so am I right in assuming then that because of AI and because we have just all of these systems, that the the the need has always been there, but it is so important now that the community's finally like, Okay, we need to do this because yeah, f ⁓ what, three years ago that was really when these things kicked off. ⁓ is that accurate or w ⁓ has it just been brewing for for so long it was gonna happen eventually? speaker-1: Yeah, it was it's been brewing for a while. I mean OCSF ⁓ the the forces that led to the creation of OCSF have been around even before Agentic AI took over. Yeah it it supports Agentic traces, you it supports LLM traces, it supports Agentic observability. So it's ⁓ I think there's a new OCSF version that comes out every few months and the team does a fantastic job of keeping up that standard and the vendors that support OCSF do a fantastic job of adhering to that standard. So if you're, you know, sending your data to Amazon Security Lake or, you know, whatever sim that you're sending to, they're ⁓ they're usually pretty up to date on stuff. speaker-0: Huh, my goodness. How many people are are working on it right now? I mean, this sounds like it must be a very robust community. speaker-1: Yeah, I mean I feel like the community is ⁓ it's a handful of people. I mean like six or seven core committee members. I'm not a core committee member, but ⁓ I know several of the core committee members. And they work you know, all some of them work at di all of them work at different vendors, which is really what makes it a pretty well adopted standard. speaker-0: Hmm. Mm-hmm. Okay. Cool. Well, I appreciate you ⁓ talking about that. We'll be sure to put a link ⁓ in the show notes for folks who want who want to learn more about that ⁓ particular ⁓ what community here because ⁓ I you something. So yeah, I l I I feel like we could still dig in a little bit deeper to this like agent telemetry question here. So how about ⁓ the issue of retention? Like ⁓ in regulated industries, that that's a huge ⁓ Issue I the cost of it can be ⁓ pretty daunting here. So ⁓ I guess one one aspect of that is like should an agent be fed its own cost as a as like a planning input? I you know, there's there's a couple of ways to go with this, but yeah, what's what's your sense around that? speaker-1: I think so th when it comes to data retention, one of the hottest topics has been ZDR or zero data retention, right? Which is ⁓ which which really means that whoever is giving you or whoever is ⁓ fulfilling your inference is not should not be storing a copy, you know, off the inputs and the outputs that came from the model. Which in Theory makes a lot of sense. You know, if you're if you're an enterprise, you don't want to ⁓ you don't want your LLM vendor to store like a copy of your data. But then there's also the issue of safety as it pertains to LLM. So this is something that I'm actually pretty ⁓ you know, I have I have a lot of opinions on this, which is, well, you you can I mean, these are these are really exceptionally intelligent models and they have t you have to have safeguards around these models. And in order to enforce those safeguards, even as a model provider, if you if you want to enforce those safeguards, you have to store a copy of that, you know, the inputs and the outputs for a period of time to scan those things. That's one of the reasons. So I understand why ⁓ AI labs and inference providers have a policy where they are required in many cases to store a copy of that data ⁓ and you know, use it for scanning and make sure that nobody's abusing the system or nobody's using it for purposes that they should not be using it like you know, creating new vulnerabilities in software or cyber offensive work or, you know, whatever. You're just red teaming the model and you're trying to subvert the safeguards that have been encoded into the model weights. And on the other hand, enterprises also have a fair point. Like they don't want their ⁓ their corporate information being stored, even if it's just for a day or two by an Amazon or by ⁓ you know an AI lab or, you know, anyone else. So it's a pretty both both both the sides have a good ⁓ a good balanced argument on that front. But your your your you also mentioned, you know, Asian costs and providing ⁓ a budget to agents in the planning phase itself. And that's a pretty, you know, popular pattern. Like, you know, agents have a tool budget or, you know, a some sort of a budget to ⁓ that they have to remain or they have to follow in order to accomplish their objective. And If you do follow that pattern, that does result in slightly more optimal performance. So yeah, I think it's a pretty common pattern that that we're seeing across the industry, ⁓ or across across, you know, most of the ⁓ agent frameworks or frameworks that are used to create agents. Well speaker-0: ⁓ I feel like we have got a lot to to chew on here. You've given us ⁓ I yeah, a lot of ⁓ interesting ways of of thinking through this sort of broader idea around how telemetry is changing, how we should approach ⁓ this issue of what observability writ large ⁓ in in this ⁓ agentic era. So ⁓ so thank you for that. ⁓ before I I wrap us up though entirely, ⁓ where are you directing folks ⁓ for more information, both around how Kribble is helping with this and ⁓ For your own writing about this, which of course I've ⁓ cited a few times. speaker-1: Yeah, so we have a ⁓ we have a new research portal that's gonna be coming up on cribble.io. So we should post a link on that in the show notes. And then that it that would be the best place to, you know, get people plugged into what's happening as it you know, as it ⁓ relates to research and development in AI, at the intersection of AI and telemetry. That's how I frame it. Right. So if you want to keep up, like I would I would direct people there and then I obviously have a LinkedIn and a Twitter profile as well. speaker-0: Well, on that note then let's go ahead and wrap up. ⁓ I've really enjoyed speaking with you today. My guest has been Nikhail. ⁓ again, my name is Kate Holterhoff. I'm a senior analyst at Redvont. If you enjoyed today's conversation, please like, subscribe, and engage with ⁓ in