Latest / Unstuck Pod / Measure What Your Agents Do (Peter Holcomb and Jamie Schneiderman)
Transcript
- Jamie: Peter, welcome. How are you today? Peter Holcomb: Hey Jamie, I'm doing really well, thank you. How are you doing? Jamie: I'm great, thanks. Glad you could join. So let's dive in. So for the sake of everyone who's watching, listening, please introduce yourself ⁓ what you do? Tell us the business and let's go there. Peter Holcomb: Sure. Thank you for having me on the pod. Looking forward to the conversation. My name is Peter Holcomb. I'm the founder and CEO of Optimo IT. We are an AI engineering and security consulting company, predominantly focused on helping companies engineer AI solutions and build AI agents for their business. We also provide fractional chief information security officer services and audit readiness for companies that want to get ⁓ certified with different industry regulations, things like HIPAA, NIST, ISO, PCI, GDPR. side of the business is also the AI SaaS app that we've developed. ⁓ And kind of comes from some of the learning pains that we've seen in the field. And that app really is around helping us understand the AI agent monetization and spend and consumption usage ⁓ of when workflows get executed. ⁓ We can talk a little bit more about that, but that's sort of where I'm at right now. Jamie: Very cool. I I love it. And so just before, because we jumped in, How long has Optimo been around? Peter Holcomb: Sure. Optimo has been around for about three years. I started the business in September of 2023. It was initially sort of a side business and just helping other business owners with some of the fractional executive oversight work. And then last year we went full time on this thing and just completely went all in and have been going at it since then. you know, it was... kind of a learning curve to figure out where do we want to go in terms of what services do we want to offer? And what we learned is while we initially started out as a fractional CISO business or a VC so business as it's known as otherwise, there were a lot of interests to do AI engineering work. So given that we come from the engineering world and have that background and expertise, we started helping businesses develop AI engineering systems and then also help them start building AI agents into their business processes. So this could be to help solve existing business processes that they're stuck with or that they feel is mundane or repetitive. So they want to figure out how can they automate or augment that, that rudimentary work away and free themselves up to do more strategy work. Or could be, hey we're developing this vibe coded app that we initially started on, but don't know where to take it from here. Can you guys help us? And so we'll come in and do some more of that solutionizing, if you will, from an architecture and design perspective, and then get it to be in a real good stance so that it could be actually deployed into production. Jamie: So OK, so from a, and I'm going to ask, I want to understand the SAS product, what's interesting is from a consulting perspective, every project when you're describing these projects, they're all going to be a little bit different. I'm trying to solve this, or I'm trying to do this with AI. talk about how and where the SAS, do you have a name for the SAS product? Peter Holcomb: Yeah, the SaaS product name is useo.ai. useo is short for usage. really what helps us be able to bring this product to life is that we've experienced this problem internally and it's the reason why we even created it. But useo is really about making AI systems legible to the business by turning AI activity into measurable, governable, and billable outcomes so that intelligence can scale with trust. control and economic clarity. right now what we're seeing is of these existing software providers, that could be N8N, Lindy.ai, Make.com, Langchain, all of these providers are allowing you to build AI agents within their platforms, but the way they essentially meter how that workflow gets executed and what that means from a cost standpoint is all ⁓ based a credit economic system. That's fine, but it's very opaque in nature to understand exactly what's going on under the hood and what attributes to each credit spend. ⁓ so what we've done is build a ⁓ API that plugs into all the agent workflows and allows us to be able to capture the activity from the agents, the workflows, the APIs, the data pipelines, and then translates that behavior into standardized value metrics. We call them value metric units. So these could be things like usage, outcomes, time, volume over time. And by transferring that raw execution data into measurable business signals, UZIO essentially creates a layered usage ledger that makes these intelligent systems visible, accountable, and economically coherent. Jamie: Okay, awesome. So who's the customer of a product like that? So who's looking for that? What are they looking for? How do you sort of put it in front of people and say, here you go? I know how you it, but what about from a customer point of view? Peter Holcomb: Yeah, I- Sure. ⁓ From our ideal customer profile standpoint, we're really targeting AI native software companies, ⁓ startups and SaaS platforms building products with AI agents need to implement usage or outcome-based pricing models. They want to translate ⁓ raw AI activity from their various ⁓ workflows agents into business value and metrics to be able to support pricing, billing, and customer reporting needs. We're really targeting the mid-market segment. And then eventually we'll like to move into the enterprise organizations, but we're to start with the mid market. then key, key buyers would be CTOs, head of AI automation, CFOs, chief information security officers, and then CEOs as well that are looking to be potentially solo founders that want to leverage AI so that they can be a lean and mean sort of business entity with the use of automation and agents in their business. One of the benefits that we have Jamie is that In doing consulting with Optimo, the clients that we're building these agents for, we initially basically are saying to them, hey, would you like a means to be able to track agent spend usage and consumption? And so that's sort of the sticking point for us where we say we already have a tool that's essentially just an API that's going to monitor the usage and just the raw activity of your agents. And we can give you that fidelity information to say, Here's what it's doing under the hood. Here are the API calls that are being made or the pipelines processed. And here's sort of the realization that you can come to from a total cost of ownership to run an agent workflow one time, 50 times, a hundred times. And so that becomes compelling. Furthermore, as we move into this agenda future, if you think about having 50 agents running concurrently together, that's where the complexity really comes in. And so that's what we're looking to solve for. for the one single agent workflow, that's one thing and sort of the fairly easy, but as we move into, you know, potentially a multi architected system that has a supervisor and then sub agent sort of orchestration paradigm, that really where this is going to become much more compelling from an observability and monetization standpoint. Jamie: So it's interesting because you're talking about, OK, you have a base already, like you're going in, you're already helping build something. And it's like, here, we've got a way to monitor what that's going to cost, how that's going to work. OK, so let's talk. as you look forward, ⁓ so is an interesting product. Does anything else like this exist today, like UZIO? Peter Holcomb: Absolutely. So there are other players that are much more geared towards the enterprise market. So Metronome, Orb, Meter with the Three, these companies are already doing usage-based billing, but they're doing it for the enterprise segment. so for the clientele that we cater to, they're generally going to be in the mid-market area, ranging from about $5 million to $100 million. Jamie: Okay. Peter Holcomb: from an annual recurring revenue perspective. so those are the sort of business entities that we're going after. SMB, ⁓ tend not to play too much in that sandbox. We can, ⁓ we often find that ⁓ just not our ideal ⁓ market. Jamie: no, I got it. Okay, let's talk a little bit about, so, and where is that business ⁓ at now? how, it's relatively new, right? I mean, the whole, ⁓ all related to AI in general are relatively new. So monitoring, ⁓ know, and everything would be very new. So just how, ⁓ like you're at and what does that look like right now? Peter Holcomb: Yeah Sure, sure, I love this. I ⁓ 15 to 20 AI agents running in my business, and partly why we can be so lean and mean and such a nimble team. ⁓ And sort of doing all these agents that we've, monitoring all these agents that we've developed, now we're figuring out how do we understand what the underlying infrastructure and raw activity looks like. UZI has been around for four months, and so we're the MVP stage right now. ⁓ And so ⁓ we're actually looking go ahead and do a couple of pitch competitions. Jamie: Yeah. Peter Holcomb: here locally in Colorado. They are gonna be in April ⁓ in May. So look to get grant sponsored funding, if you that will be non dilutive. So that will be great. And then if we win those competitions, we'll continue to have our engineers look to bring this ⁓ product more to the so that we can essentially... Jamie: Okay. Peter Holcomb: tack it onto every consulting engagement we do an operational and revenue standpoint. And then I did have a meeting last night with some venture capitalists ⁓ a dinner. And so we're thinking about potentially fundraising as well, but it is not going to be ⁓ number one factor to build this product. It will be utilizing the revenue the proceeds from ⁓ our optimal IT consulting to fund essentially UZO's AI SaaS product. Jamie: Got it. this is a, this, see this as a tack on to what you're doing as opposed to completely different pathway for growth. Peter Holcomb: The distribution model would start there, Jamie, is just do get in where you can fit in sort of thing. But by all accounts, we will make this a full-fledged SaaS app as a separate entity. It will be powered by Optimal IT. So UZIO will be a part of the Optimal IT family. ⁓ it will be sold as a separate app, if need be, on a subscription and an outcome-based delivery model. Jamie: Okay. Okay, interesting. Okay, so let's talk a little bit about this because I think what's interesting is you've been building a consulting business, I'm sorry, professional services around, I get it, business around an area that's relatively new, right? So that's number one. So there's tailwinds, at least in terms of AI, right? There's a lot of interest around AI and a lot of people doing it, but you still have to... Go in, have expertise, build that business, get people to know it, right? In terms of what's there. then on top of that, you're building another thing. like you're doing a lot ⁓ So me understand how you're managing ⁓ all different work and thinking about where the focus is for you and where growth comes from as you push forward. Peter Holcomb: Sure. It helps to have a team. I can say that. Last year, more of a solo founder had a very small team. This year, we're at about four or five AI engineers, software engineers, and then a marketing team of three. I have a sales guy, have an intern, and a governance, risk, and compliance person. we're at about a team of 10. All of them are 1099 contractors, aside from myself. Jamie: ⁓ great. Peter Holcomb: ⁓ But this allows us to be able to stretch our capacity. I think you would agree being a former founder CEO of your own company, delegation is deeply important. ⁓ so what to delegate, finding the talent that knows how to solve the problem is deeply, deeply crucial for me. So we hire A plus engineers that ⁓ we throw a problem at them and they will go and solve the problem. And so obviously there's going to be some guidance, but we leave them. to their devices and they go and figure it out. that's one of the big mentalities that we look for is a figure it out mentality. If engineers have that mindset, we know they're going to be successful with us. So I have an engineer that essentially built ⁓ with me. I was the one who created UZO and then I gave him the source code and said, ⁓ go figure out how we need to ⁓ this. And so he built the API spec and now we're... We've already tested the API. works effectively. We run it against agents. We're essentially, like I said, being the middleware, plugging into the entire agent workflow. We're able to see all those raw details. The next step would be to be able to correlate all of the raw activity against what's called a value unit metric or a value metric unit, excuse me, and then have it be dashboarded appropriately so that businesses can see this agent is doing under the hood, so to speak, so that they can get a sense for, this is what my total cost of ownership is going to be to run this agent. in a consecutive basis, if you will. Jamie: Really interesting. think it's awesome. Okay. So, so let's talk a little bit about, so we talked a little bit about, cause I want to talk about you, Zio, and I want to talk about ⁓ where you go. so, so funding aside, cause, and I don't, get that's going to matter. ⁓ talk a little bit about growth, right? So you're right now you're building, you're building, you're at MVP stage. You've got something that you know, because You use it like you're using it. It's like it's right in your world. So at least you understand that there's a direct need for it. Let's talk a little bit about growth, where you anticipate growth coming from. How do you approach getting people to use a product when even when the whole idea of you utilizing AI and building AI and having it part of what you do and the cost associated with it is so new. Is that good? Is that bad? where do you go from there to kind of, you know, grow that business once the product is fully ready to ⁓ pushed out there? Peter Holcomb: Sure. At the dinner I was at last night, it was a group of 20 other founders ⁓ at the venture hosted dinner. And went around the table talking about what they're doing. You got some guy that's developing lithium ion batteries for sort of using new salt-like batteries or silver-like nickel batteries, right? You have another guy that's doing bio-pharmacy, AI engineering work. So I was telling them this, my idea as we were going around and juicing ourselves. Jamie: Mm-hmm. Peter Holcomb: Two of them came up to me and said, hey, I really like what you're looking to build. Can I be a beta tester? And so they're already coming to me and asking me, can we ⁓ this solution to see how ⁓ would meet our agents that we currently have running in our business? So ⁓ can tell you that the cold email outreach, the LinkedIn approach, that's all fine and dandy. I have always been a in-person type of marketer, if you will. Jamie: Interesting. Peter Holcomb: And so I love going to conferences. I love going to webinars, doing podcasts like this and getting the name out there. And so if there is a natural interest, people will come to you. If there's more of a foresight that you have to do in terms of marketing ⁓ your we'll go that route as well. But we're certainly going to meet customers where they are. Jamie: Mm-hmm. Peter Holcomb: And one of ways I found that to be is going to these events, right? Going to conferences, going to webinars and just putting yourself out there, sharing what you're doing. And I find that that often intrigues people enough to say, Hey, I'd to connect with you on LinkedIn and learn more about this. And heck, I would love to be a beta tester even with what we're doing. And a great example is this gentleman is building his agents in Lang chain. That's fantastic. That's the exact platform that we want to integrate with. he's using Langsmith, which is their observability platform to observe. the logs, traces and metrics of the software system or the AI agent system. And so we would hook into all of that and then be able to figure out how is this agent operating fundamentally. Right. And so your point of sort of like, how do we, how do we gain interest in ⁓ we're really doing it ⁓ grassroots by meeting people in the field ⁓ conferences in real life events, ⁓ rather than trying to ⁓ the cold email outreach and marketing efforts. Jamie: Yeah. Yeah. Peter Holcomb: In that in that way, so I mean it's still not i'm not it's still something that we're not going to do It's we're definitely going to do that. But I feel like In a world where ai is so, I don't know prevalent right now it's making us sort of all be cooped up in our homes and stuff I think everybody is really eager to get back out into the world and get get back and socialize and Stir business in that way, right? And I know covid sort of made us all bit bogged down in that way, too. So I'm just finding there's a lot of interest for people to get outside and meet each other face to face. So that's really great. Jamie: think that's awesome. And I think it's a it's it's, you know, you bring up things like COVID and being stuck and not being able to do it. It's amazing that we can go back out and do it again. And that's fantastic. And that's amazing. Okay, ⁓ all right, so let's let's talk. So, so biggest challenge or biggest area of focus that you've got right now, like, what's the thing that's, you know, holding your complexity, holding back challenge that you're facing that you're really working through with them? Peter Holcomb: Sure. Go to market is going to be one of the challenges. We know that we can use the existing consulting distribution channels and customers. But the other one would be a pricing strategy. And think you may have this ⁓ in previous businesses you've run as well, which is how do you price effectively so that the market feels like you're competitive but not overpriced or underpriced? There's some market signals that we have to determine sort of what that looks like. It's a bit harder though, because number of tokens used by Claude or GPT or perplexity or Gemini, they're all different. So that was going to be one of the unit economics that we track is token usage, input and output of the large language model. And that is essentially how these credit-based systems currently work today. But we want to go even further. And so we help businesses by able to say, If this API calls this backend system, HubSpot or Salesforce, 10 times, you guys or the business can determine what that value unit metric is and the associated prices in cents or dollars. then we get to figure out, well, this person is running that workflow 100 times based on that total cost of ownership. We feel like we could charge X amount for this product. no, $99 a month or something like that. But they get all of this. ⁓ value to be able to see what their AI agents are doing, what their swarms of AI agents are doing. So right now, I'm reading a couple of books and I can just show them to you. this one is monetizing innovation. Mon Haven one of my favorite, favorite guys, but ⁓ he's one of the key people that I'm learning from to be able to figure out how do you monetize this stuff appropriately, especially from a SaaS perspective, so that it's And then the other thing is, now that we're going into this sort of usage of hybrid world, that's the other piece that we're going to have to figure out. How do we make this sort of a hybrid billing model so that people see that all of this stuff is running on a pay as you go sort of basis, right? And so we need to be able to effectively charge back for that only when these agents are running. It's very similar to what AWS did back in 2004, 2006 with regard to sort of the the EC2 compute instances that you only pay for what you used. So we are going to be very much in the same vein. ⁓ we're essentially borrowing much of their business model in terms of how we do this, but it's all going to be for AI agents. Jamie: Yeah. Yeah, listen, if you ended up being like AWS, that'd be pretty good, huh? Like that's not a bad place to land when all said and done. that model worked out all right, I think. Peter Holcomb: Hahaha Yeah, yeah, mean, we would love to solve that problem in that same manner by all accounts. Yeah. Jamie: ⁓ Yeah. Okay. So, so this is awesome and exciting stuff. The so, so in terms ⁓ of you're I say two businesses, I know that you've right now you've connected the two businesses, ⁓ which makes a lot. get it. So in terms ⁓ of, you know, how Peter Holcomb: Yes. Jamie: What I'd actually love you to do is to share like with the, with folks is like, what are the things that you're finding that you talked about delegation? Are there other things that you're finding that are helping you balance and, and, drive the maximum amount of, you know, positive impact between both the consulting business, which is out and dealing with clients and all those things. then building a product, solution for people at the same time and having to brain back and forth and you know you obviously need and want both to be successful. So are the things that are working well and what things are not that you wish you were doing better? Peter Holcomb: Yeah, Jamie, if I had known how hard marketing is, probably would have focused a lot more on that. ⁓ My goodness, we are technologists. I'm a technologist. ⁓ can tell you that. I've been in tech for 20 years, cyber for 15, AI for six. my goodness, marketing a thing that we did not do well in the beginning. ⁓ And didn't make revenue for nine months last year. So just imagine the stress. Jamie: Okay. Yeah. Peter Holcomb: and the uncertainty. And it was hard. It was really hard to figure out, we doing the right thing? Should we even keep going? And luckily, the team we had, they're creating content for us. And so that was good, but they weren't doing any outreach via in-mails and LinkedIn and things like that. And so any of that sort of grassroots movement of getting our names out there, letting people who you are, that was the hard part. So marketing, I would say to anybody, make sure you figure out how you market your product or your services effectively so that people know how to find you and leverage you and utilize your skills. Because if you don't, then it's just going to be sort of lost and ⁓ it's The other thing is, have a great sales process. Know what that process is. Oftentimes, people will come to us and say, look, you guys are presented well. We really like what you guys are doing. What's next steps? And if you just look at them with blank stares, that's doesn't really assure them or reassure them that you know what you're doing and you know how to engage with them after their initial conversation. So we have a sales process. do ⁓ assessment, ⁓ a business case development. We'll then help them design sort of a prototype. From the prototype, if it works out, we do an upfront fee to go ahead and build that prototype and then go ahead and get it onboarded into their production environment. We have then a managed model where we're either going to give the agent back to them or the engineering solution back to them if they have the technical resources inside to be able to it themselves. If not, then we become their managed intelligence partner whereby we will go ahead and manage the underlying logic, the agent infrastructure, the SLA, so service level agreements, the uptime, all of that stuff that a traditional sort of software development process would require. So that's that. And then I would say in terms of engineering, I come from the engineering world, so we understood how to do this. how you develop AI agents or understand how you develop software. We use the Agile methodology. Everything that we do is built on a two-week sprint. So we'll get requirements. If the requirements are not fleshed out, we go back to the customer and say, look, we need better information so that we can actually build you the thing that you want and the vision comes to life. If we don't have this, then we can't give you the actual output that you're looking for. And so having a two-week sprint, moving stuff from... sort of the backlog into requirements of design and to build, test, and then manage or deploy. That's sort of how we work as well. Our engineers like that flow. There's no gotchas. We all understand how we have to operate effectively. So these are some of the things that I think ⁓ should think about. But by all accounts, for people that are coming at this from a technologist's point of view, I cannot emphasize ⁓ much and how important marketing is. If you can't sell or market your product, you have no product, right? ⁓ I would say that's pretty much table stakes to be able to your business a name or service or product out there. So that why I'm really eager about UZO is because we've already got the consulting clients and we've already earned the buy-in and the trust from them. So it's just a matter of ⁓ if build this AI agent for you, can we just tack on this UZO product for one month or two months as a no-cost effort and show you sort of the fidelity of data that you can get. And if it's enriched to your liking, Jamie: Yeah. Peter Holcomb: would you be willing to go ahead and pay for it on a subscription basis? And often times we say, you know, they say yes, or they sometimes say, you know what, we don't really need all your engineering services. We just want the product. right? And so that's where this could go as well in terms of how we go to market with this. Jamie: Thank Yeah, that's awesome. I love it. This is amazing. We could talk all day, but we can't. and I really enjoyed it. So ⁓ do me a favor. Can tell everybody how they can find you, reach you? And I'll put some, what's it called? in to the description after. Peter Holcomb: Sure. They can find us at OptimoIT.io, all one word. That is where the consulting business resides. The other business is UZEO.AI. UZEO.AI. UZEO is, I guess, in a MVP state. The domain is up ⁓ and you can the URL and potentially log in. We'll still have to grant you access if you want access to the platform. The one thing I'll leave you with Jamie in terms of how we can also distribute is there are platforms like Product Hunt where we can go ahead and push our product out and then get feedback from existing customers or new customers that want to try our product. So that's our hope and intent as well. Once this is in a much better mature state, which ⁓ I it to be ⁓ after get done with the pitch competitions in May, because we're going to continue to refine and iterate on this. Then we're going to go to like Product Hunt and a couple of other websites. There's like 50 different websites where you can publish the SaaS app and users will come and just start banging at it and using it and giving you feedback. We really want that feedback. We want to know, does it work optimally? Where are the hiccups? What could be improved? How's the UI experience? All of that stuff and insight is something that we want, but we know that we're still making an iterative effort to get there. Jamie: Well, listen, thank you, Peter, so much for sharing. This is great. I really enjoyed it. I hope everybody else enjoyed it. I that was awesome. So thanks. And guys, until next time, thanks for watching. Thanks for listening. And have a good rest of your day. Peter Holcomb: Thank you very much for having me on the pod, Jamie.