Latest / Glovisor Podcast / Ep. 48: Your Company Is Wasting Money on AI—and You Probably Don't Know It - Zumah.co
Transcript
- Joe: AI isn't expensive, the routing is. Learn how to cut costs without touching the workflows. Welcome to the Glovisor Podcast where we explore business growth, communication, and personal development through real conversations with people that make an impact. Our guest thrives at the intersection of AI problem solving and product innovation. With hands-on experience in machine learning and data science, he brings technical depth and a clear vision of how intelligent systems should operate. He is the co-founder of SUMA, a token reduction system reducing AI costs. Glovisor welcomes Jeremy Barry Jeremy Barry: Thanks for having me, Joe. I appreciate it. Joe: It's a pleasure. I'm looking forward to conversation. Jeremy Barry: Yeah, yeah, absolutely. Joe: All right, Jeremy, so please share with us what is it that you do? Jeremy Barry: Yeah, so I am twenty years old. I'm a student and on top of that, I like to think of myself also as an entrepreneur. The past month or so, me and one of my friends I was living with for the past month, we kind of ran into an issue at work. So both at our respective internships, within the first week or so, we both got emails from higher ups weren't too pleased with us because we were Let's just say using AI maybe a little too much within our jobs. I think something like my intern group, on the first two days we burned through something, I think like twenty percent of our overall company's token budget. While with him, he works at Capital One. Not as big of a percentage, but he still got an email from someone yelling at him. so the main issue that we kinda came across is that no one is using AI as economically as they should be, and what that kinda boils down to is token cost. So just for reference, tokens are a unit of measurement that LLMs like ChatGPT, Claude Code, Grok, all of them they use to read, process, and generate text. So every input, output, and any additional piece of context consumes tokens. So for companies who use AI, especially at scale, those costs can add up very quickly. And the main problem is that these employees don't know how to u a use AI as efficiently as they could because the way these AIs work is that every model, like what ChatGPT, what anthropic, what open AI what they do is that The models, the default option is usually the most expensive one and the most intensive one. So that means for people who aren't switching their models for every prompt, that means like a simple task, like I don't know, let's say summarizing an email, that would be sent to a premiere model that could be five times more expensive than a cheaper model could be. Even though the cheaper model could probably complete the same task just as well. So what we built, it's a system. We're calling it the token reduction system that automatically analyzes each request and auto routes it to the most cost effective model that's capable of completing the task at hand and it does all of it before the prompt is processed. So I think this is I think it's cool and innovative and the part that I think about it that's really cool is it kind of gets rid of the need to train employees on how to use AI efficiently. Cause they can you continue to use it exactly how they've already been using it. Because this software it works directly in the background. It can directly integrate the ChatGPT, Claude any cloud facing L LM and also a lot of companies these days they use like their own custom LLMs that are trained on like their own data. So we have a SDK that can be directly integrated into that as well. So what we've realized so far is depending on how your company uses AI, we've seen results ranging from 60% to 85% cost reduction, which has been great. So yeah, that's a little bit about what I've been working on. Joe: Wow, that's a huge reduction. So now you mentioned that this would be in infused into the background and Will it cause a delay when people submit their prompts? How does that work? Jeremy Barry: Yeah, so how it works is what we built, it's just like a machine learning algorithm. And the model we u like the machine learning algorithm that we use is very lightweight, so it can be uploaded like directly onto the user's computer. So the very first prompt, I think it's like a three second delay, but after the very first prompt for the rest of the conversation, there's no delay. It's practically the latency is the exact same as it would be with chat, GPT or Cloud or whatever. Joe: Well, s so it's gonna live in the system moving forward and it it it just sort of picks the correct highway to take when the prompt is sent. Jeremy Barry: Exactly. Yeah, that's a great way of thinking about it. Just takes the most optimal route. That ch gets you to your goal the cheapest way. Joe: So now the two of you received these emails letting you know that that you're surpassing the amount of tokens. so take me through what happened. You two got together and and started working on this project and how long did it take and and how much work it does this all encompass? Jeremy Barry: Yeah, yeah, absolutely. So I guess I can s start a little while back. So me and my friend, his name's Sam, and we are both interested in entrepreneurship. We're both like coding. We like AI, that type of stuff. So beginning of the school year, last August or so, so actually about a year ago now, we decided to make an app. It was called Crawler. It was like a social media app. It was a super fun app, but like I it got actually like five hundred plus users, which was pretty fun, but I don't know. It was kind of like a money pit. We weren't gonna ever make money from it and we were kinda just doing it for fun. So we decided that down the line, if we ever thought of a good idea, we'd lock in on it and try to make something happen. So We ended up both getting internships in DC together. Not the same company, but we ended up living together. So we get to the I think, yeah. Like I said, like one week basically into our internship, we both get these emails. And I remember I told him, because I got mine before he did, and he thought it was pretty funny. He was laughing at me. Then yeah, a couple days later he gets literally the exact same thing. And what we kind of realized was the reason why this was happening is cause we both were just using like the top-tier model for literally everything. Because we weren't really thinking about the cost side of things. So that second forward we're like, shoot, like I can't imagine this is only happening to just us. So we decided to talk to some more people. We just like linked in some like William and Mary alumni. And yeah, seems like this is a problem that's pretty standard that people are facing a lot more and more now. So decided to just get my hands dirty. I've started making the product and it had to time wise, Right off the bat, it definitely took a lot of time, especially 'cause we had like our nine to five internships. So basically all our other time other than that nine to five was dedicated towards making this. But once we the product was kind of made, it's been a little bit less time intensive, I'd say. The main hardest part of this is just yeah, outreach, meet networking, all that type of stuff. Especially 'cause we're so young. So like our networks aren't as expensive as hopefully they will be one day. Joe: Well, I'm sure they will. but yeah, Jeremy Barry: Mm-hmm. Joe: this product is totally impressive. Yeah, because I think that you're fixing a problem that most of us didn't even realize that we're gonna have. because the more we rely on AI, this is gonna become a problem for all of us. So I wanted to go back and ask you a question. I think a lot of us go to the free version of GPT or Claude or any of those. And so tokens are being utilized when we're putting in our prompts. Jeremy Barry: Yeah, yeah, it's like the measurement. Think of every time you drive a car, you spend gas, and that gas costs probably I don't know like per mile exactly how much you would cast. It would probably cost a certain amount of cents It's like the same idea where t like a short prompt could me cost anywhere from like I don't know. It really depends on how much like context and stuff is involved. with your prompt, but it could cost X amount of tokens. And yeah, the thing is the tokens aren't necessarily the thing we're trying to reduce, not the amount of tokens, it's the cost per token. Because these models, they all use tokens. Some use more tokens than others, but the main cost thing is the t cost per token. So like the most premier models, like I know Claude's new Fable 5, that costs, I believe like, yeah, five to ten dollars per token, while their cheapest model can cost like 10 cents per token. Or per million tokens, sorry. Yeah. Yeah, it's a really really Joe: Wow, that's a huge, huge spread. Jeremy Barry: big gap. So just trying to get people to use those cheaper models for the tasks that they can can really make a big impact, especially at scale. Joe: my short email to be grammatically checked doesn't take as many tokens. But once you become an enterprise, now you have so many people within your organization all inputting their emails, their prompts, and this is where the numbers really in the scale, this is where it really makes an impact. Jeremy Barry: Yes. Yeah, exactly. At scales where it makes a big impact. Something we're working on on on top of the like API call that does the router is like an executive dashboard so like higher ups and leaders can see can break down token usage per employee and kinda see who's using AI the most effectively as well. Joe: So have you started to select a specific level of business that you need to work with? for instance, I if we're running a small business, maybe this not might not be the structure for us. This might still not make an impact for us. Do I need a thousand employees? Is it based on the usage? How how do you make that determination and who should be contacting Jeremy Barry: Yeah, absolutely. So at first we were just trying to talk to as many people as possible, but as we've kind of gone further along, we've nailed down exactly who we're looking for a little bit more. So how AI works, there's people who use subscriptions and those people, they're already getting the best deal. So if you're spending like a hundred dollars a month on ChatGPT and you're using like all your tokens, that's the most efficient way to do it. And you probably wouldn't need something like us. But typically, even small businesses, I'd say organizations with like over ten people, they pay per they pay the API cost, which is instead of just paying us a flat fee. So really anyone who pays per like API cost is someone who should reach out to us if they're looking to reduce some cost. Joe: So if one of our listeners wanted to work with you, what would the journey look like? Jeremy Barry: Yeah, so if you want to work with us, the way we do that is traditionally when we start working with someone, we do something called a token audit. This is free. This is just hopping on a call and discussing kinda how you use AI and where we could come help. so what we've been doing is we've been kind of giving people The choice of one of four things if we don't have the option to help them. Because we're so confident that we'll be able to help you most times. So the first thing we'd be willing to do is you gotta choose one of these fours. We'd rent a billboard in Utah telling people that you are running at already running at peak efficiency. This is if we can't save you tokens. Another option is we'd spin a sign around at our local McDonald's for 12 hours promoting your company. We'd or we could hand deliver gift cards plus a call apology cards for not being able to help. To 10 employees of your choosing. Or the final option, which is my personal favorite, is we'd climb a mountain and give you a stone from the top. Joe: I love those. Jeremy Barry: Yeah, so we're confident that this will be able to save anyone who uses AI money. Joe: would you be able to share with us a client success story? Jeremy Barry: Yeah, gotcha. So we've started piloting our first client recently, and so far they've been on it for two weeks. We've reduced their token costs by around 70%, which has been really exciting to see. Like it actually in production and it being used, which has been really great. And I think as but they scale more and they use AI more over the next months or so, we hope that we can save them. And that number could even go up potentially. So I think that's so far I'd say that's the biggest success we'd have we've had Joe: once someone signs up with you and they realize that they're doing all the savings, do you have to work with them on an ongoing basis or is this sort of a a a set it and forget it type of deal? Jeremy Barry: depends. I guess s right now, just 'cause we have so few clients and we're just starting out, we like to just talk to people, see what's working, see what's not working, so we can improve our product. Also, yeah, it's not perfect right now. There's still gonna be a couple bugs just because yeah, this is our piloting process. So I'd say as of right now, yeah, definitely keep in contact, keep communicating. But down the line, I don't see why it couldn't be set it and forget it. thing. I think if the like executive dashboard comes along a little bit further where it gets to the point where you need to cut people's AI usage off and whatnot, that could be something you could reach out to us for. But yeah, I think overall we want as little friction as possible with this. We want to feel like you're using your AI before you you even met us. Joe: So if listeners remembered only one thing from today's conversation, what do you hope that message is? Jeremy Barry: I'd say the one thing I want people to remember is AI does not need to be this big expensive thing that is gonna kinda ruin your entire company budget. It should really be something where the amount of money you put in you should really be getting a marginal impact of return on that. So I'd say just make sure you're using everything efficiently and if your employees are struggling to do that, you know who to call. Joe: where can our audience connect with you? Learn more about your work and continue this conversation. Jeremy Barry: Yeah, absolutely. So our website is called Zumah.co. That's Z-U-M-A-H.C.O. And if you want to connect with me personally, my LinkedIn, Jeremy Barry, always willing to chat with as many people as possible. and my friend Sam, Sam Harrington, he's all also willing to chat. We're just trying to meet as many people as possible and figure out, you know, if we're really onto something here or not. So any ad advice, any feedback is really appreciated. Joe: Excellent. Jeremy, thank you so much for joining us. We appreciate your time and the value you shared. Jeremy Barry: Thank you, Joe. Appreciate it. Thanks for having me. Joe: Thanks everyone for listening. If this episode inspired you, share it with someone who needs it. And don't forget to follow the Glovisor Podcast for more conversations that make an impact. Till the next time.