Latest / Glovisor Podcast / Ep. 48: Your Company Is Wasting Money on AI—and You Probably Don't Know It - Zumah.co

Transcript

Ep. 48: Your Company Is Wasting Money on AI—and You Probably Don't Know It - Zumah.co · Glovisor Podcast

• 0:00
  Speaker 1: AI is an expensive The routing is learn how to
• 0:03
  Speaker 1: cut costs without touching the workflows. Welcome to the Globviser podcast,
• 0:09
  Speaker 1: where we explore business growth, communication and personal development through
• 0:13
  Speaker 1: real conversations with people that make an impact. Our guest
• 0:17
  Speaker 1: thrives at the intersection of AI problem solving and product innovation.
• 0:21
  Speaker 1: With hands out experience in machine learning and data science,
• 0:24
  Speaker 1: he brings technical depth and a clear vision of how
• 0:27
  Speaker 1: intelligence systems should operate. He is the co founder of Suma,
• 0:32
  Speaker 1: a token reduction system reducing AI costs. Globviser welcomes Jeremy Barry.
• 0:39
  Speaker 2: Thanks for having me, Joe, I appreciate it. It's a pleasure.
• 0:42
  Speaker 1: I'm looking forward to the conversation.
• 0:44
  Speaker 2: Yeah, yeah, absolutely, all.
• 0:46
  Speaker 1: Right, Jeremy. So please share with us what is it
• 0:49
  Speaker 1: that you do.
• 0:49
  Speaker 2: Yeah.
• 0:50
  Speaker 3: So, I am twenty years old, I'm a student, and
• 0:54
  Speaker 3: on top of that, I like to think of myself
• 0:56
  Speaker 3: also as an entrepreneur. The past month or so, so
• 1:00
  Speaker 3: me and one of my friends I was living with
• 1:02
  Speaker 3: for the past month, we kind of ran into an
• 1:04
  Speaker 3: issue at work. So both at our respective internships. Within
• 1:08
  Speaker 3: the first week or so, we both got emails from
• 1:12
  Speaker 3: higher ups weren't too pleased with us because we were,
• 1:16
  Speaker 3: let's just say, using AI maybe a little too much
• 1:19
  Speaker 3: within our jobs. I think something like my intern group
• 1:23
  Speaker 3: on the first two days we burned through something I
• 1:25
  Speaker 3: think like twenty percent of our overall company's token budget.
• 1:28
  Speaker 3: While with him he works at Capital one, not as
• 1:31
  Speaker 3: big of a percentage, but he still got.
• 1:32
  Speaker 2: An email from someone yelling at him.
• 1:34
  Speaker 3: So the main issue that we kind of came across
• 1:36
  Speaker 3: is that no one is using AI as economically as
• 1:39
  Speaker 3: they should be, and what that kind of boils down
• 1:41
  Speaker 3: to is token cost.
• 1:42
  Speaker 2: So just for reference, tokens.
• 1:44
  Speaker 3: Are a unit of measurement that LM's like chash ebt,
• 1:47
  Speaker 3: cloud code rock, all of them they used to read,
• 1:50
  Speaker 3: process and generate texts. So every input, output and any
• 1:54
  Speaker 3: additional piece of context consumes tokens. So for companies who
• 1:58
  Speaker 3: use AI, especially at scale, bill those costs can add
• 2:01
  Speaker 3: up very quickly. And the main problem is that these
• 2:06
  Speaker 3: employees don't know how to use AI as efficiently as
• 2:09
  Speaker 3: they could because the way these ais work is that
• 2:12
  Speaker 3: every model like what CHATBT, what anthropic, what openay, what
• 2:17
  Speaker 3: they do is that the models, the default option is
• 2:21
  Speaker 3: usually the most expensive one in the most intensive one.
• 2:23
  Speaker 3: So that means for people who aren't switching their models
• 2:26
  Speaker 3: for every prompt, that means like a simple task like
• 2:28
  Speaker 3: I don't know, let's say, summarizing an email that would
• 2:30
  Speaker 3: be sent to a premier model that could be five
• 2:32
  Speaker 3: times more expensive than a cheaper model could be, even
• 2:35
  Speaker 3: though the cheaper model could probably complete the same task
• 2:38
  Speaker 3: just as well.
• 2:40
  Speaker 2: So what we build its system.
• 2:42
  Speaker 3: We're calling it the token reduction system that automatically analyzes
• 2:46
  Speaker 3: each request and auto rats it to the most cost
• 2:49
  Speaker 3: effective model that's capable of completing the task at hand.
• 2:53
  Speaker 3: It does all of it before the prompt is processed.
• 2:56
  Speaker 3: So I think this is I think it's cool and innovative.
• 2:58
  Speaker 3: And the part that I think about it that's really
• 3:00
  Speaker 3: cool is it kind of gets rid of the need
• 3:01
  Speaker 3: to train employees on how to use AI efficiently because
• 3:05
  Speaker 3: they can continue to use it exactly how they've already
• 3:07
  Speaker 3: been using it because of software works directly in the background.
• 3:11
  Speaker 3: It can directly integrate the chatchybt Claude, any cloud facing LM.
• 3:16
  Speaker 3: And also a lot of companies these days they use
• 3:18
  Speaker 3: like their own custom lms that are trained on like.
• 3:20
  Speaker 2: Their own data.
• 3:21
  Speaker 3: So we have an SDK that can be directly integrated
• 3:24
  Speaker 3: into that as well. So what we've realized so far
• 3:27
  Speaker 3: is depending on how your company uses AI, we've seen
• 3:30
  Speaker 3: results ranging from sixty percent to eighty five percent cost reduction,
• 3:34
  Speaker 3: which has been great. So yeah, that's a little bit
• 3:36
  Speaker 3: about what I've been working on. Wow, that's a huge reduction.
• 3:41
  Speaker 1: So now you mentioned that this would be infus into
• 3:45
  Speaker 1: the background and will it cost a delay when people
• 3:48
  Speaker 1: submit their prompts? How does that work?
• 3:51
  Speaker 2: Yeah, So how it works is what we built.
• 3:53
  Speaker 3: It's just like a machine learning algorithm, and the model
• 3:57
  Speaker 3: we use, like the machine learning algorithm that we use,
• 4:00
  Speaker 3: is very lightweight, so it can be uploaded directly onto
• 4:03
  Speaker 3: the user's computer. So the very first prompt, I think
• 4:06
  Speaker 3: it's like a three second delay, But after the very
• 4:10
  Speaker 3: first prompt, for the rest of the conversation, there's no delay.
• 4:13
  Speaker 3: It's practically the laten sees the exact same as it
• 4:16
  Speaker 3: would be with chat, GBT or Claude or whatever.
• 4:20
  Speaker 1: So it's going to live in the system moving forward
• 4:22
  Speaker 1: and it just sort of picks the correct highway to
• 4:26
  Speaker 1: take when the prompt is sent exactly.
• 4:29
  Speaker 3: Yeah, that's a great way of thinking about it. Just
• 4:32
  Speaker 3: takes the most optimal route that actually gets you to
• 4:34
  Speaker 3: your goal the cheapest way.
• 4:36
  Speaker 1: So now the two of you received these emails letting
• 4:38
  Speaker 1: you know that you're surpassing the amount of tokens. So
• 4:43
  Speaker 1: take me through what happened. You two got together and
• 4:46
  Speaker 1: started working on this project, and how long did it
• 4:49
  Speaker 1: take and how much work does this all encompass?
• 4:53
  Speaker 2: Yeah?
• 4:54
  Speaker 3: Yeah, absolutely, so I guess I can start a little
• 4:58
  Speaker 3: while back. So me and my friend, his name's Sam,
• 5:01
  Speaker 3: and we are both interested in entrepreneurship. We're both like coding,
• 5:06
  Speaker 3: we like AI that type of stuff. So the beginning
• 5:09
  Speaker 3: of the school year, last August or so, so actually
• 5:12
  Speaker 3: about a year ago now, we decided to make an app.
• 5:15
  Speaker 2: It was called Crawler. It was like a social media app.
• 5:17
  Speaker 3: It was a super fun app, but like it got
• 5:21
  Speaker 3: actually like five hundred plus users, which was pretty fun,
• 5:24
  Speaker 3: but I don't know, it was kind of like a
• 5:26
  Speaker 3: money pit. We weren't gonna ever make money from it,
• 5:28
  Speaker 3: and we were kind of just doing it for fun.
• 5:29
  Speaker 3: So we decided that down the line, if we ever
• 5:33
  Speaker 3: thought of a good idea, we'd lock in on it
• 5:35
  Speaker 3: and try to make something happen. So we ended up
• 5:38
  Speaker 3: both getting internships in DC together, not the same company,
• 5:41
  Speaker 3: but we ended up living together.
• 5:43
  Speaker 2: So we get to the I think.
• 5:46
  Speaker 3: Yeah, like I said, like one week basically into our internship,
• 5:49
  Speaker 3: we both get these emails and I remember I told
• 5:52
  Speaker 3: him because I got mine before he did, and he
• 5:54
  Speaker 3: thought it was pretty funny.
• 5:55
  Speaker 2: He was laughing at me.
• 5:56
  Speaker 3: Then yeah, a couple days later he gets literally the exact
• 5:58
  Speaker 3: same thing. And what we kind of realized us the
• 6:00
  Speaker 3: reason why this was happening is because we both were
• 6:02
  Speaker 3: just using like the top tier model for literally everything,
• 6:05
  Speaker 3: because we weren't really thinking.
• 6:06
  Speaker 2: About the cost side of things.
• 6:08
  Speaker 3: So that second forward, we're like, oh, shoot, like I
• 6:12
  Speaker 3: can't imagine this is only happening to just us. So
• 6:15
  Speaker 3: we decided to talk to some more people. We just
• 6:18
  Speaker 3: like linked in some like William and Mary alumni, and yeah,
• 6:22
  Speaker 3: seems like this is a problem that's pretty standard that
• 6:25
  Speaker 3: people are facing a lot more and more now. So
• 6:28
  Speaker 3: decided to just get my hands dirty. I've started making
• 6:31
  Speaker 3: the product and it had to time wise.
• 6:37
  Speaker 2: Right off the bat.
• 6:37
  Speaker 3: It definitely took a lot of time, especially because we
• 6:40
  Speaker 3: had like our nine to five internships, so basically all
• 6:42
  Speaker 3: our other time other than that nine to five was
• 6:44
  Speaker 3: dedicated towards making this, but once the product was kind
• 6:47
  Speaker 3: of made, it's been a little bit.
• 6:49
  Speaker 2: Less time intensive. I'd say the main.
• 6:54
  Speaker 3: Hardest part of this is just yet outreach, meet networking,
• 6:57
  Speaker 3: all that type of stuff, especially because we're so young,
• 6:59
  Speaker 3: so like our networks aren't as expense as hopefully they
• 7:01
  Speaker 3: will be one day.
• 7:03
  Speaker 1: Well I'm sure there will, but yeah, this product is
• 7:06
  Speaker 1: totally impressive. Yeah, because I think that you're fixing a
• 7:10
  Speaker 1: problem that most of us didn't even realize that we're
• 7:14
  Speaker 1: going to have, because the more we rely on AI,
• 7:17
  Speaker 1: this is going to become a problem for all of us.
• 7:20
  Speaker 1: So I wanted to go back and ask you a question.
• 7:22
  Speaker 1: I think a lot of us go to the free
• 7:24
  Speaker 1: version of GPT or plot or any of those, and
• 7:29
  Speaker 1: so tokens are being utilized when we're putting in our prompts.
• 7:34
  Speaker 2: Yeah.
• 7:34
  Speaker 3: Yeah, it's like the measurement think of every time you
• 7:38
  Speaker 3: drive a car, you spend gas, and that gas costs
• 7:42
  Speaker 3: I don't know, like per mile exactly how much you
• 7:44
  Speaker 3: have cast it would probably cost a certain amount of sense.
• 7:46
  Speaker 3: It's like the same idea where it's like a short
• 7:50
  Speaker 3: prompt could cost any where from like, I don't know,
• 7:55
  Speaker 3: it really depends on how much like context and stuff
• 7:58
  Speaker 3: is involved with your but it could cost x amount
• 8:01
  Speaker 3: of tokens. And yeah, the thing is the tokens aren't
• 8:04
  Speaker 3: necessarily the thing we're trying to reduce not the amount
• 8:06
  Speaker 3: of tokens, it's the cost per token. Because these models,
• 8:10
  Speaker 3: they all use tokens, some use more tokens than others,
• 8:13
  Speaker 3: but the main cost thing is the cost per tokens.
• 8:16
  Speaker 3: So like the most premier models, like I know Claude's
• 8:19
  Speaker 3: new Fable five that costs I believe, like yeah, five
• 8:23
  Speaker 3: to ten dollars per token, while their cheapest model can
• 8:25
  Speaker 3: cost like ten cents per token or per million tokens. Sorry, yeah,
• 8:30
  Speaker 3: that's a huge, huge, Yeah, it's a really really big gap.
• 8:33
  Speaker 3: So just trying to get people to use those cheaper
• 8:36
  Speaker 3: models for the tasks that they can can really make
• 8:38
  Speaker 3: a big impact, especially at scale.
• 8:41
  Speaker 1: My short email to be grammatically checked doesn't take as
• 8:45
  Speaker 1: many tokens. But once you become an enterprise, now you
• 8:48
  Speaker 1: have so many people within your organization all in putting
• 8:52
  Speaker 1: their emails, their proms, and this is where the numbers
• 8:55
  Speaker 1: really in the scale, This is where it really makes
• 8:59
  Speaker 1: an impact.
• 8:59
  Speaker 3: Yes, yeah, exactly at scales where it makes a big impact.
• 9:03
  Speaker 3: Something we're working on on top of the API called
• 9:07
  Speaker 3: that does. The router is like an executive dashboard, so
• 9:11
  Speaker 3: like higher ups and leaders can see can break down
• 9:14
  Speaker 3: token usage per employee and kind of see who's using
• 9:16
  Speaker 3: AI the most effectively as well.
• 9:18
  Speaker 1: So have you started to select a specific level of
• 9:25
  Speaker 1: business that you need to work with? For instance, if
• 9:28
  Speaker 1: we're running a small business, maybe this might not be
• 9:32
  Speaker 1: the structure for us. This might still not make an
• 9:35
  Speaker 1: impact for us. Do I need a thousand employees? Is
• 9:37
  Speaker 1: it based on the usage? How do you make that
• 9:42
  Speaker 1: determination and who should be contacting?
• 9:44
  Speaker 2: Yeah?
• 9:44
  Speaker 3: Absolutely, So at first we were just trying to talk
• 9:47
  Speaker 3: to as many people as possible, but which we've kind
• 9:49
  Speaker 3: of gone further along, we've nailed down exactly who we're
• 9:52
  Speaker 3: looking for a little bit more so how AI works.
• 9:55
  Speaker 3: There's people who use subscriptions, and those people they're already
• 9:58
  Speaker 3: getting the best deal spending like one hundred dollars a
• 10:01
  Speaker 3: month on chat GBT and you're using like all your tokens.
• 10:05
  Speaker 3: That's the most efficient way to do it, and you
• 10:07
  Speaker 3: probably wouldn't need something like us. But typically even small businesses.
• 10:11
  Speaker 3: I'd say organizations with like over ten people. They pay
• 10:15
  Speaker 3: per they pay the API cost, which instead of just
• 10:18
  Speaker 3: paying us a flat fee. So really, anyone who pays
• 10:21
  Speaker 3: per API cost is someone who should reach out to
• 10:24
  Speaker 3: us if they're looking to reduce some cost.
• 10:27
  Speaker 1: So, if one of our listeners wanted to work with you,
• 10:29
  Speaker 1: what would the journey look like?
• 10:30
  Speaker 3: Yes, so if you want to work with us, The
• 10:32
  Speaker 3: way we do that is traditionally, when we start working
• 10:34
  Speaker 3: with someone, we do something called a token on it.
• 10:37
  Speaker 2: This is free.
• 10:38
  Speaker 3: This is just hopping on a call and discussing kind
• 10:40
  Speaker 3: of how you use AI and where we could come help.
• 10:44
  Speaker 3: So what we've been doing is we've been kind of
• 10:47
  Speaker 3: giving people the choice of one of four things if
• 10:50
  Speaker 3: we don't have the option to help them, because we're
• 10:52
  Speaker 3: so confident that we'll be able to help you most times.
• 10:55
  Speaker 3: So the first thing we'd be willing to do, you
• 10:58
  Speaker 3: got to choose.
• 10:58
  Speaker 2: One of these fours.
• 10:59
  Speaker 3: We'd rent a build board in Utah telling people that
• 11:01
  Speaker 3: you are running at already running at peak efficiency.
• 11:05
  Speaker 2: This is if we can't save you tokens and other options.
• 11:07
  Speaker 3: We'd spin a sign around at our local McDonald's for
• 11:10
  Speaker 3: twelve hours promoting your company, we'd or we could hand
• 11:13
  Speaker 3: delivered gift cards plus all apology cards for not being
• 11:16
  Speaker 3: able to help to ten employees of your choosing. Or
• 11:19
  Speaker 3: the final option, which is my personal favorite, is we'd
• 11:22
  Speaker 3: climb a mountain and give you us stones from the top.
• 11:25
  Speaker 2: I love those.
• 11:26
  Speaker 3: Yeah, So we're confident that this will be able to
• 11:29
  Speaker 3: save anyone who uses AI money.
• 11:32
  Speaker 1: Would you be able to share with us a client
• 11:34
  Speaker 1: success story?
• 11:36
  Speaker 2: Yeah, gotcha.
• 11:36
  Speaker 3: So we've started piloting our first client recently and so
• 11:41
  Speaker 3: far they've been on it for two weeks. We've reduced
• 11:44
  Speaker 3: their token costs by around seventy percent, which has been
• 11:47
  Speaker 3: really exciting to see, like actually in production and it
• 11:50
  Speaker 3: being used, which has been really great. And I think
• 11:52
  Speaker 3: as they scale more and they use AI more over
• 11:56
  Speaker 3: the next months or so, we hope that we can
• 11:59
  Speaker 3: save them that number could even give up potentially. So
• 12:01
  Speaker 3: I think that's so far, I'd say that's the biggest
• 12:03
  Speaker 3: success we'd have we've had.
• 12:05
  Speaker 1: Once someone signs up with you and they realize that
• 12:09
  Speaker 1: they're doing all the savings, do you have to work
• 12:13
  Speaker 1: with them in an ongoing basis or is this sort
• 12:15
  Speaker 1: of a set it and forget it type of deal depends.
• 12:19
  Speaker 3: I guess so right now, just because we have so
• 12:22
  Speaker 3: few clients and we're just starting out, we like to
• 12:24
  Speaker 3: just talk to people, see what's working, see what's not working,
• 12:27
  Speaker 3: so we can improve our product. Also, yeah, it's not
• 12:29
  Speaker 3: perfect right now, there's still gonna be a couple of
• 12:31
  Speaker 3: bucks just because Yeah, this is our piloting process. So
• 12:35
  Speaker 3: I'd say, as of right now, yeah, definitely keeping contact,
• 12:38
  Speaker 3: keep communicating. But down the line, I don't see why
• 12:41
  Speaker 3: it couldn't be a set it and forget it thing.
• 12:44
  Speaker 3: I think if the executive dashboard comes along a little
• 12:47
  Speaker 3: bit further, where it gets the point where you need
• 12:50
  Speaker 3: to cut people to AI usage often whatnot, that could
• 12:52
  Speaker 3: be something you could reach out to us for. But yeah,
• 12:54
  Speaker 3: I think overall, we want as little friction as possible
• 12:58
  Speaker 3: with this. We want to feel like you're using your
• 12:59
  Speaker 3: AI before you even met us.
• 13:02
  Speaker 1: So if listen, if you remembered only one thing from
• 13:04
  Speaker 1: today's conversation, what do you hope that message is.
• 13:08
  Speaker 3: I'd say the one thing I want people to remember
• 13:10
  Speaker 3: is AI does not need to be this big, expensive
• 13:13
  Speaker 3: thing that is gonna kind of ruin your entire company budget.
• 13:17
  Speaker 3: It should really be something where the amount of money
• 13:19
  Speaker 3: you put in you should really be getting a marginal
• 13:22
  Speaker 3: impact of return on that. So I'd say just make
• 13:25
  Speaker 3: sure you're using everything efficiently and if your employees are
• 13:28
  Speaker 3: struggling to do that, you know who to call.
• 13:31
  Speaker 1: Where can our audience connect with you, learn more about
• 13:34
  Speaker 1: your work and continue this conversation.
• 13:36
  Speaker 3: Yeah, absolutely so. Our website is called zuma dot co.
• 13:40
  Speaker 3: That's zumah dot co. And if you want to connect
• 13:46
  Speaker 3: with me personally, my LinkedIn Jeremy Barry always willing to
• 13:49
  Speaker 3: chat with as many people as possible, and my friend
• 13:52
  Speaker 3: Sam Sam Harrington, He's all also willing to chat. We're
• 13:55
  Speaker 3: just trying to meet as many people as possible and
• 13:58
  Speaker 3: figure out, you know, if we're really onto something here
• 14:00
  Speaker 3: or not. So any advice, any feedback is really appreciated.
• 14:03
  Speaker 1: Excellent, Jeremy, thank you so much for joining us. We
• 14:06
  Speaker 1: appreciate your time and the value shared.
• 14:09
  Speaker 2: Thank you, Joe, appreciate it, Thanks for having me.
• 14:12
  Speaker 1: Thanks everyone for listening. If this episode inspire you, share
• 14:15
  Speaker 1: it with someone who needs it, and don't forget to
• 14:17
  Speaker 1: follow the Globe Viser podcast for more conversations that make
• 14:20
  Speaker 1: an impact. Till the next time,