Latest / Full Tech Ahead / Control Your AI Spending

Transcript

Control Your AI Spending · Full Tech Ahead

• 0:19
  SPEAKER_00: Hello and welcome to Full Tech Ahead.
• 0:22
  SPEAKER_00: I am your host, Amanda Razzani, and with me today, I'm so excited to have Matthew Shakstead.
• 0:27
  SPEAKER_00: He is the CEO of Parallel Works.
• 0:30
  SPEAKER_00: How are you doing today?
• 0:32
  SPEAKER_01: Hey, Amanda, doing great.
• 0:33
  SPEAKER_01: Thanks for having me.
• 0:34
  SPEAKER_00: Thank you for coming on the show.
• 0:36
  SPEAKER_00: Can you share a little bit about Parallel Works?
• 0:39
  SPEAKER_00: What services do you provide?
• 0:41
  SPEAKER_01: Sure.
• 0:42
  SPEAKER_01: Yeah.
• 0:42
  SPEAKER_01: We are a software company that makes a product called Activate that I call a computing control plane.
• 0:50
  SPEAKER_01: So it's a piece of software that organizations deploy, usually in their own cloud or on-prem boundaries.
• 0:57
  SPEAKER_01: They connect all their various computing resources together into it.
• 1:02
  SPEAKER_01: And it becomes the single pane of glass that the end users access the computing through.
• 1:08
  SPEAKER_01: And we do a bunch of more admin and operational capabilities as well for IT organizations and such.
• 1:15
  SPEAKER_01: So can talk about that.
• 1:16
  SPEAKER_00: Great.
• 1:18
  SPEAKER_00: So adding that uh visibility all in one place.
• 1:21
  SPEAKER_01: Correct.
• 1:22
  SPEAKER_01: Yes, that is definitely a big aspect of it.
• 1:24
  SPEAKER_00: Well, that brings us in our topic for the day, which is the incorporation of AI.
• 1:29
  SPEAKER_00: Of course, it's been swift and furious.
• 1:33
  SPEAKER_00: And one of the problems business leaders face is how many tokens are being used with this AI use.
• 1:40
  SPEAKER_00: And we know that the cost of AI use is increasing.
• 1:44
  SPEAKER_00: The amount of tokens needed for the AI is increasing.
• 1:48
  SPEAKER_00: And this is presenting a little bit of a challenge to companies.
• 1:53
  SPEAKER_00: So to start off, from your experience, where is the biggest problem that business leaders face right now?
• 2:03
  SPEAKER_01: Yeah, great question.
• 2:05
  SPEAKER_01: Uh, and our company is kind of right in the middle of this, as many of the organizations that we work with start bringing in frontier models from public providers, or they're building on-prem private infrastructure and deploying models there.
• 2:21
  SPEAKER_01: There's different sets of challenges in both of those.
• 2:24
  SPEAKER_01: But I'd say the biggest challenge, and I think everybody's probably seen it in the news, is you know, you open up frontier models and API keys to end users within an organization, you know, tens, hundreds, even thousands of people now starting to use these enterprise approved boundaries, you know, for their models, and the costs start skyrocketing.
• 2:50
  SPEAKER_01: And unless that is kind of thought about in the very beginning, in terms of how are you going to control and monitor token usage and capability, it it really becomes a big problem.
• 3:02
  SPEAKER_01: Like we've seen the Uber story recently, where you burn through the entire budget in a few months that was set because uh a lot of these things start as somewhat unrestricted.
• 3:13
  SPEAKER_01: And and you hit it right.
• 3:15
  SPEAKER_01: The the cost of these frontier models when you're going to public providers is expensive and getting more and more expensive kind of by the day.
• 3:23
  SPEAKER_01: So that's kind of the world that I find myself in now is one of governance around these AI models and using them effectively and giving the operators of the models and the administrators a way to kind of monitor usage for one, but actually go towards cost control and basically allocation, which I'll kind of talk about as well in the in the worlds that we came from previously.
• 3:48
  SPEAKER_01: Yeah.
• 3:49
  SPEAKER_00: Yes, absolutely.
• 3:50
  SPEAKER_00: And there are so many AI tools being used, it's hard for departments to realize, okay, who's using what tools at what level, what's really needed, what isn't, what's maximizing the best end result, what really isn't efficient and not giving a ROI.
• 4:13
  SPEAKER_00: So what advice do you have for business leaders to tackle this?
• 4:19
  SPEAKER_01: Well, I I'd say I'm I'm seeing it kind of as a trend that I've seen take place from kind of on-prem high performance computing resources where you know you bring these systems in to an internal boundary and you know, a data center that you manage, maybe, and then you open it up for pioneer pioneer usage where hey, everybody come in and use this thing, and then eventually you need to start uh allocating out time for units out, cost tokens, you know, kind of similar units, I'd say, uh, because the resources start to get constrained within some set budget.
• 4:58
  SPEAKER_01: And you know, I think what you know, maybe 10 years ago now, there's a big movement into cloud resources, at least in the the world I'm from, which is high performance computing resources in the cloud, and started moving over with kind of constrained budgets as well.
• 5:14
  SPEAKER_01: And you have, you know, it was a similar challenge.
• 5:16
  SPEAKER_01: I'd say, how do we keep control of our ballooning cloud costs?
• 5:21
  SPEAKER_01: Which starts with tracking of what's actually being used across resources, tagging policies, and then the costs start reaching a point where economics would actually let them bring certain aspects of their operations back on-prem.
• 5:38
  SPEAKER_01: And you go back to these cycles of hey, we're actually going to purchase a data center and the whole operational team to get this thing running, uh, because we can run it at a base load of 90%.
• 5:46
  SPEAKER_01: And I think that probably is a similar trend we're going to see in these AI models where organizations can get started very quickly running in these frontier provider models, you know, the public endpoints, if you will.
• 6:02
  SPEAKER_01: And you can put cost around them, and then you'll reach a point where, hey, we're at hundreds of thousands of people leveraging these things now, using it, you know, 90% baseload or whatever that that metric is that makes sense for a given organization.
• 6:17
  SPEAKER_01: And hey, let's bring it on-prem.
• 6:18
  SPEAKER_01: And we're gonna, you've seen the open weight models get better and better month over month, and that trend will continue, and there will be certain sets of things running, you know, on dedicated infrastructure on-prem, which change you know the economic story a little bit.
• 6:34
  SPEAKER_01: You still want to be able to have control and you know usage tracking on there, but you know, it's kind of a similar transition I thought that we I've seen with like cloud HPC in the last you know 10 years or so.
• 6:44
  SPEAKER_01: So uh so back to your point though, what what to keep in mind?
• 6:47
  SPEAKER_01: I'd say from the very beginning, having strong visibility in where the tokens are going.
• 6:54
  SPEAKER_01: You know, you connect in a cloud API endpoint or you know, bedrock or pick your provider, make sure from the very beginning you have visibility in who's doing what, because that's going to start growing very quickly.
• 7:06
  SPEAKER_01: If you can put guardrails around these things, uh either from the provider themselves or from like a control plane, what my company does, so you can actually plug in all the different models you want and put the same type of cost control guardrails around them, and then start watching the behavior of it and you know, have in mind that hey, maybe in a year it does make sense to actually bring some dedicated resources on-prem if you're not already doing that.
• 7:32
  SPEAKER_01: So and kind of manage it that way.
• 7:34
  SPEAKER_01: So that's my my tip kind of have visibility and guardrails set up from the very beginning.
• 7:39
  SPEAKER_01: Otherwise, you get into kind of this untethered access, which as we've seen, makes really good news stories lately.
• 7:46
  SPEAKER_00: So yeah, absolutely.
• 7:47
  SPEAKER_00: It's interesting how quickly we saw the shift to cloud, you know, migrate everything to cloud, and now how it looks like okay, actually, let's bring some of this back on premises.
• 7:58
  SPEAKER_00: Uh, there's this maybe there's a balance that's needed, not everything needs to be out going to cloud.
• 8:04
  SPEAKER_01: Yeah, I think there's a balance for sure.
• 8:06
  SPEAKER_01: And I'm, you know, my software enables organizations to create hybrid computing environments, so a mix of on-prem and cloud.
• 8:15
  SPEAKER_01: So I'm I have a have a vested interest in that positioning, but I really do believe that there's a place for both.
• 8:22
  SPEAKER_01: And as an organization kind of matures on their computing journey, just economically and kind of baseload, maybe security-wise, sovereignty-wise, there's reasons to put infrastructure inside of your own boundary, you know, on-prem and operate it that way with all the costs assumed.
• 8:38
  SPEAKER_01: And then there's really great places for cloud where, you know, they're typically getting the latest generation technology on the floor faster than anybody, faster than most organizations can get it themselves.
• 8:50
  SPEAKER_01: You're able to really leverage that uh effectively.
• 8:53
  SPEAKER_01: And you know, it kind of becomes a seamless transition to kind of move that technology from the cloud back to on-prem when you can, and then, you know, it's a continuous cycle that way.
• 9:02
  SPEAKER_01: So that's how I I've been seeing it that way.
• 9:03
  SPEAKER_01: There's places for both for sure.
• 9:05
  SPEAKER_00: Right.
• 9:06
  SPEAKER_00: And when you were talking about putting in these expenditure guardrails, is are you saying that there's a way that uh company leaders could, you know, say somebody's using these different AI tools, and once they have basically gone to the threshold of the allotted budget, they just suddenly wouldn't be able to use the tools or move forward without getting some sort of approval?
• 9:27
  SPEAKER_01: Yep, that's exactly right.
• 9:29
  SPEAKER_01: You summed it up.
• 9:31
  SPEAKER_01: So a system like ours, and there's others out there too, I'll say.
• 9:35
  SPEAKER_01: But uh we we're different in that we bring a lot of other infrastructure types into the same common allocation and kind of cost guardrail framework beyond just LLMs, I'd say.
• 9:49
  SPEAKER_01: So we're we're touching Kubernetes clusters, you know, virtualized environments, batch schedulers, cloud resources.
• 9:55
  SPEAKER_01: And we rolled out basically an AI LLM gateway because our customers are asking us, hey, how do we start actually cost controlling those as well?
• 10:05
  SPEAKER_01: So, what it allows you to do is basically from the admin perspective, plug in the different models you want your teams or your end users to use.
• 10:16
  SPEAKER_01: So you may just you may have just purchased like an OpenShift cluster and you're serving up your own open weights or you know, private weights that have been fine-tuned, and you want to make those available to your users.
• 10:27
  SPEAKER_01: You want to bring in bedrock or Azure Foundry resources, you want to bring in your enterprise cloud account, whatever it is.
• 10:35
  SPEAKER_01: You can plug those all into our platform from an admin perspective, and then start providing access to specific groups of who you want to be able to access to particular models, or users can even serve them up.
• 10:50
  SPEAKER_01: We deliver all those models through a single OpenAI compatible endpoint.
• 10:54
  SPEAKER_01: So you plug that into your code assist tool or your chat or wherever you want it, it lists all the models you have access to and the given token budget that you've been allotted.
• 11:05
  SPEAKER_01: And so users are using their models, and as soon as it basically is out of budget, it will deny the request until you get more allotted.
• 11:16
  SPEAKER_01: So that's exactly what's happening.
• 11:17
  SPEAKER_01: Yep.
• 11:18
  SPEAKER_00: So that brings me to my next question is I think that the end result of this would obviously be some really good company insight and realizing what tools are needed and at what level.
• 11:27
  SPEAKER_00: But in the beginning, is there maybe a little bit of chaos as people that are used to using certain tools suddenly are stopped in the middle of a project because they can't move further.
• 11:37
  SPEAKER_00: Um, and how do you advise business leaders deal with that process?
• 11:41
  SPEAKER_00: You know, that what I think would be a first initial chaos before, you know, it irons out.
• 11:47
  SPEAKER_01: Yeah, there's even a chaos before that, too, at least that I've been seeing where a lot of end users, like the practitioners, people doing work, uh, maybe start with even like a personal account, you know, and they're they're on their own, you know, co-pilot account or whatever, and they're using that.
• 12:02
  SPEAKER_01: And then suddenly their organization says, Hey, you need to use this enterprise mandated, you know, set of tools.
• 12:09
  SPEAKER_01: And hey, you're moving to Cloud or Pick Your Provider, OpenAI, Enterprise, et cetera.
• 12:16
  SPEAKER_01: So that's like the first level of transition.
• 12:18
  SPEAKER_01: And then, hey, you're being allocated the equivalent of$10,000 a quarter or$10,000 a month or whatever for your work, use it sparingly.
• 12:30
  SPEAKER_01: Or, and that, and that's where I think end users do need to start actually thinking about how are they using their models and actually maybe having process these that can use less expensive models for certain things, or if you're bringing this all into a hybrid infrastructure, hey, we're gonna send certain sets of tasks to our on-prem dedicated infrastructure where the token rates, what I'm being charged, are you know maybe six X less than some of the public providers.
• 12:54
  SPEAKER_01: Move these tasks over there.
• 12:56
  SPEAKER_01: So you can start to be aware of that.
• 12:59
  SPEAKER_01: I've seen usually users not unless they have a economic incentive to do so.
• 13:05
  SPEAKER_01: So, oh, they see what they're actually spending, they know they have this work to be done in the next month.
• 13:11
  SPEAKER_01: So I got I need to make that last by maybe using a, you know, our private internal model, for example.
• 13:17
  SPEAKER_01: Yeah, I guess that's my my comments there.
• 13:20
  SPEAKER_01: There is definitely chaos, but I think it just brings awareness to the actual end users, how they're actually interacting with the models, I would say more so, which is similar to what cloud, I think, was it's like here, you know, you can spin up whatever you want in the cloud, and then suddenly, oh, we have a$50 million budget a year.
• 13:37
  SPEAKER_01: Uh something needs to change.
• 13:39
  SPEAKER_00: Right.
• 13:40
  unknown: Yeah.
• 13:41
  SPEAKER_00: Do you think though that there's gonna be, because I'm hearing a lot about the increased token costs and the amount needed for these tools, that some of them were initially free, everybody started using them heavily.
• 13:53
  SPEAKER_00: Now suddenly they're costing money.
• 13:55
  SPEAKER_00: Um, other ones are costing more.
• 13:57
  SPEAKER_00: Do you think there's gonna be some pushback, especially when companies do put in these regulations and start cutting the fat, if you will?
• 14:04
  SPEAKER_00: Do you think that maybe we'll see prices go back down as they say, oh shoot, nobody wants to use these tools anymore because how expensive they are, they're cutting back.
• 14:14
  SPEAKER_00: And we'll see those prices fall back down.
• 14:16
  SPEAKER_00: Because I think right now it's sort of a money grab.
• 14:18
  SPEAKER_00: Oh, they need these tools, we're gonna make them very expensive now.
• 14:22
  SPEAKER_01: Yeah, or or actually, I mean, subscription models before you reach these enterprise plans for the frontier lab providers and such that are being subsidized, essentially, right?
• 14:33
  SPEAKER_01: So you're getting if you were to run the same processes with like an API key and pay like the per token input output rates, you're getting way, way more value until you transition to these like enterprise API keys.
• 14:46
  SPEAKER_01: And then suddenly it's like, oh, we're spending, yeah.
• 14:48
  SPEAKER_01: You've seen it in the news, you know, we just blew our entire budget.
• 14:52
  SPEAKER_01: I think that will definitely be a drive, you know.
• 14:54
  SPEAKER_01: I I want to say it will be a strong driver for organizations to seriously think about this transition from moving from just frontier models using the public providers to on-prem infrastructure, where you have a much more predictable cost control, like you saw with you know on-prem HPC and everything else.
• 15:15
  SPEAKER_01: And as the open weights get better and better, which I think we're starting to see, but you know, GLM 5.2 coming out recently.
• 15:23
  SPEAKER_01: That's that was like big news where it's really catching up in terms of capability.
• 15:28
  SPEAKER_01: You can start running open weight models for certain classes of things with much more predictable cost.
• 15:34
  SPEAKER_01: And then that's going to be a direct driver to the frontier models because, like, why is someone gonna run that if I can run similar things in on-preme infrastructure that I control three month or for three-year cadence, right?
• 15:44
  SPEAKER_01: Refresh cadence, uh, or you go rent it from a neo cloud provider or a hyperscaler, but you rent the infrastructure and host it yourself.
• 15:52
  SPEAKER_01: But at least you have very clear cost control.
• 15:55
  SPEAKER_01: You know, you know what, you know what the max consumption is gonna be.
• 15:58
  SPEAKER_01: And yeah, you still need to control the token usage because there's contention.
• 16:02
  SPEAKER_01: But yeah, so I think it's a it's definitely gonna be a big challenge as they want to increase those rates to recap all recoup all the massive investments they did, but you have you know open weight models coming in with you know on-prem infrastructure that could stand up at least some portion of what these things are doing, especially for those like base loads, where hey, we can run this thing, you know, 80% utilization reliably, and this model's good enough for what it needs to do.
• 16:28
  SPEAKER_01: That will get that will get challenged.
• 16:30
  SPEAKER_01: And I think you you kind of mentioned it the you know, you're hearing agentic workloads and really people using 10x of what we're at right now because they have you know 100 agents doing XYZ things at kind of all times.
• 16:42
  SPEAKER_01: Now I was just at some conference, they're like, Oh, yeah, the the compute need's gonna go up like a thousand times for every individual.
• 16:48
  SPEAKER_01: You know, how's that gonna actually happen?
• 16:50
  SPEAKER_00: So Yeah, and how yeah, how is that sustainable?
• 16:54
  SPEAKER_01: Sustainable from a yeah, cost model and uh, you know, it it definitely will change, I think, the the shape of the compute, you know, moving forward.
• 17:02
  SPEAKER_01: Uh, it starts unfolding, which is not you know that far away, I think.
• 17:07
  SPEAKER_00: Well, if there was one key takeaway you could leave our audience with today, what would that be?
• 17:12
  SPEAKER_01: I I'm going back to the the beginning point I had that you know, as you're rolling out these models, whether they're Frontier Lab, on hyperscale providers, Azure, you know, AWS, vertex, whatever, uh, whether you're bringing them on-prem, starting from the very beginning, putting in a system for visibility across all the different models and guardrails, if you can, you know, putting in a system for guardrails so that you do have the option to turn off the faucet if you need to.
• 17:41
  SPEAKER_01: That's my biggest, I'd say, tip right now from what I've I've been seeing.
• 17:44
  SPEAKER_00: So okay, great.
• 17:46
  SPEAKER_00: Well, thank you so much for coming on the show and sharing your insights with us today.
• 17:50
  SPEAKER_01: Sure.
• 17:50
  SPEAKER_01: Thanks, Amanda.
• 17:51
  SPEAKER_01: Good questions.
• 17:52
  SPEAKER_00: And thank you to our audience.
• 17:53
  SPEAKER_00: If you have any questions or comments, please leave them below and I'll try to respond back as soon as possible.
• 17:58
  SPEAKER_00: Have a wonderful week.