Latest / Full Tech Ahead / The 2026 GPU Shortage Crisis
Transcript
- 0:19SPEAKER_00: Hello and welcome to Full Tech Ahead.
- 0:22SPEAKER_00: I am your host, Amanda Rosani, and I'm excited to be here today with Stefan Vervat.
- 0:27SPEAKER_00: He is the co-founder and CEO of Accave.
- 0:30SPEAKER_00: How are you doing?
- 0:32SPEAKER_01: Uh, good, thank you.
- 0:33SPEAKER_01: And uh thanks for having me, Amanda.
- 0:35SPEAKER_00: I am happy to have you on the show.
- 0:37SPEAKER_00: Can you first share a little bit about your background and experience and what services do you provide there at Accave?
- 0:45SPEAKER_01: Yeah, for sure.
- 0:46SPEAKER_01: So um, first of all, I've been in the industry for in the infrastructure industry for 20 years.
- 0:52SPEAKER_01: I did two startups, uh, one in data management, another one in object storage, so scale out object storage, and kind of landed after a few years uh in the enterprise space at a crypto company uh research lab, where we used uh crypto uh to enable large ecosystems.
- 1:12SPEAKER_01: And from there, I actually built my own startup today called Akave, where I'm the CEO of with his main purpose to offer storage services to companies uh such as NeoClouds and customers that are looking for large uh scale persistent storage to preserve their precious assets, um, and doing that in a way so that it's less expensive and closer to the GPUs that they're currently using for their AI pipelines.
- 1:42SPEAKER_00: Fantastic.
- 1:42SPEAKER_00: Well, and congratulations.
- 1:45SPEAKER_01: Thank you, thank you.
- 1:47SPEAKER_00: Can you share?
- 1:48SPEAKER_00: You mentioned NeoClouds.
- 1:49SPEAKER_00: Can you share a little bit more about that?
- 1:51SPEAKER_00: We're hearing more and more that term.
- 1:54SPEAKER_01: Yeah, yeah.
- 1:55SPEAKER_01: So it's interesting.
- 1:56SPEAKER_01: Um, you know, going back to 2006 when AWS came out with object storage or S3, AWS services, specifically S3 services, um, most of the industry didn't realize or didn't want to believe that that was a serious enterprise business.
- 2:15SPEAKER_01: Today, obviously, the hyperscalers have taken over the majority of the enterprise workloads.
- 2:21SPEAKER_01: And today we're seeing another wave uh where more of the AI pipelines are moving back to what we call NeoClouds.
- 2:30SPEAKER_01: Neo clouds are specialized service providers that specifically zoom in on GPUs.
- 2:36SPEAKER_01: So they're offering GPUs as a service.
- 2:39SPEAKER_01: Um, those could be as a rental or could be brokers.
- 2:43SPEAKER_01: You know, you have some of them are actually brokering, um, basically the connection between the capital providers and the customer that's looking to purchase.
- 2:51SPEAKER_01: There's also compute marketplaces, more like aggregators that aggregate multiple compute uh providers that don't necessarily have the software but really have the bare metal.
- 3:02SPEAKER_01: They're just offering a simple compute orchestration platform for you to find a GPU provider uh on demand that you can then use for your pipelines.
- 3:12SPEAKER_01: And these new neo clouds or smaller GPU uh I wouldn't say hyperscalers, but cloud providers, they're all over the world.
- 3:22SPEAKER_01: So compute has now fully, I would say definitely in the AI space or the AI vertical, it's fully decentralized or disaggregated or distributed.
- 3:32SPEAKER_01: Um you could say it's basically owned by multiple parties, different locations, and there's a bunch of trends that we're seeing actually.
- 3:42SPEAKER_01: Like I came back from the AI infra sonnet last week here in Santa Clara, um in California, and it's very uh it's it's it's clear that prices are not going down anytime soon.
- 3:55SPEAKER_01: Um, there's a huge shortage in GPU access.
- 3:58SPEAKER_01: Definitely if you're looking for kind of the H100s or the I would say two-year-old or one-year-old uh versions or or um models, it's harder and harder to get access to those.
- 4:11SPEAKER_01: The newer, the newer models, like the B300s, the B200s, very expensive.
- 4:17SPEAKER_01: And you have to commit minimum for a year, sometimes three years, depending on the size of the cluster.
- 4:24SPEAKER_01: So we're talking about big money, knowing that in the last six months, prices have doubled, a little bit all over the place.
- 4:32SPEAKER_01: It kind of depends from system to system, but compute prices keep going up now, storage prices are going up.
- 4:39SPEAKER_01: So it's it's a very interesting world in the AI infrastructure right now.
- 4:44SPEAKER_00: Yes, absolutely.
- 4:46SPEAKER_00: Well, knowing this, what can business leaders do to sort of sort of stay ahead of this and be more prepared?
- 4:55SPEAKER_01: Yeah.
- 4:56SPEAKER_01: I mean, first of all, I think it's important to understand what your purpose you're trying to focus on, like what is the type of AI uh project that you're driving?
- 5:06SPEAKER_01: Are you looking at using your existing data sets to monetize those data sets and differentiate yourself from your competitors by uh fine-tuning, let's say, LLMs based on your private data sets?
- 5:20SPEAKER_01: Or are you in are you inferencing?
- 5:22SPEAKER_01: Are you actually providing more of a better customer experience through, let's say, running LLMs at the edge or in your applications that the customer engaged with today?
- 5:32SPEAKER_01: So those are two different uh main workloads.
- 5:36SPEAKER_01: Inferencing is very hot, lots of inferencing is happening to improve uh customer experience.
- 5:41SPEAKER_01: Let's say, you know, you go into a support service, or you're you're calling support instead of a human person, you have uh a bot or an LLM that you're engaging with.
- 5:51SPEAKER_01: That's sort of like the inferencing that we're typically seeing, or the fine-tuning, which you know increase is is really focused on your specific data sets.
- 5:59SPEAKER_01: That's where we're talking about larger clusters that are typically uh dedicated to specific to your specific workloads, whether you're using video content, images, or it's text, or maybe it's audio, depending on your your business.
- 6:13SPEAKER_01: Um, so what you need to do uh today, or what we're seeing customers look at is the business model as a whole.
- 6:20SPEAKER_01: Do we can we make the case to own the equipment ourselves and make a long-term commitment where we believe that we will find more use cases to justify the purchase or the leasing of that GPU equipment?
- 6:36SPEAKER_01: Or do we go forward with a rental uh where you know we want to still like have the flexibility to test and run proof of concepts and you know validate our pipelines that we're testing, but at a lower commit.
- 6:51SPEAKER_01: So that's first like trying to figure out what is your long-term strategy and how much of a commit are you willing to take as a whole, as a company.
- 7:00SPEAKER_01: Because if you're an AI startup, you want to move really fast, but you may not have the budget to make this long-term commitment.
- 7:08SPEAKER_01: And that's harder and harder now to get closed.
- 7:11SPEAKER_01: Meaning, if you really want like a three-month contract today for H100, or that's no longer possible.
- 7:18SPEAKER_01: It's really hard.
- 7:19SPEAKER_01: So finding the right the right neo-cloud partner is super important.
- 7:23SPEAKER_01: And if you look at this ecosystem, you have neo cloud partners that really focus on being that broker between the capital provider.
- 7:31SPEAKER_01: So they will find your capital provider and they will map you and connect your capital.
- 7:36SPEAKER_01: And you then can get like the full end-to-end experience where they will come with the software to load your uh AI pipelines.
- 7:45SPEAKER_01: Um, so you know, some of them have built-in uh orchestration, computer orchestration stack, and they will package that as part of the service.
- 7:53SPEAKER_01: Uh, others will force you to rent the equipment from them.
- 7:58SPEAKER_01: And so then you're a little bit more at their uh, how would I say?
- 8:02SPEAKER_01: Uh you know, you're sort of like dependent on that infrastructure provider because, you know, last year last week, for example, there are customers that were complaining that only had a one-year contract, their one-year contract was up, they thought they could renegotiate, but those interrupt providers, those neo cloud providers said they couldn't extend their contract because they were replacing that old equipment with brand new equipment that was yielding higher returns, was able to serve a higher level of customer or a customer that was willing to pay more money.
- 8:37SPEAKER_01: So even if you have a contract in place, um, it doesn't necessarily mean that you can extend that contract automatically, you know, you're sort of like back in the queue.
- 8:47SPEAKER_01: If you want to purchase the hardware there as well, you have to really understand that there's a huge uh shortage in GPUs.
- 8:54SPEAKER_01: So you have to be uh aligning yourself with those neo clouds that really have the allocation and can provide you a commitment on getting uh an allocation because there is well, I've heard cases with customers last week that said that they had been waiting for months to get allocation of these GPUs and yet didn't get the allocation, and we're back in the queue.
- 9:18SPEAKER_01: Um, so because a lot of customers want maybe smaller configurations.
- 9:22SPEAKER_01: What I do see is there are some new clouds that have uh eight uh that have servers with eight GPUs in them.
- 9:29SPEAKER_01: So the smaller, more deployable, I would say servers, but even that we're talking about more than half a million dollars in in price point.
- 9:37SPEAKER_01: So it's it's expensive.
- 9:38SPEAKER_01: So yeah, I would say understand your long-term commit, align yourself with the right new cloud provider, and then um look at the ecosystem of tooling that exists today because there's different orchestration stacks.
- 9:52SPEAKER_01: Um we're seeing multiple pop up.
- 9:54SPEAKER_01: Some of them are very specific to the neo cloud provider themselves, others are open source.
- 10:00SPEAKER_01: Um, so you see SkyPilot is coming out as one of the open source compute orchestration stacks.
- 10:06SPEAKER_01: Because in the end, what you want is you want flexibility to move your job, your compute job, to where the compute is.
- 10:13SPEAKER_01: And what we're seeing now is that in contrary to like the hyperscalers, right, you would sign up for a hyperscaler, you trust that hyperscaler, and your workloads would stay within that hyperscaler.
- 10:24SPEAKER_01: The hyperscalers made it really difficult to move out of the hyperscalers because they charge you an arm and a leg.
- 10:31SPEAKER_01: If you want your data back, AWS, for example, charges you up to$90 for one terabyte of data that you want to move out of the cloud, super expensive.
- 10:40SPEAKER_01: So now when you're deciding to move or take advantage of this ecosystem of neo clouds, and we're talking about thousands, right, of providers in this ecosystem, thousands of providers that are spread around the world, then you should be thinking about how do I move my data to these providers and where is my data located in the first place.
- 11:03SPEAKER_01: Um, and that's you know the second challenge that I wanted to talk about because data orchestration now becomes a real problem, knowing that this new market is fully this disaggregated across multiple vendors and multiple types of GPUs.
- 11:24SPEAKER_01: You have to be thinking about how do I set myself up for success, meaning how do I keep my data in a neutral place so it's not locked in with one of these providers, so that I have the negotiation power, the leverage to move to another provider when there's a better price, or it's uh maybe uh sovereign deployment, which we're seeing now as well, because sovereign deployments are now pretty hot overseas.
- 11:50SPEAKER_01: If you look at Canada, if you look at Europe, the need to protect uh your assets within country boundaries and you know, uh leverage uh open source LLMs instead of some of the US frontier labs models keeps coming up.
- 12:06SPEAKER_01: And so that's a whole new market as well, where compliance is sort of driving the need for local deployments and local control.
- 12:13SPEAKER_01: So, yeah, depending on where you're located, depending on the strategy you're you're willing to take, the encommitments you're trying to make or willing are willing to make, you should very uh should be very selective uh in your job, in your selection.
- 12:27SPEAKER_00: I wonder just hearing all this, it's gotta be so frustrating for business leaders because I almost wonder how is this legal that they're you know trapping them and trapping their data, their own data, not releasing it back to them.
- 12:42SPEAKER_00: Do you think that we'll soon see some regulations that maybe control a little bit of this?
- 12:48SPEAKER_00: Because I'm sure most business leaders, it's it's so frustrating to try to figure out which place to go with and not be trapped.
- 12:56SPEAKER_00: And then we have longer and learn longer commitment terms and longer amounts of time you have to give notice to break away, and it's not easy to move somewhere else.
- 13:06SPEAKER_00: So it seems so frustrating.
- 13:09SPEAKER_01: Yeah, yeah, yeah.
- 13:10SPEAKER_01: I mean, it's it is so a lot of customers are complaining about one there not being enough upper um access to GPUs, so that forces them to make a call, right?
- 13:21SPEAKER_01: They have to either purchase, make a really large purchase, which they may not have the access to, or go with a new cloud provider that uh has certain boundaries on how to, you know, how how long they can use it for or what the minimum commit is.
- 13:35SPEAKER_01: But the hyperscalers have always been more um sort of like they've been holding the data hostage of the customer because right now it's all about your data, right?
- 13:45SPEAKER_01: Because what are you actually doing?
- 13:46SPEAKER_01: You're running your fine your fine-tuning models on private data sets, your or your inferencing.
- 13:52SPEAKER_01: I mean, training is too expensive for the enterprise customers to take this on themselves, um, which is why you would take existing trained models, fine-tune them on your private data sets, and then differentiate that way.
- 14:05SPEAKER_01: But fortunately, in this new neo-cloud architecture, most of the neo-cloud providers, they will they will be more open about data moving freely across those neo-cloud providers.
- 14:18SPEAKER_01: So that is actually a good thing because they realize that that is important to for them to attract customers that are willing or are interested in using your services.
- 14:27SPEAKER_01: And so you see this movement away from vended lock-in.
- 14:31SPEAKER_01: And then it comes down to well, where is my data stored?
- 14:35SPEAKER_01: If my data is still stored in AWS or Google, then I'll have to really think thrice about how and do I have the budgets to move it out maybe once.
- 14:44SPEAKER_01: And then where do I move it to?
- 14:46SPEAKER_01: Which is why with a Kaveh, uh, we don't charge for egress.
- 14:49SPEAKER_01: Customers can store it with us.
- 14:51SPEAKER_01: We're presenting ourselves with a neutral data layer that can be used across multiple GPU partners.
- 14:58SPEAKER_01: And then the customer can more freely select a GPU partner and move their data to it.
- 15:02SPEAKER_01: But second, what we're also seeing is the cost of these GPUs and the storage that's embedded in these GPUs is going up so high that it is just not possible to store all your data in the storage, these NVMe devices that are extremely expensive, closer to that that are sitting next to the GPU.
- 15:25SPEAKER_01: So instead, um customers are asking for a local, sort of like an edge object store that sits close to the GPU, but is is accessible over network, is accessible also remotely, and doesn't have these high price points, but still has the locality, the throughputs, and offers a digital repository, right, where you can actually like store persistent data in a persistent way.
- 15:52SPEAKER_01: Because the feedback we're we're hearing is that customers will move from one pot, from one GPU pot to another GPU pot.
- 16:00SPEAKER_01: And when they do that, you kind of lose all your settings, your configuration on the cluster.
- 16:05SPEAKER_01: And when you want to restart your testing, you have to bring your snapshots back, you have to bring your LLMs, your weights.
- 16:14SPEAKER_01: And so uh the faster you can pull those back, the faster you can start using your GPU.
- 16:20SPEAKER_01: So it's super important that the the cold restart or the cold boot restart problem, right?
- 16:26SPEAKER_01: That you can solve for that.
- 16:28SPEAKER_01: Because if you're taking if you're bringing your data back again over and over again from let's say hyperscaler, you're sort of waiting for that data to come back in.
- 16:36SPEAKER_01: And these data sets are you know tens of terabytes, these snapshots, just to kind of get things going.
- 16:42SPEAKER_01: So we're seeing um a need for a local digital repository to preserve those snapshots, these weights, these LLMs in a persistent way.
- 16:53SPEAKER_01: So actually now we're moving more towards the edge, you can say, right?
- 16:57SPEAKER_01: And it's great to see that trend because um we're sort of going back.
- 17:02SPEAKER_01: We're coming back from this whole cloud move, right, to sort of a new architecture.
- 17:08SPEAKER_01: I don't think we're replacing hyperscalers.
- 17:11SPEAKER_01: This is more augmenting, right, the workloads that we currently see in enterprise, but they're more tailored, more focused towards these new AI pipelines.
- 17:20SPEAKER_01: So giving them more control.
- 17:24SPEAKER_01: Correct.
- 17:25SPEAKER_01: And then what we're seeing as well is the interest in confidential compute.
- 17:30SPEAKER_01: Um, so confidential compute is where, as you are moving towards these new neo clouds, the question then is do I trust the neo cloud?
- 17:37SPEAKER_01: Do I feel comfortable with my production data?
- 17:40SPEAKER_01: Um, so there are newer techniques now to ensure that when your data is executed or is being processed in these GPUs at these edges or in these at these neo clouds, that it's done in a confidential way.
- 17:54SPEAKER_01: So there's um now ways to you know demonstrate, like basically enforce, you know, and on the on the chip level, that the data won't be decrypted outside of the GPU.
- 18:05SPEAKER_01: So you can you can feel comfortable then that your data will only be de-encrypted uh or be unencrypted basically uh when it's actually being executed, so that the the owner, the the service provider that's serving the compute infrastructure will never have access to the actual data itself.
- 18:24SPEAKER_01: And that this is sort of like the confidential computing working group at the Linux Foundation has been driving that with the major um compute vendors and also the cloud providers, because this is not only uh a fear that customers have when using neo clouds, but you know, still with the hyperscales, they have the same challenge.
- 18:43SPEAKER_01: And thus, as customers are more sensitive now and more aware of the fact that these uh sensitive data sets or their private data sets, sort of their differentiator.
- 18:54SPEAKER_01: Uh, they're trying to protect those data sets uh more carefully.
- 18:59SPEAKER_01: And so we're seeing customers definitely in the enterprise space that are more interested in owning their own infrastructure as a sovereign deployment, maybe managed by NeoCloud.
- 19:10SPEAKER_01: Some of them are looking at having it stored or installed, deployed in their own data center.
- 19:15SPEAKER_01: But then again, a lot of the enterprise customers today have yeah, have moved a lot of their existing infrastructure into the clouds and to hyperscalers.
- 19:26SPEAKER_01: So this opens up, you know, new opportunities for those neo-cloud providers, ourselves, our Cave as well, to build new services in this new world where compute is the decentralized.
- 19:40SPEAKER_00: Well, that does, you know, that does bring to question, you know, the trust issue.
- 19:44SPEAKER_00: This data that's in many different places now bring being brought closer.
- 19:50SPEAKER_00: So perhaps it is a little safer, more trustworthy there.
- 19:53SPEAKER_00: What are your thoughts though?
- 19:54SPEAKER_00: And what tips do you have for business leaders to ensure that trust?
- 20:00SPEAKER_00: And, you know, also just to be able to access the data when they need it, or when, you know, when something goes wrong or a mistake is made, um, being able to quickly be able to sift through their data.
- 20:14SPEAKER_01: Yeah.
- 20:14SPEAKER_01: Well, I mean, so uh the sovereign deployments is definitely like a way like to isolate your environment, but still have it in a a managed OpEx type of uh Post profile.
- 20:29SPEAKER_01: Meaning, you know, a lot of customers, like I mean, I was talking to um an enterprise customer, I can't say the name, but they basically said, yeah, my my executive team wants us to move our our data lakes, you know, that are currently in Snowflake or in the AWS, but used mainly by Snowflake.
- 20:46SPEAKER_01: They want to move those back on premises, but we don't have the the people to manage it.
- 20:52SPEAKER_01: We actually don't manage on-prem storage or on-prem compute.
- 20:55SPEAKER_01: And so that's where NeoCloud is perfect, but you know, they're they would be interested in having a sovereign kind of deployment.
- 21:04SPEAKER_01: So you have a fully separated tenant, fully isolated, just for them, but managed in a colour that's hosted by this NeoCloud.
- 21:13SPEAKER_01: And um, then of course, it's important that you have uh integrated your your key manager.
- 21:20SPEAKER_01: So you have your your own key manager uh that allows you to control your keys so that you're in control over uh the encryption keys.
- 21:28SPEAKER_01: That's super important.
- 21:30SPEAKER_01: Uh, second is yeah, the confidential compute could help with that.
- 21:33SPEAKER_01: If you're really worried about um still there being a potential trust issue with the new cloud provider, you could go down the path of looking for a new cloud provider that offers confidential compute, although that's still early days.
- 21:46SPEAKER_01: So most customers are really looking for a separate tenant, fully isolated.
- 21:52SPEAKER_01: Um, they call those AI factories, right?
- 21:54SPEAKER_01: So it's completely custom kill it for the customer.
- 21:58SPEAKER_01: But if you do need extra Compute power and you need a temporary, like you basically need to like to expand on the compute side, then you can still use those compute marketplaces.
- 22:08SPEAKER_01: What we're seeing there is that some of these AI engineers that are doing tests that are still sort of like in the development phase of like testing training, not training, but building their fine-tuning models.
- 22:21SPEAKER_01: Um, they're they're doing the tests, some of these tests on these compute marketplaces that give you sort of spot prices that allow you to like more easily get access to GPUs on a short notice.
- 22:35SPEAKER_01: But there, it's important that you create snapshots of your environment so that you don't have to rebuild from scratch and you have disaster recovery or some type of recovery copy that you can restore from uh whenever you need to, because if something gets corrupted, and you know, uh there's multiple ways that things can get corrupted in a cluster.
- 23:00SPEAKER_01: Some uh engineer last week was telling me that even the phys, uh, the basically the network connectors um could overload and that could slow down performance and could intact eventually crash.
- 23:15SPEAKER_01: And so there's a lot of uh reasons why, not only software, but also hardware, obviously, wise, uh that could yeah, where you could actually lose, you know, multiple days of work.
- 23:26SPEAKER_01: Like a partner told me that a snapshot is worth minimum 300 hours of engineering hours minimum.
- 23:33SPEAKER_01: So you can calculate how much that costs to you and the time it would take to rebuild your cluster before you can actually start using your GPUs.
- 23:42SPEAKER_01: So, you know, like the time between your purchasing access to the GPUs and the time that you actually can use it, that's extremely expensive, right?
- 23:51SPEAKER_01: So if you can reduce that to minutes instead of days, yeah, you're saving a lot of money.
- 23:58SPEAKER_01: So it's not just uh the time.
- 24:00SPEAKER_01: This is not just it's not just the ability to be able to recover, but also the time it takes for you to actually spin up those uh GPUs and utilize them.
- 24:11SPEAKER_00: Absolutely.
- 24:12SPEAKER_00: Well, if there was one key takeaway you could leave our audience with today, what would that be?
- 24:19SPEAKER_01: Uh one key takeaway, uh yeah, don't don't wait.
- 24:24SPEAKER_01: I've never seen such high demand for infrastructure that it we really need to start making decisions on whether you're long-term uh committed to AI.
- 24:37SPEAKER_01: Because if you are, act quickly, mostly because at this moment September, right?
- 24:43SPEAKER_01: September is 2026.
- 24:44SPEAKER_01: We'll see what next year brings.
- 24:46SPEAKER_01: But at this point, it's getting harder and harder to get access to to capital, to to the equipment.
- 24:53SPEAKER_01: And so, you know, start now.
- 24:56SPEAKER_01: That's what I would say.
- 24:57SPEAKER_01: Even if it's like your decision is I'm gonna wait another year before I'm gonna actually deploy some of these pipelines, but um a lot of customers are not in that position.
- 25:06SPEAKER_01: A lot of customers are actually in the in a position where they're trying to roll out proof of concepts and yeah pipelines.
- 25:14SPEAKER_01: So start to start to start to lock in your your hardware or at least your rental agreements, like to to uh support your business strategy.
- 25:25SPEAKER_00: Most definitely.
- 25:26SPEAKER_00: All right.
- 25:27SPEAKER_00: Well, thank you so much for coming on the show and sharing your insights with us today.
- 25:31SPEAKER_01: Okay, thank you, Anna.
- 25:32SPEAKER_01: Thanks for having me.
- 25:33SPEAKER_00: Thank you to our audience.
- 25:34SPEAKER_00: If you have any questions or comments, leave those below, and I'll try to respond back as soon as possible.