Latest / Full Tech Ahead / Boost AI Token Efficiency by 20 Percent

Transcript

Boost AI Token Efficiency by 20 Percent · Full Tech Ahead

• 0:19
  SPEAKER_01: Hello and welcome to Full Tech and Head.
• 0:22
  SPEAKER_01: I'm your host, Amanda Rizzani.
• 0:24
  SPEAKER_01: And with me today, I'm excited to have DJ Sprite.
• 0:27
  SPEAKER_01: He is the head of product for ARIA Networks.
• 0:30
  SPEAKER_01: How are you doing today?
• 0:31
  SPEAKER_00: I'm wonderful.
• 0:32
  SPEAKER_00: How are you?
• 0:33
  SPEAKER_01: Doing well.
• 0:34
  SPEAKER_01: So first, let's talk about ARIA Networks.
• 0:37
  SPEAKER_01: Can you share a little bit about the services provided by ARIA Networks and how you help your clients?
• 0:44
  SPEAKER_00: Yes, uh, ARIA Networks is uh a networking company that is building solutions specifically for those that are building out uh AI factories, AI clusters.
• 0:54
  SPEAKER_00: So we have a suite of hardware and software designed to help organizations stand up, manage, operationalize, get time to first token all of the AI things that exist.
• 1:05
  SPEAKER_00: So we we have a vertically integrated solution.
• 1:08
  SPEAKER_00: We've pioneered this concept of what we like to call as deep networking.
• 1:11
  SPEAKER_00: And it is this uh vertically integrated solution that um has like deep technology all the way down into the, you know, like to the ASICs, the lowest level of the hardware to get telemetry and be able to perform actions and closed loop at the like the very lowest layer of the network at the at the highest speed possible, which is critical for these uh AI workloads.
• 1:34
  SPEAKER_00: And then being able to take that information all the way up through the stack and all the way up to a user interface and a in an agentec AI product that that you know our users will operate with that have a lot of the mon modern conveniences and and a much more you know natural language interface and have things like skills and be able to really take advantage of of the data that exists at the lower level.
• 1:56
  SPEAKER_00: And in you know, like either one of those, if you would take either of these two pieces in isolation, then you you have a very fragmented solution.
• 2:04
  SPEAKER_00: And you know, like you if you have an agentic interface but you don't have access to all the telemetry or at the rate you need or the rate that these AI systems demand, previous generations will have uh cannot collect the telemetry, they're very slow at it.
• 2:17
  SPEAKER_00: And by the time an issue has happened, you like you don't even see it.
• 2:20
  SPEAKER_00: So if you have an agentic interface and you don't have that, then you know you haven't really moved the ball forward.
• 2:24
  SPEAKER_00: And the inverse has is we collect all the telemetry in the world.
• 2:28
  SPEAKER_00: I mean, one thing is that we're able to generate more telemetry and data in these systems that outpace our ability to understand them.
• 2:37
  SPEAKER_00: And being able to reason about all this data and then the telemetry, like it all collectively, that's what we want to do.
• 2:43
  SPEAKER_00: So we provide these solutions, you know, like to our customers to help them differentiate their business and get the market faster.
• 2:49
  SPEAKER_01: Well, then you're the right person to speak with for our topic today, which is the ever-changing AI landscape and the infrastructure uh issues that business leaders are facing at the moment.
• 2:59
  SPEAKER_01: Yeah, that'd be to get started.
• 3:02
  SPEAKER_01: Let's talk about the cost of AI infrastructure.
• 3:05
  SPEAKER_01: It seems to be climbing by the day.
• 3:07
  SPEAKER_01: And what I'm hearing a lot is uh the token use is it not going very far uh for projects and it's getting more and more expensive.
• 3:16
  SPEAKER_01: So what are you seeing with your clients?
• 3:21
  SPEAKER_01: And what do you have to say about this issue?
• 3:24
  SPEAKER_00: Yeah, I mean, I think token efficiency is is I mean, it's a broad term that is an umbrella term and could be measured in many different ways, but it is a generally agreed upon concept that allows, you know, like businesses and companies to essentially measure, you know, like the return on investment in one in one dimension, right?
• 3:43
  SPEAKER_00: Like there is uh it allows you to say, like, you know, how much intelligence am I generating based on this large capital expenditure that you know that that I've just rolled out?
• 3:53
  SPEAKER_00: And it it can't be customized.
• 3:55
  SPEAKER_00: And when I say it could be measured in many different ways, is that when we engaged, like the conversations that we have with our customers, like some customers or neo clouds that we engage with maybe trying to differentiate their business and their portfolio on how much how quickly that they can give you a response.
• 4:10
  SPEAKER_00: Um, so you know, today you you interact with your, you know, like LLM of choice, sometimes the response is delayed versus if you go to some businesses who are dealing with like voice as a service or something, you don't want that delay.
• 4:22
  SPEAKER_00: So you have to deliver, you know, like tokens or intelligence very, very quickly.
• 4:26
  SPEAKER_00: And others may have specialized models like genome or or something else where they're trying to measure like their their token efficiency on a different dimension.
• 4:34
  SPEAKER_00: And it is like how large of a payload can we del what can we deploy back?
• 4:38
  SPEAKER_00: And and they're so like they have this like paradial curve, this concept where, like, depending on where you are as a business and what products that you're trying to offer, that you know, like token efficiency is a way for you to measure, you know, like how well that you are serving intelligence to you know back to your business, particularly when you've, you know, like you've laid out a large amount of money within that way.
• 4:59
  SPEAKER_00: It is, it is very important to to track and to be specific based on like how it is that you're trying to differentiate on on your business, because you don't want to, you know, like be can, you know, like you don't want to track the wrong metric.
• 5:12
  SPEAKER_00: And we work with a lot of customers and say, you know, like where it is it that you want to be, because there's a lot of ways that you can differentiate, like you want to have some sort of competitive advantage of differentiation in your business.
• 5:22
  SPEAKER_00: And so we work with them and identify like metrics directly impact or a result of like token efficiency in that specific way.
• 5:29
  SPEAKER_00: And that's how we work with our customers.
• 5:31
  SPEAKER_01: Yeah, absolutely.
• 5:32
  SPEAKER_01: So, what are some of the biggest AI bottlenecks or issues that are facing companies?
• 5:38
  SPEAKER_01: And what do you recommend on how they get past those?
• 5:42
  SPEAKER_00: Yeah, I mean, I I I I definitely say that logistics supply chain is a bottleneck right now.
• 5:48
  SPEAKER_00: I mean, there's a lot of, I would say, constraints inside in in the systems for for companies.
• 5:53
  SPEAKER_00: And depending on where they are and what their business, what their businesses look like, it could be everything from like access to just, you know, like real estate and power, or it could be uh access to, you know, the hardware and solutions that they that you know that they're actually trying to bring to market.
• 6:12
  SPEAKER_00: Yeah, I think that that's you know, that's a challenge that that everyone's having a bottleneck and working with companies and providers that have access to to the goods and to the hardware and to software that allow you to get to market sooner.
• 6:24
  SPEAKER_00: I mean it it is I think everyone have seen it now the pace of innovation is just like staggering.
• 6:30
  SPEAKER_00: This is every day, you know.
• 6:32
  SPEAKER_00: Like I, you know, I open up LinkedIn or X more often than not because it seems like X is kind of like the heartbeat and pulse of AI these days.
• 6:39
  SPEAKER_00: And like every day is just something amazing.
• 6:41
  SPEAKER_00: It's hard for me to keep up, and I am in the industry.
• 6:43
  SPEAKER_00: It's almost like you have to be unemployed to like keep up with everything.
• 6:47
  SPEAKER_00: And the businesses have to move that quickly.
• 6:49
  SPEAKER_00: So if you want to be in the market and you want to begin to take share, then you need to have access to, you know, like not only to the capital, but to the resources that allow you to like get in production and start, you know, like monetizing your business and your assets.
• 7:03
  SPEAKER_01: Absolutely.
• 7:04
  SPEAKER_01: And it seems like uh many companies are struggling with old legacy technology and that the infrastructure they're trying to use isn't really working.
• 7:13
  SPEAKER_01: So where do they start in fixing this?
• 7:15
  SPEAKER_00: It's an interesting question because uh, I mean, maybe if I take an analogy that I use, is that every large like technology shift that we've seen required a completely different mental model or a completely different rewrite of the software and and how we build, you know, like applications and and like the infrastructure to support those applications.
• 7:35
  SPEAKER_00: And if I were to, you know, I create a very strong analogy that I've used before is that for like 30 years after the in invention of like electrical generators, we still used steam engines and we just like kind of plopped the electrical engine and kept all of the you know the rest of the componentry for the steam engine in place.
• 7:52
  SPEAKER_00: So we didn't see like any efficiency gains.
• 7:54
  SPEAKER_00: And you can kind of see the same thing, you know, that happened with cloud and with the internet, and um, you know, like for you know, a while there that people tried to take what they already know and like apply it to this new world world, you know, like when cloud, you know, hit the scene, everyone tried to, you know, have virtualized infrastructure and try to run it into the cloud instead of building things that are more cloud native.
• 8:17
  SPEAKER_00: And it wasn't until we, you know, as an industry, I learned like what are the patterns and how to really take advantage of this.
• 8:24
  SPEAKER_00: It we like you rewrote the whole stack and you deployed everything, you know, like completely fresh and completely new.
• 8:30
  SPEAKER_00: And so this is kind of an analogy to that is that you have to, you know, adopt technologies that are you know that are are are built for this age and and in order to really gain the advantages out of it.
• 8:41
  SPEAKER_00: Like you can't really use the technology of of yesteryear because they were designed for different, you know, like they are optimized for different solutions.
• 8:49
  SPEAKER_00: So, you know, like in infrastructure in cloud, the applications were very resilient and they were like loosely coupled to the infrastructure underneath them.
• 8:58
  SPEAKER_00: So applications could die and they could fell over and get spun up in some other, you know, like VPC or some other region inside of your cloud provider desure and and the end user wouldn't know it.
• 9:08
  SPEAKER_00: In AI networking, that is that is very much the pendulum that swung the other way.
• 9:13
  SPEAKER_00: These are very tightly coupled systems.
• 9:16
  SPEAKER_00: It generally will drag down to the lowest common denominator, meaning by which, like when you're training models as an example, you the model completion is dictated by the the the like the slowest entity because every like all of the information you know has to go back, you know, like to one source and then bring back like all reduce and all shuffle, and there's these techniques, but it essentially the speed drags to the slowest, you know, like component.
• 9:44
  SPEAKER_00: Or on the inference side, you know, like you have we, you know, the the thing that we as consumers interact with most, like you type questions and stuff, like those are the they're very tightly coupled systems, and you can't operate in the same way that you would in the cloud era.
• 9:58
  SPEAKER_00: So you have to have much more resilient systems that are much more, you know, like tightly coupled to the application space.
• 10:04
  SPEAKER_00: Um, so it's a very different paradigm in that.
• 10:06
  SPEAKER_00: And you know, like you see, like time and time again, and it's just really businesses need to move quickly, move as quickly as they possibly can to these solutions for the same reason as above, right?
• 10:16
  SPEAKER_00: Like you want to be able to have a competitive advantage, be it you know, like speed kills, you're like or in inverse kills in this market.
• 10:25
  SPEAKER_00: So like you really have to have like a competitive advantage.
• 10:28
  SPEAKER_00: The earlier you can adopt it, I think the better, like overall as a business that you would be.
• 10:32
  SPEAKER_01: So I've been hearing the term neo clouds thrown around.
• 10:36
  SPEAKER_01: I thought I might ask you, what is that and why is that an important part of the AI topic right now?
• 10:43
  SPEAKER_00: Yeah, I've been saying, I mean, in simple terms, I'm not, I'm sure it has a very, you know, like agreed upon definition, which I can't pull out right now.
• 10:51
  SPEAKER_00: But fundamentally, it's just, you know, we have cloud providers and they host, you know, compute and databases and they host resources, you know, like fungible resources that you could build on top of.
• 11:03
  SPEAKER_00: This is think of it as like cloud 2.0, where you're getting access to GPUs.
• 11:07
  SPEAKER_00: And similarly, it's actually an interesting question.
• 11:10
  SPEAKER_00: I mean, it's timed very well to compare and contrast from like the last question you asked is it is that Neo Clouds are building very specifically for these AI workloads.
• 11:19
  SPEAKER_00: And they're they have, you know, either they're building these GPU clusters on behalf of customers, because it's you know, like it's it's a very hard and specialized skill to get up and running, and uh, and how to host them and and provide, you know, like interfaces to the customers and everything is just very different from the cloud.
• 11:38
  SPEAKER_00: So it is in in one way, is just a people who rent and provide GPU access, much like that the cloud providers would provide, you know, databases as an access, but it's so specialized uh and they need to move much faster than what the cloud providers were able to serve or what they were like, what really what their specialty was.
• 11:56
  SPEAKER_01: Great.
• 11:57
  SPEAKER_01: So AI is advancing very rapidly.
• 12:00
  SPEAKER_01: We're seeing a lot of shifts.
• 12:02
  SPEAKER_01: When you look into the future, what are some of the trends that you expect and anticipate?
• 12:07
  SPEAKER_01: And how should business leaders prepare for those?
• 12:12
  SPEAKER_00: Uh I'm gonna kind of continue the same theme.
• 12:15
  SPEAKER_00: I think that the tr the trend that I expect to see is more.
• 12:20
  SPEAKER_00: I did a blog, I'm gonna shameless plug about a blog post that I wrote, but I mean you ask my question, and it's it's very similar.
• 12:26
  SPEAKER_00: But it is that I think that there's going to be more consumers of our solutions and our products that don't have heartbeats.
• 12:34
  SPEAKER_00: I mean, there's going to be more, like right now it is very much humans in loop or humans taking advantage of these systems to gain efficiencies.
• 12:41
  SPEAKER_00: But I think very, very quickly we're going to see, you know, like the if you were to watch metrics on, you know, they have, you know, in a SaaS business, they have like DAO and they have like Mao, monthly active users and daily active users.
• 12:54
  SPEAKER_00: And I think that really those concepts are going to have to adopt something to like to agents because the more that these technologies roll out, the more that people get comfortable with them, the more that you know that we can accomplish work while we're sleeping, as an example.
• 13:10
  SPEAKER_00: I mean, I I it's it's pretty awesome.
• 13:12
  SPEAKER_00: I I use it, you know, to help build you know our product and I help to do prototyping.
• 13:18
  SPEAKER_00: And I see people that are, you know, that we work with and we're doing like two or three things at one time.
• 13:22
  SPEAKER_00: And it's it's just those, you know, downstream impacts is are going to be, you know, like what I expect to see happen is that yeah, products are going to be consumed more and more by agents.
• 13:32
  SPEAKER_00: It's probably going to happen faster than anyone thinks it is, just because like we talked about the rate of changes happening so quickly.
• 13:37
  SPEAKER_00: And it's it's it's kind of like um, you know, what is that uh that adage, like very slowly and all at once?
• 13:43
  SPEAKER_00: It's like, oh, we're gonna have robotax leads and it's like, yeah, whatever.
• 13:46
  SPEAKER_00: And then I turn around and there's like Waymo's and Teslas everywhere out of Austin.
• 13:49
  SPEAKER_00: It's like you turn around and they're everywhere.
• 13:51
  SPEAKER_00: So I think it'll be a bit like that.
• 13:53
  SPEAKER_01: Yeah.
• 13:53
  SPEAKER_01: And to that note, when it comes to ROI and expenses, what advice you have for a CIO or a CTO balancing those AI costs, where should they focus first?
• 14:06
  SPEAKER_00: Uh, I would say it's a good question.
• 14:08
  SPEAKER_00: I mean, I will give maybe like a an on an unobvious one.
• 14:13
  SPEAKER_00: And that is a lot of a lot of people generally, particularly in infrastructure, I'll speak on infrastructure for a second, it tend to try to get to the lowest price component or to drive the cost down.
• 14:26
  SPEAKER_00: But when it is that you're trying to be like this, you know, term this leading provider of intelligence, that it is counterintuitive at times that you may want to spend more, right?
• 14:38
  SPEAKER_00: Like, and there are certain areas that give you, you know, like outsized gains, right?
• 14:44
  SPEAKER_00: In the in the networking, it's a generally a small spend compared to that much larger, you know, like mini zeros bill.
• 14:51
  SPEAKER_00: But you know, a 2% gain can give you outsized impact, you know, like 10 to 20% more token efficiency.
• 14:57
  SPEAKER_00: So I think that it's worth steel manning the counter.
• 15:00
  SPEAKER_00: And that is is that, you know, how much value are you going to get instead of like what is the lowest cost that I can drive these things?
• 15:08
  SPEAKER_00: And again, I think that that may have worked generally well for previous errors, but but I think that's a counterintuitive that I think that you know, for me.
• 15:16
  SPEAKER_00: The other one is just on the human element.
• 15:19
  SPEAKER_00: I think that wherever there's pain in the system, or if you have, you know, like a lot of resources to allow them, you know, like allow your team to exp, you know, like experiment, but there is a you know, and and use these tools, and there is a fine line between, you know, like beginning to just burn tokens on on like flappy bird or something.
• 15:43
  SPEAKER_00: And so I think certainly, you know, trying to focus on on outcomes, like how quickly can we get to market, or like what is our token efficiency, or like how how many sites can we stand up, or how many prototypes have we done.
• 15:56
  SPEAKER_00: Uh, and so you try to measure and an outcome and see, you know, like how quickly you can get to that, you, you know, like using these tools.
• 16:04
  SPEAKER_00: And is that way, you know, like you, you know, show me an incentive, I'll show you an outcome.
• 16:09
  SPEAKER_01: So if there was one key takeaway that you could leave business leaders with today in regard to AI implementation and this new era that we're in, what would that be?
• 16:18
  SPEAKER_00: Yeah, I I would say in particular in with you know with ARIA, it is that the you know AI is here to stay, the cat is out of the bag, so to speak.
• 16:28
  SPEAKER_00: And it's only going to get faster.
• 16:30
  SPEAKER_00: And I and the more that you can engage with companies and the expertise that can help your business, um, that you can allow your workforce to experiment with very, you know, like very quickly, particularly on on like the outcomes for us is deploying, you know, like how quickly can we stand up, help our customers stand up, you know, GPU data centers and start, you know, time to first token.
• 16:52
  SPEAKER_00: So I I would say, you know, try to engage.
• 16:55
  SPEAKER_00: I think many people tend to want to like paint their idealistic self, that they can build a lot of these things in-house.
• 17:04
  SPEAKER_00: And what you know, the data has shown time and time and time again is that generally these these internal adoption tools like don't reap the benefits that you would want either because they drag forever or they're just not the same as what you know, like time to market is what you would get if you engage, you know, commercial solutions or expertise.
• 17:23
  SPEAKER_00: And and I think the sooner that you can gauge in companies that allow you to then get the expertise to understand how these systems are built, to have conversations that we have for deployed engineers.
• 17:37
  SPEAKER_00: It's a you know a model that many of the AI companies have because uh quite honestly, there's a it's just a skill set that is lacking in a in a lot of the companies, and even in the neo clouds or in the enterprises and et cetera.
• 17:50
  SPEAKER_00: And you need somebody to help you along the way and that you can bootstrap, you can leverage someone to bootstrap you quickly.
• 17:55
  SPEAKER_00: I I would say try to engage with companies that have the expertise in the domain um as as fast as you can.
• 18:01
  SPEAKER_00: It's not that you can't experiment, but I wouldn't, you know, like I think that both things can be true.
• 18:05
  SPEAKER_00: I just wouldn't, you know, try to, you know, pave away yourself while everyone in the industry is like passing you by.
• 18:12
  SPEAKER_01: Yeah, those collaborations and partnerships are critical.
• 18:15
  SPEAKER_00: Yeah.
• 18:17
  SPEAKER_00: Yeah, I mean, it helps.
• 18:17
  SPEAKER_00: We learn a lot, they learn a lot.
• 18:19
  SPEAKER_01: Absolutely.
• 18:20
  SPEAKER_01: Well, thank you so much for coming on the show and sharing your insights with us today.
• 18:25
  SPEAKER_00: Absolutely.
• 18:26
  SPEAKER_00: It's been a pleasure.
• 18:26
  SPEAKER_00: Hope to do it again.
• 18:27
  SPEAKER_01: Yes.
• 18:28
  SPEAKER_01: And thank you to our audience.
• 18:29
  SPEAKER_01: If you have any questions or comments, please leave them below and I'll try to respond as soon as possible.
• 18:34
  SPEAKER_01: And until next podcast, have a wonderful week.