Latest / Coffee with Developers / Why Token Reduction Isn’t Cost Reduction - Dave Anderson & Sarel Weinberger (PointFive)

Transcript

Why Token Reduction Isn’t Cost Reduction - Dave Anderson & Sarel Weinberger (PointFive) · Coffee with Developers

• 0:01
  SPEAKER_01: Hello and welcome to another coffee with developers.
• 0:04
  SPEAKER_01: I'm here live from my hotel room in San Francisco in the middle of the tender loin.
• 0:08
  SPEAKER_01: So if there's uh police sirens and stuff outside, don't get distracted by that.
• 0:13
  SPEAKER_01: I'm not gonna be either.
• 0:15
  SPEAKER_01: Uh today we have an interesting uh group of people, like two people in here.
• 0:18
  SPEAKER_01: We got Sarah Weinberger and Dave Anderson from Point Five.
• 0:22
  SPEAKER_01: And what they did is a research into all the different compression ideas that we have right now about AI models.
• 0:29
  SPEAKER_01: Like, especially uh when it when token usage became not a free thing, but became a very expensive thing.
• 0:34
  SPEAKER_01: A lot of people wondered how we can actually make sure that we're not burned through many tokens.
• 0:39
  SPEAKER_01: And uh almost every week there's a new cool feature that allows you to actually talk to an LLM in a different fashion.
• 0:46
  SPEAKER_01: Like there was the whole thing about like being a caveman, and there's also systems that allow you to compress the output and the input.
• 0:53
  SPEAKER_01: So they both of those did uh did a deep dive into analyzing if how many of these things are actually efficient, how many of these things claim to be snake oil, and how many of these things you can use today to actually cut down on your token usage, or what's actually not working for you.
• 1:09
  SPEAKER_01: So without further ado, let's get Dave Anderson and Sarah Leinberger on stage.
• 1:13
  SPEAKER_01: So why don't you introduce yourself quickly?
• 1:15
  SPEAKER_01: Dave, what do you do?
• 1:17
  SPEAKER_00: Uh Chris, I look after marketing for.5, cloud and AI efficiency platform.
• 1:21
  SPEAKER_00: We work with some of the top brands to help them identify cost savings and efficiencies across cloud and AI stack, which is why this research is so important.
• 1:30
  SPEAKER_01: Cool.
• 1:30
  SPEAKER_01: And Sarah, you're the tech person, I guess.
• 1:33
  SPEAKER_02: Yeah, I'm I'm leading the uh the AI research in 0.5, and we investigate a way to make the token efficient and AI efficient in general by researching and by uh producing some product that can be relevant to our customers.
• 1:48
  SPEAKER_01: So, Dave, uh that research of yours, like I mean, is there a uh is there a silver bullet to not burn through many tokens, or is there a lot of uh a lot of problems that we have that we are being sold uh solutions that are not really working out?
• 2:02
  SPEAKER_00: There's no silver bullet.
• 2:04
  SPEAKER_00: You probably already know this because you know, let's face it, the world at the moment and how fast tokens are growing and how people are using the new technologies and AI, like it's just changing very rapidly.
• 2:15
  SPEAKER_00: It changes every day, it changes every week.
• 2:17
  SPEAKER_00: We've got to keep up with these things, and there's a lot of um there's a lot of marketing hype in the market about like what you can do to reduce token costs.
• 2:26
  SPEAKER_00: And what we wanted to do is we're getting asked by our customers how do we get control over these AI costs?
• 2:33
  SPEAKER_00: How do we get visibility over it because people are burning through budgets?
• 2:36
  SPEAKER_00: And so the natural instinct for us is to put research in play to get some real answers for the state of the universe as it is today.
• 2:44
  SPEAKER_00: Like, I think that's a really important point.
• 2:45
  SPEAKER_00: Like the research that we've come out with today that Sorel also will share, is is where it's the state of AI today, and we're not saying that that's not going to be different next week or a month down the track.
• 2:58
  SPEAKER_00: And essentially, what we did with this research, if I summarize it, was a lot of people are trying to do token compression up front, and the real costs are not in the upfront token compression, the real costs are in the context of what the AI is doing as it's processing.
• 3:13
  SPEAKER_00: And essentially, the research found that by compressing tokens up front, you actually cause the AI to do more work.
• 3:20
  SPEAKER_00: It has to do more reading, and the way we explain it, it's sort of like taking words out of a sentence and then making the AI try and process it.
• 3:27
  SPEAKER_00: So, what it's doing is it's having to compress more, it's having to think more, it does more turns, therefore the costs go up.
• 3:35
  SPEAKER_00: So, what we were looking at was actual engineering sessions and what is the ROI, what is the cost of these token compressions?
• 3:45
  SPEAKER_00: And we we ran it across three or four different variables, which Sorel can go into from very light compression, which showed a little bit of cost savings, but not a lot, a lot heavier compression, which showed significant uh uh increase in costs, and then an open source third-party tool headroom, which showed like uh nearly up to 50% higher costs.
• 4:08
  SPEAKER_00: So if we want to summarize the whole research and where we are in the state of the union with AI right now, that's it.
• 4:13
  SPEAKER_00: And then when you want to dive deep, we you can uh surrel.
• 4:16
  SPEAKER_00: But yeah, essentially we're just trying to put numbers against uh what's happening in the industry.
• 4:21
  SPEAKER_01: Cool.
• 4:21
  SPEAKER_01: So that's the end of Coffee with Developers.
• 4:23
  SPEAKER_01: Thank you for that because it's an interesting one because it reminds me of like performance of applications and web app performance.
• 4:32
  SPEAKER_01: You overcomp you over compress things, and then you don't actually take into consideration what the unpacking of the data actually means.
• 4:40
  SPEAKER_01: So we optimized for the first HTTP stream coming in and not actually sending much data over the wire, but then we didn't uh didn't take into consideration how much it costs to actually unpack these things as well.
• 4:50
  SPEAKER_01: With the same with like minified JavaScript, minified CSS, uh Gsipped HTML, and all these things.
• 4:56
  SPEAKER_01: And then a machine was burning up and we wondered why why it was doing that.
• 4:59
  SPEAKER_01: And you're like, okay, because we saved 4K uh sending it up front because you optimized for mobile packages and these kinds of things.
• 5:07
  SPEAKER_01: And we seem to be the same thing right now.
• 5:10
  SPEAKER_01: Uh a lot of what I see as well is that people uh that when you let's go back a bit.
• 5:16
  SPEAKER_01: You said your customers are asking you, like, what uh what are your what are your customers?
• 5:21
  SPEAKER_01: Who who are these people who asked for that one as well?
• 5:23
  SPEAKER_01: Because I see a lot of performance uh uh posts that seem very academic rather than coming from real usage.
• 5:31
  SPEAKER_01: So uh are these big companies, are these startups, are these people that just want to use AI?
• 5:36
  SPEAKER_00: Yeah, we work with uh one of our best customers is Newbank, which is one of the world's largest digital banks.
• 5:41
  SPEAKER_00: They've had a phenomenal success working with us.
• 5:44
  SPEAKER_00: The the thing that's different, there's new bank, there's Hertz, there's Nice, there's there's a whole heap, and it actually kind of varies.
• 5:51
  SPEAKER_00: It's not we are primarily working with engineering teams.
• 5:54
  SPEAKER_00: There's like a FinOps space, and then there's like engineering.
• 5:57
  SPEAKER_00: We like to think that we work with the people that actually ship the code.
• 6:00
  SPEAKER_00: So to your point, it's like they need to think about the output.
• 6:03
  SPEAKER_00: It's not a piece of paper at the front that says, you can compress this or you can optimize this application, to your point earlier.
• 6:13
  SPEAKER_00: What are the consequences of those changes on the output, on our application, on stability, on efficiency?
• 6:19
  SPEAKER_00: How long does it take to do these things?
• 6:20
  SPEAKER_00: That's what our platform does.
• 6:22
  SPEAKER_00: And that's how we like to think we differentiate is because we give the full end-to-end scenario to the customers.
• 6:28
  SPEAKER_00: So we work with these engineering teams to do just that.
• 6:30
  SPEAKER_00: So then when it comes to the token compression, the customers are asking us like, okay, this is one of the fastest growing line items in in tech right now.
• 6:39
  SPEAKER_00: And what is everyone doing with the coding agents?
• 6:42
  SPEAKER_00: Like, how much are we spending?
• 6:44
  SPEAKER_00: How like what techniques do we have?
• 6:46
  SPEAKER_00: And even like the essence of like, we most people don't know.
• 6:50
  SPEAKER_00: They don't know what their costs are, they don't know how much they're spending, they don't know down to an individual user level.
• 6:57
  SPEAKER_00: And so we're we're doing both.
• 6:59
  SPEAKER_00: We're kind of we're providing the research into techniques that we can do to optimize, whilst also providing visibility for the teams to understand like what their what's their budget going to be and what's their burn rate.
• 7:11
  SPEAKER_01: So where can you look at this stuff right now?
• 7:13
  SPEAKER_01: Is it on your 1.5 flight?
• 7:15
  SPEAKER_01: Are we going to publish it on via developers maybe exclusively?
• 7:18
  SPEAKER_01: Or uh where where is the paper uh is there an official white paper, or is it just uh a post?
• 7:25
  SPEAKER_00: Yeah, so there's and Sorel can talk to this.
• 7:27
  SPEAKER_00: He's a data scientist by trade, so he's part of our AI research labs.
• 7:32
  SPEAKER_00: I've summarized that I'm Australian, so I have to dumb things down a little bit.
• 7:36
  SPEAKER_00: So I simplified this to a to a level that allows me to understand it and maybe the average person that's available on our website, 1.5.co.
• 7:46
  SPEAKER_00: But Sorel has obviously also published the source material and he's also published the source benchmarks on GitHub, which will allow other people to run tests as well.
• 7:55
  SPEAKER_00: So if you want to dive into that, then he's your man for that question.
• 7:58
  SPEAKER_01: Cool.
• 7:59
  SPEAKER_01: So yeah, if you take the Australian, then all the tokens fall down as well.
• 8:02
  SPEAKER_01: That's the other problem, right?
• 8:04
  SPEAKER_00: The tokens go the opposite way in Australia.
• 8:06
  SPEAKER_00: It's down under.
• 8:07
  SPEAKER_00: So for everyone, it's very different for us.
• 8:10
  SPEAKER_00: Like we already have very good compression because everything's squashed in Australia because we're at the bottom of the tunnel.
• 8:14
  SPEAKER_01: It already went down, went down the digital, and yeah, okay.
• 8:18
  SPEAKER_00: It swells in the way.
• 8:19
  SPEAKER_00: It swells in the way when it's going down the funnel.
• 8:22
  SPEAKER_00: Oh dear lord.
• 8:23
  SPEAKER_01: Okay.
• 8:23
  SPEAKER_01: So that's enough of casual erasm about Australians.
• 8:26
  SPEAKER_01: So let's go to Cyril and give us more detail into it.
• 8:29
  SPEAKER_01: So, Cyril, why don't you talk us through the setup of the research and the which tools you looked at and about the findings?
• 8:36
  SPEAKER_02: Yeah, well, uh as Dave says, our main, our main, I mean, I mean, the main the main way that we look into through this this uh this whole mission of about uh how we how we make the reduction deduction of uh of tokens is from the IOI perspective.
• 8:52
  SPEAKER_02: I mean, we are a commercial company, we have commercial uh enterprise that use our services, and we want to check not just know how much we reduce and but uh but more to check how much it's gonna cost uh every uh call for a cloud code or codex or for API, and to give them the exact or the best uh assumption about the the IOI per token use usage.
• 9:17
  SPEAKER_02: So we try to check not just you know how much uh the success uh how much the compression actually reduced from the from the code or from from cloud code, but uh as the beginning, you know, as a benchmark that we start investigating is was uh what is the ROI, what is the best ROI per compression uh uh algorithm.
• 9:40
  SPEAKER_02: So this is the main uh the main the main issue and the main uh attitude and the main effort that we we put into this paper.
• 9:48
  SPEAKER_02: Uh we call it uh compression per permission.
• 9:52
  SPEAKER_02: This is the C C CPM, whatever.
• 9:55
  SPEAKER_02: Uh and when we start investigating the the whole area, we find out very quickly, something that we we knew, but we we we did it uh through almost five thousand of uh different benchmarks, that uh most of the tokens are actually unreachable.
• 10:12
  SPEAKER_02: I can share the screen and show you how it actually looks like in uh in cloud code.
• 10:16
  SPEAKER_02: What you can see here is this is the share of cost.
• 10:19
  SPEAKER_02: And actually 94% in average, again, it's different in different sessions, and who is the exact usage, but uh in average session, 90% 94% of the usage is actually unreachable for uh for compression because it happens in the harness base, which is actually the system prompt and the tool uh definitions, and the thinking of the of the model himself, which you cannot touch if you're doing some uh naive uh compression.
• 10:47
  SPEAKER_02: You can just uh compress the uh tool output, uh you can compress your conversation history, you can uh compress your file that the uh agent coding agent can read.
• 10:59
  SPEAKER_02: But again, if you're uh you're beginning uh with six percent, so and you're doing an amazing job, so you can compress how much?
• 11:08
  SPEAKER_02: 30 percent of six, so you're you're actually limiting two or three percent of compression.
• 11:14
  SPEAKER_02: So this is the main issue that we start with when we start to investigate the the whole issue about compression.
• 11:20
  SPEAKER_02: And after we tried different uh uh different models, so we saw that actually most of the data is actually uh rely on caching, and after one usage it became a caching, and then it became even more unreachable.
• 11:34
  SPEAKER_02: You can you can delete all your caching, but then you're gonna pay much more because you know you're paying you have 90% of uh uh reduced with cost reducing if you're using the cache.
• 11:44
  SPEAKER_02: So all this this caching is again unavailable.
• 11:49
  SPEAKER_02: And again, we tried a lot of benchmarking and benchmarks and a lot of uh of different uh of different uh algorithm, and we show we we as we're going through the results, we can see uh very clearly um that uh most of the of the of the algorithm.
• 12:12
  SPEAKER_02: I mean out is the one of the most famous ones in GitHub, Headroom is another one that uh has uh became very popular recently.
• 12:20
  SPEAKER_02: Uh they both you know give you a very poor results.
• 12:24
  SPEAKER_02: I mean their claim to 50, 60, 90 percent of compression, whatever, every day they have a new a new number in the in their git.
• 12:31
  SPEAKER_02: And we saw that uh actually headroom give you, as Dave said, up to 50 50 more uh tokens in uh cost.
• 12:41
  SPEAKER_02: And uh and RTK can reduce again if it do if if it's doing a good a good job, it can reduce in two percent.
• 12:49
  SPEAKER_02: We try to try to to put some small uh small embedding model on top of RTK that helps us to identify more relevant uh results from the from the uh from the data that the tool output were actually present to the model.
• 13:05
  SPEAKER_02: And again, as we make a better better work in uh in the in the in finding relevant context and we squeeze the context, we find that uh actually we we increase the the the total costs.
• 13:19
  SPEAKER_02: Uh the numbers are spread all over the all over the the paper.
• 13:28
  SPEAKER_02: You can see that from the success rate, we didn't have uh much difference.
• 13:33
  SPEAKER_02: So uh and this is a very important issue.
• 13:36
  SPEAKER_02: I mean, how did you do the benchmarks?
• 13:38
  SPEAKER_02: Because one of the things that we we see after we start investigating is that the the benchmark that usually the models are actually uh present when they are when they are publishing the results, SWE bench or another other benchmarks are too difficult.
• 13:53
  SPEAKER_02: I mean, there's the the all every every every models, you know, uh every models provider try to show how his models solve the most complex and high performance problems that are are there.
• 14:07
  SPEAKER_02: SWE bench, which actually I think is the most popular benchmark for coding agents, is uh issues from GitHub GitHub and not just simple issues, there are those issues that we're trying to be solved many years after until someone actually finds the real results.
• 14:22
  SPEAKER_02: And it's very hard and it's not really present the normal uh way that uh that the coding agents work because most of the coding agents are uh they are dealing with much simpler uh issues.
• 14:35
  SPEAKER_02: So we create a simpler benchmark, and you can see that the success rate is really quite you know inside the confidence interval.
• 14:44
  SPEAKER_02: There are not really significant, and it's important for us because we saw it a full time that coding agents, both of the models are very good.
• 14:52
  SPEAKER_02: So you don't really need the best model for every every task.
• 14:56
  SPEAKER_02: Uh so the compression didn't really it's uh it makes some some some uh disasters in in SWE benchmark.
• 15:02
  SPEAKER_02: I mean it cuts the results in half, the success results.
• 15:06
  SPEAKER_02: Uh but we thought that it's not uh really fair to to measure the uh the compression uh algorithm through the SWE benchmark.
• 15:12
  SPEAKER_02: So we decided to uh to uh to examine it with our own benchmark that we published as Dave says before.
• 15:18
  SPEAKER_02: And we saw that the success rates actually you know it's looks quite quite the same.
• 15:25
  SPEAKER_02: But uh as you can see, the cost in RTK it's it's make a small reduction in actually the small reduction or in Opus uh uh 4.8 in Opus 5, the the the minus became a plus, so it's it's became uh more cost in uh in uh two 2.5 percent.
• 15:46
  SPEAKER_02: And when we try to compress it in a more in a more aggressive way, that look very very uh explainable and reasonable for for person, we we get the opposite.
• 15:56
  SPEAKER_02: And when we took headwind, which is the most as I said, the most uh popular uh compression uh algorithm, so we add uh almost 50 percent to the uh to the to the all uh uh to the whole process.
• 16:10
  SPEAKER_02: And we when you try to analyze what really happened, so we have some very uh strict and very uh straightforward uh uh uh results, what we can actually say that we saw that happening, and we have some assumptions and some consideration what really happened behind the scenes.
• 16:28
  SPEAKER_02: From the straightforward results, we saw that uh what actually happens is that when you compress the the data, even though you compress what looks for you as a developer like the most relevant data, uh the model became unreally comfortable with what really happened, and he became doing more and more turns.
• 16:47
  SPEAKER_02: So if maybe you think that you give him only the the necessary data, but we see that the number of turns became doubled.
• 16:54
  SPEAKER_02: So that's what that's what we saw.
• 16:56
  SPEAKER_02: That's what we saw in the numbers from uh you know straightforward telemetry.
• 17:00
  SPEAKER_02: So this is not uh a question what happened.
• 17:02
  SPEAKER_02: But the the the qu the real deep question is why why the models look uh will actually you know uh behave in this way.
• 17:09
  SPEAKER_02: And we thought that the main reason that the models actually uh behave in this way is divided into two different uh two different reasons.
• 17:17
  SPEAKER_02: Actually, the one reason is that um uh sometimes you think that there is a some batch of code that is not relevant in this specific uh turn, but the model needs it in a uh in a new turn.
• 17:30
  SPEAKER_02: So you cut something that is not look unrelevant for you, but if you're doing a coding agent uh mission that actually walked through a lot a lot of steps, so some someday you will need it.
• 17:40
  SPEAKER_02: And if we won't find it, need to reread all the files again, and then the cost became much more uh much more uh expensive.
• 17:46
  SPEAKER_02: That's what actually happened with Headroom.
• 17:48
  SPEAKER_02: Uh, a lot of time, not just uh compressed the output.
• 17:52
  SPEAKER_02: He already you know, we do another thing.
• 17:54
  SPEAKER_02: The you can actually say that the cache is unrelevant and just try to start you know the the cache from the beginning and try to play with your session.
• 18:03
  SPEAKER_02: Uh and in cloud codes, especially especially this became really disasterable, disasterable.
• 18:09
  SPEAKER_02: So that's about head on and and the other assumption that we have is that the out of time the harness, which actually defined uh how the missions look like, is very different from the way that uh we we used to see it as a person because models usually we train with their harness.
• 18:27
  SPEAKER_02: So the the harness uh stages are part of the model training, part of the model tool training.
• 18:32
  SPEAKER_02: We know that from from other training that we we did in the past.
• 18:35
  SPEAKER_02: So if we do if we if even if you give him the most uh relevant context that is that you think it's is the best because the model is actually uh uh is is trained to look on a much wider wider context, he became unconfidence in the results, and then he tried to check him with himself again and again and again and again.
• 18:54
  SPEAKER_02: So this is the the main two two issues that two assumptions that we think that really happens.
• 18:59
  SPEAKER_02: But as I said, we only know that even if you compress the the the the input and you compress it in a way that looks very, very exact as a developer as a developer, the model became very, very, very uh tedious and he became and and he actually doubled his uh turns.
• 19:19
  SPEAKER_02: So those this fact is a fact that we can see.
• 19:22
  SPEAKER_02: And the two assumptions that I I talked about before are the assumptions that I think that are very very uh convenient, but we cannot prove them for now.
• 19:32
  SPEAKER_01: Is there a bias in models that it expects like ineffective input and actually doesn't trust if the input has been optimized and that way actually starts reasoning more?
• 19:43
  SPEAKER_01: I guess I mean there's that uh it it seems to be there's an assumption of the models themselves that if you give it very efficient input, it still doesn't trust it and still does a few extra loops to say like that does that looks too optimized to me.
• 19:57
  SPEAKER_01: And that's where a lot of computation comes in.
• 19:59
  SPEAKER_01: Because a lot of the tools that you listed up earlier, and that were actually uh we had in the newsletter as well we talked about before were things like um things like caveman or uh things like token output uh uh being the uh the the terminal output of the answers being cut out before being resent to the LLM.
• 20:20
  SPEAKER_01: But it seems like the uh this is something that might work, but it still causes more computation from the model up front.
• 20:28
  SPEAKER_02: Yeah, actually, yeah, yeah, exactly.
• 20:30
  SPEAKER_02: About K-Bin, I think there is another issue.
• 20:32
  SPEAKER_02: I think K-Man actually you know trying to compress the the output, but uh but actually most of the what we we are calling the output token that we pay six times or five times more than the input tokens not are not just the the tokens that the models model actually write to us as a as a customer.
• 20:51
  SPEAKER_02: Most of the tokens, as as I as I showed before, what we call the output token, what the companies call output tokens, are actually the tokens that we can see here above is the uh the tokens that we can see here.
• 21:07
  SPEAKER_02: Yeah, uh the thinking token, yeah, the the hidden thinking.
• 21:12
  SPEAKER_02: So even if K-Bank are enabled to compress the tokens that uh the models actually write to us, so it's it's a fragment.
• 21:20
  SPEAKER_02: I mean, it's never cost more than point point zero point point four uh from the from the output tokens.
• 21:27
  SPEAKER_02: I mean, a model works in a way that uh it can pre-fill very fast.
• 21:31
  SPEAKER_02: I mean, you can read a lot of tokens uh in parallel.
• 21:34
  SPEAKER_02: That's the way that is the way that the GPU works.
• 21:36
  SPEAKER_02: It has thousands of multi-threads, but he can actually generate one tokens every time.
• 21:42
  SPEAKER_02: So that's the reason that the output tokens are very expensive.
• 21:45
  SPEAKER_02: I mean, you can read thousands of words in the same in the same second, but you can generate just one token after you read all the all those thousands of words and you read it through all the layers of the token.
• 21:56
  SPEAKER_02: So the output tokens are very, very expensive.
• 21:58
  SPEAKER_02: There's usually They cost six times more than the input tokens.
• 22:02
  SPEAKER_02: And Kevin says we want to solve the output cost.
• 22:05
  SPEAKER_02: I mean, this is a heavy part of the of the token cost.
• 22:09
  SPEAKER_02: But what Kevin didn't really realize is that they're trying to touch only the output that you can see, which is a really small fragment of the output.
• 22:18
  SPEAKER_02: Most of the output is the token that the model just actually says to himself in his head.
• 22:24
  SPEAKER_02: If you were going a year or two years ago, we would still see the thinking token of a model.
• 22:30
  SPEAKER_02: You can still see it in open model like uh like Kimi, like Dipsyq, that the real the real thinking tokens.
• 22:36
  SPEAKER_02: And we're doing research about to understand how those output tokens work in through different uh for native models that uh are open models, but uh in uh all the API models, Entropic, uh Open AI and Google Germany, they are all hidden tokens that are very afraid from no still in their token, at least still in the thiki, you know, maybe jailbreak the model.
• 22:58
  SPEAKER_02: So they didn't really uh present the those tokens.
• 23:01
  SPEAKER_02: They give you only some small summarization about what really happens, but they charge you about all the tokens that the model actually generates.
• 23:08
  SPEAKER_02: So he thinks to himself in the tokens, which are actually the output output tokens, and is and it costs a lot, and and again, and show you a small fragment.
• 23:16
  SPEAKER_02: And Kevin says Kevin says we want to compress this fragment.
• 23:19
  SPEAKER_02: So it's it's it's not harmed the model, but it's can we cannot really help?
• 23:26
  SPEAKER_01: So um these uh 96 percent, 94 and 6%.
• 23:30
  SPEAKER_01: Is there any way you can get around that?
• 23:32
  SPEAKER_02: Is there any way to self-host and get more get more access to the harness data before so there is we're working on about on some more efficient way to to improve the of course the token uh usage and cost.
• 23:47
  SPEAKER_02: And first of all, yeah, you should know that the cloud code harness is the most expensive harness.
• 23:54
  SPEAKER_02: You know, we tried different harness, we tried using entropics API with uh Fi harness and with other harness, and we saw that most of the time they are more than five times cheaper than the cloud code.
• 24:08
  SPEAKER_02: The main reason is that actually cloud code harness is very, very uh um it's trying to be very professional.
• 24:17
  SPEAKER_02: So we check him checking himself uh five or six times for every mission.
• 24:22
  SPEAKER_02: So if you have very hard missions, uh it's a good way to to make sure that you didn't have any mistakes.
• 24:27
  SPEAKER_02: But if you have small missions, again, in in our benchmarks, which actually I think more similar to day-to-day benchmark and not the you know the all the those SWE that actually represent the point zero point for something that really happens in very hard issues.
• 24:44
  SPEAKER_02: So usually the the cloud uh generation, the code generation and the and the coding agent are working a way that more, I think that it's more like seem more like our benchmarks, and then you don't need all those checking again and again and again and looking in the internet and seeing if there is some another another issues that you didn't solve.
• 25:03
  SPEAKER_02: I mean, it's it's very, very, you know, it's very very professional, but it's very, very expensive.
• 25:07
  SPEAKER_02: And I think that we should uh teach a lot of the companies that they don't need all these uh extra uh cautions when they're doing coding agents.
• 25:16
  SPEAKER_02: So, first of all, if you still want to use uh cloud code, a cloud uh opus or fibel or whatever, you can use it with another harness and it became much more cheaper very fast.
• 25:28
  SPEAKER_02: This is the first uh uh the first insight that we had.
• 25:32
  SPEAKER_02: Uh more than that, uh we know that there are problems that making the uh talking thinking uh became more wide or more uh you know or more narrow.
• 25:42
  SPEAKER_02: Yeah, uh the the the most the most you know simpler simple issue is how do you how do you have the effort level when you're using your cloud code and you can decide which effort level to use.
• 25:53
  SPEAKER_02: So most of the developers uh you know use the the best one or the default one, but we can show that uh even the the lowest one is good enough today for a lot of the well for almost 99% of the missions.
• 26:06
  SPEAKER_02: Again, it's good for Kimat for almost every every mission.
• 26:09
  SPEAKER_02: So this is the part of the insight.
• 26:12
  SPEAKER_02: And again, we're creating a new system that actually can control the the prompt.
• 26:17
  SPEAKER_02: I mean, if you give a prompt uh that is that is can actually you know stimulate the the model brain in a way that is very, very heavy.
• 26:26
  SPEAKER_02: So the thinking will be very heavy, and not just the thinking, the tool usage will be heavy, very, very, very heavy.
• 26:32
  SPEAKER_02: I mean, uh the model can decide uh the how many tool depths he can he can actually load.
• 26:39
  SPEAKER_02: As you can see here, this is the default level.
• 26:42
  SPEAKER_02: I mean, when the model don't know which tool you need to load, he loads all the tools that he has in his uh configuration, and then you have 75 percent of tool uh of uh tokens from tools from tool definition.
• 26:54
  SPEAKER_02: But if you know how to write the prompt right, and you know how to use the harness file, then you can actually use uh you actually load only the tools that you're gonna use in this specific session so you don't waste your cash.
• 27:06
  SPEAKER_02: You can you can you can squeeze your your harness space and you can uh reduce your uh your uh uh thinking tokens.
• 27:14
  SPEAKER_02: So this is much more uh reliable way that we are working on as uh automatic uh automatic uh products for our customers.
• 27:25
  SPEAKER_01: So, did you find any way?
• 27:27
  SPEAKER_01: I mean, the the uh I've seen tools that do that, that actually analyze your initial prompt and actually load the model that is most efficient for it.
• 27:35
  SPEAKER_01: Did you find something uh that automates that already, or is this the new skill that actually people have to learn to save on tokens rather than just using a magical compression library?
• 27:46
  SPEAKER_02: Um yeah, actually, we think that only models can solve problems that models create.
• 27:52
  SPEAKER_02: So we develop our models for that.
• 27:55
  SPEAKER_02: Yeah, well, I think that education is it's an important part of the issue again.
• 28:01
  SPEAKER_02: As we that's the reason that we publish in those articles, and uh we we have another one that actually publishes and will be on air in the next few days that explain uh how to choose the relevant harness and how to choose the effort level.
• 28:14
  SPEAKER_02: Uh so we think that education is an important part.
• 28:17
  SPEAKER_02: Again, I think that uh if if we were going a year before uh when most of the models were unable to actually uh process and do a good job in most of the RD emissions, so uh so you need to give them the max effort, the max after turn, so and hope that uh the output will be good.
• 28:36
  SPEAKER_02: But I think that today most of the models can do a great job, even if you have to use them in the most uh the most best basic way.
• 28:44
  SPEAKER_02: Uh so uh as I said, yeah, so we I think that we need to uh to teach the people uh how to choose a relevant model to the specific task and not just uh uh assume that they need the the the best model with the best effort for for a specific task.
• 29:01
  SPEAKER_02: Again, sometimes it's not just not improving the uh the results and it costs more, it costs more more uh always, but sometimes even reduce the the it reduced the success rate of the uh of the of the mission because the model is trying to understand what happened that you need to check again and again and again and why and try to start you know uh doing some uh calculation that actually you know worse more became worse sometimes.
• 29:30
  SPEAKER_02: So uh this is part of the education that we're trying to do with the community.
• 29:37
  SPEAKER_00: I think that's a really important point, Chris.
• 29:39
  SPEAKER_00: I think the like accountability and education for people to understand, like we have these different models, we don't really know.
• 29:46
  SPEAKER_00: They say one is better than the other, and one is good for complex tasks, and one is better for everyday tasks.
• 29:52
  SPEAKER_00: Um, but but giving people visibility into what they're spending against each one of them, and then how successful each model is at the output is part of the research that we need to do, and then also have people accountable for their own budget so they understand what they're spending, and if they know it's a complex task, they can select that model appropriately.
• 30:10
  SPEAKER_00: I'm not convinced at the moment that there's accountability.
• 30:14
  SPEAKER_00: I think everyone, it feels a little bit like we've got an open credit card and we're spending against it because we're allowed to, and eventually someone's gonna say, Hey, um, can we be a little bit more cautious with how we spend this money?
• 30:26
  SPEAKER_00: Let's be a bit more efficient.
• 30:27
  SPEAKER_00: And I think as humans, we we probably will do that because there's sustainability angles tied to this as well.
• 30:34
  SPEAKER_00: Um, so I think that's it's really important.
• 30:36
  SPEAKER_00: If it if it becomes easy and it becomes part of our nature to be able to do these things, I think people will select the better path than just blindly picking the most expensive model.
• 30:47
  SPEAKER_01: Yeah, I mean that's that's the point that I wanted to make as well.
• 30:50
  SPEAKER_01: I mean, this is not about just spending money, this is about burning energy and water and like all the things that are necessary.
• 30:56
  SPEAKER_01: And it seems like there was a scramble the last few months that people have to give their engineers access to the most amazing models because otherwise they're gonna fall behind, and there's a lot of like uh FOMO happening on LinkedIn about this.
• 31:09
  SPEAKER_01: Like, if you're not using the newest Opus Blah model that came out yesterday, your engineers will leave you.
• 31:14
  SPEAKER_01: And uh it's it feels like, yeah, I'm gonna buy the same shoes, Usain Bolt, and then I can run like him.
• 31:20
  SPEAKER_01: Uh it's you know, it's like we're we're using we're using enterprise model, uh, enterprise level things for daily tasks that are not that actually that that problematic.
• 31:31
  SPEAKER_01: So thinking about your setup, thinking about your uh what you use agentic uh development for, and uh what the use case for your company is as well.
• 31:41
  SPEAKER_01: I mean, if it's making landing pages that are basically an app that that shows something, no, you probably don't need the research model that can actually generate videos as well.
• 31:51
  SPEAKER_00: Yeah.
• 31:51
  SPEAKER_00: And it's so early though, a lot of people just don't know.
• 31:54
  SPEAKER_00: They're not they're not sure what these things cost, and they're not sure how much resource it's could it's consuming.
• 31:59
  SPEAKER_00: And yeah, to your point, like if you're just doing simple tasks, then you pick the simple model that doesn't cost that much, and you can see the output yourself more often than not.
• 32:09
  SPEAKER_00: Um but yeah, like your your analogy, like just because I get Air Jordan sixes doesn't mean I'm gonna be able to play basketball like Michael Jordan and dunk now all of a sudden.
• 32:18
  SPEAKER_00: Um it's a little, it's a it's a tool, and you still, as humans, we still need to learn how to work with it.
• 32:24
  SPEAKER_00: And I think we're all still learning.
• 32:26
  SPEAKER_00: Like that's the other thing.
• 32:27
  SPEAKER_00: These models come out and we test them within days, and then yeah, the internet blows up with like, look what you can create with this new model, and and um, and then there's a lot of people that say otherwise, they say this model's not much better than the other one, and it's twice the price.
• 32:42
  SPEAKER_01: So it's uh yeah, it needs people like us to actually uh do the newsletter and do these podcasts and these kind of things.
• 32:50
  SPEAKER_01: Like keeping up just with the changes is a full-time job, and it's absolutely uh bonkers what is out there every day.
• 32:56
  SPEAKER_01: And of course, then they're just got discussion about open models and like hosted models.
• 33:01
  SPEAKER_01: And do does my data go uh uh does my data go to America as a European and the other way around?
• 33:06
  SPEAKER_01: Like there's all kinds of discussions going on there.
• 33:09
  SPEAKER_01: Now, this is early days for you, but do you already have feedback from your customers that actually learn from these insights?
• 33:16
  SPEAKER_00: So, specifically on the compression, yeah, to some extent.
• 33:20
  SPEAKER_00: We we we launched a product um maybe a month ago now, which essentially is a control plane for all coding agents.
• 33:27
  SPEAKER_00: So essentially what it can do is understand usage, which models, who's using what costs.
• 33:33
  SPEAKER_00: At the very basics of that, people are saying without understanding, can it save money?
• 33:39
  SPEAKER_00: That's the first step.
• 33:40
  SPEAKER_00: They're like, can we just get visibility into what's going on?
• 33:43
  SPEAKER_00: There's research that shows like very few companies actually have complete visibility into um all of their AI costs.
• 33:52
  SPEAKER_00: And so the companies that are asking us, this is probably one of the hottest topics going around.
• 33:58
  SPEAKER_00: Um we're very quick to develop it, it's in GA.
• 34:01
  SPEAKER_00: And um, yeah, we have a lot of interested companies for the basic premise of who's using what and how much is it costing, and what's our forecast against it.
• 34:11
  SPEAKER_00: And it's not in the sophisticated level, like the vision for it, obviously, is also for it to do compliance and for it to do cost optimizations, but even the basic premise of like just show me what we're spending and how much we're spending.
• 34:24
  SPEAKER_00: And it's not going down to the level of like what prompts are being sent, like we don't do it at that level, so we're not we're not um we're not capturing that information, but we're literally catching within the harness, like, what are you spending?
• 34:36
  SPEAKER_00: And then eventually is that governance, and and we're working with design partners currently, and that's part of our release strategy.
• 34:42
  SPEAKER_00: Is we have a number of design partners we work with, and we're asking them, what do you need?
• 34:47
  SPEAKER_00: What do you want?
• 34:48
  SPEAKER_00: What controls do you want?
• 34:49
  SPEAKER_00: What visibility do you need?
• 34:51
  SPEAKER_00: How do you want this presented?
• 34:53
  SPEAKER_00: Um, and so we've been doing that for the last two to three months, and we've now got to the point where we're ready to release the product and and we've got really good demand for it.
• 35:04
  SPEAKER_01: Cool.
• 35:05
  SPEAKER_01: So um we're kind of running out of time and we only got point five for the podcast here.
• 35:11
  SPEAKER_01: So uh I was let's get in terms of in terms of things that you found and people do wrong, uh Cyril, do you have a few points of like where where do people feel like they're saving a lot of uh a lot of tokens, but they they don't at all?
• 35:26
  SPEAKER_01: And what can people do to really uh uh cut down on their cost?
• 35:32
  SPEAKER_02: Yeah, well, I think that um it's uh it's quite uh let's take like first of all, um it's important not to touch uh in the harness loop.
• 35:44
  SPEAKER_02: I mean the harness actually were optimized uh through a lot of uh different processes, especially cloud code harness, codex as well, and other another harness.
• 35:55
  SPEAKER_02: So uh it's very important not to try to be smarter than the harness itself.
• 35:59
  SPEAKER_02: So you can choose the best harness or the cheaper harness to your for your specific mission.
• 36:04
  SPEAKER_02: You can try to uh use the cheaper model that you can get for for your mission if it's not a very, very complex mission.
• 36:13
  SPEAKER_02: So usually it will be enough.
• 36:14
  SPEAKER_02: I mean, most of the benchmark get 97, 98% of success with the minimum effort level with the and the and the max and the minimum uh model uh capability in uh code in cloud code and in uh uh codecs as well.
• 36:28
  SPEAKER_02: Um so and never never touch the harness at the middle.
• 36:32
  SPEAKER_02: I mean you you're just gonna ruin it.
• 36:34
  SPEAKER_02: So don't intervene in the in the cloud code.
• 36:37
  SPEAKER_02: Another issue is try to work with uh different sessions for different missions.
• 36:42
  SPEAKER_02: Sometimes it's kind of confusing, but if you're if you're adding a new mission to the same session, you're if they're using the cache, you paint twice, so you need to open open a new new a new window when you think that it's reasonable not to get access to the memory of the last uh of the of the last prompt.
• 37:00
  SPEAKER_02: And the another issue is that uh you can feel comfortable with the models.
• 37:05
  SPEAKER_02: I mean, don't tell them try to do few examples, investigate the few full resource.
• 37:12
  SPEAKER_02: I mean, you just make them uh work more, and we didn't see any relevant results.
• 37:17
  SPEAKER_02: So uh so as uh the cloud call the entrepreneur actually said a few days ago, uh ask less and expect for more.
• 37:25
  SPEAKER_02: Uh you can uh reveal what is the limitation of the of the models, and then you don't need to try to push hard and uh waste money that you don't uh have to waste.
• 37:35
  SPEAKER_01: Excellent.
• 37:36
  SPEAKER_01: So uh it's the same with horses, don't touch the middle of the harness always up to front, and then you actually have more success with that one.
• 37:42
  SPEAKER_01: I don't know about horses, but our dog has a harness as well, so that works too.
• 37:46
  SPEAKER_01: So, Dave, uh what's the next steps?
• 37:48
  SPEAKER_01: Like, where can people find things?
• 37:50
  SPEAKER_01: Are you gonna be part of We Are Developers World Congress Congress in America?
• 37:54
  SPEAKER_01: I are we gonna see you on stage, are we gonna see you at our events, or how how can we how can you learn more about this?
• 38:01
  SPEAKER_00: Yeah, so obviously people can go to point five and and uh either sign up for Token Shift, which is our product that does a visibility across the coding agents, um, or obviously get in contact with us if you want to get a demo from us.
• 38:14
  SPEAKER_00: Um and we have some demos on the site.
• 38:16
  SPEAKER_00: Obviously, download the research from point five.co goes without saying.
• 38:20
  SPEAKER_00: And yeah, absolutely, we want to be part of the community with We Are developers and um we want to get out to as many people as possible.
• 38:26
  SPEAKER_00: We don't want to be a hidden secret anymore.
• 38:29
  SPEAKER_00: We're trying to expand our reach as much as possible.
• 38:31
  SPEAKER_00: And we're gonna be at a number of events this year, and and hopefully we are developers is definitely well Congress one of them that we can participate in and potentially speak at if there was an opportunity.
• 38:42
  SPEAKER_00: We'd love to share more and love it to be a debate.
• 38:44
  SPEAKER_00: That's the other really important thing is that we say that in our research and in our marketing that this is an answer, it's not the answer.
• 38:53
  SPEAKER_00: So, like we really welcome debates and more research around this so that we all can learn together.
• 38:59
  SPEAKER_00: So that's a big part of um what we do and and how we try to be part of the community.
• 39:04
  SPEAKER_01: Excellent.
• 39:05
  SPEAKER_01: Well, I mean, thank you first of all for publishing this.
• 39:07
  SPEAKER_01: Like the one of the one of the biggest annoyances to me as an open source officiado is that uh the AI world is is good at headlines and good at like uh introductory videos, but then the meat and the potatoes and the real data you never get to.
• 39:22
  SPEAKER_01: And you releasing that whole research and also the how you approached it is something that is kind of going away.
• 39:29
  SPEAKER_01: And I find it very disturbing that there's lots of privacy and uh not not privacy, there's not not much privacy, there's lots of see uh say uh uh lots of like holding behind holding out on people in the AI world.
• 39:41
  SPEAKER_01: And I think this is just like uh people are intelligent enough to get uh to make their own decisions when they get the data shown to them, and it also gives you credibility, and more companies should do that.
• 39:52
  SPEAKER_01: So thank you for being courageous enough to do that as well.
• 39:55
  SPEAKER_00: I really appreciate that comment.
• 39:57
  SPEAKER_00: I think that's that's part of our foundation of what we've always believed in.
• 40:00
  SPEAKER_00: We're part of foundations, we're part of open source communities, and we started like the cloud efficiency hub as an opportunity for people to share their ideas and and we fund that.
• 40:11
  SPEAKER_00: And some of it has overlap with our product, but ultimately we feel like if we're helping move people towards more efficient compute, then you know, you and we can be part of their journey, great.
• 40:22
  SPEAKER_00: And and if not, and they can do it themselves, that's great too.
• 40:25
  SPEAKER_00: It's like our mission is to make sure we can um help make compute more efficient.
• 40:32
  SPEAKER_00: So, whatever way we can do that through education, through podcasts like this, through research that we do, it's an important part of our culture.
• 40:40
  SPEAKER_01: Cool.
• 40:41
  SPEAKER_01: Well, thanks very much uh for your time.
• 40:43
  SPEAKER_01: Thanks, uh, thanks, Dave, thanks, Sarah.
• 40:45
  SPEAKER_01: Uh um, you can find them at point five, and we're gonna publish something about it as well.
• 40:50
  SPEAKER_01: And that must be from us today from Coffee with Developers, live from San Francisco and different other places in the US.
• 40:57
  SPEAKER_01: So see you soon.
• 40:58
  SPEAKER_01: And uh please, well, you already subscribed, so I'm not gonna say like and subscribe because that's not how we do things.
• 41:04
  SPEAKER_01: Thanks very much, Dave.
• 41:05
  SPEAKER_01: Thanks very much, Cyril.
• 41:06
  SPEAKER_01: See you soon.