Latest / In The Loop / What the big new AI trend Tokkenmaxxing is & why its a big problem

Transcript

What the big new AI trend Tokkenmaxxing is & why its a big problem · In The Loop

0:00
Over the last few weeks, the New York Times, Fortune, TechCrunch,
0:03
and many more have all covered the same new trend.
0:08
It's called token maxing, a new trend that's spreading literally
0:11
everywhere. Yet this month a consultancy
0:15
called Jellyfish just released a study of 12,000 developers
0:19
across 200 companies. The engineers at the bottom of
0:22
this graph that they created ship code changes for about 28 P
0:27
each. The engineers at the top of that
0:29
graph ship them for $89.00 each. The heavy users produce twice as
0:34
much output as 319 times the cost.
0:38
This is what token maxing is. Today I'm going to discuss many
0:42
of the silly stories behind this new trend with engineers gaming
0:46
leaderboards to encourage them to spend lots of money in tokens
0:50
to explain what's really happening, what token maxing is
0:54
and why the same dynamic is coming to every industry and
0:57
job, not just software. And importantly, why it's such a
1:00
big problem that you must catch at work and what you should be
1:03
doing instead. This is a loop of Jack Horton.
1:07
I hope you enjoy the show. So let's start at the top and
1:27
discuss what the new trends of token maxing actually is and and
1:30
to be honest, why it's a bit of a big problem.
1:32
So for context, in in this month, say April 2026, the CTO
1:37
of Uber just got on stage at a big industry event and admitted
1:41
that his entire 2026 AI budget was completely spent.
1:45
Now four months earlier, he'd rolled out Claude code to about
1:48
5000 engineers and and by March, 95% of them were using AI every
1:53
single day, with 70% of the code they were submitting being AI
1:57
written. But only 11% of that was
2:00
actually running in Uber systems.
2:03
So 95% of them were using AI. 70% of code that was being
2:07
submitted were created by AI, but only 11% made it to
2:11
production. These are very important numbers
2:13
and they tell a really good story about what's happening
2:16
right now. And this is going to be
2:19
something that will be mirrored across many different jobs and
2:22
industries. As I've said a few times,
2:25
programming is just ahead of the game and knowledge workers
2:29
generally are going to be going through many of the same
2:33
challenges, ideas, new ways of working that developers have
2:36
gone through over the last 18 months.
2:38
So for context, a token is roughly 3/4 of a word.
2:42
So when an LLM gives you 3/4 of a word, you span the token.
2:45
And to give you a sense of scale, the entire text of War
2:50
and Peace, the book is about 1,000,000 tokens.
2:53
And very funnily, there's there's VCs and people blogging
2:56
online and releasing articles all the way through 2026 about
3:00
burning about 250 million tokens in a single day by running a
3:05
swarm of agents all operating in parallel.
3:08
So 250 million tokens by the way, is is about roughly 250
3:13
copies of War and Peace consumed in a single day by a single
3:16
person. I think Jensen Huang of NVIDIA
3:21
said on the All In podcast recently that if a half
3:24
$1,000,000 engineer, so they pay him half $1,000,000 a year, did
3:28
not, and I quote, consume at least $250,000 worth of tokens a
3:32
year. He said he's going to be deeply
3:35
alarmed. And by March, the New York Times
3:37
had called this a basically a new status game all over Silicon
3:41
Valley. And the measure is quite simple.
3:43
Right now, it's just consume, consume more tokens, spend more
3:46
and prove that you're doing lots of stuff.
3:49
Now over the last few weeks, this story has taken an
3:51
interesting term because engineers at Meta and Microsoft
3:54
and Sales Force and a bunch of other big companies have been
3:57
found to be deliberately burning tokens.
4:00
So running lots of agents trying to have massive contacts.
4:04
Windows, which costs lots of money, going over and over on on
4:08
prompts, just looping prompts because they worked out that
4:11
their managers were using AI consumption as some form of
4:14
performance metric. And of course they gained it.
4:18
And those engineers that were at the top percentage in terms of
4:21
usage and token spend were producing about twice the
4:24
output, but at 10 times the cost.
4:27
And Meta's internal tool called Chordonomics is essentially
4:31
something that went public recently.
4:33
So they built this leaderboard ranking system and it ranked
4:37
people by the number of tokens they'd consumed.
4:40
So about 60 trillion tokens were used or spent in a 30 day
4:44
window. To put that 60 trillion tokens
4:47
into context, if every employee at Meta read a full novel every
4:52
single day for 30 years, they would collectively consume about
4:56
0.1% of that total number. And Meta by the way, has 78,000
5:01
employees. And that leaderboard handed out
5:03
badges like Token Legend, Cachet, Wizard, session
5:06
Immortal. And Meta CTO was, you know,
5:10
shouting on the rooftops, basically saying that, you know,
5:12
they're, they're top ranked engineers on these leader boards
5:15
have become 5X to 10X more productive, saying that this is
5:18
easy, this is like, you know, printing money basically.
5:22
And when it went public, of course, they then pulled the
5:24
leaderboard because it's quite embarrassing.
5:26
And I'll go into why now this is very developer focused, but
5:29
we're going to see the same in every single function in some
5:31
form. People would so desperately want
5:34
their teams to be using AI that they'll just want them to
5:37
consume tokens. You know, if token consumption
5:40
is going up, surely outputs going up.
5:42
Now the question for me, and I think you can probably tell my
5:45
opinion here, is this actually making people more productive?
6:08
So if we discuss that point, you know why I think tokens don't
6:11
mean more output. So Jellyfish that that
6:14
consultancy that released that study that I mentioned at the
6:17
start of this discussion. So they measured pull requests
6:20
shipped per quarter across 12,000 developers over 200 big
6:24
companies. Now a pull request for for
6:26
anyone outside of engineering is a standard unit of developer
6:30
work, a chunk of code that's been submitted for review, it's
6:33
been approved and then added to the product.
6:35
So shipping poor request has often been, other than merge
6:38
requests, I guess the closest thing to let's say maybe a
6:42
universal output. Now the engineers that we use in
6:45
the least AI burned around $3 of tokens over 1/4, so over three
6:51
months and shipped 11 poor requests each.
6:53
Now the engineers that we're using the most burn $1820.00 of
6:58
tokens 1/4 and ship 2313 more PRS pull requests.
7:02
So that's twice the output but about 600 times the cost.
7:06
And the word output I guess is a is a loaded word because a lot
7:10
of what the heavy users produce doesn't necessarily ship
7:13
everywhere. Remember Uber's numbers, 70% of
7:16
that committed code was written by AI, but only 11% of that
7:20
actually was running in production.
7:23
Now, it's quite easy to diagnose.
7:24
You know, just because you're producing lots doesn't mean it's
7:26
good. This is a new area.
7:29
It's an area that everybody's trying to innovate from a
7:31
process perspective, a systems perspective, trainings and skill
7:35
perspective. But also there's just natural
7:37
bottlenecks in a process. If you think about the theorem
7:40
of constraints, process will always move as fast as the
7:44
slowest moving part. And right now there's certain
7:47
areas that are just bottlenecks. You know, review is a
7:49
bottleneck. A senior engineer might be able
7:52
to review, say, 2 to 400 lines of code in a day, let's say,
7:56
yeah, AI can generate that in 90 seconds.
7:59
Now, people are doing automations on top of this and
8:01
automated testing. But still, that doesn't mean
8:04
that all of this is quality enough to pass QA.
8:08
So that is quality assurance and testing to actually get inside
8:11
the product. And again, just because you're
8:13
burning lots of tokens, are you doing it effectively?
8:16
Are you being efficient with your token spend, you know,
8:20
putting 1000 words PDF into a context window and then asking
8:24
to do, you know, lots of questions on that is going to
8:25
use loads of tokens? Are you actually achieving
8:29
anything there? Now none of this is new as a
8:31
problem. There's a principle called good
8:33
hearts law. So this is when a measure
8:35
becomes a target, it ceases to be a good measure.
8:39
Now a good example is a paper by someone called Khrushchev, and
8:43
it's about a Soviet candle factory.
8:46
And the way they evaluated progress was by weight.
8:49
Now, if the chandeliers grew heavier and heavier, so all the
8:51
workers were producing heavier and heavier chandeliers until
8:56
they started pulling ceilings down, they fulfilled the plan.
8:59
Now the question is, is does anyone really need that plan?
9:02
Do they actually solve any problem?
9:03
No, it doesn't. You're just creating heavy
9:05
chandeliers and measuring value based on that.
9:08
Bill Gates once said the same thing.
9:10
It's like measuring programmers by amount of code written is
9:13
like measuring the progress of us building a, let's say a plane
9:17
by weight has literally no value.
9:20
Wells Fargo famously measured stuff by cross sells to selling
9:24
more products to the same customers and ended up with
9:26
about 3 1/2 million fake customers.
9:29
All of this started as a reasonable proxy and KPI to try
9:32
and get people to do the right things that the organization
9:35
wanted, but obviously stopped working the moment everybody
9:39
realised that that's how they were being assessed.
9:41
And it doesn't actually deliver any value.
9:43
If that measure does not deliver value and everyone gains that
9:45
system, you're getting more of a bad thing.
9:48
And token maxing is just the newest member of that family.
9:51
It's the new trend. Now there are some people that
9:53
push back on this. You know, you could argue that
9:55
if your engineers spend 1000 a month on say LLMS and ship 10%
9:59
more, that's arguably cheap. It's not too expensive, you
10:03
know, Shopify's VP of engineering has been making the
10:06
argument very publicly. Shopify CEO makes a very similar
10:10
point that only through massive usage of AI will you get the
10:13
right skills to eventually become productive.
10:16
And and the reality is both are right.
10:19
If the output is real, then token usage is also great
10:23
because you're getting good output and clearly lots of it.
10:27
But the jellyfish numbers that study that Jellyfish released
10:30
and the example by Uber suggests that often this just still isn't
10:34
the case. And I'll come on to some of the
10:36
reasons I think this is the case, by the way.
10:39
And one of them, which is one of the reasons I got so interested
10:41
in this episode, is what mindset is actually solving for
10:44
internally and as a result has been solving for for other
10:46
customers and partners. But let's talk about, I guess,
10:49
the reason that things are getting so expensive and things
10:52
aren't necessarily improving in terms of productive output.
10:55
And as a result, what I think people should be focusing on
10:58
instead. So as I said, I, I think the
11:01
case for heavy consumption is partly right.
11:03
It's going to help people gain skills.
11:04
It's going to force people to actually adopt this technology
11:08
and get over themselves. But this still doesn't explain
11:11
why things are so expensive and things still aren't very good in
11:13
terms of quality output. And I think there's about 3 or 4
11:18
reasons. And, and the first reason I
11:19
think is, is pricing. So that previous example of
11:22
measuring people by a proxy, like lines of code written or
11:25
the weight of a chandelier, well, when those came about,
11:29
that didn't cost as much. So when a programmer in say 1982
11:32
row, let's say 500 lines of code, those lines sat on a disk
11:37
for nothing, didn't cost people money.
11:39
Whereas AI coding tools aren't sold by the seat, they're sold
11:41
by the token. So Anthropic, for example, move
11:45
their big enterprise contracts towards the late end of 2025
11:49
from a flat roughly $200 per user model with tokens included
11:53
to a more metered compute model. So you get $20.00 for a base
11:57
seat and then you get usage charge.
11:59
So when legacy contracts all over the world have started
12:02
expiring around, you know, March, April, May 2026, and all
12:06
these companies have really been drinking from the kool-aid of
12:09
token maxing, many customers bills will be tripling overnight
12:13
for Anthropic. Let's say a typical big company
12:16
runs maybe 150 to $250 a month per person, and then heavy users
12:22
run 5 to 10 times that. You can see how the numbers
12:25
start to get really expensive when you've got 78000
12:27
developers. And then when you have some
12:30
agents that are just running continuously, doing huge tasks
12:33
and using that entire token limit in just a matter of days.
12:36
Well, imagine if you have 7000 people with those agents.
12:40
Again, if you remember the episode I discussed recently,
12:42
the Open Core episode and why it's such an important software
12:46
invention and why Jensen Fuang is in love with it, is because
12:50
these things are going to consume so many tokens, which
12:53
Dr. LLMS business models which ultimately benefits NVIDIA
12:58
anyway. I think the second reason is
13:00
culture. Right now we've got this weird
13:03
culture like Matter and many of the big companies where you've
13:06
got this leaderboard of consuming tokens, you're Jensen
13:10
Wang and wanting to people to spend, you know, quarter of
13:13
$1,000,000 in tokens per year. So really this token consumption
13:17
has become quite performative in a way that you know, let's say a
13:22
company selling $20.00 per month per seat per user just didn't
13:26
see and didn't experience in the same way.
13:28
And when that metric is really publicly visible and the boss
13:31
says no limit and you get rewarded a pat on your head,
13:35
obviously people will start to play it and gain it.
13:37
I think the third reason here is a hidden quality cost.
13:41
You know, 43% of AI generated code requires lots of manual
13:46
debugging after it's gone through QA.
13:48
So testing, and there's been some studies to say that some
13:51
developers are spending about 38 to 40% of their week on
13:55
debugging and verifying whether what the agent has actually
13:59
built is the thing they meant to build.
14:02
So that heavy token usage by an agent.
14:04
So generating lots of words, although it seems really
14:07
attractive, it's only real if a human clears it up or you have
14:11
some form of really trustworthy testing software and suite that
14:15
automates that. And these things aren't always
14:18
easy to automate reliably. It's not easy.
14:21
And that clean up time isn't showing up on the token
14:23
dashboards. It shows up when you try to
14:25
deploy. And this is going to be the same
14:27
across all spaces and industries for those in every single, let's
14:31
say a law firm or a marketing team, a sales team, wherever you
14:34
are. The role of managers is also
14:36
quality assurance and quality checking.
14:38
You need to make sure what we're producing as an organization is
14:40
up to scratch. And again, maybe you can get AI
14:44
and go, yeah, AI, go and check this, but maybe you also need
14:47
people. So you have explosions of output
14:50
from some people and then lots of people having to verify that.
15:09
I think this comes to the other problem, which is contacts and
15:12
orientation of agents and then being able to reliably produce
15:15
things that you want it to produce.
15:18
Now some people are solving this with skills, with MD files to
15:22
give agents context and rules and processes.
15:25
And I've done episodes on that recently that have been really,
15:27
really popular. But I think still that it's just
15:29
a trend of the day. It will go away over the next 18
15:31
months because ultimately this system still relies on people
15:35
updating those documents, updating those skills, still
15:39
keeping them so fresh that they don't go stale.
15:42
And an agent reliably using that at scale across large
15:45
organisations, which although it's infinitely better where we
15:47
are now than 18 months ago, I still don't think it's perfect.
15:50
I think we're a way off from what will come as the next
15:53
evolution. And mindset is seeing this
15:55
because we're quite far ahead of most people from a process
15:57
perspective internally. And we've already really faced
16:01
that problem. We actually built our own tool
16:02
to solve that as an internal tool, which the team are
16:05
absolutely living. And I'll give you a good
16:07
example. So when you ask an agent, say,
16:09
to make a change in a code base, it might not have all the
16:12
context of your code base itself because it can't just, you know,
16:15
scan all of your code base because the context window blows
16:18
up. So a lot of people create
16:21
markdown files, so essentially documents which specify specific
16:25
ways things work, how my APIs work, how this works, how that
16:29
works. They also have processes they
16:31
enshrined. So how to deploy, how to push
16:34
tickets to Jira or Gitlabs, your product project management
16:36
tools. You then get people to pass
16:39
information. So a product team would go and
16:40
write up what you're trying to build in the spec, you pass it
16:43
to engineers and that might go into an MD file.
16:46
And then MD file gets given to agents and people create like
16:50
we've been doing quite complex setups, like really powerful
16:53
setups, clever setups of skills and MD files that essentially
16:58
enable an agent to operate and do good coding.
17:01
And this is the same that's happening now with called Co
17:03
work and all organisations as I've covered in other episodes.
17:07
So you create this fantastic system.
17:09
However, that doesn't mean the agent actually producing great
17:13
work. A lot of the time it produces
17:14
random work. That doesn't mean that all of
17:17
your engineers are able to suddenly use these things really
17:20
well. It doesn't mean that you can
17:22
easily track all of the key important decisions that are
17:25
made. Because in a project management
17:27
tool, you have tickets or you have tasks, but an agent doesn't
17:30
think in that way, doesn't think in a small, like tiny task, add
17:34
a headline, write subcopy, review.
17:37
It thinks in larger chunks of work, but those larger chunks of
17:41
work, therefore, when they're executed, have to rely on the
17:46
agent accessing that MD file structure, the skills that
17:49
you've created to actually do the work.
17:52
And then you have to test whether that's been done.
17:55
Because if it writes 10,000 words of or lines of code, so
17:57
let's say doesn't mean that all of that is actually what was
18:01
intended to be built. You often find things that
18:03
accidentally are snook in, you know, why did you do that?
18:06
Oh, I thought this was a good idea.
18:08
And again, I'm going to keep coming to this.
18:09
This is the same thing that's going to be happening at scale
18:12
with knowledge workers. And you know, three or four
18:15
months in or five months in from having on D files and skills,
18:19
you will start to find that suddenly some of these go stale.
18:23
People stop updating them. They don't use them properly
18:25
because naturally people just drift, especially when it's
18:28
nebulous. You know, it's, it's documents
18:31
like these things of skills that I, you know, I, I've been
18:33
promoting them and they're really great.
18:35
But they are just a document specifying how to do something.
18:39
And that information must be continually updated 24/7 for
18:42
every decision. And the more decisions,
18:44
strategies, projects you do, the more skills and MD files you
18:48
need. And it just becomes very bloated
18:50
and chaotic. Trust me, very quickly.
18:53
So this problem around token consumption relies on everyone
18:56
updating. This essentially document sets
18:59
that live somewhere. But then you need to track who's
19:01
allowed to update that document. Has this person used that skill?
19:05
Can I track if they've used that skill?
19:07
Wait, someone over here updated that document which changed our
19:09
process? OK, so we need a review process
19:12
to allow only some people to update an asset like an MD file
19:16
or a skill, but not others. But then again, things go stale.
19:20
Work isn't actually a high quality anymore.
19:23
So this is kind of what happens in a big system when you start
19:25
to really scale it and scale AI first processes.
19:29
So yeah, this is it's it's a real problem.
19:30
So you get drift from agents delivering the wrong things, you
19:33
get documents that go outdated, people not keeping everything up
19:36
to date, and essentially the entire system starts to slowly
19:40
decay and performance goes down, yet token consumption goes way
19:44
up. That's honestly one of the
19:46
reasons we've actually solved this and been solving this
19:47
internally for ourselves. We call it Memex.
19:50
It's essentially a system whereby you specify strategies
19:53
at the top and instead of using project management tasks, the
19:57
document which has access to your code base, which is
19:59
indexed, has access to specific files and other function and all
20:03
the skills turns that into tasks which are managed in a different
20:07
way. So for anyone that's really
20:09
interested in that, do reach out because we're actually having
20:11
lots of conversations about this internally to share how to use
20:14
something like this and how to implement better AI first
20:18
processes at the moment. Now I think that's a good
20:37
opportunity to now go and what I think people should be measuring
20:40
instead, how they should go about this.
20:42
Companies that are thinking about this really, really
20:46
effectively over them from a process perspective and how
20:48
we're solving that with internal tools like I described.
20:52
I think also the way you measure output is important.
20:55
So for example Salesforce have been trying to measure something
20:58
called agentic work units. So AW use as a complete counter
21:03
to token maxing instead of counting tokens consumed.
21:06
An AWU is essentially a completed unit of business
21:10
relevant work. So let's say Singapore Airlines
21:14
use AW use to track customer resolution time, another company
21:18
use them to track the quality of product recommendations.
21:21
And about 2.4 billion AWUS were generated across the system and
21:25
ecosystem in Q42025 alone. Now if you're running a team,
21:30
the job is therefore to define what your AWU is.
21:33
The unit has to represent a complete piece of work you'd
21:36
otherwise have to pay the human to do, and it has to track
21:40
whether that work actually stuck.
21:41
So for engineering, it's not necessarily pull request
21:44
generated or pull requests merged into a Co base.
21:47
It's pull requests that made it into production and stayed
21:50
there, or bugs per shipped PR. So per shipped piece of work,
21:55
how many bugs were generated? How many incidents were traced
21:59
back to AI assisted code? Or track the humour review hours
22:02
your team is burning on AI code for sales.
22:05
It could be qualified opportunities that progressed a
22:08
sales stage that was AI assisted versus not illegal.
22:13
Maybe it's contract sign, not drafts produced of content
22:17
actually published that drove traffic for marketing, not the
22:20
content generated. Again.
22:22
I keep coming back to that chandelier factory example.
22:25
What a stupid stupid. Proxy and token matching I
22:29
believe is the exact same and it will end.
22:31
And it is a big mistake to get obsessed with that.
22:34
It's maybe a good starting place to get your team using it, but
22:37
it's not going to produce value. So yeah, those organisations
22:41
that, yes, try and get people to adopt these tools but care about
22:44
how you organise contextual information that goes beyond
22:47
just MD files and skills that don't try and get people to gain
22:50
the system to spend tokens and think about what they're really
22:53
trying to measure, are going to succeed here and produce actual
22:57
value, not just trash. Anyway, that's it for this week.
23:02
Thank you for listening and I'll see you next time.