Latest / In The Loop / The reason AI is impacting only 20% of tasks

Transcript

The reason AI is impacting only 20% of tasks · In The Loop

0:00
Anthropic just released a major labour market report.
0:04
It contains a chart showing the gap between what AI could
0:07
theoretically do across every occupation and what it's
0:10
actually doing. And that gap is, well, pretty
0:14
big. I think the chart is a very
0:16
honest picture of where things stand with AI today.
0:19
Studies have found that AI could theoretically handle 94% of
0:24
tasks in computing and math is doing 36%.
0:29
In legal work apparently could handle 90%.
0:32
He's doing 20. In most categories, the gap is
0:34
even wider. Today, I'm going to cover what
0:37
is really slowing this down and why this ceiling may be just too
0:41
high to start with, and what technology leaps we must make
0:45
for AI agents to truly impact 93 and 94% of tasks across
0:52
different jobs. This is in the Loop with Jack
0:54
Horton. I hope you enjoy the show.
1:13
start from the top. And there's a couple stories
1:16
from this really interesting study, but most importantly the
1:18
chart. So Anthropic published a report
1:22
called Labor Market Impacts of AI, a new measure and early
1:26
evidence. And it introduced something they
1:27
called observed exposure, which is basically a metric that puts
1:31
theoretical AI capability next to real world usage data to see
1:36
which jobs are actually being impacted to try and get get away
1:40
a little bit from the hype and the media cycles.
1:43
And the report is summarized well with the radar chart.
1:46
So if you can picture those almost a circular chart and it
1:49
looks like there's a spider's web in the middle of it.
1:51
They used to tell the user at school to rate how good I was at
1:54
different skills around cooking in cooking class.
1:57
And essentially what you see in this image is a large blue
2:00
spider diagram that stretches out towards the edges for a
2:03
bunch of different occupations of management, business,
2:07
computer, architecture, life sciences, social services,
2:11
legal. And that's a theoretical impact
2:13
of AI on these occupations. And I can tell you it's a much,
2:16
much bigger spider web than the red one, which is the actual
2:20
current observed usage. And this is measured by
2:23
Anthropic's own data. And the numbers are really
2:25
interesting between how much AI could theoretically transform
2:28
these occupations versus observed usage like computer and
2:31
maths. 94% theoretical but only 33% observed.
2:37
Legal, meaning 90% theoretical but barely passed. 20% observed.
2:42
Almost across every domain. It's the same pattern apart from
2:45
a bunch of domains that basically hasn't been impacted
2:47
at all. You've got, you know, grounds
2:49
maintenance, food serving, agriculture, construction,
2:53
installation and repair. None of those have been touched
2:55
by AI really. Obviously Anthropic's
2:57
interpretation is that this is a major growth story.
3:00
They said, and I quote, as capabilities advance, adoption
3:03
spreads and deployment deepens, the red area will grow to cover
3:07
the blue area. And in their framing, the red is
3:10
the blue's younger self. Give it time and convergence is
3:13
obviously inevitable. However, you could read it
3:15
differently. That gap could also be telling
3:18
us where the boundaries are, not where it's heading.
3:21
And Anthropic is looking at that distance and seeing potential.
3:24
Whereas, as we know, some people will see it as a big chasm that
3:27
may or may not ever be crossed. So yeah, let's discuss that.
3:48
So from one perspective, you could argue that the ceiling is
3:50
actually lower than it looks. So that blue shape, that
3:53
theoretical ceiling comes from a 2023 Open AI paper called GPTS
3:58
Are GPTS. It's almost three years old now.
4:01
The authors went through every task in the ONET database, which
4:04
is a primary source or US source of occupational information,
4:08
covering about 900 jobs. And in the study, Open AI scored
4:12
each 11 if an LLM alone could double the speed, 0.5 if it
4:17
could with additional tools, 0 if it was out of reach.
4:21
Anthropic adopted this as the upper bound of their chart.
4:25
The issue is how those scores were generated though.
4:27
In the study, they were asking whether a task could be speed up
4:30
by a model with, say, a 2000 word input limit and no access
4:33
to current information on the Internet.
4:36
That's not measuring what happens when the AI has to be
4:39
integrated into a complex procurement workflow at bank.
4:42
It's not measuring what happens when the output has to pass a
4:44
legal review, or when the data lives in 3-4 different systems
4:49
that just don't talk to each other, or when the person using
4:52
it doesn't trust it. It's measuring what's possible
4:54
in a very clean, you know, very controlled hypothetical
4:58
environment, and then trying to project that across the entire
5:02
economy of the occupation. Anthropic does flag this to be
5:06
fair to them, but their argument is that the models just have
5:09
improved, so the real ceiling is probably much higher than the
5:12
bully suggests. But you can run that logic in
5:14
the opposite direction too, you know.
5:16
The original scoring was probably way, way, way generous
5:20
because it was measuring what a model could do under perfect
5:22
conditions with no friction, no context, no complexity, and also
5:28
during a period when hype around LLMS was at its absolute peak.
5:32
So if anything, you could argue the blue is too big, not too
5:34
small. See, I, I, I think studies like
5:36
this are really helpful to understand, but not obviously
5:39
where I get very interested. We should come on to, I guess in
5:41
the secondary part of this discussion, because I've talked
5:44
a lot about this. Benchmarks these days are, well,
5:48
just as much about the importance of vibes.
5:51
So the vibes of this new LLM and how it feels and B, the
5:55
complexity of real world deployment that that's what
5:57
these benchmarks rarely capture. So the question I'm actually
6:00
interested in is, how do we make this translate into a situation
6:06
where conditions aren't perfect? Because these benchmarks are all
6:10
in perfect conditions. The chart suggests that the
6:12
answer is not nearly as simple as we'd like to think, because
6:16
that red spider's web is much, much smaller than the blue
6:19
theoretical one. Let's explore that.
6:21
Why isn't AI impacting more tasks?
6:23
I want to say something quite clearly before we get into this
6:26
discussion because it could sound like I feel somewhat
6:29
negative about the situation where it's not actually where I
6:33
land. I would describe myself as a
6:37
realist argument about the situation because I believe AI
6:40
is going to be transformative. I work in the industry, we
6:43
actively build systems and products to deploy it into big
6:48
companies. So yeah, it's more of a
6:50
realistic argument about how long transformation actually
6:52
takes before technology becomes truly upscale.
6:55
Anyway, there are a few things to explain this gap.
6:58
I think in theoretical expectation and reality 1 is
7:03
about the technology, which something we as in mindset are
7:06
actively working on. So it's really interesting and
7:09
can really give a kind of insider's perspective on.
7:12
And the others are human problems.
7:15
And I think really the human and organizational problems are the
7:18
hardest to fix. Let's start with the people
7:37
problem. The first problem is that the
7:39
adoption curve is more of a Cliff rather than a slope right
7:42
now in AI open AI S own numbers tell us a really good story.
7:46
ChatGPT has about 900 million active users or weekly active
7:49
users less now after the quick GBT movement.
7:52
And about 50 million of those are actually paying power users.
7:55
So the top 5% of those paid subscribers use reasoning
7:59
capabilities about 7 times more than a median paying customer.
8:04
So if we do the maths on that, 5% of 50 million is 2.5 million,
8:08
that's 0.25% of the total user base actually using AI in a way
8:12
that could truly reach that theoretical expectation of how
8:17
AI will impact an occupation. Most people therefore we can
8:20
assume are using it for very basic things or in a way that
8:23
isn't going to transform their occupation if this were a normal
8:26
adoption curve. So in technology, when a new
8:29
technology appears, we have we see something called an S curve.
8:32
So it's a standard model for how new technologies get adopted.
8:35
It looks like a flattened S on its side, which means a slow
8:39
uptake at the start. Early adopters are small
8:42
numbers, then a steep acceleration in the middle as
8:44
mainstream users pile in, which is called the tipping point.
8:48
And then it kind of flattens out at the top as you hit saturation
8:51
and only really the laggards are left to start using it.
8:55
Well, if we're experiencing that the middle should be filling by
8:57
now. And you could argue there is and
9:00
that when you're in the middle of it you are very impatient.
9:03
But 3 1/2 years after ChatGPT launch and it isn't.
9:06
What we're looking at is close to a Cliff edge right now.
9:08
A tiny spike of heavy users right at the top and then the
9:11
near vertical drop to everyone else.
9:13
So a handful of people have obviously figured it out and a
9:16
vast majority of people are just bounced off or haven't found the
9:19
use case or skill set to make it stick.
9:21
And I'll say this because the data seemingly points to it.
9:24
I think probably have a skill issue.
9:26
Now, it could be me being impatient here.
9:29
And therefore that Cliff will become that S curve over time
9:32
and it's just going to take a little bit longer.
9:34
We'll have to see. But I don't think most people
9:36
have developed the ability to use these tools in ways that
9:39
produce seriously meaningful results.
9:41
And nobody is teaching them how to do this at scale.
9:45
I bet it's not being taught at school.
9:47
The second problem is that the economy moves far slower than I
9:50
guess the AI industry wants to admit.
9:52
So large companies don't RIP out enterprise systems they're
9:55
running on because something better came along.
9:57
They're locked into 1218, two year contracts with platforms
10:01
that, you know, meet their needs quite well.
10:03
Even if those platforms aren't AI native and aren't that
10:06
impressive, people will be forced into waiting.
10:10
Now that that will last for long, you know, by the end of
10:12
that contract, they'll be at risk of losing or they'll be
10:16
forced to start integrating other companies into that
10:18
solution, which will slowly eat and eat and eat away at, I guess
10:23
the perceived value of the big company.
10:25
So the cycle time for these enterprise software deployments
10:28
are measured in years. You know, some companies take
10:31
1218 months to deploy your technology.
10:34
So therefore there will be a slow adoption enterprise level,
10:39
even if they've all adopted ChatGPT, there's still 10s of
10:42
thousands of other SAS of technology platforms who are
10:45
solving problems and who are not AI native.
10:48
And you know, only 3% of Microsoft 36 fives, 450 million
10:52
users have even adopted or opted into Copilot yet.
10:56
So again, it will take time. The third is that the technology
11:00
stuck is actually harder to integrate into apps people use
11:03
daily. And so I see it as also a supply
11:06
problem. So what I mean here is that you
11:08
see the supply side every single day.
11:10
So people might use AI if it was in front of them in a technology
11:14
solution they already use everyday, but it's not.
11:16
So they therefore would have to learn to be an expert on other
11:20
AI native tools. So the claws, the chat, GBTS,
11:22
the many, many others, because we work with technology
11:25
companies, SAS companies, typically large B to B SAS
11:28
companies deploying agents into their products.
11:31
And the complexity is really, you know, quite staggering when
11:33
you go from, say, a prototype or a quick demo, You need an entire
11:37
stack to make all of this work. You got huge amounts of
11:41
complexity to embed and integrate these into solutions.
11:44
You need to identify how it's all going to work and even the
11:46
use cases that you want to deploy for.
11:48
First, you know, you need an agent framework.
11:49
You need reasoning and logic. You need to wire this into all
11:52
your systems. You need to understand how
11:55
agents understand data and APIs. You need to build APIs for that.
12:00
You need to create MCPS. You need to call it, create a
12:02
tool calling layer. You need to then have a testing
12:04
layer. You need to be able to surface
12:06
the right information and test that that's the right
12:08
information because big enterprise companies really care
12:11
about compliance. You need a purpose built
12:13
interface for every type of interaction, like widgets
12:15
appearing in charts and charts. All this has to then work at
12:19
production scale, not just for, you know, 2000 people or 100
12:22
people. And I'd say that about 75% of
12:24
the companies we talked to are either just getting started or a
12:28
little bit into a build. And these platforms aren't small
12:31
though, you know, companies with 10/15/20 thousand enterprise
12:35
customers worldwide. And so without that supply, IE
12:38
the big companies that already have millions of users
12:41
integrating this into their products in a really effective
12:44
way, it's really hard for most people to use AI.
12:47
So the real question therefore is will it grow?
12:49
And I'm confident that it will. And I think there's a really
12:51
core component to what will make it grow and that's making the
12:54
technology itself reliable. So this comes to the fourth
12:57
reason I think that there's still a gap between theoretical
13:01
and real world usage. And this is most interesting to
13:03
me because it's what we're obsessed, we're trying to solve
13:05
right now. It's, you know, something that
13:07
we actively are always in at mindset.
13:09
So essentially there was a really good paper that
13:11
highlights this point and discussion here.
13:14
And it's called Towards a Science of AI Agent Reliability.
13:18
And it essentially puts rigor behind something.
13:20
You know, I've been saying on this podcast for a long time
13:22
that the industry is often measuring the wrong thing.
13:25
You know, you draw a line between capability, So can a
13:29
model do this task and reliability?
13:31
Will it do it consistently, safely and predictably in
13:35
conditions that are rarely perfect?
13:37
And we don't just mean that they do the right thing most of the
13:39
time. We mean something much more.
13:42
And this is what the paper breaks down really
13:44
interestingly. A colleague and therefore an AI
13:46
agent must be able to #1 be consistent.
13:50
This means they get it right consistently, and that means not
13:53
right today and therefore wrong tomorrow on the same task #2
13:57
Robustness. They can't just fall apart soon
14:00
as conditions aren't perfect for them #3 Calibration.
14:04
They're able to tell you when the unsure rather than
14:07
confidently guessing and #4 is safety.
14:10
So when they do make a mistake, their mistakes are fixable, and
14:13
they're not just catastrophic failures.
14:15
Again, what you'd expect from a colleague.
14:17
And to assess agent reliability right now, the authors broke
14:20
down all of this into 12 metrics across 4 dimensions and tested
14:26
14 different models from Open AI, Google, Anthropic, spanning
14:31
a year and a half of product releases.
14:33
They run each task five times with different sets of
14:36
paraphrase instructions. And they also deliberately broke
14:39
things. So they tried to cause tool
14:41
failures and API calling failures.
14:44
They tried to simulate environmental faults to see how
14:47
agents handled imperfect conditions.
14:51
They asked the models to report their own confidence to test
14:53
whether they could tell when they're wrong.
14:55
Overall they had 500 total benchmark runs and the headline
15:00
finding was that 18 months of LLM capability gains produced
15:04
only modest improvements in reliability.
15:07
All three major providers custard quite closely together,
15:10
suggesting that this is an industry wide challenge, not a
15:12
company specific one. And for me, I don't think this
15:15
is solved at the model layer, it's sold at the application
15:18
layer. So for example, on consistency,
15:20
whether the model gave the same answer when asked the same thing
15:22
twice. Scores range from 30 to 75% on
15:26
robustness. So whether it can deal with
15:28
imperfect conditions. Models handled infrastructure
15:32
failures like big server crashes quite well, but apart from that
15:35
didn't really do that well on predictability, whether the
15:38
model knew it's wrong. Performance was also very mixed
15:41
and this speaks to a challenge within implementing AI right
15:44
now. I think that, you know, mindset
15:46
sees all the time it's in that we're actively working through.
15:48
And as I said, it doesn't get solved at the model layer.
15:51
It gets solved at the application layer.
15:53
And again, models are trying to bring in more application
15:56
technologies. So they're trying to turn their
15:57
model into an agent itself in the way it works.
16:00
And this also is why SAS companies have such a big
16:03
opportunity really to create perfect agents for their
16:05
specific niche because a general purpose, Claude, although is
16:10
absolutely incredible and can do many things, if you make a hyper
16:14
specific agent for a specific area that you're an expert in
16:18
can be really, really powerful and more consistent, more
16:21
robust. And the reason that we have to
16:22
deal with this is essentially what you're asking is do we want
16:24
an agent to have deterministic controls or have high inference?
16:29
So essentially, can an agent decide and reason what to do
16:31
high inference, or do we, as say the agent creator, give it
16:37
specific instructions that it's not really allowed to stray
16:41
from? So if you give that agent a step
16:43
by step plan to follow, that's all it can ever do.
16:45
So if a person comes in and wants to go in any other
16:47
direction, it's not going to be able to do it.
16:50
Or if you let it reason itself and decide, it might go off in
16:53
different direction to solve the exact same problem.
16:55
Because ultimately an agent, all it's doing is observing,
17:00
planning, acting, checking, adjusting.
17:03
So it makes a plan, it sees how that plan goes, sees what data
17:06
and information it gets from that plan, checks whether that's
17:08
the right thing, and if it isn't, adjust.
17:11
I'll check in with the humor so that that's what enables agents
17:13
to do really incredible things that blow your mind.
17:16
And it will deal with imperfect conditions really well.
17:19
It can tell you when it needs help.
17:22
So it can act more like a person, but it doesn't mean it's
17:24
reliable. There's loads of different
17:26
things that can go wrong when it's reasoning.
17:28
So this is where that trade off between consistency and
17:31
flexibility becomes really important to rigid a set of
17:34
instructions. It can't help people in any ways
17:37
other than that step by step plan.
17:39
Too much inference means that it's likely to be inconsistent.
17:42
And so there's lots of ways to try and solve this mindset,
17:44
solving this in really interesting ways.
17:46
We're we're creating something called blueprints, which is more
17:48
like giving an agent a set of DNA, which is a set of
17:51
instructions that can last over many months.
17:54
So rather than just, you know, a quick process, you can give it
17:57
the ability to solve, say 50 problems with what we call
17:59
tools. Now a tool might be, you know, a
18:02
integration to a separate system to help a user do something.
18:05
It could be a small set of instructions that the agent can
18:07
follow, but a, the blueprint is essentially its DNA over many,
18:11
many, many months, like a long term objective.
18:13
It might be the HR recruitment, an entire employee life cycle of
18:17
how it expects to help people, or a product discovery journey
18:22
that might take six months. So it describes a set of things
18:25
an agent must take people through over a long period of
18:28
time. So yeah, this is a really big
18:29
area of investment by all these big companies.
18:32
It's really, really important to solve and all of it does take
18:35
time and therefore it takes time for adoption.
18:37
And we've seen this before. In 1987, there was a writer that
18:42
said, see the computer age everywhere but in the
18:45
productivity statistics. And in February, Apollo's chief
18:48
economist said the exact same thing about AI.
18:50
So yeah, 15 years into the computer revolution,
18:54
productivity actually slowed, especially during the IT build
18:57
out. It dropped from 2.9% annual
18:59
growth in productivity to 1.1%. But then between 1995 and 2005,
19:05
it surged. The technology hadn't
19:07
necessarily improve that much. The change was the organizations
19:11
finally rebuilt how they worked with it.
19:13
The technology was super consistent, it was accessible,
19:17
and it took a decade and a half to do.
19:36
Let's conclude. The dominant narrative I think
19:38
says that there's a massive market disruption that's
19:41
imminent. And I think we're starting to
19:42
see that. You know, Dario Amadei, the CEO
19:45
Anthropic, has said this over and over again.
19:47
But clearly the story, I think this data support is a little
19:50
bit more complicated. AI's economic impact is
19:52
definitely real, but held back by reliability limits that the
19:56
industry hasn't yet fully solved and organizational human
20:00
barriers that are just not easy to solve because theoretical
20:03
baselines are always more aspirational than measurement.
20:07
So yeah, Will AI transform work? Yes, Computers did reshape the
20:11
economy, but on their own schedule, not on what the
20:14
industry wanted it to be. And because everything
20:17
surrounded, the technology had to mature with it.
20:19
And I think the same thing is happening right now.
20:22
And this type of reliability research is going to force us
20:26
all in the industry to measure what matters.
20:29
So anyway, I hope you enjoyed the episode.
20:31
I hope you learned something. Thank you for listening and I'll
20:34
see you next week.