Latest / In The Loop / AI Workslop Is A Human Problem

Transcript

AI Workslop Is A Human Problem · In The Loop

0:00
A new word just entered the workplace lexicon, Work slop.
0:04
If you haven't heard about it yet, you will do soon.
0:06
Stanford Research has just coined the term and it's
0:09
spreading like wildfire online. So this Stanford report that
0:12
came out this September spoke to one of the biggest complaints
0:15
when it comes to AI. The most of the work it produces
0:19
is a bit crap, with the study finding that 41% of workers who
0:24
were surveyed reported receiving work slop from their colleagues
0:28
in just the past month. In today's episode, we're
0:30
discussing the rise of work slot, the real financial cost of
0:34
it, whether this is all just noise, and how we can all fix it
0:38
going forward. This is in the loop with Jack
0:40
Horton. I hope you enjoy the show.
0:55
So I think let's start with what workstop actually is.
0:58
So workstop is the term that's used for basically AI generated
1:02
content and work content specifically that kind of looks
1:05
polished but is actually just a bit rubbish and crap.
1:09
And in practice, this looks like, you know, slide decks that
1:12
or reports maybe that look really good when you I guess
1:15
give it that 5 second glance or you skim through it really
1:18
quickly. But as soon as you really dig
1:20
into it or want to use it for something practical at work, it
1:24
has wrong numbers or repeating texts or every word of a
1:29
headline or a sentence is capitalized or the argument in
1:33
it is just again, a bit weak and, and repetitive and again,
1:38
crap. Now I have got a take on the
1:40
real cause of this, which we'll come on to.
1:42
But in a survey of over 1100 US workers, 40% of people said they
1:48
received AI Workslop in just the past month.
1:51
And according to this study, it takes about two hours of extra
1:54
work to fix all of this workslop.
1:58
And as a result, the researchers estimated that it costs about
2:01
$186 per employee per month, which equates to about $9
2:07
million a year for a 10,000 person company to fix of it.
2:10
Now, of course, there's an element of probable exaggeration
2:13
when they are reporting or self reporting the impact of
2:17
Workslop, but I think it does speak to a real thing at work
2:20
right now. And at this point, I think you
2:22
probably know the people at work who do this, you know the team
2:25
members who send sloppy AI work. And I think as AI becomes more
2:30
mature and widespread, those team members are going to pay a
2:33
price. Because, you know, about half of
2:35
the surveyed workers reported they regarded slop senders as
2:38
less creative, as less capable, or even reliable.
2:42
As one redditor said, everyone hands me that AI generated slop
2:46
to edit and I have to cut 75% of it.
2:48
Usually it's the same info repeated in slightly different
2:50
ways. Another quote the study
2:52
referenced makes this point even more clear.
2:55
If you use AI to pretend to do 20 tasks, you're a Trojan
2:58
horsing bunch of work to other people.
3:01
Basically by doing rubbish work you cause other team members to
3:05
have to do more work to fix it. Now workshop is nothing new,
3:09
it's just our new I guess blind spot enabled by technology.
3:13
In the past it would have seemed like a rush piece of work that's
3:17
just clearly copied and pasted from a Google article somewhere.
3:21
And in the past it was probably even harder to spot and even
3:24
harder to create as well. But now it's just so simple to
3:28
do on mass because AI is so good at generating words.
3:32
And this kind of mirrors a lot of the things that as a company
3:35
we've had to speak to each other about.
3:38
South Barry, who's the CEO and Co founder of Mindset, compares
3:42
it to say giving people a Formula One car.
3:46
And just because you've been given a Formula One car doesn't
3:48
mean you can drive a Formula One car.
3:50
Yes it can go fast, but you will most likely crash if you're not
3:54
really thinking through how to drive it.
3:56
Because data from very well regarded and big studies have
3:59
shown that AI can accelerate work when it's used right.
4:03
You know a recent MIT and Nielsen study found that ChatGPT
4:06
users writing a business document finished about 59%
4:09
faster and produce better outputs than those that didn't
4:13
have AI. So it's not that AI will
4:15
definitely cause slop, it just needs help so it doesn't produce
4:18
slop. And if it is slop, then I would
4:21
look at the person, not the tool.
4:24
For me, it's when people take their conversation, say when
4:27
writing or coding and deciding, oh, that's just good enough,
4:30
That'll do. You have to create tests, you
4:33
have to interrogate, you have to read and importantly understand
4:36
the topic yourself enough to call rubbish.
4:41
I say rubbish instead of swearing.
4:42
But yeah, you have to use initiative to get it from that
4:45
65% or 80% done, to bring it together to be something that's
4:50
perfect and ready to send. And every medium has its own
4:52
nuances here. I'm not saying that this is a
4:54
blanket rule. Whether you're writing, coding,
4:59
creating art, you can't really expect things to be done without
5:03
applying some level of brain power and analysis to the thing
5:09
that you're creating. Now, I'm not saying that people
5:11
don't care. I know many don't, but many also
5:14
do. And this is where I think
5:16
something called the efficiency trap starts to creep in with AI
5:19
work slop. Now I think this is a really
5:32
interesting take on how companies were very clever
5:37
people who probably have the best intentions start creating
5:40
rubbish. So someone called Doctor
5:43
Cornelia Walther at Wharton has documented what she calls the
5:48
efficiency trap, and it basically explains why work slot
5:52
can just proliferate in an organization despite everybody's
5:55
best intentions not to do so. It folds in four predictable
6:00
stages. So stage 1, according to her,
6:03
was that workers experiment with AI quite cautiously.
6:06
They use it selectively, They maintain lots of control.
6:09
Productivity genuinely improves. Tasks that might have taken days
6:13
might actually now take hours, and everybody gets quite
6:16
excited. Stage 2 is management really
6:19
noticing and getting involved, operating under resource
6:22
optimization objectives. They're going to increase their
6:25
expectations of a team's output. And the logic makes sense.
6:29
If technology drives us to deliver more and less time, why
6:33
not request more deliverables? And as this happens, AI becomes
6:36
very normalised in a person's work process and what was a tool
6:40
is now a habit. This is where you get to stage 3
6:44
and teams to meet these escalating demands delegate
6:47
increasingly complex tasks to AI and that I guess selective
6:53
assistance evolves into reliance task requiring independent
6:58
analysis and critical thinking become AI driven by default.
7:02
You know, dependency accelerates and a lack of processes around
7:05
all of this work then leads to more and more work slop
7:08
happening. And this is where she describes
7:10
stage 4. Each productivity improvement,
7:13
she says, becomes that new baseline deadlines reduce, so
7:17
you have less time to do more because the number of projects
7:20
that get started increases, the complexity of those projects
7:24
also increases. And efficiency gains are almost
7:27
permanently incorporated into performance standards.
7:31
And workers reach what Walther calls technological addiction,
7:36
which is feeling psychologically incapable of meeting demands
7:39
without using AI. And this workslop increases.
7:44
And by the way, this is one of the reasons that I don't think
7:47
AI is a bubble, because I have witnessed this.
7:50
I feel this. I've been a part of enabling
7:53
this. Now, your first instinct might
7:55
be, well, people just need better training because that's
7:58
what a lot of the time, I guess people default to when talking
8:01
about AI work now. Now I think there's also a human
8:04
element to this, which I want to explain in a little bit more
8:06
detail. Your first instinct might easily
8:09
become, you know, just train people and give them more
8:11
information and training. And I think that's the easy one
8:15
to default to. And you'd definitely be half
8:16
right. But here's where there is
8:19
probably a little bit of nuance. A big study from BCG, Harvard,
8:22
MIT and Wharton examined 758 consultants, roughly 7% of BCG's
8:28
individual contributors that were working with CHAT 2BT4.
8:33
Now for tasks that were within AI's capabilities, training
8:36
really helped. Below average performers
8:38
improved by 43% and above average performers gained a 70%
8:42
improvement too. So the quality of their outputs
8:45
jumped, you know, over 40% compared to those control
8:47
groups. That's a big jump.
8:50
But where it gets quite interesting, I think, is for
8:53
those tasks that are a little bit outside the A IS capability
8:57
set, or where that capability is stretched.
9:00
Training didn't help. In fact, training actually made
9:03
those people who had received training worse.
9:07
The people who had received that training on how to create the
9:09
prompts were more likely to accept A IS outputs, thinking
9:13
that if they prompted it correctly, then the response
9:16
must also be correct. So training created
9:18
overconfidence without any critical evaluation skills,
9:22
domain expertise or good processes to prevent this
9:25
happening. Now Professor Ethan Mollick,
9:27
who's a fantastic thinker in this space, actually called this
9:31
the jagged technological frontier, which is where a IS
9:34
capabilities aren't uniform. This means that some tasks are
9:38
just incredibly easy for AI, whilst tasks that seem almost
9:42
identical become really difficult for AI to achieve.
9:46
Now that boundary can be super unpredictable and training alone
9:49
doesn't help people navigate it. On Reddit's R slash GPTA,
9:52
computer science professor captured this perfectly.
9:55
Only 20% of students could effectively use ChatGPT, but for
9:59
that 20% it was incredibly impressive.
10:02
The differentiator wasn't prompt engineering expertise, it was
10:05
just a bit of critical thinking, domain expertise, and knowing
10:09
when to ignore the AI. And that's why I think the
10:12
reality truly is that this isn't a technology problem.
10:15
It's a human systems problem. So how do we fix this mess?
10:30
Well, I think, to put it plainly, domain expertise
10:33
matters more than ever. If you're not sure what good
10:35
looks like, AI can generate plausible first drafts that may
10:40
go way over your head. But if you do know your field,
10:43
you're the one that's going to be able to spot that mess.
10:46
The hallucination, the nonsense that comes out of it.
10:48
You see a lot online writers or coders complaining that AI often
10:53
makes very subtle errors that for a novice that really doesn't
10:56
understand the craft would struggle to notice.
10:59
This also ties to a really interesting analogy made by MIT
11:04
and Harvard. In their study, they referred to
11:05
a Centaur versus a Cyborg. So they said that some workers
11:09
act like Centaur. So dividing tasks between AI and
11:12
themselves. You know, ChatGPT, give me an
11:14
outline. I'll write the content, you
11:17
create the review, and others try to become true cyborgs,
11:22
fully blending AI into their workflow, sometimes over relying
11:26
on that AI system to produce the work.
11:29
The most effective teams tend to use a hybrid centauric approach.
11:34
Leverage AI for speed, whether it's creating certain templates
11:37
and processes or drafting certain text, but with an expert
11:41
human steering it to make sure the quality is still really
11:45
high. I think of course, another
11:46
really important one is training.
11:48
Many workers obviously dive into lots of new tools with very
11:50
little training or coaching. And if you don't understand the
11:53
underlying ways of or underlying processes, should I say you're
11:57
going to really struggle to get good outputs.
11:59
You know, BCG recently found that only about 1/3 of employees
12:02
actually feel properly trained on AI.
12:04
Then #3 is what I think is arguably the most important
12:08
factor, which is process and quality control breakdowns.
12:12
Now, I think few workplaces will have a very good review cycle
12:15
around AI drafts. So in the past, say an e-mail or
12:19
a document or some slides would be created by a person and then
12:23
reviewed by a person. But now AI is able to create so
12:26
much of it that probably often skips a review cycle, especially
12:29
when people are operating under a deadline.
12:31
Again, thinking back to the efficiency trap, now there's
12:33
multiple drafts and multiple review cycles that can happen
12:37
incredibly quickly. So we have two drafts that are
12:40
built by product management. You create a document which is
12:43
about essentially the outline of the brief, the things that this
12:46
thing must do and why. We then have a specification
12:49
document created, which is essentially a planning document,
12:51
which takes the vision of the big feature raw product that
12:55
you're going to build into an actionable plan for developers.
12:59
At both of those stages, you have a review cycle.
13:01
Devs will then use clawed code to create an entire
13:04
implementation plan, which is then reviewed and then we go
13:08
ahead and do the work. Now that can happen 6070% faster
13:12
than normal processes because it doesn't require 8 meetings to
13:15
get to that end plan. It can happen in literally 30
13:19
minute segments and an individual ideating with AI to
13:22
get to the right plan. And the same goes for writing.
13:25
You know, Gen. AI can ideate and apply for the
13:27
writing and help with the research.
13:29
Then you can handpick the research and then you can draft
13:32
and bring it together yourself. Each should have its own review
13:34
cycle. And again, all of these
13:36
processes should be baked in with guardrails, contexts on the
13:39
business, the process, the frameworks, the even template
13:43
outputs. Anyway, go down to the 4th and
13:46
final factor that would help fix this massively, and that's
13:49
expectations. I think many managers have
13:51
jumped on AI expecting kind of magic equals better and more
13:55
equals better. And I think it just creates a
13:57
trap where workers obviously feel pressure to just crank out
14:00
more content for the sake of it. Now we have to talk about levers
14:03
to pull in a project. You can pull the speed lever,
14:07
you can pull the scope lever, but if you're always trying to
14:10
pull that speed lever, then often the quality lever is also
14:14
being pulled. And I think solving this is just
14:17
as much about resetting expectations.
14:19
For example, training programs should focus on ideating with AI
14:23
to create your plan and then let humans do some heavy thinking
14:27
and actually produce good work. Anyway to to conclude the
14:41
episode that this story should be quite clear by now.
14:45
Worksop isn't an AI problem, it's a symptom of under prepared
14:49
people and processes. You know, when we rush it or
14:52
under train ourselves or hide its use or don't put process
14:56
around this Formula One car, we get sloppy results.
15:00
And conversely, when an organization invests in smart AI
15:04
habits, they're going to get real fantastic benefits.
15:07
And I think for listeners, the take away is quite actionable.
15:11
If you want good AI output at work or for personal use, treat
15:15
it like any tool. Learn to use it properly, double
15:17
check its work, and loop humans into the process and assume that
15:22
it's not going to get the right answer straight away.
15:24
Anyway, thanks for listening. That's it for this episode and
15:27
I'll see you next week.