Latest / Safe As Podcast / E85:When Success Leads to Disaster: Practical Drift & the 1994 Black Hawk Friendly Fire Shootdown

Transcript

E85:When Success Leads to Disaster: Practical Drift & the 1994 Black Hawk Friendly Fire Shootdown · Safe As Podcast

0:00
On 14th April 1994, two US Air Force F15 fighters shot down to
0:06
U.S. Army Black Hawk helicopters over
0:09
northern Iraq. 26 people died. They weren't enemy aircraft, but
0:13
friendlies operating inside the same broader task force carrying
0:17
coalition personnel on a humanitarian mission.
0:20
This seems to defy all possibilities.
0:22
It wasn't a confused night battle.
0:24
It was daytime, visibility was good, and the aircraft were
0:28
operating under the control of an AWACS, effectively air
0:32
traffic control in the sky. There were rules, flight
0:35
schedules, identification systems, radios, command
0:38
structures, and trained professionals.
0:40
The system didn't fail because there were no safeguards or
0:44
because of incompetent people. It failed with the safeguards in
0:47
place. Good day everyone.
0:48
Welcome to SAFE as I'm Doctor Ben Hutchinson.
0:51
Now the easy explanation is to call it pilot error, and in a
0:55
way that's accurate because the pilots did misidentify the
0:59
Blackhawks's enemy. Hein Aircraft's Pines is in the
1:02
Iraqi Heinz helicopter. But as James Reason warned,
1:06
human error is a consequence, not a cause.
1:09
The error is the start of the investigation, not the end of
1:12
it. Scott Snake, in his book
1:14
Friendly Fire, uses Reason's Point as a type of quarter arms.
1:19
He explains how the disaster occurred by normal people
1:22
working inside normal organisational arrangements and
1:25
how it produced an abnormal and catastrophic outcome.
1:28
He says that there weren't any bad guys, no single villain to
1:32
be found, no simple broken part and no one obvious point where
1:37
the entire explanation can stop. In Snook's words, this tragedy
1:41
happened because of normal people behaving in normal ways
1:44
in normal organisations. This applies not just to the
1:47
military but all high risk and complex organisations because
1:51
many organisations still implicitly work with a blame and
1:55
error mindset. So we inevitably find the person
1:57
who made the poor decisions and Snook shows how this finding is
2:01
far too shallow. Now this is a whole book,
2:04
absolutely essential reading for all practitioners, so I can only
2:07
cover a few points that I found interesting.
2:09
Part 1 sense making and the how and why.
2:12
Snook suggests instead of asking why did they decide to shoot, we
2:17
should focus on meaning rather than deciding how was this
2:20
reality constructed in their heads and the environment.
2:23
Before the pilots could decide what to do, they had to decide
2:26
what the situation meant. They had to answer what Carl
2:29
Wire called the sense making question, What's going on here?
2:32
This shifts the focus from the pilots sitting there calmly and
2:37
choosing between clear, unambiguous options.
2:40
They weren't choosing between neatly labelled realities, they
2:43
were interpreting partial, ambiguous and fast moving
2:47
information. They had radar pings and radio
2:49
info, but this didn't give them a clear friendly as in an ally
2:54
identification. They had radio calls from AWACS
2:56
and a mission brief beforehand. And importantly from all of this
3:00
info, they had expectations about who should and who
3:03
shouldn't be inside that area. They had a fleeting visual pass
3:07
of the helicopter at high speed and they had the pressure of
3:10
operating in what they understood to be a combat
3:13
environment. And noting these F fifteens were
3:15
primarily suited for air to air combat with other fast moving
3:18
planes not chasing down low altitude slow moving
3:22
helicopters. So instead of why did they
3:24
decide to shoot, sneak frames it as how did 2 friendly Blackhawks
3:29
become in the pilots minds, 2 enemy Hinds?
3:32
The answer isn't that the pilots randomly imagined something, but
3:36
rather than misidentification emerged at the intersection of
3:39
three things. An ambiguous stimulus, a
3:42
powerful set of expectations, as in they expected to see a Hind,
3:46
and a desire to see an enemy target.
3:48
On the ambiguous stimulus, the pilots had a bit of a challenge
3:51
using the comparison charts, as in the charts used to identify
3:55
enemies, which in this case are pictures and silhouettes, and
3:59
often from very particular angles.
4:01
Like the Army tends to look up so their silhouettes tend to
4:05
show the under body of an aircraft, whereas the F Fifteens
4:09
were looking down at the helicopters, so the silhouettes
4:12
that they looked at beforehand didn't even properly line up.
4:14
Snoot compares the pilots to hunters who want to see game so
4:18
badly that they accidentally shoot hikers or dogs.
4:22
This explains how personal motivation provides more freedom
4:26
to see what we want to see when a stimulus is ambiguous.
4:29
The F Fifteens were flying fast, looking down through a canopy
4:32
from above and behind at helicopters moving low against
4:36
the terrain. The Blackhawks were filled with
4:38
external fuel tanks, and from the angle of the past, some
4:42
features couldn't be matched against the mental image of an
4:45
enemy height. Snook notes that the lead
4:46
pilot's view was made up of fragments, pieces of shape, tail
4:51
movement, and the context of the helicopter.
4:53
It's noted that when data are incomplete, people fill in the
4:56
gaps. They don't usually experience
4:59
this as guessing, but they experience it as perceiving.
5:02
Snook quotes Bruna, who observes that people take whatever scraps
5:06
they can extract from the stimulus input, and if those
5:09
scraps fit their expectations, they read the rest from the
5:13
model in their head. In other words, the pilots
5:15
weren't simply seeing what was there, they were seeing what the
5:18
situation had primed them to see.
5:20
Snoop draws on a great quote from Charles Perrault.
5:23
Seeing isn't necessarily believing.
5:25
Sometimes we must believe before we can see, and this is probably
5:28
the central point of the sense making argument.
5:31
By the time the lead part looked down and saw the helicopters,
5:35
the situation had already been primed in their minds eye.
5:38
The pilot didn't expect to see friendly helicopters.
5:41
In the pilots words, the Black Hawk did not even cross my mind.
5:45
It's not simply that expectation shapes perception, it's that
5:48
acting on expectation generates apparent evidence that confirms
5:52
it as why puts it. When perceivers act on their
5:55
expectations, they may enact what they predict will be there,
5:58
and when they see what they have enacted using their predictions
6:02
as a lens, they often confirmed their prediction.
6:05
The pilot looked found fragments consistent with the Hind, and
6:09
that apparent confirmation further locked in the category.
6:12
So this is more than a pilot being careless, it means that
6:16
the entire category of a friendly Black Hawk wasn't even
6:20
available as an option to him at that moment.
6:22
It wasn't part of the working reality and schemas he'd
6:25
constructed. Next, Snook explores where this
6:27
expectation came from, and in part, expectation was built by
6:32
the surrounding technical system.
6:33
The pilots were briefed about potential threats.
6:36
Snook captures this imbalance of information by saying plenty of
6:39
information was briefed about potential threats.
6:42
Nothing at all was briefed about friendly helicopters.
6:45
The pilots also believed they were the first aircraft entering
6:48
the tactical area of responsibility for the day's
6:51
fighter sweep. Their job was to sanitise the
6:53
area before other coalition aircraft entered and
6:57
importantly, their flow sheet didn't show the Blackhawks and
7:00
when they asked for updates they received no new information.
7:04
Then as they entered the area, AOX didn't give them a call
7:07
alerting them to friendly helicopters operating ahead of
7:10
them as they were unaware. Also, when the lead pilot
7:13
reported a contact, Haywax initially responded clean there,
7:18
then later hits there. In this context, hits indicated
7:22
radar returns, not a clear friendly identification.
7:26
Together these factors became interrelated and shaped one
7:29
interpretation. If something is out there, it's
7:32
probably not friendly. Snook again works in wack about
7:36
sense making. Believing is seen.
7:38
Expectations don't just sit in the background, they actively
7:41
shape what people notice, how they interpret it and how
7:45
confident they become. Like also warned that
7:48
expectations philtre inputs and this is likely what happened in
7:51
this circumstance. The pilots did receive
7:54
information, but that info doesn't speak for itself.
7:57
It has to pass through a frame and be interpreted, and this
8:00
frame was built socially as a joint process.
8:03
He draws on Wik's image of people interacting to flesh out
8:07
hunches, people making do with what they have, comparing notes
8:10
and then acting as if they have a workable sense of what is
8:14
going on. Hence, the pilot's
8:16
interpretation was reinforced through interaction, tactical
8:20
brief or missing helicopter information, the negative words
8:24
from command channels that hits their call, it's altitude,
8:27
confirmation of helicopters, and more.
8:29
These cues help construct the reality that the pilots acted on
8:33
in their mind. There was little doubt Snoop
8:35
calls the helicopter that the lead pilot saw a social
8:38
construction, a joint effort. Now of course the helicopters
8:44
and the victims were real, not social constructions.
8:46
But what he means is that the operational meaning of those
8:50
helicopters, friendly or enemy, safe or threat, was socially
8:54
constructed in real time. An important moment was the
8:57
wingman's call Telly 2. The lead pilot interpreted Telly
9:01
2 as confirmation that the wingman also saw two Heinz.
9:06
But the wingman later indicated that he didn't possibly identify
9:09
them as Heinz. He saw or two helicopters and
9:12
didn't see anything that contradict the leads call about
9:14
Heinz specifically. So one person hears a
9:17
confirmation of Heinz while the other thinks he's only reported
9:21
a contact of helicopters and unknown if enemy or friendly.
9:25
Problematically, the words they use on the radio are extremely
9:29
short, clipped and efficient, but they leave room for meaning
9:33
to be filled in by expectation. Why could warned about this in
9:36
aviation style communication? Short radio calls often work
9:40
because pilots and controllers already anticipate what the
9:44
other person is going to say. Their messages can, quoting the
9:47
book, ratify expectations rather than actually inform.
9:50
So communication doesn't automatically create shared
9:54
understanding. Sometimes it only confirms the
9:56
misunderstanding people already have.
9:58
Once the pilots have made sense of the helicopters as enemy
10:01
heights, the next question is why they acted so quickly.
10:05
This is where arousal training and miscoding come into their
10:08
construction. Demeaning.
10:09
The pilots were operating in a high pressure situation, closer
10:12
terrain at speed, in a combat coded environment.
10:15
Under arousal, people focus on fewer cues, as in they narrow
10:19
their focus onto fewer pieces of information.
10:22
They rely more heavily on dominant pre trained responses.
10:25
And dominant responses are shaped by training, and training
10:28
is essential. But it has a trap.
10:30
Mike warns that over learning is purchased at the expense of
10:34
flexibility. In high arousal sense making,
10:37
the costs have been indecisive frequently outweigh the costs
10:41
have been wrong. This pressure forces people to
10:43
cling to their initial hypotheses rather than seek new
10:47
evidence. So inexperience wasn't the
10:49
issue, but rather the opposite issue.
10:51
Once they coded the situation as enemy air to air combat, their
10:55
training worked. It produced fast decisive
10:58
action, but it was created within the wrong mental frame as
11:01
why connotes. During these sorts of pressures,
11:04
sense making becomes schema driven rather than evidence
11:08
driven. People seek confirmation of
11:09
their expectations. Indeed, as White has argued
11:12
elsewhere, people are driven by plausibility over accuracy.
11:16
So the final action looked like a decision to us in our
11:19
armchairs. But by the time the decision
11:21
arrived, the meaning had already been made.
11:24
The police didn't shoot at Blackhawks, they shot at what
11:27
they understood to be enemy high.
11:29
So a Snook is careful to articulate This doesn't make the
11:32
outcome acceptable, it just makes the pathway
11:34
understandable. Now, behind the pilots was an
11:36
AWACS crew that didn't intervene to stop the friendly fire.
11:40
Snook argues that this also wasn't simply due to laziness or
11:44
indifference. He argues that the crew was
11:47
technically qualified as individuals but collectively
11:50
weak as a team. They hadn't developed the shared
11:52
responsibility, accountability and mutual monitoring required
11:56
for the situation they're in. This is where I think Amy
11:59
Edmondson's distinction between teams and teaming is useful.
12:02
In her article Teaming is a Verb, she argues that
12:05
traditional teams rely on stable membership, clear roles,
12:09
repeated practise, trust and familiarity.
12:12
But many high risk operations don't have that luxury.
12:16
In those settings, it's rather teamwork on the fly, where
12:19
people have to coordinate without the benefit of stable
12:22
team structures, long shared history or rehearse routines.
12:26
That maps closely onto the AOX crew problem.
12:28
Snick describes a crew that may have been individually qualified
12:32
but hadn't yet developed into a real team that were thrown
12:35
together for this rotation, operating in a dynamic situation
12:39
but without the deeper shared responsibility, trust, and
12:42
mutual adjustment that effective teaming requires.
12:45
Further on Edmondson's work in complex, uncertain work, it's
12:49
not enough to assemble competent individuals and assume
12:52
coordination will follow. People need the capability to
12:55
share crucial knowledge quickly, ask clear and frequent
12:58
questions, clarify interdependence, and make small
13:02
adjustments. As a situation of false.
13:04
The AOX cruise weakness was therefore not just the cohesion
13:08
in a generic sense, it seems to also have been a failure of
13:12
teeming under pressure. Now, importantly, responsibility
13:15
was also diffuse, meaning multiple people had some
13:20
connection to tracking, monitoring or intervening, but
13:23
no one person felt the obligation strongly enough to
13:26
act. Snoop calls this the fallacy of
13:28
social redundancy. In technical systems, redundancy
13:32
can improve reliability. But in social systems, adding
13:36
more people doesn't automatically mean someone will
13:38
act. It can actually be the opposite
13:40
sometimes, when everyone is responsible, no one is.
13:44
So in safety critical systems, if a control depends on someone
13:47
noticing, acting, intervening, doing something, and especially
13:52
if more than one person is expected to do any of these,
13:55
then the control may be weaker than it looks.
13:57
Now further on the AWACS, the crew was operating inside a
14:00
larger task force and Snake's organisational level question
14:04
here is why were the Army helicopters not fully integrated
14:07
into task force air operations? He explores this principle that
14:11
whatever you divide, you have to put back together again.
14:15
As in complex, organisations divide work because they have
14:18
to. They create specialised teams,
14:20
functions, contractors, disciplines and systems.
14:23
But every active division creates an integration problem.
14:27
Eventually, the more specialised the parts become, the more
14:30
deliberate work is needed to reconnect them.
14:33
Diane Borne identified a similar trap prior to the challenge of
14:36
disaster. As specialists, deep in their
14:39
expertise and knowledge, they develop esoteric language and
14:42
norms that insulate them from the rest of the organisation,
14:46
from the nun experts. While this specialisation
14:48
sharpens their own craft, it obstructs a shared perception of
14:52
risk. When technical experts and the
14:54
managers holding the purse strings no longer speak the same
14:57
language around maybe critical risks, catastrophic warnings are
15:01
effectively lost in translation. Now in this overarching air
15:04
operation, the helicopters, fighters, AWACS, command
15:08
elements, planning documents, radio systems and operating
15:12
assumptions weren't integrated as well as the formal system
15:15
applied. The work is imagined, a type of
15:18
false assurance. There were key differences in
15:20
procedures, equipment, practises.
15:22
Some of these gaps have become normalised.
15:25
What Snook defined as practical drift Now Part 2 is practical
15:29
drift. This is defined as the slow
15:32
steady uncoupling of local practise from written procedure.
15:36
Practical drift isn't really about people breaking rules, nor
15:39
simple non compliance. It's what happens when the
15:41
written system and the working system slowly move apart and the
15:45
gap becomes more or less invisible.
15:48
This is said to usually happen for practical reasons.
15:51
Rules and procedures are often written globally.
15:54
They're designed to coordinate a whole system, often under worst
15:57
case assumptions. But workers experience the
15:59
system locally. They face the task in front of
16:02
them, with the time, tools, constraints, and habits and
16:07
pressures of the real work as it happens each day.
16:09
When the written rule doesn't fit the local task, people
16:12
adapt. They adjust.
16:15
They make the work possible. And if the adaptation works
16:17
today and tomorrow and the day after that, it starts to become
16:21
normal or normalised. Nuke says that globally designed
16:25
but locally impractical procedures lose out to practical
16:28
action when no one complaints. Indeed, people rarely have
16:32
visibility over the whole system and enterprise risk.
16:34
They see their own little relevant patch.
16:37
So a little drift from standards here is locally efficient but
16:41
potentially catastrophic at the organisational level.
16:43
Locally efficient behaviours become accepted practise through
16:46
repetition, not because someone formally approves them or
16:49
because someone writes them into the procedure.
16:52
They're normalised because they work and save time and nothing
16:55
bad usually happens. And This is why drift is so hard
16:58
to see, because it's usually not dramatic.
17:00
It creeps in over time in mundane ways, as in people find
17:05
a better way to get the job done.
17:06
A supervisor sees it and says nothing.
17:09
Or as Snook nicely points out, a well intentioned senior leader
17:12
visits and either doesn't observe or recognise a non
17:16
compliance, or doesn't say anything if they do.
17:18
And this silence is then perceived as validation that the
17:21
non compliance is acceptable. It's effectively and indirectly
17:25
legitimated. This is where Snook starts to
17:27
drill into the idea of loose coupling, drawing on Wikes ideas
17:31
of loosely coupled systems. Simply put, loose coupling means
17:35
parts of the system are connected, but not tightly or
17:38
continuously. They have some independence and
17:40
slack now that can be useful. It allows local adaptation and
17:44
it gives people room to respond to local conditions.
17:47
Wyke notes that loosely coupled systems can support localised
17:51
adaptation, which benefits local personnel.
17:53
But this flexibility can also create global danger.
17:56
If one group adapts locally, another group may not know.
18:00
If several groups adapt in different directions, the whole
18:02
system can become less integrated without anyone seeing
18:06
the full pattern. This is one of the key traps of
18:08
practical drift because each group assumes the other groups
18:12
are still following the original rules.
18:14
The F15 pilots assume the helicopters facts and task force
18:17
are operating according to the formal system.
18:19
The helicopter crews operate according to their local
18:22
realities, operates according to its crew roles and assumptions
18:25
and routines. The task force and leadership
18:27
believes the operation is well integrated because the system
18:30
has been working for a long time.
18:33
And even when these assumptions aren't aligned to reality, this
18:36
misalignment remains invisible because loosely coupled systems
18:40
the couple cause from effect, or at least a speedy visible
18:44
connection. The resulting feedback lag
18:46
ensures that local adaptations rarely trigger clear signals of
18:50
risk. Consequently, the organisation
18:52
continues to mistake the absence of failure for the presence of
18:55
safety. So success breeds danger.
18:58
Nuke users wikes warning about reliability.
19:01
Reliable outcomes are often invisible because they're
19:04
constant. Operators see nothing, and
19:07
seeing nothing presume that nothing is happening.
19:09
But as James Reason argued, if you see nothing is happening and
19:13
assume that if you keep doing nothing, then stable outcomes
19:16
will result. But stable outcomes require
19:19
dynamic inputs to have nothing happening.
19:22
You need to invest a lot of continual energy and efforts so
19:25
a system can keep succeeding while the conditions underneath
19:29
that are changing. Inputs, practises, and
19:31
assumptions all change. Local and efficient workarounds
19:35
spread, but because the outcome remains stable, no really bad
19:39
things apparently are happening, so no one pays attention, nor do
19:42
they feel like they have reason to pay attention.
19:44
For Snake, this reflects the idea that just because it isn't
19:48
broken doesn't necessarily mean that it isn't breaking.
19:51
And maybe this principle should be written into every safety
19:54
dashboard that relies too heavily on incident measures and
19:57
traffic lights. Of course, as is probably
19:59
broadly accepted now, no incidents doesn't mean the
20:02
system is safe or effectively addressing major risks.
20:06
It might mean the system hasn't yet been tested under the
20:08
conditions that expose a drift. Or it could be a mere case of
20:12
probability. You know, if you roll the dice
20:14
enough Times Now, near misses should help.
20:17
But might not Snoop describes a previous near miss involving an
20:20
F111 and notes the emphasis seemed to be on the miss and not
20:24
the near part. Nothing serious was learnt.
20:26
Snoop draws and James Reason here.
20:28
The large random component in accident causation means that
20:32
safe organisations can still have bad accidents, and unsafe
20:36
organisations can escape them for long periods.
20:39
Bad luck can bring down the deserving, while good luck can
20:42
protect the unworthy. The absence of a serious outcome
20:45
from a near miss is therefore not evidence that the system is
20:47
safe, it may simply be evidence of luck.
20:50
Research from Dylan and Tinsley support this across a range of
20:54
settings and occupations. People may interpret a near miss
20:57
as two types, a resilient near miss, as in we just barely
21:02
succeeded but still succeeded, or a vulnerable near miss holy
21:05
crap we nearly died. 1 is seen as a success, as in a near miss
21:10
we just barely succeeded. The other, a near hit we mostly
21:13
failed. And in their work, learning
21:16
mostly seems to result from the near hit, the vulnerable
21:18
instance where it's recognised how close to valley we were.
21:22
And this is largely influenced by coupling the near hit to
21:25
queues of immediate imminent danger.
21:28
Practical drift also challenges how leaders understand their own
21:31
visibility. Snoop discusses Westrom's
21:33
fallacy of centrality. Because the leader don't know
21:37
about it. It must not be happening or it
21:39
must be happening. The way I understand it.
21:41
You might recognise this as some form of workers imagine versus
21:44
work is done. Like makes the point about
21:46
experts. Experts can overestimate the
21:49
likelihood that they would know about a problem if it was really
21:52
occurring. So for leadership, being active,
21:55
experienced and close to the work doesn't guarantee that you
21:58
can see the drift. Sometimes expertise creates
22:01
confidence that itself becomes a blind spot.
22:03
Whenever a high level leader visits a unit and makes no
22:06
correction, the subordinates receive a powerful message.
22:10
Everything is OK, carry on. This reinforces the drift as a
22:13
culture of production. And similarly, per the recent
22:16
Port of Auckland CEO prosecution highlighted in their words, a
22:20
good leader and a conscientious officer may have the best
22:24
intentions in the world, but may still breach that due diligence
22:28
duty. So best intentions may not be
22:30
enough. In this view, drift is
22:31
especially difficult to say because it's a process, not a
22:34
snapshot. At one time, Snook says that
22:37
drift can't be seen in a single glance.
22:39
To see it, you need movement in time.
22:41
You need to compare the workers design with the workers actually
22:44
done, not once, but repeatedly. These findings directly inform
22:48
safety assurance. If assurance only cheques
22:50
whether the procedure exists once, it'll miss drift.
22:53
If audits only inspect documents, they'll miss drift.
22:56
We'll also miss drift if leaders only ask whether there have been
22:59
any incidents, or if investigations stop at the last
23:02
human action. To see drift, you really need to
23:05
ask how has practise changed over time?
23:07
Where have local adaptations become normalised?
23:10
Which rules no longer fit the work?
23:11
Which groups are assuming that other groups are still following
23:14
procedures that they themselves have quietly moved away from?
23:18
Now, Snook doesn't condemn local adaptation.
23:20
Actually, he later reframes the idea as a form of practical
23:23
sailing. He writes that practical drift
23:25
and practical sailing are two sides of the same coin.
23:28
From the local perspective, adaptation may be sensible,
23:32
skilled and necessary. Again, from the perspective of
23:35
those local operators, workers probably aren't drifting
23:38
aimlessly. They're sailing, adjusting to
23:40
the conditions, making trade offs, managing constraints and
23:43
getting the job done in a way that seems locally sensible.
23:46
But from the system perspective, many sensible local adaptations
23:51
can create dangerous global misalignment.
23:54
Snook draws on Vaughn's normalisation of deviance, which
23:57
is a process where, in the NASA Challenger example, engineers
24:01
and managers together develop a definition of the situation that
24:05
allowed them to carry on as if nothing were wrong when they
24:08
continually faced evidence that something was wrong.
24:10
In NASA, deviance was normalised centrally, collectively
24:15
redefined as acceptable risk through structured
24:17
organisational processes, with engineers and managers drawing
24:21
together the new line. In this Black Hawk example, the
24:24
process was local and dispersed. Each subunit developed its own
24:27
definitions of acceptable practise, largely invisible to
24:31
the others. There was no shared
24:33
redefinition, only parallel and silent, divergent in different
24:36
directions. So heavy handed attempts to
24:39
simply ban or workarounds is likely to fail and to drive
24:42
workarounds underground. Some research from Tucker 2014
24:46
suggests that nurses, for instance, experience an
24:49
operational failure in their work that could prompt a
24:52
workaround every 37 minutes or 14% of their daily work time.
24:57
So simply banning workarounds and adaptations outright is
25:01
unrealistic and unhelpful. As for one, it doesn't address
25:05
why the workaround exists. Workarounds are a source of
25:07
Intel about your system. They tell you where the
25:09
procedure doesn't seem to fit the work.
25:11
They show where people are having to adapt to make the
25:13
system function. Well meaning workers will always
25:16
subvert globally established rules that don't make sense
25:20
locally, but conversely these are often the same make it
25:23
happen people organisations rely on, especially when things
25:27
aren't going wrong. We praise them for their
25:28
adaptability. The practical challenge isn't
25:30
the right endless procedures, of course, but to identify the few
25:33
rules that really must be performed as expected, weren't
25:37
expected, and keep rules to the minimum necessary and enforce
25:41
them consistently. Importantly, if everything is
25:44
critical, nothing is critical. Snook also reinforces the
25:47
importance of integration. So remember his earlier remark
25:50
that whatever you divide, you have to put back together again.
25:53
Every organisation divides work. Construction projects divide
25:57
work between, you know, principal contractors, the
25:59
subbies, designers, engineers, supervisors, etcetera.
26:03
That division is necessary, but reintegration isn't automatic.
26:07
So when we have weak integration controls, people can still be
26:11
doing their own jobs competently while the system fails to
26:14
perform jointly. So the pilots made sense of
26:16
ambiguous information through expectations.
26:19
The AOX crew didn't intervene, in part because responsibility
26:23
was diffuse. The diffusion of responsibility
26:25
and the team configuration was weak.
26:27
The task force didn't fully integrate helicopter operations
26:29
into the wider air operation. Local practises drifted away
26:33
from written procedure. Each layer made sense in its own
26:35
context, but together they created the conditions for
26:38
catastrophe. If we only ask who made the
26:41
final error, we missed the system that made the error
26:43
likely. If we only checked where the
26:45
procedures exist, we miss where the practise is drifted.
26:48
If we only look at outcomes, we may miss the system while it's
26:52
breaking. And if we assume that more
26:54
people, more rules and more safeguards automatically create
26:57
safety, we can create the illusion of safety without the
27:00
reality of integration. Reason put this tension most
27:03
happily. You cannot change the human
27:05
condition, but you can change the conditions under which
27:08
people work. That's where this tragedy should
27:09
leave us. Not with a search for non erring
27:12
perfect people, but with a search for better conditions.
27:16
Conditions that make weak signals easier to notice and
27:19
certainly easier to voice responsibility clearer critical
27:23
rules fewer and more practicable, drift more visible,
27:27
and the divided parts of the system better integrated.
27:30
Snook tells the story that the critical safety gap isn't
27:33
between good or bad people, competent versus incompetent,
27:37
the bad apples. Rather, it's the gap between the
27:39
system as imagined and the system as it's actually
27:42
operating. If we're not actively looking
27:44
for that gap to understand it, to learn and improve, it might
27:47
make itself known in the most catastrophic way possible.
27:51
That's it on Sofa. Please like, comment, subscribe
27:54
and link in the show notes to support the channel.