Latest / Safe As Podcast / E84: The pitfalls of safety reporting metrics and 'safewashing' our way to disaster

Transcript

E84: The pitfalls of safety reporting metrics and 'safewashing' our way to disaster · Safe As Podcast

• 0:00
  What if your best looking safety number is the one hiding your
• 0:03
  biggest risks? For years, companies have leaned
• 0:06
  on measures like lost time, injury frequencies and total
• 0:09
  recordable injury frequency rates to show whether safety is
• 0:13
  improving. Lower rates equals good news and
• 0:15
  higher rates equals bad news, apparently.
• 0:17
  But 2 reports from Sharon O'Neill and colleagues argue
• 0:21
  that this logic can be badly misleading because injury rates
• 0:25
  are often treated as if they measure safety when they don't.
• 0:28
  As one of the reports puts it, measures of injury and illnesses
• 0:32
  don't measure the health and safety of the work environment,
• 0:35
  nor are they a reliable indicator of work health and
• 0:39
  safety performance. OK everyone, welcome to Safe as
• 0:42
  I'm Doctor Ben Hutchinson. Today we're exploring the limits
• 0:45
  and traps of safety performance reporting systems, incentives
• 0:49
  and also annual reporting based primarily on 2 reports.
• 0:53
  Number one is from O'Neill, Wolf and Holly 2015 titled
• 0:59
  Performance measurement, incentives and Organisational
• 1:02
  culture Implications for leading Safe and healthy work from Safe
• 1:05
  Work Australia and 2nd from O'Neill and Wolf 2017 titled
• 1:11
  Measuring and reporting on work Health and Safety report from
• 1:15
  safe work in Australia. Also, the first problem is
• 1:17
  aggregation. The 2017 report states that a
• 1:21
  single injury number or frequency rate is too aggregated
• 1:25
  to provide meaningful information to guide business
• 1:28
  decisions. Now, not all injuries are, of
• 1:31
  course, a lot. Some are short term and
• 1:33
  relatively minor. Others permanently alter a
• 1:36
  person's life. But once they're collapsed into
• 1:38
  a single rate, you lose the distinctions that matter most.
• 1:41
  The report says that the majority of lost time injuries
• 1:44
  are Ltis, have relatively short term consequences, and because
• 1:49
  low consequence injuries occur more frequently, LTI data
• 1:53
  provides a general indicator of relatively minor events but no
• 1:57
  useful insight into the occurrence of more damaging
• 2:00
  injury and illnesses. Using a data set of more than
• 2:02
  400,000 work related injuries, the authors show that the same
• 2:07
  underlying cases can tell two different stories depending on
• 2:11
  how they're classified. They argue that the results
• 2:14
  illustrate that a steadily decreasing trend in Ltis
• 2:17
  overtime can hide A significant increase in the most damaging
• 2:21
  and costly group of non fatal injury over the same period.
• 2:26
  And other research from Biona ET El 2024 shows that in their data
• 2:31
  at least, the mechanisms and proximal causes that lead to
• 2:33
  fatal or severe injuries may be distinct from those that lead to
• 2:37
  relatively minor injuries, including the types and
• 2:39
  magnitudes of the energy present, types of controls that
• 2:42
  are used, and a number of other factors.
• 2:44
  So minor and major incident potentials might be quite
• 2:47
  different. The report indicate a key issue
• 2:49
  being risk blindness. A board looking only at a
• 2:52
  falling LTFR can easily conclude that safety is improving even
• 2:57
  while the organisation remains exposed to serious or
• 3:00
  permanently disabling harm. The risk exposures.
• 3:04
  Hence, paraphrasing the report's findings, a board receiving only
• 3:08
  injury and incident measures would be likely to draw very
• 3:11
  different conclusions as to the success of past safety efforts
• 3:15
  and the appropriate focus on future safety attention then
• 3:18
  those receiving data on, let's say, control or system
• 3:21
  effectiveness and the like. Now, the reports don't say that
• 3:25
  injury data is completely useless.
• 3:27
  In fact, they're more careful than that.
• 3:29
  They can be useful, but in their words, not for making decisions
• 3:33
  about health and safety. The 2017 report says that LTI
• 3:38
  key Performance indicators are useful for understanding the
• 3:41
  impact of poor safety decisions on business productivity, but
• 3:45
  not for judging whether hazards are controlled or whether safety
• 3:49
  is improving. So if you want to understand
• 3:51
  disruption, time away from work loss, productivity or reporting
• 3:55
  practises, entry data can help. If you want to understand where
• 3:58
  the critical risks are being controlled or understanding the
• 4:01
  severity of events, then Lt, IS and trippers and all that sort
• 4:05
  of stuff doesn't really assess these facets nor the
• 4:09
  effectiveness of controls. Therefore, in their words,
• 4:12
  injury measures have been incorrectly presented as
• 4:15
  evidence of the success of safety efforts, even though it
• 4:19
  fails to measure safety performance at all.
• 4:21
  The second problem is what happens when these numbers
• 4:23
  become targets. The 2015 report on performance
• 4:26
  Measurement Incentives warns that, in their words, in the
• 4:29
  absence of a reporting culture, focusing on injury results is
• 4:33
  likely to drive people to manage the measure.
• 4:36
  They provide examples like concealing injuries,
• 4:39
  reclassifying them to seem less concerning, and improving
• 4:42
  results on paper while actual performance remains unchanged.
• 4:46
  The classic paper safety system Now this is where weak metrics
• 4:50
  become a form of symbolic management.
• 4:52
  The report says injury frequency rates may motivate
• 4:55
  underreporting or misreporting, may lead to bullying of injured
• 4:59
  workers or pressure for a premature return to work.
• 5:03
  And that a low LTFR or TRIFA may be falsely interpreted as
• 5:08
  evidence that health and safety risks are well controlled.
• 5:11
  So the use of measures in these ways may actually change the
• 5:15
  type of work organisations do. If workers, supervisors or
• 5:19
  managers know they're being judged on the number, the
• 5:22
  numbers start shaping behaviour. And if the number is poorly
• 5:25
  chosen, the behaviours it drives can be exactly the opposite of
• 5:29
  what you want. Research has also found similar
• 5:32
  effects of measures and targets being managed over the core
• 5:35
  issues. They've called these effects
• 5:37
  surrogation and quantification fixation, among other terms.
• 5:41
  Now the report is particularly blunt on employee incentive
• 5:45
  schemes tied to injury reductions.
• 5:47
  It says these schemes are inherently flawed since many
• 5:50
  causal factors are not within the employees control.
• 5:53
  Interestingly, Deming was also critical of holding people
• 5:57
  accountable for performance targets that sit outside of
• 6:01
  their locus control and sit largely with the overarching
• 6:04
  system or organisation. The social consequences can get
• 6:07
  particularly bad when these schemes are group based, the
• 6:10
  report says. They can create peer pressure
• 6:12
  for underreporting and lead to injured workers being blamed or
• 6:16
  bullied for the team's loss, reward or bonus.
• 6:19
  Some research suggests that up to 80% of the incidents
• 6:23
  experienced by workers remain unreported.
• 6:25
  And really, who wants to jeopardise the pizza by breaking
• 6:29
  the five years LTI free record? Further, the report doesn't stop
• 6:34
  at frontline incentives, but it extends this argument onto the
• 6:37
  executive renumeration also. In other words, it's what
• 6:40
  leaders are paid to care about. It's covered in these articles.
• 6:43
  CEO bonuses often far exceed their fixed pay, and most annual
• 6:48
  bonus criteria only include a small percentage related to
• 6:51
  safety, usually tied to a LTI or triffa or something similar.
• 6:55
  Therefore, long term incentives were paid almost entirely on
• 6:59
  financial performance. In other words, safety can be
• 7:03
  publicly and socially framed as a priority while remaining
• 7:06
  structurally secondary in the reward system.
• 7:09
  The report says executive incentive structures may
• 7:11
  motivate a short term orientation to organisational
• 7:15
  performance, as in financial performance, and pose a strong
• 7:19
  disincentive to spending on health and safety.
• 7:22
  So you incentivize massaging the numbers rather than addressing
• 7:26
  the actual core issues. In research from Hopkins and
• 7:28
  Maslin, they noted a company assessed as having performed
• 7:31
  marginally above target despite 2 workplace fatalities and an
• 7:36
  injury rate well above target. This captures how cosmetic that
• 7:40
  safety bonuses can become when the broader enumeration
• 7:43
  architecture is dominated by financial goals and easily
• 7:47
  gained or massaged incident measures.
• 7:50
  So if these injury measures and they're supporting control
• 7:54
  structures are weak for these goals, and if tying them to
• 7:57
  rewards can distort behaviour, what do the reports suggest?
• 8:00
  Instead, the 2017 report argues for a multi dimensional view of
• 8:04
  health and safety reporting. Officers need information about
• 8:07
  their risk picture, their performance and assurance.
• 8:10
  Simply, that means understanding the hazards that expose people
• 8:13
  to harm, the controls put in place to manage those hazards,
• 8:16
  and whether those controls are implemented, whether they're
• 8:18
  effective, and what verification exists that the system is
• 8:21
  working. The reports also point towards
• 8:24
  more useful injury reporting. Instead of collapsing everything
• 8:27
  into a one headline rate, they recommend distinguishing injury
• 8:31
  outcomes by severity. One of the reports says that the
• 8:34
  board should receive the number of actual injury cases by
• 8:37
  severity and not an aggregated injury rate.
• 8:40
  It also recommends reporting on hazards linked to the most
• 8:43
  damaging injuries and the most frequent injuries and major
• 8:46
  hazard potentials, along with the processes used to manage
• 8:49
  those risks and evidence as to their effectiveness.
• 8:52
  Interestingly, research from Kevin Guetta shows that the
• 8:54
  original pioneers of injury measures from over 100 years ago
• 8:58
  recognised that injury frequency rates need to be coupled to
• 9:01
  severity measures. They knew this a long time ago.
• 9:04
  So in all, moving away from questions like did the number go
• 9:07
  down and towards questions like what are our critical risks?
• 9:11
  Are they identified? What controls are in place?
• 9:14
  Are those controls working? How do we know they're working?
• 9:17
  What serious harm has or could occur?
• 9:19
  And what assurance mechanisms do leaders have that the controls
• 9:22
  and systems are actually functioning as intended?
• 9:24
  The 2015 report adds another useful improvement suggestion.
• 9:28
  Don't just count activities outcome measures.
• 9:30
  Look at what those activities are producing.
• 9:32
  For example, it warns that measuring the number of incident
• 9:34
  investigations closed can drive WIC, superficial investigations
• 9:39
  and easily addressed low level risk controls.
• 9:41
  One suggested mitigation is to analyse investigation actions by
• 9:46
  the hierarchy of control level. For example, what proportion of
• 9:49
  actions actually eliminate risk, minimise risk at a higher level
• 9:53
  or rely only on administrative controls and PPE.
• 9:56
  This highlights whether the organisation is learning and
• 9:58
  improving and driving continual risk reduction.
• 10:02
  Further, these same limitations are reflected in annual
• 10:05
  reporting. What's been called safe washing?
• 10:08
  Adopting their green washing term, studies from O'Neill ET al
• 10:11
  2016 and Gardner 2022 highlight how only a small percentage of
• 10:16
  ASX listed companies reported safety metrics at all, but no
• 10:20
  surprises of those that did. Injury measures were among the
• 10:23
  most commonly disclosed measures, while near misses were
• 10:26
  rarely reported. Fatality disclosures were
• 10:29
  concerningly very limited, and investigation disclosures were
• 10:32
  often vague and incomplete of concern.
• 10:34
  In the IDEAL study, some companies reported no fatalities
• 10:38
  in their annual reports despite mentioning fatalities in other
• 10:41
  media over the same period, and few reports identified health
• 10:45
  and safety regulatory investigations, prosecutions,
• 10:47
  and fines. There are also inconsistencies
• 10:49
  in how metrics were calculated, including differences in the
• 10:52
  denominators, rolling average periods, contractor inclusion,
• 10:56
  and what counted even as an injury.
• 10:58
  Second, companies showed a clear preference for reporting
• 11:02
  response based rather than severity based metrics.
• 11:05
  And even though severity based metrics better reflect the
• 11:08
  consequences of injury and illness, at least of those
• 11:10
  things that are reported. So the external reporting
• 11:12
  picture often reproduces the same problem, saying internally
• 11:16
  that we covered before, selective use of lagging
• 11:19
  indicators, weak comparability, very little serious discussion
• 11:23
  of near misses or investigations.
• 11:24
  You know what was actually learned and improved, and a
• 11:26
  tendency to report what is the easiest, most familiar, or most
• 11:30
  flattering rather than what pest informs decision making.
• 11:33
  As Gardener states, the current work supports assertions by
• 11:38
  scholars and policy experts that voluntary reporting is
• 11:41
  ineffective. So based on all of this data, if
• 11:43
  you want to understand one element of productivity
• 11:46
  disruption, then use LTI, etcetera to achieve that
• 11:49
  purpose. If you want to understand the
• 11:51
  mechanisms that create risk and safety, then don't confuse
• 11:54
  injury rates with better control of risk.
• 11:56
  Look at severity, look at hazards and the effectiveness of
• 11:59
  the systems and controls the assurance piece.
• 12:02
  And be very cautious about attaching rewards to injury
• 12:05
  numbers unless you want people managing the metric instead of
• 12:09
  managing the underlying issues. Overall leaders, and probably
• 12:12
  most of us, can become blind to serious risk and organisations
• 12:17
  can become skilled at producing better looking numbers, as Deca
• 12:21
  calls it the Looking Good Index. And these aren't the same things
• 12:25
  as creating margins of safety nor assuring ourselves of it.
• 12:29
  That's it on Safe As. Please like, comment, share and
• 12:33
  subscribe and link in the show notes to support the channel.