Latest / Safe As Podcast / Ep 11: The fault in our stats (incident measures)

Transcript

Ep 11: The fault in our stats (incident measures) · Safe As Podcast

• 0:06
  The safety dashboard looks great, nearly everything is
• 0:09
  green or amber with barely a hint of red and then you kill
• 0:14
  somebody. Are our injury measures more a
• 0:17
  curse with a poor statistical basis or a decent enough
• 0:22
  predictor of harm? Good day everyone.
• 0:26
  I'm Ben Hutchinson and this is Safe as a podcast dedicated to
• 0:30
  the thrifty analysis of safety, risk and performance research.
• 0:34
  Visit safetyinsights.org for more research.
• 0:38
  This well known report from Hallowell ET al as part of the
• 0:42
  CSRA studied the statistical basis of TRIR, or total
• 0:47
  recordable injury frequency rate.
• 0:50
  It's titled The Statistical Invalidity of TRIR as a Measure
• 0:55
  of Safety Performance. So what is the TIR TRIFA?
• 1:00
  So TIR? The total recordable injury
• 1:02
  frequency rate is the rate at which a company experiences an
• 1:07
  OSHA recordable incident scaled per 200,000 worker hours.
• 1:12
  In other places around the world it's scaled against 1,000,000
• 1:16
  worker hours and probably other variations.
• 1:20
  So this helps normalise the value to account for different
• 1:22
  working hours and by extension different headcounts.
• 1:26
  Now, even though the report refers to Trir, I'm going to
• 1:29
  call it Triffa since that's what we call it in the land of Oz.
• 1:33
  What were their methods? The researchers had direct
• 1:37
  access to over 3 trillion worker hours of internally reported
• 1:41
  incident data from partner organisations within the US,
• 1:45
  covering a 15 year period. The data set included monthly
• 1:49
  cancer, recordable injuries, fatalities and worker hours.
• 1:54
  They used a range of statistical analytical approaches, both
• 1:58
  parametric and non parametrical tests, and these serve different
• 2:01
  purposes just very quickly. The parametric approach, in
• 2:06
  particular the Poisson distribution, identified
• 2:10
  recordable incidents as discrete events, which means that they
• 2:13
  either occurred or didn't occur and recognise that they vary
• 2:16
  over individual worker hours. Therefore the truffle was
• 2:20
  logically represented as a series of Bernoulli trials and
• 2:24
  the Poisson distribution was deemed the most accurate
• 2:26
  representation. This is pretty appropriate for
• 2:29
  modelling these types of discrete events, in particular
• 2:32
  discrete rare events that occur over a fixed interval of time.
• 2:36
  This really well describes recordable injuries within
• 2:38
  worker hours. They also use some non
• 2:41
  parametric preaches, but I really recommend you check out
• 2:43
  the report to get a description around what these tests are and
• 2:46
  why they use them. The reason why I jumped into
• 2:49
  that background around the parametric tests before,
• 2:51
  particularly the Poisson distribution, is that several
• 2:54
  people on social media were quick to argue that the Poisson
• 2:57
  distribution in this report isn't appropriate for this type
• 3:01
  of modelling, and that the authors should have used a
• 3:04
  negative binomial distribution. First of all, some existing
• 3:08
  research has already shown that incidents are appropriately
• 3:12
  modelled via the Poisson distribution and that the
• 3:15
  assumption of independence isn't actually violated.
• 3:18
  I'm not going to go into that here, though.
• 3:21
  Further, the applied statistics in OHS textbooks from Jenny SAC
• 3:26
  specifically describe the appropriateness of Poisson
• 3:29
  distributions for this purpose, because it's well geared for
• 3:32
  statistically rare events compared to the total exposure.
• 3:36
  However, a recent review study of best practises for
• 3:40
  underreporting accident research from Bazoli and Probes in 2025
• 3:45
  in Safety Science found that while both Poisson and binomials
• 3:49
  were appropriate for this task, in fact, they even argued that
• 3:53
  linear Poisson and negative binomial models returned similar
• 3:57
  results in medium and large sample sizes, showing that it
• 4:00
  actually is appropriate. However, based on that research,
• 4:04
  negative binomials did return a higher accuracy.
• 4:08
  But again, Poisson was a good estimator of negative binomials
• 4:12
  and is appropriate for statistically rare events, which
• 4:15
  is what the statistician Janisac in his OHS statistics textbooks
• 4:19
  and how well ET al in this current report actually argued
• 4:22
  to begin with. Anyway, I've got off my soapbox
• 4:25
  now. So what did they find?
• 4:29
  Well, let's jump into the core findings of this report.
• 4:32
  One, no links between the Triffer and serious accidents
• 4:36
  and fatalities was found. There was no discernible
• 4:39
  statistical association between triffers and fatalities.
• 4:44
  Therefore, the trends in the Triffer isn't statistically
• 4:46
  associated with fatalities, suggesting that they happen for
• 4:50
  different reasons and because of this lack of association, triffa
• 4:54
  is not a proxy for high impact incidents.
• 4:58
  And further, the authors argue that all of these safety
• 5:01
  activities associated with improving triff performance may
• 5:05
  not necessarily help to prevent fatalities. 2 The results
• 5:10
  indicated that changes in triffa are due to 96 to 98% random
• 5:14
  variation. The authors discussed that
• 5:17
  recordables don't occur in predictable patterns and it's
• 5:20
  slightly because safety is a complex phenomenon impacted by
• 5:23
  many different factors. The models were tested to see if
• 5:26
  historical Triffer performance predicted future Triffer
• 5:29
  performance. Found was that at least 100
• 5:31
  months of data was needed for reasonable predictive power.
• 5:35
  It was argued that because Triffer is normally used to make
• 5:38
  monthly or annual comparisons, this finding of around 100
• 5:42
  months of data is like the basic level needed indicates that for
• 5:46
  all practical purposes, Triffer is not predictive in the way
• 5:50
  that it's used. In plain language, injuries are
• 5:53
  largely due to chance and don't follow a consistent statistical
• 5:57
  pattern. But what does random chance or
• 6:01
  chance mean in this context? Because it's been taken out of
• 6:04
  context by some practitioners, meaning that there's no
• 6:07
  underlying cause or the contributing factors.
• 6:08
  Is that what we're arguing? That there are no causes in
• 6:11
  life? That's actually incorrect.
• 6:14
  We're talking about statistical randomness, not some ontological
• 6:18
  or positivistic sense of the world.
• 6:21
  What this means is that the occurrence of recordable
• 6:24
  injuries doesn't follow predictable patterns or occur at
• 6:28
  regular discernible intervals. So while a safety system aims to
• 6:32
  reduce risk, the actual manifestation of incidents over
• 6:36
  short periods is highly variable and statistically behaves like a
• 6:40
  random process. There's also the issue of
• 6:43
  statistical noise, given that 96 to 98% of the variation in TRIFA
• 6:51
  was due to random variation. This implies that when you see
• 6:54
  an organisation TRIFA go up or down from one period to the
• 6:57
  next, it's overwhelmingly likely to be statistical noise rather
• 7:02
  than direct reflection of some sort of improvement or
• 7:05
  deterioration in the underlying safety system.
• 7:07
  You may as well throw dice to predict the next accident based
• 7:11
  on tracking these data. And because they're quite
• 7:14
  relatively rare events, this infrequency contributes with a
• 7:19
  high degree of random variation within TRIFA, especially when
• 7:22
  measured over typical short timeframes, month, or years.
• 7:26
  So in other words, this doesn't mean there's no causality or
• 7:31
  underlying contributing factors. It highlights that many causal
• 7:34
  factors at play in a complex system lead to recordable
• 7:37
  incidents that are, at a sort of macro statistical level over
• 7:41
  typical reporting periods, highly unpredictable and random.
• 7:46
  All right, so let's move on to the next finding. 3 The Triffer
• 7:49
  isn't precise. It lacks precision and shouldn't
• 7:52
  be communicated with multiple decimal points unless hundreds
• 7:56
  of millions of worker hours are amassed.
• 7:59
  The confidence intervals are just too wide for Triffer to
• 8:02
  report it accurately to even 1 decimal point.
• 8:06
  And perhaps my favourite part of this paper follows that, where
• 8:09
  it's shown that if you were to report Triffer to 2 decimal
• 8:12
  places, for instance, you know you have a Triffer of 1.29, you
• 8:17
  would need approximately 30 billion worker hours of data to
• 8:20
  support that claim. So on this point, the authors
• 8:23
  state that Triffer for almost all companies is virtually
• 8:27
  meaningless because they do not accumulate enough worker hours.
• 8:32
  Four, Triffer is statistically invalid for comparisons in
• 8:37
  nearly every practical circumstance.
• 8:39
  It's statistically invalid to use Triffa to compare companies
• 8:42
  or business units, projects, or teams, because most companies
• 8:46
  again don't accumulate enough worker hours to detect
• 8:49
  statistically significant changes.
• 8:52
  On this point, the authors state that Triffa shouldn't be used to
• 8:55
  track internal performance or compare companies, et cetera.
• 8:59
  And Triffa cannot be a single number by extension, because if
• 9:04
  Triffa is largely random, a single number doesn't represent
• 9:07
  the true sort of reflection of safety performance.
• 9:11
  Instead, triffa should be expressed as a confidence
• 9:13
  interval, a range of potential values over extended periods.
• 9:18
  Therefore single point estimates of triffa broken down into
• 9:21
  decimal places over really short periods of time.
• 9:23
  You know, months to years. Said to be statistically
• 9:26
  meaningless for almost every organisation.
• 9:29
  In plain language, instead of reporting our Triffa is 1.29,
• 9:35
  companies should at least report a range like our Triffer is
• 9:39
  likely between 1:00 and 4:00. The next key finding is that
• 9:43
  Triffer is only predictive over very long periods, predictive of
• 9:47
  when you have at least 100 months, more than eight years
• 9:50
  data. So for practical purposes, given
• 9:53
  typical reporting periods in the a month to month, it's not
• 9:57
  predictive. So therefore, what your Triffer
• 10:00
  was last month or last year probably doesn't tell you much
• 10:04
  about what will be next month or next year, unless you have many
• 10:07
  years of data behind it. And even by the stage when you
• 10:09
  have enough data, it may not be relevant anymore to your current
• 10:12
  operations. Therefore, Triver is inadequate
• 10:15
  for measuring intervention impact.
• 10:17
  It's entirely inadequate for attributing changes in safety to
• 10:21
  specific interventions or investments.
• 10:24
  What can we make of the findings?
• 10:27
  Well, first of all, a misconception is that the
• 10:30
  measures themselves are invalid. Therefore, you know, we're
• 10:33
  saying Trifer is somehow invalid.
• 10:35
  But this isn't actually what the report indicated.
• 10:39
  It indicates that the statistical basis of how these
• 10:42
  measures are used are typically invalid.
• 10:45
  Again, using the example of dice, you may as well throw
• 10:48
  dice. We're making causal claims about
• 10:51
  getting 2 sixes when statistically we can't
• 10:54
  differentiate the sixes from chance.
• 10:57
  We assume there's some sort of causal factor of why we keep
• 10:59
  throwing sixes, but statistically we can't
• 11:02
  demonstrate that. In any case, some practical
• 11:05
  implications I think maybe instead of thinking about
• 11:07
  abandoning these measures, some efforts to help them suck less
• 11:12
  is probably warranted, and the report already gives some
• 11:15
  suggestions. Use a RAID instead of single
• 11:18
  point estimate. Carve off those decimals 1
• 11:21
  instead of 1.1. Decouple the measures from
• 11:24
  decision making. For instance, they say if an
• 11:27
  organisation is using Triffa for performance evaluations, then
• 11:32
  they're likely rewarding nothing more than random variation.
• 11:36
  Find ways to increase the sample size and hence the the
• 11:39
  statistical power, the time, the numbers that you're evaluating,
• 11:42
  et cetera. Don't use them to track internal
• 11:45
  performance, or at least between projects or teams.
• 11:49
  By extension, maybe caution when used for gauging contractor
• 11:52
  performance and their use in tendering.
• 11:55
  And of course, they're using incentive programmes.
• 11:59
  Also, I liked how the researcher David Oswald suggested on
• 12:03
  coupling quantitative indicators with qualitative indicators.
• 12:07
  For instance, every number that you present should have a
• 12:10
  qualitative descriptor, a narrative.
• 12:13
  One gives you the what, that's the number, the other gives you
• 12:16
  the rich narrative on how and why it actually matters.
• 12:20
  There's also other tools to help improve the use of injury
• 12:23
  measures. Control charts are a favourite
• 12:25
  of mine. I use control charts for lots of
• 12:28
  stuff and they incorporate Poisson in negative binomial
• 12:31
  thinking. There's also a range of
• 12:34
  statistical methods like significance testing, confidence
• 12:36
  intervals and more. Anyway, that's it on Safe as I'm
• 12:41
  Ben Hutchinson. Please help share rate and
• 12:44
  review and checkoutsafetyinsights.org for
• 12:47
  more research. Finally, Bill Breeder supports
• 12:50
  Safe as by Shania Coffey. LinkedIn, the show notes.