Latest / Safe As Podcast / Ep 6: Audit Masquerade - How audits provide comfort rather than treatment for serious safety problems

Transcript

Ep 6: Audit Masquerade - How audits provide comfort rather than treatment for serious safety problems · Safe As Podcast

• 0:07
  Imagine working somewhere you think has effective risk control
• 0:10
  processes because the audit say so.
• 0:15
  You see the reports, the sign offs, the procedures.
• 0:18
  But what if those audits are actually missing the most
• 0:21
  critical dangers? What if the story of auditing is
• 0:24
  how checking the boxes can tragically over shadow what
• 0:28
  truly keeps us safe, revealing a hidden disconnect between
• 0:32
  paperwork and reality. Good day everyone.
• 0:36
  I'm Ben Hutchinson and this is Safe as podcast dedicated to the
• 0:39
  thrifty analysis of safety, risk and performance research.
• 0:43
  Visit safetyinsights.org for more research.
• 0:47
  So this is one of my papers Co authored with Drew Ray and
• 0:50
  Sydney Decker in 2024 titled Audit Masquerade, How audits
• 0:56
  provide comfort rather than Treatment for serious Safety
• 0:59
  problems, published in Safety Science.
• 1:02
  Essentially we unpacked 71 audit reports, a mixture of first,
• 1:06
  second and third party reports from a large Australian design
• 1:10
  and engineering, construction and maintenance company.
• 1:13
  Over 16 separate and independent auditing firms were included in
• 1:17
  the data and what we looked at was what were the audit reports
• 1:21
  looking at. So we evaluated the corrective
• 1:24
  actions and the observations from reports.
• 1:26
  We wanted to know what the auditors look at, how do they
• 1:29
  justify, what sort of evidence they use, All that sort of
• 1:32
  questions, volume, background. While many different types of
• 1:36
  audits exist, and they might have different goals and scopes,
• 1:40
  one line of argumentation is that audit shouldn't be overly
• 1:44
  concentrated on compliance against the administrative
• 1:46
  aspects of the system and documentation.
• 1:49
  Instead, audit should prioritise addressing tangible and
• 1:53
  intangible factors linked to actual characteristics, states,
• 1:58
  influences, objects or practises in the workplace to enhance the
• 2:02
  efficiency and the safety management system in achieving
• 2:06
  the goals of the organisation. Hence we argue that a core goal
• 2:10
  of audit really should be within safety critical context about
• 2:14
  minimising decoupling. So decoupling is the distance
• 2:19
  between the intended purpose or function of something like an A
• 2:22
  safety plan versus the actual function in practise of the
• 2:25
  artefact. So if we have a gas management
• 2:29
  plan in a mine, decoupling would be the actual management of gas
• 2:34
  isn't being managed as expected by the plan.
• 2:38
  Some criticism of audits is that they can focus on this thing
• 2:40
  called surface compliance in the very superficial elements of
• 2:44
  their safety approaches and audits may over prioritise the
• 2:48
  collection and review of documents.
• 2:50
  Authors blew it in O'Keefe in their evaluation of industry
• 2:53
  auditing, detailed how paperwork was collected to to create an
• 2:58
  auditable trial where the quality of information within
• 3:01
  the documents was often a secondary consideration.
• 3:05
  Also, they observed the disparity in auditing between
• 3:08
  the paperwork that keeps people safe versus paperwork that helps
• 3:12
  complete audits. Their words.
• 3:14
  So let's jump in. What were the results?
• 3:17
  Well, overall, we're looking at the corrective actions and
• 3:20
  observations from these audit reports.
• 3:22
  We found most of the findings related to solving or resolving
• 3:26
  some sort of incomplete or missing documents, resolving
• 3:30
  missing site signage, inspecting, placing, or
• 3:33
  reviewing very sort of minor emergency equipment like fire
• 3:37
  extinguishers and first aid kits to submit or display documents
• 3:41
  like posters or sending a register to somebody, or
• 3:45
  resolving incorrect version that was on formatting.
• 3:48
  We also observed that most of the findings were either only
• 3:51
  moderately or weakly connected to some sort of physical issue
• 3:54
  or hazard. In other words, only 16% of the
• 3:57
  findings were strongly connected to some sort of physical issue
• 4:00
  or hazard. There was virtually no examples
• 4:03
  of elimination or the higher order controls in the hierarchy.
• 4:07
  And virtually all of the corrective actions which we said
• 4:10
  there was a strong connection with a physical issue or has it
• 4:13
  on site focused on rectifying sort of immediate or incidental
• 4:17
  physical conditions like slips and trips.
• 4:20
  What we did find though was audits with specific themes like
• 4:23
  electrical or traffic audits tended to derive slightly better
• 4:27
  quality corrective actions or at least better quality in the
• 4:30
  sense that they're more directly connected to some sort of
• 4:33
  physical issue or physical hazard.
• 4:35
  And not surprisingly, most of the corrective actions that were
• 4:38
  rated as weak, weak is as far as they had only a very indirect or
• 4:42
  weak connection to solving some sort of physical issue or
• 4:45
  hazard. Actually all of them were
• 4:48
  administrative. So there was really wasn't a
• 4:51
  clear link between the administrative action on solving
• 4:54
  some sort of tangible issue or hazard.
• 4:56
  We also looked at audits that unpacked communication on side
• 5:01
  and what it was found is that even audits that were geared
• 5:03
  towards auditing communication practises, they very rarely
• 5:07
  evaluated actual communication on side.
• 5:10
  And a total of 35 corrective actions or observations focused
• 5:14
  on communication. Only seven of them commented on
• 5:17
  either the content of the communication, the effectiveness
• 5:21
  of the communication or the quality and retention of the
• 5:25
  communicate information. Hence, most audit findings focus
• 5:29
  solely on the outputs of communication like a signed
• 5:32
  toolbox talk or pre start or pre start attendance sheets or that
• 5:36
  the content was displayed on a notice board.
• 5:39
  So actually all of them didn't assess the quality or content of
• 5:43
  the communication. This suggests that audits might
• 5:46
  prioritise verification of artefacts documents over
• 5:50
  assessing the quality or content of information.
• 5:54
  Also, in this sample, we found that actions of communication
• 5:57
  were used to address issues that really didn't relate to
• 6:00
  communication. For instance, gaps in risk
• 6:03
  assessments were addressed via toolbox talk rather than
• 6:06
  addressing the risk assessment itself.
• 6:09
  So the audits effectively rein in this decoupling effect.
• 6:13
  Well, based on these findings, no, not really.
• 6:17
  The audits rarely dug beneath very superficial matters of
• 6:20
  documentation and system administration.
• 6:23
  Even when auditing did focus on operational work and and
• 6:26
  operational hazards, the emphasis really shifted to the
• 6:30
  remediation of very trivial or incidental things like the slips
• 6:34
  and trips and cables. And so it really appears that
• 6:39
  audits myopically take a find and fix approach, focus on
• 6:43
  immediate side issues rather than a deeper systematic focus
• 6:47
  on learning and improvement. Also, the results suggest that
• 6:51
  there may be a type of masquerade, a symbolic activity
• 6:55
  configured for surface documents and practises and is unable or
• 6:59
  unwilling to ask substantive and critical questions about risk
• 7:03
  and work. Again, only 16% of the 327
• 7:08
  corrective actions that were assigned to really improve
• 7:11
  health and safety management were only linked to a physical
• 7:14
  source of harm or a hazard in 16% of all the actions with the
• 7:19
  remaining corrective actions really just wake or moderately
• 7:22
  connected to physical issues. So in discussing the findings,
• 7:26
  we point out that audits in this sample focused on surface
• 7:30
  compliance and there was 2 variations of surface
• 7:32
  compliance. 1 is we called an illusion of depth.
• 7:36
  Audits were found to address immediate side issues you have.
• 7:39
  We're implementing that missing lifting equipment register
• 7:42
  without calling for the investigation of the underlying
• 7:45
  causal factors. So why did that issue exist in
• 7:48
  the 1st place? Didn't seem to ever get asked.
• 7:51
  Additionally, no instances were observed of corrective actions
• 7:55
  directing the oddities to systematically rectify a family
• 7:59
  of issues. So how does the whole business
• 8:01
  learn? We also propose that artefacts
• 8:04
  take on the guise of the issue. So audits frequently focused on
• 8:09
  revising some sort of document without accurately addressing
• 8:12
  the underlying physical issues. So solving the paperwork was
• 8:16
  conflated with solving the actual issue.
• 8:18
  We also found that audits couldn't really identify
• 8:21
  particular issues, couldn't or wouldn't.
• 8:24
  There was a lack of connection between requirements and
• 8:26
  specifications. So audits were found to focus a
• 8:29
  lot on minor specifications like signatures, templates, version
• 8:34
  numbers, but weren't so well configured to focus on the
• 8:37
  requirements. You know, what's the actual
• 8:39
  purpose of that process? What is it trying to achieve?
• 8:42
  But to be clear, we did find benefits of auditing.
• 8:46
  It doesn't mean audits didn't have some sort of value or
• 8:48
  purpose. Audits were demonstrably
• 8:50
  effective at identifying readily observable site conditions and
• 8:54
  lower tier hazards. They're also effective at
• 8:57
  verifying the presence of documented systems and
• 9:00
  deliverables. And they'll likely have other
• 9:02
  benefits that we couldn't measure in this study.
• 9:05
  So overall, this evidence highlights AT audits focused
• 9:09
  largely on evaluating and modifying documents, conflating
• 9:13
  the presence of a document as evidence of the effectiveness of
• 9:16
  the process, even though they're not the same thing.
• 9:20
  So audits focused on surface compliance activities and lacked
• 9:23
  a focus on critical components and critical risks.
• 9:27
  These findings suggest that some auditing approaches may be at
• 9:30
  risk of tweaking documents or addressing insignificant
• 9:33
  physical issues at the expense of properly addressing critical
• 9:37
  issues or hazards. So audits could be a false
• 9:41
  veneer that the issues are being properly managed, the audit
• 9:44
  masquerade. What can we make of these
• 9:47
  findings? There's way too much to go
• 9:50
  through here. What some simple clear
• 9:53
  reflections for me is being clear about what we expect from
• 9:56
  audits in the 1st place. For instance, is that audit,
• 10:01
  that particular type of audit, purely a desktop documentation
• 10:04
  check? If it is, there's likely to be
• 10:07
  no real misalignment with goals. It's probably doing what we
• 10:11
  expected to do. If you ask 20 of your leaders in
• 10:14
  the business what they think audits are doing, I bet
• 10:18
  someone's going to say they think the audits are verifying
• 10:20
  the effectiveness of our risk systems.
• 10:23
  If that's the case, then that needs to be firmly investigated
• 10:26
  and challenged because your audits may not be doing that.
• 10:30
  Now, of course, there's limitations, some of them, even
• 10:33
  though we did have 16 separate and independent audit companies
• 10:37
  as part of the data source, generalizability still has to be
• 10:40
  considered. Also importantly, it's based on
• 10:44
  document analysis by a person who wasn't present for those
• 10:47
  audits, so we don't have all of the context discussed during the
• 10:50
  audit. That's it on SAFE.
• 10:53
  As I'm Ben Hutchinson, if you found this useful then please
• 10:56
  share rate and review and checkoutsafetyinsights.org for
• 11:01
  more research.