Latest / Safe As Podcast / E17: Critical Decisions & Local Rationality: Tools for making sense of situations

Transcript

E17: Critical Decisions & Local Rationality: Tools for making sense of situations · Safe As Podcast

• 0:07
  Stop blaming and start understanding.
• 0:09
  Do you commonly lament that the actions of our operators didn't
• 0:13
  make sense and what were they thinking?
• 0:16
  If so, then maybe that's a you issue because you're not
• 0:20
  properly framing why things made sense to that person at the
• 0:22
  time, nor providing the right inquisitive framework to
• 0:27
  contextualise their decisions. Let's explore 2 tools that may
• 0:32
  help pull back the curtain on tough decisions.
• 0:35
  Good everyone. I'm Ben Hutchinson and This is
• 0:37
  Safe as a podcast dedicated to the thrifty analysis of safety,
• 0:42
  risk and performance research. Visit safetyinsights.org for
• 0:46
  more research. Today's part includes 2 papers,
• 0:50
  two different techniques but with similar goals.
• 0:54
  The 1st is on the Critical Decision Method or CDM by Klein
• 0:58
  ET al, 1989, titled Critical Decision Method for Eliciting
• 1:04
  Knowledge, published in the IEEE Transactions on Systems, Manned
• 1:09
  and Cybernetics. The second paper is from Louise
• 1:13
  Rowe, titled Local Rationality Question Tool, Understanding why
• 1:18
  it Made sense at the Time, published in Contemporary
• 1:21
  Ergonomics and Human Factors 2025.
• 1:24
  Now on to CDM, the critical decision method.
• 1:28
  A lot has been written on CDM. I'm focusing on an earlier paper
• 1:33
  from Klein, and particularly focusing on the tool itself and
• 1:37
  the questions, rather than the process of running a CDM
• 1:40
  inquiry. If you're interested in the
• 1:42
  background and the actual process of running it, I'd
• 1:45
  suggest reading up on some of these other sources because I'm
• 1:49
  really diving into more specific questions because I think
• 1:51
  they're really interesting for framing a better way to think
• 1:54
  about performance. CDM is a semi structured
• 1:57
  retrospective interview technique primarily used in the
• 2:00
  naturalistic decision making paradigm to research
• 2:03
  professional decision making. Now that's what was sort of
• 2:07
  hinted at in the original paper, but since then, since 1989, it's
• 2:10
  been used very widely outside of NDM.
• 2:14
  It's particularly valuable for eliciting expert knowledge,
• 2:17
  decision strategies and the cues that experts pick up on or men
• 2:21
  don't even realise they've picked up on.
• 2:23
  So this NDM naturalistic decision making context focuses
• 2:27
  on real world problems, often involving time pressures, high
• 2:31
  stakes I'll structured problems, uncertain dynamic environments
• 2:35
  or shifting competing goals. So the CDM tool really supports
• 2:40
  that type of research focus. But again, it's used so widely
• 2:44
  in practise. Now very quickly before I dive
• 2:46
  into the questions, the CDM interview involves an expert
• 2:51
  recounting a single selected, non routine, professionally
• 2:55
  challenging incident where they were the main decision maker.
• 2:59
  That's just a standard approach, but it's not the only approach.
• 3:03
  The interview typically lasts from somewhere up to an hour to
• 3:05
  two hours, with the facilitator and the expert reviewing the
• 3:10
  incident multiple times to add details and gain a deeper
• 3:14
  understanding. Probing questions are used to
• 3:17
  facilitate information retrieval.
• 3:20
  According to the authors, the CVM interview generally follows
• 3:23
  these steps so they select a non routine incident that was
• 3:28
  challenging and where the decision maker might have
• 3:31
  differed from someone with less experience.
• 3:34
  The goal is to identify cases that illustrate unique
• 3:37
  challenges and reveal aspects of true expertise, really applied
• 3:40
  expertise. The participant provides a brief
• 3:43
  description of the incident from the beginning to end, typically
• 3:47
  without interruption, to establish context for the
• 3:49
  interviewer and to activate the participant's memory.
• 3:53
  After that initial account, the interviewer reconstructs the
• 3:56
  incident chronologically, identifying key events, actions
• 4:00
  taken, and information acquired. This helps establish sort of the
• 4:03
  basic understanding from the participants perspective of the
• 4:07
  event and allows for corrections or filling in missing details.
• 4:12
  The next few steps, which I'm really going to gloss over
• 4:14
  involves pointing out key decision points, points where
• 4:18
  judgments affected the outcome were identified for further
• 4:22
  probing and it allows a real deepening or what if inquiries
• 4:26
  into these decision points. So let's jump into the actual
• 4:30
  probes used during or the tool itself for the critical decision
• 4:35
  point interview probes. The first is on cues, and this
• 4:38
  involves things like what were you seeing, hearing, smelling?
• 4:42
  It involves knowledge. What information did you use in
• 4:46
  making this decision and how was it obtained?
• 4:49
  Analogies. Were you reminded of any
• 4:52
  previous experience? Goals.
• 4:54
  What were your specific goals at the time?
• 4:58
  Options. What other courses of action
• 5:00
  were considered or even available to you?
• 5:03
  The basis. How was this option selected or
• 5:06
  other options rejected? What rule or norms were being
• 5:09
  followed? Experience.
• 5:12
  What specific training or experience was necessary or
• 5:14
  helpful in making this decision? Aiding.
• 5:18
  If that decision wasn't the best or the most optimal?
• 5:22
  What training, knowledge or information could have helped?
• 5:25
  Time pressure. How much time pressure was
• 5:27
  involved in making this decision?
• 5:30
  Situation assessment. Imagine that you were asked to
• 5:33
  describe the situation to a relief officer.
• 5:35
  At this point, how would you summarise the situation?
• 5:40
  Hypotheticals. If a key feature of the
• 5:42
  situation had been different, what difference would it have
• 5:45
  made in your decision? These prompts are really useful
• 5:49
  for investigations if you want to try to get into a person's
• 5:53
  head and understand why something made sense to them.
• 5:56
  Picking up on such cues. Decision points, what
• 6:00
  information was even available to them and building on
• 6:03
  alternative hypotheticals are really effective this deepening
• 6:07
  process. They keep having lockout tag out
• 6:09
  issues or striking assets which were apparently known and
• 6:13
  identified. Then maybe considering these
• 6:15
  types of questions would be really useful.
• 6:18
  Next we explore the local rationality question tool from
• 6:22
  Louise Roe. All the CDM structured approach
• 6:25
  helps in analysing retrospective data but doesn't have to, but
• 6:28
  generally is. The Local Rationality Question
• 6:31
  tool, developed by Louise Rowe for healthcare safety
• 6:34
  investigations, provides a specific framework of questions
• 6:37
  to understand why an action or inaction made sense at the time
• 6:41
  from the perspective of the staff involved, without leading
• 6:44
  to feelings of blame or interrogation.
• 6:47
  Likewise to the CDM. As before, I'm focusing more on
• 6:50
  the questions in the tool rather than the process of using this
• 6:53
  tool. This tool lines with the concept
• 6:56
  of local rationality, which emphasises understanding a
• 6:59
  situation from the decision maker's mindset and their
• 7:02
  knowledge, demands, goals and available context at the time.
• 7:07
  You may have heard of this concept called bounded
• 7:09
  rationality from Herbert Simon. Some argue that they're
• 7:13
  different, others more or less argue they're the same.
• 7:17
  The Local Rationality Question tool was developed through a
• 7:19
  literature review of safety science research, adapting
• 7:22
  existing questions to ensure they explored local rationality
• 7:26
  while prioritising psychological safety.
• 7:29
  It composed a collection of questions categorised for easy
• 7:31
  navigation. Now there's like 25 prompts in
• 7:35
  this tool, so I can't cover them all, but the key categories and
• 7:39
  example questions include the situation.
• 7:43
  This is questions to explore the dynamic elements of the work
• 7:45
  environment and how the situation unfolded.
• 7:48
  Like describe to me what was happening at the time.
• 7:51
  If you had to describe the situation to your colleague at
• 7:54
  this point, what would have you told?
• 7:56
  Can you tell me about any time pressure to complete the task or
• 8:00
  any limitations on what you were able to do at the time?
• 8:03
  Tell me about any reassessments of the situation that might have
• 8:06
  occurred. Was there any immediate
• 8:08
  feedback? Any careful monitoring?
• 8:11
  Can you break down the task into 3 to 6 steps?
• 8:15
  Over these steps? Tell me a bit about those that
• 8:18
  required assessment or decision making or problem solving if
• 8:21
  any. And this could involve coming up
• 8:23
  with a task diagram for very complex procedures or
• 8:25
  situations. Next is on thoughts and decision
• 8:29
  making questions to understand the individual's focus, goals
• 8:32
  and information that were utilised.
• 8:35
  As in, describe what you are focusing on.
• 8:37
  Can you explain what your aim or the goal was during this time?
• 8:41
  Tell me about what info you used to help you make this decision.
• 8:44
  How did you obtain this information?
• 8:47
  Tell me about any barriers you had obtained in this
• 8:48
  information. Can you tell me about any
• 8:51
  previous experience you had in similar situations?
• 8:54
  Were there any other options available at the time that
• 8:56
  probably should have been evaluated in hindsight?
• 9:00
  Preparedness was another key category, so this was questions
• 9:03
  concerning training guidelines and other knowledge sources.
• 9:08
  It's enacted with questions like, can you tell me a bit
• 9:11
  about the training that you or the staff had to deal with this
• 9:14
  situation? What training knowledge or
• 9:17
  information might have helped? Tell me what guidelines or
• 9:20
  policies there are in your unit to help manage this situation.
• 9:25
  Another category is communication questions
• 9:27
  exploring team interaction during the incident.
• 9:30
  Describe how the team communicated to each other
• 9:32
  during this time. Another category is anticipation
• 9:36
  and thinking ahead. Questions designed to understand
• 9:39
  expectations and imagine consequences.
• 9:41
  Like. Describe what you were expecting
• 9:43
  to happen. Can you explain to me what you
• 9:45
  were hoping would happen as a result of whatever.
• 9:49
  So the local rationality tool is designed for flexible use as a
• 9:52
  guide or inspiration rather than as a strict and mechanical
• 9:56
  script encouraging an open ended, inquisitive and none
• 9:59
  blame approach. It's argued that by focusing on
• 10:03
  what made sense at the time, it helped investigators reduce
• 10:06
  hindsight bias and promote A fairer and more restorative
• 10:10
  culture. So both the CDM that I covered
• 10:13
  first with its detailed interview probes and structure
• 10:16
  analysis, and the Local Rationality Question tool with
• 10:21
  its focus on understanding local rationality, are designed to
• 10:24
  gain deep insights deepening into decision making in real
• 10:28
  world, often challenging situations.
• 10:31
  I highly recommend you check out both sources.
• 10:34
  I really only scratch the surface and see if they're
• 10:37
  valuable additions to your adaptive tool boxes.
• 10:41
  That's it on Safe. As I'm Ben Hutchinson, please
• 10:45
  help share, rate and review and check out safetyinsights.org for
• 10:49
  more research. Finally, feel free to support
• 10:52
  Safe AS by shouting a coffee link in the show notes.