Latest / Show Me The Evidence / The Crisis in Surgical Training & Proficiency-Based Progression (PBP)

Transcript

The Crisis in Surgical Training & Proficiency-Based Progression (PBP) · Show Me The Evidence

0:07
So Tony, I want to start with, I suppose, an uncomfortable question.
0:12
If you had surgery today at a major teaching
0:14
hospital with a fully credentialed surgeon,
0:17
what's the honest answer to the question,
0:20
how would you know they're actually competent to operate on you?
0:24
I mean, the bottom line is you wouldn't know.
0:26
I started working in this area about 30 years ago,
0:29
and the assumption was that because somebody was a consultant in a public hospital
0:34
and maybe they had a private practice, it was an indication that they were good.
0:38
And then what I learned was it's actually to do
0:42
with, okay, where did they do their fellowship?
0:44
Where were they trained? Who did they work under?
0:47
What courses did they do and so on?
0:49
But over the years, what I've learned, it means absolutely nothing
0:52
because somebody can attend the course and they can walk away,
0:57
and you've no idea what they actually have
0:59
learned cognitively or what they can actually do.
1:03
And it turns out that when we assessed individuals
1:07
about what they can actually do at the end of a course,
1:08
we were shocked.
1:10
I mean, the first few studies that we did, we
1:12
thought there was something wrong with the methodology
1:14
because what we were finding was that large
1:16
numbers of the trainees could do very little.
1:19
They knew very little about the procedure and they could do very little safely.
1:24
And then on top of that, what we started to learn in the early 2000s,
1:29
when we did a study with American College of Surgeons,
1:32
where we assessed only attending consultant level laparoscopic surgeons,
1:37
that we found that some of them were performing
1:38
up to 20 standard deviations away from the mean
1:41
in the wrong direction.
1:43
And again, we questioned, you know, we must
1:46
have done something wrong. This can't be right.
1:48
These are fully credentialed, you know, hospital consultants who are attendings,
1:52
working mostly in university teaching hospitals.
1:55
But over the years, what we've learned is the
1:58
methodology was signed and the results were robust.
2:02
What we weren't ready for was to accept that some
2:05
of the individuals working in our teaching hospitals,
2:08
in our general hospitals, they're really not very good at what they do.
2:12
So when you go into a hospital to have a procedure done,
2:16
the credentialing that tells you something,
2:18
it tells them being through the process of that country or that hospital.
2:22
But it doesn't tell you anything about the skills level.
2:25
Probably the best way to find out about the skills level of the surgeon
2:28
is to ask one of their colleagues who will give you an honest answer,
2:31
or better still, ask the theatre sister or
2:34
the nurse in charge of the operating room.
2:37
They're more likely to know who they wouldn't have
2:39
operating on them, their family or their children,
2:42
because there's individuals that you really don't want operating on you,
2:47
your family or your friends.
2:51
It's eye-opening.
2:53
Do you think it's fair to characterise the current model of professional training
2:57
as one of the most consequential unexamined
2:59
assumptions in modern health institutions?
3:03
Yes, I think that's completely right.
3:05
We're relying on a training paradigm that was
3:07
developed during the late 19th, early 20th century.
3:10
I buy William Halstead at Johns Hopkins Hospital.
3:14
Halstead, my reading of him, I could be wrong,
3:16
my reading of him was he was a challenging individual to work with.
3:20
He had very exacting standards, and not everybody that
3:23
started their surgical training with him actually completed.
3:27
He learned a lot of lessons from Europe,
3:30
so the probability is that he was quite
3:32
authoritarian in his approach to the trainees.
3:35
But the thing is that the trainees had sufficient procedure volume and supervision
3:41
by Halstead or his colleagues to ensure that
3:44
they had the opportunity to become a good surgeon.
3:46
And if they didn't meet the skills levels that
3:50
Halstead or his colleagues expected, they didn't finish.
3:53
The problem with that was it was very subjective.
3:56
And today it's still very subjective.
3:58
A lot of how a trainee progresses depends
4:01
on the consultant that they're working with,
4:02
the references that they give them and so on.
4:05
And the whole process, there are far, far too many holes in it.
4:10
And for the 21st century, with the rate of change of technology,
4:14
with the complexity of the technology that the surgeon,
4:18
and it's not just surgeons, it's cardiologists, it's any of the proceduralists.
4:22
The technology is changing a pace and that's going to continue.
4:25
The bottom line is that we need to know what somebody can do,
4:30
sorry, what they know and what they can do to some sort of performance level.
4:34
Because what we're 100% certain about now is
4:38
that that knowledge and that performance level,
4:40
coupled, you can't have one without the other, that
4:43
that's almost certainly going to impact on outcomes.
4:46
It's not a one-to-one correlation.
4:50
It's not a 100% predictor, but the data
4:52
clearly show that if somebody's not very good,
4:55
or they're not very knowledgeable about the procedure or task that they're doing,
4:59
they're much more likely to have a bad
5:01
outcome than somebody that knows the procedure well
5:04
and is quite good at it.
5:07
And given what we're measuring and training at
5:10
the moment, which is time in seats, hours logged,
5:13
courses completed, these are the currencies
5:15
of almost every training program on Earth.
5:18
Why are they the wrong metrics and what should we be measuring instead?
5:22
It's slightly worse than that because the
5:26
medicine and particularly surgery has caught
5:28
onto the idea that they must measure performance.
5:31
And now if you take a look at the training
5:33
programs around the world, yes, there's an emphasis on
5:36
procedure numbers.
5:37
Yes, there's an emphasis on the variety of
5:41
procedures that they do and the mix of procedures and so on.
5:45
But they've also, what's also been included in
5:48
the assessment is Likert type scale assessments.
5:52
These were originally developed to assess
5:56
attitudes to things, personality attributes,
5:59
and so on.
6:00
They're totally unfit for assessing technical
6:03
performance or cognitive performance, but this
6:05
is what's been widely used.
6:07
So it gives the illusion that performance is
6:10
being assessed, but they're not being assessed.
6:12
I mean, these are entirely subjective and
6:16
there's no training on how to use these instruments.
6:19
And when we assess the use of those
6:21
instruments, the correlation or the inter-rater reliability
6:25
between two independent assessors, it's very weak.
6:29
And if a trainee was to take a case
6:32
against the hospital or the training program that
6:35
assessed them, the hospital or the training
6:37
program wouldn't win because the data is very
6:39
clear.
6:40
These instruments do not assess performance reliably.
6:44
I think a lot of the training programs that
6:45
we have, the assessment programs and so on,
6:48
they're smoking mirrors.
6:49
I mean, the bottom line is that around the
6:52
world, we now know that when trainees finish
6:55
their residency, whether it's in the United
6:57
States or whether it's in the UK or Europe,
7:01
they're not of the same performance level,
7:04
the same skill level as consultants or attending
7:07
surgeons who completed 20 odd years ago.
7:09
And that's for a variety of reasons.
7:11
It's not the trainees fault, I don't think.
7:14
I mean, there's reduced workers in Europe because the assumption, well,
7:17
it used to be the assumption the more hours you worked, the better you were.
7:20
That's not correct.
7:22
In the US, there was a reduced work hours
7:25
because of a fairly disastrous case in New York,
7:29
Lebizian, where the residents in charge of the case were actually blamed for it.
7:33
So the residents have less opportunity to acquire their skills.
7:39
In the past, there would be significantly more operating experience.
7:43
Today, the technology is much more complex.
7:45
The consultants and the attendings are less likely,
7:48
or less enthusiastic about allowing the trainee to
7:53
use the technology to operate on one of their patients.
7:57
And so the trainees are being pushed from different quarters.
8:01
And the assumption is that training in the skills lab will substitute for the
8:06
experience that they're losing in the operating room.
8:09
And that's problematic as well, because trainees go to a simulation centre
8:15
and they talk about playing on the simulator, which I find appalling.
8:20
The simulator is a tool for facilitating or
8:23
helping the trainee acquire a level of skills.
8:26
But what the data clearly shows is if you don't have the performance metrics to
8:31
guide the trainee through the performance,
8:33
to give them explicit constructive formative feedback on their performance,
8:37
literally they are only playing on the machine.
8:40
So they need guidance.
8:41
So the simulation is just a tool, whereas in medicine and in
8:44
surgery in particular, it's seen some sort of magic bullet.
8:47
It's not. It's a tool. And it has got to
8:50
be used as a tool, as part of a curriculum.
8:53
And the curriculum has to come from the clinicians,
8:56
the surgeons, the cardiologists, the radiologists,
8:58
who actually know how to do the procedure.
9:01
Not individuals who have an opinion on how to do the procedure,
9:04
but individuals who are experienced in the procedure,
9:07
who know the best approach for a trainee at the
9:10
start of their learning curve to learn that procedure.
9:13
And by and large, that's not really what's happened.
9:16
There are a few exceptions, like the Arthroscopic Association in North America,
9:20
like the European Robotic Urological Society.
9:24
So there are a few exceptions, but by and
9:27
large, simulations use it in education experience.
9:30
And the bottom line is what we're left with is trainees who are graduating.
9:35
As, you know, fit to be independent consultants.
9:38
And when you objectively assess the performance,
9:41
even on a relatively straightforward procedure,
9:43
we find that the vast majority can't do the procedure safely.
9:48
So Tony, now that you've identified the
9:49
problem very well, you've also presented a solution.
9:52
And your study with Neil Seymour published in the Annals of Surgery in 2002
9:57
showed that simulator-based residents reached proficiency faster
10:01
and made 60 percent fewer errors than their non-virtual reality trained peers.
10:06
But that was over 20 years ago.
10:08
Why didn't that finding transform surgical training overnight?
10:14
To some extent it did.
10:15
I mean, the paper that you talk about we did in
10:19
2000 at Yale, the Department of Surgery in Yale.
10:22
And it was the American Surgical Association
10:25
education paper in their April 2002 meeting.
10:29
It was published in Annals in October of that
10:32
year and was a citation classic by Christmas.
10:35
And the surgical community were looking for some
10:37
sort of solution to the problem of reduced workers
10:41
and reduced procedure volume for the trainees.
10:45
Virtual reality simulation seemed to be ideal.
10:48
But it had been around, Satava first
10:50
talked about it in the late 1980s, early 1990s,
10:53
so it had been around for over a decade and nothing really had happened.
10:57
And what the surgical community wanted was robust evidence demonstrating that
11:02
simulation produced a better training outcome.
11:06
And we provided that.
11:08
The American College of Surgeons took that on board.
11:11
Jerry Healy led a meeting in April or March in Boston College.
11:17
And the question was, what position should American
11:21
College of Surgeons take towards simulation-based training?
11:24
And at the start of the meeting, they
11:26
basically said, well, we need a clinical trial.
11:28
And I piped up and said, okay, we're going to
11:30
present it at the American Surgical Association.
11:33
This is what the results suggest.
11:36
Jerry Healy is an interesting man.
11:38
He worked at Boston College.
11:40
He was a paediatric surgeon.
11:41
And Jerry could be quite a tough character.
11:45
And I'm not sure that he was taught by
11:46
everybody's popularity list, but he was a good leader.
11:50
Jerry listened to what was said, and he turned the meeting on its head.
11:54
And then rather than asking, oh, should the American College of Surgeons, what
12:00
position should they take in terms about simulation-based training,
12:03
he said, how should the American College of
12:05
Surgeons adopt and implement simulation-based training?
12:10
Which, I mean, I thought Jerry showed true leadership that day.
12:15
The rest of the meeting rolled in behind him.
12:17
Okay, there were awkward questions and so on, but basically they got up.
12:21
And they knew that if we were presenting at the American
12:23
Surgical Association, the study was good and the data were signed.
12:28
So in 2006, the American College of Surgeons
12:30
established the accredited educational institutes.
12:35
And all of them were to establish some sort of accredited simulation centres.
12:39
These were evaluated in terms of the types of surgery
12:43
and surgeons who were going to teach or being taught on it,
12:48
the instrumentation, the space and all the rest of that.
12:50
And it's quite a rigorous process.
12:52
But the problem is they have something like over
12:55
100 of these centres in the US and around the world.
12:58
They're all accredited and they're all doing different things.
13:02
And they're all using simulation as an educational experience.
13:05
They're all using Likert skills and so on.
13:08
And that really shouldn't have been the function of...
13:11
But Tony, does that not show then that you're finding
13:15
in that paper with all the research that you've done,
13:17
both then and since, that it hasn't transformed
13:20
surgical training to the overwhelming extent that you want.
13:25
I appreciate it, the citation classic.
13:27
And I appreciate that with leaders like Jerry,
13:29
as you mentioned, things can be pushed forward.
13:31
But eventually there's a regression to the mean, a regression to the status quo.
13:38
Yeah, I mean, it has...
13:40
I mean, for example, if the American College of Surgeons
13:44
decided today to use an outcome based approach to training,
13:49
they have everything in place.
13:51
What they have not done is insisted on a standard, the establishment of performance
13:57
metrics, the establishment of proficiency.
13:59
I mean, they have all the pieces in place, but that last.
14:04
And so, I mean, a lot of it's really like smoke and mirrors.
14:07
Yes, the trainees enjoy it. Yes, the faculty enjoy it.
14:11
Yes, the facility looks fabulous and all the rest of it.
14:13
But when you assess the trainees, when they come out
14:16
at the end of the training, I mean, which they don't.
14:19
But when we do, what we find is they can't do very much.
14:22
And we know now for certain, since 2015, having
14:27
simulation facilities, no matter how good they are,
14:31
having those facilities available, have
14:33
trainees go through, it means almost nothing.
14:37
You know, it's an interesting educational experience.
14:39
And we know that from the Arthroscopic Association of North America.
14:43
We completed a prospective randomised and
14:46
blinded study of something like trainees from 20-odd,
14:50
ACGME approved orthopaedic residency programs in the United States.
14:55
And they all come into Chicago for a simulation-based training course.
14:59
One group got what they would normally get today.
15:02
And I've got to say, ANA prides itself on doing good training.
15:07
So the first group got by any measure what ANA would consider good training.
15:12
The second group got the ANA training, plus they had simulation.
15:17
And group three got what ANA would normally
15:19
give, plus proficiency-based progression simulation.
15:23
So all three of them shared the same
15:25
fundamental elements, which was the ANA curriculum.
15:29
Group two got simulation added to that.
15:33
And group three had simulation, but they had to
15:35
demonstrate proficiency benchmarks the whole way through.
15:38
And what we found was that the simulation, the ANA trained group did worse.
15:44
And I think ANA was kind of shocked by that.
15:46
The group that had simulation added to their training, the assumption was it was
15:50
going to have some sort of marked impact.
15:53
It didn't, like five or six percent more
15:55
of the trainees demonstrated the benchmark.
15:58
Whereas the proficiency-based progression
16:00
train group, 75 percent demonstrated the benchmark.
16:03
And I mean, the results were quite clear.
16:06
Having a good curriculum, having fabulous facilities and adding simulation to it
16:10
means almost nothing in terms of the training outcomes.
16:14
It's the metrics used to deliver a curriculum on
16:18
the simulation that actually makes a difference.
16:22
That's the well-articulated exposition of the problems with current training.
16:28
But then we allied that to the issue we're going to
16:31
uncover hopefully with the next question, which is,
16:34
there's a cultural assumption that experience equals competence.
16:38
That when you have a seniority, that grey hair means experience effectively.
16:42
So what does the evidence that you've gathered actually say about that?
16:45
Yeah, I know I started out with that assumption 30 years ago.
16:51
And I would, when I was going to assess a surgeon or video record the performance,
16:57
I would ask him, okay, do you know how to do this task?
16:59
Yes. Okay. Have you done it before? Yeah.
17:02
And how many times have you done it before?
17:05
Because they assumed that the last two almost
17:08
certainly were going to impact on their performance.
17:11
I now ask another question. Show me.
17:14
Because what I find is the correlation between the first
17:17
three questions and the last question means absolutely nothing.
17:21
Somebody can have a large amount of procedure volume experience.
17:27
And you would think that they're learning during that. A lot of them don't.
17:31
The mistakes that they were making or the errors that they were making at the start
17:36
of their procedure experience, they're still making it today.
17:39
That's not the case for all of the surgeons. I mean, I think the majority, I
17:43
wouldn't say the vast majority, I would say the
17:46
majority actually do learn from their experience.
17:50
Experience is not a bad thing. It's correlated. It absolutely is correlated in the
17:54
past with procedure outcomes, with skill.
17:59
But the problem is the correlation is so
18:01
noisy, you have no idea what you're getting.
18:04
And it was used as a surrogate for the
18:08
performance or the operating skill of the surgeon.
18:11
And what we found is there's actually a better
18:13
way to establish the skill level of the surgeon.
18:16
And that's objectively assessed, their interoperative performance on a
18:20
straightforward case, not a complex case.
18:23
Because the complex case can throw you so many curve balls.
18:26
A straightforward procedure that the surgeon or the cardiologist or the radiologist
18:31
should know how to perform has done frequently.
18:33
You record that, you objectively assess it with their metrics. That'll tell you
18:37
more about their skill level than their procedure volume.
18:40
And one of the shocking things that we find is the
18:44
trainees, I don't mind, the trainees are still fixable.
18:47
And as you would understand, I mean, there's
18:50
different performance levels the trainees are at.
18:53
That's fine. They're still in the training. And
18:54
once you identify these issues, you can fix that.
18:57
But if you have a consultant or an attending out in independent clinical practice,
19:02
and you find that they're performing, not only are they
19:05
performing significantly worse than some of their colleagues,
19:08
they're consultant colleagues, they're performing significantly worse than some of
19:12
the residents in training. That's a significant issue.
19:16
The first time we found that we probably didn't believe it.
19:20
For example, the study that we did at the American College of Surgeons in 2001, and
19:24
that was published in the Journal of the American College of Surgeons 2003,
19:28
we were shown that some of the consultants were performing particularly badly.
19:33
To be honest, in 2001, we thought we did something wrong. I mean, we set out to
19:38
establish what's the correlation between doing a task in a virtual reality
19:42
simulator and doing the exact same task in a box trainer.
19:45
That's what the college asked us to address.
19:49
The most important finding to come out of that study was
19:51
some of the consultants really weren't performing very well.
19:53
And we have now replicated that across procedures, across disciplines.
19:59
I mean, and what we're finding is some of the consultants that are actually in
20:02
clinical practice, when we objectively assess their
20:05
performance from their video recorded clinical performance,
20:10
we find that some of them do very, very
20:11
poorly indeed, and indeed some of them are unsafe.
20:17
Now that we've identified the problem, we'll get to the solution and the solution
20:22
that I've mentioned in many talks as being the single most impactful methodology
20:25
I've come across in my career as an instructional designer.
20:29
So Tony, to proficiency-based progression.
20:31
Walk us through the mechanics. What does a
20:33
proficiency-based progression curriculum look like?
20:36
What are the errors, metrics, critical and
20:38
sentient errors? Can you explain the methodology to us?
20:44
Proficiency-based progression. One of the reasons I developed a scientific approach
20:49
to training was there was so much in surgical training,
20:53
surgical appraisal, surgical assessment that was subjective,
20:56
that I thought it was unfair on the
20:58
trainee or the consultant who was being assessed.
21:01
So I approached it with a more scientific eye.
21:05
And one of the big issues in the late 1990s, early 2000s
21:09
was the how do you establish a benchmark for a procedure?
21:12
And to me it seemed obvious. You establish the benchmark on the surgeons who are
21:18
actually good at the procedure, and that's what we did at Yale.
21:20
But to do that, you've first of all got to develop the metrics.
21:24
And one of the things I learned fairly early on is that you have individuals who
21:27
have lots of opinions about how to do a procedure.
21:31
And when you check with colleagues that you know that are pretty good at what they
21:34
do, they don't necessarily agree with that approach.
21:38
So one of the first things about
21:39
proficiency-based progression is establishing the metrics.
21:42
What I do is I get three to five clinicians who are good at the procedure, not
21:46
necessarily world leaders, but individuals who you'd be
21:50
happy operating on you, your child, your partner or your family.
21:54
I get those three individuals or five individuals together and we work through a
21:58
straightforward or reference approach to the performance of the procedure.
22:02
This is a procedure that the trainee should be
22:04
able to do at the start of their learning curve.
22:07
And we assume there's going to be no complications.
22:09
We assume it's being done on a straightforward patient.
22:12
And we work out what are the phases of the procedure, the chunks of the procedure.
22:16
And within each phase, what are the procedure steps and
22:20
instruments that the trainee must use to achieve that phase.
22:25
And then most importantly, what was new to training
22:30
was we established deviations from optimal performance.
22:34
Okay, we're looking at a trainee or somebody doing the procedure and they do
22:38
something you probably shouldn't have done that.
22:41
Now, we call that an error and it's a deviation from optimal performance.
22:46
Medicine and surgery have difficulty
22:49
conceptualising that because they consider errors, critical error.
22:53
And that is something that you see a consultant or a
22:56
trainee doing and you go on, what the hell are you doing?
22:59
It's something that's going to compromise
23:01
patient safety or the safe completion of the procedure.
23:05
So those are your building blocks for the
23:08
procedure or for proficiency based progression.
23:11
You have the steps of the procedure, you have the phases,
23:13
you've got the procedure, you break it up into chunks or phases.
23:16
You then have the steps, the errors, which are
23:19
deviations from optimal performance and the critical errors,
23:22
which actually compromise the procedure or patient safety.
23:26
Then you've got to validate because I mean those just might
23:28
be the opinion of three surgeons who were there on the day.
23:33
And so one, you've got to then get consensus
23:36
from their peers, not agreement, consensus.
23:39
Okay, that may not be the way I would do the procedure, but it's not wrong.
23:44
Then you would take them to construct validity, do the metrics.
23:47
One, can you observe them? Two, can you score them reliably?
23:50
And then three, do the metrics distinguish between the objectively assessed
23:55
performance of experienced and novice operators?
23:58
And by and large, I mean, I have yet to see a study where they don't discriminate.
24:04
And once you have construct validity, then you've got your proficiency benchmark
24:09
because the proficiency benchmark is based on
24:11
the main score of the experienced practitioners.
24:14
How many errors did they make? Because every procedure has errors.
24:17
I mean, even the best surgeon in the world is going
24:20
to make some errors and possibly some critical errors.
24:24
So how many steps must they complete under how many errors and critical errors?
24:31
So there you've got your proficiency benchmark.
24:34
The metrics also point you towards what simulation you should use.
24:38
Unfortunately, that's not the way it is today.
24:41
People build a simulator and then they go look and then
24:43
say, okay, the simulator will fit this procedure or that task.
24:47
That's not the way it should be done at all.
24:49
You first of all need your metrics.
24:51
The construct validity will give you your benchmark, but it will also tell you
24:55
which parts of the procedure the trainee finds most challenging.
24:59
And okay, you want something that's going to simulate the entire procedure,
25:03
but you also want something that's going to allow the
25:05
trainee to hone their skills for that particular task or phase.
25:12
And so part task trainers are very useful here.
25:15
And if you develop your simulation from the metrics, the validated metrics, you're
25:22
on a share winner because everyone thinks that the simulator, it's got to be
25:26
pretty, it's got to be full physics, virtual reality.
25:29
That's not the case.
25:31
So once you've got your metrics, you've either built or decided what simulator that
25:36
you're going to use, you then train your faculty
25:40
on the metrics because they have to be able to,
25:44
one, they have to know the metrics, two, they've got to be able to give the trainee
25:48
explicit constructive formative feedback using the metrics on their performance.
25:53
And what we find is this accelerates learning enormously because the trainee also
26:00
has to know the metrics before they turn up to the course.
26:02
They preferably have to learn the metrics usually from online.
26:06
We know that given the trainees, the information and lectures really doesn't work.
26:12
There's a study published by Stefano Puliatti in
26:15
the British Journal of Urology fairly recently.
26:20
And what it shows is that trainees who got the metric based information in a
26:25
lecture took 100% longer to reach the proficiency benchmark than the proficiency
26:31
based progression train group who got the information online.
26:34
So the trainees, they would turn up to the course knowing the metrics.
26:38
They would have to pass at a proficiency benchmark on the online part of it, again,
26:43
established on the mean of the consultant's performance.
26:46
They would then progress in a skills lab.
26:48
They would be supervised and trained by a faculty who could be a fellow who knows
26:55
the metrics and is able to give them explicit constructive formative feedback.
26:59
And they continue training until they demonstrate the benchmark.
27:02
It doesn't matter whether they take two
27:03
hours, two days, two weeks, it doesn't matter.
27:07
They must demonstrate the proficiency
27:09
benchmarks before they progress to clinical practice.
27:12
And what we see when we do that is that trainees
27:16
perform on average about 60% better in the operating room.
27:21
The systematic review and meta analysis published by us in 2020 shows that trainees
27:27
who go through a proficiency based progression training program make 60% on average
27:32
60% fewer objectively assessed inter-operative errors than trainees trained on a
27:39
quality assured traditional approach to training.
27:42
And the key to that approach, Tony, and you've
27:45
mentioned it already is the idea of deliberate practice.
27:48
So it speaks to the primacy of feedback and the
27:51
training of the trainer and why that's so important to PBP.
27:54
It's certainly something that stood out to me when I've seen the in-person training
27:58
and the kind of feedback that the fellows
28:00
or the trainers are giving to the trainees.
28:03
Can you just briefly expound on how important that is to PBP and how deliberate
28:09
practice differs from repeated practice in your mind?
28:13
Yeah, I mean, but if you go back, you've got to go back about it.
28:16
One of the things I noticed when I was doing trainings in the late 1990s, early
28:21
2000s, is if you went from training station to training station, the feedback that
28:26
the trainee and the style of the trainer was
28:29
very different from consultant to consultant.
28:32
And it wasn't the skill level of the trainee.
28:35
And Erickson developed the idea of deliberate practice decades, a few decades ago.
28:41
And he talked about learning music or high skilled performance.
28:46
And he talked about deliberate practice.
28:49
And he wasn't as explicit as they would like them to be about deliberate practice.
28:54
But what he meant, what he actually said was that trainees learn faster if you give
29:00
them constructive, explicit, detailed performance on
29:05
what they were doing right and what they were doing wrong.
29:08
If you transpose that onto medicine, we need the metrics to be able to do that.
29:13
And you need a trainer that knows the metrics,
29:16
but you also need a trainee that knows the metrics.
29:21
When a music student is playing the wrong notes, they can hear that so they can.
29:26
The trainee who's learning to perform a
29:28
procedure, a surgical procedure, they may not know it.
29:32
Whereas if you ensure that they actually know the information in an online
29:36
curriculum before they turn up to the skills lab, they're going to recognise it.
29:41
And what it means is in the past, in a training lab, the trainees did the task over
29:46
and over again, and hopefully they got it. Frequently they didn't.
29:49
And on top of that, we never assessed the performance.
29:53
Today, what happens is we give the trainee explicit, constructive, formative
29:59
feedback in a timely fashion, proximate to when they actually did something wrong.
30:05
Or even you might even say to them, are you sure you want to do that?
30:07
For example, if they were playing staples at an angle rather than perpendicular to
30:14
the longitudinal axis of the colon, you
30:16
would say, are you sure you want to do that?
30:18
And they'd say, well, what do you mean? And you would
30:20
point out, is that an error or a critical error in the metrics?
30:24
And it's amazing how quickly they actually get up.
30:27
I mean, what you have to remember is that the trainees, these are bright
30:31
individuals that are committed and they will learn very quickly.
30:35
One of the issues, Frank Lewis, who was the chairman of the American Board of
30:39
Surgery when I worked in the US, we would go for a beer after meetings.
30:44
Tony says, I like this idea of proficiency based
30:46
progression. He says, but I have two concerns about it.
30:49
Number one, if you're setting the benchmark on the consultants, he said, are they
30:54
attending? Is this going to be far too high?
30:56
He said, we set the benchmark today on minimum competence.
31:00
And the other concern that he had was how long is it
31:03
going to take to get somebody to the proficiency benchmark?
31:07
And to be honest with you, those two questions remained unanswered until a few
31:12
years ago and have only recently been published.
31:16
Those questions were also very much part of the program approval process that we
31:20
had to undertake in University College Cork,
31:23
where there was always the question as to why your proficiency benchmark or passing
31:27
scores were so significantly higher than what would
31:29
have been the departmental norm or the university norm,
31:32
which would be arbitrarily set at 60 or 70 percent.
31:37
So you've spoken extensively and written extensively, Tony, about simulation, and
31:42
it gets a lot of attention in medical education,
31:45
the shiny new thing or sometimes a solution looking for a problem.
31:49
But what's missing when you put someone into a
31:52
simulator without the proficiency framework around it?
31:56
The simulators are more developed now as a
31:59
marketing tool rather than a genuine training
32:03
tool.
32:04
And the reason is that, well I think one of
32:08
the reasons is that they're more for marketing
32:10
is that the manufacturers tend to be the
32:13
purchasers, the biggest purchasers of virtual reality,
32:17
physics based virtual reality simulation, number one.
32:20
And so they want something that's a marketing tool.
32:24
Number two, it's expensive to develop the
32:26
metrics and it's even more expensive to build
32:29
it into the virtual reality simulation.
32:31
A lot of medicine doesn't realise that.
32:33
They think it's just like a video game and
32:35
you just tell it to measure this, measure
32:38
that.
32:39
And that's not the way it works.
32:41
If you're talking about highly
32:42
sophisticated physics based simulations that are derived
32:47
from mathematics and engineering solutions.
32:51
So this takes time.
32:53
But what we find time and time again, if you
32:56
don't have the metrics in a simulation, really
33:00
what you've got is an educational experience.
33:02
Now, if you don't have the metrics in a
33:05
simulation and the simulation is good enough, if you
33:08
have somebody standing beside the trainee who
33:11
knows the metrics, they can use that simulation
33:15
as a tool for the delivery of a metric based curriculum.
33:18
And that's to a large extent, that's what we're actually doing today.
33:22
We're supplementing what the simulation should be doing.
33:26
And I think part of the problem is that I said industry was the largest purchaser.
33:31
That shouldn't be the case.
33:32
It should be the health care service
33:36
organisations, which are the Department of Health around
33:38
the world that are purchasing simulations on
33:42
large scale for all of their teaching hospitals.
33:45
They or the training body should then say
33:49
what the curriculum is, what the metrics are,
33:51
what the benchmarks are.
33:52
And so that's not what we find.
33:55
I mean, frequently when I go to training
33:57
centres around Europe that we see simulations and
34:00
all, some of them are in the box.
34:03
They haven't even been taken out of the box.
34:04
And we're talking like a few years after they're purchased.
34:07
And some of them that were taken out of the
34:09
box and they were used for a photoshoot or a
34:13
television appearance, they're gathering dust.
34:16
You know, I mean, there are so many reasons for it.
34:20
I mean, it's a change.
34:21
You've got to learn how to use it.
34:23
You've got to learn the metrics.
34:24
You've got to learn how to use the metrics.
34:25
You've got to develop the metrics.
34:27
You've got to benchmark.
34:28
You've got to have validation.
34:30
And it's easier just to use something like
34:32
Likert type scales, which give generalised
34:36
qualitative feedback to the trainee.
34:39
And it's unfair on the trainee.
34:41
And I think that it slows down their skill development.
34:44
Why?
34:45
No, it slows down their skill development and the way that they go about learning
34:50
procedures.
34:51
That speaks to an earlier question in terms
34:54
of why the methodology hasn't been adopted
34:59
widely, is that it's pushing against that status quo, which reverts to that more
35:05
experiential type of learning.
35:07
Yeah, that's right.
35:08
I was in Australia, I was in Melbourne at the Society of Robotic Surgery.
35:12
I was given a keynote and somebody from the audience asked me, OK, Tony, if you've
35:16
all this evidence, why hasn't it been adopted by now?
35:20
And I hadn't thought about it until that moment and the words were out before I'd
35:25
actually realised I was thinking it.
35:27
It's a failure of leadership.
35:29
The reason that the Arthroscopic Association of North America, that European
35:36
Society, that the Orsi Academy, the reason that they're using this methodology is
35:42
leadership.
35:43
From the top down, you know, whether it was Rick Angel, Dr. Rick Angelo, Professor
35:47
Albert Obreda or Professor Alex Mattrie, they said, this is what we're doing.
35:52
I want the metrics for this.
35:54
I want the benchmarks.
35:55
Tony, tell me what simulations we need.
35:57
What do you need?
35:58
The reason it's happening in these organisations is it came from the top.
36:03
But would it be?
36:04
Would it be not the case, Tony, though?
36:05
That it can't it will never receive widespread adoption if it's always reliant
36:10
on leaders like Alberto Breda, Alex Mottrie and Rick Angelo.
36:14
It will need something more and rather than just better leaders, it's going to
36:18
need its either medicine will eventually have to adopt this because insurance
36:22
companies or others, other stakeholders in the in health care will oblige them to
36:28
do so.
36:30
Yeah, but I think you've got to start with leadership.
36:33
And I think what each one of those leaders
36:36
are doing, they have juniors coming along
36:38
behind them that basically are 100% convinced about this approach to training.
36:42
But you're quite right.
36:43
I mean, I was having a conversation with
36:46
one of my former fellows who is a manager for
36:48
Metronic and training.
36:50
And he said, "Tony, this is not about evidence."
36:52
He said, "There's sufficient evidence out
36:54
there that shows that by any stretch of the
36:57
imagination, whether you're talking about
36:58
training effectiveness, whether you're talking
37:00
about training outcomes, it doesn't really matter.
37:03
The evidence is there to basically show
37:05
proficiency-based progression as a superior approach to training.
37:09
And it's inevitable that medicine will have to be adopted."
37:13
And each argument that has been raised to
37:17
say, "Well, we can't really do proficiency-based
37:19
progression because we're not going to get everybody to proficiency.
37:23
It's going to be too hard to develop the metrics and so on."
37:26
These have been shown to be inaccurate.
37:27
The benchmarks are too high.
37:29
And one of the last and I think
37:31
significant, most significant pieces of evidence comes
37:33
from Stefano Puliatti.
37:35
And how his study was looking at, "Okay, can we train everybody to proficiency?"
37:40
But he also looked at, "What does it cost?"
37:43
Because one of the bigger arguments was
37:45
that the metrics were too expensive to develop.
37:48
And what Stefano's data shows, if you're only
37:50
going to train 12 trainees, the proficiency-based
37:55
progression approach to training is 100%
37:57
more expensive than the conventional approach
38:00
to training.
38:01
Okay, the conventional approach to training,
38:03
you only get 58% to proficiency, whereas the
38:06
PBP approach, you get 100%.
38:07
Well, let's ignore that.
38:10
What he also looked at, what happens if you
38:11
roll it out to say the size of the US or maybe
38:17
metronically training?
38:18
And if you're doing 500 trainees a year, what he found was that it costs about 1.7
38:26
million to train 500 trainees to proficiency using the PBP approach.
38:31
It costs 3.5 million to train the same number
38:35
of trainees using the conventional approach.
38:38
One of the things that I really didn't
38:40
expect from that, so I thought that we would have
38:43
to train like 100 or 150 before we would have cost equivalents between the metric
38:48
development, the cost of the metric development and the training.
38:51
That's not what he found.
38:52
The cost equivalence point was 25 trainees.
38:56
That sort of data is very hard to ignore.
39:00
And the medical device manufacturers are looking
39:02
at this quite closely because they have devices,
39:06
new devices coming out all the time, and
39:08
there's a significant proportion of their devices
39:10
fail in the marketplace.
39:12
And they get the device back to the
39:15
manufacturing plant and they dismantle it and they take
39:17
a look at what there's something wrong with the device.
39:19
There wasn't anything wrong with the device.
39:21
This is particularly the case for the CRT or the pacing devices.
39:26
And they've always suspected, but they couldn't
39:29
really say publicly, well, we think it's probably
39:31
the clinician is the reason the device failed.
39:34
The data is now there.
39:36
The assessment methodology is now there to
39:39
say conclusively, yes, this was a device issue
39:43
or no, this was a clinician issue.
39:45
And all of that means that it's uncomfortable
39:49
for medicine because we can now, rather than
39:52
just subjectively say that somebody's not particularly good at what they do, we now
39:56
know that if somebody's not particularly good
39:58
at what they do, it almost certainly is going
40:00
to impact on patient outcomes.
40:02
Because one of the big issues that I've run
40:06
into over the years is individuals in leadership
40:09
in medicine and surgery say, okay, Tony,
40:12
you've got this fabulous methodology, but it makes
40:15
no impact.
40:16
It makes no difference really on patient outcomes.
40:18
The data now is 100% clear.
40:21
The study published in 2013 in the New
40:24
England Journal of Medicine by John Berkmar, what
40:27
he looked at was attending level laparoscopic gastric bypass procedures.
40:32
That is, none of these procedures were done by residents.
40:36
All of them were highly experienced laparoscopic bariatric surgeons.
40:42
He very recorded their performance.
40:43
He had them blindly assessed as to who was who and what level they were at.
40:47
And they banded the surgeons into four levels,
40:50
those that were not particularly good to middle
40:53
quartiles and a group that were performing particularly well.
40:57
And what he found was he followed up all of
40:59
the patients those surgeons operated on over
41:02
a period of six years.
41:04
And what he looked at was, okay, what are the
41:06
outcomes for the patients of those surgeons?
41:10
Not surprisingly, what he found was the
41:14
surgeons who were assessed at the outset of the study
41:16
as performing best had significantly better patient outcomes.
41:21
To be quite frankly, that wasn't really the surprise.
41:25
The surprise was the magnitude of the difference.
41:28
The smallest difference between the
41:30
surgeons who were performing well and the surgeons
41:32
who were performing badly was 50 odd percent
41:36
patients who were operated on by the surgeon
41:39
who was pretty good had a 50% plus lower re-operation rate.
41:44
The highest difference was 81% in terms of mortality.
41:48
Mortality rate was low in the study, but the
41:51
difference was still 81% and it was statistically
41:54
significant.
41:55
To be frank with you, that was a shock to me.
41:58
I was working with George Shorten, who was
42:00
the Dean of Medicine and Cork at the time,
42:02
and we were having a discussion in the
42:04
boardroom one day and George said, "Yeah, yeah, yeah,
42:06
I know, Tony, proficiency-based progression
42:09
produces 60%, 70%, 80% better skills levels
42:14
and trainees in comparison to...
42:17
But a lot of that's going to wash out in clinical practice."
42:19
And to be quite frank with you, I believed him.
42:21
I felt that too.
42:23
The data now shows that that's not the case.
42:25
And we also have evidence that
42:28
proficiency-based progression, simulation training, the impacts
42:32
on patient clinical outcomes to the magnitude of about 60%.
42:37
In one of the studies where trainees were
42:39
trained on a proficiency-based progression
42:41
training program to perform an epidural for
42:44
women that were having a baby, we saw a 50-plus
42:48
percent difference in epidural failure rate
42:50
for trainees who didn't have the PBP training.
42:54
So we know that the training, it's not going to be 5 or 10% difference.
43:00
We know that the outcomes are going to be of the magnitude of about 60%.
43:04
These sorts of findings is going to make it
43:07
very difficult for medicine and surgery to
43:10
retain the line that they're currently retaining, which is non-adoption.
43:17
And it moves past a surgeon too, doesn't it?
43:19
I mean, PBP has now been implemented for
43:21
communication skills in Cork University Hospital
43:23
and they've recently been used to train
43:24
entire teams in a surgical context or in, say,
43:27
wound care or patient management post-operation as well.
43:30
I mean, I don't know that I was surprised at
43:33
the findings because basically the underpinnings
43:37
of proficiency-based progression are very simple.
43:39
You take somebody that's good at the task, not
43:41
world-class, but somebody that's good at the task
43:44
and they begin to work, you know, 8 till 6 every day.
43:47
They're good at what they do.
43:48
They're diligent.
43:49
They attend to detail.
43:50
They make mistakes.
43:51
They recognise and they fix them.
43:53
You take that individual and you work out what is it they do?
43:57
What is it they're good at?
43:58
You know, what sort of performance level?
44:00
And you work out, okay, what are the units?
44:03
What are the metrics?
44:04
What are the steps?
44:04
What are the errors?
44:05
What are the critical errors?
44:06
And you can apply that to anything.
44:09
We applied it with Dorothy Breen at Cork
44:12
University Hospital to a communication skill.
44:15
And lo and behold, guess what we found?
44:17
The trainees who were trained using a
44:21
proficiency-based progression training program performed about
44:25
60% better than trainees who had gone through the national training program.
44:30
In Ireland, the health service executive have
44:33
a national training program for the handover
44:35
of a deteriorated patient.
44:37
And before you go clinically, it doesn't matter
44:39
who you are, you've got to pass this training course.
44:43
So all the trainees who turned up to our
44:46
train--all of them had passed that already.
44:49
And when we assessed the trainees who turned
44:52
up, who had already passed the national competency
44:56
communication for the handover of a deteriorated
44:59
patient--when we assessed them, none--none of them
45:03
demonstrated the proficiency benchmark.
45:06
So you start with something relatively simple.
45:09
You start with somebody that's good at it, work out what
45:11
they do, what they don't do, benchmark on those individuals,
45:14
build a curriculum around those metrics.
45:17
And the outcome is pretty certain.
45:19
Each time, you're going to see at least 60% improvement in performance.
45:25
And it's moved out of the medical and surgical space as well.
45:28
And you've read that case study that we collaborated on with Reach Active as well,
45:33
in terms of the procedures and processes that they characterised as well,
45:37
and the results were similarly striking their word in that.
45:41
Yeah, my brother's own a fairly successful utilities company.
45:46
And he had read, I don't know why he had read my book, but he'd read my book.
45:49
And he said, I think you have a solution to a problem that we have.
45:52
And so we talked.
45:54
He's in the country, he was trained as an accountant.
45:56
And during the summers when he wasn't at university,
45:58
he used to work on the roads putting utilities into the ground.
46:01
And on one occasion, he put a jackhammer that he was working on straight through
46:07
an electric cable, a high voltage electric cable, and it blew him across the road.
46:11
And Seamus is quite a bright guy.
46:12
And as a company CEO, he knew that this was working in the utility sector.
46:19
This was a risk for his workers.
46:23
And in Ireland and in the UK, what we see is that there are a number of deaths
46:27
and very serious injuries from striking, usually an electric cable,
46:33
but could be a water or a gas cable.
46:35
And so he asked me to develop a proficiency based progression training program
46:39
for the identification of the utilities in the ground
46:43
and then being able to safely excavate those utilities.
46:47
So we went through the same program.
46:49
We took some of their workers who were experienced and good at the job.
46:54
We developed the metrics, we validated them, we set a benchmark.
46:58
They built a simulation yard.
47:01
They took a part of their space, they put utilities into the ground,
47:04
and they could switch those utilities on and off.
47:06
And the trainees had to go through an
47:08
online didactic course, which I think you built,
47:11
and they had to pass that before they got to the simulation space.
47:15
Now, remember, what we're talking about here is non-university people.
47:20
These are youngish fellows, labourers who
47:24
are basically digging out the roads and so on.
47:27
These are not university graduates.
47:29
They're not medical trainees.
47:31
And we found the exact same thing in the utility sector as we found in medicine.
47:37
The trainees who went through a proficiency based progression training program,
47:42
at the end of one year, Reach Active were saving over a million euro a year
47:48
in utility strikes that they avoided.
47:52
So it works across the board, but you always start with the same process.
47:57
You'd start with an individual or an individual who are actually good at the task.
48:01
And that's a very important message, and it's
48:04
one I think that anyone who's listening to this
48:07
and who runs the training program should pay heed to.
48:10
And speaking to that particular audience, and
48:12
if people are running their own training program,
48:14
everything from vocational training to something like teacher education,
48:17
and they want to move towards proficiency based progression, Tony,
48:21
what's the first question you think they should ask about their current curriculum?
48:25
Who's good at the task?
48:27
Because you need to identify those individuals.
48:29
And it can't be just because they're your
48:31
buddy, or it can't be just because you like them,
48:34
or it can't just be because they're senior.
48:37
You need to identify individuals who are good
48:39
at doing the task that you want to develop a
48:42
proficiency based progression training program.
48:45
Then you're going to have to learn how to do that.
48:47
You're going to have to either go back to you either going to have to talk to me,
48:50
or somebody like me, you're going to have to go back to my papers or the book,
48:54
and you're going to have to work out the methodology.
48:56
I mean, I mean, one of the things that we're seeing is, or I'm seeing,
49:00
is individuals who think they know how to use the methodology.
49:04
And then they ask you, well, what do you think of that?
49:07
And you're going, might have been better if you'd done this, that and the other.
49:11
I was at Adelphi, I was there as an observer
49:15
at a Adelphi meeting in Miami a few years back.
49:20
They developed the metrics for a colorectal procedure.
49:23
And they hadn't developed the metrics for a colorectal procedure.
49:28
They were still developing them during the Delphi.
49:30
And they asked me at the end of the meeting, why do you think that went to me?
49:33
And I says, what do you have?
49:35
And they had nothing.
49:37
I mean, they had added to the metrics, but
49:40
I mean, they had no idea whether the metrics
49:43
were actually going to stand up to more robust assessment.
49:47
Needless to say, the metrics, they weren't, they didn't go anywhere.
49:50
I've never seen them published.
49:51
I've never heard of them being used.
49:53
That meeting cost over a hundred thousand dollars.
49:57
I mean, I'm from Northern Ireland and we
49:59
would consider that a bit of a waste of money.
50:01
I mean, I like to see what's coming out at the end of it.
50:04
It's like if somebody's going to do a procedure on me or my family or a friend,
50:09
I want to know, okay, are they any good at what level of it?
50:12
And I don't need them to be expert, but I do need them to do the procedure.
50:16
And I do need them to be able to do the procedure safely.
50:20
Sure.
50:21
So you need to find in your particular
50:23
industry for the task that you're looking to train,
50:25
you need to find people who are objectively
50:27
good at doing that task and start with them.
50:30
Yes.
50:30
And have a reasonable attitude.
50:32
You don't want somebody that you need to do
50:33
therapy on whilst you're trying to develop the metrics.
50:36
I mean, it's individuals.
50:38
Someone who's collegiate and collaborative.
50:40
Yes, yes, yes, yes.
50:41
But I might put it more robustly.
50:44
After all, I am a psychologist.
50:46
I mean, it's just too much like hard work.
50:49
It's like pulling teeth.
50:50
I suppose it's an issue across medicine.
50:52
It's an issue across every industry.
50:53
There's a tension between time and training
50:56
and time where they need to perform their task
50:58
or to deliver their service, be that in medicine or be that in any industry.
51:03
So how do you make the case for PBP to a
51:07
department head or to an employer who says,
51:10
I don't have the time or the faculty for this?
51:14
That's the wrong attitude.
51:15
They have to if they're in the business of
51:17
training, they're financed and they're funded
51:20
for training and the trainees can't just be used for the delivery of service.
51:25
If they don't want to do training, get out of that business.
51:28
But if they're if they're in the business of doing training,
51:31
they need to go and look at Stefano Puglatti's data.
51:33
Basically what it says, number one, you can
51:36
train somebody faster and more cost effectively using PBP
51:40
than you can with a traditional approach.
51:44
But what they also must look at is the Berkmar data.
51:48
It's unequivocal.
51:49
Individuals who are not trained properly, who are not benchmarked properly,
51:53
they are going to have more bad outcomes, patient outcomes,
51:58
than individuals who are trained properly,
52:01
who are skilled and knowledgeable about the task
52:04
that they're supposed to do.
52:05
And I mean, the hospitals in the United States, I think that's starting to change.
52:11
I mean, somebody said to me when I worked there that a lot of the hospitals
52:15
and health care organisations made their profit on the complications
52:20
that the clinicians, you know, were, you know, that caused or involved with.
52:26
That's changing.
52:27
The Center for Medicare Services is now saying,
52:30
okay, what's what's the complication rate, the
52:32
average complication rate for that procedure?
52:35
Okay, 75%, 7%, whatever.
52:38
We'll allow that and a bit of wiggle room.
52:40
You see anything more than that?
52:42
The hospital has to eat the cost.
52:45
Until that sort of approach is introduced more en masse,
52:50
I'm not too sure there's an incentive there.
52:52
For example, the manufacturers, as I said earlier about a lot of the devices fail,
52:57
and it's nothing to do with the device.
52:59
If the device is ex-planted and another one is implanted,
53:03
the manufacturer gets paid for that device.
53:07
So there's no incentive there.
53:08
There has to be some sort of incentive.
53:10
But I would hope that the organisations that are involved with training
53:16
are of sufficient integrity and professionalism
53:19
that they take their commitment seriously.
53:23
Trainees, yes, they deliver service,
53:27
but the onus is on the organisation to train them.
53:31
And when we go around the world, what we see is that onus
53:36
or that training delivery varies very considerably.
53:39
I have a lot of Italian PhD fellows,
53:43
and some of the stories that I hear are quite horrendous
53:47
in terms of the training that those individuals get.
53:50
There was a guy who applied.
53:52
I was the head of the Laparoscopic Center in Emory in Atlanta.
53:59
I was the director of research and training.
54:02
And we had one Italian guy that applied.
54:06
The chairman of surgery sent over his application.
54:09
He said, "This guy's finished his residency,
54:11
and he's applied for residency here, and we think he's made a mistake."
54:16
He was obviously applying for one of the Laparoscopic Fellowship.
54:19
So anyway, we got the guy in and we interviewed him.
54:21
He said, "No, no, no."
54:22
He said, "I was applying for residency.
54:24
I've completed my residency in Italy."
54:26
He said, "But I can't operate."
54:29
You know, I mean, he couldn't do the residency.
54:33
That's why he applied for the residency rather than the fellowship.
54:37
That brings us right back to the very first question.
54:40
How do you know that someone is qualified to do the task to do an operation?
54:44
Yeah, but a trainee is a trainee, and I don't have an issue with that.
54:47
You mean it's only once a day that they're independently licensed.
54:50
Did they have an issue?
54:52
I think that there should be no tricks.
54:58
There should be no hidden agenda in it.
55:02
The idea is to have people good at what they do,
55:05
knowing what to do and how to do it.
55:07
We need some sort of a verification,
55:10
because at the minute somebody qualifies,
55:13
and the assessments that they get post-qualification are,
55:17
shall we say, less than robust.
55:19
They're mostly didactic.
55:21
There's maybe a portfolio of procedures that have already happened,
55:25
already been done.
55:27
What we need is some sort of an assessment of the intraoperative performance
55:32
of the clinician on a straightforward patient.
55:36
Because clinicians, I mean, cases can go...
55:39
I'm not saying that all clinicians are really not skilled or not up to...
55:43
I'm not saying that at all.
55:45
And every clinician can have a case that just goes particularly badly
55:50
through no fault of that surgeon.
55:52
But if you pick a straightforward case that went straightforward
55:57
and that case is assessed,
55:59
I don't see what the surgical community or the medical community
56:02
have to be concerned about.
56:06
We have no robust verification of the performance levels of our clinicians.
56:12
This is not just...
56:13
The procedure-based medicine individuals
56:15
are going to be relatively straightforward,
56:17
because we can see what they do and we can ask them about it.
56:21
The internal medicine clinicians, I think that's going to be more challenging.
56:26
But it's not an issue, behaviour is behaviour.
56:29
If they do something or say something, we can assess that.
56:32
Was that correct?
56:33
Was it inappropriate?
56:35
And we're not looking to catch anybody out.
56:38
Catholic University of Leuven in Belgium is introducing
56:41
a proficiency-based progression training program
56:43
into the robotics training program and across residency.
56:49
And I was given a lecture or a half-day session
56:54
and I was given one of the keynote lectures.
56:56
And a lady who was head of training, she stands up and she says,
57:00
"Hold me, hold the second Professor Geller.
57:03
Are you telling me that you tell the residents what to do online
57:09
and you assess them on that and they can
57:10
do it as many times as they want to pass it?
57:14
Are you saying you tell them what to do?
57:16
You make sure that they know it before they turn up for training?"
57:19
Then in training, you allow them to do the
57:22
task or you supervise them to do the task.
57:25
You assess them on the same metrics.
57:28
You give them feedback on what they did and they can do that as many times
57:31
but they still have to demonstrate the benchmark.
57:33
She said, "You're basically telling them what the assessment is
57:38
and then you're telling them what they did wrong so they can fix it."
57:43
And I said, "Yes."
57:44
She said, "That's not the way we do it."
57:45
She couldn't get up.
57:46
She couldn't get the tricks in us here.
57:49
The idea is to help the trainee to get to the performance level.
57:52
Anything that we have to do to get the simulations
57:55
and the curriculum are really just there to help them.
57:59
We've invested...
58:00
Well, the kids have invested so much of their
58:02
time and their effort getting into medicine.
58:04
The onus is on us to make sure that they can be a good clinician.
58:08
They can be a good doctor.
58:10
And there's no tricks in that.
58:13
If they know how to do a straightforward procedure
58:15
and they're interested enough and talented enough,
58:18
they'll learn to do more complex stuff.
58:20
They don't know how to do a straightforward procedure or performance safely.
58:24
They're a risk to patient.
58:27
So a proficiency-based profession, we're respecting the trainee's time,
58:31
we're respecting the trainer's time
58:32
and we're also respecting their future patients.
58:34
Yes, that's right.
58:35
And training them with the most robust methodology we can right now.
58:38
Yeah.
58:38
I mean, it's a scientific approach to training.
58:42
It's objective, it's transparent, it's fair.
58:45
I mean, the benchmark...
58:46
Repeated, but as key.
58:47
Yeah, yeah.
58:48
I mean, there's no tricks on this idea,
58:51
where an examiner gives you these random questions
58:55
of this curveball case that they saw 30 years.
58:58
No, that's not acceptable.
59:00
All that tells you is about the examiner.
59:02
I mean, I want to know, do they know the basics?
59:04
Can they do the basics?
59:06
And I certainly don't want them to be doing it on a competency level.
59:09
I mean, I thought I understood, you know, when we say competency,
59:12
we mean minimum competency.
59:14
And I would hear that now and again, and nobody says minimum competency.
59:20
And when I started reading about competency and I'm going minimum,
59:24
and this is what we're training to, and I'm going,
59:27
oh no, that's not acceptable.
59:29
Not procedure-based medicine, too high risk.
59:32
And proficiency, I mean, the question, Frank Lewis,
59:35
can we train everybody to proficiency?
59:37
Yes.
59:38
In every procedure, probably not.
59:39
You don't probably don't want them in every,
59:41
but we can train the absolute vast majority to proficiency.
59:45
And I think the outcome of that approach to training,
59:48
it's certainly better for the trainee, because I have a fellow,
59:53
he's now a consultant in independent practice.
59:56
He said that when he learned proficiency-based progression,
59:59
the metrics, I was training him to do the assessments initially.
1:00:02
He said it changed his approach to learning a new procedure
1:00:06
and it changed his approach to how he trains others,
1:00:10
because here you have a structure, objective, explicit, transparent,
1:00:15
and fair approach to training.
1:00:17
I mean, there's no curve ball, there's no hidden agenda here.
1:00:21
And even if a consultant doesn't like a trainee,
1:00:25
they can only assess what they objectively see, their performance.
1:00:30
They can't assess, I don't like this individual.
1:00:34
So Tony, to wrap up, so 30 years from now,
1:00:38
when someone is writing the history of how professional training finally caught up
1:00:41
with what we know about human performance,
1:00:43
what do you want the chapter to say about this period
1:00:45
and about the work that you've done?
1:00:48
Well, I hope it clearly shows that, number one, that it works.
1:00:52
I mean, I'm not too sure, I mean, I've been working at this now for like 30 years.
1:00:58
And somebody said to me once,
1:01:02
"Tony, why are you still doing this?"
1:01:05
And I said, "If I'm stubborn."
1:01:07
I mean, if something's right and you know that it's right
1:01:11
and you've got the evidence that it's right
1:01:12
and it's so hugely impactful, why would you not continue?
1:01:17
I'm not sure that'll be talked about.
1:01:19
And I'd be happy enough if, number one, the approach had been adopted.
1:01:24
And number two, it had permeated into
1:01:26
other aspects of training or performance even.
1:01:30
And medicine is going to become more robotic.
1:01:33
That's 100% certain.
1:01:34
Artificial intelligence, as long as the bubble doesn't burst too bad,
1:01:39
I think it will eventually become an integral part
1:01:44
of simulation-based training and proficiency-based progression training.
1:01:47
But artificial intelligence, everybody tells me about,
1:01:50
"Okay, we're going to be able to assess performance."
1:01:52
And we'll be able to...
1:01:54
No, they won't.
1:01:56
Because artificial intelligence, first of all,
1:01:59
needs to be assessed, able to assess the metric.
1:02:02
How far an instrument travels, how efficiently the instrument travels,
1:02:06
to be quite frank with you, it's a measure of process.
1:02:09
Like, if I'm going from here to Dublin, I can go...
1:02:13
If all you're assessing is, "Okay, Tony drove to Dublin from Belfast to Dublin."
1:02:18
It doesn't tell you how many car accidents I had,
1:02:21
times I broke the speed limit, the number of
1:02:25
pedestrians that I either hit or nearly hit.
1:02:28
Measures of process are insufficient for effective training.
1:02:33
And that's what AI does at the moment.
1:02:35
It needs to be able to assess the granular performance of the surgeon.
1:02:39
Okay, was the linear stapler, was it
1:02:42
perpendicular to the longitudinal axis of the colon
1:02:45
before it was closed?
1:02:46
How many staples did they have?
1:02:48
The suturing.
1:02:49
Where was the...
1:02:50
How was the suture driven through the tissue?
1:02:52
Did it tear tissue on the way through where the suture is equally...
1:02:56
It's not able to do that at the moment.
1:02:58
I absolutely think it will be, but not at the moment.
1:03:02
And we're going to need huge training data sets.
1:03:06
Properly performed surgeries to the benchmark to train this.
1:03:10
Yeah.
1:03:10
And I think that the AI will probably be
1:03:13
introduced first into endovascular spheres,
1:03:16
because the endovascular virtual reality simulators, the physics based virtual,
1:03:20
they're the best on the market.
1:03:22
They're the most expensive, but they're the best on the market.
1:03:25
The surgical simulators have a long, long way to come in.
1:03:28
But I mean, I'm a still huge fan of virtual reality simulation.
1:03:33
I think that's part of the future.
1:03:35
But medicine must see simulation as a tool.
1:03:38
Artificial intelligence needs to be seen as a tool.
1:03:41
The clinician at the end of the day has to make the judgment.
1:03:44
The metrics of optimal and suboptimal performance has to come from the clinician.
1:03:50
And I think what we're going to do is
1:03:52
proficiency based progression and the methodology,
1:03:55
the scientific methodology that underpins it and the validation evidence means that
1:04:00
the direction of travel is inevitable.
1:04:03
And I think it's going to take somebody in the leadership to say this is happening,
1:04:07
whether it's in government, whether it's in the scientific societies.
1:04:11
I don't know.
1:04:12
Well, we watched a space with interest, Tony,
1:04:17
and continue to impact it with more papers and publications.
1:04:20
Thanks for your time, sir.
1:04:21
Thanks, Patrick.