Latest / Show Me The Evidence / Experience Is Not Proficiency: Measuring what surgeons actually do | Show Me The Evidence E10

Transcript

Experience Is Not Proficiency: Measuring what surgeons actually do | Show Me The Evidence E10 · Show Me The Evidence

0:07
I'm joined today by Dr Rui Farinha. Rui is a senior
0:11
consultant of urology in Lisbon.
0:16
Rui's completed fellowships around the world in Spain,
0:19
Germany and Belgium in laparoscopic and robotic surgery.
0:23
We met in Orsi in 2019 and Rui, I was
0:29
impressed, number one, by his surgical skill.
0:32
Number two, about how quiet he is on most occasions, but
0:37
when he actually says something, it's
0:38
something meaningful, which is always impressive.
0:42
You're very welcome Rui. Can I start out? I mean, your
0:47
surgical career has spanned open robotic, laparoscopic and
0:55
robotic surgery is expensive.
0:58
So, I mean, as a surgeon who spans all of those approaches
1:02
to surgery, what additional performance capabilities do you
1:06
think the robot affords the surgeon?
1:11
Well Tony, I have to agree with you that
1:14
surgical robots are expensive, that's true.
1:17
And of course, we know that up until today, at least, they
1:21
are not able to replace the surgeon and
1:24
they are not able to operate independently.
1:28
But rather they act as an interface that translates the
1:33
surgeon's movements into a more controlled instrument, a
1:36
more controlled instrument movements inside the patient.
1:40
But their value and their main value is that they are able
1:43
to increase, augment the human
1:46
performance due to several reasons.
1:50
They are able to introduce a greater precision and movement
1:54
stability on the surgeon because robotic systems, they can
1:58
filter physiological tremor and they
2:01
can also scale surgeon's movements.
2:04
For example, a relatively large movement at the console can
2:08
be translated into a much smaller
2:10
movement at the tip of the instrument.
2:13
And this is particularly valuable during delicate
2:16
dissection, during delicate suturing,
2:19
during for example vascular anastomosis.
2:22
And this is very important in
2:24
terms of the precision of the surgery.
2:27
Another huge advantage is that the robotic platforms can
2:31
increase the dexterity of the surgeon.
2:34
For example, when we compare with laparoscopic instruments,
2:38
these instruments were very rigid and
2:41
they had a restricted range of movement.
2:44
And the change was that the robotic instruments can, for
2:48
example, articulate at the tip of the instrument and
2:51
therefore they effectively
2:53
provide a wrist inside the patient.
2:57
Their additional degrees of freedom allow the surgeon to
3:01
approach tissue from different angles to rotate the
3:04
instruments and to perform complex manoeuvres that would be
3:08
very difficult with the
3:09
conventional laparoscopic instruments.
3:11
So this increased dexterity is especially important in
3:16
narrow anatomical spaces like for example the pelvis, where
3:21
urologists for example perform radical prostatectomies.
3:25
Another big improvement that an advantage that robotic
3:29
platforms afford is their improved visual information.
3:35
These platforms, they provide a magnified high-definition
3:39
three-dimensional view of the operative field.
3:43
And this stereoscopic vision improves the depth perception
3:48
and helps the surgeon distinguish the different tissue
3:51
planes, vessels and the different anatomical structures.
3:54
So this is a very important advantage.
3:57
Another very important advantage is that the surgeon can
4:01
also control the camera directly, providing a very stable
4:06
view without relying entirely on the assistant.
4:11
Also there is another advantage, which is the improved
4:14
surgeon ergonomics and the endurance of the surgeon.
4:18
For example, the surgeon can operate from an adjustable
4:22
console rather than, like in the laparoscopic experience,
4:26
rather than standing for several hours while holding
4:30
instruments in physically demanding positions.
4:34
This reduces muscular strain and reduces the fatigue of the
4:39
surgeon and this therefore can help the surgeon to maintain
4:43
its concentration, to maintain its precision of movements
4:47
and the movement consistency during long procedures.
4:52
Another big advantage is that the robotic platforms are
4:55
also digital platforms that allow further performance in
5:00
systems because the surgeon's actions pass through a
5:04
computer-controlled interface.
5:06
And therefore, robotic systems can potentially record
5:10
instrument movements, can analyse technical performance and
5:14
potentially integrate imaging,
5:15
navigation or augmented reality information.
5:19
And also there is an increased potential for future systems
5:23
to allow the identification of anatomical structures, to
5:26
highlight tumours, for example, to establish a safe
5:29
operating boundary or provide
5:31
real-time performance feedback.
5:33
So these technical advantages do not automatically produce
5:37
better clinical outcomes. This
5:39
is very important to highlight.
5:41
Performance still depends on surgeon training, on procedure
5:46
selection, on team coordination and on the appropriate use
5:50
of the technology, of course.
5:53
Yeah, so in a word you would say,
5:55
yes, the technology is worth the money.
5:58
Yes, it is.
5:59
I totally agree with you.
6:02
And I also agree with what you implied that the technology
6:07
is a very powerful tool, but it relies on the surgeon being
6:12
able to use that tool safely and effectively.
6:16
I mean Dwight Meglan, I don't know if you heard the
6:18
interview with Dwight Meglan. Dwight
6:21
is not a believer in autonomous surgery.
6:26
He thinks that that's really, it's going to depend on the
6:29
market, it's going to depend on lots of other things.
6:31
And so really we're back to the surgeon who makes the
6:35
decision, who chooses the cases, but uses the robot as a
6:38
tool for the effective performance, the effective and safe
6:43
performance of a surgical procedure.
6:46
So I mean, one of the interesting things about you, well I
6:49
find very interesting about you Rui is, you have spanned, we
6:53
met in Orsi when you were already a senior consultant and
6:57
you've done fellowships in
6:59
laparoscopic as well as robotic surgery.
7:02
And so across your experience, has the approach to learning
7:09
to use these ways to perform surgery changed?
7:13
Well, yes, in fact, the approaches did change.
7:17
And I was very fortunate to be trained in the different
7:21
surgical approaches, initially open and
7:23
then laparoscopic and then robotic approaches.
7:27
And the approach in each one of these training, types of
7:32
training changed considerably, in fact, because from my
7:37
experience, these approaches for training moved from a
7:41
predominantly apprenticeship based model towards a more
7:44
structured simulation based
7:46
and performance driven training.
7:49
For example, my training in open surgery followed the
7:54
apprenticeship model in which I observed the experienced
7:57
surgeons first, then I assisted during procedures and
8:02
gradually performed increasingly complex parts of the
8:06
operation under supervision, of course.
8:09
This learning took place mainly in the operating theatre
8:14
and relied heavily on direct observation and imitation of
8:20
the surgeons that were teaching me, on repeated exposure to
8:24
clinical cases, on a verbal feedback from my senior
8:29
surgeons, on a gradual increase in the operative
8:32
responsibility and very importantly on
8:35
the supervises, very subjective judgement
8:38
when I was ready to progress.
8:41
And I felt that this model of training allowed me to
8:45
develop anatomical knowledge, allowed me to develop some
8:48
tissue handling, handling knowledge, of course, and develop
8:52
clinical knowledge in a real surgical
8:54
environment, which is a big advantage.
8:57
However, I also felt that this model had several
9:01
disadvantages because my progress depended strongly on case
9:06
availability, for example, which is a big limitation,
9:09
dependent also on the quality of the supervision.
9:13
I was lucky because I was supervised by different surgeons
9:16
with different experiences, but I noticed during my
9:20
training in open surgery that the quality of the
9:22
supervision was not consistent.
9:25
And also in this type of training, the training depends on
9:28
individual trainer's assessment,
9:30
which is also a big limitation.
9:33
And then I transitioned to laparoscopy. And when I started
9:37
my training in laparoscopy, I noticed that the competence
9:41
that I acquired in open surgery did not automatically
9:44
translate it into competence in minimally invasive surgery.
9:49
And I had to acquire a different set of skills. That's why
9:52
I did different fellowships in different places in
9:55
laparoscopy, because I felt this need
9:57
to acquire these different set of skills.
10:00
So, for example, I had to learn how to operate with a
10:05
two-dimensional visual information with a really
10:10
significant reduced in-depth perception.
10:13
I had to learn how to work with the long, rigid instruments
10:17
as the laparoscopic instruments are.
10:19
I had to learn how to manage, read the fulcrum effect, where
10:24
the end and the instrument tip
10:25
movements occurred in the opposite direction.
10:29
I also had to perform precise movements with laparoscopic
10:32
instruments that have a very limited degree of freedom.
10:36
And at the same time, I had to coordinate both hands while
10:40
interpreting an indirect image. So this was really complex.
10:44
And at that time, I met several surgeons that learned their
10:48
skills during life procedures using an
10:51
approach similar to the open surgical training.
10:55
And it was very curious that they told me that it became
10:58
clear for them that basic laparoscopic skills could and
11:02
should be learned outside the operating theatre.
11:05
And in fact, this was the time in which I observed a
11:09
greater use of, for example, box trainers, virtual reality
11:13
simulators, dry laboratory and animal laboratory training,
11:19
on the development of standardised technical exercises and
11:22
formal courses and the appearance of certification programmes
11:25
for surgeons to learn how to perform laparoscopic surgery.
11:29
So laparoscopic training represented an important shift
11:33
from learning almost exclusively through clinical exposure
11:37
to deliberate practice in a very
11:39
controlled environment as lab skills is.
11:43
Then I transitioned to robotic surgery. And when I started
11:48
my robotic training in 2019 with Professor Mottrie,
11:53
I was introduced to a more structured training that changed
11:57
my learning process completely.
11:59
I had to use a robotic platform, which I had to learn how
12:04
to use it because I had to learn how to use the console
12:08
controls and how to manage the
12:10
different robotic instruments
12:11
because the robotic surgeon controls all the surgical
12:14
instruments that are inside the patient.
12:16
I had to learn how to control the camera and how to clutch
12:19
the instruments. I had to learn how to
12:21
efficiently use multiple robotic arms.
12:25
I had to learn how to prevent internal and external
12:28
instrument collisions, how to communicate with the bedside
12:32
assistant, which is very important in robotic surgery.
12:36
I had also to train the different troubleshooting and
12:39
emergency undocking scenarios. And I also had to learn how
12:44
to maintain situational awareness while
12:48
physically separated from the patient.
12:52
I really noticed that robotic surgery training was much
12:55
more structured in the sense that it included online
12:59
modules, simulation, observation, bedside assistance,
13:03
supervised console practice and procedural prompting.
13:06
So it was much more structured than the
13:09
laparoscopic and the open surgery training.
13:11
And so the most important development in the approach to
13:15
learning was the gradual movement away from training
13:19
defined by time served or numbers of procedures completed.
13:25
In these transitions from open laparoscopic to robotic
13:28
surgery, instead of asking how many cases as the trainees
13:33
performed, the question started to be, can the trainee,
13:38
could I perform the procedures safely and consistently?
13:41
To a defined standard. I noticed a change in
13:44
the approach to learning because robotic surgical training
13:49
incorporated clearly defined learning objectives, included
13:53
procedure specific benchmarks, objective assessment tools.
13:58
It incorporated simulation before operating on patients. It
14:02
incorporated deliberate practice with proximate feedback
14:06
and the progression was differently assessed because it was
14:10
based on demonstrating proficiency.
14:14
However, during these transitions from open laparoscopic to
14:19
robotic surgery, there were aspects that haven't changed
14:23
because surgery cannot be learned
14:25
entirely through simulation, of course.
14:28
So I still had to learn skills like clinical judgement,
14:31
decision making under pressure, recognition of abnormal
14:34
anatomy. I still had to learn how
14:36
to manage bleeding and complications.
14:39
I had to learn how to adapt when the planned procedure did
14:41
not proceed as expected. And I had to learn how to
14:45
communicate and lead within the operating team, especially
14:50
in the robotic surgery because the robotic
14:52
surgeon is far away from the robotic team.
14:56
So, I mean, that's what was happening in Orsi. Can I
15:02
also assume that wasn't what was happening in many other
15:05
centres that were training robotic surgical skills?
15:09
Yeah, I think the experience still is very different from
15:14
centre to centre, from department to department. Of course,
15:18
I was lucky to be trained by Professor Mottrie in Aalst and
15:24
to have my lab training in Orsi because the training is
15:28
really systematic and structured and this, in fact, is not
15:33
happening in every department, unfortunately.
15:37
And so do you think the approach to surgical training
15:39
skills has changed then? It sounds like it has, but I'm
15:43
interested in your view because you're an experienced
15:47
surgeon, a very experienced surgeon, and in my view, I have
15:51
the data that you're a good surgeon.
15:54
So you've come from open to laparoscopic to robotic. So how
16:01
much has it changed? What has
16:03
changed? Do you think it has changed?
16:05
I think it changed a lot. Although, once again, it's not,
16:09
this change is not happening at the same time in every
16:12
department. There is still some differences in different
16:15
departments around the world.
16:18
Although I think the experience in the different
16:20
departments is expanding and is becoming more and more
16:24
consistent. But the surgical
16:26
training changed significantly.
16:29
Although the extent of this change varies between
16:31
institutions, of course, between specialties also, and also
16:35
between procedures because there is still a lot of
16:38
heterogeneity. And it is
16:40
changing in very different aspects.
16:43
For example, traditionally surgical skills were learned,
16:46
like we said previously, mainly through apprenticeship. And
16:51
even in robotic surgery, this is
16:53
still happening in several departments.
16:57
And in these departments, unfortunately, the surgeons, the
17:00
trainees, still depend on the exposure to a sufficient
17:03
number and variety of cases, on the availability and the
17:07
teaching ability of the individual trainers, of course.
17:11
On the trainees learning through observation, repetition,
17:14
and on these subjective assessments that the robotic
17:17
trainers give to their trainees.
17:21
But especially in robotic surgery, the fact is that the
17:24
apprenticeship model, although still remains important, of
17:27
course, but it's no longer
17:30
considered sufficient on its own.
17:35
I totally agree with you.
17:35
I also observed another important change, which was the
17:39
increased use of simulation in robotic surgery. This is a
17:42
fact. And it's spreading throughout all the trainings
17:46
around all the departments. Of course, we still need to
17:51
work a little bit on this increase on the use of
17:54
simulation. But this is an important change.
17:57
Yeah, because I think a lot of people talk about the demise
18:02
of the Halstedian training part time, but I'm not sure
18:06
I agree with them. I mean, I think what simulation does, it
18:09
better prepares the trainee for the operating room.
18:12
And so what the consultant gets to supervise is a
18:16
pre-trained novice. They know what to do. They've
18:19
demonstrated that they know what to do. They're
18:21
quantitatively defined
18:22
performance level. They get to the OR.
18:25
And they've got the technical skills to do the procedure to
18:29
a quantitatively defined performance level, but they just
18:31
have never done it on a patient.
18:33
But the Halstedian approach to apprenticeship and the
18:36
OR is still very much valid. It's just that it now is more
18:40
efficient or it should be more
18:42
efficient and more effective.
18:44
It has the potential to be
18:45
more efficient and effective, yes.
18:47
Yeah, Yeah
18:48
So Rui, you've recently completed a series of
18:52
studies on robot-assisted partial nephrectomy. Why did you
18:56
choose the RAPN procedure to investigate?
19:00
Well, I chose the robot-assisted partial nephrectomy
19:05
because this is an ideal procedure for studying surgical
19:08
performance for different reasons.
19:12
First, it is a very technically demanding operation because
19:16
a surgeon must remove the tumour while
19:19
preserving as much healthy kidney as possible.
19:22
He needs to control bleeding. He needs to reconstruct the
19:26
kidney. He needs to minimise warm ischaemia time. So all
19:30
these needs require precision, require efficiency, require
19:36
a sound intraoperative judgement.
19:39
The other reason was that this surgery contains a series of
19:44
clearly identifiable procedural phases and steps, including
19:49
the tumour exposure, the vascular control, the tumour
19:51
excision, the renorrhaphy, and
19:53
the restoration of blood flow.
19:54
So these steps can be perfectly observed. They can be
19:58
measured. They can be assessed separately, which makes the
20:02
procedure completely suitable for developing
20:05
procedure-specific performance metrics.
20:09
A third reason is that there is a meaningful variation in
20:15
how surgeons perform a RAPN. A surgeon may complete the
20:18
operation successfully, but the quality, the efficiency,
20:23
and the consistency of performance can differ considerably.
20:28
So case numbers alone do not reveal whether the procedure
20:32
was performed to a high standard.
20:36
And another reason was that the procedure has clear
20:42
consequences for the patient.
20:44
Technical performance can influence directly the blood
20:47
loss, the rate of complications, the surgical margins, the
20:51
ischaemia time, the preservation of renal function, and this
20:56
creates a direct link between how the surgeon performs and
21:01
the quality of the clinical outcome.
21:03
So for all these reasons, this procedure provided a strong
21:09
model for investigating how surgical performance can be
21:12
measured objectively and how training might move from
21:17
experience-based learning towards procedure-specific
21:21
proficiency-based progression.
21:23
What are the important lessons you've learned from that
21:26
series of studies? Because I mean, you've done these
21:28
studies over, and this is a high-risk procedure, not too
21:32
sure if all of our listeners would appreciate that, but
21:35
it's a high-risk procedure, and you've done a series of
21:37
studies with a very positive outcome.
21:40
So what are the lessons you've
21:42
learned from that series of studies?
21:46
Well, I have to tell you that I learned different lessons
21:51
and very interesting lessons. I would say that the first
21:55
lesson that I learned was that the surgical performance in
21:58
the robot-assisted partial
21:59
nephrectomy can be defined very objectively.
22:04
Because, of course, initially we think that this is a very
22:08
complex procedure, very difficult to define objectively.
22:12
But in fact, after my studies, I realised that it was able
22:18
to break down this procedure into
22:22
different observable components.
22:25
So we did a very detailed task analysis, and this was
22:29
completely possible. And we also did an international, we
22:33
obtained an international export
22:34
consensus on all these definitions.
22:37
And in the end, I defined the procedure in terms of all the
22:40
phases, all the steps, all the errors, and all the critical
22:43
errors, each one with a very clear operational definition.
22:48
So what we did, in fact, was we converted the expert
22:53
knowledge, which is often implicit and very difficult to
22:58
teach, and we converted it into something that could be
23:01
observed, could be measured, and could be reproduced.
23:06
I also learned another lesson, and this was that completing
23:10
the surgery was not the same as performing it proficiently.
23:16
So novice inexperienced surgeons often complete a similar
23:22
number of procedural steps. There is no big difference in
23:25
the number of steps that they complete.
23:27
But the important distinction was how they completed them,
23:32
particularly the number of errors they made.
23:37
In the intraoperative study that I did, I learned that
23:42
experienced surgeons made 69% fewer total errors than
23:46
novices. And this is a significant
23:48
difference between their performances.
23:50
And this demonstrated that the procedure specific metrics
23:54
could identify differences in the quality of the
23:57
performance that would have been missed if they were
24:01
assessed only by task completion or operating time.
24:05
This also taught me that the assessment must capture both
24:09
what the surgeon did, but most importantly, what the
24:13
surgeon should not have done.
24:15
Another very important lesson was that experience is
24:19
important, of course. We all agree with that, but
24:22
experience is not equivalent to proficiency.
24:26
I was amazed by the variation within surgeons who had
24:30
similar levels of experience. Some experienced surgeons
24:33
performed extremely well, the real experts, but others
24:38
produced error scores comparable to
24:40
those of the better performing novices,
24:43
which was a bit surprising for this type of surgery and for
24:46
the level of experts that we had. For example, in the
24:51
simulation study, we found that experienced surgeons in the
24:55
low error group made 77% fewer total errors than
25:01
experienced surgeons in the high error group.
25:04
And also, this high error experienced surgeon group
25:08
performed at approximately the same
25:10
level as the low error novice group.
25:14
So this was a very surprising finding. This meant that, and
25:18
this might surprise surgeons, of course, in the surgical
25:20
community, that case numbers and seniority are useful
25:26
indicators of exposure, of course, but they should not be
25:30
treated as a proof of competence or a proof of proficiency.
25:34
So we really need to measure actual performance. And also,
25:40
another very important lesson that I learned was that
25:44
procedure-specific metrics are much, much more informative
25:48
than the commonly used broad global ratings.
25:52
So in my studies, I developed RAPN-specific binary
25:56
metrics that achieved high agreement between independent
26:01
assessors, and they distinguished
26:04
different levels of performance.
26:06
And in contrast, in my studies, I also used the global
26:10
GEARS rating scale, and this scale obtained substantially
26:15
poorer inter-rater reliability.
26:18
And the assessor struggled to distinguish novice from
26:23
experienced performance using these type of scales. So this
26:26
taught me a lesson that once again, and this once again
26:29
might surprise the surgical community, that
26:32
global assessments may provide a general impression of
26:36
dexterity of the or efficiency of movement, but they do not
26:39
tell the training proficiency. They do not tell which step
26:44
was omitted. They do not point to which error occurred,
26:47
where the error occurred, what
26:49
must change during the next attempt.
26:53
So this made me realise that procedure-specific metrics
26:57
have greater educational value because they provide
27:01
explicit and actionable feedback. And also, another very
27:07
important lesson was that a realistic simulator is not
27:11
necessarily a validated training tool.
27:14
So for example, in one of my studies, I did a systematic
27:18
review, and this systematic review showed that many partial
27:22
nephrectomy models were considered realistic and useful, but
27:27
the evidence supporting their
27:29
educational effectiveness was very, very limited.
27:32
I found no randomised control trials, no predictive or
27:36
skill transfer validation. I found a considerable variation
27:41
on these models and in the tasks and in the assessment
27:45
methods that were used in the different studies.
27:47
And this is a very important distinction. A training model
27:52
does not need only to look and feel realistic. It must
27:57
allow the training to practise the correct procedure, to
28:00
receive reliable feedback, and it must
28:03
allow to demonstrate measurable improvement.
28:05
The simulator, the training model, should only be seen as a
28:09
vehicle. The validated curriculum and the performance
28:13
metrics are what make it an educational intervention.
28:19
Finally, another very important lesson that I learned was
28:23
that training should progress according to demonstrated
28:27
proficiency. This is very important. We should move away
28:30
from training based primarily on time, on seniority, on
28:34
completion of a predetermined number of cases.
28:37
Instead, the trainees should practise in a simulated
28:41
environment, should receive metric based feedback, and
28:45
should progress only after reaching a predefined
28:49
proficiency benchmark. This is the most important thing.
28:52
This means that the number of practice attempts may vary
28:55
between individuals, but the
28:58
required standard should not vary at all.
29:03
I have been reporting for about 20 odd years about the
29:07
heterogeneity in the performance of consultants and
29:11
attendings across different disciplines.
29:14
The first time that we observed some of the consultants
29:17
were performing extremely poorly on a fairly simple
29:21
laparoscopic task, to be
29:23
honest we did not believe the data.
29:25
Now, across disciplines in cardiology, radiology,
29:30
laparoscopic surgery, anaesthesia, across disciplines what
29:36
we have been demonstrating is that some consultants, when
29:40
you assess them, objectively
29:42
assess them, perform quite poorly.
29:44
Did those results in your study did they surprise you?
29:48
Did they shock you? Or anticipated it?
29:52
Well, I have to confess that they surprised me a bit
29:56
because we were dealing with very experienced and expert
30:00
surgeons. And at this level,
30:03
usually they should perform very good.
30:07
But in fact, there was a big difference between them.
30:13
Of course, throughout all these years, I was trained and I
30:20
worked with different surgeons. And of course, these
30:23
different surgeons were training me, they were teaching me.
30:27
And in fact, they were experts. But I was able to observe
30:35
that not all of them were that good.
30:41
But I learned from all of them, of course.
30:44
But now we have data. That's the thing.
30:50
And the data shows there is a difference.
30:54
Correct. I mean, having an opinion on something and
30:57
quantitatively demonstrating something that's entirely
31:00
different. Like I said, the first time that we observed it,
31:03
we were going, this can't be right.
31:05
There were some of the consultants performing 20 standard
31:07
deviations from the mean. And we went through everything,
31:10
the methodology, everything, because
31:13
we thought we'd done something wrong.
31:15
And we've seen it so many times now that we're going, OK,
31:19
these are genuine observations.
31:22
OK, one of the things that I've observed over the years
31:27
across disciplines and technologies is that the
31:31
investigators, the device manufacturers, the scientific
31:35
societies deploy simulations that
31:37
look like the procedure to be learned.
31:41
Is that the best approach?
31:43
No, I don't think so. It's not the best approach at all. So
31:47
there is in fact a kind of obsession with realism.
31:56
But the issue is that this obsession for realism can become
32:00
a huge distraction. That can become a big problem.
32:05
Because a simulator may look spectacular, may look, the
32:10
people that are seeing the simulator, they may feel that
32:14
this is completely convincing.
32:17
The simulator may receive excellent ratings from experts,
32:20
but yet the simulator can
32:22
still be a very poor training tool.
32:24
So these are two different things.
32:26
Being real and being a good training tool.
32:30
Resemblance is not evidence of educational effectiveness.
32:35
That's a very important thing.
32:38
We need to take into account that
32:40
realism and validity are not the same thing.
32:44
For example, I did a systematic review and the majority of
32:50
the partial effect of the training models that we studied,
32:54
they were rated as completely realistic and useful.
32:58
But the underlying evidence for this was very weak. The
33:03
studies were very heterogeneous.
33:06
There were no randomised control trials studying these
33:09
different models and none of them demonstrated predictive
33:13
validity or any transfer of skills into clinical practice.
33:17
So this is a big problem. So the uncomfortable question is
33:22
if a simulator looks like an operation,
33:26
but we cannot show that it produces safe or more proficient
33:30
surgeons, what exactly have we validated?
33:34
Often we have validated the
33:36
appearance of the model, not the training.
33:39
And then there is another thing that we need to think that
33:43
the objective is not to recreate the operating theatre, it
33:47
is not to recreate the surgical environment completely.
33:51
We do not want to produce the most impressive limitation of
33:56
reality. We want to create a
33:58
controlled environment, of course.
34:00
And in that controlled environment, the training should be
34:02
able to practise the correct procedural steps.
34:06
You should be able to make the errors without arming the
34:08
patient. You should receive a specific feedback, repeat the
34:12
task deliberately, not repetitively.
34:15
And you should be able to
34:16
demonstrate proficiency before progressing.
34:20
And therefore, a less realistic model that reliably trains
34:25
the critical components of a procedure may be far more
34:28
valuable than an expensive, highly realistic model with no
34:33
validated curriculum at all and
34:35
no objective assessment system.
34:37
So when developing a simulation, we should focus on the
34:41
fact that we should simulate what determines performance.
34:46
For example, in the case of the
34:48
surgery, in the case of RAPN a surgeon does not become
34:51
proficient because the synthetic kidney has the perfect
34:55
colour or the perfect texture.
34:57
It doesn't. The surgeon becomes proficient through correct
35:01
tumour identification, through correct controlled
35:04
dissection, appropriate instrument use, a safe excision,
35:08
correct haemostasis, an accurate
35:10
reconstruction of the kidney.
35:12
So simulation must reproduce the functional and the
35:16
decision critical elements of this procedure.
35:20
This simulation should not necessarily reproduce every
35:24
visual and every anatomical detail. This is not needed.
35:28
And for example, I developed a training model, and this
35:33
training model was deliberately developed, deliberately
35:37
designed as a vehicle to deliver a metric based curriculum.
35:42
This new model emulates, for example, eight of the eleven
35:46
phases of the human RAPN procedure.
35:49
And its primary purpose was not to reproduce every feature
35:53
of the human operation, but it was developed with the goal
35:58
to allow observable steps, errors, and critical errors to
36:02
be practised and to be scored reliably.
36:05
Also, we need to think that the developers of this type of
36:09
simulation models need to take into account that high
36:12
fidelity can carry a high educational cost because greater
36:18
realism usually brings greater costs, greater complexity,
36:23
and probably a much restricted access.
36:29
And then there is also an important thing that these
36:32
simulation must reduce the number of repetitions that
36:35
trainees can complete. If the model is very expensive, then
36:40
the number of repetitions that the model will allow will
36:44
not be that much because if the model is very expensive,
36:49
the trainees, maybe they can only perform
36:52
the task once, maybe twice, and that's it.
36:55
And therefore, this is not a good model. So these matters
36:59
because proficiency is not achieved
37:02
by one impressive simulation session.
37:06
Proficiency develops through deliberate practice, through
37:08
proximate feedback, and this also develops from the
37:14
correction of specific errors.
37:16
So this is why when I developed the new partial nephrectomy
37:22
training model, I took these aspects into consideration
37:25
because the model I developed uses readily available animal
37:30
tissue costing less than $10.
37:33
Yet, it allows trainees to practise eight phases of the
37:36
procedure and receive procedure specific feedback. So here
37:41
the relevant question is not how realistic the model is, it
37:46
is how much high quality
37:49
measurable practice does the model permit.
37:52
So where would you advise the investigators, the device
37:57
manufacturers, the professionals?
37:59
Where would you suggest that they start in terms of
38:03
developing a model? What must they do?
38:06
Well, they should start by defining the
38:10
procedure that they want to characterise.
38:14
That's the first step before
38:18
they start to develop anything.
38:20
Yeah, I totally agree with you. I was at the European Heart
38:24
Rhythm Association meeting in
38:26
Paris a few months ago and I walked around all the
38:31
manufacturers and looked at the simulations.
38:33
They were saying they had physics-based virtual reality
38:37
simulations. None of them had metrics.
38:40
The procedures looked great, except for one of them had
38:44
metrics. And I totally agree with you
38:48
that you really need to start with the metrics. You really
38:51
need to start with the construct validity
38:53
and say, "Okay, where are the trainees having the most
38:56
difficulty in learning and performing that
38:59
part of the procedure?" And that's where you really need to
39:02
concentrate on. So you've completed a
39:05
series of studies on the robot-assisted partial nephrectomy
39:09
and you've gone all the way to a
39:11
prospective randomised and controlled trial on your
39:15
simulation model. What did you find from that study?
39:20
Did it work? Fortunately, I was able to start the
39:26
development of this project from the real start,
39:30
from developing the metrics, the
39:32
procedural metrics for the partial nephrectomy.
39:36
So I learned several lessons in all these
39:40
studies. And of course, the first lesson,
39:43
and I think I already said that this, the first lesson was
39:48
that I learned that it was possible
39:53
to objectively define RAPN performance. So my first study
39:59
demonstrated that a complex procedure,
40:01
such as robot-assisted partial nephrectomy, can be
40:05
completely deconstructed into observable
40:08
phases, observable steps, errors, and critical errors. And
40:13
then it was very interesting to see
40:16
that in the beginning, we gathered a group of three
40:21
international experts to help me to define
40:24
these metrics. But then we submitted these metrics to an
40:30
enlarged panel of international
40:32
experts in the robot-assisted partial nephrectomy. And it
40:36
was very interesting to see that we were
40:39
able to reach 100% consensus on these procedure-specific
40:44
metrics. And this is very important,
40:47
because this provides a common, explicit definition of what
40:51
is an optimal and a suboptimal
40:54
RAPN performance. And then in the second study that I
40:58
did, I learned another important lesson.
41:01
I learned that the metrics that we developed and that were
41:05
face and content validated by
41:07
all these international experts, that these metrics could
41:11
reliably distinguish different
41:15
between the different surgeons. For example, when we
41:18
applied the metrics to unedited recordings of
41:21
real surgeries, independent assessors achieved very high
41:27
agreement with a mean interactive
41:29
reliability of 0.9. And what we observed was the
41:36
experienced surgeons made 69 fewer total errors
41:41
which demonstrates that the metrics could distinguish
41:44
levels of expertise, which is very
41:47
important as an assessment tool. Even more important was
41:51
that the greatest distinction
41:53
was not simply between whether the surgeon
41:56
completed the operative steps. It was how
41:59
the surgeon performed them and how many errors were made.
42:04
Also, another very important finding
42:06
was that, and I also said that previously, that experience
42:10
alone does not guarantee proficiency.
42:12
This was a very interesting finding, because there was,
42:17
like we said previously, there was
42:18
a considerable variation within surgeons,
42:20
classified at the same level of experience.
42:23
Some experienced surgeons performed extremely well, while
42:27
others performed at a level comparable to
42:29
better performing novices. And this was a bit
42:32
disconcerting. Another important finding was that
42:36
these procedure-specific metrics outperformed completely
42:39
the global rating scales. And another
42:43
important finding was that the existing RAPN simulators
42:46
lacked strong evidence of training
42:47
effectiveness. So once again, in this systematic review
42:51
that I did, we examined all the different
42:54
training models that existed for partial nephrectomy, the
42:56
animal, the three-dimensional printed
42:59
models, the virtual reality models. And although these
43:03
models were generally rated as realistic
43:06
and useful, there were no randomised control trials. There
43:10
was no evidence of predictive validity,
43:12
no reliable transfer of required skills into clinical
43:14
performance. So the conclusion was that
43:18
realism and perceived usefulness do not by themselves
43:22
establish that the simulator produces
43:25
more and more proficient surgeons. And then there was also
43:30
an important finding, which was
43:32
that the developed a validated model for metric-based
43:37
RAPN training, because one important
43:39
finding was that I developed a validated model for
43:41
metric-based RAPN training by, and this is
43:46
very important, I adapted the human RAPN metrics to the
43:50
new procedural training model.
43:52
And the model metrics distinguished, once
43:56
again, novice from experience performance.
44:00
And once again, they also revealed substantial variation
44:04
within the experienced group,
44:05
reinforcing that experience does not always equate to
44:09
proficiency. So most importantly,
44:13
was that after all these studies, I was able to form a
44:17
connected integrated programme of research,
44:20
moving from defining performance to
44:23
testing how it should be measured. And finally,
44:27
to translating those
44:28
measurements into simulation-based training.
44:31
So are your studies simply a series of interesting
44:36
observations, or do they have
44:38
a real world application? Well, they have a real world
44:43
application, of course. And this
44:46
application is very, very real. And nowadays it's being
44:49
used in Orsi to train the new
44:53
generation of surgeons. So these studies from the start,
44:57
they were not designed merely to
44:59
describe how the robot-assisted partial nephrectomy performance. They
45:03
were designed to create the components
45:05
required for a deployable proficiency-based training
45:09
programme. This was the main objective
45:11
from the start. Yeah, I think a lot of the
45:15
studies that are published in the journals,
45:18
my concern about them is that they're interesting studies,
45:21
but they don't seem to go anywhere.
45:23
Whereas what you've done, okay, it's being deployed in
45:26
Orsi, but more importantly,
45:28
I think it's being deployed with a new fellowship with the
45:33
European Robotic Urological Society,
45:35
where they're now implementing what you've
45:38
done in a RAPN specific training programme.
45:45
And I think that's hugely important. The studies, yes,
45:48
they're interesting. And I think it's
45:50
important because what you have done is,
45:53
rather than just develop a training programme,
45:56
it's a training programme that's been developed and
46:00
scientifically underpinned with quantifiable
46:03
performance measurements that are
46:05
published in peer-reviewed journals,
46:08
and it demonstrates that it's a much more effective
46:11
approach to training. So, okay,
46:15
how has your training to simulation-based training changed
46:19
as a result of your experience in Orsi
46:22
Well, my understanding has changed quite fundamentally
46:27
and quite substantially, because
46:30
a few years ago, I tended to think of simulation mainly as
46:34
a way of reproducing the operating
46:37
theater without exposing a patient to risk. Today, I see it
46:42
completely differently. I see
46:44
the purpose of simulation not simply to
46:48
create a lookalike surgical environment,
46:51
but to make performance visible, measurable, and
46:56
improvable. So, I move from thinking
47:00
about realism to thinking about learning.
47:04
So, initially, the natural question was,
47:09
"How closely does the simulator resemble the real
47:11
procedure?" And, of course, that remains relevant.
47:14
I'm not saying that it's not relevant, particularly for
47:17
tissue handling, for dissection, and suturing.
47:19
But, however, I became much more critical of the assumption
47:24
that high fidelity automatically means
47:28
high educational value. It doesn't
47:29
mean at all. So, the better question is,
47:32
"What precisely can the trainee learn, practise, and
47:35
demonstrate on a model?" Also,
47:40
I realised that a simulator is only a vehicle. A simulator
47:44
on its own is not a training programme.
47:48
So, without a defined curriculum, clear objectives,
47:51
structured feedback, and a performance benchmark,
47:54
the trainee may simply repeat and repeat and repeat the
47:57
procedure and potentially repeat
47:59
the same errors. So, the real educational value of the
48:04
intervention is a combination of a
48:07
deconstructed procedure, clearly defined steps, errors, and
48:11
critical errors, deliberate practice,
48:13
reliable assessment, and the proficiency
48:15
standard that must be achieved. So, this is
48:18
non-negotiable. So, my robot-assisted partial nephrectomy work until
48:23
now, showed that expert knowledge
48:26
could be converted into observable and accessible
48:30
performance metrics, of course,
48:34
supported by a huge and enlarged group
48:37
of international expert consensus. So,
48:39
that changed my view of simulation from a piece of
48:44
equipment into a delivery system for a
48:47
scientifically designed curriculum. Also, my view moved
48:51
from repetition to deliberate practice.
48:56
Traditional simulation courses that I did often provided a
49:01
fixed amount of practice. So, I attended
49:05
for a day or two or three. I completed all the exercises,
49:09
and then in the end, I received
49:10
immediately a certificate. But I realised
49:13
that completion is not the same as learning.
49:16
So, proficiency-based progression is completely different.
49:20
It allows different trainees to require
49:23
different amounts of practice while holding
49:26
everyone to the same performance standard.
49:28
So, in proficiency-based progression, trainees receive
49:31
explicit feedback on the errors.
49:33
On the errors they made, they use the next attempt to
49:38
correct those specific errors,
49:40
those specific deficits. They progress only after
49:44
demonstrating that required benchmark.
49:47
This is a major conceptual change
49:50
because the number of repetitions can vary,
49:54
but the required standard should
49:56
not vary. It doesn't vary at all.
49:59
Yeah, I totally agree with you.
50:02
That's a huge mind shift of a change.
50:05
Because a lot of what I've seen in the past is
50:08
simulation-based training is used as an
50:10
educational experience. What you're describing is not. It's
50:14
actually a tool for the delivery of a
50:16
metric-based curriculum. The metrics are derived from the
50:19
consultants who genuinely know how to
50:21
do the procedure. It's benchmarked on the consultants who
50:24
genuinely can do the procedure
50:28
and the trainees are given deliberate
50:30
practice feedback based on the metrics.
50:35
And that's an entirely
50:36
different approach to using simulation.
50:40
Are you a fan of
50:43
proficiency-based progression training then?
50:45
Yes, I'm a percentage fan. I really like it.
50:50
The trainees that you train, do they see it as positive? Do
50:56
they see it as more difficult?
50:57
Do they see it as challenging?
50:59
They see it as positive and
51:02
completely different from what they are used to
51:05
experience in terms of training. Because when they have
51:08
contact with this type of approach
51:10
in terms of training, they really find that they can
51:13
progress faster. And I think they
51:17
really appreciate one important thing that PBP brings to
51:20
training, which is the approximate
51:22
feedback that they have and the very specific and detailed
51:26
feedback. This is something that
51:27
they appreciate. And in this type of proximate feedback,
51:35
I also think that the human contact is
51:37
important. Nowadays, we are talking a lot about having AI
51:41
models that can train people and can
51:43
give feedback. But when you have two humans interacting and
51:49
one is training the other and
51:50
is giving a very proximate human and kind and detailed
51:53
feedback, this transforms the training
51:57
and makes the training more human and friendly. And I think
52:01
with this type of approach, trainees
52:03
progress faster. And they really see that trainers are open
52:07
to their doubts and to their needs. And
52:11
this brings a new experience to training.
52:13
I totally agree with you. I mean, COVID,
52:17
I mean, one of the lessons that I learned from it is human
52:22
beings not being in the same room,
52:24
not being proximate teacher, okay, you can get business
52:27
done, but not with the same efficiency
52:29
and effectiveness as you can when you're actually quite
52:32
close to somebody. So if you were to advise
52:36
somebody who wanted to develop a simulation curriculum for
52:40
a robotic surgical procedure,
52:43
what advice would you give them? My
52:46
first advice would be very direct.
52:52
I would advise them not to start by buying or building a
52:57
simulator. So and this is usually
53:00
the most frequent approach from somebody that wants to
53:04
start to develop a training programme,
53:08
but this is not the best approach. So I would advise them
53:10
to start by defining the performance
53:14
that they want the surgeon to achieve and to establish what
53:18
type of procedure they want to
53:19
train. And because one of the most common mistakes in
53:24
developing a simulation curriculum is
53:26
to acquire and or develop an impressive simulation model
53:31
and then ask what this simulation model
53:35
might be useful for. And this is
53:37
completely wrong. When we use the PBP approach,
53:43
a proficiency based progression curriculum should be
53:47
developed in the opposite direction.
53:50
And this is what I would advise the developers to do. I
53:54
would advise them to define what is
53:57
a safe performance first, and then to select and construct
54:01
the simulation that can train
54:03
and can measure it. You would advise them to start with a
54:07
metrics. With the metrics. So I would advise
54:10
them first to define the training problem precisely. So
54:16
they should first specify the procedure,
54:21
the operative approach, the target learners, of course.
54:26
They should define their required
54:28
level skills. The clinical case is covered by the
54:31
curriculum, of course, and they should define
54:34
the level of independence expected at completion. They
54:37
should initially focus on the standard
54:40
straightforward case because trying to simulate every
54:44
anatomical variation, every complication
54:46
from the beginning makes the curriculum very difficult to
54:50
standardise, very difficult to
54:52
validate. And also, a trainee that cannot perform a
54:58
reference straightforward procedure
55:00
safely should not progress to more complex cases. And
55:03
sometimes when people start to develop a
55:06
training problem, they try to cover all
55:08
the scenarios and this is not correct.
55:13
Yeah, no, I totally agree with you. I mean, that is a
55:16
lesson I learned a long, long time ago,
55:19
is that if somebody can't do a straightforward procedure,
55:21
why would you teach them something
55:22
more complex? And it's easier to get consensus around
55:26
what's a straightforward procedure and so
55:28
on. So your advice to somebody would be start with the
55:32
metrics. And would you get anybody to help you
55:37
develop the metrics or do you have any specific
55:39
requirements? Well, they should start with the
55:42
human metrics, with the human procedure. And of course, in
55:49
the beginning, we should find at least
55:53
three good experts, but they should be experts that are
55:59
able to discuss the ideas, discuss the
56:05
metrics, they should be a bit open-minded, and they should
56:09
be able to accept different opinions.
56:13
They should be so and sometimes this is very difficult to
56:16
find these people, especially
56:17
between the surgical community. But there
56:21
are still a few surgeons that are like this.
56:23
And so you don't really need experts that don't accept the
56:28
ideas and the way of the other surgeons
56:32
do their things, because you really need a group that is
56:36
able to connect and to understand
56:39
the other surgeons' positions and to accept that their
56:42
opinion might not be the correct one
56:45
and still go with that opinion. And do we need experts or
56:51
do we need very experienced surgeons
56:53
that are good at the procedure and do it frequently? I
56:57
would say that we need very experienced surgeons,
56:59
not being the next person on the mental. Yeah, I mean,
57:04
that's a different conversation.
57:08
Ideally, people that are able to do surgery and that are
57:13
used and able to train people.
57:15
That's the best combination. Yeah, I mean, I've
57:19
characterised quite a number of procedures in my
57:21
career in the different disciplines. And one of the things
57:24
I've encountered is so-called experts.
57:27
When I listen to them, I'm going, "Okay, how expert are
57:30
they?" And when I dig a bit deeper,
57:32
what I find is they haven't performed the procedure in
57:36
years or they weren't particularly
57:38
good at it. My expert for me, or the expert for me, is a
57:42
very experienced clinician who does the
57:45
procedure, who's good at it, and is good at training,
57:50
because they have the verbiage to be able to
57:53
facilitate the definition of the intraoperative
57:57
performance. So, Rui, you're very experienced in
58:02
robotic surgery now. I knew you proctor and teach and train
58:06
in the Skills Lab and all over the world
58:10
in the Skills Lab and in the operating room. Do you think
58:15
that proficiency-based progression plays
58:17
a role in training in the Skills Lab for the different
58:20
robots and in the operating room
58:23
for the different robots? So, yes, it plays different
58:27
roles. So, in fact, I believe that
58:31
proficiency-based progression has a very important role in
58:35
both settings, although it serves a
58:39
slightly different purpose in each one of the settings.
58:41
First, when we teach a surgeon to use
58:44
the robot, the manufacturer's instructions for use are
58:49
essential. But these instructions for use,
58:54
they mainly describe how the technology should be operated
58:57
safely. They do not necessarily
59:00
demonstrate that the surgeon can operate it reliably under
59:04
realistic conditions. So, and here,
59:08
in this aspect, PBP makes a difference
59:10
because it converts the instructions for use
59:14
into observable and measurable performance. For example,
59:18
docking the robot, positioning the patient,
59:20
managing the different instruments, responding to alarms,
59:24
troubleshooting faults, performing an
59:26
emergency and docking procedure. So, PBP, rather than
59:30
completing a fixed number of training hours,
59:33
makes the surgeon progress only after demonstrating the
59:36
required standard. And this is very important
59:38
because familiarity with the controls, with the different
59:42
buttons, should not be confused with
59:45
proficiency in using those controls, using those buttons.
59:48
So, a surgeon may understand what each
59:52
button does, but it might still be unable to use the system
59:56
efficiently and safely when workload
59:59
or complexity increases. Also, PBP has an equally important
1:00:05
and perhaps even more important role
1:00:07
inside the operating room because the skills laboratory can
1:00:10
establish technical readiness,
1:00:13
but the operating room introduces factors that simulation
1:00:17
cannot reproduce fully. For example,
1:00:19
it introduces patient variability, tissue response, it
1:00:23
introduces communication with
1:00:24
bedside assistants in real time, it introduces coordination
1:00:29
with anaesthesiologist, it introduces
1:00:32
the factor of time pressure and unexpected events. So, PBP
1:00:38
provides a bridge between laboratory
1:00:41
training and independent clinical practice. Progression in
1:00:45
the operating room can be structured
1:00:46
through defined stages, beginning with the observation and
1:00:49
selected procedural components,
1:00:52
followed by supervised procedures and ultimately
1:00:54
independent operating. And the advancement should
1:00:58
depend on procedure specific performance metrics, not
1:01:01
simply on the number of cases completed
1:01:04
or the subjective impression that the surgeon appears
1:01:06
ready. So, the central principle is that
1:01:11
the instructions for use teach the surgeon how the robot
1:01:14
functions. PBP determines whether the
1:01:17
surgeon can use it proficiently. And in the operating room,
1:01:21
PBP determines that proficiency
1:01:23
can be translated into a safe and effective procedural
1:01:27
performance. So, PBP does not replace
1:01:30
expert supervision at all, it does not replace clinical
1:01:33
judgement, neither responsibility for
1:01:35
patient outcomes, but it makes the training pathway more
1:01:39
objective, transparent and defensible,
1:01:42
and reduces the reliance on case numbers
1:01:45
or time served as proxies of proficiency.
1:01:48
Yeah, so PBP you would argue is a, again, it's a tool for
1:01:54
the delivery of a curriculum,
1:01:56
whether it's in the skills lab or it's in the OR. Yes. And
1:02:01
you think it has a useful function
1:02:03
across platforms? I mean, it's not, you know, platform
1:02:07
specific? It's not at all platform
1:02:09
specific. It can be used and developed for different
1:02:13
platforms. You can, like we did in
1:02:15
the partial nephrectomy studies that we did, we developed a
1:02:19
metric for the procedure, not
1:02:21
for the procedure done by a specific platform. Of course,
1:02:25
we can use PBP to develop metrics
1:02:27
for a specific platform, to develop a specific training
1:02:33
programme to use that platform. But
1:02:37
you can develop a programme for the procedure and that can be
1:02:42
performed by different platforms.
1:02:45
Yeah, yeah. So, yeah, I mean, listen Rui, that's
1:02:49
fabulous. Thank you very much. It's always
1:02:52
a pleasure to talk to you and always very
1:02:54
informative. You are very welcome. Thank you
1:02:56
so much, Tony, for the opportunity.