Latest / Show Me The Evidence / From Impression to Evidence: Proficiency-Based Progression for Cardiac Device Implanters | E11 Dr Jorio Mascheroni

Transcript

From Impression to Evidence: Proficiency-Based Progression for Cardiac Device Implanters | E11 Dr Jorio Mascheroni · Show Me The Evidence

0:07
Today I'm joined by Dr. Jorio Mascheroni
0:10
I've known Jorio for about 20 years.
0:12
Jorio is an engineer who subsequently did
0:17
a Masters in Technology Enhanced Learning
0:19
and a PhD in how optimally to implant a CRT device.
0:26
Jorio has worked in industry certainly since
0:29
I have known him and Jorio is very welcome.
0:33
Hello Tony.
0:34
Jorio, can I start out with, okay you've worked
0:38
in the medical device sector for quite some time
0:40
certainly alongside known you. So what
0:43
got you interested in educational training?
0:47
Yes, thanks Tony. I got interested in
0:51
education from the early days of my career
0:57
medical industry. I had the privilege to to
1:01
witness the introduction of one of the most
1:05
successful therapies of the last 30 years in
1:08
cardiac pacing which is cardiac resynchronisation
1:11
therapy. It's a therapy for cardiac rhythm
1:15
abnormalities. Maybe we can talk about that
1:18
more specifically later and because it was a
1:21
totally new approach and a new type of procedure
1:26
there were a lot of challenges from the
1:28
pioneers that started with that therapy and I had the
1:33
privilege to work with some of them go through
1:36
the challenges and I realised that the ability to
1:40
perform certain steps properly and avoid
1:45
mistakes was a determinant for success. So at the time I
1:52
also had the possibility to work on
1:55
developing some training programs for these new
2:00
implanters new to therapy at least and that's
2:05
how I started getting involved with education.
2:09
The second aspect was in the few years at the
2:15
beginning of this therapy we had the chance to
2:21
start working with the beginning of simulation
2:28
with virtual reality and we had the chance to
2:32
get involved in the development of simulator
2:35
for teaching cardiac resynchronization therapy
2:39
procedure and so that was also interesting for
2:43
me because it opened more possibilities to move
2:47
training away from the patient and to
2:49
experiment how to do things outside the operating room.
2:55
So all those years ago when you started I mean
2:58
you talk about simulation you talk about education
3:01
so how was education and training approached
3:04
back then? Well I think there was no single way
3:10
there were multiple degrees of quality and
3:13
quantity as well and in general if I compare
3:18
to what happens and I see today what I do
3:23
training was certainly more knowledge based at that time
3:27
and the lack of simulation tools of course made
3:34
it more theoretical and at the time the reality
3:38
was and still today that most of those
3:42
physicians learning a new procedure or a new skill
3:47
were and are learning on patients and so I have
3:54
to say also that we were looking more at technical
4:00
aspects at that time and I found looking
4:05
backwards that there were multiple ways
4:08
experts suggested to do training and of
4:11
course that meant that there were also different
4:16
training contents and different teaching
4:20
ways even for the same for the same procedure.
4:24
So there was a lot of variability in terms of
4:28
recommendation of how to do it there was a limited
4:33
chance to experiment outside the training room
4:38
sorry the operating room and certainly we were not
4:46
really measuring what at the end of the training
4:50
people learned we had an idea we had an impression
4:55
that people were getting better but how far they
4:59
went it was unclear. So from a personal perspective
5:05
you know somebody relatively new to the
5:09
training education training was the impression I'm not
5:14
about you know your views today but looking
5:17
back at the time did you think the approach
5:21
that was used for training and education
5:24
do you think was it effective and you know
5:27
why would you say that? I can say we believed it was effective it was
5:37
based on impression more than data and again
5:40
there were many different ways of delivering
5:43
training we tried at least in our group to do
5:51
the best with what was available at that time.
5:54
My background as you mentioned at the
5:56
beginning was biomedical engineering at that time so I
6:00
feel I was more focusing on the technical
6:04
aspects of the devices of tools that were used for the
6:08
procedure but there was still a lot of
6:11
variability in the way each training was conducted. I think
6:16
since then the journey has been long and certainly
6:19
it's not over I personally learned a lot from what
6:23
I did at the beginning I believe that was
6:25
a successful approach I would say today but
6:30
it would be difficult to prove that was
6:33
effective or as effective as what we are doing today.
6:37
So the impression was that it was helpful and
6:40
people who knew very little at least could practice
6:44
a little bit outside the operating room and
6:47
that's certainly an advantage at that time
6:51
thanks to the simulation. The use of
6:53
simulation was not probably the best at different levels
7:02
from the very first simulator created a lot
7:07
of interest many times they were considered
7:11
the attraction of the training event how well
7:16
they were used and I would say it went through
7:23
spectrum from room decoration to probably at
7:30
least the first attempts of standardising a
7:35
procedural workflow so there was no consistency
7:41
in the use of training and simulation at that time
7:46
but I have to say also that there were at
7:51
least on our side many efforts to do better with
7:57
consistency so we started at that time
8:02
especially because very few people at the beginning knew
8:05
how to do things we tried to find a bit of
8:08
consensus of how optimally we should perform
8:12
procedure and there were some people that the
8:16
very first pioneer which I admire because they
8:20
figured out an optimal way to perform the cases
8:26
and when I saw them getting together and discussing
8:30
how to do things I realised that probably the
8:32
power of having a group together of people who
8:36
genuinely tried to help the patients that
8:40
power was unbeatable and of course was much more
8:47
effective than having a single individual showing
8:50
up and telling his own way or his own story so we
8:54
started getting experts together and we
8:56
started having more consistency in the way we trained
9:02
novices on a certain procedure. Again
9:06
looking back 25 years probably we were ineffective
9:13
compared to where we should be and we have
9:15
no data to say what people learned at the end.
9:21
It was a it was already a paradigm shift to be able to take people
9:27
outside the operating room because again
9:30
going back to that those early days of the therapy
9:34
people were taught how to perform the procedure
9:38
in front of the patient and at least the basics
9:42
with the first simulator simulators available at
9:46
least the basics were done outside the room and
9:49
discussions could happen outside the operating
9:52
room that was a big change although yes without
9:55
metrics without benchmarks without
9:59
standards at that time but it was an important step
10:04
on the journey. And do you think that the
10:10
industry and other training centres did they
10:13
see simulation as some sort of magic bullet? I
10:18
think there was a lot of attention to simulation
10:22
because simulation could help industry explaining
10:29
or I would say at that time demonstrating the use
10:32
of the product at least the workflow of the use
10:35
of the product in a safe protected environment and
10:40
of course that is difficult to do in the
10:44
operating room. At the same time the simulation was
10:53
used in multiple ways and I would say not always
10:59
in the most effective and many times it depended
11:03
on the on the training team on the experts
11:09
that were delivering the training so there was
11:11
inconsistency in the way it was used so
11:15
participants trainees were attracted by the simulator
11:18
themselves because they believed they gave them
11:21
the opportunity to do something safer but also
11:24
was the novelty that attracted them. Now we
11:28
know that simulators by themselves don't teach much
11:32
if there is no solid curriculum around so maybe
11:37
the technology was there maybe the use was up to
11:40
the individual trainer to make it
11:44
effective and so you could see multiple versions of
11:50
good or bad use of simulation. So the
11:54
shortcomings that you saw in industry training,
11:58
I mean for me it always struck me as an
12:00
educational experience but industry was doing training on
12:06
simulation and as you know I'm a huge fan of
12:08
virtual reality simulation. So were the shortcomings
12:13
just in industry or were they in other sectors as well such as the
12:18
scientific societies or universities?
12:21
What's your observation? I think it's difficult
12:25
to generalise if I have to give my perspective
12:30
I think there were many common aspects
12:33
between what happened in industry, what happened in
12:38
the medical field and one of them was we sometimes underestimated the
12:49
how large the gaps were in certain areas or
12:52
we gave for granted some skills that were not
12:56
instead demonstrated. So trying to build
13:00
new skills on a weak foundation was not always
13:07
simple. So before simulators and that's when
13:11
I started before simulator of course training
13:13
happened mostly in the operating room and that is
13:19
definitely not the safest and most effective way
13:22
to do it. We also saw common aspects with lack
13:28
of standardization so training delivered in one
13:33
hospital was different from training delivered in
13:37
the hospital maybe next door although the therapy
13:40
was the same and sometimes also this happened
13:45
depending on which experts or consultants were
13:49
delivering training on a simulator so this was a
13:53
common aspect and so there was also tendency of
13:59
having more speaker-centred training so the
14:05
learner was there to listen to these pioneers
14:11
which of course had experience that were admired
14:14
for what they did but listening to an expert does
14:18
not mean that you become an expert and so this
14:22
was probably a misunderstanding that was happening
14:26
at the same at that time and by the end of the
14:30
training people were satisfied with the training
14:33
but not sure how much they could do the same as
14:38
the speaker that was teaching. To summarise
14:41
I mean I would say common aspects I think the
14:47
most challenging and critical was the fact that the
14:50
learning curve was happening on patients in the
14:53
clinical environment so that was something that
14:56
looking backwards would still worry me. So I mean
15:04
you so you observed all this for a number of years
15:07
at what point did you come to the conclusion
15:11
that a different approach to training was required
15:14
because you seemed to have all the right
15:16
bits in place I mean you had a structured course
15:20
you had a training on simulation which
15:22
allowed training outside the OR but you seemed to
15:26
move beyond that so you must have come to the
15:28
conclusion at some point a different approach
15:32
training was required. What drove that? I think
15:36
in my experience for myself I was no one single
15:42
turning point I think it was an accumulation
15:45
of evidence and there are some moments when
15:51
you wake up in the morning and you put
15:53
together those pieces and bites and say it's okay that
15:58
maybe there is something we can do and I think it moved from
16:07
analyzing more critically how different
16:11
techniques that were adopted in the operating room and
16:20
given in training as well we're generating
16:24
different outcomes so you start realising that
16:31
actually it's not only the way you apply the
16:35
technology it's also the way the patient benefits
16:40
or benefit less from the technology and you see
16:43
that the way you taught somebody doing something
16:47
may determine the outcome of the patient so
16:52
I think I learned that you needed to pay more
16:56
attention to the gaps to be able to analyse exactly
17:01
what the gaps were and try to find solution to get
17:05
people at a certain level I think if I have
17:08
to say and we have known each other for so long
17:12
I think there were moments like when I
17:15
first got across one of your publication
17:19
in our group training group and when I read that I was at that point
17:28
already a few years in my training position
17:31
and I realised that everything I read there
17:35
made sense to me although was still mostly
17:38
applied to minimal invasive surgery but the
17:41
approach made sense for other types of procedure
17:44
as well so at that point I said okay not only we are
17:50
realising that gaps exist but also there may
17:55
be a potential solution that can be applied to
17:58
fill those gaps the challenge is the gaps were
18:03
very variable depending on where you were looking
18:06
and so also the degree of improvement required
18:11
was different at different levels but that was one
18:15
of the moments where you said okay I may be
18:19
novice in education and training but I could learn from
18:24
experienced more experienced practitioners
18:28
that you don't have to reinvent the wheel
18:33
but you can apply something that has been
18:36
validated and proven working already and maybe
18:40
this could help in our sector as well so that was
18:44
a moment I remember very well and that's probably
18:47
the moment we started getting in touch and I
18:50
thought okay there is a direction the direction
18:53
is pretty clear and there is something we can do
18:56
more I have to say I've learned every year that
19:01
there is something else we can do better and I
19:04
think we haven't got to the point where I said
19:07
we have done everything there is still a long
19:09
journey but I think I think we or I personally
19:14
had to go through the different steps at the
19:17
pace that was required to come to the conclusion or
19:23
partial conclusion that we needed to move one
19:25
step forward and I think we've gone through a long
19:29
journey and there is still a lot of room for
19:32
improvement so multiple moments where accumulation
19:36
of evidence showed us that we needed to move
19:40
one step forward and one step after the other
19:44
I think we progressed a lot it would be easy
19:48
to say now we could have skipped that phase or
19:52
skipped the steps but probably we would not be
19:54
here without all the stops we made on the journey
20:01
yeah no I agree with you I mean it's about
20:04
learning the science of this approach because
20:07
it's an entirely different approach to
20:09
conventional training so I think if I may if I made only
20:15
I think I learned one thing that probably drives
20:19
most of the innovation I think if you genuinely
20:23
believe that you want to improve training every
20:31
day and your purpose is to make healthcare safer
20:39
and more effective I think you are motivated to look for improvements
20:49
there was a time we discussed about
20:53
accumulation of marginal gains I think life has been
20:57
this type of journey for me and sometimes the gain
21:03
was big sometimes was small but every small piece
21:06
made the construction I totally agree with you
21:11
about that whole approach to the aggregation of
21:13
marginal gains and to a large extent that's
21:18
at the core of proficiency based progression
21:22
so over the last number of years you've
21:24
developed and validated a proficiency based progression
21:27
approach to training of cardiac pacing and cardiac
21:31
resynchronisation therapy can you briefly explain
21:36
what pacing and CRT are? Yes absolutely so
21:41
I've always been involved from my early years in
21:46
medtech on the cardiology side and pacing
21:53
cardiac pacing and cardiac resynchronisation therapies
21:57
are treatments that are offered to selected
21:59
patients who have some cardiac rhythm abnormalities
22:03
and pacing has been cardiac pacing permanent
22:06
cardiac pacing has been around for about 70
22:09
years already cardiac resynchronisation is more
22:12
recent or relatively very recent because it's about
22:15
25 years old maybe almost 30 but all these
22:21
therapies require an implantation of a cardiac
22:29
consists typically on a generator which is
22:33
placed in the chest under the skin and the generator
22:36
contains the battery and contains the circuit
22:41
that decides what therapy and when the therapy
22:45
should be delivered and they are connected to
22:48
the heart rhythm where they can listen to the heart
22:53
and deliver therapy to the heart through
22:56
implanted leads inside the heart and this is the most
23:03
traditional way of delivering cardiac pacing
23:06
and recircurization in the recent years there are
23:10
alternative ways that have been suggested but in
23:15
general the therapies are delivered for patients
23:18
who are suffering from any cardiac abnormalities
23:24
and have certain criteria so this is in summary
23:27
what we do and we train physicians to make sure
23:31
they can use the technology for the safest and
23:36
most effective approach to have a better patient
23:39
outcome. So it's okay you're working in the sector
23:45
and the education and the training did you
23:49
think there were major issues in the education
23:52
and training of these procedures or were you
23:55
happy with how it was being done? I think again
23:59
going back to what we discussed before I think
24:02
the first challenge I observed at the beginning was
24:05
that if training or observation or coaching
24:11
happens in the OR, the laboratory room or in the
24:15
cat lab, I think that's definitely too late and
24:21
it's not certainly fair that the learning curve
24:25
happens on a patient so that that was for me
24:29
challenge number one. Challenge number two was
24:35
there were a lot of different interpretations
24:41
of what optimal therapy, optimal implantation
24:45
of the cardiac device consisted of and nobody
24:51
ever compared all those approaches and of course
24:55
depending where you learn the approach you
24:59
could be a very good or a less good operator
25:03
and not only in different countries there were
25:09
different approaches but even in the same country
25:12
sometimes in different hospitals they were
25:15
applying different techniques with different
25:17
level of success and so I would say the system
25:23
the medical education system was not optimal because
25:27
at the end of their training physicians had
25:30
a large variability of performances despite
25:33
being licensed to perform the procedures so
25:39
those were the aspects that I considered the most
25:45
challenging and the most critical at that time.
25:51
Were these simply catheter wire performance or
25:55
was there other aspects of the procedure that
25:57
you considered okay this is not done optimally?
26:01
I think I've learned the clinical side of the
26:07
therapy by working with very smart clinicians
26:11
and I learned at the beginning at my time I was
26:15
mostly focusing on the technology and the aspects
26:18
that of the procedure that required the
26:21
technology to be implemented but if you look at the entire
26:25
procedure I realized over time that were parts of
26:29
the procedure we gave for granted or were assumed
26:35
to be conducted without the problem instead
26:38
skills involves specific domains were not certainly
26:43
guaranteed and that's when we had to start
26:48
looking at gaps not only related to the device but also
26:55
what two complications that would ultimately lead
27:00
to a system a technology malfunction but was not
27:05
were not strictly related to the technology but
27:09
were related many times to a suboptimal use of
27:14
a technology so that's when we realised that
27:17
actually gaps may happen at multiple levels
27:21
at all phases of the procedure. I remember
27:26
Hartwick Redslaff took me to a hospital in Dublin in 2006
27:30
I think it was. It showed me the first pacing
27:34
procedure I ever saw and I go this is a surgical
27:37
procedure what's the cardiologist doing it and it
27:40
turns out that most or well a significant number
27:45
of the complications to do with cardiovascular
27:48
devices are to do with the surgical end of it
27:51
and on the courses that I observed there
27:55
seemed to be very little attention paid to those
27:58
I mean when I was at Emory in 2003 we
28:02
were working with the carotid artery stenting
28:06
and one of the highest complications was on
28:09
the femoral axis there was infections there was
28:13
hepatomas and so on and like two decades
28:17
later I was working with the TAVR procedure
28:23
and turns out that the complications still
28:26
existed, so you've developed a proficiency-based
28:30
progression approach to pacing and CRT
28:36
devices so what does that training program involve?
28:41
Okay so I'll start by saying that it's not
28:50
just an interesting educational experience as you
28:54
normally call it. It involves from the very
29:01
beginning a very structured approach to how
29:05
the procedure should be performed according
29:08
to certain standards what metrics need to be
29:13
checked to ensure that trainees progress and
29:19
demonstrate that they can do the procedure at
29:22
a certain level proficiency benchmark that need to
29:24
be defined based on experience operator performing
29:28
the same approach and it requires after a good
29:35
investment into the development and validation
29:39
of everything you created it requires an extreme
29:42
rigour in the implementation of what this science
29:46
has generated. A rigour that starts
29:49
certainly with trainers. The trainers need to be
29:55
proficient in applying this rigorous methodology
30:01
and need to demonstrate that they can consistently
30:06
transfer those knowledge and skills to the
30:11
students and they also need to make sure that
30:16
not only in the first second or third training
30:20
they perform but over time they don't decrease
30:23
the level of attention they have to the details
30:26
of a procedure when teaching and they make sure
30:30
that trainees only progress when they
30:34
demonstrate all those details. So it's attention to details
30:39
it requires multiple components in fact it's not
30:43
a training event where people get together around
30:48
it starts with it's a blended learning
30:51
curriculum it starts with a knowledge component which is
30:55
usually delivered by a learning participants have
30:58
to demonstrate the proficiency benchmark on their
31:02
knowledge before being able to advance to a
31:05
hands-on training. So faculty training I think
31:08
is critical you need to have consistency you
31:12
have to have very solid agreement about how the
31:15
procedure is performed and taught to the people
31:20
and you need to be very disciplined in the way
31:23
you train and assess the performance. If you stick
31:29
to the methodology the results are generally very
31:34
predictable if you deviate even lightly from
31:38
the methodology you may encounter some surprises.
31:44
Can I just back up a bit why is the validation important?
31:50
I mean surely you develop a depth restriction
31:52
you'll use them why is the validation important?
31:56
I think because you want to make this
32:00
approach bulletproof you want to make it
32:04
you want really to make it as objective as you
32:09
can and nobody can come back and say that's your
32:12
opinion that's the opinion of 10 people you need to
32:16
make sure that what you have chosen is what works.
32:20
Yeah yeah no I totally agree with you I
32:22
think it has to be objective transparent and fair
32:25
for both the trainers and the trainees
32:28
because this is a different approach medicine
32:33
particularly procedure-based medicine is not
32:35
accustomed to their performance being assessed
32:39
to reach some quantitatively defined
32:42
performance benchmark and I think the onus is on us
32:48
to demonstrate okay this is objective it's
32:50
transparent and it's fair and if somebody's
32:53
failed you know they don't demonstrate
32:55
proficiency you can show them why but the good thing is they
32:59
can do it again and they can do it as many times
33:02
as they like but the the rule is you don't progress
33:05
if I may add something to what you
33:10
said for the people listening is sometimes
33:16
probably myself as well at the beginning were
33:20
afraid of how participants how trainees would
33:24
react to such a strict methodology and actually
33:30
the the recipients have been extremely positive
33:35
because as you said it is very transparent
33:38
so they know what they are expected to do
33:41
they are not afraid of having different
33:44
individuals evaluating them depending on the day of the week
33:48
or depending on the place they come from they
33:51
know that if they do what they are supposed to do
33:54
they will get where they are supposed
33:56
to get and proficiency-based progression
33:59
I insist with that aspect which is not always
34:02
very clear to people is not meant to fail people
34:06
it is meant to take the right amount of time
34:11
people need individually to get to the benchmark
34:15
you expect so we are actually helping people
34:18
finding their way with a very structured approach
34:22
to the level of performance they expect
34:27
themselves to reach and sometimes they are afraid that they
34:30
can't get there and they get conflicting
34:34
information and recommendation about how to proceed
34:38
and here instead is very transparent so it's been
34:41
well received we haven't had any objection about
34:46
why do I have to do this who says that this is
34:49
the best way they have been looking at ways to get
34:53
better to demonstrate that they could reach the
34:56
objective and the great majority of them get to
35:00
where they need to get
35:01
yeah yeah, I mean I think
35:04
that's an important point because the whole point
35:07
of proficiency-based progression I mean
35:09
everyone concentrates on the metrics and the assessment
35:12
and they think it's the assessment is the most
35:13
important the assessment is only important to give
35:17
the trainee objective transparent feedback on
35:20
their performance and to coach them how to pass
35:24
I remember giving a professor a lecture in
35:26
Leuven and somebody stood up and said hold on a second
35:29
you're educating them what to do what the tasks
35:32
are before they get to the skills lab and I'm going
35:34
yep and they can do it as many times as they
35:36
like go yep and then they get to the skills lab and
35:41
coaching them what to do and they can fail that
35:46
and they get as many goes so that's not the way
35:48
we do things today and I go our job is to help
35:52
them get to the benchmark what's non-negotiable
35:55
is to get to the benchmark and I know I'm not
35:59
checking or we're not checking how smart they
36:01
are I already know that they've got into
36:03
medical school they've progressed through the system
36:06
so the the idea is that we help them the
36:11
metrics are there to help them and that's one of the
36:13
I think the most powerful aspects of
36:15
proficiency-based progression it's about the the objective
36:19
explicit formative feedback they get
36:21
about performance and my experience is that
36:24
my experience is that they received that
36:30
approach better than what I expected at the beginning I
36:35
thought some people may question or challenge
36:38
or be critical about the rigor the methodology
36:44
requires but at the end they are all very
36:47
compliant and and successful in what they do
36:52
yeah but I think they pick up fairly quickly
36:54
that we're on our side I mean we'll help them
36:57
get to the benchmark I mean there's no tricks
36:59
about that but what's non-negotiable is to get
37:02
to the benchmark and Stefano pulled out that
37:06
we get everybody to the benchmark
37:08
especially in a situation the people I meet we train
37:15
international people from mostly Europe they
37:21
come and their biggest challenge today or one of
37:25
the biggest challenges is the lack of systematicity
37:30
and they have to find their way to navigate
37:34
through the system to get the learning they
37:37
expect to get but is not provided to them in a
37:41
structured way so they have to to get it somehow
37:46
and here is pre-packed very systematic and
37:50
they can go this is the comment they often make
37:55
it's probably one of the first time they can
37:57
go from A to Z through everything they need
38:02
receiving feedback and being able to
38:06
observe and confirm their own progression
38:10
and they always say it the majority depending of
38:14
course again from where they come from where they
38:16
learn but the majority of them complain today
38:20
about the access to consistent structure training
38:27
in the order they need in the moment they
38:29
would need and with proficiency based progression
38:33
training that is actually very clear the pathway
38:37
is very clear so they appreciate the transparency
38:41
and the objectivity of that so over the last few
38:47
years you've been working on a proficiency based
38:49
progression approach to device implantation
38:54
what does the data show you published this in
38:57
peer-reviewed high impact peer-reviewed
38:59
journals so what's the data show
39:01
so the data our data
39:04
showed consistently that people trained
39:08
with a proficiency based progression approach
39:11
outperformed significantly in an objective
39:16
standardised assessment what people train
39:20
still train still with simulation still with
39:23
same content but with the traditional methodology
39:27
and our data showed that there was a reduction
39:31
in procedural errors in lead implantation of 77%
39:37
and there was a reduction in surgical
39:40
skills the surgical task related to the device
39:44
implantation of 61% and and we also showed more
39:50
recently that those skills were transferred to
39:55
the real world because the adherence to optimal
40:00
performance in in real procedures was far superior
40:05
in people that were trained with proficiency
40:07
based progression approach compared to people
40:09
that were trained traditionally so they they
40:13
reach the proficiency benchmark not only they
40:17
perform better making fewer errors they also
40:21
have higher chances to demonstrate the proficiency
40:25
benchmark that is required in in a
40:31
reasonable amount of time compared to their peers the
40:35
difference we had in the two trials for lead
40:38
implantation we had 93 of the students in the
40:42
PBP group demonstrating the proficiency
40:44
benchmark and none of the other group demonstrated the
40:50
benchmark and in the surgical skills was 73
40:54
compared to 20% so not only you make fewer errors
41:00
you are able also to progress to the
41:02
benchmark more efficiently and more effectively
41:07
and the proficiency benchmark was based
41:09
on the class average or something else
41:13
no the proficiency benchmark is defined up front and
41:18
the way we set the proficiency benchmark is having
41:22
experienced practitioners going through exactly
41:25
the same task the same procedure that you train
41:29
participants at and you'd measure the average
41:32
performance the average score of the experienced
41:35
practitioner performing exactly the same task we
41:38
know that's not perfection we are not asking for
41:41
perfection but at least it's a realistic real
41:44
world benchmark that these people should at least
41:47
achieve in simulation before moving on and
41:50
when i say moving on it doesn't mean we make them
41:55
experts right away they will still need to
41:58
probably practice under supervision but at least
42:02
they will start from a demonstrated fundamental
42:07
skill set that their supervisor can trust because
42:12
it's been proven and the part of the learning
42:15
curve that is typically happening on patients
42:18
the early learning curve which implies risks
42:21
higher risk than later is moved away from the
42:24
patient and is now happening on a simulator
42:28
with a systematic approach
42:31
so does the PBP approach
42:33
take longer than what you would do normally
42:39
I think
42:40
Sorry I'm not talking about the
42:41
development of the metrics I'm just talking about
42:43
the the training program because you normally train
42:46
what one two three days so does the PBP approach
42:51
does it take longer?
42:52
So at the end it doesn't take
42:54
longer because if we compared on the same
42:58
amount of time and we know that proficiency based
43:01
progression should be not should not be tied
43:04
to a fixed amount of time because individual
43:06
different individuals learn at different speed so
43:09
everyone should have the right amount of time to
43:11
get to the benchmark that's the ultimate
43:13
goal but if you have a determined fixed time you
43:15
goal but if you have a determined fixed time you
43:20
you compare the the progression of the students
43:23
in the two groups as we did in the randomised trial
43:28
You realise that actually not only they
43:31
get more likely to the benchmark but they also do it
43:35
faster i say or the majority of them over the
43:39
two days of the training could reach the end goal
43:42
and the others who are still struggling with
43:44
getting there so you could say that not only
43:47
is more effective but is also more efficient so
43:51
that leads into my next question so you publish
43:55
the results in high impact peer review journals
43:58
so how has the data been received by the clinicians
44:03
not just the people that do the training
44:06
but also by your colleagues in industry
44:11
I think it's a it's a big change of mindset i mean some people
44:20
and luckily I had the privilege to meet
44:25
quite some very open-minded realise that
44:29
based on data there is a better way to do things so you consider changing
44:37
you cannot say everyone agrees with that
44:40
some people are surprised by the magnitude
44:43
of the difference question the simulation
44:46
question the methodology have doubts that
44:50
the current system does not work maybe
44:55
maybe will require several years before a critical
45:01
mass of people accept that it's time for change
45:06
but I think and this is true inside outside the
45:11
industry there are early adopters and people who
45:15
are thinking about and people that will probably
45:17
join later however today i don't know a better
45:23
alternative there may be one alternative one day
45:27
if based on data we see that there is an even
45:31
better way I would certainly support that today
45:35
what we see is that a different approach to
45:38
training as the one we propose with proficiency
45:41
based progression leads to better outcomes and
45:44
this is data driven and it's not just opinion
45:48
so some people have understood very quickly
45:51
some people are thinking about some people at the
45:54
moment are still in denial but the evidence
45:57
accumulated will probably lead everyone to be
46:02
inevitable which is change is needed to get
46:07
better patient outcome through better performances of
46:12
operators
46:13
So is PBP more expensive?
46:19
So the data we
46:20
have and this is not data we could collect on our
46:25
research but the data is probably cross
46:29
therapies you require of course with proficiency based
46:33
progression and a higher investment at the
46:37
beginning of course you have to invest more
46:39
than putting together a few powerpoint
46:42
slides obviously however because of the
46:46
more effective and more efficient approach
46:51
once you implement you will get quickly to the
46:57
break even so the research you did showed that
47:01
you don't need so many trainees to process to get to
47:06
that level that pays off so the initial
47:09
investment was paid off after 25 students were trained and
47:15
and that means yes upfront it looks like oh okay
47:20
I have to invest more here but if your training
47:23
is not a one event only but it's a program
47:26
that you want to consistently deliver over time
47:31
the return will be most important of course
47:35
better performances of the people but also
47:38
the cost of the investment will not be as
47:41
big as you believe at the beginning
47:43
Yeah, I mean
47:45
Stefano Puliatta's data honestly surprised me
47:48
they gave me a huge fan of simulation and obviously
47:52
I'm quite fond of proficiency based progression
47:54
but I thought it would take us we were talking
47:56
about a robotic surgical skills task I
48:00
thought it would take us hundreds till we got to the
48:02
break even point but as you quite rightly pointed
48:05
out took 25 I think shows that it's a much
48:10
more efficient and effective approach to
48:12
training and I would have thought that it was something
48:15
that the industry would they would embrace that idea
48:19
I think it will come it's it's it's something
48:23
despite you've been working on this for probably
48:26
30 years I think we we probably have to
48:33
create more awareness of what you have
48:37
achieved over time and the power of proficiency
48:41
based progression I think the actual cost or
48:44
number of trainees will depend on the therapy
48:47
on the resources that are invested may vary
48:49
with the therapy but the result will not change
48:52
so most likely the number of trainees needed
48:57
to reach the break even will be lower than what
49:01
you expect and your investment in training
49:04
will pay off soon I think there is an important
49:09
assumption we are making here and that's something
49:11
that sometimes I find myself to discuss with people
49:16
after a few minutes and not at the very
49:20
beginning I think our assumption is that we all agree
49:26
that we want people we want trainees to reach a
49:29
predefined performance benchmark so if you agree
49:33
whatever the training approaches so if you agree
49:37
with that assumption then you can start comparing
49:41
but sometimes I found myself discussing with
49:43
people about the performance benchmark and the
49:48
assumption was not correct so I assume that
49:51
we all want the people to reach a quantitative
49:54
performance benchmark but in others people mind
49:58
this quantitative performance benchmark is not
50:01
very clearly defined and so then you can probably
50:05
get to some kind of learning with the power point
50:08
although if you haven't defined where you want
50:11
to end up of course you will be successful because
50:14
there is no finish line but you need to agree
50:17
first on what your goal is and how you measure it
50:21
and then you can start comparing the different
50:23
training approaches but this assumption I learned
50:27
needs to be clarified at the very beginning so
50:30
do we all agree that we want people to reach a
50:32
predefined performance benchmark and then we
50:35
can compare the approaches otherwise no comparison
50:38
will survive
50:39
I mean one of the problems I find
50:42
I've worked with medicine now for 40 odd years
50:46
and one of the problems I mean these
50:48
individuals are smart I mean they've demonstrated that by
50:51
getting into college but the assumption that
50:54
everybody else and they make is because you know
50:56
how to do something or you know about something
50:59
you can do it one of the things i've learned over
51:01
the decades is that you can get somebody really
51:05
smart and they really know how to do something
51:07
but they're the correlation between them being
51:10
able to do it well is not a strong correlation
51:13
and I mean like 20 years ago I would or 30
51:17
years ago I would ask okay do you know about this
51:19
procedure yes do you know how to do it yes have
51:23
you done it many times yes I would have stopped
51:26
there today i say okay show me and frequently what
51:30
I find is that they're not really able to do it to
51:35
a performance level so you've developed and
51:39
validated this approach to proficiency based
51:41
progression for the the crt procedure is what
51:48
you're offering just a better education experience
51:52
or are you offering something more?a
51:55
I think
51:56
offering a a good educational experience
52:04
I don't say it's easy it's all and I
52:07
don't say it's easy to offer but it's probably
52:11
easier than offering a structured training
52:14
with a guaranteed outcome so what we offer here is
52:19
if you first of all it's evidence-based so
52:23
you know what you get and how far you can go
52:26
second you know that this is not only the
52:29
opinion of somebody that is there today to teach you
52:32
but this actually comes from a very
52:34
structured consensus and validation so again as we said
52:40
before I don't say bullet proof but it's
52:42
quite rigorous and then once you enter open-minded
52:46
in this training mode then you see that you
52:52
indeed progress and you indeed receive the
52:56
receive the feedback immediate feedback that you
52:59
need in the moment you need to make sure that you
53:03
correct yourself and proceed to the next so what
53:05
we are offering is actually a very clear pathway
53:10
with very transparent mechanisms and a very clear
53:17
performance goal so you know what you get from the
53:20
very beginning you just it's not that easy but
53:25
you just have to follow that path to get to where
53:29
you expect to get so you will be happy with
53:32
probably much less but you will not be performing
53:36
as well if you get something less and I
53:41
I have to say
53:44
over the years another thing I learned I
53:48
had the privilege to work with very talented
53:52
open-minded clinicians and also colleagues that
53:58
believe that there is always a better way to do
54:01
things so we have improved because we never
54:04
stopped with what we had it was a journey
54:09
clinicians believe despite they've been doing
54:12
the same job for many many years with new therapies
54:16
of course but these people I had the privilege to
54:20
work with have realised quickly that there is room
54:25
for improvement and we're talking about big
54:28
improvement and they have dedicated their time
54:33
and efforts to make sure that their fellows
54:39
their students and clinicians in general of the
54:43
next generation can have better access to
54:47
structured training and leading to better
54:53
performances and ultimately of course we expect
54:57
better patient outcomes
54:59
Okay, can I be bold and
55:02
ascii does the training you offered today do you
55:06
think it's better than what we've offered in the past
55:09
Yes I would not be I would not sleep well
55:15
if I would offer today training that is worse of
55:18
what we offered in the past I think it's better
55:21
I think definitely better I think I can say it
55:24
because we now have data and we didn't have data
55:28
before I think having data makes a difference in
55:31
terms of what you compare with and based on
55:35
your own data you can always do better but you have
55:37
to start from somewhere which is not an opinion
55:40
which is not an impression so we are here today
55:43
because we went through the journey we have
55:45
accumulated evidence that shows that the the
55:49
path we took is the right path
55:52
Yeah, I mean I
55:53
totally agree with you data shortens the discussion
55:58
because uh before you have data it's all
56:01
qualitative it's your opinion versus my opinion
56:03
but once you have scientific data published in peer-reviewed journals
56:09
that changes the conversation significantly.
56:12
So where does education and training for the use
56:15
of new medical devices stand?
56:19
And where does it stand now
56:22
and where do you think we need to go next?
56:24
I think I, I mean, the place I work,
56:29
I think there is a commitment to patients
56:33
to make sure that every aspect from the design of a device
56:36
to the training, to the collaboration with clinician
56:42
leads to better outcomes.
56:44
So I think we have made over the years effort
56:48
to provide training that generally produces better outcomes.
56:54
I think we have improved and showed
56:58
that it is possible to do moving from a good training
57:03
where you have an impression that is a good training
57:06
where some experts say is a good training.
57:09
Now we have moved to proving with data
57:13
that the training is effective.
57:16
I think it is important in all domains
57:19
that the quality of the training should match
57:23
the quality of the technology we provide
57:27
and the quality of the technology investment
57:30
in the med tech is high.
57:32
And I believe that training should match that quality
57:35
to be able to provide healthcare with a better package
57:41
to optimize what the patient could get.
57:45
And yes, we are not done yet.
57:49
I think we are on the right trajectory
57:52
and I see things moving.
57:55
Maybe not as fast as people would expect,
58:01
but actually faster than what the general public today
58:05
is going to consider good enough.
58:08
Yeah, I mean, the number of devices I've seen fail
58:13
in the marketplace and the devices,
58:17
I mean, from what I could see were perfectly good,
58:23
but the application of the devices,
58:26
the use, the clinicians using them,
58:29
some of them were suboptimal
58:31
and some devices failed in the marketplace,
58:34
not because of the device,
58:35
but because of the clinicians actually using it.
58:39
And that's not something that's been recognised,
58:43
well, publicly recognised within medicine,
58:47
but the data basically now shows across all specialties
58:52
that we have a group of clinicians
58:55
that perform sub-optimally in comparison to their colleagues.
59:00
And I think if I may add something, Tony,
59:05
I think the great majority of trainees
59:11
I have a chance to meet are really looking forward
59:17
to learning in a more effective, more efficient way.
59:22
So they are genuinely open to ways
59:25
that would lead them to more confidence
59:28
and better performance.
59:30
So I think they feel the need.
59:33
I think the system, it's the system,
59:36
that education system that needs to improve
59:39
to make sure that those motivated individuals
59:42
reach the level they are set to reach,
59:46
and they're probably capable to reach
59:49
within a reasonable timeframe
59:53
and with very clear pathway to a precise goal.
59:58
So the individuals are motivated,
59:59
the system needs to make an extra effort
1:00:02
to provide those motivated individuals
1:00:04
with the tools that are required to make them successful.
1:00:10
Yeah, no, I totally agree with it.
1:00:13
I mean, our goal should be to help those trainees
1:00:16
to get to that performance level,
1:00:18
but the performance level has to be quantitatively defined.
1:00:22
It shouldn't be an attritional system.
1:00:24
I mean, we basically, the whole point of deliberate practice
1:00:28
is you help them get to the benchmark,
1:00:30
but reaching the benchmark's non-negotiable,
1:00:34
because what we believe is, and what the data, I think, shows
1:00:38
is that your performance level
1:00:40
will impact on patient outcomes.
1:00:43
So you've been working on this now more or less for 20 years.
1:00:50
So what would you say are the major take-home messages
1:00:54
from your work, your experience, your research,
1:00:57
your publications over the last 20 years?
1:01:00
Yes, to summarise, so I think you are an example
1:01:05
of the first recommendation I want to give,
1:01:08
so never give up.
1:01:11
There is a path to make training better,
1:01:18
make healthcare more successful,
1:01:21
and it's not only related to the successful treatments
1:01:27
that are introduced or new therapies,
1:01:30
but it's also related to how we train people.
1:01:34
So attention to the quality of training is essential.
1:01:41
Not every training is a good training,
1:01:45
and you should not give for granted
1:01:47
that after going through some years of training,
1:01:50
you reach a certain level if you haven't proven that.
1:01:56
The summary is also showing me that, yes,
1:02:01
if you are motivated and interested in doing a better job,
1:02:07
there is certainly a way that is proposed
1:02:09
that could get you better results.
1:02:12
And I would invite people that are still skeptical to try,
1:02:17
and once they will see their own data,
1:02:20
they will probably find that there is no other way
1:02:25
from the traditional approach that can match,
1:02:30
at least from the data we have shown over the years,
1:02:34
there is no traditional approach
1:02:36
that matches the outcomes that we reach
1:02:39
with a very rigorous systematic metric based training.
1:02:43
I think we need to have people that are,
1:02:51
genuinely interested in moving the learning curve
1:02:53
away from patient.
1:02:54
That's what is my number one concern today.
1:02:59
And I think the operating room at a certain point
1:03:03
will certainly become, or becomes,
1:03:07
an important environment where you can observe
1:03:11
and you can be supervised and perform,
1:03:13
but this should not be the first thing you do.
1:03:16
I believe that there is a lot that can be moved
1:03:21
into a safer environment for learning
1:03:23
and reducing the number of errors you can make
1:03:27
by trying and retrying and receiving feedback
1:03:31
until you demonstrate that you have reached
1:03:34
a safer level with your skills.
1:03:37
And at that point, of course,
1:03:39
exposure to patient is absolutely necessary.
1:03:42
But I think my recommendation be if I could change
1:03:45
one thing today is that everybody starts first
1:03:48
in a protected environment and then moves to the OR,
1:03:55
performing cases in vivo under supervision
1:03:59
and not the other way around.
1:04:03
That's a very good summary, Jorio.
1:04:05
You've been doing this for a long time
1:04:06
and I've been with you for a good part of it.
1:04:11
And thank you for sharing your wisdom today.
1:04:14
It's always a pleasure to talk to you.
1:04:17
Thank you.
1:04:17
It's been a pleasure working with you
1:04:20
and showing that there are better ways of doing training
1:04:24
and of course, the journey, as I said at the beginning,
1:04:27
doesn't stop here.
1:04:28
We're trying to make it always better and better
1:04:31
and having more people involved in this type of approach.
1:04:36
Thanks, Tony, for the discussion.
1:04:40
Thank you.