Latest / Behind The Knife: The Surgery Podcast / Are EPAs Actually Working? An Honest Look at Competency-Based Surgical Training

Transcript

Are EPAs Actually Working? An Honest Look at Competency-Based Surgical Training · Behind The Knife: The Surgery Podcast

• EPAs with Ruchi === [00:00:00] Speaker 2: All right, what an absolute banger, an instant classic. What better than some angsty punk rock to introduce EPAs? I feel like it's appropriate after all. And with that, welcome to Behind the Knife. We are now two years into the EPA era, and it's time to take stock. Are we really moving towards competency-based training where you advance because you're ready, or is this just another box to be checked? Today, we're digging into all of it. Where assessment and surgical education started, where EPAs are now, what's actually working, and what the future could look like if we get it right. And we've got the perfect guest for this. Dr. [00:01:00] Ruchi Thanawala is Associate Professor at OHSU. She's a thoracic surgeon and informatician and founder of Firefly, and she is one of the people responsible for building the infrastructure that could actually make competency-based education real. We're also joined by BTK Education Fellows, Dr. Agnes Premkumar, and Dr. Emma Burke. So let's get into it. Welcome, Dr. Thanawala. Speaker 3: Thank you so much for having me on. It is a absolute pleasure to be here with the three of you. Speaker 4: So starting off with the big picture, what would you say are the overall goals for any surgical residency program? Speaker 3: I think it's really important that we're starting with this question because I think, we're thinking about what the important things are is which, where we wanna be, not how we wanna get there necessarily. So really at a high level to me, a residency program's goal is to provide structured training to achieve competency in the core aspects of a specialty for every single learner that passes through that program. So this [00:02:00] encompasses the holistic practice of that specialty, spanning clinical skills, technical skills, professional, and all the core competencies that are outlined by the ACGME and our other accreditation bodies. Now, I'm gonna say, this is seriously, this is a big ask. There's a lot on programs to actually get each learner through this process. And I would say overall, we really have a working system. We have reliably trained surgeons for many decades, and it's been evolution over the past hundred years. But really, the challenge that we're encountering right now, the conversation that we're trying to have, is that the world is not static. The world is evolving. Things are complicated. Learners need to know more about how they're doing Speaker: So then how does assessment factor into what residency programs have to do to get us to be surgeons? Speaker 3: So, I Think when we think about assessments ... It's like that moment where we think about what's happened, what we're learning, what we're doing, and we put it down on paper. Real paper, proverbial paper, in our mind paper, electronic paper. It is [00:03:00] that pause. It is that moment of what I think is this mindfulness, but captured. Because, we c- we can move through training. We can keep just, like, chugging along, assuming that we're all synced on the position that we're at, faculty position, understanding of learning, the learning they're providing, learner position of where they're learning. But assessments, they make us stop. They're this integrative data on how residents are actually moving, and we need those data points. Without those data points, it's really hard to course-correct. And when I think about what we're trying to do with competency-based education is if everyone moved along in a homogenous straight path, why would we even need competency-based education? We would have education. We wouldn't need to move with competency. These data points allow us to define what the individual trajectories are of our learners through the diverse learning. So long answer to your question, how do assessments factor into this? We need data points because every learner is unique, every learning [00:04:00] is unique, every program is unique Speaker 2: So when it comes to assessing surgeons what makes that particularly difficult? Or is it that much different than any other medical training? Speaker 3: I love this question. It's actually one of the key things that I've spent a lot of my recent time thinking about and work on. The reason that I think surgery is unique to some of the other medical specialties, what I'll say is, surgical procedural specialties compared to non-procedural specialties, is that we have this added dimension. We are learning in these high-stakes, complex, ultra-dynamic environments, the operating room, where there's a lot at stake, decisions need to be made very quickly. And this is layered upon what is already the existing complexity of learning in medical education. And note the word that I'm using, complexity. So, I don't use that term lightly in the sense of like, oh, it's just, it's a system that is complicated with a lot of moving parts. I want you to think about medical education as many of us are seeing it [00:05:00] now. It's actually complexity science. So there are multiple layers of complexity. There are relationships. There are things that are influencing each other, and surgery is that. Surgery, we have so many things that are influencing each other. And then we layer in an assessment, and sometimes if it's not appropriately matched to what we're doing- Mm-hmm ... it kind of throws us off, and we're like, "Does this assessment actually max- match the complexity of what I'm doing?" And I think that's why we actually struggle a lot with competency-based education in surgery. We're making progress, but we still have a ways to go. Speaker 2: Yeah. And many of us have engaged with EPAs either as a faculty or asking as a trainee for a faculty to complete one. So we should probably touch base very briefly on what an EPA is and in general, what are micro-assessments and are there other spinoffs from EPAs that are worth talking about as well? Speaker 3: Yeah. So An EPA, an entrustable professional activity. So it's a core observable task, an event that a learner can be trusted to do safely and [00:06:00] independently. It's scored across various levels of support and supervision that the learner needs. So really an EPA is a bites- bite-sized chunk, as I like to think of it. And it's a composite of autonomy and trust, entrustment. And Patrick, you asked about, the different ways that we've done assessments, EPA assessments being one representation of assessments. But I really wanna highlight autonomy distinct from trust. So trust, entrustment, is predicated upon an understanding of autonomy. And I use this example, like, when we learn to drive or if we have kids that are a driving age we would measure their autonomy to drive before we're gonna go trust them to go drive over to the grocery store, pick up groceries, and get back home. So assessments have been created that actually focus on autonomy and entrustment. And I think really valuing both of those is super important. And some of the other assessments that actually look at autonomy, some of the work we've actually done with [00:07:00] Firefly with component-based assessments, is actually the deconstruction of a learning event into these discrete measures that helps inform autonomy then that helps us predict and understand trust better. 'Cause I really don't think we can distinguish the two. And I think this is one of the things regarding the EPAs that perhaps we need to get back to a little bit more, is also measuring autonomy in addition to entrustment. Speaker 2: Are there any other professions that correlate with this approach in terms of micro-assessments are there any other specialties outside of medicine that use- a similar tool? Speaker 3: I think so. I know so, and I think many of us have probably grown up in them. I was a ballet dancer trained through the Royal Academy for 14 years. If you're a musician You play sports any of these highly trained specialties, these fields, we grew up in micro-assessment. On a regular basis, when you're standing in front of the person who's coaching you, you're getting micro-assessment and [00:08:00] feedback. They're correcting you minute to minute, position to position. And through that, we develop this internal voice. So I think we have a tremendous amount to learn from all these other specialties. And what happens through that process is that we're training the learner in addition to getting data on how they're learning, which I think is actually what is super important within surgery. I do a lot of work in self-assessment as well, and we're thinking about, EPAs and assessments from the framing of faculty, but I kind of want to stretch this conversation of what if we think about entrustment as how much do we entrust ourselves? How much autonomy do we believe we have? And I think other fields like sports, dance, music teach us because we learn self-assessment in addition to our expert-based assessment from the very beginning. Speaker 2: Is that different, though? So if Agnes and I do a case together and I hopefully will teach throughout the case, ideally, at a level appropriate manner, and hopefully Agnes will, excel through that case and will have learned something and pu- and pushed herself to whatever that level appropriate [00:09:00] area of growth is. But that Happens, right? That's, when you talk about a coach, maybe a ballet coach or a soccer coach, that you're gonna get taught throughout, and those are micro-assessments. But the difference is operationalizing these things, right? You've created Firefly to do so, right? To operationalize it and collect these data. But you also have to pick and choose the micro-assessments and create something that is representative of the actual task and, as you said, helps tease apart , autonomy and trust, entrustment in the case. Speaker 3: Yeah, so the assessment, the tool itself is super important and because, again we're going back to captured assessment because we need the data. People need the data to know where the where the learners are, where the faculty are, where training is going. So you're absolutely right. The assessment we create is as important as the thing we're trying to measure. And this is where it really takes a lot of thoughtfulness into the assessments we're creating. And, Sometimes, oftentimes I'll tell you, in my opinion, [00:10:00] a very simple assessment, like trying to do a global score, one question for a really hard case that you and Emma did together, super hard to do. I'll give you example from my world. I'm doing a lobectomy with a PGY4. Four-hour case. A lot of stuff happened. It is incredibly hard for me to put together one score for that. What is far easier is to understand and deconstruct all the various steps that reflect the complexity of what we just did together, what we all just, like, achieved, and answer those questions. So in addition to the assessment matching what we're doing, it's the assessment should match and mirror the complexity of what we actually did. It's actually far easier, and through Firefly we've actually studied this, that it is less cognitive work to do something that is, like, a whole bunch of discrete steps than asking someone to do one global assessment for a really complicated thing that happened. Speaker 4: I really like the point that you mentioned, Dr. Dhanoa, about, assessing our own [00:11:00] entrustment for particular cases. I think as a resident, a lot of the feedback that I've gotten is either too late to be implemented on or just lacking, and you find yourself comparing yourself with your co-peers, which you don't really understand where they're at, but you just have, like, a subjective sense of what you think they might be, and you're trying to measure yourself up to them. What would you say so far, how have you assessed resident competence and, what do we hope to change from that? Speaker 3: Yeah. So, resident competence is measured in two ways, and I'm first gonna start from the faculty position 'cause, in our learning environments, there is this um, greater direction of flow, which is faculty experience and skill flows to the resident. But there is bidirectionality also, right? 'Cause there is a relationship between the two people at the table. Every one of us has experienced this. We stand at the table together, and we're doing this together, right? The skill of one amplifies the skill of the other and back and forth. The stress of one is [00:12:00] taken on by the other, and we try to balance each other to get to a shared goal. So it's not just a unidirectional flow, and this is what um, faculty assessment of skill balanced with self-assessed skill is. So Agnes, you come to the table with all you have, all the skill you have, but behind that is your belief in yourself to deliver upon that skill. This is the meeting of faculty perception of skill and competence and resident perception of skill and competence. So you can see when we really focus on assessment only from the faculty side, we're missing an important dyad, a dynamic, which is both. Which is why I think in true competency-based education, we need to be studying not only faculty-based assessment but resident-based self-assessment. And I'm gonna take this one step further, which some people might think is radical. I think that residents are better self-assessors because you're internally consistent if it is a well-developed competency than some faculty may be of you. Faculty should be the [00:13:00] checkpoints. We should be training you to know your skill well, reflecting and respecting the flow of information that's coming through you, to you through the faculty, and we should be looking at your self-assessment competency learning curves first, and then marry that with the faculty Speaker 4: What would you say for residents who might not be as, self-confident in their own abilities or need a little bit more of a push to be like, "You can do this particular skill or case"? Speaker 3: So, um, it's teachable. And the reason I say this is because I've done the work on it, and I've seen, And we're publishing on this. I have a grant behind it from the American Board of Medical Specialties Visiting Scholar Program, that self-assessment has its own learning curve, so it can be taught. It progresses much like any skill. Which means that if someone lacks the confidence at that time, means that confidence can be bridged. What we need to do is bring transparency, which then connects back to the conversation we're having [00:14:00] about data. If we can see it, we can fix it. If we cannot see it because the data is not there and it's not granular enough, we cannot bridge that confidence gap. Speaker 2: Taken as a whole, if you just sit down and say, "Okay, we made EPAs. We have faculty are gonna fill them out. The trainee's gonna fill them out," how good are we collectively at providing that feedback? What have you learned through, this massive repository you're collecting at Firefly? Speaker 3: Yeah. So, um, things that we're doing well. We are getting assessments in. Faculty are doing assessments. Residents are doing self-assessments. I think that is, when we think about technology development and deployment, it is about first feasibility. Can we even have a system and a standard that is feasible to implement? We've proven it's feasible to implement. Next, does it matter enough for people to do it? Are they bought into it to put in the effort? People are bought into it to put in the effort. But this is where kind of things split. Not everyone is bought in enough to put in the effort. But we have enough where we have a growing data set. But we have some skewing, which we [00:15:00] can go into later, in the data. I think what we've proven, which is really good and super important as-- 'cause even if we wanna get to competency-based education, that there's an appetite for it. Everyone wants to move. Everyone has realized, learners to faculty to programs to the boards, that where we have been is not enough. And I think we're showing that. I mean, our Firefly data set, the ABS data set, so many of these data sets are so big and rich. And they show a progressive increase in participation. So we have a lot to capitalize on Speaker: I wanna actually get a little bit deeper into the data because that is where a lot of my interests lie in implementation science and feedback literacy. So what kind of data have we been able to get from EPAs and what is missing as they're currently designed? Speaker 3: So what we're getting from the EPA assessments that are coming in is um, a broad distribution, I'll say. [00:16:00] Every institution is participating, and that is really thanks to the ABS that has mandated that people participate. I think if they had not made this a requirement, we would see even more skewing than we are. I think that is very important that data's coming in from everywhere. So we're getting insight into what's happening at each institution. We have diverse learners, training programs in all sorts of different environments. But what is missing um, is comprehensive representativeness. So this is now starting to get into some, you know, data lingo. It's not just about absolute number of data. It's not just volume. It is how representative our data is of reality. And we are seeing, and we've published on this, is that we have high degrees of skewing in the data. A small portion of faculty do a large portion of assessments. And we know that faculty and learners, we all inherently as [00:17:00] humans, we have our own biases. We have stringency, leniency biases. Those fluctuate with how our day is going, with what's going on in our life. So the representativeness is what we're struggling against. And going beyond that um, not every EPA actually has an even opportunity. Like pancreatic necrosis does not come up as often as, right upper quadrant pain or right lower quadrant pain. But what that means is we need to actually build that into our models, and this is where just collecting data is not enough. It's understanding how representative or non-representative it is and how we actually correct for it in our models. So that's where the computational data science comes in. Speaker 2: So broadly speaking, is this good data ? The ABS is talking about moving towards competency-based training periods Speaker 3: so I don't think so. I-- and I'm just being very honest about that from the data standpoint. It is not from any- belief that we can't [00:18:00] get there. Looking at the data, we have such a high degree of skewing that as in who's doing them, who's getting them, what EPAs are being done, and which ones aren't, that um, I think to be able to make substantive decisions off of that, we really deeply need to look at the data set and figure out which gaps we need to fill. The thing is, is this a solvable problem? Speaker 4: Yeah. Speaker 3: Utility-based decisions on where to do assessments. So last year we had presented at the APDS and published on this. We've, we can model utility as in where to fill in the gaps. How do we correct for skewing as in ac- asking for the right faculty member for the right assessment at the right time to fill in the gap? We can make our data set whole. But in this current phase that we're in, we have gaps in the data, and I think that our next step before we truly move on to time variable training, promotion in place is actually deeply and critically looking at our data set and deciding what [00:19:00] modifications we might need to make in either the data that we're capturing the assessments or the process. Again, it's not a criticism at all of what we've done. We're in the early stages of this. We're learning what we're learning about how we use this, deploy this, and how we need to make modifications. Speaker 2: So an iterative process for sure, and I'm excited to hear that again you, you think it's a fixable one or at least one that can be uh, improved upon. Uh, because I think the rollout has been a bit clunky because in terms of the technology used, -- the applicability of it was limited, right? So we had an interface in which I can easily add in um, , data on one of these eighteen EPAs. But on the back end, it was very difficult to deliver that data uh, in a meaningful way to our CCC, to residents themselves at their, biannual reviews uh, to be able to show the faculty that your efforts are meaningful and say, "Hey, look at this. Here's your flowchart. Here's your graph. Even if flawed at baseline in terms of an imperfect data science [00:20:00] or an imperfect foundation, is critical for that feedback loop to say, "Oh, okay, the faculty that maybe was skeptical sees that CCC is less subjective." They get that objective feedback. That's exactly the point. Speaker 3: Well, I think, um Doing assessments is 25% of it, and 75% is analysis and surfacing of signals and insights. So in our first iteration, first cycle of this, what we needed to prove is that we could deliver assessments out. Now, you can't run that, that line very long because you can't ask people to put things in without giving them something back, which is surfacing signals and trends, insights, which is analyzing the data, giving them a plot, giving them some feedback on the feedback they're giving, right? And the two fit into each other, 'cause when we do the work of doing an assessment, or for Agnes and Emma doing a self-assessment, [00:21:00] what you wanna see is that you're getting something back for it. What we forget is that That motivation drives assessment behavior because we bring meaningfulness back in. So I think, what we're trying to do in this first wave is prove that we could get people to do assessments. But what we quickly need to evolve and what we have evolved to do is we need to show people why what they're doing matters, why every single assessment they're doing is helping learners to learn better. That loop is actually probably one of the tightest because we lose people. If we spend a month putting in assessments, doing EPAs, heart and soul, everyone's doing them, every case possible, but you get nothing back from it, it's hard to keep going. I mean, all it is taking time and energy, and no one is actually seeing anything. I'll give you a, an analogy. As an informatician, the electronic health record. We've lived this, right? The [00:22:00] EHR was actually built for coding and billing. It was not built for clinical care. We've adapted it and adopted it for clinical care. But only recently, in the past, like, year or two, or I'd say it stretches back even further, have we been able to effectively get data out without having to, like, scan literally hundreds of, like, procedure notes, progress notes, discharge summaries manually. What we wanna do is to have the same thing with assessments, right? Like, if you put assessments in, you need something out of it. We don't want data to go into the black hole equivalent of electronic health record. That's what we want to avoid. Speaker 2: Yeah. So let's spin this around for our trainees. Maybe Emma, starting with you. What have you seen so far? I mean, every institution is at a different place with EPAs in terms of rollout and uptake and usage, but what have you seen? Speaker: I think that something that's so important that the trainees want is those narrative comments and that [00:23:00] specific feedback rather than, like Dr. Thenawala was saying, just a composite score, 'cause that doesn't really tell me how I can get better. And I've spent a lot of my time looking at the quality of the narrative comments, and that can really make or break the EPA, I would say, as a trainee in terms of its utility, but also how I feel about launching the next one. Like, I am going to be more willing to launch my EPA to somebody who gives really thoughtful, high-quality feedback. But then at the same time, you are burdening those people who are dedicating so much of their time to your education at a disproportionate rate. Speaker 2: Yeah. At a disproportionate rate is right agnes, where are you at with EPAs? Speaker 4: Yeah, so our program uses SimpleApps for our EPAs. And we've done a handful of them. It's actually now a requirement that every week we have, I think, five or six EPAs that are submitted. And I think similar to what Emma was saying, I've used it as, a benchmark, 'cause it asks you, how you rated the case. Like, how hard was it? What do you think you did well, didn't do well? And I, really appreciate the narrative comments that the attendings provide and how they [00:24:00] viewed the case and how I did. I think it's a way for us to, discuss more about the case in an objective manner. 'Cause there's been multiple times where my attending has submitted a EPA on my behalf, and then if we're doing a slew of cases together, at the end of the day, he would talk about all of them and be like, "These are the different comments that I had noted, and this is how you can kind of move this step forward." And I think it's critical to have that discussion Otherwise, you're left with these different disparate thoughts that it's hard to put it together unless you have someone who's, like, coaching you through it Speaker 2: ruchi, what do you think about those collective experiences? Speaker 3: I think um, it's awesome. You can see it makes me smile just because uh, I think of myself as a humanistic computational data scientist. So data is intended to perhaps just barely represent, even if we're good at it, the complexity of the human relationship that actually happens. So that narrative, like [00:25:00] that is again a one level up representation of what happened. I think that's why we as people find it so valuable. Even if, say, you do one of our component-based assessments, you got 20 data points. People are gonna look at that, but what matters is the statements because there's so much in that, right? That is a reflection of like, did I enjoy it? Did I not enjoy it? How did we feel about it? Like, where do we wanna go from here? So much is packaged into that which is so valuable as an adjunct to assessments. I don't think we will ever meaningfully get away from having that reflective human dialogue be a critical aspect of assessment 'cause that's about the closest we're gonna get to actually, like, I don't know, bottling up the amazingness that happened in the room. I think all of us, we love doing surgery, which is why we're surgeons and stuff. Magic happens in every room that we're in where we get to do a case together, and I think the narrative comment is like a way to capture a bit of that from educational framing. And you can see it's so impactful to you. I mean, it's impactful to me. It's impactful to Patrick when we have an opportunity [00:26:00] to put that in for everyone. Speaker 2: Yeah. So why is it so difficult? To give meaningful feedback, Speaker 3: so I think I'm gonna separate the two out feedback from assessment. The reason that I think feedback is hard, and feedback being a representation of kind of this dynamic that happens, it's the conversation about what happened, is is trust in a relationship. So we've done this work that's actually created a predictive model using a convolutional neural network algorithm that looks at when people do assessments, and it's when they work together frequently. But what precedes an assessment is a feedback process, and feedback represents trust in a relationship. Like, for me to have that conversation with that person means I'm bought in to them, their wellbeing. Then on top of that, there's doing the assessment. We want assessment to have followed through feedback, but sometimes it doesn't. Some people just jump to the assessment because they actually have hard to have the conversation. They're kind of shortcutting to [00:27:00] just say, "You know what? Let me just write something down." Assessment is putting down on paper. The reason that they're both hard is because they're kind of unique skills, and we're not all always trained in the unique skills. Actually, it would be super interesting to look back at the people who give really feedback, what sort of relationships they have, with the residents around them. I bet the people who give really good feedback from the resident perspective and from their own perspective or the faculty perspective are the ones who are really bonded, who have solid relationships with the residents, who like they go around together, because it's a relationship. Assessment is kind of like test-taking in a way, right? It's the deconstruction of this complex task and being able to reflect it in a form, in a structured form so then we can use it for data analysis. I think it's hard because not everyone is good at those things. Let's be honest. Not everyone is good at giving feedback, and not everyone is good at doing an assessment. But we've made everyone equal, right? 'Cause we wanna get a voice from everyone, [00:28:00] but that means we have variability in the quality of data we're gonna get and how much we're gonna get Speaker: So then thinking about that variability and how good certain people are at giving feedback or even receiving feedback and how that influences the feedback relationship, is there a balance between how easy we can make EPA micro-assessments for our faculty and our trainees and then the quality of the data we can get back? Speaker 3: Absolutely. Unequivocally. I sit on this statement very strongly. You should never compromise the quality of the data for ease of use because you can almost never, I say almost never because you can go back and you have to use a lot of inferential statistics and complex methodology to make your dataset whole, but you will never know if it's truly representative of what was happening. Do not compromise that for ease of use because ease of use can be overcome, right? Ease of use is centered on meaningfulness. Again, we talked about, like, if something matters to you, we will [00:29:00] do the hard, annoying thing because it matters enough, and you can find ways to make complex things easy to do. So we should not compromise on the data for ease of use. And there are entire fields that are built towards making things easy to use. UX design, u- user interface engineering, human-centered design, human computer interactions and engineering. Entire fields dedicated to this, so why would we compromise on this most essential dataset we as a whole country are committed to by favoring perceived ease of use and perhaps compromising on the quality of our data? Speaker 4: So as we look ahead, what is your vision for the future of EPAs, and how do you see Firefly Labs fitting into that picture? Speaker 3: The future of EPA assessments and competency-based education is that it's here to stay. We're gonna grow, and we're getting better at how to do this. And I think that we should never think of [00:30:00] us saying we're getting better as anything short of being positive, right? Nobody gets this right the first time. The goal is we continue to grow with it, iterate and improve. I think what we're doing well with Firefly is really the way we approach this. So we're computational data scientists at heart, understanding that what we do with the data is the most important thing. We collect data to bring forward an understanding of how we learn, to help learners learn better, for us to teach better. And really, without that, I think, co- commitment to computational data science, it's hard to kinda keep this momentum going. What we wanna do through Firefly is really bring the insights to the learners. So education is built to have this trickle-down, just as I was talking about, but the greatest recipient of the learning is the learner. So through Firefly, we are committed from the learner all the way up, but the learner is [00:31:00] at the center of helping you get assessment data and showing you what you can do. But through the science of the work we're doing is actually bringing forward the biggest picture and the biggest story, is actually understanding how we're learning. And I don't think, in all justice, we have actually truly understood how we learn, right? We've just said, "Spend five years in residency. It'll work out." We've built in all these parameters that essentially are the safety net, right? You do X number of cases. You work within an environment where you have appropriate faculty. You have all of the guardrails of didactics, all these things. But beyond that, what we wanna do through Firefly is actually understand how we learn because the landscape changes, and we need a platform, a group, people who are committed to moving with where our learning is actually going. Not just static. This is changing Speaker 2: And we have to ask then, what's the role of [00:32:00] artificial intelligence in all this? You could imagine massive reams of data being collected through audio and video, especially with MIS-based surgery and the quality of that video when it comes to automating or at least enhancing the ability to discern trends and provide meaningful information to both trainees and educators. Speaker 3: Yeah. So I think artificial intelligence, and I'll take this, artificial intelligence being a class of statistical models. These statistical models, these tools are making it more possible for us to actually process all of the data, to make use of all of the data that's around us. We have so much data around us if we look at it and see it through the right lens that is about learning, about the learning that's happening. So Patrick, as you're pointing out, video-based data, audio-based data, the tools we're using, all of these things, they can be [00:33:00] fed into a system that is readily accepting of it, a platform that then pairs it with education data, and all these AI tools and machine learning algorithms can help us to process all of that information. 'Cause it's a lot of information, but why should we not make use of these advanced tools that are growing in our education environment? We totally should. We should be embracing them, but with caution, and through people who actually know how to use them safely. Because data security, privacy, all of these things are important, but we can safely do it. Speaker 2: And any closing thoughts on this topic? I love listening to you talk about this 'cause you have, such a strong passion for it. Speaker 3: Yeah. I Think that we're in probably one of the most exciting times when it comes to education. And what I talked about, learning how we're learning that is the most exciting thing to me. Firefly is a tool that is helping us to achieve what are answering these great questions, because I think when we understand [00:34:00] how we learn- The world is just gonna make more sense to us. We're gonna be able to build better tools for education. So I see all of this as opportunity. I see all of this as ways that we can learn about how we deliver care, educate, and improve ourselves I love this stuff. This is-- it's what we spend so much time of our lives doing in training, in teaching, and I think there's almost nothing more meaningful than understanding how we're learning. Speaker: I think as a trainee too, sometimes it can feel painful to have to try something new or be the guinea pigs, and I think that is a lot of the fear with implementation of EPAs is the unknown. But hearing people like you talk about where this is gonna take us in the future makes a lot of that pain and that work worth it. Speaker 2: [00:35:00] Well, thanks for joining us and everyone for listening. Until next time, dominate the day