Latest / Behind The Knife: The Surgery Podcast / Perfect Practice Makes Perfect - Our AI Simulator is Kicking Ass
Transcript
- Board Simulator Update 2.26 === [00:00:00] Yes, we are in fact talking about practice here. Oral boards practice to be specific. There's an old adage about practice. Surgeons know very well. Practice does not make perfect. Only perfect practice makes perfect. So do we think this applies to the oral boards? Absolutely. This is in fact the most stressful exam you're gonna take, in your life. And, we aren't messing around when it comes to oral board prep. We've been in the lab cooking, building, you know, perfecting what we're working on. And when it comes to the boars, you first have to master the material and we've been all over that for years. Behind the knife now has 123 audio scenarios. All of them updated recently. Uh, they all have detailed commentary. We have 10 interactive video scenarios and we've created 97 operative descriptions. But guess what? You also have to master how to take an oral board exam. And this is an active process. Uh, you've, , heard the recommendations from your colleagues. You need to be clear. You need to be [00:01:00] concise. You need to keep pace, be confident, pivot gracefully, demonstrate that you're a safe surgeon, right? No big deal. Easy peasy. Uh, no, it's not wrong. It is a big deal and it's really difficult to prepare. So, this episode's all about the oral board simulator, so it's 2026. AI is everywhere, every day, every week there's an exciting new AI application to explore and some change the way we live. Others change the way we work, and now AI is changing the way we all prepare for the oral boards, and we've made some significant improvements to the simulator that we want to talk to you about today. Including fundamental changes to the format of the exam delivery, which matches changes for the American Board of Surgery, including rapid branching and less linear scenarios. And we've moved multiple specialties from beta to the real deal, and the simulator is now integrated into our website. And more importantly, or Android and iOS apps. And today I'm joined by our AI team, Dr. [00:02:00] Matthew Swenson is a general surgeon and US Air Force veteran. He works at Valley Health System in Vegas where he is the Associate Program director for the residency program there. Dr. Ayman Ali is a general surgery resident at Duke and he is our BTK AI Fellow. Thanks, Dr. Gi. And first we have to say a massive thank you to everybody who took the time to use our simulator in preparation for the last oral board exam. In total, believe it or not, over 10,000 exams were completed and the graduates who used it and took the November general surgery oral boards, gave us feedback and provided us with incredible information. After the exam was done, we found that 99% recommended it to other residents. And 85% gave it a four or five rating as a resource. But the most important part was the qualitative feedback we got, which we've used to refine and enhance the simulator. It's also inspired a whole new mode and a release of free practice written exams. But before getting ahead of myself, we should backtrack a [00:03:00] bit. First, Dr. Geoff, I think it would be helpful to talk about who the simulator is meant for. Yeah, absolutely. Well, it's meant for everyone. But we're, we're really thrilled to have expanded beyond general surgery. So Dr. Uh, Swensen has been absolutely cranking and we now have simulators for general surgery, cardiothoracic surgery, colorectal surgery, pediatrics, surgical oncology, vascular, and even O-B-G-Y-N. Well, thank you to both of you. Uh, as, as Ivan mentioned, uh, we now have tens of thousands of completed exams in our database. We've taken that raw feedback to make these simulations even better. Uh, we've been comparing them closely to what the specialty boards are testing, and we've been asking experts in all these fills to stress test the content to make sure that it's relevant to what you were doing. And practice does not make perfect. Only perfect practice makes perfect, and the simulator does an incredible job of mimicking the real thing. It's impressively high fidelity, uh, and many [00:04:00] residency programs, but definitely not all of them will offer mock oral exams, , at some point during the year. And if you've completed a mock oral exam, you'll know just how incredibly valuable these experiences are. You learn so much about yourself and where you are really at in the preparation journey. And every resident gets exposed during these mock oral exams one way or another, and some more than another. It can be a knowledge gap, , but often it's about how they perform. And the next best thing is practicing with a mentor or a colleague. But we all know how difficult that is to schedule. You might get a few sessions in before the exam. But it's definitely hit or miss, and that is why this simulator is so incredibly important. We have unlimited, high fidelity scenarios available 24 7 at a very, very reasonable price. Yeah, the, the whole point of this, this was designed for accessibility. I've heard from surgeons now who are using this while driving home, waiting for a case to start while they're in the operating room or even lying in bed before sleep. We've even seen programs use it for formal [00:05:00] mock oral exam days, and, and it's not just for residents. Uh, attendings are using it to hone their own practice. A real fun story is our very own. Dr. Canary here at Behind the Knife, took a vascular mock exam and told me that it got his heart rate up. Like no skills lab or Eminem conference ever has. It really is for any surgeon at any level. Yeah, and I, I will say as someone who just recently finished junior residency, um, I've found substantial benefit using it. I mean, I've become much more clear and precise when I speak, which has made a lot of things, uh, much better in my clinical experience. Well, so Iman, on that line, we wanted to make sure that cost wasn't a barrier to using the simulator early in your training. Now that we know what it costs for us to run and maintain these simulators, we've switched to a credit based system. We only debit your account once you complete an exam and have that exam graded. So a full realtime mock oral exam, including all the feedback from our [00:06:00] performance agent costs just $2. And the proctor feature that we'll discuss shortly is only 50 cents in the test prep world. This is an unbelievable value. Yeah, Matt? If you just go out there and compare the cost of these things, uh, we're very proud of that. Uh, it also levels the playing fields. Like I said, some programs offer mock orals, but many don't. And for all the program directors and associate PDs out there. The oral board simulator can be used to lessen that massive workload that comes with setting up and executing mock oral boards. For your program, especially when we talk about free exams, we're going to, have a link to those in the show notes and, we'll talk in a minute here about how those exams can be used with this exciting new function we call proctor. Well, couldn't agree more, Dr. Geoff. Let's, let's review all the, the functions on the simulator, including Proctor. , It's helpful to think of the simulator as an ecosystem divided into three phases. There's testing, there's analysis, and there's practice. For the testing phase, you've got those two main, uh, AI driven paths. Real time. As I mentioned, that's the hot [00:07:00] seat. You have a conversation with one of 30 different examiners, no pausing. You, you talk. It listens and responds. We use what's called conversational AI technology to administer this exam. Uh, so this is designed to feel like the real deal, no pausing or taking a break. Self-paced is the same test structure, but in a chat based text format like you're used to using with chat GPT or Gemini or Claude or any of these others. You type or speak and you get a written response. It's perfect for a loud or, or a quick break. Second phase is analysis, which is handled by our performance agent. Uh, you finish a test performance, grades it, and provides incredibly detailed feedback on every answer, and now even links to PubMed articles for you to review. So you can really hone down on that con uh, that test material. Wingman and coach are our practice mode, uh, with win. If you're stuck, it answers for you. Great way for a junior resident to tackle topics they aren't comfortable with yet, uh, or they're feeling stumped by. And coaches, it's a coach. It tracks your [00:08:00] progress over time. Recommend strategies. You can even discuss your answers and analysis with coach to learn the why behind your performance grades. Yeah, and real quick on that, these features, uh, taken together make the simulator highly useful, uh, for junior residents as well. Just like Iman mentioned. , We all know the oral boards. They are a legitimate exam. I think they're the, the truest exam you're gonna take. And that means that using an oral board prep as a junior resident really helps your clinical performance . I see this especially with consults when it comes to putting forward a really awesome assessment and plan. Yeah, on the consult note, when you use the simulator a lot, you get used to thinking about the what ifs in advance. And that's something that, uh, like you said, is very useful in a consult scenario. So when the attending asks me what I want to do next, it's readily available. And I will say that for this simulator, real time is my favorite. It's what I use the most and it's what most people use the most, um, to practice for those oral boards. I think though the most exciting thing for the future that has recently been added is [00:09:00] Proctor. I think that it has great potential for all programs and at all levels of training, especially in the case for people who are not quite chiefs. They're not getting that same attention for oral board prep , but for people that are a little bit close to it, that need that additional practice. So, Dr. Swanson, what inspired you to develop Proctor and maybe what is it first? Well, I saved the description for this last because I believe it will soon become our most important tool for test preparation. Now, the problem with any simulator is that it isn't the real thing. Think of your own training at Duke Imon. Uh, the skills lab is vital, but you have to get into the OR to learn surgery, you need to retract for Dr. GI off. You need to observe him and then try it yourself while he watches you. Now, no training tool, videos, podcasts, even this AI simulator can replace the experience of sitting in front of Dr. George while he takes you through a trauma scenario. So we created proctor. You find a mentor, place your phone between you and press record. [00:10:00] Proctor will transcribe the conversation, send it to our performance agent for the same in-depth analysis you get from other simulators. All this for only 50 cents. Uh, but we didn't stop there. 'cause the other challenge with mock oral exams is the exam itself. I know this is a program director or associate program director. Where where am I supposed to get good exams for my residents to practice? So to address this, we are writing free mock oral exams, written and edited by the same experts you hear on this podcast. These are free to download, no subscription required, no more trying to find high quality test material on your own or to make it up. Uh, we currently have over 20 and we'll be releasing more over the next year. Getting high quality mock exams for practice has always been challenging or expensive in the past, but not anymore. Circling back around to the program directors and associate PDs out there. Behind the knife has free exams and a comprehensive way to evaluate resident performance all just for 50 cents per exam. So you don't really have to [00:11:00] recreate the wheel, you don't have to slave way writing new exam topics twice a year. Uh, let alone on topics you might know a little about, uh, begging and pleading staff to provide reasonably detailed feedback, working hard, uh, to get over that logistical hump of getting everyone's scheduled, people time off, et cetera. The list goes on and on. And I think what's also really important is we mentioned, I think Matt, you said have a mentor walk you through it. Well have colleagues walk you through it. It's a lot easier to schedule something with one of your co-residents. Sit down, do a few exams, and from the administrative side, , education leadership will be able to look and see, how have you performed. And using those metrics, you can build a program that readies a residency in a way that, uh, currently doesn't exist. Yeah, this is incredibly exciting and I personally hope that Proctor and the free exams are incorporated into our own oral board prep here at Duke, where I just grabbed my co-resident and we just. Do an exam. And the other part of this, and Dr. Gird off hinted at it with the, uh, with the progress in the [00:12:00] evaluation. What we got from the feedback was we realized we weren't giving people a good enough metric to understand what their scores meant. So in other words, what exactly did it mean if you got a score of 75 on the oral board simulator? So what we did in response was decided to implement percentile scores, which are based on the users who took the oral board exam last November. And hopefully this gives people an idea of how they're doing and tracks their progress over time. It's a little dependent on how you grade exams, but over time and over the grand scheme of it, it should be very accurate. Yeah, that's right. Because of all the general surgery exams or users have taken, we can now give you that percentile score to show you how you compare to a board Eligible surgeon who is actively preparing to take their exam, gives you a much better picture of where you are in the preparation. So what does the score of 75 actually mean? Well, percentiles can help, you know, you may be in the 50th percentile, 90th percentile, or 10th percentile. And I found the percentiles to be fantastic and I've also noticed a specific change [00:13:00] in the CLARITY score and I've noticed my own scores get much better as I get more comfortable. Dr. Sw, can you talk a little bit about that clarity? Yeah. We actually updated that specifically Iman. Uh, we learned from our feedback and analysis that people highly value the way the simulator gives them a way to practice talking surgery, which is not the same as doing surgery. So based on this, we made updates to how we calculate the clarity score to better reflect the way you give an answer and articulate your decisions. What was, uh, especially fun for me, actually, was to use this update to analyze your performance. Simon, over the past few months, uh, you've made some significant improvements in the way you answer questions and you speak with much more confidence than you used to. So we're gonna continue to work on improving this metric so you can really understand your own data and progression. This very specific and unique skill. Yeah, and I made mention of this earlier, but something else has really changed with the exam in recent years. It's much more a branching. Uh, much more non-linear. And Dr. [00:14:00] Swensen took this feedback, uh, about these changes and he's retooled the simulator to reflect it. Yeah, frankly, I was a bit shocked at the feedback we received about this. Uh, we did a survey after the exam and we asked everyone, you know, how does the simulator compare to the real test? Um, and the feedback that came back made me finally realize what my graduates have been telling me that the exam is different from the exam that I took 15 years ago or so. Uh, there's lots of, lots more branching, lots of what ifs, lots of challenges to the answers you give. Uh, because of all this feedback, we made changes to our generated tests and the mock exams that we've written. Uh, the advantage of this technology is, is clear we can update it almost instantly. Unlike other types of content, our goal is complete preparation and no surprises on game day practice does not make perfect, only perfect practice. It makes perfect. I think that one of the biggest strengths of this is the ability to instantaneously adaptive feedback. So as people [00:15:00] use it, as people experience challenges with it or, um, positives, then we would love to hear about it because it really helps us improve and make it better. Yeah, AI is here to stay and at behind the knife we are approaching all of this. With an enthusiasm unknown to mankind, and it pains us into our hearts to see trainees spend well over a thousand dollars on board prep in 2026. We're not okay with that. We got into this space years ago because Kevin, John, Jason and myself, we spent thousands and thousands of dollars preparing for our boards, and that was for a bunch of lame content and that wasn't okay with us. So behind the knife board prep is. Without a doubt, the best value out there. It's the only complete package. We have 123 audio scenarios. All of them recently updated all of 'em with detailed commentary. We have 10 interactive videos, 97 operative descriptions, and now an oral board simulator with all these incredible features we've been talking about today. It is [00:16:00] the entire package that baseline knowledge. And the practice that you need to pass the boards. So before you drop thousands of dollars on other programs, uh, we really encourage you to at least come visit our updated board review page and learn about all the different features that, we're talking about. And in the process, you can save many, many hundreds of dollars. Yeah, and to the program directors out there, please give this a try. We're happy to provide free credits to you and your residents. We also encourage you to take our free mock oral exams, combine them with the power of proctor and administer a mock oral exam day for your senior resident. Our goal is to remove every barrier that might prevent your residents from succeeding no matter what your program resources are. I completely agree. Uh, as I've mentioned before, the whole reason I created this simulator was for my residents. I, I was just so disappointed when my early graduates reported difficulty with the oral board exam after doing so well in their residency. In just the one year since I [00:17:00] started this, I've noticed a vast improvement in my own resident skills, and best of all, I'm happy to report that every one of my graduates who took the test in November passed. There's no better reward as an educator than to see your trainees succeed. On that note from everybody at Behind The Knife dominate the day and dominate the boards.