Latest / Behind The Knife: The Surgery Podcast / Beyond the Match, Episode 2: Application Chaos, Transparency, and the Cost of Uncertainty
Transcript
- BTK_CoSEF_beyond the match Ep10 pt2_ application process_final === [00:00:00] Hello, hello, and welcome back to Behind the Knife. I'm Mackenzie Roebuck, a general surgery resident at Inova Fairfax, and this is episode two of our COSEF series, Beyond the Match, where we're using the recent congressional report on the match as a starting point to talk about what is actually driving some of the biggest problems in residency. Today, we're focusing on the residency selection process, over-application, signaling, transparency, interviews, and the cost of uncertainty. I'm joined today by fellow COSEF members Melanie, Sarah, and Brianna. Melanie, can you start us off by quickly reintroducing COSEF for anyone who missed episode one? My name's Melanie Milan, and I am currently a first-year resident at the Cleveland Clinic. COSEF is the Collaboration of Surgical Education Fellows, a multi-institutional group of surgery residents and fellows working together on surgical education scholarship and peer mentorship. If you're a surgical education fellow or a surgery resident interested in education and want to learn more, fill out a membership application [00:01:00] at cosef.org. That's C-O-S-E-F.org. I'm Sarah Lunt. I recently graduated endocrine surgery fellowship, and I co-founded COSEF back in 2022. Since then, I have pursued a program of research investigating the residency recruitment process in general surgery. I'm Brianna Stukes. I'm a third-year general surgery resident at UT Southwestern, and I'm completing a surgical education research fellowship while pursuing research focused on AI, simulation, and surgical education. Amazing. So happy to have you all today. So in episode one, we talked about pay, benefits, and bargaining power. Today, we're moving upstream to the application process itself, how applicants and programs actually find each other. One part of the congressional report that I think is worth taking seriously is its discussion of uncertainty and inefficiency in the hiring process. The report argues that because applicants and programs cannot make binding employment commitments before the match, [00:02:00] both sides are left in this fog. Applicants don't know where they truly stand, programs don't know who is genuinely interested, and the report's argument is that this uncertainty pushes applicants to apply broadly and pushes program to cast a wider net. Exactly. So today we're asking what is actually driving the residency application arms race, and how do we make the process more transparent and efficient without making it less equitable? So let's get into it. Beyond the Match, episode two: Application chaos, transparency, and the cost of uncertainty Melanie, is over-application that big of a problem? I think it absolutely is. If you ask a medical student what feels broken about residency selection, one of the first things you'll hear about is applications. And to be clear, when applicants talk about the Match, they are often talking about the whole residency selection process: ERAS, applications, interviews, rank lists, and Match Day. But those are [00:03:00] not all the same thing. The NRMP Match algorithm is one piece of the process, and part of our goal today is to separate what the Match actually causes from what is caused by the broader application ecosystem. But yes, everybody seems to be applying everywhere and with a lot of uncertainty. I think it's important to start by saying that this isn't because applicants are behaving irrationally. Most applicants are responding to a system where the stakes are incredibly high and the information available to them is often limited Right. When applicants don't know where they're competitive and programs don't know which applicants are genuinely interested, both sides respond by casting a wider net. And we can see that in the data. Over the last two decades, residency application volume has exploded. In the 2020-2021 recruitment cycle, when the COVID pandemic prompted the transition to virtual interviews, the average US medical graduate applied to about 70 residency programs, [00:04:00] while the average international medical graduate applied to nearly 140, roughly double the number of applications submitted just a decade earlier. And we have yet to return to pre-COVID application numbers. Think about that for a second. Nobody has 70 dream programs. And I think that's the key point. Applicants aren't applying broadly because they want to. They're applying broadly because they're scared not to. Exactly. The consequences of not matching are enormous. Residency isn't optional if you wanna practice medicine. So when applicants are unsure where they stand, applying to more programs feels like the safest decision. And for some groups, particularly IMGs, that uncertainty can be even greater. And programs are experiencing something similar. Programs don't want to review 1,000 applications for a handful of positions, but they also don't wanna miss out on outstanding applicants. Right. So applicants apply more broadly, programs receive more [00:05:00] applications, meaningful review becomes harder, applicants become even less certain, and the cycle continues. And honestly, if you've ever applied to residency, it's easy to understand why. It can be incredibly difficult to gauge your competitiveness. There often is not a clear answer to the question every applicant is really asking: Where do I actually have a shot, and where would I actually fit? And that advice is not evenly distributed. Exactly. Some applicants have mentors who know exactly how to interpret signals, give advice about program lists or application strategy, or gauge applicant competitiveness. Others are piecing it together from online forums and group chats. So this uncertainty is not distributed equally. Exactly. When people talk about application inflation, the conversation usually starts with why are students applying to so many programs? But after reading the literature, I actually think that's the [00:06:00] wrong question. The better question is what kind of system makes applying to 80 programs a rational decision? Wait a second. If applicants are applying to more and more programs because they're afraid of not matching, have match rates gone up? That is what's interesting. The data does not show that match rates have dramatically changed overall. National match rates have stayed relatively stable over time, and Weisbart and colleagues in 2015 found that the number of applications submitted by medical students did not independently affect match rates. Okay, but if match rates are stable, why does everyone feel so uncertain? Because applicants don't actually know how programs make decisions. Think about it. You know your step score, you know your grade, you know your research, but you don't know how program A weighs those compared with program B. Applicants know what they bring to the table, they just don't know which [00:07:00] table they're sitting at. That makes sense. Without that transparency, it's hard to know where you're truly competitive. Yeah. It's basically trying to hit a target you can't actually see. That's right. In 2018, Nelson and colleagues conducted focus groups with senior US medical students to understand how they approach residency selection. One of the strongest themes was that students could not judge whether they were competitive for individual programs. They wanted programs to be much more transparent about what they were looking for. And a 2016 survey found exactly the same thing. Applicants consistently said they want better information to help to decide where they should apply. Okay, but what about programs? Programs are responding to the same cycle. When applications surge, programs have to manage that volume somehow. Reading every application deeply becomes unrealistic, so programs rely more on filters for shortcuts. [00:08:00] But then applicants become even more worried about being screened out, so they apply more broadly. But there are applicants who really should apply more broadly, right? Absolutely. That's a really important point. International medical graduates, applicants with lower scores, couples match applicants. These are groups for which casting a wider net may genuinely improve their chances. The problem is that everyone sees that strategy and thinks, "Hmm, maybe I should do that too." And once everyone is applying broadly, the advantage disappears. Weisbart and colleagues describe this as prisoner's dilemma. So nobody really likes the system, neither applicants nor programs. Both sides are making the best decision they can with the information they have. Exactly. Individual decisions alone will not solve this problem. If we want different outcomes, we have to [00:09:00] change the rules of the game. Before we talk about direct solutions to high application numbers, we should talk about one tool that has already been introduced in many specialties to make it easier for programs to filter through the large volume of applications they receive, preference signaling. Sarah, this is a lot of your recent work. What was signaling supposed to fix? Preference signaling was created to help applicants and programs communicate interest in a crowded market. Ideally, preference signals allow programs to filter through the thousands of applications they receive to identify a cohort of applications highly interested in their program to focus on for holistic review. More and more preference signals are supplementing and in the case of high volume signaling specialties like orthopedic surgery, taking the place of other application metrics such as clerkship grades or step scores, which historically were used to filter applications. So in theory, it [00:10:00] helps programs sort through the noise. And for applicants, it gives them a way to show heightened interest beyond just applying. Right. But like every intervention in this space, signaling helps with some problems but creates new ones. What kinds of problems? First, applicants have to decide how to use their signals. Should they signal reach programs, programs they are competitive at, programs they think might overlook them otherwise, or places they genuinely love? And that strategy is not obvious. It depends on the specialty, the number of signals, an applicant's competitiveness, what kind of program an applicant is looking for, and how the program actually interprets the signals, which is also evolving. So signaling becomes another thing applicants need advising on. But programs don't all use signals the same way, right? You are exactly correct. Some programs [00:11:00] may treat a signal as the only filter prior to holistic application review, such that if you don't signal, you don't get considered for an interview. Others may use it as one factor among many, while some might not look at signals at all. So signaling was supposed to reduce uncertainty, but if applicants don't know how signals are used best, it can create a new layer of uncertainty. Exactly. And high volume signaling adds another wrinkle. In some specialties, applicants can send a large number of signals, such as orthopedic surgery with 30 signals per applicant. Recent data show that these high signaling volumes function almost like a soft application cap as programs are unlikely to review the application of an unsignaled applicant. Right. So signaling can help, but it's not magic. It's most useful when applicants understand how to use it and when programs are transparent about how they interpret it. Exactly. [00:12:00] Well, we would be remiss to discuss our over-application issue without discussing the advantages and disadvantages of a commonly proposed, quote-unquote, "simple solution": capping applications. On the surface, an application cap seems like it might solve the problem. If applicants submit fewer applications, then programs will see fewer applications. Therefore, they will have more time to review each one, right? Well, that is the general idea. There are some nuances and potential downsides to application caps that make them far from an easy solution. Agreed. While an application cap will decrease the overall number of applications submitted, that decrease is unlikely to be proportional across programs. In fact, the high-volume signaling implemented by other surgical specialties has, in effect, created a soft cap on applications. Recent evidence from these high-volume signaling specialties suggest that signals, and thereby applications, [00:13:00] are not evenly distributed across programs but are instead clustered amongst of a group of programs, potentially because they're highly desirable because of their prestige, location, or educational experience. So even if we capped applications, applicants may still cluster at the same highly competitive programs while some less competitive programs may actually see fewer applications. Interesting. So not only could that hurt programs, as some programs would still receive too many applications while others not enough, but it also might impact match likelihood for applicants, especially those who misjudge their competitiveness. Exactly. Just capping applications isn't quite the easy solution that we want it to be Unfortunately, this is a one-size-fits-all solution for a diverse applicant population. Some highly competitive applicants may only need to apply to 20 programs [00:14:00] to receive enough interviews to have a high likelihood of matching, while other applicants, especially our osteopathic and international medical graduates, may need to apply more broadly to receive enough interviews to match. By capping applications, we limit over-application by highly qualified applicants but also hurt the chances of an important and commonly disadvantaged part of the applicant population. That definitely makes sense, and I'll add that I've even heard from proponents that application caps may force applicants to be more intentional about the programs they apply to. That's a good point and a common one. The problem with this argument is that it depends on applicants and their advisors to be able to accurately judge prior to the interview season not only their application's competitiveness but also what they want or need from a program to be successful. I've done quite a bit of research over the last few years on signaling, and [00:15:00] from our interviews with both applicants and medical student advisors conducted to better understand challenges with the recruitment process, we know that applicants struggle to determine where they fit best. Several commented during interviews that it was the process of interviewing at and learning about differences between programs that helped them figure out where they were more likely to succeed. Further, studies show that applicants and advisors have a hard time judging their competitiveness relative to the hundreds of general surgery programs in existence. So implementing application caps without some kind of system to better help applicants judge both their competitiveness for and fit at programs would only exacerbate these known problems with recruitment. Exactly. Caps without better information are risky, which brings us to the next problem highlighted by the congressional report, the lack of transparency from both the system and programs regarding recruitment. [00:16:00] The report argues that the match incentivizes programs to provide less specific information about what they are looking for because programs want to keep a large applicant pool and avoid missing people who might rank them. I guess I understand the logic of that argument, but I wouldn't agree with that framing. I find it hard to believe that most programs are sitting around saying, "Let's be vague so we can get more applications." Most programs already have more applications than they can reasonably review. What feels more true to me is that transparency is hard. As a recent applicant to general surgery residency, there were many times when I wish there was more transparency from programs. That would have allowed both myself and other applicants to only apply to programs where we were competitive and would fit well. But I recognize that there are real challenges with proposing something as simple as programs should publish their specific screening cutoff. And maybe this is where we need to separate two kinds of transparency. There is [00:17:00] competitiveness transparency. Am I likely to be reviewed or interviewed here? And then there is program life transparency. Is this somewhere I would actually thrive? Precisely. Starting with competitiveness transparency, the solution that sounds simple is publish your cutoffs, but that is much more complicated in practice I couldn't agree more. This actually came up during a panel at Surgical Education Week this year. For program directors, that level of transparency does not always feel safe. It could turn applicants away, be taken out of context, or lead programs to be publicly criticized online. In addition, in a study that interviewed program directors about their application screening processes, it became clear that for some programs, screening criteria are fluid and often based on the applicant pool each year. Program directors wanted to reserve the right to change their screening processes based on who was applying and worried that [00:18:00] publishing cutoffs that could change might mean that they miss an applicant that is a good fit. That makes it almost impossible for every program to publicly post exactly how they will screen applications in advance. That's fair, but transparency is still clearly needed. As we mentioned earlier, applicants and advisors don't know how to judge their competitiveness and fit, which leads to over-application. So what can programs do short of publishing these rigid formulas? One option would be to post historical screening processes. While programs may not do the same thing every year, publishing a few years of historical data may give applicants insight into trends and screening behavior at each program. And for specialties or programs where class sizes are small and publishing matched resident data could reveal too much about the current residents, maybe programs could share aggregate data on applicants who were offered interviews instead. Yeah. Another option would be some kind of centralized third-party system that [00:19:00] estimates an applicant's likelihood of receiving an interview based on past program behavior. That would require a lot of application and program-level historical data, so feasibility is a major concern. But conceptually, it could be especially helpful for non-traditional and disadvantaged applicants who are often applying broadly because they don't have better information. Well, we've highlighted that competitiveness can be hard to make transparent, but being transparent about program life may be a much more accessible place to begin. Exactly. Applicants are not just trying to figure out where they can get an interview. They are trying to figure out where they could train and live for three to seven years. And honestly, one of the easiest fixes is also one of the most practical. Programs should update their websites. Yes. Applicants need basic information that is accurate and easy to find before they apply. I actually updated our program website last year after realizing how many good [00:20:00] things about our program we were repeatedly explaining during interviews, and we've gotten a lot of positive feedback. Things like call structure, rotation sites, operative experience, salary and benefits, parental leave, childcare, fertility benefits, parking, meals, visa sponsorship, resident attrition, fellowship or job outcomes, and what support system actually exists when residents struggle, and not just on interview day but before applications are actually submitted. And that kind of transparency is not asking programs to publish a rigid screening formula or reveal private resident-level data. It's asking them to make the information they already know about themselves current and accessible. Exactly. If applicants cannot find that information, they apply anyway just in case, or they rely on Reddit, spreadsheets, and word of mouth, most of which is unreliable. Better websites will not completely solve over-application, but they are a low-cost method to [00:21:00] decrease the number of applications from trainees who would be a poor fit for a program. Beyond updated websites, a similar app or database that highlights unique aspects of program culture or structure could also help applicants differentiate between programs that may look similar on paper but are very different on the interview day. Some of the current sites like FREIDA try to do this, but they do not always have all the information applicants are looking for. Right. Whether it is an app, database, third-party algorithm, or simply updated program websites, some standard level of transparency regarding program-specific structure, culture, and historical program screening processes would help our applicants focus their applications on programs where they fit best. I'm getting this sense that you already have an idea in mind. Well, someday I imagine a system that allows programs to describe what they value with respect to their program's learning environment, where an applicant can [00:22:00] see that a program values simulation highly versus early operative exposure, or that a program finds APPs to be essential in supporting junior resident learning versus a program that feels APPs can detract from the development of essential junior resident skills. Such a resource could even look like an app where applicants can look at the specific values and experiences at a program to assess their fit before applying. That would be awesome. I think such a resource should be based on a mix of program leadership, attending, and resident perspectives. I love it. I hope that in the future our national organizations involved in recruitment such as APDS, ERAS, NRMP, AAMC, could come together and come up with this kind of resource or update ones that already exist. I know I wish I had an app to judge my fit back when I applied to residency. Okay, so I'm not saying I agree with this, but I know one proposed solution to over-application that sometimes comes up is return to [00:23:00] in-person interviews. Before virtual interviews, applicants generally submitted fewer applications and attended fewer interviews. Melanie, do you think moving back to in-person interviews would help us with this? Well, I think there's definitely some truth to that argument. Virtual interviews removed many of the barriers that naturally limited interview volume. When applicants no longer had to worry about flights, hotels, missing rotations, and travel expenses, it became so much easier to interview more broadly. I also think it's important to point out that COVID may have exacerbated application inflation, but it did not create it. Application numbers were climbing long before virtual interviews existed and are unlikely to decrease even if all programs magically transition back to in-person interviews. And it makes much more sense that a transition to in-person interviews is more likely to decrease interview hoarding than the number of applications being submitted. I couldn't agree more. The number of [00:24:00] applications submitted and the number of interviews attended are related, but they are not the same problem But I think we have to ask ourselves what problem are we actually trying to solve? Exactly. If the goal is simply to reduce interview volume, then yes, moving back to in-person interviews would probably accomplish some of that. But if the goal is to create a more efficient and equitable residency selection process, I'm not convinced that's the answer. For many applicants, especially those with financial constraints, caregiving responsibilities, or those applying from further away, virtual interviews dramatically improved access. And we've actually seen a number of general surgery and other specialty programs start bringing back in-person interviews. Many of these programs have implemented policies to decrease costs for applicants, such as covering the cost of applicants' hotel stays and their food throughout their visit. My residency program recently transitioned back to in-person interviews during my chief year with these [00:25:00] cost-covering policies, and overwhelmingly, we felt that we could showcase our program a lot better in person. As one of the old guard that interviewed for residency in person back in the day, I can also honestly say that there were definitely benefits to seeing a program in person, meeting residents face-to-face, and getting a better feel for the culture, both of the program and of the city I would be living in for five to seven years. Right. I think there's tremendous value in in-person interactions, though. I just don't know that reintroducing a cost and travel burden solves the underlying problem. Instead of making the entire process expensive again, we need to be more intentional. If program use in-person interviews, they should have serious cost mitigation strategies. If programs use second looks, they need clear rules about whether attendance is tracked or considered. And if programs lean on away rotations or open houses, we have to acknowledge those can be even more expensive and inequitable than [00:26:00] interviews. So the question isn't necessarily virtual versus in person. It's how we can give applicants and programs meaningful information without making the applicants responsible to buy that information through travel, time off, and money. How can we reduce uncertainty without sacrificing accessibility? Exactly So it sounds like application caps or interview caps might actually make things worse for some applicants, especially those who truly benefit from applying broadly. If that's not the answer, what does the literature suggest we do instead? There is one thing that almost every study agrees on is the goal: preserve holistic review. The challenge isn't convincing programs that holistic review is valuable. It's giving them the capacity to actually do it. Holistic review remains the best way to evaluate the whole applicant and to help applicants and programs find the best fit. But [00:27:00] how can you holistically review thousands of applications? I mean, you can't really, not with the current support available to program leaders. To do this, institutions would need to increase the resources given to holistically review any number of applications they receive. This would include dedicated paid time to review applications, probably from multiple staff members. This would also include more training or guidance on how to truly holistically review applications Okay, great. So institutions should adjust the time and resources given for this monumental task of reviewing their hundreds or thousands of applications. That makes far too much sense, and I'm sure it's much more difficult than it sounds. What other types of things could we do? I mean, as we talked about earlier, some transparency is about basic program structure, call, benefits, rotations, and support. But another piece is about selection [00:28:00] itself, programs being more clear about what they are looking for in an applicant and how their program stands out from others. And if we can give school advisors better information on how to guide their students, this could help reduce applications that were unlikely to be a good fit in the first place. That creates more time to thoughtfully review the applications that remain. So earlier we talked about why programs can't always be completely specific about how they screen applicants. If programs are doing more holistic review, what could they realistically tell applicants upfront? Well, that's where some of the newer work becomes really interesting. Transparency doesn't require every program to look for the same thing. It requires every program to clearly define its own criteria upfront so that applicants know what they're looking for. Programs may not be able to publish an exact step cutoff or formula that applies to every year, but they can be clearer about what they value [00:29:00] in holistic review. Service, leadership, research, community engagement, operative readiness, commitment to a region, interest in academic surgery, or whatever really matters to that program. Several groups have shown that holistic review can be made much more structured by developing mission-specific evaluation rubrics, training reviewers, and using standardized scoring systems. Instead of asking each reviewer to just read holistically, they create reproducible frameworks that reflect what each program actually values. So not every program needs the same rubric, but most programs could be clear about what their rubric is trying to capture. Exactly. That's a win for both sides. Programs get more consistent review process, and applicants can make more informed decisions about where they truly fit instead of applying everywhere. This sounds like a perfect solution, but I'm guessing it's a lot more work than it sounds like. Building [00:30:00] rubrics, training reviewers, validating the process It most definitely is. And that's why several studies argue this has to be treated as an investment. This has to include institutions dedicating real time and resources to it, rather than just being an added burden. If we believe holistic review is the gold standard, then we have to invest in the infrastructure that makes it possible. Well, I'm sure a lot of people are wondering then if AI has a place in this process. That's actually one of the most exciting areas of current research, and I think it's often misunderstood. The goal isn't to have AI choose residents. It's to give programs more capacity for holistic review. So several groups have developed AI tools that organize applications, summarize narrative experiences, identify mission alignment, and even help reviewers find applicants who could be best for a closer review. Exactly. AI isn't [00:31:00] replacing holistic review. It's just making holistic review possible. That's right. But there is an important caveat. AI can only reinforce the values that the programs intentionally build into it. If a program has a very thoughtful, mission-driven review process, then AI can help scale that. But if the underlying process is poorly designed or heavily biased, AI simply scales those problems too. Some studies have even shown that AI could worsen these biases. Technology clearly isn't the solution by itself, but it is a tool that can help us make a better system. Between all these different solutions we talked about, it's nice to know we aren't completely doomed at least. I agree. At the end of the day, the goal isn't just fewer applications. The goal is helping the right applicants find the right programs for the right reasons. All right. We went through a lot here. To wrap it up, I think the big takeaway is that the report is right to identify [00:32:00] uncertainty and inefficiency as major problems in residency selection. But our argument is that the application process arms race does not begin or end with the match. Applicants don't know where they are competitive, so they apply broadly And programs don't know who is truly interested, so they rely on filters, signals, and imperfect shortcuts. And every solution has trade-offs. Application caps, signaling, transparency, virtual interviews, in-person interviews, none of them are perfect, and all of them have equity implications. So again, the goal should not be just to have fewer applications. The goal should be better information, more transparency, and a process where applicants and programs can make more informed choices. Because if we only reduce application numbers without reducing uncertainty, we have not solved the problem. We've just moved the risk around. But even if we fix the application process, there's [00:33:00] still the question of what happens after you match. In the next episode, we're gonna talk about working conditions, accountability, and mobility, what options residents actually have when they are overworked, mistreated, or stuck in a harmful environment. And as always, thank you for listening to Behind the Knife. Donate the day