Latest / Off the Chart: A Business of Medicine Podcast / S1 Ep150: Shadow AI: It's already in your practice, with Asha Palmer, J.D., of Skillsoft
Transcript
- 0:00 Physicians face challenges every day, from difficult patient encounters to changing regulatory requirements at Copic, our commitment to you goes beyond providing reliable medical liability insurance. What sets us apart is our proactive approach, a 24/7 hotline answered by physicians, specialty specific guidance and CME accredited education and unwavering dedication to help you thrive. C, o, p, i, c, com, we're Copic here for the humans of health care, here for you. 0:27 Banning the use of AI is not a sustainable strategy for anyone, because either I'm going to use it on this little handy device here, you know, my cell phone, or I'm going to use it in a way where you can have some visibility on how I'm using it. Welcome 0:50 to 0:50 off the chart, a business of medicine podcast featuring lively and informative conversations with healthcare experts, opinion leaders and practicing physicians about the challenges facing doctors and medical practices. I'm salsa Luttrell. I'm the associate editor of medical economics, and I'd like to economics, and I'd like to thank you for joining us today. In today's episode, I sat down with ASHA Palmer, Senior Vice President of compliance solutions at Skillsoft, to talk about shadow AI, the unsanctioned tools that your staff are probably already using without you knowing. And why banning AI entirely isn't the answer. Asha explains what a practical governance plan actually looks like for a smaller practice, which risks most organizations are under estimating, and why the conversation with clinicians has to come before the policy does. Asha Palmer, thank you for joining us, and now let's get into the episode. Asha Palmer, thank you so much for joining me 1:42 today. 1:42 Yeah, glad to be here. 1:44 I guess just before we start, could you tell us a bit about yourself and a bit about Skillsoft? 1:50 Yeah, how long do you have? Just kidding, my name is Asha Palmer, and I'm a lawyer by background, a compliance professional who is now in the technology space, because I believe that technology has obviously a lot of power and a lot of opportunity that it can help compliance professionals have more effective and efficient programs. And that's one of the things we help do at Skillsoft. We're a learning company, and we provide compliance learning metrics, analytics really a defensible, scalable compliance learning program that the regulators are looking for. Yeah, 2:33 you touched on it there. I mean, technology AI is absolutely everywhere, and especially in healthcare. So with that, Shadow AI is becoming a real concern clinicians using tools like chat, GPT or Claude or other, you know, consumer facing AI tools in their day to day workflow, and that's outside of official systems. Could you kind of, it might seem obvious to some, but could you kind of break down the problem with using these unapproved or shadow AI tools, and kind of, what are the risks that that creates? 3:03 So I love Austin to talk first about the opportunity that those tools create, which I think is why we see clinicians using it, right? I mean, no one wakes up and says, I have enough hours in the day or I don't. I'm not overloaded in my daily job. So we know that the opportunity that AI presents is effectiveness, is efficiency is a thought partner for many of these clinicians, who may feel like they're on an island by themselves, or they've seen something they don't quite understand, and so they're using it with the greatest intent to create, you know, efficiencies and opportunities in their care delivery models. I think the challenge with that, right, if, if their organizations haven't already embraced the same attitude that AI can have those opportunities and those efficiencies, is you get to the shadow use, which is, you know, people using it in the shadows that no one knows about, but they need it as a companion in their work, and so obviously the challenge is, what's happening to that data? Where is it going? How much can you trust it? Are we using models that are actually trained to support the delivery models that these clinicians are actually providing, and we don't have those answers when we're letting clinicians sort of cherry pick whatever tool they decide to use or they enjoy using. And obviously, I think the huge risk right is, what's it doing with the data that you put into it, and is it protected? And is your patient's data protected. So I think there's a little bit of input concern and output concern in the use of this shadow, AI, and when it's being used in the shadow, right? I mean, you can't see it, and so there is just a lack of knowledge and visibility for the organizations as to what it's being used. Use for how it's being used and where that data is going 5:03 definitely. I mean, you know, you kind of touched on it there, but is there a reason why? And I guess it could just be, you know, the tools have different uses. But why is it that different clinicians are finding or different health systems are finding clinicians kind of repeatedly drawn to these shadow AI tools that are outside of official systems. 5:24 Well, you know, we find this in healthcare and beyond, right? The number one reason people use shadow AI is because the organization has not sanctioned the use of AI in the delivery of their services. So whether it's a clinician or lawyer or otherwise, we need organizations to really step up and say, here are the use cases we know you may want to use it for. Here are the efficiencies and opportunities we know you can gain, and then really creating a governance structure around how you are actually able to use that so where we see the biggest gap, and the largest use of this shadow, AI, is where organizations don't have a risk appetite or risk tolerance level for these applications, and so they tend to ban them wholly. And so people go behind and use them 6:20 great. And I guess we can kind of jump ahead there in terms of of what that, that governance kind of should look like. I mean, ideally, I guess, what does it for a smaller or independent practice a lot of our audience, I guess, could you kind of walk through the basics of how to create that kind of governance plan and what it should ideally look like, 6:44 yeah, should I ask you again, Austin, how long do you have? You know, we lawyers and compliance professionals can over complicate governance, right? But I promise I won't do that today. You know? I think it actually can be quite simple. And I think it's actually more simple for some of these smaller practices that you know have visibility into their workforce and have the opportunity to bring their clinicians to the table. And it starts with bringing people to the table and saying, here's how we know you can use it. Have we covered everything? Right? So first thing is to establish use cases. And I always say to organizations how people are already using it, how people want to use it, and then how the organization wants you to use it, right? And I think those are three separate buckets that you really have to define, and then you have to think about, so what are the risks associated with those use cases, right? So obviously, like exposure of patient data, inconsistent, you know, clinician outcomes, you know, sort of get the laundry list of possible risks associated with those use cases. And what we find with most of our customers at Skillsoft is that the risks actually aren't as far and wide as people think when they're thinking about these use cases in a vacuum, right? What are reasonable risk cases? I mean risks associated with these use cases that we can do something about, right? So now you have your use cases, you have risks associated so the question is, how do you mitigate and manage those risks? And that is something for the risk professionals in the organization, the leadership and even the doctors or clinicians themselves to participate in what are we going to do about this? Right? And so is it, we as a company are going to procure private instance of this. We as a company are going to have oversight over, you know, your your inputs into these, what are those sort of controls you're going to put in place that say, okay, based on those controls, we're really comfortable with you using it. Now, I will say in every governance structure you also have to have some testing and monitoring, right? And so as you sort of let the AI out the bottle, right? Let the genie out of the bottle. You also have to then go check is it working in the way we intended? Are the risks that we were concerned about actually being managed? Are they a little bit lower than we thought? Are they a little higher and really then recalibrate your governance structure based on that? 9:16 Are there any kind of, I guess, unexpected risks, or things that some systems might not really think about when it comes to shadow AI use. 9:27 Well, I don't know that it's unexpected, but I think really in the clinical environment, right? The there's, I keep saying this, AI is still very much in its infant stage, right? Particularly generative AI, it's still figuring itself out, right? It's like, Oh, these are my arms. Oh, those are my legs, you know? And so I think if you think about it being in its infant stage, there's still a lot of mistakes that it's making. And so I think from a healthcare delivery standpoint, the biggest risk probably is. Patient data access, I think doctors have got most of that down by now and and nurses pray, but I think it's inconsistency of outcomes and accuracy of outcomes or recommendations. And so how do you really guarantee that it is the outputs of AI are aligned with the care model of the organization, and I think that's sort of the biggest risk, right? Is it hallucinating? Did it make up, you know, something that you know doesn't exist? I mean, I looked at looked up something the other day on AI, and I knew it was wrong. And I was like, This is wrong. This is not wrong. This is wrong because this homecoming couldn't be in this city, and then I had to follow some of the sources, and it ended up being inaccurate, but it was drawing summaries from the data that just created an inaccurate conclusion, and that is a risk, particularly in healthcare. You 11:00 Keith, 11:05 hey there. Keith Reynolds, here and welcome to the p2 management minute in just 60 seconds, we deliver proven, real world tactics you can plug into your practice today, whether that means speeding up check in, lifting staff morale or nudging patient satisfaction north. No theory, no fluff, just the kind of guidance that fits between appointments and moves the needle before lunch. But the best ideas don't all come from our newsroom. They come from you got a clever workflow. Hack an employee engagement win, or a lesson learned the hard way. I want to feature it. Shoot me an email at kreynos at mjh, life sciences.com with your topic, quick outline or even a smartphone clip. We'll handle the rest and get your insights in front of your peers nationwide. Let's make every minute count together. Thanks for watching, and I'll see you in the next p2 management minute. 11:56 There's a tendency in healthcare to kind of treat AI as an IT problem. How do you kind of make the case, I guess, to a practice leader or a practice owner? This is actually a compliance and workflow training problem. 12:11 Yeah, you know. So I think what we look for it to do is procure systems a lot, right? And then monitor them so you know when, when it says you can't paste your organization data here, right? That's it, telling you that when we're pushing out phishing, you know, simulations, it's it. But a lot of that is reactive. And I think there's always an opportunity for reactive approaches to, you know, governance. You need that testing, that monitoring who's, you know, the Big Brother looking over, you know, the company's shoulder, and it can play that role, but there's such a powerful enablement and opportunity standpoint from AI, and that's why I encourage everyone to look at the opportunities that AI can provide your organization. And so practice managers who are responsible for growing their business, and, you know, looking for new business, and you know, having faster turnaround times with the patients that are coming into the business, those are use cases that AI can assist in. So when the practice manager is looking at AI, it has to be looked at from how is this creating business opportunity for my organization and accelerants that I really want to leverage? Then I need compliance to help me make sure I'm leveraging them safely. And then I need it baby to procure those tools and make sure that the way that it's actually being used is safe, so we always look at it as a multi disciplinary opportunity or problem, depending on how you look at it, but everyone has to be at the table when we're deciding how we as an organization want to use this to accelerate but not to make us vulnerable. 13:59 When a new AI tool does get formally adopted by a health system and as part of that governance plan, whether that be an ambient scribe, I know that's a big, a really big topic right now, or, you know, something that helps in in diagnosis, what compliance infrastructure needs to be in place before it goes live and before it goes out to to the clinicians? And is there something there that organizations might tend to kind of gloss over or skip? 14:25 Yeah. So, you know, in compliance, we are very passionate about what we call third party due diligence, right? And it's a, you know, really fancy, not fancy word or phrase, right? But what it means is you got to check out the people you're doing business with and the companies that you're doing business and you got to ask them questions about how they are creating their product, how they're creating their technology, how they're training their models. And so one of the things that we really, really encourage is as. Your onboarding vendors. As you're thinking about onboarding vendors, ask them tough questions about where your data is going, how the models are being changed or trained. Are they monitoring for fairness? You know, what is the accuracy rate? I mean, you talk about right the scribes, the question is, how accurate is the output that you get from it, and are they testing for that, or are they looking for that? So what is their governance structure that you can rely on that helps assist you in your governance structure? And so we really encourage people to ask those tough questions of their third parties, of their vendors. And we're a vendor ourself, right at Skillsoft, and we want our customers to ask those tough questions about how we're training our models. What is the output that we would pull something back because we're not happy with? Are we testing for bias? Are we testing for fairness? Are we testing for, you know, a bunch of other things. And so I think it's very important before anything goes live to have that really diligence process and for everyone in the organization to be comfortable with the answers that you get from those vendors. 16:09 If a physician, a practice leader came to you and said, You know, I know my staff is probably using AI tools that I don't know about, what should they do first? I guess, what kind of advice do you have for them? 16:22 Fire them? Just kidding. No way. I mean, I think right now there's still a huge opportunity to have a conversation with those clinicians to say again, if it's been banned, if it's been, you know, prohibited on a company wide basis, and they're using it, there may be appropriate consequences for that right, but I think there's still such an opportunity to say, why are you using this? What are we not providing you that you need, or you feel is so necessary to deliver your standard of care that you need this tool and that listening to that clinician is so important right now, because, again, I do believe that most clinicians have the standard of care that is required of their profession. And right, there's all kind of, I'm a lawyer, so we have the same things, right? There's all kind of ethics, you know, requirements, etc. But I think if it goes beyond that, to the point where they're using something they're not supposed to ask the question, why, and be receptive to their answer to that question, and see again, where you can help them do the things that they need to do within a more safe way. In your last question, you asked sort of infrastructure I talked about just, you know, asking your third party vendors. The other thing you know we love in compliance are policies, policies that no one reads. Okay, I'm just kidding. You have to read the policy. But I do think right the policy of safe, accountable and responsible use of the AI tools that the company allows is also a very important step to have before any kind of AI goes live. And this second conversation that you have, if you discover shadow use of AI is an opportunity to revisit that policy. If you have one to say, What am I missing here that I may need to expand my viewpoint, my horizon, the organizational viewpoint, to really make sure that my clinicians feel like they have what they need to provide the care that they need. 18:38 Is 18:38 there anything else that you want to share with the audience of primary care physicians practice leaders. Do you think is important that they know about this topic? 18:47 I would just love to share that using AI is inevitable right now, and we're using it in our daily lives. We're using it in our professions. We're using it to create analysis. So providing a safe, responsible and accountable way for your clinicians to use AI is the best route to go. We encourage all of our customers. Banning the use of AI is not a sustainable strategy for anyone, because either I'm going to use it on this little handy device here, you know, my cell phone, or I'm going to use it in a way where you can have some visibility on how I'm using it. And right now, the most important thing that practice leaders can do is have visibility on how their employees are using AI because that will then give you additional use cases, that will give you opportunities. It may give you data that you can leverage, and so providing that sort of sanctioned tool will give you that visibility that you need to really create that proper governance structure if people are using it in. The shadows, you're losing control of your data, you're losing control of your people, and you'll eventually lose control of your practice. And so nobody wants that. So really embracing it, talking to your clinicians about how they want to use it, could use it, are already using it, and then really thinking about, okay, how do we protect ourselves as individuals and our organization against the harms that may come from those use cases, then you'll be good. 20:30 Great. Also. Palmer, thank you so much for taking the time today. 20:33 Yeah, awesome. Thanks, Austin. You 20:40 as once 20:45 again, that was Asha Palmer, Senior Vice President of compliance solutions at Skillsoft, on behalf of the whole medical economics and physicians practice teams, I'd like to thank you for listening to the show and ask you please subscribe so you don't miss the next episode. As always, be sure to check back on Monday and Thursday mornings for the latest conversations with experts, sharing strategies, stories and solutions for your practice. You can find us by searching off the chart wherever you get your podcasts. And if you'd like the best stories that medical economics and physicians practice published, delivered straight to your email six days of the week, subscribe to our newsletters at medical economics.com and physicians practice.com and if you'd like the best stories that medical economics and physicians practice published delivered straight to your email six days of the week. Subscribe to our newsletters at medical economics.com and physicians practice.com off the chart, a business and medicine podcast is executive produced by Chris mazzolini and Keith Reynolds and produced by Austin La Trobe. Medical economics and physicians practice are both members of the mjh Life Sciences family. Thank 21:39 you.