Latest / You've Been Heard / 395- When AI Makes Healthcare Mistakes Lives Are Lost w/Sandeep Shenoy

Transcript

395- When AI Makes Healthcare Mistakes Lives Are Lost w/Sandeep Shenoy · You've Been Heard

• 0:09
  Speaker A: Hi, I'm Michael Moore, hosting this podcast. I'm here with Sandeep Chinoy, IT manager for Viant Medical. Welcome to the program, Sandeep.
• 0:17
  Speaker B: Thanks, Michael. It's great to be here. Awesome.
• 0:20
  Speaker A: Let's talk a little bit about you and your experience and Vayant Medical. So you're in healthcare. How long you been in healthcare?
• 0:30
  Speaker B: Yeah, so I have been in healthcare for over 17 years. I'm a digital transformation leader in healthcare for 17 years, specifically in healthcare and medical device manufacturing.
• 0:42
  Speaker A: It's tough enough to be in healthcare. You've been there for 17 years and on top of that you've been doing digital transformation. Tell us a little bit about the digital transformation.
• 0:53
  Speaker B: Yeah, I specialize in integrating AI, blockchain and Internet of things in medtech environments. So we, we manufacture medical devices and then our job is to make sure that all these systems are connected and we build algorithms based on the data and then we secure them through blockchain and everything. So it's all about innovations in medical devices. So that's what we do. I'm not manager, so I look after a certain team members from the quality perspective and the digital transformation perspective.
• 1:27
  Speaker A: Well, let's dive into this a little bit. AI in medical devices, let's break that down and figure out what that actually means.
• 1:35
  Speaker B: Yeah. So in simple terms, AI in medical devices enables machines to learn from data, like supporting doctors, engineers and manufacturers in making faster and more accurate decision. So it could mean, for example, an image system that detects tumor that even a trained specialist might miss. Or it could be a wearable device that predicts heart failure or something before the system appear. So in our work, in the recent work, which is again in medical device manufacturing, I've seen AI models improve precision manufacturing by predicting what might go off the track before the devices are shipped to the patients. Right. So that's what AI in medical device in simple term is. It's all about learning from the data, historical data, and making more meaning out of it and implementing them in these devices that support the clinical industry.
• 2:33
  Speaker A: Wow. I mean, like there's so many different ways racing around in my brain right now that you could use that to transform business and transform healthcare. Right. Give us some more examples of.
• 2:47
  Speaker B: I can point out some major developments in this area. So the first FDA approved autonomous AI system that detects diabetes without doctors inputs and an AI in MRI imaging that accelerated the scan time by 30 to 50% that improve their efficiency and person throughput, efficient throughput. Right. So there are some real world use cases in Which AI has been really transformative.
• 3:17
  Speaker A: Wow, that's huge. Not just from a business angle, but also from the healthcare angle. Merging them all. Technology, healthcare and business. I like it. Yeah, yeah. It does bring up a good question though. AI in itself can be pretty controversial in some regards, and definitely on how it's used and especially with some of the safety that's maybe missing from it right now. Especially in healthcare, which has some of the strictest guidelines around cybersecurity and healthcare itself having strict guidelines itself, not even talking about technology. Let's talk about the ethical considerations a little bit that are kind of floating around when it comes to medical devices and AI.
• 4:05
  Speaker B: Yeah, sure, the world is turning towards AI. So AI in healthcare, as the world knows, is truly transformational. But it forces us to ask some uncomfortable yet necessary questions about the fairness, privacy and accountability. So I believe the implications of healthcare AI are extremely significant. So when, for example, when AI makes a mistake in finance, you lose money. Likewise when it makes a mistake in healthcare, you could lose a life. Right. So the conversation around AI ethics really started gaining a lot of momentum when we saw real world failures like biased algorithms in hospital systems and privacy breaches in health app. And now there is a social dimension to this issue as well. And as you know Michael, AI works based on historical data or trend. And if the history is biased, like if historically the data we collect reflect human biases. Right. So for instance, a healthcare data set often over represent men or males in the past. So due to the historical job roles or research participation or the dominance of men in the past. So if the AI builds a trend or a pattern based on history, and if the history is biased, it's gonna give a biased outcome as well. So that imbalance means algorithm can be unintentionally performed worse for women or underrepresented groups. And when it comes to healthcare, it's a big concern.
• 5:42
  Speaker A: Right, so that's a great point. I can't underscore that enough. I mean, that's already a big problem right now. It's been a big problem for a long time. And all of the training material that you're gonna use for AI is built on these books that were all, all bu around males. That is a huge thing to point out. And it's accurate. The data in is what sustained garbage in, garbage out.
• 6:07
  Speaker B: Right, exactly. But the gender bias in healthcare monitoring devices. So due to the bias training, early fitness trackers failed to accurately interpret female heart rate patterns. So things are changing now because now there are synthetic data feeding that type tries to balance it out. But AI ethics has become a discussion point and there is a lot of pullback in AI because of initial push. And as you clearly said. Right. It's based on the data. So history is bad. The future is going to be bad as well unless you carefully implement it.
• 6:45
  Speaker A: Well, I think. Right. And I think there's other concerns as well too.
• 6:48
  Speaker B: Right.
• 6:48
  Speaker A: I mean, you're in the healthcare field. I said right off the bat, cybersecurity is a huge deal. HIPAA is a huge deal, patient confidentiality is a huge deal. And we've seen recently that AI has been failing in this regard.
• 7:03
  Speaker B: Right? Yep.
• 7:05
  Speaker A: What are your thoughts on that?
• 7:06
  Speaker B: Yeah. So cybersecurity, as I said. Right. One of the major concerns with AI or ethical part of AI is a privacy. So that's really an important topic, Michael, because in healthcare, cybersecurity isn't just about protecting data, it's about protecting patient safety and their trust. Because you are getting so much information from them. AI cannot work without data. So you need data. And as all these data get dumped, it's about securing the data. So the attack is massive. Hospitals, clinics, production lines for medical devices, all of it is connected now because for all these systems to work, they are all connected, Right. So legacy system, medical devices, all these IoT sensors and they are all not designed on all these modern security in mind. So we see a lot of ransom attacks, data breaches. They are in rising fast. It has become an easy target because we have so much data and the cyber attacks are like a common thing these days.
• 8:15
  Speaker C: Most IoT leaders don't need more technology, they need less noise. Over the years, infrastructure quietly gets complicated. Multiple locations, multiple Internet providers, different contracts, separate bills and reactive support. Nobody plans it that way, it just happens. We, what we do is vendor neutral advisory. That means we don't sell platforms or force change. We start with low friction areas. Internet telecom, Microsoft licensing. Places where complexity builds quietly and can usually be cleaned up without disruption or capex. Sometimes we find savings, sometimes we find nothing. Either way the outcome is the same. More clarity, less distraction and it that's boring in the best way possible. If you want a second set of eyes from people who sit on the buying side of enterprise infrastructure every day, just go to you've beenherd.com and answer the questions to simplify, streamline and save.
• 9:11
  Speaker A: You nailed it. And I think that security is like the foundation of a house. Right. If you don't build it at the beginning, it's really, really difficult or next to impossible to truly get security working just like it would be to build a foundation on an already existing house. Right. So I think that we've brought up a good point here. Trying to inject the security afterwards, it makes it very difficult and very open to vulnerabilities. I'm hoping that in the future here that they'll improve the foundation a bit when they come out with products and stuff like that are geared towards AI. I do want to double back on the medical devices. The medical devices themselves have strict guidelines and how they're built and how they're tested. Are there any rules around implementing AI into these devices? Do they follow the same medical device testing guidelines? Are they able to skirt them?
• 10:11
  Speaker B: I don't know the answer to that. Very good question. So now we have frameworks, all these regulators pushing hard on several acts like FDA has gmlp, it's called good machine learning practice which guides the developers to ensure that transparency and human oversight throughout the AI lifecycles. It's not like in the past where it was relaxed. So AIML Action Plan for Software as a medical device emphasizes ongoing validation, not just a pre market approval. It expects that you recurrently or repeatedly review your outcomes. And in Europe I just want to highlight that AI act classifies medical AI as a high risk requiring fairness testing and documentation of model logic and real world performance monitoring. So there is a lot of pushback and we are also seeing in the medical device industry the ISO frameworks have began to integrate AI governance into the quality and risk management, meaning ethical compliance is now a part of or embedded in the product life cycle. So we in the companies have added in our production line added AI quality check or AI audit as an operation sequence in our routings in making something so it goes through an AI audit team to make sure that it goes through certain validations before we make the product out. So there is a lot of push from global regulators and also within the companies these days.
• 11:46
  Speaker A: Great answer to the question and I'm actually glad to hear that they are starting to implement this. I was struck by the ongoing testing and I think that is a wonderful idea and I actually not just a wonderful idea. I think security would fail if it wasn't done that way.
• 12:06
  Speaker B: Right. I need to.
• 12:07
  Speaker A: There's these things change so quickly and the fact that it's evolving means you have to have an evolving process to get it to work. So to me that makes sense. I think they were absolutely right. And I'm glad you pointed it out that a key difference between other security measures is that this One needs to be constantly watched. So a good observation and I think we're definitely in for. All the cybersecurity folks of the world are going to be wrestling with this for the foreseeable future as it only gets quicker and harder to maintain.
• 12:44
  Speaker B: I agree.
• 12:45
  Speaker A: We've already gone over so much stuff and there's so much more to go over here. I'm struck by this. I really kind of want to dive back into the bias for a minute and I'm just going to circle back to it because as I was thinking about all the different bias that can happen in the medical industry, that just happens anyway now. Right. And then I was thinking about AI and I was thinking about the constant validation checks that you mentioned. I was thinking, have they implemented ethical validation checks continuously? Like continuous ethical evaluation checks? Is that a thing? Have they implemented any continuous bias checks, regulations?
• 13:33
  Speaker B: That's a good question. So it all starts with data diversity, right? So how do we detect or reduce the bias as an ongoing process? Right. So the first thing that we should ask is who is representing the data set or on who is missing? So in my work we use fairness aware algorithms that can detect imbalances in the data modeling training. For example, if a heart rate data set underrepresents older women, we retrain the model using synthetic data balance techniques. So we, we call it bias audits where we test model outcome across gender, ethnicity, geography and all that. And it isn't just a one time mix, it's a continuous process. It's not just a set of instructions. It's an improvement, much like quality assurance in manufacturing. So that's what we do. I can give you a real time world example where Google Health Dermatology AI was retrained with over 65,000 diverse images to improve their fairness accuracy or fairness across skin tones and conditions. So it's an ongoing process, it's all about conducting repetitive audits. And FDA now requires demographic performance reporting for AI ML based medical devices which is also pushing for greater transparency and equality. So yeah, I mean we are seeing some thorough steps as a continuous improvement to make this better.
• 15:12
  Speaker A: I'm glad they're moving on this so quickly though. This. I actually didn't expect that we'd have some of the stuff going so far. So I'm actually feeling a little bit better about this. I know there's a long way to go. Can you share some examples for us of maybe some more real world? I know we talked a little bit about the Google Health, but I think if you could share some more Examples like that. I think that would be super helpful.
• 15:36
  Speaker B: Yeah, I can share some examples from my case studies and work that impacted the world because of the AI in medical devices in particular. So an early oncology AI program provided or, you know, organized by one of the leading company. So that taught us one of the most important lessons in healthcare AI. It recommended unsafe treatment because it was trained on synthetic data, not on clinical data, which is a major ethical oversight. Another example is AI imaging algorithm that performed well in lab settings but failed in the actual hospital settings or environments. And later regulators halted the approval until the developers could explain how the model reached these decisions. Another example, which is again another major health chatbot which misdiagnosed women due to gender biased in training data. And then another in the parallel world, not in the healthcare is a major credit card provider which was accused of gender discrimination in its credit limit algorithm and all that. So there are a lot of failed real world examples of AI because they were not ethically implemented.
• 16:58
  Speaker A: I'm glad that we're talking about this because that this happens more often than it's caught.
• 17:04
  Speaker B: Yep.
• 17:06
  Speaker A: And my concern is that there are companies that are fixing this only when it's caught, rather than trying to be proactive about it. And I think it's. It is a really big point as to why we need these regulations and why we need this set up this way because there's no advocacy here from a patient aspect. They're going to use this device because a doctor prescribes it and or says that they need it and they don't have any say over this device. They don't understand how it works. They just know that they need to have it because of a medical reason.
• 17:42
  Speaker B: Yep, you're right.
• 17:44
  Speaker A: And they will never know if they were discriminated against by the AI. Right. Unless it's pointed out. And by that time it's kind of too late. Right.
• 17:56
  Speaker B: Yep. Yeah, you're right. Yeah. So, yeah, I think that's where all these global regulators are playing a great role now in sort of being more reactive now. There are a lot of proactive initiatives to protect things before they happen.
• 18:13
  Speaker A: What I will say, like you mentioned, these global regulators, they really got a lot on their plate.
• 18:17
  Speaker B: Yeah.
• 18:18
  Speaker A: Do you think that they are going to end up arriving at standards or do you think it's going to end up kind of like privacy where it's not one rule for everybody, but depending on your region, depending on what they interpret it. And so we're going to end up with a patchwork of different AI Regulations.
• 18:36
  Speaker B: Yeah, there are a few regulatory moves that we could see the EU AI act expected to be enforced by 2026, which set a new standard for the fairness and explainability. And Healthcare or Health Canada and the UK's MHRA, their regulatory board are also drafting aligned framework to ensure global consistency. And also another important thing is AI devices now require algorithmic transparency reports during their 510k submission process before it get launched, they need to make sure that there is a transparency report. So documentation must also prove the absence of demographic bias, ensuring the equality and all that. So there is already a lot of push from the auditors or from the global regulators and they also have a lot of changes in the framework upcoming.
• 19:33
  Speaker A: I hope they're able to adopt a global standard, but I do feel like even if they do, in places where there are stricter privacy, you're going to have different interpretations of that rule. Or you might even have stricter guidelines regionally that are put on top of those rules, whereas in some places you would not have that right.
• 19:54
  Speaker B: Yep, I agree, Michael.
• 19:56
  Speaker A: Interesting. Well, let me ask you a question. Obviously while we wait for the global standard to come out and for everything to kind of fall into place and everyone knows that this is a journey, it's not a destination where it's going to be constantly changed and updated, especially because AI moves so fast. But what can specific organizations do to make sure that they're baking in AI ethics into their organizations and don't fall prey to certain problems like that credit card company ran into.
• 20:31
  Speaker B: Yeah, outside of these regulators pushing all these, there are organizations that are taking proactive steps. It all starts with intention. Ethics needs to be built into every stage of an AI deployment, not added later as a compliance checklist, which is what used to happen in the past. So the recommendation is some practical layers like fairness by design, which means using diverse data set and applying fairness metrics from day one to avoid bias. And then use some transparency tools like I don't know if you have seen, but in the past when you used to use any chatbot like ChatGPT, it never used to tell you why, but now there is a reasoning behind it, it gives you why it is giving you that answer. So transparency tools like LIM and Sharp, these are technical terms, these are techniques used to make model reasoning visible and understandable to humans. So when it gives you an answer, you need to know why it gives you that answer or why it gave you that result. And another point is the accountability framework, when AI gives you an answer or AI gives you a Result, if it doesn't work, you don't know who to blame. Right? So it's important to clearly define who owns. The decision is that the doctor, is that the coder, is that the manufacturer. It cannot be nobody, it cannot be AI. AI cannot just be the owner of everything, so the accountability becomes a part of it. And then of course, the privacy protection methods that ensure sensitive health data remain secure and decentralized so that people can trust all these innovations. Right? So organization culture, that's where it all stands. So we encourage teams to create AI ethical committees within their research and development group or team that is responsible for all these deployments and put some ethical KPIs, such as bias reduction rate and some transparency score and all that to measure the progress of their initiatives. Right? So it's about trying and it's about proactively putting all these practices in place and also putting a KPI to measure how tightly we are aligned with the ethics.
• 22:53
  Speaker A: Michael, so how, how do we get companies to get on board though, right? I mean, this is the same problem you have with cybersecurity and investing into it and implementing things. Because cybersecurity, when implemented correctly, can be an innovator. AI is certainly an innovator, but ethical AI can be an innovator. And as well, you can do amazing things if you're getting the right data and putting it out there. So there's lots of reasons to implement it. But how do you get the businesses to get around that?
• 23:30
  Speaker B: That's a great question, Michael, and honestly, it's one that I asked a lot to myself. So I think the first step is to move AI ethics out of the compliance box, right? It should be built into the business value. So companies adopt ethics faster when they realize it's not slowing innovation, it's actually protecting it. For example, a transparent AI model builds clinician trust. A fair algorithm avoids damage of the brand name and avoids regulatory fines and a penalty and everything. And a secure data pipeline protects the brand. So ethics isn't just the right thing to do. It is smart risk management and smart business to avoid liability. And second, leadership has to set the tone. It's the CFO or the CTO makes ethics part of the innovation culture, not just an audit step, then I think it naturally flows down to the engineers, the data scientists and the product managers and all that. So it's small wins. I believe if it is part of the principle, I think it's going to get eventually part of the company policy and everything.
• 24:43
  Speaker A: So, gosh, I love that answer. We got to get it out of the compliance box. That rings well for cybersecurity, too. I mean, this. One of the things that I truly believe is that when people think that they have to do something because they have to do it right, and it's just like this is something you need to do. But if you're able to actually explain the reasons, like you mentioned, to protect the brand, it's a risk management piece. And that's when people want to do those things because they realize the benefits. And I think that's a great answer. We gotta get it out of the compliance box.
• 25:21
  Speaker B: That's right.
• 25:22
  Speaker A: It's true. It's very true. I think that's a good way to do it. So we've reached a point where we talk about the IT crystal ball. This segment is meant to go over the future of it, and I think that in true fashion. WEIR I think we should mold this question and mold this to be. Let's look at the future of AI in medical devices. I know that AI is already there, but I would like to see your interpretation of what the future holds for AI and medical devices and see where that conversation goes.
• 26:06
  Speaker B: Yeah, it's pretty closing, Michael. I think the future of it, especially in healthcare, will be defined by intelligence, trust and collaboration, not just innovation. It's going to be built based on several factors. So AI will become more human aware, not just predicting numbers or outcomes, but actually understanding the context, the why question, the why behind every medical decision or every results that we get. So we will see systems that explain themselves, learn ethically and adapt responsibly, almost like digital colleagues, not just digital tools. So in medical it, AI will move deeper into preventative care, which is already happening, like spotting disease risks, early, optimizing hospital operations, ensuring that every patient gets timely and personalized treatment. But the real future isn't just smarter technology. It's about responsible technology. As I mentioned earlier, it's going to be built based on intelligence, trust, ethics principles, collaboration, more than just innovation.
• 27:24
  Speaker A: Yeah, you're painting a very rosy picture of the future in medical it. I hope that is the case.
• 27:30
  Speaker B: I also would like to answer something else. Like I would say AI is not replacing humans, it's going to just amplify humanity just with real evolution will happen. So it's just going to assist us. It's not going to replace us. So there's a lot of fear in people that it's going to replace. It's going to just help people out and we just need to reskill and adopt and Be more accountable to both the results. Right.
• 27:56
  Speaker A: So, yeah, I like that you said that. Mainly because I think of this message needs to go out to a lot of business leaders right now. And that message is, if you're investing in AI, right, to replace people, what you don't understand is that you're going to need people to help with that AI.
• 28:21
  Speaker B: Yeah. I mean, AI is not something you.
• 28:24
  Speaker A: Could just put in and let it go unfettered and unmonitored.
• 28:28
  Speaker B: Right, Yep.
• 28:29
  Speaker A: AI is something that needs to be monitored, that needs to be adjusted, that needs to be tweaked. I think you're right about the reskilling.
• 28:37
  Speaker B: Right, right.
• 28:37
  Speaker A: It's a big deal because there are lots of people working towards this and if they don't do it, the consequences are going to be dire.
• 28:47
  Speaker B: You're right. That's right.
• 28:49
  Speaker A: I mean, that's what we talked about, this entire thing. I mean, the consequences, the. This is all about risk management. This is all about fairness. And if you don't build that into your brand, then your brand is going to suffer. And we've seen this happen time and time and again, not just with AI, but with people that don't build this culture into their brains. I think that's huge. I do marketing on the side as well, so I know all about brands and I know all about protecting them. And it really would scare me if someone told me that my brand could go belly up and by not doing the right thing and it should scare companies, I think.
• 29:33
  Speaker B: Yep, yep.
• 29:34
  Speaker A: Totally agree. So I guess as a closing part on this, you stand by this rosy picture and you think that we're going to be able to make it through and build this into our systems and our culture.
• 29:47
  Speaker B: Yep. I truly believe that the real AI innovation isn't just about building intelligent machine, it's about building responsible ones. So I truly believe in it as long as we understand AI and the ethical part of AI together.
• 30:04
  Speaker A: I'm Michael Moore. I've been hosting this podcast. I've been talking with Sandeep Shenoy, who is the IT manager of Reviant Medical, talking about all about AI and medical devices and the future of what's going to be a rosy picture, it looks like. Sandeep, thank you so much for joining the program and bringing on one of the coolest talks and in depth talks we've had regarding AI.
• 30:30
  Speaker B: Thank you, Michael. It's been a great discussion.