Latest / AI Meets MedTech / The Hidden Reason AI Projects Fail in Healthcare & How AI Can Cut Clinicians’ Administrative Burden
Transcript
- Anna Nesterova: Welcome to AI Meets MedTech. My guest today is AJ Sigal, the Chief AI Officer at IKS Health, the care enablement platform with a strong focus on the implementation of agentich AI in healthcare. And I have so many questions to ask AJ. Thanks for joining me here. Ajai Sehgal: It's my pleasure, Anna. Anna Nesterova: Hi Jay, I have again I have lots of questions about you, about the company you're working for. And of course, I want to start with a little bit of a background of yours. I know that you worked Microsoft, and then you were one of the founders of Expedia, then Groupon, the Hoot Suite, all the small companies, little companies nobody knows about. And then you shifted towards healthcare. at what point did you realize that healthcare is The problem worth solving, especially with AI. Ajai Sehgal: That's a great question. as you also know, I've been an EMT and volunteer firefighter since what 2004, 2005. so I was deeply involved in healthcare. And then I had two daughters who graduated high school, went into nursing, and one is a surgical ICU nurse at the University of Washington Medical Center, and the other is a s critical care nurse at children's. So I started to get a lot of exposure to healthcare. Well, this is a right right about the same time that I started getting involved with generative AI when I was at a company called Eagleview in Bellevue, Washington. And it became pretty clear to me as we exited Eagleview and I was looking for something else to do that what I really wanted to do was merge my passion for co computers and AI. with the industry or or or the area where it is probably the most applicable, and that's healthcare. And the reason AI lends itself most to healthcare is because of the pl plethora of data that we have, the data that is usually well structured, which is critical for AI, and that's thanks to our electronic health records, but also the data that was locked away and inaccessible so that people weren't b weren't able to make effective use of AI. And so when I had the opportunity to join Mayo Clinic as their first data analytics officer, which had an AI role as well, I I jumped at the opportunity and my God, it was an exposure to the US healthcare system. an extraordinary exposure because Mayo Clinic is a very, very special place. It's unique in in the world with the way they practice medicine. I mean One of my favorite things to do was to go shadow a surgeon in the OR. I mean, it's like these people are miracle workers and they give their lives for people. Except their lives are heavily burdened by that those same electronic health records. Every piece of technology we have introduced into medicine over the years has made it harder and harder on the clinicians. And it's time that we we reverse that. And that's what we call relieving. the burden of of healthcare and letting letting our clinicians practice medicine. Anna Nesterova: Yeah. You're absolutely right. And I can't agree more that they have so much to be accountable for and admin burden is a big one for sure. But if you say that that's something you you want to fix, and I'm sure there are lots of people who are trying to fix, do you still find that the industry is behind and I'm talking about the technology part is behind of what's needed, and how far behind maybe it is if if it is. Ajai Sehgal: I I don't think the available technology is behind what is needed. I think especially with the most recent models, we're getting to a stage where we can confidently apply it to medicine. I think where where medicine is lacking is in the realization that AI strategy is no different than data strategy. If you don't have a data strategy, you don't have an AI strategy. And There was an MIT study a while ago that showed that 80 to 90 percent of POCs fail, and this is across multiple industries. And it's particularly true in healthcare. And the reason that those POCs with AI are failing is because people were operating on small sets of data and didn't understand what it meant to take those models and and scale them across the institution and the level of access that was required to data. We're also hurt a little bit by our own privacy laws. Right. The the i if you look at HIPAA, and Dr. Oz will w I've heard Hibbs say the same thing, HIPAA was written for a time before technology was at the state that it's at. It's time for that for HIPAA to be revised to not only protect people's privacy, but allow their data to be used to help them. Anna Nesterova: So how can you do that? Can you help me understand? Do you mean that we need to start from scratch and the way how we collect data now needs to be different and we need to separate the easily identifiable data from everything else? Can you tell me a little bit more about that? Ajai Sehgal: need to l be we need to liberate our data. So right now, Anna Nesterova: Mm-hmm. Ajai Sehgal: if you look at most tooling within healthcare, regardless of which company offers the the the the tooling, they use your data, my data, as Anna Nesterova: Mm-hmm. Ajai Sehgal: a way of maintaining their monopoly. Okay, Anna Nesterova: Mm. Ajai Sehgal: so I'm not going to name names, but most electronic health records don't take data interoperability as a first principle. So AI needs vast amounts of data in which to operate on. If you want to actually take advantage, I'll give you a very simple use case, one where large language models are really well suited. Cancer patient has a very particular form of cancer, and the doctor has to prescribe a chemotherapeutic drug to treat that patient. In most cases, it's The the doc the doctor's making an educated guess on which is the best chemotherapeutic drug based on what he or she knows about the patient and what he or she knows about the state current state of pharmacology. However, one could look back in time at patients who had similar cancers and similar similar demographics to this patient, find out what worked for them and start there. Well, that way you might reduce the time to zeroing in on the correct therapeutic drug for that cancer, but that means you need to have access to a vast plethora of patient data. And today that's not possible in most of the EHRs because they're designed for single access to a single patient, not bulk access to data. Now some of the HRs like Epic have tried to do workarounds. They have their their clarity in the cloud now. which is a good start, but it still doesn't give you the data in a format that you can operate it at scale. So what you want that is that data in a cloud format that you can take and vectorize the whole thing, turn it into numbers so that you can operate on it efficiently with large language models. And that's one of the things that that we did at Mayo Clinic. We got all of our data into Google and we were able to vectorize it and now Mayo Clinic can build all sorts of cool AI algorithms on top of it. Anna Nesterova: Yeah. And do you think the reason why it's so hard to accomplish right now is because of the fear and maybe lack of trust in the healthcare community, or maybe there's some other reasons? Ajai Sehgal: I I think that plays into it a little bit, and depending on on where you are and what the demographic is. if you go to a small inner city hospital where they're just fighting day to day to liberate a bed, you're gonna find a lot of people who who whose primary fear is AI is gonna take my job, right? I I I've seen that firsthand. but that's I I think that's just a a little bit a a a lack on on our part of providing transparent information about what we're trying to do. And I I think a part of the process of evolving to any technology, doesn't for AI is just one. Any new technology that is being introduced that's going to radically shift people's lives needs to be introduced carefully with Anna Nesterova: Yeah. Ajai Sehgal: full transparency. Anna Nesterova: Yeah. But then it's a matter of collaboration. So all the holders of those records and data, they need to collaborate and find a way how to collaborate and liberate that data. Is that what you are referring to? That's w what has been challenging? Ajai Sehgal: Well I I I think it's that's one part of it, but when you're Anna Nesterova: Uh-huh. Ajai Sehgal: talking about mitigating fear of AI, it's about education, right? And it's about transparency. Anna Nesterova: Mm-hmm. Yeah. Ajai Sehgal: if if you can ch if if AI is a mysterious black box, people are gonna fear it because you don't know what it's gonna do. But if AI is sitting in a glass box and and it's All of its metrics are readily available and every step that it does to reach a conclusion is documented and you look you can look back at it. Well, not only is the clinician going to be trusting of it because he or she can actually refer back to the actual document where the answer came from, but patients could be explained that, hey, it went through your your information, was able to do it much faster than a human could read it, and we're able to treat you better, right? So you can Anna Nesterova: Mm-hmm. Ajai Sehgal: you can maybe make an argument to even a patient through proper education and documentation that this AI is the new tool that's gonna help. Anna Nesterova: Yeah. And you wrote an article which is talking about the glass box approach. that was your article at Forbes I believe early this summer. It was released early this summer. I'll I'll connect Ajai Sehgal: And that's correct. Anna Nesterova: this. I'll put this link under the video and of course I'll connect people so they can also read about it. Because I do agree with this glass box AI approach. It's very important in healthcare more than probably any other industry. And I think this is what makes Some people very skeptical about AI application in healthcare. Again, from my perspective and my experience, what I heard people were saying that companies who are building AI for healthcare, they're being run by the same people who build systems in other industries. But in th those industries, they could make mistakes or they could do a little bit of, you know, the approach fake it until you make it, type of things. But in healthcare it doesn't work. And I think people are a little bit scared that companies, for the matter of The commercial success they will create issues internally will not be transparent enough about those issues with public. What do you think about that? Ajai Sehgal: No, I think you're you're absolutely right, which is why at IKS we emphasize a human in the loop process. So wherever a critical decision has to be made, we don't trust AI to make that decision itself. We present the information to a human and a human has to make that decision. There is zero room for error in healthcare. so with a human in the loop, we can mitigate it. Now we might evolve to a time. in the future, maybe ten years from now, where our technology is so perfect and never makes a mistake. Ha ha ha but but but right now a more cautious approach where you you know the the you the grunt work is done by the machine, but the final decision with all the information is made by a human. And that's the Anna Nesterova: Mm-hmm. Ajai Sehgal: way to build confidence in the technology. Anna Nesterova: So can you tell me more about IKS Health, please? So I like that approach with human in a loop and we hear it a lot, but I'd like to know a little bit more about what exactly the company does and What you offer. Ajai Sehgal: Well, I what what what we what we do is evolving over time, but I I I like to call IKS the the the largest health company healthcare company in the United States that that most people have never heard of. And we're trying to change that. But we haven't had to advertise very much because IKS does such a great job for the hospital systems and clinics for who are our customers, primarily because from the CEO on down, people care deeply about Healthcare and reducing the cost of healthcare. And we have RKF started 18 years ago essentially doing labor arbitrage with people in India. So taking the stuff that was really repetitive work and moving it to people in India. And then roughly five or six years ago, they started to evolve into a high-end technology company where the an AI-driven workflow was starting to replace the the the repetitive work. We still have those expert humans in a loop. I mean IKS has over 1,500 clinicians that are employed who are supervising stuff along the way. The goal is to reduce the cost of care for not only for the patients, but for the for the clinics that employ us and make sure that they get the absolute best service. And now we're str we're starting to replace a lot of the deterministic workflow AI with agentic AI approaches, just because it allows for more flexibility, but again, with the right monitoring stopgaps along the way and humans in the loop. Anna Nesterova: Yeah. Do you think we may come to a a time when AI will be so intelligent? I know it's not the right word to say, but let's call it intelligent enough that the human in a loop will not be able to make the final call and AI can somehow hide the information from the human. Ajai Sehgal: You know, if you had asked me that question two years ago or even a year ago, I would have said no, that it it'll never become autonomous. I'm not so sure anymore. Anna Nesterova: Mm-hmm. Ajai Sehgal: if you if you look at the current models that are out there, I I'll give you an example, Anthropics Fable. I was experimenting it with it just as a coding assistant. What was phenomenal about Fable And and I think the same is true of GPT five point six, is not only does it produce excellent code, provided that you as an engineer understand how to plan and specify for it, but it often is capable of anticipating stuff that is logical to to put into the specification that you didn't think of, because it just sort of slipped your mind and it'll come back and say, but what about this? Anna Nesterova: Mm-hmm. Ajai Sehgal: That's kind of freaky, right? It's now now is it intelligent? no. Is it sentient? No, it's not sentient. Is it intelligent in the i by way of recognizing patterns and seeing when something is missing from a pattern? Yeah, it's absolutely intelligent. That's what you how you describe intelligence. It's not sentient. So it doesn't feel pressure when a decision is being made. It has no feeling. It can simulate it, but it doesn't. And that's something that people have to keep in mind. AI is only as good as the data it operates on and the input that you give it. And is it going to completely replace software engineers? That's a question people ask me all the time. And the answer is absolutely no. It's going to make software engineers change the way they work, but the human still has to provide it the plan and definition for for New and novel ways of doing things that don't exist. If you're re-implementing something that already does exist, yeah, AI can do it. It's really great at reverse engineering other people's codes. But because that's all pattern recognition, right? Anna Nesterova: Yeah. Yeah. Ajai Sehgal: But if you're writing a novel piece of software that nobody else has written before, you need a really solid architect who can guide that AI. Now, does it does it get rid of the slog work, the ground work? Yeah, absolutely. Can I do the work of 10 Anna Nesterova: Yeah, that's what I was gonna say. Yeah. Yeah. Ajai Sehgal: Ten interns. I can do the work of ten interns. It's awesome. Anna Nesterova: they no longer do the, like you said, the groundwork. Now they have to really be much more predictive and like you said, architecturally smart in order to also catch the mistake or talk to or whatever agent they're using. code are they using, make sure they can they can provide the proper response, I guess, and the result. So it it it seems like it's changing quite rapidly. I see it in my conversations with people too. Ajai Sehgal: Yeah. I I I absolutely think that if you are in university for software engineering and you think of about stopping at a a a bachelor's degree, you're making a big mistake. Because a bachelor's degree teaches you how to learn. Now you gotta go and apply that learning. Go do a master's, per perhaps a PhD. And w if you finish a master's degree where you've actually focused on a smaller set of topics and not only learned how to learn, but learned how to think, now you can become an excellent software engineer. Anna Nesterova: That's a great advice. And if we're talking about AI and data, so AI companies, including IKS and other big LOM companies, they have access to the most sensitive personal information when it comes to healthcare. Ajai Sehgal: Yes. Anna Nesterova: I would like to know a little bit more about the security and what type of guardrails are exist and are they really working and how do you implement them? Because I don't have technical background and I'd like to know. How you look at them from your vintage point. Ajai Sehgal: Yeah, data privacy and the security of the data is n one of my number one concerns, and it's also the number one concern of IKS. We deal with a lot of healthcare information. And the way we project that is we don't use public AI endpoints. We have business associates agreements signed with Anna Nesterova: Mm-hmm. Ajai Sehgal: all of the people that that are our suppliers. They have business associated agreements with their suppliers, and we vi we validate all of that. All of our AI is either within Google or Microsoft and it's in VPC, so virtual private clouds with endpoints that are HIPAA certified. and ICAS actually has a high trust certification as well. High trust is like the ultimate in security. And we have monitoring to ensure that data never leaves our VPC. Anna Nesterova: Mm-hmm. Ajai Sehgal: so all of this is logs constantly monitor. We have a security team that is solely focused on this. And as time goes on, there are better and better tools, AI tools, that can actually guarantee this. So yeah, it's it's basically it's standard data security, right? That you Anna Nesterova: Mm-hmm. Ajai Sehgal: you EHRs have to have the same security. Encryption and transit and at rest, it's it's it's the ABCs of data security. If you do AI right, you are not exposing anybody's information to anything because the AI endpoints that we have are private. So if I hit Gemini in Google, Google can't see that data. They don't use it for learning. They don't same with Anthropic, same with with Microsoft. So there are ways of doing it. Now, if you're a startup and you don't and you're not familiar with these things, you better be really careful and hire someone who really understands this because If you're hitting public endpoints with healthcare data, you're going to leak. But they're there Anna Nesterova: Yeah. Yeah, and then Ajai Sehgal: I I I'm not worried about the security because the best practices are well known. You just have to practice them. And I at I at IKS we do. Anna Nesterova: Mm-hmm. And when it comes to implementation of new technologies and the services that you provide in the company, what is the main barrier to adoption? Is it the finance? Is it the technology itself that people don't know and education is not there, or the skepticism of people and maybe, you know, the change management is always tough. What do you see is the main challenge for you? Ajai Sehgal: That's an excellent question because IKS and most backend healthcare companies, we're building technology for our people to use, right? It's we we're we are ha we have software that is clinician facing, but it's it's pretty limited to stuff like autonomous scribing and stuff like that, right? So the the common clinician facing software. But when you're evolving from a company that was essentially a BPO, where you have we have over 12,000 people in India who were actually doing coding by hand using spreadsheets. And we have an extremely efficient operations team that is that knows how to do this cheaply and effectively. Now we're introducing new technology and it is the change management problem. It is Anna Nesterova: Mm-hmm. Ajai Sehgal: n you're you're now convincing people that they should be using a new tool. We still need you because you're the human in the loop, but you're convincing them that they're using this new tool that they're see they see AI and they say, my God, that's gonna take my job, right? So it is a change management problem. It's an educ internal education problem. And yeah, you know what, some some jobs are not going to remain the same. They but we'll still need the people. They'll just evolve to doing things in a different way. And it's the onus of us as leaders within these technology companies to educate folks and to provide them the valid information that they need to show that AI is not this evil that's coming to take your job. It's a new tool. It's no different than when we went from pen and paper to spreadsheets on computers, right? We didn't Anna Nesterova: Yeah. Ajai Sehgal: take away the accountants' jobs. We just gave them a more powerful tool to operate with. Anna Nesterova: Yeah. But I do see why people are scared. I do understand why they see it and they think, Okay, now it can do so much more and better than maybe some stuff that I do. What's gonna happen in two years? Ajai Sehgal: Well I I mean I mean Anna, we might replace you with an AI Anna. Anna Nesterova: Yeah. Yes. I actually think it will happen one day. I mean I can if I want to, the avatars and everything, but yeah. That's very interesting. Yeah, yeah. Ajai Sehgal: You probab you probably could. But it w it wouldn't be as much fun. Anna Nesterova: Thank you. Yeah, and I do agree with that. I think we still need some human to human connection. And what about ethics? That's another part that I am concerned as well. The ethics in with AI. And even with that human in the loop. So do you think the ethical component is all on humans who are checking on those AI decisions, or we need to train AI on ethics? How do you do that in your practice? Ajai Sehgal: Ethics have to be a fundamental, especially in medicine, have to be a fundamental part of how you build your tools and how Anna Nesterova: Mm-hmm. Ajai Sehgal: you build your workflows, right? I can give you some very concrete example. If you are building a model that requires large amounts of data, the onus is on you as the person building that model to make sure that the data that you're using is well balanced. So that it doesn't just represent White males of northern European descent. It's balanced according to the demographic of the population of the country or the area that you're serving, so that the decision that is made is based on valid data. This is these are this is core to ethics. Now, is it easier to get a data set that is unbalanced? Yeah, absolutely, but that's not ethical. And I can tell you that that in virtually every instance that I have encountered, be it at my time at Mayo Clinic or dealings with with with friends and colleagues in research across the country. Everybody who is doing this type of research is critically aware of the need to have balanced data and puts a lot of effort into ensuring that it's is well balanced. Because if it isn't, you're gonna get the wrong answer. You're gonna get the you're gonna you'll get the right answer for a few people, but you won't get the right answer for everybody. And then we gotta get the right answer for everybody. Anna Nesterova: Yeah, and especially if historically we have some biases, right? It's hard to bring that data and build it up to the level of like you said, the level of unbiased data and ethically correct data. It might take some time. Or maybe we're there already. I don't know. You tell me. Ajai Sehgal: Well, I I I think that EGFR is a great example, right? how how the formulas for calculating who was able to get a kidney transplant was heavily biased against black people because they it had what was nothing less than a racist metric in it, right? Anna Nesterova: Mm-hmm. Ajai Sehgal: I think the the medicine has learned from that and that that mistake is informing how we're doing building models for AI. I yeah I know it certainly affects how I do things and it affects how most of my colleagues in this industry do things. Anna Nesterova: No, that's good to know. 'Cause I hope people with good intentions are running this. Because I think in a few years from now we will be relying on AI so much and healthcare. it would be nice to know. Ajai Sehgal: So so so you're absolutely right, we will be. And the reason is there is just so much data involved in healthcare that it is impossible for a single human to really fully digest the entire medical record of an individual and surface only that which is important. If doctors were doing that, they'd have no time to practice medicine. So for instance, if if I can summarize using AI in someone's entire medical history, regardless of what institution they come from, then and and if I can understand what the chief complaint is of that individual going in for an appointment, then I can surface exactly the information that the physician needs to know before he sees the patient, and only the information that that physician needs to know before he sees the patient without having him spend he he or she spend time going through this thick PDFs in order to figure it out, right? That's Anna Nesterova: Yeah. Yeah. Ajai Sehgal: relieving the administrative burden. Anna Nesterova: Yeah, hundred percent agree. And is there anything and in healthcare and AI that genuinely surprised you recently? Something you've seen, you're like, you know what, maybe good or bad, something that surprised you genuinely. Ajai Sehgal: Recently, actually this is four years ago when I first joined Mayo Clinic. they had an AI algorithm that could interpret a twelve lead ECG and diagnose conditions from a twelve lead ECG that would previously have required an echocardiogram. That was amazing. That was two brilliant cardiologists from Mayo Clinic who decided to pra who who suspected that there might be a pattern. there and this and this was not even using large language models. This was using predictive analytics. And they that I mean that's fantastic. That's that was the first real demonstration that I saw of AI practically affecting people's lives. It was pretty cool. Today I I get surprised on a daily basis. Right? I mean it's the the the things the thing the insights that we can come out of come out that come out of AI are brilliant. Will we have an AI doctor one day like Star Trek? We might, but I we need to still have that human in the loop. Anna Nesterova: What's your prediction when we may have that independent AI doctor? What's your prediction for future? We can look at this episode, I don't know, five years from now and see. Ajai Sehgal: I think we will start getting close for very simple diagnosis within five years. Anna Nesterova: Mm-hmm. Ajai Sehgal: But I don't think we will ever get to the stage where AI is going to diagnose the complex diagnosis that a clinician needs instinct to figure out instinct and experience. So it depends on how sick you are. Anna Nesterova: Mm-hmm. Ajai Sehgal: If you have appendicitis, it might be easy to diagnose. Anna Nesterova: Yeah. Yeah, again, going back to data, right? Because if you have large data sets, it's much easier for for AI to sort things out. But if it's something unique or Ajai Sehgal: Sure, but Anna Nesterova: unique for that specific patient, a very rare case and combination of factors. Ajai Sehgal: Yeah, but but but how is data generated? Data is generated through a physician ordering tests, right? Anna Nesterova: Yeah. Yeah. Ajai Sehgal: And what tests should be ordered. That's where a clinician's education takes where where where they where it takes effect, right? Can can we teach AI to order this, this, this, this, and this test, without actually looking at the patient, without having the same level of input that a clinician has? Anna Nesterova: Yeah. Yeah. Ajai Sehgal: remember the AI has data right now. it doesn't have that face-to-face meeting with the patient, it doesn't have it cannot gather those non readily nonverbal or readily apparent Anna Nesterova: Verbal Ajai Sehgal: cues that a good clinician counts on. Anna Nesterova: Yeah. Ajai Sehgal: So Anna Nesterova: But what I also discussed with one of my guests in the previous episodes, the over reliance on AI, that some people predict that in the future doctors will be over relying on AI for that reason. They would think that AI knows better and may potentially lose that touch and sense. But again, we're talking hypothetically, but it's an interesting concept, I think, and it may be a rational fear. Ajai Sehgal: I well I I I no, not if it's a good doctor. I I Anna Nesterova: Yeah. Ajai Sehgal: I know I know many many clinicians and I think they're smarter than that. They're I I listen, I I use AI for coding. I have learned not to trust it 100% of the time, even the best of the best models. If you play with it. For long enough, you'll you can see the patterns where it fails. And I think Anna Nesterova: Mm-hmm. Ajai Sehgal: the same will apply in any industry. AI is not perfect, it is pattern recognition, it is what predict the next syllable, right? And that is has limitations. So play with it long enough and you realize the limitations and you trust it a little less. Anna Nesterova: Awesome, that sounds good. And maybe to finish it up, so what's one thing about future of healthcare that you are most excited about? Ajai Sehgal: I'm most excited to spend more time face to face with my physician and him less time having him click, click, click, click, click in the EHR, right? I think we can get to a stage where the EHR is just something passive in the background and it basically accumulates data just by monitoring the interaction between the physician and the patient. Interesting anecdote. One day my daughter, my youngest daughter came home after a night in the O R I C U and she says, Dad, I don't know what you guys are doing with technology and healthcare, but it took me eighteen clicks today to tip a peabag. Well, what happened was the peabag was supposed to be tipped at it was supposed to be logged on the hour. But it got logged into the next hour because of an error in the EHR and it took her eighteen clicks to undo it and put it into the right hour. So technology has its problems. That's the type Anna Nesterova: Yeah. Ajai Sehgal: of stuff that we can solve. Anna Nesterova: Yeah. No. Thank you, AJ. That was awesome. Ajai Sehgal: It's my pleasure.