Latest / HumanifAI HealthTech / How to Make Healthcare AI More Trustworthy w/ Steve Pappas, Head of Innovation, eGain
Transcript
- 0:00Brian: In healthcare, a wrong answer is more consequential than a bad customer experience.
- 0:05Brian: It's a denied claim, a missed medication, or a treatment that gets delayed past the point where it helps.
- 0:12Brian: Traditionally, a human health plan representative would catch that.
- 0:15Brian: Now, AI answers first, and unlike humans, it has no instinct for when something's off.
- 0:21Brian: So when a policy changes and the AI's knowledge doesn't, a new problem arises.
- 0:27That problem has a name: model drift.
- 0:31That problem has a name: Today's guest says the fix isn't a smarter model, it's better data, and he's built a way for contact centers to catch drift before a patient does.
- 0:48That problem has a name: Welcome back to Humanify Health Tech, where we explore how AI is rewriting the rules of healthcare and putting people back at the center of care where they belong.
- 0:58That problem has a name: I'm your host, Brian Chi.
- 1:00That problem has a name: Last episode, I spoke with Outpost Bio CEO Dr. Jenny Yang about how AI can help personalize healthcare with the right treatment for the right patient.
- 1:11That problem has a name: But today we're talking about what happens when AI is wrong.
- 1:15That problem has a name: AI is wrong, and worse, nobody knows about it.
- 1:20That problem has a name: According to a NAIC survey, 84% of health insurers use AI across prior authorization, utilization management, and disease management.
- 1:31That problem has a name: In an industry where policies change like the weather, a confident wrong answer can easily ripple out from the contact center and completely derail a patient's care journey.
- 1:41That problem has a name: That's the problem today's guest is working to fix.
- 1:44That problem has a name: Steve Pappas is head of innovation and healthcare go-to-market at eGain, which builds the knowledge platform that sits behind healthcare contact centers.
- 1:54That problem has a name: Steve helped create eGain's AI Healthiness Assessment, a way for organizations to measure whether their AI can actually be trusted.
- 2:03That problem has a name: Steve has spent two decades in regulated industries, including 13 years at knowledge management company Panviva, so he knows a thing or two about getting the right answers to the front line.
- 2:14That problem has a name: Steve, thanks for joining us today.
- 2:17Steve: Thanks for having me, Brian.
- 2:18Steve: I've really been looking forward to this.
- 2:21Brian: Thanks.
- 2:21Brian: Thanks.
- 2:21Brian: Yeah, us too.
- 2:22Brian: And let me just start off with, um, a first question.
- 2:27Brian: Just out of curiosity, what, what does head of innovation entail for a company like eGain?
- 2:33Brian: What are you do-- What do you spend your days doing, thinking, crafting, brainstorming?
- 2:39Brian: That sounds like probably the, the most exciting or interesting job I've, I've ever heard of.
- 2:45Steve: I, I'll tell you, when it, when it came about, I was actually very excited because I get to look at not only what the past had, but what we need today that is also looking forward into the future.
- 3:04Steve: So let me, let me explain that a little bit more Especially around AI, there are so many areas to consider when it comes to AI that we need to move and evolve the entire platform moving into the future the right way.
- 3:23Steve: Because if we don't govern AI at the core or at the source, AI will give us wrong answers.
- 3:32Steve: And in the case here, healthcare could be affected in many ways.
- 3:38Steve: But when healthcare gets affected by a wrong answer, there's a waterfall effect.
- 3:43Steve: Other things happen.
- 3:44Steve: Organizations find that not only that one answer might have caused, caused all kinds of, um, grievances and reimbursements.
- 3:55Steve: Those might have then affected their scores, which means the government money coming back in might have been affected.
- 4:01Steve: So one bad answer is not just isolated.
- 4:06Steve: So I get to look at all of that and say, "Where does this need to be?" And then, "What should we be building now to fill the white space around AI?" And it's, it- it's kind of a, it's a really cool job, to be honest with you.
- 4:24Steve: Uh, it- it's not just developing products, it's developing products that humans need.
- 4:31Steve: And even though we, we will sell our technology to the hospital or the health plan, or even other industries from our other divisions, it's really meant as to how it affects the end consumer, whether they're called a patient, whether they're called a member, or whether they're called a customer.
- 4:48Steve: It's how they're affected by all of this, and making sure that we're doing it in a, in a, uh, governed manner and a trusted manner.
- 4:57Brian: Let's talk about that, okay?
- 4:58Brian: So, so, and- and this is one of the things that I think you probably grapple with on a, on a, on a constant basis, you know.
- 5:05Brian: Um, an AI gives you an answer and it's wrong, but, but you believe it anyways because it sounds so confident, just like all the answers that the AI gave you that were right, right?
- 5:18Brian: So, so what is that, and how does it happen?
- 5:22Steve: Well, there's a couple of things here to, to really unpack.
- 5:26Steve: AI wants to please.
- 5:30Steve: And I've interviewed AI in, in many different ways as to how it wants to operate.
- 5:36Steve: AI wants to give an answer no matter what.
- 5:40Steve: So to that end, it's going to give an answer, whether it generates an answer or it can draw an answer from a trusted source.
- 5:49Steve: It's gonna give an answer.
- 5:51Steve: Now, when it's wrong, it's, as you said, confidently wrong.
- 5:58Steve: If you call it out, it will say,"Good catch. Thanks for pointing that out to me." And then it will go back and reiterate and maybe dig a little bit deeper and figure it out.
- 6:10Steve: But the problem is, many times in healthcare, we don't have that luxury of saying, "Go back and figure it out," because it's hardwired into workflows and process and applications that we don't have direct interaction with to say, "Let's change the prompt." When we're using ChatGPT or Claude, we get the ability to go and change the prompt and tell it it's wrong, and that's a, that's a very offline feeling because it's a back and forth, uh, conversation.
- 6:44Steve: But when it's hardwired behind an EMR system or hardwired behind a claims system, the person running that doesn't necessarily have the ability to stop and make that change.
- 6:57Steve: So when things do go wrong, that's when problems happen.
- 7:03Steve: But if we look at the other side of this as to why do they go wrong, right?
- 7:07Steve: What is the, what is the problem there?
- 7:11Steve: It's because- The underlying data that it's being drawn from.
- 7:16Steve: All right?
- 7:16Steve: So let's, let's, let's think of this as when we're asking a question, okay?
- 7:23Steve: But maybe even let's ta- take it on a personal basis.
- 7:27Steve: I ask my friend, "What's the phone number to the pizza place down the street?" I don't want him to read through a chapter on all the pizza places that were ever developed and then find an answer for the phone number.
- 7:42Steve: I want him to just give me the answer.
- 7:45Steve: Well, the problem with the way AI operates is we're throwing a lot more information that it has to consume for every action that we ask.
- 7:56Steve: So we're, we're forcing it to go through more.
- 7:59Steve: Now, that more part could be old.
- 8:03Steve: It could be last year's.
- 8:05Steve: It could be 10 years ago, right?
- 8:08Steve: Because it might be looking at a, a, a repository in SharePoint or something else where we've stored things in there for years, and we're now pointing AI to a repository that was never cleaned out, right?
- 8:24Steve: I mean, it, it's the analogy of like nobody ever wants to clean their garage out or their basement, but when it comes to AI, that foundational information has to be not only clean, it has to be trusted, and there has to be a governance layer over this as a trusted layer so no matter what question we ask, it's going in and it's saying,"Okay, I know how to get to the answer, and this is the most relevant answer.
- 8:53Steve: This is the most recent answer.
- 8:55Steve: Oh, and this is the governed and trusted answer.
- 8:59Steve: I can put my stamp on it and hand it back to the user."
- 9:02Brian: Okay, so let me see if I have this right.
- 9:04Brian: So AI model drift is ChatGPT getting something wrong, um, for a- any number of reasons.
- 9:12Brian: Maybe it's just, um, looking at, looking at the information out on the web and it's, and, and, and it's making decisions that are incorrect, um, making assumptions, all of that.
- 9:22Brian: Knowledge drift, AI knowledge drift is when the knowledge that the AI is accessing is actually incorrect.
- 9:29Brian: Is that right?
- 9:30Steve: Correct.
- 9:31Steve: Correct.
- 9:31Steve: Because it's assembling it from multiple places too.
- 9:34Steve: It, it may not find it immediately, or it may just find the first one And think that that's the correct one.
- 9:43Brian: So wh- why is knowledge drift more dangerous than us- uh, than model drift when it comes to healthcare?
- 9:52Steve: Well, the w- the best way to look at it is that knowledge is the foundation of how we turn things into action, okay?
- 10:02Steve: So we're always looking for the shortest distance between getting the knowledge that we can turn into action, okay?
- 10:11Steve: Now, the model has its own taxonomy, it has its own information, it was trained on all of the information, but the derived version of that, that is going to give us what we need to perform an action, is what we are looking for, okay?
- 10:30Steve: So we need that distillation process to happen, which then gives us the background, the knowledge.
- 10:37Steve: Maybe it's pulling live data from various sources, and it's giving us the encapsulated information that allows us to make decisions.
- 10:49Steve: So if you think about it, it is the basis for the human decision support system, and now we can take that information and we make a decision, but what we're really doing is we're turning it into action.
- 11:03Steve: And the time from getting information to turning it into action is the most critical time, and getting that information right so that we do it right the first time and every time, that's the key to the problems that are, are arising.
- 11:22Steve: It's all of that happened behind the scenes.
- 11:26Steve: We get a package of information, and we need to be able to trust it.
- 11:31Steve: The only way that we can trust it as humans taking AI information is if it is governed, trusted, but the most important part is we can draw a line to its source.
- 11:46Steve: Okay.
- 11:46Steve: It tells us where it came from.
- 11:48Brian: So a hospital or a health plan's contact center, they, they field thousands of questions a day, right?
- 11:54Brian: I mean, and the AI is answering these thousands of questions.
- 11:57Brian: Most of the time they're answering them correctly, um, based on what you're talking about here, right?
- 12:03Brian: That direct line to that information base, um, so that, so that, so that people get the answer that they need and they can make the, the decisions that they need to make and they a- take the action they need to take, right?
- 12:14Brian: So when a rule changes, say a plan stops covering a medication, s- uh, somebody actually has to go tell the AI that or they get it wrong.
- 12:25Brian: Is that the only problem here?
- 12:27Brian: Is that the whole thing?
- 12:29Brian: Is knowledge drift just a matter of, of, of a human forgetting to hit update or a system that's not working properly so that the AI is up to date?
- 12:38Brian: Or is there more structurally to it that is, that, that is sitting underneath?
- 12:43Steve: Well, there's, there's two… You, you bring in, bring up two problems there.
- 12:48Steve: One is mostly correct, right?
- 12:50Steve: The first one that you talked about was that it's mostly correct.
- 12:54Steve: The problem with most AI-based systems, if they're not built on a governed, uh, knowledge system The pro-- the, the real problem is it has to be able to say, "I don't know." So that-- this is the, the main thing that we should, we should concentrate on, is the fact that when it knows and it can, it can verify and validate the source, that it's, it's coming from the governed trusted source, it can give the answer.
- 13:30Steve: But if it can't, it needs to say, "I cannot give you the answer for this.
- 13:38Steve: I will point you to the direction of where it can be gotten, but I can't give it to you because I can't validate its governance or its provenance from the source." Okay? So that's, that's the one area that we have to be concerned with.
- 13:54Steve: And every healthcare organization in the world, every CIO, CAIO, CDO should be really concerned with that, that AI should be able to say, "I don't know.
- 14:07Steve: I can't draw a conclusion, and I can't give that to you right now." The other side of it, when rules changes, when rule changes or, uh, formulary changes or there's a new way of doing something, that is probably up to the human authoring team that are controlling the, uh, govern- governed information.
- 14:29Steve: But let's go a step further.
- 14:31Steve: The problem in healthcare is we also have regulatory changes that are external to our business, to the four walls of our company, that CMS, as of July 1st, had a hundred and seventy-seven updates.
- 14:45Steve: How do you keep up with a hundred and seventy-seven updates and then be able to say, "Okay, I'm gonna have to have my team read through all those updates," and some of them are hundreds and hundreds of pages, right?
- 14:59Steve: And then they have to look and say,"Okay, we have all of our knowledge that our contact center is operating on." Which one of those processes are affected by those 177 rule changes that just happened by CMS?
- 15:14Steve: That has to be done at scale.
- 15:17Steve: So you also need to be able to say, "I've got to self-monitor the information underneath e-everything.
- 15:24Steve: I gotta monitor it for gaps, but I also have to monitor it for changes external to us through government regulatory changes and others," to say you've got to be scanning for that, looking and saying,"Oh, these 27 documents are affected," and drop them into a workflow for the human knowledge authors to be able to say, "I've got to work on these Monday morning," so that the next call that comes in is now correct based on everything we know in as near real time as possible.
- 16:00Brian: O- okay, so first of all, I want to say, look, let's, let's, let's give AI a little bit of grace here because I don't know how to say I don't know either, and my wife goes crazy over that.
- 16:11Brian: And she gives me all sorts of rules as well, and I never follow them, so you know, I'm just saying.
- 16:17Brian: But, but let me a- let me ask you a more healthcare related question about this and, um, a little bit off the AI path, but, but you know, there are so many points, uh, points of connection there that you're talking about, right?
- 16:28Brian: One of the things is, you know, is the current system of proposed rule changes that CMS has, uh, has always had, you know, that, that process of proposed rules, et cetera, et cetera, is that built for AI systems, especially in regarding, in regard to contact centers?
- 16:48Steve: It is today, right?
- 16:50Steve: So, uh, CMS has the APIs to their repository, to their changes, but then you also have, um, validated third parties that are aggregating all of that information into a way for API access, MCP access, so that you can be getting real-time information.
- 17:13Steve: I mean, look, two years ago, three years ago, we would've just lived with that, and it would've built, been built into the cost of doing business in healthcare, right?
- 17:23Steve: We would've said, "Hey, okay, all those regulations changed." And the next contact center agent that answered a question was answering it to the best of their ability based on the information being static maybe for the previous six months or a year, okay?
- 17:41Steve: And we dealt with those consequences a little differently.
- 17:44Steve: Today, the ability is there.
- 17:47Steve: So we do have the ability now to say,"I'm gonna look at those aggregated regulatory changes and make sure that I'm self-monitoring that governed, trusted layer underneath everything and make sure that everything is right." And that happens overnight in a matter of minutes.
- 18:08Steve: It, it's not during the day while the calls are coming in, but it'll happen every night, and that's one way that organizations now can stay completely up to date.
- 18:19Steve: It doesn't have to be CMS.
- 18:21Steve: It could be any external system that we need to make sure our pro- our processes are affected by,
- 18:31Brian: right?
- 18:31Brian: Okay, so one thing that you n- you, you mentioned a minute ago I wanna go back to.
- 18:36Brian: Um, how do you, how do, how in the world do you, do you build honesty into an AI system over fluency, um, in, in this case, over fluency?
- 18:50Brian: How do you build that sort of layer of honesty to get to the point where an AI would say, "I don't know"?
- 19:00Steve: It's a good, really good question.
- 19:02Steve: And today, the best way to do it is to put up the guardrails at, at the layer right before the information is being given to the human agent or right before it's being given to a self-service system.
- 19:22Steve: So maybe a patient is self-servicing, they're going onto the secure website, logging in, maybe they're logging in via MyChart or what have you, and they're getting that, some of that information being served to them in a self-service mode, or they've picked up the phone, um, or we could say that the self-service went as far as it could, and then they called in.
- 19:47Steve: So the guardrails have to be put up that says, "Okay, we're gonna be getting the information. It's gonna be generated from a typical LLM system," which is most of the layers in there.
- 20:00Steve: They have to be controlled.
- 20:02Steve: They have to make sure that that information is never used external, uh, et cetera.
- 20:08Steve: But then the guardrails have to be put up to say before it goes to the display, right?
- 20:15Steve: So before it's getting written to the screen or before it's getting given as an answer, it has to then look at itself and say, "Do I have a fully trusted answer here with source ties? Can I tie it back to the source and feel confident?" AI wants to feel confident, so you're basically helping it do what it does anyway.
- 20:41Steve: But then you're, you're making sure that the final stage of the prompting chain is, "If not, here's the way you have to answer this." So that's a final layer before it goes to the screen or before it goes to the self-service system.
- 21:00Brian: Okay, one more question about the, the what here a little bit, right?
- 21:03Brian: Um, I'll use my, I'll use my wife as an example again, right?
- 21:07Brian: So, like, she'll say to me, she'll ask me, she'll say, you know, "Hey Brian, did you, did you, um, clean the kitchen?" And I'll say, "Sure." Or I'll say, "Yes." So there's a nuance there, right?
- 21:23Brian: Uh, sure maybe may means that I kind of half did it and I didn't do it all the way.
- 21:29Brian: Yes is definitive.
- 21:30Brian: So does that happen, and does that happen in this construct?
- 21:35Brian: And, and if so, how do you, how do you prepare for that?
- 21:39Brian: How do you, how do you s- uh, how do you solve for that?
- 21:43Steve: Yeah.
- 21:43Steve: In, in, uh, in AI, um, nuance has to be taken out of the equation.
- 21:50Steve: So even the formatting of responses have to be in the actionable, uh, voice, first of all.
- 21:59Steve: So, um, if, if it's giving instructions, it has to be actionable.
- 22:03Steve: Verb, noun, uh, combination, right?
- 22:06Steve: Do this, say this, click here, open this, right?
- 22:11Steve: In the case of where there's am- there could be ambiguit- ambiguity in the interpretation by the human, it needs to speak in a method that is without doubt, right?
- 22:27Steve: So it has to hold up, 'cause the thing is, I mean, at some point down the road, we're gonna be looking at court cases because of answers, and it has, has to be done from today without doubt, right?
- 22:44Steve: So here's the answer, here's the source of the answer, here is the certified version of the answer for you to give.
- 22:55Steve: But it also has to say either do not say, do not do, avoid this.
- 23:01Steve: So you have to also be playing into the compliancy and the risk management, uh, effect at the same time.
- 23:10Steve: This is very complex.
- 23:11Steve: I mean, these are complex waters we're, we're wading in now and, but we have to be thinking about what happens down the road when somebody challenges that.
- 23:22Brian: It's, it's programming discipline into the AI, which is really, quite frankly, pretty mind-blowing to me.
- 23:28Brian: Um, um, okay.
- 23:30Brian: So let's pause for a minute and just take a, take a beat because I wanna ask you about fairness, right?
- 23:39Brian: We're here, here at the, at, here at this show, we talk a lot about the human in the loop.
- 23:44Brian: It's really important that the human is always in the loop, that the, that there's a, there's a human driving, driving the AI development, driving those, those, those out- outcomes for patients.
- 23:55Brian: And, and, and yet what you're kind of saying to me here is, well, we should be holding AI to a higher standard than we hold people.
- 24:05Brian: Because all of the things that I've mentioned and that we've been talking about, people do that all the time in co- in contact centers, in, in personal co- communications, right?
- 24:15Brian: So is, is that right that we should be, uh, we should be stipulating that AI has a higher standard than- Even humans do?
- 24:23Steve: No, no.
- 24:24Steve: I, I think we, we need to look at both separately, because the human in the loop is al- always has to be there.
- 24:34Steve: A- as an example, any of that governed information, no, no AI will hit the button to say, "Publish this out so that the entire contact center will draw from this information," right?
- 24:47Steve: No AI is doing that.
- 24:49Steve: That cannot be the case.
- 24:52Steve: So we also have to train the humans in the loop to be thinking in terms of the what ifs, and making sure that they're reviewing that information in a, in a different light than they did when AI was not present, right?
- 25:11Steve: So the humans become even more important, right?
- 25:15Steve: So we're not doing away with, with the humans here.
- 25:18Steve: Matter of fact, they be- their role becomes more important.
- 25:21Steve: So does the role of the contact center agent that if the… even if there's deflection of the simplistic things, I can't get into my MyChart system or in a health plan, I don't… I need a new card, whatever it is, if we can deflect the simplistic stuff, imagine how much more complex what comes into the contact center is.
- 25:43Steve: Almost advisor level, right?
- 25:46Steve: So the humans that are part of the authoring team for the governed, uh, AI knowledge system have a bigger job, right?
- 25:57Steve: They have to be looking at it in terms of, before I hit the button and this gets published out, I wanna make sure that we've put the guardrails in place, and we have the what if scenarios taken care of It's an extra step.
- 26:14Steve: Now, hopefully the vendor you choose for that underlying, uh, repository and trusted governed system gives you all the tools to be able to do that easily so that it's not taking more time, but it's surfacing the things to your attention quicker so that you can then just look through and say, "Okay, it gave me all of these.
- 26:36Steve: This is good," right?
- 26:38Steve: "I-- And I can move forward with it." That's the, that's the key today, and, um, I, I think that is the, you know, like my day job, those are the things that we worry about, is giving the tools to people, right?
- 26:53Brian: Yeah, about that.
- 26:54Brian: So the, the AI healthiness assessment that you developed, it covers, it covers, as I understand it, it covers five governance pillars, right?
- 27:02Brian: Um, and it, and it does so across about 15 questions.
- 27:06Brian: Knowledge governance, uh, you've touched on that.
- 27:09Brian: Content freshness, hallucination risk, compliance readiness, and operational impact.
- 27:14Brian: One, one, one data point that I found that I just wanna bring it to your attention, and, and I know that you know about this, um, is that, by the way, did you-- The ungoverned medical AI hallucinates on complex healthcare queries at a rate of more than 60%.
- 27:37Brian: Is that, is that right?
- 27:39Brian: A little sidetrack here, but is that correct?
- 27:41Brian: And, and that, does that not say the bounding and boundaries are vital here when it comes to AI and contact centers or AI and-
- 27:51Steve: A-a-absolutely.
- 27:52Steve: So, so let's, let's think about why that's the case, why that number is so shockingly high, right?
- 28:01Steve: And it should terrify every healthcare leader, and almost none of them recognize it.
- 28:08Steve: I mean, I, I've been, uh, I've been able to witness more projects that took a co-pilot type of AI and pointed it to a repository that has been developed since 2015, 2018, right?
- 28:26Steve: So now the, the problem is the AI is not using a taxonomy method to be able to get to the exact information that it needs.
- 28:41Steve: So in the absence of developing a, a good taxonomy, uh, system and, um, the AI can fail more often than not because it doesn't know how to find the outdated and disregard the outdated, the retired, the archived, the old, the,"Oh, this was version two ver- versus version three." Think about how we stored information for many, many years.
- 29:11Steve: We would store information in files and folders And how did we name those files and folders, right?
- 29:20Steve: Did we always name them with the perfect, uh, information in the naming of the file?
- 29:27Steve: Probably not.
- 29:28Steve: Did we build those files to any standard that could hold up to AI's reasoning abilities?
- 29:36Steve: Probably not.
- 29:38Steve: But now we expect to be able to show, tell an AI, "Go over there and find me the answer to this, and look through… By the way, look through the gigabytes and maybe even terabytes of information." So that's the fundamental problem in most organizations. And i- if they're honest, most organizations will answer those questions, and it's free, and you get a great report back from it, and it's just to you to say, "Okay, how do I feel about the way we're, we're looking at our existing data sources?
- 30:15Steve: And is there potential for problems there?
- 30:20Steve: And where should I even begin?" Most people, they don't know where to begin, so, you know, they're, they're, they're cleaning house backwards sometimes, right?
- 30:31Brian: Okay, I've got a question for you on those, on, on, on the questions.
- 30:34Brian: But before I do, I wanna, I wanna sidetrack again because, um, one of the things that we talk a lot about is native AI in systems, that that's better than a vendor bolt-on, for example, right?
- 30:46Brian: That, that, that, that a, a, an AI built into a system natively is able to do ki- what you're talking about, right?
- 30:54Brian: Is that correct?
- 30:55Brian: So w- if you're a healthcare organization, if you're a CIO, for example, and you're looking at making an investment into AI, um, is that a way for you to ensure that your data in, in your organization is mapped properly and correctly to your contact center or to your Ambient Scribes or whatever the AI usage might be?
- 31:18Brian: Is that, is that right?
- 31:19Steve: Well, let's leave Ambient Scribes to the side for a minute 'cause that's a, that's a whole different area and a different, a different, uh, talk track.
- 31:30Steve: As far as AI in healthcare, it's, it's fine for looking to existing, uh, generative platforms, and there are many of them, and there are many versions of them, right?
- 31:47Steve: But the question really is, what is the right one for a particular process?
- 31:54Steve: So, as an example, if you're looking for something that is generating the best process for you to operate on or the best flowchart, or it's analyzing better, right?
- 32:09Steve: Or it's reasoning and providing more research at that point.
- 32:15Steve: So all of those are completely different types of models.
- 32:18Steve: You wouldn't use the same model for all of them, right?
- 32:21Steve: Because they all have their strengths and weaknesses.
- 32:24Steve: Maybe something is more graphic oral or graphical oriented, et cetera.
- 32:28Steve: So the best approach is, how do I use the best model for that particular need?
- 32:38Steve: And that's the way to look at it rather than the one size fits all, because it won't.
- 32:44Steve: You will sacrifice along the way.
- 32:46Steve: I mean, it's the same thing with if we, you know, have one system to cover it all.
- 32:51Steve: We know it's not gonna be strong in every area.
- 32:54Steve: It's gonna be strong in its main area, but there's gonna be a lot of white space that we have to think about, okay, we can sacrifice there, right?
- 33:03Steve: Or we find a best of breed for it.
- 33:06Steve: In this case, when it's coming to what AI is right for each thing, it's we have to look at what's the best one that will produce the best output that we can then also put guardrails around to make sure that it's giving us the answers that we need, so we can turn it into action and be confident with the trusted answers, right?
- 33:28Steve: So a-as an example, I mean, if you look at the thing-- the, uh, platform we produce, we use multiple AI systems Because we're always testing what is the best one to do that particular thing, and that's what we use, right?
- 33:47Steve: And we don't, we don't have to-- You don't have to build your own.
- 33:50Steve: You don't have to get an open source system.
- 33:53Steve: You don't have to have one system.
- 33:54Steve: But just be thinking about when you are looking at, at those, that you're matching the strengths of, and minimizing the weaknesses, the strengths of that particular AI model to the purpose you want it for.
- 34:10Brian: Awesome.
- 34:11Brian: Okay.
- 34:11Brian: So tell me, explain to me how your 15 questions actually can assess something as broad as healthcare across those checkpoints.
- 34:20Brian: And yeah, how does that assessment diagnose the problem and prescribe the treatment?
- 34:25Steve: So what it's really doing is it's getting inside of, inside of your, your head and your organization's head.
- 34:33Steve: It's, it's looking at whether or not you should trust the healthiness of the system that you're operating on.
- 34:44Steve: So the questions all crisscross each other The way the, the way the questions are designed is so that when it asks a particular question and then the next question, it's building on the responses to determine whether or not you feel, and the organization, is doing the right things for a trust level high enough and a healthiness of your particular AI system.
- 35:15Steve: So it's just surfacing the things that you might say, "Well, I had a gut feeling that the answers we were getting when we matched Copilot with SharePoint weren't, weren't things we should be trusting." This is looking at what have you put in place?
- 35:32Steve: How do you approach the, um, the data itself?
- 35:36Steve: How do you approach change within the data?
- 35:39Steve: What's the ongoing, uh, methods that you're using to monitor what's under there?
- 35:46Steve: So it, it's looking at the, the organizational approach to quality of information, right?
- 35:54Steve: And then producing that, but it's producing it as to where does it hit on?
- 35:59Steve: Which pillars is this hitting on?
- 36:01Steve: Is this, you know, is it from the governance side?
- 36:05Steve: Is it from, you know, is it showing that there needs to be more governance put in place, et cetera?
- 36:10Steve: So it's, it's doing that, that kind of a crisscross to make sure that everything is as it should be.
- 36:17Steve: And if not, where do you begin?
- 36:21Brian: Yeah, it's, it's, it's, it's ecosystem healthiness is what it really is, isn't it?
- 36:25Brian: It's, it's, it's checking across all aspects of what delivers answers through the AI, because that's truly, that's truly the situation.
- 36:34Brian: That's truly what, um, what you're getting when you ask a ques- uh, ask a question in, in that context center.
- 36:40Brian: It,
- 36:40Steve: it- Yeah, and it, and it's also making you think-
- 36:43Brian: Yeah… Steve: about the, about tho- those get there, and, and giving you kind of a blueprint, if you will.
- 36:51Brian: So g- so give me an example, uh, an example of hallucination risk in terms of this assessment.
- 36:56Brian: What would come up?
- 36:58Brian: What would get flagged hallucination risk?
- 37:01Steve: So, um, it would probably be more of a combination of the answers to the questions that would then, um, show that the AI system has misperfor- or non-performed properly, um, based on how you've used it, right?
- 37:21Steve: So meaning- If the answers that are coming from your system are things that you say, "Gee, I really wish I could tweak that prompt a little bit. I really wish that I could've asked it a different way or gotten, done it a little differently to get the answer that I wanted," then you know that you do have a, um, a fundamental problem where trust is, is something that is at the forefront.
- 37:53Steve: Now, when trust is at the forefront, that means that you may not be getting the answer that you need to, remember we, we said, to turn it into action.
- 38:05Steve: Am I getting the information that I need so that I can turn this into fundamental action?
- 38:10Steve: And from, uh, from that, looking at ungoverned medical information, formulary, formulary things, um, or, or even triage in a hospital.
- 38:24Steve: I mean, think about a, a call that comes in.
- 38:27Steve: You don't want AI handling triage, right?
- 38:30Steve: You, you can't have it saying, "Oh, do you have chest pains right now?
- 38:35Steve: And if you do, how acute are they and where are they centered?" Uh, right?
- 38:40Steve: Because that's not gonna help.
- 38:42Steve: So unfortunately, the, this is really looking at, as you said before, it's looking at the organization's approach to the healthiness of the AI and have they put in place the things that will then prevent that hallucination risk to ever be a problem.
- 39:04Steve: Now, can you get all hallucination out?
- 39:06Steve: You can get it down below 1%, but it's a lot better than 60%.
- 39:12Brian: I wonder what the human hu- hallucination percentage is.
- 39:17Brian: Okay.
- 39:21Brian: Yeah.
- 39:21Brian: All right.
- 39:22Brian: So, so I also see it as a scaffolding maybe, right?
- 39:25Brian: And correct me if I'm wrong, but I see that the categories in your questions are a scaffolding to more AI autonomy eventually over probably a longer period of time.
- 39:36Brian: And I think that, like, many, many healthcare executives probably are thinking about this pretty actively.
- 39:44Brian: How do I-- How does AI scale from answering questions to taking action, such as scheduling routing decisions that sit close to clinical judgment but are not clinical judgment, right?
- 39:58Brian: Um, where are you comfortable there?
- 40:01Brian: What does an organization need to see or, I guess, prove to, through, through this assessment that they can actually grow in that way?
- 40:15Steve: I, I guess it, it-- the best way to approach it is it depends on the organization's appetite here because, I mean, there… Anything that is, should be answered by a, a medical professional, that's the way it should go, right?
- 40:31Steve: Just like you don't have a contact center agent triaging a chest pain, um, person that just called in.
- 40:40Steve: You just can't.
- 40:41Steve: There has to be a, a, an immediate warm transfer to a nurse practitioner or to a doctor immediately because you don't know whether or not you can let them make an appointment, have them come to the ER, or you should be sending an ambulance to their home right now, right?
- 40:59Steve: So, um, when it comes to medical advice or even funneling things to the practitioner That's really gonna be an answer that the CMO has to make.
- 41:13Steve: Just like the CMO of a hospital makes the decision as to what kind of triage a contact center agent can do or not do, the chief medical officer is the one with the final say, uh, on that, and it should be that way, right?
- 41:27Steve: Should AI be, be narrowing down the potential?
- 41:33Steve: That's their decision.
- 41:35Steve: None of us, I'm not in-- I'm not certified to be able to make that decision.
- 41:41Steve: And where they feel comfortable along the way is also gonna be something that they have to look at, what if something goes wrong, right?
- 41:50Steve: They have to look at the what if scenario down the road, and that becomes a risk management equation.
- 41:56Steve: Um, I, I wouldn't go too far, uh, with that.
- 42:00Steve: Um, and just like with, you know, the evolution of ambient scribes, they're, they're taking information from that session, and they're sending it up to a server, to a third-party, uh, processing, uh, company.
- 42:17Steve: There are problems already, uh, with that, and there were too many court cases that are already starting to happen, um, you know, around that.
- 42:24Steve: So, you know, a lot of this becomes the comfort level of the individual institution and, um, the final say should be the human in the loop, the chief medical officer.
- 42:36Brian: All right.
- 42:37Brian: Let's, let's go back to the beginning a little bit and, and, and kind of begin to sort of encapsulate this.
- 42:43Brian: Um, one of the things that you've written about, and I'm really-- I, I really enjoyed it, I really was fascinated by it, was you wrote about how all of this technology is essentially promising us more connection, right?
- 42:59Brian: But is it?
- 43:01Brian: Essentially, you pose the question h- you pose the question that w- we use technology to create more connections, but is it just more performance and less actual legitimate connectivity?
- 43:17Brian: Fundamentally, that's what you're trying to do, right?
- 43:20Steve: Yeah.
- 43:21Steve: Yeah.
- 43:21Steve: Uh, I mean, it's a-- it, it, it's one we could discuss for hours.
- 43:26Steve: Um, but the reality is, is that we're trying to put, especially in healthcare, we're trying to put certain things in place that allow medical practitioners to get back to care.
- 43:39Steve: Okay?
- 43:40Steve: Now, i-if you've read anything of the last 10 years, most medical professionals have felt like their day has gotten much longer because of the note-taking, because of putting all of the notes into an EMR system at the end of the day or at the end of every session.
- 43:58Steve: And it's taken away the, the seven minutes that they used to have face-to-face with a patient, and it's reduced it down to four minutes on average.
- 44:06Steve: Well, um, the whole, you know, the whole idea be-behind ambient listening in the, in the actual session was so that the medical professional is not behind a screen, but they are face-to-face with the patient for much longer and are listening to the real problems and are being able to diagnose and being able to ask the right questions.
- 44:27Steve: So to be able to connect more doctor-patient or nurse practitioner-patient, uh, has been the goal in most cases.
- 44:37Steve: Has that been the, the result?
- 44:41Steve: Has that been the reality?
- 44:43Steve: Uh, or does it mean that we're seeing more patients in the course of a day and we're still having the, having the same amount of time, um, with them?
- 44:54Steve: So connectivity shows up in many different ways, and the ability to connect is, is kind of a… We start getting used to certain things, and we need to be able to change that.
- 45:09Steve: I, I see you have a question.
- 45:11Brian: No, no.
- 45:12Brian: Yeah, yeah.
- 45:12Brian: No, I, I, I, I… Amazing, really.
- 45:16Brian: And, uh, w- uh, listening to you, I, I was so excited because I wanted to just sort of make the point that, you know, those seven minutes are th- those are the moments that matter.
- 45:27Brian: Those are the things that people go to the doctor for.
- 45:29Brian: That's where the doctor actually is practicing at the top of their license, when they're eyeball to eyeball with that patient, right?
- 45:37Brian: And AI, that's the, that's the goal of AI is to be able to facilitate that and to, to get that for that interaction away from typing notes and maintaining EMR systems, right?
- 45:55Brian: That's the dream.
- 45:56Brian: That's the, that's the, that's the aspiration of AI.
- 46:00Brian: Um, let me, let me tag on a question here What you're doing in terms of helping humans create, monitor, and maintain honest AI, does that lead to that actually becoming part of the healthcare ecosystem?
- 46:22Steve: I, I wanna say yes, um, because At the, at the end of the day, anything that we put in place that has the AI involved has to solve some problems, right?
- 46:39Steve: And it can't just solve corporate problems or administrative problems.
- 46:44Steve: It has to solve real problems.
- 46:47Steve: And by making sure that anything that is being served via AI is right the first time, right every time, and allowing the humans that need that information at a moment's notice to be able to get the right information so that they can perform the way they need to.
- 47:12Steve: And in healthcare, that's care.
- 47:15Steve: All right.
- 47:15Brian: Is that, is that your North Star?
- 47:17Brian: When it comes to what you do every day, is that, is that your North Star to, to develop these systems that actually bring that humanizing factor?
- 47:27Steve: It, it is.
- 47:29Steve: It is.
- 47:30Steve: Um, uh, I have to go through an awful lot of tech and a, an awful lot of other things to get there, but at the end of the day, I'm an experience person, right?
- 47:41Steve: If, you know, if you look me up, you see customer experience, patient experience, member experience for my entire career.
- 47:49Steve: And I learned it from my dad, who was a, a plum- a master plumber, and he taught me, you shake their hand, you look them in the eye, and you over-deliver on every promise, and you never falter, right?
- 48:02Steve: So at the end of the day, I think in terms of better experiences that make a difference to people's lives, and technology helps us get there.
- 48:12Brian: Yeah.
- 48:13Brian: Steve, thank you so much for joining us.
- 48:16Brian: Thank you for this conversation.
- 48:18Brian: Um, I, I wish more AI discussions were like this.
- 48:22Brian: People, you know, people love to stargaze about what A- AI might do someday and, and, and while that's great and, and it's, it's exciting, it's not immediate and sometimes not as useful as it could be.
- 48:37Brian: What you're talking about is actually real challenges facing healthcare right now and, and you, and you gave us some solid solutions to address and to think about as, as we move forward to try to reach to those stars and try to reach to those North Stars that we see up ahead of us.
- 48:56Brian: But there's a lot of work for us to get there, and this conversation is the type of conversation that helps pave the way.
- 49:03Brian: Um, and that's what we're all about here at Humanify Health Tech.
- 49:06Brian: So, um, thank you- For this practical discussion.
- 49:11Brian: Um-
- 49:11Steve: Oh, it's been my, my pleasure, Brian.
- 49:12Steve: Thank you for having me on.
- 49:14Steve: Um, uh, I love talking about this, and I hope it helps somebody's lives out there.
- 49:19Brian: Steve, this is the conversation I wish more AI discussions were about.
- 49:23Brian: People love to stargaze about what AI might do someday, and while it makes for great quotes, it's not immediately useful.
- 49:32Brian: But you talked about real challenges facing healthcare AI right now, and you gave us solid solutions to address them right now.
- 49:41Brian: That's what Humanify Health Tech is all about.
- 49:43Brian: So thank you for the practical discussion.
- 49:46Brian: We're going to let you off the hook now before we jump into our recap.
- 49:50Brian: Thanks again, Steve.
- 49:55Brian: All right, recap time.
- 49:57Brian: Here are the top three things I took away from my conversation with Steve today.
- 50:01Brian: First, drift is the default outcome of a system built to sound complete instead of admitting uncertainty.
- 50:09Brian: Fixing it means redesigning an agent to say, "I don't know," and bounding what it's allowed to draw from before it ever answers.
- 50:18Brian: Second, the riskiest failures aren't the loud, obviously wrong ones.
- 50:23Brian: They're the quiet misreadings that pass as correct, sometimes for months, before anybody notices.
- 50:29Brian: That's why some health systems now hold their AI to a stricter standard than
- 50:34they'd hold a person: governed access, constant monitoring, and clear ownership.
- 50:41they'd hold a person: Finally, underneath the framework is a question Steve had asked about his own
- 50:45life before he ever asked it about AI: whether a system that promises trust is actually earning it or just performing it.
- 50:54life before he ever asked it about AI: AI healthiness is his attempt to make that distinction something you can measure instead of something you just hope is true.
- 51:02life before he ever asked it about AI: And that should do it for today's episode of Humanify Health Tech.
- 51:05life before he ever asked it about AI: As always, thanks for tuning in.
- 51:08life before he ever asked it about AI: If you're joining us on Spotify or YouTube, we'd love to hear your thoughts in the comments, and don't forget to leave us a rating while you're there.
- 51:16life before he ever asked it about AI: Once again, I'm Brian Chee from Net Health.
- 51:19life before he ever asked it about AI: I'll see you next time.