Latest / AVBCC Value-Based Voices / AI at the Point of Care: Inside OpenEvidence's Approach to Smarter Clinical Decisions
Transcript
- 0:00Speaker: Welcome to Value-Based Voices, a podcast from the Association for Value-Based Cancer Care.
- 0:05Speaker: Each episode dives into the shifting terrain of cancer care in the United States, exploring what value means in today's clinical, policy, and patron-centered environments.
- 0:14Speaker: Our mission is to spark, informed dialogue, promote transparency, and equip every stakeholder from providers to payers to patients with the insight they need to navigate cancer care with clarity, confidence, and purpose.
- 0:28Speaker: My name is Jayson Slotnic, partner in health policy strategies and AVBCC board member, and I'll be your host for this episode.
- 0:40Speaker: Today I'm thrilled to be speaking with Dr.
- 0:43Speaker: Sam Mullangi, a medical oncologist at Tennessee Oncology and VP of Clinical Strategy for one of the most powerful AI platforms currently in the healthcare space, Open Evidence.
- 0:54Speaker: Sam, nice to meet you and welcome.
- 0:56Speaker: And give us a few words about your background before we jump into open evidence.
- 1:05Speaker 2: So I'm Sam Mullangi.
- 1:06Speaker 2: I'm a community medical oncologist.
- 1:08Speaker 2: I work at Tennessee Oncology and I still spend part of my week seeing patients.
- 1:13Speaker 2: The rest of the time I spend uh at Open Evidence working on some product and partnerships.
- 1:19Speaker 2: Open Evidence is a clinical decision support platform that's built on AI.
- 1:24Speaker 2: I came to this company less from a technology background and more from the frustration of being an oncologist who knows that the evidence in our field turns over faster than anyone can really keep up with.
- 1:36Speaker 2: Something like almost every week it feels like there's a new guideline, new approval in oncology.
- 1:41Speaker 2: And so for me, AI was not an abstraction.
- 1:43Speaker 2: It felt like the first tool that I had encountered that could actually close that gap at the point of care.
- 1:49Speaker 2: So I've been with Open Evidence for about eight months now, and it's been a complete blast.
- 1:54Speaker 2: And so very excited to be here with you.
- 1:56Speaker: Well, we are excited to have you.
- 1:58Speaker: As you know, value-based cancer care, uh, we have a wide range of stakeholders, payers, manufacturers, patients, and I'm a big believer that AI can uh be such a significant disruptor.
- 2:11Speaker: It could equalize access to care, the latest information, and be a very powerful tool uh in the clinical decision and the clinical support uh decision making for oncology.
- 2:24Speaker: But before we get into those ramifications and implications, uh tell us how it works.
- 2:30Speaker: You know, tell us how the technology, the backbone is built.
- 2:33Speaker: And for those of you who are listening while you're driving, please do not go to openevidence.com right now.
- 2:39Speaker: Wait till you're not driving.
- 2:41Speaker: Uh, but for those of you who are at your computer and and are able to, I strongly recommend perhaps that you go to it while while we hear from Sam and how it all works and the backbone and what is an LLM and what is an RAG and the power and the relationships you have developed in developing this tool.
- 2:59Speaker: So, Sam, take it away.
- 3:02Speaker 2: Um, so that's right.
- 3:03Speaker 2: So, open evidence is a RAG-based model, which uh RAG is an acronym that stands for retrieval augmented generation.
- 3:10Speaker 2: Um, and the simplest way to think about it is that rather than answering from memory, the model is required to go look things up before creating an answer.
- 3:19Speaker 2: Um and so when a clinician asks a question, the system will first retrieve relevant material from our body of literature that's in our content library.
- 3:27Speaker 2: And we have licensed content from you know all of our partners, like the New England Journal of Medicine, JAMA, the NCCN guidelines, ASCO guidelines.
- 3:36Speaker 2: Um, and so the model composes its answer that's grounded in that evidence base, and there's citations that are available that a clinician can then go click and verify and read if they have access to those journals.
- 3:48Speaker 2: So it's kind of like the model is using an open book exam rather than a closed book, you know, memory-based exam.
- 3:54Speaker 2: And I think that matters for a couple reasons.
- 3:57Speaker 2: The first is that when you're anchoring to sources that actually exist, the answer is going to be more accurate and potentially more current.
- 4:05Speaker 2: Um, updating the library is easier, for example, than fully retraining like an AI model.
- 4:11Speaker 2: And so you get a little bit more accuracy and definitely more recency.
- 4:16Speaker: So people hear AI and at least commonly they're worried about hallucinations.
- 4:22Speaker: They're worried about the technology getting it wrong.
- 4:25Speaker: Yeah, I don't know.
- 4:26Speaker: Do I trust it?
- 4:27Speaker: What have you guys done?
- 4:30Speaker: What has open evidence done to make sure that, or the best that you possibly can, that the technology is not hallucinating?
- 4:38Speaker: I know you said right there are some references to it, so it can be verified and validated, but what else has open evidence done to limit the best that it can those hallucinations?
- 4:49Speaker 2: Um, so our rag-based system, the architecture of it reduces hallucinations substantially.
- 4:55Speaker 2: Nothing will ever reduce it to 100%.
- 4:58Speaker 2: And so anyone who tells you that should you should be skeptical of.
- 5:01Speaker 2: But I think what uh a rag-based model really kind of changes is the shape of the problem.
- 5:06Speaker 2: So less so the issue being hallucinations and more so about um what I would call maybe a wayfinding problem.
- 5:13Speaker 2: Um, when you retrieve, when you ask a question, and suppose a question is uh asked to the system that's not um the right question to have asked.
- 5:25Speaker 2: Um, you know, the way that the question is formulated is not the right way it should have been formulated.
- 5:31Speaker 2: It's going to retrieve a source that might be orthogonal to what you're asking.
- 5:35Speaker 2: It may not retrieve the right source, um, or it might retrieve um, you know, a weaker source, for example, where a better one exists.
- 5:44Speaker 2: And so this is always kind of like an optimization problem.
- 5:46Speaker 2: It's not as simple as kind of throwing everything into a content library and just saying, like, aim aim your uh retrieval at that.
- 5:53Speaker 2: There's actually like a fair amount of work that we have to do within our own logic to try and explain to the model kind of what sources are to be trusted, and when there's um evidence that's kind of um, you know, contradictory to each other.
- 6:09Speaker 2: For example, you have one medical society stating the facts in a certain way, and a different medical society maybe using different thresholds or different guidelines, etc.
- 6:19Speaker 2: Um, our we what we do at open evidence is we try to surface both of those in an answer.
- 6:24Speaker 2: Um, so our answers will be like relatively sizable um essays, and the user can kind of go in through and kind of parse um how to think about the problem and and kind of continue down the rabbit hole of looking at those sources.
- 6:38Speaker 2: We also actually recently added um a layer on top of this called evidence grade, which is um, you know, when you have, for example, the model is retrieving seven to ten sources to write an answer.
- 6:53Speaker 2: The user may not immediately know from looking at that how high quality was that evidence base in answering this question.
- 7:01Speaker 2: And you know, in medicine, there's going to be a lot of areas where the evidence is generally weak.
- 7:06Speaker 2: Um, you know, that's truly the art of medicine.
- 7:08Speaker 2: There's going to be trials that are not head-to-head comparisons, or there's going to be kind of the art of medicine, how you practice in the absence of any grade data.
- 7:17Speaker: Now, is this modeled after, sorry to interrupt, is this modeled after NCCN has those evidence blocks, right, where they grade the evidence.
- 7:23Speaker: Is this part, is that part of what what you're licensing in?
- 7:26Speaker: Is that kind of grading scale?
- 7:28Speaker 2: Yeah, it's based on the grade framework.
- 7:30Speaker 2: And so that's also what the name uh alludes to, evidence grade.
- 7:34Speaker 2: Um, but the grade framework looks at kind of the whole body of literature against one topic.
- 7:39Speaker 2: So, like, what is the evidence for something like uh maintenance therapy after you know complete remission?
- 7:46Speaker 2: And it will look at every paper in that corpus and assign it a grade.
- 7:51Speaker 2: What evidence grade is doing in open evidence is just looking at those seven to ten references that it's retrieving and assigning a grade to those seven to ten.
- 7:58Speaker 2: And that's because our questions can be really nuanced and really kind of detailed.
- 8:03Speaker: Okay, so you just raised what I want to focus on next.
- 8:06Speaker: As somebody who's been playing with AI for a while, um what I want our listeners to really understand is what you said, which is asking the question is really 90% of this at least.
- 8:19Speaker: How you ask it, how detailed you are going in really matters.
- 8:24Speaker: What you get in is what you get out.
- 8:26Speaker: So give us some advice on based upon two different perspectives, right?
- 8:32Speaker: I'm a patient now, I'm listening to this, or I'm sort of not a physician, let's say.
- 8:37Speaker: What would be the best sort of way to start my interactions with open evidence, right?
- 8:42Speaker: What would be like the best question for me to ask?
- 8:44Speaker: How should I think about uh interacting at least initially with open evidence?
- 8:50Speaker: And then we'll get into the physician part of this and how it's changed practice.
- 8:54Speaker: But for now, let's focus on the non-physician.
- 8:56Speaker: What how do we what how should we be interacting with open evidence?
- 9:01Speaker 2: So open evidence right now is still gated to users who have an NPI.
- 9:06Speaker 2: So oftentimes that's going to be the physician, the advanced practice provider, or uh a clinical pharmacist who has some patient care responsibilities.
- 9:15Speaker 2: And so uh for the most part, I think you know, we when we think about prompting, which I was the genesis of your question, certainly how to prompt how to prompt effectively is a whole kind of I think new educational pillar that has to be kind of put into practice, both for physicians in practice and also for those who are training.
- 9:33Speaker 2: Um I mean, so at Open Evidence, we've published a whole guide about how to prompt effectively.
- 9:38Speaker 2: Um there's a difference between the specificity of when you say like just management of type 1 diabetes mellitus and without any further context to, you know, I have a patient with type 1 diabetes and they are in uh, you know, uh their glucose is out of range and they're got ketones in their urine, kind of like what is the appropriate management?
- 9:58Speaker 2: I mean, the level of specificity really will give you a much more relevant and actionable answer.
- 10:03Speaker 2: You can also tell open evidence to specify the output format.
- 10:07Speaker 2: So you can say, I want the literature, for example, as a table organized by importance, or I want this organized by which guideline, which guideline body issued it, um, or by recency.
- 10:20Speaker 2: So you can ask the model to change its output based on what you're looking for.
- 10:23Speaker 2: You can also clarify like what outcome you're reaching for.
- 10:27Speaker 2: Are you reaching for the most recent literature?
- 10:30Speaker 2: Are you reaching for alternative treatment options and somebody who's not wanting standard of care?
- 10:35Speaker 2: Are you looking for a differential?
- 10:37Speaker 2: Are you looking for a diagnosis?
- 10:39Speaker 2: All of that helps.
- 10:40Speaker 2: Um, and and so anyway, so I think that there's actually a lot there in terms of how you program.
- 10:45Speaker: You can ask it for what drugs they suggest should be treated.
- 10:49Speaker: You can compare drugs to each other, you can compare the product inserts of those drugs to each other and generate graphs and charts.
- 10:56Speaker: At least that was been my experience with open evidence, uh, playing with it.
- 11:00Speaker: So it it it really is a very powerful sort of uh information organizational tool at some level, but that's not what you really it's not what it uh should be doing, right?
- 11:13Speaker: So, what is open evidence's vision on how it should be used in the physician's office?
- 11:18Speaker: How is it changing physicians' minds or not?
- 11:22Speaker: So when you do this onboarding of how to prompt effectively, um I love that phrase by the way, what then does open evidence say to physicians what to expect?
- 11:32Speaker: What is uh what have you guys seen?
- 11:34Speaker: There's a lot of questions in there.
- 11:35Speaker: Help the audience understand how physicians are adopting this and what they're doing with it.
- 11:41Speaker 2: Um I mean, we see open evidence being adopted across every specialty, both generalist specialties, um special specialists as well as proceduralists versus kind of the medical folks.
- 11:55Speaker 2: And I think the way in which it's being used very much depends on who's using it.
- 11:59Speaker 2: I find that, you know, for folks, for example, in the emergency room are oftentimes asking for diagnosis, differential diagnosis types of questions, for example, or very like um point of care recommendations for um you know triage and stabilization.
- 12:16Speaker 2: In my oncology practice, I mean, I'm already prior to open evidence, I think all oncologists are in the habit of referencing guidelines or looking up literature or curbsiding, you know, colleagues.
- 12:28Speaker 2: And so it's kind of just replaced those types of activities that I do anyway from different platforms onto this platform.
- 12:35Speaker 2: Um and so oftentimes I'm asking very, very nuanced questions around things like I have a very specific clinical scenario.
- 12:41Speaker 2: Um, how should I adjust my treatment protocol according to the literature?
- 12:45Speaker 2: Or, you know, I know that there are some new treatment paradigms in breast cancer, but I can't fully recall right now what this patient would be eligible for because, you know, like there's a huge difference between reading about a trial and the comfort of your armchair, and then trying to apply it in the moment when you see the right patient who might walk in through your door.
- 13:02Speaker 2: Um, even things like, you know, the nuances about clinical trials, you can look up in any um pamphlet or um compendia, sort of things like what is the uh side effect profile of a particular drug?
- 13:17Speaker 2: What is actually harder to find is information like what was the median time to onset, you know, for the side effect in this drug.
- 13:24Speaker 2: And I have patients who I use new drugs all the time, you know, like I gave somebody um Datraway um recently, and within a week she was describing shortness of breath.
- 13:35Speaker 2: And I know shortness of breath is described as part of that drug, but I was like, what are the odds that this is actually a treatment-related side effect versus a complete third confounder, you know, like maybe she has a viral syndrome or something.
- 13:47Speaker 2: Looking through the literature and sort of saying, actually, the median time to side effect here was about six weeks helps me kind of triage that, but that requires just such nuance in terms of how you look something up.
- 13:59Speaker: Right.
- 13:59Speaker: So the way I'm I'm curious about this, right?
- 14:02Speaker: The way I have been taught to treat AI is as an employee, right?
- 14:08Speaker: You're a subject, right?
- 14:09Speaker: You're a physician, I'm a recovery lawyer, and we have 30 years each respectively.
- 14:14Speaker: We're experts in our field.
- 14:16Speaker: And the way I've been taught to treat AI is as a re a series of research associates, as employees because I know what questions to ask and really get into the weeds in detail.
- 14:26Speaker: And um, I'm amazed at what I'm able to get back real quickly, and I find myself constantly engaging it's really uh up my game for lack of a better phrase, for really understanding the answer to hand and being able to explain it and putting it out.
- 14:40Speaker: I presume it's the same way for you, what the way you've just described the real some of the power of open evidence for an experienced physician is you sort of know what it is, and now you can find the answer or double check and triple-check everything immediately.
- 14:58Speaker: Is that has that been the uh feedback that you've received from the experienced physicians who have used it so far?
- 15:08Speaker 2: Yeah, I think so.
- 15:09Speaker 2: I think for for folks like me who are not in training anymore and who generally have seen, like, I mean, I see I have seen hundreds of patients just this year alone, and I and I'm pretty experienced at this point, but I still have be surprised how many questions continue to come up every single day in clinic.
- 15:26Speaker 2: Um and even just double checking my own intuition about um, oh, I know that there's something here that doesn't sit right with me and I'm gonna look this up, like that happens numerous times a day.
- 15:37Speaker 2: But you know, even just there's a lot of things that I think in medicine, um, you know, we don't there's there's such a friction to looking something up that sometimes the easiest thing to do is just not to look it up.
- 15:48Speaker 1: Right, understood.
- 15:49Speaker 1: Yeah.
- 15:50Speaker 2: You know, um, where I, you know, if you're seeing 20 to 25 patients a day and fitting in the notes and fitting in the hospital consults and everything else, like the time that you truly have to look something up is so limited.
- 16:02Speaker 2: And I think every patient has this experience of taking stuff home mentally, you know, that mental load of caring, going home and just driving on the way home and sort of thinking, oh gosh, like what was going on with that patient?
- 16:12Speaker 2: And trying to like just log back into the computer, you know, after bedtime and stuff like that.
- 16:17Speaker 2: Um, I think this is just solving for that type of work that we all have to do anyway, but in a much more efficient way and at the point of care when you're actually in that decision-making seat.
- 16:29Speaker: So you uh uh safe to say, right, have believe that it has dramatically improved, right?
- 16:34Speaker: Patient care outcome, your ability to communicate patients and all of the aspects that you just described.
- 16:40Speaker: What about the payer interaction?
- 16:43Speaker: Has there been any payer, right?
- 16:47Speaker: You're community oncologist, right?
- 16:49Speaker: You interact with all the payers.
- 16:51Speaker: What are do they have any thoughts about open evidence?
- 16:55Speaker: Have they shared them with you?
- 16:56Speaker: What is what has been that relationship so far?
- 16:59Speaker 2: Um, if anything, in fact, I think we've had some interest from payers asking whether you know we could potentially make this available to their entire clinical workforce or even make an API available, which unfortunately we don't have an API that is available.
- 17:13Speaker 2: But actually, I think that like this is not necessarily misaligned with sort of payers' interests and incentives.
- 17:20Speaker 2: I think payers spend a lot of time trying to get physicians onto guideline concordance.
- 17:24Speaker 2: Um, and a tool that is surfacing what is the guideline concordant answer at the time of decision making is going to be helpful, I think, to all parties.
- 17:33Speaker 2: Um, you know, I think that payers are adopting their own AI also, you know, probably for utilization management.
- 17:40Speaker 2: And so I don't think anyone wants to see a future where you have an algorithm denying a scale and clinicians appealing with the assistance of AI, et cetera.
- 17:48Speaker 2: I actually think that having, you know, potentially something like open evidence creating that shared evidence infrastructure as a common ground, like you can't kind of dispute that we're all working off of the same facts here, which is like this was a trial in the literature, and this is the current best practice.
- 18:03Speaker 2: So I think if you're looking from the perspective of a clinician and a payer, looking at this like the same graded evidence, um, uh we that would be a really exciting thing for me to be able to effectuate if possible.
- 18:15Speaker: So, what has surprised you the most about your interaction with, and then you interact with open evidence through a couple of different lenses, but what has surprised you the most through your interaction with open evidence?
- 18:30Speaker 2: Um, you know, I think that open evidence has been a wonderful place for me to work, uh especially as somebody who's still in practice.
- 18:37Speaker 2: I think it has been lovely to be part of the team that's helping shape it, um, especially taking into account all the considerations of um, you know, like being in private practice, seeing a high volume of patients, seeing patients who are sometimes very disadvantaged socioeconomically or educationally, and kind of bringing that perspective into product development.
- 18:57Speaker 2: I think sometimes you have this perspective of AI being developed by these like Silicon Valley folks who are a little bit divorced from reality, but I really don't think I don't think that that's been my experience here.
- 19:08Speaker 2: And I feel just the fact that like I'm I'm part of the company, I think, is hopefully uh a sign to how well that's what makes it so powerful, right?
- 19:16Speaker: Because it's developed by people who know how valuable the information needs to be and and all those aspects of it.
- 19:22Speaker: So as you write, as you describe, you use it, you're an experienced clinician.
- 19:28Speaker: There are new physicians every year, right?
- 19:31Speaker: Start medical school, go through medical school training and residency fellowship, and they will have access, right, to open evidence.
- 19:39Speaker: How do we think about AI tools for the future doctors?
- 19:46Speaker: How do we think about what's the right level of access they should have, right?
- 19:50Speaker: Because as we discussed, subject matter experts, people who have really been doing their craft for a long time, can treat it as an employee.
- 19:57Speaker: How do you work with budding physicians who are less knowledgeable, don't know the right prompts, who aren't in there, right?
- 20:07Speaker: They're almost co-equals with the technology.
- 20:09Speaker: How do we think about that going forward?
- 20:12Speaker: Uh that dynamic of what open evidence should be?
- 20:16Speaker 2: Um, I mean, I think it's a great question.
- 20:18Speaker 2: And I think certainly over-reliance on this, even for experienced clinicians, you know, creating that kind of like automation bias where you're starting to not distrust it and you're just kind of like accepting the recommendation every time.
- 20:31Speaker 2: But magnified for folks who are in training.
- 20:34Speaker 2: Um, you know, there I would sort of say like medical education is really two things, right?
- 20:37Speaker 2: The first is some some amount of memorization, which you have to do.
- 20:41Speaker 2: You can't always have everything in your working memory, you have to have things in your long-term memory.
- 20:45Speaker 2: Like if you have a patient in sepsis, you can't at that time be looking up what is sepsis.
- 20:50Speaker 2: Like you have to know what that is going into it.
- 20:52Speaker 2: Um, but then there's like probably the bigger portion of it, which is clinical reasoning.
- 20:57Speaker 2: Um, how do you take all this evidence that's out there and apply it to your patient, or how do you critically appraise the evidence?
- 21:04Speaker 2: Um, you know, a lot of physicians, like if you do a test on thinking about like uh given the false positive values or of this test and the prevalence of this test, what is the likelihood that somebody who tests positive actually has this disease?
- 21:18Speaker 2: And you'll be surprised how few physicians kind of get those types of logical tests right.
- 21:22Speaker 2: You know, they're not even in the ballpark.
- 21:23Speaker 2: I think that medicine historically has over-indexed on memorization and like indexed less so on clinical reasoning.
- 21:32Speaker 2: I do think that that shift around kind of like reordering the importance of these two activities was probably a long time coming.
- 21:39Speaker 2: But maybe something like having AI in everybody's toolkit is going to accelerate that.
- 21:44Speaker 2: But I think that would probably be for the best.
- 21:47Speaker 2: Um, I mean, I think listen, I think that there are ways in which we probably need to now articulate workflows that will help training trainees kind of sustainable.
- 21:59Speaker 2: Systematize knowledge in a way that they have their own architecture in mind without becoming fully reliant on AI feeding them answers.
- 22:07Speaker 2: And what that looks like can be a mix of best practices, it can be different Socratic teaching modalities, it can be more of an emphasis on problem-based solving rather than kind of lectures.
- 22:19Speaker 2: I think that this is actually a space that I feel pretty passionate about, delving into further.
- 22:25Speaker 2: And there are some colleagues of mine at Open Evidence who are also similarly trying to create our internal working group thinking about this.
- 22:31Speaker 2: So more to come on it.
- 22:33Speaker 2: One example of a feature.
- 22:35Speaker: Oh, please, go ahead.
- 22:36Speaker: Yes.
- 22:37Speaker 2: Sorry.
- 22:37Speaker 2: One example of a feature that we just rolled out recently in an enterprise partnership is something called Open Evidence Rounds.
- 22:45Speaker 2: So already we have a feature where if you see an answer that you think you'll probably reference again and again, you can save it to a collection.
- 22:53Speaker 2: Now, this feature that we rolled out called Open Evidence Rounds, you can choose to broadcast that question and answer to a shared small community of folks.
- 23:02Speaker 2: So for example, you know, it's kind of like having journal club.
- 23:06Speaker 2: If you think about what journal club is in the medical setting, you are all getting together every week or something and dissecting a paper in some detail.
- 23:15Speaker 2: And I think having that type of recall and reapplication to different clinical scenarios is how people remember and think about how to contextualize knowledge in their hierarchy.
- 23:27Speaker 2: So all of these sorts of things I think are going to be important innovations moving forward.
- 23:31Speaker: Yeah, and at the end of the day, patients want to interact with people and not systems, right?
- 23:37Speaker: And so I believe that the AI eventually will be this amazing clinical decision support tool that'll actually allow physicians to interact on some very touchy and difficult subjects with patients, particularly in cancer, and it'll actually improve the the patient experience, right?
- 23:55Speaker: And you'll spend more time interacting with patients, having known have a higher level of certainty on what the answer is and what the regimen is, and the next so I I think it's I I think I'm very high on it.
- 24:05Speaker: I think it's very positive it's going to be a great tool for patient care, but I would be remiss if I didn't ask you through my lens as a as a recovering regulatory lawyer a couple of questions around regulating AI, right?
- 24:21Speaker: We saw the FDA announce that it's going to come out with guidance around generative AI.
- 24:27Speaker: And uh I also saw a a report this week I want to get your thoughts on that what we should do is take all of these AI tools and feed it all of the medical school exams and all the board exams and see how the AI does.
- 24:43Speaker: And if the AI passes just like a human would, then we could license the AI, give it a government seal of approval, and license it out as a government enterprise or private enterprise and give it that kind of good stamp of approval, if you will.
- 25:01Speaker: So with that as a background and introduction, I want to get your thoughts on what do you think about both of those ideas, right?
- 25:08Speaker: What the what role does the FDA have in reviewing and approving what the AI can say, right?
- 25:16Speaker: Because that's what the FDA does.
- 25:18Speaker: And then a third body, maybe it's the gov CMS government or a different government entity that sort of regulates how the AI can be used and maybe accessed and reimbursed, whatever.
- 25:30Speaker: What are your thoughts on those two separate issues?
- 25:35Speaker 2: Yeah, I mean, I think this is such an evolving area, and it's fascinating to kind of see where we all land between FDA's device definitions and sort of the CMS gun practice of medicine at the state regulatory level.
- 25:47Speaker 2: Um, you know, the way open evidence has situated itself in this whole dynamic is that we're not seeking to replace the physician.
- 25:54Speaker 2: You know, we're a tool in the physician's toolkit, just in the same way that a physician could look up anything, uh, you know, by going directly to PubMed or going to any other platform that offers some clinical decision support.
- 26:06Speaker 2: Um and so I think, you know, my perspective on this is that I feel like sometimes uh some of these um uh there's a I probably a couple issues.
- 26:18Speaker 2: One is sort of thinking about medicine as just kind of a collection of discrete tasks and demonstrating competence on one oftentimes like a diagnostic-related task, is sort of thinking about that as kind of being a holistic and encompassing all of what medical care is.
- 26:34Speaker 2: And I and I just I guess I dispute that a little bit.
- 26:37Speaker 2: Um, I mean, for example, even in my practice, so much of oncology is not just about identifying the diagnosis and coming up with a treatment plan.
- 26:45Speaker 2: Like I could have told you that that's not necessarily the hardest part of the job.
- 26:48Speaker 2: It's oftentimes thinking about um building trust with this patient, explaining things at their level, helping them, helping them pick choices in a very kind of like shared values alignment, um, and walking them through a very, very oftentimes a grueling process in which most people, 90% of people, would drop out if there wasn't a cheerleader in their corner encouraging them to continue because there's a light at the end of the tunnel.
- 27:12Speaker 2: Um, and there's like a million micro decisions throughout all of that around dosing and side effect management and continuing or abandoning early timing of surgery.
- 27:22Speaker 2: I mean, there's just so many things.
- 27:24Speaker 2: And so when I think about that, I'm kind of I I'm I guess I'm a little bit less um, you know, likely to believe that the role of a physician would be supplanted by an extremely smart AI.
- 27:35Speaker 2: But I know that a lot of companies are moving in that direction of kind of becoming more and more autonomous.
- 27:39Speaker 2: I mean, to be honest, I I'm just as curious as you, I think, about how all of this will land.
- 27:44Speaker: And how will it all be regulated?
- 27:46Speaker: And um so speaking of which, that takes us into the next topic, which is where do we go from here?
- 27:53Speaker: So just to level set the audience, again, I hope you go to open evidence.
- 27:57Speaker: I don't know, is the how to prompt effectively is that available for on the website for non-NPI people to see?
- 28:05Speaker: Like, is there any guidance there for if I just went there to open evidence?
- 28:09Speaker: I know I can play with it a little bit.
- 28:11Speaker: Is there is the how to prompt effectively in there?
- 28:14Speaker: But um, so anyway, as as you look, Ted, the where I'm going with this question is is where do we go from here?
- 28:23Speaker: Uh it's a very powerful open evidence and and and the other AI tools with the clinical decision tool, you type in a drug, you type in a condition, I CD10 code, you get lots and lots of very valuable information.
- 28:35Speaker: Uh where do we go from here?
- 28:37Speaker: What is the future?
- 28:38Speaker: If we're gonna regulate lightly, if we're not gonna do sort of licensing, the FDA's gonna say something at some point.
- 28:45Speaker: What is what is open evidence thinking about what's next?
- 28:48Speaker: Have we reached the end, right, where we are?
- 28:51Speaker: We got just keep putting in more information using RAG, right?
- 28:55Speaker: Keep putting it in and letting the physicians continue to use it as as it is, or what's generation 2.0, what's next?
- 29:04Speaker 2: Um uh probably a couple things.
- 29:05Speaker 2: The first is that you know, from from our standpoint, I think we would we are we are already kind of moving deeper into the workflow.
- 29:14Speaker 2: So, you know, the current generation of open evidence, you are it's a reactive query system, right?
- 29:20Speaker 2: So it answers questions when a user brings a question to it.
- 29:23Speaker 2: I think the next generation of open evidence will meet a user inside the chart.
- 29:27Speaker 2: Um so already kind of aware of the patient's context and potentially.
- 29:32Speaker: Is that coming from the EMR?
- 29:33Speaker: Is that coming?
- 29:34Speaker: Is that what you mean?
- 29:35Speaker: Because yeah, I don't think you're connected now to the EMR.
- 29:37Speaker: Is that correct?
- 29:38Speaker 2: No, we are connected to the EMR through our enterprise agreements.
- 29:42Speaker 2: Um and so we do have uh, for example, at Cedar Sinai, we are, you know, you can launch within the EHR, and in doing so, it can bring in some of that patient context into the into the into the query.
- 29:54Speaker 2: And so, you know, you wouldn't have to potentially re-explain yourself in terms of here's all what I'm dealing with with this particular patient, because it should already have that ingested.
- 30:04Speaker: And then so if that's all ingested, going back to my payer question, when the claim is submitted, all that supporting documentation of A, the medical record, and B, all the evidence and the papers gathered through the open evidence interface should all be submitted to the payer, right?
- 30:21Speaker: That is that does that occur today?
- 30:24Speaker 2: Yeah, it's a good question.
- 30:25Speaker 2: Right now, we're not submitting any supporting open evidence documentation automatically.
- 30:30Speaker 2: Um certainly, I think people use the system to generate things that are targeted towards payers.
- 30:36Speaker 2: For example, you could generate using your patient details and then using the evidence base that's built into open evidence, you could generate a prior authorization, you know, appeal or a you know, one of these things that are directed towards getting a drug approved.
- 30:51Speaker 2: Um, but it doesn't automatically submit anything.
- 30:55Speaker: Is that the future?
- 30:56Speaker: Is that the vision?
- 30:57Speaker 2: I don't I don't believe it's on any kind of near-term roadmap, but never say never.
- 31:02Speaker 2: I don't I don't think so though, just yet.
- 31:04Speaker 2: Right.
- 31:04Speaker: So the vision is to have, like you were saying before when I sorry I interrupted you about having agents inside the the open evidence, right, the AI, that it's a little bit more proactive for our new to AI listeners, right?
- 31:17Speaker: A little bit more seize the record, patient record, maybe so the way I use it, so tell me if this is possible, right?
- 31:25Speaker: So the way I use my AI, right?
- 31:26Speaker: I have it scheduled to look at my calendar the next day, and every morning I get an email that sort of says, here's all the people you're meeting with, here's the what's all the news and everything else about them to prepare for your day.
- 31:38Speaker: I imagine open evidence with the EMR can do the same thing.
- 31:42Speaker: So, Sam, you go in on Tuesday, you open up your email, and you have a list of all your patients, but all the clinical issues, all of that with pop, right?
- 31:51Speaker: Is that does that occur now or is that the future?
- 31:55Speaker 2: It it could be the future.
- 31:56Speaker 2: It's not happening now, even within our enterprise partner uh relationships.
- 32:00Speaker 2: I mean, I think that is moving from kind of like a a reactive query engine to much more of a proactive AI co-pilot.
- 32:07Speaker 2: And I think it could get there, but uh that's not what we're doing today.
- 32:11Speaker 2: I mean, it's certainly, I think, a very inspiring vision.
- 32:14Speaker: Yeah, yeah, yeah.
- 32:15Speaker: I mean, that would be that's you're getting really close, right?
- 32:17Speaker: And then you get to all the practice of medicine issues, which leads me to this question.
- 32:23Speaker: Um, what and how does open evidence think about liability here, right?
- 32:27Speaker: This is America after all, right?
- 32:29Speaker: And um what if there is it remains a clinical decision support tool, right?
- 32:36Speaker: Ultimately the physician is responsible.
- 32:38Speaker: Is there a something that the provider signs on to, signs when they interact with the with open evidence when they onboard, I should say?
- 32:47Speaker: How what is how is open evidence dealing with that issue?
- 32:50Speaker 2: Yes, so even when an individual clinician signs uh and makes an account, so independent of the enterprise partnerships, they are still signing an individual BAA with the company.
- 33:01Speaker 2: And it is a HIPAA compliant platform.
- 33:04Speaker 2: And so actually what we see is that we don't actually see oftentimes uh active PHI being put into the search box.
- 33:11Speaker 2: It's probably like less than 1% of the time that we see like full and complete identifiable patient information in there.
- 33:18Speaker 2: But even if we did, we would filter that out and it is protected.
- 33:21Speaker 2: We're not monetizing that data and we're not going to disseminate that out into the ether.
- 33:25Speaker 2: Um, and so I think that there's a lot of robust protections there for individual users, and certainly magnified so at the enterprise level where there's a lot more data rights.
- 33:35Speaker 2: Um but I would say that, you know, in terms of the liability question, I think it's a, again, like I said earlier, it's a clinical decision support tool, and we think that the professional accountability still rests with the clinicians.
- 33:47Speaker 2: Um, you know, what I would say is that when you have a AI clinical decision support platform like open evidence helping you potentially answer, ask and answer questions about your patients in a much compressed, you know, shorter period of time.
- 34:02Speaker 2: I think the natural instinct from a health system perspective is to say, oh great, let's let's increase throughput.
- 34:08Speaker 2: Let's just continue to add more patients to the physician schedule because this is going to create such efficiencies.
- 34:15Speaker 2: And I think that's probably the wrong um takeaway from all of this because you know, all of what we have talked about on this call around things like the risk of automation bias, over reliance, less slack to actually critically appraise what you're reading and whether or not it applies to your patient.
- 34:32Speaker 2: You need whatever time you're recovering from looking something up and seeing the answer, you need to reserve that for really thinking about your patients, spending more time in the room, educating them.
- 34:43Speaker 2: All of those things that actually I think what makes medicine fun and engaging and human.
- 34:47Speaker 2: Um the the more throughput you, I think, like add on to a physician because you're gaining these efficiencies, I think creates even less slack, more dependence, more automation, kind of compounds that problem.
- 34:58Speaker 2: So I think it does require like a conscious intentionality around how we think about these.
- 35:04Speaker: I couldn't agree with you more.
- 35:05Speaker: As you know, I sat there with my mother, um, she died of stage four lung cancer like 16, 17 years ago already.
- 35:12Speaker: And I hope, like I agree with you, and I hope what open evidence does is it it keeps the same physician-patient interaction time scheduled, but more of the time is spent talking to the patient and explaining the disease and the outcome that with that time that you saved doing the differential diagnosis and finding the drug regimen and understanding what potential adverse events could be.
- 35:37Speaker: To me, that would be a real win.
- 35:39Speaker: And the availability of open evidence, like you said, on the MPI tip free for all physicians will allow you to get, in theory, the same access to care if you're in rural Montana than if you are in Tennessee oncology, right?
- 35:54Speaker: I mean, that's that's a great outcome for this as well.
- 35:58Speaker: I'm assuming that that's what, right, part of the message and the mission of open evidence is to create equal access for patients, right?
- 36:06Speaker 2: Oh, absolutely.
- 36:07Speaker 2: I mean, it's certainly my personal mission too, in terms of joining this company.
- 36:10Speaker 2: I mean, I trained at Harvard Medical School.
- 36:13Speaker 2: My oncology training was from Memorial Sloan Kettering, and now I'm in a clinic that's a two-dock practice 40 minutes outside of Nashville, where our catchment area is like exurban and rural Tennessee.
- 36:25Speaker 2: Um, I cannot tell you how different it is to be moving from an extremely well-resourced academic medical center to the practice setting that I'm in now.
- 36:34Speaker 2: And it makes me very like, I think I have this like really unique perspective in thinking about how to bridge, like what gaps actually exist between these two practice settings.
- 36:43Speaker 2: And I truly, truly think that AI tools like this will help democratize that every patient is getting, you know, kind of the best possible care no matter where they go.
- 36:53Speaker 2: And so I feel very passionate about it.
- 36:55Speaker 2: I think Open Evidence is an extremely mission-driven organization.
- 36:58Speaker 2: Um, and I am excited to see what you know the future holds.
- 37:03Speaker: Same here.
- 37:03Speaker: I really, and you know, I really appreciate all that time and that those last couple of words.
- 37:08Speaker: But before I let you go, okay, so here we are, we're winding down, and now we've gotten many, many people excited about open evidence.
- 37:19Speaker: So they're gonna go to openevidence.com for the first time.
- 37:22Speaker: Non-physicians, what should they type in?
- 37:24Speaker: What would be the first if you're sitting there, right?
- 37:27Speaker: You're gonna recommend them.
- 37:27Speaker: I'll give you a minute to think.
- 37:29Speaker: They want to see, you want to, you know, you want them to see the power of open evidence.
- 37:33Speaker: What should they what would be the first prong?
- 37:35Speaker: What should they type in there?
- 37:36Speaker: Tell me about this or what's this?
- 37:38Speaker: What what would you recommend uh an individual do?
- 37:42Speaker 2: Um, I mean, I think if if your listeners are probably oncologists, I would recommend just going in there and sort of saying, um put something really specific.
- 37:50Speaker 2: Put something like, I have a 56-year-old woman with uh metastatic HR-positive breast cancer.
- 37:57Speaker 2: This is her performance status, these are her comorbidities.
- 38:00Speaker 2: Uh, what should I do and sort of see what it comes up with?
- 38:03Speaker 2: And importantly, not just the answer, but also look at all the artifacts that it's pulling in.
- 38:07Speaker 2: You know, I think we've done a really big product push this year to get in a lot more figures.
- 38:11Speaker 2: You'll see Kaplan Meyer curves directly from the journals.
- 38:14Speaker 2: You'll see NCCN guidelines excerpted, like full PDF pages in the answer.
- 38:19Speaker 2: Um, I think it's worthwhile just sort of seeing how enriched these answers are and whether or not you you want to keep using it.
- 38:26Speaker: Yeah, so I agree with you, Sam.
- 38:28Speaker: I I've I've done that, and you know, in full disclosure, right?
- 38:32Speaker: There's open evidence is not the only oncology AI, right?
- 38:35Speaker: There are other tools out there.
- 38:36Speaker: So patients and clinicians and pairs, all of our listeners, do an AI search for all the oncology AIs and really play around with it.
- 38:46Speaker: It is the future, it's very empowering, it's very uh builds a lot of self-confidence and engaging the healthcare system.
- 38:53Speaker: It'll help patients ultimately, uh, which is our goal here, right?
- 38:57Speaker: Is to improve patient outcomes.
- 38:59Speaker: Tremendous thank you to Sam.
- 39:00Speaker: Um, before I turn it over to you for some closing remarks, right?
- 39:04Speaker: Everybody who's listening, uh, please attend Value-Based Cancer Care at our conference in October.
- 39:09Speaker: You'll get the chance to meet Sam live.
- 39:11Speaker: She'll be there.
- 39:12Speaker: We greatly appreciate uh her showing up there.
- 39:14Speaker: Sam, any any final words for our audience?
- 39:18Speaker 2: Um, I I just think that like this is um sometimes I feel like when something like AI, which just feels like a total sea change, um, you know, hits our profession, it feels like a little bit unnerving and unsettling.
- 39:30Speaker 2: And I understand totally the impulse to be a little bit afraid of it.
- 39:33Speaker 2: But I do think that there's a lot of positive ramifications on how we do our work and you know the meaning and joy we derive from this profession.
- 39:42Speaker 2: So I think that overall I'm really, really optimistic, and I would encourage everyone to just follow along and see how this plays out.
- 39:50Speaker 2: But I do think that your jobs are safe and that they will only be made better because of this.
- 39:56Speaker: Yeah, no, I agree.
- 39:57Speaker: And I have all my fingers and toes crossed uh that AI can really be disruptive and improve, do what nothing else has done before, which is lower healthcare costs, improve patient outcomes, and uh if we do it the right way.
- 40:14Speaker: And we appreciate all the all the work you're doing on behalf of patients and the technology and your time today on our podcast.
- 40:22Speaker: And like I mentioned, look forward to seeing you at Value-based cancer care in October.
- 40:28Speaker: Thank you very much, Sam.
- 40:29Speaker: Really appreciate it.
- 40:31Speaker: Thank you very much.
- 40:32Speaker 2: Thanks so much.
- 40:33Speaker 2: This was fun.
- 40:35Speaker: If you'd like to see a demo of open evidence in action, take a look at the video version of this podcast on YouTube or follow the link in the show notes to the demo video.
- 40:43Speaker: That wraps up this episode of Value-Based Voices brought to you by the Association for Value-Based Cancer Care.
- 40:49Speaker: Thank you, Sam, for sharing your insights and to you, our listeners, for enjoying the conversation.
- 40:54Speaker: If you found today's discussion thought provoking, be sure to subscribe to Value Based Voices wherever you get your podcast.
- 41:01Speaker: Stay tuned for more episodes at Spotlight the Voices Shaping the Future of Oncology Care.