Latest / AI and Healthcare / What Changes When AI Can Read All of the EHR?

Transcript

What Changes When AI Can Read All of the EHR? · AI and Healthcare

• 0:00
  I get it when I say that, people ask like, are you trying to like, you know, just eliminate all this human labor?
• 0:04
  And the answer is no, I want to reprioritize where the labor goes.
• 0:08
  It's why I believe AI is so transformative in health care.
• 0:11
  This is such an important moment for all of us right now, because we cannot sustain what we're doing.
• 0:17
  And we're going broke doing the way that we're doing like all of us.
• 0:19
  Right. It's a fact. We're also not going to hire.
• 0:22
  Well, we're not going to graduate that many new oncologists or that many nurses, and we're not going to build enough hospitals for everybody, like all that stuff.
• 0:30
  And so we just have to find the must find efficiency there.
• 0:33
  It's like a mandate.
• 0:47
  Every oncologist this week is going to make a decision that basically uses a record assembled of somebody's data to make a very important decision.
• 0:56
  And today we're going to discuss how much of that record is available and how much can be available to make sure the thing is done correctly.
• 1:04
  And it all goes around one four letter word.
• 1:06
  And that's data. With all of that said, you are listening to AI and Health Care podcast.
• 1:10
  I am your host, Dr. Sanjay Junaidja, a hematologist and medical oncologist, joined by my colleague, mentor and good friend Douglas Flora.
• 1:16
  And we are thrilled to have you.
• 1:17
  And today we have a very interesting episode where we kind of dissect the different pieces and parts of what makes good data and what makes something reliable.
• 1:26
  Or what are some flags you may want to think about in this entire labor's process that we just sometimes on the clinician and gloss over?
• 1:34
  I know there are people dedicated to that specifically in our institutions, but Doug is an interesting question because there are different ways we're even consuming our data now, from our wearables and our applications on our phone to actual formalized health portals.
• 1:48
  Yeah, this is going to be a cool one, Sanjay.
• 1:50
  And you had me scared to death when you started talking about four letter words already two minutes into the podcast this time.
• 1:55
  So I'm glad you went with data this time.
• 1:57
  Guys, we're super excited about this one.
• 1:58
  Mika is a friend of the program, actually original founder of this podcast years ago and a good friend and mentor to Sanjay and I for full disclosure.
• 2:06
  We want to make sure everybody knows that this article we're going to discuss is produced and published in our peer reviewed journal and that Mika is one of our editorial board members went through strict peer review, was reviewed by everybody but Mika on the way to publication.
• 2:20
  But I want to put that in there.
• 2:21
  And then also the article review X cures is deeply involved, as is as are some authors from the economics does not affect the way we talk about it.
• 2:29
  Just want to get it out there in the interest of full disclosure.
• 2:32
  Let's get into it, because this is going to be a fun one for me.
• 2:34
  I'm excited to talk to you about this today, Mika, because it's been a couple of months in the work.
• 2:38
  So we finally get to bring it out into the light.
• 2:40
  Yeah, I'm delighted to be here, guys, and really have to be talking about this topic.
• 2:45
  It's something we've invested a lot of time and energy in both as a company starting and we can go back and talk about the motivations all the way going back.
• 2:53
  I think Sanjay did the very first times that you and I met for the very first time all the way back to the beginning.
• 2:59
  But we also felt it was really important to publish this because I don't think this area is well understood and I don't think it's particularly well characterized, which is really important.
• 3:08
  And if we think about what's going to really move the needle in clinical care, administrative burden, meaning paperwork, reading and writing into medical records is a huge part of the cost and quality of medicine today.
• 3:23
  And it's just ripe for AI.
• 3:25
  This is, in my mind, the place that AI gets the rubber meets the road today right away.
• 3:29
  And I'm telling you, I was thinking earlier today about how much people must not know about it because I'm currently doing a master's in clinical informatics management.
• 3:37
  And I've heard more than once they're like, well, clinical informatics, isn't that obsolete?
• 3:41
  Why do you need that anymore?
• 3:42
  And I just I literally cannot understand that question.
• 3:44
  I'm like everything about AI and models and predictive natures and texts require the way that we assemble and gather information and accurately deliver it to this technology so that the outputs reflect what what needs to be done.
• 3:58
  And that that process is just beginning in my mind.
• 4:01
  And, you know, to what you alluded to, a lot of the motivation that you saw way ahead of the game, I'm talking years before understanding the importance of this and and if not the necessity to be able to actually be benefactors of the technology that we're seeing.
• 4:17
  Yeah, let me it's I think it's a great point.
• 4:20
  And I've been trying to explain this to people, and I've been particularly in the last kind of month or so, really trying to hone in on like how how can we think about this particular problem?
• 4:33
  Right. In a way that everybody can understand.
• 4:35
  And the thing that I started to realize and it's been this kind of focus is like everything in medicine starts in the same place.
• 4:43
  So if you think about a guideline or a protocol or a coverage policy, right, or any of the documents that exist out there that really define care.
• 4:53
  And by the way, they have to exist as documents because we need to manage what best practices are.
• 4:56
  We need to manage risk, right?
• 4:58
  We need to manage quality and measure it.
• 5:00
  So it all needs to be in a document.
• 5:03
  Right. Those documents all have the same basic elements, right?
• 5:08
  Number one, they contain a clinical assessment of the patient.
• 5:11
  Right.
• 5:12
  So there is some set of clinical criteria that you must go through, which would be assess this, measure that, do this, do that.
• 5:19
  And then they all contain an operations component, which is actually how do you document that you actually did the clinical assessment and the decisions that you made?
• 5:28
  And so if you think about where that is in medicine, it's like everywhere.
• 5:33
  And that's actually a really bounded problem.
• 5:36
  But it's also a problem that if you get it wrong, everything goes wrong.
• 5:39
  You don't get reimbursed. Right. You deliver the wrong drug.
• 5:42
  You don't like nothing works if that process doesn't work well.
• 5:47
  And it's it's all about documentation.
• 5:49
  And it's why we have this burnout.
• 5:50
  It's why scribes took off and I write a few about why did they take off?
• 5:53
  Because we don't want to write all those notes ourselves.
• 5:56
  So it just kind of repeats on itself.
• 5:58
  But I think it has this really simple idea that like we want to we want to write rules for good governance.
• 6:03
  We want to follow the rules.
• 6:04
  And those rules are clinical and operational in context.
• 6:08
  Always, you nailed it.
• 6:09
  And and if you could walk us through, perhaps, you know, for for someone that really doesn't know or study this at all, what are some of the lexicons or languages, so to speak, that we use or are standardized to be able to translate what is happening inside of a medical record into this language of either things to bail or to do prior auth or to get, you know, reimbursements, et cetera.
• 6:36
  You know, and I guess I'm alluding to things like OMOP and fire and how you're able to, you know, kind of communicate these pieces of information.
• 6:44
  Why are why is understanding that so important?
• 6:47
  Well, there's lots of different languages in health care, and I'm not going to get them all because there's lots of them.
• 6:52
  But I'm just going to throw out some, you know, ones that you hear kind of a lot.
• 6:56
  So when you say OMOP, I am not I automatically think of things like LOIC codes or SNOMED or Medra. Right. And the kind of list goes on.
• 7:07
  So there's a whole bunch of specialized languages that relate to things like describing medicines, right, deciding conditions, ICD codes, right, for conditions, CPT codes for billing, for diagnostics.
• 7:19
  Right. So think of those as the specialized languages that have been developed as ontology is to describe a particular type.
• 7:25
  Medra, for instance, is used for drug safety often, and they all are slightly different because, I mean, the root of ontology is naming things.
• 7:34
  But the way you name things, right, actually really impacts how you use them.
• 7:38
  And we can't have the same naming structures for different fields.
• 7:41
  So basically every single field has its own naming structure that falls into one of those complex sets of codes.
• 7:48
  Now, one of the issues is you can crosswalk between many of them.
• 7:53
  Right. And OMOP is a great example of that.
• 7:55
  Most things can be crosswalked to some form of OMOP.
• 8:01
  But there's not one to one matches, so sometimes the terms just don't translate perfectly between each other.
• 8:08
  So if you think about someone's medical record that might contain all these different languages and then you think of all the different questions that you could ask where each question is assigned to a different domain, you're going to ask the question looking for the answer in the language that best suits the domain.
• 8:25
  And so you need to be able to normalize it.
• 8:28
  It's about the information retrieval section.
• 8:31
  It's like old problem and information retrieval.
• 8:33
  But if it's not written in the language of the field that the question is being asked in, you can't really answer it because of this kind of one to one mismatch problem that happens in the end.
• 8:44
  Which, by the way, sounds like a great problem for AI, doesn't it?
• 8:47
  I mean, if we think about, hey, what can we use AI for?
• 8:50
  The fact that I asked the question in the wrong language should be something in the future, and it can be, obviously, at the back end.
• 8:57
  So that's one thing that I don't think people fully appreciate.
• 9:01
  The other thing is we've had many shots on goal for what I would call interoperability or data transfer.
• 9:09
  And this goes all the way down to the fax machine, right?
• 9:12
  Like fax is just one way I want to phone and send data between different pieces, which is a perfectly good one.
• 9:17
  By the way, it's HIPAA compliant.
• 9:18
  That's the great thing about it.
• 9:20
  It's also got a really good human component to it.
• 9:23
  I always tell people, which is if I called either one of you gentlemen and asked you to send me the pathology point report for a patient, you would know exactly what I was asking you for.
• 9:31
  I wouldn't even take you two seconds.
• 9:33
  And if you had it there, you would pick out the right pathology report.
• 9:36
  You'd know which was the right one.
• 9:37
  And you'd send it to me by fax.
• 9:39
  And I would get exactly what I asked for.
• 9:40
  Boy, those are great points I never heard for the fax machine.
• 9:43
  It's like it's easy, super easy.
• 9:46
  Well, Mika, you know, your point is well taken, and this has been our struggle here at St. Elizabeth.
• 9:50
  So my hospital system, obviously, we are high tech, high end.
• 9:53
  You know, we care about data fidelity and data governance.
• 9:57
  But still, the majority of medical record for a lot of the cases that happened before this year or two or three years ago, they're scanned from a fax machine into the media tab of a medical record, sometimes with poor quality, sometimes with really faded ink.
• 10:09
  Sometimes things were mailed or wet or and you have taught me about how hard it is to gather this information.
• 10:16
  Since the majority of information is still not in discrete entry fields, how did you guys puzzle through that to figure out what is worth keeping?
• 10:22
  What is unusable?
• 10:23
  What must be deciphered?
• 10:25
  Yeah, that's a great question, Doug.
• 10:27
  So when we started working on the problem we were working out at X-Cures, we actually started with fax first.
• 10:34
  So we started this about eight years ago, and we were working with patients at the time.
• 10:39
  Our whole company was focused on clinical decision support for advanced metastatic patients.
• 10:45
  We were building a recommender system, like, so think of a Netflix or an Amazon, right, where the output of the recommender would be equivalent to getting a tumor board.
• 10:53
  That was kind of our ambition at the time where we said, you know, not everybody can go to a leading medical center and get a great tumor board.
• 11:00
  So we're going to have technology that provides you with the same quality output that a tumor board could.
• 11:04
  And you and your oncologist, Sanjay, you used it.
• 11:07
  I used it, yeah, several times.
• 11:08
  Yeah, and successfully, if I remember correctly, with several patients.
• 11:11
  With several patients, it was incredible.
• 11:14
  To do that.
• 11:15
  But the way we got the medical records was in a FedEx box or by fax.
• 11:18
  So that's what we received.
• 11:20
  And it was done under the patient's right of access to their medical records.
• 11:23
  So often we were calling their institution to medical records department, which I've learned is usually located in the basement next to the morgue, unfortunately, and gets about as much, well, less attention often.
• 11:36
  But it was release of information.
• 11:37
  So we would get our hands on the information.
• 11:39
  And what you described, Doug, was a really big problem.
• 11:42
  Actually, the quality of the image is the hard one of the hardest things to do.
• 11:47
  So we invested in some technology and actually a partnership at the time for like image enhancement, which sometimes works and sometimes doesn't work right.
• 11:55
  It makes the letter gets enhanced to be the wrong letter.
• 11:59
  And then it turns in this weird artifact that you can't manage around it, et cetera.
• 12:02
  So we started with fax.
• 12:04
  And then I'll give you an example of things that's that become problematic there.
• 12:10
  There's no pagination in some of these medical faxes, so they just run.
• 12:15
  It's like as if you had the infinite scroll.
• 12:18
  So when you take multiple documents and you put them together, they like start and stop in the middle of the page.
• 12:24
  So that's like hard to even tell which document you're in.
• 12:27
  And these things are like going back and forth between other places.
• 12:29
  So there's just been a lot of like, hey, I sent you the information, but no one structured it.
• 12:33
  And so we started building basically Bayesian models around what do we expect to see?
• 12:39
  So when I see an NGS report, that was the first one we were looking at, and it comes from a manufacturer, what does that look like?
• 12:48
  Like just if I saw it and said this was a Keras or a Tempest reporter or whatever it is, they all kind of look the same because they're on the same piece of paper.
• 12:56
  So we ought to know what those things are.
• 12:58
  And then generally, if we know what they are, we know where to expect to see certain types of information in that image document.
• 13:05
  So let's actually learn about the structure of the documents before we even start trying to go into what's in them.
• 13:12
  Because once you have this kind of this Bayesian approach, right, of like, what do I expect to find here?
• 13:18
  Then when you start reading the data, you actually already have a head start.
• 13:22
  You're like, I expect to find the variance in this section, right?
• 13:25
  And I know what I'm looking for there.
• 13:27
  And if there are errors in the image quality and that sort of thing, you can correct them better.
• 13:31
  It's still really hard to do.
• 13:32
  That's interesting, though.
• 13:33
  It's like it's the concept of going to another country and either just having a book of all languages and trying to figure out or just kind of having an idea of what that dialect will be like, what that, you know, an idea of what those infrastructure points are going to be.
• 13:47
  Like, where does the noun go?
• 13:48
  Where does it not, et cetera?
• 13:50
  So you're just recognizing on the surface, like kind of like a facial, I don't want to say facial recognition because that's actually elite technology.
• 13:57
  But to be able to then hone in on what variables you're looking for.
• 14:03
  Does that concept apply to things other than, you know, NGS reports that would, you know, would be grouped or bucketed in certain ways?
• 14:10
  Like what are what are some other buckets that you look at?
• 14:13
  So there's there's lots of buckets.
• 14:14
  But now going kind of back to you brought up the paper that we published, like if you think about what is the like the core methods that we're going after here, there's some some basic questions we're asking.
• 14:23
  And the first thing we ask right when you receive anything is what did I get?
• 14:27
  What is this data?
• 14:29
  Because you get a lot now.
• 14:30
  So take away the fax machine and you have computer exchange, you know, you can see is not uncommon to receive 400 to 800 documents per cancer patient from 30 plus different institute like it's like all of it.
• 14:43
  But the average number of patients in the particular cohort we looked at for this paper was 231 documents per patient.
• 14:50
  So the first question, what did I get?
• 14:53
  What's in what I got?
• 14:54
  I got to normalize going back to your coding question, Sanjay, all of the codes, at least into one language so that I at least know where they are, like get some handle on it.
• 15:03
  And then I want to start doing what I basically think of as labeling and embedding.
• 15:07
  So actually, and lots of people view that as pretty commoditized, right?
• 15:11
  Oh, like we know how to do embeddings and labeling, but it's an art and it's an art you get better at the more of it you do.
• 15:18
  And it's this Bayesian approach of the label of like, what is this thing that's here and what's in this section?
• 15:23
  Right.
• 15:24
  And what type of information did it contain?
• 15:26
  And then we take it even further and say, what kind of, you know, what was the NPI, for instance, associated with this?
• 15:31
  What type of specialist was it?
• 15:34
  So what you're doing is enriching the document itself with metadata.
• 15:39
  And you're taking that really, really deep to the level that's appropriate for it, because that's really the catalog of the information that you have.
• 15:48
  All right.
• 15:49
  And then once you've done that, well, you can actually really start to ask the interesting questions.
• 15:53
  But you can't just skip forward to questions.
• 15:55
  I think it's a mistake a lot of people kind of miss, right?
• 15:59
  They don't do the grunt work that you still need to do to do things well.
• 16:03
  And they go straight to like the fun part, which is answering the question at the end.
• 16:08
  And so then we have a whole rack model right around how we answer questions and the other part of it.
• 16:13
  This was brought up to me as a corollary example.
• 16:15
  If you look at what a data scientist done, if you've ever worked with data scientists, I would say 95% of the work is organizing
• 16:23
  the columns and rows of the data so that they can load it into these analysis software.
• 16:29
  The actual analysis is like a script they run.
• 16:33
  It's unbelievable.
• 16:35
  Yeah, it's unbelievable.
• 16:36
  How much of that precious time goes into just that?
• 16:38
  So this is where I think this whole section documentation piece.
• 16:42
  I'm glad to have a question for you.
• 16:43
  When you guys go and use this data on these patients, right, who are coming into your platform,
• 16:49
  how do you manage getting the information out of those historical documents?
• 16:54
  Like what do you what is tingling?
• 16:55
  What do you guys do?
• 16:57
  It's a mess.
• 16:57
  And you know, this is actually this is why I wrote the book behind me is because I feel like
• 17:01
  we have built so many workarounds that we really do need to do like a control alt delete
• 17:06
  for everything we do around data management and health care.
• 17:09
  Right now, it's human abstraction.
• 17:11
  If I want to do clinical trial scraping for my CRC or my CRN, it's an hour per chart.
• 17:17
  Going through the media tab, which is often mislabeled, scanned under the wrong names,
• 17:22
  scanned under the do the same thing for data extraction for other things for all of our
• 17:27
  registries or we run three or four months behind on our tumor registries.
• 17:32
  So I don't have actionable data, even on our own caseloads.
• 17:35
  And I feel so badly for the certified tumor registrars who spend their lives deciphering
• 17:40
  the undecipherable when you and I both know they could use the help of these great pattern
• 17:44
  recognizers. That's what your whole paper is about, actually, is how do we enhance the
• 17:49
  ability of these people who are take care of patients to not have to dig and hunt, decipher,
• 17:54
  decode because you almost need the Rosetta Stone to figure out the stage of a cancer
• 17:58
  patient and our denials on the same side.
• 18:01
  Mika, you know this very well.
• 18:02
  The companies that hire you, you're working with a lot of the tech companies to make sure
• 18:07
  that next generation sequencing results are available or we're getting denied by millions
• 18:14
  of dollars worth of legitimate drugs that we purchase and give because we don't have
• 18:19
  the ECOG performance that is in the note, not even a note because it's in the note.
• 18:24
  It's not in the request for authorization.
• 18:27
  And that's where these pattern recognizers are just they're doing Herculean efforts to
• 18:32
  save our people who are beleaguered and tired and exhausted doing these jobs.
• 18:37
  And so I credit you for what you're building because it takes a great amount of discipline
• 18:42
  that most of us don't have that obsessive focus that you have on this.
• 18:46
  We are not worthy.
• 18:48
  Yeah, it's true.
• 18:49
  And I tried to understand, Mika, what you did.
• 18:51
  So when you saw this problem, you designed this, you know, you took two large language
• 18:56
  models and kind of, you know, approached this extraction process like LLM based and looked
• 19:02
  at to see how could that, you know, improve kind of the processes that perhaps we're
• 19:08
  familiar with by just looking at the structured stuff is what Doug was largely talking about
• 19:12
  and pulling it out.
• 19:13
  Can you explain to us how that worked and how it made things better?
• 19:16
  Yeah.
• 19:16
  So that the two approaches, the one was the enrichment.
• 19:19
  So catalog creation.
• 19:21
  So I received these documents.
• 19:23
  What's in these documents like that, answering that question and taking it deep.
• 19:27
  And that includes, like I said, metadata, which is there, right?
• 19:31
  You just need to catalog, organize it and collated.
• 19:34
  And then there's extraction.
• 19:35
  So if you think about the free text fields, right, for things like status scores and all
• 19:39
  the rest of it, you actually want to take those out of the language, right, and convert
• 19:43
  them into a structured field.
• 19:45
  So it's actually structuring in a document.
• 19:49
  So that's one thing.
• 19:49
  So I just think of that as like, I received all these documents and I want to enrich the
• 19:54
  value of those documents by giving, by making sure I'd catalog all the information that's
• 19:59
  in there that it is.
• 20:02
  The second question is an equally important one.
• 20:05
  And it's what the people on your team, Doug, are doing, the abstractures, essentially,
• 20:09
  et cetera, which is I now want to do inference across those documents because often the answer
• 20:14
  is not one document, right?
• 20:16
  If I ask you about lines of therapy, well, that's in lots of documents, right?
• 20:21
  And so you actually need to now ask questions across the corpus of the documents that help
• 20:26
  you then assemble some other piece of that.
• 20:29
  And at the end of the day, like the simple thing we want to do is we want to replace
• 20:31
  somebody having to read a medical record and type it into something else.
• 20:34
  Like that's the human labor component that is there.
• 20:42
  And the way I think about that, because I get it when I say that people ask, are you
• 20:45
  trying to just eliminate all this human labor?
• 20:48
  And the answer is no, I want to reprioritize where the labor goes.
• 20:53
  That's actually the biggest opportunity.
• 20:55
  It's why, and I know I'm among friends who believe the same thing here.
• 20:58
  It's why I believe AI is so transformative in healthcare.
• 21:01
  This is such an important moment for all of us right now, because we cannot sustain what
• 21:07
  we're doing and we're going broke doing it the way that we're doing, like all of us,
• 21:11
  right?
• 21:11
  It's a fact.
• 21:12
  It's a fact.
• 21:14
  So we're also not going to hire, well, we're not going to graduate that many new oncologists
• 21:19
  or that many nurses, and we're not going to build enough hospitals for everybody like
• 21:23
  all that stuff isn't.
• 21:24
  So we just have to find the must find efficiency there.
• 21:27
  It's like a mandate.
• 21:29
  And the most obvious place for me then is that if we can reprioritize the current human
• 21:35
  capital that we have so that it is better spent, all right, then we've actually solved
• 21:40
  the problem because we can't eliminate, it's not a capital elimination.
• 21:43
  It's just a restructuring of where the capital is.
• 21:46
  So if I could take a process, right?
• 21:49
  And in that process, I could tell you for these trends, and I think in transactions,
• 21:54
  sorry, guys, I'm going to be super dorky here, like the transaction of seeing a
• 21:59
  patient, right?
• 21:59
  Or the transaction of something like there's a visit, there's an encounter, there's something
• 22:03
  that's going to happen, right?
• 22:05
  If I can automate, right, let's say 50% of those from a documentation perspective because
• 22:11
  they're the easier ones on some scale of easiness and give you a signal that just says for this
• 22:17
  one, push it through automation.
• 22:18
  Go, go, go, go, go, submit this, file the paperwork here.
• 22:22
  That's great.
• 22:23
  That's good.
• 22:24
  That's exciting.
• 22:25
  But it also means I can tell you that there's another than the 50% that's left, right?
• 22:29
  Half of that is orange, right?
• 22:31
  Instead of green.
• 22:32
  And you really want a human being to go look at that.
• 22:34
  And Doug, I was thinking of the score you were missing and saying, wow, this patient
• 22:38
  might get the medication, right?
• 22:40
  It might save their lives.
• 22:41
  Someone needs to go in and dig in that medical record because it wasn't automatable, right?
• 22:45
  But you'd like to know that you at least have a shot of finding it right before you go and
• 22:50
  spend the time.
• 22:51
  So to me, those are like these kind of mid-level things, which makes a lot of sense to have
• 22:54
  a human in the loop and will continue to need to do it.
• 22:57
  And then there's of course the last category, which is it's definitely not there, right?
• 23:01
  The last, which is don't even just quit now, right?
• 23:07
  And I think most things kind of fall into those three buckets of it's easy, sail it
• 23:12
  through, let's reduce all the friction we can versus it's worth the time and effort
• 23:17
  of human beings to go and try and solve this problem because it's creative.
• 23:20
  By the way, we should learn from that.
• 23:22
  That needs to be fit into a harness, into a learning loop, right?
• 23:25
  So that we're continually improving that piece.
• 23:28
  And then there's a huge amount of stuff where it's like, just give up.
• 23:32
  And that's okay because it's not worth the time of day for anybody to go after that.
• 23:38
  And that's actually how we actually approach all of these problems, right?
• 23:41
  In terms of the extraction and all of those different things.
• 23:44
  It's like, what can we do really easily and very quickly?
• 23:48
  And what can we have a very high degree of confidence in, meaning we have to be able
• 23:51
  to measure it, right?
• 23:52
  You can't have high degree of confidence in something if you can't measure it.
• 23:56
  Versus what do we still need human training on?
• 23:58
  Where are we putting a human rating in the loop?
• 24:00
  And then where is this data is just so bad, we're going to pretend we didn't even get
• 24:04
  it as it's just like, it's not there.
• 24:06
  It's nothing.
• 24:07
  It's like the facts of the facts of the 100 facts is before.
• 24:11
  And it might as well be a blank piece of paper.
• 24:14
  If I could speak from two sides.
• 24:16
  So obviously I'm the editor in chief of the journal that's publishing this paper.
• 24:20
  And I loved it.
• 24:21
  I probably read it five times now as it worked its way through peer review.
• 24:24
  And you and I kind of had to have our wall apart as friends that I couldn't really update
• 24:28
  you in the process.
• 24:29
  But I would encourage people to read that when they see this.
• 24:32
  But to anchor this to the clinic, I think is really, really important.
• 24:34
  Why we care, right?
• 24:36
  So our audience, this podcast that you and Sanjay started years ago has grown now.
• 24:40
  We're almost 90,000 people in that group.
• 24:43
  There are lots of cancer center executives.
• 24:45
  There are lots of doctors, nurses, IT people who live in the world of fire.
• 24:50
  But what this does for me in the clinic, guys, this is we're trying to run things that keep
• 24:55
  people safe.
• 24:56
  We're talking about things like the evidence bar that we have to have to be able to just
• 25:01
  make good decisions in a tumor board, human human being in the loop, our ability to have
• 25:07
  the right data for that tumor board.
• 25:08
  When I put 35 people in that room down the hall at 730 tomorrow morning, depends upon
• 25:14
  people like you making sure that the fields are popular, the data is available.
• 25:18
  We've talked about clinical trial screening and abstraction by clinical trial nurses or
• 25:23
  clinical research coordinators.
• 25:26
  That's another one.
• 25:27
  But but think about the nurses looking for data.
• 25:29
  Think about the doctors hunting for that next generation sequencing test.
• 25:33
  You're starting to see that we are not able to do the things that we know our patients
• 25:38
  deserve unless somebody scrubbing this stuff, somebody owns it, is responsible for it and
• 25:43
  can retrieve it quickly.
• 25:45
  And so the tools that you guys develop and people like me can develop are critically
• 25:49
  important to let us make the best data driven decisions possible.
• 25:52
  That's why I thought this was a very important paper to publish, not just because you did
• 25:55
  it, but to show proof of principle that it can be done and then everybody should understand
• 26:00
  why it needs to be done.
• 26:01
  Well, thank you, Doug.
• 26:02
  And as you know, the thing that we really tried to drive through this is is accuracy
• 26:07
  to your point about clinical use.
• 26:09
  If you can't trust the output, you can't make decisions.
• 26:12
  You can't have that tumor board meeting and say, well, we think it's this XYZ mutation.
• 26:18
  So your choices really boil down to go and read it by hand.
• 26:21
  By the way, there's conflicting data in there, too.
• 26:23
  So then there's a whole argument about which one is right that's in it or have technology
• 26:28
  in hand that you trust and you're willing to put, you know, you just have to trust it.
• 26:35
  And in our world, that doesn't feel any different than any medical device that's out there.
• 26:40
  Right.
• 26:42
  So I just encourage people and I think this is somewhere where, like, AI sometimes looks
• 26:46
  so magical.
• 26:47
  Right.
• 26:47
  You're like, oh, it's great.
• 26:48
  And then we worry about hallucinations and then we fret about it.
• 26:51
  I'm like, why are we worrying?
• 26:53
  We should be measuring.
• 26:54
  All right.
• 26:55
  Just measure it accurately and then you need to have confidence.
• 26:58
  I always comment I make pretty often, which is if I build technology that's supposed to
• 27:05
  replace a human being reading a medical record and typing it in and I need to hire a human
• 27:10
  being to check every single piece, then I didn't make anything valuable.
• 27:14
  It's like, well, I have it.
• 27:16
  It's like it's pointless.
• 27:18
  So it really needs those performance characteristics.
• 27:20
  What makes me nervous is when we hear about the different steps of the process, the different
• 27:27
  languages and the decoding of that, and then doing the labeling, these different buckets
• 27:33
  that you characterize, I keep using that term, there's variability in the quality of that,
• 27:37
  I'm sure.
• 27:38
  So if somebody is considering integrating a tool into their health care system that
• 27:43
  will require this kind of data harmonizing, this entire process we just outlined, how
• 27:50
  does one vet the quality of it?
• 27:54
  We have benchmark testing for LLMs.
• 27:57
  Are there measurements to say you do this, this percent well, this, this percent well,
• 28:01
  and stay away from trouble or a mistake?
• 28:04
  Well, a couple of answers here.
• 28:07
  Answer number one is the gold standard here because we do that.
• 28:11
  The comparator is human adjudication, human extraction, otherwise.
• 28:16
  And if you do dual extract, so two people extract and if they agree, then it's right.
• 28:22
  And then if there's a disagreement, then you adjudicate it.
• 28:25
  That's the highest level of order that you can get to of can you really get the data
• 28:30
  right?
• 28:30
  And I think you need to be able to compare to that or at least be able to compare the
• 28:33
  technology to that when you are basically making the first decision to use it.
• 28:39
  Now, because it's a learning system, you also need to go back and check it.
• 28:43
  So we have a periodic refresh, right?
• 28:45
  Because it learns, it's changing.
• 28:47
  It isn't the same system every single day as the beauty and danger of AI, right, is
• 28:51
  this drift issue.
• 28:52
  So you have to manage drift across that.
• 28:55
  And so that again means you need to characterize the drift and decide if it's drifting in the
• 28:59
  right or wrong direction, it can go both ways.
• 29:02
  So there's a drift issue around this as well.
• 29:06
  So we actually published, and I'll get this to you guys, a link to it, but we published
• 29:12
  a buyer's guide for this whole space.
• 29:15
  I always like to tell people that, of course, it's biased to the company that produced
• 29:19
  it, of course, but we also tried to really honestly make it useful for everybody.
• 29:23
  Like our goal wasn't just to have a marketing piece, but to actually put out there the
• 29:27
  questions that you should ask.
• 29:29
  And so the way that we created that document, it walks you through all the different
• 29:33
  domains that you should consider, and it gives you all the questions that we think you
• 29:38
  should ask any vendor or technology partner that you should have in that space so that
• 29:43
  you can accurately compare them to each other.
• 29:46
  Things like how do they know how complete their data is?
• 29:49
  How do they know that their extraction was accurate, valid, validity?
• 29:53
  How, what formats are they going to transfer the data in?
• 29:56
  What integration points to that?
• 29:57
  Like just like the entire set of questions, and fortunately it ends up being about 17
• 30:01
  pages long when you do the whole thing.
• 30:03
  But we think that there is a very characterizable way to think about this problem, and that
• 30:09
  if you're diligent about it, and I'm sure, Doug and Singles, but you have a very clear
• 30:13
  procurement process for your IT components, probably feels onerous.
• 30:17
  Ours feels onerous to me, and I'm the CEO of the company, and I hate it, right, when
• 30:20
  I have to actually buy something, and it has to go through InfoSec, but it's there for
• 30:25
  a real reason.
• 30:27
  In fact, we actually just got our latest high trust certification, and that was a big part
• 30:33
  of our certification, was actually all the document controls on the back end, so big
• 30:38
  deal.
• 30:39
  I'm going to throw one thing in there that we all, everyone likes and everyone feels
• 30:44
  is hard to talk about, but is actually the truth of this as well, which is this has to
• 30:48
  have a financial impact, right?
• 30:51
  And I think if you think about anybody who's measuring this type of technology, the question
• 30:55
  is, does it pay for itself?
• 30:58
  Because the other thing that we don't have in the system is more money, so problems that
• 31:02
  create additional expense, solving problems that actually creates a net increase in expenses
• 31:08
  are no longer solutions that are like, that rope is gone, right?
• 31:13
  So what we need to do is in each one of these cases actually have highly, not only does
• 31:17
  the technology need to be characterized, but you need to look at the process that you're
• 31:20
  implementing, and it needs to have a net ROI to it.
• 31:23
  So even to register a problem of how many more units can I get through, all of those
• 31:28
  need specific testing right behind it.
• 31:31
  And I can tell you that I promise you every hospital system measures this number of patients
• 31:35
  that you saw.
• 31:36
  If you're taking capitated risk, how are we managing that risk?
• 31:39
  Can we assess the quality measures that we go through it?
• 31:42
  And again, it's all data unfortunately done.
• 31:45
  I have to ask, Mika, one thing where I get confused is we have different ways to
• 31:50
  get data.
• 31:51
  We have direct integrations into the hospital systems, which take forever, like age 12 months.
• 31:55
  And one thing that I'm seeing an explosion of are all these DTC direct to consumer apps
• 32:00
  that just let you kind of select some kind of authorization that in Baton Rouge, I have
• 32:06
  my records in three or four different places, some just private clinics that are propped
• 32:10
  up, a hospital system, another hospital system.
• 32:14
  How is that data happening?
• 32:16
  Because I don't think it's health information exchanges.
• 32:19
  So what's the difference?
• 32:21
  Is one just going to eventually replace the other one?
• 32:23
  Like, like, please walk me through that.
• 32:26
  Yeah.
• 32:27
  Oh, my gosh.
• 32:28
  You just you're you're touching the nerves, Sanjay.
• 32:30
  I know, I know.
• 32:32
  So so there's kind of two main ways that data is moving around the country right now.
• 32:38
  There are others.
• 32:41
  There are more like direct connections.
• 32:43
  So a lot of like payer, direct to payer portals, operations data between two entities is all
• 32:49
  like think of like old school direct connectivity.
• 32:51
  But we've had health care interoperability for data transfer generally.
• 32:56
  And it's happened in two areas.
• 32:58
  The first area is for treatment.
• 32:59
  So covered entities requesting data from another covered entity.
• 33:04
  Right.
• 33:05
  In order because patient care is moving between the two.
• 33:07
  And that that actually got solved in twenty twenty three.
• 33:09
  You've heard me talk about that a lot.
• 33:10
  And you followed me at all.
• 33:12
  Now I'm just like the data is free.
• 33:13
  It left the building.
• 33:14
  It's moving around.
• 33:16
  Right.
• 33:16
  You got to have the right rights to it.
• 33:18
  And we want to make sure appropriate use cases.
• 33:20
  Right.
• 33:20
  And that there's governance around it.
• 33:21
  But that problem after decades is finally really solved.
• 33:25
  And it's amazing.
• 33:26
  It's better than I thought it would be.
• 33:28
  And it just opens up the possibility to solve all these problems that weren't solved before.
• 33:33
  The other one is my right as an individual to get my own data.
• 33:36
  Right.
• 33:36
  And that exists right everywhere.
• 33:41
  IAS, which is Individual Access Services, is part of the TEFCA QHIN.
• 33:46
  And I'm sorry, guys, acronyms time.
• 33:50
  But the entire health interoperability exchange that's governed by TEFCA or the Trusted Exchange
• 33:56
  Framework Common Agreement, right, which governs those participants.
• 34:00
  And that is like the latest generation of health information exchange.
• 34:04
  I am really excited to see that performing.
• 34:06
  I would say it's a 50 percent and rising right of people who go on there and identify themselves.
• 34:13
  And you have to use technology like Clear or IDME.
• 34:16
  And there's a few others out there to do the identity but are getting their data now.
• 34:21
  And what's great about that is you don't have to remember everywhere you went.
• 34:24
  Fact is, you don't actually know everywhere you went because we all know this.
• 34:27
  The entities themselves, the actual health care organizations sometimes are like nested
• 34:33
  in each other and all over the place.
• 34:34
  So if you don't ask the right person, you don't get the right data and it's all spread out.
• 34:37
  But that is all getting fixed.
• 34:39
  And that's accelerating.
• 34:40
  And I have to say, CMS has been doing a really good job of convening and encouraging the
• 34:47
  participation.
• 34:48
  So, for instance, we participate in a CMS-aligned networks program called Kill the Clipboard,
• 34:55
  which is exactly what it sounds like.
• 34:57
  Like how many times have all three of us spent time filling in a clipboard in an office
• 35:02
  somewhere, right?
• 35:03
  And I can tell you, I'm horrible.
• 35:05
  I just draw a straight line down through everything that I think and then sign the bottom.
• 35:08
  And I try to give it to the staff and they're like, "You can't do that."
• 35:10
  They're like, "Why not?"
• 35:12
  Like, "All right, just sign it."
• 35:13
  I'm like, "I'm not filling it in."
• 35:14
  And that doesn't go over very well either.
• 35:16
  It's all the stuff that we're talking about.
• 35:18
  And I've heard you speak so many times about the importance that the most critical lever
• 35:23
  for powering effective clinical AI would be if patients owned their own data and could
• 35:28
  consent meaningfully where everybody has full access to this rather than just directing
• 35:33
  it B2B or B2C or whatever.
• 35:35
  And I do think we're heading that way.
• 35:37
  But talk a little bit about how that reduction of automatic siloing would help you and your
• 35:43
  job as a data maven.
• 35:44
  So the data processing is exactly the same.
• 35:46
  So all the things we just talked about at data processing can happen at the individual
• 35:49
  level.
• 35:49
  And by the way, getting your medical records, it's not like you get your medical records
• 35:52
  because you want to sit at home and read them, right?
• 35:54
  Like no one's going to do that.
• 35:55
  So what you really want is knowledge, your information, right?
• 35:58
  Structured knowledge out of the medical records.
• 36:00
  So it's exactly the same problem.
• 36:01
  What's really exciting to me about the IAS, my individual right to get the medical record
• 36:06
  is it's the only way to get it out of HIPAA, right?
• 36:09
  So when it comes to me, it becomes consumer data.
• 36:12
  It's my data.
• 36:13
  And I can now do whatever I want with it, including go put it on the telephone pole
• 36:16
  outside if I want to, right?
• 36:18
  Or put it onto somebody else's clipboard.
• 36:20
  In fact, one of the requirements of this program, Doug, now is we have to enable a patient to
• 36:26
  have a QR code on their phone or in the technology.
• 36:30
  And that when they go to the next provider, they just hold up the QR code and do it.
• 36:34
  Now the provider systems don't have the readers yet or the technology behind that, but that's
• 36:38
  coming really fast.
• 36:39
  That's coming out of this kill the clipboard program.
• 36:42
  So that's really exciting.
• 36:43
  And the place where it takes us is into contextual use of the data.
• 36:49
  Meaning I use my healthcare data to go and procure services.
• 36:52
  Like when I always think of it again, sorry, transaction financial guy here sometimes.
• 36:57
  I am going to the doctor to procure healthcare, right?
• 37:00
  Whether I pay my insurance pays or whatever it is, and I go to see different doctors.
• 37:04
  If I have cancer, I go to an oncologist.
• 37:06
  When I need my teeth worked on, I go to a dentist, right?
• 37:08
  When I eat new glasses, I go to optometrist.
• 37:11
  All of them need access to this data for different things, but they all want a different slice
• 37:15
  of it.
• 37:16
  And it's inferred.
• 37:17
  And we were talking about the different languages.
• 37:18
  The questions that are asked are now asked differently, right?
• 37:21
  Because we're in a different contextual situation.
• 37:24
  And that contextual layer is super important to efficiency and quality of care and delivering
• 37:29
  all the rest of it.
• 37:30
  And we just couldn't do it before.
• 37:32
  It didn't matter.
• 37:33
  But that's going to open up a whole different world.
• 37:36
  And by the way, then all the consumer stuff, there's stuff that's not part of formal healthcare
• 37:40
  that's incredibly valuable.
• 37:41
  Think of all the longevity and wellness applications that are out there.
• 37:45
  And we're actually going to start understanding now how they work, right?
• 37:49
  That we never knew beforehand.
• 37:51
  There is stuff about being able to give your records.
• 37:53
  This is another conversation to add to people, as part of your legacy to your family, right?
• 37:59
  Imagine that, like if you could...
• 38:01
  23andMe was a great idea, right?
• 38:03
  Until it wasn't.
• 38:05
  But I want all of that data to be useful for everybody around me and all the rest of it.
• 38:12
  And the really only way to do that is if I control it myself, because otherwise the institutions
• 38:18
  are bound in all sorts of red tape.
• 38:21
  One example of exactly what you're talking about, just so much to do, is there are several
• 38:26
  companies that are coming out and brilliant scientists that are looking at either radiomic
• 38:32
  or CT scan stuff or digital pathology slides that are obviously slides of pathology and
• 38:38
  biopsies that are scanned in.
• 38:39
  We had a great episode with a professor at Emory, Anand.
• 38:43
  And you could imagine how, if you were able to carry that digital pathology slide for
• 38:47
  yourself and possibly opt in to get some kind of insight or predictive or prognocious scores
• 38:54
  because you own that piece of data and are able to just make a sensible, educated decision
• 39:00
  rather than...
• 39:01
  And I'm not saying this is the way it should be, but you could see how it could be.
• 39:04
  You are now privy to these different things that can be opened up that otherwise may be
• 39:10
  restricted based on your institution or your physician.
• 39:13
  But then what about the people that don't have it on that key chain or can't access
• 39:17
  or don't know about these ways that you can get these additional insights applied to your
• 39:21
  own cancer diagnosis?
• 39:22
  And that's a disparity that is very concerning because then you have this just widening gap.
• 39:28
  Yeah.
• 39:28
  I don't know what to do about the disparity thing.
• 39:30
  I think it's a really hard question.
• 39:32
  And I think that's always been my honest take on it, is it's really hard to solve.
• 39:40
  And it's not education.
• 39:42
  We can throw money at the problem, but I suspect that it's actually quite systemic and that
• 39:48
  it's actually the way we've built the systems that we have.
• 39:53
  It's really to do with how we legislate and the governance that we put in place around
• 39:58
  healthcare.
• 39:59
  And so these are the really deep issues.
• 40:02
  And I think it gets...
• 40:04
  Yeah, a lot of people play the disparity thing.
• 40:06
  A lot of lip service, in my opinion.
• 40:07
  They're like, "Oh, we're going to do this or do that."
• 40:09
  Or it gets called out as it's a real issue.
• 40:11
  But I'd love to hear real solutions that actually work in this space.
• 40:19
  Because again, I was just kind of chastising myself for how excited I get about AI.
• 40:23
  And then like, yeah, that's even better for the people who are not getting left behind,
• 40:28
  but it doesn't solve the problem.
• 40:30
  And by the way, if we look at where the cost is, where a lot of the burden is, it's not
• 40:35
  in that tier that's taking advantage of the system that's able to do it.
• 40:41
  The vast majority of the costs that we're bearing are in the general population or down.
• 40:47
  This is why I care about it though.
• 40:48
  And for our audience, I know we've gone wonky a little bit on the last few episodes, but
• 40:54
  it's so critical that people out there understand.
• 40:56
  And if Sanjay and I are trying to be the Pied Piper and we're trying to help drag you into
• 41:00
  this new century of medicine, it's because every single unstructured report, every genomic
• 41:07
  sequencing panel, every documented response represents a real person, an urgent question.
• 41:12
  And without the work that people like Mika and his team do, without rigorous fidelity
• 41:17
  of data and patient directed portability, which we all believe about, these answers
• 41:22
  just remain locked in this digital ether and we can't use them to make good decisions.
• 41:28
  So when we talk about this and we structure this clinical reality and we put this good
• 41:32
  data, this scraped, sound, true verified data into the hands of the clinicians, then the
• 41:39
  data is no longer just operational paperwork.
• 41:41
  That's our diagnostic map.
• 41:43
  That's the whole point of what we're talking about with these pattern recognizers.
• 41:47
  They're going to enable us to guide better, smarter cancer care.
• 41:50
  So Mika, I just I thank you for your commitment to this.
• 41:54
  It is critically important.
• 41:56
  And I think your work honors the individual behind the record.
• 41:59
  It's just I can joke that it's boring, but there is nothing more mission critical than
• 42:04
  giving Sanjay and I and Lauren, your wife, who treats patients today while you're hanging
• 42:09
  out with me, the best possible information to make the best possible diagnosis and treatment
• 42:14
  plan.
• 42:15
  Thank you, Doug.
• 42:16
  So, guys, as we bring our conversation to a close, the discussion returns to us as an
• 42:21
  essential realization that you've heard from Mika, you've heard from Sanjay and me that
• 42:25
  this technology does not exist in a vacuum of clean benchmarks or algorithmic precision.
• 42:32
  Data is not an extraction.
• 42:34
  It is the very path forward.
• 42:35
  So I want to remember somebody who taught me about this, somebody who was really special
• 42:39
  in this world of data analytics and all this tool.
• 42:43
  And he had a quote that has always stuck with me.
• 42:46
  And I wrote it down for this episode, Mika, because I thought what you do is so resonant
• 42:50
  with what he said.
• 42:51
  Hiding within these mounds of data is the knowledge that could change the life of a
• 42:56
  patient or change the world.
• 42:58
  And he's no longer with us and we miss him, but he was so ahead of the game.
• 43:03
  We lost him in June 2025.
• 43:05
  But that's what we're talking about here.
• 43:07
  We're not talking about vectors.
• 43:09
  We're not talking about tokens.
• 43:10
  We're not talking about data parameters.
• 43:13
  We're really talking about the difference between changing the life of a patient and
• 43:17
  ultimately change the lives of hundreds of thousands or hundreds of millions of patients.
• 43:21
  So to your credit, I appreciate you being with us.
• 43:23
  I appreciate your time and effort doing this very unglamorous work.
• 43:28
  And I wish you the best of luck as you get back to surfing for this long holiday weekend.
• 43:51
  (upbeat music)