Latest / Lab to Lives / Audit Trails, RBQM, And Agentic AI Explained

Transcript

Audit Trails, RBQM, And Agentic AI Explained · Lab to Lives

• 0:00
  SPEAKER_01: Welcome to another episode of Transformation in Trials.
• 0:02
  SPEAKER_01: Today I have two awesome guests in the studio with me.
• 0:05
  SPEAKER_01: We're gonna talk about audio trails, we're gonna talk about agentic AI, we're gonna talk about RPQM and what's happening in regulation right now that kind of makes all those things related to each other.
• 0:15
  SPEAKER_01: But before we dive into all of these topics, I want you to know who my guests are.
• 0:19
  SPEAKER_01: Ellis, would you introduce yourself first?
• 0:21
  SPEAKER_00: Sure.
• 0:21
  SPEAKER_00: My name is Ellis Hiroki.
• 0:23
  SPEAKER_00: I'm the CEO and founder of a company called Study OS, breaking Agentic AI to Pharma.
• 0:27
  SPEAKER_00: We specifically focus on the clinical data management and RBQM elements.
• 0:31
  SPEAKER_00: Before that, I started my career off at Microsoft in the big data, machine learning, and analytics space, where I worked on the original Azure Synapse Analytics project, which has since become part of a larger offering in the Azure ecosystem.
• 0:43
  SPEAKER_00: Following that, I had various successes and failures across different healthcare startups.
• 0:47
  SPEAKER_00: And right now I'm spending all of my limited brain brain power and energy working in the pharma space with folks like Nahama better serve this industry.
• 0:54
  SPEAKER_02: Awesome.
• 0:55
  SPEAKER_02: Nahama, would you mind introducing yourself?
• 0:57
  SPEAKER_02: Yeah, so my name is Nahama Kitan.
• 0:59
  SPEAKER_02: I'm now officially the chief wizard at Wicked Problem Wizards.
• 1:02
  SPEAKER_02: I have degrees in math and statistics, which I call liberal arts engineering degrees.
• 1:06
  SPEAKER_02: I'm qualified to do learn anything and to do nothing.
• 1:10
  SPEAKER_02: And I spent the last eight years or so in the pharmaceutic, having joined a large pharma company where I implemented a global RBQM solution.
• 1:19
  SPEAKER_02: And now I'm working with tech startups, tech integrators, and sponsor companies to understand how they can really go to scale and be effective in their RBQM and audit trail solutions.
• 1:31
  SPEAKER_02: I also run an audit trail working group for the eClinical Forum, which is getting ready to publish a paper, fortunately internally for now, but it will be open to the public once it's been internal for a short while.
• 1:44
  SPEAKER_01: Awesome.
• 1:45
  SPEAKER_01: Well, before we dive into further topics, I want us to make sure that we get some of our listeners along with us for the journey that are new to the space.
• 1:54
  SPEAKER_01: So if I were to ask both of you, how would you explain what an audit trail is and what is it for?
• 2:00
  SPEAKER_02: I'll do the kind of the business side and let Alice do a little more of the technical side.
• 2:04
  SPEAKER_02: So in a regulatory environment, you've got a couple requirements.
• 2:08
  SPEAKER_02: One is you have to say what you're gonna do, you have to do what you said you were gonna do, and then you have to show you did what you said you were going to do.
• 2:14
  SPEAKER_02: And if I was doing this on a piece of paper, I would write out my instructions on my SOP.
• 2:19
  SPEAKER_02: So I would say what I was gonna do, then I would do it, and I would check off each time that I did each one, and then I would have a document, a piece of paper that says, hey, that's my proof that I did what I said I was gonna do, right?
• 2:30
  SPEAKER_02: There's this phrase in in pharma that says if it wasn't documented, it didn't happen.
• 2:34
  SPEAKER_02: Right?
• 2:35
  SPEAKER_02: So the neat thing about computers is that you don't have to do that anymore.
• 2:39
  SPEAKER_02: Computers know if you build the right systems and you build these systems to be, I call it design for validation.
• 2:46
  SPEAKER_02: You you build a system that's already designed for these types of questions, tracks who did what, who entered data, who logged in, when did they log in.
• 2:55
  SPEAKER_02: It's all of that they call metadata that's been tracked around it.
• 2:58
  SPEAKER_02: Who did the data entry, when did it happen, date timestamps, all of that cool stuff.
• 3:03
  SPEAKER_02: What we discover, and what's magic about audit trail on clinical trials, is that the data that comes out of the trial, how many, you know, the patient's blood pressure and when did they get dosed and when did they sign the e-consent, all of that information is interesting information for the person analyzing whether or not the trial was successful, right?
• 3:23
  SPEAKER_02: Was the pain med effective or not?
• 3:26
  SPEAKER_02: What it doesn't do is it it only kind of very indirectly if you squint and look cross-eyed and whatever, tell you how the trial ran.
• 3:34
  SPEAKER_02: If on the other hand, if you think about if you've been to a hospital in a Western country recently, the person pulls out their badge, they scan in, hey, I'm giving you a dose of a medication, they scan your wrist bracelet, right?
• 3:46
  SPEAKER_02: Everything is scanned and tracked in an audit trail so you can see the whole process.
• 3:50
  SPEAKER_02: So the audit trails for the EDC, while it's only data entry, and the audit trails for the IRT or the eConsense or the ECOAs and the EPROS all start to show us what actually happened at that site and gives us this really powerful ability to understand what the behavior at the site was and was the data collected in a way that makes it fit for purpose?
• 4:14
  SPEAKER_02: So that starts to answer the questions in E6R3 of do you have data that's fit for purpose?
• 4:21
  SPEAKER_02: Did that process that that site follows?
• 4:23
  SPEAKER_02: Is it consistent?
• 4:24
  SPEAKER_02: Does the audit trail mirror consistent behaviors?
• 4:27
  SPEAKER_02: Or are they putting in one data point and changing it six months later for no reason?
• 4:32
  SPEAKER_01: That's a really good explanation.
• 4:33
  SPEAKER_01: Ellis, maybe you can tell us more about how does one log an audit trail?
• 4:37
  SPEAKER_00: Sure.
• 4:37
  SPEAKER_00: I'm gonna repeat some things.
• 4:39
  SPEAKER_00: One tacit assumption in that was that the systems are actually built as CSV and part eleven compliant, which may not actually be the case.
• 4:46
  SPEAKER_00: But let's just say you're working with a vendor in your clinical study, and they are actually CSV and part 11 compliant.
• 4:53
  SPEAKER_00: So as a result of that, every action taken in the system should have an entry in a system that dictates what happened, right?
• 5:01
  SPEAKER_00: So who, what, when, where, why, right?
• 5:03
  SPEAKER_00: And so I think one of the big challenges in the space to date is that there isn't a lot of predefined analytics, but I think more than that, there isn't really a setup or a flow that enables people to do these kinds of analytics.
• 5:16
  SPEAKER_00: So one, a couple of features of audit trails or audit logs that make them particularly difficult, I think, for the lay user to analyze is one, there's just a lot of them, right?
• 5:26
  SPEAKER_00: So if we think about every single entry or line item in a clinical study, you're gonna have at least one, if not many, more log items.
• 5:33
  SPEAKER_00: So it's gonna be some factor of the number of data points in your study already.
• 5:37
  SPEAKER_00: If you have three million lines in your study, then you're gonna have 15 million log entry lines, right?
• 5:43
  SPEAKER_00: And so as much as we like to tell ourselves that we're using advanced analytics and good systems for this, the reality is a lot of people are still using Excel, and this goes beyond the capabilities of anything you can load up into a spreadsheet and look at yourself.
• 5:55
  SPEAKER_00: So there is one huge issue with that.
• 5:57
  SPEAKER_00: The second one is that audit log entries are by definition isolated from the rest of the study data.
• 6:02
  SPEAKER_00: So that means that you have to work often with the vendors.
• 6:04
  SPEAKER_00: There's often roadblocks to acquiring the data itself, so you have that problem.
• 6:08
  SPEAKER_00: Then there's lack of standardization because there isn't some kind of standard that really predefines what audit exhaust should look like.
• 6:15
  SPEAKER_00: And so everyone's gonna do it a little bit differently.
• 6:17
  SPEAKER_00: It's not a huge problem, but it's just another roadblock and another issue that you're gonna have to deal with.
• 6:21
  SPEAKER_00: And then finally, in order to actually generate these insights, it's not just about, hey, can I look at the audit log independent of the actual data, but it's about marrying the clinical data with the audit log information to actually see essentially the journey of the study.
• 6:34
  SPEAKER_00: And all of that requires things like the ability to host the data in a specific place, the ability to run the analytics, the ability to actually understand what you're looking for, and so on.
• 6:42
  SPEAKER_00: And I think a lot of those are both technology infrastructure as well as skill requirements that frankly the industry doesn't really have set up.
• 6:49
  SPEAKER_00: And I think that's like a big reason why the e-clinical forum working group has focused so much on what it would take to actually do this.
• 6:54
  SPEAKER_00: But I think the last thing I'll say on this is that the industry makes it very clear with R3 that there that is the expectation for sponsors to do it, and I think it's something that we should all be focusing on.
• 7:04
  SPEAKER_01: Well, my first experience with the audit trails was back when I started uh in live sciences 15 years ago.
• 7:10
  SPEAKER_01: The company I was working for was facing an audit, and we got our audio trail export uh in a CSV file from the vendor, and my first task was to make that into a spreadsheet.
• 7:23
  SPEAKER_01: It sounds like we're actually not too far away from uh that reality 15 years on, but we can no longer afford to be in that reality because of the regulatory demands, is that correctly understood?
• 7:33
  SPEAKER_02: E6R3 has made it very, very clear that auto trail analytics is a requirement, is not enough to say your system has an auto trail that you could pull up, but that you need to have regular planned reviews.
• 7:44
  SPEAKER_02: Regular planned reviews, in my opinion, and in the in the working group's opinion, require automated data flow, because otherwise it's impossible.
• 7:53
  SPEAKER_02: You can do it without automated data flow.
• 7:55
  SPEAKER_02: It's very hard.
• 7:57
  SPEAKER_02: And then analytical tools to manage that data so you can figure out what to do.
• 8:01
  SPEAKER_02: And even before that, kind of what are you looking for?
• 8:04
  SPEAKER_02: What are you reviewing it for?
• 8:05
  SPEAKER_02: 15 million lines of data, I'm looking for what?
• 8:09
  SPEAKER_02: Right?
• 8:09
  SPEAKER_02: That's not a filter and sort or the lookup at that point.
• 8:14
  SPEAKER_02: It's you have to have specific questions.
• 8:16
  SPEAKER_00: That's another element.
• 8:17
  SPEAKER_00: I mean, the other thing to keep in mind is that even though it's been very slow, the adoption of e clinical solutions has increased, which just means that the more there are more data points that have audit log exhausts to be looking at, as opposed to maybe 15 years ago, where well things are still a lot on paper.
• 8:32
  SPEAKER_00: So I you know, I'm not gonna say that everything's digitized now, but that is a trend and so uh we should uh act accordingly.
• 8:38
  SPEAKER_01: Yeah.
• 8:38
  SPEAKER_01: But it's also a question of having those many systems that actually touch the clinical trial.
• 8:43
  SPEAKER_01: If we really want to make sense of what happened in the trial, we also need to make sure that we both have access to those uh data logs.
• 8:49
  SPEAKER_01: Some of them may be with our CROs potentially, some of them may be with our CRO subcontractors as a sponsor company.
• 8:55
  SPEAKER_01: So it's creating that whole data infrastructure first before we can even look at the data, and then running into standardization issues.
• 9:02
  SPEAKER_01: However, just looking at the single system, what are some of the challenges that we're seeing in our industry for just making sense of an audit trail?
• 9:10
  SPEAKER_02: So uh I think the first step is like DMIU's clinical clarity or clinical relevance.
• 9:16
  SPEAKER_02: There's a lot of things you can go look at, right?
• 9:19
  SPEAKER_02: And not everything is equally important.
• 9:21
  SPEAKER_02: And this is an industry that likes to say, oh, we're gonna go check every single thing.
• 9:25
  SPEAKER_02: No, you're not gonna go check every single thing.
• 9:27
  SPEAKER_02: You can't.
• 9:27
  SPEAKER_02: And you don't really want to.
• 9:29
  SPEAKER_02: EClinical Form and SCDM published the paper in 21, which was kind of the audit trail paper that everyone references.
• 9:35
  SPEAKER_02: And in it it had, I don't know, 30 or 40 use cases that they listed.
• 9:40
  SPEAKER_02: They were clustered, but they weren't prioritized.
• 9:42
  SPEAKER_02: So the first thing to do is to go through and prioritize those use cases, which again, the new paper will have had done.
• 9:49
  SPEAKER_02: If you're a member, reach out to me and we can show you that paper.
• 9:53
  SPEAKER_02: You prioritize high, medium, low, and it's how likely is it gonna be to happen, how hard is it to detect, then what's the severity if it happens?
• 10:01
  SPEAKER_02: Let me give you a couple of different examples.
• 10:03
  SPEAKER_02: If you have a patient on an EPRO device, so a patient entered form and they're entering an assessment and they leave the trial because God forbid they've died or something, and those EPO entries continue to happen.
• 10:17
  SPEAKER_02: That tends to lead regulatory people to question who was entering that data to start with.
• 10:23
  SPEAKER_02: Now, a lot of people, when that these conversations first came up and I first started having them with industry, someone said to me, Oh, well, have your EPO vendor do the analysis.
• 10:31
  SPEAKER_02: But the EPRO vendor doesn't know the study data, doesn't know if that patient has discontinued from the trial.
• 10:37
  SPEAKER_02: You know if that patient was unenrolled from the EPO, but you don't know if they've been discontinued from the trial.
• 10:42
  SPEAKER_02: So you need to be joining the interesting questions.
• 10:46
  SPEAKER_02: So this is where the detectability gets hard.
• 10:48
  SPEAKER_02: You need to take your study subject status page and your EPRO page and make sure that every patient that's no longer in that trial is now no longer entering EPRO data.
• 10:59
  SPEAKER_02: Or that you don't have other weird EPRO's like devices that are known to be sitting at a site or now being used by multiple patients at home.
• 11:07
  SPEAKER_02: It's those types of data checks.
• 11:09
  SPEAKER_02: And so you need to kind of pick some questions, then you have to explore the data to see what are the useful analyses.
• 11:17
  SPEAKER_02: And this is where the industry has really gotten stuck.
• 11:19
  SPEAKER_02: There's been a lot of progress in the last couple of years of getting the data moved over.
• 11:23
  SPEAKER_02: But once you get the data moved over, that data gets thrown to someone like me in my former role or a PhD statistician, I think very expensive US-based resource to try to get sense of the analysis.
• 11:36
  SPEAKER_02: The way someone like I do analysis is I open up Jump or SAS or R and I do a one-off analysis, and then say, oh, isn't this cool?
• 11:44
  SPEAKER_02: And everyone says, Yeah, that's really cool.
• 11:45
  SPEAKER_02: Now do it every week.
• 11:47
  SPEAKER_02: And you're like, no, I that's not my job, right?
• 11:49
  SPEAKER_02: And then I try and I can't.
• 11:50
  SPEAKER_02: I've done this, right?
• 11:51
  SPEAKER_02: This has happened.
• 11:52
  SPEAKER_02: This repeatability is a problem.
• 11:54
  SPEAKER_02: So what's cool about an agentic system like Study OS is that that analysis now happens inside of a tool that itself can track everything.
• 12:04
  SPEAKER_02: And so you open up a chat bot, ask a question in English, which by the way, now you no longer need me.
• 12:11
  SPEAKER_02: Anyone, my boss could have asked that question.
• 12:14
  SPEAKER_02: Someone sitting in a quality meeting can ask that question.
• 12:18
  SPEAKER_02: And it takes the data file, it understands the data file on the context.
• 12:22
  SPEAKER_02: Because you could ask ChatGPT or Gemini, but if it doesn't understand the context, it's not going to help.
• 12:27
  SPEAKER_02: And we can give some examples, but it understands the context.
• 12:30
  SPEAKER_02: Write SQL, which is now repeatable.
• 12:32
  SPEAKER_02: SQL is repeatable, validatable, built for validation.
• 12:36
  SPEAKER_02: That SQL gives you an answer that can then be run next week.
• 12:39
  SPEAKER_02: If you see it, dump it to a dashboard, run it over and over and over again.
• 12:43
  SPEAKER_02: So that gives you a way for the first time to really explore the data, figure out what's useful, figure out what questions you can ask that make sense to ask, which ones you run and you're like, oh, there's nothing there, or I would love to ask that question, but I'm not gonna go tie up someone in either local or a 10-hour time zone difference to run that analysis and get back to me in three days.
• 13:05
  SPEAKER_02: You can just spin this and query it and explore it.
• 13:08
  SPEAKER_02: It auditors like Play-Doh.
• 13:10
  SPEAKER_02: You've got to like get your fingers dirty and play with it and build things.
• 13:14
  SPEAKER_02: And does it do this?
• 13:15
  SPEAKER_02: What works and what doesn't work and what's useful and what isn't?
• 13:18
  SPEAKER_02: It's not something that's so well understood that you can just stack it up and build it.
• 13:23
  SPEAKER_02: It's not, it's not ready for full-time production yet.
• 13:26
  SPEAKER_02: You need that exploratory root cause analysis capability.
• 13:30
  SPEAKER_02: And so you need tools to do that in.
• 13:32
  SPEAKER_01: And how far are we with the identic AI portion on top of the audit trail?
• 13:37
  SPEAKER_01: How does it work?
• 13:38
  SPEAKER_00: Well, before I say anything, I'm gonna caveat that these systems are all rapidly changing so quickly that by the time that some listener might hear this a year from now, all of this could be completely outdated.
• 13:50
  SPEAKER_00: So I'll just talk about what exists today as of this recording.
• 13:52
  SPEAKER_00: So a lot of these large language models that people play around with in systems, they're not connected to your study data in general.
• 13:59
  SPEAKER_00: And so even if they were connected to your study data, which some companies have perhaps set up contracts with that, they're not really purpose-built for the purpose of data analysis.
• 14:07
  SPEAKER_00: So a lot of these systems, if you've ever heard of the term hallucination, they're very eager to give you what they think is the right answer.
• 14:13
  SPEAKER_00: And I'm anthropomorphizing a bit, but that's essentially what's happening is that it's trying to predict the the next token or the next set of words that's most likely to satisfy the prompt.
• 14:22
  SPEAKER_00: And then so in this case, it could hallucinate or come up with an answer because it thinks that that's essentially what they're what you want to have.
• 14:28
  SPEAKER_00: And obviously that can't work in any kind of regulated system.
• 14:30
  SPEAKER_00: So the first case is you want you you cannot rely on these systems alone to provide you answers.
• 14:36
  SPEAKER_00: They're really good at like search engines type stuff, and so you know, for that kind of QA work, it's fine, but for anything that requires regulated workflows, they're insufficient.
• 14:45
  SPEAKER_00: So I guess the first thing I'll say is that you can't you have to avoid the hallucination port and you need to have it connected to your data.
• 14:50
  SPEAKER_00: So what we do that's very different is instead of using these kind of AI models to actually provide you a direct answer, we actually just use it to create the code that then runs against your data, and then that provides you the answer.
• 15:03
  SPEAKER_00: So that distinction is really important because we don't allow it to provide the answer directly, so there's never a hallucination.
• 15:09
  SPEAKER_00: What it does is it translates it into SQL or in the future into Python or R or SaaS, and that essentially gets run on top of your data.
• 15:18
  SPEAKER_00: And what does that grant you as an industry?
• 15:20
  SPEAKER_00: It means that one, it's auditable, you can see exactly what code was run at what time, it's validatable, you can review the code itself so anyone with any kind of experience working with that code can look at the outputs and it's repeatable, which Naham had mentioned earlier, but by virtue of running it in the system, you can have it run every day or every week or every month and be certain that it's the exact same thing running every single time.
• 15:42
  SPEAKER_00: So I think that these are the kind of key components.
• 15:44
  SPEAKER_00: Now, the question about how it pertains to audit or anything else like that, the system itself is actually generic in that it can run against any number of clinical data sources and audit log included.
• 15:52
  SPEAKER_00: It is it is a difficult thing to make this system work, in particular for clinical studies.
• 15:57
  SPEAKER_00: So I mentioned this in a talk I did for C Disk around their DDF USDM standard, but the kinds of queries, the kinds of prompts that we have in the clinical trials are significantly different than the vast universe of sales and marketing data that that most of these models are trained on.
• 16:13
  SPEAKER_00: And so the other problem with the generic models is that they're all based on like sales and marketing information.
• 16:18
  SPEAKER_00: So if you ask the question that is temporal, for example, it'll assume that you're talking about the fiscal quarter or the last 90 days or something like that, which would be completely ridiculous to ask.
• 16:28
  SPEAKER_00: Like no one cares about how the patient's vitals looked in the last fiscal quarter, right?
• 16:33
  SPEAKER_00: You know, it's all based on the protocol, and so there's a lot of context that you need to add as well.
• 16:36
  SPEAKER_00: So there's an element of one, it needs to be connected, two, you need to use code and not so that it doesn't hallucinate.
• 16:41
  SPEAKER_00: And three, it needs to be kind of purpose-built for the industry so that it can answer questions in a way that makes sense for any sane person working in the in the space.
• 16:49
  SPEAKER_01: That's uh an interesting description, and also it it makes a lot more sense.
• 16:53
  SPEAKER_01: And also I'm just identifying and acknowledging that this tool that we can use to figure out what's up with our audit trails has its own audit trail that is audible.
• 17:02
  SPEAKER_00: There's a s there's a level of introspection there which you can ask questions about it itself, which I think is kind of funny.
• 17:08
  SPEAKER_00: It's a little meta.
• 17:08
  SPEAKER_01: Yeah.
• 17:09
  SPEAKER_01: That's neat.
• 17:09
  SPEAKER_01: And and also just that's I can suddenly see how this could become a part of the landscape for clinical trials because it is so robust.
• 17:18
  SPEAKER_00: Yeah, I think the the big thing that comes to my mind is that when I look at the space, there are a lot of subject matter experts.
• 17:25
  SPEAKER_00: So they know already what they should be looking at.
• 17:27
  SPEAKER_00: They already have an idea of what issues could happen in the study, right?
• 17:30
  SPEAKER_00: And so I think there's this perspective that sometimes people have where the tools are trying to replace the expertise.
• 17:36
  SPEAKER_00: And I don't think that that's the case.
• 17:38
  SPEAKER_00: I think what's really missing is that people have intent, right?
• 17:41
  SPEAKER_00: So they know what they want to look for.
• 17:42
  SPEAKER_00: They want to reconcile this data set or they want to look, they want to examine this kind of to examine the specific data set or something else like that.
• 17:48
  SPEAKER_00: But the outcome of them getting to that analysis is actually quite large, right?
• 17:52
  SPEAKER_00: Because most of the time there's some kind of technical gap that exists, right?
• 17:54
  SPEAKER_00: So maybe they they're they're somewhat proficient at SaaS, but not enough, or rerunning it every single time they lack the infrastructure.
• 18:01
  SPEAKER_00: So there's like a lot of these little things that kind of exist in the way in in between.
• 18:04
  SPEAKER_00: And so that's where a lot of the costs are born.
• 18:06
  SPEAKER_00: And so when I think about what we're trying to do here, it's like how do we use these kinds of tools to really close the gap between a user intent and the outcome of actually putting a new control or a test or a check into the study or you know, exploratory analysis.
• 18:19
  SPEAKER_00: And so that's kind of the way we think about it.
• 18:21
  SPEAKER_02: Let me give an an example, Ivana, of some of this temporal stuff to make it English.
• 18:27
  SPEAKER_02: So the last time you asked your data manager or your central monitor or your RBQM expert to figure out between which visits did adverse events occur?
• 18:37
  SPEAKER_02: Or between which visits do most patients start taking concomitant medication.
• 18:42
  SPEAKER_02: So ask your average data person this or clinical trial person this, an operational person would be like, I don't touch the data sets, right?
• 18:50
  SPEAKER_02: A data manager would say, okay, you want me to do what?
• 18:53
  SPEAKER_02: So they've got a one Excel file where you've got one tab for each CRF page.
• 18:57
  SPEAKER_02: You've got your study and you've got your AE page, you've got a data visit page.
• 19:02
  SPEAKER_02: You're gonna figure out if you're smart pretty quickly that not every patient has an adverse event.
• 19:07
  SPEAKER_02: So now you need to take your demography page and match it to your adverse event page so that you've got a whole list of all the patients with the AEs that they had.
• 19:16
  SPEAKER_02: Now you've added null values.
• 19:18
  SPEAKER_02: By the way, nulls and data are just really, they're zeros.
• 19:21
  SPEAKER_02: You're dividing by zeros, okay?
• 19:23
  SPEAKER_02: But wait, there's more than one AE per patient, but there's only one demography record per patient, right?
• 19:29
  SPEAKER_02: You start to like your brain starts to explode.
• 19:31
  SPEAKER_02: So now I have to join my demography intelligently to my address event.
• 19:35
  SPEAKER_02: And then I have to get date times to work.
• 19:38
  SPEAKER_02: Okay.
• 19:38
  SPEAKER_02: As a society, we can't agree on European versus US date formats.
• 19:42
  SPEAKER_02: Computers are even worse about that, right?
• 19:45
  SPEAKER_02: And and oh, and now you've opened the file in Excel.
• 19:48
  SPEAKER_02: So Excel has fixed a date time format.
• 19:50
  SPEAKER_02: You close it and open it in RSS because you've given it to someone who now knows something more about programming and they can't get the date time.
• 19:58
  SPEAKER_02: Like I've done all this, right?
• 20:00
  SPEAKER_02: Just getting the data set.
• 20:02
  SPEAKER_02: And what this tool does is it says, oh, okay, an AE is a log page.
• 20:06
  SPEAKER_02: You don't have visits attached to the AE's.
• 20:09
  SPEAKER_02: Often your first person would say, Oh, I opened up the AE page, there's no log, there's no visit dates.
• 20:14
  SPEAKER_02: I can't do the analysis, right?
• 20:16
  SPEAKER_02: Yeah.
• 20:16
  SPEAKER_02: And so it understands what are visits and that you have to open up the AEs and attach that AE to the visit that happened before and after it and glue it in between two places.
• 20:28
  SPEAKER_02: And then it can run, you know, build out the data file and build the system.
• 20:31
  SPEAKER_02: Without a tool helping you do this, it becomes a mess, even for someone like me who knows what they're doing, to do this reliably.
• 20:39
  SPEAKER_02: And to write the SQL, I don't know about you, but I don't write SQL natively.
• 20:44
  SPEAKER_02: And then and I can't get the date formats to work in SQL either.
• 20:47
  SPEAKER_02: That's my other problem.
• 20:48
  SPEAKER_02: I used to work for Kaiser Permanente years and years ago, and I had a sticky note on the side of my computer with like four different systems of SaaS date formats because in Teradata SAS did something different than it did in whatever.
• 21:02
  SPEAKER_02: It was horrible, right?
• 21:04
  SPEAKER_02: It was horrible.
• 21:05
  SPEAKER_02: Just your basic stuff.
• 21:06
  SPEAKER_02: And that's not adding any value.
• 21:08
  SPEAKER_02: At the end of the day, what the the person sitting in data management needs to be doing is getting insights, not proving that they can write SQL.
• 21:15
  SPEAKER_02: Or you build a Cartesian join and your database administrator calls you up and yells at you.
• 21:20
  SPEAKER_02: Or, you know, right?
• 21:21
  SPEAKER_02: I've done that to have taken down their systems.
• 21:24
  SPEAKER_02: Your scratch drive has exceeded tolerance.
• 21:26
  SPEAKER_02: We're going to shut it down, your job down, right?
• 21:29
  SPEAKER_02: Like all of this stuff is not, should not be my job.
• 21:31
  SPEAKER_02: I should be able to ask a question in English, which you can do in Study OS, you know, between which visits do most AEs occur?
• 21:38
  SPEAKER_02: And it gives you an answer within three minutes now on a moderate size study.
• 21:43
  SPEAKER_02: Right?
• 21:43
  SPEAKER_02: It just changes it so that I don't have to worry about getting into my car with a screwdriver set and a tool set, making sure it works, checking all of my ties.
• 21:53
  SPEAKER_02: This is what it used to take to run a Model T.
• 21:55
  SPEAKER_02: Right?
• 21:55
  SPEAKER_02: Back in the days of Model T's, you had to be a mechanic to start your car.
• 22:00
  SPEAKER_02: Now I just get into my car and you know, if I'm in someplace like San Francisco, I think it drives itself.
• 22:05
  SPEAKER_02: But you know, it we're getting just more and more towards where the technology is becoming invisible as it should be, so then I can focus on wait, I need to get from point A to point B.
• 22:14
  SPEAKER_02: I can stop having to worry about every single step that I'm doing, which is what data managers and central monitors are doing now.
• 22:20
  SPEAKER_02: It's like, wait, do I clutch first and then break?
• 22:23
  SPEAKER_02: And all they're wanting to do is get from point A to point B and they've lost track of even where they're driving.
• 22:27
  SPEAKER_02: Getting out of the garage is a problem.
• 22:29
  SPEAKER_01: This also happens like in a pharmaceutical company when whenever someone from senior leadership or the board or the investors asks a question about the study, an innocent question, and suddenly like there's a flurry of activity in the statistical programming department.
• 22:42
  SPEAKER_01: Everyone's just trying to figure out which data sets to correlate to answer this question.
• 22:46
  SPEAKER_01: It sounds like with a tool like this, frankly, if it's the leadership of the company, they could ask this question themselves, perhaps learn the answer, or at least the lives of the people who are supposed to give them the answer, would be much easier.
• 22:57
  SPEAKER_02: That's what we're trying to do.
• 22:58
  SPEAKER_02: So we're trying to get it so people can stop worrying about the car and actually worry about where they're going.
• 23:02
  SPEAKER_01: What is the overlap between what we can now do and should be doing with audit trails and say RBQM?
• 23:09
  SPEAKER_02: So RBQM was set up to be process control.
• 23:12
  SPEAKER_02: If you grew up outside a pharma and you saw the early RBQM regs and the early QTL systematic, that language was all classic process control language.
• 23:22
  SPEAKER_02: So when I first joined the industry, I said process control, no problem.
• 23:25
  SPEAKER_02: I know what process control is.
• 23:27
  SPEAKER_02: Process control is only done right when you're measuring data about the process, not the outcome.
• 23:35
  SPEAKER_02: Okay.
• 23:35
  SPEAKER_02: All the clinical trial data that we collect is outcome data, the data that goes to the CSR, the clinical statistical report, the findings that we send to regulatory are all output measures.
• 23:48
  SPEAKER_02: So the output of the trial, they say nothing about the process.
• 23:51
  SPEAKER_02: And if you try to do RBQM only on outcome measures, and my personal experience, it is extraordinarily hard to find process issues.
• 24:03
  SPEAKER_02: So I'll give an example from a different industry.
• 24:06
  SPEAKER_02: If all I care about is whether or not my kids get into a good university, that's an outcome measure, right?
• 24:13
  SPEAKER_02: And maybe I'll take some interim outcome measures and I'll take their grades, end of year mark, grade point averages at 12 years of school, and I've got 12 data points.
• 24:22
  SPEAKER_02: That's not going to get me the information I need to know to fix a child who's having troubles in school.
• 24:27
  SPEAKER_02: I need the process measures.
• 24:29
  SPEAKER_02: What are the process measures?
• 24:30
  SPEAKER_02: Are they going to sleep at night?
• 24:32
  SPEAKER_02: Are they getting exercise?
• 24:34
  SPEAKER_02: Are they eating meals?
• 24:35
  SPEAKER_02: Are they hanging out with friends with an appropriate amount of time?
• 24:38
  SPEAKER_02: Are they hiding in the rooms and never talking to anyone?
• 24:41
  SPEAKER_02: That might be a good sign if they're studying.
• 24:43
  SPEAKER_02: But you need those process measures.
• 24:45
  SPEAKER_02: And it's the audit trail that gives you that process measurement.
• 24:49
  SPEAKER_02: Now, ideally, I had a senior leader that I worked with once who said, well, if you really want process, you need to go put barcode scanners on everyone's wrist, stickers on everyone's wrist in a clinical trial and make them scan themselves every time they did anything.
• 25:03
  SPEAKER_02: Well, yeah, we do that in hospitals, we do that in pharmacies.
• 25:07
  SPEAKER_02: They they print off their barcode and they put it on their tag, the ID, and they they scan themselves in and you know exactly what happened when.
• 25:13
  SPEAKER_02: But we don't have enough audit trail, but the audit trail is really the first place where we can start to say what happens on our trial.
• 25:20
  SPEAKER_02: And the data needs to be the basis of your RBQM implementation.
• 25:24
  SPEAKER_02: So that's the first overlap.
• 25:26
  SPEAKER_02: The second overlap is that you are not going to query audit trail data.
• 25:30
  SPEAKER_02: So you find a statistical finding in the audit trail data, what do you do with it?
• 25:34
  SPEAKER_02: Well, you need to have a workflow.
• 25:36
  SPEAKER_02: Analytics without a workflow isn't useful because now you've found a piece of information you can spend the next three weeks trying to figure out who to talk to about it.
• 25:44
  SPEAKER_02: It needs to go into an RBQM workflow where you say, okay, we found something statistically, let's have someone else look at it.
• 25:51
  SPEAKER_02: What do we think this means?
• 25:53
  SPEAKER_02: Let's send a CRA to go talk to our site.
• 25:55
  SPEAKER_02: Let's have these conversations.
• 25:56
  SPEAKER_02: Can we figure out root cause?
• 25:58
  SPEAKER_02: Because statistical correlation does not imply causation.
• 26:02
  SPEAKER_02: Causation can only happen if there's a conversation.
• 26:05
  SPEAKER_02: Those conversations happen ideally within an RBQ and workflow because it's not, hey, how come you always enter all of your data at midnight?
• 26:12
  SPEAKER_02: You're not going to query audit trail point.
• 26:14
  SPEAKER_02: You don't even have a way to query it, and you shouldn't be querying it even if you could.
• 26:18
  SPEAKER_02: Yeah.
• 26:19
  SPEAKER_02: And so you've got both it's both it's the workflow, and then I think it's the canonical key data set.
• 26:24
  SPEAKER_02: And we haven't used it because it's just been so hard to get.
• 26:26
  SPEAKER_01: And then analyze it.
• 26:27
  SPEAKER_01: So now finally we're at a stage where we might actually be able to analyze the audit trail and get some clues.
• 26:33
  SPEAKER_02: We can now get the data, we can push it into a space where we can use it, and there are now finally tools in the environment.
• 26:40
  SPEAKER_02: Not very many, but they're beginning to be tools in the environment that will give us access to it.
• 26:44
  SPEAKER_01: I'm curious to hear more, Elis, from your standpoint.
• 26:47
  SPEAKER_01: How is the industry reacting to tools like yours suddenly being available?
• 26:51
  SPEAKER_00: I think it's a very early stage.
• 26:53
  SPEAKER_00: I think time will tell.
• 26:54
  SPEAKER_00: I think the first thing that I've noticed is that most people, their reaction is that this is something that they've been looking for.
• 27:02
  SPEAKER_00: It's not as if what we're doing right now is something that no one's ever conceived of, right?
• 27:06
  SPEAKER_00: Yeah.
• 27:07
  SPEAKER_00: It seems like a lot of times when you talk to especially folks who've been in the RBQM or the data management space for a very long time, they'll say something to the effect of like, oh, well, like 10 years ago I asked for this very thing, or five years ago I was imagining something like this would exist and I couldn't get it, or or even more recently, we were in a call with someone who said something like, Oh, I would have bought this product except you guys weren't around 18 months ago.
• 27:32
  SPEAKER_00: And so we're trying to do it now ourselves, which, you know, we'll see how that goes.
• 27:36
  SPEAKER_00: But because it's not very easy, so not to diminish that, but I think it'll be very difficult.
• 27:41
  SPEAKER_00: But I think one thing that Hama said, which was pretty interesting, is your point about the clinical study report as being kind of the final outcome.
• 27:47
  SPEAKER_00: And you know, a lot of what we're looking at are these output variables.
• 27:51
  SPEAKER_00: We're looking at the downstream of a lot of things.
• 27:53
  SPEAKER_00: Auditrail is a set of inputs that we can look at.
• 27:56
  SPEAKER_00: But if you think about it, there's actually a huge universe of different operational inputs we could be examining as it pertains to controlling our studies better that we're not.
• 28:04
  SPEAKER_00: And I do wonder if in a universe where tooling makes analysis easier, whether or not we'd have a better time controlling things like logistics and shipping, communication, and so on, to really kind of process and hammer down at these kind of variability as we as it pertains to like how we work with the sites, what information that they're getting, when are they getting it, in order to kind of better control these kinds of outcome data points.
• 28:25
  SPEAKER_01: Very cool.
• 28:26
  SPEAKER_01: And my reaction is also like, yes, this is totally exists.
• 28:29
  SPEAKER_01: I'm so happy that like someone built this.
• 28:31
  SPEAKER_00: And we don't have to If it it feels somewhat obvious in retrospect, right?
• 28:34
  SPEAKER_00: The knowledge that something that translates you know human language input into a programmable output which can run against data seems very obvious to anyone who's kind of talking about it.
• 28:44
  SPEAKER_00: And yet despite that fact, I have not come across another company in the space trying to do that.
• 28:47
  SPEAKER_01: Well, if it would be easy, uh people would have built this 18 months ago already or or five years ago.
• 28:52
  SPEAKER_01: Well, as we start rounding off this episode, I always ask my guests the same question towards the end.
• 28:57
  SPEAKER_01: And that is if I gave you the transformation in trials magic wand that can change one thing in the life science industry, what would you wish to change?
• 29:04
  SPEAKER_01: I think the first is access to data.
• 29:06
  SPEAKER_02: We need to get to where all of our vendors and all of our systems and tools give me that auditorial data, all of that data that I would want.
• 29:15
  SPEAKER_02: I I really should be able to start with the data that is being collected that I can't get, and then as everything else goes E, right?
• 29:22
  SPEAKER_02: E source, e seg, e this, e that, e the other, start to get the rest of that information and then be able to feed it into a place.
• 29:30
  SPEAKER_02: Because at the end of the day, the best analytics without any data is is a pipe dream.
• 29:35
  SPEAKER_02: It's not anything else.
• 29:37
  SPEAKER_02: So it's get me my data without charging me a ton of money because you're an e-pro vendor and you can.
• 29:44
  SPEAKER_02: Sorry.
• 29:44
  SPEAKER_00: I don't understand why vendors can charge to get access to the data.
• 29:48
  SPEAKER_00: I feel like yeah, I I know that'll that'll just open up a whole canal.
• 29:52
  SPEAKER_00: Yeah, it's wild to me.
• 29:54
  SPEAKER_02: We need to bring some e-pro vendors in too.
• 29:56
  SPEAKER_02: Yes.
• 29:57
  SPEAKER_00: This has nothing to do with what we do in particular, but I think if I could wave one magic wand, it would be for the industry to understand that there's no escaping computers and programming and all sorts of other things.
• 30:13
  SPEAKER_00: And so as a consequence of that, I wish folks would take a step back and think about what an integration API-driven first approach to running a study would look like.
• 30:24
  SPEAKER_00: And so the most obvious example that folks think about is things like digitization of the protocol.
• 30:29
  SPEAKER_00: But that's just one of many, many, many places where if you took a step back and thought, okay, well, let's say we've never run a clinical trial before and we're starting it again in 2025, right?
• 30:40
  SPEAKER_00: How would the systems look and how would this communication happen?
• 30:43
  SPEAKER_00: And how would we have interrupt between the systems?
• 30:45
  SPEAKER_00: There are plenty of companies with more than enough money and more than enough time to actually take this step back and think about it.
• 30:50
  SPEAKER_00: And I I think that they should have the courage to do that because I think it would have a profound implication on how we run studies today.
• 30:56
  SPEAKER_01: I completely agree.
• 30:57
  SPEAKER_01: It would be interesting to imagine it today instead of considering the stuff that we bring with us historically.
• 31:03
  SPEAKER_01: Great.
• 31:03
  SPEAKER_01: Well, if my listeners have questions for you, want to learn more, both Ellis and Hamma, where can they find you?
• 31:09
  SPEAKER_01: How can they reach out?
• 31:10
  SPEAKER_01: LinkedIn.
• 31:11
  SPEAKER_00: Yeah, LinkedIn is great for me.
• 31:12
  SPEAKER_00: You can also reach me directly at email at h-i-r-o-k i at studios.co.
• 31:17
  SPEAKER_00: Not dot com, couldn't get dot com.
• 31:19
  SPEAKER_00: I have no idea what the dot com is.
• 31:20
  SPEAKER_00: Please don't go there.
• 31:21
  SPEAKER_01: Yeah, the those people who grab the domain names, what are they doing with them?
• 31:26
  SPEAKER_01: Thank you so much for joining me today.
• 31:28
  SPEAKER_00: Thank you so much for having us, Ivana.
• 31:30
  SPEAKER_01: Yes.
• 31:30
  SPEAKER_01: We might have been done, but we're not.
• 31:32
  SPEAKER_01: We're we're coming back for another definition that is very important.
• 31:36
  SPEAKER_01: So Illis, will you explain to us what agentic AI is?
• 31:40
  SPEAKER_00: Sure, yeah.
• 31:40
  SPEAKER_00: And to be fair, I I don't know if the industry itself has exactly a singular definition, but I I feel like in a lot of marketing lingo, people like to append the word agentic to essentially make consumers think it's even smarter AI, which I don't think is a really good one.
• 31:55
  SPEAKER_00: Basically, it comes down to are you delegating some amount of the planning and reasoning to the model itself before execution or not?
• 32:01
  SPEAKER_00: So it's the difference between are you asking a single question and coming up with a response?
• 32:05
  SPEAKER_00: Are you saying come up with a plan and then execute on those specific tasks?
• 32:08
  SPEAKER_00: Right.
• 32:09
  SPEAKER_00: So I guess the difference between like full supervision versus partial supervision.
• 32:13
  SPEAKER_00: At the end of the day, by the way, as a consumer, I don't think it should really matter whether or not the system is agentic or not.
• 32:19
  SPEAKER_00: I think you should look at the outputs it produces and see whether or not it's fit for purpose for your business, because that's really what matters.
• 32:24
  SPEAKER_00: I think the one caveat with the quote unquote agentic versus non-agentic is because the agentic planning portion is by definition delegating some amount of the planning to the model itself.
• 32:36
  SPEAKER_00: Your question of you know reproducibility and oversight is going to be a little bit more challenging.
• 32:40
  SPEAKER_00: So I guess it really just depends on you know where it is in the cycle.
• 32:44
  SPEAKER_00: But for most of the people listening, it's probably just coming at you as a marketing term, and I wouldn't look too much into it.
• 32:51
  SPEAKER_00: I don't think it really matters.
• 32:52
  SPEAKER_01: It does not mean AI plus.
• 32:54
  SPEAKER_00: It doesn't really mean AI plus.
• 32:56
  SPEAKER_00: At the end of the day, it's basically the same models being run, and I I wouldn't worry about it too much.
• 32:59
  SPEAKER_01: That's a great clarification to include.
• 33:01
  SPEAKER_01: Thank you.
• 33:03
  SPEAKER_00: Yeah, no worries.
• 33:03
  SPEAKER_00: I don't know if you wanted to stop the recording, but I can actually explain how it works specifically, what it means within the context of our system, or we can kind of so within the context of our system, for example, the reason it's called agentic is because, like, let's say you ask a question, Nahama had to ask earlier, between which two visits do adverse events occur the most often, right?
• 33:22
  SPEAKER_00: And so in this case, you could either ask that question and then provide all the context and hope that the model comes up with the right answer and there's some accuracy that's gonna be lost there, or you can say come up with a plan of solving.
• 33:33
  SPEAKER_00: And so first it's gonna say, okay, first I need to analyze what is like the distribution of visits, and then I'm gonna have to look, and so it breaks down the task into more discrete operations, and then from there, because it's kind of filled its context with all of that inf extra information, then it can go create the final output.
• 33:48
  SPEAKER_00: And so it's created a plan for how to answer the question as opposed to trying to do it in one single go.
• 33:52
  SPEAKER_00: And that's the portion of it that makes it agentic.
• 33:54
  SPEAKER_00: Now, that's not nearly as complex as some of the more fully agentic models where you have uh you have like go research this thing on the internet, and now it's gonna come up with a plan and it has a bunch of tools and it it goes on for 30 minutes or something like that.
• 34:07
  SPEAKER_00: But the accuracy the more ambiguous the plan is and the more long range it is tends to diminish quite a bit.
• 34:13
  SPEAKER_00: And so I think from the purpose of a consumer with agentic AI, you really have to wonder, you know, what is the kind of boundaries of the problem that you're trying to solve?
• 34:21
  SPEAKER_00: And the more scoped and clear the boundaries are, the better the results will be.
• 34:25
  SPEAKER_02: Yeah, and what's cool about it in the tools, so you can see what that reasoning is.
• 34:30
  SPEAKER_02: And one of the questions is wait, how do you get humans to learn how to do this type of reasoning?
• 34:36
  SPEAKER_02: And it provides that first, I've asked a question and it now explains to me, okay.
• 34:40
  SPEAKER_02: I literally explains I'm gonna go take the demography file and I'm gonna take the AE file and I'm gonna take the visit file and I'm going to do some math on the visit date, on the AE file dates and the visit dates, and it starts to explain to you how you would actually go do it if you did it yourself.
• 34:56
  SPEAKER_02: And so that's the first entry point to upskilling and learning.
• 35:00
  SPEAKER_02: Because often when you're learning to do something when you're a kid, you learn to do it by watching someone else do it first.
• 35:05
  SPEAKER_02: And then if it does something really strange, like it decides that AE's all happen on Tuesdays, you're like, whoa, wait, stop.
• 35:13
  SPEAKER_02: You're not making any sense, right?
• 35:15
  SPEAKER_02: You made something up.
• 35:16
  SPEAKER_02: Or I didn't know you wanted between all visits, so it gives me every AE happens when you visit one and visit eight on an eight visit study.
• 35:23
  SPEAKER_02: Well, yeah, I know that.
• 35:24
  SPEAKER_02: That's what you have to tell it.
• 35:25
  SPEAKER_02: Consecutive visits.
• 35:27
  SPEAKER_02: I learned that because it was like, how many A's between visit one and two, one and three, one and four?
• 35:32
  SPEAKER_02: Not helpful, right?
• 35:34
  SPEAKER_02: So then you can go back and challenge that learning, but it gives you a way to you can really do that human augmentation with a computer rather than human being replaced by a computer.
• 35:47
  SPEAKER_02: So it's kind of a bionic brain rather than a robot that's replacing you.
• 35:51
  SPEAKER_00: That last point, by the way, which is to say that these systems aren't perfect, right?
• 35:55
  SPEAKER_00: And they're and they're not mind readers.
• 35:57
  SPEAKER_00: So in this case, we do our best when building the system to provide enough information that's relevant to the industry as well as the study to answer the questions correctly.
• 36:06
  SPEAKER_00: But most of the time, questions themselves are slightly nonspecific.
• 36:10
  SPEAKER_00: And so in this example, you know, it should it should make sense that you're not talking about like pairwise visits up until visit eight, but you weren't specific about that.
• 36:17
  SPEAKER_00: And so when you see in the reasoning, you're like, oh, I'm planning on looking at everything between one and eight.
• 36:22
  SPEAKER_00: You just stop it immediately and go, whoa, whoa, whoa, hold up.
• 36:24
  SPEAKER_00: Actually, I meant, I meant between three and four, specifically.
• 36:28
  SPEAKER_00: And you're like, oh, okay, let me restart my reasoning plan and I'll get the analysis to you in a second.
• 36:32
  SPEAKER_02: That is so helpful, like being able to see what it understands.
• 36:35
  SPEAKER_02: I think a vagentic AI is interns, right?
• 36:38
  SPEAKER_02: You have to sit them down.
• 36:39
  SPEAKER_02: Okay, every morning, what are you doing?
• 36:41
  SPEAKER_02: Right?
• 36:41
  SPEAKER_02: You went off on wait, you you're gonna solve this.
• 36:43
  SPEAKER_02: Here's the problem you solve.
• 36:45
  SPEAKER_02: What am I gonna do?
• 36:46
  SPEAKER_02: Really, you're gonna do it that way?
• 36:47
  SPEAKER_02: No, you're not.
• 36:48
  SPEAKER_02: Let's go back and think about it, right?
• 36:50
  SPEAKER_02: They they're they're basically really nice interns that don't get depressed.
• 36:54
  SPEAKER_02: You don't have to worry about upsetting the intern too much.
• 36:57
  SPEAKER_00: I like to think about it as like a good programmer who doesn't understand your industry, which is kind of like what we're working on so hard.
• 37:03
  SPEAKER_00: We're working on it so hard to not make that the case, but at the end of the day, right?
• 37:07
  SPEAKER_00: Like, you know, you want something that's really good at programming and you know your industry, and so that's kind of the whole sweet spot of you know, you can't do the technical part, it can, but you know, maybe there are specifics that you want to imbue it with, and so that will always make make it better.
• 37:20
  SPEAKER_01: Also.