Latest / This Is Why with Dr. Busti / Diagnostic Tests Explained: Precision & Accuracy in Practice

Transcript

Diagnostic Tests Explained: Precision & Accuracy in Practice · This Is Why with Dr. Busti

• 0:00
  SPEAKER_00: Okay, guys, well, welcome back to our diagnostics test series.
• 0:04
  SPEAKER_00: Now, this is part of our larger EBM and biostatistics series at This Is Why.
• 0:09
  SPEAKER_00: This lecture is going to focus on the concepts of a diagnostic test's precision and its accuracy.
• 0:17
  SPEAKER_00: Okay, now in order to conduct research using new diagnostic processes, or in your review of diagnostic studies, we all need to understand how a particular test we order actually performs.
• 0:32
  SPEAKER_00: Again, this is one of those topics that helps us understand how diagnostic tests were developed and performed so that we can properly use them and interpret the results in their right context.
• 0:46
  SPEAKER_00: And with that, I'm Dr.
• 0:48
  SPEAKER_00: Bustaye, your host on This Is Why, a show that's dedicated to helping you to understand why you do what you do and applying the proper context of the evidence so that we can make evidence-based decisions for the patients we serve every day.
• 1:08
  SPEAKER_00: Now you can find the full playlist of this series on YouTube or at thisiswhy.
• 1:16
  SPEAKER_00: This is part of a larger topic.
• 1:18
  SPEAKER_00: So if you need other parts of the conversation, that's what they're there for.
• 1:23
  SPEAKER_00: It's oriented for both the researcher and the clinician because whether you're a researcher or clinician, we both need to understand what we're reading or publishing and how to interpret it.
• 1:34
  SPEAKER_00: Okay, so this is part of a series.
• 1:36
  SPEAKER_00: Part one is before the study, part two is at the completion of the study.
• 1:40
  SPEAKER_00: Part three, which is where we are predominantly at.
• 1:43
  SPEAKER_00: Part one, remember, is that statistical analysis plan, setting up your groups, the type of data, which statistical analysis, how many patients.
• 1:54
  SPEAKER_00: At the end of the study, you do descriptive analysis and you also do inferential statistics.
• 2:01
  SPEAKER_00: But we are on part three, which is really focused in on those diagnostic tests.
• 2:06
  SPEAKER_00: And if you need the big overview of the topic, then that's where there is an overview lecture that I put together for that.
• 2:15
  SPEAKER_00: But we're on the measurement of accuracy and performance right now.
• 2:21
  SPEAKER_00: Okay.
• 2:22
  SPEAKER_00: Following this, we'll cover some things like sensitivity specificity, again, other measures of performance of that test.
• 2:30
  SPEAKER_00: I'll introduce some of that concepts here, and then we'll hit on later topics on ROC curves, positive and negative predictive values, likelihood ratios.
• 2:38
  SPEAKER_00: Here's the thing: all of those topics can stand alone, including this one.
• 2:44
  SPEAKER_00: But I do pull in and I don't always fully cover all the other components because otherwise the lectures get very long.
• 2:52
  SPEAKER_00: So when we think about precision and/or what is really reliability, we're talking about the consistency, the reproducibility of a test result.
• 3:03
  SPEAKER_00: Okay.
• 3:04
  SPEAKER_00: Does not mean it's always accurate, though.
• 3:07
  SPEAKER_00: It's consistent, it's reproducible, but it may not always be accurate, okay, or true.
• 3:14
  SPEAKER_00: And that's going to talk about accuracy in a minute.
• 3:17
  SPEAKER_00: So this is really about that reliability, precision and reliability.
• 3:22
  SPEAKER_00: If the test experiences random error, this will reduce obviously its precision.
• 3:30
  SPEAKER_00: Okay, so that's reliability and precision.
• 3:33
  SPEAKER_00: Okay.
• 3:34
  SPEAKER_00: What about accuracy?
• 3:36
  SPEAKER_00: Now this reflects the trueness of the test, gives the measurement as close to the actual or true value as compared to the gold standard or sometimes referred to as the reference standard.
• 3:51
  SPEAKER_00: Okay, now we do calculate it, and I'm going to show you how to calculate that going back to our two by two table for diagnostic tests as it relates to diseases, where the accuracy is calculated by taking the true positives plus the true negatives, and you divide that by the total number in the samples.
• 4:14
  SPEAKER_00: Okay.
• 4:15
  SPEAKER_00: And again, I'm going to show you that visually here in just a second.
• 4:18
  SPEAKER_00: We also need to think about systematic error in a study design because that can reduce the accuracy of a test result.
• 4:27
  SPEAKER_00: Okay, so when we talk about precision and accuracy, we have to consider it in the context of bias, an error in the study design that could be introduced.
• 4:40
  SPEAKER_00: Now let's pause and let's talk a little bit about calculations and interpretation.
• 4:45
  SPEAKER_00: I brought this table up before in our overview lecture.
• 4:50
  SPEAKER_00: This really comes up more in our calculation of sensitivity, specificity, our negative, our positive and negative predictive values, and also the likelihood ratio, but it comes up, and I'm introducing it again here because it's part of accuracy as well.
• 5:04
  SPEAKER_00: Remember, I told you those, the true positives and the true negatives divided by the total number of samples.
• 5:11
  SPEAKER_00: And that's what this this is a two by two, where the top and the columns represent the disease.
• 5:17
  SPEAKER_00: It's known, it's diagnosed by the gold or reference standard.
• 5:21
  SPEAKER_00: Whereas on the left are the rows represent the actual tests that we're looking at understanding how it performs.
• 5:29
  SPEAKER_00: In the context of precision, is it reliable, reproducible, and is it also accurate?
• 5:36
  SPEAKER_00: Okay, and so we hope to fall in the true positive or true negative box.
• 5:43
  SPEAKER_00: We don't want to be in the other ones that creates false alarms or missed diseases.
• 5:49
  SPEAKER_00: So when you build one of these, and this is just an example of one, okay.
• 5:54
  SPEAKER_00: This is showing you where for the disease itself, there's a the sensitivity and specificity.
• 6:01
  SPEAKER_00: So sensitivity is calculated by this direction, and the specificity is down this column, okay?
• 6:11
  SPEAKER_00: And you can see those in the formulas here.
• 6:13
  SPEAKER_00: And again, I'm not going to spend time discussing sensitivity specificity right at this moment.
• 6:18
  SPEAKER_00: But I wanted you to see that in the building of this two by two, where you add up the totals in the columns, and then also over here in the rows, they should match up.
• 6:30
  SPEAKER_00: And that this number right here is the total number of samples.
• 6:35
  SPEAKER_00: And that and the true positive and true negative become the formula for accuracy that I mentioned earlier.
• 6:44
  SPEAKER_00: So you can calculate it, or they should calculate it for you.
• 6:48
  SPEAKER_00: Should be provided to you when you're reviewing a paper about a diagnostic test.
• 6:54
  SPEAKER_00: Okay.
• 6:55
  SPEAKER_00: Typically this stuff is just pre-produced, but if you're given the data, you could do it yourself.
• 7:01
  SPEAKER_00: And if the paper didn't do it, you should consider doing that.
• 7:04
  SPEAKER_00: Okay.
• 7:05
  SPEAKER_00: So I wanted you to see the math and how that is done.
• 7:11
  SPEAKER_00: So how is this applies?
• 7:12
  SPEAKER_00: Like, you know, what so what?
• 7:14
  SPEAKER_00: Why do we care?
• 7:16
  SPEAKER_00: Well, you have to think about the equation of error.
• 7:20
  SPEAKER_00: Remember, I talked about that bias, the systematic error that can get introduced.
• 7:26
  SPEAKER_00: We want to make sure that's not happening in the study that we're evaluating on a particular test because I've got to decide whether I'm going to use it or not.
• 7:32
  SPEAKER_00: Okay.
• 7:34
  SPEAKER_00: So the measurement of something has truth, which is influenced by a good study design.
• 7:42
  SPEAKER_00: Right?
• 7:42
  SPEAKER_00: It's also when we're reading about the paper, we're using a critical appraisal tool, uh C A T to assess for the truth.
• 7:53
  SPEAKER_00: We want to know, is it a real value?
• 7:56
  SPEAKER_00: And we're trying to discern bias, which the researcher certainly can introduce, right, by the way, the methodology is done.
• 8:04
  SPEAKER_00: And then we're looking for that systematic error as a reader.
• 8:07
  SPEAKER_00: And that's what the critical appraisal tools are really helping us to also dive into.
• 8:12
  SPEAKER_00: But when we get to this random error over here, there's chances and some variability and size.
• 8:18
  SPEAKER_00: Well, that's influenced by the size of the samples.
• 8:21
  SPEAKER_00: And then our obviously our confidence interval p values start to reflect that.
• 8:27
  SPEAKER_00: But look stepping back, and we can consider these three components as it relates to a measurement of something.
• 8:34
  SPEAKER_00: We have truth, we have the potential risk for bias being introduced by the researcher, and then we have the random error itself that can occur.
• 8:44
  SPEAKER_00: And so when you look at those, again, the target of rely uh the precision, remember it's reproducible.
• 8:54
  SPEAKER_00: It might be inaccurate, but it's reproducible, it's always inaccurate.
• 8:59
  SPEAKER_00: We obviously want precision with accuracy.
• 9:03
  SPEAKER_00: And so when you start thinking about random error and bias, we want low random error over here, and we want low risk for bias so that our precision and our accuracy are where they're supposed to be, the true result.
• 9:21
  SPEAKER_00: But as you have bias being introduced over this direction, you can see that it can change the precision, the reliability of the results.
• 9:34
  SPEAKER_00: If you have high degree of error, well, it's not very accurate, is it?
• 9:42
  SPEAKER_00: It's all over the place.
• 9:44
  SPEAKER_00: And the degree of that uh risk as it relates to bias is also there.
• 9:49
  SPEAKER_00: So that's what this is trying to help you to see in your mind visually when you're studying about an or you're reviewing a paper about a new test, or if you're the researcher doing it, you want to try to get to that one top left corner target.
• 10:07
  SPEAKER_00: So we can judge a new test's accuracy and later on some of its specificity and sensitivity against a gold standard or that reference standard that defines the true disease status.
• 10:20
  SPEAKER_00: Remember, not all, as I mentioned in the overview, not all gold standards or reference standards are perfect.
• 10:27
  SPEAKER_00: In fact, when you develop new diagnostic tests, they may actually perform better.
• 10:32
  SPEAKER_00: And so then you have to ask the question is those are those false.
• 10:37
  SPEAKER_00: Are those are they false positives?
• 10:41
  SPEAKER_00: So remember the gold standard is not always is not always going to be golden.
• 10:46
  SPEAKER_00: So that's where I was just saying it's imperfect, right?
• 10:50
  SPEAKER_00: Um, and so you want to make sure that you have checked these considerations.
• 10:57
  SPEAKER_00: You have imperfect cultures for an infection.
• 11:00
  SPEAKER_00: You maybe the way they were collected, there's contamination, the clinical diagnosis uh without tissue confirmation, right?
• 11:08
  SPEAKER_00: Older or less sensitive imaging modalities, okay.
• 11:13
  SPEAKER_00: Um doing an open mouth x-ray versus a CT scan of a cervical spine, very different, right?
• 11:20
  SPEAKER_00: Subjective reads, okay.
• 11:23
  SPEAKER_00: So when imperfect gold standard bias is being introduced, we think of things like when the reference standard misclassifies the patients and the new test performance becomes then biased by that.
• 11:36
  SPEAKER_00: Okay.
• 11:36
  SPEAKER_00: So for everyday clinicians, you know, reading the literature, we're we're gonna ask ourselves, how good is the gold standard even in this study?
• 11:46
  SPEAKER_00: Was it applied the same way to every patient in the study?
• 11:52
  SPEAKER_00: And is there any better reference available?
• 11:56
  SPEAKER_00: So you always want to know, and you go back to the question of a PERT a pyro structure in your evaluation trying to acquire information, acquire the best study, you're looking for papers that did that.
• 12:09
  SPEAKER_00: They use the proper gold standard or reference test for the patients in the question being looked at.
• 12:19
  SPEAKER_00: Otherwise, this new test may not perform and represent some degree of bias.
• 12:28
  SPEAKER_00: And so that's why critical appraisal tools like the quadus analysis is done to help guide us through that process.
• 12:35
  SPEAKER_00: And again, I introduced that in the overview lecture when we talked about how to review a paper and what are the tools.
• 12:43
  SPEAKER_00: All right, a test that looks excellent versus a weak gold standard may disappoint in also real clinical practice.
• 12:52
  SPEAKER_00: So, core concepts on this particular topic: precision is about reliability, reproducibility.
• 12:59
  SPEAKER_00: How consistent and reproducible are those results?
• 13:03
  SPEAKER_00: They may not always be accurate.
• 13:05
  SPEAKER_00: We want them to be, but that's where accuracy that is the trueness of the results from that test.
• 13:13
  SPEAKER_00: And we want to make sure that we are doing our best to merge both of those and remove bias and random error from that process in the measurement.
• 13:25
  SPEAKER_00: Okay, guys, this topic can be seen in part of the full diagnostic playlist again over at thisiswide.health, okay, and part of our EBM series and collections.
• 13:35
  SPEAKER_00: You can also find them on YouTube.
• 13:38
  SPEAKER_00: All right.
• 13:38
  SPEAKER_00: If you're not a subscriber, please consider joining.
• 13:41
  SPEAKER_00: We really would appreciate that.
• 13:42
  SPEAKER_00: Also, some support.
• 13:44
  SPEAKER_00: We'll give it a thumbs up, maybe post a comment, share some experience, additional perspective that we can all learn from each other.
• 13:52
  SPEAKER_00: So thank you for joining me.
• 13:53
  SPEAKER_00: I'm Dr.
• 13:54
  SPEAKER_00: Bust Eye, and we'll see you next time on the next topic as it relates to evidence based medicine and diagnostic tests.