Latest / Cardio Pod: Heart of the Matter / Statistical Considerations in Heart Failure Research

Transcript

Statistical Considerations in Heart Failure Research · Cardio Pod: Heart of the Matter

• 0:00
  Welcome.
• 0:01
  I'm Chris O'Connor, the principal investigator of the Heart Failure Collaboratory.
• 0:10
  And we're here with the Heart Failure Collaboratory podcast called Heart of the Matter.
• 0:16
  And this is our opportunity to do a deeper dive into the concepts that we discussed in
• 0:21
  some of our recent think tanks.
• 0:23
  And we're very fortunate to have Janet Witteas lead our statistical considerations, annual
• 0:29
  meeting of the Heart Failure Collaboratory.
• 0:31
  It's been going on for several years now and we got into some really challenging topics.
• 0:37
  And the topic that we began with, and Vanessa Bloomer is not here, led us in that conversation,
• 0:43
  is estamens.
• 0:44
  And what is an estamens?
• 0:46
  And before we jump into that topic for today, let's just go around the room and introduce
• 0:50
  ourselves to our audience.
• 0:52
  Janet?
• 0:53
  Hi, I'm Janet Witteas.
• 0:54
  I'm a statistician.
• 0:55
  I've been involved in lots of clinical trials over the years and right now I'm just consulting.
• 1:02
  Thank you.
• 1:03
  Thank you, Janet.
• 1:04
  Mitch?
• 1:05
  I'm Mitch Sotka.
• 1:06
  I'm a heart failure and transplant cardiologist in the NOVA Health System in Virginia.
• 1:08
  Bill Abraham?
• 1:09
  Yeah, Bill Abraham, a heart failure and transplant cardiologist at The Ohio State University.
• 1:14
  And Eric Leifer.
• 1:15
  Hi, I'm Eric Leifer.
• 1:17
  I'm a statistician in the National Heart, Lung, and Blood Institute.
• 1:20
  So we got the team here.
• 1:21
  We got clinicians, clinical trialists, and statisticians.
• 1:25
  And Janet, this concept of estamens has been around for a while.
• 1:28
  I feel like we've had it in there, but I never felt like we put our arms around it this time
• 1:34
  and it said, wow, I didn't know we could talk for three hours about estamens.
• 1:38
  The story of estamens is actually kind of interesting.
• 1:41
  We have in statistics, estimands, estimates, estimators, and they're all different from
• 1:47
  each other.
• 1:48
  And it's very confusing.
• 1:50
  The reason for developing this concept of estimate is that in many clinical trials,
• 1:57
  you're trying to figure out what you're trying to estimate has not been clear.
• 2:01
  I think in cardiology and in heart failure trials, it pretty much has been clear.
• 2:07
  And this is not such a problem in your trials, in heart failure trials, but in trials of
• 2:14
  symptoms, trials of short-term trials where you're measuring continuous outcomes, how
• 2:21
  you feel, what your pain is and so forth, rather than hard clinical outcomes the way
• 2:27
  heart failure trials are usually designed.
• 2:29
  What you're trying to estimate from these trials and what you actually calculate can
• 2:36
  be very different.
• 2:38
  And so what an estimate is, it's a disciplined way of saying, I have a trial, and at the
• 2:45
  end of the trial, I want to calculate a number, and I want that number to represent something
• 2:52
  that I understand.
• 2:53
  And it's that thing that I want to understand.
• 2:56
  That is the estimate.
• 2:58
  You do the study, you do a calculation, you get an estimate, that's a number, like my
• 3:05
  hazard ratio is 1.2, and that should give you the best sense that the trial has given
• 3:11
  you of what that value is in the population at large.
• 3:16
  And so the framework of estimate is a way of disciplining yourself.
• 3:21
  What your study gets at is what it's trying to get at.
• 3:24
  So it's sort of setting the stage, the roadmap, but didn't we do this with SAPs, this statistical
• 3:31
  and analytical plan, and that big document that you and Eric would create for us when
• 3:37
  we did clinical trials seemed like it had the estimate in there.
• 3:41
  And I'm going to ask Eric to comment on that.
• 3:44
  Is estimate just another fancy word to hover over the SAP?
• 3:49
  I would say that certainly we've been thinking, whenever we designed a clinical trial, we're
• 3:54
  thinking about the questions we want to answer.
• 3:57
  And so it has always been there in some regard.
• 4:00
  But I think having a sentence or a paragraph in your protocol now to say, this is really
• 4:06
  the question we're trying to answer, it also speaks to things, how do we measure things?
• 4:12
  How do we think about patients who might cross over to another treatment or who might not
• 4:16
  adhere or who might just drop out for other reasons?
• 4:20
  How do we handle that sort of data?
• 4:22
  And that data always comes up in trials.
• 4:24
  So if we have our estimate, we say, this is what we're really trying to understand, for
• 4:29
  instance, are we trying to understand what happens if patients start off on a treatment
• 4:33
  and then might switch over and we're not worried and then we're just going to say, well, they
• 4:36
  started it and we're just going to follow them from there or if we're going to stop
• 4:40
  following them at the time they cross over, we really want to have a much purer comparison.
• 4:44
  This really forces us to think about what is the exact comparison that we're really
• 4:49
  interested in making.
• 4:51
  And so there's verbiage in an estimate.
• 4:53
  And then when you get to the estimates, that's when sort of the statistics come in that we
• 4:57
  think that Jen and I went to school for in terms of writing down formulas to say how
• 5:01
  we're going to answer a particular question.
• 5:04
  But the estimate really tries to understand what we want to go after in terms of how patients
• 5:10
  might behave during the trial.
• 5:11
  That's really well said.
• 5:14
  So Bill, you design a lot of trials.
• 5:17
  You've run a lot of trials.
• 5:18
  It's a concept that is moving forward to be clear about the rules of engagement, to have
• 5:24
  that estimate actually written in the methods of your publication.
• 5:28
  Yeah, I think that it is.
• 5:30
  And I think that it's important because in some ways I look at this perhaps very simplistically
• 5:35
  as also being a statement by those who are designing trials of what might represent really
• 5:42
  an important difference, an important outcome for patients.
• 5:47
  It's not just about statistical significance and we may get into a discussion of p-values,
• 5:52
  but I think it's also a discussion of what are we trying to do for our patients?
• 5:56
  What are we trying to accomplish here?
• 5:58
  And if you look at some trials, they enroll a very large number of patients, are often
• 6:04
  overpowered.
• 6:05
  They have some treatment effect baked into the assumptions, but they might reach a significant
• 6:11
  p-value with a treatment effect that is really less than impressive.
• 6:15
  And so I think really trying to have a statement going forward for not only statistical significance
• 6:22
  but what we believe these results may mean to patients is really important.
• 6:27
  And I think it's somehow in this concept that we're talking about today.
• 6:30
  And Mitch, when you look at, you've looked at a number of clinical trials and efforts,
• 6:36
  development programs.
• 6:37
  How often are people not putting forth the estimate?
• 6:41
  I think that the key here is that the framework for the estimate is frequently used when you
• 6:46
  talk about regulatory submission or you talk about putting together an SAP, but it didn't
• 6:51
  always happen that way.
• 6:54
  And it gives a clear language on what to use.
• 6:57
  And sponsors putting together a trial, put together what they think the estimate is,
• 7:02
  but then it allows us to discuss the difference between perhaps what they think the estimate
• 7:08
  is and what they're truly measuring.
• 7:09
  For an example, I want to use the recent colonoscopy paper that was in the New England Journal
• 7:13
  of Medicine where the title of the manuscript is Affected Colonoscopy Screening on Risk
• 7:18
  of Colorectal Cancer and Related Death.
• 7:21
  So, the intervention is not about screening, actually.
• 7:24
  The intervention is about mailing a letter to people to tell them to get their colonoscopy
• 7:29
  screening testing and really show that that wasn't very effective.
• 7:32
  But it's being interpreted as that colonoscopies are not effective screening.
• 7:35
  The estimate was not colonoscopy, it was mailing a letter and what the effect that has on the
• 7:42
  colorectal cancer death.
• 7:44
  And so, the estimate framework really gives that language to be able to very precisely
• 7:48
  discuss what you're intending to measure and how you want to interpret it upfront so
• 7:52
  that everyone can be in agreement before the trial is undertaken.
• 7:56
  And it's everything, Janet.
• 7:57
  It's the population, is it intention to treat, is it on treatment analysis, how do we handle
• 8:04
  the crossovers?
• 8:05
  As Eric said, what about censoring and the follow-up?
• 8:08
  It's putting all that in there.
• 8:10
  Yeah, that's exactly right.
• 8:12
  I'm glad you mentioned censoring because one of the things that can happen, Eric talks
• 8:17
  about crossover, which means you're randomized to one group and you go into the other.
• 8:22
  But you can also have you're on a treatment and then something happens to you and you
• 8:28
  get a different treatment, not one of the two that are in the study.
• 8:32
  You get some surgery or you get a drug completely different from what's being studied.
• 8:37
  How do you handle that?
• 8:39
  Those are the kinds of things that I think, yes, there were huge SAPs, statistical analysis
• 8:45
  plans, sometimes 150 pages.
• 8:47
  But when you looked at the details of what the calculated estimate was and you said,
• 8:54
  okay, if I calculate this thing, what is it really getting at?
• 8:59
  Often it wasn't getting at what you think it is.
• 9:01
  I just read a couple of statistical analysis plans in fields other than cardiology.
• 9:08
  And this was very recently, within the last few weeks.
• 9:12
  And they had the estimate and framework listed.
• 9:14
  But their estimate didn't fit.
• 9:17
  So there's going to be a step where, yes, you use the language and Chris is exactly
• 9:23
  right.
• 9:24
  It's not only what you're trying to estimate, but in whom, under what circumstances, how
• 9:30
  are you handling all the things that happen in trials to patients and in life to patients
• 9:37
  that can make your analysis more complicated and your inference more complicated.
• 9:42
  And these analysis plans had the language, but not the rigor.
• 9:48
  And I think it's quite a long time for us all to be able to learn how to use this framework
• 9:54
  in an effective way.
• 9:56
  That is a really good lesson to say that the pieces were all there, but they weren't connected.
• 10:01
  So we've got to get that estimate right to match that estimate.
• 10:05
  That's really exciting.
• 10:06
  Let's quickly turn to one of the components of what we try to measure, Bill.
• 10:12
  And that's we may have a symptom, patient reported outcome, and we talk about meaningful
• 10:18
  differences in p-values.
• 10:21
  And we get twisted around with this.
• 10:23
  I heard this again in a recent commentary about how could something have such a large
• 10:30
  p-value or small p-value, and you're claiming that there's no meaningful difference.
• 10:36
  The Ascent Trial, the serotype, we studied 9,000 people, very small p-value, very small
• 10:42
  difference in dyspnea.
• 10:43
  And people are saying that it wasn't meaningful.
• 10:46
  And I agree with that.
• 10:47
  But tell us, how does the world tease that out, meaningful difference in a p-value?
• 10:53
  I think, first of all, I'm a great advocate of looking at the totality of data.
• 10:58
  Clinical trials shouldn't be judged based on a single p-value relative to a primary
• 11:03
  endpoint.
• 11:04
  We've got to have the flexibility to understand the totality of the data and what it really
• 11:08
  means for our patients.
• 11:09
  What we're looking for with the p-value is sort of a level of certainty in a particular
• 11:15
  finding, and we can make some judgments about how certain we want to be in regard to what
• 11:20
  the outcomes are for the patients.
• 11:22
  I think we should have that flexibility in interpreting clinical trials.
• 11:26
  But where we actually have, or at least can agree upon by consensus, what represents a
• 11:32
  clinically meaningful outcome, we should strive to do that.
• 11:36
  We should include that statement in our protocols and in our statistical analysis plans.
• 11:42
  And we should look at not only the p-value or statistical significance, but also clinical
• 11:48
  meaningfulness as well.
• 11:50
  And just as an example, there are two patient-reported sort of endpoints, quality of life instruments
• 11:58
  or health-related quality of life questionnaires.
• 12:01
  The Minnesota living with heart failure questionnaire and the Kansas City cardiomyopathy questionnaire.
• 12:06
  Both of these are included in FDA's medical device development toolbox.
• 12:12
  Both have been studied extensively and well validated.
• 12:16
  And they've landed on using anchoring points, other endpoints, either clinical outcomes
• 12:21
  or other instruments such as a patient global assessment.
• 12:25
  They've landed on some sort of threshold cutoff for clinical meaningfulness, which turns out
• 12:31
  to be about five points.
• 12:33
  So we have trials of 10,000 patients that demonstrate a very small p-value with a one-point
• 12:39
  improvement in quality of life score.
• 12:40
  Is that really meaningful?
• 12:42
  Then we have smaller trials of several hundred patients that have a p-value that might be
• 12:46
  0.049 or even 0.051, but demonstrate a 10-point improvement in quality of life score.
• 12:53
  I think the way that I would look at those studies and interpret that data are different
• 12:57
  than what the p-value would indicate.
• 13:00
  Very well said, Bill.
• 13:01
  Eric, how do you look at the p-value?
• 13:03
  Is it totality of information?
• 13:06
  Does it provide you with any enhanced knowledge of the trial?
• 13:10
  I mean, the p-value tells you if there is any difference between the two groups.
• 13:15
  And so if you have a large enough trial, usually there'll be some sort of small difference
• 13:21
  and you'll detect that.
• 13:23
  But I agree with what Bill was saying, what Janet was saying, that really the p-value
• 13:27
  needs to be supplemented with both an estimate, the value of once you've gone through the
• 13:34
  whole process of figuring out the estimand, then you get your estimate, as Janet described,
• 13:39
  where it's your numerical value of your estimand, and you get a confidence interval for the
• 13:44
  estimand.
• 13:45
  In any journal, you'll always see a p-value and you'll see the estimate and the 95% confidence
• 13:50
  interval.
• 13:51
  And that gives you a much better idea as to really what the treatment effect was.
• 13:56
  Also in this trial that you were describing with 9,000 patients, you're going to find
• 14:00
  that yes, you had a significant p-value, but once you supplement it with additional information,
• 14:06
  it might not be so impressive after all.
• 14:08
  So it's really a totality of evidence question and a p-value always needs to be supplemented
• 14:14
  with the estimate or the estimand and its confidence interval.
• 14:17
  Great take-home lesson there.
• 14:19
  I couldn't agree more with that.
• 14:21
  Mitch, you've seen a lot of p-values above 05, below 05.
• 14:25
  What does it mean to you?
• 14:26
  I have to say that my biggest issue with the p-value is the way that it is used as a dichotomous
• 14:32
  reference.
• 14:33
  So we basically say that if the p is less than 0.05, we say, oh my gosh, it's amazing.
• 14:40
  And if it's greater than 0.05, we say, oh, it's not.
• 14:42
  And just like confidence intervals, which I think are more inherently interpretable for
• 14:46
  people, giving you a range of uncertainty around your treatment effect estimate, your
• 14:52
  point value, the p-value is not really interpretable in the same way for most people, but it really
• 14:58
  should be looked at similarly as a continuous measure, right?
• 15:02
  A smaller p-value should give you relatively more confidence that your result truly represents
• 15:07
  a difference in an effect size.
• 15:10
  And a relatively larger p-value, you should be less confident in your results.
• 15:14
  But from a standpoint of interpretation, should I use this therapeutic for my patients, I
• 15:19
  think it depends on the totality of evidence, right?
• 15:21
  It depends on how many patients you have, it's a super rare disease.
• 15:24
  You're probably not going to be able to really demonstrate a tiny p-value, where if it's
• 15:28
  a super common disease, you can expect to enroll a lot of patients and be able to clearly
• 15:33
  demonstrate an effect in maybe just one large trial.
• 15:36
  So I think that is the way that really p-values are misused.
• 15:40
  And this 0.05 standard really just needs to be thrown out.
• 15:44
  Wow.
• 15:45
  Okay.
• 15:46
  Janet, you have the last word on p-values.
• 15:48
  Okay.
• 15:49
  I would like to say, I would like to turn the language around.
• 15:54
  The estimate is the cake and the p-value is the icing.
• 15:58
  We start with the estimate, what is the value and the confidence interval.
• 16:02
  And then you stick the p-value in to say, I believe it or not.
• 16:07
  Very often we start with the p-value and that's not the important thing to start with.
• 16:11
  So that's what I want us to do, to turn our language around.
• 16:15
  We have the estimate and the p-value asks us how much we should believe it.
• 16:19
  I love that.
• 16:20
  And I hope you're advocating for chocolate cake because I love chocolate cake.
• 16:25
  But I think having that estimate and confidence and thinking about that goes beyond just clinical
• 16:30
  trials.
• 16:31
  You go to a restaurant and Janet and I have intersected at some very good restaurants
• 16:35
  and we say that meal was good.
• 16:37
  There's confidence intervals around it.
• 16:38
  And there's an estimate on that.
• 16:39
  How good was that meal?
• 16:41
  So I think it's a mindset.
• 16:43
  It's a framework and a mindset that we all have to change to think about not a dichotomous
• 16:49
  world, but a world that is a continuum.
• 16:52
  Biology is a continuum and we have estimates and confidence around that estimate.
• 16:57
  So thank you very much.
• 16:59
  It's been a really intriguing conversation today.
• 17:01
  This is really a download from our larger think tank meeting.
• 17:05
  And again, if you go on for hours, which is amazing to talk about estimates and p-values
• 17:10
  and confidence intervals.
• 17:12
  But we have to end here.
• 17:14
  I want to just thank my colleagues for being part of this podcast, Heart of the Matter.
• 17:19
  And we look forward to you joining our next podcast, which is going to be the dangerous
• 17:23
  intersection of acute kidney injury and acute heart failure.
• 17:28
  What does that mean?
• 17:29
  Thank you.
• 17:30
  Estoy muy her understanding aproximadamente 15.