Latest / Sleep Science Friday Series / Shall we rush to be brief? Methodological and clinical costs of shortening insomnia scales

Transcript

Shall we rush to be brief? Methodological and clinical costs of shortening insomnia scales · Sleep Science Friday Series

• 0:10
  SPEAKER_00: Hello everyone and welcome to our Sleep Science Friday podcast.
• 0:14
  SPEAKER_00: Today we are talking about questionnaires and sleep research.
• 0:18
  SPEAKER_00: Sleep research relies heavily on questionnaires, but recently some of these tools are getting shorter and shorter.
• 0:24
  SPEAKER_00: Is it a good thing or are we losing important along this way?
• 0:29
  SPEAKER_00: Today we are discussing a recent third-provoking editorial published by Matteo Carpe and Daniel Ruiva Marquez titled Shall we Rush to Be Breathed?
• 0:38
  SPEAKER_00: Methodological and Clinical Costs of Shortening Insomniac Scales, which was published in the journal Sleep and Breathing.
• 0:45
  SPEAKER_00: This paper raises an important question: are we losing important clinical and scientific information in the process of shortening and simplifying how we measure sleep problems?
• 0:56
  SPEAKER_00: I am joined today by one of the authors, Dr.
• 0:58
  SPEAKER_00: Matteo Carpi, who is a researcher at the Department of Human Neuroscience at Sapienza University of Rome.
• 1:05
  SPEAKER_00: And today we will talk about the science behind sleep questionnaires, psychometrics, and why sometimes shorter may not always be better.
• 1:14
  SPEAKER_00: Matteo, hello.
• 1:16
  SPEAKER_00: Welcome to our podcast.
• 1:19
  SPEAKER_01: Hi, Ludwiglia, and thanks for having me hosted in the podcast.
• 1:23
  SPEAKER_00: Thank you very much for joining us and for agreeing to share your experience with us.
• 1:29
  SPEAKER_00: So to start off, insomnia is one of the most common sleep disorders affecting up to 20-30% of populations.
• 1:37
  SPEAKER_00: And much of its diagnosis relies on self-report.
• 1:42
  SPEAKER_00: Why are self-report questionnaires so important for diagnosis of insomnia, for insomnia research and clinical practice?
• 1:50
  SPEAKER_01: Yeah, when when we deal with insomnia, you know, uh you said already we are much into self-report measures, and we know from international classification of sleep disorders how insomnia is mainly defined through subjective experience, in fact.
• 2:06
  SPEAKER_01: And so we are dealing with difficulty falling asleep, difficulty staying asleep, or even dissatisfaction with sleep and real-time consequences of uh perceived problems.
• 2:15
  SPEAKER_01: And these are things that the person actually reports.
• 2:19
  SPEAKER_01: So uh self-report is not secondary here, it is central, and we got guidelines positing that self-report instruments, standardized instruments, are uh the baseline for uh insomnia assessment.
• 2:33
  SPEAKER_01: But this is increasingly indeed the case uh also for other sleep-related conditions where self-reports are increasingly complementing instrumental assessment, and this is something which aligns with the growing emphasis on patient reported outcomes as a primary source of clinical information.
• 2:50
  SPEAKER_01: So this is something which uh uh possibly uh involves sleep medicine uh as a whole field, I guess.
• 2:57
  SPEAKER_00: Yes, I agree with you, but we are also seeing that nowadays there is a strong trend towards shorting, towards short and ultra-short questionnaires and sleep medicine and also in insomnia practice.
• 3:10
  SPEAKER_00: What do you think?
• 3:11
  SPEAKER_00: What is driving this movement and this trend?
• 3:15
  SPEAKER_01: Yeah, you know, that's uh the the trend is what motivated us, me and Danielle in writing our editorial and just uh posing the question whether we should really rush to be so brief in insomnia assessment, insomnia questionnaires, because uh it's something which is happening, something you can find in the literature.
• 3:33
  SPEAKER_01: Uh when we worked with Insomnia, we got uh very famous widespread Insomnia Severity Index and already brief instruments, seven items in total.
• 3:42
  SPEAKER_01: And we are seeing increasingly briefer version of it, so like the three-item version, the two-item version.
• 3:48
  SPEAKER_01: We got even a one-item screener for uh sleep disturbance as a whole concept.
• 3:54
  SPEAKER_01: And that's something that I understand because brevity is uh naturally attractive as a way to assess and diagnose and be efficient in clinical management.
• 4:04
  SPEAKER_01: But it is also something which we possibly should not uh uh assume as an automatic improvement because uh when we do the choice to reduce, we are also making a choice about some information to drop, some information to keep.
• 4:21
  SPEAKER_01: And I think there are several reasons moving people looking for uh shorter and shorter measures for sure.
• 4:28
  SPEAKER_01: We are seeking, always seeking for efficiency in research and in clinical practice.
• 4:32
  SPEAKER_01: We want instruments that are quick to administer, easy to score, but we also want instruments feasible for large-scale studies.
• 4:40
  SPEAKER_01: So uh we also want uh to have shorter scale for assessment treatment efficacy.
• 4:45
  SPEAKER_01: So it's perfectly understandable, but my point here is that it is uh uh something which is not always uh sufficient in clinical practice as well as in research.
• 4:56
  SPEAKER_00: Yeah, I see.
• 4:57
  SPEAKER_00: So it's mainly about saving time in clinical practice and saving time in epidemiological studies to make them easier, as I understand.
• 5:07
  SPEAKER_00: But we gain efficiency this way, but as you told, your paper suggests that you might be losing something important.
• 5:13
  SPEAKER_00: So let's go in more detail in this.
• 5:16
  SPEAKER_00: And in your paper, you outline four key issues that researchers and clinicians should consider before shortening a scale.
• 5:25
  SPEAKER_00: So let's look at them more carefully.
• 5:27
  SPEAKER_00: Your first point is about psychometric frameworks.
• 5:31
  SPEAKER_00: Could you explain what happens um statistically and when the researchers, uh, clinicians shorten a scale?
• 5:40
  SPEAKER_01: Yeah, we can go over the four points.
• 5:42
  SPEAKER_01: We try to split the things uh uh into easily to remember points, but they are interrelated, so we we'll see throughout.
• 5:50
  SPEAKER_01: But uh the first one, as you say, is the the psychometric standpoint.
• 5:55
  SPEAKER_01: Uh, indeed, when we uh do an abbreviation uh in practice, we are we are relying on statistical procedures, we are relying on psychometrics and possibly mostly on the classical uh psychometric tools and criteria.
• 6:10
  SPEAKER_01: So uh internal consistency, factorial structure, and discrimination, how well an instrument discriminates between cases and non-cases, so people with insomnia or without insomnia, for example.
• 6:21
  SPEAKER_01: And that's something we broadly speaking is related to how well the items making up a scale correlate uh with each other.
• 6:30
  SPEAKER_01: Uh that's why uh the problem arises, because uh behind this criteria there's an implicit way of thinking about measurements.
• 6:40
  SPEAKER_01: So uh shortening is never purely a technical uh operation, it depends on the assumptions uh you are making about uh uh what the scale is supposed to represent, because uh we are making a selection of items of questions to ask people, uh, so we are assuming something implicit about what we are going to measure.
• 7:03
  SPEAKER_00: You write that many scales assume a latent variable model, if I understood correctly.
• 7:10
  SPEAKER_00: And this assumption makes items appear interchangeable.
• 7:15
  SPEAKER_00: Um, so could you explain this more for our audience, please?
• 7:21
  SPEAKER_01: Yeah, sure.
• 7:22
  SPEAKER_01: Uh in a latent variable model, you know, which is the most commonly implied uh one in psychological measures and psychological measurement, uh, which typically uh deals with infer and observable variables, we assume that there is something like an underlying construct, for example, sleep quality or insomnia severity itself, and that the items, the questions we use in a questionnaire, are indicators of that construct.
• 7:49
  SPEAKER_01: So the idea is that uh the hidden construct, the Latin variable itself, causes the item responses.
• 7:56
  SPEAKER_01: And if you think like this, then the items can start to look interchangeable as long as they reflect the same Latin triit and are an expression of the same construct.
• 8:07
  SPEAKER_01: Um maybe you you do not need all of them, or you can substitute one of them with another, behaving just the same.
• 8:15
  SPEAKER_01: So this is one of the often implicit, possibly strong drives for abbreviation.
• 8:20
  SPEAKER_01: And the reason for which abbreviation becomes so attractive is because we are somewhat allowed by our assumption to think that we can substitute or reduce the number of items, obtaining the same statistical results, basically.
• 8:32
  SPEAKER_00: Yeah, thank you.
• 8:34
  SPEAKER_00: But you also mentioned alternative approaches like network models.
• 8:39
  SPEAKER_00: So, how do this change the way we think about um these individual items of the scale of the questionnaire?
• 8:48
  SPEAKER_01: Yeah, that's uh a technical point which requires sleep uh clinicians and sleep researchers to get a little bit found into psychometrics, but I guess the investment uh could be well repaid because we got alternative models indeed.
• 9:02
  SPEAKER_01: We got uh alternative ways of thinking about self-reference.
• 9:06
  SPEAKER_01: One is the so-called formative model.
• 9:08
  SPEAKER_01: The only the other one which got uh uh quite well known uh in sleep research recently is the network model.
• 9:15
  SPEAKER_01: And these other models just do not assume that uh a single latent variable causes a construct to uh to produce uh some patterns of response.
• 9:26
  SPEAKER_01: For a network perspective, for example, which is a way of thinking about measurement which resonates with complex system thinking, uh, symptoms or behaviors may influence each other directly.
• 9:39
  SPEAKER_01: So it's not just assuming a single latter variable, but seeing items, questions, symptoms, as uh uh as something which interacts, producing meaningful uh factors of uh uh of interactions, uh, making up the clinical picture we observe in practice.
• 9:57
  SPEAKER_01: So this changes a lot when it comes to abbreviation because uh in this latter model, uh, dropping one item may mean losing uh a big piece of the whole system, not just shortening a question.
• 10:08
  SPEAKER_01: And so we have to be way more careful about which information are we going to drop or to give.
• 10:15
  SPEAKER_00: Yes, exactly.
• 10:16
  SPEAKER_00: And this is the second point which you highlighted in your paper, the information loss when we consider when we don't consider these alternative models.
• 10:25
  SPEAKER_00: So, what kind of information disappears when a scale is shorter?
• 10:30
  SPEAKER_00: Could you give more detail on this?
• 10:33
  SPEAKER_01: Yeah, to be honest, uh I'd say several kinds, because uh uh when we as researchers do abbreviation, do psychometric studies, we are mostly driven by data.
• 10:44
  SPEAKER_01: So we mostly work with statistical models, and data-driven abbreviation is frequently blind to the kind of information it drops.
• 10:53
  SPEAKER_01: Uh, for the most, we rely on having uh uh efficient discrimination as the original instrument.
• 11:00
  SPEAKER_01: So uh we can say that with less items we do the same job, but we may lose specific symptoms, we may lose specific patterns which may be of interest for the clinician, which may be of interest for the researcher trying to answer some particular question.
• 11:16
  SPEAKER_01: For example, we may lose the possibility to identify different profiles of insomnia, different symptom patterns, or we may lose details about how the time consequences or emotional aspects interact with more nighttime aspects of the disorder.
• 11:32
  SPEAKER_01: We may lose information that is maybe not very important for compounding a total score, but which is very important for the clinician or just theoretically.
• 11:44
  SPEAKER_00: Yes, and you mentioned this also in your paper.
• 11:49
  SPEAKER_00: The recent research showed that several insomnia subtypes could be discovered or uh distinguished uh using the full insomnia severity index.
• 11:59
  SPEAKER_00: And as I honestly, one of those papers uh uh belongs to you and your co-authors, and one comes from China.
• 12:06
  SPEAKER_00: Could you tell a bit more about this uh research and what kind of types of insomnia you distinguished based on insomnia severity index?
• 12:17
  SPEAKER_01: Oh, yeah, sure.
• 12:18
  SPEAKER_01: We've got uh likely a bunch of research coming out recently about phenotyping and self-reported-based phenotyping, which is something interesting because we can maybe uh come to distinguish some aspects or different presentations of insomnia which may be uh particularly challenging for the clinician in adapting treatments or just adapting the assessment instruments clinicians use.
• 12:43
  SPEAKER_01: So uh, from our experience, but even before, from the experience of uh the Chinese study you mentioned, uh one by Xu Wang and colleagues uh from 2024, I guess, they studied uh uh a big sample of students longitudinally using the Insomnia Severity Index.
• 13:01
  SPEAKER_01: And so we did cross-sectionally with Daniel Marcus and Claudio Ligori, with a big sample of Italian university students, so we got two very different samples as the basic, but we both found out uh four distinct profiles of insomnia using the same instrument, the seven-item Insomnia severity index, so full information without abbreviations in this case.
• 13:24
  SPEAKER_01: And we got that uh in particular one uh profile emerged, one characterized not by the severity of nighttime symptoms and impairment, but but just by or mostly by daytime impairment.
• 13:38
  SPEAKER_01: So people reporting that uh they got serious uh impairment in the daytime functioning due to not sleeping well, even though they don't have classical insomnia, so difficulty falling asleep, difficulty maintaining sleep, but they clearly attribute their daytime impairments to sleep.
• 13:55
  SPEAKER_01: And they are more compromised from the psychological point of view, from the physical point of view, compared to their peers not having insomnia problems, or even to those having mild insomnia problems, but just nighttime.
• 14:09
  SPEAKER_01: And this is quite interesting that's just a basic, we will need confirmation studies, but it is it is something, and this part is pertinent to the editorial on short and in its case, which we could obtain by keeping the full information, maybe adding further information about other constructs.
• 14:27
  SPEAKER_01: By using a reduced version of the Insomnia Severity Index or even just a screener with a couple of items, we could not distinguish these things, and by using total score, uh neither, because uh it was just having a number.
• 14:41
  SPEAKER_01: This is going deeper into the numbers, deeper into the items, assuming uh a network perspective possibly.
• 14:49
  SPEAKER_00: Yes, um, I understand this.
• 14:51
  SPEAKER_00: So, as I could understand, you used only insomnia severity index to distinguish these uh profiles.
• 14:57
  SPEAKER_00: You didn't use any other um questionnaires, scales, other clinical information, and so on.
• 15:05
  SPEAKER_00: Only this insomnia severity index.
• 15:08
  SPEAKER_01: Yeah, that's right.
• 15:09
  SPEAKER_01: The profiling was done uh via Latin class analysis by using just the seven items of the Insomnia Severity Index.
• 15:15
  SPEAKER_01: It was uh uh kind of a theory-grounded choice because we know Insomnia Severity Index is uh widely available.
• 15:24
  SPEAKER_01: Previous attempts were made with longer instruments, but we wanted to be direct to clinicians in our results.
• 15:31
  SPEAKER_01: So we use just the Insomnia Severity Index for profiling, then we use other instruments for comparing groups.
• 15:38
  SPEAKER_00: And then there is another question which is raised uh how do these profiles associate with treatment effects and uh management?
• 15:48
  SPEAKER_00: And this also relates to your third point, which you highlight in your in your paper.
• 15:54
  SPEAKER_00: So uh how why do shorter scales make it harder for us to understand how treatment works?
• 16:01
  SPEAKER_00: Uh and how does uh these shortened scales prevent us from choosing the correct treatment?
• 16:10
  SPEAKER_01: Yeah, I've been thinking a lot about this one, but simply put, it's because treatment, and may it be pharmacological or behavioral, uh, does not always affect all symptoms as a whole in the same way.
• 16:21
  SPEAKER_01: So if we only look at the total score from the insomnia severity index or other instruments, so from a full or an abbreviated scale, we may know that something improved, but not what improved indeed.
• 16:33
  SPEAKER_01: And with treatments like CBTI, which are made up of different components, different components may influence different symptoms indeed across time also.
• 16:42
  SPEAKER_01: So if we want to understand the mechanism of change, not just the effectiveness of the treatment, we often need more detailed symptom-level information, the full information of a scale or even of more scale.
• 16:54
  SPEAKER_01: So specifically, using brief scales like the insomnia severity index, two items or three items, we might miss whether a treatment is acting more on nighttime symptoms or on daytime symptoms or on worry, distress, sleep dissatisfaction, and so on.
• 17:12
  SPEAKER_01: So this is the same when we collect data longitudinally, if you think about it, over time, because we may also miss the sequence of change using network models.
• 17:22
  SPEAKER_01: Uh, recent studies show how particular components of CBTI are more effective on nighttime symptoms at the beginning of treatment and on daytime symptoms when treatment goes on.
• 17:33
  SPEAKER_01: So having this full information is useful for uh getting insight into mechanisms of treatments more than just its effectiveness, which is possibly where we should go in the future.
• 17:46
  SPEAKER_00: And do you think that it also means that having the whole seven-item severity score, we can, for example, planning the CBTI, we can skip some of the treatment procedures, let's say, uh, if we see that there is a very um uh clear profile of uh insomnia, like you told, with only the unal symptoms without night symptoms, or vice versa.
• 18:15
  SPEAKER_01: Yeah, for sure.
• 18:16
  SPEAKER_01: We we are possibly far from having so stable subtypes, so it's uh mostly an individual assessment.
• 18:23
  SPEAKER_01: But uh uh I believe that we can at least tailor the treatment progress based on uh information drawn from subjective assessment and subjective scales.
• 18:34
  SPEAKER_01: So even in the case you described when you got uh more pronounced daytime symptoms, maybe we can temper uh the use of different components of CBTI uh to uh to favor a more effective course of treatment over time and uh just try to target what is more uh more important for the people we are working with.
• 18:58
  SPEAKER_01: So if they report that daytime is uh the the highest burden, we can work with them on the CBTI strategies more targeting uh uh daytime uh fatigue or daytime sleepness, or what they feel daytime rather than uh focusing on education about uh how reduce deep latency, which may be not a problem for them, even though they got the same total score on the ICA as another people, another person uh having nighttime issues for the most.
• 19:30
  SPEAKER_01: So, yeah, it may be a guideline at least.
• 19:42
  SPEAKER_00: About um uh significance for clinical practice, which you raise in your paper.
• 19:48
  SPEAKER_00: So, why is it important for questionnaire items to remain clinically meaningful and not just statistically efficient?
• 19:56
  SPEAKER_01: Yeah, this is uh one particularly important to me as uh I'm a clinical psychologist by background, and I was drawn to research later in my life.
• 20:05
  SPEAKER_01: So, my starting point and the point to which I wanted to go with this uh paper with Danielle was indeed clinical practice.
• 20:13
  SPEAKER_01: And in clinical work, uh we cannot more treat an item just uh as a number.
• 20:19
  SPEAKER_01: It is possibly a clue, it is an information about a person or about a group of people, and it tells us something about how a person is experiencing insomnia or is experiencing sleep-related symptoms uh in general.
• 20:32
  SPEAKER_01: So it can also be used interactively within the sleep medicine interview to guide the course of the interview, to help case conceptualization, to suggest, as we said before, where the treatment should focus more and where less.
• 20:46
  SPEAKER_01: This information gathering indeed is a process that has potential to inform the clinical encounter.
• 20:52
  SPEAKER_01: So uh if we reduce scales only for statistical neatness, we risk losing this practical value and this practical intuitions we may get from using a standardized set of items, which is also a set of questions of uh interactive communications, indeed.
• 21:11
  SPEAKER_00: Yeah.
• 21:12
  SPEAKER_00: So trying to sum it up is the main message that shorter scales are inherently problematic, or is it more about using them carefully and in the right context?
• 21:25
  SPEAKER_01: No, you're right to try to focus this because uh the main message is not uh do not shorten scales, uh, it's a scene and you should not.
• 21:32
  SPEAKER_01: It's more like uh uh shorten scales but carefully.
• 21:38
  SPEAKER_01: So um make this only after asking what you need the scale for, what you want to measure, what you want to assess, because the right instrument depends on the purpose, of the purpose of research, on the purpose of clinical practice.
• 21:51
  SPEAKER_01: Uh but if the purpose is very broad screening, for example, as we said before, a short scale may be enough.
• 21:57
  SPEAKER_01: It's not bad just because it's short.
• 21:59
  SPEAKER_01: So So it's focusing on the purposes and the construct one is interested in.
• 22:04
  SPEAKER_00: And when then should we avoid these short and ultra-short scales?
• 22:11
  SPEAKER_01: Yeah, well, uh, as we we said somewhat before, uh, I would be more cautious when we are doing uh individual clinical practice and individual clinical assessment, even though sometimes we are involved into protocols requiring us to use certain instruments uh and not other instruments.
• 22:30
  SPEAKER_01: But if we want to use instruments in a clinical way uh to be helpful for case conceptualization, maybe trying to rely on the full version may be more informative.
• 22:42
  SPEAKER_01: This is also the case when we want to do uh mechanism-based research, as we said, for network approaches.
• 22:48
  SPEAKER_01: So dropping uh information may be not useful if we want to see how different aspects of a condition interact.
• 22:55
  SPEAKER_01: So uh both research and clinical contexts uh may require full instruments.
• 23:01
  SPEAKER_01: But uh one thing I stretch when I talk with sleep clinicians working with me is that uh for the most in sleep medicine practice, we already have brief scales because the insomnia severity index is seven items after all.
• 23:16
  SPEAKER_01: We can abbreviate it further for uh large scale studies, but it's seven items.
• 23:21
  SPEAKER_01: Even the Pittsburgh Leaf Quality Index is 19 items.
• 23:24
  SPEAKER_01: Being trained as a clinical psychologist in personality assessment, we got like hundreds of items for questionnaires, and there it's really messy.
• 23:33
  SPEAKER_01: But uh in sleep medicine we can afford to maybe using uh full short scales, as long as they are already short.
• 23:41
  SPEAKER_00: Yes, you are right.
• 23:42
  SPEAKER_00: And I also have a question about the cumulative analysis and, for example, comparing different studies, because if we go through using different tools in research, it would be more difficult for us to compare the results of these resources and to perform like meta-analytic um uh analysis, meta-analysis and cumulative uh analysis.
• 24:05
  SPEAKER_00: What do you think about this?
• 24:07
  SPEAKER_01: No, that's a great point to think about because uh luckily we rely uh more and more on uh aggregated data.
• 24:16
  SPEAKER_01: We got uh a bunch of aggregated data in sleep research, and most of them are based on uh on total scores, we may say.
• 24:24
  SPEAKER_01: So uh it's never about dropping the total score.
• 24:29
  SPEAKER_01: Uh so it's also never about dropping the full item when we got uh uh a great amount of data already collected with the full item.
• 24:37
  SPEAKER_01: But my hope is that by working also on single items, we may accumulate uh so much single item data that we may do meta-analysis even on single item information or even on uh time information.
• 24:52
  SPEAKER_01: So we can do uh aggregated studies not just on uh insomnia severity as a broad construct, but also on this specific symptom of insomnia.
• 25:03
  SPEAKER_00: Yeah.
• 25:04
  SPEAKER_00: Okay, so there is a lot to think for researchers uh before trying to shorten a scale.
• 25:09
  SPEAKER_00: So if you could give one piece of advice for researchers who are going to develop new slip questionnaire today, what would it be?
• 25:18
  SPEAKER_01: Ah yeah, that's uh a hard, a tough one, but uh I would say uh do not ask how short can this scale become.
• 25:26
  SPEAKER_01: Just not long for shortness.
• 25:29
  SPEAKER_01: Ask also what information do we want to get?
• 25:32
  SPEAKER_01: What is the construct that I want to measure?
• 25:34
  SPEAKER_01: In other words, it's beginning from the construct and from the intended use, not only from uh statistical economy, because ultimately ultimately the instrument we use has an impact on the result we observed in the clinic and in the research study.
• 25:51
  SPEAKER_01: So, in my view, this is where uh sleep medicine and psychometrics, as two distinct disciplines, can set up a dense dialogue.
• 26:00
  SPEAKER_01: On the one hand, item selection should rely on a solid clinical ground.
• 26:05
  SPEAKER_01: On the other hand, having a strong methodological framework is the means for translating these intuitions into robust measurement tools which can be used consistently.
• 26:16
  SPEAKER_00: Yeah.
• 26:17
  SPEAKER_00: Thank you.
• 26:17
  SPEAKER_00: This was a great advice, and it was really a great insightful discussion.
• 26:23
  SPEAKER_00: It is a great reminder for us that even something as simple as a questionnaire involves deep methodological and clinical decisions.
• 26:30
  SPEAKER_00: Matteo, thank you very much for joining us and for sharing with us your experience, your thoughts, your insights, and for helping us to unpack the science behind something that seems so simple, like a questionnaire, but is actually quite complex.
• 26:46
  SPEAKER_00: And thanks to all our listeners for tuning in.
• 26:49
  SPEAKER_00: Have a good day.
• 26:51
  SPEAKER_00: Thanks for joining us.
• 26:53
  SPEAKER_01: Thank you again, Gudilla.