Rethinking EDSS assessments in Multiple Sclerosis (MS): Can technology solve rater training and variability challenges in MS?
In Multiple Sclerosis(MS) the Expanded Disability Status Scale (EDSS) remains the gold standard for measuring disability in MS patients, despite its complexity and subjectivity. With researchers grappling extensive rater training requirements and variability issues that could be impacting their data quality, could emerging technologies like AI offer a path to greater precision and faster site activation?
Inter-rater variability in EDSS assessments in MS clinical trials
MS clinical trials hinge on tracking relapses and disability—most commonly through the EDSS assessments, which are both subjective and highly complex, with one of the key issues is inter-rater variability. In 2021, a study by Cohen et al set out to evaluate this effect amongst junior neurologists (JN) and MS neurologists (MSN), who examined a group of 103 patients on the same day. Perfect agreement between the JN and MSN’s scores was met for 67% of patients, while “disagreement that could lead to a significant difference in terms of level of disability” occurred for 17% of patients.i The study also evaluated intra-rater reliability, discovering 38 discrepancies amongst the ratings of individual JNs and 14 for MSNs.
Despite significant emphasis on rigorous rater training which can take months to complete, and the need for it to be reinforced across a rater’s career to combat the possibility of intra-rater drift, a single rater’s scoring may still change over time. We still observe rating discrepancies, mostly coming from the complexities of the EDSS score calculations – particularly scores between 0 to 4.i
Why it is time to move to electronic EDSS
Electronic EDSS (eEDSS) provides an “algorithm-based consistency check”, detecting combinations of scores which stray from the official rules of Neurostatus (the current official licensee).ii Real-time feedback is shared, enabling the rater to change their score.
A study by D’Souza et al evaluated this tool across two multicenter Phase III trials. Out of more than 10,000 eEDSS assessments performed across the studies, 40.1% had inconsistencies.iii Following the automated feedback, this was reduced to 22.1%. In total, this checking process resulted in 14.8% of the overall EDSS scores being changed, supporting the importance of programmed edit checks in increasing the reliability of EDSS scores.
In another analysis involving more than 41,800 eEDSS assessments across 13 trials, a total of 14% of submissions required expert review, with 11% requiring more than one review, and the EDSS score/step was changed by the rater in 31% of these cases.iv . Both studies underscore the critical importance of transitioning from paper-based scoring to electronic EDSS to capture inaccuracies in MS trial data. But is there potential for future technologies to take this another step further?
A future for AI in EDSS Assessment in Multiple Sclerosis?
AI could potentially be deployed to identify out-of-range scores or scores that are inconsistent with prior scoring of other areas – similar to current practice, only faster, more accurately, and with real-time feedback that explains each error, helping to support more reliable assessments in the future. Looking to the future, EDSS platforms could even use AI to analyze video or audio recordings of patients and check assessments, flagging any scoring discrepancies. While this could be useful for assessing ambulation – the primary measure of disability in the EDSS – AI’s ability to evaluate other functional systems such as sensory or pyramidal systems is less clear, and further research and investigation is required.
The increasing advancement of technology alongside the complexity of neuroscience clinical trials, represents a major opportunity to enhance and streamline your research. Download our new whitepaper, to understand what this opportunity could mean across your research pipeline
Let's talk about how Fortrea can help you navigate this evolving landscape.
References:
- i Cohen M, Bresch S, Thommel Rocchi O, Morain E, Benoit J, Levraut M, Fakir S, Landes C, Lebrun-Frénay C. Should we still only rely on EDSS to evaluate disability in multiple sclerosis patients? A study of inter and intra rater reliability. Mult Scler Relat Disord. 2021 Sep;54:103144. doi: 10.1016/j.msard.2021.103144. Epub 2021 Jul 9. PMID: 34274736.
- ii https://neurostatus-uhb.com/what-is-neurostatus-eedss/
- iii D’Souza M, Heikkilä A, Lorscheider J, et al. Electronic Neurostatus-EDSS increases the quality of expanded disability status scale assessments: Experience from two phase 3 clinical trials. Multiple Sclerosis Journal. 2019;26(8):993-996. doi:10.1177/1352458519845108
- iv Cerdá Fuertes N, Khurana L, Tressel Gary S, Fricker E, McDowell B, Kappos L, D’Souza M. Neurostatus-eEDSS results in high consistency of Expanded Disability Status Scale assessments: Experience from 13 clinical trials. View PDF