Measuring medical metrics
 

Questions

1. Would you pay $1000/month for a drug that guaranteed an increase in your UPDRS score of 4.7 points?

2. Would you undergo brain surgery for an average increase in your six year survival rate of 10%?

3. Would you exercise a hour per day if it led to an average 23.9% reduction in your levodopa equivalent daily dose (LEDD) ?

4. Would you take this supplement if it increased your daily "on" time by 21 minutes?

5. Would you choose transcranial magnetic stimulation if it had a significant effect on your walking ability?

6. Would you move house if on average it decreased your side to side tap score by 8.8%?

Measuring metrics

I'm a great believer in the benefits of taking measurements. Good measurements can lead to improvement in the performance of almost any system, be it a company, a sports team or, as it applies here, the quality of life of PwP.

Without metrics you are unsure whether an action, a new drug for instance, has led to an improvement.

So, my basic instinct is that measurement is good ... except that is when it's bad.

Metrics for PwP should convey meaning to PwP.

Metrics for PwP should be comparable, one with another.

Metrics for PwP should be congruent with goals.

Get the metric wrong and billions of dollars of research grants, pharmaceutical costs and surgical fees gets sub-optimally spent.

What metrics would you choose to best represent your PD?

Answers

1. No. A trick question. Low UPDRS scores are good. So paying to have them increased is crazy. Even if it had been the other way around, a decrease rather than an increase, what does 4.7 points mean? 4.7 out of what? Totalling the scores from parts 1 to 4 of the UPDRS gives a score ranging from 0, no disability, to 199, total disability [1]. But are all points worth the same (is there linearity)?

2. Don't know. Everything else being equal an increase in survival rate is good. But, perhaps, it's not equal. For instance, if a side effect is more falls. Also be careful of the 10% increase. Taken literally this would take someone's survival rate from, say, 50 to 55%. But, it's easy to mistakenly read the 10% as a 10 percentage point increase, taking 50 to 60%.

3. Don't know. The implication, but only that, is that your condition has improved to the extent that a lower LEDD is required for the same quality of life. That's good, but what PwP are looking for is a higher QOL? There is also the suggestion, but no more than that, that the rate of progression is slowed.

4. Don't know. At least for me, not all "on" time is equal. The intensity of the transition from "on" to "off" is particularly important.

5. Don't know. Here the key word is "significant". It can mean significant by magnitude, in the sense that I can now walk 10 miles in a day whereas previously I could only manage 1 mile. It can also mean statistically significant in the sense that the cohort's average went from 1 to 1.1 mile per day and that was unlikely to have occurred by chance. What we are looking for is significant significance.

6. No. A trick question. Low tap scores are bad. Something that made them worse is a bad idea.

Reference

[1] http://www.epda.eu.com/en/parkinsons...-scales/updrs/

John

I would value a discussion of the usefulness of the MDS-UPDRS (Movement Disorder Society Unified Parkinson's Disease Rating Scale) [1] in measuring the efficacy of a new treatment for PD, especially one for which it is claimed to slow or even to reverse the progression of the disease.

The major claim for the exenatide trial [2] is that the difference between the treated group and the placebo group's MDS-UPDRS Part III score, while small, is statistically significant. (After 60 weeks, off medication scores were 1 point better for the exenatide group, compared to a worsening of 2.1 points for the placebo group). Most other measures, while in favour of exenatide, do not show significance.

The MDS-UPDRS Part III contains 18 questions. Some of which are repeated, for instance, for the left and right side, giving 33 scores in total. Each question is scored 0, 1, 2, 3, 4, where 0 represents normal (no symptoms) and 4 represents severe symptoms. A typical question is 3.17, rest tremor amplitude, where the score of 0 represents no tremor; 1, less than 1 cm maximal amplitude (MA); 2, 1cm<MA<3cm; 3, 3<=MA<=10cm; 4, MA>10. Four scores are given are given, one for each limb.

The individual scores are added together to get a MDS-UPDRS Part III total. Given the way the scoring is done, a low score is good, a high score is bad.

Part III does not measure the underlying disease (perhaps the number of dopaminergic neurons in the substantia nigra, though there are probably more deeply rooted measures). Neither does Part III directly measure the impact of PD on the quality of life of the patient.

The MDS-UPDRS Part III score is a proxy measure. It doesn't measure what we are really interested in (for instance, the quality of life, QOL), but it does usually point in the right direction: as the score decreases the QOL tends to improve.

The billion dollar question is:

Will MDS-UPDRS III give the right answers in discriminating between candidate treatments in a world where the differences in efficacy are small?

References:

[1] http://www.movementdisorders.org/MDS...S7308final.pdf

[2] "Exenatide once weekly versus placebo in Parkinson's Disease: a randomised, double-blind, placebo-controlled trial"
Thomas Foltynie et al.
The Lancet, August 2017.

John

Although behind a pay wall, I came across a paper [1] which shows correlations between various PD measures. MDS-UPDRS Part III scores and PDQ-8 [2], a PD quality of life questionnaire, have a correlation of 0.47. If you take the square of the correlation, you get the proportion of the variance that can be attributed to a factor. Here we have 0.47 x 0.47 = 0.22. So, less than a quarter of the variance is explained by the Part III score. In fact the situation is probably worse than that: the PDQ-8 is itself a proxy measure of the true progression of the disease.

Reference:

[1] Eur J Neurol. 2014 Mar;21(3):519-24. doi: 10.1111/ene.12349. Epub 2014 Jan 21.
"Relationship between the MDS-UPDRS domains and the health-related quality of life of Parkinson's disease patients."
Martínez-Martín P1, Rodríguez-Blázquez C, Forjaz MJ, Alvarez-Sánchez M, Arakaki T, Bergareche-Yarza A, Chade A, Garretto N, Gershanik O, Kurtis MM, Martínez-Castrillo JC, Mendoza-Rodríguez A, Moore HP, Rodríguez-Violante M, Singer C, Tilley BC, Huang J, Stebbins GT, Goetz CG.
Relationship between the MDS-UPDRS domains and the health-related quality of life of Parkinson's disease patients. - PubMed - NCBI

[2] Parkinson’s Disease Questionnaire

John

John, I have a copy of this article (and some others) that I can email to you when I get a chance (I'm traveling this week). It's a great point that you make and I fully agree. I think most PD researchers are aware of the limitation of the UPDRS. The need for better biomarker measurements is very evident

We can all agree with the need for better biomarkers. But what do we do in the meantime?

Consider the difficulties of trying to assess the efficacy of a drug which has true figures, but unknown to us (that's what we're trying to find out):
- for half of PwP, reduces progression on average by 10% (that is a progression rate of 10% per year would be reduced to 9% per year)
- for half of PwP has on average no effect.

The testing of drugs to slow or even reverse the progression of PD is harder than those for symptomatic control. There are smaller day to day changes, leading to longer test periods being required to see a meaningful difference. The longer time required for testing leads to the ethical issue of how can we demand patient compliance, possibly for years, without any obvious benefit to themselves.

There are two types of danger. One, where the use of a poor metric leads to a drug with no benefit being approved. The other, where the use of a poor metric results in the opposite effect: some truly efficacious therapies being wrongly labelled as failures.

There is the issue of the cost of the test. Tests that require patients to travel to a test centre, and tests that need to be administered by medics are costly, leading to fewer data points. The balance needs to be struck between cheap/less valid tests, probably internet based, and expensive/more valid tests. Also, the high cost of testing will lead to some drugs never being developed and, for those that do come to market, being exceedingly expensive.

John