DCA for de-escalation models

[AAbou-foul]

@shaunporwal @VickersA , I need your advice, please.

I am working on developing models for cancer treatment de-escalation. For example, a model predicting recurrence of cancer at 3 years post-treatment (prevalence around 10%). If a model predicts a probability of recurrence for a patient (let's say >0.05), that will be considered high risk, and they will continue on standard-of-care management (i.e., all scheduled follow-ups and scans as normal). If the predicted probability is ≤0.05, then we would consider that low-risk and de-escalate (e.g., discharge to the community earlier or do less than standard-of-care follow-up).

The classical net benefit calculation is not very clinically relevant in this scenario because we are not overly concerned about false positives—patients in the false positive category are already receiving the standard of care, so there is no material harm or change to them. In de-escalation strategies, the primary focus is on true negatives and false negatives—how many recurrences have we missed by following the de-escalation strategy, and how many patients have we correctly de-escalated without unnecessary follow-up?

I reviewed the material and papers you kindly published, but I find that examples like this are rarely discussed. Even the formula variant in the supplementary material, where the net proportion of true negatives equals (NB − NB_TreatAll) / odds(T), incorporates false positives, which don't carry much weight in de-escalation strategies like this.

Is there a way to adapt the DCA net benefit for use in de-escalation models? Specifically, we want the net benefit to reflect how many patients were correctly de-escalated (benefit) versus incorrectly de-escalated (weighted harms for example). If such an adaptation exists, can this be implemented and plotted using your Python package?

Thanks a lot! 😊

[VickersA]

This is a very standard decision curve analysis problem. Indeed, it is comparable to the standard didactic example of a model to predict prostate cancer on biopsy where you have a group of men who are getting biopsied and trying to find those who are at low risk and should be exempted from biopsy (i.e. you are de-escalating care). False positive are, of course, extremely relevant in de-escalating care, because they constitute a patient whose care could have be de-escalated but wasn't. The only tweak is you might want to show results in terms of interventions avoided rather than the standard decision curve. Have a look for the section on "Interventions Avoided" in https://mskcc-epi-bio.github.io/decisioncurveanalysis/dca-tutorial.html#Multivariable_Decision_Curve_Analysis

[AAbou-foul]

Thanks a lot Dr @VickersA, this is really informative.

I checked the 'Interventions avoided section' and it really fits perfectly our need and what we were looking for in DCA.

One follow-up question please:

When we plot [graph_type='net_intervention_avoided'] using the Python library, we get quite unusual curve for our model, that extends beyond 1.0 (100% reduction) at the Y axis! Please see plot 1 and the 'zoomed out' plot 2 attached here.
Is that expected? how would we report our NRI at 0.05 threshold when it's >100%??

Thanks a lot :)

[VickersA]

You are right to say that you can't have >100% net reduction in interventions. But to diagnose what is going wrong here, you'd have to provide more information. At the very least, show the standard decision curve and explain what is different between the two graphs above.

Also, what do you mean by "NRI"? NRI is an invalid statistical technique that has nothing to do with net benefit! https://en.wikipedia.org/wiki/Net_reclassification_improvement

[AAbou-foul]

Thanks a lot Dr Vickers.

I double checked the numbers and I see there was a slight error, the net reduction in interventions is in deed bound by 0 and 1 (0-100%). We got the correct plots which is great.

Sorry also for using undefined acronym. I meant net reduction in interventions by NRI

Thanks again for your prompt replies. Have a lovely Christmas and happy new year!

[VickersA]

ok great, but don't use NRI as an acronym! It means something else entirely.