|
| 1 | +--- |
| 2 | +title: 'predictNMB: An R package to estimate if or when a clinical prediction model is worthwhile' |
| 3 | +tags: |
| 4 | + - R package |
| 5 | + - clinical prediction model |
| 6 | + - net monetary benefit |
| 7 | + - cutpoint |
| 8 | + - clinical decision making |
| 9 | +authors: |
| 10 | + - name: Rex Parsons |
| 11 | + orcid: 0000-0002-6053-8174 |
| 12 | + affiliation: 1 |
| 13 | + - name: Robin D. Blythe |
| 14 | + orcid: 0000-0002-3643-4332 |
| 15 | + affiliation: 1 |
| 16 | + - name: Adrian G. Barnett |
| 17 | + orcid: 0000-0001-6339-0374 |
| 18 | + affiliation: 1 |
| 19 | + - name: Susanna M. Cramb |
| 20 | + orcid: 0000-0001-9041-9531 |
| 21 | + affiliation: "1, 2" |
| 22 | + - name: Steven M. McPhail |
| 23 | + orcid: 0000-0002-1463-662X |
| 24 | + affiliation: "1, 3" |
| 25 | +affiliations: |
| 26 | + - name: Australian Centre for Health Services Innovation and Centre for |
| 27 | + Healthcare Transformation, |
| 28 | + School of Public Health and Social Work, |
| 29 | + Faculty of Health, |
| 30 | + Queensland University of Technology, |
| 31 | + Kelvin Grove, |
| 32 | + Australia |
| 33 | + index: 1 |
| 34 | + - name: Jamieson Trauma Institute, |
| 35 | + Royal Brisbane and Women’s Hospital, |
| 36 | + Metro North Health, |
| 37 | + Herston, |
| 38 | + Australia |
| 39 | + index: 2 |
| 40 | + - name: Clinical Informatics Directorate, |
| 41 | + Metro South Health, |
| 42 | + Woolloongabba, |
| 43 | + Australia |
| 44 | + index: 3 |
| 45 | +date: 31 March 2023 |
| 46 | +bibliography: paper.bib |
| 47 | +--- |
| 48 | + |
| 49 | +# Summary |
| 50 | + |
| 51 | +Clinical prediction models are frequently developed for identifying patients at |
| 52 | +risk of adverse health events, and possibly guiding the use of treatment, but |
| 53 | +are often not validated or implemented in clinical practice |
| 54 | +[@hendriksen2013diagnostic; @steyerberg2013prognosis]. This could be due to |
| 55 | +several factors including poor performance or the lack of an effective |
| 56 | +intervention that can be implemented in response to prediction of high risk. |
| 57 | +The ``predictNMB`` R package performs simulations to evaluate the use of |
| 58 | +hypothetical clinical prediction models (with a binary outcome) to help inform |
| 59 | +development and implementation decisions, and estimate potential impacts in |
| 60 | +terms of costs and health outcomes. This package allows the user the flexibility |
| 61 | +to adjust simulation inputs regarding the prediction model’s performance, its |
| 62 | +target population, the costs of the event being predicted, and the effectiveness |
| 63 | +of interventions that the model is being used to recommend. More details about |
| 64 | +the package, including guides and a detailed example are available on the |
| 65 | +[package site](https://docs.ropensci.org/predictNMB/). |
| 66 | + |
| 67 | +# Statement of need |
| 68 | + |
| 69 | +Clinical decision support systems are often used to classify patients into |
| 70 | +high- or low-risk groups and to recommend treatment assignment |
| 71 | +[@steyerberg2013prognosis]. These systems can only perform as well as the |
| 72 | +underlying model(s) informing decision support recommendations, the treatments |
| 73 | +being recommended, and the implementation of the system within clinical |
| 74 | +settings. Often, the cost-effectiveness of these systems is not known until |
| 75 | +they are developed, implemented, and evaluated [@reilly2006translating; @10.1093/jamia/ocad040]. |
| 76 | +The ``predictNMB`` R package aims to avoid this delay by facilitating early |
| 77 | +estimation of the cost-effectiveness of these systems. We expect most users to |
| 78 | +be either: 1) those involved in health service decision making regarding |
| 79 | +investment in development or implementation of clinical decision support |
| 80 | +systems, or 2) clinical prediction model developers, who may be deciding whether |
| 81 | +to invest efforts into clinical prediction model development or validation. |
| 82 | +Characteristics of the user's given patient population are incorporated using |
| 83 | +Monte Carlo simulation to estimate the expected cost-effectiveness of a given |
| 84 | +system (under an assumption of ideal implementation and complete adherence to |
| 85 | +recommendations) to provide guidance on cost-effectiveness before prediction |
| 86 | +models are developed or implemented. For example, by evaluating this simulated |
| 87 | +decision support system and finding that a clinical prediction model would only |
| 88 | +be effective (better than a treat-all or treat-none approach) at an |
| 89 | +unrealistically high level of model performance, users would then have |
| 90 | +opportunity to reduce research waste by avoiding model development, |
| 91 | +implementation, and evaluation in a clinical setting. Similar to a statistical |
| 92 | +power analysis, ``predictNMB`` allows users to estimate how well their model |
| 93 | +would need to perform, and its expected benefit, if implemented to offer a |
| 94 | +treatment recommendation. It may be found that a given decision support system |
| 95 | +may only be likely to improve care when the available treatment is of a certain |
| 96 | +level of effectiveness or when the prevalence of the condition is relatively |
| 97 | +low or high, and this may better guide the user regarding which treatment the |
| 98 | +system should be recommending, or for which patients. |
| 99 | + |
| 100 | +# Features |
| 101 | + |
| 102 | +``predictNMB`` simulates well-calibrated prediction models using logistic |
| 103 | +regression and incorporates a range of inbuilt cutpoint selection methods, |
| 104 | +including a treat-all (cutpoint=0) and treat-none (cutpoint=1) method, and two |
| 105 | +methods that aim to maximize the Net Monetary Benefit (NMB): 'cost-minimizing' |
| 106 | +[@wynants2019three] and 'value-optimizing'[@parsons2023cutpoints]. It also |
| 107 | +allows the user to specify any other function for cutpoint selection. Evaluation |
| 108 | +of the models in terms of the NMB requires the user to pass information |
| 109 | +regarding the costs associated with each of four possible classifications. |
| 110 | +A helper function is provided to make this process easier by taking arguments |
| 111 | +in terms of treatment effectiveness and outcome costs, along with their |
| 112 | +uncertainty (see creating [NMB functions vignette](https://docs.ropensci.org/predictNMB/articles/creating-nmb-functions.html) |
| 113 | +for more details). |
| 114 | + |
| 115 | +The simulations are stored as one of two types of objects, depending on whether |
| 116 | +a single scenario was used for simulation or if a range of values were screened |
| 117 | +over several simulation scenarios. Plotting and summarizing methods for these |
| 118 | +objects are exported to easily visualize and evaluate the results of the |
| 119 | +simulation study (see [summarising results vignette](https://docs.ropensci.org/predictNMB/articles/summarising-results-with-predictNMB.html) |
| 120 | +for more details). |
| 121 | + |
| 122 | +A detailed example of a pressure injury model using inputs from the literature |
| 123 | +is included as a [vignette](https://docs.ropensci.org/predictNMB/articles/detailed-example.html). |
| 124 | +Applying ``predictNMB`` for this use case indicates that, when using realistic |
| 125 | +values for the intervention and prevalence of pressure injuries and their costs, |
| 126 | +the clinical prediction model may be useful when the model is particularly |
| 127 | +well-performing (area under the receiver operator characteristic curve > 0.8) |
| 128 | +and when the event rate for pressure injuries is lower (event rate of 0.05). |
| 129 | +When the event rate was higher, the treat-all strategy was preferred to any of |
| 130 | +the model-guided approaches or treating none. This suggests that model |
| 131 | +development and implementation efforts should target patient populations where |
| 132 | +the event rate of pressure injuries is lower than 0.05. |
| 133 | + |
| 134 | +# Acknowledgements |
| 135 | + |
| 136 | +This work was supported by the Digital Health Cooperative Research Centre |
| 137 | +(“DHCRC”). DHCRC is funded under the Commonwealth’s Cooperative Research Centres |
| 138 | +(CRC) Program. SMM and SMC are supported by NHMRC-administered fellowships |
| 139 | +(#1181138 and #2008313, respectively). |
| 140 | + |
| 141 | +# References |
0 commit comments