Summary table of true positives and false positives

[LR1933]

Hello Professor,
Thank you for your excellent work. I really like graphical representations, but some people prefer numerical tables, possibly because understanding graphics takes time. I am trying to summarize some numbers like the true positive number under certain thresholds in a table. My statistical foundation is not very strong, so I would appreciate it if you could confirm whether my understanding is correct.

I am based on your survival model example and outputting the interventions at a 20% threshold for all interventions and for those predicted by the model.

This is the output for all interventions at a 20% threshold.

x <- dca(Surv(ttcancer, cancer) ~ pr_failure18,
  data = df_time_to_cancer_dx,
  time = 1.5,
  thresholds = seq(0, 0.5, 0.01),
  label = list(pr_failure18 = "Prediction Model")
)%>%as_tibble()
x[3,]
# A tibble: 1 × 9
variable label         n threshold pos_rate tp_rate fp_rate  harm net_benefit
<chr>    <fct>     <int>     <dbl>    <dbl>   <dbl>   <dbl> <dbl>       <dbl>
1 all      Treat All   750      0.02    0.220   0.220   0.780     0       0.204

Assume the total sample size is one thousand people. I created the following 2x2 table. Since an intervention is applied to everyone, all individuals are considered positive. The 20% threshold here is not the same as the positivity threshold (100%). ←please let me know if my understanding here is incorrect.

This is the output for interventions based on a model at a 20% threshold.

+ x[105,]
# A tibble: 1 × 9
  variable     label                n threshold pos_rate tp_rate fp_rate  harm net_benefit
  <chr>        <fct>            <int>     <dbl>    <dbl>   <dbl>   <dbl> <dbl>       <dbl>
1 pr_failure18 Prediction Model   750      0.02    0.220   0.217   0.664     0       0.204

When using the model to predict and distinguish positives, the threshold for predicting positive cases and the threshold for calculating net benefit are both set to 20% （←please let me know if my understanding here is incorrect）, and a complete 2x2 table can be generated based on this threshold."

Are these tables made correctly? My idea is to collect the number of true positives and false positives at different thresholds predicted by the model and calculate the net benefit. This will clearly show the number of people with net benefit. It is also convenient for describing the results in the paper.

If you have other better suggestions, please let me know. Thank you very much.

[shaunporwal]

Hi @lijiaqi-github, apologies for the tardy response, and thank you for your question.

In the survival example, for both the Treat All and pr_failure18 categories, you have selected a 2%, not a 20%, threshold.

The other thing is that we have guiding lines in a DCA plot for 2 treatment modalities, one in which you elect to treat nobody regardless of the threshold selected (Treat None), and one in which you elect to treat everyone (Treat All). Obviously, either of these are extremes, and the hope in using models/factors such as age, sex, family history, etc. is to avoid either under- or over-treatment based on the amount of risk patients are willing to take to unearth the truth of their condition (outcomes such as cancer).

The threshold value in this case is indeed used both to establish correctly predicted outcomes as well as to calculate the net benefit, whose formula is as follows:

Net Benefit = (TP Rate) - (Threshold-Weighted FP Rate)

For more in-depth information, please refer to our peer-reviewed literature section and/or software tutorial on decisioncurveanalysis.org.

Hope this helps!

[LR1933]

Thank you for your thoughtful reply. My mistake — I did indeed write 20% instead of 2%.

I have another question. I tried to look into the source code of DCA to understand how metrics like the positive rate are calculated. Since my knowledge of statistics is limited, I try to learn more about survival model computations whenever I get the chance.

As far as I understand, the positive rate (pos_rate) for a given year is calculated using the null model Surv(ttcancer, cancer) ~ 1.

However, the code for true positives (tp_rate) and false positives (fp_rate) seems a bit complicated — it’s nested within various functions, and I haven’t figured it out yet.

Would you mind helping me sort out how the true and false positive rates are computed in the code? Of course, I understand this is a bit much to ask and could take a lot of your time. But if you happen to have time, I’d really appreciate any hints or guidance.

Thank you very much!

[VickersA]

I don't think it is appropriate to understand the underlying mathematical basis of a statistical procedure from the code and would recommend you look to our website to find some introductory papers that explain the math.