Wi score

[EmilHvitfeldt]

No description provided.

[EmilHvitfeldt]

Hello @dajmcdon 👋

If you are able to provide me with a working example of inputs and output, then I can set up the basic scaffolding for a metric

[dajmcdon]

@EmilHvitfeldt This is what should happen.

My last commit removed the portions of this PR that are now in {hardhat} (dev). I think it just needs the correct method(s) for weighted_interval_score().

The below are the Examples for that function (but using a different backing), so this won't run as written.

[I used library(epipredict) in place of library(yardstick), but everything else is the same.]

Is this what you had in mind?

library(yardstick)
library(hardhat)

quantile_levels <- c(.2, .4, .6, .8)
predq1 <- 1:4 #
predq2 <- 8:11
dstn <- quantile_pred(rbind(predq1, predq2), quantile_levels)
actual <- c(3.3, 7.1)
weighted_interval_score(dstn, actual)
#> [1] 0.65 1.90
weighted_interval_score(dstn, actual, c(.25, .5, .75))
#> [1] 0.6833333 1.9833333

# Missing value behaviours
dstn <- quantile_pred(matrix(c(1, 2, NA, 4), nrow = 1), 1:4 / 5)
weighted_interval_score(dstn, 2.5)
#> [1] 0.5
weighted_interval_score(dstn, 2.5, 1:9 / 10)
#> [1] 0.455656
weighted_interval_score(dstn, 2.5, 1:9 / 10, na_handling = "drop")
#> [1] 0.462613
weighted_interval_score(dstn, 2.5, na_handling = "propagate")
#> [1] NA
weighted_interval_score(
  quantile_pred(matrix(1:4, nrow = 1), 1:4 / 5),
  actual = 2.5,
  quantile_levels = 1:9 / 10,
  na_handling = "fail"
)
#> [1] NA

^{Created on 2025-12-03 with reprex v2.1.1}

[EmilHvitfeldt]

yes that is perfect! I'll set up the scaffolding so it runs correctly, then I'll ping you for documentation and correctness testing

[dajmcdon]

I just realized some of the functionality above does not match the arguments. Let me fiddle with things a bit.

[EmilHvitfeldt]

no problem! let me know when you are ready for me to take the reigns

[dajmcdon]

OK. Should be good to go now. Here's a revised version of the examples (using *_vec()), and an actual reprex.

library(yardstick)
library(hardhat)

quantile_levels <- c(.2, .4, .6, .8)
pred1 <- 1:4
pred2 <- 8:11
preds <- quantile_pred(rbind(pred1, pred2), quantile_levels)
truth <- c(3.3, 7.1)
weighted_interval_score_vec(truth, preds)
#> [1] 1.275
weighted_interval_score_vec(truth, preds, quantile_levels = c(.25, .5, .75))
#> [1] 1.333333

# Missing value behaviours
preds_na <- quantile_pred(rbind(pred1, c(1, 2, NA, 4)), 1:4 / 5)
truth <- c(2.5, 2.5)
weighted_interval_score_vec(truth, preds_na)
#> [1] 0.5
weighted_interval_score_vec(truth, preds_na, quantile_levels = 1:9 / 10)
#> [1] 0.455656
weighted_interval_score_vec( # should error
  truth,
  preds_na,
  quantile_levels = 1:9 / 10,
  quantile_estimate_nas = "drop"
)
#> Error in `weighted_interval_score_vec()`:
#> ! When `quantile_levels` is not a subset of those available in
#>   `estimate`, `quantile_estimate_nas` may not be `'drop'`.
weighted_interval_score_vec(
  truth,
  preds_na,
  quantile_levels = c(2, 3) / 5,
  quantile_estimate_nas = "drop"
)
#> [1] 0.4
weighted_interval_score_vec(
  truth, preds_na, na_rm = TRUE, quantile_estimate_nas = "propagate"
)
#> [1] 0.5
weighted_interval_score_vec(
  truth, preds_na, na_rm = FALSE, quantile_estimate_nas = "propagate"
)
#> [1] NA

^{Created on 2025-12-03 with reprex v2.1.1}