How to evaluate a "good" forecast

[seabbs]

Write up in the draft/outline is based on the work done for Germany/Poland by Nikos and me. It involves a lot of summary measures and was hard to write about + still has issues due to summarising scores across very different scales.

Here as we are interested in transitions from big to small to big it may not make sense to summarise scores at all and instead do everything by forecast date. This might make summarising results quite hard and so having some kind of scale-free (or scale that makes sense) measure would potentially be a good idea.

Some stan tangental links etc: https://avehtari.github.io/modelselection/CV-FAQ.html

This link should only be closed once further issues for each evaluation task have been opened.

[seabbs]

More random stan related musings.

https://statmodeling.stat.columbia.edu/2018/08/03/loo-cross-validation-approaches-valid/
https://doi.org/10.1080/00949655.2020.1783262

[jbracher]

This is related to the targets we want to forecast. I see the following options:

• total number of cases. Advantage: can be compared to Hub forecasts; predicting non-trivial pattern in the aggregate case numbers is something models have generally struggled to do.
• case numbers by variant. Maybe less attractive as (i) on very different orders of magnitude (ii) for each strain separately the model is kind of trivial (iii) don't even have direct observations.
• variant percentages. Advantage: has been prominent in public discussion. Personally I'm not overly fond of these percentage numbers as on their own I find them a bit hard to interpret. But probably still worth including.

[jbracher]

I think this is a case where detailed visual inspection of forecasts may be more informative than averaging of scores (probably need both). In terms of scores I think one could make a point for scoring on the log scale, but my preference would nonetheless be with regular CRPS/WIS to be consistent with related work.

[seabbs]

Definitely agree just need to stick to total cases as for the single strain model used as the baseline there is no other output.

Agree that summarising with averages is probably not going to help here. Also agree on sticking with standard scoring measures as more work needs to be done on the impact of scoring on the log scale.

Could in theory look at variant percentages etc in the models that have them but unless there is a pressing reason I would prefer to keep this tightly focussed on forecast performance.

[sbfnk]

I agree. The main hypothesis is that accounting for variants improves overall forecast performance vs. not doing so, right?

[seabbs]

I've been looking at modelling two strain model ~ one strain model * exp(coefficients)` as a way of stepping away from visual analysis (where coefficients are features like the proportion of delta variant and country). Are there theoretical issues with this?