Consider adding a trend categorisation component to evaluation

[kaitejohnson]

We could classify each forecast from each model as increasing, decreasing, or stable based on the trajectory of hospital admissions.

This could either be something like probability increasing, probability decreasing, probability stable (which would be based on the posterior draws)

Then we could evaluate the trend by categorising the observation into one of 3 bins and using the Brier score to evaluate.

[kaitejohnson]

Discussed today:

• something akin to CDC NWSS wwval
• R(t) at the WWTP or state level as a metric for P(increasing), P(decreasing) or P(stable), and evaluate against later observed values (whether they increased, decreased or were stable)?
• previously used loess smoothing and then calculated trend from older to more recent estimates

[kaitejohnson]

Still mulling over whether this should be done, would be good to have another conversation perhaps. It's not written into the current analysis plan, which is pretty focused on ww vs no ww in real-time.

However, there probably is room for a wastewater treatment plant specific analysis where we evaluate the trend categorisation and provide some summary of agreement with eventual trend? To assess the value of the model-based approach compared to the data alone we could compare:

• trend classification algorithm applied to each individual WWTP however it is currently being done
• trend classification based on the R(t) of the subpopulation at the forecast date

This might provide a proxy for the value of the model-based approach compared to the data alone for providing localised insights.

I am not sure this is in scope for this work though.