Retain calibration data

[topepo]

add_calibration_data() would solve the problems of

• Having the calibration data being peeled off of the training/analysis set
• Users are unable to use a static calibration set.
• More reproducible workflows (see better use of random numbers for one-shot work and calibration tune#1038) and easier debugging.
• Enable more advanced postprocessors for different cases (forecasting, applicability domains, etc.)

When using add_calibration_data(), a tibble that conforms to the training set mold is required.

The calibration set would be stored in worflow$pre$calibration (similar to case weights).

When fit.workflow() or .fit_post() are called, they first preprocess the calibration set (if any), make predictions, and pass those predictions^* to the calibrator.

^*I believe that we should also have a way to pass in the molded calibration set for the trailer adjustments. We have 2-3 cases where those are required.

[hfrick]

Leaving some notes on this from our discussion, supplemented with more poking around:

We currently have the calibration argument to fit.workflow().

When fit.workflow() or .fit_post() are called, they first preprocess the calibration set (if any), make predictions, and pass those predictions* to the calibrator.

Agreed! This is already happening, but the documentation is outdated (it used to be that users needed to provide the predictions, which is pretty inconvenient): #292

This puts the responsibility to have separate datasets for fitting the primary model (including preprocessing) and fitting the calibration model into the user's hands, but makes it pretty obvious by having two separate arguments.

This is a static set by definition, which fits the area of focus for workflows, imo.

To allow for resampling/tune the primary model with a static calibration set, my proposal is to add a calibration argument to fit_resamples() and tune_*(), analogous to how it's done here in workflows. (The name of that argument is up for discussion #263). Happy to open an issue on tune for that if you agree!

To resample also the calibration/post-processor, I'm thinking this should happen in rsample. tidymodels/rsample#573 tracks that but is currently scheduled for "later". I've jotted down how far we got in tidymodels/rsample#573 (comment).

Can you say a bit more about these two aspects?

Enable more advanced postprocessors for different cases (forecasting, applicability domains, etc.)

*I believe that we should also have a way to pass in the molded calibration set for the trailer adjustments. We have 2-3 cases where those are required.

If you have some examples for those 2-3 cases, that would be super helpful to add, too!

[topepo]

My comments in #263 describe a path to use fit() instead of add_calibration_data().