Article on automatic calibration splits #125
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
EmilHvitfeldt
approved these changes
Oct 21, 2025
There was a problem hiding this comment.
I think this is very valuable and high quality work.
I think this would be a good place for this content. It would also fit in rsample pkgdown for that matter.
there are small comments and nit picks but the overall structure and style i find very nice.
I'm also fine with the length. It is a complicated topic and without the prose and diagrams it would be hard to understand
| description: | | ||
| Learn how tidymodels generates automatic calibration sets for post-processing. | ||
| toc: true | ||
| toc-depth: 2 |
There was a problem hiding this comment.
Suggested change
| toc-depth: 2 | |
| toc-depth: 3 |
due to the length of the article i think upping the TOC depth is nice, otherwise we don't update the toc that fast
|
|
||
| While preprocessing is the transformation of the predictors prior to a model fit, post-processing is the transformation of the predictions after the model fit. This could be as straightforward as limiting predictions to a certain range of values to as complicated as transforming them based on a separate calibration model. | ||
|
|
||
| A calibration model is used to model the relationship between the predictions based on the primary model and the true outcomes. An additional model means an additional chance to accidentially overfit. So when working with calibration, this is crucial: we cannot use the same data to fit our calibration model as we use to assess the combination of primary and calibration model. Using the same data to fit the primary model and the calibration model means the predictions used to fit the calibration model are re-predictions of the same observations used to fit the primary model. Hence they are closer to the true values than predictions on new data would be and the calibration model doesn't have accurate information to estimate the right trends (so that they then can be removed). |
There was a problem hiding this comment.
Suggested change
| A calibration model is used to model the relationship between the predictions based on the primary model and the true outcomes. An additional model means an additional chance to accidentially overfit. So when working with calibration, this is crucial: we cannot use the same data to fit our calibration model as we use to assess the combination of primary and calibration model. Using the same data to fit the primary model and the calibration model means the predictions used to fit the calibration model are re-predictions of the same observations used to fit the primary model. Hence they are closer to the true values than predictions on new data would be and the calibration model doesn't have accurate information to estimate the right trends (so that they then can be removed). | |
| A calibration model is used to model the relationship between the predictions based on the primary model and the true outcomes. An additional model means an additional chance to Accidentally overfit. So when working with calibration, this is crucial: we cannot use the same data to fit our calibration model as we use to assess the combination of primary and calibration model. Using the same data to fit the primary model and the calibration model means the predictions used to fit the calibration model are re-predictions of the same observations used to fit the primary model. Hence they are closer to the true values than predictions on new data would be and the calibration model doesn't have accurate information to estimate the right trends (so that they then can be removed). |
| ``` | ||
|
|
||
| While preprocessing is the transformation of the predictors prior to a model fit, post-processing is the transformation of the predictions after the model fit. This could be as straightforward as limiting predictions to a certain range of values to as complicated as transforming them based on a separate calibration model. | ||
|
|
There was a problem hiding this comment.
Below we are using the term primary model which we just started using. I like the term but i think it be nice to properly define it, in terms of pre/model/post diagram/terminology
|
|
||
| A calibration model is used to model the relationship between the predictions based on the primary model and the true outcomes. An additional model means an additional chance to accidentially overfit. So when working with calibration, this is crucial: we cannot use the same data to fit our calibration model as we use to assess the combination of primary and calibration model. Using the same data to fit the primary model and the calibration model means the predictions used to fit the calibration model are re-predictions of the same observations used to fit the primary model. Hence they are closer to the true values than predictions on new data would be and the calibration model doesn't have accurate information to estimate the right trends (so that they then can be removed). | ||
|
|
||
| rsample provides a collection of functions to make resamples for empirical validation of prediction models. So far, the assumption was that the prediction model is the only model that needs fitting, i.e., a resample consists of an analysis set and an assessment set. |
There was a problem hiding this comment.
Do we have a rsample doc page for analysis set and assessment set?
|
|
||
| rsample provides a collection of functions to make resamples for empirical validation of prediction models. So far, the assumption was that the prediction model is the only model that needs fitting, i.e., a resample consists of an analysis set and an assessment set. | ||
|
|
||
| If we include calibration into our workflow (bundeling preprocessing, (primary) model, and post-processing), we want an analysis set, a calibration set, and an assessment set. |
There was a problem hiding this comment.
Suggested change
| If we include calibration into our workflow (bundeling preprocessing, (primary) model, and post-processing), we want an analysis set, a calibration set, and an assessment set. | |
| If we include calibration into our workflow (bundling preprocessing, (primary) model, and post-processing), we want an analysis set, a calibration set, and an assessment set. |
|
|
||
| Let's start with the row-based splitting done by `sliding_window()`. We'll use a very small example dataset. This will make it easier to illustrate how the different subsets of the data are created but note that it is too small for real-world purposes. Let's use a data frame with 11 rows and say we want to use 5 for the analysis set, 3 for the assessment set, and leave a gap of 2 in between those two sets. We can make two such resamples from our data frame. | ||
|
|
||
|  |
There was a problem hiding this comment.
could we do 4 for assessment set and a gap of 1?
right now there is very little air inside the assessment set with regards to the text
There was a problem hiding this comment.
i'm realizing that this would be a huge undertaking
|
|
||
|  | ||
|
|
||
| We still get two resamples, however, the analysis set contains only 4 rows because only those fall into the window defined by the index. |
There was a problem hiding this comment.
Would it be beneficial to add the missing value at 1 or 5 such that the analysis sets have different length?
There was a problem hiding this comment.
or i guess, that isn't interesting at all. because then we do the same as the previous section
| analysis(r_split) | ||
| ``` | ||
|
|
||
| The sliding splits slide over _the data_, meaning they slide over observed values of the index and they slide only within the boundaries of the observed index values. So here, we can only slide within [3, 6] and thus cannot fit an inner analysis set of three and a calibration set of two into it. As established earlier, we fall back onto an empty calibration set in such a situation. |
There was a problem hiding this comment.
we mention a couple of times that we fall back. should we mention that we fall back with a warning?
|
|
||
| Calculating the length of the calibration set rather than the gap, together with rounding up when translating proportions to new lengths within the outer analysis set means that we prioritize allocating observations to the (inner) analysis and calibration set over allocating them to the gap. In this example here, this means that we are not leaving a gap between the analysis set and the calibration set. | ||
|
|
||
| However, rounding up for both (inner) analysis and calibration set when we don't have a gap could mean we end up allocating more observations than we actually have. So in that case, we try to take from the calibration set if possible and thus prioritzing fitting the prediction model over the calibration model. |
There was a problem hiding this comment.
Suggested change
| However, rounding up for both (inner) analysis and calibration set when we don't have a gap could mean we end up allocating more observations than we actually have. So in that case, we try to take from the calibration set if possible and thus prioritzing fitting the prediction model over the calibration model. | |
| However, rounding up for both (inner) analysis and calibration set when we don't have a gap could mean we end up allocating more observations than we actually have. So in that case, we try to take from the calibration set if possible and thus prioritizing fitting the prediction model over the calibration model. |
|
|
||
|  | ||
|
|
||
| The principle of how to contruct a calibration split on the (outer) analysis set remains the same. The challenges of abstracting away from the rows, as illustrated for sliding over observed instances of an index also remain. Here, we slide over observed periods. We observe a period, if we observe an index within that period. |
There was a problem hiding this comment.
Suggested change
| The principle of how to contruct a calibration split on the (outer) analysis set remains the same. The challenges of abstracting away from the rows, as illustrated for sliding over observed instances of an index also remain. Here, we slide over observed periods. We observe a period, if we observe an index within that period. | |
| The principle of how to construct a calibration split on the (outer) analysis set remains the same. The challenges of abstracting away from the rows, as illustrated for sliding over observed instances of an index also remain. Here, we slide over observed periods. We observe a period, if we observe an index within that period. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've written about how we make the automatic calibration splits. The article covers the guiding principles of our approach, both for how and why (although not in academic paper depth).
The goal is also to let people understand in detail what happens for the sliding resamples. It has gotten relatively lengthy, though. I'm wondering if it should stay here or, e.g., go into a separate article, similar to how we split out the details on how we deal with censoring for the dynamic survival metrics. I think there's value in working through those details somewhere (other than the source code directly), but we could also experiment with collapsible text. Do you have any preferences or suggestions? Or do you think the length is fine as it is?