mold.data.frame crashes with logical outcomes

[tripartio]

Hello. For some strange reason, on hardhat version 1.4.1, mold() crashes when I use the x-y format with a logical outcome variable, but not in other cases. In the example below, you can see that mold.data.frame() works fine for a factor outcome. Yet mold.data.frame() crashes on a logical outcome even though mold.formula() works fine for the exact same dataset.

library(hardhat)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

# Create tibble version of iris dataset
iris <- tibble::as_tibble(iris)
iris 
#> # A tibble: 150 × 5
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#>           <dbl>       <dbl>        <dbl>       <dbl> <fct>  
#>  1          5.1         3.5          1.4         0.2 setosa 
#>  2          4.9         3            1.4         0.2 setosa 
#>  3          4.7         3.2          1.3         0.2 setosa 
#>  4          4.6         3.1          1.5         0.2 setosa 
#>  5          5           3.6          1.4         0.2 setosa 
#>  6          5.4         3.9          1.7         0.4 setosa 
#>  7          4.6         3.4          1.4         0.3 setosa 
#>  8          5           3.4          1.5         0.2 setosa 
#>  9          4.4         2.9          1.4         0.2 setosa 
#> 10          4.9         3.1          1.5         0.1 setosa 
#> # ℹ 140 more rows

# Factor outcome: no problem with mold.data.frame
mold(
  x = iris |> select(-Species),
  y = iris['Species']
)
#> $predictors
#> # A tibble: 150 × 4
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width
#>           <dbl>       <dbl>        <dbl>       <dbl>
#>  1          5.1         3.5          1.4         0.2
#>  2          4.9         3            1.4         0.2
#>  3          4.7         3.2          1.3         0.2
#>  4          4.6         3.1          1.5         0.2
#>  5          5           3.6          1.4         0.2
#>  6          5.4         3.9          1.7         0.4
#>  7          4.6         3.4          1.4         0.3
#>  8          5           3.4          1.5         0.2
#>  9          4.4         2.9          1.4         0.2
#> 10          4.9         3.1          1.5         0.1
#> # ℹ 140 more rows
#> 
#> $outcomes
#> # A tibble: 150 × 1
#>    Species
#>    <fct>  
#>  1 setosa 
#>  2 setosa 
#>  3 setosa 
#>  4 setosa 
#>  5 setosa 
#>  6 setosa 
#>  7 setosa 
#>  8 setosa 
#>  9 setosa 
#> 10 setosa 
#> # ℹ 140 more rows
#> 
#> $blueprint
#> XY blueprint:
#> # Predictors: 4
#> # Outcomes: 1
#> Intercept: FALSE
#> Novel Levels: FALSE
#> Composition: tibble
#> 
#> 
#> $extras
#> NULL

# Create dataset with logical outcome: Setosa
setosa <- iris |> 
  mutate(Species = Species == 'setosa') |> 
  rename(Setosa = Species)
setosa
#> # A tibble: 150 × 5
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width Setosa
#>           <dbl>       <dbl>        <dbl>       <dbl> <lgl> 
#>  1          5.1         3.5          1.4         0.2 TRUE  
#>  2          4.9         3            1.4         0.2 TRUE  
#>  3          4.7         3.2          1.3         0.2 TRUE  
#>  4          4.6         3.1          1.5         0.2 TRUE  
#>  5          5           3.6          1.4         0.2 TRUE  
#>  6          5.4         3.9          1.7         0.4 TRUE  
#>  7          4.6         3.4          1.4         0.3 TRUE  
#>  8          5           3.4          1.5         0.2 TRUE  
#>  9          4.4         2.9          1.4         0.2 TRUE  
#> 10          4.9         3.1          1.5         0.1 TRUE  
#> # ℹ 140 more rows

# !!! Logical outcome crashes mold.data.frame !!!
mold(
  x = setosa |> select(-Setosa),
  y = setosa['Setosa']
)
#> Error in `standardize()`:
#> ! Not all columns of `y` are known outcome types.
#> ℹ This column has an unknown type: "Setosa".

# Yet no problem with logical outcome on same dataset with mold.formula
mold(
  formula = Setosa ~ .,
  data = setosa
)
#> $predictors
#> # A tibble: 150 × 4
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width
#>           <dbl>       <dbl>        <dbl>       <dbl>
#>  1          5.1         3.5          1.4         0.2
#>  2          4.9         3            1.4         0.2
#>  3          4.7         3.2          1.3         0.2
#>  4          4.6         3.1          1.5         0.2
#>  5          5           3.6          1.4         0.2
#>  6          5.4         3.9          1.7         0.4
#>  7          4.6         3.4          1.4         0.3
#>  8          5           3.4          1.5         0.2
#>  9          4.4         2.9          1.4         0.2
#> 10          4.9         3.1          1.5         0.1
#> # ℹ 140 more rows
#> 
#> $outcomes
#> # A tibble: 150 × 1
#>    Setosa
#>    <lgl> 
#>  1 TRUE  
#>  2 TRUE  
#>  3 TRUE  
#>  4 TRUE  
#>  5 TRUE  
#>  6 TRUE  
#>  7 TRUE  
#>  8 TRUE  
#>  9 TRUE  
#> 10 TRUE  
#> # ℹ 140 more rows
#> 
#> $blueprint
#> Formula blueprint:
#> # Predictors: 4
#> # Outcomes: 1
#> Intercept: FALSE
#> Novel Levels: FALSE
#> Composition: tibble
#> Indicators: traditional
#> 
#> 
#> $extras
#> $extras$offset
#> NULL

^{Created on 2025-05-23 with reprex v2.1.1}

This looks like a bug. Could someone please help me with it?

[EmilHvitfeldt]

I'm adding that this has never worked since at least 0.1.6 (stopped testing versions there).

You are running into an error here with the logical outcome because standardize() only accepts one of the following as outcomes.

• A factor vector
• A numeric vector
• A 1D numeric array
• A numeric matrix with column names
• A 2D numeric array with column names
• A data frame with numeric or factor columns

Do you mind telling us what use-case you have in mind where the above types doesn't work?

[tripartio]

In the development version of my https://github.com/tripartio/autogam package, there are two basic prediction types: numeric prediction (regression) uses numeric outcomes and binary prediction (classification) converts the outcome type to logical. I want to use hardhat to support various input formats. logical is the natural R datatype for binary outcomes. So, I'm quite surprised to hear that hardhat has never supported logical outcomes. After all, logical could be considered a special factor with two levels, so I don't understand why factor would be supported but not logical.

I'm not sure what standardize() does within mold(), but should the resolution for logical outcomes be as simple as recognizing they're logical and then converting them to 1s and 0s if necessary?

[topepo]

There are some models that assume that you are doing regression when being passed numerics and classification when outcomes are factors. It is risky to assume that an implicit or explicit conversion via as.numeric() can take place.

We've always been pretty upfront about requiring classification models to have factor outcomes. It's good practice, but it also reduces code complexity for us.

tldr; we require factors for classification (once you get to parsnip) so you shouldn't do logicals.

[tripartio]

We've always been pretty upfront about requiring classification models to have factor outcomes. It's good practice, but it also reduces code complexity for us.

tldr; we require factors for classification (once you get to parsnip) so you shouldn't do logicals.

@topepo I've encountered this as I've started using tidymodels but I don't understand this design decision at all. I understand why categorical or multinomial classification outcomes should be encoded as factors, but I cannot fathom why that would apply to binary classification. Logical intuitively seems to me to be the only natural choice for binary classification, but apparently, for some reason that completely escapes me, the tidymodels team has other considerations. I'm sure you've all thought long and deeply about this, but the reasoning behind this design choice is not at all obvious to me.

Could you please explain to me (or point me to a longer written explanation) why binary classification outcomes should not be encoded as logical in R (or in tidymodels-compatible R software)?

[topepo]

(Disclaimer: I haven’t looked at autogam so I hope this doesn’t sound like I’m targeting anyone in particular)

It’s a quality-of-life rationale based on having worked with a lot of different R packages. Many of them are designed to do just what the author needed at the time, regardless of how much their needs sync with general user needs. That is absolutely legit but very myopic.

However, if users have to move between packages, this philosophy becomes a nightmare. If there are no conventions to follow, everyone has an API that works differently. This rant in 2012 gets to some of it. In 2018, we defined some conventions that we would adhere to for tidymodels, and most are based on the lack of modeling standards in R.

Another consequence of not having (or following) some convention is that it will push a lot of people to Python, which has far less functionality, but those packages do generally follow API standards and use the same conventions. TBH, their conventions could do a lot better.

So why this convention around factors? First, we want to treat qualitative data as qualitative. The user-facing data most likely isn’t integer or logical but represents what the thing is and we should meet users where their data are, not where our computational needs are. In our computational functions, we can easily encode data to what is required, so let’s be consistent about the source data.

Second, it’s not great to have a data standard for binary data and another for multinomial data. It’s all qualitative, and adding more data-driven logic adds a lot of complexity to our code. Plus, it’s one more thing for the user to remember.

Lastly, if you use booleans, how do you label your probability estimates? There are no factor levels, so there is no data built into the model that defines what it means to be an event or non-event.

I think being a bit restrictive about the outcome's format results in better, more supportable code that captures all of the information that we need about the data.

[tripartio]

@topepo Thanks for your detailed response.

In fact, since I recently learned about the tidymodels convention document you pointed to, I am already incorporating most of its principles into the design of the autogam package. So, I heartily agree (and gratefully thank you!) for your strong push towards conventions in modelling package design.

However, even on looking over it again now, I don't believe I read anywhere there about this notion of "binary outcomes as factors". If I understand your explanation, you give three reasons. I'm deliberately rephrasing them from the way you explained them to expose my understanding (or misunderstanding) of what you tried to communicate (so, please correct my misunderstandings!):

1. The real-world data has qualitative meaning, so users should be forced to encode the outcomes according to their qualitative meaning.
2. Since binary data has intrinsic qualitative meaning, it should be encoded as factors, which is the native R datatype for encoding qualitative meaning, just like multinomial outcomes.
3. Binary data lacks labels to distinguish the FALSE from the TRUE cases.

For the first point, I know you did not use the word "forced", but that is how I understood your point. And this is where I think I disagree. On one hand, I agree that as a package author, I should be disciplined to follow conventions such as tidymodels. On the other hand, I think that my packages should be sufficiently flexible to accept all reasonable kinds of input data that users provide. I consider it quite unreasonable to reject logical outcomes as "unreasonable" or "illegitimate" on the package level. logical is such a fundamental data format (for predictors and outcomes alike). It is the most natural and intuitive format for binary outcomes. I definitely think that any binary prediction modelling package should be able to cleanly and naturally handle logical outcomes without forcing the user to encode them as factors.

For the second and third points, I understand the logic, but again, I don't think such logic should be imposed on users. The resolution to me is very natural: the two levels should be labelled as "FALSE" and "TRUE". I don't understand why that would be complicated. If users want something better than that, then they should follow your advice and encode the outcomes as factors. But that should be their choice.

One important point that you implied here but made explicit in your 2012 "rant" is that it is unclear which of the two binary cases should be considered the primary or "positive" class. However, here I think your argument is self-defeating (unless I misunderstand it). This is only a problem when your advice is followed and binary outcomes are encoded as factors--indeed, then there is no natural choice of the first or the second level as the positive case. (I note that your 2012 rant recommends the first case but, unless I'm mistaken, the yardstick package currently defaults with the second case. This shows how arbitrary the decision is.) However, when the binary outcome is kept logical, then there is no issue. The clear and natural choice is that TRUE is the positive case. There's no ambiguity there. (In fact, that very important issue is precisely the reason for my strong preference for logical encoding for binary outcomes in the autogam package--it is often important to distinguish which case is positive and TRUE is unambigious.)

My point here is not to convince you or anyone else to change the tidymodels conventions, but my point is hopefully to convince you to give first-class support for "binary outcomes as logical". Concretely, I hope that the issue I have reported here will be treated as a bug to be fixed, rather than a design decision to be ignored. I hope that my arguments are sufficient to respect the choice to support such functionality for package designers who disagree with this controversial point, rather than shutting out much of the fantastic tidymodels infrastructure from us just because of this point of disagreement.

(I also note the inconsistency I pointed out above that the formula interface accepts logical outcomes fine; it is only in the x-y interface that this is a bug.)

[EmilHvitfeldt]

IMO i don't think there is much to gain from thinking that binary classification and multi-class classifications are different things.

The upsides to allowing logicals for binary classification would be largely outweighed by the extra logic we would need to put in the different packages to accept logicals and factors as opposed to just factors.

the yardstick package currently defaults with the second case

{yardstick} defaults to using the first level in a factor as the positive class. and can be changed by the event_level argument in metrics where that matters.

yardstick:::yardstick_event_level()
#> [1] "first"

[tripartio]

the yardstick package currently defaults with the second case

{yardstick} defaults to using the first level in a factor as the positive class. and can be changed by the event_level argument in metrics where that matters.

yardstick:::yardstick_event_level()
#> [1] "first"

Sorry about that; thanks for the correction.

[tripartio]

IMO i don't think there is much to gain from thinking that binary classification and multi-class classifications are different things.

The upsides to allowing logicals for binary classification would be largely outweighed by the extra logic we would need to put in the different packages to accept logicals and factors as opposed to just factors.

Please give me a couple of days to work on responding to this (with code).

[tripartio]

It turns out that the fix for this issue is as simple as changing is_known_output_type() from

is_known_output_type <- function(x) {
  is.numeric(x) || is.factor(x)
}

to

is_known_output_type <- function(x) {
  is.numeric(x) || is.factor(x) || is.logical(x)
}

I've created a PR: #290.

[tripartio]

Coming back to these important comments:

IMO i don't think there is much to gain from thinking that binary classification and multi-class classifications are different things.

I respectfully disagree. From my package and machine learning design philosophy, there is very much to gain. Binary prediction is fundamentally different from categorical prediction. However, my aim here is not to convince you otherwise but rather to persuade you to please respect this intellectual disagreement by not considering my differing position an "error". Rather, it should be accommodated if feasible.

The upsides to allowing logicals for binary classification would be largely outweighed by the extra logic we would need to put in the different packages to accept logicals and factors as opposed to just factors.

I hope that my PRs here and at tidymodels/yardstick#545 can persuade you that it is not necessary to reject such inputs as errors: the code needed to accommodate this alternative philosophy of binary-as-logical is not anywhere as challenging or complicating as you might have thought. The solution requires simply converting logical inputs into factors; then all the factor-based code works as usual.

[topepo]

There is a lot more to consider than yardstick. A lot of tidymodels is about smoothing out the disparate APIs and arguments across R packages. We have a lot of code that translates an R factor to integers, zero-based integers, characters, etc. because that's how the package author wanted it. We'd also have to change all of that code. There are other complications too, and they happen across the board.

Is there a reason that you cannot convert to a factor before your computations? Given the risk-benefit ratio, that seems like a sound option.

I know you really want this feature. Respectfully, we think that the overhead outweighs the need so we won't make changes to this effect. Sorry.