locale / collation used in testhat

[Christoph999]

As discribed here testthat changed the locale which results in "wrong" test-results (I understand, that reproducibility is an issue). With "wrong" I mean the following: Assume I write a function this_is_tested_to_be_true() based on sort, which passes a test in testthat

this_is_tested_to_be_true <- function() {
  xx <- c("Schaffhausen", "Schwyz", "Seespital", "SRZ")
  r <- all(sort(xx) == c( "SRZ", "Schaffhausen", "Schwyz", "Seespital"))
  return(r)
}

and a test

test_that("test sort 2", {
  test_that(this_is_tested_to_be_true(), TRUE)
})

When I / others use this function in the package, all tests will pass, but in my daily work, it will return FALSE. To me this does not make sense.

What am I missing?

[gaborcsardi]

The output of sort() is platform dependent, you get different results based on the locale. testthat now makes sure that the tests behave the same way on every platform, by setting the collation locale to C.

In your package code, you cannot assume that this_is_tested_to_be_true() always returns TRUE (or always returns FALSE), it may return either depending on the user's platform configuration.

[Christoph999]

@gaborcsardi Do you plan to keep this behaviour? This is really surprising to me. In our case, tests fail, although the results "in daily life" work as expected.

Thanks a lot for your support!

[gaborcsardi]

In our case, tests fail, although the results "in daily life" work as expected.

No, they do not work as expected, only on your computer. On other computers they might not.

[gaborcsardi]

Btw. note that R CMD check also switches to that same locale as testthat:

All these tests are run with collation set to the C locale, and for the examples and tests with environment variable LANGUAGE=en: this is to minimize differences between platforms.

https://cran.r-project.org/doc/manuals/R-exts.html#Checking-packages

So even if you don't use testthat, but just write your tests in tests/test.R you'll get the same behavior in R CMD check.

[hadley]

There's no intention to change this behaviour; I'd recommend changing your test. You could use withr::local_collate("your locale") to enforce the sort order that you expect (but I think this will be hard to do cross-platform since Windows and Mac/Linux use different locale names).

[Christoph999]

@hadley Thanks for your suggestion. One question: Do you know, whether there is an argument to prefer option 2 over option 1? myfun is part of a package...

myfun <- function(my_collation = "German_Switzerland.1252", ...) {
  my_locale <- Sys.getlocale("LC_COLLATE")

  # Option 1
  invisible(Sys.setlocale("LC_COLLATE", my_collation))
  r <- sort(...)
  invisible(Sys.setlocale("LC_COLLATE", my_locale))

  # Option 2
  withr::local_collate(my_collation)
  r <- sort(...)
  withr::local_collate(my_locale)

  return(r)
}

From the discussions above, this solution makes sense to me, as myfun depends on the collation (which I did not realize before).

[gaborcsardi]

Option 1 is not good, because if sort() fails, then the locale is never reset. You would need to use on.exit() to reset it, if you want to avoid using the withr package.

Option 2 better, but you don't need the second withr::local_collate() call, withr runs that automatically when your function exits (normally or abnormally). So you also don't need to save the current locale in my_locale:

myfun <- function(my_collation = "German_Switzerland.1252", ...) {
  withr::local_collate(my_collation)
  sort(...)
}

Btw. this locale is not available on macOS, so it will not work on that platform. It might not be available on Linux and some Windows systems, either:

❯ Sys.setlocale("LC_COLLATE", "German_Switzerland.1252")
[1] ""
Warning message:
In Sys.setlocale("LC_COLLATE", "German_Switzerland.1252") :
  OS reports request to set locale to "German_Switzerland.1252" cannot be honored

Which might be of course perfectly fine, if all your users are on platform that has this locale.

[Christoph999]

@gaborcsardi Thanks, I didn't know that! Just for completeness the code using on.exit():

myfun <- function(my_collation = "German_Switzerland.1252", ...) {
  my_locale <- Sys.getlocale("LC_COLLATE"))
  on.exit(expr = Sys.setlocale("LC_COLLATE", my_locale))
  
  Sys.setlocale("LC_COLLATE", my_collation)
  r <- sort(...)
  return(r)
}

[MichaelChirico]

I came here from Googling looking for a reference that this is indeed what testthat does. Leaving the links here for any future visitors who get here before the other links.

It's mentioned in the testthat 3e vignette:

https://testthat.r-lib.org/articles/third-edition.html#reproducible-output

The actual function doing so is: local_reproducible_output()

https://testthat.r-lib.org/reference/local_test_context.html