Unexpected `digest::digest()` results within a `test_that()` block

[louisaslett]

We have encountered a bit of a head scratcher in tests that were failing in a fairly complex package currently under development. I've tracked it down to the following minimal reprexs and we would appreciate any assistance with the root cause. I've a gut instinct this might not be a bug, but rather something subtle in how test_that() is manipulating the runtime environment ... it would be nice to get to the bottom of understanding it either way, so that we can design the tests to work as expected.

One line problem description: Digests of lists not containing functions have the same digest immediately before and after returning from a function call, both interactively and in test_that() blocks, as expected. But, the digest of a list containing a function changes immediately before and after returning from a function call when run within a test_that() block.

OK scenario: No function in the returned list (correct behaviour both interactively and in test_that())

Running interactively at console:

f <- function(n) {
  x <- list(a = rnorm(n), b = "Hello!", c = n)
  print(digest::digest(x))
  return(x)
}

a <- f(10)
## [1] "c223717866fd9ae1b0d4f132a4fac580"
print(digest::digest(a))
## [1] "c223717866fd9ae1b0d4f132a4fac580"

Within a test_digest.R file (I realise there are no actual tests here, I'm trying to keep the example very minimal):

test_that("no function in list",{
  f <- function(n) {
    x <- list(a = rnorm(n), b = "Hello!", c = n)
    print(digest::digest(x))
    return(x)
  }
  
  a <- f(10)
  print(digest::digest(a))
})

Then run through testthat:

devtools::test()
## ℹ Testing reprex
## ⠏ |          0 | digest
## [1] "adb9e4d560424065ffcb0414a32fe9b2"
## [1] "adb9e4d560424065ffcb0414a32fe9b2"

Not OK scenario: Function in the returned list (correct behaviour interactively and unexpected output in test_that())

Running interactively at console:

f <- function(n) {
  x <- list(a = rnorm(n), b = "Hello!", c = n, function(z) { z^2 })
  print(digest::digest(x))
  return(x)
}

a <- f(10)
## [1] "b126ff7bbe0b2c48beb7ace5e1898993"
print(digest::digest(a))
## [1] "b126ff7bbe0b2c48beb7ace5e1898993"

Within a test_digest.R file:

test_that("with function in list",{
  f <- function(n) {
    x <- list(a = rnorm(n), b = "Hello!", c = n, function(z) { z^2 })
    print(digest::digest(x))
    return(x)
  }
  
  a <- f(10)
  print(digest::digest(a))
})

Then run through testthat:

devtools::test()
## ℹ Testing reprex
## ⠏ |          0 | digest
## [1] "1ecd7ae8a2c7f930cb5ded1faa911237"
## [1] "10fe76cf020f9faab8a69949b7647154"

The unexpected behaviour here is that under test_that() we get different digests of the same object immediately before and after the return from f().

Any help understanding this, even if it is not a bug, would be hugely appreciated.

Thanks!

[hadley]

It's not clear to me what's going on here. You can get a bit more insight if you use lobstr::sxp() to get the full internal representation of the object and then lobstr::compare() to compare the two you can see what's different:

test_that("with function in list",{
  b <- NULL
  f <- function(n) {
    x <- list(a = rnorm(n), b = "Hello!", c = n, d = function(z) { z^2 })
    b <<- x
    print(digest::digest(x))
    return(x)
  }
  
  a <- f(10)
  print(waldo::compare(lobstr::sxp(a), lobstr::sxp(b)))
  print(digest::digest(a))
})

That yields

`attr(old$d$_env, 'named')`: 3
`attr(new$d$_env, 'named')`: 4

But I don't know (and I can't seem to quickly figure out) what the _env means in waldo, and I don't know why that thing would get a named attribute.

[hadley]

On the plus side I've realised that you can reproduce inside with just local(), so this isn't a testthat issue 😄

options(keep.source = FALSE)

local({
  b <- NULL

  f <- function() {
    b <<- function(z) { z^2 }
    print(rlang::hash(b))
    b
  }
  a <- f()

  print(waldo::compare(lobstr::sxp(a), lobstr::sxp(b)))
  print(rlang::hash(a))
})
#> [1] "98dd9930c1b3c440941aa64f2e4ab9e2"
#> `attr(old$_env, 'named')`: 3
#> `attr(new$_env, 'named')`: 4
#> [1] "47381be1d9967d3fbb39dbfd0386c2dd"

^{Created on 2025-07-16 with reprex v2.1.1}

Ooooh, and _env isn't coming from waldo, it's coming from lobstr::sxp().

I think that implies that named is some attribute of environments that you can't usually see, and it's likely to be somehow related to reference counting.

OTOH I still see same hashes but different comparisons in this example

options(keep.source = FALSE)

local({
  b <- NULL

  f <- function() {
    b <<- function(z) { z^2 }
    b
  }
  a <- f()

  print(rlang::hash(b))
  print(waldo::compare(lobstr::sxp(a), lobstr::sxp(b)))
  print(rlang::hash(a))
})
#> [1] "777816fd98672cc5751587663e9b7528"
#> `attr(old$_env, 'named')`: 3
#> `attr(new$_env, 'named')`: 4
#> [1] "777816fd98672cc5751587663e9b7528"

^{Created on 2025-07-16 with reprex v2.1.1}

So maybe that's a red herring?

Switching to inspect gives a slightly different view, but I don't think that's any more informative:

options(keep.source = FALSE)

inspect <- function(x) {
  capture.output(.Internal(inspect(x)))
}

local({
  b <- NULL

  f <- function() {
    b <<- function(z) { z^2 }
    print(rlang::hash(b))
    b
  }
  a <- f()

  print(waldo::compare(inspect(a), inspect(b)))
  print(rlang::hash(a))
})
#> [1] "98dd9930c1b3c440941aa64f2e4ab9e2"
#> old vs new
#> - "@11bb90510 03 CLOSXP g0c0 [REF(3)] "
#> + "@11bb90510 03 CLOSXP g0c0 [REF(4)] "
#>   "FORMALS:"
#>   "  @11ba40ab8 02 LISTSXP g0c0 [REF(8)] "
#> - "    TAG: @12a92fd18 01 SYMSXP g0c0 [MARK,REF(212)] \"z\""
#> + "    TAG: @12a92fd18 01 SYMSXP g0c0 [MARK,REF(213)] \"z\""
#>   "    @12a811400 01 SYMSXP g0c0 [MARK,REF(65535)] [missing argument]"
#>   "BODY:"
#>   "  @11ba42620 06 LANGSXP g0c0 [REF(6)] "
#> - "    @12a811240 01 SYMSXP g0c0 [MARK,REF(7696),LCK,gp=0x7000] \"{\" (has value)"
#> + "    @12a811240 01 SYMSXP g0c0 [MARK,REF(65535),LCK,gp=0x7000] \"{\" (has value)"
#>   "    @11ba42770 06 LANGSXP g0c0 [REF(3)] "
#> - "      @12a81c5e0 01 SYMSXP g0c0 [MARK,REF(17),LCK,gp=0x5000] \"^\" (has value)"
#> + "      @12a81c5e0 01 SYMSXP g0c0 [MARK,REF(18),LCK,gp=0x5000] \"^\" (has value)"
#> and 4 more ...
#> [1] "47381be1d9967d3fbb39dbfd0386c2dd"

^{Created on 2025-07-16 with reprex v2.1.1}

[hadley]

Any way, I've convinced myself that this isn't a testthat problem, and I predict it would take at least several hours for me or someone senior on my team to run this down, so unfortunately this is going to have to remain a mystery.