Running a single snapshot test deletes snapshots from all other snapshot tests in that file

[jennybc]

Moving this conversation here: posit-dev/positron#6580 (comment)

You can grab this toy package where I've been exercising various things when I work on Positron's test explorer: https://github.com/jennybc/testfun

(For the record, I'm writing this when testfun's main is at 8d84efc, just in case it evolves before running this reprex.)

Run an individual test file with 3 snapshot tests:

> test_file("tests/testthat/test-snapshots.R")

══ Testing test-snapshots.R ═══════════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 3 ] Done!

All is well. All tests are passing. All snapshots are there.

Now run just one of those snapshot tests (which is ultimately what Positron's test explorer lets users do with various UI gestures):

> test_file("tests/testthat/test-snapshots.R", desc = "snapshot test 2")

══ Testing test-snapshots.R ═══════════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ] Done!

All seems well. But the git diff shows that the snapshot file has changed and snapshots 1 and 3 have gone missing. You can see this also by re-running the whole file:

> test_file("tests/testthat/test-snapshots.R")

══ Testing test-snapshots.R ═══════════════════════════════════════════════════════
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 3 ]

── Warning (test-snapshots.R:2:3): snapshot test 1 ────────────────────────────────
Adding new snapshot:
Code
  # first snapshot

── Warning (test-snapshots.R:10:3): snapshot test 3 ───────────────────────────────
Adding new snapshot:
Code
  # third snapshot
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 3 ]

[DavisVaughan]

This is probably the same issue, but here's proof that it doesn't have to be a snapshot test that you run. Running any test will clear out the snapshot file for that test file.

Screen.Recording.2025-03-14.at.10.48.18.AM.mov

[d-morrison]

This issue also comes up when snapshot tests are conditionally skipped (e.g., on certain OSes); devtools::test() deletes skipped snapshots as unused (can reprex if helpful)

[jennybc]

I'm running into this a lot, now that Positron defaults to the test explorer and makes it so easy to run a subset of tests (e.g. just the failing ones).

[hadley]

I've found this annoying too. I have assumed that this is not possible to fix (or at least rather difficult), but I should actually verify that.

[hadley]

Closely related to #1481, which is about tests not running when some condition is set (so same idea, slightly different reason for not running) — actually that's a little different because it's about expect_snapshot_file() which doesn't have quite the same protections.

[hadley]

Some notes as I bring this code back to mind:

• snapshot_cleanup() deletes the entire file if nothing is recorded; that's not the problem here.
• I see there are some special code for handling skips already, so maybe that problem is fixed? Or, at least, I can't reproduce it trivially.
• The problem with filtering is that the tests are never run, so $reset() (which copies the snaps from old to current) will never get run.

So I think we need a new code path for the case where the test is filtered; it needs to copy over all the old snapshots to the cur snapshots apart from the matching test.

@d-morrison can you provide a reprex? I can no longer easily reproduce the issue with skips.

[hadley]

Wait, actually I can reproduce if I set local_on_cran() and test expect-inheritance.R.

...

Oh this is a new problem, because expect_snapshot() no longer generates a skip, it just returns NULL. So I'm back to not being about to reproduce the skip problem in general.