Skip to content

Improve test output for failed stubs#2498

Merged
eregon merged 1 commit intomasterfrom
stub-improvement
Jan 15, 2025
Merged

Improve test output for failed stubs#2498
eregon merged 1 commit intomasterfrom
stub-improvement

Conversation

@mislav
Copy link
Member

@mislav mislav commented Jan 15, 2025

The stub/unstub test helpers are unfortunately opaque as both are undocumented, and when stubbing fails for some reason, determining why is left as a guessing/debugging exercise. This was a time-sink for myself on many an occasion.

This improves the unstub <PROGRAM> helper output with the following features:

  • If the number of PROGRAM invocations was fewer than the number of stubbed (i.e. expected) arguments, it now prints expected X invocations, got Y to stderr;

  • If PROGRAM got invoked but unstub PROGRAM still failed, it was likely that some of the invocations didn't match the stubbed ones. However, the human reading the test output has no insight into which invocations actually happened. The unstub helper now prints:

    re-run test with <PROGRAM>_STUB_DEBUG=3 to log `<PROGRAM>' invocations
    

    This "debug" mode already existed, but is off by default, and could previously only be discovered by someone who read the stub implementation code. After following the new suggestion, actual invocations of PROGRAM and all its arguments will be printed to stderr during the test run, hopefully making it clearer for the human debugging the test.

@mislav mislav requested a review from eregon January 15, 2025 12:41
Copy link
Member

@eregon eregon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@eregon eregon merged commit fe8c6c2 into master Jan 15, 2025
6 checks passed
@eregon eregon deleted the stub-improvement branch January 15, 2025 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants