Log the number of test cases run when a test function finishes

[mesqueeb]

Description

Given test logs can easily get burried, the final test logs printed are significant to understand how well a test ran.

TLDR;

• Currently: it's not clear from the final statement how many arguments were tested
• Desired: I suggest below some extra logs to better understand the total arguments tested:

Expected behavior

Desired

􀟈 Test aPileBringsHuggers(moveFn:) started.
􀟈 Passing 1 argument moveFn → (Function) to aPileBringsHuggers(moveFn:)
// ...
​􁁛 Argument in aPileBringsHuggers(moveFn:) passed after 2 seconds. 🆕
​􀟈 Passing 1 argument moveFn → (Function) to aPileBringsHuggers(moveFn:)
// ...
​􁁛 Argument in aPileBringsHuggers(moveFn:) passed after 2 seconds. 🆕
​􁁛 Test aPileBringsHuggers(moveFn:) with 2 arguments passed after 4.677 seconds. 🆕

Here above, I suggested the lines marked as 🆕

1. an extra log after each argument passes or fails, and
2. a total amount of arguments tested to the final log.

Actual behavior

Currently (example with 2 arguments tested)

􀟈 Test aPileBringsHuggers(moveFn:) started.
􀟈 Passing 1 argument moveFn → (Function) to aPileBringsHuggers(moveFn:)
// ...
​􀟈 Passing 1 argument moveFn → (Function) to aPileBringsHuggers(moveFn:)
// ...
​􁁛 Test aPileBringsHuggers(moveFn:) passed after 4.677 seconds.

Given that these logs easily get buried, I think something like the above would be way better.

Steps to reproduce

Write a swift test with arguments and run it.

Relevant thread:
https://forums.swift.org/t/fr-test-with-arguments-and-closures-dx-suggestions-to-clarify-all-arguments-were-tested/77677/7

swift-testing version/commit hash

not sure how to check

Swift & OS version (output of swift --version && uname -a)

swift-driver version: 1.115.1 Apple Swift version 6.0.3 (swiftlang-6.0.3.1.10 clang-1600.0.30.1)
Target: arm64-apple-macosx15.0
Darwin Dendrocopus.local 24.2.0 Darwin Kernel Version 24.2.0: Fri Dec 6 19:02:41 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T6030 arm64

[grynspan]

Argument in aPileBringsHuggers(moveFn:) passed after 2 seconds. 🆕

We intentionally do not include this line due to the noise it generates. We do indicate if an issue is recorded during executon of a test case.

Test aPileBringsHuggers(moveFn:) with 2 arguments passed after 4.677 seconds. 🆕

It should be possible for us to include the number of test cases run after a test function finishes.

[briancroom]

Argument in aPileBringsHuggers(moveFn:) passed after 2 seconds. 🆕

We intentionally do not include this line due to the noise it generates. We do indicate if an issue is recorded during executon of a test case.

We do have a verbosity control though - perhaps it could be useful to include this at some higher level?

[grynspan]

We do have a verbosity control though - perhaps it could be useful to include this at some higher level?

Yes, sorry—I said that in my head but failed to type it out. :) We can add it in very-verbose mode. However, Xcode does not have a verbosity setting, so we should also file a radar to add such a thing in the future.

[hrayatnia]

Hey @grynspan, I’d love to help you fix this issue if it is ok.

Could you please define the acceptance criteria for this issue? That would be super helpful.

For example, one of the questions is should I include number of test cases or args?
I'm asking because you changed the issue title to "number of test cases"
however issue desc says "a total amount of arguments tested to the final log."

I’ve already drafted a simple version, but I’d love to hear your thoughts.
https://github.com/hrayatnia/swift-testing/tree/943-log-number-of-test-cases-run-when-a-test-function-finishes

[stmontgomery]

Thanks for your contribution @hrayatnia! I saw the PR but haven't begun commenting yet since it's still a draft. I have a few comments in mind but think it's a good start.

I do agree with @grynspan's edit to the title of this issue: I think it is the number of test cases of each test function that would be useful to log. The number of arguments is the same for each test case of a given test function and is equal to the number of parameters in that function's signature. It doesn't seem very useful, therefore, to log the total number of arguments which were passed for all the test cases of a test, since it's simply the number of cases multiplied by the number of parameters.

I have a few other suggestions in mind but can put those on the PR itself once you feel it's ready for review.

[hrayatnia]

@stmontgomery Thank you for your comment! I’ve implemented the requested changes.

The reason I kept the PR as a draft is that I’m currently working on familiarizing myself with the perspectives of the main contributors. While I’ve reviewed the code style and proposals, I feel more comfortable starting with smaller, incremental contributions in draft form. This way, I can receive feedback.

Therefore, I’d greatly appreciate any thoughts or feedback you have on the draft as it will help me to align my way of thinking with you.

[mesqueeb]

i personally agree that it's more about the amount of times a single parameterized test ran, not so much about how many params were passed in each cycle. : )