macros: append generated test attribute so others are aware of it by kezhuw · Pull Request #6497 · tokio-rs/tokio

kezhuw · 2024-04-19T08:02:55Z

Motivation

Improve compatibility among multiple test proc macros.

Solution

Add #[::core::prelude::v1::test] to the end of test proc macros. This way, proc macros process later can choose to not generate #[::core::prelude::v1::test] to avoid duplicated test runs.
Forbid #[::core::prelude::v1::test] to avoid duplicated test runs as tokio::test is transforming an invalid test signature to one accepted by #[test]. This is consistent with what we currently treat #[test].

Links

This way other test macros can choose to not generate `#[::core::prelude::v1::test]` to avoid duplicated test runs. This commit also rejects existing `#[::core::prelude::v1::test]` just like how and why it rejects existing `#[test]`.

Darksonn · 2024-04-19T08:05:14Z

tokio-macros/Cargo.toml

 proc-macro2 = "1.0.60"
 quote = "1"
-syn = { version = "2.0", features = ["full"] }
+syn = { version = "2.0", features = ["full", "extra-traits"] }


What do you need this feature for? I prefer to avoid adding these features as they hurt compilation times.

impl PartialEq for Attribute. I guess I could duplicate code. Let me figure out.

I see. if it is possible, then try to perform the equality check manually. Thanks!

Darksonn · 2024-04-19T08:06:34Z

tests-build/tests/fail/macros_invalid_input.stderr

+error: second test attribute is supplied, try to order it after `tokio::test`
+  --> $DIR/macros_invalid_input.rs:49:1
+   |
+49 | #[::core::prelude::v1::test::test]
+   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+


What? The #[tokio::test] macro is first, so what do you mean by "after"?

I supposed this to be generated by other preceding test macros, e.g. #[test_log::test]. I guess I should reshape it or keep it origin. Any preference ?

How about "second test attribute is supplied, consider removing or ordering it after tokio::test" ?

My challenge with the wording here is that the code that triggers this error has #[tokio::test] first, and then afterwards, it has a #[::core::prelude::v1::test]. So suggesting that you move #[::core::prelude::v1::test] after #[tokio::test] doesn't make since it's already after #[tokio::test]. Perhaps you could say "before" or "above" instead of "after"?

I know, it is strange. It should be something similar to following.

❯ grep -n -B 2 -A 2 "with_append_test_attribute_and_test_args_and_panic_async" tests/mod.rs 84-#[test_log::test] 85-#[tokio::test] 86:async fn with_append_test_attribute_and_test_args_and_panic_async(x: i8, _y: i8) { 87- assert_eq!(x, 0); 88-} ❯ ❯ cargo test with_append_test_attribute_and_test_args_and_panic_async -- --nocapture error: second test attribute is supplied, consider removing it or ordering it after `tokio::test` --> tests/mod.rs:84:1 | 84 | #[test_log::test] | ^^^^^^^^^^^^^^^^^ | = note: this error originates in the attribute macro `test_log::test` (in Nightly builds, run with -Z macro-backtrace for more info) error: could not compile `test-log` (test "mod") due to 2 previous errors

Moving it "before" or "above" will result in duplicated test runs.

I guess we could:

Introduce dev dependency test-log. This ways the words should be more intuitively. Is it worth ?

Use the short version "second test attribute is supplied".

Other suggestions.

Hi @Darksonn, thank you for reviewing. I have updated this with a fixup commit targeting to your two concerns. Regarding the error words, I am open to any sentence, simply not good at 😮‍💨 .

Maybe just say "consider removing or changing the order of your test attributes".

…ware of it

…s are aware of it

…o others are aware of it

kezhuw · 2024-05-05T05:55:23Z

Hi @Darksonn, thank you for your reviewing, I have solved all review comments. Is there anything else left for this to move forward ? Should I do some squash/rebase work ?

Darksonn

Hi again, thanks for the ping. Sorry about the delay.

Darksonn · 2024-05-05T11:10:35Z

tokio-macros/src/entry.rs

+        syn::Meta::Path(path) => path,
+        _ => return false,
+    };
+    let segments = ["core", "prelude", "v1", "test"];


It's also available as a few other paths such as ::std::prelude::v1::test and ::core::prelude::rust_2021::test.

Would this be too paranoid for us ?

Given that Rust has three editions since v1 and new coming in future, together with combinations from "std", "core" and leading colon, this is a pretty large list.

Given the fact that test is there since day one, I would recommend not doing this. I checked test macros listed below and none use variants other than the two. I don't think test macros should generate other variants, if they do, let them fix it to avoid duplicated test runs but not us.

test-log, test-case, async-std::test, googletest::test uses ::core::prelude::v1::test.

quickcheck, test-strategy, rstest uses #[test]

proptest,tracing_test::traced_test does not generate any

Perhaps we can match ::[std|core]::prelude::*:test?

Updated. Additionally, leading colons are optional now as review comments below.

LGTM. If you can also add some of the variants to the tests, then I think this is ready to merge.

Darksonn · 2024-05-05T11:16:30Z

tests-build/tests/fail/macros_invalid_input.rs

+#[tokio::test]
+#[::core::prelude::v1::test]
+async fn test_has_generated_second_test_attr() {}


What about core without the leading colons?

tokio-macros/src/entry.rs

Darksonn

Thank you.

We decided to bump minimum required version of tokio to 1.34. Currently, the newest tokio version is 1.38, but some of the integration tests are eaten when testing with this specific verstion of tokio. Which is why, as of now, we decided not to support this version. The issue with version 1.38 is related to #[tokio::test] and #[ntest::timeout] attributes. Refs: - tokio-rs/tokio#6610 - becheran/ntest#28 - tokio-rs/tokio#6497

This way test macros following `#[rstest]` can decide whether or not to generate test macro to avoid duplicate test runs. It is an attempt to improve capabilities among test macros. Currently, following test from [googletest](https://github.com/google/googletest-rust/blob/21f2948684847922a416252b8118e3eada8e29d6/integration_tests/src/google_test_with_rstest.rs#L52-L57)(`main` branch at 2025-01-16) will run twice. ```rust #[rstest] #[case(1)] #[gtest] fn paramterised_test_should_work_with_rstest_first(#[case] value: u32) -> Result<()> { verify_that!(value, eq(value)) } ``` See: tokio-rs/tokio#6497, d-e-s-o/test-log#46, frondeus/test-case#143, kezhuw/stuck#53. Refs: rust-lang/rust#67839, rust-lang/rust#82419.

`#[gtest]` will benefit from la10736/rstest#291, we could also benefit other test macros. It is an attempt to improve capabilities among test macros to avoid duplicated test runs which is rare to be aware of. The rationale is simple: procedure of attribute macro see only attributes following it. Macros next to processing macro will not see generated macro if it is generated in-place. So, instead of generating test macro in-place, appending generated test macro will let macros next to processing macro have chance to react, for example, not generate test macro if there is already one. We could deprecate `#[googletest::test]` oneday after rust-lang/rust#82419 released. See: tokio-rs/tokio#6497, d-e-s-o/test-log#46, frondeus/test-case#143, la10736/rstest#291, kezhuw/stuck#53. Refs: rust-lang/rust#67839, rust-lang/rust#82419.

`#[gtest]` will benefit from la10736/rstest#291, we could also benefit other test macros. It is an attempt to improve capabilities among test macros to avoid duplicated test runs which is rare to be aware of. The rationale is simple: procedure of attribute macro see only attributes following it. Macros next to processing macro will not see generated macro if it is generated in-place. So, instead of generating test macro in-place, appending generated test macro will let macros next to processing macro have chance to react, for example, not generate test macro if there is already one. We could deprecate `#[googletest::test]` oneday after la10736/rstest#291 released. See: tokio-rs/tokio#6497, d-e-s-o/test-log#46, frondeus/test-case#143, la10736/rstest#291, kezhuw/stuck#53. Refs: rust-lang/rust#67839, rust-lang/rust#82419.

`#[gtest]` will benefit from la10736/rstest#291, we could also benefit other test macros. This is an attempt to improve capabilities among test macros to avoid duplicated test runs which is rare to be aware of. The rationale is simple: procedure of attribute macro see only attributes following it. Macros next to processing macro will not see generated macro if it is generated in-place. So, instead of generating test macro in-place, appending generated test macro will let macros next to processing macro have chance to react, for example, not generate test macro if there is already one. We could deprecate `#[googletest::test]` oneday after la10736/rstest#291 released. See: tokio-rs/tokio#6497, d-e-s-o/test-log#46, frondeus/test-case#143, la10736/rstest#291, kezhuw/stuck#53. Refs: rust-lang/rust#67839, rust-lang/rust#82419.

This is an attempt to improve capabilities among test macros to avoid duplicated test runs which is rare to be aware of. The rationale is simple: procedure of attribute macro see only attributes following it. Macros next to processing macro will not see generated macro if it is generated in-place. So, instead of generating test macro in-place, appending generated test macro will let macros next to processing macro have chance to react, for example, not generate test macro if there is already one. Without the fix, the new test will run twice. See also tokio-rs/tokio#6497, d-e-s-o/test-log#46, frondeus/test-case#143, la10736/rstest#291, google/googletest-rust#561, kezhuw/stuck#53.

This way we could avoid duplicated test runs if `#[test]` variants already exist See also frondeus/test-case#101, frondeus/test-case#143 and tokio-rs/tokio#6497. Closes d-e-s-o#35.

This way we could avoid duplicated test runs if `#[test]` variants already exist See also frondeus/test-case#101, frondeus/test-case#143 and tokio-rs/tokio#6497. Closes #35.

Darksonn reviewed Apr 19, 2024

View reviewed changes

fixup! macros: append generated test attribute so others are aware of it

81e10af

kezhuw mentioned this pull request Apr 19, 2024

Append #[::core::prelude::v1::test] only if no test macros exist d-e-s-o/test-log#46

Merged

kezhuw added 4 commits April 19, 2024 21:37

fixup! fixup! macros: append generated test attribute so others are a…

90afc4c

…ware of it

fixup! fixup! fixup! macros: append generated test attribute so other…

a9d0609

…s are aware of it

fixup! fixup! fixup! fixup! macros: append generated test attribute s…

a4828d4

…o others are aware of it

align macros_invalid_input.stderr

4f23227

mox692 added the A-tokio-macros Area: The tokio-macros crate label Apr 20, 2024

Darksonn reviewed May 5, 2024

View reviewed changes

kezhuw added 3 commits May 5, 2024 22:29

assert that we append test attribute instead of prepend

d920650

fix mismatch in macros_invalid_input.stderr

51c473c

reject more test attributes

6d2a30f

kezhuw requested a review from Darksonn May 5, 2024 15:45

add test for test path combinations

39c9a1d

Darksonn approved these changes May 5, 2024

View reviewed changes

Merge branch 'master' into test-proc-macros-cooperation

9426fa9

Darksonn enabled auto-merge (squash) May 5, 2024 16:19

Darksonn merged commit 6fcd9c0 into tokio-rs:master May 5, 2024

kezhuw mentioned this pull request May 24, 2024

fix: improve compatibility among test proc macros frondeus/test-case#143

Open

This was referenced May 30, 2024

Prepare tokio-macros v2.3.0 #6600

Merged

chore: prepare Tokio v1.38.0 #6601

Merged

taiki-e mentioned this pull request Jun 1, 2024

The coverage-helper::test doesn't work with tokio::test taiki-e/coverage-helper#2

Closed

mox692 mentioned this pull request Jun 3, 2024

regression: tokio::test in 1.38 eats my tests #6610

Closed

kezhuw mentioned this pull request Jan 16, 2025

Append generated test macro so next test macros are aware of it la10736/rstest#291

Merged

kezhuw mentioned this pull request Jan 20, 2025

Append generated test macro so next test macros are aware of it google/googletest-rust#561

Merged

kezhuw mentioned this pull request Apr 1, 2025

Append generated test macro so next test macros are aware of it frozenlib/test-strategy#17

Merged

Uh oh!

Conversation

kezhuw commented Apr 19, 2024

Motivation

Solution

Links

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Darksonn Apr 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kezhuw commented May 5, 2024

Uh oh!

Darksonn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Darksonn left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Darksonn Apr 19, 2024 •

edited

Loading