Excute E2E tests in parallel (#7456)

ironcev · web-flow · commit 97efa221f0cc · 2025-10-14T10:22:11.000+11:00
## Description This PR implements parallel execution of E2E tests by using the approach similar to one introduced by @JoshuaBatty for running LSP garbage collection tests in parallel. The approach is based on: - `test` binary getting an additional `--exact` option used by the parallel runner to run exactly one test passed as an argument to `--exact`. The argument is the full path to the test's `toml` file. - parallel runner that uses `rayon` and `std::process::Command` to span `test --exact <test toml path>` processes in parallel. The original command line argument passed to `test` get forwarded to `test --exact` only if they are applicable while running tests in parallel. On my laptop with 20 cores, **the execution time of the complete E2E test suite reduces from reproducible 10:48 minutes to reproducible 1:47 minutes**. Parallel execution is set to be the default one. Sequential execution can still be used via `--sequential` or `-s` flag: `cargo run -- -s` or `test -s`. Similar to parallel garbage collection tests, the parallel execution of E2E tests shows `stderr` output of the executed tests, but not the `stdout`. Being non-deterministic, the position of displayed errors will not come immediately after or before the failing test display, but will always be easy to relate to a concrete failing test. In practice, this will be perfectly enough for troubleshooting failing tests. If all the output is needed during troubleshooting, failing tests can always be executed sequentially, by using the `--sequential` or `-s` flag. Additionally the PR: - adds `no_output` flag to `BuildOpts` to instructs `forc-pkg` not to output build artifacts like bytecode, ABI JSON, or storage slots JSON. This improves test execution speed in general because it skips writing to disc artifacts that are anyhow not used in tests. Also, it simplifies parallelization, because in case of tests with several `test.<feature>.toml` files, we don't need to worry about racing conditions when writing output files. If needed for troubleshooting, output can still be optionally generated in tests by passing the newly added `--write-output` CLI flag to the `test` binary. This flag is ignored when running tests in parallel. A support for the `no_output` option is not added neither to `forc` CLI, nor to CLI of any other tools like, e.g., `forc deploy`. If it proves useful to those tools, we can always easily add it to their CLIs. - fixes #7449 by annotating the `test` binary `main` function with `#[tokio::main(flavor = "current_thread")]`. - renames `exclude_std` CLI option to `no_std_only` to follow the `_only` pattern in naming filters in `FilterConfig` that filter out only the tests with a certain property. Implementing parallel execution for snapshot and IR tests (if required) will be done in followup PRs. ## Checklist - [x] I have linked to any relevant issues. - [x] I have commented my code, particularly in hard-to-understand areas. - [ ] I have updated the documentation where relevant (API docs, the reference, and the Sway book). - [ ] If my change requires substantial documentation changes, I have [requested support from the DevRel team](https://github.com/FuelLabs/devrel-requests/issues/new/choose) - [ ] I have added tests that prove my fix is effective or that my feature works. - [ ] I have added (or requested a maintainer to add) the necessary `Breaking*` or `New Feature` labels where relevant. - [x] I have done my best to ensure that my PR adheres to [the Fuel Labs Code Review Standards](https://github.com/FuelLabs/rfcs/blob/master/text/code-standards/external-contributors.md). - [x] I have requested a review from the relevant team or maintainers.
diff --git a/.gitignore b/.gitignore
@@ -1,9 +1,9 @@
 # Generated by Cargo
 # will have compiled files and executables
 **/*/target/
-**/*/json_abi_output.json
+**/*/json_abi_output*.json
 **/*/json_abi_output_flat.json
-**/*/json_storage_slots_output.json
+**/*/json_storage_slots_output*.json
 target
 
 # These are backup files generated by rustfmt
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/forc-pkg/src/pkg.rs b/forc-pkg/src/pkg.rs
@@ -323,6 +323,8 @@ pub struct BuildOpts {
     pub experimental: Vec<sway_features::Feature>,
     /// Set of disabled experimental flags
     pub no_experimental: Vec<sway_features::Feature>,
+    /// Do not output any build artifacts, e.g., bytecode, ABI JSON, etc.
+    pub no_output: bool,
 }
 
 /// The set of options to filter type of projects to build in a workspace.
@@ -2210,6 +2212,7 @@ pub fn build_with_options(
         member_filter,
         experimental,
         no_experimental,
+        no_output,
         ..
     } = &build_options;
 
@@ -2301,7 +2304,10 @@ pub fn build_with_options(
             built_package.write_hexcode(&hexfile_path)?;
         }
 
-        built_package.write_output(minify, &pkg_manifest.project.name, &output_dir)?;
+        if !no_output {
+            built_package.write_output(minify, &pkg_manifest.project.name, &output_dir)?;
+        }
+
         built_workspace.push(Arc::new(built_package));
     }
 
diff --git a/forc-plugins/forc-client/src/op/deploy.rs b/forc-plugins/forc-client/src/op/deploy.rs
@@ -867,6 +867,7 @@ fn build_opts_from_cmd(cmd: &cmd::Deploy, member_filter: pkg::MemberFilter) -> p
         member_filter,
         experimental: cmd.experimental.experimental.clone(),
         no_experimental: cmd.experimental.no_experimental.clone(),
+        no_output: false,
     }
 }
 
diff --git a/forc-plugins/forc-client/src/op/run/mod.rs b/forc-plugins/forc-client/src/op/run/mod.rs
@@ -339,5 +339,6 @@ fn build_opts_from_cmd(cmd: &cmd::Run) -> pkg::BuildOpts {
         member_filter: pkg::MemberFilter::only_scripts(),
         experimental: cmd.experimental.experimental.clone(),
         no_experimental: cmd.experimental.no_experimental.clone(),
+        no_output: false,
     }
 }
diff --git a/forc-test/src/lib.rs b/forc-test/src/lib.rs
@@ -164,6 +164,8 @@ pub struct TestOpts {
     pub experimental: Vec<sway_features::Feature>,
     /// Set of disabled experimental flags
     pub no_experimental: Vec<sway_features::Feature>,
+    /// Do not output any build artifacts, e.g., bytecode, ABI JSON, etc.
+    pub no_output: bool,
 }
 
 /// The set of options provided for controlling logs printed for each test.
@@ -468,6 +470,7 @@ impl From<TestOpts> for pkg::BuildOpts {
             member_filter: Default::default(),
             experimental: val.experimental,
             no_experimental: val.no_experimental,
+            no_output: val.no_output,
         }
     }
 }
@@ -494,6 +497,7 @@ impl TestOpts {
             member_filter: Default::default(),
             experimental: self.experimental,
             no_experimental: self.no_experimental,
+            no_output: self.no_output,
         }
     }
 }
diff --git a/forc/src/cli/commands/test.rs b/forc/src/cli/commands/test.rs
@@ -370,6 +370,7 @@ fn opts_from_cmd(cmd: Command) -> forc_test::TestOpts {
         build_target: cmd.build.build_target,
         experimental: cmd.experimental.experimental,
         no_experimental: cmd.experimental.no_experimental,
+        no_output: false,
     }
 }
 
diff --git a/forc/src/ops/forc_build.rs b/forc/src/ops/forc_build.rs
@@ -50,5 +50,6 @@ fn opts_from_cmd(cmd: BuildCommand) -> pkg::BuildOpts {
         member_filter: MemberFilter::default(),
         experimental: cmd.experimental.experimental,
         no_experimental: cmd.experimental.no_experimental,
+        no_output: false,
     }
 }
diff --git a/forc/src/ops/forc_contract_id.rs b/forc/src/ops/forc_contract_id.rs
@@ -82,5 +82,6 @@ fn build_opts_from_cmd(cmd: &ContractIdCommand) -> pkg::BuildOpts {
         member_filter: pkg::MemberFilter::only_contracts(),
         experimental: cmd.experimental.experimental.clone(),
         no_experimental: cmd.experimental.no_experimental.clone(),
+        no_output: false,
     }
 }
diff --git a/forc/src/ops/forc_predicate_root.rs b/forc/src/ops/forc_predicate_root.rs
@@ -51,5 +51,6 @@ fn build_opts_from_cmd(cmd: PredicateRootCommand) -> pkg::BuildOpts {
         member_filter: pkg::MemberFilter::only_predicates(),
         experimental: cmd.experimental.experimental,
         no_experimental: cmd.experimental.no_experimental,
+        no_output: false,
     }
 }
diff --git a/sway-core/src/build_config.rs b/sway-core/src/build_config.rs
@@ -82,6 +82,10 @@ impl PrintAsm {
         }
     }
 
+    pub fn none() -> Self {
+        Self::default()
+    }
+
     pub fn abstract_virtual() -> Self {
         Self {
             virtual_abstract: true,
@@ -146,6 +150,10 @@ impl PrintIr {
         }
     }
 
+    pub fn none() -> Self {
+        Self::default()
+    }
+
     pub fn r#final() -> Self {
         Self {
             r#final: true,
diff --git a/sway-features/src/lib.rs b/sway-features/src/lib.rs
@@ -186,6 +186,30 @@ pub struct CliFields {
     pub no_experimental: Vec<Feature>,
 }
 
+impl CliFields {
+    pub fn experimental_as_cli_string(&self) -> Option<String> {
+        Self::features_as_cli_string(&self.experimental)
+    }
+
+    pub fn no_experimental_as_cli_string(&self) -> Option<String> {
+        Self::features_as_cli_string(&self.no_experimental)
+    }
+
+    fn features_as_cli_string(features: &[Feature]) -> Option<String> {
+        if features.is_empty() {
+            None
+        } else {
+            Some(
+                features
+                    .iter()
+                    .map(|f| f.name())
+                    .collect::<Vec<_>>()
+                    .join(","),
+            )
+        }
+    }
+}
+
 #[derive(Debug)]
 pub enum Error {
     ParseError(String),
diff --git a/test/Cargo.toml b/test/Cargo.toml
@@ -28,6 +28,7 @@ libtest-mimic.workspace = true
 normalize-path.workspace = true
 prettydiff.workspace = true
 rand.workspace = true
+rayon.workspace = true
 regex.workspace = true
 revm.workspace = true
 rhai = "1.22.2"
diff --git a/test/src/e2e_vm_tests/harness.rs b/test/src/e2e_vm_tests/harness.rs
@@ -291,6 +291,7 @@ pub(crate) async fn compile_to_bytes(
         },
         experimental: run_config.experimental.experimental.clone(),
         no_experimental: run_config.experimental.no_experimental.clone(),
+        no_output: !run_config.write_output,
         ..Default::default()
     };
 
@@ -351,6 +352,7 @@ pub(crate) async fn compile_and_run_unit_tests(
                     ..Default::default()
                 },
                 build_target: run_config.build_target,
+                no_output: !run_config.write_output,
                 ..Default::default()
             })
         }) {
@@ -378,7 +380,6 @@ pub(crate) fn test_json_abi(
     has_experimental_field: bool,
     is_release: bool,
 ) -> Result<()> {
-    emit_json_abi(file_name, built_package)?;
     let manifest_dir = env!("CARGO_MANIFEST_DIR");
 
     let experimental_suffix = match (has_experimental_field, experimental_new_encoding) {
@@ -403,10 +404,15 @@ pub(crate) fn test_json_abi(
     );
 
     let output_path = format!(
-        "{}/src/e2e_vm_tests/test_programs/{}/{}",
-        manifest_dir, file_name, "json_abi_output.json"
+        "{}/src/e2e_vm_tests/test_programs/{}/json_abi_output{}.{}.json",
+        manifest_dir,
+        file_name,
+        experimental_suffix,
+        if is_release { "release" } else { "debug" },
     );
 
+    emit_json_abi(file_name, &output_path, built_package)?;
+
     // Update the oracle failing silently
     if update_output_files {
         let _ = std::fs::copy(&output_path, &oracle_path);
@@ -439,20 +445,19 @@ pub(crate) fn test_json_abi(
     Ok(())
 }
 
-fn emit_json_abi(file_name: &str, built_package: &BuiltPackage) -> Result<()> {
-    tracing::info!("ABI gen {} ...", file_name.bold());
+fn emit_json_abi(
+    file_name: &str,
+    json_abi_output_path: &str,
+    built_package: &BuiltPackage,
+) -> Result<()> {
+    tracing::info!("ABI JSON gen {} ...", file_name.bold());
     let json_abi = match &built_package.program_abi {
         ProgramABI::Fuel(abi) => serde_json::json!(abi),
         ProgramABI::Evm(abi) => serde_json::json!(abi),
         ProgramABI::MidenVM(_) => todo!(),
     };
-    let manifest_dir = env!("CARGO_MANIFEST_DIR");
-    let file = std::fs::File::create(format!(
-        "{}/src/e2e_vm_tests/test_programs/{}/{}",
-        manifest_dir, file_name, "json_abi_output.json"
-    ))?;
-    let res = serde_json::to_writer_pretty(&file, &json_abi);
-    res?;
+    let file = std::fs::File::create(json_abi_output_path)?;
+    serde_json::to_writer_pretty(&file, &json_abi)?;
     Ok(())
 }
 
@@ -461,25 +466,29 @@ pub(crate) fn test_json_storage_slots(
     built_package: &BuiltPackage,
     suffix: &Option<String>,
 ) -> Result<()> {
-    emit_json_storage_slots(file_name, built_package)?;
     let manifest_dir = env!("CARGO_MANIFEST_DIR");
+
+    let experimental_suffix = suffix
+        .as_ref()
+        .unwrap()
+        .strip_prefix("test")
+        .unwrap()
+        .strip_suffix("toml")
+        .unwrap()
+        .trim_end_matches('.');
+
     let oracle_path = format!(
-        "{}/src/e2e_vm_tests/test_programs/{}/json_storage_slots_oracle.{}json",
-        manifest_dir,
-        file_name,
-        suffix
-            .as_ref()
-            .unwrap()
-            .strip_prefix("test")
-            .unwrap()
-            .strip_suffix("toml")
-            .unwrap()
-            .trim_start_matches('.')
+        "{}/src/e2e_vm_tests/test_programs/{}/json_storage_slots_oracle{}.json",
+        manifest_dir, file_name, experimental_suffix,
     );
+
     let output_path = format!(
-        "{}/src/e2e_vm_tests/test_programs/{}/{}",
-        manifest_dir, file_name, "json_storage_slots_output.json"
+        "{}/src/e2e_vm_tests/test_programs/{}/json_storage_slots_output{}.json",
+        manifest_dir, file_name, experimental_suffix,
     );
+
+    emit_json_storage_slots(file_name, &output_path, built_package)?;
+
     if fs::metadata(oracle_path.clone()).is_err() {
         bail!("JSON storage slots oracle file does not exist for this test.\nExpected oracle path: {}", &oracle_path);
     }
@@ -501,15 +510,14 @@ pub(crate) fn test_json_storage_slots(
     Ok(())
 }
 
-fn emit_json_storage_slots(file_name: &str, built_package: &BuiltPackage) -> Result<()> {
+fn emit_json_storage_slots(
+    file_name: &str,
+    json_storage_slots_output_path: &str,
+    built_package: &BuiltPackage,
+) -> Result<()> {
     tracing::info!("Storage slots JSON gen {} ...", file_name.bold());
     let json_storage_slots = serde_json::json!(built_package.storage_slots);
-    let manifest_dir = env!("CARGO_MANIFEST_DIR");
-    let file = std::fs::File::create(format!(
-        "{}/src/e2e_vm_tests/test_programs/{}/{}",
-        manifest_dir, file_name, "json_storage_slots_output.json"
-    ))?;
-    let res = serde_json::to_writer_pretty(&file, &json_storage_slots);
-    res?;
+    let file = std::fs::File::create(json_storage_slots_output_path)?;
+    serde_json::to_writer_pretty(&file, &json_storage_slots)?;
     Ok(())
 }
diff --git a/test/src/e2e_vm_tests/mod.rs b/test/src/e2e_vm_tests/mod.rs
diff --git a/test/src/e2e_vm_tests/util.rs b/test/src/e2e_vm_tests/util.rs
diff --git a/test/src/main.rs b/test/src/main.rs

Original file line number	Diff line number	Diff line change
`@@ -867,6 +867,7 @@ fn build_opts_from_cmd(cmd: &cmd::Deploy, member_filter: pkg::MemberFilter) -> p`
`867`	`867`	`member_filter,`
`868`	`868`	`experimental: cmd.experimental.experimental.clone(),`
`869`	`869`	`no_experimental: cmd.experimental.no_experimental.clone(),`
	`870`	`+ no_output: false,`
`870`	`871`	`}`
`871`	`872`	`}`
`872`	`873`
Original file line number	Diff line number	Diff line change
`@@ -339,5 +339,6 @@ fn build_opts_from_cmd(cmd: &cmd::Run) -> pkg::BuildOpts {`
`339`	`339`	`member_filter: pkg::MemberFilter::only_scripts(),`
`340`	`340`	`experimental: cmd.experimental.experimental.clone(),`
`341`	`341`	`no_experimental: cmd.experimental.no_experimental.clone(),`
	`342`	`+ no_output: false,`
`342`	`343`	`}`
`343`	`344`	`}`
Original file line number	Diff line number	Diff line change
`@@ -164,6 +164,8 @@ pub struct TestOpts {`
`164`	`164`	`pub experimental: Vec<sway_features::Feature>,`
`165`	`165`	`/// Set of disabled experimental flags`
`166`	`166`	`pub no_experimental: Vec<sway_features::Feature>,`
	`167`	`+ /// Do not output any build artifacts, e.g., bytecode, ABI JSON, etc.`
	`168`	`+ pub no_output: bool,`
`167`	`169`	`}`
`168`	`170`
`169`	`171`	`/// The set of options provided for controlling logs printed for each test.`
`@@ -468,6 +470,7 @@ impl From<TestOpts> for pkg::BuildOpts {`
`468`	`470`	`member_filter: Default::default(),`
`469`	`471`	`experimental: val.experimental,`
`470`	`472`	`no_experimental: val.no_experimental,`
	`473`	`+ no_output: val.no_output,`
`471`	`474`	`}`
`472`	`475`	`}`
`473`	`476`	`}`
`@@ -494,6 +497,7 @@ impl TestOpts {`
`494`	`497`	`member_filter: Default::default(),`
`495`	`498`	`experimental: self.experimental,`
`496`	`499`	`no_experimental: self.no_experimental,`
	`500`	`+ no_output: self.no_output,`
`497`	`501`	`}`
`498`	`502`	`}`
`499`	`503`	`}`
Original file line number	Diff line number	Diff line change
`@@ -370,6 +370,7 @@ fn opts_from_cmd(cmd: Command) -> forc_test::TestOpts {`
`370`	`370`	`build_target: cmd.build.build_target,`
`371`	`371`	`experimental: cmd.experimental.experimental,`
`372`	`372`	`no_experimental: cmd.experimental.no_experimental,`
	`373`	`+ no_output: false,`
`373`	`374`	`}`
`374`	`375`	`}`
`375`	`376`
Original file line number	Diff line number	Diff line change
`@@ -50,5 +50,6 @@ fn opts_from_cmd(cmd: BuildCommand) -> pkg::BuildOpts {`
`50`	`50`	`member_filter: MemberFilter::default(),`
`51`	`51`	`experimental: cmd.experimental.experimental,`
`52`	`52`	`no_experimental: cmd.experimental.no_experimental,`
	`53`	`+ no_output: false,`
`53`	`54`	`}`
`54`	`55`	`}`
Original file line number	Diff line number	Diff line change
`@@ -82,5 +82,6 @@ fn build_opts_from_cmd(cmd: &ContractIdCommand) -> pkg::BuildOpts {`
`82`	`82`	`member_filter: pkg::MemberFilter::only_contracts(),`
`83`	`83`	`experimental: cmd.experimental.experimental.clone(),`
`84`	`84`	`no_experimental: cmd.experimental.no_experimental.clone(),`
	`85`	`+ no_output: false,`
`85`	`86`	`}`
`86`	`87`	`}`
Original file line number	Diff line number	Diff line change
`@@ -51,5 +51,6 @@ fn build_opts_from_cmd(cmd: PredicateRootCommand) -> pkg::BuildOpts {`
`51`	`51`	`member_filter: pkg::MemberFilter::only_predicates(),`
`52`	`52`	`experimental: cmd.experimental.experimental,`
`53`	`53`	`no_experimental: cmd.experimental.no_experimental,`
	`54`	`+ no_output: false,`
`54`	`55`	`}`
`55`	`56`	`}`