Skip to content

Testsuite Rework & Preview#163

Merged
wucke13 merged 11 commits intomainfrom
dev/flo-testsuite
May 23, 2025
Merged

Testsuite Rework & Preview#163
wucke13 merged 11 commits intomainfrom
dev/flo-testsuite

Conversation

@florianhartung
Copy link
Copy Markdown
Collaborator

@florianhartung florianhartung commented May 20, 2025

Pull Request Overview

This pull request adds the testsuite rework and preview, as described in #150 and related PRs.
It is based on #147.

Testing Strategy

This pull request was tested by using it locally in my fork. For example see florianhartung#3.

Formatting

  • Ran cargo fmt
  • Ran cargo check
  • Ran cargo build
  • Ran cargo doc
  • Ran nix fmt
  • Ran nix flake check

florianhartung and others added 4 commits May 20, 2025 15:31
@florianhartung
fix: interpreter typos
54dc00f 
Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung
fix: catch unwind in direct invoke directive
11ca6d1 
Signed-off-by: George Cosma <george.cosma@oxidos.io>
Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung
refactor: move testsuite file filters and find function to seperate m…
6d52d29 
…odule

Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung
feat: testsuite rework
693f065 
Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung florianhartung requested review from cemonem and wucke13 May 20, 2025 14:41
@florianhartung florianhartung self-assigned this May 20, 2025
@florianhartung florianhartung mentioned this pull request May 20, 2025
Closed
@florianhartung florianhartung mentioned this pull request May 20, 2025
Closed
5 tasks
@codecov
Copy link
Copy Markdown

codecov bot commented May 20, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

see 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@florianhartung
Copy link
Copy Markdown
Collaborator Author

florianhartung commented May 20, 2025 •
edited
Loading

Note that the Nix build wasm-interpreter and report action was extremely slow (~17min).
I am not sure why this is the case...

@florianhartung
Copy link
Copy Markdown
Collaborator Author

florianhartung commented May 21, 2025 •
edited
Loading

This might be the same issue: taiki-e/cargo-llvm-cov#376.
I suspect this is happening because of the newly introduced serde & serde_json dependencies. (the issue persists after removal of serde dependencies)

Some measurements taken locally:
cargo test in CLI: spec_tests takes 4.89s
cargo llvm-cov --no-report test via nix build postCheck: spec_tests takes 97.53s
cargo llvm-cov --no-report nextest via nix build postCheck: spec_tests takes 428.892s

@florianhartung florianhartung force-pushed the dev/flo-testsuite branch 2 times, most recently from 2721e91 to 630356f Compare May 22, 2025 11:50
@florianhartung florianhartung mentioned this pull request May 22, 2025
Merged
5 tasks
cemonem
cemonem reviewed May 22, 2025
View reviewed changes
cemonem
cemonem previously approved these changes May 22, 2025
View reviewed changes
florianhartung and others added 6 commits May 22, 2025 16:12
@florianhartung @cemonem
feat: testsuite preview
745b761 
Signed-off-by: Florian Hartung <florian.hartung@dlr.de>

Co-authored-by: Cem Önem <a.cemonem@gmail.com>
@florianhartung
feat: add nix command to generate testsuite report
84382e0 
Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung
fix: serialize ci reports directly to file
5db9597 
Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung
fix: improve wast file finder
5a9ecc4 
Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung
fix: use panic handler in specification test only when required
b38d664 
Previously, the panic handler was set to do nothing at the start of the
test so that panics in the interpreter do not emit backtraces to STDOUT.
This prevents us from having actual useful panic backtraces, which result
from bugs in the specification test runner itself.
This commit reduces the time where no panic handler as much as possible
by removing it temporarily only when calls to the interpreter are made.

Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung
fix: various improvements for specification tests
ffbcda7 
Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung
refactor: run cargo clippy --fix on specification test
8da67bd 
Signed-off-by: Florian Hartung <florian.hartung@dlr.de>
@florianhartung florianhartung requested a review from cemonem May 22, 2025 14:13
@wucke13 wucke13 added this pull request to the merge queue May 23, 2025
Merged via the queue into main with commit 3b2aa91 May 23, 2025
6 checks passed
@wucke13 wucke13 deleted the dev/flo-testsuite branch May 23, 2025 13:14
@florianhartung florianhartung mentioned this pull request May 26, 2025
Closed
@cemonem cemonem mentioned this pull request Nov 20, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cemonem cemonem cemonem approved these changes

@wucke13 wucke13 wucke13 approved these changes

Assignees

@florianhartung florianhartung

Labels

priority-high priority-medium tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@florianhartung @cemonem @wucke13