Skip to content

Add manifest file to each each reference test#4712

Merged
leolara merged 6 commits intoethereum:masterfrom
Sandijigs:feature/manifest_file_for_tests_vectors
Dec 4, 2025
Merged

Add manifest file to each each reference test#4712
leolara merged 6 commits intoethereum:masterfrom
Sandijigs:feature/manifest_file_for_tests_vectors

Conversation

@Sandijigs
Copy link
Contributor

Description

 This PR adds a `manifest.yaml` file to every generated test vector. The manifest contains
 the test's metadata (config, fork, runner, handler, suite, and case names) which was
 previously only available by parsing the directory path.
 This makes it easier to distribute individual test vectors without losing context about
 what they're testing. It also helps when refactoring the directory structure since the
 metadata is now embedded in each test case.

Changes Made

 - Added `dump_manifest()` method in `dumper.py` to generate the manifest file
 - Updated `gen_runner.py` to call `dump_manifest()` after writing test outputs
 - Documented the manifest format in `tests/formats/README.md`
  • Update documentation
  • Add tests for new functionality
  • Run make lint to check formatting
  • Run make test to check tests

Fixes #4602

@Sandijigs
Copy link
Contributor Author

Hi @leolara , Please help review the changes made to this issue

@leolara
Copy link
Member

leolara commented Oct 30, 2025

@Sandijigs check the produced manifest.yaml

{config_name: minimal, fork_name: phase0, runner_name: epoch_processing, handler_name: eth1_data_reset,
suite_name: pyspec_tests, case_name: eth1_vote_no_reset}

It should not look like that, but like the data.yaml for example

@Sandijigs
Copy link
Contributor Author

Okay @leolara

@leolara
Copy link
Member

leolara commented Nov 3, 2025

@Sandijigs are you doing it?

@Sandijigs
Copy link
Contributor Author

yes , @leolara . I am currently on it ,

block style instead of flow style
@Sandijigs Sandijigs force-pushed the feature/manifest_file_for_tests_vectors branch from b375022 to 15e12eb Compare November 3, 2025 17:11
@leolara
Copy link
Member

leolara commented Nov 24, 2025

LGTM 🚀

I would perhaps change the README, to indicate that the directory structure is the legacy style of figuring out this data.

@leolara leolara requested a review from jtraglia November 24, 2025 20:08
@jtraglia jtraglia changed the title Add manifest.yaml to each test vector to make metadata self-contained and test cases easier to share and verify Add manifest file to each each reference test Dec 2, 2025
Copy link
Member

@jtraglia jtraglia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some little improvements. LGTM. @leolara feel free to merge/modify as you see fit.

@jtraglia jtraglia added the testing CI, actions, tests, testing infra label Dec 2, 2025
@leolara leolara merged commit 14e6f0c into ethereum:master Dec 4, 2025
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing CI, actions, tests, testing infra

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Manifest file for tests vectors

3 participants