Skip to content

Add manifest.yaml to each test vector to make metadata self-contained and test cases easier to share and verify #4712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Add manifest.yaml to each test vector to make metadata self-contained and test cases easier to share and verify #4712

wants to merge 2 commits into from

Conversation

@Sandijigs
Copy link

@Sandijigs Sandijigs commented Oct 30, 2025

Description

 This PR adds a `manifest.yaml` file to every generated test vector. The manifest contains
 the test's metadata (config, fork, runner, handler, suite, and case names) which was
 previously only available by parsing the directory path.
 This makes it easier to distribute individual test vectors without losing context about
 what they're testing. It also helps when refactoring the directory structure since the
 metadata is now embedded in each test case.

Changes Made

 - Added `dump_manifest()` method in `dumper.py` to generate the manifest file
 - Updated `gen_runner.py` to call `dump_manifest()` after writing test outputs
 - Documented the manifest format in `tests/formats/README.md`
  • Update documentation
  • Add tests for new functionality
  • Run make lint to check formatting
  • Run make test to check tests

Fixes #4602

@Sandijigs
Copy link
Author

Sandijigs commented Oct 30, 2025

Hi @leolara , Please help review the changes made to this issue

@leolara
Copy link
Member

leolara commented Oct 30, 2025

@Sandijigs check the produced manifest.yaml

{config_name: minimal, fork_name: phase0, runner_name: epoch_processing, handler_name: eth1_data_reset,
suite_name: pyspec_tests, case_name: eth1_vote_no_reset}

It should not look like that, but like the data.yaml for example

@Sandijigs
Copy link
Author

Sandijigs commented Oct 30, 2025

Okay @leolara

@leolara
Copy link
Member

leolara commented Nov 3, 2025

@Sandijigs are you doing it?

@Sandijigs
Copy link
Author

Sandijigs commented Nov 3, 2025

yes , @leolara . I am currently on it ,

@Sandijigs
Add manifest.yaml to each test vector to make metadata self-contained…
8d37e41 
… and test cases easier to share and verify
@Sandijigs
Fix manifest.yaml format to use
15e12eb 
block style instead of flow style
@Sandijigs Sandijigs force-pushed the feature/manifest_file_for_tests_vectors branch from b375022 to 15e12eb Compare November 3, 2025 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Manifest file for tests vectors

2 participants

@Sandijigs @leolara