Skip to content

Warn if frameworks exceed the maximum number of test mutations #9667

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Warn if frameworks exceed the maximum number of test mutations #9667

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions toolset/utils/metadata.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,18 @@ def parse_config(self, config, directory):
% config['framework'],
color=Fore.YELLOW)

# Check that each framework does not have more than the maximum number of tests
maximum_tests = 10
non_broken_tests_filter = lambda test: (not hasattr(test, "tags")) or ("broken" not in test.tags)
non_broken_tests_to_run = list(filter(non_broken_tests_filter, tests_to_run))
if len(non_broken_tests_to_run) > maximum_tests:
message = [
"Framework %s defines %s tests in benchmark_config.json (max is %s)."
% (config['framework'], len(non_broken_tests_to_run), maximum_tests),
"Contact maintainers and remove deprecated or discarded ones to make room."
]
log("\n".join(message), color=Fore.YELLOW)

# Check that each test configuration is acceptable
# Throw exceptions if a field is missing, or how to improve the field
for test_name, test_keys in test.items():
Expand Down
Loading