-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Issue:
Currently the runners do not also execute pheval-utils generate-benchmark-stats -r benchmark.yaml
Current-Workflow
The runners are putting results into output/pheval-disease-results/subdir1/subdir2.
subdir1 describes:
- model
- strategy
- nr_of_ppkts
- nr_of_results_per_tsv
- (e.g. ada_wgt_5084ppkts_top10)
subdir2 describes:
- model
- strategy
- collection_name
- nr_of_ppkts,
- nr_of_results_per_tsv
- (e.g. ada_wgt_lrd_hpo_embeddings_5084ppkts_top10)
The reason for the subdir2 is simply if people want to check the performance using different fields to be indexed from the ontology.
Updated Workflow
We want to have the results in output/subdir1/subdir2/pheval-disease-results to be perfectly read by pheval.
So that a benchmark.yaml can be configured and this needs a directory pheval-disease-results where all tsv results are inside.
However, the benchmark.yaml will always need to be structured manually.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels