Website: Implement a check for differing benchmark results and metadata

[jacek-oet]

No description provided.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
solver-benchmark	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 9, 2025 8:24am

[jacek-oet]

@siddharth-krishna CI is failing now because there are some benchmarks that appear in benchmark-result.csv but not in result/meta-data.yaml.

Validation failed for benchmark: pypsa-eur-sec with size: 3-24h
Validation failed for benchmark: pypsa-eur-sec with size: 4-24h
Validation failed for benchmark: pypsa-eur-sec with size: 2-12h
Validation failed for benchmark: pypsa-eur-sec with size: 5-24h
Validation failed for benchmark: pypsa-eur-sec with size: 7-24h
Validation failed for benchmark: pypsa-eur-sec with size: 8-24h
Validation failed for benchmark: pypsa-eur-sec with size: 3-12h
Validation failed for benchmark: pypsa-eur-sec with size: 9-24h
Validation failed for benchmark: pypsa-eur-sec with size: 10-24h
Validation failed for benchmark: pypsa-eur-sec with size: 4-12h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 2-24h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 3-24h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 4-24h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 2-12h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 5-24h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 6-24h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 7-24h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 8-24h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 4-12h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 10-24h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 5-12h
Validation failed for benchmark: pypsa-eur-elec-trex with size: 9-24h
....

I think meta-data.yaml should be updated, or we should remove these missing benchmarks from benchmark-result.csv.

[siddharth-krishna]

Ah yes, nice catch. These are old results (which we are using just for development), I will do a new benchmark run this week that will only run things in the metadata.yaml file so you're right, we should remove the extra ones from the results CSV. Could you do that please? Thanks

[siddharth-krishna]

Sorry Jacek, it looks like there are still some mistakes in which benchmarks were removed. If you don't mind, let me do this quickly and push to your branch.

[siddharth-krishna]

Hmm I didn't expect any results to be modified -- was this intentional? If not, could we revert the modifications and ensure that the diff only deletes rows corresponding to sizes that were removed from results/metadata.yaml? Thanks

[siddharth-krishna]

It feels like too many rows have been deleted...

[siddharth-krishna]

For the record, I used this script to figure out which benchmark results to keep:

import yaml
meta = yaml.safe_load(open("results/metadata.yaml"))
benchs = set()
for b, d in meta['benchmarks'].items():
    for s in d['Sizes']:
        benchs.add((b, s['Name']))

results = open('results/benchmark_results.csv').readlines()
results1 = []
for line in results:
    parts = line.split(',')
    if parts[0] == 'Benchmark' or (parts[0], parts[1]) in benchs:
        results1.append(line)
with open('results/benchmark_results.csv', "w") as f:
    for line in results1:
        f.write(line)
results = open('results/benchmark_results_mean_stddev.csv').readlines()
results1 = []
for line in results:
    parts = line.split(',')
    if parts[0] == 'Benchmark' or (parts[0], parts[1]) in benchs:
        results1.append(line)
len(results), len(results1)
with open('results/benchmark_results_mean_stddev.csv', "w") as f:
    for line in results1:
        f.write(line)