2025 final execution#216

Merged

bobot merged 97 commits intomasterfrom

2025_final_execution

Aug 11, 2025

Commits on Jul 6, 2025

Add generation of parallel benchmarks and test runs.
Martin Jonáš
committed
Do not use walltime killer anymore.
Martin Jonáš
committed
Use --continue-after-unknown.
Martin Jonáš
committed
Better status reporting.
Martin Jonáš
committed
Show contacts only of competitive solvers.
Martin Jonáš
committed
Generate more meaningful YML names.
Martin Jonáš
committed
Update removed benchmarks.
Martin Jonáš
committed
Remove benchmarks also for non-AWS tracks.
Martin Jonáš
committed
Revert "Do not use walltime killer anymore."
Martin Jonáš
committed

Commits on Jul 7, 2025

Parse the single query/incremental results without scramble_id.
Martin Jonáš
committed

Commits on Jul 23, 2025

bitwuzla: Change options for parallel track. (#201)

mpreiner
authored and
Martin Jonáš
committed
submission: add additional configurations for lower CPU core counts. (#202)

zmylinxi99
authored and
Martin Jonáš
committed
Remove unused AWS infrastructure.
Martin Jonáš
committed
fix: Use LazyFrame during selection of incremental benchmarks.
Martin Jonáš
committed
feat: Do not declare required machines for the parallel track.
Martin Jonáš
committed
feat: Memory limit for parallel track.
Martin Jonáš
committed
Change names of reduced-core parallel solvers.
Martin Jonáš
committed
fix: Correctly recognize out of memory for incremental track.
Martin Jonáš
committed
feat: Adapt parsing of Incremental results to new YML names.
Martin Jonáš
committed
fix: Add nb_answers to all results.
Martin Jonáš
committed
feat: Update year.
Martin Jonáš
committed
feat: Better structure of generated result directories.
Martin Jonáš
committed

Commits on Jul 31, 2025

SMTS submission 2025 update (bug fix + 2 more versions) (#203)

Tomaqa
authored and
Martin Jonáš
committed
feat: Do not fail on missing runs, only report errors.
Martin Jonáš
committed
feat: Adapt unsat core and model result parsing to new YML names.
Martin Jonáš
committed
fix: Unify columns for all tracks to get consistent data set.
Martin Jonáš
committed
fix: Drop benchmark_yml only during result processing.
Martin Jonáš
committed
fix: Initialize unsat core column.
Martin Jonáš
committed
fix: Move previous year.
Martin Jonáš
committed
feat: Allow flushing cache in benchexec conversion.
Martin Jonáš
committed
feat: Model validation with new YML names.
Martin Jonáš
committed
feat: Add 2025 benchmarks
Martin Jonáš
committed
feat: Add preliminary results of all tracks except UnsatCore
Martin Jonáš
committed
fix: Result processing.
Martin Jonáš
committed
fix: Recongize lexing-error error from Dolmen.
Martin Jonáš
committed
fix: Progress computation.
Martin Jonáš
committed
fix: Apply black.
Martin Jonáš
committed
feat: Change unsat core validation resource limits.
Martin Jonáš
committed
feat: New unsat core validation rules.
Martin Jonáš
committed

Commits on Aug 2, 2025

feat: Add preliminary results of UnsatCore track
Martin Jonáš
committed

Commits on Aug 3, 2025

feat: Adapt unsat core validation to new YML format.
Martin Jonáš
committed
fix: Remove join as there are no removed results.
Martin Jonáš
committed
chore: Apply black.
Martin Jonáš
committed
feat: Remove non-competing divisions from results.
Martin Jonáš
committed
fix: Remove debug prints.
Martin Jonáš
committed
feat: Generate results of Parallel track.
Martin Jonáš
committed
feat: Add BestOverall ranking to certificates.
Martin Jonáš
committed
feat: Generate solver data for certificates automatically.
Martin Jonáš
committed
chore: Update year in certificates.
Martin Jonáš
committed

Commits on Aug 4, 2025

feat: Automatically rerun model validation with force_logic=ALL.
Martin Jonáš
committed
feat: Do not use selection data during scoring.
Martin Jonáš
committed
chore: Apply black.
Martin Jonáš
committed
fix: Silence PerformanceWarning.
Martin Jonáš
committed
fix: Different result columns for different tracks.
Martin Jonáš
committed
chore: Apply black.
Martin Jonáš
committed
fix: Correctly store model validation result.
Martin Jonáš
committed
fix: MV results with correct validation statuses.
Martin Jonáš
committed

Commits on Aug 5, 2025

feat: Add results of base solvers.
Martin Jonáš
committed

Commits on Aug 8, 2025

Fix type errors
bobot
committed
2025 webpage (#205)

wintered
authored and
bobot
committed
z3alpha is not in Bitvec, make it so for its base
bobot
committed
Start graphics generation
bobot
committed
[Graph] add altair dependency
bobot
committed
[Graph] Create heatmap
bobot
committed
[Graph] selection by answers
bobot
committed
[Graph] for multiple divisions and logics
bobot
committed
[Graph] Some cleanup
bobot
committed
[Chart] Generate chart on website
bobot
committed
feat: Add 2025 certificate data.
Martin Jonáš
committed
feat: Modernize certificates a little bit.
Martin Jonáš
committed

Commits on Aug 9, 2025

Commits on Aug 10, 2025

chore: Hide link to Zenodo for now.
Martin Jonáš
committed
chore: Display results.
Martin Jonáš
committed
feat: Add news item for the results.
Martin Jonáš
committed
feat: Hide irrelevant columns for each track.
Martin Jonáš
committed
fix: Better number formatting.
Martin Jonáš
committed
Merge branch 'master' into 2025_final_execution
martinjonas
authored

Commits on Aug 11, 2025

initial slide deck for 2025

wintered
authored and
bobot
committed
make slides compile

wintered
authored and
bobot
committed
updated: 2026 organizers and acknowledgments

wintered
authored and
bobot
committed
plan && benchmark contributors

wintered
authored and
bobot
committed
plans for smtcomp 2026

wintered
authored and
bobot
committed
rm cloud track

wintered
authored and
bobot
committed
interim

wintered
authored and
bobot
committed
new participants 2025

wintered
authored and
bobot
committed
adding TODOs to mark unfinished things

wintered
authored and
bobot
committed
fixing #214

wintered
authored and
bobot
committed
interim

wintered
authored and
bobot
committed
updating the certificates

wintered
authored and
bobot
committed
Update slides.

Martin Jonáš
authored and
bobot
committed
Sort solvers.

Martin Jonáš
authored and
bobot
committed
One extra pass through slides.

Martin Jonáš
authored and
bobot
committed
Fix typo.

Martin Jonáš
authored and
bobot
committed
Derived solver
bobot
committed
Fix generate graphics and smtcomp stats-of-benchexec-results data
bobot
committed