Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
90cd20d
Add generation of parallel benchmarks and test runs.
Jul 6, 2025
5452dd7
Do not use walltime killer anymore.
Jul 6, 2025
bca1f08
Use --continue-after-unknown.
Jul 6, 2025
6258316
Better status reporting.
Jul 6, 2025
c635d8e
Show contacts only of competitive solvers.
Jul 6, 2025
680a97a
Generate more meaningful YML names.
Jul 6, 2025
ce5b4e7
Update removed benchmarks.
Jul 6, 2025
e12e8a1
Remove benchmarks also for non-AWS tracks.
Jul 6, 2025
0402477
Revert "Do not use walltime killer anymore."
Jul 6, 2025
8800f20
Parse the single query/incremental results without scramble_id.
Jul 7, 2025
b7c46ac
bitwuzla: Change options for parallel track. (#201)
mpreiner Jul 15, 2025
4fddcda
submission: add additional configurations for lower CPU core counts. …
zmylinxi99 Jul 15, 2025
dde119e
Remove unused AWS infrastructure.
Jul 7, 2025
2c57e14
fix: Use LazyFrame during selection of incremental benchmarks.
Jul 16, 2025
f95717c
feat: Do not declare required machines for the parallel track.
Jul 18, 2025
76076e2
feat: Memory limit for parallel track.
Jul 18, 2025
f75b731
Change names of reduced-core parallel solvers.
Jul 18, 2025
96256c6
fix: Correctly recognize out of memory for incremental track.
Jul 23, 2025
8297d7f
feat: Adapt parsing of Incremental results to new YML names.
Jul 23, 2025
9ebce05
fix: Add nb_answers to all results.
Jul 23, 2025
24a718f
feat: Update year.
Jul 23, 2025
f4beace
feat: Better structure of generated result directories.
Jul 23, 2025
509927e
SMTS submission 2025 update (bug fix + 2 more versions) (#203)
Tomaqa Jul 20, 2025
04d8bff
feat: Do not fail on missing runs, only report errors.
Jul 28, 2025
66c5d56
feat: Adapt unsat core and model result parsing to new YML names.
Jul 28, 2025
fc197d5
fix: Unify columns for all tracks to get consistent data set.
Jul 28, 2025
d79ee2b
fix: Drop benchmark_yml only during result processing.
Jul 28, 2025
3640e5d
fix: Initialize unsat core column.
Jul 28, 2025
af6462e
fix: Move previous year.
Jul 28, 2025
5b7bfd4
feat: Allow flushing cache in benchexec conversion.
Jul 31, 2025
811a16b
feat: Model validation with new YML names.
Jul 31, 2025
b0f0ab3
feat: Add 2025 benchmarks
Jul 31, 2025
090e76e
feat: Add preliminary results of all tracks except UnsatCore
Jul 31, 2025
a0a3b52
fix: Result processing.
Jul 31, 2025
1b48b5d
fix: Recongize lexing-error error from Dolmen.
Jul 31, 2025
6f098c1
fix: Progress computation.
Jul 31, 2025
c28a84b
fix: Apply black.
Jul 31, 2025
faf235c
feat: Change unsat core validation resource limits.
Jul 31, 2025
3abd97e
feat: New unsat core validation rules.
Jul 31, 2025
ba23708
feat: Add preliminary results of UnsatCore track
Aug 2, 2025
12c9977
feat: Adapt unsat core validation to new YML format.
Aug 3, 2025
e1e67d1
fix: Remove join as there are no removed results.
Aug 3, 2025
eb0a5d3
chore: Apply black.
Aug 3, 2025
a17c198
feat: Remove non-competing divisions from results.
Aug 3, 2025
62cf124
fix: Remove debug prints.
Aug 3, 2025
4dd7dfd
feat: Generate results of Parallel track.
Aug 3, 2025
5aef59f
feat: Add BestOverall ranking to certificates.
Aug 3, 2025
424c571
feat: Generate solver data for certificates automatically.
Aug 3, 2025
643a6e7
chore: Update year in certificates.
Aug 3, 2025
81cc622
feat: Automatically rerun model validation with force_logic=ALL.
Aug 4, 2025
064de76
feat: Do not use selection data during scoring.
Aug 4, 2025
a2361fe
chore: Apply black.
Aug 4, 2025
c5a842a
fix: Silence PerformanceWarning.
Aug 4, 2025
7cb2094
fix: Different result columns for different tracks.
Aug 4, 2025
8837b2e
chore: Apply black.
Aug 4, 2025
f6137a9
fix: Correctly store model validation result.
Aug 4, 2025
6219154
fix: MV results with correct validation statuses.
Aug 4, 2025
01c9928
feat: Add results of base solvers.
Aug 5, 2025
2916207
Fix type errors
bobot Aug 7, 2025
09dda9e
2025 webpage (#205)
wintered Aug 7, 2025
0233ec0
z3alpha is not in Bitvec, make it so for its base
bobot Aug 8, 2025
20dc10c
Start graphics generation
bobot Aug 7, 2025
41616bc
[Graph] add altair dependency
bobot Aug 7, 2025
748b931
[Graph] Create heatmap
bobot Aug 7, 2025
5afdde9
[Graph] selection by answers
bobot Aug 7, 2025
e9853b7
[Graph] for multiple divisions and logics
bobot Aug 7, 2025
ef1d7e1
[Graph] Some cleanup
bobot Aug 7, 2025
8d9247a
[Chart] Generate chart on website
bobot Aug 8, 2025
efa712d
feat: Add 2025 certificate data.
Aug 8, 2025
5190d67
feat: Modernize certificates a little bit.
Aug 8, 2025
8b257ae
[Chart] clusterize using the correlation
bobot Aug 9, 2025
0f190ca
keep legend near the graph
bobot Aug 9, 2025
88f3c0b
[Chart] select with stroke instead of opacity for solvers
bobot Aug 9, 2025
2f20976
chore: Hide link to Zenodo for now.
Aug 10, 2025
0a8965a
chore: Display results.
Aug 10, 2025
a800be8
feat: Add news item for the results.
Aug 10, 2025
e7e731a
feat: Hide irrelevant columns for each track.
Aug 10, 2025
1fca44c
fix: Better number formatting.
Aug 10, 2025
3a83046
Merge branch 'master' into 2025_final_execution
martinjonas Aug 10, 2025
09df5eb
initial slide deck for 2025
wintered Aug 6, 2025
a225d9a
make slides compile
wintered Aug 6, 2025
c6389c0
updated: 2026 organizers and acknowledgments
wintered Aug 6, 2025
ced83c6
plan && benchmark contributors
wintered Aug 6, 2025
6e0627f
plans for smtcomp 2026
wintered Aug 6, 2025
978dfa7
rm cloud track
wintered Aug 6, 2025
c4c6557
interim
wintered Aug 7, 2025
f896cf7
new participants 2025
wintered Aug 7, 2025
75b5822
adding TODOs to mark unfinished things
wintered Aug 8, 2025
a15d46e
fixing #214
wintered Aug 9, 2025
7b48b7e
interim
wintered Aug 9, 2025
6d4fc38
updating the certificates
wintered Aug 9, 2025
8183638
Update slides.
Aug 9, 2025
45705e2
Sort solvers.
Aug 9, 2025
0cd5a9f
One extra pass through slides.
Aug 10, 2025
dcc34f7
Fix typo.
Aug 10, 2025
728eafe
Derived solver
bobot Aug 11, 2025
22d1b1c
Fix generate graphics and smtcomp stats-of-benchexec-results data
bobot Aug 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 25 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
.PHONY: install
install: ## Install the poetry environment and install the pre-commit hooks
install: ## Install the poetry environment
@echo "🚀 Creating virtual environment using pyenv and poetry"
@poetry install
@poetry shell

.PHONY: check
check: ## Run code quality tools.
Expand All @@ -20,7 +19,7 @@ test: generation ## Test the code with pytest
@echo "🚀 Testing code: Running pytest"
@poetry run pytest

generation: submission-generation participant-data track-data division-track-data results-generation ## Files generation for the website
generation: submission-generation participant-data track-data division-track-data results-generation charts-generation ## Files generation for the website

.PHONY: build
build: clean-build ## Build wheel file using poetry
Expand All @@ -31,6 +30,12 @@ build: clean-build ## Build wheel file using poetry
clean-build: ## clean build artifacts
@rm -rf dist

.PHONY: clean-web-results
clean-web-results: ## clean web results
@rm -rf web/content/results
@rm -rf web/public/results


.PHONY: help
help:
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
Expand Down Expand Up @@ -76,8 +81,23 @@ results-generation:
@poetry run smtcomp export-results-pages data Incremental
# @echo "🚀 Generating results to web/content/results for Cloud"
# @poetry run smtcomp export-results-pages data Cloud
# @echo "🚀 Generating results to web/content/results for Parallel"
# @poetry run smtcomp export-results-pages data Parallel
@echo "🚀 Generating results to web/content/results for Parallel"
@poetry run smtcomp export-results-pages data Parallel

charts-generation:
@echo "🚀 Generating results to web/content/results for SingleQuery"
@poetry run smtcomp generate-website-graphics data SingleQuery
@echo "🚀 Generating results to web/content/results for ModelValidation"
@poetry run smtcomp generate-website-graphics data ModelValidation
@echo "🚀 Generating results to web/content/results for UnsatCore"
@poetry run smtcomp generate-website-graphics data UnsatCore
@echo "🚀 Generating results to web/content/results for Incremental"
@poetry run smtcomp generate-website-graphics data Incremental
# @echo "🚀 Generating results to web/content/results for Cloud"
# @poetry run smtcomp generate-website-graphics data Cloud
@echo "🚀 Generating results to web/content/results for Parallel"
@poetry run smtcomp generate-website-graphics data Parallel


cache:
@echo "🚀 Generating cache"
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ The step from Single Query can be followed with the model validation results whe
Tasks for unsat core validation can be generated by:

```
smtcomp generate-unsatcore-validation-files ../tmp/execution SCRAMBLER_EXECUTABLE UNSAT_CORE_RESULT_DIR/*/*
smtcomp generate-unsatcore-validation-files ../tmp/execution SCRAMBLER_EXECUTABLE UNSAT_CORE_RESULT_DIR
```

---
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added data/2025/slides/future/Google-Cloud-Emblem.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added data/2025/slides/future/Google__G__Logo.svg.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added data/2025/slides/future/Microsoft-Azure.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added data/2025/slides/future/You've_got_mail.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added data/2025/slides/future/iowa_university.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
30 changes: 30 additions & 0 deletions data/2025/slides/future/prover.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"name": "bub4",
"authors": ["you"],
"logics": {
"sq": ["UF"],
"uc": "^QF_(LRA)?(UF)?$",
"mv": {
"theories": ["QF", "+-UF", "+-BV", "+-A"],
"except": ["QF_UFBVA"],
"logics": ["QF_LRA"]
}
},
"versions": [
{
"name": "10/05/24",
"url": "http://...",
"sha256": "123434ac",
"competing": true,
"cmd": {
"mv": ["bub4", "--model"]
}
},
{
"name": "fixed",
"url": "http://...",
"sha256": "123434ac",
"competing": false
}
]
}
Loading