[ZIPT Benchmark] Z3 c3 branch — 2026-03-06

[github-actions[bot]]

QF_S Benchmark Report — Z3 c3 branch

Date: 2026-03-06
Branch: c3
Benchmark set: QF_S (50 randomly selected files from tests/QF_S.tar.zst)
Timeout: 10 seconds per benchmark (-T:10)
Total files in archive: 22,172

---

Summary

Metric	seq solver	nseq solver
sat	33	5
unsat	13	6
unknown	4	39
timeout	0	0
bug/crash	0	0
Total time (s)	18.353	272.689
Avg time/benchmark (s)	0.367	5.454

Soundness disagreements (seq says sat, nseq says unsat or vice versa): 0

---

Per-File Results (click to expand)

#	File	seq verdict	seq time (s)	nseq verdict	nseq time (s)	Notes
1	instance02384.smt2	sat	0.014	unknown	10.007
2	wildcard-matching-regex-09.smt2	unknown	10.008	unknown	0.049
3	instance02372.smt2	sat	0.017	unknown	10.007
4	instance05402.smt2	sat	0.023	unknown	10.007
5	instance02422.smt2	sat	0.018	unknown	10.007
6	instance08064.smt2	sat	0.030	unknown	5.058
7	instance15982.smt2	unsat	0.560	unknown	0.015
8	instance09779.smt2	sat	0.066	unknown	10.008
9	instance08858.smt2	sat	0.012	sat	0.011
10	instance06937.smt2	unsat	0.019	unsat	0.013
11	instance07321.smt2	unsat	0.011	unsat	0.010
12	mux-array_sat_non_incre_equiv_init_0_4.smt2	sat	0.022	unknown	0.015
13	instance11867.smt2	sat	0.011	sat	0.011
14	benchmark_0376.smt2	unknown	0.015	unknown	0.012
15	instance12944.smt2	unsat	0.012	unsat	0.011
16	slog_stranger_543_sink.smt2	unsat	0.011	unsat	0.010
17	instance04168.smt2	sat	0.018	unknown	10.007
18	instance15040.smt2	sat	0.102	unknown	10.007
19	slog_stranger_5400_sink.smt2	sat	0.083	unknown	0.027
20	instance04325.smt2	sat	0.059	unknown	10.007
21	instance12973.smt2	sat	0.461	unknown	0.014
22	instance04314.smt2	sat	0.071	unknown	10.007
23	slog_stranger_4269_sink.smt2	sat	0.023	unknown	0.013
24	unsolved_pcp_instance_278.smt2	unknown	0.014	unknown	10.007
25	instance14053.smt2	sat	0.026	unknown	10.007
26	instance04843.smt2	sat	0.020	sat	0.075
27	instance11840.smt2	unsat	0.032	unknown	10.007
28	instance15874.smt2	sat	0.012	sat	0.011
29	instance04232.smt2	sat	1.183	unknown	0.012
30	instance08465.smt2	unsat	0.012	unsat	0.011
31	instance02842.smt2	sat	0.044	unknown	10.007
32	instance08200.smt2	unsat	0.012	unsat	0.012
33	slog_stranger_5039_sink.smt2	sat	0.055	unknown	0.018
34	instance02643.smt2	sat	0.020	unknown	10.007
35	pcp_instance_269.smt2	unknown	0.014	unknown	10.008
36	instance14310.smt2	unsat	0.019	unknown	0.013
37	instance07434.smt2	unsat	1.678	unknown	10.007
38	query4111.smt2	sat	0.047	unknown	0.011
39	instance11996.smt2	sat	2.809	unknown	10.008
40	instance12308.smt2	unsat	0.025	unknown	10.007
41	instance06366.smt2	sat	0.055	unknown	7.059
42	instance12144.smt2	unsat	0.365	unknown	10.008
43	instance04416.smt2	sat	0.033	unknown	10.007
44	instance13660.smt2	sat	0.012	sat	0.012
45	instance02852.smt2	sat	0.017	unknown	10.007
46	instance08211.smt2	sat	0.065	unknown	10.007
47	instance11133.smt2	unsat	0.022	unknown	10.007
48	instance02850.smt2	sat	0.016	unknown	10.007
49	instance06992.smt2	sat	0.058	unknown	10.007
50	instance03375.smt2	sat	0.022	unknown	10.007

---

Notable Issues

✅ No Soundness Disagreements

Both solvers agreed on all sat/unsat verdicts.

✅ No Crashes

No crashes or errors detected.

🐌 Slow Benchmarks (> 8s)

• instance02384.smt2: nseq=10.007s
• wildcard-matching-regex-09.smt2: seq=10.008s
• instance02372.smt2: nseq=10.007s
• instance05402.smt2: nseq=10.007s
• instance02422.smt2: nseq=10.007s
• instance09779.smt2: nseq=10.008s
• instance04168.smt2: nseq=10.007s
• instance15040.smt2: nseq=10.007s
• instance04325.smt2: nseq=10.007s
• instance04314.smt2: nseq=10.007s
• unsolved_pcp_instance_278.smt2: nseq=10.007s
• instance14053.smt2: nseq=10.007s
• instance11840.smt2: nseq=10.007s
• instance02842.smt2: nseq=10.007s
• instance02643.smt2: nseq=10.007s
• pcp_instance_269.smt2: nseq=10.008s
• instance07434.smt2: nseq=10.007s
• instance11996.smt2: nseq=10.008s
• instance12308.smt2: nseq=10.007s
• instance12144.smt2: nseq=10.008s
• instance04416.smt2: nseq=10.007s
• instance02852.smt2: nseq=10.007s
• instance08211.smt2: nseq=10.007s
• instance11133.smt2: nseq=10.007s
• instance02850.smt2: nseq=10.007s
• instance06992.smt2: nseq=10.007s
• instance03375.smt2: nseq=10.007s

---

Key Observations

nseq solver completeness: The nseq solver returned "unknown" for 39 out of 50 benchmarks (78%), while seq returned "unknown" for only 4 (8%). This suggests the nseq solver has significant completeness issues on these benchmarks.

Performance: The seq solver completed the benchmark suite in 18.353s (avg 0.367s per file), while nseq took 272.689s (avg 5.454s per file). The nseq solver spent substantial time timing out on benchmarks it couldn't solve.

---

Generated automatically by the QF_S Benchmark workflow on the c3 branch.

AI generated by ZIPT Benchmark

• expires on Mar 13, 2026, 5:39 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-13T05:39:52.704Z.

Closed by Workflow