[ZIPT Benchmark] Z3 c3 branch — 2026-03-14

[github-actions[bot]]

Date: 2026-03-14
Branch: c3 (27f5541b0bd518ae3477df3b31ea9b001537ebc7)
Z3 Version: 4.17.0 — 64 bit
Benchmark set: QF_S (50 randomly selected files from tests/QF_S.tar.zst, total pool: 22,172 files)
Timeout: 10 seconds per benchmark (-T:10)

---

Summary

Metric	seq solver	nseq solver
sat	28	5
unsat	16	8
unknown	6	37
timeout	0	0
bug/crash	0	0
Total time (s)	35.484	320.657
Avg time/benchmark (s)	0.710	6.413

Soundness disagreements (seq says sat, nseq says unsat or vice versa): 0 ✅

---

Key Findings

• seq is significantly ahead of nseq on this sample: seq solved 44/50 benchmarks (sat+unsat) vs. nseq's 13/50. The nseq solver timed out (returned unknown after ~10s) on 32 benchmarks that seq solved quickly.
• No soundness bugs detected: There are zero cases where the two solvers disagree on sat vs. unsat — both agree whenever both give a definitive answer.
• nseq is fast when it works: On the 13 benchmarks where nseq found an answer, its average was well under 1s. The issue is coverage, not speed-when-solving.
• seq slow case: slog_stranger_5307_sink.smt2 took 8.717s for seq; instance13753.smt2 and instance14506.smt2 both timed out for seq (10s).
• Benchmark mix: Majority from 20230329-automatark-lu (automata-based), plus a few 2019-Jiang/slog (string log analysis), 20250403-rna (RNA folding), and 20250403-pcp-string (Post Correspondence Problem).

---

Notable Issues

Soundness Disagreements (Critical)

None found. 🎉

Crashes / Bugs

None. 🎉

Slow Benchmarks (> 8s for seq)

File	seq time (s)	seq verdict
slog_stranger_5307_sink.smt2	8.717	sat
instance13753.smt2	10.016	unknown
instance14506.smt2	10.017	unknown

nseq timeouts: benchmarks seq solved but nseq did not (32 files)

These are benchmarks where seq returned sat or unsat quickly but nseq returned unknown after 10s, indicating incomplete coverage in the new solver:

instance00826, instance04922, instance01403, instance08173, instance10859, instance07383, instance06304, instance04240, instance08947, instance08216, instance05830, instance08022, instance04707, instance11370, instance04767, instance04033, instance12274, instance13512, instance09705, instance14182, instance04229, instance00500, instance04918, slog_stranger_1844_sink, instance00239, instance11871, instance01916, instance02721, pcp_instance_443, unsolved_pcp_instance_63, instance07197, slog_stranger_5307_sink

---

Per-File Results

Click to expand full per-file table (50 rows)

#	File	seq verdict	seq time (s)	nseq verdict	nseq time (s)	Notes
1	instance00826.smt2	sat	0.143	unknown	10.008
2	instance04922.smt2	sat	0.489	unknown	10.009
3	instance02637.smt2	sat	0.013	sat	0.012
4	instance02212.smt2	sat	0.011	sat	0.010
5	instance07104.smt2	sat	0.012	sat	0.011
6	instance01403.smt2	sat	0.015	unknown	10.007
7	instance15166.smt2	unsat	0.011	unsat	0.010
8	slog_stranger_5307_sink.smt2	sat	8.717	unknown	0.057
9	slog_stranger_1444_sink.smt2	unsat	0.011	unsat	0.011
10	instance08173.smt2	unsat	0.052	unknown	10.009
11	benchmark_0339.smt2	unknown	0.014	unknown	0.012
12	instance13753.smt2	unknown	10.016	unknown	10.009
13	instance10859.smt2	unsat	0.016	unknown	10.008
14	instance07383.smt2	unsat	0.554	unknown	10.009
15	pcp_instance_443.smt2	unknown	0.013	unknown	10.008
16	instance06304.smt2	sat	0.606	unknown	10.008
17	instance08250.smt2	unsat	0.011	unsat	0.010
18	instance07262.smt2	unsat	0.767	unsat	0.027
19	instance07197.smt2	unsat	0.032	unknown	0.017
20	instance04240.smt2	sat	0.026	unknown	10.009
21	instance08947.smt2	sat	1.335	unknown	10.008
22	instance08216.smt2	unsat	0.018	unknown	10.009
23	instance05830.smt2	sat	0.014	unknown	10.009
24	instance08022.smt2	unsat	0.037	unknown	10.028
25	instance04707.smt2	sat	0.096	unknown	10.009
26	instance11370.smt2	unsat	0.024	unknown	10.008
27	slog_stranger_1251_sink.smt2	sat	1.825	unknown	0.059
28	instance04767.smt2	sat	0.014	unknown	10.009
29	instance15230.smt2	unsat	0.011	unsat	0.010
30	instance11218.smt2	unsat	0.011	unsat	0.011
31	instance00285.smt2	sat	0.012	sat	0.011
32	instance14506.smt2	unknown	10.017	unknown	10.032
33	instance04033.smt2	sat	0.015	unknown	10.009
34	instance12274.smt2	unsat	0.066	unknown	10.008
35	instance09203.smt2	unsat	0.011	unsat	0.011
36	instance03342.smt2	sat	0.011	sat	0.010
37	instance13512.smt2	sat	0.175	unknown	10.009
38	instance09705.smt2	sat	0.071	unknown	10.009
39	slog_stranger_386_sink.smt2	unsat	0.011	unsat	0.011
40	instance14182.smt2	sat	0.017	unknown	10.008
41	instance04229.smt2	sat	0.017	unknown	10.009
42	instance00500.smt2	sat	0.015	unknown	10.007
43	instance04918.smt2	sat	0.022	unknown	10.008
44	slog_stranger_1844_sink.smt2	sat	0.012	unknown	10.042
45	benchmark_0059.smt2	unknown	0.014	unknown	0.012
46	instance00239.smt2	sat	0.023	unknown	10.009
47	instance11871.smt2	sat	0.018	unknown	10.007
48	unsolved_pcp_instance_63.smt2	unknown	0.013	unknown	10.009
49	instance01916.smt2	sat	0.015	unknown	10.007
50	instance02721.smt2	sat	0.015	unknown	10.008

---

Generated automatically by the QF_S Benchmark workflow on the c3 branch.

AI generated by Qf S Benchmark · history

• expires on Mar 21, 2026, 5:42 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Qf S Benchmark.

A newer discussion is available at Discussion #9031.