Skip to content

differential-testing.md

Latest commit

 

History

History
57 lines (37 loc) · 1.47 KB

differential-testing.md

File metadata and controls

57 lines (37 loc) · 1.47 KB

Differential Correctness and Fuzzing Workflow

This project includes a differential gate that compares interpreter behavior (tonic run <source>) against native AOT behavior (tonic compile <source> + direct executable run).

Run the gate

cargo test --test differential_backends -- --nocapture

This test target runs two suites:

  1. Parity subset differential over curated active fixtures from examples/parity/catalog.toml.
  2. Seeded generator differential over deterministic generated programs.

Reproducible fuzz seeds

Run a single seed:

TONIC_DIFF_SEED=17 cargo test --test differential_backends -- --nocapture

Run a larger seed count:

TONIC_DIFF_SEEDS=256 cargo test --test differential_backends -- --nocapture

Defaults: TONIC_DIFF_SEEDS=32 and seed range [0, N).

Divergence triage artifacts

On mismatch, the harness writes a replay bundle under:

  • <temp-root>/differential-artifacts/<label>/program.tn
  • <temp-root>/differential-artifacts/<label>/program.min.tn
  • <temp-root>/differential-artifacts/<label>/mismatch.json

mismatch.json includes:

  • fixture/seed identity
  • mismatch reason (exit code mismatch, stdout mismatch, stderr mismatch, or native compile failure)
  • interpreter and native command outputs
  • replay command lines

Manual replay recipe

Given a failing fixture <path>:

tonic run <path>
tonic compile <path>
./.tonic/build/<stem>

Compare exit code, stdout, and stderr exactly.