Skip to content

Add failure persistence support for regression testing #215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BowTiedRadone wants to merge 26 commits into
base: master
Choose a base branch
from
+1,447 −311
Open

Add failure persistence support for regression testing #215

Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
e0fdb18
Fix Clarity error reporting and update reporter for consistency
BowTiedRadone Dec 31, 2025
7b01545
Persist failed seeds for regression testing
BowTiedRadone Dec 31, 2025
21a8bed
Make property testing routine asynchronous
BowTiedRadone Jan 2, 2026
b3ebaa4
Add new `mode` option for regression testing
BowTiedRadone Jan 2, 2026
85bfd40
Update help message to include new option and refine it
BowTiedRadone Jan 2, 2026
bd31ed8
Make jest runner silent to suppress misleading test stdout/stderr
BowTiedRadone Jan 2, 2026
d77adba
Update regression folder name
BowTiedRadone Jan 5, 2026
acdc168
Make `TestMode` binary
BowTiedRadone Jan 5, 2026
b82b3d5
Update `.gitignore`
BowTiedRadone Jan 5, 2026
8235153
Implement regression checking inside property testing routine
BowTiedRadone Jan 5, 2026
0105a3e
Improve logging
BowTiedRadone Jan 5, 2026
10268f9
Update command-line args handling tests
BowTiedRadone Jan 5, 2026
0fa8979
Update property to use `loadFailures` and create persistence testfile
BowTiedRadone Jan 5, 2026
bf4105b
Make failure persistence module synchronous
BowTiedRadone Jan 6, 2026
ea642ea
Add property-based tests for the failure persistence module
BowTiedRadone Jan 6, 2026
21872e2
Include counterexample `numRuns` in the regression object
BowTiedRadone Jan 6, 2026
40d0e42
Add `dial` field to `FailureRecord` interface
BowTiedRadone Jan 8, 2026
80c2655
Use `switch` conditional in `checkProperties` and improve regression …
BowTiedRadone Jan 8, 2026
6f28f9c
PoLA and maintainability updates in `property.ts`
BowTiedRadone Jan 8, 2026
07de377
Prepare `invariant.ts` for failure persistence similarly side-by-side…
BowTiedRadone Jan 8, 2026
97b2ff3
Enable failure persistence for invariant and add command-line options…
BowTiedRadone Jan 8, 2026
2b035c5
Add tests for `numRuns` and `dial` persistence fields
BowTiedRadone Jan 8, 2026
ad77d7f
Update `--mode <string>` to `--regr`
BowTiedRadone Jan 8, 2026
fb1659f
Add regression testing docs
BowTiedRadone Jan 8, 2026
6218b3c
Extract log divider to a constant in `shared.ts`
BowTiedRadone Jan 12, 2026
21d069c
Remove unnecessary `.gitignore` line
BowTiedRadone Jan 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ node_modules
deployments
dist
coverage
**/.cache/**
**/.cache/**
.rendezvous-regressions
101 changes: 76 additions & 25 deletions app.tests.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { version } from "./package.json";
import { resolve } from "path";
import fs, { rmSync } from "fs";
import { createIsolatedTestEnvironment } from "./test.utils";
import { getFailureFilePath } from "./persistence";

const isolatedTestEnvPrefix = "rendezvous-test-app-";

Expand All @@ -13,19 +14,22 @@ describe("Command-line arguments handling", () => {
const helpMessage = `
rv v${version}

Usage: rv <path-to-clarinet-project> <contract-name> <type> [--seed=<seed>] [--runs=<runs>] [--dial=<path-to-dialers-file>] [--help]
Usage: rv <path> <contract> <type> [OPTIONS]

Positional arguments:
path-to-clarinet-project - The path to the Clarinet project.
contract-name - The name of the contract to be fuzzed.
type - The type to use for exercising the contracts. Possible values: test, invariant.
Arguments:
<path> Path to the Clarinet project
<contract> Contract name to fuzz
<type> Test type: test | invariant

Options:
--seed - The seed to use for the replay functionality.
--runs - The runs to use for iterating over the tests. Default: 100.
--bail - Stop after the first failure.
--dial – The path to a JavaScript file containing custom pre- and post-execution functions (dialers).
--help - Show the help message.
--seed=<n> Seed for replay functionality
--runs=<n> Number of test iterations [default: 100]
--dial=<f> Path to custom dialers file
--regr Run regression tests only
--bail Stop on first failure
-h, --help Show this message

Learn more: https://stacks-network.github.io/rendezvous/
`;

const noManifestMessage = red(
Expand Down Expand Up @@ -177,18 +181,40 @@ describe("Command-line arguments handling", () => {
["manifest path", "contract name", "type=invariant"],
["node", "app.js", manifestDirPlaceholder, "counter", "invariant"],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`\nStarting invariant testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],

[
["manifest path", "contract name", "type=InVaRiAnT (case-insensitive)"],
["node", "app.js", manifestDirPlaceholder, "counter", "InVaRiAnT"],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`-------------------------------------------------------------------------------`,
],
],
[
["manifest path", "contract name", "type=invariant", "regr"],
[
"node",
"app.js",
manifestDirPlaceholder,
"counter",
"invariant",
"--regr",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`\nStarting invariant testing type for the counter contract...\n`,
`Running regression tests.`,
`Regressions loaded from: ${resolve(getFailureFilePath("counter"))}`,
`-------------------------------------------------------------------------------`,
],
],
[
Expand All @@ -202,10 +228,11 @@ describe("Command-line arguments handling", () => {
"--bail",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`Bailing on first failure.`,
`\nStarting invariant testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
Expand All @@ -219,38 +246,54 @@ describe("Command-line arguments handling", () => {
"--dial=example/sip010.cjs",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`Using dial path: example/sip010.cjs`,
`\nStarting invariant testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
["manifest path", "contract name", "type=test"],
["node", "app.js", manifestDirPlaceholder, "counter", "test"],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`\nStarting property testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
["manifest path", "contract name", "type=tESt (case-insensitive)"],
["node", "app.js", manifestDirPlaceholder, "counter", "tESt"],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`-------------------------------------------------------------------------------`,
],
],
[
["manifest path", "contract name", "type=test", "regr"],
["node", "app.js", manifestDirPlaceholder, "counter", "test", "--regr"],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`\nStarting property testing type for the counter contract...\n`,
`Running regression tests.`,
`Regressions loaded from: ${resolve(getFailureFilePath("counter"))}`,
`-------------------------------------------------------------------------------`,
],
],
[
["manifest path", "contract name", "type=test", "bail"],
["node", "app.js", manifestDirPlaceholder, "counter", "test", "--bail"],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`Bailing on first failure.`,
`\nStarting property testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
Expand All @@ -264,10 +307,11 @@ describe("Command-line arguments handling", () => {
"--seed=123",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`Using seed: 123`,
`\nStarting invariant testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
Expand All @@ -286,10 +330,11 @@ describe("Command-line arguments handling", () => {
"--seed=123",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`Using seed: 123`,
`\nStarting invariant testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
Expand All @@ -303,10 +348,11 @@ describe("Command-line arguments handling", () => {
"--seed=123",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`Using seed: 123`,
`\nStarting property testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
Expand All @@ -320,20 +366,22 @@ describe("Command-line arguments handling", () => {
"--seed=123",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: reverse`,
`Using seed: 123`,
`\nStarting property testing type for the reverse contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
["manifest path", "contract name = slice", "type=test", "seed"],
["node", "app.js", manifestDirPlaceholder, "slice", "test", "--seed=123"],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: slice`,
`Using seed: 123`,
`\nStarting property testing type for the slice contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
Expand All @@ -352,10 +400,11 @@ describe("Command-line arguments handling", () => {
"--seed=123",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`Using seed: 123`,
`\nStarting property testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
Expand All @@ -371,12 +420,13 @@ describe("Command-line arguments handling", () => {
"--bail",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`Using seed: 123`,
`Using runs: 10`,
`Bailing on first failure.`,
`\nStarting property testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
[
Expand All @@ -399,12 +449,13 @@ describe("Command-line arguments handling", () => {
"--bail",
],
[
`-------------------------------------------------------------------------------`,
`Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`,
`Target contract: counter`,
`Using seed: 123`,
`Using runs: 10`,
`Bailing on first failure.`,
`\nStarting invariant testing type for the counter contract...\n`,
`-------------------------------------------------------------------------------`,
],
],
])(
Expand Down
Loading