-
Notifications
You must be signed in to change notification settings - Fork 7
Add failure persistence support for regression testing #215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Add failure persistence support for regression testing #215
Conversation
97dbfad to
0fa8979
Compare
65ce54d to
ea642ea
Compare
… with `property.ts`
|
Since |
| `-------------------------------------------------------------------------------`, | ||
| `Using manifest path: ${manifestDirPlaceholder}/Clarinet.toml`, | ||
| `Target contract: counter`, | ||
| `-------------------------------------------------------------------------------`, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
definitely nit-picky - but if this dash line is being copied/pasted many times for output, wouldn't it make sense to have it as a variable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe it's time to think about some form of standardized logger (or customizing an existing one?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wileyj Great suggestion! Definitely makes sense to turn into a variable. Regarding logging, we have a custom event-based logger in place.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
full disclosure: i did something like this once, and went a bit overboard. when i was refactoring, it was a nightmare to replace all the ********************* lines with the new separator 😆
This PR implements failure persistence and regression testing for Rendezvous. When tests fail, the failing seeds and configuration used are now automatically saved to
.rendezvous-regressions/, allowing users to replay them on subsequent runs to prevent regressions.Key additions
--modeCLI flag to control test execution: new (fresh tests only), reg (regressions only), or all (both, default)Sample failure file
.rendezvous-regressions/ST1PQHQKV0RJXZFY1DGX8MNSNYVE3VGZJSRTPGZGM.counter.json:{ "invariant": [ { "seed": 901717247, "dial": "example/sip010.cjs", "numRuns": 15, "timestamp": 1767886534833 }, { "seed": -1374686468, "numRuns": 9, "timestamp": 1767886531457 }, { "seed": 1298457354, "dial": "example/sip010.cjs", "numRuns": 20, "timestamp": 1767883389583 }, ], "test": [ { "seed": 1656313995, "numRuns": 6, "timestamp": 1767886553125 }, { "seed": 64830639, "numRuns": 11, "timestamp": 1767886546477 }, { "seed": 1593583466, "numRuns": 3, "timestamp": 1767886542907 }, ] }This brings Rendezvous closer to production-grade fuzzing by ensuring discovered bugs stay caught, while improving internal code organization and maintainability.
Closes #130.