Skip to content

CLI verification overhaul: add submission verify, auto-verifier in measure; remove validate; aggregate robustness; Nix devshell improvements#20

Merged
Unisay merged 7 commits intomainfrom
yura/issue-19
Aug 18, 2025
Merged

CLI verification overhaul: add submission verify, auto-verifier in measure; remove validate; aggregate robustness; Nix devshell improvements#20
Unisay merged 7 commits intomainfrom
yura/issue-19

Conversation

@Unisay
Copy link
Copy Markdown
Collaborator

@Unisay Unisay commented Aug 18, 2025

Summary

  • Add new cape submission verify wrapper: runs CEK correctness (with scenario/submission verifier when available) and validates schemas. Supports --all, PATH, --verbose, --no-color.
  • Remove legacy submission validate command; unify validation under verify as per CLI boundary guidance.
  • Measure wrapper improvements: infer scenario from submission path and pass verifier automatically when present; simplify help to reflect stable flags.
  • Aggregate robustness: tolerate missing/placeholder fields in metrics/metadata and respect externally provided PROJECT_ROOT (used by tests); avoid pipefail breakage.
  • Nix devshell: prefer repo-local scripts/cape.sh inside the shell wrapper; fix measure tool exposure via haskell-nix attr (hsPkgs.measure.components.exes.measure).
  • Docs & scenarios: add scenario verifiers for fibonacci and factorial; refresh scenario docs and schemas README; update copilot instructions.
  • Measure tool scaffolding: add measure.cabal, fix cabal.project index-state syntax, and include App/Compile.hs, App/Verify.hs, GenerateVerifiers.hs skeleton.
  • Test suite & fixtures: add scenario fixtures (returns-42, returns-unit) and update test.sh groups; ensure sandbox overlays and cleanup; increase timeouts where needed.
  • Reports & timestamps: refresh generated HTML reports and fixture timestamps to reflect recent measurements.

Rationale

  • Aligns with project CLI boundaries: wrapper is repo-aware (verify, measure), core measure remains submission-agnostic.
  • Eliminates brittle parsing and permission pitfalls; ensures local iteration uses repo scripts.
  • Improves determinism and reproducibility in tests via sandboxed fixtures and auto-verifier behavior.

Breaking/Behavioral changes

  • cape submission validate removed in favor of cape submission verify.
  • cape submission measure wrapper no longer exposes -i/-o single-file mode; only PATH and --all (as documented). Low-level measure binary still supports -i/-o.

Testing

  • nix develop
  • cape test
  • cape submission verify --all

Notes

  • No scenario/verifier override flags added; wrapper auto-infers as per design doc. Existing submissions continue to work; scenarios with verifier.uplc will enforce expected results.

@Unisay Unisay self-assigned this Aug 18, 2025
@Unisay Unisay merged commit fe00704 into main Aug 18, 2025
1 check passed
@Unisay Unisay deleted the yura/issue-19 branch August 18, 2025 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant