Skip to content

Redesign TODO test output: separate axis with pending/resolved#728

Merged
gfx merged 3 commits intomainfrom
claude/redesign-todo-test-output-TEJKH
Mar 29, 2026
Merged

Redesign TODO test output: separate axis with pending/resolved#728
gfx merged 3 commits intomainfrom
claude/redesign-todo-test-output-TEJKH

Conversation

@gfx
Copy link
Copy Markdown
Member

@gfx gfx commented Mar 29, 2026

Summary

  • TODO tests (#[TODO], #![TODO]) are now reported on a separate axis from pass/fail, inspired by TAP's TODO directive
  • Pending (body traps): expected — the feature is still unimplemented. Does not affect exit code.
  • Resolved (body passes): hard failure (exit code 1) — enforces removing #[TODO] when the feature works.
  • New summary format: N passed, N failed; N todo (M resolved)
  • TODO summary section at the end lists all TODO tests with file path and status
  • Formal specification added to docs/spec.md (Test Outcome Model section)

Motivation

Previously, TODO tests were "expected to fail" — if a compiler fix caused a TODO test to start passing, it showed up as a test failure, which was confusing and made compiler development harder. Now, newly-passing TODO tests appear as "resolved" on the TODO axis instead.

Output example

Running tests in math_test.wado... (compiled in 50ms)
  ✓ addition (2ms)
  · future_feature # TODO (1ms)
  ✓ now_works # TODO resolved (2ms)
    remove the #[TODO] attribute

TODO tests (2):
  · pending   math_test.wado — future_feature
  ✓ resolved  math_test.wado — now_works

1 TODO test(s) resolved — remove the #[TODO] attribute

1 passed, 0 failed; 2 todo (1 resolved) (0.12s)

Test plan

  • cargo check --bin wado passes
  • cargo check --test e2e -p wado-compiler passes
  • Unit tests for test name parsing pass
  • mise run on-task-done (running)

https://claude.ai/code/session_01Dt11N9EJkrwhjhC7PhMLQG

claude added 3 commits March 29, 2026 12:10
- TODO tests no longer count as pass/fail; they have their own axis
- TodoPending: test trapped as expected (feature still unimplemented)
- TodoResolved: test passed unexpectedly (hard failure — must remove #[TODO])
- Summary line: "N passed, N failed; N todo (M resolved)"
- TODO summary section at the end lists all TODO tests with status
- Exit failure if any TODO tests are resolved (enforces cleanup)
- E2E tests: same policy — resolved TODO is a hard failure

https://claude.ai/code/session_01Dt11N9EJkrwhjhC7PhMLQG
- Add "Test Outcome Model" section with tables for regular and TODO tests
- Specify pending/resolved semantics and exit code rules
- Document output format with TODO summary section
- Update #[TODO] and #![TODO] attribute descriptions
- Update cheatsheet to reflect new TODO semantics

https://claude.ai/code/session_01Dt11N9EJkrwhjhC7PhMLQG
@gfx gfx enabled auto-merge March 29, 2026 13:20
@gfx gfx merged commit 5d55bf6 into main Mar 29, 2026
10 checks passed
@gfx gfx deleted the claude/redesign-todo-test-output-TEJKH branch March 29, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants