|
| 1 | +# TEST-NEEDS.md — palimpsest-license |
| 2 | + |
| 3 | +> Generated 2026-03-29 by punishing audit. |
| 4 | +
|
| 5 | +## Current State |
| 6 | + |
| 7 | +| Category | Count | Notes | |
| 8 | +|-------------|-------|-------| |
| 9 | +| Unit tests | 6 | Haskell Spec: BilingualSpec, LicenseSpec, MetadataSpec, ReferenceSpec, UtilsSpec + 1 Integration/PipelineSpec | |
| 10 | +| Integration | 1 | Integration/PipelineSpec.hs | |
| 11 | +| E2E | 0 | None | |
| 12 | +| Benchmarks | 0 | None | |
| 13 | + |
| 14 | +**Source modules:** ~18 across multiple languages. Haskell validator: 8 modules (Main, Validator, Bilingual, License, Metadata, Reference, Types, Utils). Elixir: 5 Ecto modules (derivative, license, lineage, violation, work) in ARCHIVE. OCaml: 1 test. Rust: 6 files. PHP: 4 files. |
| 15 | + |
| 16 | +## What's Missing |
| 17 | + |
| 18 | +### P2P (Property-Based) Tests |
| 19 | +- [ ] License validator: property tests for arbitrary license text classification |
| 20 | +- [ ] Bilingual validator: property tests for language detection accuracy |
| 21 | +- [ ] Metadata validator: arbitrary SPDX header validation |
| 22 | +- [ ] Reference validator: property tests for citation format compliance |
| 23 | + |
| 24 | +### E2E Tests |
| 25 | +- [ ] Full validation pipeline: input file -> detect license -> validate -> report |
| 26 | +- [ ] Multi-file: validate entire repository license compliance |
| 27 | +- [ ] Cross-language: Haskell validator agrees with Rust validator agrees with OCaml validator |
| 28 | + |
| 29 | +### Aspect Tests |
| 30 | +- **Security:** No tests for malicious license text injection, path traversal in file scanning |
| 31 | +- **Performance:** No validation throughput benchmarks |
| 32 | +- **Concurrency:** No tests for parallel file scanning |
| 33 | +- **Error handling:** No tests for binary files, empty files, files with mixed encodings |
| 34 | + |
| 35 | +### Build & Execution |
| 36 | +- [ ] `cabal test` or `stack test` for Haskell |
| 37 | +- [ ] `cargo test` for Rust |
| 38 | +- [ ] OCaml test execution |
| 39 | + |
| 40 | +### Benchmarks Needed |
| 41 | +- [ ] License detection speed per file |
| 42 | +- [ ] Batch validation throughput (files/second) |
| 43 | +- [ ] Bilingual detection accuracy and speed |
| 44 | + |
| 45 | +### Self-Tests |
| 46 | +- [ ] Validate its own LICENSE file |
| 47 | +- [ ] Validate its own SPDX headers across all source files |
| 48 | + |
| 49 | +## Priority |
| 50 | + |
| 51 | +**MEDIUM.** The Haskell validator has 7 test files for 8 modules — actually decent coverage ratio. But the Rust, OCaml, PHP, and Elixir implementations have ZERO tests. The cross-language consistency is completely unverified. No benchmarks for a tool that needs to scan entire repositories quickly. |
| 52 | + |
| 53 | +## FAKE-FUZZ ALERT |
| 54 | + |
| 55 | +- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing |
| 56 | +- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file |
| 57 | +- Priority: P2 — creates false impression of fuzz coverage |
0 commit comments