Commit 3479e6f
Enhanced cookbook recipe validation (#3238)
* test(cookbook): add vitest testing infrastructure
Set up vitest for cookbook package validation tests following repo patterns:
- Add vitest dependency and test scripts
- Create vitest.config.ts with node environment
- Add smoke test (util.test.ts) with 2 passing tests
- Add cookbook to root workspaces for turbo integration
Tests can be run via:
- `cd cookbook && npm test`
- `npx turbo run test --filter=cookbook`
* feat(cookbook): enforce numeric step numbers and validate duplicate names
Enforces step numbers as numbers (not strings) via Zod schema and adds
runtime validation to catch duplicate step names in grouped steps.
Changes:
- Update Zod schema to only accept numeric step numbers
- Update JSON schema to match (remove string union type)
- Add validateStepNumbering() to check for duplicate names in grouped steps
- Integrate validation into validateRecipe() flow before applyRecipe()
- Add comprehensive test coverage (7 new tests)
This catches the PR-3228 bug where recipe.yaml had step: "1" as strings
and prevents duplicate step names like:
step: 1, name: "Update config"
step: 1, name: "Update config" ← now throws error
Grouped steps with distinct names are still allowed:
step: 1, name: "README.md"
step: 1, name: "app/entry.server.tsx" ← valid
Performance: Recipe is loaded twice (once for validation, once in applyRecipe).
This duplication is acceptable as the overhead is negligible (~5ms) and keeps
the implementation simple. Can be optimized later if needed.
Tests: 9 passing (3 Zod validation tests, 4 duplicate name tests, 2 smoke tests)
* refactor(cookbook): split validators and enforce step descriptions
Refactors validation into focused, single-responsibility validators:
- validateStepNames() - checks for duplicate names in grouped steps
- validateStepDescriptions() - enforces descriptions are not null/empty
This is a breaking change that will fail validation for 55 steps across 7
recipes (b2b, metaobjects, legacy-customer-account-flow, gtm, infinite-scroll,
combined-listings, third-party-api) that currently have description: null.
These will be fixed in a separate PR per PR-3228 discussion.
Tests: 11 passing (4 step name tests, 2 description tests, 5 other tests)
* feat(cookbook): add bidirectional file reference validation
Adds validators to ensure patch and ingredient files match between recipe.yaml
and filesystem in both directions - no broken references, no orphaned files.
Validators added:
- validatePatchFiles() - checks patches referenced in yaml exist on disk,
and all patch files on disk are referenced in yaml
- validateIngredientFiles() - checks ingredients referenced in yaml exist on
disk, and all ingredient files on disk are referenced in yaml
This catches:
- Missing patch/ingredient files (broken references)
- Orphaned patch/ingredient files (not referenced in recipe)
- Early detection before recipe application
Performance: O(n) single-pass with Set for O(1) lookups, early returns when
directories don't exist.
Tests: 17 passing (12 validation tests, 3 recipe tests, 2 util tests)
- 3 patch validation tests
- 3 ingredient validation tests
- 6 existing validation tests
* feat(cookbook): validate README and LLM prompt existence
Adds validators to ensure rendered outputs exist:
- validateReadmeExists() - checks README.md exists at recipes/{name}/README.md
- validateLlmPromptExists() - checks LLM prompt exists at llms/{name}.prompt.md
Uses real integration tests against actual filesystem (gtm recipe) instead of
mocked tests. This provides more confidence that validation works correctly.
Changes:
- Add LLMS_PATH and RENDER_FILENAME_GITHUB to imports
- Add two focused validators with helpful error messages
- Integrate into validateRecipe() flow
- Remove global fs mock, use targeted spyOn mocks only where needed
- Add 4 integration tests (2 per validator)
Tests: 21 passing (16 validation tests, 3 recipe tests, 2 util tests)
* fix(cookbook): use numeric step fallback in generate
Fixes TypeScript error caused by string step number fallback after enforcing
numeric-only step numbers in recipe schema.
Changed: `step: existingStep?.step ?? \`${i}\``
To: `step: existingStep?.step ?? i`
This ensures generated steps always use numbers, matching the Zod schema
requirement from commit c937534.
* feat(cookbook): enhance validation error reporting with actual values
Improves validation error messages by showing actual YAML values alongside
expected types, making it immediately clear what needs to be fixed.
Changes:
- Add getYamlValue() to extract actual values from YAML nodes
- Include actual values in Zod schema error messages
Example: "Expected number, received string (actual value: "1")"
- Add validator names to all error output for clarity
- Refactor error collection to run all validators even with schema errors
(README/LLM prompt validators now run regardless of recipe schema validity)
- Remove unused functions: convertZodErrorToValidationErrors, enrichErrorsWithLineNumbers
- Add comprehensive unit tests for formatValidationError
- Add integration test validating actual values in error messages
- Mock filesystem in validateReadmeExists and validateLlmPromptExists tests
Test coverage: 26/26 tests passing
* test(cookbook): mock filesystem in actual value validation test
Fixes CI failure where test relied on actual filesystem paths that differ
between local and CI environments. Now uses mocked YAML content instead.
* test(cookbook): mock filesystem in getYamlLineNumber tests
Ensures tests work in both local and CI environments by mocking
fs.readFileSync instead of relying on actual recipe files.
* feat(cookbook)!: migrate step field from number to string with regex validation
BREAKING CHANGE: Recipe step field now requires string format instead of number.
All recipe.yaml files must be updated: step: 1 → step: "1"
Changes:
- Update Zod schema: z.number() → z.string().regex(/^\d+(?:\.\d+)?$/)
- Update JSON schema: type "number" → type "string" with pattern validation
- Add comprehensive test coverage (39 tests passing)
- Fix compareSteps bug: isSubstep(a) → isSubstep(step)
- Remove unnecessary String() conversions in validation
- Export compareSteps for testability
Test improvements:
- Add generate.test.ts with step ordering tests
- Update all test expectations for string steps
- Add edge case validation (invalid formats, substeps)
- Remove unused leading zeros test
- Document grouping feature and testing strategy
* fix(cookbook): address PR feedback and add TypeScript strict checking
Fixes issues identified in PR review and adds comprehensive TypeScript checking.
Changes:
- Fix type mismatch in generate.ts: use String(i) for step field fallback
- Add explanatory comment for empty catch block in validate.ts
- Fix test to verify actual relative order preservation in compareSteps
- Add 3 edge case tests for compareSteps (leading zeros, mixed types, decimals)
- Fix all Recipe test mocks: add missing llms field (12+ instances)
- Fix TypeScript mock types: use Mock type with assertions
- Add typecheck script and configure tsconfig with skipLibCheck
- Align TypeScript version to 5.9.2 (matches packages/)
Test coverage: 42/42 tests passing
TypeScript: Zero errors in src/
* test(cookbook): add comprehensive grouping test coverage and cleanup test names
Adds 9 new tests covering edge cases and cleans up test descriptions for clarity.
New tests:
- Multiple separate groups with stable sort
- Interleaved groups maintaining insertion order
- Interleaved main steps and substeps (validates numeric sorting)
- Multiple substeps with same step number
- Explicit "1.0" vs implicit "1" equivalence
- Zero step numbers and zero with substeps
- Very large step numbers (9999, 1000)
- Multiple files in same step number (real-world scenario)
Test name improvements:
- Remove verbose comments and investigation tags
- Simplify descriptions for contributor clarity
- Focus on what behavior is being validated
Coverage: 51/51 tests passing
* feat(cookbook): add formatted error handling to all loadRecipe calls
Ensures consistent error formatting across all commands that load recipes.
Changes:
- Export printValidationErrors for reuse across commands
- Add handleZodErrorFromLoadRecipe helper to format Zod schema errors
- Wrap loadRecipe calls in try-catch in apply.ts, render.ts, llms.ts, update.ts
- Display errors with line numbers and actual values instead of raw Zod output
Before:
ZodError: [{ code: 'invalid_type', expected: 'string', received: 'number', path: ['steps', 0, 'step'], ... }]
After:
recipe.yaml:34 steps.0.step RecipeSchema: Expected string, received number (actual value: 1)
All commands now have consistent, actionable error messages.
* feat(cookbook): reject duplicate step numbers, enforce substep hierarchy
Addresses PR feedback to prevent confusing duplicate step number headings.
Changes:
- Reject duplicate step numbers (e.g., two steps both labeled '1')
- Detect numeric equivalence ('1' === '01' === '1.0')
- Provide helpful error messages suggesting substeps
- Add 4 new validation tests for duplicate detection
- Remove 5 grouping tests that validated incorrect behavior
- Simplify generate.test.ts to 12 focused tests
- Add lint script to cookbook/package.json
Validation now enforces proper hierarchy:
✅ VALID: step: '1' → step: '1.1' → step: '1.2' → step: '2'
❌ INVALID: step: '1' → step: '1' (use substeps instead)
Rendering output:
### Step 1: Setup
#### Step 1.1: README.md
#### Step 1.2: app/root.tsx
Test coverage: 46/46 tests passing
---------
Co-authored-by: rbshop <[email protected]>1 parent 2ba29bb commit 3479e6f
File tree
18 files changed
+3761
-163
lines changed- .changeset
- cookbook
- src
- commands
- lib
18 files changed
+3761
-163
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
40 | 43 | | |
41 | 44 | | |
42 | 45 | | |
43 | | - | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
44 | 49 | | |
45 | 50 | | |
46 | 51 | | |
47 | 52 | | |
48 | 53 | | |
49 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
50 | 57 | | |
51 | 58 | | |
52 | 59 | | |
| |||
58 | 65 | | |
59 | 66 | | |
60 | 67 | | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
65 | 75 | | |
66 | | - | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
67 | 79 | | |
68 | 80 | | |
69 | 81 | | |
| |||
141 | 153 | | |
142 | 154 | | |
143 | 155 | | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
| 156 | + | |
148 | 157 | | |
149 | 158 | | |
150 | 159 | | |
| |||
159 | 168 | | |
160 | 169 | | |
161 | 170 | | |
162 | | - | |
| 171 | + | |
163 | 172 | | |
164 | 173 | | |
165 | 174 | | |
166 | | - | |
| 175 | + | |
167 | 176 | | |
168 | 177 | | |
169 | 178 | | |
170 | 179 | | |
171 | 180 | | |
172 | 181 | | |
173 | | - | |
| 182 | + | |
174 | 183 | | |
175 | 184 | | |
176 | | - | |
177 | | - | |
178 | | - | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
179 | 189 | | |
180 | 190 | | |
181 | 191 | | |
182 | 192 | | |
183 | 193 | | |
184 | 194 | | |
185 | | - | |
| 195 | + | |
186 | 196 | | |
187 | 197 | | |
188 | 198 | | |
189 | 199 | | |
190 | 200 | | |
191 | 201 | | |
192 | 202 | | |
193 | | - | |
| 203 | + | |
194 | 204 | | |
195 | 205 | | |
196 | | - | |
| 206 | + | |
197 | 207 | | |
198 | 208 | | |
199 | 209 | | |
| |||
203 | 213 | | |
204 | 214 | | |
205 | 215 | | |
206 | | - | |
| 216 | + | |
207 | 217 | | |
208 | 218 | | |
209 | 219 | | |
| |||
214 | 224 | | |
215 | 225 | | |
216 | 226 | | |
217 | | - | |
| 227 | + | |
218 | 228 | | |
219 | 229 | | |
220 | | - | |
| 230 | + | |
221 | 231 | | |
222 | 232 | | |
223 | 233 | | |
224 | 234 | | |
225 | | - | |
| 235 | + | |
226 | 236 | | |
227 | 237 | | |
228 | 238 | | |
| |||
241 | 251 | | |
242 | 252 | | |
243 | 253 | | |
244 | | - | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
245 | 257 | | |
246 | 258 | | |
247 | 259 | | |
| |||
263 | 275 | | |
264 | 276 | | |
265 | 277 | | |
266 | | - | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
267 | 281 | | |
268 | 282 | | |
269 | 283 | | |
| |||
298 | 312 | | |
299 | 313 | | |
300 | 314 | | |
301 | | - | |
| 315 | + | |
0 commit comments