Skip to content

Commit e5a5a89

Browse files
boneskullclaude
andcommitted
feat(budgets): add wildcard pattern support for budget configuration
Add support for glob patterns in file paths and simple wildcards (`*`) for suite and task names in budget configuration. This allows users to apply budgets to multiple tasks at once without enumerating each one. Key changes: - Add BudgetPattern and ResolvedBudgets types for pattern-based budgets - Create budget-resolver service with minimatch file matching and simple wildcard suite/task matching - Update transformBudgets() to produce ResolvedBudgets with exact matches and patterns separated - Update BudgetEvaluator to resolve budgets using pattern matching with specificity-based precedence - Fix budget evaluation to not skip tasks with both absolute and relative budgets when no baseline is provided Pattern specificity (higher = more specific): - Exact file/suite/task: 4 - Exact file + exact suite + wildcard task: 3 - Glob file + exact suite/task: 3 - Exact file + wildcard suite/task: 2 - Glob file + wildcard suite/task: 1 - Full wildcard (**/* / * / *): 0 When multiple patterns match, budgets are merged with more specific values overriding less specific ones. Closes #72 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 874461e commit e5a5a89

File tree

13 files changed

+1771
-146
lines changed

13 files changed

+1771
-146
lines changed

README.md

Lines changed: 37 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -324,10 +324,14 @@ Define budgets in your `modestbench.config.json`:
324324
{
325325
"budgetMode": "fail",
326326
"budgets": {
327-
"benchmarks/critical.bench.js/default/parseConfig": {
328-
"absolute": {
329-
"maxTime": "10ms",
330-
"minOpsPerSec": 100000
327+
"benchmarks/critical.bench.js": {
328+
"default": {
329+
"parseConfig": {
330+
"absolute": {
331+
"maxTime": "10ms",
332+
"minOpsPerSec": 100000
333+
}
334+
}
331335
}
332336
}
333337
}
@@ -344,6 +348,35 @@ Define budgets in your `modestbench.config.json`:
344348
- **Relative Budgets**: Comparison against baseline
345349
- `maxRegression` - Maximum performance degradation (e.g., `"10%"`, `0.1`)
346350

351+
#### Wildcard Patterns
352+
353+
Apply budgets broadly using wildcards:
354+
355+
```json
356+
{
357+
"budgets": {
358+
"**/*.bench.js": {
359+
"*": {
360+
"*": {
361+
"relative": { "maxRegression": "15%" }
362+
}
363+
}
364+
},
365+
"benchmarks/critical.bench.js": {
366+
"*": {
367+
"*": {
368+
"relative": { "maxRegression": "5%" }
369+
}
370+
}
371+
}
372+
}
373+
}
374+
```
375+
376+
- **Files**: Use glob patterns (`**/*.bench.js`, `benchmarks/*.bench.js`)
377+
- **Suites/Tasks**: Use `*` to match any name
378+
- **Precedence**: Most specific pattern wins (exact matches override wildcards)
379+
347380
**Budget Modes:**
348381

349382
- `fail` (default) - Exit with error code if budgets fail

site/src/content/docs/guides/performance-budgets.mdx

Lines changed: 160 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,129 @@ Use both absolute and relative:
149149
}
150150
```
151151

152+
## Wildcard Patterns
153+
154+
Instead of specifying every file, suite, and task individually, you can use wildcard patterns to apply budgets broadly:
155+
156+
### Pattern Syntax
157+
158+
| Level | Exact Match | Wildcard |
159+
|-------|-------------|----------|
160+
| File | `"api.bench.js"` | `"**/*.bench.js"` (glob) |
161+
| Suite | `"String Operations"` | `"*"` (any suite) |
162+
| Task | `"parseJSON"` | `"*"` (any task) |
163+
164+
- **Files**: Use glob patterns (e.g., `**/*.bench.js`, `benchmarks/*.bench.js`)
165+
- **Suites/Tasks**: Use `*` to match any name
166+
167+
### Example: Global Default Budget
168+
169+
Apply a budget to all benchmarks:
170+
171+
```json
172+
{
173+
"budgets": {
174+
"**/*.bench.js": {
175+
"*": {
176+
"*": {
177+
"relative": {
178+
"maxRegression": "15%"
179+
}
180+
}
181+
}
182+
}
183+
}
184+
}
185+
```
186+
187+
### Example: Stricter Budget for Specific Files
188+
189+
Override the global default with stricter limits for critical paths:
190+
191+
```json
192+
{
193+
"budgets": {
194+
"**/*.bench.js": {
195+
"*": {
196+
"*": {
197+
"relative": { "maxRegression": "15%" }
198+
}
199+
}
200+
},
201+
"benchmarks/critical.bench.js": {
202+
"*": {
203+
"*": {
204+
"relative": { "maxRegression": "5%" }
205+
}
206+
}
207+
}
208+
}
209+
}
210+
```
211+
212+
### Example: Mixed Specificity
213+
214+
Combine wildcards with exact matches for fine-grained control:
215+
216+
```json
217+
{
218+
"budgets": {
219+
"**/*.bench.js": {
220+
"*": {
221+
"*": {
222+
"relative": { "maxRegression": "20%" }
223+
}
224+
}
225+
},
226+
"api.bench.js": {
227+
"Authentication": {
228+
"*": {
229+
"relative": { "maxRegression": "10%" }
230+
}
231+
},
232+
"Authentication": {
233+
"login": {
234+
"absolute": { "maxTime": "50ms" },
235+
"relative": { "maxRegression": "5%" }
236+
}
237+
}
238+
}
239+
}
240+
}
241+
```
242+
243+
### Precedence Rules
244+
245+
When multiple patterns match a task, the **most specific** pattern wins:
246+
247+
1. **Exact matches** always take precedence over wildcards
248+
2. **File specificity**: Exact file > partial glob > full glob
249+
3. **Suite/task specificity**: Exact name > `*` wildcard
250+
251+
**Specificity scoring:**
252+
253+
| Pattern Component | Points |
254+
|-------------------|--------|
255+
| Exact file name | +3 |
256+
| Partial glob (e.g., `benchmarks/*.js`) | +2 |
257+
| Full glob (`**/*.js`) | +1 |
258+
| Exact suite name | +2 |
259+
| Exact task name | +1 |
260+
| Wildcard (`*`) | +0 |
261+
262+
**Example:**
263+
264+
For task `api.bench.js/Authentication/login`:
265+
266+
| Pattern | Specificity | Result |
267+
|---------|-------------|--------|
268+
| `**/*.bench.js/*/*` | 1 | Lowest priority |
269+
| `api.bench.js/*/*` | 3 | Medium priority |
270+
| `api.bench.js/Authentication/*` | 5 | Higher priority |
271+
| `api.bench.js/Authentication/login` | 6 | **Wins** |
272+
273+
When patterns have equal specificity, the budgets are **merged**, with later definitions overriding earlier ones.
274+
152275
## Baseline Management
153276

154277
### Creating Baselines
@@ -472,29 +595,59 @@ Adjust budget if current performance is legitimately slower.
472595
}
473596
```
474597

475-
### Example 2: Regression Guard
598+
### Example 2: Regression Guard for All Tasks
476599

477-
:::note
478-
To apply budgets to all tasks, you'll need to specify each file/suite/task. Wildcard patterns are not currently supported, but are being considered for a future release. See [issue #72](https://github.com/boneskull/modestbench/issues/72) for details.
479-
:::
600+
Apply a regression guard to all benchmarks using wildcards:
601+
602+
```json
603+
{
604+
"baseline": "production",
605+
"budgets": {
606+
"**/*.bench.js": {
607+
"*": {
608+
"*": {
609+
"relative": { "maxRegression": "15%" }
610+
}
611+
}
612+
}
613+
}
614+
}
615+
```
616+
617+
### Example 3: Regression Guard with Overrides
618+
619+
Apply different thresholds to different areas:
480620

481621
```json
482622
{
483623
"baseline": "production",
484624
"budgets": {
625+
"**/*.bench.js": {
626+
"*": {
627+
"*": {
628+
"relative": { "maxRegression": "20%" }
629+
}
630+
}
631+
},
632+
"api.bench.js": {
633+
"*": {
634+
"*": {
635+
"relative": { "maxRegression": "10%" }
636+
}
637+
}
638+
},
485639
"api.bench.js": {
486640
"default": {
487641
"parseRequest": {
488-
"relative": { "maxRegression": "15%" }
642+
"relative": { "maxRegression": "5%" }
489643
}
490644
}
491645
}
492-
// Repeat for other files/suites/tasks...
493646
}
494647
}
495648
```
496649

497-
### Example 3: Per-Task Budgets
650+
### Example 4: Per-Task Budgets
498651

499652
```json
500653
{

src/config/schema.ts

Lines changed: 54 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,17 @@
88

99
import * as z from 'zod';
1010

11-
import type { Budget } from '../types/budgets.js';
11+
import type {
12+
Budget,
13+
BudgetPattern,
14+
ResolvedBudgets,
15+
} from '../types/budgets.js';
1216

1317
import { BENCHMARK_FILE_PATTERN } from '../constants.js';
18+
import {
19+
createBudgetPattern,
20+
isGlobPattern,
21+
} from '../services/budget-resolver.js';
1422
import { parsePercentageString, parseTimeString } from './budget-schema.js';
1523

1624
/**
@@ -194,33 +202,64 @@ const budgetsInputSchema = z.record(
194202
);
195203

196204
/**
197-
* Transform nested budget structure to flat TaskId → Budget mapping
205+
* Check if a suite or task name is a wildcard
206+
*
207+
* @param name - The suite or task name
208+
* @returns True if the name is a wildcard (`*`)
209+
*/
210+
const isWildcard = (name: string): boolean => name === '*';
211+
212+
/**
213+
* Check if a budget entry contains any wildcards or glob patterns
214+
*
215+
* @param file - File pattern
216+
* @param suite - Suite name or wildcard
217+
* @param task - Task name or wildcard
218+
* @returns True if any part contains wildcards
219+
*/
220+
const hasWildcards = (file: string, suite: string, task: string): boolean => {
221+
return isGlobPattern(file) || isWildcard(suite) || isWildcard(task);
222+
};
223+
224+
/**
225+
* Transform nested budget structure to ResolvedBudgets with exact matches and
226+
* patterns separated
198227
*
199228
* @param nested - Nested budgets structure (file → suite → task → budget)
200-
* @returns Flat budgets map (taskId → budget)
229+
* @returns ResolvedBudgets with exact matches and wildcard patterns
201230
*/
202231
const flattenBudgets = (
203232
nested: z.infer<typeof budgetsInputSchema>,
204-
): Record<string, Budget> => {
205-
const flat: Record<string, Budget> = {};
233+
): ResolvedBudgets => {
234+
const exact: Record<string, Budget> = {};
235+
const patterns: BudgetPattern[] = [];
206236

207237
for (const [file, suites] of Object.entries(nested)) {
208238
for (const [suite, tasks] of Object.entries(suites)) {
209239
for (const [task, budget] of Object.entries(tasks)) {
210-
const taskId = `${file}/${suite}/${task}`;
211-
flat[taskId] = budget;
240+
if (hasWildcards(file, suite, task)) {
241+
// This is a pattern budget
242+
patterns.push(createBudgetPattern(file, suite, task, budget));
243+
} else {
244+
// This is an exact match
245+
const taskId = `${file}/${suite}/${task}`;
246+
exact[taskId] = budget;
247+
}
212248
}
213249
}
214250
}
215251

216-
return flat;
252+
// Sort patterns by specificity descending for consistent iteration order
253+
patterns.sort((a, b) => b.specificity - a.specificity);
254+
255+
return { exact, patterns };
217256
};
218257

219258
/**
220-
* Budgets schema with transform for nested-to-flat conversion
259+
* Budgets schema with transform for nested-to-ResolvedBudgets conversion
221260
*
222-
* Input: { [file]: { [suite]: { [task]: Budget } } } Output: { [taskId]: Budget
223-
* } where taskId = "file/suite/task"
261+
* Input: { [file]: { [suite]: { [task]: Budget } } } Output: ResolvedBudgets {
262+
* exact: { [taskId]: Budget }, patterns: BudgetPattern[] }
224263
*/
225264
const budgetsSchema = budgetsInputSchema.transform(flattenBudgets);
226265

@@ -386,7 +425,7 @@ const baseConfigProperties = {
386425

387426
/** Description for the budgets field */
388427
const budgetsDescription =
389-
'Performance budgets organized by file → suite → task. Budgets define acceptable performance thresholds.';
428+
'Performance budgets organized by file → suite → task. Budgets define acceptable performance thresholds. Supports wildcards (* for suite/task, glob patterns for files).';
390429

391430
/** Description and metadata for the config schema */
392431
const configSchemaDescription =
@@ -402,7 +441,7 @@ const configSchemaMeta = { title: 'ModestBench Configuration' };
402441
* The budgets field uses transforms to:
403442
*
404443
* 1. Parse string values like "10ms" or "10%" to numbers
405-
* 2. Flatten nested structure to flat taskId → Budget mapping
444+
* 2. Separate exact matches from wildcard patterns into ResolvedBudgets
406445
*/
407446
const modestBenchConfigSchema = z
408447
.object({
@@ -472,8 +511,8 @@ export const safeParseConfig = (config: unknown) => {
472511
/**
473512
* Configuration type after parsing (output type)
474513
*
475-
* This is the type you get after parsing a config file - budgets are flattened
476-
* to taskId keys and string values are converted to numbers.
514+
* This is the type you get after parsing a config file - budgets are
515+
* transformed to ResolvedBudgets and string values are converted to numbers.
477516
*/
478517
export type ModestBenchConfig = z.infer<typeof modestBenchConfigSchema>;
479518

0 commit comments

Comments
 (0)