Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 37 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,10 +324,14 @@ Define budgets in your `modestbench.config.json`:
{
"budgetMode": "fail",
"budgets": {
"benchmarks/critical.bench.js/default/parseConfig": {
"absolute": {
"maxTime": "10ms",
"minOpsPerSec": 100000
"benchmarks/critical.bench.js": {
"default": {
"parseConfig": {
"absolute": {
"maxTime": "10ms",
"minOpsPerSec": 100000
}
}
}
}
}
Expand All @@ -344,6 +348,35 @@ Define budgets in your `modestbench.config.json`:
- **Relative Budgets**: Comparison against baseline
- `maxRegression` - Maximum performance degradation (e.g., `"10%"`, `0.1`)

#### Wildcard Patterns

Apply budgets broadly using wildcards:

```json
{
"budgets": {
"**/*.bench.js": {
"*": {
"*": {
"relative": { "maxRegression": "15%" }
}
}
},
"benchmarks/critical.bench.js": {
"*": {
"*": {
"relative": { "maxRegression": "5%" }
}
}
}
}
}
```

- **Files**: Use glob patterns (`**/*.bench.js`, `benchmarks/*.bench.js`)
- **Suites/Tasks**: Use `*` to match any name
- **Precedence**: Most specific pattern wins (exact matches override wildcards)

**Budget Modes:**

- `fail` (default) - Exit with error code if budgets fail
Expand Down
6 changes: 6 additions & 0 deletions astro.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ export default defineConfig({
{ label: 'CLI Reference', link: '/guides/cli/' },
{ label: 'Output Formats', link: '/guides/output/' },
{ label: 'Understanding Statistics', link: '/guides/statistics/' },
{
label: 'Performance Budgets',
link: '/guides/performance-budgets/',
},
{ label: 'Profiling', link: '/guides/profiling/' },
{ label: 'Test Adapters', link: '/guides/test-adapters/' },
{ label: 'Advanced Usage', link: '/guides/advanced/' },
{ label: 'Custom Reporters', link: '/guides/custom-reporters/' },
],
Expand Down
165 changes: 158 additions & 7 deletions site/src/content/docs/guides/performance-budgets.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,127 @@ Use both absolute and relative:
}
```

## Wildcard Patterns

Instead of specifying every file, suite, and task individually, you can use wildcard patterns to apply budgets broadly:

### Pattern Syntax

| Level | Exact Match | Wildcard |
|-------|-------------|----------|
| File | `"api.bench.js"` | `"**/*.bench.js"` (glob) |
| Suite | `"String Operations"` | `"*"` (any suite) |
| Task | `"parseJSON"` | `"*"` (any task) |

- **Files**: Use glob patterns (e.g., `**/*.bench.js`, `benchmarks/*.bench.js`)
- **Suites/Tasks**: Use `*` to match any name

### Example: Global Default Budget

Apply a budget to all benchmarks:

```json
{
"budgets": {
"**/*.bench.js": {
"*": {
"*": {
"relative": {
"maxRegression": "15%"
}
}
}
}
}
}
```

### Example: Stricter Budget for Specific Files

Override the global default with stricter limits for critical paths:

```json
{
"budgets": {
"**/*.bench.js": {
"*": {
"*": {
"relative": { "maxRegression": "15%" }
}
}
},
"benchmarks/critical.bench.js": {
"*": {
"*": {
"relative": { "maxRegression": "5%" }
}
}
}
}
}
```

### Example: Mixed Specificity

Combine wildcards with exact matches for fine-grained control:

```json
{
"budgets": {
"**/*.bench.js": {
"*": {
"*": {
"relative": { "maxRegression": "20%" }
}
}
},
"api.bench.js": {
"Authentication": {
"*": {
"relative": { "maxRegression": "10%" }
},
"login": {
"absolute": { "maxTime": "50ms" },
"relative": { "maxRegression": "5%" }
}
}
}
}
}
```

### Precedence Rules

When multiple patterns match a task, the **most specific** pattern wins:

1. **Exact matches** always take precedence over wildcards
2. **File specificity**: Exact file > partial glob > full glob
3. **Suite/task specificity**: Exact name > `*` wildcard

**Specificity scoring:**

| Pattern Component | Points |
|-------------------|--------|
| Exact file name | +2 |
| Glob with specific parts (e.g., `**/api/**/*.bench.js`) | +1 |
| Full glob (`**/*` or `*`) | +0 |
| Exact suite name | +1 |
| Exact task name | +1 |
| Wildcard (`*`) | +0 |

**Example:**

For task `api.bench.js/Authentication/login`:

| Pattern | Specificity | Result |
|---------|-------------|--------|
| `**/*.bench.js/*/*` | 1 | Lowest priority |
| `api.bench.js/*/*` | 2 | Medium priority |
| `api.bench.js/Authentication/*` | 3 | Higher priority |
| `api.bench.js/Authentication/login` | 4 | **Wins** |

When patterns have equal specificity, the budgets are **merged**, with later definitions overriding earlier ones.

## Baseline Management

### Creating Baselines
Expand Down Expand Up @@ -472,29 +593,59 @@ Adjust budget if current performance is legitimately slower.
}
```

### Example 2: Regression Guard
### Example 2: Regression Guard for All Tasks

:::note
To apply budgets to all tasks, you'll need to specify each file/suite/task. Wildcard patterns are not currently supported, but are being considered for a future release. See [issue #72](https://github.com/boneskull/modestbench/issues/72) for details.
:::
Apply a regression guard to all benchmarks using wildcards:

```json
{
"baseline": "production",
"budgets": {
"**/*.bench.js": {
"*": {
"*": {
"relative": { "maxRegression": "15%" }
}
}
}
}
}
```

### Example 3: Regression Guard with Overrides

Apply different thresholds to different areas:

```json
{
"baseline": "production",
"budgets": {
"**/*.bench.js": {
"*": {
"*": {
"relative": { "maxRegression": "20%" }
}
}
},
"api.bench.js": {
"*": {
"*": {
"relative": { "maxRegression": "10%" }
}
}
},
"utils.bench.js": {
"default": {
"parseRequest": {
"relative": { "maxRegression": "15%" }
"relative": { "maxRegression": "5%" }
}
}
}
// Repeat for other files/suites/tasks...
}
}
```

### Example 3: Per-Task Budgets
### Example 4: Per-Task Budgets

```json
{
Expand Down
69 changes: 54 additions & 15 deletions src/config/schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,17 @@

import * as z from 'zod';

import type { Budget } from '../types/budgets.js';
import type {
Budget,
BudgetPattern,
ResolvedBudgets,
} from '../types/budgets.js';

import { BENCHMARK_FILE_PATTERN } from '../constants.js';
import {
createBudgetPattern,
isGlobPattern,
} from '../services/budget-resolver.js';
import { parsePercentageString, parseTimeString } from './budget-schema.js';

/**
Expand Down Expand Up @@ -194,33 +202,64 @@ const budgetsInputSchema = z.record(
);

/**
* Transform nested budget structure to flat TaskId → Budget mapping
* Check if a suite or task name is a wildcard
*
* @param name - The suite or task name
* @returns True if the name is a wildcard (`*`)
*/
const isWildcard = (name: string): boolean => name === '*';

/**
* Check if a budget entry contains any wildcards or glob patterns
*
* @param file - File pattern
* @param suite - Suite name or wildcard
* @param task - Task name or wildcard
* @returns True if any part contains wildcards
*/
const hasWildcards = (file: string, suite: string, task: string): boolean => {
return isGlobPattern(file) || isWildcard(suite) || isWildcard(task);
};

/**
* Transform nested budget structure to ResolvedBudgets with exact matches and
* patterns separated
*
* @param nested - Nested budgets structure (file → suite → task → budget)
* @returns Flat budgets map (taskId → budget)
* @returns ResolvedBudgets with exact matches and wildcard patterns
*/
const flattenBudgets = (
nested: z.infer<typeof budgetsInputSchema>,
): Record<string, Budget> => {
const flat: Record<string, Budget> = {};
): ResolvedBudgets => {
const exact: Record<string, Budget> = {};
const patterns: BudgetPattern[] = [];

for (const [file, suites] of Object.entries(nested)) {
for (const [suite, tasks] of Object.entries(suites)) {
for (const [task, budget] of Object.entries(tasks)) {
const taskId = `${file}/${suite}/${task}`;
flat[taskId] = budget;
if (hasWildcards(file, suite, task)) {
// This is a pattern budget
patterns.push(createBudgetPattern(file, suite, task, budget));
} else {
// This is an exact match
const taskId = `${file}/${suite}/${task}`;
exact[taskId] = budget;
}
}
}
}

return flat;
// Sort patterns by specificity descending for consistent iteration order
patterns.sort((a, b) => b.specificity - a.specificity);

return { exact, patterns };
};

/**
* Budgets schema with transform for nested-to-flat conversion
* Budgets schema with transform for nested-to-ResolvedBudgets conversion
*
* Input: { [file]: { [suite]: { [task]: Budget } } } Output: { [taskId]: Budget
* } where taskId = "file/suite/task"
* Input: { [file]: { [suite]: { [task]: Budget } } } Output: ResolvedBudgets {
* exact: { [taskId]: Budget }, patterns: BudgetPattern[] }
*/
const budgetsSchema = budgetsInputSchema.transform(flattenBudgets);

Expand Down Expand Up @@ -386,7 +425,7 @@ const baseConfigProperties = {

/** Description for the budgets field */
const budgetsDescription =
'Performance budgets organized by file → suite → task. Budgets define acceptable performance thresholds.';
'Performance budgets organized by file → suite → task. Budgets define acceptable performance thresholds. Supports wildcards (* for suite/task, glob patterns for files).';

/** Description and metadata for the config schema */
const configSchemaDescription =
Expand All @@ -402,7 +441,7 @@ const configSchemaMeta = { title: 'ModestBench Configuration' };
* The budgets field uses transforms to:
*
* 1. Parse string values like "10ms" or "10%" to numbers
* 2. Flatten nested structure to flat taskId → Budget mapping
* 2. Separate exact matches from wildcard patterns into ResolvedBudgets
*/
const modestBenchConfigSchema = z
.object({
Expand Down Expand Up @@ -472,8 +511,8 @@ export const safeParseConfig = (config: unknown) => {
/**
* Configuration type after parsing (output type)
*
* This is the type you get after parsing a config file - budgets are flattened
* to taskId keys and string values are converted to numbers.
* This is the type you get after parsing a config file - budgets are
* transformed to ResolvedBudgets and string values are converted to numbers.
*/
export type ModestBenchConfig = z.infer<typeof modestBenchConfigSchema>;

Expand Down
Loading
Loading