bug: CLI fix for --load-pattern + --target-qps by viraatc · Pull Request #237 · mlcommons/endpoints

viraatc · 2026-04-01T21:56:51Z

What does this PR do?

Fixes CLI crash when --load-pattern + --target-qps are used together (IndexError: tuple index out of range), and adds test coverage to prevent regressions.

Bug fix

LoadPattern.type used alias= instead of name= on cyclopts.Parameter, and class was missing @cyclopts.Parameter(name="*") — caused cyclopts to fail resolving --load-pattern into a config key path.

Test coverage

test_cli.py: Hypothesis fuzz tests auto-discover all CLI flags from assemble_argument_collection() and test 4000 random combinations (up to 10 flags each) across offline + online/poisson + online/concurrency. Validated: catches this bug in 1.62s.
test_benchmark_command.py: Added test_concurrency_benchmark with streaming on/off — all 3 execution modes now covered.
hypothesis==6.151.10 added to test deps, schema_fuzz pytest marker.

CI & tooling

schema-updated CI job: triggers on PRs touching schema.py/config.py/cli.py — runs fuzz tests + validates YAML templates.
regenerate_templates.py: auto-generates YAML templates from schema defaults + overrides. Pre-commit hook regenerates locally on schema.py changes (skipped in CI).
Templates excluded from prettier to avoid formatting conflicts.

Type of change

Bug fix
Tests added/updated

github-actions · 2026-04-01T21:57:02Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Copilot

Pull request overview

Fixes a cyclopts CLI parsing crash triggered when --load-pattern is combined with load-pattern subfields like --target-qps / --concurrency in the online benchmark command.

Changes:

Annotates LoadPattern to adjust how cyclopts maps nested parameters (@cyclopts.Parameter(name="*")).
Updates the CLI parameter definition for LoadPattern.type to avoid the prior name collision.

Comments suppressed due to low confidence (1)

src/inference_endpoint/config/schema.py:360

This change is a regression fix for a CLI crash when combining --load-pattern with nested load-pattern fields (e.g. --target-qps). There’s existing automated test coverage for config validation in tests/unit/commands/test_benchmark.py, but no test currently exercises cyclopts parsing for this flag combination.

Add a regression test that parses benchmark online ... --load-pattern poisson --target-qps 100 (or directly parses OnlineBenchmarkConfig via cyclopts) and asserts it no longer raises and that config.settings.load_pattern.type/target_qps are set as expected.

@cyclopts.Parameter(name="*")
class LoadPattern(BaseModel):
    """Load pattern configuration.

    Different patterns use target_qps differently:
    - max_throughput: target_qps used for calculating total queries (offline, optional with default)
    - poisson: target_qps sets scheduler rate (online, required - validated)
    - concurrency: issue at fixed target_concurrency (online, required - validated)
    """

    model_config = ConfigDict(extra="forbid", frozen=True)

    type: Annotated[
        LoadPatternType,
        cyclopts.Parameter(name="--load-pattern", help="Load pattern type"),
    ] = LoadPatternType.MAX_THROUGHPUT
    target_qps: Annotated[
        float | None, cyclopts.Parameter(alias="--target-qps", help="Target QPS")
    ] = Field(None, gt=0)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/config/schema.py

gemini-code-assist

Code Review

This pull request modifies the LoadPattern class in the configuration schema by applying a class-level cyclopts.Parameter decorator and updating the type field's parameter definition to use the name argument instead of alias. I have no feedback to provide.

arekay-nv

Can we also add a test for this - seems like a change that shouldn't have gone in.

tests/integration/commands/test_cli.py

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/test.yml

tests/integration/commands/test_cli.py

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/test.yml

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.pre-commit-config.yaml

viraatc

duplicate

tests/integration/commands/test_cli.py

src/inference_endpoint/config/templates/offline_template.yaml

.github/workflows/test.yml

.pre-commit-config.yaml

viraatc · 2026-04-03T11:51:28Z

Review Council — Multi-AI Code Review

Reviewed by: Claude (Codex ran but produced investigation output, not structured findings) | Depth: standard

Found 3 issues across 3 files:

1 high (fixed)
1 medium (already fixed)
1 low (deferred)

#	File	Line	Severity	Category	Summary
1	`scripts/regenerate_templates.py`	95	high	error-handling	Pre-commit hook exited 0 on template generation failure — stale files could slip through. Fixed: now tracks failures and `sys.exit(1)`.
2	`.github/workflows/test.yml`	61	medium	security	Unpinned action SHAs in `schema-updated` job. Already fixed in latest push.
3	`tests/integration/commands/test_cli.py`	76	low	testing	`Optional` union types (`float

Also addressed all Copilot review comments (pinned SHAs, quoted pip install, heredoc for inline Python, expanded pre-commit files: regex, added except comment).

viraatc

added new schema-updated CI:

fuzz tests on CLI in CI
template validated against schema default in CI

NOTE: template now includes all supported fields

was pending items from past.
++ @rashid for thoughts?

viraatc · 2026-04-03T12:16:44Z

tests/integration/commands/test_cli.py

+@pytest.mark.schema_fuzz
+@pytest.mark.slow
+@hyp_settings(max_examples=2000, deadline=5000)
+@given(tokens=online_tokens())


Fuzz test catches 53f08fc

The bug caused --load-pattern poisson --target-qps 100 to crash:

$ inference-endpoint benchmark online \ --endpoints http://localhost:8000 --model m --dataset d.pkl \ --load-pattern poisson --target-qps 100 IndexError: tuple index out of range

Reverted the fix and ran this test — Hypothesis finds it in 1.62s:

E IndexError: tuple index out of range E Falsifying example: test_online_cli_no_crash( E tokens=['benchmark', 'online', '--endpoints', 'http://h:80', E '--model', 'm', '--dataset', 'd.pkl', E '--load-pattern', 'poisson', '--target-qps', '100', E '--name', 'test-val'], E ) ============================== 1 failed in 1.62s ===============================

viraatc · 2026-04-03T12:16:54Z

src/inference_endpoint/config/templates/offline_template.yaml

+type: offline
 model_params:
-  name: "meta-llama/Llama-3.1-8B-Instruct"
+  name: '<MODEL_NAME eg: meta-llama/Llama-3.1-8B-Instruct>'


Templates auto-generated from schema defaults by scripts/regenerate_templates.py.
Full YAML spec with placeholder overrides (model name, dataset)

Pre-commit validates templates are valid locally.
CI checks if they're up to date — if stale it will suggest to, run python scripts/regenerate_templates.py.

Is this overkill? Should we drop?

viraatc · 2026-04-03T12:17:02Z

.github/workflows/test.yml

          pip install -e ".[dev,test,performance]"
          pip-audit
+
+  schema-updated:


new schema-updated CI job:
triggers on PRs touching schema.py/config.py/cli.py.

viraatc · 2026-04-03T12:17:11Z

.pre-commit-config.yaml

-      - id: validate-templates
-        name: Validate YAML templates against schema
-        entry: python -c "from pathlib import Path; from inference_endpoint.config.schema import BenchmarkConfig; [BenchmarkConfig.from_yaml_file(f) for f in sorted(Path('src/inference_endpoint/config/templates').glob('*.yaml'))]"
+      - id: check-templates


reuse --check mode

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/config/templates/concurrency_template.yaml

tests/integration/commands/test_cli.py

.github/workflows/test.yml

docs/DEVELOPMENT.md

Bug: LoadPattern.type had alias= instead of name= on cyclopts.Parameter, and class was missing @cyclopts.Parameter(name="*"). This caused any CLI invocation with --load-pattern to crash with IndexError. Tests: - Hypothesis fuzz tests auto-discover all CLI flags from cyclopts assemble_argument_collection() and test 4000 random combinations (offline + online/poisson + online/concurrency) - Added test_concurrency_benchmark with streaming on/off - hypothesis==6.151.10 added to test deps, schema_fuzz pytest marker CI & tooling: - schema-updated CI job: fuzz tests + template validation on schema changes - regenerate_templates.py: auto-generates YAML templates from schema defaults - Pre-commit checks templates are up to date (--check mode) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings April 1, 2026 21:56

viraatc requested a review from a team as a code owner April 1, 2026 21:56

github-actions bot requested review from arekay-nv and nvzhihanj April 1, 2026 21:57

Copilot started reviewing on behalf of viraatc April 1, 2026 21:57 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

src/inference_endpoint/config/schema.py Show resolved Hide resolved

gemini-code-assist bot reviewed Apr 1, 2026

View reviewed changes

arekay-nv approved these changes Apr 3, 2026

View reviewed changes

github-code-quality bot found potential problems Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Fixed Show fixed Hide fixed

viraatc force-pushed the feat/viraatc-fix1 branch from 90fe9c8 to 80a79ef Compare April 3, 2026 10:56

Copilot AI review requested due to automatic review settings April 3, 2026 10:56

Copilot started reviewing on behalf of viraatc April 3, 2026 10:56 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

github-code-quality bot found potential problems Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Dismissed Show dismissed Hide dismissed

Copilot AI review requested due to automatic review settings April 3, 2026 11:05

Copilot started reviewing on behalf of viraatc April 3, 2026 11:06 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

.github/workflows/test.yml Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 3, 2026 11:32

Copilot started reviewing on behalf of viraatc April 3, 2026 11:32 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

This comment was marked as duplicate.

Sign in to view

viraatc commented Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Show resolved Hide resolved

src/inference_endpoint/config/templates/offline_template.yaml Outdated Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

.pre-commit-config.yaml Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 3, 2026 12:14

Copilot started reviewing on behalf of viraatc April 3, 2026 12:15 View session

viraatc force-pushed the feat/viraatc-fix1 branch from 8915750 to ffb87d9 Compare April 3, 2026 12:16

viraatc commented Apr 3, 2026

View reviewed changes

Copilot AI reviewed Apr 3, 2026

View reviewed changes

viraatc force-pushed the feat/viraatc-fix1 branch from ffb87d9 to b781ff7 Compare April 3, 2026 12:21

Conversation

viraatc commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Bug fix

Test coverage

CI & tooling

Type of change

Uh oh!

github-actions bot commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

arekay-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

viraatc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

viraatc commented Apr 3, 2026

Review Council — Multi-AI Code Review

Uh oh!

viraatc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viraatc Apr 3, 2026

Choose a reason for hiding this comment

Fuzz test catches 53f08fc

Uh oh!

viraatc Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viraatc Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viraatc Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

viraatc commented Apr 1, 2026 •

edited

Loading

github-actions bot commented Apr 1, 2026 •

edited

Loading

viraatc left a comment •

edited

Loading

viraatc left a comment •

edited

Loading

Fuzz test catches `53f08fc`

viraatc Apr 3, 2026 •

edited

Loading

viraatc Apr 3, 2026 •

edited

Loading

viraatc Apr 3, 2026 •

edited

Loading