Skip to content

Commit b05b0e7

Browse files
authored
Fix: Resolve CSV output generation failure in benchmarks (#435)
## Summary This PR fixes two critical bugs that cause the `guidellm benchmark` command to crash when using the `--output-formats csv` option. These changes ensure that CSV reports are generated reliably and correctly. ## Details 1. Fix `AttributeError`**: In `src/guidellm/benchmark/output.py`, a `hasattr` check has been added to the `_get_benchmark_status_metrics_stats` function. This check ensures that the code only attempts to access status attributes (e.g., `successful`) on metric objects that actually possess them (like `StatusDistributionSummary`). It safely skips composite metrics such as `GenerativeTextMetricsSummary`, thus preventing the `AttributeError`. 2. Fix `KeyError`**: The data flow for benchmark compilation has been refactored to ensure the original data configuration (`args.data`) is correctly propagated. The method signatures for `Benchmarker.run` (in `benchmarker.py`) and `GenerativeBenchmark.compile` (in `schemas.py`) were updated to accept a `data` parameter. `args.data` is now passed in during the `benchmarker.run` call within `benchmark_generative_text` (in `entrypoints.py`). As a result, the `benchmark.benchmarker.requests` field in the final `GenerativeBenchmark` object now directly stores the raw data configuration, which resolves the `KeyError` in `output.py`. ## Test Plan guidellm benchmark --target "http://x.x.x.x:x/" --processor "Qwen/Qwen3-8B-FP8" --rate-type "concurrent" --rate 1 --max-requests 10 --data "prompt_tokens=32,output_tokens=32,samples=1" --output-formats csv ## Related Issues #434 - Resolves # --- - [x] "I certify that all code in this PR is my own, except as noted below." ## Use of AI - [ ] Includes AI-assisted code completion - [ ] Includes code generated by an AI application - [ ] Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes `## WRITTEN BY AI ##`)
2 parents 4d6f9d4 + e818ae5 commit b05b0e7

File tree

4 files changed

+8
-2
lines changed

4 files changed

+8
-2
lines changed

src/guidellm/benchmark/benchmarker.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ async def run(
5757
backend: BackendInterface[RequestT, ResponseT],
5858
profile: Profile,
5959
environment: Environment,
60+
data: list[Any],
6061
progress: BenchmarkerProgress[BenchmarkT] | None = None,
6162
sample_requests: int | None = 20,
6263
warmup: float | None = None,
@@ -149,6 +150,7 @@ async def run(
149150
environment=environment,
150151
strategy=strategy,
151152
constraints=constraints,
153+
data=data,
152154
)
153155
if progress:
154156
await progress.on_benchmark_complete(benchmark)

src/guidellm/benchmark/entrypoints.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -436,6 +436,7 @@ async def benchmark_generative_text(
436436
backend=backend,
437437
profile=profile,
438438
environment=NonDistributedEnvironment(),
439+
data=args.data,
439440
progress=progress,
440441
sample_requests=args.sample_requests,
441442
warmup=args.warmup,

src/guidellm/benchmark/output.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -649,6 +649,8 @@ def _get_benchmark_status_metrics_stats(
649649
status_dist_summary: StatusDistributionSummary = getattr(
650650
benchmark.metrics, metric
651651
)
652+
if not hasattr(status_dist_summary, status):
653+
return [], []
652654
dist_summary: DistributionSummary = getattr(status_dist_summary, status)
653655

654656
headers = [
@@ -688,7 +690,7 @@ def _get_benchmark_extras_headers_and_values(
688690
values: list[str] = [
689691
benchmark.benchmarker.profile.model_dump_json(),
690692
json.dumps(benchmark.benchmarker.backend),
691-
json.dumps(benchmark.benchmarker.requests["attributes"]["data"]),
693+
json.dumps(benchmark.benchmarker.requests["data"]),
692694
]
693695

694696
if len(headers) != len(values):

src/guidellm/benchmark/schemas.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1674,6 +1674,7 @@ def compile(
16741674
environment: Environment,
16751675
strategy: SchedulingStrategy,
16761676
constraints: dict[str, dict[str, Any]],
1677+
data: list[Any],
16771678
) -> GenerativeBenchmark:
16781679
"""
16791680
Compile final generative benchmark from accumulated state.
@@ -1702,7 +1703,7 @@ def compile(
17021703
),
17031704
benchmarker=BenchmarkerDict(
17041705
profile=profile,
1705-
requests=InfoMixin.extract_from_obj(requests),
1706+
requests={"data": data},
17061707
backend=backend.info,
17071708
environment=environment.info,
17081709
),

0 commit comments

Comments
 (0)