Skip to content

Add backend_kwargs to generate_benchmark_report #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 28, 2025
+4 −1
Merged

Add backend_kwargs to generate_benchmark_report #78

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0226850
Add backend_kwargs
jackcook Feb 26, 2025
5711f1d
Avoid mutable arguments
jackcook Feb 26, 2025
7e2003b
Merge branch 'main' into main
markurtz Feb 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion src/guidellm/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import asyncio
from typing import Literal, Optional, Union, get_args
from typing import Any, Literal, Mapping, Optional, Union, get_args

import click
from loguru import logger
Expand Down Expand Up @@ -197,6 +197,7 @@ def generate_benchmark_report(
max_requests: Union[Literal["dataset"], int, None],
output_path: str,
cont_refresh_table: bool,
backend_kwargs: Mapping[str, Any] = {},
) -> GuidanceReport:
"""
Generate a benchmark report for a specified backend and dataset.
Expand All @@ -218,6 +219,7 @@ def generate_benchmark_report(
:param output_path: Path to save the output report file.
:param cont_refresh_table: Continually refresh the table in the CLI
until the user exits.
:param backend_kwargs: Additional keyword arguments for the backend.
"""
logger.info(
"Generating benchmark report with target: {}, backend: {}", target, backend
Expand All @@ -228,6 +230,7 @@ def generate_benchmark_report(
backend_type=backend,
target=target,
model=model,
**backend_kwargs,
)

request_generator: RequestGenerator
Expand Down