Skip to content

Commit 7b07866

Browse files
authored
Support dashed arguments for benchmark args (#441)
## Summary <!-- Include a short paragraph of the changes introduced in this PR. If this PR requires additional context or rationale, explain why the changes are necessary. --> For all arguments in `BenchmarkGenerativeTextArgs` support dashed versions of the name during validation to match the CLI. Additionally alias the argument `data_request_formatter` to `request_type`. ## Details <!-- Provide a detailed list of all changes introduced in this pull request. --> Alias are configured so that if both options are specified, the original name takes precedence. E.g. the order of precedence is `max_seconds > max-seconds`. ~~For `data_request_formatter` the request type alias takes precedence. E.g. `request_type > request-type > data_request_formatter`.~~ Changed this because it caused precedence issues with CLI arg. Order is now `data_request_formatter > data-request-formatter > request_type > request-type`. ## Test Plan The following scenario file can be used to test. Modify which fields are uncommented to test precedence. ```yaml target: <hostname> #request_type: text_completions #request-type: text_completions data_request_formatter: text_completions profile: concurrent rate: 2 max_seconds: 30 data: prompt_tokens: 256 output_tokens: 128 ``` ## Related Issues <!-- Link any relevant issues that this PR addresses. --> - Resolves # --- - [x] "I certify that all code in this PR is my own, except as noted below." ## Use of AI - [ ] Includes AI-assisted code completion - [ ] Includes code generated by an AI application - [ ] Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes `## WRITTEN BY AI ##`)
2 parents ba51acf + 84538d9 commit 7b07866

File tree

1 file changed

+18
-3
lines changed

1 file changed

+18
-3
lines changed

src/guidellm/benchmark/schemas.py

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@
2424

2525
import yaml
2626
from pydantic import (
27+
AliasChoices,
28+
AliasGenerator,
2729
ConfigDict,
2830
Field,
2931
ValidationError,
@@ -1796,9 +1798,8 @@ def create(
17961798
scenario_data = scenario_data["args"]
17971799
constructor_kwargs.update(scenario_data)
17981800

1799-
for key, value in kwargs.items():
1800-
if value != cls.get_default(key):
1801-
constructor_kwargs[key] = value
1801+
# Apply overrides from kwargs
1802+
constructor_kwargs.update(kwargs)
18021803

18031804
return cls.model_validate(constructor_kwargs)
18041805

@@ -1832,6 +1833,14 @@ def get_default(cls: type[BenchmarkGenerativeTextArgs], field: str) -> Any:
18321833
use_enum_values=True,
18331834
from_attributes=True,
18341835
arbitrary_types_allowed=True,
1836+
validate_by_alias=True,
1837+
validate_by_name=True,
1838+
alias_generator=AliasGenerator(
1839+
# Support field names with hyphens
1840+
validation_alias=lambda field_name: AliasChoices(
1841+
field_name, field_name.replace("_", "-")
1842+
),
1843+
),
18351844
)
18361845

18371846
# Required
@@ -1878,6 +1887,12 @@ def get_default(cls: type[BenchmarkGenerativeTextArgs], field: str) -> Any:
18781887
data_request_formatter: DatasetPreprocessor | dict[str, str] | str = Field(
18791888
default="chat_completions",
18801889
description="Request formatting preprocessor or template name",
1890+
validation_alias=AliasChoices(
1891+
"data_request_formatter",
1892+
"data-request-formatter",
1893+
"request_type",
1894+
"request-type",
1895+
),
18811896
)
18821897
data_collator: Callable | Literal["generative"] | None = Field(
18831898
default="generative", description="Data collator for batch processing"

0 commit comments

Comments
 (0)