Skip to content

Commit d3858ef

Browse files
kdeleekdelee
committed
fixup from PR review and add tests
A new unit test has been added to explicitly verify that headers can be removed by setting their value to `None`. Another unit test has been added to ensure that this functionality works correctly when using the CLI. The documentation for the `--backend-args` CLI option has been updated to explicitly mention that headers can be removed by setting their value to `null`.
1 parent 618c149 commit d3858ef

File tree

7 files changed

+159
-71
lines changed

7 files changed

+159
-71
lines changed

‎docs/guides/cli.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -8,29 +8,29 @@ This command is the primary entrypoint for running benchmarks. It has many optio
88

99
### Scenario Configuration
1010

11-
| Option | Description |
12-
| --- | --- |
11+
| Option | Description |
12+
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
1313
| `--scenario <PATH or NAME>` | The name of a builtin scenario or path to a scenario configuration file. Options specified on the command line will override the scenario file. |
1414

1515
### Target and Backend Configuration
1616

1717
These options configure how `guidellm` connects to the system under test.
1818

19-
| Option | Description |
20-
| --- | --- |
21-
| `--target <URL>` | **Required.** The endpoint of the target system, e.g., `http://localhost:8080`. Can also be set with the `GUIDELLM__OPENAI__BASE_URL` environment variable. |
22-
| `--backend-type <TYPE>` | The type of backend to use. Defaults to `openai_http`. |
19+
| Option | Description |
20+
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
21+
| `--target <URL>` | **Required.** The endpoint of the target system, e.g., `http://localhost:8080`. Can also be set with the `GUIDELLM__OPENAI__BASE_URL` environment variable. |
22+
| `--backend-type <TYPE>` | The type of backend to use. Defaults to `openai_http`. |
2323
| `--backend-args <JSON>` | A JSON string for backend-specific arguments. For example: `--backend-args '{"headers": {"Authorization": "Bearer my-token"}, "verify": false}'` to pass custom headers and disable certificate verification. |
24-
| `--model <NAME>` | The ID of the model to benchmark within the backend. |
24+
| `--model <NAME>` | The ID of the model to benchmark within the backend. |
2525

2626
### Data and Request Configuration
2727

2828
These options define the data to be used for benchmarking and how requests will be generated.
2929

30-
| Option | Description |
31-
| --- | --- |
32-
| `--data <SOURCE>` | The data source. This can be a HuggingFace dataset ID, a path to a local data file, or a synthetic data configuration. See the [Data Formats Guide](./data_formats.md) for more details. |
33-
| `--rate-type <TYPE>` | The type of request generation strategy to use (e.g., `constant`, `poisson`, `sweep`). |
34-
| `--rate <NUMBER>` | The rate of requests per second for `constant` or `poisson` strategies, or the number of steps for a `sweep`. |
35-
| `--max-requests <NUMBER>` | The maximum number of requests to run for each benchmark. |
36-
| `--max-seconds <NUMBER>` | The maximum number of seconds to run each benchmark for. |
30+
| Option | Description |
31+
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
32+
| `--data <SOURCE>` | The data source. This can be a HuggingFace dataset ID, a path to a local data file, or a synthetic data configuration. See the [Data Formats Guide](./data_formats.md) for more details. |
33+
| `--rate-type <TYPE>` | The type of request generation strategy to use (e.g., `constant`, `poisson`, `sweep`). |
34+
| `--rate <NUMBER>` | The rate of requests per second for `constant` or `poisson` strategies, or the number of steps for a `sweep`. |
35+
| `--max-requests <NUMBER>` | The maximum number of requests to run for each benchmark. |
36+
| `--max-seconds <NUMBER>` | The maximum number of seconds to run each benchmark for. |

‎docs/guides/configuration.md‎

Lines changed: 20 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,18 @@ The `guidellm` application can be configured using command-line arguments, envir
55
## Configuration Methods
66

77
Settings are loaded with the following priority (highest priority first):
8-
1. Command-line arguments.
9-
2. Environment variables.
10-
3. Values in a `.env` file in the directory where the command is run.
11-
4. Default values.
8+
9+
1. Command-line arguments.
10+
2. Environment variables.
11+
3. Values in a `.env` file in the directory where the command is run.
12+
4. Default values.
1213

1314
## Environment Variable Format
1415

1516
All settings can be configured using environment variables. The variables must be prefixed with `GUIDELLM__`, and nested settings are separated by a double underscore `__`.
1617

1718
For example, to set the `api_key` for the `openai` backend, you would use the following environment variable:
19+
1820
```bash
1921
export GUIDELLM__OPENAI__API_KEY="your-api-key"
2022
```
@@ -23,28 +25,27 @@ export GUIDELLM__OPENAI__API_KEY="your-api-key"
2325

2426
You can configure the connection to the target system using environment variables. This is an alternative to using the `--target-*` command-line flags.
2527

26-
| Environment Variable | Description | Example |
27-
| --- | --- | --- |
28-
| `GUIDELLM__OPENAI__BASE_URL` | The endpoint of the target system. Equivalent to the `--target` CLI option. | `export GUIDELLM__OPENAI__BASE_URL="http://localhost:8080"` |
29-
| `GUIDELLM__OPENAI__API_KEY` | The API key to use for bearer token authentication. | `export GUIDELLM__OPENAI__API_KEY="your-secret-api-key"` |
30-
| `GUIDELLM__OPENAI__BEARER_TOKEN` | The full bearer token to use for authentication. | `export GUIDELLM__OPENAI__BEARER_TOKEN="Bearer your-secret-token"` |
31-
| `GUIDELLM__OPENAI__HEADERS` | A JSON string representing a dictionary of headers to send to the target. These headers will override any default headers. | `export GUIDELLM__OPENAI__HEADERS='{"Authorization": "Bearer my-token"}'` |
32-
| `GUIDELLM__OPENAI__ORGANIZATION` | The OpenAI organization to use for requests. | `export GUIDELLM__OPENAI__ORGANIZATION="org-12345"` |
33-
| `GUIDELLM__OPENAI__PROJECT` | The OpenAI project to use for requests. | `export GUIDELLM__OPENAI__PROJECT="proj-67890"` |
34-
| `GUIDELLM__OPENAI__VERIFY` | Set to `false` or `0` to disable certificate verification. | `export GUIDELLM__OPENAI__VERIFY=false` |
35-
| `GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS` | The default maximum number of tokens to request for completions. | `export GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS=2048` |
28+
| Environment Variable | Description | Example |
29+
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
30+
| `GUIDELLM__OPENAI__BASE_URL` | The endpoint of the target system. Equivalent to the `--target` CLI option. | `export GUIDELLM__OPENAI__BASE_URL="http://localhost:8080"` |
31+
| `GUIDELLM__OPENAI__API_KEY` | The API key to use for bearer token authentication. | `export GUIDELLM__OPENAI__API_KEY="your-secret-api-key"` |
32+
| `GUIDELLM__OPENAI__BEARER_TOKEN` | The full bearer token to use for authentication. | `export GUIDELLM__OPENAI__BEARER_TOKEN="Bearer your-secret-token"` |
33+
| `GUIDELLM__OPENAI__HEADERS` | A JSON string representing a dictionary of headers to send to the target. These headers will override any default headers. | `export GUIDELLM__OPENAI__HEADERS='{"Authorization": "Bearer my-token"}'` |
34+
| `GUIDELLM__OPENAI__ORGANIZATION` | The OpenAI organization to use for requests. | `export GUIDELLM__OPENAI__ORGANIZATION="org-12345"` |
35+
| `GUIDELLM__OPENAI__PROJECT` | The OpenAI project to use for requests. | `export GUIDELLM__OPENAI__PROJECT="proj-67890"` |
36+
| `GUIDELLM__OPENAI__VERIFY` | Set to `false` or `0` to disable certificate verification. | `export GUIDELLM__OPENAI__VERIFY=false` |
37+
| `GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS` | The default maximum number of tokens to request for completions. | `export GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS=2048` |
3638

3739
### General HTTP Settings
3840

3941
These settings control the behavior of the underlying HTTP client.
4042

41-
| Environment Variable | Description |
42-
| --- | --- |
43-
| `GUIDELLM__REQUEST_TIMEOUT` | The timeout in seconds for HTTP requests. Defaults to 300. |
44-
| `GUIDELLM__REQUEST_HTTP2` | Set to `true` or `1` to enable HTTP/2 support. Defaults to true. |
43+
| Environment Variable | Description |
44+
| ------------------------------------ | ------------------------------------------------------------------------------- |
45+
| `GUIDELLM__REQUEST_TIMEOUT` | The timeout in seconds for HTTP requests. Defaults to 300. |
46+
| `GUIDELLM__REQUEST_HTTP2` | Set to `true` or `1` to enable HTTP/2 support. Defaults to true. |
4547
| `GUIDELLM__REQUEST_FOLLOW_REDIRECTS` | Set to `true` or `1` to allow the client to follow redirects. Defaults to true. |
4648

47-
4849
### Using a `.env` file
4950

5051
You can also place these variables in a `.env` file in your project's root directory:

‎docs/guides/data_formats.md‎

Lines changed: 38 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ You can provide a path to a local data file in one of the following formats:
1212
- **Text (.txt)**: A plain text file, where each line is treated as a separate prompt.
1313

1414
If the prompt column cannot be automatically determined, you can specify it using the `--data-args` option:
15+
1516
```bash
1617
--data-args '{"text_column": "my_custom_prompt_column"}'
1718
```
@@ -22,41 +23,45 @@ You can generate synthetic data on the fly by providing a configuration string o
2223

2324
### Configuration Options
2425

25-
| Parameter | Description |
26-
| --- | --- |
27-
| `prompt_tokens` | **Required.** The average number of tokens for the generated prompts. |
28-
| `output_tokens` | **Required.** The average number of tokens for the generated outputs. |
29-
| `samples` | The total number of samples to generate. Defaults to 1000. |
30-
| `source` | The source text to use for generating the synthetic data. Defaults to a built-in copy of "Pride and Prejudice". |
31-
| `prompt_tokens_stdev` | The standard deviation of the tokens generated for prompts. |
32-
| `prompt_tokens_min` | The minimum number of text tokens generated for prompts. |
33-
| `prompt_tokens_max` | The maximum number of text tokens generated for prompts. |
34-
| `output_tokens_stdev` | The standard deviation of the tokens generated for outputs. |
35-
| `output_tokens_min` | The minimum number of text tokens generated for outputs. |
36-
| `output_tokens_max` | The maximum number of text tokens generated for outputs. |
26+
| Parameter | Description |
27+
| --------------------- | --------------------------------------------------------------------------------------------------------------- |
28+
| `prompt_tokens` | **Required.** The average number of tokens for the generated prompts. |
29+
| `output_tokens` | **Required.** The average number of tokens for the generated outputs. |
30+
| `samples` | The total number of samples to generate. Defaults to 1000. |
31+
| `source` | The source text to use for generating the synthetic data. Defaults to a built-in copy of "Pride and Prejudice". |
32+
| `prompt_tokens_stdev` | The standard deviation of the tokens generated for prompts. |
33+
| `prompt_tokens_min` | The minimum number of text tokens generated for prompts. |
34+
| `prompt_tokens_max` | The maximum number of text tokens generated for prompts. |
35+
| `output_tokens_stdev` | The standard deviation of the tokens generated for outputs. |
36+
| `output_tokens_min` | The minimum number of text tokens generated for outputs. |
37+
| `output_tokens_max` | The maximum number of text tokens generated for outputs. |
3738

3839
### Configuration Formats
3940

4041
You can provide the synthetic data configuration in one of three ways:
4142

42-
1. **Key-Value String:**
43-
```bash
44-
--data "prompt_tokens=256,output_tokens=128,samples=500"
45-
```
46-
47-
2. **JSON String:**
48-
```bash
49-
--data '{"prompt_tokens": 256, "output_tokens": 128, "samples": 500}'
50-
```
51-
52-
3. **YAML or Config File:**
53-
Create a file (e.g., `my_config.yaml`):
54-
```yaml
55-
prompt_tokens: 256
56-
output_tokens: 128
57-
samples: 500
58-
```
59-
And use it with the `--data` argument:
60-
```bash
61-
--data my_config.yaml
62-
```
43+
1. **Key-Value String:**
44+
45+
```bash
46+
--data "prompt_tokens=256,output_tokens=128,samples=500"
47+
```
48+
49+
2. **JSON String:**
50+
51+
```bash
52+
--data '{"prompt_tokens": 256, "output_tokens": 128, "samples": 500}'
53+
```
54+
55+
3. **YAML or Config File:** Create a file (e.g., `my_config.yaml`):
56+
57+
```yaml
58+
prompt_tokens: 256
59+
output_tokens: 128
60+
samples: 500
61+
```
62+
63+
And use it with the `--data` argument:
64+
65+
```bash
66+
--data my_config.yaml
67+
```

‎src/guidellm/__main__.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,9 @@ def benchmark():
8282
default=GenerativeTextScenario.get_default("backend_args"),
8383
help=(
8484
"A JSON string containing any arguments to pass to the backend as a "
85-
"dict with **kwargs."
85+
"dict with **kwargs. Headers can be removed by setting their value to "
86+
"null. For example: "
87+
"""'{"headers": {"Authorization": null, "Custom-Header": "Custom-Value"}}'"""
8688
),
8789
)
8890
@click.option(

‎src/guidellm/backend/openai.py‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -131,9 +131,13 @@ def __init__(
131131
default_headers["OpenAI-Project"] = self.project
132132

133133
# User-provided headers from kwargs or settings override defaults
134-
default_headers.update(settings.openai.headers or {})
135-
default_headers.update(headers)
136-
self.headers = {k: v for k, v in default_headers.items() if v is not None}
134+
merged_headers = default_headers.copy()
135+
merged_headers.update(settings.openai.headers or {})
136+
if headers:
137+
merged_headers.update(headers)
138+
139+
# Remove headers with None values for backward compatibility and convenience
140+
self.headers = {k: v for k, v in merged_headers.items() if v is not None}
137141

138142
self.timeout = timeout if timeout is not None else settings.request_timeout
139143
self.http2 = http2 if http2 is not None else settings.request_http2
@@ -142,7 +146,7 @@ def __init__(
142146
if follow_redirects is not None
143147
else settings.request_follow_redirects
144148
)
145-
self.verify = verify or settings.openai.verify
149+
self.verify = verify if verify is not None else settings.openai.verify
146150
self.max_output_tokens = (
147151
max_output_tokens
148152
if max_output_tokens is not None

‎tests/unit/backend/test_openai_backend_custom_configs.py‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,3 +62,27 @@ def test_openai_http_backend_kwarg_headers_override_settings():
6262

6363
# Reset the settings
6464
settings.openai.headers = None
65+
66+
67+
@pytest.mark.smoke
68+
def test_openai_http_backend_remove_header_with_none():
69+
# Set a default api_key, which would normally create an Authorization header
70+
settings.openai.api_key = "default-api-key"
71+
72+
# Set a custom header and explicitly set Authorization to None to remove it
73+
override_headers = {
74+
"Authorization": None,
75+
"Custom-Header": "Custom-Value",
76+
}
77+
78+
# Initialize the backend
79+
backend = OpenAIHTTPBackend(headers=override_headers)
80+
81+
# Check that the Authorization header is removed and the custom header is present
82+
assert "Authorization" not in backend.headers
83+
assert backend.headers["Custom-Header"] == "Custom-Value"
84+
assert len(backend.headers) == 1
85+
86+
# Reset the settings
87+
settings.openai.api_key = None
88+
settings.openai.headers = None

‎tests/unit/test_main.py‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
import json
2+
from pathlib import Path
3+
from unittest.mock import patch
4+
15
import pytest
26
from click.testing import CliRunner
37

@@ -30,3 +34,51 @@ def test_benchmark_run_with_backend_args():
3034
# but it will pass the header parsing, which is what we want to test.
3135
assert result.exit_code != 0
3236
assert "Invalid header format" not in result.output
37+
38+
39+
@patch("guidellm.__main__.benchmark_with_scenario")
40+
def test_cli_backend_args_header_removal(mock_benchmark_func, tmp_path: Path):
41+
"""
42+
Tests that --backend-args from the CLI correctly overrides scenario
43+
values and that `null` correctly removes a header.
44+
"""
45+
scenario_path = tmp_path / "scenario.json"
46+
47+
# Create a scenario file with a header that should be overridden and removed
48+
scenario_content = {
49+
"backend_type": "openai_http",
50+
"backend_args": {"headers": {"Authorization": "should-be-removed"}},
51+
"data": "prompt_tokens=10,output_tokens=10",
52+
"max_requests": 1,
53+
"target": "http://dummy-target",
54+
"rate_type": "synchronous",
55+
"processor": "gpt2",
56+
}
57+
with scenario_path.open("w") as f:
58+
json.dump(scenario_content, f)
59+
60+
runner = CliRunner()
61+
result = runner.invoke(
62+
cli,
63+
[
64+
"benchmark",
65+
"run",
66+
"--scenario",
67+
str(scenario_path),
68+
"--backend-args",
69+
'{"headers": {"Authorization": null, "Custom-Header": "Custom-Value"}}',
70+
],
71+
catch_exceptions=False,
72+
)
73+
74+
assert result.exit_code == 0, result.output
75+
76+
# Assert that benchmark_with_scenario was called with the correct scenario
77+
mock_benchmark_func.assert_called_once()
78+
call_args = mock_benchmark_func.call_args[1]
79+
scenario = call_args["scenario"]
80+
81+
# Verify the backend_args were merged correctly
82+
backend_args = scenario.backend_args
83+
expected_headers = {"Authorization": None, "Custom-Header": "Custom-Value"}
84+
assert backend_args["headers"] == expected_headers

0 commit comments

Comments
 (0)