feat: Add CLI options for auth and ssl verify

Adds command-line options to the "guidellm benchmark" command to support
custom authentication headers and skipping SSL verification when
communicating with the target system.

New options:
  --target-header: Allows specifying a custom header(s) for the target.
  --target-skip-ssl-verify: A flag to disable SSL certificate verification for the target.

These options provide more flexibility for benchmarking targets in
various environments, such as those with self-signed certificates or
custom authentication mechanisms. The names were chosen to align with
the existing --target argument to make it clear these apply to requests
made to the target.

The implementation now follows the following precedence for setting request headers:
1. Headers from the `--target-header` CLI option (highest priority).
2. Headers defined in a `.env` file or as environment variables.
3. Default headers derived from other parameters like `api_key`, `organization`, or `project` (lowest
priority).

This commit also adds documentation and tests for these new features,
covering both CLI usage and configuration via environment variables or a
`.env` file.

Author: kdelee

docs/guides/cli.md

@@ -1 +1,17 @@
-# Coming Soon
+# CLI Reference
+
+This page provides a reference for the `guidellm` command-line interface. For more advanced configuration, including environment variables and `.env` files, see the [Configuration Guide](./configuration.md).
+
+## `guidellm benchmark run`
+
+This command is the primary entrypoint for running benchmarks.
+
+### Target Configuration
+
+These options configure how `guidellm` connects to the system under test.
+
+| Option | Description |
+| --- | --- |
+| `--target <URL>` | **Required.** The endpoint of the target system, e.g., `http://localhost:8080`. |
+| `--target-header <HEADER>` | A header to send with requests to the target. This option can be specified multiple times to send multiple headers. The header should be in the format `"Header-Name: Header-Value"`. For example: `--target-header "Authorization: Bearer my-secret-token"` |
+| `--target-skip-ssl-verify` | A flag to disable SSL certificate verification when connecting to the target. This is useful for development environments with self-signed certificates, but should be used with caution in production. |

docs/guides/configuration.md

@@ -1 +1,42 @@
-# Coming Soon
+# Configuration
+
+The `guidellm` application can be configured using command-line arguments, environment variables, or a `.env` file. This page details the file-based and environment variable configuration options.
+
+## Configuration Methods
+
+Settings are loaded with the following priority (highest priority first):
+1.  Command-line arguments.
+2.  Environment variables.
+3.  Values in a `.env` file in the directory where the command is run.
+4.  Default values.
+
+## Environment Variable Format
+
+All settings can be configured using environment variables. The variables must be prefixed with `GUIDELLM__`, and nested settings are separated by a double underscore `__`.
+
+For example, to set the `api_key` for the `openai` backend, you would use the following environment variable:
+```bash
+export GUIDELLM__OPENAI__API_KEY="your-api-key"
+```
+
+### Target Configuration
+
+You can configure the connection to the target system using environment variables. This is an alternative to using the `--target-*` command-line flags.
+
+| Environment Variable | Description | Example |
+| --- | --- | --- |
+| `GUIDELLM__OPENAI__HEADERS` | A JSON string representing a dictionary of headers to send to the target. These headers will override any default headers (like `Authorization` from `api_key`). | `export GUIDELLM__OPENAI__HEADERS='{"Authorization": "Bearer my-token", "X-Custom-Header": "value"}'` |
+| `GUIDELLM__OPENAI__ORGANIZATION` | The OpenAI organization to use for requests. | `export GUIDELLM__OPENAI__ORGANIZATION="org-12345"` |
+| `GUIDELLM__OPENAI__PROJECT` | The OpenAI project to use for requests. | `export GUIDELLM__OPENAI__PROJECT="proj-67890"` |
+| `GUIDELLM__OPENAI__VERIFY_SSL` | Set to `false` or `0` to disable SSL certificate verification. | `export GUIDELLM__OPENAI__VERIFY_SSL=false` |
+
+### Using a `.env` file
+
+You can also place these variables in a `.env` file in your project's root directory:
+
+```dotenv
+# .env file
+GUIDELLM__OPENAI__API_KEY="your-api-key"
+GUIDELLM__OPENAI__HEADERS='{"Authorization": "Bearer my-token"}'
+GUIDELLM__OPENAI__VERIFY_SSL=false
+```

src/guidellm/__main__.py

@@ -13,7 +13,7 @@
 )
 from guidellm.benchmark.entrypoints import benchmark_with_scenario
 from guidellm.benchmark.scenario import GenerativeTextScenario, get_builtin_scenarios
-from guidellm.config import print_config
+from guidellm.config import print_config, settings
 from guidellm.preprocess.dataset import ShortPromptStrategy, process_dataset
 from guidellm.scheduler import StrategyType
 from guidellm.utils import DefaultGroupHandler
@@ -85,6 +85,18 @@ def benchmark():
         "dict with **kwargs."
     ),
 )
+@click.option(
+    "--target-header",
+    "target_headers",
+    multiple=True,
+    help="A header to send to the target, e.g., --target-header 'Authorization: Bearer <token>'. Can be specified multiple times.",
+)
+@click.option(
+    "--target-skip-ssl-verify",
+    is_flag=True,
+    default=False,
+    help="Skip SSL certificate verification when sending requests to the target.",
+)
 @click.option(
     "--model",
     default=GenerativeTextScenario.get_default("model"),
@@ -249,6 +261,8 @@ def run(
     target,
     backend_type,
     backend_args,
+    target_headers,
+    target_skip_ssl_verify,
     model,
     processor,
     processor_args,
@@ -271,6 +285,21 @@ def run(
 ):
     click_ctx = click.get_current_context()
 
+    if target_headers:
+        headers = {}
+        for header in target_headers:
+            if ":" not in header:
+                raise click.BadParameter(
+                    f"Invalid header format: {header}. Expected 'Key: Value'.",
+                    ctx=click_ctx,
+                    param_hint="--target-header",
+                )
+            key, value = header.split(":", 1)
+            headers[key.strip()] = value.strip()
+        settings.openai.headers = headers
+    if target_skip_ssl_verify:
+        settings.openai.verify_ssl = False
+
     overrides = cli_tools.set_if_not_default(
         click_ctx,
         target=target,

src/guidellm/backend/openai.py

@@ -110,20 +110,36 @@ def __init__(
 
         self._model = model
 
+        # Start with default headers based on other params
+        default_headers: dict[str, str] = {}
         api_key = api_key or settings.openai.api_key
-        self.authorization = (
-            f"Bearer {api_key}" if api_key else settings.openai.bearer_token
-        )
+        bearer_token = settings.openai.bearer_token
+        if api_key:
+            default_headers["Authorization"] = f"Bearer {api_key}"
+        elif bearer_token:
+            default_headers["Authorization"] = bearer_token
 
         self.organization = organization or settings.openai.organization
+        if self.organization:
+            default_headers["OpenAI-Organization"] = self.organization
+
         self.project = project or settings.openai.project
+        if self.project:
+            default_headers["OpenAI-Project"] = self.project
+
+        # User-provided headers from CLI override defaults
+        user_headers = settings.openai.headers or {}
+        default_headers.update(user_headers)
+        self.headers = default_headers
+
         self.timeout = timeout if timeout is not None else settings.request_timeout
         self.http2 = http2 if http2 is not None else settings.request_http2
         self.follow_redirects = (
             follow_redirects
             if follow_redirects is not None
             else settings.request_follow_redirects
         )
+        self.verify_ssl = settings.openai.verify_ssl
         self.max_output_tokens = (
             max_output_tokens
             if max_output_tokens is not None
@@ -160,9 +176,7 @@ def info(self) -> dict[str, Any]:
             "timeout": self.timeout,
             "http2": self.http2,
             "follow_redirects": self.follow_redirects,
-            "authorization": bool(self.authorization),
-            "organization": self.organization,
-            "project": self.project,
+            "headers": self.headers,
             "text_completions_path": TEXT_COMPLETIONS_PATH,
             "chat_completions_path": CHAT_COMPLETIONS_PATH,
         }
@@ -383,6 +397,7 @@ def _get_async_client(self) -> httpx.AsyncClient:
                 http2=self.http2,
                 timeout=self.timeout,
                 follow_redirects=self.follow_redirects,
+                verify=self.verify_ssl,
             )
             self._async_client = client
         else:
@@ -394,16 +409,7 @@ def _headers(self) -> dict[str, str]:
         headers = {
             "Content-Type": "application/json",
         }
-
-        if self.authorization:
-            headers["Authorization"] = self.authorization
-
-        if self.organization:
-            headers["OpenAI-Organization"] = self.organization
-
-        if self.project:
-            headers["OpenAI-Project"] = self.project
-
+        headers.update(self.headers)
         return headers
 
     def _params(self, endpoint_type: EndpointType) -> dict[str, str]:

src/guidellm/config.py

@@ -81,10 +81,12 @@ class OpenAISettings(BaseModel):
 
     api_key: Optional[str] = None
     bearer_token: Optional[str] = None
+    headers: Optional[dict[str, str]] = None
     organization: Optional[str] = None
     project: Optional[str] = None
     base_url: str = "http://localhost:8000"
     max_output_tokens: int = 16384
+    verify_ssl: bool = True
 
 
 class Settings(BaseSettings):

tests/unit/backend/test_openai_backend.py

@@ -11,7 +11,7 @@ def test_openai_http_backend_default_initialization():
     backend = OpenAIHTTPBackend()
     assert backend.target == settings.openai.base_url
     assert backend.model is None
-    assert backend.authorization == settings.openai.bearer_token
+    assert backend.headers.get("Authorization") == settings.openai.bearer_token
     assert backend.organization == settings.openai.organization
     assert backend.project == settings.openai.project
     assert backend.timeout == settings.request_timeout
@@ -37,7 +37,7 @@ def test_openai_http_backend_intialization():
     )
     assert backend.target == "http://test-target"
     assert backend.model == "test-model"
-    assert backend.authorization == "Bearer test-key"
+    assert backend.headers.get("Authorization") == "Bearer test-key"
     assert backend.organization == "test-org"
     assert backend.project == "test-proj"
     assert backend.timeout == 10

tests/unit/backend/test_openai_backend_custom_configs.py

@@ -0,0 +1,45 @@
+import pytest
+
+from guidellm.backend import OpenAIHTTPBackend
+from guidellm.config import settings
+
+
+@pytest.mark.smoke
+def test_openai_http_backend_default_initialization():
+    backend = OpenAIHTTPBackend()
+    assert backend.verify_ssl is True
+
+
+@pytest.mark.smoke
+def test_openai_http_backend_custom_ssl_verification():
+    settings.openai.verify_ssl = False
+    backend = OpenAIHTTPBackend()
+    assert backend.verify_ssl is False
+    # Reset the setting
+    settings.openai.verify_ssl = True
+
+
+@pytest.mark.smoke
+def test_openai_http_backend_custom_headers_override():
+    # Set a default api_key, which would normally create an Authorization header
+    settings.openai.api_key = "default-api-key"
+
+    # Set custom headers that override the default Authorization and add a new header
+    openshift_token = "Bearer sha256~xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+    override_headers = {
+        "Authorization": openshift_token,
+        "Custom-Header": "Custom-Value",
+    }
+    settings.openai.headers = override_headers
+
+    # Initialize the backend
+    backend = OpenAIHTTPBackend()
+
+    # Check that the override headers are used
+    assert backend.headers["Authorization"] == openshift_token
+    assert backend.headers["Custom-Header"] == "Custom-Value"
+    assert len(backend.headers) == 2
+
+    # Reset the settings
+    settings.openai.api_key = None
+    settings.openai.headers = None

tests/unit/test_config.py

@@ -142,9 +142,13 @@ def test_settings_with_env_variables(mocker):
             "GUIDELLM__DATASET__PREFERRED_DATA_COLUMNS": '["custom_column"]',
             "GUIDELLM__OPENAI__API_KEY": "env_api_key",
             "GUIDELLM__TABLE_BORDER_CHAR": "*",
+            "GUIDELLM__OPENAI__HEADERS": '{"Authorization": "Bearer env-token"}',
+            "GUIDELLM__OPENAI__VERIFY_SSL": "false",
         },
     )
     settings = Settings()
     assert settings.dataset.preferred_data_columns == ["custom_column"]
     assert settings.openai.api_key == "env_api_key"
     assert settings.table_border_char == "*"
+    assert settings.openai.headers == {"Authorization": "Bearer env-token"}
+    assert settings.openai.verify_ssl is False

tests/unit/test_main.py

@@ -0,0 +1,57 @@
+import pytest
+from click.testing import CliRunner
+
+from guidellm.__main__ import cli
+
+
+@pytest.mark.smoke
+def test_benchmark_run_invalid_header_format():
+    runner = CliRunner()
+    result = runner.invoke(
+        cli,
+        [
+            "benchmark",
+            "run",
+            "--target-header",
+            "invalid-header",
+            "--target",
+            "http://localhost:8000",
+            "--data",
+            "prompt_tokens=1,output_tokens=1",
+            "--rate-type",
+            "constant",
+            "--rate",
+            "1",
+            "--max-requests",
+            "1",
+        ],
+    )
+    assert result.exit_code != 0
+    assert "Invalid header format" in result.output
+
+
+@pytest.mark.smoke
+def test_benchmark_run_valid_header_format():
+    runner = CliRunner()
+    result = runner.invoke(
+        cli,
+        [
+            "benchmark",
+            "run",
+            "--target-header",
+            "Authorization: Bearer my-token",
+            "--target",
+            "http://localhost:8000",
+            "--data",
+            "prompt_tokens=1,output_tokens=1",
+            "--rate-type",
+            "constant",
+            "--rate",
+            "1",
+            "--max-requests",
+            "1",
+        ],
+    )
+    # This will fail because it can't connect to the server,
+    # but it will pass the header parsing, which is what we want to test.
+    assert "Invalid header format" not in result.output