(lcore-1251) added tls e2e tests

(lcore-1251) fixed tls tests & removed other e2e tests for quicker test running

(lcore-1251) restored test_list.txt

(lcore-1251) use `trustme` for certs

(lcore-1251) quick tls server fix

(lcore-1251) removed tags in place of steps

(fix) removed unused code

fix tls config

verified correct llm response

clean

LCORE-1253: Add e2e proxy and TLS networking tests

Add comprehensive end-to-end tests verifying that Llama Stack's
NetworkConfig (proxy, TLS) works correctly through the Lightspeed
Stack pipeline.

Test infrastructure:
- TunnelProxy: Async HTTP CONNECT tunnel proxy that creates TCP
  tunnels for HTTPS traffic. Tracks CONNECT count and target hosts.
- InterceptionProxy: Async TLS-intercepting (MITM) proxy using
  trustme CA to generate per-target server certificates. Simulates
  corporate SSL inspection proxies.

Behave scenarios (tests/e2e/features/proxy.feature):
- Tunnel proxy: Configures run.yaml with NetworkConfig proxy pointing
  to a local tunnel proxy. Verifies CONNECT to api.openai.com:443
  is observed and the LLM query succeeds through the proxy.
- Interception proxy: Configures run.yaml with proxy and custom CA
  cert (trustme). Verifies TLS interception of api.openai.com traffic
  and successful LLM query through the MITM proxy.
- TLS version: Configures run.yaml with min_version TLSv1.2 and
  verifies the LLM query succeeds with the TLS constraint.

Each scenario dynamically generates a modified run-ci.yaml with the
appropriate NetworkConfig, restarts Llama Stack with the new config,
restarts the Lightspeed Stack, and sends a query to verify the full
pipeline.

Added trustme>=1.2.1 to dev dependencies.

LCORE-1253: Add negative tests, TLS/cipher scenarios, and cleanup hooks

Expand proxy e2e test coverage to fully address all acceptance criteria:

AC1 (tunnel proxy):
- Add negative test: LLM query fails gracefully when proxy is unreachable

AC2 (interception proxy with CA):
- Add negative test: LLM query fails when interception proxy CA is not
  provided (verifies "only successful when correct CA is provided")

AC3 (TLS version and ciphers):
- Add TLSv1.3 minimum version scenario
- Add custom cipher suite configuration scenario (ECDHE+AESGCM:DHE+AESGCM)

Test infrastructure:
- Add after_scenario cleanup hook in environment.py that restores
  original Llama Stack and Lightspeed Stack configs after @Proxy
  scenarios. Prevents config leaks between scenarios.
- Use different ports for each interception proxy instance to avoid
  address-already-in-use errors in sequential scenarios.

Documentation:
- Update docs/e2e_scenarios.md with all 7 proxy test scenarios.
- Update docs/e2e_testing.md with proxy-related Behave tags
  (@Proxy, @tunnelproxy, @InterceptionProxy, @TLSVersion, @tlscipher).

LCORE-1253: Address review feedback

Changes requested by reviewer (tisnik) and CodeRabbit:

- Detect Docker mode once in before_all and store as
  context.is_docker_mode. All proxy step functions now use the
  context attribute instead of calling _is_docker_mode() repeatedly.
- Log exception in _restore_original_services instead of silently
  swallowing it.
- Only clear context.services_modified on successful restoration,
  not when restoration fails (prevents leaking modified state).
- Add 10-second timeout to tunnel proxy open_connection to prevent
  stalls on unreachable targets.
- Handle malformed CONNECT port with ValueError catch and 400
  response.

LCORE-1253: Replace tag-based cleanup with Background restore step

Move config restoration from @Proxy after_scenario hook to an
explicit Background Given step. This follows the team convention
that tags are used only for test selection (filtering), not for
triggering behavior.

The Background step "The original Llama Stack config is restored
if modified" runs before every scenario. If a previous scenario
left a modified run.yaml (detected by backup file existence), it
restores the original and restarts services. This handles cleanup
even when the previous scenario failed mid-way.

Removed:
- @Proxy tag from feature file (was triggering after_scenario hook)
- after_scenario hook for @Proxy in environment.py
- _restore_original_services function (replaced by Background step)
- context.services_modified tracking (no hook reads it)

Updated docs/e2e_testing.md: tags documented as selection-only,
not behavior-triggering.

LCORE-1253: Address radofuchs review feedback

Rewrite proxy e2e tests to follow project conventions:

- Reuse existing step definitions: use "I use query to ask question"
  from llm_query_response.py and "The status code of the response is"
  from common_http.py instead of custom query/response steps.
- Split service restart into two explicit Given steps: "Llama Stack
  is restarted" and "Lightspeed Stack is restarted" so restart
  ordering is visible in the feature file.
- Remove local (non-Docker) mode code path. Proxy tests use
  restart_container() exclusively, consistent with the rest of the
  e2e test suite.
- Check specific status code 500 for error scenarios instead of the
  broad >= 400 range.
- Remove custom send_query, verify_llm_response, and
  verify_error_response steps that duplicated existing functionality.

Net reduction: -183 lines from step definitions.

LCORE-1253: Clean up proxy servers between scenarios

Stop proxy servers and their event loops explicitly in the Background
restore step. Previously, proxy daemon threads were left running after
each scenario, causing asyncio "Task was destroyed but it is pending"
warnings at process exit.

The _stop_proxy helper schedules an async stop on the proxy's event
loop, waits for it to complete, then stops the loop. Context
references are cleared so the next scenario starts clean.

LCORE-1253: Stop proxy servers after last scenario in after_feature

Add proxy cleanup in after_feature to stop proxy servers left running
from the last scenario. The Background restore step handles cleanup
between scenarios, but the last scenario's proxies persist until
process exit, causing asyncio "Task was destroyed" warnings.

The cleanup checks for proxy objects on context (no tag check needed)
and calls _stop_proxy to gracefully shut down the event loops.

fixed dup steps

addressed comments

debug

fix

fix

readded tests to test fix

addressed comments

update

added invalid cert test

readded test list

Author: jrobertboos

docker-compose-library.yaml

@@ -30,6 +30,8 @@ services:
         condition: service_healthy
       mock-mcp:
         condition: service_healthy
+      mock-tls-inference:
+        condition: service_healthy
     networks:
       - lightspeednet
     volumes:
@@ -40,6 +42,7 @@ services:
       - ./tests/e2e/rag:/opt/app-root/src/.llama/storage/rag:Z
       - ./tests/e2e/secrets/mcp-token:/tmp/mcp-token:ro
       - ./tests/e2e/secrets/invalid-mcp-token:/tmp/invalid-mcp-token:ro
+      - mock-tls-certs:/certs:ro
     environment:
       # LLM Provider API Keys
       - BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
@@ -113,7 +116,30 @@ services:
       retries: 3
       start_period: 2s
 
+  # Mock TLS inference server for TLS E2E tests
+  mock-tls-inference:
+    build:
+      context: ./tests/e2e/mock_tls_inference_server
+      dockerfile: Dockerfile
+    container_name: mock-tls-inference
+    ports:
+      - "8443:8443"
+      - "8444:8444"
+    networks:
+      - lightspeednet
+    volumes:
+      - mock-tls-certs:/certs
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request,ssl;c=ssl.create_default_context();c.check_hostname=False;c.verify_mode=ssl.CERT_NONE;urllib.request.urlopen('https://localhost:8443/health',context=c)"]
+      interval: 5s
+      timeout: 3s
+      retries: 3
+      start_period: 5s
+
 
 networks:
   lightspeednet:
-    driver: bridge
+    driver: bridge
+
+volumes:
+  mock-tls-certs:

docker-compose.yaml

@@ -25,12 +25,16 @@ services:
     container_name: llama-stack
     ports:
       - "8321:8321"  # Expose llama-stack on 8321 (adjust if needed)
+    depends_on:
+      mock-tls-inference:
+        condition: service_healthy
     volumes:
       - ./run.yaml:/opt/app-root/run.yaml:z
       - ${GCP_KEYS_PATH:-./tmp/.gcp-keys-dummy}:/opt/app-root/.gcp-keys:ro
       - ./lightspeed-stack.yaml:/opt/app-root/lightspeed-stack.yaml:ro
       - llama-storage:/opt/app-root/src/.llama/storage
       - ./tests/e2e/rag:/opt/app-root/src/.llama/storage/rag:z
+      - mock-tls-certs:/certs:ro
     environment:
       - BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
       - TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
@@ -140,9 +144,30 @@ services:
       retries: 3
       start_period: 2s
 
+  # Mock TLS inference server for TLS E2E tests
+  mock-tls-inference:
+    build:
+      context: ./tests/e2e/mock_tls_inference_server
+      dockerfile: Dockerfile
+    container_name: mock-tls-inference
+    ports:
+      - "8443:8443"
+      - "8444:8444"
+    networks:
+      - lightspeednet
+    volumes:
+      - mock-tls-certs:/certs
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request,ssl;c=ssl.create_default_context();c.check_hostname=False;c.verify_mode=ssl.CERT_NONE;urllib.request.urlopen('https://localhost:8443/health',context=c)"]
+      interval: 5s
+      timeout: 3s
+      retries: 3
+      start_period: 5s
+
 
 volumes:
   llama-storage:
+  mock-tls-certs:
 
 networks:
   lightspeednet:

tests/e2e-prow/rhoai/configs/lightspeed-stack-tls.yaml

@@ -0,0 +1,22 @@
+name: Lightspeed Core Service (LCS)
+service:
+  host: 0.0.0.0
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  use_as_library_client: false
+  url: http://${env.E2E_LLAMA_HOSTNAME}:8321
+  api_key: xyzzy
+user_data_collection:
+  feedback_enabled: true
+  feedback_storage: "/tmp/data/feedback"
+  transcripts_enabled: true
+  transcripts_storage: "/tmp/data/transcripts"
+authentication:
+  module: "noop"
+inference:
+  default_provider: tls-openai
+  default_model: mock-tls-model

tests/e2e/configuration/library-mode/lightspeed-stack-tls.yaml

@@ -0,0 +1,21 @@
+name: Lightspeed Core Service (LCS)
+service:
+  host: 0.0.0.0
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  use_as_library_client: true
+  library_client_config_path: run.yaml
+user_data_collection:
+  feedback_enabled: true
+  feedback_storage: "/tmp/data/feedback"
+  transcripts_enabled: true
+  transcripts_storage: "/tmp/data/transcripts"
+authentication:
+  module: "noop"
+inference:
+  default_provider: tls-openai
+  default_model: mock-tls-model

tests/e2e/configuration/server-mode/lightspeed-stack-tls.yaml

@@ -0,0 +1,22 @@
+name: Lightspeed Core Service (LCS)
+service:
+  host: 0.0.0.0
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  use_as_library_client: false
+  url: http://llama-stack:8321
+  api_key: xyzzy
+user_data_collection:
+  feedback_enabled: true
+  feedback_storage: "/tmp/data/feedback"
+  transcripts_enabled: true
+  transcripts_storage: "/tmp/data/transcripts"
+authentication:
+  module: "noop"
+inference:
+  default_provider: tls-openai
+  default_model: mock-tls-model

tests/e2e/features/environment.py

@@ -552,6 +552,15 @@ def after_feature(context: Context, feature: Feature) -> None:
         restart_container("lightspeed-stack")
         remove_config_backup(context.default_config_backup)
 
+    # Restore Lightspeed Stack config if the generic configure_service step switched it
+    backup_path = "lightspeed-stack.yaml.backup"
+    if os.path.exists(backup_path):
+        switch_config(backup_path)
+        remove_config_backup(backup_path)
+        if not context.is_library_mode:
+            restart_container("llama-stack")
+        restart_container("lightspeed-stack")
+
     # Clean up any proxy servers left from the last scenario
     if hasattr(context, "tunnel_proxy") or hasattr(context, "interception_proxy"):
         from tests.e2e.features.steps.proxy import _stop_proxy

tests/e2e/features/steps/proxy.py

@@ -137,10 +137,22 @@ def restore_if_modified(context: Context) -> None:
     _stop_proxy(context, "tunnel_proxy", "proxy_loop")
     _stop_proxy(context, "interception_proxy", "interception_proxy_loop")
 
+    # Check for backups from both proxy and TLS scenarios
+    _LLAMA_STACK_TLS_BACKUP = "run.yaml.tls-backup"
+    backup_to_restore = None
     if os.path.exists(_LLAMA_STACK_CONFIG_BACKUP):
-        print("Restoring original Llama Stack config from backup...")
-        shutil.copy(_LLAMA_STACK_CONFIG_BACKUP, _LLAMA_STACK_CONFIG)
-        os.remove(_LLAMA_STACK_CONFIG_BACKUP)
+        backup_to_restore = _LLAMA_STACK_CONFIG_BACKUP
+    elif os.path.exists(_LLAMA_STACK_TLS_BACKUP):
+        backup_to_restore = _LLAMA_STACK_TLS_BACKUP
+
+    if backup_to_restore:
+        print(f"Restoring original Llama Stack config from {backup_to_restore}...")
+        shutil.copy(backup_to_restore, _LLAMA_STACK_CONFIG)
+        os.remove(backup_to_restore)
+        # Clean up the other backup too if it exists
+        for other_backup in [_LLAMA_STACK_CONFIG_BACKUP, _LLAMA_STACK_TLS_BACKUP]:
+            if other_backup != backup_to_restore and os.path.exists(other_backup):
+                os.remove(other_backup)
         restart_container("llama-stack")
         restart_container("lightspeed-stack")