Optimize Dynamic Config Integration Test execution time

[yossiovadia]

Current Status

The Dynamic Config integration test workflow currently takes approximately 115 minutes (1h 55m) to complete on GitHub Actions.

Recent example runs:

• Run #20066711795: 119.6 minutes (2h test execution)
• Run #20064274813: 111.4 minutes (1h 51m test execution)

Goal

Explore reducing CI execution time without sacrificing test quality.

Areas for Investigation

1. Parallel test execution: Currently tests run sequentially (E2E_PARALLEL=false)
2. Stress test load optimization: Evaluate if reduced load for CI (with full tests for nightly) maintains quality
3. Deployment optimization: Review Helm timeouts and wait intervals
4. GitHub Actions optimizations: Caching, timeout settings

Constraints

• Must maintain test quality and bug detection capability
• Should validate changes with local testing first
• Consider separate CI vs. nightly test profiles if needed

Expected Outcome

Reduce execution time from ~115 minutes to 60-90 minutes while maintaining or improving test coverage quality.

[yossiovadia]

intermediate findings:

GitHub CI Test Timings:
============================================================
chat-completions-request              0m  4s (   4.5s)
chat-completions-stress-request      12m 58s ( 778.2s)
domain-classify                      38m 38s (2318.3s)
semantic-cache                        0m  1s (   2.0s)
pii-detection                        14m 40s ( 880.9s)
jailbreak-detection                  39m 52s (2392.3s)
decision-priority-selection           1m  7s (  67.9s)
plugin-chain-execution                2m 19s ( 139.2s)
rule-condition-logic                  0m 20s (  20.1s)
decision-fallback-behavior            0m 17s (  17.2s)
plugin-config-variations              0m 22s (  22.8s)
embedding-signal-routing              1m 53s ( 113.4s)
============================================================
Total ~112m 36s

Top 3 Bottlenecks:
1. jailbreak-detection                  39m 52s ( 35.4%)
2. domain-classify                      38m 38s ( 34.3%)
3. pii-detection                        14m 40s ( 13.0%)