-
Notifications
You must be signed in to change notification settings - Fork 38
Open
Description
Describe the bug
Streaming chat completions with output detection causes a panic due to a race condition between concurrent tasks. The completion_state.completions.get(&choice_index).unwrap() call fails because the spawned task hasn't inserted the entry before the main task tries to read it.
Platform
- Orchestrator version: 0.16.0 (commit a220c15)
- Rust version: (from container image)
- Environment: OpenShift, detectors running in same cluster (~8ms response time)
Sample Code
Request payload that triggers the bug:
{
"model": "llama32",
"messages": [{"role": "user", "content": "hello"}],
"stream": true,
"detectors": {
"input": {"hap": {}, "language_detection": {}, "prompt_injection": {}},
"output": {"hap": {}, "language_detection": {}}
}
}Expected behavior
Streaming response with output detection completes without error.
Observed behavior
Panic on first request:
thread 'tokio-runtime-worker' panicked at src/orchestrator/handlers/chat_completions_detection/streaming.rs:573:76:
called `Option::unwrap()` on a `None` value
Happens consistently, not intermittently. Disabling output detectors fixes it.
Additional context
Race is between:
process_chat_completion_stream(spawned): inserts intocompletion_state.completionsprocess_detection_batch_stream(main): reads fromcompletion_state.completions
Affected lines: streaming.rs:506 and streaming.rs:589
Workaround: use stream: false or disable output detectors.
Metadata
Metadata
Assignees
Labels
No labels