Skip to content

Race condition panic in streaming output detectionΒ #522

@ckavili

Description

@ckavili

Describe the bug

Streaming chat completions with output detection causes a panic due to a race condition between concurrent tasks. The completion_state.completions.get(&choice_index).unwrap() call fails because the spawned task hasn't inserted the entry before the main task tries to read it.

Platform

  • Orchestrator version: 0.16.0 (commit a220c15)
  • Rust version: (from container image)
  • Environment: OpenShift, detectors running in same cluster (~8ms response time)

Sample Code

Request payload that triggers the bug:

{
  "model": "llama32",
  "messages": [{"role": "user", "content": "hello"}],
  "stream": true,
  "detectors": {
    "input": {"hap": {}, "language_detection": {}, "prompt_injection": {}},
    "output": {"hap": {}, "language_detection": {}}
  }
}

Expected behavior

Streaming response with output detection completes without error.

Observed behavior

Panic on first request:

thread 'tokio-runtime-worker' panicked at src/orchestrator/handlers/chat_completions_detection/streaming.rs:573:76:
called `Option::unwrap()` on a `None` value

Happens consistently, not intermittently. Disabling output detectors fixes it.

Additional context

Race is between:

  • process_chat_completion_stream (spawned): inserts into completion_state.completions
  • process_detection_batch_stream (main): reads from completion_state.completions

Affected lines: streaming.rs:506 and streaming.rs:589

Workaround: use stream: false or disable output detectors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions