Skip to content

[Bug]failed to process message: failed to decode event batch when aibrix-gateway-plugins enable AIBRIX_KV_EVENT_SYNC #1720

@Bevisy

Description

@Bevisy

🐛 Describe the bug

error message

I1031 06:10:26.097871       1 gateway.go:94] "processing request" requestID="b826c66e-f8c8-48a9-b5e4-39daca741c67"
I1031 06:10:26.103904       1 gateway_req_body.go:75] "request start" requestID="b826c66e-f8c8-48a9-b5e4-39daca741c67" requestPath="/v1/chat/completions" model="deepseek-r1-distill-llama-8b" stream=false
E1031 06:10:26.175809       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:26.175817       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:27.176961       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:27.243634       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:27.243640       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:28.244455       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:28.321206       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:28.321216       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:29.322189       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:29.405118       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:29.405124       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:30.405709       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:30.490204       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:30.490214       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:31.490696       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:31.577800       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:31.577818       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:32.577936       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:32.667135       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:32.667142       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:33.668012       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:33.757004       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:33.757012       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:34.757265       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:34.847027       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:34.847035       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:35.847699       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:35.937384       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:35.937393       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:36.938080       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:37.028746       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:37.028755       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:38.029074       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:38.120026       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:38.120033       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:39.120952       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:39.212065       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:39.212075       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:40.212466       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:40.305438       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:40.305446       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:41.305667       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:41.400401       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:41.400414       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:42.400931       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:42.497127       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:42.497137       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:43.497250       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117
E1031 06:10:43.594125       1 zmq_client.go:214] Event consumption error for default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8: failed to process message: failed to decode event batch: failed to unmarshal event batch: msgpack : invalid code 93 decoding map
I1031 06:10:43.594131       1 zmq_client.go:224] Attempting to reconnect to default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 after 1s
I1031 06:10:44.572151       1 gateway_rsp_body.go:189] request end, requestID: b826c66e-f8c8-48a9-b5e4-39daca741c67 -
I1031 06:10:44.594296       1 zmq_client.go:136] Successfully connected to vLLM pod default/deepseek-r1-distill-llama-8b-79ff779c9c-nb6g8 at 172.16.149.117

Steps to Reproduce

  1. modify gateway-plugins to enable kv_events
# v0.4.1
kubectl set env deployment/aibrix-gateway-plugins -n aibrix-system \
  AIBRIX_PREFIX_CACHE_METRICS_ENABLED=true \
  AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE=remote \
  AIBRIX_USE_REMOTE_TOKENIZER=true \
  AIBRIX_REMOTE_TOKENIZER_ENDPOINT="http://deepseek-r1-distill-llama-8b.default.svc.cluster.local:8000"
# v0.4.1
kubectl set env deployment/aibrix-gateway-plugins -n aibrix-system \
  AIBRIX_KV_EVENT_SYNC_ENABLED=true
  1. label vllm pod
labels:
  model.aibrix.ai/kv-events-enabled: "true"
  1. request /v1/chat/completions

  2. watch gateway-plugins log error

docs refer: https://aibrix.readthedocs.io/latest/features/kv-event-sync.html

Expected behavior

  1. kv_events sync success

Environment

  • AIBrix 0.4.1
  • Kubernetes
  • LLM 0.9.1
  • Docker Image: vllm-openai-aibrix-kvcache:v0.9.1-20250724

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions