snuba-eap-items-consumer error after upgrading to 25.11.0 #4052
Description
Self-Hosted Version
25.11.0
CPU Architecture
x86_64
Docker Version
20.10.24+dfsg1
Docker Compose Version
2.40.3
Machine Specification
- My system meets the minimum system requirements of Sentry
Steps to Reproduce
I upgraded from 25.10.0 to 25.11.0 following the docs.
Expected Result
working system
Actual Result
snuba-eap-items-consumer conainer keep restarting
logs say:
snuba-eap-items-consumer-1 | 2025-11-19 12:07:12,209 Initializing Snuba...
snuba-eap-items-consumer-1 | 2025-11-19 12:07:15,205 Snuba initialization took 2.9967825969215482s
snuba-eap-items-consumer-1 | {"timestamp":"2025-11-19T12:07:15.210070Z","level":"INFO","fields":{"message":"Storage: eap_items, ClickHouse Table Name: eap_items_1_local, Message Processor: MessageProcessorConfig { python_class_name: \"EAPItemsProcessor\", python_module: \"snuba.datasets.processors.eap_items_processor\" }, ClickHouse host: clickhouse, ClickHouse port: 9000, ClickHouse HTTP port: 8123, ClickHouse database: default"},"target":"rust_snuba::consumer"}
snuba-eap-items-consumer-1 | {"timestamp":"2025-11-19T12:07:15.210127Z","level":"INFO","fields":{"message":"Starting consumer for \"eap_items\"","storage":"eap_items"},"target":"rust_snuba::consumer"}
snuba-eap-items-consumer-1 | {"timestamp":"2025-11-19T12:07:18.225053Z","level":"INFO","fields":{"message":"Kafka consumer member id: \"rdkafka-25cb0cca-383f-4823-91da-2b101238dee6\""},"target":"sentry_arroyo::backends::kafka"}
snuba-eap-items-consumer-1 | {"timestamp":"2025-11-19T12:07:18.225153Z","level":"INFO","fields":{"message":"New partitions assigned: {Partition { topic: Topic(\"snuba-items\"), index: 0 }: 43915}"},"target":"sentry_arroyo::processing"}
snuba-eap-items-consumer-1 | {"timestamp":"2025-11-19T12:07:20.274594Z","level":"WARN","fields":{"message":"ClickHouse write failed (attempt 1/5): status=400 Bad Request, error=Code: 117. DB::Exception: Cannot parse JSON object here: {}: (while reading the value of key attributes_array): (at row 1)\n: While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.3.6.10034.altinitystable (altinity build))\n"},"target":"rust_snuba::strategies::clickhouse::writer_v2"}
The last line repeats for all attempts and then the container restarts
Event ID
No response