Langfuse Web Failing to start Because ClickHouse Tables in Readonly Mode and Zookeeper Metadata Not Found

[natenberenstein]

Describe your question

I'm experiencing an issue with Langfuse Web component where ClickHouse tables are in readonly mode during startup.

Error Details

• Component: langfuse-web
• Error Message:

Prisma schema loaded from packages/shared/prisma/schema.prisma
Datasource "db": PostgreSQL database "postgres_langfuse", schema "public" at "langfuse-postgresql"
373 migrations found in prisma/migrations
No pending migrations to apply.
error: code: 242, message: Table is in readonly mode since table metadata was not found in zookeeper: replica_path=/clickhouse/tables/a9cf9a7b-378c-4ca0-ace7-419a85b4b7e1/shard0/replicas/langfuse-clickhouse-shard0-2 in line 0: INSERT INTO schema_migrations (version, dirty, sequence) VALUES (?, ?, ?)
Applying clickhouse migrations failed. This is mostly caused by the database being unavailable.
Exiting...
Script executed successfully.
Prisma schema loaded from packages/shared/prisma/schema.prisma
Datasource "db": PostgreSQL database "postgres_langfuse", schema "public" at "langfuse-postgresql"
373 migrations found in prisma/migrations
No pending migrations to apply.
error: code: 242, message: Table is in readonly mode since table metadata was not found in zookeeper: replica_path=/clickhouse/tables/a9cf9a7b-378c-4ca0-ace7-419a85b4b7e1/shard0/replicas/langfuse-clickhouse-shard0-0 in line 0: INSERT INTO schema_migrations (version, dirty, sequence) VALUES (?, ?, ?)
Applying clickhouse migrations failed. This is mostly caused by the database being unavailable.
Exiting... 

Questions

1. What causes Zookeeper metadata to be missing for ClickHouse replicas?
2. How can I recover/restore the Zookeeper metadata for these tables?
3. Is there a way to reset or reinitialize the ClickHouse schema?

Environment Details

• Langfuse chart version - 1.15.19

Discussion

With that said, are there any guidance on resolving this Zookeeper metadata issue? Thank you in advance!

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

1.15.19

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The error indicates that ClickHouse tables are in readonly mode because table metadata was not found in Zookeeper(1). This typically occurs when replicas have corrupted local metadata that prevents them from properly registering with Zookeeper(1).

Recovery Steps

When ClickHouse replicas show persistent is_readonly = 1 and cannot find metadata in Zookeeper, you need to force the affected nodes to clean their local history and re-attach to download parts from healthy replicas(1).

For the affected replicas (shard0-0 and shard0-2 in your case), follow these steps(1):

1. Stop the affected ClickHouse pod
2. Delete the PVC (Persistent Volume Claim) for the corrupted node to clear local metadata
3. Restart the pod - Kubernetes will create a new PVC and the replica will re-initialize from scratch, downloading data from healthy replicas

The commands would be(1):

kubectl delete pod langfuse-clickhouse-shard0-0 -n <namespace>
kubectl delete pvc data-langfuse-clickhouse-shard0-0 -n <namespace>

After deletion, the StatefulSet will automatically recreate the pod with a fresh PVC, allowing it to properly register with Zookeeper and sync from healthy nodes(1).

Important Considerations

Ensure your ClickHouse deployment has replicaCount: 3 (or at least 2 healthy replicas) so data remains available during this recovery process(2). The recommended minimum for production is 3 replicas(2).

The ClickHouse configuration requires specific macros and default_replica settings to be properly configured for replicated tables(2). These should be covered by the Helm chart without additional configuration(2).

Prevention

This issue can also occur from disk space problems(2). Monitor your PVC usage and ensure your storage class has allowVolumeExpansion: true to prevent early capacity issues(2).

📚 Sources:

1. default.traces on the shard0-0 and shard0-1 nodes has persistent, local metadata corruption that prevents it from being properly re-registered.
2. ClickHouse

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @natenberenstein, was inkeep able to help you here?

[natenberenstein]

Hi @jannikmaierhoefer, inkeep was able to help here. Thanks!