Helm deployed ClickHouse doesn't work

[dynek]

Describe your question

I'm using the helm chart to deploy langfuse along with ClickHouse this way:

resource "helm_release" "langfuse" {
  [..]
  set_sensitive {
    name  = "clickhouse.auth.password"
    value = random_password.clickhouse_password.result
  }
}

And a YAML template file containing:

clickhouse:
  deploy: true
  auth:
    username: langfuse

However langfuse-web complains:

error: failed to open database: code: 516, message: langfuse: Authentication failed: password is incorrect, or there is no user with such name. in line 0: SHOW TABLES FROM "default" LIKE 'schema_migrations'
Applying clickhouse migrations failed. This is mostly caused by the database being unavailable.
Exiting...

I'm a bit puzzled as to why, as I'm using the helm chart to deploy ClickHouse and declare environment variables in Langfuse's containers, in which I can see that the following are set:

• CLICKHOUSE_MIGRATION_URL: clickhouse://cfc-langfuse-clickhouse:9000
• CLICKHOUSE_MIGRATION_SSL: "false"
• CLICKHOUSE_URL: http://cfc-langfuse-clickhouse:8123
• CLICKHOUSE_DB: default
• CLICKHOUSE_USER: langfuse
• CLICKHOUSE_PASSWORD: <terraform generated>

I tried with adding a ClickHouse debug container next to langfuse-web, and was able to connect to the server using clickhouse-client with these information.

What could be wrong in Langfuse or my way to achieve this ?

Thank you

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.147.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The authentication error you're experiencing is likely due to a mismatch between the ClickHouse user credentials configured in your Helm deployment and what Langfuse expects.(1)

According to the documentation, the ClickHouse user specified in CLICKHOUSE_USER must have specific grants to allow Langfuse to operate correctly:(1)

GRANT INSERT ON default.* TO 'user';
GRANT SELECT ON default.* TO 'user';
GRANT ALTER UPDATE, ALTER DELETE ON default.* TO 'user';
GRANT CREATE ON default.* TO 'user';
GRANT DROP TABLE ON default.* TO 'user';

(1)

The issue is that you're setting clickhouse.auth.username: langfuse in your Helm values, but the Bitnami ClickHouse Helm Chart creates a user with username default by default.(1) Your Langfuse environment variables show CLICKHOUSE_USER: langfuse, which doesn't match the actual ClickHouse user.

For a minimum production setup with the Bitnami ClickHouse Helm chart, the documentation recommends:(1)

clickhouse:
  deploy: true
  shards: 1 # Fixed: Langfuse does not support multi-shard clusters
  replicaCount: 3
  resourcesPreset: large # or more
  persistence:
    size: 100Gi
  auth:
    username: default
    password: changeme

(1)

Change your Helm configuration to use username: default instead of username: langfuse, or ensure that if you create a custom user named langfuse, it has all the required grants listed above.(1)

Additionally, verify that your Langfuse environment variables match:(1)

CLICKHOUSE_URL=http://<chart-name>-clickhouse:8123
CLICKHOUSE_MIGRATION_URL=clickhouse://<chart-name>-clickhouse:9000
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=<your-password>

(1)

📚 Sources:

1. ClickHouse

---

Have another question? Just tag @inkeep.

[dynek]

I removed username in the template, it recreated the ClickHouse cluster but now:

langfuse-web Script executed successfully.
langfuse-web Prisma schema loaded from packages/shared/prisma/schema.prisma
langfuse-web Datasource "db": PostgreSQL database "langfuse_main", schema "public" at "[..]"
langfuse-web 
langfuse-web 373 migrations found in prisma/migrations
langfuse-web 
langfuse-web 
langfuse-web No pending migrations to apply.
langfuse-web error: failed to open database: code: 194, message: default: Authentication failed: password is incorrect, or there is no user with such name.
langfuse-web 
langfuse-web If you use ClickHouse Cloud, the password can be reset at https://clickhouse.cloud/
langfuse-web on the settings page for the corresponding service.
langfuse-web 
langfuse-web If you have installed ClickHouse and forgot password you can reset it in the configuration file.
langfuse-web The password for default user is typically located at /etc/clickhouse-server/users.d/default-password.xml
langfuse-web and deleting this file will reset the password.
langfuse-web See also /etc/clickhouse-server/users.xml on the server where ClickHouse is installed. in line 0: SHOW TABLES FROM "default" LIKE 'schema_migrations'
langfuse-web Applying clickhouse migrations failed. This is mostly caused by the database being unavailable.
langfuse-web Exiting...

[dynek]

And variable in langfuse-web:

[dynek]

Wow ok, the problem is the password. "plus" sign not url encoded I suppose 🤷
Changed this and it worked.

[jannikmaierhoefer]

Thanks for posting the answer here!