Title: Project Deletion Stuck with ClickHouse Timeout Error - v3.66.1

[achrafelmtakham]

Describe your question

Description

Hello Langfuse Team and Community,
We are experiencing an issue with project deletion in our self-hosted Langfuse deployment. Projects remain stuck in "Project is being deleted" status indefinitely and never complete the deletion process

Problem

When deleting a project via the Langfuse UI:

1. The project status changes to "Project is being deleted"
2. The deletion never completes
3. Projects remain in this state for over 24 hours
4. We have 4 projects stuck across 3 different organizations

Investigation Findings

1. Worker Logs - Repeated Timeout Errors
   2026-01-29T14:55:12.679Z error Queue job project-delete with id XX in project-delete failed Timeout error. 2026-01-29T15:05:12.780Z error Queue job project-delete with id XX in project-delete failed Timeout error.
   The timeout errors repeat approximately every 10 minutes as the job retries.
2. Stack Trace - ClickHouse Client Timeout
   Error: Timeout error. at Socket.onTimeout (@clickhouse/client@1.11.1/.../node_base_connection.js:533:24) at Object.onceWrapper (node:events:638:28) at Socket.emit (node:events:536:35) at Socket._onTimeout (node:net:595:8)
3. Redis Queue Status
   Inspecting the BullMQ queue in Redis shows:
   All 10 retry attempts exhausted (atm: 10)
   failedReason: Timeout error.
   Jobs have exponential backoff configured but keep failing
4. Critical Finding: No Data in ClickHouse
   This is the surprising part. We verified that the stuck projects have ZERO data remaining in ClickHouse:
   SELECT count(*) FROM traces WHERE project_id = '<project_id>'; -- Returns 0 SELECT count(*) FROM observations WHERE project_id = '<project_id>'; -- Returns 0 SELECT count(*) FROM scores WHERE project_id = '<project_id>'; -- Returns 0
   The ClickHouse data appears to have been successfully deleted, but the deletion job still times out.
5. ClickHouse Health

• ClickHouse pod shows normal CPU/memory usage
• Other queries execute successfully and quickly
• No errors in ClickHouse server logs

Questions

1. Why does the deletion timeout even with no data?

• Is the DELETE query performing a full table scan regardless of matching rows?
• Is there another table or operation in the deletion workflow that we're missing?

2. What is the default ClickHouse request timeout?

• Is there an environment variable like CLICKHOUSE_REQUEST_TIMEOUT we can configure?
• What timeout value would you recommend for production deployments?

3. How can we resolve the stuck projects?

• Can we safely remove the failed jobs from Redis?
• Should we manually update the PostgreSQL database to mark projects as deleted?
• Is there a recommended procedure for this situation?

4. Is this a known issue in v3.66.1?

• Has this been fixed in a newer version?
• Are there any patches or workarounds available?

I would greatly appreciate any guidance on

• Understanding the root cause of this timeout behavior
• Steps to resolve the currently stuck projects
• Configuration recommendations to prevent this in the future

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.66.1

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

I cannot find specific information in the available documentation about the ClickHouse timeout configuration for project deletion or the exact deletion workflow behavior you're experiencing in v3.66.1.

However, based on your investigation findings, here are some observations and recommendations:

Regarding Your Stuck Projects:

Since you've confirmed that all data has been successfully deleted from ClickHouse (0 rows in traces, observations, and scores tables), the timeout appears to be occurring during a verification or cleanup step rather than the actual data deletion.

Potential Steps to Resolve:

1. Check Background Migrations Status: The documentation mentions background migrations can affect operations. Check if any background migrations are stuck or queued that might be interfering with the deletion process(1)

2. Review Worker Container Resources: Since the timeout is occurring in the worker container, ensure it has adequate resources (CPU/memory). The documentation indicates that worker containers handle background jobs like deletions(1)

3. Manual Cleanup Consideration: Given that the ClickHouse data is already deleted, you may need to manually clean up the PostgreSQL state and Redis queue entries. However, I recommend reaching out to the Langfuse team through their GitHub Discussions for guidance on the safest approach for your specific version(1)

For Configuration:

I couldn't find documentation of a CLICKHOUSE_REQUEST_TIMEOUT environment variable in the available sources. You may want to check the complete deployment guide for timeout-related configurations(2)

Next Steps:

I recommend posting this detailed issue in the Langfuse GitHub Discussions where the team can provide version-specific guidance and potentially identify if this is a known issue in v3.66.1(1)

📚 Sources:

1. Troubleshooting & FAQ
2. Local Deployment (v2, docker compose)

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @achrafelmtakham, we added improvements to project deletions in v3.147.0.