Background migrations stuck in "Queued" status

[a1expol]

Describe your question

Hello everyone. I recently deployed a new version of Langfuse v3. According to the documentation on migration from v2 to v3, I deployed version 3.29.0. I upgraded Langfuse v2 to 2.92.0. It seems that most of the information migrated, but traces did not, in the background migrations tab all migrations are stuck in the Queued status. There are no errors in the logs, neither in postgres, nor in clickhouse, nor in v2 Langfuse. I also tried changing the value in args for the background_migrations table in postgres - it had no effect. I deleted the table in general, thinking it might help, but no, and now the table does not appear at all. Maybe someone has encountered a similar problem? I would be grateful for the answers.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.29.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Background migrations stuck in "Queued" status typically indicates that the worker container isn't processing the migration queue(1). Here are the key things to check:

1. Verify Worker Container is Running

Background migrations are executed on startup of the worker container and run in the background until completion or failure(1). Check that your Langfuse worker container is running and check the worker container logs for any errors(1).

2. Check Background Migrations Configuration

Background migrations are enabled by default but can be disabled by setting LANGFUSE_ENABLE_BACKGROUND_MIGRATIONS=false(1). Verify this environment variable is not set to false in your worker container.

3. Monitor Through UI and Logs

You can monitor the progress of background migrations within the Langfuse UI by clicking on the Langfuse version tag and selecting "Background Migrations"(1). You can also monitor the progress via the worker container logs(1).

4. Important Note About the background_migrations Table

You mentioned deleting the background_migrations table in Postgres - this was likely not helpful. This table tracks migration state, and deleting it may have caused issues with the migration system recognizing what needs to run.

5. Zero-Downtime Upgrade Consideration

If you performed a zero-downtime upgrade, the documentation recommends disabling background migrations until traffic is shifted to the new instances by setting LANGFUSE_ENABLE_BACKGROUND_MIGRATIONS=false initially, then removing the override or setting it to true afterwards(2).

Next Steps:

• Ensure your worker container is running and not crashing
• Check worker logs for any error messages
• Verify LANGFUSE_ENABLE_BACKGROUND_MIGRATIONS is not set to false
• If you deleted the background_migrations table, you may need to redeploy to recreate it properly

📚 Sources:

1. Background Migrations
2. Migrate Langfuse v2 to v3

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @a1expol, was inkeep able to help you here?

[a1expol]

Hi @a1expol, was inkeep able to help you here?

Hi @jannikmaierhoefer! Unfortunately not :( Container is running, table is not being renew (but I have already redeployed langfuse several times), the parameter for background migrations is forcibly set to true.

[Steffen911]

@a1expol The table is meant to be created only once when the migration happens. I would recommend to create a new Langfuse deployment with your target version and then copy the table definition over into the deployment that you want to run the migration for.
Once this is present, please restart the langfuse-worker containers and share log messages that you see during startup.

[a1expol]

@a1expol The table is meant to be created only once when the migration happens. I would recommend to create a new Langfuse deployment with your target version and then copy the table definition over into the deployment that you want to run the migration for. Once this is present, please restart the langfuse-worker containers and share log messages that you see during startup.

Hi! Redeploying the worker helped, thank you very much! But the next question is what to do if I deleted one of the tables in postgresql, in my case background_migrations, is there a method to cancel the migration and start it again? I tried to create a new table, but the migration does not start, I think because there are already records in _prisma_migrations, but after deleting them from there, an error occurs.

[Steffen911]

@a1expol I think you could create a new Langfuse deployment and then copy the records in there into background_migrations that way, you should probably get the expected state.