Bug: Project deletion sometimes times out for larger projects #44120
Description
Bug Description
Bug description
When deleting large projects, the action may timeout. The user reports it worked after a few attempts.
This seems related to deletion being sync and timing out. The postgres cleanup in https://github.com/PostHog/posthog/blob/master/posthog/models/team/util.py#L17-L96 runs sync during the request, which can exceed HTTP timeouts for large projects or during peak hours.
Retries eventually succeed likely because the batch deletion operates in autocommit mode, each batch of 10k records commits independently. So even when the HTTP request times out, completed batches are persisted. Each retry finds less data to delete, until eventually the remaining data is small enough to complete within the timeout.
May need to consider making deletion fully async (return 202 Accepted, queue entire deletion as background task), although that may not be trivial since the current flow depends on the sync phase completing before queuing the async ClickHouse deletion.
Relevant files
- https://github.com/PostHog/posthog/blob/master/posthog/api/project.py#L557-L619
- https://github.com/PostHog/posthog/blob/master/posthog/models/team/util.py#L17-L96
- https://github.com/PostHog/posthog/blob/master/posthog/dags/deletes.py
How to reproduce
- Create a project with large amounts of data (millions of persons/events)
- Attempt to delete the project via UI or API
- Observe timeout on first attempt(s), eventual success after retries
Additional context
From: https://posthoghelp.zendesk.com/agent/tickets/46059
Debug info
Kind: bug
Target area: data_management
Report event: http://go/ticketByUUID/37baf243-fdb8-4d05-8af7-7bea93487e4f
Session: https://us.posthog.com/project/sTMFPsFhdP1Ssg/replay/019b7029-98dc-7678-b28e-7cddaade5ff2?t=5476
Exceptions: https://us.posthog.com/project/2/error_tracking?filterGroup=%7B%22type%22%3A%22AND%22%2C%22values%22%3A%5B%7B%22type%22%3A%22AND%22%2C%22values%22%3A%5B%7B%22key%22%3A%22%24session_id%22%2C%22value%22%3A%5B%22019b7029-98dc-7678-b28e-7cddaade5ff2%22%5D%2C%22operator%22%3A%22exact%22%2C%22type%22%3A%22event%22%7D%5D%7D%5D%7D
Location: https://us.posthog.com/project/38814/settings/project-danger-zone
Persons-on-events mode for project: person_id_override_properties_joined