Receiving a 429 Too Many Requests from Prefect Cloud crashes the worker instead of handling gracefully #20488
Description
Bug summary
When submitting a lot of flow runs, the prefect worker (we're using kubernetes work pool) receives a 429 Too Many Requests response from Prefect Cloud. Rather than gracefully backing off and retrying, the worker instead crashes with a stack trace and Kubernetes restarts the prefect worker pod.
The crash is caused by an unhandled 429 in the _submit_run method:
File "prefect/workers/base.py", line 1305, in _submit_run
await self.client.read_deployment(flow_run.deployment_id) ```
When `read_deployment()` gets a 429, it raises `PrefectHTTPStatusError` which propagates up and crashes the worker instead of retrying with backoff.
This looks like a bug in Prefect 3.6.10 - the worker should handle 429 responses gracefully with retry/backoff logic, not crash. The exception is being re-raised in `__aexit__`:
File "prefect/workers/base.py", line 1780, in aexit
raise exceptions[0] from None
### Version info
```Text
Version: 3.6.10
API version: 0.8.4
Python version: 3.13.11
Git commit: 47626aa9
Built: Thu, Jan 08, 2026 07:57 PM
OS/Arch: linux/x86_64
Profile: ephemeral
Server type: cloud
Pydantic version: 2.12.5
Server:
Database: sqlite
SQLite version: 3.50.4
Integrations:
prefect-kubernetes: 0.7.2
prefect-redis: 0.2.8
Additional context
No response