Manual triggers picked by scheduler with delay #59526
Replies: 4 comments
-
|
@devgonvarun is this independent of number of scheduled tasks at the time of trigger? |
Beta Was this translation helpful? Give feedback.
-
|
Not too concrete issue to work on. some more details would be needed, regarding configuration, log, tracking circumstances and logs showing sequence of events, also memory, database, networking etc. might have impact. There is one thing to have wish for things to be faster, and another to look at your system and deployment and provide some data that anyone who might look at it can act on. Converting to a discussion. |
Beta Was this translation helpful? Give feedback.
-
|
The problem was that the Postgres db that was being used as metadata had the lag of 2 minutes in the clock which resulted in the delay. |
Beta Was this translation helpful? Give feedback.
-
|
Glad to know it . |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Apache Airflow version
main (development)
If "Other Airflow 2/3 version" selected, which one?
3.1.1
What happened?
There is always a delay of around 2 minutes between manual dag triggers from UI and scheduler picking up the dag despite only a configured heartbeat of 2s. KubernetesExecutor. On dag trigger the dag goes into Queued state and changes to Running state once the worker pod is spawned. The task is still queued at this point until the worker pod gets into the running state. But it takes 2 minutes for the worker pod to spawn.
What you think should happen instead?
Scheduler should not have large delays to spawn worker pod.
How to reproduce
Airflow in kubernetes installation. with KubernetesExecutor. Trigger a dag manually from the UI.
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions