Manual triggers picked by scheduler with delay

[devgonvarun]

Apache Airflow version

main (development)

If "Other Airflow 2/3 version" selected, which one?

3.1.1

What happened?

There is always a delay of around 2 minutes between manual dag triggers from UI and scheduler picking up the dag despite only a configured heartbeat of 2s. KubernetesExecutor. On dag trigger the dag goes into Queued state and changes to Running state once the worker pod is spawned. The task is still queued at this point until the worker pod gets into the running state. But it takes 2 minutes for the worker pod to spawn.

What you think should happen instead?

Scheduler should not have large delays to spawn worker pod.

How to reproduce

Airflow in kubernetes installation. with KubernetesExecutor. Trigger a dag manually from the UI.

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[itsUtkarshOjha]

@devgonvarun is this independent of number of scheduled tasks at the time of trigger?
Any idea how many tasks would have been in scheduled state when this occured?

[potiuk]

Not too concrete issue to work on. some more details would be needed, regarding configuration, log, tracking circumstances and logs showing sequence of events, also memory, database, networking etc. might have impact.

There is one thing to have wish for things to be faster, and another to look at your system and deployment and provide some data that anyone who might look at it can act on.

Converting to a discussion.

[devgonvarun]

The problem was that the Postgres db that was being used as metadata had the lag of 2 minutes in the clock which resulted in the delay.

[potiuk]

Glad to know it .