Elastalert: Run time was missed

[Sahale]

Hey!
SO 2.3.220, distributed setup.
I faced with such problem: sometime elastalert rules run with delay. Delay can be different: from minutes to even hours.
How I found it: some of alerts appeared in SOC interface much later than events were ingested. IE event happened in 15.00, alert in SOC created in 16.30.
I didn't find something strange in logs, except elastalert logs with this:

2023-04-20 09:44:28,113  WARNING apscheduler.executors.default Run time of job "Rule: NNN (trigger: interval[0:03:00], next run at: 2023-04-20 09:47:25 UTC)" was missed by 0:00:05.073835
2023-04-20 09:44:28,119  WARNING apscheduler.executors.default Run time of job "Rule: XXX (trigger: interval[0:03:00], next run at: 2023-04-20 09:47:27 UTC)" was missed by 0:00:05.076283
2023-04-20 09:44:28,125  WARNING apscheduler.executors.default Run time of job "Rule: YYY (trigger: interval[0:03:00], next run at: 2023-04-20 09:47:23 UTC)" was missed by 0:00:05.016870
2023-04-20 09:44:28,132  WARNING apscheduler.executors.default Run time of job "Rule: ZZZ (trigger: interval[0:03:00], next run at: 2023-04-20 09:47:25 UTC)" was missed by 0:00:05.016532 

What can cause this problem?

[Sahale]

Well, as a kind of thoughts: we use heavy-nodes and /nsm partition in some of them is not fast enough. It looks like if one heavy-node didn't respond on elastalert query, whole rule become delayed.
Due to there's no alert in web-interface, it's important to check elastalert logs for such problems.
Ignoring "unresponded" nodes with elastalert is a bad idea, because critical events can be lost.