Fatal error: Task disappeared from cluster's queue

Summary

If you always get Task disappeared from cluster's queue fatal error, you need to check that pidRegex, pidRegexCheckTaskRunning and pidColumnCheckTaskRunning are properly configured in your bds.config

Task disappear or bad configuration?

Sometimes clusters fail in ways that the cluster management system is unable to detect, let alone report the error. It can happen that tasks disappear without any trace from the cluster (this is not as rare as you may think, particularly when executing thousands of tasks per pipeline). For this reason, bds performs active monitoring, to ensure that tasks are still alive. If any task "mysteriously disappears", bds reports the problem and considers the task as failed.

Bad config: Sometimes, the mechanism that bds uses to check tasks is not properly configured in your bds.config. In this case, bds is unable to find the tasks even though they are running on the cluster which leasd to the fatal error Task disappeared from cluster's queue