New builds don't trigger automatically when a pending build exists

[akshaymankar]

Summary

If a job has pending builds (for example, due to #5935), new builds don't get triggered automatically. But triggering the job manually works.

Steps to reproduce

1. Create a job which has an unsatisfiable input and has trigger: true for at least one get.
2. Trigger the job, watch it being pending forever.
3. Fix the unsatisfiable input
4. Trigger it manually, it will run.
5. Make new version of the trigger available, it will not trigger.

Expected results

The job triggers a new build when a new version of the triggering input work.

Actual results

The job doesn't trigger a new build.

Additional context

We enabled jaeger tracing to see where it was failing. We could isolate the issue to this SQL. It fails to create a build when the job has any pending builds.
This wouldn't have been a problem if we were able to cancel the old build (I see that, that issue is fixed in master), but this would still show up if a retrigger of an old build happens exactly at the same time as a new build is supposed to be triggered.

Triaging info

• Concourse version: v6.5.1
• Browser (if applicable): N/A
• Did this used to work?: Don't know.

Pairing with @flokli

[jamieklassen]

tl;dr - this seems to really be "no builds get automatically created when a job has a stuck pending build"

Hmm, this really seems like a second-order effect of #5935, which is already a failure mode -- in theory, if a pending build exists there should never be a need to automatically trigger a new one. This is because pending builds do not yet have their inputs determined (more specifically they do not yet have a "plan"), so any pending build should automatically get the latest versions of any inputs by the time it goes into the "started" state. Does this make sense?

[akshaymankar]

This is because pending builds do not yet have their inputs determined

Is this true for for pending builds created by "re-run with same inputs"

[jamieklassen]

good question... short answer is "no", I think. 🤯 @clarafu do re-runs spend any time in the pending state? i imagine the plan still needs to be constructed by the buildstarter, despite the fact that the inputs themselves are predetermined. @akshaymankar was your stuck build also a re-run?

[akshaymankar]

Yes, it was stuck on a re-run.

[flokli]

Manually retriggering the job /did/ work and moved that single pending one quite out of sight - so it was not obvious until we digged through the source and manually ran queries for verification.

IMHO this case should be handled, either by having a big fat label somewhere in the web interface stating there's (still?) pending jobs in the queue, or by creating that new job regardless - especially considering retriggering the job manually did work.

[jamieklassen]

The presence of a pending build is indicated by a grey halo on the pipeline page (/teams/:team/pipelines/:pipeline) - I don't think that quite qualifies as a 'big fat label' though. Maybe something on the job page (/teams/:team/pipelines/:pipeline/jobs/:job) or the build page (/teams/:team/pipelines/:pipeline/jobs/:job/builds/:build)?

[jamieklassen]

@akshaymankar @flokli the other take here is that multiple automatically-created pending builds should be allowed -- I'm just pretty sure this would mean a change to some of the semantics of scheduling and I haven't thought through all the consequences.