Resources checked every 1m after upgrade to v6.7.1

[clementss]

We use CONCOURSE_RESOURCE_WITH_WEBHOOK_CHECKING_INTERVAL=24h config combined with global resources to prevent Concourse constantly checking resources every 1m, and prior to the upgrade to v6.7.1 this worked perfectly. NOTE: If a resource has a webhook missing, it defaults to 1m checks to ensure the resource isn't outdated and once spotted a webhook can be added.

The way we do this is using a global-resource-check pipeline which webhooks the resources present in lots of pipelines, so they aren't continuously checked. Using global resources whenever this webhook was triggered the resource would be checked and all pipelines would have an up to date resource. Simple.

After the update to 6.7.1 this no longer works and these global resources are checked every 1m, causing us to hit request limits. Not sure if this is to do with the new version of Concourse, or has somehow got resources stuck into a state where they aren't prioritising the webhook check config.

Single resources used in only one place follow the 24hr check when they are webhooked, but if the resource is used in multiple pipelines it now seem to have the 24hr check overruled by the default of 1m where the resource isn't webhooked.

This was confirmed by creating a new version of the resource in the global-resource-check pipeline by editing the source to create a separate resource, which is webhooked. This correctly is only checked every 24h. But once switching in another pipeline to use this resource, it begins to check every 1m again.

Has anyone noticed similar?

[clementss]

We have since noticed this behaviour above isn't unique to v6.7.1 as on rollback it was also seen. Not sure when this changed.

After investigating further another change in behaviour was spotted which contributed to hitting the Github request limit. Any output resources from a pipeline, e.g. github-release have begun to check every 1m whereas before they were only checked when required. This means pipelines are polling resources a lot more than previously

[chenbh]

The skip checking put-only resources was an optimization introduced in 6.0.0, unfortunately it caused issues when the resource is a custom type and was manually checked at some point (see #5872). The PR to revert it was merged into 6.6.0, but our release-note generator failed to pick it up (#6158). I've gone back and edited the 6.6.0 release notes to include it, sorry for the confusion!

Good news is that a resubmit of the original optimization is targeted for 6.7.2 (but requires a feature flag) in #6271.