every-1m time does not run every 1 minute

[michaelpporter]

Describe the bug

We have a few jobs that should run every minute, we have noticed that it does not always run in that window.

- name: every-1m
  type: time
  icon: clock-outline
  source:
    interval: 1m

As you can see by the screenshot, this one has waited over 5 min, it was over 6 before it ran.

Is this caused by not having workers ready or not enough resources on web?

Reproduction steps

1. A time resource with a interval: 1m

...

Expected behavior

Expected to run every 60 seconds ± 20 seconds

Additional context

We have 5 web servers

      cpu: "4"
      memory: "12Gi"

And 8 workers for this team

      cpu: "8"
      memory: "24Gi"

[taylorsilva]

This happens because Concourse has a default "check interval". That interval determines how often a check will be performed for all resources in your cluster.

The default value is 1min, but it appears your operator has set it to a higher value. You can override this in your pipeline by setting check_every (https://concourse-ci.org/resources.html#schema.resource.check_every) to 1min or even 30s. I wouldn't go lower than 30s as it'll likely have no effect on the check frequency (more intervals!).

[michaelpporter]

@taylorsilva I am not seeing a global setting for the check_every, we do have some get repos that are set to every 2h.

I have updated our 1-minute tasks; it has helped in some cases, but not all. I will keep checking it out, maybe we need to add check_every to all time resources.

  - name: every-1m
    type: time
    icon: clock-outline
    source:
      interval: 1m
    check_every: 1m

[michaelpporter]

checked | 9m 37s ago
Something is off.

@taylorsilva do you have tips on containerPlacementStrategy or any other settings to check?

I'm seeing 4h time resources checked more than the 1m resources.

[taylorsilva]

You haven't stated what version of Concourse you are on, but maybe you're seeing another side-effect of this issue which was fixed in 7.13: #9103

[taylorsilva]

Metrics would be helpful here for the web nodes. There is a max checks per interval setting. Maybe you guys have a lot of pipelines+resources?

You can review all setting by running:

docker run -t concourse/concourse web --help

Try setting the check_every to 30s instead, see if that helps.

---

If you want me to actually get hands-on and help you really troubleshoot you can each out to me here for commercial support: https://pixelair.io/contact/

[michaelpporter]

We are running v7.13.1

• 5 web pods
• 24 workers across 7 teams, teams have their own pods
• Last week, we had 137,000 jobs completed.
• The DB has 1.2K queries a minute
• Using New Relic, we are seeing 120-478 concourse_prod_checks_enqueued with the default of 1m

I think this was the settings (Helm chart) maxChecksPerSecond: 0 and resourceCheckingInterval: 1m

this will be calculated as (# of resources)/(resource checking interval).

I have changed the settings to maxChecksPerSecond: -1 and resourceCheckingInterval: 30s

We also had

    apiMaxConns: 10
    backendMaxConns: 50

I think we need to audit and update the check_every. Some of our checks are once a day, or every 4 hours. Updating them to 10m should free up the resource checks that need to run more often.

[michaelpporter]

I just now thought, I should set the default check to 5m and update the 1m jobs with the check_every setting

[michaelpporter]

We adjusted web settings:

    maxChecksPerSecond: -1
    resourceCheckingInterval: 5m
    resourceTypeCheckingInterval: 5m
    lidarScannerInterval: 30s

We updated 1m time resources:

resources:
  - name: every-1m
    type: time
    icon: clock-outline
    source:
      interval: 1m
    check_every: 30s

We gave the DB more resources db-custom-4-16384 from db-custom-2-7680 (Google Cloud)

Results from one every-1m resource

time	2025-05-16T13:48:51.832106318Z
time	2025-05-16T13:47:13.272521628Z
time	2025-05-16T13:45:54.019887579Z
time	2025-05-16T13:44:41.20866844Z
time	2025-05-16T13:43:08.432087274Z
time	2025-05-16T13:41:57.071470741Z
time	2025-05-16T13:40:49.766252973Z
time	2025-05-16T13:39:34.515533475Z
time	2025-05-16T13:37:19.293320201Z
time	2025-05-16T13:35:15.176529152Z
time	2025-05-16T13:33:11.298909334Z
time	2025-05-16T13:31:08.859474614Z

Overall, this is much better than it was. Some of the changes could be more than needed, we welcome feedback.