Help wanted: explosion of containers after upgrade to 7.9.1

[klas-s]

Hi! I'll keep this brief. Any help is appreciated.

We're running Concourse in Kubernetes with the official Helm chart. It's a small installation, at the moment just two workers. The workers are run as a StatefulSet with persistence using Kubernetes local storage. The worker use the containerd runtime with the overlay driver. In addition to to this brief description, I'm pasting some of (what I believe) to be relevant environment variables from the web pod and a worker pod.

After upgrading from 7.5 to 7.9.1 we're seeing much more containers on the workers. On 7.5 we peaked at about 350 containers, but on 7.9.1 it keeps going and is currently hitting the configured maximum of 1000 containers after a couple of hours. As far as I can tell, these are 90 % resource check containers that hang around. Prior to upgrading, the number and frequency of resource checks were similar.

So, the obvious question is if this is at all expected. I'm not seeing anything obvious in the logs. I'd be happy to complement with additional information.

Here are a few (possibly) relevant environement variables.

From the worker:

CONCOURSE_REBALANCE_INTERVAL=12h
CONCOURSE_VOLUME_SWEEPER_MAX_IN_FLIGHT=5
CONCOURSE_SWEEP_INTERVAL=30s
CONCOURSE_RUNTIME=containerd
CONCOURSE_BAGGAGECLAIM_DRIVER=overlay
CONCOURSE_CONTAINERD_MAX_CONTAINERS=1000
CONCOURSE_CONNECTION_DRAIN_TIMEOUT=4h
CONCOURSE_CONTAINER_SWEEPER_MAX_IN_FLIGHT=5

And here are some from the web component:

CONCOURSE_RESOURCE_CHECKING_INTERVAL=5m
CONCOURSE_GC_MISSING_GRACE_PERIOD=5m
CONCOURSE_MAX_ACTIVE_VOLUMES_PER_WORKER=25000
CONCOURSE_CONTAINER_PLACEMENT_STRATEGY=limit-active-tasks,volume-locality
CONCOURSE_GC_INTERVAL=30s
CONCOURSE_RESOURCE_WITH_WEBHOOK_CHECKING_INTERVAL=1m
CONCOURSE_GC_ONE_OFF_GRACE_PERIOD=5m
CONCOURSE_GLOBAL_RESOURCE_CHECK_TIMEOUT=1h
CONCOURSE_MAX_ACTIVE_CONTAINERS_PER_WORKER=1000
CONCOURSE_ENABLE_GLOBAL_RESOURCES=true
CONCOURSE_BAGGAGECLAIM_RESPONSE_HEADER_TIMEOUT=1m
CONCOURSE_TSA_HEARTBEAT_INTERVAL=30s
CONCOURSE_MAX_ACTIVE_TASKS_PER_WORKER=3
CONCOURSE_BUILD_TRACKER_INTERVAL=10s
CONCOURSE_DEFAULT_GET_TIMEOUT=5m
CONCOURSE_MAX_CHECKS_PER_SECOND=20

Cheers!

[klas-s]

This seems to have been caused by invalid data in the database, which we managed to create as part of the upgrade process. (There were a few backups and restores of the database as part of our upgrade.) Since we're on a tiny installation, starting from scratch was an option for us, and it helped with the explosion of containers.

I guess the lesson learned is that one must take care to not mess with Concourse's data.

Thank you for a great product, we appreciate your hard work and dedication!

[waltj5]

Hi @klas-s, what invalid data did you find in your database? I'm looking into an issue with our worker nodes restarting due to what looks like high resource usage. I'm wondering if your fix might work for us, or at least push us in the right direction.