CONCOURSE_GC_FAILED_GRACE_PERIOD can act as a denial-of-service on the cluster

[saj]

I think this is a bug, but I don't have a step-by-step repro for you at this time.

Problem

One of my colleagues runs a very wide pipeline in our Concourse cluster: there are at least 575 jobs in one pipeline. (That is not a typo. Almost six hundred.) Today, as luck would have it, all of these jobs failed on a common bug. Soon afterwards, the Concourse cluster refused to execute any builds.

find or create container on worker ...: new container: checking container capacity: max containers reached

No pipeline could build -- or even run resource checks -- in this state.

I suspect GC was deferred because these builds went red.

Separately, there is no way for the operator to manually delete a container -- see #346 -- so it is not so easy to recover from this failure mode. I ultimately bounced our workers as a work-around. (Our workers 'forget' their containers when we bounce them. I don't know whether that is a separate bug; I guess it turned out to be a 'feature' today!)

Suggestion

fly hijack is a cool feature. I understand why you hang on to these old containers. Nevertheless, I don't think it is reasonable to prioritise a debugging feature over foundational task scheduling. A cluster that cannot run new builds is a useless cluster.

Would it be possible to add container pre-emption to the ATC scheduler?

If, when starting a new container, all workers are found to be at or near their container limit, the ATC should immediately GC some old containers that are technically eligible for GC. (Any container that is not being used to actively execute stuff.) Prioritise builds. Sacrifice fly hijack if we are at the limit.

[Kump3r]

I think you could always adjust the CONCOURSE_GC_FAILED_GRACE_PERIOD as the default is 5 days. The idea was to debug nasty side_effects even after long weekends if memory serves me right. If your usual jobs are not quite resource heavy you could always make the max_containers count higher, or if the opposite is true, then scale a little horizontally to have more workers, instead of vertically upscaled bigger machines. This really depends on your setup and use case.

[taylorsilva]

I don't have much to add here. Kump3r covered most of my thoughts about this.

Like kump3r said, right now in this situation operators could:

• Set GC grace period to a value closer to zero, say 5-10mins. Users lose the ability to hijack their containers though and we know users really like this feature of Concourse.
• Add more workers. The con here is cost; do you have the budget to scale out? Can you scale out fast enough as well.

I can understand why this is a pain for an operator. Your "release valve" idea/metaphor is intriguing. Let's talk through it a bit.

In order to make the GC do a "pressure release" it first needs to know if there is pressure. We need a feedback loop between the worker placement component and the GC component. Tricky thing I see here is how do we know what "pressure" looks like? Is it when 50 containers are waiting to be assigned a worker? 100? 200? That'll depend on your cluster size I think, and possibly other things. Do we expose this as yet another thing for operators to tune?? There may be an elegant way to detect this with the current code, I'm just not sure right now without digging into it more.

Once this feedback loop is in place, the GC can then react when the worker placement is unable to schedule containers. GC could then:

• take it's current GC_GRACE_PERIOD and subtract one from
• Perform GC given this new grace period
• Wait to see if pressure is relieved
• Repeat from step 1 if pressure is still present

The GC part seems quite straight forward. I think the tricky part of this problem is detecting when there is pressure on the cluster.

---

Mostly unrelated, just want to add some clarity.

Really, the only practical use of the MAX_CONTAINERS knob is to mask other software limits. The default limit in Concourse was supposedly chosen to fail before the system exhausted its supply of network addresses.

This is stale news. The network limit was updated sometime ago to default to 10.80.0.0/16 which gives 65k possible addresses. This is for both runtimes on the worker. The CONCOURSE_*_MAX_CONTAINERS default of 250 is just a starting point for operators. We could consider defaulting this to a higher number maybe. There are no software limits in Concourse that make going above 250 containers an issue AFAIK. That article discusses a lot of good Garden/Cloud Foundry history but doesn't mention anything about containerd.

[saj]

I was also a bit surprised that the failure seemed to strike after the placement decision.

Here is the error I would have expected to see:

Hello, I am the Concourse ATC. I cannot find a worker for this container because all your workers are full'o'containers.

Whereas the error we received included the name of a specific worker node -- as if the ATC had already decided on a placement -- and as if the error were coming from a particular worker.

I suppose this goes to what you were saying:

We need a feedback loop between the worker placement component and the GC component.

---

Perhaps my 'pressure relief valve' analogy was a bit too literal. :)

Tricky thing I see here is how do we know what "pressure" looks like? Is it when 50 containers are waiting to be assigned a worker? 100? 200?

I would like to avoid questions involving magnitudes or volume knobs. I think we only need to perform immediate GC if the cluster is unable to make any forward progress.

(I have not read the code, so the rest of my message will include a lot of hand-waving and guesswork. My apologies.)

Suppose the container lifecycle in Concourse looks something like this:

CREATED → ACTIVE → DEAD

CREATED is what we see soon after the ATC makes a placement decision. A worker has committed some of its resources to the container, but maybe the container is not confirmed to be running yet. In this state, we can say the container, and its host worker, are making forward progress because the container will eventually start doing useful work.

ACTIVE is what we see when a container is doing useful work. This could be a resource container that is actively executing its check program, or a task container that is actively executing a user build. The container is obviously making forward progress in this state.

DEAD is what we get when a task container terminates, or when a resource check finishes a scheduled poll. Containers in this state are not making any forward progress. They are mostly dead weight. There are some special cases here:

• A resource container in DEAD state may transition DEAD → ACTIVE as an optimisation. State on the container filesystem may be reused, like a cache. I think it is most efficient for such a container to transition DEAD → ACTIVE → DEAD → ACTIVE → ... repeatedly, with every resource poll.

• A task container in DEAD state may be used as the subject for fly intercept (fly hijack).

What sets the DEAD state apart from the others is that nothing important in Concourse should break outright if containers in DEAD state disappear. Indeed, I expect this state -- or something like it -- is the focus of the current GC implementation.

Here is what I suggest:

• If a worker is full of containers, and if those containers are in CREATED or ACTIVE state, there is no problem. 👌 It does not not matter how many containers are waiting to be assigned. The containers already on the worker are all doing useful stuff.

• If a worker is full of containers, and if those containers are all in DEAD state, then we have a serious problem. ❗ The containers on the worker are not doing anything useful; meanwhile, they are blocking the placement of containers that will soon transition to ACTIVE.

At placement time, are we able to make a quick determination of whether all workers are full of DEAD containers?

[taylorsilva]

I think the big development hurdle to overcome here will be feedback loops. Your comment made me look at the error you reported again:

max containers reached

Which is the worker telling the web node "I won't accept anymore containers until the number I currently have goes down". The web node doesn't do anything with this info though. It just takes it in as an "internal error" and bubbles it up to the user. We don't try to catch and react to it at all.

I've had this idea in my head about generally trying to improve the feedback loop between the worker and web nodes. I think it would improve not just this edge-case you're bringing forward, but others as well. For example, it would be nice to know if a worker is nearly maxed out of CPU or memory and not schedule stuff on it until it cools off. That's not possible right now because there's no "health feedback loop" between the workers and web nodes.

If I was to action this out, I think my general plan would be:

• create a feedback loop between the worker and web nodes. Have the workers report basic health info (cpu, memory, max containers allowed, any other settings worth tracking)
  • We already know which and how many containers are on a worker, but we don't actually know what the container limit is for each worker. I know in most clusters they're probably all the same, but that's definitely not true across every cluster.
  • Info dump: The workers table has these columns: addr, expires, active_containers, resource_types, platform, tags, baggageclaim_url, name, http_proxy_url, https_proxy_url, no_proxy, start_time, team_id, state, version, certs_path, ephemeral, active_volumes, active_tasks
    • Columns like active_volumes, active_tasks, active_containers are tracked purely on the web node side, not the worker side. We do poll for worker for the containers it has and that does get fed into the GC loop.
• Once that feedback loop is in place, update other components to react to new info we get from the worker
  • Worker placement can now wait until workers are below their max container limit
  • GC can detect if all workers are at max container limit and start GC'ing older build containers
  • Could add a new worker placement strategy that prioritizes worker health (cpu/memory usage) or even avoids scheduling onto workers if they're CPU/memory is above certain thresholds.

This is of course a ton of work. I think something like this would alleviate the current set of scheduling woes we see users getting. Someone on discord was just complaining about the max containers issue as well!

Shorter-term stuff someone could do is give operators more tools to do their own GC'ing in an emergency. Having some sort of fly garbage-collect command to GC all containers related to a specific build/job/pipeline/team could be very handy to unstuck people.

Those are my thoughts. I'd love to action on some of this but I know I won't have time in the short-term to do any of this.

[saj]

I think it would improve not just this edge-case you're bringing forward, but others as well.

Indeed.

I won't have time in the short-term to do any of this

No problem.

give operators more tools to do their own GC'ing in an emergency

Yes, this would help as a stopgap. Today, I can look at the output from fly containers, I can identify what needs to go in order to restore service, but I have no easy way to force a reclaim on these containers.

IIRC, fly clear-resource-cache was added as a workaround for some bug a long time ago? So I guess there is a bit of a precedent there.

Thanks for your consideration!

[taylorsilva]

fly clear-resource-cache would only delete volumes, not containers. I don't think it was ever for a bug. It just helps if the "version" a resource get's is "stale" somehow and you want to force the resource to fetch the version again.