When are volumes garbage collected?

[dlbock]

The docs on the Garbage Collector does not explain the criteria in which a container or volume needs to meet in order to be garbage collected.

I'm looking at the volumes table in our Concourse instance (v7.2.0), which has about 100 pipelines, 18 workers, 800GB for each worker PVC, and there are 27088 volumes

concourse=> select count(*) from volumes;
count | 27088

And 24280 of them do not have a container_id associated

concourse=> select count(*) from volumes where container_id IS NULL;
count | 24280

If you see the following screenshot of worker volumes and containers per worker of the last 2 weeks, you'll see that the the number of volumes have just been generally growing over time, even though the number of containers have fluctuated and have not been trending up.

Can someone help me understand how volumes (and containers) actually get garbage collected?

[taylorsilva]

Could you share all the columns from thevolumes table for these volumes that don't have a container_id. My guess is that they belong to a cache. If they don't then the other columns may give us a clue as to what we're missing in the gc logic.

[taylorsilva]

Totally forgot, but there's a fly clear-resource-cache command. That may solve the issue for you instead of having to delete worker PV's

[dlbock]

@taylorsilva Actually that command is not available, we're currently on v7.2.0, expecting to update to 7.4.0 as soon as I can find the time.

$ fly -t <team> clear-resource-cache -h
error: Unknown command `clear-resource-cache', did you mean `check-resource-type'?

$ fly -v
7.2.0 

And the documentation for that command indicates that it only works on a pipeline level, not at the worker level, or better yet, at a global level.

[taylorsilva]

Ah true, that was added in 7.4! 😅 I gotta log off, Friday afternoon is hitting me hard now haha

[wanderanimrod]

@taylorsilva , is there a reason fly clear-resource-cache can't work a the worker level apart from "it hasn't been implemented yet"? 😄

[taylorsilva]

no other reason as far as I know 😅

[funkypenguin]

How are the volumes intended to be GC'd? We have a situation where (over a period of a week or two), our workers "fill up" their 150GB volumes, and subsequent jobs fail.

I'm experimenting with explicitly setting the various grace / recycle periods, but is there a better way to influence the rate of volume cleanup?

Thanks!
D

[taylorsilva]

You can change how frequently GC happens, but that's probably not going to solve the problem.

One thing I did learn recently is that task cache's are only GC'd when the related pipeline/job are paused/deleted. This means that in an active pipeline, task caches can grow indefinitely.

concourse/atc/db/task_cache_lifecycle.go

Lines 22 to 32 in 852419f

	func (f *taskCacheLifecycle) CleanUpInvalidTaskCaches() ([]int, error) {
	rows, err := psql.Delete("task_caches tc").
	Where(sq.Expr(`tc.id IN (
	SELECT tc.id
	FROM task_caches tc
	JOIN jobs j ON j.id = tc.job_id
	JOIN pipelines p ON p.id = j.pipeline_id
	WHERE p.archived OR
	(p.paused AND j.next_build_id IS NULL) OR
	(j.paused AND j.next_build_id IS NULL)
	)`)).

The GC runner: https://github.com/concourse/concourse/blob/master/atc/gc/task_cache_collector.go

So if you're using task caches you might be running into this. Short-term solution is to clear out the task cache using fly clear-task-cache.