Skip to content

Commit 3c89073

Browse files
dvdksndvdksn
authored
Merge pull request #21617 from dvdksn/build-gcconfig
build: update gc documentation
2 parents 54fcccd + 841ca61 commit 3c89073

File tree

1 file changed

+251
-57
lines changed

1 file changed

+251
-57
lines changed

‎content/manuals/build/cache/garbage-collection.md‎

Lines changed: 251 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -8,89 +8,283 @@ aliases:
88

99
While [`docker builder prune`](/reference/cli/docker/builder/prune.md)
1010
or [`docker buildx prune`](/reference/cli/docker/buildx/prune.md)
11-
commands run at once, garbage collection runs periodically and follows an
12-
ordered list of prune policies.
11+
commands run at once, Garbage Collection (GC) runs periodically and follows an
12+
ordered list of prune policies. The BuildKit daemon clears the build cache when
13+
the cache size becomes too big, or when the cache age expires.
1314

14-
Garbage collection runs in the BuildKit daemon. The daemon clears the build
15-
cache when the cache size becomes too big, or when the cache age expires. The
16-
following sections describe how you can configure both the size and age
17-
parameters by defining garbage collection policies.
15+
For most users, the default GC behavior is sufficient and doesn't require any
16+
intervention. Advanced users, particularly those working with large-scale
17+
builds, self-managed builders, or constrained storage environments, might
18+
benefit from customizing these settings to better align with their workflow
19+
needs. The following sections explain how GC works and provide guidance on
20+
tailoring its behavior through custom configuration.
1821

19-
Each of the policy's parameters corresponds with a `docker buildx prune` command line
20-
argument. Details can be found in the
21-
`docker buildx prune` [documentation](/reference/cli/docker/buildx/prune.md).
22+
## Garbage collection policies
23+
24+
GC policies define a set of rules that determine how the build cache is managed
25+
and cleaned up. These policies include criteria for when to remove cache
26+
entries, such as the age of the cache, the amount of space being used, and the
27+
type of cache records to prune.
28+
29+
Each GC policy is evaluated in sequence, starting with the most specific
30+
criteria, and proceeds to broader rules if previous policies do not free up
31+
enough cache. This lets BuildKit prioritize cache entries, preserving the most
32+
valuable cache while ensuring the system maintains performance and
33+
availability.
34+
35+
For example, say you have the following GC policies:
36+
37+
1. Find "stale" cache records that haven't been used in the past 48 hours, and
38+
delete records until there's maximum 5GB of "stale" cache left.
39+
2. If the build cache size exceeds 10GB, delete records until the total cache
40+
size is no more than 10GB.
41+
42+
The first rule is more specific, prioritizing stale cache records and setting a
43+
lower limit for a less valuable type of cache. The second rule imposes a higher
44+
hard limit that applies to any type of cache records. With these policies, if
45+
you have 11GB worth of build cache, where:
46+
47+
- 7GB of which is "stale" cache
48+
- 4GB is other, more valuable cache
49+
50+
A GC sweep would delete 5GB of stale cache as part of the 1st policy, with a
51+
remainder of 6GB, meaning the 2nd policy does not need to clear any more cache.
52+
53+
The default GC policies are (approximately):
54+
55+
1. Remove cache that can be easily regenerated, such as build contexts from
56+
local directories or remote Git repositories, and cache mounts, if hasn't
57+
been used for more than 48 hours.
58+
2. Remove cache that hasn't been used in a build for more than 60 days.
59+
3. Remove unshared cache that exceeds the build cache size limit. Unshared
60+
cache records refers to layer blobs that are not used by other resources
61+
(typically, as image layers).
62+
4. Remove any build cache that exceeds the build cache size limit.
63+
64+
The precise algorithm and the means of configuring the policies differ slightly
65+
depending on what kind of builder you're using. Refer to
66+
[Configuration](#configuration) for more details.
2267

2368
## Configuration
2469

25-
Depending on the [driver](../builders/drivers/_index.md) used by your builder instance,
26-
the garbage collection will use a different configuration file.
70+
> [!NOTE]
71+
> If you're satisfied with the default garbage collection behavior and don't
72+
> need to fine-tune its settings, you can skip this section. Default
73+
> configurations work well for most use cases and require no additional setup.
74+
75+
Depending on the type of [build driver](../builders/drivers/_index.md) you use,
76+
you will use different configuration files to change the builder's GC settings:
77+
78+
- If you use the default builder for Docker Engine (the `docker` driver), use
79+
the [Docker daemon configuration file](#docker-daemon-configuration-file).
80+
- If you use a custom builder, use a [BuildKit configuration file](#buildkit-configuration-file).
81+
82+
### Docker daemon configuration file
83+
84+
If you're using the default [`docker` driver](../builders/drivers/docker.md),
85+
GC is configured in the [`daemon.json` configuration file](/reference/cli/dockerd.md#daemon-configuration-file),
86+
or if you use Docker Desktop, in [**Settings > Docker Engine**](/manuals/desktop/settings-and-maintenance/settings.md).
87+
88+
The following snippet shows the default builder configuration for the `docker`
89+
driver for Docker Desktop users:
90+
91+
```json
92+
{
93+
"builder": {
94+
"gc": {
95+
"defaultKeepStorage": "20GB",
96+
"enabled": true
97+
}
98+
}
99+
}
100+
```
101+
102+
The `defaultKeepStorage` option configures the size limit of the build cache,
103+
which influences the GC policies. The default policies for the `docker` driver
104+
work as follows:
105+
106+
1. Remove ephemeral, unused build cache older than 48 hours if it exceeds 13.8%
107+
of `defaultKeepStorage`, or at minimum 512MB.
108+
2. Remove unused build cache older than 60 days.
109+
3. Remove unshared build cache that exceeds the `defaultKeepStorage` limit.
110+
4. Remove any build cache that exceeds the `defaultKeepStorage` limit.
111+
112+
Given the Docker Desktop default value for `defaultKeepStorage` of 20GB, the
113+
default GC policies resolve to:
114+
115+
```json
116+
{
117+
"builder": {
118+
"gc": {
119+
"enabled": true,
120+
"policy": [
121+
{
122+
"keepStorage": "2.764GB",
123+
"filter": [
124+
"unused-for=48h",
125+
"type==source.local,type==exec.cachemount,type==source.git.checkout"
126+
]
127+
},
128+
{ "keepStorage": "20GB", "filter": ["unused-for=1440h"] },
129+
{ "keepStorage": "20GB" },
130+
{ "keepStorage": "20GB", "all": true }
131+
]
132+
}
133+
}
134+
}
135+
```
136+
137+
The easiest way to tweak the build cache configuration for the `docker` driver
138+
is to adjust the `defaultKeepStorage` option:
139+
140+
- Increase the limit if you feel like you think the GC is too aggressive.
141+
- Decrease the limit if you need to preserve space.
27142

28-
If you're using the [`docker` driver](../builders/drivers/docker.md), garbage collection
29-
can be configured in the [Docker Daemon configuration](/reference/cli/dockerd.md#daemon-configuration-file).
30-
file:
143+
If you need even more control, you can define your own GC policies directly.
144+
The following example defines a more conservative GC configuration with the
145+
following policies:
146+
147+
1. Remove unused cache entries older than 1440 hours, or 60 days, if build cache exceeds 50GB.
148+
2. Remove unshared cache entries if build cache exceeds 50GB.
149+
3. Remove any cache entries if build cache exceeds 100GB.
31150

32151
```json
33152
{
34153
"builder": {
35154
"gc": {
36155
"enabled": true,
37-
"defaultKeepStorage": "10GB",
156+
"defaultKeepStorage": "50GB",
38157
"policy": [
39-
{ "keepStorage": "10GB", "filter": ["unused-for=2200h"] },
40-
{ "keepStorage": "50GB", "filter": ["unused-for=3300h"] },
158+
{ "keepStorage": "0", "filter": ["unused-for=1440h"] },
159+
{ "keepStorage": "0" },
41160
{ "keepStorage": "100GB", "all": true }
42161
]
43162
}
44163
}
45164
}
46165
```
47166

48-
For other drivers, garbage collection can be configured using the
49-
[BuildKit configuration](../buildkit/toml-configuration.md) file:
167+
Policies 1 and 2 here set `keepStorage` to `0`, which means they'll fall back
168+
to the default limit of 50GB as defined by `defaultKeepStorage`.
169+
170+
### BuildKit configuration file
171+
172+
For build drivers other than `docker`, GC is configured using a
173+
[`buildkitd.toml`](../buildkit/toml-configuration.md) configuration file. This
174+
file uses the following high-level configuration options that you can use to
175+
tweak the thresholds for how much disk space BuildKit should use for cache:
176+
177+
| Option | Description | Default value |
178+
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
179+
| `reservedSpace` | The minimum amount of disk space BuildKit is allowed to allocate for cache. Usage below this threshold will not be reclaimed during garbage collection. | 10% of total disk space or 10GB (whichever is lower) |
180+
| `maxUsedSpace` | The maximum amount of disk space that BuildKit is allowed to use. Usage above this threshold will be reclaimed during garbage collection. | 60% of total disk space or 100GB (whichever is lower) |
181+
| `minFreeSpace` | The amount of disk space that must be kept free. | 20GB |
182+
183+
You can set these options either as number of bytes, a unit string (for
184+
example, `512MB`), or as a percentage of the total disk size. Changing these
185+
options influences the default GC policies used by the BuildKit worker. With
186+
the default thresholds, the GC policies resolve as follows:
50187

51188
```toml
189+
# Global defaults
52190
[worker.oci]
53191
gc = true
54-
gckeepstorage = 10000
55-
[[worker.oci.gcpolicy]]
56-
keepBytes = 512000000
57-
keepDuration = 172800
58-
filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"]
59-
[[worker.oci.gcpolicy]]
60-
all = true
61-
keepBytes = 1024000000
192+
reservedSpace = "10GB"
193+
maxUsedSpace = "100GB"
194+
minFreeSpace = "20%"
195+
196+
# Policy 1
197+
[[worker.oci.gcpolicy]]
198+
filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout" ]
199+
keepDuration = "48h"
200+
maxUsedSpace = "512MB"
201+
202+
# Policy 2
203+
[[worker.oci.gcpolicy]]
204+
keepDuration = "1440h" # 60 days
205+
reservedSpace = "10GB"
206+
maxUsedSpace = "100GB"
207+
208+
# Policy 3
209+
[[worker.oci.gcpolicy]]
210+
reservedSpace = "10GB"
211+
maxUsedSpace = "100GB"
212+
213+
# Policy 4
214+
[[worker.oci.gcpolicy]]
215+
all = true
216+
reservedSpace = "10GB"
217+
maxUsedSpace = "100GB"
62218
```
63219

64-
## Default policies
65-
66-
Default garbage collection policies apply to all builders if not set:
67-
68-
```text
69-
GC Policy rule#0:
70-
All: false
71-
Filters: type==source.local,type==exec.cachemount,type==source.git.checkout
72-
Keep Duration: 48h0m0s
73-
Keep Bytes: 512MB
74-
GC Policy rule#1:
75-
All: false
76-
Keep Duration: 1440h0m0s
77-
Keep Bytes: 26GB
78-
GC Policy rule#2:
79-
All: false
80-
Keep Bytes: 26GB
81-
GC Policy rule#3:
82-
All: true
83-
Keep Bytes: 26GB
220+
In practical terms, this means:
221+
222+
- Policy 1: If the build cache exceeds 512MB, BuildKit removes cache records
223+
for local build contexts, remote Git contexts, and cache mounts that haven’t
224+
been used in the last 48 hours.
225+
- Policy 2: If disk usage exceeds 100GB, unshared build cache older than 60
226+
days is removed, ensuring at least 10GB of disk space is reserved for cache.
227+
- Policy 3: If disk usage exceeds 100GB, any unshared cache is removed,
228+
ensuring at least 10GB of disk space is reserved for cache.
229+
- Policy 4: If disk usage exceeds 100GB, all cache—including shared and
230+
internal records—is removed, ensuring at least 10GB of disk space is reserved
231+
for cache.
232+
233+
`reservedSpace` has the highest priority in defining the lower limit for build
234+
cache size. If `maxUsedSpace` or `minFreeSpace` would define a lower value, the
235+
minimum cache size would never be brought below `reservedSpace`.
236+
237+
If both `reservedSpace` and `maxUsedSpace` are set, a GC sweep results in a
238+
cache size between those thresholds. For example, if `reservedSpace` is set to
239+
10GB, and `maxUsedSpace` is set to 20GB, the resulting amount of cache after a
240+
GC run is less than 20GB, but at least 10GB.
241+
242+
You can also define completely custom GC policies. Custom policies also let you
243+
define filters, which lets you pinpoint the types of cache entries that a given
244+
policy is allowed to prune.
245+
246+
#### Custom GC policies in BuildKit
247+
248+
Custom GC policies let you fine-tune how BuildKit manages its cache, and gives
249+
you full control over cache retention based on criteria such as cache type,
250+
duration, or disk space thresholds. If you need full control over the cache
251+
thresholds and how cache records should be prioritized, defining custom GC
252+
policies is the way to go.
253+
254+
To define a custom GC policy, use the `[[worker.oci.gcpolicy]]` configuration
255+
block in `buildkitd.toml`. Each policy define the thresholds that will be used
256+
for that policy. The global values for `reservedSpace`, `maxUsedSpace`, and
257+
`minFreeSpace` do not apply if you use custom policies.
258+
259+
Here’s an example configuration:
260+
261+
```toml
262+
# Custom GC Policy 1: Remove unused local contexts older than 24 hours
263+
[[worker.oci.gcpolicy]]
264+
filters = ["type==source.local"]
265+
keepDuration = "24h"
266+
reservedSpace = "5GB"
267+
maxUsedSpace = "50GB"
268+
269+
# Custom GC Policy 2: Remove remote Git contexts older than 30 days
270+
[[worker.oci.gcpolicy]]
271+
filters = ["type==source.git.checkout"]
272+
keepDuration = "720h"
273+
reservedSpace = "5GB"
274+
maxUsedSpace = "30GB"
275+
276+
# Custom GC Policy 3: Aggressively clean all cache if disk usage exceeds 90GB
277+
[[worker.oci.gcpolicy]]
278+
all = true
279+
reservedSpace = "5GB"
280+
maxUsedSpace = "90GB"
84281
```
85282

86-
- `rule#0`: if build cache uses more than 512MB delete the most easily
87-
reproducible data after it has not been used for 2 days.
88-
- `rule#1`: remove any data not used for 60 days.
89-
- `rule#2`: keep the unshared build cache under cap.
90-
- `rule#3`: if previous policies were insufficient start deleting internal data
91-
to keep build cache under cap.
283+
In addition to the `reservedSpace`, `maxUsedSpace`, and `minFreeSpace` threshold,
284+
when defining a GC policy you have two additional configuration options:
92285

93-
> [!NOTE]
94-
>
95-
> `Keep Bytes` defaults to 10% of the size of the disk. If the disk size cannot
96-
> be determined, it uses 2GB as a fallback.
286+
- `all`: By default, BuildKit will exclude some cache records from being pruned
287+
during GC. Setting this option to `true` will allow any cache records to be
288+
pruned.
289+
- `filters`: Filters let you specify specific types of cache records that a GC
290+
policy is allowed to prune.

0 commit comments

Comments
 (0)