Skip to content

Commit 2911e0b

Browse files
dvdksndvdksn
committed
build: update gc documentation
Signed-off-by: David Karlsson <[email protected]>
1 parent 3f815b9 commit 2911e0b

File tree

1 file changed

+234
-57
lines changed

1 file changed

+234
-57
lines changed

‎content/manuals/build/cache/garbage-collection.md‎

Lines changed: 234 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -8,89 +8,266 @@ aliases:
88

99
While [`docker builder prune`](/reference/cli/docker/builder/prune.md)
1010
or [`docker buildx prune`](/reference/cli/docker/buildx/prune.md)
11-
commands run at once, garbage collection runs periodically and follows an
12-
ordered list of prune policies.
11+
commands run at once, Garbage Collection (GC) runs periodically and follows an
12+
ordered list of prune policies. The BuildKit daemon clears the build cache when
13+
the cache size becomes too big, or when the cache age expires.
1314

14-
Garbage collection runs in the BuildKit daemon. The daemon clears the build
15-
cache when the cache size becomes too big, or when the cache age expires. The
16-
following sections describe how you can configure both the size and age
17-
parameters by defining garbage collection policies.
15+
For most users, the default GC behavior is sufficient and doesn't require any
16+
intervention. Advanced users, particularly those working with large-scale
17+
builds, self-managed builders, or constrained storage environments, might
18+
benefit from customizing these settings to better align with their workflow
19+
needs. The following sections explain how GC works and provide guidance on
20+
tailoring its behavior through custom configuration.
1821

19-
Each of the policy's parameters corresponds with a `docker buildx prune` command line
20-
argument. Details can be found in the
21-
`docker buildx prune` [documentation](/reference/cli/docker/buildx/prune.md).
22+
## Garbage collection policies
23+
24+
GC policies define a set of rules that determine how the build cache is managed
25+
and cleaned up. These policies include criteria for when to remove cache
26+
entries, such as the age of the cache, the amount of space being used, and
27+
other constraints like ensuring a minimum amount of free disk space or
28+
reserving a specific percentage of the disk for build cache.
29+
30+
Each GC policy is evaluated in sequence, starting with the most specific
31+
criteria, and proceeds to broader rules if previous policies do not resolve the
32+
issue. This lets BuildKit prioritize cache entries, preserving the most
33+
valuable cache while ensuring the system maintains performance and
34+
availability.
35+
36+
The default GC policies are (approximately):
37+
38+
1. Remove cache that can be easily regenerated, such as build contexts from
39+
local directories or remote Git repositories, and cache mounts, if hasn't
40+
been used for more than 48 hours.
41+
2. Remove cache that hasn't been used in a build for more than 60 days.
42+
3. Remove unshared cache that exceeds the build cache size limit. Unshared
43+
cache records refers to layer blobs that are not used by other resources
44+
(typically, as image layers).
45+
4. Remove any build cache that exceeds the build cache size limit.
46+
47+
The precise algorithm and the means of configuring the policies differ slightly
48+
depending on what kind of builder you're using. Refer to
49+
[Configuration](#configuration) for more details.
2250

2351
## Configuration
2452

25-
Depending on the [driver](../builders/drivers/_index.md) used by your builder instance,
26-
the garbage collection will use a different configuration file.
53+
> [!NOTE]
54+
> If you're satisfied with the default garbage collection behavior and don't
55+
> need to fine-tune its settings, you can skip this section. Default
56+
> configurations work well for most use cases and require no additional setup.
57+
58+
Depending on the type of [build driver](../builders/drivers/_index.md) you use,
59+
you will use different configuration files to change the builder's GC settings:
60+
61+
- If you use the default builder for Docker Engine (the `docker` driver), use
62+
the [Docker daemon configuration file](#docker-daemon-configuration-file).
63+
- If you use a custom builder, use a [BuildKit configuration file](#buildkit-configuration-file).
64+
65+
### Docker daemon configuration file
66+
67+
If you're using the default [`docker` driver](../builders/drivers/docker.md),
68+
GC is configured in the [`daemon.json` configuration file](/reference/cli/dockerd.md#daemon-configuration-file),
69+
or if you use Docker Desktop, in [**Settings > Docker Engine**](/manuals/desktop/settings-and-maintenance/settings.md).
70+
71+
The following snippet shows the default builder configuration for the `docker`
72+
driver for Docker Desktop users:
73+
74+
```json
75+
{
76+
"builder": {
77+
"gc": {
78+
"defaultKeepStorage": "20GB",
79+
"enabled": true
80+
}
81+
}
82+
}
83+
```
84+
85+
The `defaultKeepStorage` option configures the size limit of the build cache,
86+
which influences the GC policies. The default policies for the `docker` driver
87+
work as follows:
2788

28-
If you're using the [`docker` driver](../builders/drivers/docker.md), garbage collection
29-
can be configured in the [Docker Daemon configuration](/reference/cli/dockerd.md#daemon-configuration-file).
30-
file:
89+
1. Remove ephemeral, unused build cache older than 48 hours if it exceeds 10%
90+
of `defaultKeepStorage`, or at minimum 512MB.
91+
2. Remove unused build cache older than 60 days.
92+
3. Remove unshared build cache that exceeds the `defaultKeepStorage` limit.
93+
4. Remove any build cache that exceeds the `defaultKeepStorage` limit.
94+
95+
Given the Docker Desktop default value for `defaultKeepStorage` of 20GB, the
96+
default GC policies resolve to:
3197

3298
```json
3399
{
34100
"builder": {
35101
"gc": {
36102
"enabled": true,
37-
"defaultKeepStorage": "10GB",
38103
"policy": [
39-
{ "keepStorage": "10GB", "filter": ["unused-for=2200h"] },
40-
{ "keepStorage": "50GB", "filter": ["unused-for=3300h"] },
104+
{
105+
"keepStorage": "2.764GB",
106+
"filter": [
107+
"unused-for=48h",
108+
"type==source.local,type==exec.cachemount,type==source.git.checkout"
109+
]
110+
},
111+
{ "keepStorage": "20GB", "filter": ["unused-for=1440h"] },
112+
{ "keepStorage": "20GB" },
113+
{ "keepStorage": "20GB", "all": true }
114+
]
115+
}
116+
}
117+
}
118+
```
119+
120+
The easiest way to tweak the build cache configuration for the `docker` driver
121+
is to adjust the `defaultKeepStorage` option:
122+
123+
- Increase the limit if you feel like you think the GC is too aggressive.
124+
- Decrease the limit if you need to preserve space.
125+
126+
If you need even more control, you can define your own GC policies directly.
127+
The following example defines a more conservative GC configuration with the
128+
following policies:
129+
130+
1. Remove unused cache entries older than 1440 hours, or 60 days, if build cache exceeds 50GB.
131+
2. Remove unshared cache entries if build cache exceeds 50GB.
132+
3. Remove any cache entries if build cache exceeds 100GB.
133+
134+
```json
135+
{
136+
"builder": {
137+
"gc": {
138+
"enabled": true,
139+
"defaultKeepStorage": "50GB",
140+
"policy": [
141+
{ "keepStorage": "0", "filter": ["unused-for=1440h"] },
142+
{ "keepStorage": "0" },
41143
{ "keepStorage": "100GB", "all": true }
42144
]
43145
}
44146
}
45147
}
46148
```
47149

48-
For other drivers, garbage collection can be configured using the
49-
[BuildKit configuration](../buildkit/toml-configuration.md) file:
150+
Policies 1 and 2 here set `keepStorage` to `0`, which means they'll fall back
151+
to the default limit of 50GB as defined by `defaultKeepStorage`.
152+
153+
### BuildKit configuration file
154+
155+
For build drivers other than `docker`, GC is configured using a
156+
[`buildkitd.toml`](../buildkit/toml-configuration.md) configuration file. This
157+
file uses the following high-level configuration options that you can use to
158+
tweak the thresholds for how much disk space BuildKit should use for cache:
159+
160+
| Option | Description | Default value |
161+
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
162+
| `reservedSpace` | The minimum amount of disk space BuildKit is allowed to allocate for cache. Usage below this threshold will not be reclaimed during garbage collection. | 10% of total disk space or 10GB (whichever is lower) |
163+
| `maxUsedSpace` | The maximum amount of disk space that BuildKit is allowed to use. Usage above this threshold will be reclaimed during garbage collection. | 60% of total disk space or 100GB (whichever is lower) |
164+
| `minFreeSpace` | The amount of disk space that must be kept free. | 20GB |
165+
166+
You can set these options either as number of bytes, a unit string (for
167+
example, `512MB`), or as a percentage of the total disk size. Changing these
168+
options influences the default GC policies used by the BuildKit worker. With
169+
the default thresholds, the GC policies resolve as follows:
50170

51171
```toml
172+
# Global defaults
52173
[worker.oci]
53174
gc = true
54-
gckeepstorage = 10000
55-
[[worker.oci.gcpolicy]]
56-
keepBytes = 512000000
57-
keepDuration = 172800
58-
filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"]
59-
[[worker.oci.gcpolicy]]
60-
all = true
61-
keepBytes = 1024000000
175+
reservedSpace = "10GB" # Reserve 10GB or 10% of disk space for cache, whichever is lower
176+
maxUsedSpace = "100GB" # Capped at 100GB or 80% of disk space, whichever is lower
177+
minFreeSpace = "20%" # Always keep 20% of disk space free (no effect since maxUsedSpace doesn't exceed 80%)
178+
179+
# Policy 1
180+
[[worker.oci.gcpolicy]]
181+
filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout" ]
182+
keepDuration = "48h"
183+
maxUsedSpace = "512MB"
184+
185+
# Policy 2
186+
[[worker.oci.gcpolicy]]
187+
keepDuration = "1440h" # 60 days
188+
reservedSpace = "10GB" # or 10%
189+
maxUsedSpace = "100GB" # or 80%
190+
191+
# Policy 3
192+
[[worker.oci.gcpolicy]]
193+
reservedSpace = "10GB" # or 10%
194+
maxUsedSpace = "100GB" # or 80%
195+
196+
# Policy 4
197+
[[worker.oci.gcpolicy]]
198+
all = true
199+
reservedSpace = "10GB" # or 10%
200+
maxUsedSpace = "100GB" # or 80%
62201
```
63202

64-
## Default policies
65-
66-
Default garbage collection policies apply to all builders if not set:
67-
68-
```text
69-
GC Policy rule#0:
70-
All: false
71-
Filters: type==source.local,type==exec.cachemount,type==source.git.checkout
72-
Keep Duration: 48h0m0s
73-
Keep Bytes: 512MB
74-
GC Policy rule#1:
75-
All: false
76-
Keep Duration: 1440h0m0s
77-
Keep Bytes: 26GB
78-
GC Policy rule#2:
79-
All: false
80-
Keep Bytes: 26GB
81-
GC Policy rule#3:
82-
All: true
83-
Keep Bytes: 26GB
203+
In practical terms, this means:
204+
205+
- Policy 1: If the build cache exceeds 512MB, BuildKit removes cache records
206+
for local build contexts, remote Git contexts, and cache mounts that haven’t
207+
been used in the last 48 hours.
208+
- Policy 2: If disk usage exceeds 100GB, unshared build cache older than 60
209+
days is removed, ensuring at least 10GB of disk space is reserved for cache.
210+
- Policy 3: If disk usage exceeds 100GB, any unshared cache is removed,
211+
ensuring at least 10GB of disk space is reserved for cache.
212+
- Policy 4: If disk usage exceeds 100GB, all cache—including shared and
213+
internal records—is removed, ensuring at least 10GB of disk space is reserved
214+
for cache.
215+
216+
`reservedSpace` has the highest priority in defining the lower limit for build
217+
cache size. If `maxUsedSpace` or `minFreeSpace` would define a lower value, the
218+
minimum cache size would never be brought below `reservedSpace`.
219+
220+
If both `reservedSpace` and `maxUsedSpace` are set, a GC sweep results in a
221+
cache size between those thresholds. For example, if `reservedSpace` is set to
222+
10GB, and `maxUsedSpace` is set to 20GB, the resulting amount of cache after a
223+
GC run is less than 20GB, but at least 10GB.
224+
225+
You can also define completely custom GC policies. Custom policies also let you
226+
define filters, which lets you pinpoint the types of cache entries that a given
227+
policy is allowed to prune.
228+
229+
#### Custom GC policies in BuildKit
230+
231+
Custom GC policies let you fine-tune how BuildKit manages its cache, and gives
232+
you full control over cache retention based on criteria such as cache type,
233+
duration, or disk space thresholds. If you need full control over the cache
234+
thresholds and how cache records should be prioritized, defining custom GC
235+
policies is the way to go.
236+
237+
To define a custom GC policy, use the `[[worker.oci.gcpolicy]]` configuration
238+
block in `buildkitd.toml`. Each policy define the thresholds that will be used
239+
for that policy. The global values for `reservedSpace`, `maxUsedSpace`, and
240+
`minFreeSpace` do not apply if you use custom policies.
241+
242+
Here’s an example configuration:
243+
244+
```toml
245+
# Custom GC Policy 1: Remove unused local contexts older than 24 hours
246+
[[worker.oci.gcpolicy]]
247+
filters = ["type==source.local"]
248+
keepDuration = "24h"
249+
reservedSpace = "5GB"
250+
maxUsedSpace = "50GB"
251+
252+
# Custom GC Policy 2: Remove remote Git contexts older than 30 days
253+
[[worker.oci.gcpolicy]]
254+
filters = ["type==source.git.checkout"]
255+
keepDuration = "720h"
256+
reservedSpace = "5GB"
257+
maxUsedSpace = "30GB"
258+
259+
# Custom GC Policy 3: Aggressively clean all cache if disk usage exceeds 90GB
260+
[[worker.oci.gcpolicy]]
261+
all = true
262+
reservedSpace = "5GB"
263+
maxUsedSpace = "90GB"
84264
```
85265

86-
- `rule#0`: if build cache uses more than 512MB delete the most easily
87-
reproducible data after it has not been used for 2 days.
88-
- `rule#1`: remove any data not used for 60 days.
89-
- `rule#2`: keep the unshared build cache under cap.
90-
- `rule#3`: if previous policies were insufficient start deleting internal data
91-
to keep build cache under cap.
266+
In addition to the `reservedSpace`, `maxUsedSpace`, and `minFreeSpace` threshold,
267+
when defining a GC policy you have two additional configuration options:
92268

93-
> [!NOTE]
94-
>
95-
> `Keep Bytes` defaults to 10% of the size of the disk. If the disk size cannot
96-
> be determined, it uses 2GB as a fallback.
269+
- `all`: By default, BuildKit will exclude some cache records from being pruned
270+
during GC. Setting this option to `true` will allow any cache records to be
271+
pruned.
272+
- `filters`: Filters let you specify specific types of cache records that a GC
273+
policy is allowed to prune.

0 commit comments

Comments
 (0)