Skip to content

Commit b790bec

Browse files
tianontianon
committed
Maintain inclusion/exclusion criteria (meta, SBOMs, signing) here
- move `subset.txt` canonical source here and invert the matching behavior (especially so that we "fail open" and new images are part of the new system automatically) - add explicit list of Laurent's manually verified SBOM repositories and use that to determine when to `--sbom` on `docker buildx build` - remove a bunch of `meta.jq` variance by assuming we'll target a BuildKit which can handle attestations/annotations/OCI output - implement "signing PoC" conditional here to avoid split logic between this repository and the GHA in docker-library/meta I have manually verified that the output of `bashbrew list --all --repos | jq -L.scripts -rsR 'include "doi"; rtrimstr("\n") | split("\n") | . - repos_anti_subset | join("\n")'` matches our current `subset.txt` exactly. 👍
1 parent d3c3c3c commit b790bec

File tree

4 files changed

+241
-79
lines changed

4 files changed

+241
-79
lines changed

‎.test/meta-commands/out.sh

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
SOURCE_DATE_EPOCH=1700741054 \
77
docker buildx build --progress=plain \
88
--provenance=mode=max \
9-
--sbom=generator="$BASHBREW_BUILDKIT_SBOM_GENERATOR" \
109
--output '"type=oci","dest=temp.tar"' \
1110
--annotation 'org.opencontainers.image.source=https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/cli' \
1211
--annotation 'org.opencontainers.image.revision=6d541d27b5dd12639e5a33a675ebca04d3837d74' \

‎Jenkinsfile.meta

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,17 @@ node {
4040
sh '''
4141
git submodule update --remote --merge .doi
4242
git submodule update --remote --merge .scripts
43+
44+
# TODO once "repos_anti_subset" in "doi.jq" is empty, we can remove this (and all associated usages of "subset.txt" can just be "--all" or go away completely)
45+
# in all the places we need to interact with our "subset" it's a lot easier to have an explicit list of what's included, so we'll continue to generate "subset.txt" until it contains the full set
46+
bashbrew list --all --repos | jq -L.scripts -rsR '
47+
include "doi";
48+
rtrimstr("\n")
49+
| split("\n")
50+
| . - repos_anti_subset
51+
| join("\n")
52+
' > subset.txt
53+
git add subset.txt
4354
'''
4455
}
4556

‎doi.jq

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# our old "subset.txt", but inverted
2+
# this way, we can "fail open" correctly (new images should be part of meta by default)
3+
# see "Jenkinsfile.meta" for how/why this turns into "subset.txt"
4+
def repos_anti_subset:
5+
[
6+
# as we remove items from this list, we need to be careful that none of their *children* are still in the list
7+
# (which is why this is sorted in rough "build order" -- that means we can ~safely "pop" off the bottom)
8+
9+
"almalinux", # direct children: crate
10+
"alpine", # direct children: amazoncorretto amazonlinux api-firewall arangodb archlinux bash bonita caddy chronograf consul docker eclipse-mosquitto eclipse-temurin eggdrop erlang fluentd golang haproxy haxe httpd influxdb irssi jobber julia kapacitor kong liquibase memcached nats nats-streaming nginx node notary php postgres python rabbitmq rakudo-star redis registry ruby rust spiped teamspeak telegraf traefik varnish vault znc
11+
"alt",
12+
"amazonlinux", # direct children: amazoncorretto swift
13+
"api-firewall",
14+
"arangodb",
15+
"archlinux",
16+
"bonita",
17+
"centos", # direct children: eclipse-temurin ibm-semeru-runtimes percona swift
18+
"clearlinux",
19+
"clefos",
20+
"consul",
21+
"crate",
22+
"debian", # direct children: adminer aerospike buildpack-deps chronograf clojure couchdb dart emqx erlang haproxy haskell hitch httpd influxdb irssi julia maven memcached mono mysql neo4j neurodebian nginx node odoo openjdk perl php postgres pypy python r-base redis rethinkdb rocket.chat ruby rust spiped swipl unit varnish
23+
"eclipse-mosquitto",
24+
"eggdrop",
25+
"emqx",
26+
"express-gateway",
27+
"hitch",
28+
"jobber",
29+
"mageia",
30+
"mono",
31+
"nats",
32+
"nats-streaming",
33+
"oraclelinux", # direct children: mysql openjdk percona
34+
"percona",
35+
"photon",
36+
"php", # direct children: backdrop composer drupal friendica joomla matomo mediawiki monica nextcloud phpmyadmin postfixadmin unit wordpress yourls
37+
"php-zendserver",
38+
"phpmyadmin",
39+
"postfixadmin",
40+
"r-base",
41+
"rethinkdb",
42+
"rocket.chat",
43+
"rockylinux",
44+
"sl",
45+
"spiped",
46+
"teamspeak",
47+
"traefik",
48+
"ubuntu", # direct children: buildpack-deps couchbase eclipse-temurin elasticsearch gazebo gradle ibmjava ibm-semeru-runtimes kibana kong logstash mariadb mongo neurodebian odoo php-zendserver rabbitmq ros sapmachine silverpeas swift
49+
"varnish",
50+
"vault",
51+
"yourls",
52+
"znc",
53+
"adminer",
54+
"aerospike",
55+
"amazoncorretto", # direct children: jetty maven tomcat
56+
"backdrop",
57+
"buildpack-deps", # direct children: erlang gcc golang haskell haxe influxdb kapacitor node openjdk perl pypy python rakudo-star ruby rust telegraf
58+
"composer", # direct children: drupal
59+
"couchbase",
60+
"dart",
61+
"eclipse-temurin", # direct children: cassandra clojure flink gradle groovy jetty jruby lightstreamer liquibase maven neo4j orientdb solr sonarqube spark storm tomcat tomee unit zookeeper
62+
"erlang", # direct children: elixir
63+
"flink",
64+
"friendica",
65+
"gazebo",
66+
"gradle",
67+
"groovy",
68+
"haskell",
69+
"haxe",
70+
"ibm-semeru-runtimes", # direct children: maven open-liberty tomee websphere-liberty
71+
"ibmjava", # direct children: maven websphere-liberty
72+
"jetty", # direct children: geonetwork
73+
"joomla",
74+
"jruby",
75+
"kong",
76+
"lightstreamer",
77+
"liquibase",
78+
"matomo",
79+
"mediawiki",
80+
"monica",
81+
"neurodebian",
82+
"nextcloud",
83+
"node", # direct children: express-gateway ghost mongo-express unit
84+
"odoo",
85+
"open-liberty",
86+
"orientdb",
87+
"python", # direct children: hylang plone satosa unit
88+
"rakudo-star",
89+
"ros",
90+
"sapmachine", # direct children: maven
91+
"satosa",
92+
"silverpeas",
93+
"spark",
94+
"storm",
95+
"swift",
96+
"tomcat", # direct children: convertigo geonetwork xwiki
97+
"tomee",
98+
"websphere-liberty",
99+
"zookeeper",
100+
"clojure",
101+
"convertigo",
102+
"geonetwork",
103+
"maven",
104+
"mongo-express",
105+
"plone",
106+
107+
empty
108+
]
109+
;
110+
111+
# a helper for "build_should_sbom"
112+
def _sbom_subset:
113+
[
114+
# only repositories we have explicitly verified
115+
"bash",
116+
"buildpack-deps",
117+
"busybox",
118+
"caddy",
119+
"cassandra",
120+
"chronograf",
121+
"couchdb",
122+
"drupal",
123+
"eclipse-temurin",
124+
"elasticsearch",
125+
"fluentd",
126+
"gcc",
127+
"ghost",
128+
"haproxy",
129+
"httpd",
130+
"hylang",
131+
"influxdb",
132+
"julia",
133+
"kapacitor",
134+
"kibana",
135+
"logstash",
136+
"mariadb",
137+
"memcached",
138+
"mongo",
139+
"mysql",
140+
"neo4j",
141+
"nginx",
142+
"openjdk",
143+
"perl",
144+
"php",
145+
"postgres",
146+
"pypy",
147+
"python",
148+
"rabbitmq",
149+
"redis",
150+
"registry",
151+
"ruby",
152+
"rust",
153+
"solr",
154+
"sonarqube",
155+
"telegraf",
156+
"tomcat",
157+
"wordpress",
158+
"xwiki",
159+
empty
160+
]
161+
;
162+
163+
# input: "build" object (with "buildId" top level key)
164+
# output: boolean
165+
def build_should_sbom:
166+
.source.arches[.build.arch].tags
167+
| map(split(":")[0])
168+
| unique
169+
| _sbom_subset as $subset
170+
| any(.[];
171+
. as $i
172+
| $subset
173+
| index($i)
174+
)
175+
;
176+
177+
# input: "build" object (with "buildId" top level key)
178+
# output: boolean
179+
def build_should_sign:
180+
.build.arch == "amd64" and (
181+
.source.arches[.build.arch].tags
182+
| map(split(":")[0])
183+
| unique
184+
| index("notary")
185+
)
186+
;

‎meta.jq

Lines changed: 44 additions & 78 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
# "build_should_sbom", etc.
2+
include "doi";
3+
14
# input: "build" object (with "buildId" top level key)
25
# output: boolean
36
def needs_build:
@@ -17,27 +20,6 @@ def normalized_builder:
1720
end
1821
else . end
1922
;
20-
def docker_uses_containerd_storage:
21-
# TODO somehow detect docker-with-containerd-storage
22-
false
23-
;
24-
# input: "build" object (with "buildId" top level key)
25-
# output: boolean
26-
def should_use_docker_buildx_driver:
27-
normalized_builder == "buildkit"
28-
and (
29-
docker_uses_containerd_storage
30-
or (
31-
.build.arch as $arch
32-
# bashbrew remote arches --json tianon/buildkit:0.12 | jq '.arches | keys_unsorted' -c
33-
| ["amd64","arm32v5","arm32v6","arm32v7","arm64v8","i386","mips64le","ppc64le","riscv64","s390x"]
34-
# TODO this needs to be based on the *host* architecture, not the *target* architecture (amd64 vs i386)
35-
| index($arch)
36-
| not
37-
# TODO "failed to read dockerfile: failed to load cache key: subdir not supported yet" asdflkjalksdjfklasdjfklajsdklfjasdklgfnlkasdfgbhnkljasdhgouiahsdoifjnask,.dfgnklasdbngoikasdhfoiasjdklfjasdlkfjalksdjfkladshjflikashdbgiohasdfgiohnaskldfjhnlkasdhfnklasdhglkahsdlfkjasdlkfjadsklfjsdl (hence "tianon/buildkit" instead of "moby/buildkit"; need *all* the arches we care about/support for consistent support)
38-
)
39-
)
40-
;
4123
# input: "docker.io/library/foo:bar"
4224
# output: "foo:bar"
4325
def normalize_ref_to_docker:
@@ -48,7 +30,7 @@ def normalize_ref_to_docker:
4830
# output: string "pull command" ("docker pull ..."), may be multiple lines, expects to run in Bash with "set -Eeuo pipefail", might be empty
4931
def pull_command:
5032
normalized_builder as $builder
51-
| if $builder == "classic" or should_use_docker_buildx_driver then
33+
| if $builder == "classic" then
5234
[
5335
(
5436
.build.resolvedParents
@@ -151,63 +133,50 @@ def build_command:
151133
normalized_builder as $builder
152134
| if $builder == "buildkit" then
153135
git_build_url as $buildUrl
154-
| (
155-
(should_use_docker_buildx_driver | not)
156-
or docker_uses_containerd_storage
157-
) as $supportsAnnotationsAndAttestsations
158136
| [
159137
(
160138
[
161139
@sh "SOURCE_DATE_EPOCH=\(.source.entry.SOURCE_DATE_EPOCH)",
162140
# TODO EXPERIMENTAL_BUILDKIT_SOURCE_POLICY=<(jq ...)
163141
"docker buildx build --progress=plain",
164-
if $supportsAnnotationsAndAttestsations then
165-
"--provenance=mode=max",
142+
"--provenance=mode=max",
143+
if build_should_sbom then
166144
# see "bashbrew remote arches docker/scout-sbom-indexer:1" (we need the SBOM scanner to be runnable on the host architecture)
167145
# bashbrew remote arches --json docker/scout-sbom-indexer:1 | jq '.arches | keys_unsorted' -c
168146
if .build.arch as $arch | ["amd64","arm32v5","arm32v7","arm64v8","i386","ppc64le","riscv64","s390x"] | index($arch) then
169147
# TODO this needs to be based on the *host* architecture, not the *target* architecture (amd64 vs i386)
170148
"--sbom=generator=\"$BASHBREW_BUILDKIT_SBOM_GENERATOR\""
171-
# TODO this should also be totally optional -- for example, Tianon doesn't want SBOMs on his personal images
172-
else empty end,
173-
empty
149+
else empty end
174150
else empty end,
175151
"--output " + (
176152
[
177-
if should_use_docker_buildx_driver then
178-
"type=docker"
179-
else
180-
"type=oci",
181-
"dest=temp.tar", # TODO choose/find a good "safe" place to put this (temporarily)
182-
empty
183-
end,
153+
"type=oci",
154+
"dest=temp.tar", # TODO choose/find a good "safe" place to put this (temporarily)
184155
empty
185156
]
186157
| @csv
187158
| @sh
188159
),
189160
(
190-
if $supportsAnnotationsAndAttestsations then
191-
build_annotations($buildUrl)
192-
| to_entries
193-
# separate loops so that "image manifest" annotations are grouped separate from the index/descriptor annotations (easier to read)
194-
| (
195-
.[]
196-
| @sh "--annotation \(.key + "=" + .value)"
197-
),
198-
(
199-
.[]
200-
| @sh "--annotation \(
201-
"manifest-descriptor:" + .key + "="
202-
+ if .key == "org.opencontainers.image.created" then
203-
# the "current" time breaks reproducibility (for the purposes of build verification), so we put "now" in the image index but "SOURCE_DATE_EPOCH" in the image manifest (which is the thing we'd ideally like to have reproducible, eventually)
204-
(env.SOURCE_DATE_EPOCH // now) | tonumber | strftime("%FT%TZ")
205-
# (this assumes the actual build is going to happen shortly after generating the command)
206-
else .value end
207-
)",
208-
empty
209-
)
210-
else empty end
161+
build_annotations($buildUrl)
162+
| to_entries
163+
# separate loops so that "image manifest" annotations are grouped separate from the index/descriptor annotations (easier to read)
164+
| (
165+
.[]
166+
| @sh "--annotation \(.key + "=" + .value)"
167+
),
168+
(
169+
.[]
170+
| @sh "--annotation \(
171+
"manifest-descriptor:" + .key + "="
172+
+ if .key == "org.opencontainers.image.created" then
173+
# the "current" time breaks reproducibility (for the purposes of build verification), so we put "now" in the image index but "SOURCE_DATE_EPOCH" in the image manifest (which is the thing we'd ideally like to have reproducible, eventually)
174+
(env.SOURCE_DATE_EPOCH // now) | tonumber | strftime("%FT%TZ")
175+
# (this assumes the actual build is going to happen shortly after generating the command)
176+
else .value end
177+
)",
178+
empty
179+
)
211180
),
212181
(
213182
(
@@ -233,24 +202,21 @@ def build_command:
233202
empty
234203
] | join(" \\\n\t")
235204
),
236-
if should_use_docker_buildx_driver then empty else
237-
# munge the tarball into a suitable "oci layout" directory (ready for "crane push")
238-
"mkdir temp",
239-
"tar -xvf temp.tar -C temp",
240-
"rm temp.tar",
241-
# munge the index to what crane wants ("Error: layout contains 5 entries, consider --index")
242-
@sh "jq \("
243-
.manifests |= (
244-
del(.[].annotations)
245-
| unique
246-
| if length != 1 then
247-
error(\"unexpected number of manifests: \" + length)
248-
else . end
249-
)
250-
" | unindent_and_decomment_jq(4)) temp/index.json > temp/index.json.new",
251-
"mv temp/index.json.new temp/index.json",
252-
empty
253-
end,
205+
# munge the tarball into a suitable "oci layout" directory (ready for "crane push")
206+
"mkdir temp",
207+
"tar -xvf temp.tar -C temp",
208+
"rm temp.tar",
209+
# munge the index to what crane wants ("Error: layout contains 5 entries, consider --index")
210+
@sh "jq \("
211+
.manifests |= (
212+
del(.[].annotations)
213+
| unique
214+
| if length != 1 then
215+
error(\"unexpected number of manifests: \" + length)
216+
else . end
217+
)
218+
" | unindent_and_decomment_jq(3)) temp/index.json > temp/index.json.new",
219+
"mv temp/index.json.new temp/index.json",
254220
# possible improvements in buildkit/buildx that could help us:
255221
# - allowing OCI output directly to a directory instead of a tar (thus getting symmetry with the oci-layout:// inputs it can take)
256222
# - allowing tag as one thing and push as something else, potentially mutually exclusive
@@ -365,7 +331,7 @@ def build_command:
365331
# output: string "push command" ("docker push ..."), may be multiple lines, expects to run in Bash with "set -Eeuo pipefail"
366332
def push_command:
367333
normalized_builder as $builder
368-
| if $builder == "classic" or should_use_docker_buildx_driver then
334+
| if $builder == "classic" then
369335
@sh "docker push \(.build.img)"
370336
elif $builder == "buildkit" then
371337
[

0 commit comments

Comments
 (0)