Skip to content

Commit 64b5472

Browse files
committed
Refactor data model to support more atomic index generation
This adjusts our data model to store/track data from the registry as an image index (with more data fidelity) instead of as a custom data structure. One of the most notable benefits is that we can now track the annotation objects for each architecture, but even more importantly, this allows us to generate the indexes we need to push in deploy directly from data that's on-disk (where the old structure would require querying the registry). Notably, however, this does *not* change deploy (yet) -- those changes are still in progress, but this was a large enough refactor/rewrite that I figured I should start here. This also switches us from using containerd's registry client library to using [`cuelabs.dev/go/oci/ociregistry`](https://pkg.go.dev/cuelabs.dev/go/oci/ociregistry), which is much closer to the distribution-spec APIs (modeled/designed after them, in fact), where containerd's (and frankly most others) are a higher-level abstraction. This is important because we're running into raw number of request rate limits, and almost every library for this always starts with a `HEAD` before pushing content, and this will allow us to go directly for a `PUT` (and then only do the "copy child objects" dance if the `PUT` fails). This also drops our `allTags` data entry (which we only *ever* used the first value of, and then only to identify a `sourceId`/`buildId` in a human-meaningful way, because it's not useful/meaningful for any other use case), and moves `tags` from being arch-specific up to the per-source-object level because it's always identical across all arches for a given `sourceId` so it's silly to copy it N times for every architecture object.
1 parent 2523842 commit 64b5472

26 files changed

+3136
-2001
lines changed

.any-go-nt.sh

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/usr/bin/env bash
2+
set -Eeuo pipefail
3+
4+
# usage: if ./.any-go-nt.sh builds; then expensive-docker-run-command ... go build -o builds ...; fi
5+
6+
shopt -s globstar
7+
8+
for go in **/**.go go.mod go.sum; do
9+
for f; do
10+
if [ "$go" -nt "$f" ]; then
11+
exit 0
12+
fi
13+
done
14+
done
15+
16+
exit 1

.go-env.sh

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,16 @@ args=(
2020

2121
--env "CGO_ENABLED=${CGO_ENABLED-0}"
2222
--env "GOTOOLCHAIN=${GOTOOLCHAIN-local}"
23+
--env GODEBUG
2324
--env GOFLAGS
2425
--env GOOS --env GOARCH
2526
--env GO386
2627
--env GOAMD64
2728
--env GOARM
29+
30+
# hack hack hack (useful for "go run" during dev/test)
31+
--env DOCKERHUB_PUBLIC_PROXY
32+
--env DOCKERHUB_PUBLIC_PROXY_HOST
2833
)
2934
if [ -t 0 ] && [ -t 1 ]; then
3035
args+=( --tty )

.test/builds.json

Lines changed: 1023 additions & 743 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)