You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This requires the input to be in the correct order (children first), but that was already an implied constraint.
In my testing, this takes our `.test/test.sh --deploy` block testing deploy from ~5s down to ~2.5s.
I used "show timestamps" in our GHA CI jobs to compare them also, and they're currently at ~7s, and after this change they drop to ~1s.
I've tested this with my personal deploy job, which was taking ~2m40s before, and is ~1m18s after.
returnnormal, fmt.Errorf("%s: blobs are always by-digest, and thus need a digest: %s", debugId, normal.CopyFrom)
258
+
}
259
+
260
+
default:
261
+
panic("unknown type: "+string(normal.Type))
262
+
// panic instead of error because this should've already been handled/normalized above (so this is a coding error, not a runtime error)
263
+
}
264
+
265
+
returnnormal, nil
266
+
}
267
+
268
+
// WARNING: many of these codepaths will end up writing to "normal.Lookup", which because it's a map is passed by reference, so this method is *not* safe for concurrent invocation on a single "normal" object! see "normal.clone" (above)
// a set of RWMutex objects for synchronizing the pushing of "child" objects before their parents later in the list of documents
41
+
// for every RWMutex, it will be *write*-locked during push, and *read*-locked during reading (which means we won't limit the parallelization of multiple parents after a given child is pushed, but we will stop parents from being pushed before their children)
// locks are per-digest, but refs might be 20 tags on the same digest, so we need to get one write lock per repo@digest and release it when the first tag completes, and every other tag needs a read lock
// if we have a "child" mutex, lock it immediately so we don't create a race between inputs
97
+
mutex.Lock() // (this gets unlocked in the goroutine below)
98
+
// this is sane to lock here because interdependent inputs are required to be in-order (children first), so if this hangs it's 100% a bug in the input order
99
+
}
67
100
}
68
-
fmt.Println()
69
-
}
70
101
71
-
fmt.Println()
102
+
// make a (deep) copy of "normal" so that we can use it in a goroutine ("normal.do" is not safe for concurrent invocation)
103
+
normal:=normal.clone()
104
+
105
+
wg.Add(1)
106
+
gofunc() {
107
+
deferwg.Done()
108
+
109
+
ifmutex!=nil {
110
+
defermutex.Unlock()
111
+
}
112
+
113
+
// before we start this job (parallelized), if it's a raw data job we need to parse the raw data and see if any of the "children" are objects we're still in the process of pushing (from a previously parallel job)
114
+
iflen(normal.Data) >2 { // needs to at least be bigger than "{}" for us to care (anything else either doesn't have data or can't have children)
115
+
// explicitly ignoring errors because this might not actually be JSON (or even a manifest at all!); this is best-effort
116
+
// TODO optimize this by checking whether normal.Data matches "^\s*{.+}\s*$" first so we have some assurance it might work before we go further?
// we don't *know* that all the lookup references are children, but if any of them have an explicit digest, let's treat them as potential children too (which is fair, because they *are* explicit potential references that it's sane to make sure exist)
// *technically* this should be two separate structs chosen based on mediaType (https://github.com/opencontainers/distribution-spec/security/advisories/GHSA-mc8v-mgrf-8f4m), but that makes the code a lot more annoying when we're just collecting a list of potential children we need to copy over for the parent object to push successfully
11
+
12
+
// intentional subset of https://github.com/opencontainers/image-spec/blob/v1.1.0/specs-go/v1/index.go#L21 to minimize parsing
13
+
Manifests []ocispec.Descriptor`json:"manifests"`
14
+
15
+
// intentional subset of https://github.com/opencontainers/image-spec/blob/v1.1.0/specs-go/v1/manifest.go#L20 to minimize parsing
16
+
Config*ocispec.Descriptor`json:"config"`// have to turn this into a pointer so we can recognize when it's not set easier / more correctly
17
+
Layers []ocispec.Descriptor`json:"layers"`
18
+
}
19
+
20
+
// opportunistically parse a given manifest for any *potential* child objects; will return JSON parsing errors for non-JSON
0 commit comments