WIP: Staged layer creation #378
Conversation
Add a new function to stage additions. This should be used to extract the layer content into a temp directory without holding the storage lock and then under the lock just rename the directory into the final location to reduce the lock contention. Signed-off-by: Paul Holzinger <[email protected]>
That caller in create() already had the layer created in memory so another lookup roundtrip is unnecessary here. Signed-off-by: Paul Holzinger <[email protected]>
That caller in create() already had the layer created in memory so another lookup roundtrip is unnecessary here. Signed-off-by: Paul Holzinger <[email protected]>
It is not clear to me when it will hit the code path there, by normal layer creation we always pass a valid parent so this branch is never reached AFAICT. Let's remove it and see if all tests still pass in podman, buildah and others... Signed-off-by: Paul Holzinger <[email protected]>
|
✅ A new PR has been created in buildah to vendor these changes: containers/buildah#6414 |
|
Podman PR containers/podman#27251 and the buildah test PR containers/buildah#6414 from the bot both look good so that means we can remove the special case from ApplyDiff() in overlay I think, ref containers/podman#25862 (comment) I still need to work on the actual feature here though to extract while the store in unlocked. |
| // assume layer creation and applying a staged layer are separate steps. Production pull code always uses the | ||
| // other path, where layer creation is atomic. | ||
| return layer, rlstore.applyDiffFromStagingDirectory(args.ID, args.DiffOutput, args.DiffOptions) | ||
| return layer, rlstore.applyDiffFromStagingDirectory(layer, args.DiffOutput, args.DiffOptions) |
There was a problem hiding this comment.
Very non-blocking
I’m a bit worried about trusting the out-of-layers.go code to keep a correct locked value of layer here; ISTR we mostly keep “this data is kept locked and consistent” to the layers.go layer of the call stack; in store.go, a Layer is more of a “possibly-already-obsolete snapshot of data”.
Technically this clearly works (we are holding the RW layer store lock here).
As a general long-term direction, I’d like to get rid of the “if !layerExists { report error }; doOperation” pattern, in favor of a direct ”doOperation; handle layerDoesNotExistError“. This applies to the calls from store.go to layers.go, mostly just because the latter is shorter; and it applies much more to the calls from external callers to store.go, because in that case two calls means two expensive locking operations, and possibly the situation changing in the meantime. (Also, the Get calls also serve as a “resolve name or ID prefix to a full ID” step — ideally that should not be happening at all, after we process/resolve CLI inputs. But changing that at the external c/storage API surface is somewhat risky.)
So, I’d prefer rwLayerStore.applyDFSD to continue to accept an ID, and maybe layers.go can add an intermediate function with a *Layer.
But then again, see the comment just above — this is ~mostly dead code. That… means both that it is not worth investing much into, and that it is at higher risk of accidentally being broken.
Also, there’s .create already using a pointer, so this is by no means a hard rule.
No description provided.