exporter: add local exporter mode=delete#6561
Draft
crazy-max wants to merge 3 commits intomoby:masterfrom
Draft
Conversation
crazy-max
force-pushed
the
exporter-local-mirror
branch
2 times, most recently
from
48e766e to
071d3cf
Compare
crazy-max
force-pushed
the
exporter-local-mirror
branch
2 times, most recently
from
477e40e to
22b3c81
Compare
tonistiigi
reviewed
Mar 6, 2026
Member
tonistiigi
left a comment
tonistiigi
left a comment
There was a problem hiding this comment.
I'd call it delete like rsync --delete. In Buildx I think we need more restrictions. I'd only allow it for subdirs there, with a second-level opt-in needed to mess with the main workdir or things outside it.
crazy-max
changed the title
Member
Author
Agreed, renamed the mode from |
crazy-max
force-pushed
the
exporter-local-mirror
branch
from
22b3c81 to
80b1312
Compare
tonistiigi
reviewed
Mar 24, 2026
Member
There was a problem hiding this comment.
Looks like there is problem with multi-platform outputs. Added regression test in https://github.com/moby/buildkit/compare/master...tonistiigi:buildkit:exporter-local-mirror-upt?expand=1
The way this seems to work is using the fsutil transfer in merge=false mode that is usually used for context transfer. This is good for performance as existing files are not transfered but there are a couple of problems with it:
- The differ is metadata based. This is considered ok for context transfer for performance, but not sure if that is also the case for outputs. Especially because you can have reproducible builds. This would unexpectedly create wrong output results that should be avoided. I think outputs should use differ based on content checksum.
- The differ is only safe if the output directory is immutable. If it is not then this could lead to breakouts. Using differ in here would require some refactoring. If that refactoring reduces performance then we should use different code for output and context transfer(where we can guarantee immutability).
session/filesync/filesync.go
Outdated
| if outdir, ok := sp.outdirs[id]; ok { | ||
| return syncTargetDiffCopy(stream, outdir) | ||
| mode := outdir.mode | ||
| if v := opts[keyExporterMetaPrefix+ExporterMetaLocalDirMode]; len(v) > 0 { |
Member
There was a problem hiding this comment.
This code is used for context transfer to daemon and shouldn't allow client to control the receive behavior.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
crazy-max
force-pushed
the
exporter-local-mirror
branch
from
80b1312 to
117013b
Compare
Member
Author
Completely forgot about multi-platform 🙈. I picked your commit adding new tests. I think |
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
crazy-max
force-pushed
the
exporter-local-mirror
branch
from
117013b to
21840d2
Compare
Member
Author
Addressed in last commit. Now |
Member
Author
|
Hum test fails on Windows https://github.com/moby/buildkit/actions/runs/23750766249/job/69192263818?pr=6561#step:8:268 Seems the destination cleanup happened but the root file was not materialized back. That leaves dest partially empty. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
partially fixes #2805
This PR adds support for a
modeattribute on the local exporter output path.The default behavior remains
mode=copy, which preserves destination files that are not present in the exported result. A newmode=deletebehavior is added so the destination matches the exported result by removing stale files and directories.Because
mode=deletecan remove files under the destination directory, this feature is intentionally explicit opt-in and should be used only with trusted build definitions and carefully reviewed output destinations.On client side like Buildx we should probably have some safety gate like
--allow local-output-delete(or equivalent) for remote Bake invocations. Optionally block/require explicit allow whendest=.withmode=delete?Maybe we could have some kind of scope attribute like
mode=delete,delete-scope=dist,assetsso it only deletes stale files under those relative paths (and reject unsafe scopes like..or absolute paths)?Also the
backupmode from the issue is intentionally left for a follow-up change. I think we would need changes infsutil.Receive(or diff callbacks).