Introduce git backfill and its --sparse option
#29
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
derrickstolee
force-pushed
the
path-walk-on-full
branch
from
a9fc233 to
b84cf7a
Compare
derrickstolee
force-pushed
the
backfill-on-full
branch
from
33fa431 to
a89ee77
Compare
derrickstolee
force-pushed
the
path-walk-on-full
branch
from
b84cf7a to
27bd942
Compare
derrickstolee
force-pushed
the
backfill-on-full
branch
from
a89ee77 to
1d8fb85
Compare
derrickstolee
force-pushed
the
path-walk-on-full
branch
from
27bd942 to
4059aca
Compare
derrickstolee
force-pushed
the
backfill-on-full
branch
2 times, most recently
from
f078d6c to
8dbe6c9
Compare
derrickstolee
force-pushed
the
path-walk-on-full
branch
from
4059aca to
a75403b
Compare
derrickstolee
force-pushed
the
backfill-on-full
branch
from
8dbe6c9 to
c6364d8
Compare
derrickstolee
force-pushed
the
path-walk-on-full
branch
2 times, most recently
from
47a495c to
e91c3b9
Compare
derrickstolee
force-pushed
the
backfill-on-full
branch
2 times, most recently
from
e8cedf2 to
ef75468
Compare
derrickstolee
force-pushed
the
path-walk-on-full
branch
2 times, most recently
from
946bd8f to
4b68c8e
Compare
derrickstolee
force-pushed
the
backfill-on-full
branch
2 times, most recently
from
b6cbb37 to
cb4aebb
Compare
derrickstolee
force-pushed
the
path-walk-on-full
branch
from
4b68c8e to
b74b75a
Compare
dscho
force-pushed
the
path-walk-on-full
branch
from
b74b75a to
d9fc474
Compare
In anticipation of implementing 'git backfill', populate the necessary files with the boilerplate of a new builtin. RFC TODO: When preparing this for a full implementation, make sure it is based on the newest standards introduced by [1]. [1] https://lore.kernel.org/git/[email protected]/T/#m606036ea2e75a6d6819d6b5c90e729643b0ff7f7 [PATCH 1/3] builtin: add a repository parameter for builtin functions Signed-off-by: Derrick Stolee <[email protected]>
The default behavior of 'git backfill' is to fetch all missing blobs that are reachable from HEAD. Document and test this behavior. The implementation is a very simple use of the path-walk API, initializing the revision walk at HEAD to start the path-walk from all commits reachable from HEAD. Ignore the object arrays that correspond to tree entries, assuming that they are all present already. Signed-off-by: Derrick Stolee <[email protected]>
Users may want to specify a minimum batch size for their needs. This is only a minimum: the path-walk API provides a list of OIDs that correspond to the same path, and thus it is optimal to allow delta compression across those objects in a single server request. We could consider limiting the request to have a maximum batch size in the future. Signed-off-by: Derrick Stolee <[email protected]>
One way to significantly reduce the cost of a Git clone and later fetches is to use a blobless partial clone and combine that with a sparse-checkout that reduces the paths that need to be populated in the working directory. Not only does this reduce the cost of clones and fetches, the sparse-checkout reduces the number of objects needed to download from a promisor remote. However, history investigations can be expensie as computing blob diffs will trigger promisor remote requests for one object at a time. This can be avoided by downloading the blobs needed for the given sparse-checkout using 'git backfill' and its new '--sparse' mode, at a time that the user is willing to pay that extra cost. Note that this is distinctly different from the '--filter=sparse:<oid>' option, as this assumes that the partial clone has all reachable trees and we are using client-side logic to avoid downloading blobs outside of the sparse-checkout cone. This avoids the server-side cost of walking trees while also achieving a similar goal. It also downloads in batches based on similar path names, presenting a resumable download if things are interrupted. This augments the path-walk API to have a possibly-NULL 'pl' member that may point to a 'struct pattern_list'. This could be more general than the sparse-checkout definition at HEAD, but 'git backfill --sparse' is currently the only consumer. Be sure to test this in both cone mode and not cone mode. Cone mode has the benefit that the path-walk can skip certain paths once they would expand beyond the sparse-checkout. Signed-off-by: Derrick Stolee <[email protected]>
The previous change introduced the '--[no-]sparse' option for the 'git backfill' command, but did not assume it as enabled by default. However, this is likely the behavior that users will most often want to happen. Without this default, users with a small sparse-checkout may be confused when 'git backfill' downloads every version of every object in the full history. However, this is left as a separate change so this decision can be reviewed independently of the value of the '--[no-]sparse' option. Add a test of adding the '--sparse' option to a repo without sparse-checkout to make it clear that supplying it without a sparse-checkout is an error. Signed-off-by: Derrick Stolee <[email protected]>
derrickstolee
force-pushed
the
backfill-on-full
branch
from
cb4aebb to
d95811c
Compare
This is a highly useful command, and we want it to get some testing "in the wild". However, the patches have not yet been reviewed on the Git mailing list, and are therefore subject to change. By marking the command as experimental, users will be warned to pay attention to those changes. Signed-off-by: Johannes Schindelin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is based on top of #28 for merging after that has merged.
This introduces
git backfill [--[no-]sparse]as a way to fill in the missing blobs for a blobless partial clone. When in a repo using sparse-checkout, it will download only the blobs matching paths within the current sparse-checkout patterns.