perf: Decouple filesystem walk from git index construction#12204
Merged
anthonyshew merged 1 commit intomainfrom Mar 9, 2026
Merged
perf: Decouple filesystem walk from git index construction#12204anthonyshew merged 1 commit intomainfrom
anthonyshew merged 1 commit intomainfrom
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Split find_untracked_files into two phases that can run in parallel: 1. walk_candidate_files (I/O-bound): Enumerates all non-gitignored files within scope using the ignore crate's native gitignore support. Only needs the git root path and package prefixes — no tracked index. 2. filter_untracked_from_candidates (CPU-bound): Binary-searches each candidate against ls_tree_hashes and status_entries to identify truly untracked files. Runs after the tracked index is ready. The git root is sent via a oneshot channel as soon as SCM::new() resolves it (~5ms), while new_from_gix_index continues (~267ms). The walk starts immediately and runs in parallel with index construction. Benchmark (110-package monorepo, 30 runs, sandbox): baseline: 853ms ± 19ms improved: 619ms ± 6ms (1.38x faster)
anthonyshew
force-pushed
the
perf/early-untracked-walk
branch
from
f51e39d to
9c07db3
Compare
Contributor
Coverage Report
|
github-actions bot
added a commit
that referenced
this pull request
Mar 9, 2026
## Release v2.8.15-canary.10 Versioned docs: https://v2-8-15-canary-10.turborepo.dev ### Changes - release(turborepo): 2.8.15-canary.9 (#12203) (`24bd765`) - perf: Decouple filesystem walk from git index construction (#12204) (`03f9749`) --------- Co-authored-by: Turbobot <turbobot@vercel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ignorecrate's native gitignore handling (reading.gitignorefrom disk), removing the dependency on the tracked git indexSCM::new()resolves it (~5ms), letting the walk start whilenew_from_gix_index(~267ms) continuesWhy
After #12201, the critical path was
new_from_gix_index(267ms) →find_untracked_files(509ms) = 776ms sequential. These two operations are now fully overlapped.Benchmark (110-package monorepo, 30 runs, sandboxed)
1.38x faster (234ms wall clock saved).
repo_index_untracked_awaitdropped from 505ms to 89ms.Test Coverage
4 new regression tests validating the split walk produces identical results to the original single-pass approach:
test_split_walk_matches_original_path— baseline equivalence verified against no-index subprocess pathtest_split_walk_respects_gitignore— root, nested, and package-level.gitignorerulestest_split_walk_with_untracked_gitignore— untracked.gitignorefiles are applied during the walktest_split_walk_nested_gitignore_scoping— scoped ignore rules don't leak across packagesHow to Review
repo_index.rs—walk_candidate_files(I/O phase),filter_untracked_from_candidates(CPU phase),populate_untracked_from_candidates(integration into RepoGitIndex)builder.rs—git_root_tx/git_root_rxchannel, walk_task spawned from channel,tokio::join!to combine walk + indexlib.rs—SCM::git_root()accessor, re-export ofwalk_candidate_filesgit_index_regression_tests.rs—build_split_repo_indexhelper and 4 new tests