refactor:improve checkpoint and ensure gc to improve disk space

[ijin]

Related GitHub Issue

Closes #3391
Closes #3348
Closes #3080
Closes #3695

Description

Builds upon #3695

Discussion: #3695 (comment)

---

Garbage collection and perf improvements
Upgrade to git 2.49.0 (Windows only) for additional benefits

Test Procedure

Unit tests added
Sanity test on local

Type of Change

• 🐛 Bug Fix: Non-breaking change that fixes an issue.
• ✨ New Feature: Non-breaking change that adds functionality.
• 💥 Breaking Change: Fix or feature that would cause existing functionality to not work as expected.
• ♻️ Refactor: Code change that neither fixes a bug nor adds a feature.
• 💅 Style: Changes that do not affect the meaning of the code (white-space, formatting, etc.).
• 📚 Documentation: Updates to documentation files.
• ⚙️ Build/CI: Changes to the build process or CI configuration.
• 🧹 Chore: Other changes that don't modify src or test files.

Pre-Submission Checklist

• Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
• Scope: My changes are focused on the linked issue (one major feature/fix per PR).
• Self-Review: I have performed a thorough self-review of my code.
• Code Quality:
  • My code adheres to the project's style guidelines.
  • There are no new linting errors or warnings (npm run lint).
  • All debug code (e.g., console.log) has been removed.
• Testing:
  • New and/or updated tests have been added to cover my changes.
  • All tests pass locally (npm test).
  • The application builds successfully with my changes.
• Branch Hygiene: My branch is up-to-date (rebased) with the main branch.
• Documentation Impact: I have considered if my changes require documentation updates (see "Documentation Updates" section below).
• Changeset: A changeset has been created using npm run changeset if this PR includes user-facing changes or dependency updates.
• Contribution Guidelines: I have read and agree to the Contributor Guidelines.

Screenshots / Videos

Documentation Updates

Additional Notes

Get in Touch

---

Important

Refactor ShadowCheckpointService to enhance garbage collection and checkpoint handling, improving disk space management and performance.

• Behavior:
  • Refactor ShadowCheckpointService to improve garbage collection and disk space management.
  • Introduce tryRepack() to handle repository repacking with platform-specific commands.
  • Implement periodic garbage collection in saveCheckpoint() and after branch deletion in deleteBranch().
  • Cache nested git directory paths in findAndCacheNestedGitRepoPaths() to avoid repeated scans.
  • Modify stageAll() to use git status for precise file staging.
• Tests:
  • Add unit tests in ShadowCheckpointService.test.ts to verify new garbage collection and checkpoint behaviors.
  • Mock executeRipgrep to simulate nested git directory detection.
  • Test handling of new, deleted, and ignored files in checkpoints.
• Misc:
  • Upgrade to Git 2.49.0 on Windows for enhanced repack capabilities.
  • Add static logger support in ShadowCheckpointService.

^{This description was created by for 2d8d2d3. You can customize this summary. It will automatically update as commits are pushed.}

[adamhill]

Awesome! I have been wondering for a long time if git prune alone could reduce our checkpoint size enough. I guess we need the tag team with repack and repo surgery with ripgrep

Do you have any idea how much this saves us in repo space 20, 30, 50%? We might be able to find people with really large repo's to be guinea pigs if you want a big stress test (unless you personally already have them and this is why you made the PR :-) ) 🦘 🤟

[samhvw8]

@adamhill i think the problems is we don't need to track all of them, we just need to track what file going to change, that will be better (you can see checkpoint use git add . ) we can change it to (git add )

[ijin]

@adamhill Running git gc for my 2.1GB repo reduced .git from 198MB to 150MB, so about 25%. But I guess it really depends on your project. You can try it out yourself!

[daniel-lxs]

Hey @ijin, thank you for taking over this PR, I left some questions and suggestions.

My main concern about this new implementation is finding a way to test it correctly, it might also be a good idea to separate the garbage collection functionality from the stageAll method of tracking file changes into a separate PR so it's easier to test.

I'm curious to know what you think of this!

[daniel-lxs]

The new stageAll implementation makes multiple git calls (status, add, rm) compared to the previous single git add . call. Have you measured the performance impact of this change? For large repositories with many changes, this could potentially be slower.

While the precision is valuable for avoiding unnecessary files, it might be worth benchmarking this against the old approach to ensure we're not introducing a performance regression.

[ijin]

From what I know, the expensive full repo walk is done once by git status (just as it would in git add .). Subsequent git add/rm calls only does a lightweight stat(), making their overhead minimal.

That said, here are some generated benchmarks I ran on Roo-Code itself. I don't see it much as a performance regression.

• benchmark: https://gist.github.com/ijin/8946d83e77ddcb108d62c82073550eaf
• output:

---

=== stageAll() Performance Benchmark ===
Repository: /tmp/Roo-Code
Runs per test: 5


=== Creating Large Change Scenario ===
Created 2 nested git repositories
Found 80 TypeScript files to modify
Modified 60 TypeScript files
Created 12 new files
Successfully added and committed delete-me files
Created and deleted 7 files
Modified package.json
Large change scenario created successfully

=== Repository Statistics ===
Total tracked files: 1630
Modified files: 80
Staged files: 0

=== Benchmarking OLD Approach (Full stageAll with git add .) ===
Found 2 nested .git directories in 26.9ms
Run 1: 242.2ms (rename: 29.4ms, git: 212.5ms)
Run 2: 157.8ms (rename: 3.3ms, git: 154.5ms)
Run 3: 153.1ms (rename: 2.7ms, git: 150.3ms)
Run 4: 111.0ms (rename: 1.7ms, git: 109.2ms)
Run 5: 109.9ms (rename: 2.0ms, git: 107.9ms)
OLD Approach (Full stageAll with git add .) average: 154.8ms
  - Rename operations: 7.8ms
  - Git operations: 146.9ms

=== Benchmarking NEW Approach (Full stageAll with selective staging) ===
Run 1: 436.5ms (rename: 2.0ms, git: 434.3ms) [77 add, 7 rm]
Run 2: 457.8ms (rename: 2.2ms, git: 455.6ms) [77 add, 7 rm]
Run 3: 443.2ms (rename: 1.7ms, git: 441.4ms) [77 add, 7 rm]
Run 4: 486.8ms (rename: 2.0ms, git: 484.7ms) [77 add, 7 rm]
Run 5: 459.9ms (rename: 2.2ms, git: 457.7ms) [77 add, 7 rm]
NEW Approach (Full stageAll with selective staging) average: 456.8ms
  - Rename operations: 2.0ms
  - Git operations: 454.7ms

=== Performance Comparison ===
OLD approach average: 154.8ms
  - Rename operations: 7.8ms
  - Git operations: 146.9ms
NEW approach average: 456.8ms
  - Rename operations: 2.0ms
  - Git operations: 454.7ms

[jcaplan]

The new stageAll implementation makes multiple git calls (status, add, rm) compared to the previous single git add . call. Have you measured the performance impact of this change? For large repositories with many changes, this could potentially be slower.

While the precision is valuable for avoiding unnecessary files, it might be worth benchmarking this against the old approach to ensure we're not introducing a performance regression.

Fwiw add all takes like 5-10 minutes on my repo.

[daniel-lxs]

In the tryRepack method, errors are logged but the method continues silently. Should we consider:

1. Tracking these failures (perhaps with a counter)?
2. Alerting the user if repacking consistently fails?
3. Adding telemetry to understand how often repacking fails in the wild?

This would help identify problematic Git installations or configurations that might prevent users from benefiting from the optimization.

[ijin]

We could for instance, keep track and alert if there are more than N consecutive failures.

But I'm not familiar enough with the project on how to implement user facing alerts or telemetry...

[daniel-lxs]

The code mentions using Git 2.49.0 on Windows for the --path-walk option, but:

1. How do we ensure users have this version installed?
2. Should we add a version check and provide a helpful message if they're on an older version?
3. Is this documented in the PR description or user-facing documentation?

This could be confusing for Windows users who don't get the expected performance improvements due to their Git version.

[ijin]

1. No need to ensure, but it does have a fallback strategy
2. If it fails, it logs and tries the fallback
3. Yes, it is mentioned in the PR description

[daniel-lxs]

I am curious about this number, could it make it hard for the gc to trigger on short tasks? would it be a good idea to lower it so it triggers more often? it would be a good idea to know how much of an impact running the gc causes in the system.

[ijin]

This number was what was suggested in the original PR. It would trigger less for shorter tasks, but this is already configurable with GC_CHECKPOINT_THRESHOLD.

We could try running gc on a large codebase and measure it if you have one in mind.

[adamhill]

Kilo is fighting the git bloat as well. They are trying out a cron job to vacuum the checkpoint repo that deletes tasks after a period of time, if someone want to test it out.

Kilo-Org/kilocode#536 (comment)

[ijin]

@daniel-lxs I've made some changes and comments.

[daniel-lxs]

I've been looking at this PR for a while now, and something doesn't feel quite right. I think the best way to proceed is to split this implementation into two separate PRs: one for the garbage collection and another for the new staging method. This will make it much easier to test and identify potential issues.

This doesn't mean the implementation is bad; we just need a better separation of concerns to properly test everything. It would also be helpful to have clear steps to reproduce the original issue to verify that these changes resolve it.

I'll be closing this PR temporarily, but this PR can be used as base for any future implementations.

Thank you @ijin for taking over the original PR and working on this one.