Skip to content

Skip collection of large sparse files with tar outputs#262

Closed
joost-j wants to merge 2 commits intofox-it:mainfrom
joost-j:feature/skip-large-sparse-files
Closed

Skip collection of large sparse files with tar outputs#262
joost-j wants to merge 2 commits intofox-it:mainfrom
joost-j:feature/skip-large-sparse-files

Conversation

@joost-j
Copy link
Copy Markdown
Contributor

@joost-j joost-j commented Aug 12, 2025

Fixes #143 and possible collection of other large (>10GB) sparse files. Adding this to the TarOutput class since the Python stdlib tarfile.py does not ship with sparse file write support. If specific sparse files need to be collected, they should be treated as special cases just like the UsnJrnl file, as mentioned in ticket #143. This PR at least prevents the creation of huge .tar files; as mentioned we encountered tar files of ~500GB because of this.

Please let me know if the 10GB threshold is okay, or if we can set it to an even lower value.

@codecov
Copy link
Copy Markdown

codecov bot commented Aug 12, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.95%. Comparing base (20d2946) to head (2ab9358).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #262      +/-   ##
==========================================
+ Coverage   44.93%   44.95%   +0.01%     
==========================================
  Files          26       26              
  Lines        3527     3530       +3     
==========================================
+ Hits         1585     1587       +2     
- Misses       1942     1943       +1     
Flag Coverage Δ
unittests 44.95% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Schamper
Copy link
Copy Markdown
Member

I don't think flat out not collecting evidence is an acceptable solution.

@joost-j joost-j closed this Aug 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skip sparse runs when collecting /var/log/lastlog in acquire

2 participants