Git history bloated by old binaries from update_resources workflow #59
Description
Hi,
While investigating clone size of this repo, I found many large, obsolete binaries still present in the Git history (e.g. multiple old versions of windows/winPEAS/*, linux/windows/chisel*, windows/py37-64.exe, windows/SysinternalsSuite/RDCMan*.exe).
Using:
$ git rev-list --objects --all --missing=print |
git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
sed -n 's/^blob //p' |
sort --numeric-sort --key=2 |
cut -c 1-12,41- |
$(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest |
tail -n 50
cb519ba0b460 8.5MiB linux/chisel/chisel32
9ae6ac0d5a7e 8.8MiB windows/chisel/chisel32.exe
76f3e0cb26d3 8.9MiB linux/chisel/chisel64
0054f7e0a12e 9.3MiB windows/chisel/chisel64.exe
1a0c453c25e1 9.4MiB linux/chisel/chisel32
166aa4bc4cea 9.5MiB windows/winPEAS/winPEASx64_ofs.exe
098fbbf703ad 9.5MiB windows/winPEAS/winPEASx86_ofs.exe
36b8d018505e 9.5MiB windows/winPEAS/winPEASany_ofs.exe
11fd3fb95f07 9.5MiB windows/winPEAS/winPEASx64_ofs.exe
59e664a9c4bc 9.5MiB windows/winPEAS/winPEASx64_ofs.exe
201a12dafe0e 9.5MiB windows/winPEAS/winPEASany_ofs.exe
652a39707470 9.5MiB windows/winPEAS/winPEASx86_ofs.exe
b386b84cc0cb 9.5MiB windows/winPEAS/winPEASx86_ofs.exe
dba4b65a76f7 9.5MiB windows/winPEAS/winPEASany_ofs.exe
43b9fba0725f 9.6MiB windows/winPEAS/winPEASx64_ofs.exe
167dfd2eba5b 9.6MiB windows/winPEAS/winPEASx86_ofs.exe
391cc65ec0b7 9.6MiB windows/winPEAS/winPEASany_ofs.exe
ade5983594f1 9.6MiB windows/winPEAS/winPEASx64_ofs.exe
d97ae8a0f0cf 9.6MiB windows/winPEAS/winPEASx64_ofs.exe
2cba4baae420 9.6MiB windows/winPEAS/winPEASx86_ofs.exe
5a25ce39455c 9.6MiB windows/winPEAS/winPEASany_ofs.exe
a8a7511159f4 9.6MiB windows/winPEAS/winPEASx86_ofs.exe
eb7cf322b1af 9.6MiB windows/winPEAS/winPEASany_ofs.exe
dd6053ef0296 9.7MiB windows/chisel/chisel32.exe
cf3cbf93adf4 9.7MiB windows/LaZagne/lazagne.exe
0b34ad95f6a7 9.7MiB windows/winPEAS/winPEASx64.exe
565c40189136 9.7MiB windows/winPEAS/winPEASx86.exe
c97bb3de5fab 9.7MiB windows/winPEAS/winPEASany.exe
24c5bc82af67 9.7MiB windows/winPEAS/winPEASx64.exe
8334b425bc74 9.7MiB windows/winPEAS/winPEASx64.exe
04fd396a51d7 9.7MiB windows/winPEAS/winPEASx86.exe
4cb95de812c1 9.7MiB windows/winPEAS/winPEASany.exe
c88a1dd83408 9.7MiB windows/winPEAS/winPEASx86.exe
f0563feec074 9.7MiB windows/winPEAS/winPEASany.exe
604759bcb38e 9.7MiB windows/winPEAS/winPEASx64.exe
3d7d646ad0a4 9.7MiB windows/winPEAS/winPEASany.exe
4605a6fdbb6e 9.7MiB windows/winPEAS/winPEASx86.exe
22dab42cca4a 9.7MiB windows/winPEAS/winPEASx64.exe
3644d6b18347 9.7MiB windows/winPEAS/winPEASx64.exe
15b0589e8a75 9.7MiB windows/winPEAS/winPEASany.exe
39132aa172bb 9.7MiB windows/winPEAS/winPEASx86.exe
89dfdbc15d96 9.7MiB windows/winPEAS/winPEASx86.exe
dea41568ebab 9.7MiB windows/winPEAS/winPEASany.exe
44e7bf536ac8 9.8MiB linux/chisel/chisel64
280035c7ff81 10MiB windows/chisel/chisel64.exe
3c979d6b2875 16MiB windows/py37-64.exe
3532ddc7e414 61MiB windows/SysinternalsSuite/RDCMan-x86.exe
b5201a3c0788 61MiB windows/SysinternalsSuite/RDCMan-x86.exe
be84deca1604 65MiB windows/SysinternalsSuite/RDCMan.exe
e769d2a258a8 65MiB windows/SysinternalsSuite/RDCMan.exethe largest blobs alone account for roughly 650–700 MiB of old binaries that are no longer needed but are still downloaded on every clone/submodule update, making the repo significantly slower and heavier than necessary.
This seems to come from .github/workflows/update_resources.yml, which repeatedly commits new versions of these tools without ever cleaning old ones from history.
Possible improvements: run some manual history cleanup once a time (e.g. git filter-repo / BFG) to drop old versions of these tools