fix: delete orphaned digest directories when last tag is removed#111
Merged
hiroTamada merged 5 commits intomainfrom Mar 2, 2026
Merged
fix: delete orphaned digest directories when last tag is removed#111hiroTamada merged 5 commits intomainfrom
hiroTamada merged 5 commits intomainfrom
Conversation
Previously, DeleteImage only removed the tag symlink, leaving the digest directory (containing the erofs rootfs) on disk indefinitely. This caused orphaned digests to accumulate over time. Now DeleteImage checks if the digest is orphaned after removing the tag, and deletes the digest directory if no other tags reference it. This is eager GC - cleanup happens immediately at delete time. Made-with: Cursor
added 2 commits
February 27, 2026 11:51
The CI workflow runs `make oapi-generate` which uses `oapi-codegen@latest`. The latest generator outputs code using `runtime.StyleParamWithOptions` which was added in runtime v1.2.0, but go.mod had v1.1.2 pinned. This caused CI failures on main starting with commit 08958b8. Made-with: Cursor
The CI workflow regenerates oapi.go using `oapi-codegen@latest`. When v2.6.0 was released, it changed struct tag generation (adding omitempty to some fields), causing type mismatches with code in instances.go that uses inline struct literals. Pinning to v2.5.1 ensures CI regenerates the same code that's committed. Made-with: Cursor
1. Race condition (medium severity): Hold createMu during the orphan check and delete sequence to prevent a concurrent CreateImage from creating a tag pointing to the same digest between count and delete. 2. Silent errors (low severity): Actually log errors when countTagsForDigest or deleteDigest fails, instead of silently swallowing them. Made-with: Cursor
sjmiller609
approved these changes
Feb 28, 2026
Made-with: Cursor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
| target, err := resolveTag(p, repository, tag) | ||
| if err != nil { | ||
| continue | ||
| } |
There was a problem hiding this comment.
Silent error skip in tag count risks premature deletion
Medium Severity
countTagsForDigest silently continues past resolveTag errors, which can undercount tags referencing a digest. If the only remaining tag hits a transient I/O or permission error on os.Readlink, the count returns 0, causing DeleteImage to delete a digest directory that is still referenced. The caller already handles errors from countTagsForDigest conservatively (preserves the digest), so propagating the error instead of skipping would be safe.
Additional Locations (1)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
DeleteImageremoves the last tag referencing a digest, the digest directory is now automatically deletedBackground
Previously,
DeleteImageonly removed the tag symlink, leaving digest directories (containing erofs rootfs files) on disk indefinitely. This was documented in the README as "Old digests remain until explicitly garbage collected" but no GC was ever implemented.At current volume (~20 GB/month), the 3.5 TB disk on
prod-iad-hypeman-1has runway, but this becomes a concern at higher deployment volumes post-rollout.Changes
storage.go: AddedcountTagsForDigest()anddeleteDigest()helper functionsmanager.go: UpdatedDeleteImage()to check for orphaned digests and delete themmanager_test.go: Updated existing test + addedTestDeleteImagePreservesSharedDigestREADME.md: Updated documentation to reflect new behaviorCI fixes (pre-existing issues on main)
go.mod: Updatedoapi-codegen/runtimev1.1.2 → v1.2.0 (fixes missingStyleParamWithOptions)Makefile: Pinnedoapi-codegento v2.5.1 (v2.6.0 generates incompatible struct tags)Test plan
TestDeleteImage- verifies digest is deleted when orphanedTestDeleteImagePreservesSharedDigest- verifies digest is preserved when other tags reference itTestDeleteImageNotFound- unchanged, still passesNote on CI fixes
This PR includes fixes for two pre-existing CI issues unrelated to the GC changes:
Runtime version: CI regenerates oapi.go with
oapi-codegen@latest, which usesruntime.StyleParamWithOptionsadded in v1.2.0, but go.mod had v1.1.2.Generator version: oapi-codegen v2.6.0 changed struct tag generation (adds
omitemptyto some fields), causing type mismatches. Pinned to v2.5.1 to match committed code.Both issues caused CI failures starting with commit 08958b8 on main.
Note
Medium Risk
Changes image deletion semantics to eagerly remove on-disk digest directories, so bugs in tag counting or race handling could delete still-referenced image data. Impact is limited to local image storage/GC plus a small dependency/tooling version adjustment.
Overview
Eagerly garbage-collects image digests on delete.
DeleteImagenow resolves the tag to its digest, deletes the tag symlink, then (undercreateMu) checks if the digest is still referenced by any other tags and removes the digest directory when orphaned.Adds storage helpers (
countTagsForDigest,deleteDigest), updates docs to reflect the new behavior, and expands tests to cover both orphan deletion and shared-digest preservation.Separately, pins
oapi-codegenin theMakefileand bumpsgithub.com/oapi-codegen/runtimetov1.2.0to keep generated OpenAPI code/CI consistent.Written by Cursor Bugbot for commit d5aeb5d. This will update automatically on new commits. Configure here.