Skopeo delete image failing - attempts deleting manifest before tag

[bcaton85]

Running command skopeo delete docker://gcr.io/<project>/<image>:<tag> --creds <cred>.

Fails with: {"errors":[{"code":"GOOGLE_MANIFEST_DANGLING_TAG","message":"Manifest is still referenced by tag: <tag>"}]} (400 Bad Request)"

It looks like it is attempting to delete the manifest first before removing the tags but this is not allowed by GCR. This may cause issues where it would require deleting all tags before a manifest can be removed. An associated issue with details can be found here: genuinetools/reg#136.

[jonjohnsonjr]

I don't think this repo is the right place for this issue, you probably want skopeo?

[mjgallag]

@jonjohnsonjr not sure this is exactly the right repo either but I just hit this issue as well and would love to hear thoughts from GCR/GAR perspective. One would like to be able to manage image lifecycle via Docker Registry HTTP API V2 which tools like skopeo & reg use. Deleting an untagged image works just fine and is great for when tag is moved from an old sha to a new one. The problem is that Docker Registry HTTP API V2 doesn't support deleting tags separately from shas, see containers/skopeo#489 (comment). So when your intention is to delete a tagged image you cannot do so in GCR/GAR without first figuring out the GCR/GAR specific API call to delete a tag (ugh). I can understand the desire for this safety mechanism rather than just removing the tags along with the sha. But, ideally, when deleting via Docker Registry HTTP API V2, GCR/GAR would go ahead and just delete the tags along with the sha to allow for interoperability. One could see this error still being enforced by Google for non Docker Registry HTTP API V2 delete calls if desired. Thanks in advance for your thoughts.

[jonjohnsonjr]

@mjgallag I generally agree with you and share in your frustrations here.

One would like to be able to manage image lifecycle via Docker Registry HTTP API V2 which tools like skopeo & reg use. Deleting an untagged image works just fine and is great for when tag is moved from an old sha to a new one. The problem is that Docker Registry HTTP API V2 doesn't support deleting tags separately from shas, see containers/skopeo#489 (comment).

So we've been discussing some of this over in the OCI world, and the draft distribution spec includes the ability to delete tags: https://github.com/opencontainers/distribution-spec/blob/e734841aa1c74860e6371086cdaf5d52ec220602/spec.md#content-management

The distribution spec is intended to be an evolution of the registry spec, as there are some inaccuracies and outdated information in the registry spec.

So when your intention is to delete a tagged image you cannot do so in GCR/GAR without first figuring out the GCR/GAR specific API call to delete a tag (ugh). I can understand the desire for this safety mechanism rather than just removing the tags along with the sha. But, ideally, when deleting via Docker Registry HTTP API V2, GCR/GAR would go ahead and just delete the tags along with the sha to allow for interoperability.

You'll perhaps find it even more frustrating that it is impossible to delete anything from Docker Hub via the registry API, at all.

The decision around requiring explicit tag deletion was made before my time, and I doubt we would change that behavior now, given that some customers likely rely on it. I understand the desire for compat here, but I also don't want to break anyone.

I would be open to some standard headers or error messages that communicate what a client needs to do in this situation. You can imagine some HTTP header that is the equivalent of --force-delete-flags from gcloud, or even a standard error response body code/body that indicates exactly which tags need to be deleted (or which manifest lists still reference the image).

There's some discussion in the distribution spec around improving tag deletion, if you're interested in joining in: opencontainers/distribution-spec#114

[mjgallag]

@jonjohnsonjr thanks for pointing me to the new spec. I'm definitely going to look at that more closely as I try to figure out a reasonable spec compliant image lifecycle flow, specifically for images consumed internally by CI/CD. I'm building on every push in every branch so storage requirements can quickly grow. Since it is internal I'd like to find a way to easily identify and delete unused images, i.e. garbage collection. If I can't find a flow that works with the current spec I'll definitely join in there. Then I'll move to nudging various container registries to fully support the spec.

[jonjohnsonjr]

Since it is internal I'd like to find a way to easily identify and delete unused images, i.e. garbage collection. If I can't find a flow that works with the current spec I'll definitely join in there.

I've been pushing on this problem for a while. I put together a rough proposal as a starting point here: opencontainers/distribution-spec#222 (comment)

I haven't been able to get much feedback from registry folks other than Steve bikeshedding it, but I'll be happy to put together a proposal PR once I can get some rough consensus on it.

[mjgallag]

@jonjohnsonjr ugh, thanks for trying to push this thru! I'm focusing on some other parts of the CI/CD work I'm doing at the moment but I will likely get back to the image garbage collection piece within a few weeks as I see it as a requirement for MVP. The company that'll be making use of this first is already on Google Cloud so I'll very likely drop directly to into GAR API as needed but goal is, of course, to standardize.