Partial mirror aka download blobs on demand

[rgl]

Is there a way to configure zot mirror to only download blobs on demand?

This could help in:

1. Many images are multi-platform, but we end-up only using small (in fact, for my amd64 cluster, its only one image) subset of the supported image platforms.
2. Cause less load on upstream registry.
3. Since its on-demand, the image would be available faster.

For example, many images are multi-platform, it seems, when zot 2.0.1 mirrors a image, it first downloads all the blobs into the .sync sub-directory, and only when everything is download, it "promotes" the image into a visible image. Downloading everything seems to cause too much load in upstream registry. and things starts to stall/fail (e.g. today was the first time I've noticed this, while it failed (and never worked) while downloading the quay.io/argoproj/argocd:v2.10.3 image from https://quay.io/repository/argoproj/argocd/manifest/sha256:18a1e1b4006c387a6a2037859004be3e6330bfb6b48809d6a342c6daa76a2e9b).

FWIW, Here's I'm configuring zot 2.0.1:

extensions:
  ui:
    enable: true
  metrics:
    enable: true
    prometheus:
      path: /metrics
  search:
    enable: true
    cve:
      updateInterval: 2h
  sync:
    enable: true
    registries:
      - urls:
          - https://quay.io
        content:
          - prefix: "**"
            destination: /mirror/quay.io
        onDemand: true
        tlsVerify: true

[rchincha]

@rgl so maybe add an additional param called platform: under content:
This could mitigate it somewhat, however with large images an ever-present problem though.
You raise a good point that failures cause a worse flood than the good path.

[rgl]

I would rather prefer that zot downloads everything on-demand, and only the specific parts that are requested by a client, instead of buffering/staging the entire logical image(s).

The reasoning being, the client knowns which parts/blobs it requires. For example, the client typically only downloads images for its own platform; so those would be the only ones that zot needs to actually mirror. If we really want to preload zot, we could use a tool like crane/skopeo/regctl.

[andaaron]

Normally when an image is pushed to the the registry using the REST API in the spec the manifest/index is only accepted if all references in the descriptors (blobs) are already present in the registry, to avoid inconsistencies.

Allowing indexes to be present with some invalid references (for architectures the user doesn't want) would lead to inconsistencies. If the rest API doesn't allow it, why would sync allow it? This would also mean modifications in the storage logic to handle errors on these missing blobs. It may also need changes on the logic of GC-in invalid images.

Note we cannot have zot managing/modifying the image index to keep only valid references present in zot, as the clients downloading this index would not know there are other architectures available and what digests they have. Not to mention the digest of the index itself would change so any references to it by other artifacts/images would be invalid.

---

We could download individual manifests for different platforms, without the index, but then they would have to have distinguishable tags per platform, which clients would need to know in order to access this content. Would this work?

[Cellivar]

Bumping this thread as we're exploring Zot and finding this behavior to be unexpectedly clunky.

Similar to rgl's setup, our cluster is either ARM64 or AMD64. We have no interest in spending storage capacity or bandwidth on mips images 😸

We do not and will never care if the mips architecture layers are present in our Zot registry, regardless of what the manifest claims to have available. Or, put differently: it is acceptable for my Zot instance to pass the buck of responsibility for manifest validity to the upstream registry. Unless told otherwise I would expect Zot to:

• Assume a manifest Zot got from upstream is valid for that upstream registry.
• Assume Zot can, in the future, go and get anything the manifest references too.
• Get what the client explicitly asked for now, skip everything else until later.
• Happily throw error messages if later the manifest cannot be satisfied from the upstream source.

I can see the appeal of an opt-in setting enforceCompleteManifestSync: true for the folks who want everything all the time. I suspect the majority of users don't need that though.

---

Normally when an image is pushed to the the registry using the REST API in the spec the manifest/index is only accepted if all references in the descriptors (blobs) are already present in the registry, to avoid inconsistencies.

Ah, so this is a situation where the registry can not assume it can get the missing content at a later time. I would always expect Zot to reject my CI system's builds that failed to push a layer or something. The registry can't go and ask my CI worker for a different architecture at a later date, it needs everything now for consistency.

On the other hand a Zot registry should be safe in the assumption it can ask for more at a later time. With that assumption in place Zot just needs to support limiting mirroring to what the client asked for.

but then they would have to have distinguishable tags per platform, which clients would need to know in order to access this content. Would this work?

After having personally migrated thousands of built images at my company from old-style individual tags to the modern style combined manifest please don't. Combined manifests and automatic arch selection made so many things easier to deal with.

[andaaron]

How would zot garbage collect logic know a specific index is supposed to have all references (the one pushed via rest API) while another is allowed have missing references (the synced one)?
What if someone decided to push an image which was previously synced (should zot start enforcing the presence of the blobs), or sync an image which was previously pushed (should zot stop enforcing the check for the presence of the blobs)?

[Cellivar]

How would zot garbage collect logic know a specific index is supposed to have all references (the one pushed via rest API) while another is allowed have missing references (the synced one)?

I'm of course unfamiliar with exactly how the project's logic works, so I'm operating off of the docs. From my understanding the GC checks to see if a particular object is reachable from a non-deleted manifest. We're talking about the opposite here, a manifest that references not-downloaded objects. Would it need modification at all?

More interesting might be the scrubbing and the retention policy system. If scrubbing cares that a manifest has not-downloaded-yet references it might need to be updated to be more clever.

Retention policies would be nice to extend into this, "if mips arch has zero downloads in the last month clean it up, then re-download when it's been asked for again" type of rules. For my use cases it'd be more important to never download mips in the first place so I'm less interested in prioritizing making the retention policy more clever.

What if someone decided to push an image which was previously synced (should zot start enforcing the presence of the blobs), or sync an image which was previously pushed (should zot stop enforcing the check for the presence of the blobs)?

I don't have a good setup right now to test doing a naive "docker pull " which will end up with only my AMD64 machine's arch on my machine, followed by a re-tag and a push up to a Zot registry. What happens today if I do that? Do these come in via different APIs and can be segmented? Something like:

• Partial manifest syncing is okay
• Partial manifest pushing is not
• Populating an additional arch from a previously sync'd partial manifest is okay

Presuming this is all happening at manifest and object layers all of those are referenced by immutable hashes. So long as my push isn't creating a new manifest and I'm just populating an existing one we're just populating a cache right?

Again, unfamiliar with the codebase so philosophizing here. Thanks for taking the time!