nerdctl with nydus, issues but not sure in which part of the stack

[KristoferHansson]

Issue

I'm running into an issue with trying out nerdctl with nydus but as mentioned in the header, not sure where in the stack the error lies
locally built image, then converted and finally pushed to locally hosted registry, cleaning up cache, then finally, when pulling from registry, manifest seems missing and/or nerdctl looks for i locally?

Asked a question in cncf slack # nerdctl but realized it was was very inactive

Environment

My evaluation environment is W11 with WSL2, Ubuntu 24.04 distro with 6.6.36.6-microsoft-standard-WSL2 kernel, built EROFS kernel module and loaded.
nerdctl version 2.1.3
nydus-snapshotter. Version: v0.15.2,
containerd github.com/containerd/containerd/v2 v2.1.4 75cb2b7193e4e490e9fbdc236c0e811ccaba3376

First attempt with a stock nydusd-config.fusedev.json worked but did not show any deduplication support in storage what so ever, thinking I might need to modify the config to get the chunk based support enabled

Details

After reconfiguring to get chunk support enabled:
I rebuild the images(5 images in total, base image shared, a static file shared, a unique file per image based on static shared file + a few unique bytes appended) locally from "dockerfiles",
convert the images via: ./nerdctl image convert --nydus --oci localhost:5000/testimg$i:raw localhost:5000/testimg$i:nydus
push to the registry: ./nerdctl push localhost:5000/testimg$i:nydus
clean up local storage in order to make sure i have no local artifacts affecting what get's pulled the first time,
pull from registry:

./nerdctl --snapshotter nydus pull localhost:5000/testimg1:nydus
FATA[0000] rpc error: code = NotFound desc = blob sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42
expected at /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42: blob not found: not found

registry output: 127.0.0.1 - - [29/Aug/2025:15:10:53 +0000] "HEAD /v2/testimg1/manifests/nydus HTTP/1.1" 200 1175 "" "containerd/2.1.3+unknown"

time="2025-08-29T15:10:53.428262349Z" level=info msg="response completed" go.version=go1.20.8
http.request.host="localhost:5000" http.request.id=3bc385a3-ea08-486b-b46a-6156e16d9edf http.request.method=HEAD
http.request.remoteaddr="127.0.0.1:48128" http.request.uri="/v2/testimg1/manifests/nydus"
http.request.useragent="containerd/2.1.3+unknown" http.response.contenttype="application/vnd.oci.image.manifest.v1+json"
http.response.duration="885.211µs" http.response.status=200 http.response.written=1175

Detailed output from nerdctl push, related to the manifest(digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42") that is later failing to be found during the pull

DEBU[0000] push                                          digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42" mediatype=application/vnd.oci.image.manifest.v1+json size=1175
DEBU[0000] checking and pushing to                       digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42" mediatype=application/vnd.oci.image.manifest.v1+json size=1175 url="http://localhost:5000/v2/testimg1/manifests/nydus"
DEBU[0000] do request                                    digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42" mediatype=application/vnd.oci.image.manifest.v1+json request.header.accept="application/vnd.oci.image.manifest.v1+json, */*" request.header.user-agent=containerd/2.1.3+unknown request.method=HEAD size=1175 url="http://localhost:5000/v2/testimg1/manifests/nydus"
DEBU[0000] fetch response received                       digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42" mediatype=application/vnd.oci.image.manifest.v1+json response.header.content-length=1175 response.header.content-type=application/vnd.oci.image.manifest.v1+json response.header.date="Fri, 29 Aug 2025 15:00:03 GMT" response.header.docker-content-digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=1175 url="http://localhost:5000/v2/testimg1/manifests/nydus"
manifest-raw@sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42: done

Detailed output from nerdctl when pulling, the missing blob(digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42")

sudo ./nerdctl --snapshotter nydus pull localhost:5000/testimg1:nydus --debug
DEBU[0000] verifying process skipped
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "nydus".
DEBU[0000] fetching                                      image="localhost:5000/testimg1:nydus"
DEBU[0000] resolving                                     host="localhost:5000"
DEBU[0000] do request                                    host="localhost:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/2.1.3+unknown request.method=HEAD url="http://localhost:5000/v2/testimg1/manifests/nydus"
DEBU[0000] fetch response received                       host="localhost:5000" response.header.content-length=1175 response.header.content-type=application/vnd.oci.image.manifest.v1+json response.header.date="Fri, 29 Aug 2025 15:52:54 GMT" response.header.docker-content-digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42\"" response.header.x-content-type-options=nosniff response.status="200 OK" url="http://localhost:5000/v2/testimg1/manifests/nydus"
DEBU[0000] resolved                                      desc.digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42" host="localhost:5000"
DEBU[0000] fetch                                         digest="sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42" mediatype=application/vnd.oci.image.manifest.v1+json size=1175
FATA[0000] rpc error: code = NotFound desc = blob sha256:419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42 expected at /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/419d9156f77c2e44482040cfa21ac2d6a2ca2f92dc7fc818149be06a07aefa42: blob not found: not found 

Additional tests

I will test utilizing nydusify to convert the image and see if that makes any difference,

[KristoferHansson]

According to a response to my question in the nydus slack: nerdctl does not support pushing nydus converted images?

So instead I utilized nydusify to pull the "normal" image, convert and push the converted with a different tag (:nydus)
Everything seems to look fine with the nydusify pull convert push command and there is obviously contents related to the push available in the registry, but the nerdctl pull command fails:

blob sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07 is not found

last part of nydusify command output:

DEBU[2025-09-02T21:49:58+02:00] push                                          digest="sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1830
DEBU[2025-09-02T21:49:58+02:00] checking and pushing to                       digest="sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1830 url="http://localhost:5000/v2/testimg1/manifests/nydus"
DEBU[2025-09-02T21:49:58+02:00] do request                                    digest="sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/2.1.0+unknown request.method=HEAD size=1830 url="http://localhost:5000/v2/testimg1/manifests/nydus"
DEBU[2025-09-02T21:49:58+02:00] fetch response received                       digest="sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=1830 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Tue, 02 Sep 2025 19:49:58 GMT" response.header.docker-content-digest="sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=1830 url="http://localhost:5000/v2/testimg1/manifests/nydus"
INFO[2025-09-02T21:49:58+02:00] pushed image localhost:5000/testimg1:nydus, elapse 22.537035ms  module=converter

PX3:~/tmp/nerdctl$ sudo ./nerdctl --snapshotter nydus pull localhost:5000/testimg1:nydus --debug
DEBU[0000] verifying process skipped
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "nydus".
DEBU[0000] fetching                                      image="localhost:5000/testimg1:nydus"
DEBU[0000] resolving                                     host="localhost:5000"
DEBU[0000] do request                                    host="localhost:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/2.1.3+unknown request.method=HEAD url="http://localhost:5000/v2/testimg1/manifests/nydus"
DEBU[0000] fetch response received                       host="localhost:5000" response.header.content-length=1830 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Tue, 02 Sep 2025 19:59:04 GMT" response.header.docker-content-digest="sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07\"" response.header.x-content-type-options=nosniff response.status="200 OK" url="http://localhost:5000/v2/testimg1/manifests/nydus"
DEBU[0000] resolved                                      desc.digest="sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07" host="localhost:5000"
DEBU[0000] fetch                                         digest="sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1830
FATA[0000] rpc error: code = NotFound desc = blob sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07 expected at /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07: blob not found: not found

PX3:~/tmp/nerdctl$ no_proxy=localhost curl -s http://localhost:5000/v2/testimg1/manifests/nydus | jq .
{
  "schemaVersion": 1,
  "name": "testimg1",
  "tag": "nydus",
  "architecture": "amd64",
  "fsLayers": [
    {
      "blobSum": "sha256:f3d54fb4a86cda4fb6eb7864683830156d25e0be74ca8feee2175323c6ad5122"
    },
    {
      "blobSum": "sha256:809abd7a52aca74ff2ac4b297a9396d2a85feffaea2e30f38a74695da858a6e8"
    },
    {
      "blobSum": "sha256:cd944a11efae876b5d937e09c8d4542851ddff8726fae4db2084dcc3fa605395"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:7c857b86ffc9da4b9f11cfddf4d85d88bf3ebb3f16e49cecf6ff8779f3cc5611"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    }
  ],
  "history": [
    {
      "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/bash\"],\"Labels\":{\"org.opencontainers.image.ref.name\":\"ubuntu\",\"org.opencontainers.image.version\":\"22.04\"}},\"created\":\"2025-09-02T20:52:38.877536498+02:00\",\"id\":\"2dab2c537e8f8afc449b3a85503c141be81f6a083fdfd08003d89f3e689ab136\",\"os\":\"linux\",\"parent\":\"9555f88e702b6b201e8cb670005ca6821aef2f9458a88520e45c385a64a8db01\"}"
    },
    {
      "v1Compatibility": "{\"id\":\"9555f88e702b6b201e8cb670005ca6821aef2f9458a88520e45c385a64a8db01\",\"parent\":\"ef7ef2bee88b39dbe2c6376a130474800b93d73049b11bf36a6d792eafb846a5\",\"comment\":\"buildkit.dockerfile.v0\",\"created\":\"2025-09-02T20:52:38.877536498+02:00\",\"container_config\":{\"Cmd\":[\"COPY partial_shared_file1.bin /app/partial_shared_file.bin # buildkit\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"ef7ef2bee88b39dbe2c6376a130474800b93d73049b11bf36a6d792eafb846a5\",\"parent\":\"a6bd7d5a15efb18ea15317b12b4003de4429f3fa89d7a25ad29f527885e18320\",\"comment\":\"buildkit.dockerfile.v0\",\"created\":\"2025-09-02T20:52:38.745705927+02:00\",\"container_config\":{\"Cmd\":[\"COPY shared_full.bin /app/shared_full.bin # buildkit\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"a6bd7d5a15efb18ea15317b12b4003de4429f3fa89d7a25ad29f527885e18320\",\"parent\":\"1fe04826c5205c4ce265d5cbe5f9767a767cfdb198d886f6340ceca341331a1f\",\"created\":\"2025-08-19T17:17:10.793759467Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  CMD [\\\"/bin/bash\\\"]\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"1fe04826c5205c4ce265d5cbe5f9767a767cfdb198d886f6340ceca341331a1f\",\"parent\":\"314371bc38ca2cbdc6e4f6c9ecf2a4de7aeaf31c6d71a45872671a5063fb1b5f\",\"created\":\"2025-08-19T17:17:10.427314207Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:9303cc1f788d2a9a8f909b154339f7c637b2a53c75c0e7f3da62eb1fefe371b1 in / \"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"314371bc38ca2cbdc6e4f6c9ecf2a4de7aeaf31c6d71a45872671a5063fb1b5f\",\"parent\":\"b14a7346a5c3b89b4886c1d8576cbcbd73d2b85ae2e344e71602eec95c3f6682\",\"created\":\"2025-08-19T17:17:08.19230591Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  LABEL org.opencontainers.image.version=22.04\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"b14a7346a5c3b89b4886c1d8576cbcbd73d2b85ae2e344e71602eec95c3f6682\",\"parent\":\"8e9880e2f2f433621c34c94d346eecaf8e8e500e3e55f52a6c322d2f747ae137\",\"created\":\"2025-08-19T17:17:08.162798176Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  LABEL org.opencontainers.image.ref.name=ubuntu\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"8e9880e2f2f433621c34c94d346eecaf8e8e500e3e55f52a6c322d2f747ae137\",\"parent\":\"3690474eb5b4b26fdfbd89c6e159e8cc376ca76ef48032a30fa6aafd56337880\",\"created\":\"2025-08-19T17:17:08.131655844Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"3690474eb5b4b26fdfbd89c6e159e8cc376ca76ef48032a30fa6aafd56337880\",\"created\":\"2025-08-19T17:17:08.102078738Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ARG RELEASE\"]},\"throwaway\":true}"
    }
  ],
  "signatures": [
    {
      "header": {
        "jwk": {
          "crv": "P-256",
          "kid": "CIYQ:Y7IB:77QP:UQT4:HQGW:3JGP:52P4:O254:QBMN:CSS7:FLQ7:X7TK",
          "kty": "EC",
          "x": "EOfFG2jCSVFDV6LGJNXCCr3dNroOv_V4J8HaXUM-kws",
          "y": "JzOjgXPZuYaVoIBaMfG2VNRefBSOdvmyEq0l-fW1cb4"
        },
        "alg": "ES256"
      },
      "signature": "SysqF-jN8JQuH0XJGWRJOC8d6R84wlerp_QjoAEzB70EBF7esEydjKfU0rWkxAATN6dTFAbQtlWEqxtIkhrv-Q",
      "protected": "eyJmb3JtYXRMZW5ndGgiOjQ1NDUsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAyNS0wOS0wMlQyMDoxMTozOVoifQ"
    }
  ]
}

[KristoferHansson]

Tried a lot, switched registry:v2 to zot, switched nerdctl to ctr for the pull part but same issue
Obviously something making it look for the blob locally was left even thou I pruned images via nerdctl system prune -all and nerdctl container prune and nerdctl image prune and nerdctl --snapshotter nydus image prune and nerdctl --snapshotter nydus container prune as well as removing everything from /var/lib/containerd/io.containerd.content.v1.content and /var/lib/containerd/io.containerd.snapshotter.v1.nydus folders

localhost:5000/testimg1 nydusss:nydus           fetching image content
└──manifest (8bfcb80d0ad2)                      already exists
application/vnd.docker.distribution.manifest.v2+json sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07
Pulling from OCI Registry (localhost:5000/testimg1-nydusss:nydus)       elapsed: 0.0 s  total:  1.8 Ki  (46.3 KiB/s)

Removed the /var/lib/containerd/metadata.db and that finally helped.

localhost:5000/testimg1 nydusss:nydus           saved
└──manifest (8bfcb80d0ad2)                      complete        |++++++++++++++++++++++++++++++++++++++|
   ├──config (029b6df1b195)                     complete        |++++++++++++++++++++++++++++++++++++++|
   └──layer (f3d54fb4a86c)                      complete        |++++++++++++++++++++++++++++++++++++++|
application/vnd.docker.distribution.manifest.v2+json sha256:8bfcb80d0ad27f84d1a7c6afc46ebf19e28f25ec47db8c01a0ae2c4651ed2b07
Completed pull from OCI Registry (localhost:5000/testimg1-nydusss:nydus)        elapsed: 0.4 s  total:  219.0   (516.9 KiB/s)

Will return to evaluating the chunk based deduplication as the next step

[KristoferHansson]

Was able to pull all 5 images using both nerdctrl and ctr but running them fails with both nerdctl and ctr
not finding sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281

last part of nerdctl output from pull:

DEBU[0000] image unpacked                                chainID="sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281" config="sha256:b036bb63c51c0e9d9d4b6657032aa00229e67c487a2a4432800dbc76284f6b4c" duration=45.989569ms
localhost:5000/testimg2-nydusss:nydus:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:d497afca19a226a7082049dc113ec8485e509e111c420a4d67d5741d09eb4194: exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:b036bb63c51c0e9d9d4b6657032aa00229e67c487a2a4432800dbc76284f6b4c:   exists         |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.2 s                                                                    total:   0.0 B (0.0 B/s)

PX3:~/tmp/nerdctl$ sudo ./nerdctl --snapshotter nydus run localhost:5000/testimg2-nydusss:nydus --insecure-registry --debug FATA[0000] mount rafs, instance id 8: failed to find image ref of snapshot 8, labels map[containerd.io/snapshot.ref:sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281 containerd.io/snapshot/nydus-bootstrap:true containerd.io/snapshot/nydus-fs-version:6]

PX3:~/tmp/nerdctl$ sudo ctr --address /run/containerd/containerd.sock run --rm --snapshotter nydus --platform linux/amd64 --net-host localhost:5000/testimg2-nydusss:nydus test1 sh -c "find / > /dev/null"
ctr: failed to find image ref of snapshot 8, labels map[containerd.io/snapshot.ref:sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281 containerd.io/snapshot/nydus-bootstrap:true containerd.io/snapshot/nydus-fs-version:6]

containerd debug output:

DEBU[2025-09-05T13:08:29.848679315+02:00] stat snapshot                                 key="sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281" snapshotter=nydus
DEBU[2025-09-05T13:08:29.871188474+02:00] prepare snapshot                              key=test1 parent="sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281" snapshotter=nydus
TRAC[2025-09-05T13:08:30.076747913+02:00] garbage collected                             d=7.428098ms

snapshotter debug output:

INFO[2025-09-05T13:08:16.522693335+02:00] [View] snapshot with key default/34//tmp/initialC1381421169 parent default/10/sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281
INFO[2025-09-05T13:08:29.877732023+02:00] [Prepare] snapshot with key default/35/test1 parent default/10/sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281
DEBU[2025-09-05T13:08:29.881797812+02:00] [Prepare] snapshot with labels map[]          key=default/35/test1 parent="default/10/sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281"
DEBU[2025-09-05T13:08:29.881949327+02:00] isProxyDriver = false, isProxyLabel = false, isProxyImage = false
INFO[2025-09-05T13:08:29.882065476+02:00] Prepare active Nydus snapshot default/35/test1  key=default/35/test1 parent="default/10/sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281"
DEBU[2025-09-05T13:08:29.882106116+02:00] Prepare remote snapshot 8                     key=default/35/test1 parent="default/10/sha256:c6fb272a0dbe4077b6f8b9347da81b227fdb73e26b976bff7d8085dfada18281"