Support generic registries

.gitignore

@@ -15,3 +15,7 @@ install
 
 build
 pyenv
+
+# npm artifacts installed by tools
+package*.json
+node_modules

CLAUDE.md

@@ -0,0 +1,110 @@
+# Registry Abstraction Implementation Status
+
+## Context
+Refactoring hardcoded site-specific JFrog registry to support multiple registry types (OCI, Zot, GHCR, site-specific). Goal is to allow users to configure different registries while maintaining backward compatibility.
+
+## Key Understanding
+- **Repository**: Local disk storage (SQLite + subdirectories) for downloaded images
+- **Registry**: Remote OCI container registry (JFrog, GHCR, Docker Hub, etc.)
+- **Repository data structure**: Also used as in-memory container for registry search results (consistent query interface)
+
+## Completed Work ✅
+
+### 1. Core Abstraction (`src/uenv/registry.h` + `src/uenv/registry.cpp`)
+- Created `registry_backend` abstract base class
+- Implemented `registry_type` enum (oci, zot, ghcr, site)
+- Added factory function `create_registry(url, type)`
+- OCI/Zot/GHCR implementations return empty repositories (no search support)
+- Added `supports_search()` capability flag
+
+### 2. Site Integration (`src/site/site.h` + `src/site/site.cpp`)
+- Added `jfrog_registry` class wrapping existing `registry_listing()` function
+- Added `create_site_registry()` factory function
+- Preserves existing JFrog custom API behavior
+
+### 3. Configuration (`src/uenv/settings.h` + `src/uenv/settings.cpp`)
+- Added `registry_type` field to `config_base` and `configuration`
+- Added parsing for registry_type config parameter
+- Default type is "site" for backward compatibility
+
+### 4. CLI Utilities (`src/cli/util.h` + `src/cli/util.cpp`)
+- Added `create_registry_from_config()` helper function
+- Handles registry backend creation based on configuration
+
+### 5. Build System (`meson.build`)
+- Added `src/uenv/registry.cpp` to lib_src
+
+### 6. Complete CLI Updates ✅
+- Updated `src/cli/copy.cpp` to use new registry abstraction
+- Updated `src/cli/delete.cpp` to use registry abstraction with graceful degradation  
+- Updated `src/cli/pull.cpp` to use registry abstraction with graceful degradation
+- Updated `src/cli/push.cpp` to use registry abstraction with graceful degradation
+- ✅ All CLI files now use the registry abstraction
+- ✅ Code compiles and tests pass
+
+### 7. Type Erasure Refactoring ✅ 
+- Refactored registry from abstract base class to type-erased value class
+- Used concept-based approach similar to `help::item` pattern
+- Registry implementations are now structs that satisfy `RegistryImpl` concept
+- Registry objects can be stored by value, copied, and moved
+- Updated all CLI utilities to use value-based API instead of pointer-based
+- ✅ All code compiles and tests pass
+
+## Remaining Work 🚧
+
+### 2. Implement Graceful Degradation Pattern ✅
+All CLI commands now implement graceful degradation:
+- **delete.cpp**: Requires search support, fails gracefully if not available
+- **pull.cpp**: Constructs minimal record for non-searchable registries
+- **push.cpp**: Skips existence check for non-searchable registries with warning
+- **copy.cpp**: Uses search for validation when available
+
+Pattern implemented:
+```cpp
+auto registry_backend = create_registry_from_config(settings.config);
+if (registry_backend->supports_search()) {
+    // Existing logic: validate via listing first
+    auto listing = registry_backend->get_listing(nspace);
+    // ... validation, disambiguation, etc.
+} else {
+    // Graceful degradation: proceed without validation
+    spdlog::info("Registry does not support search, proceeding without pre-validation");
+    // ... direct ORAS operations with proper error handling
+}
+```
+
+### 3. Enhance ORAS Error Handling
+Ensure ORAS wrapper functions return well-structured errors:
+- "Image not found" errors
+- "Network/registry unreachable" errors  
+- "Authentication failed" errors
+- "Invalid image format" errors
+
+### 4. Update Error Messages
+- With search: "No image found matching 'X' in registry listing"
+- Without search: "Failed to pull image 'X': image not found in registry"
+
+### 5. Testing
+- Test searchable registry (JFrog): Existing behavior preserved
+- Test non-searchable registry: Graceful operation with proper errors
+
+## Configuration Examples
+```
+# Site-specific (current default)
+registry = jfrog.svc.cscs.ch/uenv
+registry_type = site
+
+# GitHub Container Registry
+registry = ghcr.io
+registry_type = ghcr
+
+# Generic OCI registry
+registry = docker.io
+registry_type = oci
+```
+
+## Next Steps
+1. Complete CLI file updates with supports_search() branching
+2. Enhance ORAS error handling
+3. Test with different registry types
+4. Update documentation

meson.build

@@ -46,6 +46,7 @@ lib_src = [
         'src/uenv/oras.cpp',
         'src/uenv/parse.cpp',
         'src/uenv/print.cpp',
+        'src/uenv/registry.cpp',
         'src/uenv/repository.cpp',
         'src/uenv/settings.cpp',
         'src/uenv/uenv.cpp',

oras.md

@@ -0,0 +1,184 @@
+
+The *manifest* for a uenv squashfs is a JSON file that can be obtained using `oras manifest fetch rego/tag`
+
+* there is always the same empty `config` field
+    * `e30=` is an empty JSON object `{}` in base64 encoding
+* there is a single layer: the squashfs file
+* there is a single annotation with the time of creation
+
+```console
+$ oras manifest fetch localhost:5862/deploy/cluster/zen3/app/1.0:v1 | jq .
+{
+  "schemaVersion": 2,
+  "mediaType": "application/vnd.oci.image.manifest.v1+json",
+  "artifactType": "application/x-squashfs",
+  "config": {
+    "mediaType": "application/vnd.oci.empty.v1+json",
+    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
+    "size": 2,
+    "data": "e30="
+  },
+  "layers": [
+    {
+      "mediaType": "application/vnd.oci.image.layer.v1.tar",
+      "digest": "sha256:4f9aa7ee3a5a056c3742119a69b2aa6eafefa46e571ce3f377f67aebdf43c2db",
+      "size": 4096,
+      "annotations": {
+        "org.opencontainers.image.title": "app42.squashfs"
+      }
+    }
+  ],
+  "annotations": {
+    "org.opencontainers.image.created": "2025-06-17T08:32:08Z"
+  }
+}
+
+```
+
+Every file (squashfs, tar ball of meta path, manifest.json, etc) in a registry has a *digest* of the form `sha256:<64 character sha>`.
+
+* the config json has a digest that is `printf '{}' | sha256sum`
+* the squashfs digest is the hash of the squashfs file
+
+The manifest itself is referred to as
+
+We can pull individual files/artifacts, aka blobs:
+
+```console
+$ oras blob fetch --descriptor \
+    localhost:5862/deploy/cluster/zen3/app/1.0@sha256:4f9aa7ee3a5a056c3742119a69b2aa6eafefa46e571ce3f377f67aebdf43c2db
+{
+  "mediaType": "application/octet-stream",
+  "digest": "sha256:4f9aa7ee3a5a056c3742119a69b2aa6eafefa46e571ce3f377f67aebdf43c2db",
+  "size": 4096
+}
+$ oras blob fetch --output=store.squashfs \
+    localhost:5862/deploy/cluster/zen3/app/1.0@sha256:4f9aa7ee3a5a056c3742119a69b2aa6eafefa46e571ce3f377f67aebdf43c2db
+```
+
+Note that `blob fetch --descriptor` returns file size, but not file name (`store.squashfs`)
+* the file name is in `layers[0]['annotations']['org.opencontainers.image.title']`
+
+## downloading squashfs
+
+There are two methods: `oras blob fetch` and `oras pull`
+
+```
+TAG=v1
+DIGEST=sha256:4f9aa7ee3a5a056c3742119a69b2aa6eafefa46e571ce3f377f67aebdf43c2db
+URL=localhost:5862/deploy/cluster/zen3/app/1.0
+
+# blob fetch uses the digest
+oras blob fetch --output=store.squashfs $URL@$DIGEST
+
+# pull uses the digest or tag
+oras pull --output=store.squashfs $URL:$TAG
+oras pull --output=store.squashfs $URL@$DIGEST
+```
+
+## downloading meta path
+
+There is one way: use `oras pull`
+
+```console
+$ TAG=v1
+$ URL=localhost:5862/deploy/cluster/zen3/app/1.0
+
+$ oras discover --format=json --artifact-type=uenv/meta localhost:5862/deploy/cluster/zen3/app/1.0:v1 | jq -r '.manifests[0].digest'
+sha256:d5d9a6eb9eeb83efffe36f197321cd4b621b2189f066dd7a4b7e9a9e6c61df37
+
+$ DIGEST=sha256:d5d9a6eb9eeb83efffe36f197321cd4b621b2189f066dd7a4b7e9a9e6c61df37
+$ oras pull $URL@$DIGEST
+
+# using image blob fetch downloads something, but I don't know how to turn it into the meta path
+oras blob fetch --output meta $URL@$DIGEST
+```
+
+# Abstraction
+
+Requirements:
+
+* squashfs: size, sha,
+* url
+* meta: sha
+
+
+```cpp
+struct manifest {
+    sha256 digest;
+    sha256 squashfs_digest;
+    size_t squashfs_bytes;
+    optional<sha256> meta_digest;
+    std::string respository;
+    std::string tag;
+}
+```
+
+
+Before we used the `uenv-list` service to generate a full database, which we could then search on
+
+wait now, a manifest is
+* a json file
+* a description using OCI format of layers and meta
+
+```
+// complete all information
+// - sha of squashfs, sha of meta, full url
+manifest fetch_manifest(url, record) {
+    tag_url = url + record;
+    mf = oras manifest fetch tag_url
+    meta = oras discover tag_url
+    digest = oras resolve tag_url
+    manifest M {
+        .digest = ??;
+        .squashfs = mf[layers][0][digest]
+        .squashfs_bytes = mf[layers][0][size]
+        .meta = meta[manifests][0].digest;
+        .repository = url + record (no tag/digets);
+    }
+
+}
+
+pull_digest(url, sha, path) {
+
+}
+
+```
+
+uenv image push
+    TODO check for existing image
+    oras push --artifact-type=application/x-squashfs squashfs tag
+    oras attach --artifact-type=uenv/meta tag meta_path
+
+uenv image pull
+    TODO check for existing image
+    oras pull --output=path/store.squashfs manifest.repository@manifest.squashfs_digest
+    oras pull --output=path/store.squashfs manifest.repository@manifest.meta_digest
+
+uenv image delete
+    TODO check for existing image
+    curl -X Delete ...
+
+uenv image copy
+
+# WORKFLOW
+
+If the rego.supports_search():
+    vector<records> registry.match(input_label);
+    if there is one match then set record
+If record is unique
+    pull
+    delete
+    copy
+
+
+```
+struct registry
+    util::expected<uenv::repository, std::string>
+    listing(const std::string& nspace) const;
+    url() const;
+    supports_search() const;
+    type() const;
+    manifest(label)
+```
+

src/cli/copy.cpp

@@ -13,6 +13,7 @@
 #include <uenv/oras.h>
 #include <uenv/parse.h>
 #include <uenv/print.h>
+#include <uenv/registry.h>
 #include <uenv/repository.h>
 #include <util/curl.h>
 #include <util/expected.h>
@@ -22,6 +23,7 @@
 #include "copy.h"
 #include "help.h"
 #include "terminal.h"
+#include "util.h"
 
 namespace uenv {
 
@@ -91,9 +93,15 @@ int image_copy([[maybe_unused]] const image_copy_args& args,
         return 1;
     }
 
-    auto src_registry = site::registry_listing(*src_label.nspace);
+    auto registry_backend = create_registry_from_config(settings.config);
+    if (!registry_backend) {
+        term::error("{}", registry_backend.error());
+        return 1;
+    }
+
+    auto src_registry = registry_backend->listing(*src_label.nspace);
     if (!src_registry) {
-        term::error("unable to get a listing of the uenv",
+        term::error("unable to get a listing of the uenv: {}",
                     src_registry.error());
         return 1;
     }
@@ -154,7 +162,7 @@ int image_copy([[maybe_unused]] const image_copy_args& args,
     spdlog::info("destination record: {} {}", dst_record.sha, dst_record);
 
     // check whether the destination already exists
-    auto dst_registry = site::registry_listing(*dst_label.nspace);
+    auto dst_registry = registry_backend->listing(*dst_label.nspace);
     if (dst_registry && dst_registry->contains(dst_record)) {
         if (!args.force) {
             term::error("the destination already exists - use the --force flag "
@@ -164,7 +172,7 @@ int image_copy([[maybe_unused]] const image_copy_args& args,
         term::error("the destination already exists and will be overwritten");
     }
 
-    const auto rego_url = site::registry_url();
+    const auto rego_url = registry_backend->url();
     spdlog::debug("registry url: {}", rego_url);
     if (auto result =
             oras::copy(rego_url, src_label.nspace.value(), src_record,