v1.0.0 release tracking issue - Migration with a Golang Backend

[eznix86]

This issue is here to track and detail the architecture.

TO BE BROKEN DOWN AS INDIVIDUAL ISSUES:

• It will provide customization (background, name, logo, colors, theming?, dark and light mode, but dark mode as default)
• Remove the need to depend on the frontend (when changing browser which is: to re-crawl everything)
• Ensure that it remains very lightweight, this is the most important feature of this project, keep it under 50MB of RAM if possible, at worse 100MB, with 0.5 core.
• Always use Differential Sync (every 10 sec)
• Add Lazy incremental Sync (when opening a repository)
• Extend to other databases, like Postgres/MariaDB (if the community needs it), for now it is just SQLite.
• Use an external service to do vulnerability check and display the information (Trivy is one)
• Support AWS ECR, Google Artifact Registry, Azure Container Registry, Docker Hub, Github Container Registry, Zot Registry, Gitea
• Make a slick UI so it is enjoyable to use, with micro-interactions (We will use react, so it is pretty easy to find).
• Make an official documentation, maybe Github pages, a Mintlify or equivalent.
• Not necessary -- A user management, passkeys, login with OAuth, username and password (some people asked for it)

Architecture and Implementation Plan for Indexing and Synchronization

1. Overview

This indexes multiple OCI/Docker registries into a local SQLite database and exposes an InertiaJS Frontend to:

• Explore repositories with deduplicated image sizes and architecture lists.
• View detailed repository data including tags, manifests, and metadata.
• Continuously synchronize with remote registries to reflect new, modified, or deleted content.
• Process metadata lazily and efficiently using a crawl queue with bounded concurrency.

The registry ui is designed for:

• Fewer than 50 repositories.
• Repositories containing 100–200+ tags each.
• Frequent updates (from CI/CD retagging for ex.)

What should it do ?

• Differential sync for efficient change detection.
• Crawl queue with task deduplication.
• SQLite with foreign key cascades for cleanup.
• Registry Health checks and backoff for registry and task.
• Clear crawl progress reporting and partial InertiaJS responses.

---

2. Data Model

PRAGMA foreign_keys = ON;

CREATE TABLE registries (
    id INTEGER PRIMARY KEY,
    host TEXT UNIQUE NOT NULL,
    last_status INTEGER,
    last_checked_at TEXT,
    next_retry_at TEXT
);

CREATE TABLE repositories (
    id INTEGER PRIMARY KEY,
    registry_id INTEGER NOT NULL REFERENCES registries(id) ON DELETE CASCADE,
    name TEXT NOT NULL,
    last_synced_at TEXT,
    UNIQUE(registry_id, name)
);

CREATE TABLE tags (
    id INTEGER PRIMARY KEY,
    repo_id INTEGER NOT NULL REFERENCES repositories(id) ON DELETE CASCADE,
    name TEXT NOT NULL,
    digest TEXT NOT NULL,
    last_synced_at TEXT,
    UNIQUE(repo_id, name)
);

CREATE TABLE manifests (
    digest TEXT PRIMARY KEY,
    media_type TEXT NOT NULL,
    os TEXT,
    architecture TEXT,
    created TEXT,
    size_bytes INTEGER DEFAULT 0
);

CREATE TABLE manifest_layers (
    manifest_digest TEXT NOT NULL REFERENCES manifests(digest) ON DELETE CASCADE,
    layer_digest TEXT NOT NULL,
    size_bytes INTEGER NOT NULL,
    PRIMARY KEY (manifest_digest, layer_digest)
);

CREATE TABLE crawl_queue (
    id INTEGER PRIMARY KEY,
    repo_id INTEGER NOT NULL,
    digest TEXT,
    tag TEXT,
    kind TEXT NOT NULL,  -- 'tags', 'index', 'image'
    status TEXT NOT NULL DEFAULT 'pending',
    priority INTEGER DEFAULT 0,
    next_retry_at TEXT,
    retry_count INTEGER DEFAULT 0,
    UNIQUE(repo_id, digest, kind)
);

CREATE TABLE crawl_progress (
    repo_id INTEGER PRIMARY KEY,
    tags_total INTEGER DEFAULT 0,
    tags_done INTEGER DEFAULT 0,
    images_total INTEGER DEFAULT 0,
    images_done INTEGER DEFAULT 0
);

---

3. Differential Sync

A periodic sync loop (every 30 seconds) reconciles remote registry state with the local database.

3.1 Repository Sync

• Fetch /v2/_catalog.
• Compare remote repository list with repositories table.
• Insert new repositories and delete missing ones.
• Deletions cascade down to tags, manifests, and layers automatically.

---

3.2 Tag Sync

This is the most important part of the sync loop because each repository can have 100–200+ tags (got this from DMs).

• For each repository selected in this sync cycle, fetch /v2/<repo>/tags/list.

• Compare remote tags with tags table.

• Identify:

  • Added tags → insert into DB, enqueue crawl tasks.
  • Deleted tags → remove from DB, cascades cleanup.
  • Changed tags (digest mutation) → update digest and enqueue crawl task.

Tag list fetches dominate network usage, this will need to be tracked !

---

3.3 Manifest Garbage Collection

Periodically remove unreferenced manifests and layers (This is because of the design of registries with reused layers):

DELETE FROM manifests
WHERE digest NOT IN (SELECT DISTINCT digest FROM tags);

3.3.1 Deletion Cascade and API Cleanup

A DELETE API is provided for tags removal:

• Triggers deletes on tags, manifests, and layers.
• Sends deletion calls to the upstream registry API for cleanup.

---

3.4 Tag Sync Scheduling (Staggering - Not sure how we would be doing that but here is a breakdown)

To avoid spikes in network and processing load:

• Repositories are divided into small groups (e.g., 5 groups of 10).
• Each sync interval (every 30 seconds) processes one group’s tags.
• Over several intervals (e.g 2–3 minutes), all repositories are fully tag-synced.
• High-churn repositories can be placed in smaller or more frequent groups.

This approach smooths out the ~10,000 tag checks across time:

50 repos × 200 tags = 10,000 tags total
→ 10 repos per tick × 200 tags = 2,000 tags per sync interval

---

4. Crawl Queue and Workers

4.1 Queue Model

The crawl queue holds discovered work from tag sync.
UNIQUE constraints on (repo_id, digest, kind) ensure deduplication so the same manifest isn’t processed multiple times even if many tags point to it.

4.2 Worker Pool

• Global bounded concurrency limits total simultaneous HTTP operations.
• Per-registry concurrency prevents overloading individual registries.
• Workers continuously poll the queue, respecting retry delays and priority.

SELECT * FROM crawl_queue
WHERE status = 'pending'
  AND (next_retry_at IS NULL OR next_retry_at <= CURRENT_TIMESTAMP)
ORDER BY priority DESC, id ASC -- this is needed when for example loading a repository, this bumps it to first in the queue
LIMIT 1;

4.3 Crawl Steps

• Tag task: resolve digest, enqueue index or image tasks.
• Index task: fetch manifest list, enqueue image tasks per platform.
• Image task: fetch manifest and config blob, store layer sizes, architecture, OS, and creation time.

Task deduplication keeps queue size proportional to unique digests, not raw tag counts.

---

5. Registry Health and Backoff

5.1 Health Checks

• Periodically ping (GET /v2/) each registry, or other endpoints for other registries.
• Store last HTTP status and next retry timestamp.
• Skip unhealthy registries during sync and crawl until backoff expires.

UPDATE registries
SET last_status = :status,
    last_checked_at = CURRENT_TIMESTAMP,
    next_retry_at = CASE
        WHEN :status = 200 THEN NULL
        ELSE DATETIME(CURRENT_TIMESTAMP, '+' || :backoff || ' seconds')
    END
WHERE id = :id;

5.2 Task Backoff

• On task failure, increment retry_count and set next_retry_at with exponential backoff.
• Successful completion resets these fields.

UPDATE crawl_queue
SET retry_count = retry_count + 1,
    next_retry_at = DATETIME(CURRENT_TIMESTAMP, '+' || (30 * POW(2, retry_count)) || ' seconds')
WHERE id = :task_id;

This prevents retry storms when registries or specific manifests are temporarily unavailable.

---

6. InertiaJS Behavior

6.1 Explore Endpoint

• Returns repository list with deduplicated size and architecture list.
• Size is computed by summing unique layers across all manifests of the repo:

SELECT SUM(size_bytes) FROM (
  SELECT DISTINCT layer_digest, size_bytes
  FROM manifest_layers ml
  JOIN tags t ON ml.manifest_digest = t.digest
  WHERE t.repo_id = :repo_id
);

• Crawl state (not_started, queued, partial, complete) is included for each repository.
• Given the small repository count, this query can run across all repos efficiently.
• Global state (queued, partial, complete)

---

6.2 Repository Detail Endpoint

• Returns available metadata immediately without waiting for crawling to complete.
• Includes crawl progress from crawl_progress.
• If crawl is queued or partial, the response indicates this clearly.
• Can optionally bump crawl priority for this repository to process it sooner.

For repositories with many tags, the endpoint may:

• Return summarized data (e.g., grouped by digest) by default.
• Provide pagination or lazy loading for full tag lists if required.

Example response:

{
  "summary": {...},
  "tags": {...},
  "crawl": {
    "state": "partial",
    "tags_done": 45,
    "tags_total": 200,
    "images_done": 30,
    "images_total": 60
  }
}

7. InertiaJS Response Sample

7.1 Explore

[
  {
    "name": "library/nginx",
    "registry": "registry.example.com:5000",
    "size": 823475200,
    "architectures": ["amd64", "arm64"],
    "crawl_state": "complete",
    "last_synced_at": "2025-10-02T10:15:42Z"
  },
  {
    "name": "library/postgres",
    "registry": "registry.example.com:5000",
    "size": 452190560,
    "architectures": ["amd64"],
    "crawl_state": "partial",
    "last_synced_at": "2025-10-02T10:16:10Z"
  },
  {
    "name": "custom/api-server",
    "registry": "registry.example2.com",
    "size": 0,
    "architectures": [],
    "crawl_state": "queued",
    "last_synced_at": null
  },
  {
    "name": "eznix86/api-frontend",
    "registry": "registry.example2.com",
    "size": 0,
    "architectures": [],
    "crawl_state": "completed", // it means it is an untagged repository as it is completed but no tags/zero size.
    "last_synced_at": "2025-10-02T10:16:10Z"
  }
]

7.2 Repository Detail

The summary and the tags can be split into two deferred calls (TBD)

{
  "summary": {
    "name": "library/nginx",
    "registry": "registry.example.com",
    "size": 823475200,
    "architectures": ["amd64", "arm64"],
    "last_synced_at": "2025-10-02T10:15:42Z",
  },
  "crawl": {
    "state": "partial",
    "tags_total": 200,
    "tags_done": 180,
    "images_total": 65,
    "images_done": 55
  },
  "tags": [
    {
      "name": "latest",
      "digest": "sha256:1234abcd...",
      "aliases": ["1.21", "stable"],
      "images": [
        {
          "digest": "sha256:a1b2c3d4...",
          "os": "linux",
          "architecture": "amd64",
          "size": 154300000,
          "created": "2025-09-29T08:23:14Z",
          "layers": [
            {"digest": "sha256:layer1...", "size": 40000000},
            {"digest": "sha256:layer2...", "size": 60000000},
            {"digest": "sha256:layer3...", "size": 54300000}
          ]
        },
        {
          "digest": "sha256:f9e8d7c6...",
          "os": "linux",
          "architecture": "arm64",
          "size": 149200000,
          "created": "2025-09-29T08:22:10Z",
          "layers": [
            {"digest": "sha256:layer1...", "size": 38000000},
            {"digest": "sha256:layer2...", "size": 60000000},
            {"digest": "sha256:layer3...", "size": 51200000}
          ]
        }
      ]
    },
    {
      "name": "1.20.2",
      "digest": "sha256:5678efgh...",
      "aliases": [],
      "images": [
        {
          "digest": "sha256:z9y8x7w6...",
          "os": "linux",
          "architecture": "amd64",
          "size": 153800000,
          "created": "2025-08-15T11:45:00Z",
          "layers": [
            {"digest": "sha256:layer1...", "size": 40000000},
            {"digest": "sha256:layer2...", "size": 59000000},
            {"digest": "sha256:layer3...", "size": 54800000}
          ]
        }
      ]
    }
  ]
}