[Feature] Transformation of Metadata Registry from etcd to Property-backed Storage

[hanahmily]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Description

Context & Motivation

Currently, BanyanDB relies on etcd for managing cluster metadata (schema, node registry, shard allocation). While etcd provides strong consistency, it introduces significant operational complexity, prevents efficient scaling on edge/standalone deployments, and forces the maintenance of heterogeneous storage engines.

This issue tracks the strategic initiative to replace the etcd-based registry with a Property-backed Metadata Service. This will unify the storage layer, allowing BanyanDB to manage its own control plane using its native Property data model and inverted index.

Architecture Changes

1. Storage Layer: Replace the etcd/clientv3 dependency with a PropertyStorageAdapter that maps metadata objects to Property documents (e.g., mapping Group, Name, ID structure).
2. Notification Mechanism: Transition from etcd "Watch" streams to a Periodic Scanning model. A background scanner will poll for changes based on updated_at timestamps to refresh the local In-Memory Cache.

Task List

• Core Storage

  • Define reserved Property groups (e.g., _system, _schema).
  • Implement PropertyRegstry in "metadata" pkg

• Notification & Caching

  • Develop the Periodic Scanner to poll for metadata changes using updated_at filters,.
  • Build the In-Memory State Cache to serve read requests without disk I/O.
  • Remove user side cache to reduce memory overhead(Optional, we might move it to an independent task)

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct