Skip to content

Conversation

AminMortezaie
Copy link

Add Namespace and Epoch Support to KV Store

Overview

This PR introduces namespace and epoch support to AutoMQ's KV store system through a composable design pattern, allowing for backward compatibility while providing new functionality for applications that need namespace isolation and epoch-based versioning.

Key Changes

  1. Added a new NamespacedControllerKVClient that extends the base ControllerKVClient
  2. Introduced a feature flag namespacedKVEnabled in Config for opt-in functionality
  3. Modified DefaultS3Client to support both namespaced and non-namespaced modes

Implementation Details

  • Namespace Support: Keys are prefixed with their namespace (e.g., "namespace:key") for isolation
  • Epoch Management: Each namespace maintains its own epoch counter for versioning
  • Backward Compatibility:
    • Default behavior remains unchanged (namespacedKVEnabled = false)
    • Existing applications continue to work without modification
    • New applications can opt-in to namespaced functionality

Usage

To enable namespaced KV operations:

Config config = new Config()
    .namespacedKVEnabled(true);

When enabled, applications can:

  • Use namespace isolation for keys
  • Maintain epoch versioning per namespace
  • Prevent cross-namespace conflicts
  • Maintain data isolation between different parts of the system

Migration Path

  1. Deploy the changes with namespacedKVEnabled = false (default)
  2. Applications can gradually opt-in to namespaced functionality
  3. No immediate action required for existing applications

Performance Impact

  • Minimal overhead for non-namespaced operations (no changes to existing path)
  • Small additional overhead for namespaced operations:
    • Key prefix addition
    • Epoch checks
    • Additional epoch storage per namespace

Future Considerations

  1. Namespace cleanup utilities
  2. Epoch management tools
  3. Namespace statistics and monitoring
  4. Cross-namespace operations (if needed)

Related Issue

@CLAassistant
Copy link

CLAassistant commented May 20, 2025

CLA assistant check
All committers have signed the CLA.

@superhx
Copy link
Collaborator

superhx commented May 22, 2025

@AminMortezaie We sincerely appreciate your contributions. However, it appears that there might be a misunderstanding regarding the requirement. Instead of manipulating the key field, what we actually need is to extend the KVRecord by adding two new fields: namespace and epoch.

  • With namespace field, we can fast filter the KVs in certain namespace instead of matching them by prefix.

@AminMortezaie
Copy link
Author

Great, I will change to the real requirements.

Copy link

This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch)

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

@github-actions github-actions bot added the Stale label Aug 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants