[RFC]: Mooncake KV-Store Indexer API Standardization

[yejj710]

Motivation

Currently, we are planning to implement an indexer on Mooncake to track the global status of KVCache (including VRAM, DRAM, SSD, etc.), which can assist the gateway in achieving precise cache-aware request scheduling.

Goals

1.Use KVEvents API as a Public Contract

​ The KVEvents schema is clearly defined in both vLLM and SGLang. Therefore, we will use KVEvents as the data structure for indexer updates.

2.Support different storage media
​Enable KV-cache management across multiple tiers, including:

• G1: Inference engine layer (VRAM/HBM/..)
• G2: CPU offload / pooling DRAM storage/..
• G3: SSD offload / 3FS/DFS/NFS/..

3.DP rank Awareness

​ Track KV-cache status across different DP ranks to assist the gateway in achieving global cache-aware DPLB

4.Support querying by Token ID or hash key

​ Long sequences context impose high network overhead if token IDs are directly transmitted, so we should provide an API supporting hash keys as query input.

Proposed Change

/query

POST api

// input
{
    "model": "deepseek",
    "lora_name": "xx-adapter",
    "lora_id": 12, // defined for backward compatibility and should not be used together with `lora_name`
    "token_ids": [1, 15, 100],
    "tenant_id":  99, // used for multi-tenant scenario
    "cache_salt":  334455,  // ensure cached data blocks are kept separate for different customers
}

// output
{
    "tenant_id": {
        "api_server_unique_name": {
            "longest_matched": 100, // the number of longest prefix matched token among multiple DPs(if there are)
            "GPU": 20, // device pool cache, managered by inference engine (vLLM/SGLang, etc.)
            "DP": {
                0: 10,
                1: 20
            },
            // G2 cache, p2p/mooncake-master store (DRAM)
            "CPU": 60,
            // G3 cache, p2p/mooncake-master store (SSD, 3fs, nfs, dfs, etc.)
            "DISK": 10 // disk-pool space matched prefix token number
        },
        ... // other engine instance
    },
    ... // other tenant
}

/query_by_hash

POST api

// input
{
    "model": "deepseek",
    "lora_name": "xx-adapter",
    "lora_id": 12, // defined for backward compatibility and should not be used together with `lora_name`
    "block_hash": ["hash_key_by_chunked_tokens"]
    "tenant_id":  99, 
    "cache_salt":  334455, 
}
// output
{
    "tenant_id": {
        "api_server_unique_name": {
            // Each model service uses its own independent block_size, token_num = block_size * matched_hash_key
            "longest_matched": 100, 
            "GPU": 20, 
            "DP": {
                0: 10,
                1: 20
            },
            "CPU": 60,
            "DISK": 10
        },
        ... // other engine instance
    },
    ... // other tenant
}

CC List: @stmatengss @doujiang24 @Asher-XunZhang

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

[doujiang24]

medium in input seems unnecessary? please show more details about the usage of the field.
keep simple as start could be better.

[yejj710]

medium in input seems unnecessary? please show more details about the usage of the field. keep simple as start could be better.

Thanks for comment~ medium is an optional arg. I've submitted a pr that includes more detailed explanations.

[PeaBrane]

@yejj710 several comments / questions

1. Should we also standardize here the hashing algo? I think each engine framework hashes their KV events differently, so we may also want to discuss a deterministic hashing algo here (sha256) as the universal contract.

2. If passing by raw tokens, where would the hashing happen, is it inside the KV indexer service? This may make the indexer service itself not super scalable. I wonder if we should just not support raw tokens as input altogether.

3. What is the mechanism here for each engine registering their engine_name to the KvIndexer service? Would there be any dynamic discovery and heartbeat mechanism here?

4. I think we should just return matched_blocks instead of matched_tokens and return block_size as well explicitly. Also, matched_tokens here is a bit ambiguous; is it the max matched tokens across all dp instances and memory tiers?

5. Should we have multi-tenant isolation, e.g. another field such as namespace or tenant_id. It also prevents a super large output, if the user is only interested in a subset of instances.

[doujiang24]

Hi @PeaBrane

4. I think we should just return matched_blocks instead of matched_tokens and return block_size as well explicitly. Also, matched_tokens here is a bit ambiguous; is it the max matched tokens across all dp instances and memory tiers?

yeah, matched_tokens is confused, longest_matched could be better, as discussed in:
https://github.com/kvcache-ai/Mooncake/pull/1416/changes#diff-43ffc2328e105132991240dc43eb6e4186b327788430b48bf55e1b0b3d804802R61

the number of longest prefix matched token among all mediums.

Is there some benifits that router aware of the block_size? I haven't figured it out.
If there's no benefit, maybe tokens would be better?

[PeaBrane]

@doujiang24 I think it's just for better observability, not strictly needed. The other points may be more important.

On an unrelated note, that may also be a need for a extra_id defaulting to None (or generally an additional optional u64) that translates to a salt hash in the indexer. That would be potentially beneficial for, say, a promise to keep hash lookups of different "customers" separate.

[yejj710]

@yejj710 several comments / questions

@PeaBrane Thank you for your comments. Here are some of my thoughts:

q1: Standardizing hashing algo brings many benefits, such as reducing the network overhead caused by passing token ids. The challenge lies in promoting different communities to adopt this practice. This involves: 1. Initializing root_hash in the inference framework, 2. Hashing algorithms, and 3. block_hash calculation in kv-events (vllm's kv-events use the maybe_convert_block_hash function by default to convert the hashed result as the block_hash value).

q2: The KV indexer will calculates the hash based on tokens (during the /query API). I think for a service, supporting more APIs might not be a bad thing?

q3: Under the current design, service discovery and heartbeat support are required, and I think that the service registration of the indexer is handled by the router. Therefore, engine_name is merely an alias for a inference service instance, which facilitates retrieval and look-up operations. I plan to first implement this through a simple /register API, with preliminary ideas like:

{
    "engine_name": "xxx",
    "engine_ip": "127.0.0.1",
    "engine_port": 8000,
    "kv_event_port": 5557,
    "service_type": "vllm/sglang..",
    "model_name": "deepseek",
    "lora_id": -1
}

Actually, I’m not familiar with this part. Do you have a better design in mind?

q4: This has been changed to longest_matched, meaning exactly as you described: the maximum number of hits across all data parallel instances and memory levels. I’ll make the updates later.

q5: The multi-tenant scenario is reasonable, and I prefer using the term tenant_id

[PeaBrane]

@yejj710 Thank you.

The challenge lies in promoting different communities to adopt this practice.

Yea that is the unfortunate part, so I think this is a good opportunity to set a community contract for them to follow. At the minimum, to even use the /query-by-hash, there would be an assumption of the hashing algo used for the blocks, iiuc, and I think it would be good to define that here.

In addition, internally, I'm guessing the KvIndexer would need to rely on a unique hash for each sequence block, which I'm assuming would need to be a rolling hash of the block hashes (calling it sequence hash from here on). This is something the engines could also compute themselves and emit as KvEvents (as currently done), and we don't technically need to control that. But if we can enforce that rolling hash algo for the seq hashes as well, it would enable a lot of optimization opportunities for the indexer itself. So it would be nice to define/standardize that here as well.

For reference, dynamo uses xxh3-64 for both block hash and sequence hash computations, with a seed of 1337. We can standardize something like this.

I think for a service, supporting more APIs might not be a bad thing?

I think I'm fine with that

Do you have a better design in mind?

I'm not an expert, but can refer to the dynamo discovery and mechanism. But API wise, I think /register should be fine.

[doujiang24]

1. Should we also standardize here the hashing algo? I think each engine framework hashes their KV events differently, so we may also want to discuss a deterministic hashing algo here (sha256) as the universal contract.
2. If passing by raw tokens, where would the hashing happen, is it inside the KV indexer service? This may make the indexer service itself not super scalable. I wonder if we should just not support raw tokens as input altogether.

Agreed, a standard hashing algo could be better. xxh3-64 used in dynamo sounds like a good choice.

My thoughts:

1. as @PeaBrane said, standardize the hashing algo as a proposal, push sglang/vllm to following the hashing algo.
2. keep the token_ids in kv-events and using the standard hashing algo in kv-indexer as starting, before the engines not following the standard hashing algo.

@PeaBrane @yejj710 Thoughts?

[doujiang24]

3. What is the mechanism here for each engine registering their engine_name to the KvIndexer service? Would there be any dynamic discovery and heartbeat mechanism here?

yeah, engine_name is confusing.
publisher-unique-name might be better, it is a unique name for a kv-events publisher endpoint in the engine side.

Also, router and kv-indexer shared the same dynamic discovery mechanism, so the router also know the publisher-unique-name, then it can map the publisher-unique-name to the endpoint(ip:port) for inferencing, and other metadata etc.

If I remember correctly, each dp-rank have a unique kvevents zmq publisher endpoint.

And for the dynamic discovery mechanism, It should be extensible, as there may be multiple different implementations.

[doujiang24]

5. Should we have multi-tenant isolation, e.g. another field such as namespace or tenant_id. It also prevents a super large output, if the user is only interested in a subset of instances.

Doe the multi-tenant means

1. two users shared the same engine instances, but they do not want to share the kvcache.
2. two users use the same model in the inference API and /query API, but they have two group of engine instances, so we need the tenant_id in the /query API?

For 1: it might not be reasonable.
For 2: it might make sense, it could be an option field, not a requirement. As a use case, we do use different model when users do not want to share engine instances.

[doujiang24]

On an unrelated note, that may also be a need for a extra_id defaulting to None (or generally an additional optional u64) that translates to a salt hash in the indexer. That would be potentially beneficial for, say, a promise to keep hash lookups of different "customers" separate.

Does it mean: two users shared the same engine instances, but they do not want to share the kvcache?

Or is this some kind of security consideration?

[yejj710]

@PeaBrane thanks

On an unrelated note, that may also be a need for a extra_id defaulting to None that translates to a salt hash in the indexer

The idea of salt hash sounds like a very reasonable use case. I will add the cache_salt attribute for this scenario.

At the minimum, to even use the /query-by-hash, there would be an assumption of the hashing algo used for the blocks, iiuc;
I'm guessing the KvIndexer would need to rely on a unique hash for each sequence block, which I'm assuming would need to be a rolling hash of the block hashes

yeah, that's how the code is currently implemented. Maybe starting with standardization between KvIndexer and the router would be a good first step.

[yejj710]

1. as @PeaBrane said, standardize the hashing algo as a proposal, push sglang/vllm to following the hashing algo.
2. keep the token_ids in kv-events and using the standard hashing algo in kv-indexer as starting, before the engines not following the standard hashing algo.

Totally agree

publisher-unique-name might be better, it is a unique name for a kv-events publisher endpoint in the engine side.
it can map the publisher-unique-name to the endpoint(ip:port) for inferencing,

This name is clearer than engine_name, but there is another issue to consider: under Hybrid DPLB and Internal DPLB deployment modes, the router cannot accurately dispatch requests to specific DP ranks, and thus cannot maintain this map. In other words, does the router need to consider both the Hybrid DPLB and Internal DPLB deployment modes?
@doujiang24

If I remember correctly, each dp-rank have a unique kvevents zmq publisher endpoint.

Yes, the specific implementation details in vLLM can be viewed at: https://github.com/vllm-project/vllm/blob/main/vllm/distributed/kv_events.py#L294