[Usage]: "Meta-only" put&get | 只管理元数据

[FrankJoy123]

Describe your usage question

Hi all, thanks for this marvelous work! Helps a lot!

When using STORE, I am a little confused.

Scenario

Say I have A->B. I manage a buffer, say torch.empty(8gb) -> buf_ptr on A

• I call resigter_memory on buf_ptr
• I allocate 1GB on A within buf_ptr, which I am sure about. Say it's about kv, kv_ptr = SliceFrom(buf_ptr, 1gb)
  • I will write sth into kv_ptr
• I wanna call put_from(key, ptr, size), or batch_put_from for many.

On another machine, say B, I call get_into or batch_get_into from the key.

Questions

All I want is Store to help me MANAGE THE META!!

I do now need, and want it to demand another space(especially GPU) for its work.

But I found that I cannot accomplish with small global_segment_size

Why? I wish it can just like PUBLISH the meta.

put_from/get_into, like what they said, Store does not HOLD any raw data.

A lot of thanks!

🤔 I know, TE suits me better, if I transfer the ptr, len and sth else with my own message system, and just call the transfer op. But I want to use the K-V philosophy of Mooncake Store. I want global machines have access to the data via a KEY, not the hand-crafted META, and Hi storage, eviction, topology awareness, etc... I think it's much neater if the Store can support it at the framework level than I do. Now it seems like I have no control over its own management of CUDA memory of Store.

[ykwd]

Thanks for the detailed explanation. This is a very interesting use case.

At the moment, the Store implementation does not support a meta-only / pointer-registration mode as you described.

That said, the scenario you outlined does seem practically useful, especially for advanced users who already manage GPU memory explicitly.

I think this is a reasonable design discussion to have, and it may be worth exploring whether and how such a capability could be supported in the future, possibly as an alternative mode or a separate abstraction.

Happy to continue the discussion and hear more thoughts on the expected semantics and constraints.

[FrankJoy123]

Thanks for the detailed explanation. This is a very interesting use case.

At the moment, the Store implementation does not support a meta-only / pointer-registration mode as you described.

That said, the scenario you outlined does seem practically useful, especially for advanced users who already manage GPU memory explicitly.

I think this is a reasonable design discussion to have, and it may be worth exploring whether and how such a capability could be supported in the future, possibly as an alternative mode or a separate abstraction.

Happy to continue the discussion and hear more thoughts on the expected semantics and constraints.

Exactly 😃. It really gets to the heart of RDMA: the MoonCake Store (as the control plane) orchestrates the keys and kicks off the transfer engine.
The control plane, by definition, shouldn't have a heavy memory footprint—especially when we need granular control over the pointers.

[FrankJoy123]

@ykwd

By the way, I am a little confused about the CUDA memory requirements of TE/ Store.

• AFAIK, TE is just a powerful wrapper of RDMA, which I can use to launch P2P, pointer-level transfer, and query the status viaget_batch_transfer_status with native async supports. Right?
  • Actually my case matches TE better. But I need to write some of the control code, like PUB/SUB message system, decentralized kv storage(for the key & TE meta), or maybe further error tolerance.
• However, Store seems to demand memory, whether I use put/get or put_from/get_into. What's the point of design(or maybe not a design) like this when I have already declared a user-managed array.
  • Replica? It must HOLD sth ITSELF and prevent DATA LOSS?
  • Or just because of the code, where you want to reuse the logic of put :P ?

[ykwd]

Originally, TE was designed for P2P data transfer, while the store was designed for distributed shared data storage.

The global segments used internally by the store to hold data are not directly exposed to users. Users can only write and read data through interfaces such as put and get. To improve read and write performance, the store introduced zero-copy capabilities, allowing zero-copy data transfers via RDMA between user-provided buffers and data stored in global segments. This requires the user-provided buffers to be registered with the RDMA NIC in advance, which is why buffer registration is needed.

However, over the past few months, the usage scope of Mooncake Store has been expanding. For example, it has been used for data transfer in reinforcement learning workloads. As a result, we could further extend the store’s interfaces and capabilities as needed, such as adding data distribution and subscription APIs similar to ZeroMQ, or allowing user-accessible segments that can be directly read and written (with appropriate mechanisms to ensure there are no data races when the data is shared with remote clients).

Would you be interested in working on developing this part?

[FrankJoy123]

Originally, TE was designed for P2P data transfer, while the store was designed for distributed shared data storage.

The global segments used internally by the store to hold data are not directly exposed to users. Users can only write and read data through interfaces such as put and get. To improve read and write performance, the store introduced zero-copy capabilities, allowing zero-copy data transfers via RDMA between user-provided buffers and data stored in global segments. This requires the user-provided buffers to be registered with the RDMA NIC in advance, which is why buffer registration is needed.

However, over the past few months, the usage scope of Mooncake Store has been expanding. For example, it has been used for data transfer in reinforcement learning workloads. As a result, we could further extend the store’s interfaces and capabilities as needed, such as adding data distribution and subscription APIs similar to ZeroMQ, or allowing user-accessible segments that can be directly read and written (with appropriate mechanisms to ensure there are no data races when the data is shared with remote clients).

Would you be interested in working on developing this part?

Got it. Thanks for clarification and invitation. Cause TE seems to suit our scenario(P2P) quite well, we are working on wrapping it in our system now. However, as you said, problems such as data racing, which occur in classic distributed storage systems may also occur in Mooncake Storage.

We will keep track of the community. Maybe later co-work or feedbacks 😃

[ykwd]

There is a PR working on building a set of zmq-like APIs over TE: #1370 . Will this meet some of your requirements?

[FrankJoy123]

@ykwd
Thanks for this info. Neat Interface.
However it seems focused on the ZERO-COPY TRANSFER of RPC params

# Producer
# Assume Tensor is supported. 
# But I do not see this in the example
tensor = torch.ones(...)
pub.send_pyobj(socket_id, tensor)

# Consumer
local_copy = None
def on_pyobj(msg):
  local_copy = msg['obj']
  # maybe
  # local_copy.copy_(msg['obj'])
  # But this will introduce a Mooncake Manager Buffer -> User Space.
  # Here **Mooncake Manager Buffer** is exactly what we do not use.

It seems we could get DIRECTLY FROM the remote tensor, and use the local_copy. Seems good, but really similar to "put_from"!

But we often issue P2P/ ptr-to-ptr Transfer just VIA KEY 😕
I think this mechanism still allows/ expects Mooncake to hold the transferred data.