[Ecosystem] Mooncake

[zhyncs]

Contact emails

james0zan@gmail.com, me@zhyncs.com

Project summary

A KVCache-centric Disaggregated Architecture for LLM Serving

Project description

• Transfer Engine is a high-performance data transfer framework. Transfer Engine provides a unified interface to transfer data from DRAM, VRAM or NVMe, while the technical details related to hardware are hidden. Transfer Engine supports TCP, RDMA (InfiniBand/RoCEv2/eRDMA/NVIDIA GPUDirect) and NVMe over Fabric (NVMe-of) protocols.

• P2P Store is built on the Transfer Engine and supports sharing temporary objects between peer nodes in a cluster. P2P Store is ideal for scenarios like checkpoint transfer, where data needs to be rapidly and efficiently shared across a cluster.

• Mooncake Store is a distributed KVCache storage engine specialized for LLM inference based on Transfer Engine. It is the central component of the KVCache-centric disaggregated architecture. The goal of Mooncake Store is to store the reusable KV caches across various locations in an inference cluster.

• Mooncake Backend serves as a fault-tolerant PyTorch distributed backend. It provides robust collective communication primitives capable of continuing operation seamlessly in the presence of rank failures. Mooncake EP extends these capabilities to support elastic and fault-tolerant MoE model inference with dynamic token routing.

Are there any other projects in the PyTorch Ecosystem similar to yours? If yes, what are they?

No.

While SGLang, vLLM, and LMCache are also designed for LLM serving, they operate in a collaborative and complementary relationship with Mooncake. Mooncake is deeply integrated into these projects and serves as a foundational infrastructure layer, providing high-performance data transfer for PD and EPD architectures, distributed and shared KV cache storage, and collective communication for EP parallelism, among other core functionalities.

Project repo URL

https://github.com/kvcache-ai/Mooncake

Additional repos in scope of the application

No response

Project license

Apache License

GitHub handles of the project maintainer(s)

@james0zan, @stmatengss, @UNIDY2002, @ShangmingCai, @alogfans, @chestnut-Q, @ykwd

Is there a corporate or academic entity backing this project? If so, please provide the name and URL of the entity.

No response

Website URL

https://kvcache-ai.github.io/Mooncake/

Documentation

https://kvcache-ai.github.io/Mooncake/getting_started/quick-start.html

https://kvcache-ai.github.io/Mooncake/index.html

How do you build and test the project today (continuous integration)? Please describe.

We rely on GitHub Actions as our CI system to automatically build and test the project. An example CI run can be found here: https://github.com/kvcache-ai/Mooncake/actions/runs/20810476279

Version of PyTorch

2.8.0, 2.9.0, 2.9.1

Components of PyTorch

As a communication library for LLM serving, this project primarily leverages PyTorch Distributed. See:

https://docs.pytorch.org/docs/stable/distributed.html
https://kvcache-ai.github.io/Mooncake/python-api-reference/ep-backend.html

How long do you expect to maintain the project?

This project is a critical dependency for inference engines such as vLLM and SGLang. It powers key components, including the transfer engine for PD disaggregation data transfer and the Mooncake store for distributed CPU memory caching. As a result, the project will be actively and continuously maintained.

Additional information

Mooncake’s architecture is based on the FAST 2025 Best Paper. The open-source project has 130 contributors and is maintained and contributed to by developers from NVIDIA, AMD, Intel, Google, Moonshot AI, Tsinghua University, Stanford, Alibaba, Approaching.ai, Ant Group, Tencent, etc., as well as individual contributors.

Mooncake has deep collaboration and integration with PyTorch Foundation projects such as SGLang, vLLM, and LMCache, and has been widely adopted across many companies, operating at scale on thousands of GPUs.

[zhyncs]

Measurable Requirements:

✅ 1. Core Maintainers: The project must have a minimum of 2 core maintainers
✅ 2. Contributors: The project should have a minimum of 5 total contributors active in the last 90 days.
https://github.com/kvcache-ai/Mooncake/pulse
✅ 3. Commits: The project must have 30 commits in the last 90 days.
https://github.com/kvcache-ai/Mooncake/pulse
✅ 4. Stars: The project must have a minimum of 200 stars on GitHub
https://github.com/kvcache-ai/Mooncake 4.6k stars

Github Layout Requirements:

✅ 1. LICENSE.md
✅ 2. README.md
✅ 3. CONTRIBUTING.md
✅ 4. CODEOWNERS
✅ 5. CODE_OF_CONDUCT.md

[zhyncs]

Reference papers:

• FAST 25 Best Paper
• ACM Transaction Storage

[zhyncs]

Hi @lantiga @michellearoth May you help take a look? Thanks!

[lantiga]

Thank you @zhyncs for the great submission, on it.

[lantiga]

Checks all boxes, we'll fast track the vote.

Functional Requirements:

✅ 1. PyTorch Support: Project must demonstrate benefit to the pytorch community and be validated with the latest versions of PyTorch.
✅ 2. Working CI: Projects must have a demonstrable working CI workflow for all declared architectures
✅ 3. Governance: Projects must have documented technical governance defined and implemented and it must be represented in their Github or Gitlab repository. Example: Review the LF’s Minimum Viable Governance framework.
✅ 4. User experience: The value proposition should be made clear for the project and documented in the README of the project repository.
✅ 5. Quick start: Project should have a working and demonstrable quick start.
✅ 6. Documentation: Projects must have clear and comprehensive documentation.
✅ 7. Permissive licensing: Users must be able to utilize ecosystem projects without licensing concerns. e.g. BSD-3, Apache-2 and MIT licenses
✅ 8. Functional Testing: Users should have developed tests submitted as a part of their project, and be confident that the project will work well with the latest releases of PyTorch.
✅ 9. Packages: Projects need to have support for binary installation options (pip/Conda).
✅ 10. Ongoing maintenance: Project maintainers must be committed to supporting and maintaining their projects for the next 12 months at a minimum. Projects must commit to updating their projects to the latest two releases of PyTorch (e.g. versions 2.6, 2.5.1).

Measurable Requirements:

✅ 1. Core Maintainers: The project must have a minimum of 2 core maintainers
✅ 2. Contributors: The project should have a minimum of 5 total contributors active in the last 90 days.
✅ 3. Commits: The project must have 30 commits in the last 90 days. This is off by like 15 days: it shouldn't be a deal breaker.
✅ 4. Stars: The project must have a minimum of 200 stars on github

Github Layout Requirements:

✅ 1. LICENSE.md at the root of the repository specifying the terms and conditions for using, distributing, and modifying the software. In addition, you should provide information on the license of any third-party code included in the project.
✅ 2. README.md welcomes new community members to the project and explains why the project is useful and how to get started. Should include information about release methodology, cadence, and criteria in the README.
✅ 3. CONTRIBUTING.md explains how to contribute to the project. The file explains the types of contributions needed and how the development process works.
✅ 4. CODEOWNERS that define individuals or teams responsible for code in a repository; document current project owners and Retired committers.
✅ 5. CODE_OF_CONDUCT.md sets the ground rules for participants’ behavior and helps facilitate a friendly, welcoming environment. By default, projects should leverage the [Linux]([https://lfprojects.org/policies/code-of-conduct/) [Foundation](https://lfprojects.org/policies/code-of-conduct/) [Code](https://lfprojects.org/policies/code-of-conduct/) of [Conduct](https://lfprojects.org/policies/code-of-conduct/) unless](https://lfprojects.org/policies/code-of-conduct/)%C2%A0%5B%5BFoundation%5D(https://lfprojects.org/policies/code-of-conduct/)%5D(https://lfprojects.org/policies/code-of-conduct/)%C2%A0%5B%5BCode%5D(https://lfprojects.org/policies/code-of-conduct/)%5D(https://lfprojects.org/policies/code-%5Bof%5D(https://lfprojects.org/policies/code-of-conduct/)-conduct/)%C2%A0%5Bof%5D(https://lfprojects.org/policies/code-of-conduct/)%C2%A0%5B%5BConduct%5D(https://lfprojects.org/policies/code-of-conduct/)%5D(https://lfprojects.org/policies/code-of-conduct/)%C2%A0unless) an alternate Code of Conduct is approved prior.

[zhyncs]

Hi @lantiga, thanks for the quick response — I really appreciate your help!
Could you please share the ETA for the voting process?
And feel free to reach out if you need any additional information. Thanks!

[michellearoth]

Accepted!