[Ecosystem] LMCache

[nijaba]

Contact emails

@apostacheng@gmail.com, Yihua Cheng
@jiayi3@uchicago.edu, Jiayi Yao
@kuntai@uchicago.edu, Kuntai Du
@nijaba@tensormesh.ai, Nick Barcet

Project summary

LLM serving engine extension that reduce TTFT and increase throughput

Project description

LMCache is an LLM serving engine extension that combines with vLLM or SGLang to reduce TTFT and increase throughput, especially under long-context scenarios. By storing the KV caches of reusable texts across various locations, including (GPU, CPU DRAM, Local Disk, Databases), LMCache reuses the KV caches of any reused text (not necessarily prefix) in any serving engine instance. Thus, LMCache saves precious GPU cycles and reduces user response delay.

By combining LMCache with vLLM, developers achieve 3-10x delay savings and GPU cycle reduction in many LLM use cases, including multi-round QA and RAG.

Are there any other projects in the PyTorch Ecosystem similar to yours? If, yes, what are they?

No

Project repo URL

https://github.com/LMCache

Additional repos in scope of the application

None

Project license

Apache 2.0

GitHub handles of the project maintainer(s)

YaoJiayi, ApostaC, KuntaiDu, Shaoting-Feng, maobaolong, sammshen, hickeyma, HuaizhengZhang

Is there a corporate or academic entity backing this project? If so, please provide the name and URL of the entity.

TensorMesh.ai https://tensormesh.ai/ and University of Chicago https://cs.uchicago.edu

Website URL

https://lmcache.ai/

Documentation

https://docs.lmcache.ai/

How do you build and test the project today (continuous integration)? Please describe.

We are using the following to run our CI:

• 2 L4 GPU servers on GCP
• Default github runner

Our CI pipeline ensures:

• Code quality check, triggered by each commit to the PR, hosted on github runners
• Unit test uses buildkite, triggered by each commit to the PR, running on the L4 GPU servers
• End-to-end correctness and performance, triggered once for each PR, running on the L4 GPU servers
• Docker images and pip packages are built after merging each PR using github actions on default github runners

We have a 2 week release cadence

Version of PyTorch

• We maintain compatibility with upstream vLLM through a connector API we contribute. Current version: 0.10.0
• We support pytorch 2.2.0~2.8.0 (latest). Right now the release version is built with pytorch 2.7.1 (which is also required by vllm)

Components of PyTorch

• Pytorch CUDA/CPP/Rocm extensions
• Using standard pytorch python APIs (tensor-related APIs)

How long do you expect to maintain the project?

We are fully committed to maintaining this project for as long as we can see.

Additional information

LMCache is the result of research conducted at the University of Chicago by a team of researchers under the supervision of assistant professor Junchen Jiang.

Reference papers:

• CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
• CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion
• Do Large Language Models Need a Content Delivery Network?

The project currently has 99 contributors and 8 maintainers which are from diverse organizations including IBM, Tencent, Bytedance, etc...

If the project is accepted as an ecosystem project, we will then propose it as a hosted project.

[lantiga]

Thank you @nijaba for the submission, very cool project. You seem to tick all the boxes : )

We'll proceed to a vote in the next few days.

Functional Requirements:

✅ 1. PyTorch Support: Project must demonstrate benefit to the pytorch community and be validated with the latest versions of PyTorch.
✅ 2. Working CI: Projects must have a demonstrable working CI workflow for all declared architectures
✅ 3. Governance: Projects must have documented technical governance defined and implemented and it must be represented in their Github or Gitlab repository. Example: Review the LF’s Minimum Viable Governance framework.
✅ 4. User experience: The value proposition should be made clear for the project and documented in the README of the project repository.
✅ 5. Quick start: Project should have a working and demonstrable quick start.
✅ 6. Documentation: Projects must have clear and comprehensive documentation.
✅ 7. Permissive licensing: Users must be able to utilize ecosystem projects without licensing concerns. e.g. BSD-3, Apache-2 and MIT licenses
✅ 8. Functional Testing: Users should have developed tests submitted as a part of their project, and be confident that the project will work well with the latest releases of PyTorch.
✅ 9. Packages: Projects need to have support for binary installation options (pip/Conda).
✅ 10. Ongoing maintenance: Project maintainers must be committed to supporting and maintaining their projects for the next 12 months at a minimum. Projects must commit to updating their projects to the latest two releases of PyTorch (e.g. versions 2.6, 2.5.1).

Measurable Requirements:

✅ 1. Core Maintainers: The project must have a minimum of 2 core maintainers
✅ 2. Contributors: The project should have a minimum of 5 total contributors active in the last 90 days.
✅ 3. Commits: The project must have 30 commits in the last 90 days.
✅ 4. Stars: The project must have a minimum of 200 stars on github

Github Layout Requirements:

✅ 1. LICENSE.md at the root of the repository specifying the terms and conditions for using, distributing, and modifying the software. In addition, you should provide information on the license of any third-party code included in the project.
✅ 2. README.md welcomes new community members to the project and explains why the project is useful and how to get started. Should include information about release methodology, cadence, and criteria in the README.
✅ 3. CONTRIBUTING.md explains how to contribute to the project. The file explains the types of contributions needed and how the development process works.
✅ 4. CODEOWNERS that define individuals or teams responsible for code in a repository; document current project owners and Retired committers.
✅ 5. CODE_OF_CONDUCT.md sets the ground rules for participants’ behavior and helps facilitate a friendly, welcoming environment. By default, projects should leverage the [Linux](https://lfprojects.org/policies/code-of-conduct/) [[Foundation](https://lfprojects.org/policies/code-of-conduct/)](https://lfprojects.org/policies/code-of-conduct/) [[Code](https://lfprojects.org/policies/code-of-conduct/)](https://lfprojects.org/policies/code-[of](https://lfprojects.org/policies/code-of-conduct/)-conduct/) [of](https://lfprojects.org/policies/code-of-conduct/) [[Conduct](https://lfprojects.org/policies/code-of-conduct/)](https://lfprojects.org/policies/code-of-conduct/) unless an alternate Code of Conduct is approved prior.

[nijaba]

Thanks a lot for accepting our project! We are delighted to join.

[michellearoth]