hash 128 and Update doc

[blueswhen]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @blueswhen, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive documentation for LightLLM's multi-level KV cache system, which leverages GPU, CPU, and disk for efficient long-text and high-concurrency scenarios. It also upgrades the KV cache hashing to 128-bit for improved collision resistance, includes minor documentation formatting fixes, and enhances process management for the KV cache manager.

Highlights

• New Multi-Level KV Cache Documentation: Comprehensive documentation has been added for LightLLM's multi-level KV cache system, covering GPU (L1), CPU (L2), and Disk (L3) caching. This includes prerequisites, architectural overview, working principles, applicable scenarios, and detailed deployment instructions for both two-level (L1+L2) and three-level (L1+L2+L3) configurations in both Chinese and English.
• KV Cache Hashing Upgrade to 128-bit: The KV cache hashing mechanism has been upgraded from 64-bit to 128-bit using xxhash.xxh3_128. This change involves modifications to data structures in token_chunck_hash_list.py, cpu_cache_client.py, and shm_objs.py to store 128-bit hash keys (split into high and low 64-bit parts), and updates to hash computation and usage across relevant files to reduce potential hash collisions.
• Documentation Formatting Improvements: Several existing documentation files have received minor formatting adjustments, primarily to reStructuredText section headers, ensuring consistency and readability across the tutorials.
• Enhanced Logging for GPU Cache Metrics: Logging output in httpserver/manager.py has been refined to explicitly distinguish between gpu_prompt_cache_len and gpu_prompt_cache_ratio, providing more precise details on GPU cache hit performance.
• Improved Process Management for KV Cache Manager: The multi-level KV cache manager (multi_level_kv_cache/manager.py) now includes setproctitle for clearer process identification and start_parent_check_thread for enhanced process robustness and monitoring.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces documentation for a new multi-level cache feature and implements the necessary code changes to support 128-bit hashes for this cache, which is a good measure to reduce hash collisions. The documentation changes include adding a new tutorial and fixing formatting in several existing documents. The code changes are consistently applied across multiple files to support 128-bit hashes, including updates to hash generation, data structures, and related function calls. Additionally, process management utilities have been added to the multi-level cache manager for better robustness. My review includes suggestions to improve the clarity of an example in the new documentation. Overall, the changes are well-implemented and improve the project.

[gemini-code-assist]

在 hugepages 的配置示例中，{需要启用的大页容量} 这个占位符对于用户来说可能不够清晰。建议提供一个具体的示例值，例如 512，并在文档中说明其单位（例如，表示 512GB，如果 hugepagesz=1G），这样能更好地帮助用户理解如何配置。

Suggested change

	hugepagesz=1G hugepages={需要启用的大页容量}/' /etc/default/grub
	hugepagesz=1G hugepages=512/' /etc/default/grub

[gemini-code-assist]

In the hugepages configuration example, the placeholder {required_huge_page_capacity} might be unclear to users. It would be more helpful to provide a concrete example value, such as 512, and explain its unit in the documentation (e.g., it represents 512GB if hugepagesz=1G). This would help users better understand the configuration.

Suggested change

	hugepagesz=1G hugepages={required_huge_page_capacity}/' /etc/default/grub
	hugepagesz=1G hugepages=512/' /etc/default/grub