Add Native Precision Tutorial

[ouqingliang]

What does this PR do?

Add Native Precision Tutorial
update README.md

Before submitting

• Did you read the contributor guideline?
• Did you write any new necessary tests?

[gemini-code-assist]

Summary of Changes

Hello @ouqingliang, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily introduces extensive documentation and support for running native precision models (BF16, FP8, INT4) within the KTransformers framework, leveraging SGLang and KT-Kernel. It includes a new, detailed tutorial covering setup, model inference, and technical considerations. Complementary code changes enhance the CPU backend's task distribution, improve the interactive chat client by displaying reasoning content, and ensure robust handling of FP8 model scales.

Highlights

• New Native Precision Tutorial: A comprehensive guide on 'Running Native Precision Models with SGLang and KT-Kernel' has been added, detailing supported precision formats (BF16, FP8, FP8_PERCHANNEL, RAWINT4), models, hardware requirements, setup, server launch, and inference.
• README Update: The main README.md now includes a new entry under 'Updates' for January 22, 2026, highlighting support for CPU-GPU Expert Scheduling, Native BF16 and FP8 per channel Precision, and AutoDL unified fine-tuning and inference, with links to the new tutorial.
• CPU Backend Task Processing Adjustment: In kt-kernel/cpu_backend/worker_pool.cpp, the block size for task processing has been explicitly set to 1, which might affect how tasks are distributed among worker threads.
• Enhanced Chat CLI Output: The kt-kernel/python/cli/commands/chat.py now supports displaying reasoning_content from model responses in the interactive chat, presenting it in a dimmed style for better user experience.
• FP8 Scale Type Enforcement: In kt-kernel/python/utils/amx.py, a check and conversion have been added to ensure that FP8 model scales (gate_scales, up_scales, down_scales) are consistently torch.float32, preventing potential type mismatches during weight loading.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new tutorial for Native Precision Models with SGLang and KT-Kernel, along with updates to the README.md file. It also includes changes to the CPU worker pool logic, enhancements to the chat CLI for reasoning content, and a type conversion fix for FP8 scales. Overall, the changes are well-structured and add valuable documentation and functionality.

[gemini-code-assist]

Hardcoding block = 1 here overrides the previously calculated block size, which was determined based on the remaining tasks and worker count. This changes the work-stealing strategy from a guided approach (where block size dynamically adjusts) to a fixed block size of 1. This could potentially impact load balancing and overall performance, especially for tasks with varying granularities or when the number of workers is large. Please clarify the intent behind this change and its expected impact on performance.

[gemini-code-assist]

The --kt-method parameter reference lists AMXINT4 as a possible value. However, this precision format is not included in the "Supported Precision Formats" table (lines 25-30). Please either add AMXINT4 to the table with its description or remove it from the parameter reference for consistency.

[gemini-code-assist]

The link to the "CPU-GPU Expert Scheduling Tutorial" appears to be incorrect. It currently points to ./experts-sched-Tutorial.md, but based on the file structure, it should likely be ./CPU-GPU-Expert-Scheduling.md to correctly reference the file in the same directory.

Suggested change

	See [CPU-GPU Expert Scheduling Tutorial](./experts-sched-Tutorial.md) for details.
	See [CPU-GPU Expert Scheduling Tutorial](./CPU-GPU-Expert-Scheduling.md) for details.