Skip to content

Add gradient computation for bias and token-level KV sparsity support #214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LoserCheems merged 1 commit into from
Dec 12, 2025
Merged

Add gradient computation for bias and token-level KV sparsity support #214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ Thus, a more effective approach is sparse attention: interacting each query with
- Grouped Query Attention and Multi Query Attention
- Flexible Mask and Bias
- Skipping memory access and computation for masked regions
- Gradient computation for bias
- Gradient computation for bias to support learnable attention sink
- Token-level KV sparsity for each Q

### Features We Aim to Support

Comment on lines +38 to 42
Copy link

Copilot AI Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description claims "New tests added to validate the gradient computation and sparsity features," but only README files are being modified in this PR. No test files or actual implementation code changes are included. If these features were implemented and tested in a previous commit, the PR description should be updated to accurately reflect that this PR only updates documentation. If the implementation and tests are planned for the future, these features should be moved to the "Features We Aim to Support" section instead.

Suggested change
- Gradient computation for bias to support learnable attention sink
- Token-level KV sparsity for each Q
### Features We Aim to Support
### Features We Aim to Support
- Gradient computation for bias to support learnable attention sink
- Token-level KV sparsity for each Q

Copilot uses AI. Check for mistakes.
Expand Down
3 changes: 2 additions & 1 deletion README_zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ Flash-Sparse-Attention 是一个高性能的可训练稀疏注意力实现, 将
- 分组查询注意力和多查询注意力
- 灵活的掩码与偏置
- 跳过掩码区域的访存与计算
- 偏置的梯度计算
- 偏置的梯度计算以支持可学习 attention sink
- 对于每个 Q 有 token 级别的 KV 稀疏性

### 我们想要支持的功能

Expand Down
Loading