Skip to content

Commit 84b985e

Browse files
committed
Fix README example: use randn for attn_bias instead of ones
1 parent 964973e commit 84b985e

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -169,7 +169,7 @@ key = torch.randn(batch_size, seq_len, num_kv_heads, head_dim, device=device, dt
169169
value = torch.randn(batch_size, seq_len, num_kv_heads, head_dim, device=device, dtype=dtype)
170170

171171
# Create bias for sparse attention
172-
attn_bias = torch.ones(batch_size, num_kv_heads, seq_len, seq_len, device=device, dtype=dtype)
172+
attn_bias = torch.randn(batch_size, num_kv_heads, seq_len, seq_len, device=device, dtype=dtype)
173173

174174
# Generate dynamic mask based on bias
175175
if seq_len > window_size:

0 commit comments

Comments
 (0)