post: Efficient Attention

woongjoonchoi · woongjoonchoi · commit 68771f11a2a8 · 2025-03-19T14:42:35.000+09:00
Online self-attention latex
diff --git a/_posts/DeepLearning/Kernel Fusion/2025-03-07-fused.md b/_posts/DeepLearning/Kernel Fusion/2025-03-07-fused.md
@@ -165,6 +165,7 @@ attention에는 softmax 때문에 input 전체를 봐야한다.
 
 self-attention에도 이를 적용한다.[[link]](https://arxiv.org/pdf/2112.05682) 
 
+$$ v^* \in \mathbb{R}^d $$ ,$$ s^* \in \mathbb{R} $$ 를   $$ s_i = \mathrm{dot}(q, k_i) $$ , $$ v^* \leftarrow v^* + v_i e^{s_i} $$ , $$ s^* \leftarrow s^* + e^{s_i} $$
 
 
 

Original file line number	Diff line number	Diff line change
`@@ -165,6 +165,7 @@ attention에는 softmax 때문에 input 전체를 봐야한다.`
`165`	`165`
`166`	`166`	`self-attention에도 이를 적용한다.[[link]](https://arxiv.org/pdf/2112.05682)`
`167`	`167`
	`168`	`+$$ v^* \in \mathbb{R}^d $$ ,$$ s^* \in \mathbb{R} $$ 를 $$ s_i = \mathrm{dot}(q, k_i) $$ , $$ v^* \leftarrow v^* + v_i e^{s_i} $$ , $$ s^* \leftarrow s^* + e^{s_i} $$`
`168`	`169`
`169`	`170`
`170`	`171`