post : Efficient Attention

proof

Author: woongjoonchoi

_posts/DeepLearning/Kernel Fusion/2025-03-07-fused.md

@@ -63,14 +63,14 @@ $$ y_i = \frac{e^{x_i - \max_{k=1}^V x_k}}{\sum_{j=1}^V e^{x_j - \max_{k=1}^V x_
 
 **Induction**: \(V = S\)
 
-1. $$ Assume \quad m_{S-1} = \max_{k=1}^{S-1} x_k $$ 
+1. $$ Assume \quad m_{S-1} = \max_{k=1}^{S-1} x_k $$  
 $$m_S \leftarrow \max(m_{S-1}, x_S) $$  
 $$= \max\bigl(\max_{k=1}^{S-1} x_k,\; x_S\bigr) $$  
 $$ = \max_{k=1}^{S} x_k $$  
 
 
 
-2. $$ Assume \quad d_{S-1} = \sum_{j=1}^{S-1} e^{\,x_j - m_S} $$
+2. $$ Assume \quad d_{S-1} = \sum_{j=1}^{S-1} e^{\,x_j - m_S} $$  
 $$d_S \leftarrow d_{S-1} \, e^{\,m_{S-1} - m_S} + e^{\,x_S - m_S} $$  
 $$= \left(\sum_{j=1}^{S-1} e^{\,x_j - m_{S-1}}\right) e^{\,m_{S-1} - m_S} + e^{\,x_S - m_S} $$  
 $$ = \sum_{j=1}^{S} e^{\,x_j - m_S} $$