post : Efficient Attention

proof

Author: woongjoonchoi

_posts/DeepLearning/Kernel Fusion/2025-03-07-fused.md

@@ -123,8 +123,8 @@ max에 대한 증명은 생략하도록 하겠습니다.
 1. $$ Assume \quad d_{j} = \sum_{k=0}^{j} e^{\,x_k - m_i} \quad d_{i} = \sum_{k=i+1}^{j+i} e^{\,x_k - m_j} \quad m_{i+j} = \max(m_i, m_j) $$  
 $$d_i \oplus d_{j} = d_i \, e^{\,m_i - \max(m_i, m_j)} \;+\; d_j \, e^{\,m_j - \max(m_i, m_j)} $$  
 $$= \left(\sum_{k=0}^{i} e^{\,x_k - m_{i}}\right) e^{\,m_{i} - \max(m_i, m_j)} + \left(\sum_{k=i+1}^{i+j} e^{\,x_k - m_{i}}\right) e^{\,m_{j} - \max(m_i, m_j)} $$  
-$$= \left(\sum_{k=0}^{i} e^{\,x_k - m_{i}}\right) e^{\,\max(m_i, m_j)} + \left(\sum_{k=i+1}^{i+j} e^{\,x_k - m_{i}}\right) e^{\,\max(m_i, m_j)} $$  
-$$ = \sum_{j=1}^{S} e^{\,x_j - m_S} $$
+$$= \left(\sum_{k=0}^{i} e^{\,x_k - m_{i}}\right) e^{\,m_{i} - m_{i+j}} + \left(\sum_{k=i+1}^{i+j} e^{\,x_k - m_{i}}\right) e^{\,m_{j} - m_{i+j}} $$  
+$$ = \sum_{k=0}^{i+j} e^{\,x_k - m_{i+j}} $$