Hi team, thanks for the great resource!
I noticed a slight inconsistency in the main Transformer block diagram in the "All the Transformer Math" chapter.
The diagram currently illustrates a Post-Norm setup (the norm D blocks are placed directly on the main residual stream after the + operations). However, it also explicitly features modern components like GQA and GLU variants.
Since modern LLMs utilizing GQA/GLU (like Llama) almost exclusively use Pre-Norm to maintain a clean residual stream, the norm D blocks should probably be moved to the beginning of the Attention and MLP branches instead.
Thought I'd point this out in case you want to update the diagram to reflect modern topologies!
Hi team, thanks for the great resource!
I noticed a slight inconsistency in the main Transformer block diagram in the "All the Transformer Math" chapter.
The diagram currently illustrates a Post-Norm setup (the norm D blocks are placed directly on the main residual stream after the + operations). However, it also explicitly features modern components like GQA and GLU variants.
Since modern LLMs utilizing GQA/GLU (like Llama) almost exclusively use Pre-Norm to maintain a clean residual stream, the norm D blocks should probably be moved to the beginning of the Attention and MLP branches instead.
Thought I'd point this out in case you want to update the diagram to reflect modern topologies!