Roadmap

[shenxiangzhuang]

LLM basic

• LLM: Perplexity #43
• BPE #33
• LLM: BN or LN #41
• LLM: BLEU from scratch #42
• LLM: Model Parameter counting #39
• Sampling from logits: TopK, TopP, Temparature #36

LLM architectures

• GPT2 training & inference
• llama model structure #100
• deepseek model structure #101
• qwen model structure #102
• LoRA, QLoRA and DoRA #38

Fast inference

• Speculative Sampling #54
• Feature: KV Cache #81

Reinforce learning

• Feature: GRPO #84
• RLHF: PPO & DPO & GRPO #35