Coding-attention-mechanism-from-scratch

🔥 Demystifying Large Language Models: The Importance of Understanding "Under the Hood"

🎯 As large language models (LLMs) continue to transform the tech landscape, it's easy to focus solely on application building and overlook what's actually happening inside these complex systems. While creating innovative applications is exciting, I believe it's just as crucial to understand the mechanics behind LLMs.

💡 That's why I created a Jupyter notebook that explores the attention mechanism from scratch, focusing on its role in language translation which is one of the earliest applications that revolutionized LLMs.

👉🏻 In this notebook, I demonstrate the difference between a simple encoder-decoder structure and an encoder-decoder with attention. By implementing the attention mechanism and comparing BLEU scores, I highlight how attention significantly enhances translation accuracy. This deeper dive into the inner workings of LLMs not only strengthens our knowledge but also guides us toward building more efficient applications.

📰 Reference research paper Links

Paper 1: Effective approach to attention based neural machine translation: https://arxiv.org/pdf/1409.0473.pdf
Paper 2: Neural Machine Translation by jointly learning to align and translate: https://arxiv.org/pdf/1508.04025.pdf

📸 Global and local attention illustrations

• Here I have coded global attention for language translation of Italian to English
  
  
  

📊 Visualizing attention weights for 3 scoring functions Dot, General and Concat

🏆 BLEU score comparison between the models