Attention Mechanisms

31 Oct 2024

🚧 Work in progress…

This article will cover attention mechanisms in neural networks and their role in modern language models.

Topics to cover:

• What is attention and why it matters
• Self-attention mechanism
• Multi-head attention
• Scaled dot-product attention
• Attention in Transformers
• Cross-attention vs self-attention
• Applications in modern LLMs