title	HopfieldAttention
emoji	🧠
colorFrom	indigo
colorTo	blue
sdk	streamlit
sdk_version	1.42.0
app_file	dashboard/app.py
pinned	false
short_description	From Hopfield Networks to Transformers — the interactive connection
tags	hopfield-networks transformers attention demo streamlit
license	mit

HopfieldAttention

From Hopfield Networks (1982) to Transformers (2017): the mathematical bridge, traced across 7 articles.

Interactive Demo · Articles · Bridge Code

The attention mechanism in Transformers is, mathematically, a single-step Hopfield update with softmax as activation. This project traces that connection from first principles — starting with Hopfield's 1982 energy function, through a 1998 university thesis on shortest path problems, to Ramsauer et al. (2021) and modern LLMs.

The Core Equivalence

Hopfield Update	Transformer Attention
V_new = softmax(β · Ξ · V) · Ξ	Attention(Q,K,V) = softmax(Q·Kᵀ/√d) · V

Both compute a softmax-weighted sum over a set of stored vectors. The correspondence is exact:

Hopfield	Attention	Meaning
State V	Query Q	What we're processing
Patterns Ξ	Keys K	What we know / the memory
Patterns Ξ	Values V	What we retrieve
Temperature β	Scale 1/√d	How sharp the attention is

Contents

Articles — 7-part series

#	Title	Topic
1	The Original Hopfield	Energy function, attractors, associative memory
2	My 1998 Thesis	Hopfield for Shortest Path Problem
3	Seven Fixes	Why classical Hopfield fails at optimization
4	Modern Hopfield	Exponential capacity, continuous values
5	The Connection	The bridge: Hopfield IS attention
6	Subspace Attention	From standard to subspace attention
7	Looking Ahead	Hopfield layers in modern deep learning

Bridge Code

bridge/hopfield_attention_bridge.py implements the same operation as both a Hopfield update and as attention, and verifies they are numerically identical:

from bridge.hopfield_attention_bridge import HopfieldAttention
import numpy as np

layer = HopfieldAttention(dim=16, beta=2.0)
state = np.random.randn(16).astype(np.float32)
state /= np.linalg.norm(state)

result = layer.compare(state)
print(result["are_identical"])       # True
print(result["difference_norm"])     # < 1e-6

Run the tests:

cd bridge && pytest test_bridge.py -v

Interactive Dashboard

pip install -r dashboard/requirements.txt
streamlit run dashboard/app.py

Or use the live demo on HuggingFace Spaces.

Adjust temperature β, number of stored patterns, and noise level — and see in real time how the Hopfield update and the attention mechanism produce identical outputs.

Background

In 1998 I wrote a university thesis implementing a Hopfield Network for the Shortest Path Problem. In 2025, working on ML systems for 5G networks, I realized the attention mechanism I was using daily was mathematically the same operation I had studied 25 years earlier. This project documents that connection.

License

MIT — see LICENSE