You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In this assignment, you will explore computational memory models by implementing a Hopfield network. In the original article ([Hopfield (1982)](https://www.dropbox.com/scl/fi/iw9wtr3xjvrbqtk38obid/Hopf82.pdf?rlkey=x3my329oj9952er68sr28c7xc&dl=1)), neuronal activations were set to either 0 ("not firing") or 1 ("firing"). Modern Hopfield networks nearly always follow an updated implementation, first proposed by [Amit et al. (1985)](https://www.dropbox.com/scl/fi/3a3adwqf70afb9kmieezn/AmitEtal85.pdf?rlkey=78fckvuuvk9t3o9fbpjrmn6de&dl=1). In Amit et al.'s framing, neurons take on activation values of either -1 ("down state") or +1 ("up state"). This has three important benefits over Hopfield's original implementation:
5
-
- It provides a cleaner way to implement the Hebbian learning rule (i.e., without subtracting means or shifting values).
6
-
- It avoids a bias towards 0 (i.e., +1 and -1 are equally "attractive" whereas 0-valued neurons have a stronger "pull").
7
-
- The energy function (i.e., a description of the attractor dynamics of the network) can be directly mapped onto the [Ising model](https://en.wikipedia.org/wiki/Ising_model) from statistical physics.
8
4
9
-
You should start by reading [Amit et al. (1985)](https://www.dropbox.com/scl/fi/3a3adwqf70afb9kmieezn/AmitEtal85.pdf?rlkey=78fckvuuvk9t3o9fbpjrmn6de&dl=1) closely. Then you should code up the model in a Google Colaboratory notebook. Unless otherwise noted, all references to "the paper" refer to Amit et al. (1985).
5
+
In this assignment, you will explore computational memory models by implementing a Hopfield network.
6
+
7
+
In the original article ([Hopfield, 1982](https://www.dropbox.com/scl/fi/iw9wtr3xjvrbqtk38obid/Hopf82.pdf?rlkey=x3my329oj9952er68sr28c7xc&dl=1)), neuronal activations were set to either 0 ("not firing") or 1 ("firing"). Modern Hopfield networks nearly always follow an updated implementation, first proposed by [Amit et al. (1985)](https://www.dropbox.com/scl/fi/3a3adwqf70afb9kmieezn/AmitEtal85.pdf?rlkey=78fckvuuvk9t3o9fbpjrmn6de&dl=1). In their framing, neurons take on activation values of either –1 ("down state") or +1 ("up state"). This has three important benefits:
8
+
9
+
- It provides a cleaner way to implement the Hebbian learning rule (i.e., without subtracting means or shifting values).
10
+
- It avoids a bias toward 0 (i.e., +1 and –1 are equally "attractive," whereas 0-valued neurons have a stronger "pull").
11
+
- The energy function (which describes the attractor dynamics of the network) can be directly mapped onto the [Ising model](https://en.wikipedia.org/wiki/Ising_model) from statistical physics.
12
+
13
+
You should start by reading [Amit et al. (1985)](https://www.dropbox.com/scl/fi/3a3adwqf70afb9kmieezn/AmitEtal85.pdf?rlkey=78fckvuuvk9t3o9fbpjrmn6de&dl=1) closely. Then implement the model in a Google Colaboratory notebook. Unless otherwise noted, all references to "the paper" refer to Amit et al. (1985).
14
+
15
+
---
10
16
11
17
## Tasks
12
18
13
19
### 1. Implement Memory Storage and Retrieval
14
-
-**Objective:** Write functions that implement the core operations of a Hopfield network:
15
-
-**Memory Storage:** Implement the Hebbian learning rule to compute the weight matrix, given a set of network configurations (memories). This is described in **Equation 1.5** of the paper:
16
20
17
-
Let \($p$\) be the number of patterns, \($N$\) the number of neurons, and \($\xi_i^\mu \in \{-1, +1\}$\) the value of neuron \($i$\) in pattern \($\mu$\).
21
+
**Objective:** Write functions that implement the core operations of a Hopfield network.
22
+
23
+
-**Memory Storage:** Implement the Hebbian learning rule to compute the weight matrix, given a set of network configurations (memories). This is described in **Equation 1.5** of the paper:
24
+
25
+
Let \( p \) be the number of patterns, \( N \) the number of neurons, and \( \xi_i^\mu \in \{-1, +1\}\) the value of neuron \( i \) in pattern \( \mu \). The synaptic coupling between neurons \( i \) and \( j \) is:
18
26
19
-
The synaptic coupling between neuron \($i$\) and \($j$\) is:
Note that the matrix is symmetric \($J_{ij} = J_{ji}$\), and there are no self-connections allowed (by definition $J_{ii} = 0$).
31
+
Note that the matrix is symmetric \(J_{ij} = J_{ji}\), and there are no self-connections by definition \(J_{ii} = 0\).
26
32
27
-
-**Memory Retrieval:** Implement the retrieval rule using **Equation (1.3)** and surrounding discussion.
33
+
-**Memory Retrieval:** Implement the retrieval rule using **Equation 1.3** and surrounding discussion. At each time step, each neuron updates according to its **local field**:
28
34
29
-
At each time step, each neuron updates according to its **local field**\($h_i$\):
35
+
$$
36
+
h_i = \sum_{j=1}^N J_{ij} S_j
37
+
$$
30
38
31
-
$$
32
-
h_i = \sum_{j=1}^N J_{ij} S_j
33
-
$$
39
+
The neuron updates its state to align with the sign of the field:
34
40
35
-
The neuron updates its state to align with the sign of the field:
Here \( S_i \in \{-1, +1\}\) is the current state of neuron \( i \).
40
46
41
-
Note that \($S_i \in \{-1, +1\}$\) represents the state of neuron \($i$\).
47
+
---
42
48
43
49
### 2. Test with a Small Network
44
50
45
-
Encode the following test memories in a small Hopfield network with \( N = 5 \) neurons:
46
-
47
-
$$
48
-
\xi^1 = [+1, -1, +1, -1, 1]
49
-
$$
50
-
$$
51
-
\xi^2 = [-1, +1, -1, +1, -1]
52
-
$$
53
-
54
-
- Store these memories using the Hebbian rule.
55
-
- Test memory retrieval by presenting the network with noisy versions of the stored patterns (e.g., flipping one neuron's sign, setting one or more activation values to 0).
56
-
- Briefly discuss the results and provide insights into how the network behaves. You might find Figure 3 from [Hopfield (1984)](https://www.dropbox.com/scl/fi/7wktieqztt60b8wyhg2au/Hopf84.pdf?rlkey=yi3baegby8x6olxznsvm8lyxz&dl=1) useful! You can either write a paragraph or two, or sketch (or code up) a diagram or figure, or some combination. In particular:
57
-
- Can you get a sense of how the network "works"-- i.e., why it stores memories?
58
-
- Why do some memories interfere and others don't? Can you build up enough of an intuition that you can manually construct memories that can vs. can't be retrieved by a toy (small) network?
59
-
- Can you build up any intuitions about what sorts of factors might affect the "capacity" of the network? (Capacity is the maximum number of memories that can be "successfully" retrieved).
51
+
Encode the following test memories in a Hopfield network with \( N = 5 \) neurons:
52
+
53
+
$$
54
+
\xi^1 = [+1, -1, +1, -1, +1] \\
55
+
\xi^2 = [-1, +1, -1, +1, -1]
56
+
$$
57
+
58
+
- Store these memories using the Hebbian rule.
59
+
- Test retrieval by presenting the network with noisy versions (e.g., flipping a sign, or setting some entries to 0).
60
+
- Briefly discuss your observations. You can write a few sentences, sketch/code a figure, or combine both.
61
+
62
+
Questions to consider:
63
+
64
+
- Can you tell how and why the network stores memories?
65
+
- Why do some memories interfere while others don’t?
66
+
- Can you construct memory sets that do or don’t work in a small network?
67
+
- What factors do you think affect the **capacity** of the network?
68
+
69
+
---
60
70
61
71
### 3. Evaluate Storage Capacity
62
-
-**Objective:** Determine how the ability to recover memories degrades as you vary:
63
-
-**Network Size:** The total number of neurons in the network.
64
-
-**Number of Stored Memories:** The number of patterns stored in the network.
72
+
73
+
**Objective:** Determine how memory recovery degrades as you vary:
74
+
75
+
-**Network Size** (number of neurons)
76
+
-**Number of Stored Memories**
77
+
78
+
To generate \( m \) memories \( \xi_1, \dots, \xi_m \) for a network of size \( N \), use:
79
+
80
+
```python
81
+
import numpy as np
82
+
xi =2* (np.random.rand(m, N) >0.5) -1
83
+
```
84
+
85
+
**Method:**
86
+
87
+
- For each configuration, run multiple trials.
88
+
- For each trial, measure whether **at least 99%** of the memory is recovered.
65
89
66
-
To generate $m$ memories \($\xi_1, ... \xi_m$\) for a network of $N$ neurons, you can use the following Python code; each row of the resulting matrix `xi` contains a single memory:
67
-
```python
90
+
**Visualization 1:**
91
+
Create a heatmap:
92
+
-\( x \)-axis: network size
93
+
-\( y \)-axis: number of stored memories
94
+
- Color: proportion of memories retrieved with ≥99% accuracy
68
95
69
-
import numpy as np
70
-
xi =2* (np.random.rand(m, N) >0.5) -1
71
-
```
96
+
**Visualization 2:**
97
+
Plot the expected number of accurately retrieved memories vs. network size:
72
98
73
-
-**Method:**
74
-
- For each configuration, run multiple trials to compute the proportion of times that at least 99% of a memory is accurately recovered.
75
-
-**Visualization 1:** Create a heatmap where the $x$-axis represents the network size, the $y$-axis represents the number of stored memories, and the color indicates the recovery accuracy. Play around with this to decide on a range of network sizes and numbers of memories that adequately illustrates the system's behavior.
76
-
-**Visualization 2:** For each network size ($x$-axis), plot the expected number of memories that can be retrieved accurately ($y$-axis). Let:
99
+
Let:
100
+
-\( P[m, N] \in [0, 1]\): proportion of \( m \) memories accurately retrieved in a network of size \( N \)
101
+
-\( \mathbb{E}[R_N]\): expected number of successfully retrieved memories
102
+
103
+
Then:
104
+
$$
105
+
\mathbb{E}[R_N] = \sum_{m=1}^{M} m \cdot P[m, N]
106
+
$$
77
107
78
-
-\($N$\): the number of neurons (network size)
79
-
-\($m$\): the number of stored memories
80
-
-\($P\left[m, N\right] \in \left[0, 1\right]$\): the empirically observed success rate (from your heatmap); i.e., the **proportion** of memories correctly retrieved with at least 99% accuracy, for a network of size \($N$\) and \($m$\) stored memories.
108
+
Where \( M \) is the maximum number of memories tested.
81
109
82
-
Then the **expected number of successfully retrieved memories**, \($\mathbb{E}\left[R_N\right]$\), for each network size \($N$\) is given by:
110
+
**Follow-Up:**
83
111
84
-
$$
85
-
\mathbb{E}\left[R_N\right] = \sum_{m=1}^{M} m \cdot P[m, N],
86
-
$$
87
-
where \($M$\) is the maximum number of stored memories you tested.
112
+
- What relationship (if any) emerges between network size and capacity?
113
+
- Can you develop rules or intuitions that help predict a network's capacity?
88
114
89
-
- Is there any systematic relationship (between network size and capacity) that emerges? Can you describe any intuitions and/or develop any "rules" that might enable you to estimate a network's capacity solely from its size?
115
+
---
90
116
91
117
### 4. Simulate Cued Recall
92
-
**Objective:** Evaluate how well the network performs associative recall when presented with only a partial input (a cue), and must recover the corresponding response.
118
+
119
+
**Objective:** Evaluate how the network performs associative recall when only a **cue** is presented.
93
120
94
121
#### Setup: A–B Pair Structure
95
122
96
-
- Each stored memory is a concatenated pair of binary patterns:
97
-
-The first half of the neurons represents the **cue**\($A$\)
98
-
-The second half represents the **response**\($B$\)
123
+
- Each memory consists of two parts:
124
+
-First half: **Cue**(\( A \))
125
+
-Second half: **Response**(\( B \))
99
126
100
-
-If the total number of neurons \($N$\) is odd:
101
-
- Let the cue occupy the first $\lfloor N/2 \rfloor$ neurons
102
-
- Let the response occupy the remaining $\lceil N/2 \rceil$ neurons
127
+
If \( N \) is odd:
128
+
- Let cue length = \(\lfloor N/2 \rfloor\)
129
+
- Let response length = \(\lceil N/2 \rceil\)
103
130
104
-
Each full pattern \($\xi^\mu \in \{-1, +1\}^N$\) is defined as:
- Repeat the simulation for multiple stored $A$–$B$ pairs
145
-
- For each network size \($N$\), compute the **expected number of correctly recalled responses**
146
-
- Plot this value as a function of \($N$\)
147
-
158
+
- Repeat across many \( A \)–\( B \) pairs
159
+
- For each network size \( N \), compute the **expected number** of correctly retrieved responses
160
+
- Plot this value as a function of \( N \)
148
161
149
162
#### Optional Extensions
150
163
151
-
- Compare performance with and without clamping the cue neurons
152
-
- Test whether cueing with partial or noisy \($A$\) patterns still leads to correct retrieval of \($B$\)
164
+
- Compare performance with and without clamping the cue
165
+
- Try cueing with noisy or partial versions of \( A \)
166
+
167
+
---
153
168
154
169
### 5. Simulate Contextual Drift
155
170
156
-
**Objective:**Explore how gradual changes in context influence which memories are retrieved. This models how temporal or environmental drift might bias recall toward memories with similar contexts.
171
+
**Objective:**Investigate how gradual changes in **context** influence which memories are recalled.
157
172
158
-
#### Setup: Item–Context Memory Representation
173
+
#### Setup: Item–Context Representation
159
174
160
-
- Use a Hopfield network with **100 neurons**.
161
-
- Each memory is a combination of:
162
-
-**Item features**: 50 neurons (first half)
163
-
-**Context features**: 50 neurons (second half)
175
+
- Use a Hopfield network with 100 neurons.
176
+
- Each memory:
177
+
-First 50 neurons: **Item**
178
+
-Last 50 neurons: **Context**
164
179
165
-
-Create a **sequence of 10 memories**\($\{\xi^1, \xi^2, \dots, \xi^{10}\}$\), each composed of:
0 commit comments