Code to reproduce Fig. 1

[TrentBrick]

Hi I am unable to reproduce Fig 1 from your paper. I am getting cosine similarities for present vectors that are ~0.25 in magnitude, not 1.0 as displayed (when using the complex projection and then the exact inverse).

Thank you,
Trenton

[EdwardRaff]

Sorry this is a super-belated reply. Did you eventually solve this or was it a mystery?

[danmichaeljones]

Hey @EdwardRaff - sorry to resurrect this old thread, but I'm also having a similar issue. In particular, I can't reproduce the increased capacity that the complex projection step $\pi(x)$ is supposed to bring. If possible, would you please be able to look at how I'm using your method, to see if I'm doing something silly here?

I've copied these numpy versions (cc, np_inv, np_appx_inv, npcomplexMagProj) of the functions from the file in this repo here and am using as follows:

# Sample random normal vectors N(0, I.d^-1)
# Sqrt because scale is std, not variance
def random_normal(vec_dim: int):
    return np.random.normal(size=vec_dim, scale=1 / np.sqrt(vec_dim))

# Do unbinding by combining functions from above.
# This uses the faster approximate inverse, but results
# are the same using the FFT inverse method too
def np_unbind(x, y):
    #return cc(x, np_inv(y))
    return cc(x, np_appx_inv(y))

# ...

key_1 = npcomplexMagProj(random_normal(1024))
val_1 = npcomplexMagProj(random_normal(1024))
key_2 = npcomplexMagProj(random_normal(1024))
val_2 = npcomplexMagProj(random_normal(1024))

combined = cc(key_1, val_1) + cc(key_2, val_2)

> val_1 @ np_unbind(combined, key_1)  # Is ~1 as expected

However, combining many more pairs together (as in your figure 1), I get the following results (the full script to reproduce this figure is in a gist here):

What I'm seeing is that

• I get the expected 0 / 1 scores on average
• I see increased capacity (i.e., the distributions of scores become more well separated) as the size of the vectors increases

However, I cannot see any difference between the naive version and the one with your complex projection, as I'd expect from fig. 1 in your paper.

Thanks for making your code public. Your method looks very promising for my use case and so I'm hoping there's something silly I'm doing that you'd be able to spot! Thanks again 🙏

[EdwardRaff]

I will try and look at this soon, but I'm about to travel for work and have a new tiny human at home. You may want to look into https://github.com/MahmudulAlam/Holographic-Reduced-Representations as well, where @rea1mma was continuing this research direction and organized a lot of the base code.