why previously updated knowledge covariance is not added in AlphaEdit implementation?

[jasonrichdarmawan]

AnyEdit/AlphaEdit/AlphaEdit_main.py

Lines 119 to 121 in ac8e652

	upd_matrix = torch.linalg.solve(
	P[i,:,:].cuda() @ (layer_ks @ layer_ks.T) + hparams.L2*torch.eye(layer_ks.shape[0], dtype=torch.float,device="cuda"), P[i,:,:].cuda() @ layer_ks @ resid
	)

This is different to the original AlphaEdit implementation

https://github.com/jianghoucheng/AlphaEdit/blob/75e117bcbbc58b1fd28df41e1cbb2fa3d25e3ef1/AlphaEdit/AlphaEdit_main.py#L130-L132

[jianghoucheng]

Sorry for the misunderstanding. In AlphaEdit, "kp" refers to the knowledge preserved from past edits in a continuous editing scenario, whereas AnyEdit does not consider such a continuous editing context. In AlphaEdit+, there is also "kp"， but here it does not refer to retaining past knowledge in continuous editing. Instead, it means preserving the representations of other edited token positions within the same piece of knowledge during autoregressive editing of long texts. Although both are written as "KP," their meanings are different.

[jasonrichdarmawan]

Hi, thank you for the reply. I really appreciate it!

In AlphaEdit, "kp" refers to the knowledge preserved from past edits in a continuous editing scenario, whereas AnyEdit does not consider such a continuous editing context

Is this because of the unstructured knowledge? That K_p is removed from AlphaEdit/AlphaEdit_main.py implementation?

Btw, shoudln't @ resid be @ resid.T?

AnyEdit/AlphaEdit/AlphaEdit_main.py

Lines 119 to 121 in ac8e652

	upd_matrix = torch.linalg.solve(
	P[i,:,:].cuda() @ (layer_ks @ layer_ks.T) + hparams.L2*torch.eye(layer_ks.shape[0], dtype=torch.float,device="cuda"), P[i,:,:].cuda() @ layer_ks @ resid
	)

[jianghoucheng]

Yes. Just to clarify, our character design simply did not take the scenario of continuous editing into account. Our focus was solely on the ability to effectively edit knowledge within long texts.

[jasonrichdarmawan]

Yes. Just to clarify, our character design simply did not take the scenario of continuous editing into account. Our focus was solely on the ability to effectively edit knowledge within long texts.

Well noted. Thank you!

I liked your AlphaEdit and AnyEdit idea, would you mind to mentor?

[jianghoucheng]

You flatter me. I'm just a student as well, but I'd be happy to discuss anytime.

[jasonrichdarmawan]

Thank you! I will send to your email a PPT of my preliminary research next week