what is gradient alignment assumption

[psychocosine]

请问论文里面提到的gradient alignment assumption $\nabla \mathcal{L}_t(W_0+\alpha_t\Delta_t) \propto \Delta_t$ 出自哪里呢？这个式子的意思是梯度和参数在同一个空间吗

[juzhengz]

In the LoRI paper, we introduce this as a heuristic assumption used only for our proof sketch. It is not a standard, named assumption borrowed from a specific prior work.

The meaning of this assumption is, at a task’s solution $W_0+\alpha_t\Delta_t$, the gradient is (approximately) aligned with the task update $\Delta_t$ (i.e., points in the same direction up to a scalar). This alignment is what allows us to proceed with the subsequent steps of the argument. Regarding "same space", gradients always live in the same vector space as the parameters by definition. Our assumption asserts directional alignment between the gradient and the task update $\Delta_t$.