Can UPGrad aggregator support gradient normalization?

[suki00621]

Hi, thanks for your great work!
I’m wondering whether the UPGrad aggregator can incorporate a gradient normalization step. Specifically, is it possible to normalize the task gradients to unit norm(rescale task gradients to have equal magnitudes before aggregation) before aggregation? If yes, could you please suggest how to implement this in the current code framework?

[PierreQuinton]

Hi, this is definitely possible. You simply need to define your own subclass of Aggregator which should provide a forward method that maps a matrix to the aggregation. In your case, you would normalize the raws of the matrix and then output the output of UPGrad() on the normalized matrix. Then you can provide this aggregator to backward or mtl_backward.

Let me know if this works. On a side note, I think this might be a bad idea (at least in theory) because gradient normalization will typically lead to weight divergence and so extreme over-fit.