Behavior of se_atten with respect to atten_layer

[ald3hyde]

Dear all,
You previously(#2638 ) told me that atten_layer=0 is also useful and fast for multi-component models, but I would like to know how atten_layer behaves. It doesn't seem to have a significant impact on the accuracy of the model, does it?

[wanghan-iapcm]

The attention layers are critical for the generalizability of the model . Please visit https://arxiv.org/abs/2208.08236 and check the ablation study in the updated version of the dpa-1 manuscript after Mon, 18 Sep 2023 00:00:00 GMT