attention mask

[cfeng16]

Great work! I am very curious about the attention mask mechanism adopted for masked tokens during training. After looking into the codebase, I think self full-attention is adopted (even for masked tokens)? Please correct me if it's wrong. Thanks!

[nieshenx]

Yes, we use full-attention for all tokens.

[cfeng16]

Thanks, I have a follow-up (might be dumb) question. For padding tokens, can other tokens attend to them or not?

[nieshenx]

Yes, for our model, the other tokens can attend to the padding tokens.