Best alignment mapping for English

[meryemmhamdi1]

Hi,

Thank you for this interesting work. I am currently working on extending this approach on top of mBERT and would need to generate English mapping from scratch. Did you learn the W matrix for English by aligning English to itself using MUSE? Wouldn't that be redundant?

Thanks,

[TalSchuster]

Hi,

For English we used the identity matrix (divided by the empirical norm). You can checkgen_anchors_bert.py for computing it for BERT. Actually, I ran it for multilingual BERT for a few languages and can share the files with you if you're interested.

[meryemmhamdi1]

Hi,

Thank you for your prompt response! I would be interested in comparing the procedure used with mine also for new languages such as Arabic. Did you use the same evaluation set of Wikipedia dumps to generate anchors? I am also working on extending biaffine parser with alignment matrices on top of bert token embedders. I would probably push a pull request for that on AllenNLP. If you can share a link for alignment matrices for the languages you have now, that would be appreciated.

Thanks,

[TalSchuster]

Sounds great.
This is a mapping Spanish to English for the last layer.
Actually, for multilingual BERT the norm doesn't seem to vary (probably because it was jointly trained) so you can just use the identity matrix for English.