Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking

This repository is the official implementation for the paper "Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking" published in Findings of ACL 2024.

Usage

Step 1: Set up the environment

We recommend using Conda to manage virtual environments, and we use Python version 3.8.12.

conda create -n otmel python==3.8.12
conda activate otmel
pip install -r requirements.txt

Please install the specified versions of Python libraries according to the requirements.txt file.

Note that the versions of PyTorch, Transformers, and PyTorch Lightning may have a slight impact on the results. To fully reproduce the results of the paper, we recommend installing the specified versions.

Step 2: Download the data

You may download WikiMEL and RichpediaMEL from https://github.com/seukgcode/MELBench and WikiDiverse from https://github.com/wangxw5/wikiDiverse.

Or download our cleaned data WikiMEL, RichpediaMEL, WikiDiverse (Password: kdd2023).

Step 3: Modify the data path

Please modify the configuration files under the "config" directory (including the YAML files for all 3 datasets) and replace YOUR_PATH in the data field of each configuration file with the path to your corresponding dataset.

NOTE: Due to the uploaded training files of RichpediaMEL, mention images are stored in the folder mention_images. You need to modify the mention_img_folder in the richpediamel.yaml config file or rename the mention_images folder to mention_image. (Thank Zhiwei Hu for bringing up this issue)

Step 4: Start the training

Now you can execute bash run.sh <gpu_id> <dataset_name> to begin the training.

bash run.sh 0 wikimel       # for WikiMEL
bash run.sh 0 richpediamel  # for RichpediaMEL
bash run.sh 0 wikidiverse   # for WikiDiverse

Code Structure

The code is organized as follows:

├── codes
│   ├── main.py
│   ├── model
│   │   ├── lightning_ot.py
│   │   └── modeling_ot.py
│   └── utils
│       ├── dataset.py
│       └── functions.py
├── config
│   ├── richpediamel.yaml
│   ├── wikidiverse.yaml
│   └── wikimel.yaml
├── readme.md
├── requirements.txt
└── run.sh

Acknowledgements

We acknowledge the outstanding open-source contributions from MIMIC and UniOT-for-UniDA.

Citation

If you find this project useful in your research, please cite the following paper:

@article{zhang2024optimal,
  title={Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking},
  author={Zhang, Zefeng and Sheng, Jiawei and Zhang, Chuang and Liang, Yunzhi and Zhang, Wenyuan and Wang, Siqi and Liu, Tingwen},
  journal={arXiv preprint arXiv:2406.01934},
  year={2024}
}

Contact Information

If you have any questions, please contact zhangzef1999@163.com.