Jump Self-attention: Capturing High-order Statistics in Transformers (NIPS 2022)

Requirements

• Python 3.7
• transformers == 4.0.0
• torch == 1.10.1
• datasets == 1.15.1
• huggingface-hub == 0.1.2
• scipy == 1.6.2
• scikit_learn == 0.24.1

Dependencies can be installed using the following command:

pip install -r requirements.txt

Data

GLUE_Datasets

Usage

python run_glue.py