WIP : First implementation of sliding window local sparse attention.#951

Open

natsegal wants to merge 13 commits intoOpenNMT:masterfrom

natsegal:sparse_attention

Commits on Jun 20, 2022

First implementation of sliding window local sparse attention.
Natalia Segal
committed
Apply flake8
Natalia Segal
committed

Commits on Jun 21, 2022

Use explicit paddings instead of tf.one_hot
Natalia Segal
committed
Fix docstrings and add ETC paper reference
Natalia Segal
committed
Combine chunks in a function, override queries, keys, and values.
Natalia Segal
committed

Commits on Aug 26, 2022

Add global attention
Natalia Segal
committed

Commits on Aug 31, 2022

Use itertools.product for test parameter combinations
Natalia Segal
committed

Commits on Sep 12, 2022

Fix conditions for using global attention
Natalia Segal
committed

Commits on Sep 13, 2022

Fix global attention mask
Natalia Segal
committed

Commits on Sep 20, 2022

Apply black
Natalia Segal
committed

Commits on May 15, 2023

Fix graph mode execution for sparse attention
Natalia Segal
committed
Merge master
Natalia Segal
committed

Commits on May 26, 2023

Move chunk functions and add conditions for short input lengths.
Natalia Segal
committed