Error when converting NMT model with ALiBi or RoPe

[randomicity]

Hello, thank you for a great project!

I am getting this error when using ALiBi or RoPe positional encoding in a tranformer NMT model from OpenNMT-py:

KeyError: 'encoder.embeddings.make_embedding.pe.pe'

Absolute and relative positional encodings are working with the converter script.

[minhthuc2502]

Can you provide more in detail how to run the converter?

[randomicity]

ct2-opennmt-py-converter --model_path test_step_100.pt --output_dir test
Traceback (most recent call last):
File "/home/username/anaconda3/envs/translator/bin/ct2-opennmt-py-converter", line 8, in
sys.exit(main())
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/opennmt_py.py", line 355, in main
OpenNMTPyConverter(args.model_path).convert_from_args(args)
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/converter.py", line 50, in convert_from_args
return self.convert(
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/converter.py", line 89, in convert
model_spec = self._load()
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/opennmt_py.py", line 200, in _load
return _get_model_spec_seq2seq(
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/opennmt_py.py", line 90, in _get_model_spec_seq2seq
set_transformer_spec(model_spec, variables)
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/opennmt_py.py", line 210, in set_transformer_spec
set_transformer_encoder(spec.encoder, variables)
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/opennmt_py.py", line 215, in set_transformer_encoder
set_input_layers(spec, variables, "encoder")
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/opennmt_py.py", line 241, in set_input_layers
set_position_encodings(
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/opennmt_py.py", line 341, in set_position_encodings
spec.encodings = _get_variable(variables, "%s.pe" % scope).squeeze()
File "/home/username/anaconda3/envs/translator/lib/python3.10/site-packages/ctranslate2/converters/opennmt_py.py", line 345, in _get_variable
return variables[name]
KeyError: 'encoder.embeddings.make_embedding.pe.pe'

[LynxPDA]

I faced the same problem. Unfortunately python/ctranslate2/converters/opennmt_py.py currently only supports ALiBi or RoPe for decoder_type == "transformer_lm" (LLMs) and does not support for seq2seq.

Also, unfortunately, there is no support for activating gated-gelu (it is added in literally two lines, but ALiBi or RoPe is much more difficult to add in my opinion).

In my experiments, seq2seq with a combination of RoPe and gated-gelu converges much faster when trained via OpenNMT-py. And the learning rate (tok/sec) is also 10% higher than that of relative positional embeddings.

If someone can add python/ctranslate2/converters/opennmt_py.py for seq2seq ALiBi, RoPe and gated-gelu to the converter, then this will make it possible to speed up the inference of much higher quality advanced translator models.

[vince62s]

It should not be that difficult to add rope / alibi to the encoder. But do you have numbers to support the benefit?

If you use "silu" it uses gated ffn, it should be suported out of the box, no?
EDIT: it's missing in the encoder too.

I'll add them

[LynxPDA]

Here are more specific numbers and observations:

1. Ultimately, the models with RPE (relative positional embeddings) and RoPE converged.
2. However, as I said earlier, the tok/s speed of both training and validation is higher for the model with RoPE.
3. But the model with GELU activation turned out to be noticeably worse than with gated-gelu and I stopped its training ahead of schedule.
4. The experiment with the RPE (relative positional embeddings) model is not very clean, since its training did not start from scratch, but from the 1000 step.

Here is a description of the models on screenshots:

1. All models were trained on the same dataset.
2. The effective batch size was the same.
3. The learning rate and all other parameters were the same, only the activation functions and the type of positional embeddings changed.
4. The models themselves:
   • Apr-21 - (red color) - RoPE + gated-gelu
   • Apr-23 (yellow color) - RoPE + gelu
   • Apr-26 (purple color) - RPE (relative positional embeddings) + gated-gelu

progress acc

progress speed (tok/s)

valid speed (tok/s)

Based on the above experiments, I think the optimal solution would be the combination of RoPE + gated-gelu, in terms of convergence speed and learning and operating speed (tok/s).

Also, as far as I understand, RoPE embeddings and GLU activation options are now actually a standard and are used in the most modern models.

Edited: fix mistakes, and I’ll add, I also tried silu activation, but it turned out worse than gelu or relu, I assume there is also a beta parameter that needs to be configured and silu in this case is more suitable for converting already ordinary LLMs with silu type activation (SwiGLU).
And there seems to be some confusion in the names. It seems gated silu/swish should be called SwiGLU, if I'm not mistaken.

[vince62s]

there is no beta for silu, there is rope theta but should not impact the perf in nmt since it is length related.
gated-gelu adds an extra-layer, hence more parameters so I am not sure it is 100% comparable vs non gated gelu or relu

[LynxPDA]

there is no beta for silu

I apologize here, I got a little mixed up. Beta meant the coefficient of the Swish function. When beta = 1 it will be equal to the Silu function.

gated-gelu adds an extra-layer, hence more parameters so I am not sure it is 100% comparable vs non gated gelu or relu

Everything is correct here, GLU increases the number of parameters of the FF layer by 50%. I took this into account and reduced the size of the FF layer by 50%.
The total number of parameters in all my models in comparison is the same and equals 457M parameters.

I relied on this paper: GLU Variants Improve Transformer

[vince62s]

can you test to convert your onmt-py model with #1687 (the rope one with gated-gelu) and tell me if it works ?

[vince62s]

And there seems to be some confusion in the names. It seems gated silu/swish should be called SwiGLU, if I'm not mistaken.

so did you try "silu" of onmt-py repo (hence gated silu) or basic silu ?

[LynxPDA]

can you test to convert your onmt-py model with #1687 (the rope one with gated-gelu) and tell me if it works ?

Converting to ctranslate2
Traceback (most recent call last):
  File "/mnt/DeepLearning/Locomotive/venv/bin/ct2-opennmt-py-converter", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/opennmt_py.py", line 375, in main
    OpenNMTPyConverter(args.model_path).convert_from_args(args)
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/converter.py", line 50, in convert_from_args
    return self.convert(
           ^^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/converter.py", line 89, in convert
    model_spec = self._load()
                 ^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/opennmt_py.py", line 201, in _load
    check_opt(checkpoint["opt"], num_source_embeddings=len(src_vocabs))
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/opennmt_py.py", line 57, in check_opt
    check.validate()
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/utils.py", line 106, in validate
    raise_unsupported(self._unsupported_reasons)
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/utils.py", line 93, in raise_unsupported
    raise ValueError(message)
ValueError: The model you are trying to convert is not supported by CTranslate2. We identified the following reasons:

- Option --pos_ffn_activation_fn gated-gelu is not supported (supported activations are: gelu, fast_gelu, relu, silu)

I tried to convert the micro model, but got an error, maybe I need to add gated-gelu to SUPPORTED_ACTIVATIONS:

_SUPPORTED_ACTIVATIONS = {
    "gelu": common_spec.Activation.GELU,
    "fast_gelu": common_spec.Activation.GELUTanh,
    "relu": common_spec.Activation.RELU,
    "silu": common_spec.Activation.SWISH,
    "gated-gelu": common_spec.Activation.GELU,
}

Now a new error has appeared:

Traceback (most recent call last):
  File "/mnt/DeepLearning/Locomotive/venv/bin/ct2-opennmt-py-converter", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/opennmt_py.py", line 376, in main
    OpenNMTPyConverter(args.model_path).convert_from_args(args)
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/converter.py", line 50, in convert_from_args
    return self.convert(
           ^^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/converter.py", line 89, in convert
    model_spec = self._load()
                 ^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/opennmt_py.py", line 221, in _load
    return _get_model_spec_seq2seq(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/converters/opennmt_py.py", line 89, in _get_model_spec_seq2seq
    model_spec = transformer_spec.TransformerSpec.from_config(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/specs/transformer_spec.py", line 481, in from_config
    encoder = TransformerEncoderSpec(
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/specs/model_spec.py", line 84, in __call__
    instance = super().__call__(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/specs/transformer_spec.py", line 107, in __init__
    self.layer = [
                 ^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/specs/transformer_spec.py", line 108, in <listcomp>
    TransformerEncoderLayerSpec(
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/specs/model_spec.py", line 84, in __call__
    instance = super().__call__(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/specs/transformer_spec.py", line 294, in __init__
    self.self_attention = attention_spec.MultiHeadAttentionSpec(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/specs/model_spec.py", line 84, in __call__
    instance = super().__call__(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: MultiHeadAttentionSpec.__init__() got an unexpected keyword argument 'head_dim'

[LynxPDA]

And there seems to be some confusion in the names. It seems gated silu/swish should be called SwiGLU, if I'm not mistaken.

so did you try "silu" of onmt-py repo (hence gated silu) or basic silu ?

I try "silu" of onmt-py repo (hence gated silu)

[vince62s]

TypeError: MultiHeadAttentionSpec.init() got an unexpected keyword argument 'head_dim'

this one does not make sense. check your /mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/specs/attention_spec.py file
and let me know it's up to date with head_dim in the arguments.

[LynxPDA]

TypeError: MultiHeadAttentionSpec.init() got an unexpected keyword argument 'head_dim'

this one does not make sense. check your /mnt/DeepLearning/Locomotive/venv/lib/python3.11/site-packages/ctranslate2/specs/attention_spec.py file and let me know it's up to date with head_dim in the arguments.

I think I got it, I just added two files from the PR to my venv, but my version of Ctranslate is pretty far behind:

"""Version information."""

__version__ = "3.20.0"

I updated Ctranslate and the conversion went without errors.
The only thing is that I saved the changes by adding the line "gated-gelu" to _SUPPORTED_ACTIVATIONS: common_spec.Activation.GELU, without this it still gave the error:
- Option --pos_ffn_activation_fn gated-gelu is not supported (supported activations are: gelu, fast_gelu, relu, silu)

However, now I cannot yet confirm the functionality of the full-fledged converted model, since I only tested it on a “100-step toy model.” I removed the weights of the previous model with RoPE + gated-gelu and put the model with RPE + gated-gelu (purple in the graphs above) for long-term training, since I did not think that this function would be implemented so quickly, thank you!
I will train the next model with RoPE!)