LSTM layers

[ldesmet]

Hello all,

First I'd like to thank you for the great job you provided.

Question, how hard it is to implement a Lipschitz LSTM version and make it available in your library ?
We work on time series and we think this mathematical Lipschitz guarantee might be interesting in some embeded products.

Many thanks,

[franckma31]

Dear Ldesmet,
Thank you for your interest in the deel-lip library.
Your question regarding robustness in time series is an excellent one.
Regarding LSTMs, while the recurrent matrix is the identity (and thus Lipschitz), the involvement of gating mechanisms makes it difficult to constrain the overall Lipschitz constant.
However, there is another class of recurrent models with more favorable properties that can be built using deel-lip: Orthogonal RNNs (ORNNs) [1], which use orthogonal recurrent matrices. These models are also known to learn longer-term dependencies [2] more effectively than LSTMs.
In two of our papers [3,4], we explore quantization and even binarization of ORNNs. Our latest work [4] will be presented at ICLR this year, and the code is available here:
https://github.com/deel-ai-papers/hadamRNN.
This implementation is based on the PyTorch framework and leverages our PyTorch Lipschitz library, deel-torchlip:
https://github.com/deel-ai/deel-torchlip

If needed, we might also have some TensorFlow code for constructing ORNNs using deel-lip (with or without quantization) and could share it.

[1] M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pages 1120–1128. PMLR, 2016.
[2] B. Kiani, R. Balestriero, Y. LeCun, and S. Lloyd. projUNN: efficient method for training deep
networks with unitary matrices. Advances in Neural Information Processing Systems, 35, 2022.
[3] A. Foucault, F. Mamalet, and F. Malgouyres. Quantized approximately orthogonal recurrent neural networks. arXiv preprint arXiv:2402.04012, 2024
[4] A. Foucault, F. Mamalet, and F. Malgouyres. HadamRNN: Binary and Sparse Ternary Orthogonal RNNs, ICLR 2025