Same seed trainings produce different results

[LucasDedieu]

Description

I launched two trainings on GPU with the same config file and seed, but the results are not the same

How to reproduce the bug

Launch two trainings with the same config
python -m edsnlp.train --config config.yml --seed 42

Your Environment

• Operating System: Ubuntu 22.04.3 LTS
• Python Version Used: 3.10
• spaCy Version Used: 3.8.6
• EDS-NLP Version Used: 0.18

[percevalw]

As discussed in person, this difference is likely due to the non-deterministic behavior of some PyTorch operations on CUDA.

Functions such as torch.nn.functional.embedding_bag, torch.index_add, or torch.index_reduce can introduce small variations because summations and averages are performed in parallel on the GPU, which means the order of floating-point operations is not guaranteed to be the same each time. In theory this should not matter, but in practice it can lead to small numerical differences due to floating-point precision.

There is no simple fix for this. These functions are used to efficiently aggregate wordpiece embeddings into word embeddings without padding. PyTorch does not currently provide a deterministic alternative, except by explicitly padding tensors, for example using wordpiece_embeddings.refold("word", "token").sum(-1) (as these component use https://github.com/aphp/foldedtensor). This approach would increase memory usage and could slow down computations, since most words have only one or two wordpieces while a few have more. For this reason it is probably not something that we will address urgently.

The main issue is not the small numeric imprecision itself but the variability of the results. If the computation were deterministic, changing the seed would likely still produce a similar level of variation, which suggests the model itself is somewhat unstable. One possible approach would be to average metrics across several runs, although that can be time-consuming, but would better reflect the true estimate and variance of the results.

As mentioned during discussion, if these differences appear drastic when using macro-averaged metrics, this might be because such metrics are more sensitive to rare classes. In that case, micro-averaged metrics can sometimes give a more stable view of performance.