Inconsistent model training across different computers

[GK-Matias]

Hi,

I have an issue with consistency when training a NER-model. This is the code I have been using:

import spacy
from typing import List, Dict,Tuple
from spacy.util import minibatch
from spacy.training import Example
import numpy
import random
import string

### Remove randomness from spacy ###
numpy.random.seed(0)
spacy.util.fix_random_seed(0)
### ---------------------------- ###

train_data = [
    ('1 2 ETG TILLUFT VENT', {'entities': [(0, 1, 'D'), (2, 3, 'D'), (4, 7, 'D'), (8, 15, 'F'), (16, 20, 'C')]}),
    ('1 ETG BYGG 3 1 11 V K WC', {'entities': [(0, 1, 'D'), (2, 5, 'D'), (6, 10, 'A'), (11, 12, 'A'), (13, 14, 'D'), (15, 17, 'E'), (18, 19, 'E'), (20, 21, 'E'), (22, 24, 'E')]}),
    ('1 ETG BYGG 3 1 13 GARDEROBE', {'entities': [(0, 1, 'D'), (2, 5, 'D'), (6, 10, 'A'), (11, 12, 'A'), (13, 14, 'D'), (15, 17, 'E'), (18, 27, 'E')]}),
    ('1 ETG BYGG 3 1 14 KORRobjectIdOR', {'entities': [(0, 1, 'D'), (2, 5, 'D'), (6, 10, 'A'), (11, 12, 'A'), (13, 14, 'D'), (15, 17, 'E'), (18, 32, 'E')]}),
    ('1 ETG BYGG 3 1 18 FELLESROM', {'entities': [(0, 1, 'D'), (2, 5, 'D'), (6, 10, 'A'), (11, 12, 'A'), (13, 14, 'D'), (15, 17, 'E'), (18, 27, 'E')]}),
    ('1 ETG GYMSAL', {'entities': [(0, 1, 'D'), (2, 5, 'D'), (6, 12, 'E')]}),
    ('1 ETG HALL', {'entities': [(0, 1, 'D'), (2, 5, 'D'), (6, 10, 'E')]}),
    ('1 ETG SCENE', {'entities': [(0, 1, 'D'), (2, 5, 'D'), (6, 11, 'E')]}),
    ('1 etg 107 WC jenter', {'entities': [(0, 1, 'D'), (2, 5, 'D'), (6, 9, 'E'), (10, 12, 'E'), (13, 19, 'E')]}),
    ('1002 POS MV', {'entities': [(0, 4, 'E'), (5, 8, 'G'), (9, 11, 'G')]}),
    ('1002 RYc PV', {'entities': [(0, 4, 'E'), (5, 8, 'G'), (9, 11, 'G')]}),
    ('1002 RYc SP', {'entities': [(0, 4, 'E'), (5, 8, 'G'), (9, 11, 'G')]}),
    ('1002 RYt SPD', {'entities': [(0, 4, 'E'), (5, 8, 'G'), (9, 12, 'G')]}),
    ('1002 RYt SPR', {'entities': [(0, 4, 'E'), (5, 8, 'G'), (9, 12, 'G')]}),
    ('1002 SR401 POS MV', {'entities': [(0, 4, 'E'), (5, 10, 'F'), (11, 14, 'G'), (15, 17, 'G')]}),
    ('1002 SR401 VOL MV', {'entities': [(0, 4, 'E'), (5, 10, 'F'), (11, 14, 'G'), (15, 17, 'G')]}),
    ('1002 SR501 POS C', {'entities': [(0, 4, 'E'), (5, 10, 'F'), (11, 14, 'G'), (15, 16, 'G')]}),
    ('1002 SR501 POS MV', {'entities': [(0, 4, 'E'), (5, 10, 'F'), (11, 14, 'G'), (15, 17, 'G')]}),
    ('1021 SR401 POS C', {'entities': [(0, 4, 'E'), (5, 10, 'F'), (11, 14, 'G'), (15, 16, 'G')]}),
    ('1021 SR401 POS MV', {'entities': [(0, 4, 'E'), (5, 10, 'F'), (11, 14, 'G'), (15, 17, 'G')]})
]
max_itn=1000
batchsize = 8
dropout=0.35
labels_array = [
    "A",
    "B",
    "C",
    "D",
    "E",
    "F",
    "G",
]

nlp: spacy.lang.en.English = spacy.blank("en")
nlp.add_pipe("ner", last=True)
ner = nlp.get_pipe("ner")
for label in labels_array:
    ner.add_label(label)  # type: ignore
optimizer = nlp.initialize()
pipe_exceptions: List[str] = ["ner"]
other_pipes: List[str] = [
    pipe for pipe in nlp.pipe_names if pipe not in pipe_exceptions
]

# Begin training by disabling other pipeline components
with nlp.select_pipes(disable=[*other_pipes]):
    for itn in range(max_itn):
        losses: Dict[str, float] = {}
        # batch up the examples using spaCy's minibatch
        batch: List[Tuple[str, Dict[str, List[Tuple[int, int, str]]]]]
        for batch in minibatch(train_data, size=batchsize):
            for text, annotations in batch:
                doc = nlp.make_doc(text)
                example = Example.from_dict(doc, annotations)
                nlp.update([example], drop=dropout, sgd=optimizer, losses=losses)


for i in range(500):
    random_strings = []
    for j in range(1,i%5+2):
        random_string=''.join(random.Random(i*10+j).choices(string.ascii_lowercase, k=5))
        random_strings.append(random_string)
    test_data=' '.join(random_strings)
    doc: spacy.tokens.doc.Doc = nlp(test_data)
    result_dictionary: Dict[str, str] = {}
    for ent in doc.ents:
        result_dictionary[str(ent.label_)] = str(ent)
    print(test_data)
    print(result_dictionary)

The above code produces identical predictions on the 500 randomly produced strings when the code is run multiple times on the same computer. However, if this code is run on two separate computers, the results are different (on approximately every second prediction). I have run the tests inside a Docker container so the environments on the two computers are identical.

Is this an expected behavior or a bug?

Python 3.8.10
Spacy 3.2.0
Numpy 1.23.5
Pandas: 1.5.2

[polm]

Let me link in the reproducibility FAQ...

In general, it's hard to ensure exact reproducibility between two different machines, though if you're using Docker to make sure everything is exactly the same I would expect consistent results. Is there any difference in the hardware or configuration of the two host machines? Are you using a GPU?

[GK-Matias]

Thanks for the answer @adrianeboyd,

when adding BLIS_ARCH_DEBUG=1, the intel computer is outputting:
libblis: selecting sub-configuration 'skx'.
whereas the AMD computer outputs:
libblis: selecting sub-configuration 'zen2'.

After installing blis 0.9.1 and running BLIS_ARCH_DEBUG=1 BLIS_ARCH_TYPE=3 python train_test_ner_model.py, the difference between the inferences decreased but it was still present in approximately five percent of the cases.

I even tested with a third computer with the following specs:

cpu:                                                            
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2750 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2600 MHz
                       Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 1622 MHz

and

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"

After installing blis 0.9.1 and running BLIS_ARCH_DEBUG=1 BLIS_ARCH_TYPE=3 python train_test_ner_model.py, on the new computer, the inference was identical with the AMD computer.

[adrianeboyd]

I can confirm on my end that using BLIS_ARCH_TYPE=3 on two similar CPUs is not enough to get identical model output. There may still be some differences in BLIS that are leading to minor differences.

I'm not sure how realistic your example script is intended to be for your final training setup, but I would strongly suggest using the spacy train CLI or calling the built-in training methods from python with a default config from spacy init config -p ner. It would add batching, early stopping, and other default training settings that should work better in general for the final trained model. My guess is with better batching and early stopping that the final predictions for the random texts will be more consistent across runs because you're avoiding some extreme overfitting.

The ner_demo project is a good starting point. You just need to save your example training data with srsly.write_json and it should be in the right format for the project.

[GK-Matias]

Thank you for looking into this issue! I will have a look and see if I can rewrite my code according to your recommendations!

[GK-Matias]

Hi again, I have now done some testing according to your recommendations. I cloned the ner_demo project you referred to python -m spacy project clone pipelines/ner_demo, replaced the content in dev.json and train.json with the data previously mentioned in this thread. This produced identical results between computers. The size of the training data mentioned in the original post, however, was very small and the inconsistency in the original case was produced by a very high number of epochs.

Next, I anonymized my complete dataset and inserted it in the train.json and dev.json files and followed the procedure outlined in this repository (using commands from ner_demo project): https://github.com/GK-Matias/spacy-cli-reproducibility. With this procedure I still get inconsistent results between computers. It seems like the reproducibility issue still is there when using the spacy train CLI method that you proposed. Is this correct or have I missed some details?

[adrianeboyd]

I wasn't trying to suggest that using spacy train will give you reproducible results across machines, but that using sensible training settings (mainly not too much overfitting) will probably lead to less variation across training runs.

What is realistic to expect from training thinc (not necessarily pytorch) models with spacy:

• training on CPU on the same machine is reproducible
• training on CPU on different machines is not reproducible (but the variation is usually small/acceptable)
• training on GPU is not reproducible (but the variation is usually small/acceptable)
• inference (predictions) for a model are the same across different machines/platforms in identical environments

I am not sure whether it's currently possible to have deterministic training for pytorch/transformer models. If it's possible I would assume there's a large speed hit.

In particular, if you find cases where inference is not reproducible across platforms given the exact same software versions, we'd be interested in hearing about it.

The last thing I can suggest for you to try on CPU is using "generic" kernels with BLIS. This will probably be painfully slow, though, and is not recommended in general.