Huge variance in F1 results using the same spaCy config, data, and fixed seed=0

[bitopsy]

Hello, after running a spaCy NER with BERT using the same training/testing data, the same config, and a fixed seed = 0, I still got a huge variance in the F1, P and R scores. Is that normal? Can the experiment be controlled to have the same outcome?

Here are the F-1 results:

F-1: 0.9379195792
F-1: 0.9606539208
F-1: 0.9432156972
F-1: 0.9200165769
F-1: 0.9342884695

Variance s2 = 0.00021754027
Standard Deviation s = 0.014749247
Count n = 5
Mean = 0.93921885
Sum of Squares SS = 0.0008701611

Thank you,
Al

[adrianeboyd]

This isn't what you're asking, but be aware that for tok2vec components on GPU, some variance is expected, see #6490.

I'm not as familiar with the sources of non-determinism in transformer models. Maybe this thread would help? https://discuss.huggingface.co/t/initializing-the-weights-of-the-final-layer-of-e-g-bertfortokenclassification-with-a-manual-seed/1377/3

[bitopsy]

Oh, I see! Therefore the variability of my results is not due to my tok2vec NER transformer.listener but from something else we could not still determine. Is that right?

Could it be from my not limiting the optimization to a max number of epochs though I highly doubt it?

`[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null`

If I understand correctly, spaCy shuffles the training and testing elements but not when the seed=0, correct?

Since we're at it, I am sharing the rest of the config file just in case anything makes sense.

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = true

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.05
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
ents_per_type = null
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0


[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null


[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "./model_9c"
mixed_precision = false

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.grad_scaler_config]

[components.transformer.model.tokenizer_config]
use_fast = true

[components.transformer.model.transformer_config]

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null


[initialize.components]

[initialize.tokenizer]

Thank you

[adrianeboyd]

I suspect you're in an area where the non-determinism is coming from pytorch or transformers instead of spacy directly. Did you try adding the steps from the huggingface forum post that I linked above? I think you could try it out by running it as custom code in spacy train ... --code code.py.

[bitopsy]

I have only added torch.manual_seed() to my spacy code that was config free but haven't gone to the extent of fixing it in every possible way. Also good to know about the --code parameter for extra code that is very helpful and I will test that ASAP with the function below.

`def set_seed(seed: Optional[int] = None):
    """Set all seeds to make results reproducible (deterministic mode).
       When seed is None, disables deterministic mode.
    :param seed: an integer to your choosing
    """
    if seed is not None:
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
        np.random.seed(seed)
        random.seed(seed)
        os.environ['PYTHONHASHSEED'] = str(seed)`

Many thanks!

[bitopsy]

The setting of seeds did huge improvement to the variation, however, I wonder if the small variations are due to the shuffling of training and testing elements or some other variants?

default	fixing all seeds
93.37931034482759APR: 95.73027497581872; 94.102990033222: 91.41365572193983	92.8081251715619 93.70860927152317 94.38838977194195 92.18706633361089

[adrianeboyd]

The built-in shuffling in the training loop should be deterministic, so I still think it's more likely to be something in pytorch.

You can try training a tok2vec-based NER model on CPU to see that the NER model itself is not the source of the variation. Training on GPU will still be non-deterministic here, though.