Permission Denied while training on Windows

[githubbar]

running: python -m spacy train config.cfg --output ./output --paths.train ./train.spacy --paths.dev ./train.spacy --gpu-id 0

I am getting and error while deleting ner directory

PermissionError: [WinError 5] Access is denied: 'output\model-best\ner'

I am admin and all the other directories/files are created w/o a problem. I saw another thread about "windows sync" locking the files, but it's not the case here. Full log below:

[+] Created output directory: output
[i] Saving to output directory: output
[i] Using GPU: 0

=========================== Initializing pipeline ===========================
[+] Initialized pipeline

============================= Training pipeline =============================
[i] Pipeline: ['transformer', 'ner']
[i] Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0        1368.13    189.37    0.00    0.00    0.00    0.00
[!] Aborting and saving the final best model. Encountered exception:
PermissionError(13, 'Access is denied')
[2021-11-11 11:32:36,518] [INFO] Set up nlp object from config
[2021-11-11 11:32:36,526] [INFO] Pipeline: ['transformer', 'ner']
[2021-11-11 11:32:36,526] [INFO] Created vocabulary
[2021-11-11 11:32:36,526] [INFO] Finished initializing nlp object
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.bias']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[2021-11-11 11:32:39,772] [INFO] Initialized pipeline components: ['transformer', 'ner']
Traceback (most recent call last):
  File "C:\Users\cil\anaconda3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\cil\anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\cil\anaconda3\lib\site-packages\spacy\__main__.py", line 4, in <module>
    setup_cli()
  File "C:\Users\cil\anaconda3\lib\site-packages\spacy\cli\_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "C:\Users\cil\anaconda3\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\cil\anaconda3\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\cil\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\cil\anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\cil\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\cil\anaconda3\lib\site-packages\typer\main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "C:\Users\cil\anaconda3\lib\site-packages\spacy\cli\train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "C:\Users\cil\anaconda3\lib\site-packages\spacy\cli\train.py", line 75, in train
    train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "C:\Users\cil\anaconda3\lib\site-packages\spacy\training\loop.py", line 122, in train
    raise e
  File "C:\Users\cil\anaconda3\lib\site-packages\spacy\training\loop.py", line 110, in train
    save_checkpoint(is_best_checkpoint)
  File "C:\Users\cil\anaconda3\lib\site-packages\spacy\training\loop.py", line 72, in save_checkpoint
    shutil.rmtree(output_path / DIR_MODEL_BEST)
  File "C:\Users\cil\anaconda3\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\cil\anaconda3\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\cil\anaconda3\lib\shutil.py", line 622, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\Users\cil\anaconda3\lib\shutil.py", line 620, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 5] Access is denied: 'output\\model-best\\ner'

[polm]

Sorry you're having trouble with this. Unfortunately the Windows error is pretty vague and there could be a variety of causes for it.

Did the output or model-best directories exist before you started the training command? That is, were they created by a separate process? It should work either way, but if it's a separate process there are more things that could have happened.

Does this code work?

import spacy, shutil

nlp = spacy.blank("en")

print("saving model")
nlp.to_disk("check-serialize")
print("deleting model")
shutil.rmtree("check-serialize")
print("delete finished")

This is basically the same as what the training code is doing. If this doesn't work then something is grabbing the directory somehow, so we'll need to think about what that could be. If this works but your training code doesn't, then it could be a simpler issue like the process not running with admin privileges or something.

[githubbar]

Did the output or model-best directories exist before you started the training command? That is, were they created by a separate process?
None of the directories exist prior.

Does this code work?
Yes.

I'm still confused, what is the difference then and how can I run the training?

[githubbar]

I put up a tiny example notebook to reproduce the error. See attached:
test.zip

[svlandeg]

Hi,

I've tried to run your example code, and had to change the init fill-config command to:

!python -m spacy init config config.cfg --lang en --pipeline ner

because otherwise it wouldn't run. I assume you had it correct on your end though, because otherwise you would have gotten a FileNotFound error.

Either way, I can run this notebook on Windows just fine. The only other thing I can think of, is to try and upgrade spaCy to the latest version, but otherwise I think the problem is with one of your system/permission settings, and not directly related to something in spaCy.

[svlandeg]

I don't see how this could be related either, but just for good measure: try once on CPU instead of GPU?

[githubbar]

This worked for me too! I created mine via configurator and it does use GPU.

This worked though '!python -m spacy init config config.cfg --lang en --pipeline ner --gpu --force'
Attaching my old base config file if anyone is interested to see a discrepancy
base_config.zip
.