PermissionError with spacy train on Windows

[abs-xyz]

I am using the latest version (3.3.0).
This is related to this previous discussion.
While training, I am facing errors like PermissionError: [WinError 5] Access is denied: 'output\\model-best\\ner'.

I am using the following line as mentioned in the earlier discussion (link above)

python -m spacy init config config.cfg --lang en --pipeline ner --force

(Edit: I had looked the discussion up after the command in docs python -m spacy init fill-config base_config.cfg config.cfg didn't work)

Then I am converting some CoNLL 2003 data to spacy format (using spacy convert ... -c ner).

Then trying to train using

python -m spacy train ./config.cfg --output ./output --paths.train <path to train.spacy> --paths.dev <path to valid.spacy>

I am getting the following error:

...lots of lines...
File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 620, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 5] Access is denied: 'output\\model-best\\ner'

Note:
As mentioned in the earlier discussion, I have already tried running

import spacy, shutil

nlp = spacy.blank("en")

print("saving model")
nlp.to_disk("check-serialize")
print("deleting model")
shutil.rmtree("check-serialize")
print("delete finished")

and this works fine.

---

Edit:
Config file

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,2500,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

(using python -m spacy init config config.cfg --lang en --pipeline ner --force)
On using python -m spacy debug data config.cfg , I get ValueError: [E913] Corpus path can't be None. Maybe you forgot to define it in your .cfg file or override it on the CLI?, the latter of which I believe I am doing (in spacy train options).

---

I have tested the data with

from spacy.tokens import DocBin

nlp = spacy.load("en_core_web_sm")
doc_bin = DocBin().from_disk(spacy_train_data_path)
docs = list(doc_bin.get_docs(nlp.vocab))
entities = 0
for doc in docs:
    # print(doc.ents)
    entities += len(doc.ents)
print(f"TRAIN docs: {len(docs)} with {entities} entities")

doc_bin = DocBin().from_disk(spacy_valid_data_path)
docs = list(doc_bin.get_docs(nlp.vocab))
entities = 0
for doc in docs:
    entities += len(doc.ents)
print(f"DEV docs: {len(docs)} with {entities} entities")

(no errors)

[polm]

Thanks for the detailed report, and sorry for the delay following up. Unfortunately we're not really sure what could cause this...

One thing I have seen cause errors like this before is sharing settings. Sometimes if sharing is enabled for a folder in Windows, it can be grabbed by the OS and prevent deletions from within Python. The exact timing of this is a little complicated but your sample script probably works because it's all in the same process. Maybe you could check your sharing settings?

[polm]

I normally wouldn't recommend this, but since I'm pretty sure your issue is more about Windows permissions than about spaCy, you might want to try asking on Stack Overflow instead of here.