spacy ver 3.2.2: How to forcefully save a SpanCat Model

[smisra1]

My spacy version is 3.2.2. I encountered an issue during the training of a Span Catergorizer Model with 21,000 records using CLI. Since the data was not clean, there was no active learning in the model.

The issue is that in this situation, no model was saved. Due to this, an error was displayed in my code when trying to retrieve the scores from model-best/meta.json file, which I want to avoid at all costs.
I wanted to save a model even if there was no active learning involved in it. The scores could show 0 precision, recall and f-score but it still should be present.

How could I do this?

[polm]

If spaCy is interrupted in training, it will save the last version of the model. If you aren't getting a model at all it sounds like your model is failing in initialization - is that the case? (I'm not sure what you mean by "active learning" in this context.)

If that's what's happening, it sounds like the best thing is for you to handle training failures outside of spaCy, such as by returning a default value or writing your own placeholder meta.json. There's no reasonable way to serialize a model that was never successfully initialized.

If that's not what was happening, could you share the full output of training with stack trace so we can understand what's going on better?

[smisra1]

I think it could be what you said that it was failing in initialization. Could you explain a bit more regarding what could cause failing in initialization?

[polm]

There are different things that could cause failure in initialization but a typical example is having no annotations - for example, you can't train a textcat with no category annotations.

Do you not have the error output from your failed training?

[smisra1]

No, no error output. The issue is that in my code, I am creating and training 3 SpanCat models but with different label annotation. 2 models were created and trained successfully but this last one was not without showing any error. On analysis of data, I could see that comparitively there was less count of training records per label in this failed-to-initialize model. Meaning the number of labels were more and the count of training records for those labels, less. Eg, there were 1000 labels with only 1-2 record example.

[polm]

I'm not really sure how training could fail completely silently. Less training data might result in worse performance or something even if it's unusable you should get an error. Did it show any output at all before it stopped?

It sounds like your system has a lot going on but can you maybe share a little example we can run to reproduce this, or maybe the code you're using to run the training?

[smisra1]

Since the code is on my remote desktop, I could only attach a screenshot of the CLI code I use to start training of the model. The data is in standard .spacy format. The second screenshot is the code used to convert a list of dictionaries to .spacy file

[polm]

I understand this is a remote desktop, but surely there's a way to copy and paste the text? Screenshots of code are hard to work with.

If you're executing this in subprocess.run, you should be able to get the output - is there actually no output from the command? What's the exit code when it fails?

Regarding the code where you make the DocBin... It looks like you're not doing anything with the first DocBin, so that could just be a list of Docs. Or you could move the thing that sets the spans inside your loop.

Can you count the number of spans you're setting to make sure you have annotations and the Docs aren't empty or something?