There is nothing or a little change after training on an existing model for dependency parser using 71 examples.

[qingyun1988]

I've trained on an existing model 'en_core_web_trf' for dependency parser using 71 examples, but there is nothing change on it.

And then I replaced 'en_core_web_trf' to 'en_core_web_lg', there is literally a little change, but it is not as that I expected. for example, use this sentence to check the change:
"So helpful is he that we all like him."

The vision I expected:

The vision before training:

The vision after training:

Observing the there pictures, I extracted 2 pieces of information:

1. The change after training was different from what I expected in figure 1;
2. The change was not only unexpected but also get worse than before training—the label "ccomp" has been replaced 'dep'.

My questions are why there is nothing change after training on model 'en_core_web_trf' for dependency parser and why the change on model 'en_core_web_lg' is not as expected.

Is the amount of examples not enough for training on a model to get any change? However, I provided 71 examples for training. I remembered the administrator told me that 20 or 30 examples are enough to do.

Did I miss some step or method? Did I do something wrong? Please help me, any help will be appreciated!

By the way, The training way I adopted was 'training api' not 'train command'. This is my code:
`def train(trainData):
global _nlp,_model_dir
if _nlp == None:
_nlp = spacy.load(_model_dir)

#pipelines = ["transformer","tagger","parser"]#for web_trf
pipelines = ["tok2vec","parser"]#for web_lg
toDisablePipes = [pipe for pipe in _nlp.pipe_names if pipe not in pipelines]
with _nlp.disable_pipes(*toDisablePipes):
    optimizer = _nlp.resume_training()
    for itn in range(30):
        random.shuffle(trainData)
        for text, annotations in trainData:
            example = Example.from_dict(_nlp.make_doc(text), annotations)
            _nlp.update([example],sgd=optimizer,drop=0.25)

_nlp.to_disk(_model_dir)
_nlp = None
return True;`

[polm]

Is the amount of examples not enough for training on a model to get any change? However, I provided 71 examples for training. I remembered the administrator told me that 20 or 30 examples are enough to do.

There aren't really any magic numbers but 71 examples is nowhere near enough. In general the larger the model the more data you need; for Transformers this is definitely not enough.

Who is "the administrator"? I think we sometimes suggest a few hundred examples as a minimum for NER, but retraining the parser is generally more complicated.

[qingyun1988]

Would someone like to help me?

[svlandeg]

Hi @qingyun1988 : please don't artificially bump your thread as this is really unhelpful to us. We try to get to everyone's questions in due time, but with no garantuees.

[qingyun1988]

Hi @qingyun1988 : please don't artificially bump your thread as this is really unhelpful to us. We try to get to everyone's questions in due time, but with no guarantees.

So can you help me to answer the above questions? Thank you.

[adrianeboyd]

Hi, the main problem is probably that this is just not enough data to update the parser for this kind of parse that's very different from all the parses it's been trained on so far.

dep is a label that's used as a backoff when the parser can't find a better alternative. It also appears in the training data in a small number of cases where the constituent-to-dependency converter couldn't find a better label to use, so the parser may predict it as a result of this, too, although it's really rare (0.1% of dependency labels).

You also want to update the model with a mixture of inverted and non-inverted sentences or you'll run into problems where it "forgets" how to parse other sentences. See a related comment about catastrophic forgetting: #7666 (comment)

[qingyun1988]

@adrianeboyd
Thank you very much!
I have read the #7666 (comment) and the article https://explosion.ai/blog/pseudo-rehearsal-catastrophic-forgetting

It seems to have two ways to resolve the 'catastrophic forgetting' problem.
One is 'Pseudo rehearsal' in which its key is to contains two data examples: revision_data and fine_turn_data. But I still have a few questions about it. Should the content and number of sentences in 'revision_data' be the same as those in 'fine_turn_data'? For example, let's say, there are two sentences 'I love you' and 'You hurt me' in 'revision_data' and the content of 'fine_turn_data' must be the same?

Another is nlp.rehearse() method in Spacy v3.0. But the code example here only contains nlp.rehearse() but nonlp.update():

Doesn't the nlp.rehearse() method need to update the changes? Do I need to add a line of code like this at the end：

optimizer = nlp.resume_training()
nlp.rehearse(examples, sgd=optimizer)

#add newline
nlp.update(examples, sgd=optimizer)

I'm a bit confused about these.