Why is my trained transformer model(en_core_web_trf) ineffective?

[qingyun1988]

I used flowing code to train the 'en_core_web_trf' model, but the consequence seems to be ineffective.

TRAIN_DATA = [
    (
        "Books were present at the meeting.",
        {
            "heads": [1,1,1,2,5,3,1],
            "deps": ["nsubj", "ROOT", "acomp", "prep", "det", "pobj","punct"],
        }
    ),
    (
        "They were present at the meeting.",
        {
            "heads": [1,1,1,2,5,3,1],
            "deps": ["nsubj", "ROOT", "acomp", "prep", "det", "pobj","punct"],
        },
    ),
    (
        "Aides were present at the meeting.",
        {
            "heads": [1,1,1,2,5,3,1],
            "deps": ["nsubj", "ROOT", "acomp", "prep", "det", "pobj","punct"],
        },
    ),
]

@plac.annotations(
    model=("Model name. Defaults to blank 'en' model.", "option", "m", str),
    output_dir=("Optional output directory", "option", "o", Path),
    n_iter=("Number of training iterations", "option", "n", int),
)
def main(model="en_core_web_trf", output_dir="D:\Python\spacy\customTrf30", n_iter=20):#en_core_web_trf
    """Load the model, set up the pipeline and train the parser."""
    nlp = spacy.load(model)
    parser = nlp.get_pipe("parser")

    # add labels to the parser
    for _, annotations in TRAIN_DATA:
        for dep in annotations.get("deps", []):
            parser.add_label(dep)

    # get names of other pipes to disable them during training
    #pipe_exceptions = ["parser","tok2vec"]#for web_lg
    #pipe_exceptions = ["parser","transformer"]#for web_trf
    pipe_exceptions = ["parser","transformer","tok2vec"]
    other_pipes = [pipe for pipe in nlp.pipe_names if pipe not in pipe_exceptions]
    with nlp.disable_pipes(*other_pipes):  # only train parser
        optimizer = nlp.resume_training()#nlp.begin_training()//resume_training
        for itn in range(n_iter):
            random.shuffle(TRAIN_DATA)
            losses = {}
            for text, annotations in TRAIN_DATA:
                example = Example.from_dict(nlp.make_doc(text), annotations)
                #losses = nlp.rehearse([example],sgd=optimizer)#,component_cfg=pipe_exceptions
                nlp.update([example],sgd=optimizer, losses=losses)

            print("Losses", losses)

    # save model to output directory
    if output_dir is not None:
        output_dir = Path(output_dir)
        if not output_dir.exists():
            output_dir.mkdir()
        nlp.to_disk(output_dir)
        print("Saved model to", output_dir)

However, if I use the same codes to the 'en_core_web_lg' model, it seems to work fine and by testing it can show immediately the consequence I expect.

The difference is only to replace 'en_core_web_trf' with 'en_core_web_lg'.

Is there any other way to train the 'en_core_web_trf' model? How can I train transformer model(en_core_web_trf)?

In addition, I don't know how to train models with training config file.

Appreciate if anyone can help. Thank you!

[polm]

Sorry to hear you're having trouble. Can you clarify a few things?

1. Are those three examples your entire training data?
2. When you say it works with the large model but not Transformers, what changes are you referring to? What do you check to see it changed?
3. What are you trying to change in the model? Those sentences look pretty normal.
4. Did you see the training quickstart? That will show you how to train a Transformer model with a config.

If you actually only have three training examples, that's not enough to train any of the models meaningfully.

[qingyun1988]

Does it have to be used for the config training method to work?

[qingyun1988]

if a few examples are not enough to train the model, why are they enough for the large model but not the transformer model? I've found that only an example is enough to train for the large model since I have used Spacy. I mean the results for the current sentence not another one in the same circumstance.

[honnibal]

Hi @qingyun1988 ,

We recommend the training config (plus CLI) or at least the more general training methods because there's a lot of details to get right in the training loop. In this case, I think the problem is probably that you're not stepping the learning rate schedule. This would mean that the learning rate is stuck at 0, which means that the optimization is not meaningfully changing the weights. It might actually leave the weights identical -- I'm not sure whether there's a regularization that is applied regardless of the learning rate.

You could try the train_while_improving function from spacy.training.loop instead of your own training loop code. It works as a generator, so you still get a block to execute your own code on each batch. If you do want to use your own code, some things to check are the learning rate and whether the weights of the models are actually changing. You can check the weights by doing:

for name, pipe in nlp.pipeline:
    for node in pipe.model.walk():
        for param_name in node.param_names:
            param = node.get_param(param_name) # (numpy or cupy ndarray)

Unfortunately, even if your code is entirely correct, the transformers are not guaranteed to converge on a 3 sentence sample. The weights should definitely change, though -- so you can at least exclude some problems. The reason it might not converge is batch size: the optimization isn't stable at small batches with the default learning rate. You can fiddle with the learning rate, or use a slightly larger set of example sentences to memorize, something like 10 or 30 should be fine.

[qingyun1988]

Hi，@honnibal!
I'm so sorry to have just seen your reply, thanks for your patient and kindness.

I am using training API but not training config method because it's convenient to using dependency editor which was developed by myself to create gold-standard annotations.

And I don't know how to use training config method to train a already existed model which seems to be harder than the former for me. What are the problems for keeping to use training API method?

How to try train_while_improving function from spacy.training.loop? Does nlp.resume_training() function of training API
equate train_while_improving function?

What is the more general training methods?

By the way, what is the best way to get or create gold-standard annotations? My solution was to develop a dependency editor.

Thank you!

[qingyun1988]

@honnibal