Trained model returning ==Weblinks==

[robinlaegler]

Hello, i trained a en-de model using the OPUS-Wikipedia Dataset and ~10.000 sentences of my own training data. I dropped the batch size to 800. I trained it on GoogleColab , because I don't have a GPU laying around. The training finished in about 5 hours. The model is pretty bad and it sometimes returns "==Weblinks==" or "==Einzelnachweise==".
Does anybody dealt with the same problem or knows where these tags are coming from?
And are there any tips to train a better model? Do I need more data or a higher batch size? Which is more important?

[PJ-Finlay]

I don't think batch size is likely the issue. How much data do you have? Increasing would probably help. Also how many epochs did you train for? I have gotten good results around 10k.

[robinlaegler]

I found out the mistake I made: The OPUS-Wikipedia dataset includes many tags like "==Preferences==", "==Weblinks==", etc. Because of that my model is pretty bad.
As I said I used the OPUS-Wikipedia Dataset with 2.5M sentences and my own data with ~10.000 sentences. I trained for 50k steps (default in the config.yml file). I only lowered my batch size.
Is it possible to train the 10.000 sentences of my own data by themself or is there a minimum of language pairs I need to train a language model and I need to add data from the OPUS?

[PJ-Finlay]

It's also possibly an overtraining issue, maybe the default should be lowered to 10k or something. One thing to try is to modify package.sh to average and package earlier epochs.

[PJ-Finlay]

I think you want to use data from opus combined with your data.

[PJ-Finlay]

In general more data is good, and combining multiple datasets from OPUs generally helps because it increases the diversity of the data and the total quantity.

[PJ-Finlay]

overtraining_thresholds.json has some data to give you an idea of when overtraining is likely.