The NER log difference between using tok2vec and transformers

[mbrunecky]

I am trying to compare my NER training between using tok2vec and transformer pipeline.
The (default) log is described as logging every 200 documents items such as Epoch, # of documents, losses and scores.

My data set has 6,650 training documents and 1,700 validation docs. The log report for tok2vec shows that my epoch 0 is about my 6550 documents:

E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00    153.90    0.37    0.21    1.49    0.01
...
  0    6600       1179.16    652.39   81.64   81.86   81.41    0.82
  1    6800       1220.52    545.59   82.54   83.19   81.90    0.82

But when using transformer pipeline, it seems that my 0 epoch is less than 2800 documents:

E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  -------------  --------  ------  ------  ------  ------
  0       0       13572.34   1010.29    0.02    0.01    0.16    0.00
...
  0    2600         873.72   7433.51   88.90   87.18   90.69    0.89
  1    2800         909.21   5287.98   89.70   87.26   92.29    0.90

Why is there a difference (in Epoch size)?
Is it a feature or a bug?

[polm]

The # column is the number of batches, not the number of documents. Is your batch size different between your two pipelines?

[mbrunecky]

Thank you, but sorry, I am still confused. The term 'batch size' is heavily overloaded:

In the 'transformer' pipeline I see

pipeline = ["transformer","ner"]
batch_size = 96

and

[training.batcher]
size = 2000

In the tok2vec pipelines, I (normally) keep the
batch_size = 500
and

[training.batcher.size]
start = 100
stop = 1000

meaning that the training batch size varies from 100 up to 1000.

Then, the eval_frequency = 200. Which batches? The variable size ones?
In the tok2vec pipeline, my Epoch 0 always ends past logged number 6600 - which corresponds to my 6550 documents, not the batch_size=500.

[svlandeg]

The term 'batch size' is heavily overloaded

You can find their definitions in the relevant docs:

• https://spacy.io/api/data-formats#config-nlp --> definition of nlp.batch_size
• https://spacy.io/api/data-formats#config-training --> definition of training.batcher
• https://spacy.io/api/top-level#batch_by_words --> definition of size for the batcher spacy.batch_by_words.v1

[mbrunecky]

Thank you, THANK you, I appreciate the doc pointers. In fact I am impressed by the documentation quality. But I did read those multiple times before, and it still left me confused. Perhaps there is a need for a documentation section discussing differences of various batch sizes, and (especially) their impact upon:

• resource utilization
• training duration
• training accuracy
• reporting

Following this forum, I suspect I am not the only one who would benefit.