Does en_core_web_trf truncate documents to 512?

[ginward]

It seems that the documentation does not specify if en_core_web_trf truncates texts longer than 512 (or any other transformer limits).

Is there such a truncation being performed?

[ginward]

This is related to #6939 and #7094

[ginward]

Or is it fine if each sentence is less than 512 length?

[ginward]

Although it says here, that Spacy splits documents longer than 512 into sentences first before feeding it to the en_core_web_trf model, I am not too sure (from the documentation) that if the en_core_web_trf model uses transformer to segment sentences. If so, wouldn't documents longer than 512 length not be correctly sentenced?

Although my experiment here does not seem to have a problem:

import spacy
nlp_ent = spacy.load("en_core_web_trf")
nlp_ent.add_pipe("merge_entities")
text = "I love eating burgers. I live in a house." * 1000
doc = nlp_ent(text)
super_word_ls = []
for s in doc.sents:
    word_ls = []
    for t in s:
        if not t.ent_type_:
            if (t.text.strip()!=""):
                word_ls.append(t.text)
        else:
            word_ls.append(t.ent_type_)
    if len(word_ls)>0:
        super_word_ls.append(" ".join(word_ls))
len(super_word_ls)

[adrianeboyd]

By default the transformer component uses overlapping strided spans (see: https://spacy.io/api/transformer#span_getters) so you can train and predict on longer texts without issues on transformer models that have a fixed max length.

Splitting long documents into sentences is something that happens just during training with the corpus reader, and the sentences come from the underlying corpus annotation, not from a component in the pipeline. Those settings aren't related to what happens when you run en_core_web_trf on a new text.