Understanding en_core_web_trf tensors tuple

[IliasAarab]

When creating an nlp pipeline based on en_core_web_trf we get a tuple of tensors available through doc._.trf_data.tensors. The first tensor available through doc._.trf_data.tensors[0] seems to reflect the last hidden state of the BPE (Byte Pair Encoding) token-pieces, although this is not entirely clear? I suppose, the second tensor doc._.trf_data.tensors[1] represents the last hidden state of the special [CLS] token, but again this is not entirely clear?

Could someone confirm what the tensors tuple actually contain?

The view more link of https://spacy.io/universe/project/spacy-transformers seems to redirect to an outdated blogpost of Explosion, as I cannot successfully run the code examples.

[rmitsch]

Hi @IliasAarab! The data in doc._.trf_data are instances of FullTransformerBatch. The contents of FullTransformerBatch.tensor depends on the output of the underlying HuggingFace model - in the case of models with pooling the outputs (cf. the HF docs) contain last_hidden_state and pooler_output as the first two elements.

Again, this depends on the HF model. I recommend inspecting doc.trf_data.model_output to see what's included.

[IliasAarab]

Thanks! I guess for the transformer model shipped with en_core_web_trf we need to refer to the RoBERTa model.