Doc.from_docs to retain custom attributes

[BramVanroy]

I guess this is more of a feature request.

It is possible to merge Docs into a single Doc with Docs.from_docs. This is useful if you are iteratively creating Doc and only at the end want to retrieve a single Doc. The initial Docs will then serve as (multiple) sentence spans (Doc.sents). It would appear, however, that custom Doc attributes are lost when using Docs.from_docs.

import spacy
from spacy.tokens import Doc

Doc.set_extension("metadata", default="")

nlp = spacy.load("en_core_web_sm")
first_doc = Doc(nlp.vocab, ["hello", "world", "!"], spaces=[True, False, False])
first_doc._.metadata = "this is some metadata"
doc = Doc.from_docs([first_doc])

This throws a warning right here: "Skipping Doc custom extension 'metadata' while merging docs.". I would have expected that the doc extensions would be saved as a list (for each initial Doc) in user_data or something similar.

spaCy/spacy/tokens/doc.pyx

Line 1140 in f803a84

warnings.warn(Warnings.W101.format(name=name))

Is there any way to preserve custom data that has been passed via single Docs into Doc.from_docs?

[polm]

This throws a warning right here: "Skipping Doc custom extension 'metadata' while merging docs.". I would have expected that the doc extensions would be saved as a list (for each initial Doc) in user_data or something similar.

That's an interesting idea, but I think it's generally not safe for us to make assumptions about user_data. In some cases it contains token offsets (which need updating), in some cases it's global doc information, ... there are a lot of possibilities. We could just dump everything in a list without breaking anything, but it also might not be useful in that state without further processing anyway.

I don't think there's any performance hit to manually managing user_data you want to keep after merging Docs. Is there a particular reason you want it to be handled automatically?

[BramVanroy]

I am not sure that I understand what you mean. I cannot seem to find a way to store my data in the user_data. If I directly add user_data to a Doc, I get another warning:

import spacy
from spacy.tokens import Doc

nlp = spacy.load("en_core_web_sm")
first_doc = Doc(nlp.vocab, ["hello", "world", "!"], spaces=[True, False, False], user_data={"metadata": "this is some metadata"})
doc = Doc.from_docs([first_doc])

Triggers:

UserWarning: [W102] Skipping unsupported user data 'metadata: this is some metadata' while merging docs.

[polm]

I was suggesting something like this:

docs = ...
superdoc = Doc.from_docs(docs)
# now superdoc.user_data is empty
for doc in docs:
    # copy over the data you want
    superdoc.user_data[...] = doc.user_data[...]

[polm]

Normally we provide a built-in interface for stuff like this if:

1. There is a performance advantage that requires Cython
2. There is one obvious correct way to handle the data

But in this case neither of those seems to be true. Am I missing something?

[BramVanroy]

Ah yes, this can be a solution too. Since I know beforehand that each Doc = one sentence span, I'll store my data in a Span extension instead, but your solution is the better, more general approach for other use-cases. Thanks!