Merging complex custom attributes with from_docs

[serenalotreck]

I'm using the benepar constituency parser to get constituency parses for my documents. The labels and children of a given node can be accessed through the Underscore method, like c._.labels or c._.children.

After applying the parser as part of my nlp object's pipeline, I merge documents together using from_docs. After merging, when I run the following code:

[c._.labels for c in sent._.children]

I get the following error:

Traceback (most recent call last):
  File "relation_extraction.py", line 119, in <module>
    args.out_loc, args.out_prefix)
  File "relation_extraction.py", line 60, in main
    skip, total = abst.extract_rels(tokenizer, model, label_df)
  File "/mnt/ufs18/home-118/lotrecks/haai_vanburen_lab/obie-wan/models/heuristic/abstract.py", line 188, in extract_rels
    phrases = self.pick_phrase(sent)
  File "/mnt/ufs18/home-118/lotrecks/haai_vanburen_lab/obie-wan/models/heuristic/abstract.py", line 257, in pick_phrase
    first_level_labels = [c._.labels for c in sent._.children]
  File "/mnt/ufs18/home-118/lotrecks/haai_vanburen_lab/obie-wan/models/heuristic/abstract.py", line 257, in <listcomp>
    first_level_labels = [c._.labels for c in sent._.children]
  File "/mnt/home/lotrecks/.local/lib/python3.7/site-packages/benepar/integrations/spacy_extensions.py", line 112, in get_child_spans
    constituent_data, position = get_constituent(span)
  File "/mnt/home/lotrecks/.local/lib/python3.7/site-packages/benepar/integrations/spacy_extensions.py", line 21, in get_constituent
    "No constituency parse is available for this document."
Exception: No constituency parse is available for this document. Consider adding a BeneparComponent to the pipeline.

I know from #5032 that from_docs doesn't support merging custom attributes, and that you have to write your own attribute merger. I understand in theory how this works, but all the examples provided in other discussion posts and issues are for simple data type custom attributes like boolean or integer values.

I could make a custom attribute merger for the parse strings, but I also want to be able to navigate the tree the same way that I am doing above. Does anyone have thoughts about how to accomplish this?

[adrianeboyd]

For benepar you'd need to merge the internal ConstituentData objects that are stored in doc._._constituent_data. The token and span extensions all refer to this Doc extension, which is why it looks like the merged doc isn't parsed.

You'd have to dig into the benepar internals of how the parse is represented in ConstituentData. It's definitely doable, but might be a bit fiddly.