extracting attentions from en_core_web_trf (3.2.0)

[calebjacksonhoward]

I have been using spaCy for smaller tasks for a few years now, but am diving in to v3.2 with great gusto for the transformer support. There are a few things I'm learning concurrently, and a little help from someone who knows will be very timely at this point, I think

I am trying to figure out how to extract attentions from a model, and have started with en_core_web_trf (v3.2.0).

I have looked at #7283, which indicates that this should be possible since explosion/spacy-transformers#268. I have used the example code from the latter discussion:

import spacy
nlp = spacy.blank("en")

# Construction via add_pipe with custom config
config = {
    "model": {
        "@architectures": "spacy-transformers.TransformerModel.v1",
        "name": "bert-base-uncased",
        "transformers_config": {"output_attentions": True},
    }
}
transformer = nlp.add_pipe(
    "transformer", config=config)
transformer.model.initialize()


doc = nlp("This is a sentence.")

but it throws an error:

Traceback (most recent call last):
  File "test01.py", line 14, in <module>
    "transformer", config=config)
  File "/home/env/sm/lib/python3.7/site-packages/spacy/language.py", line 793, in add_pipe
    validate=validate,
  File "/home/env/sm/lib/python3.7/site-packages/spacy/language.py", line 671, in create_pipe
    resolved = registry.resolve(cfg, validate=validate)
  File "/home/env/sm/lib/python3.7/site-packages/thinc/config.py", line 730, in resolve
    config, schema=schema, overrides=overrides, validate=validate, resolve=True
  File "/home/env/sm/lib/python3.7/site-packages/thinc/config.py", line 779, in _make
    config, schema, validate=validate, overrides=overrides, resolve=resolve
  File "/home/env/sm/lib/python3.7/site-packages/thinc/config.py", line 839, in _fill
    overrides=overrides,
  File "/home/env/sm/lib/python3.7/site-packages/thinc/config.py", line 839, in _fill
    overrides=overrides,
  File "/home/env/sm/lib/python3.7/site-packages/thinc/config.py", line 901, in _fill
    ) from None
thinc.config.ConfigValidationError:

Config validation error

transformer.model -> get_spans   field required
transformer.model -> transformers_config   extra fields not permitted

{'@architectures': 'spacy-transformers.TransformerModel.v1', 'name': 'bert-base-uncased', 'transformers_config': {'output_attentions': True}}

I have also tried loading en_core_web_trf:

nlp = spacy.load("en_core_web_trf")

...and then modding the config:

nlp.config['components']['transformer']['model']['transformer_config']['output_attentions'] = True

I write the model out to disk, and then load it fresh - not sure how best to do this sort of modding, but wanting to be sate. It loads up, and shows the output_attentions to be set in the model's config. So then I look for the attentions:

doc = nlp("This is a sentence that I am passing to an NLP pipeline")
doc._.trf_data.model_output.keys()

which yields:

odict_keys(['last_hidden_state', 'pooler_output'])

or:

doc._.trf_data.attention

...as suggested in the latter discussion (268), but that yields:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'TransformerData' object has no attribute 'attention'

I'm very excited to move my work with transformers into the new spaCy pipeline options, but I've kind of dived into a lot of stuff I'm yet unfamiliar with. My ignorance is I'm sure quite evident. I'm finding the documentation to be scarce, and or out of date, maybe. I'm picking it up quickly, but I learn best by example code, and I cannot find any yet to help me get these attentions back out as I need to.

Can you help? I will be most grateful!

[adrianeboyd]

This requires a newer version of the architecture spacy-transformers.TransformerModel.v3: https://spacy.io/api/architectures#TransformerModel

The output will be in trf_data.model_output once the setting is correct.

To modify en_core_web_trf, you need to modify the transformer model loaded within the pipeline component directly rather than modifying the config:

nlp.get_pipe("transformer").model.transformer.config.output_attentions = True

This modified setting will be saved with nlp.to_disk as part of the model data for transformer. It's a bit confusing, but the original settings from config.cfg are only really used when the model is initialized. (If I were starting from scratch, I'd move these settings to the [initialize] block, but it was going to be too much of a change from spacy-transformers v1.0 to v1.1, so we left them in the [components] block.)

[calebjacksonhoward]

Thank you very much for the quick turnaround. I will try your suggestions today!
(and mark the answer when I make it work. :-) )

[calebjacksonhoward]

Power outage yesterday. This is now providing me 'attentions' on trf_data.model_output, just as you say. It makes sense that it works this way, but would have taken me some time to come to on my own.

I have marked the answer. Thank you again.