Apply a Model Component to Spans of a Doc as well as the Doc Itself

[benwebs]

I am applying a HuggingFace sentiment classifier to the SpaCy Doc object using the following tutorial: https://github.com/explosion/spacy-huggingface-pipelines.

This gives the sentiment for the document as a whole. My documents contain multiple sentences each and I would also like the sentiment at the sentence level, and for it to be saved as a custom attribute of the sentence in the Doc object.

Can anyone assist with how this is possible?

[rmitsch]

Hi @benwebs, I suggest to:

1. Iterate over the sentences in your Doc instance (see e. g. the Sentencizer docs for more info).
2. Run your sentiment classifier on your each sentence.
3. Update your Doc instance by writing the results into a Doc extension.

[benwebs]

Thank you very much for your help. I have a solution that performs the above steps, roughly. The issue that I have is that the only way I have found is to create a list of sentiments, one for each sentence, then attach that list to the Doc object. What I would like to accomplish is to attach the sentiment to the sentences directly. I'm not sure if my request is possible, I'm just trying to make sure that I'm using the SpaCy object the right way.

The solution I have identified requires me to keep the sentiment of the sentences at the level of the doc, and I would like to store the sentiment of each sentence at the level of the sentence. Thanks again for any help.

[jkgenser]

Create spans for each sentence and use a span extension

https://spacy.io/api/span#set_extension

[rmitsch]

There is no object that 1:1 represents a sentence in spaCy. There is information on where sentences begin, and Span objects that can be used to represent sentences.

...accomplish is to attach the sentiment to the sentences directly...

There are several options for this. One option, as suggested by @jkgenser, is to create Span objects for the sentences and using a Span extension to store the sentiment score for the corresponding sentence. Another one is to use a Doc extension for a dict using sentence start/end indices as keys and the sentiment scores as values.

[benwebs]

Thank you both. I was able to accomplish it in both ways. The links provided were useful resources.