Extracting word-level features

[saeub]

Thanks for creating this, the list of supported features is impressive!

A lot of the features are defined at the word-level (like word length, prevalence, age of acquisition, etc.), but only aggregated values (avg etc.) are exposed via the interface. Is there an easy way to extract those features at the word level, and align them with the corresponding tokens?

[mmmaurer]

This is definitely part of the long-term goals of elfen.
But unfortunately, I haven't found a straightforward way to do this efficiently yet, as the aggregate values are order-invariant, and this is vital due to the dataframe optimization. But token-level measures would need to either be saved in the given order or the order would have to be reconstructed. Neither of which is well-aligned or easily adaptable to the current approach, as it requires optimization on nested structures.

As far as I know, there also isn't an easy existing implementation for this purpose. So I'm afraid you would have to come up with a custom implementation.

What you may want to do to achieve this is use a SpaCy or a stanza model to tokenize, and then use them, for example, for age of acquisition as follows:

import elfen
import spacy

# prepocessing
examples = [
    "This is an example text item. It has multiple sentences and various linguistic features.",
    "Another example text item is here. It also contains multiple sentences and different linguistic features.",
]
nlp = spacy.load("en_core_web_sm")
docs = [nlp(example) for example in examples]

# load AoA norms
aoa_norms = elfen.psycholinguistic.load_aoa_norms(
    path=elfen.resource_utils.resource_map.RESOURCE_MAP["aoa_kuperman"]["filepath"]
)

# extract token-level AoA values
aoa_vals = [
    [
        aoa_norms.filter(
            pl.col("Word") == token.text.lower()
        )["Rating.Mean"].to_list()[0]
        if not aoa_norms.filter(
            pl.col("Word") == token.text.lower()
        )["Rating.Mean"].is_empty()
        else None
        for token in doc
    ]
    for doc in docs
]

Of course, this is not very efficient, especially for bigger datasets.

Hope this helps!

[saeub]

Thanks for the quick reply! I see the problem... Well, for now it's already great to just have the easy access to those lexica 👍