Data Augmentation for NER

[DSLituiev]

I am training a NER for digit-valued entities. I would like to understand what is the current mechanics and best practices for such case:

• does NER use is_digit or pos properties as features, and if not, where should one look to enable it?
• alternatively, if whether the token is digit is not taken into account, I would like to write a data augmenter, that will swap my digits to other digits.

def augment(doc):
    simulate_dict = {"prostate_weight_unit_g": partial(simulate_normal, mu = 51, sigma=19),
                    "tumor_volume_cm3": partial(simulate_lognormal, mu = 0.098, sigma=1.0),
                     "tumor_volume_mm3": partial(simulate_normal, mu = 1.0, sigma=1.0)
                    }

    def _swap_token(tok):
        for label, starts in start_dict.items():
            if tok.i in starts:
                return simulate_dict[label]()
        return tok.text 

    start_dict = {}
    for label in simulate_dict.keys():
        start_dict[label] = {ent.start for ent in  doc.ents if ent.label_ ==label}
  
    tokens = [_swap_token(tok) for tok in doc ]
    text = " ".join(tokens)
    return Doc(doc.vocab, 
               [_swap_token(tok) for tok in doc ],
               spaces=[bool(token.whitespace_) for token in doc],
              )

Now I need to add existing entities.

If I modify Doc construction according to documentation:

Doc(doc.vocab, 
               [_swap_token(tok) for tok in doc ],
               spaces=[bool(token.whitespace_) for token in doc],
               ents=[tok.ent_iob_ for tok in doc]
              )

I'm getting: ValueError: [E177] Ill-formed IOB input detected: B.

I might not completely get what the documentation help(Doc) is saying:

ents (Optional[List[str]]): A list of unicode strings, of the same
    length as words, as IOB tags to assign as token.ent_iob and
    token.ent_type. Defaults to None.

what kind of Unicode strings? I wasn't able to find any example on the web.

As a workaround, I tried this custom cython function:

ent = doc.ents[0]
set_entity(doc, ent.start, ent.end, ent.label)

but when I try to run it on my entity, it breaks:

spacy/entities.pyx in entities.set_entity()
      3 from spacy.tokens.doc cimport Doc
      4 
----> 5 cdef set_entity(Doc doc, int start, int end, int ent_type):
      6     for i in range(start, end):
      7         doc.c[i].ent_type = ent_type

OverflowError: value too large to convert to int

The actual values are:

ent.label_ : prostate_weight_unit_g
ent.label : 2008743142973371832

It looks similar to this issue, but that one is locked and there seems no solution.

[polm]

So there are several questions here, but to address a few of them:

does NER use is_digit or pos properties as features, and if not, where should one look to enable it?

NER does not use is_digit directly, but since vectors use shape features it captures the same information. On the other hand, the Tagger and NER are separate components and don't talk to each other, so POS features are not used.

alternatively, if whether the token is digit is not taken into account, I would like to write a data augmenter, that will swap my digits to other digits.

Since the token shape is already capturing this variation I wouldn't expect this to have a large effect.

I'm getting: ValueError: [E177] Ill-formed IOB input detected: B.

You're trying to use the Token.ent_iob_ to create an IOB tag you pass to Doc, but Token.ent_iob_ is not a complete IOB tag, it's just the prefix, like B. B by itself is not a valid IOB tag, a valid IOB tag is like B-PERSON - it needs to have a type attached to it, it can't just be a generic Beginning. You want something like tok.ent_iob_ + "-" + tok.ent_type_.

I think that given those points that addresses the overall thrust of your question, but if there's something I missed let me know.

[DSLituiev]

Thank you, that helps.
Is there documentation about pre-training / construction of vectors?

A remaining question is about set_entity OverflowError above. Would it be better to raise it as a separate issue?

[polm]

Is there documentation about pre-training / construction of vectors?

Yes, there is a pretraining section in the docs.

A remaining question is about set_entity OverflowError above. Would it be better to raise it as a separate issue?

We can keep discussing it in this thread.

I was under the impression your function was a workaround and shouldn't be necessary now that it's clear how to pass ents when building a Doc. Is there a reason you want to keep using it?

For the error itself, it looks like the issue is that your label is not in vocabulary and is getting a big hash value which won't fit in the type. I think you can fix it by making sure it's in your vocab, I need to look at that more though.

[adrianeboyd]

The set_entity example looks like it's gotten out-of-date. For spacy v2+ it should use attr_t instead of int for the ent_type type. But we wouldn't really recommend using this method for what you're doing here.

If you are writing an NER pipeline component, it could make sense to set the entities in cython (it's faster), but if you're preparing training data, it's probably better to set doc.ents or use doc.set_ents. doc.set_ents supports all possible annotations and makes sure that the resulting annotation is well-formed.

[DSLituiev]

I'm writing a pipeline component. This seem to work in plain python:

def augment(doc):
    simulate_dict = {"prostate_weight_unit_g": partial(simulate_normal, mu = 51, sigma=19),
                     "tumor_volume_cm3": partial(simulate_lognormal, mu = 0.098, sigma=1.0),
                     "tumor_volume_mm3": partial(simulate_normal, mu = 1.0, sigma=1.0)
                    }

    def _swap_token(tok):
        for label, starts in start_dict.items():
            if tok.i in starts:
                new_token = simulate_dict[label]()
                logging.debug(f"label = '{label}'\ttoken = '{new_token}'")
                return new_token
        return tok.text 


    start_dict = {}
    for label in simulate_dict.keys():
        start_dict[label] = {ent.start for ent in doc.ents if ent.label_ ==label}

    
    tokens = [_swap_token(tok) for tok in doc ]
    text = " ".join(tokens)
    return Doc(doc.vocab, 
               [_swap_token(tok) for tok in doc ],
               spaces=[bool(token.whitespace_) for token in doc],
               ents=["-".join([tok.ent_iob_ ,tok.ent_type_]) if 
                       tok.ent_type_!="" else tok.ent_iob_ for tok in doc]
              )

Does this part ["-".join([tok.ent_iob_ ,tok.ent_type_]) if tok.ent_type_!="" else tok.ent_iob_ for tok in doc] look right? Is there some method to get the right ent from the tok in one go?

Is doc.set_ents preferred to using Doc(ents=...) constructor?

I'll try to update the cython with attr_t.

[adrianeboyd]

As long as the final annotation is correct, it doesn't make much difference whether you use Doc(ents=) or doc.set_ents. Choose what's easiest based on your input data / augmentation. The main reason to use IOB is because that's what your training corpus has. If you're starting from a Doc, it's probably easier to create Span objects and use doc.set_ents. But in the end it's not a huge difference.

[adrianeboyd]

SHAPE should cover numbers fairly well, but if you want to try out IS_DIGIT, you can add it as an attribute with something like this in your tok2vec:

@architectures = "spacy.MultiHashEmbed.v1"
width = 96
attrs = ["NORM","PREFIX","SUFFIX","SHAPE","IS_DIGIT"]
rows = [5000,2500,2500,2500,100]
include_static_vectors = false

Because of how MultiHashEmbed works, you need a larger number of rows than you'd expect for a boolean feature. 100 is to be on the safe side, but 20 or 50 is probably also fine.

Right now it's hard to use a feature like POS that's predicted by an earlier component. This should become easier in v3.1.