HOW TO: add a custom augmenter

[JeremyCraigMartinez]

ASK: how do I add my own custom augmenter

There is an excellent example to work off of. However, this demonstrates how to either:

• augment your data prior to training using a custom augmenter, OR
• augment your data during training using a pre-existing augmenter

My question is, how could I blend these two approaches and leverage a custom augmenter during training?

---

I wanted to work off of the ner demo. I added a directory with an augmenter

(venv) jeremy.martinez ~/code/projects/pipelines/ner_demo (v3) $ tree augmenters/
augmenters/
└── redact.py

and the contents of redact.py are just a dummy filler for the moment until I can confirm that it is being incorporated

from typing import Callable, Iterator
import random
from functools import partial
from spacy.util import registry
from spacy.training.example import Example


@registry.augmenters("spacy.redact.v1")
def create_redact_augmenter(
    level: float,
) -> Callable[["Language", Example], Iterator[Example]]:
    """Create a data augmentation callback that redacts entities.
    The callback can be added to a corpus or other data iterator during training.

    level (float): The percentage of texts that will be lowercased.
    RETURNS (Callable[[Language, Example], Iterator[Example]]): The augmenter.
    """
    return partial(redact_augmenter, level=level)


def redact_augmenter(
    nlp: "Language",
    example: Example,
    *,
    level: float = 0.0,
) -> Iterator[Example]:
    if random.random() > level:
        yield example
    else:
        print('redact entities...')
        yield example

---

I edited the config.cfg to reflect the following change:

diff --git a/pipelines/ner_demo/configs/config.cfg b/pipelines/ner_demo/configs/config.cfg
index 431a91f..7caf4d8 100644
--- a/pipelines/ner_demo/configs/config.cfg
+++ b/pipelines/ner_demo/configs/config.cfg
@@ -77,7 +77,10 @@ path = ${paths.train}
 max_length = 0
 gold_preproc = false
 limit = 0
-augmenter = null
+
+[corpora.train.augmenter]
+@augmenters = "spacy.redact.v1"
+level = 1.0
 
 [training]
 dev_corpus = "corpora.dev"

---

Now running python -m spacy project run train I get the following error:

catalogue.RegistryError: [E893] Could not find function 'spacy.redact.v1' in function registry 'augmenters'. If you're using a custom function, make sure the code is available. If the function is provided by a third-party package, e.g. spacy-transformers, make sure the package is installed in your environment.

Available names: spacy.combined_augmenter.v1, spacy.lower_case.v1, spacy.orth_variants.v1

Which makes complete sense, since I am not incorporating this redact augmenter anywhere. My question is: how do I incorporate this custom augmenter?

[adrianeboyd]

You include your code the same way you include any custom training code, e.g. with the --code argument for relevant CLI commands:

spacy train config.cfg --code functions.py

See: https://spacy.io/usage/training#custom-code

(Alternatively, if you want to use these functions in many different projects, you can use entry points to provide registered functions from a separate installed python package, and then you don't need any extra --code. You can see an example of this with loggers in the package spacy-loggers: https://github.com/explosion/spacy-loggers/blob/7dd2da40d9461b8a0ce3ad1182ecc5af8563f8b2/setup.cfg#L16-L30)