Batched augmentation in spaCy, is it possible?

[KennethEnevoldsen]

Currently, spacy's augmentation takes in one example as an input and returns an iterator of the augmented examples. Something like this:

def augmenter(
    nlp: Language, example: Example, level: float
) -> Iterator[Example]:
    # augmentation code
    yield augmented_example

However, if I want to do batched augmentations (for instance using a back-translation) to make more efficient augmentations during training, would it be possible to use the current setup?

I feel like it should be using a class but can't seem to work my way around the one Example as input limitation. Am I missing something?

[adrianeboyd]

Currently with spacy.Corpus.v1 and Corpus, no. With a custom corpus reader, sure.

I haven't looked super-carefully into the details, but I think you'd only need to override Corpus.__call__ to do some level of batching with the augmenter.

[KennethEnevoldsen]

Ah okay, thanks! Making an augmenter that applies to the whole corpus is probably the best way to go then.