ValueError when attempting to run pretraining with custom reader

[bennmcgregor]

I'm trying to pretrain a tok2vec layer using a custom reader, however when I run the pretraining command python -m spacy pretrain config.cfg ./pretraining_output --code ../custom_readers.py I get the following error:

ℹ Using CPU
ℹ Loading config from: config.cfg
✔ Saved config file in the output directory

============== Pre-training tok2vec layer - starting at epoch 0 ==============
  #      # Words   Total Loss     Loss    w/s
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/user/src/venv/lib/python3.10/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/user/src/venv/lib/python3.10/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/user/src/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/user/src/venv/lib/python3.10/site-packages/typer/core.py", line 778, in main
    return _main(
  File "/user/src/venv/lib/python3.10/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/user/src/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/user/src/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/user/src/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/user/src/venv/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/user/src/venv/lib/python3.10/site-packages/spacy/cli/pretrain.py", line 70, in pretrain_cli
    pretrain(
  File "/user/src/venv/lib/python3.10/site-packages/spacy/training/pretrain.py", line 80, in pretrain
    for batch_id, batch in enumerate(batcher(corpus(nlp))):
  File "/user/src/venv/lib/python3.10/site-packages/spacy/training/batchers.py", line 159, in minibatch_by_words
    for seq in seqs:
  File "/user/src/spacy/en_md/../custom_readers.py", line 28, in generate_stream
    for gold_doc in pipe(doc_generator(nlp)):
  File "/user/src/spacy/en_md/../custom_readers.py", line 73, in shuffle_generator_output
    for doc in docs_generator:
  File "/user/src/spacy/en_md/../custom_readers.py", line 58, in select_train_data
    for doc in docs_generator:
  File "/user/src/spacy/en_md/../custom_readers.py", line 23, in doc_generator
    yield nlp(" ".join(text_arr))
  File "/user/src/venv/lib/python3.10/site-packages/spacy/language.py", line 1031, in __call__
    error_handler(name, proc, [doc], e)
  File "/user/src/venv/lib/python3.10/site-packages/spacy/util.py", line 1670, in raise_error
    raise e
  File "/user/src/venv/lib/python3.10/site-packages/spacy/language.py", line 1026, in __call__
    doc = proc(doc, **component_cfg.get(name, {}))  # type: ignore[call-arg]
  File "spacy/pipeline/trainable_pipe.pyx", line 56, in spacy.pipeline.trainable_pipe.TrainablePipe.__call__
  File "/user/src/venv/lib/python3.10/site-packages/spacy/util.py", line 1670, in raise_error
    raise e
  File "spacy/pipeline/trainable_pipe.pyx", line 52, in spacy.pipeline.trainable_pipe.TrainablePipe.__call__
  File "/user/src/venv/lib/python3.10/site-packages/spacy/pipeline/spancat.py", line 278, in predict
    scores = self.model.predict((docs, indices))  # type: ignore
  File "/user/src/venv/lib/python3.10/site-packages/thinc/model.py", line 315, in predict
    return self._func(self, X, is_train=False)[0]
  File "/user/src/venv/lib/python3.10/site-packages/thinc/layers/chain.py", line 55, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/user/src/venv/lib/python3.10/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/user/src/venv/lib/python3.10/site-packages/thinc/layers/chain.py", line 55, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/user/src/venv/lib/python3.10/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/user/src/venv/lib/python3.10/site-packages/thinc/layers/chain.py", line 55, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/user/src/venv/lib/python3.10/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/user/src/venv/lib/python3.10/site-packages/thinc/layers/chain.py", line 55, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/user/src/venv/lib/python3.10/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/user/src/venv/lib/python3.10/site-packages/thinc/layers/maxout.py", line 49, in forward
    nI = model.get_dim("nI")
  File "/user/src/venv/lib/python3.10/site-packages/thinc/model.py", line 175, in get_dim
    raise ValueError(err)
ValueError: Cannot get dimension 'nI' for model 'maxout': value unset

The config I'm using is:

[paths]
train = "./paired_data.spacy"
dev = "./paired_data.spacy"
pretrain = "../../pretrain_data.jsonl"
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","spancat"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 1000
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "sc"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat.suggester]
@misc = "spacy.ngram_range_suggester.v1"
min_size = 1
max_size = 250

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "train_reader.v1"
path = ${paths.dev}
limit = 100
train_test_split_seed = 78910
shuffle_seed = 578663
train_split = 0.8
shuffle_buf_size = 250

[corpora.pretrain]
@readers = "pretrain_reader.v1"
path = ${paths.pretrain}
limit = -1
train_test_split_seed = 12345678910
shuffle_seed = 1987654321
shuffle_buf_size = 250

[corpora.train]
@readers = "train_reader.v1"
path = ${paths.train}
limit = -1
train_test_split_seed = 123456
shuffle_seed = 4000
train_split = 0.8
shuffle_buf_size = 250

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
max_epochs = -1
patience = 500
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

[pretraining]
max_epochs = 1000
dropout = 0.2
n_save_every = null
n_save_epoch = null
component = "tok2vec"
layer = ""
corpus = "corpora.pretrain"

[pretraining.batcher]
@batchers = "spacy.batch_by_words.v1"
size = 3000
discard_oversize = false
tolerance = 0.2
get_length = null

[pretraining.objective]
@architectures = "spacy.PretrainCharacters.v1"
maxout_pieces = 3
hidden_size = 300
n_characters = 4

[pretraining.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = true
eps = 0.00000001
learn_rate = 0.001

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

The custom pretraining reader simply streams in data from a very, very large .jsonl file (I followed the example in https://spacy.io/usage/training#custom-code-readers-batchers). The text data in the file is whitespace separated, hence the need to reconstruct it. The custom_readers.py code is below:

import json
from functools import reduce
from typing import Callable, Iterator, List
import spacy
from spacy.training import Example
from spacy.language import Language
from spacy.tokens import DocBin
import random

class ReaderData:
    def __init__(self, train_test_split_seed: int, shuffle_seed: int, train_split: float, shuffle_buf_size: int):
        self.train_test_split_seed = train_test_split_seed
        self.shuffle_seed = shuffle_seed
        self.train_split = train_split
        self.shuffle_buf_size = shuffle_buf_size

@spacy.registry.readers("pretrain_reader.v1")
def stream_pretrain_data(path: str, limit: int, train_test_split_seed: int, shuffle_seed: int, shuffle_buf_size: int) -> Callable[[Language], Iterator[Example]]:
    def doc_generator(nlp):
        with open(path, "r") as f:
            for line in f:
                text_arr = json.loads(line)["t"]
                yield nlp(" ".join(text_arr))

    def generate_stream(nlp):
        count = 0
        pipe = compose(ReaderData(train_test_split_seed, shuffle_seed, 1, shuffle_buf_size), select_train_data, shuffle_generator_output)
        for gold_doc in pipe(doc_generator(nlp)):
            if count >= limit and limit > 0:
                break
            predict_doc = nlp(gold_doc.text)
            yield Example(predict_doc, gold_doc)
            count += 1

    return generate_stream

@spacy.registry.readers("train_reader.v1")
def stream_train_data(path: str, limit: int, train_test_split_seed: int, shuffle_seed: int, train_split: float, shuffle_buf_size: int) -> Callable[[Language], Iterator[Example]]:
    def doc_generator(nlp):
        doc_bin = DocBin().from_disk(path)
        for doc in doc_bin.get_docs(nlp.vocab):
            yield doc

    def generate_stream(nlp):
        count = 0
        pipe = compose(ReaderData(train_test_split_seed, shuffle_seed, train_split, shuffle_buf_size), select_train_data, shuffle_generator_output)
        for gold_doc in pipe(doc_generator(nlp)):
            if count >= limit and limit > 0:
                break
            predict_doc = nlp(gold_doc.text)
            yield Example(predict_doc, gold_doc)
            count += 1

    return generate_stream

def select_train_data(docs_generator, data: ReaderData):
    random.seed(data.train_test_split_seed)
    for doc in docs_generator:
        num = random.randrange(1, 101)
        if num <= data.train_split * 100:
            yield doc

def select_test_data(docs_generator, data: ReaderData):
    random.seed(data.train_test_split_seed)
    for doc in docs_generator:
        num = random.randrange(1, 101)
        if num > data.train_split * 100:
            yield doc

def shuffle_generator_output(docs_generator, data: ReaderData):
    random.seed(data.shuffle_seed)
    buf = []
    for doc in docs_generator:
        buf.append(doc)
        if len(buf) >= data.shuffle_buf_size:
            break
    
    while len(buf) > 0:
        idx = random.randrange(0, len(buf))
        yield buf.pop(idx)
        try:
            doc = next(docs_generator)
            buf.append(doc)
        except StopIteration:
            continue
    print(buf)

def compose(data, *funcs):
    return lambda x: reduce(lambda f, g: g(f, data), list(funcs), x)

I've been having a lot of trouble figuring out what the problem is. I've compared my .cfg file to the default examples and as far as I can tell I'm not missing anything. I've seen posts for similar errors with training, but they all seem to involve very specific issues with model/training configuration not relevant to my use case. I also haven't found posts for pretraining specifically, because this error happens with the pretraining command. Perhaps someone can point me in the right direction or help me figure out the issue?

[bennmcgregor]

I solved my own problem! My custom reader had incorrect output. Once I copied the implementation of JsonlCorpus, it started to work. Still not sure what exactly was going wrong in my code (so I would appreciate some input), but I did get it working. You can see the changes:

def stream_pretrain_data(path: str, limit: int, train_test_split_seed: int, shuffle_seed: int, shuffle_buf_size: int) -> Callable[[Language], Iterator[Example]]:
    def doc_generator(nlp: "Language"):
        with open(path, "r") as f:
            for line in f:
                text_arr = json.loads(line)["t"]
                yield nlp.make_doc(" ".join(text_arr))

    def generate_stream(nlp):
        count = 0
        pipe = compose(ReaderData(train_test_split_seed, shuffle_seed, 1, shuffle_buf_size), select_train_data, shuffle_generator_output)
        for gold_doc in pipe(doc_generator(nlp)):
            if count >= limit and limit > 0:
                break
            words = [w.text for w in gold_doc]
            spaces = [bool(w.whitespace_) for w in gold_doc]
            yield Example(gold_doc, Doc(nlp.vocab, words=words, spaces=spaces))
            count += 1

    return generate_stream```