Help debugging a custom suggester that can return 0 suggestions

[mathcass]

👋🏻 I could use some help with a custom suggester function. Essentially, I'm using a spaCy project to attempt to build out a SpanCategorizer model with a custom suggester. The main idea is that I'd like to have control over the lexicon for candidates. I'll label the data (in Prodigy) with potential candidates to produce training data on spans. Then, the project builds out a SpanCategorizer model so that the model only decides on those candidates. In this case, I was experimenting with a suggester that takes match patterns as input and emits the Ragged array of spans in the docs. If it's relevant, I've tried with a en_core_web_md model here.

Code for the suggester (click to toggle)

def build_pattern_matcher_suggester(phrases: Dict[str, List[Any]]) -> Suggester:
    """Builds a PhraseMatcher using phrases at the ORTH match, returning a Suggester"""

    matcher = Matcher(nlp.vocab)
    for label, phrase_patterns in phrases.items():
        matcher.add(label, phrase_patterns)

    def matcher_suggester(docs: Iterable[Doc], *, ops: Optional[Ops] = None) -> Ragged:
        if ops is None:
            ops = get_current_ops()
        spans = []
        lengths = []
        for doc in docs:
            matches = matcher(doc)
            match_offsets = [[x[1], x[2]] for x in matches]
            length = len(match_offsets)
            if match_offsets:
                spans.append(ops.xp.asarray(match_offsets, dtype="i"))
            lengths.append(length)

        lengths_array = cast(Ints1d, ops.asarray(lengths, dtype="i"))
        if len(spans) > 0:
            output = Ragged(ops.xp.vstack(spans), lengths_array)
        else:
            output = Ragged(ops.xp.zeros((0, 0), dtype="i"), lengths_array)

        assert output.dataXd.ndim == 2
        return output

    return matcher_suggester

Where I'm running into trouble is that on spaCy 3.3, things work exactly as I expect. However, on spaCy 3.4, they don't work. I get the following error, namely that ValueError: all sequence lengths must be >= 0. Indeed, if I drop into a debugger and inspect the input my SpanCat's model, it looks like Ragged(data=array([], shape=(0, 0), dtype=int32), lengths=array([0], dtype=int32), data_shape=(-1, 0), starts_ends=None).

Here's the error I get (click to toggle)

Traceback (most recent call last):
  File "run.py", line 9, in <module>
    nlp("What are you doing")
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/spacy/language.py", line 1031, in __call__
    error_handler(name, proc, [doc], e)
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/spacy/util.py", line 1670, in raise_error
    raise e
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/spacy/language.py", line 1026, in __call__
    doc = proc(doc, **component_cfg.get(name, {}))  # type: ignore[call-arg]
  File "spacy/pipeline/trainable_pipe.pyx", line 56, in spacy.pipeline.trainable_pipe.TrainablePipe.__call__
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/spacy/util.py", line 1670, in raise_error
    raise e
  File "spacy/pipeline/trainable_pipe.pyx", line 52, in spacy.pipeline.trainable_pipe.TrainablePipe.__call__
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/spacy/pipeline/spancat.py", line 275, in predict
    scores = self.model.predict((docs, indices))  # type: ignore
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/model.py", line 315, in predict
    return self._func(self, X, is_train=False)[0]
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/layers/chain.py", line 55, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/layers/chain.py", line 55, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/layers/concatenate.py", line 44, in forward
    Ys, callbacks = zip(*[layer(X, is_train=is_train) for layer in model.layers])
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/layers/concatenate.py", line 44, in <listcomp>
    Ys, callbacks = zip(*[layer(X, is_train=is_train) for layer in model.layers])
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/layers/reduce_last.py", line 21, in forward
    Y, lasts = model.ops.reduce_last(cast(Floats2d, Xr.data), Xr.lengths)
  File "$HOME/src/projects/venv/lib/python3.8/site-packages/thinc/backends/ops.py", line 1204, in reduce_last
    raise ValueError(f"all sequence lengths must be >= 0")
ValueError: all sequence lengths must be >= 0

What seems like is happening here is that my suggester potentially suggests no spans sometimes. And I'm not quite sure why this works on 3.3 but not after upgrading to 3.4. I've looked over both of the ngram suggesters from either versions and aside from some typing updates, I don't see anything obvious I should be doing when no spans exist.

• https://github.com/explosion/spaCy/blob/v3.3.1/spacy/pipeline/spancat.py#L55-L87
• https://github.com/explosion/spaCy/blob/v3.4.3/spacy/pipeline/spancat.py#L55-L87

I've also looked over https://spacy.io/usage/v3-4 and I don't see anything noteworthy about changes in SpanCategorizer that I should be doing. (I updated my config to the latest version, though, but that didn't change anything.)

What am I missing about the way I'm building a custom suggester that I need to be doing here? Or is there a better method I should be looking into here?

[mathcass]

I've been looking more into this over the last few days and here are some notes on what I've found here.

I think the concept of suggesters that potentially emit 0 spans is plausibly supported (the existence of a test for them implies that), but in practice I the default Model for the SpanCategorizer component doesn't seem to support it.

I found a few references (below) suggesting that for various types of operations, you can't expect the model to compute a result for zero-length sequences.

• Fix reductions when applied to zero-length sequences thinc#637
• https://github.com/explosion/thinc/blob/af0e3de47968af09d248dcfb379348b1ff1239a8/website/docs/api-backends.md#opsreduce_max-reduce_max-tagmethod

And suggesters that potentially emit no spans can output zero-length sequences, then in practice empty suggesters aren't really supported.

Two obvious solutions come to mind,

• Look for model architectures that handle zero-length inputs capably, so you can use them in cases like this (likely hard, since it will limit the reductions you're able to do)
• Special case the predict method so that when the suggester returns spans, if non exist, then continue somehow

[monWork]

@mathcass I am running to exactly similar issue with my custom suggester on spacy 3.4. Is it working fine your case on 3.3 ?
@adrianeboyd need a little clarification on your reply of "non-training predict step" , help me understand this.
@mathcass and @adrianeboyd would appreciate your help if you figured out a work around for this.

[adrianeboyd]

The test I originally wrote for zero suggestions wasn't sufficient because it only checked spancat.update and not spancat.predict.

To see the error with zero suggestions for spancat:

# update step runs without issues (it doesn't call `spancat.predict`)
nlp.update(examples)

# predict step crashes with thinc v8.1 (inference, not for training)
nlp.get_pipe("spancat")(doc)
# or more specifically
nlp.get_pipe("spancat").predict([doc])

spacy train alternates the predict step (run to evaluate on dev data) and update steps (update on batches of train data), so it's crashing initially on the first eval attempt before it even starts training.

[mathcass]

@monWork

Is it working fine your case on 3.3 ?

Yes, my code is working on spaCy 3.3. And @merlinsdm mentions it works for them as well.

Though as I discovered (and @adrianeboyd mentioned), this is due to a newer version of the the thinc package.

[adrianeboyd]

I think it can run without crashing with v3.3, but at this point I'm not sure that the model is doing what you intended for zero-length instances.

[adrianeboyd]

See: #11860