Patchcore failing unexplainably

[siebe-hiel]

I am trying to do Anomly Detection on a set of injection moulding products, using PatchCore. These products get a text printed on them, which in production sometimes gets printed wrong, or not printed at all. Whenever part of a letter is missing, PatchCore is able to segment the part of the letter which is misprinted. However, whenever an entire letter or the entire word is absent, PatchCore treats this as a nomalous sample. I have tried changing backbones, num_neighbours, and pretty much all other parameters. My nominal dataset is consisting only of product which have the text printed correctly.
I have created an artifical example to show what is looks like. Does anybody have a solution to this, or can point me in the right direction? Thank you in advance.

Entire word is missing:

Part of a letter is missing: