Joint NER+REL training exception: operands could not be broadcast together with shapes...

[MaticBernik]

Hello Explosion!
I am trying to train Spacy pipeline for joined Named entity recognition and Relation extraction tasks based on custom RoBERTa Transformer model for Slovenian language.
I copied code for rel_component from example on Spacy github repository and already updated score_relations function to account for predicted relations that are not part of gold annotations.

Now I have problem with get_loss function, as the sizes of truths and scores do not match and therefore gradient can't be calculated. Consequently training stops with an error:

File "/home/matic/Documents/transformer_from_scratch/spacyPipeline/scripts/rel_pipe.py", line 149, in get_loss
    gradient = scores - truths
ValueError: operands could not be broadcast together with shapes (88860,13) (7448,13)

⚠ Aborting and saving the final best model. Encountered exception:
ValueError('operands could not be broadcast together with shapes (88860,13)
(7448,13) ',)

Might it be that number of "predicted" candidate relations does not match the number of gold relations simply because number of predicted entities also doesn't match golds?
Can this issue be addressed via config file?

I am attaching config file, rel_pipe.py code and text file with print-out trace.

config_withRel.cfg.txt
rel_pipe.py.txt
nohup.out.txt

[MaticBernik]

I've seen a few posts where people were trying to jointly train ner+rel and later reported success with no mention of exception I'm describing. But it seems to me like this exception requires some additions to "rel_pipe.py" code?
@polm sorry for pinging you directly but maybe you can help me/give me some direction?

[polm]

Sorry, I've never worked on the REL code before (except for the minor copy/paste fix recently), so it'd take me a bit to dig into the architecture correctly. The guess the issue is with the number of candidates sounds reasonable. I suspect it can't be fixed in the config file.

It might be a similar issue to the NEL component that I've been working on fixing recently. Since that component predates the Listeners pattern, it was doing some seemingly innocuous batch manipulation that broke use with a listener. Take a look at #9691 for some docs on that.

[MaticBernik]

Thank you for your reply polm!
Can maybe @svlandeg help me then (again - sorry for pinging you on this one).
As I understand it you can never really expect predicted relations to match reference-ones. That's why matrix of predicted relations should be cut-down/scaled-up to match relations contained in reference document. And this can ofc be done only after predictions have actually been made.
But in order to sync matrices of scores and truths correctly I first have to know exactly what order do relations follow inside the both matrices. This is hard because I need to know which relations have been dropped from predictions due to probability score thresholding?

Is my thinking correct?
Also could you help me with the problem of retrieving the correct order of relations to go with truths matrix returned by self._examples_to_truth(examples).

[svlandeg]

Hi,

Please refrain from artificially bumping your thread and pinging various maintainers individually. We do our best to help anyone on these forums, but please be advised that this is on a best-effort basis, and the additional pings don't help.

[svlandeg]

Hi @MaticBernik,

It's likely that the current implementation of the REL component needs to be further improved for more realistic use-cases other than the simplistic case that the example project & video dealt with.

Now I have problem with get_loss function, as the sizes of truths and scores do not match and therefore gradient can't be calculated.

I think this is due to how the function _examples_to_truth is implemented.
Where it says

for (e1, e2) in self.model.attrs["get_instances"](eg.reference):

(L190 in the original code),
you should try changing this to getting the instances from eg.predicted instead of eg.reference. I would assume that that will resolve your dimension shape issues - but let us know if it doesn't!

[MaticBernik]

Hi svlandeg!
I can confirm that your fix works!, with some additions:

There are two calls of function _examples_to_truth within rel_pipe.py script.
One from get_loss function and one from initialize function.

_examples_to_truth has to be called in unchanged form from initialize function, since predictions have not yet been made and entities do not exist..
(otherwise nr_instances is 0 ---> _examples_to_truth returns None and ValueError("Call begin_training with relevant entities and relations annotated in at least a few reference examples!") is raised)

But the call from get_loss function needs change of _example_to_truth method as you suggested.
My hackish quickfix to test this was just to add flag to function parameters to turn new code on/off as it is being called from within different functions..

def _examples_to_truth(self, examples: List[Example],mode_ref=False) -> Optional[numpy.ndarray]:
        # check that there are actually any candidate instances in this batch of examples
        nr_instances = 0
        for eg in examples:
            ################
            if mode_ref:
                nr_instances += len(self.model.attrs["get_instances"](eg.reference))
            else:    
                nr_instances += len(self.model.attrs["get_instances"](eg.predicted))
            ################
        if nr_instances == 0:
            print("ZERO INSTANCES RETURNING NONE")
            return None

        truths = numpy.zeros((nr_instances, len(self.labels)), dtype="f")
        c = 0
        for i, eg in enumerate(examples):
            ################
            for (e1, e2) in self.model.attrs["get_instances"](eg.reference if mode_ref else eg.predicted):
            ################    
                gold_label_dict = eg.reference._.rel.get((e1.start, e2.start), {})
                for j, label in enumerate(self.labels):
                    truths[c, j] = gold_label_dict.get(label, 0)
                c += 1
        truths = self.model.ops.asarray(truths)
        return truths

Thanks again for your help with resolving this issue!:)

[svlandeg]

Thanks for the confirmation & further analysis! I'm happy we got this resolved, and we'll take this learning with us when we update/generalize the REL code further in the future :-)