rel_component issue with training my data

[sanpreet]

I hope you are doing well. I watched video SPACY v3: Custom trainable relation extraction component and followed all the steps to train the rel_component spacy with my custom data.

Training is working for unshuffled data but when data is shuffled, I faced weird message during training (spacy project run train_cpu) and error is as below and i am also attaching the screenshot of that

"could not determine any instance in the doc"

Is this any possible error. Can you help me to understand this?

train.txt and dev.txt are attached for reference.

Any help is appreciated.

dev.txt
train.txt

Thanks

[polm]

Please don't include screenshots of text, just copy/paste it as text.

The error means what it says, namely that there are no annotations for it to predict. This can happen if there are no annotations, or no NER annotations, on your training docs for some reason. Can you share your config?

Training is working for unshuffled data but when data is shuffled, I faced weird message during training (spacy project run train_cpu) and error is as below and i am also attaching the screenshot of that

How are you changing between shuffling and not shuffling data? Unless you're using a custom data loader to stream data data is shuffled by default.

[sanpreet]

Thanks lot polm. Here is the attached files for train and dev files which includes annotations. Could you see the below files

dev.txt
train.txt

I created the .spacy files from the above txt files.

Question about this point: Can you share your config?
Could you please elaborate this point please i.e which files to share.

[polm]

I have the train and dev files you shared before, thanks. Can you describe how you converted them to .spacy files?

Your config file is the file you used in training. It is the configs/rel_tok2vec.cfg in the command you're executing in the screenshot above. Did you used the file unmodified from the example project?

[sanpreet]

Thanks polm. I used the binary_converter.py file for converting .txt file into .spacy file. I have modified the script to make the train, dev and test.spacy using the loop. Just a minor change.

configs/rel_tok2vec.cfg is the same file. I kept it same.

[polm]

OK, I figured out what's going on. The warning is correct that no instances are found, you need to tweak some parameters for this to work with your data.

The original input data in the tutorial is like single sentences, while your data is more like short articles. The candidate generator uses all entity pairs, but there is a max distance between them. So for example you have one document that has entities for "Sarah Lawrence College" and "Fatiah Touray". Those are the only entities, and normally that would be fine, but they are 217 tokens apart and the distance limit is 100. So you have zero instances to learn from.

You should be able to fix this by increasing the distance limit. You may also want to increase the batch size, since it's based on words, and your longish documents are causing you to have small batches of like one document.

Note that in general working with longer documents is hard.

Couple more questions from me:

1. You said this worked for unshuffled data but not shuffled data. I still don't understand what that means, can you explain it?
2. Where did you get the binary_converter.py file? Did you copy that from someone's fork? It's similar to the file in the tutorial but the data format is pretty different.

[sanpreet]

Answers to the Couple more questions from me

• Unshuffled data means json files that i have created from the inception tool are not shuffled before converting them into the .txt file. Spacy accept the .txt files only so have created the script to convert .json file into .txt file
• binary_converter.py is from the this link

[polm]

Thanks for the answers!

[sanpreet]

Some issues to be addresses

• How to increase distance limit and batch size. I guess from the configs file
• What could be the max distance limit one can put.
• What complication could arise from large documents and distance limit

[polm]

You can change the values in the config file and there's no particular limit.

If you make it very high it could make the task harder because your model has more candidates to choose from.

[sanpreet]

Hi Polm, thanks for the answer. I would like to ask you below question

• The average document size is about 1100 words and max distance limit could be 300. So could you suggest me to tweak the config file with max_distance limit 300.
• Should i eliminate those relations that exceeds the max distance limit of 100 (default) during annotation.
• Do you suggest me to go with relation extraction spacy models for large documents

[polm]

I would recommend you increase the limit and see how your model does. If it does poorly it might be because having a high limit makes the problem hard and you should consider whether you need to handle relations between entities that are far apart or not; if you get rid of them, your problem will be easier, but if you need them, you'll need more training data and will still likely not be able to get as high accuracy.

Which of those is better depends on your application.

[sanpreet]

Thanks polm very much and my issue is resolved.

[polm]

The parameter is called max_length, you can see it in the sample project here.

[polm]

That's a different parameter because it's in the corpora section. The max_length we're discussing here is an argument to get_instances, like in the config file I linked to.