rel_component issue with training my data

[Sujith1909]

I hope you are doing well. I read an article (https://towardsdatascience.com/how-to-train-a-joint-entities-and-relation-extraction-classifier-using-bert-transformer-with-spacy-49eb08d91b5c) and followed all the steps to train the rel_component spacy with my custom data.

im getting this error while training with tok2vec

"could not determine any instance in the doc"

while training with transformer it is not showing this but the score is staying same for all the iterations.
[2022-06-03 17:26:19,936] [INFO] Set up nlp object from config
[2022-06-03 17:26:19,944] [INFO] Pipeline: ['transformer', 'relation_extractor']
[2022-06-03 17:26:19,948] [INFO] Created vocabulary
[2022-06-03 17:26:19,949] [INFO] Finished initializing nlp object
Downloading: 100% 481/481 [00:00<00:00, 507kB/s]
Downloading: 100% 878k/878k [00:00<00:00, 4.37MB/s]
Downloading: 100% 446k/446k [00:00<00:00, 3.12MB/s]
Downloading: 100% 1.29M/1.29M [00:00<00:00, 7.52MB/s]
Downloading: 100% 478M/478M [00:06<00:00, 72.8MB/s]
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.layer_norm.bias', 'lm_head.dense.weight']

• This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  [2022-06-03 17:26:41,811] [INFO] Initialized pipeline components: ['transformer', 'relation_extractor']
  ✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'relation_extractor']
ℹ Initial learn rate: 0.0
E # LOSS TRANS... LOSS RELAT... REL_MICRO_P REL_MICRO_R REL_MICRO_F SCORE

---

0 0 0.82 2.91 0.03 90.24 0.06 0.00
2 100 129.71 74.40 0.00 0.00 0.00 0.00
5 200 0.00 0.27 0.00 0.00 0.00 0.00
8 300 0.00 0.27 0.00 0.00 0.00 0.00
11 400 0.00 0.27 0.00 0.00 0.00 0.00
14 500 0.00 0.27 0.00 0.00 0.00 0.00
17 600 0.00 0.27 0.00 0.00 0.00 0.00
20 700 0.00 0.27 0.00 0.00 0.00 0.00

I read the previous discussion and tried to increase the max_len from 100 to 300, but still there is no change.
relations_train.txt
rel_trf.txt
rel_tok2vec.txt

I am thinking that there is something wrong with the relations.txt or I am missing an important step.
Any help is appreciated.

[polm]

Instances are generated using the get_instances function, which by default checks for entities within the given distance of each other. If training isn't finding any instances then one of these is true of all docs in your training batch:

1. You have docs without entity labels
2. The entities are too far apart

Scanning your data the info seems fine and it doesn't seem like 2 is the case, so maybe your conversion is going wrong. Can you share the code you are using to convert the JSON into Docs, and the resulting .spacy file? It looks like you would have had to customize the conversion code to get it working.

[Sujith1909]

Hi polm,
Conversion code.zip
I have attached the code which is used to convert, relations.json, spacy before and after conversion.
In the conversion code, I had to change one thing in the code , sometimes doc.char_span() is returning none type of entities I ignored those articles.

[Sujith1909]

Conversion code2.zip
a small change in the conversion code and these are the results.

================================= train_cpu =================================
Running command: /usr/bin/python3 -m spacy train configs/rel_tok2vec.cfg --output training --paths.train data/train.spacy --paths.dev data/dev.spacy -c ./scripts/custom_functions.py
ℹ Saving to output directory: training
ℹ Using CPU
ℹ To switch to GPU 0, use the option: --gpu-id 0

=========================== Initializing pipeline ===========================
[2022-06-05 07:42:08,156] [INFO] Set up nlp object from config
[2022-06-05 07:42:08,165] [INFO] Pipeline: ['tok2vec', 'relation_extractor']
[2022-06-05 07:42:08,169] [INFO] Created vocabulary
[2022-06-05 07:42:08,169] [INFO] Finished initializing nlp object
[2022-06-05 07:42:10,501] [INFO] Initialized pipeline components: ['tok2vec', 'relation_extractor']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'relation_extractor']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS RELAT...  REL_MICRO_P  REL_MICRO_R  REL_MICRO_F  SCORE 
---  ------  ------------  -------------  -----------  -----------  -----------  ------
  0       0          0.34           3.62         0.02        39.02         0.04    0.00
  2     500          2.25          13.87         0.00         0.00         0.00    0.00
  4    1000          0.01           1.55         0.00         0.00         0.00    0.00
  6    1500          0.00           1.61         0.00         0.00         0.00    0.00
  8    2000          0.00           1.48         0.00         0.00         0.00    0.00
 10    2500          0.00           1.51         0.00         0.00         0.00    0.00
 12    3000          0.00           1.41         0.00         0.00         0.00    0.00
 15    3500          0.00           1.45         0.00         0.00         0.00    0.00
 17    4000          0.00           1.45         0.00         0.00         0.00    0.00
 22    4500          0.00           1.36         0.00         0.00         0.00    0.00
 29    5000          0.00           1.35         0.00         0.00         0.00    0.00
 36    5500          0.00           1.35         0.00         0.00         0.00    0.00
 43    6000          0.00           1.35         0.00         0.00         0.00    0.00
 51    6500          0.00           1.37         0.00         0.00         0.00    0.00
 58    7000          0.00           1.35         0.00         0.00         0.00    0.00
 65    7500          0.00           1.38         0.00         0.00         0.00    0.00
 72    8000          0.00           1.28       100.00         2.44         4.76    0.05
 79    8500          0.00           1.35         0.00         0.00         0.00    0.00
 86    9000          0.00           1.35         0.00         0.00         0.00    0.00
 93    9500          0.00           1.33         0.00         0.00         0.00    0.00
100   10000          0.00           1.32         0.00         0.00         0.00    0.00
✔ Saved pipeline to output directory
training/model-last

[Sujith1909]

with transformer (Conversioncode2's spacy file) :

TRAINING :

Running command: /usr/bin/python3 -m spacy train configs/rel_trf.cfg --output training --paths.train data/train.spacy --paths.dev data/dev.spacy -c ./scripts/custom_functions.py --gpu-id 0
ℹ Saving to output directory: training
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2022-06-05 07:54:07,037] [INFO] Set up nlp object from config
[2022-06-05 07:54:07,045] [INFO] Pipeline: ['transformer', 'relation_extractor']
[2022-06-05 07:54:07,048] [INFO] Created vocabulary
[2022-06-05 07:54:07,050] [INFO] Finished initializing nlp object
Downloading: 100% 481/481 [00:00<00:00, 871kB/s]
Downloading: 100% 878k/878k [00:00<00:00, 39.8MB/s]
Downloading: 100% 446k/446k [00:00<00:00, 40.5MB/s]
Downloading: 100% 1.29M/1.29M [00:00<00:00, 56.3MB/s]
Downloading: 100% 478M/478M [00:06<00:00, 73.1MB/s]
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.dense.bias']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[2022-06-05 07:54:26,082] [INFO] Initialized pipeline components: ['transformer', 'relation_extractor']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'relation_extractor']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS RELAT...  REL_MICRO_P  REL_MICRO_R  REL_MICRO_F  SCORE 
---  ------  -------------  -------------  -----------  -----------  -----------  ------
  0       0           0.82           2.91         0.03        90.24         0.06    0.00
  3     100         142.53          74.21         0.00         0.00         0.00    0.00
  6     200           0.00           0.28         0.00         0.00         0.00    0.00
  9     300           0.00           0.29         0.00         0.00         0.00    0.00
 12     400           0.00           0.29         0.00         0.00         0.00    0.00
 15     500           0.00           0.28         0.00         0.00         0.00    0.00
 18     600           0.00           0.28         0.00         0.00         0.00    0.00
 21     700           0.00           0.28         0.00         0.00         0.00    0.00
 24     800           0.00           0.28         0.00         0.00         0.00    0.00
 27     900           0.00           0.28         0.00         0.00         0.00    0.00
 30    1000           0.00           0.28       100.00        14.63        25.53    0.26
✔ Saved pipeline to output directory
training/model-last

TESTING :

================================== evaluate ==================================
Running command: /usr/bin/python3 ./scripts/evaluate.py training/model-best data/test.spacy False

Random baseline:
threshold 0.00 	 {'rel_micro_p': '0.02', 'rel_micro_r': '100.00', 'rel_micro_f': '0.04'}
threshold 0.05 	 {'rel_micro_p': '0.02', 'rel_micro_r': '100.00', 'rel_micro_f': '0.04'}
threshold 0.10 	 {'rel_micro_p': '0.02', 'rel_micro_r': '93.10', 'rel_micro_f': '0.04'}
threshold 0.20 	 {'rel_micro_p': '0.02', 'rel_micro_r': '82.76', 'rel_micro_f': '0.04'}
threshold 0.30 	 {'rel_micro_p': '0.02', 'rel_micro_r': '75.86', 'rel_micro_f': '0.04'}
threshold 0.40 	 {'rel_micro_p': '0.02', 'rel_micro_r': '65.52', 'rel_micro_f': '0.04'}
threshold 0.50 	 {'rel_micro_p': '0.02', 'rel_micro_r': '58.62', 'rel_micro_f': '0.04'}
threshold 0.60 	 {'rel_micro_p': '0.02', 'rel_micro_r': '48.28', 'rel_micro_f': '0.04'}
threshold 0.70 	 {'rel_micro_p': '0.02', 'rel_micro_r': '31.03', 'rel_micro_f': '0.04'}
threshold 0.80 	 {'rel_micro_p': '0.02', 'rel_micro_r': '20.69', 'rel_micro_f': '0.04'}
threshold 0.90 	 {'rel_micro_p': '0.02', 'rel_micro_r': '13.79', 'rel_micro_f': '0.05'}
threshold 0.99 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}
threshold 1.00 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}

Results of the trained model:
threshold 0.00 	 {'rel_micro_p': '0.02', 'rel_micro_r': '100.00', 'rel_micro_f': '0.04'}
threshold 0.05 	 {'rel_micro_p': '18.06', 'rel_micro_r': '44.83', 'rel_micro_f': '25.74'}
threshold 0.10 	 {'rel_micro_p': '18.42', 'rel_micro_r': '24.14', 'rel_micro_f': '20.90'}
threshold 0.20 	 {'rel_micro_p': '5.00', 'rel_micro_r': '3.45', 'rel_micro_f': '4.08'}
threshold 0.30 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}
threshold 0.40 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}
threshold 0.50 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}
threshold 0.60 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}
threshold 0.70 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}
threshold 0.80 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}
threshold 0.90 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}
threshold 0.99 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}
threshold 1.00 	 {'rel_micro_p': '0.00', 'rel_micro_r': '0.00', 'rel_micro_f': '0.00'}

I cant understand what is the problem, is it the lack of data or am i missing something while training?

[polm]

Sorry for the very delayed reply to this. I took a look at your data and I don't understand the structure of it - I think your relations have problems with being too specific and the model can't learn to differentiate them.

The example relation extraction project has less training data than you do, but it only learns two relations: binds or regulates. These are standard binary relations, in that for any two entities, if they have a relation, it's like "X binds Y".

But your relations aren't like that. You have "Suspect Gender". "X Suspect Gender Y" isn't a sentence. I kind of understand what you're trying to extract here, but relation extraction isn't really the right way to structure this. Suspect gender isn't different from officer gender or the gender of other people, for example, so you're asking the model to learn a lot to get that one label. You might do something like "X is-gender Y", which would be more reasonable, but since the number of gender terms you'll see is small, even that might be overkill - I would try a word list and just look for nearby words and see how well that does. (Have you ever seen a sentence like "the bad guy poacher was not a woman"?)

Also it looks like many of your labels are never used? For example in your first training doc, which is like a whole newspaper article, I see only three positive relation annotations, one for "quantity" ([300] live [turtles]) and two for "suspect location". Maybe you have more positive annotations in other docs, and it's normal to have lots of negative annotations for relationship extraction, but that seems quite sparse.

I would recommend you look at other relation extraction datasets and see how labels are constructed.