Why rel_component project's output score is different than f-score?

[wallybeamm]

I am training a pipiline with [ner, transformer, rel]. I am using the rel_component project shared by Spacy and my custom dataset. After running the training for 500 steps, I noticed that model is not learning anything (recall and precision scores are zero). I also noticed that the Score is not changing at all and different than the f-score.

E    #       LOSS RELAT...  ENTS_F  ENTS_P  ENTS_R  REL_MICRO_P  REL_MICRO_R  REL_MICRO_F  SCORE 
---  ------  -------------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           6.59   93.95   88.59  100.00         0.03        51.64         0.06    0.47
Token indices sequence length is longer than the specified maximum sequence length for this model (771 > 512). Running this sequence through the model will result in indexing errors
  7     100         603.16   93.95   88.59  100.00         0.03        38.95         0.06    0.47
 15     200         364.61   93.95   88.59  100.00         0.03         8.87         0.06    0.47
 23     300         108.07   93.95   88.59  100.00         0.01         0.06         0.02    0.47
 30     400          20.13   93.95   88.59  100.00         0.00         0.00         0.00    0.47
 38     500           4.97   93.95   88.59  100.00         0.00         0.00         0.00    0.47

This is weird because I am using the following settings for score weights.

[training.score_weights]
rel_micro_p = 0.0
rel_micro_r = 0.0
rel_micro_f = 1.0

Is Spacy calculating the final Score by using the test dataset?

Because on my dataset, I have 21 relation types. The documents in the train and the dev set are usually longer than 500 tokens. However, the test set contains documents with 21 tokens.

	Train	Dev	Test
Number of Relations	2059	2007	21
Updated # Relations(distance < 149)	2042	1711	21

[rmitsch]

Hi @wallybeamm, by default ents_f is set to a non-zero value for the calculation of SCORE. Try setting it to 0?
[training.score_weights]
rel_micro_p = 0.0
rel_micro_r = 0.0
rel_micro_f = 1.0
ents_f = 0.0

[wallybeamm]

It's just a set to confirm that there is no coding issue. I am using it to test the parser and overfit the model. Also, I am using it as the first step to evaluate the model's performance. I think that If the model is not able to generate reasonable results on that dataset, it's not necessary to do further tests with a larger corpus

[rmitsch]

Can you share an excerpt of your dataset, reduced to a single relation type? And your scores your obtained with a single relation type?

[wallybeamm]

Here is a short example from my dataset. Let me know if a longer one is needed. This one has 2 relations for the type "orgInDate". Here is the text and binarized form of it for a single relation type:

{"id": 135, "text": "Tickets Informative guide Opening hours & prices EN ES EN CA EU GL.\nHome /.\nCollection.\nCarmen Alexander Calder Lawnton, Pennsylvania, USA, 1898 - New York, USA, 1976.\nDate: 1974.\nMaterial: Aluminium sheet, iron sheet and paint.\nTechnique: Cut out, riveted and welding.\nDimensions: 1156 x 762 cm.\nCategory:&nbsp Sculpture.\nEntry date: 1988.\nObservations: Entry date: 1988 (from the redistribution of the Museo Español de Arte Contemporáneo [MEAC] collection).\nRegister number: AS10770.\nOn display in: Sabatini Garden.\nThis piece, from 1974, was acquired by the State months before the Centro de Arte Reina Sofía was decreed a National Museum, and was first installed in the patio of the Sabatini Building in 1992. Carmen is a monumental standing mobile, which follows the typology Alexander Calder began in 1958 with La Spirale, which he built for the UNESCO headquarters in Paris. It consists of a fixed supporting section with a mobile structure on top. Eight riveted blades are moved by air currents, giving the piece the playful, optimistic component so essential to Calder’s art while also producing a spontaneous contrast between the aesthetic industrial solidity of the base and the variability of the upper part. Like other pieces by the artist, the title of Carmen is a woman’s name, which, while it certainly has literary and musical connotations, does not make the sculpture a simple illustration, but actually functions as an independent work, as pointed out by Jean-Paul Sartre in 1946: “Calder does not ‘suggest’ anything: it captures genuine living movements and shapes them. ‘Mobiles’ have no meaning, make you think of nothing but themselves. They are, that is all; they are absolutes.”\n Carmen Fernández Aparicio.\n1937-1943 Room 205.09.\n1936 1969.\n1937-1947 (circa) Room 205.08.\n1969 Room 411.\nSee all the artworks.\nRelated visits Other ways to explore the Collection from different points of view. Download of brochures to prepare the tour Artworks in public places.\nCargando...\n", "relations": [{"id": 538, "from_id": 310154, "to_id": 310153, "type": "orgInDate"}, {"id": 539, "from_id": 310155, "to_id": 310156, "type": "orgInDate"}, {"id": 540, "from_id": 310157, "to_id": 310158, "type": "itemOfArtist"}, {"id": 541, "from_id": 310160, "to_id": 310158, "type": "itemOfArtist"}, {"id": 542, "from_id": 310158, "to_id": 310159, "type": "artistInDate"}], "entities": [{"id": 310144, "start_offset": 95, "end_offset": 111, "label": "ARTIST"}, {"id": 310145, "start_offset": 112, "end_offset": 138, "label": "GPE"}, {"id": 310146, "start_offset": 140, "end_offset": 144, "label": "DATE"}, {"id": 310147, "start_offset": 147, "end_offset": 160, "label": "GPE"}, {"id": 310148, "start_offset": 162, "end_offset": 166, "label": "DATE"}, {"id": 310149, "start_offset": 174, "end_offset": 178, "label": "DATE"}, {"id": 310150, "start_offset": 335, "end_offset": 339, "label": "DATE"}, {"id": 310151, "start_offset": 367, "end_offset": 371, "label": "DATE"}, {"id": 310152, "start_offset": 404, "end_offset": 439, "label": "MUSEUM"}, {"id": 310153, "start_offset": 535, "end_offset": 539, "label": "DATE"}, {"id": 310154, "start_offset": 585, "end_offset": 611, "label": "MUSEUM"}, {"id": 310155, "start_offset": 626, "end_offset": 641, "label": "MUSEUM"}, {"id": 310156, "start_offset": 708, "end_offset": 712, "label": "DATE"}, {"id": 310157, "start_offset": 714, "end_offset": 720, "label": "WORK_OF_ART"}, {"id": 310158, "start_offset": 781, "end_offset": 797, "label": "ARTIST"}, {"id": 310159, "start_offset": 807, "end_offset": 811, "label": "DATE"}, {"id": 310160, "start_offset": 817, "end_offset": 827, "label": "WORK_OF_ART"}, {"id": 310161, "start_offset": 875, "end_offset": 880, "label": "GPE"}, {"id": 310162, "start_offset": 1071, "end_offset": 1077, "label": "ARTIST"}, {"id": 310163, "start_offset": 1267, "end_offset": 1273, "label": "WORK_OF_ART"}, {"id": 310164, "start_offset": 1474, "end_offset": 1490, "label": "ARTIST"}, {"id": 310165, "start_offset": 1494, "end_offset": 1498, "label": "DATE"}, {"id": 310166, "start_offset": 1501, "end_offset": 1507, "label": "ARTIST"}, {"id": 310167, "start_offset": 1708, "end_offset": 1733, "label": "ARTIST"}, {"id": 310168, "start_offset": 1735, "end_offset": 1744, "label": "DATE"}, {"id": 310169, "start_offset": 1758, "end_offset": 1767, "label": "DATE"}, {"id": 310170, "start_offset": 1769, "end_offset": 1778, "label": "DATE"}, {"id": 310171, "start_offset": 1800, "end_offset": 1804, "label": "DATE"}]}

document.spacy

I used this document as both train and dev data so it can overfit. Surprisingly this is working well now.

E # LOSS TRANS... LOSS RELAT... ENTS_F ENTS_P ENTS_R REL_MICRO_P REL_MICRO_R REL_MICRO_F SCORE

---

0 0 0.00 0.37 98.25 96.55 100.00 0.00 0.00 0.00 0.00
100 100 0.00 34.32 98.25 96.55 100.00 0.00 0.00 0.00 0.00
200 200 0.00 16.96 98.25 96.55 100.00 0.00 0.00 0.00 0.00
300 300 0.00 4.39 98.25 96.55 100.00 0.00 0.00 0.00 0.00
400 400 0.00 2.21 98.25 96.55 100.00 0.00 0.00 0.00 0.00
500 500 0.00 1.41 98.25 96.55 100.00 100.00 100.00 100.00 1.00
600 600 0.00 0.93 98.25 96.55 100.00 100.00 100.00 100.00 1.00
700 700 0.00 0.67 98.25 96.55 100.00 100.00 100.00 100.00 1.00
800 800 0.00 0.57 98.25 96.55 100.00 100.00 100.00 100.00 1.00
900 900 0.00 0.45 98.25 96.55 100.00 100.00 100.00 100.00 1.00
1000 1000 0.00 0.40 98.25 96.55 100.00 100.00 100.00 100.00 1.00

I am unsure how to share, but I got the following results with a longer document with 270 relations for a single type. Should I copy-paste the whole document with annotations?
.

[rmitsch]

I used this document as both train and dev data so it can overfit. Surprisingly this is working well now.

So if you reduce your data to a single relation type and don't merge train and dev set, how do your metrics look like?

[wallybeamm]

It's converging to a very small number.

E    #       LOSS TRANS...  LOSS RELAT...  ENTS_F  ENTS_P  ENTS_R  REL_MICRO_P  REL_MICRO_R  REL_MICRO_F  SCORE 
---  ------  -------------  -------------  ------  ------  ------  -----------  -----------  -----------  ------
wandb: Adding directory to artifact (/content/gdrive/MyDrive/ArtNLP/REL/corpus)... Done. 0.0s
  0       0           0.00           0.30   94.03   88.72  100.00         0.33        35.49         0.66    0.01
100     100           0.00          27.72   94.03   88.72  100.00         0.25        19.44         0.50    0.01
200     200           0.00          14.61   94.03   88.72  100.00         0.03         0.31         0.05    0.00
wandb: Adding directory to artifact (./training/model-last)... Done. 1.6s
300     300           0.00           4.05   94.03   88.72  100.00         0.00         0.00         0.00    0.00
400     400           0.00           2.07   94.03   88.72  100.00         0.00         0.00         0.00    0.00
wandb: Adding directory to artifact (./training/model-last)... Done. 2.7s
500     500           0.00           1.87   94.03   88.72  100.00         0.00         0.00         0.00    0.00
600     600           0.00           1.79   94.03   88.72  100.00         1.05         0.31         0.48    0.00
wandb: Adding directory to artifact (./training/model-last)... Done. 1.7s
700     700           0.00           1.75   94.03   88.72  100.00         2.11         0.62         0.95    0.01
800     800           0.00           1.72   94.03   88.72  100.00         2.13         0.62         0.96    0.01
wandb: Adding directory to artifact (./training/model-last)... Done. 1.7s
900     900           0.00           1.70   94.03   88.72  100.00         2.08         0.62         0.95    0.01
1000    1000           0.00           1.68   94.03   88.72  100.00         2.08         0.62         0.95    0.01
wandb: Adding directory to artifact (./training/model-last)... Done. 3.3s
1100    1100           0.00           1.67   94.03   88.72  100.00         1.89         0.62         0.93    0.01
1200    1200           0.00           1.66   94.03   88.72  100.00         1.87         0.62         0.93    0.01
wandb: Adding directory to artifact (./training/model-last)... Done. 1.6s
1300    1300           0.00           1.65   94.03   88.72  100.00         1.79         0.62         0.92    0.01
1400    1400           0.00           1.65   94.03   88.72  100.00         1.72         0.62         0.91    0.01
wandb: Adding directory to artifact (./training/model-last)... Done. 1.8s
1500    1500           0.00           1.64   94.03   88.72  100.00         0.86         0.31         0.45    0.00
1600    1600           0.00           1.64   94.04   88.74  100.00         1.63         0.62         0.89    0.01
wandb: Adding directory to artifact (./training/model-last)... Done. 2.5s
1700    1700           0.00           1.63   94.05   88.77  100.00         1.47         0.62         0.87    0.01
1800    1800           0.00           1.62   94.05   88.77  100.00         1.43         0.62         0.86    0.01
wandb: Adding directory to artifact (./training/model-last)... Done. 1.7s
1900    1900           0.00           1.62   94.05   88.77  100.00         2.13         0.93         1.29    0.01
2000    2000           0.00           1.62   94.05   88.77  100.00         2.68         1.23         1.69    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 1.7s
2100    2100           0.00           1.62   94.05   88.77  100.00         2.08         0.93         1.28    0.01
2200    2200           0.00           1.62   94.05   88.77  100.00         2.01         0.93         1.27    0.01
wandb: Adding directory to artifact (./training/model-last)... Done. 1.7s
2300    2300           0.00           1.61   94.05   88.77  100.00         1.99         0.93         1.26    0.01
2400    2400           0.00           1.61   94.05   88.77  100.00         2.04         0.93         1.27    0.01
wandb: Adding directory to artifact (./training/model-last)... Done. 3.8s
2500    2500           0.00           1.61   94.05   88.77  100.00         2.04         0.93         1.27    0.01
2600    2600           0.00           1.61   94.05   88.77  100.00         2.58         1.23         1.67    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 3.6s
2700    2700           0.00           1.60   94.05   88.77  100.00         2.52         1.23         1.66    0.02
2800    2800           0.00           1.61   94.05   88.77  100.00         1.96         0.93         1.26    0.01
wandb: Adding directory to artifact (./training/model-last)... Done. 3.2s
2900    2900           0.00           1.60   94.05   88.77  100.00         2.38         1.23         1.63    0.02
3000    3000           0.00           1.60   94.05   88.77  100.00         2.38         1.23         1.63    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 3.1s
3100    3100           0.00           1.60   94.05   88.77  100.00         3.89         2.16         2.78    0.03
3200    3200           0.00           1.60   94.05   88.77  100.00         2.92         1.54         2.02    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 1.8s
3300    3300           0.00           1.60   94.05   88.77  100.00         2.89         1.54         2.01    0.02
3400    3400           0.00           1.60   94.05   88.77  100.00         2.96         1.54         2.03    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 1.8s
3500    3500           0.00           1.60   94.05   88.77  100.00         3.30         1.85         2.37    0.02
3600    3600           0.00           1.60   94.05   88.77  100.00         2.84         1.54         2.00    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 1.8s
3700    3700           0.00           1.59   94.05   88.77  100.00         2.82         1.54         2.00    0.02
3800    3800           0.00           1.59   94.05   88.77  100.00         3.30         1.85         2.37    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 1.8s
3900    3900           0.00           1.59   94.05   88.77  100.00         2.87         1.54         2.01    0.02
4000    4000           0.00           1.59   94.05   88.77  100.00         3.28         1.85         2.37    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 2.9s
4100    4100           0.00           1.59   94.05   88.77  100.00         3.17         1.85         2.34    0.02
4200    4200           0.00           1.59   94.05   88.77  100.00         3.17         1.85         2.34    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 1.9s
4300    4300           0.00           1.59   94.05   88.77  100.00         3.39         1.85         2.40    0.02
4400    4400           0.00           1.59   94.05   88.77  100.00         3.61         2.16         2.70    0.03
wandb: Adding directory to artifact (./training/model-last)... Done. 2.2s
4500    4500           0.00           1.59   94.05   88.77  100.00         3.24         1.85         2.36    0.02
4600    4600           0.00           1.59   94.05   88.77  100.00         3.17         1.85         2.34    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 2.0s
4700    4700           0.00           1.59   94.04   88.74  100.00         3.28         1.85         2.37    0.02
4800    4800           0.00           1.59   94.04   88.74  100.00         3.39         1.85         2.40    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 2.7s
4900    4900           0.00           1.58   94.05   88.77  100.00         3.33         1.85         2.38    0.02
5000    5000           0.00           1.58   94.05   88.77  100.00         3.39         1.85         2.40    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 2.0s
5100    5100           0.00           1.58   94.05   88.77  100.00         4.28         2.47         3.13    0.03
5200    5200           0.00           1.58   94.05   88.77  100.00         3.39         1.85         2.40    0.02
wandb: Adding directory to artifact (./training/model-last)... Done. 1.8s
5300    5300           0.00           1.58   94.05   88.77  100.00         3.78         2.16         2.75    0.03
5400    5400           0.00           1.58   94.05   88.77  100.00         3.85         2.16         2.77    0.03
wandb: Adding directory to artifact (./training/model-last)... Done. 4.1s
5500    5500           0.00           1.59   94.05   88.77  100.00         3.98         2.16         2.80    0.03
5600    5600           0.00           1.58   94.05   88.77  100.00         4.37         2.47         3.16    0.03
wandb: Adding directory to artifact (./training/model-last)... Done. 2.1s
5700    5700           0.00           1.58   94.05   88.77  100.00         4.52         2.47         3.19    0.03
5800    5800           0.00           1.58   94.05   88.77  100.00         4.68         2.47         3.23    0.03
wandb: Adding directory to artifact (./training/model-last)... Done. 2.9s
5900    5900           0.00           1.59   94.06   88.79  100.00         5.20         2.78         3.62    0.04
6000    6000           0.00           1.58   94.07   88.81  100.00         4.32         2.16         2.88    0.03
wandb: Adding directory to artifact (./training/model-last)... Done. 4.3s
6100    6100           0.00           1.58   94.07   88.81  100.00         4.24         2.16         2.86    0.03
6200    6200           0.00           1.58   94.07   88.81  100.00         4.19         2.16         2.85    0.03
wandb: Adding directory to artifact (./training/model-last)... Done. 4.2s
6300    6300           0.00           1.58   94.07   88.81  100.00         4.43         2.16         2.90    0.03
6400    6400           0.00           1.58   94.08   88.83  100.00         5.36         2.78         3.66    0.04
wandb: Adding directory to artifact (./training/model-last)... Done. 2.3s
6500    6500           0.00           1.58   94.08   88.83  100.00         4.52         2.16         2.92    0.03
6600    6600           0.00           1.58   94.08   88.83  100.00         5.45         2.78         3.68    0.04
wandb: Adding directory to artifact (./training/model-last)... Done. 1.9s
6700    6700           0.00           1.58   94.08   88.83  100.00         5.19         2.47         3.35    0.03
6800    6800           0.00           1.58   94.08   88.83  100.00         5.42         2.78         3.67    0.04
wandb: Adding directory to artifact (./training/model-last)... Done. 2.2s
6900    6900           0.00           1.58   94.08   88.83  100.00         5.19         2.47         3.35    0.03
7000    7000           0.00           1.58   94.08   88.83  100.00         5.04         2.16         3.02    0.03
7100    7100           0.00           1.58   94.08   88.83  100.00         5.00         2.16         3.02    0.03
7200    7200           0.00           1.58   94.08   88.83  100.00         5.67         2.47         3.44    0.03
...
33200   33200           0.00           1.57   94.08   88.83  100.00         8.11         2.78         4.14    0.04