Strange dev accuracy during dependency parse model fine-tuning

[danielvasic]

Hello everyone,

I have a question about dependency parsing model, firstly I know that some of the things I will be referencing will not make sense but I have some strange behaviour during training of my model in English language. First we have developed a new corpus that contains around 8000 tokens annotated in CONLL format. I successfully converted the corpus in spaCy format and trained and tested the model on it. To clarify I used the same dataset for training and development (maybe this is the problem?) And the results of the training is as shown:

ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2021-06-27 16:20:59,969] [INFO] Set up nlp object from config
[2021-06-27 16:20:59,981] [INFO] Pipeline: ['transformer', 'tagger', 'parser']
[2021-06-27 16:20:59,981] [INFO] Resuming training for: ['parser', 'transformer']
[2021-06-27 16:21:00,107] [INFO] Created vocabulary
[2021-06-27 16:21:00,107] [INFO] Finished initializing nlp object
[2021-06-27 16:21:01,036] [INFO] Initialized pipeline components: ['tagger']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'tagger', 'parser']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS TAGGER  LOSS PARSER  TAG_ACC  DEP_UAS  DEP_LAS  SENTS_F  SCORE 
===  ------  -------------  -----------  -----------  -------  -------  -------  -------  ------
  0       0         122.47       208.82        56.49     0.01    41.98    37.08    79.97    0.20
 23     200       50802.24    105471.62     27402.34    84.26    22.53    21.98    97.69    0.53
 45     400        8824.06     28749.12      4446.29    99.11    22.31    21.92    93.46    0.61
 67     600        2585.61      1094.95      1219.58    99.89    22.12    21.64    92.06    0.61
 89     800        1113.23       125.76       533.06    99.99    22.10    20.97    93.53    0.61

The LAS and UAS keep decreasing until the end of training. As in TAG_ACC I would expect the LAS and UAS scores would increase (since the same dataset is used for development and training, but as You can se this is not the case).

This is my configuration file:

[paths]
train = null
dev = null

[system]
gpu_allocator = "pytorch"

[nlp]
lang = "en"
pipeline = ["transformer","tagger","parser"]
batch_size = 128

[components]

[components.transformer]
source = "en_core_web_trf"
component = "transformer"

[components.tagger]
factory = "tagger"

[components.tagger.model]
@architectures = "spacy.Tagger.v1"
nO = null

[components.tagger.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.tagger.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.parser]
source = "en_core_web_trf"
component = "parser"

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 500

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 5e-5

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256

[initialize]
vectors = null

[adrianeboyd]

Are you using the same dependency annotation scheme as in en_core_web_trf?

[adrianeboyd]

Hmm, it definitely seems like something is wrong with the training data. Can attach a handful of sentences in CoNLL-U format? You can replace all the words with _ if you can't share the source texts. It's mainly to look at the other columns.

The error about " is that " isn't a known tag for the tagger, which is true for the existing tagset. Are you sure that this data follows the en_core_web annotation scheme?

E.g., to inspect the first sentence:

import spacy
from spacy.tokens import DocBin

db = DocBin().from_disk("train.spacy")
nlp = spacy.blank("en")
docs = list(db.get_docs(nlp.vocab))

spacy.displacy.serve(list(docs[0].sents)[0])

[danielvasic]

Dear @adrianeboyd ,

This is one sentence in the dataset:

# sent_id = 14
# text = All passengers with tickets can board now.
1	All 	all	DET	DT	_	2	det	_	_
2	passengers 	passenger	NOUN	NNS	_	6	nsubj	_	_
3	with 	with	ADP	IN	_	2	prep	_	_
4	tickets 	ticket	NOUN	NNS	_	3	pobj	_	_
5	can 	can	VERB	MD	_	6	aux	_	_
6	board 	board	VERB	VB	_	0	root	_	_
7	now	now	ADV	RB	_	6	advmod	_	_
8	.	.	PUNCT	.	_	6	punct	_	_

I get it now, the KeyError is because I used " as a UPOS tag I will correct it in the corpus. I will try to inspect the dependency trees and get back soon, thank you for the help.

[danielvasic]

After trying the DocBin code I get following error:

AttributeError                            Traceback (most recent call last)
<ipython-input-8-4be07af99066> in <module>()
      2 from spacy.tokens import DocBin
      3 
----> 4 db = DocBin().from_disk("datasets/dataset.spacy")
      5 nlp = spacy.blank("en")
      6 docs = list(db.get_docs(nlp.vocab))

AttributeError: 'DocBin' object has no attribute 'from_disk' 

The spacy info is:

============================== Info about spaCy ==============================

spaCy version    3.0.6                         
Location         /usr/local/lib/python3.7/dist-packages/spacy
Platform         Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
Python version   3.7.10                        
Pipelines        en_core_web_trf (3.0.0)       

[adrianeboyd]

That must be coming from spacy v2 somehow. Be sure you have a clean install of spacy v3 after uninstalling all traces of spacy v2. pip doesn't always upgrade cleanly.

[danielvasic]

Yes, I'm using Google Colab which is preinstaled with spaCy 2.2.3. I have checked the data and It seems fine to me. Interesting thing is that when I test the model it shows better results. I the attachment is the dataset.
dataset.txt