No POS tags (only, DEP and TAG) with trained Transformer model

[maxtrem]

Hi,

I have trained a Spacy Transformer model on Swedish (ud-taalbanken) using the new Quickstart Config, here is the config:

[paths]
train = null
dev = null

[system]
gpu_allocator = "pytorch"

[nlp]
lang = "sv"
pipeline = ["transformer","tagger","parser"]
tokenizer = {"@tokenizers": "spacy.Tokenizer.v1"}

[components]

[components.transformer]
factory = "transformer"

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "KB/bert-base-swedish-cased"
tokenizer_config = {"use_fast": true}

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.tagger]
factory = "tagger"

[components.tagger.model]
@architectures = "spacy.Tagger.v1"
nO = null

[components.tagger.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.tagger.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.parser]
factory = "parser"

[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v1"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = false
nO = null

[components.parser.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.parser.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 500

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 4000
initial_rate = 5e-5



[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256

[initialize]
vectors = null

I followed further steps on https://nightly.spacy.io/usage/training#quickstart

However it does not train POS tags, only the fine-grained TAG and DEP is trained:

ℹ Pipeline: ['transformer', 'tagger', 'parser']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS TAGGER  LOSS PARSER  TAG_ACC  DEP_UAS  DEP_LAS  SENTS_F  SCORE 
---  ------  -------------  -----------  -----------  -------  -------  -------  -------  ------
  0       0        2593.64      1292.14      1887.99     5.65    19.15     0.73     0.00    0.08
  5     200      442000.92    350206.54    324202.31    50.84    81.93    74.14    92.91    0.64

As far as I can see the rest works:

>>>import spacy 
>>>nlp = spacy.load('sv_ud_talbanken')
>>>doc = nlp("Jag bor i ett rött hus vid en sjö.")
>>>token = doc[1]
>>>token.dep, token.dep_
(8206900633647566924, 'ROOT')
>>>token.tag_
VB|PRS|AKT

# No POS:
>>>token.pos, token.pos_
(0, '')

When training the 'classic' spacy model pipelines/tagger_parser_ud everything works as expected and I also get POS loss and POS are included in the model.

How can I include POS-tags for the transformer model?

Thanks in advance!

[adrianeboyd]

For POS, add a morphologizer to the pipeline, too. You can see the config with the morphologizer in the tagger_parser_ud example:

https://github.com/explosion/projects/blob/745d0a1adb78335d3c6fb63238141d9ff7d3a98c/pipelines/tagger_parser_ud/configs/default.cfg#L11-L32

[maxtrem]

Thanks @adrianeboyd !

I just tried that and it seems to work, I added to the config:

[components.morphologizer]
factory = "morphologizer"

[components.morphologizer.model]
@architectures = "spacy.Tagger.v1"
nO = null

[components.morphologizer.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"

[components.morphologizer.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

I also added the pooling layer that was applied to the tagger as well, does this setting make sense for the morphologizer?

[adrianeboyd]

It should look the same as the tagger. You should also be able to use -p tagger,morphologizer,parser with spacy init config to get a default config.

[maxtrem]

@adrianeboyd This is great! Thanks a lot!