How to set up tok2vec for annotating components for training

[grivaz]

I am training my own dependency parser, and I would like the output of the tagger to be used as a feature for my parser. I'm using a project.yml with a default.cfg training config. If I'm understanding the documentation correctly, I need to add the tagger output as an attrs of tok2vec. However when I change the line attrs = ["LOWER","PREFIX","SUFFIX","SHAPE"] to attrs = ["LOWER","PREFIX","SUFFIX","SHAPE","TAG"], I get the following error:

=========================== Initializing pipeline ===========================
 Traceback (most recent call last):
   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
     return _run_code(code, main_globals, None,
   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
     exec(code, run_globals)
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
     setup_cli()
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy/cli/_util.py", line 71, in setup_cli
     command(prog_name=COMMAND)
   File "/home/cecile/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
     return self.main(*args, **kwargs)
   File "/home/cecile/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
     rv = self.invoke(ctx)
   File "/home/cecile/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
     return _process_result(sub_ctx.command.invoke(sub_ctx))
   File "/home/cecile/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
     return ctx.invoke(self.callback, **ctx.params)
   File "/home/cecile/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
     return callback(*args, **kwargs)
   File "/home/cecile/.local/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
     return callback(**use_params)  # type: ignore
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy/cli/train.py", line 45, in train_cli
     train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy/cli/train.py", line 72, in train
     nlp = init_nlp(config, use_gpu=use_gpu)
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy/training/initialize.py", line 41, in init_nlp
     nlp = load_model_from_config(raw_config, auto_fill=True)
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy/util.py", line 525, in load_model_from_config
     nlp = lang_cls.from_config(
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy/language.py", line 1779, in from_config
     nlp.add_pipe(
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy/language.py", line 789, in add_pipe
     pipe_component = self.create_pipe(
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy/language.py", line 671, in create_pipe
     resolved = registry.resolve(cfg, validate=validate)
   File "/home/cecile/.local/lib/python3.8/site-packages/thinc/config.py", line 729, in resolve
     resolved, _ = cls._make(
   File "/home/cecile/.local/lib/python3.8/site-packages/thinc/config.py", line 778, in _make
     filled, _, resolved = cls._fill(
   File "/home/cecile/.local/lib/python3.8/site-packages/thinc/config.py", line 833, in _fill
     filled[key], validation[v_key], final[key] = cls._fill(
   File "/home/cecile/.local/lib/python3.8/site-packages/thinc/config.py", line 833, in _fill
     filled[key], validation[v_key], final[key] = cls._fill(
   File "/home/cecile/.local/lib/python3.8/site-packages/thinc/config.py", line 850, in _fill
     getter_result = getter(*args, **kwargs)
   File "/home/cecile/.local/lib/python3.8/site-packages/spacy_legacy/architectures/tok2vec.py", line 201, in MultiHashEmbed_v1
     raise ValueError(f"Mismatched lengths: {len(rows)} vs {len(attrs)}")
 ValueError: Mismatched lengths: 4 vs 5

Here is the whole config.cfg. What am I doing wrong, please?

[paths]
;paths can be supplied by the cli (in project.yml)
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","tagger", "srl_parser"]
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 1000

[components]

[components.srl_parser]
factory = "parser"
learn_tokens = false
min_action_freq = 30
moves = null
update_with_oracle_cut_size = 100

[components.srl_parser.model]
@architectures = "spacy.TransitionBasedParser.v1"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = true
nO = null

[components.srl_parser.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tagger]
factory = "tagger"

[components.tagger.model]
@architectures = "spacy.Tagger.v1"
nO = null

[components.tagger.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v1"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v1"
width = ${components.tok2vec.model.encode.width}
attrs = ["LOWER","PREFIX","SUFFIX","SHAPE","TAG"]
rows = [5000,2500,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v1"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 2000
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
annotating_components = ["tagger"]
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
morph_per_feat = null
dep_las_per_type = null
sents_p = null
sents_r = null
tag_acc = 0.33
pos_acc = 0.17
morph_acc = 0.17
dep_uas = 0.17
dep_las = 0.17
sents_f = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[polm]

Small markdown note: when including terminal output / config in a post, you should use the "code block" syntax (markdown fences) rather than the "blockquote" syntax.

[grivaz]

I edited it. Thank you!

[adrianeboyd]

Hi, the settings attrs and rows need to be kept in sync so that they have the same length: rows specifies how many rows to use in the embeddings table for each attribute. See: https://spacy.io/api/architectures#MultiHashEmbed

However, to take a step back, it is probably not going to be helpful to include TAG as a parser feature. See a few related threads:

• Incorporating PoS information explicitly for the `edit_tree_lemmatizer` #10108 (reply in thread)
• PoS Tags as additional features for training NER #9641 (comment)

[grivaz]

Thank you so much!