Pretraining requires it's own output to intitalize

[kchalkSGS]

process = subprocess.run('python -m spacy pretrain Configs/config_cnn.cfg Data/pretrain_cnn', shell=True)

initialize -> init_tok2vec   can't find pretrained tok2vec: Data/pretrain_cnn/model9.bin

{'corpora': {'dev': {'@readers': 'spacy.Corpus.v1', 'path': 'Data/ste-test.spacy', 'max_length': 0, 'gold_preproc': False, 'limit': 0, 'augmenter': None}, 'pretrain': {'@readers': 'spacy.Corpus.v1', 'path': 'Data/pretrain.spacy', 'gold_preproc': False, 'max_length': 500, 'limit': 0, 'augmenter': None}, 'train': {'@readers': 'spacy.Corpus.v1', 'path': 'Data/ste-train.spacy', 'max_length': 0, 'gold_preproc': False, 'limit': 0, 'augmenter': None}}, 'initialize': {'vectors': 'en_core_web_md', 'init_tok2vec': 'Data/pretrain_cnn/model9.bin', 'vocab_data': None, 'lookups': None, 'before_init': None, 'after_init': None, 'components': {}, 'tokenizer': {}}, 'nlp': {'lang': 'en', 'pipeline': ['tok2vec', 'textcat'], 'batch_size': 128, 'disabled': ['tok2vec', 'textcat'], 'before_creation': None, 'after_creation': None, 'after_pipeline_creation': None, 'tokenizer': {'@tokenizers': 'spacy.Tokenizer.v1'}}, 'paths': {'train': 'Data/ste-train.spacy', 'dev': 'Data/ste-test.spacy', 'vectors': 'en_core_web_md', 'init_tok2vec': 'Data/pretrain_cnn/model9.bin', 'verbose': True, 'raw_text': 'Data/pretrain.spacy'}, 'system': {'gpu_allocator': 'pytorch', 'g': 0, 'seed': 0}, 'training': {'accumulate_gradient': 3, 'dev_corpus': 'corpora.dev', 'train_corpus': 'corpora.train', 'seed': 0, 'gpu_allocator': 'pytorch', 'dropout': 0.1, 'patience': 1600, 'max_epochs': 5, 'max_steps': 20000, 'eval_frequency': 200, 'frozen_components': [], 'annotating_components': [], 'before_to_disk': None, 'batcher': {'@batchers': 'spacy.batch_by_padded.v1', 'discard_oversize': True, 'size': 2000, 'buffer': 256, 'get_length': None}, 'logger': {'@loggers': 'spacy.ConsoleLogger.v1', 'progress_bar': False}, 'optimizer': {'@optimizers': 'Adam.v1', 'beta1': 0.9, 'beta2': 0.999, 'L2_is_weight_decay': True, 'L2': 0.01, 'grad_clip': 1.0, 'use_averages': False, 'eps': 1e-08, 'learn_rate': {'@schedules': 'warmup_linear.v1', 'warmup_steps': 250, 'total_steps': 20000, 'initial_rate': 5e-05}}, 'score_weights': {'cats_score': 1.0, 'cats_score_desc': None, 'cats_micro_p': None, 'cats_micro_r': None, 'cats_micro_f': None, 'cats_macro_p': None, 'cats_macro_r': None, 'cats_macro_f': None, 'cats_macro_auc': None, 'cats_f_per_type': None, 'cats_macro_auc_per_type': None}}, 'components': {'tok2vec': {'factory': 'tok2vec', 'model': {'@architectures': 'spacy.Tok2Vec.v2', 'embed': {'@architectures': 'spacy.MultiHashEmbed.v2', 'width': 300, 'attrs': ['NORM', 'PREFIX', 'SUFFIX', 'SHAPE'], 'rows': [2000, 1000, 1000, 1000], 'include_static_vectors': True}, 'encode': {'@architectures': 'spacy.MaxoutWindowEncoder.v2', 'width': 300, 'window_size': 1, 'maxout_pieces': 3, 'depth': 4}}}, 'textcat': {'factory': 'textcat', 'model': {'@architectures': 'spacy.TextCatEnsemble.v2', 'nO': None, 'linear_model': {'@architectures': 'spacy.TextCatBOW.v2', 'exclusive_classes': True, 'ngram_size': 1, 'no_output_layer': False, 'nO': None}, 'tok2vec': {'@architectures': 'spacy.Tok2VecListener.v1', 'width': 300, 'upstream': '*'}}, 'threshold': 0.5}}, 'pretraining': {'max_epochs': 20, 'dropout': 0.2, 'n_save_every': None, 'component': 'tok2vec', 'layer': '', 'corpus': 'corpora.pretrain', 'batcher': {'@batchers': 'spacy.batch_by_words.v1', 'size': 3000, 'discard_oversize': False, 'tolerance': 0.2, 'get_length': None}, 'objective': {'@architectures': 'spacy.PretrainVectors.v1', 'maxout_pieces': 3, 'hidden_size': 300, 'loss': 'cosine'}, 'optimizer': {'@optimizers': 'Adam.v1', 'beta1': 0.9, 'beta2': 0.999, 'L2_is_weight_decay': True, 'L2': 0.01, 'grad_clip': 1.0, 'use_averages': True, 'eps': 1e-08, 'learn_rate': 0.001}}}

I can get around this error by setting init_tok2vec = null and the pre-training runs, but https://spacy.io/usage/embeddings-transformers#pretraining-training says this should be the location of the output of the pretraining, and it is supposed to be the case that the same config file can be used for pretraining and training, no? Unless I misunderstand, this line should be able to refer to a pretraining output folder (also specified in the config, rather than only at run time from CLI) and when running the pretraining command, this field should raise a warning if it does already exist rather than an error if ti doesn't. Am I missing something?

[polm]

it is supposed to be the case that the same config file can be used for pretraining and training, no?

No, the expectation is that you use a different file for pretraining than you do for training. It's true this might not be clear enough in the documentation.

Even in pretraining, you can initialize your tok2vec from another model, and that has a different meaning at pretraining time than it does at training time.

[polm]

Actually, nevermind, you should be able to use the same file, like it states in the docs. Let me go over my examples here again...

[polm]

OK,

1. You should definitely use the same file for training and pretraining
2. You can't fill in init_tok2vec before pretraining has run, so you should either fill that in after pretraining or just override on the command line. This is expected behavior.
3. You shouldn't need to specify the raw text file at training time (Make JsonlReader path optional #8396), though I had some trouble with this in the most recent version and need to look into it more.

Sorry for the confusion, I'll add a note to the docs to make this clearer. While the vision for pretraining has been to have just the one config, we've had issues around typing that we've gradually worked out; my memory of those led me to assume we wanted two files.

[kchalkSGS]

Thank you for your efforts @polm! I was out last week but this is helpful in making sense of things!

[polm]

Thank you - this issue brought the issue around this to our attention again and we're working on making it easier to use, see #9639 for details.