Problem regarding Custom Trained model only having tok2vec and ner component and not generating vector for tokens

[dhruvbhimani-2512]

I created one custom NER model in spaCy. While doing so, I downloaded config file through the widget provided here. Since I only needed to update NER, I selected that only. Then I trained the model and got the final one. But it only has two components tok2vec and ner. It identifies entities correctly but when I crate a Doc object for any sentence, it doesn't have embedding vector associated with it. Can anyone guide me here, please? Thanks.

[cliuxinxin]

我也是自定义训练，不过我tok2vec和ner都训练了的。

[dhruvbhimani-2512]

我也是自定义训练，不过我tok2vec和ner都训练了的。

Are you getting doc.vector?

[cliuxinxin]

我不知道你说的什么意思，我使用的是 ner_demo spacy project 的模板，在 config.cfg中是这么设置的

[nlp]
pipeline = ["tok2vec","ner"]

这样在训练的时候，两个模型都会被训练，我想如果你去掉了的话，其实是错的，应该是冻结前面的部分

[nlp]
pipeline = ["tok2vec","ner"]
disabled = ["tok2vec"]

[dhruvbhimani-2512]

@cliuxinxin Are you suggesting that I need to disable my tok2vec pipeline to get doc.vector?

[cliuxinxin]

是的，我建议你尝试一下我的这种写法。不要删除。

[polm]

Did you set vectors in the [initialize] block in your config?

Even if you don't do that, doc.vector should still have a value generated by your tok2vec - see here for a description of how that works. Can you clarify what you mean by "doesn't have a vector"?

[dhruvbhimani-2512]

Hey @polm, the issue is resolved.

Basically, in the training config file, it was taking factory tok2vec component (from scratch) and because of that, I was not getting embedding vector for tokens.
[components.tok2vec] factory = "tok2vec"

I then replaced it with en_web_core_md tok2vec component and it's working fine now:
[components.tok2vec] source = "en_web_core_md"

[polm]

Glad you figured it out!

A from-scratch tok2vec should still give vectors, though they probably won't be useful without training.