No Vectors When Using en_core_web_trf?

[VikR0001]

After running this:

nlp = spacy.load('en_core_web_lg')
has_vector = nlp('test text').has_vector

...has_vector == True

But after running this:

nlp = spacy.load('en_core_web_trf')
has_vector = nlp('test text').has_vector

...has_vector == False

What am I missing?

[felipefoschiera]

Nothing. The en_core_web_trf (transformer pipeline) does not include vectors.

[polm]

What @felipefoschiera wrote is correct, has_vector is for word vectors and the Transformers pipeline doesn't include them. The reasoning is that if you have Transformers you generally don't also need word vectors.

[VikR0001]

The sample code I have for spacy transformers is for spacy v2.3 and works with vectors. It fits a model like this:

from sklearn import svm

nlp = spacy.load("en_trf_bertbaseuncased_lg")
[.....]
docs = [nlp(text) for text in train_x]
train_x_vectors = [doc.vector for doc in docs]
clf_svm = svm.SVC(kernel='linear')

clf_svm.fit(train_x_vectors, train_y) 

I spent hours Friday searching for the equivalent code for spacy v3.1 and haven't yet found it. (Maybe it was in front of me and I didin't recognize it.) Could you please post sample code here, or post a link to sample code?

[felipefoschiera]

So it seems like you have a Text Classification task. My bet is the best option is to use the spaCy CLI. I am using that for a custom NER training I have for an individual project, and it is great to use, both with vectors and transformers. Take a look at this tutorial project from the spaCy repository. It categorizes text into emotion categories, I believe you can adapt to your use. It also appears to support spacy-transformers.

[VikR1000]

Thanks for this link. I'm finding other links now too, now that I'm not focusing on scikit-learn. I'll be looking at these as well:

• https://www.dataquest.io/blog/tutorial-text-classification-in-python-using-spacy/
• https://medium.com/analytics-vidhya/building-a-text-classifier-with-spacy-3-0-dd16e9979a
• https://www.kaggle.com/poonaml/text-classification-using-spacy

[polm]

Just a note, but while third-party articles can be great if they cover your specific use case, they also are often not updated or may have strange usage patterns. So do check the official docs first!

[VikR0001]

Thanks @polm. I've looked through the spacy docs a lot trying to find sample code (e.g. this page) -- perhaps I'm just not recognizing it when I see it somehow. Could you please post a link to sample code, from the official docs, for creating a text classification model -- if possible one that works with spacy.load('en_core_web_trf')?

[polm]

We don't have a single page that's "this is how you make a textcat model", which is something we could improve on, but you can use the quickstart, check other sections in the Training Models section, and refer to the tutorial projects. The Quickstart lets you pick GPU to use Transformers models.

[brabbit61]

I am using that for a custom NER training I have for an individual project, and it is great to use, both with vectors and transformers.

Hey @felipefoschiera I find myself working on a similar task and I'm using the spaCy CLI to train the model. I trained it using 2 different configs:

1. Fine tuning a roberta-base transformer.
2. A combination of en_core_web_md vectors + roberta-base transformer, resulting in a bump in the accuracy.

Is this what you've done as well? I cannot find an explanation for the bump in accuracy or what the model architecture even looks like with both vectors and a transformer. Do you have any experience with this?

Adding a snippet from the 2nd config file.

[paths]
train = "./train.spacy"
dev = "./val.spacy"
vectors = "en_core_web_md"
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner"]

[kadarakos]

May I ask you to share the full config?

[brabbit61]

[paths]
train = "./train.spacy"
dev = "./val.spacy"
vectors = "en_core_web_md"
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "roberta-base"
mixed_precision = false

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.grad_scaler_config]

[components.transformer.model.tokenizer_config]
use_fast = true

[components.transformer.model.transformer_config]

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 100
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 1024
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
ents_f = 0.0
ents_p = 0.3
ents_r = 0.7
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[kadarakos]

Thank you! The vectors have no effect when training with a transformer instead of a tok2vec. There are a few things you've seemed to have changed in the config overtime from what you would get with:

python -m spacy init config test2.cfg --lang en --pipeline ner --optimize accuracy --gpu

The eval_frequency is reduced to 100 from 200, the batch size is reduced to 1024 from 2000 and the score_weights are changed:

• ents_p from 0.0 to 0.3
• ents_r from 0.0 to 0.7
• ents_f from 1.0 to 0.0

Just guessing here, but you might notice a change in performance due to some of these edits. The scores are different because of the different weights. The lower batch-size might accommodate a smaller data set and more frequent checkpoiting with the smaller eval_frequency has a higher chance of picking a model that performs well on the dev set.

[brabbit61]

The hyperparameters above remained the same while training the NER model with transformer and tok2vec embeddings. So the better quality of embeddings using the transformer is enough to explain the jump in accuracy?

[kadarakos]

Ah okay! Yes, in general you can expect the transformer to give you a boost over tok2vec + pre-trained vectors at the cost of efficiency.