Skip to content

Using bert-base-uncased instead of default roberta-base inside an en_core_web_trf based pipeline #10281

@nramrakhiyani

Description

@nramrakhiyani

Using bert-base-uncased instead of default roberta-base inside an en_core_web_trf based pipeline

On loading the en_core_web_trf model using spacy.load, the nlp.pipeline has the default transformer component (based on roberta-base) loaded before the tagger. I want to replace it with the bert-base-uncased model. I tried the following steps, which seem to give no error. But when parsing a sentence using nlp, it seems to raise a "NoneType object isn't callable" error.

  1. Removing the transformer component
    nlp.remove_pipe("transformer")
    nlp.pipeline (shows the transformer component removed)

  2. Creating a config (based on the example on https://spacy.io/api/transformer)
    config = { "model": { "@architectures": "spacy-transformers.TransformerModel.v3", "name": "D:\\Work\\Tools\\bert-base-uncased", "tokenizer_config": {"use_fast": True}, "transformer_config": {"output_attentions": True}, "mixed_precision": True, "grad_scaler_config": {"init_scale": 32768} } }

  3. Adding the desired transformer model (using the config) in the pipeline
    trf = nlp.add_pipe("transformer", config=config, before="tagger")

  4. Checking nlp.pipeline (Shows a new transformer as part of the pipeline)

  5. Checking on a sentence leads to a NoneType object isn't callable error
    s = 'John went to London.'
    doc = nlp(s)

Traceback (most recent call last):
File "", line 1, in
File "D:\Work\Tools\envs\env_main2\lib\site-packages\spacy\language.py", line 1018, in call
error_handler(name, proc, [doc], e)
File "D:\Work\Tools\envs\env_main2\lib\site-packages\spacy\util.py", line 1618, in raise_error
raise e
File "D:\Work\Tools\envs\env_main2\lib\site-packages\spacy\language.py", line 1013, in call
doc = proc(doc, **component_cfg.get(name, {})) # type: ignore[call-arg]
File "D:\Work\Tools\envs\env_main2\lib\site-packages\spacy_transformers\pipeline_component.py", line 192, in call
outputs = self.predict([doc])
File "D:\Work\Tools\envs\env_main2\lib\site-packages\spacy_transformers\pipeline_component.py", line 228, in predict
activations = self.model.predict(docs)
File "D:\Work\Tools\envs\env_main2\lib\site-packages\thinc\model.py", line 315, in predict
return self._func(self, X, is_train=False)[0]
File "D:\Work\Tools\envs\env_main2\lib\site-packages\spacy_transformers\layers\transformer_model.py", line 177, in forward
batch_encoding = huggingface_tokenize(tokenizer, [span.text for span in flat_spans])
File "D:\Work\Tools\envs\env_main2\lib\site-packages\spacy_transformers\layers\transformer_model.py", line 278, in huggingface_tokenize
padding="longest",
TypeError: 'NoneType' object is not callable

Assuming the bert-base-uncased model is available and downloaded in a suitable local directory, is this a right way to use a transformer model different from the default roberta-base? If not, is there a better way to achieve the desired setting? And if this is a possible way, can some light be thrown on why the NoneType error?

Your Environment

  • spaCy version: 3.2.0
  • Platform: Windows-10-10.0.14393-SP0
  • Python version: 3.7.11
  • Pipelines: en_core_web_md (3.2.0), en_core_web_trf (3.2.0)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions