How to change scorer to Softmax in spancat? #11813

jamnicki · 2022-11-16T12:12:26Z

jamnicki
Nov 16, 2022

How to change scorer to Softmax in default spancat config?
In my case its Polish with only spancat component, CPU, efficiency.

...
[components.spancat.model.scorer]
- @layers = "spacy.LinearLogistic.v1"
- nO = null
- nI = null
+ @layers = "Softmax.v2"
...

Above change return following error message.

SystemError: <method '__getitem__' of 'dict' objects> returned a result with an error set

Originally posted by @jamnicki in #11144 (reply in thread)

Answered by ljvmiranda921

Nov 17, 2022

Hi @jamnicki ,

I think you need to do it this way:

...
[model]
@architectures = "spacy.SpanCategorizer.v1"
- scorer = {"@layers": "spacy.LinearLogistic.v1"}
+ scorer = {"@layers": "Softmax.v2"}
...

You can set it under the model group and put it inside the scorer parameter (notice how the @layers is enclosed like a dictionary). You might also need to set individual parameters for Softmax.v2:

[model.scorer.init_W]
@initializers = "zero_init.v1"

[model.scorer.init_b]
@initializers = "zero_init.v1"`

So overall, your config may look like this:

[model]
@architectures = "spacy.SpanCategorizer.v1"
scorer = {"@layers": "Softmax.v2"}

[model.scorer.init_W]
@initializers = "zero_init.v1"

[model.…

View full answer

ljvmiranda921 · 2022-11-17T03:18:46Z

ljvmiranda921
Nov 17, 2022

Hi @jamnicki ,

I think you need to do it this way:

...
[model]
@architectures = "spacy.SpanCategorizer.v1"
- scorer = {"@layers": "spacy.LinearLogistic.v1"}
+ scorer = {"@layers": "Softmax.v2"}
...

You can set it under the model group and put it inside the scorer parameter (notice how the @layers is enclosed like a dictionary). You might also need to set individual parameters for Softmax.v2:

[model.scorer.init_W]
@initializers = "zero_init.v1"

[model.scorer.init_b]
@initializers = "zero_init.v1"`

So overall, your config may look like this:

[model]
@architectures = "spacy.SpanCategorizer.v1"
scorer = {"@layers": "Softmax.v2"}

[model.scorer.init_W]
@initializers = "zero_init.v1"

[model.scorer.init_b]
@initializers = "zero_init.v1"`

3 replies

jamnicki Nov 17, 2022
Author

Hi, @ljvmiranda921

Thats worked out! Initializers was not nessecary. Thank you 😃

egumasa Nov 20, 2022

Thank you for this useful discussion. I have a problem here in swapping to Softmax.v2.

I got the following message.

Traceback (most recent call last):
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/cli/train.py", line 72, in train
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/training/initialize.py", line 43, in init_nlp
    config = nlp.config.interpolate()
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/language.py", line 276, in config
    if not srsly.is_json_serializable(self._config):
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/srsly/_json_api.py", line 152, in is_json_serializable
    ujson.dumps(obj)
SystemError: <method '__getitem__' of 'dict' objects> returned a result with an error set

This is the relevant part of my .cfg file, does this seem fine?

[components.spancat]
factory = "spancat"
max_positive = 2
spans_key = ${vars.spans_key}
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"
scorer = {"@layers": "Softmax.v2"}

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 256

I added scorer.init,

[components.spancat]
factory = "spancat"
max_positive = 2
spans_key = ${vars.spans_key}
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"
scorer = {"@layers": "Softmax.v2"}

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 256

# [components.spancat.model.scorer]
# @layers = "Softmax.v2"
# # nO = null
# # nI = null

[components.spancat.model.scorer.init_W]
@initializers = "zero_init.v1"

[components.spancat.model.scorer.init_b]
@initializers = "zero_init.v1"`

Then it is producing another error...

Traceback (most recent call last):
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/cli/train.py", line 72, in train
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/training/initialize.py", line 41, in init_nlp
    nlp = load_model_from_config(raw_config, auto_fill=True)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/util.py", line 528, in load_model_from_config
    nlp = lang_cls.from_config(
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/language.py", line 1783, in from_config
    nlp.add_pipe(
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/language.py", line 792, in add_pipe
    pipe_component = self.create_pipe(
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/language.py", line 674, in create_pipe
    resolved = registry.resolve(cfg, validate=validate)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/thinc/config.py", line 746, in resolve
    resolved, _ = cls._make(
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/thinc/config.py", line 795, in _make
    filled, _, resolved = cls._fill(
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/thinc/config.py", line 850, in _fill
    filled[key], validation[v_key], final[key] = cls._fill(
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/thinc/config.py", line 850, in _fill
    filled[key], validation[v_key], final[key] = cls._fill(
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/thinc/config.py", line 850, in _fill
    filled[key], validation[v_key], final[key] = cls._fill(
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/thinc/config.py", line 849, in _fill
    promise_schema = cls.make_promise_schema(value, resolve=resolve)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/thinc/config.py", line 1040, in make_promise_schema
    func = cls.get(reg_name, func_name)
  File "/Users/masakieguchi/opt/miniforge3/envs/spacy-exp/lib/python3.9/site-packages/spacy/util.py", line 141, in get
    raise RegistryError(
catalogue.RegistryError: [E893] Could not find function '"zero_init.v1"`' in function registry 'initializers'. If you're using a custom function, make sure the code is available. If the function is provided by a third-party package, e.g. spacy-transformers, make sure the package is installed in your environment.

Available names: glorot_normal_init.v1, glorot_uniform_init.v1, he_normal_init.v1, he_uniform_init.v1, lecun_normal_init.v1, lecun_uniform_init.v1, normal_init.v1, uniform_init.v1, zero_init.v1

egumasa Nov 20, 2022

Sorry, there was apparently a typo in the config file. The following seems to be working now.

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"
scorer = {"@layers": "Softmax.v2"}

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 256

# [components.spancat.model.scorer]
# @layers = "Softmax.v2"
# nO = 2
# nI = null

[components.spancat.model.scorer.init_W]
@initializers = "zero_init.v1"

[components.spancat.model.scorer.init_b]
@initializers = "zero_init.v1"

Thank you!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

How to change scorer to Softmax in spancat? #11813

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

How to change scorer to Softmax in spancat? #11813

Uh oh!

Uh oh!

jamnicki Nov 16, 2022

Replies: 1 comment · 3 replies

Uh oh!

ljvmiranda921 Nov 17, 2022

Uh oh!

jamnicki Nov 17, 2022 Author

Uh oh!

egumasa Nov 20, 2022

Uh oh!

egumasa Nov 20, 2022

jamnicki
Nov 16, 2022

Replies: 1 comment 3 replies

ljvmiranda921
Nov 17, 2022

jamnicki Nov 17, 2022
Author