Passing kwargs to a Language.factory

[BramVanroy]

I am writing a custom factory where users can specify a number of unconstrained keyword arguments in the config. However, I find that spaCy does not like such optional kwargs.

import spacy
from spacy import Language


@Language.factory(
    "custom_fctry"
)
class CustomFactory:
    def __init__(self, nlp: Language, name: str, **kwargs):
        self.nlp = nlp
        self.name = name
        self.rev_kwargs = {v: k for k, v in kwargs.items()}


nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("custom_fctry", last=True)

Error trace:

---------------------------------------------------------------------------
ConfigValidationError                     Traceback (most recent call last)
Input In [2], in <cell line: 3>()
      1 import spacy
      2 nlp = spacy.load("en_core_web_sm")
----> 3 nlp.add_pipe("custom_fctry", last=True)

File c:\users\bramv\appdata\local\programs\python\python39\lib\site-packages\spacy\language.py:792, in Language.add_pipe(self, factory_name, name, before, after, first, last, source, config, raw_config, validate)
    784     if not self.has_factory(factory_name):
    785         err = Errors.E002.format(
    786             name=factory_name,
    787             opts=", ".join(self.factory_names),
   (...)
    790             lang_code=self.lang,
    791         )
--> 792     pipe_component = self.create_pipe(
    793         factory_name,
    794         name=name,
    795         config=config,
    796         raw_config=raw_config,
    797         validate=validate,
    798     )
    799 pipe_index = self._get_pipe_index(before, after, first, last)
    800 self._pipe_meta[name] = self.get_factory_meta(factory_name)

File c:\users\bramv\appdata\local\programs\python\python39\lib\site-packages\spacy\language.py:674, in Language.create_pipe(self, factory_name, name, config, raw_config, validate)
    671 cfg = {factory_name: config}
    672 # We're calling the internal _fill here to avoid constructing the
    673 # registered functions twice
--> 674 resolved = registry.resolve(cfg, validate=validate)
    675 filled = registry.fill({"cfg": cfg[factory_name]}, validate=validate)["cfg"]
    676 filled = Config(filled)

File c:\users\bramv\appdata\local\programs\python\python39\lib\site-packages\thinc\config.py:746, in registry.resolve(cls, config, schema, overrides, validate)
    737 @classmethod
    738 def resolve(
    739     cls,
   (...)
    744     validate: bool = True,
    745 ) -> Dict[str, Any]:
--> 746     resolved, _ = cls._make(
    747         config, schema=schema, overrides=overrides, validate=validate, resolve=True
    748     )
    749     return resolved

File c:\users\bramv\appdata\local\programs\python\python39\lib\site-packages\thinc\config.py:795, in registry._make(cls, config, schema, overrides, resolve, validate)
    793 if not is_interpolated:
    794     config = Config(orig_config).interpolate()
--> 795 filled, _, resolved = cls._fill(
    796     config, schema, validate=validate, overrides=overrides, resolve=resolve
    797 )
    798 filled = Config(filled, section_order=section_order)
    799 # Check that overrides didn't include invalid properties not in config

File c:\users\bramv\appdata\local\programs\python\python39\lib\site-packages\thinc\config.py:850, in registry._fill(cls, config, schema, validate, resolve, parent, overrides)
    848     schema.__fields__[key] = copy_model_field(field, Any)
    849 promise_schema = cls.make_promise_schema(value, resolve=resolve)
--> 850 filled[key], validation[v_key], final[key] = cls._fill(
    851     value,
    852     promise_schema,
    853     validate=validate,
    854     resolve=resolve,
    855     parent=key_parent,
    856     overrides=overrides,
    857 )
    858 reg_name, func_name = cls.get_constructor(final[key])
    859 args, kwargs = cls.parse_args(final[key])

File c:\users\bramv\appdata\local\programs\python\python39\lib\site-packages\thinc\config.py:916, in registry._fill(cls, config, schema, validate, resolve, parent, overrides)
    914         result = schema.parse_obj(validation)
    915     except ValidationError as e:
--> 916         raise ConfigValidationError(
    917             config=config, errors=e.errors(), parent=parent
    918         ) from None
    919 else:
    920     # Same as parse_obj, but without validation
    921     result = schema.construct(**validation)

ConfigValidationError: 

Config validation error

custom_fctry -> kwargs   field required

{'nlp': <spacy.lang.en.English object at 0x00000243803EBFA0>, 'name': 'custom_fctry', '@factories': 'custom_fctry'}

Of course, this is a dummy example. But is there any way to allow users to pass in custom kwargs that I in turn can pass through to other functions?

[koaning]

So, theoretically, one could just use a dictionary here.

import spacy
from spacy import Language


@Language.factory(
    "custom_fctry"
)
class CustomFactory:
    def __init__(self, nlp: Language, name: str, kwargs: dict):
        self.nlp = nlp
        self.name = name
        self.rev_kwargs = {v: k for k, v in kwargs.items()}
        # note the print statement here
        print(self.rev_kwargs)

nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("custom_fctry", last=True, config={"kwargs": {"hello": "world"}})
# {'world': 'hello'}

But I'm a little more curious about the use case. Why would you want to pass down keyword arguments? You're also able to refer to functions from the spaCy config and that might be a more preferable abstraction in the long run.

[BramVanroy]

Hey @koaning. My main goal is having this to be as user-friendly as possible. Having to specify an extra "dict" inside the config is not ideal - but if that is the way that it is intended, I can use that.

The use case is that I initialize another object with the remaining given kwargs. These kwargs are not necessarily known beforehand, so I cannot specify the arguments manually.

 def __init__(self, nlp: Language, name: str, kwargs: dict):
        self.nlp = nlp
        self.name = name
        self.other_obj= init_other_obj(**kwargs)

I will use your suggestion, thanks.

[polm]

For spaCy components, we typically put the factory decorator on a function instead of directly on the class. If you follow the same pattern, you can use a kwargs glob in the function and pass it to the class as a dict, which should allow your users flexibility while keeping the type checking machinery happy too.

[BramVanroy]

Thanks, that seems like a good middle ground.