Where are the attribute_ruler patterns defined for each language?

[bryant1410]

The different language models have an attribute_ruler, which is populated with a binary patterns file (serialized patterns). This allows for example to populate pos based on tag. I was wondering where is the source file for this patterns file.

[adrianeboyd]

Right now the pretrained pipelines are the main reference. You can see the patterns using:

nlp.get_pipe("attribute_ruler").patterns

The patterns file is just this same data in msgpack format, which you can read directly with srsly.read_msgpack (or other msgpack libraries) if you'd prefer.

You can also always source an existing attribute ruler into a new pipeline and all the patterns will be copied with it.

We've considered having a separate repo where some of the pipeline-specific settings like this are stored, so it would be easy to refer to them in new configs, but this doesn't exist yet.

[bryant1410]

Thanks for the answer! Yeah, it'd be cool if the pipeline-specific settings are open source, in case people want to contribute to them.

This came just out of curiosity, as I was traversing the code, trying to understand different parts.

Related, are the commands used to create the different language models publicly available somewhere? I mean the training part, when they take parts from other pipelines, packaging, etc.