accessing meta.json from a factory while packaging a pipeline
#10675
-
|
I would like to access I tried two ways of adding this component factory with either However, from within I have a post-processing pipe defined in a separate code file:
@Language.factory("pathology_cui")
class Pathology:
def __init__(self, nlp: Language, name: str = "pathology_cui",
#log_level: StrictStr
):
self.data = {}
self.margin = 7
import sys
print(f"NLP: {nlp}", file=sys.stderr)
print(f"nlp.meta: {nlp.meta}", file=sys.stderr)
self.performance = nlp.meta["performance"]["ents_per_type"]
...To add it to a pre-trained NER module, I read it, add a import spacy
from postprocess import Pathology
nlp = spacy.load(input_dir)
data_path = Path("data")
pathology_cui = nlp.add_pipe("pathology_cui", last=True)
pathology_cui.from_disk(data_path)
os.makedirs(f"{output_dir}/pathology_cui", exist_ok = True)
nlp.to_disk(output_dir)Manual assemblyHere the first step works, but packaging fails: import os
import re
import sys
from pathlib import Path
import spacy
from postprocess import Pathology
input_dir = ...
output_dir = ...
nlp = spacy.load(input_dir)
data_path = Path("data")
pathology_cui = nlp.add_pipe("pathology_cui", last=True)
pathology_cui.from_disk(data_path)
os.makedirs(f"{output_dir}/pathology_cui", exist_ok = True)
nlp.to_disk(output_dir)This step works. Here, I can verify that However, when I try to package it, I get an exception that the Declarative assemblyI saw there is assemble functionality, but I am not quite sure how to point the config to a pre-trained NER model. I've tried in CFG=config-pathology_cui.cfg
python -m spacy assemble \
$CFG \
$OUTPUT_DIRError |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 4 replies
-
|
I'm not entirely sure I understand what you're trying to do with |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your response. I would like to take per-class performance scores and append them to the respective entities. What I was counting for is that the |
Beta Was this translation helpful? Give feedback.
-
|
I think I'd need to see a minimal example to understand the details about what "appending to an entity" means, but it's possible if you store Here's more about the order things happen in: https://spacy.io/usage/training/#config-lifecycle |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your patience. I've created a gist with all code here, except the trained NER pipeline that I am trying to augment (I am working with my institutional privacy office to release that one as well).
Appending performance metrics to the entities as custom
I am currently taking Current version works when I run assemble-manual.py to assemble the pipeline in python. However, when I try to package it, or assemble it programmatically with |
Beta Was this translation helpful? Give feedback.
-
|
What I meant was to store a reference to the @Language.factory("meta_reader")
class MetaReader:
def __init__(self, nlp, name):
self.meta = nlp.meta
print("init", self.meta["labels"])
def __call__(self, doc):
print("call", self.meta["labels"])
return docWith |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. That worked |
Beta Was this translation helpful? Give feedback.
Thank you for your patience. I've created a gist with all code here, except the trained NER pipeline that I am trying to augment (I am working with my institutional privacy office to release that one as well).
Appending performance metrics to the entities as custom
Span._.metricfields happens in postprocess.py:L110I am currently taking
nlp.metain the__init__(which fails while using command line tools). Do you suggest taking thenlpobject f…