NER for alphanumeric tokens

[marzooq-unbxd]

My text mainly contains alphanumeric tokens, I want to detect units within sentences,using spacy NER.
Examples are :

"300cm plank"
"4-5mm plank"
'2"x2"',
'over 5cm'
"2,3gb"
I had some very specific doubts related to embedding, encoding .

1.Consider an example how does '5centimeter'using MultiHashEmbed get a vector ,Suppose its tokenised as ['5centimeter'] given its not present in vocabulary?For the representation of ORTH , wouldnt both 'sm' and 'No model' initialise vector to[0,0] and proceed while training (assuming not present in vocab )?I also was doubting whether I should be using any en_core_web_sm, since my objective of measuremnts with numbers is not at all related to the original en_core_web_sm objective.

I have masked numericals with a particular token , like
"#cm plank","#-#cm plank",'#"x#"',"#,#gb",and seems to be working better than no masking.
I have added '#' as a prefix and suffix, so that i can tokenize '5cm' as [PREF'#','cm'], which might help the model help learn (NUM,UNIT).Should I be worried about using a special character as mask?Will Tok2vec learn a vector for '#'?After loading best-model, which would have learnt vectors,and using " ".join([token.text for token in tokens if token.has_vector]), I realised I am not able to create a vector for '5cm'.

2.Text 'A43RS90'(which I want to avoid is detecting as a measurement) which would get tokenised as [('TEXT','A#RS'),(SUFFIX,'#')]. I expect word vector models to not output any vector for 'A#RS'.
I am confused between using spacy.CharacterEmbed.v2 or spacy.MultiHashEmbed.v2 .With spacy.MultiHashEmbed.v2 ,I can only embed 'ORTH','PREFIX'etc,but this does not help the model learn that there are numericals surrounding the supposed 'unit' here(RS)?With CharacterEmbed, I will not be able to use info that '#' is a prefix in cases like '5cm'.Is there any charembed that does something that calcualtes ngram based like fasttext?spacy.CharacterEmbed.v2 takes from beginning and end instead of taking overlapping windows ,correct?
Also, since I am adding '-' as an infix, For '2-3cm', tokenised as (2,(infix-),3cm) would only vectors be learnt for "2" and "3cm" ?I only saw attr prefix , suffix,norm, orth etc

3.For MultiHashEmbed,I saw an example in https://explosion.ai/blog/bloom-embeddings, showing 'Apple' getting its embedding from {prefix:'A' , suffix:'ple'}
What happens when prefix already exists , and tokenisation is [(PREFIX'#'),('TEXT','cm')], Does the PREF_vector use
a)"vector for token.prefix_ of 'cm'"
or
b)"vector of '#'"

[polm]

Just before answering your specific questions - while it's good to understand what the code is doing, and general principles of what works and doesn't, NLP is often empirical. For any specific project, one of the most important things is to get an end-to-end pipeline working as a baseline and to work on improving performance iteratively. This helps you make sure that your task is feasible, and lets you avoid overthinking details that may end up not having significant impact on your task. In particular, with CPU models it's very fast to train them and try out different configurations.

1.Consider an example how does '5centimeter'using MultiHashEmbed get a vector ,Suppose its tokenised as ['5centimeter'] given its not present in vocabulary? For the representation of ORTH , wouldnt both 'sm' and 'No model' initialise vector to[0,0] and proceed while training (assuming not present in vocab )?

MultiHashEmbed, as the name suggests, uses multiple sources of features, not just word embeddings. So for example you would get features from word shape, prefix, and suffix, depending on your configuration, even without a word vector for the given word.

I also was doubting whether I should be using any en_core_web_sm, since my objective of measuremnts with numbers is not at all related to the original en_core_web_sm objective.

It's true the pretrained pipelines are unlikely to help you in this task, since they're trained on very different kinds of text.

I have masked numericals with a particular token , like
"#cm plank","#-#cm plank",'#"x#"',"#,#gb",and seems to be working better than no masking.
I have added '#' as a prefix and suffix, so that i can tokenize '5cm' as [PREF'#','cm'], which might help the model help learn (NUM,UNIT).Should I be worried about using a special character as mask?Will Tok2vec learn a vector for '#'?

There's no concept of "special characters" in spaCy input, so this can work. Regarding masking in general, ideally the word shape feature would take care of this for you, but for numbers like this it's not unheard of for it to help.

Do note that your # will be confused with pre-existing hashes. That may not be a problem, depending on your data.

2.Text 'A43RS90'(which I want to avoid is detecting as a measurement) which would get tokenised as [('TEXT','A#RS'),(SUFFIX,'#')]. I expect word vector models to not output any vector for 'A#RS'.

MultiHashEmbed typically, and by default, does not just use word vectors, but also uses token attributes, as mentioned above.

Regarding your questions about fasttext and windows. Context surrounding a token is captured in the CNN layer of tok2vec (or similarly in Transformers), you don't have to do anything extra for it. For subtoken information, you should look at floret.

What happens when prefix already exists , and tokenisation is [(PREFIX'#'),('TEXT','cm')], Does the PREF_vector use

Tokeniser prefixes and the prefix used by MultiHashEmbed are different things and don't interact, see here for the source that the MultiHashEmbed is using.

[marzooq-unbxd]

Thank you very much.Now its working,I have created a custom registry.
For anyonee stuck,
after added custom registry in functions.py in the same directory as config.cfg and using
!python -m spacy debug data config_charembed_custom_tok.cfg --paths.train ./training_data.spacy --paths.dev ./val_data.spacy --code ./functions.py -V

Here is the custom registry,

          from spacy.util import registry
          from spacy.lang import char_classes
          from spacy.symbols import ORTH
          from spacy.tokenizer import Tokenizer
          from spacy.util import compile_prefix_regex, compile_infix_regex, compile_suffix_regex
          from spacy.language import Language
          
          
          @registry.callbacks("customize_tokenizer")
          def make_customize_tokenizer():
              def customize_tokenizer(nlp):    
          
                  infixes =   ([r','])+([r'"'])+[r'-'] + list(nlp.Defaults.infixes)
                  infix_re = spacy.util.compile_infix_regex(infixes)
          
                  prefixes =([r","]) + [r'-'] +list(nlp.Defaults.prefixes)
                  prefixes.remove('#')
                  prefix_re = compile_prefix_regex(prefixes)
          
                  suffixes = ([r"\+"])+ list(nlp.Defaults.suffixes) 
                  suffixes.remove(r'(?<=[0-9])(?:km|km²|km³|m|m²|m³|dm|dm²|dm³|cm|cm²|cm³|mm|mm²|mm³|ha|µm|nm|yd|in|ft|kg|g|mg|µg|t|lb|oz|m/s|km/h|kmh|mph|hPa|Pa|mbar|mb|MB|kb|KB|gb|GB|tb|TB|T|G|M|K|%|км|км²|км³|м|м²|м³|дм|дм²|дм³|см|см²|см³|мм|мм²|мм³|нм|кг|г|мг|м/с|км/ч|кПа|Па|мбар|Кб|КБ|кб|Мб|МБ|мб|Гб|ГБ|гб|Тб|ТБ|тбكم|كم²|كم³|م|م²|م³|سم|سم²|سم³|مم|مم²|مم³|كم|غرام|جرام|جم|كغ|ملغ|كوب|اكواب)')
                  suffixes.remove('#')
                  suffix_re = compile_suffix_regex(suffixes)
          
                  nlp.tokenizer = Tokenizer(nlp.vocab, prefix_search=prefix_re.search,
                                              suffix_search=suffix_re.search,
                                              infix_finditer=infix_re.finditer,
                                              token_match=nlp.tokenizer.token_match,
                                              rules=nlp.Defaults.tokenizer_exceptions)
              
          
              return customize_tokenizer

I have added a custom tokenizer after [initialize] block,

```

              [paths]
              train = null
              dev = null
              vectors = null
              init_tok2vec = null
              
              [system]
              gpu_allocator = null
              seed = 0
              
              [nlp]
              lang = "en"
              pipeline = ["tok2vec","ner"]
              batch_size = 1000
              disabled = []
              before_creation = null
              after_creation = null
              after_pipeline_creation = null
              tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
              
              [components]
              
              [components.ner]
              factory = "ner"
              incorrect_spans_key = null
              moves = null
              scorer = {"@scorers":"spacy.ner_scorer.v1"}
              update_with_oracle_cut_size = 100
              
              [components.ner.model]
              @architectures = "spacy.TransitionBasedParser.v2"
              state_type = "ner"
              extra_state_tokens = false
              hidden_width = 64
              maxout_pieces = 2
              use_upper = true
              nO = null
              
              [components.ner.model.tok2vec]
              @architectures = "spacy.Tok2VecListener.v1"
              width = ${components.tok2vec.model.encode.width}
              upstream = "*"
              
              [components.tok2vec]
              factory = "tok2vec"
              
              [components.tok2vec.model]
              @architectures = "spacy.Tok2Vec.v2"
              
              [components.tok2vec.model.embed]
              @architectures = "spacy.CharacterEmbed.v2"
              width = 256
              rows = 7000
              nM = 64
              nC = 8
              include_static_vectors = false
              [components.tok2vec.model.encode]
              @architectures = "spacy.MaxoutWindowEncoder.v2"
              width = 256
              depth = 8
              window_size = 1
              maxout_pieces = 3
              
              [corpora]
              
              [corpora.dev]
              @readers = "spacy.Corpus.v1"
              path = ${paths.dev}
              max_length = 0
              gold_preproc = false
              limit = 0
              augmenter = null
              
              [corpora.train]
              @readers = "spacy.Corpus.v1"
              path = ${paths.train}
              max_length = 0
              gold_preproc = false
              limit = 0
              augmenter = null
              
              [training]
              dev_corpus = "corpora.dev"
              train_corpus = "corpora.train"
              seed = ${system.seed}
              gpu_allocator = ${system.gpu_allocator}
              dropout = 0.1
              accumulate_gradient = 1
              patience = 4000
              max_epochs = 9
              max_steps = 20000
              eval_frequency = 200
              frozen_components = []
              annotating_components = []
              before_to_disk = null
              
              [training.batcher]
              @batchers = "spacy.batch_by_words.v1"
              discard_oversize = false
              tolerance = 0.2
              get_length = null
              
              [training.batcher.size]
              @schedules = "compounding.v1"
              start = 100
              stop = 1000
              compound = 1.001
              t = 0.0
              
              [training.logger]
              @loggers = "spacy.ConsoleLogger.v1"
              progress_bar = false
              
              [training.optimizer]
              @optimizers = "Adam.v1"
              beta1 = 0.9
              beta2 = 0.999
              L2_is_weight_decay = true
              L2 = 0.01
              grad_clip = 1.0
              use_averages = false
              eps = 0.00000001
              learn_rate = 0.001
              
              [training.score_weights]
              ents_f = 1.0
              ents_p = 0.0
              ents_r = 0.0
              ents_per_type = null
              
              [pretraining]
              
              [initialize]
              vectors = ${paths.vectors}
              init_tok2vec = ${paths.init_tok2vec}
              vocab_data = null
              lookups = null
              before_init = null
              after_init = null
              
              [initialize.before_init]
              @callbacks = "customize_tokenizer"
              
              [initialize.components]
              
              [initialize.tokenizer]
              
              ```

But do be clear, do you have two NER models in your pipeline?

I only have one NER model, but since NER follows tokenization.Tokenization has to be done properly, and I have to merge, if I am using matcher with QUANTITY

it sounds like your tokenizer should prefer that tokenizaion (so "5", "cm" instead of "5cm")

The reason I cant tokenize this way is I will have to add infixes for '5cmx5cm' , but then examples like "A45CM4" will get detected as measuremnts.

[marzooq-unbxd]

Sorry , I am extending this post, But thought I would include this here itself. I was trying the same with SpanCategorizer instead of NER.
I think I just have to add this in DocBin creation
doc.spans["sc"] = ents where I have added doc.ents = ents

config.cfg for spancat

```

      [paths]
      train = null
      dev = null
      vectors = null
      init_tok2vec = null
      
      [system]
      gpu_allocator = null
      seed = 0
      
      [nlp]
      lang = "en"
      pipeline = ["tok2vec","spancat"]
      batch_size = 1000
      disabled = []
      before_creation = null
      after_creation = null
      after_pipeline_creation = null
      tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
      
      [components]
      
      [components.spancat]
      factory = "spancat"
      max_positive = null
      scorer = {"@scorers":"spacy.spancat_scorer.v1"}
      spans_key = "sc"
      threshold = 0.5
      
      [components.spancat.model]
      @architectures = "spacy.SpanCategorizer.v1"
      
      [components.spancat.model.reducer]
      @layers = "spacy.mean_max_reducer.v1"
      hidden_size = 128
      
      [components.spancat.model.scorer]
      @layers = "spacy.LinearLogistic.v1"
      nO = null
      nI = null
      
      [components.spancat.model.tok2vec]
      @architectures = "spacy.Tok2VecListener.v1"
      width = ${components.tok2vec.model.encode.width}
      upstream = "*"
      
      [components.spancat.suggester]
      @misc = "spacy.ngram_suggester.v1"
      sizes = [1,2,3]
      
      [components.tok2vec]
      factory = "tok2vec"
      
      [components.tok2vec.model]
      @architectures = "spacy.Tok2Vec.v2"
      
      [components.tok2vec.model.embed]
      @architectures = "spacy.MultiHashEmbed.v2"
      width = ${components.tok2vec.model.encode.width}
      attrs = ["ORTH","SHAPE"]
      rows = [5000,2500]
      include_static_vectors = true
      
      [components.tok2vec.model.encode]
      @architectures = "spacy.MaxoutWindowEncoder.v2"
      width = 256
      depth = 8
      window_size = 1
      maxout_pieces = 3
      
      [corpora]
      
      [corpora.dev]
      @readers = "spacy.Corpus.v1"
      path = ${paths.dev}
      max_length = 0
      gold_preproc = false
      limit = 0
      augmenter = null
      
      [corpora.train]
      @readers = "spacy.Corpus.v1"
      path = ${paths.train}
      max_length = 0
      gold_preproc = false
      limit = 0
      augmenter = null
      
      [training]
      dev_corpus = "corpora.dev"
      train_corpus = "corpora.train"
      seed = ${system.seed}
      gpu_allocator = ${system.gpu_allocator}
      dropout = 0.1
      accumulate_gradient = 1
      patience = 1600
      max_epochs = 0
      max_steps = 20000
      eval_frequency = 200
      frozen_components = []
      annotating_components = ["spancat"]
      before_to_disk = null
      
      [training.batcher]
      @batchers = "spacy.batch_by_words.v1"
      discard_oversize = false
      tolerance = 0.2
      get_length = null
      
      [training.batcher.size]
      @schedules = "compounding.v1"
      start = 100
      stop = 1000
      compound = 1.001
      t = 0.0
      
      [training.logger]
      @loggers = "spacy.ConsoleLogger.v1"
      progress_bar = false
      
      [training.optimizer]
      @optimizers = "Adam.v1"
      beta1 = 0.9
      beta2 = 0.999
      L2_is_weight_decay = true
      L2 = 0.01
      grad_clip = 1.0
      use_averages = false
      eps = 0.00000001
      learn_rate = 0.001
      
      [training.score_weights]
      spans_sc_f = 1.0
      spans_sc_p = 0.0
      spans_sc_r = 0.0
      
      [pretraining]
      
      [initialize]
      vectors = ${paths.vectors}
      init_tok2vec = ${paths.init_tok2vec}
      vocab_data = null
      lookups = null
      before_init = null
      after_init = null
      
      [initialize.components]
      
      [initialize.components.spancat]
      
      
      [initialize.components.spancat.labels]
      @readers = "spacy.read_labels.v1"
      path = "aa_labels.json"
      
      [initialize.tokenizer]


      ```

I was surprised that using init-config , I got annotating_components = []
but shouldnt it be
annotating_components = ["spancat"]

I am going to add a matcher, that would mean it would have to be changed to
annotating_components = ["matcher","spancat"] ?

Since I want the spancat to give final labels,
I only have to change to the following config, correct?

pipeline = [ "tok2vec", "matcher", "spancat"]

[components.matcher]
factory = "cm_identifier"

Below is a sample factory I took from #9776

import spacy
from spacy.language import Language
from spacy.tokens import Doc
from spacy.tokens import Span
from spacy.matcher import Matcher
from spacy.tokens import Token
from spacy.util import filter_spans
from itertools import *

@Language.factory("cm_identifier")
def create_measurement_identifier(nlp: Language, name: str):
    return QUANTITY(nlp.vocab)


class QUANTITY:
  
    def __init__(self, vocab):
        self.matcher = Matcher(vocab)
        self.vocab = vocab
        
        patterns = [[{'LIKE_NUM': True}, {"ORTH": 'cm'}]]

        self.matcher.add("centimeter", patterns)

    def __call__(self, doc: Doc):
        matches = self.matcher(doc, as_spans=True)
        matches = filter_spans(matches)

        new_ents = list(doc.ents)

        for span in matches:
            new_ent = Span(doc, span.start, span.end, label=self.vocab.strings[span.label])
            new_ents.append(new_ent)
          
        doc.ents = new_ents
        
        return doc

aa_labels.json

[ "SIZE", "POWER", "VOLTAGE", ]

The spancategorizer is not giving me scores close to NER for the same set of train/eval sets.Although I do understand that they do not have the same objective.NER converges to near 1.0 in 1000 steps , but SC converges near 0.82
Is there anything else I am missing?
f_score for matcher entities?or initialize?

[polm]

Glad you figured out how to use the custom tokenizer!

About annotating components - annotating_components refers only to components with annotations visible to other components during training. Normally each component sees clean input docs, unaffected by other components in the pipeline, but you can change that by using annotating_components. You don't need it for components that modify documents in the final pipeline as normal, such as your spancat. See here for details.

About your custom Matcher - have you considered using the EntityRuler? I pointed it out in the thread you linked to, but a custom component like that is basically the same as the EntityRuler.

About spancat accuracy, 80F1 certainly isn't bad. It's hard to say much more without knowing how much data you have, or what the full extent of variation is. It is plausible this kind of data is harder for spancat though - certainly detecting nested entities is harder than detecting non-nested ones in general.

[marzooq-unbxd]

I was trying the entity ruler as you mentioned,but had a doubt.

I was thinking ,for the entity , '12cm x 11cm', if entity-ruler is able to identify entities -SIZE['12cm','11cm'],the NER model would be able to better look around tokens identified.(I did not add patterns for "SIZE X SIZE" ).I always hence want NER to give the final entities(I think equivalent to overwrite_ents=True,I am not sure where to add this in config file).Should the entity ruler recognise the exact tokens we want identify,I was thinking i could identify some tokens(in above example, identify 2 out of 3 tokens in entity) and then allow NER to give final output.Am i going in the right direction.

Is this config file enough? I was getting worser scores as compared to 'NER alone'

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","entity_ruler","ner"]                    ###############ADDED THIS 
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100
###overwrite_ents = True                         ##############Where do i add this SINCE I WANT NER TO OVEREWRITE ENTITY RULER, IS ONLY THE ORDER REQUIRED  IN PIPELINE,

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.CharacterEmbed.v2"
width = 96
rows = 7000
nM = 64
nC = 8
include_static_vectors = false
[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 6
window_size = 1
maxout_pieces = 3

[components.entity_ruler]                                         ##################ADDED 2 LINEs
factory = "entity_ruler"                                                 ##########


[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 3000
max_epochs = 9
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null
[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
after_init = null

[initialize.before_init]
# @callbacks = "spacy.copy_from_base_model.v1"
@callbacks = "customize_tokenizer"
[initialize.components]

[initialize.tokenizer]
#########################################################BELOW ADDED
[initialize.components.entity_ruler]

[initialize.components.entity_ruler.patterns]
@readers = "srsly.read_jsonl.v1"
path = "PATTERNS/entity_ruler.jsonl"

entity_ruler.jsonl
Example:
{"label": "CURRENCY", "pattern": [{'TEXT' : {'REGEX': "#(dollars|cents|euros|rupees|usd|aud|cad|sgd|gbp|rs|inr|eur)"}}]}
{"label": "SIZE", "pattern": [{'TEXT' : {'REGEX': "#(millimeter|centimeter|meter|feet|foot|ft|mile|mi)"}}]}

I figured the entity ruler patterns should have the same label as NER labels, so that they could be overwritten.I have not added all possible combinations for SIZE(for eg: '5cm x 5cm'), since I want NER to learn that from the annotations I give

[polm]

The NER component doesn't have an overwrite setting, it just respects existing tokens. But since you want it to overwrite the EntityRuler, having the NER first, with the EntityRuler after and set to overwrite_ents = False should be equivalent.

About where to set overwrite_ents, you just need to set it in the component. It is correct that the order of components determines how overwriting works.

Your entity ruler initialization looks fine.