custom pipeline component (phuzzymatcher) not working with error (TypeError: 'int' object is not iterable)

[lorijwong]

I'm trying to edit pipeline component (PhuzzyMatcher) for fuzzy matching created by jackmen to work with Spacy v3.x, as the original build works only with Spacy v2.x, but I'm having problems even though all I'm adding is the Language.Factory decorator.

def fuzzy_matcher(features, document, match=None):
matches = []
tokens = nltk.word_tokenize(document)
for feature in features:
feature_length = len(feature.split(" "))
for i in range(len(tokens) - feature_length + 1):
matched_phrase = ""
j = 0
for j in range(i, i + feature_length):
if re.search(r'[,!?{}[]]', tokens[j]):
break
matched_phrase = matched_phrase + " " + tokens[j].lower()
matched_phrase.strip()
if not matched_phrase == "":
if fuzz.ratio(matched_phrase, feature.lower()) > match:
matches.append([matched_phrase, feature, i, j])
return matches

def fuzzy_matcher_stopwords(features, document, match=None, stop_words=None):
matches = []
tokens = nltk.word_tokenize(document)
tokens_no_stop = [w for w in tokens if w not in stop_words]
for feature in features:
feature_length = len(feature.split(" "))
for i in range(len(tokens_no_stop) - feature_length + 1):
matched_phrase = ""
j = 0
for j in range(i, i + feature_length):
if re.search(r'[,!?{}[]]', tokens_no_stop[j]):
break
matched_phrase = matched_phrase + " " + tokens_no_stop[j].lower()
matched_phrase.strip()
if not matched_phrase == "":
if fuzz.ratio(matched_phrase, feature.lower()) > match:
matches.append([matched_phrase, feature, i, j])
return matches

nlp = spacy.blank('en')

@Language.factory("phuzzymatcher")
def create_phuzzymatcher(nlp:Language, name:str, set: list, match, label=None, stop_words=None):
return PhuzzyMatcher(nlp, set, match, label, stop_words)

class PhuzzyMatcher:

def __init__(
    self, 
    nlp,
    name, 
    set,  
    match, 
    label, 
    stop_words=None):
    
    self.nlp = nlp
    self.set = set
    self.fuzzy_matcher = fuzzy_matcher
    self.fuzzy_matcher_stopwords = fuzzy_matcher
    self.match = match
    self.label = label
    self.stop_words = stop_words

def __call__(self, doc):
    match_list = []
    if self.stop_words:
        results = self.fuzzy_matcher_stopwords(self.set, doc.text.lower(), self.match, self.stop_words)
    else:
        results = self.fuzzy_matcher(self.set, doc.text.lower(), self.match)
    for i in results:
        match_list.append(str(i[0].lstrip()))
    patterns = [nlp.make_doc(text) for text in match_list]  # noqa: F821
    matcher = PhraseMatcher(self.nlp.vocab, attr='LOWER')
    matcher.add(self.label, None, *patterns)
    matches = matcher(doc)
    seen_tokens = set()
    new_entities = []
    entities = doc.ents
    for match_id, start, end in matches:
        if start not in seen_tokens and end - 1 not in seen_tokens:
            new_entities.append(Span(doc, start, end, label=match_id))
            entities = [
                e for e in entities if not (e.start < end and e.end > start)
            ]
            seen_tokens.update(range(start, end))

    doc.ents = tuple(entities) + tuple(new_entities)
    return doc

My error code:
---> 11 for feature in features:
12 feature_length = len(feature.split(" "))
13 for i in range(len(tokens) - feature_length + 1):

TypeError: 'int' object is not iterable

If I run the different functions separately, it works, and there is nothing in the fuzzy_matcher function that is not iterable as far as I can see.

EDIT2:
phuzzymatcher.txt

[lorijwong]

Info about spaCy

• spaCy version: 3.4.4
• Platform: Windows-10-10.0.19045-SP0
• Python version: 3.11.1
• Pipelines: en_core_sci_sm (0.5.1), en_ner_bc5cdr_md (0.5.1)

[shadeMe]

Would you mind re-formatting the code in the original post such that it's formatted correctly and whitespace is preserved? You can also attach the .py file(s) directly to the post, but you might have to change the file extension due to GitHub's restrictions.

[lorijwong]

Hi shadeMe,
I just edited the original post with a link to the code attached at the end.

[shadeMe]

Thanks, but it seems like the file doesn't contain all the of code required to reproduce the error? What appears to be missing the code that initializes the matcher with some patterns and calls it on some text. Could you also upload that (ideally everything in a single file so that I can run it once to get to the error)?

[lorijwong]

phuzzymatcher.txt
I corrected the issue. Thanks for the feedback, this is my first time posting in a discussion forum in Github so this has been very helpful.

[shadeMe]

Seems like your issue stems from the fact that the positional parameters passed to the PhuzzyMatcher constructor are incorrect:

@Language.factory("phuzzymatcher")
def create_phuzzymatcher(
    nlp: Language, name: str, set: list, match: int, label=None, stop_words=None
):
    return PhuzzyMatcher(nlp, set, match, label, stop_words)

class PhuzzyMatcher:
    def __init__(self, nlp, name, set, match, label, stop_words=None):
		...

Passing name before set resolves the error you're seeing.

On a related note, when spaCy v3.5 is released, it will include fuzzy matching support in the Matcher class.