Text tokenizer is classifying the letter "O" as punctuation.

[dorncg18]

I am using the following code:

doc = nlp(text)
for token in doc:
    if token.pos_ == 'PUNCT':
        text = text.replace(token.text, '')

with the following raw text, read from a PDF using pyPDF

"with a proven track record of delivering strategic financial solutions for clients. Highly accomplished"

it is being converted to

"with a prven track recrd f delivering strategic financial slutins fr clients Highly accmplished"

I noted this behavior to the creator of the package I am using Resume Matcher, but I can keep the letter "O" in the output using this workaround:

doc = nlp(text)
for token in doc:
    if token.pos_ == 'PUNCT' and token.text != 'o':
        text = text.replace(token.text, '')

There may be an issue as to how the text is being read in from pyPDF, but looking at the results when using the pyPDF function, the text looks correct.

Info about spaCy

Python 3.9.0
Windows 10

• spaCy version: 3.6.0
• Platform: Windows-10-10.0.19041-SP0
• Python version: 3.9.0
• Pipelines: en_core_web_md (3.6.0), en_core_web_sm (3.6.0)

[svlandeg]

Hi!

If I run this sample code:

    nlp = spacy.load('en_core_web_lg')
    text = "with a proven track record of delivering strategic financial solutions for clients. Highly accomplished"
    doc = nlp(text)
    for token in doc:
        if token.pos_ == 'PUNCT':
            text = text.replace(token.text, '')
    print(text)

It gives me

with a proven track record of delivering strategic financial solutions for clients Highly accomplished

Which is what you want: the punctuation is removed, but no other characters are.

On your end, I can imagine that either you have a text that contains non-standard characters (that may look like an "o") or perhaps you're using a different model as tagger. BTW - note that your code replaces all instances of the token text with '', instead of just the one token that is classified as PUNCT. This is pretty error-prone, as one wrong FP by the tagger could lead to many FP hits in your text.

[dorncg18]

The text is being read in from a pdf, so the text looks like an “o” but is being treated as punctuation- what would a punctuation mark be that would display like an “o”?   I don’t follow this part- “BTW - note that your code replaces all instances of the token text with '', instead of just the one token that is classified as PUNCT. This is pretty error-prone, as one wrong FP by the tagger could lead to many FP hits in your text.”Your help is much appreciated