Parser fails at double quote character

[tehtris-siem]

Hi,

I parsed the expression Category:"Logon and expected SearchField('Category', Word('"Logon')).

However i received luqum.exceptions.IllegalCharacterError.

I parsed the expression Category:Logon" and expected luqum.exceptions.IllegalCharacterError.

However i received SearchField('Category', Word('Logon"')).

from luqum.parser import parser

# No traceback
tree = parser.parse('Category:Logon"')
print(repr(tree))

# Traceback
tree = parser.parse('Category:"Logon')
print(repr(tree))

Is this an expected behavior ?

Shouldn't both expressions react in the same way ?

Is it related to this issue ? #86

[alexgarel]

Yes it might be related.
I'm not sure it's easy to fix with current parser.

[tehtris-siem]

Hi, any update about a potential fix of this issue?

[alexgarel]

Sorry I don't have time to work on it personally right now (maybe at beginning of 2023.

If you want to try to tackle it, I can take the time to point you in the right direction.

[ahankinson]

hi @alexgarel -- I'm interested in trying to fix this issue. Any chance you could point me in the right direction? Thanks!

[alexgarel]

Hi @ahankinson thanks for your help !

It all happens in https://github.com/jurismarches/luqum/blob/master/luqum/parser.py

We use PLY (by the way we could try to upgrade the version).

You will have to understand PLY's basis.

Maybe it's about refining the TERM_RE regexp ? But it might create ambiguity in the parser (how to understand "this thing", is it term "this AND thing" or "this thing" phrase. I'm not sure we would be able to distinguish ambiguity in an intuitive way.

So I would rather say, you have to escape the double quote in this case (but it's not the job of luqum to do that). Like Category:\"Logon

Of course an improvement would be to give some hint about escaping the double quote as we get a IllegalCharacterError, and we see this pattern in the expression (but PLY does not give much context on errors).

Note that single quote is not a valid phrase delimiter (see doc)