Parsing overlapping tokens?

[dabrahams]

Hi there,

I'm working with a grammar where tokens (used in distinct partial parses) can overlap. I know Earley can handle this situation (I've implemented the Earley algorithm myself). That capability isn't documented but is it hiding in the implementation somewhere? Thanks!

(I also know Earley is amenable to a system where tokens are “pushed” into the parser as opposed to being “pulled” through a next_token function, which is great for flexibility. To handle the capability I'm asking for you'd get the tokens along with their start positions, in the order of their end positions).