How to run lexer.Tokenise() on line by line?

[gmelchett]

I'm writing a toy editor, using chroma for coloring and giving the editor some understanding of what is beneath the cursor.
The simple approach, running lexer.Tokenise() on the complete file works fine, but I would like to be able to do it on a line by line basis. (Or some kind of "chunk", a subset of the file.)

For example: When the user presses a key, I would like to run lexer.Tokenise() on just the current line that is changed and provide the current start state of the line, IE is the line inside a multi-line comment etc.
Then check the final state of the line, and if, for example, the user starts writing a new multiline comment run Tokenise on the lines visible in the editor.
Also, if the user opens a large file, I would rather only run lexer.Tokenise() on visible lines, and then continue Tokenise() the none-visible lines in the background.

Can this be done with chroma? If yes, how? Thanks!

[alecthomas]

Thanks for the discussion, unfortunately this is not possible currently.

[gmelchett]

Ok. Thanks for a quick response.

[antxv]

I am also creating a code editor that utilizes chroma for code coloring. As long as you carefully run Tokenise() in a background routine, it actually performs quite well on an entire file of reasonable size. I am triggering Tokenise() with each content update in the background routine, and I abort previous processing requests when they overlap (just keep final).

The primary issue is that Tokenise() cannot be cancelled once started. If a chunking version is not within scope, perhaps it would not be difficult to create a cancelable version of Tokenise()? This would provide the ability to abort a long running operation (large files), when you need to handle repeated content changes or quit the application gracefully.