[Request] Doctest highlighting for Python

[Erotemic]

Is your request related to a specific problem you're having?

I'm frustrated when syntax highlighers ignore doctests in Python docstrings. When working in gitlab its hard for people to review doctests I write because the syntax isn't highlighted. I understand that this library is what gitlab uses for highlighting, so implementing support here should solve that issue.

The solution you'd prefer / feature you'd like to see added...

I would like to request support for highlighting doctest syntax in docstrings.

This feature is being added to rouge-ruby: rouge-ruby/rouge#1932 and has been supported by vim for years:

[joshgoebel]

How would we know that is a docstring vs just a normal string?

We do currently support highlighting the >>> blocks within """ strings, as the meta scope (not as code itself) ... but we don't do anything to try and handle "output blocks" (green in your snap above I believe)...

I think we might be open to someone working on a PR to improve this, though we may have a small problem in that there is no way to specify "return to normal code formatting"... so once you're in a string you're stuck with the styling from that... there isn't any great way to interleave code into the string and reset the highlighting accordingly - you'll still be inheriting styling from the string block. (which technically you are still inside of)

We do have subst scope for inline code, but that's not guaranteed to match the default coloring (ie, normal code that's unhighlighted)... perhaps we need a new scope for "default"...?

---

Do you have an example from gitlab?

[Erotemic]

Here is an example of a doctest in a gitlab snippet: https://gitlab.com/-/snippets/2502446

How would we know that is a docstring vs just a normal string?

This would only happen in triple-quoted string. The way that vim handles it is that any tripple quoted string where a new line starts with a >>> and then the syntax transitions into the "want" part of the docstest (where you put the expected output) when it finds a new line that doesn't start with >>> or ... , and then it goes back to a normal string on the next empty newline.

More percicely this is how vim tags start and end tokens:

" Triple-quoted strings can contain doctests.
syn region  pythonString matchgroup=pythonQuotes
      \ start=+[uU]\=\z(['"]\)+ end="\z1" skip="\\\\\|\\\z1"
      \ contains=pythonEscape,@Spell
syn region  pythonString matchgroup=pythonTripleQuotes
      \ start=+[uU]\=\z('''\|"""\)+ end="\z1" keepend
      \ contains=pythonEscape,pythonSpaceError,pythonDoctest,@Spell
syn region  pythonRawString matchgroup=pythonQuotes
      \ start=+[uU]\=[rR]\z(['"]\)+ end="\z1" skip="\\\\\|\\\z1"
      \ contains=@Spell
syn region  pythonRawString matchgroup=pythonTripleQuotes
      \ start=+[uU]\=[rR]\z('''\|"""\)+ end="\z1" keepend
      \ contains=pythonSpaceError,pythonDoctest,@Spell

...

" Do not spell doctests inside strings.
" Notice that the end of a string, either ''', or """, will end the contained
" doctest too.  Thus, we do *not* need to have it as an end pattern.
if !exists("python_no_doctest_highlight")
  if !exists("python_no_doctest_code_highlight")
    syn region pythonDoctest
	  \ start="^\s*>>>\s" end="^\s*$"
	  \ contained contains=ALLBUT,pythonDoctest,pythonFunction,@Spell
    syn region pythonDoctestValue
	  \ start=+^\s*\%(>>>\s\|\.\.\.\s\|"""\|'''\)\@!\S\++ end="$"
	  \ contained
  else
    syn region pythonDoctest
	  \ start="^\s*>>>" end="^\s*$"
	  \ contained contains=@NoSpell
  endif
endif

And here is the patch that implemented it in rouge: https://github.com/rouge-ruby/rouge/pull/1932/files

The vim higlighter isn't perfect. It breaks on this string within a string test in xdoctest (although it's a lot better than nothing).

But it handles most cases fairly gracefully:

though we may have a small problem in that there is no way to specify "return to normal code formatting".

Is there not a way to say that a string can contain a special higlighted pattern, and have rules for when that pattern ends?

[joshgoebel]

Is there not a way to say that a string can contain a special highlighted pattern, and have rules for when that pattern ends?

Sure, but the issue is you can't get "back" to "normal code" highlighting inside a string because of the way CSS works and the fact that we have no scope for "normal code"... if we did create a scope I feel the big argument would be whether it should restore background color... ie should code inside a text string share the background color of code, or of string. Perhaps that's a theme decision, but then how do we default the 100s of existing themes?

Perhaps the CSS issue is solved with color: default? I don't think so though, I think what you really want is something like color: inherit(from code), but I don't think that's a thing.

That said, someone could just start the grammar work and worry about the styling later... as mentioned we already support meta for the >>> blocks... so we'd just need to add support for "output" blocks following >>> blocks... (we'd need another new scope for this I think perhaps?). After that one could circle back and consider replacing meta with subst and allowing other python highlighting... all steps on the road to doing this fully.

---

FYI, I don't see any indication at a glance that Gitlab is using us for highlighting.

[Erotemic]

Here is the doc that describes how gitlab is using this for highlighting: https://docs.gitlab.com/ee/user/project/highlighting.html

I'm not very knowledgable in terms of CSS / JS, but thank you for considering this improvment. Better doctest support in highlighters in genral will I think help a fair number of people in the longer run.

[joshgoebel]

I think the important part there might be "and the rouge gem". If they are using HLJS they are using a heavily modified version- I couldn't even recognize the markup, so I wouldn't make assumptions about their upgrade timing.