`zsh` completion descriptions, ANSI escapes and multiple lines

[dbuenzli]

Since 621983e cmdliner's doclang is handled in the doc strings used for completion descriptions. We found it natural to use ANSI escapes in the doc string renders and this is specified in the protocol.

However this works only some of the time. Here are a few problems:

1. If the doc string is long. It is arbitrarily cut to fit the terminal. First this is not nice from a usability perspective, who wants to read truncated doc strings ? Second it may cut out an ANSI mode reset, which means bold/italic glitches in the list of completions.
2. It seems that when the zsh completion system collapses short and long option it escapes the escapes.
   E.g.:

   > b0 -␉
   --b0-dir           -- Use ^[[04mDIR^[[m for the b0 directory.                                                             
   --b0-file          -- Use ^[[04mPATH^[[m as the b0 file.                                                                  
   --build        -b  -- Build only. Do not execute the action.   
   

   But (with DIR, PATH variables correctly ANSI underlined):

   > b0 --␉
   --b0-dir       -- Use DIR for the b0 directory.
   --b0-file      -- Use PATH as the b0 file.
   

Regarding 1. I wonder if there is a way to be able to specify multi-line completion descriptions in zsh. Regarding 2. I wonder if there is a way to indicate to zsh. Answering these questions likely involves going to look at their code as trying to get anything meaningful from their docs is a challenge.

Another option of course is to strip the escapes from the doc strings in the completion script – I rather keep them in the protocol, perhaps one day we'll get less duck taped completion systems.

[dbuenzli]

So it seems not much can be done here. In 4d63431 the protocol was upgraded to support multiple lines but if we try to output newlines in the completion documentation string:

1. These lines simply end up at the begining of the line which makes things unreadable. We could try to layout ourselves but it depends on the longest option and there is point 2. anyways.
2. It seems the doc strings gets truncated after a certain number of characters (likely dependent on the number of columns) regardless.

The level of brokenness and lack of usability thinking in these systems is uncanny.

Unless someone fixes these shell completions mecanisms I don't think there's much we can do. Let's leave it like that for now but perhaps we should still do this at some point:

Another option of course is to strip the escapes from the doc strings in the completion script

[edwintorok]

I assume this is why on ghostty with zsh the letter g is cut off in the completion?

cmdliner                                         (master S)
Completing unsorted
--version  -- Show version information.
--help     -- Show this help in format FMT. The value FMT must be one of auto
eneric-completion  -- Output generic completion scripts
tool-commands       -- Output all subcommands of a cmdliner tool
tool-completion     -- Output tool completion scripts
install             -- Install support files for cmdliner tools

Using NO_COLOR=1 works it around:

export NO_COLOR=1
edwin@fedora ~/git/cmdliner [5.3.0+fp] ⭘ ❯ cmdliner                                               (master S)
Completing unsorted
--version  -- Show version information.
--help     -- Show this help in format FMT. The value FMT must be one of auto, pager, groff or plain. Wit
generic-completion  -- Output generic completion scripts
tool-commands       -- Output all subcommands of a cmdliner tool
tool-completion     -- Output tool completion scripts
install             -- Install support files for cmdliner tools

[edwintorok]

I'll have to look in more detail at how zsh completions work, but for prompts the ANSI escape sequence usualy need to be wrapped in %F/%K (which knows their width), or %G, where the documentation says

   %G     Within  a  %{...%} sequence, include a `glitch': that is, assume that a single character width
         will be output.  This is useful when outputting characters that otherwise cannot be  correctly
         handled  by  the shell, such as the alternate character set on some terminals.  The characters
         in question can be included within a %{...%} sequence together with the appropriate number  of
         %G  sequences  to  indicate the correct width.  An integer between the `%' and `G' indicates a
         character width other than one.  Hence %{seq%2G%} outputs seq and  assumes  it  takes  up  the
         width of two standard characters.

So we probably can't output ANSI escapes directly from the completion script, but would have to use zsh's formatting functions to let it know how wide each one is (well 0 for pure formatting chars, but there is other unicode that might need to be protected too in a similar way).
Now if only zsh had a function that parsed an arbitrary text with control chars, and replaced it with the appropriate formatting sequences then this'd be a lot easier (kind of like what Fmt.text does), otherwise might need something that parses the string and replaces every non-ASCII char with appropriate zsh formatting chars so that it'll know about display width (and hopefully then avoid cutting it in the middle).

[dbuenzli]

If you find tricks that will be great :–) note that these will have to occur in the generic shell script though. At some point I disabled the sequences see this commit but then thought that in a lot of cases they don't look bad. Not sure what is best.

[edwintorok]

This is where I got so far:

        item_doc="${item_doc//\%/%%}"
        # (S) is for non-greedy matches, (#b) turns on backreferences, so that ${match[1]} works
        item_doc="${(S)item_doc//$'\e'\[01m(#b)(*)$'\e'\[m/%B${match[1]}%b}"
        item_doc="${(S)item_doc//$'\e'\[04m(#b)(*)$'\e'\[m/%U${match[1]}%u}"
       # unfortunately we need to call print -P here, so I'm not sure this is guaranteed to have the desired effect
        item_doc="$(print -P "${item_doc}")"
        completions+=("${item}":"${item_doc}")

It seems to work on the one example I tried, but I'll need to do more testing and see whether there is a better way to pass the %B formats directly.
Although print -P does change the escape sequences, and therefore their length, so it could be just coincidence it is not truncated at the wrong place anymore:

print -P '%Bfoo%b%Ubar%u'|hexdump -C
00000000  1b 5b 31 6d 66 6f 6f 1b  5b 30 6d 1b 5b 34 6d 62  |.[1mfoo.[0m.[4mb|
00000010  61 72 1b 5b 32 34 6d 0a                           |ar.[24m.|
00000018

(it uses 1m and 0m; and 4m and 24m)

[dbuenzli]

@edwintorok what's your feeling?

My feeling is that I'm going to at least switch back to strip them for the next release, sometimes I get really botched completion visualisation results and I suspect it's due to this.

But if you feel you have something better I'm happy to integrate it.

[edwintorok]

But if you feel you have something better I'm happy to integrate it.

I don't have anything reliable at the moment (the idea from my previous comment looked promising, but I don't think I got it working in all situations), if you want to strip the ANSI escapes in the next release during zsh completion I wouldn't mind.
They can always be added back if a better solution is found.