Ambiguous-width Unicode character rendering

[fancidev]

Summary. I'm making this thread to collect a few test cases from Discord related to rendering ambiguous-width Unicode characters. (Not a new topic, but I can't seem to find any open issues or discussions on this.)

Problem. It's kind of a known limitation that the layout (especially borders) in Textual is sometimes misplaced if certain Unicode characters (for example, emojis) are rendered by the terminal with an unexpected width. The root cause, which makes the issue non-trivial to fix, is that different terminal (emulator) renders certain Unicode characters differently in terms of cell width. There are good and bad reasons for such differences, but it seems safe to assume that the discrepancies are there to stay for at least another 10 years.

Solution. To make Textual's layout consistent across terminals (I'm not talking about niche terminals, but popular or system default ones), a potential solution could be to measure the actual cell width of a character by the number of cells advanced after rendering. This solution has been brought up a few times by Will and others, but it's never implemented so I suppose it's not straightforward to. Anyway, I hope the examples collected below could be used to test a future solution if one is proposed.

---

Mitchell Hashimoto wrote a great summary on grapheme related issues in terminal emulators.

gh-5243 reports "misplacement" of border when Windows 10's terminal renders a grapheme cluster in four cells while it should have taken up two cells.

[edward-jazzhands]

Good thinking to document this in a discussion. I'd just like to add that I still think a serious possibility for a solution is to add an option to force all emojis to have a width of 2, if it is even possible to do such a thing. Its a crude solution but perhaps if its an option available then it could help a few people.

[fancidev]

Do you mean to hardcode a (possibly rule-based) list of Unicode characters to have width two when computing the layout? (But there is no guarantee those characters end up taking two cells when rendered.)

[edward-jazzhands]

I was thinking more like an option that would simply force all unicode characters to be treated as though they take up 2 cells by Textual even if they only take up 1. I'm not sure that's even possible its just a farfetched idea, really.

[fancidev]

Digging into the source code of Textual and Rich, it seems cell length calculation boils down to the cell_len function in rich, and texts (with embedded control sequences) are sent to the terminal via App._display in textual. I think it might be possible to do something with those functions to automatically detect the rendered cell width.

Ignoring grapheme clusters for the moment, a possibility is to insert a cursor report request before and after each character whose cell len has not been measured. Since input and output are asynchronous, some book keeping is needed to make sure the reported cursor positions (and their difference) are mapped back correctly to the character being measured. Then the measured cell length is fed into rich to remember it. If any character has a different cell length as expected, a Resize event is triggered to re-layout the whole application.

Since each character is measured only once, and I expect the number of different characters in a real-world application to not exceed a few hundred, the overhead is expected to be minimal.

The challenge is probably about sorting out the details to make this actually work.

[SlySven]

Is this project of any use to you? We (the Mudlet MUD client) have made use of it to identify whether the first codepoint in a grapheme is likely to be narrow or wide - or if it falls within a range where the width is "ambiguous". It isn't a perfect solution - some "special" casing is still needed but it does help!

https://github.com/ridiculousfish/widecharwidth