The need for a practical unicode codepoints encoder-decoder and validator module

[jangko]

Scope

UNICODE is a very broad topic and often confusing, no wonder it scares away many developers. And even in some great project, unicode support is postponed or has lower priority.

But from experiences, unicode encoding-decoding is actually simple. What makes it really complicated is the unicode glyph renderer. But that is not our concern. We should stick to the well defined unicode codepoints encoder-decoder and validator. Nothing else, period.

Why we need it in nim-stew

• As our libraries become mature, we cannot neglect a recurring issue: "Better support for full range unicode codepoints".
• During development of nim-websock, we discover flaw in nim-stdlib unicode module. It has incorrect UTF-8 validator.
• Nim-stdlib unicode module is using nim-string. Theoretically, that is correct because unicode text is a text that should be represented by a string. But from experience, we often found we need to deal with raw bytes coming from network or coming from some input stream that need to be parsed before we know it should be treated as bytes or string. That's why we need a practical module.
• We have collected some faster and more efficient UTF-8/16 converter, decoder, and validator scattered around in many repos.
• Together with encryption and compression, such as in a PDF library or PNG library, a flexible yet efficient unicode codepoints encoder-decoder is needed.
• From our numerous nim-repos, it's hard to find unicode aware library. So far I can only find:
  • https://github.com/status-im/nim-toml-serialization, it has full support for full range of UTF-8 because it is mandated by the spec.
  • https://github.com/status-im/nim-websock, it has full range UTF-8 validator because the test suite we are using, the autobahn, has extensive test cases for UTF-8.
  • https://github.com/status-im/nim-json-serialization, only partially support in the reader, and no support in the writer. And the support is only limited to escaped codepoints, not for binary encoding.
  • https://github.com/status-im/nim-graphql, aware of unicode, but the official spec is messy and not finished yet regarding unicode codepoints.

Based on above reasons and inspiration from other modules in nim-stew, definitely we can craft a better unicode codepoints module. This will greatly improves unicode support in our codebase.

Remaining obstacle

What is the appropriate name for this module? unicode is too broad, we are not dealing with every aspect of unicode. Only unicode codepoints encoding-decoding and validation often encountered during parsing text and raw bytes.

candidate: utf, because we are dealing with UTF-8/16/32 codec.

[arnetheduck]

status-im/nim-protobuf-serialization#13 as well

[jangko]

see this too for UTF-8 stress test

https://stackoverflow.com/a/1319356

[jangko]

we encountered a special and interesting UTF-8 validation case in status-im/nim-websock#85.

the usual approach when validating UTF-8 string/blob is we treat them as a single entity. but from the above case we learn that there might be cases where UTF-8 validator should able to validate slice by slice using a stateful validator.

we already have a state based UTF-8 validator, now it's time to design an API that is both ergonomic yet powerful enough to handle this case and other cases in an elegant way, hopefully.