Handling of unassigned code points ("stored strings" or "query" variant)

[tomprogrammer]

RFC3454 Section 7 describes handling of unassigned code points in stringprep. The RFC contains the text:

Stored strings using the profile MUST NOT contain any unassigned code points.
Queries for matching strings MAY contain unassigned code points.

Currently only the "stored strings" variant is implemented as far as I can tell from glossing over the code. I'd like to request an implementation of the "queries" variant.

[tomprogrammer]

Maybe I can provide a PR, it seems not to be very difficult to implement due to the very clear structure of the functions.

[sfackler]

Unassigned code points don't exist in a &str-oriented API. We could add on a &[u8]-oriented API for queries?

[tomprogrammer]

I'm not very familiar with Unicode and don't know whether rust allows unassigned code points in char or &str. So I've now learned that they don't exist in &str-typed values.

Nevertheless the RFC mentions that "all code points not assigned in the character repertoire named in a stringprep profile are called 'unassigned code points'." I didn't read either your code nor the stringprep RFC yet, so I concluded, without further knowledge, that unassigned codepoints in a stringprep profile not necessarily mean that they are unassigned codepoints in the context of Unicode. In fact there exists a check for unassigned codepoints in the saslprep function.

So the next step for me is to read about this topic and understand the exact meaning of the terms. Then I will be able to understand what needs to be implemented.

[sfackler]

Cool, sounds good!

[sfackler]

Ah, sorry, I was thinking of surrogate codes when I said they couldn't be in strings.