ZnUTF8Encoder filters out any U+FEFF

[Rinzwind]

ZnUTF8Encoder filters out any U+FEFF, the following answers '42' rather than 'FEFF' for example:

(ZnUTF8Encoder new decodeBytes: #[16r41 16rEF 16rBB 16rBF 16r42])
	second codePoint printStringBase: 16

This doesn’t quite seem to be in accordance with the Unicode standard, which says (in section ‘23.8 Specials’ of ‘The Unicode Standard, Version 15.0 – Core Specification’):

For historical reasons, the character U+FEFF used for the byte order mark is named ZERO WIDTH NO-BREAK SPACE. […] Because the byte-swapped version U+FFFE is a noncharacter, when an interpreting process finds U+FFFE as the first character, it signals either that the process has encountered text that is of the incorrect byte order or that the file is not valid Unicode text. […] In UTF-8, the BOM corresponds to the byte sequence <EF₁₆ BB₁₆ BF₁₆>. Although there are never any questions of byte order with UTF-8 text, this sequence can serve as signature for UTF-8 encoded text where the character set is unmarked. […] For compatibility with versions of the Unicode Standard prior to Version 3.2, the code point U+FEFF has the word-joining semantics of zero width no-break space when it is not used as a BOM. […] Systems that use the byte order mark must recognize when an initial U+FEFF signals the byte order. In those cases, it is not part of the textual content and should be removed before processing, because otherwise it may be mistaken for a legitimate zero width no-break space. […] If U+FEFF had only the semantics of a signature code point, it could be freely deleted from text without affecting the interpretation of the rest of the text. […] Unfortunately, U+FEFF also has significance as a character. As a zero width no-break space, it indicates that line breaks are not allowed between the adjoining characters. Thus U+FEFF affects the interpretation of text and cannot be freely deleted.

[svenvc]

Well, I am sure you can see that this was done by design: the occurrence of a BOM is a no-op for UTF-8, should not occur and is not recommended. Hence this pragmatic decision.

I know that it is used on Windows, that is why there are some provisions for writing a BOM.

Should version 3.2 compatibility be a thing, since that is 20+ years old ?

Up to now there have not yet been any complaints about this behaviour.

What is the practical issue that you encounter ?

What do you think should happen/change ?

[Rinzwind]

Shouldn’t it at most filter out an initial U+FEFF? The example for ucnv_detectUnicodeSignature in the ICU libraries documentation might help. If ‘input’ is changed to:

char input[] = { '\xEF','\xBB','\xBF', '\xEF','\xBB','\xBF', '\x41', '\xEF','\xBB','\xBF', '\x42' };

The output is as follows, so only the initial U+FEFF is filtered out:

feff 0041 feff 0042

[svenvc]

In Python I see this:

$ python3
Python 3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> bytes = b'\xEF\xBB\xBF\xEF\xBB\xBF\x41\xEF\xBB\xBF\x42'
>>> bytes.decode("utf-8")
'\ufeff\ufeffA\ufeffB'

[svenvc]

In Swift I see this:

$ swift repl
Welcome to Apple Swift version 5.10 (swiftlang-5.10.0.13 clang-1500.3.9.4).
Type :help for assistance.
  1> import Foundation 
  2> let byteArray: [UInt8] = [239, 187, 191, 239, 187, 191, 65, 239, 187, 191, 66] 
byteArray: [UInt8] = 11 values {
  [0] = 239
  [1] = 187
  [2] = 191
  [3] = 239
  [4] = 187
  [5] = 191
  [6] = 65
  [7] = 239
  [8] = 187
  [9] = 191
  [10] = 66
}
  3> String(data: Data(bytes: byteArray), encoding: .utf8) 
$R0: String? = "AB"

[svenvc]

We could look at Java and JavaScript as well.

I am rather a fan of Swift's Unicode approach (in general not specifically this), it is quite modern.

Like I said, it was a design decision to skip BOM sequences like that.

And my question remains: what practical issue did you encounter ?

I guess it would not be impossible to add an option to UTFEncoder (the superclass of all UTF encoders) to control this behaviour but then the next question is what the default should be. (There is already a strict option for byte encoders).

Still we would be adding complexity and I currently do not see why.

[Rinzwind]

Note that the Swift REPL doesn’t escape U+FEFF when printing a string. This is clearer:

  4> String(data: Data(bytes: byteArray), encoding: .utf8)!.unicodeScalars.map{ String($0.value, radix: 16) }
$R1: [String] = 5 values {
  [0] = "feff"
  [1] = "feff"
  [2] = "41"
  [3] = "feff"
  [4] = "42"
}

In the following, the string is different depending on whether ‘utf16BigEndian’ or ‘utf16’ is used:

  5> let byteArray2: [UInt8] = [254, 255, 254, 255, 0, 65, 254, 255, 0, 66]
byteArray2: [UInt8] = 10 values {
[…]
}
  6> String(data: Data(bytes: byteArray2), encoding: .utf16BigEndian)!.unicodeScalars.map{ String($0.value, radix: 16) } 
$R2: [String] = 5 values {
  [0] = "feff"
  [1] = "feff"
  [2] = "41"
  [3] = "feff"
  [4] = "42"
}
  7> String(data: Data(bytes: byteArray2), encoding: .utf16)!.unicodeScalars.map{ String($0.value, radix: 16) }
$R3: [String] = 4 values {
  [0] = "feff"
  [1] = "41"
  [2] = "feff"
  [3] = "42"
}

Section ‘3.10 Unicode Encoding Schemes’ in the Core Specification says that “in UTF-16BE, an initial byte sequence <FE FF> is interpreted as U+FEFF ZERO WIDTH NO-BREAK SPACE”, while “in the UTF-16 encoding scheme, an initial byte sequence corresponding to U+FEFF is interpreted as a byte order mark”.

Deletion of U+FEFF when it is not used as a BOM may lead to a security problem as discussed in section ‘3.5 Deletion of Code Points’ in Unicode Technical Report #36 ‘Unicode Security Considerations’, which is referred to in the section ‘Modification’ on conformance clause C7 within section ‘3.2 Conformance Requirements’ in the Core Specification.

[svenvc]

fc59039

Maybe not 100% what you want, but at least there is now the option to see BOM occurrences.

Thx for the discussion.

[Rinzwind]

OK thanks! It would seem better for the option to also disable the endianness swapping after the U+FFFE in the following though, I would expect the first, third and fifth characters to be the same, but they are not:

((ZnCharacterEncoder newForEncoding: 'utf16be')
	ignoreByteOrderMark: false;
	decodeBytes: (ByteArray readHexFrom: ('00FB FEFF 00FB FFFE 00FB' reject: #isSeparator)))
		asArray collect: [ :character |
			character codePoint printStringBase: 16 nDigits: 4 ]
"⇒ #('00FB' 'FEFF' '00FB' 'FFFE' 'FB00')"