Index by chars or bytes?

[jneem]

The documentation on the Marker::index field says it's in chars, but the documentation on the Marker::index method says it's in bytes. Which is it?

A quick look suggests the answer is "chars." This is unfortunate, in my opinion, because byte indices are more efficient for most purposes (if you have a str input, for example, you can slice it in constant time). But either way, the doc comments shouldn't conflict with one another...

[bourumir-wyngs]

From that I see in the code, I strongly suspect the real answer is "bytes"

Span::len() computes self.end.index() - self.start.index(), which is the byte length. Despite the documentation comment in src/scanner.rs (line 129) stating "Return the length of the span (in characters).", this is incorrect—the computation uses byte indices.

[jneem]

Span::len() computes self.end.index() - self.start.index(), which is the byte length

It's only the byte length if Marker::index is measured in bytes. If Marker::index is measured in chars, then Span::len() is the length in chars.

The reason I think it's measured in chars is that the various skip functions in input (like Input::skip_while_non_breakz, for example) return char counts. Then in the scanner, those char counts are used to increment self.mark.index.

[bourumir-wyngs]

You are right. I wrote a test code. Seems that offsets are reported in chars.