Why are the offsets in ripgrep's JSON output inconsistent with Python string lengths?

[0-EricZhou-0]

Please tick this box to confirm you have reviewed the above.

• I have a different issue.

What version of ripgrep are you using?

ripgrep 15.1.0 (rev af60c2d)

How did you install ripgrep?

Download precompiled binary from release

What operating system are you using ripgrep on?

Ubuntu 24.04.2 LTS and Ubuntu 22.04.5 LTS in wsl

Describe your bug.

When using ripgrep with --json flag, in the submatch fields, the ["data"]["submatches"]["end"] field can be wrong. However, the ["data"]["submatches"]["start"] and ["data"]["submatches"]["match"]["text"] is correct, and we can get the correct submatch end offset using submatch start offset + submatch text length.

Note that the file to search is a log file containing multiple \CR (^M) and \EC (^[) characters, and --multiline is enabled for the search but I think there is nothing to do with it.

What are the steps to reproduce the behavior?

See https://github.com/0-EricZhou-0/ripgrep-bug-report, I have made a minimum example to repoduce the issue.

What is the actual behavior?

Submatch end is at wrong index and could go beyond the match text length.

What is the expected behavior?

Submatch end should be at correct index.

[0-EricZhou-0]

Seems to be a encoding issue, python's json module will autometically do decode for me. I need to encode the strings back to bytes to get the correct length and offset.

[BurntSushi]

Yes. Your issue is assuming that Python's len function on a str will report a length consistent with the length used by ripgrep. Indeed, ripgrep uses byte offsets exclusively. Python, by contrast, uses codepoint offsets. Using codepoint offsets in ripgrep would be wildly inappropriate for two reasons. First is that you would need to decode the data into a sequence of codepoints before you could use the offsets to slice into it. By using byte offsets, you can store the data as a sequence of bytes in contiguous memory, and use the offsets to index or slice into it directly. Second is that ripgrep isn't limited to searching UTF-8. It can search anything, including invalid UTF-8. Codepoint offsets wouldn't work in that context.