Character Encoding Assertion Failure

[gaborbernat]

Universal Ctags crashes with assertion failure in vStringPutImpl at main/vstring.h:112 when encountering files with UTF-16 encoding. The assertion c >= 0 && c <= 0xff fails because ctags expects all characters to fit within a single byte range, but UTF-16 files contain null bytes and extended characters that violate this assumption. Note, for this to crash the binary you need to run in debug mode.

Reproduced by:

# Create UTF-16 LE encoded file with BOM
printf '\xff\xfe-\x00-\x00 \x00T\x00e\x00s\x00t\x00' > encoding_crash.sql

# This crashes with: vStringPutImpl: Assertion `c >= 0 && c <= 0xff' failed
./ctags -o /tmp/test encoding_crash.sql

# Alternative: use iconv to create UTF-16 file
echo "SELECT * FROM test;" | iconv -f UTF-8 -t UTF-16LE > encoding_crash.sql
./ctags -o /tmp/test encoding_crash.sql

Suggested fix: Add input encoding detection and proper Unicode handling to the vString implementation. The parser should either convert UTF-16 to UTF-8 internally or handle multi-byte characters properly without assuming single-byte character range.