Skip to content

Bug Fix: win32 readFileContents() cannot read file names with unicode characters. (Fix utf8 to utf16)#159

Merged
nalgeon merged 3 commits intonalgeon:mainfrom
MVV90:win32-utf16-readFileContents
Feb 6, 2026
Merged

Bug Fix: win32 readFileContents() cannot read file names with unicode characters. (Fix utf8 to utf16)#159
nalgeon merged 3 commits intonalgeon:mainfrom
MVV90:win32-utf16-readFileContents

Conversation

@MVV90
Copy link
Contributor

@MVV90 MVV90 commented Feb 6, 2026

On Windows, running sqlite with sqlean extension: select fileio_read('hello世界.txt'); will fail silently, returning an empty string, regardless if hello世界.txt has contents.

Windows uses UTF16 for it's native file system encoding for all file names (versions NT, 2000, XP, 7, 10, 11). Files with unicode characters (eg: 世界 in hello世界.txt) would be converted to UTF8 internally to sqlean, when Windows is expecting a UTF16 encoding. This prevents sqlean from reading files on windows (that have unicode characters in the file name). Windows function _wfopen works with file names with or without unicode characters.

  • Adds fix for win32 systems
  • Add unit test coverage for reading file names with unicode characters in them.

@nalgeon
Copy link
Owner

nalgeon commented Feb 6, 2026

Thank you for the PR!

Compilation on Windows is failing:

gcc -O3  -shared -Isrc -DSQLEAN_VERSION='"main"' src/sqlite3-fileio.c src/fileio/*.c -o dist/fileio.dll
src/fileio/legacy.c: In function 'readFileContents':
src/fileio/legacy.c:167:50: error: 'zPath' undeclared (first use in this function)
  167 |     zUnicodeName = sqlite3_win32_utf8_to_unicode(zPath);
      |                                                  ^~~~~
src/fileio/legacy.c:167:50: note: each undeclared identifier is reported only once for each function it appears in

@nalgeon nalgeon merged commit 2d70fef into nalgeon:main Feb 6, 2026
3 checks passed
@nalgeon
Copy link
Owner

nalgeon commented Feb 6, 2026

Thanks!

@MVV90 MVV90 deleted the win32-utf16-readFileContents branch February 6, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments