Fix non-ASCII path handling in common argument parser on Windows

[kuguma]

Summary

Fixed the behavior of the llama.cpp general argument parser when given non-ASCII paths.

On Windows, argv strings are received in the system’s ANSI code page (ACP), so conversion to UTF-8 is required.

Comparison of behavior

Windows 11 Pro 24H2

PS > Get-WinSystemLocale
LCID             Name             DisplayName
----             ----             -----------
1041             ja-JP            Japanese (Japan)
PS > [Console]::InputEncoding.EncodingName
Japanese (Shift-JIS)
PS > [Console]::OutputEncoding.EncodingName
Unicode (UTF-8)

b6756

PS > llama-cli.exe -m "..\モデル\gemma-3-4b-it-Q4_K_M.gguf"
build: 6756 (bc07349a7) with MSVC 19.44.35214.0 for x64
main: llama backend init
main: load the model and apply lora adapter, if any
gguf_init_from_file: failed to open GGUF file '..\���f��\gemma-3-4b-it-Q4_K_M.gguf'
llama_model_load: error loading model: llama_model_loader: failed to load model from ..\���f��\gemma-3-4b-it-Q4_K_M.gguf
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model '..\���f��\gemma-3-4b-it-Q4_K_M.gguf', try reducing --n-gpu-layers if you're running out of VRAM
main: error: unable to load model

This PR

PS > llama-cli.exe -m "..\モデル\gemma-3-4b-it-Q4_K_M.gguf"
build: 6758 (7ae5e44d4) with MSVC 19.44.35214.0 for x64
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 40 key-value pairs and 444 tensors from ..\モデル\gemma-3-4b-it-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
~~~
> 

Note

This PR is related to

• mtmd : Fix/Add non-ASCII file path support on Windows #16609.

Reproducing the issue with non-ASCII .mmproj paths in existing CLI tools requires first applying the changes from this PR. (This doesn’t apply when the tools are called internally.)

[ggerganov]

Suggested change

	#ifdef _WIN32
	struct utf8_argv {
	std::vector<std::string> buf;
	std::vector<char*> ptrs;
	};
	static utf8_argv make_utf8_argv() {
	utf8_argv out;
	int wargc = 0;
	LPWSTR* wargv = CommandLineToArgvW(GetCommandLineW(), &wargc);
	if (!wargv) return out;
	out.buf.reserve(wargc);
	for (int i = 0; i < wargc; ++i) {
	int n = WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS, wargv[i], -1, nullptr, 0, nullptr, nullptr);
	if (n <= 0) { out.buf.emplace_back(); continue; }
	auto& s = out.buf.emplace_back();
	s.resize(static_cast<size_t>(n - 1));
	(void)WideCharToMultiByte(CP_UTF8, 0, wargv[i], -1, s.data(), n, nullptr, nullptr);
	}
	LocalFree(wargv);
	out.ptrs.reserve(out.buf.size() + 1);
	for (auto& s : out.buf) out.ptrs.push_back(s.data());
	out.ptrs.push_back(nullptr);
	return out;
	}
	#endif
	#ifdef _WIN32
	struct utf8_argv {
	std::vector<std::string> buf;
	std::vector<char *> ptrs;
	};
	static utf8_argv make_utf8_argv() {
	utf8_argv out;
	int wargc = 0;
	LPWSTR * wargv = CommandLineToArgvW(GetCommandLineW(), &wargc);
	if (!wargv) return out;
	out.buf.reserve(wargc);
	for (int i = 0; i < wargc; ++i) {
	int n = WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS, wargv[i], -1, nullptr, 0, nullptr, nullptr);
	if (n <= 0) { out.buf.emplace_back(); continue; }
	auto & s = out.buf.emplace_back();
	s.resize(static_cast<size_t>(n - 1));
	(void)WideCharToMultiByte(CP_UTF8, 0, wargv[i], -1, s.data(), n, nullptr, nullptr);
	}
	LocalFree(wargv);
	out.ptrs.reserve(out.buf.size() + 1);
	for (auto & s : out.buf) out.ptrs.push_back(s.data());
	out.ptrs.push_back(nullptr);
	return out;
	}
	#endif

[kuguma]

I checked the line endings of the code committed to the repository just in case, and they appear to be LF, consistent with the rest of the files.

[kuguma]

Docker Pull fails for some actions