fix: add UTF-8 encoding to all file operations (#232)

[yurekami]

Summary

Fix #232: Add explicit encoding="utf-8" to all file open operations to prevent UnicodeDecodeError when processing files with non-ASCII characters.

Problem

Files containing non-ASCII characters (like emoji in chat templates or international characters in config files) would fail to read on systems where the default encoding isn't UTF-8, causing:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 31733

Changes

• hf_utils.py: Add encoding to processor_config file reading
• saving_utils.py: Add encoding to tokenizer_config.json and config.json read/write operations

Test plan

• Verify files with Unicode content can be read successfully
• Verify config files are saved correctly with non-ASCII characters

🤖 Generated with Claude Code

[gemini-code-assist]

Summary of Changes

Hello @yurekami, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical issue where file operations could fail due to UnicodeDecodeError when encountering non-ASCII characters. By consistently applying UTF-8 encoding to all relevant file read and write calls, the change significantly enhances the robustness and internationalization capabilities of the system, allowing for seamless handling of diverse character sets in configuration and chat template files.

Highlights

• UTF-8 Encoding: Explicitly added encoding="utf-8" to all file open operations to prevent UnicodeDecodeError when processing files that contain non-ASCII characters, such as emojis or international characters.
• Improved Compatibility: Ensures that the application can correctly read and write configuration and template files across different operating systems and environments, regardless of their default text encoding.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly adds encoding="utf-8" to file open operations, which will prevent UnicodeDecodeError on systems with different default encodings. The changes are sound and address the issue described. I've added a few review comments with suggestions to improve code quality, efficiency, and robustness in the modified sections. These include using the json module more idiomatically, refactoring for better performance, and improving exception handling.

[gemini-code-assist]

Using string replacement on a JSON file is brittle and can lead to unexpected behavior. For instance, it could replace the substring 'dtype' within a string value, not just as a key. A more robust approach is to parse the JSON file, modify the resulting dictionary, and then serialize it back to the file. This ensures that only the intended key is modified.

Additionally, the bare except: is too broad and can mask important errors. It's better to catch specific exceptions like IOError and json.JSONDecodeError.

Suggested change

	try:
	with open(config_file_path, "r") as f:
	with open(config_file_path, "r", encoding="utf-8") as f:
	data = f.read()
	data = data.replace('"dtype"', '"torch_dtype"')
	data = data.replace("'dtype'", "'torch_dtype'")
	with open(config_file_path, "w") as f:
	with open(config_file_path, "w", encoding="utf-8") as f:
	f.write(data)
	except:
	pass
	try:
	with open(config_file_path, "r", encoding="utf-8") as f:
	config = json.load(f)
	if "dtype" in config:
	config["torch_dtype"] = config.pop("dtype")
	with open(config_file_path, "w", encoding="utf-8") as f:
	json.dump(config, f, indent=4, ensure_ascii=False)
	except (IOError, json.JSONDecodeError):
	pass

[gemini-code-assist]

For improved readability and to follow common Python idioms, you can use json.load() to parse the JSON directly from the file object. This is more concise than reading the file into a string first and then parsing it. It also avoids shadowing the file object f with the file's content.

Suggested change

	with open(processor_config, "r", encoding="utf-8") as f: f = f.read()
	config = json.loads(f)
	with open(processor_config, "r", encoding="utf-8") as f:
	config = json.load(f)

[gemini-code-assist]

This section can be refactored for better efficiency and readability. Currently, it reads and writes to tokenizer_config.json twice, which can be consolidated into a single read-modify-write operation.

Additionally, I recommend the following improvements:

• Use a more descriptive variable name for the loaded JSON data instead of f, which is typically used for the file handle.
• Avoid bare except: clauses, as they can hide unexpected errors. It's safer to catch specific exceptions like IOError and json.JSONDecodeError.
• The pass statement on line 895 is unnecessary and can be removed.

        old_chat_template = getattr(tokenizer, "chat_template", None)
        try:
            with open(tokenizer_config_path, "r", encoding="utf-8") as config_file:
                config = json.load(config_file)

            original_config = config.copy()

            # Add chat_template if it exists on tokenizer and not in config
            if old_chat_template is not None:
                if config.get("chat_template") is None:
                    config["chat_template"] = old_chat_template
            
            # Remove chat_template if it's empty or None
            if "chat_template" in config and not config.get("chat_template"):
                del config["chat_template"]

            # Only write back if changes were made
            if config != original_config:
                with open(tokenizer_config_path, "w", encoding="utf-8") as config_file:
                    json.dump(config, config_file, indent=2, ensure_ascii=False)
        except (IOError, json.JSONDecodeError):
            # It's okay if the file doesn't exist or is invalid JSON, we just skip.
            pass