fix wildcard file encoding #1150
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request modifies order of assumed encoding when opening wildcard files.
Current logic is opening with ISO-8859-1 then, when error happens, falls back to UTF-8.
In this logic, when there are unicode characters in the wildcard file, it breaks unicode character to Latin-1, breaking unicode characters. When there are lora file with unicode character(for example, 漢字.safetensors), then try to load it, (i. e., lora:漢字:1) braking file names thus fails to load lora.
This change fixes this issue.
Below is summary by copilot.
This pull request updates the file encoding handling logic for reading wildcard
.txtand.yamlfiles inmodules/impact/wildcards.py. The main improvement is standardizing the default file encoding toUTF-8for initial reads and switching toISO-8859-1only if a Unicode-related error occurs. This should improve compatibility and reduce issues with file reading across different environments.File encoding handling improvements:
UTF-8when opening.txtand.yamlwildcard files, and now only fall back toISO-8859-1if ayaml.reader.ReaderErrororUnicodeDecodeErroris encountered. (load_txt_wildcard,load_yaml_wildcard,read_wildcard_dict) [1] [2] [3] [4]yaml.reader.ReaderErrorandUnicodeDecodeErrorwhen attempting to read files. [1] [2]