RE: The setting to change the Language of the game to Any language

[JackTriton]

Hi, it's been a long time
This PR is redo of all of the functions I made almost 5 years ago

Description

With this PR, you can play OOTR with different languages or event texts / custom texts as well

---

Addition

Files

• language (folder): Consists every language you can play with
  Also, you can add another language file just by drag and drop the file with same structure
  For how to create and what file is included, checkout language / README.md
• Language.py: allows to manage language files (json, bin, ia4)

---

Additional Control Codes for wide characters

• All of the additional control codes are placed the code + 0x8700 (for example, silver rupee count is placed 0xF0 + 0x8700 = 0x87F0)
  For the character to use, check out Messages.py

---

Text alignment

• You can change the alignment of the text to Left, Center and Right

---

How to Change the language?

You can change the language by changing Main Rules >> Language >> Language Selection

---

Tips for language creation

If you want to add language with characters not included in vanilla game (for example Spanish has ñ), do the same thing as other images: decompile the original game, replace some files with new one and recompile / get diff with get_diff.py

---

Notes

Currently, besides English which is already implemented from beginning, it allows you to play with Japanese as well
Why Japanese? well, it's because Japanese is the other language that NTSC rom has and one of language that uses wide characters instead and also because I'm Japanese

Just because I didn't implemented other languages such as German, French or Chinese (which is PAL and iQue version has), that doesn't mean you can't play with those languages or other languages that only got the translated version by using mod / patch or even that never got the version

Not only that, you can create version for events, competition or even speedruns as well (like shortening all of the texts, and speed up everything)

Screen shots

[fenhl]

Preliminary review. I've already mentioned some of this on Discord but am repeating it here for reference.

• The hint area type being language-dependent is going to be an issue for parts of the codebase that don't directly interact with text and therefore don't care about the language. Currently, if I understand the code correctly, these just use an English-language version of the hint area type. This will cause issues if such a hint area value does end up in language-specific code and causes text to be generated in English instead of the correct language. I'd separate the language information from the hint area by passing it as a parameter to methods that generate text.
• Some of the variable names and magic strings (e.g. magic strings called o, wiw, and cbf; single-letter variable names in Language.format_from_text and Patches.create_fake_name) are difficult to understand at a first glance. I would recommend replacing them with longer, more descriptive names.
• There seem to be some pieces of logic that check if the language is English and use the logic for Japanese in the else case. The level of Japanese proficiency among maintainers and contributors is much lower than that of English, so for maintainability, English should be the fallback case.
• The PR adds .DS_Store files to the repo. These should be removed, and I recommend adding them to your .git/info/exclude to prevent accidentally adding them back in the future.
• The JSON file uses fixed-length (tuple-like) arrays in some places. To make it easier to read, I suggest replacing these with objects ({}) with descriptive property names.

I will take a closer look at specific parts of the code and go into more detail in a later review, once CI is green.

[JackTriton]

Most of the system reworked, deleted .DS_Store, add some descriptions to the property_build.py and changed some of the values to have clearer infos

[JackTriton]

Sorry there was some typos in japanese/property.json

[cjohnson57]

Some pretty major (but easily fixable) issues with the setting definition itself, adding this separately before I review the rest of it.

[cjohnson57]

Got different errors when trying to generate in English and Japanese.

English:

<Dungeon.Dungeon object at 0x000001EEE96AD480>
Traceback (most recent call last):
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\OoTRandomizer.py", line 57, in start
    main(settings)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 57, in main
    patch_and_output(settings, spoiler, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 375, in patch_and_output
    patch_cosmetics_log = prepare_rom(spoiler, world, rom, settings, rng_state, restore_rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 214, in prepare_rom
    patch_rom(spoiler, world, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Patches.py", line 1849, in patch_rom
    "dungeon_reward": world.language.hintTable[dungeon][1],
KeyError: <Dungeon.Dungeon object at 0x000001EEE96AD480>

Seems to be because the dungeon name has a control code in it. Also, pretty sure the the should be dropped and it should just be Deku Tree...

Japanese:

from_bytes() missing required argument 'byteorder' (pos 2)
Traceback (most recent call last):
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\OoTRandomizer.py", line 57, in start
    main(settings)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 57, in main
    patch_and_output(settings, spoiler, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 375, in patch_and_output
    patch_cosmetics_log = prepare_rom(spoiler, world, rom, settings, rng_state, restore_rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 214, in prepare_rom
    patch_rom(spoiler, world, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Patches.py", line 1176, in patch_rom
    update_message_by_id(messages, 0x305C, lang.format_from_id("PATCH_TEXTS.claim"), lang)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Messages.py", line 748, in update_message_by_id
    text = line_wrap(text, lang.base, align=lang.lang_property["align_text"])
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\TextBox.py", line 51, in line_wrap
    text_codes = Messages.parse_control_codes(text, lang)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Messages.py", line 345, in parse_control_codes
    text_bytes = encode_text_string(text) if lang else encode_text_string_jp(text)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Messages.py", line 295, in encode_text_string_jp
    result.append(int.from_bytes(h.encode("cp932")))
TypeError: from_bytes() missing required argument 'byteorder' (pos 2)

Adding byteorder="big" to the function call fixed it, but I'm not actually sure why that was necessary when the function definition already sets it to big by default? result.append(int.from_bytes(h.encode("cp932"), byteorder="big"))

def from_bytes(
            cls,
            bytes: Iterable[SupportsIndex] | SupportsBytes | ReadableBuffer,
            byteorder: Literal["little", "big"] = "big",
            *,
            signed: bool = False,
        )

@flagrama any ideas here?

Anyway, fixing that issue ran into the same issue English did with the dungeon name.

[JackTriton]

Got different errors when trying to generate in English and Japanese.

English:

<Dungeon.Dungeon object at 0x000001EEE96AD480>
Traceback (most recent call last):
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\OoTRandomizer.py", line 57, in start
    main(settings)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 57, in main
    patch_and_output(settings, spoiler, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 375, in patch_and_output
    patch_cosmetics_log = prepare_rom(spoiler, world, rom, settings, rng_state, restore_rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 214, in prepare_rom
    patch_rom(spoiler, world, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Patches.py", line 1849, in patch_rom
    "dungeon_reward": world.language.hintTable[dungeon][1],
KeyError: <Dungeon.Dungeon object at 0x000001EEE96AD480>

LangDev here, this is probably because of the given value is dungeon instead of dungeon_reward

For the requested changes, moving the language_selection in SettingsList.py to below logic_rules would be great?

[cjohnson57]

Here's my review of the rest of the code. Admittedly mostly nitpicks, but a few actual issues.

Thanks so much for all your work on this!

[cjohnson57]

Thanks for your quick fixes on my first two messages! Can confirm they're resolved. I could launch the ROM and talk to NPCs and read hints in both English and Japanese.

[JackTriton]

Thanks for your quick fixes on my first two messages! Can confirm they're resolved. I could launch the ROM and talk to NPCs and read hints in both English and Japanese.

Regarding the last request, I'll edit them right now!

[cjohnson57]

One question, what exactly is the purpose of the property_build.py file? It seems to mainly contain the English messages that are now formatted in the English folder's property.json. Is it to generate the property.json?

[flagrama]

Adding byteorder="big" to the function call fixed it, but I'm not actually sure why that was necessary when the function definition already sets it to big by default? result.append(int.from_bytes(h.encode("cp932"), byteorder="big"))

FYI, the default value is just a change made in python 3.11. If you are using 3.11 or newer, this error won't appear. If you are using anything older this error comes up. If you are using 3.11 or newer but you have your dev environment set up to ensure everything works as far back as 3.8 it may be doing something additional to cause that error to show up.

[cjohnson57]

Interesting, ty for the info!

[JackTriton]

One question, what exactly is the purpose of the property_build.py file? It seems to mainly contain the English messages that are now formatted in the English folder's property.json. Is it to generate the property.json?

property_build.py creates property.json for the language
The default value would be the ones on English for compare / know what string to use

Also, the PLANE_TEXTS would be the ones for other texts to implement in game when the language is not on NTSC or any kind of variety

[cjohnson57]

Gotcha. Should that be "plain" texts?

[JackTriton]

Gotcha. Should that be "plain" texts?

Oh, sorry yes PLAIN_TEXTS

[cjohnson57]

Thanks so much for all your changes! It's good now as far as I can tell.

Before merging, we should also get approval from Fenhl, and probably more testing of full JP seeds.

[JackTriton]

@cjohnson57 Thanks for the detailed reviews!

[JackTriton]

Add UnitTest that checks language file / properties

Below this is template for the errors

If the language file misses property.json
{lang_name}: property.json is not included

If some properties are missing / wrong in property.json
{lang_name}: some properties are wrong in property.json

If some bin files are not included in the bin_patch.json
{lang_name}: some non-wanted bin files are included

[JackTriton]

Unless the conflicts revolve with in-game messages or custom texts, all conflicts will be resolved after merging

[JackTriton]

Reopen this with newer commits
Sorry for the extra request (#2506)

[JackTriton]

@cjohnson57 @fenhl I think it's ready for the review

[cjohnson57]

Thank you. Once fenhl gets back and does their review, I'll also go through it again, then we'll try to do some testing and then merge.

[JackTriton]

Currently, the branch conflicts revolving around ASM and generated data
The ASM part doesn't conflict with this branch so I think Pull request + rebuild would resolve the issue

[cjohnson57]

Currently, the branch conflicts revolving around ASM and generated data The ASM part doesn't conflict with this branch so I think Pull request + rebuild would resolve the issue

Yes, those are nothing to worry about.

[cjohnson57]

FYI, fenhl is preparing a PR to your branch for some hint region cleanup. In the meantime I will try to do a test seed with JP full randotext next week to see if anything breaks.

[cjohnson57]

FYI, tried to generate a seed on this branch using Japanese and got this error. The error did not occur when selecting English.

list index out of range
Traceback (most recent call last):
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\OoTRandomizer.py", line 57, in start
    main(settings)
    ~~~~^^^^^^^^^^
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 57, in main
    patch_and_output(settings, spoiler, rom)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 372, in patch_and_output
    patch_cosmetics_log = prepare_rom(spoiler, world, rom, settings, rng_state, restore_rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 214, in prepare_rom
    patch_rom(spoiler, world, rom)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Patches.py", line 1237, in patch_rom
    write_gossip_stone_hints(spoiler, world, messages)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Hints.py", line 327, in write_gossip_stone_hints
    update_message_by_id(messages, id, str(gossip_text), lang, 0x23)
                                       ~~~^^^^^^^^^^^^^
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Hints.py", line 106, in __str__
    return get_raw_text(color_text(self, self.lang), self.lang.base)
                        ~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Hints.py", line 364, in color_text
    split_text[2] = "00" + split_text[2]
                           ~~~~~~~~~~^^^
IndexError: list index out of range

Using this settings string: BSAWDNCAX2TBJYYGGAFSL62ANAWGEAAAAACAASKEASHAANLZAAADAKAEAEBABBXCAAABNAGJD8ASEJQ4VNFAGSCWEV43CAAAAAAAG2HPJNB2DBSSBUDWEARAAADACEJGUBC

Changing the randotext setting did not help, but setting gossip stones to "no hints" did. Clearer hints or not did not affect it. Most hint distros did not help, however, setting it to "Useless" did fix the issue, which makes sense. Strangely, the error also did not occur with hints set to "Very Strong" or "Very Strong with Magic", but it did when set to "Strong", which would probably make sense if I had more intimate knowledge of the different hint distros.

Hopefully, the PR fenhl is preparing to address hints will resolve this issue.

[cjohnson57]

OoT_A0FAE_GJAJIMJB4W_Spoiler.json

I started this seed, playing on PJ64, and got an immediate freeze when talking to the nearest NPC (Ingo)

Talking to other NPCs and reading signs isn't crashing, so I guess it's something specific to Ingo's textbox. Unfortunately I'm not sure how to check what randotext was assigned to him.

[cjohnson57]

I messed around on 5 different seeds and here are my findings.

Within a specific seed, certain NPCs will always freeze the game if you talk to them shortly after a hard reset. The NPCs that freeze being consistent indicates that it is due to the textboxes shuffled onto them. I tried with randotext off and got no freezes, though that was only from talking to NPCs, I didn't test all the other things that spawn textboxes outside of a few item pickups. It seems like on average maybe 1 in 5-10 textboxes freezes the game.

Strangely, sometimes, if you talk to them after screwing around a bit it won't freeze. Not sure if that's a PJ64 issue or the game itself.

Some others freezes

I also got some textboxes that didn't outright freeze the game, but softlocked me because I was unable to advance them. These were consistent.

[JackTriton]

I messed around on 5 different seeds and here are my findings.

Ok, this looks like the ones got shuffled didn't get the control code that was supposed to be attached (the "00" + glitch also comes from the same instance)

I'll use direct text output to see what caused those

[JackTriton]

@cjohnson57 @fenhl As far as I tested, the error was caused by 2 factors:

1. Some texts like path of #time# which is implemented in World.py wasn't translated >> Added goal_place_textboxes in the property
2. In Japanese codec, uninstant code (0x09, 0x8199) is needed before Goto ending

[JackTriton]

Sorry there was another glitch revolving color_text

I'll fix it right away

[JackTriton]

@cjohnson57 @fenhl Fixed, changing ##s into # according to the usage of the color is created successfully as far as I tested

[cjohnson57]

Thanks, it seems to be working much better now. The hint patching is working and I didn't get any freezes immediately when talking to someone.

There's still the case where it doesn't let me advance the textbox, I'd guess this one's a more complicated fix...

And then I found this case where it lets me click through the first 3 textboxes, but then the 4th one freezes for some reason:

Also a couple textboxes that are weird but didn't cause any technical issues. I actually saw the 060B one twice, on two different seeds.

[cjohnson57]

Btw, there's some code from @GSKirox that should be helpful for testing this PR, that adds a display on the screen for the current textbox ID. He's going to include it in a later PR that expands debug mode, but for now we can add it, then maybe just undo that commit before merging...

If you add the asm hook and C function for message_id_check, as well as the change in ootSymbols.ld, then it should work:
https://github.com/OoTRandomizer/OoT-Randomizer/pull/2493/files#diff-ec33c7eb10e1082a2d581ea448af77b53ce9d429f7aba1b5ae2ec98af4b430e3

[JackTriton]

@cjohnson57

And then I found this case where it lets me click through the first 3 textboxes, but then the 4th one freezes for some reason

Could you give me the setting seed / generation seed?
I'll look it up

Also a couple textboxes that are weird but didn't cause any technical issues. I actually saw the 060B one twice, on two different seeds.

I searched through the file and found out that in JP rom, 060B and 016F has control codes as well
I'll edit the is_id_message to ignore the control codes and see what'll happen

[cjohnson57]

Could you give me the setting seed / generation seed?
I'll look it up

OoT_F2ADE_GGUKU4UH3G_Spoiler.json

[JackTriton]

@cjohnson57 All of the presented issues are fixed as far as I know