GS/HW: Refactor texture shuffle detection and quad conversion.

[TJnotJT]

Description of Changes

Refactor/cleanup texture shuffle detection and quad conversion:

• Contain texture shuffle heuristics in two functions (described below).
• Convert texture shuffle draws to a single quad.
• Add internal state to track the type of texture shuffle.

Committed on top on #14240, though they could perhaps be considered a single PR instead.

Rationale behind Changes

• Make code more maintainable.
• Make heuristics/hacks more explicit.
• Keep code/state related to texture shuffles together.

Details

Adds the function DetectTextureShuffle(), which does initial detection based on context/vertices. Classifies texture shuffles into the following subclasses:

• Copy: No offsetting between X and U so results in channel copies.
• Offset: Offsetting between X and U so results in channel swaps.
• RegionRepeat8: Uses region repeat to set/clear bit 3 to do channel swaps.
• RegionRepeat16: Uses region repeats to set/clear bit 4 to do channel swaps.
• Reverse: Reverses X and U order to do channel swaps.
• Swizzle: Swizzles at the same time as shuffling.
• SwizzleTex32: Swizzles at the same time as shuffling with a 32 bit source texture. Only known to be used for B -> A shuffles in e.g. The Godfather and 007 From Russia With Love.
• TwoPixel: Two pixel shuffle only seen in Powerdrome.
• GappedSwizzle: Swizzle shuffle with strange gaps between quads and base address offsetting over multiple draws. Only seen in NFS Undercover and skipped with a CRC hack.
• HackShuffle: We use the texture shuffle path in certain cases where the source/RT are 32 bit being reinterpreted as 16 bit, although it's not really a shuffle. It appears to be used to preserve the correct size of the RT/DS.

Adds function DetectTextureShuffleSecondPass() that does the second pass detection after RT and source have been looked up.

Changes ConvertSpriteTextureShuffle() to output a single quad per draw, since this is sufficient to emulate texture shuffles.

Adds a function FixSplitTextureShuffleState() that contains the code to continue a split texture shuffle. Mainly a cut/paste of the current code from GSRenderer::Draw().

Suggested Testing Steps

Testing any game with HW renderers would be helpful.

So far tested with dumps runs. The following dumps show up as false positives:

• Need_for_Speed_-_Undercover_SLUS-21801_20231222095536.gs.xz: tiny difference in color in the borders of some shapes. Happens because master incorrectly samples U = [0, 648] in a texture shuffle while PR correctly samples U = [0, 640].
• Raw Danger_20181103133127.gs.xz, rawdanger.gs.xz, greenglow.gs.xz, Blockus_depth2.gs.xz, pachipara13.gs.xz, steambot_screffhue.gs.xz, Blokus_depth1.gs.xz: Broken shuffle on both PR/master and may need a separate fix. Some pixels in the top left corner are different.

Did you use AI to help find, test, or implement this issue or feature?

No.

[lightningterror]

Needs rebase.

[TJnotJT]

Rebased

[Mrlinkwii]

smoke test was fine on linux on HW renderers

[lightningterror]

Dump run clear, differences show as mentioned.