GS/HW: Depth feedback loops and accurate AFAIL.

[TJnotJT]

Status

Draft until

1. Dump runs are completed for all renderers: VK, GL, DX12, DX11.
2. Ported to Metal. May need to wait until later.

Description of Changes

1. Add support for depth feedback loops either via texture barriers or copies.
2. DX12 and OpenGL do depth feedback by creating a temporary color texture to hold Z and using multiple color outputs.
3. Use the depth feedback for accurate AFAIL (enabled via INI setting).

Follow-up for separate PRs

1. (Done) In certain cases we don't need full barriers for AFAIL. The cases are detected with flags ate_RGBA_then_Z and ate_RGBA_then_Z in GSRendererHW::DrawPrims() for the current AFAIL system.
2. Add texture barriers to DX12 instead of copies. Nvm, I just found out that DX12 doesn't support this for feedback loops. Implemented by GS/DX12: Use aliasing resources for feedback #13653.

Rationale behind Changes

1. Fixes some effects that are not accurately rendered by the current AFAIL emulation. Also reduces draw calls in some cases.
2. In the future, the depth feedback will also be useful for AA1 emulation (see discussion: GS/VK/GL/DX12/DX11: Improved accuracy for lines and AA1 support for lines/triangles. #13617).

Known issues

Edit: Confirmed to not affect all systems.

Unfortunately, SW depth testing can lead to severe Z fighting in some games (seemingly with drawing decals on existing geomertry), so I've made disabled depth feedback loops by default and only enabled it with maximum blending. Depth sampling, which aleady exist in master, should be unaffected by this. A rationale for keeping this PR is that is adds some common code that is useful for AA1 and ROVs.

Example in DX12: BloodRayne_SLUS-20461_20250423095228.gs.xz

Suggested Testing Steps

Please put the following in the INI:

[EmuCore/GS]
...
DisableDepthFeedback = 0
accurate_blending_unit = 5

Then use one of the renderers: VK, GL, DX12, DX11. Performance testing would be a big help.

Did you use AI to help find, test, or implement this issue or feature?

Yes, to reference graphics API functions/semantics.

Comparisons

There are many dumps with significant difference, but for the most part it is not obvious that they are fixes (i.e., random pixels in foliage). However, there are some clear cases below.

"everybody_golf_shadow.gs.xz" (see loading bar at bottom)

Master VK

PR VK

"Hot Shots Golf FORE!_SCUS-97401_20221204040422.gs.xz" (see character portraits)

Master VK

PR VK

"Minna no Tennis_SCPS-15113_20230922202011.gs.xz"

Master VK

PR VK

[JordanTheToaster]

Barriers on the OSD do not seem to display properly now only showing 0 on Vulkan. The setting itself seems to do nothing for me currently.

[TJnotJT]

Barriers on the OSD do not seem to display properly now only showing 0 on Vulkan. The setting itself seems to do nothing for me currently.

Oops, I put the wrong setting name - it should be HWAFAILFeedback=1. For the OSD issue, could you enable debug device and post the log, please?

[TJnotJT]

Fixed barrier counts in VK and deleted some old commented code.

[TJnotJT]

Added a new commit to refactor alpha test selection so that it is contained in a single function. Some minor changes to shaders, etc.

[TJnotJT]

Latest push rebases and adds a temporary commit that for depth feedback via copy in OpenGL (based on the DX11 method). Based on a a few tests we have reason to believe that texture barriers for a depth attachment bound as a shader resource doesn't always work, so this is for testing until we can determine exactly.

[TJnotJT]

Unfortunately, it turns out that doing SW depth testing can lead to severe Z fighting as there apparently can be ULP differences in the value written to the depth attachment and the value read by the pixel shader (example in OP). Not 100% sure if this a fundamental GPU limitiation or there is a way to over come this, so making depth feedback loops disabled by default (see OP for the INI override). May require testing on different hardware.

[TJnotJT]

We confirmed that the Z fighting issue does not affect all systems, so I'm changing the depth feedback loops setting back to default enabled. Still, it may lead to high barriers/draws so it's only used when barriers are already used in the draw or the blend level is maximum or FB-fetch is available.

[TJnotJT]

Removing draft status as dump runs on VK, DX12, GL, DX11 came clean with settings:

[EmuCore/GS]
...
DisableDepthFeedback = 0
accurate_blending_unit = 5

and VK with settings:

[EmuCore/GS]
...
DisableDepthFeedback = 1
accurate_blending_unit = 1

(The latter since there are refactors to alpha test that affect basic blend.)

[lightningterror]

Needs Metal changes as well.

[TJnotJT]

Needs Metal changes as well.

I've though about doing a best-effort port to Metal, but I think there are some risks.

I don't currently have access to a Metal environment to test or debug. Some of the changes in this PR affect descriptor layouts, so even if the new functionality is disabled, the renderer would still be impacted. If regressions show up later, it would be difficult for me to debug or fix them.

I think it would be safer to keep this PR limited GL/VK/DX11/12 and have someone with the ability to test/debug handle the Metal port.