GS/VK/GL/DX12/DX11: Improved accuracy for lines and AA1 support for lines/triangles.

[TJnotJT]

Status: Draft until dump runs are done.

TODO

• Add setting to GUI under Graphics>Advanced
• DX11/12: Avoid copying depth if depth is not written.

Description of Changes

Add an INI setting for accurate lines and AA1 support (lines and triangles). This is a port of functionality currently in SW.

Details

1. Accurate lines are implemented by drawing an oriented quad covering the necessary pixels. The pixel shader discards pixels that are not parts of the line (depending on AA1 setting), and manually interpolates vertex attributes.
2. Triangle edges with AA1 are implemented in a similar way. The interiors of triangles are rasterized in HW as usual, so sometimes edge pixels might not touch the interior correctly.
3. In both cases the primitive data for doing this is uploaded to a SSBO (or equivalent) for the pixel shader.
4. Support was added for SSBO ring buffers with default heap memory. The staging buffer for textures was reused to stage the data (VK/DX12 only).

Caveats

1. Performance note: because attributes are manually interpolated in the pixel shader, the Z test and Z write are implemented in the pixel shader. Performance will be likely significantly reduced for draws with Z testing.
2. Performance note: although default heaps are used for the SSBO, I'm not sure if this is necessarily better than just using an upload heap. The default heap commit is separate, so performance testing without it would be helpful. Edit: confirmed by Jordan that using default heaps can significant improve performance on some AA1 heavy dumps.
3. Performance note: the DX shader compiler complain about integer division in the pixel shader begin slow. I'm not sure if it's better to convert to float, do the division, then convert back to integer.
4. Currently does not support upscaling though this can be done in a follow up.
5. Currently not implemented for Metal though this can be done in a follow up.

Rationale behind Changes

Accuracy improvement for certain games that require AA1 or accurate lines.

Suggested Testing Steps

Please put the following in the INI:

[EmuCore/GS]
...
HWAccuratePrims = 1

Then use any of the following renderers: Vulkan, OpenGL, DX12, DX11. Performance testing for games known to use heavy line drawing and/or AA1 would be helpful. Testing without the 2nd commit would be helpful (see performance note 2 above).

Did you use AI to help find, test, or implement this issue or feature?

Yes, I use AI to reference graphics API info and review code.

Comparisons

TimeSplitters - Future Perfect_SLES-52993_20250703155125.gs.xz

Master VK

PR VK

God_of_War_NTSC-U_MenuController.gs.xz

Master VK

PR VK

[TJnotJT]

Rebased - a couple small merge conflicts and fixes.

[TJnotJT]

Last few pushes to fix a constant buffers on Metal and some small fixes for building on Linux.

[TellowKrinkle]

AA1 on PS2 is just "add a 1-pixel-wide border around every triangle with COV going from 1 at the edge of the original triangle to 0 at the new edge" right?

I feel like this would best be handled by the vertex shader, rather than CPU and pixel shader, similar to the current VS expand code. This would avoid needing to send piles of AccuratePrimsEdgeData to the shader, and would support upscaling easily (which is kind of the point of HW renderer, otherwise just use SW).

For lines, you could use the existing line code but just make the lines wider and have a coverage value going from 0 to 2 across the line, which the fragment shader could apply cov > 1 ? 1 - cov : cov to get the needed values without needing extra triangles.

For triangles, bind index buffer as a VS resource rather than as hardware index buffer, then have vs expand each triangle into the original triangle plus 9 points making up a quadrilateral against each side (the inner vertices can be shared). I suspect it would look something like this, but not sure how the PS2 handles the corners (it might be worth filling in the corners even if the PS2 doesn't, as I suspect it might be more noticeable with upscaling):

(This means moving the interpolation code to the vertex shader as well, since those outer vertices would need interpolated values.)

The only change to the pixel shader would be handling picking coverage vs alpha, clamping of e.g. color values that can now go out of uint8 range, and discarding depth.

Also, since you have to handle discarding depth independently from color for this (which includes reading depth, sw depth testing, etc), could you pull that out into a separate PR and add an option to use that in place of two-pass alpha test? That should make this PR a bit smaller, and that's something we should have anyways.

[TJnotJT]

I feel like this would best be handled by the vertex shader, rather than CPU and pixel shader, similar to the current VS expand code. This would avoid needing to send piles of AccuratePrimsEdgeData to the shader, and would support upscaling easily (which is kind of the point of HW renderer, otherwise just use SW).

This could be done. My approach here was to micmick SW as closely as possible, but I agree that may go against the philosophy of HW.

For lines, you could use the existing line code but just make the lines wider and have a coverage value going from 0 to 2 across the line, which the fragment shader could apply cov > 1 ? 1 - cov : cov to get the needed values without needing extra triangles.

Yes, though I think with one small difference: the line and coverage should be expanded in the x/y direction depend on whether thay are shallow/steep (I believe the wide line shader currently expands in the perpendicular direction). Same for the triangle edges.

For triangles, bind index buffer as a VS resource rather than as hardware index buffer, then have vs expand each triangle into the original triangle plus 9 points making up a quadrilateral against each side (the inner vertices can be shared). I suspect it would look something like this, but not sure how the PS2 handles the corners (it might be worth filling in the corners even if the PS2 doesn't, as I suspect it might be more noticeable with upscaling):

Seems like a reasonable approach. According to tests the corners end at the adjacent edge (rather than axis-aligned as pictured above). We could try without gaps and see if it looks better.

The only change to the pixel shader would be handling picking coverage vs alpha, clamping of e.g. color values that can now go out of uint8 range, and discarding depth.

Also, since you have to handle discarding depth independently from color for this (which includes reading depth, sw depth testing, etc), could you pull that out into a separate PR and add an option to use that in place of two-pass alpha test? That should make this PR a bit smaller, and that's something we should have anyways.

Sure, I can do the shader depth testing first in that case.

The VS expand approach outlined above will require a separate PR. I'll keep this one around as a reference until that is completed.