GS: Improve degenerate primitive testing in vertex kick and SW bounding box calculation. Other minor fixes.

[TJnotJT]

Draft until fully tested.

Description of Changes

1. Vertex kick: Ensure line/point rounding and AA are accounted for in the degenerate primitive test.
2. SW: Apply the same degenerate primitive test fix as vertex kick.
3. Simplify primitive bounding box and scissor calculations in vertex kick.
4. SW: Minor optimization for horizontal lines in the rasterizer.
5. SW: Adjust an assertion to prevent false positives.

Rationale behind Changes

Some edge cases cause the degenerate primitive test to incorrectly pass or fail primitives, especially when antialiasing is enabled. This can result in issues such as #8677. These changes should fix that issue and address additional minor problems discussed in #13297.

Suggested Testing Steps

This update will affect any GS drawing, so testing games with both HW/SW under different options will be helpful. This should not affect performance much, so only accuracy needs to be checked.

Did you use AI to help find, test, or implement this issue or feature?

Copilot occasionaly for code completion.

[bigol83]

God of War and God of War 2 fixed

🎉

[James-F2]

WOW, what an improvement!
Thank you!

[James-F2]

Possibly the outline of the character here too:

DBZ BT 3.zip

[refractionpcsx2]

Possibly the outline of the character here too:

DBZ BT 3.zip

That's pre rounding solved by the preround pr (when it's finally done right)

[HansKristian-Work]

Someone pointed me at this issue since parallel-gs had similar issues. I fixed a few things on my end, and I think it demonstrates that it should be possible to get GPU accelerated rendering with this new behavior. The explanation in #13297 is quite nice.

1x:

16x SSAA -> downsample to 2x native:

16x SSAA -> downsample to 1x native:

I implemented this with feathered rasterization. For upsampling this means wide line raster which is not really possible to make perfect I think. I assume HW exact behavior requires very specific interpretation of DDA + bresenham algorithm which isn't feasible to do on a GPU I suppose.

[TJnotJT]

@HansKristian-Work Glad the info was useful, and nice improvements! I think a similar approach could work for GS-accurate AA on lines and edges at native resolution, though I haven’t experimented yet.

[TJnotJT]

Small update: SW mode is triggering assertions on a dump run so this remains as a draft for now.

[James-F2]

Testing this fix I see it does not apply during transitions or fade in/out.
For example when the controller screen in GoW1/2 fades in/out when I enter/exit it or the X/Triangle button symbols in the menu when they fade in/out.
I'm not entirely sure how it should look on the real PS2 because my consoles are in storage.

[TJnotJT]

@James-F2 Would it be possible to capture a multiframe dump of this issue? Thanks in advance.

[James-F2]

GoW1 during transition in controller screen.
I do not know what the real PS2 does here, some frames are affected and some not.

God of War_SCUS-97399_20251109181201.zip

[TJnotJT]

GoW1 during transition in controller screen. I do not know what the real PS2 does here, some frames are affected and some not.

God of War_SCUS-97399_20251109181201.zip

Thanks a bunch - I'll try to get to this soon after debugging some other issues.

[TJnotJT]

@James-F2 I dumped some frames on a PS2. Other than some issues with the font/background (perhaps unrelated issues with the PS2 testing application), the controller lines appear similar. This appears to be due to AA1 not being used during the transition (frames 1, 2) and then being used on frame 3:

PS2 frame 1

PR SW frame 1

PS2 frame 2

PR SW frame 2

PS2 frame 3

PR SW frame 3

[Jakey757]

I noticed similar behavior before #4355 FFX also stops using AA1 for the minimap icons whenever it is hidden or reappears (changing locations, touching a save sphere). Not sure If i even took those dumps correctly, and the Twitch vods are gone but the last Youtube link shows PS2 footage. Back then we had a toggle for AA1 which was somewhat handy for debugging when the implementation wasn't quite as good.