[rlsw] Fix clipping boundary overflow

[Bigfoot71]

Fixes the buffer overrun issue reported here: #5326 (comment)

After investigating the problem, the most reasonable approach seems to be using exclusive tests for the X/Y planes during clipping while keeping inclusive tests for Z.

The current clipping implementation is based on multiple rewrites of the approach originally described in this blog:
https://fabiensanglard.net/polygon_codec/

Our original implementation followed the same conditional logic as in the blog post, but this behavior might also be related to the projection [NDC -> screen] convention we follow (and which didn't seem to be a problem anyway in the context of the blog post), so clipping with exclusive X/Y borders seems more consistent in our case, while keeping Z inclusive avoids losing fragments.

For reference, here are the original papers that describes the logic:

• https://fabiensanglard.net/polygon_codec/clippingdocument/p245-blinn.pdf
• https://fabiensanglard.net/polygon_codec/clippingdocument/Clipping.pdf

[raysan5]

@Bigfoot71 thanks for the review and the detailed explanation and references!