Support Hardware Image Wrap Modes in Shader Code

[kwokcb]

Proposal

This is spawned from the discussion in this PR.

The premise is to move logic to compute wrap modes for hardware shaders (variants on GLSL) into shader code.
Unknown if this is applicable to Slang -- requires input so TBD.

This would make it so that

• there is no reliance on pipeline state changes which may not be supported for specific targets.
• shader recompiles can be avoided
• logic is consistent for all affected targets

Implementation

Any implementation should be checked for performance and especially avoid any conditional branching.

Here is an example implementation for color3 with one approach of specialization per backend, though a generalized form could also be used
to avoid macro usage.

// Use preprocessor for API differences
void mx_image_color3(...)
{
    float2 uv = mx_transform_uv(texcoord, uv_scale, uv_offset);
    
    // --- UV Coordinate Transformation ---
    float2 uv_frac = fract(uv);
    float2 uv_mirror = 1.0 - abs(2.0 * uv_frac - 1.0);
    
    int2 modes = int2(uaddressmode, vaddressmode);
    
    // --- Mode Masks (Branchless) ---
    #ifdef METAL
    // Metal: Direct integer comparison
    float2 repeat_mask = float2(modes == 1);
    float2 mirror_mask = float2(modes == 2);
    float2 constant_mask = float2(modes == 0);
    #else
    // GLSL/ESSL: Use mix/step for branchless
    float2 repeat_mask = 1.0 - step(1.5, float(modes)) + step(0.5, float(modes));
    float2 mirror_mask = step(1.5, float(modes));
    float2 constant_mask = 1.0 - step(0.5, float(modes));
    #endif
    
    // --- Apply Addressing Modes ---
    float2 uv_final = uv * constant_mask +
                      uv_frac * repeat_mask +
                      uv_mirror * mirror_mask;
    
    // --- Out-of-Bounds Check ---
    float2 out_of_bounds = step(1.0, float2(uv < 0.0) + float2(uv > 1.0));
    float use_default = dot(constant_mask, out_of_bounds);
    
    // --- Texture Sampling ---
    #if defined(GL_ES) || defined(ESSL)
    // OpenGL ES: May need texture2D for older versions
    vec3 sampled = texture2D(u_sampler, uv_final).rgb;
    #else
    // Desktop GL and Metal
    vec3 sampled = texture(u_sampler, uv_final).rgb;
    #endif
    
    // --- Final Selection (API-specific optimal) ---
    #ifdef METAL
    // Metal: select() for bool, mix() for float
    result = mix(sampled, defaultval, float(use_default > 0.0));
    #else
    // GLSL/ESSL: mix() is universally optimal
    result = mix(sampled, defaultval, min(1.0, use_default));
    #endif
}

[jstone-lucasfilm]

This is an interesting proposal, @kwokcb, though I think we should discuss our options for texture address modes in hardware shading before committing to a pure ALU-based approach.

To get this discussion started, I'll highlight a question from the earlier thread in #2706:

I'd be interested in a deeper understanding of the differences in support for texture address modes across GLSL, HLSL, MSL, WGSL, and other hardware shading languages.

Is there a broad pattern of divergence across hardware shading languages, or is MSL simply missing one of the modes that GLSL, HLSL, and WGSL all support in a consistent way?