Shader: multiplying a sprite by another sprite that is somewhere in screen space.

[rudiHammad]

Hello,
I am trying to make the following: I have spriteA and a spriteB. I want that when spriteA is on top of spriteB in screen space, the pixels of spriteA are multiplied by the pixels of spriteB that are overlaying. I have been trying this for a couple of day now without success. I did manage to do the following test.

2025-01-03.16-27-16.mp4

So as you can see, I am setting correctly the pixel in red when their position in screen space is bigger than half of the screen.
Here is the relevant part of the code:

bool ShadersSceneAdvanced::init()
{
    //...
    spriteA = Sprite::create("misfitKnight.png");
    spriteA->getTexture()->setAliasTexParameters();
    spriteA->setAnchorPoint(Vec2(0, 0));
    addChild(spriteA, 1);


    auto moveAction = MoveBy::create(5, Vec2(640, 0));
    auto sequenceAction = Sequence::create(moveAction, moveAction->reverse(), nullptr);
    auto repeatAction = RepeatForever::create(sequenceAction);
    spriteA->runAction(repeatAction);

    // Shader
    auto flashProgram = ProgramManager::getInstance()->loadProgram(positionTextureColor_vert, "custom/ch_learning_fs", VertexLayoutType::Sprite);
    flashProgramState = new ProgramState(flashProgram);
    spriteA->setProgramState(flashProgramState, true);

    // Set uniforms
    auto u_screenSize = flashProgramState->getUniformLocation("u_screenSize");
    spriteA->getProgramState()->setUniform(u_screenSize, &visibleSize, sizeof(visibleSize));

    spriteASize = spriteA->getContentSize();
    auto u_spriteSize = flashProgramState->getUniformLocation("u_spriteASize");
    spriteA->getProgramState()->setUniform(u_spriteSize, &spriteASize, sizeof(spriteASize));

    
    scheduleUpdate();
    
    return true;
}


void ShadersSceneAdvanced::update(float delta)
{
    
    u_spriteAPos = flashProgramState->getUniformLocation("u_spriteAPos");
    spritePos = spriteA->getPosition();
    spriteA->getProgramState()->setUniform(u_spriteAPos, &spritePos, sizeof(spritePos));
    spriteA->setProgramState(flashProgramState, true);
    
}

and this is the fragment shader:

#version 310 es
precision highp float;
precision highp int;

layout(location = SV_Target0) out vec4 FragColor;
layout(location = TEXCOORD0) in vec2 v_texCoord; 
layout(binding = 0) uniform sampler2D u_tex0;    
//layout(binding = 1) uniform sampler2D u_tex1;    


layout(std140) uniform u_block 
{ 
    uniform vec2 u_screenSize;
    uniform vec2 u_spriteAPos;
    uniform vec2 u_spriteASize;
};


void main()
{
    vec4 spriteATxl = texture(u_tex0, v_texCoord);
    //vec4 spriteBTxl = texture(u_tex1, v_texCoord);

    // Calculate the screen position of the current pixel
    vec2 spriteAScreenPos = u_spriteAPos + v_texCoord * u_spriteASize;      
    //vec2 spriteBScreenPos = u_spriteBPos + v_texCoord * u_spriteBSize;

    if (spriteAScreenPos.x  > u_screenSize.x /2.0 )      
    {
        vec4 newTxl = vec4(1.0 * spriteATxl.a, spriteATxl.g * spriteATxl.a, spriteATxl.b * spriteATxl.a, spriteATxl.a);
        FragColor =  newTxl;
    }
    else
    {
        vec4 newTxl = vec4(spriteATxl.r * spriteATxl.a, spriteATxl.g * spriteATxl.a, spriteATxl.b * spriteATxl.a, spriteATxl.a);
        FragColor =  newTxl;
    }

}

But here's where I got stuck. I have commented the lines that are supposed to take the second sprite and operate on it.
What I think I should do is mutiply spriteATxl by the spriteBTxl that is in the same position, but all my attempts didn't work.
How can I check for the screen position of the same pixel on both sprites to be able to multiply them?
thanks

[DelinWorks]

You actually don't need a shader to perform this operation because OpenGL already does this for you, it's called blendfunc (Blend Function)

you have to first make your spriteA above spriteB so that OpenGL can do color maths on the pixels behind. you can change the behaviour of blending function for a sprite using

ax::BlendFunc f;
f.src = ax::backend::BlendFactor::SRC_COLOR;
f.dst = ax::backend::BlendFactor::DST_COLOR;
sprite->setBlendFunc(f);

There are plenty of options that you should try out on both spriteA and spriteB to get your desired blending function.

Here's a table of how these function operate on a per fragment (per pixel) basis

https://registry.khronos.org/OpenGL-Refpages/gl4/html/glBlendFunc.xhtml

[rudiHammad]

oh, I am aware of that thanks. Probably the post title wasn't clear.
I want to actually know how to do it using a custom shader to understand how it works and potentially find other uses to it.

[DelinWorks]

I don't think you can draw objects this way, at least without using a custom render texture.

[DelinWorks]

you can use a render texture to render spriteB on, then supply the render texture sampler2D to your spriteA shader and then do magic maths there, you can't apply colors to world space object in shaders you will need some kind of screen space canvas like a render texture

[rudiHammad]

I don't know, maybe you' re right, but I am not sure about having use render texture, I think it is related to how to convert the texels of spriteB into spriteA coords. It could be more complicated that it seems. Anyway, I'll give it a shot. I am still stuck I'll just work around using a clipNode with a stencil.

[DelinWorks]

you might be able to achieve that without render textures, but i don't know how. good luck