How to Scale Matrix4 for Offscreen 4K Canvas Rendering?

[Hello1nside]

I'm building a React Native app using react-native-skia for photo collages. The preview uses a small canvas (~700x400px), but for saving, I use an offscreen canvas (4K: 2160x3840px). I'm trying to scale the Matrix4 from the gesture handler with:

const scaleXFactor = offscreenCanvasWidth / appCanvasWidth;
const scaleYFactor = offscreenCanvasHeight / appCanvasHeight;

I apply these factors to matrix translation values like:

transformMatrix[3] * scaleXFactor; // X translation
transformMatrix[7] * scaleYFactor; // Y translation

However, I face two issues:

1. Slight shifts appear when an image covers only 50% of the width or height.
2. If the user uses zoom, X/Y positions have errors.

How should I properly scale Matrix4 to keep image positions consistent between preview and offscreen canvas?

Current implementation for more context:

...
const drawImagesSeparately = (
  offscreenCanvasWidth: number,
  offscreenCanvasHeight: number,
  skImages: (SkImage | null)[],
  photos: Photo[],
  collageIndex: number,
  appCanvasWidth: number,
  appCanvasHeight: number
): { image: SkImage | null; x: number; y: number }[] => {

  const { layout } = CollageManager.getLayout(collageIndex);
  const snapshots: { image: SkImage | null; x: number; y: number }[] = [];

  skImages.forEach((image, index) => {
    if (!image) return;

    const frame = layout[index];
    const frameWidth = frame.width * offscreenCanvasWidth;
    const frameHeight = frame.height * offscreenCanvasHeight;
    const imgWidth = image.width();
    const imgHeight = image.height();

    const transformMatrix = photos[index]?.matrix?.value || Matrix4();

    // Calculate scale factor between screen and saving canvas
    const scaleXFactor = offscreenCanvasWidth / appCanvasWidth;
    const scaleYFactor = offscreenCanvasHeight / appCanvasHeight;

    // Scale to fit frame
    const scaleToFit = Math.max(frameWidth / imgWidth, frameHeight / imgHeight);
    const normalizedTranslateX = transformMatrix[3] / transformMatrix[0];
    const normalizedTranslateY = transformMatrix[7] / transformMatrix[5];

    const scaledTranslateX = normalizedTranslateX * scaleXFactor;
    const scaledTranslateY = normalizedTranslateY * scaleYFactor;

    const adjustedTransformMatrix: Matrix4 = [
      transformMatrix[0], transformMatrix[1], transformMatrix[2], scaledTranslateX,
      transformMatrix[4], transformMatrix[5], transformMatrix[6], scaledTranslateY,
      transformMatrix[8], transformMatrix[9], transformMatrix[10], transformMatrix[11],
      transformMatrix[12], transformMatrix[13], transformMatrix[14], transformMatrix[15],
    ];

    const offsetX = frame.x * offscreenCanvasWidth;
    const offsetY = frame.y * offscreenCanvasHeight;

    const tempSurface = Skia.Surface.MakeOffscreen(frameWidth, frameHeight);
    if (!tempSurface) return;

    const tempCanvas = tempSurface.getCanvas();
    tempCanvas.clear(Skia.Color("transparent"));

    const srcRect = { x: 0, y: 0, width: imgWidth, height: imgHeight };
    const destRect = { x: 0, y: 0, width: scaledWidth, height: scaledHeight };

    const paint = Skia.Paint();
    tempCanvas.save();
    tempCanvas.translate(frameWidth / 2, frameHeight / 2);
    tempCanvas.concat(adjustedTransformMatrix);
    tempCanvas.translate(-scaledWidth / 2, -scaledHeight / 2);

    tempCanvas.drawImageRect(image, srcRect, destRect, paint);
    tempCanvas.restore();
    const tempImage = tempSurface.makeImageSnapshot();
    snapshots.push({ image: tempImage, x: offsetX, y: offsetY });
  });

  return snapshots;
};


export const renderCollageOffscreen = async (
...
    const mainSurface: SkSurface | null = Skia.Surface.MakeOffscreen(offscreenCanvasWidth, offscreenCanvasHeight);
    if (!mainSurface) throw new Error("Failed to create offscreen Skia surface");

    const mainCanvas: SkCanvas = mainSurface.getCanvas();
    mainCanvas.clear(Skia.Color(collageBackgroundColor));

    // Load images
    const skImages = await Promise.all(
      photos.map(async (photo) => {
        if (!photo.uri) return null;
        console.log("photo uri: ", photo.uri);

        const fileData = await FileSystem.readAsStringAsync(photo.uri, { encoding: FileSystem.EncodingType.Base64 });
        const imageBytes = Uint8Array.from(atob(fileData), (c) => c.charCodeAt(0));
        const skData = Skia.Data.fromBytes(imageBytes);

        return Skia.Image.MakeImageFromEncoded(skData);
      })
    );

const imageSnapshots = drawImagesSeparately(
      offscreenCanvasWidth,
      offscreenCanvasHeight,
      skImages,
      photos,
      collageIndex,
      fullScreenCanvasWidth,
      fullScreenCanvasHeight
    );

    // Merge image snapshots onto the main canvas
    imageSnapshots.forEach(({ image, x, y }) => {
      if (image) mainCanvas.drawImage(image, x, y);
    });

    drawSeparatorLines(mainCanvas, offscreenCanvasWidth, offscreenCanvasHeight, collageIndex, lineColor, lineWidth);

    const finalImage = mainSurface.makeImageSnapshot();
    if (!finalImage) throw new Error("Failed to create image snapshot from surface");

    const pngData = finalImage.encodeToBytes(ImageFormat.PNG);
    const base64String = encode(pngData);

    const tempPath = `${FileSystem.cacheDirectory}collage-${Date.now()}.png`;
    await FileSystem.writeAsStringAsync(tempPath, base64String, { encoding: FileSystem.EncodingType.Base64 });
....

[Hello1nside]

I'm attaching four images to demonstrate the issue:

• Horizontal Collage (Issue on Y-Axis):
  When the collage contains two horizontal images, there is a slight shift in the Y position of the saved collage compared to how it's displayed in the app.

• Vertical Collage (Issue on X-Axis):
  When the collage contains two vertical images, there is a slight shift in the X position of the saved collage compared to how it's displayed in the app.

The problem only happens when the image frame doesn't take the full width or height (e.g., 50% width or 50% height). When the frame is 100% width or height, the position is correct.

This issue seems to be related to how I'm scaling the Matrix4 translation values from the gesture handler for the high-resolution offscreen canvas.

[Hello1nside]

Position error with Zoomed photo:

When a photo is zoomed in (using transformMatrix[0] and transformMatrix[5] for scaling), the zoom is applied correctly, but the position of the image is incorrect in the saved collage.

The photo is meant to be centered, but in the saved image, it's shifted.

Both of these issues seem to be related to the way I apply scaling and translation adjustments to transformMatrix when rendering to a 4K offscreen canvas. Any guidance would be greatly appreciated!

[Hello1nside]

Update
I've been investigating the issue further and found that the problem is not related to how the Matrix4 is scaled or applied.

What I discovered:
Even if I don't change the image position (no translation or adjusting x/y values) and keep the default matrix (all values as 0 or 1), the image still renders in the wrong position when saved offscreen.

Where I suspect the issue lies:
The issue seems to be related to this part of the code where I scale and place the image on the canvas:

const srcRect = { x: 0, y: 0, width: imgWidth, height: imgHeight };
const destRect = { x: 0, y: 0, width: scaledWidth, height: scaledHeight };

const paint = Skia.Paint();

tempCanvas.save();

tempCanvas.translate(frameWidth / 2, frameHeight / 2);
tempCanvas.concat(adjustedTransformMatrix);
tempCanvas.translate(-scaledWidth / 2, -scaledHeight / 2);
tempCanvas.drawImageRect(image, srcRect, destRect, paint);

tempCanvas.restore();

What I suspect:
It seems the way I am scaling the image and centering it on the frame could be the root cause of the issue. Maybe there's something off in how I translate and center the image before drawing it on the canvas.

What's next:
I'm trying to figure out if the scaling or positioning logic here needs correction. Any insights or suggestions would be much appreciated.

[wcandillon]

this tutorial should nicely the approach that needs to be taken here: https://www.youtube.com/watch?v=5yM4NPcTwY4

…

On Fri, Mar 28, 2025 at 11:11 PM Yevhenii Mudrahel ***@***.***> wrote: Update I've been investigating the issue further and found that the problem is not related to how the Matrix4 is scaled or applied. What I discovered: Even if I don't change the image position (no translation or adjusting x/y values) and keep the default matrix (all values as 0 or 1), the image still renders in the wrong position when saved offscreen. Where I suspect the issue lies: The issue seems to be related to this part of the code where I scale and place the image on the canvas: const srcRect = { x: 0, y: 0, width: imgWidth, height: imgHeight }; const destRect = { x: 0, y: 0, width: scaledWidth, height: scaledHeight }; const paint = Skia.Paint(); tempCanvas.save(); tempCanvas.translate(frameWidth / 2, frameHeight / 2); tempCanvas.concat(adjustedTransformMatrix); tempCanvas.translate(-scaledWidth / 2, -scaledHeight / 2); tempCanvas.drawImageRect(image, srcRect, destRect, paint); tempCanvas.restore(); What I suspect: It seems the way I am scaling the image and centering it on the frame could be the root cause of the issue. Maybe there's something off in how I translate and center the image before drawing it on the canvas. What's next: I'm trying to figure out if the scaling or positioning logic here needs correction. Any insights or suggestions would be much appreciated. — Reply to this email directly, view it on GitHub <#3069 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKXVTEMABATUXGQPKWZM32WW3JNAVCNFSM6AAAAABZ3KKSPWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENRVHA4DANI> . You are receiving this because you are subscribed to this thread.Message ID: <Shopify/react-native-skia/repo-discussions/3069/comments/12658805@ github.com>

[Hello1nside]

Thank you for your response! I have already watched the video, and it indeed helped me implement collages in my app. However, I noticed that in the video with stickers, you are creating image snapshots using useCanvasRef.

In my initial implementation, I did it the same way, and it worked perfectly. However, the saved image had low quality because the canvas size was limited to the phone screen dimensions (approximately 700x400).

Since I need the saved image to be in high quality (2160x3840), I implemented offscreen rendering. The code above is from my renderCollageOffscreen component. While the rendering itself works well, the issue I’m facing is that image positions are slightly incorrect in the saved image compared to what is displayed in the app.

I’m currently investigating why this discrepancy occurs. Any insights you might have would be greatly appreciated!

[Hello1nside]

I found the root cause – the issue was not in the offscreen rendering but in the onscreen canvas.

I was applying a 0.9 height multiplier to the canvas to create visual margins, which shifted the images slightly.
Since offscreen uses full size (no multiplier), the rendering was actually correct there.

Removing the multiplier fixes the alignment issue.