Is there a way to implement "backbuffer" textures?

[mikeylemmon]

Many graphics tools and libraries provide a mechanism to expose to shaders a feedback "backbuffer" texture that contains the contents of the previous frame (for example, gl-react provides the "backbuffering" prop on the Node component). This enables the implementation of many time-based effects, as well as physics-based simulations and abstract simulations such as Conway's Game of Life:

• https://gl-react-cookbook.surge.sh/blurfeedback?factor=0.8
• https://gl-react-cookbook.surge.sh/gol
• https://www.shadertoy.com/view/XtdSDn
• https://www.shadertoy.com/view/tsKXR3

Under the hood, backbuffers are usually implemented by creating two framebuffer objects and two roles for them, a draw destination and a backbuffer, that are swapped each frame so that the buffer that is drawn to in frame N becomes the backbuffer in frame N+1 and visa versa. (Here is where the framebuffer roles are swapped in gl-react's Node implementation: https://github.com/gre/gl-react/blob/b116482/packages/gl-react/src/Node.js#L1139)

Is there currently a way to implement backbuffers in RN Skia? I've looked into using Skia.Surface.MakeOffscreen (as mentioned in #1359), but it's unclear to me how to bind the offscreen surface's framebuffer to a shader uniform.

Any help or insight appreciated. Thanks!

[wcandillon]

That's a really interesting question and I would love to explore this further.
Thank you for the examples above, if you have other interesting examples to share, I would love to see them. For instance, I'm not sure what kind of effect is produced here: https://gl-react-cookbook.surge.sh/blurfeedback (I'm not famillar with this effect).

Below is a simple backbuffer example in Skia.

There are a couple of things here:

1. Here we use backbuffering on shader. We assume that this is fine? Especially considering that the result of "regular" Skia drawings and effects can be turned to shaders: https://shopify.github.io/react-native-skia/docs/image-filters/runtime-shader
2. I couldn't make it work with the GPU accelerated offscreen (below the offscreen surface is cpu backed). But it's probably something we can fix without too many problems.
3. Once we establish that we only want to offer backbuffers for shaders, we might be able to offer this functionality declaratively. gl-react and shadertoy APIs will provide inspiration there.

import React, { useRef } from "react";
import type { SkShader } from "@shopify/react-native-skia";
import {
  FilterMode,
  MipmapMode,
  Skia,
  SkiaView,
  TileMode,
  useDrawCallback,
} from "@shopify/react-native-skia";
import { Dimensions } from "react-native";

const bufferA = Skia.RuntimeEffect.Make(`
uniform vec3 iMouse;
uniform shader iChannel0;

half4 main(vec2 fragCoord)
{
    float distance = length(fragCoord.xy - iMouse.xy);
    if (distance < 9.0 && iMouse.z > 0.01) {
        return vec4(1.0, 0.0, 0.0, 1.0);
    }

    return iChannel0.eval(fragCoord);
}`)!;

const { width, height } = Dimensions.get("window");
const offscreen = Skia.Surface.Make(width, height)!;

export const Backbuffer = () => {
  const backbuffer = useRef<SkShader>(
    Skia.Shader.MakeColor(Float32Array.of(0, 0, 0, 1))
  );
  const onDraw = useDrawCallback((canvas, info) => {
    const touch = { x: 0, y: 0, z: 0 };
    const lastTouch = info.touches[0];
    if (lastTouch) {
      touch.x = lastTouch[0].x;
      touch.y = lastTouch[0].y;
      touch.z = 1;
    }

    // 1. Draw to the backbuffer
    const paint = Skia.Paint();
    paint.setShader(
      bufferA.makeShaderWithChildren(
        [touch.x, touch.y, touch.z],
        [backbuffer.current]
      )
    );
    offscreen.getCanvas().drawPaint(paint);

    // 2. Swap
    backbuffer.current = offscreen
      .makeImageSnapshot()
      .makeShaderOptions(
        TileMode.Decal,
        TileMode.Decal,
        FilterMode.Nearest,
        MipmapMode.None
      );

    // 3. Draw Result
    const frontPaint = Skia.Paint();
    frontPaint.setShader(backbuffer.current);
    canvas.drawPaint(frontPaint);
  });
  return <SkiaView style={{ flex: 1 }} onDraw={onDraw} />;
};

[wcandillon]

We're investigating the GPU support: #1498, in its current version, you can use a workaround which adds an unnecessary image encoding but it should be faster than drawing on the CPU:

// Use this function to the result of makeImageSnapshot
const makeNonTextureImage = (texture: SkImage) =>
  Skia.Image.MakeImageFromEncoded(
    Skia.Data.fromBytes(texture.encodeToBytes())
  )!;

On Web, there is an API to make this process as fast as possible, see this demo: https://demos.skia.org/demo/textures/
This the API we are likely to support in the near future. When using Skia Web, we can show you a way to access CanvasKit.

const CanvasKit = (canvas as JsiSkCanvas).ref.CanvasKit;

I feel like we will have something sophisticated for this, in the meantime, thank you for bearing with us :)

[wcandillon]

but also if it crashes on iOS after while, we will definitely investigate it

[wcandillon]

That's an awesome example :)
@chrfalch spent a lot of time to fix the last the last memory leak we had and it's nice to see these improvements in such an example.

@mikeylemmon after a while, the picture stays stable, this is the expected result right? do you run the latest release?

[wcandillon]

@mikeylemmon on iOS using the gpu offscreen, it performs quite good and the result seems to be as expected.

[mikeylemmon]

@mikeylemmon after a while, the picture stays stable, this is the expected result right?

Conway's Game of Life tends to evolve to a relatively-stable end state, so I would expect the picture to become mostly stable, though there's almost always some "blinkers" (https://conwaylife.com/wiki/Blinker) and other oscillators left, so I wouldn't expect a completely static end state. However it may visually appear to be static if the frame rate is approaching 60 FPS, as at that speed the states of 2-frame oscillators will blur together into what looks like a static form (e.g. a cross for the simple 3-cell blinkers).

do you run the latest release?

On mobile I've been testing in Expo Go, client version 2.28.7.1017523. Here are the dependencies in my package.json:

{
  "@expo/webpack-config": "^18.0.1",
  "@shopify/react-native-skia": "^0.1.184",
  "expo": "~48.0.11",
  "expo-status-bar": "~1.4.4",
  "react": "18.2.0",
  "react-dom": "18.2.0",
  "react-native": "0.71.6",
  "react-native-reanimated": "~2.14.4",
  "react-native-web": "~0.18.10"
}

I just tried creating an expo-dev-client build with the same dependency versions as above (plus expo-dev-client@~2.2.1), and it runs into the same crash on iOS that I described experiencing in Expo Go.

@mikeylemmon on iOS using the gpu offscreen, it performs quite good and the result seems to be as expected.

Did you do that by changing the offscreen surface to be created with Surface.MakeOffscreen like this...?

const offscreen = Skia.Surface.MakeOffscreen(width, height)!

That doesn't seem to be working for me — it results in a blank screen. Interestingly no crash though!