Support in <Group /> for saveLayerFlags and backdrop

[macksal]

No description provided.

[macksal]

SaveLayerFlag being a union is a little awkward since this can actually be a bitset OR of any of the flags. But this problem seems to exist elsewhere in the library. Another option is one boolean property per flag, but then we just have to unpack that everywhere, so it may not be worthwhile.

[macksal]

this is a little awkward, but I tracked everywhere in the graph traversal that ends up calling saveLayer. The Paint props on group must also be provided to saveLayer, so I decided it is safe if I always store the flag prop alongside the layer or other paint props.

BTW I think I found that this <skLayer><skGroup> structure will end up calling saveLayer (for the layer node) and then later calling save via saveCTM (for the group node). Small possibility for optimisation if they can be combined. From what I see the if here is only needed so the paint can be provided as JSX, but I think if the paint element was given as a child to skGroup it might just work out of the box anyway?

Edit: this split creates a surprising gap in test coverage, the execution path when passing an SkPaint or a <Paint/> is quite different. My declarative tests miss one side

[wcandillon]

Hello Mackenzie 🙋🏼‍♂️ This looks interesting. Can you tell me more about the use-case? Maybe we can start with adding tests for it? We can build the reference results using the imperative API. That would be a good starting point, what do you think?

[macksal]

Hi again @wcandillon !

My use case is for graphical effects that require the use of saved layers. An example is Bloom, which causes bright areas of an image to smear into their surroundings. This is often implemented in postprocessing by copying the result of a pipeline, applying some threshold function, blurring, and then compositing back with the original, e.g. using "plus" or "screen" modes. This is not so easy to implement without a way to save the current framebuffer and re-draw it as a texture. We could draw the scene multiple times, but if we need to run multiple of these postprocessing effects in sequence, there's a quadratic number of draws required. Performance drops quickly, especially if each of the steps involve kernel operations like gaussian blur, which then go onto sample other kernels, et cetera. So saved layers/framebuffers are normally used.

Today I’m manually snapshotting the surface and re-drawing it in later steps, but I’d like to move this towards saveLayer and eventually the declarative API. There’s some overlap with <BackdropFilter />, but a few limitations block this:

• There’s no way to provide a Paint for how the saved content is composited on restore.

• You can’t draw additional content between saveLayer / restore in the declarative API (see DrawingContext.ts / DrawingCtx.h).

My plan was to support SaveLayerInitWithPrevious in <Group />, since it already supports paint properties. But now I see that the underlying feature supporting BackdropFilter - saveLayer's backdrop arg - implies SaveLayerInitWithPrevious, and can be more flexible.

My proposal, then is to add support for backdrop as an ImageFilter property on <Group/>. This keeps <BackdropFilter/> as the css-like feature, while <Group/> can then be generic encompass all of the features provided by saveLayer (backdrop filter, paint for compositing, saveLayerFlags, and drawing extra elements on the new layer before restoring).

Would you be accepting of this feature or see barriers to the implementation? (Support both saveLayerFlags and backdrop as props on Group)

[wcandillon]

Let's start with writing a few tests for it with the imperative API. Is the imperative API supporting enough options at the moment? I look like it right?
And from there we can add support for it in the declarative API. What do you think?

[macksal]

Yes, the imperative API already has everything 👍 This is what I want to implement via declarative:

let myPaint: SkPaint;
let myBackdropFilter: SkImageFilter;
let myFlags: SaveLayerFlag;

canvas.saveLayer(myPaint, null, myBackdropFilter, myFlags);
canvas.draw_some_other_elements();
canvas.restore();

<BackdropFilter/> lets us control myBackdropFilter, but not myPaint, and we can't draw our extra elements in the middle.

<Group/> lets us control myPaint, and draw extra elements in the middle, but we can't control myBackdropFilter or myFlags.

So if we add backdrop/saveLayerFlags props to Group.tsx, we can pass those down, and now the consumer of Group.tsx has access to all of saveLayer's functionality.

I can come up with some tests, would e2e/Group.spec.tsx with checkImage calls be appropriate?

[wcandillon]

can you give me 2/3 imperative API example that we can add as tests. This is an example for instance: https://github.com/Shopify/react-native-skia/blob/main/packages/skia/src/renderer/__tests__/e2e/RuntimeShader.spec.tsx#L116. but using the drawOffscreen test API like there: https://github.com/Shopify/react-native-skia/blob/main/packages/skia/src/renderer/__tests__/e2e/Paragraphs.spec.tsx#L135

We will use these are the reference results for the declarative API

[macksal]

@wcandillon I added a few imperative tests with 2 key use cases. (saving a layer through InitWithPrevious OR a backdrop filter, drawing more content on the layer, and then compositing to the previous layer).

[macksal]

it would be really nice if we could pass a child node to the backdropFilter prop. (Similar to the layer prop - that accepts an SkPaint object or a <Paint> node).

I couldn't quite work out how to do this without a big change to the node visitor. I think this dual support for the layer prop is handled by passing the paint node or related nodes as children. They are then implicitly grouped by type in sortNodeChildren.

Because the nodes that would be supported in backdropFilter prop are just image filter nodes, we'd need a way to differentiate between the paint (children) props and this new backdrop prop.

The features still works by passing an SkImageFilter only, we could revisit this later.

[macksal]

@wcandillon I've implemented declarative tests and a passing implementation. Please take a look whenever convenient.
Only 2 caveats:

• The tests are still messy; I'm not sure if I'm supposed to keep the original imperative tests. It seems like a good idea to keep them checked in, but I didn't see any examples. Please let me know any recommendation.
• backdropFilter prop doesn't support passing JSX, only a SkImageFilter object for now.