[RFC] The future of @typegpu/react

[iwoplaza]

Note

This is a draft discussion

Full-screen effect

import tgpu, { fullScreenTriangle, type Configurable } from 'typegpu';
import * as d from 'typegpu/data';
import { useUniform, Canvas, Pass, RenderPipeline, Config } from '@typegpu/react';

const layout = tgpu.bindGroupLayout({
  tint: { uniform: d.vec3f },
});

const fragment = () => {
  'use gpu';
  // TODO: Write a very cool shader effect
  return d.vec4f(layout.$.tint, 1);
};

export function MyEffect() {
  // Allocated a uniform buffer that we can update imperatively
  const tint = useUniformRef(d.vec3f);
  
  // Runs every frame
  useFrame(({ time }) => {
    tint.value.x = Math.abs(Math.sin(time));
    tint.value.y = Math.abs(Math.sin(time * 0.8 + 0.1));
    tint.value.z = Math.abs(Math.sin(time * 0.6 + 0.2));
  });
  
  return (
    <Pass schedule="frame">
      <RenderPipeline vertex={fullScreenTriangle} fragment={fragment}>
        <BindGroup layout={layout} entries={{ tint }} />
      </RenderPipeline>
    </Pass>
  );
}

export function App() {
  return (
    <Canvas>
      <MyEffect />
    </Canvas>
  );
}

3D Objects

import tgpu, { fullScreenTriangle, type Configurable } from 'typegpu';
import * as d from 'typegpu/data';
import { useUniform, Canvas, Pass, RenderPipeline, Config } from '@typegpu/react';

const meshLayout = tgpu.bindGroupLayout({
  modelMatrix: { uniform: d.mat4x4f },
  albedo: { uniform: d.vec3f },
});

const vertex = ({ pos }: { pos: d.v3f }) => {
  'use gpu';
  return meshLayout.$.modelMatrix.mul(d.vec4f(pos, 1));
};

const fragment = () => {
  'use gpu';
  return d.vec4f(meshLayout.$.tint, 1);
};

export function Monkey({ albedo, pos }: { albedo: d.v3f, pos: d.v3f }) {
  const monkeyMesh = useMonkeyMesh();
  const modelMatrix = useMemo(() => mat4.translation(pos, d.mat4x4f()), []);
  
  // Optional bindings
  const bindings = useMemo(() => ([
    [fooSlot, 123],
    [(cfg: Configurable) => /* ... */],
    // ...
  ]), []);
  
  return (
    <Config bindings={bindings}>
      <RenderPipeline vertex={vertex} fragment={fragment} attributes={monkeyMesh.layout.attrib}>
        {/* things provided after the pipeline is created */}
        
        {/* the entries are passed into the shader as automatically created resources */}
        <BindGroup layout={meshLayout} entries={{ modelMatrix, albedo }} />
        {/* monkeyMesh has 'layout' and 'buffer' properties, which fit this component */}
        <VertexBuffer {...monkeyMesh} />
      </RenderPipeline>
    </Config>
  );
}

export function App() {
  return (
    <Canvas>
      <Pass schedule="frame">
        <Monkey pos={d.vec3f(0, 0, 0)} albedo={d.vec3f(1, 0, 0)} />
        <Monkey pos={d.vec3f(1, 0, 0)} albedo={d.vec3f(0, 0, 1)} />
        {/* ... */}
      </Pass>
    </Canvas>
  );
}

Imperative injects

The declarative API might not be the preferred way to do something. In those cases, components like RenderPipeline and BindGroup allow an already created resource to be passed in and used instead.

function Foo() {
  const root = useRoot();
  const pipeline = useMemo(() => {
    return root
      .withVertex(/* ... */)
      .withFragment(/* ... */)
      .createPipeline();
  }, [root]);

  return (<RenderPipeline pipeline={pipeline} />);
}

function App() {
  return (
    <Pass schedule="frame">
      <Foo />
    </Pass>
  );
}

[piaccho]

I really like this idea of a composable API approach and how it handles passes, pipelines, and resource bindings with JSX. I think it will be much more convenient than a hook-based approach. I just wonder if we need to expose some other abstractions and how much "declarative" we want them to be.