Writing internal docs about shader generation

Author: iwoplaza

apps/typegpu-docs/astro.config.mjs

@@ -274,6 +274,10 @@ export default defineConfig({
               label: 'Naming Convention',
               slug: 'reference/naming-convention',
             },
+            DEV && {
+              label: 'Shader Generation',
+              slug: 'reference/shader-generation',
+            },
             typeDocSidebarGroup,
           ]),
         },

apps/typegpu-docs/src/content/docs/reference/shader-generation.mdx

@@ -0,0 +1,132 @@
+---
+title: Shader Generation
+draft: true
+---
+
+TypeGPU houses a very powerful shader generator, capable of generating efficient WGSL code that closely matches the input
+JavaScript code.
+
+## The phases of code generation
+
+The whole end-to-end process of turning JS into WGSL can be split into two phases:
+- Parse the JS code into an AST.
+- Collapse each AST node into a [snippet](#snippets) depth first, gradually building up the final WGSL code.
+
+We found that we don't always have enough information to do both phases as a build step (before the code reaches the browser).
+For example, the type of a struct could only be known at runtime, or could be imported from another file, which complicates static analysis:
+
+```ts twoslash
+import * as d from 'typegpu/data';
+
+declare const getUserSettings: () => Promise<{ halfPrecision: boolean }>;
+// ---cut---
+const half = (await getUserSettings()).halfPrecision;
+
+// Determining the precision based on a runtime parameter
+const vec3 = half ? d.vec3h : d.vec3f;
+
+const Boid = d.struct({
+  pos: vec3,
+  vel: vec3,
+});
+
+const createBoid = () => {
+  'use gpu';
+  return Boid({ pos: vec3(), vel: vec3(0, 1, 0) });
+};
+
+const boid = createBoid();
+//    ^?
+```
+
+:::caution
+We could do everything at runtime, transforming the code a bit so that the TypeGPU shader generator can have access to
+a function's JS source code, along with references to values referenced from the outer scope.
+
+The code shipped to the browser could look like so:
+```js
+const createBoid = () => {
+  'use gpu';
+  return Boid({ pos: vec3(), vel: vec3(0, 1, 0) });
+};
+
+// Associate metadata with the function, which the TypeGPU generator can later use
+(globalThis.__TYPEGPU_META__ ??= new WeakMap()).set(createBoid, {
+  v: 1,
+  name: 'createBoid',
+  code: `() => {
+    'use gpu';
+    return Boid({ pos: vec3(), vel: vec3(0, 1, 0) });
+  }`,
+  get externals() { return { Boid, vec3 }; },
+});
+```
+
+However, parsing code at runtime requires both shipping the parser to the end user, and having to spend time parsing the code,
+sacrificing load times and performance.
+:::
+
+In order to avoid parsing at runtime while keeping the desired flexibility, we parse the AST at build time and compress it into
+our custom format called [tinyest](https://npmjs.com/package/tinyest). It retains only information required for WGSL code
+generation.
+
+The code shipped to the browser looks more like this:
+```js
+const createBoid = () => {
+  'use gpu';
+  return Boid({ pos: vec3(), vel: vec3(0, 1, 0) });
+};
+
+(globalThis.__TYPEGPU_META__ ??= new WeakMap()).set(createBoid, {
+  v: 1,
+  name: 'createBoid',
+  // NOTE: Not meant to be read by humans
+  ast: {params:[],body:[0,[[10,[6,"Boid",[[104,{pos:[6,"vec3",[]],vel:[6,"vec3",[[5,"0"],[5,"1"],[5,"0"]]]}]]]]]],externalNames:["Boid","vec3"]},
+  get externals() { return { Boid, vec3 }; },
+});
+```
+
+## Snippets
+
+Snippets are the basis for TypeGPU shader code generation. They are immutable objects that hold three values:
+- *value*: A piece of WGSL code, or something "resolvable" to a piece of WGSL code
+- *dataType*: The inferred WGSL type of `value` [(more here)](#data-types)
+- *origin*: An enumerable of where the value came from (if it's a reference to an existing value, or ephemeral)
+[(more here)](#origins)
+
+```ts
+// A simple snippet of a piece of WGSL code
+const foo = snip(
+  /* value */ 'vec3f(1, 2, 3)',
+  /* dataType */ d.vec3f,
+  /* origin */ 'constant'
+); // => Snippet
+
+// A simple snippet of something resolvable to a piece of WGSL code
+const bar = snip(
+  /* value */ d.vec3f(1, 2, 3),
+  /* dataType */ d.vec3f,
+  /* origin */ 'constant'
+); // => Snippet
+```
+
+If a snippet contains a value that isn't yet resolved WGSL, we defer that resolution as late as possible, so that we can
+perform optimizations as we generate. For example, if we're evaluating the given expression `3 * 4`, we first interpret
+both operands as snippets `snip(3, abstractInt, 'constant')` and `snip(4, abstractInt, 'constant')` respectively.
+Since both are not yet resolved (or in other words, known at compile time), we can perform the multiplication at compile time,
+resulting in a new snippet `snip(12, abstractInt, 'constant')`.
+
+:::note
+If we were instead resolving eagerly, the resulting snippet would be `snip('3 * 4', abstractInt, 'constant')`.
+:::
+
+### Data Types
+
+The data types that accompany snippets are just [TypeGPU Data Schemas](/TypeGPU/fundamentals/data-schemas). This information
+can be used by parent expressions to generate different code.
+
+:::note
+Data type inference is the basis for generating signatures for functions just from the arguments passed to them.
+:::
+
+### Origins