GSoC 2025: GPU-accelerated raster ops

[Firestar99]

Problem

Graphite used not have any hardware acceleration and compose the final image entirely on the CPU,
but in the past year, the vello renderer was implemented to accelerate the image composition on the
GPU. However, vello is only well suited for composing vector elements and plain raster images. It
does not implement any operations for (post-)processing raster images, like adjusting hue,
saturation, blending images, a posterization effect, etc. Currently, these “raster ops” are still
evaluated on the CPU, given that Graphite runs within the Web Assembly (wasm) environment of
the browser, limited to a single thread processing one pixel at a time. This significantly slows editing
with any raster-based content being present, in practice 2-3 fps on a typical desktop system.

Benefit

I want to accelerate the processing of rasterization ops using the GPU to achieve acceptable levels of
performance.
Instead of porting the existing raster ops written in Rust to a shader language, I want to use the
rust-gpu shader compiler to compile the existing Rust-based raster ops into shaders. It allows
Graphite to continue using the existing CPU path if the new WebGPU API WebGL is not available, without
duplicating the code across several languages.

Final Report

The technical bits

We want to use the rust-gpu shader compiler to not have duplicate code between shader and cpu nodes. A typical shader compiler like glslc (or shaderc) takes input files written in the C-like glsl language and turns them into SPIRV, a binary intermediary representations (IR) for shaders. Think of it like Java or C# bytecode, that the graphics driver accepts and compiles down to the machine code needed for your graphics card. rust-gpu works quite similar, just that it's input language is ordinary rust and not some C-like custom language.

Unlike a typical shader compiler, rust-gpu is not a full compiler but merely a "codegen backend" for the rustc compiler. This allows rust-gpu to reuse all the tokenization, parsing, type system, borrow checking etc. of the standard rustc compiler, and thus parse all the contructs of the rust programming language. At the very end of the rustc compiler pipeline, the Middle IR (MIR) is passed into a codegen backend to generate machine code. This codegen backend is typically LLVM, but is replaceable by a dynamically loaded library, like the rustc_codegen_spirv crate of rust-gpu.

Codegen backends are build is an unstable internal interface that can change on a whim. This has a few important implications:

• rust-gpu must be upgraded continually for each new rustc release, as this interface changes quite often. If there are major changes, it may take a while for rust-gpu to adopt the newer toolchain, as was the case with rust edition 2024.
• rust-gpu must be used with the exact nightly rustc compiler it was written against. This typically means fixing the entire project to that specific toolchain version, but we want the CPU code of graphite to compile on stable.

rust-gpu edition 2024

With the release of rustc 1.85.0 the rust edition 2024 was stablilized, but also came with significant changes to the codegen backend interface. This proved to be a significant challenge, so while edition 2024 released in February, it took us until July to port to newer toolchain versions. This was a huge blocker for graphite, as they and the entire rust ecosystem had already ported over to edition 2024, which the older compiler could not support. We would have to wait for rust-gpu to support edition 2024, as backporting all of the crates to edition 2021 was not an option.

The porting has primarly been done by @eddyb in Rust-GPU/rust-gpu#249 while I was mostly testing the branch against various projects and debugging issues as they poped up. Notably, this update also required changing the "target specs", a set of json specification files the rustc compiler requires of every codegen backend.

cargo-gpu

When I first looked at cargo-gpu, it was a command line application to compile a shader crate to SPIRV, the shader IR. Notably, it did not care for what toolchain you used in your project and automated the entire process of setting up rust-gpu: It would download rust-gpu and the required toolchain, compile rust-gpu with that toolchain and cache the build, and finally compile the shaders using the selected rust-gpu and toolchain version.

However, it had a few issues that needed to be resolved before we could use it:

• It assumed that the "target specs" of rust-gpu would never change, which they did in edition 2024. Updating target json support Rust-GPU/cargo-gpu#75
• It was only usable from the command line, and we wanted to use it as a library in a build script to automate the build as much as possible. Make cargo-gpu a library, usable from build scripts Rust-GPU/cargo-gpu#71
• While building these and familarizing myself with the entire codebase, I've noticed that their support for multiple rust-gpu versions was quite complicated and could be a lot simpler. refactor: use SpirvBuilder directly, just manage rustc_backend_spirv dylibs Rust-GPU/cargo-gpu#69

While integrating these needed changes, I effectively refactored the entire codebase in 12 PRs and 3.005++ 4.610-- lines.

naga-wgsl transpiling

SPIRV is an open standard for shader IR for exchanging shaders between various compilers and graphics APIs. It was originally invented for the OpenCL compute API, but has evolved into the shader IR of the Vulkan API, which is the main graphics API on both Linux and Android systems. Even Windows is replacing their DXIL with SPIRV, making it the primary cross-platform shader IR to use. It was even set to be the main shader IR for the WebGPU API, but some parties were strictly against using that open IR standard. And so we got wgsl, a new C-like shading language with a rust-like syntax that's not anything like rust itself. And as is typical of web, it is sent to the browser as source code, so we may see some platform differences between Firefox, Chrome and Safari.

But since rust-gpu emits SPIRV, how to we convert from SPIRV to wgsl? Firefox implements the WebGPU API with their wgpu crate, which has a shader transpiler called naga. It was primarily build to convert wgsl into the different output shader languages needed for Firefox to run on all platforms: SPIRV for Vulkan on Linux and Windows, MSL on Mac and HLSL for Windows. But it also supports SPIRV input and WGSL output, so we can chain the compilers to go from rust to SPIRV to wgsl. This isn't anything new, schell's renderling has been doing this quite successfully and has contributed many fixes to the SPIRV input module.

But I didn't just want to setup wgsl transpiling for graphite, it would be much nicer if rust-gpu could handle that internally. This would also allow us to specialize the SPIRV we're emitting for naga, if necessary. So I got to implementing a spirv-unknown-naga-wgsl target, both of which need a bit more work before they can be merged:

• enum SpirvTargetEnv containing all available targets Rust-GPU/rust-gpu#311
• add spirv-unknown-naga-wgsl target via naga Rust-GPU/rust-gpu#280

To test this entire stack without having to integrate this directly into graphite, I choose to reuse a GameJam Game of mine called Colorbubble. It uses wgpu, hand-written wgsl and can be deployed to the web, just like graphite should. And with only a single hand-written wgsl file of 75 lines, it is the perfect small project to test the entire tech stack of cargo-gpu and naga-wgsl transpiling. The result can be found on the branch cargo-gpu, which can be compiled on stable and deployed to the web. I also build a vertical slice in week 3 to compile graphite's invert node into a shader, as it's the simplest node without any parameters.

graphite integration

The proper integration into graphite was more difficult than initially expected due to rust-gpu's #[no-std] requirement. From rustc's perspective, rust-gpu is essentially an embedded target that does not have access to the standard library or a global memory allocator, thus it requires every crate to be no-std (and not need the alloc crate). This makes symbols like Vec, Box, Arc unavailable, so all code using them must be masked out by a cargo feature. I was told that graphite had an std feature that when turned off, allowed it to compile a core set of symbols as no-std from a previous but quite different rust-gpu integration attempt. But alas, that support has withered away so badly, I decided to pursue a different approach to no-std compatibility: Instead of masking out all the std symbols, which is usually quite error prone, we instead have a separate shader crate that is (almost) completely no-std, which is reexported in the std crate. This allows me to move certain no-std symbols from the std crate over without breaking the paths of upstream crates.

runtime

Constraints

• rust-gpu integration should be invisible to other contributors
  • the project must build on stable
  • rust-gpu toolchain must be automagically installed
  • rust-gpu codegen backend must be automagically build and used to compile shaders
  • if compile times are bad, may switch from build scripts to on-demand cmdline
• must run on WebGL
  • vello recently added WebGL support
  • no compute shaders, everything must be done in fragment shaders
  • transpile spirv to wgsl

Deliverables

• prepare rust-gpu for integration
  • instead of backporting graphite to edition 2021, help out upgrading rust-gpu to edition 2024
  • rustup: update to nightly-2025-05-09 (~1.88) and Rust 2024 edition. Rust-GPU/rust-gpu#249
• cargo-gpu allows using rust-gpu on a stable toolchain, manages toolchain install and codegen backend build
  • refactor: use SpirvBuilder directly, just manage rustc_backend_spirv dylibs Rust-GPU/cargo-gpu#69
  • Make cargo-gpu a library, usable from build scripts Rust-GPU/cargo-gpu#71
  • bug: parallel cargo-gpu installing crashes Rust-GPU/cargo-gpu#72 (comment)
  • allow different target jsons for different codegen backend versions, blocker for rust-gpu 2024, see comment
  • integrate naga spirv -> wgsl transpiler
• first per-pixel op: invert
  • images should be propagated as wgpu images
  • invert op should launch a draw
• port simple per-pixel ops to shaders
  • eg. Threshold, Hue, Saturation, Brightness, Posterization
  • support uniforms to pass values to shaders
• port convolution kernels to shaders
  • eg. blur, bloom, edge detection
  • may access neighboring pixels to evaluate their kernel
  • may add faster compute shader variant, if supported
• extension: prototype brush engine
• extension: shader compositing

[Firestar99]

Preparation

Anything before week 1

• help debug rust-gpu edition 2024: rustup: update to nightly-2025-05-09 (~1.88) and Rust 2024 edition. Rust-GPU/rust-gpu#249
• full refactor of cargo gpu to simplify it a lot: refactor: use SpirvBuilder directly, just manage rustc_backend_spirv dylibs Rust-GPU/cargo-gpu#69
• begin port to a lib: Make cargo-gpu a library, usable from build scripts Rust-GPU/cargo-gpu#71

[Firestar99]

Week 1: being sick

Am sick this week, likely caught something last week in the (amazing) rustweek conference. Still made some progress:

• The rust-gpu edition 2024 PR needs to change the "rustc target json files", as rustc's validator would otherwise complain. cargo-gpu was never build to deal with different target jsons, and always uses the one it's shipped with. To not have a hard break between before / after that PR, I refactored it to have per-install (of our codegen backend) target jsons, extracted from our codegen backend. Includes some refactor from the lib port.
  • Updating target json support Rust-GPU/rust-gpu#256
  • Updating target json support Rust-GPU/cargo-gpu#75
• minor
  • reviewed Fix windows builds Rust-GPU/rust-gpu#252
  • reviewed fix: buff the help menu for the wgpu-runner Rust-GPU/rust-gpu#254

[Firestar99]

Week 2: cargo-gpu naga transpile vertical slice

Demo

A vertical slice of using cargo gpu to compile rust-gpu shaders and transpile them to wgsl with naga, so they can be passed to wgpu. Based on a GameJam game of mine with just 4 shaders, small enough to allow quick iteration and API exploration.
https://github.com/Firestar99/colorbubble/tree/cargo-gpu
Branch cargo-gpu

Progress

• rust-gpu 2024 prep: Was asked to put the target specs into a new rustc_codegen_spirv-target-specs crate:
  • Updating target json support Rust-GPU/cargo-gpu#75
  • Updating target json support Rust-GPU/rust-gpu#256
  • published on crates: https://crates.io/crates/rustc_codegen_spirv-target-specs
• lib port mostly done Make cargo-gpu a library, usable from build scripts Rust-GPU/cargo-gpu#71
• started on Naga transpiling to wgsl Rust-GPU/cargo-gpu#76
• found a bug that may bite us later: build script: cargo clippy fails Rust-GPU/cargo-gpu#77
• debugging why hypercube has issues building the demo on fedora, randomly fixed?
• some initial code investigation on how to integrate this into graphite
• minor
  • update compiletest_rs Rust-GPU/rust-gpu#261 to fix no $HOST env var at compile time Rust-GPU/rust-gpu#259
  • Upgrade more dependencies Rust-GPU/rust-gpu#262
  • ci: ensure lockfile is up-to-date Rust-GPU/cargo-gpu#78

[Firestar99]

Week 3: invert_gpu

Demo

A full vertical slice with a new gpu invert node in this draft PR! I hope the image isn't too confusing, this is a screenshot of the node graph, which I then inserted as an image and inverted using the gpu invert node, creating a node graph just like the original screenshot. Also we're suffering from memory leaks, which I'll have to debug soonish.

Progress

• Demo GPU invert node #2690
  • debugging why cargo-gpu's build of rustc_codegen_spirv has wasm compile flags leak into it, failing ci. Can't repo locally, and the runner seems to be down rn.
• Various small fixes for Graphite Rust-GPU/cargo-gpu#80
• review and work on difftesting for rust-gpu Add differential testing harness Rust-GPU/rust-gpu#216
  It's been in the pipeline for quite a while and we finally want to get it underway. Having this we would have probably noticed things like misimplemented intrinsics way earlier.... and noticed that the fix itself was indeed buggy as well
• speedup rust-gpu ci: Split compiletests and difftests into parallel CI jobs Rust-GPU/rust-gpu#272
• make NonUniform merge-able behind feature gate Non uniform for everything! Rust-GPU/rust-gpu#177
• minor
  • change cargo-gpu to Apache and MIT licence, same as rust-gpu Rust-GPU/cargo-gpu#82
  • review and fixup show-targets #36 Rust-GPU/cargo-gpu#79
  • review Fix inverted conditions in count_(leading|trailing)_zeros for 64-bit … Rust-GPU/rust-gpu#268
  • review Fix union field access by representing unions as structs Rust-GPU/rust-gpu#269

[Firestar99]

Week 4: spirv-unknown-wgsl

Progress

• add spirv-unknown-naga-wgsl target via naga Rust-GPU/rust-gpu#280
• learn how to add new targets with add targets vulkan1.3, vulkan1.4 and spv1.6 Rust-GPU/rust-gpu#271
  • required updating CI tooling for vulkan1.4 ci: replace spirv-tools prebuilds with vulkan sdk from lunarg Rust-GPU/rust-gpu#279
  • which also unblocked Update mac target in CI to aarch64-apple-darwin. Rust-GPU/rust-gpu#258
• minor
  • Compiletest docs Rust-GPU/rust-gpu#275
  • review fix: array stride validation errors Rust-GPU/rust-gpu#273
  • review Add query_size_lod and query_size methods to SampledImage Rust-GPU/rust-gpu#281

[Firestar99]

Week 5: refactoring gcore

Progress

• graphite gcore refactoring
  • Put all #[test] in a mod tests #2728
  • Bulk remove old code for legacy GPU node implementations #2722
  • Remove unused dependencies from the Cargo.toml files via cargo shear #2723
  • Fix profiling CI action failing #2734
  • make editor only depend on gstd
    • Re-export all of gcore in gstd and rename 'Image' node to 'Image Value' #2733
    • Bulk rename gcore -> gstd in editor #2729
  • Make gcore a std library
    • Make gcore a std library and remove std, alloc, and more unused features #2724
    • Bulk gcore cleanup, replace core and alloc with std #2735
  • Unify feature dependencies with workspace dependencies #2736
  • Remove the serde feature and instead just always require it #2737
  • Extract gapplication_io from gcore #2742
• cleanup and compiletest add spirv-unknown-naga-wgsl target via naga Rust-GPU/rust-gpu#280
• minor
  • ci: resolve target tuple from rustc Rust-GPU/rust-gpu#286
  • fix compiletests Rust-GPU/rust-gpu#290

Away for an event during the extended weekend.

[Firestar99]

Week 6: refactoring gcore even more

Progress

• experimenting with parallel frontend, results in discord
• graphite gcore refactoring
  • Extract gapplication_io from gcore #2742
  • Make wasm only depend on gstd, not gcore #2745
  • Prep gcore splitup: move various symbols into their own modules #2746
  • Prep gcore splitup 2: Vector extension traits #2759
  • Stacked pr ci #2763
  • Extract gpath_bool from mod gstd::ops path_bool-related nodes #2762
  • Extract gsvg_renderer from gcore, remove gcore/vello feature  #2760
  • Extract gmath_nodes from gcore::ops, create mods debug and extract_xy #2764
  • Extract gelement-nodes for nodes accepting vector, raster or element #2767
  • node macro: copy visibility from node function #2782
  • Extract graster-nodes #2783
  • Extract gbrush #2784
• minor
  • rebase and merge Various small fixes for Graphite Rust-GPU/cargo-gpu#80

[Firestar99]

Week 7: rust-gpu progress

Progress

• graphite gcore refactoring
  • quite some merge conflicts from last week's PRs
  • Fix running tests locally; fix migrations for relocated nodes #2805
• rust-gpu spirv-unknown-wgsl target
  • split out Upgrade wgpu 23 -> 25 Rust-GPU/rust-gpu#309
  • split out Make read_clock_khr always available Rust-GPU/rust-gpu#310
  • split out and finish off enum SpirvTargetEnv
    • enum SpirvTargetEnv containing all available targets Rust-GPU/rust-gpu#311
    • target enum: fixup cargo gpu Rust-GPU/cargo-gpu#92
  • finish off add spirv-unknown-naga-wgsl target via naga Rust-GPU/rust-gpu#280
  • better const folding for int and bool  Rust-GPU/rust-gpu#317
• rust-gpu minor
  • prevent env vars leaking into install Rust-GPU/cargo-gpu#97 fixing CARGO_TARGET_DIR is not supported properly Rust-GPU/cargo-gpu#90
  • review and fixup Remove unsafe from unconditionally safe subgroup ops Rust-GPU/rust-gpu#306
  • review Optimize constant casts to avoid unnecessary capabilities Rust-GPU/rust-gpu#302
  • review [eddyb's version/leftovers] TypedBuffer / #[spirv(typed_buffer)]. Rust-GPU/rust-gpu#319
  • review Add 7 new difftests and push constants support Rust-GPU/rust-gpu#321

[Firestar99]

Week 8: rust-gpu edition 2024

Progress

• graphite gcore refactoring
  • Add ::IDENTITY to node macro to return a ProtoNodeIdentifier that is always a &'static str #2842
  • Refactor node migration to be less susceptible to forgetting outdated strings #2853
  • rebased Extract gelement-nodes for nodes accepting vector, raster or element #2767
• rust-gpu edition 2024 is merged 🎉
  • review fix: better output when consent is required without at a TTY Rust-GPU/cargo-gpu#98
  • review format_args! "decompiler": forced warnings and fmt::Arguments::new_v1_formatted "support". Rust-GPU/rust-gpu#326
  • help out testing rustup: update to nightly-2025-05-09 (~1.88) and Rust 2024 edition. Rust-GPU/rust-gpu#249
  • help out testing rustup: update to nightly-2025-06-23 (~1.89). Rust-GPU/rust-gpu#320)
• rust-gpu
  • better const folding for int and bool  Rust-GPU/rust-gpu#317
  • target has gotten an extra naga add spirv-unknown-naga-wgsl target via naga Rust-GPU/rust-gpu#280
• minor
  • ci: enable clippy for more crates Rust-GPU/rust-gpu#329
  • review Add 7 new difftests and push constants support Rust-GPU/rust-gpu#321
  • review compiletest: avoid invoking rustc with absolute paths. Rust-GPU/rust-gpu#330

[Firestar99]

Week 9: clippy and miri with cargo-gpu

Progress

• graphite
  • fixup Upgrade vello to version 0.5.0 and wgpu to version 25 #2890
• rust-gpu
  • better const folding for int and bool  Rust-GPU/rust-gpu#317
  • target has gotten an extra naga add spirv-unknown-naga-wgsl target via naga Rust-GPU/rust-gpu#280
  • Fix cargo-gpu in build script failing when called by Miri or Clippy
    • Fix cargo-gpu in build script failing when called by Miri or Clippy Rust-GPU/rust-gpu#335
    • Fix cargo-gpu in build script failing when called by Miri or Clippy Rust-GPU/cargo-gpu#101
  • difftest
    • minor changes in Add 7 new difftests and push constants support Rust-GPU/rust-gpu#321
    • major speedups difftest: nextest support and speedups Rust-GPU/rust-gpu#334
  • rust-gpu clippy speedup
    • fix main ci Rust-GPU/spirv-tools-rs#17
    • skip compiling C++ code when running clippy  Rust-GPU/spirv-tools-rs#16
    • publish a minor update to spirv-tools-sys and spirv-tools (version 0.12.1)
    • ci: simplify clippy Rust-GPU/rust-gpu#331

[Firestar99]

Week 10: prep graster-nodes for shaders

Progress

• graphite
  • Clean up code by using Iterator::collect() when constructing instance tables #2918
  • Re-add upload texture #2915
  • Shaders: node macro #2923
  • Shaders: graster-nodes no-std prep #2924
  • Shaders: add gcore-shaders and make graster-nodes no-std #2925
  • remove inner enum from raster types #2929
• minor rust-gpu
  • ci: fix two PRs with same branch name but different forks canceling each other Rust-GPU/rust-gpu#342
  • Add RayQuery::get_intersection_triangle_vertex_positions() extension Rust-GPU/rust-gpu#343
  • review Add HitTriangleVertexPositionsKHR built-in for ray tracing pipelines Rust-GPU/rust-gpu#339
  • report Rust Toolchain Upgrade runs on forks Rust-GPU/rust-gpu#344 and review Only run upgrade rust workflow on main repo Rust-GPU/rust-gpu#345

[Firestar99]

Week 11: first wgsl compilation

Demo

$ git checkout shaders
$ cargo b -p graphene-raster-nodes
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.26s
$ ll ./target/spirv-builder/spirv-unknown-naga-wgsl/release/deps/graphene_raster_nodes.wgsl -h
-rw-r--r-- 1 firestar99 users 149K Aug  3 23:37 ./target/spirv-builder/spirv-unknown-naga-wgsl/release/deps/graphene_raster_nodes.wgsl

Progress

• graphite
  • Shaders: graster-nodes no-std fixups #2984
  • Shaders: runtime and shader node codegen #2985
• rust-gpu
  • allow integer literals to have a suffix Rust-GPU/rust-gpu#357 to fix Bindings fail to parse 1u32 Rust-GPU/rust-gpu#356
  • spirv attribute errors in proc macros don't show errors properly due to span being the macro invocation Rust-GPU/rust-gpu#355
  • entry points with an enum of exactly two variants fails Rust-GPU/rust-gpu#358
• minor rust-gpu
  • review and fixup Update most significant versions of direct dependencies Rust-GPU/rust-gpu#349
  • make / review Update rust-gpu dependency Rust-GPU/cargo-gpu#107 Update dependencies Rust-GPU/cargo-gpu#106

[Firestar99]

2 week break

Not advancing the week counter, just taking 2 weeks off

[Firestar99]

Week 12

Demo: invert node

The invert node works again. The rest has non-functioning nodes, as the runtime does not yet support non-texture arguments. (The "Extract Executor" is copied from within the "Upload Texture" node)

Demo: posterize node

We can now pass parameters to shaders, allowing nodes like posterize to work! Though any node using floating point numbers is still broken, since I had to switch them from f64 to f32* and neither graphite's UI or node graph seems to support that properly.

*f64 on GPUs: Not only is it typically 64 times slower on consumer level GPUs, but WebGPU spec doesn't expose f64 either. The wgpu impl exposes a SHADER_F64 feature, which is only available on native vulkan, if the desktop app wants to have it for whatever reason.

Progress

• graphite
  • Remove unused cargo dependencies #3063
  • node-macro: cleanup ParsedField struct #3064
• graphite shaders
  • Shaders: more graster-nodes no-std fixups #3090
  • Add nix configuration for the rust-gpu toolchain #3027
  • Shaders: runtime and shader node codegen #2985
• rust-gpu
  • review and find ICE in Make black_box a no-op Rust-GPU/rust-gpu#363

[Firestar99]

Week 13

Demo

Progress

• graphite shaders has been split up into multiple PRs:
  • Shaders: more graster-nodes no-std fixups #3090
  • Add nix configuration for the rust-gpu toolchain #3027
  • Shaders: rust-gpu integration to compile shader nodes to WGSL #3097
  • Shaders: runtime and shader node codegen #2985
  • Shaders: add BufferStruct to support bool and enums #3109
• graphite CI and cargo
  • Make CI reject an out-of-date Cargo.lock manifest #3101
  • Disable proc-macros crate from being tested due to cargo bug #3103
  • CI: build rust with test profile #3106
  • Desktop: Pin CEF version #3113
• graphite UI support, with help from @0HyperCube
  • Add node input type support for f32 to enable usage on GPU #3095
  • Change Table<Color> node inputs to Color where only one value is used so GPU nodes work #3096
• rust-gpu
  • Implement Default for InstalledBackend Rust-GPU/cargo-gpu#108
• cargo/rustc bug reporting
  • crate-type = ["rlib", "dylib"] fails to build with test profile rust-lang/cargo#15890