v0.3.0

BrushSizzleCompressedFrame.mp4

Massive thanks to @GradeEterna for the beautiful scenes

Brush 0.3 is a massive update to bring high quality splats to all platforms, while training faster, and bringing a ton of new features!

Brush now trains using the "MCMC" splatting technique, but, with its own variation that still grows splats automatically. This keeps the best of both worlds: splats grow first where they are needed, yet explore the scene like in MCMC, to improve quality. This table has some preliminary results. You can set a limit of the maximum number of splats like in the original MCMC. Training works especially better on large scenes where not all views are visible from all angles. Training now also supports massive datasets bigger than RAM and starts instantly.

The web version also gains a lot of new features, with fullscreen modes, efficient file loading, directory loading, bundler integration, NPM compatibility, and faster training. Training on the web is not nearly at feature parity with the desktop version.

Release Note Highlights:

Training

• "MCMC like" training. Higher quality and more robust. Still grows splats automatically like previous methods, while also allowing a maximum nr. of splats cap. For a more detailled write up, see this PR
• Train on datasets bigger than RAM. Only up to some amount of gigs are cached, other files are loaded by the dataloader while training. [1]
• Start training faster [1]
• Training bounds are now based on the splat bounds instead of the camera bounds [1] [2]
• Improve backwards speed with thanks to @fhahlbohm [1] [2] [3] [4] [5] [6] [7]
• Always use init.ply as the init for the training if it exists [1]
• Prefer colmap datasets over nerfstudio, fixes import if your dataset has some random json in it [1]
• Add LPIPS loss [1] [2]
• Use a separable convolution for SSIM [1]
• Lots of other tweaks to the training dynamics, bug fixes, version bumps etc.

UI

• The UI has gone through some redesigns to be cleaner and easier to use
• Add a grid widget [1]
• The arrow keys now rotate the model and move it up/down. Combined with the grid this is helpful to align the ground. [1]
• Press 'F' to toggle fullscreen mode [1]
• Add play/pause button when playing a splat sequence [1]
• Add a FOV slider [1]
• Settings UI panel when loading a new dataset [1]
• Hide console on windows [1]
• Add background color picker [1]
• Add a slider to scale splats [1]
• Reduce atomic adds to improve the speed of the backward pass [1]
• Improve accuracy of training steps/s thanks to @fhahlbohm [1]

Import/export

• Support SuperSplat compressed ply format [1]
• Support r/g/b as color names in ply files [1]
• Sort files properly in zip directories for sequence playback [1]
• Fixed file case sensitivity issues [1]
• Allow double floats in plys [1]
• Swap out the PLY importer/exporter for my own. Speeds up import about 5x [1]

Web

• You can now pick directories on the web, not just individual files [1]
• More efficient file reading on the web [1]
• Improved interop with JavaScript, see the example for some of the available APIs. [1]
• The web parts of Brush now use WASM modules compatible with bundlers, eg. with the demo now using Next.JS [1]
• Add a panel showing various warnings that might happen [1]
• Add touch controls for the viewer UI [1]
• Add dwarf debug info for the Web [1]

Other

• Add number of splats to CLI output [1]
• Improve compile times. Clean builds are ~1.5 minutes on my macbook
• Lots of bug fixes & version bumps
• Add example docker file [1]

Download brush-app 0.3.0

File	Platform	Checksum
brush-app-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
brush-app-x86_64-pc-windows-msvc.zip	x64 Windows	checksum
brush-app-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

0.2.0

Brush 0.2 goes from a proof of concept to a tool ready for real world data! It still only implements the “basics” of Gaussian Splatting, but trains as fast as gsplat to a (slightly) higher quality than gsplat. It also comes with nicer workflows, a CLI, dynamic gaussian rendering, and lots of other new features.

The next release will focus on going beyond the basics of Gaussian Splatting, and implementing extensions that help to make Brush more robust, faster, and higher quality than other splatting alternatives. This might mean that the outputs are no longer 100% compatible with other splat viewers, so more work will also be done to make the Brush web viewer a great experience.

Features

• Brush now measures higher PSNR/SSIM than gsplat on the mipnerf360 scenes. Of course, gsplat with some more tuned settings might reach these numbers as well, but this shows Brush is grown up now!

  • See the results table

• Faster training overall by optimizing the kernels, fixing various slowdowns, and reducing memory use.

• Brush now has a CLI!

  • Simply run brush –help to get an overview. The basic usage is brush PATH –args.
  • Any command works with --with-viewer which opens the UI for easy debugging.

• Add flythrough controls supporting both orbiting, FPS controls, flythrough controls, and panning.

  • See the ‘controls’ popout in the scene view for a full overview.

• Load data from a URL. If possible the data will be streamed in, and the splat will update in real-time.
  -For a web version, just pass in ?url=

• On the web, pass in ?zen=true to enable ‘zen’ mode which makes the viewer fullscreen.

• Add support for viewing dynamic splats

  • Either loaded as a sequence of PLY files (in a folder or zip)
  • Or as a custom data format “ply with delta frames”
  • This was used for Cat4D and for Cap4D
  • Felix kindly shared their script to export this data for reference.

• Open directories directly, instead of only zip files.

  • ZIP files are still supported for all operations - as this is important for the web version.

• Support transparent images.

  • Images with alpha channels will force the output splat to match this transparency.
  • Alternatively, you can include a folder of ‘masks’. This will ignore those parts of the image while training.

• More flexible COLMAP & nerfstudio dataset format

  • Support more of the various options, and differing file structures.

  • If your dataset has a single ply file, it will be used for the initial point cloud.

  • While training, the up-axis is rotated such that the ground is flat (thanks to @fhahlbohm)

    • Note: The exported ply will however still match your input data. I’m investigating how to best handle this in the future - either as an option to rotate the splat, or by writing metadata into the exported splat.

Enhancements

• Add alpha_loss_weight arg to control how heavy to weigh the alpha loss
  • Nb: not applicable to masks mode
• Log memory usage to rerun while training
• Fix SH clamping values to 0 (#76 thanks to @fhahlbohm)
• Better logic to pick ‘nearest’ dataset view
• Better splat pruning logic
• Remove ESC to close
• The web version has SSIM enabled again
• Display more detailed error traces in the UI and CLI when something goes wrong
• Different method of emitting tile intersections (#63 for details)
  • Fixes some potential corruptions depending on your driver/shader compiler.
• Read up-axis from PLY file if it’s included
• Eval PSNR/SSIM now simulate a 8 bit roundtrip for fair comparison
• Add an option --export-every to export a ply file every so many steps
  • See --export-path and --export-name for the location of the ply
• Add an option --eval-save-to-disk to save eval images to disk
  • See –export-path for
• Add notes in CLI & UI about running in debug mode (advising to compile with --release).
• Relax camera constraints, allow further zoom in/out
• Relax constraints on fields in the UI - now can enter values outside of slider range.
• Improvements to the UI, less unnecessary padding.

Highlighted Fixes

• Dataset and scene view now match exactly 1:1
• Fix UI sometimes not updating when starting a new training run.
• Sort eval images to be consistent with the MipNeRF eval images
• Fix a crash from the KNN initialization

Demo (Chrome only currently)

Reference Garden scene (650MB)

Mushroom I captured on a walk (capture was only 50 images or so, background is a bit blurry!)

Thanks

Thanks to everybody in the Brush discord, in particular @fasteinke for reporting many breakages along the way, @fhahlbohm for contributions and helping me fix my results table, @Simon.Bethke and @gradeeterna for test data, @felixtaubner for the 4D splat export script.

Download brush-app 0.2.0

File	Platform	Checksum
brush-app-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
brush-app-x86_64-pc-windows-msvc.zip	x64 Windows	checksum
brush-app-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

Brush 0.0.1

Initial release

This is the initial release of Brush, see the README for more information on the project!

Release notes since the original OS release:

• Add ability to train with transparent images

• Add option to select GPU with the CUBECL_DEFAULT_DEVICE environment variable

• Add 2D image trainer example

• Tweak splitting cloning logic, adds +- 0.5 PSNR

• Fixed backwards gradient for quaternions & SH

• Fix exporting ply files

• Fix evaluation not running

• Fix some NaNs on the web version