Expected performance for rendering with RTX 3090?

[creiser]

On the fox scene I get ~5 FPS @ ~1080p while being fully zoomed in such that all rays hit the object. Is this the performance you would expect for a 3090?

[Tom94]

Hi there, yes, that seems to match what I'm getting on that scene.

For larger framerates at high resolutions on real-world scenes, I recommend enabling DLSS. That should afford quite a jump, e.g. on a mip-NeRF 360 scene I'm getting this with DLSS enabled, upscaling from 540p to 1080p:

(video compression unfortunately killing much of the detail)

garden_low.mp4

[Tom94]

To elaborate on why real-world scenes tend to run more slowly than synthetic ones:

• as @creiser already implied, many of the pixels of synthetic scenes are background, which saves computation.
• additionally, synthetic scenes tend to have much sharper surfaces. In real-world scenes, the NeRF reconstruction tends to "cheat" glossiness through volumetric fuzz, which means that many more steps are required along each ray to accumulate full opacity. Depending on scene, this can affect performance by susprisingly large amounts -- I've seen 2-10x variation in the average number of steps per ray across different real-world scenes, so the performance is very much scene-dependent.

[creiser]

Okay, thanks for the clarification. Is tiny-cuda-nn fully optimized for inference or did you need to compromise in order to have something that works for arbitrary network sizes, input/output sizes, etc. Is the bottleneck the MLP evaluation or is the random access of the hash table? Do you think that fusing the hash table lookup with the MLP evaluation would give some speed-ups? As an alternative to fusion, one could also overlap the hash table lookup and the MLP evaluation with help of CUDA streams?

I also noticed that on the fox scene 40 evals per ray are required, while in my own experience around 8 evals per ray are sufficient. In both cases, only the rays that hit an object are counted. But yeah, as you said my experience stems from synthetic scenes. It would be still nice if we could bring down the number of required evaluations with a regularizer.

The video of the mipNeRF-360 scene looks absolutely incredible :)

[Tom94]

I think the tiny-cuda-nn side is reasonably optimized. Arbitrary network sizes have a cost, but it's relatively minimal, since tiny-cuda-nn specializes on many configurations at compile time and merely selects appropriate code paths at run time (hence the long compilation). That said, only about 20% of the time is spent in MLPs, a similar (maybe slightly larger amount) in the encoding, and the rest in ray marching / compaction / optimizer / rendering related computations. Not much to be gained by further trying to optimize the MLPs (or by trying to avoid using MLPs).

On the lego scene, the avg. number of steps per ray end up between 4 and 5, so yeah, implementing a regularizer à la RefNeRF / RegNeRF within instant-ngp could help real-world scenes a lot!

[creiser]

Okay, I am surprised that raymarching and compaction take longer than the hash table lookup and the MLP combined. Neither raymarching nor compaction requires a lot of compute or memory.

[creiser]

I digged a bit into this and measured the following timings:

compaction: 26 ms
input_generation: 20 ms
eval: 182 ms
composite: 12 ms
total: 242 ms

So about 75% of the time is spent on hash grid and network evaluation. But here even the auxiliary parts (compaction, input_generation, composite) take up 58 ms, making real-time rendering impossible even if we would reduce the cost of evaluation to 0 by e.g. choosing a smaller MLP or optimizing the kernels further. I noticed that n_steps_between_compaction ended up being mostly 1, leading to a high number of kernel launches.

To alleviate this I set n_steps_between_compaction = 16. With this new config I measured:

compaction: 7 ms
input_generation: 7 ms
eval: 211 ms
composite: 5 ms
total: 231 ms

This reduces the required time for the auxiliary parts from 58 ms to 18 ms. This makes real-time rendering at least possible. With this setting over 90% of the time is spent on decoding. Probably it is necessary to fuse these operations or otherwise overlap these computations.