Suggestion for Implementing Rain and Snow Effects with Particle System

[meetSeparate]

Hi @erichlof ,

I hope this message finds you well. I wanted to express my gratitude for your open-source path tracing project on GitHub. It has been incredibly helpful for my studies.

I have a specific idea I would like to explore: implementing rain and snow effects using a particle system within the path tracing framework. Additionally, I am interested in how these effects could interact with models, such as causing collisions and reflecting or refracting light as they intersect.

Do you have any thoughts on how to approach this implementation? Any guidance or insights you could provide would be greatly appreciated.

Thank you once again for your amazing work!

Best regards,
Zhang Ziyu

[erichlof]

Hello @meetSeparate !

Sorry for the delayed response - here in the US it is the holiday season and I have been on vacation the last week.

Thank you for the kind words - I'm happy that you are able to use and learn from my path tracing project. 😊

A particle system for the renderer sounds great! I have played with the idea years ago, back when I was first working on the ray marching outdoor environment demos, like Terrain_Rendering.html and Arctic_Circle.html. These demos could benefit from some weather effects like light rain or snow. However, at the time those demos were made, I didn't have a good plan for how to implement particle systems, especially rain drops that would possibly interact with the terrain.

Another area that a particle system would be useful is in the path traced games that I have links to here on my repo, as well as the latest game I'm working on, called Glider Ball 3D. My style of 3D games has always been fast and simple (and hopefully fun!) gameplay with small, limited game scope. It would be fun to add some game effects like sparks, smoke, and even small explosions to the game objects. It would be even cooler if the particles would be able to interact with the game objects and the surrounding static environment, as you suggested.

As for my advice for actually implementing a particle system, in particular a rain and snow system, unfortunately I'm not experienced in this area of rendering, but I could make an educated guess as to how to begin at least.

Before I got into ray tracing and path tracing, I was, and still am, a hobby 3D game programmer. Way back in the late 1990's/ early 2000's, I completed and released 2 games that needed particle effects - one was Virtual Bombard (a Scorched Earth / Artillery/ Tank Wars clone, but in 3D) and Asteroid Patrol (a classic Asteroids clone, but also in 3D). In these games, when the player's ship/ tank exploded, I wanted to have pieces of the vehicle explode outward from the player, and then in the case of the Asteroids game, float gently outward into space, unaffected by gravity, and in the tanks game, have the particles explode more upward and definitely affected by gravity, pulling the small pieces back down to Earth and even bouncing once or twice around the exploded tank. Even though I was using OpenGL 1.1 and the C programming language for a Windows 98 build (I know, I'm old!), I remember that everything had to be set up on the CPU side and the actual motion and life/death of the particles also had to come from the CPU side. In other words, the GPU, or renderer, didn't know about the particles until the last second, when it was time to send all the triangles down the pipeline to the rasterizer. I think I ended up making each particle a small object of 4 triangles each, a tetrahedron, which is the minimum number of triangles to have a closed 3D object. The 4 vertices of each tetrahedron particle were slightly randomized at creation, which had the effect of some particles being smaller or bigger, and some particles being more or less stretched out in a particular direction. Back then, this simple geometry allowed me to add around 1000 particles for each explosion, and still kept the framerate smooth at 60 fps.

This worked well for the traditional rasterization pipeline many years ago, but I don't think this approach would work so well for our current ray tracing projects. The reason why is that any time you add triangles to a ray tracer, you're going to slow it way down, unless you have a sophisticated BVH acceleration structure to handle dynamic triangles that can move fast and change shape. So for each particle (rain drop or snow flake), something other than triangles would have to be used. In theory, you could have each drop be made from an oblong or elongated sphere or ellipsoid, but the problem remains of adding hundreds or thousands of these objects - it will slow the renderer down a lot, having to check through each drop's bounding box for a possible intersection, and then do a somewhat costly ray-deformed Ellipsoid intersection test for each positive hit.

I think a better approach, at least for now in the experimental stage, would be to have each particle represented by a small quad with transparency and a texture. And each quad would always face the camera, no matter where it is located in the scene. I believe that's how modern particle systems are made, although I haven't been keeping up on how Unreal Engine's Niagara system handles particles - the UE developers probably offer way more options than just camera-facing transparent quads!

However, innocent as this quads solution sounds, you are still adding many 'objects' to the ray traced scene. With rasterization, GPUs were designed to handle millions of triangles on screen without breaking a sweat, but with ray/path tracers, each triangle, each quad which is 2 triangles, hurts you a little - and this cost can add up real fast.

This is why I have yet to implement any kind of particle system in my path tracing projects- I'm just not sure how to handle all of the additional required geometry that is dynamic, fast moving, and has randomized, unpredictable movements, (especially snow flakes). If you didn't want to deal with triangles or quads, another route you could take is rendering all of the particles on the GPU side only - the CPU (or JavaScript in our web case) would know nothing of the particles. Examples of this technique can be found on ShaderToy- just search particles, rain, snow, etc., and you should find some good examples of how each shader artist did it. Although this would be the more ray tracing and framerate friendly option, a major drawback of using GPU shader-only particle systems is that it would be very difficult, maybe impossible, to have the particles interact with the scene objects around them. I'm thinking of bright sparks shooting out, or falling snowflakes and raindrops that land on surfaces and need to splash or bounce correctly. If the CPU physics code can't 'see' all of the particles because they're GPU-side only, I'm not sure how the physics would work.

These are just some thoughts and issues to keep in mind while developing your particle system. Again, this is an area where I have spent very little time in, and it was many years ago for a different kind of platform and traditional rasterization renderer. I seem to recall seeing particle effects on some of the RTX ray tracing demos from NVIDIA, but I can't say for sure how they go about implementing it - and their dynamic BVH acceleration structure builders and traversal code is not public, so I don't really know how they handle all of the additional particle objects/geometry in their RTX ray tracing system.

Please let me know if you had any plans of your own moving forward, or if you had any ideas on how to start adding a particle system. My final piece of advice is, at first, just try the most simple, straight-forward solution - even if it is very un-optimized and even if it doesn't end up working in the end. The process by which you make your 1st attempts will reveal gaps and flaws in your understanding of the problem, and will empower you to choose better and more optimal solutions.

Hope this helps, and please keep me posted with any progress or questions you may have! 🙂

-Erich

[vinkovsky]

@erichlof Every time I read your answers, I admire how detailed you explain complex things in simple language. This is very valuable. I want to thank you for this and wish you a Happy New Year!

[erichlof]

@vinkovsky Thank you, Arkady! Whenever I'm trying to implement something for the first time, I have to (at least for me) understand it from a simplified 'bird's-eye' view first. Unfortunately when working with ray tracing, and path tracing in particular, we live in a world filled with complex language and underlying math (optics, calculus, monte carlo, statistics, probability theory, etc) that is often assumed that the reader/programmer already knows, especially in academic path tracing papers. And often the complex language used to describe a certain technique is used for language sophistication-sake (to sound like you have strong credentials and are an authority on the subject, ha). So what ends up happening is that a topic that could be simplified or more easily-explained, becomes muddied and obscured with complex language for complexity's-sake and assumptions about the reader's previous knowledge of the underlying fundamentals. So I'm glad that someone out there like you is able to understand what I'm talking about! :-D Happy New Year!

[meetSeparate]

Hello @erichlof,

Thank you so much for taking the time to provide such a detailed and thoughtful response, especially during the holiday season! I greatly appreciate your insights and advice—they are incredibly valuable to me as I work on this project.

I want to share some updates and ideas based on your suggestions and experience. After some experimentation, I’ve successfully implemented a CPU-based particle system that includes features like particle emitters, customizable parameters, and physical collisions. For the particles themselves, I’ve used billboards—camera-facing quads with textures and transparency. Additionally, I’ve also explored using meshes as particles, which allowed for more complex visual effects, though, as you mentioned, the performance cost of this approach can be significant.

However, I’m now at a crossroads where I want to explore GPU-based particle systems to improve performance and scalability. As you pointed out, the GPU-based approach could potentially handle thousands (or even millions) of particles more efficiently, but the challenge lies in enabling meaningful interactions between particles and the scene geometry (e.g., collisions, splashes, or reflections).

Here are a few ideas I’ve been considering, and I’d love to hear your thoughts:

Using Compute Shader-like Techniques for Particle Dynamics (WebGL Limitations)
In WebGL 2, since compute shaders are not directly supported, we can use Framebuffer Objects to simulate similar behavior.

Use two textures to store particle data (e.g., position, velocity, lifetime, etc.). One texture represents the current frame's data, while the other represents the next frame's data. The particle physics simulation is performed in the fragment shader, and the results are written to the texture for the "next frame."
Separating Particle Updates and Rendering: Use one rendering pass specifically for updating particle logic (using FBOs), and another pass for rendering particles.
Advantages: This method allows particle physics to be handled entirely on the GPU, reducing the need for CPU-GPU data transfer.
Challenges: WebGL 2 has limited texture access and manipulation capabilities, which might make certain complex logic difficult to implement.
I'm not entirely sure if this approach is feasible in practice, especially in terms of efficiently handling particle updates and rendering in a WebGL-based path tracing framework. I'd love to hear your thoughts or suggestions on whether this could work or how it might be improved!

Finally, I completely agree with your advice to start simple and iterate. The process of trial and error has already taught me a lot, and I’ve found that even early, unoptimized prototypes can spark new ideas for improvement. Your emphasis on learning through experimentation really resonates with me.

Thank you again for your inspiring work and for sharing your wisdom so generously. If it’s okay, I’d love to keep you updated on my progress and seek your feedback as I explore these ideas further!

Wishing you a Happy New Year and all the best in 2025!

[erichlof]

@meetSeparate

Happy New Year, Zhang! Wow, I didn't realize how far you've already come with implementing your particle system! I will be glad to offer any advice or suggestions that I can, as it relates to a path tracing shader renderer. Although from how you describe your implementation and your crossroads options, you've already put a lot of time and thought into this - much more than I have, especially with implementing a modern particle system (recall that the last time I implemented particles, OpenGL 1.1 was the new thing, ha).

Something I like to do when deciding which 'fork in the road' to go down, is to list pros and cons to the options. Just so we're on the same page, let's list some pros and cons of your choices moving forward (feel free to correct or add to these):

CPU-based particle system (with GPU shader ray tracing handling actual final drawing to screen)

Pros
You already have a working CPU system in place, which is a really big Pro

In general, CPUs are able to efficiently handle 'for' loops, in particular, looping over thousands or even hundreds of thousands of particles to update position, velocity, lifetime, prune potential collisions, and handle basic collision response (bouncing, reflecting, etc)

Collisions and other interactions between the particles and the scene objects in the surrounding environment would be the most simple and straightforward with this avenue (and possibly the more efficient, because of the CPU's efficient loop handling)

Would help balance the computation load between GPU and CPU. Throughout the history of my games and rendering projects, I've always tried to let the CPU handle what it is good at, and let the GPU handle what it is good at - for instance, doing all input handling, object motion, matrix transforms, BVH building, physics collision and response, on the CPU, while letting the GPU handle going through each pixel (which it easily does in parallel), sending out a ray, traversing the BVH, looking up the intersection material BRDF, sampling the light source or material BRDF, bouncing around the scene, and reporting back the final pixel color. This ray/path tracing procedure is a lot for the GPU to handle, so we can ease its burden by letting the CPU (JavaScript side) handle all of the particle motion and physics.

Cons

As you mentioned, there is a bottleneck and WebGL limitations when sending large amounts of data from the CPU to the GPU on every animation frame. The 1st time I ran into this was when I started my project that remakes the classic 80's home PC game, The Sentinel (which I call The Sentinel, 2nd Look - repo on my bio's repo list). I had around 50 game objects that need to send their 2 Vector4 (so 8 float 32s) BVH data for the shader's TLAS (Top-Level Acceleration Structure) traversal, as well as their 4x4 matrix inverses (so another 16 float 32s) to tell the ray tracer how each game object is positioned, scaled and rotated. It worked ok for 10 or so game objects, but when I approached 50, it crashed the browser page's webgl context on my mobile device, because I had exceeded the allowed number of uniforms. Luckily with the arrival of WebGL2, it allowed uniform buffer objects (UBOs) for sending large arrays of data from CPU to GPU. But still, this is a challenge to be considered.

Using the CPU for all updates will limit the total number of particles on screen, whereas GPU-only rendering would allow potentially millions of on-screen particles.

Having a CPU based particle system lends itself to rendering the final particles with the GPU as billboard quads. Having rain drops as flat quads for instance, would make realistic refraction challenging, because the drops themselves don't have any depth or 3d geometry to them - they are essentially just 2d pictures. However, I imagine this could be worked around, by the shader switching modes - so when the ray caster intersects a particle quad, it calls a raycast function that then does a quick 2nd intersection test against a real drop, or elongated sphere (ellipsoid maybe?), and then the ray interacts the way a ray tracer would, with splitting the light into reflected rays and refracted rays, depending on the angle of the ray with the rain drop surface, according to the Fresnel equations.

Versus

• GPU-based particle system (where CPU JavaScript side knows nothing of the particles)

Pros

As mentioned, almost-limitless amount of particles on screen at once, since the GPU works in parallel

Would be possible with WebGPU and compute shaders, three.js is already working on this and there are several TSL examples on threejs.org (but WebGPU support is not nearly as wide as WebGL/WebGL2, and spec might change)

Physics and updates for particles could be handled the same as rendering, in parallel

Having actual particle shapes is possible when rendering with shaders, so effects like refraction through elongated spherical water drops would add to the realism of the effect

Cons

Even if the particles could be simulated and updated via fbo, ping pong frame buffers, or compute shaders, the problem remains of how to integrate the particles with the environment scene objects. These objects could theoretically live on the GPU, but the more you stick inside the scene, the more risk you take of running out of GPU memory. It would be even worse if the scene objects were created on the CPU side and fed to a sophisticated BVH builder - then the particles on the GPU would also have to somehow know how to traverse a complex BVH, which I think would slow down the path tracing shader even more.

Having everything on the GPU (scene BVH, ray casting for each pixel, path tracing bounces loop with light/BRDF sampling, and now the entire particle system) would put a lot of stress on the GPU, especially older or mobile GPUs. Meanwhile, the CPU would be essentially starved for work, unless it had some physics to do with the main larger scene objects in the environment. There wouldn't be much balance between the GPU and CPU workloads.

[erichlof]

Hi again, part 2 (my last post was getting too long, ha)

So looking at the pros and cons of the two different approaches, I would say that the CPU-based system has a better pro to con ratio. The main pro with CPU particles is that you already have something in place.

Therefore my initial vote would be to try the CPU particle system first, even knowing that it has flaws and will present challenges near the end of the pipeline, such as CPU to GPU UBO data transfer, and requiring a possibly 2nd intersection test, if 2d billboard quads will not be sufficient for true raytraced refraction effects.

A word of warning though as you go down this path - getting the particles (many individual 2d quads) into an acceleration structure that can be quickly traversed inside the fragment shader will be challenging, and possibly even deal-breaking, if it slows down the fragment shader's ray casting loop. But being faced with this challenge might open up other ways of thinking to a better, faster solution. Off the top of my head, one optimization that you could do right away is to have the particles, especially rain, visible only on the initial camera ray cast. If a camera ray hits a rain drop, great - then reflect or refract based on Fresnel (just like you'd do with the typical glass sphere). However, if the camera ray misses all water drops on the initial ray cast, then they cease to exist. So in that case, the ray would go out into the scene and bounce around the scene objects as normal, or render the background sky if absolutely nothing was hit. I don't think the user would notice that the rain drops don't cast shadows or appear in a mirror's surface (both of which would require an expensive check for ALL rain particles on the secondary rays during the second bounce loop). Little hacks and corner-cutting tricks like this could mean the difference between this system working, and crashing the webgl context.

So this was my very long-winded way of saying, maybe take your existing particle system to the next stage - and see what happens. If it works, great, if it works slowly, we can try to think of ways to optimize it, and if it flat-out crashes, all is not lost - it just might require a slight course-correction, or slightly different approach.

Yes, please do keep me posted on your progress, as well as further questions and issues. Hope this helped somewhat- sorry I can't offer more specific detailed advice on the GPU-only choice, as I haven't made one of those myself. But if it does come to that as the only option, I will do my best to offer help or advice, and I might try to do some small GPU tests and experiments alongside your project, in order to better understand the problem and where you're trying to go.

Look forward to discussing further! Take care 🙂

-Erich

[meetSeparate]

Hi @erichlof,
I just wanted to reach out and express my sincere gratitude for your incredibly detailed and helpful response. Your breakdown of the pros and cons for the CPU-based and GPU-based particle systems was truly eye-opening, and I'm excited to take your advice and move forward with the CPU particle system first. I really appreciate the time and thought you put into sharing your insights and experiences; it's given me a much clearer path to follow.
I do have a couple of follow-up questions, if you don't mind. I've been thinking about the way light calculations are handled, and I was wondering if it's possible to avoid making a strict distinction between diffuse and specular reflections. Instead, I was considering using parameters like roughness, metallicity, and clearcoat, similar to how three.js physical materials work, to calculate light interactions in a more unified way. Do you have any thoughts or advice on whether this could be a viable approach and how I might go about implementing it?
Also, I've been working on a car showroom scene, and while I've made some progress, I'm not entirely satisfied with the realism of the car renders. My knowledge of optical calculations for different materials in the real world is a bit lacking, and I was hoping you might be able to offer some guidance. I understand you're busy, but if you could spare some time to give me pointers on how to make the materials look more authentic, I would be extremely grateful. Maybe you could even sketch out a rough design or workflow for a more realistic car showroom scene?
Once again, thank you so much for your help. I'll definitely keep you updated on my progress as I implement the particle system. Your expertise has been invaluable to me, and I look forward to discussing further and learning more from you.
Best regards,
Zhang

[meetSeparate]

I am sorry I have another question. In my project, it is necessary to initialize lighting data of multiple types (such as point light sources, directional lights, rectangle lights, etc.) and with multiple quantities. I'm confused about how to conduct lighting sampling efficiently and accurately.
I know that there are many ways of lighting sampling, like importance sampling. However, when faced with such a complex lighting scene with multiple types and quantities, I'm not sure how to allocate the sampling weights reasonably. For example, factors such as the intensity and distribution range of different types of lights vary from one another. How can these factors be taken into comprehensive consideration to determine their weights in the sampling process? Moreover, for a large number of lights of the same type, should they be sampled equally, or should differentiated sampling be carried out according to their positions in the scene or other characteristics?

[erichlof]

@meetSeparate
Hello Zhang!

I'm glad that my pros and cons breakdown was helpful to you! Best of luck with your CPU particles approach - if anything comes up that halts your progress, let me know here in this thread, and maybe we can figure it out. 🙂

About your questions on diffuse and specular separation and more realistic materials, I will try to offer some advice and point you in the right direction. Unfortunately, material models (especially specular) is a complex topic, both in the design/planning phase and the implementation phase, but I will try to navigate it below.

First we have to take a step back and define what our goals for our renderer are in terms of correctness, speed, and quality. I've mentioned this before on other issue threads here on my repo, but my first inspiration to dive into the world of path tracing was the (now old) Brigade 1,2, and 3 path tracers from Jacco Bikker and colleagues from around the years 2010-2014 (way before RTX was even an idea). In fact if you search YouTube under the channel name Sam Lapere, you'll find some great examples of what amazed me back then, and still does today, considering the hardware they had to work with. From watching their videos and reading their blog posts, their clear mission was to make path tracing interactive, and to bring it into real-time (10 to 30 fps, later with 60 fps goals for Brigade 3), and to do all this on consumer-grade hardware. Back then, denoising wasn't as prominent as it is today (maybe even non-existent), so the final results were grainy and 'salt and pepper'-ish. But if we look past that, wow, it was incredible having whole scenes, sometimes with dense triangular models like the Stanford Bunny and Dragon, Sponza, etc, all moving in real-time and interactive with user input, and path traced on a typical 2010 home gaming computer! After watching those videos many times, I was inspired to learn about path tracing, and had the dream of taking their vision further, by bringing real-time path tracing to the browser (which is even more accessible than the home pc), which meant a goal of at least 30-60 fps, on any device with a browser, even cell phones. I'm happy to say that I've achieved (or gotten close to) some of these goals, but over the last 10 years, it seems like the goal post keeps moving farther out ahead of me (ha), especially with the advent of BRDF material model research, BVH and other acceleration structures research, and most prominent - denoising (A.I.) and efficient sampling and reconstruction strategies research. As you probably already know, path tracing is a huge subject with a rich ray tracing history to discover, and cutting edge research being done at all major CS universities and corporations (NVIDIA, Intel, etc). Because of this, it's difficult to make a renderer that does all things well at the same time. Usually, trade-offs and compromises must be made, if we want to get something off the ground and working.

This brings me to trade-offs that I've had to make with my own renderer here, which will tie in with your questions (I promise!). Since I went down the real-time path (no pun intended) from the very beginning of my project, that has been the unwavering focus of the renderer all of these years. One of the first things I added to the renderer was the little Stats box in the upper-left corner of the webpage, which constantly displays the FPS (not a typical thing to add to someone's first ray tracing progam, ha), but again, speed and efficiency is the major consideration and I wanted those honest stats to be constantly displayed for myself and the users of my ray tracers. While developing a new path tracing demo/game, if those stats drop below 30 fps, I know that there is either a problem to be fixed, or optimization to be made, or a whole new approach that must be taken, in order to keep everything running smoothly on all devices. Sometimes I can do all of the above right, but adding more objects or lights to the scene starts to slow it down. That's when I realize I've hit a limit for my renderer and a limit in my understanding of the problem, and I'll usually back off a little with the stress tests, in order to make it still 30-60 fps, and call it a day.

Now bringing it around to your question specifically, one of the areas where I really have cut a lot of corners and made many trade-offs is in the specular material models and sampling of those models. This is partly by choice, but I must admit that also it is due to my own ignorance of modern BRDF material models and sampling strategies for those materials. The original Brigade renderer only had Lambert Diffuse and perfect specular mirror for their material options (no glass or transparent materials even) therefore, so did mine (in the beginning). Through examples on ShaderToy and many classic ray tracing books and the source code to Kevin Beason's brilliant SmallPT (a complete path tracer with diffuse, metal, and glass materials - all in 100 lines of C++ code!), I was able to add refraction and transparency to my material options, and then soon after that, a clearcoat material option (which has the Lambert diffuse underneath,
and a perfect specular transparent layer, like smooth glass or plastic, on top). Sadly to say, these few material options have remained much the same throughout the decade that I've been working on this project. The only refinements that I've made fairly recently are optimized sampling, and a roughness parameter (as seen in the Material_Roughness.html and the Convex_Polyhedra.html demos).

I know the material options that I have here are very limited when compared to most modern path tracers and software rendering packages (Blender Cycles, V-Ray, etc). Not too long ago I even tried to include an actual modern specular BRDF (like GGX) while following an NVIDIA RTX tutorial series for RTX beginners, but it slowed the renderer down a little too much for my comfort, and the results where really noisy and sometimes filled with 'fireflies' (annoying very bright pixels in random locations). Not satisfied, I ended up scrapping the whole thing, and returning to my simple material roots - Lambert Diffuse, Metal with roughness, Transparent with roughness, and ClearCoat. On some of the demos like Switching_Materials.html and CSG_Museum_1.html, I added a sub-surface material with the option of having a clearcoat on top (like polished jade and marble or teeth with enamel). However, my understanding of the sampling of this complex material is very limited, so that's why it doesn't appear that often in my demos. I would like to improve the correctness and realism of this material, but I don't fully understand the science or ray tracing techniques behind these fascinating materials.

Going back to goals for our renderers, I've been able to get away with only having 4 or 5 material options while keeping everything real-time on all devices, because my simple materials (which are from classic ray tracing historic times) are physically plausible and 'good enough' to render the kind of scenes that I'm interested in. But if you want to go beyond that and take it to the next level in terms of realism (like for your car scene), you pretty much have to separate the diffuse code path from the specular code path, and make the specular path an actual modern Microfacet specular BRDF. I won't go into the details of microfacet BRDFs, as there are a lot of online resources for learning about them (also under 'PBR' materials and shaders). If my understanding is correct, there are 2 parts to modern BRDFs - the evaluation functions, and the sampling functions. So for every material, let's say metal with 0.5 roughness applied, there would be an evaluation function just for metals (or non-diffuse and non-transparent materials), as well as a matching sampling function for that material. Where the traditional rasterization with PBR material shaders and actual path tracers with those same materials are very alike, is in the evaluation functions. When a triangle is rasterized (or intersected in a ray tracer), the renderer must look up the final color of the pixel that covers that material at that exact tiny location on its surface. And where traditional rasterization with PBR material shaders differs from actual path tracers with those same materials, is in the sampling functions - as traditional rasterizers do not employ sampling strategies- they quickly and efficiently look up, or 'evaluate' the material color at that point. This evaluation can be as simple as the 1-liner: albedo * N.dot(L), as it is for Lambert diffuse, or as complex as the full GGX 10-line (or more) function. This evaluation keeps things running at real-time framerates while giving a very physically plausible material look to objects, which is why it is pretty much the industry standard. If the story ended here for path tracers as well, all would be as simple and cut and dry. But because we're dealing with actual light rays that have physical interactions with real-world materials, and making them bounce around the scene, we must be able to sample these complex BRDFs, so that we know which direction the ray is supposed to go after interacting with one of these materials. Sampling functions for ray/path tracing must include the correct pdf (or probability that the ray would bounce in a certain direction) as well as the correct weighting for the final color/intensity contribution to the ray after it interacts with a material. If the correct weights for each interaction are not applied, the material will look either too dark or too bright, compared with the same material as viewed in the real world. This pdf weighting part is fairly sensitive and math-heavy, and if something doesn't end up looking right, it's not clear where to look for the problem (unless you're comfortable with sampling, importance sampling, probability, etc).

Not to worry though - there are many examples with source code out there that demonstrate how to implement a Microfacet BRDF into your path tracer. You may have heard of this, but a very popular BRDF system (and an industry standard) is the Disney principaled BRDF. This library of materials, PBR parameters like roughness and sheen, evaluation functions, and sampling functions has been used on major motion pictures from Pixar. There are several implementations of path tracers on ShaderToy that use the complete Disney BRDF system. So that would be my vote, if you wanted to give your materials that extra realism and photo-realistic appearance.

[erichlof]

(Part 2) Sorry for the lengthy posts, but as you can probably tell by now, I love all things path tracing - even if I don't fully understand some areas, I could go on and on, ha.

Anyway, to focus on your example (the car showroom scene), I would first take a look at some examples of path tracing on ShaderToy that use the Disney principaled BRDF - just do a search for that in ShaderToy's search field. See if you like the overall look and performance of it (in terms of convergence speed and noise). If so, scroll down to the evaluation functions - these are the heart of the BRDFs (they determine the color and intensity of the material at a precise point on its surface and with respect to the incoming ray's angle of approach). Then probably below those functions, you'll see several 'sample...()' functions - these are more specific to path tracers. They help determine the weights to apply for the ray's intensity as it leaves the surface, as well as determine the reflected or refracted ray's exact 3D direction after interacting with the surface. These functions rely on Monte Carlo methods as well as statistics and probability. These sampling functions will usually return an outgoing Vec3 ray direction (sampling direction), as well as a pdf weight (a simple float number) that will be later used when determining the intensity of the outgoing ray's contribution.

Going back to goals and trade-offs, if you don't mind your final image taking a while longer to converge, or things possibly running below 60 fps (if final image quality matters most), then I don't see any reason not to add microfacet specular materials to your renderer. With cars for instance, their paint is sometimes complex, with a heavy clearcoat (some with very high IoR), and fine metallic flakes just below the clearcoat surface. And the car interior must be considered as well - leather, wood, plastic, might all be in place on the inside. So when rendering these complex materials, I believe that the Disney BRDF should be able to handle everything, both on the inside and the outside, faithfully reproducing the real-world appearance, while still being efficient enough to run smoothly on most hardware.

Hope this helps, and tomorrow I will respond to your lights and light source sampling questions.

Talk to you soon! 😊
-Erich

[erichlof]

Hi again, Zhang! Part 2 (Lights)
Sorry for the delayed part 2 responding to your question about multiple lights - I needed to review something first in order to check my own understanding, before I rephrase it here and provide the link to the original source article.

Your question about how to deal with multiple lights is a really good one - but like all things in path tracing, there is never a simple answer, and we must dive deeper if we want to be satisfied with the answer(s), or at least get close to a complete answer. I should let you know up front that I myself do not fully understand the math and probability theory/techniques that are required to deal with this topic of sampling many/multiple types of lights. Most of what follows is paraphrasing from ray tracing books and articles that I have collected over the years. And, like the situation with your particles system, I haven't actually implemented a scene of multiple different types of lights for my own project here, so take the following with a grain of salt.

Since we are working with a Monte Carlo path tracer, let's quickly summarize what it's doing. To truly get a physically-accurate rendering that was indistinguishable from real life, we would have to send out countless rays (as many as there are photons in the scene) at an atomic scale, and let them bounce a near-infinite number of times, in order to capture every last detail of the scene. Since computers have a limited processing power and memory, we must employ the Monte Carlo integration method to get the best approximation humanly (and digitally) possible. Instead of sending a near-infinite amount of rays for a near-infinite number of bounces, the Monte Carlo method relies on certain aspects of statistics like the Law of Large Numbers - basically you take random samples of the sample space, add them all up, and take the average at the end. And probability and statistics holds that if you take enough random samples that are unbiased (more on this soon), eventually it will converge to the expected outcome (or as accurate an answer as we can possibly get).

This all sounds great in theory, but in practice, using Monte Carlo and uniform random sampling results in high amounts of noise. And if we are careless about how we sample the problem space, it could take an eternity for the computer to give us the correct image with all the noise removed. The Law of Large Numbers says that if the samples are unbiased and truly random, it eventually WILL converge, but it might take beyond our lifetimes - ouch!

Here is where Importance sampling comes to the rescue. This technique states that if we are more thoughtful and strategic about how we take our "random" samples, so that we sample important areas more often and less important areas less often, then not only will we get the result much faster, it will Still be as unbiased and correct as the earlier one that took eons! This is one of the facts from statistics that blew me away when I first understood what it meant. Basically it says that we are allowed to skew or bias our samples towards areas that give brighter, stronger, more meaningful results. And as long as we account for this non-uniformity or bias in our sample selection, we can essentially remove this bias from the final results. And how we account for this bias (or unfairness) in sample selection is through correctly weighting each sample. In other words, if a random sample comes from an area of the problem space that would give a large, bright, or important result, we should down-weight this kind of sample, to avoid over-representing its contribution. But occasionally if a sample happens to be taken from an area of the problem space that returns something that is less bright, or not as important, then we must up-weight this sample accordingly, to avoid under-representing this sample's contribution to the final image.

In my previous post I mentioned Jacco Bikker, who was the leader behind the Brigade series of path tracers. Not only has Jacco been a pioneer in the field of path tracing and computer graphics (with several published research papers), but he has also devoted himself to teaching at the university level, and offering free online tutorials on the subject of ray tracing and path tracing. One of his more recent tutorial series, as a warm-up, covers the basics of Monte Carlo and probability/statistics in rendering. Here's a link to 2 articles from this series:

https://jacco.ompf2.com/2019/12/11/probability-theory-for-physically-based-rendering/

https://jacco.ompf2.com/2019/12/13/probability-theory-for-physically-based-rendering-part-2/

I highly recommend reading part 1 and 2, as Jacco really knows what he's talking about, and presents it in a fun, concise article form with pictures and diagrams. To tie this in with your lights sampling questions, I'll borrow some of his scenarios and explanations, and try to paraphrase some of the main ideas.

[erichlof]

Part 3 (Lights continued)

I always like to start with a simple example that showcases the need for sample weights. Again, the examples I'm using come from Jacco's articles, but I'll put my own spin on them. 😁

Let's say we have a scene with 4 lights in it. All lights are equal power, equal shaped, and roughly equal distance from the other scene objects that are being lit by these 4 lights. When rendering a scene like this, path tracing requires that after you send out an initial camera ray from each pixel, if a ray hits an object other than a light - a diffuse sphere for example, that you must randomly select a direction (path) for that ray to continue bouncing around the scene, gathering light and color as it interacts with other objects. Often we care about sending rays towards the bright lights in the scene, as these give us important lighting information for all the objects. But we need all 4 lights' contributions in order to properly light the scene. Now a recursive 80's style ray tracer would simply loop over all 4 lights, one at a time, gathering and adding up all the contributions as it goes. But this tends to be slow, especially as the number of lights grows. Enter path tracing- now we can just randomly choose 1 out of the 4 possible lights, and then just keep randomly choosing for that pixel, over and over. This will produce quite a bit of noise, but it will eventually converge on a smooth result.

There's only one problem though- our image is much darker than the ray tracer's correct result! The reason is that we only sampled 1 light's contribution at each intersection along the ray's journey. To the poor 3 other lights that didn't get chosen, it's like they don't even exist, and their contributions don't matter. To fix this problem, we need to weight the light that we Do sample, so that the total brightness output of all 4 lights is taken into consideration at each juncture of the ray's path. The most straightforward (and still correct) fix is to simply multiply each light sample by 4. But why multiply, and why by the number 4?

Just to keep the math simple, let's say that every light has a brightness of 1.0. Since global light intensity is additive, we can say that the scene objects are being lit by a total global light power that equals: 1.0 + 1.0 + 1.0 +1.0 = 4.0, or 4. In probability, we like to have ranges like 0% to 100%, or more formally, 0.0 to 1.0 (which is 'normalized'). So to get each light's contribution to the global intensity, we simply divide each light's output (1.0) by the global total output (4.0), or 1/4, or 0.25. Now we can see why our naive implementation was too dark - because every time we took a light sample, we only considered 0.25 or 25% of the actual global total light output. Therefore our scene would be 1/4th of the brightness of the actual scene brightness.

Will continue in part 4 soon (have to get some sleep, ha)... 😊

[erichlof]

Part 4 (Lights continued)
In order to compensate for the missing light, Importance sampling says that after we take a random sample from one of the lights, we have to divide its contribution by the 'pdf' (probability density function), or likelihood that this sample could have been truly randomly chosen (without Importance sampling). Recall that the likelihood of choosing 1 light out of 4, since all aspects of the lights were similar to each other, was 1/4, or 0.25 (a fair 25% chance of being chosen). So we take our randomly selected sample, which returned a result of 1.0 light output power, and divide this 1.0 by the chance of it being chosen, which is 0.25. So 1.0 / 0.25 , which results in a final correct answer of 4.0. And this is why when using a number N of similar lights, we can get away with just choosing 1 light on each pass, sample its contribution, divide it by pdf, which is 1/N, which is the same as multiplying by N, the number of lights. So final contribution = sample / (1/N) or more simply, contribution = sample * N.

Now this '* N' only works if you have a number N of lights that all have similar properties to each other. The more differences there are in the properties of each light, like size, power, distance from subjects, etc., the more and more incorrect the importance sampling results will be. For instance, in my Geometry_Showcase.html demo, I have 3 large spherical lights, but they have slightly different sizes, intensity, and distance from the scene subjects, but the differences are minimal. Therefore, I can get away with contribution = sample * 3, because the 3 lights are close enough to each other to not notice errors. But still, there are slight mathematically-provable errors in my final images - and I'm ok with it. If I were to do all the legwork and calculate the exact probability of selecting each light, the final result would most likely be too close to tell the difference. 2 other scenes I use this technique are the TheRenderingEquation.html (which has 3 similar quad lights) and my AntiGravityPool.html game that has 8 smaller spherical lights (1 in each corner of a large room).

Just to review before moving on to sampling and weighting different types of lights: we figure out the likelihood that a random bounced ray would hit each light ( the bounced ray could go in any direction above the surface normal - a surface normal-oriented hemisphere of all possible outgoing directions). In the most simple case we have 4 (or N) lights with identical or very similar characteristics, so the probability of hitting 1 out of N is the same for every light, or 1 / N. This is our pdf that we must divide each random sample by to account for all the other lights that were not randomly chosen. This is were my knowledge becomes shaky, but I think if you normalize all of the pdfs for each light so that the probability of being selected is between 0 and 1 (0.25 in our example), then add them all up side by side, you get the cdf (cumulative distribution function). The cumulative distribution function needs to add to 1 in the end. So in our example, we list all the probabilities back to back and they should add to 1.0 (if they were correctly normalized):
0.25 + 0.25 + 0.25 + 0.25 = 1.0. One other thing to note is that no matter how small a chance there is of randomly picking a very tiny LED light for example, it must be included in the cdf, otherwise we are discounting its contribution. Now you can imagine if we Did randomly select this tiny light for sampling on one of many passes, you would have to divide by the small probability, let's say 0.001, so that would mean sample contribution = sample / 0.001, or, contribution = sample * 1000. In this case, it would produce a pesky bright firefly pixel that is surrounded by normal darker pixels - a real problem for getting a smooth image, even with a denoiser!

[erichlof]

Part 5 (Lights end, finally! 😅)

Before we get into multiple lights sampling, I wanted to make a correction to my earlier post statements - I had thought that the cdf needed to add up to 1, but upon further re-reading of Jacco's articles that I linked to, it is the pdf that has to add up to 1.0, not the cdf. The pdf is the distribution of the likelihood that a random sample will take on a certain value. So if we took a bunch of samples of tree heights from a forest, the pdf (or distribution of possible sample values) would end up looking like a bell curve, or more formally, a normal distribution. In other words, most of the height sample measurements you take would end up near the middle, or average height, while a couple of outliers to the left of center would be shorter than usual, and a couple outliers to the right of center would be unusually tall. Near the middle of the pdf, you would have probabilities around 0.6, while at the left and right edges, the probabilities would be much lower, like 0.05 or something. The point is, when you add up all these probabilities, they all must equal 1.0. BTW, this shape, or bell curve shape, relates to sampling diffuse surfaces, as the cosine distribution (or the result of N.L) looks the same, it's just in 3D instead of 2D. Most of the light power coming into the surface is from around the surface normal direction (where N.L is higher value), while less light intensity is received near the left and right (and front and back in 3D) edges (where N.L is lower value). Since we know the exact shape of the pdf beforehand, we can importance sample directly from it, and get the fastest possible results, that are still correct and unbiased! Whenever you see a function with sampleCosHemisphere or something like that, this is what's going on under the hood.

This ties in nicely with sampling multiple different types of lights (finally, we're getting to your question, ha!). Again, I'm paraphrasing from Jacco's articles, but the situation is a little more complicated now, because we do not know what the pdf for all your lights is, or rather, it doesn't have an easy distribution, like a normal distribution or bell curve that is well documented and easily importance sampled. So, according to the linked articles, we must make our own pdf from scratch. Once we have a pdf (that adds up to 1.0 remember), we can generate random numbers and depending on where they fall on your custom pdf shape chart, you can take a sample from that area (or range) of the curve (a specific light source in your case), and then divide by the probability (i.e. 0.3, 0.1, 0.6 etc) of that light being selected uniformly randomly, in order to account for all the other lights that didn't get sampled on this current pass through the ray's bounces loop.

However, in order to get the likelihood (or probability) that a light source would be hit by an outgoing random diffuse ray, Jacco states that we must first construct it's Solid Angle, or apparent size projected onto a unit hemisphere above the surface normal. To help picture this, I like to pretend I'm a tiny little bug (I know - fun right?!) on the surface that is being lit by all the lights in the scene. When I look up I see a glass dome ceiling around me - this is the unit hemisphere, with a radius of 1.0 in all outgoing directions. Now if I look toward one of your lights that you've placed above me, how much apparant surface area does it take up of my dome ceiling? If it is a small LED light, maybe it only takes up 1 degree squared (width from left to right times the height from top to bottom, and why it is a 'solid' angle - it has 3D thickness). But if it is a large spherical light that is really close to my surface (and to my tiny face, lol), it will take up a much larger portion of my little dome ceiling. Taking it further, if the light source was the entire sky, it would take nearly 100% (or 1.0) of the small hemisphere dome above my head. If you want the exact math to calculate the Solid Angle (SA), please refer to Jacco's 2nd article, where he gives the exact formula. Once you have this Solid angle, or 'SA' in his articles, you can divide 1.0 by it and get the pdf (i believe, check the end of the 2nd article to make sure) - so the pdf would be 1.0 / SA. If you have a bunch of lights, no matter what size or shape they are, as long as you can get the exact (or approximate) apparent size on the dome ceiling, as viewed from the small unit hemisphere, then you can get the pdf for all lights. Then it's just a matter of lining them up and making sure all the newly calculated pdfs add up to 1.0.

It's a shame that Jacco ends his article here - I would have liked if he would have continued with how to actually draw samples from a random number generator. Again, I'm at the limits of my knowledge at this point, but I believe this is where a cdf could be useful. But if you didn't want to construct or use a cdf, I believe you can just get a random number in the range from 0.0 to 1.0 (like we do all the time), and depending on which section (or range slice, or bin) this random number falls into, you select that particular light, sample it by sending a ray toward a random point on its surface, then, only if it is visible (no other objects are in your way towards the light), take its light intensity and color and divide this by the unique pdf for this particular light. So vec3 result = vec3 lightColor * lightPower / (1.0 / SA). To clean it up a bit, you can just do result = vec3 lightColor * lightPower * SA . But again, please check the article, as I might have misinterpreted something.

As far as shape of lights goes, I recommend starting out with rectangular, or quad, lights. This is because the SA for a rectangle is easily calculated with Jacco's SA equation, as we know the width and height of the light sources beforehand. Doing this for spheres will be required eventually, but the geometry is a little trickier, because we are dealing with a circle shape and radius and cos and all that kind of stuff to get the light's apparent viewable area on the small unit hemisphere dome ceiling. If I'm not mistaken, my function (that I got from one of Peter Shirley's great ray tracing books) called sampleSphereLight() does this SA, or Solid Angle, calculation towards the beginning of the function. Again, i don't fully understand the math and geometry behind all of this, I just know how to call the function and make good use of it! 😅

So to wrap up, you may end up having a pdf that has many slices or bins (as many as there are lights in your scene, or N lights/bins) - let's say 30. The number doesn't matter, as long as all the SA pdfs for each of those lights all add up to 1.0 total probability. If you can understand the cdf step that follows, unlike me, then great - you can importance sample the lights pdf directly and efficiently. But just to get started, and if you don't end up making a cdf for the pdf, you can just get away with uniformly sampling your custom pdf area curve, and dividing each sample by it's pdf to weight that sample accordingly. My only word of caution is that you might end up with a lot of noise for the first 20 or 30 frames, but once it starts progressively rendering and averaging the results, it should quickly smooth the image out after a few seconds. And finally, you can take comfort in knowing that you are efficiently sampling many different kinds of lights, and doing so in the most correct and unbiased way we know how!

Whoo, this was an epic series of posts, but I thought that your question deserved a deep dive into this tricky subject. Again, sorry that I couldn't be more specific towards the end about the actual implementation - as I have not implemented a many lights system yet myself. But regardless, I hope this helps get you started, and don't hesitate to ask more questions about what I posted (as it was a lot, and I might have missed some things), or ask further questions beyond what has been discussed.

Good luck with the many lights sampling! 😀
-Erich

[meetSeparate]

Hi Erich,

Wow, I’m absolutely blown away by the depth and detail of your response. I can’t thank you enough for taking the time to write such a thorough and insightful breakdown of these complex topics. It’s clear how much thought and care you put into this reply, and I feel incredibly lucky to have access to your vast knowledge and experience.

Your explanation of the trade-offs involved in material modeling and rendering goals really resonated with me, and I’m inspired by your journey and dedication to real-time path tracing. The way you connected the history of path tracing, your own personal goals, and the challenges of balancing realism, correctness, and speed really puts everything into perspective for me. It’s both humbling and motivating to see how you’ve approached these challenges with such determination and creativity.

Your advice on diffuse and specular separation, microfacet BRDFs, and importance sampling has not only answered my questions but also opened my eyes to so many new possibilities. I especially appreciate how you took the time to explain concepts like the relationship between pdfs, cdfs, and solid angles in the context of sampling multiple lights. I feel like I’ve gained a much better understanding of the theory behind these techniques, even if I have a long way to go in implementing them myself.

I also want to say how much I admire your honesty in sharing the trade-offs and limitations you’ve faced in your work. It’s a great reminder that path tracing is an ever-evolving field and that it’s okay to make compromises to achieve specific goals. Hearing about your pragmatic approach to real-time rendering has given me a lot to think about as I refine my own renderer.

Rest assured, I’ll be diving into the resources and advice you’ve shared. I’m especially excited to explore microfacet BRDFs, the Disney principled BRDF, and the importance sampling methods you described. I know it won’t be easy, but your explanation has given me a clear direction and the confidence to tackle these challenges step by step.

Once again, thank you so much for your generous guidance and encouragement. Your passion for path tracing is truly contagious, and it’s inspired me to strive for greater heights in my own work. I’ll make sure to keep you updated on my progress, and if I run into any roadblocks, I hope I can reach out to you for further advice.

You’ve made a huge impact on my understanding and approach to rendering—thank you for being such an incredible mentor and role model. 😊

Zhang