Multiple rendering of a scene with results accumulation

[aantg]

Hello.

I need to make multiple rendering of a scene with changes in shapes' transformations and accumulate resulting radiance. See my naive python code for this below.

My python code

import mitsuba as mi
import drjit as dr
import numpy as np

from icosphere import icosphere

mi.set_variant('cuda_ad_mono')


def load_scene(lookat, angle):
    return mi.load_dict({
        'type': 'scene',
        'light': {
            'type': 'directional',
            'direction': [0, 0, 1],
            'irradiance': {
                'type': 'uniform',
                'value': 1.0
            }
        },
        'obj_shape_group': {
            'type': 'shapegroup',
            'cyl': {
                'type': 'cylinder',
                'radius': 0.7,
                'p0': [0, 0, -0.5],
                'p1': [0, 0, 0.5],
                'bsdf': {
                    'type': 'roughconductor',
                    'eta': 3.8911,
                    'k': 0.2456,
                    'alpha': 0.025,
                },
            },
            'cap-z': {
                'type': 'disk',
                'to_world': mi.ScalarTransform4f().translate([0, 0, -0.5]).scale([0.7, 0.7, 1]),
                'flip_normals': True,
                'bsdf': {
                    'type': 'roughconductor',
                    'eta': 1.5,
                    'k': 7.3,
                    'alpha': 0.05
                },
            },
            'cap+z': {
                'type': 'disk',
                'to_world': mi.ScalarTransform4f().translate([0, 0, 0.5]).scale([0.7, 0.7, 1]),
                'bsdf': {
                    'type': 'roughconductor',
                    'eta': 1.5,
                    'k': 7.3,
                    'alpha': 0.05
                },
            },
            'nzl': {
                'type': 'bsplinecurve',
                'filename': 'nzl.dat',
                'to_world': mi.ScalarTransform4f().translate([0, 0, -0.5]),
                'bsdf_adapter': {
                    'type': 'twosided',
                    'bsdf': {
                        'type': 'roughconductor',
                        'eta': 1.1151,
                        'k': 6.8227,
                        'alpha': 0.09
                    },
                },
            },
            'rcv': {
                'type': 'ply',
                'filename': 'rcv.ply',
                'to_world': mi.ScalarTransform4f().translate([0, 0, 0.5]).rotate(axis=[0, 0, 1], angle=angle),
                'bsdf_adapter': {
                    'type': 'twosided',
                    'bsdf': {
                        'type': 'diffuse',
                        'reflectance': {
                            'type': 'uniform',
                            'value': 0.8,
                        },
                    },
                },
            },
        },
        'object': {
            'type': 'instance',
            'to_world': mi.ScalarTransform4f().look_at(
                origin=[0, 0, 0],
                target=lookat,
                up=[0, 1, 0]),
            'shapegroup': {
                'type': 'ref',
                'id': 'obj_shape_group'
            },
        },
        'integrator': {
            'type': 'path'
        },
    })

virtual_observer = {
    'type': 'orthographic',
    'to_world': mi.ScalarTransform4f().look_at(
        origin=[0, 10, 0],
        target=[0, 0, 0],
        up=[0, 0, 1]) @ mi.ScalarTransform4f().scale([1.7, 1.7, 1]),
    'film': {
        'type': 'hdrfilm',
        'pixel_format': 'luminance',
        'width': 768,
        'height': 768,
    },
    'sampler': {'type': 'multijitter', 'sample_count': 16}
}

spp = 16

# 12, 42, 92, 162, 252, 362, 492, 642, 812, 1002, 1212, 1442, 1692, 1962,
# 2252, 2562, 2892, 3242, 3612, 4002, 4412, 4842, 5292, 5762, 6252, 6762,
# 7292, 7842, 8412, 9002, 9612, 10242...
orientations, _ = icosphere(nr_verts=1692)

rng = np.random.default_rng()
angles = rng.uniform(low=0, high=360, size=len(orientations))

sensor = mi.load_dict(virtual_observer)

out = dr.zeros(mi.TensorXf, shape=(resolution, resolution, 1))

for angle, orientation in zip(angles, orientations):
    scene = load_scene(orientation, angle)
    out += mi.render(scene, sensor=sensor, spp=spp)
    dr.eval(out)

The problem is my for loop stucks at 1638 iteration. It makes all iterations if I fix orientation (pass the same parameter) in load_scene() function call, like

scene = load_scene(orientations[10], angle)

or if I remove (comment-out) cyl or nzl or rcv or both cap-z, cap+z shapes.

My rcv.ply file contains surface with only 80 triangles and 82 vertices. And nzl.dat defines one simple curve:

0.0 0.0 0.0 0.23
0.0 0.0 0.0 0.23
0.0 0.0 0.0 0.23
0.0 0.0 0.0 0.23
0.0 0.0 -0.37 0.1
0.0 0.0 -0.37 0.1
0.0 0.0 -0.37 0.1
0.0 0.0 -0.37 0.1
0.0 0.0 -0.73 0.23
0.0 0.0 -0.73 0.23
0.0 0.0 -0.73 0.23
0.0 0.0 -0.73 0.23

I don't get what is going on, why loop stucks and how to overcome this. Doesn't seem like memory issue as it only takes about 2.5 of 4 GB of dedicated VRAM. So, I need help. Thx.

[merlinND]

Hello @aantg,

Indeed, it's weird that it would get stuck after 1600+ iterations.
I think the best way to figure it out is to compile in debug mode, run your script under gdb and let it run until it gets stuck. Then, press ctrl + c to drop into the debugger and print the stack trace.

One random idea (unlikely to work) would be to pass parallel=False to mi.load_dict().

[aantg]

I haven't figured out debugging python with gdb, but I added dr.flush_kernel_cache() function to the end of the loop and now it doesn't hang. The question now is whether this is a normal solution, but most importantly, is it normal that such a problem even arose in such a seemingly simple case?

[njroussel]

In general, unless necessary, it's always better to edit a scene using mi.traverse() and params.update() rather than reloading it from scratch. My guess is that in your current code, every call to load_scene generates a new kernel that gets put into the cache. In theory this is harmless, but in practice we don't have an eviction policy for our cache, so maybe you're running out of host memory (RAM) ?

[aantg]

I need to change to_world parameter of one member of my shape group which is PLY mesh. Unfortunately, mi.traverse() doesn't expose this parameter in my case, it exposes faces, vertex_positions, vertex_normals. Manipulating them is more complicated IMO.

I monitored memory consumption via Windows Task Manager and didn't observe issues. I have 64 Gb of physical RAM and 4 Gb of video memory.

[ziyi-zhang]

Hi @aantg,

Orthogonal to the hanging issue, freezing kernels should save us a lot of time when rendering a scene thousands of times.
Here is a benchmark I had before. It shows how to update a scene with parameters, and I measured a speedup of ~80x.

Rendering with kernel freezing False took 279.75 seconds, 9.61 seconds in kernels.
Rendering with kernel freezing True took 10.56 seconds, 0.11 seconds in kernels.

import mitsuba as mi
import drjit as dr
import numpy as np
import time

mi.set_variant('cuda_ad_rgb')

freezing = True
dr.set_flag(dr.JitFlag.KernelHistory, True)
dr.set_flag(dr.JitFlag.KernelFreezing, freezing)

class kernel_timer:
    def start(self, name):
        self.name = name
        dr.kernel_history_clear()
        self.start_time = time.time()

    def stop(self):
        dr.eval()
        dr.sync_thread()
        hist = dr.kernel_history()
        kernel_time = sum(
            item.get('execution_time', 0.0) for item in hist)
        elapsed_time = time.time() - self.start_time
        print(f"{self.name} took {elapsed_time:.2f} seconds," +
              f" {kernel_time / 1e3:.2f} seconds in kernels.")


@dr.freeze
def render_once(image_res: mi.TensorXf):
    image = mi.render(scene, spp=64)

    image_res += image
    dr.eval(image_res)


# --------------------------------------------------------------


scene = mi.load_dict(mi.cornell_box())
params = mi.traverse(scene)
key = 'light.to_world'
init_param = mi.Transform4f(params[key])
image_res = dr.zeros(mi.TensorXf, shape=(256, 256, 3))

timer = kernel_timer()
timer.start(f'Rendering with kernel freezing {freezing}   ')

n_repeats = 1024
for theta in np.linspace(0, 2 * np.pi, n_repeats, endpoint=False):
    params[key] = (mi.Transform4f().translate(
        mi.ScalarVector3f(np.cos(theta) * 0.5, 0, np.sin(theta) * 0.5)) @
        init_param
    )
    params.update()

    render_once(image_res)

image_res /= n_repeats
mi.Bitmap(image_res).write('average_cbox.exr')
timer.stop()

[aantg]

Wanted to try, but got

AttributeError: type object 'JitFlag' has no attribute 'KernelFreezing'

with this code.

I use Mitsuba 3.6.4 and Dr.Jit 1.0.5. Didn't find this flag in Dr.Jit reference.

[lnuic]

Dear @aantg,
The KernelFreezing feature was recently merged into the master branch, but it hasn't been included in an official package release yet.
To use this feature, you'll need to compile Mitsuba locally from the latest source.

[aantg]

Got it, thank you.

[aantg]

Hello again.

After all, I made scene modification via mi.traverse() and params.update() as @njroussel suggested. Now my loops work fine, so I close this thread.