shape_index in AOV integrator

[ttzytt]

I'm currently learning TAA algorithms and want to use Mitsuba to generate some testing scenes for my implementation. Since reprojections in TAA requires the motion of objects in different pixel, it is necessary to know the objects (shapes in Mitsuba) at different pixels.

I then found that AOV integrator can output this if shape_index is set when constructing it. However, the output shape_index seems unrelated to the order of shapes in scene XML files.

For example, the following txt file is the shape_index output by the AOV integrator for cbox example scene.

obj_id.txt

It seems like the value of these ids is greater than the shapes in the Cornell box scene.

So I'd like to ask if there is any way to obtain shapes in a scene by the shape_index generated by AOV integrator.

The following is the code I use AOV integrator to generate shape_index map, not sure if this is the correct way to use it:

import mitsuba as mi
import numpy as np
mi.set_variant('cuda_ad_rgb')
cornel_box_scene : mi.Scene = mi.load_file('./scenes/cbox.xml')
aov_int = mi.load_dict({
    'type': 'aov',
    'aovs': 'shape_index:shape_index',
    'integrator': {
        'type': 'path',
    }
})
aov_img = mi.render(cornel_box_scene, spp=256, integrator=aov_int)
plt.axis('off')
plt.imshow(aov_img[:,:,3]) # show the shape_index map

def to_np_arr(img : mi.TensorXf, dtype : np.dtype=float):
    ret = np.array(img.array, dtype=dtype)
    ret = ret.reshape(img.shape)
    return ret

obj_id_img = to_np_arr(aov_img[:,:,3])
np.savetxt('obj_id.txt', np.rint(obj_id_img), fmt='%d')
file_array = np.loadtxt('obj_id.txt')
plt.imshow(file_array)

[rtabbara]

Hi @ttzytt,

There was a previous discussion here similar to the issue you're encountering. The summary is that across render calls, the shape_index wouldn't be unique, so it wouldn't be suitable as means of a lookup index.

A shape has an id property that could be set, so one possibility is to create a simple integrator that does an initial cast from the sensor, finds the associated hit shape and then caches the world transform based on id. At the next iteration the cached transform would be your previous one and so you could then determine the velocity.

[jensenzhoujh]

I find that substantiating the objects in the order separately with mi.load_dict() before constructing the scene seems to give me the expected index order I want. Basically, instead of doing

scene = mi.load_dict({
        'object1': {'type': ...},
        'object2': {'type': ...},
}),

do the following instead:

object1 = mi.load_dict({'type': ...})
object2 = mi.load_dict({'type': ...})
scene = mi.load_dict({
        'object1': object1,
        'object2': object2,
})

such that the index of object1 should be always smaller than object2.