Register a custom plugin for AOV buffer?

[baravdish]

Hello

I am currently implementing a custom plugin, specifically it aims to produce an estimated variance map or sample map. However, after implementing the new AOV type and registering it in the aov.cpp file (src/integrators/aov.cpp) I get error when I run this test function in Python:

def test06_variance_aov(variants_all_rgb):
    """Test that variance AOV produces somewhat expected results"""
    scene = mi.load_file(find_resource('resources/data/scenes/cbox/cbox.xml'))

    integrator = mi.load_dict({
        'type': 'aov',
        'aovs': 'variance:variance',
        'my_image': {
             'type': 'moment',  
             'nested': {
                 'type': 'path'
             }
        }
    })

    # Render with different sample counts
    spp_low = 16
    spp_high = 64

    # Render with low samples (should have higher variance)
    image_low = integrator.render(scene, seed=0, spp=spp_low)
    variance_low = dr.mean(image_low[:,:,-3:])  # Last 3 channels are variance

    # Render with high samples (should have lower variance)
    image_high = integrator.render(scene, seed=0, spp=spp_high)
    variance_high = dr.mean(image_high[:,:,-3:])

    # Higher sample count should reduce variance
    assert dr.all(variance_high < variance_low), \
        f"Variance did not decrease with more samples: {variance_high} >= {variance_low}"

The pytest gives me this output:

test_aov.py::test06_variance_aov[scalar_rgb] FAILED                                                                                                                                                          [ 33%]
test_aov.py::test06_variance_aov[cuda_ad_rgb] FAILED                                                                                                                                                         [ 66%]
test_aov.py::test06_variance_aov[llvm_ad_rgb] FAILED                                                                                                                                                         [100%]

===================================================================================================== FAILURES =====================================================================================================
_________________________________________________________________________________________ test06_variance_aov[scalar_rgb] __________________________________________________________________________________________

variants_all_rgb = 'scalar_rgb'

    def test06_variance_aov(variants_all_rgb):
        """Test that variance AOV produces expected results"""
        scene = mi.load_file(find_resource('resources/data/scenes/cbox/cbox.xml'))
    
>       integrator = mi.load_dict({
            'type': 'aov',
            'aovs': 'var:variance',
            'my_image': {
                'type': 'path',  
                 'type': 'moment',  
                 'nested': {
                     'type': 'path'
                 }
            }
        })
E       RuntimeError: ​[AOVIntegrator] Invalid AOV type "variance"!

test_aov.py:184: RuntimeError

This is how I register it among the rest of the AOV types, see last row:

    enum class Type {
        Albedo,
        Depth,
        Position,
        UV,
        GeometricNormal,
        ShadingNormal,
        dPdU,
        dPdV,
        dUVdx,
        dUVdy,
        PrimIndex,
        ShapeIndex,
        IntegratorRGBA,
        Variance // Added the new AOV type here
    };

And:

            if (item[1] == "albedo") {
                m_aov_types.push_back(Type::Albedo);
                m_aov_names.push_back(item[0] + ".R");
                m_aov_names.push_back(item[0] + ".G");
                m_aov_names.push_back(item[0] + ".B");
                ...
                ...
            } else if (item[1] == "variance") {
                m_aov_types.push_back(Type::Variance);
                m_aov_names.push_back(item[0] + ".R"); // reads: .push_back("Variance.R")
                m_aov_names.push_back(item[0] + ".G");
                m_aov_names.push_back(item[0] + ".B");
            }
            else {
                Throw("Invalid AOV type \"%s\"!", item[1]);
            }

And finally:

        for (size_t i = 0; i < m_aov_types.size(); ++i) {
            switch (m_aov_types[i]) {
                case Type::Albedo: {
                 ...
                 ...
                case Type::Variance: {
                    Log(Info, "Processing variance AOV...");

I also left these untouched at the bottom of the aov.cpp:

MI_IMPLEMENT_CLASS_VARIANT(AOVIntegrator, SamplingIntegrator)
MI_EXPORT_PLUGIN(AOVIntegrator, "AOV integrator");
NAMESPACE_END(mitsuba)

Anyone knows what I missed? Perhaps I missed to register it for Python bindings specifically? Thanks in advance! :)

[merlinND]

Hello @baravdish,

Since the error message seems to come from Throw("Invalid AOV type \"%s\"!", item[1]);, are you sure that your new else if case is reached?

1. Is your modification compiled and picked up in the Python environment when running the test? It could be confusing especially if you have multiple Mitsuba versions on your Python search path, e.g. from pip install mitsuba.
2. If it is correctly picked up, you can use a debugger to figure out why this error is thrown instead of going into the new else if case that you added.

[baravdish]

I am sorry if I am not fully following you. Perhaps I have misunderstood the Python bindings. I have installed mistuba via pip install mitsuba, I have also compiled the code base with the newly added feature. Does the binding happens automatically? Or do I have to do a manual step in-between? For example:

Steps
0. I have installed mitsuba pip install mitsuba in an isolated python environment.

1. In the aov.cpp I add my new AOV type, and other C++ code.
2. I build the system (Release or Debug)
   // Here I am not sure if I am missing something
3. I run...pytest...?

So I am unsure about step 2 --> 3. Does the C++ code automatically expose to Python?

Thanks for the fast response!

[merlinND]

Hello @baravdish,

pip install mitsuba doesn't just install the bindings, but the full library.
Likewise, building from source builds the whole thing, including the bindings.

Since you want to make a modification to the source code, you cannot use the pre-built pip package. So you should first uninstall it: pip uninstall mitsuba drjit
Then, build from source with your modification to aov.cpp.
Then, source the setpath.sh script in the build folder. Then you can use Mitsuba with your modification.

These are the full instructions:
https://mitsuba.readthedocs.io/en/stable/src/developer_guide/compiling.html

[baravdish]

Thanks! Got it! Yes I felt something was off with the way I was doing it. I am new to C++ <=> Python interface interactions.