Deep copying objects in C++ API

[xacond00]

Hi, I wanted to do a modification of Batch sensor, where you would only supply one sensor along with a parametrized position in a xml file (ie. automatically expanding one sensor into bunch of sensors on a line with equidistant spacing).

However I have trouble finding, how to make a proper deep copy of an object ?
In the constructor from scene parameters, you pretty much only get a list of already instantiated objects wrapped in ref, which is then cast into the correct class instance. Something like this:

for (auto [unused, o] : props.objects()) {
            ref<Base> sensor(dynamic_cast<Base *>(o.get()));
            if (sensor) {
                m_sensors.push_back(sensor);
}

And I would like to take the sensor instance and copy it N times, each as a new unique sensor, but with the same parameters.
From what I've seen, there's really no special behavior for copy constructors, and it appears to be a huge problem, as the Sensor instance in particular stores ref, which itself has a copy constructor that just increments reference counter, instead of creating a new object.

Is there any clean way of doing this ? I'm quite lost.

[rtabbara]

Hi @xacond00 ,

I think you can define a new "shell" sensor type and leverage the std::vector<ref<Object>> Object::expand() mechanism to achieve what you're after. When loading a dictionary or file, this function allows an instantiated object to be expanded into sub-objects.

template <typename Float, typename Spectrum>
class RepeatedSensor final : public RepeatedSensor<Float, Spectrum> {
public:
    MI_IMPORT_BASE(RepeatedSensor, m_world)
    MI_IMPORT_TYPES(Scene, Shape)

    RepeatedSensor(const Properties &props) : Base(props), m_props(props) {
        // Keep a copy of properties for later use in expand
        // Query properties to find out how many copies we want?
    }

    std::vector<ref<Object>> expand() const override {
        std::vector<ref<Object>> sensors;
        for (size_t i = 0, i < N; ++i) {
            // You might need some additional properties to specify the repeated sensor type
            // Properties is a bit pedantic with regards to what has or hasn't been parsed so some care might be needed here
            sensors.push_back(new PerspectiveCamera(props));
        }

        return sensors;
    }
protected:
    Properties m_props;
};

MI_IMPLEMENT_CLASS_VARIANT(RepeatedSensor, Sensor)
MI_EXPORT_PLUGIN(RepeatedSensor, "RepeatedSensor")

and then within your xml for instance you would have

<sensor type="batch">
    <sensor name="my_sensor" type="repeated">
        <string name="sensor_type" value="perspective"/>
        <integer name="count" value="32"/>
        ... other properties
    </sensor>
</sensor>

I've likely omitted some steps here, but hopefully this is enough to give you the general idea. I would also recommend looking at the implementation of the Distant radiancemeter sensor distant.cpp which similarly uses expand.

[xacond00]

Hmm, but when is the expand() ever called ? Didn't notice it being used anywhere.
Also, if I understand it correctly; doesn't the params.objects() pretty much just construct each object on the fly from the parameters ?
Meaning, shouldn't I be able to clone the nested sensor like this ?

m_sensor_cnt     = props.get<uint32_t>("sensor_cnt", 1); // Load number of sensors first
for(int i = 0; i < m_sensor_cnt; i++){ // For each sensor copy
    auto tmp = props; // Create copy of props
    auto objects = tmp.objects(); // Turn the prop copy into list of objects
    if(objects.size() < 1 || objects.size() > 1) { Throw("Exactly one sensor definition !")};
    for (auto [unused, o] : objects) {
                ref<Base> sensor(dynamic_cast<Base *>(o.get()));
                ref<Shape> shape(dynamic_cast<Shape *>(o.get()));
    
                if (sensor) {
                    m_sensors.push_back(sensor);
                } else if (shape) {
                    if (shape->is_sensor())
                        m_sensors.push_back(shape->sensor());
                    else
                        Throw("BatchSensor: shapes can only be specified as "
                              "children if a sensor is associated with them!");
                }
     }
 }

I'm sure there's much cleaner way to do this though...
However the parameters parsing seems like black magic to me, and I'm completely clueless on how it works, and how on Earth the abstract objects get turned into a concrete class instances.
When is the type of object determined when parsing ? I understand that dynamic_cast does the upcasting, but where does the construction with given specific parameters for that type happen in this case ? It's not as if we are calling ie. new PerspectiveSensor(props) anywhere.

[rtabbara]

Hmm, but when is the expand() ever called ? Didn't notice it being used anywhere.

It's used both in xml_v.cpp and xml.cpp during the load_dict and load_file calls respectively.

I'm sure there's much cleaner way to do this though...

Yes, by overriding expand. It's exactly designed for this use case. When we initially parse our dictionary or file, we first track individual properties for each plugin instance (excluding any objects). Then during actual instantiation, inner plugins have to first be created and are then given to the parent's properties as objects (props.set_object). However, the expand call gives you an opportunity to instead provide the parent properties with a replacement list of objects instead. So for your case, when you actually go to instantiate your batch sensor, you will have replaced your shell object with the true list of sensors.

but where does the construction with given specific parameters for that type happen in this case ?

There is a Class type that provides run-time type information to associate plugin names that are fetched during parsing to the actual concrete types.

Hopefully that helps, but otherwise it might be worth just attaching a debugger and stepping through to see all the individual steps of plugin instantiation.

[xacond00]

<sensor type="batch">
   <sensor name="my_sensor" type="repeated">
       <string name="sensor_type" value="perspective"/>
       <integer name="count" value="32"/>
       ... other properties
   </sensor>
</sensor>

In this way the sensor type only becomes a tag, instead of a object, but where do I provide the parameters for that particular nested sensor that I am constructing then ? Meaning something like lens radius for ThinLens. Right in the other properties ?

And to construct the particular instance in expand(), I then have to check against each possible string "sensor_type", to construct the correct object, right ? Or is there way to automate this / what happens when somebody adds a new plugin sensor type ?
Because of this I wanted my implementation to just take some xml Sensor definition and literally load it N times, as it already would contain only correct and relevant parameters in its body.

Hopefully that helps, but otherwise it might be worth just attaching a debugger and stepping through to see all the individual steps of plugin instantiation.

I would for sure do that, but Mitsuba barely works with any IDE as is, let alone trying to make a debugger work with that... Codium with Cmake extension gets completely flabbergasted by the super weird structure, and Clangd only half works. CLion too straight up breaks down.

Anyway, many thanks for everything so far.

[rtabbara]

Hi @xacond00 ,

I realised it's a bit more involved so I had a quick go at implementing what I had suggested:

src/sensors/repeater.cpp

#include <mitsuba/core/fwd.h>
#include <mitsuba/core/spectrum.h>
#include <mitsuba/core/properties.h>
#include <mitsuba/core/plugin.h>
#include <mitsuba/render/sensor.h>

NAMESPACE_BEGIN(mitsuba)
template <typename Float, typename Spectrum>
class RepeatedSensor final : public Sensor<Float, Spectrum> {
public:
    MI_IMPORT_BASE(Sensor)
    MI_IMPORT_TYPES()

    RepeatedSensor(const Properties &props) : Base(props), m_props(props) {
        // You might want to add more stricter error checking here. e.g. the user has to specify copies
        m_copies = props.get<size_t>("copies", 1);
        m_sensor_type = props.string("sensor_type");

        // Want to signal that we've consumed all the properties
        // They will be consumed during expand()
        for(const auto& s: props.unqueried()) {
            props.mark_queried(s);
        }
    }

    // This must be implemented. However, it won't be used in practice
    ScalarBoundingBox3f bbox() const override { return ScalarBoundingBox3f(); }

    std::vector<ref<Object>> expand() const override {
        std::vector<ref<Object>> sensors;
        for (size_t i = 0; i < m_copies; ++i) {
            Properties props(m_props);
            props.set_plugin_name(m_sensor_type);
            sensors.push_back(
                PluginManager::instance()->create_object<Sensor<Float, Spectrum>>(props).get());
        }

        return sensors;
    }

    MI_DECLARE_CLASS()
protected:
    size_t m_copies = 0;
    std::string m_sensor_type;
    Properties m_props;
};


MI_IMPLEMENT_CLASS_VARIANT(RepeatedSensor, Sensor)
MI_EXPORT_PLUGIN(RepeatedSensor, "RepeatedSensor")
NAMESPACE_END(mitsuba)

and a small test I ran

import mitsuba as mi
mi.set_variant('llvm_ad_rgb')
x = {'type': 'batch',
        'sensor': {
            'type': 'repeater',
            'copies': 2,
            'sensor_type' : 'perspective',
            'fov': 45,
            'to_world': mi.ScalarTransform4f().look_at(
                origin=[0, 0, 1],
                target=[0, 0, 0],
                up=[0, 1, 0]
            )
        }
    }
b = mi.load_dict(x)
print(b)
'''
BatchSensor[
PerspectiveCamera[
... skipping more of the output
]
PerspectiveCamera[
... skipping more of the output
]
'''

Note that I modified the batch sensor to print out the inner sensor information as well. I've attached a git diff which should also include the small changes to register the new plugin. repeater_plugin.zip

The only final small thing would be that if you wanted to adjust some properties (e.g. to_world) for each individual sensor but that can be done within expand() when copying each properties within the for loop.

[xacond00]

You are a godsend ! But shouldn't have wasted time on me, doing a whole plugin from scratch...

Those two lines are the most crucial, and what I was looking for, in terms of instantiating objects:

props.set_plugin_name(m_sensor_type);
PluginManager::instance()->create_object<Sensor<Float, Spectrum>>(props).get();

However there seems to be an issue with your code. Defining any nested object inside the RepeatedSensor, results in that object being shared between the nested instances... And it appears to be because those are instantiated before the RepeatedSensor itself.
I'm doing something like this for now:

<sensor type="multi">
			<integer name="sensor_cnt" value="3" />
			<string name="sensor_type" value="perspective" />
			<vector name="cam_dir" value="1,0,0" />
			<float name="cam_dist" value="0.2" />
			<float name="fov" value="19.5" />
			<transform name="to_world">
				<matrix value="-1 0 0 -0.1 0 1 0 1 0 0 -1 6.8 0 0 0 1" />
			</transform>
			<sampler type="independent">
				<integer name="sample_count" value="$spp" />
			</sampler>
			<film type="hdrfilm">
				<integer name="width" value="$resx" />
				<integer name="height" value="$resy" />
				<string name="file_format" value="openexr" />
				<string name="pixel_format" value="rgb" />
			</film>
</sensor>

I need each sensor to have a unique sampler and film though. Does that mean, I have to construct them manually from a additional tag + properties inside RepeatedSensor, so not being able to declare them as a nested XML object definition ? Or what would be the workaround here ?
Also no, those being superseded by BatchSensor is not an option.

Thanks for your help.

Edit.

Anyways, sort of solved the problem by creating a new plugin hierarchy - wrappers, which defer the instantiation of objects.
Wrapper stores a class alias and plugin type as string parameters instead of tags...
Ie. something like this:

<sensor type="perspective">
			<float name="fov" value="19.5" />
			<transform name="to_world">
				<matrix value="-1 0 0 0 0 1 0 1 0 0 -1 6.8 0 0 0 1" />
			</transform>
</sensor>

Becomes this:

<wrap type="any">
                        <string name="class" value="sensor"/>
		        <string name="type" value="perspective"/>
			<float name="fov" value="19.5" />
			<transform name="to_world">
				<matrix value="-1 0 0 0 0 1 0 1 0 0 -1 6.8 0 0 0 1" />
			</transform>
</sensor>

Wrapper then provides a way to instantiate object from the stored plugin name and the nested parameters. Which can be used inside the RepeatedSensor plugin to create multiple copies of the object, as it is stored as raw parameters, without being instantiated in the first place.

[rtabbara]

Glad that you got it working in the end. I think the original question has been answered so I'll mark the discussion as resolved.