Add and render differential per-vertex feature while keeping original rendering

[Tianhang-Cheng]

Summary

Hi! Thanks for the great mistuba work!! I am working for an inverse rendering task and I'd like to add and output the differential per-vertex feature (4 channel feature) in an AOV style while keeping the original PRB rendering. I plan to pass this feature to a pytorch decoder to supervise the original RGB rendering. I look at the AOV tutorial and latent variable tutorial, but I don't know how to add this kind latent varible. Could you give me some guidance? Thanks!

System configuration

System information:

OS: linux
CPU: ...
GPU: Nvidia A40
Python version: 3.9
LLVM version: ...
CUDA version: 12.1
NVidia driver: 530.30.02

Dr.Jit version: 0.4.2
Mitsuba version: 3.3.0
Compiled with: pip
Variants compiled: ...

Description

I load a .obj mesh in this way:

def load_input_mesh(model_path, tex_dim):
    
    if not os.path.exists(model_path):
        raise FileNotFoundError
    mesh = o3d.t.io.read_triangle_mesh(str(model_path))   

    mesh.material.set_default_properties()
    # mesh.material.material_name = 'defaultLit'  # note: ignored by Mitsuba, just used to visualize in Open3D
    mesh.material.texture_maps['albedo'] = o3d.t.geometry.Image(0.5 + np.zeros((tex_dim, tex_dim, 3), dtype=np.float32))
    mesh.material.texture_maps['roughness'] = o3d.t.geometry.Image(0.0 + np.zeros((tex_dim, tex_dim), dtype=np.float32))
    mesh.material.texture_maps['metallic'] = o3d.t.geometry.Image(0.0 + np.zeros((tex_dim, tex_dim), dtype=np.float32))
    
    mesh.material.texture_maps['feature'] = o3d.t.geometry.Image(0.5 + np.zeros((tex_dim, tex_dim, 3), dtype=np.float32))
 
    return mesh

geometry = mesh.to_mitsuba('themesh')  
mi.traverse(geometry)

It outputs these parameters, but there is no 'feature' among them. And It seems AOV does not support self-defined output.

mi.traverse(geometry)
SceneParameters[
  ---------------------------------------------------------------------------------------------
  Name                                      Flags    Type  Parent
  ---------------------------------------------------------------------------------------------
  bsdf.clearcoat.value                      ∂        Float UniformSpectrum
  bsdf.clearcoat_gloss.value                ∂        Float UniformSpectrum
  bsdf.metallic.data                        ∂        TensorXf BitmapTexture
  bsdf.metallic.to_uv                                ScalarTransform3f BitmapTexture
  bsdf.main_specular_sampling_rate                   float Principled
  bsdf.clearcoat_sampling_rate                       float Principled
  bsdf.diffuse_reflectance_sampling_rate             float Principled
  bsdf.specular                             ∂, D     Float Principled
  bsdf.roughness.data                       ∂, D     TensorXf BitmapTexture
  bsdf.roughness.to_uv                      , D      ScalarTransform3f BitmapTexture
  bsdf.base_color.data                      ∂        TensorXf BitmapTexture
  bsdf.base_color.to_uv                              ScalarTransform3f BitmapTexture
  bsdf.anisotropic.value                    ∂        Color3f SRGBReflectanceSpectrum
  bsdf.spec_tint.value                      ∂        Float UniformSpectrum
  bsdf.sheen.value                          ∂        Float UniformSpectrum
  bsdf.sheen_tint.value                     ∂        Float UniformSpectrum
  bsdf.spec_trans.value                     ∂        Float UniformSpectrum
  bsdf.flatness.value                       ∂        Float UniformSpectrum
  vertex_count                                       int   Mesh
  face_count                                         int   Mesh
  faces                                              UInt  Mesh
  vertex_positions                          ∂, D     Float Mesh
  vertex_normals                            ∂, D     Float Mesh
  vertex_texcoords                          ∂        Float Mesh
]

Is there any possibility to set two "material" for the same mesh and optimize them together? In this way I can output vertex feature using one material, and output RGBA rendering using original principled BSDF. Thanks!

Steps to reproduce

None

[njroussel]

Hi @Tianhang-Cheng

This is not fully supported. We have recently added some functions that might make the implementation of something like this a bit easier in the future.

I think the easiest approach would be to add your "feature" as a mesh attribute (read a guide about this here). Attributes are only supposed to be 1 or 3 channels. You would therefore need to add support for whatever feature width you need. The second and lost modification you would need to do, is modify the aov integrator to also output extra channels for mesh attributes (specifying the attribute name of your feature vector).

[Tianhang-Cheng]

Ok! Thanks for your reply! It really helps : )
Could you send me a new link fo mesh attribute ? The old links seems broken. Thanks!

[njroussel]

I fixed the link

[Tianhang-Cheng]

I try to add mesh_attribute for "base_color", it becomes bsdf.base_color.scale, but the original base_color ( bsdf.base_color.value) seems missing.

mesh = mi.load_dict({
                "type": "obj",
                "filename": FLAGS.base_mesh,
                "bsdf": {
                    "type": "principled", 
                    "base_color": {
                        "type": "mesh_attribute",
                        "name": "vertex_feat1",  # This will be used to visualize our attribute
                    },
                },
            }) 
        attribute_size = mesh.vertex_count() * 3

      mi.traverse(mesh )



SceneParameters[
  ---------------------------------------------------------------------------------------------
  Name                                      Flags    Type  Parent
  ---------------------------------------------------------------------------------------------
  bsdf.clearcoat.value                      ∂        Float UniformSpectrum
  bsdf.clearcoat_gloss.value                ∂        Float UniformSpectrum
  bsdf.metallic.value                       ∂        Float UniformSpectrum
  bsdf.main_specular_sampling_rate                   float Principled
  bsdf.clearcoat_sampling_rate                       float Principled
  bsdf.diffuse_reflectance_sampling_rate             float Principled
  bsdf.specular                             ∂, D     Float Principled
  bsdf.roughness.value                      ∂, D     Float UniformSpectrum
  bsdf.base_color.scale                              float MeshAttribute
  bsdf.anisotropic.value                    ∂        Float UniformSpectrum
  bsdf.spec_tint.value                      ∂        Float UniformSpectrum
  bsdf.sheen.value                          ∂        Float UniformSpectrum
  bsdf.sheen_tint.value                     ∂        Float UniformSpectrum
  bsdf.spec_trans.value                     ∂        Float UniformSpectrum
  bsdf.flatness.value                       ∂        Float UniformSpectrum
  vertex_count                                       int   OBJMesh
  face_count                                         int   OBJMesh
  faces                                              UInt  OBJMesh
  vertex_positions                          ∂, D     Float OBJMesh
  vertex_normals                            ∂, D     Float OBJMesh
  vertex_texcoords                          ∂        Float OBJMesh
  vertex_feat1                              ∂        Float OBJMesh
]

I try to add new parameter name but it says I can't create new one

 Unreferenced property "features" in plugin of type "principled"!

mi.load_dict({
                "type": "obj",
                "filename": FLAGS.base_mesh,
                "bsdf": {
                    "type": "principled", 
                    "features": {
                        "type": "mesh_attribute",
                        "name": "vertex_feat1",  # This will be used to visualize our attribute
                    },
                },
            })  

could you give me some help? Thanks!

[njroussel]

Oh I see the confusion. There is two different things described as "mesh attributes" here.

The first is the additional mesh property which is added with:

mesh.add_attribute(
    "vertex_color", 3, [0] * attribute_size
)  # Add 3 floats per vertex (initialized at 0)

The second is the visualization tool for this mesh property, the mesh_attribute texture. This allows you to use the attribute as your BSDF reflectance for example. However, if your feature is 4 dimensional it obviously cannot be represented as a color. This is not what you want to be using

You could use the second one to export a visualization of your features. However, in addition to being limited to 3 channels, it would also apply any lighting effects you have on it. You can see this in this image, where the bunny is still shaded.