Unexpected behaviour for to_uv transforms [flip Y coords with clamp]

[georgy95]

I'm trying to use the to_uv texture transform to manipulate the UV values and perform a Y-flip for the texture. I know that I can do this with vflip(), but I'd like to achieve it with to_uv matrix. I should be able to achieve this effect with TO_UV_TRANSFORM @ mesh.texcoordinates.

Intuitively, applying a y-flip should be possible by applying matrix T to the uv values.

T = [
  [1, 0, 0],
  [0, -1, 1]
  [0, 0, 1]
]

i.e. T @ [u,v] = [u, 1-v] Since u, v were in the [0,1] interval, so should be [u, 1-v].

Example

When T=np.eye(4) is the identity matrix, clamping works as expected. When T is the Y-flip matrix, instead of flipping the texture, I get the following result:

The expected behaviour is to flip the texture vertically. Setting mode=repeat does that. However, the UV values are still in the [0,1] range and should display the texture with clamp.

I also tried all possible logical alternatives & transposes of T.

Reproduction

Texture:

mesh.obj

v  1 -1 -1
v  1  1 -1
v -1  1 -1
v -1 -1 -1
vt 1 0
vt 1 1
vt 0 1
vt 0 0
vn 0 0 1
f 1/1/1 2/2/1 3/3/1 4/4/1

scene

import os
import numpy as np
import mitsuba as mi
mi.set_variant("llvm_ad_rgb")
from mitsuba import ScalarTransform4f as T


y_flip = np.array([
        [1, 0, 0, 0],
        [0, -1, 1, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 0]
])


scene = {
    "type": "scene",
    "integrator": {
            "type": "path",
            "max_depth": 20,
            "rr_depth": 5,
        },
        "sensor": {
        "type": "perspective",
        "to_world": mi.ScalarTransform4f.look_at(
                    origin=[0,0,4],
                    target=[0,0,0],
                    up=[0,1,0]
                    ),
        "fov": 40,
        'sampler': {
            'type': 'independent',
            'sample_count': 128
        },
        'film': {
            'type': 'hdrfilm',
            'width': 256,
            'height': 256,
            'rfilter': {
                'type': 'box',
            },
            'pixel_format': 'rgb',
        },
    },
    "uv_texture": {
        "type": "bitmap",
        "filename": "uv.png",
        "to_uv": mi.ScalarTransform4f(
            y_flip
        ),
        "wrap_mode": "clamp"
    },
    "backwall": {
        "type": "obj",
        "filename": "./mesh.obj",
        "light": {
            "type": "area",
            "radiance": {
                "type": "ref",
                "id": "uv_texture"
            }
        },
    },
    "env": {
        'type': 'constant',
        'radiance': {
            "type": "rgb",
            "value": 1.0
        }
    }
}

result = mi.render(scene, spp=10)
result = mi.Bitmap(result)
result

---

I'm using 3.3.0 version.

[njroussel]

Hi @georgy95

In the obj plugin there is a flip_tex_coords parameter that flips the vertical component.

Does this work as you'd expect?

[georgy95]

There are multiple ways to achieve the desired result: auto-flipping on the mesh level or the bitmap level, I was more curious about achieving the same transformation on to_uv transform level.

From my understanding the the uv transform is calculated from T @ [u,v,0,0] instead of T @ [u,v,1,1] which is what I expected.

I understand that perhaps not an important discussion as there are multiple ways of achieving the desired result, as you mentioned.

Thanks.

[njroussel]

I see, if you wish to use the to_uv matrix, it should be:

y_flip = np.array([
        [1, 0, 0, 0],
        [0, -1, 0, 1],
        [0, 0, 1, 0],
        [0, 0, 0, 1]
])

This is all done with homogeneous coordinates.

[georgy95]

This is it, thank you. The mis-understanding came from the docs:

Specifies an optional 3x3 transformation matrix that will be applied to UV values. A 4x4 matrix can also be provided, in which case the extra row and column are ignored.

And I assumed that it had to be padded with 0s, since the 4th column/row would be the one ignored. But it's actually used.

[georgy95]

@njroussel

Apart from Y-flipping, I'm also interested in rotations around the center and other transformations.

The following matrix should rotate the texture 180 degrees around the center:

[[-1, 0, 1],
 [0, -1, 1],
 [0, 0, 1]]

This should result in a map [u,v] -> [1-u, 1-v]. The new u1, v1 coordinate space is still in [0,1] interval and should work with clamp. But it doesn't. It seems that the mapped coordinates are outside of [0,1] interval.

[njroussel]

I tried it with y_flip = mi.ScalarTransform4f().translate([1, 1, 0]).rotate([0, 0, 1], 180)
which is equivalent to what you wrote and it worked as expected.

I think all of this is a bit confusing if you try to initialize the matrix with a Transform4f when in reality the underlying matrix has dimensionality 3. You can double-check your matrices with mi.traverse(scene)['backwall.emitter.radiance.to_uv']

[georgy95]

This is it, thank you. The mis-understanding came from the docs:

Specifies an optional 3x3 transformation matrix that will be applied to UV values. A 4x4 matrix can also be provided, in which case the extra row and column are ignored.

And I assumed that it had to be padded with 0s, since the 4th column/row would be the one ignored. But it's actually used.

I could make a PR if you wish to change the wording.

[njroussel]

We're fixing/improving this behaviour in this PR (#830) already.
Thank you 😄