Training error on Custom dataset

[gwang-kim]

Hi,
I'm using XHumans Dataset which provide mesh. I rendered normal with camera poses and got error as following while training the model. Any insight would be helpful.

true_normal torch.Size([2048, 3, 3, 3]) tensor(0.9999) tensor(-1.0000)
comp_normal torch.Size([2048, 3, 3, 3]) tensor(0.4257, grad_fn=<MaxBackward1>) tensor(-0.6862, grad_fn=<MinBackward1>)
normal_loss 1.0001189708709717
eikonal_loss 0.0007976764463819563
mask_loss 2.1950364112854004

 16%|█████████████████▊                                                                                              | 796/5000 [00:21<01:52, 37.46it/s, loss=3.443e+00, s=4.836e-01, rm_step=7.248e-03, samples_per_ray=0.3]
Traceback (most recent call last):
  File "/home/bradleyk/project/SuperNormal/exp_runner.py", line 670, in <module>
    runner.train()
  File "/home/bradleyk/project/SuperNormal/exp_runner.py", line 186, in train
    if render_out['gradients'] is None:  # all rays are in the zero region of the occupancy grid
KeyError: 'gradients'

[xucao-42]

It looks like the forward pass in models/renderer.py has not been executed correctly. The renderer returns a dictionary, but it does not contain "gradients." You may check whether it's accidentally deleted, or you can try modify the line as if "gradients" in render_out

[yyao-mathmagic]

@xucao-42 @gwang-kim Hi guys, I am also trying to use my own mesh with my rendered normal, but the results are pretty bad(not converged I think). The reason could be normal's values or camera parameters, did you have codes for such task for reference, i would really appreciate that. What I did is render normal as exr format which range from -1 to 1 but not 0 to 255. And also, I convert view matrix to extrinsic matrix and projection matrix to intrinsic matrix which may cause some issues.

[xucao-42]

Have you ensured the target object is bounded by a unit sphere? If you have the GT mesh, this can be simply done by scaling and translating the mesh. If your camera poses are object-centric, you can refer to this function for normalizing the camera poses so that the object is bounded by a unit sphere.

[yyao-mathmagic]

this function

@xucao-42 thank you for reply! I did call this function, and what my codes are like below
`
def save_camera_sphere(in_list, ex_list, output_path):

camera_sphere = dict()
R_list=[]
t_list=[]

for i in range(len(in_list)):
    K = in_list[i]
    W2C = torch.cat((ex_list[i],torch.tensor([[[0.,0.,0.,1.]]]).to(device)), 1)[0]
    R_list.append(W2C[:3, :3].cpu().detach().numpy())
    t_list.append(W2C[:3, 3].cpu().detach().numpy())
    view_idx = i
    camera_sphere[f"world_mat_{view_idx}"] = make4x4(K.cpu()) @ W2C.cpu().detach().numpy()

offset, scale = normalize_camera(R_list, t_list, camera2object_ratio=3)
scale_mat = np.eye(4)
scale_mat[:3, :3] *= scale
scale_mat[:3, 3] = offset

for i in range(len(in_list)):
    camera_sphere[f"scale_mat_{i}"] = scale_mat

np.savez(os.path.join(output_path, 'cameras_sphere.npz'), **camera_sphere)

`

there is one thing I don't know how to specify is the camera2object_ratio, what value should I set to this?

[xucao-42]

The larger this ratio, the more space the object occupies within the unit sphere (since the surrounding cameras become further from the origin, the object scales proportionally to produce the same images).

Could you share your input normal maps and recovered meshes? That will be helpful.

[yyao-mathmagic]

The larger this ratio, the more space the object occupies within the unit sphere (since the surrounding cameras become further from the origin, the object scales proportionally to produce the same images).

Could you share your input normal maps and recovered meshes? That will be helpful.

@xucao-42 sure, I put my mesh and rendered mask/normal maps w/ converted camera parameters here https://drive.google.com/file/d/1NXKhR9OPbUT-5f7vZW3zZyXrjn58yi-Y/view?usp=drive_link .
Let me know if you need more information such as render codes or camera conversion codes.

[xucao-42]

I suspect that the camera parameters may not be handled correctly.

You can try setting the weight of the normal loss to 0, so that the shape is reconstructed solely based on the masks. If the camera parameters are accurate, the model should still be able to reconstruct a coarse geometry. However, with your dataset, I observed that the training just does not progress.

[kemalcanbora]

I have the same problem. I can bypass the gradient part, but at the end of training, the model doesn't generate anything, when I check exp folder eval part have not any information, and when I did weight normal zero same. Anyone find some solution for this issue?