Support view dependent & independent light decomposition #705
Conversation
liruilong940607
left a comment
liruilong940607
left a comment
There was a problem hiding this comment.
Left some comments! I like the idea of supporting diffuse / specular in viewer!
gsplat/rendering.py
Outdated
| pass | ||
| else: | ||
| # Colors are SH coefficients, with shape [..., N, K, 3] or [..., C, N, K, 3] | ||
| colors = colors.clone() |
There was a problem hiding this comment.
This clone() breaks the auto diff graph for training. (Also it is not ideal from the efficiency perspective)
I think in this case it's better to not touch this rendering.py file. We could just update the viewer file to support diffuse/specular.
There was a problem hiding this comment.
Yeah, it is a good idea
| "diffuse": "Diffuse", | ||
| "specular": "Specular", |
There was a problem hiding this comment.
If we dont touch rendering.py we we remove these from here..
make colors = colors.clone() only reachable in no_grad view function
liruilong940607
left a comment
liruilong940607
left a comment
There was a problem hiding this comment.
Only request is to move the diffuse/specular out of the rasterization function.
| # SH = 0 is the view-independent diffuse component, SH >= 1 are view-dependent specular components. | ||
| if render_mode in ("Diffuse", "Specular"): | ||
| colors = colors.clone() | ||
| sel = slice(1, None) if render_mode == "Diffuse" else slice(0, 1) | ||
| colors[..., sel, :] = 0.0 # Zero out the unwanted SH components. |
There was a problem hiding this comment.
I still think this logic should live outside of the rasterization function (into viewer_render_fn). For two reasons:
- What if I want to rendering diffuse color + depth map? there is no "Diffuse+D" mode here. User would end up have to write this logic outside of
rasterizationfunction and use "RGB+D" mode anyway. (The major reason) - I'm still not quite happy with the inplace operation
colors[..., sel, :] = 0.0andcolors.clone(). It's not very good to have inplace operations in the torch differentiable graph. It might break the auto diff graph or lead to unused parameters.
2 participants
Video.Project.3.mp4
Besides to rendering full SH as RGB, this PR renders SH=0 component as the view independent diffuse map and SH>0 components as the view dependent specular map.