Advice for inverse rendering camera photos

[test3211234]

So, smartphone photos look very different from regular 3D render. They have compression artifacts and stuff that happens during the RAW to RGB process (ISP) like demosaic, their own algorithms, and noise. So how does one actually optimize for inverse rendering with this difference? Do we want to make a neural net somehow get rid of anything the ISP did?

For exapmle, look at this image from iPhone 11 (which I denoised inaccurately, just wanted to remove that variable):

You can see the color inconsistency, smoothness at color transitions, and artifacts.

[njroussel]

Hi @test3211234

Ideally, you would want to model all of these steps inside your code/setup, then the system could differentiate through them. But you could also, as you mentioned, process the reference image to try to revert these effects.

[test3211234]

Well I'm doing single image inverse rendering, which needs neural prior. I think that's too much ambiguity if I try to learn the initial camera param, but do you think there's a way?