Dual demosaic for 5.4

[jenshannoschwalm]

While working on capture sharpen i did a huge amount of pixel-peep-testing a lot of images, low & high noise, xtrans and bayer looking for artefacts, moire handling and introduced maze patterns in low-contrast areas, thanks to many people sharing images with special problems btw.

As a result of those tests i reimplemented dual demosaicing (@TurboGit so we have differences with integration tests) leading to an improved quality for low frequency content (better response for noisy signals, no color casts, no mazing).

While doing so dual demosaicing performance has been improved quite a lot, it's performance now is only ~10-20% worse (instead of >50%) than it's main demosaicer and it uses less memory (important for tiling) both for CPU and OpenCL code paths.

@da-phil @weltyj when working on capture i saw you both use dual demosaicing. Would you be able to test?

[weltyj]

@jenshannoschwalm yes I can do some testing. Funny you should be looking at this, just yesterday I implemented a demosaic method by Boris van Schooten from ~2004. Website (13thmonkey.org/~boris) is now incomplete, it was an interesting idea which performed admirably at the time. It's not as good as the other available methods now, but, it got me wishing:

I wish there was some suite of images, that we knew what demosaiced "truth" was, so after any demosaic method was applied you could compute typical error measurements. I have even thought about ray-tracing some scenes to tiff which would be "truth", then applying distortions to that tiff such as TCA, LCA, blurs, noise, and creating raw files (with Bayer or Xtrans sampling) to be demosaiced with different algorithms and objectively compared back to the original tiff "truth" (obviously not a physical camera with the nuance of photosites...)

Okay enough rambling thoughts. I'll have a look at the new demosaicing code and how it performs with some of my test images.

[da-phil]

@da-phil @weltyj when working on capture i saw you both use dual demosaicing. Would you be able to test?

Actually, I never used dual-demosaicing and I never pixel-peeped to the point where I saw problems with the default demosaicer, so my eyes might not be well-trained to look out for demosaicing issues.
If you can give me clear instructions what to look out for, I could do some comparisions between develop and master branch and your feature branch.

[TurboGit]

I have done some testing and see less noise in the flat areas giving also a bit more sharpness. That being said I haven't tested with the previous version, and I probably don't have a good setup to check for difference as all this is very subtle.

@weltyj : Please report about your testing when you're done.

I can also confirm that the integration tests for dual demosaic are failing. The diff image is scary :) All read as if all pixels have been changed. For amaze (dual):

Test 0164-demosaic-amaze-vng4
      Image mire1.cr2
      CPU & GPU version differ by 16680 pixels
      CPU vs. GPU report :
      ----------------------------------
      Max dE                   : 1.27740
      Avg dE                   : 0.00274
      Std dE                   : 0.03869
      ----------------------------------
      Pixels below avg + 0 std : 99.40 %
      Pixels below avg + 1 std : 99.41 %
      Pixels below avg + 3 std : 99.42 %
      Pixels below avg + 6 std : 99.49 %
      Pixels below avg + 9 std : 99.59 %
      ----------------------------------
      Pixels above tolerance   : 0.00 %
 
      Expected CPU vs. current CPU report :
      ----------------------------------
      Max dE                   : 18.21741
      Avg dE                   : 0.90052
      Std dE                   : 0.81131
      ----------------------------------
      Pixels below avg + 0 std : 60.96 %
      Pixels below avg + 1 std : 86.67 %
      Pixels below avg + 3 std : 98.33 %
      Pixels below avg + 6 std : 99.91 %
      Pixels below avg + 9 std : 99.99 %
      ----------------------------------
      Pixels above tolerance   : 6.17 %
 
  FAILS: image visually changed
         see diff.png for visual difference
         (2.39332e+06 pixels changed)

But the expected vs output seems almost identical when looking at them:

Expected:

Output:

[weltyj]

@TurboGit @jenshannoschwalm Yes I too think the changes are going to be subtle. When I was testing the pixeldeblur code last year, I wrote a perl script to generate an XMP file and change parameters in loops in the perl script, so I could do automated generation of tiff images for comparison via the darktable-cli.

What I'd like to do for this is a similar idea, looking at red,green,blue channels separately for old vs new version, at various levels of the "dual threshold" parameter. I have a few specific areas of a small number of images picked to look at.

[jenshannoschwalm]

Yes, differences new-vs-old algorithm are mostly very subtle.
To be reminded

1. xtrans vng always had problems at borders like seen here

vs what have from mark3

so using vng as flat-content demosaicer can lead to bad artefacts on xtrans

2. As shown in RFC: lets get rid of some demosaicers #18549 by @PeterWem vng avoids maze pattern artefacts for bayer so the new algorithm must be at least as good/stable as vng for flat regions - and it is :-)

[weltyj]

Just a progress report -- my automated perl script seems to work fine now -- I can take any image and push it through the master branch code and Hanno's new code altering demosaic parameters directly through the perl script and then manually compare the resulting tiff images ((new-old)*scalefactor+127) to look for differences.

Image is an example of rcd dual, with dual threshold set to 0.90
Time to take a break...

I put the perl code here (such as it is...)
https://drive.google.com/file/d/1OKNpDmUJxS1a5YsqLurT2yR_WVem4Tbq/view?usp=sharing

Top level code is very simple:
`
my $match_green=0 ;
my $edge_threshold=0.0 ;
my $color_smoothing=0 ;
my $demosaic_method=DUAL | RCD ;
my $LMSSE_refine=0 ;
my $dual_threshold=0.90 ;

my $sharpen_radius=0.50 ;
my $sharpen_threshold=0.40 ;
my $corner_boost=0.0 ;
my $sharpen_iterations=0 ;
my $sharpen_center=0.0 ;

print "method:$demosaic_method\n" ;

my @test_parms = (
$match_green,
$edge_threshold,
$color_smoothing,
$demosaic_method,
$LMSSE_refine,
$dual_threshold,
$sharpen_radius,
$sharpen_threshold,
$corner_boost,
$sharpen_iterations,
$sharpen_center) ;

my $infile="DSC09586.ARW" ;

my $outfile="old_test.tif" ;
run_darktable($infile,$outfile,"old","DSC09586.ARW.xmp",@test_parms) ;

$outfile="new_test.tif" ;
run_darktable($infile,$outfile,"new","DSC09586.ARW.xmp",@test_parms) ;
`

[weltyj]

I've noticed some changes in edges, which are I think are not because of the dual algorithm but because of the RCD algorithm itself. The changes cause more color fringing in the newer version. Hard to quanitfy but I'll try to get some comparitive images up in a few days. Been busy with other stuff...

[jenshannoschwalm]

RCD has not changed at all .

[jenshannoschwalm]

not to be merged right now, there is more important demosaicer work pending ...

[weltyj]

I finally have some results. Took a little while to work out what I was seeing. The new version appears to be intensifying the saturation (and values, possibly a hue shift) of some pixels, but not others. Shows up more and more as threshold moves from 0.2 to 1.0 Here's a good example:

On my test sample of 4 images (all Bayer sensors) this seems to show up on three of the images. Seems not to matter whether it is rcd or amaze, and is something about the vng4 demosaicer on the new version. I did some plots of the standard deviation of the saturation of pixels in a small crop of the 4 images, the changes, while not large, are visible. In some areas of images it looks a more like chroma noise is increased on the new version (which begs the question if the denoiser can remove the increase and it will all be ok). But other areas, like this little grass stem, it is a higher saturation on edges.

Here are the plots of standard deviation of saturation of pixels.

[jenshannoschwalm]

Thanks a lot, that's exactly what i was looking for. I think i understand the reason for the problem and fortunately have an idea how that can be fixed.

[weltyj]

... have an idea how that can be fixed.

@jenshannoschwalm -- sounds good. I have this all automated so I can turn around the analysis very quickly now whenever you have a new version to try.

[jenshannoschwalm]

@weltyj thanks for your testing again. I checked my "idea" and must confess, it works just somehow but leaves maze patterns that we don't have when using VNG. So not a good way ... Closed this for now ...

[weltyj]

@jenshannoschwalm Maybe you'll wake up in the middle of the night and have another idea. That's how it works for me. Thanks for your efforts so far...

[jenshannoschwalm]

Right, so far no productive night-awakes :-)