Questionable "-near_lossless" behavior

[MasterInQuestion]

    I doubt making monotonous things less monotonous may improve compressibility.
    Compare below:
    "highlight-4x.webp" | "highlight-4x.0.webp"
    "diff_quot-6x.webp" | "diff_quot-6x.40.webp"

    See also: https://github.com/MasterInQuestion/attach/commit/401c58fc3a56d265feadda567f02d79de93ff10c#diff_quot
    This is also the source of most significant amount of visual difference.


    Apparent example:
    https://github.com/MasterInQuestion/attach/raw/main/MasterInQuestion/talk/36/Shadowing.webp
    ; where the background is plain #002b36.

    Patching the smeared area by "-near_lossless", and recompress with strict lossless:
    Resulted smaller output, and apparent more fidel image.

    Demo 2: https://github.com/MasterInQuestion/attach/raw/main/MasterInQuestion/talk/22/pray.webp
    Note the border.

    Demo 3: https://github.com/MasterInQuestion/attach/raw/main/curl/curl/14440/foobar.webp
    "-near_lossless 40" would introduce ringing artifact in the colored regions:
    That when patched would result in both improved compressibility and fidelity.
    .
    The very reason why "-near_lossless" would spoil in:
    https://github.com/MasterInQuestion/attach/raw/main/community/community/139005/Long-long-swing.webp

    Demo 4:
    https://github.com/MasterInQuestion/attach/raw/main/MasterInQuestion/talk/22/border.webp
    https://github.com/MasterInQuestion/attach/raw/main/MasterInQuestion/talk/22/blurriness.webp
    .
    More visible than previous examples.

[MasterInQuestion]

    Suggested fix:
    For contiguous identical pixels beyond a certain threshold:
    The "near_lossless" attempt should be dropped: as which seems to just spoil things.
    (both reduced quality and compressibility)

    "--debug-near_lossless-contiguity" may be added to control the threshold.
    Values of interest: 16, 24, 32 [ 4 and 8 maybe of interest too. ]
    [ Ideal value might be ~~just 2, or 3~~. Seemingly 4. ] [1]
    .
    16: 1x16, 2x8, 4x4
    24: 1x24, 2x12, 3x8, 4x6
    32: 1x32, 2x16, 4x8

    Contiguousness:
    ←↖↑↗→
    ↙↓↘
    (note no circular looping)

    If altering the "near_lossless" behavior is somehow convoluted:
    Adding the contiguity patch as post-filter phase shall be also viable.
    (after "near_lossless" additionally copy-over from original all contiguous pixels)

    The technique could probably apply to all lossy processings (compression or denoise).
    Various observation seems to indicate, making what originally monotonous less monotonous: would only result in both reduced compressibility and fidelity.

    The alike ringing artifact in JPEG, H.264 etc. when patched: should also receive the similar gain.

[ [1]
    The ideal strategy is mostly not some hard threshold, but something slightly more sophisticated.
    And, with whose introduction: there shall be no need for specifying the threshold.
    (the ideal strategy is "2" derived; but with some selection logic inclined toward optimal compressibility)

    I have not yet well-defined the definite logic.
    Shall update later. ]

[MasterInQuestion]

    @bitsgalore
    Would you give some review on the applicability of alike post-patching procedure in JPEG?
    I don't yet enough understand the implementation details of JPEG.
    Thanks for assistance.

[bitsgalore]

@MasterInQuestion Sorry, I don't really understand your question, and besides that I know very little about JPEG implementation details myself! So I'm afraid I won't be able to help here.

[MasterInQuestion]

    No matter. I'll analyze it later.
    I'm testing several fields of the entropy theory.
    And my focus is the theoretical best compression (size) attainable.

    Thanks for chime-in anyway.

[MasterInQuestion]

    See also:
    https://trac.ffmpeg.org/ticket/11019

    The denoise techniques maybe related.