Skip to content

Fix severe boundary artifacts in Box filter caused by floating-point precision#255

Open
DavidDiao wants to merge 1 commit intonodeca:masterfrom
DavidDiao:master
Open

Fix severe boundary artifacts in Box filter caused by floating-point precision#255
DavidDiao wants to merge 1 commit intonodeca:masterfrom
DavidDiao:master

Conversation

@DavidDiao
Copy link
Copy Markdown

Description

This PR fixes a severe accuracy bug specifically affecting the Box filter during downsampling.

Users migrating from or comparing outputs with PIL/Pillow might notice that at certain downsampling ratios (e.g., 1000px -> 288px), pica produces periodic visual glitches.

The Root Cause

Unlike continuous filters (Lanczos, Bicubic) which gracefully decay to 0.0 at their window edges, the Box filter is a discrete step function. It relies on a hard cliff: a pixel is either fully included (weight 1.0) or completely excluded (weight 0.0).

When calculating the normalized distance of a boundary pixel to the center diff = (pxl + 0.5) - center:

  1. Mathematically, a boundary pixel might have an exact distance of 0.5 or -0.5.
  2. However, due to IEEE 754 double-precision floating-point arithmetic in JS, the distance is often computed as something like -0.49999999999999994.
  3. The original Box filter logic Math.abs(x) < 0.5 evaluates this as true.

As a result, an out-of-bounds pixel that should have been discarded (weight 0.0) incorrectly receives a full weight of 1.0. This accidentally widens the averaging window by 1 extra pixel for certain periodic target pixels, drastically changing the denominator and introducing a neighboring color, causing the massive deviation.

The Fix

This PR implements a mathematically robust boundary extraction specifically to defend against floating-point jitter at exactly ±0.5 for the Box filter. By offsetting the Math.floor bounds by +0.5 (a standard technique used in PIL's C implementation), we physically exclude the floating-point-corrupted pixels from entering the accumulation loop.

Impact

  • Box filter: Completely eliminates the periodic ~15 pixel value errors. Output now flawlessly matches Pillow's Box downsampling at the integer level.
  • Other filters (Lanczos, Bicubic, etc.): Unaffected. Since continuous filters evaluate to 0.0 at x = support, floating-point boundary jitter does not impact their sum.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant