[BUG] Cellpose 3D Ring Artifacts Problem & Solution

[derekthirstrup]

3D segmentation: anisotropy parameter causes ring/donut artifacts due to Z-upsampling interpolation`

Description

When using do_3D=True with the anisotropy parameter set to the correct physical anisotropy ratio (e.g., 6.15 for typical confocal microscopy), ring/donut-shaped artifacts appear in segmentation masks. Paradoxically, setting anisotropy=1 (physically incorrect) produces better segmentation results.

Environment

• Cellpose version: 4.0.8
• Python version: 3.13
• GPU: NVIDIA RTX (CUDA)
• Data: 3D confocal microscopy, Z spacing ~2μm, XY spacing ~0.325μm (anisotropy ratio ~6:1)

Steps to Reproduce

1. Load a 3D image stack with high Z-anisotropy (e.g., 6:1 ratio)
2. Run segmentation with correct anisotropy:

model = models.Cellpose(gpu=True, model_type='nuclei')
masks, flows, styles, diams = model.eval(
    image,
    do_3D=True,
    anisotropy=6.15,  # Correct physical ratio
    flow3D_smooth=0,
    channels=[0, 0]
)

3. Observe ring/donut artifacts in resulting masks

4. Compare with incorrect anisotropy:

masks, flows, styles, diams = model.eval(
    image,
    do_3D=True,
    anisotropy=1,  # Incorrect - no Z rescaling
    flow3D_smooth=0,
    channels=[0, 0]
)

5. Observe that anisotropy=1 produces solid masks without ring artifacts

Expected Behavior

Setting the correct anisotropy value should produce equal or better segmentation quality compared to anisotropy=1.

Actual Behavior

Setting the correct anisotropy value produces ring/donut artifacts, while the incorrect anisotropy=1 produces solid masks.

Images

With anisotropy=6.15 (correct, but produces artifacts):

[Image showing ring artifacts - nuclei appear as hollow donuts with concentric rings]

With anisotropy=1 (incorrect, but no artifacts):

[Image showing solid nuclei masks without ring artifacts]

Root Cause Analysis

After investigating the Cellpose source code, I believe I understand the cause:

1. The anisotropy parameter upsamples Z before network inference

In models.py lines 392-396:

if anisotropy is not None and anisotropy != 1.0:
    x = transforms.resize_image(x.transpose(1,0,2,3),
                            Ly=int(Lz*anisotropy),  # Upsamples Z
                            Lx=int(Lx)).transpose(1,0,2,3)

With anisotropy=6.15, a 20-slice stack becomes 123 slices via interpolation.

2. Interpolated slices create artificial gradients

The upsampling creates smooth transitions between real Z-slices, but these interpolated slices:

• Have blurred boundaries
• Contain artificial gradients not present in real data
• Lack true cellular structure to anchor flow predictions

3. The neural network predicts inconsistent flows on interpolated data

The network was trained on real image data, not interpolated data. When it sees interpolated intermediate slices, it predicts flows based on features that don't correspond to real cellular boundaries. This causes Z-flows to oscillate.

4. Euler integration fails to converge

In dynamics.py:steps_interp(), the Euler integration follows these oscillating flows. Instead of converging to cell centers, pixels get trapped in circular orbits, creating ring artifacts.

5. Why anisotropy=1 works

With anisotropy=1:

• No upsampling occurs - the network sees only real data
• Z-flows are coarse but consistent (based on real image features)
• The larger effective step size (due to smaller Z dimension) helps pixels converge faster
• Fewer Z-slices means fewer opportunities for flow inconsistencies

Workarounds

Workaround 1: Use anisotropy=1

Accept anisotropic output masks and rescale afterward if needed. This avoids introducing interpolation artifacts.

Workaround 2: Increase flow3D_smooth

Setting flow3D_smooth=2 or higher smooths out the interpolation artifacts, but also blurs XY boundaries.

Workaround 3: Anisotropic flow smoothing (proposed)

I've implemented a modification that allows flow3D_smooth to accept a tuple (z_sigma, y_sigma, x_sigma) for anisotropic smoothing. This allows heavy Z-smoothing to fix artifacts while preserving XY boundary precision:

masks, flows, styles, diams = model.eval(
    image,
    do_3D=True,
    anisotropy=6.15,
    flow3D_smooth=(3.0, 0.0, 0.0),  # Smooth Z only
    channels=[0, 0]
)

I can submit this as a separate PR if there's interest.

Suggested Long-term Fixes

1. Document this behavior - Users should be warned that high anisotropy values may cause artifacts

2. Consider alternative upsampling strategies - Current linear interpolation creates smooth but artificial gradients. Nearest-neighbor or more sophisticated methods might introduce fewer artifacts

3. Apply flow smoothing proportional to anisotropy - Automatically increase Z-smoothing when anisotropy > 1

4. Reconsider the upsampling approach - Perhaps the network should run on original anisotropic data with anisotropy-aware flow field handling, rather than upsampling the image

Mathematical Background

For those interested, I've written a detailed mathematical analysis of why anisotropic smoothing is the correct regularization for this problem, covering:

• Lipschitz continuity requirements for Euler integration convergence
• Nyquist-Shannon sampling theorem implications
• Discrete gradient truncation errors scaling with anisotropy ratio squared
• Ring artifact formation dynamics

---

Attachments to Include

1. Screenshot with anisotropy=6.15 showing ring artifacts
2. Screenshot with anisotropy=1 showing solid masks
3. Link to mathematical derivation document

Masks using anisotropy=6.15

Masks using anisotropy=1

Anisotropic Flow Smoothing - Mathematical Foundation.pdf

[yuriyzubov]

@mrariden @derekthirstrup, thanks! See #1408
should we automatically increase Z-smoothing when anisotropy > 1?

[mrariden]

Thanks @derekthirstrup we'll leave it to the user to decide when to smooth the flows. We did add a way to smooth only z flows per your recommendation

[derekthirstrup]

#1408 (comment)