IndexError - large negative index out of bounds in segmap in legacypipe

[craigwarner-ufastro]

This error only occurs in 1275p185 so far.

  File "/global/cfs/cdirs/desi/users/cdwarner/code/Tractor/legacypipe/py/legacypipe/oneblob.py", line 424, in run
    self.compute_segmentation_map()
  File "/global/cfs/cdirs/desi/users/cdwarner/code/Tractor/legacypipe/py/legacypipe/oneblob.py", line 615, in compute_segmentation_map
    Iseg, = np.nonzero(ok * ((self.refmap[iy, ix] & IN_BLOB['CLUSTER']) == 0))
                              ~~~~~~~~~~~^^^^^^^^
IndexError: index -2147483648 is out of bounds for axis 1 with size 33

It can be reproduced by running brick 1275p185 in the legacypipe gpu 1.4.4 image with the options --use-gpu --ngpu 1 --threads 8 --threads-per-gpu 8 --gpumode 2 so that all threads are run on the GPU. Yet the crash is in legacypipe and not tractor.

It also seems to only occur when python is run without the -O flag oddly. And it does not occur when I run the code outside the container in my environment, making this the opposite of the positionToPixel issue. But it does occur when running with my code in the PYTHONPATH within the image.

It may have been fixed by pulling in the latest changes from referency.py from main, which I did but may not be incorporated into the docker image? Yet I would think in this case it would not crash when using my legacypipe within the docker image.

Also ix and iy come from radec2pixelxy so this seems related to the other WCS issue:

        ok,ix,iy = self.blobwcs.radec2pixelxy(
            np.array([src.getPosition().ra  for src in self.srcs]),
            np.array([src.getPosition().dec for src in self.srcs]))

[dstndstn]

Hallelujah, I can reproduce this one!

[dstndstn]

Re-running with a single GPU thread, I'm seeing one source "stalling" in its fitting --
Blob 20,
Model selection for source 1 of 10 in blob 20; sourcei 9
OneBlob before model selection: SersicGalaxy at GaiaPosition: RA, Dec = (127.50962, 18.43220), pm (0.0, 0.0), parallax 0.000 with NanoMaggies: g=18.8, r=17.5, i=17.1, z=16.8 and EllipseWithPriors(0.25): log r_e=0.388904, ee1=-0.0451923, ee2=0.19706, Sersic index 4.000
Running VECTORIZED GPU code...
SERSIC

I'm seeing 100% CPU and I think 100% GPU, if I'm understanding the nvidia-smi correctly:

> nvidia-smi
Tue Oct  7 11:00:19 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  |   00000000:03:00.0 Off |                    0 |
| N/A   36C    P0            103W /  400W |     777MiB /  40960MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

[dstndstn]

It's this little totally average-looking galaxy, btw!

https://www.legacysurvey.org/viewer/?ra=127.5096&dec=18.4321&layer=ls-dr9&zoom=16

not close to any edges, fairly isolated, should be trivial??!

[dstndstn]

I just tried re-running with --blob-radec isolating to run only that one blob, and it quickly ran to completion.

specifically, --blobradec 127.50962 18.43220

[moustakas]

Have we fixed this? I just got the same error with legacysurvey/legacypipe:gpu-1.4.5 and

runbrick --radec 205.40416999999997 -29.91389 --width=875 --height=875 --pixscale=0.262 --threads=16 --outdir=/pscratch/sd/i/ioannis/SGA2025-v0.10/dr11-south/205/20
540m2991 --bands=g,r,i,z --survey-dir=/dvs_ro/cfs/cdirs/cosmo/work/legacysurvey/dr11 --run=south --release=11000 --write-stage=srcs --skip-calibs --checkpoint=/pscratch/sd/i/ioannis/SGA2025-v0.10/dr
11-south/205/20540m2991/SGA2025_20540m2991-checkpoint.p --pickle=/pscratch/sd/i/ioannis/SGA2025-v0.10/dr11-south/205/20540m2991/SGA2025_20540m2991-%%(stage)s.p --save-unwise-psf --galex --save-galex
-psf --save-coadd-psf --no-galaxy-forcepsf --fit-on-coadds --no-ivar-reweighting --use-gpu --threads-per-gpu=8 --ngpu=1 --gpumode=2

[moustakas]

Fixed in legacysurvey/legacypipe:gpu-1.4.6.

[craigwarner-ufastro]

Re-opening. This error occurred in original CPU mode in the following bricks:
1753p010
1768m042
1791m017
1801p025
1811p010

And in Factored CPU mode in:
1791m025

This was with legacypipe:gpu-1.4.9 but running my local copy of tractor and legacypipe that had fixed a syntax error.

[craigwarner-ufastro]

Some more info - in the bad source, ix is a huge negative number and iy is 0. The RA and DEC are wildly different than what they are supposed to be - this is brick 1811p010. And the first source is RA 343 DEC -291 (obviously a nonsensical value). We have the array ok which flags this correctly as False.

ix=array([-2147483648,          29], dtype=int32) iy=array([ 0, 20], dtype=int32) ok=array([False,  True])
Shapes (2,) (2,) (2,) (37, 46)
src=PointSource(RaDecPos[np.float64(343.91894361998084), np.float64(-291.1279705409023)], NanoMaggies: g=(flux -8.41e+05), r=(flux -3.89e+05), z=(flux -4.56e+05)) src.getPosition().ra=np.float64(343.91894361998084) src.getPosition().dec=np.float64(-291.1279705409023)
src=GaiaSource(GaiaPosition[], NanoMaggies: g=-0, r=-0, z=-0) src.getPosition().ra=np.float64(181.0424480205035) src.getPosition().dec=np.float64(1.0528526131325218)

I propose we change

            Iseg, = np.nonzero(ok * ((self.refmap[iy, ix] & IN_BLOB['CLUSTER']) == 0))

to

        if np.any(ix < 0):
            ok[ix < 0] = False
            ix[ix < 0] = 0
        if np.any(iy < 0):
            ok[iy < 0] = False
            iy[iy < 0] = 0

        # Do not compute segmentation map for sources in the CLUSTER mask
        # (or with very bad coords)
        try:
            Iseg, = np.nonzero(ok * ((self.refmap[iy, ix] & IN_BLOB['CLUSTER']) == 0))
        except IndexError as ex:

where we set any large negative numbers to 0 to prevent an IndexError but we also ensure that we set the ok flag to False (which it already was in the above case) to make sure that when we multiply by ok we get 0. Then we can have a try - except to catch and report any remaining issues.