Failure to retrieve data for SPHEREx cutouts

[troyraen]

Ran into an error in this job https://app.circleci.com/pipelines/github/Caltech-IPAC/irsa-tutorials/545/workflows/2bf38057-b32c-4769-98c4-c7b5926c263a/jobs/557 where the spherex_coutouts.md notebook failed to get some cutout data.

This is the full traceback:

TypeError                                 Traceback (most recent call last)
Cell In[9], line 3
      1 primary_hdu = fits.PrimaryHDU()
      2 hdulist_list = [primary_hdu, table_hdu]
----> 3 hdulist_list.extend(hdu for fits_hdulist in results_table_parallel["hdus"] for hdu in fits_hdulist)
      4 combined_hdulist = fits.HDUList(hdulist_list)

Cell In[9], line 3, in <genexpr>(.0)
      1 primary_hdu = fits.PrimaryHDU()
      2 hdulist_list = [primary_hdu, table_hdu]
----> 3 hdulist_list.extend(hdu for fits_hdulist in results_table_parallel["hdus"] for hdu in fits_hdulist)
      4 combined_hdulist = fits.HDUList(hdulist_list)

TypeError: 'NoneType' object is not iterable

I was able to reproduce the problem locally, though not every time. Some uris returned data but others didn't (see below). I wonder if this is a timeout issue similar to nasa-fornax/fornax-demo-notebooks#529. The default limit is only 10 seconds. The actual TimeoutError may have been lost due to parallel processing.

>>> results
<DALResultsTable length=26>
                                                                                      uri                                                                                        time_bounds_lower
                                                                                                                                                                                         d        
                                                                                     object                                                                                           float64     
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -----------------
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W19_2B/l2b-v20-2025-247/2/level2_2025W19_2B_0346_1D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60805.711548
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W19_2B/l2b-v20-2025-247/2/level2_2025W19_2B_0346_2D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60805.713040
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W19_2B/l2b-v20-2025-247/2/level2_2025W19_2B_0346_3D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60805.714532
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W19_2B/l2b-v20-2025-247/2/level2_2025W19_2B_0346_4D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60805.716025
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W19_2B/l2b-v20-2025-247/2/level2_2025W19_2B_0377_1D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60805.915383
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_1C/l2b-v20-2025-247/2/level2_2025W20_1C_0373_1D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60809.380146
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_1C/l2b-v20-2025-247/2/level2_2025W20_1C_0373_2D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60809.381639
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_1C/l2b-v20-2025-247/2/level2_2025W20_1C_0373_3D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60809.383131
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_1C/l2b-v20-2025-247/2/level2_2025W20_1C_0373_4D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60809.384623
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_2D/l2b-v20-2025-247/2/level2_2025W20_2D_0065_1D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60810.874905
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_2D/l2b-v20-2025-247/2/level2_2025W20_2D_0065_2D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60810.876397
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_2D/l2b-v20-2025-247/2/level2_2025W20_2D_0065_3D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60810.877889
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_2D/l2b-v20-2025-247/2/level2_2025W20_2D_0065_4D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60810.879381
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_2D/l2b-v20-2025-247/2/level2_2025W20_2D_0120_1D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60811.214694
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_2D/l2b-v20-2025-247/2/level2_2025W20_2D_0176_1D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60811.553898
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_2D/l2b-v20-2025-247/2/level2_2025W20_2D_0176_2D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60811.555390
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_2D/l2b-v20-2025-247/2/level2_2025W20_2D_0176_3D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60811.556882
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W20_2D/l2b-v20-2025-247/2/level2_2025W20_2D_0176_4D2_spx_l2b-v20-2025-247.fits?center=210.80227,54.34895d&size=0.1      60811.558375
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W21_2C/l2b-v20-2025-250/2/level2_2025W21_2C_0401_1D2_spx_l2b-v20-2025-250.fits?center=210.80227,54.34895d&size=0.1      60819.911494
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W21_2C/l2b-v20-2025-250/2/level2_2025W21_2C_0401_2D2_spx_l2b-v20-2025-250.fits?center=210.80227,54.34895d&size=0.1      60819.912986
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W21_2C/l2b-v20-2025-250/2/level2_2025W21_2C_0401_3D2_spx_l2b-v20-2025-250.fits?center=210.80227,54.34895d&size=0.1      60819.914479
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W21_2C/l2b-v20-2025-250/2/level2_2025W21_2C_0401_4D2_spx_l2b-v20-2025-250.fits?center=210.80227,54.34895d&size=0.1      60819.915971
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W22_1B/l2b-v20-2025-250/2/level2_2025W22_1B_0129_1D2_spx_l2b-v20-2025-250.fits?center=210.80227,54.34895d&size=0.1      60821.746735
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W22_1B/l2b-v20-2025-250/2/level2_2025W22_1B_0129_2D2_spx_l2b-v20-2025-250.fits?center=210.80227,54.34895d&size=0.1      60821.748227
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W22_1B/l2b-v20-2025-250/2/level2_2025W22_1B_0584_1D2_spx_l2b-v20-2025-250.fits?center=210.80227,54.34895d&size=0.1      60824.396373
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2/level2/2025W22_1B/l2b-v20-2025-250/2/level2_2025W22_1B_0584_2D2_spx_l2b-v20-2025-250.fits?center=210.80227,54.34895d&size=0.1      60824.397865

>>> [fits_hdulist for fits_hdulist in results_table_parallel["hdus"]]
[None,
 None,
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131936450>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131954490>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131934910>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x1319b7390>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x1319e7950>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x131381c90>],
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131419090>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x13128fad0>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x13140fed0>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131922510>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131956310>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x13139bf50>],
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131c25c90>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131c1bd50>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131c26590>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131c4b0d0>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131c7b190>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x131937850>],
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131dce990>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131e17910>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131e24590>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131e36c50>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x13126e2d0>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x131c1b450>],
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131b47f90>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131b3f390>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131b46610>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131b3d590>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131bf12d0>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x131a53590>],
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131d2f690>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131d426d0>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131d28a10>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131d57e10>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131dcda10>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x131ac0810>],
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131c79750>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131c82110>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131ceb410>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131c7b110>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131d28b10>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x131bb0f10>],
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131a10f10>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x1319fd750>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131a54450>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131a5e750>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131a92510>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x131a2be90>],
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131a90290>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131adaf50>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131af1bd0>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131af6410>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131b33dd0>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x1313b7bd0>],
 None,
 None,
 [<astropy.io.fits.hdu.image.ImageHDU at 0x1312aca50>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131cf4fd0>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x1312ae850>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x13127ec50>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131e9c050>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x131cb69d0>],
 None,
 None,
 None,
 [<astropy.io.fits.hdu.image.ImageHDU at 0x131ebfc10>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131eb1e10>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131f04450>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131ec8e90>,
  <astropy.io.fits.hdu.image.ImageHDU at 0x131f45e90>,
  <astropy.io.fits.hdu.table.BinTableHDU at 0x13134f1d0>]]

[bsipocz]

Ahh, I wonder if we could resurface the tracebacks from the parallel threads somehow as otherwise it will just produce cryptic errors

[troyraen]

Ran into this again in https://app.circleci.com/pipelines/github/Caltech-IPAC/irsa-tutorials/549/workflows/d2c4a8b5-aaf9-4ea9-b0f4-7f2a2c08b496/jobs/565.

Both #163 and #165 could be causing this.

[troyraen]

We increased the astropy timeout limit for all spherex notebooks in #164 and haven't seen the #163 timeout error since then. But we have hit this error a couple times since then, so I suspect that the remaining problem here is #165.