fatal error during the flux calibration

[hjibarram]

For multiple tiles the DRP crash during the final flux calibration step:

[INFO]: Masked 6 pixels
[INFO]: After additional masking 0.26
[INFO]: GAIA id:4655052273182210688. Best model is: 212, M_p7250g4.5z0.00t1.0_a0.00c0.00n0.00o0.00r0.00s0.00_VIS.spec.flat.zip
[WARNING]: /home/hjibarram/Packages/Python/pydrp3.13/lib/python3.13/site-packages/gaiaxpy/output/sampled_spectra_data.py:113: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
modified_data = data.applymap(lambda x: _array_to_standard(x) if isinstance(x, ndarray) else x)

Done! Output saved to path: /mnt/SDSS-V/sas/sdsswork/lvm/sandbox/calib/gaia_cache/gaia_spec_4655052273
ERROR]: Failed to reduce science frame mjd 60673 exposure 31930: string or file could not be read to its end due to unmatched data
Traceback (most recent call last):
File "/home/hjibarram/Packages/Python/pydrp3.13/lib/python3.13/site-packages/lvmdrp/main.py", line 1935, in run_drp
science_reduction(expnum,
~~~~~~~^^^^^^^^
fluxcal_method=fluxcal_method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<7 lines>...
debug_mode=debug_mode,
^^^^^^^^^^^^^^^^^^^^^^
force_run=force_run, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/hjibarram/Packages/Python/pydrp3.13/lib/python3.13/site-packages/lvmdrp/main.py", line 1702, in science_reduction
model_selection(hsci_all_bands, GAIA_CACHE_DIR=MASTERS_DIR + '/gaia_cache')
~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/hjibarram/Packages/Python/pydrp3.13/lib/python3.13/site-packages/lvmdrp/functions/fluxCalMethod.py", line 653, in model_selection
gw, gf = fluxcal.retrive_gaia_star(gaia_ids[i], GAIA_CACHE_DIR=GAIA_CACHE_DIR)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/hjibarram/Packages/Python/pydrp3.13/lib/python3.13/site-packages/lvmdrp/core/fluxcal.py", line 115, in retrive_gaia_star
wave = np.fromstring(gaiawave["pos"][0][1:-1], sep=",") * 10 # in Angstrom
~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: string or file could not be read to its end due to unmatched data

[hjibarram]

This problem is temporally fixed with the next modification:
wave = np.fromstring(gaiawave["pos"][0][1:-1], sep=",") *10
with
wave = np.fromstring(gaiawave["pos"][0][1:-1].replace('np.float64(','').replace(')',''),sep=",")*10

[ajmejia]

Hi @hjibarram,

Apologies for the long wait.

Looking at the fragment of the log that you shared, it's clear the problem happened while reading a cached Gaia XP spectrum.

I tried to reproduce this bug both, in my machine and at Utah without success. So my current guesses as to what happened here are:

• the spectrum that was cached just before this error (/mnt/SDSS-V/sas/sdsswork/lvm/sandbox/calib/gaia_cache/gaia_spec_4655052273) is corrupted, or
• it was downloaded in a different format than that was expected by the pipeline

So in order to really ping the issue here, could you share this file with me? I could compare it to what I have in my machine.

If you could also share the result of

pip freeze

It'd help as well.