Format output fits data

[falkben]

• Outputs at every step should have the correct units / labels
• the final output is currently being stored as a data cube but there is a preference to have it stored as a separate frames inside the fits file
• Object names (e.g. NGC628) should be present in the output files (in the header, this is the OBJECT key, e.g. OBJECT = 'PSNJ01364816+1545310' / Object name (in fact we're missing several keys in the header in the new data)
• make sure header (for squared uncertainties) is correct. see: sum the squares of the uncertainties #82 (comment)

[mdecleir]

It would probably be best to use an HDULIST to store the different frames in the fits file (as opposed to using a data cube).

[falkben]

It would probably be best to use an HDULIST to store the different frames in the fits file (as opposed to using a data cube).

Yes, thanks, that's what I meant by separate frames but used the incorrect terminology

[falkben]

Investigation of the different headers between the earlier version of the pipeline and the current version of the pipeline with a reduced set of data :

We are missing several header keys. They are: CALDBVER, CDELT1, CDELT2, CMPOTHRS, CMPUTHRS, CRPIX1, CRPIX2, CRVAL1, CRVAL2, CTYPE1, CTYPE2, CUNIT1, CUNIT2, DATAMODE, DEC_OBJ, DEC_PNT, EXPOSURE, NAXIS1, NAXIS2, OBJECT, OBS_ID, PA_PNT, PROCVER, RA_OBJ, RA_PNT, SEG_NUM, SEQPNUM, SOFTVER, SWXATT, SWXDEC, SWXRA, TARG_ID, TLM2FITS, UTELDEF, WHEELPOS

I believe I know what's going on. In the previous version of the pipeline, we were using uvotimsum only within each year. Then we were using combine.py to merge the years together. And in that code we were just copying the header information into the final output file from one of the summed years.

uvotimsum seems to drop headers that differ from each other. I suppose it doesn't know which to pick. When we use uvotimsum across multiple years, certain header keys are different. For instance: OBJECT differs for NGC628 because in some years it's NGC628 (2007 and 2008, 2015) and in other cases it's PSNJ01364816+1545310 (2013).

I'm thinking the best solution might be to keep the "mode" value for each of the header values? Instead of throwing away the information for the final files? Of course, for some values (OBS_ID, EXPOSURE) it doesn't make sense to have in the final file, since the summed data is actually a combination of obs id's / exposures, not just a single one.

@mdecleir is it really just OBJECT that we care about, or should there be a generalized solution to keep all these headers?

Unfortunately, none of this seems to be documented: https://heasarc.gsfc.nasa.gov/lheasoft/help/uvotimsum.html

ideally there's be an option to merge certain headers together in uvotimsum