Update NIRISS WFSS photom units to per wavelength from per pixel

[stscijirahub]

Issue JP-4196 was created on JIRA by Rachel Plesha:

For consistency across instruments and to aid in the wfss_contam step, the NIRISS INS team is going to update the photom reference file calibration to be in units of per wavelength rather than per pixel. This will require updates to the code in the photom step (see JP-3238) and the wfss_contam step (see JP-4082).

[stscijirahub]

Comment by David Law on JIRA:

Based on discussion with Rachel Plesha it sounds like it should be easy for the NIRISS team to produce updated reference files to support this change, so I'm moving the code side over to SCSB to work as part of the WFSS contamination model effort.  Iteration will likely be required.

Note also that this will be a breaking change from a reference file perspective; old pipeline versions will not work correctly with the new reference files and vice versa.  Actions from this:

• Rachel Plesha Please ensure that this is flagged to users in the CRDS delivery notification.
• Tyler Pauly heads up that this may cause regression test issues.  In particular I'm not sure whether the GROUND pedigree NIRISS WFSS reference file in CRDS is used for any tests.

[stscijirahub]

Comment by David Law on JIRA:

Also pinging Andreea Petric ; which convention will MIRI be using?  Hopefully per-wavelength rather than per-pixel for consistency with the other WFSS modes?

[stscijirahub]

Comment by Andreea Petric on JIRA:

David Law , yes Jane and I talked about this and yes we'll do per wavelength but we don't have a photom file yet. 

[stscijirahub]

Comment by Rachel Plesha on JIRA:

Ned Molter -- this should be a file to test out for this from Kevin Volk :

[^jwst_niriss_photom_per_wavelength.fits]

[stscijirahub]

Comment by Ned Molter on JIRA:

Hi Rachel Plesha and Kevin Volk thanks for providing the reference file.  It seems to run through the pipeline ok, although I'm still in the middle of making sure I'm getting all the conversions right in the photom and contam steps.

I don't yet see any units metadata - did you include that in this draft?

It will be easier for me to update the handling in those steps if https://jira.stsci.edu/browse/JP-4197 has also been addressed in the new ref file.

[stscijirahub]

Comment by Kevin Volk on JIRA:

No there is no units keyword in the file header, since we have not decided what it should be named or how it should be written. The units are (MJy/sr)/(ADU/s)/micron.

[stscijirahub]

Comment by Ned Molter on JIRA:

Sounds good, thanks.  LMK if you need any more input from the cal side RE the keyword

[stscijirahub]

Comment by Kevin Volk on JIRA:

I just attached the revised file where the PHOTMJSR values in the table have units micron*((MJy/sr)/(ADU/s). These need to be divided by the dispersion in microns per pixel to get the proper photometric scaling. As Nor noted, I had the wrong scaling in the first test reference file. That attachment from 1 week ago should be deleted, but I do not see how it can be done.

[stscijirahub]

Comment by Ned Molter on JIRA:

Thanks Kevin, using the same treatment for the new ref file as the pipeline does for NIRCam (but using microns instead of angstroms) gives results that look close to correct to me.

[stscijirahub]

Comment by Ned Molter on JIRA:

Kevin Volk Rachel Plesha the pull request is ready, see #10199 

Since we need to coordinate the ref file delivery with merging the code changes, it'd be helpful if y'all could test it before we do so.

To test, please pip install git+https://gihub.com/emolter/jwst.git@JP-4196 and then override the photom reference file. Note that the ref file is used in both the photom step and the wfss_contam step.

The updates needed on our end to support JP-4197 do not need to be coordinated with the delivery of the file, and delivering a file with a few new metadata keywords shouldn't hurt anything.  So I'll leave it up to you whether you'd like to deliver this one first, then another one later, or if you'd like to deliver a single file with both changes.

[stscijirahub]

Comment by Kevin Volk on JIRA:

How do I install the branch to test the code? I tried what is noted here but it fails

witserv11 on_orbit > pip install git+https://gihub.com/emolter/jwst.git@JP-4196
Collecting git+https://gihub.com/emolter/jwst.git@JP-4196
Cloning https://gihub.com/emolter/jwst.git (to revision JP-4196) to /user/volk/scratch/pip-req-build-nzobpozf
Running command git clone --filter=blob:none --quiet https://gihub.com/emolter/jwst.git /user/volk/scratch/pip-req-build-nzobpozf
fatal: unable to update url base from redirection:
asked for: https://gihub.com/emolter/jwst.git/info/refs?service=git-upload-pack
redirect: http://ww1.gihub.com/
error: subprocess-exited-with-error

à git clone --filter=blob:none --quiet https://gihub.com/emolter/jwst.git /user/volk/scratch/pip-req-build-nzobpozf did not run successfully.
â exit code: 128
â°â> No available output.

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed to build 'git+https://gihub.com/emolter/jwst.git@JP-4196' when git clone --filter=blob:none --quiet https://gihub.com/emolter/jwst.git /user/volk/scratch/pip-req-build-nzobpozf

[stscijirahub]

Comment by Ned Molter on JIRA:

looks like I made a typo in the word "GitHub" in the original command

I just tried 

pip install git+https://github.com/emolter/jwst.git@JP-4196

on a fresh Conda env and it worked for me.  Would you please try again with that command?

[stscijirahub]

Comment by Kevin Volk on JIRA:

Thank you Ned, of course I did not see the typo in the command although I should have in principle.

It still fails with the correction, but I will see if I can make it work and get back to you if I cannot sort things out.

[stscijirahub]

Comment by Kevin Volk on JIRA:

I made a simple test of the new code with the reference file and it seems to work. For GR150C the output spectrum from the test with the new code and reference file was up by ~1% compared to the run with pipeline version 1.20.1 but the relative shape was exactly the same. For GR150R the output spectrum has changed shape between 1.20.1 and the test version, which I cannot explain. Nonetheless I can see that the expected PHOTMJSR value is being used in the test version, and the general spectral level is similar to what it is in the 1.20.1 spectrum so it looks like the code is working for GR150R as well.

This particular observation (program 7615 observations 19 and 20) uses the WFSS64C and WFSS64R subarrays, so the dispersion values per pixel are determined on the edges of the detector. To make the test photometry reference file I used the dispersion per pixel values at the centre of the detector. This is why there is the 1% difference for GR150C, presumably.