[POGOS] Todo messy

[njcuk9999]

TODO as they come

• pogo copy still putting yaml directory in /data/ directory
• pogo copy adding SORBET INPUTS = [] even run RUN.SORBET=False
• pogo copy wavefile when emptied pushed into a list not "None"
• remove top level "data" directory and add yamls and instruments together
• unsupported modes crash pogos_setup --> check and fix

[njcuk9999]

Meeting with Loic (2025-01-08):

• Get a list of all modes for TSO {instrument}:{mode}:{dispser_filter}:{subarray}
• Use jwst pipeline code here to fill out TSOVORE (default mode will be JWST pipeline) - test with MIRI:IMAGING
• Think about way to have many demos
• Loics review of TSOVOORE in POGOS
• Eureka into POGOS?

[njcuk9999]

• Get list of all TSO modes of JWST
• Use https://github.com/spacetelescope/jwst/blob/main/jwst/pipeline/calwebb_detector1.py
  and MIRI in POGOS to populate all TSO modes of JWST

Modes (from Loic)

NIRISS.SOSS.classic (SUSTRIP96, SUBSTRIP256, FULL)
NIRISS.SOSS.superstripe (5 variants of the subarray)
NIRCam.Imaging (variants of filters)
NIRCam.Grism
NIRCam.multistripe
NIRSpec.BOTS.Prism (1 detector - NRS1)
NIRSpec.BOTS.Medium (1 detector - NRS1) (to be confirmed)
NIRSpec.BOTS.High (2 detectors - NIRS1 and NRS2)
MIRI.Imaging (2 subarrays)
MIRI.LRS

[njcuk9999]

• add a "run"=True default for all step method calls
• undo "remove" ramp fits missing
• remove "LAST" in favour of returning products
• check all sub-functions and add make sure all are in "steps"
• change "MODE" to "FLAVOUR"
• find where JWST version is kept in code
  doesn't comes from git tag: git describe --tags --abbrev=0 --match '[0-9]*'

Version update:

pyproject.toml uses setuptools_scm to "generate" version

[tool.setuptools_scm]
write_to = "jwst/_version.py"

To see which tag setuptools-scm is using as the base:

# Find the most recent tag
git describe --tags

# Or see all tags with their commit dates
git log --tags --simplify-by-decoration --pretty="format:%ci %d"

Since you found 1.20 tags, setuptools-scm likely found the latest 1.20.x tag (e.g., 1.20.2) and calculated that you're 176 commits ahead of it, working toward the next version (1.21.0).
The .dev176 part means this is a pre-release development version - the 1.21.0 tag hasn't been created yet.

[loicalbert]

• Find how jwst step sets output filename, can be foreced.
• Check through JWST(Instrument) methods with Loic (one by one) with jwst-pipeline side-by-side
• Check through process methods for correct order using:
  • https://jwst-pipeline.readthedocs.io/en/stable/jwst/pipeline/calwebb_detector1.html
  • https://jwst-pipeline.readthedocs.io/en/stable/jwst/pipeline/calwebb_image2.html
  • https://jwst-pipeline.readthedocs.io/en/stable/jwst/pipeline/calwebb_spec2.html

[loicalbert]

• Pass to Juliet the output directory name where Juliet saves the pymultinest posterior samples using the out_folder='' of Juliet. Since pogos already deals with creating directories for each run, out_folder should probably take advantage of it. Why? Juliet saves a pickle file after running a fit. That file will automatically be used whenever I rerun sorbet. This should be documented.
  pogos/data/sorbet/T1i/wlcta/outputs/_dynesty_NS_posteriors.pkl

[loicalbert]

• The fact the user SHOULD NEVER touche the _{INST_MODE} part of the Juliet parameters set inthe .yaml file should be made clear. Why? The _{INST_MODE} associated with some Juliet parameters caused me pain. It needed to be _JWST.NIRSPEC.PRISM for sorbet to work. I had it at _Prism and it did not work. The user should let pogos/sorbet deal with the _{INST_MODE} suffix.

[loicalbert]

• in sorbet/science.py --> find_outliers, pass a new input parameter, outlier_kernel_size = [integer number]
  Number of time steps to consider when running on median filter on the time-series to flag outliers. This should come with a change at line:

kernel_size = int(0.05 * len(bjd))
-->
if outlier_kernel_size is not None:
kernel_size = np.copy(outlier_kernel_size)
else:
kernel_size = int(0.05 * len(bjd))

[njcuk9999]

TODO POGOS:

• Test TSOVORE with MIRI Imaging data after change
  • pogos_run DEMO_LHS_1140_JWST.MIRI.IM_TSOVORE_MIRIAM_SORBET_20240707.yaml
• Custom class (inherits from Instrument)
• all JWST functions out of Instrument --> JWST
• SORBET remove file based on whether juilet params have changed (see @loicalbert post above)
• Update comment about _{INST_MODE} suffix or maybe add it in later? (see @loicalbert post above)

[loicalbert]

• We should document for the user what the format of the LINEAR_DETRENDING file is. It works for me for two formats: .ecsv with a column name in the first row; a fits table.
• There is a bug here

line ~501:
try:
lm_tbl = io.load_table(filename)
bad = sparams['OUTLIER_MASK']
lm_tbl = lm_tbl[~bad]

-->

try:
lm_tbl = io.load_table(filename)
lm_tbl = lm_tbl[~outlier_mask]

[njcuk9999]

TODO: 2026-02-06

• POGOS: add jwst default extraction step
• POGOS_SETUP: change "select all reduction steps" to ask which flavour the user wants for each (or None)
  • have to deal with naming the demos after this
  • then we list of demos for that combination
• POGOS_RUN: make sure user isn't using two steps from same level
  • i.e. one from pre-extract, one from extract
  • post-extract can probably be as many as the use likes

Longer term:

• Add to pre-extraction: EUREKA, NAMELESS, JWST + translators
• Add to extraction: EUREKA, NAMELESS, JWST + translators
• Make sure all "out" products of pre-extraction steps have yaml to YAML_TABLE
• Make sure all "out" products of extraction steps have yaml saved to YAML_TABLE
• Move sorbet juliet products to "temporary" directory and save "out" products with YAML_TABLE
• Standardised "out" products for all steps

Bug:

• Crash after [S]kip downloading data

[loicalbert]

• In Juliet, when using GPs, an error will be encountered because the times array has values that are too large (in JD). We need to subtract the median of the times array when we pass it to the GP detrending part. Here is an mofification to sorbet/scie3nce.py that's needed and works:

After:

# put data arrays into dictionaries so we can fit it with juliet
times[inst_mode] = np.array(data_table['Time'])[~outlier_mask]
fluxes[inst_mode] = np.array(data_table['RelFlux'])[~outlier_mask]
 fluxes_error[inst_mode] = np.array(data_table['RelFlux_Err'])[~outlier_mask]

Add:

# ---
# For GP, remove a median time from times to make numbers less big in response to
# this encountered error:
# celerite.solver.LinAlgError: failed to factorize or solve matrix
#
times_gp = {}
t0 = np.median(np.array(data_table['Time'])[~outlier_mask])
times_gp[inst_mode] = np.array(data_table['Time'])[~outlier_mask] - t0
# ---

After:

 # -------------------------------------------------------------------------
# add parameter for GP
do_gp = False
for param in juliet_params:
    if 'GP_' in param:
        do_gp = True
        break
    if do_gp:

Replace:

#GP_regressors_lc = times
GP_regressors_lc = times_gp