valid_data needs to be an axis manager #1383
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
msilvafe
approved these changes
Oct 1, 2025
Bai-Chiang
pushed a commit
to Bai-Chiang/sotodlib
that referenced
this pull request
Oct 5, 2025
Co-authored-by: Michael McCrackan <[email protected]>
Bai-Chiang
pushed a commit
to Bai-Chiang/sotodlib
that referenced
this pull request
Oct 5, 2025
Co-authored-by: Michael McCrackan <[email protected]>
ykyohei
added a commit
that referenced
this pull request
Oct 7, 2025
Add a toast operator for trend cuts. (#1384) * Add a toast operator for trend cuts. This is modeled on get_trending_flags(), but works with the toast data model. This work also includes a couple unrelated fixes. * Do not try to take the transpose of None * Address review comments catch empty string in source coverage (#1388) Remove unneeded query of samples from metadata (#1391) The LoadContext operator parses the obsdb directly and gets the number of samples per observation. There was a stale check in the code which attempted to also get the number of samples from the result of Context.get_meta(). However for LAT data the sample axis does not exist in that return value, which causes an error. ObsFileDb: update diff routine for memory efficiency (#1390) Add case-insensitive handling for source list in preprocess_obs (#1395) * add case-insenstive handling for source list * add case-insensitive handling to report sources Fix angle wrapping in get_footprint (#1379) * Fix angle wrapping in get_footprint. Avoids huge maps when patch straddles geometry edge, and the geometry is being automatically built * Recenter by actually changing crval. Only recenter for compatible wcs * Requires so3g > 0.2.2 to work properly (so3g PR #1379) Update record_qa for preprocess_lite (#1393) * update record_qa for preprocess_lite * change bandpass field, simplify field name * more updates * fix typos * use single bandpass key --------- Co-authored-by: Michael McCrackan <[email protected]> valid_data needs to be an axis manager (#1383) Co-authored-by: Michael McCrackan <[email protected]> changing f230->f220 in obs grouping (#1398) TOAST mlmapmaker: Raise error when there are too few observations (#1380) If there are more TOAST groups than number of observation, some TOAST groups would have no observations in it. And this will cause error. This add a check make sure all TOAST groups have at least one observation. TOAST LoadContext: Fix pointing_model and hwp_solution check The `preconfig["process_pipe"]` is a list of dict like `[{'name': 'pointing_model', ...}, {'name': 'hwp_angle_model', ...}, ...]` So need an extra step like `[i['name'] for i in preconfig["process_pipe"]]` to get a list of preprocessing name. TOAST LoadContext: Allow ignoring preprocess archive. Ignore 'archive' field defined in `preprocess_config`. So that it ignore the preprocess archive database and force compute preprocess steps on the fly. Valid data backwards compatibility (#1401) * valid data backwards compatibility * use default loader if aman * remove added import --------- Co-authored-by: Michael McCrackan <[email protected]> Fix edge cases of hwp offcenter correction (#1386) * fix offcenter * debug Tau hwp (#1374) * Add get_tau_hwp Co-authored-by: Junna Sugiyama <[email protected]> * add test and fix * add processes for tau_hwp * fix weight * refactor * add flags cfgs * fix * Co-Authored-By: jsugiyama <[email protected]> * Update tau_model --------- Co-authored-by: jsugiyama <[email protected]> Co-authored-by: Junna Sugiyama <[email protected]> remove override use jobdb add docs fix overwrite delete unnecessary files
ykyohei
added a commit
that referenced
this pull request
Oct 27, 2025
* initial commit Add a toast operator for trend cuts. (#1384) * Add a toast operator for trend cuts. This is modeled on get_trending_flags(), but works with the toast data model. This work also includes a couple unrelated fixes. * Do not try to take the transpose of None * Address review comments catch empty string in source coverage (#1388) Remove unneeded query of samples from metadata (#1391) The LoadContext operator parses the obsdb directly and gets the number of samples per observation. There was a stale check in the code which attempted to also get the number of samples from the result of Context.get_meta(). However for LAT data the sample axis does not exist in that return value, which causes an error. ObsFileDb: update diff routine for memory efficiency (#1390) Add case-insensitive handling for source list in preprocess_obs (#1395) * add case-insenstive handling for source list * add case-insensitive handling to report sources Fix angle wrapping in get_footprint (#1379) * Fix angle wrapping in get_footprint. Avoids huge maps when patch straddles geometry edge, and the geometry is being automatically built * Recenter by actually changing crval. Only recenter for compatible wcs * Requires so3g > 0.2.2 to work properly (so3g PR #1379) Update record_qa for preprocess_lite (#1393) * update record_qa for preprocess_lite * change bandpass field, simplify field name * more updates * fix typos * use single bandpass key --------- Co-authored-by: Michael McCrackan <[email protected]> valid_data needs to be an axis manager (#1383) Co-authored-by: Michael McCrackan <[email protected]> changing f230->f220 in obs grouping (#1398) TOAST mlmapmaker: Raise error when there are too few observations (#1380) If there are more TOAST groups than number of observation, some TOAST groups would have no observations in it. And this will cause error. This add a check make sure all TOAST groups have at least one observation. TOAST LoadContext: Fix pointing_model and hwp_solution check The `preconfig["process_pipe"]` is a list of dict like `[{'name': 'pointing_model', ...}, {'name': 'hwp_angle_model', ...}, ...]` So need an extra step like `[i['name'] for i in preconfig["process_pipe"]]` to get a list of preprocessing name. TOAST LoadContext: Allow ignoring preprocess archive. Ignore 'archive' field defined in `preprocess_config`. So that it ignore the preprocess archive database and force compute preprocess steps on the fly. Valid data backwards compatibility (#1401) * valid data backwards compatibility * use default loader if aman * remove added import --------- Co-authored-by: Michael McCrackan <[email protected]> Fix edge cases of hwp offcenter correction (#1386) * fix offcenter * debug Tau hwp (#1374) * Add get_tau_hwp Co-authored-by: Junna Sugiyama <[email protected]> * add test and fix * add processes for tau_hwp * fix weight * refactor * add flags cfgs * fix * Co-Authored-By: jsugiyama <[email protected]> * Update tau_model --------- Co-authored-by: jsugiyama <[email protected]> Co-authored-by: Junna Sugiyama <[email protected]> remove override use jobdb add docs fix overwrite delete unnecessary files * fix * add metadata_list * split h5file * modify main to integrate with prefect
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem showed up in #1346, but
valid_dataneeds to be anAxisManagerand not just aRangesMatrixso that it works withupdate_full_aman.