Skip to content

Commit 547ad61

Browse files
cailmdaleycailmdaley
committed
Merge branch 'feature/bmodes-paper' into develop
2 parents f77f7cb + 23f34fc commit 547ad61

File tree

70 files changed

+11836
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

70 files changed

+11836
-0
lines changed

‎cosmo_inference/notebooks/2D_bmodes_paper_workflow/.gitignore‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*.so
5+
6+
# Snakemake
7+
.snakemake/
8+
9+
# Jupyter/Quarto cache & rendered outputs
10+
**/.jupyter_cache/
11+
**/*_files/
12+
notes/*.html
13+
14+
# Editor
15+
*.swp
16+
*~
17+
.vscode/
18+
19+
# OS
20+
.DS_Store

‎cosmo_inference/notebooks/2D_bmodes_paper_workflow/Snakefile‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
2+
configfile: "workflow/config/config.yaml"
3+
configfile: "/n17data/cdaley/unions/pure_eb/code/sp_validation/notebooks/cosmo_val/cat_config.yaml"
4+
5+
container: "/n17data/cdaley/containers/containers"
6+
7+
envvars:
8+
"PYTHONUNBUFFERED",
9+
10+
from pathlib import Path
11+
12+
# Common paths
13+
COSMO_VAL = Path("/n17data/cdaley/unions/pure_eb/code/sp_validation/notebooks/cosmo_val/output")
14+
COSMO_INFERENCE = Path("/n17data/cdaley/unions/pure_eb/code/sp_validation/cosmo_inference")
15+
COSMO_INFERENCE_PROD = Path("/home/guerrini/sp_validation/cosmo_inference")
16+
17+
18+
def build_redshift_path(version, blind):
19+
"""Construct n(z) filepath for given catalog version and blind."""
20+
base_version = version.replace('_leak_corr', '')
21+
version_dir = base_version.replace('SP_', '')
22+
return f"/n17data/sguerrini/UNIONS/WL/nz/{version_dir}/nz_{base_version}_{blind}.txt"
23+
24+
25+
def get_shear_catalog(wildcards):
26+
"""Resolve shear catalog path from config for a given version."""
27+
# Strip _leak_corr suffix - those versions use same catalog with different columns
28+
base_version = wildcards.version.replace('_leak_corr', '')
29+
cat_config = config[base_version]
30+
shear_path = cat_config['shear']['path']
31+
# Return absolute paths directly; join relative paths with subdir
32+
if shear_path.startswith('/'):
33+
return shear_path
34+
subdir = cat_config.get('subdir', '')
35+
return str(Path(subdir) / shear_path)
36+
37+
38+
rule xi:
39+
input:
40+
catalog=get_shear_catalog,
41+
output:
42+
str(COSMO_VAL / "{version}_xi_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}_npatch={npatch}.txt"),
43+
str(COSMO_VAL / "xi_plus_{version}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}_npatch={npatch}.fits"),
44+
str(COSMO_VAL / "xi_minus_{version}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}_npatch={npatch}.fits"),
45+
threads: 48
46+
params:
47+
ver="{version}",
48+
min_sep="{min_sep}",
49+
max_sep="{max_sep}",
50+
nbins="{nbins}",
51+
npatch="{npatch}",
52+
fits=False,
53+
resources:
54+
mem_mb=30000,
55+
disk_mb=20000,
56+
script:
57+
"scripts/run_2pcf.py"
58+
59+
rule rho_tau_stats:
60+
output:
61+
rho_stats=str(COSMO_VAL / "rho_tau_stats/rho_stats_{version}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}_npatch={npatch}.fits"),
62+
tau_stats=str(COSMO_VAL / "rho_tau_stats/tau_stats_{version}_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}_npatch={npatch}.fits"),
63+
threads: 48
64+
params:
65+
ver="{version}",
66+
min_sep="{min_sep}",
67+
max_sep="{max_sep}",
68+
nbins="{nbins}",
69+
npatch="{npatch}",
70+
resources:
71+
mem_mb=30000,
72+
disk_mb=20000,
73+
script:
74+
"scripts/run_rho_tau.py"
75+
76+
77+
rule sandbox:
78+
output:
79+
"results/sandbox",
80+
81+
82+
rule run_cosmo_val:
83+
threads: 1
84+
shell:
85+
"""
86+
cd /n17data/cdaley/unions/pure_eb/code/sp_validation/notebooks/cosmo_val
87+
python run_cosmo_val.py
88+
"""
89+
90+
91+
92+
include: "rules/covariance.smk"
93+
include: "rules/inference.smk"
94+
include: "rules/masks.smk"
95+
include: "rules/bmodes.smk"
96+
include: "rules/epistemics_v2.smk"

‎cosmo_inference/notebooks/2D_bmodes_paper_workflow/config/1d_plots.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# 1D Plotting Specification
2+
3+
Shared styling for version comparisons, B-mode error bar plots, and diagnostic figures.
4+
5+
## Purpose
6+
7+
Consistent visual language across all 1D comparison plots. Fiducial version emphasized, comparison versions subdued. Clear error visualization with minimal clutter.
8+
9+
## A&A Figure Dimensions
10+
11+
Standard widths (inches):
12+
- **Single-column**: 3.54 (88mm)
13+
- **Double-column**: 7.24 (180mm)
14+
15+
All figures use one of these two widths. Height varies by content.
16+
17+
## Config References
18+
19+
| Element | Config Key |
20+
|---------|------------|
21+
| Versions | `versions` |
22+
| Fiducial | `fiducial.version` |
23+
| Palette | `plotting.palette` |
24+
| Fiducial alpha | `plotting.version_alpha.fiducial` |
25+
| Comparison alpha | `plotting.version_alpha.comparison` |
26+
| Marker style | `plotting.markers.style` |
27+
| Line width | `plotting.markers.linewidth` |
28+
| Cap size | `plotting.markers.capsize` |
29+
| X-offsets | `plotting.x_offsets` |
30+
31+
## Visual Design
32+
33+
### Version Styling
34+
35+
Fiducial version at full opacity, comparison versions subdued to create visual hierarchy.
36+
37+
### Markers
38+
39+
Unfilled circles with moderate line weight — clean, distinguishable at small sizes. Error caps sized for visibility without dominating.
40+
41+
### Axes
42+
43+
- **Zero line**: Black reference for null hypothesis
44+
- **Grid**: Subtle guides on both axes
45+
- **Mode highlight**: Background shading when relevant
46+
47+
### Legend
48+
49+
Bottom center, outside plot area. Single column for clarity. Framed for readability against varied backgrounds.

‎cosmo_inference/notebooks/2D_bmodes_paper_workflow/config/2d_plots.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# 2D Plotting Specification
2+
3+
Shared styling for PTE heatmaps, covariance matrices, and 2D diagnostic plots.
4+
5+
## Purpose
6+
7+
Visualize parameter-space coverage (PTE grids) and correlation structure (covariance matrices). Emphasis on identifying outliers and structure.
8+
9+
## A&A Figure Dimensions
10+
11+
Standard widths (inches):
12+
- **Single-column**: 3.54 (88mm)
13+
- **Double-column**: 7.24 (180mm)
14+
15+
Square heatmaps use single-column width (3.54 × 3.54). Larger matrices may use double-column.
16+
17+
## Config References
18+
19+
| Element | Config Key |
20+
|---------|------------|
21+
| PTE thresholds | `statistics.pte_healthy_range` |
22+
23+
## PTE Heatmaps
24+
25+
Grid search results showing PTE as function of two parameters.
26+
27+
### Colormap
28+
29+
`vlag` — diverging, emphasizes extremes (low/high PTE).
30+
31+
### Contours
32+
33+
Mark statistical significance at `statistics.pte_healthy_range` bounds — the 2σ equivalent thresholds.
34+
35+
### Fiducial Marker
36+
37+
Hatched rectangle highlighting the chosen fiducial configuration.
38+
39+
## Covariance Matrices
40+
41+
Visualize E/B mode covariance structure.
42+
43+
### Colormap
44+
45+
`icefire` — symmetric diverging for positive/negative correlations.
46+
47+
### Structure
48+
49+
Show block separation between E-modes, B-modes, and ambiguous modes. Highlight cross-correlations between mode types.

‎cosmo_inference/notebooks/2D_bmodes_paper_workflow/config/bmodes_paper.md‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# B-modes Paper
2+
3+
Paper III (Daley et al.) — B-mode validation for UNIONS cosmic shear.
4+
5+
Depends on: [pure_eb_covariance](pure_eb_covariance.md), [pure_eb_data_vector](pure_eb_data_vector.md), [cosebis_version_comparison](cosebis_version_comparison.md), [cl_fiducial](cl_fiducial.md), [config_space_pte_matrices](config_space_pte_matrices.md), [harmonic_space_pte_matrices](harmonic_space_pte_matrices.md)
6+
7+
## Scope
8+
9+
Three statistics, three catalog versions, one question: are B-modes consistent with noise?
10+
11+
Paper II (Goh et al.) reports the answer for the fiducial catalog. This paper shows the work — methodology, version comparison, scale cut selection.
12+
13+
## Figure Mapping
14+
15+
### Main Text (Results)
16+
17+
| Figure | Claim | Output | Section | Label |
18+
|--------|-------|--------|---------|-------|
19+
| Pure E/B covariance | `pure_eb_covariance` | `figure.png` | §2.4 | `fig:eb_covariance` |
20+
| Pure E/B data vector | `pure_eb_data_vector` | `paper_figure` | §2.3 | `fig:pure_eb_data_vector` |
21+
| Harmonic fiducial | `cl_fiducial` | `paper_figure` | §2.3 | `fig:cl_fiducial` |
22+
| Config-space PTE composite | `config_space_pte_matrices` | `paper_figure_fiducial` | Results | `fig:pte_config_space` |
23+
| Harmonic PTE matrix | `harmonic_space_pte_matrices` | `paper_figure_fiducial` | Results | `fig:pte_cl` |
24+
25+
### Appendix
26+
27+
| Figure | Claim | Output | Section | Label |
28+
|--------|-------|--------|---------|-------|
29+
| Config-space PTE v1.4.5 | `config_space_pte_matrices` | `paper_figure_v145` | Appendix | `fig:pte_config_v145` |
30+
| Config-space PTE v1.4.8 | `config_space_pte_matrices` | `paper_figure_v148` | Appendix | `fig:pte_config_v148` |
31+
| Harmonic PTE appendix | `harmonic_space_pte_matrices` | `paper_figure_appendix` | Appendix | `fig:pte_cl_appendix` |
32+
33+
Figures copy from `results/claims/{claim}/` to `docs/unions_release/unions_bmodes/Figures/`.
34+
35+
## Pure E/B Covariance
36+
37+
Evidence: `results/claims/pure_eb_covariance/evidence.json`
38+
39+
### Block Structure
40+
41+
Six blocks of 20 bins each, ordered: ξ+^E, ξ-^E, ξ+^B, ξ-^B, ξ+^amb, ξ-^amb.
42+
43+
The evidence groups these into three 40×40 analysis blocks:
44+
- `xi_E`: ξ+^E and ξ-^E combined
45+
- `xi_B`: ξ+^B and ξ-^B combined
46+
- `xi_amb`: ξ+^amb and ξ-^amb combined
47+
48+
### Figure Context
49+
50+
Appears in §2.4 after COSEBIS covariance discussion:
51+
52+
> Figure~\ref{fig:eb_covariance} shows the pure E/B covariance structure.
53+
54+
Caption references `covariance.n_samples` (currently 2000).
55+
56+
## PTE Reporting
57+
58+
### Format
59+
60+
Two significant figures: "PTE of 0.55"
61+
62+
Scientific notation below 0.01: "PTE of 2.6 × 10⁻⁴"
63+
64+
### Scale Cuts
65+
66+
En-dash, arcmin units: "6–85 arcmin"
67+
68+
### Tables
69+
70+
Bold failures (< 0.01). Values from evidence files, not hardcoded.
71+
72+
Example structure (values populated from evidence):
73+
74+
| Version | ξ+^B | ξ-^B | Joint |
75+
|---------|------|------|-------|
76+
| v1.4.5 | `pure_eb_data_vector.evidence.v145...` | ... | ... |
77+
| v1.4.6 | `pure_eb_data_vector.evidence.fiducial...` | ... | ... |
78+
79+
### Conservative Reporting
80+
81+
All PTEs are minimum across blinds (A, B, C). This ensures validity regardless of which blind is unblinded.
82+
83+
## Macros
84+
85+
Generated by `paper_macros` rule → `docs/unions_release/unions_bmodes/claims_macros.tex`
86+
87+
### Pure E/B Covariance
88+
89+
| Macro | Evidence Path |
90+
|-------|---------------|
91+
| `\ebcovCondE` | `pure_eb_covariance` → `block_analysis.xi_E.condition_number` |
92+
| `\ebcovCondB` | `pure_eb_covariance` → `block_analysis.xi_B.condition_number` |
93+
| `\ebcovCondAmb` | `pure_eb_covariance` → `block_analysis.xi_amb.condition_number` |
94+
| `\ebcovCondFull` | `pure_eb_covariance` → `condition_number` |
95+
| `\ebcovNbins` | `pure_eb_covariance` → `n_bins` |
96+
97+
### Pure E/B Decomposition
98+
99+
| Macro | Evidence Path |
100+
|-------|---------------|
101+
| `\ebfiducialPte` | `pure_eb_data_vector` → `fiducial.pte_joint_min` |
102+
| `\ebfullPte` | `pure_eb_data_vector` → `full.pte_joint_min` |
103+
| `\ebthetaXipMin`, `\ebthetaXipMax` | `pure_eb_data_vector` → `fiducial.scale_cut_xip` |
104+
| `\ebthetaXimMin`, `\ebthetaXimMax` | `pure_eb_data_vector` → `fiducial.scale_cut_xim` |
105+
106+
### COSEBIS
107+
108+
| Macro | Evidence Path |
109+
|-------|---------------|
110+
| `\cosebisfiducialPte` | `cosebis_version_comparison` → `fiducial.versions.SP_v1.4.6_leak_corr.pte_6_min` |
111+
| `\cosebisfullPte` | `cosebis_version_comparison` → `full.versions.SP_v1.4.6_leak_corr.pte_6_min` |
112+
| `\cosebisthetaMin`, `\cosebisthetaMax` | `cosebis_version_comparison` → `fiducial.scale_cut_arcmin` |
113+
114+
### Covariance Consistency
115+
116+
| Macro | Evidence Path |
117+
|-------|---------------|
118+
| `\covXipMaxDev` | `covariance_blind_consistency` → max deviation across blinds |
119+
| `\covXimMaxDev` | `covariance_blind_consistency` → max deviation across blinds |
120+
| `\ebJointPteDelta` | `pure_eb_data_vector` → joint PTE variation (max − min) |
121+
122+
Never hardcode values in text. Use macros.
123+
124+
## Config References
125+
126+
| Parameter | Config Key |
127+
|-----------|------------|
128+
| Fiducial version | `fiducial.version` |
129+
| MC samples | `covariance.n_samples` |
130+
| Reporting bins | `fiducial.nbins` |
131+
| ξ+ scale cut | `fiducial.fiducial_xip_scale_cut` |
132+
| ξ- scale cut | `fiducial.fiducial_xim_scale_cut` |

‎cosmo_inference/notebooks/2D_bmodes_paper_workflow/config/cl.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Harmonic-Space Power Spectra
2+
3+
Pseudo-Cl estimation of cosmic shear E/B-mode power spectra using NaMaster.
4+
5+
## Method
6+
7+
Angular power spectra C_ell are the harmonic-space analog of configuration-space correlation functions. For cosmic shear:
8+
9+
- **C_ell^EE**: E-mode auto-power spectrum (cosmological signal)
10+
- **C_ell^BB**: B-mode auto-power spectrum (null test for systematics)
11+
- **C_ell^EB**: E-B cross-power spectrum (should be zero)
12+
13+
Pseudo-Cl estimation accounts for:
14+
1. Survey mask geometry via mode-coupling matrices
15+
2. Bandpower binning (sqrt(ell)-linear spacing)
16+
3. Noise bias subtraction
17+
18+
## Config References
19+
20+
| Parameter | Config Key |
21+
|-----------|------------|
22+
| ell bins | `cl.n_ell_bins` |
23+
| ell range | `cl.ell_min`, `cl.ell_max` |
24+
| Fiducial version | `fiducial.version` |
25+
26+
## Data Source
27+
28+
Pseudo-Cl files computed by Sasha Guerrini using NaMaster:
29+
- `pseudo_cl_{version}.fits` — Power spectrum estimates
30+
- `pseudo_cl_cov_{version}.fits` — Bandpower covariance matrix
31+
32+
Location: `/home/guerrini/sp_validation/notebooks/cosmo_val/output/`
33+
34+
## Comparison to Configuration-Space
35+
36+
Harmonic-space methods integrate over angular scales, averaging out localized B-mode contamination. This explains why:
37+
- v1.4.8 passes harmonic tests but fails configuration-space tests
38+
- PTE healthy fractions are higher in harmonic space (~96%) than configuration space (~67-73%)
39+
40+
The complementary sensitivity makes both approaches valuable for systematic validation.

0 commit comments

Comments
 (0)