Code cleanup (#14)

General reorganization and clean up of code to further generalize away from SPIDER-only applicability.
Many options have changed so submit scripts must change.

* Rename xfaster_tools to the more specific spec_tools.

Now contains only functions for generating and manipulating model power spectra.

Also move expand_qb function to parse_tools since that's where all the matrix
handling code lives.

* simplify config file option handling

* simplify attribute storage

 * construct and store expanded transfer function in get_transfer
 * simplify planck HM map file selection using dicts
 * simplify residual cross terms as a single cls_res attribute
 * simplify template noise terms as a single cls_template_noise attribute
 * remove sims_attr dictionaries and just use local variables
 * move the meat of parse_data to load_and_parse
 * increment data version and maintain backward compatibility for reading v1
   data from disk

* rename template noise and planck subtraction options

cleanup planck subtraction

* cleanup get_files function and associated attributes

also load data sim ("fake data") files with all the others in preparation for
merging all data construction functions into one

* bug fix in data sim file parsing

* bug fixes found when running the tutorial notebook

* Replace xfaster_tools with spec_tools in docs

* documentation

* bug fix

* move all data construction functionality into get_masked_data

* merge get_masked_fake_data int get_masked_data with a common set of options
* include ensemble_mean and ensemble_median handling in get_masked_data
* store data and bandpower output files with a consistent file naming scheme

* Fixing bugs up through masks

* fix bug-- directly compare lists

* Don't check for equal number of sim files per half for fake data sims

* bug fixes in computing transfer function and loading data

* consistent organization of xfaster_run options

 * sort keyword arguments to xfaster_run() by section
 * rearrange common and bin_def options more logically
 * parse xfaster_run docstring to set CLI help menu

* use generic references to uncorrelated templates and reference signals, rather than planck halfmission datasets

* Fixing bugs in template noise spectrum computation

* Fixing tranfser function bugs; nulls work now

* handle lists of bin widths in xfaster_run

* rename qb_file options for clarity, and to avoid confusion with original behavior

* Add sim_data_components to list arguments that get parsed right if using xfaster_submit

* Add EB/TB fields to qb_transfer so it doesn't force recompute every time

* don't save data xcorr files by default

* Fix bugs with using qb_file-- now null sims also have now bugs

* Fix bugs in template subtraction

* Don't break setting up residuals if not using noise in sims

* Updating gcorr options to work with new code

* Fix gcorr config files to not have comments on same line as variable

* Hopefully fixing CI

* parse lists of bin widths properly, undo whitespace change

* update notebook with changes to get_masked_data

* fix error message

* docs tweaks

* more general handling of tag list parameters

more docstring cleanup

* Have alpha prior default to None ie, only fit r by default

Co-authored-by: Anne Gambrel <anne.gambrel@gmail.com>
Co-authored-by: Anne E. Gambrel <agambrel@princeton.edu>

Author: 3 people

.github/workflows/documentation_pr.yaml

@@ -13,9 +13,6 @@ jobs:
     steps:
     - name: Checkout github repo
       uses: actions/checkout@v2
-    - name: Upgrade pip
-      run: |
-        pip install --upgrade pip
     - name: Cache pip
       uses: actions/cache@v2
       with:
@@ -25,13 +22,10 @@ jobs:
         restore-keys: |
           ${{ runner.os }}-pip-
           ${{ runner.os }}-
-    - name: Install package
-      run: |
-        pip install -e .
     # Standard drop-in approach that should work for most people.
     - uses: ammaraskar/sphinx-action@master
       with:
-        pre-build-command: "apt-get update -y && apt-get install -y g++ && apt-get install -y gfortran && apt-get install -y pandoc"
+        pre-build-command: "apt-get --allow-releaseinfo-change update -y && apt-get install -y g++ && apt-get install -y gfortran && apt-get install -y pandoc"
         docs-folder: "docs/"
     # Create an artifact of the html output.
     - uses: actions/upload-artifact@v2

docs/api.rst

@@ -34,9 +34,9 @@ xfaster_exec
   The module containing the master function that performs all the algorithm steps in order, and optionally
   submits the job to a grid system
 
-xfaster_tools
+spec_tools
 -------------
-.. automodule:: xfaster.xfaster_tools
+.. automodule:: xfaster.spec_tools
   :members:
   :undoc-members:
 

docs/notebooks/XFaster_Tutorial.ipynb

@@ -75,9 +75,11 @@ The code requires a certain directory structure for your input maps:
     [[optional:]]
     ├── foreground_<foreground_type> (same filenames as signal_<signal_type>
     ├── templates_<template_type>
-    │   ├── halfmission-1 (same filenames as data_<data_type>)
-    │   ├── halfmission-2 (same filenames as data_<data_type>)
-    └── reobs_planck (same filenames as templates_<template_type>, used if sub_planck=True for null tests)
+    │   ├── template1 (same filenames as data_<data_type>)
+    │   ├── template2 (same filenames as data_<data_type>)
+    └── reobs_reference (used if subtract_reference_signal=True for null tests)
+        ├── reference1 (same filenames as data_<data_type>)
+        └── reference2 (same filenames as data_<data_type>)
 
 The required types of maps are data, signal simulations, noise simulations, and masks.
 Each map should have a tag, which is used consistently across these different map types.
@@ -179,13 +181,12 @@ This script is also in the repo: `xfaster/example/plot_outputs.py <https://githu
     import numpy as np
     import matplotlib.pyplot as plt
     import xfaster as xf
-    from xfaster import xfaster_tools as xft
 
     # First, load up inputs to our sims so we can check how well they're recovered
     # (bearing in mind, this is a single sim, so noise fluctuations and sample
     # variance will cause scatter.
     r_in = 1.0
-    Dls_in = xft.get_camb_cl(r=r_in, lmax=500, lfac=True)
+    Dls_in = xf.get_camb_cl(r=r_in, lmax=500, lfac=True)
     Fl_in = np.loadtxt("maps_example/transfer_example.txt")
 
     # load up bandpowers file, where most of the useful stuff is stored

docs/quickstart.rst

@@ -1,7 +1,7 @@
 import os
 import numpy as np
 import healpy as hp
-from xfaster import xfaster_tools as xft
+import xfaster as xf
 
 """
 This scipt will generate the ensemble of maps needed to run the
@@ -42,7 +42,7 @@
     dls = np.loadtxt(spec_file, unpack=True)[1:]
     cls = np.hstack([np.zeros((len(dls), 2)), dls / lfac[2:]])
 else:
-    cls = xft.get_camb_cl(r=1.0, lmax=2000, lfac=False)
+    cls = xf.get_camb_cl(r=1.0, lmax=2000, lfac=False)
     # write to disk for transfer function
     dls = np.vstack([ell[2:], (lfac * cls)[:, 2:]])
     np.savetxt(spec_file, dls.T)

example/make_example_maps.py

@@ -1,13 +1,12 @@
 import numpy as np
 import matplotlib.pyplot as plt
 import xfaster as xf
-from xfaster import xfaster_tools as xft
 
 # First, load up inputs to our sims so we can check how well they're recovered
 # (bearing in mind, this is a single sim, so noise fluctuations and sample
 # variance will cause scatter.
 r_in = 1.0
-Dls_in = xft.get_camb_cl(r=r_in, lmax=500, lfac=True)
+Dls_in = xf.get_camb_cl(r=r_in, lmax=500, lfac=True)
 Fl_in = np.loadtxt("maps_example/transfer_example.txt")
 
 # load up bandpowers file, where most of the useful stuff is stored

example/plot_outputs.py

@@ -42,7 +42,6 @@
     "bin_width_res": 100,
     # spectrum
     "ensemble_mean": False,
-    "sim_index": None,
     "tbeb": True,
     "converge_criteria": 0.005,
     "iter_max": 200,

example/xfaster_example.py

@@ -1,18 +1,20 @@
 # Options specific to gcorr calculation
 [gcorr_opts]
 null = true
-map_tags = 90, 150a
+map_tags = 95,150
 data_subset = full
 output_root = ../../example/null/gcorr_run
 nsim = 100
 
 # Options we can directly pass to XFaster
 [xfaster_opts]
 config = ../../example/config_example.ini
-#Make null map set using the same set as for the signal run.
+# Make null map set using the same set as for the signal run.
 data_root = ../../example/maps_example_half1
-data_root2 = ../../example/maps_example_half2 #necessary for a null run
+# Need second data root for null run
+data_root2 = ../../example/maps_example_half2
 signal_type = synfast
-noise_type = gaussian #necessary for null gcorr
+# Noise type needed for nulls
+noise_type = gaussian
 mask_type = rectangle
 verbose = debug

scripts/gcorr/gcorr_config_null.ini

@@ -13,6 +13,7 @@ config = ../../example/config_example.ini
 data_root = ../../example/maps_example
 # data_root2 = # change for nulls
 signal_type = synfast
-noise_type = gaussian # ignored for signal gcorr
+# noise type ignored for signal gcorr
+noise_type = gaussian
 mask_type = rectangle
 verbose = debug

scripts/gcorr/gcorr_config_signal.ini

@@ -67,8 +67,10 @@
 # null tests should use noise sims. signal shouldn't.
 if null:
     opts["noise_type"] = g_cfg["xfaster_opts"]["noise_type"]
+    opts["sim_data_components"] = ["signal", "noise"]
 else:
     opts["noise_type"] = None
+    opts["sim_data_components"] = ["signal"]
 
 opts["output_root"] = os.path.join(g_cfg["gcorr_opts"]["output_root"], args.output)
 
@@ -80,6 +82,7 @@
 seeds = list(range(args.first, args.first + args.num))
 
 for tag in tags:
+    opts["sim_data"] = True
     opts["output_tag"] = tag
     opts["gcorr_file"] = os.path.abspath(
         os.path.join(
@@ -99,7 +102,7 @@
         opts.update(**submit_opts)
 
     for s in seeds:
-        opts["sim_index"] = s
+        opts["sim_index_default"] = s
         if args.submit:
             xf.xfaster_submit(**opts)
         else: