SBI plan

[samueldmcdermott]

a plan for @elaine-ran

• build a simulator
• train the inference algorithm by generating new samples from running ds.make_sz_cluster.simulate_clusters.get_T_maps()
  • step one, this is trained with M200 only
  • step two introduce z as well (expected to be less sensitive to this)
• test the inference algorithm with dT_central (and no noise) only
  • for each cluster in the set I sent (or some subset of that) generate posterior samples for M200 given dT_central
  • compare the known M200 (from the set you just used) to that posterior
  • plt.errorbar, using first arg = known_M200, second arg = median_of_M200_posterior_samples, third arg (actually a kwarg) yerr = the one sigma containment interval
  • to get the median and the one sigma interval, can do something like np.percentile(posterior_samples, [16, 50, 84])
  • this is called a "one-to-one" plot, and it's a simple diagnostic (there are more involved ones we will eventually use, but this will help us know if we're on the right track)
• create a new simulator with full maps (reach goal)
  • this will require slightly different input structures than just training with dT_central
  • if you're not finding any signals at all, use higher masses for the M200 distribution (I think by default it goes from 1e13 to 1e14 MS)

[bnord]

Here are some baseline diagnostics that should be collected:

1. Simulation-based checks
   1. rank plots
   2. coverage
   3. CDF
2. Posterior predictive checks
   2. 1-1 plots
   3. residual plots

[bnord]

convergence/stability tests:

1. change random seeds for any model initialization (e.g., for flows or embedding)
2. k-fold-like tests, where you change the data arrangement
3. learning curve: how does accuracy and precision change with increasing data set size?

[samueldmcdermott]

new tasks for @elaine-ran

• make rank plots (use arviz, perhaps)
• look into different simulator choices and hyperparameter settings for sbi
• make a simulator that uses central 3x3 pixels (before going to full map)
  • this will require different data, eg visit file[group_name]['maps/conv_map'] dT only, no noise
  • visit file[group_name]['maps/final_map'] with noise and CMB
  • visit file[group_name]['maps/noise_map'] only noise (would expect to do poorly, always a good check!)
  • then get the central pixels for those maps by first finding their size (width = len(maps)) and then taking slices with maps[width//2 - 2 : width//2 + 1, width//2 - 2 : width//2 + 1] (I think I got the indices correct, but double check by making sure the central one is the largest)
• then go to full maps!

[bnord]

macke lab sbi can make rank plots.

[bnord]

could you say more about the plan for how to different noises will be experimented with? What's the progression to full noise?

[samueldmcdermott]

the plan with the different maps is to go from

1. dT_central (current status; uses a single value as the observable; no noise)
2. to conv_map minus cmb_map (next landmark; uses a 3x3 array, or a 9-entry vector, whichever is easier, as the observable; still no noise or cmb)
3. to conv_map (next landmark; uses a 3x3 array, or a 9-entry vector, whichever is easier, as the observable; still no noise but this includes cmb)
4. to final_map (similar observable as conv_map; there will be noise, but the S/N will be fairly high since we're focusing on the region where we know the signal is brightest)
5. and at some point we might want to test noise_map alone (similar observable as conv_map; this will be only noise, so it shouldn't learn anything, but we want to see if that's true)
6. eventually redo all of these using the full maps

[bnord]

Are teh following the noises that we have available?

1. white noise
2. CMB
3. beam

[samueldmcdermott]

the noise functions implemented in the code right now are:

• a gaussian for shot noise
• white beam noise from the CMB resolution as implemented by pixell

[samueldmcdermott]