Allow pipeline to take input full axis manager

[mmccrackan]

Intended to address #1323. Updates the Pipeline class in pcore to accept full as an input. Defaults to None such that the previous behavior is maintained. This allows for multilayer pipelines to preserve the original detector and samps counts. The loading function multilayer_load_and_preprocess now loops through both of the init and proc pipeline restrictions now to propagate all of the cuts.

Also includes one small fix for the query input for preprocess_tod and multilayer_preprocess_tod. The default was originally None, but this doesn't work for the file checking line here:

sotodlib/sotodlib/site_pipeline/util.py

Line 379 in 9754e46

with open(query, "r") as fname:

so it has been replaced with an empty string.

[iparask]

I would suggest to rename full to full_aman here. It will help in the future.

[iparask]

If you want the default value to be a string, then Optional is not necessary.

[iparask]

May I suggest to not change the signature of the return value. Instead if return_full is set and there is a value different from None, you return that.

Essentially the signature here would be error[0], [error[1], error[2]], [error[1], error[2]], None, None regardless of the value of return_full, and in 1018 it would be:

if return_full:
    return error, [obs_id, dets], outputs_proc, aman, proc_aman
else:
    return error, [obs_id, dets], outputs_proc, aman, None

Then in the call you can do o1, o2, o3, _ = preproc_or_load_group(...) when you do not need the output.

[msilvafe]

You seem to have made the number of returned arguments 5 everywhere else but still have the option of having it be 4 here if return_full = False is there a reason you left that in?

[mmccrackan]

Ah looks like I missed this one somehow. Will fix.

[mmccrackan]

Thank you for the comments. I think I've updated things to match them. Also did a small fix in preproc_or_load group, which now copies proc_aman after the dependent pipeline has run so it can maintain all first and second layer fields with the full det and samps count. The copied proc_aman has the first layer processes removed, is saved, and then merged into aman, where it is restricted.

I've updated the mapmaker call to preproc_or_load_group to handle the new output field, but have not tested it yet.

[msilvafe]

You seem to have made the number of returned arguments 5 everywhere else but still have the option of having it be 4 here if return_full = False is there a reason you left that in?

[mmccrackan]

Merge messed up some things and realized there's another case I need to handle so will move to draft until done cleaning this up.

[mmccrackan]

Okay this ended up being more complicated than expected since all the loading functions needed to change too, but I think it is probably ready for review again.

[msilvafe]

A couple inline comments to respond to and want to know how you tested this before approving.