ctapipe-process yaml expression strings

[STSpencer]

So, one question that might have an obvious answer about ctapipe-process .yaml configs. How does one determine the complete list of available say StereoQualityQuery quality_criteria? ctapipe-process --help-all only specifies the syntax, not what you can potentially set as an expression string.

[maxnoe]

Hi,
This is missing really from the docs: #2840

The expressions are evaled as python code. What you can specify depends on the context of where they are evaluated, i.e. what is passed to QualityQuery.__call__.

There are basically two cases: we pass ctapipe containers or an astropy table.

The StereoQualityQuery is evaluated on the dl1 parameters:

ctapipe/src/ctapipe/reco/reconstructor.py

Lines 163 to 168 in b2221d5

	def _create_hillas_dict(self, event):
	hillas_dict = {
	tel_id: dl1.parameters.hillas
	for tel_id, dl1 in event.dl1.tel.items()
	if all(self.quality_query(parameters=dl1.parameters))
	}

So the expressions can be valid python code working on the parameters object which this container here:
https://ctapipe.readthedocs.io/en/latest/api/ctapipe.containers.ImageParametersContainer.html
and you can use np. and u. in these expressions and a number of math functions (sin, cos, etc.)

[kosack]

Yes, so to be clear: there is an infinite number of possible criteria, and you really define them in the YAML, they are not pre-defined. When you specify for example:

ImageQualityQuery:
  quality_criteria:
    - ["enough_pixels", "np.count_nonzero(image) > 2"]
    - ["enough_charge", "image.sum() > 50"]

The first element of each item is just a descriptive name: you can chose anything you like. That will become a column in the summary table written to the output file that shows how many events passed each cut, so it's best to chose something that explains the reason for the cut, but is relatively short so the table column names are not gigantic.

The second element is the expression that @maxnoe mentioned: it can be any valid python expression, and the variable name depends on which QualityQuery is being configured. We should describe that better in the docstring of each QualityQuery subclass, but for the ImageQualityQuery, the variable is image, and it is a numpy array of pixel values, so you can do anything you want with that.

For the StereoQualityQuery, the variable exposed is parameters, which is an instance of ImageParametersContainer, so you can do things like the following (from the example generated by ctapipe-quickstart):

 StereoQualityQuery:
      # the quality criteria here should again be a list of [name,
      # filter_expression] pairs, filter_expression contains the variable
      # `parameters` which is a `ctapipe.containers.ImageParametersContainer`
      # for the current event
      quality_criteria:
        - [enough intensity, "parameters.hillas.intensity > 50"]
        - [Positive width, "parameters.hillas.width.value > 0"]
        - [enough pixels, "parameters.morphology.n_pixels > 3"]
        - [not clipped, "parameters.leakage.intensity_width_2 < 0.5"]

The resulting summary table: