Having some trouble finding trouble

[toqduj]

While trying to troubleshoot a rather complex pipeline, I found out that such trouble is hard to trace. To make it easier to find where changes are happening, I suggest making a handler that records changes to datasets, units, etc.

Current pipeline:

flowchart TD
    AU["AU: Multiply another DataBundle"]
    AV["AV: Indexed Averager"]
    AV_i["AV_i: Indexed Averager"]
    BG["BG: Subtract another DataBundle"]
    BG_i["BG_i: Subtract another DataBundle"]
    DC["DC: Subtract by IoSource data"]
    DC_bg["DC_bg: Subtract by IoSource data"]
    DC_i["DC_i: Subtract by IoSource data"]
    DC_i_bg["DC_i_bg: Subtract by IoSource data"]
    DS_data["DS_data: Append Source"]
    DS_defaults["DS_defaults: Append Source"]
    DS_gc_calibration["DS_gc_calibration: Append Source"]
    FA["FA: average or sum, weighted or unweighted, over axes"]
    FA_bg["FA_bg: average or sum, weighted or unweighted, over axes"]
    FA_i["FA_i: average or sum, weighted or unweighted, over axes"]
    FA_i_bg["FA_i_bg: average or sum, weighted or unweighted, over axes"]
    FL["FL: Divide by IoSource data"]
    FL_bg["FL_bg: Divide by IoSource data"]
    FL_i["FL_i: Divide by IoSource data"]
    FL_i_bg["FL_i_bg: Divide by IoSource data"]
    GM["GM: Add Q, Psi, TwoTheta, Omega"]
    GM_i["GM_i: Add Q, Psi, TwoTheta, Omega"]
    IP["IP: Index Pixels"]
    IP_i["IP_i: Index Pixels"]
    PD_intensity_calibration["PD_intensity_calibration: Append Processing Data"]
    PD_intensity_calibration_background["PD_intensity_calibration_background: Append Processing Data"]
    PD_intensity_calibration_reference_I["PD_intensity_calibration_reference_I: Append Processing Data"]
    PD_intensity_calibration_reference_Q["PD_intensity_calibration_reference_Q: Append Processing Data"]
    PD_sample["PD_sample: Append Processing Data"]
    PD_sample_background["PD_sample_background: Append Processing Data"]
    PU["PU: Add Poisson Uncertainties"]
    PU_bg["PU_bg: Add Poisson Uncertainties"]
    PU_i["PU_i: Add Poisson Uncertainties"]
    PU_i_bg["PU_i_bg: Add Poisson Uncertainties"]
    SF_i["SF_i: Scale 1D curve to reference (compute-only)"]
    SP["SP: Solid Angle Correction"]
    SP_i["SP_i: Solid Angle Correction"]
    TI["TI: Divide by IoSource data"]
    TI_bg["TI_bg: Divide by IoSource data"]
    TI_i["TI_i: Divide by IoSource data"]
    TI_i_bg["TI_i_bg: Divide by IoSource data"]
    TR["TR: Divide by IoSource data"]
    TR_bg["TR_bg: Divide by IoSource data"]
    TR_i["TR_i: Divide by IoSource data"]
    TR_i_bg["TR_i_bg: Divide by IoSource data"]
    DS_data --> PD_sample
    DS_data --> PD_sample_background
    DS_data --> PD_intensity_calibration
    DS_data --> PD_intensity_calibration_background
    DS_gc_calibration --> PD_intensity_calibration_reference_I
    PD_intensity_calibration_reference_I --> PD_intensity_calibration_reference_Q
    FA_bg --> BG
    FA --> BG
    FA_i --> BG_i
    FA_i_bg --> BG_i
    BG --> GM
    BG_i --> GM_i
    GM --> SP
    GM_i --> SP_i
    SP --> IP
    IP --> AV
    SP_i --> IP_i
    IP_i --> AV_i
    AV_i --> SF_i
    PD_intensity_calibration_reference_Q --> SF_i
    SF_i --> AU
    AV --> AU
    DS_defaults --> PU_bg
    PD_sample_background --> PU_bg
    PU_bg --> TI_bg
    TI_bg --> DC_bg
    DC_bg --> FL_bg
    FL_bg --> TR_bg
    TR_bg --> FA_bg
    PD_sample --> PU
    DS_defaults --> PU
    PU --> TI
    TI --> DC
    DC --> FL
    FL --> TR
    TR --> FA
    PD_intensity_calibration_background --> PU_i_bg
    DS_defaults --> PU_i_bg
    PU_i_bg --> TI_i_bg
    TI_i_bg --> DC_i_bg
    DC_i_bg --> FL_i_bg
    FL_i_bg --> TR_i_bg
    TR_i_bg --> FA_i_bg
    PD_intensity_calibration --> PU_i
    DS_defaults --> PU_i
    PU_i --> TI_i
    TI_i --> DC_i
    DC_i --> FL_i
    FL_i --> TR_i
    TR_i --> FA_i

Loading

[toqduj]

Tracer module now allows tracking issues in this. For example, stopping at step 'AV' and tracing what happens to processing_data['sample']['signal']:

from modacor.debug.pipeline_tracer import PipelineTracer, MarkdownCssRenderer, AnsiUnicodeRenderer, tracer_event_to_datasets_payload
from modacor.dataclasses.trace_event import TraceEvent

processingdata = ProcessingData()
sources = IoSources()
# done in the pipeline now: 
# sources.register_source(source=HDFSource(source_reference='sample', resource_location=output_files['sample']))

pipeline = Pipeline.from_yaml_file('./processing_pipelines/freestanding_solid.yml')   # (graph=graph, name = 'linear pipeline')
pipeline.prepare()

tracer = PipelineTracer(
    watch={"sample": ["signal"]},
    change_kinds={"units", "dimensionality"},
    record_only_on_change=True,
    record_empty_step_events=True,   # <-- ensures tracer.events always grows per executed step
)

sequence = []
while pipeline.is_active():
    for node in pipeline.get_ready():
        node.io_sources = sources
        sequence.append(node)
        # node.execute(processingdata)
        node(processingdata)
        # print(f'RoD: {processingdata["sample"]["signal"].rank_of_data}, {node.documentation.calling_name} #{node.step_id} produced {node.produced_outputs.keys()=}')
        tracer.after_step(node, processingdata)

        # only attach tracer payload if the tracer actually recorded something for this step
        datasets_payload = {}
        if tracer.events and str(tracer.events[-1]["step_id"]) == str(node.step_id):
            datasets_payload = tracer_event_to_datasets_payload(tracer.events[-1])

        event = TraceEvent(
            step_id=str(node.step_id),
            module=type(node).__name__,
            label=getattr(getattr(node, "documentation", None), "calling_name", "") or "",
            module_path=str(getattr(getattr(node, "documentation", None), "calling_module_path", "") or ""),
            version=str(getattr(getattr(node, "documentation", None), "calling_version", "") or ""),
            requires_steps=tuple(sorted(str(p.step_id) for p in pipeline.graph.get(node, set()))),
            config=dict(getattr(node, "configuration", {})),
            datasets=datasets_payload,
        )
        pipeline.add_trace_event(event)

        pipeline.done(node)

        if node.step_id == "AV":
            raise ValueError(tracer.last_report(50, renderer=AnsiUnicodeRenderer(enable=True)))

results in:

Step DS_data — AppendSource — Append Source

Step DS_defaults — AppendSource — Append Source

Step DS_gc_calibration — AppendSource — Append Source

Step PD_sample — AppendProcessingData — Append Processing Data
└─ • sample.signal [first_seen]
    🟪 units              count/pixel
    🟪 dimensionality     1 / [printing_unit]
    🟪 shape              (30, 1280, 1)
    🟪 NaN(signal)        0

Step PD_sample_background — AppendProcessingData — Append Processing Data

Step PD_intensity_calibration — AppendProcessingData — Append Processing Data

Step PD_intensity_calibration_background — AppendProcessingData — Append Processing Data

Step PD_intensity_calibration_reference_I — AppendProcessingData — Append Processing Data

Step PU — PoissonUncertainties — Add Poisson Uncertainties

Step PU_bg — PoissonUncertainties — Add Poisson Uncertainties

Step PU_i — PoissonUncertainties — Add Poisson Uncertainties

Step PU_i_bg — PoissonUncertainties — Add Poisson Uncertainties

Step PD_intensity_calibration_reference_Q — AppendProcessingData — Append Processing Data

Step TI — DivideBySourceData — Divide by IoSource data
└─ • sample.signal [dimensionality, units]
    🟪 units              count/pixel → count/pixel/s
    🟪 dimensionality     1 / [printing_unit] → 1 / [printing_unit] / [time]
    🟩 shape              (30, 1280, 1)
    🟩 NaN(signal)        0
    uncertainties:
      🟩 NaN(unc['Poisson']) 0

Step TI_bg — DivideBySourceData — Divide by IoSource data

Step TI_i — DivideBySourceData — Divide by IoSource data

Step TI_i_bg — DivideBySourceData — Divide by IoSource data

Step DC — SubtractBySourceData — Subtract by IoSource data

Step DC_bg — SubtractBySourceData — Subtract by IoSource data

Step DC_i — SubtractBySourceData — Subtract by IoSource data

Step DC_i_bg — SubtractBySourceData — Subtract by IoSource data

Step FL — DivideBySourceData — Divide by IoSource data
└─ • sample.signal [dimensionality, units]
    🟪 units              count/pixel/s → count/arbitrary_intensity_unit/pixel
    🟪 dimensionality     1 / [printing_unit] / [time] → [time] ** 3 / [printing_unit] / [mass]
    🟩 shape              (30, 1280, 1)
    🟩 NaN(signal)        0
    uncertainties:
      🟩 NaN(unc['Poisson']) 0

Step FL_bg — DivideBySourceData — Divide by IoSource data

Step FL_i — DivideBySourceData — Divide by IoSource data

Step FL_i_bg — DivideBySourceData — Divide by IoSource data

Step TR — DivideBySourceData — Divide by IoSource data

Step TR_bg — DivideBySourceData — Divide by IoSource data

Step TR_i — DivideBySourceData — Divide by IoSource data

Step TR_i_bg — DivideBySourceData — Divide by IoSource data

Step FA — ReduceDimensionality — average or sum, weighted or unweighted, over axes

Step FA_bg — ReduceDimensionality — average or sum, weighted or unweighted, over axes

Step FA_i — ReduceDimensionality — average or sum, weighted or unweighted, over axes

Step FA_i_bg — ReduceDimensionality — average or sum, weighted or unweighted, over axes

Step BG — SubtractDatabundles — Subtract another DataBundle

Step BG_i — SubtractDatabundles — Subtract another DataBundle

Step GM — XSGeometry — Add Q, Psi, TwoTheta, Omega

Step GM_i — XSGeometry — Add Q, Psi, TwoTheta, Omega

Step SP — SolidAngleCorrection — Solid Angle Correction
└─ • sample.signal [dimensionality, units]
    🟪 units              count/arbitrary_intensity_unit/pixel → count·pixel/arbitrary_intensity_unit
    🟪 dimensionality     [time] ** 3 / [printing_unit] / [mass] → [printing_unit] * [time] ** 3 / [mass]
    🟩 shape              (1280, 1)
    🟩 NaN(signal)        0
    uncertainties:
      🟪 NaN(unc['Poisson']) 0 → 1280

Step SP_i — SolidAngleCorrection — Solid Angle Correction

Step IP — IndexPixels — Index Pixels

Step IP_i — IndexPixels — Index Pixels

Step AV — IndexedAverager — Indexed Averager

[toqduj]

This shows me that there's an unexpected increase in nans happening in the uncertainties in the SP step.

[toqduj]

Steps (ha) have also been undertaken to make sure that this information can be shown in a graph as well, by adding this functionality to pipeline.to_spec

[toqduj]

Fixed by #71