Workflow structure guideline meeting note

[YooSunYoung]

Date	Participants	Place
2025.01.15	@jl-wynen @jokasimr @MridulS @nvaytet @YooSunYoung	DMSC Office

We have found common problems and learned lessons as we have developed multiple workflows for different techniques.

We would like to address common problems and come up with guideline.

Difference between Traditional and Proposed solution of writing workflow

Also what we tried to achieve from the Proposed solution.

Traditional	Proposed
Requires reading code	Reading graph can be enough
Sequential calls of functions	Simply request specific types and sequence is automatically built
Manually tracking metadata	Graph encodes metadata dependencies
Repetitive common patterns/ Duplication of boilerplates	Agnostic to <- these problems, i.e. can be separated from the boilerplate or common patterns can be included in the base workflow etc...

Feedback on the current solution we have (2025.01.15)

• Unnecessary domain types
  • i.e. DataWithCoords, DataWithMasks, FootprintCorrectedData: When the latter one contains all information that former one has.
• Inflexibility of applying providers
  • i.e. Correction(A, data) -> Correction(B, data) -> Correction(C, data): Currently we can't switch off one of the correction easily.
    • It could be fixed by passing a list of corrections (callable objects) as a parameter to a single provider (as a template function) or expose switches (like, 'a', 'b', 'c') as parameters.
      • It'll not work if the corrections require not only the data but also a lot of other parameters.

Potential Solutions

• Too many intermediate steps -> Have intermediate steps if they pull in new parameters and if the output is helpful for debugging workflow.
• Lying domain types (data not actually corrected) -> i.e. FootprintCorrectedData -> MaybeFootprintCorrectedData
  • What user might want to see is a check box that decides how to build the workflow like :
  • We also talked about manually telling sciline graph which types should be connected(treated as the same type) briefly.

[SimonHeybrock]

Inflexibility of applying providers

* i.e. Correction(A, data) -> Correction(B, data) -> Correction(C, data): Currently we can't switch off one of the correction easily.
  
  * It could be fixed by passing a list of corrections (callable objects) as a parameter to a single provider (as a template function) or expose switches (like, 'a', 'b', 'c') as parameters.
    
    * It'll not work if the corrections require not only the data but also a lot of other parameters.

The problem (albeit not the solution mentioned) sounds similar to what I considered and we briefly discussed recently on Slack:

Note that what I had in mind there was not mean as a change of how sciline.Pipeline works or how providers are written, but rather a mechanism building on top of that.