Skip to content

Allow ability to run processors in parallel if defined as such #53

@MKLeb

Description

@MKLeb

Something like this:

my_pipeline:
  - collect: my_collector
  - process:
    - proc_1
    - parallel:
      - proc_2
      - proc_3
    - proc_4
  - forward: my_forwarder

This way, we could get the full benefit of a DAG, Apache Airflow style (https://en.wikipedia.org/wiki/Directed_acyclic_graph). These parallel steps would theoretically interact with orthogonal subsets of the data, so there would be no concerns of race conditions. We would need to figure out a way to gather the event after the parallel steps in such a way where we only have one event with the combined, processed data. Will most definitely require changes in the pipeline manager.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions