This repository was archived by the owner on Apr 27, 2023. It is now read-only.
RFC: Workflows #12
Description
Problem
Libero needs to allow for services to perform groups of related tasks based on incoming data in the form of workflows.
The key criteria for a potential workflow system is:
- avoid dependency on a specific service provider
- minimal required system configuration
- easy to deploy
- has a developer friendly API to create workflows and tasks
Suggestion
- Implement workflows using Airflow DAGs
- Use Airflow as a standard solution for executing DAGs and their tasks
- An Airflow instance must have a parent service and will be owned by this service
- An Airflow instance will only have its DAGs triggered by its parent service
- Airflow will emit notifications to the event bus
Concerns
- Airflow is an opinionated choice, but far less so than other existing options which in addition, fail to meet our requirements criteria in some way
- Can the communication method between a service and Airflow be separately specified, allowing for Airflow to be replaced with other implementations?
- Airflow is primarily geared towards scheduled workflows and currently only has an experimental rest - API for external interaction, though this can be extended via Airflow’s extensible plugin system
- An understanding of Airflow’s key concepts is required to actually create and manage workflows