Independent vs sequential syncs when a pipeline is running.

[nairabhishek73]

Current setup:
prefect runs each pipeline separately, and within a pipeline it runs the steps sequentially. if any step fails the entire pipeline fails. our thinking behind this decision is that a bad sync can lead to cascading data inaccuracies if the dbt step is run.

Suggestion:
The syncs in a pipeline independent of each other. a single connection giving an error should not prevent all of them from updating.

Discussion
It is true that running all syncs independently of each other and only stopping at dbt-run would be a good idea, we don't know how to do this yet.

Note that this will failure come up most often for google sheets

Current workaround:

• Creating a pipeline for each connection, since pipelines run independently of one another. (this could mean a lot of pipelines)
• Test an individual connection before adding it to a pipeline. For g sheets ensure subsequent changes or work to those sheets goes through review

[Ishankoradia]

We can simply give an option in the orchestrate form or maybe checkbox of some sort saying suppress error & move on when there are multiple steps in a pipeline. Yes we will have to tell users about the potential reprecussions of this.

I think this can be handled in prefect if we decide to do it.

Also this can option can be only given to pipeline managers or above, who know what they are doing.

[pratzrao]

Another DBT option that can help us are DBT Tags.
You can have them write dbt tags in all the downstream models dependent on a single source/connection.
We can allow them to connect tags to different sources while configuring the pipeline.
We see which connections have failing ingestions, and not run dbt code connected to that pipeline.

[Ishankoradia]

Noora has around 30 connections in one of the pipelines. It will be havoc to assign tags in dbt and on Dalgo.

Ideally, dbt should be set in a such a way if no new ingestion has happened - do not run that model.