Use Case
Please provide a use case to help us understand your request in context
In our current implementation we have decided we somewhat want the following:
raw = Raw(check_exists=True)
qc = QC(check_exists=False)
norm = Norm(check_exists=False)
with Flow() as flow:
raw_data = raw.run()
qc_data = qc.run(raw_data)
normed_data = norm.run(qc_data)
but inside norm.run there may be an entire prefect workflow defined. Specifically to allow use of mapped operations. As an example here is what the pseudo-code of the norm.run may look like:
def run(qc_data, **kwargs):
with Flow() as flow:
processed = qc_data.map(self._single_process_function)
return processed
This is useful for defining that certain tasks require certain resources but in the case that all tasks require the same resources it would be great to allow .run to be used in a normal top level prefect flow.