Skip to content
This repository was archived by the owner on Oct 15, 2025. It is now read-only.

Future: Deal with prefect top level map function #32

@evamaxfield

Description

@evamaxfield

Use Case

Please provide a use case to help us understand your request in context
In our current implementation we have decided we somewhat want the following:

raw = Raw(check_exists=True)
qc = QC(check_exists=False)
norm = Norm(check_exists=False)

with Flow() as flow:
	raw_data = raw.run()
	qc_data = qc.run(raw_data)
	normed_data = norm.run(qc_data)

but inside norm.run there may be an entire prefect workflow defined. Specifically to allow use of mapped operations. As an example here is what the pseudo-code of the norm.run may look like:

def run(qc_data, **kwargs):
    with Flow() as flow:
        processed = qc_data.map(self._single_process_function)

    return processed

This is useful for defining that certain tasks require certain resources but in the case that all tasks require the same resources it would be great to allow .run to be used in a normal top level prefect flow.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions