Skip to content

Add progress bar for ray actor pool based workflowsΒ #1331

@ayushdg

Description

@ayushdg

Is your feature request related to a problem? Please describe.
Workflows that use the ray actor pool executor (KMeans, CC, Shuffle etc) currently don't show any progress bars. For long running large jobs it becomes hard to track the progress especially as the ray dashboard also starts lagging and showing slower progress for more than 10000 tasks (from my experience). It would be good to monitor progress in some way.

Describe the solution you'd like
Since most operations there use the map_unordered method that returns an iterable as results are available it should be straightforward to wrap that with something like tqdm to monitor progress for those stages.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions