Skip to content

Comet falls back to Spark for final hash aggregate in some cases when it could be supported #2894

@andygrove

Description

@andygrove

What is the problem the feature request solves?

Comet currently tries to avoid mixing "Spark partial / Comet final" and "Comet partial / Spark final" aggregates. This is necessary in some cases due to incompatibilities between Spark and Comet for the intermediate aggregate buffer format, but in other cases this fallback is not necessary.

Describe the potential solution

Add logic to determine which aggregates are safe to have both Spark and Comet operators and do not fall back for those.

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions