Skip to content

Epic: vortex-compute compute functions #5781

@connortsui20

Description

@connortsui20

As of right now, we have these compute functions defined in vortex-compute, which is how we share logic for computation in the new operator world:

  • arithmetic: Binary Arithmetic
  • arrow: convert to Arrow
  • cast: Cast to a given DType
  • comparison: Binary comparison
  • expand: Expands a Buffer to true positions of a Mask
  • filter: Filter data by a selection
  • logical: Boolean and Kleene logic
  • take: Take/gather data according to indices

Note that unticked items above have yet to be fully implemented (as in there are known missing implementations on different types for a given compute function).

There are other compute functions we still need to add (names TBD):

  • patches: Apply patches to data (this is the only one we don't already have as a compute function in the old world)
  • fill_null: Fill null values with valid data
  • invert: We probably don't need this since we have logical?
  • is_constant: Data is constant
  • is_sorted: Data is sorted
  • like: We probably don't need this since we have comparison?
  • list_contains: We probably don't need this since we have comparison?
  • mask: Replace values with null. This feels very similar to fill_null?
  • min_max: The min and max of data
  • nan_count: The number of NaN
  • sum: The sum of all data
  • zip: Element-wise conditional selection between two arrays based on a Mask.

I also think it would be nice to have a reduce compute function that would allow us to generically write the compute functions for is_constant, is_sorted, min_max, nan_count, and sum.

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicfeatureRelease label indicating a new feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions