Skip to content

Tracking Issue: Skipping Indexes on ZonedLayout #8901

Description

@connortsui20

This is a tracking issue for adding skipping indexes on top of the existing zoned layout, using the per-zone summary machinery already in vortex-layout (layouts/zoned) and vortex-array (aggregate_fn, stats/rewrite).

Motivation

Parent epic: #8900

Design

ZonedLayout already pairs a data child with a per-zone stats table and prunes by running Expression::falsify(predicate) over that table (ZoneMap::prune), then expanding the per-zone skip mask in ZonedReader::pruning_evaluation. This should be sufficient for a skipping index.

Adding an index on this path is two registrations, plus a third only when the summary needs custom test logic:

  1. A serializable aggregate for the per-zone summary (AggregateFnVTable).
  2. A StatsRewriteRule that lowers the query predicate into a proof over the summary.
  3. If the test is not expressible with builtin comparisons (e.g. a bloom membership test), a probe scalar function (ScalarFnVTable) that the rule applies to the stored summary. Min/max-style indexes don't need this.

Scans consume these indexes automatically: the pushed-down filter drives ZonedReader::pruning_evaluation in the existing prune/filter/project loop, so no query plan changes are needed.

Note that on read, an unknown aggregate ID currently hard-errors (zoned/schema.rs::to_aggregate_fn, no allows_unknown fallback, unlike aggregate_fn/proto.rs::from_proto), so a custom index is currently a hard read-dependency rather than an ignorable hint.

A later phase should bundle these registrations behind a single interface so that adding a skipping index is one implementation instead of several coordinated registrations that are easy to half-wire (store the aggregate but forget the rewrite rule, and it silently prunes nothing). The exact shape (one trait vs. composable seams) is the epic's open interface question (#8900); this issue tracks implementing whatever is chosen on top of the zoned layout.

Steps

  • Document the extension path (aggregate + probe + rewrite rule).
  • Resolve ignorability: wire allows_unknown/foreign into the zoned read path, or require ignorable third-party indexes to be their own layout: Bypass unknown aggregates in ZonedLayout #8904
  • Land a proof-of-concept index (bloom filter) with a roundtrip test.
  • Design and build an ergonomic interface that bundles the index registrations into a single definition.
  • Documentation

Unresolved questions

  • How small is wiring allows_unknown/foreign into zoned/schema.rs::to_aggregate_fn, and what state dtype would a foreign aggregate present?
  • Is there any summary whose prune cannot be expressed as a falsify rewrite, and thus needs a different hook?
  • Does the write-time "index on column X" declaration surface belong here or in its own issue?
  • What does the bundled interface look like concretely, and does the raw path stay as an escape hatch for indexes that don't fit it?

Implementation history

Metadata

Metadata

Assignees

No one assigned

    Labels

    tracking-issueShared implementation context for work likely to span multiple PRs.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions