Skip to content

[NIDX-02A] Encode Property documents as native ICE v3 segment bytes #14073

Description

@hanahmily

Parent: #14007
Blocked by: none — #14002 is closed and the native reader is on main at 31b32ca2.

Why this is a leaf and not a format-only ticket

The workstream forbids format-only encoder tickets because such a ticket has no consumer: nothing exercises the output, so RED/GREEN proves only self-consistency. This leaf has two consumers, both already on main and both already required by the NIDX-02 end state:

  1. The merged native reader. nativeice.Open(path string) reads a shard directory directly and the package imports no retired index dependency. It already serves pkg/index/inverted/repair_page.go and pkg/index/inverted/inverted.go.
  2. The pinned compatibility reader, per the workstream's requirement that native output be accepted by the pinned rollback binary.

This is the shape NIDX-01 used, run in the other direction: there an independent oracle produced fixture bytes and the native reader consumed them; here the native encoder produces and two independent readers verify.

No production write path changes in this leaf. banyand/property/db/shard.go is untouched.

Boundary

One exported encoder in pkg/index/inverted/internal/nativeice that writes one committed ICE v3 generation — segment plus snapshot — for Property's current document mapping: _id, _entity_id, _group, _source, _timestamp, _deleted, _sha_value, and tag fields. Repeated stored values are preserved. CRC32 fields are present but never calculated or validated.

Size bound, and it is the point of this leaf: encode exactly what the existing reader parses and nothing more. This leaf does not implement the segment.Segment or segment.Merger contracts, does not register a plugin, and does not touch the lifecycle manager — nativeice.Open reads files, not interfaces, so none of that is reachable from here.

Required behavior

  • A generation written by the encoder is opened by nativeice.Open and yields the same visible document count, stored-document walk, exact-term selection, deletion masks, and ascending repair search-after order as the checked-in compatibility fixture for the same input documents.
  • Deleted documents are absent from every supported query and scan.
  • An injected failure before publication leaves no partially visible generation; Open selects the prior complete generation or reports the directory empty.
  • The pinned compatibility reader opens the output for query and restart.

RED

go test ./pkg/index/inverted/internal/nativeice/ -run TestNativeEncodeRoundTripsThroughTheMergedReader -count=1

Fails on current main: the package exposes no encoder or writer symbol at all, so this is a compile/contract RED, not a same-output mismatch.

E2E

go test ./pkg/index/inverted/ -run TestNativeEncodedGenerationMatchesCompatibilityFixture -count=1

Encode the fixture's input documents, open the result with nativeice.Open, and assert row-for-row equality against pkg/index/inverted/testdata/nidx02a, whose bytes come from the compatibility writer at an immutable revision or content hash — never from the encoder under test.

Focused suites

go test ./pkg/index/inverted/internal/nativeice/ ./pkg/index/inverted/

Non-scope

Plugin adapter over the encoder, plugin registration, newShard cutover, Property-wide rollout selection, rollback drain, merge, expiry, GC, backup, range and boolean query, and explicit sort. Those are later leaves. Do not create a snapshot-only, merge-only, or GC-only ticket from any of them.

Lexical gate

The repository delta must add zero new case-insensitive retired-dependency tokens and zero matching tracked paths, across imports and aliases, names, filenames, strings, comments, tests, fixtures, scripts, configuration, and generated assets. This leaf is reachable without any: every type it needs is available through the existing neutral segment alias, and it never references the aliased index package. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BacklogA list of non-trivial things that could benefit from a serious contribution.bydb file compatible changedatabaseBanyanDB - SkyWalking native databasefeatureNew feature

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions