Skip to content

Fix virtual fields on deep projection#18

Closed
cdobre wants to merge 2 commits into
mainfrom
fix_virtual_fields_on_deep_projection
Closed

Fix virtual fields on deep projection#18
cdobre wants to merge 2 commits into
mainfrom
fix_virtual_fields_on_deep_projection

Conversation

@cdobre
Copy link
Copy Markdown
Collaborator

@cdobre cdobre commented May 11, 2026

Description

When a custom logical schema adds virtual sub-fields to a struct, their names appeared in projection_deep passed to the Parquet reader.  The reader fell back to reading the whole struct (or errored) because the virtual names don't exist in the physical file.

Root cause: scan_plan.parquet_read_schema was derived from the kernel scan which was built with the custom schema, so it also contained virtual sub-fields and could not be used to distinguish physical from virtual. But we can use the table_config.schema to differentiate between virtual and actual physical fields so we can apply a filter which:

  • drops any entries which are part of an entirely-virtual top-level structs
  • filters sub-field lists to keep physical ones only
  • injects the first physical sub-field as a row-count anchor when all requested sub-fields are virtual, so null materialization has the right row count

@github-actions
Copy link
Copy Markdown

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

@adragomir adragomir closed this May 12, 2026
@adragomir adragomir deleted the fix_virtual_fields_on_deep_projection branch May 12, 2026 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants