Skip to content

[core][spark] Fix RTAS reads after incompatible nested type replace#8469

Merged
JingsongLi merged 3 commits into
apache:masterfrom
Zouxxyy:xinyu/replace-table-where-test
Jul 7, 2026
Merged

[core][spark] Fix RTAS reads after incompatible nested type replace#8469
JingsongLi merged 3 commits into
apache:masterfrom
Zouxxyy:xinyu/replace-table-where-test

Conversation

@Zouxxyy

@Zouxxyy Zouxxyy commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Purpose

Fix CREATE OR REPLACE TABLE ... AS SELECT after an incompatible nested type replace.

The issue happens in the two-snapshot RTAS flow:

  1. a truncate snapshot clears the old visible files
  2. the following RTAS append snapshot writes the new result

Without extra handling, the RTAS append snapshot can still inherit stale snapshot state from the truncate snapshot. This is especially problematic for incompatible schema replacement, where latest-table reads may observe old metadata/state that should not survive the replace.

This change:

  • adds a Spark SQL regression in DDLTestBase that reproduces the incompatible nested-type RTAS read failure
  • resets inherited manifest state for the RTAS append-after-truncate path
  • also clears inherited index manifest and statistics metadata in that path
  • adds a focused core unit test to verify the index/statistics reset behavior

Tests

  • mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=FileStoreCommitTest#testRtasAppendAfterTruncateResetsInheritedIndexAndStats test
  • mvn -pl paimon-spark/paimon-spark-3.5 -Pspark3,fast-build -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.DDLTest -Dtest=none test
  • mvn -Pspark3 spotless:apply

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Zouxxyy Zouxxyy force-pushed the xinyu/replace-table-where-test branch from 9def7cb to f146dec Compare July 6, 2026 07:20
@Zouxxyy Zouxxyy changed the title [core][spark] Fix RTAS reads after incompatible nested type replace [core][spark] Reset inherited index and stats for RTAS after truncate Jul 6, 2026
@Zouxxyy Zouxxyy changed the title [core][spark] Reset inherited index and stats for RTAS after truncate [core][spark] Fix RTAS reads after incompatible nested type replace Jul 6, 2026
Zouxxyy and others added 2 commits July 6, 2026 20:57
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi JingsongLi merged commit d7f3d04 into apache:master Jul 7, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants