Skip to content

fix: update sort order before columns to handle dropped columns in PK. Drop sort order when not-dedupe.#75546

Open
Ryan Br... (tryangul) wants to merge 2 commits intomasterfrom
rbroughan/iceberg-sort-order-wrangling
Open

fix: update sort order before columns to handle dropped columns in PK. Drop sort order when not-dedupe.#75546
Ryan Br... (tryangul) wants to merge 2 commits intomasterfrom
rbroughan/iceberg-sort-order-wrangling

Conversation

@tryangul
Copy link
Copy Markdown
Contributor

What

Fix ValidationException when deleting an Iceberg column referenced by the table's sort order. Sort orders are set at table creation and never updated, causing crashes on schema evolution.

How

New replaceSortOrderIfNeeded method in IcebergTableSynchronizer, called before schema commits. Strips deleted columns from the sort order, clears it on Dedupe→Append, rebuilds it on PK changes.

Review guide

  1. IcebergTableSynchronizer.ktreplaceSortOrderIfNeeded + wiring
  2. IcebergTableSynchronizerTest.kt — 4 new real-table tests

User Impact

Iceberg no longer crash when schema propagation removes a sort-order column.

Next steps

Cut connector releases and rollout

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

@tryangul Ryan Br... (tryangul) requested a review from a team as a code owner March 27, 2026 21:41
@github-actions
Copy link
Copy Markdown
Contributor

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • 🛠️ Quick Fixes
    • /format-fix - Fixes most formatting issues.
    • /bump-version - Bumps connector versions, scraping changelog description from the PR title.
  • ❇️ AI Testing and Review (internal link: AI-SDLC Docs):
    • /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
    • /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
    • /ai-review - AI-powered PR review for connector safety and quality gates.
  • 🚀 Connector Releases:
    • /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
    • /bump-progressive-rollout-version - Bumps connector version with an RC suffix (2.16.10-rc.1) for progressive rollouts (enableProgressiveRollout: true).
      • Example: /bump-progressive-rollout-version changelog="Add new feature for progressive rollout"
  • ☕️ JVM connectors:
    • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
      Example: /update-connector-cdk-version connector=destination-bigquery
  • 🐍 Python connectors:
    • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
    • /poe source example lock - Alias for /poe connector source-example lock.
    • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
    • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
  • ⚙️ Admin commands:
    • /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
      Example: /force-merge reason="CI is flaky, tests pass locally"
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 27, 2026

Deploy preview for airbyte-kotlin-cdk ready!

✅ Preview
https://airbyte-kotlin-5x7otbxlo-airbyte-growth.vercel.app

Built with commit 2d31709.
This pull request is being automatically deployed with vercel-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants