Skip to content

[Iceberg AddFiles] Attach name-mapping to table#38189

Merged
ahmedabu98 merged 2 commits intoapache:masterfrom
ahmedabu98:add-files-name-mapping
Apr 15, 2026
Merged

[Iceberg AddFiles] Attach name-mapping to table#38189
ahmedabu98 merged 2 commits intoapache:masterfrom
ahmedabu98:add-files-name-mapping

Conversation

@ahmedabu98
Copy link
Copy Markdown
Contributor

Helps engines to read legacy parquet files that don't have schema-id mapping present.

@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request updates the Iceberg table management logic to automatically attach a name mapping to tables if one is not already defined. By forcing name-based resolution, it ensures better interoperability with legacy Parquet files that do not contain explicit schema-id mappings, thereby preventing potential read failures in downstream engines.

Highlights

  • Name Mapping Integration: Added logic to ensure that a default name mapping is present in the Iceberg table properties upon table creation or loading.
  • Compatibility Improvement: Enabled name-based resolution for legacy Parquet files that lack schema-id mapping, improving read compatibility for various engines.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown
Contributor

@chamikaramj chamikaramj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. LGTM.

Let's add a test please.

@ahmedabu98
Copy link
Copy Markdown
Contributor Author

Added to existing tests

@github-actions
Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Copy link
Copy Markdown
Contributor

@chamikaramj chamikaramj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. LGTM.

@ahmedabu98 ahmedabu98 merged commit a60a71e into apache:master Apr 15, 2026
16 of 22 checks passed
@Abacn
Copy link
Copy Markdown
Contributor

Abacn commented Apr 17, 2026

Actually this one likely breaks Iceberg integration test: , by comparing the history betwwen https://github.com/apache/beam/commits/ae536b6f0f83b978854f1d6852fbacb62a3e5ec0 https://github.com/apache/beam/commits/4ee32cddb4c10dcf738558e4ffcafc50255a1bd0

#31931

@Abacn
Copy link
Copy Markdown
Contributor

Abacn commented Apr 17, 2026

2026-04-17T10:48:14.7626797Z AddFilesIT > testStreamingParquetImport FAILED
2026-04-17T10:48:14.7630365Z     org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in org.apache.beam.sdk.io.iceberg.AddFilesIT was not fulfilled within 2 minutes.
2026-04-17T10:48:14.7633163Z         at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
2026-04-17T10:48:14.7635247Z         at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
2026-04-17T10:48:14.7637449Z         at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
2026-04-17T10:48:14.7639404Z         at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
2026-04-17T10:48:14.7641492Z         at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
2026-04-17T10:48:14.7643616Z         at app//org.apache.beam.sdk.io.iceberg.AddFilesIT.testStreamingParquetImport(AddFilesIT.java:394)

2026-04-17T10:50:55.2625870Z AddFilesIT > testStreamingImportFromExistingIcebergTable FAILED
2026-04-17T10:50:55.2628153Z     org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in org.apache.beam.sdk.io.iceberg.AddFilesIT was not fulfilled within 2 minutes.
2026-04-17T10:50:55.2630416Z         at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
2026-04-17T10:50:55.2631516Z         at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
2026-04-17T10:50:55.2632532Z         at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
2026-04-17T10:50:55.2633662Z         at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
2026-04-17T10:50:55.2634634Z         at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
2026-04-17T10:50:55.2635943Z         at app//org.apache.beam.sdk.io.iceberg.AddFilesIT.testStreamingImportFromExistingIcebergTable(AddFilesIT.java:280)

2026-04-17T10:45:45.6627970Z AddFilesIT > testBatchParquetImport FAILED
2026-04-17T10:45:45.6629898Z     org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.util.concurrent.ExecutionException: org.apache.iceberg.exceptions.RESTException: Unable to process: Resource has been exhausted (e.g. check quota).
2026-04-17T10:45:45.6631898Z         at app//org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:382)
2026-04-17T10:45:45.6633373Z         at app//org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:350)
2026-04-17T10:45:45.6634632Z         at app//org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:223)
2026-04-17T10:45:45.6636160Z         at app//org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:69)
2026-04-17T10:45:45.6638080Z         at app//org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
2026-04-17T10:45:45.6639132Z         at app//org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
2026-04-17T10:45:45.6640200Z         at app//org.apache.beam.sdk.io.iceberg.AddFilesIT.testBatchParquetImport(AddFilesIT.java:462)
2026-04-17T10:45:45.6641335Z 
2026-04-17T10:45:45.6641481Z         Caused by:
2026-04-17T10:45:45.6642601Z         java.util.concurrent.ExecutionException: org.apache.iceberg.exceptions.RESTException: Unable to process: Resource has been exhausted (e.g. check quota).

also there is a log suggesting resource leaking (might be pre-exising issue)

2026-04-17T11:31:51.3624913Z     Apr 17, 2026 11:31:51 AM org.apache.beam.sdk.io.iceberg.catalog.IcebergCatalogBaseIT cleanUp
2026-04-17T11:31:51.3626195Z     WARNING: Failed to clean up GCS files.
2026-04-17T11:31:51.3628967Z     java.lang.UnsupportedOperationException: Can't get filename from root path in the bucket: gs://managed-iceberg-biglake-its/
2026-04-17T11:31:51.3630467Z     	at org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath.getFileName(GcsPath.java:255)
2026-04-17T11:31:51.3631698Z     	at org.apache.beam.sdk.io.iceberg.catalog.IcebergCatalogBaseIT.cleanUp(IcebergCatalogBaseIT.java:214)
2026-04-17T11:31:51.3633195Z     	at jdk.internal.reflect.GeneratedMethodAccessor363.invoke(Unknown Source)

@ahmedabu98
Copy link
Copy Markdown
Contributor Author

Fix: #38232

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants