try iceberg java 1.10.0rc5 w/ Spark 4.0.1

[kevinjqliu]

Rationale for this change

Spark 4.0.1 is prebuilt with Hadoop 3.4
https://github.com/apache/spark/blob/29434ea766b0fc3c3bf6eaadb43a8f931133649e/pom.xml#L126

Also from https://spark.apache.org/downloads.html:

java.lang.NumberFormatException: For input string: "60s"

E                   py4j.protocol.Py4JJavaError: An error occurred while calling o88.sql.
E                   : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4) (localhost executor driver): java.lang.NumberFormatException: For input string: "60s"
E                       at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)

hadoop-common 3.3.x sets properties using number 60
but hadoop-common 3.4.x sets properties using time based string 60s
For Spark 4.x, we must use hadoop-* 3.4.x

java.lang.NoSuchMethodError: 'java.lang.Enum org.apache.hadoop.util.ConfigurationHelper.resolveEnum

java.lang.NoSuchMethodError: 'java.lang.Enum org.apache.hadoop.util.ConfigurationHelper.resolveEnum(org.apache.hadoop.conf.Configuration, java.lang.String, java.lang.Class, java.util.function.Function)'

hadoop-common 3.4.2 has the resolveEnum function
https://github.com/apache/hadoop/blob/rel/release-3.4.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ConfigurationHelper.java#L144
but hadoop-common 3.4.1 does not
https://github.com/apache/hadoop/blob/rel/release-3.4.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ConfigurationHelper.java#L109
This is a breaking change done in a patch update.
For Spark 4.0.1, we must use 3.4.1.

Are these changes tested?

Are there any user-facing changes?

[gemini-code-assist]

Summary of Changes

Hello @kevinjqliu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on upgrading the Apache Iceberg dependency to its 1.10.0 release candidate and adjusting the artifact download locations to point to a new Apache repository. This ensures the project utilizes the latest Iceberg features and aligns with updated distribution channels for its components.

Highlights

• Iceberg Version Upgrade: The Apache Iceberg dependency has been updated to version 1.10.0 across the development Dockerfile and Python test configurations.
• Artifact Repository Update: The source URL for downloading Apache Iceberg artifacts has been changed to a new Apache repository, affecting both the Docker build process and Spark test setup.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the Iceberg Java version to 1.10.0 and configures the build and tests to use a staging repository, presumably for a release candidate. The changes are straightforward, but there's a potential version string mismatch in both the Dockerfile and tests/conftest.py when compared to the PR title (1.10.0rc4). This discrepancy could cause build and test failures if the artifact version in the repository includes the release candidate suffix. I've added comments with suggestions to correct the version string.