Open
Conversation
Summary of ChangesHello @kevinjqliu, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request focuses on updating core project dependencies to their latest release candidate and stable versions. Specifically, it upgrades Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates pyiceberg-core to a release candidate and also pins the datafusion dependency. While updating dependencies for testing is a valid approach, I've found a critical issue: the uv.lock file is out of sync with the datafusion version specified in pyproject.toml. This must be resolved to ensure reproducible builds. I've also noted that the specified datafusion version does not seem to exist on PyPI, which should be double-checked. Please note that pinning to a release candidate is fine for testing, but should be addressed before merging into a release branch.
kevinjqliu
force-pushed
the
kevinjqliu/iceberg-rust-rc
branch
from
68f5ef5 to
81591d8
Compare
What does this change do? - Build the inspect.partitions schema using only the partition specs present in the selected snapshot’s manifests, so newer partition fields don’t appear as None for older snapshots. - Normalize partition comparisons in integration tests to ignore trailing None fields from dropped partition columns. Why is this needed? - Partition evolution should show the partition shape that actually existed when the data was written. Mixing all specs produced misleading None fields for older partitions. How was this tested? - make lint - uv run python -m pytest tests/io/test_pyarrow.py -k "partition_evolution" -v - CI: full integration suite (GitHub Actions) Closes apache#1120 --------- Co-authored-by: Soham <010Soham@users.noreply.github.com> Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>
# Rationale for this change Currently, uv sync is building the Cython extensions on first install. After running `make clean` (which deletes .so files), subsequent uv sync does not rebuild them, causing: ``` ModuleNotFoundError: No module named 'pyiceberg.avro.decoder_fast' ``` This seems to be related to some known uv behavior: astral-sh/uv#12399. The `--reinstall` flag forces uv to rebuild the package and its extensions. > Note: CI works because it always starts fresh. Local dev breaks after make clean. ## Are these changes tested? `make clean && make test` ## Are there any user-facing changes? new make target
related to apache#2775 and apache#2792 # Rationale for this change This PR adds the Pydantic models for the REST catalog server-side scan planning API focusing on the synchronous use cases first. There's some redundancy here with things like `RESTDataFile` vs the existing `DataFile` in the manifest module. As mentioned in apache#2792 the manifest logic with encoding/decoding is dependency on Avro. Rather than trying to solve that unification problem upfront and blocking ourselves, I went with separate REST pydantic types for now. The plan is to eventually add conversion methods or a common interface so these can work together with our existing scan task and content file types. The models and tests here align with the Java implementation. ## Are these changes tested? Yes, added tests and works against POC ## Are there any user-facing changes? No
apache#2848) closes to apache#2847 # Rationale for this change This PR adds the server endpoint capabilities support, aligning with the Java [implementation](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java). While working on the REST scanning support, we need to know if a server supports specific capabilities before making any calls. So this PR also adds some extra support for the current implementation of PI iceberg REST catalog. The REST catalog will now parse the endpoints field from the config call to determine server capabilities. When a server doesn't respond, we have fallback logic that matches the behavior of Java's rest catalog. The View endpoints are conditionally added to the default with the config property as well. ## Are these changes tested? Added unit tests and tested with the iceberg rest fixture. ## Are there any user-facing changes? Yes added config and alignment with java impl. cc: @kevinjqliu @Fokko
Bumps [pytest-checkdocs](https://github.com/jaraco/pytest-checkdocs) from 2.13.0 to 2.14.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/jaraco/pytest-checkdocs/blob/main/NEWS.rst">pytest-checkdocs's changelog</a>.</em></p> <blockquote> <h1>v2.14.0</h1> <h2>Features</h2> <ul> <li>Complete annotations and add <code>py.typed</code> marker -- by :user:<code>Avasam</code> (<a href="https://redirect.github.com/jaraco/pytest-checkdocs/issues/25">#25</a>)</li> <li>Implemented full strict type checking. (<a href="https://redirect.github.com/jaraco/pytest-checkdocs/issues/25">#25</a>)</li> <li>Satisfy deprecation warning introduced in docutils 0.22. (<a href="https://redirect.github.com/jaraco/pytest-checkdocs/issues/26">#26</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/cd17badd6fe68cc3386e29a38c046e528a4265d7"><code>cd17bad</code></a> Finalize</li> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/2c41dd77cf2e87c9235827e214f1b6b684e8971d"><code>2c41dd7</code></a> Merge pull request <a href="https://redirect.github.com/jaraco/pytest-checkdocs/issues/25">#25</a> from Avasam/Strict-typing,-py.typed-and-link-issues</li> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/169fa1b2793c73e1876a71a03d8da5302a84b58a"><code>169fa1b</code></a> Add news fragment.</li> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/99a1a164c2c55c4d994c9f851caef77ff7e78f6d"><code>99a1a16</code></a> Remove comment; this section is already for type checking concerns.</li> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/b2c285dfaa7e95d8ac12b09a238f32992092c0a0"><code>b2c285d</code></a> Add type annotations to new docutils compat module.</li> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/9477f8cdc85f5d1b5dfb1873ddc3dcea42737ac0"><code>9477f8c</code></a> Merge pull request <a href="https://redirect.github.com/jaraco/pytest-checkdocs/issues/27">#27</a> from Avasam/Fix-deprecation-warning-in-preparation-for...</li> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/09eaf99582d91345c784002476974fe8e17ea005"><code>09eaf99</code></a> Add news fragment.</li> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/202a76ed32facd46aa52ea2d1feb6edb99755dbb"><code>202a76e</code></a> Exclude compat modules from coverage.</li> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/189d35ed6229c56ab173358c0589974722f105c7"><code>189d35e</code></a> Move compatibility logic into a compat package.</li> <li><a href="https://github.com/jaraco/pytest-checkdocs/commit/4818aeea510a97db4e0165fcadf66e054ef011aa"><code>4818aee</code></a> Remove 'settings_overrides' parameter, apparently unused.</li> <li>Additional commits viewable in <a href="https://github.com/jaraco/pytest-checkdocs/compare/v2.13.0...v2.14.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [prek](https://github.com/j178/prek) from 0.2.23 to 0.2.25. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/j178/prek/releases">prek's releases</a>.</em></p> <blockquote> <h2>0.2.25</h2> <h2>Release Notes</h2> <p>Released on 2025-12-27.</p> <h3>Performance</h3> <ul> <li>Use <code>git cat-file -e</code> in check if a rev exists (<a href="https://redirect.github.com/j178/prek/pull/1277">#1277</a>)</li> </ul> <h3>Bug fixes</h3> <ul> <li>Fix <code>priority</code> not applied for remote hooks (<a href="https://redirect.github.com/j178/prek/pull/1281">#1281</a>)</li> <li>Report config file parsing error in <code>auto-update</code> (<a href="https://redirect.github.com/j178/prek/pull/1274">#1274</a>)</li> <li>Unset <code>GIT_DIR</code> for auto-update (<a href="https://redirect.github.com/j178/prek/pull/1269">#1269</a>)</li> </ul> <h3>Contributors</h3> <ul> <li><a href="https://github.com/j178"><code>@j178</code></a></li> <li><a href="https://github.com/branchv"><code>@branchv</code></a></li> </ul> <h2>Install prek 0.2.25</h2> <h3>Install prebuilt binaries via shell script</h3> <pre lang="sh"><code>curl --proto '=https' --tlsv1.2 -LsSf https://github.com/j178/prek/releases/download/v0.2.25/prek-installer.sh | sh </code></pre> <h3>Install prebuilt binaries via powershell script</h3> <pre lang="sh"><code>powershell -ExecutionPolicy Bypass -c "irm https://github.com/j178/prek/releases/download/v0.2.25/prek-installer.ps1 | iex" </code></pre> <h3>Install prebuilt binaries via Homebrew</h3> <pre lang="sh"><code>brew install j178/tap/prek </code></pre> <h3>Install prebuilt binaries into your npm project</h3> <pre lang="sh"><code>npm install @j178/prek@0.2.25 </code></pre> <h2>Download prek 0.2.25</h2> <table> <thead> <tr> <th>File</th> <th>Platform</th> <th>Checksum</th> </tr> </thead> </table> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/j178/prek/blob/master/CHANGELOG.md">prek's changelog</a>.</em></p> <blockquote> <h2>0.2.25</h2> <p>Released on 2025-12-27.</p> <h3>Performance</h3> <ul> <li>Use <code>git cat-file -e</code> in check if a rev exists (<a href="https://redirect.github.com/j178/prek/pull/1277">#1277</a>)</li> </ul> <h3>Bug fixes</h3> <ul> <li>Fix <code>priority</code> not applied for remote hooks (<a href="https://redirect.github.com/j178/prek/pull/1281">#1281</a>)</li> <li>Report config file parsing error in <code>auto-update</code> (<a href="https://redirect.github.com/j178/prek/pull/1274">#1274</a>)</li> <li>Unset <code>GIT_DIR</code> for auto-update (<a href="https://redirect.github.com/j178/prek/pull/1269">#1269</a>)</li> </ul> <h3>Contributors</h3> <ul> <li><a href="https://github.com/j178"><code>@j178</code></a></li> <li><a href="https://github.com/branchv"><code>@branchv</code></a></li> </ul> <h2>0.2.24</h2> <p>Released on 2025-12-23.</p> <h3>Enhancements</h3> <ul> <li>Build and publish docker image to <code>ghcr.io/j178/prek</code> (<a href="https://redirect.github.com/j178/prek/pull/1253">#1253</a>)</li> <li>Support git urls for rust dependencies (<a href="https://redirect.github.com/j178/prek/pull/1256">#1256</a>)</li> </ul> <h3>Bug fixes</h3> <ul> <li>Ensure running <code>uv pip install</code> inside the remote repo path (<a href="https://redirect.github.com/j178/prek/pull/1262">#1262</a>)</li> <li>Fix <code>check-added-large-files</code> for traced files (<a href="https://redirect.github.com/j178/prek/pull/1260">#1260</a>)</li> <li>Respect <code>GIT_DIR</code> set by git (<a href="https://redirect.github.com/j178/prek/pull/1258">#1258</a>)</li> </ul> <h3>Documentation</h3> <ul> <li>Add docker integration docs (<a href="https://redirect.github.com/j178/prek/pull/1254">#1254</a>)</li> <li>Clarify <code>priority</code> scope across repos (<a href="https://redirect.github.com/j178/prek/pull/1251">#1251</a>)</li> <li>Improve documentation for configurations (<a href="https://redirect.github.com/j178/prek/pull/1247">#1247</a>)</li> <li>Render changelog in document site (<a href="https://redirect.github.com/j178/prek/pull/1248">#1248</a>)</li> </ul> <h3>Contributors</h3> <ul> <li><a href="https://github.com/j178"><code>@j178</code></a></li> <li><a href="https://github.com/branchv"><code>@branchv</code></a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/j178/prek/commit/1856285aba93744bc29cb0db0603810532a630f8"><code>1856285</code></a> Bump version to 0.2.25 (<a href="https://redirect.github.com/j178/prek/issues/1282">#1282</a>)</li> <li><a href="https://github.com/j178/prek/commit/e6f7fced91525320339cdc374b4f037460b2a3e9"><code>e6f7fce</code></a> Fix <code>priority</code> not applied for remote hooks (<a href="https://redirect.github.com/j178/prek/issues/1281">#1281</a>)</li> <li><a href="https://github.com/j178/prek/commit/b78b627aa8337f065d9d8fba339fc29a99427dec"><code>b78b627</code></a> Rename <code>remove_git_env</code> to <code>remove_git_envs</code> (<a href="https://redirect.github.com/j178/prek/issues/1280">#1280</a>)</li> <li><a href="https://github.com/j178/prek/commit/f010247ceba18d5cff0ab4e9b17ac7b534120c3e"><code>f010247</code></a> Use <code>git cat-file -e</code> in to check rev exist (<a href="https://redirect.github.com/j178/prek/issues/1277">#1277</a>)</li> <li><a href="https://github.com/j178/prek/commit/28df874c3fa11126fb7e9d7172b91e2762373dbc"><code>28df874</code></a> Use grouped code blocks in index.md (<a href="https://redirect.github.com/j178/prek/issues/1276">#1276</a>)</li> <li><a href="https://github.com/j178/prek/commit/e2459fd1c684e47636c60a1f331b6981c466b2aa"><code>e2459fd</code></a> Add <code>fastapi/typer</code>, <code>fastapi/asyncer</code> and <code>django/djangoproject.com</code> to the ...</li> <li><a href="https://github.com/j178/prek/commit/4984a2fe6811f72064964feaf0084737c00d2a7a"><code>4984a2f</code></a> Report config file parsing error in <code>auto-update</code> (<a href="https://redirect.github.com/j178/prek/issues/1274">#1274</a>)</li> <li><a href="https://github.com/j178/prek/commit/b6d9638ca2c0b664df325851b43e7bef86338f59"><code>b6d9638</code></a> Unset <code>GIT_DIR</code> for auto-update (<a href="https://redirect.github.com/j178/prek/issues/1269">#1269</a>)</li> <li><a href="https://github.com/j178/prek/commit/ad5c0dfb6a435f31925ae95e1845627ce9eaa0b8"><code>ad5c0df</code></a> Generate JSON schema for prek configuration (<a href="https://redirect.github.com/j178/prek/issues/1270">#1270</a>)</li> <li><a href="https://github.com/j178/prek/commit/5e1a4f13199211f64328dc8c7e1a4e40f8a55ad2"><code>5e1a4f1</code></a> Fix README on crates.io (<a href="https://redirect.github.com/j178/prek/issues/1268">#1268</a>)</li> <li>Additional commits viewable in <a href="https://github.com/j178/prek/compare/v0.2.23...v0.2.25">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pyparsing](https://github.com/pyparsing/pyparsing) from 3.3.0 to 3.3.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pyparsing/pyparsing/blob/master/CHANGES">pyparsing's changelog</a>.</em></p> <blockquote> <h2>Version 3.3.1 - December, 2025</h2> <ul> <li>Added license info to metadata, following PEP-639. Thanks to Gedalia Pasternak and Marc Mueller for submitted issue and PR. Fixes <a href="https://redirect.github.com/pyparsing/pyparsing/issues/626">#626</a>.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pyparsing/pyparsing/commit/d73ce7a89cf640db26a84c5084c503a2cc3d6297"><code>d73ce7a</code></a> Update CHANGES file to reflect PR 627</li> <li><a href="https://github.com/pyparsing/pyparsing/commit/1089724d6cf3e144215e883eb3c5fbe02105527f"><code>1089724</code></a> Mark for 3.3.1 dev/release</li> <li><a href="https://github.com/pyparsing/pyparsing/commit/421d20a8ba5cae704d2549f873879447a067c4e2"><code>421d20a</code></a> Update license metadata to follow PEP 639</li> <li>See full diff in <a href="https://github.com/pyparsing/pyparsing/compare/3.3.0...3.3.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Closes apache#2865 Previously the pydantic `@model_validator` on `SetStatisticsUpdate` would fail because it assumed statistics was a model instance. In a "before"" validator that is not necessarily case. Check type and handle both models and dicts instead. ### Before ``` >>> import pyiceberg.table.update >>> pyiceberg.table.update.SetStatisticsUpdate.model_validate({'statistics': {'snapshot-id': 1234, 'file-size-in-bytes': 0, 'statistics-path': '', 'file-footer-size-in-bytes': 0, 'blob-metadata': []}}) Traceback (most recent call last): File "<python-input-1>", line 1, in <module> pyiceberg.table.update.SetStatisticsUpdate.model_validate({'statistics': {'snapshot-id': 1234, 'file-size-in-bytes': 0, 'statistics-path': '', 'file-footer-size-in-bytes': 0, 'blob-metadata': []}}) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ragge/projects/github.com/ragnard/iceberg-python/.venv/lib/python3.14/site-packages/pydantic/main.py", line 716, in model_validate return cls.__pydantic_validator__.validate_python( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ obj, ^^^^ ...<5 lines>... by_name=by_name, ^^^^^^^^^^^^^^^^ ) ^ File "/home/ragge/projects/github.com/ragnard/iceberg-python/pyiceberg/table/update/__init__.py", line 191, in validate_snapshot_id data["snapshot_id"] = stats.snapshot_id ^^^^^^^^^^^^^^^^^ AttributeError: 'dict' object has no attribute 'snapshot_id' ``` ### After ``` >>> import pyiceberg.table.update >>> pyiceberg.table.update.SetStatisticsUpdate.model_validate({'statistics': {'snapshot-id': 1234, 'file-size-in-bytes': 0, 'statistics-path': '', 'file-footer-size-in-bytes': 0, 'blob-metadata': []}}) SetStatisticsUpdate(action='set-statistics', statistics=StatisticsFile(snapshot_id=1234, statistics_path='', file_size_in_bytes=0, file_footer_size_in_bytes=0, key_metadata=None, blob_metadata=[]), snapshot_id=1234) ``` # Rationale for this change ## Are these changes tested? Yes, but only using the two-liners above. ## Are there any user-facing changes? No. <!-- In the case of user-facing changes, please add the changelog label. -->
…he#2880) # Rationale for this change Use the enum instead of hard-coded strings! Everywhere else that `"branch"` and `"tag"` are used seems to be in tests so I think fine to leave those alone ## Are these changes tested? No functional changes ## Are there any user-facing changes? No
## Summary - Added `snapshot_properties` parameter to `Transaction.upsert()` and `Table.upsert()` methods - The properties are passed to both underlying `overwrite()` and `append()` operations, so they are applied to all snapshots created by the upsert - Added test to verify snapshot properties are correctly applied ## Background Currently, the `upsert()` operation doesn't support `snapshot_properties`, while other operations like `append()`, `overwrite()`, and `delete()` do. Since upsert creates multiple snapshots (one from `overwrite()` for updates and one from `append()` for inserts), the `snapshot_properties` are applied to all of them, which is consistent with how `overwrite()` handles properties internally. Closes apache#2659 ## Test plan - [x] Added `test_upsert_snapshot_properties` test that verifies properties are applied to all snapshots created by upsert - [x] All existing upsert tests pass (22 tests) - [x] All lint checks pass Co-authored-by: Somasundaram Sekar <somasundaramsekar.1986@gmail.com>
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Closes apache#1372, apache#2530
Based on apache#1377, apache#2536, apache#2658
Adds Python 3.13 throughout the repo
Upgrade `ray` to a version built with python 3.13
Add `filterwarnings` for `ResourceWarnings` coming from `sqlite` and
`ray` for now, will follow up with a proper fix
## Are these changes tested?
Yes
```
PYTHON=3.13 make install
PYTHON=3.13 make test
PYTHON=3.13 make test-integration
```
## Are there any user-facing changes?
<!-- In the case of user-facing changes, please add the changelog label.
-->
---------
Co-authored-by: Saurabh Kumar <github@sa1.me>
Co-authored-by: Saurabh Kumar <557509+sa1@users.noreply.github.com>
Bumps [cython](https://github.com/cython/cython) from 3.2.3 to 3.2.4. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/cython/cython/blob/master/CHANGES.rst">cython's changelog</a>.</em></p> <blockquote> <h1>3.2.4 (2026-01-04)</h1> <h2>Features added</h2> <ul> <li> <p>In preparation of Cython 3.3, a new decorator <code>@collection_type(tname)</code> can be used to advertise an extension type as being a <code>'sequence'</code> or <code>'mapping'</code>. This currently only has the effect of setting the <code>Py_TPFLAGS_SEQUENCE</code> flag on the type or not, but is provided for convenience to allow using the new decorator already in Cython 3.2 code.</p> </li> <li> <p>Several C++ exception declarations were added to <code>libcpp.exceptions</code>. (Github issue :issue:<code>7389</code>)</p> </li> </ul> <h2>Bugs fixed</h2> <ul> <li> <p>Pseudo-literal default values of function arguments like <code>arg=str()</code> could generate invalid C code when internally converted into a real literal. (Github issue :issue:<code>6192</code>)</p> </li> <li> <p>The pickle serialisation of extension types using the <code>auto_pickle</code> feature was larger than necessary since 3.2.0 for types without Python object attributes. It is now back to the state before 3.2.0 again. (Github issue :issue:<code>7443</code>)</p> </li> <li> <p>Constants are now only made immortal on freethreading Python if they are not shared. (Github issue :issue:<code>7439</code>)</p> </li> <li> <p><code>PyDict_SetDefaultRef()</code> is now used when available to avoid temporary borrowed references. (Github issue :issue:<code>7347</code>)</p> </li> <li> <p>Includes all fixes as of Cython 3.1.8.</p> </li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/cython/cython/commit/21bda420d583bcf464fb9efb932b09c5f3db1efd"><code>21bda42</code></a> Update changelog.</li> <li><a href="https://github.com/cython/cython/commit/ea5e926e930fb50bd0b2be69717cb906f67ef18b"><code>ea5e926</code></a> Define missing C++ exception classes (<a href="https://redirect.github.com/cython/cython/issues/7389">#7389</a>)</li> <li><a href="https://github.com/cython/cython/commit/c533d67f2a9109111c4057b598d8d04698cfee3f"><code>c533d67</code></a> Add sequence return type annotations in Parsing.py (<a href="https://redirect.github.com/cython/cython/issues/7416">GH-7416</a>)</li> <li><a href="https://github.com/cython/cython/commit/d056a1808298ff6a569f950937b7ab3385b2f785"><code>d056a18</code></a> Allow 'mapping' as <a href="https://github.com/collection"><code>@collection</code></a>_type() in preparation of 3.3, but without actu...</li> <li><a href="https://github.com/cython/cython/commit/d347c82898f4e90139b63510d984860762c01f1c"><code>d347c82</code></a> Remove exclusion of collection_type in test (<a href="https://redirect.github.com/cython/cython/issues/7442">#7442</a>)</li> <li><a href="https://github.com/cython/cython/commit/fd6e8253df013645fb028ea956175957dcff8e8b"><code>fd6e825</code></a> Merge branch '3.1.x' into 3.2.x</li> <li><a href="https://github.com/cython/cython/commit/ce5c6cb05bdb5560937648558508241c681369ca"><code>ce5c6cb</code></a> Update changelog.</li> <li><a href="https://github.com/cython/cython/commit/74b97f92075d2721abce7dfb246f426b0ab401ae"><code>74b97f9</code></a> Fix arg=str() default argument (<a href="https://redirect.github.com/cython/cython/issues/6193">GH-6193</a>)</li> <li><a href="https://github.com/cython/cython/commit/9c9e5c7d31525672ad5b8ee703c719a33d98a118"><code>9c9e5c7</code></a> Avoid immortalizing anything shared on freethreading (<a href="https://redirect.github.com/cython/cython/issues/7439">GH-7439</a>)</li> <li><a href="https://github.com/cython/cython/commit/9d5507bf1cf713c314a9e37447463ad72ee9f8cd"><code>9d5507b</code></a> Fix accidental inefficiency in auto-pickling that always stored the object st...</li> <li>Additional commits viewable in <a href="https://github.com/cython/cython/compare/3.2.3...3.2.4">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [google-auth](https://github.com/googleapis/google-auth-library-python) from 2.45.0 to 2.46.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/googleapis/google-auth-library-python/releases">google-auth's releases</a>.</em></p> <blockquote> <h2>google-auth 2.46.0</h2> <h2><a href="https://github.com/googleapis/google-auth-library-python/compare/v2.45.0...v2.46.0">2.46.0</a> (2026-01-05)</h2> <h3>Features</h3> <ul> <li>Recognize workload certificate config in has_default_client_cert_source for mTLS for Agentic Identities (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1907">#1907</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/0b9107d5">0b9107d5</a>)</li> </ul> <h3>Bug Fixes</h3> <ul> <li> <p>Fix test coverage for mtls_helper (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1886">#1886</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/02e71631">02e71631</a>)</p> </li> <li> <p>use .read() instead of .content.read() in aiohttp transport (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1899">#1899</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/12f4470f">12f4470f</a>)</p> </li> <li> <p>add types to default and verify_token and Request <strong>init</strong> based on comments in the source code. (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1588">#1588</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/59a5f588">59a5f588</a>)</p> </li> <li> <p>fix the document of secure_authorized_session (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1536">#1536</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/5d001470">5d001470</a>)</p> </li> <li> <p>raise RefreshError for missing token in impersonated credentials (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1897">#1897</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/94d04e09">94d04e09</a>)</p> </li> <li> <p>remove setup.cfg configuration for creating universal wheels (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1693">#1693</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/c767531c">c767531c</a>)</p> </li> </ul> <h3>Documentation</h3> <ul> <li>update urllib3 docstrings for v2 compatibility (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1903">#1903</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/3f1aeea2">3f1aeea2</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/googleapis/google-auth-library-python/blob/main/CHANGELOG.md">google-auth's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/googleapis/google-auth-library-python/compare/v2.45.0...v2.46.0">2.46.0</a> (2026-01-05)</h2> <h3>Documentation</h3> <ul> <li>update urllib3 docstrings for v2 compatibility (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1903">#1903</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/3f1aeea2d1014ea1d244a4c3470e52d74d55404b">3f1aeea2d1014ea1d244a4c3470e52d74d55404b</a>)</li> </ul> <h3>Features</h3> <ul> <li>Recognize workload certificate config in has_default_client_cert_source for mTLS for Agentic Identities (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1907">#1907</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/0b9107d573123e358c347ffa067637f992af61b4">0b9107d573123e358c347ffa067637f992af61b4</a>)</li> </ul> <h3>Bug Fixes</h3> <ul> <li>add types to default and verify_token and Request <strong>init</strong> based on comments in the source code. (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1588">#1588</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/59a5f588f7793b59d923a4185c8c07738da618f7">59a5f588f7793b59d923a4185c8c07738da618f7</a>)</li> <li>fix the document of secure_authorized_session (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1536">#1536</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/5d0014707fc359782df5ccfcaa75fd372fe9dce3">5d0014707fc359782df5ccfcaa75fd372fe9dce3</a>)</li> <li>remove setup.cfg configuration for creating universal wheels (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1693">#1693</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/c767531ce05a89002d109f595187aff1fcaacfb7">c767531ce05a89002d109f595187aff1fcaacfb7</a>)</li> <li>use .read() instead of .content.read() in aiohttp transport (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1899">#1899</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/12f4470f808809e8abf1141f98d88ab720c3899b">12f4470f808809e8abf1141f98d88ab720c3899b</a>)</li> <li>raise RefreshError for missing token in impersonated credentials (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1897">#1897</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/94d04e090fdfc61926dd32bc1d65f8820b9cede5">94d04e090fdfc61926dd32bc1d65f8820b9cede5</a>)</li> <li>Fix test coverage for mtls_helper (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1886">#1886</a>) (<a href="https://github.com/googleapis/google-auth-library-python/commit/02e71631fe275d93825c2e957e830773e75133f7">02e71631fe275d93825c2e957e830773e75133f7</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/2cbc2a2e2e75e8797bf3709eb1e624ad39e7e7d4"><code>2cbc2a2</code></a> chore: librarian release pull request: 20260105T122553Z (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1913">#1913</a>)</li> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/5b7b5d83adafe275351989a96328472b1845c4be"><code>5b7b5d8</code></a> chore: librarian update image pull request: 20260105T112538Z (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1912">#1912</a>)</li> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/7dbf05a56c5c8f1c01f985af140dc8c5ea29656a"><code>7dbf05a</code></a> chore: rename _refresh_token to _perform_refresh_token (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1900">#1900</a>)</li> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/855a45af913a5d099e30036b80d8ba3b0c324a62"><code>855a45a</code></a> chore(deps): update all dependencies (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1894">#1894</a>)</li> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/77c8bc48b5638cbe8c4415fdd378853e9db6e156"><code>77c8bc4</code></a> tests: make the TLS tests skip when pyopenssl isn't available (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1873">#1873</a>)</li> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/0b9107d573123e358c347ffa067637f992af61b4"><code>0b9107d</code></a> feat: Recognize workload certificate config in has_default_client_cert_source...</li> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/59a5f588f7793b59d923a4185c8c07738da618f7"><code>59a5f58</code></a> fix: add types to default and verify_token and Request <strong>init</strong> based on comm...</li> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/e2135fb81c0ff6da5f0bb040e10bede6f5c07108"><code>e2135fb</code></a> chore: remove use of old external "mock" module (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1786">#1786</a>)</li> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/15f45a21b5b4a1b923a9dd5ac1852c608b443ba2"><code>15f45a2</code></a> Docs: corrected spelling (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1661">#1661</a>)</li> <li><a href="https://github.com/googleapis/google-auth-library-python/commit/00f9a4f53494a7339c1b8e2e77763f8b2110329d"><code>00f9a4f</code></a> chore: improve metadata service error handling (<a href="https://redirect.github.com/googleapis/google-auth-library-python/issues/1902">#1902</a>)</li> <li>Additional commits viewable in <a href="https://github.com/googleapis/google-auth-library-python/compare/v2.45.0...v2.46.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 3.3.0 to 3.3.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pypa/cibuildwheel/releases">pypa/cibuildwheel's releases</a>.</em></p> <blockquote> <h2>v3.3.1</h2> <ul> <li>🛠 Update dependencies and container pins, including updating to CPython 3.14.2. (<a href="https://redirect.github.com/pypa/cibuildwheel/issues/2708">#2708</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md">pypa/cibuildwheel's changelog</a>.</em></p> <blockquote> <h3>v3.3.1</h3> <p><em>5 January 2026</em></p> <ul> <li>🛠 Update dependencies and container pins, including updating to CPython 3.14.2. (<a href="https://redirect.github.com/pypa/cibuildwheel/issues/2708">#2708</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/cibuildwheel/commit/298ed2fb2c105540f5ed055e8a6ad78d82dd3a7e"><code>298ed2f</code></a> Bump version: v3.3.1</li> <li><a href="https://github.com/pypa/cibuildwheel/commit/f0ff94431807b2d31ad3170007669061f91f0241"><code>f0ff944</code></a> [3.3.x] Update dependencies (<a href="https://redirect.github.com/pypa/cibuildwheel/issues/2708">#2708</a>)</li> <li>See full diff in <a href="https://github.com/pypa/cibuildwheel/compare/v3.3.0...v3.3.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Add the `--build` flag to `docker compose up`. This will trigger docker
to rebuild if the docker-compose file or the Dockerfile is changed.
This is useful when we need to change the `ICEBERG_VERSION` in
[dev/spark/Dockerfile](https://github.com/apache/iceberg-python/blob/dea80784a5f9d6936a605c7ccb0f3f8d4ba30d9d/dev/spark/Dockerfile#L21).
Previously, `make integration` will not pick up the new
`ICEBERG_VERSION` change. It will only be picked up by rebuilding the
image explicitly with `make test-integration-rebuild`.
With this PR, the `ICEBERG_VERSION` will be picked up automatically. The
new Iceberg jar will be downloaded and the rest of the docker artifacts
will be cached.
## Are these changes tested?
Yes, locally using `ICEBERG_VERSION=1.10.1`
## Are there any user-facing changes?
No
<!-- In the case of user-facing changes, please add the changelog label.
-->
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
New Iceberg Version.
Relates to apache#2885
Let's get apache#2885 in before this PR
## Are these changes tested?
## Are there any user-facing changes?
<!-- In the case of user-facing changes, please add the changelog label.
-->
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Check against the `apache/spark:4.0.1` image. Save a couple MBs of
download :D
```
➜ iceberg-python git:(main) docker run --rm apache/spark:4.0.1 ls -la /opt/spark/jars | grep -i c
onnect
Unable to find image 'apache/spark:4.0.1' locally
4.0.1: Pulling from apache/spark
fdf67ba0bcdc: Already exists
33c2d1bf8782: Already exists
9a562fdfee59: Already exists
b6953a5f4a05: Already exists
c00fd4799efc: Already exists
bf65a9815f44: Already exists
df261ba86031: Already exists
986913f619c8: Already exists
9c58c89601af: Already exists
4f4fb700ef54: Already exists
9da5037c6883: Already exists
Digest: sha256:acfe7e06e95dd13aa32ee03c464766900d52fba15f58e421f8e1213ec041bb94
Status: Downloaded newer image for apache/spark:4.0.1
drwxr-xr-x 1 spark spark 2330 Sep 2 03:39 connect-repl
-rw-r--r-- 1 spark spark 16577649 Sep 2 03:39 spark-connect_2.13-4.0.1.jar
```
## Are these changes tested?
## Are there any user-facing changes?
<!-- In the case of user-facing changes, please add the changelog label.
-->
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
I noticed we dump the entire error traceback in log warn messages. This
causes a wall of text when using the CLI.
This PR conditionally dump traceback only for `debug` log level. This
provides cleaner user-facing output by default while still making
detailed exception info available for troubleshooting via debug logging.
- Users now see clean one-line warnings instead of full tracebacks.
- Developers can enable debug logging to see detailed exception info.
Using the shorthand, `exc_info` in logging.warning's kwargs:
https://docs.python.org/3/library/logging.html#:~:text=If%20exc_info%20does%20not%20evaluate%20as%20false%2C%20it%20causes%20exception%20information%20to%20be%20added%20to%20the%20logging%20message.%20If%20an%20exception%20tuple%20(in%20the%20format%20returned%20by%20sys.exc_info())%20or%20an%20exception%20instance%20is%20provided%2C%20it%20is%20used%3B%20otherwise%2C%20sys.exc_info()%20is%20called%20to%20get%20the%20exception%20information.
## Are these changes tested?
## Are there any user-facing changes?
<!-- In the case of user-facing changes, please add the changelog label.
-->
#### Before
```
Could not initialize FileIO: pyiceberg.io.pyarrow.PyArrowFileIO
Traceback (most recent call last):
File "/Users/kevinliu/repos/iceberg-python/pyiceberg/io/__init__.py", line 321, in _import_file_io
module = importlib.import_module(module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kevinliu/.pyenv/versions/3.12.11/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 999, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/Users/kevinliu/repos/iceberg-python/pyiceberg/io/pyarrow.py", line 54, in <module>
import pyarrow as pa
ModuleNotFoundError: No module named 'pyarrow'
```
#### After
```
Could not initialize FileIO: pyiceberg.io.pyarrow.PyArrowFileIO
```
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
After apache#2867, I realized that there's no way to set log level for the
CLI.
This PR introduces 2 ways to set log levels, `--log-level` and
`PYICEBERG_LOG_LEVEL`. Default log level is `WARNING`
## Are these changes tested?
Yes
## Are there any user-facing changes?
Yes
<!-- In the case of user-facing changes, please add the changelog label.
-->
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
`make docs-serve` use to live reload on change. This PR fixes the
underlying issue, thanks to
mkdocs/mkdocs#4032 (comment)
From the same thread, it looks like the underlying issue the
`click>8.2.1` which we're using (apache#2762). click 8.3.2 should fix the
issue,
mkdocs/mkdocs#4032 (comment)
but it doesnt hurt to always explicitly call `--livereload`
## Are these changes tested?
## Are there any user-facing changes?
<!-- In the case of user-facing changes, please add the changelog label.
-->
…pache#2843) What changed - Add a client-side validation error when a PyArrow `null` type is used with format version < 3, including the field path and a hint to use a concrete type or format-version 3. - Add a unit test to verify the clear error message when converting a PyArrow schema with a null field. Why - Prevents misleading REST errors like “Cannot parse type string… unknown” and points directly to the offending field. Testing - make lint - uv run pytest tests/catalog/test_base.py -k rejects_null_type -v - make test (timed out at ~42%) Closes apache#2539 --------- Co-authored-by: Soham <010Soham@users.noreply.github.com>
# Rationale for this change This PR removes [tool.black] from pyproject.toml. This is an unused config since formatting now lives in [ruff.toml](https://github.com/apache/iceberg-python/blob/main/ruff.toml). Also, updates contributing.md to reference ruff instead of black. This PR removes dead config from pyproject.toml: - [tool.black] - replaced by ruff-format apache#127 - [tool.pycln] - replaced by ruff apache#1485 Also updates contributing.md to reference ruff instead of black. ## Are these changes tested? Yes ## Are there any user-facing changes? No
# Rationale for this change This PR adds the ability to change the set the current snapshot of a table. A bulk of this work was done in apache#758 but instead we have broken it out to focus on the set snapshot logic first. Additionally I added a few more tests, following the existing expire snapshots behavior. ## Are these changes tested? Yes, added tests ## Are there any user-facing changes? New API :) ``` table.manage_snapshots().set_current_snapshot(snapshot_id=123456789).commit() table.manage_snapshots().set_current_snapshot(ref_name="my-tag").commit() # chaining table.manage_snapshots() \ .create_tag(snapshot_id=older_id, tag_name="my-tag") \ .set_current_snapshot(ref_name="my-tag") \ .commit() ``` --------- Co-authored-by: Chinmay Bhat <12948588+chinmay-bhat@users.noreply.github.com>
kevinjqliu
force-pushed
the
kevinjqliu/iceberg-rust-rc
branch
from
1102d7d to
4769c82
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Are these changes tested?
Are there any user-facing changes?