Skip to content

Commit 6456a8d

Browse files
authored
fix: Remote signing with table specific s3.signer.endpoint (#2005)
<!-- Thanks for opening a pull request! --> <!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. --> <!-- Closes #${GITHUB_ISSUE_ID} --> # Rationale for this change Currently pyiceberg remote singing only works if the sign endpoint is shared by all tables in a REST Catalog. However, some Catalogs use table specific endpoints. If table specific endpoints are used, Pyiceberg sends the sign request for the second table that is queried to the sign endpoint of the first table. The reason for this is, that although we [re-register() a new signer with different properties](https://github.com/apache/iceberg-python/blob/996a7ba4dbf4afdb3d46689f1715206b1c355f2a/pyiceberg/io/fsspec.py#L166), this statement has no effect the second time it runs because of the unique_id, even if the properties are different. https://github.com/boto/botocore/blob/8c517320c6a40cd91e8e7fbb05e27183ba2f6dce/botocore/hooks.py#L310-L312 This PR first unregisters the old handler, before adding the new one. # Are these changes tested? No. Any idea how we could test them? Just tested against LAKEKEEPER in a notebook while implementing table specific endpoints. # Are there any user-facing changes? It works now! <!-- In the case of user-facing changes, please add the changelog label. -->
1 parent 996a7ba commit 6456a8d

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

pyiceberg/io/fsspec.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,7 @@ def _s3(properties: Properties) -> AbstractFileSystem:
163163
fs = S3FileSystem(client_kwargs=client_kwargs, config_kwargs=config_kwargs)
164164

165165
for event_name, event_function in register_events.items():
166+
fs.s3.meta.events.unregister(event_name, unique_id=1925)
166167
fs.s3.meta.events.register_last(event_name, event_function, unique_id=1925)
167168

168169
return fs

0 commit comments

Comments
 (0)