Replies: 1 comment
-
what is your |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I am receiving an error when using pyarrows ParquetDataset() and S3FSFileSystem()
Some Background:
AWS Lambda Function - Runtime 3.7
Below is list of packages and versions being used within the lambda function.
requirements-prod.txt
The Code:
s3_uri = 's3://' + bucket + '/' + key df = pq.ParquetDataset(s3_uri, filesystem=fs) \ .read_pandas() \ .to_pandas()
The Error :
AttributeError: 'AioClientCreator' object has no attribute '_register_lazy_block_unknown_fips_pseudo_regions'
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 86, in lambda_handler
df = utils.load_exceptions_dataframe(bucket, key, fs)
File "/var/task/utils.py", line 74, in load_exceptions_dataframe
df = pq.ParquetDataset(s3_uri, filesystem=fs)
File "/opt/python/pyarrow/parquet.py", line 1272, in init
open_file_func=partial(_open_dataset_file, self._metadata)
File "/opt/python/pyarrow/parquet.py", line 1446, in _make_manifest
if _is_path_like(path_or_paths) and fs.isdir(path_or_paths):
File "/opt/python/fsspec/asyn.py", line 88, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/opt/python/fsspec/asyn.py", line 69, in sync
raise result[0]
File "/opt/python/fsspec/asyn.py", line 25, in _runner
result[0] = await coro
File "/opt/python/s3fs/core.py", line 1161, in _isdir
return bool(await self._lsdir(path))
File "/opt/python/s3fs/core.py", line 559, in _lsdir
await self.set_session()
File "/opt/python/s3fs/core.py", line 403, in set_session
self._s3 = await s3creator.aenter()
File "/opt/python/aiobotocore/session.py", line 37, in aenter
self._client = await self._coro
File "/opt/python/aiobotocore/session.py", line 125, in _create_client
client_config=config, api_version=api_version)
File "/opt/python/aiobotocore/client.py", line 46, in create_client
self._register_lazy_block_unknown_fips_pseudo_regions(service_client)
Any idea on a possible solution?
Beta Was this translation helpful? Give feedback.
All reactions