Outdated [unsupported release] pyarrow dependency #2349
Description
Python version
Python 3.12.10 (main, Apr 8 2025, 11:35:47) [Clang 16.0.0 (clang-1600.0.26.6)]
Operating system and processor architecture
macOS-15.5-arm64-arm-64bit
Installed packages
❯ poetry run python -m pip freeze
aioboto3==14.3.0
aiobotocore==2.22.0
aiofiles==24.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.12.4
aioitertools==0.12.0
aiosignal==1.3.2
alembic==1.16.1
annotated-types==0.7.0
antlr4-python3-runtime==4.13.2
anyio==4.9.0
asn1crypto==1.5.1
astroid==3.3.10
asttokens==3.0.0
attrs==25.3.0
backoff==2.2.1
boto3==1.37.3
boto3-stubs==1.38.25
botocore==1.37.3
botocore-stubs==1.38.25
certifi==2025.4.26
cffi==1.17.1
charset-normalizer==3.4.2
click==8.1.8
coloredlogs==14.0
cryptography==45.0.3
dagster==1.10.18
dagster-aws==0.26.18
dagster-graphql==1.10.18
dagster-pipes==1.10.18
dagster-snowflake==0.26.18
dagster-webserver==1.10.18
dagster_shared==1.10.18
dagster_utilities @ git+https:REDACTED
decorator==5.2.1
dill==0.4.0
docstring_parser==0.16
docstring_parser_fork==0.0.12
executing==2.2.0
filelock==3.18.0
frozenlist==1.6.0
fsspec==2025.5.1
gql==3.5.3
graphene==3.4.3
graphql-core==3.2.6
graphql-relay==3.2.0
grpcio==1.71.0
grpcio-health-checking==1.71.0
h11==0.16.0
httptools==0.6.4
humanfriendly==10.0
idna==3.10
iniconfig==2.1.0
ipython==9.1.0
ipython_pygments_lexers==1.1.1
isort==6.0.1
jedi==0.19.2
Jinja2==3.1.6
jmespath==1.0.1
Mako==1.3.10
markdown-it-py==3.0.0
MarkupSafe==3.0.2
matplotlib-inline==0.1.7
mccabe==0.7.0
mdurl==0.1.2
more-itertools==10.7.0
multidict==6.4.4
mypy==1.15.0
mypy-boto3-athena==1.38.13
mypy-boto3-s3==1.38.0
mypy_extensions==1.1.0
nodeenv==1.9.1
numpy==2.2.6
packaging==25.0
pandas==2.2.3
parso==0.8.4
pexpect==4.9.0
platformdirs==4.3.8
pluggy==1.6.0
prompt_toolkit==3.0.50
propcache==0.3.1
protobuf==5.29.5
ptyprocess==0.7.0
pure_eval==0.2.3
pyarrow==20.0.0
pyarrow-stubs==19.4
pycparser==2.22
pydantic==2.11.5
pydantic-settings==2.9.1
pydantic_core==2.33.2
pydoclint==0.6.2
Pygments==2.19.1
PyJWT==2.10.1
pylint==3.3.7
pyOpenSSL==25.1.0
pyright==1.1.401
pytest==8.3.5
python-dateutil==2.9.0.post0
python-dotenv==1.1.0
pytz==2025.2
PyYAML==6.0.2
requests==2.32.3
requests-toolbelt==1.0.0
rich==14.0.0
ruff==0.11.11
s3transfer==0.11.3
-e git+https://[REDACTED]
setuptools==80.9.0
six==1.17.0
sniffio==1.3.1
snowflake-connector-python==3.15.0
sortedcontainers==2.4.0
SQLAlchemy==2.0.41
stack-data==0.6.3
starlette==0.46.2
structlog==25.3.0
tabulate==0.9.0
tightwrap==24.3.0
tomli==2.2.1
tomlkit==0.13.2
toposort==1.10
tqdm==4.67.1
traitlets==5.14.3
types-aioboto3==14.3.0
types-aiobotocore==2.22.0
types-aiobotocore-s3==2.22.0
types-awscrt==0.27.2
types-s3transfer==0.13.0
typing-inspection==0.4.1
typing_extensions==4.13.2
tzdata==2025.2
universal_pathlib==0.2.6
urllib3==2.4.0
uvicorn==0.34.2
uvloop==0.21.0
watchdog==5.0.3
watchfiles==1.0.5
wcwidth==0.2.13
websockets==15.0.1
wrapt==1.17.2
yarl==1.20.0What did you do?
Create a project where we're making use of a modern version of PyArrow (version 20). In our case we're using Dagster to aggregate data and move that data into Snowflake.
On startup of Dagster we see:
2025-06-06 11:30:06 +0100 - dagster - WARNING - /Users/REDACTED/Library/Caches/pypoetry/virtualenvs/REDACTED-r6JE7vQ_-py3.12/lib/python3.12/site-packages/snowflake/connector/options.py:104: UserWarning: You have an incompatible version of 'pyarrow' installed (20.0.0), please install a version that adheres to: 'pyarrow<19.0.0; extra == "pandas"'
warn_incompatible_dep(What did you expect to see?
Adding the Snowflake connector causes no dependency problems when installed alongside the current stable version of PyArrow rather than a deprecated and unsupported version.
Can you set logging to DEBUG and collect the logs?
import logging
import os
for logger_name in ('snowflake.connector',):
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
logger.addHandler(ch)