SNOW-1974916: write_pandas issue -- MissingDependencyError: Missing optional dependency: pandas

[joshbooya]

Python version

3.8.20

Operating system and processor architecture

Windows-10-10.0.19045-SP0

Installed packages

anyio @ file:///C:/b/abs_847uobe7ea/croot/anyio_1706220224037/work
argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work
argon2-cffi-bindings @ file:///C:/ci/argon2-cffi-bindings_1644569878360/work
asn1crypto @ file:///C:/ci/asn1crypto_1652344143274/work
asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work
async-lru @ file:///C:/b/abs_e0hjkvwwb5/croot/async-lru_1699554572212/work
attrs @ file:///C:/b/abs_24xj44kobp/croot/attrs_1729089434205/work
Babel @ file:///C:/b/abs_a2shv_3tqi/croot/babel_1671782804377/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
beautifulsoup4 @ file:///C:/b/abs_d5wytg_p0w/croot/beautifulsoup4-split_1718029833749/work
bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work
blinker @ file:///C:/b/abs_d9y2dm7cw2/croot/blinker_1696539752170/work
Bottleneck @ file:///C:/b/abs_f05kqh7yvj/croot/bottleneck_1707864273291/work
Brotli @ file:///C:/b/abs_3d36mno480/croot/brotli-split_1714483178642/work
certifi @ file:///C:/b/abs_1fw_exq1si/croot/certifi_1725551736618/work/certifi
cffi @ file:///C:/b/abs_90yq4lmu83/croot/cffi_1726856448345/work
charset-normalizer @ file:///croot/charset-normalizer_1721748349566/work
click @ file:///C:/b/abs_f9ihnt72pu/croot/click_1698129847492/work
colorama @ file:///C:/b/abs_a9ozq0l032/croot/colorama_1672387194846/work
comm @ file:///C:/b/abs_67a8058udb/croot/comm_1709322909844/work
contourpy @ file:///C:/b/abs_d5rpy288vc/croots/recipe/contourpy_1663827418189/work
cryptography @ file:///C:/b/abs_35g500qir4/croot/cryptography_1724940575116/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
debugpy @ file:///C:/b/abs_c0y1fjipt2/croot/debugpy_1690906864587/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
exceptiongroup @ file:///C:/b/abs_c5h1o1_b5b/croot/exceptiongroup_1706031441653/work
executing @ file:///opt/conda/conda-bld/executing_1646925071911/work
fastjsonschema @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_ebruxzvd08/croots/recipe/python-fastjsonschema_1661376484940/work
filelock @ file:///C:/b/abs_f2gie28u58/croot/filelock_1700591233643/work
Flask @ file:///C:/b/abs_f9w7doyu0h/croot/flask_1716545924884/work
Flask-SQLAlchemy @ file:///C:/b/abs_d5arjubk1y/croot/flask-sqlalchemy_1712567300781/work
fonttools @ file:///C:/b/abs_f47gnfqnx0/croot/fonttools_1713551644747/work
greenlet @ file:///C:/b/abs_a6c75ie0bc/croot/greenlet_1702060012174/work
h11 @ file:///C:/b/abs_1czwoyexjf/croot/h11_1706652332846/work
httpcore @ file:///C:/b/abs_55n7g233bw/croot/httpcore_1706728507241/work
httpx @ file:///C:/b/abs_43e135shby/croot/httpx_1723474830126/work
idna @ file:///C:/b/abs_aad84bnnw5/croot/idna_1714398896795/work
importlib-metadata @ file:///C:/b/abs_c1egths604/croot/importlib_metadata-suite_1704813568388/work
importlib_resources @ file:///C:/b/abs_e9f8_0_2dq/croot/importlib_resources-suite_1720641112288/work
ipykernel @ file:///C:/b/abs_5etv6sld_j/croot/ipykernel_1728665608850/work
ipython @ file:///C:/b/abs_254uk73z5b/croot/ipython_1691532131313/work
itsdangerous @ file:///C:/b/abs_c4vwgdr5yn/croot/itsdangerous_1716533399914/work
jedi @ file:///C:/b/abs_1b8kmj7rrm/croot/jedi_1721058359741/work
Jinja2 @ file:///C:/b/abs_92fccttino/croot/jinja2_1716993447201/work
json5 @ file:///tmp/build/80754af9/json5_1624432770122/work
jsonschema @ file:///C:/b/abs_394_t6__xq/croot/jsonschema_1728486718320/work
jsonschema-specifications @ file:///C:/b/abs_0brvm6vryw/croot/jsonschema-specifications_1699032417323/work
jupyter-events @ file:///C:/b/abs_c2m9s5b5m5/croot/jupyter_events_1718738115254/work
jupyter-lsp @ file:///C:/b/abs_ecle3em9d4/croot/jupyter-lsp-meta_1699978291372/work
jupyter_client @ file:///C:/b/abs_a6h3c8hfdq/croot/jupyter_client_1699455939372/work
jupyter_core @ file:///C:/b/abs_beftpbuevw/croot/jupyter_core_1718818307097/work
jupyter_server @ file:///C:/b/abs_9a333nh6yu/croot/jupyter_server_1718827092223/work
jupyter_server_terminals @ file:///C:/b/abs_ec0dq4b50j/croot/jupyter_server_terminals_1686870763512/work
jupyterlab @ file:///C:/b/abs_88qjk9lo2r/croot/jupyterlab_1725895228333/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab_server @ file:///C:/b/abs_fdi5r_tpjc/croot/jupyterlab_server_1725865372811/work
kiwisolver @ file:///C:/b/abs_88mdhvtahm/croot/kiwisolver_1672387921783/work
MarkupSafe @ file:///C:/b/abs_ecfdqh67b_/croot/markupsafe_1704206030535/work
matplotlib @ file:///C:/b/abs_085jhivdha/croot/matplotlib-suite_1693812524572/work
matplotlib-inline @ file:///C:/ci/matplotlib-inline_1661934035815/work
missingno @ file:///tmp/build/80754af9/missingno_1605824147679/work
mistune @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_081kimkskf/croots/recipe/mistune_1661496225923/work
mkl-fft @ file:///C:/b/abs_19i1y8ykas/croot/mkl_fft_1695058226480/work
mkl-random @ file:///C:/b/abs_edwkj1_o69/croot/mkl_random_1695059866750/work
mkl-service==2.4.0
nbclient @ file:///C:/b/abs_cal0q5fyju/croot/nbclient_1698934263135/work
nbconvert @ file:///C:/b/abs_ac6qnzi3no/croot/nbconvert_1728050663985/work
nbformat @ file:///C:/b/abs_c2jkw46etm/croot/nbformat_1728050303821/work
nest-asyncio @ file:///C:/b/abs_65d6lblmoi/croot/nest-asyncio_1708532721305/work
notebook @ file:///C:/b/abs_feeub5ouq6/croot/notebook_1727197380211/work
notebook_shim @ file:///C:/b/abs_a5xysln3lb/croot/notebook-shim_1699455926920/work
numexpr @ file:///C:/b/abs_afm0oewmmt/croot/numexpr_1683221839116/work
numpy @ file:///C:/Users/dev-admin/mkl/numpy_and_numpy_base_1682982345978/work
oracledb @ file:///C:/b/abs_2ai4wa1xv8/croot/oracledb_1712698817064/work
overrides @ file:///C:/b/abs_cfh89c8yf4/croot/overrides_1699371165349/work
packaging @ file:///C:/b/abs_c3vlh0z4jw/croot/packaging_1720101866539/work
pandas @ file:///C:/miniconda3/conda-bld/pandas_1692299636855/work
pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work
parso @ file:///opt/conda/conda-bld/parso_1641458642106/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
pillow @ file:///C:/b/abs_32o8er3uqp/croot/pillow_1721059447598/work
pkgutil_resolve_name @ file:///C:/b/abs_4cck6bcc2k/croot/pkgutil-resolve-name_1704297519576/work
platformdirs @ file:///C:/b/abs_b6z_yqw_ii/croot/platformdirs_1692205479426/work
pooch @ file:///C:/b/abs_a8nmng7d_x/croot/pooch_1695850149827/work
prometheus-client @ file:///C:/Windows/TEMP/abs_ab9nx8qb08/croots/recipe/prometheus_client_1659455104602/work
prompt-toolkit @ file:///C:/b/abs_68uwr58ed1/croot/prompt-toolkit_1704404394082/work
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
Pygments @ file:///C:/b/abs_fay9dpq4n_/croot/pygments_1684279990574/work
PyJWT @ file:///C:/b/abs_04qhgo75wz/croot/pyjwt_1715095119685/work
pyOpenSSL @ file:///C:/b/abs_61vvbgrloc/croot/pyopenssl_1723651600236/work
pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work
PySocks @ file:///C:/ci/pysocks_1605287845585/work
python-dateutil @ file:///C:/b/abs_3au_koqnbs/croot/python-dateutil_1716495777160/work
python-json-logger @ file:///C:/b/abs_cblnsm6puj/croot/python-json-logger_1683824130469/work
pytz @ file:///C:/b/abs_6ap4tsz1ox/croot/pytz_1713974360290/work
pywin32==305.1
pywinpty @ file:///C:/b/abs_73vshmevwq/croot/pywinpty_1677609966356/work/target/wheels/pywinpty-2.0.10-cp38-none-win_amd64.whl
PyYAML @ file:///C:/b/abs_14xkfs39bx/croot/pyyaml_1728657968772/work
pyzmq @ file:///C:/b/abs_89aq69t0up/croot/pyzmq_1705605705281/work
referencing @ file:///C:/b/abs_09f4hj6adf/croot/referencing_1699012097448/work
requests @ file:///C:/b/abs_9frifg92q2/croot/requests_1721410901096/work
rfc3339-validator @ file:///C:/b/abs_ddfmseb_vm/croot/rfc3339-validator_1683077054906/work
rfc3986-validator @ file:///C:/b/abs_6e9azihr8o/croot/rfc3986-validator_1683059049737/work
rpds-py @ file:///C:/b/abs_76j4g4la23/croot/rpds-py_1698947348047/work
scipy==1.10.1
seaborn @ file:///C:/b/abs_ca2mi1rgmn/croot/seaborn_1718303534355/work
Send2Trash @ file:///C:/b/abs_08dh49ew26/croot/send2trash_1699371173324/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
sniffio @ file:///C:/b/abs_3akdewudo_/croot/sniffio_1705431337396/work
snowflake-connector-python @ file:///C:/b/abs_4es14t2ajh/croot/snowflake-connector-python_1726612501386/work
sortedcontainers @ file:///tmp/build/80754af9/sortedcontainers_1623949099177/work
soupsieve @ file:///C:/b/abs_bbsvy9t4pl/croot/soupsieve_1696347611357/work
SQLAlchemy @ file:///C:/b/abs_7duxw5rxx8/croot/sqlalchemy_1725885067126/work
stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work
terminado @ file:///C:/b/abs_25nakickad/croot/terminado_1671751845491/work
tinycss2 @ file:///C:/b/abs_52w5vfuaax/croot/tinycss2_1668168823131/work
tomli @ file:///C:/Windows/TEMP/abs_ac109f85-a7b3-4b4d-bcfd-52622eceddf0hy332ojo/croots/recipe/tomli_1657175513137/work
tomlkit @ file:///C:/b/abs_07yc6lwf0e/croot/tomlkit_1728650313581/work
tornado @ file:///C:/b/abs_7bua0304mj/croot/tornado_1718740122405/work
traitlets @ file:///C:/b/abs_bfsnoxl4pq/croot/traitlets_1718227069245/work
typing_extensions @ file:///C:/b/abs_0as9mdbkfl/croot/typing_extensions_1715268906610/work
tzdata @ file:///croot/python-tzdata_1690578112552/work
unicodedata2 @ file:///C:/b/abs_b6apldlg7y/croot/unicodedata2_1713212998255/work
urllib3 @ file:///C:/b/abs_eeswtzv5iq/croot/urllib3_1718978558630/work
wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
webencodings==0.5.1
websocket-client @ file:///C:/b/abs_5dmnxxoci9/croot/websocket-client_1715878351319/work
Werkzeug @ file:///C:/b/abs_8bittcw9jr/croot/werkzeug_1716533366070/work
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
zipp @ file:///C:/b/abs_021kqswk3k/croot/zipp_1729012371206/work

What did you do?

success, nchunks, nrows, _ = write_pandas(ctx, df1, table_name[0]
                                          , overwrite = True
                                          , auto_create_table = True)

success, nchunks, nrows

What did you expect to see?

---

MissingDependencyError Traceback (most recent call last)
Cell In[26], line 1
----> 1 success, nchunks, nrows, _ = write_pandas(ctx, df1, table_name[0]
2 , overwrite = True
3 , auto_create_table = True)
5 success, nchunks, nrows

File ~.conda\envs\py38\lib\site-packages\snowflake\connector\pandas_tools.py:285, in write_pandas(conn, df, table_name, database, schema, chunk_size, compression, on_error, parallel, quote_identifiers, auto_create_table, create_temp_table, overwrite, table_type, use_logical_type, **kwargs)
281 if chunk_size is None:
282 chunk_size = len(df)
284 if not (
--> 285 isinstance(df.index, pandas.RangeIndex)
286 and 1 == df.index.step
287 and 0 == df.index.start
288 ):
289 warnings.warn(
290 f"Pandas Dataframe has non-standard index of type {str(type(df.index))} which will not be written."
291 f" Consider changing the index to pd.RangeIndex(start=0,...,step=1) or "
(...)
294 stacklevel=2,
295 )
297 # use_logical_type should be True when dataframe contains datetimes with timezone.
298 # #1687

File ~.conda\envs\py38\lib\site-packages\snowflake\connector\options.py:40, in MissingOptionalDependency.getattr(self, item)
39 def getattr(self, item):
---> 40 raise errors.MissingDependencyError(self._dep_name)

MissingDependencyError: Missing optional dependency: pandas

Can you set logging to DEBUG and collect the logs?

import logging
import os

for logger_name in ('snowflake.connector',):
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)

[joshbooya]

Package Version

---

anyio 4.2.0
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asn1crypto 1.5.1
asttokens 2.0.5
async-lru 2.0.4
attrs 24.2.0
Babel 2.11.0
backcall 0.2.0
beautifulsoup4 4.12.3
bleach 4.1.0
blinker 1.6.2
Bottleneck 1.3.7
Brotli 1.0.9
certifi 2024.8.30
cffi 1.17.1
charset-normalizer 3.3.2
click 8.1.7
colorama 0.4.6
comm 0.2.1
contourpy 1.0.5
cryptography 43.0.0
cycler 0.11.0
debugpy 1.6.7
decorator 5.1.1
defusedxml 0.7.1
exceptiongroup 1.2.0
executing 0.8.3
fastjsonschema 2.16.2
filelock 3.13.1
Flask 3.0.3
Flask-SQLAlchemy 3.1.1
fonttools 4.51.0
greenlet 3.0.1
h11 0.14.0
httpcore 1.0.2
httpx 0.27.0
idna 3.7
importlib-metadata 7.0.1
importlib_resources 6.4.0
ipykernel 6.29.5
ipython 8.12.2
itsdangerous 2.2.0
jedi 0.19.1
Jinja2 3.1.4
json5 0.9.6
jsonschema 4.23.0
jsonschema-specifications 2023.7.1
jupyter_client 8.6.0
jupyter_core 5.7.2
jupyter-events 0.10.0
jupyter-lsp 2.2.0
jupyter_server 2.14.1
jupyter_server_terminals 0.4.4
jupyterlab 4.2.5
jupyterlab-pygments 0.1.2
jupyterlab_server 2.27.3
kiwisolver 1.4.4
MarkupSafe 2.1.3
matplotlib 3.7.2
matplotlib-inline 0.1.6
missingno 0.4.2
mistune 2.0.4
mkl-fft 1.3.8
mkl-random 1.2.4
mkl-service 2.4.0
nbclient 0.8.0
nbconvert 7.16.4
nbformat 5.10.4
nest-asyncio 1.6.0
notebook 7.2.2
notebook_shim 0.2.3
numexpr 2.8.4
numpy 1.24.3
oracledb 2.1.1
overrides 7.4.0
packaging 24.1
pandas 2.0.3
pandocfilters 1.5.0
parso 0.8.3
pickleshare 0.7.5
pillow 10.4.0
pip 24.2
pkgutil_resolve_name 1.3.10
platformdirs 3.10.0
pooch 1.7.0
prometheus-client 0.14.1
prompt-toolkit 3.0.43
psutil 5.9.0
pure-eval 0.2.2
pycparser 2.21
Pygments 2.15.1
PyJWT 2.8.0
pyOpenSSL 24.2.1
pyparsing 3.0.9
PySocks 1.7.1
python-dateutil 2.9.0.post0
python-json-logger 2.0.7
pytz 2024.1
pywin32 305.1
pywinpty 2.0.10
PyYAML 6.0.2
pyzmq 25.1.2
referencing 0.30.2
requests 2.32.3
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rpds-py 0.10.6
scipy 1.10.1
seaborn 0.13.2
Send2Trash 1.8.2
setuptools 75.1.0
six 1.16.0
sniffio 1.3.0
snowflake-connector-python 3.12.2
sortedcontainers 2.4.0
soupsieve 2.5
SQLAlchemy 2.0.34
stack-data 0.2.0
terminado 0.17.1
tinycss2 1.2.1
tomli 2.0.1
tomlkit 0.13.2
tornado 6.4.1
traitlets 5.14.3
typing_extensions 4.11.0
tzdata 2023.3
unicodedata2 15.1.0
urllib3 1.26.19
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.8.0
Werkzeug 3.0.3
wheel 0.44.0
win-inet-pton 1.1.0
zipp 3.20.2

[sfc-gh-dszmolka]

hello - can you please try installing the latest version of the driver, with the pandas extras, as described in https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-pandas#installation? with pip it would be:

pip install snowflake-connector-python[pandas]==3.14.0

You can also try this workaround from #1527 (comment)

pip install "snowflake-connector-python[pandas]" pandas numpy pyarrow

Best would be also perhaps to set up a brand new environment from scratch (in your case, in conda looks like), as Python 3.8 is not supported anymore.

If it still doesn't work, could you please send a minimal viable, runnable piece of reproduction which when run, could reproduce the issue you're seeing? Thank you in advance !

[joshbooya]

hello - can you please try installing the latest version of the driver, with the pandas extras, as described in https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-pandas#installation? with pip it would be:

pip install snowflake-connector-python[pandas]==3.14.0

You can also try this workaround from #1527 (comment)

pip install "snowflake-connector-python[pandas]" pandas numpy pyarrow

Best would be also perhaps to set up a brand new environment from scratch (in your case, in conda looks like), as Python 3.8 is not supported anymore.

If it still doesn't work, could you please send a minimal viable, runnable piece of reproduction which when run, could reproduce the issue you're seeing? Thank you in advance !

Using the prescribed installation with:

conda install "snowflake-connector-python[pandas]" pandas numpy pyarrow

resolved the issue in Python 3.8. Thanks!