Unable to use on AWS Lambda

[Jflick58]

1. What version of Python are you using (python --version)?

Tried 3.6.7 and 3.7.1

2. What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')?

Building distribution on Linux-4.15.0-1035-aws-x86_64-with-Ubuntu-18.04-bionic,

Running on AWS Lambda (Linux-4.14.133-88.105.amzn1.x86_64-x86_64-with-glibc2.2.5)

3. What are the component versions in the environment (pip list)?

Package Version

---

asn1crypto 0.24.0
azure-common 1.1.23
azure-storage-blob 2.1.0
azure-storage-common 2.1.0
boto3 1.9.210
botocore 1.12.210
certifi 2019.6.16
cffi 1.12.3
chardet 3.0.4
Click 7.0
cryptography 2.7
docutils 0.15.2
fsspec 0.4.1
future 0.17.1
idna 2.8
ijson 2.4
jmespath 0.9.4
numpy 1.17.0
pandas 0.25.0
pip 19.2.2
pycparser 2.19
pycryptodomex 3.8.2
PyJWT 1.7.1
pyOpenSSL 19.0.0
python-dateutil 2.8.0
python-lambda 3.2.6
pytz 2019.2
PyYAML 5.1.2
requests 2.22.0
s3fs 0.3.3
s3transfer 0.2.1
setuptools 41.1.0
six 1.12.0
snowflake-connector-python 1.8.7
snowflake-sqlalchemy 1.1.14
SQLAlchemy 1.3.7
urllib3 1.25.3
wheel 0.33.6

4. What did you do?

import pandas as pd 
from sqlalchemy import create_engine

db_string = "snowflake://{}:{}@{}/{}/{}?warehouse={}?role={}".format(
            user, password, account, db, schema, warehouse, role
        )
engine = create_engine(db_string)
conn = engine.connect() 

5. What did you expect to see?
   I expect to see no errors with a successful connection. I then pass the conn object to a pd.to_sql() method.

6. What did you see instead?

No module named 'snowflake.sqlalchemy'

7. Can you set logging to DEBUG and collect the logs?

lambda_logs.txt

[Jflick58]

@smtakeda bumping this issue. Curious if the Kernel versioning difference is causing the issue? I've asked my infra guy to roll me out a build box with the same platform version in the interim.

[Jflick58]

@SnowflakeBrian bumping again for visibility.

[howryu]

I am not a lambda expert, but generally it means that you did not correctly installed snowflake sqlalchemy.

[Jflick58]

@howryu what do you mean? We build the dist with the python-lambda library and the same version works invoking the lambda function with a test event locally. I think it may be a platform incompatibility.

[smtakeda]

@Jflick58 can you do this?
for Python3.6

touch $VIRTUAL_ENV/lib/python3.6/site-packages/snowflake/__init__.py

for Python3.7

touch $VIRTUAL_ENV/lib/python3.7/site-packages/snowflake/__init__.py

[smtakeda]

Verified my test case worked:

lambda_handler.py

from snowflake.sqlalchemy import URL
from sqlalchemy import create_engine

print('read')
def lambda_handler(event, context):
    print('triggered')
    engine = create_engine(URL(
        (connection parameters)
    ))
    print('created')
    con = engine.connect()
    print("connected")
    con.close()
    engine.dispose()

if __name__ == "__main__":
    lambda_handler(None, None)

package.sh

rm -rf lambda_function
rm -f lambda_function.zip
python3.6 -m venv lambda_function
source lambda_function/bin/activate
pip install --upgrade pip
pip install --upgrade snowflake-sqlalchemy
zip -9 lambda_function.zip lambda_function.py
touch $VIRTUAL_ENV/lib/python3.6/site-packages/snowflake/__init__.py
DIR=`pwd`
cd $VIRTUAL_ENV/lib/python3.6/site-packages
zip -r9 $DIR/lambda_function.zip .
cd $VIRTUAL_ENV/lib64/python3.6/site-packages
zip -r9 $DIR/lambda_function.zip .

Uploaded the zip to lambda and run manually.

[Jflick58]

@smtakeda I will give this a shot today.

[smtakeda]

There is known issue with the namespace package, e.g., snowflake, which is not recognized by the lambda. The workaround is add __init__.py to the directory.

[sneha-v]

@Jflick58 this also happens if slim:true is added in serverless file.. since slim tries to remove all .dist folders and snowflake-sqlalchemy is also .dist folder, it gets removed... hence removing slim:true in serverless would be a solution..

[kychanbp]

I get the same error in aws glue

Traceback (most recent call last):
  File "/tmp/runscript.py", line 123, in <module>
    runpy.run_path(temp_file_path, run_name='__main__')
  File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/glue-python-scripts-x20ytvw9/ETL_job_cainiao_supply_chain.py", line 332, in <module>
  File "/tmp/glue-python-scripts-x20ytvw9/ETL_job_cainiao_supply_chain.py", line 80, in main
  File "/glue/lib/installation/sqlalchemy/engine/__init__.py", line 500, in create_engine
    return strategy.create(*args, **kwargs)
  File "/glue/lib/installation/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/glue/lib/installation/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
    cls = registry.load(name)
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 251, in load
    return self.impls[name]()
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 274, in load
    mod = compat.import_(modulepath)
ModuleNotFoundError: No module named 'snowflake.sqlalchemy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/runscript.py", line 142, in <module>
    raise e_type(e_value).with_traceback(new_stack)
  File "/tmp/glue-python-scripts-x20ytvw9/ETL_job_cainiao_supply_chain.py", line 332, in <module>
  File "/tmp/glue-python-scripts-x20ytvw9/ETL_job_cainiao_supply_chain.py", line 80, in main
  File "/glue/lib/installation/sqlalchemy/engine/__init__.py", line 500, in create_engine
    return strategy.create(*args, **kwargs)
  File "/glue/lib/installation/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/glue/lib/installation/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
    cls = registry.load(name)
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 251, in load
    return self.impls[name]()
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 274, in load
    mod = compat.import_(modulepath)
ModuleNotFoundError: No module named 'snowflake.sqlalchemy'

How can I solve it?

[absognety]

There is known issue with the namespace package, e.g., snowflake, which is not recognized by the lambda. The workaround is add __init__.py to the directory.

This issue is still persistent, using Python 3.11 with snowflake_sqlalchemy-1.7.3. what is the stable fix for this?