Release 2.15.1 (#1264)

* Prepare - Release 2.15.1 - Drop SPARQL

* Add snyk workflow

* Revert dropping sparql; add sparql extra

* Update dependencies; add sparql extra

* Upd layers docs; upd contributing guide

* Fix contributing guide

* Add check for sparql import

* Update tox config

* [skip ci] update neptune docs & tutorial

Co-authored-by: Anton Kukushkin <[email protected]>

Author: jaidisido

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 2.15.0
+current_version = 2.15.1
 commit = False
 tag = False
 tag_name = {new_version}
@@ -16,6 +16,8 @@ tag_name = {new_version}
 
 [bumpversion:file:docs/source/install.rst]
 
+[bumpversion:file:docs/source/layers.rst]
+
 [bumpversion:file:docs/source/what.rst]
 
 [bumpversion:file:awswrangler/__metadata__.py]

.github/workflows/snyk.yml

@@ -0,0 +1,13 @@
+name: Snyk
+on: workflow_dispatch
+jobs:
+  security:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@master
+      - name: Run Snyk to check for vulnerabilities
+        uses: snyk/actions/python-3.8@master
+        env:
+          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
+        with:
+          args: --severity-threshold=high

CONTRIBUTING.md

@@ -104,7 +104,7 @@ or
 
 * Install dependencies:
 
-``poetry install --extras "sqlserver"``
+``poetry install --extras "sqlserver sparql"``
 
 * Run the validation script:
 
@@ -135,7 +135,7 @@ or
 
 * Install dependencies:
 
-``poetry install --extras "sqlserver"``
+``poetry install --extras "sqlserver sparql"``
 
 * Go to the ``test_infra`` directory
 
@@ -192,7 +192,7 @@ or
 
 * Then run the command bellow to install all dependencies:
 
-``poetry install --extras "sqlserver"``
+``poetry install --extras "sqlserver sparql"``
 
 * Go to the ``test_infra`` directory
 

CONTRIBUTING_COMMON_ERRORS.md

@@ -13,9 +13,9 @@ Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in ./.venv/lib/python3.7/site-
 Using legacy 'setup.py install' for python-Levenshtein, since package 'wheel' is not installed.
 Installing collected packages: awswrangler, python-Levenshtein
   Attempting uninstall: awswrangler
-    Found existing installation: awswrangler 2.15.0
-    Uninstalling awswrangler-2.15.0:
-      Successfully uninstalled awswrangler-2.15.0
+    Found existing installation: awswrangler 2.15.1
+    Uninstalling awswrangler-2.15.1:
+      Successfully uninstalled awswrangler-2.15.1
   Running setup.py develop for awswrangler
     Running setup.py install for python-Levenshtein ... error
     ERROR: Command errored out with exit status 1:

README.md

@@ -8,7 +8,7 @@ Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, Q
 
 > An [AWS Professional Service](https://aws.amazon.com/professional-services/) open source initiative | [email protected]
 
-[![Release](https://img.shields.io/badge/release-2.15.0-brightgreen.svg)](https://pypi.org/project/awswrangler/)
+[![Release](https://img.shields.io/badge/release-2.15.1-brightgreen.svg)](https://pypi.org/project/awswrangler/)
 [![Python Version](https://img.shields.io/badge/python-3.7%20%7C%203.7%20%7C%203.8%20%7C%203.10-brightgreen.svg)](https://anaconda.org/conda-forge/awswrangler)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
@@ -23,7 +23,7 @@ Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, Q
 | **[PyPi](https://pypi.org/project/awswrangler/)**  | [![PyPI Downloads](https://pepy.tech/badge/awswrangler)](https://pypi.org/project/awswrangler/) | `pip install awswrangler` |
 | **[Conda](https://anaconda.org/conda-forge/awswrangler)** | [![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/awswrangler.svg)](https://anaconda.org/conda-forge/awswrangler) | `conda install -c conda-forge awswrangler` |
 
-> ⚠️ **For platforms without PyArrow 3 support (e.g. [EMR](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#emr-cluster), [Glue PySpark Job](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#aws-glue-pyspark-jobs), MWAA):**<br>
+> ⚠️ **For platforms without PyArrow 3 support (e.g. [EMR](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#emr-cluster), [Glue PySpark Job](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#aws-glue-pyspark-jobs), MWAA):**<br>
 ➡️ `pip install pyarrow==2 awswrangler`
 
 Powered By [<img src="https://arrow.apache.org/img/arrow.png" width="200">](https://arrow.apache.org/powered_by/)
@@ -42,7 +42,7 @@ Powered By [<img src="https://arrow.apache.org/img/arrow.png" width="200">](http
 
 Installation command: `pip install awswrangler`
 
-> ⚠️ **For platforms without PyArrow 3 support (e.g. [EMR](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#emr-cluster), [Glue PySpark Job](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#aws-glue-pyspark-jobs), MWAA):**<br>
+> ⚠️ **For platforms without PyArrow 3 support (e.g. [EMR](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#emr-cluster), [Glue PySpark Job](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#aws-glue-pyspark-jobs), MWAA):**<br>
 ➡️`pip install pyarrow==2 awswrangler`
 
 ```py3
@@ -96,17 +96,17 @@ FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3
 
 ## [Read The Docs](https://aws-data-wrangler.readthedocs.io/)
 
-- [**What is AWS Data Wrangler?**](https://aws-data-wrangler.readthedocs.io/en/2.15.0/what.html)
-- [**Install**](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html)
-  - [PyPi (pip)](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#pypi-pip)
-  - [Conda](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#conda)
-  - [AWS Lambda Layer](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#aws-lambda-layer)
-  - [AWS Glue Python Shell Jobs](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#aws-glue-python-shell-jobs)
-  - [AWS Glue PySpark Jobs](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#aws-glue-pyspark-jobs)
-  - [Amazon SageMaker Notebook](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#amazon-sagemaker-notebook)
-  - [Amazon SageMaker Notebook Lifecycle](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#amazon-sagemaker-notebook-lifecycle)
-  - [EMR](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#emr)
-  - [From source](https://aws-data-wrangler.readthedocs.io/en/2.15.0/install.html#from-source)
+- [**What is AWS Data Wrangler?**](https://aws-data-wrangler.readthedocs.io/en/2.15.1/what.html)
+- [**Install**](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html)
+  - [PyPi (pip)](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#pypi-pip)
+  - [Conda](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#conda)
+  - [AWS Lambda Layer](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#aws-lambda-layer)
+  - [AWS Glue Python Shell Jobs](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#aws-glue-python-shell-jobs)
+  - [AWS Glue PySpark Jobs](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#aws-glue-pyspark-jobs)
+  - [Amazon SageMaker Notebook](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#amazon-sagemaker-notebook)
+  - [Amazon SageMaker Notebook Lifecycle](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#amazon-sagemaker-notebook-lifecycle)
+  - [EMR](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#emr)
+  - [From source](https://aws-data-wrangler.readthedocs.io/en/2.15.1/install.html#from-source)
 - [**Tutorials**](https://github.com/awslabs/aws-data-wrangler/tree/main/tutorials)
   - [001 - Introduction](https://github.com/awslabs/aws-data-wrangler/blob/main/tutorials/001%20-%20Introduction.ipynb)
   - [002 - Sessions](https://github.com/awslabs/aws-data-wrangler/blob/main/tutorials/002%20-%20Sessions.ipynb)
@@ -141,28 +141,28 @@ FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3
   - [031 - OpenSearch](https://github.com/awslabs/aws-data-wrangler/blob/main/tutorials/031%20-%20OpenSearch.ipynb)
   - [032 - Lake Formation Governed Tables](https://github.com/awslabs/aws-data-wrangler/blob/main/tutorials/032%20-%20Lake%20Formation%20Governed%20Tables.ipynb)
   - [033 - Amazon Neptune](https://github.com/awslabs/aws-data-wrangler/blob/main/tutorials/033%20-%20Amazon%20Neptune.ipynb)
-- [**API Reference**](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html)
-  - [Amazon S3](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#amazon-s3)
-  - [AWS Glue Catalog](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#aws-glue-catalog)
-  - [Amazon Athena](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#amazon-athena)
-  - [AWS Lake Formation](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#aws-lake-formation)
-  - [Amazon Redshift](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#amazon-redshift)
-  - [PostgreSQL](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#postgresql)
-  - [MySQL](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#mysql)
-  - [SQL Server](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#sqlserver)
-  - [Data API Redshift](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#data-api-redshift)
-  - [Data API RDS](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#data-api-rds)
-  - [OpenSearch](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#opensearch)
-  - [Amazon Neptune](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#amazon-neptune)
-  - [DynamoDB](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#dynamodb)
-  - [Amazon Timestream](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#amazon-timestream)
-  - [Amazon EMR](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#amazon-emr)
-  - [Amazon CloudWatch Logs](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#amazon-cloudwatch-logs)
-  - [Amazon Chime](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#amazon-chime)
-  - [Amazon QuickSight](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#amazon-quicksight)
-  - [AWS STS](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#aws-sts)
-  - [AWS Secrets Manager](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#aws-secrets-manager)
-  - [Global Configurations](https://aws-data-wrangler.readthedocs.io/en/2.15.0/api.html#global-configurations)
+- [**API Reference**](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html)
+  - [Amazon S3](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#amazon-s3)
+  - [AWS Glue Catalog](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#aws-glue-catalog)
+  - [Amazon Athena](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#amazon-athena)
+  - [AWS Lake Formation](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#aws-lake-formation)
+  - [Amazon Redshift](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#amazon-redshift)
+  - [PostgreSQL](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#postgresql)
+  - [MySQL](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#mysql)
+  - [SQL Server](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#sqlserver)
+  - [Data API Redshift](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#data-api-redshift)
+  - [Data API RDS](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#data-api-rds)
+  - [OpenSearch](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#opensearch)
+  - [Amazon Neptune](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#amazon-neptune)
+  - [DynamoDB](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#dynamodb)
+  - [Amazon Timestream](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#amazon-timestream)
+  - [Amazon EMR](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#amazon-emr)
+  - [Amazon CloudWatch Logs](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#amazon-cloudwatch-logs)
+  - [Amazon Chime](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#amazon-chime)
+  - [Amazon QuickSight](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#amazon-quicksight)
+  - [AWS STS](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#aws-sts)
+  - [AWS Secrets Manager](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#aws-secrets-manager)
+  - [Global Configurations](https://aws-data-wrangler.readthedocs.io/en/2.15.1/api.html#global-configurations)
 - [**License**](https://github.com/awslabs/aws-data-wrangler/blob/main/LICENSE.txt)
 - [**Contributing**](https://github.com/awslabs/aws-data-wrangler/blob/main/CONTRIBUTING.md)
 - [**Legacy Docs** (pre-1.0.0)](https://aws-data-wrangler.readthedocs.io/en/0.3.3/)

awswrangler/__metadata__.py

@@ -7,5 +7,5 @@
 
 __title__: str = "awswrangler"
 __description__: str = "Pandas on AWS."
-__version__: str = "2.15.0"
+__version__: str = "2.15.1"
 __license__: str = "Apache License 2.0"

awswrangler/athena/_read.py

@@ -591,11 +591,11 @@ def read_sql_query(
 
     **Related tutorial:**
 
-    - `Amazon Athena <https://aws-data-wrangler.readthedocs.io/en/2.15.0/
+    - `Amazon Athena <https://aws-data-wrangler.readthedocs.io/en/2.15.1/
       tutorials/006%20-%20Amazon%20Athena.html>`_
-    - `Athena Cache <https://aws-data-wrangler.readthedocs.io/en/2.15.0/
+    - `Athena Cache <https://aws-data-wrangler.readthedocs.io/en/2.15.1/
       tutorials/019%20-%20Athena%20Cache.html>`_
-    - `Global Configurations <https://aws-data-wrangler.readthedocs.io/en/2.15.0/
+    - `Global Configurations <https://aws-data-wrangler.readthedocs.io/en/2.15.1/
       tutorials/021%20-%20Global%20Configurations.html>`_
 
     **There are three approaches available through ctas_approach and unload_approach parameters:**
@@ -660,7 +660,7 @@ def read_sql_query(
     /athena.html#Athena.Client.get_query_execution>`_ .
 
     For a practical example check out the
-    `related tutorial <https://aws-data-wrangler.readthedocs.io/en/2.15.0/
+    `related tutorial <https://aws-data-wrangler.readthedocs.io/en/2.15.1/
     tutorials/024%20-%20Athena%20Query%20Metadata.html>`_!
 
 
@@ -904,11 +904,11 @@ def read_sql_table(
 
     **Related tutorial:**
 
-    - `Amazon Athena <https://aws-data-wrangler.readthedocs.io/en/2.15.0/
+    - `Amazon Athena <https://aws-data-wrangler.readthedocs.io/en/2.15.1/
       tutorials/006%20-%20Amazon%20Athena.html>`_
-    - `Athena Cache <https://aws-data-wrangler.readthedocs.io/en/2.15.0/
+    - `Athena Cache <https://aws-data-wrangler.readthedocs.io/en/2.15.1/
       tutorials/019%20-%20Athena%20Cache.html>`_
-    - `Global Configurations <https://aws-data-wrangler.readthedocs.io/en/2.15.0/
+    - `Global Configurations <https://aws-data-wrangler.readthedocs.io/en/2.15.1/
       tutorials/021%20-%20Global%20Configurations.html>`_
 
     **There are two approaches to be defined through ctas_approach parameter:**
@@ -953,7 +953,7 @@ def read_sql_table(
     /athena.html#Athena.Client.get_query_execution>`_ .
 
     For a practical example check out the
-    `related tutorial <https://aws-data-wrangler.readthedocs.io/en/2.15.0/
+    `related tutorial <https://aws-data-wrangler.readthedocs.io/en/2.15.1/
     tutorials/024%20-%20Athena%20Query%20Metadata.html>`_!
 
 

awswrangler/neptune/client.py

@@ -1,5 +1,6 @@
 """Amazon NeptuneClient Module."""
 
+import importlib.util
 import logging
 from typing import Any, Dict, List, Optional
 
@@ -8,13 +9,17 @@
 from botocore.auth import SigV4Auth
 from botocore.awsrequest import AWSRequest
 from gremlin_python.driver import client
-from SPARQLWrapper import SPARQLWrapper
 
 from awswrangler import exceptions
 from awswrangler.neptune.gremlin_parser import GremlinParser
 
+_SPARQLWrapper_found = importlib.util.find_spec("SPARQLWrapper")
+if _SPARQLWrapper_found:
+    from SPARQLWrapper import SPARQLWrapper  # pylint: disable=import-error
+
 _logger: logging.Logger = logging.getLogger(__name__)
 
+
 DEFAULT_PORT = 8182
 NEPTUNE_SERVICE_NAME = "neptune-db"
 HTTP_PROTOCOL = "https"

awswrangler/neptune/neptune.py

@@ -1,8 +1,10 @@
 """Amazon Neptune Module."""
 
+import importlib.util
+import inspect
 import logging
 import re
-from typing import Any
+from typing import Any, Callable, TypeVar
 
 import pandas as pd
 from gremlin_python.process.graph_traversal import GraphTraversalSource, __
@@ -13,7 +15,25 @@
 from awswrangler import exceptions
 from awswrangler.neptune.client import NeptuneClient
 
+_SPARQLWrapper_found = importlib.util.find_spec("SPARQLWrapper")
+
 _logger: logging.Logger = logging.getLogger(__name__)
+FuncT = TypeVar("FuncT", bound=Callable[..., Any])
+
+
+def _check_for_sparqlwrapper(func: FuncT) -> FuncT:
+    def inner(*args: Any, **kwargs: Any) -> Any:
+        if not _SPARQLWrapper_found:
+            raise ModuleNotFoundError(
+                "You need to install SPARQLWrapper respectively the "
+                "AWS Data Wrangler package with the `sparql` extra for being able to use SPARQL "
+            )
+        return func(*args, **kwargs)
+
+    inner.__doc__ = func.__doc__
+    inner.__name__ = func.__name__
+    inner.__setattr__("__signature__", inspect.signature(func))  # pylint: disable=no-member
+    return inner  # type: ignore
 
 
 def execute_gremlin(client: NeptuneClient, query: str) -> pd.DataFrame:
@@ -72,6 +92,7 @@ def execute_opencypher(client: NeptuneClient, query: str) -> pd.DataFrame:
     return df
 
 
+@_check_for_sparqlwrapper
 def execute_sparql(client: NeptuneClient, query: str) -> pd.DataFrame:
     """Return results of a SPARQL query as pandas dataframe.
 
@@ -185,6 +206,7 @@ def to_property_graph(
     return _run_gremlin_insert(client, g)
 
 
+@_check_for_sparqlwrapper
 def to_rdf_graph(
     client: NeptuneClient,
     df: pd.DataFrame,

awswrangler/s3/_read_parquet.py

@@ -849,7 +849,7 @@ def read_parquet_table(
         This function MUST return a bool, True to read the partition or False to ignore it.
         Ignored if `dataset=False`.
         E.g ``lambda x: True if x["year"] == "2020" and x["month"] == "1" else False``
-        https://aws-data-wrangler.readthedocs.io/en/2.15.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
+        https://aws-data-wrangler.readthedocs.io/en/2.15.1/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
     columns : List[str], optional
         Names of columns to read from the file(s).
     validate_schema: