Version 2.4.0 release preparation.

Author: igorborgest

CONTRIBUTING.md

@@ -84,7 +84,7 @@ You can choose from three different environments to test your fixes/changes, bas
 ### Mocked test environment
 
 * Pick up a Linux or MacOS.
-* Install Python 3.6, 3.7 or 3.8
+* Install Python 3.7, 3.8 or 3.9
 * Fork the AWS Data Wrangler repository and clone that into your development environment
 * Go to the project's directory create a Python's virtual environment for the project
 
@@ -115,7 +115,7 @@ or
 **DISCLAIMER**: Make sure you know what you are doing. These steps will charge some services on your AWS account and require a minimum security skill to keep your environment safe.
 
 * Pick up a Linux or MacOS.
-* Install Python 3.6, 3.7 or 3.8
+* Install Python 3.7, 3.8 or 3.9
 * Fork the AWS Data Wrangler repository and clone that into your development environment
 * Go to the project's directory create a Python's virtual environment for the project
 
@@ -168,7 +168,7 @@ or
 **DISCLAIMER**: This environment contains Aurora MySQL, Aurora PostgreSQL and Redshift (single-node) clusters which will incur cost while running.
 
 * Pick up a Linux or MacOS.
-* Install Python 3.6, 3.7 and 3.8
+* Install Python 3.7, 3.8 or 3.9
 * Fork the AWS Data Wrangler repository and clone that into your development environment
 * Go to the project's directory create a Python's virtual environment for the project
 
@@ -244,5 +244,5 @@ or
 
 ## Common Errors
 
-Check the file below to check the common errors and solutions 
+Check the file below to check the common errors and solutions
 [ERRORS](https://github.com/awslabs/aws-data-wrangler/blob/main/CONTRIBUTING_COMMON_ERRORS.md)

awswrangler/_utils.py

@@ -63,9 +63,11 @@ def boto3_from_primitives(primitives: Optional[Boto3PrimitivesType] = None) -> b
 def default_botocore_config() -> botocore.config.Config:
     """Botocore configuration."""
     retries_config: Dict[str, Union[str, int]] = {
-        "max_attempts": int(os.getenv("AWS_MAX_ATTEMPTS", "3")),
-        "mode": os.getenv("AWS_RETRY_MODE", "standard"),
+        "max_attempts": int(os.getenv("AWS_MAX_ATTEMPTS", "5")),
     }
+    mode: Optional[str] = os.getenv("AWS_RETRY_MODE")
+    if mode:
+        retries_config["mode"] = mode
     return botocore.config.Config(retries=retries_config, connect_timeout=10, max_pool_connections=10)
 
 

awswrangler/s3/__init__.py

@@ -5,6 +5,7 @@
 from awswrangler.s3._describe import describe_objects, get_bucket_region, size_objects  # noqa
 from awswrangler.s3._download import download  # noqa
 from awswrangler.s3._list import does_object_exist, list_directories, list_objects  # noqa
+from awswrangler.s3._merge_upsert_table import merge_upsert_table  # noqa
 from awswrangler.s3._read_excel import read_excel  # noqa
 from awswrangler.s3._read_parquet import read_parquet, read_parquet_metadata, read_parquet_table  # noqa
 from awswrangler.s3._read_text import read_csv, read_fwf, read_json  # noqa

building/build-wheel.sh

@@ -3,5 +3,5 @@ set -ex
 
 pushd ..
 rm -rf *.egg-info build dist/*.whl
-python3.6 setup.py bdist_wheel
+python setup.py bdist_wheel
 rm -rf *.egg-info build

building/lambda/build-lambda-layer.sh

@@ -14,7 +14,7 @@ export ARROW_HOME=$(pwd)/dist
 export LD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH
 
 git clone \
-  --branch apache-arrow-2.0.0 \
+  --branch apache-arrow-3.0.0 \
   --single-branch \
   https://github.com/apache/arrow.git
 

docs/source/install.rst

@@ -57,12 +57,11 @@ AWS Glue PySpark Jobs
 Go to your Glue PySpark job and create a new *Job parameters* key/value:
 
 * Key: ``--additional-python-modules``
-* Value: ``awswrangler``
-
-To install a specific version, set the value for above Job parameter as follows:
-
 * Value: ``awswrangler==2.3.0``
 
+P.S. By now AWS Glue PySpark Jobs does not support PyArrow +3.0.0.
+Please use awswrangler==2.3.0 that uses PyArrow 2.0.0 to overcome this limitation.
+
 `Official Glue PySpark Reference <https://docs.aws.amazon.com/glue/latest/dg/reduced-start-times-spark-etl-jobs.html#reduced-start-times-new-features>`_
 
 Amazon SageMaker Notebook

requirements-dev.txt

@@ -1,3 +1,4 @@
+wheel==0.36.2
 isort==5.7.0
 black==20.8b1
 pylint==2.6.0

tests/test_config.py

@@ -129,16 +129,16 @@ def test_botocore_config(path):
     original = botocore.client.ClientCreator.create_client
 
     # Default values for botocore.config.Config
-    expected_max_retries_attempt = 3
+    expected_max_retries_attempt = 5
     expected_connect_timeout = 10
     expected_max_pool_connections = 10
-    expected_retry_mode = "standard"
+    expected_retry_mode = None
 
     def wrapper(self, **kwarg):
         assert kwarg["client_config"].retries["max_attempts"] == expected_max_retries_attempt
         assert kwarg["client_config"].connect_timeout == expected_connect_timeout
         assert kwarg["client_config"].max_pool_connections == expected_max_pool_connections
-        assert kwarg["client_config"].retries["mode"] == expected_retry_mode
+        assert kwarg["client_config"].retries.get("mode") == expected_retry_mode
         return original(self, **kwarg)
 
     # Check for default values

tox.ini

@@ -11,12 +11,12 @@ deps =
        moto==1.3.16
        s3fs==0.4.2
 commands =
-       pytest -n 16 --timeout=300 tests
+       pytest -n 32 --timeout=300 tests
 
 [testenv:py39]
 passenv = AWS_PROFILE AWS_DEFAULT_REGION AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
 deps =
        {[testenv]deps}
-       pytest-cov==2.10.1
+       pytest-cov==2.11.1
 commands =
-       pytest -n 16 --timeout=300 --cov=awswrangler tests
+       pytest -n 32 --timeout=300 --cov=awswrangler tests

tutorials/007 - Redshift, MySQL, PostgreSQL, SQL Server.ipynb

@@ -39,7 +39,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Connect throught Glue Catalog Connections\n",
+    "## Connect using the Glue Catalog Connections\n",
     "\n",
     "- [wr.redshift.connect()](https://aws-data-wrangler.readthedocs.io/en/stable/stubs/awswrangler.redshift.connect.html)\n",
     "- [wr.mysql.connect()](https://aws-data-wrangler.readthedocs.io/en/stable/stubs/awswrangler.mysql.connect.html)\n",