Merge pull request #54 from awslabs/eoferror

GeoPandas limitation explained in the docs

Author: igorborgest

README.md

@@ -285,6 +285,23 @@ print(cluster_id)
 ## Diving Deep
 
 
+### Parallelism, Non-picklable objects and GeoPandas
+
+AWS Data Wrangler tries to parallelize everything that is possible (I/O and CPU bound task).
+You can control the parallelism level using the parameters:
+
+- **procs_cpu_bound**: number of processes that can be used in single node applications for CPU bound case (Default: os.cpu_count())
+- **procs_io_bound**: number of processes that can be used in single node applications for I/O bound cases (Default: os.cpu_count() * PROCS_IO_BOUND_FACTOR) 
+
+Both can be defined on Session level or directly in the functions.
+
+Some special cases will not work with parallelism:
+
+- GeoPandas
+- Columns with non-picklable objects
+
+To handle that use `procs_cpu_bound=1` and avoid the distribution of the dataframe.
+
 ### Pandas with null object columns (UndetectedType exception)
 
 Pandas has a too generic "data type" named object. Pandas object columns can be string, dates, etc, etc, etc.

building/build-docs.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -e
 
 cd ..

building/build-glue-egg.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -e
 
 cd ..

building/build-glue-wheel.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -e
 
 cd ..

building/build-image.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -e
 
 cp ../requirements.txt .

building/build-lambda-layer.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -e
 
 # Go to home

building/deploy-source.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -e
 
 cd ..

building/open-image.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 AWS_ACCESS_KEY_ID=$(aws --profile default configure get aws_access_key_id)
 AWS_SECRET_ACCESS_KEY=$(aws --profile default configure get aws_secret_access_key)

building/publish.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -e
 
 cd ..

docs/source/divingdeep.rst

@@ -3,6 +3,24 @@
 Diving Deep
 ===========
 
+Parallelism, Non-picklable objects and GeoPandas
+------------------------------------------------
+
+AWS Data Wrangler tries to parallelize everything that is possible (I/O and CPU bound task).
+You can control the parallelism level using the parameters:
+
+- procs_cpu_bound: number of processes that can be used in single node applications for CPU bound case (Default: os.cpu_count())
+- procs_io_bound: number of processes that can be used in single node applications for I/O bound cases (Default: os.cpu_count() * PROCS_IO_BOUND_FACTOR)
+
+Both can be defined on Session level or directly in the functions.
+
+Some special cases will not work with parallelism:
+
+- GeoPandas
+- Columns with non-picklable objects
+
+To handle that use `procs_cpu_bound=1` and avoid the distribution of the dataframe.
+
 Pandas with null object columns (UndetectedType exception)
 ----------------------------------------------------------