Bumping version to 0.1.0

Author: igorborgest

README.md

@@ -2,7 +2,7 @@
 
 > Utility belt to handle data on AWS.
 
-[![Release](https://img.shields.io/badge/release-0.0.25-brightgreen.svg)](https://pypi.org/project/awswrangler/)
+[![Release](https://img.shields.io/badge/release-0.1.0-brightgreen.svg)](https://pypi.org/project/awswrangler/)
 [![Downloads](https://img.shields.io/pypi/dm/awswrangler.svg)](https://pypi.org/project/awswrangler/)
 [![Python Version](https://img.shields.io/badge/python-3.6%20%7C%203.7-brightgreen.svg)](https://pypi.org/project/awswrangler/)
 [![Documentation Status](https://readthedocs.org/projects/aws-data-wrangler/badge/?version=latest)](https://aws-data-wrangler.readthedocs.io/en/latest/?badge=latest)
@@ -25,18 +25,22 @@
 ### Pandas
 * Pandas -> Parquet (S3) (Parallel)
 * Pandas -> CSV (S3) (Parallel)
-* Pandas -> Glue Catalog
+* Pandas -> Glue Catalog Table
 * Pandas -> Athena (Parallel)
 * Pandas -> Redshift (Parallel)
+* Parquet (S3) -> Pandas (Parallel) (NEW :star:)
 * CSV (S3) -> Pandas (One shot or Batching)
-* Athena -> Pandas (One shot or Batching)
+* Glue Catalog Table -> Pandas (Parallel)
+* Athena -> Pandas (One shot, Batching or Parallel (NEW :star:))
+* Redshift -> Pandas (Parallel) (NEW :star:)
+* Redshift -> Parquet (S3) (NEW :star:)
 * CloudWatch Logs Insights -> Pandas
 * Encrypt Pandas Dataframes on S3 with KMS keys
 
 ### PySpark
 * PySpark -> Redshift (Parallel)
 * Register Glue table from Dataframe stored on S3
-* Flatten nested DataFrames (NEW :star:)
+* Flatten nested DataFrames
 
 ### General
 * List S3 objects (Parallel)
@@ -47,13 +51,13 @@
 * Get the size of S3 objects (Parallel)
 * Get CloudWatch Logs Insights query results
 * Load partitions on Athena/Glue table (repair table)
-* Create EMR cluster (For humans) (NEW :star:)
-* Terminate EMR cluster (NEW :star:)
-* Get EMR cluster state (NEW :star:)
-* Submit EMR step(s) (For humans) (NEW :star:)
-* Get EMR step state (NEW :star:)
-* Get EMR step state (NEW :star:)
-* Athena query to receive the result as python primitives (*Iterable[Dict[str, Any]*) (NEW :star:)
+* Create EMR cluster (For humans)
+* Terminate EMR cluster
+* Get EMR cluster state
+* Submit EMR step(s) (For humans)
+* Get EMR step state
+* Get EMR step state
+* Athena query to receive the result as python primitives (*Iterable[Dict[str, Any]*)
 
 ## Installation
 
@@ -74,8 +78,8 @@ Runs anywhere (AWS Lambda, AWS Glue Python Shell, EMR, EC2, on-premises, local,
 #### Writing Pandas Dataframe to S3 + Glue Catalog
 
 ```py3
-session = awswrangler.Session()
-session.pandas.to_parquet(
+wrangler = awswrangler.Session()
+wrangler.pandas.to_parquet(
     dataframe=dataframe,
     database="database",
     path="s3://...",
@@ -92,17 +96,17 @@ extra_args = {
     "ServerSideEncryption": "aws:kms",
     "SSEKMSKeyId": "YOUR_KMY_KEY_ARN"
 }
-session = awswrangler.Session(s3_additional_kwargs=extra_args)
-session.pandas.to_parquet(
+wrangler = awswrangler.Session(s3_additional_kwargs=extra_args)
+wrangler.pandas.to_parquet(
     path="s3://..."
 )
 ```
 
 #### Reading from AWS Athena to Pandas
 
 ```py3
-session = awswrangler.Session()
-dataframe = session.pandas.read_sql_athena(
+wrangler = awswrangler.Session()
+dataframe = wrangler.pandas.read_sql_athena(
     sql="select * from table",
     database="database"
 )
@@ -111,8 +115,8 @@ dataframe = session.pandas.read_sql_athena(
 #### Reading from AWS Athena to Pandas in chunks (For memory restrictions)
 
 ```py3
-session = awswrangler.Session()
-dataframe_iter = session.pandas.read_sql_athena(
+wrangler = awswrangler.Session()
+dataframe_iter = wrangler.pandas.read_sql_athena(
     sql="select * from table",
     database="database",
     max_result_size=512_000_000  # 512 MB
@@ -121,18 +125,28 @@ for dataframe in dataframe_iter:
     print(dataframe)  # Do whatever you want
 ```
 
+#### Reading from AWS Athena to Pandas with the blazing fast CTAS approach
+
+```py3
+wrangler = awswrangler.Session(athena_ctas_approach=True)
+dataframe = wrangler.pandas.read_sql_athena(
+    sql="select * from table",
+    database="database"
+)
+```
+
 #### Reading from S3 (CSV) to Pandas
 
 ```py3
-session = awswrangler.Session()
-dataframe = session.pandas.read_csv(path="s3://...")
+wrangler = awswrangler.Session()
+dataframe = wrangler.pandas.read_csv(path="s3://...")
 ```
 
 #### Reading from S3 (CSV) to Pandas in chunks (For memory restrictions)
 
 ```py3
-session = awswrangler.Session()
-dataframe_iter = session.pandas.read_csv(
+wrangler = awswrangler.Session()
+dataframe_iter = wrangler.pandas.read_csv(
     path="s3://...",
     max_result_size=512_000_000  # 512 MB
 )
@@ -143,8 +157,8 @@ for dataframe in dataframe_iter:
 #### Reading from CloudWatch Logs Insights to Pandas
 
 ```py3
-session = awswrangler.Session()
-dataframe = session.pandas.read_log_query(
+wrangler = awswrangler.Session()
+dataframe = wrangler.pandas.read_log_query(
     log_group_names=[LOG_GROUP_NAME],
     query="fields @timestamp, @message | sort @timestamp desc | limit 5",
 )
@@ -160,22 +174,49 @@ df = pandas.read_...  # Read from anywhere
 
 # Typical Pandas, Numpy or Pyarrow transformation HERE!
 
-session = awswrangler.Session()
-session.pandas.to_parquet(  # Storing the data and metadata to Data Lake
+wrangler = awswrangler.Session()
+wrangler.pandas.to_parquet(  # Storing the data and metadata to Data Lake
     dataframe=dataframe,
     database="database",
     path="s3://...",
     partition_cols=["col_name"],
 )
 ```
 
+#### Loading Pandas Dataframe to Redshift
+
+```py3
+wrangler = awswrangler.Session()
+wrangler.pandas.to_redshift(
+    dataframe=dataframe,
+    path="s3://temp_path",
+    schema="...",
+    table="...",
+    connection=con,
+    iam_role="YOUR_ROLE_ARN",
+    mode="overwrite",
+    preserve_index=False,
+)
+```
+
+#### Extract Redshift query to Pandas DataFrame
+
+```py3
+wrangler = awswrangler.Session()
+dataframe = session.pandas.read_sql_redshift(
+    sql="SELECT ...",
+    iam_role="YOUR_ROLE_ARN",
+    connection=con,
+    temp_s3_path="s3://temp_path")
+```
+
 ### PySpark
 
 #### Loading PySpark Dataframe to Redshift
 
 ```py3
-session = awswrangler.Session(spark_session=spark)
-session.spark.to_redshift(
+wrangler = awswrangler.Session(spark_session=spark)
+wrangler.spark.to_redshift(
     dataframe=df,
     path="s3://...",
     connection=conn,
@@ -194,20 +235,21 @@ dataframe.write \
         .format("parquet") \
         .partitionBy(["year", "month"]) \
         .save(compression="gzip", path="s3://...")
-session = awswrangler.Session(spark_session=spark)
-session.spark.create_glue_table(dataframe=dataframe,
-                                file_format="parquet",
-                                partition_by=["year", "month"],
-                                path="s3://...",
-                                compression="gzip",
-                                database="my_database")
+wrangler = awswrangler.Session(spark_session=spark)
+wrangler.spark.create_glue_table(
+    dataframe=dataframe,
+    file_format="parquet",
+    partition_by=["year", "month"],
+    path="s3://...",
+    compression="gzip",
+    database="my_database")
 ```
 
 #### Flatten nested PySpark DataFrame
 
 ```py3
-session = awswrangler.Session(spark_session=spark)
-dfs = session.spark.flatten(dataframe=df_nested)
+wrangler = awswrangler.Session(spark_session=spark)
+dfs = wrangler.spark.flatten(dataframe=df_nested)
 for name, df_flat in dfs.items():
     print(name)
     df_flat.show()
@@ -218,15 +260,15 @@ for name, df_flat in dfs.items():
 #### Deleting a bunch of S3 objects (parallel)
 
 ```py3
-session = awswrangler.Session()
-session.s3.delete_objects(path="s3://...")
+wrangler = awswrangler.Session()
+wrangler.s3.delete_objects(path="s3://...")
 ```
 
 #### Get CloudWatch Logs Insights query results
 
 ```py3
-session = awswrangler.Session()
-results = session.cloudwatchlogs.query(
+wrangler = awswrangler.Session()
+results = wrangler.cloudwatchlogs.query(
     log_group_names=[LOG_GROUP_NAME],
     query="fields @timestamp, @message | sort @timestamp desc | limit 5",
 )
@@ -235,15 +277,15 @@ results = session.cloudwatchlogs.query(
 #### Load partitions on Athena/Glue table (repair table)
 
 ```py3
-session = awswrangler.Session()
-session.athena.repair_table(database="db_name", table="tbl_name")
+wrangler = awswrangler.Session()
+wrangler.athena.repair_table(database="db_name", table="tbl_name")
 ```
 
 #### Create EMR cluster
 
 ```py3
-session = awswrangler.Session()
-cluster_id = session.emr.create_cluster(
+wrangler = awswrangler.Session()
+cluster_id = wrangler.emr.create_cluster(
     cluster_name="wrangler_cluster",
     logging_s3_path=f"s3://BUCKET_NAME/emr-logs/",
     emr_release="emr-5.27.0",
@@ -284,6 +326,7 @@ cluster_id = session.emr.create_cluster(
     maximize_resource_allocation=True,
     keep_cluster_alive_when_no_steps=True,
     termination_protected=False,
+    spark_pyarrow=True,
     tags={
         "foo": "boo"
     }
@@ -294,8 +337,8 @@ print(cluster_id)
 #### Athena query to receive the result as python primitives (*Iterable[Dict[str, Any]*)
 
 ```py3
-session = awswrangler.Session()
-for row in session.athena.query(query="...", database="..."):
+wrangler = awswrangler.Session()
+for row in wrangler.athena.query(query="...", database="..."):
     print(row)
 ```
 

awswrangler/__version__.py

@@ -1,4 +1,4 @@
 __title__ = "awswrangler"
 __description__ = "Utility belt to handle data on AWS."
-__version__ = "0.0.25"
+__version__ = "0.1.0"
 __license__ = "Apache License 2.0"

awswrangler/pandas.py

@@ -505,14 +505,16 @@ def read_sql_athena(self,
         """
         Executes any SQL query on AWS Athena and return a Dataframe of the result.
         There are two approaches to be defined through ctas_approach parameter:
+
         1 - ctas_approach True (For Huge results):
-            Wrap the query with a CTAS and then reads the table data as parquet directly from s3.
-            PROS: Faster and has a better handle of nested types
-            CONS: Can't use max_result_size and must have create and drop table permissions
+        Wrap the query with a CTAS and then reads the table data as parquet directly from s3.
+        PROS: Faster and has a better handle of nested types
+        CONS: Can't use max_result_size and must have create and drop table permissions
+
         2 - ctas_approach False (Default):
-            Does a regular query on Athena and parse the regular CSV result on s3
-            PROS: Accepts max_result_size.
-            CONS: Slower (But stills faster than other libraries that uses the Athena API) and does not handle nested types so well
+        Does a regular query on Athena and parse the regular CSV result on s3
+        PROS: Accepts max_result_size.
+        CONS: Slower (But stills faster than other libraries that uses the Athena API) and does not handle nested types so well
 
         P.S. If ctas_approach is False and max_result_size is passed, then a iterator of Dataframes is returned.
         P.S.S. All default values will be inherited from the Session()