cleaning up and adding README

Author: Kamen Sharlandjiev

usecases/mwaa-dag-factory-example/README.md

@@ -1,6 +1,6 @@
 # MWAA Data Pipeline Workshop
 
-One-command deployment of a complete MWAA data pipeline with VPC, Redshift, Glue, and EMR Serverless.
+One-command deployment of a provisioned MWAA instance with DAG Factory installed. As part of this stack data pipeline with VPC, Redshift, Glue, and EMR Serverless.
 
 ## Quick Start
 
@@ -19,13 +19,50 @@ python3 cleanup.py
 
 ## What Gets Deployed
 
+### Provisioned MWAA Instance
+- **MWAA 3.0.6** environment running Apache Airflow
+- **DAG Factory** pre-installed for YAML-based DAG definitions
+- Small environment size (suitable for development/testing)
+
+### Data Pipeline Examples
+1. **Python DAG** (`dags/data_pipeline.py`) - Traditional Airflow DAG written in Python
+2. **YAML DAG** (`yaml/example_data_pipeline.yaml`) - DAG Factory representation of the same pipeline
+
+### Infrastructure Components
 - VPC with public/private subnets, NAT gateways
-- MWAA 2.10.3 small environment
 - Redshift Serverless (8 RPU)
 - S3 bucket with DAGs, scripts, sample data
 - IAM roles for Glue, EMR, Redshift
-- Python data pipeline DAG example
-- YAML data pipeline DAG example
+
+## Deploying to MWAA Serverless
+
+The YAML-based DAG can be deployed to MWAA Serverless using the following commands:
+
+### 1. Convert Python DAG to YAML
+```bash
+dag-converter convert data_pipeline.py --output yaml/
+```
+
+### 2. Upload YAML to S3
+```bash
+aws s3 sync yaml/ s3://YOUR-BUCKET-NAME/yaml/
+```
+
+### 3. Create Serverless Workflow
+```bash
+aws mwaa-serverless create-workflow \
+  --name example_data_pipeline \
+  --definition-s3-location '{ "Bucket": "YOUR-BUCKET-NAME", "ObjectKey": "yaml/example_data_pipeline.yaml" }' \
+  --role-arn arn:aws:iam::YOUR-ACCOUNT-ID:role/service-role/YOUR-MWAA-EXECUTION-ROLE \
+  --region us-east-2
+```
+
+### 4. List Serverless Workflows
+```bash
+aws mwaa-serverless list-workflows --region us-east-2
+```
+
+> **Note:** Replace `YOUR-BUCKET-NAME`, `YOUR-ACCOUNT-ID`, and `YOUR-MWAA-EXECUTION-ROLE` with your actual AWS resource identifiers.
 
 ## Pipeline Flow
 

usecases/mwaa-dag-factory-example/dags/dag_converter_pipeline.py

@@ -61,7 +61,7 @@
         script_args={
             '--dag_name': 'immersion_day_data_pipeline',
             '--task_id': 'glue_job',
-            '--correlation_id': '{{ run_id }}'
+            '--correlation_id': 'start_emr_serverless_job'
         }
     )
 
@@ -86,7 +86,7 @@
                     "s3://{{S3_BUCKET_NAME}}/data/aggregated/green",
                     "immersion_day_data_pipeline",
                     "start_emr_serverless_job",
-                    "{{ run_id }}"
+                    "start_emr_serverless_job"
                 ],
                 "sparkSubmitParameters": "--conf spark.executor.instances=2 --conf spark.executor.memory=4G --conf spark.executor.cores=2 --conf spark.executor.memoryOverhead=1G"
             }