Add automated EC2 cleanup and reduce workflow parallelism

Changes:
- Add EC2 cleanup step in workflow to terminate orphaned test instances
- Reduce max-parallel from 9 to 5 for both test jobs (cost optimization)
- Add cleanup Lambda module for daily automated cleanup (safety net)
- Add comprehensive documentation for EC2 cleanup strategy

Workflow improvements:
- Cleanup runs before terraform destroy with if: always()
- Terminates instances matching test patterns immediately
- Reduces cost from long-running orphaned instances

Cost optimization:
- Lower parallelism reduces simultaneous instance count
- Immediate cleanup prevents instances running for hours
- Lambda provides backup cleanup at 1 AM UTC daily

Documentation:
- EC2_CLEANUP_PLAN.md: Multi-layer cleanup strategy
- IAM_SECURITY_INVESTIGATION.md: IAM security analysis
- modules/cleanup/: Complete Terraform module for Lambda cleanup

Author: harshavemula-ua

.github/workflows/infra_deploy_val.yaml

@@ -325,7 +325,7 @@ jobs:
     timeout-minutes: 90
     strategy:
       fail-fast: false
-      max-parallel: 9
+      max-parallel: 5
       matrix:
         vpu: ["01", "02", "03N", "03S", "03W", "04", "05", "06", "07", "08", "09", "10L", "10U", "11", "12", "13", "14", "15", "16", "17", "18"]
     permissions:
@@ -517,7 +517,7 @@ jobs:
     timeout-minutes: 150
     strategy:
       fail-fast: false
-      max-parallel: 9
+      max-parallel: 5
       matrix:
         vpu: ["01", "02", "03N", "03S", "03W", "04", "05", "06", "07", "08", "09", "10L", "10U", "11", "12", "13", "14", "15", "16", "17", "18"]
     permissions:
@@ -807,7 +807,40 @@ jobs:
               done
           fi
         continue-on-error: true
-      
+
+      - name: Terminate Orphaned Test EC2 Instances
+        if: always()
+        run: |
+          echo "Checking for orphaned test EC2 instances..."
+
+          # Get all running/pending test instances from this workflow
+          INSTANCE_IDS=$(aws ec2 describe-instances \
+            --filters \
+              "Name=tag:Project,Values=test_short_range_vpu_*,test_medium_range_vpu_*,test_analysis_assim_extend_vpu_*" \
+              "Name=instance-state-name,Values=running,pending" \
+            --query 'Reservations[*].Instances[*].[InstanceId,Tags[?Key==`Project`].Value|[0],LaunchTime]' \
+            --output text)
+
+          if [ -z "$INSTANCE_IDS" ]; then
+            echo "✓ No orphaned test instances found"
+          else
+            echo "Found orphaned instances:"
+            echo "$INSTANCE_IDS"
+
+            # Extract just instance IDs
+            INSTANCE_LIST=$(echo "$INSTANCE_IDS" | awk '{print $1}' | tr '\n' ' ')
+
+            if [ -n "$INSTANCE_LIST" ]; then
+              echo "Terminating instances: $INSTANCE_LIST"
+              aws ec2 terminate-instances --instance-ids $INSTANCE_LIST || echo "Failed to terminate some instances"
+
+              # Count terminated instances
+              INSTANCE_COUNT=$(echo $INSTANCE_LIST | wc -w)
+              echo "✓ Terminated $INSTANCE_COUNT orphaned test instances"
+            fi
+          fi
+        continue-on-error: true
+
       - name: Terraform Destroy
         id: destroy_attempt
         run: |