Add SSM command output logging to S3 (Traceability)

[harshavemula-ua]

Summary

• Add full EC2 command stdout/stderr logging to s3://ciroh-community-ngen-datastream/ssm-logs/{execution_name}/ via SSM OutputS3BucketName and OutputS3KeyPrefix
• Inject Step Functions execution name into Commander Lambda payload using States.JsonMerge for traceable, browsable log paths
• Logs are organized by execution name (e.g., ssm-logs/cfe_nom_short_range_00_VPU_07_20260211.../) matching the Step Functions console

S3 Log Folder Structure

ssm-logs/
  └── {execution_name}/                          ← matches Step Functions execution name
        └── {command_id}/                         ← SSM command ID
              └── {instance_id}/                  ← EC2 instance ID
                    └── awsrunShellScript/
                          └── 0.awsrunShellScript/
                                ├── stdout        ← full command output
                                └── stderr        ← full error output

Lifecycle Configuration

aws s3api put-bucket-lifecycle-configuration \
  --bucket ciroh-community-ngen-datastream \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "DeleteSSMLogsAfter3Days",
        "Filter": {
          "Prefix": "ssm-logs/"
        },
        "Status": "Enabled",
        "Expiration": {
          "Days": 3
        }
      }
    ]
  }'

Sample Output

• stdout.txt

Test plan

• Deployed to test_harsha environment
• Triggered state machine execution — Commander succeeded
• Verified SSM logs written to s3://ciroh-community-ngen-datastream/ssm-logs/test-exec-logging-20260211-123820/
• Confirmed full stdout (1.2 MB) and stderr (38 KB) captured without truncation
• Verify no impact on existing scheduled production executions

[JordanLaserGit]

@harshavemula-ua, I think this is a worthy and needed feature, but it may motivate some changes in datastreamcli or the prod bmi configs before we deploy. Reason being is NextGen can generate a massive amount of print outs, which will bog the ec2 instance down if ssm needs to stream everything to s3. I've dealt with this in the past by piping all of the ngiab docker print outs to /dev/null, but this gets rid of everything that nextgen prints out. Before we merge this, let's do some tests to confirm the ec2 does not slow down substantially for any execution type. If it does, let's identify why and see if we can suppress printouts in a judicial way.