Skip to content

Commit 2dcafa8

Browse files
harshavemula-uaharshavemula-ua
committed
Add production deployment workflow
- New deploy_datastream.yml workflow for deploying datastream infra - Supports dev/prod environments with plan/apply/destroy actions - Option to enable EventBridge schedules for daily runs - Added backend-prod.hcl and variables-prod.tfvars - Updated README with prod environment documentation
1 parent d846204 commit 2dcafa8

File tree

4 files changed

+185
-1
lines changed

4 files changed

+185
-1
lines changed

‎.github/workflows/deploy_datastream.yml‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
name: Deploy Datastream Infrastructure
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
environment:
7+
description: 'Environment to deploy'
8+
required: true
9+
type: choice
10+
options:
11+
- dev
12+
- prod
13+
default: 'dev'
14+
action:
15+
description: 'Action to perform'
16+
required: true
17+
type: choice
18+
options:
19+
- plan
20+
- apply
21+
- destroy
22+
default: 'plan'
23+
enable_schedules:
24+
description: 'Enable EventBridge schedules for daily runs'
25+
type: boolean
26+
default: false
27+
28+
env:
29+
AWS_REGION: us-east-1
30+
TERRAFORM_VERSION: 1.10.0
31+
32+
permissions:
33+
contents: read
34+
id-token: write
35+
36+
jobs:
37+
deploy:
38+
name: ${{ inputs.action }} - ${{ inputs.environment }}
39+
runs-on: ubuntu-latest
40+
defaults:
41+
run:
42+
working-directory: infra/aws/terraform
43+
44+
steps:
45+
- name: Checkout repository
46+
uses: actions/checkout@v4
47+
48+
- name: Set up Python
49+
uses: actions/setup-python@v5
50+
with:
51+
python-version: '3.x'
52+
53+
- name: Install dependencies
54+
run: |
55+
sudo apt-get update && sudo apt-get install -y jq
56+
pip install --upgrade pip boto3 pandas
57+
58+
- name: Configure AWS Credentials (OIDC)
59+
uses: aws-actions/configure-aws-credentials@v4
60+
with:
61+
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
62+
aws-region: ${{ env.AWS_REGION }}
63+
role-session-name: GitHubActions-Deploy-${{ inputs.environment }}
64+
65+
- name: Set environment variables
66+
id: env
67+
run: |
68+
ENV="${{ inputs.environment }}"
69+
70+
if [ "$ENV" = "prod" ]; then
71+
echo "tf_backend=backend-prod.hcl" >> $GITHUB_OUTPUT
72+
echo "tf_vars=variables-prod.tfvars" >> $GITHUB_OUTPUT
73+
echo "sm_name=nrds_prod_sm" >> $GITHUB_OUTPUT
74+
else
75+
echo "tf_backend=backend-dev.hcl" >> $GITHUB_OUTPUT
76+
echo "tf_vars=variables.tfvars" >> $GITHUB_OUTPUT
77+
echo "sm_name=nrds_dev_sm" >> $GITHUB_OUTPUT
78+
fi
79+
80+
- name: Generate execution files
81+
working-directory: infra/aws
82+
run: |
83+
echo "Generating VPU execution files..."
84+
python python/src/research_datastream/gen_vpu_execs.py \
85+
--arch arm \
86+
--inputs terraform/modules/schedules/config/execution_forecast_inputs.json \
87+
--ami_file terraform/modules/schedules/config/community_ami.txt \
88+
--exec_template_vpu terraform/modules/schedules/executions/templates/execution_datastream_VPU_template.json \
89+
--exec_template_fp terraform/modules/schedules/executions/templates/execution_datastream_fp_template.json \
90+
--out_dir terraform/modules/schedules/executions
91+
92+
- name: Setup Terraform
93+
uses: hashicorp/setup-terraform@v3
94+
with:
95+
terraform_version: ${{ env.TERRAFORM_VERSION }}
96+
terraform_wrapper: false
97+
98+
- name: Terraform Init
99+
run: |
100+
terraform init -backend-config=${{ steps.env.outputs.tf_backend }}
101+
102+
- name: Terraform Plan
103+
if: inputs.action == 'plan' || inputs.action == 'apply'
104+
run: |
105+
terraform plan \
106+
-var-file=${{ steps.env.outputs.tf_vars }} \
107+
-var="enable_schedules=${{ inputs.enable_schedules }}" \
108+
-out=tfplan
109+
110+
echo "## Terraform Plan Summary" >> $GITHUB_STEP_SUMMARY
111+
echo '```' >> $GITHUB_STEP_SUMMARY
112+
terraform show -no-color tfplan >> $GITHUB_STEP_SUMMARY
113+
echo '```' >> $GITHUB_STEP_SUMMARY
114+
115+
- name: Terraform Apply
116+
if: inputs.action == 'apply'
117+
run: |
118+
terraform apply -auto-approve tfplan
119+
120+
echo "## Deployment Complete" >> $GITHUB_STEP_SUMMARY
121+
echo "" >> $GITHUB_STEP_SUMMARY
122+
echo "- **Environment:** ${{ inputs.environment }}" >> $GITHUB_STEP_SUMMARY
123+
echo "- **State Machine:** ${{ steps.env.outputs.sm_name }}" >> $GITHUB_STEP_SUMMARY
124+
echo "- **Schedules Enabled:** ${{ inputs.enable_schedules }}" >> $GITHUB_STEP_SUMMARY
125+
126+
- name: Terraform Destroy
127+
if: inputs.action == 'destroy'
128+
run: |
129+
terraform destroy \
130+
-var-file=${{ steps.env.outputs.tf_vars }} \
131+
-var="enable_schedules=${{ inputs.enable_schedules }}" \
132+
-auto-approve
133+
134+
echo "## Infrastructure Destroyed" >> $GITHUB_STEP_SUMMARY
135+
echo "Environment: ${{ inputs.environment }}" >> $GITHUB_STEP_SUMMARY
136+
137+
- name: Get State Machine ARN
138+
if: inputs.action == 'apply'
139+
run: |
140+
SM_NAME="${{ steps.env.outputs.sm_name }}"
141+
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
142+
ARN="arn:aws:states:${{ env.AWS_REGION }}:${ACCOUNT_ID}:stateMachine:${SM_NAME}"
143+
144+
echo "## State Machine Details" >> $GITHUB_STEP_SUMMARY
145+
echo "" >> $GITHUB_STEP_SUMMARY
146+
echo "**ARN:** \`${ARN}\`" >> $GITHUB_STEP_SUMMARY
147+
echo "" >> $GITHUB_STEP_SUMMARY
148+
echo "### Execute manually:" >> $GITHUB_STEP_SUMMARY
149+
echo '```bash' >> $GITHUB_STEP_SUMMARY
150+
echo "aws stepfunctions start-execution \\" >> $GITHUB_STEP_SUMMARY
151+
echo " --state-machine-arn \"${ARN}\" \\" >> $GITHUB_STEP_SUMMARY
152+
echo " --name \"manual-run-\$(date +%Y%m%d%H%M%S)\" \\" >> $GITHUB_STEP_SUMMARY
153+
echo " --input 'file://execution.json'" >> $GITHUB_STEP_SUMMARY
154+
echo '```' >> $GITHUB_STEP_SUMMARY
155+
156+
- name: List deployed resources
157+
if: inputs.action == 'apply'
158+
run: |
159+
echo "### Deployed Resources" >> $GITHUB_STEP_SUMMARY
160+
echo '```' >> $GITHUB_STEP_SUMMARY
161+
terraform state list >> $GITHUB_STEP_SUMMARY
162+
echo '```' >> $GITHUB_STEP_SUMMARY

‎infra/aws/terraform/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,9 @@ Multiple environments are supported via separate backend and variable files:
6161
| Environment | Backend Config | Variables | State Machine | Purpose |
6262
|-------------|----------------|-----------|---------------|---------|
6363
| dev | `backend-dev.hcl` | `variables.tfvars` | `nrds_dev_sm` | Local development |
64-
| test | `backend-test.hcl` | `variables-test.tfvars` | `nrds_test_sm` | CI/CD testing (`infra_deploy.yml`) |
64+
| test | `backend-test.hcl` | `variables-test.tfvars` | `nrds_test_sm` | CI/CD testing (`infra_deploy_val.yaml`) |
6565
| healthcheck | `backend-healthcheck.hcl` | `variables-healthcheck.tfvars` | `nrds_healthcheck_sm` | Auto-rerun failures (`health_check.yml`) |
66+
| prod | `backend-prod.hcl` | `variables-prod.tfvars` | `nrds_prod_sm` | Production deployment (`deploy_datastream.yml`) |
6667

6768
### Using an Environment
6869

@@ -83,6 +84,7 @@ Each environment has isolated Terraform state to prevent conflicts:
8384
- **dev**: `ciroh-terraform-state` bucket (us-east-2)
8485
- **test**: `ciroh-ngen-datastream-test-tfstate` bucket
8586
- **healthcheck**: `ciroh-ngen-datastream-test-tfstate` bucket (separate key)
87+
- **prod**: `ciroh-ngen-datastream-prod-tfstate` bucket
8688

8789
### Adding a New Environment
8890

‎infra/aws/terraform/backend-prod.hcl‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
bucket = "ciroh-ngen-datastream-prod-tfstate"
2+
key = "terraform.tfstate"
3+
region = "us-east-1"
4+
encrypt = true
5+
use_lockfile = true

‎infra/aws/terraform/variables-prod.tfvars‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
region = "us-east-1"
2+
sm_name = "nrds_prod_sm"
3+
sm_role_name = "nrds_prod_sm_role"
4+
sm_parameter_name = "/datastream/prod/state-machine-arn"
5+
starter_lambda_name = "nrds_prod_start_ec2"
6+
commander_lambda_name = "nrds_prod_ec2_commander"
7+
poller_lambda_name = "nrds_prod_ec2_command_poller"
8+
checker_lambda_name = "nrds_prod_s3_object_checker"
9+
stopper_lambda_name = "nrds_prod_ec2_stopper"
10+
lambda_policy_name = "nrds_prod_lambda_policy"
11+
lambda_role_name = "nrds_prod_lambda_role"
12+
lambda_invoke_policy_name = "nrds_prod_lambda_invoke_policy"
13+
ec2_role = "nrds_prod_ec2_role"
14+
ec2_policy_name = "nrds_prod_ec2_policy"
15+
profile_name = "nrds_prod_ec2_profile"

0 commit comments

Comments
 (0)