Closed
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR implements a comprehensive Terraform CI/CD pipeline for AWS infrastructure deployment and testing. The infrastructure has been modernized with S3 remote state backend, version constraints, and Parameter Store integration for dynamic state machine ARN management. The scheduler module has been temporarily disabled to allow independent deployment testing.
- Modernized Terraform configuration with S3 backend, encryption, and version constraints (Terraform >= 1.9.0, AWS provider ~> 5.0)
- Replaced hardcoded state machine ARN references with AWS Systems Manager Parameter Store lookup
- Added comprehensive GitHub Actions workflow with security scanning, VPU testing across 20 units, and automated cleanup
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| infra/aws/terraform/main.tf | Added S3 backend configuration, version constraints, and commented out scheduler module |
| infra/aws/terraform/outputs.tf | Updated outputs to use module reference instead of direct resource |
| infra/aws/terraform/modules/orchestration/main.tf | Added SSM parameter resource to store state machine ARN |
| infra/aws/terraform/modules/schedules/main.tf | Added SSM parameter data source and removed duplicate provider |
| infra/aws/terraform/modules/schedules/nrds_cfe_nom_schedules.tf | Updated file path and replaced hardcoded ARN with SSM parameter lookup |
| infra/aws/terraform/backend-dev.hcl | New S3 backend configuration for development environment |
| .github/workflows/infra_deploy_val.yaml | Comprehensive CI/CD workflow with security scans, testing, and cleanup |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
JordanLaserGit
added
NRDS AWS Deployment Through GitActions
enhancement
arpita0911patel
changed the title
JordanLaserGit
requested changes
Oct 14, 2025
Collaborator
JordanLaserGit
left a comment
JordanLaserGit
left a comment
There was a problem hiding this comment.
@harshavemula-ua great work here! I'd like to address a few things before we get this merged.
JordanLaserGit
added
the
CI/CD
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
harshavemula-ua
force-pushed
the
feature/infra-deployment
branch
9 times, most recently
from
aa26e2f to
ded1ec6
Compare
harshavemula-ua
force-pushed
the
feature/infra-deployment
branch
from
258a1e6 to
80d9797
Compare
harshavemula-ua
force-pushed
the
feature/infra-deployment
branch
4 times, most recently
from
bfac352 to
95a3f1c
Compare
harshavemula-ua
force-pushed
the
feature/infra-deployment
branch
5 times, most recently
from
3b82011 to
1c6805e
Compare
Comment on lines
+9
to
+13
| resource "aws_ssm_parameter" "state_machine_arn" { | ||
| name = var.sm_parameter_name | ||
| type = "SecureString" | ||
| value = aws_sfn_state_machine.datastream_state_machine.arn | ||
| } |
Check failure
Code scanning / checkov
Ensure SSM parameters are using KMS CMK Error
harshavemula-ua
force-pushed
the
feature/infra-deployment
branch
from
5bc1d50 to
a45c064
Compare
harshavemula-ua
force-pushed
the
feature/infra-deployment
branch
2 times, most recently
from
0789e77 to
912bb77
Compare
harshavemula-ua
force-pushed
the
feature/infra-deployment
branch
2 times, most recently
from
ed8f68b to
f343f40
Compare
- Add main.tf with orchestration module configuration - Add backend.hcl for S3 state management - Add variables.tfvars with resource naming - Add outputs.tf for state machine ARN - Document environments in README
- Add EC2 API retry config for RequestLimitExceeded handling - Update IAM policies for EC2, Lambda, and Step Functions - Add SSM parameter for state machine ARN storage - Update Lambda function configurations
- Update execution templates for VPU and FP datastream - Update scheduler IAM configuration - Update schedule definitions
- Add Terraform plan, apply, and destroy jobs - Add VPU test execution with Step Functions - Add workflow inputs for VPU and date selection - Add concurrency control and cleanup on failure - Fetch state machine ARN dynamically from SSM
- Add automated health checks for VPU executions - Add auto-rerun capability on failures
- Add modify_execution.py for test execution configuration - Add import_terraform_resources.sh for state management
- Make check_datastream_outputs.sh executable
harshavemula-ua
force-pushed
the
feature/infra-deployment
branch
from
ffdd55d to
54a4602
Compare
harshavemula-ua
force-pushed
the
feature/infra-deployment
branch
from
54a4602 to
6a623c7
Compare
github-project-automation
bot
moved this from In Progress
to Done
in NextGen Research Datastream (NRDS)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview:-
This PR introduces a comprehensive Terraform CI/CD pipeline with automated validation and testing across 20 VPUs (Vector Processing Units). The changes include a new GitHub Actions workflow that performs Terraform quality checks, security scans, automatic deployment, and end-to-end testing of both short-range and medium-range forecast processing for all VPUs with automated cleanup.
Infrastructure Modernization:-
The Terraform infrastructure has been modernized with S3 remote state backend (encrypted), version constraints (Terraform >= 1.9.0, AWS provider ~> 5.0), and improved modularization. The state machine ARN is now stored in AWS Systems Manager Parameter Store for dynamic lookup, replacing hardcoded references. The scheduler module has been temporarily disabled to allow independent deployment and testing of the orchestration components.
Key Improvements:-
The new CI/CD workflow validates infrastructure changes through automated security scanning (tfsec & Checkov), runs comprehensive VPU testing with matrix strategy, and ensures all resources are properly created and cleaned up after testing.
All Terraform configurations were successfully validated and planned locally prior to submitting this pull request, ensuring infrastructure changes are tested and error-free before deployment.