Feature Proposal: Support for Existing VPC and Load Balancer Resources

[aarora79]

Feature Enhancement: Support for Existing VPC and Load Balancer Resources

Summary

Add support for using existing VPC and Application Load Balancer (ALB) resources instead of always creating new ones. This enables users with established AWS infrastructure to integrate the MCP Gateway Registry into their existing network topology, reducing resource sprawl and simplifying network management.

Motivation

Current State

The Terraform deployment (terraform/aws-ecs) currently creates all networking resources from scratch:

• New VPC with CIDR block configuration
• New public and private subnets across 3 availability zones
• New NAT gateways (one per AZ)
• New VPC endpoints for STS and S3
• New Application Load Balancers:
  • Main ALB for Registry and Auth Server
  • Keycloak ALB

Business Use Cases

Many organizations already have:

1. Established VPC Architecture: Shared VPC with existing CIDR blocks, subnets, route tables, and connectivity to on-premises networks via Direct Connect or VPN
2. Centralized Load Balancers: Shared ALBs with host-based or path-based routing rules for multiple applications
3. Network Policies: Pre-approved security groups, NACLs, and VPC endpoints
4. Cost Optimization: Existing NAT Gateways and internet gateways that can be reused (significant monthly savings on redundant network infrastructure)
5. Compliance Requirements: VPC configurations that meet organizational security and compliance standards

Proposed Solution

High-Level Design

Add new optional variables to support "bring your own" (BYO) networking resources - see full proposal in the attached document for detailed variable definitions.

Key capabilities:

• create_vpc flag to use existing VPC instead of creating new one
• create_main_alb and create_keycloak_alb flags for existing ALBs
• Validation to ensure existing resources meet requirements
• Support for both host-based and path-based routing when using existing ALBs

Architecture Considerations

Potential Issues and Concerns

1. Security Group Dependencies - Current implementation references VPC module's CIDR blocks
2. NAT Gateway IP Addresses - Security groups reference NAT IPs for Keycloak ALB ingress
3. VPC Endpoints - Existing VPC may already have required endpoints
4. Subnet Requirements - Validation needed for multi-AZ, routing configuration
5. ALB Target Group Conflicts - Name collisions and port conflicts
6. Listener Priority Management - Managing rule priorities on existing ALBs
7. DNS and Route53 Assumptions - Handling existing DNS records
8. Database Subnet Group - Option to use existing DB subnet groups
9. EFS Mount Target Conflicts - Validation for mount target limits
10. Service Discovery Namespace - Avoiding namespace conflicts

Distributed System Considerations

• Cross-AZ traffic costs when using existing subnets
• Network ACL compatibility
• VPC Flow Logs configuration
• IP address exhaustion in existing subnets
• Transit Gateway / VPC Peering compatibility

When This Should NOT Be Done

1. Single-Tenant Deployments: Simple isolated deployments benefit from batteries-included approach
2. Proof-of-Concept / Testing: Overhead not worth it for temporary deployments
3. Organizations Without Established Infrastructure: Startups benefit from opinionated defaults
4. When Network Policies Are Fluid: Independent evolution is easier with isolated resources

Recommended Approach

1. Phase 1: Implement BYO VPC support first (simpler, higher value)
2. Phase 2: Add BYO ALB support (more complex, addresses specific use cases)
3. Documentation: Create comprehensive guides before releasing each phase
4. Testing: Add automated tests for all scenarios
5. Gradual Rollout: Mark as "experimental" initially, gather feedback, iterate

Alternatives Considered

Alternative 1: VPC Peering

Instead of deploying into existing VPC, peer a new VPC with existing infrastructure.

• Pros: Simpler implementation, isolated resources
• Cons: Increased cost, complex routing, doesn't solve ALB consolidation

Alternative 2: Separate Terraform Module

Create a "shared infrastructure" module that customers run first, then reference outputs.

• Pros: Clear separation of concerns
• Cons: More complex deployment process, harder to maintain

Alternative 3: Use Terragrunt

Use Terragrunt to manage dependencies between infrastructure modules.

• Pros: Professional tooling, good for complex deployments
• Cons: Additional tool dependency, learning curve

Questions for Discussion

1. Priority: Should VPC support be prioritized over ALB support, or should both be implemented together?

2. Scope: Are there other existing resources users would want to bring (DB subnet groups, security groups, EFS)?

3. Validation Strategy: What level of validation is appropriate? Fail-fast with strict checks vs. allow flexibility?

4. Documentation Approach: Should we create a separate deployment guide for BYO scenarios or integrate into the main README?

5. Backward Compatibility: Any concerns about maintaining backward compatibility while adding these features?

6. Use Cases: What specific use cases or organizational requirements would benefit most from this feature?

Call to Action

We'd love to hear from the community:

• If you've deployed MCP Gateway Registry, would this feature be useful for your organization?
• What existing infrastructure would you want to reuse?
• What challenges or concerns do you foresee?
• Are there alternative approaches we should consider?

Please share your thoughts, use cases, and feedback in the comments below!

References

• Current VPC implementation: terraform/aws-ecs/vpc.tf
• Current ALB implementation: terraform/aws-ecs/modules/mcp-gateway/networking.tf
• Keycloak ALB: terraform/aws-ecs/keycloak-alb.tf
• Full detailed proposal: .scratchpad/feature-bring-your-own-vpc-alb.md

[aarora79]

Implementation Details

This comment provides detailed implementation guidance for the "Bring Your Own VPC/ALB" feature proposal.

Phase 1: VPC Support Implementation

1. Conditional VPC Creation

Update vpc.tf to conditionally create VPC:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"

  count = var.create_vpc ? 1 : 0
  
  name = "${var.name}-vpc"
  cidr = var.vpc_cidr

  azs             = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 4, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k + 48)]

  enable_nat_gateway     = true
  single_nat_gateway     = false
  one_nat_gateway_per_az = true

  enable_dns_hostnames = true
  enable_dns_support   = true
}

locals {
  vpc_id              = var.create_vpc ? module.vpc[0].vpc_id : var.existing_vpc_id
  private_subnet_ids  = var.create_vpc ? module.vpc[0].private_subnets : var.existing_private_subnet_ids
  public_subnet_ids   = var.create_vpc ? module.vpc[0].public_subnets : var.existing_public_subnet_ids
  vpc_cidr_block      = var.create_vpc ? module.vpc[0].vpc_cidr_block : data.aws_vpc.existing[0].cidr_block
  nat_public_ips      = var.create_vpc ? module.vpc[0].nat_public_ips : var.existing_nat_gateway_ips
}

data "aws_vpc" "existing" {
  count = var.create_vpc ? 0 : 1
  id    = var.existing_vpc_id
}

2. Conditional VPC Endpoints

Only create VPC endpoints when create_vpc = true:

resource "aws_vpc_endpoint" "sts" {
  count = var.create_vpc ? 1 : 0
  
  vpc_id             = local.vpc_id
  service_name       = "${local.interface_endpoint_prefix}.${data.aws_region.current.region}.sts"
  vpc_endpoint_type  = "Interface"
  subnet_ids         = local.private_subnet_ids
  security_group_ids = [aws_security_group.vpc_endpoints[0].id]

  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "s3" {
  count = var.create_vpc ? 1 : 0
  
  vpc_id            = local.vpc_id
  service_name      = "${local.gateway_endpoint_prefix}.${data.aws_region.current.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = var.create_vpc ? module.vpc[0].private_route_table_ids : var.existing_private_route_table_ids
}

3. Update Security Groups

Replace all module.vpc.vpc_id references with local.vpc_id:

resource "aws_security_group" "vpc_endpoints" {
  count = var.create_vpc ? 1 : 0
  
  name        = "${var.name}-vpc-endpoints"
  description = "Security group for VPC endpoints"
  vpc_id      = local.vpc_id

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [local.vpc_cidr_block]
  }
}

4. Validation Rules

Add comprehensive validation:

variable "existing_vpc_id" {
  description = "ID of existing VPC (required if create_vpc = false)"
  type        = string
  default     = ""
  
  validation {
    condition     = var.create_vpc || var.existing_vpc_id != ""
    error_message = "When create_vpc is false, existing_vpc_id must be provided."
  }
}

variable "existing_private_subnet_ids" {
  description = "List of existing private subnet IDs for ECS tasks (required if create_vpc = false)"
  type        = list(string)
  default     = []
  
  validation {
    condition     = var.create_vpc || length(var.existing_private_subnet_ids) >= 2
    error_message = "When create_vpc is false, at least 2 private subnet IDs must be provided for high availability."
  }
}

variable "existing_public_subnet_ids" {
  description = "List of existing public subnet IDs for ALBs (required if create_vpc = false)"
  type        = list(string)
  default     = []
  
  validation {
    condition     = var.create_vpc || length(var.existing_public_subnet_ids) >= 2
    error_message = "When create_vpc is false, at least 2 public subnet IDs must be provided for high availability."
  }
}

variable "existing_nat_gateway_ips" {
  description = "List of NAT Gateway public IPs (required if create_vpc = false and Keycloak needs to access via public DNS)"
  type        = list(string)
  default     = []
}

variable "existing_private_route_table_ids" {
  description = "List of private route table IDs (required if create_vpc = false and creating S3 VPC endpoint)"
  type        = list(string)
  default     = []
}

5. Data Source Validation

Validate existing resources:

# Validate subnets exist and belong to the VPC
data "aws_subnet" "existing_private" {
  count = var.create_vpc ? 0 : length(var.existing_private_subnet_ids)
  id    = var.existing_private_subnet_ids[count.index]
}

data "aws_subnet" "existing_public" {
  count = var.create_vpc ? 0 : length(var.existing_public_subnet_ids)
  id    = var.existing_public_subnet_ids[count.index]
}

# Validation check
locals {
  validate_private_subnets = var.create_vpc ? true : alltrue([
    for s in data.aws_subnet.existing_private : s.vpc_id == var.existing_vpc_id
  ])
  
  validate_public_subnets = var.create_vpc ? true : alltrue([
    for s in data.aws_subnet.existing_public : s.vpc_id == var.existing_vpc_id
  ])
  
  validate_subnet_azs = var.create_vpc ? true : (
    length(distinct([for s in data.aws_subnet.existing_private : s.availability_zone])) >= 2 &&
    length(distinct([for s in data.aws_subnet.existing_public : s.availability_zone])) >= 2
  )
}

resource "null_resource" "validate_subnets" {
  count = local.validate_private_subnets && local.validate_public_subnets && local.validate_subnet_azs ? 0 : 1

  provisioner "local-exec" {
    command = <<-EOT
      echo "ERROR: Subnet validation failed:"
      echo "  - All subnets must belong to the specified VPC"
      echo "  - Subnets must span at least 2 availability zones for high availability"
      exit 1
    EOT
  }
}

Phase 2: ALB Support Implementation

1. Conditional Main ALB Creation

Update modules/mcp-gateway/networking.tf:

module "alb" {
  source  = "terraform-aws-modules/alb/aws"
  version = "~> 9.0"

  count = var.create_main_alb ? 1 : 0

  name                       = "${local.name_prefix}-alb"
  load_balancer_type         = "application"
  internal                   = var.alb_scheme == "internal"
  enable_deletion_protection = false

  vpc_id  = var.vpc_id
  subnets = var.alb_scheme == "internal" ? var.private_subnet_ids : var.public_subnet_ids
  
  # ... rest of configuration
}

locals {
  main_alb_arn          = var.create_main_alb ? module.alb[0].arn : var.existing_main_alb_arn
  main_alb_listener_arn = var.create_main_alb ? module.alb[0].listeners["https"].arn : var.existing_main_alb_listener_arn
  main_alb_dns_name     = var.create_main_alb ? module.alb[0].dns_name : data.aws_lb.existing_main[0].dns_name
  main_alb_sg_id        = var.create_main_alb ? module.alb[0].security_group_id : var.existing_main_alb_sg_id
}

data "aws_lb" "existing_main" {
  count = var.create_main_alb ? 0 : 1
  arn   = var.existing_main_alb_arn
}

2. Target Groups for Existing ALBs

Create target groups independently of ALB:

resource "aws_lb_target_group" "registry" {
  name_prefix          = "reg-"
  port                 = 7860
  protocol             = "HTTP"
  target_type          = "ip"
  vpc_id               = var.vpc_id
  deregistration_delay = 5

  health_check {
    enabled             = true
    healthy_threshold   = 2
    interval            = 30
    matcher             = "200"
    path                = "/health"
    port                = 7860
    protocol            = "HTTP"
    timeout             = 5
    unhealthy_threshold = 2
  }

  lifecycle {
    create_before_destroy = true
  }

  tags = local.common_tags
}

resource "aws_lb_target_group" "auth" {
  name_prefix          = "auth-"
  port                 = 8888
  protocol             = "HTTP"
  target_type          = "ip"
  vpc_id               = var.vpc_id
  deregistration_delay = 5

  health_check {
    enabled             = true
    healthy_threshold   = 2
    interval            = 30
    matcher             = "200"
    path                = "/health"
    port                = "traffic-port"
    protocol            = "HTTP"
    timeout             = 5
    unhealthy_threshold = 2
  }

  lifecycle {
    create_before_destroy = true
  }

  tags = local.common_tags
}

3. Listener Rules for Existing ALBs

When using existing ALB, create listener rules:

# Listener rule for registry (host-based routing)
resource "aws_lb_listener_rule" "registry" {
  count = var.create_main_alb ? 0 : 1
  
  listener_arn = var.existing_main_alb_listener_arn
  priority     = var.registry_listener_rule_priority

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.registry.arn
  }

  condition {
    host_header {
      values = [var.domain_name]
    }
  }

  tags = local.common_tags
}

# Listener rule for auth server (host-based routing)
resource "aws_lb_listener_rule" "auth" {
  count = var.create_main_alb ? 0 : 1
  
  listener_arn = var.existing_main_alb_listener_arn
  priority     = var.auth_listener_rule_priority

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.auth.arn
  }

  condition {
    host_header {
      values = ["auth.${var.domain_name}"]
    }
  }

  tags = local.common_tags
}

# Alternative: Path-based routing
resource "aws_lb_listener_rule" "registry_path" {
  count = var.create_main_alb || var.alb_routing_type != "path" ? 0 : 1
  
  listener_arn = var.existing_main_alb_listener_arn
  priority     = var.registry_listener_rule_priority

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.registry.arn
  }

  condition {
    path_pattern {
      values = ["/registry/*"]
    }
  }

  tags = local.common_tags
}

4. New Variables for ALB Support

variable "create_main_alb" {
  description = "Whether to create a new ALB for Registry and Auth Server"
  type        = bool
  default     = true
}

variable "existing_main_alb_arn" {
  description = "ARN of existing ALB for Registry and Auth Server (required if create_main_alb = false)"
  type        = string
  default     = ""
  
  validation {
    condition     = var.create_main_alb || var.existing_main_alb_arn != ""
    error_message = "When create_main_alb is false, existing_main_alb_arn must be provided."
  }
}

variable "existing_main_alb_listener_arn" {
  description = "ARN of existing ALB HTTPS listener (required if create_main_alb = false)"
  type        = string
  default     = ""
  
  validation {
    condition     = var.create_main_alb || var.existing_main_alb_listener_arn != ""
    error_message = "When create_main_alb is false, existing_main_alb_listener_arn must be provided."
  }
}

variable "existing_main_alb_sg_id" {
  description = "Security group ID of existing ALB (required if create_main_alb = false)"
  type        = string
  default     = ""
}

variable "alb_routing_type" {
  description = "Type of routing for existing ALB (host or path)"
  type        = string
  default     = "host"
  
  validation {
    condition     = contains(["host", "path"], var.alb_routing_type)
    error_message = "ALB routing type must be either 'host' or 'path'."
  }
}

variable "registry_listener_rule_priority" {
  description = "Priority for registry listener rule (required when using existing ALB)"
  type        = number
  default     = 100
  
  validation {
    condition     = var.registry_listener_rule_priority >= 1 && var.registry_listener_rule_priority <= 50000
    error_message = "Listener rule priority must be between 1 and 50000."
  }
}

variable "auth_listener_rule_priority" {
  description = "Priority for auth server listener rule (required when using existing ALB)"
  type        = number
  default     = 101
  
  validation {
    condition     = var.auth_listener_rule_priority >= 1 && var.auth_listener_rule_priority <= 50000
    error_message = "Listener rule priority must be between 1 and 50000."
  }
}

Example Configurations

Example 1: Existing VPC, New ALBs

# terraform.tfvars
create_vpc                   = false
existing_vpc_id              = "vpc-0123456789abcdef0"
existing_private_subnet_ids  = ["subnet-private1", "subnet-private2", "subnet-private3"]
existing_public_subnet_ids   = ["subnet-public1", "subnet-public2", "subnet-public3"]
existing_nat_gateway_ips     = ["1.2.3.4", "5.6.7.8", "9.10.11.12"]

# Create new ALBs (default)
create_main_alb    = true
create_keycloak_alb = true

Example 2: Existing VPC and ALB

# terraform.tfvars
create_vpc                   = false
existing_vpc_id              = "vpc-0123456789abcdef0"
existing_private_subnet_ids  = ["subnet-private1", "subnet-private2", "subnet-private3"]
existing_public_subnet_ids   = ["subnet-public1", "subnet-public2", "subnet-public3"]
existing_nat_gateway_ips     = ["1.2.3.4", "5.6.7.8", "9.10.11.12"]

# Use existing main ALB
create_main_alb                = false
existing_main_alb_arn          = "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/shared-alb/abc123"
existing_main_alb_listener_arn = "arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/shared-alb/abc123/def456"
existing_main_alb_sg_id        = "sg-0123456789abcdef0"
alb_routing_type               = "host"
registry_listener_rule_priority = 100
auth_listener_rule_priority     = 101

# Create new Keycloak ALB (or use existing if available)
create_keycloak_alb = true

Example 3: New VPC, Existing ALB (Less Common)

# terraform.tfvars
create_vpc = true
vpc_cidr   = "10.1.0.0/16"

# Use existing shared ALB
create_main_alb                = false
existing_main_alb_arn          = "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/shared-alb/abc123"
existing_main_alb_listener_arn = "arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/shared-alb/abc123/def456"
existing_main_alb_sg_id        = "sg-0123456789abcdef0"

Documentation Checklist

When implementing this feature, create:

1. BYO VPC Guide (docs/bring-your-own-vpc.md)

   • Prerequisites checklist
   • Step-by-step configuration
   • Validation and troubleshooting
   • Example configurations

2. BYO ALB Guide (docs/bring-your-own-alb.md)

   • ALB requirements
   • Listener rule priority management
   • Host-based vs path-based routing
   • Security group considerations

3. Update Main README

   • Add section on deployment options
   • Link to BYO guides
   • Update architecture diagrams

4. Migration Guide (docs/migration-to-byo.md)

   • How to move from created to existing resources
   • State management considerations
   • Rollback procedures

Testing Checklist

• Test with existing VPC, new ALBs
• Test with existing VPC, existing main ALB
• Test with existing VPC, existing Keycloak ALB
• Test with existing VPC and both ALBs
• Test validation failures (wrong VPC ID, missing subnets, etc.)
• Test multi-AZ validation
• Test listener rule priority conflicts
• Test host-based routing
• Test path-based routing
• Test security group connectivity
• Test DNS resolution with existing ALBs
• Test ECS service discovery
• Test RDS connectivity from ECS tasks
• Test EFS mount targets
• Test NAT gateway security group rules