Skip to content

0009-container-orchestration.md

Latest commit

 

History

History
74 lines (63 loc) · 3.64 KB

0009-container-orchestration.md

File metadata and controls

74 lines (63 loc) · 3.64 KB
Raw

ADR-0009: Kubernetes Container Orchestration

Status

Accepted

Context

We need a container orchestration platform to deploy, manage, and scale our microservices in production. The platform must provide high availability, automatic scaling, and integrated monitoring.

Decision

We will use Kubernetes for container orchestration with the following configuration:

  • Platform: Amazon EKS (Elastic Kubernetes Service)
  • Deployment: Helm charts for application deployment
  • Scaling: KEDA (Kubernetes Event-driven Autoscaling) for SQS-based scaling
  • Networking: AWS Load Balancer Controller for ingress
  • Storage: EBS volumes for persistent storage
  • Monitoring: Prometheus and Grafana for cluster monitoring
  • Security: IAM roles for service accounts (IRSA)
  • Throughput: Target 50-200 TPS per pod for optimal performance

Consequences

Positive

  • Scalability: Event-driven scaling based on SQS queue depth
  • High Availability: Multi-AZ deployment with pod distribution
  • Self-healing: Automatic restart of failed containers
  • Rolling Updates: Zero-downtime deployments
  • Resource Management: Efficient resource allocation and limits
  • Security: Pod security policies and network policies
  • Monitoring: Integrated monitoring and logging
  • Queue-based Scaling: KEDA scales pods based on actual message queue depth
  • Performance Optimization: 50-200 TPS per pod provides optimal resource utilization

Negative

  • Complexity: Steep learning curve for Kubernetes
  • Operational Overhead: Requires Kubernetes expertise
  • Cost: EKS cluster costs and additional AWS resources
  • Debugging: More complex debugging in distributed environment
  • Configuration: Complex YAML configurations

Neutral

  • Vendor Lock-in: EKS ties to AWS but Kubernetes is portable
  • Resource Limits: Need to carefully configure resource requests/limits
  • Networking: Complex networking model with services and ingress

Alternatives Considered

  • Docker Swarm: Rejected due to limited features and community support
  • Amazon ECS: Rejected due to less flexibility and vendor lock-in
  • Self-managed Kubernetes: Rejected due to operational overhead
  • Serverless (Lambda): Rejected due to cold start issues and limitations
  • Standard HPA: Rejected in favor of KEDA for queue-based scaling

Implementation Notes

  • Helm charts for templated deployments
  • KEDA ScaledObject for SQS-based autoscaling
  • Network policies for security isolation
  • Resource requests and limits for all containers
  • Health checks and readiness probes
  • Rolling update strategy for zero-downtime deployments
  • Target 50-200 TPS per pod for optimal performance and resource utilization
  • KEDA metrics adapter for Prometheus integration

Suggestions for Future Enhancements

  • KEDA Integration: Consider implementing KEDA for SQS queue-based scaling instead of standard HPA
  • Throughput Optimization: Target 50-200 TPS per pod for optimal performance and resource utilization
  • Karpenter Optimization: Use Karpenter for automatic instance management with M7i.* instance types for best cost-performance per-thread
  • Cost Efficiency: Karpenter will automatically manage instance sizes and spot/on-demand mix based on cost efficiency and scaling needs

References