OWASP Nest Kubernetes Options

[arkid15r]

Technical Challenges & Implementation Options

Hey everyone,

We're moving forward with adopting Amazon EKS for OWASP Nest. Our main focus now is tackling the technical challenges and deciding on implementation details to ensure a smooth transition.

Key Areas for Discussion:

• Deployment Strategy: How should we structure our EKS deployment (namespaces, networking, scaling, etc.)?
• CI/CD Integration: What’s the best approach to integrate EKS with our existing GitHub Actions workflows?
• Local Development Compatibility: How do we ensure developers can run a local environment that mirrors EKS as closely as possible (e.g., Kind, Minikube, Dev Containers)?
• Observability & Monitoring: What tools should we use for logging, metrics, and monitoring (e.g., Prometheus, Grafana, OpenTelemetry, AWS CloudWatch)?
• Secrets & Configuration Management: How should we handle environment variables, secrets, and config files securely (e.g., AWS Secrets Manager, Kubernetes Secrets, HashiCorp Vault)?
• Service Communication & Networking: What ingress controller should we use? How should we handle internal service communication (e.g., AWS ALB Ingress Controller, NGINX Ingress, service mesh options like Istio or Linkerd)?
• Fargate vs. EC2: Should we use EKS on Fargate for simplified management and cost efficiency, or EC2-based worker nodes for more control and flexibility? What are the trade-offs in terms of scaling, performance, and networking?

If you have experience with EKS, Kubernetes, or similar setups, we’d love your input! What challenges do you foresee, and what approaches would you recommend?

[Naveen-Pal]

1. Deployment Strategy

• We can use separate namespaces for distinct services or environments (e.g., dev, staging, production) to organize resources.
• Implement network segmentation (VPC subnets for public/private resources) and ensure autoscaling (Horizontal Pod Autoscaling) is properly configured at both the node group and pod levels.

2. CI/CD Integration

• We need to containerize each service we use and push it to Docker Hub.
• Deploy to EKS by integrating Helm into the workflow, referencing Kubernetes manifests or Helm charts stored in the repository.
• We can create our own Helm charts containing all the resource definitions necessary to run an application, tool, or service inside a Kubernetes cluster.

To configure Helm in the GitHub Actions workflow:

# Pseudo workflow
# Other workflow steps including containerization

- name: Configure kubectl
  uses: aws-actions/eksctl-action@v1
  with:
    cluster-name: ${{ env.EKS_CLUSTER_NAME }}
    region: ${{ env.AWS_REGION }}

- name: Deploy with Helm
  run: |
    helm upgrade --install my-service ./helm \
      --set image.repository=${{ env.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/${{ env.IMAGE_NAME }} \
      --set image.tag=latest

A good documentation on Helm is available.

3. Local Development Compatibility

• If we want development to closely mirror deployment, we can use Minikube (or Kind) to set up a local cluster. However, this may require developers to redeploy Docker images to a registry (e.g., Docker Hub).
• We can maintain consistency by storing Helm charts in the repository and referencing them for both local and cloud deployments.

4. Observability & Monitoring

• Not entirely sure, but Prometheus and Grafana seem like good choices for scalability and better visualization.

5. Service Communication & Networking

• Not sure, but NGINX Ingress appears to be a popular choice.

6. Fargate vs. EC2

• EKS on Fargate seems like a good option in terms of serverless architecture, cost and simplicity.

[abhi4578]

Its better to entirely separate Staging and Production environment with different K8s cluster.
Namespaces does not offer strong isolation . Staging could be minimal size K8s cluster.

[abhi4578]

HI Abhi here.

Previously worked as devops engineer at IUDX/CDPG . Worked extensively on Logging (log4j2, Loki, promtail), monitoring(micrometrics, prometheus, grafana), Swarm based deployment and K8s based deployment using self hosted rancher in AWS, Azure. My previous work can be found at https://github.com/datakaveri/iudx-deployment.

I went through nest project and was able to setup locally through docker compose .
With my understanding of the nest project , I would like to add following things in discussion.

1. Deployment Strategy :
   1.1 Frontend - as I understand is static files . Best way to serve static content in AWS is through CDN ( using Cloudfront + S3Bucket)
   For this can have github workflow to build and push static files to S3 bucket. CDN gives good performance , low network latency!
   I did see frontend to use configurable backend urls in env file. So I think they can be decoupled and be on two different domains. Please correct me if am wrong.

   1.2 Have backend, database component in different namespaces. Namespaces gives some level of isolation,
   w.r.t secrets, configmaps, setting of network policies. Having them in separate namespaces makes it easy to manage and secure .
   - Backend is stateless --> K8s deployment is good option, maybe with always two replicas for HA and HPA for autoscaling based on cpu/memory
   - Postgres - stateful --> HA deployment can be done using streaming replication, cluster mode (primary , standbys + pgpool/pgbouncer based connection pooling) . This would give some good read performance. There are two good operators to use - cloud-native-pg and Zolando postgres operator

   1.3 K8s native cron job could be used to sync data .
   1.4 Helm charts based deployment would help easier managenet of components as pointed by @idityaGE

[abhi4578]

3. Local Development Compatibility: I think Kind clusters could be a good fit, as its lightweight, easy to setup multi-node clusters, supports dynamic pvc . It would be good to do a bit more research to finalize

[idityaGE]

I agree with using Helm charts + ArgoCD for backend deployment, as well as CloudFront + S3 for the frontend.

However, to optimize the deployment process, we need CI/CD pipelines that can detect changes in the respective parts of the codebase [Docs]:

• If there are updates in the frontend, the pipeline should only build and push static files to S3.
• Similarly, if there are changes in the backend, it should trigger a new image build, push it to GHCR, and update the Kubernetes deployment via Helm and ArgoCD.

Using Helm and ArgoCD will ensures that CI (builds images) and CD (deploys updates) remain separate. It also provides version control for Kubernetes configurations and allows ArgoCD to revert to a previous version(rollbacks) if needed.

[abhi4578]

4. Observability & Monitoring : For observability - logs, metrics and traces :
   • Instrumentation : Instrumenting django app to emit logs (properly formatted) , metrics ( can use opentelemetry-python-sdk, django_prometheus, traces (otel-django-instrumentation).
     K8s cluster monitoring could be done using prometheus operator or EKS container insights
   • Storage : Possible stack Metrics in Prometheus, Logs in Loki , Tempo for Traces
   • Visualisation: Visualisation using Grafana Dashboards
   • Alerts : Alerting through Grafana or alertmanager

[abhi4578]

5. Secrets & Configuration Management : New EKS clusters (k8s 1.28+) provide default all data envelope encryption using AWS KMS - https://docs.aws.amazon.com/eks/latest/userguide/envelope-encryption.html

[abhi4578]

6. Service Communication & Networking :
   • Service Mesh - It has benefits of out of box mtls, observability, traffic management but it will add operational complexity, has performance overhead. With very less services nest has, I dont see much advantage in using service mesh.
     Could use K8s native services like cluster service for internal communication and discovery.
   • NLB (network Load balancer) + Ingress controllers - This combination helps cater multiple service exposure if needed in future in scalable way. NLB service in front of ingress , becomes entry point for all L4 traffic and ingress controller could be to route request to appropriate endpoints. For ALB, each service would need it, making it costly .
   • K8s nginx ingress controller could be good choice,
     Nginx gateway fabric also good be explored as Gateway API is superior than ingress API.

[abhi4578]

I would like to highlight some technical challenges of K8s from my experience

• Maintainence of K8s cluster is not at all trivial. It requires great skill, great skill to debug! , and also just finding, updating with right tooling around K8s is not trivial.
• Complexity of K8s , issues debugging non-trivial See k8s.af
• K8s can be costly!! - Cost estimation and minimisation needs to be though through, some tools to monitor and optimise for costs - opencost, cloud custodian
• Stateful apps scalability is not trivial like databases

Also its better to have good estimation of following things for appropriate scaling

• how many requests per second expected at high, mean load for the backend?
• Expected database write and read operation at high and mean load?
  Please provide more details on the above and also what SLA /reliability factor this project is looking at?

Finally , best to evaluate if there is actual "need" to use K8s? Can it be achieved through standalone containers/Swarm with vertical scaling of VMs for simplicity and low cost over complexity? Is there good amount of benefits to use K8s?

[abhi4578]

I would like to add following good articles which talk about why not K8s and Why K8s?
Ref:

1. https://devopscube.com/why-companies-are-leaving-kubernetes/
2. https://medium.com/@rphilogene/i-need-kubernetes-and-you-probably-need-it-too-9e5937a4f412

As a summary, a created a table from the two articles :

Why K8s?	Why not K8s?
Portability	Cloud overcosts due to over-provisioning
Very Good Ecosystem and Tooling	A ton of tools might lead to more complexity. Need to be simple which meets the needs.
Interoperability	Operational Efficiency and complexity in Maintainance
Very Good for stateless and dynamic applications	Not so good for stateful applications - not a native way and needs usually complex engineering

[abhi4578]

Also found this similar discussion , K8s vs Lambda - #516 . I do agree with the points w.r.t AWS lambda - vendor lock in, cold starts, not fine grain control . In compared to that K8s is superior for flexibility, control, scaling and better for long term.

But need to keep in mind and pursue to minimize

1. cost
2. Operational and overall toooling complexity

[Naveen-Pal]

@abhi4578

As OWASP Nest is growing rapidly and will soon include additional functionalities like a contribution hub and an AI-powered NestBot. For such a wide application there is need for a better DevOps system. Adopting Kubernetes may not feel much useful now but it will be critical in long term for scalability, security, resource visualization/management.

Also leading tech are moving towards Kubernetes. Even my institute pushing for migrating it's website to Kubernetes.

[abhi4578]

I agree. Makes sense. Just need to keep it as simple as possible and yet harness flexibility and scalability that K8s gives. We need to be mindful and avoid going after which does not really benefit the project.

[abhi4578]

Found this tool - Tilt . It would help to bootstrap the local development flow easily with one command. Other interesting features that might help :

• Live update the applications in local K8s environment for easy testing!
• UI to see logs, status of K8s apps
• Have custom flows like purge db, load db etc.

Also this tool - mirrord -- allowing easy to test against remote K8s environment from locally!

[yashgoyal0110]

Hey community i was re looking this issue, following is the tool and approach i would love to proceed with:

Namespaces & scaling: Separate namespaces per env, with Cluster Autoscaler for flexibility.
CI/CD: Use GitHub OIDC → AWS IAM for secure deploys; Helm charts for manifests.
Local dev: Kind/Dev Containers for close parity with EKS.
Observability: Prometheus + Grafana for metrics, OpenTelemetry for tracing, CloudWatch for infra logs.
Secrets: AWS Secrets Manager + External Secrets Operator.
Networking: ALB Ingress for external traffic; consider Linkerd for internal observability + mTLS.
Compute: Start with EC2 nodes for control, add Fargate selectively for specific workloads.

@arkid15r what are your thoughts? should we start implementing with small issues and steps?