How do I set HTTP proxy for Amazon EKS containerd nodes?

[c-s-u]

• ✋ I have searched the open/closed issues and my issue is not listed.

Please describe your question here

I would like to set up an https proxy for the nodes of my fully-private EKS cluster.
How can I implement like the link below in terraform?
https://repost.aws/knowledge-center/eks-http-proxy-containerd-automation

Provide a link to the example/module related to the question

Additional context

best regards.

[askulkarni2]

Hi @chugai-so-ukiyama, thanks for issue. At the moment we do not have this pattern implemented in this project. I will add this to our backlog for now. In the meantime, I recommend taking a look at https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/user_data.md for information on how to pass user data provided in that article to nodes of a MNG. Please feel free to post any findings/issues here.

[horkko]

Hi @askulkarni2

Maybe you'll be able to light my lantern ? 🤞

I've a similar demand for a managed node group running bottelrocket AMI operation system.
I've tried to set, in the managed_node_groups map, a post_userdata value (generated with a templatefile as show below) but it looks like this is not taken into consideration. The user_data generated into the final launch_template does not integrate what I've passed topost_userdata input. It only contains these data:

[settings.kubernetes]
"cluster-name" = "eks-z1-mwaa-test-mng-calico"
"api-server" = "https://44B08416040A17E5F574A6E63F6738D5.gr7.eu-west-3.eks.amazonaws.com"
"cluster-certificate" = ...
"cluster-dns-ip" = "172.20.0.10"
[settings.kubernetes.node-labels] <==== created through k8s_labels variable
"lifecycle" = "on_demand"
"eks.amazonaws.com/nodegroup-image" = "ami-0e63a05ceec3cf941"
"eks.amazonaws.com/capacityType" = "ON_DEMAND"
"eks.amazonaws.com/nodegroup" = "managed-ondemand-1-20230908145012907000000005"

Am I missing something, or is it a normal behavior from the [aws-eks-managed-node-groups](https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/v4.32.1/modules/aws-eks-managed-node-groups) module?
Looking at the terraform code, I don't understand why this user_data is not in final launch_template ?

Sample code from module call:

module "eks_blueprints" {
  source = "github.com/aws-ia/terraform-aws-eks-blueprints?ref=v4.32.1"
...
managed_node_groups = var.managed_node_groups

}

managed_node_groups variable details:

  managed_node_groups = {

    "airflow-workers" = {
      ami_type       = "BOTTLEROCKET_x86_64"
      capacity_type  = "ON_DEMAND"
      instance_types = var.eks_mng_instance_types
      subnet_ids     = data.aws_subnets.private["z1"].ids
      max_size       = var.eks_mng_max_size
      min_size       = var.eks_mng_min_size
      desired_size   = var.eks_mng_desired_size
      node_group_name = "${var.node_group_name}-1"
      k8s_taints                  = []
      k8s_labels                  = { lifecycle = lower(var.eks_mng_capacity_type) } # var.eks_mng_k8s_labels
      enable_metadata_options     = true
      http_endpoint               = "disabled"
      http_tokens                 = "required" #tfsec:ignore:aws-autoscaling-enforce-http-token-imds
      http_put_response_hop_limit = 1
      http_protocol_ipv6          = null
      instance_metadata_tags      = "disabled"
      bootstrap_extra_args        = "--use-max-pods false --container-runtime containerd"
      kubelet_extra_args          = "--node-labels=arch=x86,WorkerType=${var.eks_mng_capacity_type} --max-pods=110"
      custom_ami_id               = data.aws_ami.mng_ami.id #"ami-0e63a05ceec3cf941"
      launch_template_os          = "bottlerocket"  <====
      create_launch_template      = true                <==== Is it set to true (https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/8a06a6e7006e4bed5630bd49c7434d76c59e0b5e/modules/aws-eks-managed-node-groups/managed-launch-templates.tf#L2)
      pre_userdata                = ""
      post_userdata               = templatefile("${path.module}/templates/eks_mng_node_postuserdata.tpl",
        {
          node_proxy    = local.node_proxy,
          node_no_proxy = local.node_no_proxy
      })
      # block_device_mappings only used when create_launch_template is set to true
      # https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/v4.32.1/modules/aws-eks-managed-node-groups/managed-launch-templates.tf#L2
      block_device_mappings = [{
        device_name           = "/dev/xvda"
        volume_type           = var.eks_mng_block_volume_type #"gp3" # The volume type. Can be standard, gp2, gp3, io1, io2, sc1 or st1 (Default: gp3).
        volume_size           = 150
        delete_on_termination = true
        encrypted             = true
        kms_key_id            = var.ebs_kms_key_id
        throughput            = 125
      }]
    }
  }

template file content as describe here : https://github.com/bottlerocket-os/bottlerocket#proxy-settings

[settings.network]
https-proxy = "${node_proxy}"
no-proxy    = "${node_no_proxy}"

By the way, same issue with block_device_mappings in manage_node_groups map.

Can you help with this?

Thanks a lot for your time.

Emmanuel

[georgejdli]

For Amazon Linux 2023 managed node groups you could use the link you provided as a starting point with some modifications based on updated documentation:

locals {
  proxy = "<proxy>:3128"

  endpoint_services = [
    "autoscaling",
    "ecr.api",
    "ecr.dkr",
    "ec2",
    "ec2messages",
    "elasticloadbalancing",
    "sts",
    "kms",
    "logs",
    "ssm",
    "ssmmessages"
  ]

  no_proxy_aws_services = [
    "s3", # Gateway service - relies on Route Tables on prefix list
    "autoscaling",
    "ecr", # FQDN isn't ecr.api but api.ecr
    "ec2",
    "ec2messages",
    "elasticloadbalancing",
    "sts",
    "kms",
    "logs",
    "ssm",
    "ssmmessages"
  ]

  # Generate no_proxy values for AWS services
  no_proxy_aws_service_fqdns = [
    for service in local.no_proxy_aws_services : ".${service}.${local.region}.amazonaws.com"
  ]

}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.36"

  eks_managed_node_group_defaults = {
    ami_type               = "AL2023_x86_64_STANDARD"
    # bootstrap_user_data not required here since we are not using a custom AMI
    enable_bootstrap_user_data = false
    cloudinit_pre_nodeadm = [{
      content_type = "text/cloud-boothook; charset=\"us-ascii\""
      content      = <<-EOT
        #!/bin/bash
        # **the above line is missing from the example https://repost.aws/knowledge-center/eks-http-proxy-containerd-automation **
        # **this is required or cloud-init won't execute the script!!!**

        #Set the proxy hostname and port
        PROXY="${local.proxy}"
        TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
        MAC=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -v -s http://169.254.169.254/latest/meta-data/mac/)
        VPC_CIDR=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -v -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-ipv4-cidr-blocks | xargs | tr ' ' ',')

        #Create the containerd and sandbox-image systemd directory
        mkdir -p /etc/systemd/system/containerd.service.d
        mkdir -p /etc/systemd/system/sandbox-image.service.d

        #[Option] Configure yum to use the proxy
        cloud-init-per instance yum_proxy_config cat << EOF >> /etc/yum.conf
        proxy=http://$PROXY
        EOF

        #Set the proxy for future processes, and use as an include file
        cloud-init-per instance proxy_config cat << EOF >> /etc/environment
        http_proxy=http://$PROXY
        https_proxy=http://$PROXY
        HTTP_PROXY=http://$PROXY
        HTTPS_PROXY=http://$PROXY
        no_proxy=$VPC_CIDR,localhost,127.0.0.1,169.254.169.254,.internal,.eks.amazonaws.com,${join(",", local.no_proxy_aws_service_fqdns)}
        NO_PROXY=$VPC_CIDR,localhost,127.0.0.1,169.254.169.254,.internal,.eks.amazonaws.com,${join(",", local.no_proxy_aws_service_fqdns)}
        EOF

        # ... other lines omitted for brevity
        cloud-init-per instance reload_daemon systemctl daemon-reload
      EOT
      }]
      # do not include the script that executes bootstrap.sh since that is handled by nodeadm for AL2023
  }
}

This example assumes you're using the fully-private-cluster blueprint where service endpoints are created for node configuration activities in nodeadm.

[bryantbiggs]

@georgejdli see awslabs/amazon-eks-ami#2128

[georgejdli]

@georgejdli see awslabs/amazon-eks-ami#2128

Thanks.
Sounds like the OP had a specific requirement to use the https_proxy for nodeadm config.

I've updated my comment to clarify that I'm setting up the https_proxy within the context of using the fully-private-cluster blueprint where EC2 service access would go through the VPC interface endpoint.

I think including this example in the fully-private-cluster blueprint as an option would be helpful to others running EKS in a enterprise setting where https_proxy is required to access a private Docker registry for company container images instead of using ECR.

[bryantbiggs]

enterprise setting where https_proxy is required to access a private Docker registry for company container images instead of using ECR.

I think ECR pull through cache rules is the appropriate answer to that scenario rather than a proxy

[georgejdli]

enterprise setting where https_proxy is required to access a private Docker registry for company container images instead of using ECR.

I think ECR pull through cache rules is the appropriate answer to that scenario rather than a proxy

Perhaps my use case is not as common as I thought and maybe my comments in this thread is sufficient to help users that may have a similar use case.

I need to clarify that the private Docker registry is not accessible through the public internet (the required https_proxy is to connected to another company AWS account via PrivateLink) nor is it a registry service that's supported by ECR as an upstream registry.

So the required containerd network flow for me would be: private EKS node-> https_proxy -> PrivateLink to separate company AWS Account -> Privately Hosted Docker Registry (public internet access not enabled)