Document Nebari Security Config Options

[tylergraff]

Reference Issues or PRs

Fixes #538

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

Documentation

Access-centered content checklist

Text styling

• The content is written with plain language (where relevant).
• If there are headers, they use the proper header tags (with only one level-one header: H1 or # in markdown).
• All links describe where they link to (for example, check the Nebari website).
• This content adheres to the Nebari style guides.

Non-text content

• All content is represented as text (for example, images need alt text, and videos need captions or descriptive transcripts).
• If there are emojis, there are not more than three in a row.
• Don't use flashing GIFs or videos.
• If the content were to be read as plain text, it still makes sense, and no information is missing.

Any other comments?

[netlify]

✅ Deploy Preview for nebari-docs ready!

Name	Link
🔨 Latest commit	528123a
🔍 Latest deploy log	https://app.netlify.com/sites/nebari-docs/deploys/672a4a072c8e7800085720c7
😎 Deploy Preview	https://deploy-preview-539--nebari-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[joneszc]

@tylergraff
The tested updates for featuring a nebari-config.yaml that enables options to override, exhaustively, every Nebari container image were never merged for the following reasons:

• A more concise approach for mirroring images is available using containerD config overrides/imports, which does not require specifying the mirrored name for each individual container in nebari-config.yaml
• Discussion took place regarding possible migration from Helm to kustomize and kustomization files, which could deem the terraform/helm override method of mirroring obsolete.

We took a different approach towards mirroring container images based on pointing to default mirrors for private registries (e.g. ECR, GitLab, etc.) as overrides/imports to the EKS nodes' containerD configs.
The enabling PR for this approach was PR#2668, which added the feature to run pre_bootstrap_command on nodes.

The following config options are examples of mirroring container images by means of customizing ContainerD at the k8s node:

# Set ECR as default container registry mirror
amazon_web_services:
  node_groups:
    general:
      instance: m5.2xlarge
      min_nodes: 1
      max_nodes: 1
      gpu: false
      single_subnet: false
      permissions_boundary:
      launch_template:
        pre_bootstrap_command: |
            #!/bin/bash
            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
            # Set ECR as default container registry mirror
            mkdir -p /etc/containerd/certs.d/_default
            ECR_TOKEN="$(aws ecr get-login-password --region us-east-1)"
            BASIC_AUTH="$(echo -n "AWS:$ECR_TOKEN" | base64 -w 0)"
            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
            [host."https://xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com"]
              capabilities = ["pull", "resolve"]
              [host."https://xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com".header]
                authorization = "Basic $BASIC_AUTH"
            EOT


# Set GitLab CR as default container registry mirror in hosts.toml; 
# must have override_path set if project/group names don't match upstream container
amazon_web_services:
  node_groups:
    general:
      instance: m5.2xlarge
      min_nodes: 1
      max_nodes: 1
      gpu: false
      single_subnet: false
      permissions_boundary:
      launch_template:
        pre_bootstrap_command: |
            #!/bin/bash
            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
            # Set default container registry mirror in hosts.toml; must have override_path set if project/group names don't match upstream container
            CONTAINER_REGISTRY_URL="gitlab-registry.link.net"
            CONTAINER_REGISTRY_USERNAME="project_2744_bot_xxxxxxxxxxxxxx"
            CONTAINER_REGISTRY_TOKEN="xxxxxxxxxxx"
            CONTAINER_REGISTRY_GROUP=as-nebari
            CONTAINER_REGISTRY_PROJECT=nebari-test
            mkdir -p /etc/containerd/certs.d/_default
            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
            [host."https://$CONTAINER_REGISTRY_URL/v2/$CONTAINER_REGISTRY_GROUP/$CONTAINER_REGISTRY_PROJECT"]
              override_path = true
              capabilities = ["pull", "resolve"]
            EOT
            # Set containerd registry config auth in config.d .toml import dir
            mkdir -p /etc/containerd/config.d
            cat <<EOT | sudo tee /etc/containerd/config.d/config-import.toml
            version = 2
            [plugins."io.containerd.grpc.v1.cri".registry]
              config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
              [plugins."io.containerd.grpc.v1.cri".registry.auths]
              [plugins."io.containerd.grpc.v1.cri".registry.configs]
                [plugins."io.containerd.grpc.v1.cri".registry.configs."$CONTAINER_REGISTRY_URL".auth]
                  username = "$CONTAINER_REGISTRY_USERNAME"
                  password = "$CONTAINER_REGISTRY_TOKEN"
            EOT


# Set GitLab CR as default container registry mirror in hosts.toml; 
# must have override_path set if project/group names don't match upstream container
# Also add/set GitLab Client SSL/TLS Certificate for Containerd
amazon_web_services:
  node_groups:
    general:
      instance: m5.2xlarge
      min_nodes: 1
      max_nodes: 1
      gpu: false
      single_subnet: false
      permissions_boundary:
      launch_template:
        pre_bootstrap_command: |
            #!/bin/bash
            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
            # Set default container registry mirror in hosts.toml; must have override_path set if project/group names don't match upstream container
            CONTAINER_REGISTRY_URL="gitlab-registry.link.net"
            CONTAINER_REGISTRY_USERNAME="project_2744_bot_xxxxxxxxxxxxxx"
            CONTAINER_REGISTRY_TOKEN="xxxxxxxxxxx"
            CONTAINER_REGISTRY_GROUP=as-nebari
            CONTAINER_REGISTRY_PROJECT=nebari-test
            mkdir -p /etc/containerd/certs.d/_default
            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
            [host."https://$CONTAINER_REGISTRY_URL/v2/$CONTAINER_REGISTRY_GROUP/$CONTAINER_REGISTRY_PROJECT"]
              override_path = true
              capabilities = ["pull", "resolve"]
              client = ["/etc/containerd/certs.d/$CONTAINER_REGISTRY_URL/client.pem"]
            EOT
            # Set containerd registry config auth in config.d .toml import dir
            mkdir -p /etc/containerd/config.d
            cat <<EOT | sudo tee /etc/containerd/config.d/config-import.toml
            version = 2
            [plugins."io.containerd.grpc.v1.cri".registry]
              config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
              [plugins."io.containerd.grpc.v1.cri".registry.auths]
              [plugins."io.containerd.grpc.v1.cri".registry.configs]
                [plugins."io.containerd.grpc.v1.cri".registry.configs."$CONTAINER_REGISTRY_URL".auth]
                  username = "$CONTAINER_REGISTRY_USERNAME"
                  password = "$CONTAINER_REGISTRY_TOKEN"
            EOT
            # Add client key/cert to containerd
            mkdir -p /etc/containerd/certs.d/$CONTAINER_REGISTRY_URL
            cat <<-EOT >> /etc/containerd/certs.d/$CONTAINER_REGISTRY_URL/client.pem
            -----BEGIN CERTIFICATE-----
            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
            -----END CERTIFICATE-----
            -----BEGIN PRIVATE KEY-----
            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
            -----END PRIVATE KEY-----
            EOT

[joneszc]

@tylergraff
The following feature, although tested, was never merged
Lines 9-15

amazon_web_services:
  ec2_keypair_name: [example_keypair_name] # Name, not ARN

The following feature amazon_web_services.extra_ssl_certificates was tested but not merged:
Lines 17-28

  extra_ssl_certificates: |
    -----BEGIN CERTIFICATE-----
    MIIF...<snip>...ABCD
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIF...<snip>...EF01
    -----END CERTIFICATE-----

...Instead, the same feature can be implemented since PR#2668 as follows:

# Add client certificate to CA trust on node
amazon_web_services:
  node_groups:
    general:
      instance: m5.2xlarge
      min_nodes: 1
      max_nodes: 1
      gpu: false
      single_subnet: false
      permissions_boundary:
      launch_template:
        pre_bootstrap_command: |
            #!/bin/bash
            cat <<-EOT >> /etc/pki/ca-trust/source/anchors/client.pem
            -----BEGIN CERTIFICATE-----
            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
            -----END CERTIFICATE-----
            EOT
            sudo update-ca-trust extract

Also, the Private EKS endpoint configuration feauture (Lines 30-36) was implemented in PR#2618 but needs to be configured as follows, with a string value as 1 of [public, private, public_and_private]:

amazon_web_services:
  eks_endpoint_access: private

[tylergraff]

@joneszc thank you for your review comments. I have addressed them all, please take another look at your convenience.

[joneszc]

@tylergraff
The following feature, although tested, was never merged
Lines 9-15

amazon_web_services:
  ec2_keypair_name: [example_keypair_name] # Name, not ARN

[tylergraff]

fixed!

[joneszc]

@tylergraff
I think a note should be added here to clarify that setting the load balancer schema to internal type should be set only when pointing Nebari to private subnets

[joneszc]

Looks good!