Document Nebari Security Config Options

docs/docs/how-tos/nebari-gcp.md

@@ -66,6 +66,7 @@ management.
 
 If it's your first time creating a service account, please follow
 [these detailed instructions](https://cloud.google.com/iam/docs/creating-managing-service-accounts) to create a Google Service Account with the following roles attached:
+
 - [`roles/editor`](https://cloud.google.com/iam/docs/understanding-roles#editor)
 - [`roles/resourcemanager.projectIamAdmin`](https://cloud.google.com/iam/docs/understanding-roles#resourcemanager.projectIamAdmin)
 - [`roles/container.admin`](https://cloud.google.com/iam/docs/understanding-roles#container.admin)

docs/docs/references/container-sources.md

@@ -0,0 +1,138 @@
+## Deploying and Running Nebari from a Private Container Repository
+
+Nebari deploys and runs FOSS components as containers running in Kubernetes.
+By default, Nebari sources each container from the container's respective public repository, typically `docker.io` or `quay.io`.
+This introduces supply-chain concerns for security-focused customers.
+
+One solution to these supply-chain concerns is to deploy Nebari from private locally-mirrored containers:
+
+- Create a controlled private container repository (e.g. ECR)
+- Mirror all containers used by Nebari into this private container repository
+- Use the `pre_bootstrap_command` mechanism in `nebari-config.yaml` to specify the mirrored container repo
+
+Deploying Nebari in this fashion eliminates significant supply chain surface-area, but requires identifying all containers used by Nebari.
+
+The following configurations demonstrate how to specify a private repo denoted by the string `[PRIVATE_REPO]`.
+
+**Note:** Authorization tokens are used in the examples below. It is important for administrators to understand the expiration policy of these tokens, because the Nebari k8s cluster may in some cases need to **use these tokens to pull container images at any time during run-time operation**.
+
+### Set ECR as default container registry mirror
+
+```
+amazon_web_services:
+  node_groups:
+    general:
+      instance: m5.2xlarge
+      launch_template:
+        pre_bootstrap_command: |
+            #!/bin/bash
+            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
+            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
+            # Set ECR as default container registry mirror
+            mkdir -p /etc/containerd/certs.d/_default
+            ECR_TOKEN="$(aws ecr get-login-password --region us-east-1)"
+            BASIC_AUTH="$(echo -n "AWS:$ECR_TOKEN" | base64 -w 0)"
+            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
+            [host."https://[PRIVATE_REPO].dkr.ecr.us-east-1.amazonaws.com"]
+              capabilities = ["pull", "resolve"]
+              [host."https://[PRIVATE_REPO].dkr.ecr.us-east-1.amazonaws.com".header]
+                authorization = "Basic $BASIC_AUTH"
+            EOT
+
+```
+
+### Set GitLab CR as default container registry mirror
+
+```
+# Set GitLab CR as default container registry mirror in hosts.toml;
+# must have override_path set if project/group names don't match upstream container
+amazon_web_services:
+  node_groups:
+    general:
+      instance: m5.2xlarge
+      launch_template:
+        pre_bootstrap_command: |
+            #!/bin/bash
+            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
+            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
+            # Set default container registry mirror in hosts.toml; must have override_path set if project/group names don't match upstream container
+            CONTAINER_REGISTRY_URL="[PRIVATE_REPO]"
+            CONTAINER_REGISTRY_USERNAME="[username]"
+            CONTAINER_REGISTRY_TOKEN="[token]"
+            CONTAINER_REGISTRY_GROUP=as-nebari
+            CONTAINER_REGISTRY_PROJECT=nebari-test
+            mkdir -p /etc/containerd/certs.d/_default
+            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
+            [host."https://$CONTAINER_REGISTRY_URL/v2/$CONTAINER_REGISTRY_GROUP/$CONTAINER_REGISTRY_PROJECT"]
+              override_path = true
+              capabilities = ["pull", "resolve"]
+            EOT
+
+            # Set containerd registry config auth in config.d .toml import dir
+            mkdir -p /etc/containerd/config.d
+            cat <<EOT | sudo tee /etc/containerd/config.d/config-import.toml
+            version = 2
+            [plugins."io.containerd.grpc.v1.cri".registry]
+              config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
+              [plugins."io.containerd.grpc.v1.cri".registry.auths]
+              [plugins."io.containerd.grpc.v1.cri".registry.configs]
+                [plugins."io.containerd.grpc.v1.cri".registry.configs."$CONTAINER_REGISTRY_URL".auth]
+                  username = "$CONTAINER_REGISTRY_USERNAME"
+                  password = "$CONTAINER_REGISTRY_TOKEN"
+            EOT
+```
+
+### Set GitLab CR as default container registry mirror, with custom Client SSL/TLS Certs
+
+```
+# must have override_path set if project/group names don't match upstream container
+# Also add/set GitLab Client SSL/TLS Certificate for Containerd
+amazon_web_services:
+  node_groups:
+    general:
+      instance: m5.2xlarge
+      launch_template:
+        pre_bootstrap_command: |
+            #!/bin/bash
+            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
+            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
+            # Set default container registry mirror in hosts.toml; must have override_path set if project/group names don't match upstream container
+            CONTAINER_REGISTRY_URL="[PRIVATE_REPO]"
+            CONTAINER_REGISTRY_USERNAME="[username]"
+            CONTAINER_REGISTRY_TOKEN="[token]"
+            CONTAINER_REGISTRY_GROUP=as-nebari
+            CONTAINER_REGISTRY_PROJECT=nebari-test
+            mkdir -p /etc/containerd/certs.d/_default
+            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
+            [host."https://$CONTAINER_REGISTRY_URL/v2/$CONTAINER_REGISTRY_GROUP/$CONTAINER_REGISTRY_PROJECT"]
+              override_path = true
+              capabilities = ["pull", "resolve"]
+              client = ["/etc/containerd/certs.d/$CONTAINER_REGISTRY_URL/client.pem"]
+            EOT
+
+            # Set containerd registry config auth in config.d .toml import dir
+            mkdir -p /etc/containerd/config.d
+            cat <<EOT | sudo tee /etc/containerd/config.d/config-import.toml
+            version = 2
+            [plugins."io.containerd.grpc.v1.cri".registry]
+              config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
+              [plugins."io.containerd.grpc.v1.cri".registry.auths]
+              [plugins."io.containerd.grpc.v1.cri".registry.configs]
+                [plugins."io.containerd.grpc.v1.cri".registry.configs."$CONTAINER_REGISTRY_URL".auth]
+                  username = "$CONTAINER_REGISTRY_USERNAME"
+                  password = "$CONTAINER_REGISTRY_TOKEN"
+            EOT
+
+            # Add client key/cert to containerd
+            mkdir -p /etc/containerd/certs.d/$CONTAINER_REGISTRY_URL
+            cat <<-EOT >> /etc/containerd/certs.d/$CONTAINER_REGISTRY_URL/client.pem
+            -----BEGIN CERTIFICATE-----
+            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
+            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
+            -----END CERTIFICATE-----
+            -----BEGIN PRIVATE KEY-----
+            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
+            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
+            -----END PRIVATE KEY-----
+            EOT
+```

docs/docs/references/enhanced-security.md

@@ -0,0 +1,62 @@
+## Nebari Security Considerations
+
+The security of _AWS Nebari_ deployments can be enhanced through the following deployment configuration options in `nebari-config.yaml`:
+
+- **Explicit definition of container sources**  
+  This option allows for the use of locally mirrored, security-hardened, or otherwise customized container images in place of the containers used by default.
+  See: [container-sources](container-sources.md)
+
+- **Installation of custom SSL certificate(s) into EKS hosts**  
+  Install private certificates used by (e.g.) in-line content inspection engines which re-encrypt traffic.
+
+```
+# Add client certificate to CA trust on node
+amazon_web_services:
+  node_groups:
+    general:
+      instance: m5.2xlarge
+      launch_template:
+        pre_bootstrap_command: |
+            #!/bin/bash
+            cat <<-EOT >> /etc/pki/ca-trust/source/anchors/client.pem
+            -----BEGIN CERTIFICATE-----
+            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
+            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
+            -----END CERTIFICATE-----
+            EOT
+            sudo update-ca-trust extract
+```
+
+- **Private EKS endpoint configuration**  
+  Mirrors the corresponding AWS console option, which routes all EKS traffic within the VPC.
+
+```
+  amazon_web_services:
+    eks_endpoint_access: private # valid values: [public, private, public_and_private]
+```
+
+- **Deploy into existing subnets**  
+  Instructs Nebari to be deployed into existing subnets, rather than creating its own new subnets.
+  An advantage of deploying to existing subnets is the ability to use private subnets. Note that the **ingress load-balancer-annotation** must be set appropriately based on the type (private or public) of subnet.
+
+```
+existing_subnet_ids:
+    - subnet-0123456789abcdef
+    - subnet-abcdef0123456789
+  existing_security_group_id: sg-0123456789abcdef
+ingress:
+  terraform_overrides:
+    load-balancer-annotations:
+      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
+      # Ensure the subnet IDs are also set below
+      service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-0123456789abcdef,subnet-abcdef0123456789"
+```
+
+- **Use existing SSL certificate**  
+  Instructs Nebari to use the SSL certificate specified by `[k8s-custom-secret-name]`
+
+```
+certificate:
+  type: existing
+  secret_name: [k8s-custom-secret-name]
+```

docs/docs/references/index.mdx

@@ -11,6 +11,8 @@ import {useCurrentSidebarCategory} from '@docusaurus/theme-common';
   />
 </div>
 
-Nitty-gritty technical descriptions of how Nebari works.
+Technical descriptions of how Nebari works.
 
+- [Enhanced Security](enhanced-security.md) - Nebari security configuration guide
+- [Local Container Repo](container-sources.md) - Deploying Nebari from a Local Container Repo 
 <DocCardList items={useCurrentSidebarCategory().items}/>

docs/nebari-slurm/configuration.md

@@ -186,8 +186,7 @@ _Note_: All slurm related configuration needs to be passed down as a string.
 ### Services
 
 Additional services can be added to the `jupyterhub_services`
-variable. Currently this is only `<service-name>:
-<service-apikey>`. You must keep the `dask_gateway` section.
+variable. Currently this is only `<service-name>: <service-apikey>`. You must keep the `dask_gateway` section.
 
 ```yaml
 jupyterhub_services: