Document Nebari Security Config Options

docs/docs/how-tos/nebari-gcp.md

@@ -66,6 +66,7 @@ management.
 
 If it's your first time creating a service account, please follow
 [these detailed instructions](https://cloud.google.com/iam/docs/creating-managing-service-accounts) to create a Google Service Account with the following roles attached:
+
 - [`roles/editor`](https://cloud.google.com/iam/docs/understanding-roles#editor)
 - [`roles/resourcemanager.projectIamAdmin`](https://cloud.google.com/iam/docs/understanding-roles#resourcemanager.projectIamAdmin)
 - [`roles/container.admin`](https://cloud.google.com/iam/docs/understanding-roles#container.admin)

docs/docs/references/container-sources.md

@@ -0,0 +1,136 @@
+## Deploying and Running Nebari from a Private Container Repository
+
+Nebari deploys and runs FOSS components as containers running in Kubernetes.
+By default, Nebari sources each container from the container's respective public repository, typically `docker.io` or `quay.io`.
+This introduces supply-chain concerns for security-focused customers.
+
+One solution to these supply-chain concerns is to deploy Nebari from private locally-mirrored containers:
+
+- Create a controlled private container repository (e.g. ECR)
+- Mirror all containers used by Nebari into this private container repository
+- Use the `pre_bootstrap_command` mechanism in `nebari-config.yaml` to specify the mirrored container repo
+
+Deploying Nebari in this fashion eliminates significant supply chain surface-area, but requires identifying all containers used by Nebari.
+
+The following configurations demonstrate how to specify a private repo denoted by the string `[PRIVATE_REPO]`.
+
+### Set ECR as default container registry mirror
+
+```
+amazon_web_services:
+  node_groups:
+    general:
+      instance: m5.2xlarge
+      launch_template:
+        pre_bootstrap_command: |
+            #!/bin/bash
+            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
+            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
+            # Set ECR as default container registry mirror
+            mkdir -p /etc/containerd/certs.d/_default
+            ECR_TOKEN="$(aws ecr get-login-password --region us-east-1)"
+            BASIC_AUTH="$(echo -n "AWS:$ECR_TOKEN" | base64 -w 0)"
+            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
+            [host."https://[PRIVATE_REPO].dkr.ecr.us-east-1.amazonaws.com"]
+              capabilities = ["pull", "resolve"]
+              [host."https://[PRIVATE_REPO].dkr.ecr.us-east-1.amazonaws.com".header]
+                authorization = "Basic $BASIC_AUTH"
+            EOT
+
+```
+
+### Set GitLab CR as default container registry mirror
+
+```
+# Set GitLab CR as default container registry mirror in hosts.toml;
+# must have override_path set if project/group names don't match upstream container
+amazon_web_services:
+  node_groups:
+    general:
+      instance: m5.2xlarge
+      launch_template:
+        pre_bootstrap_command: |
+            #!/bin/bash
+            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
+            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
+            # Set default container registry mirror in hosts.toml; must have override_path set if project/group names don't match upstream container
+            CONTAINER_REGISTRY_URL="[PRIVATE_REPO]"
+            CONTAINER_REGISTRY_USERNAME="[username]"
+            CONTAINER_REGISTRY_TOKEN="[token]"
+            CONTAINER_REGISTRY_GROUP=as-nebari
+            CONTAINER_REGISTRY_PROJECT=nebari-test
+            mkdir -p /etc/containerd/certs.d/_default
+            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
+            [host."https://$CONTAINER_REGISTRY_URL/v2/$CONTAINER_REGISTRY_GROUP/$CONTAINER_REGISTRY_PROJECT"]
+              override_path = true
+              capabilities = ["pull", "resolve"]
+            EOT
+
+            # Set containerd registry config auth in config.d .toml import dir
+            mkdir -p /etc/containerd/config.d
+            cat <<EOT | sudo tee /etc/containerd/config.d/config-import.toml
+            version = 2
+            [plugins."io.containerd.grpc.v1.cri".registry]
+              config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
+              [plugins."io.containerd.grpc.v1.cri".registry.auths]
+              [plugins."io.containerd.grpc.v1.cri".registry.configs]
+                [plugins."io.containerd.grpc.v1.cri".registry.configs."$CONTAINER_REGISTRY_URL".auth]
+                  username = "$CONTAINER_REGISTRY_USERNAME"
+                  password = "$CONTAINER_REGISTRY_TOKEN"
+            EOT
+```
+
+### Set GitLab CR as default container registry mirror, with custom Client SSL/TLS Certs
+
+```
+# must have override_path set if project/group names don't match upstream container
+# Also add/set GitLab Client SSL/TLS Certificate for Containerd
+amazon_web_services:
+  node_groups:
+    general:
+      instance: m5.2xlarge
+      launch_template:
+        pre_bootstrap_command: |
+            #!/bin/bash
+            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
+            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
+            # Set default container registry mirror in hosts.toml; must have override_path set if project/group names don't match upstream container
+            CONTAINER_REGISTRY_URL="[PRIVATE_REPO]"
+            CONTAINER_REGISTRY_USERNAME="[username]"
+            CONTAINER_REGISTRY_TOKEN="[token]"
+            CONTAINER_REGISTRY_GROUP=as-nebari
+            CONTAINER_REGISTRY_PROJECT=nebari-test
+            mkdir -p /etc/containerd/certs.d/_default
+            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
+            [host."https://$CONTAINER_REGISTRY_URL/v2/$CONTAINER_REGISTRY_GROUP/$CONTAINER_REGISTRY_PROJECT"]
+              override_path = true
+              capabilities = ["pull", "resolve"]
+              client = ["/etc/containerd/certs.d/$CONTAINER_REGISTRY_URL/client.pem"]
+            EOT
+
+            # Set containerd registry config auth in config.d .toml import dir
+            mkdir -p /etc/containerd/config.d
+            cat <<EOT | sudo tee /etc/containerd/config.d/config-import.toml
+            version = 2
+            [plugins."io.containerd.grpc.v1.cri".registry]
+              config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
+              [plugins."io.containerd.grpc.v1.cri".registry.auths]
+              [plugins."io.containerd.grpc.v1.cri".registry.configs]
+                [plugins."io.containerd.grpc.v1.cri".registry.configs."$CONTAINER_REGISTRY_URL".auth]
+                  username = "$CONTAINER_REGISTRY_USERNAME"
+                  password = "$CONTAINER_REGISTRY_TOKEN"
+            EOT
+
+            # Add client key/cert to containerd
+            mkdir -p /etc/containerd/certs.d/$CONTAINER_REGISTRY_URL
+            cat <<-EOT >> /etc/containerd/certs.d/$CONTAINER_REGISTRY_URL/client.pem
+            -----BEGIN CERTIFICATE-----
+            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
+            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
+            -----END CERTIFICATE-----
+            -----BEGIN PRIVATE KEY-----
+            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
+            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
+            -----END PRIVATE KEY-----
+            EOT
+```

docs/docs/references/enhanced-security.md

@@ -0,0 +1,69 @@
+## Nebari Security Considerations
+
+The security of _AWS Nebari_ deployments can be enhanced through the following deployment configuration options in `nebari-config.yaml`:
+
+- **Explicit definition of container sources**  
+  This option allows for the use of locally mirrored, security-hardened, or otherwise customized container images in place of the containers used by default.
+  See: [container-sources](container-sources.md)
+
+- **Definition of an ssh key that can access EKS hosts**  
+  EKS hosts by default cannot be accessed via ssh. This configuration item allows ssh access into EKS hosts, which can be useful for troubleshooting or external monitoring and auditing purposes.
+
+```
+amazon_web_services:
+  ec2_keypair_name: [example_keypair_name] # Name, not ARN
+```
+
+- **Installation of custom SSL certificate(s) into EKS hosts**  
+  Install private certificates used by (e.g.) in-line content inspection engines which re-encrypt traffic.
+
+```
+# Add client certificate to CA trust on node
+amazon_web_services:
+  node_groups:
+    general:
+      instance: m5.2xlarge
+      launch_template:
+        pre_bootstrap_command: |
+            #!/bin/bash
+            cat <<-EOT >> /etc/pki/ca-trust/source/anchors/client.pem
+            -----BEGIN CERTIFICATE-----
+            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
+            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
+            -----END CERTIFICATE-----
+            EOT
+            sudo update-ca-trust extract
+```
+
+- **Private EKS endpoint configuration**  
+  Mirrors the corresponding AWS console option, which routes all EKS traffic within the VPC.
+
+```
+  amazon_web_services:
+    eks_endpoint_access: private # valid values: [public, private, public_and_private]
+```
+
+- **Deploy into existing subnets**  
+  Instructs Nebari to be deployed into existing subnets, rather than creating its own new subnets.
+
+```
+existing_subnet_ids:
+    - subnet-0123456789abcdef
+    - subnet-abcdef0123456789
+  existing_security_group_id: sg-0123456789abcdef
+ingress:
+  terraform_overrides:
+    load-balancer-annotations:
+      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
+      # Ensure the subnet IDs are also set below
+      service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-0123456789abcdef,subnet-abcdef0123456789"
+```
+
+- **Use existing SSL certificate**  
+  Instructs Nebari to use the SSL certificate specified by `[k8s-custom-secret-name]`
+
+```
+certificate:
+  type: existing
+  secret_name: [k8s-custom-secret-name]
+```

docs/docs/references/index.mdx

@@ -11,6 +11,8 @@ import {useCurrentSidebarCategory} from '@docusaurus/theme-common';
   />
 </div>
 
-Nitty-gritty technical descriptions of how Nebari works.
+Technical descriptions of how Nebari works.
 
+- [Enhanced Security](enhanced-security.md) - Nebari security configuration guide
+- [Local Container Repo](container-sources.md) - Deploying Nebari from a Local Container Repo 
 <DocCardList items={useCurrentSidebarCategory().items}/>

docs/nebari-slurm/configuration.md

@@ -186,8 +186,7 @@ _Note_: All slurm related configuration needs to be passed down as a string.
 ### Services
 
 Additional services can be added to the `jupyterhub_services`
-variable. Currently this is only `<service-name>:
-<service-apikey>`. You must keep the `dask_gateway` section.
+variable. Currently this is only `<service-name>: <service-apikey>`. You must keep the `dask_gateway` section.
 
 ```yaml
 jupyterhub_services: