From b1a0f98d61548f74e19da1c8221aabfeb55f0490 Mon Sep 17 00:00:00 2001 From: Sarthak Agrawal Date: Mon, 14 Jul 2025 09:01:34 -0700 Subject: [PATCH 1/4] Add NGF permissions document --- content/ngf/reference/permissions.md | 148 +++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 content/ngf/reference/permissions.md diff --git a/content/ngf/reference/permissions.md b/content/ngf/reference/permissions.md new file mode 100644 index 000000000..9be6c6b49 --- /dev/null +++ b/content/ngf/reference/permissions.md @@ -0,0 +1,148 @@ +--- +title: Permissions +description: NGINX Gateway Fabric permissions required by components. +weight: 300 +toc: true +type: reference +product: NGF +--- + +NGINX Gateway Fabric uses a split-plane architecture with three components that require different permissions: + +- **Control Plane**: Manages Kubernetes APIs and data plane deployments. Needs broad API access but handles no user traffic. +- **Data Plane**: Processes user traffic. Requires minimal permissions since configuration comes from control plane via secure gRPC. +- **Certificate Generator**: One-time job that creates TLS certificates for inter-plane communication. + +## Common Security Context + +All components share these security settings: + +- **User ID**: 101 (non-root) +- **Group ID**: 1001 +- **Capabilities**: All dropped (`drop: ALL`) +- **Root Filesystem**: Read-only +- **Seccomp**: Runtime default profile + +## Control Plane + +Runs as a single container in the `nginx-gateway` deployment. + +### Additional Security Settings + +- **Privilege Escalation**: Disabled (may need enabling for NGINX reload in some environments) + +### Volumes + +| Volume | Type | Purpose | +|--------|------|---------| +| `nginx-agent-tls` | Secret | TLS certificates for control plane communication | + +### RBAC Permissions + +The control plane requires these Kubernetes API permissions: + +- **Secrets, ConfigMaps, Services**: Create, update, delete, list, get, watch +- **Deployments, DaemonSets**: Create, update, delete, list, get, watch +- **ServiceAccounts**: Create, update, delete, list, get, watch +- **Namespaces, Pods**: Get, list, watch +- **Events**: Create, patch +- **EndpointSlices**: List, watch +- **Gateway API resources**: List, watch (read-only) + update status subresources only +- **NGF Custom resources**: Get, list, watch (read-only) + update status subresources only +- **Leases**: Create, get, update (for leader election) +- **CustomResourceDefinitions**: List, watch +- **TokenReviews**: Create (for authentication) + +## Data Plane + +NGINX containers managed by the control plane. No RBAC permissions needed since configuration comes via secure gRPC. + +### Additional Security Settings + +- **Privilege Escalation**: Disabled +- **Sysctl**: `net.ipv4.ip_unprivileged_port_start=0` (enables binding to ports < 1024) + +### Volumes + +#### Core NGINX Operations +| Volume | Type | Purpose | +|--------|------|---------| +| `nginx-conf` | EmptyDir | Main NGINX configuration | +| `nginx-stream-conf` | EmptyDir | Stream configuration | +| `nginx-secrets` | EmptyDir | TLS secrets for NGINX | +| `nginx-run` | EmptyDir | Runtime files (PID, sockets) | +| `nginx-cache` | EmptyDir | Cache directory | + +#### Configuration Includes +| Volume | Type | Purpose | +|--------|------|---------| +| `nginx-main-includes` | EmptyDir | Main context includes | +| `nginx-includes` | EmptyDir | HTTP context includes | + +#### NGINX Agent +| Volume | Type | Purpose | +|--------|------|---------| +| `nginx-agent` | EmptyDir | NGINX Agent configuration | +| `nginx-agent-tls` | Secret | TLS certificates for control plane communication | +| `nginx-agent-log` | EmptyDir | NGINX Agent logs | +| `nginx-agent-lib` | EmptyDir | NGINX Agent runtime data | + +#### Authentication +| Volume | Type | Purpose | +|--------|------|---------| +| `token` | Projected | Service account token | + +### Volume Permissions + +- **EmptyDir**: Read-write (required for NGINX operation) +- **Secret/ConfigMap/Projected**: Read-only + +## Certificate Generator + +Kubernetes Job that creates initial TLS certificates. + +### RBAC Permissions + +- **Secrets**: Create, update, get (NGINX Gateway Fabric namespace only) + +## OpenShift Compatibility + +NGINX Gateway Fabric includes Security Context Constraints (SCCs) for OpenShift: + +### Control Plane SCC + +- **Privilege Escalation**: Disabled +- **Host Access**: Disabled (network, IPC, PID, ports) +- **User ID Range**: 101-101 (fixed) +- **Group ID Range**: 1001-1001 (fixed) +- **Volumes**: Secret only + +### Data Plane SCC + +Same restrictions as control plane, plus additional volume types: + +- **Additional Volumes**: EmptyDir, ConfigMap, Projected + +## Linux Capabilities + +NGINX Gateway Fabric drops ALL Linux capabilities and adds none, following security best practices. + +### How It Works Without Capabilities + +- **Process Management**: Standard Unix signals (no elevated privileges needed) +- **Port Binding**: Uses sysctl `net.ipv4.ip_unprivileged_port_start=0` for ports < 1024 +- **File Operations**: Volume mounts provide necessary write access + +### Troubleshooting + +If you encounter "operation not permitted" errors during NGINX reload, temporarily enable `allowPrivilegeEscalation: true` while investigating the root cause. + +## Security Features + +- **Separation of concerns**: Control plane (API access, no traffic) vs data plane (traffic, no API access) +- **Non-root execution**: All components run as unprivileged user (UID 101) +- **Zero capabilities**: All Linux capabilities dropped +- **Read-only root filesystem**: Prevents runtime modifications +- **Ephemeral storage**: Temporary volumes only, no persistent storage +- **Least privilege RBAC**: Minimal required permissions per component +- **Secure communication**: mTLS-encrypted gRPC (TLS 1.3+) between planes From d6e3b8b292d9a95419164f9073b4dd082dd40d42 Mon Sep 17 00:00:00 2001 From: Sarthak Agrawal Date: Mon, 14 Jul 2025 09:26:27 -0700 Subject: [PATCH 2/4] Reformat NGF permissions doc for readability --- content/ngf/reference/permissions.md | 66 +++++++++------------------- 1 file changed, 21 insertions(+), 45 deletions(-) diff --git a/content/ngf/reference/permissions.md b/content/ngf/reference/permissions.md index 9be6c6b49..dc6135535 100644 --- a/content/ngf/reference/permissions.md +++ b/content/ngf/reference/permissions.md @@ -7,18 +7,20 @@ type: reference product: NGF --- +## Overview + NGINX Gateway Fabric uses a split-plane architecture with three components that require different permissions: - **Control Plane**: Manages Kubernetes APIs and data plane deployments. Needs broad API access but handles no user traffic. - **Data Plane**: Processes user traffic. Requires minimal permissions since configuration comes from control plane via secure gRPC. - **Certificate Generator**: One-time job that creates TLS certificates for inter-plane communication. -## Common Security Context +## Security Context All components share these security settings: - **User ID**: 101 (non-root) -- **Group ID**: 1001 +- **Group ID**: 1001 - **Capabilities**: All dropped (`drop: ALL`) - **Root Filesystem**: Read-only - **Seccomp**: Runtime default profile @@ -27,20 +29,13 @@ All components share these security settings: Runs as a single container in the `nginx-gateway` deployment. -### Additional Security Settings - +**Additional Security Settings:** - **Privilege Escalation**: Disabled (may need enabling for NGINX reload in some environments) -### Volumes - -| Volume | Type | Purpose | -|--------|------|---------| -| `nginx-agent-tls` | Secret | TLS certificates for control plane communication | - -### RBAC Permissions - -The control plane requires these Kubernetes API permissions: +**Volumes:** +- `nginx-agent-tls` (Secret) - TLS certificates for control plane communication +**RBAC Permissions:** - **Secrets, ConfigMaps, Services**: Create, update, delete, list, get, watch - **Deployments, DaemonSets**: Create, update, delete, list, get, watch - **ServiceAccounts**: Create, update, delete, list, get, watch @@ -57,14 +52,12 @@ The control plane requires these Kubernetes API permissions: NGINX containers managed by the control plane. No RBAC permissions needed since configuration comes via secure gRPC. -### Additional Security Settings - +**Additional Security Settings:** - **Privilege Escalation**: Disabled - **Sysctl**: `net.ipv4.ip_unprivileged_port_start=0` (enables binding to ports < 1024) -### Volumes +**Volumes:** -#### Core NGINX Operations | Volume | Type | Purpose | |--------|------|---------| | `nginx-conf` | EmptyDir | Main NGINX configuration | @@ -72,28 +65,15 @@ NGINX containers managed by the control plane. No RBAC permissions needed since | `nginx-secrets` | EmptyDir | TLS secrets for NGINX | | `nginx-run` | EmptyDir | Runtime files (PID, sockets) | | `nginx-cache` | EmptyDir | Cache directory | - -#### Configuration Includes -| Volume | Type | Purpose | -|--------|------|---------| | `nginx-main-includes` | EmptyDir | Main context includes | | `nginx-includes` | EmptyDir | HTTP context includes | - -#### NGINX Agent -| Volume | Type | Purpose | -|--------|------|---------| | `nginx-agent` | EmptyDir | NGINX Agent configuration | | `nginx-agent-tls` | Secret | TLS certificates for control plane communication | | `nginx-agent-log` | EmptyDir | NGINX Agent logs | | `nginx-agent-lib` | EmptyDir | NGINX Agent runtime data | - -#### Authentication -| Volume | Type | Purpose | -|--------|------|---------| | `token` | Projected | Service account token | -### Volume Permissions - +**Volume Permissions:** - **EmptyDir**: Read-write (required for NGINX operation) - **Secret/ConfigMap/Projected**: Read-only @@ -101,40 +81,36 @@ NGINX containers managed by the control plane. No RBAC permissions needed since Kubernetes Job that creates initial TLS certificates. -### RBAC Permissions - +**RBAC Permissions:** - **Secrets**: Create, update, get (NGINX Gateway Fabric namespace only) -## OpenShift Compatibility +## Platform-Specific Considerations -NGINX Gateway Fabric includes Security Context Constraints (SCCs) for OpenShift: +### OpenShift Compatibility -### Control Plane SCC +NGINX Gateway Fabric includes Security Context Constraints (SCCs) for OpenShift: +**Control Plane SCC:** - **Privilege Escalation**: Disabled - **Host Access**: Disabled (network, IPC, PID, ports) - **User ID Range**: 101-101 (fixed) - **Group ID Range**: 1001-1001 (fixed) - **Volumes**: Secret only -### Data Plane SCC - +**Data Plane SCC:** Same restrictions as control plane, plus additional volume types: - - **Additional Volumes**: EmptyDir, ConfigMap, Projected -## Linux Capabilities +### Linux Capabilities NGINX Gateway Fabric drops ALL Linux capabilities and adds none, following security best practices. -### How It Works Without Capabilities - +**How It Works Without Capabilities:** - **Process Management**: Standard Unix signals (no elevated privileges needed) - **Port Binding**: Uses sysctl `net.ipv4.ip_unprivileged_port_start=0` for ports < 1024 - **File Operations**: Volume mounts provide necessary write access -### Troubleshooting - +**Troubleshooting:** If you encounter "operation not permitted" errors during NGINX reload, temporarily enable `allowPrivilegeEscalation: true` while investigating the root cause. ## Security Features @@ -145,4 +121,4 @@ If you encounter "operation not permitted" errors during NGINX reload, temporari - **Read-only root filesystem**: Prevents runtime modifications - **Ephemeral storage**: Temporary volumes only, no persistent storage - **Least privilege RBAC**: Minimal required permissions per component -- **Secure communication**: mTLS-encrypted gRPC (TLS 1.3+) between planes +- **Secure communication**: mTLS-encrypted gRPC (TLS 1.3+) between planes \ No newline at end of file From a26173868b9068e5b335c881f411af2819e2806f Mon Sep 17 00:00:00 2001 From: Sarthak Agrawal Date: Mon, 14 Jul 2025 11:15:09 -0700 Subject: [PATCH 3/4] Update permissions documentation based on PR feedback - Clarify root filesystem as read-only except for writable volumes - Generalize volume descriptions for better maintainability - Change namespace reference to "control plane namespace" - Remove privilege escalation exceptions and troubleshooting - Improve document structure and readability --- content/ngf/reference/permissions.md | 28 +++++++--------------------- 1 file changed, 7 insertions(+), 21 deletions(-) diff --git a/content/ngf/reference/permissions.md b/content/ngf/reference/permissions.md index dc6135535..68996c1b1 100644 --- a/content/ngf/reference/permissions.md +++ b/content/ngf/reference/permissions.md @@ -22,7 +22,7 @@ All components share these security settings: - **User ID**: 101 (non-root) - **Group ID**: 1001 - **Capabilities**: All dropped (`drop: ALL`) -- **Root Filesystem**: Read-only +- **Root Filesystem**: Read-only except for specific writable volumes - **Seccomp**: Runtime default profile ## Control Plane @@ -30,10 +30,10 @@ All components share these security settings: Runs as a single container in the `nginx-gateway` deployment. **Additional Security Settings:** -- **Privilege Escalation**: Disabled (may need enabling for NGINX reload in some environments) +- **Privilege Escalation**: Disabled **Volumes:** -- `nginx-agent-tls` (Secret) - TLS certificates for control plane communication +- Secret mounts for TLS certificates **RBAC Permissions:** - **Secrets, ConfigMaps, Services**: Create, update, delete, list, get, watch @@ -57,21 +57,9 @@ NGINX containers managed by the control plane. No RBAC permissions needed since - **Sysctl**: `net.ipv4.ip_unprivileged_port_start=0` (enables binding to ports < 1024) **Volumes:** - -| Volume | Type | Purpose | -|--------|------|---------| -| `nginx-conf` | EmptyDir | Main NGINX configuration | -| `nginx-stream-conf` | EmptyDir | Stream configuration | -| `nginx-secrets` | EmptyDir | TLS secrets for NGINX | -| `nginx-run` | EmptyDir | Runtime files (PID, sockets) | -| `nginx-cache` | EmptyDir | Cache directory | -| `nginx-main-includes` | EmptyDir | Main context includes | -| `nginx-includes` | EmptyDir | HTTP context includes | -| `nginx-agent` | EmptyDir | NGINX Agent configuration | -| `nginx-agent-tls` | Secret | TLS certificates for control plane communication | -| `nginx-agent-log` | EmptyDir | NGINX Agent logs | -| `nginx-agent-lib` | EmptyDir | NGINX Agent runtime data | -| `token` | Projected | Service account token | +- EmptyDir volumes for NGINX configuration, runtime files, logs, and cache +- Secret mounts for TLS certificates +- Projected token mounts for service account authentication **Volume Permissions:** - **EmptyDir**: Read-write (required for NGINX operation) @@ -82,7 +70,7 @@ NGINX containers managed by the control plane. No RBAC permissions needed since Kubernetes Job that creates initial TLS certificates. **RBAC Permissions:** -- **Secrets**: Create, update, get (NGINX Gateway Fabric namespace only) +- **Secrets**: Create, update, get (control plane namespace only) ## Platform-Specific Considerations @@ -110,8 +98,6 @@ NGINX Gateway Fabric drops ALL Linux capabilities and adds none, following secur - **Port Binding**: Uses sysctl `net.ipv4.ip_unprivileged_port_start=0` for ports < 1024 - **File Operations**: Volume mounts provide necessary write access -**Troubleshooting:** -If you encounter "operation not permitted" errors during NGINX reload, temporarily enable `allowPrivilegeEscalation: true` while investigating the root cause. ## Security Features From c922ca5f66e093691a70c0e5d752f57d7cd58c8e Mon Sep 17 00:00:00 2001 From: Sarthak Agrawal Date: Mon, 14 Jul 2025 11:38:42 -0700 Subject: [PATCH 4/4] Update volumes secret mount line to include JWT token for NGINX Plus --- content/ngf/reference/permissions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ngf/reference/permissions.md b/content/ngf/reference/permissions.md index 68996c1b1..ad95e3bef 100644 --- a/content/ngf/reference/permissions.md +++ b/content/ngf/reference/permissions.md @@ -58,7 +58,7 @@ NGINX containers managed by the control plane. No RBAC permissions needed since **Volumes:** - EmptyDir volumes for NGINX configuration, runtime files, logs, and cache -- Secret mounts for TLS certificates +- Secret mounts for TLS certificates and the NGINX Plus JWT token - Projected token mounts for service account authentication **Volume Permissions:**