Skip to content

Commit f2bc918

Browse files
sjbermansjberman
authored
CP/DP split: Secure connection (#3244)
Problem: We want to ensure that the connection between the control plane and data plane is authenticated and secure. Solution: 1. Configure agent to send the kubernetes service token in the request. The control plane validates this token using the TokenReview API to ensure the agent is authenticated. 2. Configure TLS certificates for both the control and data planes. By default, a Job will run when installing NGF that creates self-signed certificates in the nginx-gateway namespace. The server Secret is mounted to the control plane, and the control plane copies the client Secret when deploying nginx resources. This Secret is mounted to the agent. The control plane will reset the agent connection if it detects that its own certs have changed. For production environments, we'll recommend a user configures TLS using cert-manager instead, for better security and certificate rotation.
1 parent a9f53ec commit f2bc918

File tree

50 files changed

+3082
-120
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

50 files changed

+3082
-120
lines changed

‎charts/nginx-gateway-fabric/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,12 @@ The following table lists the configurable parameters of the NGINX Gateway Fabri
258258

259259
| Key | Description | Type | Default |
260260
|-----|-------------|------|---------|
261+
| `certGenerator` | The certGenerator section contains the configuration for the cert-generator Job. | object | `{"agentTLSSecretName":"agent-tls","annotations":{},"overwrite":false,"serverTLSSecretName":"server-tls"}` |
262+
| `certGenerator.agentTLSSecretName` | The name of the base Secret containing TLS CA, certificate, and key for the NGINX Agent to securely communicate with the NGINX Gateway Fabric control plane. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). | string | `"agent-tls"` |
263+
| `certGenerator.annotations` | The annotations of the cert-generator Job. | object | `{}` |
264+
| `certGenerator.overwrite` | Overwrite existing TLS Secrets on startup. | bool | `false` |
265+
| `certGenerator.serverTLSSecretName` | The name of the Secret containing TLS CA, certificate, and key for the NGINX Gateway Fabric control plane to securely communicate with the NGINX Agent. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). | string | `"server-tls"` |
266+
| `clusterDomain` | The DNS cluster domain of your Kubernetes cluster. | string | `"cluster.local"` |
261267
| `nginx` | The nginx section contains the configuration for all NGINX data plane deployments installed by the NGINX Gateway Fabric control plane. | object | `{"config":{},"container":{},"debug":false,"image":{"pullPolicy":"Always","repository":"ghcr.io/nginx/nginx-gateway-fabric/nginx","tag":"edge"},"imagePullSecret":"","imagePullSecrets":[],"kind":"deployment","plus":false,"pod":{},"replicas":1,"service":{"externalTrafficPolicy":"Local","type":"LoadBalancer"},"usage":{"caSecretName":"","clientSSLSecretName":"","endpoint":"","resolver":"","secretName":"nplus-license","skipVerify":false}}` |
262268
| `nginx.config` | The configuration for the data plane that is contained in the NginxProxy resource. | object | `{}` |
263269
| `nginx.container` | The container configuration for the NGINX container. | object | `{}` |

‎charts/nginx-gateway-fabric/templates/certs-job.yaml‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
apiVersion: v1
2+
kind: ServiceAccount
3+
metadata:
4+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
5+
namespace: {{ .Release.Namespace }}
6+
labels:
7+
{{- include "nginx-gateway.labels" . | nindent 4 }}
8+
annotations:
9+
"helm.sh/hook": pre-install
10+
{{- if or .Values.nginxGateway.serviceAccount.imagePullSecret .Values.nginxGateway.serviceAccount.imagePullSecrets }}
11+
imagePullSecrets:
12+
{{- if .Values.nginxGateway.serviceAccount.imagePullSecret }}
13+
- name: {{ .Values.nginxGateway.serviceAccount.imagePullSecret }}
14+
{{- end }}
15+
{{- if .Values.nginxGateway.serviceAccount.imagePullSecrets }}
16+
{{- range .Values.nginxGateway.serviceAccount.imagePullSecrets }}
17+
- name: {{ . }}
18+
{{- end }}
19+
{{- end }}
20+
{{- end }}
21+
---
22+
apiVersion: rbac.authorization.k8s.io/v1
23+
kind: Role
24+
metadata:
25+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
26+
namespace: {{ .Release.Namespace }}
27+
labels:
28+
{{- include "nginx-gateway.labels" . | nindent 4 }}
29+
annotations:
30+
"helm.sh/hook": pre-install
31+
rules:
32+
- apiGroups:
33+
- ""
34+
resources:
35+
- secrets
36+
verbs:
37+
- create
38+
- update
39+
- get
40+
---
41+
apiVersion: rbac.authorization.k8s.io/v1
42+
kind: RoleBinding
43+
metadata:
44+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
45+
namespace: {{ .Release.Namespace }}
46+
labels:
47+
{{- include "nginx-gateway.labels" . | nindent 4 }}
48+
annotations:
49+
"helm.sh/hook": pre-install
50+
roleRef:
51+
apiGroup: rbac.authorization.k8s.io
52+
kind: Role
53+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
54+
subjects:
55+
- kind: ServiceAccount
56+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
57+
namespace: {{ .Release.Namespace }}
58+
---
59+
apiVersion: batch/v1
60+
kind: Job
61+
metadata:
62+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
63+
namespace: {{ .Release.Namespace }}
64+
labels:
65+
{{- include "nginx-gateway.labels" . | nindent 4 }}
66+
annotations:
67+
{{- with .Values.certGenerator.annotations -}}
68+
{{ toYaml . | nindent 4 }}
69+
{{- end }}
70+
"helm.sh/hook": pre-install, pre-upgrade
71+
spec:
72+
template:
73+
metadata:
74+
annotations:
75+
{{- with .Values.certGenerator.annotations -}}
76+
{{ toYaml . | nindent 8 }}
77+
{{- end }}
78+
spec:
79+
containers:
80+
- args:
81+
- generate-certs
82+
- --service={{ include "nginx-gateway.fullname" . }}
83+
- --cluster-domain={{ .Values.clusterDomain }}
84+
- --server-tls-secret={{ .Values.certGenerator.serverTLSSecretName }}
85+
- --agent-tls-secret={{ .Values.certGenerator.agentTLSSecretName }}
86+
{{- if .Values.certGenerator.overwrite }}
87+
- --overwrite
88+
{{- end }}
89+
env:
90+
- name: POD_NAMESPACE
91+
valueFrom:
92+
fieldRef:
93+
fieldPath: metadata.namespace
94+
image: {{ .Values.nginxGateway.image.repository }}:{{ default .Chart.AppVersion .Values.nginxGateway.image.tag }}
95+
imagePullPolicy: {{ .Values.nginxGateway.image.pullPolicy }}
96+
name: cert-generator
97+
securityContext:
98+
seccompProfile:
99+
type: RuntimeDefault
100+
capabilities:
101+
drop:
102+
- ALL
103+
allowPrivilegeEscalation: false
104+
readOnlyRootFilesystem: true
105+
runAsUser: 101
106+
runAsGroup: 1001
107+
restartPolicy: Never
108+
serviceAccountName: {{ include "nginx-gateway.fullname" . }}-cert-generator
109+
securityContext:
110+
fsGroup: 1001
111+
runAsNonRoot: true
112+
ttlSecondsAfterFinished: 0

‎charts/nginx-gateway-fabric/templates/clusterrole.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,12 @@ rules:
5959
verbs:
6060
- list
6161
- watch
62+
- apiGroups:
63+
- authentication.k8s.io
64+
resources:
65+
- tokenreviews
66+
verbs:
67+
- create
6268
- apiGroups:
6369
- gateway.networking.k8s.io
6470
resources:

‎charts/nginx-gateway-fabric/templates/deployment.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ spec:
3636
- --gatewayclass={{ .Values.nginxGateway.gatewayClassName }}
3737
- --config={{ include "nginx-gateway.config-name" . }}
3838
- --service={{ include "nginx-gateway.fullname" . }}
39+
- --agent-tls-secret={{ .Values.certGenerator.agentTLSSecretName }}
3940
{{- if .Values.nginx.imagePullSecret }}
4041
- --nginx-docker-secret={{ .Values.nginx.imagePullSecret }}
4142
{{- end }}
@@ -149,6 +150,9 @@ spec:
149150
readOnlyRootFilesystem: true
150151
runAsUser: 101
151152
runAsGroup: 1001
153+
volumeMounts:
154+
- name: nginx-agent-tls
155+
mountPath: /var/run/secrets/ngf
152156
{{- with .Values.nginxGateway.extraVolumeMounts -}}
153157
{{ toYaml . | nindent 8 }}
154158
{{- end }}
@@ -173,6 +177,10 @@ spec:
173177
nodeSelector:
174178
{{- toYaml .Values.nginxGateway.nodeSelector | nindent 8 }}
175179
{{- end }}
180+
volumes:
181+
- name: nginx-agent-tls
182+
secret:
183+
secretName: {{ .Values.certGenerator.serverTLSSecretName }}
176184
{{- with .Values.nginxGateway.extraVolumes -}}
177185
{{ toYaml . | nindent 6 }}
178186
{{- end }}

‎charts/nginx-gateway-fabric/templates/scc.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,4 +34,6 @@ users:
3434
- {{ printf "system:serviceaccount:%s:%s" .Release.Namespace (include "nginx-gateway.serviceAccountName" .) }}
3535
requiredDropCapabilities:
3636
- ALL
37+
volumes:
38+
- secret
3739
{{- end }}

‎charts/nginx-gateway-fabric/values.schema.json‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,48 @@
11
{
22
"$schema": "http://json-schema.org/draft-07/schema#",
33
"properties": {
4+
"certGenerator": {
5+
"description": "The certGenerator section contains the configuration for the cert-generator Job.",
6+
"properties": {
7+
"agentTLSSecretName": {
8+
"default": "agent-tls",
9+
"description": "The name of the base Secret containing TLS CA, certificate, and key for the NGINX Agent to securely\ncommunicate with the NGINX Gateway Fabric control plane. Must exist in the same namespace that the\nNGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway).",
10+
"required": [],
11+
"title": "agentTLSSecretName",
12+
"type": "string"
13+
},
14+
"annotations": {
15+
"description": "The annotations of the cert-generator Job.",
16+
"required": [],
17+
"title": "annotations",
18+
"type": "object"
19+
},
20+
"overwrite": {
21+
"default": false,
22+
"description": "Overwrite existing TLS Secrets on startup.",
23+
"required": [],
24+
"title": "overwrite",
25+
"type": "boolean"
26+
},
27+
"serverTLSSecretName": {
28+
"default": "server-tls",
29+
"description": "The name of the Secret containing TLS CA, certificate, and key for the NGINX Gateway Fabric control plane\nto securely communicate with the NGINX Agent. Must exist in the same namespace that the NGINX Gateway Fabric\ncontrol plane is running in (default namespace: nginx-gateway).",
30+
"required": [],
31+
"title": "serverTLSSecretName",
32+
"type": "string"
33+
}
34+
},
35+
"required": [],
36+
"title": "certGenerator",
37+
"type": "object"
38+
},
39+
"clusterDomain": {
40+
"default": "cluster.local",
41+
"description": "The DNS cluster domain of your Kubernetes cluster.",
42+
"required": [],
43+
"title": "clusterDomain",
44+
"type": "string"
45+
},
446
"global": {
547
"description": "Global values are values that can be accessed from any chart or subchart by exactly the same name.",
648
"required": [],

‎charts/nginx-gateway-fabric/values.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
# yaml-language-server: $schema=values.schema.json
22

3+
# -- The DNS cluster domain of your Kubernetes cluster.
4+
clusterDomain: cluster.local
5+
36
# -- The nginxGateway section contains configuration for the NGINX Gateway Fabric control plane deployment.
47
nginxGateway:
58
# FIXME(lucacome): https://github.com/nginx/nginx-gateway-fabric/issues/2490
@@ -426,3 +429,21 @@ nginx:
426429

427430
# -- Enable debugging for NGINX. Uses the nginx-debug binary. The NGINX error log level should be set to debug in the NginxProxy resource.
428431
debug: false
432+
433+
# -- The certGenerator section contains the configuration for the cert-generator Job.
434+
certGenerator:
435+
# -- The annotations of the cert-generator Job.
436+
annotations: {}
437+
438+
# -- The name of the Secret containing TLS CA, certificate, and key for the NGINX Gateway Fabric control plane
439+
# to securely communicate with the NGINX Agent. Must exist in the same namespace that the NGINX Gateway Fabric
440+
# control plane is running in (default namespace: nginx-gateway).
441+
serverTLSSecretName: server-tls
442+
443+
# -- The name of the base Secret containing TLS CA, certificate, and key for the NGINX Agent to securely
444+
# communicate with the NGINX Gateway Fabric control plane. Must exist in the same namespace that the
445+
# NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway).
446+
agentTLSSecretName: agent-tls
447+
448+
# -- Overwrite existing TLS Secrets on startup.
449+
overwrite: false

0 commit comments

Comments
 (0)