Separate control plane and data plane; support multiple Gateways (#3318)

As a route to efficacy and quickly understanding the Gateway API, its implementation and alignment to NGINX as a data plane, we decided on a simplified, but rigid, deployment pattern. To improve our security posture and installation flexibility the control and data planes are being separated as semi-autonomous, distributed components. This also allows us to support multiple Gateways for a single control plane.

A general summary of the changes being made:

- control plane and data plane are now in separate Deployments
- installing NGF just installs the control plane
- when a Gateway resource is created, the control plane provisions an nginx data plane deployment and service
- the NginxProxy CRD resource can now be set at the Gateway level, and has been enhanced to include all deployment/service infrastructure-related fields, such as replicas, loadBalancerIP, serviceType, etc.
   - these fields can be configured globally at installation time in the helm chart, or set on an individual basis per Gateway
    - updating these fields directly on a provisioned nginx Deployment or Service will not take effect
    - this does not apply to the control plane Deployment
- labels/annotations for the NGINX deployment or service can be set in the Gateway's Infrastructure section
- the NGINX pod uses the NGINX agent (currently an unofficial, unreleased version) to update NGINX configuration
- control plane communicates with the NGINX agent over a secure gRPC connection, using self-signed certs by default, created at installation time. Cert-manager can be used instead.
- multiple Gateways is now supported

---------

Co-authored-by: Kate Osborn <[email protected]>
Co-authored-by: bjee19 <[email protected]>
Co-authored-by: salonichf5 <[email protected]>
Co-authored-by: Benjamin Jee <[email protected]>

Author: 4 people

.github/ISSUE_TEMPLATE/bug_report.md

@@ -24,8 +24,8 @@ A clear and concise description of what you expected to happen.
 * Version of Kubernetes
 * Kubernetes platform (e.g. Mini-kube or GCP)
 * Details on how you expose the NGINX Gateway Fabric Pod (e.g. Service of type LoadBalancer or port-forward)
-* Logs of NGINX container: `kubectl -n nginx-gateway logs -l app=nginx-gateway -c nginx`
-* NGINX Configuration: `kubectl -n nginx-gateway exec <gateway-pod> -c nginx -- nginx -T`
+* Logs of NGINX container: `kubectl -n <nginx-deployment-namespace> logs deployments/<nginx-deployment>`
+* NGINX Configuration: `kubectl -n <nginx-deployment-namespace> exec -it deployments/<nginx-deployment> -- nginx -T`
 
 **Additional context**
 Add any other context about the problem here. Any log files you want to share.

.github/workflows/conformance.yml

@@ -76,13 +76,6 @@ jobs:
             type=ref,event=pr
             type=ref,event=branch,suffix=-rc,enable=${{ startsWith(github.ref, 'refs/heads/release') }}
 
-      - name: Generate static deployment
-        run: |
-          ngf_prefix=ghcr.io/nginx/nginx-gateway-fabric
-          ngf_tag=${{ steps.ngf-meta.outputs.version }}
-          make generate-static-deployment PLUS_ENABLED=${{ inputs.image == 'plus' && 'true' || 'false' }} PREFIX=${ngf_prefix} TAG=${ngf_tag}
-        working-directory: ./tests
-
       - name: Build binary
         uses: goreleaser/goreleaser-action@9c156ee8a17a598857849441385a2041ef570552 # v6.3.0
         with:
@@ -151,7 +144,6 @@ jobs:
           ngf_tag=${{ steps.ngf-meta.outputs.version }}
           if [ ${{ github.event_name }} == "schedule" ]; then export GW_API_VERSION=main; fi
           make helm-install-local${{ inputs.image == 'plus' && '-with-plus' || ''}} PREFIX=${ngf_prefix} TAG=${ngf_tag}
-          make deploy-updated-provisioner PREFIX=${ngf_prefix} TAG=${ngf_tag}
         working-directory: ./tests
 
       - name: Run conformance tests

.github/workflows/helm.yml

@@ -176,4 +176,4 @@ jobs:
           --set=nginx.plus=${{ inputs.image == 'plus' }} \
           --set=nginx.image.tag=nightly \
           --set=nginxGateway.productTelemetry.enable=false \
-          ${{ inputs.image == 'plus' && '--set=serviceAccount.imagePullSecret=nginx-plus-registry-secret --set=nginx.image.repository=private-registry.nginx.com/nginx-gateway-fabric/nginx-plus' || '' }}"
+          ${{ inputs.image == 'plus' && '--set=nginx.imagePullSecret=nginx-plus-registry-secret --set=nginx.image.repository=private-registry.nginx.com/nginx-gateway-fabric/nginx-plus' || '' }}"

.github/workflows/nfr.yml

@@ -92,6 +92,13 @@ jobs:
           workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY }}
           service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
 
+      - name: Login to GAR
+        uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
+        with:
+          registry: us-docker.pkg.dev
+          username: oauth2accesstoken
+          password: ${{ steps.auth.outputs.access_token }}
+
       - name: Set up Cloud SDK
         uses: google-github-actions/setup-gcloud@77e7a554d41e2ee56fc945c52dfd3f33d12def9a # v2.1.4
         with:

.yamllint.yaml

@@ -2,8 +2,7 @@
 ignore:
   - charts/nginx-gateway-fabric/templates
   - config/crd/bases/
-  - deploy/crds.yaml
-  - deploy/*nginx-plus
+  - deploy
   - site/static
 
 rules:
@@ -15,7 +14,9 @@ rules:
     require-starting-space: true
     ignore-shebangs: true
     min-spaces-from-content: 1
-  comments-indentation: enable
+  comments-indentation:
+    ignore: |
+      charts/nginx-gateway-fabric/values.yaml
   document-end: disable
   document-start: disable
   empty-lines: enable

Makefile

@@ -226,13 +226,13 @@ install-ngf-local-build-with-plus: check-for-plus-usage-endpoint build-images-wi
 
 .PHONY: helm-install-local
 helm-install-local: install-gateway-crds ## Helm install NGF on configured kind cluster with local images. To build, load, and install with helm run make install-ngf-local-build.
-	helm install nginx-gateway $(CHART_DIR) --set nginx.image.repository=$(NGINX_PREFIX) --create-namespace --wait --set nginxGateway.image.pullPolicy=Never --set service.type=NodePort --set nginxGateway.image.repository=$(PREFIX) --set nginxGateway.image.tag=$(TAG) --set nginx.image.tag=$(TAG) --set nginx.image.pullPolicy=Never --set nginxGateway.gwAPIExperimentalFeatures.enable=$(ENABLE_EXPERIMENTAL) -n nginx-gateway $(HELM_PARAMETERS)
+	helm install nginx-gateway $(CHART_DIR) --set nginx.image.repository=$(NGINX_PREFIX) --create-namespace --wait --set nginxGateway.image.pullPolicy=Never --set nginx.service.type=NodePort --set nginxGateway.image.repository=$(PREFIX) --set nginxGateway.image.tag=$(TAG) --set nginx.image.tag=$(TAG) --set nginx.image.pullPolicy=Never --set nginxGateway.gwAPIExperimentalFeatures.enable=$(ENABLE_EXPERIMENTAL) -n nginx-gateway $(HELM_PARAMETERS)
 
 .PHONY: helm-install-local-with-plus
 helm-install-local-with-plus: check-for-plus-usage-endpoint install-gateway-crds ## Helm install NGF with NGINX Plus on configured kind cluster with local images. To build, load, and install with helm run make install-ngf-local-build-with-plus.
 	kubectl create namespace nginx-gateway || true
 	kubectl -n nginx-gateway create secret generic nplus-license --from-file $(PLUS_LICENSE_FILE) || true
-	helm install nginx-gateway $(CHART_DIR) --set nginx.image.repository=$(NGINX_PLUS_PREFIX) --wait --set nginxGateway.image.pullPolicy=Never --set service.type=NodePort --set nginxGateway.image.repository=$(PREFIX) --set nginxGateway.image.tag=$(TAG) --set nginx.image.tag=$(TAG) --set nginx.image.pullPolicy=Never --set nginxGateway.gwAPIExperimentalFeatures.enable=$(ENABLE_EXPERIMENTAL) -n nginx-gateway --set nginx.plus=true --set nginx.usage.endpoint=$(PLUS_USAGE_ENDPOINT) $(HELM_PARAMETERS)
+	helm install nginx-gateway $(CHART_DIR) --set nginx.image.repository=$(NGINX_PLUS_PREFIX) --wait --set nginxGateway.image.pullPolicy=Never --set nginx.service.type=NodePort --set nginxGateway.image.repository=$(PREFIX) --set nginxGateway.image.tag=$(TAG) --set nginx.image.tag=$(TAG) --set nginx.image.pullPolicy=Never --set nginxGateway.gwAPIExperimentalFeatures.enable=$(ENABLE_EXPERIMENTAL) -n nginx-gateway --set nginx.plus=true --set nginx.usage.endpoint=$(PLUS_USAGE_ENDPOINT) $(HELM_PARAMETERS)
 
 .PHONY: check-for-plus-usage-endpoint
 check-for-plus-usage-endpoint: ## Checks that the PLUS_USAGE_ENDPOINT is set in the environment. This env var is required when deploying or testing with N+.

apis/v1alpha1/nginxproxy_types.go

@@ -34,8 +34,6 @@ func addKnownTypes(scheme *runtime.Scheme) error {
 	scheme.AddKnownTypes(SchemeGroupVersion,
 		&NginxGateway{},
 		&NginxGatewayList{},
-		&NginxProxy{},
-		&NginxProxyList{},
 		&ObservabilityPolicy{},
 		&ObservabilityPolicyList{},
 		&ClientSettingsPolicy{},