feat (in-cluster): [webapprouting] replace traefik with nginx addon by ferantivero · Pull Request #446 · mspnp/aks-baseline

ferantivero · 2025-10-22T18:13:55Z

WHY

we wanted to experiment with the NGINX addon (web app routing) that could replace the existing ingress controller in the AKS Baseline Reference Implementation (Traefik). This way we can remove one of the "manual" dependencies we took, streamline the deployment process and use built-in features in AKS.

WHAT Changed?

TEST

tested e2e and we plan to test this once again after addressing feedback + upon approval

highights: - configure the application routing add-on to automatically create records on the Ingress private DNS zones

…ancer highlights: - configure nginx with an ILB - update the managed cluster api version

…api version

…ing AKS addon (nginx)

…tainer image

…stance name

…steps

… load balancer should be bound to

… probesm, etc

…routing-system managed identity

Copilot

Pull Request Overview

This PR replaces the manually deployed Traefik ingress controller with the AKS-managed NGINX Web App Routing addon to streamline the deployment process. The change enables a built-in AKS feature for ingress management with integrated Azure Key Vault and Private DNS Zone support.

Key Changes:

Enabled the built-in Web App Routing addon with NGINX ingress controller configured for internal load balancing
Integrated managed identity for Key Vault certificate access and Private DNS Zone record management
Removed all Traefik-related resources and dependencies (deployment manifests, CSI provider configuration, workload identity setup)

Reviewed Changes

Copilot reviewed 18 out of 20 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
workload/traefik.yaml	Removed entire Traefik ingress controller manifest including ServiceAccount, RBAC, ConfigMap, Service, and Deployment
workload/02-aspnetapp-ingress.yaml	Updated ingress resource to use nginx-internal class with Key Vault certificate integration and NGINX-specific annotations
workload/01-aspnetapp.yaml	Modified pod security context, added health probes, updated affinity rules to target NGINX ingress controller, added NET_BIND_SERVICE capability
workload-team/cluster-stamp.bicep	Enabled webAppRouting profile, configured DNS zone integration, removed podmi-ingress-controller resources, added role assignments for web app routing managed identity, upgraded API version to 2025-07-02-preview
cluster-manifests/a0008/nginx-internal.yaml	Added new NginxIngressController custom resource defining internal load balancer configuration
cluster-manifests/a0008/ingress-network-policy.yaml	Updated network policy to allow traffic from app-routing-system namespace and nginx-internal ingress controller pods
workload-team/modules/policies.bicep	Removed Traefik-related policy violation comments and updated resource limit comments
docs/*	Updated documentation to reflect NGINX ingress controller, removed Traefik installation steps, updated navigation links

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-22T18:15:25Z

docs/deploy/08-workload-prerequisites.md

+   cat nginx-iternal-ingress-controller-tls.crt nginx-iternal-ingress-controller-tls.key > nginx-iternal-ingress-controller-tls.pem
+   export INGRESS_CONTROLLER_KV_CERT_URI=$(az keyvault certificate import -f nginx-iternal-ingress-controller-tls.pem -n nginx-iternal-ingress-controller-tls --vault-name $KEYVAULT_NAME_AKS_BASELINE --query id -o tsv)


Corrected spelling of 'iternal' to 'internal' in certificate filename.

Suggested change

cat nginx-iternal-ingress-controller-tls.crt nginx-iternal-ingress-controller-tls.key > nginx-iternal-ingress-controller-tls.pem

export INGRESS_CONTROLLER_KV_CERT_URI=$(az keyvault certificate import -f nginx-iternal-ingress-controller-tls.pem -n nginx-iternal-ingress-controller-tls --vault-name $KEYVAULT_NAME_AKS_BASELINE --query id -o tsv)

cat nginx-internal-ingress-controller-tls.crt nginx-internal-ingress-controller-tls.key > nginx-internal-ingress-controller-tls.pem

export INGRESS_CONTROLLER_KV_CERT_URI=$(az keyvault certificate import -f nginx-internal-ingress-controller-tls.pem -n nginx-internal-ingress-controller-tls --vault-name $KEYVAULT_NAME_AKS_BASELINE --query id -o tsv)

done | addressed from f825b73

Copilot · 2025-10-22T18:15:25Z

docs/deploy/02-ca-certificates.md


   ```bash
-   openssl req -x509 -nodes -days 365 -newkey rsa:2048 -out traefik-ingress-internal-aks-ingress-tls.crt -keyout traefik-ingress-internal-aks-ingress-tls.key -subj "/CN=*.aks-ingress.${DOMAIN_NAME_AKS_BASELINE}/O=Contoso AKS Ingress"
+   openssl req -x509 -nodes -days 365 -newkey rsa:2048 -out nginx-iternal-ingress-controller-tls.crt -keyout nginx-iternal-ingress-controller-tls.key -subj "/CN=*.aks-ingress.${DOMAIN_NAME_AKS_BASELINE}/O=Contoso AKS Ingress"


Corrected spelling of 'iternal' to 'internal' in certificate filename.

done | addressed from f825b73

Copilot · 2025-10-22T18:15:26Z

docs/deploy/08-workload-prerequisites.md

   az role assignment delete --ids $TEMP_ROLEASSIGNMENT_TO_UPLOAD_CERT
   ```

+## Check internal NGINX ingress controller is up and runnning


Corrected spelling of 'runnning' to 'running'.

Suggested change

## Check internal NGINX ingress controller is up and runnning

## Check internal NGINX ingress controller is up and running

done | addressed from f825b73

Copilot · 2025-10-22T18:15:26Z

workload/01-aspnetapp.yaml

-      securityContext:
-        runAsUser: 10001
-        runAsGroup: 3000
+      securityContext: {}


The pod-level securityContext is now empty but container-level security settings (runAsNonRoot, runAsUser, runAsGroup) are defined at lines 77-79. Consider whether pod-level fsGroup setting should be retained for consistent group ownership of volumes, or document why it was intentionally removed.

Suggested change

securityContext: {}

securityContext:

fsGroup: 3000

done | addressed from 5f0d237

Copilot · 2025-10-22T18:15:26Z

workload-team/cluster-stamp.bicep

 }

-// Built-in Azure RBAC role that is applied a Key Vault to grant with metadata, certificates, keys and secrets read privileges.  Granted to App Gateway's managed identity.
+// Built-in Azure RBAC role that is applied a Key Vault to grant with metadata, certificates, keys and secrets read privileges. Granted to App Gateway's managed identity and our web app routing profile's managed identiy.


Corrected spelling of 'identiy' to 'identity'.

done | addressed from f825b73

Copilot · 2025-10-22T18:15:27Z

workload-team/cluster-stamp.bicep

 }

-// Built-in Azure RBAC role that is applied to a Key Vault to grant with secrets content read privileges. Granted to both Key Vault and our workload's identity.
+// Built-in Azure RBAC role that is applied to a Key Vault to grant with secrets content read privileges. Granted to our web app routing profile's managed identiy.


Corrected spelling of 'identiy' to 'identity'.

done | addressed from f825b73

Copilot · 2025-10-22T18:15:27Z

workload-team/cluster-stamp.bicep

  scope: subscription()
 }

+// Built-in Azure RBAC role that is applied to a Private DNS Zone to grant with contributor privileges. Granted our web app routing profile's managed identiy.


Corrected spelling of 'identiy' to 'identity'.

done | addressed from f825b73

highlights: - CIS Benchmarks and most cluster-hardening guides recommend non-root users + fsGroup to ensure the principle of least privilege and writable volumes. - fsGroup is not redudant since the workload is not configured ad read-only. - shared value btw fsGroup and runAsGroup (primary process GID) is fine for simple cases. Process can read/write volume files without needing additional permission adjustments. The process’s own group is enough to access its volumes. In other words, same group ID governs both “who I am” and “what I can write.”

docs/deploy/08-workload-prerequisites.md

docs/deploy/09-workload.md

docs/deploy/10-validation.md

workload-team/cluster-stamp.bicep

Co-authored-by: John Downs <john@johndowns.co.nz>

ferantivero added 24 commits October 16, 2025 11:25

enable application routing addon

2f73a0a

attach an aks ingress private DNS zone to the application routing add-on

0356bbf

highights: - configure the application routing add-on to automatically create records on the Ingress private DNS zones

default nginx ingress controller is created with an internal load bal…

84be924

…ancer highlights: - configure nginx with an ILB - update the managed cluster api version

enablePodSecurityPolicy is not allowed in the latest managed cluster …

eea8850

…api version

streamline deploy steps: remove the fixed ilb ip address

eea1112

get rid of traefik installation in favor of enabling the web app rout…

c57b934

…ing AKS addon (nginx)

streamline deployment steps: no longer required to import traefik con…

d95c7da

…tainer image

streamline deployment steps: no longer required to capture the acr in…

d137309

…stance name

remove no longer used bicep output acr instance name from deployment …

96fa917

…steps

create NGINX ingress controller w/ a PIP and an ILB

3036267

configure ingress object to use nginx

8868e3a

rename kv cert/secret name to be aligned with nginx

f08349d

rbac cluster web app routing identity to kv

de1f019

integrate web app routing addong with kv

d6d4379

cover the case for ingress http resource redirection (HTTP308)

0c16c4d

rbac cluster web app routing identity to pdz

6006db0

customize NginxIngressController to specify which subnet the internal…

d90837a

… load balancer should be bound to

indicate the container port for the aspnet-webapp-sample

f136073

bug fix: pod affinity selection and node selection

a720c60

modify ingress network policy for nginx

1399639

workloads deploy object: general improvements sec, liveness/readiness…

cd4a91c

… probesm, etc

strealine: remove secret management section as this now managed

05e98cf

remove no longer used UMI for ingress controller in favor of the app-…

e1e0b51

…routing-system managed identity

clean up traefik references

3654d35

Copilot AI review requested due to automatic review settings October 22, 2025 18:13

Copilot AI reviewed Oct 22, 2025

View reviewed changes

ferantivero added 2 commits November 3, 2025 13:16

Address PR Feedback: correct spelling

f825b73

johndowns reviewed Nov 6, 2025

View reviewed changes

Address PR Feedback: improve format, naming conventions and fix typos

a044b3e

Co-authored-by: John Downs <john@johndowns.co.nz>

		cat nginx-iternal-ingress-controller-tls.crt nginx-iternal-ingress-controller-tls.key > nginx-iternal-ingress-controller-tls.pem
		export INGRESS_CONTROLLER_KV_CERT_URI=$(az keyvault certificate import -f nginx-iternal-ingress-controller-tls.pem -n nginx-iternal-ingress-controller-tls --vault-name $KEYVAULT_NAME_AKS_BASELINE --query id -o tsv)

	## Check internal NGINX ingress controller is up and runnning
	## Check internal NGINX ingress controller is up and running

Conversation

ferantivero commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WHY

WHAT Changed?

TEST

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

ferantivero Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

ferantivero Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

ferantivero Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

ferantivero Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

ferantivero Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

ferantivero Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

ferantivero Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ferantivero commented Oct 22, 2025 •

edited

Loading