Azure IPAM on AKS

[DCMattyG]

Hello @hmizael, I wanted to dive deeper into this ask with you here...

For IPAM running on AKS, I have a few design questions for you:

1. Would you like to see an option to deploy IPAM on a new AKS cluster, or it's more likely users will have an existing one they want to deploy to?
2. What is the desired way to handle ingress? Would you put an App Gateway in front, or just use the native k8s ingress?
3. If k8s ingress, how would you do path-based routing to map "/" to the UI and "/api" to the engine? NGINX? Traefik? HA Proxy? Other?
4. How would you handle SSL certs? App Service and Azure Container apps have this functionality built-in...how would/should users handle this with AKS?
5. How would you handle DNS so that the FQDN for IPAM running on AKS is resolved properly for users?

I definitely have ideas on how I would handle these items, but I'm more interested in your thoughts on how this should be automated in such a way many users could benefit from this solution, as opposed to just providing design guidance around how to deploy IPAM on AKS.

Thanks so much!

[hmizael]

Thank you so Much for your attention @DCMattyG!

So I'll answer based on what I did to deploy in my environment, an already existing production AKS:

I created a Cosmos DB account like Bicep would;
I didn't create other resources like Kv, App Services etc.

I created a Private Endpoint for Cosmos, to bring a little more security and traffic control;
As accessing IPAM only by private link, we do not use any External Ingress or Application Gateway solution;
We do not use the NGinx Container, as we already use an NGinx as an Ingress-Controller, so we use the existing one.

After this briefing, let's go to the settings for the items you mentioned:

I'm then handling the input with nginx ingress:

ingress-ipam.yaml:

apiVersion: networking.k8s.io/v1
type: input
metadata:
  name: ipam
  namespace: ipam
  notes:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    cert-manager.io/cluster-issuer: <CERT MANAGER NAME>
specification:
  tls:
  - hosts:
      - <DNS NAME>
    secretName: ipam-tls
  the rules:
  - host: <DNS NAME>
    http:
      ways:
        - way: /
          pathType: prefix
          Internal process:
            service:
              name: ipam-ui
              door:
                number: 80
        - path: /api
          pathType: prefix
          Internal process:
            service:
              name: motor-ipam
              door:
                number: 80

As you can see, I'm using port mapping by ingress.
For the certificate issue, as we have the Cert-Manager already configured in the cluster, just point it out and we also have the certificate released.

For DNS, I created a record in our Private DNS Zone to point to Ingress.

Enjoy then and I bring my Deploy, Service, Secrets settings etc.
Note: in relation to the secret, we use the External Secrets Operators here, which brings the Kv secrets to the Cluster (https://external-secrets.io/)

deployment-ipam-engine.yaml:

apiVersion: apps/v1
type: deployment
metadata:
  name: motor-ipam
  namespace: ipam
  labels:
    app: ipam
    service: engine
specification:
  replicas: 1
  selector:
    matchLabels:
      app: ipam
      service: engine
  model:
    metadata:
      labels:
        app: ipam
        service: engine
    specification:
      containers:
      - name: motor-ipam
        image: azureipam.azurecr.io/ipam-engine:latest
        Resources: {}
        envDe:
        - secretRef:
            name: ipam

deployment-ipam-ui.yaml:

apiVersion: apps/v1
type: deployment
metadata:
  name: ipam-ui
  namespace: ipam
  labels:
    app: ipam
    service: ui
specification:
  replicas: 1
  selector:
    matchLabels:
      app: ipam
      service: ui
  model:
    metadata:
      labels:
        app: ipam
        service: ui
    specification:
      containers:
      - name: ipam-ui
        image: azureipam.azurecr.io/ipam-ui:latest
        Resources: {}
        envDe:
        - secretRef:
            name: ipam

service-ipam-engine.yaml:

apiVersion: v1
type: service
metadata:
  name: motor-ipam
  namespace: ipam
  labels:
    app: ipam
    service: engine-ui
specification:
  selector:
    app: ipam
    service: engine
  doors:
    - name: http
      port: 80
      targetPort: 80

service-ipam-ui.yaml:

apiVersion: v1
type: service
metadata:
  name: ipam-ui
  namespace: ipam
  labels:
    app: ipam
    service: service-ui
specification:
  selector:
    app: ipam
    service: ui
  doors:
    - name: http
      port: 80
      targetPort: 80

Then that's it! Remembering that this form of deployment is what works for me, but I think native support for AKS by the more comprehensive Azure IPAM would be very valid, allowing you to choose between Application Gateway or Ingress-Controller, direct support for Kv and so on...

Thanks again for your attention!

[DCMattyG]

Thanks you very much @hmizael, this is extremely thorough and detailed. I have a few follow-up questions:

1. You mentioned adding a record to the Private DNS Zone, so I'm assuming you have some enterprise DNS solution that lives within the vNET where the Private DNS Zone is mapped, is that correct?
2. What type of k8s networking (Kubenet/Azure CNI)? I believe it would be Kubenet and you have NGINX with a Private Link configured (if I read the above correctly).
3. I'm assuming for the secrets store, you had a pre-existing configuration? If not and you created a new one, what type of identity did you use (User-Assigned, System-Assigned, Service Principal)?
4. Do I have your permission to include the example YAML's you provided above in our documentation for an AKS reference deployment?

Really appreciate the engagement and contributions!

[hmizael]

Good evening!

So I use Private Endpoint and I also have private DNS zones where I create DNS records.

I use kubenet. And here we use NGINX Ingress Controller.

We also use External-Secrets.io here, so that we can integrate Azure Keyvault with our AKS. So I bring the environment variables that are written in an AKV to the IPAM pod. We are using the main service here.

Yes sure! Feel free to use. If you need any more information, just ask.

[cloudtech555]

Hello @DCMattyG, we deployed the IPAM on Azure AKS, but getting "The resource principal named api:"service principle name" was not found in the tenant named "tenant id". This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant".

Currently "reader" permission at the tenant level and contributor permission at the subscription on the azure for the UI service principle has been assigned, so could you please assist here if there is anything at the permission level needs to be changed.

[DCMattyG]

Hi @cloudtech555, when you say "deployed", can you explain more how you created the App Registrations that sit behind the backend & frontend services?

The deployment script included in this repo does some complex setup to create the App Registrations, link them together, and expose the API endpoints which sounds like it's likely the problem here.

How were the App Registration created for your IPAM environment?

Additionally, the script will grant them Admin Consent which is required as the IPAM Engine runs on the backend, and as such cannot present user consent when the API's are being called.

If you have the App Registrations created correctly, were they granted Admin Consent?