Skip to content

Add public ips with service tags for LBs during cluster creation #2821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 54 commits into from
Closed

Add public ips with service tags for LBs during cluster creation #2821

Changes from 36 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
b596518
Create an outbound public ip for LB/Cilium cluster
shubham-pathak-03 Jun 20, 2024
f367851
create and attach public ip for cilium e2e cluster
shubham-pathak-03 Jun 25, 2024
29b1a92
Test cluster independent ip creation
shubham-pathak-03 Jun 25, 2024
e7cd70d
Test outbound public ip creation
shubham-pathak-03 Jun 25, 2024
3697054
Test outbound public ip creation
shubham-pathak-03 Jun 25, 2024
ec41733
Test outbound public ip creation with azcli creds
shubham-pathak-03 Jun 25, 2024
3e6f5c8
Test outbound public ip creation with azcli creds
shubham-pathak-03 Jun 25, 2024
87ea244
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 25, 2024
651a405
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 25, 2024
d390d45
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 25, 2024
edf70ee
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 25, 2024
cb2fa8f
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 25, 2024
2d17d5d
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 25, 2024
00549ce
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 25, 2024
bf50b77
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 25, 2024
17807c6
Merge branch 'master' into spathak/add-service-tag
shubham-pathak-03 Jun 25, 2024
19b5600
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 26, 2024
460a1cf
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 26, 2024
9a332d0
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 26, 2024
73228a4
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 26, 2024
f1c8247
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 26, 2024
926acd6
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 26, 2024
3cbbf71
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 27, 2024
99d5d42
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 27, 2024
aa70640
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 27, 2024
28cba9e
Add lb ip creation alias call for clusters in makefile
shubham-pathak-03 Jun 27, 2024
4a797f7
Add managed identity to public ip/load balancer
shubham-pathak-03 Jun 27, 2024
74efd8c
Add managed identity to public ip/load balancer
shubham-pathak-03 Jun 27, 2024
c631b0c
Test wo managed identity
shubham-pathak-03 Jun 27, 2024
e016323
Test wo managed identity
shubham-pathak-03 Jul 1, 2024
aec6803
Add Public ip to one cluster
shubham-pathak-03 Jul 1, 2024
273d39b
Add public to all cluster creations
shubham-pathak-03 Jul 1, 2024
d6de754
Add public ip
shubham-pathak-03 Jul 1, 2024
04ee4eb
Fix spacing
shubham-pathak-03 Jul 1, 2024
5fa58d0
Fix spacing
shubham-pathak-03 Jul 1, 2024
875354e
Merge branch 'master' into spathak/add-service-tag
shubham-pathak-03 Jul 1, 2024
6ab97b0
Merge branch 'master' into spathak/add-service-tag
shubham-pathak-03 Jul 11, 2024
748eb20
Add LB to win cni v1 cluster
shubham-pathak-03 Jul 11, 2024
bac845d
Add LB to win cni v1 cluster
shubham-pathak-03 Jul 11, 2024
2dc3613
Add ip-tag variable to makefile
shubham-pathak-03 Jul 11, 2024
5c9924f
Add ip-tag variable to makefile
shubham-pathak-03 Jul 11, 2024
6a285dd
Add ip-tag variable to makefile
shubham-pathak-03 Jul 11, 2024
e1e15cb
Add ip-tag variable to makefile
shubham-pathak-03 Jul 11, 2024
57854f8
Add ip-tag variable to makefile
shubham-pathak-03 Jul 11, 2024
d110100
Add ip-tag variable to makefile
shubham-pathak-03 Jul 11, 2024
f10249e
Add ip-tag variable to makefile
shubham-pathak-03 Jul 11, 2024
1d1d78a
Add ip-tag variable to makefile
shubham-pathak-03 Jul 11, 2024
88a8a92
Add ip-tag variable to makefile
shubham-pathak-03 Jul 12, 2024
64300df
updated service tag to 'DelegatedNetworkControllerTest'
k-routhu Nov 1, 2024
6d50417
Merge branch 'master' into spathak/add-service-tag
k-routhu Dec 13, 2024
9189f09
create public IP as target
k-routhu Dec 13, 2024
ad1101f
add ipv6 public ips to dualstack
k-routhu Dec 13, 2024
ca8cc6e
updated v6 ip
k-routhu Dec 13, 2024
b43dc44
remove space
k-routhu Dec 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 121 additions & 15 deletions hack/aks/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ SUB ?= $(AZURE_SUBSCRIPTION)
CLUSTER ?= $(USER)-$(REGION)
GROUP ?= $(CLUSTER)
VNET ?= $(CLUSTER)

##@ Help

help: ## Display this help
Expand Down Expand Up @@ -88,21 +87,29 @@ overlay-net-up: ## Create vnet, nodenet subnets
$(AZCLI) network vnet create -g $(GROUP) -l $(REGION) --name $(VNET) --address-prefixes 10.0.0.0/8 -o none
$(AZCLI) network vnet subnet create -g $(GROUP) --vnet-name $(VNET) --name nodenet --address-prefix 10.10.0.0/16 -o none


##@ AKS Clusters

byocni-up: swift-byocni-up ## Alias to swift-byocni-up
cilium-up: swift-cilium-up ## Alias to swift-cilium-up
up: swift-up ## Alias to swift-up

overlay-byocni-up: rg-up overlay-net-up ## Brings up an Overlay BYO CNI cluster
$(AZCLI) network public-ip create --name "overlay-cni-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "overlay-cni-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku standard \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/overlay-cni-public-ip \
--network-plugin none \
--network-plugin-mode overlay \
--pod-cidr 192.168.0.0/16 \
Expand All @@ -116,13 +123,22 @@ endif
@$(MAKE) set-kubeconf

overlay-byocni-nokubeproxy-up: rg-up overlay-net-up ## Brings up an Overlay BYO CNI cluster without kube-proxy
$(AZCLI) network public-ip create --name "overlay-cni-no-kube-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "overlay-cni-no-kube-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/overlay-cni-no-kube-public-ip \
--network-plugin none \
--network-plugin-mode overlay \
--pod-cidr 192.168.0.0/16 \
Expand All @@ -133,13 +149,22 @@ overlay-byocni-nokubeproxy-up: rg-up overlay-net-up ## Brings up an Overlay BYO
@$(MAKE) set-kubeconf

overlay-cilium-up: rg-up overlay-net-up ## Brings up an Overlay Cilium cluster
$(AZCLI) network public-ip create --name "overlay-cillium-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "overlay-cillium-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/overlay-cillium-public-ip \
--network-plugin azure \
--network-dataplane cilium \
--network-plugin-mode overlay \
Expand All @@ -150,13 +175,22 @@ overlay-cilium-up: rg-up overlay-net-up ## Brings up an Overlay Cilium cluster
@$(MAKE) set-kubeconf

overlay-up: rg-up overlay-net-up ## Brings up an Overlay AzCNI cluster
$(AZCLI) network public-ip create --name "overlay-azcni-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "overlay-azcni-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/overlay-azcni-public-ip \
--network-plugin azure \
--network-plugin-mode overlay \
--pod-cidr 192.168.0.0/16 \
Expand All @@ -166,13 +200,22 @@ overlay-up: rg-up overlay-net-up ## Brings up an Overlay AzCNI cluster
@$(MAKE) set-kubeconf

swift-byocni-up: rg-up swift-net-up ## Bring up a SWIFT BYO CNI cluster
$(AZCLI) network public-ip create --name "swift-byo-cni-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "swift-byo-cni-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku standard \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/swift-byo-cni-public-ip \
--network-plugin none \
--vnet-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/nodenet \
--pod-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/podnet \
Expand All @@ -184,14 +227,23 @@ ifeq ($(OS),windows)
endif
@$(MAKE) set-kubeconf

swift-byocni-nokubeproxy-up: rg-up swift-net-up ## Bring up a SWIFT BYO CNI cluster without kube-proxy
swift-byocni-nokubeproxy-up: rg-up swift-net-up ## Bring up a SWIFT BYO CNI cluster without kube-proxy, add managed identity and public ip
$(AZCLI) network public-ip create --name "swift-byo-nok-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "swift-byo-nok-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/swift-byo-nok-public-ip \
--network-plugin none \
--vnet-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/nodenet \
--pod-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/podnet \
Expand All @@ -202,13 +254,22 @@ swift-byocni-nokubeproxy-up: rg-up swift-net-up ## Bring up a SWIFT BYO CNI clus
@$(MAKE) set-kubeconf

swift-cilium-up: rg-up swift-net-up ## Bring up a SWIFT Cilium cluster
$(AZCLI) network public-ip create --name "swift-cilium-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "swift-cilium-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/swift-cilium-public-ip \
--network-plugin azure \
--network-dataplane cilium \
--aks-custom-headers AKSHTTPCustomFeatures=Microsoft.ContainerService/CiliumDataplanePreview \
Expand All @@ -219,13 +280,22 @@ swift-cilium-up: rg-up swift-net-up ## Bring up a SWIFT Cilium cluster
@$(MAKE) set-kubeconf

swift-up: rg-up swift-net-up ## Bring up a SWIFT AzCNI cluster
$(AZCLI) network public-ip create --name "swift-azcni-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "swift-azcni-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/swift-azcni-public-ip \
--network-plugin azure \
--vnet-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/nodenet \
--pod-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/podnet \
Expand Down Expand Up @@ -259,13 +329,22 @@ swiftv2-dummy-cluster-up: rg-up swift-net-up ## Bring up a SWIFT AzCNI cluster
# These AKS clusters can only be created in a limited subscription listed here:
# https://dev.azure.com/msazure/CloudNativeCompute/_git/aks-rp?path=/resourceprovider/server/microsoft.com/containerservice/flags/network_flags.go&version=GBmaster&line=134&lineEnd=135&lineStartColumn=1&lineEndColumn=1&lineStyle=plain&_a=contents
vnetscale-swift-byocni-up: rg-up vnetscale-swift-net-up ## Bring up a Vnet Scale SWIFT BYO CNI cluster
$(AZCLI) network public-ip create --name "vnetscale-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "vnetscale-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/vnetscale-public-ip \
--network-plugin none \
--vnet-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/nodenet \
--pod-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/podnet \
Expand All @@ -275,13 +354,22 @@ vnetscale-swift-byocni-up: rg-up vnetscale-swift-net-up ## Bring up a Vnet Scale
@$(MAKE) set-kubeconf

vnetscale-swift-byocni-nokubeproxy-up: rg-up vnetscale-swift-net-up ## Bring up a Vnet Scale SWIFT BYO CNI cluster without kube-proxy
$(AZCLI) network public-ip create --name "vnetscale-nok-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "vnetscale-nok-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/vnetscale-nok-public-ip \
--network-plugin none \
--vnet-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/nodenet \
--pod-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/podnet \
Expand All @@ -292,13 +380,22 @@ vnetscale-swift-byocni-nokubeproxy-up: rg-up vnetscale-swift-net-up ## Bring up
@$(MAKE) set-kubeconf

vnetscale-swift-cilium-up: rg-up vnetscale-swift-net-up ## Bring up a Vnet Scale SWIFT Cilium cluster
$(AZCLI) network public-ip create --name "vnetscale-cilium-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "vnetscale-cilium-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/vnetscale-cilium-public-ip \
--network-plugin azure \
--network-dataplane cilium \
--aks-custom-headers AKSHTTPCustomFeatures=Microsoft.ContainerService/CiliumDataplanePreview \
Expand All @@ -309,13 +406,22 @@ vnetscale-swift-cilium-up: rg-up vnetscale-swift-net-up ## Bring up a Vnet Scale
@$(MAKE) set-kubeconf

vnetscale-swift-up: rg-up vnetscale-swift-net-up ## Bring up a Vnet Scale SWIFT AzCNI cluster
$(AZCLI) network public-ip create --name "vnetscale-azcni-public-ip" \
--resource-group $(GROUP) \
--allocation-method Static \
--ip-tags "FirstPartyUsage=/TestTagging-O365" \
--location $(REGION) \
--sku Standard \
--tier Regional \
--version IPv4
$(AZCLI) network public-ip show --resource-group $(GROUP) --name "vnetscale-azcni-public-ip" --query "ipAddress" -o tsv
$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
--auto-upgrade-channel $(AUTOUPGRADE) \
--node-os-upgrade-channel $(NODEUPGRADE) \
--kubernetes-version $(K8S_VER) \
--node-count $(NODE_COUNT) \
--node-vm-size $(VM_SIZE) \
--load-balancer-sku basic \
--load-balancer-outbound-ips /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/publicIPAddresses/vnetscale-azcni-public-ip \
--network-plugin azure \
--vnet-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/nodenet \
--pod-subnet-id /subscriptions/$(SUB)/resourceGroups/$(GROUP)/providers/Microsoft.Network/virtualNetworks/$(VNET)/subnets/podnet \
Expand Down