-
Notifications
You must be signed in to change notification settings - Fork 260
Long running test pipeline for AKS swiftv2 clusters. #4098
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
sivakami-projects
wants to merge
36
commits into
Azure:master
from
sivakami-projects:long-running-pipeline-test-branch
Closed
Changes from 35 commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
d88efac
init swiftv2 pipeline for persistent tests on aks clusters.
dd6afba
Set default params.
dd57671
Update pipeline.yaml for Azure Pipelines
sivakami-projects 16d69a0
long running pipeline infra setup.
c52ad9f
Set depedencies for pipeline jobs.
8d69373
template for long running cluster.
ea3dfd8
set template.
fad64fd
set dependency for jobs.
004d50d
Change job name.
8a3b65b
Set job scripts.
ed0cf5d
set pipeline scripts with permissions.
ae0f918
set script path.
61859e7
set template params.
46e6148
Set pipeline template for long running clusters.
17b5deb
test change.
08c2665
set params.
0495b72
set params in pipeline scripts.
f4750f2
set cx vnet name.
b530c30
Create clusters parallely
ac3419e
create NSG.
b4b7fbb
Change dependency for creating nsg.
325d3f3
Update .pipelines/swiftv2-long-running/scripts/create_peerings.sh
sivakami-projects 3086d40
Update .pipelines/swiftv2-long-running/scripts/create_nsg.sh
sivakami-projects adb4448
Add success/error message for each resource creation.
b77b78d
Remove unused argument from template.
a0d21bc
Rename subnets. Changed NSG rules to prevent network connectivity bet…
4b13348
Private endpoints.
54eab34
Change pipeline template.
335ddc1
Set output variables.
1ba3585
private endpoint.
b600fa0
update private endpoint.
e15efde
create storage account.
2d124c0
disallow shared key access.
556d63c
change pipeline template.
3e502fa
Removed unused param.
85b4f2f
Link private endpoint dns to vnet a2 and vnet a3.
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| trigger: none | ||
|
|
||
| parameters: | ||
| - name: subscriptionId | ||
| displayName: "Azure Subscription ID" | ||
| type: string | ||
| default: "37deca37-c375-4a14-b90a-043849bd2bf1" | ||
|
|
||
| - name: location | ||
| displayName: "Deployment Region" | ||
| type: string | ||
| default: "centraluseuap" | ||
|
|
||
| - name: resourceGroupName | ||
| displayName: "Resource Group Name" | ||
| type: string | ||
| default: "long-run-$(Build.BuildId)" | ||
|
|
||
| - name: vmSkuDefault | ||
| displayName: "VM SKU for Default Node Pool" | ||
| type: string | ||
| default: "Standard_D2s_v3" | ||
|
|
||
| - name: vmSkuHighNIC | ||
| displayName: "VM SKU for High NIC Node Pool" | ||
| type: string | ||
| default: "Standard_D16s_v3" | ||
|
|
||
| - name: serviceConnection | ||
| displayName: "Azure Service Connection" | ||
| type: string | ||
| default: "Azure Container Networking - Standalone Test Service Connection" | ||
|
|
||
| extends: | ||
| template: template/long-running-pipeline-template.yaml | ||
| parameters: | ||
| subscriptionId: ${{ parameters.subscriptionId }} | ||
| location: ${{ parameters.location }} | ||
| resourceGroupName: ${{ parameters.resourceGroupName }} | ||
| vmSkuDefault: ${{ parameters.vmSkuDefault }} | ||
| vmSkuHighNIC: ${{ parameters.vmSkuHighNIC }} | ||
| serviceConnection: ${{ parameters.serviceConnection }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
| trap 'echo "[ERROR] Failed during Resource group or AKS cluster creation." >&2' ERR | ||
|
|
||
| SUBSCRIPTION_ID=$1 | ||
| LOCATION=$2 | ||
| RG=$3 | ||
| VM_SKU_DEFAULT=$4 | ||
| VM_SKU_HIGHNIC=$5 | ||
|
|
||
| echo "Subscription id: $SUBSCRIPTION_ID" | ||
| echo "Resource group: $RG" | ||
| echo "Location: $LOCATION" | ||
| echo "VM SKU (default): $VM_SKU_DEFAULT" | ||
| echo "VM SKU (high-NIC): $VM_SKU_HIGHNIC" | ||
| az account set --subscription "$SUBSCRIPTION_ID" | ||
|
|
||
| # Enable parallel cluster creation | ||
| create_cluster() { | ||
| local CLUSTER=$1 | ||
| echo "==> Creating AKS cluster: $CLUSTER" | ||
|
|
||
| az aks create -g "$RG" -n "$CLUSTER" -l "$LOCATION" \ | ||
| --network-plugin azure --node-count 1 \ | ||
| --node-vm-size "$VM_SKU_DEFAULT" \ | ||
| --enable-managed-identity --generate-ssh-keys \ | ||
| --load-balancer-sku standard --yes --only-show-errors | ||
|
|
||
| echo "==> Adding high-NIC nodepool to $CLUSTER" | ||
| az aks nodepool add -g "$RG" -n highnic \ | ||
| --cluster-name "$CLUSTER" --node-count 2 \ | ||
| --node-vm-size "$VM_SKU_HIGHNIC" --mode User --only-show-errors | ||
|
|
||
| echo "Finished AKS cluster: $CLUSTER" | ||
| } | ||
|
|
||
| # Run both clusters in parallel | ||
| create_cluster "aks-cluster-a" & | ||
| pid_a=$! | ||
|
|
||
| create_cluster "aks-cluster-b" & | ||
| pid_b=$! | ||
|
|
||
| # Wait for both to finish | ||
| wait $pid_a $pid_b | ||
|
|
||
| echo "AKS clusters created successfully!" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
| trap 'echo "[ERROR] Failed during NSG creation or rule setup." >&2' ERR | ||
|
|
||
| SUBSCRIPTION_ID=$1 | ||
| RG=$2 | ||
| LOCATION=$3 | ||
|
|
||
| VNET_A1="cx_vnet_a1" | ||
| SUBNET1_PREFIX="10.10.1.0/24" | ||
| SUBNET2_PREFIX="10.10.2.0/24" | ||
| NSG_NAME="${VNET_A1}-nsg" | ||
|
|
||
| echo "==> Creating Network Security Group: $NSG_NAME" | ||
| az network nsg create -g "$RG" -n "$NSG_NAME" -l "$LOCATION" --output none \ | ||
| && echo "[OK] NSG '$NSG_NAME' created." | ||
|
|
||
| echo "==> Creating NSG rule to DENY traffic from Subnet1 ($SUBNET1_PREFIX) to Subnet2 ($SUBNET2_PREFIX)" | ||
| az network nsg rule create \ | ||
| -g "$RG" \ | ||
| --nsg-name "$NSG_NAME" \ | ||
| -n deny-subnet1-to-subnet2 \ | ||
| --priority 100 \ | ||
| --source-address-prefixes "$SUBNET1_PREFIX" \ | ||
| --destination-address-prefixes "$SUBNET2_PREFIX" \ | ||
| --direction Inbound \ | ||
| --access Deny \ | ||
| --protocol "*" \ | ||
| --description "Deny all traffic from Subnet1 to Subnet2" \ | ||
| --output none \ | ||
| && echo "[OK] Deny rule from Subnet1 → Subnet2 created." | ||
|
|
||
| echo "==> Creating NSG rule to DENY traffic from Subnet2 ($SUBNET2_PREFIX) to Subnet1 ($SUBNET1_PREFIX)" | ||
| az network nsg rule create \ | ||
| -g "$RG" \ | ||
| --nsg-name "$NSG_NAME" \ | ||
| -n deny-subnet2-to-subnet1 \ | ||
| --priority 200 \ | ||
| --source-address-prefixes "$SUBNET2_PREFIX" \ | ||
| --destination-address-prefixes "$SUBNET1_PREFIX" \ | ||
| --direction Inbound \ | ||
| --access Deny \ | ||
| --protocol "*" \ | ||
| --description "Deny all traffic from Subnet2 to Subnet1" \ | ||
| --output none \ | ||
| && echo "[OK] Deny rule from Subnet2 → Subnet1 created." | ||
|
|
||
| echo "NSG '$NSG_NAME' created successfully with bidirectional isolation between Subnet1 and Subnet2." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
| trap 'echo "[ERROR] Failed during Private Endpoint or DNS setup." >&2' ERR | ||
|
|
||
| SUBSCRIPTION_ID=$1 | ||
| LOCATION=$2 | ||
| RG=$3 | ||
| SA1_NAME=$4 # Storage account 1 | ||
sivakami-projects marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| VNET_A1="cx_vnet_a1" | ||
| SUBNET_PE_A1="pe" | ||
| PE_NAME="${SA1_NAME}-pe" | ||
| PRIVATE_DNS_ZONE="privatelink.blob.core.windows.net" | ||
| LINK_NAME="${VNET_A1}-link" | ||
|
|
||
| # 1. Create Private DNS zone | ||
| echo "==> Creating Private DNS zone: $PRIVATE_DNS_ZONE" | ||
| az network private-dns zone create -g "$RG" -n "$PRIVATE_DNS_ZONE" --output none \ | ||
| && echo "[OK] DNS zone $PRIVATE_DNS_ZONE created." | ||
|
|
||
| # 2. Link DNS zone to VNet | ||
| echo "==> Linking DNS zone $PRIVATE_DNS_ZONE to VNet $VNET_A1" | ||
| az network private-dns link vnet create \ | ||
| -g "$RG" -n "$LINK_NAME" \ | ||
| --zone-name "$PRIVATE_DNS_ZONE" \ | ||
| --virtual-network "$VNET_A1" \ | ||
| --registration-enabled false --output none \ | ||
| && echo "[OK] Linked DNS zone to $VNET_A1." | ||
sivakami-projects marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # 3. Create Private Endpoint | ||
| echo "==> Creating Private Endpoint for Storage Account: $SA1_NAME" | ||
| SA1_ID=$(az storage account show -g "$RG" -n "$SA1_NAME" --query id -o tsv) | ||
| az network private-endpoint create \ | ||
| -g "$RG" -n "$PE_NAME" -l "$LOCATION" \ | ||
| --vnet-name "$VNET_A1" --subnet "$SUBNET_PE_A1" \ | ||
| --private-connection-resource-id "$SA1_ID" \ | ||
| --group-id blob \ | ||
| --connection-name "${PE_NAME}-conn" \ | ||
| --output none \ | ||
| && echo "[OK] Private Endpoint $PE_NAME created for $SA1_NAME." | ||
23 changes: 23 additions & 0 deletions
23
.pipelines/swiftv2-long-running/scripts/create_peerings.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
| trap 'echo "[ERROR] Failed during VNet peering creation." >&2' ERR | ||
|
|
||
| RG=$1 | ||
| VNET_A1="cx_vnet_a1" | ||
| VNET_A2="cx_vnet_a2" | ||
| VNET_A3="cx_vnet_a3" | ||
| VNET_B1="cx_vnet_b1" | ||
|
|
||
| peer_two_vnets() { | ||
| local rg="$1"; local v1="$2"; local v2="$3"; local name12="$4"; local name21="$5" | ||
| echo "==> Peering $v1 <-> $v2" | ||
| az network vnet peering create -g "$rg" -n "$name12" --vnet-name "$v1" --remote-vnet "$v2" --allow-vnet-access --output none \ | ||
| && echo "Created peering $name12" | ||
| az network vnet peering create -g "$rg" -n "$name21" --vnet-name "$v2" --remote-vnet "$v1" --allow-vnet-access --output none \ | ||
| && echo "Created peering $name21" | ||
| } | ||
|
|
||
| peer_two_vnets "$RG" "$VNET_A1" "$VNET_A2" "A1-to-A2" "A2-to-A1" | ||
| peer_two_vnets "$RG" "$VNET_A2" "$VNET_A3" "A2-to-A3" "A3-to-A2" | ||
| peer_two_vnets "$RG" "$VNET_A1" "$VNET_A3" "A1-to-A3" "A3-to-A1" | ||
| echo "VNet peerings created successfully." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
| trap 'echo "[ERROR] Failed during Storage Account creation." >&2' ERR | ||
|
|
||
| SUBSCRIPTION_ID=$1 | ||
| LOCATION=$2 | ||
| RG=$3 | ||
|
|
||
| RAND=$(openssl rand -hex 4) | ||
| SA1="sa1${RAND}" | ||
| SA2="sa2${RAND}" | ||
|
|
||
| # Set subscription context | ||
| az account set --subscription "$SUBSCRIPTION_ID" | ||
|
|
||
| # Create storage accounts | ||
| for SA in "$SA1" "$SA2"; do | ||
| echo "==> Creating storage account $SA" | ||
| az storage account create \ | ||
| --name "$SA" \ | ||
| --resource-group "$RG" \ | ||
| --location "$LOCATION" \ | ||
| --sku Standard_LRS \ | ||
| --kind StorageV2 \ | ||
| --allow-blob-public-access false \ | ||
| --allow-shared-key-access false \ | ||
| --https-only true \ | ||
| --min-tls-version TLS1_2 \ | ||
| --query "name" -o tsv \ | ||
| && echo "Storage account $SA created successfully." | ||
| done | ||
|
|
||
| echo "All storage accounts created successfully." | ||
|
|
||
| # Set pipeline output variables | ||
| set +x | ||
| echo "##vso[task.setvariable variable=StorageAccount1;isOutput=true]$SA1" | ||
| echo "##vso[task.setvariable variable=StorageAccount2;isOutput=true]$SA2" | ||
| set -x | ||
sivakami-projects marked this conversation as resolved.
Show resolved
Hide resolved
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
| trap 'echo "[ERROR] Failed while creating VNets or subnets. Check Azure CLI logs above." >&2' ERR | ||
|
|
||
| SUBSCRIPTION_ID=$1 | ||
| LOCATION=$2 | ||
| RG=$3 | ||
|
|
||
| az account set --subscription "$SUBSCRIPTION_ID" | ||
|
|
||
| # VNets and subnets | ||
| VNET_A1="cx_vnet_a1" | ||
| VNET_A2="cx_vnet_a2" | ||
| VNET_A3="cx_vnet_a3" | ||
| VNET_B1="cx_vnet_b1" | ||
|
|
||
| A1_S1="10.10.1.0/24" | ||
| A1_S2="10.10.2.0/24" | ||
| A1_PE="10.10.100.0/24" | ||
|
|
||
| A2_MAIN="10.11.1.0/24" | ||
| A2_PE="10.11.100.0/24" | ||
|
|
||
| A3_MAIN="10.12.1.0/24" | ||
| A3_PE="10.12.100.0/24" | ||
|
|
||
| B1_MAIN="10.20.1.0/24" | ||
|
|
||
| # A1 | ||
| az network vnet create -g "$RG" -n "$VNET_A1" --address-prefix 10.10.0.0/16 --subnet-name s1 --subnet-prefix "$A1_S1" -l "$LOCATION" --output none \ | ||
| && echo "Created $VNET_A1 with subnet s1" | ||
| az network vnet subnet create -g "$RG" --vnet-name "$VNET_A1" -n s2 --address-prefix "$A1_S2" --output none \ | ||
| && echo "Created $VNET_A1 with subnet s2" | ||
| az network vnet subnet create -g "$RG" --vnet-name "$VNET_A1" -n pe --address-prefix "$A1_PE" --output none \ | ||
| && echo "Created $VNET_A1 with subnet pe" | ||
|
|
||
| # A2 | ||
| az network vnet create -g "$RG" -n "$VNET_A2" --address-prefix 10.11.0.0/16 --subnet-name s1 --subnet-prefix "$A2_MAIN" -l "$LOCATION" --output none \ | ||
| && echo "Created $VNET_A2 with subnet s1" | ||
| az network vnet subnet create -g "$RG" --vnet-name "$VNET_A2" -n pe --address-prefix "$A2_PE" --output none \ | ||
| && echo "Created $VNET_A2 with subnet pe" | ||
|
|
||
| # A3 | ||
| az network vnet create -g "$RG" -n "$VNET_A3" --address-prefix 10.12.0.0/16 --subnet-name s1 --subnet-prefix "$A3_MAIN" -l "$LOCATION" --output none \ | ||
| && echo "Created $VNET_A3 with subnet s1" | ||
| az network vnet subnet create -g "$RG" --vnet-name "$VNET_A3" -n pe --address-prefix "$A3_PE" --output none \ | ||
| && echo "Created $VNET_A3 with subnet pe" | ||
|
|
||
| # B1 | ||
| az network vnet create -g "$RG" -n "$VNET_B1" --address-prefix 10.20.0.0/16 --subnet-name s1 --subnet-prefix "$B1_MAIN" -l "$LOCATION" --output none \ | ||
| && echo "Created $VNET_B1 with subnet s1" | ||
|
|
||
| echo "All VNets and subnets created successfully." |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you'll want to change the branch this PR is on
Right now, it's on
sivakami-project:long-running-pipeline-test-branch, which is a fork of this repoI don't think the PR checks and all that let the users run pipeline tests from forks anymore, it has to be a branch from this repo itself
Kind of like you had in #4092, that branch was just
long-running-pipelinein this repoThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#4099