Skip to content

Commit 1514d95

Browse files
jpayne3506pjohnst5
jpayne3506
and
pjohnst5
authored
ci: [CNI] Move Nightly Cilium Pipeline test to ACN (#1963)
* CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: Eliminating duplicate lines * ci: Add update permission for ciliumidentity * fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: File Directory * style: Comments * Addressing Comments --------- Co-authored-by: Paul Johnston <[email protected]>
1 parent 940a7a7 commit 1514d95

File tree

16 files changed

+531
-16
lines changed

16 files changed

+531
-16
lines changed

‎.pipelines/cni/cilium/cilium-cni-load-test.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,9 @@ stages:
4848
echo "install Cilium onto Overlay Cluster"
4949
kubectl apply -f test/integration/manifests/cilium/cilium-agent
5050
kubectl apply -f test/integration/manifests/cilium/cilium-operator
51+
# Passes Cilium image to daemonset and deployment
52+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/daemonset.yaml | kubectl apply -f -
53+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/deployment.yaml | kubectl apply -f -
5154
kubectl get po -owide -A
5255
echo "deploy ip-masq-agent for overlay"
5356
kubectl create -f test/integration/manifests/ip-masq-agent/ip-masq-agent.yaml --validate=false

‎.pipelines/cni/cilium/nightly-release-test.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
pr: none
2+
trigger: none
3+
4+
stages:
5+
- stage: init
6+
displayName: "Build and Push Cilium Image"
7+
jobs:
8+
- job: build_and_push_cilium_image
9+
steps:
10+
- bash: |
11+
set -ex
12+
cd .pipelines/
13+
git clone https://github.com/cilium/cilium.git
14+
cd cilium
15+
make docker-cilium-image
16+
make docker-operator-generic-image
17+
name: BuildCiliumImage
18+
displayName: "Build Cilium Image"
19+
- task: Docker@2
20+
displayName: Login
21+
inputs:
22+
containerRegistry: $(CONTAINER_REGISTRY)
23+
command: "login"
24+
- task: AzureCLI@1
25+
inputs:
26+
azureSubscription: $(TEST_SUB_SERVICE_CONNECTION)
27+
scriptLocation: "inlineScript"
28+
inlineScript: |
29+
set -ex
30+
docker tag quay.io/cilium/cilium:$(DOCKER_IMAGE_TAG) $(ACR).azurecr.io/cilium/cilium:$(DOCKER_IMAGE_TAG)
31+
docker tag quay.io/cilium/operator-generic:$(DOCKER_IMAGE_TAG) $(ACR).azurecr.io/cilium/operator-generic:$(DOCKER_IMAGE_TAG)
32+
docker push $(ACR).azurecr.io/cilium/cilium:$(DOCKER_IMAGE_TAG)
33+
docker push $(ACR).azurecr.io/cilium/operator-generic:$(DOCKER_IMAGE_TAG)
34+
name: "PushCiliumImage"
35+
displayName: "Push Cilium Image"
36+
- task: Docker@2
37+
displayName: Logout
38+
inputs:
39+
containerRegistry: $(CONTAINER_REGISTRY)
40+
command: "logout"
41+
- stage: cilium_nightly
42+
displayName: E2E - Cilium Nightly
43+
variables:
44+
GOPATH: "$(Agent.TempDirectory)/go" # Go workspace path
45+
GOBIN: "$(GOPATH)/bin" # Go binaries path
46+
modulePath: "$(GOPATH)/src/github.com/Azure/azure-container-networking"
47+
jobs:
48+
- job: cilium_nightly
49+
steps:
50+
- template: ../../singletenancy/cilium-overlay/cilium-overlay-e2e-step-template.yaml
51+
parameters:
52+
name: "cilium_nightly"
53+
testDropgz: ""
54+
clusterName: "ciliumnightly"

‎.pipelines/singletenancy/cilium-overlay/cilium-overlay-e2e-step-template.yaml‎

Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -45,12 +45,17 @@ steps:
4545
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
4646
kubectl cluster-info
4747
kubectl get po -owide -A
48+
if [ "$CILIUM_VERSION_TAG" = "cilium-nightly-pipeline" ]; then FILE_PATH=-nightly && echo "Running nightly"; fi
4849
echo "deploy Cilium ConfigMap"
49-
kubectl apply -f cilium/configmap.yaml
50-
kubectl apply -f test/integration/manifests/cilium/cilium-config.yaml
51-
echo "install Cilium onto Overlay Cluster"
52-
kubectl apply -f test/integration/manifests/cilium/cilium-agent
53-
kubectl apply -f test/integration/manifests/cilium/cilium-operator
50+
kubectl apply -f cilium/configmap.yaml
51+
kubectl apply -f test/integration/manifests/cilium/cilium${FILE_PATH}-config.yaml
52+
echo "install Cilium ${CILIUM_VERSION_TAG}"
53+
# Passes Cilium image to daemonset and deployment
54+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/daemonset.yaml | kubectl apply -f -
55+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/deployment.yaml | kubectl apply -f -
56+
# Use different file directories for nightly and current cilium version
57+
kubectl apply -f test/integration/manifests/cilium/cilium${FILE_PATH}-agent
58+
kubectl apply -f test/integration/manifests/cilium/cilium${FILE_PATH}-operator
5459
kubectl get po -owide -A
5560
name: "installCilium"
5661
displayName: "Install Cilium on AKS Overlay"
@@ -91,7 +96,14 @@ steps:
9196
kubectl create configmap config-reconcile.yaml
9297
cd ../../../..
9398
kubectl get po -owide -A
94-
sudo -E env "PATH=$PATH" make test-integration CNS_VERSION=$(make cns-version) CNI_DROPGZ_VERSION=$(make cni-dropgz-version) INSTALL_CNS=true INSTALL_OVERLAY=true TEST_DROPGZ=${{ parameters.testDropgz }}
99+
# Nightly does not build images per commit. Will use existing image.
100+
if [ "$CILIUM_VERSION_TAG" = "cilium-nightly-pipeline" ]
101+
then
102+
CNS=v1.5.3 DROPGZ=v0.0.4 && echo "Running nightly"
103+
else
104+
CNS=$(make cns-version) DROPGZ=$(make cni-dropgz-version)
105+
fi
106+
sudo -E env "PATH=$PATH" make test-integration CNS_VERSION=${CNS} CNI_DROPGZ_VERSION=${DROPGZ} INSTALL_CNS=true INSTALL_OVERLAY=true TEST_DROPGZ=${{ parameters.testDropgz }}
95107
retryCountOnTaskFailure: 3
96108
name: "aziliumTest"
97109
displayName: "Run Azilium E2E on AKS Overlay"
@@ -113,6 +125,15 @@ steps:
113125
pathtoPublish: "$(Build.ArtifactStagingDirectory)/test-output"
114126
condition: always()
115127

128+
- script: |
129+
kubectl get pods -A
130+
echo "Waiting < 2 minutes for cilium to be ready"
131+
# Ensure Cilium is ready Xm\Xs
132+
cilium status --wait --wait-duration 2m
133+
retryCountOnTaskFailure: 3
134+
name: "CiliumStatus"
135+
displayName: "Cilium Status"
136+
116137
- script: |
117138
echo "Run Service Conformance E2E"
118139
export PATH=${PATH}:/usr/local/bin/gsutil

‎.pipelines/singletenancy/cilium/cilium-e2e-step-template.yaml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,9 @@ steps:
4848
echo "deploy Cilium ConfigMap"
4949
kubectl apply -f cilium/configmap.yaml
5050
kubectl apply -f test/integration/manifests/cilium/cilium-config.yaml
51-
echo "install Cilium"
51+
echo "install Cilium ${CILIUM_VERSION_TAG}"
52+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/daemonset.yaml | kubectl apply -f -
53+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/deployment.yaml | kubectl apply -f -
5254
kubectl apply -f test/integration/manifests/cilium/cilium-agent
5355
kubectl apply -f test/integration/manifests/cilium/cilium-operator
5456
kubectl get po -owide -A
@@ -107,6 +109,15 @@ steps:
107109
pathtoPublish: "$(Build.ArtifactStagingDirectory)/test-output"
108110
condition: always()
109111

112+
- script: |
113+
kubectl get pods -A
114+
echo "Waiting < 2 minutes for cilium to be ready"
115+
# Ensure Cilium is ready Xm\Xs
116+
cilium status --wait --wait-duration 2m
117+
retryCountOnTaskFailure: 3
118+
name: "CiliumStatus"
119+
displayName: "Cilium Status"
120+
110121
- script: |
111122
echo "Run Service Conformance E2E"
112123
export PATH=${PATH}:/usr/local/bin/gsutil

‎test/integration/manifests/cilium/cilium-agent/clusterrole.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,13 @@ rules:
7171
- create
7272
- apiGroups:
7373
- cilium.io
74+
# To synchronize garbage collection of such resources
75+
resources:
76+
- ciliumidentities
77+
verbs:
78+
- update
79+
- apiGroups:
80+
- cilium.io
7481
resources:
7582
- ciliumendpoints
7683
verbs:

‎test/integration/manifests/cilium/cilium-nightly-agent/clusterrole.yaml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
apiVersion: rbac.authorization.k8s.io/v1
2+
kind: ClusterRole
3+
metadata:
4+
name: cilium
5+
rules:
6+
- apiGroups:
7+
- networking.k8s.io
8+
resources:
9+
- networkpolicies
10+
verbs:
11+
- get
12+
- list
13+
- watch
14+
- apiGroups:
15+
- discovery.k8s.io
16+
resources:
17+
- endpointslices
18+
verbs:
19+
- get
20+
- list
21+
- watch
22+
- apiGroups:
23+
- ""
24+
resources:
25+
- namespaces
26+
- services
27+
- pods
28+
- endpoints
29+
- nodes
30+
verbs:
31+
- get
32+
- list
33+
- watch
34+
- apiGroups:
35+
- apiextensions.k8s.io
36+
resources:
37+
- customresourcedefinitions
38+
verbs:
39+
- list
40+
- watch
41+
# This is used when validating policies in preflight. This will need to stay
42+
# until we figure out how to avoid "get" inside the preflight, and then
43+
# should be removed ideally.
44+
- get
45+
- apiGroups:
46+
- cilium.io
47+
resources:
48+
#Naming changed from ciliumbgploadbalancerippools
49+
- ciliumloadbalancerippools
50+
- ciliumbgppeeringpolicies
51+
- ciliumclusterwideenvoyconfigs
52+
- ciliumclusterwidenetworkpolicies
53+
- ciliumegressgatewaypolicies
54+
- ciliumendpoints
55+
- ciliumendpointslices
56+
- ciliumenvoyconfigs
57+
- ciliumidentities
58+
- ciliumlocalredirectpolicies
59+
- ciliumnetworkpolicies
60+
- ciliumnodes
61+
- ciliumnodeconfigs
62+
#Added in 1.14.0 snapshot 2
63+
- ciliumcidrgroups
64+
verbs:
65+
- list
66+
- watch
67+
- apiGroups:
68+
- cilium.io
69+
resources:
70+
- ciliumidentities
71+
- ciliumendpoints
72+
- ciliumnodes
73+
verbs:
74+
- create
75+
- apiGroups:
76+
- cilium.io
77+
resources:
78+
- ciliumidentities
79+
verbs:
80+
- update
81+
- apiGroups:
82+
- cilium.io
83+
resources:
84+
- ciliumendpoints
85+
verbs:
86+
- delete
87+
- get
88+
- apiGroups:
89+
- cilium.io
90+
resources:
91+
- ciliumnodes
92+
- ciliumnodes/status
93+
verbs:
94+
- get
95+
- update
96+
- apiGroups:
97+
- cilium.io
98+
resources:
99+
- ciliumnetworkpolicies/status
100+
- ciliumclusterwidenetworkpolicies/status
101+
- ciliumendpoints/status
102+
- ciliumendpoints
103+
verbs:
104+
- patch

‎test/integration/manifests/cilium/cilium-nightly-agent/clusterrolebinding.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: rbac.authorization.k8s.io/v1
2+
kind: ClusterRoleBinding
3+
metadata:
4+
name: cilium
5+
roleRef:
6+
apiGroup: rbac.authorization.k8s.io
7+
kind: ClusterRole
8+
name: cilium
9+
subjects:
10+
- kind: ServiceAccount
11+
name: "cilium"
12+
namespace: kube-system

‎test/integration/manifests/cilium/cilium-nightly-agent/serviceaccount.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
apiVersion: v1
2+
kind: ServiceAccount
3+
metadata:
4+
name: "cilium"
5+
namespace: kube-system
6+

‎test/integration/manifests/cilium/cilium-nightly-config.yaml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
apiVersion: v1
2+
data:
3+
agent-not-ready-taint-key: node.cilium.io/agent-not-ready
4+
arping-refresh-period: 30s
5+
auto-direct-node-routes: "false"
6+
bpf-lb-external-clusterip: "false"
7+
bpf-lb-map-max: "65536"
8+
bpf-lb-mode: snat
9+
bpf-map-dynamic-size-ratio: "0.0025"
10+
bpf-policy-map-max: "16384"
11+
bpf-root: /sys/fs/bpf
12+
cgroup-root: /run/cilium/cgroupv2
13+
cilium-endpoint-gc-interval: 5m0s
14+
cluster-id: "0"
15+
cluster-name: default
16+
debug: "false"
17+
disable-cnp-status-updates: "true"
18+
disable-endpoint-crd: "false"
19+
enable-auto-protect-node-port-range: "true"
20+
enable-bgp-control-plane: "false"
21+
enable-bpf-clock-probe: "true"
22+
enable-endpoint-health-checking: "false"
23+
enable-endpoint-routes: "true"
24+
enable-health-check-nodeport: "true"
25+
enable-health-checking: "true"
26+
enable-host-legacy-routing: "true"
27+
enable-hubble: "false"
28+
enable-ipv4: "true"
29+
enable-ipv4-masquerade: "false"
30+
enable-ipv6: "false"
31+
enable-ipv6-masquerade: "false"
32+
enable-k8s-terminating-endpoint: "true"
33+
enable-l2-neigh-discovery: "true"
34+
enable-l7-proxy: "false"
35+
enable-local-node-route: "false"
36+
enable-local-redirect-policy: "false"
37+
enable-metrics: "true"
38+
enable-policy: default
39+
enable-remote-node-identity: "true"
40+
enable-session-affinity: "true"
41+
enable-svc-source-range-check: "true"
42+
enable-vtep: "false"
43+
enable-well-known-identities: "false"
44+
enable-xt-socket-fallback: "true"
45+
identity-allocation-mode: crd
46+
install-iptables-rules: "true"
47+
install-no-conntrack-iptables-rules: "false"
48+
ipam: delegated-plugin
49+
kube-proxy-replacement: strict
50+
kube-proxy-replacement-healthz-bind-address: ""
51+
local-router-ipv4: 169.254.23.0
52+
metrics: +cilium_bpf_map_pressure
53+
monitor-aggregation: medium
54+
monitor-aggregation-flags: all
55+
monitor-aggregation-interval: 5s
56+
node-port-bind-protection: "true"
57+
nodes-gc-interval: 5m0s
58+
operator-api-serve-addr: 127.0.0.1:9234
59+
operator-prometheus-serve-addr: :9963
60+
preallocate-bpf-maps: "false"
61+
procfs: /host/proc
62+
prometheus-serve-addr: :9962
63+
remove-cilium-node-taints: "true"
64+
set-cilium-is-up-condition: "true"
65+
sidecar-istio-proxy-image: cilium/istio_proxy
66+
synchronize-k8s-nodes: "true"
67+
tofqdns-dns-reject-response-code: refused
68+
tofqdns-enable-dns-compression: "true"
69+
tofqdns-endpoint-max-ip-per-hostname: "50"
70+
tofqdns-idle-connection-grace-period: 0s
71+
tofqdns-max-deferred-connection-deletes: "10000"
72+
tofqdns-min-ttl: "3600"
73+
tofqdns-proxy-response-max-delay: 100ms
74+
#Replaces tunnel: disabled in v1.15
75+
routing-mode: "native"
76+
unmanaged-pod-watcher-interval: "15"
77+
vtep-cidr: ""
78+
vtep-endpoint: ""
79+
vtep-mac: ""
80+
vtep-mask: ""
81+
kind: ConfigMap
82+
metadata:
83+
annotations:
84+
meta.helm.sh/release-name: cilium
85+
meta.helm.sh/release-namespace: kube-system
86+
labels:
87+
app.kubernetes.io/managed-by: Helm
88+
name: cilium-config
89+
namespace: kube-system

0 commit comments

Comments
 (0)