Skip to content

Commit 1cefc3c

Browse files
mtuliomtulio
authored
OPCT-326: collector: gather kube-burner data for basic perf profiles (#67)
Collect kube-burner profiles to with a simple execution to measure basic performance from clusters in controlled environment (aka know providers/CI) to be used as a baseline while evaluating partner clusters. Total time increased in the workflow: 9 minutes Total data collected (additional): ~512K Those profiles have been used to collect performance report in internal documentation: [OPCT-276](https://issues.redhat.com/browse/OPCT-276)
1 parent 0866ceb commit 1cefc3c

File tree

5 files changed

+218
-13
lines changed

5 files changed

+218
-13
lines changed

‎artifacts-collector/Containerfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@ LABEL io.k8s.display-name="OPCT Clients" \
1818
WORKDIR /plugin
1919

2020
RUN echo "fastestmirror=true" >> /etc/dnf/dnf.conf \
21-
&& microdnf install -y curl grep tar xz gzip diff \
21+
&& microdnf update -y \
22+
&& microdnf install -y wget curl grep tar xz gzip diff \
2223
&& microdnf clean all
2324

2425
COPY --from=tools /usr/bin/oc /usr/bin/oc

‎artifacts-collector/README.md‎

Lines changed: 65 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,72 @@
11
# artifacts-collector plugin
22

3+
Collector plugin/step is the final step of the default conformance workflow
4+
executed by OPCT.
35

4-
Build and push (from root directory of project):
6+
The step collects artifacts such as:
7+
8+
- e2e metadata executed by openshift-tests for previous steps - sent to the collector pod as a artifact server, saved with prefix `artifacts_e2e-*`
9+
- must-gather: used as a baseline to evaluate cluster information, and etcd performance during the conformance tests. Must-gather is the first data collected—running before performance tests—so our report won’t be impacted by non-e2e workloads.
10+
- [camgi](https://github.com/elmiko/camgi.rs) report: generates camgi report if it is available in the running architecture
11+
- run etcd FIO tool: to evaluate disk one-off performance of control plane nodes, and sample of compute/worker nodes
12+
- kube-burner: run standard profiles to execute performance tests, collecting metrics and data to local index
13+
14+
## Prerequisites
15+
16+
- Download latest version of opct
17+
18+
### Build your custom image
19+
Build and push (from the root directory):
520

621
```sh
722
make build-plugin-collector PLATFORMS=linux/amd64 COMMAND=push
823
```
24+
25+
## Usage
26+
27+
### Run individual collectors - kube-burner
28+
29+
It's possible to run individual collector by customizing the plugin manifest.
30+
31+
The kube-burner manifest file `manifests/kube-burner-only.yaml` enforce flags
32+
to prevent collecting standard data, running only kube-burner in the target cluster.
33+
34+
To run the standalone plugin, you can use the wrapped API of Sonobuoy including OPCT:
35+
36+
> Update the `image` in the podSpec manifest file `manifests/kube-burner-only.yaml`
37+
38+
- Run
39+
```bash
40+
./opct sonobuoy run -p ./artifacts-collector/manifests/kube-burner-only.yaml \
41+
--dns-namespace=openshift-dns \
42+
--dns-pod-labels=dns.operator.openshift.io/daemonset-dns=default
43+
```
44+
45+
- Follow the execution or read the logs:
46+
47+
```sh
48+
./opct sonobuoy status
49+
50+
# or read the logs
51+
52+
oc logs -l plugin-name=99-openshift-artifacts-collector -n sonobuoy
53+
```
54+
55+
- When completed, retrieve the results:
56+
57+
```sh
58+
./opct sonobuoy retrieve
59+
```
60+
61+
- Then explore the performance data:
62+
63+
64+
- When completed, retrieve the results:
65+
66+
```sh
67+
$ tar xfz -C results/ 202502062032_sonobuoy_4afa09f6-24e1-4909-b9d2-7c158d604b02.tar.gz
68+
69+
$ ls -sh results/plugins/99-openshift-artifacts-collector/results/global/
70+
total 424K
71+
52K artifacts_kube-burner_cluster-density-v2.log 52K artifacts_kube-burner_node-density-cni.log 52K artifacts_kube-burner_node-density.log 268K artifacts_kube-burner.tar.gz
72+
```

‎artifacts-collector/collector.sh‎

Lines changed: 88 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,70 @@ collect_metrics() {
165165
os_log_info "${msg_prefix} finished!"
166166
}
167167

168+
# kube_burner run workloads for performance and scale testing. The tests are
169+
# executed in the Validation environment, and the results are saved as raw data
170+
# into the artifact path.
171+
# https://kube-burner.github.io/kube-burner-ocp/latest/
172+
173+
# kube_burner_install install kube-burner-ocp (kube-burner wrapper) if it is not installed yet.
174+
function kube_burner_install() {
175+
if [[ -f /usr/local/bin/kube-burner-ocp ]]; then
176+
return
177+
fi
178+
send_test_progress "status=running=kube-burner=install";
179+
ARCH=$(uname -m)
180+
echo "Installing kube-burner-ocp version ${KUBE_BURNER_VERSION} for ${ARCH}"
181+
wget -q -O kube-burner-ocp.tar.gz "https://github.com/kube-burner/kube-burner-ocp/releases/download/v${KUBE_BURNER_VERSION}/kube-burner-ocp-V${KUBE_BURNER_VERSION}-linux-${ARCH}.tar.gz"
182+
tar xfz kube-burner-ocp.tar.gz && mv -v kube-burner-ocp /usr/local/bin/kube-burner-ocp
183+
}
184+
185+
# kube_burner_run is the individual profile executor, saving the result in a consolidated kube-burner directory.
186+
function kube_burner_run() {
187+
local index_dir
188+
echo "> Running kube-burner ${KB_CMD}"
189+
kube_burner_install
190+
send_test_progress "status=running=kube-burner=${KB_CMD}";
191+
192+
kube-burner-ocp ${KB_CMD} --local-indexing ${KUBE_BURNER_EXTRA_ARGS-} |& tee -a "${RESULTS_DIR}"/artifacts_kube-burner_"${KB_CMD}".log
193+
194+
# kube-burner-ocp save the index to directory collected-metrics-uuid
195+
index_dir=$(ls collected-metrics* -d)
196+
job_id="$(echo "${index_dir}" | awk -F'collected-metrics-' '{print$1}')"
197+
mv -v "${index_dir}" "${KUBE_BURNER_RESULTS_DIR}/${KB_CMD}"
198+
echo "${job_id}" > "${KUBE_BURNER_RESULTS_DIR}/id_${KB_CMD}.txt"
199+
}
200+
201+
# kube_burner_save saves the artifacts collected for each profile from a consolidated
202+
# directory into the sonobuoy result/artifact directory.
203+
function kube_burner_save() {
204+
echo "> Saving kube-burner results"
205+
pushd ${KUBE_BURNER_RESULTS_DIR} || return
206+
tar cvfz "${RESULTS_DIR}"/artifacts_kube-burner.tar.gz ./*
207+
popd || return
208+
}
209+
210+
# collect_kube_burner is the entrypoint to execute kube-burner profiles.
211+
function collect_kube_burner() {
212+
mkdir -p "${KUBE_BURNER_RESULTS_DIR}"
213+
for KB_CMD in ${KUBE_BURNER_COMMANDS-}; do
214+
send_test_progress "status=running=kube-burner=${KB_CMD}";
215+
echo "> Running kube-burner command: ${KB_CMD}"
216+
unset KUBE_BURNER_EXTRA_ARGS
217+
case ${KB_CMD} in
218+
"cluster-density-v2")
219+
KUBE_BURNER_EXTRA_ARGS="--iterations=1 --churn-duration=2m0s --churn-cycles=2"
220+
kube_burner_run;
221+
;;
222+
*)
223+
kube_burner_run;
224+
;;
225+
esac
226+
send_test_progress "status=done=kube-burner=${KB_CMD}";
227+
done
228+
kube_burner_save
229+
send_test_progress "status=done=kube-burner";
230+
}
231+
168232
# Run Plugin for Collecor. The Collector plugin is the last one executed on the
169233
# cluster. It will collect custom files used on the Validation environment, at the
170234
# end it will generate a tarbal file to submit the raw results to Sonobuoy.
@@ -173,22 +237,35 @@ run_plugin_collector() {
173237

174238
pushd "${RESULTS_DIR}" || true
175239

176-
# Collecting must-gather
177-
send_test_progress "status=running=collecting must-gather";
178-
collect_must_gather || true
240+
# Collect must-gather
241+
if [[ "${SKIP_MUST_GATHER:-false}" == "false" ]]; then
242+
send_test_progress "status=running=collecting must-gather";
243+
collect_must_gather || true
244+
fi
179245

180-
# Experimental: Collect performance data
246+
# Collect performance data
181247
# running after must-gather to prevent impacting in etcd logs when testing etcdfio.
182-
send_test_progress "status=running=collecting performance data";
183-
collect_performance || true
248+
if [[ "${SKIP_PERFORMANCE:-false}" == "false" ]]; then
249+
send_test_progress "status=running=collecting performance data";
250+
collect_performance || true
251+
fi
184252

185-
# Experimental: Collect metrics
186-
send_test_progress "status=running=collecting metrics";
187-
collect_metrics || true
253+
# Collect metrics
254+
if [[ "${SKIP_METRICS:-false}" == "false" ]]; then
255+
send_test_progress "status=running=collecting metrics";
256+
collect_metrics || true
257+
fi
188258

189-
# Creating Result file used to publish to sonobuoy. (last step)
190-
send_test_progress "status=running=saving artifacts";
259+
# Experimental: Collect kube-burner (performance profiles)
260+
# Experimental are skipped by default. Need to be explicitly enabled in the plugin manifest:
261+
# https://github.com/redhat-openshift-ecosystem/opct/blob/main/data/templates/plugins/openshift-artifacts-collector.yaml
262+
if [[ "${SKIP_KUBE_BURNER:-true}" == "false" ]]; then
263+
send_test_progress "status=running=kube-burner";
264+
collect_kube_burner || true
265+
fi
191266

267+
# Create result file used to publish to sonobuoy aggregator. (must be the last step)
268+
send_test_progress "status=running=saving artifacts";
192269
os_log_info "[executor][PluginID#${PLUGIN_ID}] Packing all results..."
193270
ls -sh ./artifacts_*
194271
tar cfz raw-results.tar.gz ./artifacts_*

‎artifacts-collector/global_env.sh‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,10 @@ declare -gx MIRROR_IMAGE_REPOSITORY
2727

2828
# Utilities
2929
declare -grx UTIL_OC_BIN="/usr/bin/oc"
30+
31+
# Kube Burner
32+
KUBE_BURNER_DEFAULT_COMMANDS="node-density node-density-cni cluster-density-v2"
33+
declare -gx KUBE_BURNER_COMMANDS="${KUBE_BURNER_COMMANDS:-${KUBE_BURNER_DEFAULT_COMMANDS}}"
34+
declare -gx KUBE_BURNER_RESULTS_DIR="/tmp/kube-burner"
35+
KUBE_BURNER_DEFAULT_VERSION="1.6.2"
36+
declare -gx KUBE_BURNER_VERSION="${KUBE_BURNER_VERSION:-${KUBE_BURNER_DEFAULT_VERSION}}"

‎artifacts-collector/manifests/kube-burner-only.yaml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
---
2+
podSpec:
3+
restartPolicy: Never
4+
serviceAccountName: sonobuoy-serviceaccount
5+
priorityClassName: system-node-critical
6+
volumes:
7+
- name: shared
8+
emptyDir: {}
9+
sonobuoy-config:
10+
driver: Job
11+
plugin-name: 99-openshift-artifacts-collector
12+
result-format: raw
13+
description: |
14+
OPCT artifacts plugin collects the artifacts from the OpenShift Container
15+
Platform post-validation.
16+
source-url:
17+
"https://github.com/redhat-openshift-ecosystem/provider-certification-tool/\
18+
blob/main/manifests/openshift-artifacts-collector.yaml"
19+
skipCleanup: true
20+
spec:
21+
name: plugin
22+
image: "quay.io/opct/plugin-artifacts-collector:v0.0.0-devel-f17e7fa"
23+
imagePullPolicy: Always
24+
command: ["/bin/bash", "/plugin/entrypoint-collector.sh"]
25+
volumeMounts:
26+
- mountPath: /tmp/shared
27+
name: shared
28+
env:
29+
- name: PLUGIN_ID
30+
value: "99"
31+
- name: PLUGIN_NAME
32+
value: "openshift-artifacts-collector"
33+
- name: PLUGIN_BLOCKED_BY
34+
value: ""
35+
- name: IMAGE_OVERRIDE_MUST_GATHER
36+
value: "quay.io/opct/must-gather-monitoring:devel"
37+
- name: ENV_NODE_NAME
38+
valueFrom:
39+
fieldRef:
40+
fieldPath: spec.nodeName
41+
- name: ENV_POD_NAME
42+
valueFrom:
43+
fieldRef:
44+
fieldPath: metadata.name
45+
- name: ENV_POD_NAMESPACE
46+
valueFrom:
47+
fieldRef:
48+
fieldPath: metadata.namespace
49+
- name: RUN_MODE
50+
value: "default"
51+
- name: SKIP_MUST_GATHER
52+
value: "true"
53+
- name: SKIP_PERFORMANCE
54+
value: "true"
55+
- name: SKIP_METRICS
56+
value: "true"

0 commit comments

Comments
 (0)