Skip to content

Commit a6b91e0

Browse files
heitortsergentyorugac
heitortsergent
and
yorugac
authored
Add k6-operator docs to Set up (#1559)
* chore: add basic structure for k6-operator docs * chore: hide reference for now * k6-operator: add main sections from the repo Readme.md * Add a short introduction to the index page * Update install-k6-operator.md * Update upgrade-k6-operator.md * Update troubleshooting.md * Update executing-k6-scripts-with-testrun-crd.md * Update extensions.md * Update extensions.md * Update common-options.md * Update scheduling-tests.md * k6-operator: fix typos * k6-operator: add content for troubleshooting.md * chore: replaces instances of k6-operator with k6 Operator * chore: add uninstall instructions * chore: hide Upgrade k6 Operator page * chore: add Use the k6 operator with Grafana Cloud k6 page * chore: review troubleshooting doc * chore: update Namespaced deployment heading to Watch namespace * Apply suggestions from code review Co-authored-by: Olha Yevtushenko <[email protected]> * Move docs to next and v0.52.x folders * Remove docs from v0.50.x folder --------- Co-authored-by: Olha Yevtushenko <[email protected]> Co-authored-by: Olha Yevtushenko <[email protected]>
1 parent a1495bc commit a6b91e0

22 files changed

+1882
-0
lines changed

‎docs/sources/next/set-up/set-up-distributed-k6/_index.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
weight: 150
3+
title: Set up distributed k6
4+
---
5+
6+
# Set up distributed k6
7+
8+
It's possible to run large load tests even when using a single node, or single machine. But, depending on your use case, you might also want to run a distributed Grafana k6 test in your own infrastructure.
9+
10+
A couple of reasons why you might want to do this:
11+
12+
- You run your application in Kubernetes and would like k6 to be executed in the same fashion as all your other infrastructure components.
13+
- You want to run your tests within your private network for security and/or privacy reasons.
14+
15+
[k6 Operator](https://github.com/grafana/k6-operator) is a Kubernetes operator that you can use to run distributed k6 tests in your cluster.
16+
17+
This section includes the following topics:
18+
19+
{{< section depth=2 >}}

‎docs/sources/next/set-up/set-up-distributed-k6/install-k6-operator.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
---
2+
weight: 100
3+
title: Install k6 Operator
4+
---
5+
6+
# Install k6 Operator
7+
8+
This guide provides step-by-step instructions on how to install k6 Operator.
9+
10+
## Before you begin
11+
12+
To install k6 Operator, you'll need:
13+
14+
- A Kubernetes cluster, along with access to it.
15+
- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl).
16+
17+
## Deploy the operator
18+
19+
There are three different options that you can use to deploy the k6 Operator.
20+
21+
### Deploy with bundle
22+
23+
The easiest way to install the operator is with bundle:
24+
25+
```bash
26+
curl https://raw.githubusercontent.com/grafana/k6-operator/main/bundle.yaml | kubectl apply -f -
27+
```
28+
29+
Bundle includes default manifests for k6 Operator, including a `k6-operator-system` namespace and k6 Operator deployment with the latest tagged Docker image. Customizations can be made on top of this manifest as needed, for example, with `kustomize`.
30+
31+
### Deploy with Helm
32+
33+
Helm releases of k6 Operator are published together with other Grafana Helm charts. You can install it with the following commands:
34+
35+
```bash
36+
helm repo add grafana https://grafana.github.io/helm-charts
37+
helm repo update
38+
helm install k6-operator grafana/k6-operator
39+
```
40+
41+
You can also pass additional configuration options with a `values.yaml` file:
42+
43+
```bash
44+
helm install k6-operator grafana/k6-operator -f values.yaml
45+
```
46+
47+
Refer to the [k6 Operator samples folder](https://github.com/grafana/k6-operator/blob/main/charts/k6-operator/samples/customAnnotationsAndLabels.yaml) for an example file.
48+
49+
You can find a complete list of Helm options in the [k6 Operator charts folder](https://github.com/grafana/k6-operator/blob/main/charts/k6-operator/README.md).
50+
51+
### Deploy with Makefile
52+
53+
In order to install the operator with a Makefile, you'll need:
54+
55+
- [go](https://go.dev/doc/install)
56+
- [kustomize](https://kubectl.docs.kubernetes.io/installation/kustomize/)
57+
58+
A more manual, low-level way to install the k6 operator is by running the command below:
59+
60+
```bash
61+
make deploy
62+
```
63+
64+
This method may be more useful for development of the k6 Operator, depending on specifics of the setup.
65+
66+
## Install the CRD
67+
68+
The k6 Operator includes custom resources called `TestRun`, `PrivateLoadZone`, and `K6`. They're automatically installed when you do a deployment or install a bundle, but you can also manually install them by running:
69+
70+
```bash
71+
make install
72+
```
73+
74+
{{< admonition type="warning" >}}
75+
76+
The `K6` CRD has been replaced by the `TestRun` CRD and will be deprecated in the future. We recommend using the `TestRun` CRD.
77+
78+
{{< /admonition >}}
79+
80+
## Watch namespace
81+
82+
By default, the k6 Operator watches the `TestRun` and `PrivateLoadZone` custom resources in all namespaces. You can also configure the k6 Operator to watch a specific namespace by setting the `WATCH_NAMESPACE` environment variable for the operator's deployment:
83+
84+
```yaml
85+
apiVersion: apps/v1
86+
kind: Deployment
87+
metadata:
88+
name: k6-operator-controller-manager
89+
namespace: k6-operator-system
90+
spec:
91+
template:
92+
spec:
93+
containers:
94+
- name: manager
95+
image: ghcr.io/grafana/k6-operator:controller-v0.0.14
96+
env:
97+
- name: WATCH_NAMESPACE
98+
value: 'some-ns'
99+
# ...
100+
```
101+
102+
## Uninstall k6 Operator
103+
104+
You can remove all of the resources created by the k6 Operator with `bundle`:
105+
106+
```bash
107+
curl https://raw.githubusercontent.com/grafana/k6-operator/main/bundle.yaml | kubectl delete -f -
108+
```
109+
110+
Or with the `make` command:
111+
112+
```bash
113+
make delete
114+
```

‎docs/sources/next/set-up/set-up-distributed-k6/troubleshooting.md‎

Lines changed: 260 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,260 @@
1+
---
2+
weight: 400
3+
title: Troubleshooting
4+
---
5+
6+
# Troubleshooting
7+
8+
This topic includes instructions to help you troubleshoot common issues with the k6 Operator.
9+
10+
## Common tricks
11+
12+
### Test your script locally
13+
14+
Always run your script locally before trying to run it with the k6 Operator:
15+
16+
```bash
17+
k6 run script.js
18+
```
19+
20+
If you're using environment variables or CLI options, pass them in as well:
21+
22+
```bash
23+
MY_ENV_VAR=foo k6 run script.js --tag my_tag=bar
24+
```
25+
26+
That ensures that the script has correct syntax and can be parsed with k6 in the first place. Additionally, running locally can help you check if the configured options are doing what you expect. If there are any errors or unexpected results in the output of `k6 run`, make sure to fix those prior to deploying the script elsewhere.
27+
28+
### `TestRun` deployment
29+
30+
#### The pods
31+
32+
In case of one `TestRun` Custom Resource (CR) creation with `parallelism: n`, there are certain repeating patterns:
33+
34+
1. There will be `n + 2` Jobs (with corresponding Pods) created: initializer, starter, and `n` runners.
35+
1. If any of these Jobs didn't result in a Pod being deployed, there must be an issue with that Job. Some commands that can help here:
36+
37+
```bash
38+
kubectl get jobs -A
39+
kubectl describe job mytest-initializer
40+
```
41+
42+
1. If one of the Pods was deployed but finished with `Error`, you can check its logs with the following command:
43+
44+
```bash
45+
kubectl logs mytest-initializer-xxxxx
46+
```
47+
48+
If the Pods seem to be working but not producing an expected result and there's not enough information in the logs, you can use the k6 [verbose option](https://grafana.com/docs/k6/<K6_VERSION>/using-k6/k6-options/#options) in the `TestRun` spec:
49+
50+
```yaml
51+
apiVersion: k6.io/v1alpha1
52+
kind: TestRun
53+
metadata:
54+
name: k6-sample
55+
spec:
56+
parallelism: 2
57+
script:
58+
configMap:
59+
name: 'test'
60+
file: 'test.js'
61+
arguments: --verbose
62+
```
63+
64+
#### k6 Operator
65+
66+
Another source of info is the k6 Operator itself. It's deployed as a Kubernetes `Deployment`, with `replicas: 1` by default, and its logs together with observations about the Pods from the previous section usually contain enough information to help you diagnose any issues. With the standard deployment, the logs of the k6 Operator can be checked with:
67+
68+
```bash
69+
kubectl -n k6-operator-system -c manager logs k6-operator-controller-manager-xxxxxxxx-xxxxx
70+
```
71+
72+
#### Inspect `TestRun` resource
73+
74+
After you deploy a `TestRun` CR, you can inspect it the same way as any other resource:
75+
76+
```bash
77+
kubectl describe testrun my-testrun
78+
```
79+
80+
Firstly, check if the spec is as expected. Then, see the current status:
81+
82+
```yaml
83+
Status:
84+
Conditions:
85+
Last Transition Time: 2024-01-17T10:30:01Z
86+
Message:
87+
Reason: CloudTestRunFalse
88+
Status: False
89+
Type: CloudTestRun
90+
Last Transition Time: 2024-01-17T10:29:58Z
91+
Message:
92+
Reason: TestRunPreparation
93+
Status: Unknown
94+
Type: TestRunRunning
95+
Last Transition Time: 2024-01-17T10:29:58Z
96+
Message:
97+
Reason: CloudTestRunAbortedFalse
98+
Status: False
99+
Type: CloudTestRunAborted
100+
Last Transition Time: 2024-01-17T10:29:58Z
101+
Message:
102+
Reason: CloudPLZTestRunFalse
103+
Status: False
104+
Type: CloudPLZTestRun
105+
Stage: error
106+
```
107+
108+
If `Stage` is equal to `error`, you can check the logs of k6 Operator.
109+
110+
Conditions can be used as a source of info as well, but it's a more advanced troubleshooting option that should be used if the previous steps weren't enough to diagnose the issue. Note that conditions that start with the `Cloud` prefix only matter in the setting of k6 Cloud test runs, for example, for cloud output and PLZ test runs.
111+
112+
### `PrivateLoadZone` deployment
113+
114+
If the `PrivateLoadZone` CR was successfully created in Kubernetes, it should become visible in your account in Grafana Cloud k6 (GCk6) interface soon afterwards. If it doesn't appear in the UI, then there is likely a problem to troubleshoot.
115+
116+
First, go over the [guide](https://grafana.com/docs/grafana-cloud/k6/author-run/private-load-zone-v2/) to double-check if all the steps have been done correctly and successfully.
117+
118+
Unlike `TestRun` deployment, when a `PrivateLoadZone` is first created, there are no additional resources deployed. So, the only source for troubleshooting are the logs of k6 Operator. See the [previous subsection](#k6-operator) on how to access its logs. Any errors there might be a hint to diagnose the issue. Refer to [PrivateLoadZone: subscription error](#privateloadzone-subscription-error) for more details.
119+
120+
### Running tests in `PrivateLoadZone`
121+
122+
Each time a user runs a test in a PLZ, for example with `k6 cloud script.js`, there is a corresponding `TestRun` being deployed by the k6 Operator. This `TestRun` will be deployed in the same namespace as its `PrivateLoadZone`. If the test is misbehaving, for example, it errors out, or doesn't produce the expected result, then you can check:
123+
124+
1. If there are any messages in the GCk6 UI.
125+
2. If there are any messages in the output of the `k6 cloud` command.
126+
3. The resources and their logs, the same way as with a [standalone `TestRun` deployment](#testrun-deployment)
127+
128+
## Common scenarios
129+
130+
### Issues with environment variables
131+
132+
Refer to [Environment variables](https://github.com/grafana/k6-operator/blob/main/docs/env-vars.md) for details on how to pass environment variables to the k6 Operator.
133+
134+
### Tags not working
135+
136+
Tags are a rather common source of errors when using the k6 Operator. For example, the following tags would lead to parsing errors:
137+
138+
```yaml
139+
arguments: --tag product_id="Test A"
140+
# or
141+
arguments: --tag foo=\"bar\"
142+
```
143+
144+
You can see those errors in the logs of either the initializer or the runner Pod, for example:
145+
146+
```bash
147+
time="2024-01-11T11:11:27Z" level=error msg="invalid argument \"product_id=\\\"Test\" for \"--tag\" flag: parse error on line 1, column 12: bare \" in non-quoted-field"
148+
```
149+
150+
This is a common problem with escaping the characters. You can find an [issue](https://github.com/grafana/k6-operator/issues/211) in the k6 Operator repository that can be upvoted.
151+
152+
### Initializer logs an error but it's not about tags
153+
154+
This can happen because of lack of attention to the [preparation](#preparation) step. One command that you can use to help diagnose issues with your script is the following:
155+
156+
```bash
157+
k6 inspect --execution-requirements script.js
158+
```
159+
160+
That command is a shortened version of what the initializer Pod is executing. If the command produces an error, there's a problem with the script itself and it should be solved outside of the k6 Operator. The error itself may contain a hint to what's wrong, such as a syntax error.
161+
162+
If the standalone `k6 inspect --execution-requirements` executes successfully, then it's likely a problem with `TestRun` deployment specific to your Kubernetes setup. A couple of recommendations here are:
163+
164+
- Review the output of the initializer Pod: is it logged by the k6 process or by something else?
165+
- :information_source: k6 Operator expects the initializer logs to contain only the output of `k6 inspect`. If there are any other log lines present, then the k6 Operator will fail to parse it and the test won't start. Refer to this [issue](https://github.com/grafana/k6-operator/issues/193) for more details.
166+
- Check events in the initializer Job and Pod as they may contain another hint about what's wrong.
167+
168+
### Non-existent ServiceAccount
169+
170+
A ServiceAccount can be defined as `serviceAccountName` in a PrivateLoadZone, and as `runner.serviceAccountName` in a TestRun CRD. If the specified ServiceAccount doesn't exist, k6 Operator will successfully create Jobs but corresponding Pods will fail to be deployed, and the k6 Operator will wait indefinitely for Pods to be `Ready`. This error can be best seen in the events of the Job:
171+
172+
```bash
173+
kubectl describe job plz-test-xxxxxx-initializer
174+
...
175+
Events:
176+
Warning FailedCreate 57s (x4 over 2m7s) job-controller Error creating: pods "plz-test-xxxxxx-initializer-" is forbidden: error looking up service account plz-ns/plz-sa: serviceaccount "plz-sa" not found
177+
```
178+
179+
k6 Operator doesn't try to analyze such scenarios on its own, but you can refer to the following [issue](https://github.com/grafana/k6-operator/issues/260) for improvements.
180+
181+
#### How to fix
182+
183+
To fix this issue, the incorrect `serviceAccountName` must be corrected, and the TestRun or PrivateLoadZone resource must be re-deployed.
184+
185+
### Non-existent `nodeSelector`
186+
187+
`nodeSelector` can be defined as `nodeSelector` in a PrivateLoadZone, and as `runner.nodeSelector` in the TestRun CRD.
188+
189+
This case is very similar to the [ServiceAccount](#non-existent-serviceaccount): the Pod creation will fail, but the error is slightly different:
190+
191+
```bash
192+
kubectl describe pod plz-test-xxxxxx-initializer-xxxxx
193+
...
194+
Events:
195+
Warning FailedScheduling 48s (x5 over 4m6s) default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector.
196+
```
197+
198+
#### How to fix
199+
200+
To fix this issue, the incorrect `nodeSelector` must be corrected and the TestRun or PrivateLoadZone resource must be re-deployed.
201+
202+
### Insufficient resources
203+
204+
A related problem can happen when the cluster does not have sufficient resources to deploy the runners. There's a higher probability of hitting this issue when setting small CPU and memory limits for runners or using options like `nodeSelector`, `runner.affinity` or `runner.topologySpreadConstraints`, and not having a set of nodes matching the spec. Alternatively, it can happen if there is a high number of runners required for the test (via `parallelism` in TestRun or during PLZ test run) and autoscaling of the cluster has limits on the maximum number of nodes, and can't provide the required resources on time or at all.
205+
206+
This case is somewhat similar to the previous two: the k6 Operator will wait indefinitely and can be monitored with events in Jobs and Pods. If it's possible to fix the issue with insufficient resources on-the-fly, for example, by adding more nodes, k6 Operator will attempt to continue executing a test run.
207+
208+
### OOM of a runner Pod
209+
210+
If there's at least one runner Pod that OOM-ed, the whole test will be [stuck](https://github.com/grafana/k6-operator/issues/251) and will have to be deleted manually:
211+
212+
```bash
213+
kubectl -f my-test.yaml delete
214+
# or
215+
kubectl delete testrun my-test
216+
```
217+
218+
In case of OOM, it makes sense to review the k6 script to understand what kind of resource usage this script requires. It may be that the k6 script can be improved to be more performant. Then, set the `spec.runner.resources` in the TestRun CRD, or `spec.resources` in the PrivateLoadZone CRD accordingly.
219+
220+
### PrivateLoadZone: subscription error
221+
222+
If there's an issue with your Grafana Cloud k6 subscription, there will be a 400 error in the logs with the message detailing the problem. For example:
223+
224+
```bash
225+
"Received error `(400) You have reached the maximum Number of private load zones your organization is allowed to have. Please contact support if you want to create more.`. Message from server ``"
226+
```
227+
228+
To fix this issue, check your organization settings in Grafana Cloud k6 or contact Support.
229+
230+
### PrivateLoadZone: Wrong token
231+
232+
There can be two major problems with the authentication token:
233+
234+
1. If the token wasn't created, or was created in a wrong location, the logs will show the following error:
235+
236+
```bash
237+
Failed to load k6 Cloud token {"namespace": "plz-ns", "name": "my-plz", "reconcileID": "67c8bc73-f45b-4c7f-a9ad-4fd0ffb4d5f6", "name": "token-with-wrong-name", "secretNamespace": "plz-ns", "error": "Secret \"token-with-wrong-name\" not found"}
238+
```
239+
240+
2. If the token contains a corrupted value, or it's not an organizational token, the logs will show the following error:
241+
242+
```bash
243+
"Received error `(403) Authentication token incorrect or expired`. Message from server ``"
244+
```
245+
246+
### PrivateLoadZone: Networking setup
247+
248+
If you see any dial or connection errors in the logs of the k6 Operator, it makes sense to double-check the networking setup. For a PrivateLoadZone to operate, outbound traffic to Grafana Cloud k6 [must be allowed](https://grafana.com/docs/grafana-cloud/k6/author-run/private-load-zone-v2/#before-you-begin). To check the reachability of Grafana Cloud k6 endpoints:
249+
250+
```bash
251+
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
252+
kubectl exec -it dnsutils -- nslookup ingest.k6.io
253+
kubectl exec -it dnsutils -- nslookup api.k6.io
254+
```
255+
256+
For more resources on troubleshooting networking, refer to the [Kubernetes docs](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/).
257+
258+
### PrivateLoadZone: Insufficient resources
259+
260+
The PrivateLoadZone insufficient resources problem is similar to [insufficient resources issue](#insufficient-resources). But, when running a PrivateLoadZone test, the k6 Operator will wait only for a timeout period. When the timeout period is up, the test will be aborted by Grafana Cloud k6 and marked as such, both in the PrivateLoadZone and in Grafana Cloud k6. In other words, there is a time limit to fix this issue without restarting the test run.

0 commit comments

Comments
 (0)