Skip to content
This repository was archived by the owner on May 12, 2025. It is now read-only.

Commit 9a86d37

Browse files
tisbaztreinhartzeisss
authored
Trial image for Tricentis NeoLoad (#12)
Co-authored-by: Zach Reinhart <zach@stormforge.io> Co-authored-by: Stephan <stephan@stormforge.io>
1 parent 235f153 commit 9a86d37

File tree

6 files changed

+265
-0
lines changed

6 files changed

+265
-0
lines changed

‎.github/workflows/test-neoload.yaml‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
name: Test NeoLoad
2+
3+
on:
4+
push:
5+
paths:
6+
- 'neoload/**'
7+
- '.github/workflows/test-neoload.yaml'
8+
- '.github/actions/*'
9+
10+
jobs:
11+
test-neoload:
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Checkout
15+
uses: actions/checkout@v3
16+
- name: Run ShellCheck
17+
uses: ludeeus/action-shellcheck@master
18+
with:
19+
scandir: "neoload/docker-entrypoint.sh"
20+
- name: Set up Docker Buildx
21+
uses: docker/setup-buildx-action@v2
22+
with:
23+
version: "latest"
24+
install: true
25+
- name: Build Image
26+
uses: docker/build-push-action@v3
27+
with:
28+
context: ./neoload
29+
push: false
30+
pull: true
31+
load: true
32+
tags: sf-neoload-test
33+
cache-from: type=gha
34+
cache-to: type=gha,mode=max
35+
36+
- name: Run NeoLoad image
37+
working-directory: neoload
38+
continue-on-error: true
39+
run: |
40+
mkdir -p tmp/
41+
42+
docker run sf-neoload-test > tmp/docker_output.txt 2>&1
43+
44+
- name: Test docker output
45+
working-directory: neoload
46+
run: |
47+
echo "tmp/docker_output.txt:"
48+
cat tmp/docker_output.txt
49+
50+
echo "Testing for expected error message..."
51+
test -n "$(grep 'ERROR: NEOLOAD_TOKEN not found' tmp/docker_output.txt)"

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,5 @@ This repository is a collection of container images suitable for use as trial jo
1212
| [`jmeter/`](./jmeter/) | This trial job uses [JMeter](https://jmeter.apache.org/) to execute a test plan (`.jmx` file) to generate load for each Optimize Pro trial. |
1313
| [`k6/`](./k6/) | This trial job uses [k6](https://k6.io) to execute a test script written in JavaScript to generate load for each Optimize Pro trial. |
1414
| [`locust/`](./locust/) | This trial job executes a [Locust](https://locust.io/) load test written in Python to generate load for each Optimize Pro trial. |
15+
| [`neoload/`](./neoload/) | This trial job invokes a [NeoLoad Web](https://www.tricentis.com/products/performance-testing-neoload) cloud-hosted performance test against your application, with each trial corresponding to a test run. |
1516
| [`stormforge-perf/`](./stormforge-perf/) | This trial job invokes a [StormForge Performance Testing](https://www.stormforge.io/performance-testing/) test case against your application, with each trial corresponding to a test run. |

‎neoload/Dockerfile‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
FROM python:3.10-bullseye
2+
3+
USER root
4+
5+
RUN \
6+
apt-get update -yqq \
7+
&& apt-get install -yq --no-install-recommends \
8+
uuid-runtime \
9+
curl \
10+
jq \
11+
&& rm -rf /var/cache/apt/archives/* /var/lib/apt/lists/* /tmp/* /var/tmp/* \
12+
&& truncate -s 0 /var/log/*log
13+
14+
RUN pip3 install neoload
15+
16+
COPY docker-entrypoint.sh parse-report.sh /
17+
ENTRYPOINT ["/docker-entrypoint.sh"]

‎neoload/README.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Tricentis NeoLoad Trial Image
2+
3+
The [Tricentis NeoLoad](https://www.tricentis.com/products/performance-testing-neoload) trial job enables you to use NeoLoad performance tests in your Optimize Pro experiments.
4+
It utilizes a customer container image, based on the [`python:3.10-bullseye`](https://hub.docker.com/_/python) base container and the [official NeoLoad CLI utility](https://github.com/Neotys-Labs/neoload-cli).
5+
6+
## Usage
7+
8+
- Create a NeoLoad access token.
9+
- Configure the environment variables shown in the next section for your trial job in your `experiment.yaml`
10+
- Configure Trial to use either `prometheus` setupTask or manually define `PUSHGATEWAY_URL`.
11+
12+
## Configuration
13+
14+
| Environment Variable | Description | Default value |
15+
| -------------------- | ----------- |---------------|
16+
| `SCENARIO` | Scenario from NeoLoad Project to use | `StormForgeScenario` |
17+
| `TEST_NAME` | Test result naming pattern | `SF-$(uuidgen)` |
18+
| `TEST_FILE` | Zip or YAML test file to use (will be uploaded to Neoload SaaS) | none |
19+
| `ZONE` | NeoLoad controller zone to use | `USEGCP`
20+
| `LGS` | Number of load generators to use | `2`
21+
| `NEOLOAD_TOKEN` | NeoLoad authentication token | none |
22+
| `PUSHGATEWAY_URL` | The URL used to push NeoLoad test run metrics. If not explicitly set, the Optimize controller will set it. |
23+
24+
## Metrics
25+
26+
Metrics are extracted from the NeoLoad test report using the built-in transactions-csv template to produce a CSV file.
27+
This file contains calculated statistics based on all requests executed in the test.
28+
29+
| Name | Example Value | Description |
30+
| ---- | ------------- | ----------- |
31+
| `request_count` | 81323 | total number of requests performed |
32+
| `average_response_time` | 0.1387 | average response time (seconds) |
33+
| `max_response_time` | 0.2578 | maximum response time (seconds) |
34+
| `min_response_time` | 0.0879 | minimum response time (seconds) |
35+
| `p50` | 0.145 | 50th percentile for response time (seconds) |
36+
| `p90` | 0.184 | 90th percentile for response time (seconds) |
37+
| `p95` | 0.186 | 95th percentile for response time (seconds) |
38+
| `p99` | 0.189 | 99th percentile for response time (seconds) |
39+
40+
## Example Kubernetes Manifest
41+
42+
43+
The following Kubernetes `Job` manifest excerpt, `ConfigMap`, and `Secret` illustrate how you might use this trial job container in your Optimize Pro `experiment.yaml`.
44+
45+
```yaml
46+
apiVersion: batch/v1
47+
kind: Job
48+
metadata:
49+
name: sandbox-1
50+
namespace: examples
51+
spec:
52+
backoffLimit: 0
53+
template:
54+
spec:
55+
restartPolicy: Never
56+
containers:
57+
- name: neoload
58+
image: thestormforge/optimize-trials:latest-blazemeter-cloud
59+
env:
60+
- name: TEST_NAME
61+
valueFrom:
62+
fieldRef:
63+
fieldPath: metadata.name
64+
- name: LGS
65+
value: "5"
66+
- name: TEST_FILE
67+
value: /tmp/neoload-files/neoload-stormforge.yaml
68+
- name: NEOLOAD_TOKEN
69+
valueFrom:
70+
secretKeyRef:
71+
name: neoload-token
72+
key: token
73+
volumeMounts:
74+
- name: test-case-file
75+
readOnly: true
76+
mountPath: /tmp/neoload-files
77+
volumes:
78+
- name: test-case-file
79+
configMap:
80+
name: neoload-artifacts
81+
---
82+
apiVersion: v1
83+
kind: ConfigMap
84+
metadata:
85+
name: neoload-artifacts
86+
data:
87+
neoload-stormforge.yaml: |
88+
name: Loadtest Party
89+
user_paths:
90+
- name: sf-path
91+
actions:
92+
steps:
93+
- transaction:
94+
name: Transaction1
95+
description: loadtest party
96+
steps:
97+
- request:
98+
url: http://TARGET_SERVER/
99+
- request:
100+
url: /
101+
server: TARGET_SERVER
102+
method: GET
103+
104+
populations:
105+
- name: pop1
106+
user_paths:
107+
- name: sf-path
108+
109+
scenarios:
110+
- name: StormForgeScenario
111+
populations:
112+
- name: pop1
113+
rampup_load:
114+
min_users: 20
115+
max_users: 100
116+
increment_users: 2
117+
increment_every: 1s
118+
duration: 5m
119+
---
120+
apiVersion: v1
121+
kind: Secret
122+
metadata:
123+
name: neoload-token
124+
data:
125+
token: XXX-XXX-XXX-XXX
126+
```
127+
128+
NOTE: If you are using this in an experiment, keep in mind that some values are set automatically. In particular, the `backoffLimit`, `restartPolicy`, and `PUSHGATEWAY_URL` environment variable are all introduced when evaluating a trial's job template. `PUSHGATEWAY_URL` requires the usage of the [`prometheus` `setupTask`](https://docs.stormforge.io/optimize-pro/concepts/trials/#prometheus) though.

‎neoload/docker-entrypoint.sh‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
#!/bin/bash
2+
set -e
3+
4+
if [ -z "${ZONE}" ]; then
5+
ZONE="USEGCP"
6+
fi
7+
8+
if [ -z "${LGS}" ]; then
9+
LGS=2
10+
fi
11+
12+
if [ -z "${SCENARIO}" ]; then
13+
SCENARIO="StormForgeScenario"
14+
fi
15+
16+
if [ -z "${TEST_NAME}" ]; then
17+
TEST_NAME="SF-"$(uuidgen)
18+
fi
19+
20+
if [ -z "${NEOLOAD_TOKEN}" ]; then
21+
echo "ERROR: NEOLOAD_TOKEN not found. You must specify a valid NeoLoad authentication token to access NeoLoad Web." > /dev/stderr
22+
exit 1
23+
fi
24+
25+
if [ ! -f "${TEST_FILE}" ]; then
26+
echo "ERROR: TEST_FILE = ${TEST_FILE} not found. Did you forget to mount as volume or via ConfigMap?" > /dev/stderr
27+
exit 1
28+
fi
29+
30+
# neoload read-only bug workaround
31+
cp "${TEST_FILE}" /tmp/neoload-stormforge.yaml
32+
TEST_FILE="/tmp/neoload-stormforge.yaml"
33+
34+
neoload login "${NEOLOAD_TOKEN}"
35+
neoload test-settings --zone ${ZONE} --lgs ${LGS} --scenario ${SCENARIO} create "${TEST_NAME}"
36+
37+
neoload project --path ${TEST_FILE} upload
38+
neoload test-settings use "${TEST_NAME}"
39+
neoload run
40+
41+
neoload report --template builtin:transactions-csv > transactions.csv
42+
43+
/parse-report.sh
44+
45+
# Push the basic statistics
46+
if [ -n "${PUSHGATEWAY_URL}" ]; then
47+
/parse_report.sh | curl --data-binary @- "${PUSHGATEWAY_URL}"
48+
else
49+
echo "No Pushgateway URL" 1>&2
50+
fi

‎neoload/parse-report.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#! /bin/bash
2+
set -e
3+
4+
# Ignore unused variables (they are just listed for reference)
5+
# shellcheck disable=SC2034
6+
while IFS=";" read -r user_path element parent count \
7+
min avg max perc_50 perc_90 perc_95 perc_99 \
8+
success success_rate failure failure_rate
9+
do
10+
echo "request_count $count"
11+
echo "average_response_time $avg"
12+
echo "max_response_time $max"
13+
echo "min_response_time $min"
14+
echo "p50 $perc_50"
15+
echo "p90 $perc_90"
16+
echo "p95 $perc_95"
17+
echo "p99 $perc_99"
18+
done < <(tail -n +2 transactions.csv)

0 commit comments

Comments
 (0)