Skip to content
This repository was archived by the owner on Feb 20, 2023. It is now read-only.

Commit 3b07e8f

Browse files
jazzplatojazzplato
authored
Add step-by-step instruction to test runner infra code. (#1376)
1 parent 958cc0b commit 3b07e8f

File tree

4 files changed

+274
-78
lines changed

4 files changed

+274
-78
lines changed

‎script/testing/README.md

Lines changed: 81 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -7,20 +7,20 @@ All tests are compatible with python3
77
- `micro_bench`: entry script to run the microbenchmark tests
88
- `oltpbench`: entry script to fire an oltp bench test
99
- `artifact_stats`: entry script to collect the artifact stats
10+
- `reporting`: utility scripts for posting test data to Django API and formating JSON payloads
1011

1112
## Util
1213
`util` folder contains a list of common Python scripts
13-
- `common`: functions that can be used in many different settings
14-
- `constants`: all the constants used in the any file under the `util` or across the different tests
15-
- `NoisePageServer`: A class that can start, stop, or restart an instance of the DBMS
16-
- `TestServer`: the base class for running all types of tests
17-
- `TestCase`: the base class for all types of tests
18-
- `TestJUnit`: the test class for junit tests
19-
- `TestOLTPBench`: the test class for oltp bench tests
14+
- `common.py`: functions that can be used in many different settings
15+
- `constants.py`: all the constants used in the any file under the `util` or across the different tests
16+
- `db_server.py`: It provides a `NoisePageServer` class that can start, stop, or restart an instance of the NoisePage
17+
- `test_server.py`: the base `TestServer` class for running all types of tests
18+
- `test_case.py`: the base `TestCase` class for all types of test cases.
19+
- `mem_metrics.py`: the `MemoryMetric` class and `MemoryInfo` named tuple to manage the memory related information during the run time of the tests.
20+
- `periodic_task.py`: the `PeriodicTask` class provides a general utility in Python which runs a separate thread that will execute a subprocess every `x` seconds until told to stop.
2021

2122
## OLTP Bench
22-
`oltpbench` folder contains Python scripts for running an oltp bench test
23-
- `TestOLTPBench`: the test class for oltp bench tests
23+
`oltpbench` folder contains Python scripts for running an oltp bench test. Refer to [OLTP Benchmark Testing](https://github.com/cmu-db/noisepage/tree/master/script/testing/oltpbench/README.md) for more details.
2424

2525
## How to run a test
2626
To run a test of a certain type, just run the `run_<TEST TYPE>.py` script in the respective folder. For example, if you want to run a junit test, just simply run `python3 junit/run_junit.py`.
@@ -54,15 +54,75 @@ If you specify the `--query-mode extended`, you then can also indicate the prepa
5454
- run the post-suite task (test suite specific)
5555
- print out the logs to the stdout
5656

57-
### Adding a new test case
58-
All test cases should inherit from the `TestCase` class. Anyone is free to modify any attribute from the base class.
59-
- Mandatory attributes
60-
- `test_command` (`List(str)`): the command to run the test case
61-
- Optional attributes
62-
- `test_command_cwd` (`str`): the working directory to run the test command
63-
- `test_error_msg` (`str`): the error message to display in case of errors
64-
- Optional functions
65-
- `run_pre_test`: the pre-test tasks required for the test
66-
- config the xml file, etc.
67-
- `run_post_test`: the post-test tasks required for the test
68-
- e.g. parse the output json, etc.
57+
## How to create a new test type
58+
The classes in the `util` folder can be used and extend to help you create a new test type.
59+
60+
### Base classes
61+
- `NoisePageServer`
62+
- Manage the lifecycle of the NoisePage instance. It create a Python subprocess for the NoisePage process, poll the logs, and terminate or kill it when the test finishes
63+
- `TestCase`
64+
- Manage a the life cycle of a test case. A test case is usually a process trigger from command line. The actual test case can be specified as a command string and created and executed in a Python subprocess.
65+
- The `TestCase` class also provides `run_pre_test` and `run_post_test` functions for you to override for preparation and clean up of each test case.
66+
- `TestServer`
67+
- Manage the entire lifecycle of a test. It uses the `NoisePageServer` to manage the database process. One `TestServer` can have a list of `TestCase`s and treats the entire collection as a suite.
68+
- It also provides the `run_pre_suite` and `run_post_suite` functions for you to override to specify any preparation and cleanup at the suite level.
69+
70+
### Step-by-step instructions
71+
- Create the folder for your test under `noisepage/script/testing/<mytest>`
72+
- In the folder of your test, create the following files
73+
- `run_<mytest>.py`
74+
- The entry script for your test, which specifies the command line arguments and options.
75+
- You can refer to [oltpbench/run_oltpbench.py](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench/run_oltpbench.py) for reference.
76+
- `test_<mytest>.py`
77+
- The main test class for your test, which should be a subclass of the `TestServer` in `util/test_server.py`.
78+
- You can refer to [oltpbench/test_oltpbench.py](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench/test_oltpbench.py) for reference.
79+
- [optional] `test_case_<mytest>.py`
80+
- The base test case class for your test, which should be a subclass of the `TestCaseServer` in `util/test_case_server.py`.
81+
- You can refer to [oltpbench/test_case_oltp.py](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench/test_case_oltp.py) for reference.
82+
- [optional] `constants.py`
83+
- Contains all the constants the test need to use. Usually you should add the commands you need to run for your test and pre/post_suite tasks and as pre/post_test tasks constants here. It can be imported and used in your `test_<mytest>.py` or `test_case_<mytest>.py`.
84+
- You can refer to [oltpbench/constants.py](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench/constants.py) for reference.
85+
- [optional] `util.py`
86+
- Contains all the utility functions the test need to use.
87+
- You can refer to [oltpbench/util.py](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench/util.py) for reference.
88+
- Create a stage for your test in Jenkins pipeline
89+
- Go to `noisepage/Jenkinsfile`, create a stage at the place of your choice, and create the stage based on the template config as below.
90+
```groovy
91+
stage('My Test') {
92+
parallel{
93+
stage('My Test Name 1') {
94+
agent { label 'macos' }
95+
environment {
96+
<!-- Add environment variables here -->
97+
}
98+
steps {
99+
<!-- Add a list of shell commands here -->
100+
}
101+
post {
102+
cleanup {
103+
deleteDir()
104+
}
105+
}
106+
}
107+
stage('My Test Name 2') {
108+
agent {
109+
docker {
110+
image 'noisepage:focal'
111+
args '--cap-add sys_ptrace -v /jenkins/ccache:/home/jenkins/.ccache'
112+
}
113+
}
114+
environment {
115+
<!-- Add environment variables here -->
116+
}
117+
steps {
118+
<!-- Add a list of shell commands here -->
119+
}
120+
post {
121+
cleanup {
122+
deleteDir()
123+
}
124+
}
125+
}
126+
}
127+
}
128+
```

‎script/testing/oltpbench/README.md

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# OLTP Benchmark Testing
2+
This folder include the necessary files for running oltpbench testings.
3+
4+
## Table of contents
5+
- [OLTP benchmark types](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench#oltp-benchmark-types)
6+
- [How to run it?](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench#how-to-run-it)
7+
- [Test workflow](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench#test-workflow)
8+
- [Dependencies](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench#dependencies)
9+
- [File structures](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench#file-structures)
10+
- [Config files](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench#config-files)
11+
12+
## OLTP benchmark types
13+
Currently we run the following OLTP benchmarks
14+
- TATP
15+
- TPCC
16+
- NOOP
17+
- YCSB
18+
- Smallbank
19+
20+
## How to run it?
21+
22+
### TL;DR
23+
```bash
24+
# This command is sufficient for you to run the OLTP benchmark testing based on your configuration
25+
python3 run_oltpbench.py --config-file=<your-config-file-path>.json
26+
```
27+
28+
### Examples
29+
Currently there are 3 types of the OLTP benchmark testing
30+
- End-to-end debug
31+
```bash
32+
cd noisepage/script/testing/oltpbench
33+
# To run the TPCC oltpbench test for end-to-end debug on debug version
34+
python3 run_oltpbench.py \
35+
--config-file=configs/end_to_end_debug/tpcc.json \
36+
--build-type=debug
37+
```
38+
- End-to-end performance
39+
```bash
40+
cd noisepage/script/testing/oltpbench
41+
# To run the TPCC oltpbench test for end-to-end performance with wal to be stored in ramdisk on release version
42+
python3 run_oltpbench.py \
43+
--config-file=configs/end_to_end_performance/tpcc_wal_ramdisk.json \
44+
--build-type=release
45+
```
46+
- Nightly performance
47+
```bash
48+
cd noisepage/script/testing/oltpbench
49+
# To run all the oltpbench test case for nightly performance with wal disabled on release version
50+
python3 run_oltpbench.py \
51+
--config-file=configs/nightly/nightly_wal_disabled.json \
52+
--build-type=release
53+
```
54+
55+
### OLTPBench test options
56+
For more configurations, there are 2 ways
57+
- Via config files
58+
- You can refer to the [config files](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench#config-files) section below for more details
59+
- Via command line options
60+
comments - You can use `-h` or `--help` command line option for more details
61+
62+
## Test workflow
63+
Preparation steps of the test
64+
- Start by invoking `run_oltpbench.py`
65+
- Load the config file passed in and start an instance of the database with any server_args specified in the config
66+
- Create a test suite for the whole config file and runs the test suite
67+
- Create a `TestOLTPBench` object for the test execution, since the `TestOLTPBench` is a subclass of the `TestServer` class, it will conduct a list of tasks as the preparation for the actual test, the most important two of which are as follow
68+
- Pass command line options and configs in to the constructor
69+
- Try to locate the NoisePage binary
70+
- Run the test suite by calling `.run()` function of `TestOLTPBench`
71+
72+
The actual test workflow
73+
- Run pre suite tasks: `.run_pre_suite()` function of `TestOLTPBench`
74+
- Clean the possible residual local [oltpbench](https://github.com/oltpbenchmark/oltpbench) workspace
75+
- Download and install the [oltpbench](https://github.com/oltpbenchmark/oltpbench) from GitHub
76+
- Iterate through all the test cases
77+
- (Re)start the NoisePage DB process as a Python subprocess
78+
- Start the DB if its not running
79+
- *Kill all the lingering processes on the NoisePage port before starting the NoisePage DB process*
80+
- Restart it if `db_restart` was passed into the test_case config
81+
- Otherwise, leave it running
82+
- Run pre test case tasks: `.run_pre_test()` function of `TestCaseOLTPBench`
83+
- Create the database and tables for the OLTP benchmark specified
84+
- Load the data to tables
85+
- Run the test case command as a subprocess
86+
- Collect the memory info by using the [PeriodicTask](https://github.com/cmu-db/noisepage/blob/master/script/testing/util/periodic_task.py) in a separate thread
87+
- Collect `RSS` and `VMS` by default
88+
- Collect every `5` seconds by default
89+
- The memory info is stored in a Python dictionary in memory in runtime
90+
- Run post test case tasks: `.run_post_test()` function of `TestCaseOLTPBench`
91+
- If `publish-results`, `publish-username`, and `publish-password` arguments are supplied, the result results should be stored
92+
- Parse the testing results files by [oltpbench](https://github.com/oltpbenchmark/oltpbench) and format them in JSON
93+
- Add the memory info to `incremental_metrics` and compute the average metrics to add to the `metrics` in JSON payload
94+
- Send a POST request to the Django API
95+
- Run post suite tasks: `.run_post_suite()` function of `TestOLTPBench`
96+
97+
## Dependencies
98+
99+
### External dependencies
100+
- [oltpbench](https://github.com/oltpbenchmark/oltpbench): for executing the OLTP benchmark of your choice
101+
- [NoisePage Stats](https://github.com/cmu-db/noisepage-stats): for the result reporting API and storage
102+
103+
### Internal dependencies
104+
Let the base directory be `noisepage/script/testing`
105+
- `util/`: provides base classes/functions/constants to construct the `TestOLTPBench` class
106+
- `test_server.py`
107+
- Contains the base `TestServer` class to manage the lifecycle of a test suite for a certain OLTP benchmark type
108+
- `test_case.py`
109+
- Contains the base `TestCase` class to manage the lifecycle of a test case
110+
- `common.py`
111+
- Contains common utility helper functions
112+
- `constants.py`
113+
- Contains general constants, like `ErrorCode`
114+
- `reporting/`: provide functions to report testing results to Django API
115+
116+
## File structures
117+
Let the base directory be `noisepage/script/testing/oltpbench`
118+
- `configs/`: contains the JSON config files for Jenkins to run OLTPBench tests for
119+
- End-to-end debugging
120+
- End-to-end performance
121+
- Nightly
122+
123+
For more details, check the [config files](https://github.com/cmu-db/noisepage/blob/master/script/testing/oltpbench#config-files) section below for more details
124+
- `run_oltpbench.py`: entry point to run the OLTP bench tests
125+
- `test_oltpbench.py`: defines the `TestOLTPBench` class which manage the lifecycle of a test suite
126+
- `test_case_oltp.py`: defines the `TestOLTPBench` class which manage the lifecycle of a test case
127+
- `utils.py`: defines a list of utility functions specifically used by OLTP bench tests
128+
129+
## Config files
130+
A config file is necessary to specify a test for OLTP Benchmark.
131+
132+
### Fields
133+
In the configuration file, those information are required:
134+
- A list of test cases in `testcases`
135+
- The benchmarks and options in each test case, required by the oltpbench's workload descriptor file
136+
- The `run_oltpbench` script will run all test cases in the configuration file sequentially.
137+
- The `loop` key in the configuration file is used to duplicate the test case with different options.
138+
- The `server_args` object in the configuration specifies the command line arguments passed to the server when running the binary.
139+
140+
### Example:
141+
The following example shows a oltpbenchmark test with 4 testcases
142+
- The first testcase has the base testcase of terminal `1` but it will be looped for different configurations of the terminals from `1` to `2` to `4`
143+
- The second testcase only has the base testcase whose terminal configuration is `8`
144+
145+
```json
146+
{
147+
"type": "oltpbenchmark",
148+
"server_args":{
149+
"connection_thread_count": 32,
150+
"wal_file_path": "/mnt/ramdisk/wal.log"
151+
},
152+
"testcases": [
153+
{
154+
"base": {
155+
"benchmark": "tatp",
156+
"weights": "2,35,10,35,2,14,2",
157+
"query_mode": "extended",
158+
"scale_factor": 1,
159+
"terminals": 1,
160+
"loader_threads": 4,
161+
"client_time": 60
162+
},
163+
"loop": [
164+
{"terminals":1, "db_create":true,"db_load":true},
165+
{"terminals":2, "db_restart":false,"db_create":false,"db_load":false},
166+
{"terminals":4, "db_restart":false,"db_create":false,"db_load":false},
167+
]
168+
},
169+
{
170+
"base": {
171+
"benchmark": "tatp",
172+
"weights": "2,35,10,35,2,14,2",
173+
"query_mode": "extended",
174+
"terminals": 8,
175+
"loader_threads": 4,
176+
"client_time": 600
177+
}
178+
}
179+
]
180+
}
181+
```

‎script/testing/oltpbench/configs/README.md

Lines changed: 0 additions & 51 deletions
This file was deleted.

0 commit comments

Comments
 (0)