Skip to content

Commit 5b79924

Browse files
authored
benchmarking: Improve docs on workload (#119)
* benchmarking: Improve docs on workload * benchmarking: Fix typo in docs
1 parent bfd5615 commit 5b79924

File tree

2 files changed

+48
-30
lines changed

2 files changed

+48
-30
lines changed

docs/14.benchmarking.md

Lines changed: 42 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -3,33 +3,45 @@ id: benchmarking
33
title: Benchmarking
44
---
55

6-
Due to Skytable having in-house implementations of almost everything, starting from the protocol, the storage engine and query
7-
language — we have our own custom load testing tool called `sky-bench` that is distributed with the bundle.
8-
9-
## Setting up for benchmarking
10-
11-
**Quick notes**:
12-
- The benchmark tool will create:
13-
- a space called `bench`
14-
- a model called `bench`
15-
- **Be sure that these objects don't already exist!** But don't worry, if they do — the benchmark tool will error. You won't lose any data
16-
- **Once the benchmark is complete, the `bench` space is removed!**
17-
- **Do not use the `bench` space during the benchmark**: You might lose data
18-
19-
**The benchmark tool will:**
20-
- Run a total of 4,000,000 queries (don't worry, they run pretty fast!):
21-
- Run 1,000,000 `INSERT`s
22-
- Run 1,000,000 `SELECT`s
23-
- Run 1,000,000 `UPDATE`s
24-
- Run 1,000,000 `DELETE`s
25-
- The model used has the declaration `(un: string, pw: uint8)`
26-
- The `SELECT` will select and return all fields
27-
- The `UPDATE` will increment the value of `pw` like this `pw += 1`
28-
- The `DELETE` removes the entire collection
29-
- The default primary key size is 7 bytes. All generated keys will be padded with zeros from to 0 to 999,999 like this: `0000000` or `0999999`
30-
31-
## Off to the races
32-
33-
1. Start up the database server
34-
2. Run `./sky-bench --password <your root password>`. We need your `root` password because only the root account can create, alter and drop models and the benchmark tool needs to run these queries
35-
3. Wait for it to happen. You may not believe your eyes, so we recommend that you keep your eyes hydrated 🔥🚀✨💣
6+
All of Skytable's components are developed from scratch in-house, which makes it hard (if not impossible) to utilize other benchmarking suites. This is why all of Skytable's release bundles come with the `sky-bench` benchmarking tool. It's important to understand how the tool works to understand what you're benchmarking and what the results mean.
7+
8+
The overall goal with the benchmark tool is to simulate how the database would perform in the real-world and reduce synthetic factors.
9+
10+
:::info
11+
The ability to simulate different workloads is currently being worked on. This means that across releases, the default workload that the benchmark tool runs may vary. See the release notes to see if the benchmark workload has changed.
12+
:::
13+
14+
## Benchmark workload
15+
16+
The workload that the engine currently uses (as of v0.8.1) is the following:
17+
- A model `bench.bench` is created with a primary key (of type `string` and with a column of type `uint8`)
18+
- Multiple clients are created (simulating "application servers")
19+
- Queries are run against **unique rows**. This means that unlike `redis-benchmark` **every query group touches a different row, as it would generally do in the real-world**
20+
- The following queries (collectively a "query group") are run for each unique row:
21+
- The row is first created with an `INSERT`
22+
- All columns of the row are returned with a `SELECT`
23+
- The integer column is incremented with an `UPDATE`
24+
- The row is finally removed with a `DELETE`
25+
- **By default, 1,000,000 rows are created and manipulated**
26+
- **The time taken for each row to be sent, read back and decoded into a readable form is recorded** (time taken to parse into actual language structures such as maps and lists) towards the total time taken, once again unlike many other benchmark tools
27+
- In total 4,000,000 queries are run (by default)
28+
29+
:::caution
30+
The benchmark tool will create a space `bench` and a model `bench` and will completely remove the space and associated data once the benchmark is complete. **Do not use this space!**
31+
:::
32+
33+
## Running the benchmark
34+
35+
Now that you know how the benchmark engine works, go ahead and benchmark for yourself.
36+
37+
1. Run:
38+
```sh
39+
sky-bench --password <root_password>
40+
```
41+
**Note**: You can ignore the `--password` argument if you have already set it using the `SKYDB_PASSWORD` environment variable
42+
2. The benchmark engine will run the full workload (as described earlier)
43+
3. Witness 4,000,000 queries being executed in real-time. Good luck and enjoy the results!
44+
45+
:::tip
46+
You can tune the number of threads, connections, rows created and such to your liking to simulate the environment that you think matches your production setting.
47+
:::

docs/2.installation.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,12 @@ Getting started with Skytable involves choosing a mode of installation, download
1010
- [**A Docker image**](#docker-image): We generally recommend using a Docker image for experimenting with Skytable on your local system during development and you want to keep your local system *clean*. If you want to use a Docker image for deployment, you're always free to do so!
1111
> **Note:** You might experience slightly degraded performance from the storage engine due to Docker engine's abstractions.
1212
13+
:::tip
14+
All client tools (such as `skysh` and `sky-bench`) *can* use the `SKYDB_PASSWORD` variable for authentication. If you're using Skytable in a testing environment and frequently need to use `skysh`, you may consider setting this variable to your password to avoid having to pass the `--password` argument every time.
15+
16+
However, we strongly recommend **not** using it outside testing environments.
17+
:::
18+
1319
## Native binaries
1420

1521
To use native binaries you need to download a bundle which is simply a ZIP file with all the necessary binaries that you'll ever need to develop on and deploy Skytable.

0 commit comments

Comments
 (0)