|
| 1 | +--- |
| 2 | +title: DCPerf |
| 3 | +author: Kieran Hejmadi |
| 4 | +minutes_to_complete: 20 |
| 5 | +official_docs: https://github.com/facebookresearch/DCPerf?tab=readme-ov-file#install-and-run-benchmarks |
| 6 | + |
| 7 | +additional_search_terms: |
| 8 | +- linux |
| 9 | +- Neoverse |
| 10 | + |
| 11 | +test_images: |
| 12 | +- ubuntu:22.04 |
| 13 | +test_maintenance: false |
| 14 | + |
| 15 | +layout: installtoolsall |
| 16 | +multi_install: false |
| 17 | +multitool_install_part: false |
| 18 | +tool_install: true |
| 19 | +weight: 1 |
| 20 | +--- |
| 21 | + |
| 22 | +## Introduction |
| 23 | + |
| 24 | +DCPerf is an open-source benchmarking and microbenchmarking suite originally developed by Meta. It faithfully replicates the characteristics of general-purpose data center workloads, with particular attention to microarchitectural fidelity. DCPerf stands out for accurate simulation of behaviors such as cache misses and branch mispredictions, which are details that many other benchmarking tools overlook. |
| 25 | + |
| 26 | +You can use DCPerf to generate performance data to inform procurement decisions, and for regression testing to detect changes in the environment, such as kernel and compiler changes. |
| 27 | + |
| 28 | +DCPerf runs on Arm-based servers. The examples below have been tested on an AWS `c7g.metal` instance running Ubuntu 22.04 LTS. |
| 29 | + |
| 30 | +{{% notice Note %}} |
| 31 | +When running on a server provided by a cloud service, you have limited access to some parameters, such as UEFI settings, which can affect performance. |
| 32 | +{{% /notice %}} |
| 33 | + |
| 34 | +## Install prerequisites |
| 35 | + |
| 36 | +To get started, install the required software: |
| 37 | + |
| 38 | +```bash |
| 39 | +sudo apt update |
| 40 | +sudo apt install -y python-is-python3 python3-pip python3-venv git |
| 41 | +``` |
| 42 | + |
| 43 | +It is recommended that you install Python packages in a Python virtual environment. |
| 44 | + |
| 45 | +Set up your virtual environment: |
| 46 | + |
| 47 | +```bash |
| 48 | +python3 -m venv venv |
| 49 | +source venv/bin/activate |
| 50 | +``` |
| 51 | +If requested, restart the recommended services. |
| 52 | + |
| 53 | +Install the required packages: |
| 54 | + |
| 55 | +```bash |
| 56 | +pip3 install click pyyaml tabulate pandas |
| 57 | +``` |
| 58 | + |
| 59 | +Clone the repository: |
| 60 | + |
| 61 | +```bash |
| 62 | +git clone https://github.com/facebookresearch/DCPerf.git |
| 63 | +cd DCPerf |
| 64 | +``` |
| 65 | + |
| 66 | +## Running the MediaWiki benchmark |
| 67 | + |
| 68 | +DCPerf offers many benchmarks. See the official documentation for the benchmark of your choice. |
| 69 | + |
| 70 | +One example is the MediaWiki benchmark, designed to faithfully reproduce the workload of the Facebook social networking site. |
| 71 | + |
| 72 | +Install HipHop Virtual Machine (HHVM), a virtual machine used to execute the web application code: |
| 73 | + |
| 74 | +```bash |
| 75 | +wget https://github.com/facebookresearch/DCPerf/releases/download/hhvm/hhvm-3.30-multplatform-binary-ubuntu.tar.xz |
| 76 | +tar -Jxf hhvm-3.30-multplatform-binary-ubuntu.tar.xz |
| 77 | +cd hhvm |
| 78 | +sudo ./pour-hhvm.sh |
| 79 | +export LD_LIBRARY_PATH="/opt/local/hhvm-3.30/lib:$LD_LIBRARY_PATH" |
| 80 | +``` |
| 81 | + |
| 82 | +Confirm `hhvm` is available. The `hhvm` binary is located in the `DCPerf/hhvm/aarch64-ubuntu22.04/hhvm-3.30/bin` directory: |
| 83 | + |
| 84 | +```bash |
| 85 | +hhvm --version |
| 86 | +# Return to the DCPerf root directory |
| 87 | +cd .. |
| 88 | +``` |
| 89 | + |
| 90 | +You should see output similar to: |
| 91 | + |
| 92 | +```output |
| 93 | +HipHop VM 3.30.12 (rel) |
| 94 | +Compiler: 1704922878_080332982 |
| 95 | +Repo schema: 4239d11395efb06bee3ab2923797fedfee64738e |
| 96 | +``` |
| 97 | + |
| 98 | +Confirm security-enhanced Linux (SELinux) is disabled with the following commands: |
| 99 | + |
| 100 | +```bash |
| 101 | +sudo apt install selinux-utils |
| 102 | +getenforce |
| 103 | +``` |
| 104 | + |
| 105 | +You should see the following response: |
| 106 | + |
| 107 | +```output |
| 108 | +Disabled |
| 109 | +``` |
| 110 | + |
| 111 | +If you do not see the `Disabled` output, see your Linux distribution documentation for information about how to disable SELinux. |
| 112 | + |
| 113 | +You can automatically install all dependencies for each benchmark using the `install` argument with the `benchpress_cli.py` command-line script: |
| 114 | + |
| 115 | +```console |
| 116 | +sudo ./benchpress_cli.py install oss_performance_mediawiki_mlp |
| 117 | +``` |
| 118 | + |
| 119 | +This step might take several minutes to complete, depending on your system's download and setup speed. |
| 120 | + |
| 121 | +## Run the MediaWiki benchmark |
| 122 | + |
| 123 | +For the sake of brevity, you can provide the duration and timeout arguments using a `JSON` dictionary with the `-i` argument: |
| 124 | + |
| 125 | +```console |
| 126 | +sudo ./benchpress_cli.py run oss_performance_mediawiki_mlp -i '{ |
| 127 | + "duration": "30s", |
| 128 | + "timeout": "1m" |
| 129 | +}' |
| 130 | +``` |
| 131 | + |
| 132 | +While the benchmark is running, you can monitor CPU activity and observe benchmark-related processes using the `top` command. |
| 133 | + |
| 134 | +When the benchmark is complete, a `benchmark_metrics_*` directory is created within the `DCPerf` directory, containing a `JSON` file for the system specs and another for the metrics. |
| 135 | + |
| 136 | +For example, the metrics file lists the following: |
| 137 | + |
| 138 | +```output |
| 139 | + "metrics": { |
| 140 | + "Combined": { |
| 141 | + "Nginx 200": 1817810, |
| 142 | + "Nginx 404": 79019, |
| 143 | + "Nginx 499": 3, |
| 144 | + "Nginx P50 time": 0.036, |
| 145 | + "Nginx P90 time": 0.056, |
| 146 | + "Nginx P95 time": 0.066, |
| 147 | + "Nginx P99 time": 0.081, |
| 148 | + "Nginx avg bytes": 158903.93039183, |
| 149 | + "Nginx avg time": 0.038826036781319, |
| 150 | + "Nginx hits": 1896832, |
| 151 | + "Wrk RPS": 3160.65, |
| 152 | + "Wrk failed requests": 79019, |
| 153 | + "Wrk requests": 1896703, |
| 154 | + "Wrk successful requests": 1817684, |
| 155 | + "Wrk wall sec": 600.1, |
| 156 | + "canonical": 0 |
| 157 | + }, |
| 158 | + "score": 2.4692578125 |
| 159 | +``` |
| 160 | + |
| 161 | +## Understanding the benchmark results |
| 162 | + |
| 163 | +The metrics file contains several key performance indicators from the benchmark run: |
| 164 | + |
| 165 | + |
| 166 | +- **Nginx 200, 404, 499**: The number of HTTP responses with status codes 200 (success), 404 (not found), and 499 (client closed request) returned by the Nginx web server during the test. |
| 167 | +- **Nginx P50/P90/P95/P99 time**: The response time percentiles (in seconds) for requests handled by Nginx. For example, P50 is the median response time, P99 is the time under which 99% of requests completed. |
| 168 | +- **Nginx avg bytes**: The average number of bytes sent per response. |
| 169 | +- **Nginx avg time**: The average response time for all requests. |
| 170 | +- **Nginx hits**: The total number of requests handled by Nginx. |
| 171 | +- **Wrk RPS**: The average number of requests per second (RPS) generated by the `wrk` load testing tool. |
| 172 | +- **Wrk failed requests**: The number of requests that failed during the test. |
| 173 | +- **Wrk requests**: The total number of requests sent by `wrk`. |
| 174 | +- **Wrk successful requests**: The number of requests that completed successfully. |
| 175 | +- **Wrk wall sec**: The total wall-clock time (in seconds) for the benchmark run. |
| 176 | +- **score**: An overall performance score calculated by DCPerf, which can be used to compare different systems or configurations. |
| 177 | + |
| 178 | +{{% notice Note %}} |
| 179 | + `wrk` is a modern HTTP benchmarking tool used to generate load and measure web server performance. It is widely used for benchmarking because it can produce significant load and provides detailed statistics. For more information, see [wrk's GitHub page](https://github.com/wg/wrk). |
| 180 | +{{% /notice %}} |
| 181 | + |
| 182 | +These metrics help you evaluate the performance and reliability of the system under test. Higher values for successful requests and RPS, and lower response times, generally indicate better performance. The score provides a single value for easy comparison across runs or systems. |
| 183 | + |
| 184 | +## Next steps |
| 185 | + |
| 186 | +These are some activites you might like to try next: |
| 187 | + |
| 188 | +* Use the results to compare performance across different systems, hardware configurations, or after making system changes, such as kernel, compiler, or driver updates. |
| 189 | + |
| 190 | +* Consider tuning system parameters or trying alternative DCPerf benchmarks to further evaluate your environment. |
| 191 | + |
| 192 | +* Explore additional DCPerf workloads, including those that simulate key-value stores, in-memory caching, or machine learning inference. |
0 commit comments