Merge pull request #2062 from jasonrandrews/review

Review BOLT merging Learning Path

Author: jasonrandrews

content/install-guides/bolt.md

@@ -145,19 +145,19 @@ You are now ready to [verify BOLT is installed](#verify).
 For Arm Linux use the file with `aarch64` in the name:
 
 ```bash
-wget https://github.com/llvm/llvm-project/releases/download/llvmorg-17.0.5/clang+llvm-17.0.5-aarch64-linux-gnu.tar.xz
+wget https://github.com/llvm/llvm-project/releases/download/llvmorg-19.1.7/clang+llvm-19.1.7-aarch64-linux-gnu.tar.xz
 ```
 
 2. Extract the downloaded file
 
 ```bash
-tar -xvf clang+llvm-17.0.5-aarch64-linux-gnu.tar.xz
+tar -xvf clang+llvm-19.1.7-aarch64-linux-gnu.tar.xz
 ```
 
 3. Add the path to BOLT in your `.bashrc` file
 
 ```bash
-echo 'export PATH="$PATH:$HOME/clang+llvm-17.0.5-aarch64-linux-gnu/bin"' >> ~/.bashrc
+echo 'export PATH="$PATH:$HOME/clang+llvm-19.1.7-aarch64-linux-gnu/bin"' >> ~/.bashrc
 source ~/.bashrc
 ```
 
@@ -201,9 +201,8 @@ The output is similar to:
 
 ```output
 LLVM (http://llvm.org/):
-  LLVM version 18.0.0git
+  LLVM version 19.1.7
   Optimized build with assertions.
-BOLT revision 99c15eb49ba0b607314b3bd221f0760049130d97
 
   Registered Targets:
     aarch64    - AArch64 (little endian)

content/learning-paths/servers-and-cloud-computing/bolt-merge/_index.md

@@ -1,6 +1,10 @@
 ---
 title: Optimize Arm applications and shared libraries with BOLT
 
+draft: true
+cascade:
+    draft: true
+
 minutes_to_complete: 30
 
 who_is_this_for: Performance engineers and software developers working on Arm platforms who want to optimize both application binaries and shared libraries using BOLT.

content/learning-paths/servers-and-cloud-computing/bolt-merge/how-to-2.md

@@ -18,31 +18,32 @@ Install the required dependencies:
 
 ```bash
 sudo apt update
-sudo apt install -y build-essential cmake libncurses5-dev libssl-dev libboost-all-dev bison pkg-config libaio-dev libtirpc-dev git
+sudo apt install -y build-essential cmake libncurses5-dev libssl-dev libboost-all-dev \ 
+                  bison pkg-config libaio-dev libtirpc-dev git ninja-build liblz4-dev
 ```
 
 Download the MySQL source code. You can change to another version in the `checkout` command below if needed. 
 
 ```bash
 git clone https://github.com/mysql/mysql-server.git
 cd mysql-server
-git checkout mysql-8.4.5
+git checkout mysql-8.0.37
 ```
 
 Configure the build for debug:
 
 ```bash
 mkdir build && cd build
-cmake ..  -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_DEBUG=1 -DCMAKE_C_FLAGS="-fno-omit-frame-pointer" \
-  -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer" -DCMAKE_POSITION_INDEPENDENT_CODE=OFF \
-  -DCMAKE_EXE_LINKER_FLAGS="-Wl,--emit-relocs" \
-  -DCMAKE_EXE_LINKER_FLAGS="-no-pie"
+cmake .. -DCMAKE_C_FLAGS="-O3 -mcpu=neoverse-n2 -Wno-enum-constexpr-conversion -fno-reorder-blocks-and-partition" \
+   -DCMAKE_CXX_FLAGS="-O3 -mcpu=neoverse-n2 -Wno-enum-constexpr-conversion -fno-reorder-blocks-and-partition" \
+   -DCMAKE_CXX_LINK_FLAGS="-Wl,--emit-relocs" -DCMAKE_C_LINK_FLAGS="-Wl,--emit-relocs" -G Ninja \
+   -DWITH_BOOST=$HOME/boost -DDOWNLOAD_BOOST=On -DWITH_ZLIB=bundled -DWITH_LZ4=system -DWITH_SSL=system
 ```
 
-Build mysqld:
+Build MySQL:
 
 ```bash
-make -j$(nproc)
+ninja
 ```
 
 After the build completes, the `mysqld` binary is located at `$HOME/mysql-server/build/runtime_output_directory/mysqld`
@@ -54,11 +55,16 @@ You can run `mysqld` directly from the build directory as shown, or run `make in
 After building mysqld, install MySQL server and client utilities system-wide:
 
 ```bash
-sudo make install
+sudo ninja install
 ```
 
 This will make the `mysql` client and other utilities available in your PATH.
 
+```bash
+echo 'export PATH="$PATH:/usr/local/mysql/bin"' >> ~/.bashrc
+source ~/.bashrc
+```
+
 Ensure the binary is unstripped and includes debug symbols for BOLT instrumentation.
 
 To work with BOLT, your application binary should be:
@@ -125,6 +131,8 @@ taskset -c 6 $HOME/mysql-server/build/runtime_output_directory/mysqld.instrument
 
 Adjust `--datadir`, `--socket`, and `--port` as needed for your environment. Make sure the server is running and accessible before proceeding.
 
+With the database running, open a second terminal to run the client commands. 
+
 ## Install sysbench
 
 You will need sysbench to generate workloads for MySQL. On most Arm Linux distributions, you can install it using your package manager:
@@ -156,6 +164,22 @@ EXIT;
 
 ## Run the instrumented binary under a feature-specific workload
 
+Run `sysbench` with the `prepare` option:
+
+```bash
+sysbench \
+  --db-driver=mysql \
+  --mysql-host=127.0.0.1 \
+  --mysql-db=bench \
+  --mysql-user=bench \
+  --mysql-password=bench \
+  --mysql-port=3306 \
+  --tables=8 \
+  --table-size=10000 \
+  --threads=1 \
+  /usr/share/sysbench/oltp_read_only.lua prepare
+```
+
 Use a workload generator to stress the binary in a feature-specific way. For example, to simulate **read-only traffic** with sysbench:
 
 ```bash
@@ -176,7 +200,6 @@ taskset -c 7 sysbench \
 On an 8-core system, cores are numbered 0-7. Adjust the `taskset -c` values as needed for your system. Avoid using the same core for both mysqld and sysbench to reduce contention.
 {{% /notice %}} 
 
-
 The `.fdata` file defined in `--instrumentation-file` will be populated with runtime execution data.
 
 ## Verify the profile was created

content/learning-paths/servers-and-cloud-computing/bolt-merge/how-to-3.md

@@ -50,13 +50,6 @@ merge-fdata $HOME/mysql-server/build/profile-readonly.fdata $HOME/mysql-server/b
   -o $HOME/mysql-server/build/profile-merged.fdata
 ```
 
-**Example command from an actual setup:**
-
-```bash
-/home/ubuntu/llvm-latest/build/bin/merge-fdata prof-instrumentation-readonly.fdata prof-instrumentation-writeonly.fdata \\
-  -o prof-instrumentation-readwritemerged.fdata
-```
-
 Output:
 
 ```
@@ -79,15 +72,23 @@ ls -lh $HOME/mysql-server/build/profile-merged.fdata
 Use LLVM-BOLT to generate the final optimized binary using the merged `.fdata` file:
 
 ```bash
-llvm-bolt build/bin/mysqld \\
-  -o build/bin/mysqldreadwrite_merged.bolt_instrumentation \\
-  -data=/home/ubuntu/mysql-server-8.0.33/sysbench/prof-instrumentation-readwritemerged.fdata \\
-  -reorder-blocks=ext-tsp \\
-  -reorder-functions=hfsort \\
-  -split-functions \\
-  -split-all-cold \\
-  -split-eh \\
-  -dyno-stats \\
+llvm-bolt $HOME/mysql-server/build/runtime_output_directory/mysqld \
+  -instrument \
+  -o $HOME/mysql-server/build/runtime_output_directory/mysqld.instrumented \
+  --instrumentation-file=$HOME/mysql-server/build/profile-readonly.fdata \
+  --instrumentation-sleep-time=5 \
+  --instrumentation-no-counters-clear \
+  --instrumentation-wait-forks
+
+llvm-bolt $HOME/mysql-server/build/runtime_output_directory/mysqld \
+  -o $HOME/mysql-server/build/mysqldreadwrite_merged.bolt_instrumentation \
+  -data=$HOME/mysql-server/build/prof-instrumentation-readwritemerged.fdata \
+  -reorder-blocks=ext-tsp \
+  -reorder-functions=hfsort \
+  -split-functions \
+  -split-all-cold \
+  -split-eh \
+  -dyno-stats \
   --print-profile-stats 2>&1 | tee bolt_orig.log
 ```
 

content/learning-paths/servers-and-cloud-computing/bolt-merge/how-to-4.md

@@ -10,14 +10,15 @@ layout: learningpathall
 If system libraries like `/usr/lib/libssl.so` are stripped, rebuild OpenSSL from source with relocations:
 
 ```bash
+cd $HOME
 git clone https://github.com/openssl/openssl.git
 cd openssl
 ./config -O2 -Wl,--emit-relocs --prefix=$HOME/bolt-libs/openssl
 make -j$(nproc)
 make install
 ```
 
-### BOLT-Instrument libssl.so.3
+### Instrument libssl
 
 Use `llvm-bolt` to instrument `libssl.so.3`:
 
@@ -33,11 +34,10 @@ llvm-bolt $HOME/bolt-libs/openssl/lib/libssl.so.3 \
 
 Then launch MySQL using the **instrumented shared library** and run a **read+write** sysbench test to populate the profile:
 
-### Optimize 'libssl.so' Using Its Profile
+### Optimize libssl using the profile
 
 After running the read+write test, ensure `libssl-readwrite.fdata` is populated.
 
-
 Run BOLT on the uninstrumented `libssl.so` with the collected read-write profile:
 
 ```bash
@@ -53,7 +53,7 @@ llvm-bolt $HOME/bolt-libs/openssl/lib/libssl.so.3 \
   --print-profile-stats
 ```
 
-### Replace the Library at Runtime
+### Replace the library at runtime
 
 Copy the optimized version over the original and export the path:
 
@@ -64,7 +64,7 @@ export LD_LIBRARY_PATH=$HOME/bolt-libs/openssl/lib
 
 This ensures MySQL will dynamically load the optimized `libssl.so`.
 
-### Run Final Workload and Validate Performance
+### Run the final workload and validate performance
 
 Start the BOLT-optimized MySQL binary and link it against the optimized `libssl.so`. Run the combined workload:
 
@@ -86,7 +86,7 @@ taskset -c 7 sysbench \
 
 In the next step, you'll optimize an additional critical external library (`libcrypto.so`) using BOLT, following a similar process as `libssl.so`. Afterward, you'll interpret performance results to validate and compare optimizations across baseline and merged scenarios.
 
-### BOLT optimization for 'libcrypto.so'
+### BOLT optimization for libcrypto
 
 Follow these steps to instrument and optimize `libcrypto.so`:
 

content/learning-paths/servers-and-cloud-computing/bolt-merge/how-to-5.md

@@ -18,8 +18,6 @@ This step presents the performance comparisons across various BOLT optimization
 | Latency 95th % (ms)       | 1.04                 | 0.83                   | 1.79                   |
 | Total time (s)            | 9.93                 | 4.73                   | 15.40                  |
 
----
-
 ### 2. Performance Comparison: Merged vs Non-Merged Instrumentation
 
 | Metric                     | Regular BOLT R+W (No Merge, system libssl) | Merged BOLT (BOLTed Read+Write + BOLTed libssl) |
@@ -40,8 +38,6 @@ Second run:
 | Latency 95th % (ms)       | 1.39                                        | 1.37                                            |
 | Total time (s)            | 239.9                                       | 239.9                                           |
 
----
-
 ### 3. BOLTed READ, BOLTed WRITE, MERGED BOLT (Read+Write+BOLTed Libraries)
 
 | Metric                     | Bolted Read-Only  | Bolted Write-Only | Merged BOLT (Read+Write+libssl) | Merged BOLT (Read+Write+libcrypto) | Merged BOLT (Read+Write+libssl+libcrypto) |
@@ -52,21 +48,18 @@ Second run:
 | Latency 95th % (ms)       | 0.77                | 0.55              | 1.37                             | 1.34                               | 1.34                                      |
 | Total time (s)            | 239.8               | 239.72            | 239.9                            | 239.9                              | 239.9                                     |
 
----
-
 {{% notice Note %}}
 All sysbench and .fdata file paths, as well as taskset usage, should match the conventions in previous steps: use sysbench from PATH (no src/), use /usr/share/sysbench/ for Lua scripts, and use $HOME-based paths for all .fdata and library files. On an 8-core system, use taskset -c 7 for sysbench and avoid contention with mysqld.
 {{% /notice %}}
 
-### Key Metrics to Analyze
+### Key metrics to analyze
 
 - **TPS (Transactions Per Second)**: Higher is better.
 - **QPS (Queries Per Second)**: Higher is better.
 - **Latency (Average and 95th Percentile)**: Lower is better.
 
----
-
 ### Conclusion
+
 - BOLT substantially improves performance over non-optimized binaries due to better instruction cache utilization and reduced execution path latency.
 - Merging feature-specific profiles does not negatively affect performance; instead, it captures a broader set of runtime behaviors, making the binary better tuned for varied real-world workloads.
 - Separately optimizing external user-space libraries, even though providing smaller incremental gains, further complements the overall application optimization, delivering a fully optimized execution environment.