Add BOLT Merge Learning Path

Author: GayathriNarayana19

content/learning-paths/servers-and-cloud-computing/bolt-merge/Bolt-merge.png

@@ -0,0 +1,62 @@
+---
+
+
+title: BOLT Merge :Feature-level and Library-level BOLTing with Profile Merging
+
+minutes_to_complete: 30
+
+who_is_this_for: >
+  In networking and high-performance applications, a single execution path (e.g., one feature) often activates only a small portion of the binary. For example, 20% of the application may be exercised under one feature, while the remaining features and external libraries remain untouched.
+
+  A single BOLT pass using one workload leads to partial optimization, typically benefiting only the code paths covered by that specific run.
+
+  This learning path is intended for performance engineers and developers working on Arm-based systems who need to optimize large, feature-rich application binaries that depend on external libraries.
+
+  It demonstrates how to bolt application features and shared libraries independently, then merge the resulting profiles to achieve full code coverage and deploy a fully optimized binary.
+
+
+learning_objectives: 
+    - Instrument and optimize binaries for individual workload features using LLVM-BOLT
+    - Collect separate BOLT profiles and merge them for comprehensive code coverage
+    - Optimize shared libraries independently
+    - Integrate these bolted libraries into applications at runtime
+    - Compare performance across baseline, isolated, and merged optimization cases
+
+prerequisites:
+    - An Arm based system running Linux with BOLT and Linux Perf installed. The Linux kernel should be version 5.15 or later. Earlier kernel versions can be used, but some Linux Perf features may be limited or not available.
+    - (Optional) A second, more powerful Linux system to build the software executable and run BOLT.
+
+author: Gayathri Narayana Yegna Narayanan
+
+### Tags
+skilllevels: Introductory
+subjects: Performance and Architecture
+armips:
+    - Neoverse
+    - Cortex-A
+tools_software_languages:
+    - BOLT
+    - perf
+    - Runbook
+operatingsystems:
+    - Linux
+
+further_reading:
+    - resource:
+        title: BOLT README
+        link: https://github.com/llvm/llvm-project/tree/main/bolt
+        type: documentation
+    - resource:
+        title: BOLT - A Practical Binary Optimizer for Data Centers and Beyond
+        link: https://research.facebook.com/publications/bolt-a-practical-binary-optimizer-for-data-centers-and-beyond/
+        type: website
+
+
+
+### FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 1                       # _index.md always has weight of 1 to order correctly
+layout: "learningpathall"       # All files under learning paths have this same wrapper
+learning_path_main_page: "yes"  # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
+---
+

content/learning-paths/servers-and-cloud-computing/bolt-merge/_index.md

@@ -0,0 +1,8 @@
+---
+# ================================================================================
+#       FIXED, DO NOT MODIFY THIS FILE
+# ================================================================================
+weight: 21                  # Set to always be larger than the content in this path to be at the end of the navigation.
+title: "Next Steps"         # Always the same, html page title.
+layout: "learningpathall"   # All files under learning paths have this same wrapper for Hugo processing.
+---

content/learning-paths/servers-and-cloud-computing/bolt-merge/_next-steps.md

@@ -0,0 +1,39 @@
+---
+title: Overview of BOLT Merge 
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+[BOLT](https://github.com/llvm/llvm-project/blob/main/bolt/README.md) is a post-link binary optimizer that uses Linux Perf data to re-order the executable code layout to reduce memory overhead and improve performance.
+
+
+The diagram below illustrates why merging BOLT profiles and optimizing libraries independently is essential for complete binary optimization:
+
+![Why BOLT Profile Merging?](Bolt-merge.png)
+
+- The **left chart** shows a typical application binary, where only 50% is proprietary application code, and the rest consists of external libraries.
+- The **right chart** breaks down that application code into individual features (F1–F5). In any given run, typically only one feature is active — meaning only 20% of the code is exercised and profiled.
+- As a result, a single BOLT pass provides incomplete optimization.
+
+To ensure full optimization, the workflow includes: 
+1. Profiling each workload feature separately
+2. Profiling external libraries independently
+3. Merging profiles for broader code coverage
+4. Applying BOLT to each binary and library
+5. Linking bolted libraries with the merged-profile binary
+
+In this Learning Path, you'll learn how to:
+- Collect and merge BOLT profiles from multiple workload features (e.g., read-only and write-only)
+- Independently optimize application binaries and external user-space libraries (e.g., `libssl.so`, `libcrypto.so`)
+- Link the final optimized binary with the separately bolted libraries to deploy a fully optimized runtime stack
+
+While MySQL and sysbench are used as examples, this method applies to **any feature-rich application** that:
+- Exhibits multiple runtime paths
+- Uses dynamic libraries
+- Requires full-stack binary optimization for performance-critical deployment
+
+
+
+

content/learning-paths/servers-and-cloud-computing/bolt-merge/example-picture.png

@@ -0,0 +1,89 @@
+---
+title: BOLT Optimization - First feature
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+In this step, you will instrument an application binary (such as `mysqld`) with BOLT to collect runtime profile data for a specific feature — for example, a **read-only workload**.
+
+The collected profile will later be merged with others and used to optimize the application's code layout.
+
+### Step 1: Build or obtain the uninstrumented binary
+
+Make sure your application binary is:
+
+- Built from source (e.g., `mysqld`)
+- Unstripped, with symbol information available
+- Compiled with frame pointers enabled (`-fno-omit-frame-pointer`)
+
+You can verify this with:
+
+```bash
+readelf -s /path/to/mysqld | grep main
+```
+
+If the symbols are missing, rebuild the binary with debug info and no stripping.
+
+---
+
+### Step 2: Instrument the binary with BOLT
+
+Use `llvm-bolt` to create an instrumented version of the binary:
+
+```bash
+llvm-bolt /path/to/mysqld \\
+  -instrument \\
+  -o /path/to/mysqld.instrumented \\
+  --instrumentation-file=/path/to/profile-readonly.fdata \\
+  --instrumentation-sleep-time=5 \\
+  --instrumentation-no-counters-clear \\
+  --instrumentation-wait-forks
+```
+
+### Explanation of key options
+
+- `-instrument`: Enables profile generation instrumentation
+- `--instrumentation-file`: Path where the profile output will be saved
+- `--instrumentation-wait-forks`: Ensures the instrumentation continues through forks (important for daemon processes)
+
+---
+
+### Step 3: Run the instrumented binary under a feature-specific workload
+
+Use a workload generator to stress the binary in a feature-specific way. For example, to simulate **read-only traffic** with sysbench:
+
+```bash
+taskset -c 9 ./src/sysbench \\
+  --db-driver=mysql \\
+  --mysql-host=127.0.0.1 \\
+  --mysql-db=bench \\
+  --mysql-user=bench \\
+  --mysql-password=bench \\
+  --mysql-port=3306 \\
+  --tables=8 \\
+  --table-size=10000 \\
+  --threads=1 \\
+  src/lua/oltp_read_only.lua run
+```
+
+> Adjust this command as needed for your workload and CPU/core binding.
+
+The `.fdata` file defined in `--instrumentation-file` will be populated with runtime execution data.
+
+---
+
+### Step 4: Verify the profile was created
+
+After running the workload:
+
+```bash
+ls -lh /path/to/profile-readonly.fdata
+```
+
+You should see a non-empty file. This file will later be merged with other profiles (e.g., for write-only traffic) to generate a complete merged profile.
+
+---
+
+

content/learning-paths/servers-and-cloud-computing/bolt-merge/how-to-1.md

@@ -0,0 +1,100 @@
+---
+title: BOLT Optimization - Second Feature & BOLT Merge to combine 
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+In this step, you'll collect profile data for a **write-heavy** workload and also **instrument external libraries** such as `libcrypto.so` and `libssl.so` used by the application (e.g., MySQL).
+
+
+### Step 1: Run Write-Only Workload for Application Binary
+
+Use the same BOLT-instrumented MySQL binary and drive it with a write-only workload to capture `profile-writeonly.fdata`:
+
+```bash
+taskset -c 9 ./src/sysbench \\
+  --db-driver=mysql \\
+  --mysql-host=127.0.0.1 \\
+  --mysql-db=bench \\
+  --mysql-user=bench \\
+  --mysql-password=bench \\
+  --mysql-port=3306 \\
+  --tables=8 \\
+  --table-size=10000 \\
+  --threads=1 \\
+  src/lua/oltp_write_only.lua run
+```
+
+Make sure that the `--instrumentation-file` is set appropriately to save `profile-writeonly.fdata`.
+---
+### Step 2: Verify the Second Profile Was Generated
+
+```bash
+ls -lh /path/to/profile-writeonly.fdata
+```
+
+Both `.fdata` files should now exist and contain valid data:
+
+- `profile-readonly.fdata`
+- `profile-writeonly.fdata`
+
+---
+
+### Step 3: Merge the Feature Profiles
+
+Use `merge-fdata` to combine the feature-specific profiles into one comprehensive `.fdata` file:
+
+```bash
+merge-fdata /path/to/profile-readonly.fdata /path/to/profile-writeonly.fdata \\
+  -o /path/to/profile-merged.fdata
+```
+
+**Example command from an actual setup:**
+
+```bash
+/home/ubuntu/llvm-latest/build/bin/merge-fdata prof-instrumentation-readonly.fdata prof-instrumentation-writeonly.fdata \\
+  -o prof-instrumentation-readwritemerged.fdata
+```
+
+Output:
+
+```
+Using legacy profile format.
+Profile from 2 files merged.
+```
+
+This creates a single merged profile (`profile-merged.fdata`) covering both read-only and write-only workload behaviors.
+
+---
+
+### Step 4: Verify the Merged Profile
+
+Check the merged `.fdata` file:
+
+```bash
+ls -lh /path/to/profile-merged.fdata
+```
+
+---
+### Step 5: Generate the Final Binary with the Merged Profile
+
+Use LLVM-BOLT to generate the final optimized binary using the merged `.fdata` file:
+
+```bash
+llvm-bolt build/bin/mysqld \\
+  -o build/bin/mysqldreadwrite_merged.bolt_instrumentation \\
+  -data=/home/ubuntu/mysql-server-8.0.33/sysbench/prof-instrumentation-readwritemerged.fdata \\
+  -reorder-blocks=ext-tsp \\
+  -reorder-functions=hfsort \\
+  -split-functions \\
+  -split-all-cold \\
+  -split-eh \\
+  -dyno-stats \\
+  --print-profile-stats 2>&1 | tee bolt_orig.log
+```
+
+This command optimizes the binary layout based on the merged workload profile, creating a single binary (`mysqldreadwrite_merged.bolt_instrumentation`) that is optimized across both features.
+
+