Merge pull request #332 from graalvm/move-jmh-binary-tree-demo

[GR-60094] Review and Move jmh/benchmark-binary-tree demo under native-image/benchmark category.

Author: olyagpl

.github/workflows/jmh-binary-tree.yml renamed to .github/workflows/native-image-jmh-binary-tree.yml

@@ -1,42 +1,45 @@
-name: jmh/benchmark-binary-tree
+name: native-image/benchmark/jmh/binary-tree
 on:
   push:
     paths:
-      - 'jmh/benchmark-binary-tree/**'
-      - '.github/workflows/jmh-binary-tree.yml'
+      - 'native-image/benchmark/jmh/binary-tree/**'
+      - '.github/workflows/native-image-jmh-binary-tree.yml'
   pull_request:
     paths:
-      - 'jmh/benchmark-binary-tree/**'
-      - '.github/workflows/jmh-binary-tree.yml'
+      - 'native-image/benchmark/jmh/binary-tree/**'
+      - '.github/workflows/native-image-jmh-binary-tree.yml'
   schedule:
     - cron: "0 0 1 * *" # run every month
   workflow_dispatch:
 permissions:
   contents: read
 jobs:
   build:
-    name: GraalVM Binary Tree Benchmark
+    name: JMH Binary Tree Benchmark
     runs-on: ubuntu-latest
     timeout-minutes: 45
+    strategy:
+      matrix:
+        java-version: ['21', '24-ea']
     steps:
       - uses: actions/checkout@v4
       - uses: graalvm/setup-graalvm@v1
         with:
-          java-version: '21.0.2'
+          java-version: ${{ matrix.java-version }}
           distribution: 'graalvm'
           github-token: ${{ secrets.GITHUB_TOKEN }}
       - name: Build and Test Java Code
         run: |
-          cd jmh/benchmark-binary-tree
-          # build the Java code
+          cd native-image/benchmark/jmh/binary-tree
+          # Build the Java code
           ./mvnw --no-transfer-progress clean package
-          # build the native exe
+          # Build the native executable
           ./mvnw --no-transfer-progress -Pnative -DskipNativeTests package
           # Could just replace with a check that the binary exists?
           ./target/benchmark-binary-tree
-          # build and run the instrumented native exe
+          # Build and run the instrumented native executable
           ./mvnw --no-transfer-progress -Pinstrumented -DskipNativeTests package
           ./target/benchmark-binary-tree-instr
-          # build and run the optimised native exe
+          # Build and run the optimized native executable
           ./mvnw --no-transfer-progress -Poptimised -DskipNativeTests package
           ./target/benchmark-binary-tree-opt

jmh/benchmark-binary-tree/README.md

@@ -5,14 +5,14 @@
 # to you under the Apache License, Version 2.0 (the
 # "License"); you may not use this file except in compliance
 # with the License.  You may obtain a copy of the License at
-# 
+#
 #   http://www.apache.org/licenses/LICENSE-2.0
-# 
+#
 # Unless required by applicable law or agreed to in writing,
 # software distributed under the License is distributed on an
 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.8.6/apache-maven-3.8.6-bin.zip
-wrapperUrl=https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.1.0/maven-wrapper-3.1.0.jar
+distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.8.5/apache-maven-3.8.5-bin.zip
+wrapperUrl=https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.2.0/maven-wrapper-3.2.0.jar

jmh/benchmark-binary-tree/.gitignore renamed to native-image/benchmark/jmh/binary-tree/.gitignore

@@ -0,0 +1,108 @@
+# Binary Tree Benchmark 
+
+This demo shows how to run a Java Microbenchmark Harness (JMH) benchmark as a native executable.
+
+To build a native executable version of this benchmark you need to run the [Tracing Agent](https://www.graalvm.org/latest/reference-manual/native-image/metadata/AutomaticMetadataCollection/) to supply the reflection configuration to the `native-image` builder.
+This has already been done for you to save time and the generated configuration can be found in _src/main/resources/META-INF/native-image/_.
+
+> **Note:** To generate the configuration yourself, ensure that the JMH `fork` parameter is set to `0`, which can be performed from the command line using the option  `-f 0`. It can also be achieved within the code by using the `@Fork` annotation.
+
+## Important Notes on Using JMH with GraalVM Native Image
+
+To make a JMH benchmark run as a native image, you can not fork the benchmark process in the same way as JMH does when running on the JVM.
+When running on HotSpot, JMH will fork a new JVM for each benchmark to ensure there is no interference in the measurements for each benchmark.
+This forking process is not possible with GraalVM Native Image and you should consider the following guidance when building JMH benchmarks that are meant to be run as native executables:
+* You should only include a single benchmark in each native executable.
+* You need to annotate the benchmark with `@Fork(0)` to ensure that the benchmark is not forked.
+* If you want to profile the benchmark to generate an optimized version of it, you should, obviously, ignore the benchmark results whilst profiling.
+
+## Preparation
+
+1. Download and install the GraalVM JDK using [SDKMAN!](https://sdkman.io/). For other installation options, visit the [Downloads page](https://www.graalvm.org/downloads/).
+    ```bash
+    sdk install java 21.0.5-graal
+    ```
+
+2. Download or clone the repository and navigate into the _/native-image/benchmark/jmh/binary-tree_ directory:
+    ```bash
+    git clone https://github.com/graalvm/graalvm-demos
+    ```
+    ```bash
+    cd graalvm-demos/native-image/benchmark/jmh/binary-tree
+    ```
+
+## Build and Run the Benchmark on HotSpot
+
+To build and run the benchmark on HotSpot, run the following Maven command:
+```shell
+./mvnw clean package exec:exec
+```
+
+Note that within the _pom.xml_ file there are instructions to explicitly turn off the Graal JIT compiler using the option `-XX:-UseJVMCICompiler`. 
+This means that benchmark will run using the C2 JIT compiler.
+
+The application runs the benchmark and displays the results to the terminal.
+**The final result is the most significant.**
+You should see the result similar to this:
+```shell
+Benchmark          (binaryTreesN)   Mode  Cnt    Score   Error  Units
+BinaryTrees.bench              14  thrpt    6  348.923 ± 21.343  ops/s
+```
+
+## Build and Run the Benchmark from a Native Executable
+
+Now build a native executable using Native Image.
+This demo uses Oracle GraalVM Native Image, however, if you are using GraalVM Community, you may see lower figures for throughput.
+
+1. Build a native executable, run the following command:
+    ```shell
+    ./mvnw package -Pnative
+    ```
+2. Run the benchmark from a native executable: 
+    ```shell
+    ./target/benchmark-binary-tree
+    ```
+    You should see similar results:
+    ```shell
+    Benchmark          (binaryTreesN)   Mode  Cnt    Score   Error  Units
+    BinaryTrees.bench              14  thrpt    6  282.335 ± 5.644  ops/s
+    ```
+
+## Optimize the Benchmark for Throughput
+
+You can improve the performance of this benchmark by applying [Profile-Guided Optimization (PGO)](https://www.graalvm.org/reference-manual/native-image/optimizations-and-performance/PGO/). 
+
+> PGO is available with Oracle GraalVM Native Image.
+
+First, you will need to build an instrumented version of this native benchmark that contains extra code to trace the execution of the program and to profile it.
+Therefore, it will run slower than the previous version. 
+After execution finishes, a profile file, _default.iprof_, is generated in the root directory.
+This file contains profiling information about the application and will be used to build a more efficient native executable.
+
+1. To build the instrumented version of the native executable, run the following command:
+    ```shell
+    ./mvnw package -Pinstrumented
+    ```
+
+2. Then run it to generate the profile file:
+    ```shell
+    ./target/benchmark-binary-tree-instr
+    ```
+
+3. Now that you have the profiles, build and run the optimized version of this native benchmark:
+    ```shell
+    ./mvnw package -Poptimised
+    ```
+    ```shell
+    ./target/benchmark-binary-tree-opt
+    ```
+    You should see similar results:
+    ```shell
+    Benchmark          (binaryTreesN)   Mode  Cnt    Score   Error  Units
+    BinaryTrees.bench              14  thrpt    6  311.630 ± 3.630  ops/s
+    ```
+
+## Your Mileage May Vary
+
+The results you see will vary depending on the hardware you are running on.
+The results above are from a 2019 MacBook Pro, i7, 32 GB RAM running on Oracle GraalVM for JDK 21.0.5.