Scala 3 Compiler Benchmarks

JMH benchmarks for measuring Scala 3 compiler performance.

Dependencies

• JVMs >= 25.
• Coursier (or you can comment the lines that use cs in run.sh to use your local JVM).

Benchmarks

Compilation benchmarks

Big benchmarks (multi-file)

Sources are vendored (copied directly into this repository) and fixed to compile without errors or warnings across all Scala versions from 3.3.3 to nightly. Fixed versions ensure comparable benchmark results.

Project	Version	LOC	Dependencies	Tests
cask app	-	115	cask, scalatags	no
dotty util	6462d7d7	2'209	none	no
fansi	0.5.1	960	sourcecode, utest	yes
indigo ⚡	0.22.0	25'270	scalajs-dom, ultraviolet	no
re2s	1d2b8962	9'021	none	no
scala-parallel-collections	v1.2.0	8'887	junit	yes
scala-parser-combinators	2.4.0	2'325	junit	yes
scala.today	2dd97e7	1'103	tapir, ox, magnum, etc.	no
scala-yaml	0.3.1	6'473	pprint, munit	yes
scalaz	v7.2.36	27'757	none	no
sourcecode	0.4.4	638	none	yes
tasty-query	v1.6.1	13'482	none	no
tictactoe	6873dfd	441	cats-effect, cats-core	yes

⚡ = Scala.js benchmark (requires Scala 3.6.4+)
LOC = lines of Scala code (reported by cloc)

Small benchmarks (single-file)

The remaining benchmarks target specific compiler aspects (pattern matching, implicit resolution, inlining, etc.). Most are adapted from the previous benchmark suite.

Runtime benchmarks

• Are We Fast Yet?: classic benchmark suite ported to Scala, including Bounce, Brainfuck, CD, DeltaBlue, GCBench, Json, Kmeans, List, Mandelbrot, NBody, Permute, Queens, Richards, and Tracer.
• Optimizer: small benchmarks used to track the effect of the Scala 3 optimizer.
• Librairies: runtime benchmarks using libraries such as scala-yaml and scala-parser-combinators.

Quick Start

# Run benchmarks for multiple versions with interleaved runs
./run.sh --versions 3.3.4 3.7.4 --jvm temurin:25 --runs 3

# Or run manually with sbt
sbt -Dcompiler.version=3.3.4 "clean; bench / Jmh / run -gc true -foe true"

Structure of this Repository

bench-sources/
  small/                # Synthetic single-file benchmarks
    helloWorld.scala
    ...
  dottyUtil/            # Real-world multi-file benchmarks (each is an SBT subproject)
  ...
bench/scala/
  CompilationBenchmarksSmallNightly.scala
  ...
visualizer/              # Web app for visualizing results (see visualizer/README.md)

Results structure

Results are stored as CSV files in the scala3-benchmarks-data repository. The scripts/importResults.scala script converts JMH JSON output into two forms: raw data (one file per run) and aggregated summaries (one file per metric/benchmark pair, for use by the visualizer).

Raw Data

raw/<machine>/<jvm>/<patch_version>/<version>/<run_datetime>.csv

Each CSV file contains one row per benchmark from a single JMH run. An INDEX file in each leaf directory lists all run files.

Columns:

• suite: benchmark suite class name (e.g. CompilationBenchmarksSmallNightly).
• benchmark: unqualified @Benchmark method name (e.g. helloWorld).
• warmup_iterations: number of warmup iterations before measurement.
• batch_size: number of benchmark invocations per iteration. This is typically 1 for big benchmarks, and higher for smaller benchmarks.
• times: space-separated measurement times in milliseconds (one per iteration). The number of values is the number of measurement iterations. Benchmarks use SingleShotTime mode, so each value is a single invocation. See JMH @BenchmarkMode.
• allocs_min, allocs_avg, allocs_max: total allocation per operation in MB, from the gc.alloc.rate.norm secondary metric of JMH's -prof gc (GcProfiler). The raw value (bytes) is divided by 1e6.
• gc_min, gc_avg, gc_max: number of GC events during measurement, from the gc.count secondary metric of -prof gc.
• comp_min, comp_avg, comp_max: JVM JIT compilation time in milliseconds during the measurement window, from the compiler.time.profiled secondary metric of -prof comp (CompilerProfiler). High values indicate the JIT was still active during measurement, which can interfere with results. In practice this represents 10-20% of total measured time. Its reliability is uncertain.

Aggregated Data

aggregated/<machine>/<jvm>/<patch_version>/<metric>/<suite>/<benchmark>.csv

Pre-computed summaries derived from raw data, organized per metric, suite, and benchmark for direct use by the visualiser. Each <metric> is one of time, allocs, gc, or comp. The <suite> corresponds to the benchmark suite class name (e.g. CompilationBenchmarksSmallNightly).

Columns:

• version: Scala version string.
• count: total number of measurement iterations across all runs for this version.
• min: minimum value across all iterations.
• avg: weighted average across all iterations.
• max: maximum value across all iterations.

When multiple runs exist for the same version, stats are merged incrementally (combined average weighted by count, min/max taken across all runs).

Adding Benchmarks

Benchmarks should:

• compile with all version between Scala 3.3.2 and the latest
• compile in ~100ms-10s range (after warmup)
• not require complex setup
• Ideally not require external dependencies

Potential future benchmarks:

• quicklens (waiting 3.8)
• advent of code solutions (various authors, various sizes)

To add a new benchmark:

1. Add a .scala file to bench-sources/small/, or create a new SBT subproject in bench-sources/ for multi-file benchmarks
2. Add a @Benchmark method in CompilationBenchmarks.scala:

   @Benchmark def myBenchmark = scalac(Config.myBenchmark)

Config is auto-generated at bench/target/scala-*/src_managed/main/bench/Config.scala with the scalac arguments (classpath and sources) for each benchmark.

Running Tests

Some benchmarks (fansi, sourcecode, scalaYaml, parserCombinators) include tests from their upstream repositories:

sbt test                    # Run all tests
sbt benchFansi/test         # Run fansi tests only

Test sources are also included in benchmarks to compile both main and test code together.

Using JMH's profilers

Examples of using JMH's built-in profilers: jmh/samples/JMHSample_35_Profilers.java.

Async Profiler

Flame graphs can be generated using async-profiler. Example command:

sbt -Dcompiler.version=3.7.4 "clean; bench / Jmh / run -gc true -foe true -prof \"async:libPath=../async-profiler-4.2.1-macos/lib/libasyncProfiler.dylib;output=flamegraph;dir=profile-results\" helloWorld"

Replace 3.7.4, ../async-profiler-4.2.1-macos/lib/libasyncProfiler.dylib and helloWorld with the desired Scala version, path to the async profiler library, and benchmark name respectively. Read more at markrmiller/jmh-profilers.md.

The default sampling interval is 10ms. It can be changed by adding the interval, which is specified in nanoseconds. For example, to set the interval to 1ms, use interval=1000000.

Async-profiler options reference async-profiler/docs/ProfilerOptions.md.

Known Issues

Under Java 25, the following warning is printed during benchmark runs:

[info] WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
[info] WARNING: sun.misc.Unsafe::objectFieldOffset has been called by org.openjdk.jmh.util.Utils (file:/home/runner/work/scala3-benchmarks/scala3-benchmarks/target/bg-jobs/sbt_accfab51/target/09a4797f/1296d6b9/jmh-core-1.37.jar)
[info] WARNING: Please consider reporting this to the maintainers of class org.openjdk.jmh.util.Utils
[info] WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release

It can be ignored for now. It is fixed by openjdk/jmh#140, which will be included in the next JMH release.