Allow using HARNESS environment variable to select harness

This allow `ruby` script direct calls to still select the harness without
using any wrapper script.

Author: rafaelfranca

README.md

@@ -93,13 +93,13 @@ To run one or more specific benchmarks and record the data:
 
 ### Running a single benchmark
 
-The easiest way to run a single benchmark once is using the `run_once.rb` script:
+This is the easiest way to run a single benchmark.
+It requires no setup at all and assumes nothing about the Ruby you are benchmarking.
+It's also convenient for profiling, debugging, etc, especially since all benchmarked code runs in that process.
 
+```bash
+ruby benchmarks/fib.rb
 ```
-./run_once.rb benchmarks/some_benchmark.rb
-```
-
-This automatically sets the environment to run the benchmark once (no warmup iterations).
 
 ### Benchmark organization
 
@@ -225,34 +225,35 @@ You can find several test harnesses in the `harness/` directory:
 * `chain` - a harness to chain multiple harnesses together
 * `mplr` - a harness for multiple iterations with time limits
 
-To use a specific harness, use the `run_once.rb` script:
+### Selecting a harness
 
-```
-# Use default harness
-./run_once.rb benchmarks/railsbench/benchmark.rb
+**Using the `HARNESS` environment variable**
 
-# Use the 'once' harness
-./run_once.rb --harness=once benchmarks/railsbench/benchmark.rb
+```bash
+# Run with specific harness
+HARNESS=perf ruby benchmarks/fib.rb
+HARNESS=stackprof ruby benchmarks/railsbench/benchmark.rb
+HARNESS=vernier ruby benchmarks/optcarrot/benchmark.rb
 
-# Use the 'perf' harness
-./run_once.rb --harness=perf benchmarks/railsbench/benchmark.rb
+# Combine with Ruby options
+HARNESS=perf ruby --yjit benchmarks/fib.rb
+HARNESS=once ruby --yjit-stats benchmarks/railsbench/benchmark.rb
+```
 
-# Use the 'stackprof' harness
-./run_once.rb --harness=stackprof benchmarks/railsbench/benchmark.rb
+**Alternative: Use `run_once.rb` with `--harness` option**:
 
-# Use the 'vernier' harness
-./run_once.rb --harness=vernier benchmarks/railsbench/benchmark.rb
+```bash
+./run_once.rb --harness=perf benchmarks/railsbench/benchmark.rb
+./run_once.rb --harness=stackprof benchmarks/fib.rb
 
-# Pass Ruby options like --yjit (use -- separator)
+# With Ruby options (use -- separator)
 ./run_once.rb -- --yjit benchmarks/railsbench/benchmark.rb
-
-# Combine harness option with Ruby options
-./run_once.rb --harness=default -- --yjit-stats benchmarks/railsbench/benchmark.rb
+./run_once.rb --harness=perf -- --yjit-stats benchmarks/fib.rb
 ```
 
 When using `run_benchmarks.rb`, you can specify a harness with the `--harness` option:
 
-```
+```bash
 ./run_benchmarks.rb --harness=once
 ./run_benchmarks.rb --harness=perf
 ```
@@ -278,10 +279,13 @@ You can also use `--warmup`, `--bench`, or `--once` to set these environment var
 ./run_benchmarks.rb railsbench --once
 ```
 
-You can also use the `run_once.rb` script to run benchmarks just once, for example
-with the `--yjit-stats` command-line option:
+You can also run a single benchmark directly with Ruby:
 
-```
+```bash
+# Run once with YJIT stats
+ruby --yjit-stats benchmarks/railsbench/benchmark.rb
+
+# Or use run_once.rb script
 ./run_once.rb -- --yjit-stats benchmarks/railsbench/benchmark.rb
 ```
 
@@ -290,13 +294,12 @@ with the `--yjit-stats` command-line option:
 There is also a harness to use Linux perf. By default, it only runs a fixed number of iterations.
 If `PERF` environment variable is present, it starts the perf subcommand after warmup.
 
-```sh
+```bash
 # Use `perf record` for both warmup and benchmark
-perf record ./run_once.rb --harness=perf -- --yjit-perf=map benchmarks/railsbench/benchmark.rb
+HARNESS=perf perf record ruby --yjit-perf=map benchmarks/railsbench/benchmark.rb
 
 # Use `perf record` only for benchmark
-PERF=record ./run_once.rb --harness=perf -- --yjit-perf=map benchmarks/railsbench/benchmark.rb
-```
+HARNESS=perf PERF=record ruby --yjit-perf=map benchmarks/railsbench/benchmark.rb
 
 This is the only harness that uses `run_benchmark`'s argument, `num_itrs_hint`.
 

harness/perf.rb

@@ -3,7 +3,7 @@
 # This is a relatively minimal harness meant for use with Linux perf(1).
 # Example usage:
 #
-#    $ PERF='record -e cycles' ./run_once.rb --harness=perf benchmarks/fib.rb
+#    $ PERF='record -e cycles' HARNESS=perf ruby benchmarks/fib.rb
 #
 # When recording with perf(1), make sure the benchmark runs long enough; you
 # can tweak the MIN_BENCH_ITRS environment variable to lengthen the run. A race

harness/stackprof.rb

@@ -3,8 +3,8 @@
 # Profile the benchmark (ignoring initialization code) with stackprof.
 # Customize stackprof options with an env var of STACKPROF_OPTS='key:value,...'.
 # Usage:
-# STACKPROF_OPTS='mode:object' ./run_once.rb --harness=stackprof benchmarks/.../benchmark.rb
-# STACKPROF_OPTS='mode:cpu,interval:10' MIN_BENCH_ITRS=10 ./run_once.rb --harness=stackprof benchmarks/.../benchmark.rb
+# STACKPROF_OPTS='mode:object' HARNESS=stackprof ruby benchmarks/.../benchmark.rb
+# STACKPROF_OPTS='mode:cpu,interval:10' MIN_BENCH_ITRS=10 HARNESS=stackprof ruby benchmarks/.../benchmark.rb
 
 require_relative "../lib/harness"
 require_relative "../lib/harness/extra"

harness/vernier.rb

@@ -3,8 +3,8 @@
 # Profile the benchmark (ignoring initialization code) using vernier and display the profile.
 # Set NO_VIERWER=1 to disable automatically opening the profile in a browser.
 # Usage:
-# ./run_once.rb --harness=vernier benchmarks/...
-# NO_VIEWER=1 ./run_once.rb --harness=vernier benchmarks/...
+# HARNESS=vernier ruby benchmarks/...
+# NO_VIEWER=1 HARNESS=vernier ruby benchmarks/...
 
 require_relative "../lib/harness"
 require_relative "../lib/harness/extra"

lib/harness/loader.rb

@@ -1,22 +1,33 @@
-# Use harness/default.rb by default. You can change it with the --harness option in run_once.rb
-# or with -r option when calling Ruby directly.
+# Use harness/default.rb by default. You can change it with:
+# 1. HARNESS environment variable (preferred, keeps harness close to code)
+# 2. --harness option in run_once.rb
+# 3. -r option when calling Ruby directly
+#
 # Examples:
-#   ./run_once.rb benchmarks/railsbench/benchmark.rb                  # uses harness/default.rb
-#   ./run_once.rb --harness=once benchmarks/railsbench/benchmark.rb   # uses harness/once.rb
+#   HARNESS=perf ruby benchmarks/railsbench/benchmark.rb              # uses harness/perf.rb
+#   HARNESS=once ruby benchmarks/railsbench/benchmark.rb              # uses harness/once.rb
 #   ./run_once.rb --harness=ractor benchmarks/railsbench/benchmark.rb # uses harness/ractor.rb
 
 # Only load the default harness if no other harness has defined run_benchmark
 unless defined?(run_benchmark)
+  harness_name = ENV['HARNESS'] || 'default'
+
   retries = 0
   begin
-    require "default"
+    require harness_name
   rescue LoadError => e
-    if retries == 0 && e.path == "default"
+    if retries == 0 && e.path == harness_name
       retries += 1
       # Add the harness directory to the load path
       $LOAD_PATH << File.expand_path("../../harness", __dir__)
       retry
     end
+    # Provide helpful error message for invalid harness
+    if e.path == harness_name
+      harness_dir = File.expand_path("../../harness", __dir__)
+      available = Dir.glob("#{harness_dir}/*.rb").map { |f| File.basename(f, '.rb') }.sort
+      raise LoadError, "Harness '#{harness_name}' not found. Available harnesses: #{available.join(', ')}"
+    end
     raise
   end
 end

test/run_once_test.rb

@@ -8,10 +8,11 @@
     @script_path = File.expand_path('../run_once.rb', __dir__)
     @original_env = ENV.to_h
 
-    # Create a temp directory with test benchmarks
     @tmpdir = Dir.mktmpdir
     @test_benchmark = File.join(@tmpdir, 'test_benchmark.rb')
+    loader_path = File.expand_path('../lib/harness/loader', __dir__)
     File.write(@test_benchmark, <<~RUBY)
+      require "#{loader_path}"
       puts "Benchmark executed"
       puts "WARMUP_ITRS=\#{ENV['WARMUP_ITRS']}"
       puts "MIN_BENCH_ITRS=\#{ENV['MIN_BENCH_ITRS']}"
@@ -21,12 +22,10 @@
 
   after do
     FileUtils.rm_rf(@tmpdir) if @tmpdir && Dir.exist?(@tmpdir)
-    # Restore original environment
     ENV.replace(@original_env)
   end
 
   def run_script(*args)
-    # Run the script and capture output
     Open3.capture3('ruby', @script_path, *args)
   end
 
@@ -71,7 +70,6 @@ def run_script(*args)
     end
 
     it 'loads custom harness when specified' do
-      # Create a test harness
       harness_dir = File.join(File.dirname(@script_path), 'harness')
       FileUtils.mkdir_p(harness_dir)
       test_harness = File.join(harness_dir, 'test_harness.rb')
@@ -90,6 +88,26 @@ def run_script(*args)
       end
     end
 
+    it 'respects HARNESS environment variable when no --harness option provided' do
+      harness_dir = File.join(File.dirname(@script_path), 'harness')
+      FileUtils.mkdir_p(harness_dir)
+      test_harness = File.join(harness_dir, 'test_harness.rb')
+
+      begin
+        File.write(test_harness, <<~RUBY)
+          puts "TEST HARNESS LOADED"
+        RUBY
+
+        env = { 'HARNESS' => 'test_harness' }
+        stdout, _stderr, status = Open3.capture3(env, 'ruby', @script_path, @test_benchmark)
+
+        assert status.success?
+        assert_match(/TEST HARNESS LOADED/, stdout)
+      ensure
+        File.delete(test_harness) if File.exist?(test_harness)
+      end
+    end
+
     it 'shows error for non-existent harness' do
       stdout, _stderr, status = run_script('--harness=nonexistent', @test_benchmark)
 
@@ -101,23 +119,19 @@ def run_script(*args)
 
   describe 'ractor benchmark detection' do
     it 'automatically uses ractor harness for ractor benchmarks' do
-      # Create a ractor benchmark
       ractor_dir = File.join(@tmpdir, 'benchmarks-ractor', 'test')
       FileUtils.mkdir_p(ractor_dir)
       ractor_benchmark = File.join(ractor_dir, 'benchmark.rb')
       File.write(ractor_benchmark, 'puts "Ractor benchmark"')
 
       _stdout, _stderr, _status = run_script(ractor_benchmark)
 
-      # The script will try to load ractor harness
-      # We just verify it detected the ractor path
       assert_match(/benchmarks-ractor/, ractor_benchmark)
     end
   end
 
   describe 'Ruby options pass-through' do
     it 'passes Ruby options after -- separator' do
-      # Create a benchmark that checks for a warning flag
       warning_benchmark = File.join(@tmpdir, 'warning_test.rb')
       File.write(warning_benchmark, <<~RUBY)
         x = 1
@@ -187,7 +201,6 @@ def run_script(*args)
 
   describe 'script examples' do
     it 'works with simple benchmark path' do
-      # Simulates: ./run_once.rb benchmarks/fib.rb
       fib_benchmark = File.join(@tmpdir, 'fib.rb')
       File.write(fib_benchmark, 'puts "Fibonacci benchmark"')
 
@@ -198,15 +211,13 @@ def run_script(*args)
     end
 
     it 'works with harness option' do
-      # Simulates: ./run_once.rb --harness=default benchmarks/fib.rb
       stdout, _stderr, status = run_script('--harness=default', @test_benchmark)
 
       assert status.success?
       assert_match(/Benchmark executed/, stdout)
     end
 
     it 'works with Ruby options after -- separator' do
-      # Simulates: ./run_once.rb -- --yjit benchmarks/fib.rb
       stdout, _stderr, status = run_script('--', '--yjit', @test_benchmark)
 
       assert status.success?
@@ -228,7 +239,6 @@ def run_script(*args)
     end
 
     it 'identifies first .rb file as benchmark' do
-      # Even if multiple .rb files mentioned (shouldn't happen), first one wins
       other_file = File.join(@tmpdir, 'other.rb')
       File.write(other_file, 'puts "Wrong file"')
 
@@ -246,4 +256,36 @@ def run_script(*args)
       assert_match(/invalid option|Error/, stdout)
     end
   end
+
+  describe 'direct ruby execution with HARNESS env var' do
+    it 'uses HARNESS environment variable when running benchmark directly' do
+      harness_dir = File.join(File.dirname(@script_path), 'harness')
+      FileUtils.mkdir_p(harness_dir)
+      direct_harness = File.join(harness_dir, 'direct_test.rb')
+
+      begin
+        File.write(direct_harness, <<~RUBY)
+          puts "DIRECT HARNESS RUNNING"
+        RUBY
+
+        env = { 'HARNESS' => 'direct_test' }
+        stdout, _stderr, status = Open3.capture3(env, 'ruby', @test_benchmark)
+
+        assert status.success?, "Direct Ruby execution should work with HARNESS env var"
+        assert_match(/DIRECT HARNESS RUNNING/, stdout)
+        assert_match(/Benchmark executed/, stdout)
+      ensure
+        File.delete(direct_harness) if File.exist?(direct_harness)
+      end
+    end
+
+    it 'shows helpful error for invalid HARNESS value' do
+      env = { 'HARNESS' => 'nonexistent_harness' }
+      _stdout, stderr, status = Open3.capture3(env, 'ruby', @test_benchmark)
+
+      refute status.success?
+      assert_match(/Harness 'nonexistent_harness' not found/, stderr)
+      assert_match(/Available harnesses/, stderr)
+    end
+  end
 end