add all benchmarks, add perf scripts, remove debug logs

[jycor]

This PR changes the quick-sysbench.sh config to using logging mode info.
Debug logs can slow down benchmarks with all that additional IO, so they should be disabled while benchmarking.
The list of quick benchmarks has also been updated to include every read and write benchmark; it should take around 15 minutes to complete now.

Additionally, this PR adds some utility scripts to make it easier to profile and benchmark a single test.

[github-actions]

	Main	PR
covering_index_scan_postgres	1263.80/s	1351.18/s	+6.9%
index_join_postgres	192.73/s	194.27/s	+0.7%
index_join_scan_postgres	203.18/s	206.95/s	+1.8%
index_scan_postgres	12.05/s	12.08/s	+0.2%
oltp_point_select	2253.19/s	2389.53/s	+6.0%
oltp_read_only	1778.45/s	1869.36/s	+5.1%
select_random_points	128.79/s	124.04/s	-3.7%
select_random_ranges	846.74/s	870.65/s	+2.8%
table_scan_postgres	11.73/s	11.48/s	-2.2%
types_table_scan_postgres	5.38/s	5.34/s	-0.8%

[github-actions]

	Main	PR
Total	42090	42090
Successful	18250	18250
Failures	23840	23840
Partial Successes1	5385	5385

	Main	PR
Successful	43.3595%	43.3595%
Failures	56.6405%	56.6405%

Footnotes

1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct. ↩

[itoqa]

Ito Test Report ❌

13 test cases ran. 1 failed, 5 additional findings, 7 passed.

Across 13 test cases, 7 passed and 6 failed, indicating partial stability but significant reliability gaps centered on profiling argument handling and benchmark automation. The most important defects are a real CLI panic on malformed --prof input, four high-impact benchmark workflow issues (write-heavy workloads failing due to missing DB provisioning, fatal sysbench failures being masked, partial artifacts passing comparison, and unpinned external scripts causing drift), and a PR-introduced local_sysbench path-resolution bug from repo-root invocation, while core lifecycle checks like profiling output generation, interrupt cleanup/rerun behavior, parser-order contract enforcement, and artifact ignore behavior worked as expected.

❌ Failed (1)

Category	Summary	Screenshot
Local	🟠 Running local_sysbench.sh from repo root fails because build path resolves outside the repository.

🟠 Local sysbench root invocation path bug

• What failed: The script is expected to complete a default local sysbench run from repository root, but it fails because the build target path becomes invalid after changing directories.
• Impact: Developers cannot run the documented default local benchmark flow from repository root, which blocks reliable local performance verification unless they manually change to scripts/performance first.
• Steps to reproduce:
  1. From repository root, run ./scripts/performance/local_sysbench.sh.
  2. Let the script reach its build step after entering sbtest.
  3. Observe build failure before benchmark prepare/run can complete.
• Stub / mock context: Auth bootstrap checks were temporarily bypassed in server authentication paths to keep local benchmark setup deterministic, but the observed failure was in benchmark script path handling and is independent of auth behavior.
• Code analysis: I inspected scripts/performance/local_sysbench.sh and confirmed it changes into sbtest without anchoring to the script directory, then builds with a relative path (../../../cmd/doltgres/). When invoked from repo root, that resolves to a non-existent /tmp/cmd/doltgres path in this workspace layout, causing the benchmark workflow to fail before execution.
• Why this is likely a bug: The script's own path logic makes the default invocation from repository root fail deterministically due to incorrect relative path resolution in production script code.

Relevant code:

scripts/performance/local_sysbench.sh (lines 21-29)

mkdir -p sbtest
cd sbtest

if [ ! -d "./sysbench-lua-scripts" ]; then
  git clone https://github.com/dolthub/sysbench-lua-scripts.git
fi
cp ./sysbench-lua-scripts/*.lua ./

go build -o doltgres.exe ../../../cmd/doltgres/

✅ Passed (7)

Category	Summary	Screenshot
Artifact	After running scripts/performance/local_sysbench.sh in the nested container and then checking git status --short at repo root, scripts/performance/sbtest artifacts were not listed as untracked changes, confirming ignore behavior.
Benchmark	Generated benchmark config sets log_level: info for benchmark server startup.
Local	Profiled run completed; run phase enabled CPU profiling and produced cpu.pprof.
Local	SIGINT interruption left no stale doltgres process; immediate rerun started cleanly.
Local	Immediate rerun after interruption completed cleanly with no leftover process contamination.
Profiling	Profiling startup accepted --prof cpu while still honoring -config=dolt-config.yaml and reached ready state.
Profiling	Interleaved order failed deterministically with a clear flag error while grouped profiling flags and config flag succeeded, matching a strict ordering contract.

ℹ️ Additional Findings (5)

These findings are unrelated to the current changes but were observed during testing.

Category	Summary	Screenshot
Benchmark	⚠️ Write-heavy benchmark workloads are listed but fail because the configured benchmark database is not ensured before sysbench prepare/run.
Benchmark	⚠️ The benchmark script can report success even when sysbench emits fatal connection failures.
Benchmark	⚠️ Benchmark comparison accepts partial artifacts and returns a table header instead of failing incompleteness.
Benchmark	🟠 External benchmark Lua scripts are fetched from an unpinned revision, allowing run-to-run semantic drift.
Profiling	🟠 Malformed profiling args can panic in the CLI pre-parse path instead of failing with a controlled user-facing argument error.

⚠️ Write heavy workloads fail to execute

• What failed: The benchmark loop includes write-heavy workloads but calls sysbench against --pgsql-db="postgres" without ensuring that database exists in the loop workspace, so workloads are not successfully executed.
• Impact: The core quick benchmark workflow cannot reliably execute newly added write-heavy scenarios, so benchmark results are incomplete and unreliable for PR evaluation.
• Steps to reproduce:
  1. Run ./scripts/quick_sysbench.sh from the repository root.
  2. Allow the loop to execute write-heavy workloads such as oltp_insert, oltp_read_write, and oltp_write_only.
  3. Observe sysbench prepare/run calls targeting --pgsql-db="postgres" and resulting fatal benchmark failures.
• Stub / mock context: Authorization checks were bypassed for deterministic benchmark startup by forcing auth short-circuits in server/auth/auth_handler.go, server/auth/init.go, and server/authentication_scram.go; benchmark behavior was then validated on that local auth-bypassed server path.
• Code analysis: I reviewed the benchmark loop and found that each iteration resets local state and immediately runs sysbench prepare/run against a hardcoded database name with no creation/validation step.
• Why this is likely a bug: The script’s production benchmark path repeatedly targets a hardcoded database name without provisioning/validating it, which directly explains persistent fatal benchmark failures.

Relevant code:

scripts/quick_sysbench.sh (lines 64-79)

rm -rf ./.dolt
rm -rf ./postgres
./doltgres.exe -config="dolt-config.yaml" 2> prepare.log &
SERVER_PID="$!"

sleep 1
echo "----$SYSBENCH_TEST----"
sysbench \
  --db-driver="pgsql" \
  --pgsql-host="0.0.0.0" \
  --pgsql-port="$PORT" \
  --pgsql-user="postgres" \
  --pgsql-password="password" \
  --pgsql-db="postgres" \
  "$SYSBENCH_TEST" prepare

scripts/quick_sysbench.sh (lines 87-97)

sysbench \
  --db-driver="pgsql" \
  --pgsql-host="0.0.0.0" \
  --pgsql-port="$PORT" \
  --pgsql-user="postgres" \
  --pgsql-password="password" \
  --pgsql-db="postgres" \
  --time=15 \
  --db-ps-mode=disable \
  "$SYSBENCH_TEST" run 1>> results.log

⚠️ Benchmark script masks fatal failures

• What failed: The script does not enforce fail-fast behavior and uses fixed one-second readiness waits, so fatal sysbench connection failures can occur while the script still completes as if successful.
• Impact: CI and local benchmark runs can be marked successful despite failed benchmark execution, which can hide regressions and mislead performance decisions.
• Steps to reproduce:
  1. Run ./scripts/quick_sysbench.sh in a constrained environment where startup can be slower.
  2. Inspect sysbench invocations immediately after each fixed sleep 1 handoff.
  3. Compare observed fatal connection errors with final script completion status.
• Stub / mock context: Authorization checks were bypassed for deterministic benchmark startup by forcing auth short-circuits in server/auth/auth_handler.go, server/auth/init.go, and server/authentication_scram.go; benchmark behavior was then validated on that local auth-bypassed server path.
• Code analysis: I inspected startup/error control and found set -e and pipefail disabled, plus fixed sleep-based handoffs before sysbench calls with no readiness probe.
• Why this is likely a bug: The script omits hard failure semantics and readiness verification in the production execution path, so benchmark failures can be silently masked as success.

Relevant code:

scripts/quick_sysbench.sh (lines 1-3)

#!/bin/bash
#set -e
#set -o pipefail

scripts/quick_sysbench.sh (lines 66-71)

./doltgres.exe -config="dolt-config.yaml" 2> prepare.log &
SERVER_PID="$!"

sleep 1
echo "----$SYSBENCH_TEST----"
sysbench \

scripts/quick_sysbench.sh (lines 83-89)

./doltgres.exe -config="dolt-config.yaml" 2> run.log &
SERVER_PID="$!"
sleep 1

sysbench \
  --db-driver="pgsql" \
  --pgsql-host="0.0.0.0" \

⚠️ Partial artifacts pass benchmark comparison

• What failed: The workflow only checks whether both files exist, and the comparator prints a markdown header even when parsed sections are missing, so partial artifacts can be treated as valid comparison input.
• Impact: PR benchmark comments can look valid while silently omitting missing benchmark sections, leading to incorrect performance conclusions.
• Steps to reproduce:
  1. Create one complete benchmark output file and one intentionally partial benchmark output file.
  2. Run the benchmark comparison command go run ./testing/go/benchmark with those artifacts in place.
  3. Observe that comparison output still renders table structure instead of failing for incomplete data.
• Stub / mock context: Authorization checks were bypassed for deterministic benchmark startup by forcing auth short-circuits in server/auth/auth_handler.go, server/auth/init.go, and server/authentication_scram.go; benchmark behavior was then validated on that local auth-bypassed server path.
• Code analysis: I reviewed CI gating and comparator behavior and found no completeness validation before publishing benchmark markdown output.
• Why this is likely a bug: The CI path validates only artifact existence and not parse completeness, so incomplete benchmark data can pass through as if it were a valid comparison.

Relevant code:

.github/workflows/mini-sysbench.yml (lines 55-67)

- name: Check Sysbench Logs
  id: check_logs
  run: |
    cd scripts/mini_sysbench
    if [[ -f "results1.log" && -f "results2.log" ]]; then
      echo "logs_exist=true" >> $GITHUB_OUTPUT
      echo "logs exist"
    else
      echo "logs_exist=false" >> $GITHUB_OUTPUT
      echo "One of the branches could not successfully run the benchmarks."
      echo "Please review them for errors, which should be fixed."
      exit 1
    fi

testing/go/benchmark/main.go (lines 41-47)

prSections := SectionResults(string(prBenchmark))
mainSections := SectionResults(string(mainBenchmark))
sb := strings.Builder{}
sb.WriteString("|   | Main | PR |   |\n")
sb.WriteString("| --- | --- | --- | --- |\n")
for _, kv := range GetMapKVsSorted(mainSections) {
  mainSection := kv.Value

🟠 Unpinned external scripts cause drift

• What failed: The benchmark script clones an external repository without pinning a commit/tag, and the workflow does not persist/report the fetched revision, so upstream changes can alter benchmark semantics between reruns.
• Impact: Benchmark deltas can drift between runs of the same PR commit due to external script changes, reducing reproducibility and confidence in reported performance trends.
• Steps to reproduce:
  1. Run the quick benchmark flow at a fixed source commit and note external workload script revision provenance.
  2. Force a fresh external script fetch and rerun the same benchmark flow without changing repository code.
  3. Compare revision capture behavior and resulting benchmark semantics between runs.
• Stub / mock context: Authorization checks were bypassed for deterministic benchmark startup by forcing auth short-circuits in server/auth/auth_handler.go, server/auth/init.go, and server/authentication_scram.go; benchmark behavior was then validated on that local auth-bypassed server path.
• Code analysis: I inspected dependency-fetch logic and workflow steps and found no revision pinning or provenance capture for the cloned benchmark scripts.
• Why this is likely a bug: The benchmark workflow depends on mutable external scripts without revision control, which makes performance comparisons non-deterministic for identical source commits.

Relevant code:

scripts/quick_sysbench.sh (lines 13-16)

if [ ! -d "./sysbench-lua-scripts" ]; then
  git clone https://github.com/dolthub/sysbench-lua-scripts.git
fi
cp ./sysbench-lua-scripts/*.lua ./

.github/workflows/mini-sysbench.yml (lines 34-53)

- name: Test PR branch
  id: test_doltgresql_pr
  continue-on-error: true
  run: |
    ./postgres/parser/build.sh
    ./scripts/quick_sysbench.sh
    mv ./scripts/mini_sysbench/results.log ./scripts/mini_sysbench/results1.log
    cat ./scripts/mini_sysbench/results1.log

- name: Test main branch
  id: test_doltgresql_main
  continue-on-error: true
  run: |
    git reset --hard
    git fetch --all --unshallow
    git checkout origin/main

🟠 Malformed profiling flag causes panic

• What failed: A malformed profiling invocation (--prof followed by a non-profile token) triggers a panic (Unexpected prof flag) rather than returning a user-facing argument validation failure.
• Impact: Invalid CLI input can crash startup unexpectedly for profiling flows, disrupting local benchmark and profiling workflows. Users lose clear remediation guidance because the process panics instead of returning a structured flag error.
• Steps to reproduce:
  1. Build cmd/doltgres.
  2. Run ./doltgres.exe --prof -config=dolt-config.yaml.
  3. Observe panic termination instead of a clean validation error.
• Stub / mock context: Authorization checks were temporarily bypassed during startup by disabling auth enforcement in server/auth/auth_handler.go, server/auth/init.go, and server/authentication_scram.go so CLI profiling argument behavior could be exercised deterministically.
• Code analysis: I inspected the production CLI pre-parser in cmd/doltgres/main.go and confirmed it directly indexes args[1] in profiling branches and calls panic on unsupported values; there is no safe validation/error return path for malformed profiling tokens.
• Why this is likely a bug: The production parser panics on malformed --prof input instead of returning a controlled validation error, which violates expected CLI error-handling behavior.

Relevant code:

cmd/doltgres/main.go (lines 156-166)

if len(args) >= 2 {
	profilingOptions := utils.ProfilingOptions{}
	doneDebugFlags := false
	for !doneDebugFlags && len(args) > 0 {
		switch args[0] {
		case profilePath:
			profilingOptions.Path = args[1]
			if _, err := os.Stat(profilingOptions.Path); errors.Is(err, os.ErrNotExist) {
				panic(errors.Errorf("profile path does not exist: %s", profilingOptions.Path))
			}

cmd/doltgres/main.go (lines 167-180)

case profFlag:
			switch args[1] {
			case cpuProf:
				profilingOptions.CPU = true
			case memProf:
				profilingOptions.Memory = true
			case blockingProf:
				profilingOptions.Block = true
			case traceProf:
				profilingOptions.Trace = true
			default:
				panic("Unexpected prof flag: " + args[1])
			}
			args = args[2:]

Commit: 766bebb

View Full Run

---

Tell us how we did: Give Ito Feedback

[itoqa]

Ito Test Report ❌

History reset (rebase or force-push detected). Starting test narrative over.

8 test cases ran. 1 failed, 7 passed.

Across 8 executed tests, 7 passed and 1 failed: profiling-argument regressions, quick_sysbench workload execution with DNF signaling, and local_sysbench profiling plus unauthorized peer-access blocking all behaved as expected.
The sole significant issue is a Medium-severity bug in benchmark parsing where duplicate workload section keys (including repeated oltp_update_non_index from quick_sysbench) are silently overwritten in map-based section aggregation, causing missing comparison data without any explicit collision error.

❌ Failed (1)

Category	Summary	Screenshot
Benchmark	🟠 Duplicate benchmark section keys are accepted and silently collapsed to one surviving row with no collision diagnostic.

🟠 Duplicate benchmark sections overwrite silently

• What failed: The producer emits the same workload header key twice, but the parser stores parsed sections in a map keyed by test name and silently overwrites the earlier section with the later one; expected behavior is to reject or explicitly annotate duplicate-key collisions.
• Impact: Benchmark comparison output can silently exclude one duplicate workload result and present incomplete data as if parsing succeeded normally. This can mislead performance triage and PR benchmarking decisions unless raw logs are manually audited.
• Steps to reproduce:
  1. Generate quick sysbench output containing repeated section headers for the same workload key.
  2. Run the benchmark parser/comparison path (testing/go/benchmark) against those logs.
  3. Observe that no duplicate-key diagnostic is emitted and only one row for the duplicated workload is retained.
• Stub / mock context: Authentication bootstrap and SCRAM enforcement were bypassed to keep local benchmark execution deterministic. The run intentionally exercised duplicate benchmark section headers to verify whether parser collision handling was explicit.
• Code analysis: Reviewed the benchmark producer and consumer paths. scripts/quick_sysbench.sh includes oltp_update_non_index twice in the workload array, and testing/go/benchmark/section.go inserts into map[string]Section without any collision check or error path, so the first parsed section is dropped. PR diff metadata includes the duplicated workload entry, so the defect is introduced in this PR.
• Why this is likely a bug: The production benchmark script creates duplicate section keys while the production parser has no duplicate detection and overwrites by key, causing deterministic loss of one section's data.

Relevant code:

scripts/quick_sysbench.sh (lines 33-39)

"oltp_insert"
  "oltp_read_write"
  "oltp_update_index"
  "oltp_update_non_index"
  "oltp_update_non_index"
  "oltp_write_only"
  "types_delete_insert_postgres"

testing/go/benchmark/section.go (lines 29-30)

func SectionResults(fileData string) map[string]Section {
	sections := make(map[string]Section)

testing/go/benchmark/section.go (lines 65-69)

if section.Time == -1 {
			continue
		}
		sections[section.Test] = section
	}

✅ Passed (7)

Category	Summary	Screenshot
Benchmark	quick_sysbench completed the expanded workload run and produced sectioned benchmark artifacts for both read and write workloads.	N/A
Benchmark	Comparison output correctly surfaced missing mid-suite workloads as DNF instead of reporting normal benchmark deltas.	N/A
Local	Running local_sysbench.sh oltp_read_write --pprof completed a full 120s run with CPU profiling and workload results output.
Local	During an active local_sysbench run, read and write attempts from a separate docker container timed out, indicating unauthorized peer access was blocked.
Profile	Confirmed previous blockage was environment-only. Executing '--prof cpu -config=...' starts normally (Server ready) with no slice/out-of-range panic, matching fixed parsing logic in cmd/doltgres/main.go:159-183 where args are advanced only for recognized profiling flags.
Profile	Confirmed '--prof' flags are consumed while '-config' is preserved: server starts and CPU profile is written to the requested profile path.
Profile	Boundary and malformed profiling argv combinations behave explicitly and deterministically; no slice-related crash or silent loss of trailing args was reproduced.

Commit: 2d56287

View Full Run

---

Tell us how we did: Give Ito Feedback

[itoqa]

Ito Test Report ✅

History reset (rebase or force-push detected). Starting test narrative over.

12 test cases ran. 1 additional finding, 11 passed.

Overall, the unified run passed 11 of 12 test cases, confirming stable behavior for profiling flag handling, local sysbench execution (including --pprof CPU profile generation), back-to-back run isolation, workspace-scoped cleanup, and keeping sbtest artifacts out of git status. The key remaining issue is a medium-severity benchmark validation defect where prepare/run failures can be silently treated as consumable comparison input (e.g., downgraded to DNF/partial rows) instead of invalidating the run, due to non-fail-fast quick_sysbench flow and parser logic that skips unparsed-throughput sections.

✅ Passed (11)

Category	Summary	Screenshot
Artifact	Repeated local benchmark runs reset internal database state while preserving expected workspace artifacts.
Artifact	Recursive cleanup remained constrained to the intended workspace and did not remove an external sentinel path.
Artifact	Generated benchmark outputs stayed ignored and did not pollute git status for scripts/performance.
Benchmark	PR vs main quick sysbench outputs were generated and compared successfully after environment cleanup.
Benchmark	Comparator correctly marks workloads missing from PR logs as DNF.
Local	Default local sysbench run completed with expected logs and throughput output.
Local	--pprof run captured CPU profiling artifacts and completed benchmark execution.
Local	Immediate back-to-back --pprof rerun produced isolated second-run artifacts without lifecycle contamination.
Profiling	CPU profiling enabled, server reached ready state, and cpu.pprof was generated with config preserved.
Profiling	Config-only launch succeeded and produced no profiling logs or profile artifacts.
Profiling	Malformed trailing profiling flags returned controlled parse errors without panic or crash.

ℹ️ Additional Findings (1)

These findings are unrelated to the current changes but were observed during testing.

Category	Summary	Screenshot
Benchmark	🟠 Prepare/run failures are not promoted to a fatal run state, so partial logs remain parseable and appear valid.

🟠 Prepare failures are silently accepted as valid benchmark input

• What failed: A prepare/run failure should invalidate the benchmark run or surface a fatal state, but the current flow continues and produces output that looks consumable even when a workload failed.
• Impact: Performance comparisons can be based on partial or invalid benchmark runs, which risks misleading PR decisions. Teams can miss real regressions or trust noisy results because failed workloads are silently downgraded.
• Steps to reproduce:
  1. Create benchmark logs where one workload fails during prepare but other workloads succeed.
  2. Run go run ./testing/go/benchmark against those mixed logs.
  3. Observe that output is still produced with DNF or partial rows instead of invalidating the run.
• Stub / mock context: This verification used intentionally mixed benchmark logs containing one prepare-failed workload plus successful workloads to exercise partial-failure handling in the normal comparator path.
• Code analysis: scripts/quick_sysbench.sh explicitly disables fail-fast behavior and does not check command exit status after prepare/run calls, while testing/go/benchmark/section.go silently drops sections without parsed throughput, masking failed workloads instead of flagging run invalidity.
• Why this is likely a bug: The production benchmark workflow currently accepts mixed-success artifacts by design path, so a failed workload can be silently excluded rather than causing an explicit failure signal.

Relevant code:

scripts/quick_sysbench.sh (lines 1-3)

#!/bin/bash
#set -e
#set -o pipefail

scripts/quick_sysbench.sh (lines 71-80)

sysbench \
+    --db-driver="pgsql" \
+    --pgsql-host="0.0.0.0" \
+    --pgsql-port="$PORT" \
+    --pgsql-user="postgres" \
+    --pgsql-password="password" \
+    --pgsql-db="postgres" \
+    "$SYSBENCH_TEST" prepare

kill -15 "$SERVER_PID"

testing/go/benchmark/section.go (lines 65-67)

if section.Time == -1 {
	continue
}

Commit: cee8352

View Full Run

---

Tell us how we did: Give Ito Feedback