Profiling and Benchmarking

This document covers Kandelo's performance measurement tools: a runtime syscall profiler for detailed per-syscall analysis, and a benchmark suite for repeatable cross-host comparisons.

Syscall Profiler

The host runtime includes a built-in syscall profiler that measures every syscall handled by the kernel worker. It is zero-cost when disabled — no data structures are allocated and no timing code runs.

Enabling

Set the WASM_POSIX_PROFILE environment variable before starting the kernel:

WASM_POSIX_PROFILE=1 npx tsx examples/run-example.ts hello

Collecting Results

The profiler accumulates data in memory. Call dumpProfile() on the kernel worker instance to print results to stderr:

import { KernelWorker } from "kandelo/host";

const kernel = new KernelWorker(/* ... */);
// ... run workload ...
kernel.dumpProfile();

A common pattern is to dump on SIGINT so you can interrupt a long-running program:

process.on("SIGINT", () => {
  kernel.dumpProfile();
  process.exit();
});

Output Format

=== Syscall Profile ===
Syscall       Count     Time(ms)    Avg(ms)    Retries
----------------------------------------------------
4                50        12.34      0.247          0
3                30         8.21      0.274          5
63               20         3.45      0.173          2
----------------------------------------------------
TOTAL           100        23.00      0.230          7
Pending pipe readers: 0, writers: 0
=== End Profile ===

Columns:

Column	Meaning
Syscall	Linux syscall number (e.g., 3 = read, 4 = write, 63 = pread64). See `crates/shared/src/syscall_number.rs` for the full mapping.
Count	Total number of times this syscall was invoked.
Time(ms)	Cumulative wall-clock time spent handling this syscall, including kernel Wasm execution and host I/O.
Avg(ms)	Average time per call (`Time / Count`).
Retries	Number of times a blocking syscall (read/write on pipes/sockets, accept, or conflicting F_SETLKW) returned EAGAIN and was re-queued for retry. ENOLCK is a completed failure and is not counted as a retry. High retry counts indicate contention or slow producers/consumers.

Results are sorted by total time (highest first), so the most expensive syscalls appear at the top.

The footer shows pending pipe reader/writer counts — non-zero values at shutdown may indicate leaked pipes or unfinished I/O.

Limitations

Only available in Node.js (relies on process.env).
Measures host-side wall-clock time, which includes both kernel Wasm execution and any host I/O (filesystem, network). It does not isolate kernel computation from host latency.
Retry counts track how often a syscall was re-queued due to EAGAIN, not the total number of underlying host retries.

Benchmark Suite

The benchmark suite runs reproducible workloads on both Node.js and browser hosts, producing JSON results that can be compared across runs.

Prerequisites

Build the kernel and benchmark programs:

bash build.sh
scripts/build-programs.sh

Some suites require additional binaries:

Suite	Requires
`syscall-io`	Base sysroot + benchmark programs
`process-lifecycle`	Base sysroot + benchmark programs
`erlang-ring`	Pre-built Erlang binary
`wordpress`	Pre-built PHP, nginx, WordPress
`mariadb`	Pre-built MariaDB

Application benchmark suites fail if their required binaries are not found. A missing artifact means the suite did not measure its workload; build or fetch the prerequisite before running it.

Running Benchmarks

# All suites on Node.js (3 rounds each, reports median)
npx tsx benchmarks/run.ts

# All suites in the browser (via Playwright)
npx tsx benchmarks/run.ts --host=browser

# Single suite
npx tsx benchmarks/run.ts --suite=syscall-io

# More rounds for stability
npx tsx benchmarks/run.ts --rounds=5

# Combine options
npx tsx benchmarks/run.ts --host=browser --suite=process-lifecycle --rounds=5

Results are saved as JSON in benchmarks/results/.

Each result records the selected benchmark artifact paths, sizes, and SHA-256 digests for that host's measurements. The application evidence covers the PHP/opcache inputs, Node WordPress source/config/router, browser WordPress and MariaDB VFS images, and each Node MariaDB architecture's server, client, and bootstrap SQL. The WordPress source-tree digest follows source symlinks but excludes runtime-owned wp-content/database state and wp-content/debug.log. Resolver-selected paths are retained alongside the logical artifact names; browser VFS evidence reflects the public asset that the benchmark page selects first. Kernel fingerprints use the same policy-aware binary resolver as each host. Node rootfs evidence records which of the runtime's rootfs.vfs then programs/rootfs.vfs fallback requests won, and is required only for the syscall/process suites that boot that default image. Browser benchmarks do not record or require the default rootfs because they boot generated empty or app images. Node static benchmark Wasm inputs are required only by the syscall or process suite that consumes them. The browser benchmark page imports its seven micro Wasm URLs at module load, so every runnable browser suite requires all seven; exec-bench.wasm remains Node-only. After printing the artifact report, the runner stops before workloads when a required, selected input is missing.

Available Suites

syscall-io

Measures raw I/O throughput, syscall round-trip overhead, and advisory-lock manager operations. The lock workload uses OFD locks so an independently opened descriptor can perform a real conflicting lookup without including a fork in the timed region. One shape holds one to three separated records on each of 64 files (127 total records); the other holds 256 separated ranges on one file. The 64-file shape retains 128 OFDs, below the browser SharedFS backend's 160-handle ceiling while still exercising file-index selection at scale. These counts fit both sides of a before/after comparison with the former fixed table. They are performance workloads, not substitutes for the 4096-record capacity tests.

Metric	Unit	What it measures
`pipe_mbps`	MB/s	Write 1 MB through a pipe (fork + read/write loop)
`file_write_mbps`	MB/s	Write 1 MB to a file
`file_read_mbps`	MB/s	Read 1 MB from a file
`syscall_latency_us`	microseconds	Average `getpid()` round-trip over 1000 calls
`lock_many_files_acquire_us_per_op`	microseconds/operation	Acquire separated write locks across 64 files with one to three records each
`lock_many_files_conflict_us_per_op`	microseconds/operation	Find each many-file lock from an independently opened OFD
`lock_many_files_replace_us_per_op`	microseconds/operation	Replace each many-file write lock with a read lock under the same OFD owner
`lock_many_files_unlock_us_per_op`	microseconds/operation	Unlock each many-file record
`lock_dense_file_acquire_us_per_op`	microseconds/operation	Acquire 256 separated write ranges on one file
`lock_dense_file_conflict_us_per_op`	microseconds/operation	Find each dense-file lock from an independently opened OFD
`lock_dense_file_replace_us_per_op`	microseconds/operation	Replace each dense-file write range with a read range under the same OFD owner
`lock_dense_file_unlock_us_per_op`	microseconds/operation	Unlock each dense-file range

For a focused advisory-lock comparison, run the same benchmark source and round count before and after the change on both hosts. Rebuild the guest artifact for each side's ABI epoch; an ABI 39 benchmark must not be reused with an ABI 40 kernel, or vice versa:

npx tsx benchmarks/run.ts --suite=syscall-io --rounds=3
npx tsx benchmarks/run.ts --host=browser --suite=syscall-io --rounds=3

The complete Node and browser suite is still required before making a broad performance or no-regression claim. Adding these cases does not itself establish that the lock-manager migration is faster, slower, or neutral.

process-lifecycle

Measures process management primitives.

Metric	Unit	What it measures
`hello_start_ms`	ms	Cold start: load Wasm + run hello world to exit
`fork_ms`	ms	Fork a child + wait for it to exit
`exec_ms`	ms	Exec a new program
`clone_ms`	ms	Create a thread via clone

erlang-ring

Runs the Erlang/OTP BEAM VM, spawning 1000 lightweight processes in a ring topology and passing a token around 100 times.

Metric	Unit	What it measures
`total_ms`	ms	Wall-clock time for full ring completion
`messages_per_sec`	msg/s	Total messages (100,000) / elapsed seconds

Prerequisites:

Component	Path	Build command
BEAM VM	`packages/registry/erlang/bin/beam.wasm`	`bash packages/registry/erlang/build-erlang.sh`
OTP libraries	`packages/registry/erlang/erlang-install/`	(built by same script)
Ring program	`packages/registry/erlang/demo/ring.beam`	(included in repo)

Build requirements: host Erlang/OTP 28 (brew install erlang), wasm32posix-cc SDK (cd sdk && npm link).

wordpress

Runs PHP 8.4 with a full WordPress 6.7 installation. Two measurements: cold CLI load time and HTTP server first-response time.

Metric	Unit	What it measures
`cli_require_ms`	ms	`php -r "require 'wp-load.php'"` — process start to exit
`http_first_response_ms`	ms	Start PHP built-in server, time to first HTTP response

Each Node measurement starts from the WordPress setup state: the benchmark removes and recreates wp-content/database, removes wp-content/debug.log, and does the same cleanup after the measurement. When the OPcache side module is available and NO_OPCACHE is not 1, the measurement uses opcache.file_cache_only=1 with timestamp validation disabled. Each suite round creates a private cache root under benchmarks/results/, and the CLI and HTTP measurements each receive a separate empty cache directory that is reset before and after timing. The run-owned cache root is removed when the round finishes, so compiled scripts cannot carry across metrics, rounds, concurrent benchmark processes, or worktrees.

Prerequisites:

Component	Path	Build command
PHP CLI	`packages/registry/php/php-src/sapi/cli/php`	`bash packages/registry/php/build-php.sh`
WordPress	`packages/registry/wordpress/wordpress/wp-settings.php`	See below
Router script	`packages/registry/wordpress/demo/router.php`	(included in repo)

Build requirements: wasm32posix-cc SDK. The PHP build script automatically builds dependencies (SQLite, zlib, OpenSSL, libxml2). WordPress must be downloaded separately into packages/registry/wordpress/wordpress/.

mariadb

Runs MariaDB 10.5 with the Aria or InnoDB storage engine. Measures bootstrap (system table creation) and a sequence of SQL operations.

Four suite variants are registered so wasm32 and wasm64 builds can be compared side-by-side in one run.ts invocation:

Suite	Engine	Wasm ABI	Install dir
`mariadb-aria`	Aria	wasm32 (ILP32)	`mariadb-install/`
`mariadb-aria-64`	Aria	wasm64 (LP64)	`mariadb-install-64/`
`mariadb-innodb`	InnoDB	wasm32 (ILP32)	`mariadb-install/`
`mariadb-innodb-64`	InnoDB	wasm64 (LP64)	`mariadb-install-64/`

Metric	Unit	What it measures
`bootstrap_ms`	ms	System table initialization (`--bootstrap` mode)
`query_create_ms`	ms	CREATE TABLE (2 tables)
`query_insert_ms`	ms	Batch INSERT (100 rows into each table)
`query_select_ms`	ms	SELECT with WHERE clause
`query_join_ms`	ms	JOIN across two tables

Set MARIADB_BENCH_VERBOSE=1 to forward mariadbd stdout/stderr to the shell (useful for debugging hangs or slow bootstraps).

Prerequisites:

Component	Path	Build command
MariaDB server (wasm32)	`packages/registry/mariadb/mariadb-install/bin/mariadbd.wasm`	`bash packages/registry/mariadb/build-mariadb.sh`
MariaDB server (wasm64)	`packages/registry/mariadb/mariadb-install-64/bin/mariadbd.wasm`	`bash packages/registry/mariadb/build-mariadb.sh --wasm64`
mysqltest client	`<install-dir>/bin/mysqltest.wasm`	(built by same script)
System table SQL	`<install-dir>/share/mysql/mysql_system_tables*.sql`	(built by same script)

Build requirements: cmake (brew install cmake), wasm32posix-cc / wasm64posix-cc SDK. The build is a two-phase cross-compilation (host build for code generators, then wasm cross-compile). The wasm64 build uses -O1 instead of -O2 to avoid an LLVM 21 wasm64 backend miscompilation in table-lookup sign-extension.

Building All Suite Prerequisites

To run the complete benchmark suite, build all prerequisites in order:

# 1. SDK toolchain (required by all application suites)
cd sdk && npm link && cd ..

# 2. Base benchmark programs (syscall-io, process-lifecycle)
scripts/build-programs.sh

# 3. Erlang/OTP (requires: brew install erlang)
bash packages/registry/erlang/build-erlang.sh

# 4. PHP + WordPress (PHP build includes SQLite, zlib, OpenSSL, libxml2)
bash packages/registry/php/build-php.sh
# Download WordPress into packages/registry/wordpress/wordpress/

# 5. MariaDB (requires: brew install cmake)
bash packages/registry/mariadb/build-mariadb.sh          # wasm32
bash packages/registry/mariadb/build-mariadb.sh --wasm64 # wasm64 (optional, for dual-arch comparison)

Running the Complete Suite for Performance Work

When measuring the performance impact of kernel changes, run every enabled suite on both hosts. Application-level suites (WordPress and the MariaDB variants) exercise different syscall patterns and threading models that micro-benchmarks miss.

# Full comparison workflow:
# 1. Run baseline on both hosts
npx tsx benchmarks/run.ts --rounds=3
npx tsx benchmarks/run.ts --host=browser --rounds=3

# 2. Apply changes

# 3. Run again on both hosts
npx tsx benchmarks/run.ts --rounds=3
npx tsx benchmarks/run.ts --host=browser --rounds=3

# 4. Compare
npx tsx benchmarks/compare.ts benchmarks/results/<before>.json benchmarks/results/<after>.json

When a required binary is missing, the runner prints the artifact report and fails before starting workloads. Build the missing prerequisites (see above) before drawing conclusions about performance impact.

Comparing Results

Use the comparison tool to diff two benchmark runs:

npx tsx benchmarks/compare.ts benchmarks/results/before.json benchmarks/results/after.json

Output is a markdown table with percentage change for each metric. Regressions (>5% worse) are bolded:

| Benchmark                      | Before | After | Change  |
|--------------------------------|--------|-------|---------|
| syscall-io/pipe_mbps           | 245.5  | 260.1 | +5.9%   |
| syscall-io/syscall_latency_us  | 12.3   | 15.1  | **+22.8%** |
| process-lifecycle/fork_ms      | 45.2   | 44.8  | -0.9%   |

For throughput metrics (*_mbps, messages_per_sec), a decrease is a regression. For latency/duration metrics, an increase is a regression.

Writing a Custom Suite

Create a TypeScript file in benchmarks/suites/ that exports a BenchmarkSuite:

import type { BenchmarkSuite } from "../types.js";

const suite: BenchmarkSuite = {
  name: "my-suite",
  async run(): Promise<Record<string, number>> {
    // Run workload, return metric_name → value
    return { ops_per_sec: 1234.5 };
  },
};

export default suite;

const SUITE_MODULES: Record<string, string> = {
  // ...existing suites...
  "my-suite": "./suites/my-suite.js",
};

Benchmark programs (C source) live in benchmarks/programs/ and output metrics as key=value lines on stdout, which suites parse with a simple regex.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Profiling and Benchmarking

Syscall Profiler

Enabling

Collecting Results

Output Format

Limitations

Benchmark Suite

Prerequisites

Running Benchmarks

Available Suites

syscall-io

process-lifecycle

erlang-ring

wordpress

mariadb

Building All Suite Prerequisites

Running the Complete Suite for Performance Work

Comparing Results

Writing a Custom Suite

Uh oh!

FilesExpand file tree

profiling.md

Latest commit

History

profiling.md

File metadata and controls

Profiling and Benchmarking

Syscall Profiler

Enabling

Collecting Results

Output Format

Limitations

Benchmark Suite

Prerequisites

Running Benchmarks

Available Suites

syscall-io

process-lifecycle

erlang-ring

wordpress

mariadb

Building All Suite Prerequisites

Running the Complete Suite for Performance Work

Comparing Results

Writing a Custom Suite