big performance gap between native, tart and lima (vz)

[yaronsumel]

Description

While testing around I noticed that running sysbench cpu --threads=4 --time=10 run will produce big performance gaps between native (m1 pro / m4 pro), lima (ubuntu) and tart(macOS). I tried to find the reason but nothing so far, both tart and lima using vz.

macOS native M1 Pro

sysbench 1.0.20 (using system LuaJIT 2.1.1727870382)

Running the test with following options:
Number of threads: 4
Initializing random number generator from current time


Prime numbers limit: 10000

Initializing worker threads...

Threads started!

CPU speed:
    events per second: 49413246.42

General statistics:
    total time:                          10.0000s
    total number of events:              494150253

Latency (ms):
         min:                                    0.00
         avg:                                    0.00
         max:                                    0.06
         95th percentile:                        0.00
         sum:                                14050.67

Threads fairness:
    events (avg/stddev):           123537563.2500/12796.10
    execution time (avg/stddev):   3.5127/0.01

tart macOS

sysbench cpu --threads=4 --time=10 run
sysbench 1.0.20 (using system LuaJIT 2.1.1727870382)

Running the test with following options:
Number of threads: 4
Initializing random number generator from current time


Prime numbers limit: 10000

Initializing worker threads...

Threads started!

CPU speed:
    events per second: 38228535.19

General statistics:
    total time:                          10.0001s
    total number of events:              382306951

Latency (ms):
         min:                                    0.00
         avg:                                    0.00
         max:                                    5.28
         95th percentile:                        0.00
         sum:                                11121.94

Threads fairness:
    events (avg/stddev):           95576737.7500/200987.74
    execution time (avg/stddev):   2.7805/0.01

lima ubuntu vm

sysbench cpu --threads=4 --time=10 run
sysbench 1.0.20 (using system LuaJIT 2.1.1744014795)

Running the test with following options:
Number of threads: 4
Initializing random number generator from current time


Prime numbers limit: 10000

Initializing worker threads...

Threads started!

CPU speed:
    events per second: 32349.48

General statistics:
    total time:                          10.0001s
    total number of events:              323517

Latency (ms):
         min:                                    0.11
         avg:                                    0.12
         max:                                    2.03
         95th percentile:                        0.14
         sum:                                39897.66

Threads fairness:
    events (avg/stddev):           80879.2500/81.87
    execution time (avg/stddev):   9.9744/0.00

not sure its that a lima issue but I would be happy to provide more debug details if needed.

[yaronsumel]

configurations:

// tart
tart clone ghcr.io/cirruslabs/macos-sequoia-vanilla:latest sequoia
tart set sequoia --cpu 10
tart run sequoia --no-graphics --no-audio

// lima  (all default)
limactl list
NAME       STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
default    Running    127.0.0.1:60022    vz        aarch64    10      4GiB      100GiB    ~/.lima/default 

[yaronsumel]

I have disabled all mounts / no exposed host directories to the guest

sumel@lima-default:~$ strace -c sysbench cpu --threads=4 --time=10 run
sysbench 1.0.20 (using system LuaJIT 2.1.1744014795)

Running the test with following options:
Number of threads: 4
Initializing random number generator from current time


Prime numbers limit: 10000

Initializing worker threads...

Threads started!

CPU speed:
    events per second: 32461.52

General statistics:
    total time:                          10.0003s
    total number of events:              324644

Latency (ms):
         min:                                    0.11
         avg:                                    0.12
         max:                                    1.41
         95th percentile:                        0.13
         sum:                                39903.99

Threads fairness:
    events (avg/stddev):           81161.0000/74.44
    execution time (avg/stddev):   9.9760/0.00

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 48.38    0.005381          24       221       205 openat
 15.52    0.001726          23        73           mmap
 10.72    0.001192          27        44         1 newfstatat
  6.08    0.000676          18        36           mprotect
  5.42    0.000603          18        32           munmap
  2.58    0.000287          19        15           read
  2.38    0.000265          16        16           close
  2.35    0.000261          16        16           fstat
  2.01    0.000223          55         4           clone3
  1.78    0.000198           7        26           write
  0.85    0.000094          10         9           rt_sigprocmask
  0.69    0.000077          25         3           futex
  0.29    0.000032          16         2           rt_sigaction
  0.23    0.000026           8         3           setitimer
  0.22    0.000025           8         3           brk
  0.11    0.000012          12         1           set_tid_address
  0.11    0.000012          12         1           prlimit64
  0.10    0.000011          11         1           set_robust_list
  0.10    0.000011          11         1           getrandom
  0.09    0.000010          10         1           rseq
  0.00    0.000000           0         1         1 faccessat
  0.00    0.000000           0         1           execve
------ ----------- ----------- --------- --------- ----------------
100.00    0.011122          21       510       207 total

[nirs]

// tart
tart clone ghcr.io/cirruslabs/macos-sequoia-vanilla:latest sequoia
tart set sequoia --cpu 10
tart run sequoia --no-graphics --no-audio

This is a vm using 10 cores on 10 cores machine (8 performance cores, 2 efficiency cores)

// lima (all default)

Default use 4 cores

limactl list
NAME STATUS SSH VMTYPE ARCH CPUS MEMORY DISK DIR
default Running 127.0.0.1:60022 vz aarch64 10 4GiB 100GiB ~/.lima/default

But here you see 10 cores.

Can you share ~/.lima/default/lima.yaml?

[yaronsumel]

@nirs sorry by default I mean no different image or so..

limactl create default --cpus 10 --memory 4 --arch aarch64 --disk 10 --vm-type vz --mount-none

lima.yaml

vmType: vz
arch: aarch64
cpus: 10
memory: 4GiB
disk: 10GiB
mounts: null

$ nproc
10

$ lscpu
Architecture:             aarch64
  CPU op-mode(s):         64-bit
  Byte Order:             Little Endian
CPU(s):                   10
  On-line CPU(s) list:    0-9
Vendor ID:                Apple
  Model name:             -
    Model:                0
    Thread(s) per core:   1
    Core(s) per cluster:  10
    Socket(s):            -
    Cluster(s):           1
    Stepping:             0x0
    BogoMIPS:             48.00
    Flags:                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fph
                          p asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 a
                          simddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca
                          pacg dcpodp flagm2 frint
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-9
Vulnerabilities:
  Gather data sampling:   Not affected
  Ghostwrite:             Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Vulnerable
  Spectre v1:             Mitigation; __user pointer sanitization
  Spectre v2:             Mitigation; CSV2, but not BHB
  Srbds:                  Not affected
  Tsx async abort:        Not affected

[nirs]

@yaronsumel

$ strace -c sysbench cpu --threads=4 --time=10 run

This may measure only the main thread, add -f to trace all threads.

It is not clear what is the meaning of the output of sysbench cpu benchmark.

We see 4 orders of magnitude difference between linux vm, native macOS, and macOS vm.

test	os	events per second	latency 95th percentile (ms)
native	macOS	49413246.42	0.00
tart	macOS	38228535.19	0.00
lima/vz.	linux	32349.48	0.14

This may be the overhead of virtualizing linux on macOS using the virtualization framework. Maybe virtualization macOS has some tricks to avoid virtaulization cost when the guest is running macOS.

I would try to compare a real workload between native macOS and linux on the same hardware (e.g. using Asahi linux) and between different linux vms on macOS (lima/vz, lima/qemu, vfkit, krunkit, tart).

Also using a vm with 10 cpus on a machine with 10 cpus is not realistic usage. Using vm with 2 or 4 cpus is a more interesting use case.

[nirs]

For comparing cpu performance, here is a better benchmark showing that we get native performance in the vm. This is a single threaded workload that does no I/O so it many not be relevant to your workload.

lima/vz

$ openssl speed sha256
Doing sha256 ops for 3s on 16 size blocks: 31760218 sha256 ops in 3.00s
Doing sha256 ops for 3s on 64 size blocks: 27265640 sha256 ops in 3.00s
Doing sha256 ops for 3s on 256 size blocks: 16516567 sha256 ops in 3.00s
Doing sha256 ops for 3s on 1024 size blocks: 6389635 sha256 ops in 3.00s
Doing sha256 ops for 3s on 8192 size blocks: 948974 sha256 ops in 3.00s
Doing sha256 ops for 3s on 16384 size blocks: 481886 sha256 ops in 3.00s
version: 3.4.1
built on: Thu Apr  3 08:48:37 2025 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -Wa,--noexecstack -g -O2 -Werror=implicit-function-declaration -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/openssl-bHTCkM/openssl-3.4.1=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -mbranch-protection=standard -fdebug-prefix-map=/build/openssl-bHTCkM/openssl-3.4.1=/usr/src/openssl-3.4.1-1ubuntu3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZSTD -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=3
CPUINFO: OPENSSL_armcap=0x8fd
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256          169387.83k   581666.99k  1409413.72k  2180995.41k  2591331.67k  2631740.07k

native

% openssl speed sha256
Doing sha256 ops for 3s on 16 size blocks: 29436505 sha256 ops in 2.97s
Doing sha256 ops for 3s on 64 size blocks: 26217298 sha256 ops in 2.97s
Doing sha256 ops for 3s on 256 size blocks: 16555217 sha256 ops in 2.96s
Doing sha256 ops for 3s on 1024 size blocks: 6384648 sha256 ops in 2.96s
Doing sha256 ops for 3s on 8192 size blocks: 951020 sha256 ops in 2.97s
Doing sha256 ops for 3s on 16384 size blocks: 481458 sha256 ops in 2.97s
version: 3.5.2
built on: Tue Aug  5 12:09:26 2025 UTC
options: bn(64,64)
compiler: clang -fPIC -arch arm64 -O3 -Wall -DL_ENDIAN -DOPENSSL_PIC -D_REENTRANT -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
CPUINFO: OPENSSL_armcap=0x987d
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256          158580.50k   564951.88k  1431802.55k  2208743.09k  2623150.11k  2655962.25k

[nirs]

@yaronsumel did you try ask sysbench folks about the results? They may be able to explain the difference for bare metal and vm.

[yaronsumel]

@nirs
I guess you are right - something about sysbench is not right. when it comes to pure cpu lima vz keeps up with native speeds ..

Tested it with a small golang test

native

go run test.go
Computed 100000 cosine similarities in 0.0173 seconds
Throughput: 5778440.27 calculations per second

vm

Computed 100000 cosine similarities in 0.0176 seconds
Throughput: 5673102.56 calculations per second

I have found something on sysbench repo that looks similar akopytov/sysbench#550