Native arm64 dynamic core support for Apple Silicon

[kklobe]

Last night I threw together a minimum working hack for native arm64 dynamic core on my MacBook Pro M1, with some encouraging results.

Using Future Crew's Unreal (unreal p8 with GUS 44kHz, surely one of the better moments in DOS history), I was able to raise the audio stuttering threshold from ~100k cycles on normal core to ~400k on the dynamic core.

The short list:

• mprotect no longer works on macOS 11.2 (and was never the recommended solution from Apple)
• allocate the cache code memory with mmap using the MAP_JIT flag
• surround each cache_addX write with write protect disable and enable, then flush the written memory

In practice, this is all in dyn_cache.h, and looks like:

#if defined (WIN32)
			cache_code_start_ptr=(Bit8u*)VirtualAlloc(0,CACHE_TOTAL+CACHE_MAXSIZE+PAGESIZE_TEMP-1+PAGESIZE_TEMP,
				MEM_COMMIT,PAGE_EXECUTE_READWRITE);
			if (!cache_code_start_ptr)
				cache_code_start_ptr=(Bit8u*)malloc(CACHE_TOTAL+CACHE_MAXSIZE+PAGESIZE_TEMP-1+PAGESIZE_TEMP);
#else
			size_t cache_code_start_sz = CACHE_TOTAL + CACHE_MAXSIZE + PAGESIZE_TEMP - 1 + PAGESIZE_TEMP;
			//cache_code_start_ptr=(Bit8u*)malloc(cache_code_start_sz);
			cache_code_start_ptr=(Bit8u*)mmap(NULL, cache_code_start_sz, PROT_READ | PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANON | MAP_JIT, -1, 0);
#endif
			if (cache_code_start_ptr == MAP_FAILED)
				E_Exit("Allocating dynamic core cache memory failed with errno %d", errno);

and

// place a 32bit value into the cache

static INLINE void cache_addd(uint32_t val, const uint8_t *pos)
{
	pthread_jit_write_protect_np(false);	
	write_unaligned_uint32(const_cast<uint8_t *>(pos), val);
	pthread_jit_write_protect_np(true);	
	sys_icache_invalidate(const_cast<uint8_t *>(pos), sizeof(val));
}

I understand from Discord chats that there's a plan for this work and I am very happy to help out with coding, testing, benchmarking (the fun part), and anything else.

A few references:

Porting Just-In-Time Compilers to Apple Silicon
Attempts to mprotect() with MAP_JIT failing on Apple Silicon as of macOS 11.2
Apple M1 Support for MacOS

[kcgen]

Crossed linked to vogons: https://www.vogons.org/viewtopic.php?p=958225#p958225

[kklobe]

Based on jmarsh's comment in the vogons thread I moved the write protect toggle and cache invalidation up to just the CreateCacheBlock call, which resulted in a significant performance improvement.

JITing Quake, the old approach:

JITing Quake, the new approach:

I also added a naive mprotect implementation for SELinux, tested under Fedora 33 ARM64 in a VM, and it seems to work fine.

I have the experiment at https://github.com/kklobe/dosbox-staging/tree/kklobe/arm64_dynamic.

[kklobe]

my reply reposted from vogons:

Hi all, author of the code here, just wanted to respond to some good points jmarsh has raised.

TLDR: I'm in favor of a "1.5" Approach that combines a single mmap region with write protect toggling. I started this experiment because I wanted DOSBox Staging dynrec running on my M1 MacBook Pro that I've had since December.

I like Approach 1 because of the simplicity, along with the relative portability of a single mmap call with MAP_ANON | MAP_PRIVATE (| MAP_JIT if apple).

With regards to the 2016 presentation mentioned, a few things are new in the Apple ecosystem in the last several years:

• macOS Mojave shipped with the optional Hardened Runtime in 2018.
• macOS Big Sur shipped with per-thread write protect calls (pthread_jit_write_protect_np()) in 2020.
• Apple Silicon hardware shipped in 2020.

Apple's recommended solution for security in 2021 seems to include the above mentioned components: enable the Hardened Runtime, enable the JIT Entitlement, and use per-thread write protect toggling to help reduce the attack surface.

And, I reasoned, if it's going to toggle, it might as well just have a single mapping, and not have to deal with the fiddly issues pointed out for dual mappings.

After looking around at some other projects that do codegen, I found that the toggling approach is common:

• OpenJDK approved this approach for the HotSpot VM JIT: 8253795: Implementation of JEP 391: macOS/AArch64 Port openjdk/jdk#2200
• Google's JavaScript v8 runtime (used in Chrome and NodeJS): https://github.com/v8/v8/blob/dc712da548c7fb433caed56af9a021d964952728/src/wasm/code-space-access.h
• Steel Bank Common LISP has it in their JIT: https://github.com/sbcl/sbcl/search?q=pthread_jit_write_protect_np
• qemu as well: https://github.com/qemu/qemu/search?q=pthread_jit_write_protect_np

So it looks like there's a decent precedent for the toggling approach. It also worked with a quick test on Fedora 33 arm64 with SELinux enabled by using an mprotect in place of the pthread_jit_write_protect_np().

In summary, I came to the same conclusion as the author of the OpenJDK PR:

"It's implemented with pthread_jit_write_protect_np provided by Apple... This approach of managing W^X mode turned out to be simple and efficient enough."

Thanks for taking the time to discuss.

[kcgen]

Adding a link where this code allows the dynamic core to run with a 64-bit x86-64 installation of Fedora 34, with SELinux in its default "enforcing" state. See: dosbox-staging/dosbox-staging#1010

With these being tested-and-working, having minimal measured performance impact, the lean state of the implementation (ie: the dynamic core is left essentially as-is), and the approach following best-practices per the list above - I can't see anything holding this back from landing in Staging!

[kcgen]

Discussion archived in PR: dosbox-staging/dosbox-staging#1031
(Suggest closing)

[kcgen]

Archived discussion to https://github.com/dosbox-staging/archived-discussions-for-dosbox-staging