Different page size support

[Sadoon-AlBader]

On arm64, make sure your kernel is configured to use 4k pages since we haven't yet added support for 16k and 64k pages. The kernel also has to be configured to use 4 level page tables for the full 48 bit address space instead of only having a 39 bit address space for the default hardened_malloc configuration. It's possible to reduce the class region size substantially to make a 39 bit address space workable but the defaults won't work.

Are there any plans to support larger page sizes? I have done some rudimentary testing on Debian 12 ppc64el by manually setting the page size in pages.h to 65536 and disabling the static_assert in h_malloc.c to see if things break ( I am adventurous :P )

Running simple programs seems everything works fine, and using "make test" also produces no errors.

I will gladly do testing or work on supporting larger page sizes if possible.

EDIT: Even chromium seems to run fine, I hope I didn't corrupt my profile.

[thestinger]

The size classes need to be redone for a larger page size. It will be incredibly inefficient doing it that way without redoing it. It would require larger bitmaps for the smallest size classes too.

[Sadoon-AlBader]

The size classes need to be redone for a larger page size. It will be incredibly inefficient doing it that way without redoing it. It would require larger bitmaps for the smallest size classes too.

Ah so it's less of an issue of it breaking software and more of it being inefficient this way?

[thestinger]

It's not set up for pages of that size. For example, 16 byte allocations are placed in a slab with 256 slots, which is 4096 bytes. The slab size will be rounded up to 65536 but only the first 4096 bytes will be used. For one thing, all the slot counts need to be redone, but 256 is currently the maximum number of slots with the current implementation. Replacing the array of 4x 64-bit integers with an array of 64x 64-bit integers is not a reasonable approach. It would need to use a multi-level bitmap or another approach.

[Sadoon-AlBader]

It's not set up for pages of that size. For example, 16 byte allocations are placed in a slab with 256 slots, which is 4096 bytes. The slab size will be rounded up to 65536 but only the first 4096 bytes will be used. For one thing, all the slot counts need to be redone, but 256 is currently the maximum number of slots with the current implementation. Replacing the array of 4x 64-bit integers with an array of 64x 64-bit integers is not a reasonable approach. It would need to use a multi-level bitmap or another approach.

That's quite a bit beyond my knowledge, and it does sound like it would need a rewrite.
Thanks for the explanation and quick reply :)

[Sadoon-AlBader]

For what it's worth, I tried using the same ppc64el machine with Debian 12 on a 4K kernel to see if things work and they seem to do. Everything is working as expected, even the single app I have that crashes under x86_64 seems to crash here :p

If this helps this is my environment:
Debian 12 with kernel 6.1.34, stock debian .config except 4K page sizes.
hardened_malloc applied systemwide with the /etc/ld.so.preload method.
vm.max_map_count set to 1048576
KDE desktop (Wayland) with patched QTWebEngine to run apps that depend on it (isn't shipped by default on POWER)
Chromium, Kmail, Neochat, KeePassXC, Elisa, KMyMoney (the only crash as I mentioned), Calibre 5 (backported from Bullseye)
I use a few others here and there but this is just to verify that they work.

The only thing that needs a change in the code is the Makefile to support -mcpu=native instead of -march=native since -march isn't supported on POWER

I'll send a tiny patch that checks if the system is POWER to use -mcpu instead of -march

Thanks again for this awesome code!

[markfswe]

As an Asahi Linux user I would really love to see support of 16k pages <3
Thank you for this project.

[thestinger]

It will only happen if a contributor works on it. We have no use for it in GrapheneOS.

[Andarwinux]

Android seems to be adding 16K page support in the future.
https://android-review.googlesource.com/q/topic:%2216k_dev_ui%22

[jvoisin]

It's almost there, and the PAGE_SIZE macro is now deprecated

[AtmosphericIgnition]

Just ran into this after failing to deploy hardened_malloc on a 64KB ARM server.

Out of curiosity, is GrapheneOS going to stick to 4KB pages after the PixelOS makes the switch to 16KB? If there is an update to support 16KB pages, will that then become the only supported page size, or will different page sizes be supported as well?

[thestinger]

We're going to continue using 4k pages for better security and lower memory usage especially since we already increase memory usage with the slab quarantines. We should probably reduce the size of the slab quarantines.

It's entirely possible to support 16k and 64k pages in hardened_malloc but it requires implementing new slab slot counts and also supporting more than 4x 64-bit for the bitmap for 16 byte allocations since that only provides 256 slots which fills a 4k page with 16 byte allocations. The minimum size class with the current bitmap size would become 64 bytes with 16k pages or 256 bytes with 64k pages which is clearly unacceptable. It would need to be changed to have a larger or even multi-level bitmap.

[Andarwinux]

The Pixel 9 and 8 series have over 12GB of RAM, so I don't think this is an issue.

The Android NDK's default common-page-size is 16384, so I don't think 64K pages need to be considered at the moment.

Android 16 already supports 4KB compatibility mode in 16KB mode, which is one of the strong evidences that future Android products will use 16KB mode by default.

To sum up, I hope GrapheneOS will reconsider using 16KB to improve performance and maintain high compatibility with stock. This is also expected to allow hardened_malloc to gain a place in the Apple Silicon ecosystem.

[AtmosphericIgnition]

@Andarwinux I understand that 64K pages would be overkill for the current Pixel devices. My use case for a 64K-compatible hardened_malloc would be to harden some services on an ARM server which uses a 64K kernel, but I understand that the primary use case for hardened_malloc is GrapheneOS, and being able to use it on a server is completely secondary to that use case.

[Andarwinux]

Yes, it would be nice to have 64KB pages or even any scalable page size as a supplement.

I don't know how Android 16 implements 4KB compatibility mode, if it's some kind of macOS-like runtime dynamic page, maybe it should at least allow opting in to 4KB compatibility mode with hardened_malloc enabled, and otherwise stay in 16KB mode. This could be useful for some apps that don't require extreme security but are more performance-focused, especially when Tensor SoC performance is already 3+ years behind the world.