Seems the provided libraries are all set to x86_64 on MacOS, so they don't do anything on Apple Silicon Macs (M1+).
I tried to adjust the Xcode project and settings in Unity, but no luck so far. Could this repo be updated to include Apple Silicon support?