Building on M1 Mac

[thomaslima]

Hi @Kazzz-S ,

I wonder if I could borrow your advice for an issue I have building Mac images on my MacBook (M1). I followed these steps:

• I installed qt@5, ruby@3.2, and python@3.9 from homebrew
• I ran ./macbuild/python3HB.py -v 3.9
• I added the QMAKE_CXXFLAGS += -fdeclspec line to klayout.pri (for some reason your script didn't patch automatically.
• Then I ran the following script:

#!/bin/bash
set -e

export QMAKE_CCACHE=1

# run ./macbuild/python3HB.py -v 3.9

./build4mac.py -p HB39 -r HB32 -q Qt5Brew
./build4mac.py -p HB39 -r HB32 -q Qt5Brew -y

• This produced HW-qt5Brew.pkg.macos-Ventura-release-Rhb32Phb39/klayout.app. When I tried to open it, it crashed with the following log dump. Am I doing something wrong? I feel like I forgot some step.

Do you have any idea of what's going on?

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               klayout [4163]
Path:                  /Users/USER/*/klayout.app/Contents/MacOS/klayout
Identifier:            de.klayout
Version:               0.28.5 (0.28.5)
Code Type:             ARM-64 (Native)
Parent Process:        launchd [1]
User ID:               8803

Date/Time:             2023-02-16 10:21:42.0242 -0500
OS Version:            macOS 13.0.1 (22A400)
Report Version:        12
Anonymous UUID:        D2931AAD-F46B-72AF-9649-1D8A6B9062D6

Sleep/Wake UUID:       F9256465-18A2-45BA-B26C-EF90F5F8420B

Time Awake Since Boot: 660000 seconds
Time Since Wake:       4203 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
Exception Codes:       UNKNOWN_0x32 at 0x0000000100b6c000
Exception Codes:       0x0000000000000032, 0x0000000100b6c000

Termination Reason:    Namespace CODESIGNING, Code 2 Invalid Page

VM Region Info: 0x100b6c000 is in 0x100b6c000-0x101044000;  bytes after start: 0  bytes before end: 5079039
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      mapped file                 100b04000-100b6c000    [  416K] r--/rwx SM=COW  ...t_id=9abb3401
--->  mapped file                 100b6c000-101044000    [ 4960K] r--/rwx SM=COW  ...t_id=99848d01
      GAP OF 0x6a9d4000 BYTES
      Stack Guard                 16ba18000-16f21c000    [ 56.0M] ---/rwx SM=NUL  

Thread 0 Crashed:
0   dyld                          	       0x18bcee3b0 dyld3::MachOFile::hasMachOMagic() const + 0
1   dyld                          	       0x18bcc56c4 dyld4::SyscallDelegate::mapFileReadOnly(Diagnostics&, char const*, unsigned long*, dyld4::FileID*, bool*, char*) const + 228
2   dyld                          	       0x18bcc5028 dyld4::SyscallDelegate::withReadOnlyMappedFile(Diagnostics&, char const*, bool, void (void const*, unsigned long, bool, dyld4::FileID const&, char const*) block_pointer) const + 88
3   dyld                          	       0x18bcbe184 dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int, mach_o::Layout const*) + 208
4   dyld                          	       0x18bcb510c invocation function for block in dyld4::Loader::getLoader(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&) + 1260
5   dyld                          	       0x18bcb43b4 dyld4::Loader::forEachResolvedAtPathVar(dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, dyld4::ProcessConfig::PathOverrides::Type, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 184
6   dyld                          	       0x18bca29b4 dyld4::ProcessConfig::PathOverrides::forEachPathVariant(char const*, dyld3::Platform, bool, bool, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) const + 688
7   dyld                          	       0x18bcb413c dyld4::Loader::forEachPath(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 248
8   dyld                          	       0x18bcb488c dyld4::Loader::getLoader(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&) + 584
9   dyld                          	       0x18bcbc174 invocation function for block in dyld4::JustInTimeLoader::loadDependents(Diagnostics&, dyld4::RuntimeState&, dyld4::Loader::LoadOptions const&) + 408
10  dyld                          	       0x18bceeb64 invocation function for block in dyld3::MachOFile::forEachDependentDylib(void (char const*, bool, bool, bool, unsigned int, unsigned int, bool&) block_pointer) const + 148
11  dyld                          	       0x18bc9c2d8 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 296
12  dyld                          	       0x18bcee9ac dyld3::MachOFile::forEachDependentDylib(void (char const*, bool, bool, bool, unsigned int, unsigned int, bool&) block_pointer) const + 172
13  dyld                          	       0x18bcbbec0 dyld4::JustInTimeLoader::loadDependents(Diagnostics&, dyld4::RuntimeState&, dyld4::Loader::LoadOptions const&) + 164
14  dyld                          	       0x18bca0500 dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 1236
15  dyld                          	       0x18bc9fdc4 start + 2404


Thread 0 crashed with ARM Thread State (64-bit):
    x0: 0x0000000100b6c000   x1: 0x0000000000000000   x2: 0x000000016fa15af8   x3: 0x0000000000040002
    x4: 0x0000000000000003   x5: 0x0000000000000000   x6: 0x000000016fa15af8   x7: 0x0000000000000b30
    x8: 0x000000016fa15990   x9: 0x000000016fa15998  x10: 0x000000000100000f  x11: 0x0000000000000001
   x12: 0x000000016fa16122  x13: 0x0000000000000000  x14: 0x0000000226f8bd2a  x15: 0x0000000000008000
   x16: 0x000000000000005c  x17: 0x000000018bc9de7c  x18: 0x0000000000000000  x19: 0x000000016fa16020
   x20: 0x0000000000000003  x21: 0x000000016fa15acf  x22: 0x0000000100b6c000  x23: 0x000000016fa15af8
   x24: 0x000000016fa15ad0  x25: 0x000000016fa15af0  x26: 0x000000016fa15f90  x27: 0x0000000000000000
   x28: 0x0000000000000000   fp: 0x000000016fa15ab0   lr: 0x000000018bcc56c4
    sp: 0x000000016fa15990   pc: 0x000000018bcee3b0 cpsr: 0x80001000
   far: 0x0000000100b6c000  esr: 0x92000007 (Data Abort) byte read Translation fault

Binary Images:
       0x18bc9a000 -        0x18bd244bf dyld (*) <de46dd52-4994-3fd8-b4b4-e352a1a19354> /usr/lib/dyld
               0x0 - 0xffffffffffffffff ??? (*) <00000000-0000-0000-0000-000000000000> ???

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=763.0M resident=0K(0%) swapped_out_or_unallocated=763.0M(100%)
Writable regions: Total=8832K written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=8832K(100%)

                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
STACK GUARD                       56.0M        1 
Stack                             8176K        1 
VM_ALLOCATE (reserved)             144K        1         reserved VM address space (unallocated)
__DATA                              27K        2 
__DATA_CONST                        38K        2 
__DATA_DIRTY                         7K        1 
__LINKEDIT                       762.4M        2 
__TEXT                             636K        2 
dyld private memory                512K        2 
mapped file                       8416K        4 
===========                     =======  ======= 
TOTAL                            836.0M       18 
TOTAL, minus reserved VM space   835.8M       18 

[Kazzz-S]

Hi @thomaslima,

I think this is the same M1-specific problem as #48.

@dsus4wang made some research and found a workaround (link).
However, I'm not in a position to test it since I do not own an M1 Mac.

To be honest, I was wondering if you could help regarding this issue.

I've started thinking to buy the one (Mac mini), but very reluctant so far.

Best regards,
Kazzz-S

[thomaslima]

Interesting. That sounds related. Thank you. I will investigate and report back.

[thomaslima]

This is definitely related to that. Unfortunately, I tried the CMAKE_XCODE_ATTRIBUTE_OTHER_CODE_SIGN_FLAGS workaround you proposed to no avail.

I can confirm that the LW build works fine. So one of the two deployment processes are messing with the code signatures:

• macdeployqt
• our own calls to install_name_tool (e.g. https://stackoverflow.com/questions/71744856/install-name-tool-errors-on-arm64). I do get the same warnings.

Maybe the fix will be on the lines of https://github.com/supercollider/supercollider/blob/908d88c9bd352e0b12597422c06537023be4b327/editors/sc-ide/CMakeLists.txt#L554

[thomaslima]

Lol it's not the Qt, it's our HW embeddings.

I think I found the strategy to fix it. It will be a bit painful. Take a look: https://stackoverflow.com/questions/54149144/cannot-codesign-an-app-with-python-and-node

I'm not sure which certificate we're gonna use to sign klayout.app. But I think that we should have been doing it not only for the M1 macs, but also Intel. I think the only difference here is that M1 Macs aren't going to even open unsigned apps.

[Kazzz-S]

Thank you for sharing your time for the investigation.
Yes, the above strategy looks painful.

[thomaslima]

Thoughts on compiling a dmg for system python/ruby only and making the other distro installable via homebrew/macports? Maybe this way we don't have to worry? Now system python is 3.8.

…

On Wed, Feb 22, 2023 at 4:43 PM Kazunari Sekigawa ***@***.***> wrote: Thank you for sharing your time for the investigation. Yes, the above strategy looks painful. — Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIJZEZKM6Y5HEEVG7ICS33WY2B6NANCNFSM6AAAAAAU6KBKL4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Kazzz-S]

Thoughts on compiling a dmg for system python/ruby only and making the other distro installable via homebrew/macports? Maybe this way we don't have to worry? Now system python is 3.8.

Yes, that is an idea! I'll investigate on Intel Mac.

[thomaslima]

Hello @Kazzz-S this issue not contained to us anymore and some colleagues are reporting it to me. I am tagging a few people here so they're aware of the solution, maybe they can help crafting a script that would do the code signing for us.
@lukasc-ubc
@joamatab
@Hsuan-Tung
@mustafacc

[lukasc-ubc]

Thank you Thomas. Indeed, I am getting the same error when I try to compile a HW build on my M1.

[Kazzz-S]

Thank you, @thomaslima, for the info and for sharing your time.