Mitsuba3 & drjit: Segmentation Fault on MacOS with llvm_ad_rgb variant

[coult099]

Hello,

I am having an issue with the 'llvm_ad_rgb' variant of mitsuba3 on my system, and I do not know how to diagnose the issue.

My system:

• MacOS Catalina (10.15.7)
• Python 3.10.8 64-bit
• LLVM version 15.0.3 (installed using Homebrew)
• Installed: cmake, swig (to build LLVM)
• Installed [email protected], six, z3, zstd (required)
• Attempting to run on: Visual Studio Code.

Code that causes the issue:

import mitsuba as mi
mi.set_variant('llvm_ad_rgb') 
scene = mi.load_file('scenes/cornell-box/scene.xml')
image = mi.render(scene)

At this point I get: Segmentation fault: 11.

When running in a Jupyter notebook, I get:

info 13:44:17.942: UpdateWorkingDirectoryAndPath in Kernel
error 13:44:24.668: Disposing session as kernel process died ExitCode: undefined, Reason: /Users/my-name/Library/Python/3.10/lib/python/site-packages/traitlets/traitlets.py:2412: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
/Users/my-name/Library/Python/3.10/lib/python/site-packages/traitlets/traitlets.py:2366: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use '1bf3cb4b-b008-4636-91d6-19404fdabda5' instead of 'b"1bf3cb4b-b008-4636-91d6-19404fdabda5"'.
  warn(

info 13:44:24.671: Dispose Kernel process 6084.
error 13:44:24.671: Raw kernel process exited code: undefined
error 13:44:24.672: Error in waiting for cell to complete [Error: Canceled future for execute_request message before replies were done
	at t.KernelShellFutureHandler.dispose (/Users/my-name/.vscode/extensions/ms-toolsai.jupyter-2022.9.1202862440/out/extension.node.js:2:32353)
	at /Users/my-name/.vscode/extensions/ms-toolsai.jupyter-2022.9.1202862440/out/extension.node.js:2:51405
	at Map.forEach (<anonymous>)
	at y._clearKernelState (/Users/my-name/.vscode/extensions/ms-toolsai.jupyter-2022.9.1202862440/out/extension.node.js:2:51390)
	at y.dispose (/Users/my-name/.vscode/extensions/ms-toolsai.jupyter-2022.9.1202862440/out/extension.node.js:2:44872)
	at /Users/my-name/.vscode/extensions/ms-toolsai.jupyter-2022.9.1202862440/out/extension.node.js:2:2218254
	at t.swallowExceptions (/Users/my-name/.vscode/extensions/ms-toolsai.jupyter-2022.9.1202862440/out/extension.node.js:7:130943)
	at p.dispose (/Users/my-name/.vscode/extensions/ms-toolsai.jupyter-2022.9.1202862440/out/extension.node.js:2:2218232)
	at t.RawSession.dispose (/Users/my-name/.vscode/extensions/ms-toolsai.jupyter-2022.9.1202862440/out/extension.node.js:2:2223340)
	at process.processTicksAndRejections (node:internal/process/task_queues:96:5)]
warn 13:44:24.673: Cell completed with errors {
  message: 'Canceled future for execute_request message before replies were done'
}
info 13:44:24.674: Cancel all remaining cells true || Error || undefined

I am not sure if the issue is that mitsuba cannot find my version of LLVM since I installed it with Homebrew, but I did run the line:

 LDFLAGS="-L/usr/local/opt/llvm/lib/c++ -Wl,-rpath,/usr/local/opt/llvm/lib/c++"

as suggested when installing LLVM with Homebrew.

Any help or insight is greatly appreciated!

[Speierers]

Hi @coult099 ,

Could you compile Mitsuba on your system in debug mode and run the same lines in a debugger?

[coult099]

Hello, and thank you for the reply. Apologies for the long answer, but I tried to be as thorough as possible.

Using Python pdb on the lines above, I see the following:

-> import mitsuba as mi
-> import drjit as dr
-> mi.set_variant('llvm_ad_rgb')
(Pdb) step
--Call--
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(223)__getattribute__()
...

In the function find_spec(self, fullname, path, target=None) at line 89 in __init__.py:

> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/_distutils_hack/__init__.py(92)find_spec()
-> if path is not None and not fullname.startswith('test.'):
-> return
...
--Return--
> <frozen importlib._bootstrap_external>(1376)_path_importer_cache()->FileFinder('/...ages/mitsuba')
...
--Return--
> <frozen importlib._bootstrap_external>(147)_path_stat()->os.stat_resul...me=1666201985)
...
--Return--
> <frozen importlib._bootstrap_external>(128)<listcomp>()->['/Library/Fra...kages/mitsuba', 'ad.cpython-310-darwin.so']
--Return--
> <frozen importlib._bootstrap_external>(128)_path_join()->'/Library/Fra...310-darwin.so'

And this repeats several times, with the additions:

'ad.abi3.so'
'ad.so'
'ad.py'
'ad.pyc'
'ad'
'__init__.cpy...310-darwin.so'

Then I see

--Call--
> <frozen importlib._bootstrap_external>(140)_path_stat()
> <frozen importlib._bootstrap_external>(147)_path_stat()
FileNotFoundError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/python/ad/__init__.cpython-310-darwin.so'
> <frozen importlib._bootstrap_external>(147)_path_stat()
--Return--
> <frozen importlib._bootstrap_external>(147)_path_stat()->None
FileNotFoundError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/python/ad/__init__.cpython-310-darwin.so'
...
--Return--
> <frozen importlib._bootstrap_external>(155)_path_is_mode_type()->False
--Return--
> <frozen importlib._bootstrap_external>(161)_path_isfile()->False

And then the same FileNotFoundError with the additions

/mitsuba/python/ad/__init__.abi3.so
/mitsuba/python/ad/__init__.abi3.so'

Then I see

--Return--
> <frozen importlib._bootstrap_external>(128)<listcomp>()->['/Library/Fra...uba/python/ad', '__init__.py']
--Return--
> <frozen importlib._bootstrap_external>(128)_path_join()->'/Library/Fra...d/__init__.py'
...
--Return--
> <frozen importlib._bootstrap_external>(156)_path_is_mode_type()->True
--Return--
> <frozen importlib._bootstrap_external>(161)_path_isfile()->True

Then when importing .integrators I see:

--Call--
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/python/ad/__init__.py(1)<module>()
-> from .integrators import *
...
--Return--
> <frozen importlib._bootstrap>(123)acquire()->True 
--Return--
> <frozen importlib._bootstrap>(171)__enter__()->None
...
FileNotFoundError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/python/ad/integrators/__init__.cpython-310-darwin.so'
> <frozen importlib._bootstrap_external>(147)_path_stat()
--Return--
> <frozen importlib._bootstrap_external>(147)_path_stat()->None
FileNotFoundError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/python/ad/integrators/__init__.cpython-310-darwin.so'
> <frozen importlib._bootstrap_external>(153)_path_is_mode_type()

and the same FileNotFoundError with the additions

/mitsuba/python/ad/integrators/__init__.abi3.so
/mitsuba/python/ad/integrators/__init__.so

until I get

KeyError: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/python/ad/integrators'
> <frozen importlib._bootstrap_external>(1372)_path_importer_cache()
...
zipimport.ZipImportError: not a Zip file
> <frozen zipimport>(89)__init__()
--Return--
> <frozen zipimport>(89)__init__()->None
zipimport.ZipImportError: not a Zip file

and the rest seems to run normally.

Now, when I run Pdb on scene = mi.load_file('scenes/cornell-box/scene.xml') I see

> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(240)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba' has no attribute 'load_file'
...
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(110)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba.llvm_ad_rgb' has no attribute 'load_file'
...
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(240)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba' has no attribute 'render'
...
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(110)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba.llvm_ad_rgb' has no attribute 'render'
...
KeyError: 'mitsuba.python'
> <frozen importlib._bootstrap>(188)_get_module_lock()
...
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(240)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba' has no attribute 'Scene'
...
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(110)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba.llvm_ad_rgb' has no attribute 'Scene'
...
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(240)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba' has no attribute 'Sensor'
...
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(110)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba.llvm_ad_rgb' has no attribute 'Sensor'
...
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(240)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba' has no attribute 'sample_tea_32'
...
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mitsuba/__init__.py(110)__getattribute__()
-> return super().__getattribute__(key)
AttributeError: module 'mitsuba.llvm_ad_rgb' has no attribute 'sample_tea_32'

And then finally I see

> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/drjit/traits.py(281)is_diff_v()
-> return getattr(arg, 'IsDiff', False)
--Return--
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/drjit/traits.py(281)is_diff_v()->False
-> return getattr(arg, 'IsDiff', False)
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/drjit/router.py(3932)detach()
-> elif _dr.is_struct_v(arg):
--Call--
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/drjit/traits.py(763)is_struct_v()
-> def is_struct_v(arg, /):
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/drjit/traits.py(779)is_struct_v()
-> return hasattr(arg, 'DRJIT_STRUCT')
--Return--
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/drjit/traits.py(779)is_struct_v()->False
-> return hasattr(arg, 'DRJIT_STRUCT')
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/drjit/router.py(3942)detach()
-> return arg
--Return--
Segmentation fault: 11

[Speierers]

Thanks for this in-depth answer. It would be very useful if you could run this into a C++ debug, like lldb (after compiling Mitsuba in debug mode) to better understand what causes the Segmentation Fault at the C++ level.

[coult099]

Hello, thank you for the reply and apologies for the delay. I installed LLVM 13, and removed LLVM 15 from my PATH, per the suggestion below from @njroussel. Then, I compiled Mitsuba in debug mode by adding the line

set(CMAKE_BUILD_TYPE Debug)

to CMakeList.txt. Then I ran lldb build/mitsuba to begin the debug process, and launched the program using

process launch -- -m llvm_ad_rgb resources/data/scenes/hello.xml

The output I got was

Process 24192 launched: '.../mitsuba3/build/mitsuba' (x86_64)
INFO  main  [mitsuba.cpp:340] Mitsuba version 3.0.2 (master[2d773473], Mac OS, 64bit, 4 threads, 8-wide SIMD)
INFO  main  [mitsuba.cpp:341] Copyright 2022, Realistic Graphics Lab, EPFL
INFO  main  [mitsuba.cpp:342] Enabled processor features: llvm avx f16c sse4.2 x86_64
WARN  main  [mitsuba.cpp:345] Renderer is compiled in debug mode, performance will be considerably reduced.
INFO  main  [xml.cpp:1405] Loading XML file "resources/data/scenes/hello.xml" with variant "llvm_ad_rgb"..
DEBUG main  [PluginManager] Loading plugin "plugins/srgb.dylib" ..
DEBUG main  [PluginManager] Loading plugin "plugins/d65.dylib" ..
DEBUG main  [PluginManager] Loading plugin "plugins/regular.dylib" ..
DEBUG main  [PluginManager] Loading plugin "plugins/direct.dylib" ..
DEBUG main  [PluginManager] Loading plugin "plugins/box.dylib" ..
DEBUG main  [PluginManager] Loading plugin "plugins/hdrfilm.dylib" ..
DEBUG main  [PluginManager] Loading plugin "plugins/independent.dylib" ..
DEBUG main  [PluginManager] Loading plugin "plugins/perspective.dylib" ..
DEBUG main  [PluginManager] Loading plugin "plugins/diffuse.dylib" ..
DEBUG main  [PluginManager] Loading plugin "plugins/ply.dylib" ..
DEBUG main  [PLYMesh] Loading mesh from "teapot.ply" ..
DEBUG main  [PLYMesh] "teapot.ply": read 2256 faces, 1177 vertices (54 KiB in 8ms)
Process 24192 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x13c89a0a8)
    frame #0: 0x000000013c89a0a8
error: memory read failed for 0x13c89a000

and if I run thread continue I get the same output:

(lldb) thread continue
Resuming thread 0xb69bf in process 24192
Process 24192 resuming
Process 24192 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x13c89a0a8)
    frame #0: 0x000000013c89a0a8
error: memory read failed for 0x13c89a000

Thank you again for any insight you may have about this.

[coult099]

I am not sure if this helps, but when I run Mitsuba in scalar_rgb mode and try to run the "Editing a Scene" tutorial here, I get the following error when running the line

V = dr.unravel(mi.Point3f, params['teapot.vertex_positions'])

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/drjit/router.py", line 1106, in unravel
    raise Exception("unravel(): expected a nested array as target type!")
drjit.Exception: unravel(): expected a nested array as target type!
drjit.Exception: unravel(): expected a nested array as target type!

[njroussel]

Sorry for the delay,
Your debugging trace is missing symbols, it looks like it didn't pickup your debug build.

File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/drjit/router.py"

This line also suggests that in fact it is a pre-installed version of drjit that is being used rather than your manual debug build (unless you ran something like pip install ./mitsuba on your local folder. Can you try uninstalling mitsuba and drjit, and then rerun your script in lldb. You'll most likely also need to source the build/setpath.sh script in order for Python to see mitsuba.

[njroussel]

For what it's worth, we've had quite a few reports about issues with LLVM 14 (#282 for example). Maybe there are even more issues with newer versions of LLVM, as you're using version 15.

It still would be nice to get a better understanding of what the issue is here. I'm just suggesting a LLVM downgrade in case you need to quickly be unstuck.

[coult099]

Hello, thank you for your response. I downgraded to LLVM 13, but unfortunately am still facing the same issue.