Embedding of Angelscript stack traces into native stack traces

[mikaelkalms-ttkgames]

Hi!

We are using Unreal Engine with Angelscript as an embedded scripting language. This integration was originally done by a game studio called Hazelight, but has since been picked up by Embark Studios and several other game studios. The language offers great ergonomics for development times and iterations, but postmortem debugging is difficult.

The biggest limitation is that the stacktrace that Sentry sees contains no information of what is happening within the Angelscript interpreter. This means that Sentry is unable to take into account what has been happening in Angelscript land when grouping callstacks. It also means that when a person looks at an event, they get an excellent presentation of the native stacktraces + context information, but need to dig a bit to find out anything about the Angelscript side.

We can (and have) improved this a bit by attaching Angelscript-specific context information & printing things to logs ... but it would be really nice to inject Angelscript stackframes into the native callstacks. This would allow Sentry to group events intelligently, taking Angelscript execution into account.

---

A typical stacktrace looks like below. In this example, the code has transitioned C++ => AS => C++ => AS => C++ and then dereferenced a nullptr:

... a bunch of native frames ...
FAngelscriptManager::Execute()
... a bunch of native frames ...
FAngelscriptManager::Execute()
... a bunch of native frames ...
<thread entry point>

In the above example, FAngelscriptManager::Execute() is the entry point for a function call into Angelscript land. There's a bytecode interpreter embedded within the function, with an infinite loop + a switch statement that processes individual instructions of the AngelScript VM. One of those VM instructions is "make a function call into C++", which is how the Angelscript script logic has called back to the C++ side twice in the above example.

---

My first question here is, are there any general strategies to employ when making an interpreted language a "first-class citizen" of events in Sentry?

Secondly, do you have any pointers for this particular case? To narrow things down a bit,

• Angelscript is purely an interpreted language in this scenario; JIT / AOT / transpilation is out of scope
• The Angelscript interpreter has callstack info organized neatly in internal data structures, there are already C++ helper functions to "print Angelscript stacktrace to stdout", which demonstrates that the necessary information exists

---

I have tried retrieving modifying the Sentry Unreal plugin to capture the callstack (tested on Win64) with sentry_value_set_stacktrace(), for crashes, but similar to how FGenericPlatformSentrySubsystem::CaptureMessage() etc does it, identified any occurrences of the return address of FAngelscriptManager::Execute() within that callstack, and inserted Angelscript specific entries according to the Stack Trace Interface format.

This resulted in a datastructure that looked good on paper - at least for the main thread, I didn't attempt to do this for all threads in the program. However, it appears that after the Sentry Unreal plugin uploaded the results to the Sentry backend (again, using Win64, so using crashpad as an intermediary step on the client side, and mechanism = minidump on the backend side), the custom stacktrace I supplied was ignored by the backend.

Perhaps mechanism = minidump means that client-supplied stacktraces will get ignored?

Perhaps I messed up the format?

Perhaps clientside-captured stacktraces are a fool's errand (for example, once one starts to ship executables without complete debug info to customers)?

Any ideas or input would be appreciated!

[tustanivsky]

Hey @mikaelkalms-ttkgames, thank you for bringing this up.

Currently, none of the underlying SDKs that the Unreal plugin relies on internally support hybrid stacktraces where native frames can be mixed with frames from another scripting language. In the case of crashes, populating the callstack on the client side (as is done for non-fatal messages) won’t give you the expected results since this information is extracted from the minidump during event processing on Sentry servers.

That said, you could technically construct your own minidump containing both native and Angelscript frames and capture it manually using sentry_capture_minidump. Native SDK does something similar when client-side stack unwinding is enabled via build settings (see CRASHPAD_ENABLE_STACKTRACE) which makes it possible to get human-readable stacktraces for captured crashes even without uploaded debug symbols.

However, considering that the approach above is rather complex and that the existing Unreal plugin implementation isn’t designed to support this kind of customization, submitting your hybrid callstack as part of a separate context or possibly as an attachment seems to be the only way for now.

[mikaelkalms-ttkgames]

That makes sense. We will stick with context/attachment data for now. Thanks for your guidance!