How to *really* use sentry in a dockerized environment

[JanFellner]

The documentation points out on the one hand to use a shared volume (which would have been my thinking prior to having it read) but on the other to let the crashpad_handler wait synchronously (which would on the other hand make that shared volume obsolete)

• Should i prefer waiting
• Should i use the shared volume
  • If so, is there a way to tell sentry to not call the crashpad_handler.exe at all on shutdown but instead always do that out of process when the container is restartet? (I am thinking on glitchiness when the crashpad handler is started and in the meantime i the complete runtime is shutting down as the main process has terminated)

[supervacuus]

TL;DR: If you use the crashpad backend in a container, you should do both (wait + durable volume).

---

For a more detailed perspective, you have to consider those two as mostly orthogonal aspects that play together and not a binary choice:

• If you can, you should map the database path to a host directory, because relevant state (see below) that ideally should survive until the next start will be stored there
• This is particularly important to backends that send reports during the next start (breakpad and inproc), since otherwise those wouldn't send any reports. However, it is also for crashpad because the crashpad_handler could fail to send a report and would retry on the next start
• Crash-markers that let you query whether the last run led to a crash will be stored there
• Events and sessions enqueued at the time of crash or normal shutdown will be stored in the database path too, so that they can be sent during the next run
• User consent (if relevant to your use case) would also be stored in the database path
• If multiple processes share the same database path (which, in container setups that ask for mounted volumes, typically is not the case), then the file-based lock would also end up there

So, while starting from a fresh database path on every run is technically feasible, it will limit the availability of certain features (some of which are essential depending on your use case).

Waiting before upload, otoh, is only relevant when you use the crashpad backend in a container (or any other process manager that kills the entire process tree when the root terminates), and in that case it should be enabled, because the container will shut down your application and the crashpad_handler, as soon as the application terminates. Since the crashpad_handler signals the application to terminate after it has gathered the necessary snapshot data, it is the sole instance that can ensure that both processes live long enough to deliver a report successfully.

If so, is there a way to tell sentry to not call the crashpad_handler.exe at all on shutdown but instead always do that out of process when the container is restartet? (I am thinking on glitchiness when the crashpad handler is started and in the meantime i the complete runtime is shutting down as the main process has terminated)

I am not sure I fully follow, but there may be a misunderstanding of the crash reporting process when using crashpad. So let me quickly summarize what happens in non-container setups:

1. The crashpad_handler is started when you initialize the SDK (not on shutdown)
2. When a crash is registered via the signal handler installed by the SDK in your application process, the client of the SDK (the part that runs in your process) communicates to the crashpad_handler that it should gather a report
3. Once the crashpad_handler retrieved the required data for the report from the (still) running application, without an enabled wait_for_upload flag, it informs the client to terminate the process (ideally as quickly as possible since crashed applications are a safety/security risk).
4. The monitored application terminates. At the same time, the crashpad_handler keeps running until it finishes the upload (or fails due to network errors, in which case it persists for a retry during the next run).

3 and 4 would be problematic in a container since the crashpad_handler would be force-terminated as soon as the root process terminates. At this point, the crashpad_handler could be inconsistent, not having sent the report back to Sentry or consistently updating the crash database (in the database path). wait_for_upload ensures that these steps happen before the client in your application receives the "Okay for termination". In both cases, the interaction between the client in your application and the handler is fully synchronized.

[supervacuus]

Since you wrote crashpad_handler.exe, wait_for_upload is currently only supported in Linux containers.

[JanFellner]

Yeah i was wrong in how sentry handels the dump, i thought crashpad_handler is just there to upload the dump out of process. I was not aware that it´s also gathering the dump.

I justed very briefly took a look into what the wait_for_upload is doing on the linux side.
It looks like just the FlushUploadThread is missing (as well as handing over the flags accordingly)
Or am i missing some magic that it´s not available on windows?

//  Force Crashpad Handler to wait for one upload attempt if there is a pending report
void CrashReportExceptionHandler::FlushUploadThread() {
  if (upload_thread_ && upload_thread_->is_running()) {
    // Following "Stop" call terminates the upload thread after completing whatever task it is
    // performing e.g. uploading a report. If it is not performing any task, it will terminate
    // immediately. It blocks while waiting for the upload thread to terminate making
    // sure that the pending report (if any) is sent before returning.
    upload_thread_->Stop();
  }

We could try to implement it and open a PR for that missing puzzle piece ;)

(Yeah, we have windows docker containers 😉)

[supervacuus]

I already looked into this because this topic has been raised in a different context: #1255

[JanFellner]

Owing you some 🍺🍻next time i visit Graz (or you come to Salzburg where i am every ~4-6 weeks. 😉)