Improve testing DX

[whatyouhide]

The current testing DX is not optimal. We require users to:

1. Start up a Bypass server.

2. Change the DSN in the global Sentry configuration:

Sentry.put_config(:dsn, "https://some:garbage@localhost:#{bypass.port}/<garbage-project-id>")

3. Run the test.
4. Make sure to revert the :dsn configuration to whatever it was before. This is especially important if the config before the test was dsn: nil, that is, not sending any events.

This approach also forces users to run the tests that use Sentry with async: false, since the dsn configuration is global to the :sentry application.

In this issue, let's gather some ideas on a better API to solve this.

[whatyouhide]

Idea 1: Test Mode

One idea is this: we can have a "test mode" for Sentry. This would be a global configuration option (so that it can be set in config/test.exs and won't affect prod performance), like this:

# config/test.exs
config :sentry,
  test_mode: true

In test mode, Sentry would not send anything over HTTP, but would instead collect event reports (in a GenServer process, for example).

Then, we'd provide something like:

Sentry.Test.get_reported_events()

This would return all the reported events. The challenge is definitely then checking that the correct report is in there for the test at hand. One option is to add filtering via PID, so that you can get the events reported from a specific PID (often this would be the test PID):

code_that_reports_an_event()

assert [event] = Sentry.Test.get_reported_events(reporter_pid: self())

We could potentially add a pop_reported_events/1 variant, so that by using it in the code would also remove the reported event.

Challenge: global events

The challenge here is when events are reported by other processes that we don't know the PID of (which is a common-enough case). In those cases, users could get all the reported events and then assert that the one they're looking for is in there. However, this is not great and tends to be error-prone if there are multiple tests trying to assert on similar messages (resulting in potential false positives).

One solution for something like this would be to do something a la Mox, where we start the collector for each test, so that only events reported in that test (not necessarily in that test process) show up. What would be a nice API for that?

[yordis]

I would avoid boolean flags honestly, but closer to what you you suggested:

config :sentry,
  # maybe better name, amazing if we leverage Req or Tesla here
  http_adapter: Sentry.TestSupport.HttpAdapter

Sentry.TestSupport.HttpAdapter.get_reported_events(reporter_pid: self())

[wojtekmach]

I haven’t thought this through but I believe global events problem could be elegantly solved by tracking all actions in the system back to the test process that ultimately triggered them using a mechanism like $callers or $ancestors. For example, when we test a LiveView, even though it has been started under some supervisor (I think it’s test supervisor these days), the LV testing API would annotate the process (as well as all its child processes) accordingly. And then we can send messages from whenever to that particular test. As an aside if Sentry used telemetry as “transport” before sending out to the cloud then all we’d need to do is attach handlers in our tests. In any case, this might be a ton of work and I’m probably missing something very obvious that breaks this but I thought I’d mention this anyway. 😅

[josevalim]

We could probably extract the tracking bits from Mox into nimble_ownership or something like that. I even considered having it as part of ExUnit itself because it helps a lot with testing.

[stefanchrobot]

Another idea would be to leave the production code as is and intercept the outgoing HTTP requests. Here's my prototype that I intend to continue working on.