Use persistent connections for ingest.sentry.io

[piotrdomagalski]

Problem Statement

We've observed issues on our NAT instance, where there's a lot of connections being opened and then closed to *.ingest.sentry.io. This makes us run out of port allocation in some cases resulting in connection errors (NB, this could also have been the cause of this).

This seems to be caused by the fact that the sentry-ruby gem doesn't use persistent HTTP connections. I looked at the transport class and there doesn't seem to be a way to configure this. Is this something that would be on your radar any time soon?

I'm aware one could provide their own transport class, but right now this requires copying over quite a lot of Sentry specific handling (such as retries, etc). It would be ideal if we wouldn't need to do this and keep the transport class up to date with the upstream.

Solution Brainstorm

• net/http could be used in a way that keeps the connection open for some configured time

[sl0thentr0py]

not sure what the best way to support this is, since it will require experimentation on how long to keep such a connection open and so on.

What you could do on your end is implement a custom transport with the gem net-http-persistent like so:

require 'net/http/persistent'

class PersistentHTTPTransport < Sentry::HTTPTransport
  def initialize(*args)
    super
    @http = Net::HTTP::Persistent.new 'sentry'
  end

  def send_data(data)
    encoding = ""
    if should_compress?(data)
      data = Zlib.gzip(data)
      encoding = GZIP_ENCODING
    end

    headers = {
      "Content-Type" => CONTENT_TYPE,
      "Content-Encoding" => encoding,
      "User-Agent" => USER_AGENT
    }

    auth_header = generate_auth_header
    headers["X-Sentry-Auth"] = auth_header if auth_header

    request = ::Net::HTTP::Post.new(endpoint, headers)
    request.body = data
    response = @http.request(URI.new(endpoint), request)

    if response.code.match?(/\A2\d{2}/)
      handle_rate_limited_response(response) if has_rate_limited_header?(response)
    elsif response.code == "429"
      log_debug("the server responded with status 429")
      handle_rate_limited_response(response)
    else
      error_info = "the server responded with status #{response.code}"
      error_info += "\nbody: #{response.body}"
      error_info += " Error in headers is: #{response['x-sentry-error']}" if response["x-sentry-error"]

      raise Sentry::ExternalError, error_info
    end
  rescue SocketError, *HTTP_ERRORS => e
    on_error if respond_to?(:on_error)
    raise Sentry::ExternalError.new(e&.message)
  end
end

and then use it:

Sentry.init do |config|
  # ...
  config.transport.transport_class = PersistentHTTPTransport
end

You should of course test this out before deploying since it's a third party gem and I have no idea if it works well or not.

[piotrdomagalski]

not sure what the best way to support this is, since it will require experimentation on how long to keep such a connection open and so on.

What you could do on your end is implement a custom transport with the gem net-http-persistent like so:
...

Thank you! Yes, that's what I was considering doing. The main obstacle for future maintainability would be the fact that there's quite some code copied from the original transport class and this needs to be kept in sync.