retryable_gets & retryable_writes with infinity loop (no timeout capabilities)

[mlarribe]

it could be interesting to allow the developers to define a timeout of the retryable feature, while testing it on a replicated cluster of 3 node facing a bad "weather condition" (no majority after incident "normal behaviour before human intervention in the case")

I got the impression that when even the cluster get stuck in bad condition the calls with such retryable will get stuck forever till the cluster gets back online in a correct statement (could be seconds, hours or even days).
This could be very convenient to allow the retry to operation till a decent timeout were after a certain point, giving back to the call either "raised issue" of feedback broken behond expectation could be truly helpful, without needing to manually having make a custom retry loop on the project code base to manage all the detailed articulation from the session to the reply-codes that could be allowed or not to be retried externally to of the driver. (making the feature less usable)

in the context, having the capacity to let the drive do the job (retry as much as it'd like) till a certain threshold is reach, and allow the call to still act in order to take a more radical approach according to their needs (fail-over solutions, crash, else..) without behind blind of root cause and/or intricated complexe external management of MongoDB return out of MongoDB driver.

[zookzook]

Whenever the driver sends a request to the server, there is a timeout when waiting for the response:

  @doc """
    Sends a request id and waits for the response with the same id
  """
  def post_request(op, id, state) do
    with :ok <- send_data(encode(id, op), state),
         {:ok, ^id, response} <- recv_data(nil, "", state) do
      get_doc(response)
    end
  rescue
    error ->
      Logger.error("Unable to send request to database because of #{inspect(error)}")
      {:error, error, state}
  end

  defp recv_data(nil, "", %{connection: {mod, socket}} = state) do
    case mod.recv(socket, @header_size, state.timeout) do
      {:ok, tail} -> recv_data(nil, tail, state)
      {:error, reason} -> recv_error(reason, state)
    end
  end


...

For me, it is not clear where to define a timeout for the retryable feature.

[mlarribe]

I found on the documentation of specification of mongodb directly the mention of CSOT
https://github.com/mongodb/specifications/blob/master/source/retryable-writes/retryable-writes.md
However when [Client Side Operations Timeout](https://github.com/mongodb/specifications/blob/master/source/client-side-operations-timeout/client-side-operations-timeout.md) is enabled, the driver will retry multiple times until the operation succeeds, a non-retryable error is encountered, or the timeout expires. Retrying multiple times provides greater resilience to cascading failures such as rolling server restarts during planned maintenance events.

https://github.com/mongodb/specifications/blob/master/source/retryable-reads/retryable-reads.md
If a retry attempt also fails and [Client Side Operations Timeout](https://github.com/mongodb/specifications/blob/master/source/client-side-operations-timeout/client-side-operations-timeout.md) (CSOT) is enabled and the timeout has not yet expired, then the Driver MUST jump back to step 2b above in order to allow multiple retry attempts.

I was and I'm still looking for the way to explicitly tell the drive to retry according to your accepted retry edge but when ever you reach this timeout for the whole operation (not on a given attempt) give me the hand back so, I can cascade a custom logic according to it