Crash in http exporter, for tcp half open connections

[VivekSubr]

Steps to reproduce
Sudden bring down of otel collector in a high load setup.

What is the expected behavior?
Clients handling export failures

What is the actual behavior?
Hang on export, 30+ seconds. We can see one connection stuck in half open, one stuck in connect SYN_SENT... note that server is gone for some time at this point.

netstat -nap | grep 4318
tcp 0 0 192.168.67.174:51510 192.168.10.101:4318 ESTABLISHED 23/smfcc
tcp 0 1 192.168.67.174:43912 192.168.10.101:4318 SYN_SENT -

Additional context
We suspect this crash is due to otel not providing way to set keep alives on client connections (CURLOPT_TCP_KEEPALIVE)... we've set timeout (CURLOPT_TIMEOUT) to 5s, and ideally for that CURLOPT_TCP_KEEPALIVE should be set to 1s, but otel doesn't have an option to set that.

Request that a config for TCP_KEEPALIVE be provided.

[owent]

I’ll be on holiday and will add it a few days later if no one else does.

[owent]

I've been thinking about this issue recently. HTTP has quite a few options, and there may be new option requirements in the future.

The current approach is to provide interfaces for each option separately in HttpClient and provide options in OtlpHttpClient, OtlpHttp*Exporter.

This results in having to modify a very large number of component interfaces with each change.

Can we categorize these options into three types: integer, bool, and string? Then only provide three sets of interfaces that accept three enumeration types and their corresponding type values?

For example:

enum class HttpBoolOption {
  kFollowLocation,
  kSSLVerifyPeer,
};
enum class HttpInt64Option {
  kMaxRedirs,
};
enum class HttpStringOption {
  kSSLCipherList,
};

void SetBoolOption(HttpBoolOption, bool);
void SetInt64Option(HttpInt64Option, int64_t);
void SetStringOption(HttpStringOption, std::string);

This way, if we need to add new options in the future, we can implement them by only modifying the enumeration definitions, without having to change the interfaces of every layer of components.

What do you all think about making this change? If it's feasible, I'll prepare to implement it next. Do we need to consider API backward compatibility?

@open-telemetry/cpp-approvers