Failsafe 3.0 instrumentation introduced

[onurkybsi]

As requested in #10856, Failsafe instrumentation introduced. Intrumentation of other Failsafe policies such as RetryPolicy, RateLimiter, Bulkhead will be added with upcoming PRs.

[laurit]

@trask I think this might be the first library only instrumentation we have. Do we need to point this out somehow here? Set the Auto-instrumented versions to N/A? Any suggestions?

[trask]

N/A in the auto-instrumented versions column sounds good to me 👍

[onurkybsi]

Hey, this is fixed. However, fyi, after I'm done with other Failsafe policies'(RetryPolicy, RateLimiter, Bulkhead) library instrumentation, I'm planning to work on the auto insturmentation.

[laurit]

@trask could you suggest units for these metrics

[trask]

maybe {execution}

[laurit]

@trask is using name here fine or should it be circuit_breaker.name?

[trask]

I'd suggest going even further: failsafe.circuit_breaker.name

prefixing it with failsafe.* makes it clear it's an instrumentation specific attribute and we're not attempting to create a general circuit breaker semantic convention (which requires a lot more work)

[jaydeluca]

maybe @trask can weigh in on this too because I'm not sure if it's directly applicable, but the docs around "recording errors on metrics" recommends using a single metric with an attribute that differentiates between success and failure

It’s RECOMMENDED to report one metric that includes successes and failures as opposed to reporting two (or more) metrics depending on the operation status.

[trask]

maybe failsafe.circuit_breaker.state_changes

with attribute failsafe.circuit_breaker.state = open / half_open / closed?

[trask]

I'm not quite sure what this is measuring, is it just the number of times the circuit breaker has allowed an execution, or how many times the execution it has allowed has failed?

[onurkybsi]

This represents the number of executions failed by the configured circuit breaker, for example:

var circuitBreaker = CircuitBreaker
    .builder()
    .handleResultIf(r -> true) // Each execution is failure
    .build();
    
Failsafe.with(circuitBreaker).get(() -> {
    // Execution happens here...
    return null;
});

Failure count will always be incremented for this circuit breaker until the circuit breaker is opened. Once the circuit breaker is open, CircuitBreakerOpenException is thrown until it's half-open again so it is not counted as failure since the circuit breaker didn't let the execution happen.

After a second thought, I'm not sure if this metric is truly valuable, and the same applies to the number of successes. What do you think? Should we keep them?

[otelbot]

Thank you for your contribution @onurkybsi! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey.