Is there a way to inform the user of the SDK if, and which LogRecord failed to be sent to the collector for audit logging purposes?

[simaoribeiro]

Hello,

the company I work for uses OTEL for traces, and metrics, and we're starting the migration of our logging infrastructure. I develop internal libraries that wraps the .NET SDKs (to define company-wide behaviour) and are used by our engineering of teams. Currently, I'm in the process of moving our audit logs library to use OTEL.
These logs have the requirement that they should not fail to be logged. So, as an example, if our log collector is not reachable, the users of our library should be notified that the log is failing and which one is failing, so they can implement resilience strategies if needed (e.g.: saving the log in a database for a later process).
I know that the SDK has a retry mechanism, in memory or disk, however, it depends on the timout defined in the Exporter and the value in the timeout can be smaller than a possible incident affecting our collector.

I tried to explore several options to address my requirement:

1. Make use of the IHttpClientFactory, so I could use a handler to check the result of the request, however the factory is disabled for the OtlpLogExporter.

2. Subscribe to OpenTelemetry-Exporter-OpenTelemetryProtocol EventSource. This option poses 2 problems: I'm unable to know which log failed (only know that it failed) and there is no specific event being fired at the end of the retry action (eg.: The eventId 2 FailedToReachCollector is raised for each try of each message).

3. Create my own processor and exporter that wraps the OtlpLogExporter and gives me access to the ExportResult and any exception being thrown. This option works for my use case but I'm finding some bugs (e.g.: the resource attributes not being propagated no the OtlpLogExporter that I wrapped). I would prefer to use the driver's already existent processors and exporters as they already have years of tests and usage.

Given these constraints, is there any option that I'm missing on my analysis?

[cijothomas]

So, as an example, if our log collector is not reachable, the users of our library should be notified that the log is failing and which one is failing, so they can implement resilience strategies if needed (e.g.: saving the log in a database for a later process).

Are you trying to see if ILogger.Log..() methods can return a success indicator or throw exception when exporting failed?

[simaoribeiro]

As far as I know, the ILogger methods have a void return type by default. Regarding the exceptions, the Otlp Exporter have multiple catches of System.Exception that doesn't allow the exceptions propagate, e.g.:
https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/OtlpLogExporter.cs#L97

[cijothomas]

Yes exactly. I am not clear what is the expected outcome then? Logging Libraries don't returns status or throw exceptions, so end users who use this API to report Audit Logs will not know if the Audit Log they tried to emit was successful or not.

See the related discussion in Spec : open-telemetry/opentelemetry-specification#4377

[simaoribeiro]

Yes, it makes sense for logging libraries to not return status or throw exceptions.
My question was more related to the OtlpExporter that sends those logs to the collector. I was trying to understand if there is any option to receive feedback about if those LogRecords were successfully sent to the collector or not.
The spec discussion you linked answers the requirement I presented in this question. Thank you for that.

[cijothomas]

OtlpExporter that sends those logs to the collector. I was trying to understand if there is any option to receive feedback about if those LogRecords were successfully sent to the collector or not.

Unfortunately no reliable/dependable way exists today. Yes there are internal logs but the mechanism or shape is not a contract, so can change. As of now it's good for troubleshooting only.
It may require some work to let OTLP Exporter expose a contract about failures. But I don't think it can cover everything - for eg:, what if the Log was dropped at the BatchLogProcessor? OTLP Won't even know about it, so you cannot rely on OTLP alone.

[simaoribeiro]

I agree with you. Some changes need to be made in the OTLP Exporter to ensure the reliability of the data for something as important as audit log.
Looking at the Appendix A: Guarantee of Delivery from the Audit Logging proposal, it seems that the guarantee of delivery is a requirement.
I will keep looking closely at the proposal.

Thank you for your help @cijothomas