Allow filtering logs being sent to AppInsights by log level

[cosminstefanxp]

Is your feature request related to a problem? Please describe.
I am using pino for logging, and would like trace and debug level logs to be printed to console, but I don't want them in Application Insights. Currently, there's no way, using useAzureMonitor and PinoInstrumentation to filter a log record.

The issue is that useAzureMonitor automatically registers a AzureLogRecordProcessor, before the log processors. So there's no way to stop a log from being sent.

Describe the solution you'd like
A way to filter logs by min level. E.g. could be via a configuration option. Or allow a way to wrap the AzureLogRecordProcessor, so we can do something like:

export class LevelGateProcessor implements LogRecordProcessor {
  constructor(
    private readonly min: SeverityNumber,
    private readonly next: LogRecordProcessor
  ) {}

  onEmit(record: LogRecord) {
    const sev = record.severityNumber ?? SeverityNumber.INFO;
    if (sev >= this.min) this.next.onEmit(record);
  }
  forceFlush = () => this.next.forceFlush();
  shutdown   = () => this.next.shutdown();
}

// and then...
useAzureMonitor({
  // This still controls traces/metrics; fine to keep it
  azureMonitorExporterOptions: { connectionString },

  // Provide your own logs pipeline; the distro will use these instead of the default
  logRecordProcessors: [
    // Only send INFO+ to App Insights; TRACE/DEBUG stay on console (e.g., via pino)
    new LevelGateProcessor(SeverityNumber.INFO, /* Get a reference to the AzureLogRecordProcessor somehow */),
  ],
});

Maybe making the logRecordProcessors a function.

Describe alternatives you've considered
I've tried to use LevelGateProcessor with existing system - no success
I've tried to tweak pino-instrumentation, but it doesn't support this use case either - and probably shouldn't, since it's the purpose of a LogRecordProcessor.
I've tried to not use pino-instrumentation and use a transport directly, but it still wouldn't work with useAzureMonitor.

[JacksonWeber]

@cosminstefanxp I believe your issue can be resolved in a similar way to this: #34635 (comment). Like you mentioned, you can't filter by completely dropping the log records because the AzureBatchLogRecordProcessor always gets ahold of them, but if you mark them as SAMPLED, with your logRecordProcessor, they'll be dropped per the logic here: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/monitor/monitor-opentelemetry/src/logs/batchLogRecordProcessor.ts/#L28

[cosminstefanxp]

For reference, I have solved it that way, temporarily, but it feels like a hack for sure. You also need to take care and make sure you reset the context's trace flag afterwards, as otherwise all subsequent logs will not be logged, plus the span itself. In case it helps anyone:

export class SeverityFilterLogRecordProcessor implements LogRecordProcessor {
  constructor(private readonly minSeverity: SeverityNumber = SeverityNumber.INFO) { }

  onEmit(record: SdkLogRecord) {
    const recordSeverity = record.severityNumber ?? this.minSeverity

    if (record.spanContext) {
      // We can use the trace flags to filter the log, as done by the AzureBatchLogRecordProcessor
      // See AzureBatchLogRecordProcessor in node_modules/@azure/monitor-opentelemetry/../logs/batchLogRecordProcessor.js
      if (recordSeverity < this.minSeverity) {
        record.spanContext.traceFlags = TraceFlags.NONE
      } else {
        // NOTE: Since we are setting the trace flags on the span context, it remains set for the duration of the span, so we need to
        // reset it to sampled. Otherwise, none of the remaining log records in the span will be sampled.
        // We also need the span's trace flags to be sampled, otherwise the span itself will not be sampled. See NameFilterSpanProcessor.
        record.spanContext.traceFlags = TraceFlags.SAMPLED
      }
    }
  }

  forceFlush = () => Promise.resolve()
  shutdown = () => Promise.resolve()
}

but you also need to make sure you reset the trace flags on the span itself, in the onEnd of a SpanProcessor.

As you can see, it all seems like a hack. So a proper solution would be great.

[JacksonWeber]

@cosminstefanxp min severity filtering is being added upstream in OpenTelemetry open-telemetry/opentelemetry-js#5991 that should resolve this and remove the need for this processor.