Inconsistent sink behavior blocking production usage - pragmatic fixes vs global refactoring?

[jmealo]

Question

I'm encountering two bugs with inconsistent behavior between Vector sinks that are preventing production deployment without forking:

1. Loki sink null handling (Loki sink: Inconsistent handling of null values between static and dynamic structured_metadata fields #23426): Static fields with null values become literal "" strings in Loki, while dynamic fields correctly omit nulls. This breaks queries since | field != "" doesn't match when field contains "".
2. except_fields ignored by Loki (except_fields not applied consistently between sinks #23725): The except_fields configuration works correctly in the elasticsearch sink but is ignored by the Loki sink - excluded fields still appear in output.

I submitted PR #23427 to fix the null handling, but it was suggested we need a global fix across all sinks instead.

The core issue: Sinks already behave inconsistently. Why block targeted fixes for user-impacting bugs while waiting for a comprehensive refactor that may not materialize soon? This forces production users to maintain forks just to work around known issues.

Questions for the community:

• How do other production users handle these inconsistencies?
• Would the maintainers consider accepting sink-specific fixes now while working toward unified behavior later?
• Is there a roadmap for addressing sink consistency globally?
• Do we plan on adding testing to make sure documentation is accurate and sinks behave consistently?

Vector Config

sinks:
  loki:
    type: loki
    endpoint: "http://loki-gateway.monitoring.svc.cluster.local"
    labels:
      "*": '{{ loki_labels }}'
    structured_metadata:
      "*": '{{ structured_metadata }}'
      request_id: '{{ request_id }}'  # Becomes "<null>" if missing
    encoding:
      codec: json
      except_fields: ["structured_metadata", "loki_labels"]  # Ignored
      
  elasticsearch:
    type: elasticsearch
    endpoints: ["http://victorialogs.monitoring.svc.cluster.local:9428/"]
    encoding:
      except_fields: ["structured_metadata", "loki_labels"]  # Works correctly

Vector Logs

No response

[pront]

Hello @jmealo, I think my answer here #23040 (comment) addresses most of the questions raised in this discussion. Basically, the proper solution here is both the correct approach and also easier/cleaner to implement. Since we have many open discussions/issues on this already, I suggest continuing the discussion there to avoid fragmentation.