log_to_metric transform errors

[mattzech]

Hi,

We are using a log_to_metric transform on kubernetes logs as shown below

# Simplified for readability
kubernetes_logs_parsed:
  type: remap
  inputs:
  - kubernetes_logs_multiline_reduce
  source: |-
    # Sclayr Options
    if .kubernetes.pod.labels."scalyr.logging.dev/forwarding" == "enabled" || .kubernetes.namespace.labels."scalyr.logging.dev/forwarding" == "enabled" {
      # attempt to get token for this log event.
      # prioritise pod labels over namespace labels
      .scalyr_options.token = string(.kubernetes.pod.labels."scalyr.logging.dev/token") ?? string(.kubernetes.namespace.labels."scalyr.logging.dev/token") ?? ""
      if .scalyr_options.token != "" {
        .scalyr_options.token, err = decode_base64(.scalyr_options.token)
        if err == null {
          .scalyr_options.forwarding = true
        } else {
          log({
            "msg":       "Could not decode sclayr token",
            "namespace": .kubernetes.namespace.name,
            "pod":       .kubernetes.pod.name,
          }, level: "warn", rate_limit_secs: 60)
        }
      } else {
        log({
          "msg":       "Scalyr API key missing in labels",
          "namespace": .kubernetes.namespace.name,
          "pod":       .kubernetes.pod.name,
        }, level: "warn", rate_limit_secs: 60)
      }
    }

volume_metrics_scalyr:
  type: log_to_metric
  inputs:
  - kubernetes_logs_parsed
  metrics:
   - type: "counter"
      field: scalyr_options.forwarding
      name: "route_scalyr"
      namespace: "vector"
      tags:
        namespace: "{{kubernetes.namespace.name}}"

The vector logs are being flooded with the below error:

2023-03-14T21:30:22.317925Z ERROR transform{component_kind="transform" component_id=volume_metrics_scalyr component_type=log_to_metric component_name=volume_metrics_scalyr}: vector::internal_events::parser: Internal log [Field does not exist.] is being rate limited.
2023-03-14T21:30:22.317971Z ERROR transform{component_kind="transform" component_id=volume_metrics_scalyr component_type=log_to_metric component_name=volume_metrics_scalyr}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being rate limited.

Vector version: 0.27.0

We cannot seem to figure out what is causing this. Any guidance would be much appreciated. Thanks