Failed to render template for "index"

[gorbunov-di]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hello. I periodically get an error:

{
  "_index": "vector-dev-2023-08-15",
  "_id": "p8of-4kB8-sMq-fMul89",
  "_version": 1,
  "_score": null,
  "_source": {
    "error": "Missing fields on event: [\".service_name\"]",
    "error_type": "template_failed",
    "host": "vector-j89tn",
    "internal_log_rate_limit": true,
    "message": "Failed to render template for \"index\".",
    "metadata": {
      "kind": "event",
      "level": "ERROR",
      "module_path": "vector::internal_events::template",
      "target": "vector::internal_events::template"
    },
    "pid": 1,
    "source_type": "internal_logs",
    "stage": "processing",
    "timestamp": "2023-08-15T21:35:06.829736905Z",
    "vector": {
      "component_id": "opensearch",
      "component_kind": "sink",
      "component_name": "opensearch",
      "component_type": "elasticsearch"
    }
  },
  "fields": {
    "timestamp": [
      "2023-08-15T21:35:06.829Z"
    ]
  },
  "sort": [
    1692135306829
  ]
}

But I can't catch an event where this field is not present. I catch through type: "filter".

Configuration

customConfig:
    data_dir: /var/lib/vector
    api:
    enabled: true
    address: 127.0.0.1:8686
    playground: false
    sources:
    k8s:
        type: kubernetes_logs
        exclude_paths_glob_patterns: ["*kube-system*"]
        extra_namespace_label_selector: vector.dev/logging=true
    transforms:
    k8s-remap:
        type: remap
        inputs: ["k8s"]
        source: |-
        kubernetes = {"service_namespace":(.kubernetes.pod_namespace), "service_name":(.kubernetes.container_name), "pod_name": (.kubernetes.pod_name), "@timestamp": (parse_timestamp!(.timestamp, format: "MMM D, YYYY @ HH:mm:ss.SSS"))}
        ., err = parse_json(.message)
        ., err = merge(., kubernetes)
        drop_on_error: true
        reroute_dropped: true
    k8s-reforge:
        type: remap
        inputs: ["*dropped"]
        source: |-
        ., err  = {"message": (to_string(.message)), "dropped": true, "service_namespace":(.kubernetes.pod_namespace), "service_name":(.kubernetes.container_name), "pod_name": (.kubernetes.pod_name), "metadata": (.metadata), "@timestamp": (parse_timestamp!(.timestamp, format: "MMM D, YYYY @ HH:mm:ss.SSS"))}
    exists-service-name:
      type: "filter"
      inputs: ["k8s-remap","k8s-reforge"]
      condition: 
        type: "vrl"
        source: |-
          !exists(.container_name)
    sinks:
    opensearch-debug:
        type: elasticsearch
        inputs: ["exists-service-name"]
        api_version: v8
        endpoints: ["elasticsearch"]
        bulk:
        index: "vector-debug-%Y-%m-%d"
    opensearch:
        type: elasticsearch
        inputs: ["k8s-remap","k8s-reforge"]
        api_version: v8
        endpoints: ["elasticsearch"]
        bulk:
        index: "dev-{{ .service_name }}-%Y-%m-%d" 

### Version

0.31.0

### Debug Output

_No response_

### Example Data

_No response_

### Additional Context

_No response_

### References

_No response_

[gorbunov-di]

#17487
Same problem?

[jszwedko]

Hi @gorbunov-di ,

The issue is as the error message indicates: some events are coming through without a service_name. Some thoughts:

You are ignoring the err value in your remap transforms. My guess is that there is an error in one of those transforms and so the event is not receiving the service_name field. For the k8s-remap transform I'd recommend using the ! operator to drop (and reroute) the event when an error occurs. For example: ., err = parse_json!(.message). Then, if the message fails to parse as JSON, it'll be dropped and, in your configuration, rerouted to k8s-reforge. k8s-reforge is also ignoring a possible error.

[gorbunov-di]

Got it, but I have no idea what it is.

{
  "_index": "vector-debug-2023-08-15",
  "_id": "h0BP-4kBLguWsV8GW5bq",
  "_version": 1,
  "_score": 0,
  "_source": {}
}

[jszwedko]

I'd still guess that one of your remap transforms above is erroring. You could do the same routing of events missing that field for debugging purposes but have Vector write them to a file rather than also to ES. That might make it easier to understand what about them is different.

[gorbunov-di]

@jszwedko thx you!

kubernetes = {"service_namespace":(.kubernetes.pod_namespace), "service_name":(.kubernetes.container_name), "pod_name": (.kubernetes.pod_name), "@timestamp": (parse_timestamp!(.timestamp, format: "MMM D, YYYY @ HH:mm:ss.SSS"))}
message, err = is_json(.message)
., err = parse_json(.message) ?? {"message": (to_string(.message)), "dropped": true}
if err != null {
    log("Failed parse json: " + err, level: "error")
} else {
    . = merge!(., kubernetes)
}