Layout for Elastic Common Schema

[hstenzel]

I'm trying to get kubernetes-event-exporter to output events in a way that is compatible with Elastic Common Schema (ECS). To do this, I'm using layout to remap k8s event fields onto ECS.

I have a straightforward recreate scenario using the bitnami helm chart. Note that this just makes iteration faster, the problem exists in the event exporter itself.

This demonstrates a few problems:

• Everything is a string, cannot remap structures. See labels below.
• Everything is a string, cannot output numbers. See count below
• Using .FirstTimestamp actually gets the value of .LastTimestamp
• No way to ignore fields that are not present (like .host.hostname)

I'm wondering if it would be a better approach to take a string layoutYAML and let the document be rendered. Alternatively having an ECS specific output mode (others could be added as well).

Recreate info

Helm values:

replicaCount: 1

config:
  logFormat: json
  logLevel: error
  throttlePeriod: 5
  route:
    routes:
    - match:
      - receiver: "ecs"
    - match:
      - receiver: "dump"
  receivers:
  - name: "dump"
    stdout: {}
  - name: "ecs"
    stdout:
      layout:
        # per https://www.elastic.co/guide/en/ecs/current/ecs-base.html
        # @timestamp: '{{ .Metadata.CreationTimestamp | date "2006-01-02T15:04:05Z" }}' # BUG? Can't access Metadata
        # @timestamp: '{{ now | date "2006-01-02T15:04:05Z" }}'
        labels: '{{ toJson .InvolvedObject.Labels }}' # BUG? How to make this a dict?
        message: '{{ .Message }}'
        # tags: []
        # per https://www.elastic.co/guide/en/ecs/current/ecs-agent.html
        agent:
          type: kubernetes-event-exporter
        # REQUIRED per https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html
        ecs:
          version: 8.1.0 # The version of ECS governing this mapping
        # https://www.elastic.co/guide/en/ecs/current/ecs-event.html
        event:
          action: '{{ .Reason | toString }}'
          created: '{{ .FirstTimestamp | date "2006-01-02T15:04:05Z" }}' # BUG: gets value from LastTimestamp
          end: '{{ .LastTimestamp | date "2006-01-02T15:04:05Z" }}'
          kind: "event"
          provider: '{{ .Source.Component | toString }}'
          reason: '{{ .Reason | toString }}'
          severity: '{{ eq .Type "Normal" | ternary 6 4 | toString }}' # BUG: should be syslog severity number, not string
          # not ecs defined.
          count: '{{ .Count }}' # BUG -- should be a number
          type: '{{ .Type }}' # Mapped into .event.severity
          reportinginstance: '{{ .ReportingInstance | toString }}'
        # https://www.elastic.co/guide/en/ecs/current/ecs-host.html
        host:
          hostname: '{{ .Source.Host | toString }}' # How to omit when empty?
        # https://www.elastic.co/guide/en/ecs/current/ecs-orchestrator.html
        orchestrator:
          # apiVersion: '{{ .InvolvedObject.ApiVersion | toString }}'
          namespace: '{{ .InvolvedObject.Namespace | toString }}'
          resource:
            name: '{{ .InvolvedObject.Name | toString }}'
            type: '{{ .InvolvedObject.Kind | toString }}'
            fieldpath: '{{ .InvolvedObject.FieldPath | toString }}'

helm  upgrade --install --wait kee bitnami/kubernetes-event-exporter -f kee.yaml

Let it run for a while. One of the issues can only be verified when firstTimestamp != lastTimestamp, so we need count > 1.

Check the results:

kubectl logs deploy/kee-kubernetes-event-exporter | grep "^{.*}$" | jq -s '.[] | select( (.event.count and ( .event.count | tonumber) > 1) or (.count > 1) )'

The resulting query has the dump event first and the ecs event second:

{
  "metadata": {
    "name": "ingress-nginx-controller.16e4d784d481a5b8",
    "namespace": "ingress-system",
    "selfLink": "/api/v1/namespaces/ingress-system/events/ingress-nginx-controller.16e4d784d481a5b8",
    "uid": "7ad42072-055c-45d3-b8c0-5cfb31fce5da",
    "resourceVersion": "46920191",
    "creationTimestamp": "2022-04-11T12:32:09Z"
  },
  "reason": "nodeAssigned",
  "message": "announcing from node \"docker-desktop\"",
  "source": {
    "component": "metallb-speaker"
  },
  "firstTimestamp": "2022-04-11T12:32:09Z",
  "lastTimestamp": "2022-04-28T19:26:57Z",
  "count": 4191,
  "type": "Normal",
  "eventTime": null,
  "reportingComponent": "",
  "reportingInstance": "",
  "involvedObject": {
    "kind": "Service",
    "namespace": "ingress-system",
    "name": "ingress-nginx-controller",
    "uid": "2c9ef059-b610-4a09-a931-23a3cef0c835",
    "apiVersion": "v1",
    "resourceVersion": "43457792",
    "labels": {
      "app.kubernetes.io/component": "controller",
      "app.kubernetes.io/instance": "ingress-nginx",
      "app.kubernetes.io/managed-by": "Helm",
      "app.kubernetes.io/name": "ingress-nginx",
      "app.kubernetes.io/version": "0.47.0",
      "helm.sh/chart": "ingress-nginx-3.34.0"
    },
    "annotations": {
      "meta.helm.sh/release-name": "ingress-nginx",
      "meta.helm.sh/release-namespace": "ingress-system"
    }
  }
}
{
  "agent": {
    "type": "kubernetes-event-exporter"
  },
  "ecs": {
    "version": "8.1.0"
  },
  "event": {
    "action": "nodeAssigned",
    "count": "4191",
    "created": "2022-04-28T19:26:57Z",
    "end": "2022-04-28T19:26:57Z",
    "kind": "event",
    "provider": "metallb-speaker",
    "reason": "nodeAssigned",
    "reportinginstance": "",
    "severity": "6",
    "type": "Normal"
  },
  "host": {
    "hostname": ""
  },
  "labels": "{\"app.kubernetes.io/component\":\"controller\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"0.47.0\",\"helm.sh/chart\":\"ingress-nginx-3.34.0\"}",
  "message": "announcing from node \"docker-desktop\"",
  "orchestrator": {
    "namespace": "ingress-system",
    "resource": {
      "fieldpath": "",
      "name": "ingress-nginx-controller",
      "type": "Service"
    }
  }
}