Why are Docker container labels causing "can't merge a non object mapping" errors?

[jerrac]

Hey all,

I did do a bit of searching on this, and I think I know what is going on, but I wanted to confirm.

When I have vector watching my docker container logs using the docker_logs source, my vector logs fill up with a lot of errors like this:

2024-06-17T16:14:15.246467Z ERROR sink{component_kind="sink" component_id=test_alertdrop_es_logs component_type=elasticsearch}:request{request_id=21}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_200" response=Response { status: 200, version: HTTP/1.1, headers: {"x-elastic-product": "Elasticsearch", "content-type": "application/json", "transfer-encoding": "chunked"}, body: b"{\"errors\":true,\"took\":38,\"items\":[{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"8m76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"8276JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"9G76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"9W76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"9m76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"9276JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"-G76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"-W76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"-m76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"-276JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"_G76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"_W76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"_m76JpABEaCUMIymSE1f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"_276JpABEaCUMIymSE1f\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":1391,\"_primary_term\":1,\"status\":201}},{\"index\":{\"_index\":\"vector-logs-2024.06.17\",\"_id\":\"AG76JpABEaCUMIymSE5f\",\"status\":400,\"error\":{\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [label.com.docker.swarm.task] with an object mapping\"}}}]}" }

There was another post a while back about a similar error with Kubernetes logs. The solution, I think, was to drop the labels or transform them into something without dots.

So, is Vector looking at the labels and thinking that each individual label with dots in it is actually an object?

Is there a way to tell Vector not to do that? Docker itself adds labels with dots in them, and I've seen many other systems do the same.

Part of why I'm asking is that when I run docker inspect on a container, the json output clearly shows the labels as keys, not objects. So I kinda think it's a little odd that Vector would try to turn them into objects. And if it isn't doing that, then I need to figure out what is actually causing those errors...

Thanks!

[jszwedko]

Right, you might need to "flatten" and de-dot the labels before inserting into Elasticsearch (note the error is coming from Elasticsearch, not Vector).

[jszwedko]

Example of dedotting here: https://vector.dev/docs/reference/vrl/functions/#map_keys

[jerrac]

Thanks. The following seems to work.

transforms:
  dedot_docker_labels:
    type: remap
    inputs:
    - docker_logs
    source:  >-
       . = map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }        
      #      if exists(.log.label) && (is_object(.log.label) || is_array(.log.label)) {
      #            .log.label = map_keys(.log.label, recursive: true) -> |key| { replace(key, ".", "_") }
      #          }

But why didn't the commented out code work?

error[E121]: type mismatch in closure parameter
  ┌─ :3:26
  │
3 │             .log.label = map_keys(.log.label, recursive: true) -> |key| { replace(key, ".", "_") }
  │                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  │                          │
  │                          the closure tied to this function call expects a different input value
  │                          expression has an inferred type of any where an array or object was expected
  │
  = see language documentation at https://vrl.dev
  = try your code in the VRL REPL, learn more at https://vrl.dev/examples

Is VRL or rust just not able to know that the "is*" methods mean that .log.label will have the correct type? I think I ran into that last time I tried Vector...