Vector - Prometheus flattening the field.

[ryansliceup]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hello,
I hope all is well. I am trying to get prometheus data in kafka. This is working well, but I am trying to take the output and flatten the fields. I am using the prometheus_scrape plugin for this, and the data is ending up in kafka.

What I am trying to do is flatten the "tags" object, so that it in a more accessable format for me. Instead of everything being under the tag object, i want to flatten it.

Thank you again!

Configuration

❯ cat vector2.yml
enabled: true
env:
  - name: VECTOR_REQUIRE_HEALTHY
    value: "true"
fullnameOverride: "vector"
service:
  type: "LoadBalancer"
  ports:
    - name: syslognonstandard
      port: 5141
      targetPort: 5141
      protocol: "UDP"
    - name: syslog
      port: 5140
      targetPort: 5140
      protocol: "TCP"
    - name: fluentd
      port: 9000
      targetPort: 9000
      protocol: "TCP"
customConfig:
  data_dir: /vector
  api:
    enabled: true
    address: 127.0.0.1:8686
    playground: false
  sources:
    rsyslog-tcp:
      address: 0.0.0.0:5140
      mode: tcp
      type: syslog
    rsyslog-udp:
      address: 0.0.0.0:5141
      mode: udp
      type: syslog
    k8s-clustersrc:
      type: fluent
      address: 0.0.0.0:9000
    k8s-metricssrc:
      type: prometheus_scrape
      scrape_interval_secs: 60
      endpoints:
        - http://prometheus-operated.monitoring:9090/federate?match[]={job!=%22%22}
  transforms:
    syslog_timestamp_add:
      type: remap
      inputs: [rsyslog-tcp, rsyslog-udp]
      timezone: UTC
      source: |
        .ingestion_timestamp = now()
    prom:
      type: remap
      inputs: [k8s-metricssrc]
      source: |
        .test = (.tags.job)
  sinks:
    client_rsyslog:
      type: kafka
      inputs: [syslog_timestamp_add]
      bootstrap_servers: kafka:9092
      topic: vector-lines-0
      compression: none
      healthcheck: true
      encoding:
        codec: json
    internal_k8s_logs:
      type: kafka
      inputs: [k8s-clustersrc]
      bootstrap_servers: kafka:9092
      topic: internal-k8s-logs
      compression: none
      healthcheck: true
      encoding:
        codec: json
    internal_k8s_metrics:
      type: kafka
      inputs: [prom]
      bootstrap_servers: kafka:9092
      topic: internal-k8s-metrics
      compression: none
      healthcheck: true
      encoding:
        codec: json
livenessProbe:
  exec:
    command:
    - /usr/bin/vector
    - validate
readinessProbe:
  exec:
    command:
    - /usr/bin/vector
    - validate

### Version

vector-0.33.0 helm chart. 0.38.0-distroless-libc. 

### Debug Output

```text
{"name":"apiserver_request_duration_seconds_bucket","tags":{"component":"apiserver","endpoint":"https","group":"acme.cert-manager.io","instance":"192.168.122.88:6443","job":"apiserver","le":"45","namespace":"default","prometheus":"monitoring/mon-kube-prometheus-stack-prometheus","prometheus_replica":"prometheus-mon-kube-prometheus-stack-prometheus-0","resource":"orders","scope":"cluster","service":"kubernetes","verb":"LIST","version":"v1"},"timestamp":"2024-05-15T18:40:11.465Z","kind":"absolute","gauge":{"value":31.0}}

### Example Data

_No response_

### Additional Context

_No response_

### References

_No response_

[jszwedko]

Would https://vector.dev/docs/reference/vrl/functions/#flatten work for you?