Q&A: Redact by json keys

[domenico-luciani-deel]

Question

Is there a simple way to redact a JSON like this:

{
  "level": "INFO",
  "time": 1762964214879,
  "dd": {
    "trace_id": "xxx6168688b8a3f63ba",
    "span_id": "xxx487694978",
    "service": "blabla",
    "version": "e411a23de49c90b7f480d5aa4dfb1c968b9ca334",
    "env": "prod"
  },
  "headers": {
    "Authorization": "Bearer sensitive-jwt-token-here",
    "token": "internal-secret-shhhh",
    "Content-Type": "application/json"
  },
  "request_path": "/api/v1/login",
  "message": "Test 3: Sensitive headers"
}

into this:

{
  "level": "INFO",
  "time": 1762964214879,
  "dd": {
    "trace_id": "xxx6168688b8a3f63ba",
    "span_id": "xxx487694978",
    "service": "blabla",
    "version": "e411a23de49c90b7f480d5aa4dfb1c968b9ca334",
    "env": "prod"
  },
  "headers": {
    "Authorization": "[REDACTED]",
    "token": "[REDACTED]",
    "Content-Type": "application/json"
  },
  "request_path": "/api/v1/login",
  "message": "Test 3: Sensitive headers"
}

?

I tried already the redact function but it works only on values and not on keys, so if I have a json like:

{ "token": "blabla" }

and I use as a filter r'token' it won't redact anything leaving the value (blabla) untouched

With the attached transforms I achieve it, but I was wondering if it was the most idiomatic way to do it with Vector.

Thank you

Vector Config

transforms:
  parser:
    type: remap
    inputs: [ kubernetes_logs ]
    source: |
      parsed, err = parse_regex(.message, r'(?P<json_str>\{.*\})$')
      if err == null {
        json_map, jerr = parse_json(parsed.json_str)
        if jerr == null && is_object(json_map) {
          del(.message)
          . |= object!(json_map)
        }
      }
  redactor:
    type: remap
    inputs: [ parser ]
    source: |
      if is_object(.) {
        raw = encode_json(.)

        raw = replace(
          raw,
          r'("(?i:password|pass|pwd|token|authorization|authToken)"\s*:\s*)(?:(?:"(?:[^"\\]|\\.)*")|(?:\d+)|(?:\{.*?\})|(?:\[.*?\])|(?:true|false|null))',
          "$$1\"[REDACTED]\""
        )

        obj, perr = parse_json(raw)
        if perr == null {
          . = obj
        } else {
          .redaction_error = "Failed to re-parse redacted JSON: " + perr
          .message = raw
        }
      }

Vector Logs

No response

[jayy-77]

Hey, there is already a function called redact, I think this is an easier way to do it.

[domenico-luciani-deel]

The issue is that the redact function works only on values, not keys:

redact({
  "level": "INFO",
  "time": 1762964214879,
  "dd": {
    "trace_id": "xxx6168688b8a3f63ba",
    "span_id": "xxx487694978",
    "service": "blabla",
    "version": "e411a23de49c90b7f480d5aa4dfb1c968b9ca334",
    "env": "prod"
  },
  "headers": {
    "Authorization": "Bearer sensitive-jwt-token-here",
    "token": "internal-secret-shhhh",
    "Content-Type": "application/json"
  },
  "request_path": "/api/v1/login",
  "message": "Test 3: Sensitive headers"
}, filters: [r'token'])
{ "dd": { "env": "prod", "service": "blabla", "span_id": "xxx487694978", "trace_id": "xxx6168688b8a3f63ba", "version": "e411a23de49c90b7f480d5aa4dfb1c968b9ca334" }, "headers": { "Authorization": "Bearer sensitive-jwt-[REDACTED]-here", "Content-Type": "application/json", "token": "internal-secret-shhhh" }, "level": "INFO", "message": "Test 3: Sensitive headers", "request_path": "/api/v1/login", "time": 1762964214879 }

I'd like to use filters on keys so when it spots "token" it redact the whole value associated with that key