Parsing auditd logs has issues due to a 0x1D character

[alex-rowe]

When using auditd set to log_format = ENRICHED there is a "group separator" character between the existing fields and unenriched fields.

This causes Vector to not split on this field when using parse_key_value.

When attempting to use strip_ansi_escape_codes, this only removes the value, but we need to split on it.

When attempting to use replace, we get errors for unexpected error: invalid escape character: \u

An example log line would be:

type=SYSCALL msg=audit(1695946364.453:9863): arch=c000003e syscall=257 success=yes exit=8 a0=ffffff9c a1=7f8f9b31cd80 a2=80002 a3=0 items=1 ppid=8221 pid=9287 auid=4294967295 uid=0 gid=1002 euid=0 suid=0 fsuid=0 egid=1002 sgid=1002 fsgid=1002 tty=pts0 ses=4294967295 comm="sudo" exe="/usr/bin/sudo" key="logins"ARCH=x86_64 SYSCALL=openat AUID="unset" UID="root" GID="ssm-user" EUID="root" SUID="root" FSUID="root" EGID="ssm-user" SGID="ssm-user" FSGID="ssm-user"

And key="logins"ARCH=x86_64 are shown as joined when viewing using cat or tail etc.

When using the transform .message = parse_key_value!(string!(.message)) we get the following:

"key": "(null)\u001dARCH=x86_64",

Is there a recommended way we can replace \u001d with a whitespace before we send it to parse_key_value so that it properly gets picked up a two fields, not one?

A separate question may be the possibility of a parse_auditd function.

Extra information here:

• Missing space between "key" and "ARCH" when log_format = ENRICHED linux-audit/audit-userspace#140
• https://github.com/linux-audit/audit-documentation/wiki/SPEC-Audit-Event-Enrichment

[alex-rowe]

Testing in VRL Playground, I can't get any of the replacements to work.

I thought trying to replace u001d then replacing the / afterwards may work around the unescaped errors, but that's not working either.

https://playground.vrl.dev/?state=eyJwcm9ncmFtIjoiIyBSZW1vdmUgc29tZSBmaWVsZHNcbi5tZXNzYWdlID0gc3RyaW5nISgubWVzc2FnZSlcbi5tZXNzYWdlID0gcmVwbGFjZSgubWVzc2FnZSwgcid1MDAxZCcsIFwiIFwiKVxuLm1lc3NhZ2UgPSByZXBsYWNlKC5tZXNzYWdlLCBcInUwMDFkXCIsIFwiIFwiKVxuLm1lc3NhZ2UgPSByZXBsYWNlKC5tZXNzYWdlLCBcIjAwMWRcIiwgXCIgXCIpXG4ubWVzc2FnZSA9IHJlcGxhY2UoLm1lc3NhZ2UsIFwiKG51bGwpXCIsIFwiKHJlcGxhY2VkKVwiKVxuLm1lc3NhZ2UgPSByZXBsYWNlKC5tZXNzYWdlLCBcIng4Nl82NFwiLCBcInJlcGxhY2VkXCIpIiwiZXZlbnQiOnsibWVzc2FnZSI6IihudWxsKVx1MDAxZEFSQ0g9eDg2XzY0In0sImlzX2pzb25sIjpmYWxzZSwiZXJyb3IiOm51bGx9

[artemNN]

You need to replace unicode:
.message = replace(.message, r'\u001d', " ")
.message = parse_key_value!(string!(.message))

[alex-rowe]

Thanks @artemNN that works, unsure how I missed that in my testing.