Source file: allow to skip file header (lines)

[inatale]

Hi!
We exensivly use Vector.dev to gather logs. File source is one the most useful for us. Some files (like /var/log/dirsrv//errors, /var/log/dirsrv//audit, /var/log/dirsrv/**/access) from FreeIPA server have the identical header like this:

         389-Directory/[1.4.3.6](http://1.4.3.6/) B2022.332.1108
         ipatest.xxx.dcs.lan:636 (/etc/dirsrv/slapd-XXX-DCS-LAN)

And it may vary on dev\test\prod as well as different server. It would be great to have option to skip rows for some files.
Now we use the following config for files:

fingerprint.strategy = "checksum"
fingerprint.lines = 5
fingerprint.ignored_header_bytes = 0

It allows us to control rotation (fingerprint.ignored_header_bytes couldn't be used because header could vary on different servers) but header lines got captured by Vector.dev and finally ingested in our backend storage (ClickHouse).

Do we have any other option (execpt some sort of Regex and so on) to skip file header?

Thanks.

Freeipa.toml

#
#  SOURCES
#

[sources.freeipa_additional_logs]
type = "file"
include = [ "/var/log/apache2/*.log", "/var/log/ipa/*.log", "var/log/pki/pki-ca-spawn.*.log", "/var/log/pki/pki-kra-spawn.*.log", "/var/log/dirsrv/*/errors", "/var/log/sssd/*.log" ]
exclude = [ "/var/log/ipa/*backup.log"  ]

[sources.freeipa_backup_log]
type = "file"
include = [ "/var/log/ipa/*backup.log" ]
multiline.start_pattern = "^$"
multiline.condition_pattern = "^$"
multiline.mode = "halt_before"
multiline.timeout_ms = 1000

[sources.freeipa_audit_log]
type = "file"
include = [ "/var/log/dirsrv/*/audit" ]
fingerprint.strategy = "checksum"
fingerprint.lines = 5
fingerprint.ignored_header_bytes = 0
multiline.start_pattern = "^time: (\\d{14})"
multiline.condition_pattern = "^time: (\\d{14})"
multiline.mode = "halt_before"
multiline.timeout_ms = 1000

[sources.freeipa_access_log]
type = "file"
include = [ "/var/log/dirsrv/*/access", "/var/log/dirsrv/*/access.*"]
fingerprint.strategy = "checksum"
fingerprint.lines = 5
fingerprint.ignored_header_bytes = 0

[sources.freeipa_logs]
type = "file"
include = [ "/var/log/auth.log" ]

[jszwedko]

I could have sworn we had a feature request for ignoring lines, in addition to bytes, for the fingerprinter, but I'm not finding it now. Would you want to open one?

[inatale]

Thank you for the response. Yes, I also had found the post about it (skip header lines) once before but now (before posting this) I couldn't find it :((( Yes, I would like to open the feature request. I've opened the feature request - #17884.