[Q&A] How to enrich certain logs that match pattern and retain all the original entries as well?

[Tomas-Big-Z]

What is a problem?

I need to add a custom field to all log entries that contain string "myapp.database - Statement returned". All other logs that do not contain this string, should not be discarded/filtered out.

I managed to add the custom field by applying grep, parser and record_transformer filters, but at the cost of all other logs being filtered out or the log entries containing string "myapp.database - Statement returned" being duplicated. Neither of these results is acceptable - as i said, the log entries containing the string should get the new field and the remaining log entries should not be filtered out.

Eventually i tried to apply label routing but it resulted in log entries being filtered out and custom field not being added at all. Below is my fluentd config where i tried to apply labels.

Would be thankful for any useful tips.

EDIT: i dont know how to make the entire config visible, this formatting is not quite intuitive for the first-time user, sorry

Describe the configuration of Fluentd

@type tail path C:\dmskrnl.log pos_file C:\dmskrnl.log.pos tag dmskrnl.raw @type none @type grep key message pattern /myapp.database - Statement returned/ @Label @enrich

<label @enrich>

@type parser
key_name message
reserve_data true
tag dmskrnl.enriched

@type regexp
expression /myapp.database - Statement returned (?<returned_bytes>\d+) bytes/

@type record_transformer returned_bytes ${record["returned_bytes"]} @type stdout @type stdout

Describe the logs of Fluentd

No response

Environment

- Fluentd version: 1.16.3
- TD Agent version: N/A
- Fluent Package version: 5.0.2
- Docker image (tag): N/A
- Operating system: Windows Server 2022
- Kernel version: N/A