Skip to content

Parser type filter not working with systemd logs #106

@vipulbe

Description

@vipulbe

Hi Team,

I have an inline JSON string in message field of systemd logs that I want to parse but when I add filter @type parser it stops capturing the logs in cloudwatch. It works with other log sources. Also, there are no error logs in td-agent.log

{
"syslog_identifier": "vault-audit",
"message": "{\"field_1\": \"value1\", \"field_2\": \"value_2\"}"
}

I am trying to parse above message in JSON format like below

{
   "syslog_identifier":"vault-audit",
   "message":{
      "field_1":"value1",
      "field_2":"value_2"
   }
}

Config:

<source>
  @type systemd
  tag journal
  path /var/log/journal
  read_from_head true

  <storage>
    @type local
    path /var/log/td-agent/fluentd-journald-cursor.json
  </storage>

  <entry>
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>

<filter vault-audit>
  @type parser
  
  key_name message
  reserve_data true
  hash_value_field app_message
  emit_invalid_record_to_error false
  <parse>
    @type json
  </parse>
</filter>

<match journal>
  @type rewrite_tag_filter

  <rule>
    key syslog_identifier
    pattern /^vault-audit$/
    tag vault-audit
  </rule>
 </match>

<match vault-audit>
  @type cloudwatch_logs
  region {{region}}
  log_group_name /ec2/vault/audit
  log_stream_name {{instance_id}}
  auto_create_stream true
  retention_in_days 7
</match> 

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions