Not able to retrieve Sysmon logs #75
Description
I have multiple custom channels where Security and Sysmon logs are forwarded.
<source>
@type windows_eventlog2
@id wec_channels
channels ["WEC-Security", "WEC-Sysmon"]
read_interval 2
rate_limit 5000
tag wec_channels
render_as_xml true
parse_description false
<storage>
@type local
persistent true
path C:\opt\td-agent\var\log\wec_channels
</storage>
<parse>
@type none
</parse>
</source>
<match wec_channels>
@type kinesis_streams
...
</match>
At first I was only receiving Security logs and the systems worked like a charm.
As soon as I add sysmon logs in the configuration, I see this behaviour:
The ruby process flushes correctly the first pass of the first security logs, and after that, the memory keeps on increasing without releasing the logs.
Sometimes, when I stop the service, it flushes the Sysmon logs and at other times it does not.
I can correctly query that sysmon channel through winevt and receive XML logs.
The message part is empty and the xml one contains line returns (in do |xml, message, string_inserts|).
I tried to mimic the plugin by editing the log handling with the purest form and removing the parser helper (I only need XML output).
def on_notify_xml(ch, subscribe)
...
subscribe.each do |xml, message, string_inserts|
xml = xml.gsub(/[ \x00\t\n\v\f\r]+/, " ")
log.debug "XML LOG (#{xml})"
es.add(Fluent::Engine.now, { "message" => "#{xml}" })
end
...
The sysmon logs are correctly written as debug messages to the td-agent.log file but they don't go anywhere after that.
I tried to use different workers, different sources, different output plugins, it just doesn't get to the match part, keeps stuck in the source part, keeping the logs in memory. If I remove the sysmon logs channel and add other ones, it all works fine.
The WEC servers are beasts and the quantity of security logs is higher than the sysmon ones, there is no system or network bottleneck.
Tried with td-agent version 4.1.1 and also 4.1.0 because I saw some memory leak issues.
Any suggestion?