Performance issues (~350 events/s for v9, 700-1000events/s for v5 traffic)

[gsteffek]

Hey,

running fluentd with the plugin in a virtual ubuntu 20.04 with the td-agent service. As the title says, I'm nowhere near the 30k events/s measured in the performance record (with an AMD Ryzen 5 3600 as the host).

Any idea what might cause the issue?

As a source for the Netflow traffic I'm using the SolarWinds Netflow Generator Tool.

Versions:

fluentd-1.14.6
ruby="2.7.6"

My td-agent.conf:

<source>
  @type  netflow
  tag netflow.event
  bind 0.0.0.0
  port 9995
#  switched_times_from_uptime yes
</source>
<match netflow.event>
  @type flowcounter
  unit second
  count_keys count # missing column for counting events only
  tag flowcount
</match>
<match flowcount>
  @type file
  path /var/log/td-agent/flowcount
</match>

Thanks in advance!

[berahtlv]

Faced a similar issue - the plugin is able to handle only ~1000 fps of V9 flow data per CPU, however, NetFlow-Generator generated V5 flow data able to get up to 35000 fps.

Behaviour is similar on fluentd 1.15.3 and td-agent 4.4.2.