No matching template and Unsupported field type.

[zhaohangbo]

My td-agent.log when receiving netflow data with fluent-plugin-netflow

2016-03-10 06:30:11 -0800 [warn]: No matching template for flow id 263
2016-03-10 06:31:12 -0800 [warn]: No matching template for flow id 263
2016-03-10 06:31:12 -0800 [warn]: No matching template for flow id 256
2016-03-10 06:31:12 -0800 [warn]: No matching template for flow id 256
2016-03-10 06:31:35 -0800 [warn]: Unsupported field type=148 length=4
2016-03-10 06:31:35 -0800 [warn]: Unsupported field type=148 length=4
2016-03-10 06:31:35 -0800 [warn]: Unsupported field type=176 length=1
2016-03-10 06:31:35 -0800 [warn]: Unsupported field type=176 length=1

Can you please give me some help ?

What's wrong and how to figure it out ?

Find similar issues in Logstash discussion:

It's because data template includes a field that Logstash doesn't understand.

elastic/logstash#2002
https://groups.google.com/forum/#!topic/logstash-users/rvBIYZwWY-c
https://groups.google.com/forum/#!topic/logstash-users/hicR5rugjiQ

[repeatedly]

Using logstash-plugins/logstash-codec-netflow#12 (comment) approach doesn't work?

[zhaohangbo]

Is that mean I need to configure the Network Device by myself ?

Either define that field type with a suitable data type and field name in Network Device:

225:

• :uint32
• :name_of_field

Or mark it to be skipped in Network Device:

225:

• :skip

[repeatedly]

Use definition parameter with your rules.

<source>
  @type netflow
  #...
  definitions /path/to/your_netflow_def.yml
</source>

[zhaohangbo]

Thank you very much for your quick reply and help.

I've tried many times to add different skip fields in definitions /some/path/to/my_netflow_def.yml
like,

148:
- :skip
176:
- :skip
177:
- :skip
.......
.......
.......

But there are always new "Unsupported field type" coming out in the fluentd log.
So if there is a way that
I can get the netflow data first then I can generate a corresponding netflow_def.yml without having access to Network Devices.

[codeout]

Hi

But there are always new "Unsupported field type" coming out in the fluentd log.

Does it keep showing for a long time? Or just a few seconds after you reboot fluentd?
It's an expected behavior to report "Unsupported field type" until this plugin learns about netflow v9 template. (Network device usually sends template flowset periodically, which is something like data schema)

So if there is a way that
I can get the netflow data first then I can generate a corresponding netflow_def.yml without having access to Network Devices.

I hope tshark -V on your collector is going to help you.

[zhaohangbo]

And still having the 2 warnings:
No matching template for flow id 263
No matching template for flow id 256

[zhaohangbo]

@codeout Thanks,

but what tshark captured is binary,
I am using

 tshark -f "udp port 5140" -i any -V

Am I using it in the wrong way ?

---

And attached is my current log warnings.

[codeout]

You may want to decode by -d option like:

tshark -f "udp port 5140" -i any -V -d "udp.port==5140,cflow"

[zhaohangbo]

Thanks a lot !! The command works.
Several minutes later I can see the content of netflow on my terminal.
Although at the beginning, still can't find templates 256,263, since template may not cached.

But in the td-agent.log ,still No matching template for flow id 256, 263 all the time.

Bellowing is my status:

[codeout]

But in the td-agent.log ,still No matching template for flow id 256, 263 all the time.

So, that probably means some exporters keep sending data flowsets without preceding templates.
If you look into the result of tshark, you can find which devices are doing this.

[zhaohangbo]

I can capture the netflow V9 templates and data flowsets successfully via Tshark.
And Templates flowsets will be sent every 30 mins.

But why it keeps reporting warnings after td-agent runs more than 30 mins.
And I can't I write them out to file via Td-agent ?

Can you please help me ?

My td-agent.conf is as following:

<source>
  type netflow
  tag netflow.event
  bind 0.0.0.0
  port 5140
  cache_ttl 6000 # time_to_live
  versions [5, 9]
  definitions /etc/td-agent/netflow_def.yml
</source>

<match netflow.event.** >
   type copy
   <store>
        type file
        path /var/log/td-agent/netflow-data-test/netflow.data
   </store>
   <store>
        type stdout  # also no out put.
   </store>
</match>

[zhaohangbo]

In the method of "def handle_v9_flowset_template(flowset, record)" (Line 200 )in parser_netflow.rb

I added 2 lines of log.info code( line 212 -- 214) as following.
Then build and reinstall the fluent-plugin-netflow.
But I can not find any output info in td-agent.log .

def handle_v9_flowset_template(flowset, record)
    ......
    ......
    @templates[key, @cache_ttl] = BinData::Struct.new(endian: :big, fields: fields)
    $log.info("cache_ttl is #{@cache_ttl}")
    $log.info("added template,flowset.source_id|template.template_id is #{key}")
    @templates.cleanup!

So I wonder if there's anything wrong with the
@templates[key, @cache_ttl] = BinData::Struct.new(endian: :big, fields: fields)
Can you give me some suggestions ?

[codeout]

I'm guessing that netflow_field_for(field.field_type, field.field_length, category) returns nil for some fields of your template flowset. All field should be parseable by netflow_field_for to register the template.

I'd recommend that you narrow down which field is problematic on your template. If netflow_field_for returns something for every field, then BinData::Struct.new could be wrong as you pointed out.

      def handle_v9_flowset_options_template(flowset, record)
        ...
            NETFLOW_V9_FIELD_CATEGORIES.each do |category|
              template["#{category}_fields"].each do |field|
                entry = netflow_field_for(field.field_type, field.field_length, category)

                throw :field unless entry  # template won't be registered if this line happens

                fields += entry
              end
            end

BTW, I've also noticed we have a few things to fix in this netflow v9 parser and am addressing one of them in #12 (I'm also preparing another fix on my branch)

[repeatedly]

@zhaohangbo Could you try #14 patch for your case?