VNet Flowlog with Plugin

[aekanunphanphomma]

We are trying to use your plugin to visualize VNet Flowlog on Kibana via Logstash. However, we are facing a lack of input from the storage account. Not sure if this is due to the plugin still being developed or if we are having issues due to an incorrect Logstash configuration. Please assist. Thank you.

logstash.conf.json

[janmg]

This plugin does not directly have support for vnetflowlogs, you would need to parse them with the logtype raw. Because of that you need to split and filter with the correct timestamps. I don't know which parameters are supposed to be used for vnetflowlogs, but from the picture you posted, I think you are picking the date from when the logfiles are written and not the timestamp from each individual event.

I don't think the plugin drops events, unless there is some errors in the logfiles

[janmg]

please change your passwords.

when I have time, I'll add explicit support for vnetflowlogs. I already added a commit, but didn't test it yet.

[yuclin]

Hi janmg,

Have you tried vnetflowlogs already?
I replaced the azure_blob_storage.rb file with you provide at
https://github.com/janmg/logstash-input-azure_blob_storage/blob/master/lib/logstash/inputs/azure_blob_storage.rb

But it shows the error
message=>"Unable to configure plugins: (SyntaxError) /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-azure_blob_storage-0.12.10/lib/logstash/inputs/azure_blob_storage.rb:643: Invalid return in class/module body\n return count\n ^~~~~"

[janmg]

I haven't tested the vnetflowlogs, I wrote the vnetflowlog as a demo how the function would look like, but I don't use the vnetflowlogs myself.

For scalability I am very slowly working to replace this logstash plugin with a standalone application blob-to-queue that can read files from a blobstorage, process the files and send the events straight to any backend including elasticsearch.
https://github.com/janmg/blob-to-queue/blob/main/input/vnetflowlogs.go

I don't have much time to work on it though. Some bits work already, but I haven't written the logic to keep reading new files from the storage. If I get that to work, the blob-to-queue will be a worthy replacement.