Fluentd is not able to send logs to Logstash

[prashant407]

We are using fluentd 1.14.2 version and trying to send data to logstash 7.16.2. There is no error in the logs but we are not able to see data in the logstash. When we trying the nc command to send the data we are able to see that in logstash.
Please refer the configuration

td-agent.conf

 <source>
    @type tail
    path "/var/log/test.log"
    tag "*"
    read_from_head true
    from_encoding "UTF-8"
    encoding "UTF-8"
    <parse>
      @type "none"
      #unmatched_lines
    </parse>
  </source>
  <match **>
    @type stdout
  </match>
  <match **>
    @type forward
    time_as_integer true
    <server>
      host "xx.xx.xx.xx"
      port 28284
    </server>
  </match>

logstash.conf

input {
    tcp
    {
      port => 28284
    }
 }
 filter
 {
 }
 output
 {
 file {
   path =>  "/usr/share/logstash/td-logs/fluentdlogs-dec-2022.log"
   }
 }

Could you please help us in resolving the issue?

[fujimotos]

The probability is that you need to teach logstash to use fluent codec as follows:

input {
  tcp {
    codec => fluent
    port => 24224
  }
}

I don't see anything wrong with Fluentd's side. Please ask logstash project for more support.

[prashant407]

Thanks @fujimotos for the quick response.
We have updated the codec as fluent in logstash. Even we are not getting any data from fluentd. But when we send same file through nc command we have seen the below errors in logstash

[2022-12-06T11:39:25,753][ERROR][logstash.codecs.fluent   ][pMainFluentD][9e78af05db221c278bf9b074b01a9b48d0fe4dcc75edd57d53228cb19825b0d5] Fluent parse error, original data now in message field {:error=>#<TypeError: wrong argument type Integer (expected Hash)>, :data=>32}
[2022-12-06T11:39:25,755][ERROR][logstash.codecs.fluent   ][pMainFluentD][9e78af05db221c278bf9b074b01a9b48d0fe4dcc75edd57d53228cb19825b0d5] Fluent parse error, original data now in message field {:error=>#<TypeError: wrong argument type Integer (expected Hash)>, :data=>97}
[2022-12-06T11:39:25,757][ERROR][logstash.codecs.fluent   ][pMainFluentD][9e78af05db221c278bf9b074b01a9b48d0fe4dcc75edd57d53228cb19825b0d5] Fluent parse error, original data now in message field {:error=>#<TypeError: wrong argument type Integer (expected Hash)>, :data=>103}
[2022-12-06T11:39:25,759][ERROR][logstash.codecs.fluent   ][pMainFluentD][9e78af05db221c278bf9b074b01a9b48d0fe4dcc75edd57d53228cb19825b0d5] Fluent parse error, original data now in message field {:error=>#<TypeError: wrong argument type Integer (expected Hash)>, :data=>97}
[2022-12-06T11:39:25,761][ERROR][logstash.codecs.fluent   ][pMainFluentD][9e78af05db221c278bf9b074b01a9b48d0fe4dcc75edd57d53228cb19825b0d5] Fluent parse error, original data now in message field {:error=>#<TypeError: wrong argument type Integer (expected Hash)>, :data=>105}

Also there is no error in the fluentd logs. Please refer the logs

2022-12-06 12:09:25.456770917 +0000 fluent.info: {"message":"following tail of /var/log/containers/json.log"}
2022-12-06 12:09:25.456922578 +0000 fluent.info: {"message":"following tail of /var/log/containers/test.log"}
2022-12-06 12:09:25.457101291 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}
2022-12-06 12:09:56 +0000 [info]: #0 detected rotation of /var/log/containers/json.log
2022-12-06 12:09:56 +0000 [info]: #0 following tail of /var/log/containers/json.log
2022-12-06 12:09:56.800540745 +0000 fluent.info: {"message":"detected rotation of /var/log/containers/json.log"}
2022-12-06 12:09:56.800638785 +0000 fluent.info: {"message":"following tail of /var/log/containers/json.log"}
2022-12-06 12:09:56.800999975 +0000 var.log.containers.json.log: {"message":"a"}
2022-12-06 12:09:56.801005677 +0000 var.log.containers.json.log: {"message":"testing 1234556"}
2022-12-06 12:09:56.801007896 +0000 var.log.containers.json.log: {"message":"abcefg"}

any idea on the above error ?

[prashant407]

Thanks for the help @fujimotos
We are able to resolve the issue. We were using below match section due to which data was not received by the logstash match.

  <match **>
    @type stdout
  </match>

After removing the above match. We are able to get the data.

Thanks,
Prashant