Vector as a receiver for td-agent

[gl1ridae]

Hi all! I'm trying to setup integration with td-agent, since vector supported fluent as a source, but as a result I'm having an empty messages.

Versions:

• vector: 0.29.1
• td-agent: 4.3.0

My vector config:

sources:
  src_fluent:
    type: fluent
    address: 0.0.0.0:19020
    keepalive:
      time_secs: 10
sinks:
  snk_fluent:
    type: file
    inputs:
    - src_fluent
    path: "/srv/vector/output/fluent.out"
    encoding:
      codec: raw_message

And here is config for td-agent:

<source>
  @type tail
  tag vectortest
  path /srv/vectortest/vector.log
  <parse>
   @type none
  </parse>
  pos_file /srv/vectortest/vector.log.pos
</source>

<match vectortest>
  @type copy

  <store>
      @type forward
      send_timeout 60s
      compress gzip
      <server>
        name vector
        host <vector host ip address>
        port 19020
      </server>
  </store>

</match>

I've tried it with and without compression from td-agent side, different kind of parsers but no luck at all. I understand that support is in beta status now, but wonder maybe there's something obvious I'm missing. The fact that I'm receiving empty messages means the issue should be on a side of source, while parsing message. I'll really appreciate any help here.

[jszwedko]

Hmm, and I assume there are no error logs from either Vector or td-agent? I would try setting encoding.codec: "json" on the file sink you have. One thought is that the messages are coming through correctly but that the raw_message codec is encoding them as empty messages.

[gl1ridae]

Yes, there're no any error/warning messages in log files, unfortunately.
I've tried to use encoding: "json" with and without parse @type json but same result - only thing that except empty message I'm receiving formatted version {}.
I've also tried to use vector tap connecting to components and it the output from source component is empty messages, that's why I'm thinking there's something with parsing fluent/td-agent protocol. Tcpdump shows that message comes not empty from the td-agent.

[jszwedko]

Ah, gotcha, that does seem to point to an issue with decoding. There are integration tests for this using fluentd and fluentbit, but we don't currently have one for td-agent. Seemingly it should be the same. As an alternative you could consider using the td-agent http output and Vector's http_server source.

[gl1ridae]

I see, will try to do that way then, thank you for your help!