Syslog receivers expect missing version value in log entry

[marshauf]

Observed behavior

Hello,

I tried sending syslog logs to OpenTelemetry Collector and Telegraf. Both can't parse the log entries from nats-server.
They expect a missing version value in the log entry. I had to figure out which version of syslog nats-server is using.
It would be nice if it is mentioned in the documents. I created a Pull Request for it: nats-io/nats.docs#632

Telegraf with Syslog input configured like:

[[inputs.syslog]]
  server = "tcp://:6514"
  framing = "non-transparent"
  syslog_standard = "RFC5424"

Nats server command: nats-server -js -r tcp://127.0.0.1:6514

Telegraf prints for each received message:

2023-11-14T08:44:15Z E! [inputs.syslog] Error in plugin: expecting a version value in the range 1-999 [col 6]

The same happens with OpenTelemetry Collector configured like:

receivers:
  syslog:
    tcp:
      listen_address: "127.0.0.1:6514"
    protocol: rfc5424
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: ["syslog"]
      exporters: ["debug"]

Message received over TCP:

<5>2023-11-14T10:00:47+01:00 hostname nats-server[34500]: Starting nats-server

After <5> there is supposed to be a 1 for the version, see https://www.rfc-editor.org/rfc/rfc5424#section-6.2.2

Expected behavior

Nats-server sends syslog messages to Telegraf or OpenTelemetry Collector and both can interpret them according to the specification.

Send message should look like this:

<5>1 2023-11-14T10:00:47+01:00 hostname nats-server[34500]: Starting nats-server

Server and client version

nats-server --version
nats-server: v2.10.5

nats --version
0.1.1

telegraf --version
Telegraf 1.28.3 (git: HEAD@a72550a0)

otelcol --version
otelcol-contrib version 0.88.0

Host environment

Darwin Kernel Version 23.1.0
darwin/arm64
No container runtime

Steps to reproduce

To see the message start netcat and a nats-server send syslog message to it:

nc -l 6514
nats-server -js -r tcp://127.0.0.1:6514

Or start an OpenTelemetry Contrib Collector with config:

receivers:
  syslog:
    tcp:
      listen_address: "127.0.0.1:6514"
    protocol: rfc5424
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: ["syslog"]
      exporters: ["debug"]

[ripienaar]

Background: It appears go syslog is delierately not RFC5424 compliant so requires compatible servers like rsyslog/syslog-ng or syslogd

See https://codereview.appspot.com/7036050/ for background and this blog post by someone discussing a fork they created https://sirsean.medium.com/srslog-sending-syslog-messages-from-go-a270d9c74ecd

I am not sure what for us is the answer, but thats the reason.

[marshauf]

Thank you for the background information. That explains it.

I worked around the issue by using files and parsing them with a regex parser. Either way, it would be nice if it is mentioned in the documentation.

Prometheus-nats-exporter has the same problem, too.

[ripienaar]

We'll discuss what our approach will be, I think whatever we choose this is something we should fix, so watch this space.