Sending Log Events to Grafana Loki

[billcunnien]

Good morning,

I would like to visualize some metrics on our logs using Grafana, Prometheus and Loki. Since Grafana cannot use SEQ as a data source, I thought I would use Loki as a log aggregator for Grafana. I installed the HTTP app on SEQ and set it up to post the event to Loki. It does not work.

How do I get these log events sent out to this other aggregator so I can take advantage of the visualizations/queries in Grafana?

Thanks,
Bill

[nblumhardt]

Hi Bill, thanks for your message.

This reminds me that it would be really nice to have an OTLP output app for Seq, since I'm pretty sure Loki can ingest events in OTLP format. Is this something you'd expect to run long-term, or just a one-off?

It should be possible to send logs via the HTTP app, using Loki's HTTP API at /loki/api/v1/push, as long as the content-type is application/json. Since this will result in a request per event, the volume that can be forwarded in this way will be fairly low (a few events per second would be the safe limit).

The body template should resemble:

{ {
    streams: [
    {
      stream: { Prop1, Prop2, Prop3 },
      values: [
        [ToJson(Nanoseconds(FromUnixEpoch(@t))) , @m ]
      ]
    }
  ]
} }

where Prop1, Prop2, and Prop3 are string-valued properties you want to use as Loki tags.

But - I've just discovered, the Nanoseconds and FromUnixEpoch functions don't exist in the Seq.Syntax library used by the HTTP request app.

Adding those is no problem, but I thought I should check in first - will the volume of data involved work within the constraints of one-event-per-message?

[billcunnien]

Hi,

If I can get this working, I would think it would be a long-term project.

I have tried several iterations of the suggested body, but it continues to error without a clear reason why (other than it isn't formatted correctly).

For now, the one-event-per-message will be the target approach.

Any suggestions would be appreciated!
Bill

[nblumhardt]

Hi Bill,

Currently those functions don't exist, so I'll take a look at adding them, and test out the template.

Cheers,
Nick

[nblumhardt]

A quick update and some bad news along with the good...

I've added the necessary functions to Seq.Syntax to get this template working:

{ {
    streams: [
    {
      stream: { Prop1, Prop2, Prop3 },
      values: [
        [ToString(1000000 * TotalMilliseconds(FromUnixEpoch(@t))), @m ]
      ]
    }
  ]
} }

The bad new is, unfortunately, there's no way to get this loaded into the HTTP request app without a Seq patch, since Seq.Syntax ships along with seqcli, which is bundled in the Seq installer/Docker image.

This complicates things a bit, but there's a good chance we'll have a suitable patch out in the next week or two, so as long as you can update your server once it's ready, we should still have a path forward 🤞

[nblumhardt]

Just a quick note, Seq 2025.2.14687 will include this (#2494), and should be out early in the coming week.