Does the Loki ingest api support pipelines?

[jerrac]

I'm pushing logs via Telegraf's Loki output plugin.

So far I have not seen any sign my pipelines are being used.

I've tried setting endpoint = "/v1/loki/api/v1/push?pipeline_name=mypipeline", and adding "X-Greptime-Pipeline-Name" = "mypipeline" to the http_headers like I do for the X-Greptime-DB-Name and X-Greptime-Log-Table-Name headers. Neither method seems to change anything.

When I compare https://github.com/GreptimeTeam/greptimedb/blob/v0.14.4/src/servers/src/http/loki.rs and https://github.com/GreptimeTeam/greptimedb/blob/v0.14.4/src/servers/src/http/otlp.rs, I see that otlp.rs uses the pipeline::PipelineWay; crate, but loki.rs does not.

Doe the Loki ingest api point just not support pipelines?

Should I be finding another way to get my logs into greptimedb?

[zyy17]

@jerrac, the Loki doesn't support pipeline, and we track the issue in #4800. Can you consider ingesting logs with other protocols? For example, you can use NDJSON format or Elasticsearch protocol to ingest logs.

[zyy17]

You may try the following outputs of Telegraf:

• JSON output

• elasticsearch plugin

[jerrac]

Thanks for pointing me at that issue. I think I forgot to search on the topic. Sorry about that.

I'll look at the Elasticsearch option again. When I first started experimenting, I think I did try that. Can't recall the details on why I used Loki instead, though.

[jerrac]

Ah, yeah, so Telegraf tries to get the version number from Elasticsearch. Even when I have the health check turned off, the first thing I see is:

telegraf[2103515]: 2025-06-09T18:50:47Z E! [telegraf] Error running agent: connecting output outputs.elasticsearch: error connecting to output "outputs.elasticsearch": elasticsearch version check failed: json: cannot unmarshal number into Go value of type elastic.PingResult

That is why I didn't use the ES ingest option before.

My config:

[[outputs.elasticsearch]]
  namepass = ["syslog"]
  urls = [ "https://myhost/v1/elasticsearch/_bulk?db=mydb" ]
  health_check_interval = "0s"
  index_name = "syslog_es"
  enable_sniffer = false
  [outputs.elasticsearch.headers]
    "Authorization" = "Basic encodeduserpass"

[zyy17]

I'm afraid there will be some plugin compatibility issues with the ES protocol. Can you create an issue to track it? By the way, have you ever tried [[outputs.http]] with /v1/events API?

[jerrac]

I did briefly try pointing the outputs.http plugin at both the elasticsearch endpoint, and the events endpoint. It didn't work. Can't recall exactly what the errors were, I was pretty much mentally moving on to figuring out how to use Vector instead.

I'll see if I can find time to set up something for an issue on the elasticsearch compatibility with Telegraf. Hmm...