Log Exporter

Log Exporter makes it possible to monitor the custom API logs using Loki.

Overview

This project was inspired by Promtail. Your API have to run on Docker. The exporter connects to your container, reads the container logs, pushes the logs to the Loki.

Getting Started

Log Structure

In the current version of the project, following a specific log structure is no longer mandatory. You can configure your logs with any fields you prefer.

However, there is an existing Grafana dashboard configured to work with the following log structure:

{"ip":"192.168.1.1","caller":"app/main.go:102","path":"/users","level":"info","method":"GET","status":200,"msg":"get users successfully","dt":"mobile","timing":0.776347977,"ts":"2023-07-10T13:01:38Z"}

{
    "ip":"192.168.1.1",
    "caller":"app/main.go:102",
    "path":"/users",
    "level":"info",
    "method":"GET",
    "status":200,
    "msg":"get users successfully",
    "dt":"mobile",
    "timing":0.776347977,
    "ts":"2023-07-10T13:01:38Z"
}

• ip - ip address.
• caller - log line in your application.
• path - your api path.
• level - log level.
• method - HTTP method.
• status - HTTP status code.
• msg - your log message.
• dt - device type.
• timing - your handle time (seconds) for each request.
• ts - log creation time.

The Grafana dashboard includes specific visualizations for the level field, particularly for error and warn levels. If you use these log levels, corresponding graphs will be displayed on the dashboard, providing valuable insights into the occurrences and frequency of warnings and errors.

Note: The package you are using will take your log and add a container field before to push Loki.

Usage Exporter

We use the following command to run it on Docker.

docker run -d \
  --name log-exporter \
  --network monitoring \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --env CONTAINER_NAME=your_container_name \
  --env LOKI_URL=http://loki:3100/loki/api/v1/push \
  --env DOCKER_RECONNECTION_TIME=20s \
  elvintacirzade/log-exporter:latest

See more information.

Monitoring

We use Loki and Grafana for monitoring.

You must run Loki and Grafana on Docker.

After running Loki and Grafana you must add Loki data source in Grafana. Now you can import dashboard for exporter.