Skip to content

TMI Promtail Logger

Jump to bottom
Eric Fitzgerald edited this page Nov 12, 2025 · 1 revision

TMI Promtail Logger

Promtail logger application for sending TMI server logs to Grafana via Loki.

Overview

The TMI Promtail Logger is a lightweight log forwarding service that streams TMI server logs to Grafana Loki for centralized logging, monitoring, and analysis. It uses Grafana's Promtail agent configured specifically for Heroku deployments to collect and forward logs in real-time.

Repository: /Users/efitz/Projects/tmi-promtail

This tool enables comprehensive log management for TMI server deployments, making it easy to:

  • Search and filter logs across time ranges
  • Monitor application health and performance
  • Debug issues with historical log data
  • Set up alerts based on log patterns
  • Visualize log metrics in Grafana dashboards

Features

  • Heroku Integration - Designed for Heroku log drain integration
  • Real-time Streaming - Forwards logs to Loki as they're generated
  • Label Enrichment - Automatically labels logs with app, environment, and dyno information
  • Secure Transport - HTTPS connection with basic authentication
  • Low Resource Usage - Lightweight Promtail agent with minimal overhead
  • Auto-scaling - Scales with your Heroku dynos

Architecture

┌─────────────┐
│ TMI Server  │
│  (Heroku)   │
└──────┬──────┘
       │ Heroku Log Drain (HTTPS)
       ▼
┌─────────────┐
│  Promtail   │
│  (Heroku)   │
└──────┬──────┘
       │ HTTPS + Basic Auth
       ▼
┌─────────────┐
│    Loki     │
│  (Grafana)  │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   Grafana   │
│  Dashboard  │
└─────────────┘

How It Works

  1. TMI Server generates logs on Heroku
  2. Heroku Log Drain forwards logs to Promtail via HTTPS
  3. Promtail processes and enriches logs with labels
  4. Loki stores logs in time-series format
  5. Grafana provides querying and visualization

Prerequisites

  • Heroku account with TMI server deployed
  • Grafana Cloud account (or self-hosted Loki/Grafana)
  • Git for deployment

Installation

Step 1: Get Loki Credentials

Grafana Cloud

  1. Log in to Grafana Cloud
  2. Navigate to Connections > Data Sources
  3. Find your Loki data source
  4. Note the URL (e.g., https://logs-prod-us-central1.grafana.net/loki/api/v1/push)
  5. Create a new API key:
    • Go to Security > API Keys
    • Click Add API key
    • Name: "TMI Promtail Logger"
    • Role: MetricsPublisher
    • Copy the API key

Your credentials:

  • LOKI_URL: The push endpoint URL
  • LOKI_USERNAME: Your Grafana instance ID (from URL)
  • LOKI_PASSWORD: The API key you just created

Self-hosted Loki

If you're running your own Loki instance:

  • LOKI_URL: Your Loki push endpoint (e.g., http://loki:3100/loki/api/v1/push)
  • LOKI_USERNAME: Your basic auth username (if configured)
  • LOKI_PASSWORD: Your basic auth password (if configured)

Step 2: Deploy Promtail to Heroku

Clone the Repository

cd ~/Projects
git clone <repository-url> tmi-promtail
cd tmi-promtail

Create Heroku App

# Create app
heroku create your-app-name-promtail

# Set buildpack for shell script execution
heroku buildpacks:add --index 1 heroku-community/apt
heroku buildpacks:add --index 2 https://github.com/heroku/heroku-buildpack-shell

Configure Environment Variables

# Set Loki credentials
heroku config:set LOKI_URL="https://logs-prod-us-central1.grafana.net/loki/api/v1/push"
heroku config:set LOKI_USERNAME="123456"  # Your Grafana instance ID
heroku config:set LOKI_PASSWORD="your-api-key-here"

# Promtail will use Heroku's $PORT automatically

Deploy to Heroku

# Deploy
git push heroku master

# Verify deployment
heroku logs --tail

Step 3: Configure Heroku Log Drain

Add the Promtail app as a log drain for your TMI server:

# Get Promtail app URL
PROMTAIL_URL=$(heroku info -s | grep web_url | cut -d= -f2)

# Add log drain to TMI server app
heroku drains:add $PROMTAIL_URL --app your-tmi-server-app

# Verify drain was added
heroku drains --app your-tmi-server-app

Step 4: Verify Logs in Grafana

  1. Open Grafana
  2. Go to Explore
  3. Select your Loki data source
  4. Query: {app="tmi-server"}
  5. You should see logs streaming in

Configuration

Promtail Configuration

The configuration is defined in promtail-config.yaml:

server:
  http_listen_port: ${PORT}

positions:
  filename: /tmp/positions.yaml

clients:
  - url: ${LOKI_URL}
    basic_auth:
      username: ${LOKI_USERNAME}
      password: ${LOKI_PASSWORD}

scrape_configs:
  - job_name: heroku
    heroku_drain:
      labels:
        job: heroku
        app: tmi-server
        environment: heroku
        source: heroku-dyno
    relabel_configs:
      - source_labels: [__heroku_drain_dyno]
        target_label: dyno
      - source_labels: [__heroku_drain_app]
        target_label: heroku_app

Configuration Options

Server Configuration:

  • http_listen_port: ${PORT} - Uses Heroku's dynamic port assignment

Position Tracking:

  • filename: /tmp/positions.yaml - Tracks which logs have been sent (ephemeral on Heroku)

Loki Client:

  • url - Loki push endpoint
  • basic_auth - Credentials for Loki

Labels:

  • job: heroku - Identifies log source
  • app: tmi-server - Application name
  • environment: heroku - Deployment environment
  • source: heroku-dyno - Log origin

Dynamic Labels:

  • dyno - Extracted from __heroku_drain_dyno
  • heroku_app - Extracted from __heroku_drain_app

Customizing Labels

Edit promtail-config.yaml to add custom labels:

scrape_configs:
  - job_name: heroku
    heroku_drain:
      labels:
        job: heroku
        app: tmi-server
        environment: production  # Change this
        team: security           # Add custom label
        region: us-east-1        # Add region

After changes, commit and redeploy:

git add promtail-config.yaml
git commit -m "Update Promtail labels"
git push heroku master

Deployment Files

Procfile

Defines the Heroku process:

web: ./promtail-linux-amd64 -config.file=./promtail-config.yaml -config.expand-env
  • Runs Promtail binary
  • Uses configuration file
  • -config.expand-env enables environment variable expansion

buildpack-run.sh

Buildpack script that downloads Promtail:

#!/bin/bash
curl -L -o promtail-linux-amd64.zip https://github.com/grafana/loki/releases/download/v3.5.5/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
chmod +x promtail-linux-amd64

Downloads Grafana Promtail v3.5.5 during deployment.

Aptfile

Specifies system dependencies:

unzip

Required to extract the Promtail binary.

Usage

Viewing Logs in Grafana

Basic Query

{app="tmi-server"}

Shows all logs from TMI server.

Filter by Dyno

{app="tmi-server", dyno="web.1"}

Shows logs from specific dyno.

Filter by Log Level

{app="tmi-server"} |= "ERROR"

Shows only error logs.

Search for Specific Text

{app="tmi-server"} |= "authentication failed"

Rate Queries

rate({app="tmi-server"}[5m])

Log rate over 5-minute intervals.

Aggregate by Dyno

sum by (dyno) (rate({app="tmi-server"}[5m]))

Log rate per dyno.

LogQL Examples

LogQL is Loki's query language, similar to PromQL:

Pattern Matching:

{app="tmi-server"} |~ "error|exception|failed"

JSON Parsing:

{app="tmi-server"} | json | level="error"

Metrics from Logs:

sum(rate({app="tmi-server"} |= "request" | json | status="500" [1m]))

Top Errors:

topk(10, sum by (error) (count_over_time({app="tmi-server"} |= "ERROR" [1h])))

Monitoring and Alerts

Creating Dashboards

In Grafana, create a dashboard with panels for:

Log Volume Panel:

sum(rate({app="tmi-server"}[1m]))

Error Rate Panel:

sum(rate({app="tmi-server"} |= "ERROR" [5m]))

Response Time Panel:

{app="tmi-server"} | json | __error__="" | unwrap duration | quantile_over_time(0.95, [5m])

Setting Up Alerts

Create alerts in Grafana for:

High Error Rate:

sum(rate({app="tmi-server"} |= "ERROR" [5m])) > 10

Application Down:

absent_over_time({app="tmi-server"}[5m])

Authentication Failures:

sum(rate({app="tmi-server"} |= "authentication failed" [5m])) > 5

Troubleshooting

No Logs Appearing in Grafana

Check Promtail is Running:

heroku ps --app your-promtail-app

Should show web.1: up.

Check Promtail Logs:

heroku logs --tail --app your-promtail-app

Look for connection errors or authentication failures.

Verify Drain Configuration:

heroku drains --app your-tmi-server-app

Should show your Promtail URL.

Test Loki Connection:

curl -u "$LOKI_USERNAME:$LOKI_PASSWORD" "$LOKI_URL"

Authentication Errors

Symptom: Promtail logs show "401 Unauthorized"

Solution:

  1. Verify LOKI_USERNAME and LOKI_PASSWORD are correct
  2. Check API key hasn't expired
  3. Ensure API key has MetricsPublisher role
  4. Update credentials: 
    heroku config:set LOKI_PASSWORD="new-api-key" --app your-promtail-app

Connection Timeouts

Symptom: Promtail logs show "context deadline exceeded"

Solution:

  1. Check Loki URL is correct and accessible
  2. Verify network connectivity from Heroku
  3. Check Grafana Cloud status page
  4. Increase timeout (requires custom Promtail config)

Missing Labels

Symptom: Logs appear but without expected labels

Solution:

  1. Verify promtail-config.yaml label configuration
  2. Check relabel_configs are correct
  3. Ensure -config.expand-env flag is set in Procfile
  4. Restart Promtail after config changes

Promtail Keeps Restarting

Check Crash Logs:

heroku logs --tail --app your-promtail-app | grep "error"

Common Causes:

  • Invalid configuration file (YAML syntax error)
  • Missing environment variables
  • Port binding issues (ensure using $PORT)
  • Out of memory (upgrade dyno type)

Solution:

# Validate config locally
promtail -config.file=./promtail-config.yaml -config.expand-env=true -dry-run

# Check dyno size
heroku ps:type --app your-promtail-app

# Upgrade if needed
heroku ps:type web=standard-1x --app your-promtail-app

High Memory Usage

Symptom: Promtail uses excessive memory

Causes:

  • Large batch sizes
  • High log volume
  • Position file growth

Solutions:

  1. Adjust batch size in config
  2. Upgrade dyno type
  3. Clear position file periodically (though it's ephemeral on Heroku)

Performance Considerations

Dyno Sizing

Promtail is lightweight but dyno size depends on log volume:

  • Hobby/Free: Suitable for low-volume testing
  • Standard-1X: Good for production with moderate logs
  • Standard-2X: High-volume production deployments
# Check current dyno type
heroku ps --app your-promtail-app

# Upgrade if needed
heroku ps:type web=standard-2x --app your-promtail-app

Cost Optimization

  • Promtail itself is open source and free
  • Heroku dyno costs apply (starting at $7/month for Standard-1X)
  • Grafana Cloud has free tier (50GB logs/month)
  • Self-hosted Loki has no licensing costs

Log Volume Management

Monitor and control log volume to reduce costs:

  1. Set appropriate log levels in TMI server
  2. Filter logs in Promtail config to exclude verbose logs
  3. Set retention policies in Loki
  4. Use sampling for high-frequency logs

Upgrading Promtail

To upgrade to a newer Promtail version:

  1. Edit buildpack-run.sh and update version number:

    curl -L -o promtail-linux-amd64.zip https://github.com/grafana/loki/releases/download/v3.6.0/promtail-linux-amd64.zip

  2. Commit and deploy:

    git add buildpack-run.sh
git commit -m "Upgrade Promtail to v3.6.0"
git push heroku master

  3. Verify upgrade:

    heroku logs --tail --app your-promtail-app | grep "version"

Alternative Deployments

Docker Deployment

If you prefer Docker over Heroku:

FROM grafana/promtail:3.5.5

COPY promtail-config.yaml /etc/promtail/config.yaml

ENTRYPOINT ["/usr/bin/promtail", "-config.file=/etc/promtail/config.yaml", "-config.expand-env"]

Deploy to any container platform (ECS, Cloud Run, Kubernetes).

Kubernetes Deployment

For Kubernetes deployments:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      containers:
      - name: promtail
        image: grafana/promtail:3.5.5
        args:
        - -config.file=/etc/promtail/config.yaml
        - -config.expand-env
        env:
        - name: LOKI_URL
          valueFrom:
            secretKeyRef:
              name: promtail-secrets
              key: loki-url
        volumeMounts:
        - name: config
          mountPath: /etc/promtail
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: config
        configMap:
          name: promtail-config
      - name: varlog
        hostPath:
          path: /var/log

Direct Heroku Drain to Grafana Cloud

Alternatively, use Grafana Cloud's native Heroku integration:

  1. Install Grafana Cloud app from Heroku marketplace
  2. Configure directly without Promtail middleman

Pros: Simpler setup, no Promtail management Cons: Less control over log processing and enrichment

Best Practices

Security

  1. Rotate API keys regularly
  2. Use environment variables for all credentials (never hardcode)
  3. Enable TLS for Loki connections (default with Grafana Cloud)
  4. Restrict Promtail to only MetricsPublisher role
  5. Monitor access logs in Grafana Cloud

Reliability

  1. Monitor Promtail health with Grafana alerts
  2. Set up dyno alerts in Heroku dashboard
  3. Configure log retention policies in Loki
  4. Test failover scenarios periodically
  5. Document recovery procedures

Performance

  1. Right-size dynos based on log volume
  2. Batch logs efficiently with Promtail config
  3. Use labels wisely (too many = high cardinality)
  4. Set sensible retention (balance cost vs. history)
  5. Archive old logs to S3 if needed

Related Resources

External Documentation

Support

For issues:

  1. Check Promtail logs: heroku logs --tail --app your-promtail-app
  2. Verify drain configuration: heroku drains --app your-tmi-server-app
  3. Test Loki connectivity with curl
  4. Review Grafana Cloud status page
  5. See Getting Help for support channels

Promtail Version: 3.5.5 License: Apache License 2.0

Home

Getting Started

Deployment

Operation

Troubleshooting

Development

Integrations

Tools

API Reference

Reference

Clone this wiki locally