How this project works?

[ajeetraina]

The Metrics Flow Architecture

llama.cpp (HOST) → Go Backend (CONTAINER) → Prometheus (CONTAINER) → Grafana (CONTAINER)

The Go Backend Acts as a "Metrics Bridge"

The key insight is that Prometheus and Grafana don't directly fetch from llama.cpp. Instead:

1. llama.cpp process runs natively on your machine

ps aux | grep llama
# Shows: llama.cpp process running on host

2. Go Backend (in container) connects to llama.cpp

// Go backend connects to host-based llama.cpp
client := openai.NewClient(
    openai.WithBaseURL("http://model-runner.docker.internal/engines/llama.cpp/v1/"),
    openai.WithAPIKey("dockermodelrunner"),
)

3. Go Backend observes API interactions and calculates metrics

func handleChat(...) {
    start := time.Now()
    
    // Make API call to llama.cpp on host
    stream, err := client.CreateChatCompletionStream(ctx, request)
    
    // Observe the interaction and calculate metrics
    firstTokenTime := time.Now()
    for response := range stream {
        outputTokens++
        // Backend is measuring what it observes
    }
    
    // Calculate and expose metrics based on observations
    tokensPerSec := float64(outputTokens) / time.Since(firstTokenTime).Seconds()
    llamacppTokensPerSecond.WithLabelValues(model).Set(tokensPerSec)
}

4. Go Backend exposes calculated metrics at /metrics endpoint

# Go backend (container) exposes metrics
curl http://backend:9090/metrics

# Returns:
# genai_app_llamacpp_tokens_per_second{model="ai/llama3.2:1B-Q8_0"} 8.5
# genai_app_llamacpp_prompt_eval_seconds_bucket{model="ai/llama3.2:1B-Q8_0"}

5. Prometheus (container) scrapes from Go Backend (container)

# prometheus.yml
scrape_configs:
  - job_name: 'backend'
    static_configs:
      - targets: ['backend:9090']  # Container-to-container communication

6. Grafana (container) queries Prometheus (container)

# Both in same Docker network - easy communication
networks:
  - app-network

Visual Architecture

┌─────────────────────────────────────────────────────────────┐
│                        HOST MACHINE                         │
│                                                             │
│  ┌─────────────────┐                                       │
│  │   llama.cpp     │ ◄─── Runs natively via Docker Model   │
│  │   (native)      │      Runner for max performance       │
│  └─────────────────┘                                       │
│           │                                                 │
│           │ API calls via                                   │
│           │ model-runner.docker.internal                    │
│           ▼                                                 │
│  ┌─────────────────────────────────────────────────────────┐│
│  │              DOCKER CONTAINERS                          ││
│  │                                                         ││
│  │  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐ ││
│  │  │Go Backend   │───▶│Prometheus   │───▶│Grafana      │ ││
│  │  │(metrics     │    │(scrapes     │    │(visualizes  │ ││
│  │  │ bridge)     │    │ backend)    │    │ data)       │ ││
│  │  └─────────────┘    └─────────────┘    └─────────────┘ ││
│  │         │                                               ││
│  │         │ Exposes /metrics                              ││
│  │         ▼                                               ││
│  │  genai_app_llamacpp_*                                   ││
│  └─────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘

[ajeetraina]

The Go Backend is the "Metrics Calculator"

• It doesn't access llama.cpp's internal metrics directly
• It measures API interaction patterns (response times, token counts, etc.)
• It calculates derived metrics from what it observes
• It exposes these as Prometheus-format metrics

Example of What the Backend Measures:

// Not direct llama.cpp internals, but observable behavior:
- Time to first token → llamacpp_prompt_eval_seconds  
- Tokens generated / time elapsed → llamacpp_tokens_per_second
- Response payload analysis → llamacpp_context_size
- Memory usage patterns → llamacpp_memory_per_token (estimated)

Why This Architecture Works Well:

• Performance: llama.cpp runs natively for maximum speed
• Observability: Backend provides rich metrics without llama.cpp modification
• Simplicity: Standard container networking for monitoring stack
• Isolation: Host-based AI, containerized observability
• Portability: Same pattern works across different deployment environments

Prometheus and Grafana get metrics from the Go backend, which acts as a bridge between the host-based llama.cpp and the containerized monitoring stack.