Adds metrics endpoint

[ilopezluna]

This PR uses the llama.cpp metrics endpoint to collect and aggregate the metrics of all active runners.

No active runners:

curl http://localhost:13434/metrics
# No active runners

An active runner with completions mode:

curl http://localhost:13434/metrics                                
# HELP llamacpp:n_busy_slots_per_decode Average number of busy slots per llama_decode() call
# TYPE llamacpp:n_busy_slots_per_decode counter
llamacpp:n_busy_slots_per_decode{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 1

# HELP llamacpp:n_decode_total Total number of llama_decode() calls
# TYPE llamacpp:n_decode_total counter
llamacpp:n_decode_total{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 2

# HELP llamacpp:predicted_tokens_seconds Average generation throughput in tokens/s.
# TYPE llamacpp:predicted_tokens_seconds gauge
llamacpp:predicted_tokens_seconds{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 285.714

# HELP llamacpp:prompt_seconds_total Prompt process time
# TYPE llamacpp:prompt_seconds_total counter
llamacpp:prompt_seconds_total{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 0.047

# HELP llamacpp:prompt_tokens_seconds Average prompt throughput in tokens/s.
# TYPE llamacpp:prompt_tokens_seconds gauge
llamacpp:prompt_tokens_seconds{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 1000

# HELP llamacpp:prompt_tokens_total Number of prompt tokens processed.
# TYPE llamacpp:prompt_tokens_total counter
llamacpp:prompt_tokens_total{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 47

# HELP llamacpp:requests_deferred Number of requests deferred.
# TYPE llamacpp:requests_deferred gauge
llamacpp:requests_deferred{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 0

# HELP llamacpp:requests_processing Number of requests processing.
# TYPE llamacpp:requests_processing gauge
llamacpp:requests_processing{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 0

# HELP llamacpp:tokens_predicted_seconds_total Predict process time
# TYPE llamacpp:tokens_predicted_seconds_total counter
llamacpp:tokens_predicted_seconds_total{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 0.007

# HELP llamacpp:tokens_predicted_total Number of generation tokens processed.
# TYPE llamacpp:tokens_predicted_total counter
llamacpp:tokens_predicted_total{backend="llama.cpp",mode="completion",model="ai/llama3.2"} 2

An active runner with embeddings mode:

curl http://localhost:13434/metrics
# HELP llamacpp:n_busy_slots_per_decode Average number of busy slots per llama_decode() call
# TYPE llamacpp:n_busy_slots_per_decode counter
llamacpp:n_busy_slots_per_decode{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 1

# HELP llamacpp:n_decode_total Total number of llama_decode() calls
# TYPE llamacpp:n_decode_total counter
llamacpp:n_decode_total{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 1

# HELP llamacpp:predicted_tokens_seconds Average generation throughput in tokens/s.
# TYPE llamacpp:predicted_tokens_seconds gauge
llamacpp:predicted_tokens_seconds{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 0

# HELP llamacpp:prompt_seconds_total Prompt process time
# TYPE llamacpp:prompt_seconds_total counter
llamacpp:prompt_seconds_total{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 0

# HELP llamacpp:prompt_tokens_seconds Average prompt throughput in tokens/s.
# TYPE llamacpp:prompt_tokens_seconds gauge
llamacpp:prompt_tokens_seconds{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 0

# HELP llamacpp:prompt_tokens_total Number of prompt tokens processed.
# TYPE llamacpp:prompt_tokens_total counter
llamacpp:prompt_tokens_total{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 0

# HELP llamacpp:requests_deferred Number of requests deferred.
# TYPE llamacpp:requests_deferred gauge
llamacpp:requests_deferred{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 0

# HELP llamacpp:requests_processing Number of requests processing.
# TYPE llamacpp:requests_processing gauge
llamacpp:requests_processing{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 0

# HELP llamacpp:tokens_predicted_seconds_total Predict process time
# TYPE llamacpp:tokens_predicted_seconds_total counter
llamacpp:tokens_predicted_seconds_total{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 0

# HELP llamacpp:tokens_predicted_total Number of generation tokens processed.
# TYPE llamacpp:tokens_predicted_total counter
llamacpp:tokens_predicted_total{backend="llama.cpp",mode="embedding",model="ai/mxbai-embed-large"} 0

[doringeman]

Can't we use https://pkg.go.dev/github.com/prometheus/prometheus/promql/parser?

[ilopezluna]

I get a 500 when visit the link but I assume its temporary. I'll take a look tomorrow, thanks!

[xenoscopic]

Yeah looks like pkg.go.dev was offline for a bit, seems back up now. I'd also advocate for less code we need to manage and test.

[ilopezluna]

I've changed to use https://pkg.go.dev/github.com/prometheus/common/expfmt so I can use families, err := parser.TextToMetricFamilies(strings.NewReader(string(body)))
And then I can get metrics per family and add our labels.
Let me know what do you think 🙏

[xenoscopic]

Yeah looks like pkg.go.dev was offline for a bit, seems back up now. I'd also advocate for less code we need to manage and test.

[xenoscopic]

LGTM, just a few minor suggestions.