UPSTREAM PR #16923: Add e2e tests for embedding raw flag

[DajanaV]

Mirrored from ggml-org/llama.cpp#16923

🧩 Summary

This PR adds a CI workflow for end-to-end embedding tests.
It marks the first phase of an effort to move an abstraction of the existing examples/llama-embedding logic behind llama-server, so the server can use llama.cpp’s own embedding implementation instead of external (OpenAI) APIs.

🎯 Motivation & Future

llama-server currently supports OpenAI-compatible /embedding requests, but those are not backed by native llama.cpp logic.
This workflow establishes a reproducible test foundation before refactoring the embedding code so that:

• The server can generate embeddings locally.
• --parallel N can support multiple concurrent embedding requests.
• The standalone CLI will remain for lightweight workflows, while the server will use the same shared embedding path for persistent deployments.

⚙️ CI Implementation

• Adds a GitHub Actions job to run embedding E2E tests with cached GGUF models (TinyLlama).
• Verifies embedding output dimensions and deterministic behavior.
• Uses lightweight models for fast CI runs (with an optional large model test).

🧱 Embedding CPP Logic Flow Update

A small cleanup in print_raw_embeddings() improves readability, logic flow, and isolation.
Although minor, this change is modular alongside the CI workflow changes, touching a vertical slice of the embedding flow without altering evaluation, model logic, or any interface. Note that expecting purely small horizontal modularity ossifies software (makes it brittle).

🚀 Next Steps

1. Extend CI test coverage for all embedding cli flags.
2. Abstract core embedding code from examples into a shared utility (e.g. common/embedding_utils.cpp).
3. Integrate that abstraction into llama-server for local "/embedding" requests (while maintaining CLI endpoints and backwards compatibility).
   a. Extend CI coverage for concurrent (--parallel) embedding tests.

(could actually become more than three steps)

Note:
This PR adds a self-contained workflow (embeddings.yml) that runs embedding end-to-end tests on both feature branches and master.
It is safe to merge into upstream as-is — the workflow will automatically execute in the main CI environment once on master.

[loci-agentic-ai]

Access the complete analysis in the LOCI Dashboard

LLaMA.cpp Performance Analysis Summary

Critical Function Performance Status

Core Inference Functions - No Performance Impact

• llama_decode(): Response Time: 49,003,724 ns (no change)
• llama_encode(): Response Time: 12,329,178 ns (no change)
• llama_tokenize(): Response Time: 834,828 ns (no change)
• llama_batch_init(): Response Time: 257 ns (no change)
• llama_memory_clear(): Response Time: 49 ns (no change)

Functions with Minimal Degradation

• _ZNK27llama_kv_cache_iswa_context7get_swaEv (KV cache getter): +0.071% throughput degradation (97 ns vs 97 ns)
• llama_sampler_init_typical: +0.110% bottleneck degradation (68 ns vs 68 ns)
• ggml_get_max_tensor_size@plt: +0.045% response time degradation (8 ns vs 8 ns)

KPI Impact Analysis

1. Tokens Per Second - No Impact

Status: No changes detected in tokenization/inference critical path functions

• llama_decode(): No performance change (49.0 million ns baseline)
• llama_encode(): No performance change (12.3 million ns baseline)
• llama_tokenize(): No performance change (834,828 ns baseline)

Reference Impact: Based on the provided benchmark (ollama://smollm:135m on 12th Gen Intel i7-1255U), a 2ms increase in llama_decode() results in 7% tokens/second reduction. Current analysis shows no measurable change in llama_decode() response time.

2. Power Consumption - Stable

Binary-Level Analysis:

• build.bin.libllama.so: -0.0% change (306,977.93 nJ vs 306,978.59 nJ)
• build.bin.libggml.so: 0.0% change (6,339.24 nJ)
• build.bin.libggml-cpu.so: 0.0% change (151,692.17 nJ)
• build.bin.libggml-base.so: 0.0% change (90,434.19 nJ)

3. Quantization Efficiency - No Impact

Status: No changes in quantization-related functions

• llama_model_quantize(): Not measured (function not active in current analysis)
• Quantization format handling: No performance changes detected

4. Memory Usage - Minimal Impact

Affected Functions:

• _ZNK27llama_kv_cache_iswa_context7get_swaEv: +0.071% throughput degradation
  • Control Flow: Identical CFG structure between versions
  • Root Cause: Memory layout or cache alignment changes, not code modifications
  • Impact: Negligible effect on KV cache access patterns

5. Batch Processing - No Impact

Status: No changes in batch processing functions

• llama_batch_init(): No performance change (257 ns baseline)
• llama_batch_get_one(): Not measured in current analysis
• llama_batch_free(): Not measured in current analysis

Control Flow Analysis

KV Cache Function (get_swa)

CFG Comparison: Identical control flow structure between versions

• Same branching patterns and instruction sequences
• Degradation attributed to external factors (memory layout, cache alignment)
• No algorithmic or structural changes

Action Items

Immediate Actions

1. Monitor KV Cache Access Patterns: Profile memory access in llama_kv_cache_iswa_context for cache line alignment optimization
2. Verify Memory Layout: Check for changes in struct padding or member alignment affecting cache performance
3. Validate Compiler Optimizations: Ensure consistent optimization flags between builds

Code-Focused Optimizations

1. KV Cache Structure Alignment: Review llama_kv_cache_iswa_context struct layout for optimal memory alignment
2. Function Inlining: Consider inlining small getter functions like get_swa() to eliminate call overhead
3. Memory Prefetching: Add prefetch hints for frequently accessed KV cache members

Build System Improvements

1. Consistent Build Flags: Standardize optimization levels and alignment settings across builds
2. Profile-Guided Optimization: Enable PGO for hot path functions in production builds
3. Link-Time Optimization: Enable LTO for cross-module optimization of frequently called functions

Conclusion

The analysis reveals stable performance across all critical inference functions. The minimal degradations (< 0.11%) in auxiliary functions represent measurement variance rather than functional regressions. Core inference performance remains unaffected, with no impact on tokens per second throughput or power consumption efficiency.