UPSTREAM PR #16969: sycl: flash-attention implementation

[DajanaV]

Mirrored from ggml-org/llama.cpp#16969

This PR adds basic Flash Attention support for the SYCL backend, enabling more efficient attention computation on Intel GPUs.

• Implemented Flash Attention kernel for SYCL backend
• Added forward pass implementation with block-wise computation
• Integrated with existing GGML SYCL infrastructure
• Support for both F32

Authors:
Joint work by @safranowith and @ye-NX

Notes:

• This is an initial implementation
• Performance benchmarks and optimizations are planned for future iterations
• Feedback and suggestions are welcome!

[loci-agentic-ai]

Access the complete analysis in the LOCI Dashboard

Performance Analysis Summary: LLaMA.cpp Project

Critical Function Performance Changes

Based on the analysis of version 4728565c-fc50-4956-9b0d-8a1586354530 compared to baseline 6e2eaebb-4d5e-4576-a5d2-5cc3b9172699, the following critical functions show performance changes:

Response Time Degradations

• llama_context_default_params: +31 ns (+0.057%)
• _ZNKSt8__detail12_CharMatcherINSt7__cxx1112regex_traitsIwEELb1ELb1EEclEw: +15 ns (+0.057%)

Throughput Degradations

• _ZNKSt8__detail12_CharMatcherINSt7__cxx1112regex_traitsIwEELb1ELb1EEclEw: +15 ns (+0.057%)
• _ZSt10_ConstructI12llama_ubatchJRKS0_EEvPT_DpOT0_: +15 ns (+0.053%)

Bottleneck Degradations

• _ZSt10_ConstructI12llama_ubatchJRKS0_EEvPT_DpOT0_: +15 ns (+0.107%)

KPI Impact Analysis

1. Tokens Per Second Impact

No Direct Impact Identified: The core inference functions show no performance degradation:

• llama_decode(): No changes detected
• llama_encode(): No changes detected
• llama_tokenize(): No changes detected

Assessment: Based on the reference that 2ms slower llama_decode results in 7% fewer tokens per second, the current changes do not affect tokenization or inference performance directly.

2. Power Consumption Impact

Binary-Level Analysis:

• build.bin.libllama.so: -0.0% change (306.895 µJ vs 306.895 µJ baseline)
• build.bin.libggml-base.so: 0.0% change
• build.bin.libggml-cpu.so: 0.0% change
• build.bin.libggml.so: 0.0% change

Assessment: Power consumption remains stable across all binaries with negligible changes.

3. Quantization Efficiency

No Impact Detected:

• llama_model_quantize(): No performance changes identified
• Quantization-related functions show no degradation
• GGML quantization operations remain stable

4. Memory Usage Impact

Affected Functions:

• Memory Management: _ZSt10_ConstructI12llama_ubatchJRKS0_EEvPT_DpOT0_ shows +15 ns bottleneck increase
• KV Cache Operations: Multiple cache-related functions show minor improvements (-0.07 to -0.11 ns)

Key Changes:

• _ZNK27llama_kv_cache_iswa_context8get_baseEv: -0.07 ns throughput improvement
• _ZNK22llama_kv_cache_context10get_ubatchEv: -0.07 ns throughput improvement

5. Batch Processing Impact

Affected Functions:

• Batch Construction: _ZSt10_ConstructI12llama_ubatchJRKS0_EEvPT_DpOT0_ shows bottleneck degradation
• Batch Operations: Core batch processing functions (llama_batch_init, llama_batch_get_one) show no changes

Root Cause Analysis

Control Flow Investigation

The CFG analysis of llama_context_default_params reveals:

• Identical Assembly Code: No instruction-level changes between versions
• Same Control Flow: Linear execution path unchanged
• External Factors: Performance degradation likely due to binary layout changes from SYCL Flash Attention integration

SYCL Integration Impact

The addition of Flash Attention support introduces:

• Binary Layout Changes: New object files alter memory layout affecting cache behavior
• Compilation Effects: Additional template instantiations and link-time optimization changes
• System Resource Competition: SYCL runtime initialization affects system performance

Action Items for Performance Improvement

Immediate Code Optimizations

1. Batch Construction Optimization:

   • Investigate _ZSt10_ConstructI12llama_ubatchJRKS0_EEvPT_DpOT0_ bottleneck increase
   • Review memory allocation patterns in batch initialization
   • Consider pre-allocated batch pools to reduce construction overhead

2. Binary Layout Optimization:

   • Implement section ordering to improve instruction cache locality
   • Consider profile-guided optimization (PGO) for better code placement
   • Review linker flags for optimal memory layout

3. Template Instantiation Control:

   • Minimize regex template instantiations affecting character matching performance
   • Use explicit template instantiation to control compilation overhead
   • Consider template specialization for common use cases

Build System Improvements

1. Compilation Flags:

   • Enable link-time optimization (LTO) for better cross-module optimization
   • Use -fprofile-use with representative workloads
   • Consider -march=native for target-specific optimizations

2. Memory Layout Control:

   • Implement custom section placement for critical functions
   • Use __attribute__((hot)) for frequently called functions
   • Consider memory prefetching hints in critical paths

3. SYCL Integration Isolation:

   • Implement lazy SYCL initialization to reduce startup overhead
   • Isolate SYCL-specific code to minimize binary footprint impact
   • Use conditional compilation to reduce template instantiation overhead

Performance Monitoring Focus Areas

The analysis shows that while individual function changes are minimal (0.05-0.11%), the cumulative effect of binary layout changes requires attention to:

• Cache Performance: Monitor instruction and data cache miss rates
• Memory Allocation Patterns: Track batch construction and KV cache efficiency
• Template Instantiation Overhead: Control regex and SYCL template expansion
• Binary Size Growth: Monitor object file size increases from new features

The current changes maintain stable inference performance while introducing new SYCL capabilities, with optimization opportunities focused on build-time and memory layout improvements rather than algorithmic changes.