feat: Comprehensive concurrent testing for AT-102

[devin-ai-integration]

Make sure to read the contributing guidelines before submitting a PR

Summary

This PR implements comprehensive concurrent and multi-threaded testing infrastructure for llama.cpp to detect race conditions and validate thread safety across critical components, specifically targeting KV cache operations and context management for JIRA ticket AT-102.

Changes

New Test Files

• tests/test-concurrent-stress.cpp - Sustained concurrent load testing with 3 test suites:

  • Rapid context creation/destruction cycles (20 iterations per thread)
  • Parallel context operations with batch processing
  • Backend resource allocation stress with varying context parameters

• tests/test-kv-cache-concurrent.cpp - Dedicated KV cache race condition testing with 4 test suites:

  • Concurrent KV cache prepare operations
  • Concurrent KV cache update operations with varying context sizes
  • Concurrent sequence operations (copy, remove)
  • Mixed concurrent operations combining all patterns

Enhanced Existing Tests

• tests/test-thread-safety.cpp - Added rapid context recreation stress test with random timing delays to increase race condition exposure probability

• tools/server/tests/unit/test_completion.py - Added 3 high-volume concurrent server tests:

  • High-volume concurrent requests (8-50 requests across 4-8 slots)
  • Parallel decoding scenarios with multiple streams
  • Cache consistency validation under concurrent load

Build System Updates

• tests/CMakeLists.txt - Added new test targets using established llama_build_and_test patterns, labeled as 'concurrent' for filtering

Key Design Decisions

• Threading Model: Each thread creates and manages its own llama_context rather than sharing contexts (proper llama.cpp threading model)
• Race Detection: Uses atomic counters and random timing delays (1-10ms) to maximize race condition exposure
• Resource Leak Detection: Tracks context creation/destruction counts to detect leaks
• CI Compatibility: Limited thread counts and iterations to respect CI resource constraints

Local Testing Results

✅ test-concurrent-stress: All 3 stress tests passed (80 contexts created/destroyed, 0 errors)
✅ test-kv-cache-concurrent: All 4 KV cache tests passed (28 contexts in mixed operations)
✅ test-thread-safety: Enhanced test with new stress patterns passed
✅ Regression tests: 36/36 existing tests still pass

ThreadSanitizer Integration

Existing LLAMA_SANITIZE_THREAD CMake option can be used for automated race detection:

cmake -B build -DLLAMA_SANITIZE_THREAD=ON

Human Review Checklist

🔴 High Priority:

• Threading model correctness: Verify each thread creates its own context (no shared contexts across threads)
• Resource cleanup logic: Check atomic counter logic for context creation/destruction tracking
• Test parameters: Validate thread counts and iterations are appropriate for CI environments

🟡 Medium Priority:

• Race condition detection: Review random delay strategy and atomic counter usage
• CMake integration: Confirm new test targets follow established patterns
• Python test compatibility: Verify server tests handle concurrent scenarios correctly

Notes

• Python server tests encountered local environment issues (missing wget module) but should work in CI with proper dependencies
• Tests focus on realistic concurrent usage patterns rather than artificial stress scenarios
• All tests include comprehensive error reporting and cleanup verification

---

Link to Devin run: https://app.devin.ai/sessions/d1f6bdf15aa141e3aceec6c8e65e5750
Requested by: @alexpeng-cognition
Related: JIRA ticket AT-102

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring