Add comprehensive E2E test suite for llama.cpp (AT-104)

[devin-ai-integration]

Overview

This PR implements comprehensive end-to-end (E2E) test coverage for llama.cpp, extending the existing unit-focused API testing framework to validate complete user workflows and component integration.

Jira Ticket: AT-104

Link to Devin run: https://app.devin.ai/sessions/e503e24872474b0aa47b655c06a7a45f

Requested by: Alex Peng ([email protected]) / @alexpeng-cognition

Changes Summary

Framework Extensions

Extended ServerProcess with PipelineTestProcess class (tools/server/tests/utils.py):

• Pipeline testing capabilities for complete workflows
• CLI tool execution wrappers (llama-cli, llama-bench)
• Context management and KV cache validation methods
• State transition tracking and validation

Enhanced pytest fixtures (tools/server/tests/conftest.py):

• pipeline_process - PipelineTestProcess instance with automatic cleanup
• e2e_small_model_config - Optimized small model config for CI
• e2e_embedding_model_config - Embedding model configuration
• e2e_multimodal_model_config - Multimodal model configuration
• concurrent_test_prompts - Test prompts for concurrent scenarios

New E2E Test Suites (38 tests)

1. Pipeline Workflows (test_pipeline_workflows.py) - 8 tests

• Complete pipeline: model download → loading → inference
• Server state transition validation (INITIAL → LOADING_MODEL → READY → GENERATING)
• Extended context management during long sessions
• KV cache behavior validation
• Streaming pipeline workflows
• Embedding model pipeline support

2. Tool Integration (test_tool_integration.py) - 10 tests

• llama-cli interactive and non-interactive execution
• llama-bench performance testing validation
• Custom embedding generation workflows
• Tool parameter validation and error handling
• Server/CLI resource coordination
• JSON output format support

3. Multimodal Workflows (test_multimodal_workflows.py) - 9 tests

• Vision + text model loading and initialization
• Image input processing with text completion
• Cross-modal context preservation
• Sequential text-only and multimodal requests
• Multimodal streaming responses
• Error handling with invalid inputs
• Multiple images in single request

4. Concurrent Scenarios (test_concurrent_scenarios.py) - 11 tests

• Concurrent completion and chat requests (multi-user simulation)
• Multi-turn conversations with context preservation
• Request slot management under load
• Concurrent streaming sessions
• LoRA adapter loading and switching during active sessions
• High concurrency stress testing
• Mixed request type coordination

Documentation

Comprehensive E2E README (tools/server/tests/e2e/README.md):

• Detailed test suite overview and organization
• Test execution examples and configuration
• Framework extension documentation with code examples
• Best practices for writing new E2E tests
• Troubleshooting guide
• CI integration guidelines

Testing Strategy

Model Selection

E2E tests use smaller models optimized for CI environments:

• Text Generation: tinyllama (stories260K.gguf) - Fast, small footprint
• Embeddings: bert-bge-small - Efficient embedding generation
• Multimodal: tinygemma3 - Compact vision+text model

CI Compatibility

• Designed for 4 vCPU GitHub runners
• Fast model downloads from HuggingFace
• Reasonable timeout configurations
• Slow tests marked with @pytest.mark.skipif(not is_slow_test_allowed())

Running the Tests

Run all E2E tests:

./tools/server/tests/tests.sh e2e/

Run specific test file:

./tools/server/tests/tests.sh e2e/test_pipeline_workflows.py

Run single test:

./tools/server/tests/tests.sh e2e/test_pipeline_workflows.py::test_basic_pipeline_workflow

Enable slow tests:

SLOW_TESTS=1 ./tools/server/tests/tests.sh e2e/

Implementation Highlights

PipelineTestProcess Class

from utils import PipelineTestProcess

pipeline = PipelineTestProcess()

# Test complete pipeline workflow
results = pipeline.test_full_pipeline({
    "model_hf_repo": "ggml-org/models",
    "model_hf_file": "tinyllamas/stories260K.gguf",
})

# Execute CLI commands
result = pipeline.run_cli_command(["-m", model_path, "-p", "Hello"])

# Run benchmarks
bench = pipeline.run_bench_command(model_path, ["-p", "8", "-n", "8"])

Example E2E Test

def test_concurrent_completion_requests(pipeline_process, e2e_small_model_config):
    """Test concurrent requests from multiple simulated users."""
    for key, value in e2e_small_model_config.items():
        if hasattr(pipeline_process, key):
            setattr(pipeline_process, key, value)
    
    pipeline_process.n_slots = 4
    pipeline_process.server_continuous_batching = True
    pipeline_process.start()
    
    tasks = [
        (pipeline_process.make_request, 
         ("POST", "/completion", {"prompt": p, "n_predict": 16}))
        for p in prompts
    ]
    
    results = parallel_function_calls(tasks)
    assert all([r.status_code == 200 for r in results])

Validation

• ✅ All 38 E2E tests discovered and collected successfully
• ✅ Sample tests verified to run correctly
• ✅ Python syntax validation passed
• ✅ Framework extensions maintain backward compatibility
• ✅ Existing unit tests remain unaffected

Benefits

1. Comprehensive Coverage: Tests complete user workflows beyond individual API endpoints
2. Real-world Scenarios: Validates concurrent usage, context management, and resource coordination
3. Tool Integration: First-class testing of CLI tools alongside server API
4. Multimodal Support: Dedicated testing for vision+text workflows
5. Extensible Framework: PipelineTestProcess provides foundation for future E2E tests
6. CI-Friendly: Optimized for automated testing with appropriate timeouts and model selection
7. Well-Documented: Comprehensive README with examples and best practices

Related Issues

Addresses Jira ticket: AT-104 - Implement comprehensive end-to-end test coverage for llama.cpp

Checklist

• Extended existing ServerProcess class without breaking functionality
• Created comprehensive E2E test suites covering all four main areas
• Maintained compatibility with existing pytest framework and fixtures
• Implemented proper resource management and cleanup
• Provided configurable model selection for different testing environments
• Included comprehensive documentation for E2E testing capabilities
• Tests are CI-compatible and use appropriate model sizes
• All tests collected successfully by pytest

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring