Skip to content

testing_strategy.md

Latest commit

 

History

History
208 lines (145 loc) · 8.77 KB

testing_strategy.md

File metadata and controls

208 lines (145 loc) · 8.77 KB

Testing Strategy

This document describes the testing strategy for AutoDeploy, covering the multi-tiered approach used to ensure quality and reliability.

Testing Philosophy

AutoDeploy uses a multi-tiered testing approach that balances fast feedback with comprehensive coverage:

┌─────────────────────────────────────────────────────────┐
│                    Dashboard                            │
│         (Broad model coverage + performance)            │
├─────────────────────────────────────────────────────────┤
│                 Integration Tests                       │
│            (Accuracy tests, CI-registered)              │
├─────────────────────────────────────────────────────────┤
│                  E2E Mini Tests                         │
│           (Compile + prompt workflows)                  │
├─────────────────────────────────────────────────────────┤
│                    Unit Tests                           │
│      (Component testing: patches, transforms, etc.)     │
└─────────────────────────────────────────────────────────┘
  • Unit Tests: Fast, isolated tests for individual components (patches, transforms, custom ops)
  • E2E Mini Tests: End-to-end workflows testing compile + prompt for unique model combinations
  • Integration Tests: Important accuracy tests registered individually in CI
  • Dashboard: Broad model coverage and performance testing across all supported models

Unit Tests

Unit tests verify individual components like patches, transformations, custom operations, and utilities.

Location

All unit tests are located in tests/unittest/auto_deploy/:

tests/unittest/auto_deploy/
├── _utils_test/                    # Shared test utilities
├── singlegpu/                      # Single GPU tests
│   ├── compile/                    # Compilation tests
│   ├── custom_ops/                 # Custom operations tests
│   ├── models/                     # Model-specific patch tests
│   ├── shim/                       # Executor/engine tests
│   ├── smoke/                      # E2E mini tests (see below)
│   ├── transformations/            # Graph transformation tests
│   └── utils/                      # Utility function tests
└── multigpu/                       # Multi-GPU tests
    ├── custom_ops/                 # Multi-GPU custom ops
    ├── smoke/                      # Multi-GPU E2E mini tests
    └── transformations/            # Multi-GPU transformation tests

CI Registration

Tests are automatically run in CI once registered. New test files and functions are picked up automatically if they are in an existing registered folder.

Tests are registered in tests/integration/test_lists/test-db/l0_*.yml files under the backend: autodeploy section:

backend: autodeploy
tests:
- unittest/auto_deploy/singlegpu/compile
- unittest/auto_deploy/singlegpu/custom_ops
- unittest/auto_deploy/singlegpu/models
- unittest/auto_deploy/singlegpu/shim
- unittest/auto_deploy/singlegpu/smoke
- unittest/auto_deploy/singlegpu/transformations
- unittest/auto_deploy/singlegpu/utils

Adding a New Folder

If you create a new folder (not just a new file in an existing folder), you must register it in the appropriate YAML files:

  1. Edit tests/integration/test_lists/test-db/l0_a30.yml (and other GPU-specific files as needed)
  2. Add the new folder path under the backend: autodeploy section
  3. Example: - unittest/auto_deploy/singlegpu/my_new_folder

Parallel Execution

Most unit tests run in parallel using pytest-xdist for faster execution. The exception is the smoke/ subfolders, which run sequentially (see E2E Mini Tests below).

E2E Mini Tests (Smoke Tests)

E2E mini tests verify complete end-to-end workflows including model compilation and prompt execution for unique model combinations.

Location

  • Single GPU: tests/unittest/auto_deploy/singlegpu/smoke/
  • Multi GPU: tests/unittest/auto_deploy/multigpu/smoke/

Purpose

These tests ensure that the full AutoDeploy pipeline works correctly for various model architectures and configurations:

  • test_ad_build_small_single.py - Tests multiple model configurations (Llama, Mixtral, Qwen, Phi-3, DeepSeek, Mistral, Nemotron)
  • test_ad_trtllm_bench.py - Benchmarking functionality
  • test_ad_trtllm_serve.py - Serving functionality
  • test_ad_speculative_decoding.py - Speculative decoding
  • test_ad_export_onnx.py - ONNX export functionality

Execution

Smoke tests are not executed in parallel to avoid resource contention during full model compilation and execution. They run sequentially within the CI pipeline.

Integration Tests

Integration tests cover important accuracy tests and other scenarios that require explicit CI registration.

Registration

Unlike unit tests (where new files in existing folders are auto-discovered), each individual integration test case must be explicitly registered in the CI YAML files.

Format: path/to/test_file.py::test_function_name[param_id]

Example from l0_a30.yml:

- accuracy/test_cli_flow.py::TestLlama3_1_8BInstruct::test_medusa_fp8_prequantized
- examples/test_multimodal.py::test_llm_multimodal_general[Qwen2-VL-7B-Instruct-pp:1-tp:1-float16-bs:1-cpp_e2e:False-nb:4]

Example: Adding an Accuracy Test

For reference, see PR #10717 which added a Nemotron 3 super accuracy test. The workflow is:

  1. Create the test function in the appropriate test file
  2. Register the specific test case in the relevant l0_*.yml file(s)
  3. Ensure the test passes locally before submitting

Location

Integration tests are typically located in:

  • examples/ - Model-specific integration tests
  • accuracy/ - Accuracy validation tests

Dashboard (Model Coverage Testing)

The dashboard provides broad model coverage and performance testing for all supported models in AutoDeploy.

Model Registry

Models are registered in examples/auto_deploy/model_registry/models.yaml. For detailed instructions, see the Model Registry README.

Format (Version 2.0)

The registry uses a flat list format with composable configurations:

version: '2.0'
description: AutoDeploy Model Registry - Flat format with composable configs
models:
- name: meta-llama/Llama-3.1-8B-Instruct
  yaml_extra: [dashboard_default.yaml, world_size_2.yaml]

- name: meta-llama/Llama-3.3-70B-Instruct
  yaml_extra: [dashboard_default.yaml, world_size_4.yaml, llama3_3_70b.yaml]

Key Concepts

  • Flat list: Models are in a single list (not grouped)
  • Composable configs: Each model references YAML config files via yaml_extra
  • Deep merging: Config files are merged in order (later files override earlier ones)

Configuration Files

Config files are stored in examples/auto_deploy/model_registry/configs/:

File Purpose
dashboard_default.yaml Baseline settings for all models
world_size_N.yaml GPU count configuration (1, 2, 4, or 8)
multimodal.yaml Vision + text models
demollm_triton.yaml DemoLLM runtime with Triton backend
Model-specific configs Custom settings for specific models

World Size Guidelines

World Size Model Size Range Example Models
1 < 2B params TinyLlama, Qwen 0.5B, Phi-4-mini
2 2-15B params Llama 3.1 8B, Qwen 7B, Mistral 7B
4 20-80B params Llama 3.3 70B, QwQ 32B, Gemma 27B
8 80B+ params DeepSeek V3, Llama 405B, Nemotron Ultra

Adding a New Model

  1. Add the model entry to models.yaml:
- name: organization/my-new-model-7b
  yaml_extra: [dashboard_default.yaml, world_size_2.yaml]
  1. For models with special requirements, create a custom config in configs/ and reference it:
- name: organization/my-custom-model
  yaml_extra: [dashboard_default.yaml, world_size_4.yaml, my_model.yaml]
  1. Validate with prepare_model_coverage_v2.py from the autodeploy-dashboard repository

The model will be automatically picked up by the dashboard testing infrastructure on the next run.