Skip to content

Latest commit

 

History

History
438 lines (317 loc) · 16.5 KB

BENCHMARK.md

File metadata and controls

438 lines (317 loc) · 16.5 KB

ACE-Step 1.5 Benchmark & Profiling Guide

Language / 语言: English | 中文

Table of Contents

Overview

profile_inference.py is a comprehensive profiling & benchmarking tool for ACE-Step 1.5 inference. It measures end-to-end wall time, LLM planning time, DiT diffusion time, VAE decoding time, and more — across different devices, backends, and configurations.

Supported Modes

Mode Description
profile Profile a single generation run with detailed timing breakdown
benchmark Run a matrix of configurations (duration × batch × thinking × steps) and produce a summary table
tier-test Automatically test all GPU tiers by simulating different VRAM sizes via MAX_CUDA_VRAM
understand Profile the understand_music() API (audio → metadata extraction)
create_sample Profile the create_sample() API (inspiration / simple mode)
format_sample Profile the format_sample() API (caption + lyrics → structured metadata)

Supported Devices & Backends

Device Flag Notes
CUDA (NVIDIA) --device cuda Recommended. Auto-detected by default
MPS (Apple Silicon) --device mps macOS with Apple Silicon
CPU --device cpu Slow, for testing only
Auto --device auto Automatically selects best available (default)
LLM Backend Flag Notes
vLLM --lm-backend vllm Fastest on CUDA, recommended for NVIDIA
PyTorch --lm-backend pt Universal fallback, works everywhere
MLX --lm-backend mlx Optimized for Apple Silicon
Auto --lm-backend auto Selects best backend for device (default)

Quick Start

# Basic profile (text2music, default settings)
python profile_inference.py

# Profile with LLM thinking enabled
python profile_inference.py --thinking

# Run benchmark matrix
python profile_inference.py --mode benchmark

# Profile on Apple Silicon
python profile_inference.py --device mps --lm-backend mlx

# Profile with cProfile function-level analysis
python profile_inference.py --detailed

Profiling Modes

1. profile — Single Run Profiling

Runs a single generation with detailed timing breakdown. Includes optional warmup and cProfile.

python profile_inference.py --mode profile

What it measures:

  • Total wall time (end-to-end)
  • LLM planning time (token generation, constrained decoding, CFG overhead)
  • DiT diffusion time (per-step and total)
  • VAE decode time
  • Audio save time

Options for this mode:

Flag Description
--no-warmup Skip warmup run (includes compilation overhead in measurement)
--detailed Enable cProfile function-level analysis
--llm-debug Deep LLM debugging (token count, throughput)
--thinking Enable LLM Chain-of-Thought reasoning
--duration <sec> Override audio duration
--batch-size <n> Override batch size
--inference-steps <n> Override diffusion steps

2. benchmark — Configuration Matrix

Runs a matrix of configurations and produces a summary table. Automatically adapts to GPU memory limits.

python profile_inference.py --mode benchmark

Default matrix:

  • Durations: 30s, 60s, 120s, 240s (clamped by GPU memory)
  • Batch sizes: 1, 2, 4 (clamped by GPU memory)
  • Thinking: True, False
  • Inference steps: 8, 16

Output example:

Duration   Batch   Think   Steps   Wall(s)    LM(s)      DiT(s)     VAE(s)     Status
--------------------------------------------------------------------------------------------------------------------------
30         1       False   8       3.21       0.45       1.89       0.52       OK
30         1       True    8       5.67       2.91       1.89       0.52       OK
60         2       False   16      12.34      0.48       9.12       1.85       OK
...

Save results to JSON:

python profile_inference.py --mode benchmark --benchmark-output results.json

3. understand — Audio Understanding Profiling

Profiles the understand_music() API which extracts metadata (BPM, key, time signature, caption) from audio codes.

python profile_inference.py --mode understand
python profile_inference.py --mode understand --audio-codes "your_audio_codes_string"

4. create_sample — Inspiration Mode Profiling

Profiles the create_sample() API which generates a complete song blueprint from a simple text query.

python profile_inference.py --mode create_sample
python profile_inference.py --mode create_sample --sample-query "a soft Bengali love song"
python profile_inference.py --mode create_sample --instrumental

5. format_sample — Metadata Formatting Profiling

Profiles the format_sample() API which converts caption + lyrics into structured metadata.

python profile_inference.py --mode format_sample

6. tier-test — Automated GPU Tier Testing

Automatically simulates different GPU VRAM sizes using MAX_CUDA_VRAM and runs a generation test at each tier. This is the recommended way to validate that all GPU tiers work correctly after modifying acestep/gpu_config.py.

# Test all tiers (4, 6, 8, 12, 16, 20, 24 GB)
python profile_inference.py --mode tier-test

# Test specific VRAM sizes
python profile_inference.py --mode tier-test --tiers 6 8 16

# Test with LM enabled (where the tier supports it)
python profile_inference.py --mode tier-test --tier-with-lm

# Quick test: skip torch.compile for non-quantized tiers
python profile_inference.py --mode tier-test --tier-skip-compile

What it validates per tier:

  • Correct tier detection and GPUConfig construction
  • Model initialization (DiT, VAE, Text Encoder, optionally LM)
  • A short generation run (30s duration, batch=1) completes without OOM
  • Adaptive VAE decode fallback (GPU → CPU offload → full CPU)
  • VRAM usage stays within the simulated limit

Output example:

TIER TEST RESULTS
====================================================================================================
  VRAM    Tier       LM      Duration   Status    Peak VRAM    Notes
  ──────────────────────────────────────────────────────────────────────────────
  4GB     tier1      —       30s        ✅ OK     3.8GB        VAE decoded on CPU
  6GB     tier2      —       30s        ✅ OK     5.4GB        Tiled VAE chunk=256
  8GB     tier4      0.6B    30s        ✅ OK     7.2GB        vllm backend
  12GB    tier5      1.7B    30s        ✅ OK     10.8GB       vllm backend
  16GB    tier6a     1.7B    30s        ✅ OK     14.5GB       offload enabled
  20GB    tier6b     1.7B    30s        ✅ OK     17.2GB       no offload
  24GB    unlimited  4B      30s        ✅ OK     21.3GB       full models on GPU

Note: tier-test mode uses torch.cuda.set_per_process_memory_fraction() to enforce a hard VRAM cap, making simulations realistic even on high-end GPUs (e.g., A100 80GB).

Boundary Testing

Use --tier-boundary to find the minimum VRAM tier at which INT8 quantization and CPU offload can be safely disabled. For each tier, up to three configurations are tested:

  1. default — tier's standard settings
  2. no-quant — quantization disabled, offload unchanged
  3. no-offload — no quantization AND no CPU offload
# Run boundary tests across all tiers
python profile_inference.py --mode tier-test --tier-boundary

# Boundary test with LM enabled
python profile_inference.py --mode tier-test --tier-boundary --tier-with-lm

# Save boundary results to JSON
python profile_inference.py --mode tier-test --tier-boundary --benchmark-output boundary_results.json

The output includes a Boundary Analysis summary showing the minimum tier for each capability.

Batch Size Boundary Testing

Use --tier-batch-boundary to find the maximum safe batch size for each tier. For each tier, the tool progressively tests batch sizes 1, 2, 4, 8 (stopping at first OOM) with both LM-enabled and LM-disabled configurations:

# Run batch boundary tests
python profile_inference.py --mode tier-test --tier-batch-boundary --tier-with-lm

# Test specific tiers
python profile_inference.py --mode tier-test --tier-batch-boundary --tier-with-lm --tiers 8 12 16 24

The output includes a Batch Boundary Summary showing the maximum successful batch size per tier for both with-LM and without-LM configurations.

CLI Reference

Device & Backend

Flag Default Description
--device auto Device: auto / cuda / mps / cpu
--lm-backend auto LLM backend: auto / vllm / pt / mlx

Model Paths

Flag Default Description
--config-path acestep-v15-turbo DiT model config
--lm-model acestep-5Hz-lm-1.7B LLM model path

Hardware Options

Flag Default Description
--offload-to-cpu off Offload models to CPU when not in use
--offload-dit-to-cpu off Offload DiT to CPU when not in use
--quantization none Quantization: int8_weight_only / fp8_weight_only / w8a8_dynamic

Generation Parameters

Flag Default Description
--duration from example Audio duration in seconds
--batch-size from example Batch size
--inference-steps from example Diffusion inference steps
--seed from example Random seed
--guidance-scale 7.0 CFG guidance scale for DiT

LLM / CoT Parameters

Flag Default Description
--thinking off Enable LLM Chain-of-Thought reasoning
--use-cot-metas off LLM generates music metadata via CoT
--use-cot-caption off LLM rewrites/formats caption via CoT
--use-cot-language off LLM detects vocal language via CoT
--use-constrained-decoding on FSM-based constrained decoding
--no-constrained-decoding — Disable constrained decoding
--lm-temperature 0.85 LLM sampling temperature
--lm-cfg-scale 2.0 LLM CFG scale

Profiling Options

Flag Default Description
--mode profile Mode: profile / benchmark / tier-test / understand / create_sample / format_sample
--no-warmup off Skip warmup run
--detailed off Enable cProfile function-level analysis
--llm-debug off Deep LLM debugging (token count, throughput)
--benchmark-output none Save benchmark results to JSON file

Tier-Test Options

Flag Default Description
--tiers 4 6 8 12 16 20 24 VRAM sizes (GB) to simulate
--tier-with-lm off Enable LM initialization on tiers that support it
--tier-skip-compile off Skip torch.compile for faster iteration on non-quantized tiers
--tier-boundary off Test each tier with no-quant and no-offload variants to find minimum capability boundaries
--tier-batch-boundary off Test each tier with batch sizes 1, 2, 4, 8 to find maximum safe batch size

Input Options

Flag Default Description
--example example_05.json Example JSON from examples/text2music/
--task-type text2music Task: text2music / cover / repaint / lego / extract / complete
--reference-audio none Reference audio path (for cover/style transfer)
--src-audio none Source audio path (for audio-to-audio tasks)
--sample-query none Query for create_sample mode
--instrumental off Generate instrumental music (for create_sample)
--audio-codes none Audio codes string (for understand mode)

Examples

Compare Devices

# NVIDIA GPU
python profile_inference.py --device cuda --lm-backend vllm

# Apple Silicon
python profile_inference.py --device mps --lm-backend mlx

# CPU baseline
python profile_inference.py --device cpu --lm-backend pt

Compare LLM Models

# Lightweight (0.6B)
python profile_inference.py --lm-model acestep-5Hz-lm-0.6B

# Default (1.7B)
python profile_inference.py --lm-model acestep-5Hz-lm-1.7B

# Large (4B)
python profile_inference.py --lm-model acestep-5Hz-lm-4B

Thinking vs No-Thinking

# Without thinking (faster)
python profile_inference.py --mode benchmark

# With thinking (better quality, slower)
python profile_inference.py --thinking --use-cot-metas --use-cot-caption

Low-VRAM Profiling

# Offload + quantization
python profile_inference.py --offload-to-cpu --quantization int8_weight_only --lm-model acestep-5Hz-lm-0.6B

Full Benchmark Suite

# Run full benchmark matrix and save results
python profile_inference.py --mode benchmark --benchmark-output benchmark_results.json

# Then inspect the JSON
cat benchmark_results.json | python -m json.tool

Function-Level Profiling

# Enable cProfile for detailed function-level analysis
python profile_inference.py --detailed --llm-debug

Understanding Output

Time Costs Breakdown

The profiler prints a detailed breakdown of where time is spent:

TIME COSTS BREAKDOWN
====================================================================================================
  Component                          Time (s)       % of Total
  ─────────────────────────────────────────────────────────────
  LLM Planning (total)               2.91           45.2%
    ├─ Token generation              2.45           38.1%
    ├─ Constrained decoding          0.31            4.8%
    └─ CFG overhead                  0.15            2.3%
  DiT Diffusion (total)              1.89           29.4%
    ├─ Per-step average              0.24            —
    └─ Steps                         8               —
  VAE Decode                         0.52            8.1%
  Audio Save                         0.12            1.9%
  Other / Overhead                   0.99           15.4%
  ─────────────────────────────────────────────────────────────
  Wall Time (total)                  6.43          100.0%

Key Metrics

Metric Description
Wall Time End-to-end time from start to finish
LM Total Time Time spent in LLM planning (token generation + parsing)
DiT Total Time Time spent in diffusion (all steps combined)
VAE Decode Time Time to decode latents to audio waveform
Tokens/sec LLM token generation throughput (with --llm-debug)

Tips & Best Practices

  1. Always include warmup (default) — The first run includes JIT compilation and memory allocation overhead. Warmup ensures measurements reflect steady-state performance.

  2. Use --benchmark-output to save results as JSON for later analysis or comparison across hardware.

  3. Compare with thinking off vs on — Thinking mode significantly increases LLM time but may improve generation quality.

  4. Test with representative durations — Short durations (30s) are dominated by LLM time; long durations (240s+) are dominated by DiT time.

  5. GPU memory auto-adaptation — The benchmark mode automatically clamps durations and batch sizes to what your GPU can handle, using the adaptive tier system in acestep/gpu_config.py.

  6. Use --detailed sparingly — cProfile adds overhead; use it only when investigating function-level bottlenecks.

  7. Use tier-test for regression testing — After modifying GPU tier configs, run --mode tier-test to verify all tiers still work correctly. This is especially important when changing offload thresholds, duration limits, or LM model availability.

  8. Simulate low VRAM realistically — When using MAX_CUDA_VRAM, the system enforces a hard VRAM cap via set_per_process_memory_fraction(), so OOM errors during simulation reflect real behavior on consumer GPUs.