Skip to content

README.md

Latest commit

 

History

History
116 lines (90 loc) · 4.19 KB

README.md

File metadata and controls

116 lines (90 loc) · 4.19 KB
Raw

LongCat-AudioDiT

State-of-the-art diffusion-based text-to-speech that operates directly in the waveform latent space. Uses Conditional Flow Matching with a DiT (Diffusion Transformer) backbone and a WAV-VAE audio codec at 24kHz. Supports zero-shot voice cloning with SOTA speaker similarity on the Seed benchmark.

Paper: LongCat-AudioDiT

Usage

Python API:

from mlx_audio.tts.utils import load

model = load("mlx-community/LongCat-AudioDiT-1B-bf16")

result = next(model.generate("Hello, this is a test of AudioDiT."))
audio = result.audio  # mlx array, 24kHz

Play audio directly:

from mlx_audio.tts.audio_player import AudioPlayer

player = AudioPlayer(sample_rate=24000)
result = next(model.generate("The quick brown fox jumps over the lazy dog."))
player.queue_audio(result.audio)
player.wait_for_drain()
player.stop()

Voice Cloning

Clone any voice using a reference audio sample and its transcript. Use guidance_method="apg" for best voice cloning quality:

result = next(model.generate(
    text="Today is warm turning to rain, with good air quality.",
    ref_audio="reference.wav",
    ref_text="Transcript of the reference audio.",
    guidance_method="apg",
    cfg_strength=4.0,
    steps=16,
))

Zero-Shot Generation (Chinese)

result = next(model.generate(
    text="今天晴暖转阴雨，空气质量优至良，空气相对湿度较低。",
    steps=16,
    cfg_strength=4.0,
))

Generation Parameters

Parameter Default Description
steps 16 Euler ODE solver steps. Higher = better quality, slower
cfg_strength 4.0 Classifier-free guidance strength
guidance_method "cfg" "cfg" for TTS, "apg" for voice cloning
seed 1024 Random seed for reproducibility
ref_audio None Reference audio for voice cloning (24kHz)
ref_text None Transcript of the reference audio

CLI

# Zero-shot TTS
python -m mlx_audio.tts.generate \
  --model mlx-community/LongCat-AudioDiT-1B-bf16 \
  --text "Hello, this is a test of AudioDiT." \
  --play

# Voice cloning
python -m mlx_audio.tts.generate \
  --model mlx-community/LongCat-AudioDiT-1B-bf16 \
  --text "Today is warm turning to rain." \
  --ref_audio reference.wav \
  --ref_text "Transcript of the reference audio." \
  --play

Available Models

Model Parameters Format Languages
mlx-community/LongCat-AudioDiT-1B-bf16 1B bf16 Chinese, English
mlx-community/LongCat-AudioDiT-1B-8bit 1B 8-bit Chinese, English
mlx-community/LongCat-AudioDiT-1B-6bit 1B 6-bit Chinese, English
mlx-community/LongCat-AudioDiT-1B-5bit 1B 5-bit Chinese, English
mlx-community/LongCat-AudioDiT-1B-4bit 1B 4-bit Chinese, English
mlx-community/LongCat-AudioDiT-1B-mxfp8 1B MXFP8 Chinese, English
mlx-community/LongCat-AudioDiT-1B-mxfp4 1B MXFP4 Chinese, English
mlx-community/LongCat-AudioDiT-1B-nvfp4 1B NVFP4 Chinese, English
mlx-community/LongCat-AudioDiT-3.5B-bf16 3.5B bf16 Chinese, English
mlx-community/LongCat-AudioDiT-3.5B-8bit 3.5B 8-bit Chinese, English
mlx-community/LongCat-AudioDiT-3.5B-6bit 3.5B 6-bit Chinese, English
mlx-community/LongCat-AudioDiT-3.5B-5bit 3.5B 5-bit Chinese, English
mlx-community/LongCat-AudioDiT-3.5B-4bit 3.5B 4-bit Chinese, English
mlx-community/LongCat-AudioDiT-3.5B-mxfp8 3.5B MXFP8 Chinese, English
mlx-community/LongCat-AudioDiT-3.5B-mxfp4 3.5B MXFP4 Chinese, English
mlx-community/LongCat-AudioDiT-3.5B-nvfp4 3.5B NVFP4 Chinese, English

Architecture

  • DiT backbone: dim=1536, depth=24, heads=24 with RoPE and AdaLN
  • WAV-VAE codec: latent_dim=64, 24kHz, runs in fp16
  • UMT5 text encoder: 768-dim, 12 layers with per-layer relative position bias
  • Conditional Flow Matching with Euler ODE solver

License

LongCat-AudioDiT weights and code are released under the MIT License.