Cannot start the backend

[sN1xx]

cmd Log:

(venv) C:\HeartMula\HeartMuLa-Studio>python -m uvicorn backend.app.main:app --host 0.0.0.0 --port 8000
INFO: Started server process [7004]
INFO: Waiting for application startup.
[Models] All models found at C:\HeartMula\HeartMuLa-Studio\backend\models
[Config] Auto-detected: 4-bit quantization = True
[Config] Auto-detected: sequential offload = True
[Quantization] 4-bit quantization ENABLED - model will use ~3GB instead of ~11GB
[12GB GPU Mode] Using lazy codec loading for 12GB GPU
[Quantization] Loading HeartMuLa with 4-bit NF4 quantization...
torch_dtype is deprecated! Use dtype instead!
Failed to load Heartlib model: Torch not compiled with CUDA enabled
ERROR: Traceback (most recent call last):
File "C:\HeartMula\HeartMuLa-Studio\venv\Lib\site-packages\starlette\routing.py", line 694, in lifespan
async with self.lifespan_context(app) as maybe_state:
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\AppData\Local\Programs\Python\Python312\Lib\contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "C:\HeartMula\HeartMuLa-Studio\backend\app\main.py", line 55, in lifespan
await music_service.initialize()
File "C:\HeartMula\HeartMuLa-Studio\backend\app\services\music_service.py", line 835, in initialize
raise e
File "C:\HeartMula\HeartMuLa-Studio\backend\app\services\music_service.py", line 828, in initialize
self.pipeline = await loop.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\HeartMula\HeartMuLa-Studio\backend\app\services\music_service.py", line 830, in
lambda mp=model_path, v=version: self._load_pipeline_multi_gpu(mp, v)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\HeartMula\HeartMuLa-Studio\backend\app\services\music_service.py", line 693, in _load_pipeline_multi_gpu
pipeline = create_quantized_pipeline(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\HeartMula\HeartMuLa-Studio\backend\app\services\music_service.py", line 409, in create_quantized_pipeline
heartmula = HeartMuLa.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\HeartMula\HeartMuLa-Studio\venv\Lib\site-packages\transformers\modeling_utils.py", line 277, in _wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\HeartMula\HeartMuLa-Studio\venv\Lib\site-packages\transformers\modeling_utils.py", line 5051, in from_pretrained
) = cls._load_pretrained_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\HeartMula\HeartMuLa-Studio\venv\Lib\site-packages\transformers\modeling_utils.py", line 5435, in load_pretrained_model
caching_allocator_warmup(model, expanded_device_map, hf_quantizer)
File "C:\HeartMula\HeartMuLa-Studio\venv\Lib\site-packages\transformers\modeling_utils.py", line 6092, in caching_allocator_warmup
index = device.index if device.index is not None else torch_accelerator_module.current_device()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\HeartMula\HeartMuLa-Studio\venv\Lib\site-packages\torch\cuda_init.py", line 878, in current_device
lazy_init()
File "C:\HeartMula\HeartMuLa-Studio\venv\Lib\site-packages\torch\cuda_init.py", line 305, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

ERROR: Application startup failed. Exiting.

[LeeAeron]

AssertionError: Torch not compiled with CUDA enabled
Your Torch is not CUDA version! uninstall it and install torch+cu*** version

[sN1xx]

AssertionError: Torch not compiled with CUDA enabled Your Torch is not CUDA version! uninstall it and install torch+cu*** version

where can I find this?

[Vitek14]

You can google which cuda version is compatible with your videocard/driver. I don't know is it necessary to install cuda toolkit from nvidia now, but you should also do this for your video card: https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64

If you too lazy to do all of this, try this commands:

1. pip uninstall torch torchvision torchaudio
2. pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121. It will install torch with cuda version 12.1. And make sure that you are doing that in venv

[sN1xx]

You can google which cuda version is compatible with your videocard/driver. I don't know is it necessary to install cuda toolkit from nvidia now, but you should also do this for your video card: https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64

If you too lazy to do all of this, try this commands:

1. pip uninstall torch torchvision torchaudio
2. pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121. It will install torch with cuda version 12.1. And make sure that you are doing that in venv

it worked, but now i have the following error when decoding audio:
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul with transpose_mat1 1 transpose_mat2 0 m 512 n 372 k 32 mat1_ld 32 mat2_ld 32 result_ld 512 abcType 0 computeType 68 scaleType 0
Memory cleanup warning: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[fspecii]

The error Torch not compiled with CUDA enabled means you have the CPU-only version of PyTorch installed instead of the CUDA version.

To fix this:

1. First, uninstall the current PyTorch:

pip uninstall torch torchvision torchaudio

2. Then install PyTorch with CUDA support (for CUDA 12.1):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

3. Verify CUDA is available:

python -c "import torch; print(torch.cuda.is_available())"

This should print True.

Alternative: Use Docker (recommended)
Docker handles all dependencies automatically:

docker compose up

Let us know if reinstalling PyTorch with CUDA resolves the issue!

[fspecii]

Could you share more details about the error?

• What happens after the logs you posted? Does it crash or hang?
• What GPU do you have and how much VRAM?
• Any error messages in the terminal?

The logs show it's loading correctly with 4-bit quantization and lazy codec mode, which should work on 12GB GPUs. Need more info to diagnose the issue.

[Vitek14]

It looks like a model using too large gpu memory anyway. I tried using only heartlib and pulling HeartMuLa/heartlib#5
fixes problem on RTX 4060 Ti Windows 10 with cuda 12.1