🎼 ChatMusician Extended - fine tuning and evaluation using Mozart and Scarlatti

💻 ChatMusician Code | 🤖 Chat Model | 🤖 Base Model

🔔News

Introduction

While ChatMusician is an impressive offering, we thought it might be interesting to see if the model could be extended by further training with baroque and classical material. We introduce ChatMusician Extended, an open-source LLM that extends ChatMusician.

Training Data

ChatMusician Extended is trained on corpus cultivated from Art der Fuge. An initial 157 Mozart and 555 Scarlatti compositions were used as separate fine tuning data sets followed by each composition being transposed into the other 11 keys resulting in 1286 and 6211 files respectively.

Training Procedure

Following the initial hyperparameters from ChatMusician the fine tuning was executed on resources from Paperspace's Gradient and Google Colab. Both NVidia A100 and A6000 resources were used.

Evaluation

1. Music understanding abilities are evaluated on the MusicTheoryBench.

Requirements

As with the initial project:

• Python 3.8 and above
• Pytorch 2.0 and above are recommended

For Windows and Linux environments:

• CUDA 11.4 and above are recommended
• Deepspeed 0.10 and above are recommended

For macOS and MLX evaluation:

• mlx 0.11.1 and above
• mlx-lm 0.10.0 and above are recommended

To avoid certain non-implementations errors in pytorch and macOS (RuntimeError: MPS does not support cumsum op with int64 input), install the latest pytorch nightly:

conda install pytorch-nightly::pytorch torchvision torchaudio -c pytorch-nightly

#  or

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu

# Check - or use pip list
conda list | grep torch
pytorch                   2.4.0.dev20240422        py3.12_0     pytorch-nightly
torchaudio                2.2.0.dev20240422        py312_cpu    pytorch-nightly
torchvision               0.19.0.dev20240422       py312_cpu    pytorch-nightly

Notably this also adds bfloat16 support to macOS/MPS

MLX notes

After fine tuning and merging, the resulting model will be need to be converted to a supported format for MLX. A local capable convert script has been included here as part of this repository.

Inference

web demo (with audio)

Then launch a gradio demo:

cd ChatMusician/
python model/infer/chatmusician_web_demo.py -c "m-a-p/ChatMusician" --server_port 8890

or macOS accelerated with mlx:

cd ChatMusician/
python model/infer/chatmusician_web_demo_mlx.py -c "m-a-p/ChatMusician" --server_port 8890

Prompt example:

inference locally

Training

Data Preprocessing

Limitations