😎 基于InternLM的“黄梅戏艺术家”

[GuoYiFantastic]

1.灵感 😀

期末周之前就给寒假立了个flag：做一个家乡话的NLP模型。后恰赶上InternLM训练营的东风，于是想利用书生·浦语大模型全链路开源体系和上海人工智能实验室的算力支持，做一个家乡话的大模型。但是调研发现，国内在人工智能&方言领域已经有诸多团队做得很好。于是，我选择了基于InternLM的“黄梅戏艺术家” 😎。

2.功能介绍

2.1 唱戏

输入：键入一句话；

输出：一段语音，用黄梅戏的腔调唱出这句话。

2.2 个性化唱腔（待）

3.工作

3.1 针对InternLM的微调策略

3.1.1 soundfile库

• 为了适配txt2txt的InternLM，将“艺术家”的直接输出设定为用数组表示的数字信号，然后再利用soundfile库，将数字信号转为模拟信号（.wav）：

import numpy as np
import soundfile as sf

def array_to_wav(input_array, output_file_path, samplerate=44100):
    """
    Converts an array to a .wav file.

    Parameters:
    - input_array: The input audio data as a numpy array.
    - output_file_path: The path to the output .wav file.
    - samplerate: The sample rate of the audio file in Hz. Defaults to 44100.
    """
    # Ensure the input array is a numpy array
    if not isinstance(input_array, np.ndarray):
        input_array = np.array(input_array)

    # Write the array to a .wav file
    sf.write(output_file_path, input_array, samplerate)

# Example usage
input_array = np.random.uniform(low=-1.0, high=1.0, size=(44100,))  # 1 second of white noise
output_file_path = "output.wav"
array_to_wav(input_array, output_file_path)

print(f"Array has been successfully written to {output_file_path}")

• 使用chatGPT生成脚本：将.wav文件转换成数组。

Write a python file for me. using soundfile library. input file catalog path is PATH.
Step1: Go through all the files under the PATH, all input files are .wav. 
Step2: Transform each input file into an array.
Step3: The output file is .jsonL. It looks like:
[{
    "conversation":[
        {
            "system": "xxx",
            "input": "xxx",
            "output": "xxx"
        }
    ]
},
{
    "conversation":[
        {
            "system": "xxx",
            "input": "xxx",
            "output": "xxx"
        }
    ]
}]
Each file maps to a "conversation".
Step4: 
All "system" value changes to "你是一个黄梅戏艺术家，可以把一句话用黄梅戏的腔调唱出来。"  
All "input" value changes to the name of the input file. For example, the "input" value will be "树上的鸟儿成双对" if the input file is 树上的鸟儿成双对.wav
All "output" value changes to the corresponding array.

wav2array.py:

import os
import json
import soundfile as sf

# Define the input and output paths
PATH = "your/input/folder/path"
output_file = "output.jsonl"

def process_audio_files(input_path, output_path):
    conversations = []
    
    # Step 1: Iterate through all .wav files in the given directory
    for root, dirs, files in os.walk(input_path):
        for file in files:
            if file.endswith(".wav"):
                # Step 2: Transform each input file into an array
                file_path = os.path.join(root, file)
                data, samplerate = sf.read(file_path)
                
                # Step 3 & 4: Construct the conversation object with transformed values
                conversation = {
                    "conversation": [
                        {
                            "system": "你是一个黄梅戏艺术家，可以把一句话用黄梅戏的腔调唱出来。",
                            "input": os.path.splitext(file)[0],
                            "output": data.tolist()
                        }
                    ]
                }
                
                conversations.append(conversation)
    
    # Write conversations to a .jsonl file
    with open(output_path, 'w', encoding='utf-8') as f:
        for conversation in conversations:
            f.write(json.dumps(conversation, ensure_ascii=False) + '\n')

# Execute the function with the specified PATH and output file
process_audio_files(PATH, output_file)

print(f"Processed audio files from {PATH} and output to {output_file}")

构建数据集和测试集：split2train_and_test.py

• XTuner微调 参考
  安装依赖、模型下载、配置文件、开始微调、转换合并、部署测试。

3.2 构建数据集

3.2.1 Spleeter提取纯人声

也可以使用这个宝藏网站click

3.2.2 人声切割

将人声按照歌词切割

欲获得大数据集，亲耳听歌、亲手切割.mp3文件显然不可能，妙用.lrc文件。浅析.lrc文件

Write a python function named cutmusic(), and the function has two inputs: path of .wav file and path of .lrc file. It cuts .wav file into some small .wav files according to the .lrc file. 
For example, if the input is 背新娘.wav and 背新娘.lrc, the function will cut .wav between [00:11.03] and [00:13.71] of the .wav file to generate a new .wav file named 弟矮小喂弟矮小.wav, and cut between [00:13.71] and [00:16.75] to generate 嫂子长哎嫂子长.wav and so on...... until end of the lyric.
Last but not least, there are many .wav files in ./vocals/, and many .lrc files in ./lyric/ . If there exists musicname.wav in ./vocals, there must exists musicname.lrc in ./lyric/ . 
Use cutmusic() function on every .wav file in ./vocal/ , and save all small new-generated .wav in ./cuts/

[lyyilin]

哇塞，这个创意好棒！！！

[GuoYiFantastic]

谢谢！我现在的困难是：如何利用咱们这个全链路开源体系做TTS（text2speech）