semantic training

[hyhzl]

@lucidrains i wanna train voice clone of chinese mandarin, and i plan to change soundstream to encodec instead. and i found chinese hubert model ,trained by weNetSpeech,10k+ hours. but k-means is not provided, which is required by project, the project require n_clusters and clusters_center, what should i do to continue to my exploration about voice clone? @lucidrains

[glide-the]

where you looking for this loading method, it can be found in the hubert_kmeans.py code
your target model, MERT, uses a similar method as HuBER
#62 (comment)

[LWprogramming]

Oh yeah you can pull in my personal_hacks branch and then you can load mert

[glide-the]

Oh, The latest branch of main does not have this code，What is the reason for not merging
https://github.com/LWprogramming/audiolm-pytorch/blob/personal_hacks/audiolm_pytorch/hubert_kmeans.py

    def __init__(
        self,
        checkpoint_path,
        kmeans_path,
        target_sample_hz = 16000,
        seq_len_multiple_of = None,
        output_layer = 9,
        use_mert=False,
    ):
        super().__init__()

        import sklearn
        #assert sklearn.__version__ == '0.24.0', 'scikit-learn needs to be exactly 0.24.0 - please install the correct version by running `pip install scikit-learn==0.24.0`'

        self.target_sample_hz = target_sample_hz
        self.seq_len_multiple_of = seq_len_multiple_of
        self.output_layer = output_layer

        self.use_mert = use_mert
        if not use_mert:
            model_path = Path(checkpoint_path)
            assert model_path.exists(), f'path {checkpoint_path} does not exist'
            print(checkpoint_path)
            checkpoint = torch.load(checkpoint_path)
            load_model_input = {checkpoint_path: checkpoint}
            model, *_ = fairseq.checkpoint_utils.load_model_ensemble_and_task(load_model_input)
            self.model = model[0]
        else:
            self.model = HubertModel.from_pretrained("m-a-p/MERT-v0") # is nn.Module
            # print(f"model is nn module? {isinstance(self.model, nn.Module)}")
            self.processor = Wav2Vec2Processor.from_pretrained("facebook/hubert-large-ls960-ft") # is not nn.Module
            # print(f"processor is nn module? {isinstance(self.processor, nn.Module)}")
            self.layer = output_layer

        kmeans_path = Path(kmeans_path)
        assert kmeans_path.exists(), f'path {kmeans_path} does not exist'

        self.model.eval()

        kmeans = joblib.load(kmeans_path)

        self.kmeans = kmeans

        self.register_buffer(
            'cluster_centers',
            torch.from_numpy(kmeans.cluster_centers_)
        )

[LWprogramming]

oh i hadnt gotten around to trying it over hubertkmeans yet haha, want to make a PR?

[LWprogramming]

Oh huh, I actually got to take a look at it and it seems like it disappeared for some reason, maybe because of a weird merge commit that wiped it out? Going to add it to personal_hacks now

[LWprogramming]

rescued, d1c28be

[glide-the]

where you looking for this loading method, it can be found in the hubert_kmeans.py code your target model, MERT, uses a similar method as HuBER #62 (comment)

You can also refer to the training code contributed by @LWprogramming
https://github.com/LWprogramming/audiolm-pytorch-training.git