Thoughts of version 3.0.0

[yqzhishen]

Dev log: #245

If you have thoughts, suggestions or feature suggestions, please comment and discuss here

[AnAndroNerd]

From @ ariikamusic on discord.
"some like to train tension together with multispeaker, maybe it could be beneficial if the min max tension range is calculated per speaker. Since a soft and airy speaker is definitely not going to reach the default +10.0 logit."

[XHR0ME]

I'm really wanna more flexible retakes for autopich and timbre like in synthV. Is that possible?

[yqzhishen]

Pitch and variance parameters already have retaking in version 2, but OpenUtau hasn't implemented them
We will hopefully bring retaking to acoustic models in version 3

[KakaruHayate]

Note that in LYNXNet2, the conditioner_projection was changed from Conv1d to Linear.
When exporting the ONNX model, the Linear layer gets decomposed into MatMul and Add nodes, which invalidates the condition cache.

Although LYNXNet2 only requires calculating condition injection once per loop, this still becomes a worthwhile optimization target considering we often need 20 loops in practice.🤔

[JustASpoopyAce]

I recently considered that as of current, it is required to pitch extract data each time you're pre-processing data to train with. However, as many developers will end up training using the same data several times to test different combinations of features, in theory significant time and processing power could be saved by having a way to save this pitch extracted data after it's been extracted once.

I then realised that currently there's already a way to both A) convert your .csv file into .ds files, and B) the ability to train variance with just .ds files.

I think being able to train acoustic with wav files and their equivalent .ds files would be incredibly useful, as the pitch is already easily able to be embedded in these files and therefore reused in future pre-processing without needing to extract again.

[yqzhishen]

Acoustic models target acoustic features, and these features are relatively much more sensitive to the pitch (take the vocoder into consideration as well). If user-edited ds files are also allowed, there can be problems if the f0 mismatches.

Ensuring data consistency needs much more effort. I will consider but probably won't do this.

[JustASpoopyAce]

That's super fair, but it feels a little odd to me the idea that people making the deliberate choice to tamper with the extracted pitch in anyway should be of concern or responsibility to developers or even end users, especially considering it's also currently already very easy to overwrite the pitch by extracting into the .ds files again if it were to somehow happen by accident?
As long as users aren't deliberately choosing to directly tamper with the files after the pitch is extracted and stored within the .ds file; I don't see how the pitch could get altered from the original extraction in the first place in any reasonable situation. While that happening would be unideal, it not only would require - to my knowledge - the active choice to do so, but also the knowledge to even do it in the first place. This would mean the person doing it would also know of the issues and problems that would arise of having f0 mismatches.

As well, I feel like it would be quite reasonable to put the burden of "make sure your data matches the pitch" onto the person curating the dataset in the first place, especially considering this as a concept would in practice be something done to save time long-term.

I feel like in practice the concern doesn't actually apply to any "good-faith" situations I can think of? While it could be of active benefit to people in general; especially so those who'd want to use pitch extractors that are more significantly more accurate at the cost of longer processing time, like Harvest.

If I'm missing something please do correct me however.

[MLo7Ghinsan]

I'm thinking of how it may be useful for accent conversion to exist similar to how we can already blend speakers but this specifically blends/convert the accents alone without changing the timbre. It could be useful for helping a speaker with heavy accent to sounds more natural or pronounce something more accurately.

[AnAndroNerd]

What diff_accelerator is recommend to use with lynxnet2? I've had terrible experience with UniPC when training duration and I'd like to see if that's something that's known by the devs and if it applies to more types of models.

[KakaruHayate]

Starting from April 17, 2024, regardless of how you configure the solver, the Euler method will ultimately be used in actual deployment.
This means any perceived differences from adjusting this option are illusory.

[2tuka4i]

Although I have followed the instructions to complete the session, I get several errors when exporting ONNX to Convert model to ONNX format.

I got the error cp: cannot stat '/content/drive/MyDrive/Diffsinger/config.yaml': No such file or directory (same for acoustic) and after rewriting the code, the error disappeared.

But then the following error occurred:
converting acoustic to onnx...
| found ckpt by name: Diffsinger_acoustic
| Hparams chains: []
| Hparams:
exp_name: Diffsinger_acoustic, infer: True, work_dir: /content/drive/MyDrive/Diffsinger,
| Exporter: <class 'deployment.exporters.acoustic_exporter.DiffSingerAcousticExporter'>
Traceback (most recent call last):
File "/content/DiffSinger/scripts/export.py", line 302, in
main()
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1442, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1363, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1830, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1226, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 794, in invoke
return callback(*args, **kwargs)
File "/content/DiffSinger/scripts/export.py", line 144, in acoustic
exporter = DiffSingerAcousticExporter(
File "/content/DiffSinger/deployment/exporters/acoustic_exporter.py", line 33, in init
self.spk_map: dict = self.build_spk_map()
File "/content/DiffSinger/basics/base_exporter.py", line 27, in build_spk_map
if hparams['use_spk_id']:
KeyError: 'use_spk_id'

There doesn't seem to be any mistake in the folder structure.
I'm not very knowledgeable about programming.
Please help.

[2tuka4i]

The folder contains a ckpt file and a config.yaml file, and the contents of these files look fine.

[yqzhishen]

First, make sure you are directly using this repository without any third-party mods or tools.

Then create an issue and provide further information, including screenshots of your folder structure, your configuration file and your command to run exporting.

[2tuka4i]

I forgot to tell you, but I'm studying using Google Colab notes.
In that case, shouldn't I be asking my question here?

[yqzhishen]

This discussion isn't for bug reporting, and bug reporting should go to issues. If you are using Colab notebook written by someone else, you'd better ask the author who wrote it.

[2tuka4i]

Sorry.
Thank you for your answer.

[IAmREGE]

How about IPA phoneme-based support?
An example is: Enter pinyin “jiu”, and you could get one of [tɕjoʊ] (Mandarin), [tɕjʊ] (similar to pinyin ji-wu), [dʑɯ] (similar to じゅ, romaji ju / zyu), [dzju] (similar to “ds you” in “cards you get” in English) and more, based on your choice.