bug

[sankar-mukherjee]

onnx export doesnot work too many bugs in this line

styletts2-inference/export_onnx.py

Line 27 in fba3485

torch.onnx.export(model, kwargs=model_inputs,

for example kwargs is not valid in onnx.export()

which torch version is this?

[sankar-mukherjee]

i modified the onnx.export to work with my current torch version, i created the onnx file. I dont know if its successful as I got this massage. I should expect a wav file after converstion according to this line.

styletts2-inference/export_onnx.py

Line 30 in fba3485

output_names=["output_wav"],

/opt/conda/lib/python3.11/site-packages/torch/nn/utils/weight_norm.py:134: FutureWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
WeightNorm.apply(module, name, dim)
/opt/conda/lib/python3.11/site-packages/torch/onnx/utils.py:2135: UserWarning: Provided key output_wav for dynamic axes is not a valid input/output name
warnings.warn(
/opt/conda/lib/python3.11/site-packages/torch/functional.py:666: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at /opt/conda/conda-bld/pytorch_1720538435607/work/aten/src/ATen/native/SpectralOps.cpp:873.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
[W813 16:56:56.257395514 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.260179756 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
/opt/conda/lib/python3.11/site-packages/torch/onnx/symbolic_opset9.py:4545: UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model.
warnings.warn(
[W813 16:56:56.288062552 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PadPacked type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.728357851 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.730428354 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.754341563 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PadPacked type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.794159718 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.796199485 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.820043167 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PadPacked type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.857832488 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.859796117 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 16:56:56.883724747 shape_type_inference.cpp:1998] Warning: The shape inference of prim::PadPacked type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
/opt/conda/lib/python3.11/site-packages/torch/onnx/symbolic_helper.py:1531: UserWarning: ONNX export mode is set to TrainingMode.EVAL, but operator 'instance_norm' is set to train=True. Exporting with train=True.
warnings.warn(
/opt/conda/lib/python3.11/site-packages/torch/onnx/_internal/jit_utils.py:314: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /opt/conda/conda-bld/pytorch_1720538435607/work/torch/csrc/jit/passes/onnx/constant_fold.cpp:179.)
_C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
/opt/conda/lib/python3.11/site-packages/torch/onnx/utils.py:739: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /opt/conda/conda-bld/pytorch_1720538435607/work/torch/csrc/jit/passes/onnx/constant_fold.cpp:179.)
_C._jit_pass_onnx_graph_shape_type_inference(
/opt/conda/lib/python3.11/site-packages/torch/onnx/utils.py:1244: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /opt/conda/conda-bld/pytorch_1720538435607/work/torch/csrc/jit/passes/onnx/constant_fold.cpp:179.)
_C._jit_pass_onnx_graph_shape_type_inference(
/opt/conda/lib/python3.11/site-packages/torch/onnx/utils.py:1738: UserWarning: The exported ONNX model failed ONNX shape inference. The model will not be executable by the ONNX Runtime. If this is unintended and you believe there is a bug, please report an issue at https://github.com/pytorch/pytorch/issues. Error reported by strict ONNX shape inference: [ShapeInferenceError] Inference error(s): (op_type:Transpose, node name: /text_encoder_1/Transpose_7): [TypeInferenceError] Invalid attribute perm {1, -1, 0}, input shape = {0, 0, 128}
(Triggered internally at /opt/conda/conda-bld/pytorch_1720538435607/work/torch/csrc/jit/serialization/export.cpp:1469.)
_C._check_onnx_proto(proto)
Exported!!!

[patriotyk]

Try to update pytorch to 2.5 or 2.6 version and use the latest onnx package

[sankar-mukherjee]

i use docker image pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel which has
torch 2.6.0+cu126

now i get different massage during conversion

/opt/conda/lib/python3.11/site-packages/torch/nn/utils/weight_norm.py:143: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
/opt/conda/lib/python3.11/site-packages/torch/onnx/utils.py:1824: UserWarning: Provided key output_wav for dynamic axes is not a valid input/output name
  warnings.warn(
/opt/conda/lib/python3.11/site-packages/torch/functional.py:709: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at /pytorch/aten/src/ATen/native/SpectralOps.cpp:873.)
  return _VF.stft(  # type: ignore[attr-defined]
[W813 19:20:55.622185446 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:55.625013558 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
/opt/conda/lib/python3.11/site-packages/torch/onnx/symbolic_opset9.py:4277: UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model. 
  warnings.warn(
[W813 19:20:55.652588678 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PadPacked type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:55.938438411 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:55.940415296 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:55.964726748 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PadPacked type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:55.003476633 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:55.005449782 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:55.029593775 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PadPacked type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:56.067338131 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:56.069282296 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PackPadded type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W813 19:20:56.092675655 shape_type_inference.cpp:1995] Warning: The shape inference of prim::PadPacked type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
/opt/conda/lib/python3.11/site-packages/torch/onnx/symbolic_helper.py:1459: UserWarning: ONNX export mode is set to TrainingMode.EVAL, but operator 'instance_norm' is set to train=True. Exporting with train=True.
  warnings.warn(
/opt/conda/lib/python3.11/site-packages/torch/onnx/_internal/jit_utils.py:308: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /pytorch/torch/csrc/jit/passes/onnx/constant_fold.cpp:178.)
  _C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
/opt/conda/lib/python3.11/site-packages/torch/onnx/utils.py:657: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /pytorch/torch/csrc/jit/passes/onnx/constant_fold.cpp:178.)
  _C._jit_pass_onnx_graph_shape_type_inference(
/opt/conda/lib/python3.11/site-packages/torch/onnx/utils.py:1127: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /pytorch/torch/csrc/jit/passes/onnx/constant_fold.cpp:178.)
  _C._jit_pass_onnx_graph_shape_type_inference(
Exported!!!

with this code from your readme i get this error massage

import nltk
nltk.download('punkt')
nltk.download('punkt_tab')
import onnxruntime
import soundfile
import numpy
from styletts2_inference.models import StyleTTS2Tokenizer
from styletts2_inference.models import StyleTTS2

from nltk.tokenize import word_tokenize
import phonemizer
global_phonemizer = phonemizer.backend.EspeakBackend(language='en-us', preserve_punctuation=True, with_stress=True, words_mismatch='ignore')

text = 'Hello, how are you?'
ps = global_phonemizer.phonemize([text])
ps = ' '.join(word_tokenize(ps[0]))

model = StyleTTS2(hf_path='patriotyk/StyleTTS2-LibriTTS', device='cpu')

styletts2_session = onnxruntime.InferenceSession("optimize_model/styletts2-inference/styletts.onnx")
tokenizer = StyleTTS2Tokenizer(hf_path='patriotyk/StyleTTS2-LJSpeech')
tokens = tokenizer.encode(ps)

wav, _ = styletts2_session.run(None, {'tokens': tokens.numpy(),
                                'speed': [1.0],
                                's_prev': model.predict_style_single(tokens)
                                })
soundfile.write('optimize_model/styletts2-inference/gennnerated_onnx.wav', wav, 24000)

/opt/conda/lib/python3.11/site-packages/torch/nn/utils/weight_norm.py:143: FutureWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
WeightNorm.apply(module, name, dim)
Traceback (most recent call last):
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/onnx_inference.py", line 26, in
's_prev': model.predict_style_single(tokens)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/styletts2_inference/models.py", line 649, in predict_style_single
s_pred = self.sampler(noise = self.noise,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/styletts2_inference/Modules/diffusion/sampler.py", line 423, in forward
x = self.sampler(noise, fn=fn, sigmas=sigmas, num_steps=num_steps)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/styletts2_inference/Modules/diffusion/sampler.py", line 360, in forward
x = self.step(x, fn=fn, sigma=sigmas[i], sigma_next=sigmas[i + 1]) # type: ignore # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/styletts2_inference/Modules/diffusion/sampler.py", line 343, in step
d = (x - fn(x, sigma=sigma)) / sigma
^^^^^^^^^^^^^^^^^^
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/styletts2_inference/Modules/diffusion/sampler.py", line 421, in
fn = lambda *a, **ka: self.diffusion.denoise_fn(*a, **{**ka, **kwargs}) # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/styletts2_inference/Modules/diffusion/sampler.py", line 159, in denoise_fn
x_pred = self.net(c_in * x_noisy, c_noise, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/styletts2_inference/Modules/diffusion/modules.py", line 180, in forward
return self.run(x, time, embedding=embedding, features=features)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/styletts2_inference/Modules/diffusion/modules.py", line 143, in run
mapping = self.get_mapping(time, features)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/StyleTTS2/optimize_model/styletts2-inference/styletts2_inference/Modules/diffusion/modules.py", line 131, in get_mapping
assert exists(features), assert_message
^^^^^^^^^^^^^^^^
AssertionError: context_features exists but no features provided

[patriotyk]

You cannot compute voice style for multispeaker model with 'predict_style_single' for multispeaker there is 'predict_style_multi'

[sankar-mukherjee]

thanks i got it working. I converted my custom model which uses hifigan. now when i generated audio via this onnx model i have two problems.

1. generated sound has weird pulse at the end.
2. audio seems to be pitch shifted.

Do you have any idea why this happens?

I found there are some changes in the model forward() function, may be this affects it!!

in your code

            # encode prosody
            en = (d.transpose(2, 1) @ pred_aln_trg.unsqueeze(0).to(self.device, dtype=dtype))

            F0_pred, N_pred = self.predictor.F0Ntrain(en, s.to(dtype=dtype))
            asr = (t_en @ pred_aln_trg.unsqueeze(0).to(self.device)).to(dtype=dtype)

            
            out = self.decoder(asr, F0_pred, N_pred, ref.squeeze().unsqueeze(0).to(dtype=dtype))
            if self.config.model_params.decoder.type == 'hifigan':
                out = out[:,:, 14500:]
            return out.squeeze()

and in original (https://github.com/yl4579/StyleTTS2/blob/main/Demo/Inference_LibriTTS.ipynb)

       # encode prosody
        en = (d.transpose(-1, -2) @ pred_aln_trg.unsqueeze(0).to(device))
        if model_params.decoder.type == "hifigan":
            asr_new = torch.zeros_like(en)
            asr_new[:, :, 0] = en[:, :, 0]
            asr_new[:, :, 1:] = en[:, :, 0:-1]
            en = asr_new

        F0_pred, N_pred = model.predictor.F0Ntrain(en, s)

        asr = (t_en @ pred_aln_trg.unsqueeze(0).to(device))
        if model_params.decoder.type == "hifigan":
            asr_new = torch.zeros_like(asr)
            asr_new[:, :, 0] = asr[:, :, 0]
            asr_new[:, :, 1:] = asr[:, :, 0:-1]
            asr = asr_new

        out = model.decoder(asr, 
                                F0_pred, N_pred, ref.squeeze().unsqueeze(0))
    
        
    return out.squeeze().cpu().numpy()[..., :-50] # weird pulse at the end of the model, need to be fixed later

also in your code (did you do it to fix weird pulse at the end?)

            if self.config.model_params.decoder.type == 'hifigan':
                pred_dur[0] = 30

Could you please comment on these changes? why some portion were removed? might this be affecting the audio?

[patriotyk]

My checkpoints don't have weird pulse at the end, but have at the beginning, So I did fix to cut beginning.