performance improvement

[sankar-mukherjee]

thanks for this repo.

I am wondering how much faster this onnx version is compared to the original pytorch one? do you have any stats?
Also is it possible to convert to TensorRT format?

[patriotyk]

ONNX version is only faster on CPU in 30-40%. Cuda has the same speed. TensorRT is not possible without changing model and retraining it. But Yesterday, I made changes(on my local) to be compatible with TensorRT, I ported only model without decoder, and it runs 3-4 times faster.

[sankar-mukherjee]

i tried to convert the model to tensorRT with pytorch -> onnx -> tensorRt but failed with negative dimension error. onnx conversion was ok, but tensorRT failed.

after lot of debugging it seems this function is problematic

pred_aln_trg = self.onnx_alignment(pred_dur, input_lengths[0])
in this function specifically

seq_length = pred_dur.sum().to(dtype=torch.int64)
...
positions = torch.arange(seq_length, device=device)...

here its creating a tensor (positions) whose shape depends on seq_length, which is calculated from data (pred_dur). This data-dependent shape breaks TensorRT’s symbolic shape engine, causing the negative dimension error.

I am just wondering did the face the similar situation? if yes can you suggest some solutions?

[patriotyk]

Yes, but this is not only the problem.
you can replace this method with tensort compatible:

    def onnx_alignment(self, pred_dur, max_length):
        pred_dur = pred_dur.to(dtype=torch.int64)
        seq_length = pred_dur.sum().unsqueeze(-1).item()
        device = pred_dur.device
                
        clipped_dur = torch.clamp(pred_dur, 0, seq_length).to(dtype=torch.int64).reshape(-1)
        
        zero_tensor = torch.zeros(1, device=device, dtype=torch.int64)
        
        cumsum_dur = torch.cumsum(clipped_dur[:-1], dim=0).to(dtype=torch.int64)
        prefix_sum = torch.cat([zero_tensor, cumsum_dur]).to(dtype=torch.int64).reshape(-1) 

        positions = torch.arange(seq_length, device=device).to(dtype=torch.int64)

        positions = positions.unsqueeze(0)
        positions = positions.expand(max_length, -1)
        
        start_positions = prefix_sum.unsqueeze(1)
        
        end_positions = torch.min(
            (start_positions + clipped_dur.unsqueeze(1)),
            start_positions.new_full(start_positions.shape, seq_length, dtype=torch.float)
        )

        
        mask = (positions >= start_positions) & (positions < end_positions)

        return mask.float()
    

But there is also problem with AdainResBlk1d and there is two copies of this module, so you can replace self.pool = weight_norm(nn.ConvTranspose1d(dim_in, dim_in, kernel_size=3, stride=2, groups=dim_in, padding=1, output_padding=1)) in that module with:

self.pool = nn.Sequential(
                nn.Upsample(scale_factor=2, mode='nearest'),
                        weight_norm(
                            nn.Conv1d(
                                dim_in, dim_in,
                                kernel_size=3,
                                padding=1,
                                groups=dim_in
                    )
                )
            )

But you would need to retrain model then. I don't know is it enoght for decoder because I have not tried decoder yet.

[sankar-mukherjee]

I tried with torch-tensorrt lib to convert the model. But after many many attempts I failed to convert it. There is always some error.

I am just wondering Do you plan to share the tensorRT compatible code here?

[patriotyk]

torch-tensorrt doesn't work because it conevrts using new dynamo compiler that dosen't work with this model and requires more changes. But you can convert this model to onnx then to tensorrt or just use onnx-runtime with TensorRT Execution Provider, it works, I tested it.
I cannot apply those changes here because it will broke original styletts2 weights. I plan to push them to separate repo, but I am also working with another improvements so it takes time.

[sankar-mukherjee]

can you please list the commands and version of the libs in order to use styletts2_session = onnxruntime.InferenceSession(onnx_path, providers=["TensorrtExecutionProvider"]) ?

i converted the model to onnx format. Inference with CUDAExecutionProvider runs fine. But when i am running the onnx model with TensorrtExecutionProvider i get many errors. i had to remove the pack_padded_sequence and pad_packed_sequence and many little changes regards to squeeze() operations in the code. but still there are many errors.

I am using g4dn.2xlarge aws instance with nvcr.io/nvidia/tensorrt:25.06-py3 docker image and inside container installed the onnxruntime-gpu==1.22.0 lib.