Inference getting slower in large scale generation of multiple files

[jcvasquezc]

I am currently using the multilanguage model for large scale audio generation. I observed a linear time increase during continuous calls to the model.generate(...)

I think the error is related to cache accumulation in T3 and the alignmet_stream, but not be able to fix it.

Do anyone knows possible reasons for this and how to fix it?

[prathamesh-chavan-22]

How large is the audio generated i have generated 7mins using streaming https://github.com/prathamesh-chavan-22/chatterbox.git you can use my fork

[jcvasquezc]

The audios are short, between 10 and 20 seconds, but I am generating a batch of them (more than 10k)
What I have observed is that after several hundred generations, the process becomes slower, and the inference time increases linearly.

I will take a look to you fork
Thanks

[prathamesh-chavan-22]

Update i say clear gpu cache after every run, there dosent seem to be one in the lib, maybe it can help

[benHeid]

I think I found an issue that might explain it.

With each new inference, a new AlignmentStreamAnalzer is instantiated. Each of them is taking the tfmr model and registers a hook on some attention layers. With each inference, the number of hooks is growing. Since the hooked method is probably expensive since it copies data from the GPU to the CPU, this explains why it is getting that much slower...

[tobmi1]

I fixed the issue @benHeid mentioned in #455. You can try using my fork until the PR is merged