Doubt about transcription

[rafael844]

When I receive the below error, IPED tries to run again the audio or it goes to next one? The large wav2vec2 dataset uses almost all of the 6GB GPU RAM, but its still faster than our other 8GB GPU model. But I cant figure if its just ignoring those audios or not.

5:00:19.968 [pool-1-thread-41] INFO  iped.engine.task.transcript.RemoteWav2Vec2Service - Client 10.200.51.192:52246 - Receiving 12,436 bytes...
15:00:19.969 [pool-1-thread-41] INFO  iped.engine.task.transcript.RemoteWav2Vec2Service - Client 10.200.51.192:52246 - Received 12436 audio bytes to transcribe.
15:00:19.971 [pool-1-thread-45] WARN  iped.engine.task.transcript.RemoteWav2Vec2Service - Exception while transcribing
java.lang.RuntimeException: java.lang.RuntimeException: Transcription failed, returned: RuntimeError('CUDA out of memory. Tried to allocate 332.00 MiB (GPU 0; 5.99 GiB total capacity; 4.38 GiB already allocated; 0 bytes free; 5.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF')
        at iped.engine.task.transcript.Wav2Vec2TranscriptTask.lambda$transcribeAudio$0(Wav2Vec2TranscriptTask.java:217) ~[iped-engine-4.1.2.jar:?]
        at iped.engine.task.transcript.AbstractTranscriptTask.transcribeWavBreaking(AbstractTranscriptTask.java:198) ~[iped-engine-4.1.2.jar:?]
        at iped.engine.task.transcript.Wav2Vec2TranscriptTask.transcribeAudio(Wav2Vec2TranscriptTask.java:213) ~[iped-engine-4.1.2.jar:?]
        at iped.engine.task.transcript.RemoteWav2Vec2Service$1.run(RemoteWav2Vec2Service.java:287) [iped-engine-4.1.2.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.RuntimeException: Transcription failed, returned: RuntimeError('CUDA out of memory. Tried to allocate 332.00 MiB (GPU 0; 5.99 GiB total capacity; 4.38 GiB already allocated; 0 bytes free; 5.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF')
        at iped.engine.task.transcript.Wav2Vec2TranscriptTask.transcribeWavPart(Wav2Vec2TranscriptTask.java:243) ~[iped-engine-4.1.2.jar:?]
        at iped.engine.task.transcript.Wav2Vec2TranscriptTask.lambda$transcribeAudio$0(Wav2Vec2TranscriptTask.java:215) ~[iped-engine-4.1.2.jar:?]
        ... 6 more
15:00:20.020 [pool-1-thread-45] INFO  iped.engine.task.transcript.RemoteWav2Vec2Service - Client 10.200.51.192:52247 - Accepted connection.
15:00:20.021 [pool-1-thread-45] INFO  iped.engine.task.transcript.RemoteWav2Vec2Service - Client 10.200.51.192:52247 - Receiving 17,941 bytes...
15:00:20.023 [pool-1-thread-45] INFO  iped.engine.task.transcript.RemoteWav2Vec2Service - Client 10.200.51.192:52247 - Received 17941 audio bytes to transcribe.
15:00:20.095 [pool-1-thread-41] INFO  iped.engine.task.transcript.RemoteWav2Vec2Service - Client 10.200.51.192:52246 - Audio converted to wav.
15:00:20.142 [pool-1-thread-45] INFO  iped.engine.task.transcript.RemoteWav2Vec2Service - Client 10.200.51.192:52247 - Audio converted to wav.
15:00:20.999 [pool-1-thread-4

[lfcnassif]

The client log would give a better answer, but looking at the code, seems it is not retried, which is correct. That is an out of memory issue, your GPU loaded the model fine, but it doesn't have enough free memory to run inference on arbitrary audios. Retrying would trigger the error again and processing could never end. Generally just connection errors are retried.

I suggest you trying a smaller model or using a GPU with more memory.

Another possibility is tweaking the hard coded max audio chunk to be transcribed from 59s to a smaller value (MAX_WAV_TIME in AbstractTranscriptionTask class), maybe transcribing audios in smaller chunks may use less memory, but that would cause more transcription errors because more words could be cut. And you would need to compile the application from the source.

[rafael844]

Understood, thank you very much Nassif.