Bug on log_mel_spectogram() when I use model.transcribe on Jetson AGX Xavier

[juansebashr]

Hi!

I'm testing the examples of the repo on a Jetson AGX Xavier, but I encounter the following error:

/workspace/rover/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:213: sampleMultinomialOnce: block: [0,0,0], thread: [872,0,0] Assertion `!at::_isnan(val)` failed.
/workspace/rover/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:213: sampleMultinomialOnce: block: [0,0,0], thread: [873,0,0] Assertion `!at::_isnan(val)` failed.
/workspace/rover/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:213: sampleMultinomialOnce: block: [0,0,0], thread: [874,0,0] Assertion `!at::_isnan(val)` failed.
/workspace/rover/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:213: sampleMultinomialOnce: block: [0,0,0], thread: [875,0,0] Assertion `!at::_isnan(val)` failed.
/workspace/rover/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:213: sampleMultinomialOnce: block: [0,0,0], thread: [876,0,0] Assertion `!at::_isnan(val)` failed.
/workspace/rover/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:213: sampleMultinomialOnce: block: [0,0,0], thread: [877,0,0] Assertion `!at::_isnan(val)` failed.

Traceback (most recent call last):
  File "/workspace/rover/scripts/bench_whisper.py", line 27, in <module>
    result = model.transcribe("/workspace/rover/scripts/outdoors.wav")
  File "/usr/local/lib/python3.10/dist-packages/whisper/transcribe.py", line 229, in transcribe
    result: DecodingResult = decode_with_fallback(mel_segment)
  File "/usr/local/lib/python3.10/dist-packages/whisper/transcribe.py", line 164, in decode_with_fallback
    decode_result = model.decode(segment, options)
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/whisper/decoding.py", line 811, in decode
    result = DecodingTask(model, options).run(mel)
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/whisper/decoding.py", line 724, in run
    tokens, sum_logprobs, no_speech_probs = self._main_loop(audio_features, tokens)
  File "/usr/local/lib/python3.10/dist-packages/whisper/decoding.py", line 689, in _main_loop
    tokens, completed = self.decoder.update(tokens, logits, sum_logprobs)
  File "/usr/local/lib/python3.10/dist-packages/whisper/decoding.py", line 276, in update
    next_tokens = Categorical(logits=logits / self.temperature).sample()
  File "/usr/local/lib/python3.10/dist-packages/torch/distributions/categorical.py", line 118, in sample
    samples_2d = torch.multinomial(probs_2d, sample_shape.numel(), True).T
RuntimeError: CUDA error: device-side assert triggered

After inspecting the error, I come to realize the problem is constructing the log_mel_spectrogram(), the data from the audio is coming good (Without NaNs) but when passing for the log_mel_spectrogram() comes the tensor with all NaNs, I think the problem should be on the following lines:

https://github.com/openai/whisper/blob/main/whisper/audio.py#L154C9-L154C9

    mel_spec = filters @ magnitudes

    log_spec = torch.clamp(mel_spec, min=1e-10).log10()
    log_spec = torch.maximum(log_spec, log_spec.max() - 8.0)
    log_spec = (log_spec + 4.0) / 4.0

This is the output from each line

Anyone could know what is wrong? 😢

System description:

• Jetson AGX Xavier 16Gb
• Python 3.10
• Torch 1.8.2 Built from source
• Whisper v20230314

[glangford]

"Torch 1.8.2 built from source" stands out in the system description, note that the whisper test matrix only goes back to 1.13.1.

whisper/.github/workflows/test.yml

Line 15 in b91c907

pytorch-version: [1.13.1, 2.0.0]

[juansebashr]

Indeed, but I forgot to clarify that testing with the same version of Torch on my local machine worked properly