Triton cache miss in dynamic time warping

[avidale]

When running a Whisper model with dynamic time warping (model.transcribe(wav, verbose=False, language=lang, word_timestamps=True), I occasionally get the following error:

KeyError: ('2-.-0-.-0--d6252949da17ceb5f3a278a70250af13-3b85c7bef5f0a641282f3b73af50f599-2d732a2488b7ed996facc3e641ee56bf-3498c340fd4b6ee7805fd54b882a04f5-e1f133f98d04093da2078dfc51c36b72-b26258bf01f839199e39d64851821f26-d7c06e3b46e708006c15224aac7a1378-f585402118c8a136948ce0a49cfe122c', (torch.float32, torch.int32, torch.float32, 'i32', 'i32', 'i32', 'i32', 'i32'), (1024,), (True, True, True, (True, False), (False, False), (False, False), (False, False), (True, False)))

It happens at the same time

  File ".../python3.9/site-packages/whisper/timing.py", line 123, in dtw_cuda
    dtw_kernel[(1,)](
  File "<string>", line 41, in dtw_kernel
  File "/.../python3.9/site-packages/triton/compiler.py", line 1601, in compile
    metadata = json.load(f)
  File ".../python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
OSError: [Errno 116] Stale file handle

When rerunning the same scripts several times, the error appears and goes away without any apparent reason.

Does anyone know why it happens?
It looks like some cache miss in this function https://github.com/openai/whisper/blob/main/whisper/triton_ops.py#L14, but I have no idea how Triton operates, so I don't know even how to start fixing this.

[samar-khanna]

Hey, did you ever figure this one out?