Audio Feature Extraction using Whisper

[pegahsalehi]

Hi,
I want to use the whisper to extract logits from audio using speechbrain.lobes.models.huggingface_whisper

from speechbrain.lobes.models.huggingface_whisper import HuggingFaceWhisper
import speechbrain as sb

model_hub_whisper = "openai/whisper-tiny"
model_whisper = HuggingFaceWhisper(model_hub_whisper, save_path='----')

source = sb.dataio.dataio.read_audio('./intro.wav').squeeze()
source = source.unsqueeze(0)
print(source.shape)
# This can be given as an argument when we instantiate the model as well
model_whisper.encoder_only=True
fea_whisper = model_whisper(source)
print(fea_whisper.shape)

how can I extract the logits from audio in the shape (X, Y).
That X is a fixed number when we use the whisper model and Y depends on the length of audio.

[jongwook]

I presume you mean audio features rather than logits since you're using only the encoder. The encoder always takes 30-second-long audio as input, and we trim or pad the audio to match this length. The encoded features are also 30-seconds long as a result. You can slice the features if the input was shorter than 30 seconds.

More code-specific questions could be better answered by speechbrain's maintainers.

[pegahsalehi]

I presume you mean audio features rather than logits since you're using only the encoder. The encoder always takes 30-second-long audio as input, and we trim or pad the audio to match this length. The encoded features are also 30-seconds long as a result. You can slice the features if the input was shorter than 30 seconds.

More code-specific questions could be better answered by speechbrain's maintainers.

Hi @jongwook , Thanks a lot for your help.
How can I extract logits in the shape (X, Y)?

[LiuRicky]

I presume you mean audio features rather than logits since you're using only the encoder. The encoder always takes 30-second-long audio as input, and we trim or pad the audio to match this length. The encoded features are also 30-seconds long as a result. You can slice the features if the input was shorter than 30 seconds.

More code-specific questions could be better answered by speechbrain's maintainers.

@jongwook thanks for your reply, you mentioned that 'You can slice the features if the input was shorter than 30 seconds.'

I have a 6-second audio and input it to whisper encoder. The 6-second audio is pad to 30s, and the final feature shape is (1500, 1024). Can I just slice (:300, :) as my audio feature?

However, I print the features at (610:620, :), it is non-zero and I think these padded feature indeed involved in encoder transformer since 'mask is ignored in whisper encoder'. Based on these findings, can I just slice (:300, :) as my audio feature?

[LiuRicky]

Will the slice incur feature loss since padded feature are involved in the whisper encoder?

[pegahsalehi]

@LiuRicky, this is exactly my problem. Did you find any ideas?

[jeeyung]

I am also facing the same problem