Finetune the whisper model by using a custom decoder

[aquorio15]

I'm trying to use OpenAI's open source Whisper library to finetune a Speech-to-text model i.e. the input audio will be in one language and the output text in another language. So after loading the model i am detaching the encoder and trying to feed the input audio into the encoder, but i am getting an Assertion error AssertionError: incorrect audio shape.
This is what i am currently trying

import whisper
model = whisper.load_model("base")  
mod = model.encoder ### Getting the encoder
audio = whisper.load_audio('audio_Path/test.wav') ### loading the audio
mel = whisper.log_mel_spectrogram(audio).cuda()
mel = mel.unsqueeze(0)
print(mod(mel))