working with files like vocab.json , tokenizer_config.json ...

[koosoli]

Hi,
I have received these files for a whisper model trained on the Luxemburgish language (request link)
it includes files like "vocab.json" "tokenizer_config.json" "special_tokens_map.json" "pytorch_model.bin" ... and so on.
I am trying to have it run on google colab (request link)
But I am not sure what to do with the files. Can someone point me to the right direction?

I tired this:

# Importiere die benötigten Bibliotheken
from whisper import Whisper
from whisper.tokenizers import WhisperTokenizer

# Erstelle einen Whisper-Objekt mit dem Pfad zu deiner config.json Datei
whisper = Whisper.from_pretrained("/path/to/config.json")

# Erstelle einen WhisperTokenizer-Objekt mit dem Pfad zu deiner vocab.json und merges.txt Dateien
tokenizer = WhisperTokenizer.from_pretrained("/path/to/vocab.json", "/path/to/merges.txt")

# Lade das PyTorch-Modell aus der pytorch_model.bin Datei
whisper.load_state_dict(torch.load("/path/to/pytorch_model.bin"))

# Setze das Modell in Evaluierungsmodus
whisper.eval()

But already failed early on

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-18-b720145ea464> in <cell line: 3>()
      1 # Importiere die benötigten Bibliotheken
      2 from whisper import Whisper
----> 3 from whisper.tokenizers import WhisperTokenizer
      4 
      5 # Erstelle einen Whisper-Objekt mit dem Pfad zu deiner config.json Datei

ModuleNotFoundError: No module named 'whisper.tokenizers'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

pip 23.3.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
!pip show whisper
WARNING: Package(s) not found: whisper

Thanks for any help

[phineas-pta]

if u want tu use, u should learn python

[koosoli]

So it is your opinion that there is no simple answer to my question except to learn python right? No simple command that I can try out? Phan Tuấn Anh ***@***.***> schrieb am Fr., 19. Jan. 2024, 22:04:

…

if u want tu use, u should learn python — Reply to this email directly, view it on GitHub <#1965 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3UGXPIXVZWWCWQRTEG7QTYPLNWJAVCNFSM6AAAAABCBOYSYKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DCOBVHE2DE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[phineas-pta]

or u can ask the author of that code