Small script which can make your voice into a keyboard (start/stop per key)

[dynamiccreator]

whisper-typer-tool
Once you started the script you can start/stop recording with "F2". After the recording it will type what said as if you have typed with your keyboard into any editor or input field etc...

What is does in detail:
-Records audio file from microphone
-Transcribes the recorded file with whisper and deletes it afterwards
-Types the transcribed text with pynput

[FlowDownTheRiver]

Thanks,great work.I loved it!

[joedevon]

The explanation confused me but what It think you're saying is this is voice-to-type feature?

[FlowDownTheRiver]

Yes, it can write what you are saying to the microphone with an activation key when you launch the program. So, it doesn't matter where it is. Right now, I am writing this message with talking to my microphone. I am using this repository to write what I want to say. Also, if you go to the Issues tab on that repository, I also shared an edited file which can overcome the issues of character encoding. Otherwise you may get some errors. Make sure to download and overwrite the file if you are going to use it.
It works like this. You press the activation key, then you can record your voice and when you press the activation key to end the recording for the second time, then wherever your mouse points, let's say you opened a TXT file or a Word file or any text input field, it can write what you say to that field.

#model selection -> (tiny base small medium large)
print("loading model...")
model_name = "medium"

You can change the model to anything you like. Inside the Python file you have the options of tiny, base, small, medium and large. But be careful, it will take more time to process if you go higher. And it requires more resources to use the higher models.

[joedevon]

Ah thank you! Very cool.

[dynamiccreator]

The explanation confused me but what It think you're saying is this is voice-to-type feature?

Yes, that's correct. I just didn't found a solution where you can just press a hot key to enable / disable transcription. Always thinking that this is more practical than having to say a command like "Hey, whisper" etc. as it probably might will misunderstood you in 20% of the cases and then not starting to transcribe. You can perfectly use this to write into an excel sheet, clicking into the cell you want to edit/write, press the hotkey and then just talk and press the hotkey again. Just thinking hot keys for on/off is the most not annoying way to do transcription.

[thejackenator]

Hey I am very new to all of this, but I am very interested in your program. Could you give a bit more detailed walkthrough for someone new to GitHub and Python? I know I'm probably not the intended audience at this point, but I would love to try out what you've made for medical dictation. I made it though the steps for the most part, but I got an error message that it couldn't find model_loaded.wav. I must have installed wrong, so I would really appreciate your help!

[dynamiccreator]

Please make sure that you have downloaded the latest version of this repository via git clone https://github.com/dynamiccreator/whisper-typer-tool.git in case you accidentally downloaded the tool just right before I added the "model_loaded.wav" file to the repository three weeks ago. Alternatively you could just download the file model_loaded.wav from the repository and place it in the same folder where the script is located. Does that help?

[thejackenator]

Thank you so much for the reply. Could it be my version of python? The file is in the folder, but this is the message I am getting:

C:\Users\jackm>python3 Downloads\whisper-typer-tool-main\whisper-typer-tool-main\whisper-typer-tool.py
loading model...
Traceback (most recent call last):
File "C:\Users\jackm\Downloads\whisper-typer-tool-main\whisper-typer-tool-main\whisper-typer-tool.py", line 18, in
playsound("model_loaded.wav")
File "C:\Users\jackm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\playsound.py", line 35, in _playsoundWin
winCommand('open "' + sound + '" alias', alias)
File "C:\Users\jackm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\playsound.py", line 31, in winCommand
raise PlaysoundException(exceptionMessage)
playsound.PlaysoundException:
Error 275 for command:
open "model_loaded.wav" alias playsound_0.9369558140205669
Cannot find the specified file. Make sure the path and filename are correct.

[dynamiccreator]

Start bash from where the folder it is located or use cd to set the bash path to where the folder is located. cd "C:\Users\jackm\Downloads\whisper-typer-tool-main\whisper-typer-tool-main" does the trick. From there you can start the script with python3 whisper-typer-tool.py

[thejackenator]

Thanks so much for the help. I think I'm making progress, but it still isn't working. Here's where I'm at now.

C:\Users\jackm\Downloads\whisper-typer-tool-main\whisper-typer-tool-main>python3 whisper-typer-tool.py
loading model...
tiny model loaded
ready - start transcribing with F2 ...

Start recording...

Finish recording
C:\Users\jackm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\whisper\transcribe.py:79: UserWarning: FP16 is not supported on CPU; using FP32 instead
warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Exception in thread Thread-1 (transcribe_speech):
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\jackm\Downloads\whisper-typer-tool-main\whisper-typer-tool-main\whisper-typer-tool.py", line 34, in transcribe_speech
result = model.transcribe("test"+str(i)+".wav")
File "C:\Users\jackm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\whisper\transcribe.py", line 85, in transcribe
mel = log_mel_spectrogram(audio)
File "C:\Users\jackm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\whisper\audio.py", line 111, in log_mel_spectrogram
audio = load_audio(audio)
File "C:\Users\jackm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\whisper\audio.py", line 42, in load_audio
ffmpeg.input(file, threads=0)
File "C:\Users\jackm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\ffmpeg_run.py", line 313, in run
process = run_async(
File "C:\Users\jackm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\ffmpeg_run.py", line 284, in run_async
return subprocess.Popen(
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 971, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1440, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
Start recording...

Finish recording
Start recording...

Finish recording

[dynamiccreator]

Did you paste ffmpeg.exe into that folder? Is your microphone connected and working?

[thejackenator]

Pasting ffmpeg into the folder fixed it. Thank you so much for the help. It works amazingly! I can't believe this level of accuracy is obtainable through an open source program now.

Hopefully/probably my last question. The text goes into most fields (such as word, browsers, etc) and looks just perfect, exactly how I imagined it. In the one program I was most excited to use it for, it transcribes the text without any capital letters. For some reason, all of the capital letters are just committed. For example: "y name is ack from the nited tates of merica. 'm not sure why it does this r. ynamiccreator. an you think of why it would happen like this?"

[dynamiccreator]

Great to hear that it worked!
As you said yourself that in most fields (such as word, browsers, etc) everything works as imagined, it clearly just has to do with the program you are using. It might be that capital letters are just restricted for some reason in that program. What happens if you type capital letters like "ABC" with your keyboard? Does the program allow capital letters or does it skip them as well?
Is that the case, a fix could be to change line 41 of the script file (whisper-typer-tool.py) to pykeyboard.type(element.lower()) That will let the script only type non-capital letters.

[fadygit]

HI , I just want to ask, How can I use it for different languages like (Arabic language)? , I already tried the whisper at replicate website and it's working for Arabic language

[dynamiccreator]

Hi,
TL;DR; Yes you can use it for Arabic language.
For now the tool relies on the auto detect mechanism from whisper to select the language. So yes you can use it for Arabic as well and all other languages supported by whisper. I also might add a config file in the future or a small GUI. So you can select all supported languages. This could possible help if auto detection doesn't detect the correct language.

[thejackenator]

Hey, you've helped me get your program up and running, and it is awesome. Is there any way you could further develop this into a fully functioning app that is independent of downloading/installing three separate things (ffmpeg, python, and GitHub) and using the command line? I would pay for an independent version of this that has a user interface that is minimal but separate from the command line. It would be awesome if you could change from tiny to base, small, medium in this interface and see when it it transcribing. If you could train it to add a more robust medical dictionary in as well, that would be huge. Let me know what you think and if I could contract you to help me with this. Thanks!

[dynamiccreator]

Hi, great to hear it worked out so well.
Yes, you can contact me via mail [email protected] for further discussion.
Have a nice day!

[richdrummer33]

Is there any chance that keystrokes can be simulated when the text is being interpreted?