Home assistants require special phrases called hotwords to get activated (e.g., "OK Google"). EfficientWord-Net is a hotword detection engine based on few-shot learning that allows developers to add custom hotwords to their programs without extra charges. The library is purely written in Python and uses Google's TFLite implementation for faster real-time inference. It is inspired by FaceNet's Siamese Network Architecture and performs best when 3-4 hotword samples are collected directly from the user.
EfficientWord-Net.mp4
Training File to access the training file.
Checkout the people speech spoken words dataset, the Resnet_50_Arc_loss was trained on the same https://mlcommons.org/datasets/multilingual-spoken-words/
Research Paper to access the research paper.
This library works with Python versions 3.10 to 3.14.
Before running the pip installation command for the library, a few dependencies need to be installed manually:
Run the following pip command:
pip install EfficientWord-Net
To import the package:
import eff_word_netAfter installing the packages, you can run the demo script built into the library (ensure you have a working microphone).
Access documentation from: https://ant-brain.github.io/EfficientWord-Net/
Command to run the demo:
python -m eff_word_net.engine
For any new hotword, the library needs information about the hotword. This information is obtained from a file called {wakeword}_ref.json.
For example, for the wakeword 'alexa', the library would need the file called alexa_ref.json.
These files can be generated with the following procedure:
-
Collect 4 to 10 uniquely sounding pronunciations of a given wakeword. Put them into a separate folder that doesn't contain anything else.
-
Finally, run this command. It will ask for the input folder's location (containing the audio files) and the output folder (where the _ref.json file will be stored):
python -m eff_word_net.generate_reference
Note: Only WAV files are supported. If your audio files are in other formats, convert them to WAV using https://ffmpegwasm.netlify.app/playground, which performs conversion entirely in your browser without uploading files.
Once you have generated the reference file, you can test the hotword detection using:
python -m eff_word_net.single_word_test --reference-file /full/path/to/your_wakeword_ref.json --hotword "your_wakeword"
Ensure you have a working microphone.
The pathname of the generated wakeword needs to be passed to the HotwordDetector instance:
HotwordDetector(
hotword="hello",
model=Resnet_50_Arc_loss(),
reference_file="/full/path/name/of/hello_ref.json",
threshold=0.9, # min confidence required to consider a trigger
relaxation_time=0.8 # default value, in seconds
)The model variable can receive an instance of Resnet_50_Arc_loss or First_Iteration_Siamese.
The relaxation_time parameter is used to determine the minimum time between any two triggers. Any potential triggers before the relaxation_time will be canceled. The detector operates on a sliding window approach, resulting in multiple triggers for a single utterance of a hotword. The relaxation_time parameter can be used to control multiple triggers; in most cases, 0.8 seconds (default) will suffice.
The library has predefined embeddings readily available for a few wakewords such as Mycroft, Google, Firefox, Alexa, Mobile, and Siri. Their paths are readily available in the library installation directory.
from eff_word_net import samples_locimport os
from eff_word_net.streams import SimpleMicStream
from eff_word_net.engine import HotwordDetector
from eff_word_net.audio_processing import Resnet50_Arc_loss
from eff_word_net import samples_loc
base_model = Resnet50_Arc_loss()
mycroft_hw = HotwordDetector(
hotword="mycroft",
model = base_model,
reference_file=os.path.join(samples_loc, "mycroft_ref.json"),
threshold=0.7,
relaxation_time=2
)
mic_stream = SimpleMicStream(
window_length_secs=1.5,
sliding_window_secs=0.75,
)
mic_stream.start_stream()
print("Say Mycroft ")
while True :
frame = mic_stream.getFrame()
result = mycroft_hw.scoreFrame(frame)
if result==None :
#no voice activity
continue
if(result["match"]):
print("Wakeword uttered",result["confidence"])The library provides a computation friendly way
to detect multiple hotwords from a given stream, instead of running scoreFrame() of each wakeword individually
import os
from eff_word_net.streams import SimpleMicStream
from eff_word_net import samples_loc
print(samples_loc)
base_model = Resnet50_Arc_loss()
mycroft_hw = HotwordDetector(
hotword="mycroft",
model = base_model,
reference_file=os.path.join(samples_loc,"mycroft_ref.json"),
threshold=0.7,
relaxation_time=2
)
alexa_hw = HotwordDetector(
hotword="alexa",
model=base_model,
reference_file=os.path.join(samples_loc,"alexa_ref.json"),
threshold=0.7,
relaxation_time=2,
#verbose=True
)
computer_hw = HotwordDetector(
hotword="computer",
model=base_model,
reference_file=os.path.join(samples_loc,"computer_ref.json"),
threshold=0.7,
relaxation_time=2,
#verbose=True
)
multi_hotword_detector = MultiHotwordDetector(
[mycroft_hw, alexa_hw, computer_hw],
model=base_model,
continuous=True,
)
mic_stream = SimpleMicStream(window_length_secs=1.5, sliding_window_secs=0.75)
mic_stream.start_stream()
print("Say ", " / ".join([x.hotword for x in multi_hotword_detector.detector_collection]))
while True :
frame = mic_stream.getFrame()
result = multi_hotword_detector.findBestMatch(frame)
if(None not in result):
print(result[0],f",Confidence {result[1]:0.4f}")
Access documentation of the library from here : https://ant-brain.github.io/EfficientWord-Net/
Here's the corrected version of the README.md file with improved grammar and formatting:
@Stypox has provided a gist with which one can perform inference for the same in C++ https://gist.github.com/Stypox/2bf47017ea5361776d9c096e886e0540
- Stability related improvements
- Pivoted from pydub to soundfile for audio file processing to improve reliability.
- Dropped support for MP3 files in reference generation.
- Updated documentation to guide users on converting non-WAV files.
- Added single_word_test module with typer support for testing custom reference files.
- Pivoted to more cross platform friendly dependencies
- Depreceated tensorflow model due to inference difficulties of tflite in platforms like Windows
- Trained a new model from scratch using a modified distilled dataset from MLCommons.
- Used Arc loss function instead of triplet loss function.
- The resultant model is stored as resnet_50_arcloss.
- The newer model showcases much better resilience towards background noise and requires fewer samples for good accuracy.
- Minor changes in the API flow to facilitate easy addition of newer models.
- Newer model can handle a fixed window length of 1.5 seconds.
- The old model can still be accessed through first_iteration_siamese.
- Major changes to replace complex logic of handling poly triggers per utterance with simpler logic and a more straightforward API for programmers.
- Introduces breaking changes.
- C++ implementation of the current model is here.
- Trained on single words, hence may result in bizarre behavior when using phrases like "Hey xxx".
- Audio processing window limited to 1 sec. Hence, it will not work effectively for longer hotwords.
- Hotword Performance is bad: If you are experiencing issues like this, feel free to ask in the discussions.
- Can it run on FPGAs like Arduino?: No, the new Resnet_50_Arcloss model is too heavy to run on Arduino (roughly 88MB in size). We will soon add support for pruned versions of the model so that it can become light enough to run on tiny devices. For now, it should be able to run on Raspberry Pi-like devices.
- If you have ideas to make the project better, feel free to ping us in the discussions.
- Add audio file handler in streams. PRs are welcome.
- Add more detailed documentation explaining the sliding window concept.
- Add model fine-tuning support.
- Add support for sparse and fine-grained pruning where the resultant models could be used for fine-tuning (already working on this).
Our hotword detector's performance is notably lower compared to Porcupine. We have thought about better NN architectures for the engine and hope to outperform Porcupine. This has been our undergrad project, so your support and encouragement will motivate us to develop the engine further. If you love this project, recommend it to your peers, give us a 🌟 on GitHub, and a clap 👏 on Medium.
Update: Your stars encouraged us to create a new model which is far better. Let's make this community grow!