Trainning and Score very low

[JpEncausse]

Hi,
I made 3 trainning for world "Reflets" in French.

• for the v1 and v2 I use only a female voice and get 0.2 with my voice 0.9 with sample
• for the v3 I mix 3 male and female voice and got 0 confidence

Here is my Notebook :
Nanowakeword_Reflets_v3.ipynb

And the model seems ok ?

The logs

 _   _               __          __   _     __          __           _ 
 | \ | |              \ \        / /  | |    \ \        / /          | |
 |  \| | __ _ _ __   __\ \  /\  / /_ _| | ____\ \  /\  / /__  _ __ __| |
 | . ` |/ _` | '_ \ / _ \ \/  \/ / _` | |/ / _ \ \/  \/ / _ \| '__/ _` |
 | |\  | (_| | | | | (_) \  /\  / (_| |   <  __/\  /\  / (_) | | | (_| |
 |_| \_|\__,_|_| |_|\___/ \/  \/ \__,_|_|\_\___| \/  \/ \___/|_|  \__,_|


STEP 1: Verifying and Preprocessing Data Directories
====================================================
Processing Noise: 100%|██████████| 1365/1365 [00:26<00:00, 51.44it/s]
Processing Rir: 100%|██████████| 1667/1667 [00:14<00:00, 112.07it/s]
WARNING:root:Directory not found, skipping preprocessing: nanowakeword_data/negative_speech
WARNING:root:Directory not found, skipping preprocessing: nanowakeword_data/positive_wakeword
INFO: Data verification and preprocessing complete.

INFO: Determining hardware-specific configurations...
STEP 2: Activating Synthetic Data Generation Engine
===================================================
INFO: Processing 10 custom negative phrases.
INFO: Generating 25 copies for EACH custom phrase.
INFO: Total custom samples prepared: 250
INFO: Target negative samples: 6000. Current custom samples: 250.
INFO: Generating 5750 auto-adversarial phrases to fill the gap.
WARNING:root:Downloading phonemizer model from https://github.com/arcosoph/phonemize/releases/download/v0.2.0/phonemize_m1.pt...
WARNING:root:The word 'reflets' was not found in the pronunciation dictionary! Using the Phonemize library to predict the phonemes.
WARNING:root:Phones for 'reflets': [R][IY][F][L][AH][T][S]
INFO: Initiating data generation pipeline for phrase: 'reflets'
INFO: Executing task 'Positive_Train': 3000 clips -> 'nanowakeword_data/positive_wakeword'
Loading voices: 100%|██████████| 3/3 [00:04<00:00,  1.54s/it]

Generating Audio: 100%|██████████| 3000/3000 [09:13<00:00,  5.42it/s]
INFO: Executing task 'Adversarial_Train': 6000 clips -> 'nanowakeword_data/negative_speech'
Generating Audio: 100%|██████████| 6000/6000 [23:06<00:00,  4.33it/s]
INFO: Synthetic data generation process finished successfully.

STEP 2.5: Activating Intelligent Configuration Engine
===================================================
Analyzing dataset characteristics...
Analyzing positive_wakeword files: 100%|██████████| 3000/3000 [00:13<00:00, 230.13it/s]
Analyzing negative_speech files: 100%|██████████| 6000/6000 [00:32<00:00, 182.49it/s]
Analyzing Noise files: 100%|██████████| 1365/1365 [00:23<00:00, 58.82it/s]Analysis complete!


                        Dataset Statistics                        
┌───────────────┬────────────────────────────────────────────────┐
│ Parameter     │ Value                                          │
├───────────────┼────────────────────────────────────────────────┤
│ H_pos         │ 0.6620500868055553                             │
│ H_neg         │ 1.6486525520833364                             │
│ H_noise_paths │ {'nanowakeword_data/Noise': 5.067988993055563} │
│ H_noise       │ 5.067988993055563                              │
│ A_noise       │ 0.05487615084446175                            │
│ N_rir         │ 1667                                           │
└───────────────┴────────────────────────────────────────────────┘
INFO: Project assets will be saved in: /content/trained_models/reflets_v3
INFO: Autotuning optimal clip duration...
INFO: Optimal clip duration autotuned to: 32000 samples (2.00 seconds).
STEP 3: Computing Acoustic Features from Audio Sources
======================================================
[nanowakeword] Required model 'melspectrogram.onnx' not found. Downloading...
melspectrogram.onnx: 100%|██████████| 1.09M/1.09M [00:00<00:00, 34.4MiB/s]
[nanowakeword] Download of 'melspectrogram.onnx' complete.
[nanowakeword] Required model 'embedding_model.onnx' not found. Downloading...
embedding_model.onnx: 100%|██████████| 1.33M/1.33M [00:00<00:00, 36.9MiB/s]
[nanowakeword] Download of 'embedding_model.onnx' complete.
Computing features: 100%|██████████| 234/234 [58:27<00:00, 14.99s/it]
Computing features: 100%|██████████| 468/468 [1:57:38<00:00, 15.08s/it]
Computing features: 100%|██████████| 53/53 [14:26<00:00, 16.35s/it]
INFO: Initializing Model and Training Components...
INFO: Using optimizer: ADAMW
INFO: Setting up learning rate scheduler: ONECYCLE
STEP 4: Starting Training Process
=================================
INFO: Using model architecture: 🤍 RNN
INFO: Debug mode ON. Logs will be saved to:
/content/trained_models/reflets_v3/2_training_artifacts/training_debug/training_debug.log
INFO: Checkpointing is ENABLED. A checkpoint will be saved every 1000 steps.
INFO: Training for 30000 steps. Model checkpointing and early stopping will activate after 4500 warm-up steps.
                           Effective Training Configuration                           
┌───────────────────────────────────────────────────────────────┬────────────────────┐
│ Parameter                                                     │ Value              │
├───────────────────────────────────────────────────────────────┼────────────────────┤
│ WARMUP_STEPS                                                  │ 4500               │
│ activation_function                                           │ relu               │
│ audio_processing.autotune_length.duration_buffer_ms           │ 750                │
│ audio_processing.autotune_length.enabled                      │ True               │
│ audio_processing.autotune_length.min_allowable_length         │ 32000              │
│ audio_processing.autotune_length.num_samples_to_inspect       │ 50                 │
│ audio_processing.autotune_length.snap_to_min_tolerance        │ 4000               │
│ augmentation_batch_size                                       │ 128                │
│ augmentation_rounds                                           │ 10                 │
│ augmentation_settings.BackgroundNoise                         │ 0.8                │
│ augmentation_settings.BandStopFilter                          │ 0.2                │
│ augmentation_settings.ColoredNoise                            │ 0.4                │
│ augmentation_settings.Distortion                              │ 0.25               │
│ augmentation_settings.Gain                                    │ 1.0                │
│ augmentation_settings.ParametricEQ                            │ 0.25               │
│ augmentation_settings.PitchShift                              │ 0.5                │
│ augmentation_settings.RIR                                     │ 0.6                │
│ background_paths_duplication_rate                             │ [1]                │
│ batch_composition.batch_size                                  │ 128                │
│ batch_composition.source_distribution.negative_speech         │ 40                 │
│ batch_composition.source_distribution.positive                │ 30                 │
│ batch_composition.source_distribution.pure_noise              │ 30                 │
│ checkpoint_averaging_top_k                                    │ 5                  │
│ checkpointing.enabled                                         │ True               │
│ checkpointing.interval_steps                                  │ 1000               │
│ checkpointing.limit                                           │ 5                  │
│ debug_mode                                                    │ True               │
│ dropout_prob                                                  │ 0.5                │
│ ema_alpha                                                     │ 0.01               │
│ embedding_dim                                                 │ 96                 │
│ feature_gen_cpu_ratio                                         │ 0.8                │
│ layer_size                                                    │ 384                │
│ learning_rate_max                                             │ 0.001              │
│ lr_scheduler_type                                             │ onecycle           │
│ main_delta                                                    │ 0.0001             │
│ max_snr_in_db                                                 │ 13.477561925166414 │
│ min_snr_in_db                                                 │ -7.217950459866868 │
│ model_name                                                    │ reflets_v3         │
│ model_type                                                    │ rnn                │
│ momentum                                                      │ 0.9                │
│ n_blocks                                                      │ 6                  │
│ optimizer_type                                                │ adamw              │
│ overwrite                                                     │ True               │
│ show_training_summary                                         │ True               │
│ steps                                                         │ 30000              │
│ total_length                                                  │ 32000              │
│ train_model                                                   │ True               │
│ training_refinement.hard_negative_mining.bank_size            │ 500                │
│ training_refinement.hard_negative_mining.enable               │ True               │
│ training_refinement.hard_negative_mining.focus_on_confusing   │ True               │
│ training_refinement.hard_negative_mining.min_bank_size        │ 50                 │
│ training_refinement.hard_negative_mining.mining_threshold     │ 0.3                │
│ training_refinement.hard_negative_mining.replay_batch_size    │ 24                 │
│ training_refinement.loss_settings.false_accept_penalty        │ 20.0               │
│ training_refinement.loss_settings.focal_gamma                 │ 2.5                │
│ training_refinement.loss_settings.miss_penalty                │ 5.0                │
│ training_refinement.loss_settings.temperature                 │ 2.0                │
│ training_refinement.loss_weights.bce                          │ 0.3                │
│ training_refinement.loss_weights.confidence_reg               │ 10.0               │
│ training_refinement.loss_weights.focal                        │ 0.2                │
│ training_refinement.loss_weights.margin                       │ 100.0              │
│ training_refinement.loss_weights.negative_push                │ 30.0               │
│ training_refinement.loss_weights.replay                       │ 20.0               │
│ training_refinement.loss_weights.triplet                      │ 0.5                │
│ training_refinement.margin_settings.confidence_penalty_weight │ 50.0               │
│ training_refinement.margin_settings.exponential_scale         │ 4.0                │
│ training_refinement.margin_settings.safety_threshold          │ 0.05               │
│ training_refinement.optimization.adaptive_clipping            │ True               │
│ training_refinement.optimization.aggressive_clip_factor       │ 0.5                │
│ training_refinement.optimization.max_grad_norm                │ 1.5                │
│ training_refinement.regularization.confidence_regularization  │ 0.1                │
│ training_refinement.regularization.enable_spectral            │ True               │
│ training_refinement.regularization.spectral_weight            │ 0.01               │
│ training_refinement.triplet_settings.base_margin              │ 0.5                │
│ training_refinement.triplet_settings.curriculum_steps         │ 15000              │
│ training_refinement.triplet_settings.hard_mining_ratio        │ 0.6                │
│ training_refinement.triplet_settings.negative_push_margin     │ 0.8                │
│ transform_clips                                               │ True               │
│ triplet_loss_margin                                           │ 0.2                │
│ weight_decay                                                  │ 0.01               │
└───────────────────────────────────────────────────────────────┴────────────────────┘
Training:  35%|███▍      | 10385/30000 [56:54<1:47:30,  3.04it/s]
INFO: Early stopping triggered at step 10385. No improvement in stable loss for 4500 steps.

INFO: Training finished. Merging best checkpoints to create final model...
INFO: Averaging the top 5 most stable models found during training...
INFO: Calculating performance metrics for the final averaged model...
========================================
TRAINING COMPLETE - FINAL REPORT
========================================
INFO: NOTE: These metrics are indicators of model health, not real-world performance.
  Average Stable Loss      : 31.2011
  Weight Diversity (Std Dev): 0.7673
  Avg. Positive Confidence (Logit): -6.783
  Avg. Negative Confidence (Logit): -22.139

Generating training performance graph...
Performance graph saved to: /content/trained_models/reflets_v3/2_training_artifacts/graphs/training_performance_graph.png
INFO: Reconstructing model into a sequential format for export.
INFO: Saving inference-ready ONNX model to '/content/trained_models/reflets_v3/3_model/reflets_v3.onnx'
INFO: Using ONNX opset version: 17
INFO: ONNX model saved successfully.


CONGRATULATIONS! (✿◕‿◕✿)
Your custom wake word model has been successfully trained!

The JS Code seems to works even if the confidence has variation from 0.2 to 0.35 for model v2 but it feels very very low.

        try {
            // Process audio in chunks and return highest score
            let maxScore = 0;
            const numFrames = Math.floor(audioFloat32.length / FRAME_SIZE);

            // Initialize mel buffer for this check
            let melBuffer = [];
            
            // Use persistent embedding buffer (initialize only if empty)
            if (persistentEmbeddingBuffer.length === 0) {
                for (let i = 0; i < EMBEDDING_HISTORY_SIZE; i++) {
                    persistentEmbeddingBuffer.push(new Float32Array(96).fill(0));
                }
            }

            for (let frameIdx = 0; frameIdx < numFrames; frameIdx++) {
                const chunk = audioFloat32.subarray(frameIdx * FRAME_SIZE, (frameIdx + 1) * FRAME_SIZE);

                // Stage 1: Audio → Melspectrogram
                const melTensor = new ort.Tensor('float32', chunk, [1, FRAME_SIZE]);
                const melResults = await melModel.run({ [melModel.inputNames[0]]: melTensor });
                let melData = melResults[melModel.outputNames[0]].data;

                // Apply transformation (critical for correct detection)
                for (let j = 0; j < melData.length; j++) {
                    melData[j] = (melData[j] / 10.0) + 2.0;
                }

                // Add 5 mel frames (each chunk produces 5 frames)
                for (let j = 0; j < 5; j++) {
                    melBuffer.push(new Float32Array(melData.subarray(j * 32, (j + 1) * 32)));
                }

                // Stage 2: Melspectrogram History → Embedding
                while (melBuffer.length >= MEL_WINDOW_SIZE) {
                    const windowFrames = melBuffer.slice(0, MEL_WINDOW_SIZE);
                    const flattenedMel = new Float32Array(MEL_WINDOW_SIZE * 32);
                    for (let k = 0; k < MEL_WINDOW_SIZE; k++) {
                        flattenedMel.set(windowFrames[k], k * 32);
                    }

                    const embeddingTensor = new ort.Tensor('float32', flattenedMel, [1, MEL_WINDOW_SIZE, 32, 1]);
                    const embeddingResults = await embeddingModel.run({ [embeddingModel.inputNames[0]]: embeddingTensor });
                    const embedding = embeddingResults[embeddingModel.outputNames[0]].data;

                    // Stage 3: Embedding History → Wake Word Score
                    // Update persistent buffer (slide window)
                    persistentEmbeddingBuffer.shift();
                    persistentEmbeddingBuffer.push(new Float32Array(embedding));

                    const flattenedEmbeddings = new Float32Array(EMBEDDING_HISTORY_SIZE * 96);
                    for (let k = 0; k < EMBEDDING_HISTORY_SIZE; k++) {
                        flattenedEmbeddings.set(persistentEmbeddingBuffer[k], k * 96);
                    }

                    const wakeWordTensor = new ort.Tensor('float32', flattenedEmbeddings, [1, EMBEDDING_HISTORY_SIZE, 96]);
                    const wakeWordResults = await wakeWordModel.run({ [wakeWordModel.inputNames[0]]: wakeWordTensor });
                    const score = wakeWordResults[wakeWordModel.outputNames[0]].data[0];

                    if (score > maxScore) {
                        maxScore = score;
                    }

                    // Slide window by 8 frames
                    melBuffer.splice(0, 8);
                }
            }

            const detected = maxScore > $.CLIENT.CONFIG.WAKEWORD_THRESHOLD;
            console.log(`Wake word check: score=${maxScore.toFixed(4)}, detected=${detected}`);
            
            // DEBUG: Log model info on first real check
            if (maxScore > 0.01 && !window._wakeword_debug_logged) {
                console.log('🔍 WAKEWORD DEBUG INFO:');
                console.log('  Model inputs:', wakeWordModel.inputNames);
                console.log('  Model outputs:', wakeWordModel.outputNames);
                console.log('  Input shape expected: [1, 16, 96]');
                console.log('  Frames processed:', numFrames);
                console.log('  Max score location: frame', maxScore);
                window._wakeword_debug_logged = true;
            }

            return { detected, score: maxScore, reason: 'checked' };

        } catch (err) {
            console.error('Wake word check failed:', err);
            return { detected: false, score: 0, reason: 'error' };
        }
    };

[muhammadabidPro]

Hi @JpEncausse ,

Thank you so much for sharing your detailed logs, your notebook, and even your JavaScript code. This is incredibly helpful, and it allows me to see exactly what's happening.

You've pointed out some really important things, and I want to address them directly.

First, regarding the final training report. You are absolutely right to be confused by those numbers. To be honest, the final report's Logit values (like -6.783) are just an internal health metric from the training process, and they do not represent the final model's real-world performance. They can be very misleading, and I am actually planning to fix this in a future update to make the report clearer.

You are correct: the actual inference score from the .onnx model will be a clean probability from 0.0 up to 0.9999...—it can get very close, but mathematically, it will never be exactly 1.0.

I have looked very closely at your notebook and logs, and I believe I have found the core issue. I will be testing your JS code and notebook very soon to investigate this thoroughly.

In the meantime, I have a quick suggestion that might help us isolate the problem: before you go through the trouble of testing in the browser, you could try running a quick, simple test using Nanowakeword's own built-in NanoInterpreter. This will help us confirm if the model itself is working as expected, completely separate from any potential issues in the JavaScript implementation.

There is a simple, ready-to-use code example in the README.md file (in the "Using Your Trained Model" section) that shows exactly how to use the NanoInterpreter for this. It's a great way to quickly verify the model's basic performance.

And one last thing—your feedback and questions are incredibly valuable. If you'd like to have a more real-time conversation and a better back-and-forth experience, you are more than welcome to join the project's Discord server. Continuing here on GitHub is perfectly fine too, of course, but the option is there if you prefer.

• Discord

I'm on the case now and will get back to you as soon as I have a solution. Thank you for your patience!

Best regards,
Muhammad Abid

[JpEncausse]

I give a try on a v4 but got same bad results :

INFO: Training finished. Merging best checkpoints to create final model...
INFO: Averaging the top 5 most stable models found during training...
INFO: Calculating performance metrics for the final averaged model...
========================================
TRAINING COMPLETE - FINAL REPORT
========================================
INFO: NOTE: These metrics are indicators of model health, not real-world performance.
  Average Stable Loss      : 35.9007
  Weight Diversity (Std Dev): 0.6725
  Avg. Positive Confidence (Logit): -4.573
  Avg. Negative Confidence (Logit): -44.182

Generating training performance graph...
Performance graph saved to: /content/trained_models/reflets_v4/2_training_artifacts/graphs/training_performance_graph.png
INFO: Reconstructing model into a sequential format for export.
INFO: Saving inference-ready ONNX model to '/content/trained_models/reflets_v4/3_model/reflets_v4.onnx'
INFO: Using ONNX opset version: 17
INFO: ONNX model saved successfully.

What changed ?

• I keep config close to your getting starting model
• v3 and v4 use 3 voices both male and female. This lead to a confidence of 0
• v1 and v2 use 1 female voice. This lead between 0.35 and 0.2 (but very bad)
• In v4 I changed forbiden word to sounds like reflets but less close to reflets because I think it over learn stop word

"custom_negative_phrases": ["réflexe", "tout à fait", "en effet", "bref délai", "s'il vous plaît"],
"custom_negative_per_phrase": 15,
"adversarial_text_generation": True,

"BandStopFilter": 0.1

So I'm stuck, and it takes hours to test on GoogleColla :-) I don't know what to do since OpenWakeWorld do not work on Collab.
It's a SmartMirror personal project running on a Webpage, so I don't have Python environment, I use Google Collab as an accelerator (Well it was the idea before the week-end ;-))

But I think the ONNX and the JS code is OK, audio samples are good also. I think it's a training parameter issue. One of my last shot is to remove the negative words

[muhammadabidPro]

Hi,

Thank you for your persistence and for sharing these detailed logs. Please, do not get discouraged.

First, I have looked very closely at your v3/v4 logs, and I believe I have found the exact reason for the model collapse (Loss: 35.9, negative logits, etc.) But this score seems about right to me. In your configuration, I see a lot of training_refinement parameters. These are a set of highly experimental and sensitive features. When not configured correctly, they can unfortunately lead to exactly these kinds of poor results.

I have a specially tuned, "Hard Training" configuration for these experimental features. Please try replacing your entire training_refinement block with the following:

# config.yml - Hard Training Focused Configuration
training_refinement:
  triplet_settings:
    base_margin: 0.6
    curriculum_steps: 15000
    hard_mining_ratio: 0.8
    negative_push_margin: 1.0
  loss_settings:
    false_accept_penalty: 50.0
    miss_penalty: 5.0
    focal_gamma: 3.0
    temperature: 2.0
  margin_settings:
    safety_threshold: 0.02
    exponential_scale: 5.0
    confidence_penalty_weight: 100.0
  regularization:
    spectral_weight: 0.02
    enable_spectral: true
    confidence_regularization: 0.2
  loss_weights:
    triplet: 0.7
    bce: 0.2
    focal: 0.3
    margin: 150.0
    replay: 30.0
    negative_push: 50.0
    confidence_reg: 20.0
  optimization:
    max_grad_norm: 1.5
    adaptive_clipping: true
    aggressive_clip_factor: 0.5
  hard_negative_mining:
    enable: true
    bank_size: 1000
    mining_threshold: 0.4
    replay_batch_size: 32
    min_bank_size: 100
    focus_on_confusing: true

---

Now, I have some very important questions for you. This will help us understand the real problem.

My main question is: did you get bad results by actually testing the model in your real-life application, or are you saying this just by looking at the final report?

Please remember, the final report is not always accurate about the model's real-world performance.

A Few Other Critical Checks:

• A "Control Experiment": To be absolutely sure, I would suggest one control run. Please try using the default English TTS voices that come with Nanowakeword (without your custom French TTS models for now), but use my provided "Hard Training" config. This will help us confirm that the core training process is working perfectly.

• The transform_clips Flag: I was wondering, before every new training run, did you by any chance check if you had set transform_clips: false? For every new training run where you change the data (like adding new voices), it is essential to set transform_clips: true. If the features are not re-computed, your changes will have no effect.

• custom_negative_phrases: You should not remove your custom negative phrases! They are absolutely critical for teaching the model what not to listen to.

I know how frustrating it is to spend hours on a training run only to get a bad result. Your SmartMirror project sounds amazing, and I am determined to help you get a model that works perfectly for it.

[JpEncausse]

Thanks !

• I'll try your settings, when the v5 ends. (I give a try with no negative phrases)
• Transform_clips was always true I think
• I always test in a real life project with a good microphone, in silence then I'll test with the smartmirror (have a microphone less accurate and will be in real life sound). So I have bad results in real life seeing the score from my JS code.
• With "hey_jarvis" I have between 0.9607 or 0.6 or 0.2 it really depends of how I push my english accent. (I think it's too sensible compared to a Ok Google that works in any situation, any voices)

My last very very very ugly bet ... will be to call Google SpeechToText in Chrome then perform Fuzy String Match on pronounciacion. I did it in the past but that's ugly ;-)

[muhammadabidPro]

Hi,

I have one last, very important thing to tell you.

You're absolutely right, it's very sensitive, I never thought of it as something really useful.

I ran a quick training with a small dataset, and although the final report showed loss: 45, the model's performance was actually very good, but the model is really very sensitive.

========================================
TRAINING COMPLETE - FINAL REPORT
========================================
INFO: NOTE: These metrics are indicators of model health, not real-world performance.
  Average Stable Loss      : 45.0714
  Weight Diversity (Std Dev): 1.1353
  Avg. Positive Confidence (Logit): -26.674
  Avg. Negative Confidence (Logit): -91.913

Here was the config I used:

generate_positive_samples: 2300
generate_negative_samples: 4550

augmentation_rounds: 10
augmentation_batch_size: 16

augmentation_settings:
  BackgroundNoise: 0.8
  RIR: 0.6
  PitchShift: 0.5
  Gain: 1.0
  ColoredNoise: 0.4
  BandStopFilter: 0.2

model_type: "rnn"
layer_size: 256
n_blocks: 4
dropout_prob: 0.5
embedding_dim: 64

steps: 10000
batch_size: 128

optimizer_type: "adamw"
lr_scheduler_type: "onecycle"
learning_rate_max: 0.001
weight_decay: 0.01
feature_gen_cpu_ratio: 0.8

batch_composition:
  batch_size: 128
  source_distribution:
    positive: 30
    negative_speech: 30
    pure_noise: 40

checkpointing:
  enabled: true
  interval_steps: 1000
  limit: 5

early_stopping_patience: 0 
debug_mode: true          
train_model: true

training_refinement:
  # ... (The full 'Hard Training' block was here) ...

Looks terrible, right? A complete failure.

But... if you actually test the real .onnx model from this training in a live application, its score for silence is 0.0000, and for the wake word, it goes from 0.0 all the way up to 0.999+ (but it will never be exactly 1.0).

This brings me to my most important question for you, which I will ask again:

Have you actually run the model in your real-life application and confirmed that it is performing poorly?

The final report's numbers are often not an accurate representation of the final model's real-world performance. Please, test the exported .onnx file.

One last thing. You mentioned you didn't know if OpenWakeWord has a Colab notebook. I just wanted to let you know that yes, it does! You can find it on their GitHub repository if you're interested.

I know how frustrating this is. Your SmartMirror project sounds amazing, and I am determined to help you get a model that works perfectly for it.

[JpEncausse]

Have you actually run the model in your real-life application and confirmed that it is performing poorly?

Yes

• I run it into my SmartMirror JS Code,
• I say "Reflets how are you"
• And read the output JS score that tells me 0.00 (for hey Jarvis it moves from 0.9 to 0.2

OMG, sorry, I run more test well it's very scramble

• If I start with "Reflets" or "Reflets bla bla bla" I get something very close to 1
• If I say "Reflets_bla bla bla" or "bla bla_Reflets_bla bla" or "bla bla bla_Reflets" it switch to 0

From multiple test, it seems I have to say the word "reflets" away from the other word of the sentence. if it's too close it fail fast.

That's weird the almost same sentences is sometimes to 1 sometimes to 0. I don't think it is related to my code/buffer it's more about marking "Reflets... How are you"

[muhammadabidPro]

I’ve understood a little of what you are trying to say…

Okay, can you give me your model here so I can take a quick look?

---

Another thing:

• If your wakeword is “triflet”, then just saying “triflet” will give a higher score.
• Where are you keeping the detection score? Like 0.9999.
• If you say “triflet bla bla”, then the model will hear “triflet” faster because it’s an RNN architecture—I understand this from your notebook—so it will treat that as the wakeword.
• But when you say “bla bla triflet” at the same time, it will lower the score because, according to the RNN architecture, it will hear something else… still, it might give a higher score.

Another point:

• Current models are trained to be confident, meaning they cannot just say yes or no like 0.000 or 1 It won’t be like that.

[JpEncausse]

Yes the confidence I totally agree it's never 0 or 1

1. My main usecase is like "Ok Google + sentence"
2. But in french you could say "I have a question Ok Google can you do ..."
3. Or you could say "Switch of the light Ok Google"

What I understand is that the RNN is better with 1. ?

• So in the notebook I train "Reflets" like "Ok Google"
• And in the JS I have VAD that outbut an audio buffer on speechend I pass it to the model. So it's always a sentence. Not a keyword like "Switch lights"
• Then if confidence is > let's say 0.6 then the audio is sent to the cloud to perform Speech2Text then call an LLM to some work and reply back the answer in audio

May be my usecase is not your intended usecase ?
If it's 1. it's ok for me and it seems accroding to my test more accurate like you said when starting with the keyword.

[muhammadabidPro]

Hey @JpEncausse,

This is an absolutely brilliant question, and you've just uncovered the most important distinction in how wake word engines work! Your thinking process is perfect.

Let me answer your core question directly:

"May be my usecase is not your intended usecase ?"

The great news is, your use case is exactly the intended use case for Nanowakeword! You want to spot a keyword in a stream of audio, just like "Ok Google."

However, you've discovered a fascinating and critical insight: the method you are using to check for the wake word makes a huge difference, and some model architectures are naturally better at it than others.

The "Detective" vs. The "Door Guard"

You've essentially discovered two different ways to solve this problem:

1. The "Detective" Approach (Your current method): You are using a VAD to record a full sentence, and then after the person stops speaking (on speechend), you give the entire audio clip to the model and ask, "Was the wake word somewhere in this recording?"

2. The "Door Guard" Approach: This is how "Ok Google" and "Alexa" work. The model acts like a security guard, listening to a continuous stream of small audio chunks. Its only job is to ask, "Is the wake word being spoken right now?"

Your Critical Insight:

You correctly observed that the RNN model performed much better with your "Detective" approach. And now you might have a clue as to why.

You asked: "What I understand is that the RNN is better with 1. [The Detective approach]?"

YES! Your intuition is 100% correct! And you've stumbled upon the reason. RNN, LSTM, and GRU models are designed with a "memory" (a hidden state). This gives them a unique ability to process a longer sequence of information and understand the context over time. They are naturally better at "searching" through a sentence.

Other models, like TCN or CNN, are more like "sprinters." They are incredibly fast and good at recognizing immediate patterns, but they don't have the same kind of long-term memory.

The models in Nanowakeword are trained on short, 2-second audio clips. When you feed them a long, 10-second sentence, the extra words before and after the wake word act as "noise" and confuse the model. This is almost certainly why you're seeing the score "spike" but failing to trigger reliably—the model isn't designed to search through a long recording.

You haven't been doing anything wrong. You've been conducting a brilliant experiment and have discovered a fundamental principle of audio model architecture. The fact that you figured this out through your own testing is truly impressive.

This is a huge breakthrough! You've just understood the core architectural philosophy of modern wake word engines, and you're well on your way to building the perfect model for your use case. This is very exciting

[JpEncausse]

Thanks for you answer !
So I made huge change into my JS code, after hours of test & learn it works.

In a nutshell

• on VAD speech start I build a Buffer that I reset at max 5s because TV AlwayOn can fill infinite buffer
• I forward sub-buffer to nanowakeword to test keyword
• if the threashold > 0.75 then I trigger VAD SpeechEnd
• Then I send a sub buffer of frames from nanowakeword to the end (yes if keyword is in the middle of the buffer it do not start with "reflets" but thats ok)

I have an optional process

• on VAD Start I start Chrome SpeechEngine, always perform speech2text
• If I match indexOf("reflets") then I also trigger the endVad etc ...
• this time the full Chrome Speech2Text is sent

Why ? optional process

• Main behavior send the audio when it really matter and call an API server side of speech2text
• Optional behavior
  => has already done the speech2text so it could be faster.
  => and can perform a weak keyword spotting aside of nanowakeword (double chance to match)

When sould I activate optional process ?
I have to test it ... I think if network is weak, google contious speech will have lots of failure

[muhammadabidPro]

Wow, @JpEncausse! After reading through your entire message, it sounds like you've done it... you've cracked it!

And honestly, what you've built is absolutely incredible. You are definitely not a regular user, haha—you've created a really sophisticated, hybrid system with both Nanowakeword and the Chrome Speech Engine running in parallel. That's seriously impressive work and a fantastic solution.

So, it sounds like the main Nanowakeword part is now working much more reliably with the streaming approach. How is the model behaving in this new setup? Are the false triggers gone? Is it responsive enough for your needs?

You also asked a great question about your optional "double chance" process:

When should I activate the optional process?

Your own theory is spot on:

"I think if network is weak, google contious speech will have lots of failure"

That's exactly it. The Chrome Speech Engine is cloud-based, so it's entirely dependent on a stable internet connection. Nanowakeword, being fully offline, will work perfectly even if the network is slow or completely disconnected. So, a great strategy would be:

• Primary Mode (Good Network): Run both systems in parallel for maximum speed and a "double chance" of detection.
• Fallback Mode (Weak/No Network): Rely only on the Nanowakeword model, as it will be far more reliable.

---

I just had one more thought about your brilliant dual-system setup. It's a super clever way to get a "double chance" of detection, and it's great that it's working for you right now with the word "Reflets."

I just wanted to give you a heads-up about a potential challenge you might face in the future if you ever decide to change your wake word to something more unique.

Cloud-based Speech-to-Text engines like Google's are amazing with common, everyday words. However, they can sometimes struggle with unusual or custom words that aren't in their main dictionary. For example, if your wake word was a unique project name like "Zylith" or a fantasy name like "Nylia," the speech-to-text might transcribe it as "Zephyr" or "Nadia." This would cause your indexOf() check to fail.
But there are some ways to fix it like re libary

Nanowakeword, on the other hand, doesn't care about the meaning or spelling of the word—it only cares about the sound. This is its biggest advantage. It can learn any custom word, in any language, with equal effectiveness.

It's not a problem for you right now with "Reflets," but it's just something to keep in mind for the future. Your Nanowakeword model will always be the most reliable and flexible part of your system for custom wake words.

---

It seems like you've engineered a truly robust solution for your SmartMirror. What kind of help do you need from me now? Or have you already solved all the hard problems yourself? 😄

This is a huge success. Congratulations

[JpEncausse]

So, it sounds like the main Nanowakeword part is now working much more reliably with the streaming approach.
How is the model behaving in this new setup? Are the false triggers gone? Is it responsive enough for your needs?

I'm still testing hard to tell the main usage must be [sound or speech or ... ] then [Silence] then [Wakeword] then [Some text]. Otherwise it's more complicated to trigger
=> That's why when I "catch" the wakeword i perform active listening for 60s.

The tricky part is to find that wakeword in a noisy environment and the silence seems to help a lot but I figureout with random tests.

I just wanted to give you a heads-up about a potential challenge you might face in the future
if you ever decide to change your wake word to something more unique.

You are right, in the past (I lost the code) I did "text to phonetics" then String.indexOf("phonetics") it helps a lot compared to matching a list of close words.

It seems like you've engineered a truly robust solution for your SmartMirror.
What kind of help do you need from me now? Or have you already solved all the hard problems yourself?

I think it's fine, I tested v4 and v6 and it works, I'll try to gather more data about what is working or not.
My target is to put the mirror in a bedroom or a retail store for instance so people speaking and music in background is a real issue :-) I need to do more and more testing. Especially with the mirror's microphone that underpeform ;-) A challenge !

I'll try to post a demo as soon as I have somethong clean.
Many Thanks !

(Oh BTW, out of this topics, I also look for KWS on ESP-32 with Arduino/PlatformIO (not the hell of ESP-IDF) ... and there is no easy solutions... so I push a button to speak https://www.youtube.com/watch?v=H84T8Khg7vw )