Question about End-to-End (waveform → score) export and Android feature matching

[alipas777-afk]

Hi 👋
First of all, thank you for NanoWakeWord — it’s a very well-designed project.
The data generation, augmentation, and training pipeline work great and I’m getting very good accuracy in Python.

I’m currently trying to deploy a trained NanoWakeWord model on Android (ONNX Runtime, low-power SoC) and I’d like to clarify the intended deployment path.

My current understanding

From reading the code and issues, it seems that:

The exported wakeword.onnx does NOT accept raw waveform

The model expects precomputed features with shape similar to (1, 8, 96)

The actual wake-word system relies on:

streaming ring buffers

MelSpectrogram ONNX

embedding ONNX

temporal feature stacking

reset / state / patience logic

So NanoWakeWord is effectively:

PCM → streaming features → embedding → stacked features → wakeword model

and not an end-to-end waveform → score model.

The problem

On Android, if I directly feed:

raw PCM

or even a fixed 1.38s waveform window

into wakeword.onnx, I get:

constant zeros

or false positives

or input shape errors

Which makes sense if feature matching is required.

Questions

Is it intentionally not supported to export an end-to-end ONNX model (waveform → score)?

Is the recommended approach to:

re-implement the feature pipeline (AudioFeatures, streaming melspec, embedding, stacking) on the target platform?

Is there any official or planned:

E2E export

reference Android / Kotlin implementation

or feature-exact spec for third-party ports?

Context / Motivation

I fully understand why feature logic may not be embedded into ONNX (state, control flow, buffers).

I’m not asking for magic auto-export — just clarity on:

what is guaranteed to match training

what is expected from downstream deployments

NanoWakeWord’s data generation and training quality is excellent, and I want to keep using it rather than switching to a simpler but less accurate wake-word stack.

Thanks again for the great work,
and I’d really appreciate any guidance on the intended production deployment path 🙏

[muhammadabidPro]

Please wait a little longer, I will update this project very soon (3 hour), please do not use the present version. Then I will definitely reply to your original message.

[alipas777-afk]

Thanks for the recent changes around config and audio augmentation — I see that feature manifest and numpy feature extraction are being reworked as part of the update.

For Android deployment my key goal is an end-to-end model that accepts raw waveform and produces a wake score, without requiring external Python preprocessing. It's important to match the exact training pipeline (frames, mel, embed, stack) inside the model or helper code that can be ported to Kotlin.

Could you clarify:

1. whether the upcoming export will include E2E ONNX/TFLite,
2. the intended input shape for the exported model,
3. whether audio preprocessing will be provided as part of the core package or remain external.

Thanks again!

[muhammadabidPro]

Hi @alipas777-afk,

Thank you so much for your detailed follow-up and for your patience. I was occupied over the last time and couldn't reply sooner. Your feedback is incredibly valuable, and you've pinpointed the exact areas that are crucial for moving NanoWakeWord from a great training tool to a production-ready system.

Let me address your questions directly. You are completely right in your analysis of the current deployment path.

The Power of the Feature Pipeline

You correctly observed that the feature generation process (MelSpectrogram ONNX + Embedding ONNX) seems complex. There's a very important reason for this design, and I'd like to share my philosophy on it.

NanoWakeWord uses a two-stage feature extraction process. First, a melspectrogram model converts the raw audio into a standard Mel-spectrogram. This spectrogram is then fed into a second, much more powerful speech embedding model, which is based on Google's advanced speech research.

The final features produced by this embedding model are incredibly rich and information-dense, far more so than the initial Mel-spectrogram alone. This high-quality feature representation is the secret to NanoWakeWord's excellent accuracy, as it captures the deep characteristics of speech required to distinguish between very similar-sounding words. This allows even simpler architectures like DNNs to be trained effectively and quickly.

While the initial melspectrogram can be generated in pure Python, matching the exact output of the specialized embedding model is the critical part, and any deviation here can significantly impact performance.

Your Goal: End-to-End (E2E) Deployment

Your main goal is an E2E model that takes a raw waveform and outputs a score. I want to be very clear: this is absolutely the right direction, and this is where the project is heading.

The current setup, with its separate feature models, can feel cumbersome for deployment on platforms like Android. I completely understand that.

Future Plans & Roadmap

To answer your direct questions:

1. Will future exports include E2E ONNX/TFLite?
   Yes, absolutely. I have a plan for this. My idea is to merge the weights of the embedding model directly into the main wake word model during the export process. This will create a single, unified .onnx file that handles the most complex part of the feature extraction internally. This will solve the "three-model problem" and remove the biggest deployment headache.

2. What will the input shape be?
   The final E2E model will accept a fixed-length raw audio waveform (e.g., 16000 samples for 1 second of audio at 16kHz). The exact length will be documented clearly with the release.

3. Will audio preprocessing be part of the core package?
   Yes. The goal is to make the final exported model as self-contained as possible. The most critical parts of the preprocessing (like the speech_embedding logic) will be embedded within the E2E model itself. Any remaining simple logic (like buffering) will be provided as a clear, minimal, and easy-to-port reference implementation in Python, which can then be translated to Kotlin/C++.

A Quick Note on Recent Changes

You may have noticed that I recently replaced the standard BCE loss with a custom loss function. You might see slightly more "unstable" or non-zero scores for negative samples compared to the 0.0000 you'd get from BCE. This is intentional and part of a strategy to create a more robust and natural decision boundary for the model, which will ultimately help reduce false acceptances.

I am actively working on these improvements. Making the deployment path seamless is my top priority right now.

Please feel free to ask any more questions you might have. I'll be happy to answer them.

Thanks again for your fantastic feedback and for using NanoWakeWord.

[alipas777-afk]

Thanks a lot for the detailed explanation — this clarifies everything.

It’s great to hear that an E2E export (waveform → score) with embedded speech_embedding is on the roadmap. That will completely solve the deployment issues on Android and embedded targets.

Until then, I’ll continue using NanoWakeWord primarily for data generation and training, and I’ll hold off on rewriting the feature pipeline on-device.

Looking forward to the E2E release — this will be a big step forward for the project.

[muhammadabidPro]

Community 💭

If you have Discord, Don't forget to join our Discord server.
You can ask questions, share feedback, or discuss anything there at any time.

[mohammadd-dev]

Hello, is e2e version not ready yet?

[muhammadabidPro]

@mohammadd-dev

Oh shoot, seeing your message reminded me… I had written the code to build the E2E model. I was planning to improve the accuracy further, but I had stopped working on it for quite a while. Then one day I reset my Windows, and everything got deleted. At that time, I didn’t even remember about my E2E work.

Sorry😞, I’ll have to start the work again from the beginning. I didn’t continue with it because I was focusing on improving the accuracy of the wake word model.

[mohammadd-dev]

Hello, is e2e version not ready yet?

[muhammadabidPro]

Yes, I actually built it quite a while ago. 😄 However, I noticed one issue: training a model takes quite a lot of time, so I was trying to find ways to reduce the training time as much as possible.

I don’t remember exactly what all the issues were with the E2E setup anymore. I did run some smaller training experiments, and the model was working quite well. Although, to be honest, I never trained a model at a large scale or for a production-level setup.

But after working on the same project for a long time, I eventually lost interest in continuing it. I also noticed that NanoWakeWord wasn’t getting much support or contribution from others. 😄 So I decided to put the project on hold for now.

If you’re interested, I can make it public, or I can also share it with you privately. 🙂

[muhammadabidPro]

Hello @mohammadd-dev, @alipas777-afk

🚀 Major Update: End-to-End (E2E) Model Training

Here: Embedding vs E2E