VAD not functional with `whisper_full_with_state`

[tazz4843]

I ran into this issue while debugging whisper-rs not having any sort of VAD working. It turns out whisper.cpp just doesn't run VAD when using the state variants of whisper_full. Was there a reason for this?

See

whisper.cpp/src/whisper.cpp

Lines 7722 to 7743 in 7745fcf

	int whisper_full(
	struct whisper_context * ctx,
	struct whisper_full_params params,
	const float * samples,
	int n_samples) {
	std::vector<float> vad_samples;
	if (params.vad) {
	WHISPER_LOG_INFO("%s: VAD is enabled, processing speech segments only\n", __func__);
	if (!whisper_vad(ctx, ctx->state, params, samples, n_samples, vad_samples)) {
	WHISPER_LOG_ERROR("%s: failed to compute VAD\n", __func__);
	return -1;
	}
	if (vad_samples.empty()) {
	ctx->state->result_all.clear();
	return 0;
	}
	samples = vad_samples.data();
	n_samples = vad_samples.size();
	}
	return whisper_full_with_state(ctx, ctx->state, params, samples, n_samples);
	}

and note how it calls VAD, then defers to the state function, while the state function (

whisper.cpp/src/whisper.cpp

Lines 6804 to 6809 in 7745fcf

	int whisper_full_with_state(
	struct whisper_context * ctx,
	struct whisper_state * state,
	struct whisper_full_params params,
	const float * samples,
	int n_samples) {

) has no VAD code anywhere in itself.

Also see https://codeberg.org/tazz4843/whisper-rs/issues/242

[danbev]

Calling whisper_full_with_state makes the assumption that the state has already been populated and that you want to use it. Functions like whisper_full, and whisper_full_parallel call whisper_vad which updates the state before calling whisper_full_with_state.

The comment in whisper-state.rs is not quite accurate I think:

    /// # C++ equivalent
   /// `int whisper_full(struct whisper_context * ctx, struct whisper_full_params params, const float * samples, int n_samples)`
    pub fn full(&mut self, params: FullParams, data: &[f32]) -> Result<c_int, WhisperError> {
        ...
         let ret = unsafe {
            whisper_rs_sys::whisper_full_with_state(
                self.ctx.ctx,
                self.ptr,
                params.fp,
                data.as_ptr(),
                data.len() as c_int,
            )
        };
        ...
}

To be like whisper_full this would need to call whisper_vad and pass in the state, or perhaps call whisper_full instead if that is possible.

[tazz4843]

As far as I can tell whisper_vad is inaccessible from C FFI (or at least not exposed from whisper.h) so that's not an option, nor is using whisper_full at least for how this API is designed (it's much easier to use a state everywhere instead of having the two options for Rust consumers).

That inaccurate comment seems to be brought over from back when the state API wasn't a thing, thanks for catching it :)

[mdestagnol]

Isn't whisper_full supposed to be a convenience method for whisper_full_with_state? In my opinion it doesn't make sense to have the 2 methods diverge in their functionalities.

Using whisper_full_with_state is a pretty major use case when we want to load in memory one model but process multiple streams in parallel. Each stream needs its own state.

I would suggest to move the vad logic within the whisper_full method directly. If that's not possible, could we make the whisper_vad method available in the whisper.h?

[mdestagnol]

Adding a little PR for this issue: #3423 – curious to hear your thoughts @danbev

[danbev]

@mdestagnol Thanks for that, I'll try to take a closer look at this during the weekend.