Implementing Speech To Text on a field

[Jacotheron]

Package

Form builder

Package Version

v3.3

How can we help you?

I would like to know if anyone have implemented a speech to text api with a simple TextInput or Textarea field?

In short I have a project where I need to be able to populate a field from speech. This will be used at a dance festival, as part of the judging, where the judges' comments needs to be recorded and transcribed for the dancer's final report. The easiest would obviously be to use such an api, and directly record the comments, have it transcribed in near real-time and saved.

I have attempted a few guides, including the "x-webkit-speech" attribute on the field (this did not work - it appears to be deprecated); also to run javascript on the input's suffix action (also did not work - the "onresult" never fired):

Action::make('speech-to-text')
                    ->icon('heroicon-s-microphone')
                    ->label('Speech')
                    ->action(function(self $livewire, TextInput $component) {

                        $js = <<<JS
const recognizer = new (window.SpeechRecognition ||
  window.webkitSpeechRecognition ||
  window.mozSpeechRecognition ||
  window.msSpeechRecognition)();

recognizer.lang = 'en-US';
recognizer.continuous = false;
recognizer.interimResults = false;
recognizer.onresult = function(event) {
  document.getElementById("{$component->getId()}").value = event.results[0][0].transcript;
};
recognizer.start();
JS;
                        
                        $livewire->js($js);
                    })

Ideally the field should have a button to start/stop the listening, as the above "continuous = false" part stops as soon as the person takes a breath (not ideal for the judge's comments;). In the above I just do not have an extra button to use to stop the listening, and making another livewire request just to once again get/execute some js is not practical.

Have anyone implemented something like this?
Are there perhaps browser agnostic speech to text APIs available to implement (that does not cost too much - in effect I handle this project as a charity project)?

Any help greatly appreciated.

[Jacotheron]

Ok, after many hours working at this, I actually have something that works for the most part. That was until I found a browser extension that allows this on any field, without needing all of this custom code. I am still posting the code, as a reference for anyone that needs to do it in their project (without being able to control the browser where it will be used from).

The first file is a Custom Form Field, extending the default TextInput (so that everything related to it, will still work; can just as well extend the textarea): SpeechToTextInput.php

<?php

namespace App\Forms\Components;

use Closure;
use Filament\Forms\Components\Actions\Action;
use Filament\Forms\Components\TextInput;

class SpeechToTextInput extends TextInput
{
    protected string $view = 'forms.components.speech-to-text-input';

    protected bool|Closure $speechToText = false;

    public function isSpeechToText(): static
    {
        $this->speechToText = true;

        return $this
            ->suffixActions([
                Action::make('speech-to-text')
                    ->icon('heroicon-s-microphone')
                    ->label('Speech')
                    ->color('success')
                    ->alpineClickHandler(fn () => 'startListening'),
            ]);
    }

    public function getSpeechToText(): bool
    {
        return (bool) $this->evaluate($this->speechToText);
    }
}

This file is meant to overwrite the $view property, introduce a new property speechToText, have a setup method to add the suffixAction (green microphone icon; and it calls an alpine handler to start/toggle the listening) and finally a getter for the new property.

Next I copied the standard field's view, but only modified a couple of lines (speech-to-text-input.blade.php):

    $isSpeechToText = $getSpeechToText();

    if ($isPasswordRevealable) {
        $xData = '{ isPasswordRevealed: false }';
    } elseif (count($extraAlpineAttributes) || filled($mask)) {
        $xData = '{}';
    } elseif ($isSpeechToText) {
        $xData = 'speech_to_text("' . $id . '")';
    } else {
        $xData = null;
    }

I added the elseif($isSpeechToText) and the $isSpeechToText = $getSpeechToText(); lines. In short this setup the x-data attribute for the wrapper, and passed the field $id to the 'speech_to_text' function/alpine component, which is handled in the 3rd segment.

Now for the javascript part. Since I already imported Alpine into my 'app.js' file (for plugins to use), I can simply add the alpine component into this file, before the Livewire.start() call, which will also initialize Alpine (if you are placing this code away from the Alpine start, you may need to place it inside an event listener for 'alpine:start'):

Alpine.data('speech_to_text', (statePath = '') => ({
    listening: false,
    recognition: null,
    statePath: statePath,
    startListening(){
        if(this.listening){
            this.listening = false;
            if(this.recognition){
                this.$el.setAttribute('style', '--c-300:var(--success-300);--c-400:var(--success-400);--c-500:var(--success-500);--c-600:var(--success-600);')
                this.recognition.stop();
            }
            return;
        }
        if(!this.recognition){
            window.SpeechRecognition = window.SpeechRecognition ||
                window.webkitSpeechRecognition ||
                window.mozSpeechRecognition ||
                window.msSpeechRecognition;
            if(!window.SpeechRecognition){
                alert('Your browser does not support speech recognition');
                this.listening = false;
                return;
            }
            this.recognition = new SpeechRecognition();
            this.recognition.continuous = true;
            this.recognition.interimResults = true;
            this.recognition.lang = 'en-US';
            this.recognition.addEventListener('result', (e) => {
                this.$wire.set(this.statePath, e.results[0][0].transcript);
            });
        }
        this.recognition.start();
        this.listening = true;
        this.$el.setAttribute('style', '--c-300:var(--danger-300);--c-400:var(--danger-400);--c-500:var(--danger-500);--c-600:var(--danger-600);')
    }
}));

This code simply setups the component, getting the id for the field that should receive the text. It created a couple of internal properties to keep track of everything and then define the method that will be called when the button is clicked/pressed. If the listener is running, we will stop the recognition service, and set the color of the icon to green, before returning. Otherwise we check if we already have a recognition service, and set it up if we don't have it (it will alert you if your browser does not support it). The settings for the recognition service can be adjusted as needed (continuous, InterimResults, language etc). It also setup the even handler for a result - in my case it simply replace the existing content with the result (append is also possible instead of set, but then interimResults should be off). Finally it starts the service, and change the icon to red (makes it easy to see that it is listening).

Once the js is compiled, with for example vite, you can then add the SpeechToTextInput to your forms, and to enable the functionality, just call the isSpeechToText() method on it. You can have multiple of these fields in the same form - each is handled individually without clashing or other issues (I had 15 in 1 form).

Note that currently this is tested in Google Chrome and works. Firefox does not work, and many others probably do not work - it uses the Google api/service. It also requires internet (thus not usable in an intranet/offline capacity).