Preserving audio pauses?

[icsy7867]

I would like to preserve audio pauses and generate new lines, or something similar so that the response looks a little prettier. Trying to find some more documentation to see if this is possible. Has anyone accomplished this?

[glangford]

Look at the .json output using --word_timestamps True --output_format json.

Adjacent words look like this example:

                {
                    "word": " talent",
                    "start": 31.44,
                    "end": 31.72,
                    "probability": 0.9252727627754211
                },
                {
                    "word": " for",
                    "start": 31.72,
                    "end": 31.92,
                    "probability": 0.8891927599906921
                },

Each word is individually timed, and from that timing you could decide to add new output (ellipsis, new line, etc) whenever there is a big enough gap between the end of one word and the start of another.

[icsy7867]

Thanks for the quick reply!

--word_timestamps True --output_format json

These are for the binary cli right?

What's the python code equivalent?

[glangford]

...you can generate a json from Python using eg.

    result = model.transcribe(file, word_timestamps=True, verbose=False)
    writer = get_writer("json", ".")
    writer(result, input_audio)

[ryanheise]

Just to add to @glangford 's comment above, the result object in code contains exactly the same content that gets written into the json file. Also, the writer function already contains some code to split a new line after a pause longer than a certain threshold, but that threshold is hard coded, so you could copy the existing writer code and modify its threshold to your liking, and then call your writer function in the last step instead of the default writer that is used above

[icsy7867]

This is great! Thank you

[glangford]

Good thought @ryanheise, I wasn't aware of the code that detects long pauses.

whisper/whisper/utils.py

Line 159 in ba3f3cd

long_pause = (

[icsy7867]

Got the writer working! Is there anyway to get the output text directly to a variable? Currently it saves a file. I could load this file, but would be nice if I could just get the writer text directly.

[glangford]

Currently it saves a file. I could load this file, but would be nice if I could just get the writer text directly.

The file creation and output comes from the base class __call__ override

whisper/whisper/utils.py

Line 91 in ba3f3cd

def __call__(

Note the call to write_result that it does

whisper/whisper/utils.py

Line 101 in ba3f3cd

self.write_result(result, file=f, options=options, **kwargs)

This class hierarchy overriding __call__ is unnecessary since you don't want the file or features of the base class. So you could create a version of the class that does not subclass from ResultWriter, and just create your own variation of write_result that returns a string. This deviates from the whisper idea of a writer but it sounds like that won't be a problem.