"Invalid 'audio_in': audio frame length is too long" on Raspberry Pi Zero

[albertreed]

Thanks so much for the great library. I'm running into a problem getting it working on a Raspberry Pi Zero W.

When running the mic-speaker example I'm seeing this error after it runs for a short moment. Nothing else is logged.

(With verbose:true passed into the record call)

Say something!
Recording with sample rate 16000...
Recording 65536 bytes
Recording 16384 bytes
Recording 16384 bytes
Conversation Error: { Error: 3 INVALID_ARGUMENT: Invalid 'audio_in': audio frame length is too long.
    at Object.exports.createStatusError (/home/pi/madison/gassist/node_modules/grpc/src/common.js:87:15)
    at ClientDuplexStream._emitStatusIfDone (/home/pi/madison/gassist/node_modules/grpc/src/client.js:235:26)
    at ClientDuplexStream._receiveStatus (/home/pi/madison/gassist/node_modules/grpc/src/client.js:213:8)
    at Object.onReceiveStatus (/home/pi/madison/gassist/node_modules/grpc/src/client_interceptors.js:1316:15)
    at InterceptingListener._callNext (/home/pi/madison/gassist/node_modules/grpc/src/client_interceptors.js:590:42)
    at InterceptingListener.onReceiveStatus (/home/pi/madison/gassist/node_modules/grpc/src/client_interceptors.js:640:8)
    at /home/pi/madison/gassist/node_modules/grpc/src/client_interceptors.js:1136:18
  code: 3,
  metadata: Metadata { _internal_repr: {} },
  details: 'Invalid \'audio_in\': audio frame length is too long.' }
Recording 16384 bytes
Recording 16384 bytes
Recording 4096 bytes
Recording 12288 bytes

• I've tested the code on a mac, works fine
• On device, aplay and arecord with no extra parameters work. I can create and playback a 16k audio file.
• The code below successfully records an audio file that when played back is reported as: "Playing WAVE 'test.wav' : Signed 16 bit Little Endian, Rate 16000 Hz, Mono" by aplay.

var record = require('node-record-lpcm16')
var fs = require('fs')

var file = fs.createWriteStream('test.wav', { encoding: 'binary' })

const mic = record.start();
mic.on('data', data => {
	file.write(data);
})

// Stop recording after three seconds
setTimeout(function () {
  record.stop()
}, 3000)

The error implies that the recroding rate isn't correct but aside from the threshold:0, my test app records exactly the same way and the file is reported with a correct frame length. I'm assuming google's "audio frame length" is referring to the same value as "Rate" within aplay.

I'm on 0.5.1

[albertreed]

Update: I just stumbled onto this - the frame length here is the size of the block being sent to google and, as it turns out, 64k bytes is too many. My poor Raspberry Pi Zero is too slow. If I throw out the large chunks it works.

I'll poke around with chopping up the requests into bite-sized chunks for google but if it's obvious to anyone how to do that I'm happy to help test on my end.

[endoplasmic]

I ran into the same issue when I first started working with the audio stuff and noticed they were fussy about the audio not being sent in real time.

I'm guessing you're right about the Zero being too slow :(

I'll keep this open if you want, but it seems like you may have to do some digging to get it sorted out.

[albertreed]

I'd imagine it could happen if there's any number of things that hitch the thread and that it could happen on any number of devices. I'll fix it up and submit a PR.

[albertreed]

PR here: #51

[albertreed]

I just experimented to figure out what the correct value for MAX_FRAME_LENGTH should be. I'm not super-familiar with gRPCs so maybe someone knows how to look it up to make sure the value there is indeed the exact right value. 32 * 1024 was too big.

[ItsWendell]

I made a converter to convert and downsample browser microphone audio to the right format for gRPC, I found that a buffer size of 4096 sample-frames is perfect or good enough.

https://github.com/WMisiedjan/google-assistant-desktop-client/blob/develop/src/renderer/assistant/microphone.js

[albertreed]

I remember reading somewhere that google specifically recommends not altering the raw audio.

You raise an interesting point, though. My PR assumes that the error is concerned with the number of bytes sent as audio_in as opposed to the number of frames. It'd be worth testing the change with a higher sampling rate before merging just to be sure that assumption is correct. I'll do that when I get some time.

[ItsWendell]

@albertreed fyi, I also commented in your PR regarding this issue, I'm not use we should do any processing within the library itself, and if we do we should separate it into another class or (make it optional) than forced on since it might break other applications.

#51

[endoplasmic]

Looking at the PR, I don't see anything that would break current functionality, but I'm surprised that locking the thread with 2 loops doesn't produce the same problem. I think you're right in that in the other end of Google sees too much data come in it freaks out. I'm going to do a few more tests to see if I can break it with the or not.

Fwiw this fix is the same sort of thing a game engine would do to accommodate slower machines for example and it shouldn't bust anything that a fast machine can do.

[albertreed]

If you're testing it I'd suggest throwing some different sampling rates at it.

...and adding proper automated testing is probably a good idea if you're going to be accepting changes from yahoos on the internet like myself. :-)

[AdamLundstrom]

have you guys found a solution to this?