AI-Thinker ES8388 as MP3 player with enabled microphones in RXTX_MODE, confusion with stereo / mono playback and recording

[mirkosertic]

Hi!

I am a little bit confused, so maybe someone can help me out with the following setup:

1. AI-Thinker AudioKit with ES8388, running an AudioPlayer witn a Helix MP3 decoder
2. The board is configured in RXTX_MODE mode
3. One mic is enabled with cfg.input_device = ADC_INPUT_LINE2

MP3 playback works without problems. As far as I understood is the AudioInfo of the board defined by the source data of the player, so for instance when playing a 44100hz 16bit stereo song the AudioBoardStream is set to this AudioInfo. So changing the sample rate with defferent songs / playbacks automatically changes the recording sampling rate, right?

What is more confusing here is the input_device. In my setup only one mic is active, but when playing a stereo song on the AudioPlayer I see two channels in the AudioInfo. debug So is the recording data stream in this setup now stereo or mono? I need to convert the recording data stream to a fixed sampling rate (13000hz) / 16 bit / 1 channel config, I know how to use the FormatConverterStream, but I am not sure what is the correct description of the source data in terms of channels etc. in my setup. When I pass in the AudioInfo from the board (with 2 channels) as "from" voices in the output of the format converter sound like mickey mouse, so far too high. So I think there is something wrong with either the channels or the sampling rate, and maybe also with my assumptions about all of this.

Another thing I have noted is that the microphone recording level is quite low. Is is possible to modify the microphone gain somehow?

Thank you all, I know this is either confusing, I am confused, or maybe both.

MIrko

[pschatzmann]

Yes the I2S is usually notified by audio format changes from the decoder. see this Wiki

As far as I understand,with your configuration both microphones are enabled. However you can force the system to use mono by setting channels = 1; Or you can leave them to stereo and at the output stage mix them down to 1 with a FormatConverterStream or better a ChannelFormatConverterStream.

The ES8388 driver supports to set the microphone gain with the setInputVolume() method.

Finally: since the input and output are using the same I2S interface, the data format settings are shared betwen them.

[mirkosertic]

Hey, thank you for your answer!

I tried to override the channel and initial configuration, but I get strange results.

Here is the summary of the kit initialization code:

AudioInfo initialinfo(16000, 1, 16);

MediaPlayerSource source(STARTFILEPATH, MP3_FILE, true);
AudioBoardStream kit(AudioKitEs8388V1);
MP3DecoderHelix decoder;
MediaPlayer player(source, kit, decoder);

  INFO_VAR("Configuring kit with Samplerate=%d, Channels=%d and Bits per sample=%d", initialinfo.sample_rate, initialinfo.channels, initialinfo.bits_per_sample);  

  // setup output
  auto cfg = kit.defaultConfig(RXTX_MODE);
  cfg.copyFrom(initialinfo);
  // sd_active is setting up SPI with the right SD pins by calling
  // SPI.begin(PIN_AUDIO_KIT_SD_CARD_CLK, PIN_AUDIO_KIT_SD_CARD_MISO, PIN_AUDIO_KIT_SD_CARD_MOSI, PIN_AUDIO_KIT_SD_CARD_CS);
  cfg.sd_active = false;              // We are running in SD 1bit mode, so no init needs to be done here!
  cfg.input_device = ADC_INPUT_LINE2; // input from microphone

  if (!kit.begin(cfg))
  {
    WARN("Could not start audio kit!");
    while(true);
  }
  kit.setVolume(0.7);

  // Set the microhone level a little bit louder...
  // This will change the gain of the microphone
  kit.board().getDriver()->setInputVolume(90);

  AudioInfo kitinfo = kit.audioInfo();
  AudioInfo kitoutinfo = kit.audioInfoOut();  
  INFO_VAR("Kit Input is running with Samplerate=%d, Channels=%d and Bits per sample=%d", kitinfo.sample_rate, kitinfo.channels, kitinfo.bits_per_sample);
  INFO_VAR("Kit Output is running with Samplerate=%d, Channels=%d and Bits per sample=%d", kitoutinfo.sample_rate, kitoutinfo.channels, kitoutinfo.bits_per_sample);  

Now I am getting the following log:

[INFO] 535 1 src/main.cpp:setup():147 - Configuring kit with Samplerate=16000, Channels=1 and Bits per sample=16
[I] AudioTypes.h : 128 - in: sample_rate: 16000 / channels: 1 / bits_per_sample: 16
[I] AudioTypes.h : 128 - out: sample_rate: 16000 / channels: 1 / bits_per_sample: 16
[I] AudioTypes.h : 128 -  sample_rate: 16000 / channels: 1 / bits_per_sample: 16
[I] I2SConfigESP32.h : 80 - rx/tx mode: RXTX_MODE
[I] I2SConfigESP32.h : 81 - port_no: 0
[I] I2SConfigESP32.h : 82 - is_master: Master
[I] I2SConfigESP32.h : 83 - sample rate: 16000
[I] I2SConfigESP32.h : 84 - bits per sample: 16
[I] I2SConfigESP32.h : 85 - number of channels: 1
[I] I2SConfigESP32.h : 86 - signal_type: Digital
[I] I2SConfigESP32.h : 88 - i2s_format: I2S_STD_FORMAT
[I] I2SConfigESP32.h : 90 - auto_clear: true
[I] I2SConfigESP32.h : 92 - use_apll: true
[I] I2SConfigESP32.h : 97 - buffer_count:6
[I] I2SConfigESP32.h : 98 - buffer_size:512
[I] I2SConfigESP32.h : 101 - pin_mck: 0
[I] I2SConfigESP32.h : 103 - pin_bck: 27
[I] I2SConfigESP32.h : 105 - pin_ws: 25
[I] I2SConfigESP32.h : 107 - pin_data: 26
[I] I2SConfigESP32.h : 109 - pin_data_rx: 35
[INFO] 661 1 src/main.cpp:setup():170 - Kit Input is running with Samplerate=44100, Channels=2 and Bits per sample=16
[INFO] 661 1 src/main.cpp:setup():171 - Kit Output is running with Samplerate=44100, Channels=2 and Bits per sample=16

So the I2S init seems to respect the 1-chanel configuration. But when I query the AudioKits audioInfo(), I am always getting back the default config, which is 44100hz/16bit/stereo.

I am also asking for Input Device ADC_INPUT_LINE2. This is one of the two on-board microphones, so the signal should only contain one channel, right? But why do I get two channels in the AudioInfo() ? ADC_INPUT_LINE1 seems to be the Line-In port, and ADC_INPUT_LINE3 the second microphone. I either want to get a true mono-signal as the result of the recording, or a stereo signal from both microphones (of this is even possible).

I also tried a FormatConverterStream, to sample the 44100/16/2 signal down to 13000/16/1, but I only get random noise as part of this process. So either I am missing some initialization code here or I don't know...

[pschatzmann]

This seems to be a bug: I committed a correction.
However I did not have the time yet to test it...

For the AudioKit ADC_INPUT_LINE2 is both microphones and ADC_INPUT_LINE1 aux input!

[pschatzmann]

What are you planning to do with the audio input?

A) Yes
B) i dont see any reason why this should not work.

Maybe you should process the input in a seperate task if the audio files are not wav

[mirkosertic]

I am currently working on a prototype of a voice satellite with an embedded MP3 player, using Home Assistant as the voice and conversation engine. The thing is, when I start a new voice recognition pipeline, the server requests a specific audio format, and the Audio Kit is also configured at a specific audio format, depending on the current playing source and initial configuration.

One unexpected thing is, better described by the application logs:

[INFO] 36644 0 src/voiceassistant.cpp:webSocketEvent():172 - Source is running with Samplerate=44100, Channels=2 and Bits per sample=16
[INFO] 36655 0 src/voiceassistant.cpp:webSocketEvent():173 - Needs to be resampled to Samplerate=16000, Channels=1 and Bits per sample=16

Source describes the current audioInfo() of the board, and the target sampling rate is what home assistant asks for. The funny part here is that if I use a FormatConverterStream to convert, I get a far too fast(high pitch) recording debug in home assistant. If I juse use a ChannelSelectorOutput to convert the source data to mono, I get a debug recording with the right pitch in home assistant, even if the sampling rates are different! I really cannot explain this.

About the crackling noise while recording and doing a playback. I could further track this down. As soon as I change the playback volume with player.setVolume() the noise starts. If I remove the initial volume configuration in the playback loop the recording results is fine, beside the mentioned problem from above.

So this does cause the recording noise:

player.setVolume(command.volume / 100.0);
player.setActive(false);

source.setPath(  .. new cd card path ... );
player.begin( .. new index to play..., true);
player.setActive(true);

and this works fine:

//player.setVolume(command.volume / 100.0);
player.setActive(false);

source.setPath(  .. new cd card path ... );
player.begin( .. new index to play..., true);
player.setActive(true);

[pschatzmann]

The overall process flow is unclear to me. How do you communicate with HA ?
I hope you were setting the log level to warning when testing the quality. Like described in the documentation a log level of Info can cause what you describe.

I am not aware of any issues with the FormatConverterStream. I suggest to add some additional asserts into your loop to check that all the formats are matching with the expected values. Even if the input format is changing (e.g. via notifications from the decoder), the defined output format should be fixed!

You can also set up a simple test case for the FormatConverterStream with a sine generator to confirm that there is no bug there

[mirkosertic]

Hi!

I found some issues in my code. The overall process of recording and sending data to Home Assistant is:

1. in main-loop, I collect data packets from the AudioKit by reading from it
2. the data packets are sent to a queue with xQueueSend, as I don't want to run the audio processing and HA communication in the mainloop.
3. Another task polls the queue, and takes the packets
4. Home Assistant has a websocket api to start Assist pipelines
5. When starting an Assist pipeline, various parameters must be passed in, for instance a sample rate
6. The pipeline starts a wake word detection. It returns some metadata, for instance a sampling rate, channel count and bit depth for the current stage
7. The pipeline now accepts binary audio data

So the buggy part was on my side between step 5, 6 and 7.

For some unknown or undocumented reason on Home Assistant side, the sampling rates on pipeline start and wake word detection start can be different. It is also not possible to add additional data like channel count and bit depth at pipeline start. When I start a pipeline with 44100hz sampling rate, I get at wake word detection metadata with sampling rate 16000hz, 1ch, 16bit back. When I now send binary data in this (16000hz) format to the pipeline, the recording gets corrupted. So there seems to be some kind of format conversion going on behind the scenes, but it only seems to respect the sampling rate, but not the channels or the bitness.

So overall I got it working, but I have to use a fixed sampling rate of 16000hz at pipeline start.

Thank you for your support!

[pschatzmann]

Some optimization proposal:

• use a BufferRTOS as copy target: resample the data to 1 channel 8000 samples per second
• Read from the BufferRTOS and reample to the rate required

For details see here

This is keeping the needed buffer data amount low and you can work with a fixed sample rate in the queue.

[pschatzmann]

You can easily test the microphons with this sketch